From patchwork Thu Mar 30 10:21:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kai Huang X-Patchwork-Id: 9653627 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CFB1B60349 for ; Thu, 30 Mar 2017 10:22:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8B9228531 for ; Thu, 30 Mar 2017 10:22:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AD3902853B; Thu, 30 Mar 2017 10:22:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, T_DKIM_INVALID autolearn=no version=3.3.1 Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 21CAC28531 for ; Thu, 30 Mar 2017 10:22:58 +0000 (UTC) Received: from [127.0.0.1] (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id C445F21DFA8FE; Thu, 30 Mar 2017 03:22:58 -0700 (PDT) X-Original-To: intel-sgx-kernel-dev@lists.01.org Delivered-To: intel-sgx-kernel-dev@lists.01.org Received: from mail-pg0-x242.google.com (mail-pg0-x242.google.com [IPv6:2607:f8b0:400e:c05::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 1B0B021DFA8FE for ; Thu, 30 Mar 2017 03:22:57 -0700 (PDT) Received: by mail-pg0-x242.google.com with SMTP id 81so8965312pgh.3 for ; Thu, 30 Mar 2017 03:22:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=RmN0+9fD1ZrFOZV/u8hVXZRbfFCidOef5qKAS0gHads=; b=K4zFS0worACYRdfC3KN8tM7rN0vpeIAUeqi2knX6d27or+XUSzUG6ndcv12bWTje+c gN2/N57bL9Wk0NFzN+LHAV9WNnSzf9GMgsKtQXfB6fgCyLeUUlkkcajTvXt6SfmtDhF8 tWGdSNPSNl49gsqZwfSY1OoR6PEhaWkgkkPVd2N9bVylqyuFiuK4Ljf/0phm30hJ1+FT pk8eBRkIbmBZHKCKiLRLgCK9lo8VLXcDu/rEwHWy9AOrzGOfNRfKJqPGlT9Nzxsly5j0 o6fMDUALjgqSoSZN1BhfJCYOf1ewYPDDd678W6BhHISgd0hFYoMpHLNL+y1pfeeUMsyy g73w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=RmN0+9fD1ZrFOZV/u8hVXZRbfFCidOef5qKAS0gHads=; b=inU5DWhJHDnnFFyJl7s9MMDCA2h8JuK5p53sSGGnXLTb6sekRJxI2kxKX4OmQEsnfm mV+T7mOKAaz5v38thP36/NCw4//NGukKFW8Drj+KKOhKQxVfFW//VuL5qx4wXDoI+ftr LlCMSKzWV42PwhCz0qzYCub+rvWxgeSIi7jJl4xnydKwsdM0lprsDiL09kLC5nIazbpH 2pazfmPuNksIB/42FsGVJ8VExm6LKTT+jpO3baPwN8LIz3K8ivYPomWYZ5gnxkCz3pBw zFGKaOXiHQ+hXJNAb3s71kLlOG4bwz9e8cbKcOGmb5TmLM0HQWAWC8+rJBqArTDAwgGP TheQ== X-Gm-Message-State: AFeK/H0vDbH1vl+z+jJSAuNeWwE5fy3RABapvKhflA10QHbiatvSRSkO2CsWh+CnXv8x7w== X-Received: by 10.84.143.1 with SMTP id 1mr6206459ply.70.1490869376720; Thu, 30 Mar 2017 03:22:56 -0700 (PDT) Received: from localhost.localdomain (118-92-235-112.dsl.dyn.ihug.co.nz. [118.92.235.112]) by smtp.gmail.com with ESMTPSA id h71sm3584464pfj.65.2017.03.30.03.22.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Mar 2017 03:22:56 -0700 (PDT) From: Kai Huang X-Google-Original-From: Kai Huang To: intel-sgx-kernel-dev@lists.01.org Date: Thu, 30 Mar 2017 23:21:27 +1300 Message-Id: <20170330102127.21672-1-kai.huang@linux.intel.com> X-Mailer: git-send-email 2.9.3 Subject: [intel-sgx-kernel-dev] [PATCH] intel_sgx: refine page cache initialization for multiple EPC banks X-BeenThere: intel-sgx-kernel-dev@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Project: Intel® Software Guard Extensions for Linux*: https://01.org/intel-software-guard-extensions" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: jarkko.sakkinen@intel.com MIME-Version: 1.0 Errors-To: intel-sgx-kernel-dev-bounces@lists.01.org Sender: "intel-sgx-kernel-dev" X-Virus-Scanned: ClamAV using ClamSMTP Currently sgx_page_cache_init is called for each EPC bank in case of multiple EPC banks, which is wrong because sgx_page_cache_init creates ksgxswapd kernel thread which should only be created once in global. This patch refines page cache initlization code by introducing sgx_epc_bank_init which initializes one EPC bank, and sgx_page_cache_init is changed to take all EPC banks as parameter and initialize them one by one by calling sgx_epc_bank_init. And ksgxswapd will be created once in sgx_page_cache_init. ioremap for EPC bank is also moved to sgx_epc_bank_init. This is more reasonable, as EPC page's virtual address handling is also part of page cache management code. Signed-off-by: Kai Huang --- drivers/platform/x86/intel_sgx.h | 4 +- drivers/platform/x86/intel_sgx_main.c | 40 +++------------- drivers/platform/x86/intel_sgx_page_cache.c | 71 ++++++++++++++++++++++++++--- 3 files changed, 73 insertions(+), 42 deletions(-) diff --git a/drivers/platform/x86/intel_sgx.h b/drivers/platform/x86/intel_sgx.h index adb5b17..a98915a 100644 --- a/drivers/platform/x86/intel_sgx.h +++ b/drivers/platform/x86/intel_sgx.h @@ -252,8 +252,8 @@ enum sgx_free_flags { }; int ksgxswapd(void *p); -int sgx_page_cache_init(resource_size_t start, unsigned long size); -void sgx_page_cache_teardown(void); +int sgx_page_cache_init(struct sgx_epc_bank *banks, int nr_banks); +void sgx_page_cache_teardown(struct sgx_epc_bank *banks, int nr_banks); struct sgx_epc_page *sgx_alloc_page(struct sgx_tgid_ctx *tgid_epc_cnt, unsigned int flags); int sgx_free_page(struct sgx_epc_page *entry, struct sgx_encl *encl, diff --git a/drivers/platform/x86/intel_sgx_main.c b/drivers/platform/x86/intel_sgx_main.c index 6c2d240..e242969 100644 --- a/drivers/platform/x86/intel_sgx_main.c +++ b/drivers/platform/x86/intel_sgx_main.c @@ -264,7 +264,6 @@ static int sgx_dev_init(struct device *dev) { unsigned int wq_flags; int ret; - int i; pr_info("intel_sgx: " DRV_DESCRIPTION " v" DRV_VERSION "\n"); @@ -277,25 +276,9 @@ static int sgx_dev_init(struct device *dev) pr_info("intel_sgx: Number of EPCs %d\n", sgx_nr_epc_banks); - for (i = 0; i < sgx_nr_epc_banks; i++) { - pr_info("intel_sgx: EPC memory range 0x%lx-0x%lx\n", - sgx_epc_banks[i].start, sgx_epc_banks[i].end); -#ifdef CONFIG_X86_64 - sgx_epc_banks[i].mem = ioremap_cache(sgx_epc_banks[i].start, - sgx_epc_banks[i].end - sgx_epc_banks[i].start); - if (!sgx_epc_banks[i].mem) { - sgx_nr_epc_banks = i; - ret = -ENOMEM; - goto out_iounmap; - } -#endif - ret = sgx_page_cache_init(sgx_epc_banks[i].start, - sgx_epc_banks[i].end - sgx_epc_banks[i].start); - if (ret) { - sgx_nr_epc_banks = i+1; - goto out_iounmap; - } - } + ret = sgx_page_cache_init(sgx_epc_banks, sgx_nr_epc_banks); + if (ret) + return ret; wq_flags = WQ_UNBOUND | WQ_FREEZABLE; #ifdef WQ_NON_REENETRANT @@ -305,7 +288,7 @@ static int sgx_dev_init(struct device *dev) if (!sgx_add_page_wq) { pr_err("intel_sgx: alloc_workqueue() failed\n"); ret = -ENOMEM; - goto out_iounmap; + goto out_page_cache_init; } sgx_dev.parent = dev; @@ -318,11 +301,8 @@ static int sgx_dev_init(struct device *dev) return 0; out_workqueue: destroy_workqueue(sgx_add_page_wq); -out_iounmap: -#ifdef CONFIG_X86_64 - for (i = 0; i < sgx_nr_epc_banks; i++) - iounmap(sgx_epc_banks[i].mem); -#endif +out_page_cache_init: + sgx_page_cache_teardown(sgx_epc_banks, sgx_nr_epc_banks); return ret; } @@ -378,15 +358,9 @@ static int sgx_drv_probe(struct platform_device *pdev) static int sgx_drv_remove(struct platform_device *pdev) { - int i; - misc_deregister(&sgx_dev); destroy_workqueue(sgx_add_page_wq); -#ifdef CONFIG_X86_64 - for (i = 0; i < sgx_nr_epc_banks; i++) - iounmap(sgx_epc_banks[i].mem); -#endif - sgx_page_cache_teardown(); + sgx_page_cache_teardown(sgx_epc_banks, sgx_nr_epc_banks); return 0; } diff --git a/drivers/platform/x86/intel_sgx_page_cache.c b/drivers/platform/x86/intel_sgx_page_cache.c index 852066c..5d87666 100644 --- a/drivers/platform/x86/intel_sgx_page_cache.c +++ b/drivers/platform/x86/intel_sgx_page_cache.c @@ -420,17 +420,31 @@ int ksgxswapd(void *p) return 0; } -int sgx_page_cache_init(resource_size_t start, unsigned long size) +static int sgx_epc_bank_init(struct sgx_epc_bank *bank) { - unsigned long i; + unsigned long i, size; struct sgx_epc_page *new_epc_page, *entry; struct list_head *parser, *temp; + pr_info("intel_sgx: EPC memory range 0x%lx-0x%lx\n", + bank->start, bank->end); + + if (!bank->start || !bank->end) + return -ENODEV; + + size = bank->end - bank->start; + +#ifdef CONFIG_X86_64 + bank->mem = ioremap_cache(bank->start, size); + if (!bank->mem) + return -ENOMEM; +#endif + for (i = 0; i < size; i += PAGE_SIZE) { new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL); if (!new_epc_page) goto err_freelist; - new_epc_page->pa = start + i; + new_epc_page->pa = bank->start + i; spin_lock(&sgx_free_list_lock); list_add_tail(&new_epc_page->free_list, &sgx_free_list); @@ -439,10 +453,8 @@ int sgx_page_cache_init(resource_size_t start, unsigned long size) spin_unlock(&sgx_free_list_lock); } - sgx_nr_high_pages = 2 * sgx_nr_low_pages; - ksgxswapd_tsk = kthread_run(ksgxswapd, NULL, "ksgxswapd"); - return 0; + err_freelist: list_for_each_safe(parser, temp, &sgx_free_list) { spin_lock(&sgx_free_list_lock); @@ -451,17 +463,62 @@ int sgx_page_cache_init(resource_size_t start, unsigned long size) spin_unlock(&sgx_free_list_lock); kfree(entry); } +#ifdef CONFIG_X86_64 + iounmap(bank->mem); +#endif return -ENOMEM; } -void sgx_page_cache_teardown(void) +static void sgx_epc_bank_teardown(struct sgx_epc_bank *bank) +{ + if (!bank->start || !bank->end) + return; + + /* + * Don't free 'struct sgx_epc_page' for EPC page in this bank. All + * 'struct sgx_epc_page' for all EPC pages will be destroyed together. + */ +#ifdef CONFIG_X86_64 + if (bank->mem) + iounmap(bank->mem); +#endif +} + +int sgx_page_cache_init(struct sgx_epc_bank *banks, int nr_banks) +{ + int i, ret = -ENODEV; + + for (i = 0; i < nr_banks; i++) { + ret = sgx_epc_bank_init(banks + i); + if (ret) + break; + } + + if (!i) + return ret; + + /* + * We can continue to use EPC banks that have been initialized + * successfully, even we fail on initializing one particular EPC bank. + */ + sgx_nr_high_pages = 2 * sgx_nr_low_pages; + ksgxswapd_tsk = kthread_run(ksgxswapd, NULL, "ksgxswapd"); + + return 0; +} + +void sgx_page_cache_teardown(struct sgx_epc_bank *banks, int nr_banks) { struct sgx_epc_page *entry; struct list_head *parser, *temp; + int i; if (ksgxswapd_tsk) kthread_stop(ksgxswapd_tsk); + for (i = 0; i < nr_banks; i++) + sgx_epc_bank_teardown(banks + i); + spin_lock(&sgx_free_list_lock); list_for_each_safe(parser, temp, &sgx_free_list) { entry = list_entry(parser, struct sgx_epc_page, free_list);