From patchwork Tue Apr 4 07:29:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kai Huang X-Patchwork-Id: 9660899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A25416032D for ; Tue, 4 Apr 2017 07:29:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8B18128285 for ; Tue, 4 Apr 2017 07:29:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7FF7228488; Tue, 4 Apr 2017 07:29:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, T_DKIM_INVALID autolearn=no version=3.3.1 Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 21F3028285 for ; Tue, 4 Apr 2017 07:29:53 +0000 (UTC) Received: from [127.0.0.1] (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id EFA612193930D; Tue, 4 Apr 2017 00:29:53 -0700 (PDT) X-Original-To: intel-sgx-kernel-dev@lists.01.org Delivered-To: intel-sgx-kernel-dev@lists.01.org Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 2198A2193930D for ; Tue, 4 Apr 2017 00:29:53 -0700 (PDT) Received: by mail-pg0-x244.google.com with SMTP id 81so35401603pgh.3 for ; Tue, 04 Apr 2017 00:29:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=WKOeKdGi58BSD+cSIAjIuREDo3/Tbik83HB6RM9yeCQ=; b=muaY+Qxt8w1v1vCl+dGkrL2l67MBnA6Rs0M6JV6ZeIim6eulWqLLmvFo9Llyio79eb SgHrEboPmppsJtrvXysq1UPw2KH7qKKk25SMKIaj83OXBhA8eC7tn8TmNAJEW3aLYxaL FxsgOZFys0/oFHWVpBF5Ce8NtoBOBpj1Mi2nTLya7XisPhwWmNPx9ojsExov9cgsXDiD sduidkkR9GNZDZ0paUSvcPMmJroJTfUa+LW5HsR/a7zwmz7XmQQeo/uHnUpjPlER+T6/ w5D6ogyoDmlV4anJ3KaP6484Jv+z9oNj9CzAjCYwaiPXpsXvVPiw8yQgafEyGU9t1CMT Z8Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=WKOeKdGi58BSD+cSIAjIuREDo3/Tbik83HB6RM9yeCQ=; b=GiYJKG+EaWCteU84S6aUHgAkBuMYzV+FlpQdMQr7q772L5jcJLihy1Mvm2cPokqO2Y 0YzG+VfLdMIvgIQUdu/MIIMUEXeTnDAdzqW/3ozqr43RO+/4N0LTrXNmJ0r45Zm/Fc1v 4WiUdQwvtQjcZurrj6C5FVff0OPVAY26Vj88xVhiOajEVteXpyIkGrkzWmizZfzQk5dY hy8iY7RX5yX7UCP/oriE2NQz0Fhd26Rsl3YrgAUvOHCExdA/esrZi63U1nKeG5O8DOiu /JpJrKprWjJeFyjEFWHHPcqjVnqVL3VNPvdWvSb+oRmsTHyG+CcPe0f5TjlayNfENskl mzXA== X-Gm-Message-State: AFeK/H0kNul7IlS82fS2MIPTDKkLUA7XiqC6qjpFSaTseFSkCYM1LKuhidLcwi27YBLyvQ== X-Received: by 10.98.218.76 with SMTP id w12mr21457311pfl.162.1491290992773; Tue, 04 Apr 2017 00:29:52 -0700 (PDT) Received: from localhost.localdomain (118-92-235-112.dsl.dyn.ihug.co.nz. [118.92.235.112]) by smtp.gmail.com with ESMTPSA id y7sm29724831pfk.93.2017.04.04.00.29.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Apr 2017 00:29:52 -0700 (PDT) From: Kai Huang X-Google-Original-From: Kai Huang To: intel-sgx-kernel-dev@lists.01.org, jarkko.sakkinen@linux.intel.com Date: Tue, 4 Apr 2017 19:29:33 +1200 Message-Id: <20170404072938.4800-3-kai.huang@linux.intel.com> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20170404072938.4800-1-kai.huang@linux.intel.com> References: <20170404072938.4800-1-kai.huang@linux.intel.com> Subject: [intel-sgx-kernel-dev] [PATCH 2/7] intel_sgx: refine page cache initialization for multiple EPC banks X-BeenThere: intel-sgx-kernel-dev@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Project: Intel® Software Guard Extensions for Linux*: https://01.org/intel-software-guard-extensions" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-sgx-kernel-dev-bounces@lists.01.org Sender: "intel-sgx-kernel-dev" X-Virus-Scanned: ClamAV using ClamSMTP Currently sgx_page_cache_init is called for each EPC bank in case of multiple EPC banks, which is wrong because sgx_page_cache_init creates ksgxswapd kernel thread which should only be created once in global. This patch refines page cache initlization code by introducing sgx_epc_bank_init which initializes one EPC bank, and sgx_page_cache_init is changed to take all EPC banks as parameter and initialize them one by one by calling sgx_epc_bank_init. And ksgxswapd will be created once in sgx_page_cache_init. ioremap for EPC bank is also moved to sgx_epc_bank_init. This is more reasonable, as EPC page's virtual address handling is also part of page cache management code. Signed-off-by: Kai Huang --- drivers/platform/x86/intel_sgx.h | 4 +- drivers/platform/x86/intel_sgx_main.c | 40 +++------------- drivers/platform/x86/intel_sgx_page_cache.c | 71 ++++++++++++++++++++++++++--- 3 files changed, 73 insertions(+), 42 deletions(-) diff --git a/drivers/platform/x86/intel_sgx.h b/drivers/platform/x86/intel_sgx.h index adb5b17..a98915a 100644 --- a/drivers/platform/x86/intel_sgx.h +++ b/drivers/platform/x86/intel_sgx.h @@ -252,8 +252,8 @@ enum sgx_free_flags { }; int ksgxswapd(void *p); -int sgx_page_cache_init(resource_size_t start, unsigned long size); -void sgx_page_cache_teardown(void); +int sgx_page_cache_init(struct sgx_epc_bank *banks, int nr_banks); +void sgx_page_cache_teardown(struct sgx_epc_bank *banks, int nr_banks); struct sgx_epc_page *sgx_alloc_page(struct sgx_tgid_ctx *tgid_epc_cnt, unsigned int flags); int sgx_free_page(struct sgx_epc_page *entry, struct sgx_encl *encl, diff --git a/drivers/platform/x86/intel_sgx_main.c b/drivers/platform/x86/intel_sgx_main.c index 6c2d240..e242969 100644 --- a/drivers/platform/x86/intel_sgx_main.c +++ b/drivers/platform/x86/intel_sgx_main.c @@ -264,7 +264,6 @@ static int sgx_dev_init(struct device *dev) { unsigned int wq_flags; int ret; - int i; pr_info("intel_sgx: " DRV_DESCRIPTION " v" DRV_VERSION "\n"); @@ -277,25 +276,9 @@ static int sgx_dev_init(struct device *dev) pr_info("intel_sgx: Number of EPCs %d\n", sgx_nr_epc_banks); - for (i = 0; i < sgx_nr_epc_banks; i++) { - pr_info("intel_sgx: EPC memory range 0x%lx-0x%lx\n", - sgx_epc_banks[i].start, sgx_epc_banks[i].end); -#ifdef CONFIG_X86_64 - sgx_epc_banks[i].mem = ioremap_cache(sgx_epc_banks[i].start, - sgx_epc_banks[i].end - sgx_epc_banks[i].start); - if (!sgx_epc_banks[i].mem) { - sgx_nr_epc_banks = i; - ret = -ENOMEM; - goto out_iounmap; - } -#endif - ret = sgx_page_cache_init(sgx_epc_banks[i].start, - sgx_epc_banks[i].end - sgx_epc_banks[i].start); - if (ret) { - sgx_nr_epc_banks = i+1; - goto out_iounmap; - } - } + ret = sgx_page_cache_init(sgx_epc_banks, sgx_nr_epc_banks); + if (ret) + return ret; wq_flags = WQ_UNBOUND | WQ_FREEZABLE; #ifdef WQ_NON_REENETRANT @@ -305,7 +288,7 @@ static int sgx_dev_init(struct device *dev) if (!sgx_add_page_wq) { pr_err("intel_sgx: alloc_workqueue() failed\n"); ret = -ENOMEM; - goto out_iounmap; + goto out_page_cache_init; } sgx_dev.parent = dev; @@ -318,11 +301,8 @@ static int sgx_dev_init(struct device *dev) return 0; out_workqueue: destroy_workqueue(sgx_add_page_wq); -out_iounmap: -#ifdef CONFIG_X86_64 - for (i = 0; i < sgx_nr_epc_banks; i++) - iounmap(sgx_epc_banks[i].mem); -#endif +out_page_cache_init: + sgx_page_cache_teardown(sgx_epc_banks, sgx_nr_epc_banks); return ret; } @@ -378,15 +358,9 @@ static int sgx_drv_probe(struct platform_device *pdev) static int sgx_drv_remove(struct platform_device *pdev) { - int i; - misc_deregister(&sgx_dev); destroy_workqueue(sgx_add_page_wq); -#ifdef CONFIG_X86_64 - for (i = 0; i < sgx_nr_epc_banks; i++) - iounmap(sgx_epc_banks[i].mem); -#endif - sgx_page_cache_teardown(); + sgx_page_cache_teardown(sgx_epc_banks, sgx_nr_epc_banks); return 0; } diff --git a/drivers/platform/x86/intel_sgx_page_cache.c b/drivers/platform/x86/intel_sgx_page_cache.c index 852066c..2a277ba 100644 --- a/drivers/platform/x86/intel_sgx_page_cache.c +++ b/drivers/platform/x86/intel_sgx_page_cache.c @@ -420,17 +420,31 @@ int ksgxswapd(void *p) return 0; } -int sgx_page_cache_init(resource_size_t start, unsigned long size) +static int sgx_epc_bank_init(struct sgx_epc_bank *bank) { - unsigned long i; + unsigned long i, size; struct sgx_epc_page *new_epc_page, *entry; struct list_head *parser, *temp; + pr_info("intel_sgx: EPC memory range 0x%lx-0x%lx\n", + bank->start, bank->end); + + if (!bank->start || !bank->end) + return -ENODEV; + + size = bank->end - bank->start; + +#ifdef CONFIG_X86_64 + bank->mem = ioremap_cache(bank->start, size); + if (!bank->mem) + return -ENOMEM; +#endif + for (i = 0; i < size; i += PAGE_SIZE) { new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL); if (!new_epc_page) goto err_freelist; - new_epc_page->pa = start + i; + new_epc_page->pa = bank->start + i; spin_lock(&sgx_free_list_lock); list_add_tail(&new_epc_page->free_list, &sgx_free_list); @@ -439,10 +453,8 @@ int sgx_page_cache_init(resource_size_t start, unsigned long size) spin_unlock(&sgx_free_list_lock); } - sgx_nr_high_pages = 2 * sgx_nr_low_pages; - ksgxswapd_tsk = kthread_run(ksgxswapd, NULL, "ksgxswapd"); - return 0; + err_freelist: list_for_each_safe(parser, temp, &sgx_free_list) { spin_lock(&sgx_free_list_lock); @@ -451,13 +463,55 @@ int sgx_page_cache_init(resource_size_t start, unsigned long size) spin_unlock(&sgx_free_list_lock); kfree(entry); } +#ifdef CONFIG_X86_64 + iounmap(bank->mem); +#endif return -ENOMEM; } -void sgx_page_cache_teardown(void) +static void sgx_epc_bank_teardown(struct sgx_epc_bank *bank) +{ + if (!bank->start || !bank->end) + return; + + /* + * Don't free 'struct sgx_epc_page' for EPC page in this bank. All + * 'struct sgx_epc_page' for all EPC pages will be destroyed together. + */ +#ifdef CONFIG_X86_64 + if (bank->mem) + iounmap(bank->mem); +#endif +} + +int sgx_page_cache_init(struct sgx_epc_bank *banks, int nr_banks) +{ + int i, ret = -ENODEV; + + for (i = 0; i < nr_banks; i++) { + ret = sgx_epc_bank_init(banks + i); + if (ret) + break; + } + + if (!i) + return ret; + + /* + * We can continue to use EPC banks that have been initialized + * successfully, even we fail on initializing one particular EPC bank. + */ + sgx_nr_high_pages = 2 * sgx_nr_low_pages; + ksgxswapd_tsk = kthread_run(ksgxswapd, NULL, "ksgxswapd"); + + return 0; +} + +void sgx_page_cache_teardown(struct sgx_epc_bank *banks, int nr_banks) { struct sgx_epc_page *entry; struct list_head *parser, *temp; + int i; if (ksgxswapd_tsk) kthread_stop(ksgxswapd_tsk); @@ -469,6 +523,9 @@ void sgx_page_cache_teardown(void) kfree(entry); } spin_unlock(&sgx_free_list_lock); + + for (i = 0; i < nr_banks; i++) + sgx_epc_bank_teardown(banks + i); } static struct sgx_epc_page *sgx_alloc_page_fast(void)