From patchwork Sun Jul 9 08:09:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kai Huang X-Patchwork-Id: 9831685 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A1FA160318 for ; Sun, 9 Jul 2017 08:11:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 982AE24151 for ; Sun, 9 Jul 2017 08:11:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8CEF7237A5; Sun, 9 Jul 2017 08:11:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_MED, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BD6AF237A5 for ; Sun, 9 Jul 2017 08:11:47 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dU7Hf-0004dR-59; Sun, 09 Jul 2017 08:09:39 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dU7He-0004ct-9M for xen-devel@lists.xen.org; Sun, 09 Jul 2017 08:09:38 +0000 Received: from [85.158.137.68] by server-6.bemta-3.messagelabs.com id 2B/1E-02181-1C4E1695; Sun, 09 Jul 2017 08:09:37 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrAIsWRWlGSWpSXmKPExsXiVRvsrHvgSWK kwa6fWhZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8bGR70sBdeiKu78WM3UwHjKoYuRi0NIYCKj xKTVq5lBHBaBLlaJXWfus4I4EgLLWCWOtXUBZTiBnDiJ40//M0HYlRJTds0GiwsJKEt0fTvKD jFqEZPEjlNXWUESbAJqEluXtLNANNhKLDj3BaxBREBa4trny4wgNrOAh8THj0vBbGGBUIkN61 +zg9gsAqoS549uAovzCsRL7F27kg1ijrzErraLQPM5ODgFEiSaL0RA3BAv8eviT7ASTqBVX/Y vgLrNRuLXsfVMExiFFzAyrGLUKE4tKkst0jWy0EsqykzPKMlNzMzRNTQw1stNLS5OTE/NSUwq 1kvOz93ECAzRegYGxh2M7Sf8DjFKcjApifKK9SZECvEl5adUZiQWZ8QXleakFh9ilOHgUJLgn fw4MVJIsCg1PbUiLTMHGC0waQkOHiURXufpQGne4oLE3OLMdIjUKUZLjkkHtn9h4ljxdheQfD Xh/zcmIZa8/LxUKXHeVJB5AiANGaV5cONgEX2JUVZKmJeRgYFBiKcgtSg3swRV/hWjOAejkjB vIMgUnsy8Eritr4AOYgI6iK0uAeSgkkSElFQDY7ftZ5WyYKX6T0VKx9zWPbrgf9os8vjdJiM7 40YOccH55y7s+n4ncovnrScHNZnWuTg2R/ev9sy4wjfdMkQ8VHadbYdUSzxHKAejd2Wx46/Jn Xfz/h1ZIKGcaPiox+bohan9Vdq+TItsvKuOFPXtirI9cETsU0xZb4JtgOnhM5u1Tnie445XYi nOSDTUYi4qTgQAHp87l+MCAAA= X-Env-Sender: kaih.linux@gmail.com X-Msg-Ref: server-12.tower-31.messagelabs.com!1499587775!87780335!1 X-Originating-IP: [74.125.83.67] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.25; banners=-,-,- X-VirusChecked: Checked Received: (qmail 31441 invoked from network); 9 Jul 2017 08:09:36 -0000 Received: from mail-pg0-f67.google.com (HELO mail-pg0-f67.google.com) (74.125.83.67) by server-12.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 9 Jul 2017 08:09:36 -0000 Received: by mail-pg0-f67.google.com with SMTP id j186so9086760pge.1 for ; Sun, 09 Jul 2017 01:09:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=w0hiUrnBqrzlXlofVOds7McqgC+/fGWpJiVQVYN4ip8=; b=eKY5kusrTHNNlkL7FlgD+mY8TIwtGYvpXqpAP5YBLBWuFS3mOvJHM7X+CczO4AWWnB 9bJNkYT0gGYAo8C4a3U+WToVLIdPMFJB1xp6t09wCI9FK40fxIy0nEaeCl6YhJy2g827 9VNQAaawl4a1dSUcgPcTW9d2jwuJtcumnUUO7cIiqGmU1kQKmZ85ZoZVUE9/Iwioi6dK tLAsl5izqaVPK7xL46KxHzXzTx0HRV/KsC6o1bh3JTQpca7EDXOCbRDymUzbTEDLps39 qktdaAVksHrsDJGQB3HCQKxF57J6VqMwGfxR7cXXBaddEa8neNihxBGghqjMEQREBTXI Hp7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=w0hiUrnBqrzlXlofVOds7McqgC+/fGWpJiVQVYN4ip8=; b=itiHNJD2OtO1fZ8FTW0WiBt+EEOYC1Ud1jIjqeNPuSk3BenXCPNwrOJGlLIRvUSBPD 7I5gI6e4+QXdhNH6K3DyUHTwGB/1TOyZkLNuCemuX475nJj68UOQStpPmwn4AdNJBoJ4 5lBA9x0LhTdrSzkGWo5gcxksqGzCOILtD0yElhyXdf6vRCPe4dx9QvOSs3NldaXPDuR2 /ZmivgQ6rYzrOl1BFZUKCnHWrUjXituV+l+3FFWliz1+2AjUX7phzqKqFMsHou5cnfUa OFkAWP/vnSf59ZYt+rU4m37IEVqHT7CEctog/I+LhpvydtrbRXLYAYOTrZ9ess65fKSr iXNA== X-Gm-Message-State: AIVw113r/X2HSMonpBAKTAnxV2MBf17k0wkwcKADKM0zzlnuv8lP9MoV Bs62nk9vMDCrYOD+ X-Received: by 10.84.210.130 with SMTP id a2mr5705672pli.301.1499587774521; Sun, 09 Jul 2017 01:09:34 -0700 (PDT) Received: from localhost.localdomain (118-92-234-57.dsl.dyn.ihug.co.nz. [118.92.234.57]) by smtp.gmail.com with ESMTPSA id d70sm22148172pga.49.2017.07.09.01.09.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 09 Jul 2017 01:09:34 -0700 (PDT) From: Kai Huang X-Google-Original-From: Kai Huang To: xen-devel@lists.xen.org Date: Sun, 9 Jul 2017 20:09:06 +1200 Message-Id: X-Mailer: git-send-email 2.11.0 In-Reply-To: <4b8baf9779038897e6ba2ed4ac0a3e9663db2756.1499586046.git.kai.huang@linux.intel.com> References: <4b8baf9779038897e6ba2ed4ac0a3e9663db2756.1499586046.git.kai.huang@linux.intel.com> In-Reply-To: References: Cc: andrew.cooper3@citrix.com, kevin.tian@intel.com, jbeulich@suse.com Subject: [Xen-devel] [PATCH 07/15] xen: x86: add functions to populate and destroy EPC for domain X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Add per-domain structure to store SGX per-domain info. Currently only domain's EPC base and size are stored. Also add new functions for further use: - hvm_populate_epc # populate EPC when EPC base & size are notified. - hvm_reset_epc # Reset domain's EPC to be invalid. Used when domain goes to S3-S5, or being destroyed. - hvm_destroy_epc # destroy and free domain's EPC. Signed-off-by: Kai Huang --- xen/arch/x86/hvm/vmx/sgx.c | 315 +++++++++++++++++++++++++++++++++++++ xen/arch/x86/hvm/vmx/vmx.c | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 14 ++ xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + 4 files changed, 334 insertions(+) diff --git a/xen/arch/x86/hvm/vmx/sgx.c b/xen/arch/x86/hvm/vmx/sgx.c index f4c9b2f933..14379151e8 100644 --- a/xen/arch/x86/hvm/vmx/sgx.c +++ b/xen/arch/x86/hvm/vmx/sgx.c @@ -9,6 +9,8 @@ #include #include #include +#include +#include #include #include @@ -90,6 +92,319 @@ void unmap_epc_page(void *addr) /* Nothing */ } +/* ENCLS opcode */ +#define ENCLS .byte 0x0f, 0x01, 0xcf + +/* + * ENCLS leaf functions + * + * However currently we only needs EREMOVE.. + */ +enum { + ECREATE = 0x0, + EADD = 0x1, + EINIT = 0x2, + EREMOVE = 0x3, + EDGBRD = 0x4, + EDGBWR = 0x5, + EEXTEND = 0x6, + ELDU = 0x8, + EBLOCK = 0x9, + EPA = 0xA, + EWB = 0xB, + ETRACK = 0xC, + EAUG = 0xD, + EMODPR = 0xE, + EMODT = 0xF, +}; + +/* + * ENCLS error code + * + * Currently we only need SGX_CHILD_PRESENT + */ +#define SGX_CHILD_PRESENT 13 + +static inline int __encls(unsigned long rax, unsigned long rbx, + unsigned long rcx, unsigned long rdx) +{ + int ret; + + asm volatile ( "ENCLS;\n\t" + : "=a" (ret) + : "a" (rax), "b" (rbx), "c" (rcx), "d" (rdx) + : "memory", "cc"); + + return ret; +} + +static inline int __eremove(void *epc) +{ + unsigned long rbx = 0, rdx = 0; + + return __encls(EREMOVE, rbx, (unsigned long)epc, rdx); +} + +static int sgx_eremove(struct epc_page *epg) +{ + void *addr = map_epc_page_to_xen(epg); + int ret; + + BUG_ON(!addr); + + ret = __eremove(addr); + + unmap_epc_page(addr); + + return ret; +} + +/* + * Reset domain's EPC with EREMOVE. free_epc indicates whether to free EPC + * pages during reset. This will be called when domain goes into S3-S5 state + * (with free_epc being false), and when domain is destroyed (with free_epc + * being true). + * + * It is possible that EREMOVE will be called for SECS when it still has + * children present, in which case SGX_CHILD_PRESENT will be returned. In this + * case, SECS page is kept to a tmp list and after all EPC pages have been + * called with EREMOVE, we call EREMOVE for all the SECS pages again, and this + * time SGX_CHILD_PRESENT should never occur as all children should have been + * removed. + * + * If unexpected error returned by EREMOVE, it means the EPC page becomes + * abnormal, so it will not be freed even free_epc is true, as further use of + * this EPC can cause unexpected error, potentially damaging other domains. + */ +static int __hvm_reset_epc(struct domain *d, unsigned long epc_base_pfn, + unsigned long epc_npages, bool_t free_epc) +{ + struct list_head secs_list; + struct list_head *p, *tmp; + unsigned long i; + int ret = 0; + + INIT_LIST_HEAD(&secs_list); + + for ( i = 0; i < epc_npages; i++ ) + { + struct epc_page *epg; + unsigned long gfn; + mfn_t mfn; + p2m_type_t t; + int r; + + gfn = i + epc_base_pfn; + mfn = get_gfn_query(d, gfn, &t); + if ( unlikely(mfn_eq(mfn, INVALID_MFN)) ) + { + printk("Domain %d: Reset EPC error: invalid MFN for gfn 0x%lx\n", + d->domain_id, gfn); + put_gfn(d, gfn); + ret = -EFAULT; + continue; + } + + if ( unlikely(!p2m_is_epc(t)) ) + { + printk("Domain %d: Reset EPC error: (gfn 0x%lx, mfn 0x%lx): " + "is not p2m_epc.\n", d->domain_id, gfn, mfn_x(mfn)); + put_gfn(d, gfn); + ret = -EFAULT; + continue; + } + + put_gfn(d, gfn); + + epg = epc_mfn_to_page(mfn_x(mfn)); + + /* EREMOVE the EPC page to make it invalid */ + r = sgx_eremove(epg); + if ( r == SGX_CHILD_PRESENT ) + { + list_add_tail(&epg->list, &secs_list); + continue; + } + + if ( r ) + { + printk("Domain %d: Reset EPC error: (gfn 0x%lx, mfn 0x%lx): " + "EREMOVE returns %d\n", d->domain_id, gfn, mfn_x(mfn), r); + ret = r; + if ( free_epc ) + printk("WARNING: EPC (mfn 0x%lx) becomes abnormal. " + "Remove it from useable EPC.", mfn_x(mfn)); + continue; + } + + if ( free_epc ) + { + /* If EPC page is going to be freed, then also remove the mapping */ + if ( clear_epc_p2m_entry(d, gfn, mfn) ) + { + printk("Domain %d: Reset EPC error: (gfn 0x%lx, mfn 0x%lx): " + "clear p2m entry failed.\n", d->domain_id, gfn, + mfn_x(mfn)); + ret = -EFAULT; + } + free_epc_page(epg); + } + } + + list_for_each_safe(p, tmp, &secs_list) + { + struct epc_page *epg = list_entry(p, struct epc_page, list); + int r; + + r = sgx_eremove(epg); + if ( r ) + { + printk("Domain %d: Reset EPC error: mfn 0x%lx: " + "EREMOVE returns %d for SECS page\n", + d->domain_id, epc_page_to_mfn(epg), r); + ret = r; + list_del(p); + + if ( free_epc ) + printk("WARNING: EPC (mfn 0x%lx) becomes abnormal. " + "Remove it from useable EPC.", + epc_page_to_mfn(epg)); + continue; + } + + if ( free_epc ) + free_epc_page(epg); + } + + return ret; +} + +static void __hvm_unpopulate_epc(struct domain *d, unsigned long epc_base_pfn, + unsigned long populated_npages) +{ + unsigned long i; + + for ( i = 0; i < populated_npages; i++ ) + { + struct epc_page *epg; + unsigned long gfn; + mfn_t mfn; + p2m_type_t t; + + gfn = i + epc_base_pfn; + mfn = get_gfn_query(d, gfn, &t); + if ( unlikely(mfn_eq(mfn, INVALID_MFN)) ) + { + /* + * __hvm_unpopulate_epc only called when creating the domain on + * failure, therefore we can just ignore this error. + */ + printk("%s: Domain %u gfn 0x%lx returns invalid mfn\n", __func__, + d->domain_id, gfn); + put_gfn(d, gfn); + continue; + } + + if ( unlikely(!p2m_is_epc(t)) ) + { + printk("%s: Domain %u gfn 0x%lx returns non-EPC p2m type: %d\n", + __func__, d->domain_id, gfn, (int)t); + put_gfn(d, gfn); + continue; + } + + put_gfn(d, gfn); + + if ( clear_epc_p2m_entry(d, gfn, mfn) ) + { + printk("clear_epc_p2m_entry failed: gfn 0x%lx, mfn 0x%lx\n", + gfn, mfn_x(mfn)); + continue; + } + + epg = epc_mfn_to_page(mfn_x(mfn)); + free_epc_page(epg); + } +} + +static int __hvm_populate_epc(struct domain *d, unsigned long epc_base_pfn, + unsigned long epc_npages) +{ + unsigned long i; + int ret; + + for ( i = 0; i < epc_npages; i++ ) + { + struct epc_page *epg = alloc_epc_page(); + unsigned long mfn; + + if ( !epg ) + { + printk("%s: Out of EPC\n", __func__); + ret = -ENOMEM; + goto err; + } + + mfn = epc_page_to_mfn(epg); + ret = set_epc_p2m_entry(d, i + epc_base_pfn, _mfn(mfn)); + if ( ret ) + { + printk("%s: set_epc_p2m_entry failed with %d: gfn 0x%lx, " + "mfn 0x%lx\n", __func__, ret, i + epc_base_pfn, mfn); + free_epc_page(epg); + goto err; + } + } + + return 0; + +err: + __hvm_unpopulate_epc(d, epc_base_pfn, i); + return ret; +} + +int hvm_populate_epc(struct domain *d, unsigned long epc_base_pfn, + unsigned long epc_npages) +{ + struct sgx_domain *sgx = to_sgx(d); + int ret; + + if ( hvm_epc_populated(d) ) + return -EBUSY; + + if ( !epc_base_pfn || !epc_npages ) + return -EINVAL; + + if ( (ret = __hvm_populate_epc(d, epc_base_pfn, epc_npages)) ) + return ret; + + sgx->epc_base_pfn = epc_base_pfn; + sgx->epc_npages = epc_npages; + + return 0; +} + +/* + * +* + * This function returns error immediately if there's any unexpected error + * during this process. + */ +int hvm_reset_epc(struct domain *d, bool_t free_epc) +{ + struct sgx_domain *sgx = to_sgx(d); + + if ( !hvm_epc_populated(d) ) + return 0; + + return __hvm_reset_epc(d, sgx->epc_base_pfn, sgx->epc_npages, free_epc); +} + +void hvm_destroy_epc(struct domain *d) +{ + hvm_reset_epc(d, true); +} + static bool_t sgx_enabled_in_bios(void) { uint64_t val, sgx_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE | diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c index c53b24955a..243643111d 100644 --- a/xen/arch/x86/hvm/vmx/vmx.c +++ b/xen/arch/x86/hvm/vmx/vmx.c @@ -416,6 +416,9 @@ static int vmx_domain_initialise(struct domain *d) static void vmx_domain_destroy(struct domain *d) { + if ( hvm_epc_populated(d) ) + hvm_destroy_epc(d); + if ( !has_vlapic(d) ) return; diff --git a/xen/include/asm-x86/hvm/vmx/sgx.h b/xen/include/asm-x86/hvm/vmx/sgx.h index ff420e006e..40f860662a 100644 --- a/xen/include/asm-x86/hvm/vmx/sgx.h +++ b/xen/include/asm-x86/hvm/vmx/sgx.h @@ -13,6 +13,7 @@ #include #include #include +#include /* HVM_PARAM_SGX */ #define SGX_CPUID 0x12 @@ -61,4 +62,17 @@ struct epc_page *epc_mfn_to_page(unsigned long mfn); void *map_epc_page_to_xen(struct epc_page *epg); void unmap_epc_page(void *addr); +struct sgx_domain { + unsigned long epc_base_pfn; + unsigned long epc_npages; +}; + +#define to_sgx(d) (&((d)->arch.hvm_domain.vmx.sgx)) +#define hvm_epc_populated(d) (!!((d)->arch.hvm_domain.vmx.sgx.epc_base_pfn)) + +int hvm_populate_epc(struct domain *d, unsigned long epc_base_pfn, + unsigned long epc_npages); +int hvm_reset_epc(struct domain *d, bool_t free_epc); +void hvm_destroy_epc(struct domain *d); + #endif /* __ASM_X86_HVM_VMX_SGX_H__ */ diff --git a/xen/include/asm-x86/hvm/vmx/vmcs.h b/xen/include/asm-x86/hvm/vmx/vmcs.h index 889091da42..6cfa5c3310 100644 --- a/xen/include/asm-x86/hvm/vmx/vmcs.h +++ b/xen/include/asm-x86/hvm/vmx/vmcs.h @@ -20,6 +20,7 @@ #include #include +#include extern void vmcs_dump_vcpu(struct vcpu *v); extern void setup_vmcs_dump(void); @@ -62,6 +63,7 @@ struct vmx_domain { unsigned long apic_access_mfn; /* VMX_DOMAIN_* */ unsigned int status; + struct sgx_domain sgx; }; struct pi_desc {