| Message ID | 20200319011130.8556-4-sean.j.christopherson@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | x86/sgx: Make vDSO callable from C | expand |
On 3/18/2020 6:11 PM, Sean Christopherson wrote: > Make __vdso_sgx_enter_enclave() callable from C by preserving %rbx > and taking @leaf in %rcx instead of %rax. Being able to invoke the vDSO > from C reduces the overhead of runtimes that are tightly coupled with > their enclaves, e.g. that can rely on the enclave to save and restore > non-volatile registers, as the runtime doesn't need an assembly wrapper > to preserve non-volatile registers and/or shuffle stack arguments. > > Note, both %rcx and %rbx are consumed by EENTER/ERESUME, i.e. consuming > them doesn't violate the primary tenet of __vdso_sgx_enter_enclave() > that "thou shalt not restrict how information is exchanged between an > enclave and its host process". > > Suggested-by: Nathaniel McCallum <npmccallum@redhat.com> > Cc: Cedric Xing <cedric.xing@intel.com> > Cc: Jethro Beekman <jethro@fortanix.com> > Cc: Andy Lutomirski <luto@amacapital.net> > Cc: linux-sgx@vger.kernel.org > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com> > --- > > Cedric, please go to town on this, you're much better at the userspace > stack and debugging interactions, e.g. I don't think the CFA directives > need updating, but my knowledge of that stuff is skeeetchy. > > Note, if the previous patch to make the %rsp restoration relative is > dropped, then %rbx should be restored relative to %rbp instead of %rsp. > > arch/x86/entry/vdso/vsgx_enter_enclave.S | 33 ++++++++++++++---------- > 1 file changed, 19 insertions(+), 14 deletions(-) > > diff --git a/arch/x86/entry/vdso/vsgx_enter_enclave.S b/arch/x86/entry/vdso/vsgx_enter_enclave.S > index 14f07d5e47ae..7a0565476a29 100644 > --- a/arch/x86/entry/vdso/vsgx_enter_enclave.S > +++ b/arch/x86/entry/vdso/vsgx_enter_enclave.S > @@ -16,25 +16,25 @@ > > /** > * __vdso_sgx_enter_enclave() - Enter an SGX enclave > + * @rdi: Pass-through value for RDI > + * @rsi: Pass-through value for RSI > + * @rdx: Pass-through value for RDX > * @leaf: ENCLU leaf, must be EENTER or ERESUME > + * @r8: Pass-through value for R8 > + * @r9: Pass-through value for R9 > * @tcs: TCS, must be non-NULL > * @e: Optional struct sgx_enclave_exception instance > * @handler: Optional enclave exit handler > * > - * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the > - * x86-64 ABI, i.e. cannot be called from standard C code. > - * > - * Input ABI: > - * @leaf %eax > - * @tcs 8(%rsp) > - * @e 0x10(%rsp) > - * @handler 0x18(%rsp) > - * > - * Output ABI: > - * @ret %eax > + * **Important!** __vdso_sgx_enter_enclave() does not ensure full compliance > + * with the x86-64 ABI, e.g. doesn't explicitly clear EFLAGS.DF after EEXIT. > + * Except for non-volatile general purpose registers, preserving/setting state > + * in accordance with the x86-64 ABI is the responsibility of the enclave and > + * its runtime, i.e. __vdso_sgx_enter_enclave() cannot be called from C code > + * without careful consideration by both the enclave and its runtime. > * > * All general purpose registers except RAX, RBX and RCX are passed as-is to > - * the enclave. RAX, RBX and RCX are consumed by EENTER and ERESUME and are > + * the enclave. RAX, RBX and RCX are consumed by EENTER and ERESUME and are > * loaded with @leaf, asynchronous exit pointer, and @tcs respectively. > * > * RBP and the stack are used to anchor __vdso_sgx_enter_enclave() to the > @@ -70,9 +70,11 @@ > */ > #ifdef SGX_KERNEL_DOC > /* C-style function prototype to coerce kernel-doc into parsing the comment. */ > -int __vdso_sgx_enter_enclave(int leaf, void *tcs, > +int __vdso_sgx_enter_enclave(unsigned long rdi, unsigned long rsi, > + unsigned long rdx, unsigned int leaf, > + unsigned long r8, unsigned long r9, void *tcs, > struct sgx_enclave_exception *e, > - sgx_enclave_exit_handler_t handler); > + sgx_enclave_exit_handler_t handler) > #endif > SYM_FUNC_START(__vdso_sgx_enter_enclave) > /* Prolog */ > @@ -82,7 +84,9 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) > .cfi_rel_offset %rbp, 0 > mov %rsp, %rbp > .cfi_def_cfa_register %rbp > + push %rbx A CFI directive is needed here: .cfi_rel_offset %rbx, -8 > > + mov %ecx, %eax > .Lenter_enclave: > /* EENTER <= leaf <= ERESUME */ > cmp $0x2, %eax > @@ -108,6 +112,7 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) > jne .Linvoke_userspace_handler > > .Lout: > + pop %rbx > leave > .cfi_def_cfa %rsp, 8 > ret >
On Thu, Mar 19, 2020 at 01:03:41PM -0700, Xing, Cedric wrote: > On 3/18/2020 6:11 PM, Sean Christopherson wrote: > > #endif > > SYM_FUNC_START(__vdso_sgx_enter_enclave) > > /* Prolog */ > >@@ -82,7 +84,9 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) > > .cfi_rel_offset %rbp, 0 > > mov %rsp, %rbp > > .cfi_def_cfa_register %rbp > >+ push %rbx > A CFI directive is needed here: > > .cfi_rel_offset %rbx, -8 Darn, I suspected as much, but wasn't 100% positive. Shouldn't have hedged. :-) Is the rule of thumb for adding directives that one is needed any time there is a new saved value of a register, or if the relative address of the last saved value changes? Are CFI directives only used for non-volatile registers? > >+ mov %ecx, %eax > > .Lenter_enclave: > > /* EENTER <= leaf <= ERESUME */ > > cmp $0x2, %eax > >@@ -108,6 +112,7 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) > > jne .Linvoke_userspace_handler > > .Lout: > >+ pop %rbx > > leave > > .cfi_def_cfa %rsp, 8 > > ret > >
On Thu, Mar 19, 2020 at 01:11:45PM -0700, Sean Christopherson wrote: > On Thu, Mar 19, 2020 at 01:03:41PM -0700, Xing, Cedric wrote: > > On 3/18/2020 6:11 PM, Sean Christopherson wrote: > > > #endif > > > SYM_FUNC_START(__vdso_sgx_enter_enclave) > > > /* Prolog */ > > >@@ -82,7 +84,9 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) > > > .cfi_rel_offset %rbp, 0 > > > mov %rsp, %rbp > > > .cfi_def_cfa_register %rbp > > >+ push %rbx > > A CFI directive is needed here: > > > > .cfi_rel_offset %rbx, -8 > > Darn, I suspected as much, but wasn't 100% positive. Shouldn't have > hedged. :-) > > Is the rule of thumb for adding directives that one is needed any time > there is a new saved value of a register, or if the relative address of > the last saved value changes? Are CFI directives only used for > non-volatile registers? AFAIK the convention is just that if you push a register you need to offset it so that the FDE that is put into .eh_frame by GCC has the information from which locations of the stack the register values are copied. BTW, why use GCC-style ".L-mangled" label names instead of having more readable ones? /Jarkko
On Fri, Mar 20, 2020 at 05:07:00AM +0200, Jarkko Sakkinen wrote: > On Thu, Mar 19, 2020 at 01:11:45PM -0700, Sean Christopherson wrote: > > On Thu, Mar 19, 2020 at 01:03:41PM -0700, Xing, Cedric wrote: > > > On 3/18/2020 6:11 PM, Sean Christopherson wrote: > > > > #endif > > > > SYM_FUNC_START(__vdso_sgx_enter_enclave) > > > > /* Prolog */ > > > >@@ -82,7 +84,9 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) > > > > .cfi_rel_offset %rbp, 0 > > > > mov %rsp, %rbp > > > > .cfi_def_cfa_register %rbp > > > >+ push %rbx > > > A CFI directive is needed here: > > > > > > .cfi_rel_offset %rbx, -8 > > > > Darn, I suspected as much, but wasn't 100% positive. Shouldn't have > > hedged. :-) > > > > Is the rule of thumb for adding directives that one is needed any time > > there is a new saved value of a register, or if the relative address of > > the last saved value changes? Are CFI directives only used for > > non-volatile registers? > > AFAIK the convention is just that if you push a register you need to > offset it so that the FDE that is put into .eh_frame by GCC has the > information from which locations of the stack the register values are > copied. > > BTW, why use GCC-style ".L-mangled" label names instead of having more > readable ones? Readable as in dropping the .L part? E.g. enclu_eenter_eresume: I assumed we'd want to keep the local labels out of the symbols file, but maybe getting them in there would be a good thing?
On Fri, Mar 20, 2020 at 04:26:49PM -0700, Sean Christopherson wrote: > On Fri, Mar 20, 2020 at 05:07:00AM +0200, Jarkko Sakkinen wrote: > > On Thu, Mar 19, 2020 at 01:11:45PM -0700, Sean Christopherson wrote: > > > On Thu, Mar 19, 2020 at 01:03:41PM -0700, Xing, Cedric wrote: > > > > On 3/18/2020 6:11 PM, Sean Christopherson wrote: > > > > > #endif > > > > > SYM_FUNC_START(__vdso_sgx_enter_enclave) > > > > > /* Prolog */ > > > > >@@ -82,7 +84,9 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) > > > > > .cfi_rel_offset %rbp, 0 > > > > > mov %rsp, %rbp > > > > > .cfi_def_cfa_register %rbp > > > > >+ push %rbx > > > > A CFI directive is needed here: > > > > > > > > .cfi_rel_offset %rbx, -8 > > > > > > Darn, I suspected as much, but wasn't 100% positive. Shouldn't have > > > hedged. :-) > > > > > > Is the rule of thumb for adding directives that one is needed any time > > > there is a new saved value of a register, or if the relative address of > > > the last saved value changes? Are CFI directives only used for > > > non-volatile registers? > > > > AFAIK the convention is just that if you push a register you need to > > offset it so that the FDE that is put into .eh_frame by GCC has the > > information from which locations of the stack the register values are > > copied. > > > > BTW, why use GCC-style ".L-mangled" label names instead of having more > > readable ones? > > Readable as in dropping the .L part? E.g. > > enclu_eenter_eresume: > > I assumed we'd want to keep the local labels out of the symbols file, but > maybe getting them in there would be a good thing? Right, that was the legit reason. Nope, lets keep them then. /Jarkko
diff --git a/arch/x86/entry/vdso/vsgx_enter_enclave.S b/arch/x86/entry/vdso/vsgx_enter_enclave.S index 14f07d5e47ae..7a0565476a29 100644 --- a/arch/x86/entry/vdso/vsgx_enter_enclave.S +++ b/arch/x86/entry/vdso/vsgx_enter_enclave.S @@ -16,25 +16,25 @@ /** * __vdso_sgx_enter_enclave() - Enter an SGX enclave + * @rdi: Pass-through value for RDI + * @rsi: Pass-through value for RSI + * @rdx: Pass-through value for RDX * @leaf: ENCLU leaf, must be EENTER or ERESUME + * @r8: Pass-through value for R8 + * @r9: Pass-through value for R9 * @tcs: TCS, must be non-NULL * @e: Optional struct sgx_enclave_exception instance * @handler: Optional enclave exit handler * - * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. - * - * Input ABI: - * @leaf %eax - * @tcs 8(%rsp) - * @e 0x10(%rsp) - * @handler 0x18(%rsp) - * - * Output ABI: - * @ret %eax + * **Important!** __vdso_sgx_enter_enclave() does not ensure full compliance + * with the x86-64 ABI, e.g. doesn't explicitly clear EFLAGS.DF after EEXIT. + * Except for non-volatile general purpose registers, preserving/setting state + * in accordance with the x86-64 ABI is the responsibility of the enclave and + * its runtime, i.e. __vdso_sgx_enter_enclave() cannot be called from C code + * without careful consideration by both the enclave and its runtime. * * All general purpose registers except RAX, RBX and RCX are passed as-is to - * the enclave. RAX, RBX and RCX are consumed by EENTER and ERESUME and are + * the enclave. RAX, RBX and RCX are consumed by EENTER and ERESUME and are * loaded with @leaf, asynchronous exit pointer, and @tcs respectively. * * RBP and the stack are used to anchor __vdso_sgx_enter_enclave() to the @@ -70,9 +70,11 @@ */ #ifdef SGX_KERNEL_DOC /* C-style function prototype to coerce kernel-doc into parsing the comment. */ -int __vdso_sgx_enter_enclave(int leaf, void *tcs, +int __vdso_sgx_enter_enclave(unsigned long rdi, unsigned long rsi, + unsigned long rdx, unsigned int leaf, + unsigned long r8, unsigned long r9, void *tcs, struct sgx_enclave_exception *e, - sgx_enclave_exit_handler_t handler); + sgx_enclave_exit_handler_t handler) #endif SYM_FUNC_START(__vdso_sgx_enter_enclave) /* Prolog */ @@ -82,7 +84,9 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) .cfi_rel_offset %rbp, 0 mov %rsp, %rbp .cfi_def_cfa_register %rbp + push %rbx + mov %ecx, %eax .Lenter_enclave: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax @@ -108,6 +112,7 @@ SYM_FUNC_START(__vdso_sgx_enter_enclave) jne .Linvoke_userspace_handler .Lout: + pop %rbx leave .cfi_def_cfa %rsp, 8 ret
Make __vdso_sgx_enter_enclave() callable from C by preserving %rbx and taking @leaf in %rcx instead of %rax. Being able to invoke the vDSO from C reduces the overhead of runtimes that are tightly coupled with their enclaves, e.g. that can rely on the enclave to save and restore non-volatile registers, as the runtime doesn't need an assembly wrapper to preserve non-volatile registers and/or shuffle stack arguments. Note, both %rcx and %rbx are consumed by EENTER/ERESUME, i.e. consuming them doesn't violate the primary tenet of __vdso_sgx_enter_enclave() that "thou shalt not restrict how information is exchanged between an enclave and its host process". Suggested-by: Nathaniel McCallum <npmccallum@redhat.com> Cc: Cedric Xing <cedric.xing@intel.com> Cc: Jethro Beekman <jethro@fortanix.com> Cc: Andy Lutomirski <luto@amacapital.net> Cc: linux-sgx@vger.kernel.org Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com> --- Cedric, please go to town on this, you're much better at the userspace stack and debugging interactions, e.g. I don't think the CFA directives need updating, but my knowledge of that stuff is skeeetchy. Note, if the previous patch to make the %rsp restoration relative is dropped, then %rbx should be restored relative to %rbp instead of %rsp. arch/x86/entry/vdso/vsgx_enter_enclave.S | 33 ++++++++++++++---------- 1 file changed, 19 insertions(+), 14 deletions(-)