| Message ID | 20200319011130.8556-1-sean.j.christopherson@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | x86/sgx: Make vDSO callable from C | expand |
On Wed, Mar 18, 2020 at 06:11:22PM -0700, Sean Christopherson wrote: > Nathaniel pointed out that __vdso_sgx_enter_enclave() is tantalizingly > close to being callable from C (with caveats and a cooperative enclave). > The missing pieces are preserving %rbx and taking @leaf as a standard > paramter (hey look, %rcx is available!). > > As requested, update the selftest to invoke the vDSO from C, and fix > bugs along the way. Don't suppose you've changed your mind about waiting > until after upstreaming to add a proper framework? > > I also did a lot more testing on my internal code base to verify the > %rsp tweaks play nice with the exit handler, and to prove that an exit > handler can even cleanly handle exceptions in the enclave. > > Here's the relevant code from my internal test suite, might want to have > eye bleach ready ;-). I tested the exception handling by overwriting > @ms in enter_enclave() with a bogus value, e.g. 0xdeadull << 0x48. The > nested call to the vDSO (in the exit handler) with EMCD_EXCEPTION has the > enclave dump its register. The host (this runs as cgo in a Golang process) > dumps the exception info. > > I also verified the relative %rsp fixup by corrupting %rsp in the enclave, > verifying it exploded as expected (I added extra consumption in the vDSO to > force this) and then adding code to undo the damage in the exit handler and > verifying things got back to normal. > > Last thread of thought, IMO taking a context param (as suggested by > Nathaniel) is unnecessary. As shown below, the runtime effectively has a > context param in the form of "struct sgx_enclave_exception *e", the handler > just needs to be willing to cast (or use container_of()). There are a > multitude of other ways to pass info as well. > > vdso_sgx_enter_enclave_t __enter_enclave; > > vDSO::vDSO() > { > if (__enter_enclave) > abort(); > > uintptr_t vdso = (uintptr_t)getauxval(AT_SYSINFO_EHDR); > vdso_init_from_sysinfo_ehdr(vdso); > > __enter_enclave = (vdso_sgx_enter_enclave_t)vdso_sym("LINUX_2.6", "__vdso_sgx_enter_enclave"); > if (!__enter_enclave) > abort(); > } > > struct sgx_vdso_ctxt { > sgx_exception_t e; > long ret; > Enclave * enclave; > jmp_buf buf; > }; > > static int exit_handler(long rdi, long rsi, long rdx, long ursp, long r8, long r9, > void *tcs, int ret, struct sgx_enclave_exception *e) > { > struct sgx_vdso_ctxt *ctxt = (struct sgx_vdso_ctxt *)e; > int r; > > if (!ret) { > if (ctxt) > ctxt->ret = rdi; > return 0; > } > > if (!ctxt) > exit(EFAULT); > > r = __enter_enclave(ECMD_EXCEPTION, (unsigned long)&ctxt->e.regs, 0, EENTER, > 0, 0, tcs, NULL, exit_handler); > if (r) > exit(-r); > > ctxt->enclave->set_exception(ctxt->e); > > longjmp(ctxt->buf, ret); > } > > sgx_status_t vDSO::enter_enclave(Enclave &enclave, const long fn, void *ms, tcs_t *tcs) > { > struct sgx_vdso_ctxt ctxt; > ctxt.enclave = &enclave; > > int ret = setjmp(ctxt.buf); > if (ret) { > if (ret != -EFAULT) > return SGX_ERROR_INVALID_PARAMETER; > > SE_URTS_ERROR("\nEnclave exception, base = %lx, size = %lx", > enclave.get_base(), enclave.get_size()); > > return SGX_ERROR_UNHANDLED_EXCEPTION; > } > > ret = __enter_enclave(fn, (unsigned long)ms, 0, EENTER, 0, 0, tcs, > &ctxt.e.fault, exit_handler); > if (ret == -EINVAL) > return SGX_ERROR_INVALID_PARAMETER; > > return (sgx_status_t)ctxt.ret; > } > > Sean Christopherson (8): > x86/sgx: vdso: Remove an incorrect statement the enter enclave comment > x86/sgx: vdso: Make the %rsp fixup on return from handler relative > x86/sgx: vdso: Make __vdso_sgx_enter_enclave() callable from C code > x86/sgx: vdso: Define a typedef for __vdso_sgx_enter_enclave > selftests/x86: sgx: Zero out @result before invoking vDSO sub-test > selftests/x86: sgx: Pass EENTER to vDSO wrapper instead of hardcoding > selftests/x86: sgx: Stop clobbering non-volatile registers > selftests/x86: Add selftest to invoke __vsgx_enter_enclave() from C > > arch/x86/entry/vdso/vsgx_enter_enclave.S | 72 ++----------------- > arch/x86/include/uapi/asm/sgx.h | 61 ++++++++++++++++ > .../selftests/x86/sgx/encl_bootstrap.S | 6 +- > tools/testing/selftests/x86/sgx/main.c | 17 ++++- > tools/testing/selftests/x86/sgx/sgx_call.S | 1 - > tools/testing/selftests/x86/sgx/sgx_call.h | 2 +- > 6 files changed, 85 insertions(+), 74 deletions(-) > > -- > 2.24.1 > Might be a grazy idea but better to go through this anyway. Given the premise that we need the carry the callback anyway in all cases, why won't just have the callback. Why we absolutely need the code path that fills exception info given that we no matter what need to have a callback route? Would simplify considerably to have only clear flow. /Jarkko
On Fri, Mar 20, 2020 at 02:57:24AM +0200, Jarkko Sakkinen wrote: > On Wed, Mar 18, 2020 at 06:11:22PM -0700, Sean Christopherson wrote: > > Sean Christopherson (8): > > x86/sgx: vdso: Remove an incorrect statement the enter enclave comment > > x86/sgx: vdso: Make the %rsp fixup on return from handler relative > > x86/sgx: vdso: Make __vdso_sgx_enter_enclave() callable from C code > > x86/sgx: vdso: Define a typedef for __vdso_sgx_enter_enclave > > selftests/x86: sgx: Zero out @result before invoking vDSO sub-test > > selftests/x86: sgx: Pass EENTER to vDSO wrapper instead of hardcoding > > selftests/x86: sgx: Stop clobbering non-volatile registers > > selftests/x86: Add selftest to invoke __vsgx_enter_enclave() from C > > > > arch/x86/entry/vdso/vsgx_enter_enclave.S | 72 ++----------------- > > arch/x86/include/uapi/asm/sgx.h | 61 ++++++++++++++++ > > .../selftests/x86/sgx/encl_bootstrap.S | 6 +- > > tools/testing/selftests/x86/sgx/main.c | 17 ++++- > > tools/testing/selftests/x86/sgx/sgx_call.S | 1 - > > tools/testing/selftests/x86/sgx/sgx_call.h | 2 +- > > 6 files changed, 85 insertions(+), 74 deletions(-) > > > > -- > > 2.24.1 > > > > Might be a grazy idea but better to go through this anyway. > > Given the premise that we need the carry the callback anyway in all > cases, why won't just have the callback. > > Why we absolutely need the code path that fills exception info given > that we no matter what need to have a callback route? > > Would simplify considerably to have only clear flow. Invoking the callback uses a retpoline, which is non-trivial overhead. For runtimes that need an assembly wrapper for other reasons, and aren't using the untrusted stack, forcing them to implement a handler would be painful.
On Fri, Mar 20, 2020 at 04:25:12PM -0700, Sean Christopherson wrote: > On Fri, Mar 20, 2020 at 02:57:24AM +0200, Jarkko Sakkinen wrote: > > On Wed, Mar 18, 2020 at 06:11:22PM -0700, Sean Christopherson wrote: > > > Sean Christopherson (8): > > > x86/sgx: vdso: Remove an incorrect statement the enter enclave comment > > > x86/sgx: vdso: Make the %rsp fixup on return from handler relative > > > x86/sgx: vdso: Make __vdso_sgx_enter_enclave() callable from C code > > > x86/sgx: vdso: Define a typedef for __vdso_sgx_enter_enclave > > > selftests/x86: sgx: Zero out @result before invoking vDSO sub-test > > > selftests/x86: sgx: Pass EENTER to vDSO wrapper instead of hardcoding > > > selftests/x86: sgx: Stop clobbering non-volatile registers > > > selftests/x86: Add selftest to invoke __vsgx_enter_enclave() from C > > > > > > arch/x86/entry/vdso/vsgx_enter_enclave.S | 72 ++----------------- > > > arch/x86/include/uapi/asm/sgx.h | 61 ++++++++++++++++ > > > .../selftests/x86/sgx/encl_bootstrap.S | 6 +- > > > tools/testing/selftests/x86/sgx/main.c | 17 ++++- > > > tools/testing/selftests/x86/sgx/sgx_call.S | 1 - > > > tools/testing/selftests/x86/sgx/sgx_call.h | 2 +- > > > 6 files changed, 85 insertions(+), 74 deletions(-) > > > > > > -- > > > 2.24.1 > > > > > > > Might be a grazy idea but better to go through this anyway. > > > > Given the premise that we need the carry the callback anyway in all > > cases, why won't just have the callback. > > > > Why we absolutely need the code path that fills exception info given > > that we no matter what need to have a callback route? > > > > Would simplify considerably to have only clear flow. > > Invoking the callback uses a retpoline, which is non-trivial overhead. > For runtimes that need an assembly wrapper for other reasons, and aren't > using the untrusted stack, forcing them to implement a handler would be > painful. The non-callback route only exists because we did not know that we have to do the callback route. It does not make sense to add something for any other reason than absolute necessity. Things would simplify in the vDSO implementation considerably. Now it is overwhelmingly complex for no good reason. /Jarkko
On Sat, Mar 21, 2020 at 02:55:28AM +0200, Jarkko Sakkinen wrote: > On Fri, Mar 20, 2020 at 04:25:12PM -0700, Sean Christopherson wrote: > > On Fri, Mar 20, 2020 at 02:57:24AM +0200, Jarkko Sakkinen wrote: > > > On Wed, Mar 18, 2020 at 06:11:22PM -0700, Sean Christopherson wrote: > > > > Sean Christopherson (8): > > > > x86/sgx: vdso: Remove an incorrect statement the enter enclave comment > > > > x86/sgx: vdso: Make the %rsp fixup on return from handler relative > > > > x86/sgx: vdso: Make __vdso_sgx_enter_enclave() callable from C code > > > > x86/sgx: vdso: Define a typedef for __vdso_sgx_enter_enclave > > > > selftests/x86: sgx: Zero out @result before invoking vDSO sub-test > > > > selftests/x86: sgx: Pass EENTER to vDSO wrapper instead of hardcoding > > > > selftests/x86: sgx: Stop clobbering non-volatile registers > > > > selftests/x86: Add selftest to invoke __vsgx_enter_enclave() from C > > > > > > > > arch/x86/entry/vdso/vsgx_enter_enclave.S | 72 ++----------------- > > > > arch/x86/include/uapi/asm/sgx.h | 61 ++++++++++++++++ > > > > .../selftests/x86/sgx/encl_bootstrap.S | 6 +- > > > > tools/testing/selftests/x86/sgx/main.c | 17 ++++- > > > > tools/testing/selftests/x86/sgx/sgx_call.S | 1 - > > > > tools/testing/selftests/x86/sgx/sgx_call.h | 2 +- > > > > 6 files changed, 85 insertions(+), 74 deletions(-) > > > > > > > > -- > > > > 2.24.1 > > > > > > > > > > Might be a grazy idea but better to go through this anyway. > > > > > > Given the premise that we need the carry the callback anyway in all > > > cases, why won't just have the callback. > > > > > > Why we absolutely need the code path that fills exception info given > > > that we no matter what need to have a callback route? > > > > > > Would simplify considerably to have only clear flow. > > > > Invoking the callback uses a retpoline, which is non-trivial overhead. > > For runtimes that need an assembly wrapper for other reasons, and aren't > > using the untrusted stack, forcing them to implement a handler would be > > painful. > > The non-callback route only exists because we did not know that we have > to do the callback route. It does not make sense to add something for > any other reason than absolute necessity. The non-callback route exists because it's simpler for the runtime to use if it already has an asm wrapper and doesn't do stack shenanigans, e.g. the runtime doesn't need to play games to get context information when handling an exit. And using a callback in conjunction with calling the vDSO from C is by no means necessary, e.g. if the runtime is happy to die on exceptions, or if it's doing clever clobbering, function annotation, longjump(), etc... to avoid consuming corrupted state on an exception. > Things would simplify in the vDSO implementation considerably. Now it is > overwhelmingly complex for no good reason. No it wouldn't, it literally saves zero instructions (unless the vDSO wanted to crash the caller on a NULL pointer exception). It'd do nothing more than move this code: /* Invoke userspace's exit handler if one was provided. */ .Lhandle_exit: cmp $0, 0x20(%rbp) jne .Linvoke_userspace_handler up to the beginning of the code as /* Return -EINVAL if there's no userspace exit handler. */ cmp $0, 0x20(%rbp) je .Linvalid_input and the exit handler invocation would get inlined. Unless I'm missing some clever massaging of code, that's it. At best the above could be dropped, but that's a whopping two instructions and a single uop.
Nathaniel pointed out that __vdso_sgx_enter_enclave() is tantalizingly close to being callable from C (with caveats and a cooperative enclave). The missing pieces are preserving %rbx and taking @leaf as a standard paramter (hey look, %rcx is available!). As requested, update the selftest to invoke the vDSO from C, and fix bugs along the way. Don't suppose you've changed your mind about waiting until after upstreaming to add a proper framework? I also did a lot more testing on my internal code base to verify the %rsp tweaks play nice with the exit handler, and to prove that an exit handler can even cleanly handle exceptions in the enclave. Here's the relevant code from my internal test suite, might want to have eye bleach ready ;-). I tested the exception handling by overwriting @ms in enter_enclave() with a bogus value, e.g. 0xdeadull << 0x48. The nested call to the vDSO (in the exit handler) with EMCD_EXCEPTION has the enclave dump its register. The host (this runs as cgo in a Golang process) dumps the exception info. I also verified the relative %rsp fixup by corrupting %rsp in the enclave, verifying it exploded as expected (I added extra consumption in the vDSO to force this) and then adding code to undo the damage in the exit handler and verifying things got back to normal. Last thread of thought, IMO taking a context param (as suggested by Nathaniel) is unnecessary. As shown below, the runtime effectively has a context param in the form of "struct sgx_enclave_exception *e", the handler just needs to be willing to cast (or use container_of()). There are a multitude of other ways to pass info as well. vdso_sgx_enter_enclave_t __enter_enclave; vDSO::vDSO() { if (__enter_enclave) abort(); uintptr_t vdso = (uintptr_t)getauxval(AT_SYSINFO_EHDR); vdso_init_from_sysinfo_ehdr(vdso); __enter_enclave = (vdso_sgx_enter_enclave_t)vdso_sym("LINUX_2.6", "__vdso_sgx_enter_enclave"); if (!__enter_enclave) abort(); } struct sgx_vdso_ctxt { sgx_exception_t e; long ret; Enclave * enclave; jmp_buf buf; }; static int exit_handler(long rdi, long rsi, long rdx, long ursp, long r8, long r9, void *tcs, int ret, struct sgx_enclave_exception *e) { struct sgx_vdso_ctxt *ctxt = (struct sgx_vdso_ctxt *)e; int r; if (!ret) { if (ctxt) ctxt->ret = rdi; return 0; } if (!ctxt) exit(EFAULT); r = __enter_enclave(ECMD_EXCEPTION, (unsigned long)&ctxt->e.regs, 0, EENTER, 0, 0, tcs, NULL, exit_handler); if (r) exit(-r); ctxt->enclave->set_exception(ctxt->e); longjmp(ctxt->buf, ret); } sgx_status_t vDSO::enter_enclave(Enclave &enclave, const long fn, void *ms, tcs_t *tcs) { struct sgx_vdso_ctxt ctxt; ctxt.enclave = &enclave; int ret = setjmp(ctxt.buf); if (ret) { if (ret != -EFAULT) return SGX_ERROR_INVALID_PARAMETER; SE_URTS_ERROR("\nEnclave exception, base = %lx, size = %lx", enclave.get_base(), enclave.get_size()); return SGX_ERROR_UNHANDLED_EXCEPTION; } ret = __enter_enclave(fn, (unsigned long)ms, 0, EENTER, 0, 0, tcs, &ctxt.e.fault, exit_handler); if (ret == -EINVAL) return SGX_ERROR_INVALID_PARAMETER; return (sgx_status_t)ctxt.ret; } Sean Christopherson (8): x86/sgx: vdso: Remove an incorrect statement the enter enclave comment x86/sgx: vdso: Make the %rsp fixup on return from handler relative x86/sgx: vdso: Make __vdso_sgx_enter_enclave() callable from C code x86/sgx: vdso: Define a typedef for __vdso_sgx_enter_enclave selftests/x86: sgx: Zero out @result before invoking vDSO sub-test selftests/x86: sgx: Pass EENTER to vDSO wrapper instead of hardcoding selftests/x86: sgx: Stop clobbering non-volatile registers selftests/x86: Add selftest to invoke __vsgx_enter_enclave() from C arch/x86/entry/vdso/vsgx_enter_enclave.S | 72 ++----------------- arch/x86/include/uapi/asm/sgx.h | 61 ++++++++++++++++ .../selftests/x86/sgx/encl_bootstrap.S | 6 +- tools/testing/selftests/x86/sgx/main.c | 17 ++++- tools/testing/selftests/x86/sgx/sgx_call.S | 1 - tools/testing/selftests/x86/sgx/sgx_call.h | 2 +- 6 files changed, 85 insertions(+), 74 deletions(-)