| Message ID | cover.1697614386.git.andrea.porta@suse.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | arm64: Make Aarch32 compatibility enablement optional at boot | expand |
Hi, On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote: > Aarch32 compatibility mode is enabled at compile time through > CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support > (for both processes and syscalls) be enabled at boot time using > a kernel parameter. Also, it provides a mean for distributions > to set their own default without sacrificing compatibility support, > that is users can override default behaviour through the kernel > parameter. I proposed something similar in the past: https://lkml.kernel.org/linux-fsdevel/20210916131816.8841-1-will@kernel.org/ bu the conclusion there (see the reply from Kees) was that it was better to either use existing seccomp mechanisms or add something to control which binfmts can be loaded. Will
On Wed, Oct 18, 2023, at 14:27, Will Deacon wrote: > Hi, > > On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote: >> Aarch32 compatibility mode is enabled at compile time through >> CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support >> (for both processes and syscalls) be enabled at boot time using >> a kernel parameter. Also, it provides a mean for distributions >> to set their own default without sacrificing compatibility support, >> that is users can override default behaviour through the kernel >> parameter. > > I proposed something similar in the past: > > https://lkml.kernel.org/linux-fsdevel/20210916131816.8841-1-will@kernel.org/ > > bu the conclusion there (see the reply from Kees) was that it was better > to either use existing seccomp mechanisms or add something to control > which binfmts can be loaded. Right, I was going to reply along the same lines here: x86 is a bit of a special case that needs this, but I believe all the other architectures already guard the compat syscall execution on test_thread_flag(TIF_32BIT) that is only set by the compat binfmt loader. Doing the reverse is something that has however come up in the past several times and that could be interesting: In order to run userspace emulation (qemu-user, fex, ...) we may want to allow calling syscalls and ioctls for foreign ABIs in a native task, and at that point having a mechanism to control this capability globally or per task would be useful as well. The compat mode (arm32 on arm64) is the easiest case here, but the same thing could be done for emulating the very subtle architecture differences (x86-64 on arm64, arm64 on x86_64, arm32 on x86-compat, or any of the above on riscv or loongarch). Arnd
On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote: > Aarch32 compatibility mode is enabled at compile time through > CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support > (for both processes and syscalls) be enabled at boot time using > a kernel parameter. Also, it provides a mean for distributions > to set their own default without sacrificing compatibility support, > that is users can override default behaviour through the kernel > parameter. Can you elaborate on *why* people want such a policy? > *** Notes about syscall management *** > VBAR_EL1 register, which holds the exception table address, > is setup very early in the boot process, before parse_early_param(). > This means that it's not possible to access boot parameter before > setting the register. Also, setting the aforementioned register > for secondary cpus is done later in the boot flow. > Several ways to work around this has been considered, among which: > > * resetting VBAR_EL1 to point to one of two vector tables (the > former with 32-bit exceptions handler enabled and the latter > pointing to unhandled stub, just as if CONFIG_COMPAT is enabled) > depending on the proposed boot parameter. This has the disadvantage > to produce a somewhat messy patchset involving several lines, > has higher cognitive load since there are at least three places > where the register is getting changed (not near to each other), > and have implications on other code segments (namely kpti, kvm > and vdso), requiring special care. > > * patching the vector table contents once the early param is available. > This has most of the implications of the previous option > (except maybe not impacting other code segments), plus it sounds > a little 'hackish'. > > The chosen approach involves conditional executing 32-bit syscalls > depending on the parameter value. Why does the compat syscall path need to do anything? On arm64 it's not possible to issue compat syscalls from a native 64-bit task. If you prevent the loading of AArch32 binaries, none of the compat syscalls will be reachable at all. That's the proper way to implement this, and we already have logic for that as part of the mismatched AArch32 support. > This of course results in a little performance loss, but has the following > advantages: A performance loss for what relative to what? How much of a performance loss? Mark. > * all the cons from previously explained alternatives are solved > * users of 32-bit apps on 64-bit kernel are already suffering from > performance losses due to 32-bit apps not fully leveraging the 64-bit > processor, so they are already aware of this > * users of 32-bit apps on 64-bit kernel are believed > to be a minority and most of the time there are sources available > to be recompiled for 64-bit as a workaround for better performance > > It worth mentioning that users of 64-bit apps are, of course, > unaffected. > > Based on the work from Nikolay Borisov, see: > Link: https://lkml.org/lkml/2023/6/23/387 > > Andrea della Porta (4): > arm64: Introduce aarch32_enabled() > arm64/process: Make loading of 32bit processes depend on > aarch32_enabled() > arm64/entry-common: Make Aarch32 syscalls' availability depend on > aarch32_enabled() > arm64: Make Aarch32 emulation boot time configurable > > .../admin-guide/kernel-parameters.txt | 7 ++++ > arch/arm64/Kconfig | 9 +++++ > arch/arm64/include/asm/compat.h | 12 +++++++ > arch/arm64/kernel/entry-common.c | 33 +++++++++++++++++-- > arch/arm64/kernel/process.c | 2 +- > 5 files changed, 59 insertions(+), 4 deletions(-) > > -- > 2.35.3 >
On 13:27 Wed 18 Oct , Will Deacon wrote: > Hi, > > On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote: > > Aarch32 compatibility mode is enabled at compile time through > > CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support > > (for both processes and syscalls) be enabled at boot time using > > a kernel parameter. Also, it provides a mean for distributions > > to set their own default without sacrificing compatibility support, > > that is users can override default behaviour through the kernel > > parameter. > > I proposed something similar in the past: > > https://lkml.kernel.org/linux-fsdevel/20210916131816.8841-1-will@kernel.org/ > > bu the conclusion there (see the reply from Kees) was that it was better > to either use existing seccomp mechanisms or add something to control > which binfmts can be loaded. > > Will I see. Seccomp sounds like a really good idea, since just blocking the compat binfmt would not avoid the call to 32-bit syscalls per se: it's true that ARM64 enforce the transition from A64 to A32 only on exception return and PSTATE.nRW flag can change only from EL1, maybe though some exploitation may arise in the future to do just that (I'm not aware of any or come up with a proof off the top of my head, but I can't exclude it either). So, assuming by absurd a switch to A32 is feasible, the further step of embedding A32 instruction in a A64 ELF executable is a breeze. Hence blocking the syscall (and not only the binfmt loading) could prove necessary. I know all of this is higly speculative right now, maybe it's worth thinking nonetheless. Andrea
On 14:44 Wed 18 Oct , Arnd Bergmann wrote: > On Wed, Oct 18, 2023, at 14:27, Will Deacon wrote: > > Hi, > > > > On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote: > >> Aarch32 compatibility mode is enabled at compile time through > >> CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support > >> (for both processes and syscalls) be enabled at boot time using > >> a kernel parameter. Also, it provides a mean for distributions > >> to set their own default without sacrificing compatibility support, > >> that is users can override default behaviour through the kernel > >> parameter. > > > > I proposed something similar in the past: > > > > https://lkml.kernel.org/linux-fsdevel/20210916131816.8841-1-will@kernel.org/ > > > > bu the conclusion there (see the reply from Kees) was that it was better > > to either use existing seccomp mechanisms or add something to control > > which binfmts can be loaded. > > Right, I was going to reply along the same lines here: x86 is > a bit of a special case that needs this, but I believe all the > other architectures already guard the compat syscall execution > on test_thread_flag(TIF_32BIT) that is only set by the compat > binfmt loader. Are you referring to the fact that x86 can switch at will between 32- and 64- bit code? Regarding the TIF_32BIT flag, thanks for the head-up. I still believe though that this mechanism can somehow break down in the future, since prohibiting 32 bit executable loading *and* blocking 32 bit compat syscall are two separate path of execution, held together by the architecture prohibiting to switch to A32 instructions by design. Breaking the first rule and embedding wisely crafted A32 instruction in an executable is easy, while the difficult part is finding some 'reentrancy' to be able to do the execution state switch, as pinted out in https://lore.kernel.org/lkml/ZTD0DAes-J-YQ2eu@apocalypse/. I agree it's highly speculative and not something to be concerned right now, it's just a head up, should the need arise in the future. > Doing the reverse is something that has however come up in the > past several times and that could be interesting: In order to > run userspace emulation (qemu-user, fex, ...) we may want to > allow calling syscalls and ioctls for foreign ABIs in a native > task, and at that point having a mechanism to control this > capability globally or per task would be useful as well. > > The compat mode (arm32 on arm64) is the easiest case here, but the > same thing could be done for emulating the very subtle architecture > differences (x86-64 on arm64, arm64 on x86_64, arm32 on x86-compat, > or any of the above on riscv or loongarch). > > Arnd Really interesting, Since it's more related to emulation needs (my patch has another focus due to the fact that A64 can execute A32 natively), I'll take a look at this separately. Andrea
On Thu, Oct 19, 2023, at 12:52, Andrea della Porta wrote: > On 14:44 Wed 18 Oct , Arnd Bergmann wrote: >> On Wed, Oct 18, 2023, at 14:27, Will Deacon wrote: >> >> Right, I was going to reply along the same lines here: x86 is >> a bit of a special case that needs this, but I believe all the >> other architectures already guard the compat syscall execution >> on test_thread_flag(TIF_32BIT) that is only set by the compat >> binfmt loader. > > Are you referring to the fact that x86 can switch at will between 32- and 64- > bit code? No. > Regarding the TIF_32BIT flag, thanks for the head-up. I still believe though > that this mechanism can somehow break down in the future, since prohibiting > 32 bit executable loading *and* blocking 32 bit compat syscall are two > separate path of execution, held together by the architecture prohibiting > to switch to A32 instructions by design. Breaking the first rule and embedding > wisely crafted A32 instruction in an executable is easy, while the difficult > part is finding some 'reentrancy' to be able to do the execution state switch, > as pinted out in https://lore.kernel.org/lkml/ZTD0DAes-J-YQ2eu@apocalypse/. > I agree it's highly speculative and not something to be concerned right > now, it's just a head up, should the need arise in the future. There are (at least) five separate aspects to compat mode that are easy to mix up: 1. Instruction decoding -- switching between the modes supported by the CPU (A64/A32/T32) 2. Word size -- what happens to the upper 32 bits of a register in an arithmetic operation 3. Personality -- Which architecture string gets returned by the uname syscall (aarch64 vs armv8) as well as the format of /proc/cpuinfo 4. system call entry points -- how a process calls into native or compat syscalls, or possibly foreign OS emulation 5. Binary format -- elf32 vs elf64 executables On most architectures with compat mode, 4. and 5. are fundamentally tied together today: a compat task can only call compat syscalls and a native task can only call native syscalls. x86 is the exception here, as it uses different instructions (int80, syscall, sysenter) and picks the syscall table based on that instruction. I think 1. and 2. are also always tied to 5 on arm, but this is not necessarily true for other architectures. 3. used to be tied to 5 on some architectures in the past, but should be independent now. >> Doing the reverse is something that has however come up in the >> past several times and that could be interesting: In order to >> run userspace emulation (qemu-user, fex, ...) we may want to >> allow calling syscalls and ioctls for foreign ABIs in a native >> task, and at that point having a mechanism to control this >> capability globally or per task would be useful as well. >> >> The compat mode (arm32 on arm64) is the easiest case here, but the >> same thing could be done for emulating the very subtle architecture >> differences (x86-64 on arm64, arm64 on x86_64, arm32 on x86-compat, >> or any of the above on riscv or loongarch). > > Really interesting, Since it's more related to emulation needs (my patch > has another focus due to the fact that A64 can execute A32 natively), > I'll take a look at this separately. A64 mode (unlike some other architectures, notably mips64) cannot execute A32 or T32 instructions without a mode switch, the three are entirely incompatible on the binary level. Many ARMv8-CPUs support both Aarch64 mode and Aarch32 (A32/T32), but a lot of the newer ones (e.g. Apple M1/M2, Cortex-R82 or Cortex-A715) only do Aarch64 and need user-space emulation to run 32-bit binaries. Arnd
On 13:52 Wed 18 Oct , Mark Rutland wrote: > On Wed, Oct 18, 2023 at 01:13:18PM +0200, Andrea della Porta wrote: > > Aarch32 compatibility mode is enabled at compile time through > > CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support > > (for both processes and syscalls) be enabled at boot time using > > a kernel parameter. Also, it provides a mean for distributions > > to set their own default without sacrificing compatibility support, > > that is users can override default behaviour through the kernel > > parameter. > > Can you elaborate on *why* people want such a policy? > Formerly, the reason was to reduce kernel attack surface by excluding compat syscall, wherever applicable. Much less important but still a point, I would also say this could be a good chance to get rid of somewhat old and stale 32-bit libraries and programs, but this is of course debatable. > > *** Notes about syscall management *** > > VBAR_EL1 register, which holds the exception table address, > > is setup very early in the boot process, before parse_early_param(). > > This means that it's not possible to access boot parameter before > > setting the register. Also, setting the aforementioned register > > for secondary cpus is done later in the boot flow. > > Several ways to work around this has been considered, among which: > > > > * resetting VBAR_EL1 to point to one of two vector tables (the > > former with 32-bit exceptions handler enabled and the latter > > pointing to unhandled stub, just as if CONFIG_COMPAT is enabled) > > depending on the proposed boot parameter. This has the disadvantage > > to produce a somewhat messy patchset involving several lines, > > has higher cognitive load since there are at least three places > > where the register is getting changed (not near to each other), > > and have implications on other code segments (namely kpti, kvm > > and vdso), requiring special care. > > > > * patching the vector table contents once the early param is available. > > This has most of the implications of the previous option > > (except maybe not impacting other code segments), plus it sounds > > a little 'hackish'. > > > > The chosen approach involves conditional executing 32-bit syscalls > > depending on the parameter value. > > Why does the compat syscall path need to do anything? I probably didn't catch your point here, compat syscall does not need to do anything and they do not (just like they works right now with CONFIG_COMPAT alone), except for the conditional instruction that excludes them at runtime. Of course this conditional *is* doing something and somewhat redundant if compat is disabled, but in this scenario I think it's unavoidable. > > On arm64 it's not possible to issue compat syscalls from a native 64-bit task. > If you prevent the loading of AArch32 binaries, none of the compat syscalls > will be reachable at all. > > That's the proper way to implement this, and we already have logic for that as > part of the mismatched AArch32 support. > > > This of course results in a little performance loss, but has the following > > advantages: > > A performance loss for what relative to what? of a compat syscall as it is now enabling CONFIG_COMPAT vs the patched syscall handlers that need a further conditional instruction to check whether comapt is enabled or not. > > How much of a performance loss? I did not take measurement yet since it was just a qualitative consideration more than a quantitative one, also considering that chances are that it would affect just very little population. The conditional instruction time taken to execute is reasonably near to negligible if compared to any syscall execution.
Aarch32 compatibility mode is enabled at compile time through CONFIG_COMPAT Kconfig option. This patchset lets 32-bit support (for both processes and syscalls) be enabled at boot time using a kernel parameter. Also, it provides a mean for distributions to set their own default without sacrificing compatibility support, that is users can override default behaviour through the kernel parameter. *** Notes about syscall management *** VBAR_EL1 register, which holds the exception table address, is setup very early in the boot process, before parse_early_param(). This means that it's not possible to access boot parameter before setting the register. Also, setting the aforementioned register for secondary cpus is done later in the boot flow. Several ways to work around this has been considered, among which: * resetting VBAR_EL1 to point to one of two vector tables (the former with 32-bit exceptions handler enabled and the latter pointing to unhandled stub, just as if CONFIG_COMPAT is enabled) depending on the proposed boot parameter. This has the disadvantage to produce a somewhat messy patchset involving several lines, has higher cognitive load since there are at least three places where the register is getting changed (not near to each other), and have implications on other code segments (namely kpti, kvm and vdso), requiring special care. * patching the vector table contents once the early param is available. This has most of the implications of the previous option (except maybe not impacting other code segments), plus it sounds a little 'hackish'. The chosen approach involves conditional executing 32-bit syscalls depending on the parameter value. This of course results in a little performance loss, but has the following advantages: * all the cons from previously explained alternatives are solved * users of 32-bit apps on 64-bit kernel are already suffering from performance losses due to 32-bit apps not fully leveraging the 64-bit processor, so they are already aware of this * users of 32-bit apps on 64-bit kernel are believed to be a minority and most of the time there are sources available to be recompiled for 64-bit as a workaround for better performance It worth mentioning that users of 64-bit apps are, of course, unaffected. Based on the work from Nikolay Borisov, see: Link: https://lkml.org/lkml/2023/6/23/387 Andrea della Porta (4): arm64: Introduce aarch32_enabled() arm64/process: Make loading of 32bit processes depend on aarch32_enabled() arm64/entry-common: Make Aarch32 syscalls' availability depend on aarch32_enabled() arm64: Make Aarch32 emulation boot time configurable .../admin-guide/kernel-parameters.txt | 7 ++++ arch/arm64/Kconfig | 9 +++++ arch/arm64/include/asm/compat.h | 12 +++++++ arch/arm64/kernel/entry-common.c | 33 +++++++++++++++++-- arch/arm64/kernel/process.c | 2 +- 5 files changed, 59 insertions(+), 4 deletions(-)