| Message ID | 20210312104859.16337-1-yunqiang.su@cipunited.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [v7,RESEND] MIPS: force use FR=0 or FRE for FPXX binaries | expand |
On Fri, Mar 12, 2021 at 10:48:59AM +0000, YunQiang Su wrote: > The MIPS FPU may have 3 mode: > FR=0: MIPS I style, all of the FPR are single. > FR=1: all 32 FPR can be double. > FRE: redirecting the rw of odd-FPR to the upper 32bit of even-double FPR. > > The binary may have 3 mode: > FP32: can only work with FR=0 and FRE mode > FPXX: can work with all of FR=0/FR=1/FRE mode. > FP64: can only work with FR=1 mode > > Some binary, for example the output of golang, may be mark as FPXX, > while in fact they are FP32. It is caused by the bug of design and linker: > Object produced by pure Go has no FP annotation while in fact they are FP32; > if we link them with the C module which marked as FPXX, > the result will be marked as FPXX. If these fake-FPXX binaries is executed > in FR=1 mode, some problem will happen. > > In Golang, now we add the FP32 annotation, so the future golang programs > won't have this problem. While for the existing binaries, we need a > kernel workaround. what about just rebuilding them ? They are broken, so why should we fix broken user binaries with kernel hacks ? > Currently, FR=1 mode is used for all FPXX binary, it makes some wrong > behivour of the binaries. Since FPXX binary can work with both FR=1 and FR=0, > we force it to use FR=0 or FRE (for R6 CPU). I'm not sure, if I want to take this patch. Maciej, what's your take on this ? Thomas.
On Mon, 15 Mar 2021, Thomas Bogendoerfer wrote: > > In Golang, now we add the FP32 annotation, so the future golang programs > > won't have this problem. While for the existing binaries, we need a > > kernel workaround. > > what about just rebuilding them ? They are broken, so why should we fix > broken user binaries with kernel hacks ? I agree. I went ahead and double-checked myself what the situation is here since I could not have otherwise obtained the answer to the question I asked, and indeed as I suspected even the simplest Go program will include a dynamic libgo reference (`libgo.so.17' with the snapshot of GCC 11 I have built for the MIPS target). So a userland workaround is as simple as relinking this single library for the FP32 model. This will make the dynamic loader force the FR=0 mode for all the executables that load the library. Since as YunQiang says they're going to rebuild Golang with FP32 annotation anyway, which will naturally apply to the dynamic libgo library as well, this will fix the problem with the existing binaries in the current distribution. Given that this is actually a correct fix (another one is required for the linker bug) I see no reason to clutter the kernel with a hack. Especially as users will have to update a component anyway, in this case the Go runtime rather than the kernel (which is better even, as you don't have to reboot). Once Golang has been modernised to use the FPXX mode the problem will go away, and given the frequent version bumps in libgo's soname the current breakage won't be an issue for whatever future version of Debian includes it as the whole distribution will of course have been rebuilt against the new library and any old broken executables kept by the user with a system upgrade will continue using the old FP32 dynamic library. > > Currently, FR=1 mode is used for all FPXX binary, it makes some wrong > > behivour of the binaries. Since FPXX binary can work with both FR=1 and FR=0, > > we force it to use FR=0 or FRE (for R6 CPU). > > I'm not sure, if I want to take this patch. > > Maciej, what's your take on this ? Given what I have written previously and especially above I maintain my objection. I don't understand why we're supposed to do people's homework though and solve their problems. Maciej
Thomas Bogendoerfer <tsbogend@alpha.franken.de> 于2021年3月15日周一 下午11:00写道： > > On Fri, Mar 12, 2021 at 10:48:59AM +0000, YunQiang Su wrote: > > The MIPS FPU may have 3 mode: > > FR=0: MIPS I style, all of the FPR are single. > > FR=1: all 32 FPR can be double. > > FRE: redirecting the rw of odd-FPR to the upper 32bit of even-double FPR. > > > > The binary may have 3 mode: > > FP32: can only work with FR=0 and FRE mode > > FPXX: can work with all of FR=0/FR=1/FRE mode. > > FP64: can only work with FR=1 mode > > > > Some binary, for example the output of golang, may be mark as FPXX, > > while in fact they are FP32. It is caused by the bug of design and linker: > > Object produced by pure Go has no FP annotation while in fact they are FP32; > > if we link them with the C module which marked as FPXX, > > the result will be marked as FPXX. If these fake-FPXX binaries is executed > > in FR=1 mode, some problem will happen. > > > > In Golang, now we add the FP32 annotation, so the future golang programs > > won't have this problem. While for the existing binaries, we need a > > kernel workaround. > > what about just rebuilding them ? They are broken, so why should we fix > broken user binaries with kernel hacks ? > In fact without O32_FP64_SUPPORT option is enabled, the FP=0 mode is always used for FPXX. In fact it doesn't change the behaviour of kernel. > > Currently, FR=1 mode is used for all FPXX binary, it makes some wrong > > behivour of the binaries. Since FPXX binary can work with both FR=1 and FR=0, > > we force it to use FR=0 or FRE (for R6 CPU). > > I'm not sure, if I want to take this patch. > In fact, I'd prefer to use a config option to control this behivour. > Maciej, what's your take on this ? > > Thomas. > > -- > Crap can work. Given enough thrust pigs will fly, but it's not necessarily a > good idea. [ RFC1925, 2.3 ]
Maciej W. Rozycki <macro@orcam.me.uk> 于2021年3月18日周四 上午7:16写道： > > On Mon, 15 Mar 2021, Thomas Bogendoerfer wrote: > > > > In Golang, now we add the FP32 annotation, so the future golang programs > > > won't have this problem. While for the existing binaries, we need a > > > kernel workaround. > > > > what about just rebuilding them ? They are broken, so why should we fix > > broken user binaries with kernel hacks ? > > I agree. > > I went ahead and double-checked myself what the situation is here since I > could not have otherwise obtained the answer to the question I asked, and > indeed as I suspected even the simplest Go program will include a dynamic > libgo reference (`libgo.so.17' with the snapshot of GCC 11 I have built > for the MIPS target). So a userland workaround is as simple as relinking > this single library for the FP32 model. This will make the dynamic loader > force the FR=0 mode for all the executables that load the library. > In fact gccgo has no problem here at all. The problem is about Google's golang: golang.org > Since as YunQiang says they're going to rebuild Golang with FP32 > annotation anyway, which will naturally apply to the dynamic libgo library > as well, this will fix the problem with the existing binaries in the > current distribution. Given that this is actually a correct fix (another > one is required for the linker bug) I see no reason to clutter the kernel > with a hack. Especially as users will have to update a component anyway, > in this case the Go runtime rather than the kernel (which is better even, > as you don't have to reboot). > Normally Go has no runtime. For Debian, we patched golang-1.15, and all go objects in Debian bullseye will be OK. While, the objects in Debian buster or previous version or other distribution may still broken. The user may need to run these application on a kernel with O32_FP64 support enabled. > Once Golang has been modernised to use the FPXX mode the problem will go > away, and given the frequent version bumps in libgo's soname the current > breakage won't be an issue for whatever future version of Debian includes > it as the whole distribution will of course have been rebuilt against the > new library and any old broken executables kept by the user with a system > upgrade will continue using the old FP32 dynamic library. > Yes. As show on commit-msg, the patches for Golang have been accepted. https://go-review.googlesource.com/c/go/+/239217 https://go-review.googlesource.com/c/go/+/237058 The bad news is that (Google's) Go has no runtime. > > > Currently, FR=1 mode is used for all FPXX binary, it makes some wrong > > > behivour of the binaries. Since FPXX binary can work with both FR=1 and FR=0, > > > we force it to use FR=0 or FRE (for R6 CPU). > > > > I'm not sure, if I want to take this patch. > > > > Maciej, what's your take on this ? > > Given what I have written previously and especially above I maintain my > objection. I don't understand why we're supposed to do people's homework > though and solve their problems. > > Maciej
Hi, On 2021/3/19 上午9:38, YunQiang Su wrote: > Maciej W. Rozycki <macro@orcam.me.uk> 于2021年3月18日周四 上午7:16写道： >> On Mon, 15 Mar 2021, Thomas Bogendoerfer wrote: >> >>>> In Golang, now we add the FP32 annotation, so the future golang programs >>>> won't have this problem. While for the existing binaries, we need a >>>> kernel workaround. >>> what about just rebuilding them ? They are broken, so why should we fix >>> broken user binaries with kernel hacks ? >> I agree. >> >> I went ahead and double-checked myself what the situation is here since I >> could not have otherwise obtained the answer to the question I asked, and >> indeed as I suspected even the simplest Go program will include a dynamic >> libgo reference (`libgo.so.17' with the snapshot of GCC 11 I have built >> for the MIPS target). So a userland workaround is as simple as relinking >> this single library for the FP32 model. This will make the dynamic loader >> force the FR=0 mode for all the executables that load the library. >> > In fact gccgo has no problem here at all. > The problem is about Google's golang: golang.org > >> Since as YunQiang says they're going to rebuild Golang with FP32 >> annotation anyway, which will naturally apply to the dynamic libgo library >> as well, this will fix the problem with the existing binaries in the >> current distribution. Given that this is actually a correct fix (another >> one is required for the linker bug) I see no reason to clutter the kernel >> with a hack. Especially as users will have to update a component anyway, >> in this case the Go runtime rather than the kernel (which is better even, >> as you don't have to reboot). >> > Normally Go has no runtime. For Debian, we patched golang-1.15, and all > go objects in Debian bullseye will be OK. > > While, the objects in Debian buster or previous version or other distribution > may still broken. > > The user may need to run these application on a kernel with O32_FP64 > support enabled. Yes, Ingenic X2000 processor has encountered this problem. In fact, all MIPS32r5 and MIPS32r6 processors may encounter this problem (because O32_FP64 is selected by default on MIPS32r5 and MIPS32r6), and we have indeed encountered this when running docker on debian10, our solution is similar to the current Yunqiang's method. Thanks and best regards! >> Once Golang has been modernised to use the FPXX mode the problem will go >> away, and given the frequent version bumps in libgo's soname the current >> breakage won't be an issue for whatever future version of Debian includes >> it as the whole distribution will of course have been rebuilt against the >> new library and any old broken executables kept by the user with a system >> upgrade will continue using the old FP32 dynamic library. >> > Yes. As show on commit-msg, the patches for Golang have been accepted. > https://go-review.googlesource.com/c/go/+/239217 > https://go-review.googlesource.com/c/go/+/237058 > > The bad news is that (Google's) Go has no runtime. > >>>> Currently, FR=1 mode is used for all FPXX binary, it makes some wrong >>>> behivour of the binaries. Since FPXX binary can work with both FR=1 and FR=0, >>>> we force it to use FR=0 or FRE (for R6 CPU). >>> I'm not sure, if I want to take this patch. >>> >>> Maciej, what's your take on this ? >> Given what I have written previously and especially above I maintain my >> objection. I don't understand why we're supposed to do people's homework >> though and solve their problems. >> >> Maciej > >
On Fri, 19 Mar 2021, YunQiang Su wrote:
> The bad news is that (Google's) Go has no runtime.
Dynamic shared objects (libraries) were invented in early 1990s for two
reasons:
1. To limit the use of virtual memory. Memory conservation may not be as
important nowadays in many applications where vast amounts of RAM are
available, though of course this does not apply everywhere, and still
it has to be weighed up whether any waste of resources is justified and
compensated by a gain elsewhere.
2. To make it easy to replace a piece of code shared among many programs,
so that you don't have to relink them all (or recompile if sources are
available) when say an issue is found or a feature is added that is
transparent to applications (for instance a new protocol or a better
algorithm). This still stands very much nowadays.
People went through great efforts to support shared libraries, sacrificed
performance for it even back then when the computing power was much lower
than nowadays. Support was implemented in Linux for the a.out binary
format even, despite the need to go through horrible hoops to get a.out
shared libraries built. Some COFF environments were adapted for shared
library support too.
I don't know why Google choose not to have their runtime support library
(the Go library) as a dynamic shared object 20-something years on, but it
comes at a price. So you either have to relink (recompile) all the
affected applications like in the old days or find a feasible workaround.
As I noted in the discussion the use of FR=0 would be acceptable for FPXX
binaries as far as I am concerned for R2 through R5, but not the FRE mode
for R6.
Maciej
On Fri, 19 Mar 2021, YunQiang Su wrote: > > what about just rebuilding them ? They are broken, so why should we fix > > broken user binaries with kernel hacks ? > > > > In fact without O32_FP64_SUPPORT option is enabled, the FP=0 mode is > always used for FPXX. As I noted in the discussion the choice of the FR mode for FPXX binaries is neutral performance-wise for R2 through R5, so as I previously stated it would be fine with me to use FR=0 rather than FR=1 for FPXX binaries with these architecture levels. > In fact it doesn't change the behaviour of kernel. That is not true for R6. The use of the FRE mode with R6 does regress support for FPXX binaries performance-wise, which makes it unacceptable. And O32_FP64_SUPPORT is unconditionally selected for R6, so FR=0 (or FRE) is currently not used for FPXX with R6. Maciej
> -----邮件原件----- > 发件人: Maciej W. Rozycki <macro@orcam.me.uk> > 发送时间: 2021年3月19日 22:32 > 收件人: YunQiang Su <wzssyqa@gmail.com> > 抄送: Thomas Bogendoerfer <tsbogend@alpha.franken.de>; YunQiang Su > <yunqiang.su@cipunited.com>; linux-mips <linux-mips@vger.kernel.org>; > Jiaxun Yang <jiaxun.yang@flygoat.com>; Philippe Mathieu-Daudé > <f4bug@amsat.org>; stable@vger.kernel.org > 主题: Re: [PATCH v7 RESEND] MIPS: force use FR=0 or FRE for FPXX binaries > > On Fri, 19 Mar 2021, YunQiang Su wrote: > > > The bad news is that (Google's) Go has no runtime. > > Dynamic shared objects (libraries) were invented in early 1990s for two > reasons: > > 1. To limit the use of virtual memory. Memory conservation may not be as > important nowadays in many applications where vast amounts of RAM are > available, though of course this does not apply everywhere, and still > it has to be weighed up whether any waste of resources is justified and > compensated by a gain elsewhere. > > 2. To make it easy to replace a piece of code shared among many programs, > so that you don't have to relink them all (or recompile if sources are > available) when say an issue is found or a feature is added that is > transparent to applications (for instance a new protocol or a better > algorithm). This still stands very much nowadays. > > People went through great efforts to support shared libraries, sacrificed > performance for it even back then when the computing power was much > lower than nowadays. Support was implemented in Linux for the a.out > binary format even, despite the need to go through horrible hoops to get a.out > shared libraries built. Some COFF environments were adapted for shared > library support too. > > I don't know why Google choose not to have their runtime support library > (the Go library) as a dynamic shared object 20-something years on, but it > comes at a price. So you either have to relink (recompile) all the affected > applications like in the old days or find a feasible workaround. > I also have no idea why (even hate). While there do be some program languages created in recently years, prefer static link. > As I noted in the discussion the use of FR=0 would be acceptable for FPXX > binaries as far as I am concerned for R2 through R5, but not the FRE mode for > R6. There will no FPXX for r6. All of (if not mistake) R6 O32 is FP64. FRE here is only for compatible with pre-R6 objects. I will send a V8 to switch r6 back. > > Maciej
On Mon, 22 Mar 2021, yunqiang.su@cipunited.com wrote: > > I don't know why Google choose not to have their runtime support library > > (the Go library) as a dynamic shared object 20-something years on, but it > > comes at a price. So you either have to relink (recompile) all the > affected > > applications like in the old days or find a feasible workaround. > > > > I also have no idea why (even hate). > While there do be some program languages created in recently years, prefer > static link. Hmm, lost wisdom, or an orchestrated effort? Or a false illusion that since we're virtually fully open source now, we can always rebuild the world? Well, indeed this is technically possible, but whether it is feasible is another matter. Your case serves as a counterexample. > > As I noted in the discussion the use of FR=0 would be acceptable for FPXX > > binaries as far as I am concerned for R2 through R5, but not the FRE mode > for > > R6. > > There will no FPXX for r6. All of (if not mistake) R6 O32 is FP64. > FRE here is only for compatible with pre-R6 objects. That doesn't seem like a good choice to me. While R6 programs are indeed best built as FP64, libraries are best built as FPXX, so that users can link or load with whatever binary modules they have, including pre-R6 ones. As much as we may dislike it sources will not always be available or rebuilding them may be beyond the capabilities of whoever has the binaries, so I think the system should be as permissive as possible. So you may end up with running code that is largely R6 (libraries), and partly pre-R6 (application code) that ends up linked as FPXX. And the kernel has to support it in the best way possible too and avoid slow emulation where not necessary e.g. in R6 libm code used in the FPXX arrangement, which the FRE mode will inevitably lead to. Maciej
> -----邮件原件----- > 发件人: Maciej W. Rozycki <macro@orcam.me.uk> > 发送时间: 2021年3月29日 23:06 > 收件人: yunqiang.su@cipunited.com > 抄送: 'YunQiang Su' <wzssyqa@gmail.com>; 'Thomas Bogendoerfer' > <tsbogend@alpha.franken.de>; 'linux-mips' <linux-mips@vger.kernel.org>; > 'Jiaxun Yang' <jiaxun.yang@flygoat.com>; 'Philippe Mathieu-Daudé' > <f4bug@amsat.org>; stable@vger.kernel.org > 主题: Re: 回复: [PATCH v7 RESEND] MIPS: force use FR=0 or FRE for FPXX > binaries > > On Mon, 22 Mar 2021, yunqiang.su@cipunited.com wrote: > > > > I don't know why Google choose not to have their runtime support > > > library (the Go library) as a dynamic shared object 20-something > > > years on, but it comes at a price. So you either have to relink > > > (recompile) all the > > affected > > > applications like in the old days or find a feasible workaround. > > > > > > > I also have no idea why (even hate). > > While there do be some program languages created in recently years, > > prefer static link. > > Hmm, lost wisdom, or an orchestrated effort? Or a false illusion that since > we're virtually fully open source now, we can always rebuild the world? Well, > indeed this is technically possible, but whether it is feasible is another matter. > Your case serves as a counterexample. > > > > As I noted in the discussion the use of FR=0 would be acceptable > > > for FPXX binaries as far as I am concerned for R2 through R5, but > > > not the FRE mode > > for > > > R6. > > > > There will no FPXX for r6. All of (if not mistake) R6 O32 is FP64. > > FRE here is only for compatible with pre-R6 objects. > > That doesn't seem like a good choice to me. > > While R6 programs are indeed best built as FP64, libraries are best built as > FPXX, so that users can link or load with whatever binary modules they have, > including pre-R6 ones. As much as we may dislike it sources will not always > be available or rebuilding them may be beyond the capabilities of whoever > has the binaries, so I think the system should be as permissive as possible. > So you may end up with running code that is largely R6 (libraries), and partly > pre-R6 (application code) that ends up linked as FPXX. > Yes. It is the situation we talk about R6 in early days. While after some talk, we decide to figure out pure R6 systems. And I agree with your concern, since the Android is such an example: 64bit is R6, and 32bit is R2. So, the compatible of R2 object on R6 CPU is some important. > And the kernel has to support it in the best way possible too and avoid slow > emulation where not necessary e.g. in R6 libm code used in the FPXX > arrangement, which the FRE mode will inevitably lead to. > Yes. > Maciej
diff --git a/arch/mips/kernel/elf.c b/arch/mips/kernel/elf.c index 7b045d2a0b51..4d4db619544b 100644 --- a/arch/mips/kernel/elf.c +++ b/arch/mips/kernel/elf.c @@ -232,11 +232,16 @@ int arch_check_elf(void *_ehdr, bool has_interpreter, void *_interp_ehdr, * that inherently require the hybrid FP mode. * - If FR1 and FRDEFAULT is true, that means we hit the any-abi or * fpxx case. This is because, in any-ABI (or no-ABI) we have no FPU - * instructions so we don't care about the mode. We will simply use - * the one preferred by the hardware. In fpxx case, that ABI can - * handle both FR=1 and FR=0, so, again, we simply choose the one - * preferred by the hardware. Next, if we only use single-precision - * FPU instructions, and the default ABI FPU mode is not good + * instructions so we don't care about the mode. + * In fpxx case, that ABI can handle all of FR=1/FR=0/FRE mode. + * Here, we need to use FR=0/FRE mode instead of FR=1, because some binaries + * may be mark as FPXX by mistake due to bugs of design and linker: + * The object produced by pure Go has no FP annotation, + * then is treated as any-ABI by linker, although in fact they are FP32; + * if any-ABI object is linked with FPXX object, the result will be mark as FPXX. + * Then the problem happens: run FP32 binaries in FR=1 mode. + * - If we only use single-precision FPU instructions, + * and the default ABI FPU mode is not good * (ie single + any ABI combination), we set again the FPU mode to the * one is preferred by the hardware. Next, if we know that the code * will only use single-precision instructions, shown by single being @@ -248,8 +253,9 @@ int arch_check_elf(void *_ehdr, bool has_interpreter, void *_interp_ehdr, */ if (prog_req.fre && !prog_req.frdefault && !prog_req.fr1) state->overall_fp_mode = FP_FRE; - else if ((prog_req.fr1 && prog_req.frdefault) || - (prog_req.single && !prog_req.frdefault)) + else if (prog_req.fr1 && prog_req.frdefault) + state->overall_fp_mode = cpu_has_mips_r6 ? FP_FRE : FP_FR0; + else if (prog_req.single && !prog_req.frdefault) /* Make sure 64-bit MIPS III/IV/64R1 will not pick FR1 */ state->overall_fp_mode = ((raw_current_cpu_data.fpu_id & MIPS_FPIR_F64) && cpu_has_mips_r2_r6) ?
The MIPS FPU may have 3 mode: FR=0: MIPS I style, all of the FPR are single. FR=1: all 32 FPR can be double. FRE: redirecting the rw of odd-FPR to the upper 32bit of even-double FPR. The binary may have 3 mode: FP32: can only work with FR=0 and FRE mode FPXX: can work with all of FR=0/FR=1/FRE mode. FP64: can only work with FR=1 mode Some binary, for example the output of golang, may be mark as FPXX, while in fact they are FP32. It is caused by the bug of design and linker: Object produced by pure Go has no FP annotation while in fact they are FP32; if we link them with the C module which marked as FPXX, the result will be marked as FPXX. If these fake-FPXX binaries is executed in FR=1 mode, some problem will happen. In Golang, now we add the FP32 annotation, so the future golang programs won't have this problem. While for the existing binaries, we need a kernel workaround. Currently, FR=1 mode is used for all FPXX binary, it makes some wrong behivour of the binaries. Since FPXX binary can work with both FR=1 and FR=0, we force it to use FR=0 or FRE (for R6 CPU). Reference: https://web.archive.org/web/20180828210612/https://dmz-portal.mips.com/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking https://go-review.googlesource.com/c/go/+/239217 https://go-review.googlesource.com/c/go/+/237058 Signed-off-by: YunQiang Su <yunqiang.su@cipunited.com> Cc: stable@vger.kernel.org # 4.19+ --- v6->v7: Use FRE mode for pre-R6 binaries on R6 CPU. v5->v6: Rollback to V3, aka remove config option. v4->v5: Fix CONFIG_MIPS_O32_FPXX_USE_FR0 usage: if -> ifdef v3->v4: introduce a config option: CONFIG_MIPS_O32_FPXX_USE_FR0 v2->v3: commit message: add Signed-off-by and Cc to stable. v1->v2: Fix bad commit message: in fact, we are switching to FR=0 arch/mips/kernel/elf.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-)