| Message ID | 3d34a965-ab9c-d549-0c63-c717ab5d2edc@tweek.dk (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Use CRC32 and a 1MiB dictionary for XZ compressed modules | expand |
Hello? Anyone? Best regards, - Martin On Fri, 15 Sep 2023, Martin Nybo Andersen wrote: > Kmod is now using the kernel decompressor which doesn't handle CRC64 > and dictionaries larger than 1MiB. > > Fixes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > Signed-off-by: Martin Nybo Andersen <tweek@tweek.dk> > --- > scripts/Makefile.modinst | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > index c59cc57286ba..ffbafbd3aeea 100644 > --- a/scripts/Makefile.modinst > +++ b/scripts/Makefile.modinst > @@ -144,7 +144,7 @@ endif > quiet_cmd_gzip = GZIP $@ > cmd_gzip = $(KGZIP) -n -f $< > quiet_cmd_xz = XZ $@ > - cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< > + cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< > quiet_cmd_zstd = ZSTD $@ > cmd_zstd = $(ZSTD) -T0 --rm -f -q $< > > -- > 2.40.1 > >
> > Hello? Hello, > Anyone? > > Best regards, > - Martin > > On Fri, 15 Sep 2023, Martin Nybo Andersen wrote: > >> Kmod is now using the kernel decompressor which doesn't handle CRC64 >> and dictionaries larger than 1MiB. >> >> Fixes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 >> Signed-off-by: Martin Nybo Andersen <tweek@tweek.dk> >> --- >> scripts/Makefile.modinst | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst >> index c59cc57286ba..ffbafbd3aeea 100644 >> --- a/scripts/Makefile.modinst >> +++ b/scripts/Makefile.modinst >> @@ -144,7 +144,7 @@ endif >> quiet_cmd_gzip = GZIP $@ >> cmd_gzip = $(KGZIP) -n -f $< >> quiet_cmd_xz = XZ $@ >> - cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< >> + cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< I wonder whether it should be guarded with ifdef CONFIG_MODULE_DECOMPRESS But on the other hand, the difference between 1M and 2M is likely very small in terms of compression ratio. Cheers, Tor Vic >> quiet_cmd_zstd = ZSTD $@ >> cmd_zstd = $(ZSTD) -T0 --rm -f -q $< >> >> -- >> 2.40.1 >> >> >
On Fri, 22 Sep 2023, Tor Vic wrote: >> >> Hello? > > Hello, :) >> Anyone? >> >> Best regards, >> - Martin >> >> On Fri, 15 Sep 2023, Martin Nybo Andersen wrote: >> >>> Kmod is now using the kernel decompressor which doesn't handle CRC64 >>> and dictionaries larger than 1MiB. >>> >>> Fixes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 >>> Signed-off-by: Martin Nybo Andersen <tweek@tweek.dk> >>> --- >>> scripts/Makefile.modinst | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-) >>> >>> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst >>> index c59cc57286ba..ffbafbd3aeea 100644 >>> --- a/scripts/Makefile.modinst >>> +++ b/scripts/Makefile.modinst >>> @@ -144,7 +144,7 @@ endif >>> quiet_cmd_gzip = GZIP $@ >>> cmd_gzip = $(KGZIP) -n -f $< >>> quiet_cmd_xz = XZ $@ >>> - cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< >>> + cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< > > I wonder whether it should be guarded with > ifdef CONFIG_MODULE_DECOMPRESS > > But on the other hand, the difference between 1M and 2M is likely very small > in terms of compression ratio. That would work, since kmod decompress the modules in userspace if CONFIG_MODULE_DECOMPRESS if off. What I'm aiming for is just that the kernel should be able to decompress its own modules, when it advertises it. Cheers, Martin > Cheers, > Tor Vic > >>> quiet_cmd_zstd = ZSTD $@ >>> cmd_zstd = $(ZSTD) -T0 --rm -f -q $< >>> >>> -- >>> 2.40.1 >>> >>> >> > >
In linux-modules@v.k.o probably some more experts can comment on this: On Fri, Sep 15, 2023 at 12:15:39PM +0200, Martin Nybo Andersen wrote: > Kmod is now using the kernel decompressor which doesn't handle CRC64 > and dictionaries larger than 1MiB. > > Fixes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > Signed-off-by: Martin Nybo Andersen <tweek@tweek.dk> > --- > scripts/Makefile.modinst | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > index c59cc57286ba..ffbafbd3aeea 100644 > --- a/scripts/Makefile.modinst > +++ b/scripts/Makefile.modinst > @@ -144,7 +144,7 @@ endif > quiet_cmd_gzip = GZIP $@ > cmd_gzip = $(KGZIP) -n -f $< > quiet_cmd_xz = XZ $@ > - cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< > + cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< > quiet_cmd_zstd = ZSTD $@ > cmd_zstd = $(ZSTD) -T0 --rm -f -q $< > > -- > 2.40.1
On Fri, Sep 22, 2023 at 5:52 PM Martin Nybo Andersen <tweek@tweek.dk> wrote: > > On Fri, 22 Sep 2023, Tor Vic wrote: > > >> > >> Hello? > > > > Hello, > > :) > > >> Anyone? > >> > >> Best regards, > >> - Martin > >> > >> On Fri, 15 Sep 2023, Martin Nybo Andersen wrote: > >> > >>> Kmod is now using the kernel decompressor which doesn't handle CRC64 > >>> and dictionaries larger than 1MiB. > >>> > >>> Fixes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > >>> Signed-off-by: Martin Nybo Andersen <tweek@tweek.dk> > >>> --- > >>> scripts/Makefile.modinst | 2 +- > >>> 1 file changed, 1 insertion(+), 1 deletion(-) > >>> > >>> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > >>> index c59cc57286ba..ffbafbd3aeea 100644 > >>> --- a/scripts/Makefile.modinst > >>> +++ b/scripts/Makefile.modinst > >>> @@ -144,7 +144,7 @@ endif > >>> quiet_cmd_gzip = GZIP $@ > >>> cmd_gzip = $(KGZIP) -n -f $< > >>> quiet_cmd_xz = XZ $@ > >>> - cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< > >>> + cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< > > > > I wonder whether it should be guarded with > > ifdef CONFIG_MODULE_DECOMPRESS > > > > But on the other hand, the difference between 1M and 2M is likely very small > > in terms of compression ratio. > > That would work, since kmod decompress the modules in userspace if > CONFIG_MODULE_DECOMPRESS if off. > > What I'm aiming for is just that the kernel should be able to decompress > its own modules, when it advertises it. Hello, sorry for the delay. I am OK with picking the patch to my tree. I think we can use --check=crc32 --lzma2=dict=1MiB unconditionally. (But, if the in-kernel decompressor is improved to understand CRC64 in the future, we can loosen this restriction again.) Just small change requests. Please do not use the Fixes tag to point at the Debian bug tracker. Instead, you can do Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 or Closes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 Also, mention that this change happened in kmod commit 09c9f8c5df04 ("libkmod: Use kernel decompression when available") in the commit description might be helpful. I added Lukas to CC. He is the author of the change on kmod side. > Cheers, > Martin > > > Cheers, > > Tor Vic > > > >>> quiet_cmd_zstd = ZSTD $@ > >>> cmd_zstd = $(ZSTD) -T0 --rm -f -q $< > >>> > >>> -- > >>> 2.40.1 > >>> > >>> > >> > > > >
On Fri, 22 Sep 2023, Masahiro Yamada wrote: [...] > Hello, sorry for the delay. > > I am OK with picking the patch to my tree. Hi, That sound great :) > I think we can use --check=crc32 --lzma2=dict=1MiB > unconditionally. > > (But, if the in-kernel decompressor is improved > to understand CRC64 in the future, we can loosen > this restriction again.) Of course. > Just small change requests. > > Please do not use the Fixes tag to point at the Debian bug tracker. > > > Instead, you can do > > Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > > or > > Closes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > > > > Also, mention that this change happened in > > kmod commit 09c9f8c5df04 ("libkmod: Use kernel decompression when available") > > in the commit description might be helpful. > > > > > I added Lukas to CC. > He is the author of the change on kmod side. Is this ok? Kmod is now (since kmod commit 09c9f8c5df04 ("libkmod: Use kernel decompression when available")) using the kernel decompressor, when loading compressed modules. However, the kernel XZ decompressor is XZ Embedded, which doesn't handle CRC64 and dictionaries larger than 1MiB. Use CRC32 and 1MiB dictionary when XZ compressing and installing kernel modules. Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 Signed-off-by: Martin Nybo Andersen <tweek@tweek.dk> --- scripts/Makefile.modinst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index c59cc57286ba..ffbafbd3aeea 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -144,7 +144,7 @@ endif quiet_cmd_gzip = GZIP $@ cmd_gzip = $(KGZIP) -n -f $< quiet_cmd_xz = XZ $@ - cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< + cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< quiet_cmd_zstd = ZSTD $@ cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
On Fri, Sep 22, 2023 at 8:04 PM Martin Nybo Andersen <tweek@tweek.dk> wrote: > > > On Fri, 22 Sep 2023, Masahiro Yamada wrote: > > [...] > > > Hello, sorry for the delay. > > > > I am OK with picking the patch to my tree. > > Hi, > > That sound great :) > > > I think we can use --check=crc32 --lzma2=dict=1MiB > > unconditionally. > > > > (But, if the in-kernel decompressor is improved > > to understand CRC64 in the future, we can loosen > > this restriction again.) > > Of course. > > > Just small change requests. > > > > Please do not use the Fixes tag to point at the Debian bug tracker. > > > > > > Instead, you can do > > > > Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > > > > or > > > > Closes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > > > > > > > > Also, mention that this change happened in > > > > kmod commit 09c9f8c5df04 ("libkmod: Use kernel decompression when available") > > > > in the commit description might be helpful. > > > > > > > > > > I added Lukas to CC. > > He is the author of the change on kmod side. > > Is this ok? Yes, perfect! Thanks. > > Kmod is now (since kmod commit 09c9f8c5df04 ("libkmod: Use kernel > decompression when available")) using the kernel decompressor, when > loading compressed modules. > > However, the kernel XZ decompressor is XZ Embedded, which doesn't > handle CRC64 and dictionaries larger than 1MiB. > > Use CRC32 and 1MiB dictionary when XZ compressing and installing > kernel modules. > > Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > Signed-off-by: Martin Nybo Andersen <tweek@tweek.dk> > --- > scripts/Makefile.modinst | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > index c59cc57286ba..ffbafbd3aeea 100644 > --- a/scripts/Makefile.modinst > +++ b/scripts/Makefile.modinst > @@ -144,7 +144,7 @@ endif > quiet_cmd_gzip = GZIP $@ > cmd_gzip = $(KGZIP) -n -f $< > quiet_cmd_xz = XZ $@ > - cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< > + cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< > quiet_cmd_zstd = ZSTD $@ > cmd_zstd = $(ZSTD) -T0 --rm -f -q $< > -- > 2.40.1
On Fri, Sep 22, 2023 at 10:34 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Fri, Sep 22, 2023 at 8:04 PM Martin Nybo Andersen <tweek@tweek.dk> wrote: > > > > > > On Fri, 22 Sep 2023, Masahiro Yamada wrote: > > > > [...] > > > > > Hello, sorry for the delay. > > > > > > I am OK with picking the patch to my tree. > > > > Hi, > > > > That sound great :) > > > > > I think we can use --check=crc32 --lzma2=dict=1MiB > > > unconditionally. > > > > > > (But, if the in-kernel decompressor is improved > > > to understand CRC64 in the future, we can loosen > > > this restriction again.) > > > > Of course. > > > > > Just small change requests. > > > > > > Please do not use the Fixes tag to point at the Debian bug tracker. > > > > > > > > > Instead, you can do > > > > > > Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > > > > > > or > > > > > > Closes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 > > > > > > > > > > > > Also, mention that this change happened in > > > > > > kmod commit 09c9f8c5df04 ("libkmod: Use kernel decompression when available") > > > > > > in the commit description might be helpful. > > > > > > > > > > > > > > > I added Lukas to CC. > > > He is the author of the change on kmod side. > > > > Is this ok? > > > Yes, perfect! > > Thanks. > Applied to linux-kbuild. Thanks.
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index c59cc57286ba..ffbafbd3aeea 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -144,7 +144,7 @@ endif quiet_cmd_gzip = GZIP $@ cmd_gzip = $(KGZIP) -n -f $< quiet_cmd_xz = XZ $@ - cmd_xz = $(XZ) --lzma2=dict=2MiB -f $< + cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $< quiet_cmd_zstd = ZSTD $@ cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
Kmod is now using the kernel decompressor which doesn't handle CRC64 and dictionaries larger than 1MiB. Fixes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050582 Signed-off-by: Martin Nybo Andersen <tweek@tweek.dk> --- scripts/Makefile.modinst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.40.1