diff mbox series

[v2] kbuild: allow setting zstd compression level for modules

Message ID 20240530210342.1540045-1-maximilian@mbosch.me (mailing list archive)
State New
Headers show
Series [v2] kbuild: allow setting zstd compression level for modules | expand

Commit Message

Maximilian Bosch May 30, 2024, 9:03 p.m. UTC 
From: "torvic9@mailbox.org" <torvic9@mailbox.org>

Zstd offers a very fine-grained control of compression ratios.
Add a Kconfig option that allows setting the desired compression
level for module compression.

Signed-off-by: Tor Vic <torvic9@mailbox.org>
Tested-by: Piotr Gorski <lucjan.lucjanov@gmail.com>
Tested-by: Maximilian Bosch <maximilian@mbosch.me>
---
 kernel/module/Kconfig    | 8 ++++++++
 scripts/Makefile.modinst | 2 +-
 2 files changed, 9 insertions(+), 1 deletion(-)

Comments

Masahiro Yamada Aug. 6, 2024, 5:29 a.m. UTC | #1 
On Fri, May 31, 2024 at 6:13 AM Maximilian Bosch <maximilian@mbosch.me> wrote:
>
> From: "torvic9@mailbox.org" <torvic9@mailbox.org>
>
> Zstd offers a very fine-grained control of compression ratios.
> Add a Kconfig option that allows setting the desired compression
> level for module compression.
>
> Signed-off-by: Tor Vic <torvic9@mailbox.org>
> Tested-by: Piotr Gorski <lucjan.lucjanov@gmail.com>
> Tested-by: Maximilian Bosch <maximilian@mbosch.me>



Please use:

$ ZSTD_CLEVEL=N  make modules_install


I hope this is available widely enough.
https://github.com/facebook/zstd/commit/6e9512a70cfe099ac6fdf4dda58b61c2e79bbfd9




> ---
>  kernel/module/Kconfig    | 8 ++++++++
>  scripts/Makefile.modinst | 2 +-
>  2 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/module/Kconfig b/kernel/module/Kconfig
> index 33a2e991f608..076b18dd3941 100644
> --- a/kernel/module/Kconfig
> +++ b/kernel/module/Kconfig
> @@ -317,6 +317,14 @@ config MODULE_COMPRESS_ZSTD
>
>  endchoice
>
> +config MODULE_COMPRESS_ZSTD_LEVEL
> +       int "Compression level (1-19)"
> +       depends on MODULE_COMPRESS_ZSTD
> +       range 1 19
> +       default 3
> +       help
> +         Compression level used by zstd for compressing modules.
> +
>  config MODULE_DECOMPRESS
>         bool "Support in-kernel module decompression"
>         depends on MODULE_COMPRESS_GZIP || MODULE_COMPRESS_XZ || MODULE_COMPRESS_ZSTD
> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> index ab0c5bd1a60f..480d47eca36a 100644
> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -101,7 +101,7 @@ quiet_cmd_gzip = GZIP    $@
>  quiet_cmd_xz = XZ      $@
>        cmd_xz = $(XZ) --lzma2=dict=2MiB -f $<
>  quiet_cmd_zstd = ZSTD    $@
> -      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
> +      cmd_zstd = $(ZSTD) -$(CONFIG_MODULE_COMPRESS_ZSTD_LEVEL) -T0 --rm -f -q $<
>
>  $(dst)/%.ko.gz: $(dst)/%.ko FORCE
>         $(call cmd,gzip)
> --
> 2.42.0
>
>
Miguel Ojeda Aug. 6, 2024, 7:50 a.m. UTC | #2 
On Thu, May 30, 2024 at 11:17 PM Maximilian Bosch <maximilian@mbosch.me> wrote:
>
> From: "torvic9@mailbox.org" <torvic9@mailbox.org>
>
> Zstd offers a very fine-grained control of compression ratios.
> Add a Kconfig option that allows setting the desired compression
> level for module compression.
>
> Signed-off-by: Tor Vic <torvic9@mailbox.org>
> Tested-by: Piotr Gorski <lucjan.lucjanov@gmail.com>
> Tested-by: Maximilian Bosch <maximilian@mbosch.me>

Missing Signed-off-by?

> +config MODULE_COMPRESS_ZSTD_LEVEL
> +       int "Compression level (1-19)"
> +       depends on MODULE_COMPRESS_ZSTD
> +       range 1 19
> +       default 3

Zstd supports levels up to 22, as well as "negative" ones (`--fast`),
right? I imagine there is a reason for the 1-19 range, but it would be
nice to mention the rationale.

Thanks!

Cheers,
Miguel
Sedat Dilek Aug. 6, 2024, 8:50 a.m. UTC | #3 
On Tue, Aug 6, 2024 at 9:53 AM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> On Thu, May 30, 2024 at 11:17 PM Maximilian Bosch <maximilian@mbosch.me> wrote:
> >
> > From: "torvic9@mailbox.org" <torvic9@mailbox.org>
> >
> > Zstd offers a very fine-grained control of compression ratios.
> > Add a Kconfig option that allows setting the desired compression
> > level for module compression.
> >
> > Signed-off-by: Tor Vic <torvic9@mailbox.org>
> > Tested-by: Piotr Gorski <lucjan.lucjanov@gmail.com>
> > Tested-by: Maximilian Bosch <maximilian@mbosch.me>
>
> Missing Signed-off-by?
>
> > +config MODULE_COMPRESS_ZSTD_LEVEL
> > +       int "Compression level (1-19)"
> > +       depends on MODULE_COMPRESS_ZSTD
> > +       range 1 19
> > +       default 3
>
> Zstd supports levels up to 22, as well as "negative" ones (`--fast`),
> right? I imagine there is a reason for the 1-19 range, but it would be
> nice to mention the rationale.
>
> Thanks!
>
> Cheers,
> Miguel
>

Hi,

I provided some numbers when using L19 (-19 option) ZSTD for
kernel-modules compression (see [1]).

The savings is approx. 25MiB for generated Debian linux-image package.
The same number in /lib/modules/<kernel-release> - L19 compressed VS.
L3 (default) compressed.
Details see [1].

What is also to mention:
You need a ZSTD version which supports L19 compression and a KMOD
version which supports appropriate kernel-decompression routines for
this and maybe a Linux-kernel version which has appropriate support
from kernel-side.
At least there should be a note about user-space and kernel-space
requirements - maybe in the KConfig help-text?

In the thread below Emil V. had some hints [2], especially now that I
understand the ZSTD_CLEVEL info.

Thanks Masahiroy:

#### Restricted usage of Environment Variables
Using environment variables to set compression/decompression
parameters has security implications. Therefore,
we intentionally restrict its usage. Currently, only `ZSTD_CLEVEL` is
supported for setting compression level.
If the value of `ZSTD_CLEVEL` is not a valid integer, it will be
ignored with a warning message.
Note that command line options will override corresponding environment
variable settings

As a 2nd NOTE:
Masahiroy advocated to remove the -T0 multithreading option and
offered/demonstrated alternatives (see [3]).

If you like you can add for this patch:

Tested-by: Sedat Dilek <sedat.dilek@dhl.com> # ZSTD compressed
kernel-modules with L19 on Linux v6.10.3 AMD64

Thanks.

Best regards,
-Sedat-

Link: https://lore.kernel.org/all/CA+icZUUQadYjAXiCNx7PmKDV20WctvnzkXC3R2F7FM4Gzcm39Q@mail.gmail.com/
[1]
Link: https://lore.kernel.org/all/CACvgo52-KbJHUde8kP5-=HdHoCFEcAt=mmy1i=MR7Er7xZz67w@mail.gmail.com/
[2]
Link: https://lore.kernel.org/all/CAK7LNASB1qFv1GdxTHvLpcJu1NbWrze5VsfFRYbFMeiFv_JPhQ@mail.gmail.com/
[3]
Link: https://github.com/facebook/zstd/commit/6e9512a70cfe099ac6fdf4dda58b61c2e79bbfd9
(Masahiroy)
Tor Vic Aug. 6, 2024, 9:07 a.m. UTC | #4 
> Masahiro Yamada <masahiroy@kernel.org> hat am 06.08.2024 05:29 GMT geschrieben:
> 
>  
> On Fri, May 31, 2024 at 6:13 AM Maximilian Bosch <maximilian@mbosch.me> wrote:
> >
> > From: "torvic9@mailbox.org" <torvic9@mailbox.org>
> >
> > Zstd offers a very fine-grained control of compression ratios.
> > Add a Kconfig option that allows setting the desired compression
> > level for module compression.
> >
> > Signed-off-by: Tor Vic <torvic9@mailbox.org>
> > Tested-by: Piotr Gorski <lucjan.lucjanov@gmail.com>
> > Tested-by: Maximilian Bosch <maximilian@mbosch.me>

On a sidenote: I do not use this patch anymore since ages, instead I simply set the preset to "-7" which IMO is a good compromise of speed and compression.

That said, I think we can drop the "-T0" parallel compression, as has recently been done with XZ.

Tor

> 
> 
> 
> Please use:
> 
> $ ZSTD_CLEVEL=N  make modules_install
> 
> 
> I hope this is available widely enough.
> https://github.com/facebook/zstd/commit/6e9512a70cfe099ac6fdf4dda58b61c2e79bbfd9
> 
> 
> 
> 
> > ---
> >  kernel/module/Kconfig    | 8 ++++++++
> >  scripts/Makefile.modinst | 2 +-
> >  2 files changed, 9 insertions(+), 1 deletion(-)
> >
> > diff --git a/kernel/module/Kconfig b/kernel/module/Kconfig
> > index 33a2e991f608..076b18dd3941 100644
> > --- a/kernel/module/Kconfig
> > +++ b/kernel/module/Kconfig
> > @@ -317,6 +317,14 @@ config MODULE_COMPRESS_ZSTD
> >
> >  endchoice
> >
> > +config MODULE_COMPRESS_ZSTD_LEVEL
> > +       int "Compression level (1-19)"
> > +       depends on MODULE_COMPRESS_ZSTD
> > +       range 1 19
> > +       default 3
> > +       help
> > +         Compression level used by zstd for compressing modules.
> > +
> >  config MODULE_DECOMPRESS
> >         bool "Support in-kernel module decompression"
> >         depends on MODULE_COMPRESS_GZIP || MODULE_COMPRESS_XZ || MODULE_COMPRESS_ZSTD
> > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> > index ab0c5bd1a60f..480d47eca36a 100644
> > --- a/scripts/Makefile.modinst
> > +++ b/scripts/Makefile.modinst
> > @@ -101,7 +101,7 @@ quiet_cmd_gzip = GZIP    $@
> >  quiet_cmd_xz = XZ      $@
> >        cmd_xz = $(XZ) --lzma2=dict=2MiB -f $<
> >  quiet_cmd_zstd = ZSTD    $@
> > -      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
> > +      cmd_zstd = $(ZSTD) -$(CONFIG_MODULE_COMPRESS_ZSTD_LEVEL) -T0 --rm -f -q $<
> >
> >  $(dst)/%.ko.gz: $(dst)/%.ko FORCE
> >         $(call cmd,gzip)
> > --
> > 2.42.0
> >
> >
> 
> 
> -- 
> Best Regards
> Masahiro Yamada
diff mbox series

Patch

diff --git a/kernel/module/Kconfig b/kernel/module/Kconfig
index 33a2e991f608..076b18dd3941 100644
--- a/kernel/module/Kconfig
+++ b/kernel/module/Kconfig
@@ -317,6 +317,14 @@  config MODULE_COMPRESS_ZSTD
 
 endchoice
 
+config MODULE_COMPRESS_ZSTD_LEVEL
+	int "Compression level (1-19)"
+	depends on MODULE_COMPRESS_ZSTD
+	range 1 19
+	default 3
+	help
+	  Compression level used by zstd for compressing modules.
+
 config MODULE_DECOMPRESS
 	bool "Support in-kernel module decompression"
 	depends on MODULE_COMPRESS_GZIP || MODULE_COMPRESS_XZ || MODULE_COMPRESS_ZSTD
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index ab0c5bd1a60f..480d47eca36a 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -101,7 +101,7 @@  quiet_cmd_gzip = GZIP    $@
 quiet_cmd_xz = XZ      $@
       cmd_xz = $(XZ) --lzma2=dict=2MiB -f $<
 quiet_cmd_zstd = ZSTD    $@
-      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
+      cmd_zstd = $(ZSTD) -$(CONFIG_MODULE_COMPRESS_ZSTD_LEVEL) -T0 --rm -f -q $<
 
 $(dst)/%.ko.gz: $(dst)/%.ko FORCE
 	$(call cmd,gzip)