| Message ID | 20240722090622.16524-2-petr.pavlu@suse.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | module: Split modules_install compression and in-kernel decompression | expand |
On Mon, Jul 22, 2024 at 6:07 PM Petr Pavlu <petr.pavlu@suse.com> wrote: > > The kernel configuration allows specifying a module compression mode. If > one is selected then each module gets compressed during > 'make modules_install' and additionally one can also enable support for > a respective direct in-kernel decompression support. This means that the > decompression support cannot be enabled without the automatic compression. > > Some distributions, such as the (open)SUSE family, use a signer service for > modules. A build runs on a worker machine but signing is done by a separate > locked-down server that is in possession of the signing key. The build > invokes 'make modules_install' to create a modules tree, collects > information about the modules, asks the signer service for their signature, > appends each signature to the respective module and compresses all modules. > > When using this arrangment, the 'make modules_install' step produces > unsigned+uncompressed modules and the distribution's own build recipe takes > care of signing and compression later. > > The signing support can be currently enabled without automatically signing > modules during 'make modules_install'. However, the in-kernel decompression > support can be selected only after first enabling automatic compression > during this step. > > To allow only enabling the in-kernel decompression support without the > automatic compression during 'make modules_install', separate the > compression options similarly to the signing options, as follows: > > > Enable loadable module support > [*] Module compression > Module compression type (GZIP) ---> > [*] Automatically compress all modules > [ ] Support in-kernel module decompression > > * "Module compression" (MODULE_COMPRESS) is a new main switch for the > compression/decompression support. It replaces MODULE_COMPRESS_NONE. > * "Module compression type" (MODULE_COMPRESS_<type>) chooses the > compression type, one of GZ, XZ, ZSTD. > * "Automatically compress all modules" (MODULE_COMPRESS_ALL) is a new > option to enable module compression during 'make modules_install'. It > defaults to Y. > * "Support in-kernel module decompression" (MODULE_DECOMPRESS) enables > in-kernel decompression. > > Signed-off-by: Petr Pavlu <petr.pavlu@suse.com> > --- My preference is to add CONFIG_MODULE_DECOMPRESS_GZIP CONFIG_MODULE_DECOMPRESS_XZ CONFIG_MODULE_DECOMPRESS_ZSTD instead of CONFIG_MODULE_COMPRESS_ALL. For example, if MODULE_DECOMPRESS config MODULE_DECOMPRESS_GZIP bool "Support in-kernel GZIP decompression for module" default MODULE_COMPRESS_GZIP config MODULE_DECOMPRESS_XZ bool "Support in-kernel XZ decompression for module" default MODULE_COMPRESS_XZ config MODULE_DECOMPRESS_ZSTD bool "Support in-kernel ZSTD decompression for module" default MODULE_COMPRESS_ZSTD endif OR, maybe config MODULE_DECOMPRESS_GZIP bool "Support in-kernel GZIP decompression for module" select MODULE_DECOMPRESS config MODULE_DECOMPRESS_XZ bool "Support in-kernel XZ decompression for module" select MODULE_DECOMPRESS config MODULE_DECOMPRESS_ZSTD bool "Support in-kernel ZSTD decompression for module" select MODULE_DECOMPRESS config MODULE_DECOMPRESS bool You can toggle MODULE_COMPRESS_GZIP and MODULE_DECOMPRESS_GZIP independently Of course, the current kernel/module/decompress.c does not work when multiple (or zero) CONFIG_MODULE_DECOMPRESS_* is enabled. It needs a little modification. I will wait for Lius's comment. > kernel/module/Kconfig | 61 ++++++++++++++++++++-------------------- > scripts/Makefile.modinst | 2 ++ > 2 files changed, 33 insertions(+), 30 deletions(-) > > diff --git a/kernel/module/Kconfig b/kernel/module/Kconfig > index 4047b6d48255..bb7f7930fef6 100644 > --- a/kernel/module/Kconfig > +++ b/kernel/module/Kconfig > @@ -278,64 +278,65 @@ config MODULE_SIG_HASH > default "sha3-384" if MODULE_SIG_SHA3_384 > default "sha3-512" if MODULE_SIG_SHA3_512 > > -choice > - prompt "Module compression mode" > +config MODULE_COMPRESS > + bool "Module compression" > help > - This option allows you to choose the algorithm which will be used to > - compress modules when 'make modules_install' is run. (or, you can > - choose to not compress modules at all.) > - > - External modules will also be compressed in the same way during the > - installation. > - > - For modules inside an initrd or initramfs, it's more efficient to > - compress the whole initrd or initramfs instead. > - > + Enable module compression to reduce on-disk size of module binaries. > This is fully compatible with signed modules. > > - Please note that the tool used to load modules needs to support the > - corresponding algorithm. module-init-tools MAY support gzip, and kmod > - MAY support gzip, xz and zstd. > + The tool used to work with modules needs to support the selected > + compression type. kmod MAY support gzip, xz and zstd. Other tools > + might have a limited selection of the supported types. > > - Your build system needs to provide the appropriate compression tool > - to compress the modules. > + Note that for modules inside an initrd or initramfs, it's more > + efficient to compress the whole ramdisk instead. > > - If in doubt, select 'None'. > + If unsure, say N. > > -config MODULE_COMPRESS_NONE > - bool "None" > +choice > + prompt "Module compression type" > + depends on MODULE_COMPRESS > help > - Do not compress modules. The installed modules are suffixed > - with .ko. > + Choose the supported algorithm for module compression. > > config MODULE_COMPRESS_GZIP > bool "GZIP" > help > - Compress modules with GZIP. The installed modules are suffixed > - with .ko.gz. > + Support modules compressed with GZIP. The installed modules are > + suffixed with .ko.gz. > > config MODULE_COMPRESS_XZ > bool "XZ" > help > - Compress modules with XZ. The installed modules are suffixed > - with .ko.xz. > + Support modules compressed with XZ. The installed modules are > + suffixed with .ko.xz. > > config MODULE_COMPRESS_ZSTD > bool "ZSTD" > help > - Compress modules with ZSTD. The installed modules are suffixed > - with .ko.zst. > + Support modules compressed with ZSTD. The installed modules are > + suffixed with .ko.zst. > > endchoice > > +config MODULE_COMPRESS_ALL > + bool "Automatically compress all modules" > + default y > + depends on MODULE_COMPRESS > + help > + Compress all modules during 'make modules_install'. > + > + Your build system needs to provide the appropriate compression tool > + for the selected compression type. External modules will also be > + compressed in the same way during the installation. > + > config MODULE_DECOMPRESS > bool "Support in-kernel module decompression" > - depends on MODULE_COMPRESS_GZIP || MODULE_COMPRESS_XZ || MODULE_COMPRESS_ZSTD > + depends on MODULE_COMPRESS > select ZLIB_INFLATE if MODULE_COMPRESS_GZIP > select XZ_DEC if MODULE_COMPRESS_XZ > select ZSTD_DECOMPRESS if MODULE_COMPRESS_ZSTD > help > - > Support for decompressing kernel modules by the kernel itself > instead of relying on userspace to perform this task. Useful when > load pinning security policy is enabled. > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst > index 0afd75472679..bce4a9adb893 100644 > --- a/scripts/Makefile.modinst > +++ b/scripts/Makefile.modinst > @@ -51,9 +51,11 @@ $(foreach x, % :, $(if $(findstring $x, $(dst)), \ > $(error module installation path cannot contain '$x'))) > > suffix-y := > +ifdef CONFIG_MODULE_COMPRESS_ALL > suffix-$(CONFIG_MODULE_COMPRESS_GZIP) := .gz > suffix-$(CONFIG_MODULE_COMPRESS_XZ) := .xz > suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst > +endif > > modules := $(patsubst $(extmod_prefix)%.o, $(dst)/%.ko$(suffix-y), $(modules)) > install-$(CONFIG_MODULES) += $(modules) > -- > 2.35.3 >
On 7/22/24 12:23, Masahiro Yamada wrote: > On Mon, Jul 22, 2024 at 6:07 PM Petr Pavlu <petr.pavlu@suse.com> wrote: >> >> The kernel configuration allows specifying a module compression mode. If >> one is selected then each module gets compressed during >> 'make modules_install' and additionally one can also enable support for >> a respective direct in-kernel decompression support. This means that the >> decompression support cannot be enabled without the automatic compression. >> >> Some distributions, such as the (open)SUSE family, use a signer service for >> modules. A build runs on a worker machine but signing is done by a separate >> locked-down server that is in possession of the signing key. The build >> invokes 'make modules_install' to create a modules tree, collects >> information about the modules, asks the signer service for their signature, >> appends each signature to the respective module and compresses all modules. >> >> When using this arrangment, the 'make modules_install' step produces >> unsigned+uncompressed modules and the distribution's own build recipe takes >> care of signing and compression later. >> >> The signing support can be currently enabled without automatically signing >> modules during 'make modules_install'. However, the in-kernel decompression >> support can be selected only after first enabling automatic compression >> during this step. >> >> To allow only enabling the in-kernel decompression support without the >> automatic compression during 'make modules_install', separate the >> compression options similarly to the signing options, as follows: >> >>> Enable loadable module support >> [*] Module compression >> Module compression type (GZIP) ---> >> [*] Automatically compress all modules >> [ ] Support in-kernel module decompression >> >> * "Module compression" (MODULE_COMPRESS) is a new main switch for the >> compression/decompression support. It replaces MODULE_COMPRESS_NONE. >> * "Module compression type" (MODULE_COMPRESS_<type>) chooses the >> compression type, one of GZ, XZ, ZSTD. >> * "Automatically compress all modules" (MODULE_COMPRESS_ALL) is a new >> option to enable module compression during 'make modules_install'. It >> defaults to Y. >> * "Support in-kernel module decompression" (MODULE_DECOMPRESS) enables >> in-kernel decompression. >> >> Signed-off-by: Petr Pavlu <petr.pavlu@suse.com> >> --- > > > > My preference is to add > CONFIG_MODULE_DECOMPRESS_GZIP > CONFIG_MODULE_DECOMPRESS_XZ > CONFIG_MODULE_DECOMPRESS_ZSTD > instead of > CONFIG_MODULE_COMPRESS_ALL. > > > > > For example, > > > if MODULE_DECOMPRESS > > config MODULE_DECOMPRESS_GZIP > bool "Support in-kernel GZIP decompression for module" > default MODULE_COMPRESS_GZIP > > config MODULE_DECOMPRESS_XZ > bool "Support in-kernel XZ decompression for module" > default MODULE_COMPRESS_XZ > > config MODULE_DECOMPRESS_ZSTD > bool "Support in-kernel ZSTD decompression for module" > default MODULE_COMPRESS_ZSTD > > endif > > > > > > OR, maybe > > > > config MODULE_DECOMPRESS_GZIP > bool "Support in-kernel GZIP decompression for module" > select MODULE_DECOMPRESS > > config MODULE_DECOMPRESS_XZ > bool "Support in-kernel XZ decompression for module" > select MODULE_DECOMPRESS > > config MODULE_DECOMPRESS_ZSTD > bool "Support in-kernel ZSTD decompression for module" > select MODULE_DECOMPRESS > > config MODULE_DECOMPRESS > bool > > > > > You can toggle MODULE_COMPRESS_GZIP and > MODULE_DECOMPRESS_GZIP independently I can implement this, but what would be a use case to enable multiple module decompression types in the kernel? > > > Of course, the current kernel/module/decompress.c does not > work when multiple (or zero) CONFIG_MODULE_DECOMPRESS_* is > enabled. It needs a little modification. One issue is with the file /sys/module/compression which shows the module decompression type supported by the kernel. If multiple types are allowed then I think they should all get listed there. This could however create some compatibility problems. For instance, kmod reads this file and currently expects to find exactly one type, so it would need updating as well. Thanks, Petr
On Thu, Jul 25, 2024 at 9:59 PM Petr Pavlu <petr.pavlu@suse.com> wrote: > > On 7/22/24 12:23, Masahiro Yamada wrote: > > On Mon, Jul 22, 2024 at 6:07 PM Petr Pavlu <petr.pavlu@suse.com> wrote: > >> > >> The kernel configuration allows specifying a module compression mode. If > >> one is selected then each module gets compressed during > >> 'make modules_install' and additionally one can also enable support for > >> a respective direct in-kernel decompression support. This means that the > >> decompression support cannot be enabled without the automatic compression. > >> > >> Some distributions, such as the (open)SUSE family, use a signer service for > >> modules. A build runs on a worker machine but signing is done by a separate > >> locked-down server that is in possession of the signing key. The build > >> invokes 'make modules_install' to create a modules tree, collects > >> information about the modules, asks the signer service for their signature, > >> appends each signature to the respective module and compresses all modules. > >> > >> When using this arrangment, the 'make modules_install' step produces > >> unsigned+uncompressed modules and the distribution's own build recipe takes > >> care of signing and compression later. > >> > >> The signing support can be currently enabled without automatically signing > >> modules during 'make modules_install'. However, the in-kernel decompression > >> support can be selected only after first enabling automatic compression > >> during this step. > >> > >> To allow only enabling the in-kernel decompression support without the > >> automatic compression during 'make modules_install', separate the > >> compression options similarly to the signing options, as follows: > >> > >>> Enable loadable module support > >> [*] Module compression > >> Module compression type (GZIP) ---> > >> [*] Automatically compress all modules > >> [ ] Support in-kernel module decompression > >> > >> * "Module compression" (MODULE_COMPRESS) is a new main switch for the > >> compression/decompression support. It replaces MODULE_COMPRESS_NONE. > >> * "Module compression type" (MODULE_COMPRESS_<type>) chooses the > >> compression type, one of GZ, XZ, ZSTD. > >> * "Automatically compress all modules" (MODULE_COMPRESS_ALL) is a new > >> option to enable module compression during 'make modules_install'. It > >> defaults to Y. > >> * "Support in-kernel module decompression" (MODULE_DECOMPRESS) enables > >> in-kernel decompression. > >> > >> Signed-off-by: Petr Pavlu <petr.pavlu@suse.com> > >> --- > > > > > > > > My preference is to add > > CONFIG_MODULE_DECOMPRESS_GZIP > > CONFIG_MODULE_DECOMPRESS_XZ > > CONFIG_MODULE_DECOMPRESS_ZSTD > > instead of > > CONFIG_MODULE_COMPRESS_ALL. > > > > > > > > > > For example, > > > > > > if MODULE_DECOMPRESS > > > > config MODULE_DECOMPRESS_GZIP > > bool "Support in-kernel GZIP decompression for module" > > default MODULE_COMPRESS_GZIP > > > > config MODULE_DECOMPRESS_XZ > > bool "Support in-kernel XZ decompression for module" > > default MODULE_COMPRESS_XZ > > > > config MODULE_DECOMPRESS_ZSTD > > bool "Support in-kernel ZSTD decompression for module" > > default MODULE_COMPRESS_ZSTD > > > > endif > > > > > > > > > > > > OR, maybe > > > > > > > > config MODULE_DECOMPRESS_GZIP > > bool "Support in-kernel GZIP decompression for module" > > select MODULE_DECOMPRESS > > > > config MODULE_DECOMPRESS_XZ > > bool "Support in-kernel XZ decompression for module" > > select MODULE_DECOMPRESS > > > > config MODULE_DECOMPRESS_ZSTD > > bool "Support in-kernel ZSTD decompression for module" > > select MODULE_DECOMPRESS > > > > config MODULE_DECOMPRESS > > bool > > > > > > > > > > You can toggle MODULE_COMPRESS_GZIP and > > MODULE_DECOMPRESS_GZIP independently > > I can implement this, but what would be a use case to enable multiple module > decompression types in the kernel? I just thought there is a possibility where the singer service A compresses modules in GZIP, and the singer service B in XZ, etc. If the compression type is predictable at the Kbuild time, it is fine. > > > > > > > Of course, the current kernel/module/decompress.c does not > > work when multiple (or zero) CONFIG_MODULE_DECOMPRESS_* is > > enabled. It needs a little modification. > > One issue is with the file /sys/module/compression which shows the module > decompression type supported by the kernel. If multiple types are allowed then > I think they should all get listed there. This could however create some > compatibility problems. For instance, kmod reads this file and currently > expects to find exactly one type, so it would need updating as well. OK, understood. Then, Acked-by: Masahiro Yamada <masahiroy@kernel.org> -- Best Regards Masahiro Yamada
diff --git a/kernel/module/Kconfig b/kernel/module/Kconfig index 4047b6d48255..bb7f7930fef6 100644 --- a/kernel/module/Kconfig +++ b/kernel/module/Kconfig @@ -278,64 +278,65 @@ config MODULE_SIG_HASH default "sha3-384" if MODULE_SIG_SHA3_384 default "sha3-512" if MODULE_SIG_SHA3_512 -choice - prompt "Module compression mode" +config MODULE_COMPRESS + bool "Module compression" help - This option allows you to choose the algorithm which will be used to - compress modules when 'make modules_install' is run. (or, you can - choose to not compress modules at all.) - - External modules will also be compressed in the same way during the - installation. - - For modules inside an initrd or initramfs, it's more efficient to - compress the whole initrd or initramfs instead. - + Enable module compression to reduce on-disk size of module binaries. This is fully compatible with signed modules. - Please note that the tool used to load modules needs to support the - corresponding algorithm. module-init-tools MAY support gzip, and kmod - MAY support gzip, xz and zstd. + The tool used to work with modules needs to support the selected + compression type. kmod MAY support gzip, xz and zstd. Other tools + might have a limited selection of the supported types. - Your build system needs to provide the appropriate compression tool - to compress the modules. + Note that for modules inside an initrd or initramfs, it's more + efficient to compress the whole ramdisk instead. - If in doubt, select 'None'. + If unsure, say N. -config MODULE_COMPRESS_NONE - bool "None" +choice + prompt "Module compression type" + depends on MODULE_COMPRESS help - Do not compress modules. The installed modules are suffixed - with .ko. + Choose the supported algorithm for module compression. config MODULE_COMPRESS_GZIP bool "GZIP" help - Compress modules with GZIP. The installed modules are suffixed - with .ko.gz. + Support modules compressed with GZIP. The installed modules are + suffixed with .ko.gz. config MODULE_COMPRESS_XZ bool "XZ" help - Compress modules with XZ. The installed modules are suffixed - with .ko.xz. + Support modules compressed with XZ. The installed modules are + suffixed with .ko.xz. config MODULE_COMPRESS_ZSTD bool "ZSTD" help - Compress modules with ZSTD. The installed modules are suffixed - with .ko.zst. + Support modules compressed with ZSTD. The installed modules are + suffixed with .ko.zst. endchoice +config MODULE_COMPRESS_ALL + bool "Automatically compress all modules" + default y + depends on MODULE_COMPRESS + help + Compress all modules during 'make modules_install'. + + Your build system needs to provide the appropriate compression tool + for the selected compression type. External modules will also be + compressed in the same way during the installation. + config MODULE_DECOMPRESS bool "Support in-kernel module decompression" - depends on MODULE_COMPRESS_GZIP || MODULE_COMPRESS_XZ || MODULE_COMPRESS_ZSTD + depends on MODULE_COMPRESS select ZLIB_INFLATE if MODULE_COMPRESS_GZIP select XZ_DEC if MODULE_COMPRESS_XZ select ZSTD_DECOMPRESS if MODULE_COMPRESS_ZSTD help - Support for decompressing kernel modules by the kernel itself instead of relying on userspace to perform this task. Useful when load pinning security policy is enabled. diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst index 0afd75472679..bce4a9adb893 100644 --- a/scripts/Makefile.modinst +++ b/scripts/Makefile.modinst @@ -51,9 +51,11 @@ $(foreach x, % :, $(if $(findstring $x, $(dst)), \ $(error module installation path cannot contain '$x'))) suffix-y := +ifdef CONFIG_MODULE_COMPRESS_ALL suffix-$(CONFIG_MODULE_COMPRESS_GZIP) := .gz suffix-$(CONFIG_MODULE_COMPRESS_XZ) := .xz suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst +endif modules := $(patsubst $(extmod_prefix)%.o, $(dst)/%.ko$(suffix-y), $(modules)) install-$(CONFIG_MODULES) += $(modules)