| Message ID | 20220213074335.15774-1-lucas.demarchi@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v2] libkmod: Prefer builtin index over builtin.alias | expand |
On Sat, Feb 12, 2022 at 09:43:35PM -1000, Lucas De Marchi wrote: > The modules.builtin.alias.bin is way larger than the > modules.builtin.bin. On a normal "distro kernel": > > 21k modules.builtin.alias.bin > 11k modules.builtin.bin > > >From the kernel we get both modules.builtin and modules.builtin.modinfo. > depmod generates modules.builtin.bin and modules.builtin.alias.bin > from them respectively. modules.bultin is not going away: it's not > deprecated by the new index added. So, let's just stop duplicating the > information inside modules.builtin.alias.bin and just use the other > index. The modules.builtin contains only module names. The modules.builtin.modinfo contains full info about builtin modules including names. I thought that if there is complete information about the modules, then reading the index with only the names does not make sense. And only in the absence of modules.builtin.modinfo, you need to fallback to the index with the names. > --- > libkmod/libkmod-module.c | 12 +++++++----- > 1 file changed, 7 insertions(+), 5 deletions(-) > > diff --git a/libkmod/libkmod-module.c b/libkmod/libkmod-module.c > index 6f7747c..6423339 100644 > --- a/libkmod/libkmod-module.c > +++ b/libkmod/libkmod-module.c > @@ -576,13 +576,15 @@ KMOD_EXPORT int kmod_module_new_from_lookup(struct kmod_ctx *ctx, > err = kmod_lookup_alias_from_aliases_file(ctx, alias, list); > CHECK_ERR_AND_FINISH(err, fail, list, finish); > > + DBG(ctx, "lookup modules.builtin %s\n", alias); > + err = kmod_lookup_alias_from_builtin_file(ctx, alias, list); > + CHECK_ERR_AND_FINISH(err, fail, list, finish); > + > DBG(ctx, "lookup modules.builtin.modinfo %s\n", alias); > err = kmod_lookup_alias_from_kernel_builtin_file(ctx, alias, list); > - if (err == -ENOSYS) { > - /* Optional index missing, try the old one */ > - DBG(ctx, "lookup modules.builtin %s\n", alias); > - err = kmod_lookup_alias_from_builtin_file(ctx, alias, list); > - } > + /* Optional index missing, ignore */ > + if (err == -ENOSYS) > + err = 0; > CHECK_ERR_AND_FINISH(err, fail, list, finish); Yep. Looks good for me.
On Sun, Feb 13, 2022 at 02:13:39PM +0100, Alexey Gladkov wrote: >On Sat, Feb 12, 2022 at 09:43:35PM -1000, Lucas De Marchi wrote: >> The modules.builtin.alias.bin is way larger than the >> modules.builtin.bin. On a normal "distro kernel": >> >> 21k modules.builtin.alias.bin >> 11k modules.builtin.bin >> >> >From the kernel we get both modules.builtin and modules.builtin.modinfo. >> depmod generates modules.builtin.bin and modules.builtin.alias.bin >> from them respectively. modules.bultin is not going away: it's not >> deprecated by the new index added. So, let's just stop duplicating the >> information inside modules.builtin.alias.bin and just use the other >> index. > >The modules.builtin contains only module names. The modules.builtin.modinfo >contains full info about builtin modules including names. > >I thought that if there is complete information about the modules, then >reading the index with only the names does not make sense. And only in the >absence of modules.builtin.modinfo, you need to fallback to the index >with the names. yeah, but most of the time we really need only the module name, so we can optimize for that. And since we are not getting rid of the other index, we can simply use it first the same way that for modules we first do lookup on lookup modules.dep and only later on modules.aliases. > >> --- >> libkmod/libkmod-module.c | 12 +++++++----- >> 1 file changed, 7 insertions(+), 5 deletions(-) >> >> diff --git a/libkmod/libkmod-module.c b/libkmod/libkmod-module.c >> index 6f7747c..6423339 100644 >> --- a/libkmod/libkmod-module.c >> +++ b/libkmod/libkmod-module.c >> @@ -576,13 +576,15 @@ KMOD_EXPORT int kmod_module_new_from_lookup(struct kmod_ctx *ctx, >> err = kmod_lookup_alias_from_aliases_file(ctx, alias, list); >> CHECK_ERR_AND_FINISH(err, fail, list, finish); >> >> + DBG(ctx, "lookup modules.builtin %s\n", alias); >> + err = kmod_lookup_alias_from_builtin_file(ctx, alias, list); >> + CHECK_ERR_AND_FINISH(err, fail, list, finish); >> + >> DBG(ctx, "lookup modules.builtin.modinfo %s\n", alias); >> err = kmod_lookup_alias_from_kernel_builtin_file(ctx, alias, list); >> - if (err == -ENOSYS) { >> - /* Optional index missing, try the old one */ >> - DBG(ctx, "lookup modules.builtin %s\n", alias); >> - err = kmod_lookup_alias_from_builtin_file(ctx, alias, list); >> - } >> + /* Optional index missing, ignore */ >> + if (err == -ENOSYS) >> + err = 0; >> CHECK_ERR_AND_FINISH(err, fail, list, finish); > >Yep. Looks good for me. thanks Lucas De Marchi > >-- >Rgrds, legion >
On Mon, Feb 14, 2022 at 11:43:10PM -0800, Lucas De Marchi wrote: > On Sun, Feb 13, 2022 at 02:13:39PM +0100, Alexey Gladkov wrote: > > On Sat, Feb 12, 2022 at 09:43:35PM -1000, Lucas De Marchi wrote: > > > The modules.builtin.alias.bin is way larger than the > > > modules.builtin.bin. On a normal "distro kernel": > > > > > > 21k modules.builtin.alias.bin > > > 11k modules.builtin.bin > > > > > > >From the kernel we get both modules.builtin and modules.builtin.modinfo. > > > depmod generates modules.builtin.bin and modules.builtin.alias.bin > > > from them respectively. modules.bultin is not going away: it's not > > > deprecated by the new index added. So, let's just stop duplicating the > > > information inside modules.builtin.alias.bin and just use the other > > > index. > > > > The modules.builtin contains only module names. The modules.builtin.modinfo > > contains full info about builtin modules including names. > > > > I thought that if there is complete information about the modules, then > > reading the index with only the names does not make sense. And only in the > > absence of modules.builtin.modinfo, you need to fallback to the index > > with the names. > > yeah, but most of the time we really need only the module name, so we > can optimize for that. And since we are not getting rid of the other > index, we can simply use it first the same way that for modules we first > do lookup on lookup modules.dep and only later on modules.aliases. Yes and no. We can be sure that we don't need aliases. The argument passed to utilities can be very similar to the name of a module: $ modinfo fs-ext4 name: ext4 filename: (builtin) softdep: pre: crc32c license: GPL file: fs/ext4/ext4 description: Fourth Extended Filesystem author: Remy Card, Stephen Tweedie, Andrew Morton, Andreas Dilger, Theodore Ts'o and others alias: fs-ext4 alias: ext3 alias: fs-ext3 By the way, crc32c is also an alias: $ modinfo crc32c filename: /lib/modules/5.14.0.61-centos-alt1.el9/kernel/arch/x86/crypto/crc32c-intel.ko alias: crypto-crc32c-intel alias: crc32c-intel alias: crypto-crc32c alias: crc32c license: GPL description: CRC32c (Castagnoli) optimization using Intel Hardware. author: Austin Zhang <austin.zhang@intel.com>, Kent Liu <kent.liu@intel.com> rhelversion: 9.0 srcversion: 1F2B6A533C8243A4017180A alias: cpu:type:x86,ven*fam*mod*:feature:*0094* depends: retpoline: Y intree: Y name: crc32c_intel vermagic: 5.14.0.61-centos-alt1.el9 SMP preempt mod_unload modversions This is because there are multiple implementations of crc32c. So, information about alias of builtin modules is almost always needed. > > > > > --- > > > libkmod/libkmod-module.c | 12 +++++++----- > > > 1 file changed, 7 insertions(+), 5 deletions(-) > > > > > > diff --git a/libkmod/libkmod-module.c b/libkmod/libkmod-module.c > > > index 6f7747c..6423339 100644 > > > --- a/libkmod/libkmod-module.c > > > +++ b/libkmod/libkmod-module.c > > > @@ -576,13 +576,15 @@ KMOD_EXPORT int kmod_module_new_from_lookup(struct kmod_ctx *ctx, > > > err = kmod_lookup_alias_from_aliases_file(ctx, alias, list); > > > CHECK_ERR_AND_FINISH(err, fail, list, finish); > > > > > > + DBG(ctx, "lookup modules.builtin %s\n", alias); > > > + err = kmod_lookup_alias_from_builtin_file(ctx, alias, list); > > > + CHECK_ERR_AND_FINISH(err, fail, list, finish); > > > + > > > DBG(ctx, "lookup modules.builtin.modinfo %s\n", alias); > > > err = kmod_lookup_alias_from_kernel_builtin_file(ctx, alias, list); > > > - if (err == -ENOSYS) { > > > - /* Optional index missing, try the old one */ > > > - DBG(ctx, "lookup modules.builtin %s\n", alias); > > > - err = kmod_lookup_alias_from_builtin_file(ctx, alias, list); > > > - } > > > + /* Optional index missing, ignore */ > > > + if (err == -ENOSYS) > > > + err = 0; > > > CHECK_ERR_AND_FINISH(err, fail, list, finish); > > > > Yep. Looks good for me. > > thanks > Lucas De Marchi > > > > > -- > > Rgrds, legion > > >
On Sat, Feb 19, 2022 at 01:40:42PM +0100, Alexey Gladkov wrote: >On Mon, Feb 14, 2022 at 11:43:10PM -0800, Lucas De Marchi wrote: >> On Sun, Feb 13, 2022 at 02:13:39PM +0100, Alexey Gladkov wrote: >> > On Sat, Feb 12, 2022 at 09:43:35PM -1000, Lucas De Marchi wrote: >> > > The modules.builtin.alias.bin is way larger than the >> > > modules.builtin.bin. On a normal "distro kernel": >> > > >> > > 21k modules.builtin.alias.bin >> > > 11k modules.builtin.bin >> > > >> > > >From the kernel we get both modules.builtin and modules.builtin.modinfo. >> > > depmod generates modules.builtin.bin and modules.builtin.alias.bin >> > > from them respectively. modules.bultin is not going away: it's not >> > > deprecated by the new index added. So, let's just stop duplicating the >> > > information inside modules.builtin.alias.bin and just use the other >> > > index. >> > >> > The modules.builtin contains only module names. The modules.builtin.modinfo >> > contains full info about builtin modules including names. >> > >> > I thought that if there is complete information about the modules, then >> > reading the index with only the names does not make sense. And only in the >> > absence of modules.builtin.modinfo, you need to fallback to the index >> > with the names. >> >> yeah, but most of the time we really need only the module name, so we >> can optimize for that. And since we are not getting rid of the other >> index, we can simply use it first the same way that for modules we first >> do lookup on lookup modules.dep and only later on modules.aliases. > >Yes and no. We can be sure that we don't need aliases. The argument passed >to utilities can be very similar to the name of a module: > >$ modinfo fs-ext4 >name: ext4 >filename: (builtin) >softdep: pre: crc32c >license: GPL >file: fs/ext4/ext4 >description: Fourth Extended Filesystem >author: Remy Card, Stephen Tweedie, Andrew Morton, Andreas Dilger, Theodore Ts'o and others >alias: fs-ext4 >alias: ext3 >alias: fs-ext3 > >By the way, crc32c is also an alias: > >$ modinfo crc32c >filename: /lib/modules/5.14.0.61-centos-alt1.el9/kernel/arch/x86/crypto/crc32c-intel.ko >alias: crypto-crc32c-intel >alias: crc32c-intel >alias: crypto-crc32c >alias: crc32c >license: GPL >description: CRC32c (Castagnoli) optimization using Intel Hardware. >author: Austin Zhang <austin.zhang@intel.com>, Kent Liu <kent.liu@intel.com> >rhelversion: 9.0 >srcversion: 1F2B6A533C8243A4017180A >alias: cpu:type:x86,ven*fam*mod*:feature:*0094* >depends: >retpoline: Y >intree: Y >name: crc32c_intel >vermagic: 5.14.0.61-centos-alt1.el9 SMP preempt mod_unload modversions > >This is because there are multiple implementations of crc32c. So, information >about alias of builtin modules is almost always needed. I was thinking about "optimization" related to libkmod/modprobe (which is what matters during boot and for the systemd integration). Not for modinfo which is only called by developers and auxiliary tools. But then I measured it doing 50k random lookups. The speed up exists but is under 1% and within the error margin, particularly because we need to maintain the order doing the alias lookup first. So the main benefit is really "we are not getting rid of the other index, so could as well just not duplicate the info". Another reason for not getting rid of the module is to be able to force the lookup to be by module name, ignoring the alias. That was the reason I sent this other series: https://lore.kernel.org/linux-modules/Yg4DOfCUvIpDDBRd@bombadil.infradead.org/T/#t so one can get the modinfo from the crc32 module, and not from its aliases: $ modinfo --modname crc32 module explicitly: name: crc32 filename: (builtin) license: GPL file: lib/crc32 description: Various CRC32 calculations author: Matt Domsch <Matt_Domsch@dell.com> Lucas De Marchi
diff --git a/libkmod/libkmod-module.c b/libkmod/libkmod-module.c index 6f7747c..6423339 100644 --- a/libkmod/libkmod-module.c +++ b/libkmod/libkmod-module.c @@ -576,13 +576,15 @@ KMOD_EXPORT int kmod_module_new_from_lookup(struct kmod_ctx *ctx, err = kmod_lookup_alias_from_aliases_file(ctx, alias, list); CHECK_ERR_AND_FINISH(err, fail, list, finish); + DBG(ctx, "lookup modules.builtin %s\n", alias); + err = kmod_lookup_alias_from_builtin_file(ctx, alias, list); + CHECK_ERR_AND_FINISH(err, fail, list, finish); + DBG(ctx, "lookup modules.builtin.modinfo %s\n", alias); err = kmod_lookup_alias_from_kernel_builtin_file(ctx, alias, list); - if (err == -ENOSYS) { - /* Optional index missing, try the old one */ - DBG(ctx, "lookup modules.builtin %s\n", alias); - err = kmod_lookup_alias_from_builtin_file(ctx, alias, list); - } + /* Optional index missing, ignore */ + if (err == -ENOSYS) + err = 0; CHECK_ERR_AND_FINISH(err, fail, list, finish);