| Message ID | 20230313171036.268407-2-bage@linutronix.de (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | kbuild: deb-pkg: Allow parallel build | expand |
On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote: > > Currently, the only way to build the deb-pkg generated package parallely > is adding -jN to the MAKEFLAGS environment variable. The package ignores > the usual parallel build option that is described in Debian Policy §4.9.1. "dpkg-buildpackage -b -j16" worked for me. > Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable > that ends up being set by Debian's build tools. > > Link: https://www.debian.org/doc/debian-policy/ch-source.html > Signed-off-by: Bastian Germann <bage@linutronix.de> > --- > scripts/package/deb-build-option | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/scripts/package/deb-build-option b/scripts/package/deb-build-option > index b079b0d121d4..dd170e2b3018 100755 > --- a/scripts/package/deb-build-option > +++ b/scripts/package/deb-build-option > @@ -7,6 +7,12 @@ if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUI > echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}- > fi > > +for build_opt in $DEB_BUILD_OPTIONS; do > + if [ "${build_opt#parallel=}" != "$build_opt" ]; then > + echo -j${build_opt#parallel=} > + fi > +done > + > version=$(dpkg-parsechangelog -S Version) > version_upstream="${version%-*}" > debian_revision="${version#${version_upstream}}" > -- > 2.39.2 >
Am 13.03.23 um 18:59 schrieb Masahiro Yamada: > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote: >> >> Currently, the only way to build the deb-pkg generated package parallely >> is adding -jN to the MAKEFLAGS environment variable. The package ignores >> the usual parallel build option that is described in Debian Policy §4.9.1. > > > > "dpkg-buildpackage -b -j16" worked for me. This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call: /usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all So it is not used to run the actual build, just the top level `make -f debian/rules` invocation. You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use the usual way of parallelizing. Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile or dpkg-buildpackage.
On Tue, Mar 14, 2023 at 3:56 AM Bastian Germann <bage@linutronix.de> wrote: > > Am 13.03.23 um 18:59 schrieb Masahiro Yamada: > > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote: > >> > >> Currently, the only way to build the deb-pkg generated package parallely > >> is adding -jN to the MAKEFLAGS environment variable. The package ignores > >> the usual parallel build option that is described in Debian Policy §4.9.1. > > > > > > > > "dpkg-buildpackage -b -j16" worked for me. > > This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call: > /usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS but also MAKEFLAGS. This is clearly explained in "man dpkb-buildpackage". -j, --jobs[=jobs|auto] ... Will add itself to the MAKEFLAGS environment variable, which should cause all subsequent make invocations to inherit the option, thus forcing the parallel setting on the packaging ... Your statement sounds like 'MAKEFLAGS=-j<N> dpkg-buildpackage' is the only way to build packages in parallel. Apparently, dpkg-buildpackage provides a much shorter way and invokes internal Make in parallel. > > So it is not used to run the actual build, just the top level `make -f debian/rules` invocation. > You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use > the usual way of parallelizing. What is the "usual" way in this context? Do you mean 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' is the usual way for parallel building? If so, I agree. This patch caters to this case. But, I think your code should go to debian/rules instead of scripts/package/deb-build-option. Kbuild's "make clean" works much faster with the parallel option. Also, the commit description should explain the benefit of this patch. > > Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile > or dpkg-buildpackage. Try: $ make deb-pkg -j16 and $ dpkg-buildpackage -b -j16 Both run the package building in parallel. You see a big difference in build speed w/wo the -j option. (and the build logs are interleaved, since they are emitted by multiple threads) -- Best Regards Masahiro Yamada
Am 15.03.23 um 15:18 schrieb Masahiro Yamada: > "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS > but also MAKEFLAGS. > > > This is clearly explained in "man dpkb-buildpackage". > > > -j, --jobs[=jobs|auto] > ... > Will add itself to the MAKEFLAGS environment variable, which should > cause all subsequent make invocations to inherit the option, thus > forcing the parallel setting on the packaging ... I see. The testing/unstable dpkg-buildpackage version changed the -j behaviour. -j only sets DEB_BUILD_OPTIONS=parallel= now. The man page now says: --jobs-force[=jobs|auto] This option (since dpkg 1.21.10) is equivalent to the --jobs option except that it will enable forced parallel mode, by adding the make -j option with the computed number of parallel jobs to the MAKEFLAGS environment variable. > Your statement sounds like > > 'MAKEFLAGS=-j<N> dpkg-buildpackage' > > is the only way to build packages in parallel. It is in v1.21.10 or later (or using --jobs-force which does the same thing). > Apparently, dpkg-buildpackage provides a much shorter way > and invokes internal Make in parallel. > > > > >> >> So it is not used to run the actual build, just the top level `make -f debian/rules` invocation. >> You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use >> the usual way of parallelizing. > > > What is the "usual" way in this context? The usual way is dpkg-buildpackage -jN but that does not work with later versions. > Do you mean > 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' > is the usual way for parallel building? > > If so, I agree. This patch caters to this case. > > > But, I think your code should go to debian/rules > instead of scripts/package/deb-build-option. > Kbuild's "make clean" works much faster with the parallel option. > > > Also, the commit description should explain the > benefit of this patch. I will clarify the dpkg-buildpackage behaviour change and will move the stuff to debian/rules in v2.
On Wed, Mar 15, 2023 at 3:24 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Tue, Mar 14, 2023 at 3:56 AM Bastian Germann <bage@linutronix.de> wrote: > > > > Am 13.03.23 um 18:59 schrieb Masahiro Yamada: > > > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote: > > >> > > >> Currently, the only way to build the deb-pkg generated package parallely > > >> is adding -jN to the MAKEFLAGS environment variable. The package ignores > > >> the usual parallel build option that is described in Debian Policy §4.9.1. > > > > > > > > > > > > "dpkg-buildpackage -b -j16" worked for me. > > > > This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call: > > /usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all > > > "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS > but also MAKEFLAGS. > > > This is clearly explained in "man dpkb-buildpackage". > > > -j, --jobs[=jobs|auto] > ... > Will add itself to the MAKEFLAGS environment variable, which should > cause all subsequent make invocations to inherit the option, thus > forcing the parallel setting on the packaging ... > > > > > Your statement sounds like > > 'MAKEFLAGS=-j<N> dpkg-buildpackage' > > is the only way to build packages in parallel. > > Apparently, dpkg-buildpackage provides a much shorter way > and invokes internal Make in parallel. > > > > > > > > So it is not used to run the actual build, just the top level `make -f debian/rules` invocation. > > You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use > > the usual way of parallelizing. > > > What is the "usual" way in this context? > > > Do you mean > 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' > is the usual way for parallel building? > > If so, I agree. This patch caters to this case. > > > But, I think your code should go to debian/rules > instead of scripts/package/deb-build-option. > Kbuild's "make clean" works much faster with the parallel option. > > > Also, the commit description should explain the > benefit of this patch. > > > > > > Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile > > or dpkg-buildpackage. > > > Try: > > $ make deb-pkg -j16 > > and > > $ dpkg-buildpackage -b -j16 > > > Both run the package building in parallel. > > You see a big difference in build speed w/wo the -j option. > (and the build logs are interleaved, since they > are emitted by multiple threads) > > I have no strong opinion in having a DEB_BUILD_OPTIONS... As you stated Masahiro I have seen it mostly in debian/rules. See the example from [1]: ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS))) NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS))) MAKEFLAGS += -j$(NUMJOBS) endif Normally, I use in my kernel build-script: $ make deb-pkg -j${NUMJOBS} My €0,02. -Sedat- [1] https://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules-options
On Thu, Mar 16, 2023 at 12:51 AM Bastian Germann <bage@linutronix.de> wrote: > > Am 15.03.23 um 15:18 schrieb Masahiro Yamada: > > "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS > > but also MAKEFLAGS. > > > > > > This is clearly explained in "man dpkb-buildpackage". > > > > > > -j, --jobs[=jobs|auto] > > ... > > Will add itself to the MAKEFLAGS environment variable, which should > > cause all subsequent make invocations to inherit the option, thus > > forcing the parallel setting on the packaging ... > > I see. The testing/unstable dpkg-buildpackage version changed the -j behaviour. > -j only sets DEB_BUILD_OPTIONS=parallel= now. > > The man page now says: > > --jobs-force[=jobs|auto] > This option (since dpkg 1.21.10) is equivalent to the --jobs option except that > it will enable forced parallel mode, by adding the make -j option with the computed > number of parallel jobs to the MAKEFLAGS environment variable. > > > Your statement sounds like > > > > 'MAKEFLAGS=-j<N> dpkg-buildpackage' > > > > is the only way to build packages in parallel. > > It is in v1.21.10 or later (or using --jobs-force which does the same thing). > > > Apparently, dpkg-buildpackage provides a much shorter way > > and invokes internal Make in parallel. > > > > > > > > > >> > >> So it is not used to run the actual build, just the top level `make -f debian/rules` invocation. > >> You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use > >> the usual way of parallelizing. > > > > > > What is the "usual" way in this context? > > The usual way is dpkg-buildpackage -jN but that does not work with later versions. > > > Do you mean > > 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' > > is the usual way for parallel building? > > > > If so, I agree. This patch caters to this case. > > > > > > But, I think your code should go to debian/rules > > instead of scripts/package/deb-build-option. > > Kbuild's "make clean" works much faster with the parallel option. > > > > > > Also, the commit description should explain the > > benefit of this patch. > > I will clarify the dpkg-buildpackage behaviour change and will move the stuff to debian/rules > in v2. Please do so. Now I understand the point of this patch. Indeed, my dpkg-buildpackage is old. (v1.21.1 on Ubuntu 22.04) Thanks. -- Best Regards Masahiro Yamada
On Thu, Mar 16, 2023 at 12:57 AM Sedat Dilek <sedat.dilek@gmail.com> wrote: > > On Wed, Mar 15, 2023 at 3:24 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Tue, Mar 14, 2023 at 3:56 AM Bastian Germann <bage@linutronix.de> wrote: > > > > > > Am 13.03.23 um 18:59 schrieb Masahiro Yamada: > > > > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@linutronix.de> wrote: > > > >> > > > >> Currently, the only way to build the deb-pkg generated package parallely > > > >> is adding -jN to the MAKEFLAGS environment variable. The package ignores > > > >> the usual parallel build option that is described in Debian Policy §4.9.1. > > > > > > > > > > > > > > > > "dpkg-buildpackage -b -j16" worked for me. > > > > > > This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call: > > > /usr/bin/make -f ./Makefile ARCH=x86 KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1 olddefconfig all > > > > > > "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS > > but also MAKEFLAGS. > > > > > > This is clearly explained in "man dpkb-buildpackage". > > > > > > -j, --jobs[=jobs|auto] > > ... > > Will add itself to the MAKEFLAGS environment variable, which should > > cause all subsequent make invocations to inherit the option, thus > > forcing the parallel setting on the packaging ... > > > > > > > > > > Your statement sounds like > > > > 'MAKEFLAGS=-j<N> dpkg-buildpackage' > > > > is the only way to build packages in parallel. > > > > Apparently, dpkg-buildpackage provides a much shorter way > > and invokes internal Make in parallel. > > > > > > > > > > > > > > So it is not used to run the actual build, just the top level `make -f debian/rules` invocation. > > > You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use > > > the usual way of parallelizing. > > > > > > What is the "usual" way in this context? > > > > > > Do you mean > > 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b' > > is the usual way for parallel building? > > > > If so, I agree. This patch caters to this case. > > > > > > But, I think your code should go to debian/rules > > instead of scripts/package/deb-build-option. > > Kbuild's "make clean" works much faster with the parallel option. > > > > > > Also, the commit description should explain the > > benefit of this patch. > > > > > > > > > > Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile > > > or dpkg-buildpackage. > > > > > > Try: > > > > $ make deb-pkg -j16 > > > > and > > > > $ dpkg-buildpackage -b -j16 > > > > > > Both run the package building in parallel. > > > > You see a big difference in build speed w/wo the -j option. > > (and the build logs are interleaved, since they > > are emitted by multiple threads) > > > > > > I have no strong opinion in having a DEB_BUILD_OPTIONS... > > As you stated Masahiro I have seen it mostly in debian/rules. > > See the example from [1]: > > ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS))) > NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS))) > MAKEFLAGS += -j$(NUMJOBS) > endif > Yup, this code looks good to me.
diff --git a/scripts/package/deb-build-option b/scripts/package/deb-build-option index b079b0d121d4..dd170e2b3018 100755 --- a/scripts/package/deb-build-option +++ b/scripts/package/deb-build-option @@ -7,6 +7,12 @@ if [ -z "${CROSS_COMPILE}${cross_compiling}" -a "${DEB_HOST_ARCH}" != "${DEB_BUI echo CROSS_COMPILE=${DEB_HOST_GNU_TYPE}- fi +for build_opt in $DEB_BUILD_OPTIONS; do + if [ "${build_opt#parallel=}" != "$build_opt" ]; then + echo -j${build_opt#parallel=} + fi +done + version=$(dpkg-parsechangelog -S Version) version_upstream="${version%-*}" debian_revision="${version#${version_upstream}}"
Currently, the only way to build the deb-pkg generated package parallely is adding -jN to the MAKEFLAGS environment variable. The package ignores the usual parallel build option that is described in Debian Policy §4.9.1. Derive make's -j parameter from the DEB_BUILD_OPTIONS environment variable that ends up being set by Debian's build tools. Link: https://www.debian.org/doc/debian-policy/ch-source.html Signed-off-by: Bastian Germann <bage@linutronix.de> --- scripts/package/deb-build-option | 6 ++++++ 1 file changed, 6 insertions(+)