Message ID | 1414961549-31666-1-git-send-email-jason@lakedaemon.net (mailing list archive) |
---|---|
State | New, archived |
Headers | show |
Dne 2.11.2014 v 21:52 Jason Cooper napsal(a): > After this patch: > > f4d4ffc03efc kbuild: dtbs_install: new make target > > was added the kernel tree, Linus Walleij noticed that 'make dtbs' forced > a following 'make zImage' to rebuild the entire tree, even though > nothing had changed. His report: > > After this patch a while back I have observed the following behaviour > of the kernel build: > > make zImage > make zImage > -> incremental build, just relink > > make zImage > make dtbs > make zImage > -> The whole kernel gets rebuilt > > So now if I happen to recompile my device trees, I suddenly want > the entire zImage to be rebuilt to? It's by definition not changes > that affect the kernel build :-( > > I noticed this because my build scripts calls make dtbs && make > zImage, and started to rebuild absolutely everything all the time. > > To fix this, we make only the dtbs_install target depend on the prepare > target. It's needed to make sure KERNELVERSION is calculated prior to > installing. If a mere 'make prepare' causes a rebuild of the whole kernel, then there is something fishy in the ARM Makefiles. However, if you only need the KERNELRELEASE variable, then you do not need to depend on prepare. The main Makefile makes sure that silentoldconfig is ran and therefore KERNELRELEASE set for all targets except make *config. Michal
On Tue, Nov 25, 2014 at 10:41:35PM +0100, Michal Marek wrote: > Dne 2.11.2014 v 21:52 Jason Cooper napsal(a): > > After this patch: > > > > f4d4ffc03efc kbuild: dtbs_install: new make target > > > > was added the kernel tree, Linus Walleij noticed that 'make dtbs' forced > > a following 'make zImage' to rebuild the entire tree, even though > > nothing had changed. His report: > > > > After this patch a while back I have observed the following behaviour > > of the kernel build: > > > > make zImage > > make zImage > > -> incremental build, just relink > > > > make zImage > > make dtbs > > make zImage > > -> The whole kernel gets rebuilt > > > > So now if I happen to recompile my device trees, I suddenly want > > the entire zImage to be rebuilt to? It's by definition not changes > > that affect the kernel build :-( > > > > I noticed this because my build scripts calls make dtbs && make > > zImage, and started to rebuild absolutely everything all the time. > > > > To fix this, we make only the dtbs_install target depend on the prepare > > target. It's needed to make sure KERNELVERSION is calculated prior to > > installing. > > If a mere 'make prepare' causes a rebuild of the whole kernel, then > there is something fishy in the ARM Makefiles. However, if you only need > the KERNELRELEASE variable, then you do not need to depend on prepare. > The main Makefile makes sure that silentoldconfig is ran and therefore > KERNELRELEASE set for all targets except make *config. Running make prepare and then re-running a build doesn't rebuild anything for me. However, I always build with O= Also works if I do a make dtbs too. So, everything seems to work as expected here. I think this needs a bit more debugging to see why the whole kernel is being rebuilt - the kernel build system has methods to tell you why stuff is being built, which would be a good place to start.
On Tue, Nov 25, 2014 at 11:51:50PM +0000, Russell King - ARM Linux wrote: > On Tue, Nov 25, 2014 at 10:41:35PM +0100, Michal Marek wrote: > > Dne 2.11.2014 v 21:52 Jason Cooper napsal(a): > > > After this patch: > > > > > > f4d4ffc03efc kbuild: dtbs_install: new make target > > > > > > was added the kernel tree, Linus Walleij noticed that 'make dtbs' forced > > > a following 'make zImage' to rebuild the entire tree, even though > > > nothing had changed. His report: > > > > > > After this patch a while back I have observed the following behaviour > > > of the kernel build: > > > > > > make zImage > > > make zImage > > > -> incremental build, just relink > > > > > > make zImage > > > make dtbs > > > make zImage > > > -> The whole kernel gets rebuilt > > > > > > So now if I happen to recompile my device trees, I suddenly want > > > the entire zImage to be rebuilt to? It's by definition not changes > > > that affect the kernel build :-( > > > > > > I noticed this because my build scripts calls make dtbs && make > > > zImage, and started to rebuild absolutely everything all the time. > > > > > > To fix this, we make only the dtbs_install target depend on the prepare > > > target. It's needed to make sure KERNELVERSION is calculated prior to > > > installing. > > > > If a mere 'make prepare' causes a rebuild of the whole kernel, then > > there is something fishy in the ARM Makefiles. However, if you only need > > the KERNELRELEASE variable, then you do not need to depend on prepare. > > The main Makefile makes sure that silentoldconfig is ran and therefore > > KERNELRELEASE set for all targets except make *config. > > Running make prepare and then re-running a build doesn't rebuild anything > for me. However, I always build with O= Also works if I do a make > dtbs too. So, everything seems to work as expected here. > > I think this needs a bit more debugging to see why the whole kernel is > being rebuilt - the kernel build system has methods to tell you why stuff > is being built, which would be a good place to start. Linus, since you were the one experiencing the regression, care to take a whack at that? thx, Jason.
On 2014-11-26 00:51, Russell King - ARM Linux wrote: > On Tue, Nov 25, 2014 at 10:41:35PM +0100, Michal Marek wrote: >> Dne 2.11.2014 v 21:52 Jason Cooper napsal(a): >>> After this patch: >>> >>> f4d4ffc03efc kbuild: dtbs_install: new make target >>> >>> was added the kernel tree, Linus Walleij noticed that 'make dtbs' forced >>> a following 'make zImage' to rebuild the entire tree, even though >>> nothing had changed. His report: >>> >>> After this patch a while back I have observed the following behaviour >>> of the kernel build: >>> >>> make zImage >>> make zImage >>> -> incremental build, just relink >>> >>> make zImage >>> make dtbs >>> make zImage >>> -> The whole kernel gets rebuilt >>> >>> So now if I happen to recompile my device trees, I suddenly want >>> the entire zImage to be rebuilt to? It's by definition not changes >>> that affect the kernel build :-( >>> >>> I noticed this because my build scripts calls make dtbs && make >>> zImage, and started to rebuild absolutely everything all the time. >>> >>> To fix this, we make only the dtbs_install target depend on the prepare >>> target. It's needed to make sure KERNELVERSION is calculated prior to >>> installing. >> >> If a mere 'make prepare' causes a rebuild of the whole kernel, then >> there is something fishy in the ARM Makefiles. However, if you only need >> the KERNELRELEASE variable, then you do not need to depend on prepare. >> The main Makefile makes sure that silentoldconfig is ran and therefore >> KERNELRELEASE set for all targets except make *config. > > Running make prepare and then re-running a build doesn't rebuild anything > for me. However, I always build with O= Also works if I do a make > dtbs too. So, everything seems to work as expected here. Good :). > I think this needs a bit more debugging to see why the whole kernel is > being rebuilt - the kernel build system has methods to tell you why stuff > is being built, which would be a good place to start. Right, this is make V=2. Michal
On Wed, Nov 26, 2014 at 1:57 AM, Jason Cooper <jason@lakedaemon.net> wrote: > On Tue, Nov 25, 2014 at 11:51:50PM +0000, Russell King - ARM Linux wrote: >> I think this needs a bit more debugging to see why the whole kernel is >> being rebuilt - the kernel build system has methods to tell you why stuff >> is being built, which would be a good place to start. > > Linus, since you were the one experiencing the regression, care to take > a whack at that? OK this full example should be possible to repeat for anyone with a cross compiler, and uses a separate build dir, and yeah there was something fishy: the bug only appears if you specify CONFIG_DEBUG_SECTION_MISMATCH=y: make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 u300_defconfig make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage -> no rebuild, just relink make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 u300_defconfig make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y -> no rebuild, just relink make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 u300_defconfig make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 dtbs make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage -> no rebuild, just relink make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 u300_defconfig make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 dtbs make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y -> Rebuilds everything. This doesn't occur before the offending commit. So it only happens when specifying extra environment variables on the command line. So I can get rid of the rebuilding by making my scripts less strict by removing CONFIG_DEBUG_SECTION_MISMATCH=y. But is this really correct? Two last examples with more detail: make[1]: Entering directory `/home/linus/linux/build-u300' CHK include/config/kernel.release GEN ./Makefile CHK include/generated/uapi/linux/version.h CHK include/generated/utsrelease.h Using .. as source for kernel make[2]: `include/generated/mach-types.h' is up to date. CALL ../scripts/checksyscalls.sh - due to target missing CHK include/generated/compile.h Kernel: arch/arm/boot/Image is ready Kernel: arch/arm/boot/zImage is ready make[1]: Leaving directory `/home/linus/linux/build-u300' make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y V=2 make[1]: Entering directory `/home/linus/linux/build-u300' CHK include/config/kernel.release GEN ./Makefile CHK include/generated/uapi/linux/version.h CHK include/generated/utsrelease.h Using .. as source for kernel CC scripts/mod/empty.o - due to command line change CC scripts/mod/devicetable-offsets.s - due to command line change MKELF scripts/mod/elfconfig.h - due to: scripts/mod/empty.o HOSTCC scripts/mod/modpost.o - due to: scripts/mod/elfconfig.h HOSTCC scripts/mod/sumversion.o - due to: scripts/mod/elfconfig.h GEN scripts/mod/devicetable-offsets.h - due to: scripts/mod/devicetable-offsets.s HOSTCC scripts/mod/file2alias.o - due to: scripts/mod/elfconfig.h scripts/mod/devicetable-offsets.h HOSTLD scripts/mod/modpost - due to: scripts/mod/modpost.o scripts/mod/file2alias.o scripts/mod/sumversion.o make[2]: `include/generated/mach-types.h' is up to date. CC kernel/bounds.s - due to command line change GEN include/generated/bounds.h - due to: kernel/bounds.s CC arch/arm/kernel/asm-offsets.s - due to: include/generated/bounds.h GEN include/generated/asm-offsets.h - due to: arch/arm/kernel/asm-offsets.s CALL ../scripts/checksyscalls.sh - due to target missing CC init/main.o - due to: include/generated/bounds.h CC arch/arm/common/firmware.o - due to: include/generated/bounds.h CC arch/arm/mm/dma-mapping.o - due to: include/generated/bounds.h CC arch/arm/kernel/elf.o - due to: include/generated/bounds.h CC arch/arm/mach-u300/core.o - due to: include/generated/bounds.h AS arch/arm/kernel/entry-common.o - due to: include/generated/asm-offsets.h LD arch/arm/common/built-in.o - due to: arch/arm/common/firmware.o CHK include/generated/compile.h CC arch/arm/kernel/irq.o - due to: include/generated/bounds.h CC arch/arm/mach-u300/dummyspichip.o - due to: include/generated/bounds.h CC init/do_mounts.o - due to: include/generated/bounds.h CC arch/arm/mm/extable.o - due to: include/generated/bounds.h CC arch/arm/kernel/opcodes.o - due to: include/generated/bounds.h (...) If I revert commit f4d4ffc03efc: make[1]: Entering directory `/home/linus/linux/build-u300' CHK include/config/kernel.release GEN ./Makefile CHK include/generated/uapi/linux/version.h Using .. as source for kernel CHK include/generated/utsrelease.h make[2]: `include/generated/mach-types.h' is up to date. CC scripts/mod/empty.o - due to command line change CC scripts/mod/devicetable-offsets.s - due to command line change CALL ../scripts/checksyscalls.sh - due to target missing MKELF scripts/mod/elfconfig.h - due to: scripts/mod/empty.o HOSTCC scripts/mod/modpost.o - due to: scripts/mod/elfconfig.h HOSTCC scripts/mod/sumversion.o - due to: scripts/mod/elfconfig.h GEN scripts/mod/devicetable-offsets.h - due to: scripts/mod/devicetable-offsets.s HOSTCC scripts/mod/file2alias.o - due to: scripts/mod/elfconfig.h scripts/mod/devicetable-offsets.h HOSTLD scripts/mod/modpost - due to: scripts/mod/modpost.o scripts/mod/file2alias.o scripts/mod/sumversion.o CHK include/generated/compile.h Kernel: arch/arm/boot/Image is ready Kernel: arch/arm/boot/zImage is ready make[1]: Leaving directory `/home/linus/linux/build-u300' So in both cases a changed command line is detected and empty.o and devicetable-offsets.s is rebuilt. But after the commit the rebuild goes on to rebuild kernel/bounds.s which generates bounds.h which in turn rebuilds the whole world. I don't know why the behaviour changed like this :( Yours, Linus Walleij
On Wed, Nov 26, 2014 at 02:57:42PM +0100, Linus Walleij wrote: > make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- > KBUILD_OUTPUT=build-u300 u300_defconfig > make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- > KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y > make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- > KBUILD_OUTPUT=build-u300 dtbs > make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- > KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y > > -> Rebuilds everything. > > This doesn't occur before the offending commit. So it only happens > when specifying extra environment variables on the command line. I suspect if you also provide the CONFIG_DEBUG_SECTION_MISMATCH=y on the dtbs line, everything will work properly. The problem is that dtbs line executes a prepare, which I guess rebuilds the bounds stuff. So, the dtbs target results in it being rebuilt without the section mismatch, and then you re-execute a make with it, causing the bounds stuff to be rebuilt again.
On Wed, Nov 26, 2014 at 3:41 PM, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote: > On Wed, Nov 26, 2014 at 02:57:42PM +0100, Linus Walleij wrote: >> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- >> KBUILD_OUTPUT=build-u300 u300_defconfig >> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- >> KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y >> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- >> KBUILD_OUTPUT=build-u300 dtbs >> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- >> KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y >> >> -> Rebuilds everything. >> >> This doesn't occur before the offending commit. So it only happens >> when specifying extra environment variables on the command line. > > I suspect if you also provide the CONFIG_DEBUG_SECTION_MISMATCH=y on the > dtbs line, everything will work properly. Yay, it works! :) > The problem is that dtbs line executes a prepare, which I guess rebuilds > the bounds stuff. So, the dtbs target results in it being rebuilt without > the section mismatch, and then you re-execute a make with it, causing > the bounds stuff to be rebuilt again. Yep. Not very intuitive to require passing section mismatch debug flags to DTB rebuilding but whatever, it's not so bad I can't live with it. Strange that it worked before. Yours, Linus Walleij
On 2014-11-28 15:10, Linus Walleij wrote: > On Wed, Nov 26, 2014 at 3:41 PM, Russell King - ARM Linux > <linux@arm.linux.org.uk> wrote: >> On Wed, Nov 26, 2014 at 02:57:42PM +0100, Linus Walleij wrote: >>> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- >>> KBUILD_OUTPUT=build-u300 u300_defconfig >>> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- >>> KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y >>> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- >>> KBUILD_OUTPUT=build-u300 dtbs >>> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- >>> KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y >>> >>> -> Rebuilds everything. >>> >>> This doesn't occur before the offending commit. So it only happens >>> when specifying extra environment variables on the command line. >> >> I suspect if you also provide the CONFIG_DEBUG_SECTION_MISMATCH=y on the >> dtbs line, everything will work properly. > > Yay, it works! :) > >> The problem is that dtbs line executes a prepare, which I guess rebuilds >> the bounds stuff. So, the dtbs target results in it being rebuilt without >> the section mismatch, and then you re-execute a make with it, causing >> the bounds stuff to be rebuilt again. > > Yep. Not very intuitive to require passing section mismatch debug > flags to DTB rebuilding but whatever, it's not so bad I can't live with > it. Still, the dependency on the 'prepare' target is superfluous, if the only requirement is that $(KERNELRELEASE) be set properly. Michal
On Mon, 01 Dec 2014 14:55:48 +0100 , Michal Marek <mmarek@suse.cz> wrote: > On 2014-11-28 15:10, Linus Walleij wrote: > > On Wed, Nov 26, 2014 at 3:41 PM, Russell King - ARM Linux > > <linux@arm.linux.org.uk> wrote: > >> On Wed, Nov 26, 2014 at 02:57:42PM +0100, Linus Walleij wrote: > >>> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- > >>> KBUILD_OUTPUT=build-u300 u300_defconfig > >>> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- > >>> KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y > >>> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- > >>> KBUILD_OUTPUT=build-u300 dtbs > >>> make -f Makefile -j5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- > >>> KBUILD_OUTPUT=build-u300 zImage CONFIG_DEBUG_SECTION_MISMATCH=y > >>> > >>> -> Rebuilds everything. > >>> > >>> This doesn't occur before the offending commit. So it only happens > >>> when specifying extra environment variables on the command line. > >> > >> I suspect if you also provide the CONFIG_DEBUG_SECTION_MISMATCH=y on the > >> dtbs line, everything will work properly. > > > > Yay, it works! :) > > > >> The problem is that dtbs line executes a prepare, which I guess rebuilds > >> the bounds stuff. So, the dtbs target results in it being rebuilt without > >> the section mismatch, and then you re-execute a make with it, causing > >> the bounds stuff to be rebuilt again. > > > > Yep. Not very intuitive to require passing section mismatch debug > > flags to DTB rebuilding but whatever, it's not so bad I can't live with > > it. > > Still, the dependency on the 'prepare' target is superfluous, if the > only requirement is that $(KERNELRELEASE) be set properly. So, I've lost track of where things stand on this issue. What is the current proposed fix? Make dtbs depend on 'scripts' and dtbs_install depend on 'prepare scripts'? Or is there a better way to make sure KERNELRELEASE is set for the dtbs_install target? g.
On Thu, Dec 04, 2014 at 11:22:01AM +0000, Grant Likely wrote: > On Mon, 01 Dec 2014 14:55:48 +0100, Michal Marek <mmarek@suse.cz> wrote: > > Still, the dependency on the 'prepare' target is superfluous, if the > > only requirement is that $(KERNELRELEASE) be set properly. > > So, I've lost track of where things stand on this issue. What is the > current proposed fix? Make dtbs depend on 'scripts' and dtbs_install > depend on 'prepare scripts'? Or is there a better way to make sure > KERNELRELEASE is set for the dtbs_install target? Marek's first reply in the thread contains the solution: if you only need the KERNELRELEASE variable, then you do not need to depend on prepare. The main Makefile makes sure that silentoldconfig is ran and therefore KERNELRELEASE set for all targets except make *config. which seems to suggest that "prepare" is not needed for either target.
diff --git a/arch/arm/Makefile b/arch/arm/Makefile index 034a94904d69..ae1c278128f8 100644 --- a/arch/arm/Makefile +++ b/arch/arm/Makefile @@ -312,7 +312,9 @@ $(INSTALL_TARGETS): $(Q)$(MAKE) $(build)=$(boot)/dts MACHINE=$(MACHINE) $(boot)/dts/$@ PHONY += dtbs dtbs_install -dtbs dtbs_install: prepare scripts +dtbs: scripts + $(Q)$(MAKE) $(build)=$(boot)/dts MACHINE=$(MACHINE) $@ +dtbs_install: prepare scripts $(Q)$(MAKE) $(build)=$(boot)/dts MACHINE=$(MACHINE) $@ # We use MRPROPER_FILES and CLEAN_FILES now