| Message ID | 20240223092338.2433632-1-wenst@chromium.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [RFC] kbuild: create a list of all built DTB files | expand |
On Fri, Feb 23, 2024 at 6:23 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > It is useful to have a list of all composite *.dtb files, along with > their individual components, generated from the current build. > > With this commit, 'make dtbs' creates arch/*/boot/dts/dtbs-components, > which lists the composite dtb files created in the current build. It > maintains the order of the dtb-y additions in Makefiles although the > order is not important for DTBs. > > This compliments the list of all *.dtb and *.dtbo files in dtbs-list, > which only includes the files directly added to dtb-y. > > For example, consider this case: > > foo-dtbs := foo_base.dtb foo_overlay.dtbo > dtb-y := bar.dtb foo.dtb > > In this example, the new list will include foo.dtb with foo_base.dtb and > foo_overlay.dtbo on the same line, but not bar.dtb. > > Signed-off-by: Chen-Yu Tsai <wenst@chromium.org> > --- > Hi, > > I hacked up this new thing to list out the individual components of each > composite dtb. I think this information would be useful for FIT image > generation or other toolchains to consume. For example, instead of > including each dtb, a toolchain could realize that some are put together > using others, and if the bootloader supports it, put together commands > to reassemble the end result from the original parts. > > This is based on and complements Masahiro-san's recent dtbs-list work. This is another format of my previous per-dtb "*.dtlst" (but I did not pick up 3/4, 4/4 because I did not know what we need after all). This should be discussed together with how Simon's script will look like. I can understand your Makefile code, but I still do not know how the entire overlay stuff will work in a big picture. > > .gitignore | 1 + > scripts/Makefile.build | 16 ++++++++++++++++ > scripts/Makefile.lib | 8 ++++++-- > 3 files changed, 23 insertions(+), 2 deletions(-) > > diff --git a/.gitignore b/.gitignore > index c59dc60ba62e..bb5b3bbca4ef 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -52,6 +52,7 @@ > *.xz > *.zst > Module.symvers > +dtbs-components > dtbs-list > modules.order > > diff --git a/scripts/Makefile.build b/scripts/Makefile.build > index 4971f54c855e..ba85c2385c9e 100644 > --- a/scripts/Makefile.build > +++ b/scripts/Makefile.build > @@ -72,6 +72,7 @@ endif > subdir-builtin := $(sort $(filter %/built-in.a, $(real-obj-y))) > subdir-modorder := $(sort $(filter %/modules.order, $(obj-m))) > subdir-dtbslist := $(sort $(filter %/dtbs-list, $(dtb-y))) > +subdir-dtbscomp := $(sort $(filter %/dtbs-components, $(multi-dtb-y))) > > targets-for-builtin := $(extra-y) > > @@ -390,6 +391,7 @@ $(obj)/%.asn1.c $(obj)/%.asn1.h: $(src)/%.asn1 $(objtree)/scripts/asn1_compiler > $(subdir-builtin): $(obj)/%/built-in.a: $(obj)/% ; > $(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ; > $(subdir-dtbslist): $(obj)/%/dtbs-list: $(obj)/% ; > +$(subdir-dtbscomp): $(obj)/%/dtbs-components: $(obj)/% ; > > # > # Rule to compile a set of .o files into one .a file (without symbol table) > @@ -422,6 +424,20 @@ $(obj)/modules.order: $(obj-m) FORCE > $(obj)/dtbs-list: $(dtb-y) FORCE > $(call if_changed,gen_order) > > +# > +# Rule to create dtbs-components > +# > +# This is a list of composite dtb(s), along with each dtb's components, > +# from the current Makefile and its sub-directories. > + > +cmd_gen_dtb_components = { $(foreach m, $(real-prereqs), \ > + $(if $(filter %/$(notdir $@), $m), cat $m, \ > + echo $m: $(addprefix $(obj)/,$($(notdir $(m:%.dtb=%-dtbs))))); \ > + ) :; } > $@ > + > +$(obj)/dtbs-components: $(multi-dtb-y) FORCE > + $(call if_changed,gen_dtb_components) > + > # > # Rule to compile a set of .o files into one .a file (with symbol table) > # > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib > index dbcac396329e..7c2127a84ac2 100644 > --- a/scripts/Makefile.lib > +++ b/scripts/Makefile.lib > @@ -61,7 +61,6 @@ real-search = $(foreach m, $1, $(if $(call suffix-search, $m, $2, $3 -), $(call > multi-obj-y := $(call multi-search, $(obj-y), .o, -objs -y) > multi-obj-m := $(call multi-search, $(obj-m), .o, -objs -y -m) > multi-obj-ym := $(multi-obj-y) $(multi-obj-m) > - > # Replace multi-part objects by their individual parts, > # including built-in.a from subdirectories > real-obj-y := $(call real-search, $(obj-y), .o, -objs -y) > @@ -91,6 +90,11 @@ real-dtb-y := $(call real-search, $(dtb-y), .dtb, -dtbs) > # Base DTB that overlay is applied onto > base-dtb-y := $(filter %.dtb, $(call real-search, $(multi-dtb-y), .dtb, -dtbs)) > > +ifdef need-dtbslist > +multi-dtb-y += $(addsuffix /dtbs-components, $(subdir-ym)) > +always-y += dtbs-components > +endif > + > always-y += $(dtb-y) > > # Add subdir path > @@ -406,7 +410,7 @@ cmd_dtc = $(HOSTCC) -E $(dtc_cpp_flags) -x assembler-with-cpp -o $(dtc-tmp) $< ; > quiet_cmd_fdtoverlay = DTOVL $@ > cmd_fdtoverlay = $(objtree)/scripts/dtc/fdtoverlay -o $@ -i $(real-prereqs) > > -$(multi-dtb-y): FORCE > +$(filter-out %/dtbs-components, multi-dtb-y): FORCE > $(call if_changed,fdtoverlay) > $(call multi_depend, $(multi-dtb-y), .dtb, -dtbs) > > -- > 2.44.0.rc0.258.g7320e95886-goog > -- Best Regards Masahiro Yamada
On Sun, Feb 25, 2024 at 4:21 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Fri, Feb 23, 2024 at 6:23 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > It is useful to have a list of all composite *.dtb files, along with > > their individual components, generated from the current build. > > > > With this commit, 'make dtbs' creates arch/*/boot/dts/dtbs-components, > > which lists the composite dtb files created in the current build. It > > maintains the order of the dtb-y additions in Makefiles although the > > order is not important for DTBs. > > > > This compliments the list of all *.dtb and *.dtbo files in dtbs-list, > > which only includes the files directly added to dtb-y. > > > > For example, consider this case: > > > > foo-dtbs := foo_base.dtb foo_overlay.dtbo > > dtb-y := bar.dtb foo.dtb > > > > In this example, the new list will include foo.dtb with foo_base.dtb and > > foo_overlay.dtbo on the same line, but not bar.dtb. > > > > Signed-off-by: Chen-Yu Tsai <wenst@chromium.org> > > --- > > Hi, > > > > I hacked up this new thing to list out the individual components of each > > composite dtb. I think this information would be useful for FIT image > > generation or other toolchains to consume. For example, instead of > > including each dtb, a toolchain could realize that some are put together > > using others, and if the bootloader supports it, put together commands > > to reassemble the end result from the original parts. > > > > This is based on and complements Masahiro-san's recent dtbs-list work. > > > > This is another format of my previous per-dtb "*.dtlst" > (but I did not pick up 3/4, 4/4 because I did not know what we need after all). > > This should be discussed together with how Simon's script will look like. > > I can understand your Makefile code, but I still do not know > how the entire overlay stuff will work in a big picture. How would you like to proceed? I can through together some changes on top of Simon's patches as an initial proposal if that helps? I can use your format if you prefer. ChenYu > > > > .gitignore | 1 + > > scripts/Makefile.build | 16 ++++++++++++++++ > > scripts/Makefile.lib | 8 ++++++-- > > 3 files changed, 23 insertions(+), 2 deletions(-) > > > > diff --git a/.gitignore b/.gitignore > > index c59dc60ba62e..bb5b3bbca4ef 100644 > > --- a/.gitignore > > +++ b/.gitignore > > @@ -52,6 +52,7 @@ > > *.xz > > *.zst > > Module.symvers > > +dtbs-components > > dtbs-list > > modules.order > > > > diff --git a/scripts/Makefile.build b/scripts/Makefile.build > > index 4971f54c855e..ba85c2385c9e 100644 > > --- a/scripts/Makefile.build > > +++ b/scripts/Makefile.build > > @@ -72,6 +72,7 @@ endif > > subdir-builtin := $(sort $(filter %/built-in.a, $(real-obj-y))) > > subdir-modorder := $(sort $(filter %/modules.order, $(obj-m))) > > subdir-dtbslist := $(sort $(filter %/dtbs-list, $(dtb-y))) > > +subdir-dtbscomp := $(sort $(filter %/dtbs-components, $(multi-dtb-y))) > > > > targets-for-builtin := $(extra-y) > > > > @@ -390,6 +391,7 @@ $(obj)/%.asn1.c $(obj)/%.asn1.h: $(src)/%.asn1 $(objtree)/scripts/asn1_compiler > > $(subdir-builtin): $(obj)/%/built-in.a: $(obj)/% ; > > $(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ; > > $(subdir-dtbslist): $(obj)/%/dtbs-list: $(obj)/% ; > > +$(subdir-dtbscomp): $(obj)/%/dtbs-components: $(obj)/% ; > > > > # > > # Rule to compile a set of .o files into one .a file (without symbol table) > > @@ -422,6 +424,20 @@ $(obj)/modules.order: $(obj-m) FORCE > > $(obj)/dtbs-list: $(dtb-y) FORCE > > $(call if_changed,gen_order) > > > > +# > > +# Rule to create dtbs-components > > +# > > +# This is a list of composite dtb(s), along with each dtb's components, > > +# from the current Makefile and its sub-directories. > > + > > +cmd_gen_dtb_components = { $(foreach m, $(real-prereqs), \ > > + $(if $(filter %/$(notdir $@), $m), cat $m, \ > > + echo $m: $(addprefix $(obj)/,$($(notdir $(m:%.dtb=%-dtbs))))); \ > > + ) :; } > $@ > > + > > +$(obj)/dtbs-components: $(multi-dtb-y) FORCE > > + $(call if_changed,gen_dtb_components) > > + > > # > > # Rule to compile a set of .o files into one .a file (with symbol table) > > # > > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib > > index dbcac396329e..7c2127a84ac2 100644 > > --- a/scripts/Makefile.lib > > +++ b/scripts/Makefile.lib > > @@ -61,7 +61,6 @@ real-search = $(foreach m, $1, $(if $(call suffix-search, $m, $2, $3 -), $(call > > multi-obj-y := $(call multi-search, $(obj-y), .o, -objs -y) > > multi-obj-m := $(call multi-search, $(obj-m), .o, -objs -y -m) > > multi-obj-ym := $(multi-obj-y) $(multi-obj-m) > > - > > # Replace multi-part objects by their individual parts, > > # including built-in.a from subdirectories > > real-obj-y := $(call real-search, $(obj-y), .o, -objs -y) > > @@ -91,6 +90,11 @@ real-dtb-y := $(call real-search, $(dtb-y), .dtb, -dtbs) > > # Base DTB that overlay is applied onto > > base-dtb-y := $(filter %.dtb, $(call real-search, $(multi-dtb-y), .dtb, -dtbs)) > > > > +ifdef need-dtbslist > > +multi-dtb-y += $(addsuffix /dtbs-components, $(subdir-ym)) > > +always-y += dtbs-components > > +endif > > + > > always-y += $(dtb-y) > > > > # Add subdir path > > @@ -406,7 +410,7 @@ cmd_dtc = $(HOSTCC) -E $(dtc_cpp_flags) -x assembler-with-cpp -o $(dtc-tmp) $< ; > > quiet_cmd_fdtoverlay = DTOVL $@ > > cmd_fdtoverlay = $(objtree)/scripts/dtc/fdtoverlay -o $@ -i $(real-prereqs) > > > > -$(multi-dtb-y): FORCE > > +$(filter-out %/dtbs-components, multi-dtb-y): FORCE > > $(call if_changed,fdtoverlay) > > $(call multi_depend, $(multi-dtb-y), .dtb, -dtbs) > > > > -- > > 2.44.0.rc0.258.g7320e95886-goog > > > > > -- > Best Regards > Masahiro Yamada
On Thu, Feb 29, 2024 at 11:38 AM Chen-Yu Tsai <wenst@chromium.org> wrote: > > On Sun, Feb 25, 2024 at 4:21 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Fri, Feb 23, 2024 at 6:23 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > It is useful to have a list of all composite *.dtb files, along with > > > their individual components, generated from the current build. > > > > > > With this commit, 'make dtbs' creates arch/*/boot/dts/dtbs-components, > > > which lists the composite dtb files created in the current build. It > > > maintains the order of the dtb-y additions in Makefiles although the > > > order is not important for DTBs. > > > > > > This compliments the list of all *.dtb and *.dtbo files in dtbs-list, > > > which only includes the files directly added to dtb-y. > > > > > > For example, consider this case: > > > > > > foo-dtbs := foo_base.dtb foo_overlay.dtbo > > > dtb-y := bar.dtb foo.dtb > > > > > > In this example, the new list will include foo.dtb with foo_base.dtb and > > > foo_overlay.dtbo on the same line, but not bar.dtb. > > > > > > Signed-off-by: Chen-Yu Tsai <wenst@chromium.org> > > > --- > > > Hi, > > > > > > I hacked up this new thing to list out the individual components of each > > > composite dtb. I think this information would be useful for FIT image > > > generation or other toolchains to consume. For example, instead of > > > including each dtb, a toolchain could realize that some are put together > > > using others, and if the bootloader supports it, put together commands > > > to reassemble the end result from the original parts. > > > > > > This is based on and complements Masahiro-san's recent dtbs-list work. > > > > > > > > This is another format of my previous per-dtb "*.dtlst" > > (but I did not pick up 3/4, 4/4 because I did not know what we need after all). > > > > This should be discussed together with how Simon's script will look like. > > > > I can understand your Makefile code, but I still do not know > > how the entire overlay stuff will work in a big picture. > > How would you like to proceed? I can through together some changes on top > of Simon's patches as an initial proposal if that helps? > > I can use your format if you prefer. How would you select base+addonX among other base+addonY or base+addonZ configurations?
On Thu, Feb 29, 2024 at 11:35 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Thu, Feb 29, 2024 at 11:38 AM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > On Sun, Feb 25, 2024 at 4:21 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > On Fri, Feb 23, 2024 at 6:23 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > > > It is useful to have a list of all composite *.dtb files, along with > > > > their individual components, generated from the current build. > > > > > > > > With this commit, 'make dtbs' creates arch/*/boot/dts/dtbs-components, > > > > which lists the composite dtb files created in the current build. It > > > > maintains the order of the dtb-y additions in Makefiles although the > > > > order is not important for DTBs. > > > > > > > > This compliments the list of all *.dtb and *.dtbo files in dtbs-list, > > > > which only includes the files directly added to dtb-y. > > > > > > > > For example, consider this case: > > > > > > > > foo-dtbs := foo_base.dtb foo_overlay.dtbo > > > > dtb-y := bar.dtb foo.dtb > > > > > > > > In this example, the new list will include foo.dtb with foo_base.dtb and > > > > foo_overlay.dtbo on the same line, but not bar.dtb. > > > > > > > > Signed-off-by: Chen-Yu Tsai <wenst@chromium.org> > > > > --- > > > > Hi, > > > > > > > > I hacked up this new thing to list out the individual components of each > > > > composite dtb. I think this information would be useful for FIT image > > > > generation or other toolchains to consume. For example, instead of > > > > including each dtb, a toolchain could realize that some are put together > > > > using others, and if the bootloader supports it, put together commands > > > > to reassemble the end result from the original parts. > > > > > > > > This is based on and complements Masahiro-san's recent dtbs-list work. > > > > > > > > > > > > This is another format of my previous per-dtb "*.dtlst" > > > (but I did not pick up 3/4, 4/4 because I did not know what we need after all). > > > > > > This should be discussed together with how Simon's script will look like. > > > > > > I can understand your Makefile code, but I still do not know > > > how the entire overlay stuff will work in a big picture. > > > > How would you like to proceed? I can through together some changes on top > > of Simon's patches as an initial proposal if that helps? > > > > I can use your format if you prefer. > > > How would you select base+addonX among > other base+addonY or base+addonZ configurations? I assume you are alluding to the existing in-tree composite DTs that share the same board compatible strings? Under the current FIT image design with compatible strings populated from the FDTs, I don't think there's any way to automatically select among them. The FIT image simply does not have the information available. Nor do the overlays themselves. The toolchain can only either include all of them and let the bootloader figure things out, or filter out all the duplicates. With the composite list, at least it will be able to consistently keep only the base DT and drop the ones with the addons. In one of my previous replies to v9 I mentioned adding a user provided mapping between "configuration" compatible string and FDT filename. The mapping could be maintained in-tree for those base+addonXYZ FDTs if desired. Also, Simon's FIT image "extensions" proposal [1] adds more metadata to the FIT image to cover these addons that currently don't have distinct compatible strings. ChenYu [1] https://lore.kernel.org/u-boot/CAPnjgZ06s64C2ux1rABNAnMv3q4W++sjhNGCO_uPMH_9sTF7Mw@mail.gmail.com/
On Mon, Mar 4, 2024 at 1:37 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > On Thu, Feb 29, 2024 at 11:35 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Thu, Feb 29, 2024 at 11:38 AM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > On Sun, Feb 25, 2024 at 4:21 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > > > On Fri, Feb 23, 2024 at 6:23 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > > > > > It is useful to have a list of all composite *.dtb files, along with > > > > > their individual components, generated from the current build. > > > > > > > > > > With this commit, 'make dtbs' creates arch/*/boot/dts/dtbs-components, > > > > > which lists the composite dtb files created in the current build. It > > > > > maintains the order of the dtb-y additions in Makefiles although the > > > > > order is not important for DTBs. > > > > > > > > > > This compliments the list of all *.dtb and *.dtbo files in dtbs-list, > > > > > which only includes the files directly added to dtb-y. > > > > > > > > > > For example, consider this case: > > > > > > > > > > foo-dtbs := foo_base.dtb foo_overlay.dtbo > > > > > dtb-y := bar.dtb foo.dtb > > > > > > > > > > In this example, the new list will include foo.dtb with foo_base.dtb and > > > > > foo_overlay.dtbo on the same line, but not bar.dtb. > > > > > > > > > > Signed-off-by: Chen-Yu Tsai <wenst@chromium.org> > > > > > --- > > > > > Hi, > > > > > > > > > > I hacked up this new thing to list out the individual components of each > > > > > composite dtb. I think this information would be useful for FIT image > > > > > generation or other toolchains to consume. For example, instead of > > > > > including each dtb, a toolchain could realize that some are put together > > > > > using others, and if the bootloader supports it, put together commands > > > > > to reassemble the end result from the original parts. > > > > > > > > > > This is based on and complements Masahiro-san's recent dtbs-list work. > > > > > > > > > > > > > > > > This is another format of my previous per-dtb "*.dtlst" > > > > (but I did not pick up 3/4, 4/4 because I did not know what we need after all). > > > > > > > > This should be discussed together with how Simon's script will look like. > > > > > > > > I can understand your Makefile code, but I still do not know > > > > how the entire overlay stuff will work in a big picture. > > > > > > How would you like to proceed? I can through together some changes on top > > > of Simon's patches as an initial proposal if that helps? > > > > > > I can use your format if you prefer. > > > > > > How would you select base+addonX among > > other base+addonY or base+addonZ configurations? > > I assume you are alluding to the existing in-tree composite DTs that > share the same board compatible strings? Yes. It is possible to implement it, but I do not see a point to implement what we do not know how to use. > > Under the current FIT image design with compatible strings populated from > the FDTs, I don't think there's any way to automatically select among them. > The FIT image simply does not have the information available. Nor do the > overlays themselves. The toolchain can only either include all of them > and let the bootloader figure things out, or filter out all the duplicates. > With the composite list, at least it will be able to consistently keep > only the base DT and drop the ones with the addons. It makes the purpose of this work even more obscure. For the purpose of avoiding duplication, we can take the first DTB (or the smallest size) when we encounter a duplicated compatible string. > > In one of my previous replies to v9 I mentioned adding a user provided > mapping between "configuration" compatible string and FDT filename. The > mapping could be maintained in-tree for those base+addonXYZ FDTs if > desired. That is one way, but I do not think such a configuration file is maintainable. Rob suggested overwriting the compatible string, but I do not think we got consensus. > Also, Simon's FIT image "extensions" proposal [1] adds more metadata to > the FIT image to cover these addons that currently don't have distinct > compatible strings. I think this is yet another way, but I am not sure how to derive the extension compatible string. Even if we decide to implement base/overlay split, we may not need to add anything to Makefile. We already have .*.cmd files, and we can know if it is a combined DTB or not, by parsing the .*.cmd from the python script. It might be a bit messy, but it is what we do in scripts/clang-tools/gen_compile_commands.py > > > ChenYu > > [1] https://lore.kernel.org/u-boot/CAPnjgZ06s64C2ux1rABNAnMv3q4W++sjhNGCO_uPMH_9sTF7Mw@mail.gmail.com/ -- Best Regards Masahiro Yamada
On Wed, Mar 6, 2024 at 12:30 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Mon, Mar 4, 2024 at 1:37 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > On Thu, Feb 29, 2024 at 11:35 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > On Thu, Feb 29, 2024 at 11:38 AM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > > > On Sun, Feb 25, 2024 at 4:21 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > > > > > On Fri, Feb 23, 2024 at 6:23 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > > > > > > > It is useful to have a list of all composite *.dtb files, along with > > > > > > their individual components, generated from the current build. > > > > > > > > > > > > With this commit, 'make dtbs' creates arch/*/boot/dts/dtbs-components, > > > > > > which lists the composite dtb files created in the current build. It > > > > > > maintains the order of the dtb-y additions in Makefiles although the > > > > > > order is not important for DTBs. > > > > > > > > > > > > This compliments the list of all *.dtb and *.dtbo files in dtbs-list, > > > > > > which only includes the files directly added to dtb-y. > > > > > > > > > > > > For example, consider this case: > > > > > > > > > > > > foo-dtbs := foo_base.dtb foo_overlay.dtbo > > > > > > dtb-y := bar.dtb foo.dtb > > > > > > > > > > > > In this example, the new list will include foo.dtb with foo_base.dtb and > > > > > > foo_overlay.dtbo on the same line, but not bar.dtb. > > > > > > > > > > > > Signed-off-by: Chen-Yu Tsai <wenst@chromium.org> > > > > > > --- > > > > > > Hi, > > > > > > > > > > > > I hacked up this new thing to list out the individual components of each > > > > > > composite dtb. I think this information would be useful for FIT image > > > > > > generation or other toolchains to consume. For example, instead of > > > > > > including each dtb, a toolchain could realize that some are put together > > > > > > using others, and if the bootloader supports it, put together commands > > > > > > to reassemble the end result from the original parts. > > > > > > > > > > > > This is based on and complements Masahiro-san's recent dtbs-list work. > > > > > > > > > > > > > > > > > > > > This is another format of my previous per-dtb "*.dtlst" > > > > > (but I did not pick up 3/4, 4/4 because I did not know what we need after all). > > > > > > > > > > This should be discussed together with how Simon's script will look like. > > > > > > > > > > I can understand your Makefile code, but I still do not know > > > > > how the entire overlay stuff will work in a big picture. > > > > > > > > How would you like to proceed? I can through together some changes on top > > > > of Simon's patches as an initial proposal if that helps? > > > > > > > > I can use your format if you prefer. > > > > > > > > > How would you select base+addonX among > > > other base+addonY or base+addonZ configurations? > > > > I assume you are alluding to the existing in-tree composite DTs that > > share the same board compatible strings? > > > Yes. > It is possible to implement it, but I do not see a point > to implement what we do not know how to use. > > > > > > > Under the current FIT image design with compatible strings populated from > > the FDTs, I don't think there's any way to automatically select among them. > > The FIT image simply does not have the information available. Nor do the > > overlays themselves. The toolchain can only either include all of them > > and let the bootloader figure things out, or filter out all the duplicates. > > With the composite list, at least it will be able to consistently keep > > only the base DT and drop the ones with the addons. > > It makes the purpose of this work even more obscure. > > For the purpose of avoiding duplication, > we can take the first DTB (or the smallest size) > when we encounter a duplicated compatible string. Yes, that is also one way of doing it. I'm just not sure if it's fool proof. Taking the first one requires the Makefile be correctly ordered? Maybe that is implied because the base dtb needs to be built first? Also not sure about using size, as you could have an overlay that deletes stuff, and the resulting composite DTB could be streamlined and made smaller. > > > > In one of my previous replies to v9 I mentioned adding a user provided > > mapping between "configuration" compatible string and FDT filename. The > > mapping could be maintained in-tree for those base+addonXYZ FDTs if > > desired. > > > That is one way, but I do not think such a configuration file > is maintainable. I see. > Rob suggested overwriting the compatible string, > but I do not think we got consensus. Yeah, that's the simplest way. But IIRC on IRC someone mentioned that this doesn't work for stacking multiple overlays. I think prepending or appending compatible strings was proposed (subject to compiler support), but that doesn't really help our case, as all the composite DTBs would have the same fallback board compatible string. > > Also, Simon's FIT image "extensions" proposal [1] adds more metadata to > > the FIT image to cover these addons that currently don't have distinct > > compatible strings. > > I think this is yet another way, but I am not sure > how to derive the extension compatible string. I believe it is meant to be firmware specific, or at least defined by the first firmware / bootloader to implement support for that board. And also specific to a particular board family. So it may or may not live in the overlay itself. If not, then it would be an external file. If you do want it in the overlay to avoid maintaining an extra file, it would need to be brought up with the DT folks. This would be metadata associated with the overlay, not hardware descriptions, so I wonder about the acceptance. But I do think it is a better fit for the "board + a variety of modules" case. > Even if we decide to implement base/overlay split, > we may not need to add anything to Makefile. > > We already have .*.cmd files, and we can know > if it is a combined DTB or not, by parsing the .*.cmd > from the python script. > > It might be a bit messy, but it is what we do > in scripts/clang-tools/gen_compile_commands.py If that is an acceptable practice, I think that would work. Not sure how the dependency needs to be written though. For now I guess we should concentrate the discussions on Simon's FIT image series. Thanks ChenYu > > > > > > ChenYu > > > > [1] https://lore.kernel.org/u-boot/CAPnjgZ06s64C2ux1rABNAnMv3q4W++sjhNGCO_uPMH_9sTF7Mw@mail.gmail.com/ > -- > Best Regards > Masahiro Yamada
diff --git a/.gitignore b/.gitignore index c59dc60ba62e..bb5b3bbca4ef 100644 --- a/.gitignore +++ b/.gitignore @@ -52,6 +52,7 @@ *.xz *.zst Module.symvers +dtbs-components dtbs-list modules.order diff --git a/scripts/Makefile.build b/scripts/Makefile.build index 4971f54c855e..ba85c2385c9e 100644 --- a/scripts/Makefile.build +++ b/scripts/Makefile.build @@ -72,6 +72,7 @@ endif subdir-builtin := $(sort $(filter %/built-in.a, $(real-obj-y))) subdir-modorder := $(sort $(filter %/modules.order, $(obj-m))) subdir-dtbslist := $(sort $(filter %/dtbs-list, $(dtb-y))) +subdir-dtbscomp := $(sort $(filter %/dtbs-components, $(multi-dtb-y))) targets-for-builtin := $(extra-y) @@ -390,6 +391,7 @@ $(obj)/%.asn1.c $(obj)/%.asn1.h: $(src)/%.asn1 $(objtree)/scripts/asn1_compiler $(subdir-builtin): $(obj)/%/built-in.a: $(obj)/% ; $(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ; $(subdir-dtbslist): $(obj)/%/dtbs-list: $(obj)/% ; +$(subdir-dtbscomp): $(obj)/%/dtbs-components: $(obj)/% ; # # Rule to compile a set of .o files into one .a file (without symbol table) @@ -422,6 +424,20 @@ $(obj)/modules.order: $(obj-m) FORCE $(obj)/dtbs-list: $(dtb-y) FORCE $(call if_changed,gen_order) +# +# Rule to create dtbs-components +# +# This is a list of composite dtb(s), along with each dtb's components, +# from the current Makefile and its sub-directories. + +cmd_gen_dtb_components = { $(foreach m, $(real-prereqs), \ + $(if $(filter %/$(notdir $@), $m), cat $m, \ + echo $m: $(addprefix $(obj)/,$($(notdir $(m:%.dtb=%-dtbs))))); \ + ) :; } > $@ + +$(obj)/dtbs-components: $(multi-dtb-y) FORCE + $(call if_changed,gen_dtb_components) + # # Rule to compile a set of .o files into one .a file (with symbol table) # diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index dbcac396329e..7c2127a84ac2 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -61,7 +61,6 @@ real-search = $(foreach m, $1, $(if $(call suffix-search, $m, $2, $3 -), $(call multi-obj-y := $(call multi-search, $(obj-y), .o, -objs -y) multi-obj-m := $(call multi-search, $(obj-m), .o, -objs -y -m) multi-obj-ym := $(multi-obj-y) $(multi-obj-m) - # Replace multi-part objects by their individual parts, # including built-in.a from subdirectories real-obj-y := $(call real-search, $(obj-y), .o, -objs -y) @@ -91,6 +90,11 @@ real-dtb-y := $(call real-search, $(dtb-y), .dtb, -dtbs) # Base DTB that overlay is applied onto base-dtb-y := $(filter %.dtb, $(call real-search, $(multi-dtb-y), .dtb, -dtbs)) +ifdef need-dtbslist +multi-dtb-y += $(addsuffix /dtbs-components, $(subdir-ym)) +always-y += dtbs-components +endif + always-y += $(dtb-y) # Add subdir path @@ -406,7 +410,7 @@ cmd_dtc = $(HOSTCC) -E $(dtc_cpp_flags) -x assembler-with-cpp -o $(dtc-tmp) $< ; quiet_cmd_fdtoverlay = DTOVL $@ cmd_fdtoverlay = $(objtree)/scripts/dtc/fdtoverlay -o $@ -i $(real-prereqs) -$(multi-dtb-y): FORCE +$(filter-out %/dtbs-components, multi-dtb-y): FORCE $(call if_changed,fdtoverlay) $(call multi_depend, $(multi-dtb-y), .dtb, -dtbs)
It is useful to have a list of all composite *.dtb files, along with their individual components, generated from the current build. With this commit, 'make dtbs' creates arch/*/boot/dts/dtbs-components, which lists the composite dtb files created in the current build. It maintains the order of the dtb-y additions in Makefiles although the order is not important for DTBs. This compliments the list of all *.dtb and *.dtbo files in dtbs-list, which only includes the files directly added to dtb-y. For example, consider this case: foo-dtbs := foo_base.dtb foo_overlay.dtbo dtb-y := bar.dtb foo.dtb In this example, the new list will include foo.dtb with foo_base.dtb and foo_overlay.dtbo on the same line, but not bar.dtb. Signed-off-by: Chen-Yu Tsai <wenst@chromium.org> --- Hi, I hacked up this new thing to list out the individual components of each composite dtb. I think this information would be useful for FIT image generation or other toolchains to consume. For example, instead of including each dtb, a toolchain could realize that some are put together using others, and if the bootloader supports it, put together commands to reassemble the end result from the original parts. This is based on and complements Masahiro-san's recent dtbs-list work. .gitignore | 1 + scripts/Makefile.build | 16 ++++++++++++++++ scripts/Makefile.lib | 8 ++++++-- 3 files changed, 23 insertions(+), 2 deletions(-)