| Message ID | 20231202035511.487946-3-sjg@chromium.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | arm64: Add a build target for Flat Image Tree | expand |
Hi Simon, Thank you for the patch. On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > Add a script which produces a Flat Image Tree (FIT), a single file > containing the built kernel and associated devicetree files. > Compression defaults to gzip which gives a good balance of size and > performance. > > The files compress from about 86MB to 24MB using this approach. > > The FIT can be used by bootloaders which support it, such as U-Boot > and Linuxboot. It permits automatic selection of the correct > devicetree, matching the compatible string of the running board with > the closest compatible string in the FIT. There is no need for > filenames or other workarounds. > > Add a 'make image.fit' build target for arm64, as well. Use > FIT_COMPRESSION to select a different algorithm. > > The FIT can be examined using 'dumpimage -l'. > > This features requires pylibfdt (use 'pip install libfdt'). It also > requires compression utilities for the algorithm being used. Supported > compression options are the same as the Image.xxx files. For now there > is no way to change the compression other than by editing the rule for > $(obj)/image.fit > > While FIT supports a ramdisk / initrd, no attempt is made to support > this here, since it must be built separately from the Linux build. FIT images are very useful, so I think this is a very welcome addition to the kernel build system. It can get tricky though: given the versatile nature of FIT images, there can't be any one-size-fits-them-all solution to build them, and striking the right balance between what makes sense for the kernel and the features that users may request will probably lead to bikeshedding. As we all love bikeshedding, I thought I would start selfishly, with a personal use case :-) This isn't a yak-shaving request though, I don't see any reason to delay merging this series. Have you envisioned building FIT images with a subset of DTBs, or adding DTBOs ? Both would be fairly trivial extensions to this script by extending the supported command line arguments. It would perhaps be more difficult to integrate in the kernel build system though. This leads me to a second question: would you consider merging extensions to this script if they are not used by the kernel build system, but meant for users who manually invoke the script ? More generally, is the script meant to be used stand-alone as well, in which case its command line arguments need to remain backward-compatible, or do you see it as being internal to the kernel ? > Signed-off-by: Simon Glass <sjg@chromium.org> > --- > > Changes in v9: > - Move the compression control into Makefile.lib > > Changes in v8: > - Drop compatible string in FDT node > - Correct sorting of MAINTAINERS to before ARM64 PORT > - Turn compress part of the make_fit.py comment in to a sentence > - Add two blank lines before parse_args() and setup_fit() > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > - Use '$(<D)/dts' instead of '$(dir $<)dts' > - Add 'mkimage' details Documentation/process/changes.rst > - Allow changing the compression used > - Tweak cover letter since there is only one clean-up patch > > Changes in v7: > - Add Image as a dependency of image.fit > - Drop kbuild tag > - Add dependency on dtbs > - Drop unnecessary path separator for dtbs > - Rebase to -next > > Changes in v5: > - Drop patch previously applied > - Correct compression rule which was broken in v4 > > Changes in v4: > - Use single quotes for UIMAGE_NAME > > Changes in v3: > - Drop temporary file image.itk > - Drop patch 'Use double quotes for image name' > - Drop double quotes in use of UIMAGE_NAME > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > - Avoid hard-coding "arm64" for the DT architecture > > Changes in v2: > - Drop patch previously applied > - Add .gitignore file > - Move fit rule to Makefile.lib using an intermediate file > - Drop dependency on CONFIG_EFI_ZBOOT > - Pick up .dtb files separately from the kernel > - Correct pylint too-many-args warning for write_kernel() > - Include the kernel image in the file count > - Add a pointer to the FIT spec and mention of its wide industry usage > - Mention the kernel version in the FIT description > > Documentation/process/changes.rst | 9 + > MAINTAINERS | 7 + > arch/arm64/Makefile | 7 +- > arch/arm64/boot/.gitignore | 1 + > arch/arm64/boot/Makefile | 6 +- > scripts/Makefile.lib | 16 ++ > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > 7 files changed, 334 insertions(+), 3 deletions(-) > create mode 100755 scripts/make_fit.py > > diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst > index bb96ca0f774b..cad51bd5bd62 100644 > --- a/Documentation/process/changes.rst > +++ b/Documentation/process/changes.rst > @@ -62,6 +62,7 @@ Sphinx\ [#f1]_ 1.7 sphinx-build --version > cpio any cpio --version > GNU tar 1.28 tar --version > gtags (optional) 6.6.5 gtags --version > +mkimage (optional) 2017.01 mkimage --version > ====================== =============== ======================================== > > .. [#f1] Sphinx is needed only to build the Kernel documentation > @@ -189,6 +190,14 @@ The kernel build requires GNU GLOBAL version 6.6.5 or later to generate > tag files through ``make gtags``. This is due to its use of the gtags > ``-C (--directory)`` flag. > > +mkimage > +------- > + > +This tool is used when building a Flat Image Tree (FIT), commonly used on ARM > +platforms. The tool is available via the ``u-boot-tools`` package or can be > +built from the U-Boot source code. See the instructions at > +https://docs.u-boot.org/en/latest/build/tools.html#building-tools-for-linux > + > System utilities > **************** > > diff --git a/MAINTAINERS b/MAINTAINERS > index 9d46229fe21b..d2d17f0d6e64 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -3037,6 +3037,13 @@ F: drivers/mmc/host/sdhci-of-arasan.c > N: zynq > N: xilinx > > +ARM64 FIT SUPPORT > +M: Simon Glass <sjg@chromium.org> > +L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) > +S: Maintained > +F: arch/arm64/boot/Makefile > +F: scripts/make_fit.py > + > ARM64 PORT (AARCH64 ARCHITECTURE) > M: Catalin Marinas <catalin.marinas@arm.com> > M: Will Deacon <will@kernel.org> > diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile > index 1bd4fae6e806..6b893dc454b7 100644 > --- a/arch/arm64/Makefile > +++ b/arch/arm64/Makefile > @@ -150,7 +150,7 @@ libs-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a > # Default target when executing plain make > boot := arch/arm64/boot > > -BOOT_TARGETS := Image vmlinuz.efi > +BOOT_TARGETS := Image vmlinuz.efi image.fit > > PHONY += $(BOOT_TARGETS) > > @@ -162,7 +162,9 @@ endif > > all: $(notdir $(KBUILD_IMAGE)) > > -vmlinuz.efi: Image > +image.fit: dtbs > + > +vmlinuz.efi image.fit: Image > $(BOOT_TARGETS): vmlinux > $(Q)$(MAKE) $(build)=$(boot) $(boot)/$@ > > @@ -215,6 +217,7 @@ virtconfig: > define archhelp > echo '* Image.gz - Compressed kernel image (arch/$(ARCH)/boot/Image.gz)' > echo ' Image - Uncompressed kernel image (arch/$(ARCH)/boot/Image)' > + echo ' image.fit - Flat Image Tree (arch/$(ARCH)/boot/image.fit)' > echo ' install - Install uncompressed kernel' > echo ' zinstall - Install compressed kernel' > echo ' Install using (your) ~/bin/installkernel or' > diff --git a/arch/arm64/boot/.gitignore b/arch/arm64/boot/.gitignore > index af5dc61f8b43..abaae9de1bdd 100644 > --- a/arch/arm64/boot/.gitignore > +++ b/arch/arm64/boot/.gitignore > @@ -2,3 +2,4 @@ > Image > Image.gz > vmlinuz* > +image.fit > diff --git a/arch/arm64/boot/Makefile b/arch/arm64/boot/Makefile > index 1761f5972443..b835c0880d1c 100644 > --- a/arch/arm64/boot/Makefile > +++ b/arch/arm64/boot/Makefile > @@ -16,7 +16,8 @@ > > OBJCOPYFLAGS_Image :=-O binary -R .note -R .note.gnu.build-id -R .comment -S > > -targets := Image Image.bz2 Image.gz Image.lz4 Image.lzma Image.lzo Image.zst > +targets := Image Image.bz2 Image.gz Image.lz4 Image.lzma Image.lzo \ > + Image.zst image.fit > > $(obj)/Image: vmlinux FORCE > $(call if_changed,objcopy) > @@ -39,6 +40,9 @@ $(obj)/Image.lzo: $(obj)/Image FORCE > $(obj)/Image.zst: $(obj)/Image FORCE > $(call if_changed,zstd) > > +$(obj)/image.fit: $(obj)/Image FORCE > + $(call cmd,fit) > + > EFI_ZBOOT_PAYLOAD := Image > EFI_ZBOOT_BFD_TARGET := elf64-littleaarch64 > EFI_ZBOOT_MACH_TYPE := ARM64 > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib > index 1a965fe68e01..1c60d594932c 100644 > --- a/scripts/Makefile.lib > +++ b/scripts/Makefile.lib > @@ -496,6 +496,22 @@ quiet_cmd_uimage = UIMAGE $@ > -a $(UIMAGE_LOADADDR) -e $(UIMAGE_ENTRYADDR) \ > -n '$(UIMAGE_NAME)' -d $< $@ > > +# Flat Image Tree (FIT) > +# This allows for packaging of a kernel and all devicetrees files, using > +# compression. > +# --------------------------------------------------------------------------- > + > +MAKE_FIT := $(srctree)/scripts/make_fit.py > + > +# Use this to override the compression algorithm > +FIT_COMPRESSION ?= gzip > + > +quiet_cmd_fit = FIT $@ > + cmd_fit = $(MAKE_FIT) -f $@ --arch $(UIMAGE_ARCH) --os linux \ > + --name '$(UIMAGE_NAME)' \ > + --compress $(FIT_COMPRESSION) -k $< \ > + $(<D)/dts > + > # XZ > # --------------------------------------------------------------------------- > # Use xzkern to compress the kernel image and xzmisc to compress other things. > diff --git a/scripts/make_fit.py b/scripts/make_fit.py > new file mode 100755 > index 000000000000..e616b0d7a84a > --- /dev/null > +++ b/scripts/make_fit.py > @@ -0,0 +1,291 @@ > +#!/usr/bin/env python3 > +# SPDX-License-Identifier: GPL-2.0+ > +# > +# Copyright 2023 Google LLC > +# Written by Simon Glass <sjg@chromium.org> > +# > + > +"""Build a FIT containing a lot of devicetree files > + > +Usage: > + make_fit.py -A arm64 -n 'Linux-6.6' -O linux > + -f arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk > + /tmp/kern/arch/arm64/boot/dts/ -E -c gzip > + > +Creates a FIT containing the supplied kernel and a directory containing the > +devicetree files. > + > +Use -E to generate an external FIT (where the data is placed after the > +FIT data structure). This allows parsing of the data without loading > +the entire FIT. > + > +Use -c to compress the data, using bzip2, gzip, lz4, lzma, lzo and > +zstd algorithms. > + > +The resulting FIT can be booted by bootloaders which support FIT, such > +as U-Boot, Linuxboot, Tianocore, etc. > + > +Note that this tool does not yet support adding a ramdisk / initrd. > +""" > + > +import argparse > +import collections > +import os > +import subprocess > +import sys > +import tempfile > +import time > + > +import libfdt > + > + > +# Tool extension and the name of the command-line tools > +CompTool = collections.namedtuple('CompTool', 'ext,tools') > + > +COMP_TOOLS = { > + 'bzip2': CompTool('.bz2', 'bzip2'), > + 'gzip': CompTool('.gz', 'pigz,gzip'), > + 'lz4': CompTool('.lz4', 'lz4'), > + 'lzma': CompTool('.lzma', 'lzma'), > + 'lzo': CompTool('.lzo', 'lzop'), > + 'zstd': CompTool('.zstd', 'zstd'), > +} > + > + > +def parse_args(): > + """Parse the program ArgumentParser > + > + Returns: > + Namespace object containing the arguments > + """ > + epilog = 'Build a FIT from a directory tree containing .dtb files' > + parser = argparse.ArgumentParser(epilog=epilog) > + parser.add_argument('-A', '--arch', type=str, required=True, > + help='Specifies the architecture') > + parser.add_argument('-c', '--compress', type=str, default='none', > + help='Specifies the compression') > + parser.add_argument('-E', '--external', action='store_true', > + help='Convert the FIT to use external data') > + parser.add_argument('-n', '--name', type=str, required=True, > + help='Specifies the name') > + parser.add_argument('-O', '--os', type=str, required=True, > + help='Specifies the operating system') > + parser.add_argument('-f', '--fit', type=str, required=True, > + help='Specifies the output file (.fit)') > + parser.add_argument('-k', '--kernel', type=str, required=True, > + help='Specifies the (uncompressed) kernel input file (.itk)') > + parser.add_argument('srcdir', type=str, nargs='*', > + help='Specifies the directory tree that contains .dtb files') > + > + return parser.parse_args() > + > + > +def setup_fit(fsw, name): > + """Make a start on writing the FIT > + > + Outputs the root properties and the 'images' node > + > + Args: > + fsw (libfdt.FdtSw): Object to use for writing > + name (str): Name of kernel image > + """ > + fsw.INC_SIZE = 65536 > + fsw.finish_reservemap() > + fsw.begin_node('') > + fsw.property_string('description', f'{name} with devicetree set') > + fsw.property_u32('#address-cells', 1) > + > + fsw.property_u32('timestamp', int(time.time())) > + fsw.begin_node('images') > + > + > +def write_kernel(fsw, data, args): > + """Write out the kernel image > + > + Writes a kernel node along with the required properties > + > + Args: > + fsw (libfdt.FdtSw): Object to use for writing > + data (bytes): Data to write (possibly compressed) > + args (Namespace): Contains necessary strings: > + arch: FIT architecture, e.g. 'arm64' > + fit_os: Operating Systems, e.g. 'linux' > + name: Name of OS, e.g. 'Linux-6.6.0-rc7' > + compress: Compression algorithm to use, e.g. 'gzip' > + """ > + with fsw.add_node('kernel'): > + fsw.property_string('description', args.name) > + fsw.property_string('type', 'kernel_noload') > + fsw.property_string('arch', args.arch) > + fsw.property_string('os', args.os) > + fsw.property_string('compression', args.compress) > + fsw.property('data', data) > + fsw.property_u32('load', 0) > + fsw.property_u32('entry', 0) > + > + > +def finish_fit(fsw, entries): > + """Finish the FIT ready for use > + > + Writes the /configurations node and subnodes > + > + Args: > + fsw (libfdt.FdtSw): Object to use for writing > + entries (list of tuple): List of configurations: > + str: Description of model > + str: Compatible stringlist > + """ > + fsw.end_node() > + seq = 0 > + with fsw.add_node('configurations'): > + for model, compat in entries: > + seq += 1 > + with fsw.add_node(f'conf-{seq}'): > + fsw.property('compatible', bytes(compat)) > + fsw.property_string('description', model) > + fsw.property_string('fdt', f'fdt-{seq}') > + fsw.property_string('kernel', 'kernel') > + fsw.end_node() > + > + > +def compress_data(inf, compress): > + """Compress data using a selected algorithm > + > + Args: > + inf (IOBase): Filename containing the data to compress > + compress (str): Compression algorithm, e.g. 'gzip' > + > + Return: > + bytes: Compressed data > + """ > + if compress == 'none': > + return inf.read() > + > + comp = COMP_TOOLS.get(compress) > + if not comp: > + raise ValueError(f"Unknown compression algorithm '{compress}'") > + > + with tempfile.NamedTemporaryFile() as comp_fname: > + with open(comp_fname.name, 'wb') as outf: > + done = False > + for tool in comp.tools.split(','): > + try: > + subprocess.call([tool, '-c'], stdin=inf, stdout=outf) > + done = True > + break > + except FileNotFoundError: > + pass > + if not done: > + raise ValueError(f'Missing tool(s): {comp.tools}\n') > + with open(comp_fname.name, 'rb') as compf: > + comp_data = compf.read() > + return comp_data > + > + > +def output_dtb(fsw, seq, fname, arch, compress): > + """Write out a single devicetree to the FIT > + > + Args: > + fsw (libfdt.FdtSw): Object to use for writing > + seq (int): Sequence number (1 for first) > + fmame (str): Filename containing the DTB > + arch: FIT architecture, e.g. 'arm64' > + compress (str): Compressed algorithm, e.g. 'gzip' > + > + Returns: > + tuple: > + str: Model name > + bytes: Compatible stringlist > + """ > + with fsw.add_node(f'fdt-{seq}'): > + # Get the compatible / model information > + with open(fname, 'rb') as inf: > + data = inf.read() > + fdt = libfdt.FdtRo(data) > + model = fdt.getprop(0, 'model').as_str() > + compat = fdt.getprop(0, 'compatible') > + > + fsw.property_string('description', model) > + fsw.property_string('type', 'flat_dt') > + fsw.property_string('arch', arch) > + fsw.property_string('compression', compress) > + fsw.property('compatible', bytes(compat)) > + > + with open(fname, 'rb') as inf: > + compressed = compress_data(inf, compress) > + fsw.property('data', compressed) > + return model, compat > + > + > +def build_fit(args): > + """Build the FIT from the provided files and arguments > + > + Args: > + args (Namespace): Program arguments > + > + Returns: > + tuple: > + bytes: FIT data > + int: Number of configurations generated > + size: Total uncompressed size of data > + """ > + fsw = libfdt.FdtSw() > + setup_fit(fsw, args.name) > + seq = 0 > + size = 0 > + entries = [] > + > + # Handle the kernel > + with open(args.kernel, 'rb') as inf: > + comp_data = compress_data(inf, args.compress) > + size += os.path.getsize(args.kernel) > + write_kernel(fsw, comp_data, args) > + > + for path in args.srcdir: > + # Handle devicetree files > + if os.path.isdir(path): > + for dirpath, _, fnames in os.walk(path): > + for fname in fnames: > + if os.path.splitext(fname)[1] != '.dtb': > + continue > + pathname = os.path.join(dirpath, fname) > + seq += 1 > + size += os.path.getsize(pathname) > + model, compat = output_dtb(fsw, seq, pathname, > + args.arch, args.compress) > + entries.append([model, compat]) > + > + finish_fit(fsw, entries) > + > + # Include the kernel itself in the returned file count > + return fsw.as_fdt().as_bytearray(), seq + 1, size > + > + > +def run_make_fit(): > + """Run the tool's main logic""" > + args = parse_args() > + > + out_data, count, size = build_fit(args) > + with open(args.fit, 'wb') as outf: > + outf.write(out_data) > + > + ext_fit_size = None > + if args.external: > + mkimage = os.environ.get('MKIMAGE', 'mkimage') > + subprocess.check_call([mkimage, '-E', '-F', args.fit], > + stdout=subprocess.DEVNULL) > + > + with open(args.fit, 'rb') as inf: > + data = inf.read() > + ext_fit = libfdt.FdtRo(data) > + ext_fit_size = ext_fit.totalsize() > + > + comp_size = len(out_data) > + print(f'FIT size {comp_size:#x}/{comp_size / 1024 / 1024:.1f} MB', end='') > + if ext_fit_size: > + print(f', header {ext_fit_size:#x}/{ext_fit_size / 1024:.1f} KB', end='') > + print(f', {count} files, uncompressed {size / 1024 / 1024:.1f} MB') > + > + > +if __name__ == "__main__": > + sys.exit(run_make_fit())
Hi Laurent, On Sun, 3 Dec 2023 at 08:33, Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote: > > Hi Simon, > > Thank you for the patch. > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > Add a script which produces a Flat Image Tree (FIT), a single file > > containing the built kernel and associated devicetree files. > > Compression defaults to gzip which gives a good balance of size and > > performance. > > > > The files compress from about 86MB to 24MB using this approach. > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > and Linuxboot. It permits automatic selection of the correct > > devicetree, matching the compatible string of the running board with > > the closest compatible string in the FIT. There is no need for > > filenames or other workarounds. > > > > Add a 'make image.fit' build target for arm64, as well. Use > > FIT_COMPRESSION to select a different algorithm. > > > > The FIT can be examined using 'dumpimage -l'. > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > requires compression utilities for the algorithm being used. Supported > > compression options are the same as the Image.xxx files. For now there > > is no way to change the compression other than by editing the rule for > > $(obj)/image.fit > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > this here, since it must be built separately from the Linux build. > > FIT images are very useful, so I think this is a very welcome addition > to the kernel build system. It can get tricky though: given the > versatile nature of FIT images, there can't be any > one-size-fits-them-all solution to build them, and striking the right > balance between what makes sense for the kernel and the features that > users may request will probably lead to bikeshedding. As we all love > bikeshedding, I thought I would start selfishly, with a personal use > case :-) This isn't a yak-shaving request though, I don't see any reason > to delay merging this series. OK, sounds good! > > Have you envisioned building FIT images with a subset of DTBs, or adding > DTBOs ? Both would be fairly trivial extensions to this script by > extending the supported command line arguments. It would perhaps be more > difficult to integrate in the kernel build system though. This leads me > to a second question: would you consider merging extensions to this > script if they are not used by the kernel build system, but meant for > users who manually invoke the script ? More generally, is the script > meant to be used stand-alone as well, in which case its command line > arguments need to remain backward-compatible, or do you see it as being > internal to the kernel ? The script as written is internal to the kernel, but I am sure it could be expanded in some ways. I am waiting to see it merged before worrying too much about what might happen in the future! [..] Regards, Simon
Hello Simon, On 02.12.23 04:54, Simon Glass wrote: > Add a script which produces a Flat Image Tree (FIT), a single file > containing the built kernel and associated devicetree files. > Compression defaults to gzip which gives a good balance of size and > performance. > > The files compress from about 86MB to 24MB using this approach. > > The FIT can be used by bootloaders which support it, such as U-Boot > and Linuxboot. It permits automatic selection of the correct > devicetree, matching the compatible string of the running board with > the closest compatible string in the FIT. There is no need for > filenames or other workarounds. > > Add a 'make image.fit' build target for arm64, as well. Use > FIT_COMPRESSION to select a different algorithm. > > The FIT can be examined using 'dumpimage -l'. > > This features requires pylibfdt (use 'pip install libfdt'). It also > requires compression utilities for the algorithm being used. Supported > compression options are the same as the Image.xxx files. For now there > is no way to change the compression other than by editing the rule for > $(obj)/image.fit > > While FIT supports a ramdisk / initrd, no attempt is made to support > this here, since it must be built separately from the Linux build. > > Signed-off-by: Simon Glass <sjg@chromium.org> kernel_noload support is now in barebox next branch and I tested this series against it: Tested-by: Ahmad Fatoum <a.fatoum@pengutronix.de> # barebox > +"""Build a FIT containing a lot of devicetree files > + > +Usage: > + make_fit.py -A arm64 -n 'Linux-6.6' -O linux > + -f arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk > + /tmp/kern/arch/arm64/boot/dts/ -E -c gzip > + > +Creates a FIT containing the supplied kernel and a directory containing the > +devicetree files. > + > +Use -E to generate an external FIT (where the data is placed after the > +FIT data structure). This allows parsing of the data without loading > +the entire FIT. > + > +Use -c to compress the data, using bzip2, gzip, lz4, lzma, lzo and > +zstd algorithms. > + > +The resulting FIT can be booted by bootloaders which support FIT, such > +as U-Boot, Linuxboot, Tianocore, etc. Feel free to add barebox to the list. Did you check whether Linuxboot and Tianocore support kernel_noload? > + fsw.property_u32('load', 0) > + fsw.property_u32('entry', 0) I still think load and entry dummy values are confusing and should be dropped. > + with fsw.add_node(f'fdt-{seq}'): > + # Get the compatible / model information > + with open(fname, 'rb') as inf: > + data = inf.read() > + fdt = libfdt.FdtRo(data) > + model = fdt.getprop(0, 'model').as_str() > + compat = fdt.getprop(0, 'compatible') > + > + fsw.property_string('description', model) > + fsw.property_string('type', 'flat_dt') > + fsw.property_string('arch', arch) > + fsw.property_string('compression', compress) > + fsw.property('compatible', bytes(compat)) > + > + with open(fname, 'rb') as inf: > + compressed = compress_data(inf, compress) > + fsw.property('data', compressed) > + return model, compat After Doug's elaboration, extracting multiple compatibles is fine by me. Cheers, Ahmad
Hi Ahmad, On Tue, 5 Dec 2023 at 04:48, Ahmad Fatoum <a.fatoum@pengutronix.de> wrote: > > Hello Simon, > > On 02.12.23 04:54, Simon Glass wrote: > > Add a script which produces a Flat Image Tree (FIT), a single file > > containing the built kernel and associated devicetree files. > > Compression defaults to gzip which gives a good balance of size and > > performance. > > > > The files compress from about 86MB to 24MB using this approach. > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > and Linuxboot. It permits automatic selection of the correct > > devicetree, matching the compatible string of the running board with > > the closest compatible string in the FIT. There is no need for > > filenames or other workarounds. > > > > Add a 'make image.fit' build target for arm64, as well. Use > > FIT_COMPRESSION to select a different algorithm. > > > > The FIT can be examined using 'dumpimage -l'. > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > requires compression utilities for the algorithm being used. Supported > > compression options are the same as the Image.xxx files. For now there > > is no way to change the compression other than by editing the rule for > > $(obj)/image.fit > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > this here, since it must be built separately from the Linux build. > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > kernel_noload support is now in barebox next branch and I tested this > series against it: > > Tested-by: Ahmad Fatoum <a.fatoum@pengutronix.de> # barebox > OK great thank you. > > +"""Build a FIT containing a lot of devicetree files > > + > > +Usage: > > + make_fit.py -A arm64 -n 'Linux-6.6' -O linux > > + -f arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk > > + /tmp/kern/arch/arm64/boot/dts/ -E -c gzip > > + > > +Creates a FIT containing the supplied kernel and a directory containing the > > +devicetree files. > > + > > +Use -E to generate an external FIT (where the data is placed after the > > +FIT data structure). This allows parsing of the data without loading > > +the entire FIT. > > + > > +Use -c to compress the data, using bzip2, gzip, lz4, lzma, lzo and > > +zstd algorithms. > > + > > +The resulting FIT can be booted by bootloaders which support FIT, such > > +as U-Boot, Linuxboot, Tianocore, etc. > > Feel free to add barebox to the list. Did you check whether Linuxboot and > Tianocore support kernel_noload? Only what I was told by people in those projects. They may not even look at the load address, but I am not an expert on that. > > > + fsw.property_u32('load', 0) > > + fsw.property_u32('entry', 0) > > I still think load and entry dummy values are confusing and should be dropped. This is what the spec requires at present. But I agree we should change it. I will dig into that at some point to see what is needed. > > > + with fsw.add_node(f'fdt-{seq}'): > > + # Get the compatible / model information > > + with open(fname, 'rb') as inf: > > + data = inf.read() > > + fdt = libfdt.FdtRo(data) > > + model = fdt.getprop(0, 'model').as_str() > > + compat = fdt.getprop(0, 'compatible') > > + > > + fsw.property_string('description', model) > > + fsw.property_string('type', 'flat_dt') > > + fsw.property_string('arch', arch) > > + fsw.property_string('compression', compress) > > + fsw.property('compatible', bytes(compat)) > > + > > + with open(fname, 'rb') as inf: > > + compressed = compress_data(inf, compress) > > + fsw.property('data', compressed) > > + return model, compat > > After Doug's elaboration, extracting multiple compatibles is fine by me. OK good. Regards, Simon
On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > Hi Simon, > > Thank you for the patch. > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > Add a script which produces a Flat Image Tree (FIT), a single file > > containing the built kernel and associated devicetree files. > > Compression defaults to gzip which gives a good balance of size and > > performance. > > > > The files compress from about 86MB to 24MB using this approach. > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > and Linuxboot. It permits automatic selection of the correct > > devicetree, matching the compatible string of the running board with > > the closest compatible string in the FIT. There is no need for > > filenames or other workarounds. > > > > Add a 'make image.fit' build target for arm64, as well. Use > > FIT_COMPRESSION to select a different algorithm. > > > > The FIT can be examined using 'dumpimage -l'. > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > requires compression utilities for the algorithm being used. Supported > > compression options are the same as the Image.xxx files. For now there > > is no way to change the compression other than by editing the rule for > > $(obj)/image.fit > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > this here, since it must be built separately from the Linux build. > > FIT images are very useful, so I think this is a very welcome addition > to the kernel build system. It can get tricky though: given the > versatile nature of FIT images, there can't be any > one-size-fits-them-all solution to build them, and striking the right > balance between what makes sense for the kernel and the features that > users may request will probably lead to bikeshedding. As we all love > bikeshedding, I thought I would start selfishly, with a personal use > case :-) This isn't a yak-shaving request though, I don't see any reason > to delay merging this series. > > Have you envisioned building FIT images with a subset of DTBs, or adding > DTBOs ? Both would be fairly trivial extensions to this script by > extending the supported command line arguments. It would perhaps be more > difficult to integrate in the kernel build system though. This leads me > to a second question: would you consider merging extensions to this > script if they are not used by the kernel build system, but meant for > users who manually invoke the script ? More generally, is the script We'd also be interested in some customization, though in a different way. We imagine having a rule file that says X compatible string should map to A base DTB, plus B and C DTBO for the configuration section. The base DTB would carry all common elements of some device, while the DTBOs carry all the possible second source components, like different display panels or MIPI cameras for instance. This could drastically reduce the size of FIT images in ChromeOS by deduplicating all the common stuff. > meant to be used stand-alone as well, in which case its command line > arguments need to remain backward-compatible, or do you see it as being > internal to the kernel ? [...] ChenYu
On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > Hi Simon, > > > > Thank you for the patch. > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > containing the built kernel and associated devicetree files. > > > Compression defaults to gzip which gives a good balance of size and > > > performance. > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > and Linuxboot. It permits automatic selection of the correct > > > devicetree, matching the compatible string of the running board with > > > the closest compatible string in the FIT. There is no need for > > > filenames or other workarounds. > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > FIT_COMPRESSION to select a different algorithm. > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > requires compression utilities for the algorithm being used. Supported > > > compression options are the same as the Image.xxx files. For now there > > > is no way to change the compression other than by editing the rule for > > > $(obj)/image.fit > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > this here, since it must be built separately from the Linux build. > > > > FIT images are very useful, so I think this is a very welcome addition > > to the kernel build system. It can get tricky though: given the > > versatile nature of FIT images, there can't be any > > one-size-fits-them-all solution to build them, and striking the right > > balance between what makes sense for the kernel and the features that > > users may request will probably lead to bikeshedding. As we all love > > bikeshedding, I thought I would start selfishly, with a personal use > > case :-) This isn't a yak-shaving request though, I don't see any reason > > to delay merging this series. > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > DTBOs ? Both would be fairly trivial extensions to this script by > > extending the supported command line arguments. It would perhaps be more > > difficult to integrate in the kernel build system though. This leads me > > to a second question: would you consider merging extensions to this > > script if they are not used by the kernel build system, but meant for > > users who manually invoke the script ? More generally, is the script > > We'd also be interested in some customization, though in a different way. > We imagine having a rule file that says X compatible string should map > to A base DTB, plus B and C DTBO for the configuration section. The base > DTB would carry all common elements of some device, while the DTBOs > carry all the possible second source components, like different display > panels or MIPI cameras for instance. This could drastically reduce the > size of FIT images in ChromeOS by deduplicating all the common stuff. Do you envision the "mapping" compatible string mapping to a config section in the FIT image, that would bundle the base DTB and the DTBOs ? > > meant to be used stand-alone as well, in which case its command line > > arguments need to remain backward-compatible, or do you see it as being > > internal to the kernel ? > > [...] > > ChenYu
Hi, On Thu, 7 Dec 2023 at 07:38, Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote: > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > Hi Simon, > > > > > > Thank you for the patch. > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > containing the built kernel and associated devicetree files. > > > > Compression defaults to gzip which gives a good balance of size and > > > > performance. > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > and Linuxboot. It permits automatic selection of the correct > > > > devicetree, matching the compatible string of the running board with > > > > the closest compatible string in the FIT. There is no need for > > > > filenames or other workarounds. > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > requires compression utilities for the algorithm being used. Supported > > > > compression options are the same as the Image.xxx files. For now there > > > > is no way to change the compression other than by editing the rule for > > > > $(obj)/image.fit > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > this here, since it must be built separately from the Linux build. > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > to the kernel build system. It can get tricky though: given the > > > versatile nature of FIT images, there can't be any > > > one-size-fits-them-all solution to build them, and striking the right > > > balance between what makes sense for the kernel and the features that > > > users may request will probably lead to bikeshedding. As we all love > > > bikeshedding, I thought I would start selfishly, with a personal use > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > to delay merging this series. > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > extending the supported command line arguments. It would perhaps be more > > > difficult to integrate in the kernel build system though. This leads me > > > to a second question: would you consider merging extensions to this > > > script if they are not used by the kernel build system, but meant for > > > users who manually invoke the script ? More generally, is the script > > > > We'd also be interested in some customization, though in a different way. > > We imagine having a rule file that says X compatible string should map > > to A base DTB, plus B and C DTBO for the configuration section. The base > > DTB would carry all common elements of some device, while the DTBOs > > carry all the possible second source components, like different display > > panels or MIPI cameras for instance. This could drastically reduce the > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > Do you envision the "mapping" compatible string mapping to a config > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > meant to be used stand-alone as well, in which case its command line > > > arguments need to remain backward-compatible, or do you see it as being > > > internal to the kernel ? It is great to see all this discussion! I did send a proposal to the U-Boot ML about extensions but it was mixed up with other things, so I'll start a new thread. For now, I am really just waiting for this to be applied, before talking too much about future possibilities. Regards, SImon
On Thu, Dec 07, 2023 at 01:52:53PM -0700, Simon Glass wrote: > On Thu, 7 Dec 2023 at 07:38, Laurent Pinchart wrote: > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > Hi Simon, > > > > > > > > Thank you for the patch. > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > containing the built kernel and associated devicetree files. > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > performance. > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > devicetree, matching the compatible string of the running board with > > > > > the closest compatible string in the FIT. There is no need for > > > > > filenames or other workarounds. > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > requires compression utilities for the algorithm being used. Supported > > > > > compression options are the same as the Image.xxx files. For now there > > > > > is no way to change the compression other than by editing the rule for > > > > > $(obj)/image.fit > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > to the kernel build system. It can get tricky though: given the > > > > versatile nature of FIT images, there can't be any > > > > one-size-fits-them-all solution to build them, and striking the right > > > > balance between what makes sense for the kernel and the features that > > > > users may request will probably lead to bikeshedding. As we all love > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > to delay merging this series. > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > extending the supported command line arguments. It would perhaps be more > > > > difficult to integrate in the kernel build system though. This leads me > > > > to a second question: would you consider merging extensions to this > > > > script if they are not used by the kernel build system, but meant for > > > > users who manually invoke the script ? More generally, is the script > > > > > > We'd also be interested in some customization, though in a different way. > > > We imagine having a rule file that says X compatible string should map > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > DTB would carry all common elements of some device, while the DTBOs > > > carry all the possible second source components, like different display > > > panels or MIPI cameras for instance. This could drastically reduce the > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > Do you envision the "mapping" compatible string mapping to a config > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > meant to be used stand-alone as well, in which case its command line > > > > arguments need to remain backward-compatible, or do you see it as being > > > > internal to the kernel ? > > It is great to see all this discussion! I did send a proposal to the > U-Boot ML about extensions but it was mixed up with other things, so > I'll start a new thread. > > For now, I am really just waiting for this to be applied, before > talking too much about future possibilities. Sure, I don't see any reason to delay this series until you have fixed everybody's system images problems :-)
On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > Add a script which produces a Flat Image Tree (FIT), a single file > containing the built kernel and associated devicetree files. > Compression defaults to gzip which gives a good balance of size and > performance. > > The files compress from about 86MB to 24MB using this approach. > > The FIT can be used by bootloaders which support it, such as U-Boot > and Linuxboot. It permits automatic selection of the correct > devicetree, matching the compatible string of the running board with > the closest compatible string in the FIT. There is no need for > filenames or other workarounds. > > Add a 'make image.fit' build target for arm64, as well. Use > FIT_COMPRESSION to select a different algorithm. > > The FIT can be examined using 'dumpimage -l'. > > This features requires pylibfdt (use 'pip install libfdt'). It also > requires compression utilities for the algorithm being used. Supported > compression options are the same as the Image.xxx files. For now there > is no way to change the compression other than by editing the rule for > $(obj)/image.fit > > While FIT supports a ramdisk / initrd, no attempt is made to support > this here, since it must be built separately from the Linux build. > > Signed-off-by: Simon Glass <sjg@chromium.org> > --- for the kbuild parts: Reviewed-by: Nicolas Schier <n.schier@avm.de> > Changes in v9: > - Move the compression control into Makefile.lib > > Changes in v8: > - Drop compatible string in FDT node > - Correct sorting of MAINTAINERS to before ARM64 PORT > - Turn compress part of the make_fit.py comment in to a sentence > - Add two blank lines before parse_args() and setup_fit() > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > - Use '$(<D)/dts' instead of '$(dir $<)dts' > - Add 'mkimage' details Documentation/process/changes.rst > - Allow changing the compression used > - Tweak cover letter since there is only one clean-up patch > > Changes in v7: > - Add Image as a dependency of image.fit > - Drop kbuild tag > - Add dependency on dtbs > - Drop unnecessary path separator for dtbs > - Rebase to -next > > Changes in v5: > - Drop patch previously applied > - Correct compression rule which was broken in v4 > > Changes in v4: > - Use single quotes for UIMAGE_NAME > > Changes in v3: > - Drop temporary file image.itk > - Drop patch 'Use double quotes for image name' > - Drop double quotes in use of UIMAGE_NAME > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > - Avoid hard-coding "arm64" for the DT architecture > > Changes in v2: > - Drop patch previously applied > - Add .gitignore file > - Move fit rule to Makefile.lib using an intermediate file > - Drop dependency on CONFIG_EFI_ZBOOT > - Pick up .dtb files separately from the kernel > - Correct pylint too-many-args warning for write_kernel() > - Include the kernel image in the file count > - Add a pointer to the FIT spec and mention of its wide industry usage > - Mention the kernel version in the FIT description > > Documentation/process/changes.rst | 9 + > MAINTAINERS | 7 + > arch/arm64/Makefile | 7 +- > arch/arm64/boot/.gitignore | 1 + > arch/arm64/boot/Makefile | 6 +- > scripts/Makefile.lib | 16 ++ > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > 7 files changed, 334 insertions(+), 3 deletions(-) > create mode 100755 scripts/make_fit.py > > diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst > index bb96ca0f774b..cad51bd5bd62 100644 > --- a/Documentation/process/changes.rst > +++ b/Documentation/process/changes.rst > @@ -62,6 +62,7 @@ Sphinx\ [#f1]_ 1.7 sphinx-build --version > cpio any cpio --version > GNU tar 1.28 tar --version > gtags (optional) 6.6.5 gtags --version > +mkimage (optional) 2017.01 mkimage --version > ====================== =============== ======================================== > > .. [#f1] Sphinx is needed only to build the Kernel documentation > @@ -189,6 +190,14 @@ The kernel build requires GNU GLOBAL version 6.6.5 or later to generate > tag files through ``make gtags``. This is due to its use of the gtags > ``-C (--directory)`` flag. > > +mkimage > +------- > + > +This tool is used when building a Flat Image Tree (FIT), commonly used on ARM > +platforms. The tool is available via the ``u-boot-tools`` package or can be > +built from the U-Boot source code. See the instructions at > +https://docs.u-boot.org/en/latest/build/tools.html#building-tools-for-linux > + > System utilities > **************** > > diff --git a/MAINTAINERS b/MAINTAINERS > index 9d46229fe21b..d2d17f0d6e64 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -3037,6 +3037,13 @@ F: drivers/mmc/host/sdhci-of-arasan.c > N: zynq > N: xilinx > > +ARM64 FIT SUPPORT > +M: Simon Glass <sjg@chromium.org> > +L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) > +S: Maintained > +F: arch/arm64/boot/Makefile > +F: scripts/make_fit.py > + > ARM64 PORT (AARCH64 ARCHITECTURE) > M: Catalin Marinas <catalin.marinas@arm.com> > M: Will Deacon <will@kernel.org> > diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile > index 1bd4fae6e806..6b893dc454b7 100644 > --- a/arch/arm64/Makefile > +++ b/arch/arm64/Makefile > @@ -150,7 +150,7 @@ libs-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a > # Default target when executing plain make > boot := arch/arm64/boot > > -BOOT_TARGETS := Image vmlinuz.efi > +BOOT_TARGETS := Image vmlinuz.efi image.fit > > PHONY += $(BOOT_TARGETS) > > @@ -162,7 +162,9 @@ endif > > all: $(notdir $(KBUILD_IMAGE)) > > -vmlinuz.efi: Image > +image.fit: dtbs > + > +vmlinuz.efi image.fit: Image > $(BOOT_TARGETS): vmlinux > $(Q)$(MAKE) $(build)=$(boot) $(boot)/$@ > > @@ -215,6 +217,7 @@ virtconfig: > define archhelp > echo '* Image.gz - Compressed kernel image (arch/$(ARCH)/boot/Image.gz)' > echo ' Image - Uncompressed kernel image (arch/$(ARCH)/boot/Image)' > + echo ' image.fit - Flat Image Tree (arch/$(ARCH)/boot/image.fit)' > echo ' install - Install uncompressed kernel' > echo ' zinstall - Install compressed kernel' > echo ' Install using (your) ~/bin/installkernel or' > diff --git a/arch/arm64/boot/.gitignore b/arch/arm64/boot/.gitignore > index af5dc61f8b43..abaae9de1bdd 100644 > --- a/arch/arm64/boot/.gitignore > +++ b/arch/arm64/boot/.gitignore > @@ -2,3 +2,4 @@ > Image > Image.gz > vmlinuz* > +image.fit > diff --git a/arch/arm64/boot/Makefile b/arch/arm64/boot/Makefile > index 1761f5972443..b835c0880d1c 100644 > --- a/arch/arm64/boot/Makefile > +++ b/arch/arm64/boot/Makefile > @@ -16,7 +16,8 @@ > > OBJCOPYFLAGS_Image :=-O binary -R .note -R .note.gnu.build-id -R .comment -S > > -targets := Image Image.bz2 Image.gz Image.lz4 Image.lzma Image.lzo Image.zst > +targets := Image Image.bz2 Image.gz Image.lz4 Image.lzma Image.lzo \ > + Image.zst image.fit > > $(obj)/Image: vmlinux FORCE > $(call if_changed,objcopy) > @@ -39,6 +40,9 @@ $(obj)/Image.lzo: $(obj)/Image FORCE > $(obj)/Image.zst: $(obj)/Image FORCE > $(call if_changed,zstd) > > +$(obj)/image.fit: $(obj)/Image FORCE > + $(call cmd,fit) > + > EFI_ZBOOT_PAYLOAD := Image > EFI_ZBOOT_BFD_TARGET := elf64-littleaarch64 > EFI_ZBOOT_MACH_TYPE := ARM64 > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib > index 1a965fe68e01..1c60d594932c 100644 > --- a/scripts/Makefile.lib > +++ b/scripts/Makefile.lib > @@ -496,6 +496,22 @@ quiet_cmd_uimage = UIMAGE $@ > -a $(UIMAGE_LOADADDR) -e $(UIMAGE_ENTRYADDR) \ > -n '$(UIMAGE_NAME)' -d $< $@ > > +# Flat Image Tree (FIT) > +# This allows for packaging of a kernel and all devicetrees files, using > +# compression. > +# --------------------------------------------------------------------------- > + > +MAKE_FIT := $(srctree)/scripts/make_fit.py > + > +# Use this to override the compression algorithm > +FIT_COMPRESSION ?= gzip > + > +quiet_cmd_fit = FIT $@ > + cmd_fit = $(MAKE_FIT) -f $@ --arch $(UIMAGE_ARCH) --os linux \ > + --name '$(UIMAGE_NAME)' \ > + --compress $(FIT_COMPRESSION) -k $< \ > + $(<D)/dts > + > # XZ > # --------------------------------------------------------------------------- > # Use xzkern to compress the kernel image and xzmisc to compress other things. > diff --git a/scripts/make_fit.py b/scripts/make_fit.py > new file mode 100755 > index 000000000000..e616b0d7a84a > --- /dev/null > +++ b/scripts/make_fit.py > @@ -0,0 +1,291 @@ > +#!/usr/bin/env python3 > +# SPDX-License-Identifier: GPL-2.0+ > +# > +# Copyright 2023 Google LLC > +# Written by Simon Glass <sjg@chromium.org> > +# > + > +"""Build a FIT containing a lot of devicetree files > + > +Usage: > + make_fit.py -A arm64 -n 'Linux-6.6' -O linux > + -f arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk > + /tmp/kern/arch/arm64/boot/dts/ -E -c gzip > + > +Creates a FIT containing the supplied kernel and a directory containing the > +devicetree files. > + > +Use -E to generate an external FIT (where the data is placed after the > +FIT data structure). This allows parsing of the data without loading > +the entire FIT. > + > +Use -c to compress the data, using bzip2, gzip, lz4, lzma, lzo and > +zstd algorithms. > + > +The resulting FIT can be booted by bootloaders which support FIT, such > +as U-Boot, Linuxboot, Tianocore, etc. > + > +Note that this tool does not yet support adding a ramdisk / initrd. > +""" > + > +import argparse > +import collections > +import os > +import subprocess > +import sys > +import tempfile > +import time > + > +import libfdt > + > + > +# Tool extension and the name of the command-line tools > +CompTool = collections.namedtuple('CompTool', 'ext,tools') > + > +COMP_TOOLS = { > + 'bzip2': CompTool('.bz2', 'bzip2'), > + 'gzip': CompTool('.gz', 'pigz,gzip'), > + 'lz4': CompTool('.lz4', 'lz4'), > + 'lzma': CompTool('.lzma', 'lzma'), > + 'lzo': CompTool('.lzo', 'lzop'), > + 'zstd': CompTool('.zstd', 'zstd'), > +} > + > + > +def parse_args(): > + """Parse the program ArgumentParser > + > + Returns: > + Namespace object containing the arguments > + """ > + epilog = 'Build a FIT from a directory tree containing .dtb files' > + parser = argparse.ArgumentParser(epilog=epilog) > + parser.add_argument('-A', '--arch', type=str, required=True, > + help='Specifies the architecture') > + parser.add_argument('-c', '--compress', type=str, default='none', > + help='Specifies the compression') > + parser.add_argument('-E', '--external', action='store_true', > + help='Convert the FIT to use external data') > + parser.add_argument('-n', '--name', type=str, required=True, > + help='Specifies the name') > + parser.add_argument('-O', '--os', type=str, required=True, > + help='Specifies the operating system') > + parser.add_argument('-f', '--fit', type=str, required=True, > + help='Specifies the output file (.fit)') > + parser.add_argument('-k', '--kernel', type=str, required=True, > + help='Specifies the (uncompressed) kernel input file (.itk)') > + parser.add_argument('srcdir', type=str, nargs='*', > + help='Specifies the directory tree that contains .dtb files') > + > + return parser.parse_args() > + > + > +def setup_fit(fsw, name): > + """Make a start on writing the FIT > + > + Outputs the root properties and the 'images' node > + > + Args: > + fsw (libfdt.FdtSw): Object to use for writing > + name (str): Name of kernel image > + """ > + fsw.INC_SIZE = 65536 > + fsw.finish_reservemap() > + fsw.begin_node('') > + fsw.property_string('description', f'{name} with devicetree set') > + fsw.property_u32('#address-cells', 1) > + > + fsw.property_u32('timestamp', int(time.time())) > + fsw.begin_node('images') > + > + > +def write_kernel(fsw, data, args): > + """Write out the kernel image > + > + Writes a kernel node along with the required properties > + > + Args: > + fsw (libfdt.FdtSw): Object to use for writing > + data (bytes): Data to write (possibly compressed) > + args (Namespace): Contains necessary strings: > + arch: FIT architecture, e.g. 'arm64' > + fit_os: Operating Systems, e.g. 'linux' > + name: Name of OS, e.g. 'Linux-6.6.0-rc7' > + compress: Compression algorithm to use, e.g. 'gzip' > + """ > + with fsw.add_node('kernel'): > + fsw.property_string('description', args.name) > + fsw.property_string('type', 'kernel_noload') > + fsw.property_string('arch', args.arch) > + fsw.property_string('os', args.os) > + fsw.property_string('compression', args.compress) > + fsw.property('data', data) > + fsw.property_u32('load', 0) > + fsw.property_u32('entry', 0) > + > + > +def finish_fit(fsw, entries): > + """Finish the FIT ready for use > + > + Writes the /configurations node and subnodes > + > + Args: > + fsw (libfdt.FdtSw): Object to use for writing > + entries (list of tuple): List of configurations: > + str: Description of model > + str: Compatible stringlist > + """ > + fsw.end_node() > + seq = 0 > + with fsw.add_node('configurations'): > + for model, compat in entries: > + seq += 1 > + with fsw.add_node(f'conf-{seq}'): > + fsw.property('compatible', bytes(compat)) > + fsw.property_string('description', model) > + fsw.property_string('fdt', f'fdt-{seq}') > + fsw.property_string('kernel', 'kernel') > + fsw.end_node() > + > + > +def compress_data(inf, compress): > + """Compress data using a selected algorithm > + > + Args: > + inf (IOBase): Filename containing the data to compress > + compress (str): Compression algorithm, e.g. 'gzip' > + > + Return: > + bytes: Compressed data > + """ > + if compress == 'none': > + return inf.read() > + > + comp = COMP_TOOLS.get(compress) > + if not comp: > + raise ValueError(f"Unknown compression algorithm '{compress}'") > + > + with tempfile.NamedTemporaryFile() as comp_fname: > + with open(comp_fname.name, 'wb') as outf: > + done = False > + for tool in comp.tools.split(','): > + try: > + subprocess.call([tool, '-c'], stdin=inf, stdout=outf) > + done = True > + break > + except FileNotFoundError: > + pass > + if not done: > + raise ValueError(f'Missing tool(s): {comp.tools}\n') > + with open(comp_fname.name, 'rb') as compf: > + comp_data = compf.read() > + return comp_data > + > + > +def output_dtb(fsw, seq, fname, arch, compress): > + """Write out a single devicetree to the FIT > + > + Args: > + fsw (libfdt.FdtSw): Object to use for writing > + seq (int): Sequence number (1 for first) > + fmame (str): Filename containing the DTB > + arch: FIT architecture, e.g. 'arm64' > + compress (str): Compressed algorithm, e.g. 'gzip' > + > + Returns: > + tuple: > + str: Model name > + bytes: Compatible stringlist > + """ > + with fsw.add_node(f'fdt-{seq}'): > + # Get the compatible / model information > + with open(fname, 'rb') as inf: > + data = inf.read() > + fdt = libfdt.FdtRo(data) > + model = fdt.getprop(0, 'model').as_str() > + compat = fdt.getprop(0, 'compatible') > + > + fsw.property_string('description', model) > + fsw.property_string('type', 'flat_dt') > + fsw.property_string('arch', arch) > + fsw.property_string('compression', compress) > + fsw.property('compatible', bytes(compat)) > + > + with open(fname, 'rb') as inf: > + compressed = compress_data(inf, compress) > + fsw.property('data', compressed) > + return model, compat > + > + > +def build_fit(args): > + """Build the FIT from the provided files and arguments > + > + Args: > + args (Namespace): Program arguments > + > + Returns: > + tuple: > + bytes: FIT data > + int: Number of configurations generated > + size: Total uncompressed size of data > + """ > + fsw = libfdt.FdtSw() > + setup_fit(fsw, args.name) > + seq = 0 > + size = 0 > + entries = [] > + > + # Handle the kernel > + with open(args.kernel, 'rb') as inf: > + comp_data = compress_data(inf, args.compress) > + size += os.path.getsize(args.kernel) > + write_kernel(fsw, comp_data, args) > + > + for path in args.srcdir: > + # Handle devicetree files > + if os.path.isdir(path): > + for dirpath, _, fnames in os.walk(path): > + for fname in fnames: > + if os.path.splitext(fname)[1] != '.dtb': > + continue > + pathname = os.path.join(dirpath, fname) > + seq += 1 > + size += os.path.getsize(pathname) > + model, compat = output_dtb(fsw, seq, pathname, > + args.arch, args.compress) > + entries.append([model, compat]) > + > + finish_fit(fsw, entries) > + > + # Include the kernel itself in the returned file count > + return fsw.as_fdt().as_bytearray(), seq + 1, size > + > + > +def run_make_fit(): > + """Run the tool's main logic""" > + args = parse_args() > + > + out_data, count, size = build_fit(args) > + with open(args.fit, 'wb') as outf: > + outf.write(out_data) > + > + ext_fit_size = None > + if args.external: > + mkimage = os.environ.get('MKIMAGE', 'mkimage') > + subprocess.check_call([mkimage, '-E', '-F', args.fit], > + stdout=subprocess.DEVNULL) > + > + with open(args.fit, 'rb') as inf: > + data = inf.read() > + ext_fit = libfdt.FdtRo(data) > + ext_fit_size = ext_fit.totalsize() > + > + comp_size = len(out_data) > + print(f'FIT size {comp_size:#x}/{comp_size / 1024 / 1024:.1f} MB', end='') > + if ext_fit_size: > + print(f', header {ext_fit_size:#x}/{ext_fit_size / 1024:.1f} KB', end='') > + print(f', {count} files, uncompressed {size / 1024 / 1024:.1f} MB') > + > + > +if __name__ == "__main__": > + sys.exit(run_make_fit()) > -- > 2.43.0.rc2.451.g8631bc7472-goog >
On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote: > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > Hi Simon, > > > > > > Thank you for the patch. > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > containing the built kernel and associated devicetree files. > > > > Compression defaults to gzip which gives a good balance of size and > > > > performance. > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > and Linuxboot. It permits automatic selection of the correct > > > > devicetree, matching the compatible string of the running board with > > > > the closest compatible string in the FIT. There is no need for > > > > filenames or other workarounds. > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > requires compression utilities for the algorithm being used. Supported > > > > compression options are the same as the Image.xxx files. For now there > > > > is no way to change the compression other than by editing the rule for > > > > $(obj)/image.fit > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > this here, since it must be built separately from the Linux build. > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > to the kernel build system. It can get tricky though: given the > > > versatile nature of FIT images, there can't be any > > > one-size-fits-them-all solution to build them, and striking the right > > > balance between what makes sense for the kernel and the features that > > > users may request will probably lead to bikeshedding. As we all love > > > bikeshedding, I thought I would start selfishly, with a personal use > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > to delay merging this series. > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > extending the supported command line arguments. It would perhaps be more > > > difficult to integrate in the kernel build system though. This leads me > > > to a second question: would you consider merging extensions to this > > > script if they are not used by the kernel build system, but meant for > > > users who manually invoke the script ? More generally, is the script > > > > We'd also be interested in some customization, though in a different way. > > We imagine having a rule file that says X compatible string should map > > to A base DTB, plus B and C DTBO for the configuration section. The base > > DTB would carry all common elements of some device, while the DTBOs > > carry all the possible second source components, like different display > > panels or MIPI cameras for instance. This could drastically reduce the > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > Do you envision the "mapping" compatible string mapping to a config > section in the FIT image, that would bundle the base DTB and the DTBOs ? That's exactly the idea. The mapping compatible string could be untied from the base board's compatible string if needed (which we probably do). So something like: config { config-1 { compatible = "google,krane-sku0"; fdt = "krane-baseboard", "krane-sku0-overlay"; }; }; With "krane-sku0-overlay" being an overlay that holds the differences between the SKUs, in this case the display panel and MIPI camera (not upstreamed) that applies to SKU0 in particular. Sorry for not giving a more concrete idea. ChenYu > > > meant to be used stand-alone as well, in which case its command line > > > arguments need to remain backward-compatible, or do you see it as being > > > internal to the kernel ? > > > > [...] > > > > ChenYu > > -- > Regards, > > Laurent Pinchart
On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > <laurent.pinchart@ideasonboard.com> wrote: > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > Hi Simon, > > > > > > > > Thank you for the patch. > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > containing the built kernel and associated devicetree files. > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > performance. > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > devicetree, matching the compatible string of the running board with > > > > > the closest compatible string in the FIT. There is no need for > > > > > filenames or other workarounds. > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > requires compression utilities for the algorithm being used. Supported > > > > > compression options are the same as the Image.xxx files. For now there > > > > > is no way to change the compression other than by editing the rule for > > > > > $(obj)/image.fit > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > to the kernel build system. It can get tricky though: given the > > > > versatile nature of FIT images, there can't be any > > > > one-size-fits-them-all solution to build them, and striking the right > > > > balance between what makes sense for the kernel and the features that > > > > users may request will probably lead to bikeshedding. As we all love > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > to delay merging this series. > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > extending the supported command line arguments. It would perhaps be more > > > > difficult to integrate in the kernel build system though. This leads me > > > > to a second question: would you consider merging extensions to this > > > > script if they are not used by the kernel build system, but meant for > > > > users who manually invoke the script ? More generally, is the script > > > > > > We'd also be interested in some customization, though in a different way. > > > We imagine having a rule file that says X compatible string should map > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > DTB would carry all common elements of some device, while the DTBOs > > > carry all the possible second source components, like different display > > > panels or MIPI cameras for instance. This could drastically reduce the > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > Do you envision the "mapping" compatible string mapping to a config > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > That's exactly the idea. The mapping compatible string could be untied > from the base board's compatible string if needed (which we probably do). > > So something like: > > config { > config-1 { > compatible = "google,krane-sku0"; > fdt = "krane-baseboard", "krane-sku0-overlay"; > }; > }; > > With "krane-sku0-overlay" being an overlay that holds the differences > between the SKUs, in this case the display panel and MIPI camera (not > upstreamed) that applies to SKU0 in particular. The kernel DT makefiles already contain information on what overlays to apply to what base boards, in order to test the overlays and produce "full" DTBs. Maybe that information could be leveraged to create the configurations in the FIT image ? > Sorry for not giving a more concrete idea. > > > > > meant to be used stand-alone as well, in which case its command line > > > > arguments need to remain backward-compatible, or do you see it as being > > > > internal to the kernel ? > > > > > > [...]
Hi Laurent, On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote: > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > <laurent.pinchart@ideasonboard.com> wrote: > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > containing the built kernel and associated devicetree files. > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > performance. > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > devicetree, matching the compatible string of the running board with > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > filenames or other workarounds. > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > is no way to change the compression other than by editing the rule for > > > > > > $(obj)/image.fit > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > to the kernel build system. It can get tricky though: given the > > > > > versatile nature of FIT images, there can't be any > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > balance between what makes sense for the kernel and the features that > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > to delay merging this series. > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > extending the supported command line arguments. It would perhaps be more > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > to a second question: would you consider merging extensions to this > > > > > script if they are not used by the kernel build system, but meant for > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > We imagine having a rule file that says X compatible string should map > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > DTB would carry all common elements of some device, while the DTBOs > > > > carry all the possible second source components, like different display > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > That's exactly the idea. The mapping compatible string could be untied > > from the base board's compatible string if needed (which we probably do). > > > > So something like: > > > > config { > > config-1 { > > compatible = "google,krane-sku0"; > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > }; > > }; > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > between the SKUs, in this case the display panel and MIPI camera (not > > upstreamed) that applies to SKU0 in particular. > > The kernel DT makefiles already contain information on what overlays to > apply to what base boards, in order to test the overlays and produce > "full" DTBs. Maybe that information could be leveraged to create the > configurations in the FIT image ? Although the "full" DTBs created may only be a subset of all possible combinations (I believe Rob just started with creating one "full" DTB for each overlay, cfr. the additions I made in commit a09c3e105a208580 ("arm64: dts: renesas: Apply overlays to base dtbs")), that could definitely be a start. Now, since the kernel build system already creates "full" DTBs, does that mean that all of the base DTBs, overlays, and "full" DTBs will end up in the FIT image? Gr{oetje,eeting}s, Geert
On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > Add a script which produces a Flat Image Tree (FIT), a single file > containing the built kernel and associated devicetree files. > Compression defaults to gzip which gives a good balance of size and > performance. > > The files compress from about 86MB to 24MB using this approach. > > The FIT can be used by bootloaders which support it, such as U-Boot > and Linuxboot. It permits automatic selection of the correct > devicetree, matching the compatible string of the running board with > the closest compatible string in the FIT. There is no need for > filenames or other workarounds. > > Add a 'make image.fit' build target for arm64, as well. Use > FIT_COMPRESSION to select a different algorithm. > > The FIT can be examined using 'dumpimage -l'. > > This features requires pylibfdt (use 'pip install libfdt'). It also > requires compression utilities for the algorithm being used. Supported > compression options are the same as the Image.xxx files. For now there > is no way to change the compression other than by editing the rule for > $(obj)/image.fit > > While FIT supports a ramdisk / initrd, no attempt is made to support > this here, since it must be built separately from the Linux build. > > Signed-off-by: Simon Glass <sjg@chromium.org> > --- > > Changes in v9: > - Move the compression control into Makefile.lib > > Changes in v8: > - Drop compatible string in FDT node > - Correct sorting of MAINTAINERS to before ARM64 PORT > - Turn compress part of the make_fit.py comment in to a sentence > - Add two blank lines before parse_args() and setup_fit() > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > - Use '$(<D)/dts' instead of '$(dir $<)dts' > - Add 'mkimage' details Documentation/process/changes.rst > - Allow changing the compression used > - Tweak cover letter since there is only one clean-up patch > > Changes in v7: > - Add Image as a dependency of image.fit > - Drop kbuild tag > - Add dependency on dtbs > - Drop unnecessary path separator for dtbs > - Rebase to -next > > Changes in v5: > - Drop patch previously applied > - Correct compression rule which was broken in v4 > > Changes in v4: > - Use single quotes for UIMAGE_NAME > > Changes in v3: > - Drop temporary file image.itk > - Drop patch 'Use double quotes for image name' > - Drop double quotes in use of UIMAGE_NAME > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > - Avoid hard-coding "arm64" for the DT architecture > > Changes in v2: > - Drop patch previously applied > - Add .gitignore file > - Move fit rule to Makefile.lib using an intermediate file > - Drop dependency on CONFIG_EFI_ZBOOT > - Pick up .dtb files separately from the kernel > - Correct pylint too-many-args warning for write_kernel() > - Include the kernel image in the file count > - Add a pointer to the FIT spec and mention of its wide industry usage > - Mention the kernel version in the FIT description > > Documentation/process/changes.rst | 9 + > MAINTAINERS | 7 + > arch/arm64/Makefile | 7 +- > arch/arm64/boot/.gitignore | 1 + > arch/arm64/boot/Makefile | 6 +- > scripts/Makefile.lib | 16 ++ > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > 7 files changed, 334 insertions(+), 3 deletions(-) > create mode 100755 scripts/make_fit.py I'll need Masahiro's Ack on the scripts/ changes before I can take this one. Will
On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > Hi Laurent, > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > <laurent.pinchart@ideasonboard.com> wrote: > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > performance. > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > versatile nature of FIT images, there can't be any > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > balance between what makes sense for the kernel and the features that > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > to delay merging this series. > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > to a second question: would you consider merging extensions to this > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > We imagine having a rule file that says X compatible string should map > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > carry all the possible second source components, like different display > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > from the base board's compatible string if needed (which we probably do). > > > > > > So something like: > > > > > > config { > > > config-1 { > > > compatible = "google,krane-sku0"; > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > }; > > > }; > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > between the SKUs, in this case the display panel and MIPI camera (not > > > upstreamed) that applies to SKU0 in particular. > > > > The kernel DT makefiles already contain information on what overlays to > > apply to what base boards, in order to test the overlays and produce > > "full" DTBs. Maybe that information could be leveraged to create the > > configurations in the FIT image ? > > Although the "full" DTBs created may only be a subset of all possible > combinations (I believe Rob just started with creating one "full" DTB > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > definitely be a start. > > Now, since the kernel build system already creates "full" DTBs, does > that mean that all of the base DTBs, overlays, and "full" DTBs will > end up in the FIT image? I suppose we could add an option to the packing tool to be able to _not_ add the "full" DTBs if they can also be assembled with a base DTB and overlays. Think of it as a firmware compatibility option: if the firmware supports overlays, then you almost always want the deconstructed parts, not the fully assembled ones. Vice versa. If we don't we could end up with two configurations that have the same compatible string? ChenYu > Gr{oetje,eeting}s, > > Geert > > -- > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org > > In personal conversations with technical people, I call myself a hacker. But > when I'm talking to journalists I just say "programmer" or something like that. > -- Linus Torvalds
On Thu, Dec 14, 2023 at 1:03 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > Hi Laurent, > > > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > > <laurent.pinchart@ideasonboard.com> wrote: > > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > performance. > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > > versatile nature of FIT images, there can't be any > > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > > balance between what makes sense for the kernel and the features that > > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > > to delay merging this series. > > > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > > to a second question: would you consider merging extensions to this > > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > > We imagine having a rule file that says X compatible string should map > > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > > carry all the possible second source components, like different display > > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > > from the base board's compatible string if needed (which we probably do). > > > > > > > > So something like: > > > > > > > > config { > > > > config-1 { > > > > compatible = "google,krane-sku0"; > > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > > }; > > > > }; > > > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > > between the SKUs, in this case the display panel and MIPI camera (not > > > > upstreamed) that applies to SKU0 in particular. > > > > > > The kernel DT makefiles already contain information on what overlays to > > > apply to what base boards, in order to test the overlays and produce > > > "full" DTBs. Maybe that information could be leveraged to create the > > > configurations in the FIT image ? > > > > Although the "full" DTBs created may only be a subset of all possible > > combinations (I believe Rob just started with creating one "full" DTB > > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > > definitely be a start. > > > > Now, since the kernel build system already creates "full" DTBs, does > > that mean that all of the base DTBs, overlays, and "full" DTBs will > > end up in the FIT image? > > I suppose we could add an option to the packing tool to be able to _not_ > add the "full" DTBs if they can also be assembled with a base DTB and > overlays. Think of it as a firmware compatibility option: if the firmware > supports overlays, then you almost always want the deconstructed parts, > not the fully assembled ones. Vice versa. > > If we don't we could end up with two configurations that have the same > compatible string? Right. We would end up with such situations because applying an overlay does not change the compatible string. With this code in arch/arm64/boot/dts/ti/Makefile: k3-am642-tqma64xxl-mbax4xxl-sdcard-dtbs := \ k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-sdcard.dtbo k3-am642-tqma64xxl-mbax4xxl-wlan-dtbs := \ k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-wlan.dtbo $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-sdcard.dtb 2>/dev/null| head -n15 | tail -n2 model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", "ti,am642"; $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-wlan.dtb 2>/dev/null| head -n15 | tail -n2 model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", "ti,am642"; These two go into image.fit, but one of them is completely dead since there is no way to distinguish them. $ fdtdump arch/arm64/boot/image.fit ... conf-10 { compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", "ti,am642"; description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; fdt = "fdt-10"; kernel = "kernel"; }; ... conf-25 { compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", "ti,am642"; description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; fdt = "fdt-25"; kernel = "kernel"; }; I agree with Chen-Yu. FIT should not include full DTBs. Bootloaders should assemble the final DTB from base and overlays on-the-fly. The FIT spec allows the "fdt" property to list multiple image nodes. o config-1 |- description = "configuration description" |- kernel = "kernel sub-node unit name" |- fdt = "fdt sub-node unit-name" [, "fdt overlay sub-node unit-name", ...] |- loadables = "loadables sub-node unit-name" |- script = " |- compatible = "vendor > > ChenYu > > > > Gr{oetje,eeting}s, > > > > Geert > > > > -- > > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org > > > > In personal conversations with technical people, I call myself a hacker. But > > when I'm talking to journalists I just say "programmer" or something like that. > > -- Linus Torvalds -- Best Regards Masahiro Yamada
On Thu, Dec 14, 2023 at 3:12 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Thu, Dec 14, 2023 at 1:03 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > > > Hi Laurent, > > > > > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > > > versatile nature of FIT images, there can't be any > > > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > > > balance between what makes sense for the kernel and the features that > > > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > > > to delay merging this series. > > > > > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > > > to a second question: would you consider merging extensions to this > > > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > > > We imagine having a rule file that says X compatible string should map > > > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > > > carry all the possible second source components, like different display > > > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > > > from the base board's compatible string if needed (which we probably do). > > > > > > > > > > So something like: > > > > > > > > > > config { > > > > > config-1 { > > > > > compatible = "google,krane-sku0"; > > > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > > > }; > > > > > }; > > > > > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > > > between the SKUs, in this case the display panel and MIPI camera (not > > > > > upstreamed) that applies to SKU0 in particular. > > > > > > > > The kernel DT makefiles already contain information on what overlays to > > > > apply to what base boards, in order to test the overlays and produce > > > > "full" DTBs. Maybe that information could be leveraged to create the > > > > configurations in the FIT image ? > > > > > > Although the "full" DTBs created may only be a subset of all possible > > > combinations (I believe Rob just started with creating one "full" DTB > > > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > > > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > > > definitely be a start. > > > > > > Now, since the kernel build system already creates "full" DTBs, does > > > that mean that all of the base DTBs, overlays, and "full" DTBs will > > > end up in the FIT image? > > > > I suppose we could add an option to the packing tool to be able to _not_ > > add the "full" DTBs if they can also be assembled with a base DTB and > > overlays. Think of it as a firmware compatibility option: if the firmware > > supports overlays, then you almost always want the deconstructed parts, > > not the fully assembled ones. Vice versa. > > > > If we don't we could end up with two configurations that have the same > > compatible string? > > > Right. > > We would end up with such situations because applying > an overlay does not change the compatible string. > > > > With this code in arch/arm64/boot/dts/ti/Makefile: > > k3-am642-tqma64xxl-mbax4xxl-sdcard-dtbs := \ > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-sdcard.dtbo > k3-am642-tqma64xxl-mbax4xxl-wlan-dtbs := \ > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-wlan.dtbo > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-sdcard.dtb > 2>/dev/null| head -n15 | tail -n2 > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > "ti,am642"; > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-wlan.dtb > 2>/dev/null| head -n15 | tail -n2 > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > "ti,am642"; > > > > > > These two go into image.fit, but one of them is completely dead > since there is no way to distinguish them. > > > $ fdtdump arch/arm64/boot/image.fit > > ... > > conf-10 { > compatible = "tq,am642-tqma6442l-mbax4xxl", > "tq,am642-tqma6442l", "ti,am642"; > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > fdt = "fdt-10"; > kernel = "kernel"; > }; > > ... > > conf-25 { > compatible = "tq,am642-tqma6442l-mbax4xxl", > "tq,am642-tqma6442l", "ti,am642"; > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > fdt = "fdt-25"; > kernel = "kernel"; > }; > > > > > > I agree with Chen-Yu. > > FIT should not include full DTBs. > > Bootloaders should assemble the final DTB > from base and overlays on-the-fly. > > > The FIT spec allows the "fdt" property to list > multiple image nodes. > > > o config-1 > |- description = "configuration description" > |- kernel = "kernel sub-node unit name" > |- fdt = "fdt sub-node unit-name" [, "fdt overlay sub-node unit-name", ...] > |- loadables = "loadables sub-node unit-name" > |- script = " > |- compatible = "vendor This is a question for U-Boot (and barebox). images { base { ... }; addon1 { ... }; addon2 { ... }; }; configurations { ... fdt = "base", "addon1", "addon2"; }; Is U-Boot's "bootm" command able to dynamically construct the full DTB from "base" + "addon1" + "addon2" and pass to the kernel? When I used overlay from U-Boot command line last time, I typed complicated commands, following this manual: https://docs.u-boot.org/en/latest/usage/fdt_overlays.html One more question to confirm if I can use this for my practical use-cases. Is U-Boot able to handle FIT (includes kernel + DTs) and a separate initrd? # bootm <fit-address>:<conf-name> <ramdisk-address> Presumably, it would be difficult to inject initramdisk into image.fit later, so I am hoping bootm would work like that, but I did not delve into U-Boot code. If it works, is it possible to verify the integrity of initrd? The kernel and DTs inside FIT will be verified, but not sure if it is possible for ramdisk.
Hi Yamada-san, On Thu, Dec 14, 2023 at 7:12 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > On Thu, Dec 14, 2023 at 1:03 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > > > versatile nature of FIT images, there can't be any > > > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > > > balance between what makes sense for the kernel and the features that > > > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > > > to delay merging this series. > > > > > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > > > to a second question: would you consider merging extensions to this > > > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > > > We imagine having a rule file that says X compatible string should map > > > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > > > carry all the possible second source components, like different display > > > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > > > from the base board's compatible string if needed (which we probably do). > > > > > > > > > > So something like: > > > > > > > > > > config { > > > > > config-1 { > > > > > compatible = "google,krane-sku0"; > > > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > > > }; > > > > > }; > > > > > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > > > between the SKUs, in this case the display panel and MIPI camera (not > > > > > upstreamed) that applies to SKU0 in particular. > > > > > > > > The kernel DT makefiles already contain information on what overlays to > > > > apply to what base boards, in order to test the overlays and produce > > > > "full" DTBs. Maybe that information could be leveraged to create the > > > > configurations in the FIT image ? > > > > > > Although the "full" DTBs created may only be a subset of all possible > > > combinations (I believe Rob just started with creating one "full" DTB > > > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > > > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > > > definitely be a start. > > > > > > Now, since the kernel build system already creates "full" DTBs, does > > > that mean that all of the base DTBs, overlays, and "full" DTBs will > > > end up in the FIT image? > > > > I suppose we could add an option to the packing tool to be able to _not_ > > add the "full" DTBs if they can also be assembled with a base DTB and > > overlays. Think of it as a firmware compatibility option: if the firmware > > supports overlays, then you almost always want the deconstructed parts, > > not the fully assembled ones. Vice versa. > > > > If we don't we could end up with two configurations that have the same > > compatible string? > > Right. > > We would end up with such situations because applying > an overlay does not change the compatible string. That is correct. Which is one of the reasons for not using overlays for this, cfr. the details in my reply in the other thread "Re: Proposal: FIT support for extension boards / overlays" https://lore.kernel.org/all/CAMuHMdXQdMeXUOAAw5nDO4+q5_HFvUc86Wi8ykMwjUwPex6wvQ@mail.gmail.com/ Gr{oetje,eeting}s, Geert
On Thu, Dec 14, 2023 at 03:12:07PM +0900, Masahiro Yamada wrote: > On Thu, Dec 14, 2023 at 1:03 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > > > Hi Laurent, > > > > > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > > > versatile nature of FIT images, there can't be any > > > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > > > balance between what makes sense for the kernel and the features that > > > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > > > to delay merging this series. > > > > > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > > > to a second question: would you consider merging extensions to this > > > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > > > We imagine having a rule file that says X compatible string should map > > > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > > > carry all the possible second source components, like different display > > > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > > > from the base board's compatible string if needed (which we probably do). > > > > > > > > > > So something like: > > > > > > > > > > config { > > > > > config-1 { > > > > > compatible = "google,krane-sku0"; > > > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > > > }; > > > > > }; > > > > > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > > > between the SKUs, in this case the display panel and MIPI camera (not > > > > > upstreamed) that applies to SKU0 in particular. > > > > > > > > The kernel DT makefiles already contain information on what overlays to > > > > apply to what base boards, in order to test the overlays and produce > > > > "full" DTBs. Maybe that information could be leveraged to create the > > > > configurations in the FIT image ? > > > > > > Although the "full" DTBs created may only be a subset of all possible > > > combinations (I believe Rob just started with creating one "full" DTB > > > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > > > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > > > definitely be a start. > > > > > > Now, since the kernel build system already creates "full" DTBs, does > > > that mean that all of the base DTBs, overlays, and "full" DTBs will > > > end up in the FIT image? > > > > I suppose we could add an option to the packing tool to be able to _not_ > > add the "full" DTBs if they can also be assembled with a base DTB and > > overlays. Think of it as a firmware compatibility option: if the firmware > > supports overlays, then you almost always want the deconstructed parts, > > not the fully assembled ones. Vice versa. > > > > If we don't we could end up with two configurations that have the same > > compatible string? > > > Right. > > We would end up with such situations because applying > an overlay does not change the compatible string. > > > > With this code in arch/arm64/boot/dts/ti/Makefile: > > k3-am642-tqma64xxl-mbax4xxl-sdcard-dtbs := \ > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-sdcard.dtbo > k3-am642-tqma64xxl-mbax4xxl-wlan-dtbs := \ > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-wlan.dtbo > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-sdcard.dtb > 2>/dev/null| head -n15 | tail -n2 > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > "ti,am642"; > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-wlan.dtb > 2>/dev/null| head -n15 | tail -n2 > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > "ti,am642"; > > > > > > These two go into image.fit, but one of them is completely dead > since there is no way to distinguish them. I'd asked Rob about the problem of multiple platforms that don't have a unique compatible string, and never did quite conclude if the spec needs to spell out one way or another if they really need to be unique. There's some valid use cases where you might not, but then there's cases like the above where that really seems like a dts bug (and those platforms should have their dts* files reworked to be like other carrier+som+different peripherals and likely be tq,am642-tqma6442l-mbax4xxl-wlan and so on). > > > $ fdtdump arch/arm64/boot/image.fit > > ... > > conf-10 { > compatible = "tq,am642-tqma6442l-mbax4xxl", > "tq,am642-tqma6442l", "ti,am642"; > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > fdt = "fdt-10"; > kernel = "kernel"; > }; > > ... > > conf-25 { > compatible = "tq,am642-tqma6442l-mbax4xxl", > "tq,am642-tqma6442l", "ti,am642"; > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > fdt = "fdt-25"; > kernel = "kernel"; > }; > > > > > > I agree with Chen-Yu. > FIT should not include full DTBs. > > Bootloaders should assemble the final DTB > from base and overlays on-the-fly. I think we need to think about that more because both FIT and UKI try and solve the problem of a single verifiyable (in the various security / secure boot meanings of the word) image. So saying "but not DTBs" will I believe become "let the distribution people worry about it". Or am I misunderstanding your comment, and this is only in the context of real or quasi overlays?
Hi Masahiro, On Thu, Dec 14, 2023 at 7:34 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Thu, Dec 14, 2023 at 3:12 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Thu, Dec 14, 2023 at 1:03 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > > > > > Hi Laurent, > > > > > > > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > > > > versatile nature of FIT images, there can't be any > > > > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > > > > balance between what makes sense for the kernel and the features that > > > > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > > > > to delay merging this series. > > > > > > > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > > > > to a second question: would you consider merging extensions to this > > > > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > > > > We imagine having a rule file that says X compatible string should map > > > > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > > > > carry all the possible second source components, like different display > > > > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > > > > from the base board's compatible string if needed (which we probably do). > > > > > > > > > > > > So something like: > > > > > > > > > > > > config { > > > > > > config-1 { > > > > > > compatible = "google,krane-sku0"; > > > > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > > > > }; > > > > > > }; > > > > > > > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > > > > between the SKUs, in this case the display panel and MIPI camera (not > > > > > > upstreamed) that applies to SKU0 in particular. > > > > > > > > > > The kernel DT makefiles already contain information on what overlays to > > > > > apply to what base boards, in order to test the overlays and produce > > > > > "full" DTBs. Maybe that information could be leveraged to create the > > > > > configurations in the FIT image ? > > > > > > > > Although the "full" DTBs created may only be a subset of all possible > > > > combinations (I believe Rob just started with creating one "full" DTB > > > > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > > > > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > > > > definitely be a start. > > > > > > > > Now, since the kernel build system already creates "full" DTBs, does > > > > that mean that all of the base DTBs, overlays, and "full" DTBs will > > > > end up in the FIT image? > > > > > > I suppose we could add an option to the packing tool to be able to _not_ > > > add the "full" DTBs if they can also be assembled with a base DTB and > > > overlays. Think of it as a firmware compatibility option: if the firmware > > > supports overlays, then you almost always want the deconstructed parts, > > > not the fully assembled ones. Vice versa. > > > > > > If we don't we could end up with two configurations that have the same > > > compatible string? > > > > > > Right. > > > > We would end up with such situations because applying > > an overlay does not change the compatible string. > > > > > > > > With this code in arch/arm64/boot/dts/ti/Makefile: > > > > k3-am642-tqma64xxl-mbax4xxl-sdcard-dtbs := \ > > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-sdcard.dtbo > > k3-am642-tqma64xxl-mbax4xxl-wlan-dtbs := \ > > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-wlan.dtbo > > > > > > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-sdcard.dtb > > 2>/dev/null| head -n15 | tail -n2 > > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > > "ti,am642"; > > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-wlan.dtb > > 2>/dev/null| head -n15 | tail -n2 > > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > > "ti,am642"; > > > > > > > > > > > > These two go into image.fit, but one of them is completely dead > > since there is no way to distinguish them. > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > ... > > > > conf-10 { > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > "tq,am642-tqma6442l", "ti,am642"; > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > fdt = "fdt-10"; > > kernel = "kernel"; > > }; > > > > ... > > > > conf-25 { > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > "tq,am642-tqma6442l", "ti,am642"; > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > fdt = "fdt-25"; > > kernel = "kernel"; > > }; > > > > > > > > > > > > I agree with Chen-Yu. > > > > FIT should not include full DTBs. > > > > Bootloaders should assemble the final DTB > > from base and overlays on-the-fly. > > > > > > The FIT spec allows the "fdt" property to list > > multiple image nodes. > > > > > > o config-1 > > |- description = "configuration description" > > |- kernel = "kernel sub-node unit name" > > |- fdt = "fdt sub-node unit-name" [, "fdt overlay sub-node unit-name", ...] > > |- loadables = "loadables sub-node unit-name" > > |- script = " > > |- compatible = "vendor > > > > > > This is a question for U-Boot (and barebox). > > > > > images { > base { > ... > }; > > addon1 { > ... > }; > > addon2 { > ... > }; > }; > > configurations { > ... > fdt = "base", "addon1", "addon2"; > }; > > > > > Is U-Boot's "bootm" command able to dynamically construct > the full DTB from "base" + "addon1" + "addon2" > and pass to the kernel? > > > > When I used overlay from U-Boot command line last time, > I typed complicated commands, following this manual: > https://docs.u-boot.org/en/latest/usage/fdt_overlays.html > > So far this is not possible with bootm, no. But if we can add extensions to the FIT spec, then it should be possible to implement this. Is it (or will it be) possible to get Linux to build the DT + overlay combinations? > > > One more question to confirm if I can use this > for my practical use-cases. > > Is U-Boot able to handle FIT (includes kernel + DTs) > and a separate initrd? > > # bootm <fit-address>:<conf-name> <ramdisk-address> > > > Presumably, it would be difficult to inject initramdisk > into image.fit later, so I am hoping bootm would work like that, > but I did not delve into U-Boot code. > > The ramdisk is handled by the FIT configuration. I suppose it would be possible to add a way to bypass the logic in select_ramdisk(), but I wonder what is the use case for this? > > If it works, is it possible to verify the integrity of initrd? > The kernel and DTs inside FIT will be verified, but not sure > if it is possible for ramdisk. I do have thoughts about a possible new FIT feature to allow external files, which could perhaps include an initrd. Regards, Simon
On Fri, Dec 29, 2023 at 2:39 PM Simon Glass <sjg@chromium.org> wrote: > > Hi Masahiro, > > On Thu, Dec 14, 2023 at 7:34 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Thu, Dec 14, 2023 at 3:12 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > On Thu, Dec 14, 2023 at 1:03 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > > > On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > > > > > > > Hi Laurent, > > > > > > > > > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > > > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > > > > > versatile nature of FIT images, there can't be any > > > > > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > > > > > balance between what makes sense for the kernel and the features that > > > > > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > > > > > to delay merging this series. > > > > > > > > > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > > > > > to a second question: would you consider merging extensions to this > > > > > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > > > > > We imagine having a rule file that says X compatible string should map > > > > > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > > > > > carry all the possible second source components, like different display > > > > > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > > > > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > > > > > from the base board's compatible string if needed (which we probably do). > > > > > > > > > > > > > > So something like: > > > > > > > > > > > > > > config { > > > > > > > config-1 { > > > > > > > compatible = "google,krane-sku0"; > > > > > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > > > > > }; > > > > > > > }; > > > > > > > > > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > > > > > between the SKUs, in this case the display panel and MIPI camera (not > > > > > > > upstreamed) that applies to SKU0 in particular. > > > > > > > > > > > > The kernel DT makefiles already contain information on what overlays to > > > > > > apply to what base boards, in order to test the overlays and produce > > > > > > "full" DTBs. Maybe that information could be leveraged to create the > > > > > > configurations in the FIT image ? > > > > > > > > > > Although the "full" DTBs created may only be a subset of all possible > > > > > combinations (I believe Rob just started with creating one "full" DTB > > > > > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > > > > > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > > > > > definitely be a start. > > > > > > > > > > Now, since the kernel build system already creates "full" DTBs, does > > > > > that mean that all of the base DTBs, overlays, and "full" DTBs will > > > > > end up in the FIT image? > > > > > > > > I suppose we could add an option to the packing tool to be able to _not_ > > > > add the "full" DTBs if they can also be assembled with a base DTB and > > > > overlays. Think of it as a firmware compatibility option: if the firmware > > > > supports overlays, then you almost always want the deconstructed parts, > > > > not the fully assembled ones. Vice versa. > > > > > > > > If we don't we could end up with two configurations that have the same > > > > compatible string? > > > > > > > > > Right. > > > > > > We would end up with such situations because applying > > > an overlay does not change the compatible string. > > > > > > > > > > > > With this code in arch/arm64/boot/dts/ti/Makefile: > > > > > > k3-am642-tqma64xxl-mbax4xxl-sdcard-dtbs := \ > > > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-sdcard.dtbo > > > k3-am642-tqma64xxl-mbax4xxl-wlan-dtbs := \ > > > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-wlan.dtbo > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-sdcard.dtb > > > 2>/dev/null| head -n15 | tail -n2 > > > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > > > "ti,am642"; > > > > > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-wlan.dtb > > > 2>/dev/null| head -n15 | tail -n2 > > > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > > > "ti,am642"; > > > > > > > > > > > > > > > > > > These two go into image.fit, but one of them is completely dead > > > since there is no way to distinguish them. > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > ... > > > > > > conf-10 { > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > "tq,am642-tqma6442l", "ti,am642"; > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > fdt = "fdt-10"; > > > kernel = "kernel"; > > > }; > > > > > > ... > > > > > > conf-25 { > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > "tq,am642-tqma6442l", "ti,am642"; > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > fdt = "fdt-25"; > > > kernel = "kernel"; > > > }; > > > > > > > > > > > > > > > > > > I agree with Chen-Yu. > > > > > > FIT should not include full DTBs. > > > > > > Bootloaders should assemble the final DTB > > > from base and overlays on-the-fly. > > > > > > > > > The FIT spec allows the "fdt" property to list > > > multiple image nodes. > > > > > > > > > o config-1 > > > |- description = "configuration description" > > > |- kernel = "kernel sub-node unit name" > > > |- fdt = "fdt sub-node unit-name" [, "fdt overlay sub-node unit-name", ...] > > > |- loadables = "loadables sub-node unit-name" > > > |- script = " > > > |- compatible = "vendor > > > > > > > > > > > > This is a question for U-Boot (and barebox). > > > > > > > > > > images { > > base { > > ... > > }; > > > > addon1 { > > ... > > }; > > > > addon2 { > > ... > > }; > > }; > > > > configurations { > > ... > > fdt = "base", "addon1", "addon2"; > > }; > > > > > > > > > > Is U-Boot's "bootm" command able to dynamically construct > > the full DTB from "base" + "addon1" + "addon2" > > and pass to the kernel? > > > > > > > > When I used overlay from U-Boot command line last time, > > I typed complicated commands, following this manual: > > https://docs.u-boot.org/en/latest/usage/fdt_overlays.html > > > > > > So far this is not possible with bootm, no. But if we can add > extensions to the FIT spec, then it should be possible to implement > this. Isn't this already part of the FIT spec? There's nothing special here. It's one configuration that lists a base DTB plus some addon overlays. The FIT spec says: - fdt : Unit name of the corresponding fdt blob (component image node of a "fdt type"). Additional fdt overlay nodes can be supplied which signify that the resulting device tree blob is generated by the first base fdt blob with all subsequent overlays applied. Are you saying that U-boot currently lacks a mechanism to select the config? A quick skim over U-boot code suggests that boards need to implement board_fit_config_name_match()? > Is it (or will it be) possible to get Linux to build the DT + overlay > combinations? It is possible to build overlays separately, and have the build system apply them. Taking an example from the Renesas tree (line wrap mine): dtb-$(CONFIG_ARCH_R8A779G0) += r8a779g0-white-hawk.dtb dtb-$(CONFIG_ARCH_R8A779G0) += r8a779g0-white-hawk-ard-audio-da7212.dtbo r8a779g0-white-hawk-ard-audio-da7212-dtbs := r8a779g0-white-hawk.dtb \ r8a779g0-white-hawk-ard-audio-da7212.dtbo dtb-$(CONFIG_ARCH_R8A779G0) += r8a779g0-white-hawk-ard-audio-da7212.dtb The overlays are applied using the fdtoverlay command from the device tree compiler suite. ChenYu > > One more question to confirm if I can use this > > for my practical use-cases. > > > > Is U-Boot able to handle FIT (includes kernel + DTs) > > and a separate initrd? > > > > # bootm <fit-address>:<conf-name> <ramdisk-address> > > > > > > Presumably, it would be difficult to inject initramdisk > > into image.fit later, so I am hoping bootm would work like that, > > but I did not delve into U-Boot code. > > > > > > The ramdisk is handled by the FIT configuration. I suppose it would be > possible to add a way to bypass the logic in select_ramdisk(), but I > wonder what is the use case for this? > > > > > If it works, is it possible to verify the integrity of initrd? > > The kernel and DTs inside FIT will be verified, but not sure > > if it is possible for ramdisk. > > I do have thoughts about a possible new FIT feature to allow external > files, which could perhaps include an initrd. > > Regards, > Simon
Hello Yamada-san, On 14.12.23 08:33, Masahiro Yamada wrote: >> The FIT spec allows the "fdt" property to list >> multiple image nodes. >> >> >> o config-1 >> |- description = "configuration description" >> |- kernel = "kernel sub-node unit name" >> |- fdt = "fdt sub-node unit-name" [, "fdt overlay sub-node unit-name", ...] >> |- loadables = "loadables sub-node unit-name" >> |- script = " >> |- compatible = "vendor > > > > > > This is a question for U-Boot (and barebox). > > > > > images { > base { > ... > }; > > addon1 { > ... > }; > > addon2 { > ... > }; > }; > > configurations { > ... > fdt = "base", "addon1", "addon2"; > }; > > > > > Is U-Boot's "bootm" command able to dynamically construct > the full DTB from "base" + "addon1" + "addon2" > and pass to the kernel? barebox can apply overlays to the DT, but doesn't do so yet from the extra entries in configuration fdt properties. This should be straight-forward to add though, if the need arises. > Is U-Boot able to handle FIT (includes kernel + DTs) > and a separate initrd? > > # bootm <fit-address>:<conf-name> <ramdisk-address> This is possible in barebox, provided that the FIT image doesn't already have a ramdisk and that CONFIG_BOOTM_FORCE_SIGNED_IMAGES=n: bootm -r /mnt/nfs/ramdisk.gz /mnt/nfs/image.fit (Or the equivalent variables if not wanting to use the shell.) > Presumably, it would be difficult to inject initramdisk > into image.fit later, so I am hoping bootm would work like that, > but I did not delve into U-Boot code. > > > > If it works, is it possible to verify the integrity of initrd? > The kernel and DTs inside FIT will be verified, but not sure > if it is possible for ramdisk. If one wants to preclude mix & match attacks, the configuration needs to be verified fully, so if signing is required, it's probably better to amend the FIT later on with the new configuration instead of signing the initrd separately and combining them at runtime. Cheers, Ahmad > > > > >
Hi Masahiro, On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > Add a script which produces a Flat Image Tree (FIT), a single file > > containing the built kernel and associated devicetree files. > > Compression defaults to gzip which gives a good balance of size and > > performance. > > > > The files compress from about 86MB to 24MB using this approach. > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > and Linuxboot. It permits automatic selection of the correct > > devicetree, matching the compatible string of the running board with > > the closest compatible string in the FIT. There is no need for > > filenames or other workarounds. > > > > Add a 'make image.fit' build target for arm64, as well. Use > > FIT_COMPRESSION to select a different algorithm. > > > > The FIT can be examined using 'dumpimage -l'. > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > requires compression utilities for the algorithm being used. Supported > > compression options are the same as the Image.xxx files. For now there > > is no way to change the compression other than by editing the rule for > > $(obj)/image.fit > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > this here, since it must be built separately from the Linux build. > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > --- > > > > Changes in v9: > > - Move the compression control into Makefile.lib > > > > Changes in v8: > > - Drop compatible string in FDT node > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > - Turn compress part of the make_fit.py comment in to a sentence > > - Add two blank lines before parse_args() and setup_fit() > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > - Add 'mkimage' details Documentation/process/changes.rst > > - Allow changing the compression used > > - Tweak cover letter since there is only one clean-up patch > > > > Changes in v7: > > - Add Image as a dependency of image.fit > > - Drop kbuild tag > > - Add dependency on dtbs > > - Drop unnecessary path separator for dtbs > > - Rebase to -next > > > > Changes in v5: > > - Drop patch previously applied > > - Correct compression rule which was broken in v4 > > > > Changes in v4: > > - Use single quotes for UIMAGE_NAME > > > > Changes in v3: > > - Drop temporary file image.itk > > - Drop patch 'Use double quotes for image name' > > - Drop double quotes in use of UIMAGE_NAME > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > - Avoid hard-coding "arm64" for the DT architecture > > > > Changes in v2: > > - Drop patch previously applied > > - Add .gitignore file > > - Move fit rule to Makefile.lib using an intermediate file > > - Drop dependency on CONFIG_EFI_ZBOOT > > - Pick up .dtb files separately from the kernel > > - Correct pylint too-many-args warning for write_kernel() > > - Include the kernel image in the file count > > - Add a pointer to the FIT spec and mention of its wide industry usage > > - Mention the kernel version in the FIT description > > > > Documentation/process/changes.rst | 9 + > > MAINTAINERS | 7 + > > arch/arm64/Makefile | 7 +- > > arch/arm64/boot/.gitignore | 1 + > > arch/arm64/boot/Makefile | 6 +- > > scripts/Makefile.lib | 16 ++ > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > 7 files changed, 334 insertions(+), 3 deletions(-) > > create mode 100755 scripts/make_fit.py > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > one. Any thoughts on this request, please? Regards, Simon
On Fri, Dec 29, 2023 at 3:39 PM Simon Glass <sjg@chromium.org> wrote: > > Hi Masahiro, > > On Thu, Dec 14, 2023 at 7:34 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Thu, Dec 14, 2023 at 3:12 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > On Thu, Dec 14, 2023 at 1:03 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > > > On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > > > > > > > Hi Laurent, > > > > > > > > > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > > > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > > > > > versatile nature of FIT images, there can't be any > > > > > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > > > > > balance between what makes sense for the kernel and the features that > > > > > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > > > > > to delay merging this series. > > > > > > > > > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > > > > > to a second question: would you consider merging extensions to this > > > > > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > > > > > We imagine having a rule file that says X compatible string should map > > > > > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > > > > > carry all the possible second source components, like different display > > > > > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > > > > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > > > > > from the base board's compatible string if needed (which we probably do). > > > > > > > > > > > > > > So something like: > > > > > > > > > > > > > > config { > > > > > > > config-1 { > > > > > > > compatible = "google,krane-sku0"; > > > > > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > > > > > }; > > > > > > > }; > > > > > > > > > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > > > > > between the SKUs, in this case the display panel and MIPI camera (not > > > > > > > upstreamed) that applies to SKU0 in particular. > > > > > > > > > > > > The kernel DT makefiles already contain information on what overlays to > > > > > > apply to what base boards, in order to test the overlays and produce > > > > > > "full" DTBs. Maybe that information could be leveraged to create the > > > > > > configurations in the FIT image ? > > > > > > > > > > Although the "full" DTBs created may only be a subset of all possible > > > > > combinations (I believe Rob just started with creating one "full" DTB > > > > > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > > > > > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > > > > > definitely be a start. > > > > > > > > > > Now, since the kernel build system already creates "full" DTBs, does > > > > > that mean that all of the base DTBs, overlays, and "full" DTBs will > > > > > end up in the FIT image? > > > > > > > > I suppose we could add an option to the packing tool to be able to _not_ > > > > add the "full" DTBs if they can also be assembled with a base DTB and > > > > overlays. Think of it as a firmware compatibility option: if the firmware > > > > supports overlays, then you almost always want the deconstructed parts, > > > > not the fully assembled ones. Vice versa. > > > > > > > > If we don't we could end up with two configurations that have the same > > > > compatible string? > > > > > > > > > Right. > > > > > > We would end up with such situations because applying > > > an overlay does not change the compatible string. > > > > > > > > > > > > With this code in arch/arm64/boot/dts/ti/Makefile: > > > > > > k3-am642-tqma64xxl-mbax4xxl-sdcard-dtbs := \ > > > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-sdcard.dtbo > > > k3-am642-tqma64xxl-mbax4xxl-wlan-dtbs := \ > > > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-wlan.dtbo > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-sdcard.dtb > > > 2>/dev/null| head -n15 | tail -n2 > > > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > > > "ti,am642"; > > > > > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-wlan.dtb > > > 2>/dev/null| head -n15 | tail -n2 > > > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > > > "ti,am642"; > > > > > > > > > > > > > > > > > > These two go into image.fit, but one of them is completely dead > > > since there is no way to distinguish them. > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > ... > > > > > > conf-10 { > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > "tq,am642-tqma6442l", "ti,am642"; > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > fdt = "fdt-10"; > > > kernel = "kernel"; > > > }; > > > > > > ... > > > > > > conf-25 { > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > "tq,am642-tqma6442l", "ti,am642"; > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > fdt = "fdt-25"; > > > kernel = "kernel"; > > > }; > > > > > > > > > > > > > > > > > > I agree with Chen-Yu. > > > > > > FIT should not include full DTBs. > > > > > > Bootloaders should assemble the final DTB > > > from base and overlays on-the-fly. > > > > > > > > > The FIT spec allows the "fdt" property to list > > > multiple image nodes. > > > > > > > > > o config-1 > > > |- description = "configuration description" > > > |- kernel = "kernel sub-node unit name" > > > |- fdt = "fdt sub-node unit-name" [, "fdt overlay sub-node unit-name", ...] > > > |- loadables = "loadables sub-node unit-name" > > > |- script = " > > > |- compatible = "vendor > > > > > > > > > > > > This is a question for U-Boot (and barebox). > > > > > > > > > > images { > > base { > > ... > > }; > > > > addon1 { > > ... > > }; > > > > addon2 { > > ... > > }; > > }; > > > > configurations { > > ... > > fdt = "base", "addon1", "addon2"; > > }; > > > > > > > > > > Is U-Boot's "bootm" command able to dynamically construct > > the full DTB from "base" + "addon1" + "addon2" > > and pass to the kernel? > > > > > > > > When I used overlay from U-Boot command line last time, > > I typed complicated commands, following this manual: > > https://docs.u-boot.org/en/latest/usage/fdt_overlays.html > > > > > > So far this is not possible with bootm, no. But if we can add > extensions to the FIT spec, then it should be possible to implement > this. > > Is it (or will it be) possible to get Linux to build the DT + overlay > combinations? As Chen-Yu replied, dtb files specified in the -dtbs syntax in Makefiles are assembled at build time using fdtoverlay. Once they are built, you will never know how they are built unless you parse Makefiles. Your script simply picks up *.dtb files found under arch/*/boot/dts/. There are three possibilities in *.dtb files: [1] *.dtb assembled from other base and overlays [2] *.dtb directly compiled from a single *.dts [3] *.dtb meant to be used as a base of overlay, but not meant for direct use. It would be challenging to include only appropriate *.dtb (and *.dtbo) files into the FIT image. I wrote the following patch set, which might be useful to address this issue. https://lore.kernel.org/linux-kbuild/20240109120738.346061-1-masahiroy@kernel.org/T/#t > > > > > > > One more question to confirm if I can use this > > for my practical use-cases. > > > > Is U-Boot able to handle FIT (includes kernel + DTs) > > and a separate initrd? > > > > # bootm <fit-address>:<conf-name> <ramdisk-address> > > > > > > Presumably, it would be difficult to inject initramdisk > > into image.fit later, so I am hoping bootm would work like that, > > but I did not delve into U-Boot code. > > > > > > The ramdisk is handled by the FIT configuration. I suppose it would be > possible to add a way to bypass the logic in select_ramdisk(), but I > wonder what is the use case for this? I believe ramdisk is likely used to boot the arm64 Linux system. Since the FIT image generated by this lacks ramdisk, it looks useless to me unless there is a way to pass a ramdisk somehow. Barebox is good because it already supports FIT + separate ramdisk. I usually use U-Boot for my work, so I need to inject a ramdisk if I want to use this patch. > > > > If it works, is it possible to verify the integrity of initrd? > > The kernel and DTs inside FIT will be verified, but not sure > > if it is possible for ramdisk. > > I do have thoughts about a possible new FIT feature to allow external > files, which could perhaps include an initrd. > > Regards, > Simon >
Hi Simon, On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > Hi Masahiro, > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > containing the built kernel and associated devicetree files. > > > Compression defaults to gzip which gives a good balance of size and > > > performance. > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > and Linuxboot. It permits automatic selection of the correct > > > devicetree, matching the compatible string of the running board with > > > the closest compatible string in the FIT. There is no need for > > > filenames or other workarounds. > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > FIT_COMPRESSION to select a different algorithm. > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > requires compression utilities for the algorithm being used. Supported > > > compression options are the same as the Image.xxx files. For now there > > > is no way to change the compression other than by editing the rule for > > > $(obj)/image.fit > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > this here, since it must be built separately from the Linux build. > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > --- > > > > > > Changes in v9: > > > - Move the compression control into Makefile.lib > > > > > > Changes in v8: > > > - Drop compatible string in FDT node > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > - Turn compress part of the make_fit.py comment in to a sentence > > > - Add two blank lines before parse_args() and setup_fit() > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > - Add 'mkimage' details Documentation/process/changes.rst > > > - Allow changing the compression used > > > - Tweak cover letter since there is only one clean-up patch > > > > > > Changes in v7: > > > - Add Image as a dependency of image.fit > > > - Drop kbuild tag > > > - Add dependency on dtbs > > > - Drop unnecessary path separator for dtbs > > > - Rebase to -next > > > > > > Changes in v5: > > > - Drop patch previously applied > > > - Correct compression rule which was broken in v4 > > > > > > Changes in v4: > > > - Use single quotes for UIMAGE_NAME > > > > > > Changes in v3: > > > - Drop temporary file image.itk > > > - Drop patch 'Use double quotes for image name' > > > - Drop double quotes in use of UIMAGE_NAME > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > Changes in v2: > > > - Drop patch previously applied > > > - Add .gitignore file > > > - Move fit rule to Makefile.lib using an intermediate file > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > - Pick up .dtb files separately from the kernel > > > - Correct pylint too-many-args warning for write_kernel() > > > - Include the kernel image in the file count > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > - Mention the kernel version in the FIT description > > > > > > Documentation/process/changes.rst | 9 + > > > MAINTAINERS | 7 + > > > arch/arm64/Makefile | 7 +- > > > arch/arm64/boot/.gitignore | 1 + > > > arch/arm64/boot/Makefile | 6 +- > > > scripts/Makefile.lib | 16 ++ > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > create mode 100755 scripts/make_fit.py > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > one. > > Any thoughts on this request, please? > > Regards, > Simon > As I mentioned before, I am concerned with having the same "compatible" entries, with different contents, as you use the "compatible" string as an ID to selecting the target config node, right? $ fdtdump arch/arm64/boot/image.fit ... conf-10 { compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", "ti,am642"; description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; fdt = "fdt-10"; kernel = "kernel"; }; ... conf-25 { compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", "ti,am642"; description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; fdt = "fdt-25"; kernel = "kernel"; };
On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > Hi Simon, > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > Hi Masahiro, > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > containing the built kernel and associated devicetree files. > > > > Compression defaults to gzip which gives a good balance of size and > > > > performance. > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > and Linuxboot. It permits automatic selection of the correct > > > > devicetree, matching the compatible string of the running board with > > > > the closest compatible string in the FIT. There is no need for > > > > filenames or other workarounds. > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > requires compression utilities for the algorithm being used. Supported > > > > compression options are the same as the Image.xxx files. For now there > > > > is no way to change the compression other than by editing the rule for > > > > $(obj)/image.fit > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > this here, since it must be built separately from the Linux build. > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > --- > > > > > > > > Changes in v9: > > > > - Move the compression control into Makefile.lib > > > > > > > > Changes in v8: > > > > - Drop compatible string in FDT node > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > - Add two blank lines before parse_args() and setup_fit() > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > - Allow changing the compression used > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > Changes in v7: > > > > - Add Image as a dependency of image.fit > > > > - Drop kbuild tag > > > > - Add dependency on dtbs > > > > - Drop unnecessary path separator for dtbs > > > > - Rebase to -next > > > > > > > > Changes in v5: > > > > - Drop patch previously applied > > > > - Correct compression rule which was broken in v4 > > > > > > > > Changes in v4: > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > Changes in v3: > > > > - Drop temporary file image.itk > > > > - Drop patch 'Use double quotes for image name' > > > > - Drop double quotes in use of UIMAGE_NAME > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > Changes in v2: > > > > - Drop patch previously applied > > > > - Add .gitignore file > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > - Pick up .dtb files separately from the kernel > > > > - Correct pylint too-many-args warning for write_kernel() > > > > - Include the kernel image in the file count > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > - Mention the kernel version in the FIT description > > > > > > > > Documentation/process/changes.rst | 9 + > > > > MAINTAINERS | 7 + > > > > arch/arm64/Makefile | 7 +- > > > > arch/arm64/boot/.gitignore | 1 + > > > > arch/arm64/boot/Makefile | 6 +- > > > > scripts/Makefile.lib | 16 ++ > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > create mode 100755 scripts/make_fit.py > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > one. > > > > Any thoughts on this request, please? > > > > Regards, > > Simon > > > > > > As I mentioned before, I am concerned with having > the same "compatible" entries, with different contents, > as you use the "compatible" string as an ID to selecting > the target config node, right? > > > > > > $ fdtdump arch/arm64/boot/image.fit > > ... > > conf-10 { > compatible = "tq,am642-tqma6442l-mbax4xxl", > "tq,am642-tqma6442l", "ti,am642"; > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > fdt = "fdt-10"; > kernel = "kernel"; > }; > > ... > > conf-25 { > compatible = "tq,am642-tqma6442l-mbax4xxl", > "tq,am642-tqma6442l", "ti,am642"; > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > fdt = "fdt-25"; > kernel = "kernel"; > }; I had asked Rob a while ago about if having the same compatible for two functionally different machines is a feature, or a bug, and I don't think either of us fully agreed either way. I'd be leaning towards saying the above example is a bug in the dts files, it's just not been a bug people have worried about before due to (sadly) how little the top-level compatible has been used.
On 14/12/2023 08.33, Masahiro Yamada wrote: > On Thu, Dec 14, 2023 at 3:12 PM Masahiro Yamada <masahiroy@kernel.org> wrote: >> > One more question to confirm if I can use this > for my practical use-cases. > > Is U-Boot able to handle FIT (includes kernel + DTs) > and a separate initrd? > > # bootm <fit-address>:<conf-name> <ramdisk-address> > > > Presumably, it would be difficult to inject initramdisk > into image.fit later, so I am hoping bootm would work like that, > but I did not delve into U-Boot code. I recently had occasion to use this, and it actually already works out-of-the-box, but perhaps it could be better documented. Though you need not only the ramdisk address but also the size, as in <addr>:<size>, and of course CONFIG_SUPPORT_RAW_INITRD. My use case is bootstrapping: I have one FIT image (consisting of kernel, dtbs and an initramfs) which is the one that will be written to the target. But for bootstrapping, I (obviously) need to boot with a different initramfs that contains the bootstrap logic. Since this project uses fastboot, what I do is: upload the alternative initramfs, move it out of the way ('cause fastboot only supports one single target buffer), upload the FIT image, and "bootm $fitaddr $initrdaddr:$initrdsize". > If it works, is it possible to verify the integrity of initrd? No, I don't think so. In my case the FIT image is signed, and the kernel and chosen dtb does get verified, but not the contents of the initrd. I'm not sure how that should happen - in any case, in the fastboot case, the host can run arbitrary shell commands so not much U-Boot can do. Rasmus
On Tue, Jan 9, 2024 at 9:47 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Fri, Dec 29, 2023 at 3:39 PM Simon Glass <sjg@chromium.org> wrote: > > > > Hi Masahiro, > > > > On Thu, Dec 14, 2023 at 7:34 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > On Thu, Dec 14, 2023 at 3:12 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > > > On Thu, Dec 14, 2023 at 1:03 PM Chen-Yu Tsai <wenst@chromium.org> wrote: > > > > > > > > > > On Sun, Dec 10, 2023 at 1:31 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > > > > > > > > > Hi Laurent, > > > > > > > > > > > > On Sat, Dec 9, 2023 at 4:29 PM Laurent Pinchart > > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > > On Sat, Dec 09, 2023 at 10:13:59PM +0900, Chen-Yu Tsai wrote: > > > > > > > > On Thu, Dec 7, 2023 at 11:38 PM Laurent Pinchart > > > > > > > > <laurent.pinchart@ideasonboard.com> wrote: > > > > > > > > > On Thu, Dec 07, 2023 at 10:27:23PM +0800, Chen-Yu Tsai wrote: > > > > > > > > > > On Sun, Dec 03, 2023 at 05:34:01PM +0200, Laurent Pinchart wrote: > > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > > > > > > > FIT images are very useful, so I think this is a very welcome addition > > > > > > > > > > > to the kernel build system. It can get tricky though: given the > > > > > > > > > > > versatile nature of FIT images, there can't be any > > > > > > > > > > > one-size-fits-them-all solution to build them, and striking the right > > > > > > > > > > > balance between what makes sense for the kernel and the features that > > > > > > > > > > > users may request will probably lead to bikeshedding. As we all love > > > > > > > > > > > bikeshedding, I thought I would start selfishly, with a personal use > > > > > > > > > > > case :-) This isn't a yak-shaving request though, I don't see any reason > > > > > > > > > > > to delay merging this series. > > > > > > > > > > > > > > > > > > > > > > Have you envisioned building FIT images with a subset of DTBs, or adding > > > > > > > > > > > DTBOs ? Both would be fairly trivial extensions to this script by > > > > > > > > > > > extending the supported command line arguments. It would perhaps be more > > > > > > > > > > > difficult to integrate in the kernel build system though. This leads me > > > > > > > > > > > to a second question: would you consider merging extensions to this > > > > > > > > > > > script if they are not used by the kernel build system, but meant for > > > > > > > > > > > users who manually invoke the script ? More generally, is the script > > > > > > > > > > > > > > > > > > > > We'd also be interested in some customization, though in a different way. > > > > > > > > > > We imagine having a rule file that says X compatible string should map > > > > > > > > > > to A base DTB, plus B and C DTBO for the configuration section. The base > > > > > > > > > > DTB would carry all common elements of some device, while the DTBOs > > > > > > > > > > carry all the possible second source components, like different display > > > > > > > > > > panels or MIPI cameras for instance. This could drastically reduce the > > > > > > > > > > size of FIT images in ChromeOS by deduplicating all the common stuff. > > > > > > > > > > > > > > > > > > Do you envision the "mapping" compatible string mapping to a config > > > > > > > > > section in the FIT image, that would bundle the base DTB and the DTBOs ? > > > > > > > > > > > > > > > > That's exactly the idea. The mapping compatible string could be untied > > > > > > > > from the base board's compatible string if needed (which we probably do). > > > > > > > > > > > > > > > > So something like: > > > > > > > > > > > > > > > > config { > > > > > > > > config-1 { > > > > > > > > compatible = "google,krane-sku0"; > > > > > > > > fdt = "krane-baseboard", "krane-sku0-overlay"; > > > > > > > > }; > > > > > > > > }; > > > > > > > > > > > > > > > > With "krane-sku0-overlay" being an overlay that holds the differences > > > > > > > > between the SKUs, in this case the display panel and MIPI camera (not > > > > > > > > upstreamed) that applies to SKU0 in particular. > > > > > > > > > > > > > > The kernel DT makefiles already contain information on what overlays to > > > > > > > apply to what base boards, in order to test the overlays and produce > > > > > > > "full" DTBs. Maybe that information could be leveraged to create the > > > > > > > configurations in the FIT image ? > > > > > > > > > > > > Although the "full" DTBs created may only be a subset of all possible > > > > > > combinations (I believe Rob just started with creating one "full" DTB > > > > > > for each overlay, cfr. the additions I made in commit a09c3e105a208580 > > > > > > ("arm64: dts: renesas: Apply overlays to base dtbs")), that could > > > > > > definitely be a start. > > > > > > > > > > > > Now, since the kernel build system already creates "full" DTBs, does > > > > > > that mean that all of the base DTBs, overlays, and "full" DTBs will > > > > > > end up in the FIT image? > > > > > > > > > > I suppose we could add an option to the packing tool to be able to _not_ > > > > > add the "full" DTBs if they can also be assembled with a base DTB and > > > > > overlays. Think of it as a firmware compatibility option: if the firmware > > > > > supports overlays, then you almost always want the deconstructed parts, > > > > > not the fully assembled ones. Vice versa. > > > > > > > > > > If we don't we could end up with two configurations that have the same > > > > > compatible string? > > > > > > > > > > > > Right. > > > > > > > > We would end up with such situations because applying > > > > an overlay does not change the compatible string. > > > > > > > > > > > > > > > > With this code in arch/arm64/boot/dts/ti/Makefile: > > > > > > > > k3-am642-tqma64xxl-mbax4xxl-sdcard-dtbs := \ > > > > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-sdcard.dtbo > > > > k3-am642-tqma64xxl-mbax4xxl-wlan-dtbs := \ > > > > k3-am642-tqma64xxl-mbax4xxl.dtb k3-am64-tqma64xxl-mbax4xxl-wlan.dtbo > > > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-sdcard.dtb > > > > 2>/dev/null| head -n15 | tail -n2 > > > > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > > > > "ti,am642"; > > > > > > > > > > > > $ fdtdump arch/arm64/boot/dts/ti/k3-am642-tqma64xxl-mbax4xxl-wlan.dtb > > > > 2>/dev/null| head -n15 | tail -n2 > > > > model = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", "tq,am642-tqma6442l", > > > > "ti,am642"; > > > > > > > > > > > > > > > > > > > > > > > > These two go into image.fit, but one of them is completely dead > > > > since there is no way to distinguish them. > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > > > ... > > > > > > > > conf-10 { > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > fdt = "fdt-10"; > > > > kernel = "kernel"; > > > > }; > > > > > > > > ... > > > > > > > > conf-25 { > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > fdt = "fdt-25"; > > > > kernel = "kernel"; > > > > }; > > > > > > > > > > > > > > > > > > > > > > > > I agree with Chen-Yu. > > > > > > > > FIT should not include full DTBs. > > > > > > > > Bootloaders should assemble the final DTB > > > > from base and overlays on-the-fly. > > > > > > > > > > > > The FIT spec allows the "fdt" property to list > > > > multiple image nodes. > > > > > > > > > > > > o config-1 > > > > |- description = "configuration description" > > > > |- kernel = "kernel sub-node unit name" > > > > |- fdt = "fdt sub-node unit-name" [, "fdt overlay sub-node unit-name", ...] > > > > |- loadables = "loadables sub-node unit-name" > > > > |- script = " > > > > |- compatible = "vendor > > > > > > > > > > > > > > > > > > This is a question for U-Boot (and barebox). > > > > > > > > > > > > > > > images { > > > base { > > > ... > > > }; > > > > > > addon1 { > > > ... > > > }; > > > > > > addon2 { > > > ... > > > }; > > > }; > > > > > > configurations { > > > ... > > > fdt = "base", "addon1", "addon2"; > > > }; > > > > > > > > > > > > > > > Is U-Boot's "bootm" command able to dynamically construct > > > the full DTB from "base" + "addon1" + "addon2" > > > and pass to the kernel? > > > > > > > > > > > > When I used overlay from U-Boot command line last time, > > > I typed complicated commands, following this manual: > > > https://docs.u-boot.org/en/latest/usage/fdt_overlays.html > > > > > > > > > > So far this is not possible with bootm, no. But if we can add > > extensions to the FIT spec, then it should be possible to implement > > this. > > > > Is it (or will it be) possible to get Linux to build the DT + overlay > > combinations? > > > As Chen-Yu replied, dtb files specified in the -dtbs syntax in Makefiles > are assembled at build time using fdtoverlay. > > Once they are built, you will never know how they are built > unless you parse Makefiles. > > Your script simply picks up *.dtb files found under arch/*/boot/dts/. > There are three possibilities in *.dtb files: > > [1] *.dtb assembled from other base and overlays > [2] *.dtb directly compiled from a single *.dts > [3] *.dtb meant to be used as a base of overlay, > but not meant for direct use. > > > It would be challenging to include only appropriate *.dtb > (and *.dtbo) files into the FIT image. Irrespective of how and which DTB files are built, I think it would help if we could decouple the compatible string in the configuration node from the compatible string in the DTB. Many of the overlays currently in-tree are extensions of a given board, enabling a certain feature. Same could be said for all the out-of-tree Raspberry Pi overlays. Fundamentally it is still the same board, and the DTB's compatible shouldn't change. However the compatible string in the configuration node is only used by the firmware for selection purposes. It could be much more expansive including the extension features. We could have an extra config file that lists DTB files to exclude from the FIT image, and replacement compatible strings for the configuration node for certain DTB files. Using an in-tree example: instead of "gw,imx8mm-gw73xx-0x" mapping to imx8mm-venice-gw73xx-0x-*.dtb in addition to the base DT file imx8mm-venice-gw73xx-0x.dtb, each could be mapped to a different extension compatible, so "gw,imx8mm-gw73xx-0x-imx219" would map to imx8mm-venice-gw73xx-0x-imx219.dtb. How these configuration compatible strings should be defined is another matter... > I wrote the following patch set, which might be useful > to address this issue. > > https://lore.kernel.org/linux-kbuild/20240109120738.346061-1-masahiroy@kernel.org/T/#t I think this would help with selecting between the combined DTB vs base DTB plus DTBO overlays, and perhaps automatic exclusion of combined DTBs when board compatibles collide. But I do think device vendors and maintainers would want the extra flexibility I illustrated above. Regards ChenYu > > > One more question to confirm if I can use this > > > for my practical use-cases. > > > > > > Is U-Boot able to handle FIT (includes kernel + DTs) > > > and a separate initrd? > > > > > > # bootm <fit-address>:<conf-name> <ramdisk-address> > > > > > > > > > Presumably, it would be difficult to inject initramdisk > > > into image.fit later, so I am hoping bootm would work like that, > > > but I did not delve into U-Boot code. > > > > > > > > > > The ramdisk is handled by the FIT configuration. I suppose it would be > > possible to add a way to bypass the logic in select_ramdisk(), but I > > wonder what is the use case for this? > > > I believe ramdisk is likely used to boot the arm64 Linux system. > > Since the FIT image generated by this lacks ramdisk, > it looks useless to me unless there is a way to pass a ramdisk somehow. > > > Barebox is good because it already supports FIT + separate ramdisk. > > I usually use U-Boot for my work, so I need to inject a ramdisk > if I want to use this patch. > > > > > > > > > > If it works, is it possible to verify the integrity of initrd? > > > The kernel and DTs inside FIT will be verified, but not sure > > > if it is possible for ramdisk. > > > > I do have thoughts about a possible new FIT feature to allow external > > files, which could perhaps include an initrd. > > > > Regards, > > Simon > > > > > -- > Best Regards > Masahiro Yamada
Hi Masahiro, Tom, On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > Hi Simon, > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > Hi Masahiro, > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > containing the built kernel and associated devicetree files. > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > performance. > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > devicetree, matching the compatible string of the running board with > > > > > the closest compatible string in the FIT. There is no need for > > > > > filenames or other workarounds. > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > requires compression utilities for the algorithm being used. Supported > > > > > compression options are the same as the Image.xxx files. For now there > > > > > is no way to change the compression other than by editing the rule for > > > > > $(obj)/image.fit > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > --- > > > > > > > > > > Changes in v9: > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > Changes in v8: > > > > > - Drop compatible string in FDT node > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > - Allow changing the compression used > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > Changes in v7: > > > > > - Add Image as a dependency of image.fit > > > > > - Drop kbuild tag > > > > > - Add dependency on dtbs > > > > > - Drop unnecessary path separator for dtbs > > > > > - Rebase to -next > > > > > > > > > > Changes in v5: > > > > > - Drop patch previously applied > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > Changes in v4: > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > Changes in v3: > > > > > - Drop temporary file image.itk > > > > > - Drop patch 'Use double quotes for image name' > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > Changes in v2: > > > > > - Drop patch previously applied > > > > > - Add .gitignore file > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > - Pick up .dtb files separately from the kernel > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > - Include the kernel image in the file count > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > MAINTAINERS | 7 + > > > > > arch/arm64/Makefile | 7 +- > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > scripts/Makefile.lib | 16 ++ > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > one. > > > > > > Any thoughts on this request, please? > > > > > > Regards, > > > Simon > > > > > > > > > > > As I mentioned before, I am concerned with having > > the same "compatible" entries, with different contents, > > as you use the "compatible" string as an ID to selecting > > the target config node, right? > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > ... > > > > conf-10 { > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > "tq,am642-tqma6442l", "ti,am642"; > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > fdt = "fdt-10"; > > kernel = "kernel"; > > }; > > > > ... > > > > conf-25 { > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > "tq,am642-tqma6442l", "ti,am642"; > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > fdt = "fdt-25"; > > kernel = "kernel"; > > }; > > I had asked Rob a while ago about if having the same compatible for two > functionally different machines is a feature, or a bug, and I don't > think either of us fully agreed either way. I'd be leaning towards > saying the above example is a bug in the dts files, it's just not been a > bug people have worried about before due to (sadly) how little the > top-level compatible has been used. Yes I believe this is a bug in the files. What should the script do in this case? Print a warning, perhaps? Regards, Simon
Hi, On Wed, 17 Jan 2024 at 06:14, Simon Glass <sjg@chromium.org> wrote: > > Hi Masahiro, Tom, > > On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > > Hi Simon, > > > > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > Hi Masahiro, > > > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > containing the built kernel and associated devicetree files. > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > performance. > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > devicetree, matching the compatible string of the running board with > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > filenames or other workarounds. > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > is no way to change the compression other than by editing the rule for > > > > > > $(obj)/image.fit > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > --- > > > > > > > > > > > > Changes in v9: > > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > > > Changes in v8: > > > > > > - Drop compatible string in FDT node > > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > > - Allow changing the compression used > > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > > > Changes in v7: > > > > > > - Add Image as a dependency of image.fit > > > > > > - Drop kbuild tag > > > > > > - Add dependency on dtbs > > > > > > - Drop unnecessary path separator for dtbs > > > > > > - Rebase to -next > > > > > > > > > > > > Changes in v5: > > > > > > - Drop patch previously applied > > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > > > Changes in v4: > > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > > > Changes in v3: > > > > > > - Drop temporary file image.itk > > > > > > - Drop patch 'Use double quotes for image name' > > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > > > Changes in v2: > > > > > > - Drop patch previously applied > > > > > > - Add .gitignore file > > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > > - Pick up .dtb files separately from the kernel > > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > > - Include the kernel image in the file count > > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > > MAINTAINERS | 7 + > > > > > > arch/arm64/Makefile | 7 +- > > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > > scripts/Makefile.lib | 16 ++ > > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > > one. > > > > > > > > Any thoughts on this request, please? > > > > > > > > Regards, > > > > Simon > > > > > > > > > > > > > > > > As I mentioned before, I am concerned with having > > > the same "compatible" entries, with different contents, > > > as you use the "compatible" string as an ID to selecting > > > the target config node, right? > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > ... > > > > > > conf-10 { > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > "tq,am642-tqma6442l", "ti,am642"; > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > fdt = "fdt-10"; > > > kernel = "kernel"; > > > }; > > > > > > ... > > > > > > conf-25 { > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > "tq,am642-tqma6442l", "ti,am642"; > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > fdt = "fdt-25"; > > > kernel = "kernel"; > > > }; > > > > I had asked Rob a while ago about if having the same compatible for two > > functionally different machines is a feature, or a bug, and I don't > > think either of us fully agreed either way. I'd be leaning towards > > saying the above example is a bug in the dts files, it's just not been a > > bug people have worried about before due to (sadly) how little the > > top-level compatible has been used. > > Yes I believe this is a bug in the files. > > What should the script do in this case? Print a warning, perhaps? Is there anything I should do here? Would a warning be helpful, or just confusing? Regards, Simon
On Fri, Jan 26, 2024 at 1:04 AM Simon Glass <sjg@chromium.org> wrote: > > Hi, > > On Wed, 17 Jan 2024 at 06:14, Simon Glass <sjg@chromium.org> wrote: > > > > Hi Masahiro, Tom, > > > > On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > > > > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > > > Hi Simon, > > > > > > > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > Hi Masahiro, > > > > > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > performance. > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > > --- > > > > > > > > > > > > > > Changes in v9: > > > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > > > > > Changes in v8: > > > > > > > - Drop compatible string in FDT node > > > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > > > - Allow changing the compression used > > > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > > > > > Changes in v7: > > > > > > > - Add Image as a dependency of image.fit > > > > > > > - Drop kbuild tag > > > > > > > - Add dependency on dtbs > > > > > > > - Drop unnecessary path separator for dtbs > > > > > > > - Rebase to -next > > > > > > > > > > > > > > Changes in v5: > > > > > > > - Drop patch previously applied > > > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > > > > > Changes in v4: > > > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > > > > > Changes in v3: > > > > > > > - Drop temporary file image.itk > > > > > > > - Drop patch 'Use double quotes for image name' > > > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > > > > > Changes in v2: > > > > > > > - Drop patch previously applied > > > > > > > - Add .gitignore file > > > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > > > - Pick up .dtb files separately from the kernel > > > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > > > - Include the kernel image in the file count > > > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > > > MAINTAINERS | 7 + > > > > > > > arch/arm64/Makefile | 7 +- > > > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > > > scripts/Makefile.lib | 16 ++ > > > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > > > one. > > > > > > > > > > Any thoughts on this request, please? > > > > > > > > > > Regards, > > > > > Simon > > > > > > > > > > > > > > > > > > > > > As I mentioned before, I am concerned with having > > > > the same "compatible" entries, with different contents, > > > > as you use the "compatible" string as an ID to selecting > > > > the target config node, right? > > > > > > > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > > > ... > > > > > > > > conf-10 { > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > fdt = "fdt-10"; > > > > kernel = "kernel"; > > > > }; > > > > > > > > ... > > > > > > > > conf-25 { > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > fdt = "fdt-25"; > > > > kernel = "kernel"; > > > > }; > > > > > > I had asked Rob a while ago about if having the same compatible for two > > > functionally different machines is a feature, or a bug, and I don't > > > think either of us fully agreed either way. I'd be leaning towards > > > saying the above example is a bug in the dts files, it's just not been a > > > bug people have worried about before due to (sadly) how little the > > > top-level compatible has been used. > > > > Yes I believe this is a bug in the files. > > > > What should the script do in this case? Print a warning, perhaps? > > Is there anything I should do here? Would a warning be helpful, or > just confusing? I do not think it is useful. You would almost always get a warning, and there is no way to fix it. With arm64 defconfig, image.fit will include a thousand DTBs. The config node of my board was listed 214th. (I found it by fdtdump) Then, I learned > bootm ${loadaddr}#conf-214 is the correct command to boot my board. Of course, the "214" will be different in the future. The node names, conf-*, are useless. Only the useful way is to enable CONFIG_FIT_BEST_MATCH in U-Boot, but this relies on the uniqueness of a compatible string, which is not true. (I do not know how to do it in barebox) I think using the file name as a config node mitigates the issue because a file name is considered unique. For example, with composite DTBs: imx8mm-venice-gw72xx-0x-imx219-dtbs := imx8mm-venice-gw72xx-0x.dtb imx8mm-venice-gw72xx-0x-imx219.dtbo imx8mm-venice-gw72xx-0x-rpidsi-dtbs := imx8mm-venice-gw72xx-0x.dtb imx8mm-venice-gw72xx-0x-rpidsi.dtbo configurations { imx8mm-venice-gw72xx-0x-imx219 { ... }; imx8mm-venice-gw72xx-0x-rpidsi { ... } }; Then, we can distinguish them by node, even if they have the same compatible string. At least we can do > bootm ${loadaddr}#imx8mm-venice-gw72xx-0x-imx219 For the issue including stale DTBs, you can use my patch: [PATCH 1/4] kbuild: create a list of all built DTB files (https://lore.kernel.org/linux-kbuild/CAK7LNASOxi-gzve+_d-sCW9z_eEJ5TMMnzPEvN2Nj2AwgVjF9g@mail.gmail.com/T/#ma3595627a96a04554a78cbbd22056831e13db260) You can change scripts/make_fit.py to take the DTB files instead of the directory to search. Optionally, you can support '@' syntax to take command arguments from a file. scripts/make_fit.py ... @arch/$(SRCARCH)/boot/dts/dtbs-list For the separate base and overlays support, you can use my patch as a base: [PATCH 3/4] kbuild: create a list of base and overlays for each DTB (https://lore.kernel.org/linux-kbuild/CAK7LNASOxi-gzve+_d-sCW9z_eEJ5TMMnzPEvN2Nj2AwgVjF9g@mail.gmail.com/T/#m32c5bdde9098901b7c7776b932827493a05c82d5) Lastly, you do not need to require mkimage for args.external. You can simply concatenate files. -- Best Regards Masahiro Yamada
On Tue, Jan 30, 2024 at 3:16 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Fri, Jan 26, 2024 at 1:04 AM Simon Glass <sjg@chromium.org> wrote: > > > > Hi, > > > > On Wed, 17 Jan 2024 at 06:14, Simon Glass <sjg@chromium.org> wrote: > > > > > > Hi Masahiro, Tom, > > > > > > On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > > > > > > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > > > > Hi Simon, > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > > > Hi Masahiro, > > > > > > > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > performance. > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > > > --- > > > > > > > > > > > > > > > > Changes in v9: > > > > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > > > > > > > Changes in v8: > > > > > > > > - Drop compatible string in FDT node > > > > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > > > > - Allow changing the compression used > > > > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > > > > > > > Changes in v7: > > > > > > > > - Add Image as a dependency of image.fit > > > > > > > > - Drop kbuild tag > > > > > > > > - Add dependency on dtbs > > > > > > > > - Drop unnecessary path separator for dtbs > > > > > > > > - Rebase to -next > > > > > > > > > > > > > > > > Changes in v5: > > > > > > > > - Drop patch previously applied > > > > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > > > > > > > Changes in v4: > > > > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > > > > > > > Changes in v3: > > > > > > > > - Drop temporary file image.itk > > > > > > > > - Drop patch 'Use double quotes for image name' > > > > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > > > > > > > Changes in v2: > > > > > > > > - Drop patch previously applied > > > > > > > > - Add .gitignore file > > > > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > > > > - Pick up .dtb files separately from the kernel > > > > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > > > > - Include the kernel image in the file count > > > > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > > > > MAINTAINERS | 7 + > > > > > > > > arch/arm64/Makefile | 7 +- > > > > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > > > > scripts/Makefile.lib | 16 ++ > > > > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > > > > one. > > > > > > > > > > > > Any thoughts on this request, please? > > > > > > > > > > > > Regards, > > > > > > Simon > > > > > > > > > > > > > > > > > > > > > > > > > > As I mentioned before, I am concerned with having > > > > > the same "compatible" entries, with different contents, > > > > > as you use the "compatible" string as an ID to selecting > > > > > the target config node, right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > > > > > ... > > > > > > > > > > conf-10 { > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > fdt = "fdt-10"; > > > > > kernel = "kernel"; > > > > > }; > > > > > > > > > > ... > > > > > > > > > > conf-25 { > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > fdt = "fdt-25"; > > > > > kernel = "kernel"; > > > > > }; > > > > > > > > I had asked Rob a while ago about if having the same compatible for two > > > > functionally different machines is a feature, or a bug, and I don't > > > > think either of us fully agreed either way. I'd be leaning towards > > > > saying the above example is a bug in the dts files, it's just not been a > > > > bug people have worried about before due to (sadly) how little the > > > > top-level compatible has been used. I much prefer being able to use compatibles over filenames. > > > > > > Yes I believe this is a bug in the files. > > > > > > What should the script do in this case? Print a warning, perhaps? > > > > Is there anything I should do here? Would a warning be helpful, or > > just confusing? > > > > I do not think it is useful. > You would almost always get a warning, and there is no way to fix it. The above case is due to overlays. Why would you have a FIT image with both a base tree and applied overlays? In any case, maybe we need to record in dtb overlays that have been applied (which you asked about recently on dtc list). Not sure what that looks like though. Overlays have a 'top-level' compatible that we add in either separately or merged with the base's top-level compatible? Rob
On Thu, Feb 1, 2024 at 7:03 AM Rob Herring <robh@kernel.org> wrote: > > On Tue, Jan 30, 2024 at 3:16 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Fri, Jan 26, 2024 at 1:04 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > Hi, > > > > > > On Wed, 17 Jan 2024 at 06:14, Simon Glass <sjg@chromium.org> wrote: > > > > > > > > Hi Masahiro, Tom, > > > > > > > > On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > > > > > > > > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > > > > > Hi Simon, > > > > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > > > > > Hi Masahiro, > > > > > > > > > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > > > > --- > > > > > > > > > > > > > > > > > > Changes in v9: > > > > > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > > > > > > > > > Changes in v8: > > > > > > > > > - Drop compatible string in FDT node > > > > > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > > > > > - Allow changing the compression used > > > > > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > > > > > > > > > Changes in v7: > > > > > > > > > - Add Image as a dependency of image.fit > > > > > > > > > - Drop kbuild tag > > > > > > > > > - Add dependency on dtbs > > > > > > > > > - Drop unnecessary path separator for dtbs > > > > > > > > > - Rebase to -next > > > > > > > > > > > > > > > > > > Changes in v5: > > > > > > > > > - Drop patch previously applied > > > > > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > > > > > > > > > Changes in v4: > > > > > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > > > > > > > > > Changes in v3: > > > > > > > > > - Drop temporary file image.itk > > > > > > > > > - Drop patch 'Use double quotes for image name' > > > > > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > > > > > > > > > Changes in v2: > > > > > > > > > - Drop patch previously applied > > > > > > > > > - Add .gitignore file > > > > > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > > > > > - Pick up .dtb files separately from the kernel > > > > > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > > > > > - Include the kernel image in the file count > > > > > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > > > > > MAINTAINERS | 7 + > > > > > > > > > arch/arm64/Makefile | 7 +- > > > > > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > > > > > scripts/Makefile.lib | 16 ++ > > > > > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > > > > > one. > > > > > > > > > > > > > > Any thoughts on this request, please? > > > > > > > > > > > > > > Regards, > > > > > > > Simon > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I mentioned before, I am concerned with having > > > > > > the same "compatible" entries, with different contents, > > > > > > as you use the "compatible" string as an ID to selecting > > > > > > the target config node, right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > > > > > > > ... > > > > > > > > > > > > conf-10 { > > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > > fdt = "fdt-10"; > > > > > > kernel = "kernel"; > > > > > > }; > > > > > > > > > > > > ... > > > > > > > > > > > > conf-25 { > > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > > fdt = "fdt-25"; > > > > > > kernel = "kernel"; > > > > > > }; > > > > > > > > > > I had asked Rob a while ago about if having the same compatible for two > > > > > functionally different machines is a feature, or a bug, and I don't > > > > > think either of us fully agreed either way. I'd be leaning towards > > > > > saying the above example is a bug in the dts files, it's just not been a > > > > > bug people have worried about before due to (sadly) how little the > > > > > top-level compatible has been used. > > I much prefer being able to use compatibles over filenames. > > > > > > > > > Yes I believe this is a bug in the files. > > > > > > > > What should the script do in this case? Print a warning, perhaps? > > > > > > Is there anything I should do here? Would a warning be helpful, or > > > just confusing? > > > > > > > > I do not think it is useful. > > You would almost always get a warning, and there is no way to fix it. > > The above case is due to overlays. Why would you have a FIT image with > both a base tree and applied overlays? Because they are different hardware. If FIT includes only base DTBs, how to use a base with extensions? > > In any case, maybe we need to record in dtb overlays that have been > applied (which you asked about recently on dtc list). Not sure what > that looks like though. Overlays have a 'top-level' compatible that we > add in either separately or merged with the base's top-level > compatible? If there is a way to make "compatible" unique, that will be good. But, in my understanding, we can replace a property value, but not modify it.
On Wed, Jan 31, 2024 at 8:09 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Thu, Feb 1, 2024 at 7:03 AM Rob Herring <robh@kernel.org> wrote: > > > > On Tue, Jan 30, 2024 at 3:16 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > On Fri, Jan 26, 2024 at 1:04 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > Hi, > > > > > > > > On Wed, 17 Jan 2024 at 06:14, Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > Hi Masahiro, Tom, > > > > > > > > > > On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > > > > > > > > > > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > > > > > > Hi Simon, > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > > > > > > > Hi Masahiro, > > > > > > > > > > > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > > > > > --- > > > > > > > > > > > > > > > > > > > > Changes in v9: > > > > > > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > > > > > > > > > > > Changes in v8: > > > > > > > > > > - Drop compatible string in FDT node > > > > > > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > > > > > > - Allow changing the compression used > > > > > > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > > > > > > > > > > > Changes in v7: > > > > > > > > > > - Add Image as a dependency of image.fit > > > > > > > > > > - Drop kbuild tag > > > > > > > > > > - Add dependency on dtbs > > > > > > > > > > - Drop unnecessary path separator for dtbs > > > > > > > > > > - Rebase to -next > > > > > > > > > > > > > > > > > > > > Changes in v5: > > > > > > > > > > - Drop patch previously applied > > > > > > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > > > > > > > > > > > Changes in v4: > > > > > > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > > > > > > > > > > > Changes in v3: > > > > > > > > > > - Drop temporary file image.itk > > > > > > > > > > - Drop patch 'Use double quotes for image name' > > > > > > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > > > > > > > > > > > Changes in v2: > > > > > > > > > > - Drop patch previously applied > > > > > > > > > > - Add .gitignore file > > > > > > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > > > > > > - Pick up .dtb files separately from the kernel > > > > > > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > > > > > > - Include the kernel image in the file count > > > > > > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > > > > > > MAINTAINERS | 7 + > > > > > > > > > > arch/arm64/Makefile | 7 +- > > > > > > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > > > > > > scripts/Makefile.lib | 16 ++ > > > > > > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > > > > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > > > > > > one. > > > > > > > > > > > > > > > > Any thoughts on this request, please? > > > > > > > > > > > > > > > > Regards, > > > > > > > > Simon > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I mentioned before, I am concerned with having > > > > > > > the same "compatible" entries, with different contents, > > > > > > > as you use the "compatible" string as an ID to selecting > > > > > > > the target config node, right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > conf-10 { > > > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > > > fdt = "fdt-10"; > > > > > > > kernel = "kernel"; > > > > > > > }; > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > conf-25 { > > > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > > > fdt = "fdt-25"; > > > > > > > kernel = "kernel"; > > > > > > > }; > > > > > > > > > > > > I had asked Rob a while ago about if having the same compatible for two > > > > > > functionally different machines is a feature, or a bug, and I don't > > > > > > think either of us fully agreed either way. I'd be leaning towards > > > > > > saying the above example is a bug in the dts files, it's just not been a > > > > > > bug people have worried about before due to (sadly) how little the > > > > > > top-level compatible has been used. > > > > I much prefer being able to use compatibles over filenames. > > > > > > > > > > > > Yes I believe this is a bug in the files. > > > > > > > > > > What should the script do in this case? Print a warning, perhaps? > > > > > > > > Is there anything I should do here? Would a warning be helpful, or > > > > just confusing? > > > > > > > > > > > > I do not think it is useful. > > > You would almost always get a warning, and there is no way to fix it. > > > > The above case is due to overlays. Why would you have a FIT image with > > both a base tree and applied overlays? > > > > Because they are different hardware. Meaning the base tree is valid on its own without any overlays? > If FIT includes only base DTBs, how to use a base with extensions? I would expect that you package up base and overlays or DTs with already applied overlays, but not both together. That would be based on whether your bootloader can apply overlays or not. This problem boils down to your firmware knows or gains the knowledge of some set of extra features or h/w pop options. The result is you need base plus X, Y, Z whether those are a list of overlays or an encoding of filename or something else. For example, FIT entries could have a field that just lists those X, Y, Z features. But I'd much rather have something that works outside of FIT images. > > In any case, maybe we need to record in dtb overlays that have been > > applied (which you asked about recently on dtc list). Not sure what > > that looks like though. Overlays have a 'top-level' compatible that we > > add in either separately or merged with the base's top-level > > compatible? > > > If there is a way to make "compatible" unique, that will be good. > > But, in my understanding, we can replace a property value, > but not modify it. Currently yes, but that shouldn't be too hard to add. The dtc modification is the easy part. The hard part is figuring out the policy around how we would use that. But I don't really know what you want to accomplish with FIT here. IMO, if you need filenames, then use a filesystem. They work pretty well for storing large collections of files. Rob
On Fri, Feb 2, 2024 at 6:03 AM Rob Herring <robh@kernel.org> wrote: > > On Wed, Jan 31, 2024 at 8:09 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Thu, Feb 1, 2024 at 7:03 AM Rob Herring <robh@kernel.org> wrote: > > > > > > On Tue, Jan 30, 2024 at 3:16 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > > > > > On Fri, Jan 26, 2024 at 1:04 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > Hi, > > > > > > > > > > On Wed, 17 Jan 2024 at 06:14, Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > > > Hi Masahiro, Tom, > > > > > > > > > > > > On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > > > > > > > > > > > > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > > > > > > > Hi Simon, > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > > > > > > > > > Hi Masahiro, > > > > > > > > > > > > > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > > > > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > > > > > > --- > > > > > > > > > > > > > > > > > > > > > > Changes in v9: > > > > > > > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > > > > > > > > > > > > > Changes in v8: > > > > > > > > > > > - Drop compatible string in FDT node > > > > > > > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > > > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > > > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > > > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > > > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > > > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > > > > > > > - Allow changing the compression used > > > > > > > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > > > > > > > > > > > > > Changes in v7: > > > > > > > > > > > - Add Image as a dependency of image.fit > > > > > > > > > > > - Drop kbuild tag > > > > > > > > > > > - Add dependency on dtbs > > > > > > > > > > > - Drop unnecessary path separator for dtbs > > > > > > > > > > > - Rebase to -next > > > > > > > > > > > > > > > > > > > > > > Changes in v5: > > > > > > > > > > > - Drop patch previously applied > > > > > > > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > > > > > > > > > > > > > Changes in v4: > > > > > > > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > > > > > > > > > > > > > Changes in v3: > > > > > > > > > > > - Drop temporary file image.itk > > > > > > > > > > > - Drop patch 'Use double quotes for image name' > > > > > > > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > > > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > > > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > > > > > > > > > > > > > Changes in v2: > > > > > > > > > > > - Drop patch previously applied > > > > > > > > > > > - Add .gitignore file > > > > > > > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > > > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > > > > > > > - Pick up .dtb files separately from the kernel > > > > > > > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > > > > > > > - Include the kernel image in the file count > > > > > > > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > > > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > > > > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > > > > > > > MAINTAINERS | 7 + > > > > > > > > > > > arch/arm64/Makefile | 7 +- > > > > > > > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > > > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > > > > > > > scripts/Makefile.lib | 16 ++ > > > > > > > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > > > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > > > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > > > > > > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > > > > > > > one. > > > > > > > > > > > > > > > > > > Any thoughts on this request, please? > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Simon > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I mentioned before, I am concerned with having > > > > > > > > the same "compatible" entries, with different contents, > > > > > > > > as you use the "compatible" string as an ID to selecting > > > > > > > > the target config node, right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > conf-10 { > > > > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > > > > fdt = "fdt-10"; > > > > > > > > kernel = "kernel"; > > > > > > > > }; > > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > conf-25 { > > > > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > > > > fdt = "fdt-25"; > > > > > > > > kernel = "kernel"; > > > > > > > > }; > > > > > > > > > > > > > > I had asked Rob a while ago about if having the same compatible for two > > > > > > > functionally different machines is a feature, or a bug, and I don't > > > > > > > think either of us fully agreed either way. I'd be leaning towards > > > > > > > saying the above example is a bug in the dts files, it's just not been a > > > > > > > bug people have worried about before due to (sadly) how little the > > > > > > > top-level compatible has been used. > > > > > > I much prefer being able to use compatibles over filenames. > > > > > > > > > > > > > > > Yes I believe this is a bug in the files. > > > > > > > > > > > > What should the script do in this case? Print a warning, perhaps? > > > > > > > > > > Is there anything I should do here? Would a warning be helpful, or > > > > > just confusing? > > > > > > > > > > > > > > > > I do not think it is useful. > > > > You would almost always get a warning, and there is no way to fix it. > > > > > > The above case is due to overlays. Why would you have a FIT image with > > > both a base tree and applied overlays? > > > > > > > > Because they are different hardware. > > Meaning the base tree is valid on its own without any overlays? If the base board is directly added to dtb-y, we need to assume it is valid as a standalone board. k3-am654-gp-evm-dtbs is a base of other composite DTBs, and it is also added to dtb-y. dtb-$(CONFIG_ARCH_K3) += k3-am654-base-board.dtb Not only the base board. There are multiple composite DTBs that have the same compatible string. k3-am654-gp-evm-dtbs := k3-am654-base-board.dtb k3-am654-base-board-rocktech-rk101-panel.dtbo k3-am654-evm-dtbs := k3-am654-base-board.dtb k3-am654-icssg2.dtbo k3-am654-base-board.dtb, k3-am654-gp-evm-dtbs, and k3-am654-evm-dtbs have the same top-level compatible string. > > > If FIT includes only base DTBs, how to use a base with extensions? > > I would expect that you package up base and overlays or DTs with > already applied overlays, but not both together. That would be based > on whether your bootloader can apply overlays or not. Correct. > This problem boils down to your firmware knows or gains the knowledge > of some set of extra features or h/w pop options. The result is you > need base plus X, Y, Z whether those are a list of overlays or an > encoding of filename or something else. For example, FIT entries could > have a field that just lists those X, Y, Z features. But I'd much > rather have something that works outside of FIT images. > > > > In any case, maybe we need to record in dtb overlays that have been > > > applied (which you asked about recently on dtc list). Not sure what > > > that looks like though. Overlays have a 'top-level' compatible that we > > > add in either separately or merged with the base's top-level > > > compatible? > > > > > > If there is a way to make "compatible" unique, that will be good. > > > > But, in my understanding, we can replace a property value, > > but not modify it. > > Currently yes, but that shouldn't be too hard to add. The dtc > modification is the easy part. The hard part is figuring out the > policy around how we would use that. > > But I don't really know what you want to accomplish with FIT here. > IMO, if you need filenames, then use a filesystem. They work pretty > well for storing large collections of files. Whom does the term "you" point to? If you have questions about the motivation for this patch, ask its author. There are two ways to look up the config node; node-name or compatible string. The key-value lookup works only when the key is unique. Apparently, the compatible string is not unique when overlay comes into play. > > Rob -- Best Regards Masahiro Yamada
Hi, On Wed, 31 Jan 2024 at 15:03, Rob Herring <robh@kernel.org> wrote: > > On Tue, Jan 30, 2024 at 3:16 AM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > On Fri, Jan 26, 2024 at 1:04 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > Hi, > > > > > > On Wed, 17 Jan 2024 at 06:14, Simon Glass <sjg@chromium.org> wrote: > > > > > > > > Hi Masahiro, Tom, > > > > > > > > On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > > > > > > > > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > > > > > Hi Simon, > > > > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > > > > > Hi Masahiro, > > > > > > > > > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > > performance. > > > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > > > > --- > > > > > > > > > > > > > > > > > > Changes in v9: > > > > > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > > > > > > > > > Changes in v8: > > > > > > > > > - Drop compatible string in FDT node > > > > > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > > > > > - Allow changing the compression used > > > > > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > > > > > > > > > Changes in v7: > > > > > > > > > - Add Image as a dependency of image.fit > > > > > > > > > - Drop kbuild tag > > > > > > > > > - Add dependency on dtbs > > > > > > > > > - Drop unnecessary path separator for dtbs > > > > > > > > > - Rebase to -next > > > > > > > > > > > > > > > > > > Changes in v5: > > > > > > > > > - Drop patch previously applied > > > > > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > > > > > > > > > Changes in v4: > > > > > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > > > > > > > > > Changes in v3: > > > > > > > > > - Drop temporary file image.itk > > > > > > > > > - Drop patch 'Use double quotes for image name' > > > > > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > > > > > > > > > Changes in v2: > > > > > > > > > - Drop patch previously applied > > > > > > > > > - Add .gitignore file > > > > > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > > > > > - Pick up .dtb files separately from the kernel > > > > > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > > > > > - Include the kernel image in the file count > > > > > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > > > > > MAINTAINERS | 7 + > > > > > > > > > arch/arm64/Makefile | 7 +- > > > > > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > > > > > scripts/Makefile.lib | 16 ++ > > > > > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > > > > > one. > > > > > > > > > > > > > > Any thoughts on this request, please? > > > > > > > > > > > > > > Regards, > > > > > > > Simon > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I mentioned before, I am concerned with having > > > > > > the same "compatible" entries, with different contents, > > > > > > as you use the "compatible" string as an ID to selecting > > > > > > the target config node, right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > > > > > > > ... > > > > > > > > > > > > conf-10 { > > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > > fdt = "fdt-10"; > > > > > > kernel = "kernel"; > > > > > > }; > > > > > > > > > > > > ... > > > > > > > > > > > > conf-25 { > > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > > fdt = "fdt-25"; > > > > > > kernel = "kernel"; > > > > > > }; > > > > > > > > > > I had asked Rob a while ago about if having the same compatible for two > > > > > functionally different machines is a feature, or a bug, and I don't > > > > > think either of us fully agreed either way. I'd be leaning towards > > > > > saying the above example is a bug in the dts files, it's just not been a > > > > > bug people have worried about before due to (sadly) how little the > > > > > top-level compatible has been used. > > I much prefer being able to use compatibles over filenames. So do I. There is no check that each dts has a unique compatible string (e.g. in the first position). Perhaps we should add that and have vendors fix up their strings? [..] > > Rob Regards, Simon
Hi Masahiro, On Tue, 30 Jan 2024 at 02:16, Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Fri, Jan 26, 2024 at 1:04 AM Simon Glass <sjg@chromium.org> wrote: > > > > Hi, > > > > On Wed, 17 Jan 2024 at 06:14, Simon Glass <sjg@chromium.org> wrote: > > > > > > Hi Masahiro, Tom, > > > > > > On Tue, 9 Jan 2024 at 07:33, Tom Rini <trini@konsulko.com> wrote: > > > > > > > > On Tue, Jan 09, 2024 at 11:01:42PM +0900, Masahiro Yamada wrote: > > > > > Hi Simon, > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 8:47 AM Simon Glass <sjg@chromium.org> wrote: > > > > > > > > > > > > Hi Masahiro, > > > > > > > > > > > > On Wed, Dec 13, 2023 at 5:14 AM Will Deacon <will@kernel.org> wrote: > > > > > > > > > > > > > > On Fri, Dec 01, 2023 at 08:54:42PM -0700, Simon Glass wrote: > > > > > > > > Add a script which produces a Flat Image Tree (FIT), a single file > > > > > > > > containing the built kernel and associated devicetree files. > > > > > > > > Compression defaults to gzip which gives a good balance of size and > > > > > > > > performance. > > > > > > > > > > > > > > > > The files compress from about 86MB to 24MB using this approach. > > > > > > > > > > > > > > > > The FIT can be used by bootloaders which support it, such as U-Boot > > > > > > > > and Linuxboot. It permits automatic selection of the correct > > > > > > > > devicetree, matching the compatible string of the running board with > > > > > > > > the closest compatible string in the FIT. There is no need for > > > > > > > > filenames or other workarounds. > > > > > > > > > > > > > > > > Add a 'make image.fit' build target for arm64, as well. Use > > > > > > > > FIT_COMPRESSION to select a different algorithm. > > > > > > > > > > > > > > > > The FIT can be examined using 'dumpimage -l'. > > > > > > > > > > > > > > > > This features requires pylibfdt (use 'pip install libfdt'). It also > > > > > > > > requires compression utilities for the algorithm being used. Supported > > > > > > > > compression options are the same as the Image.xxx files. For now there > > > > > > > > is no way to change the compression other than by editing the rule for > > > > > > > > $(obj)/image.fit > > > > > > > > > > > > > > > > While FIT supports a ramdisk / initrd, no attempt is made to support > > > > > > > > this here, since it must be built separately from the Linux build. > > > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > > > --- > > > > > > > > > > > > > > > > Changes in v9: > > > > > > > > - Move the compression control into Makefile.lib > > > > > > > > > > > > > > > > Changes in v8: > > > > > > > > - Drop compatible string in FDT node > > > > > > > > - Correct sorting of MAINTAINERS to before ARM64 PORT > > > > > > > > - Turn compress part of the make_fit.py comment in to a sentence > > > > > > > > - Add two blank lines before parse_args() and setup_fit() > > > > > > > > - Use 'image.fit: dtbs' instead of BUILD_DTBS var > > > > > > > > - Use '$(<D)/dts' instead of '$(dir $<)dts' > > > > > > > > - Add 'mkimage' details Documentation/process/changes.rst > > > > > > > > - Allow changing the compression used > > > > > > > > - Tweak cover letter since there is only one clean-up patch > > > > > > > > > > > > > > > > Changes in v7: > > > > > > > > - Add Image as a dependency of image.fit > > > > > > > > - Drop kbuild tag > > > > > > > > - Add dependency on dtbs > > > > > > > > - Drop unnecessary path separator for dtbs > > > > > > > > - Rebase to -next > > > > > > > > > > > > > > > > Changes in v5: > > > > > > > > - Drop patch previously applied > > > > > > > > - Correct compression rule which was broken in v4 > > > > > > > > > > > > > > > > Changes in v4: > > > > > > > > - Use single quotes for UIMAGE_NAME > > > > > > > > > > > > > > > > Changes in v3: > > > > > > > > - Drop temporary file image.itk > > > > > > > > - Drop patch 'Use double quotes for image name' > > > > > > > > - Drop double quotes in use of UIMAGE_NAME > > > > > > > > - Drop unnecessary CONFIG_EFI_ZBOOT condition for help > > > > > > > > - Avoid hard-coding "arm64" for the DT architecture > > > > > > > > > > > > > > > > Changes in v2: > > > > > > > > - Drop patch previously applied > > > > > > > > - Add .gitignore file > > > > > > > > - Move fit rule to Makefile.lib using an intermediate file > > > > > > > > - Drop dependency on CONFIG_EFI_ZBOOT > > > > > > > > - Pick up .dtb files separately from the kernel > > > > > > > > - Correct pylint too-many-args warning for write_kernel() > > > > > > > > - Include the kernel image in the file count > > > > > > > > - Add a pointer to the FIT spec and mention of its wide industry usage > > > > > > > > - Mention the kernel version in the FIT description > > > > > > > > > > > > > > > > Documentation/process/changes.rst | 9 + > > > > > > > > MAINTAINERS | 7 + > > > > > > > > arch/arm64/Makefile | 7 +- > > > > > > > > arch/arm64/boot/.gitignore | 1 + > > > > > > > > arch/arm64/boot/Makefile | 6 +- > > > > > > > > scripts/Makefile.lib | 16 ++ > > > > > > > > scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ > > > > > > > > 7 files changed, 334 insertions(+), 3 deletions(-) > > > > > > > > create mode 100755 scripts/make_fit.py > > > > > > > > > > > > > > I'll need Masahiro's Ack on the scripts/ changes before I can take this > > > > > > > one. > > > > > > > > > > > > Any thoughts on this request, please? > > > > > > > > > > > > Regards, > > > > > > Simon > > > > > > > > > > > > > > > > > > > > > > > > > > As I mentioned before, I am concerned with having > > > > > the same "compatible" entries, with different contents, > > > > > as you use the "compatible" string as an ID to selecting > > > > > the target config node, right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > $ fdtdump arch/arm64/boot/image.fit > > > > > > > > > > ... > > > > > > > > > > conf-10 { > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > fdt = "fdt-10"; > > > > > kernel = "kernel"; > > > > > }; > > > > > > > > > > ... > > > > > > > > > > conf-25 { > > > > > compatible = "tq,am642-tqma6442l-mbax4xxl", > > > > > "tq,am642-tqma6442l", "ti,am642"; > > > > > description = "TQ-Systems TQMa64xxL SoM on MBax4xxL carrier board"; > > > > > fdt = "fdt-25"; > > > > > kernel = "kernel"; > > > > > }; > > > > > > > > I had asked Rob a while ago about if having the same compatible for two > > > > functionally different machines is a feature, or a bug, and I don't > > > > think either of us fully agreed either way. I'd be leaning towards > > > > saying the above example is a bug in the dts files, it's just not been a > > > > bug people have worried about before due to (sadly) how little the > > > > top-level compatible has been used. > > > > > > Yes I believe this is a bug in the files. > > > > > > What should the script do in this case? Print a warning, perhaps? > > > > Is there anything I should do here? Would a warning be helpful, or > > just confusing? > > > > I do not think it is useful. > You would almost always get a warning, and there is no way to fix it. > > > > With arm64 defconfig, image.fit will include a thousand DTBs. > > > The config node of my board was listed 214th. > (I found it by fdtdump) > > > Then, I learned > > > bootm ${loadaddr}#conf-214 > > is the correct command to boot my board. > > > Of course, the "214" will be different in the future. > > The node names, conf-*, are useless. > > > > Only the useful way is to enable CONFIG_FIT_BEST_MATCH in U-Boot, > but this relies on the uniqueness of a compatible string, > which is not true. See my other email, but I think this is a problem we should fix. > > (I do not know how to do it in barebox) > > > > > > I think using the file name as a config node > mitigates the issue because a file name is > considered unique. > > > For example, with composite DTBs: > > imx8mm-venice-gw72xx-0x-imx219-dtbs := imx8mm-venice-gw72xx-0x.dtb > imx8mm-venice-gw72xx-0x-imx219.dtbo > imx8mm-venice-gw72xx-0x-rpidsi-dtbs := imx8mm-venice-gw72xx-0x.dtb > imx8mm-venice-gw72xx-0x-rpidsi.dtbo > > > > > configurations { > imx8mm-venice-gw72xx-0x-imx219 { > ... > }; > > imx8mm-venice-gw72xx-0x-rpidsi { > ... > } > }; > > > > Then, we can distinguish them by node, even if they have > the same compatible string. > At least we can do > > > bootm ${loadaddr}#imx8mm-venice-gw72xx-0x-imx219 > > Sure, but the goal with FIT is to be able to boot with the correct dtb automatically. What you have above would require some sort of boot script, or some other information about the configuration-node name. > > > For the issue including stale DTBs, you can use my patch: > [PATCH 1/4] kbuild: create a list of all built DTB files > (https://lore.kernel.org/linux-kbuild/CAK7LNASOxi-gzve+_d-sCW9z_eEJ5TMMnzPEvN2Nj2AwgVjF9g@mail.gmail.com/T/#ma3595627a96a04554a78cbbd22056831e13db260) > > > You can change scripts/make_fit.py to take > the DTB files instead of the directory to search. > > Optionally, you can support '@' syntax to > take command arguments from a file. > > > scripts/make_fit.py ... @arch/$(SRCARCH)/boot/dts/dtbs-list Thank you for doing that. I will take a look and send v10. > > > > > > > For the separate base and overlays support, > you can use my patch as a base: > [PATCH 3/4] kbuild: create a list of base and overlays for each DTB > (https://lore.kernel.org/linux-kbuild/CAK7LNASOxi-gzve+_d-sCW9z_eEJ5TMMnzPEvN2Nj2AwgVjF9g@mail.gmail.com/T/#m32c5bdde9098901b7c7776b932827493a05c82d5) > > > > > > Lastly, you do not need to require mkimage for > args.external. > You can simply concatenate files. Yes, that's true. Do you think there is a problem with using mkimage? Regards, Simon
diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst index bb96ca0f774b..cad51bd5bd62 100644 --- a/Documentation/process/changes.rst +++ b/Documentation/process/changes.rst @@ -62,6 +62,7 @@ Sphinx\ [#f1]_ 1.7 sphinx-build --version cpio any cpio --version GNU tar 1.28 tar --version gtags (optional) 6.6.5 gtags --version +mkimage (optional) 2017.01 mkimage --version ====================== =============== ======================================== .. [#f1] Sphinx is needed only to build the Kernel documentation @@ -189,6 +190,14 @@ The kernel build requires GNU GLOBAL version 6.6.5 or later to generate tag files through ``make gtags``. This is due to its use of the gtags ``-C (--directory)`` flag. +mkimage +------- + +This tool is used when building a Flat Image Tree (FIT), commonly used on ARM +platforms. The tool is available via the ``u-boot-tools`` package or can be +built from the U-Boot source code. See the instructions at +https://docs.u-boot.org/en/latest/build/tools.html#building-tools-for-linux + System utilities **************** diff --git a/MAINTAINERS b/MAINTAINERS index 9d46229fe21b..d2d17f0d6e64 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3037,6 +3037,13 @@ F: drivers/mmc/host/sdhci-of-arasan.c N: zynq N: xilinx +ARM64 FIT SUPPORT +M: Simon Glass <sjg@chromium.org> +L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) +S: Maintained +F: arch/arm64/boot/Makefile +F: scripts/make_fit.py + ARM64 PORT (AARCH64 ARCHITECTURE) M: Catalin Marinas <catalin.marinas@arm.com> M: Will Deacon <will@kernel.org> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 1bd4fae6e806..6b893dc454b7 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -150,7 +150,7 @@ libs-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a # Default target when executing plain make boot := arch/arm64/boot -BOOT_TARGETS := Image vmlinuz.efi +BOOT_TARGETS := Image vmlinuz.efi image.fit PHONY += $(BOOT_TARGETS) @@ -162,7 +162,9 @@ endif all: $(notdir $(KBUILD_IMAGE)) -vmlinuz.efi: Image +image.fit: dtbs + +vmlinuz.efi image.fit: Image $(BOOT_TARGETS): vmlinux $(Q)$(MAKE) $(build)=$(boot) $(boot)/$@ @@ -215,6 +217,7 @@ virtconfig: define archhelp echo '* Image.gz - Compressed kernel image (arch/$(ARCH)/boot/Image.gz)' echo ' Image - Uncompressed kernel image (arch/$(ARCH)/boot/Image)' + echo ' image.fit - Flat Image Tree (arch/$(ARCH)/boot/image.fit)' echo ' install - Install uncompressed kernel' echo ' zinstall - Install compressed kernel' echo ' Install using (your) ~/bin/installkernel or' diff --git a/arch/arm64/boot/.gitignore b/arch/arm64/boot/.gitignore index af5dc61f8b43..abaae9de1bdd 100644 --- a/arch/arm64/boot/.gitignore +++ b/arch/arm64/boot/.gitignore @@ -2,3 +2,4 @@ Image Image.gz vmlinuz* +image.fit diff --git a/arch/arm64/boot/Makefile b/arch/arm64/boot/Makefile index 1761f5972443..b835c0880d1c 100644 --- a/arch/arm64/boot/Makefile +++ b/arch/arm64/boot/Makefile @@ -16,7 +16,8 @@ OBJCOPYFLAGS_Image :=-O binary -R .note -R .note.gnu.build-id -R .comment -S -targets := Image Image.bz2 Image.gz Image.lz4 Image.lzma Image.lzo Image.zst +targets := Image Image.bz2 Image.gz Image.lz4 Image.lzma Image.lzo \ + Image.zst image.fit $(obj)/Image: vmlinux FORCE $(call if_changed,objcopy) @@ -39,6 +40,9 @@ $(obj)/Image.lzo: $(obj)/Image FORCE $(obj)/Image.zst: $(obj)/Image FORCE $(call if_changed,zstd) +$(obj)/image.fit: $(obj)/Image FORCE + $(call cmd,fit) + EFI_ZBOOT_PAYLOAD := Image EFI_ZBOOT_BFD_TARGET := elf64-littleaarch64 EFI_ZBOOT_MACH_TYPE := ARM64 diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 1a965fe68e01..1c60d594932c 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -496,6 +496,22 @@ quiet_cmd_uimage = UIMAGE $@ -a $(UIMAGE_LOADADDR) -e $(UIMAGE_ENTRYADDR) \ -n '$(UIMAGE_NAME)' -d $< $@ +# Flat Image Tree (FIT) +# This allows for packaging of a kernel and all devicetrees files, using +# compression. +# --------------------------------------------------------------------------- + +MAKE_FIT := $(srctree)/scripts/make_fit.py + +# Use this to override the compression algorithm +FIT_COMPRESSION ?= gzip + +quiet_cmd_fit = FIT $@ + cmd_fit = $(MAKE_FIT) -f $@ --arch $(UIMAGE_ARCH) --os linux \ + --name '$(UIMAGE_NAME)' \ + --compress $(FIT_COMPRESSION) -k $< \ + $(<D)/dts + # XZ # --------------------------------------------------------------------------- # Use xzkern to compress the kernel image and xzmisc to compress other things. diff --git a/scripts/make_fit.py b/scripts/make_fit.py new file mode 100755 index 000000000000..e616b0d7a84a --- /dev/null +++ b/scripts/make_fit.py @@ -0,0 +1,291 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0+ +# +# Copyright 2023 Google LLC +# Written by Simon Glass <sjg@chromium.org> +# + +"""Build a FIT containing a lot of devicetree files + +Usage: + make_fit.py -A arm64 -n 'Linux-6.6' -O linux + -f arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk + /tmp/kern/arch/arm64/boot/dts/ -E -c gzip + +Creates a FIT containing the supplied kernel and a directory containing the +devicetree files. + +Use -E to generate an external FIT (where the data is placed after the +FIT data structure). This allows parsing of the data without loading +the entire FIT. + +Use -c to compress the data, using bzip2, gzip, lz4, lzma, lzo and +zstd algorithms. + +The resulting FIT can be booted by bootloaders which support FIT, such +as U-Boot, Linuxboot, Tianocore, etc. + +Note that this tool does not yet support adding a ramdisk / initrd. +""" + +import argparse +import collections +import os +import subprocess +import sys +import tempfile +import time + +import libfdt + + +# Tool extension and the name of the command-line tools +CompTool = collections.namedtuple('CompTool', 'ext,tools') + +COMP_TOOLS = { + 'bzip2': CompTool('.bz2', 'bzip2'), + 'gzip': CompTool('.gz', 'pigz,gzip'), + 'lz4': CompTool('.lz4', 'lz4'), + 'lzma': CompTool('.lzma', 'lzma'), + 'lzo': CompTool('.lzo', 'lzop'), + 'zstd': CompTool('.zstd', 'zstd'), +} + + +def parse_args(): + """Parse the program ArgumentParser + + Returns: + Namespace object containing the arguments + """ + epilog = 'Build a FIT from a directory tree containing .dtb files' + parser = argparse.ArgumentParser(epilog=epilog) + parser.add_argument('-A', '--arch', type=str, required=True, + help='Specifies the architecture') + parser.add_argument('-c', '--compress', type=str, default='none', + help='Specifies the compression') + parser.add_argument('-E', '--external', action='store_true', + help='Convert the FIT to use external data') + parser.add_argument('-n', '--name', type=str, required=True, + help='Specifies the name') + parser.add_argument('-O', '--os', type=str, required=True, + help='Specifies the operating system') + parser.add_argument('-f', '--fit', type=str, required=True, + help='Specifies the output file (.fit)') + parser.add_argument('-k', '--kernel', type=str, required=True, + help='Specifies the (uncompressed) kernel input file (.itk)') + parser.add_argument('srcdir', type=str, nargs='*', + help='Specifies the directory tree that contains .dtb files') + + return parser.parse_args() + + +def setup_fit(fsw, name): + """Make a start on writing the FIT + + Outputs the root properties and the 'images' node + + Args: + fsw (libfdt.FdtSw): Object to use for writing + name (str): Name of kernel image + """ + fsw.INC_SIZE = 65536 + fsw.finish_reservemap() + fsw.begin_node('') + fsw.property_string('description', f'{name} with devicetree set') + fsw.property_u32('#address-cells', 1) + + fsw.property_u32('timestamp', int(time.time())) + fsw.begin_node('images') + + +def write_kernel(fsw, data, args): + """Write out the kernel image + + Writes a kernel node along with the required properties + + Args: + fsw (libfdt.FdtSw): Object to use for writing + data (bytes): Data to write (possibly compressed) + args (Namespace): Contains necessary strings: + arch: FIT architecture, e.g. 'arm64' + fit_os: Operating Systems, e.g. 'linux' + name: Name of OS, e.g. 'Linux-6.6.0-rc7' + compress: Compression algorithm to use, e.g. 'gzip' + """ + with fsw.add_node('kernel'): + fsw.property_string('description', args.name) + fsw.property_string('type', 'kernel_noload') + fsw.property_string('arch', args.arch) + fsw.property_string('os', args.os) + fsw.property_string('compression', args.compress) + fsw.property('data', data) + fsw.property_u32('load', 0) + fsw.property_u32('entry', 0) + + +def finish_fit(fsw, entries): + """Finish the FIT ready for use + + Writes the /configurations node and subnodes + + Args: + fsw (libfdt.FdtSw): Object to use for writing + entries (list of tuple): List of configurations: + str: Description of model + str: Compatible stringlist + """ + fsw.end_node() + seq = 0 + with fsw.add_node('configurations'): + for model, compat in entries: + seq += 1 + with fsw.add_node(f'conf-{seq}'): + fsw.property('compatible', bytes(compat)) + fsw.property_string('description', model) + fsw.property_string('fdt', f'fdt-{seq}') + fsw.property_string('kernel', 'kernel') + fsw.end_node() + + +def compress_data(inf, compress): + """Compress data using a selected algorithm + + Args: + inf (IOBase): Filename containing the data to compress + compress (str): Compression algorithm, e.g. 'gzip' + + Return: + bytes: Compressed data + """ + if compress == 'none': + return inf.read() + + comp = COMP_TOOLS.get(compress) + if not comp: + raise ValueError(f"Unknown compression algorithm '{compress}'") + + with tempfile.NamedTemporaryFile() as comp_fname: + with open(comp_fname.name, 'wb') as outf: + done = False + for tool in comp.tools.split(','): + try: + subprocess.call([tool, '-c'], stdin=inf, stdout=outf) + done = True + break + except FileNotFoundError: + pass + if not done: + raise ValueError(f'Missing tool(s): {comp.tools}\n') + with open(comp_fname.name, 'rb') as compf: + comp_data = compf.read() + return comp_data + + +def output_dtb(fsw, seq, fname, arch, compress): + """Write out a single devicetree to the FIT + + Args: + fsw (libfdt.FdtSw): Object to use for writing + seq (int): Sequence number (1 for first) + fmame (str): Filename containing the DTB + arch: FIT architecture, e.g. 'arm64' + compress (str): Compressed algorithm, e.g. 'gzip' + + Returns: + tuple: + str: Model name + bytes: Compatible stringlist + """ + with fsw.add_node(f'fdt-{seq}'): + # Get the compatible / model information + with open(fname, 'rb') as inf: + data = inf.read() + fdt = libfdt.FdtRo(data) + model = fdt.getprop(0, 'model').as_str() + compat = fdt.getprop(0, 'compatible') + + fsw.property_string('description', model) + fsw.property_string('type', 'flat_dt') + fsw.property_string('arch', arch) + fsw.property_string('compression', compress) + fsw.property('compatible', bytes(compat)) + + with open(fname, 'rb') as inf: + compressed = compress_data(inf, compress) + fsw.property('data', compressed) + return model, compat + + +def build_fit(args): + """Build the FIT from the provided files and arguments + + Args: + args (Namespace): Program arguments + + Returns: + tuple: + bytes: FIT data + int: Number of configurations generated + size: Total uncompressed size of data + """ + fsw = libfdt.FdtSw() + setup_fit(fsw, args.name) + seq = 0 + size = 0 + entries = [] + + # Handle the kernel + with open(args.kernel, 'rb') as inf: + comp_data = compress_data(inf, args.compress) + size += os.path.getsize(args.kernel) + write_kernel(fsw, comp_data, args) + + for path in args.srcdir: + # Handle devicetree files + if os.path.isdir(path): + for dirpath, _, fnames in os.walk(path): + for fname in fnames: + if os.path.splitext(fname)[1] != '.dtb': + continue + pathname = os.path.join(dirpath, fname) + seq += 1 + size += os.path.getsize(pathname) + model, compat = output_dtb(fsw, seq, pathname, + args.arch, args.compress) + entries.append([model, compat]) + + finish_fit(fsw, entries) + + # Include the kernel itself in the returned file count + return fsw.as_fdt().as_bytearray(), seq + 1, size + + +def run_make_fit(): + """Run the tool's main logic""" + args = parse_args() + + out_data, count, size = build_fit(args) + with open(args.fit, 'wb') as outf: + outf.write(out_data) + + ext_fit_size = None + if args.external: + mkimage = os.environ.get('MKIMAGE', 'mkimage') + subprocess.check_call([mkimage, '-E', '-F', args.fit], + stdout=subprocess.DEVNULL) + + with open(args.fit, 'rb') as inf: + data = inf.read() + ext_fit = libfdt.FdtRo(data) + ext_fit_size = ext_fit.totalsize() + + comp_size = len(out_data) + print(f'FIT size {comp_size:#x}/{comp_size / 1024 / 1024:.1f} MB', end='') + if ext_fit_size: + print(f', header {ext_fit_size:#x}/{ext_fit_size / 1024:.1f} KB', end='') + print(f', {count} files, uncompressed {size / 1024 / 1024:.1f} MB') + + +if __name__ == "__main__": + sys.exit(run_make_fit())
Add a script which produces a Flat Image Tree (FIT), a single file containing the built kernel and associated devicetree files. Compression defaults to gzip which gives a good balance of size and performance. The files compress from about 86MB to 24MB using this approach. The FIT can be used by bootloaders which support it, such as U-Boot and Linuxboot. It permits automatic selection of the correct devicetree, matching the compatible string of the running board with the closest compatible string in the FIT. There is no need for filenames or other workarounds. Add a 'make image.fit' build target for arm64, as well. Use FIT_COMPRESSION to select a different algorithm. The FIT can be examined using 'dumpimage -l'. This features requires pylibfdt (use 'pip install libfdt'). It also requires compression utilities for the algorithm being used. Supported compression options are the same as the Image.xxx files. For now there is no way to change the compression other than by editing the rule for $(obj)/image.fit While FIT supports a ramdisk / initrd, no attempt is made to support this here, since it must be built separately from the Linux build. Signed-off-by: Simon Glass <sjg@chromium.org> --- Changes in v9: - Move the compression control into Makefile.lib Changes in v8: - Drop compatible string in FDT node - Correct sorting of MAINTAINERS to before ARM64 PORT - Turn compress part of the make_fit.py comment in to a sentence - Add two blank lines before parse_args() and setup_fit() - Use 'image.fit: dtbs' instead of BUILD_DTBS var - Use '$(<D)/dts' instead of '$(dir $<)dts' - Add 'mkimage' details Documentation/process/changes.rst - Allow changing the compression used - Tweak cover letter since there is only one clean-up patch Changes in v7: - Add Image as a dependency of image.fit - Drop kbuild tag - Add dependency on dtbs - Drop unnecessary path separator for dtbs - Rebase to -next Changes in v5: - Drop patch previously applied - Correct compression rule which was broken in v4 Changes in v4: - Use single quotes for UIMAGE_NAME Changes in v3: - Drop temporary file image.itk - Drop patch 'Use double quotes for image name' - Drop double quotes in use of UIMAGE_NAME - Drop unnecessary CONFIG_EFI_ZBOOT condition for help - Avoid hard-coding "arm64" for the DT architecture Changes in v2: - Drop patch previously applied - Add .gitignore file - Move fit rule to Makefile.lib using an intermediate file - Drop dependency on CONFIG_EFI_ZBOOT - Pick up .dtb files separately from the kernel - Correct pylint too-many-args warning for write_kernel() - Include the kernel image in the file count - Add a pointer to the FIT spec and mention of its wide industry usage - Mention the kernel version in the FIT description Documentation/process/changes.rst | 9 + MAINTAINERS | 7 + arch/arm64/Makefile | 7 +- arch/arm64/boot/.gitignore | 1 + arch/arm64/boot/Makefile | 6 +- scripts/Makefile.lib | 16 ++ scripts/make_fit.py | 291 ++++++++++++++++++++++++++++++ 7 files changed, 334 insertions(+), 3 deletions(-) create mode 100755 scripts/make_fit.py