mbox series

[XEN,v4,00/32] Toolstack build system improvement, toward non-recursive makefiles

Message ID 20220811164845.38083-1-anthony.perard@citrix.com (mailing list archive)
Headers show
Series Toolstack build system improvement, toward non-recursive makefiles | expand

Message

Anthony PERARD Aug. 11, 2022, 4:48 p.m. UTC 
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.toolstack-build-system-v4

Changes in v4:
- several new patches
- some changes to other patches listed in their changelogs

Changes in v3:
- rebased
- several new patches, starting with 13/25 "tools/libs/util: cleanup Makefile"
- introducing macros to deal with linking with in-tree xen libraries
- Add -Werror to CFLAGS for all builds in tools/

Changes in v2:
- one new patch
- other changes described in patch notes

Hi everyone,

I've been looking at reworking the build system we have for the "tools/", and
transforming it to something that suit it better. There are a lot of
dependencies between different sub-directories so it would be nice if GNU make
could actually handle them. This is possible with "non-recursive makefiles".

With non-recursive makefiles, make will have to load/include all the makefiles
and thus will have complete overview of all the dependencies. This will allow
make to build the necessary targets in other directory, and we won't need to
build sub-directories one by one.

To help with this transformation, I've chosen to go with a recent project
called "subdirmk". It help to deal with the fact that all makefiles will share
the same namespace, it is hooked into autoconf, we can easily run `make` from
any subdirectory. Together "autoconf" and "subdirmk" will also help to get
closer to be able to do out-of-tree build of the tools, but I'm mainly looking
to have non-recursive makefile.

Link to the project:
    https://www.chiark.greenend.org.uk/ucgi/~ian/git/subdirmk.git/

But before getting to the main course, I've got quite a few cleanup and some
changes to the makefiles. I start the patch series with patches that remove old
left over stuff, then start reworking makefiles. They are some common changes like
removing the "build" targets in many places as "all" would be the more common
way to spell it and "all" is the default target anyway. They are other changes
related to the conversion to "subdirmk", I start to use the variable $(TARGETS)
in several makefiles, this variable will have a special meaning in subdirmk
which will build those target by default.

As for the conversion to non-recursive makefile, with subdirmk, I have this WIP
branch, it contains some changes that I'm trying out, some notes, and the
conversion, one Makefile per commit. Cleanup are still needed, some makefile
not converted yet, but it's otherwise mostly done.

    https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.toolstack-build-system-v1-wip-extra

With that branch, you could tried something like:
    ./configure; cd tools/xl; make
and `xl` should be built as well as all the xen library needed.
Also, things like `make clean` or rebuild should be faster in the all tools/
directory.

Cheers,

Anthony PERARD (32):
  tools/debugger/gdbsx: Fix and cleanup makefiles
  tools/firmware/hvmloader: rework Makefile
  tools/fuzz/x86_instruction_emulator: rework makefile
  tools/hotplug: cleanup Makefiles
  tools/libfsimage: Cleanup makefiles
  tools/xenpaging: Rework makefile
  tools/xentop: rework makefile
  tools/xentrace: rework Makefile
  .gitignore: Cleanup ignores of tools/libs/*/{headers.chk,*.pc}
  tools/libs/util: cleanup Makefile
  tools/flask/utils: list build targets in $(TARGETS)
  libs/libs.mk: Rename $(LIB) to $(TARGETS)
  libs/libs.mk: Remove the need for $(PKG_CONFIG_INST)
  libs/libs.mk: Rework target headers.chk dependencies
  tools: Introduce $(xenlibs-rpath,..) to replace $(SHDEPS_lib*)
  tools: Introduce $(xenlibs-ldlibs, ) macro
  tools: Introduce $(xenlibs-ldflags, ) macro
  tools/helper: Cleanup Makefile
  tools/console: Use $(xenlibs-ldlibs,)
  tools: Add -Werror by default to all tools/
  tools: Remove -Werror everywhere else
  tools/hotplug: Generate "hotplugpath.sh" with configure
  libs/light/gentypes.py: allow to generate headers in subdirectory
  git-checkout.sh: handle running git-checkout from a different
    directory
  libs: Avoid exposing -Wl,--version-script to other built library
  libs: Fix auto-generation of version-script for unstable libs
  tools/include: Rework Makefile
  libs/light: Rework acpi table build targets
  libs/light: Rework generation of include/_libxl_*.h
  libs/light: Rework targets prerequisites
  libs/light: Makefile cleanup
  tools/golang/xenlight: Rework gengotypes.py and generation of *.gen.go

 tools/configure.ac                            |  2 +
 tools/Makefile                                |  2 +-
 tools/console/client/Makefile                 |  4 +-
 tools/console/daemon/Makefile                 |  7 +-
 tools/debugger/gdbsx/Makefile                 | 20 +++---
 tools/debugger/gdbsx/gx/Makefile              | 15 ++--
 tools/debugger/gdbsx/xg/Makefile              | 25 ++-----
 tools/debugger/kdd/Makefile                   |  1 -
 tools/firmware/hvmloader/Makefile             | 19 +++--
 tools/flask/utils/Makefile                    | 11 ++-
 tools/fuzz/cpu-policy/Makefile                |  2 +-
 tools/fuzz/x86_instruction_emulator/Makefile  | 35 +++++----
 tools/golang/xenlight/Makefile                |  8 ++-
 tools/helpers/Makefile                        | 23 +++---
 tools/hotplug/FreeBSD/Makefile                | 11 +--
 tools/hotplug/Linux/Makefile                  | 16 ++---
 tools/hotplug/Linux/systemd/Makefile          | 16 ++---
 tools/hotplug/NetBSD/Makefile                 |  9 +--
 tools/hotplug/common/Makefile                 | 20 ++----
 tools/include/Makefile                        | 28 ++++----
 tools/libfsimage/common/Makefile              | 11 +--
 tools/libfsimage/ext2fs-lib/Makefile          |  9 ---
 tools/libfsimage/ext2fs/Makefile              |  9 ---
 tools/libfsimage/fat/Makefile                 |  9 ---
 tools/libfsimage/iso9660/Makefile             | 11 ---
 tools/libfsimage/reiserfs/Makefile            |  9 ---
 tools/libfsimage/ufs/Makefile                 |  9 ---
 tools/libfsimage/xfs/Makefile                 |  9 ---
 tools/libfsimage/zfs/Makefile                 |  9 ---
 tools/libs/call/Makefile                      |  1 +
 tools/libs/ctrl/Makefile                      |  3 -
 tools/libs/devicemodel/Makefile               |  1 +
 tools/libs/evtchn/Makefile                    |  1 +
 tools/libs/foreignmemory/Makefile             |  1 +
 tools/libs/gnttab/Makefile                    |  1 +
 tools/libs/guest/Makefile                     |  3 -
 tools/libs/hypfs/Makefile                     |  1 +
 tools/libs/light/Makefile                     | 72 +++++++++++--------
 tools/libs/stat/Makefile                      |  2 +-
 tools/libs/store/Makefile                     |  1 +
 tools/libs/toolcore/Makefile                  |  1 +
 tools/libs/toollog/Makefile                   |  1 +
 tools/libs/util/Makefile                      |  6 +-
 tools/libs/vchan/Makefile                     |  3 -
 tools/misc/Makefile                           |  1 -
 tools/tests/cpu-policy/Makefile               |  2 +-
 tools/tests/depriv/Makefile                   |  2 +-
 tools/tests/resource/Makefile                 |  1 -
 tools/tests/tsx/Makefile                      |  1 -
 tools/tests/xenstore/Makefile                 |  1 -
 tools/xcutils/Makefile                        |  2 -
 tools/xenmon/Makefile                         |  1 -
 tools/xenpaging/Makefile                      | 25 ++++---
 tools/xenpmd/Makefile                         |  1 -
 tools/xentop/Makefile                         | 23 +++---
 tools/xentrace/Makefile                       | 29 +++-----
 tools/xl/Makefile                             |  2 +-
 tools/Rules.mk                                | 55 ++++++++++----
 tools/debugger/gdbsx/Rules.mk                 |  2 +-
 tools/firmware/Rules.mk                       |  2 -
 tools/libfsimage/Rules.mk                     | 26 +++----
 tools/libfsimage/common.mk                    | 11 +++
 tools/libs/libs.mk                            | 48 +++++++------
 tools/libs/light/libxl_x86_acpi.c             |  2 +-
 tools/ocaml/common.make                       |  2 +-
 .gitignore                                    | 38 ----------
 config/Tools.mk.in                            |  1 +
 scripts/git-checkout.sh                       |  4 +-
 tools/configure                               | 29 +++++++-
 .../fuzz/x86_instruction_emulator/.gitignore  |  7 ++
 tools/golang/xenlight/gengotypes.py           | 10 ++-
 tools/hotplug/common/hotplugpath.sh.in        | 16 +++++
 tools/libs/.gitignore                         |  2 +
 tools/libs/light/gentypes.py                  |  9 ++-
 tools/xenstore/Makefile.common                |  1 -
 75 files changed, 371 insertions(+), 442 deletions(-)
 create mode 100644 tools/libfsimage/common.mk
 create mode 100644 tools/fuzz/x86_instruction_emulator/.gitignore
 create mode 100644 tools/hotplug/common/hotplugpath.sh.in

Comments

Christian Lindig Aug. 12, 2022, 7:53 a.m. UTC | #1 
On 11 Aug 2022, at 17:48, Anthony PERARD <anthony.perard@citrix.com<mailto:anthony.perard@citrix.com>> wrote:

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.toolstack-build-system-v4

Changes in v4:
- several new patches
- some changes to other patches listed in their changelogs
[..]
Anthony PERARD (32):
 tools/debugger/gdbsx: Fix and cleanup makefiles
 tools/firmware/hvmloader: rework Makefile
 tools/fuzz/x86_instruction_emulator: rework makefile
 tools/hotplug: cleanup Makefiles
 tools/libfsimage: Cleanup makefiles
 tools/xenpaging: Rework makefile
 tools/xentop: rework makefile
 tools/xentrace: rework Makefile
 .gitignore: Cleanup ignores of tools/libs/*/{headers.chk,*.pc}
 tools/libs/util: cleanup Makefile
 tools/flask/utils: list build targets in $(TARGETS)
 libs/libs.mk: Rename $(LIB) to $(TARGETS)
 libs/libs.mk: Remove the need for $(PKG_CONFIG_INST)
 libs/libs.mk: Rework target headers.chk dependencies
 tools: Introduce $(xenlibs-rpath,..) to replace $(SHDEPS_lib*)
 tools: Introduce $(xenlibs-ldlibs, ) macro
 tools: Introduce $(xenlibs-ldflags, ) macro
 tools/helper: Cleanup Makefile
 tools/console: Use $(xenlibs-ldlibs,)
 tools: Add -Werror by default to all tools/
 tools: Remove -Werror everywhere else
 tools/hotplug: Generate "hotplugpath.sh" with configure
 libs/light/gentypes.py: allow to generate headers in subdirectory
 git-checkout.sh: handle running git-checkout from a different
   directory
 libs: Avoid exposing -Wl,--version-script to other built library
 libs: Fix auto-generation of version-script for unstable libs
 tools/include: Rework Makefile
 libs/light: Rework acpi table build targets
 libs/light: Rework generation of include/_libxl_*.h
 libs/light: Rework targets prerequisites
 libs/light: Makefile cleanup
 tools/golang/xenlight: Rework gengotypes.py and generation of *.gen.go

Acked-by: Christian Lindig <christian.lindig@citrix.com<mailto:christian.lindig@citrix.com>>
Jan Beulich Aug. 15, 2022, 7:01 a.m. UTC | #2 
On 11.08.2022 18:48, Anthony PERARD wrote:
> Patch series available in this git branch:
> https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.toolstack-build-system-v4
> 
> Changes in v4:
> - several new patches
> - some changes to other patches listed in their changelogs

I've committed a non-contiguous initial part of this series, omitting
patches where acks still look to be missing, and hence only up to a
point where I could be reasonably certain that the skipped patches
won't interact with later ones. If you can indicate which further
sufficiently reviewed/acked parts of the series are independent of
earlier parts which aren't fully ready yet, I'd be happy to throw in
more.

Jan
Anthony PERARD Aug. 19, 2022, 1:40 p.m. UTC | #3 
On Mon, Aug 15, 2022 at 09:01:12AM +0200, Jan Beulich wrote:
> I've committed a non-contiguous initial part of this series, omitting
> patches where acks still look to be missing, and hence only up to a
> point where I could be reasonably certain that the skipped patches
> won't interact with later ones. If you can indicate which further
> sufficiently reviewed/acked parts of the series are independent of
> earlier parts which aren't fully ready yet, I'd be happy to throw in
> more.

Hi Jan,

Thanks. I think the following are ready, they don't depend on
uncommitted patches:

15: tools: Introduce $(xenlibs-rpath,..) to replace $(SHDEPS_lib*)
16: tools: Introduce $(xenlibs-ldlibs, ) macro
18: tools/helper: Cleanup Makefile
19: tools/console: Use $(xenlibs-ldlibs,)


Also, while "tools: Add -Werror by default to all tools/" seems ready,
it's probably better to have committed at the same time as the follow-up
"tools: Remove -Werror everywhere else".

Cheers,