backports: speed up building with bash

Commit Message

Johannes Berg Dec. 3, 2019, 3:06 p.m. UTC

From: Johannes Berg <[e-mail]>

When building with /bin/sh -> bash, things are SUPER slow (at least
for me), because bash takes a LOOONG time to look at the environment
variables (and we typically have around 6k), adding ~300ms to each
bash invocation for me.

The reason we export them is that we need them in all of the sub-
makes, and those only read auto.conf, which we can't change.

Work around this by overriding 'make' itself, and using --eval to
read *our* .config file into each make that gets called. This way,
the variables are present in all make invocations in the same way
as they would be through the environment, but don't get passed to
shell invocations.

Signed-off-by: Johannes Berg <[e-mail]>
---
 backport/Makefile.build | 3 ++-
 backport/scripts/make   | 3 +++
 2 files changed, 5 insertions(+), 1 deletion(-)
 create mode 100755 backport/scripts/make

Comments

Johannes Berg Dec. 3, 2019, 3:15 p.m. UTC | #1

On Tue, 2019-12-03 at 16:06 +0100, Johannes Berg wrote:
> From: Johannes Berg <[e-mail]>
> 
> When building with /bin/sh -> bash, things are SUPER slow (at least
> for me), because bash takes a LOOONG time to look at the environment
> variables (and we typically have around 6k), adding ~300ms to each
> bash invocation for me.

I think this was under-estimating it a bit - it's more like 1.3 seconds?

Anyway ...

johannes

--
To unsubscribe from this list: send the line "unsubscribe backports" in

Johannes Berg Dec. 3, 2019, 3:18 p.m. UTC | #2

On Tue, 2019-12-03 at 16:06 +0100, Johannes Berg wrote:
> From: Johannes Berg <[e-mail]>
> 
> When building with /bin/sh -> bash, things are SUPER slow (at least
> for me), because bash takes a LOOONG time to look at the environment
> variables (and we typically have around 6k), adding ~300ms to each
> bash invocation for me.

The real results are like this:

Completely clean build, after git clean -fdxq, but I have ccache primed
so all the stuff is build overhead - gcc is ccache and just returns
directly from the cache.

without the patch:
real	9m17.180s
user	24m38.063s
sys	0m32.138s


with the patch:
real	0m18.157s
user	0m24.553s
sys	0m13.342s


I'll let those results speak for themselves ...

johannes

--
To unsubscribe from this list: send the line "unsubscribe backports" in

Johannes Berg Dec. 3, 2019, 3:21 p.m. UTC | #3

On Tue, 2019-12-03 at 16:18 +0100, Johannes Berg wrote:
> On Tue, 2019-12-03 at 16:06 +0100, Johannes Berg wrote:
> > From: Johannes Berg <[e-mail]>
> > 
> > When building with /bin/sh -> bash, things are SUPER slow (at least
> > for me), because bash takes a LOOONG time to look at the environment
> > variables (and we typically have around 6k), adding ~300ms to each
> > bash invocation for me.
> 
> The real results are like this:
> 
> Completely clean build, after git clean -fdxq, but I have ccache primed
> so all the stuff is build overhead - gcc is ccache and just returns
> directly from the cache.
> 
> without the patch:
> real	9m17.180s
> user	24m38.063s
> sys	0m32.138s
> 
> 
> with the patch:
> real	0m18.157s
> user	0m24.553s
> sys	0m13.342s

That was with bash, with dash:

without the patch:
real	0m33.018s
user	0m56.248s
sys	0m22.898s

with the patch:
real	0m18.113s
user	0m18.036s
sys	0m12.903s

johannes

--
To unsubscribe from this list: send the line "unsubscribe backports" in

Patch

diff --git a/backport/Makefile.build b/backport/Makefile.build
index a848b37ed64b..7c479ace9f4a 100644
--- a/backport/Makefile.build
+++ b/backport/Makefile.build
@@ -1,6 +1,7 @@ 
--include .config
 export
 
+export MAKE=$(BACKPORT_DIR)/scripts/make
+
 .PHONY: modules
 modules:
 	@$(MAKE) -C $(KLIB_BUILD) M=$(BACKPORT_DIR) modules
diff --git a/backport/scripts/make b/backport/scripts/make
new file mode 100755
index 000000000000..f5330a3730c8
--- /dev/null
+++ b/backport/scripts/make
@@ -0,0 +1,3 @@ 
+#!/bin/sh
+
+exec make --eval '-include $(BACKPORT_DIR)/.config' "$@"