| Message ID | 20201012011139.GA82449@mattapan.m5p.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [SECOND,RESEND] tools/python: Pass linker to Python build process | expand |
On Sun, Oct 11, 2020 at 06:11:39PM -0700, Elliott Mitchell wrote: > Unexpectedly the environment variable which needs to be passed is > $LDSHARED and not $LD. Otherwise Python may find the build `ld` instead > of the host `ld`. > > Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects > it can load at runtime, not executables. > > This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS > to $LDFLAGS which breaks many linkers. > > Signed-off-by: Elliott Mitchell <ehem+xen@m5p.com> > --- > This is now the *third* time this has been sent to the list. Mark Pryor > has tested and confirms Python cross-building is working. There is one > wart left which I'm unsure of the best approach for. > > Having looked around a bit, I believe this is a Python 2/3 compatibility > issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or > $LD variable, whereas Python 3 does have this. Alas this is pointless > due to the above (unless you can cause distutils.py to do the final link > step separately). I think this is well-reasoned but I don't have time to figure out and verify the details. Marek, do you have any comment on this? > --- > tools/pygrub/Makefile | 9 +++++---- > tools/python/Makefile | 9 +++++---- > 2 files changed, 10 insertions(+), 8 deletions(-) > > diff --git a/tools/pygrub/Makefile b/tools/pygrub/Makefile > index 3063c4998f..37b2146214 100644 > --- a/tools/pygrub/Makefile > +++ b/tools/pygrub/Makefile > @@ -3,20 +3,21 @@ XEN_ROOT = $(CURDIR)/../.. > include $(XEN_ROOT)/tools/Rules.mk > > PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS) > -PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS) > +PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS) > INSTALL_LOG = build/installed_files.txt > > .PHONY: all > all: build > .PHONY: build > build: > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > > .PHONY: install > install: all > $(INSTALL_DIR) $(DESTDIR)/$(bindir) > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \ > - setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \ > + LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \ > + --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > --root="$(DESTDIR)" --install-scripts=$(LIBEXEC_BIN) --force > set -e; if [ $(bindir) != $(LIBEXEC_BIN) -a \ > "`readlink -f $(DESTDIR)/$(bindir)`" != \ > diff --git a/tools/python/Makefile b/tools/python/Makefile > index 8d22c03676..b675f5b4de 100644 > --- a/tools/python/Makefile > +++ b/tools/python/Makefile > @@ -5,19 +5,20 @@ include $(XEN_ROOT)/tools/Rules.mk > all: build > > PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS) > -PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS) > +PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS) > INSTALL_LOG = build/installed_files.txt > > .PHONY: build > build: > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py build > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > > .PHONY: install > install: > $(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN) > > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \ > - setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \ > + LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \ > + --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > --root="$(DESTDIR)" --force > > $(INSTALL_PYTHON_PROG) scripts/convert-legacy-stream $(DESTDIR)$(LIBEXEC_BIN) > -- > 2.20.1 > > > -- > (\___(\___(\______ --=> 8-) EHM <=-- ______/)___/)___/) > \BS ( | ehem+sigmsg@m5p.com PGP 87145445 | ) / > \_CS\ | _____ -O #include <stddisclaimer.h> O- _____ | / _/ > 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445 > >
On Sun, Oct 11, 2020 at 06:11:39PM -0700, Elliott Mitchell wrote: > Unexpectedly the environment variable which needs to be passed is > $LDSHARED and not $LD. Otherwise Python may find the build `ld` instead > of the host `ld`. > > Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects > it can load at runtime, not executables. > > This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS > to $LDFLAGS which breaks many linkers. > > Signed-off-by: Elliott Mitchell <ehem+xen@m5p.com> Acked-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > --- > This is now the *third* time this has been sent to the list. Mark Pryor > has tested and confirms Python cross-building is working. There is one > wart left which I'm unsure of the best approach for. > > Having looked around a bit, I believe this is a Python 2/3 compatibility > issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or > $LD variable, whereas Python 3 does have this. Alas this is pointless > due to the above (unless you can cause distutils.py to do the final link > step separately). > --- > tools/pygrub/Makefile | 9 +++++---- > tools/python/Makefile | 9 +++++---- > 2 files changed, 10 insertions(+), 8 deletions(-) > > diff --git a/tools/pygrub/Makefile b/tools/pygrub/Makefile > index 3063c4998f..37b2146214 100644 > --- a/tools/pygrub/Makefile > +++ b/tools/pygrub/Makefile > @@ -3,20 +3,21 @@ XEN_ROOT = $(CURDIR)/../.. > include $(XEN_ROOT)/tools/Rules.mk > > PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS) > -PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS) > +PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS) > INSTALL_LOG = build/installed_files.txt > > .PHONY: all > all: build > .PHONY: build > build: > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > > .PHONY: install > install: all > $(INSTALL_DIR) $(DESTDIR)/$(bindir) > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \ > - setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \ > + LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \ > + --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > --root="$(DESTDIR)" --install-scripts=$(LIBEXEC_BIN) --force > set -e; if [ $(bindir) != $(LIBEXEC_BIN) -a \ > "`readlink -f $(DESTDIR)/$(bindir)`" != \ > diff --git a/tools/python/Makefile b/tools/python/Makefile > index 8d22c03676..b675f5b4de 100644 > --- a/tools/python/Makefile > +++ b/tools/python/Makefile > @@ -5,19 +5,20 @@ include $(XEN_ROOT)/tools/Rules.mk > all: build > > PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS) > -PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS) > +PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS) > INSTALL_LOG = build/installed_files.txt > > .PHONY: build > build: > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py build > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > > .PHONY: install > install: > $(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN) > > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \ > - setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \ > + LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \ > + --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > --root="$(DESTDIR)" --force > > $(INSTALL_PYTHON_PROG) scripts/convert-legacy-stream $(DESTDIR)$(LIBEXEC_BIN)
On Tue, Oct 13, 2020 at 01:26:06PM +0000, Wei Liu wrote: > On Sun, Oct 11, 2020 at 06:11:39PM -0700, Elliott Mitchell wrote: > > Unexpectedly the environment variable which needs to be passed is > > $LDSHARED and not $LD. Otherwise Python may find the build `ld` instead > > of the host `ld`. > > > > Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects > > it can load at runtime, not executables. > > > > This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS > > to $LDFLAGS which breaks many linkers. > > > > Signed-off-by: Elliott Mitchell <ehem+xen@m5p.com> > > --- > > This is now the *third* time this has been sent to the list. Mark Pryor > > has tested and confirms Python cross-building is working. There is one > > wart left which I'm unsure of the best approach for. > > > > Having looked around a bit, I believe this is a Python 2/3 compatibility > > issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or > > $LD variable, whereas Python 3 does have this. Alas this is pointless > > due to the above (unless you can cause distutils.py to do the final link > > step separately). > > I think this is well-reasoned but I don't have time to figure out and > verify the details. Yes, it looks like distutils in Python 2 was even more limited than the one in Python 3. > Marek, do you have any comment on this? > > > --- > > tools/pygrub/Makefile | 9 +++++---- > > tools/python/Makefile | 9 +++++---- > > 2 files changed, 10 insertions(+), 8 deletions(-) > > > > diff --git a/tools/pygrub/Makefile b/tools/pygrub/Makefile > > index 3063c4998f..37b2146214 100644 > > --- a/tools/pygrub/Makefile > > +++ b/tools/pygrub/Makefile > > @@ -3,20 +3,21 @@ XEN_ROOT = $(CURDIR)/../.. > > include $(XEN_ROOT)/tools/Rules.mk > > > > PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS) > > -PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS) > > +PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS) > > INSTALL_LOG = build/installed_files.txt > > > > .PHONY: all > > all: build > > .PHONY: build > > build: > > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > > > > .PHONY: install > > install: all > > $(INSTALL_DIR) $(DESTDIR)/$(bindir) > > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \ > > - setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \ > > + LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \ > > + --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > > --root="$(DESTDIR)" --install-scripts=$(LIBEXEC_BIN) --force > > set -e; if [ $(bindir) != $(LIBEXEC_BIN) -a \ > > "`readlink -f $(DESTDIR)/$(bindir)`" != \ > > diff --git a/tools/python/Makefile b/tools/python/Makefile > > index 8d22c03676..b675f5b4de 100644 > > --- a/tools/python/Makefile > > +++ b/tools/python/Makefile > > @@ -5,19 +5,20 @@ include $(XEN_ROOT)/tools/Rules.mk > > all: build > > > > PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS) > > -PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS) > > +PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS) > > INSTALL_LOG = build/installed_files.txt > > > > .PHONY: build > > build: > > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py build > > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build > > > > .PHONY: install > > install: > > $(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN) > > > > - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \ > > - setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > > + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \ > > + LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \ > > + --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ > > --root="$(DESTDIR)" --force > > > > $(INSTALL_PYTHON_PROG) scripts/convert-legacy-stream $(DESTDIR)$(LIBEXEC_BIN) > > -- > > 2.20.1 > > > > > > -- > > (\___(\___(\______ --=> 8-) EHM <=-- ______/)___/)___/) > > \BS ( | ehem+sigmsg@m5p.com PGP 87145445 | ) / > > \_CS\ | _____ -O #include <stddisclaimer.h> O- _____ | / _/ > > 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445 > > > >
On Thu, Oct 15, 2020 at 03:02:59AM +0200, Marek Marczykowski-G??recki wrote: > On Tue, Oct 13, 2020 at 01:26:06PM +0000, Wei Liu wrote: > > On Sun, Oct 11, 2020 at 06:11:39PM -0700, Elliott Mitchell wrote: > > > Having looked around a bit, I believe this is a Python 2/3 compatibility > > > issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or > > > $LD variable, whereas Python 3 does have this. Alas this is pointless > > > due to the above (unless you can cause distutils.py to do the final link > > > step separately). > > > > I think this is well-reasoned but I don't have time to figure out and > > verify the details. > > Yes, it looks like distutils in Python 2 was even more limited than > the one in Python 3. Actually feels like two steps forward, one step back. Separate $LDSHARED, yet $CFLAGS is appended to $LDFLAGS during invocation. Architecture name is included the output Python extension filename, yet the only way to override requires writing Python code. I've got ideas to workaround this, but they're rather gross.
On Thu, Oct 15, 2020 at 03:01:48AM +0200, Marek Marczykowski-Górecki wrote: > On Sun, Oct 11, 2020 at 06:11:39PM -0700, Elliott Mitchell wrote: > > Unexpectedly the environment variable which needs to be passed is > > $LDSHARED and not $LD. Otherwise Python may find the build `ld` instead > > of the host `ld`. > > > > Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects > > it can load at runtime, not executables. > > > > This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS > > to $LDFLAGS which breaks many linkers. > > > > Signed-off-by: Elliott Mitchell <ehem+xen@m5p.com> > > Acked-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> Thanks. Applied. Wei.
diff --git a/tools/pygrub/Makefile b/tools/pygrub/Makefile index 3063c4998f..37b2146214 100644 --- a/tools/pygrub/Makefile +++ b/tools/pygrub/Makefile @@ -3,20 +3,21 @@ XEN_ROOT = $(CURDIR)/../.. include $(XEN_ROOT)/tools/Rules.mk PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS) -PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS) +PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS) INSTALL_LOG = build/installed_files.txt .PHONY: all all: build .PHONY: build build: - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build .PHONY: install install: all $(INSTALL_DIR) $(DESTDIR)/$(bindir) - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \ - setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \ + LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \ + --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ --root="$(DESTDIR)" --install-scripts=$(LIBEXEC_BIN) --force set -e; if [ $(bindir) != $(LIBEXEC_BIN) -a \ "`readlink -f $(DESTDIR)/$(bindir)`" != \ diff --git a/tools/python/Makefile b/tools/python/Makefile index 8d22c03676..b675f5b4de 100644 --- a/tools/python/Makefile +++ b/tools/python/Makefile @@ -5,19 +5,20 @@ include $(XEN_ROOT)/tools/Rules.mk all: build PY_CFLAGS = $(CFLAGS) $(PY_NOOPT_CFLAGS) -PY_LDFLAGS = $(LDFLAGS) $(APPEND_LDFLAGS) +PY_LDFLAGS = $(SHLIB_LDFLAGS) $(APPEND_LDFLAGS) INSTALL_LOG = build/installed_files.txt .PHONY: build build: - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py build + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py build .PHONY: install install: $(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN) - CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) \ - setup.py install --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ + CC="$(CC)" CFLAGS="$(PY_CFLAGS)" LDSHARED="$(CC)" \ + LDFLAGS="$(PY_LDFLAGS)" $(PYTHON) setup.py install \ + --record $(INSTALL_LOG) $(PYTHON_PREFIX_ARG) \ --root="$(DESTDIR)" --force $(INSTALL_PYTHON_PROG) scripts/convert-legacy-stream $(DESTDIR)$(LIBEXEC_BIN)
Unexpectedly the environment variable which needs to be passed is $LDSHARED and not $LD. Otherwise Python may find the build `ld` instead of the host `ld`. Replace $(LDFLAGS) with $(SHLIB_LDFLAGS) as Python needs shared objects it can load at runtime, not executables. This uses $(CC) instead of $(LD) since Python distutils appends $CFLAGS to $LDFLAGS which breaks many linkers. Signed-off-by: Elliott Mitchell <ehem+xen@m5p.com> --- This is now the *third* time this has been sent to the list. Mark Pryor has tested and confirms Python cross-building is working. There is one wart left which I'm unsure of the best approach for. Having looked around a bit, I believe this is a Python 2/3 compatibility issue. "distutils" for Python 2 likely lacked a separate $LDSHARED or $LD variable, whereas Python 3 does have this. Alas this is pointless due to the above (unless you can cause distutils.py to do the final link step separately). --- tools/pygrub/Makefile | 9 +++++---- tools/python/Makefile | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-)