mbox series

[v2,0/9] iOS and Apple Silicon host support

Message ID 20201019013928.72770-1-j@getutm.app (mailing list archive)
Headers show
Series iOS and Apple Silicon host support | expand

Message

Joelle van Dyne Oct. 19, 2020, 1:39 a.m. UTC
These set of changes brings QEMU TCG to iOS devices and future Apple Silicon
devices. They were originally developed last year and have been working in the
UTM app. Recently, we ported the changes to master, re-wrote a lot of the build
script changes for meson, and broke up the patches into more distinct units.

A summary of the changes:

* `CONFIG_IOS` and `CONFIG_IOS_JIT` defined when building for iOS and
  iOS specific changes (as well as unsupported code) are gated behind it.
* A new dependency, libucontext is added since iOS does not have native ucontext
  and broken support for sigaltstack. libucontext is available as a new option
  for coroutine backend.
* On stock iOS devices, there is a workaround for running JIT code without
  any special entitlement. It requires the JIT region to be mirror mapped with
  one region RW and another one RX. To support this style of JIT, TCG is changed
  to support writing to a different code_ptr. These changes are gated by the
  `CONFIG_IOS_JIT`.
* For (recent) jailbroken iOS devices as well as upcoming Apple Silicon devices,
  there are new rules for applications supporting JIT (with the proper
  entitlement). These rules are implemented as well.

Since v2:

* Changed getting mirror pointer from a macro to inline functions
* Split constification of TCG code pointers to separate patch
* Removed slirp updates (will send future patch once slirp changes are in)
* Removed shared library patch (will send future patch)

-j

osy (9):
  configure: option to disable host block devices
  configure: cross-compiling without cross_prefix
  qemu: add support for iOS host
  coroutine: add libucontext as external library
  tcg: add const hints for code pointers
  tcg: implement mirror mapped JIT for iOS
  tcg: mirror mapping RWX pages for iOS optional
  tcg: support JIT on Apple Silicon
  block: check availablity for preadv/pwritev on mac

 docs/devel/ios.rst           |  40 +++++++++
 configure                    | 104 ++++++++++++++++++++--
 meson.build                  |  32 ++++++-
 include/exec/exec-all.h      |  10 +++
 include/sysemu/tcg.h         |   2 +-
 include/tcg/tcg-apple-jit.h  |  85 ++++++++++++++++++
 include/tcg/tcg.h            |  28 +++++-
 tcg/aarch64/tcg-target.h     |  23 ++++-
 tcg/arm/tcg-target.h         |   9 +-
 tcg/i386/tcg-target.h        |  24 ++++-
 tcg/mips/tcg-target.h        |   8 +-
 tcg/ppc/tcg-target.h         |   8 +-
 tcg/riscv/tcg-target.h       |   9 +-
 tcg/s390/tcg-target.h        |  13 ++-
 tcg/sparc/tcg-target.h       |   8 +-
 tcg/tci/tcg-target.h         |   9 +-
 accel/tcg/cpu-exec-common.c  |   2 +
 accel/tcg/cpu-exec.c         |   9 +-
 accel/tcg/tcg-all.c          |  27 +++++-
 accel/tcg/translate-all.c    | 168 ++++++++++++++++++++++++++++++++---
 block.c                      |   2 +-
 block/file-posix.c           |  50 ++++++++---
 bsd-user/main.c              |   2 +-
 linux-user/main.c            |   2 +-
 net/slirp.c                  |  16 ++--
 qga/commands-posix.c         |   6 ++
 target/arm/arm-semi.c        |   2 +
 target/m68k/m68k-semi.c      |   2 +
 target/nios2/nios2-semi.c    |   2 +
 tcg/tcg.c                    |  64 ++++++++-----
 util/coroutine-ucontext.c    |   9 ++
 .gitmodules                  |   3 +
 libucontext                  |   1 +
 meson_options.txt            |   2 +
 qemu-options.hx              |  11 +++
 tcg/aarch64/tcg-target.c.inc |  48 ++++++----
 tcg/arm/tcg-target.c.inc     |  33 ++++---
 tcg/i386/tcg-target.c.inc    |  28 +++---
 tcg/mips/tcg-target.c.inc    |  64 +++++++------
 tcg/ppc/tcg-target.c.inc     |  55 +++++++-----
 tcg/riscv/tcg-target.c.inc   |  51 ++++++-----
 tcg/s390/tcg-target.c.inc    |  25 +++---
 tcg/sparc/tcg-target.c.inc   |  33 ++++---
 tcg/tcg-ldst.c.inc           |   2 +-
 tcg/tcg-pool.c.inc           |   9 +-
 tcg/tci/tcg-target.c.inc     |   8 +-
 tests/qtest/meson.build      |   7 +-
 47 files changed, 919 insertions(+), 236 deletions(-)
 create mode 100644 docs/devel/ios.rst
 create mode 100644 include/tcg/tcg-apple-jit.h
 create mode 160000 libucontext

Comments

Thomas Huth Oct. 19, 2020, 8:29 a.m. UTC | #1
On 19/10/2020 03.39, Joelle van Dyne wrote:
> These set of changes brings QEMU TCG to iOS devices and future Apple Silicon
> devices. They were originally developed last year and have been working in the
> UTM app. Recently, we ported the changes to master, re-wrote a lot of the build
> script changes for meson, and broke up the patches into more distinct units.
> 
> A summary of the changes:
> 
> * `CONFIG_IOS` and `CONFIG_IOS_JIT` defined when building for iOS and
>   iOS specific changes (as well as unsupported code) are gated behind it.
> * A new dependency, libucontext is added since iOS does not have native ucontext
>   and broken support for sigaltstack. libucontext is available as a new option
>   for coroutine backend.
> * On stock iOS devices, there is a workaround for running JIT code without
>   any special entitlement. It requires the JIT region to be mirror mapped with
>   one region RW and another one RX. To support this style of JIT, TCG is changed
>   to support writing to a different code_ptr. These changes are gated by the
>   `CONFIG_IOS_JIT`.
> * For (recent) jailbroken iOS devices as well as upcoming Apple Silicon devices,
>   there are new rules for applications supporting JIT (with the proper
>   entitlement). These rules are implemented as well.
> 
> Since v2:
> 
> * Changed getting mirror pointer from a macro to inline functions
> * Split constification of TCG code pointers to separate patch
> * Removed slirp updates (will send future patch once slirp changes are in)
> * Removed shared library patch (will send future patch)
> 
> -j
> 
> osy (9):
>   configure: option to disable host block devices
>   configure: cross-compiling without cross_prefix
>   qemu: add support for iOS host
>   coroutine: add libucontext as external library
>   tcg: add const hints for code pointers
>   tcg: implement mirror mapped JIT for iOS
>   tcg: mirror mapping RWX pages for iOS optional
>   tcg: support JIT on Apple Silicon
>   block: check availablity for preadv/pwritev on mac

Is there maybe also a way to compile-test this automatically in a macOS
container on Cirrus-CI or Travis, too? ... otherwise I'm a little bit afraid
that this support might bit-rot quite fast again, since most developers
don't have the corresponding setup...

 Thomas
Joelle van Dyne Oct. 26, 2020, 3:30 p.m. UTC | #2
On Mon, Oct 19, 2020 at 1:30 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 19/10/2020 03.39, Joelle van Dyne wrote:
> > These set of changes brings QEMU TCG to iOS devices and future Apple Silicon
> > devices. They were originally developed last year and have been working in the
> > UTM app. Recently, we ported the changes to master, re-wrote a lot of the build
> > script changes for meson, and broke up the patches into more distinct units.
> >
> > A summary of the changes:
> >
> > * `CONFIG_IOS` and `CONFIG_IOS_JIT` defined when building for iOS and
> >   iOS specific changes (as well as unsupported code) are gated behind it.
> > * A new dependency, libucontext is added since iOS does not have native ucontext
> >   and broken support for sigaltstack. libucontext is available as a new option
> >   for coroutine backend.
> > * On stock iOS devices, there is a workaround for running JIT code without
> >   any special entitlement. It requires the JIT region to be mirror mapped with
> >   one region RW and another one RX. To support this style of JIT, TCG is changed
> >   to support writing to a different code_ptr. These changes are gated by the
> >   `CONFIG_IOS_JIT`.
> > * For (recent) jailbroken iOS devices as well as upcoming Apple Silicon devices,
> >   there are new rules for applications supporting JIT (with the proper
> >   entitlement). These rules are implemented as well.
> >
> > Since v2:
> >
> > * Changed getting mirror pointer from a macro to inline functions
> > * Split constification of TCG code pointers to separate patch
> > * Removed slirp updates (will send future patch once slirp changes are in)
> > * Removed shared library patch (will send future patch)
> >
> > -j
> >
> > osy (9):
> >   configure: option to disable host block devices
> >   configure: cross-compiling without cross_prefix
> >   qemu: add support for iOS host
> >   coroutine: add libucontext as external library
> >   tcg: add const hints for code pointers
> >   tcg: implement mirror mapped JIT for iOS
> >   tcg: mirror mapping RWX pages for iOS optional
> >   tcg: support JIT on Apple Silicon
> >   block: check availablity for preadv/pwritev on mac
>
> Is there maybe also a way to compile-test this automatically in a macOS
> container on Cirrus-CI or Travis, too? ... otherwise I'm a little bit afraid
> that this support might bit-rot quite fast again, since most developers
> don't have the corresponding setup...
That's a tough one. In theory we can use the cirrus-ci environment to
build for iOS but there's no easy way to get the dependencies. For
UTM, we have this complicated script
https://github.com/utmapp/UTM/blob/master/scripts/build_dependencies.sh
which downloads and builds all the dependencies from source. Adding
something like that to QEMU would tie it to specific versions of those
libraries as well as the download link. Not sure if that's desirable.

I am currently refactoring the code so the bigger changes (mirror JIT)
are separate features that doesn't depend on iOS. Other features work
on Darwin so they can be tested in the macOS builds. But there will be
iOS specific code that I don't think would be easy to CI and that's a
general limitation of open source projects on iOS.

-j

>
>  Thomas
>