[PATCHSET,RFC,0/7] Rewrite rsrc node handling

Message

Jens Axboe Oct. 26, 2024, 10:08 p.m. UTC

Hi,

Caveat - this series is very much an RFC. Not because I don't think the
idea is sound (and the way it should be done), but because the series
itself is not very clean. It should be split a bit more. So just look
at the final result, not as much the individual patches just yet.

For a full explanation of the goal of the series, see patch #3. tldr
is that our currently rsrc node handling can block freeing of resources
for an indeterminite amount of time, which is very unfortunate for
potentially long lived request. For example, networked workloads and
using fixed files, where a previously long lived socket has the full
resource tables of the entire ring pinned. That can lead to files being
held open for a very long time.

This series handles the resource nodes separately, so a request pins
just the resources it needs, and only for the duration of that request.
In doing so, it also unifies how these resources are tracked. As it
stands, the current kernel duplicates state across user_bufs and
buf_data, and ditto for the file_table and file_data. Not only is some
of it duplicated (like the node arrays), it also needs to alloc and
copy the tags that are potentially associated with the resource. With
the unification, state is only in one spot for each type of resource,
and tags are handled at registration time rather than needing to be
retained for the duration of the resource. As with cleaning up of
structures, it also shrinks io_ring_ctx by 64b (should be more, it
adds holes too in spots), and the actual resource node goes from
needing 48b and 16b of put info, to 40b.

Lightly tested - it passes the liburing test suite, and doesn't leak
any memory. And it removes a net of about 250 lines of code, as can
be seen from the diffstat below. In my opinion it's also easier to
follow.

Can also be found here:

https://git.kernel.dk/cgit/linux/log/?h=io_uring-rsrc

 include/linux/io_uring_types.h |  25 +-
 io_uring/cancel.c              |   4 +-
 io_uring/fdinfo.c              |  10 +-
 io_uring/filetable.c           |  71 ++--
 io_uring/filetable.h           |  26 +-
 io_uring/io_uring.c            |  51 +--
 io_uring/msg_ring.c            |   4 +-
 io_uring/net.c                 |  15 +-
 io_uring/notif.c               |   3 +-
 io_uring/opdef.c               |   2 +
 io_uring/register.c            |   3 +-
 io_uring/rsrc.c                | 586 +++++++++++----------------------
 io_uring/rsrc.h                |  97 +++---
 io_uring/rw.c                  |  12 +-
 io_uring/splice.c              |  42 ++-
 io_uring/splice.h              |   1 +
 io_uring/uring_cmd.c           |  16 +-
 17 files changed, 366 insertions(+), 602 deletions(-)