| Message ID | 20241025122247.3709133-5-ming.lei@redhat.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | io_uring: support sqe group and leased group kbuf | expand |
On 10/25/24 6:22 AM, Ming Lei wrote: > SQE group is defined as one chain of SQEs starting with the first SQE that > has IOSQE_SQE_GROUP set, and ending with the first subsequent SQE that > doesn't have it set, and it is similar with chain of linked SQEs. > > Not like linked SQEs, each sqe is issued after the previous one is > completed. All SQEs in one group can be submitted in parallel. To simplify > the implementation from beginning, all members are queued after the leader > is completed, however, this way may be changed and leader and members may > be issued concurrently in future. > > The 1st SQE is group leader, and the other SQEs are group member. The whole > group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and > the two flags can't be set for group members. For the sake of > simplicity, IORING_OP_LINK_TIMEOUT is disallowed for SQE group now. > > When the group is in one link chain, this group isn't submitted until the > previous SQE or group is completed. And the following SQE or group can't > be started if this group isn't completed. Failure from any group member will > fail the group leader, then the link chain can be terminated. > > When IOSQE_IO_DRAIN is set for group leader, all requests in this group and > previous requests submitted are drained. Given IOSQE_IO_DRAIN can be set for > group leader only, we respect IO_DRAIN by always completing group leader as > the last one in the group. Meantime it is natural to post leader's CQE > as the last one from application viewpoint. > > Working together with IOSQE_IO_LINK, SQE group provides flexible way to > support N:M dependency, such as: > > - group A is chained with group B together > - group A has N SQEs > - group B has M SQEs > > then M SQEs in group B depend on N SQEs in group A. > > N:M dependency can support some interesting use cases in efficient way: > > 1) read from multiple files, then write the read data into single file > > 2) read from single file, and write the read data into multiple files > > 3) write same data into multiple files, and read data from multiple files and > compare if correct data is written > > Also IOSQE_SQE_GROUP takes the last bit in sqe->flags, but we still can > extend sqe->flags with io_uring context flag, such as use __pad3 for > non-uring_cmd OPs and part of uring_cmd_flags for uring_cmd OP. Since it's taking the last flag, maybe a better idea to have the last flag mean "more flags in (for example) __pad3" and put the new flag there? Not sure you mean in terms of "io_uring context flag", would it be an enter flag? Ring required to be setup with a certain flag? Neither of those seem super encouraging, imho. Apart from that, just a few minor nits below. > +void io_fail_group_members(struct io_kiocb *req) > +{ > + struct io_kiocb *member = req->grp_link; > + > + while (member) { > + struct io_kiocb *next = member->grp_link; > + > + if (!(member->flags & REQ_F_FAIL)) { > + req_set_fail(member); > + io_req_set_res(member, -ECANCELED, 0); > + } > + member = next; > + } > +} > + > +static void io_queue_group_members(struct io_kiocb *req) > +{ > + struct io_kiocb *member = req->grp_link; > + > + if (!member) > + return; > + > + req->grp_link = NULL; > + while (member) { > + struct io_kiocb *next = member->grp_link; > + > + member->grp_leader = req; > + if (unlikely(member->flags & REQ_F_FAIL)) { > + io_req_task_queue_fail(member, member->cqe.res); > + } else if (unlikely(req->flags & REQ_F_FAIL)) { > + io_req_task_queue_fail(member, -ECANCELED); > + } else { > + io_req_task_queue(member); > + } > + member = next; > + } > +} Was going to say don't check for !member, you have the while loop. Which is what you do in the helper above. You can also drop the parens in this one. > +static enum group_mem io_prep_free_group_req(struct io_kiocb *req, > + struct io_kiocb **leader) > +{ > + /* > + * Group completion is done, so clear the flag for avoiding double > + * handling in case of io-wq > + */ > + req->flags &= ~REQ_F_SQE_GROUP; > + > + if (req_is_group_leader(req)) { > + /* Queue members now */ > + if (req->grp_link) > + io_queue_group_members(req); > + return GROUP_LEADER; > + } else { > + if (!req_is_last_group_member(req)) > + return GROUP_OTHER_MEMBER; > + > + /* > + * Prepare for freeing leader which can only be found from > + * the last member > + */ > + *leader = req->grp_leader; > + (*leader)->flags &= ~REQ_F_SQE_GROUP_LEADER; > + req->grp_leader = NULL; > + return GROUP_LAST_MEMBER; > + } > +} Just drop the second indentation here. > @@ -927,7 +1051,8 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) > * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires > * the submitter task context, IOPOLL protects with uring_lock. > */ > - if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL)) { > + if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL) || > + (req->flags & REQ_F_SQE_GROUP)) { > req->io_task_work.func = io_req_task_complete; > io_req_task_work_add(req); > return; Minor detail, but might be nice with a REQ_F_* flag for this in the future. > @@ -1450,8 +1596,16 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) > struct io_kiocb *req = container_of(node, struct io_kiocb, > comp_list); > > - if (!(req->flags & REQ_F_CQE_SKIP)) > - io_req_commit_cqe(ctx, req); > + if (unlikely(req->flags & (REQ_F_CQE_SKIP | REQ_F_SQE_GROUP))) { > + if (req->flags & REQ_F_SQE_GROUP) { > + io_complete_group_req(req); > + continue; > + } > + > + if (req->flags & REQ_F_CQE_SKIP) > + continue; > + } > + io_req_commit_cqe(ctx, req); > } > __io_cq_unlock_post(ctx); > > @@ -1661,8 +1815,12 @@ static u32 io_get_sequence(struct io_kiocb *req) > struct io_kiocb *cur; > > /* need original cached_sq_head, but it was increased for each req */ > - io_for_each_link(cur, req) > - seq--; > + io_for_each_link(cur, req) { > + if (req_is_group_leader(cur)) > + seq -= cur->grp_refs; > + else > + seq--; > + } > return seq; > } > > @@ -2124,6 +2282,67 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, > return def->prep(req, sqe); > } > > +static struct io_kiocb *io_group_sqe(struct io_submit_link *group, > + struct io_kiocb *req) > +{ > + /* > + * Group chain is similar with link chain: starts with 1st sqe with > + * REQ_F_SQE_GROUP, and ends with the 1st sqe without REQ_F_SQE_GROUP > + */ > + if (group->head) { > + struct io_kiocb *lead = group->head; > + > + /* > + * Members can't be in link chain, can't be drained, but > + * the whole group can be linked or drained by setting > + * flags on group leader. > + * > + * IOSQE_CQE_SKIP_SUCCESS can't be set for member > + * for the sake of simplicity > + */ > + if (req->flags & (IO_REQ_LINK_FLAGS | REQ_F_IO_DRAIN | > + REQ_F_CQE_SKIP)) > + req_fail_link_node(lead, -EINVAL); > + > + lead->grp_refs += 1; > + group->last->grp_link = req; > + group->last = req; > + > + if (req->flags & REQ_F_SQE_GROUP) > + return NULL; > + > + req->grp_link = NULL; > + req->flags |= REQ_F_SQE_GROUP; > + group->head = NULL; > + > + return lead; > + } else { > + if (WARN_ON_ONCE(!(req->flags & REQ_F_SQE_GROUP))) > + return req; > + group->head = req; > + group->last = req; > + req->grp_refs = 1; > + req->flags |= REQ_F_SQE_GROUP_LEADER; > + return NULL; > + } > +} Same here, drop the 2nd indentation. > diff --git a/io_uring/timeout.c b/io_uring/timeout.c > index 9973876d91b0..ed6c74f1a475 100644 > --- a/io_uring/timeout.c > +++ b/io_uring/timeout.c > @@ -149,6 +149,8 @@ static void io_req_tw_fail_links(struct io_kiocb *link, struct io_tw_state *ts) > res = link->cqe.res; > link->link = NULL; > io_req_set_res(link, res, 0); > + if (req_is_group_leader(link)) > + io_fail_group_members(link); > io_req_task_complete(link, ts); > link = nxt; > } > @@ -543,6 +545,10 @@ static int __io_timeout_prep(struct io_kiocb *req, > if (is_timeout_link) { > struct io_submit_link *link = &req->ctx->submit_state.link; > > + /* so far disallow IO group link timeout */ > + if (req->ctx->submit_state.group.head) > + return -EINVAL; > + For now, disallow IO group linked timeout
On Mon, Oct 28, 2024 at 06:12:34PM -0600, Jens Axboe wrote: > On 10/25/24 6:22 AM, Ming Lei wrote: > > SQE group is defined as one chain of SQEs starting with the first SQE that > > has IOSQE_SQE_GROUP set, and ending with the first subsequent SQE that > > doesn't have it set, and it is similar with chain of linked SQEs. > > > > Not like linked SQEs, each sqe is issued after the previous one is > > completed. All SQEs in one group can be submitted in parallel. To simplify > > the implementation from beginning, all members are queued after the leader > > is completed, however, this way may be changed and leader and members may > > be issued concurrently in future. > > > > The 1st SQE is group leader, and the other SQEs are group member. The whole > > group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and > > the two flags can't be set for group members. For the sake of > > simplicity, IORING_OP_LINK_TIMEOUT is disallowed for SQE group now. > > > > When the group is in one link chain, this group isn't submitted until the > > previous SQE or group is completed. And the following SQE or group can't > > be started if this group isn't completed. Failure from any group member will > > fail the group leader, then the link chain can be terminated. > > > > When IOSQE_IO_DRAIN is set for group leader, all requests in this group and > > previous requests submitted are drained. Given IOSQE_IO_DRAIN can be set for > > group leader only, we respect IO_DRAIN by always completing group leader as > > the last one in the group. Meantime it is natural to post leader's CQE > > as the last one from application viewpoint. > > > > Working together with IOSQE_IO_LINK, SQE group provides flexible way to > > support N:M dependency, such as: > > > > - group A is chained with group B together > > - group A has N SQEs > > - group B has M SQEs > > > > then M SQEs in group B depend on N SQEs in group A. > > > > N:M dependency can support some interesting use cases in efficient way: > > > > 1) read from multiple files, then write the read data into single file > > > > 2) read from single file, and write the read data into multiple files > > > > 3) write same data into multiple files, and read data from multiple files and > > compare if correct data is written > > > > Also IOSQE_SQE_GROUP takes the last bit in sqe->flags, but we still can > > extend sqe->flags with io_uring context flag, such as use __pad3 for > > non-uring_cmd OPs and part of uring_cmd_flags for uring_cmd OP. > > Since it's taking the last flag, maybe a better idea to have the last > flag mean "more flags in (for example) __pad3" and put the new flag > there? Not sure you mean in terms of "io_uring context flag", would it > be an enter flag? Ring required to be setup with a certain flag? Neither > of those seem super encouraging, imho. I meant: If "more flags in __pad3" is enabled in future we may claim it as one feature to userspace, such as IORING_FEAT_EXT_FLAG. Will improve the above commit log. > > Apart from that, just a few minor nits below. > > > +void io_fail_group_members(struct io_kiocb *req) > > +{ > > + struct io_kiocb *member = req->grp_link; > > + > > + while (member) { > > + struct io_kiocb *next = member->grp_link; > > + > > + if (!(member->flags & REQ_F_FAIL)) { > > + req_set_fail(member); > > + io_req_set_res(member, -ECANCELED, 0); > > + } > > + member = next; > > + } > > +} > > + > > +static void io_queue_group_members(struct io_kiocb *req) > > +{ > > + struct io_kiocb *member = req->grp_link; > > + > > + if (!member) > > + return; > > + > > + req->grp_link = NULL; > > + while (member) { > > + struct io_kiocb *next = member->grp_link; > > + > > + member->grp_leader = req; > > + if (unlikely(member->flags & REQ_F_FAIL)) { > > + io_req_task_queue_fail(member, member->cqe.res); > > + } else if (unlikely(req->flags & REQ_F_FAIL)) { > > + io_req_task_queue_fail(member, -ECANCELED); > > + } else { > > + io_req_task_queue(member); > > + } > > + member = next; > > + } > > +} > > Was going to say don't check for !member, you have the while loop. Which > is what you do in the helper above. You can also drop the parens in this > one. OK, will remove the check `!member` and all parens. > > > +static enum group_mem io_prep_free_group_req(struct io_kiocb *req, > > + struct io_kiocb **leader) > > +{ > > + /* > > + * Group completion is done, so clear the flag for avoiding double > > + * handling in case of io-wq > > + */ > > + req->flags &= ~REQ_F_SQE_GROUP; > > + > > + if (req_is_group_leader(req)) { > > + /* Queue members now */ > > + if (req->grp_link) > > + io_queue_group_members(req); > > + return GROUP_LEADER; > > + } else { > > + if (!req_is_last_group_member(req)) > > + return GROUP_OTHER_MEMBER; > > + > > + /* > > + * Prepare for freeing leader which can only be found from > > + * the last member > > + */ > > + *leader = req->grp_leader; > > + (*leader)->flags &= ~REQ_F_SQE_GROUP_LEADER; > > + req->grp_leader = NULL; > > + return GROUP_LAST_MEMBER; > > + } > > +} > > Just drop the second indentation here. OK. > > > @@ -927,7 +1051,8 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) > > * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires > > * the submitter task context, IOPOLL protects with uring_lock. > > */ > > - if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL)) { > > + if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL) || > > + (req->flags & REQ_F_SQE_GROUP)) { > > req->io_task_work.func = io_req_task_complete; > > io_req_task_work_add(req); > > return; > > Minor detail, but might be nice with a REQ_F_* flag for this in the > future. > > > @@ -1450,8 +1596,16 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) > > struct io_kiocb *req = container_of(node, struct io_kiocb, > > comp_list); > > > > - if (!(req->flags & REQ_F_CQE_SKIP)) > > - io_req_commit_cqe(ctx, req); > > + if (unlikely(req->flags & (REQ_F_CQE_SKIP | REQ_F_SQE_GROUP))) { > > + if (req->flags & REQ_F_SQE_GROUP) { > > + io_complete_group_req(req); > > + continue; > > + } > > + > > + if (req->flags & REQ_F_CQE_SKIP) > > + continue; > > + } > > + io_req_commit_cqe(ctx, req); > > } > > __io_cq_unlock_post(ctx); > > > > @@ -1661,8 +1815,12 @@ static u32 io_get_sequence(struct io_kiocb *req) > > struct io_kiocb *cur; > > > > /* need original cached_sq_head, but it was increased for each req */ > > - io_for_each_link(cur, req) > > - seq--; > > + io_for_each_link(cur, req) { > > + if (req_is_group_leader(cur)) > > + seq -= cur->grp_refs; > > + else > > + seq--; > > + } > > return seq; > > } > > > > @@ -2124,6 +2282,67 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, > > return def->prep(req, sqe); > > } > > > > +static struct io_kiocb *io_group_sqe(struct io_submit_link *group, > > + struct io_kiocb *req) > > +{ > > + /* > > + * Group chain is similar with link chain: starts with 1st sqe with > > + * REQ_F_SQE_GROUP, and ends with the 1st sqe without REQ_F_SQE_GROUP > > + */ > > + if (group->head) { > > + struct io_kiocb *lead = group->head; > > + > > + /* > > + * Members can't be in link chain, can't be drained, but > > + * the whole group can be linked or drained by setting > > + * flags on group leader. > > + * > > + * IOSQE_CQE_SKIP_SUCCESS can't be set for member > > + * for the sake of simplicity > > + */ > > + if (req->flags & (IO_REQ_LINK_FLAGS | REQ_F_IO_DRAIN | > > + REQ_F_CQE_SKIP)) > > + req_fail_link_node(lead, -EINVAL); > > + > > + lead->grp_refs += 1; > > + group->last->grp_link = req; > > + group->last = req; > > + > > + if (req->flags & REQ_F_SQE_GROUP) > > + return NULL; > > + > > + req->grp_link = NULL; > > + req->flags |= REQ_F_SQE_GROUP; > > + group->head = NULL; > > + > > + return lead; > > + } else { > > + if (WARN_ON_ONCE(!(req->flags & REQ_F_SQE_GROUP))) > > + return req; > > + group->head = req; > > + group->last = req; > > + req->grp_refs = 1; > > + req->flags |= REQ_F_SQE_GROUP_LEADER; > > + return NULL; > > + } > > +} > > Same here, drop the 2nd indentation. OK. > > > diff --git a/io_uring/timeout.c b/io_uring/timeout.c > > index 9973876d91b0..ed6c74f1a475 100644 > > --- a/io_uring/timeout.c > > +++ b/io_uring/timeout.c > > @@ -149,6 +149,8 @@ static void io_req_tw_fail_links(struct io_kiocb *link, struct io_tw_state *ts) > > res = link->cqe.res; > > link->link = NULL; > > io_req_set_res(link, res, 0); > > + if (req_is_group_leader(link)) > > + io_fail_group_members(link); > > io_req_task_complete(link, ts); > > link = nxt; > > } > > @@ -543,6 +545,10 @@ static int __io_timeout_prep(struct io_kiocb *req, > > if (is_timeout_link) { > > struct io_submit_link *link = &req->ctx->submit_state.link; > > > > + /* so far disallow IO group link timeout */ > > + if (req->ctx->submit_state.group.head) > > + return -EINVAL; > > + > > For now, disallow IO group linked timeout OK. thanks, Ming
On 10/29/24 01:50, Ming Lei wrote: > On Mon, Oct 28, 2024 at 06:12:34PM -0600, Jens Axboe wrote: >> On 10/25/24 6:22 AM, Ming Lei wrote: >>> SQE group is defined as one chain of SQEs starting with the first SQE that >>> has IOSQE_SQE_GROUP set, and ending with the first subsequent SQE that >>> doesn't have it set, and it is similar with chain of linked SQEs. >>> >>> Not like linked SQEs, each sqe is issued after the previous one is >>> completed. All SQEs in one group can be submitted in parallel. To simplify >>> the implementation from beginning, all members are queued after the leader >>> is completed, however, this way may be changed and leader and members may >>> be issued concurrently in future. >>> >>> The 1st SQE is group leader, and the other SQEs are group member. The whole >>> group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and >>> the two flags can't be set for group members. For the sake of >>> simplicity, IORING_OP_LINK_TIMEOUT is disallowed for SQE group now. >>> >>> When the group is in one link chain, this group isn't submitted until the >>> previous SQE or group is completed. And the following SQE or group can't >>> be started if this group isn't completed. Failure from any group member will >>> fail the group leader, then the link chain can be terminated. >>> >>> When IOSQE_IO_DRAIN is set for group leader, all requests in this group and >>> previous requests submitted are drained. Given IOSQE_IO_DRAIN can be set for >>> group leader only, we respect IO_DRAIN by always completing group leader as >>> the last one in the group. Meantime it is natural to post leader's CQE >>> as the last one from application viewpoint. >>> >>> Working together with IOSQE_IO_LINK, SQE group provides flexible way to >>> support N:M dependency, such as: >>> >>> - group A is chained with group B together >>> - group A has N SQEs >>> - group B has M SQEs >>> >>> then M SQEs in group B depend on N SQEs in group A. >>> >>> N:M dependency can support some interesting use cases in efficient way: >>> >>> 1) read from multiple files, then write the read data into single file >>> >>> 2) read from single file, and write the read data into multiple files >>> >>> 3) write same data into multiple files, and read data from multiple files and >>> compare if correct data is written >>> >>> Also IOSQE_SQE_GROUP takes the last bit in sqe->flags, but we still can >>> extend sqe->flags with io_uring context flag, such as use __pad3 for >>> non-uring_cmd OPs and part of uring_cmd_flags for uring_cmd OP. >> >> Since it's taking the last flag, maybe a better idea to have the last >> flag mean "more flags in (for example) __pad3" and put the new flag >> there? Not sure you mean in terms of "io_uring context flag", would it >> be an enter flag? Ring required to be setup with a certain flag? Neither >> of those seem super encouraging, imho. > > I meant: > > If "more flags in __pad3" is enabled in future we may claim it as one > feature to userspace, such as IORING_FEAT_EXT_FLAG. > > Will improve the above commit log. And we can't take it in either case. The field is in a union, and other opcodes use that part of the SQE. Enabling a generic feature for a subset of requests only is not a good idea.
On 10/25/24 6:22 AM, Ming Lei wrote: > SQE group is defined as one chain of SQEs starting with the first SQE that > has IOSQE_SQE_GROUP set, and ending with the first subsequent SQE that > doesn't have it set, and it is similar with chain of linked SQEs. > > Not like linked SQEs, each sqe is issued after the previous one is > completed. All SQEs in one group can be submitted in parallel. To simplify > the implementation from beginning, all members are queued after the leader > is completed, however, this way may be changed and leader and members may > be issued concurrently in future. > > The 1st SQE is group leader, and the other SQEs are group member. The whole > group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and > the two flags can't be set for group members. For the sake of > simplicity, IORING_OP_LINK_TIMEOUT is disallowed for SQE group now. > > When the group is in one link chain, this group isn't submitted until the > previous SQE or group is completed. And the following SQE or group can't > be started if this group isn't completed. Failure from any group member will > fail the group leader, then the link chain can be terminated. > > When IOSQE_IO_DRAIN is set for group leader, all requests in this group and > previous requests submitted are drained. Given IOSQE_IO_DRAIN can be set for > group leader only, we respect IO_DRAIN by always completing group leader as > the last one in the group. Meantime it is natural to post leader's CQE > as the last one from application viewpoint. > > Working together with IOSQE_IO_LINK, SQE group provides flexible way to > support N:M dependency, such as: > > - group A is chained with group B together > - group A has N SQEs > - group B has M SQEs > > then M SQEs in group B depend on N SQEs in group A. > > N:M dependency can support some interesting use cases in efficient way: > > 1) read from multiple files, then write the read data into single file > > 2) read from single file, and write the read data into multiple files > > 3) write same data into multiple files, and read data from multiple files and > compare if correct data is written > > Also IOSQE_SQE_GROUP takes the last bit in sqe->flags, but we still can > extend sqe->flags with io_uring context flag, such as use __pad3 for > non-uring_cmd OPs and part of uring_cmd_flags for uring_cmd OP. Did you run the liburing tests with this? I rebased it on top of the flags2 patch I just sent out, and it fails defer-taskrun and crashes link_drain. Don't know if others fail too. I'll try the original one too, but nothing between those two should make a difference. It passes just fine with just the flags2 patch, so I'm a bit suspicious this patch is the issue.
On 10/31/24 3:24 PM, Jens Axboe wrote: > On 10/25/24 6:22 AM, Ming Lei wrote: >> SQE group is defined as one chain of SQEs starting with the first SQE that >> has IOSQE_SQE_GROUP set, and ending with the first subsequent SQE that >> doesn't have it set, and it is similar with chain of linked SQEs. >> >> Not like linked SQEs, each sqe is issued after the previous one is >> completed. All SQEs in one group can be submitted in parallel. To simplify >> the implementation from beginning, all members are queued after the leader >> is completed, however, this way may be changed and leader and members may >> be issued concurrently in future. >> >> The 1st SQE is group leader, and the other SQEs are group member. The whole >> group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and >> the two flags can't be set for group members. For the sake of >> simplicity, IORING_OP_LINK_TIMEOUT is disallowed for SQE group now. >> >> When the group is in one link chain, this group isn't submitted until the >> previous SQE or group is completed. And the following SQE or group can't >> be started if this group isn't completed. Failure from any group member will >> fail the group leader, then the link chain can be terminated. >> >> When IOSQE_IO_DRAIN is set for group leader, all requests in this group and >> previous requests submitted are drained. Given IOSQE_IO_DRAIN can be set for >> group leader only, we respect IO_DRAIN by always completing group leader as >> the last one in the group. Meantime it is natural to post leader's CQE >> as the last one from application viewpoint. >> >> Working together with IOSQE_IO_LINK, SQE group provides flexible way to >> support N:M dependency, such as: >> >> - group A is chained with group B together >> - group A has N SQEs >> - group B has M SQEs >> >> then M SQEs in group B depend on N SQEs in group A. >> >> N:M dependency can support some interesting use cases in efficient way: >> >> 1) read from multiple files, then write the read data into single file >> >> 2) read from single file, and write the read data into multiple files >> >> 3) write same data into multiple files, and read data from multiple files and >> compare if correct data is written >> >> Also IOSQE_SQE_GROUP takes the last bit in sqe->flags, but we still can >> extend sqe->flags with io_uring context flag, such as use __pad3 for >> non-uring_cmd OPs and part of uring_cmd_flags for uring_cmd OP. > > Did you run the liburing tests with this? I rebased it on top of the > flags2 patch I just sent out, and it fails defer-taskrun and crashes > link_drain. Don't know if others fail too. I'll try the original one > too, but nothing between those two should make a difference. It passes > just fine with just the flags2 patch, so I'm a bit suspicious this patch > is the issue. False alarm, it was my messup adding the group flag. Works just fine. I'm attaching the version I tested, on top of that flags2 patch. Since we're on the topic - my original bundle patch used a bundle OP to define an sqe grouping, which didn't need to use an sqe flag. Any particular reason why you went with a flag for this one? I do think it comes out nicer with a flag for certain things, like being able to link groups. Maybe that's the primary reason.
On 10/31/24 3:39 PM, Jens Axboe wrote: > On 10/31/24 3:24 PM, Jens Axboe wrote: >> On 10/25/24 6:22 AM, Ming Lei wrote: >>> SQE group is defined as one chain of SQEs starting with the first SQE that >>> has IOSQE_SQE_GROUP set, and ending with the first subsequent SQE that >>> doesn't have it set, and it is similar with chain of linked SQEs. >>> >>> Not like linked SQEs, each sqe is issued after the previous one is >>> completed. All SQEs in one group can be submitted in parallel. To simplify >>> the implementation from beginning, all members are queued after the leader >>> is completed, however, this way may be changed and leader and members may >>> be issued concurrently in future. >>> >>> The 1st SQE is group leader, and the other SQEs are group member. The whole >>> group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and >>> the two flags can't be set for group members. For the sake of >>> simplicity, IORING_OP_LINK_TIMEOUT is disallowed for SQE group now. >>> >>> When the group is in one link chain, this group isn't submitted until the >>> previous SQE or group is completed. And the following SQE or group can't >>> be started if this group isn't completed. Failure from any group member will >>> fail the group leader, then the link chain can be terminated. >>> >>> When IOSQE_IO_DRAIN is set for group leader, all requests in this group and >>> previous requests submitted are drained. Given IOSQE_IO_DRAIN can be set for >>> group leader only, we respect IO_DRAIN by always completing group leader as >>> the last one in the group. Meantime it is natural to post leader's CQE >>> as the last one from application viewpoint. >>> >>> Working together with IOSQE_IO_LINK, SQE group provides flexible way to >>> support N:M dependency, such as: >>> >>> - group A is chained with group B together >>> - group A has N SQEs >>> - group B has M SQEs >>> >>> then M SQEs in group B depend on N SQEs in group A. >>> >>> N:M dependency can support some interesting use cases in efficient way: >>> >>> 1) read from multiple files, then write the read data into single file >>> >>> 2) read from single file, and write the read data into multiple files >>> >>> 3) write same data into multiple files, and read data from multiple files and >>> compare if correct data is written >>> >>> Also IOSQE_SQE_GROUP takes the last bit in sqe->flags, but we still can >>> extend sqe->flags with io_uring context flag, such as use __pad3 for >>> non-uring_cmd OPs and part of uring_cmd_flags for uring_cmd OP. >> >> Did you run the liburing tests with this? I rebased it on top of the >> flags2 patch I just sent out, and it fails defer-taskrun and crashes >> link_drain. Don't know if others fail too. I'll try the original one >> too, but nothing between those two should make a difference. It passes >> just fine with just the flags2 patch, so I'm a bit suspicious this patch >> is the issue. > > False alarm, it was my messup adding the group flag. Works just fine. > I'm attaching the version I tested, on top of that flags2 patch. > > Since we're on the topic - my original bundle patch used a bundle OP to > define an sqe grouping, which didn't need to use an sqe flag. Any > particular reason why you went with a flag for this one? > > I do think it comes out nicer with a flag for certain things, like being > able to link groups. Maybe that's the primary reason. Various hickups, please just see the patches here, works now: https://git.kernel.dk/cgit/linux/log/?h=io_uring-group
diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 6d3ee71bd832..d524be7f6b35 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -201,6 +201,8 @@ struct io_submit_state { /* batch completion logic */ struct io_wq_work_list compl_reqs; struct io_submit_link link; + /* points to current group */ + struct io_submit_link group; bool plug_started; bool need_plug; @@ -436,6 +438,7 @@ enum { REQ_F_FORCE_ASYNC_BIT = IOSQE_ASYNC_BIT, REQ_F_BUFFER_SELECT_BIT = IOSQE_BUFFER_SELECT_BIT, REQ_F_CQE_SKIP_BIT = IOSQE_CQE_SKIP_SUCCESS_BIT, + REQ_F_SQE_GROUP_BIT = IOSQE_SQE_GROUP_BIT, /* first byte is taken by user flags, shift it to not overlap */ REQ_F_FAIL_BIT = 8, @@ -465,6 +468,7 @@ enum { REQ_F_BL_EMPTY_BIT, REQ_F_BL_NO_RECYCLE_BIT, REQ_F_BUFFERS_COMMIT_BIT, + REQ_F_SQE_GROUP_LEADER_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -488,6 +492,8 @@ enum { REQ_F_BUFFER_SELECT = IO_REQ_FLAG(REQ_F_BUFFER_SELECT_BIT), /* IOSQE_CQE_SKIP_SUCCESS */ REQ_F_CQE_SKIP = IO_REQ_FLAG(REQ_F_CQE_SKIP_BIT), + /* IOSQE_SQE_GROUP */ + REQ_F_SQE_GROUP = IO_REQ_FLAG(REQ_F_SQE_GROUP_BIT), /* fail rest of links */ REQ_F_FAIL = IO_REQ_FLAG(REQ_F_FAIL_BIT), @@ -541,6 +547,8 @@ enum { REQ_F_BL_NO_RECYCLE = IO_REQ_FLAG(REQ_F_BL_NO_RECYCLE_BIT), /* buffer ring head needs incrementing on put */ REQ_F_BUFFERS_COMMIT = IO_REQ_FLAG(REQ_F_BUFFERS_COMMIT_BIT), + /* sqe group lead */ + REQ_F_SQE_GROUP_LEADER = IO_REQ_FLAG(REQ_F_SQE_GROUP_LEADER_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); @@ -643,6 +651,8 @@ struct io_kiocb { void *async_data; /* linked requests, IFF REQ_F_HARDLINK or REQ_F_LINK are set */ atomic_t poll_refs; + /* reference for group leader request */ + int grp_refs; struct io_kiocb *link; /* custom credentials, valid IFF REQ_F_CREDS is set */ const struct cred *creds; @@ -652,6 +662,14 @@ struct io_kiocb { u64 extra1; u64 extra2; } big_cqe; + + union { + /* links all group members for leader */ + struct io_kiocb *grp_link; + + /* points to group leader for member */ + struct io_kiocb *grp_leader; + }; }; struct io_overflow_cqe { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 86cb385fe0b5..f298dd5fbaa9 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -124,6 +124,7 @@ enum io_uring_sqe_flags_bit { IOSQE_ASYNC_BIT, IOSQE_BUFFER_SELECT_BIT, IOSQE_CQE_SKIP_SUCCESS_BIT, + IOSQE_SQE_GROUP_BIT, }; /* @@ -143,6 +144,8 @@ enum io_uring_sqe_flags_bit { #define IOSQE_BUFFER_SELECT (1U << IOSQE_BUFFER_SELECT_BIT) /* don't post CQE if request succeeded */ #define IOSQE_CQE_SKIP_SUCCESS (1U << IOSQE_CQE_SKIP_SUCCESS_BIT) +/* defines sqe group */ +#define IOSQE_SQE_GROUP (1U << IOSQE_SQE_GROUP_BIT) /* * io_uring_setup() flags @@ -554,6 +557,7 @@ struct io_uring_params { #define IORING_FEAT_REG_REG_RING (1U << 13) #define IORING_FEAT_RECVSEND_BUNDLE (1U << 14) #define IORING_FEAT_MIN_TIMEOUT (1U << 15) +#define IORING_FEAT_SQE_GROUP (1U << 16) /* * io_uring_register(2) opcodes and arguments diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 33856560ff87..59e9a01319de 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -112,14 +112,15 @@ IOSQE_IO_HARDLINK | IOSQE_ASYNC) #define SQE_VALID_FLAGS (SQE_COMMON_FLAGS | IOSQE_BUFFER_SELECT | \ - IOSQE_IO_DRAIN | IOSQE_CQE_SKIP_SUCCESS) + IOSQE_IO_DRAIN | IOSQE_CQE_SKIP_SUCCESS | \ + IOSQE_SQE_GROUP) #define IO_REQ_CLEAN_FLAGS (REQ_F_BUFFER_SELECTED | REQ_F_NEED_CLEANUP | \ REQ_F_POLLED | REQ_F_INFLIGHT | REQ_F_CREDS | \ REQ_F_ASYNC_DATA) #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | REQ_F_LINK | REQ_F_HARDLINK |\ - IO_REQ_CLEAN_FLAGS) + REQ_F_SQE_GROUP | IO_REQ_CLEAN_FLAGS) #define IO_TCTX_REFS_CACHE_NR (1U << 10) @@ -912,6 +913,129 @@ static __always_inline void io_req_commit_cqe(struct io_ring_ctx *ctx, } } +/* Can only be called after this request is issued */ +static inline struct io_kiocb *get_group_leader(struct io_kiocb *req) +{ + if (req->flags & REQ_F_SQE_GROUP) { + if (req_is_group_leader(req)) + return req; + return req->grp_leader; + } + return NULL; +} + +void io_fail_group_members(struct io_kiocb *req) +{ + struct io_kiocb *member = req->grp_link; + + while (member) { + struct io_kiocb *next = member->grp_link; + + if (!(member->flags & REQ_F_FAIL)) { + req_set_fail(member); + io_req_set_res(member, -ECANCELED, 0); + } + member = next; + } +} + +static void io_queue_group_members(struct io_kiocb *req) +{ + struct io_kiocb *member = req->grp_link; + + if (!member) + return; + + req->grp_link = NULL; + while (member) { + struct io_kiocb *next = member->grp_link; + + member->grp_leader = req; + if (unlikely(member->flags & REQ_F_FAIL)) { + io_req_task_queue_fail(member, member->cqe.res); + } else if (unlikely(req->flags & REQ_F_FAIL)) { + io_req_task_queue_fail(member, -ECANCELED); + } else { + io_req_task_queue(member); + } + member = next; + } +} + +/* called only after the request is completed */ +static bool req_is_last_group_member(struct io_kiocb *req) +{ + return req->grp_leader != NULL; +} + +static void io_complete_group_req(struct io_kiocb *req) +{ + struct io_kiocb *lead; + + if (req_is_group_leader(req)) { + req->grp_refs -= 1; + return; + } + + lead = get_group_leader(req); + + /* member CQE needs to be posted first */ + if (!(req->flags & REQ_F_CQE_SKIP)) + io_req_commit_cqe(req->ctx, req); + + /* Set leader as failed in case of any member failed */ + if (unlikely((req->flags & REQ_F_FAIL))) + req_set_fail(lead); + + WARN_ON_ONCE(lead->grp_refs <= 0); + if (!--lead->grp_refs) { + /* + * We are the last member, and ->grp_leader isn't cleared, + * so our leader can be found & freed with the last member + */ + if (!(lead->flags & REQ_F_CQE_SKIP)) + io_req_commit_cqe(lead->ctx, lead); + } else { + /* we are done with the group now */ + req->grp_leader = NULL; + } +} + +enum group_mem { + GROUP_LEADER, + GROUP_LAST_MEMBER, + GROUP_OTHER_MEMBER, +}; + +static enum group_mem io_prep_free_group_req(struct io_kiocb *req, + struct io_kiocb **leader) +{ + /* + * Group completion is done, so clear the flag for avoiding double + * handling in case of io-wq + */ + req->flags &= ~REQ_F_SQE_GROUP; + + if (req_is_group_leader(req)) { + /* Queue members now */ + if (req->grp_link) + io_queue_group_members(req); + return GROUP_LEADER; + } else { + if (!req_is_last_group_member(req)) + return GROUP_OTHER_MEMBER; + + /* + * Prepare for freeing leader which can only be found from + * the last member + */ + *leader = req->grp_leader; + (*leader)->flags &= ~REQ_F_SQE_GROUP_LEADER; + req->grp_leader = NULL; + return GROUP_LAST_MEMBER; + } +} + static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; @@ -927,7 +1051,8 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires * the submitter task context, IOPOLL protects with uring_lock. */ - if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL)) { + if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL) || + (req->flags & REQ_F_SQE_GROUP)) { req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); return; @@ -1411,6 +1536,27 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, comp_list); if (unlikely(req->flags & IO_REQ_CLEAN_SLOW_FLAGS)) { + if (req->flags & REQ_F_SQE_GROUP) { + struct io_kiocb *leader = NULL; + enum group_mem mem = io_prep_free_group_req(req, &leader); + + if (mem == GROUP_LEADER) { + node = req->comp_list.next; + continue; + } else if (mem == GROUP_LAST_MEMBER) { + /* + * Link leader to current request's next, + * this way works because the iterator + * always check the next node only. + * + * Be careful when you change the iterator + * in future + */ + wq_stack_add_head(&leader->comp_list, + &req->comp_list); + } + } + if (req->flags & REQ_F_REFCOUNT) { node = req->comp_list.next; if (!req_ref_put_and_test(req)) @@ -1450,8 +1596,16 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); - if (!(req->flags & REQ_F_CQE_SKIP)) - io_req_commit_cqe(ctx, req); + if (unlikely(req->flags & (REQ_F_CQE_SKIP | REQ_F_SQE_GROUP))) { + if (req->flags & REQ_F_SQE_GROUP) { + io_complete_group_req(req); + continue; + } + + if (req->flags & REQ_F_CQE_SKIP) + continue; + } + io_req_commit_cqe(ctx, req); } __io_cq_unlock_post(ctx); @@ -1661,8 +1815,12 @@ static u32 io_get_sequence(struct io_kiocb *req) struct io_kiocb *cur; /* need original cached_sq_head, but it was increased for each req */ - io_for_each_link(cur, req) - seq--; + io_for_each_link(cur, req) { + if (req_is_group_leader(cur)) + seq -= cur->grp_refs; + else + seq--; + } return seq; } @@ -2124,6 +2282,67 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, return def->prep(req, sqe); } +static struct io_kiocb *io_group_sqe(struct io_submit_link *group, + struct io_kiocb *req) +{ + /* + * Group chain is similar with link chain: starts with 1st sqe with + * REQ_F_SQE_GROUP, and ends with the 1st sqe without REQ_F_SQE_GROUP + */ + if (group->head) { + struct io_kiocb *lead = group->head; + + /* + * Members can't be in link chain, can't be drained, but + * the whole group can be linked or drained by setting + * flags on group leader. + * + * IOSQE_CQE_SKIP_SUCCESS can't be set for member + * for the sake of simplicity + */ + if (req->flags & (IO_REQ_LINK_FLAGS | REQ_F_IO_DRAIN | + REQ_F_CQE_SKIP)) + req_fail_link_node(lead, -EINVAL); + + lead->grp_refs += 1; + group->last->grp_link = req; + group->last = req; + + if (req->flags & REQ_F_SQE_GROUP) + return NULL; + + req->grp_link = NULL; + req->flags |= REQ_F_SQE_GROUP; + group->head = NULL; + + return lead; + } else { + if (WARN_ON_ONCE(!(req->flags & REQ_F_SQE_GROUP))) + return req; + group->head = req; + group->last = req; + req->grp_refs = 1; + req->flags |= REQ_F_SQE_GROUP_LEADER; + return NULL; + } +} + +static __cold struct io_kiocb *io_submit_fail_group( + struct io_submit_link *link, struct io_kiocb *req) +{ + struct io_kiocb *lead = link->head; + + /* + * Instead of failing eagerly, continue assembling the group link + * if applicable and mark the leader with REQ_F_FAIL. The group + * flushing code should find the flag and handle the rest + */ + if (lead && !(lead->flags & REQ_F_FAIL)) + req_fail_link_node(lead, -ECANCELED); + + return io_group_sqe(link, req); +} + static __cold int io_submit_fail_link(struct io_submit_link *link, struct io_kiocb *req, int ret) { @@ -2162,11 +2381,18 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, { struct io_ring_ctx *ctx = req->ctx; struct io_submit_link *link = &ctx->submit_state.link; + struct io_submit_link *group = &ctx->submit_state.group; trace_io_uring_req_failed(sqe, req, ret); req_fail_link_node(req, ret); + if (group->head || (req->flags & REQ_F_SQE_GROUP)) { + req = io_submit_fail_group(group, req); + if (!req) + return 0; + } + /* cover both linked and non-linked request */ return io_submit_fail_link(link, req, ret); } @@ -2210,11 +2436,29 @@ static struct io_kiocb *io_link_sqe(struct io_submit_link *link, return req; } +static inline bool io_group_assembling(const struct io_submit_state *state, + const struct io_kiocb *req) +{ + if (state->group.head || req->flags & REQ_F_SQE_GROUP) + return true; + return false; +} + +/* Failed request is covered too */ +static inline bool io_link_assembling(const struct io_submit_state *state, + const struct io_kiocb *req) +{ + if (state->link.head || (req->flags & (IO_REQ_LINK_FLAGS | + REQ_F_FORCE_ASYNC | REQ_F_FAIL))) + return true; + return false; +} + static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, const struct io_uring_sqe *sqe) __must_hold(&ctx->uring_lock) { - struct io_submit_link *link = &ctx->submit_state.link; + struct io_submit_state *state = &ctx->submit_state; int ret; ret = io_init_req(ctx, req, sqe); @@ -2223,11 +2467,20 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, trace_io_uring_submit_req(req); - if (unlikely(link->head || (req->flags & (IO_REQ_LINK_FLAGS | - REQ_F_FORCE_ASYNC | REQ_F_FAIL)))) { - req = io_link_sqe(link, req); - if (!req) - return 0; + if (unlikely(io_link_assembling(state, req) || + io_group_assembling(state, req))) { + if (io_group_assembling(state, req)) { + req = io_group_sqe(&state->group, req); + if (!req) + return 0; + } + + /* covers non-linked failed request too */ + if (io_link_assembling(state, req)) { + req = io_link_sqe(&state->link, req); + if (!req) + return 0; + } } io_queue_sqe(req); return 0; @@ -2240,8 +2493,27 @@ static void io_submit_state_end(struct io_ring_ctx *ctx) { struct io_submit_state *state = &ctx->submit_state; - if (unlikely(state->link.head)) - io_queue_sqe_fallback(state->link.head); + if (unlikely(state->group.head || state->link.head)) { + /* the last member must set REQ_F_SQE_GROUP */ + if (state->group.head) { + struct io_kiocb *lead = state->group.head; + struct io_kiocb *last = state->group.last; + + /* fail group with single leader */ + if (unlikely(last == lead)) + req_fail_link_node(lead, -EINVAL); + + last->grp_link = NULL; + if (state->link.head) + io_link_sqe(&state->link, lead); + else + io_queue_sqe_fallback(lead); + } + + if (unlikely(state->link.head)) + io_queue_sqe_fallback(state->link.head); + } + /* flush only after queuing links as they can generate completions */ io_submit_flush_completions(ctx); if (state->plug_started) @@ -2259,6 +2531,7 @@ static void io_submit_state_start(struct io_submit_state *state, state->submit_nr = max_ios; /* set only head, no need to init link_last in advance */ state->link.head = NULL; + state->group.head = NULL; } static void io_commit_sqring(struct io_ring_ctx *ctx) @@ -3699,7 +3972,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS | IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP | IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING | - IORING_FEAT_RECVSEND_BUNDLE | IORING_FEAT_MIN_TIMEOUT; + IORING_FEAT_RECVSEND_BUNDLE | IORING_FEAT_MIN_TIMEOUT | + IORING_FEAT_SQE_GROUP; if (copy_to_user(params, p, sizeof(*p))) { ret = -EFAULT; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 12e8fca73891..ab84b09505fe 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -72,6 +72,7 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags void io_add_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); +void io_fail_group_members(struct io_kiocb *req); struct file *io_file_get_normal(struct io_kiocb *req, int fd); struct file *io_file_get_fixed(struct io_kiocb *req, int fd, @@ -343,6 +344,11 @@ static inline void io_tw_lock(struct io_ring_ctx *ctx, struct io_tw_state *ts) lockdep_assert_held(&ctx->uring_lock); } +static inline bool req_is_group_leader(struct io_kiocb *req) +{ + return req->flags & REQ_F_SQE_GROUP_LEADER; +} + /* * Don't complete immediately but use deferred completion infrastructure. * Protected by ->uring_lock and can only be used either with diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 9973876d91b0..ed6c74f1a475 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -149,6 +149,8 @@ static void io_req_tw_fail_links(struct io_kiocb *link, struct io_tw_state *ts) res = link->cqe.res; link->link = NULL; io_req_set_res(link, res, 0); + if (req_is_group_leader(link)) + io_fail_group_members(link); io_req_task_complete(link, ts); link = nxt; } @@ -543,6 +545,10 @@ static int __io_timeout_prep(struct io_kiocb *req, if (is_timeout_link) { struct io_submit_link *link = &req->ctx->submit_state.link; + /* so far disallow IO group link timeout */ + if (req->ctx->submit_state.group.head) + return -EINVAL; + if (!link->head) return -EINVAL; if (link->last->opcode == IORING_OP_LINK_TIMEOUT)
SQE group is defined as one chain of SQEs starting with the first SQE that has IOSQE_SQE_GROUP set, and ending with the first subsequent SQE that doesn't have it set, and it is similar with chain of linked SQEs. Not like linked SQEs, each sqe is issued after the previous one is completed. All SQEs in one group can be submitted in parallel. To simplify the implementation from beginning, all members are queued after the leader is completed, however, this way may be changed and leader and members may be issued concurrently in future. The 1st SQE is group leader, and the other SQEs are group member. The whole group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and the two flags can't be set for group members. For the sake of simplicity, IORING_OP_LINK_TIMEOUT is disallowed for SQE group now. When the group is in one link chain, this group isn't submitted until the previous SQE or group is completed. And the following SQE or group can't be started if this group isn't completed. Failure from any group member will fail the group leader, then the link chain can be terminated. When IOSQE_IO_DRAIN is set for group leader, all requests in this group and previous requests submitted are drained. Given IOSQE_IO_DRAIN can be set for group leader only, we respect IO_DRAIN by always completing group leader as the last one in the group. Meantime it is natural to post leader's CQE as the last one from application viewpoint. Working together with IOSQE_IO_LINK, SQE group provides flexible way to support N:M dependency, such as: - group A is chained with group B together - group A has N SQEs - group B has M SQEs then M SQEs in group B depend on N SQEs in group A. N:M dependency can support some interesting use cases in efficient way: 1) read from multiple files, then write the read data into single file 2) read from single file, and write the read data into multiple files 3) write same data into multiple files, and read data from multiple files and compare if correct data is written Also IOSQE_SQE_GROUP takes the last bit in sqe->flags, but we still can extend sqe->flags with io_uring context flag, such as use __pad3 for non-uring_cmd OPs and part of uring_cmd_flags for uring_cmd OP. Suggested-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Ming Lei <ming.lei@redhat.com> --- include/linux/io_uring_types.h | 18 ++ include/uapi/linux/io_uring.h | 4 + io_uring/io_uring.c | 306 +++++++++++++++++++++++++++++++-- io_uring/io_uring.h | 6 + io_uring/timeout.c | 6 + 5 files changed, 324 insertions(+), 16 deletions(-)