From patchwork Fri Oct 25 12:22:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13850616 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 106191E8845 for ; Fri, 25 Oct 2024 12:23:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729858991; cv=none; b=mhrxwtNJNBdrQZl/tu3O7PXkemVzrFNzRKSx7zOtayRdyzG+pQfvV3+1/3nlNxtpFgylcrj3fehy7dDPWpu3iHaxR3a5gKUvQ8Ako168XrQFqsqYhL0mkz9sFXuY0vaa14RL9txTIq1DOy+u9O022bbhiY9Ek3mIDtFdv7GvH2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729858991; c=relaxed/simple; bh=LZ/n3hEnsp/MJH05VXcNhuGICUOKdYLSrROs6pR+ILw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BJAQ4QsKRX3YGifdYkRFG4UXQqIn+SD2y7axayowTufYQ7mU27x4Fa3o9jwtM9tWAHProfIzFW8x/6O98sll5BErL+d29VOzCm+h9Jg260xOSjc6b27q1EO2Ql6wFLLc3pJgUuRucIfHBHt793Of/RQcXX3pMGD803ChGHhUg/k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=FLbS4pQn; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FLbS4pQn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1729858987; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tEHSsCvK6YvCrDa7THoJ0nmcyDA/rlY7BBNXgdGj4+0=; b=FLbS4pQncoh5v4CLhtjKTcy8rQ7iEZ1nk34OTp9mdsteSdrLY2qVHSSNeLjNsB0vva2kQ/ 08rdgYhQKGZWFAQ+r1cVL4LesE4dpGxTa+lbXyzIDYKS+wUyIDDChL2JZFwAOBKdfsa9u+ pTImkVjqdGqj0RBhdMQ3ZUreLfYXoN8= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-301-pVdE9p3IMWO5txCh7DM0vg-1; Fri, 25 Oct 2024 08:23:06 -0400 X-MC-Unique: pVdE9p3IMWO5txCh7DM0vg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D29FE195609E; Fri, 25 Oct 2024 12:23:04 +0000 (UTC) Received: from localhost (unknown [10.72.116.106]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9DEEE196BB7D; Fri, 25 Oct 2024 12:23:03 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org, Pavel Begunkov Cc: linux-block@vger.kernel.org, Uday Shankar , Akilesh Kailash , Ming Lei Subject: [PATCH V8 1/7] io_uring: add io_link_req() helper Date: Fri, 25 Oct 2024 20:22:38 +0800 Message-ID: <20241025122247.3709133-2-ming.lei@redhat.com> In-Reply-To: <20241025122247.3709133-1-ming.lei@redhat.com> References: <20241025122247.3709133-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Add io_link_req() helper, so that io_submit_sqe() can become more readable. Signed-off-by: Ming Lei --- io_uring/io_uring.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 58b401900b41..02f7dd58d44d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2155,19 +2155,11 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, return 0; } -static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, - const struct io_uring_sqe *sqe) - __must_hold(&ctx->uring_lock) +/* + * Return NULL if nothing to be queued, otherwise return request for queueing */ +static struct io_kiocb *io_link_sqe(struct io_submit_link *link, + struct io_kiocb *req) { - struct io_submit_link *link = &ctx->submit_state.link; - int ret; - - ret = io_init_req(ctx, req, sqe); - if (unlikely(ret)) - return io_submit_fail_init(sqe, req, ret); - - trace_io_uring_submit_req(req); - /* * If we already have a head request, queue this one for async * submittal once the head completes. If we don't have a head but @@ -2181,7 +2173,7 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, link->last = req; if (req->flags & IO_REQ_LINK_FLAGS) - return 0; + return NULL; /* last request of the link, flush it */ req = link->head; link->head = NULL; @@ -2197,9 +2189,30 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, fallback: io_queue_sqe_fallback(req); } - return 0; + return NULL; } + return req; +} + +static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, + const struct io_uring_sqe *sqe) + __must_hold(&ctx->uring_lock) +{ + struct io_submit_link *link = &ctx->submit_state.link; + int ret; + ret = io_init_req(ctx, req, sqe); + if (unlikely(ret)) + return io_submit_fail_init(sqe, req, ret); + + trace_io_uring_submit_req(req); + + if (unlikely(link->head || (req->flags & (IO_REQ_LINK_FLAGS | + REQ_F_FORCE_ASYNC | REQ_F_FAIL)))) { + req = io_link_sqe(link, req); + if (!req) + return 0; + } io_queue_sqe(req); return 0; } From patchwork Fri Oct 25 12:22:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13850617 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E10BB2036EC for ; Fri, 25 Oct 2024 12:23:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729858994; cv=none; b=Y6YLSd+pWc0AR/01YH+HdbwuS9Fo/xrU9rP6CNaY4FA4Kr7lcPtdSDs4qajhz9iqNFgWDqNr/cWVOD5dy0OXapDrLsdk0MKb/LHHcJeSHFVMHTiDWqX1l654VKfrmkvYkSLT1I65mUhcHa2OiTu+MRATgDHO+eb4eLWWOqIRxVQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729858994; c=relaxed/simple; bh=z3vjFtkyi8S1A7ur2dmbu1WFpyaXcQe4CCIPF5e6/Aw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mANAD/BDCldFkVu9rCB8ElPZXGqhANqrx29is9/4isn6r/e9NJM/rvw78wwiTeVo0Vx3OSGXqfNfgfKNxHGvlHcoO97AZpdzH5uc+grAWwYVLZC2os8o1MNu9HHF4psvRbZsLZq8E09EKDA+w/zMPcQTcu1W5GWEoh97b7CFtLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=eHpDRFTR; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="eHpDRFTR" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1729858991; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PO8uGQ7tOYm3CIZufgCXnY8v91zqH0P/3o10fpPhS8s=; b=eHpDRFTRPjI7a/HUhj44tTLsrQHtgT4/GMNKMMlFTeIgmvQXEzA/qUrf/yTKqHFrI3ZoY6 53fdHBXIyHQnhjFPkdOqqoiEp8gV2SZTLuPKAsk7p7BVG13E68ITEDBn0xiLp6h32fvth2 6aYqrIuldOg8x8joybuukR/1GVq24bw= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-658-5AqOSgdLP2alENJAioQPFg-1; Fri, 25 Oct 2024 08:23:10 -0400 X-MC-Unique: 5AqOSgdLP2alENJAioQPFg-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0EF711955F67; Fri, 25 Oct 2024 12:23:09 +0000 (UTC) Received: from localhost (unknown [10.72.116.106]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id E1DB819560A2; Fri, 25 Oct 2024 12:23:07 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org, Pavel Begunkov Cc: linux-block@vger.kernel.org, Uday Shankar , Akilesh Kailash , Ming Lei Subject: [PATCH V8 2/7] io_uring: add io_submit_fail_link() helper Date: Fri, 25 Oct 2024 20:22:39 +0800 Message-ID: <20241025122247.3709133-3-ming.lei@redhat.com> In-Reply-To: <20241025122247.3709133-1-ming.lei@redhat.com> References: <20241025122247.3709133-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 Add io_submit_fail_link() helper and put linking fail logic into this helper. This way simplifies io_submit_fail_init(), and becomes easier to add sqe group failing logic. Signed-off-by: Ming Lei --- io_uring/io_uring.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 02f7dd58d44d..749ecc18049d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2118,22 +2118,17 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, return def->prep(req, sqe); } -static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, +static __cold int io_submit_fail_link(struct io_submit_link *link, struct io_kiocb *req, int ret) { - struct io_ring_ctx *ctx = req->ctx; - struct io_submit_link *link = &ctx->submit_state.link; struct io_kiocb *head = link->head; - trace_io_uring_req_failed(sqe, req, ret); - /* * Avoid breaking links in the middle as it renders links with SQPOLL * unusable. Instead of failing eagerly, continue assembling the link if * applicable and mark the head with REQ_F_FAIL. The link flushing code * should find the flag and handle the rest. */ - req_fail_link_node(req, ret); if (head && !(head->flags & REQ_F_FAIL)) req_fail_link_node(head, -ECANCELED); @@ -2152,9 +2147,24 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, else link->head = req; link->last = req; + return 0; } +static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, + struct io_kiocb *req, int ret) +{ + struct io_ring_ctx *ctx = req->ctx; + struct io_submit_link *link = &ctx->submit_state.link; + + trace_io_uring_req_failed(sqe, req, ret); + + req_fail_link_node(req, ret); + + /* cover both linked and non-linked request */ + return io_submit_fail_link(link, req, ret); +} + /* * Return NULL if nothing to be queued, otherwise return request for queueing */ static struct io_kiocb *io_link_sqe(struct io_submit_link *link, From patchwork Fri Oct 25 12:22:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13850620 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AF9E1E8845 for ; Fri, 25 Oct 2024 12:24:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859043; cv=none; b=onW4sTr4j/IrDOqkSI4zCg5mzQmFlKdkl4aPyDI+9cU2z2ycA0DFCoNtDUAFBM6d39b6KVL9ky/du6VhF5pNrbJiHU0JgBdtsu3TTErVCqL0EU8u6Nq2gKzQ2ak+b2uFqOdWb5IDgLUl1b4f/EMJtSbCcGFCGFzuKL0CpU4+BSo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859043; c=relaxed/simple; bh=2lW3tx2drdKGhaeUZ2vPyduwEZzVpqIgcMsEP5dYjM4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O7SRrheO4yhUIWv4Lx+go4oOuV3UbRzSL2FoRR2BneQjouffq68T3J4baibACQbmPllxDh1v/66wxWay53vmBgE9WYrQHIQGXzyzxCWSHR+9dWNiB3DFynkzLYc6SalQtp6lm+VDXxXOxWxe+Exy784Wy/vTcJMkwq98/FahB7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=YtNCFLxy; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="YtNCFLxy" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1729859040; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DIMBfrwdYfNOX2UOeCTE0NWQIo7aiZOeYzAIGl4DTI0=; b=YtNCFLxyOLeISwshjcGvMlxEokXyR6nGYek6EH11WL+qIz/W/Pxj5cP59t2RJ8MJLvQ6UX rGe4ynoteBKKQvIr5uQArOCtMQYhfKSONpN4S7B6ujqGasra7v5OZnuo0J/hrn17VJbueg SIAxeBamrlz7Cc9sRZLunPqvJ2HKnYw= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-664--FMOeRDWPISgRPgoDVPnag-1; Fri, 25 Oct 2024 08:23:57 -0400 X-MC-Unique: -FMOeRDWPISgRPgoDVPnag-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 39464197904E; Fri, 25 Oct 2024 12:23:13 +0000 (UTC) Received: from localhost (unknown [10.72.116.106]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 046101956088; Fri, 25 Oct 2024 12:23:11 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org, Pavel Begunkov Cc: linux-block@vger.kernel.org, Uday Shankar , Akilesh Kailash , Ming Lei Subject: [PATCH V8 3/7] io_uring: add helper of io_req_commit_cqe() Date: Fri, 25 Oct 2024 20:22:40 +0800 Message-ID: <20241025122247.3709133-4-ming.lei@redhat.com> In-Reply-To: <20241025122247.3709133-1-ming.lei@redhat.com> References: <20241025122247.3709133-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 Add helper of io_req_commit_cqe() for simplifying __io_submit_flush_completions() a bit. No functional change, and the added helper will be reused in sqe group code with same lock rule. Signed-off-by: Ming Lei --- io_uring/io_uring.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 749ecc18049d..33856560ff87 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -898,6 +898,20 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) return posted; } +static __always_inline void io_req_commit_cqe(struct io_ring_ctx *ctx, + struct io_kiocb *req) +{ + if (unlikely(!io_fill_cqe_req(ctx, req))) { + if (ctx->lockless_cq) { + spin_lock(&ctx->completion_lock); + io_req_cqe_overflow(req); + spin_unlock(&ctx->completion_lock); + } else { + io_req_cqe_overflow(req); + } + } +} + static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; @@ -1436,16 +1450,8 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); - if (!(req->flags & REQ_F_CQE_SKIP) && - unlikely(!io_fill_cqe_req(ctx, req))) { - if (ctx->lockless_cq) { - spin_lock(&ctx->completion_lock); - io_req_cqe_overflow(req); - spin_unlock(&ctx->completion_lock); - } else { - io_req_cqe_overflow(req); - } - } + if (!(req->flags & REQ_F_CQE_SKIP)) + io_req_commit_cqe(ctx, req); } __io_cq_unlock_post(ctx); From patchwork Fri Oct 25 12:22:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13850621 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 970411F80DB for ; Fri, 25 Oct 2024 12:24:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859044; cv=none; b=EgvXJnvA1J5tmbIO4S1j6TvPm0XeNYDCoyyv9CkqCS37jPcLhfmORewL/KKLiX8KwwXR93WxkwAc5fVhAve7YFXw/KfDkKLxNOTiJTgFZcqLNoIhdh9iDsrI0GJSgi8+f8Br9nMdx+5wMxXHk5Xcyket6KxsvI2Sn4vpTeWUoKg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859044; c=relaxed/simple; bh=mIkect+q3jHn2RguNheKTyUR+abvmb2NPgZO8U2rjyQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hrCVOFwh5trSkXSPaMlxUDahLzYiGdCIF+hLQCmAPua3FpR7JdUtWHoD5lhPZmODSvQLnN3hEU+XsYwJ9VnffjyvmcFY1EKc+hM+P7wBOgrr4WpHIFt0br1MPfKs573xpe0sKkCRyQUEODL944LbtYFAfHm1f96kmWLTa5iEn20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ApP+mhex; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ApP+mhex" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1729859040; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9iiDdZ5FQYHLb/3BLsGM6vo3oczVaqF6Q2mhWwPTN9A=; b=ApP+mhex01uGSdzfNavwnFmwltOWZVaSkxpdBxIYGC2u87JRaUEviXl0dlcwrkwe3UwUHG xbbNAL6tPQTRzQwsXO4Ig+yqyDdTzorbEmjzSpaXjBMtTc6QWNex+mqoO7rHeGnTFnCVeb FBlM24+Vd9wXG8mMxAqqymJoLWcZTVs= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-678-0qfRmg5AN-qhGLzgUjBKmw-1; Fri, 25 Oct 2024 08:23:57 -0400 X-MC-Unique: 0qfRmg5AN-qhGLzgUjBKmw-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 11AFD19792DC; Fri, 25 Oct 2024 12:23:17 +0000 (UTC) Received: from localhost (unknown [10.72.116.106]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 23A3A1956088; Fri, 25 Oct 2024 12:23:15 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org, Pavel Begunkov Cc: linux-block@vger.kernel.org, Uday Shankar , Akilesh Kailash , Ming Lei , Kevin Wolf Subject: [PATCH V8 4/7] io_uring: support SQE group Date: Fri, 25 Oct 2024 20:22:41 +0800 Message-ID: <20241025122247.3709133-5-ming.lei@redhat.com> In-Reply-To: <20241025122247.3709133-1-ming.lei@redhat.com> References: <20241025122247.3709133-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 SQE group is defined as one chain of SQEs starting with the first SQE that has IOSQE_SQE_GROUP set, and ending with the first subsequent SQE that doesn't have it set, and it is similar with chain of linked SQEs. Not like linked SQEs, each sqe is issued after the previous one is completed. All SQEs in one group can be submitted in parallel. To simplify the implementation from beginning, all members are queued after the leader is completed, however, this way may be changed and leader and members may be issued concurrently in future. The 1st SQE is group leader, and the other SQEs are group member. The whole group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and the two flags can't be set for group members. For the sake of simplicity, IORING_OP_LINK_TIMEOUT is disallowed for SQE group now. When the group is in one link chain, this group isn't submitted until the previous SQE or group is completed. And the following SQE or group can't be started if this group isn't completed. Failure from any group member will fail the group leader, then the link chain can be terminated. When IOSQE_IO_DRAIN is set for group leader, all requests in this group and previous requests submitted are drained. Given IOSQE_IO_DRAIN can be set for group leader only, we respect IO_DRAIN by always completing group leader as the last one in the group. Meantime it is natural to post leader's CQE as the last one from application viewpoint. Working together with IOSQE_IO_LINK, SQE group provides flexible way to support N:M dependency, such as: - group A is chained with group B together - group A has N SQEs - group B has M SQEs then M SQEs in group B depend on N SQEs in group A. N:M dependency can support some interesting use cases in efficient way: 1) read from multiple files, then write the read data into single file 2) read from single file, and write the read data into multiple files 3) write same data into multiple files, and read data from multiple files and compare if correct data is written Also IOSQE_SQE_GROUP takes the last bit in sqe->flags, but we still can extend sqe->flags with io_uring context flag, such as use __pad3 for non-uring_cmd OPs and part of uring_cmd_flags for uring_cmd OP. Suggested-by: Kevin Wolf Signed-off-by: Ming Lei --- include/linux/io_uring_types.h | 18 ++ include/uapi/linux/io_uring.h | 4 + io_uring/io_uring.c | 306 +++++++++++++++++++++++++++++++-- io_uring/io_uring.h | 6 + io_uring/timeout.c | 6 + 5 files changed, 324 insertions(+), 16 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 6d3ee71bd832..d524be7f6b35 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -201,6 +201,8 @@ struct io_submit_state { /* batch completion logic */ struct io_wq_work_list compl_reqs; struct io_submit_link link; + /* points to current group */ + struct io_submit_link group; bool plug_started; bool need_plug; @@ -436,6 +438,7 @@ enum { REQ_F_FORCE_ASYNC_BIT = IOSQE_ASYNC_BIT, REQ_F_BUFFER_SELECT_BIT = IOSQE_BUFFER_SELECT_BIT, REQ_F_CQE_SKIP_BIT = IOSQE_CQE_SKIP_SUCCESS_BIT, + REQ_F_SQE_GROUP_BIT = IOSQE_SQE_GROUP_BIT, /* first byte is taken by user flags, shift it to not overlap */ REQ_F_FAIL_BIT = 8, @@ -465,6 +468,7 @@ enum { REQ_F_BL_EMPTY_BIT, REQ_F_BL_NO_RECYCLE_BIT, REQ_F_BUFFERS_COMMIT_BIT, + REQ_F_SQE_GROUP_LEADER_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -488,6 +492,8 @@ enum { REQ_F_BUFFER_SELECT = IO_REQ_FLAG(REQ_F_BUFFER_SELECT_BIT), /* IOSQE_CQE_SKIP_SUCCESS */ REQ_F_CQE_SKIP = IO_REQ_FLAG(REQ_F_CQE_SKIP_BIT), + /* IOSQE_SQE_GROUP */ + REQ_F_SQE_GROUP = IO_REQ_FLAG(REQ_F_SQE_GROUP_BIT), /* fail rest of links */ REQ_F_FAIL = IO_REQ_FLAG(REQ_F_FAIL_BIT), @@ -541,6 +547,8 @@ enum { REQ_F_BL_NO_RECYCLE = IO_REQ_FLAG(REQ_F_BL_NO_RECYCLE_BIT), /* buffer ring head needs incrementing on put */ REQ_F_BUFFERS_COMMIT = IO_REQ_FLAG(REQ_F_BUFFERS_COMMIT_BIT), + /* sqe group lead */ + REQ_F_SQE_GROUP_LEADER = IO_REQ_FLAG(REQ_F_SQE_GROUP_LEADER_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); @@ -643,6 +651,8 @@ struct io_kiocb { void *async_data; /* linked requests, IFF REQ_F_HARDLINK or REQ_F_LINK are set */ atomic_t poll_refs; + /* reference for group leader request */ + int grp_refs; struct io_kiocb *link; /* custom credentials, valid IFF REQ_F_CREDS is set */ const struct cred *creds; @@ -652,6 +662,14 @@ struct io_kiocb { u64 extra1; u64 extra2; } big_cqe; + + union { + /* links all group members for leader */ + struct io_kiocb *grp_link; + + /* points to group leader for member */ + struct io_kiocb *grp_leader; + }; }; struct io_overflow_cqe { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 86cb385fe0b5..f298dd5fbaa9 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -124,6 +124,7 @@ enum io_uring_sqe_flags_bit { IOSQE_ASYNC_BIT, IOSQE_BUFFER_SELECT_BIT, IOSQE_CQE_SKIP_SUCCESS_BIT, + IOSQE_SQE_GROUP_BIT, }; /* @@ -143,6 +144,8 @@ enum io_uring_sqe_flags_bit { #define IOSQE_BUFFER_SELECT (1U << IOSQE_BUFFER_SELECT_BIT) /* don't post CQE if request succeeded */ #define IOSQE_CQE_SKIP_SUCCESS (1U << IOSQE_CQE_SKIP_SUCCESS_BIT) +/* defines sqe group */ +#define IOSQE_SQE_GROUP (1U << IOSQE_SQE_GROUP_BIT) /* * io_uring_setup() flags @@ -554,6 +557,7 @@ struct io_uring_params { #define IORING_FEAT_REG_REG_RING (1U << 13) #define IORING_FEAT_RECVSEND_BUNDLE (1U << 14) #define IORING_FEAT_MIN_TIMEOUT (1U << 15) +#define IORING_FEAT_SQE_GROUP (1U << 16) /* * io_uring_register(2) opcodes and arguments diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 33856560ff87..59e9a01319de 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -112,14 +112,15 @@ IOSQE_IO_HARDLINK | IOSQE_ASYNC) #define SQE_VALID_FLAGS (SQE_COMMON_FLAGS | IOSQE_BUFFER_SELECT | \ - IOSQE_IO_DRAIN | IOSQE_CQE_SKIP_SUCCESS) + IOSQE_IO_DRAIN | IOSQE_CQE_SKIP_SUCCESS | \ + IOSQE_SQE_GROUP) #define IO_REQ_CLEAN_FLAGS (REQ_F_BUFFER_SELECTED | REQ_F_NEED_CLEANUP | \ REQ_F_POLLED | REQ_F_INFLIGHT | REQ_F_CREDS | \ REQ_F_ASYNC_DATA) #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | REQ_F_LINK | REQ_F_HARDLINK |\ - IO_REQ_CLEAN_FLAGS) + REQ_F_SQE_GROUP | IO_REQ_CLEAN_FLAGS) #define IO_TCTX_REFS_CACHE_NR (1U << 10) @@ -912,6 +913,129 @@ static __always_inline void io_req_commit_cqe(struct io_ring_ctx *ctx, } } +/* Can only be called after this request is issued */ +static inline struct io_kiocb *get_group_leader(struct io_kiocb *req) +{ + if (req->flags & REQ_F_SQE_GROUP) { + if (req_is_group_leader(req)) + return req; + return req->grp_leader; + } + return NULL; +} + +void io_fail_group_members(struct io_kiocb *req) +{ + struct io_kiocb *member = req->grp_link; + + while (member) { + struct io_kiocb *next = member->grp_link; + + if (!(member->flags & REQ_F_FAIL)) { + req_set_fail(member); + io_req_set_res(member, -ECANCELED, 0); + } + member = next; + } +} + +static void io_queue_group_members(struct io_kiocb *req) +{ + struct io_kiocb *member = req->grp_link; + + if (!member) + return; + + req->grp_link = NULL; + while (member) { + struct io_kiocb *next = member->grp_link; + + member->grp_leader = req; + if (unlikely(member->flags & REQ_F_FAIL)) { + io_req_task_queue_fail(member, member->cqe.res); + } else if (unlikely(req->flags & REQ_F_FAIL)) { + io_req_task_queue_fail(member, -ECANCELED); + } else { + io_req_task_queue(member); + } + member = next; + } +} + +/* called only after the request is completed */ +static bool req_is_last_group_member(struct io_kiocb *req) +{ + return req->grp_leader != NULL; +} + +static void io_complete_group_req(struct io_kiocb *req) +{ + struct io_kiocb *lead; + + if (req_is_group_leader(req)) { + req->grp_refs -= 1; + return; + } + + lead = get_group_leader(req); + + /* member CQE needs to be posted first */ + if (!(req->flags & REQ_F_CQE_SKIP)) + io_req_commit_cqe(req->ctx, req); + + /* Set leader as failed in case of any member failed */ + if (unlikely((req->flags & REQ_F_FAIL))) + req_set_fail(lead); + + WARN_ON_ONCE(lead->grp_refs <= 0); + if (!--lead->grp_refs) { + /* + * We are the last member, and ->grp_leader isn't cleared, + * so our leader can be found & freed with the last member + */ + if (!(lead->flags & REQ_F_CQE_SKIP)) + io_req_commit_cqe(lead->ctx, lead); + } else { + /* we are done with the group now */ + req->grp_leader = NULL; + } +} + +enum group_mem { + GROUP_LEADER, + GROUP_LAST_MEMBER, + GROUP_OTHER_MEMBER, +}; + +static enum group_mem io_prep_free_group_req(struct io_kiocb *req, + struct io_kiocb **leader) +{ + /* + * Group completion is done, so clear the flag for avoiding double + * handling in case of io-wq + */ + req->flags &= ~REQ_F_SQE_GROUP; + + if (req_is_group_leader(req)) { + /* Queue members now */ + if (req->grp_link) + io_queue_group_members(req); + return GROUP_LEADER; + } else { + if (!req_is_last_group_member(req)) + return GROUP_OTHER_MEMBER; + + /* + * Prepare for freeing leader which can only be found from + * the last member + */ + *leader = req->grp_leader; + (*leader)->flags &= ~REQ_F_SQE_GROUP_LEADER; + req->grp_leader = NULL; + return GROUP_LAST_MEMBER; + } +} + static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; @@ -927,7 +1051,8 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires * the submitter task context, IOPOLL protects with uring_lock. */ - if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL)) { + if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL) || + (req->flags & REQ_F_SQE_GROUP)) { req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); return; @@ -1411,6 +1536,27 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, comp_list); if (unlikely(req->flags & IO_REQ_CLEAN_SLOW_FLAGS)) { + if (req->flags & REQ_F_SQE_GROUP) { + struct io_kiocb *leader = NULL; + enum group_mem mem = io_prep_free_group_req(req, &leader); + + if (mem == GROUP_LEADER) { + node = req->comp_list.next; + continue; + } else if (mem == GROUP_LAST_MEMBER) { + /* + * Link leader to current request's next, + * this way works because the iterator + * always check the next node only. + * + * Be careful when you change the iterator + * in future + */ + wq_stack_add_head(&leader->comp_list, + &req->comp_list); + } + } + if (req->flags & REQ_F_REFCOUNT) { node = req->comp_list.next; if (!req_ref_put_and_test(req)) @@ -1450,8 +1596,16 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); - if (!(req->flags & REQ_F_CQE_SKIP)) - io_req_commit_cqe(ctx, req); + if (unlikely(req->flags & (REQ_F_CQE_SKIP | REQ_F_SQE_GROUP))) { + if (req->flags & REQ_F_SQE_GROUP) { + io_complete_group_req(req); + continue; + } + + if (req->flags & REQ_F_CQE_SKIP) + continue; + } + io_req_commit_cqe(ctx, req); } __io_cq_unlock_post(ctx); @@ -1661,8 +1815,12 @@ static u32 io_get_sequence(struct io_kiocb *req) struct io_kiocb *cur; /* need original cached_sq_head, but it was increased for each req */ - io_for_each_link(cur, req) - seq--; + io_for_each_link(cur, req) { + if (req_is_group_leader(cur)) + seq -= cur->grp_refs; + else + seq--; + } return seq; } @@ -2124,6 +2282,67 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, return def->prep(req, sqe); } +static struct io_kiocb *io_group_sqe(struct io_submit_link *group, + struct io_kiocb *req) +{ + /* + * Group chain is similar with link chain: starts with 1st sqe with + * REQ_F_SQE_GROUP, and ends with the 1st sqe without REQ_F_SQE_GROUP + */ + if (group->head) { + struct io_kiocb *lead = group->head; + + /* + * Members can't be in link chain, can't be drained, but + * the whole group can be linked or drained by setting + * flags on group leader. + * + * IOSQE_CQE_SKIP_SUCCESS can't be set for member + * for the sake of simplicity + */ + if (req->flags & (IO_REQ_LINK_FLAGS | REQ_F_IO_DRAIN | + REQ_F_CQE_SKIP)) + req_fail_link_node(lead, -EINVAL); + + lead->grp_refs += 1; + group->last->grp_link = req; + group->last = req; + + if (req->flags & REQ_F_SQE_GROUP) + return NULL; + + req->grp_link = NULL; + req->flags |= REQ_F_SQE_GROUP; + group->head = NULL; + + return lead; + } else { + if (WARN_ON_ONCE(!(req->flags & REQ_F_SQE_GROUP))) + return req; + group->head = req; + group->last = req; + req->grp_refs = 1; + req->flags |= REQ_F_SQE_GROUP_LEADER; + return NULL; + } +} + +static __cold struct io_kiocb *io_submit_fail_group( + struct io_submit_link *link, struct io_kiocb *req) +{ + struct io_kiocb *lead = link->head; + + /* + * Instead of failing eagerly, continue assembling the group link + * if applicable and mark the leader with REQ_F_FAIL. The group + * flushing code should find the flag and handle the rest + */ + if (lead && !(lead->flags & REQ_F_FAIL)) + req_fail_link_node(lead, -ECANCELED); + + return io_group_sqe(link, req); +} + static __cold int io_submit_fail_link(struct io_submit_link *link, struct io_kiocb *req, int ret) { @@ -2162,11 +2381,18 @@ static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, { struct io_ring_ctx *ctx = req->ctx; struct io_submit_link *link = &ctx->submit_state.link; + struct io_submit_link *group = &ctx->submit_state.group; trace_io_uring_req_failed(sqe, req, ret); req_fail_link_node(req, ret); + if (group->head || (req->flags & REQ_F_SQE_GROUP)) { + req = io_submit_fail_group(group, req); + if (!req) + return 0; + } + /* cover both linked and non-linked request */ return io_submit_fail_link(link, req, ret); } @@ -2210,11 +2436,29 @@ static struct io_kiocb *io_link_sqe(struct io_submit_link *link, return req; } +static inline bool io_group_assembling(const struct io_submit_state *state, + const struct io_kiocb *req) +{ + if (state->group.head || req->flags & REQ_F_SQE_GROUP) + return true; + return false; +} + +/* Failed request is covered too */ +static inline bool io_link_assembling(const struct io_submit_state *state, + const struct io_kiocb *req) +{ + if (state->link.head || (req->flags & (IO_REQ_LINK_FLAGS | + REQ_F_FORCE_ASYNC | REQ_F_FAIL))) + return true; + return false; +} + static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, const struct io_uring_sqe *sqe) __must_hold(&ctx->uring_lock) { - struct io_submit_link *link = &ctx->submit_state.link; + struct io_submit_state *state = &ctx->submit_state; int ret; ret = io_init_req(ctx, req, sqe); @@ -2223,11 +2467,20 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, trace_io_uring_submit_req(req); - if (unlikely(link->head || (req->flags & (IO_REQ_LINK_FLAGS | - REQ_F_FORCE_ASYNC | REQ_F_FAIL)))) { - req = io_link_sqe(link, req); - if (!req) - return 0; + if (unlikely(io_link_assembling(state, req) || + io_group_assembling(state, req))) { + if (io_group_assembling(state, req)) { + req = io_group_sqe(&state->group, req); + if (!req) + return 0; + } + + /* covers non-linked failed request too */ + if (io_link_assembling(state, req)) { + req = io_link_sqe(&state->link, req); + if (!req) + return 0; + } } io_queue_sqe(req); return 0; @@ -2240,8 +2493,27 @@ static void io_submit_state_end(struct io_ring_ctx *ctx) { struct io_submit_state *state = &ctx->submit_state; - if (unlikely(state->link.head)) - io_queue_sqe_fallback(state->link.head); + if (unlikely(state->group.head || state->link.head)) { + /* the last member must set REQ_F_SQE_GROUP */ + if (state->group.head) { + struct io_kiocb *lead = state->group.head; + struct io_kiocb *last = state->group.last; + + /* fail group with single leader */ + if (unlikely(last == lead)) + req_fail_link_node(lead, -EINVAL); + + last->grp_link = NULL; + if (state->link.head) + io_link_sqe(&state->link, lead); + else + io_queue_sqe_fallback(lead); + } + + if (unlikely(state->link.head)) + io_queue_sqe_fallback(state->link.head); + } + /* flush only after queuing links as they can generate completions */ io_submit_flush_completions(ctx); if (state->plug_started) @@ -2259,6 +2531,7 @@ static void io_submit_state_start(struct io_submit_state *state, state->submit_nr = max_ios; /* set only head, no need to init link_last in advance */ state->link.head = NULL; + state->group.head = NULL; } static void io_commit_sqring(struct io_ring_ctx *ctx) @@ -3699,7 +3972,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS | IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP | IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING | - IORING_FEAT_RECVSEND_BUNDLE | IORING_FEAT_MIN_TIMEOUT; + IORING_FEAT_RECVSEND_BUNDLE | IORING_FEAT_MIN_TIMEOUT | + IORING_FEAT_SQE_GROUP; if (copy_to_user(params, p, sizeof(*p))) { ret = -EFAULT; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 12e8fca73891..ab84b09505fe 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -72,6 +72,7 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags void io_add_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); +void io_fail_group_members(struct io_kiocb *req); struct file *io_file_get_normal(struct io_kiocb *req, int fd); struct file *io_file_get_fixed(struct io_kiocb *req, int fd, @@ -343,6 +344,11 @@ static inline void io_tw_lock(struct io_ring_ctx *ctx, struct io_tw_state *ts) lockdep_assert_held(&ctx->uring_lock); } +static inline bool req_is_group_leader(struct io_kiocb *req) +{ + return req->flags & REQ_F_SQE_GROUP_LEADER; +} + /* * Don't complete immediately but use deferred completion infrastructure. * Protected by ->uring_lock and can only be used either with diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 9973876d91b0..ed6c74f1a475 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -149,6 +149,8 @@ static void io_req_tw_fail_links(struct io_kiocb *link, struct io_tw_state *ts) res = link->cqe.res; link->link = NULL; io_req_set_res(link, res, 0); + if (req_is_group_leader(link)) + io_fail_group_members(link); io_req_task_complete(link, ts); link = nxt; } @@ -543,6 +545,10 @@ static int __io_timeout_prep(struct io_kiocb *req, if (is_timeout_link) { struct io_submit_link *link = &req->ctx->submit_state.link; + /* so far disallow IO group link timeout */ + if (req->ctx->submit_state.group.head) + return -EINVAL; + if (!link->head) return -EINVAL; if (link->last->opcode == IORING_OP_LINK_TIMEOUT) From patchwork Fri Oct 25 12:22:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13850618 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 809361E8845 for ; Fri, 25 Oct 2024 12:23:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859010; cv=none; b=sQYjvyEj/aF2x6k87OJZScfHIoZZzJCmSaIiy5P3YApIwYEl96yJK/Rz4lyUTq/2TQWc0gmaqGpHc7avjWK2wQpA5igpobGeGKuEyvuMfhGVnpswIqVoQCLHvrjIZqUzjRsin2a2u0ukvltWyV4oGQsXEUCGKqZGA+eFX5lnI9A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859010; c=relaxed/simple; bh=fSMWNn+ONn5dZNsGxSl5FveuCPKsZHGURPJCVH24JSI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=q74RUbPi1RE8dmchI8ovM/WAFE1Zz7RAszJMaZoDJO6sZhsFVDV3M3TyYI5fBmEiIgMOzFy1ME0wPN5imOdaUKOgQrnAvmm7C5QlP4tUgsxtNJNxNbjt8II1AwrfD9dpjZ33GVIns+7T+egPXIQ1dWraec9/2A2ISKNkmapOdp0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=e7dWe+rd; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="e7dWe+rd" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1729859005; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XGKgZZzVBhXd6AA47MqISSpolRGetJmGOpxv7n5vAmQ=; b=e7dWe+rdnFyVGfu7kFpzkVVVTiollrjq4zswkT6ddefsaDTv+tGlBqg8OJnbLdHA1kT0MX 9vl8yILiohUpxDfmBgnR+acjWQygXlKkcqTBLsho7fueTL+e0ySIDk+DpwNkSOXKPTo3jK qUWvU+pn/nDOVt5Seg+eCYnCo3NpNJ4= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-332-wLMQAlM1PpeWlNv23d0WWA-1; Fri, 25 Oct 2024 08:23:22 -0400 X-MC-Unique: wLMQAlM1PpeWlNv23d0WWA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 295D419560A1; Fri, 25 Oct 2024 12:23:21 +0000 (UTC) Received: from localhost (unknown [10.72.116.106]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2398A196BB7E; Fri, 25 Oct 2024 12:23:19 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org, Pavel Begunkov Cc: linux-block@vger.kernel.org, Uday Shankar , Akilesh Kailash , Ming Lei Subject: [PATCH V8 5/7] io_uring: support leased group buffer with REQ_F_GROUP_KBUF Date: Fri, 25 Oct 2024 20:22:42 +0800 Message-ID: <20241025122247.3709133-6-ming.lei@redhat.com> In-Reply-To: <20241025122247.3709133-1-ming.lei@redhat.com> References: <20241025122247.3709133-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 SQE group introduces one new mechanism to share resource among one group of requests, and all member requests can consume the resource leased by group leader efficiently in parallel. This patch uses the added SQE group to lease kernel buffer from group leader(driver) to members(io_uring) in sqe group: - this kernel buffer is owned by kernel device(driver), and has very short lifetime, such as, it is often aligned with block IO lifetime - group leader leases the kernel buffer from driver to member requests of io_uring subsystem - member requests uses the leased buffer to do FS or network IO, or more operations in future; IOSQE_IO_DRAIN bit isn't used for group member IO, so it is mapped to GROUP_KBUF; the actual use becomes very similar with buffer select. - this kernel buffer is returned back after all member requests consume it io_uring builtin provide/register buffer isn't one good match for this use case: - complicated dependency on add/remove buffer this buffer has to be added/removed to one global table by add/remove OPs, and all consumer OPs have to sync with the add/remove OPs; either consumer OPs have to by issued one by one with IO_LINK; or two extra syscall are added for one time of buffer consumption, this way slows down ublk io handling, and may lose zero copy value - application becomes more complicated - application may panic and the kernel buffer is left in io_uring, which complicates io_uring shutdown handling since returning back buffer needs to cowork with buffer owner - big change is needed in io_uring provide/register buffer - the requirement is just to lease the kernel buffer to io_uring subsystem for very short time, not necessary to move it into io_uring and make it global This way looks a bit similar with kernel's pipe/splice, but there are some important differences: - splice is for transferring data between two FDs via pipe, and fd_out can only read data from pipe, but data can't be written to; this feature can lease buffer from group leader(driver subsystem) to members(io_uring subsystem), so member request can write data to this buffer if the buffer direction is allowed to write to. - splice implements data transfer by moving pages between subsystem and pipe, that means page ownership is transferred, and this way is one of the most complicated thing of splice; this patch supports scenarios in which the buffer can't be transferred, and buffer is only borrowed to member requests for consumption, and is returned back after member requests consume the leased buffer, so buffer lifetime is aligned with group leader lifetime, and buffer lifetime is simplified a lot. Especially the buffer is guaranteed to be returned back. - splice can't run in async way basically It can help to implement generic zero copy between device and related operations, such as ublk, fuse, vdpa. Signed-off-by: Ming Lei --- include/linux/io_uring_types.h | 40 +++++++++++++++++++++++ io_uring/io_uring.c | 22 ++++++++++--- io_uring/io_uring.h | 5 +++ io_uring/kbuf.c | 58 ++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 31 ++++++++++++++++++ io_uring/net.c | 25 ++++++++++++++- io_uring/rw.c | 26 ++++++++++++++- 7 files changed, 201 insertions(+), 6 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index d524be7f6b35..890bcb5d0c26 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -6,6 +6,7 @@ #include #include #include +#include #include enum { @@ -39,6 +40,26 @@ enum io_uring_cmd_flags { IO_URING_F_COMPAT = (1 << 12), }; +struct io_uring_kernel_buf; +typedef void (io_uring_buf_giveback_t) (const struct io_uring_kernel_buf *); + +/* kernel owned buffer, leased to io_uring OPs */ +struct io_uring_kernel_buf { + unsigned long len; + unsigned short nr_bvecs; + unsigned char dir; /* ITER_SOURCE or ITER_DEST */ + + /* offset in the 1st bvec */ + unsigned int offset; + const struct bio_vec *bvec; + + /* called when we are done with this buffer */ + io_uring_buf_giveback_t *grp_kbuf_ack; + + /* private field, user don't touch it */ + struct bio_vec __bvec[]; +}; + struct io_wq_work_node { struct io_wq_work_node *next; }; @@ -469,6 +490,7 @@ enum { REQ_F_BL_NO_RECYCLE_BIT, REQ_F_BUFFERS_COMMIT_BIT, REQ_F_SQE_GROUP_LEADER_BIT, + REQ_F_GROUP_KBUF_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -549,6 +571,15 @@ enum { REQ_F_BUFFERS_COMMIT = IO_REQ_FLAG(REQ_F_BUFFERS_COMMIT_BIT), /* sqe group lead */ REQ_F_SQE_GROUP_LEADER = IO_REQ_FLAG(REQ_F_SQE_GROUP_LEADER_BIT), + /* + * Group leader leases kbuf to io_uring. Set for leader when the + * leader starts to lease kbuf, and set for member in case that + * the member needs to consume the group kbuf + * + * For group member, this flag is mapped from IOSQE_IO_DRAIN which + * isn't used for group members + */ + REQ_F_GROUP_KBUF = IO_REQ_FLAG(REQ_F_GROUP_KBUF_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); @@ -629,6 +660,15 @@ struct io_kiocb { * REQ_F_BUFFER_RING is set. */ struct io_buffer_list *buf_list; + + /* + * store kernel buffer leased from sqe group lead, valid + * IFF REQ_F_GROUP_KBUF is set + * + * The buffer meta is immutable since it is shared by + * all member requests + */ + const struct io_uring_kernel_buf *grp_kbuf; }; union { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 59e9a01319de..fcf58d5e698a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -117,7 +117,7 @@ #define IO_REQ_CLEAN_FLAGS (REQ_F_BUFFER_SELECTED | REQ_F_NEED_CLEANUP | \ REQ_F_POLLED | REQ_F_INFLIGHT | REQ_F_CREDS | \ - REQ_F_ASYNC_DATA) + REQ_F_ASYNC_DATA | REQ_F_GROUP_KBUF) #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | REQ_F_LINK | REQ_F_HARDLINK |\ REQ_F_SQE_GROUP | IO_REQ_CLEAN_FLAGS) @@ -397,6 +397,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq) static void io_clean_op(struct io_kiocb *req) { + if (req->flags & REQ_F_GROUP_KBUF) + io_drop_leased_grp_kbuf(req); if (req->flags & REQ_F_BUFFER_SELECTED) { spin_lock(&req->ctx->completion_lock); io_kbuf_drop(req); @@ -1022,6 +1024,12 @@ static enum group_mem io_prep_free_group_req(struct io_kiocb *req, io_queue_group_members(req); return GROUP_LEADER; } else { + /* + * Clear GROUP_KBUF since we are done with leased group + * buffer + */ + if (req->flags & REQ_F_GROUP_KBUF) + req->flags &= ~REQ_F_GROUP_KBUF; if (!req_is_last_group_member(req)) return GROUP_OTHER_MEMBER; @@ -2223,9 +2231,15 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, if (sqe_flags & IOSQE_CQE_SKIP_SUCCESS) ctx->drain_disabled = true; if (sqe_flags & IOSQE_IO_DRAIN) { - if (ctx->drain_disabled) - return io_init_fail_req(req, -EOPNOTSUPP); - io_init_req_drain(req); + /* IO_DRAIN is mapped to GROUP_KBUF for group members */ + if (ctx->submit_state.group.head) { + req->flags &= ~REQ_F_IO_DRAIN; + req->flags |= REQ_F_GROUP_KBUF; + } else { + if (ctx->drain_disabled) + return io_init_fail_req(req, -EOPNOTSUPP); + io_init_req_drain(req); + } } } if (unlikely(ctx->restricted || ctx->drain_active || ctx->drain_next)) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index ab84b09505fe..f83e7c13e679 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -349,6 +349,11 @@ static inline bool req_is_group_leader(struct io_kiocb *req) return req->flags & REQ_F_SQE_GROUP_LEADER; } +static inline bool req_is_group_member(struct io_kiocb *req) +{ + return !req_is_group_leader(req) && (req->flags & REQ_F_SQE_GROUP); +} + /* * Don't complete immediately but use deferred completion infrastructure. * Protected by ->uring_lock and can only be used either with diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index d407576ddfb7..e86dacc7a822 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -838,3 +838,61 @@ int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma) io_put_bl(ctx, bl); return ret; } + +int io_lease_group_kbuf(struct io_kiocb *req, + const struct io_uring_kernel_buf *grp_kbuf) +{ + if (!(req->flags & REQ_F_SQE_GROUP_LEADER)) + return -EINVAL; + + if (req->flags & REQ_F_BUFFER_SELECT) + return -EINVAL; + + if (!grp_kbuf->grp_kbuf_ack || !grp_kbuf->bvec) + return -EINVAL; + + /* + * Allow io_uring OPs to borrow this leased kbuf, which is returned + * back by calling `grp_kbuf_ack` when the group leader is freed. + * + * Not like pipe/splice, this kernel buffer is always owned by the + * provider, and has to be returned back. + */ + req->grp_kbuf = grp_kbuf; + req->flags |= REQ_F_GROUP_KBUF; + return 0; +} + +int io_import_group_kbuf(struct io_kiocb *req, unsigned long buf_off, + unsigned int len, int dir, struct iov_iter *iter) +{ + struct io_kiocb *lead = req->grp_leader; + const struct io_uring_kernel_buf *kbuf; + unsigned long offset; + + if (!req_is_group_member(req)) + return -EINVAL; + + if (!lead || !(lead->flags & REQ_F_GROUP_KBUF)) + return -EINVAL; + + kbuf = lead->grp_kbuf; + offset = kbuf->offset; + + if (dir != kbuf->dir) + return -EINVAL; + + if (unlikely(buf_off > kbuf->len)) + return -EFAULT; + + if (unlikely(len > kbuf->len - buf_off)) + return -EFAULT; + + offset += buf_off; + iov_iter_bvec(iter, dir, kbuf->bvec, kbuf->nr_bvecs, offset + len); + + if (offset) + iov_iter_advance(iter, offset); + + return 0; +} diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 36aadfe5ac00..d54cd4312db9 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -89,6 +89,11 @@ struct io_buffer_list *io_pbuf_get_bl(struct io_ring_ctx *ctx, unsigned long bgid); int io_pbuf_mmap(struct file *file, struct vm_area_struct *vma); +int io_lease_group_kbuf(struct io_kiocb *req, + const struct io_uring_kernel_buf *grp_kbuf); +int io_import_group_kbuf(struct io_kiocb *req, unsigned long buf_off, + unsigned int len, int dir, struct iov_iter *iter); + static inline bool io_kbuf_recycle_ring(struct io_kiocb *req) { /* @@ -220,4 +225,30 @@ static inline unsigned int io_put_kbufs(struct io_kiocb *req, int len, { return __io_put_kbufs(req, len, nbufs, issue_flags); } + +static inline bool io_use_leased_grp_kbuf(struct io_kiocb *req) +{ + /* can't use group kbuf in case of buffer select or fixed buffer */ + if (req->flags & REQ_F_BUFFER_SELECT) + return false; + + return req->flags & REQ_F_GROUP_KBUF; +} + +static inline void io_drop_leased_grp_kbuf(struct io_kiocb *req) +{ + const struct io_uring_kernel_buf *gbuf = req->grp_kbuf; + + if (gbuf) + gbuf->grp_kbuf_ack(gbuf); +} + +/* zero remained bytes of kernel buffer for avoiding to leak daata */ +static inline void io_req_zero_remained(struct io_kiocb *req, struct iov_iter *iter) +{ + size_t left = iov_iter_count(iter); + + if (iov_iter_rw(iter) == READ && left > 0) + iov_iter_zero(left, iter); +} #endif diff --git a/io_uring/net.c b/io_uring/net.c index 2040195e33ab..c7d58a0c38c3 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -88,6 +88,13 @@ struct io_sr_msg { */ #define MULTISHOT_MAX_RETRY 32 +#define user_ptr_to_u64(x) ( \ +{ \ + typecheck(void __user *, (x)); \ + (u64)(unsigned long)(x); \ +} \ +) + int io_shutdown_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_shutdown *shutdown = io_kiocb_to_cmd(req, struct io_shutdown); @@ -384,7 +391,7 @@ static int io_send_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe) kmsg->msg.msg_name = &kmsg->addr; kmsg->msg.msg_namelen = addr_len; } - if (!io_do_buffer_select(req)) { + if (!io_do_buffer_select(req) && !io_use_leased_grp_kbuf(req)) { ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &kmsg->msg.msg_iter); if (unlikely(ret < 0)) @@ -599,6 +606,15 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (issue_flags & IO_URING_F_NONBLOCK) flags |= MSG_DONTWAIT; + if (io_use_leased_grp_kbuf(req)) { + ret = io_import_group_kbuf(req, + user_ptr_to_u64(sr->buf), + sr->len, ITER_SOURCE, + &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ret; + } + retry_bundle: if (io_do_buffer_select(req)) { struct buf_sel_arg arg = { @@ -889,6 +905,8 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, *ret = IOU_STOP_MULTISHOT; else *ret = IOU_OK; + if (io_use_leased_grp_kbuf(req)) + io_req_zero_remained(req, &kmsg->msg.msg_iter); io_req_msg_cleanup(req, issue_flags); return true; } @@ -1161,6 +1179,11 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) goto out_free; } sr->buf = NULL; + } else if (io_use_leased_grp_kbuf(req)) { + ret = io_import_group_kbuf(req, user_ptr_to_u64(sr->buf), + sr->len, ITER_DEST, &kmsg->msg.msg_iter); + if (unlikely(ret)) + goto out_free; } kmsg->msg.msg_flags = 0; diff --git a/io_uring/rw.c b/io_uring/rw.c index 4bc0d762627d..5a2025d48804 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -245,7 +245,8 @@ static int io_prep_rw_setup(struct io_kiocb *req, int ddir, bool do_import) if (io_rw_alloc_async(req)) return -ENOMEM; - if (!do_import || io_do_buffer_select(req)) + if (!do_import || io_do_buffer_select(req) || + io_use_leased_grp_kbuf(req)) return 0; rw = req->async_data; @@ -489,6 +490,11 @@ static bool __io_complete_rw_common(struct io_kiocb *req, long res) } req_set_fail(req); req->cqe.res = res; + if (io_use_leased_grp_kbuf(req)) { + struct io_async_rw *io = req->async_data; + + io_req_zero_remained(req, &io->iter); + } } return false; } @@ -630,11 +636,16 @@ static inline loff_t *io_kiocb_ppos(struct kiocb *kiocb) */ static ssize_t loop_rw_iter(int ddir, struct io_rw *rw, struct iov_iter *iter) { + struct io_kiocb *req = cmd_to_io_kiocb(rw); struct kiocb *kiocb = &rw->kiocb; struct file *file = kiocb->ki_filp; ssize_t ret = 0; loff_t *ppos; + /* group buffer is kernel buffer and doesn't have userspace addr */ + if (io_use_leased_grp_kbuf(req)) + return -EOPNOTSUPP; + /* * Don't support polled IO through this interface, and we can't * support non-blocking either. For the latter, this just causes @@ -841,6 +852,12 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) ret = io_import_iovec(ITER_DEST, req, io, issue_flags); if (unlikely(ret < 0)) return ret; + } else if (io_use_leased_grp_kbuf(req)) { + ret = io_import_group_kbuf(req, rw->addr, rw->len, ITER_DEST, + &io->iter); + if (unlikely(ret)) + return ret; + iov_iter_save_state(&io->iter, &io->iter_state); } ret = io_rw_init_file(req, FMODE_READ, READ); if (unlikely(ret)) @@ -1024,6 +1041,13 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret, ret2; loff_t *ppos; + if (io_use_leased_grp_kbuf(req)) { + ret = io_import_group_kbuf(req, rw->addr, rw->len, ITER_SOURCE, + &io->iter); + if (unlikely(ret)) + return ret; + } + ret = io_rw_init_file(req, FMODE_WRITE, WRITE); if (unlikely(ret)) return ret; From patchwork Fri Oct 25 12:22:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13850622 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44468203706 for ; Fri, 25 Oct 2024 12:24:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859047; cv=none; b=FYvf/N3OIqh1DI+poYs+vwyLrZ/2V6cGXUCc9JqMIc1LeqNNw3LnmYwyCxJQIPN6eW6DVq/UIS+75pMvbsxtxS5yn98KAXwatC3xihUiMKvNNZiUwJOwiRmVdTCPGnAUzeiqQpOcRVscvyvsO4szL3HykQ02RGraK59EHnpBCcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859047; c=relaxed/simple; bh=vABzLKNvlOxdyet/MKsBKs8rtzdGuPcPBOr6mAJI6sM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DJdjRlfWCdPuKxgKoNgJwhs0Yzw4g99aEOh/ccU/EQ/Znks/xRKTT/19MfHGBxO0IBcLvOVw+K9culWvQpy2vSDBnxlym8NFmuMySDpgUw40olFq7jHty0vNb08zuRdslTGmgTn5xRMgJoe2SCPnJktz4rnFWQ0ajYujHOxJ39g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bUARgdsU; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bUARgdsU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1729859044; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cywKv9TEoDTr7tj5I18bYRcgvSakRA47XCPWDnlogL0=; b=bUARgdsUzLx0jXb2Yfz+sBI8K9d5t1VgWPOg3YueBeZuAxOA+99OD5wniOsEV9GV9/73v5 GFtGZ/tNm9ls0NoMguamCaSfLH0C03JFYivM82gH4xMMmrjWQqIBJGDF70ipcazdmFJVxP i9QzZNcyIpraDBhNNQdRd/EeeNwJOKo= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-164-HydeUNApNS2RR6AwEGz7nw-1; Fri, 25 Oct 2024 08:23:57 -0400 X-MC-Unique: HydeUNApNS2RR6AwEGz7nw-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 281A5191379F; Fri, 25 Oct 2024 12:23:25 +0000 (UTC) Received: from localhost (unknown [10.72.116.106]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 26A4E196BB7D; Fri, 25 Oct 2024 12:23:23 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org, Pavel Begunkov Cc: linux-block@vger.kernel.org, Uday Shankar , Akilesh Kailash , Ming Lei Subject: [PATCH V8 6/7] io_uring/uring_cmd: support leasing device kernel buffer to io_uring Date: Fri, 25 Oct 2024 20:22:43 +0800 Message-ID: <20241025122247.3709133-7-ming.lei@redhat.com> In-Reply-To: <20241025122247.3709133-1-ming.lei@redhat.com> References: <20241025122247.3709133-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Add API of io_uring_cmd_lease_kbuf() for driver to lease its kernel buffer to io_uring. The leased buffer can only be consumed by io_uring OPs in group wide, and the uring_cmd has to be one group leader. This way can support generic device zero copy over device buffer in userspace: - create one sqe group - lease one device buffer to io_uring by the group leader of uring_cmd - io_uring member OPs consume this kernel buffer by passing IOSQE_IO_DRAIN which isn't used for group member, and mapped to GROUP_KBUF. - the kernel buffer is returned back after all member OPs are completed Signed-off-by: Ming Lei --- include/linux/io_uring/cmd.h | 7 +++++++ io_uring/uring_cmd.c | 13 +++++++++++++ 2 files changed, 20 insertions(+) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index c189d36ad55e..bdf7abfa0d8a 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -60,6 +60,8 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, /* Execute the request from a blocking context */ void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd); +int io_uring_cmd_lease_kbuf(struct io_uring_cmd *ioucmd, + const struct io_uring_kernel_buf *grp_kbuf); #else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd) @@ -82,6 +84,11 @@ static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, static inline void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd) { } +static inline int io_uring_cmd_lease_kbuf(struct io_uring_cmd *ioucmd, + const struct io_uring_kernel_buf *grp_kbuf) +{ + return -EOPNOTSUPP; +} #endif /* diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 6994f60d7ec7..2c9c2c60c6cd 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -15,6 +15,7 @@ #include "alloc_cache.h" #include "rsrc.h" #include "uring_cmd.h" +#include "kbuf.h" static struct uring_cache *io_uring_async_get(struct io_kiocb *req) { @@ -175,6 +176,18 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, } EXPORT_SYMBOL_GPL(io_uring_cmd_done); +int io_uring_cmd_lease_kbuf(struct io_uring_cmd *ioucmd, + const struct io_uring_kernel_buf *grp_kbuf) +{ + struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); + + if (unlikely(ioucmd->flags & IORING_URING_CMD_FIXED)) + return -EINVAL; + + return io_lease_group_kbuf(req, grp_kbuf); +} +EXPORT_SYMBOL_GPL(io_uring_cmd_lease_kbuf); + static int io_uring_cmd_prep_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe) { From patchwork Fri Oct 25 12:22:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13850619 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 983321E1A39 for ; Fri, 25 Oct 2024 12:23:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859018; cv=none; b=IVfBuMe/2L3F7NAP2XSP63H+yk+40zukB+wGTzUirMSYEpzbyF8iWuXE6+kKk79pKWD//dgu99MTJbsNIN3YBqsYhuUQnlSp4R3Vq91gGsOBKCLIgju/BNed17HXW8nRq6515paJNPMYGpiga4CWIJrJ8q1ERq/XJ4K1KS89erA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729859018; c=relaxed/simple; bh=tAKJ9uQERdUXsFTXzK0hCtlYYLjOFJvSKdnHq2fVe1A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Khue4Vvw+jFejqqjingdg7l4T12OVSQtJL4qgDVj9TFQ6pDsPBW/XLZ47iWyOHOAqIZc/0t0CC95XzQK1Gtdcgv00sXkPqUKO/pdZqE8UJ/8Nq98dgkxtWhQ3/lwLdu+ZzauxnKGjmUWRtrCWAY49MmQITUvQIrmL61350MPCGw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=GdpVnD0/; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GdpVnD0/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1729859014; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iTG9uVZ1qTRHGlOY52yOlg+0h9oMmOzlV+ulUB080YI=; b=GdpVnD0/nDNwNJxaQ+rcerbsOFMk03q29jJQI2TsZ53Fy/iEWTjKcoSy0Dejq6Mqimm+dg 2YtxTVhpbq5mbCf6G+Fb46/ZUbQWJlhivMSp6cOhfs0DlIMCo6GejBvB/NwnTmbnu6EE5i c7Dc11NbfFG+dqiWCrYZst6qz2wKQe0= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-661-4dlm_SxcOqWIxu7ucRYLPA-1; Fri, 25 Oct 2024 08:23:30 -0400 X-MC-Unique: 4dlm_SxcOqWIxu7ucRYLPA-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1A57B1955F41; Fri, 25 Oct 2024 12:23:29 +0000 (UTC) Received: from localhost (unknown [10.72.116.106]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0D1241956056; Fri, 25 Oct 2024 12:23:27 +0000 (UTC) From: Ming Lei To: Jens Axboe , io-uring@vger.kernel.org, Pavel Begunkov Cc: linux-block@vger.kernel.org, Uday Shankar , Akilesh Kailash , Ming Lei Subject: [PATCH V8 7/7] ublk: support leasing io buffer to io_uring Date: Fri, 25 Oct 2024 20:22:44 +0800 Message-ID: <20241025122247.3709133-8-ming.lei@redhat.com> In-Reply-To: <20241025122247.3709133-1-ming.lei@redhat.com> References: <20241025122247.3709133-1-ming.lei@redhat.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Suopport to lease block IO buffer for userpace to run io_uring operations(FS, network IO), then ublk zero copy can be supported. userspace code: git clone https://github.com/ublk-org/ublksrv.git -b uring_group And both loop and nbd zero copy(io_uring send and send zc) are covered. Performance improvement is quite obvious in big block size test, such as 'loop --buffered_io' perf is doubled in 64KB block test("loop/007 vs loop/009"). Signed-off-by: Ming Lei --- drivers/block/ublk_drv.c | 159 ++++++++++++++++++++++++++++++++-- include/uapi/linux/ublk_cmd.h | 11 ++- 2 files changed, 160 insertions(+), 10 deletions(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index a6c8e5cc6051..91d32ebcad0c 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -51,6 +51,8 @@ /* private ioctl command mirror */ #define UBLK_CMD_DEL_DEV_ASYNC _IOC_NR(UBLK_U_CMD_DEL_DEV_ASYNC) +#define UBLK_IO_PROVIDE_IO_BUF _IOC_NR(UBLK_U_IO_PROVIDE_IO_BUF) + /* All UBLK_F_* have to be included into UBLK_F_ALL */ #define UBLK_F_ALL (UBLK_F_SUPPORT_ZERO_COPY \ | UBLK_F_URING_CMD_COMP_IN_TASK \ @@ -71,6 +73,9 @@ struct ublk_rq_data { struct llist_node node; struct kref ref; + + bool allocated_bvec; + struct io_uring_kernel_buf buf[0]; }; struct ublk_uring_cmd_pdu { @@ -189,11 +194,15 @@ struct ublk_params_header { __u32 types; }; +static inline struct request *__ublk_check_and_get_req(struct ublk_device *ub, + struct ublk_queue *ubq, int tag, size_t offset); static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq); static inline unsigned int ublk_req_build_flags(struct request *req); static inline struct ublksrv_io_desc *ublk_get_iod(struct ublk_queue *ubq, int tag); +static void ublk_io_buf_giveback_cb(const struct io_uring_kernel_buf *buf); + static inline bool ublk_dev_is_user_copy(const struct ublk_device *ub) { return ub->dev_info.flags & UBLK_F_USER_COPY; @@ -588,6 +597,11 @@ static inline bool ublk_need_req_ref(const struct ublk_queue *ubq) return ublk_support_user_copy(ubq); } +static inline bool ublk_support_zc(const struct ublk_queue *ubq) +{ + return ubq->flags & UBLK_F_SUPPORT_ZERO_COPY; +} + static inline void ublk_init_req_ref(const struct ublk_queue *ubq, struct request *req) { @@ -851,6 +865,71 @@ static size_t ublk_copy_user_pages(const struct request *req, return done; } +/* + * The built command buffer is immutable, so it is fine to feed it to + * concurrent io_uring provide buf commands + */ +static int ublk_init_zero_copy_buffer(struct request *req) +{ + struct ublk_rq_data *data = blk_mq_rq_to_pdu(req); + struct io_uring_kernel_buf *imu = data->buf; + struct req_iterator rq_iter; + unsigned int nr_bvecs = 0; + struct bio_vec *bvec; + unsigned int offset; + struct bio_vec bv; + + if (!ublk_rq_has_data(req)) + goto exit; + + rq_for_each_bvec(bv, req, rq_iter) + nr_bvecs++; + + if (!nr_bvecs) + goto exit; + + if (req->bio != req->biotail) { + int idx = 0; + + bvec = kvmalloc_array(nr_bvecs, sizeof(struct bio_vec), + GFP_NOIO); + if (!bvec) + return -ENOMEM; + + offset = 0; + rq_for_each_bvec(bv, req, rq_iter) + bvec[idx++] = bv; + data->allocated_bvec = true; + } else { + struct bio *bio = req->bio; + + offset = bio->bi_iter.bi_bvec_done; + bvec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); + } + imu->bvec = bvec; + imu->nr_bvecs = nr_bvecs; + imu->offset = offset; + imu->len = blk_rq_bytes(req); + imu->dir = req_op(req) == REQ_OP_READ ? ITER_DEST : ITER_SOURCE; + imu->grp_kbuf_ack = ublk_io_buf_giveback_cb; + + return 0; +exit: + imu->bvec = NULL; + return 0; +} + +static void ublk_deinit_zero_copy_buffer(struct request *req) +{ + struct ublk_rq_data *data = blk_mq_rq_to_pdu(req); + struct io_uring_kernel_buf *imu = data->buf; + + if (data->allocated_bvec) { + kvfree(imu->bvec); + data->allocated_bvec = false; + } +} + static inline bool ublk_need_map_req(const struct request *req) { return ublk_rq_has_data(req) && req_op(req) == REQ_OP_WRITE; @@ -862,13 +941,25 @@ static inline bool ublk_need_unmap_req(const struct request *req) (req_op(req) == REQ_OP_READ || req_op(req) == REQ_OP_DRV_IN); } -static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req, +static int ublk_map_io(const struct ublk_queue *ubq, struct request *req, struct ublk_io *io) { const unsigned int rq_bytes = blk_rq_bytes(req); - if (ublk_support_user_copy(ubq)) + if (ublk_support_user_copy(ubq)) { + if (ublk_support_zc(ubq)) { + int ret = ublk_init_zero_copy_buffer(req); + + /* + * The only failure is -ENOMEM for allocating providing + * buffer command, return zero so that we can requeue + * this req. + */ + if (unlikely(ret)) + return 0; + } return rq_bytes; + } /* * no zero copy, we delay copy WRITE request data into ublksrv @@ -886,13 +977,16 @@ static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req, } static int ublk_unmap_io(const struct ublk_queue *ubq, - const struct request *req, + struct request *req, struct ublk_io *io) { const unsigned int rq_bytes = blk_rq_bytes(req); - if (ublk_support_user_copy(ubq)) + if (ublk_support_user_copy(ubq)) { + if (ublk_support_zc(ubq)) + ublk_deinit_zero_copy_buffer(req); return rq_bytes; + } if (ublk_need_unmap_req(req)) { struct iov_iter iter; @@ -1038,6 +1132,7 @@ static inline void __ublk_complete_rq(struct request *req) return; exit: + ublk_deinit_zero_copy_buffer(req); blk_mq_end_request(req, res); } @@ -1680,6 +1775,45 @@ static inline void ublk_prep_cancel(struct io_uring_cmd *cmd, io_uring_cmd_mark_cancelable(cmd, issue_flags); } +static void ublk_io_buf_giveback_cb(const struct io_uring_kernel_buf *buf) +{ + struct ublk_rq_data *data = container_of(buf, struct ublk_rq_data, buf[0]); + struct request *req = blk_mq_rq_from_pdu(data); + struct ublk_queue *ubq = req->mq_hctx->driver_data; + + ublk_put_req_ref(ubq, req); +} + +static int ublk_provide_io_buf(struct io_uring_cmd *cmd, + struct ublk_queue *ubq, int tag) +{ + struct ublk_device *ub = cmd->file->private_data; + struct ublk_rq_data *data; + struct request *req; + + if (!ub) + return -EPERM; + + req = __ublk_check_and_get_req(ub, ubq, tag, 0); + if (!req) + return -EINVAL; + + pr_devel("%s: qid %d tag %u request bytes %u\n", + __func__, tag, ubq->q_id, blk_rq_bytes(req)); + + data = blk_mq_rq_to_pdu(req); + + /* + * io_uring guarantees that the callback will be called after + * the provided buffer is consumed, and it is automatic removal + * before this uring command is freed. + * + * This request won't be completed unless the callback is called, + * so ublk module won't be unloaded too. + */ + return io_uring_cmd_lease_kbuf(cmd, data->buf); +} + static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags, const struct ublksrv_io_cmd *ub_cmd) @@ -1731,6 +1865,10 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, ret = -EINVAL; switch (_IOC_NR(cmd_op)) { + case UBLK_IO_PROVIDE_IO_BUF: + if (unlikely(!ublk_support_zc(ubq))) + goto out; + return ublk_provide_io_buf(cmd, ubq, tag); case UBLK_IO_FETCH_REQ: /* UBLK_IO_FETCH_REQ is only allowed before queue is setup */ if (ublk_queue_ready(ubq)) { @@ -2149,11 +2287,14 @@ static void ublk_align_max_io_size(struct ublk_device *ub) static int ublk_add_tag_set(struct ublk_device *ub) { + int zc = !!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY); + struct ublk_rq_data *data; + ub->tag_set.ops = &ublk_mq_ops; ub->tag_set.nr_hw_queues = ub->dev_info.nr_hw_queues; ub->tag_set.queue_depth = ub->dev_info.queue_depth; ub->tag_set.numa_node = NUMA_NO_NODE; - ub->tag_set.cmd_size = sizeof(struct ublk_rq_data); + ub->tag_set.cmd_size = struct_size(data, buf, zc); ub->tag_set.flags = BLK_MQ_F_SHOULD_MERGE; ub->tag_set.driver_data = ub; return blk_mq_alloc_tag_set(&ub->tag_set); @@ -2449,8 +2590,12 @@ static int ublk_ctrl_add_dev(struct io_uring_cmd *cmd) goto out_free_dev_number; } - /* We are not ready to support zero copy */ - ub->dev_info.flags &= ~UBLK_F_SUPPORT_ZERO_COPY; + /* zero copy depends on user copy */ + if ((ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY) && + !ublk_dev_is_user_copy(ub)) { + ret = -EINVAL; + goto out_free_dev_number; + } ub->dev_info.nr_hw_queues = min_t(unsigned int, ub->dev_info.nr_hw_queues, nr_cpu_ids); diff --git a/include/uapi/linux/ublk_cmd.h b/include/uapi/linux/ublk_cmd.h index c8dc5f8ea699..dc720a979186 100644 --- a/include/uapi/linux/ublk_cmd.h +++ b/include/uapi/linux/ublk_cmd.h @@ -94,6 +94,8 @@ _IOWR('u', UBLK_IO_COMMIT_AND_FETCH_REQ, struct ublksrv_io_cmd) #define UBLK_U_IO_NEED_GET_DATA \ _IOWR('u', UBLK_IO_NEED_GET_DATA, struct ublksrv_io_cmd) +#define UBLK_U_IO_PROVIDE_IO_BUF \ + _IOWR('u', 0x23, struct ublksrv_io_cmd) /* only ABORT means that no re-fetch */ #define UBLK_IO_RES_OK 0 @@ -127,9 +129,12 @@ #define UBLKSRV_IO_BUF_TOTAL_SIZE (1ULL << UBLKSRV_IO_BUF_TOTAL_BITS) /* - * zero copy requires 4k block size, and can remap ublk driver's io - * request into ublksrv's vm space - */ + * io_uring provide kbuf command based zero copy + * + * Not available for UBLK_F_UNPRIVILEGED_DEV, because we rely on ublk + * server to fill up request buffer for READ IO, and ublk server can't + * be trusted in case of UBLK_F_UNPRIVILEGED_DEV. +*/ #define UBLK_F_SUPPORT_ZERO_COPY (1ULL << 0) /*