From patchwork Fri Nov 26 11:58:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15DDAC433EF for ; Fri, 26 Nov 2021 12:35:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377477AbhKZMiY (ORCPT ); Fri, 26 Nov 2021 07:38:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229847AbhKZMgW (ORCPT ); Fri, 26 Nov 2021 07:36:22 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22E70C08EA6C; Fri, 26 Nov 2021 03:58:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=aQAe4eMPPySy8vIxEl+CCv5r89gANpG908vFhfZszbA=; b=VQNzi5UtQVnrpXA9UEc4INe4KD AAh2zjPtaV4I822UPkJASwWGs6cQPcf1zwQ0oL8jSIPfss9a6x/fvToSNMOtXkfpUCFiaVgKSOwlB oqmg/zHGffHLFR6LXLlEtcWmfqTpeI15SQiJl9NMHkAEVLdi9UP4rH4iqx9qwtxRtFCDUTGP6Kqwx X7Ol1zUjxCXp5ZQVhL0Gtqg7bzCU9bwXP9kBJH8tNsglUsTNICCvK9MA5xQSZnhX+YrHC5NxJdMw2 Sik2wl0GNGfKHxauPrFTwdby9Si5pHquq8zhs6kYcrgm228Xg5uUp5M4fJQ0+IdqLR8lcy12ge1nZ vtq0cSew==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZrw-00ASHH-IG; Fri, 26 Nov 2021 11:58:21 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 01/14] RDMA/qib: rename copy_io to qib_copy_io Date: Fri, 26 Nov 2021 12:58:04 +0100 Message-Id: <20211126115817.2087431-2-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add the proper module prefix to avoid conflicts with a function in the scheduler. Signed-off-by: Christoph Hellwig --- drivers/infiniband/hw/qib/qib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/qib/qib_verbs.c b/drivers/infiniband/hw/qib/qib_verbs.c index ef91bff5c23ca..0080f0be72fef 100644 --- a/drivers/infiniband/hw/qib/qib_verbs.c +++ b/drivers/infiniband/hw/qib/qib_verbs.c @@ -425,7 +425,7 @@ static inline u32 clear_upper_bytes(u32 data, u32 n, u32 off) } #endif -static void copy_io(u32 __iomem *piobuf, struct rvt_sge_state *ss, +static void qib_copy_io(u32 __iomem *piobuf, struct rvt_sge_state *ss, u32 length, unsigned flush_wc) { u32 extra = 0; @@ -975,7 +975,7 @@ static int qib_verbs_send_pio(struct rvt_qp *qp, struct ib_header *ibhdr, qib_pio_copy(piobuf, addr, dwords); goto done; } - copy_io(piobuf, ss, len, flush_wc); + qib_copy_io(piobuf, ss, len, flush_wc); done: if (dd->flags & QIB_USE_SPCL_TRIG) { u32 spcl_off = (pbufn >= dd->piobcnt2k) ? 2047 : 1023; From patchwork Fri Nov 26 11:58:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65A45C433F5 for ; Fri, 26 Nov 2021 12:35:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377487AbhKZMi3 (ORCPT ); Fri, 26 Nov 2021 07:38:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38076 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234137AbhKZMg2 (ORCPT ); Fri, 26 Nov 2021 07:36:28 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94FC0C08EA6F; Fri, 26 Nov 2021 03:58:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=BZ5PLUQpkwxEusMxJULA8/aX7ao2K6qNcU+5BMpwrAI=; b=U8eUC0O6eGRFHijfSLYkVEw0Tj Gfk1aDceaCQ9H9opBXohhF4z7iTnhlEgHN2RepPx9b+CZzGkrwGdLaP2x0FXfJ4hZhOunTQeFLfo2 VYKPuQyx9gKF97dqmcFy2T3XWzDAdpse5flriIGIB3D6j0un4zcO3Ek6+oFeyBZkH2mCGCmJT2VXb iqerEGQsTD9YCrMXWapjmu1gGJU6Q22TKmimH8l0qenQHZs/UPWSb7wL+vG25B5uITnUf4F2rFQ8I F3DZiAsL6U0eE76Q2lMpd7Wc9Yz9hbsLWvLsiHOb1GOrdsVr4bLqzjBLxv/zg3bxUWVMaC8HjcEyD FGJIRUfw==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZrx-00ASHc-Qu; Fri, 26 Nov 2021 11:58:22 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 02/14] fork: move copy_io to block/blk-ioc.c Date: Fri, 26 Nov 2021 12:58:05 +0100 Message-Id: <20211126115817.2087431-3-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Move the copying of the I/O context to the block layer as that is where we can use the proper low-level interfaces. Signed-off-by: Christoph Hellwig --- block/blk-ioc.c | 27 +++++++++++++++++++++++++++ include/linux/iocontext.h | 23 +++++++++++++---------- kernel/fork.c | 26 -------------------------- 3 files changed, 40 insertions(+), 36 deletions(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index 70c99e85aee50..3b31cfad4b75b 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -322,6 +322,33 @@ struct io_context *get_task_io_context(struct task_struct *task, return NULL; } +int __copy_io(unsigned long clone_flags, struct task_struct *tsk) +{ + struct io_context *ioc = current->io_context; + struct io_context *new_ioc; + + /* + * Share io context with parent, if CLONE_IO is set + */ + if (clone_flags & CLONE_IO) { + get_io_context_active(ioc); + + WARN_ON_ONCE(atomic_read(&ioc->nr_tasks) <= 0); + atomic_inc(&ioc->nr_tasks); + + tsk->io_context = ioc; + } else if (ioprio_valid(ioc->ioprio)) { + new_ioc = get_task_io_context(tsk, GFP_KERNEL, NUMA_NO_NODE); + if (unlikely(!new_ioc)) + return -ENOMEM; + + new_ioc->ioprio = ioc->ioprio; + put_io_context(new_ioc); + } + + return 0; +} + /** * ioc_lookup_icq - lookup io_cq from ioc * @ioc: the associated io_context diff --git a/include/linux/iocontext.h b/include/linux/iocontext.h index 0a9dc40b7be84..bcd47d104d8e6 100644 --- a/include/linux/iocontext.h +++ b/include/linux/iocontext.h @@ -129,14 +129,6 @@ static inline void get_io_context_active(struct io_context *ioc) atomic_inc(&ioc->active_ref); } -static inline void ioc_task_link(struct io_context *ioc) -{ - get_io_context_active(ioc); - - WARN_ON_ONCE(atomic_read(&ioc->nr_tasks) <= 0); - atomic_inc(&ioc->nr_tasks); -} - struct task_struct; #ifdef CONFIG_BLOCK void put_io_context(struct io_context *ioc); @@ -144,10 +136,21 @@ void put_io_context_active(struct io_context *ioc); void exit_io_context(struct task_struct *task); struct io_context *get_task_io_context(struct task_struct *task, gfp_t gfp_flags, int node); +int __copy_io(unsigned long clone_flags, struct task_struct *tsk); +static inline int copy_io(unsigned long clone_flags, struct task_struct *tsk) +{ + if (!current->io_context) + return 0; + return __copy_io(clone_flags, tsk); +} #else struct io_context; static inline void put_io_context(struct io_context *ioc) { } static inline void exit_io_context(struct task_struct *task) { } -#endif +static inline int copy_io(unsigned long clone_flags, struct task_struct *tsk) +{ + return 0; +} +#endif /* CONFIG_BLOCK */ -#endif +#endif /* IOCONTEXT_H */ diff --git a/kernel/fork.c b/kernel/fork.c index 3244cc56b697d..3161d7980155e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1556,32 +1556,6 @@ static int copy_files(unsigned long clone_flags, struct task_struct *tsk) return error; } -static int copy_io(unsigned long clone_flags, struct task_struct *tsk) -{ -#ifdef CONFIG_BLOCK - struct io_context *ioc = current->io_context; - struct io_context *new_ioc; - - if (!ioc) - return 0; - /* - * Share io context with parent, if CLONE_IO is set - */ - if (clone_flags & CLONE_IO) { - ioc_task_link(ioc); - tsk->io_context = ioc; - } else if (ioprio_valid(ioc->ioprio)) { - new_ioc = get_task_io_context(tsk, GFP_KERNEL, NUMA_NO_NODE); - if (unlikely(!new_ioc)) - return -ENOMEM; - - new_ioc->ioprio = ioc->ioprio; - put_io_context(new_ioc); - } -#endif - return 0; -} - static int copy_sighand(unsigned long clone_flags, struct task_struct *tsk) { struct sighand_struct *sig; From patchwork Fri Nov 26 11:58:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18E72C4332F for ; Fri, 26 Nov 2021 12:35:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377492AbhKZMi3 (ORCPT ); Fri, 26 Nov 2021 07:38:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234633AbhKZMg3 (ORCPT ); Fri, 26 Nov 2021 07:36:29 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01B82C08EA70; Fri, 26 Nov 2021 03:58:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=uFkJ4coOuJs8MZM0mjjmaO2O9RaMqiZrB6AeS46Mv/0=; b=g2kQe3ju5FIntq5kwlYSknO9C6 W89rqIuqF5WCJF83hXAPqNR1XmqQaEcN4I7vpO01jjamJ3/3POesnhqpM5AxBcnQ7AE+pSwkpUTtN 63kNJc+ubeR5HxUnpzflT9m50e3/5iA3JcCOWIrtFlj0jcT28T0zHEYzgdRdq4pwnT3MLotPwS2To kjmMUD2qSLRM70IiZQO5HqzNgpNP1WBIaq8QALivtrOc0U8nznvF92gx7ZSchHCfBBZfDLdRBrMXq dSFalbuVgZDpAcMgRtwfNRsOGgSBQmPiTiup4p5hJ5e3yVWsSm51on4eVvEoNwR4y9nNNiI9U6pA/ GsY3UIoQ==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZrz-00ASHx-7t; Fri, 26 Nov 2021 11:58:24 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 03/14] bfq: simplify bfq_bic_lookup Date: Fri, 26 Nov 2021 12:58:06 +0100 Message-Id: <20211126115817.2087431-4-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Remove the unused bfqd argument, and hardcode ioc to current->io_context. Signed-off-by: Christoph Hellwig --- block/bfq-iosched.c | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 85554b8009703..c990c6409c119 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -433,26 +433,21 @@ static struct bfq_io_cq *icq_to_bic(struct io_cq *icq) /** * bfq_bic_lookup - search into @ioc a bic associated to @bfqd. - * @bfqd: the lookup key. - * @ioc: the io_context of the process doing I/O. * @q: the request queue. */ -static struct bfq_io_cq *bfq_bic_lookup(struct bfq_data *bfqd, - struct io_context *ioc, - struct request_queue *q) +static struct bfq_io_cq *bfq_bic_lookup(struct request_queue *q) { - if (ioc) { - unsigned long flags; - struct bfq_io_cq *icq; + struct bfq_io_cq *icq; + unsigned long flags; - spin_lock_irqsave(&q->queue_lock, flags); - icq = icq_to_bic(ioc_lookup_icq(ioc, q)); - spin_unlock_irqrestore(&q->queue_lock, flags); + if (!current->io_context) + return NULL; - return icq; - } + spin_lock_irqsave(&q->queue_lock, flags); + icq = icq_to_bic(ioc_lookup_icq(current->io_context, q)); + spin_unlock_irqrestore(&q->queue_lock, flags); - return NULL; + return icq; } /* @@ -2457,7 +2452,7 @@ static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, * returned by bfq_bic_lookup does not go away before * bfqd->lock is taken. */ - struct bfq_io_cq *bic = bfq_bic_lookup(bfqd, current->io_context, q); + struct bfq_io_cq *bic = bfq_bic_lookup(q); bool ret; spin_lock_irq(&bfqd->lock); From patchwork Fri Nov 26 11:58:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 012E0C4332F for ; Fri, 26 Nov 2021 12:35:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377531AbhKZMiq (ORCPT ); Fri, 26 Nov 2021 07:38:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234976AbhKZMgq (ORCPT ); Fri, 26 Nov 2021 07:36:46 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D30AC08EA71; Fri, 26 Nov 2021 03:58:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=sK7cYKiS0ZeDgTJQcBOYJUCVPKi7GjzADEXG7w+vkFA=; b=jpj9W9dva4hbxt1LE7FoGKeilD nKJf93/e408eCDoxSMPX4YuQpBaKtKuhDledbgkJVAeeHTOlzFBnuaW0WA36QlF2NwwBct8Fpu3pW vFbhCCdaxrmxMBLIxcDqZdJ7CwBncvJ2K1JO/xWGMTBDggfAaVltos/1wlEwQVTrJ7DgJgNoSQHhX pCgSe7CBVr3lpxKcVoAUjwKyQvyEnoeKeJe3fAq6fbrvad5BDDCLAbQBGpaU4J+StTxK6NLfDsXUw NNz94o/a8eMqlgR13H6R82ee5DdAW1eeQRg2GumuvohapvQGZlsePyncmqPxrPXhlO1zl4iwxHnmu ENoTpK4w==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZs0-00ASIN-Gx; Fri, 26 Nov 2021 11:58:25 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 04/14] bfq: use bfq_bic_lookup in bfq_limit_depth Date: Fri, 26 Nov 2021 12:58:07 +0100 Message-Id: <20211126115817.2087431-5-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org No need to create a new I/O context if there is none present yet in ->limit_depth. Signed-off-by: Christoph Hellwig --- block/bfq-iosched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index c990c6409c119..ecc2e57e68630 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -663,7 +663,7 @@ static bool bfqq_request_over_limit(struct bfq_queue *bfqq, int limit) static void bfq_limit_depth(unsigned int op, struct blk_mq_alloc_data *data) { struct bfq_data *bfqd = data->q->elevator->elevator_data; - struct bfq_io_cq *bic = icq_to_bic(blk_mq_sched_get_icq(data->q)); + struct bfq_io_cq *bic = bfq_bic_lookup(data->q); struct bfq_queue *bfqq = bic ? bic_to_bfqq(bic, op_is_sync(op)) : NULL; int depth; unsigned limit = data->q->nr_requests; From patchwork Fri Nov 26 11:58:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B05B6C433F5 for ; Fri, 26 Nov 2021 12:35:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377537AbhKZMi4 (ORCPT ); Fri, 26 Nov 2021 07:38:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230471AbhKZMg4 (ORCPT ); Fri, 26 Nov 2021 07:36:56 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9530FC08EA72; Fri, 26 Nov 2021 03:58:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=u0+ePx08fdN0TLP47LRnrql+1HJ4ogpHxxrOWZhKaak=; b=kEh2Xran84vn+1SYDvp7aE9Kli X/fY/S8hIaYl0mM9zwrd4oSE8Bi0YF0c2IKpB142P2a6M+j4PSeznmi5T0FnS4tBtz7ZOIL675pLj BuXBD26B48IRaFT1gRKGyFxMtHOVfJ5nBWQ6ygJxBAVzFcBGu27Ks6jAEhQhEqOyPfHdYSRrCuaO0 wOJXB1YFKbqB9pijbdqhoRXu3yx6q2P1FqRa1SbUj1WML+UNN73WkVa1gtyCZD1i5bjNQRCsMXSbm OWufKqCjiwvY1eVdsuZTPCdOaZFtsumqSlA27S7hlzn3ePyTSRIlamXKcF807oiFBf1NpZMYc6nKu u1v1fuVA==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZs1-00ASIq-RZ; Fri, 26 Nov 2021 11:58:26 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 05/14] Revert "block: Provide blk_mq_sched_get_icq()" Date: Fri, 26 Nov 2021 12:58:08 +0100 Message-Id: <20211126115817.2087431-6-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This reverts commit 4896c4e64ba5d5d5acdbcf68c5910dd4f6d8fa62. The helper is not needed any more. Signed-off-by: Christoph Hellwig --- block/blk-mq-sched.c | 26 +++++++++++--------------- block/blk-mq-sched.h | 1 - 2 files changed, 11 insertions(+), 16 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 98c6a97729f24..b942b38000e53 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -18,8 +18,9 @@ #include "blk-mq-tag.h" #include "blk-wbt.h" -struct io_cq *blk_mq_sched_get_icq(struct request_queue *q) +void blk_mq_sched_assign_ioc(struct request *rq) { + struct request_queue *q = rq->q; struct io_context *ioc; struct io_cq *icq; @@ -27,27 +28,22 @@ struct io_cq *blk_mq_sched_get_icq(struct request_queue *q) if (unlikely(!current->io_context)) create_task_io_context(current, GFP_ATOMIC, q->node); - /* May not have an IO context if context creation failed */ + /* + * May not have an IO context if it's a passthrough request + */ ioc = current->io_context; if (!ioc) - return NULL; + return; spin_lock_irq(&q->queue_lock); icq = ioc_lookup_icq(ioc, q); spin_unlock_irq(&q->queue_lock); - if (icq) - return icq; - return ioc_create_icq(ioc, q, GFP_ATOMIC); -} -EXPORT_SYMBOL(blk_mq_sched_get_icq); -void blk_mq_sched_assign_ioc(struct request *rq) -{ - struct io_cq *icq; - - icq = blk_mq_sched_get_icq(rq->q); - if (!icq) - return; + if (!icq) { + icq = ioc_create_icq(ioc, q, GFP_ATOMIC); + if (!icq) + return; + } get_io_context(icq->ioc); rq->elv.icq = icq; } diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index add651ec06da7..25d1034952b65 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -8,7 +8,6 @@ #define MAX_SCHED_RQ (16 * BLKDEV_DEFAULT_RQ) -struct io_cq *blk_mq_sched_get_icq(struct request_queue *q); void blk_mq_sched_assign_ioc(struct request *rq); bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio, From patchwork Fri Nov 26 11:58:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE9F2C433FE for ; Fri, 26 Nov 2021 12:35:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230471AbhKZMi5 (ORCPT ); Fri, 26 Nov 2021 07:38:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231321AbhKZMg5 (ORCPT ); Fri, 26 Nov 2021 07:36:57 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC152C08EA73; Fri, 26 Nov 2021 03:58:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=jYRVhHW2gVb5d5i1xf3+2qqpIhLx3KXODjLr4PtmDyE=; b=eCapykDDNWjfyVTdMYXtBOYKLv prgYEG325JO21OUPvNXInsoHWxwlwDFRS20CKdcVoNwJY7Wn70QOJ+1r6y9MEap/CPiTCutMV9TZA CG0XXhQwyub/VWp3SkEqZ/OAYA06KkJ2Nz4QfkUh0PBSIZmLK5sM5bpMx2L11/zk7eeIwhZfu4ZSJ jtv4jDGnPnxH/jGKpmDt1YTRibVqr3rBY/dl9A1sS36Iu6nXz3XO0m+uca/M5AtTqea7lbu4xhRp3 wL/MW/0Qz7xH+7C/dUwxCKSOBkDNesfbsw9deu3xBC2vwldONP9n0QYz4dQcEqz8UAsohoGikjO/q I4uRiaGA==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZs3-00ASJM-2j; Fri, 26 Nov 2021 11:58:28 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 06/14] block: mark put_io_context_active static Date: Fri, 26 Nov 2021 12:58:09 +0100 Message-Id: <20211126115817.2087431-7-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Signed-off-by: Christoph Hellwig --- block/blk-ioc.c | 2 +- include/linux/iocontext.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index 3b31cfad4b75b..f3ff495756cb4 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -175,7 +175,7 @@ void put_io_context(struct io_context *ioc) * Undo get_io_context_active(). If active reference reaches zero after * put, @ioc can never issue further IOs and ioscheds are notified. */ -void put_io_context_active(struct io_context *ioc) +static void put_io_context_active(struct io_context *ioc) { struct io_cq *icq; diff --git a/include/linux/iocontext.h b/include/linux/iocontext.h index bcd47d104d8e6..3ba45953d5228 100644 --- a/include/linux/iocontext.h +++ b/include/linux/iocontext.h @@ -132,7 +132,6 @@ static inline void get_io_context_active(struct io_context *ioc) struct task_struct; #ifdef CONFIG_BLOCK void put_io_context(struct io_context *ioc); -void put_io_context_active(struct io_context *ioc); void exit_io_context(struct task_struct *task); struct io_context *get_task_io_context(struct task_struct *task, gfp_t gfp_flags, int node); From patchwork Fri Nov 26 11:58:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 964ECC433EF for ; Fri, 26 Nov 2021 12:36:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377574AbhKZMj0 (ORCPT ); Fri, 26 Nov 2021 07:39:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236177AbhKZMhZ (ORCPT ); Fri, 26 Nov 2021 07:37:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 036C5C08EA74; Fri, 26 Nov 2021 03:58:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=90gN+8A6JEywMJ3UpPZQ7B+PFcu57iDbBHkoNmyeA0w=; b=JxeJ+8tfnpMtAc9TsF9sB4h51l sOoK1GY3Ps30tXWbYE5KZSdLGsL1EEZDry+uFtR9T+saBxkHJPB/WaxnMjKxmHvdyVJbsDfiFKW3t yoevZ8Svzu7poSJvRNNclM9pWBazBmid5MnocN79XSfq1YYvy9on/mMqhRRt2Z1Gs8vAaI7yfeX2Q pM4eaXNS5t+/v9yWUVgaBKj1rp4avJoPPBI1lDl4o3R+PJl1DM6GzRZTILJdHJsqvshR+S0rzwWOe BKBYA9CUXRzNRSIv6cLabT+4ZKsBl/9b706gDUX/44xynzZpYzyRHburd7ovhkLHqi0LMp0wTuUnc naby8mmA==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZs4-00ASJm-An; Fri, 26 Nov 2021 11:58:29 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 07/14] block: move blk_mq_sched_assign_ioc to blk-ioc.c Date: Fri, 26 Nov 2021 12:58:10 +0100 Message-Id: <20211126115817.2087431-8-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Move blk_mq_sched_assign_ioc so that many interfaces from the file can be marked static. Rename the function to ioc_find_get_icq as well and return the icq to simplify the interface. Signed-off-by: Christoph Hellwig --- block/bfq-iosched.c | 2 +- block/blk-ioc.c | 39 +++++++++++++++++++++++++++++++++++---- block/blk-mq-sched.c | 31 ------------------------------- block/blk-mq-sched.h | 2 -- block/blk.h | 6 +----- 5 files changed, 37 insertions(+), 43 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index ecc2e57e68630..2d484d3f7f22a 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -6666,7 +6666,7 @@ static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, */ static void bfq_prepare_request(struct request *rq) { - blk_mq_sched_assign_ioc(rq); + rq->elv.icq = ioc_find_get_icq(rq->q); /* * Regardless of whether we have an icq attached, we have to diff --git a/block/blk-ioc.c b/block/blk-ioc.c index f3ff495756cb4..f4f84a2072be7 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -24,7 +24,7 @@ static struct kmem_cache *iocontext_cachep; * * Increment reference count to @ioc. */ -void get_io_context(struct io_context *ioc) +static void get_io_context(struct io_context *ioc) { BUG_ON(atomic_long_read(&ioc->refcount) <= 0); atomic_long_inc(&ioc->refcount); @@ -248,7 +248,8 @@ void ioc_clear_queue(struct request_queue *q) __ioc_clear_queue(&icq_list); } -int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, int node) +static int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, + int node) { struct io_context *ioc; int ret; @@ -397,8 +398,8 @@ EXPORT_SYMBOL(ioc_lookup_icq); * The caller is responsible for ensuring @ioc won't go away and @q is * alive and will stay alive until this function returns. */ -struct io_cq *ioc_create_icq(struct io_context *ioc, struct request_queue *q, - gfp_t gfp_mask) +static struct io_cq *ioc_create_icq(struct io_context *ioc, + struct request_queue *q, gfp_t gfp_mask) { struct elevator_type *et = q->elevator->type; struct io_cq *icq; @@ -441,6 +442,36 @@ struct io_cq *ioc_create_icq(struct io_context *ioc, struct request_queue *q, return icq; } +struct io_cq *ioc_find_get_icq(struct request_queue *q) +{ + struct io_context *ioc; + struct io_cq *icq; + + /* create task io_context, if we don't have one already */ + if (unlikely(!current->io_context)) + create_task_io_context(current, GFP_ATOMIC, q->node); + + /* + * May not have an IO context if it's a passthrough request + */ + ioc = current->io_context; + if (!ioc) + return NULL; + + spin_lock_irq(&q->queue_lock); + icq = ioc_lookup_icq(ioc, q); + spin_unlock_irq(&q->queue_lock); + + if (!icq) { + icq = ioc_create_icq(ioc, q, GFP_ATOMIC); + if (!icq) + return NULL; + } + get_io_context(icq->ioc); + return icq; +} +EXPORT_SYMBOL_GPL(ioc_find_get_icq); + static int __init blk_ioc_init(void) { iocontext_cachep = kmem_cache_create("blkdev_ioc", diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index b942b38000e53..0d7257848f7ef 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -18,37 +18,6 @@ #include "blk-mq-tag.h" #include "blk-wbt.h" -void blk_mq_sched_assign_ioc(struct request *rq) -{ - struct request_queue *q = rq->q; - struct io_context *ioc; - struct io_cq *icq; - - /* create task io_context, if we don't have one already */ - if (unlikely(!current->io_context)) - create_task_io_context(current, GFP_ATOMIC, q->node); - - /* - * May not have an IO context if it's a passthrough request - */ - ioc = current->io_context; - if (!ioc) - return; - - spin_lock_irq(&q->queue_lock); - icq = ioc_lookup_icq(ioc, q); - spin_unlock_irq(&q->queue_lock); - - if (!icq) { - icq = ioc_create_icq(ioc, q, GFP_ATOMIC); - if (!icq) - return; - } - get_io_context(icq->ioc); - rq->elv.icq = icq; -} -EXPORT_SYMBOL_GPL(blk_mq_sched_assign_ioc); - /* * Mark a hardware queue as needing a restart. For shared queues, maintain * a count of how many hardware queues are marked for restart. diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index 25d1034952b65..0250139724539 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -8,8 +8,6 @@ #define MAX_SCHED_RQ (16 * BLKDEV_DEFAULT_RQ) -void blk_mq_sched_assign_ioc(struct request *rq); - bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs, struct request **merged_request); bool blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio, diff --git a/block/blk.h b/block/blk.h index a57c84654d0a1..187cb2654ffde 100644 --- a/block/blk.h +++ b/block/blk.h @@ -363,14 +363,10 @@ static inline unsigned int bio_aligned_discard_max_sectors( /* * Internal io_context interface */ -void get_io_context(struct io_context *ioc); +struct io_cq *ioc_find_get_icq(struct request_queue *q); struct io_cq *ioc_lookup_icq(struct io_context *ioc, struct request_queue *q); -struct io_cq *ioc_create_icq(struct io_context *ioc, struct request_queue *q, - gfp_t gfp_mask); void ioc_clear_queue(struct request_queue *q); -int create_task_io_context(struct task_struct *task, gfp_t gfp_mask, int node); - #ifdef CONFIG_BLK_DEV_THROTTLING_LOW extern ssize_t blk_throtl_sample_time_show(struct request_queue *q, char *page); extern ssize_t blk_throtl_sample_time_store(struct request_queue *q, From patchwork Fri Nov 26 11:58:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29585C4332F for ; Fri, 26 Nov 2021 12:36:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236177AbhKZMj0 (ORCPT ); Fri, 26 Nov 2021 07:39:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235419AbhKZMhZ (ORCPT ); Fri, 26 Nov 2021 07:37:25 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49C90C08EA75; Fri, 26 Nov 2021 03:58:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=PDMej3uV0A69WfPJSYZoi9ft4XTEBwKSjC8fReC4QEY=; b=HYRwctFE5c7GTgKbjdufN09ivL jki4BjRwgcrxQGeAIQ/XL6jSkI4XAkVJm2fitp4ZcTAL4QNlfUbT4ZJfmQoDq6lwUFYojHBxpcEt+ HOPQ4KfIW4aM8+JypqsTh9SiOGwLYJC2cgjcK34qq9QtFoEscXslIbVEGd6i7ghjStUe2zHo0KDjn oO+DxSrizS8XIZFJcAs5bN5ynAWXzXJ2DPdO/vxpiecZg7JAATxwyfJofnW7mQXynnyi36o6F5MWg I8tORaOKdsLUHvT1xFtPt1YkPrfj9axAGd8i48Gk/jDrSuWAOiSRv9pfHFUisxTAZu1wJJpUDEwed T+JeSlWg==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZs5-00ASKG-KU; Fri, 26 Nov 2021 11:58:30 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 08/14] block: move the remaining elv.icq handling to the I/O scheduler Date: Fri, 26 Nov 2021 12:58:11 +0100 Message-Id: <20211126115817.2087431-9-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org After the prepare side has been moved to the only I/O scheduler that cares, do the same for the cleanup and the NULL initialization. Signed-off-by: Christoph Hellwig --- block/bfq-iosched.c | 12 +++++++++++- block/blk-ioc.c | 1 + block/blk-mq.c | 14 +++----------- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 2d484d3f7f22a..8295b0f96cbfe 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -6569,6 +6569,16 @@ static void bfq_finish_requeue_request(struct request *rq) rq->elv.priv[1] = NULL; } +static void bfq_finish_request(struct request *rq) +{ + bfq_finish_requeue_request(rq); + + if (rq->elv.icq) { + put_io_context(rq->elv.icq->ioc); + rq->elv.icq = NULL; + } +} + /* * Removes the association between the current task and bfqq, assuming * that bic points to the bfq iocontext of the task. @@ -7388,7 +7398,7 @@ static struct elevator_type iosched_bfq_mq = { .limit_depth = bfq_limit_depth, .prepare_request = bfq_prepare_request, .requeue_request = bfq_finish_requeue_request, - .finish_request = bfq_finish_requeue_request, + .finish_request = bfq_finish_request, .exit_icq = bfq_exit_icq, .insert_requests = bfq_insert_requests, .dispatch_request = bfq_dispatch_request, diff --git a/block/blk-ioc.c b/block/blk-ioc.c index f4f84a2072be7..3ba15c867dfa6 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -167,6 +167,7 @@ void put_io_context(struct io_context *ioc) if (free_ioc) kmem_cache_free(iocontext_cachep, ioc); } +EXPORT_SYMBOL_GPL(put_io_context); /** * put_io_context_active - put active reference on ioc diff --git a/block/blk-mq.c b/block/blk-mq.c index 82491ab676fb1..7bdef269a5d94 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -400,7 +400,6 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data, if (rq->rq_flags & RQF_ELV) { struct elevator_queue *e = data->q->elevator; - rq->elv.icq = NULL; INIT_HLIST_NODE(&rq->hash); RB_CLEAR_NODE(&rq->rb_node); @@ -631,16 +630,9 @@ void blk_mq_free_request(struct request *rq) struct request_queue *q = rq->q; struct blk_mq_hw_ctx *hctx = rq->mq_hctx; - if (rq->rq_flags & RQF_ELVPRIV) { - struct elevator_queue *e = q->elevator; - - if (e->type->ops.finish_request) - e->type->ops.finish_request(rq); - if (rq->elv.icq) { - put_io_context(rq->elv.icq->ioc); - rq->elv.icq = NULL; - } - } + if ((rq->rq_flags & RQF_ELVPRIV) && + q->elevator->type->ops.finish_request) + q->elevator->type->ops.finish_request(rq); if (rq->rq_flags & RQF_MQ_INFLIGHT) __blk_mq_dec_active_requests(hctx); From patchwork Fri Nov 26 11:58:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3023C43217 for ; Fri, 26 Nov 2021 12:36:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349220AbhKZMjd (ORCPT ); Fri, 26 Nov 2021 07:39:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345276AbhKZMh0 (ORCPT ); Fri, 26 Nov 2021 07:37:26 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FC2AC08EA76; Fri, 26 Nov 2021 03:58:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=75Yd+UjqZ8mEVC5ZdOQmcq+vXP3FIOnKl9cEqCAA/c0=; b=F+bAXzkZvw6q/0b2SNKRR/ph95 8MY6tXc+k6Mmcg3v7qm16ASN6/jEEybgpLtaf4tHVvB2y+5GQ9Hs4aGI362328nsjpSqgkuBgi5bz lAj6vl0sdqmgk5IB//NNUhgt9jpFxF7SjnXsLOIphjJDF8Ez5LH6WEHlRbi9OviZa6kuxUhBjK2Z3 L+9VBCnjABpBghEoHCD4l83vqxKaQhbswoX3KjOx2X0ptajyZzgC/qrGZEy+bo6IS+tIFIg0hm2KE OANxj+6D3DvH66QivU3I2IzTu4DjTTgZmrBaDDlWp/nlUDAaLDT5DcZtTGGwFzL2RvCZcEeGDJ1db g690wvgw==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZs6-00ASKg-Tn; Fri, 26 Nov 2021 11:58:31 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 09/14] block: remove get_io_context_active Date: Fri, 26 Nov 2021 12:58:12 +0100 Message-Id: <20211126115817.2087431-10-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Fold it into it's only caller, and remove a lof of the debug checks that are not needed. Signed-off-by: Christoph Hellwig --- block/blk-ioc.c | 8 +++----- include/linux/iocontext.h | 16 ---------------- 2 files changed, 3 insertions(+), 21 deletions(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index 3ba15c867dfa6..cc4eb2ba87f74 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -173,7 +173,7 @@ EXPORT_SYMBOL_GPL(put_io_context); * put_io_context_active - put active reference on ioc * @ioc: ioc of interest * - * Undo get_io_context_active(). If active reference reaches zero after + * Put an active reference to an ioc. If active reference reaches zero after * put, @ioc can never issue further IOs and ioscheds are notified. */ static void put_io_context_active(struct io_context *ioc) @@ -333,11 +333,9 @@ int __copy_io(unsigned long clone_flags, struct task_struct *tsk) * Share io context with parent, if CLONE_IO is set */ if (clone_flags & CLONE_IO) { - get_io_context_active(ioc); - - WARN_ON_ONCE(atomic_read(&ioc->nr_tasks) <= 0); + atomic_long_inc(&ioc->refcount); + atomic_inc(&ioc->active_ref); atomic_inc(&ioc->nr_tasks); - tsk->io_context = ioc; } else if (ioprio_valid(ioc->ioprio)) { new_ioc = get_task_io_context(tsk, GFP_KERNEL, NUMA_NO_NODE); diff --git a/include/linux/iocontext.h b/include/linux/iocontext.h index 3ba45953d5228..c1229fbd6691c 100644 --- a/include/linux/iocontext.h +++ b/include/linux/iocontext.h @@ -113,22 +113,6 @@ struct io_context { struct work_struct release_work; }; -/** - * get_io_context_active - get active reference on ioc - * @ioc: ioc of interest - * - * Only iocs with active reference can issue new IOs. This function - * acquires an active reference on @ioc. The caller must already have an - * active reference on @ioc. - */ -static inline void get_io_context_active(struct io_context *ioc) -{ - WARN_ON_ONCE(atomic_long_read(&ioc->refcount) <= 0); - WARN_ON_ONCE(atomic_read(&ioc->active_ref) <= 0); - atomic_long_inc(&ioc->refcount); - atomic_inc(&ioc->active_ref); -} - struct task_struct; #ifdef CONFIG_BLOCK void put_io_context(struct io_context *ioc); From patchwork Fri Nov 26 11:58:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0217BC433F5 for ; Fri, 26 Nov 2021 12:36:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377579AbhKZMj2 (ORCPT ); Fri, 26 Nov 2021 07:39:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377198AbhKZMh0 (ORCPT ); Fri, 26 Nov 2021 07:37:26 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05030C08EA77; Fri, 26 Nov 2021 03:58:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+BFx8j7M87EAoFMxSgcYqvBCcWuO6nRbLCHiZUMRy8o=; b=sRM/131K2LBUYEpJ9EwYqxibDM yD+zYnd5iNTEs6xAhhaHMvnUOXk2Vb+HAR45SbSmBMVtI58FntJkLzIFn5HYjwXM6SnBI3wstPFGT C2mPQkaaqsHFxYLTRrKBGw+BmIbFL8t2JVwposFx4iRwH1zTr4ID7ZsD4bYgBeXGPTRYbTTyNa9+a wnzk/sDnWw+jNsxk++pP7/FSZ1HDzM0PdbtJDsjK38TBFDDCpoVEj6ZIKLJESfUDqpl603hOi41iS 0s3xZU2ziNShT9UVzViyaw/ZeARS+F5PRaIn6YvWIL8g2RtYqgeROOgkPCxvodvCdSyTqix7ha3e7 X6auSHeg==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZs8-00ASL6-9V; Fri, 26 Nov 2021 11:58:33 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 10/14] block: factor out a alloc_io_context helper Date: Fri, 26 Nov 2021 12:58:13 +0100 Message-Id: <20211126115817.2087431-11-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Factor out a helper that just allocate an I/O context. Signed-off-by: Christoph Hellwig --- block/blk-ioc.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index cc4eb2ba87f74..b42fbb82d5c0c 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -249,18 +249,15 @@ void ioc_clear_queue(struct request_queue *q) __ioc_clear_queue(&icq_list); } -static int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, - int node) +static struct io_context *alloc_io_context(gfp_t gfp_flags, int node) { struct io_context *ioc; - int ret; ioc = kmem_cache_alloc_node(iocontext_cachep, gfp_flags | __GFP_ZERO, node); if (unlikely(!ioc)) - return -ENOMEM; + return NULL; - /* initialize */ atomic_long_set(&ioc->refcount, 1); atomic_set(&ioc->nr_tasks, 1); atomic_set(&ioc->active_ref, 1); @@ -268,6 +265,18 @@ static int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, INIT_RADIX_TREE(&ioc->icq_tree, GFP_ATOMIC); INIT_HLIST_HEAD(&ioc->icq_list); INIT_WORK(&ioc->release_work, ioc_release_fn); + return ioc; +} + +static int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, + int node) +{ + struct io_context *ioc; + int ret; + + ioc = alloc_io_context(gfp_flags, node); + if (!ioc) + return -ENOMEM; /* * Try to install. ioc shouldn't be installed if someone else From patchwork Fri Nov 26 11:58:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A2E7C433FE for ; Fri, 26 Nov 2021 12:36:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377198AbhKZMjb (ORCPT ); Fri, 26 Nov 2021 07:39:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353151AbhKZMh1 (ORCPT ); Fri, 26 Nov 2021 07:37:27 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4ABD9C08EA78; Fri, 26 Nov 2021 03:58:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=UqCtwnOlJ4I/jJrVI1zrTSkH7vvwV/GW4x2e0d3vkK0=; b=KQ/j8Kf9ed3SFkU4qrdKptnIqr TOG1RoJ3aEDHFZV1kWn86VxfI3/oIdVE6b4omkNdGF/h2oFnEm6ohlZi9YLq7/aLlTsBd2La6Y9Ax NDitrODOHgx1/f9gLqsAQ0y3CPqXikWl/zkURFqwPtzgYUIgYd9dQVO6ATDtDJwyXuU8qCMkj2+y6 Dfv9NsdgZdfcqGNOP7bSjG0XlhrRt07eYZlReWPH1xcsD8+tricYwbLUqPUt988HOsyaqWNcNGYtL xLnntbZjA0am5CTwvnFgj/CDGiEofv8hoQJvR2y3qexM1oqZQlt245mmW0j7JVtIy3bgR7/1HJWt5 X7dDtRgA==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZs9-00ASLZ-Jr; Fri, 26 Nov 2021 11:58:34 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 11/14] block: use alloc_io_context in __copy_io Date: Fri, 26 Nov 2021 12:58:14 +0100 Message-Id: <20211126115817.2087431-12-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In __copy_io we know that the newly allocate task_struct does not have an I/O context yet and is not exiting. So just allocate the I/O context struct and install it directly. There is no need to lock the task either as it is just being created. Signed-off-by: Christoph Hellwig --- block/blk-ioc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index b42fbb82d5c0c..f06d1040442c3 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -336,7 +336,6 @@ struct io_context *get_task_io_context(struct task_struct *task, int __copy_io(unsigned long clone_flags, struct task_struct *tsk) { struct io_context *ioc = current->io_context; - struct io_context *new_ioc; /* * Share io context with parent, if CLONE_IO is set @@ -347,12 +346,10 @@ int __copy_io(unsigned long clone_flags, struct task_struct *tsk) atomic_inc(&ioc->nr_tasks); tsk->io_context = ioc; } else if (ioprio_valid(ioc->ioprio)) { - new_ioc = get_task_io_context(tsk, GFP_KERNEL, NUMA_NO_NODE); - if (unlikely(!new_ioc)) + tsk->io_context = alloc_io_context(GFP_KERNEL, NUMA_NO_NODE); + if (!tsk->io_context) return -ENOMEM; - - new_ioc->ioprio = ioc->ioprio; - put_io_context(new_ioc); + tsk->io_context->ioprio = ioc->ioprio; } return 0; From patchwork Fri Nov 26 11:58:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83128C43219 for ; Fri, 26 Nov 2021 12:36:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235151AbhKZMje (ORCPT ); Fri, 26 Nov 2021 07:39:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229924AbhKZMh1 (ORCPT ); Fri, 26 Nov 2021 07:37:27 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FADBC08EAE7; Fri, 26 Nov 2021 03:58:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ddHoe7mVTenzSy5UaLglWpZv5uH0jZUNX5NQjcGtb8Y=; b=YK60G6yJ9bp6KelWJx4vEBmXML xSvTlBYZciRT5CXutXZZIttj/P5rhjAVGry6nbNLHczOPoAEH3FdLn6foC9wShg1JU7diYZmZ2GwK O0ogbh3NoabTqAoMa2+Xr9PTGLS0k/8odouAhpZToprEFVT9gJweJSUX44SF9/ouKM7XGzjmRoPzW Cs6h7QO9YrRJba2I17B7N2cfybQHNrk8xgnHjnV4euq6oqevME0X32iiatEmujE0o9/t3Il7TXhJU sRFxJgxkTdVYb4r4+8n/EIdhH4Oi1VNdh4eYIFBDnPvNLLSDO35Gf7f4vGvPMPjBLzO5Mh25AnBmM 84ZQARqw==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZsA-00ASM0-RE; Fri, 26 Nov 2021 11:58:35 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 12/14] block: return the io_context from create_task_io_context Date: Fri, 26 Nov 2021 12:58:15 +0100 Message-Id: <20211126115817.2087431-13-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Grab a reference to the newly allocated or existing io_context in create_task_io_context and return it. This simplifies the callers and removes the need for double lookups. Signed-off-by: Christoph Hellwig --- block/blk-ioc.c | 66 ++++++++++++++++++++++--------------------------- 1 file changed, 30 insertions(+), 36 deletions(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index f06d1040442c3..5bfe810496fca 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -268,15 +268,14 @@ static struct io_context *alloc_io_context(gfp_t gfp_flags, int node) return ioc; } -static int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, - int node) +static struct io_context *create_task_io_context(struct task_struct *task, + gfp_t gfp_flags, int node) { struct io_context *ioc; - int ret; ioc = alloc_io_context(gfp_flags, node); if (!ioc) - return -ENOMEM; + return NULL; /* * Try to install. ioc shouldn't be installed if someone else @@ -292,11 +291,11 @@ static int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, else kmem_cache_free(iocontext_cachep, ioc); - ret = task->io_context ? 0 : -EBUSY; - + ioc = task->io_context; + if (ioc) + get_io_context(ioc); task_unlock(task); - - return ret; + return ioc; } /** @@ -319,18 +318,15 @@ struct io_context *get_task_io_context(struct task_struct *task, might_sleep_if(gfpflags_allow_blocking(gfp_flags)); - do { - task_lock(task); - ioc = task->io_context; - if (likely(ioc)) { - get_io_context(ioc); - task_unlock(task); - return ioc; - } + task_lock(task); + ioc = task->io_context; + if (unlikely(!ioc)) { task_unlock(task); - } while (!create_task_io_context(task, gfp_flags, node)); - - return NULL; + return create_task_io_context(task, gfp_flags, node); + } + get_io_context(ioc); + task_unlock(task); + return ioc; } int __copy_io(unsigned long clone_flags, struct task_struct *tsk) @@ -449,30 +445,28 @@ static struct io_cq *ioc_create_icq(struct io_context *ioc, struct io_cq *ioc_find_get_icq(struct request_queue *q) { - struct io_context *ioc; - struct io_cq *icq; - - /* create task io_context, if we don't have one already */ - if (unlikely(!current->io_context)) - create_task_io_context(current, GFP_ATOMIC, q->node); + struct io_context *ioc = current->io_context; + struct io_cq *icq = NULL; - /* - * May not have an IO context if it's a passthrough request - */ - ioc = current->io_context; - if (!ioc) - return NULL; + if (unlikely(!ioc)) { + ioc = create_task_io_context(current, GFP_ATOMIC, q->node); + if (!ioc) + return NULL; + } else { + get_io_context(ioc); - spin_lock_irq(&q->queue_lock); - icq = ioc_lookup_icq(ioc, q); - spin_unlock_irq(&q->queue_lock); + spin_lock_irq(&q->queue_lock); + icq = ioc_lookup_icq(ioc, q); + spin_unlock_irq(&q->queue_lock); + } if (!icq) { icq = ioc_create_icq(ioc, q, GFP_ATOMIC); - if (!icq) + if (!icq) { + put_io_context(ioc); return NULL; + } } - get_io_context(icq->ioc); return icq; } EXPORT_SYMBOL_GPL(ioc_find_get_icq); From patchwork Fri Nov 26 11:58:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCEEAC4167B for ; Fri, 26 Nov 2021 12:36:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377259AbhKZMjf (ORCPT ); Fri, 26 Nov 2021 07:39:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232097AbhKZMh3 (ORCPT ); Fri, 26 Nov 2021 07:37:29 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD1BAC08EAE8; Fri, 26 Nov 2021 03:58:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=H7FEXl9MZgmEtU6V92VxqIMPFtemYzuFK5Tr4nLAwe0=; b=M8P09Stm7lyegv9a5lskPOwcUt /efI18zri/4O2JqAV5riXD4mDWJqBgKF9c6u4n8fqEkUG21ZwxA2GvI8AJeShtLlYZyCHIy+dA5PI RIRnprLmY+64vwfgEq6VDJU9Oqjga/FLxzTGBFv3UH6QpYb4MI6WKdBY7BCYNPykfigdm/V3iAtEj IsVjtssWq71vAhNAvFC+X6B49NJ72/ugV8dIox973SZgxS2VGALOVeKtaHT0MEtudisraJuFWGOkI FMU7VWLNanRqAzC4Znb4tH5b4IpZfOib++EpQt0g2x65vLt4Kmp6jnn9+C+d0xpls2MuIKi1APhAV 38gerq/Q==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZsC-00ASMQ-4e; Fri, 26 Nov 2021 11:58:37 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 13/14] block: simplify ioc_create_icq Date: Fri, 26 Nov 2021 12:58:16 +0100 Message-Id: <20211126115817.2087431-14-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Remove the ioc and gfp_mask argument, which are hard coded by the caller. Signed-off-by: Christoph Hellwig --- block/blk-ioc.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index 5bfe810496fca..c56648f7cad47 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -389,9 +389,7 @@ EXPORT_SYMBOL(ioc_lookup_icq); /** * ioc_create_icq - create and link io_cq - * @ioc: io_context of interest * @q: request_queue of interest - * @gfp_mask: allocation mask * * Make sure io_cq linking @ioc and @q exists. If icq doesn't exist, they * will be created using @gfp_mask. @@ -399,19 +397,19 @@ EXPORT_SYMBOL(ioc_lookup_icq); * The caller is responsible for ensuring @ioc won't go away and @q is * alive and will stay alive until this function returns. */ -static struct io_cq *ioc_create_icq(struct io_context *ioc, - struct request_queue *q, gfp_t gfp_mask) +static struct io_cq *ioc_create_icq(struct request_queue *q) { + struct io_context *ioc = current->io_context; struct elevator_type *et = q->elevator->type; struct io_cq *icq; /* allocate stuff */ - icq = kmem_cache_alloc_node(et->icq_cache, gfp_mask | __GFP_ZERO, + icq = kmem_cache_alloc_node(et->icq_cache, GFP_ATOMIC | __GFP_ZERO, q->node); if (!icq) return NULL; - if (radix_tree_maybe_preload(gfp_mask) < 0) { + if (radix_tree_maybe_preload(GFP_ATOMIC) < 0) { kmem_cache_free(et->icq_cache, icq); return NULL; } @@ -461,7 +459,7 @@ struct io_cq *ioc_find_get_icq(struct request_queue *q) } if (!icq) { - icq = ioc_create_icq(ioc, q, GFP_ATOMIC); + icq = ioc_create_icq(q); if (!icq) { put_io_context(ioc); return NULL; From patchwork Fri Nov 26 11:58:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12640675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB02FC4167E for ; Fri, 26 Nov 2021 12:36:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345276AbhKZMjf (ORCPT ); Fri, 26 Nov 2021 07:39:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234489AbhKZMha (ORCPT ); Fri, 26 Nov 2021 07:37:30 -0500 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3264BC08EAE9; Fri, 26 Nov 2021 03:58:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+tBLAaFkzip4baS+mnayza4tDle8mH9Q1mxOznfall8=; b=jofJ4mr/aWX50ISkivAC4mMZjg 9yDX3lKG8CQDkQbkULZYzvH4ikPNadZxYaL56H+yRZdns+aCqgoMArWgA1ys8AvWo4QjnQSuw3Tg8 lqWYo3X+n9CJRKjw+Sltyh9IC+roO6eTS6miYpqcjx4ySuMGwyPzle36FxImWmBmdATjwlM7xqTdk hVxr33F6AddUlij+gBOvE1hMx6oWLho9jvGqYS/iR5VxPc4JxPScPlQWMMF/frlbfkUkNJfiObEPl jd/L/s3tYsFEf108L8K03f5zYouCiviQwERYfnbDxV35A78GOrGplCkWxdQHUFJayR9yUWvVb9HmS lO24UhmQ==; Received: from [2001:4bb8:191:f9ce:bae8:5658:102a:5491] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1mqZsD-00ASMq-HZ; Fri, 26 Nov 2021 11:58:38 +0000 From: Christoph Hellwig To: Jens Axboe Cc: Paolo Valente , Jan Kara , Dennis Dalessandro , Mike Marciniszyn , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH 14/14] block: simplify ioc_lookup_icq Date: Fri, 26 Nov 2021 12:58:17 +0100 Message-Id: <20211126115817.2087431-15-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211126115817.2087431-1-hch@lst.de> References: <20211126115817.2087431-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Remove the ioc argument as it always points to current->io_context. Signed-off-by: Christoph Hellwig --- block/bfq-iosched.c | 2 +- block/blk-ioc.c | 8 ++++---- block/blk.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 8295b0f96cbfe..0c612a9116967 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -444,7 +444,7 @@ static struct bfq_io_cq *bfq_bic_lookup(struct request_queue *q) return NULL; spin_lock_irqsave(&q->queue_lock, flags); - icq = icq_to_bic(ioc_lookup_icq(current->io_context, q)); + icq = icq_to_bic(ioc_lookup_icq(q)); spin_unlock_irqrestore(&q->queue_lock, flags); return icq; diff --git a/block/blk-ioc.c b/block/blk-ioc.c index c56648f7cad47..536fb496ad763 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -353,14 +353,14 @@ int __copy_io(unsigned long clone_flags, struct task_struct *tsk) /** * ioc_lookup_icq - lookup io_cq from ioc - * @ioc: the associated io_context * @q: the associated request_queue * * Look up io_cq associated with @ioc - @q pair from @ioc. Must be called * with @q->queue_lock held. */ -struct io_cq *ioc_lookup_icq(struct io_context *ioc, struct request_queue *q) +struct io_cq *ioc_lookup_icq(struct request_queue *q) { + struct io_context *ioc = current->io_context; struct io_cq *icq; lockdep_assert_held(&q->queue_lock); @@ -430,7 +430,7 @@ static struct io_cq *ioc_create_icq(struct request_queue *q) et->ops.init_icq(icq); } else { kmem_cache_free(et->icq_cache, icq); - icq = ioc_lookup_icq(ioc, q); + icq = ioc_lookup_icq(q); if (!icq) printk(KERN_ERR "cfq: icq link failed!\n"); } @@ -454,7 +454,7 @@ struct io_cq *ioc_find_get_icq(struct request_queue *q) get_io_context(ioc); spin_lock_irq(&q->queue_lock); - icq = ioc_lookup_icq(ioc, q); + icq = ioc_lookup_icq(q); spin_unlock_irq(&q->queue_lock); } diff --git a/block/blk.h b/block/blk.h index 187cb2654ffde..3be0fdf76c9a5 100644 --- a/block/blk.h +++ b/block/blk.h @@ -364,7 +364,7 @@ static inline unsigned int bio_aligned_discard_max_sectors( * Internal io_context interface */ struct io_cq *ioc_find_get_icq(struct request_queue *q); -struct io_cq *ioc_lookup_icq(struct io_context *ioc, struct request_queue *q); +struct io_cq *ioc_lookup_icq(struct request_queue *q); void ioc_clear_queue(struct request_queue *q); #ifdef CONFIG_BLK_DEV_THROTTLING_LOW