From patchwork Thu Jan 5 21:24:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 13090495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED9DBC3DA7A for ; Thu, 5 Jan 2023 21:24:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235729AbjAEVYn (ORCPT ); Thu, 5 Jan 2023 16:24:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235725AbjAEVYm (ORCPT ); Thu, 5 Jan 2023 16:24:42 -0500 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBD1E6319A; Thu, 5 Jan 2023 13:24:41 -0800 (PST) Received: by mail-pj1-x1032.google.com with SMTP id v13-20020a17090a6b0d00b00219c3be9830so3381322pjj.4; Thu, 05 Jan 2023 13:24:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=TrXn/UsH93lS2A5CebmmFeBERFF7L0+KOIci0sQ1Uks=; b=TH3z2zdM6CYRaMvFVyXRvtLK7NTJD3DGsChzYPOzW0OE4ZDlFOsOzaP6/eB4Ul1i0S RvBDUXmn0foulgSyG8vos47nKdRPIfR1ZnRG9+QooLDJD4bl8UGdwD2Ekgoiq64Pj6Ct 4g73y9tPKA7rpecPy84mOTVWDD66qkegYW7JiecBtg/BMFhtOseOae82jL/t556MGJ2f cqQuDHRAVVGGiSwrS54wu2eiQnpNHtlF5dvuGzJnlPp6fh07+vN8MTivXhRuStVsOX8q fksqIdmbke68jfL5DfePwyqOxw1Aee4pUqlp9fy6nBSDR0SbpZYaxXJRiu/eeaT8u9e9 SB9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=TrXn/UsH93lS2A5CebmmFeBERFF7L0+KOIci0sQ1Uks=; b=I+/tTGlUU+EIQSyTTzNSVgBM/y7AXwxrk4b4CzIjw4VKKst5lcYABqd07RGGpvmK67 ohfQg/rgTsQKbexbzjSZnKoYy28W1Q1zptV5x6CQVQi0Sawwb5gvcWyQbg+OR2KF4ZjK dwwZ64hhxkSRAJlPSVT3V3HbcBNosfHKR+Z5m7/06Q8PP2F4m5zlrv1KbMOL9LH8ebAS 4SfyVHTHR9SzysnRLyuNoZGEerF0XXx0GaQARlgYsRg+78mQlUvfPuqCuviMwhtRTrzZ IxCTTdb7Q9ie4oFQstL1P+2fyZJeoVUxjCX+3HcV+eOGYJCqo4PT6XLzRl5ypxGAUXPP IOig== X-Gm-Message-State: AFqh2kq7hHcKwZDjHFnoOsklK+cqHOVLSDCst0ZRFPYTqi0Uyy4EPWVl CPq7qa77GPBLAcofVc2RUZM= X-Google-Smtp-Source: AMrXdXuyJ1YtBdHGDBrXpTjR4oFElnKEIBA94XIVucv+SZ35U5AmIuwdEI9LLOpzEQXjNNyrwP0wag== X-Received: by 2002:a17:90a:f80e:b0:225:d290:dcb0 with SMTP id ij14-20020a17090af80e00b00225d290dcb0mr44797327pjb.4.1672953881187; Thu, 05 Jan 2023 13:24:41 -0800 (PST) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id y1-20020a17090a134100b002260b06befbsm1736859pjf.16.2023.01.05.13.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 13:24:40 -0800 (PST) Sender: Tejun Heo From: Tejun Heo To: axboe@kernel.dk, josef@toxicpanda.com, hch@lst.de Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCH 1/4] blkcg: Drop unnecessary RCU read [un]locks from blkg_conf_prep/finish() Date: Thu, 5 Jan 2023 11:24:29 -1000 Message-Id: <20230105212432.289569-2-tj@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230105212432.289569-1-tj@kernel.org> References: <20230105212432.289569-1-tj@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Holding the queue lock now implies RCU read lock, so no need to use rcu_read_[un]lock() explicitly. This shouldn't cause any behavior changes. While at it, drop __acquires() annotation on the queue lock too. The __acquires() part was already out of sync and it doesn't catch anything that lockdep can't. Signed-off-by: Tejun Heo --- block/blk-cgroup.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index ce6a2b7d3dfb..99674e23cf88 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -672,12 +672,11 @@ struct block_device *blkcg_conf_open_bdev(char **inputp) * * Parse per-blkg config update from @input and initialize @ctx with the * result. @ctx->blkg points to the blkg to be updated and @ctx->body the - * part of @input following MAJ:MIN. This function returns with RCU read - * lock and queue lock held and must be paired with blkg_conf_finish(). + * part of @input following MAJ:MIN. This function returns with queue lock + * held and must be paired with blkg_conf_finish(). */ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, char *input, struct blkg_conf_ctx *ctx) - __acquires(rcu) __acquires(&bdev->bd_queue->queue_lock) { struct block_device *bdev; struct gendisk *disk; @@ -699,7 +698,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, if (ret) goto fail; - rcu_read_lock(); spin_lock_irq(&q->queue_lock); if (!blkcg_policy_enabled(q, pol)) { @@ -728,7 +726,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, /* Drop locks to do new blkg allocation with GFP_KERNEL. */ spin_unlock_irq(&q->queue_lock); - rcu_read_unlock(); new_blkg = blkg_alloc(pos, disk, GFP_KERNEL); if (unlikely(!new_blkg)) { @@ -742,7 +739,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, goto fail_exit_queue; } - rcu_read_lock(); spin_lock_irq(&q->queue_lock); if (!blkcg_policy_enabled(q, pol)) { @@ -778,7 +774,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, radix_tree_preload_end(); fail_unlock: spin_unlock_irq(&q->queue_lock); - rcu_read_unlock(); fail_exit_queue: blk_queue_exit(q); fail: @@ -805,10 +800,8 @@ EXPORT_SYMBOL_GPL(blkg_conf_prep); * with blkg_conf_prep(). */ void blkg_conf_finish(struct blkg_conf_ctx *ctx) - __releases(&ctx->bdev->bd_queue->queue_lock) __releases(rcu) { spin_unlock_irq(&bdev_get_queue(ctx->bdev)->queue_lock); - rcu_read_unlock(); blkdev_put_no_open(ctx->bdev); } EXPORT_SYMBOL_GPL(blkg_conf_finish); From patchwork Thu Jan 5 21:24:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 13090496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F339C3DA7A for ; Thu, 5 Jan 2023 21:24:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235750AbjAEVYs (ORCPT ); Thu, 5 Jan 2023 16:24:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235735AbjAEVYp (ORCPT ); Thu, 5 Jan 2023 16:24:45 -0500 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C513860861; Thu, 5 Jan 2023 13:24:43 -0800 (PST) Received: by mail-pj1-x1029.google.com with SMTP id z4-20020a17090a170400b00226d331390cso2282946pjd.5; Thu, 05 Jan 2023 13:24:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=77BfG96ZVCPgJEacxhG3fJgeDXHfTQZdy/9k2/xE4DY=; b=khcIIaQqkFEsI4VrL1Y0Lgt9YBLm65okXiQJur2IXuzYwoGgsaJIXm3Rj8ZLapMJMz I95BTDfIOJD7tiCvICQCjkmRWaKSVkQfoKOseeJHf4XsJLam8pSg1IQgeACFtmd4XVPg xwV7Llwt78O9X0LTfk4XSmSH41qCBLNzn028W5eEP17vAfQQrAy6hkYVA+xCNVaJbEsn JyNHv34Wf5n8pYKIzS60TpK6RO89B8DYJUIRfMw76iZQTKTdPKBvdjvr5UVlhzBCa8Zb yF643AG32f3e+OabA6ZttdF+X9enhm1ksTUQQ6alE8VO6qwkNUgezj8gJwXAbmDnOs9R MGYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=77BfG96ZVCPgJEacxhG3fJgeDXHfTQZdy/9k2/xE4DY=; b=qSg1mh1MR1zv+SNV1oYsPNt9D/cX9hfZfVmY2VGZjhDjIZV7nwKp38syUvxqlqHL1N V6EknZsvWRIKz+eAiXgwQgNRcsEHh//1rZPdnuVuoYDC6IM7LE4kI1qvISh4hN6ddIRK ju7lg9DENLoZNUU3bMlrBVjITdZOgGxfUnBATpnyyEEMkIjvhiWFkqUkc820846YhFgK AZzsOPPtNY1vH3J2oetX9ux34NIKZ657/cIBqtc1jFqQxIhEXX0qgcY0DxO43sGcs9JP 8AqatggwjqueraWaecwWbDgWtGsKWYdDasouhSx2zf9T48K6QBeeZkctZgcn1LAPEsZV QItA== X-Gm-Message-State: AFqh2koTTmb0+yXmQK7hlbm8RcEMXAUYfNxsjPmc8pzJ+iAb8YtQ/eWx JwPLh3Zkgl9h/OgnGU2e2qI= X-Google-Smtp-Source: AMrXdXsIpxGA5eOmOcOpTkcy678ni/Cqf0YEZZ9mRHSSgmEf8dax8jPozHHllKgrqE9Bi9xEOkEe8Q== X-Received: by 2002:a17:902:b58e:b0:191:24a:63e3 with SMTP id a14-20020a170902b58e00b00191024a63e3mr51668397pls.50.1672953883081; Thu, 05 Jan 2023 13:24:43 -0800 (PST) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id d14-20020a170902654e00b0017f73dc1549sm3097981pln.263.2023.01.05.13.24.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 13:24:42 -0800 (PST) Sender: Tejun Heo From: Tejun Heo To: axboe@kernel.dk, josef@toxicpanda.com, hch@lst.de Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCH 2/4] blkcg: Restructure blkg_conf_prep() and friends Date: Thu, 5 Jan 2023 11:24:30 -1000 Message-Id: <20230105212432.289569-3-tj@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230105212432.289569-1-tj@kernel.org> References: <20230105212432.289569-1-tj@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org We want to support lazy init of rq-qos policies so that iolatency is enabled lazily on configuration instead of gendisk initialization. The way blkg config helpers are structured now is a bit awkward for that. Let's restructure: * blkcg_conf_open_bdev() is renamed to blkg_conf_open_bdev(). The blkcg_ prefix was used because the bdev opening step is blkg-independent. However, the distinction is too subtle and confuses more than helps. Let's switch to blkg prefix so that it's consistent with the type and other helper names. * struct blkg_conf_ctx now remembers the original input string and is always initialized by the new blkg_conf_init(). * blkg_conf_open_bdev() is updated to take a pointer to blkg_conf_ctx like blkg_conf_prep() and can be called multiple times safely. Instead of modifying the double pointer to input string directly, blkg_conf_open_bdev() now sets blkg_conf_ctx->body. * blkg_conf_finish() is renamed to blkg_conf_exit() for symmetry and now must be called on all blkg_conf_ctx's which were initialized with blkg_conf_init(). Combined, this allows the users to either open the bdev first or do it altogether with blkg_conf_prep() which will help implementing lazy init of rq-qos policies. Users are updated accordingly. No behavior change is intended by this patch. v2: bfq wasn't updated in v1 causing a build error. Fixed. Signed-off-by: Tejun Heo Cc: Josef Bacik Cc: Christoph Hellwig --- block/bfq-cgroup.c | 8 ++-- block/blk-cgroup.c | 105 +++++++++++++++++++++++++++--------------- block/blk-cgroup.h | 10 ++-- block/blk-iocost.c | 58 +++++++++++++---------- block/blk-iolatency.c | 8 ++-- block/blk-throttle.c | 16 ++++--- 6 files changed, 127 insertions(+), 78 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index a6e8da5f5cfd..97925793aee4 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -1115,9 +1115,11 @@ static ssize_t bfq_io_set_device_weight(struct kernfs_open_file *of, struct bfq_group *bfqg; u64 v; - ret = blkg_conf_prep(blkcg, &blkcg_policy_bfq, buf, &ctx); + blkg_conf_init(&ctx, buf); + + ret = blkg_conf_prep(blkcg, &blkcg_policy_bfq, &ctx); if (ret) - return ret; + goto out; if (sscanf(ctx.body, "%llu", &v) == 1) { /* require "default" on dfl */ @@ -1139,7 +1141,7 @@ static ssize_t bfq_io_set_device_weight(struct kernfs_open_file *of, ret = 0; } out: - blkg_conf_finish(&ctx); + blkg_conf_exit(&ctx); return ret ?: nbytes; } diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 99674e23cf88..d8e0625cd12d 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -626,68 +626,92 @@ u64 __blkg_prfill_u64(struct seq_file *sf, struct blkg_policy_data *pd, u64 v) EXPORT_SYMBOL_GPL(__blkg_prfill_u64); /** - * blkcg_conf_open_bdev - parse and open bdev for per-blkg config update - * @inputp: input string pointer + * blkg_conf_init - initialize a blkg_conf_ctx + * @ctx: blkg_conf_ctx to initialize + * @input: input string * - * Parse the device node prefix part, MAJ:MIN, of per-blkg config update - * from @input and get and return the matching bdev. *@inputp is - * updated to point past the device node prefix. Returns an ERR_PTR() - * value on error. + * Initialize @ctx which can be used to parse blkg config input string @input. + * Once initialized, @ctx can be used with blkg_conf_open_bdev() and + * blkg_conf_prep(), and must be cleaned up with blkg_conf_exit(). + */ +void blkg_conf_init(struct blkg_conf_ctx *ctx, char *input) +{ + *ctx = (struct blkg_conf_ctx){ .input = input }; +} +EXPORT_SYMBOL_GPL(blkg_conf_init); + +/** + * blkg_conf_open_bdev - parse and open bdev for per-blkg config update + * @ctx: blkg_conf_ctx initialized with blkg_conf_init() * - * Use this function iff blkg_conf_prep() can't be used for some reason. + * Parse the device node prefix part, MAJ:MIN, of per-blkg config update from + * @ctx->input and get and store the matching bdev in @ctx->bdev. @ctx->body is + * set to point past the device node prefix. + * + * This function may be called multiple times on @ctx and the extra calls become + * NOOPs. blkg_conf_prep() implicitly calls this function. Use this function + * explicitly if bdev access is needed without resolving the blkcg / policy part + * of @ctx->input. Returns -errno on error. */ -struct block_device *blkcg_conf_open_bdev(char **inputp) +int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx) { - char *input = *inputp; + char *input = ctx->input; unsigned int major, minor; struct block_device *bdev; int key_len; + if (ctx->bdev) + return 0; + if (sscanf(input, "%u:%u%n", &major, &minor, &key_len) != 2) - return ERR_PTR(-EINVAL); + return -EINVAL; input += key_len; if (!isspace(*input)) - return ERR_PTR(-EINVAL); + return -EINVAL; input = skip_spaces(input); bdev = blkdev_get_no_open(MKDEV(major, minor)); if (!bdev) - return ERR_PTR(-ENODEV); + return -ENODEV; if (bdev_is_partition(bdev)) { blkdev_put_no_open(bdev); - return ERR_PTR(-ENODEV); + return -ENODEV; } - *inputp = input; - return bdev; + ctx->body = input; + ctx->bdev = bdev; + return 0; } /** * blkg_conf_prep - parse and prepare for per-blkg config update * @blkcg: target block cgroup * @pol: target policy - * @input: input string - * @ctx: blkg_conf_ctx to be filled + * @ctx: blkg_conf_ctx initialized with blkg_conf_init() + * + * Parse per-blkg config update from @ctx->input and initialize @ctx + * accordingly. On success, @ctx->body points to the part of @ctx->input + * following MAJ:MIN, @ctx->bdev points to the target block device and + * @ctx->blkg to the blkg being configured. * - * Parse per-blkg config update from @input and initialize @ctx with the - * result. @ctx->blkg points to the blkg to be updated and @ctx->body the - * part of @input following MAJ:MIN. This function returns with queue lock - * held and must be paired with blkg_conf_finish(). + * blkg_conf_open_bdev() may be called on @ctx beforehand. On success, this + * function returns with queue lock held and must be followed by + * blkg_conf_exit(). */ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, - char *input, struct blkg_conf_ctx *ctx) + struct blkg_conf_ctx *ctx) { - struct block_device *bdev; struct gendisk *disk; struct request_queue *q; struct blkcg_gq *blkg; int ret; - bdev = blkcg_conf_open_bdev(&input); - if (IS_ERR(bdev)) - return PTR_ERR(bdev); - disk = bdev->bd_disk; + ret = blkg_conf_open_bdev(ctx); + if (ret) + return ret; + + disk = ctx->bdev->bd_disk; q = disk->queue; /* @@ -765,9 +789,7 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, } success: blk_queue_exit(q); - ctx->bdev = bdev; ctx->blkg = blkg; - ctx->body = input; return 0; fail_preloaded: @@ -777,7 +799,6 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, fail_exit_queue: blk_queue_exit(q); fail: - blkdev_put_no_open(bdev); /* * If queue was bypassing, we should retry. Do so after a * short msleep(). It isn't strictly necessary but queue @@ -793,18 +814,26 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, EXPORT_SYMBOL_GPL(blkg_conf_prep); /** - * blkg_conf_finish - finish up per-blkg config update - * @ctx: blkg_conf_ctx initialized by blkg_conf_prep() + * blkg_conf_exit - clean up per-blkg config update + * @ctx: blkg_conf_ctx initialized with blkg_conf_init() * - * Finish up after per-blkg config update. This function must be paired - * with blkg_conf_prep(). + * Clean up after per-blkg config update. This function must be called on all + * blkg_conf_ctx's initialized with blkg_conf_init(). */ -void blkg_conf_finish(struct blkg_conf_ctx *ctx) +void blkg_conf_exit(struct blkg_conf_ctx *ctx) { - spin_unlock_irq(&bdev_get_queue(ctx->bdev)->queue_lock); - blkdev_put_no_open(ctx->bdev); + if (ctx->blkg) { + spin_unlock_irq(&bdev_get_queue(ctx->bdev)->queue_lock); + ctx->blkg = NULL; + } + + if (ctx->bdev) { + blkdev_put_no_open(ctx->bdev); + ctx->body = NULL; + ctx->bdev = NULL; + } } -EXPORT_SYMBOL_GPL(blkg_conf_finish); +EXPORT_SYMBOL_GPL(blkg_conf_exit); static void blkg_iostat_set(struct blkg_iostat *dst, struct blkg_iostat *src) { diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index 1e94e404eaa8..fe09e8b4c2a8 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -208,15 +208,17 @@ void blkcg_print_blkgs(struct seq_file *sf, struct blkcg *blkcg, u64 __blkg_prfill_u64(struct seq_file *sf, struct blkg_policy_data *pd, u64 v); struct blkg_conf_ctx { + char *input; + char *body; struct block_device *bdev; struct blkcg_gq *blkg; - char *body; }; -struct block_device *blkcg_conf_open_bdev(char **inputp); +void blkg_conf_init(struct blkg_conf_ctx *ctx, char *input); +int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx); int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, - char *input, struct blkg_conf_ctx *ctx); -void blkg_conf_finish(struct blkg_conf_ctx *ctx); + struct blkg_conf_ctx *ctx); +void blkg_conf_exit(struct blkg_conf_ctx *ctx); /** * bio_issue_as_root_blkg - see if this bio needs to be issued as root blkg diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 6955605629e4..22a3639a7a05 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -3091,9 +3091,11 @@ static ssize_t ioc_weight_write(struct kernfs_open_file *of, char *buf, return nbytes; } - ret = blkg_conf_prep(blkcg, &blkcg_policy_iocost, buf, &ctx); + blkg_conf_init(&ctx, buf); + + ret = blkg_conf_prep(blkcg, &blkcg_policy_iocost, &ctx); if (ret) - return ret; + goto err; iocg = blkg_to_iocg(ctx.blkg); @@ -3112,12 +3114,14 @@ static ssize_t ioc_weight_write(struct kernfs_open_file *of, char *buf, weight_updated(iocg, &now); spin_unlock(&iocg->ioc->lock); - blkg_conf_finish(&ctx); + blkg_conf_exit(&ctx); return nbytes; einval: - blkg_conf_finish(&ctx); - return -EINVAL; + ret = -EINVAL; +err: + blkg_conf_exit(&ctx); + return ret; } static u64 ioc_qos_prfill(struct seq_file *sf, struct blkg_policy_data *pd, @@ -3172,19 +3176,22 @@ static const match_table_t qos_tokens = { static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, size_t nbytes, loff_t off) { - struct block_device *bdev; + struct blkg_conf_ctx ctx; struct gendisk *disk; struct ioc *ioc; u32 qos[NR_QOS_PARAMS]; bool enable, user; - char *p; + char *body, *p; int ret; - bdev = blkcg_conf_open_bdev(&input); - if (IS_ERR(bdev)) - return PTR_ERR(bdev); + blkg_conf_init(&ctx, input); - disk = bdev->bd_disk; + ret = blkg_conf_open_bdev(&ctx); + if (ret) + goto err; + + body = ctx.body; + disk = ctx.bdev->bd_disk; ioc = q_to_ioc(disk->queue); if (!ioc) { ret = blk_iocost_init(disk); @@ -3201,7 +3208,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, enable = ioc->enabled; user = ioc->user_qos_params; - while ((p = strsep(&input, " \t\n"))) { + while ((p = strsep(&body, " \t\n"))) { substring_t args[MAX_OPT_ARGS]; char buf[32]; int tok; @@ -3290,7 +3297,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, blk_mq_unquiesce_queue(disk->queue); blk_mq_unfreeze_queue(disk->queue); - blkdev_put_no_open(bdev); + blkg_conf_exit(&ctx); return nbytes; einval: spin_unlock_irq(&ioc->lock); @@ -3300,7 +3307,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, ret = -EINVAL; err: - blkdev_put_no_open(bdev); + blkg_conf_exit(&ctx); return ret; } @@ -3351,22 +3358,25 @@ static const match_table_t i_lcoef_tokens = { static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input, size_t nbytes, loff_t off) { - struct block_device *bdev; + struct blkg_conf_ctx ctx; struct request_queue *q; struct ioc *ioc; u64 u[NR_I_LCOEFS]; bool user; - char *p; + char *body, *p; int ret; - bdev = blkcg_conf_open_bdev(&input); - if (IS_ERR(bdev)) - return PTR_ERR(bdev); + blkg_conf_init(&ctx, input); + + ret = blkg_conf_open_bdev(&ctx); + if (ret) + goto err; - q = bdev_get_queue(bdev); + body = ctx.body; + q = bdev_get_queue(ctx.bdev); ioc = q_to_ioc(q); if (!ioc) { - ret = blk_iocost_init(bdev->bd_disk); + ret = blk_iocost_init(ctx.bdev->bd_disk); if (ret) goto err; ioc = q_to_ioc(q); @@ -3379,7 +3389,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input, memcpy(u, ioc->params.i_lcoefs, sizeof(u)); user = ioc->user_cost_model; - while ((p = strsep(&input, " \t\n"))) { + while ((p = strsep(&body, " \t\n"))) { substring_t args[MAX_OPT_ARGS]; char buf[32]; int tok; @@ -3426,7 +3436,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input, blk_mq_unquiesce_queue(q); blk_mq_unfreeze_queue(q); - blkdev_put_no_open(bdev); + blkg_conf_exit(&ctx); return nbytes; einval: @@ -3437,7 +3447,7 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input, ret = -EINVAL; err: - blkdev_put_no_open(bdev); + blkg_conf_exit(&ctx); return ret; } diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index ecdc10741836..3b3667f397a9 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -842,9 +842,11 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, u64 oldval; int ret; - ret = blkg_conf_prep(blkcg, &blkcg_policy_iolatency, buf, &ctx); + blkg_conf_init(&ctx, buf); + + ret = blkg_conf_prep(blkcg, &blkcg_policy_iolatency, &ctx); if (ret) - return ret; + goto out; iolat = blkg_to_lat(ctx.blkg); p = ctx.body; @@ -880,7 +882,7 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, iolatency_clear_scaling(blkg); ret = 0; out: - blkg_conf_finish(&ctx); + blkg_conf_exit(&ctx); return ret ?: nbytes; } diff --git a/block/blk-throttle.c b/block/blk-throttle.c index 6fb5a2f9e1ee..75841d1d9bf4 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -1369,9 +1369,11 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of, int ret; u64 v; - ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, buf, &ctx); + blkg_conf_init(&ctx, buf); + + ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, &ctx); if (ret) - return ret; + goto out_finish; ret = -EINVAL; if (sscanf(ctx.body, "%llu", &v) != 1) @@ -1390,7 +1392,7 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of, tg_conf_updated(tg, false); ret = 0; out_finish: - blkg_conf_finish(&ctx); + blkg_conf_exit(&ctx); return ret ?: nbytes; } @@ -1562,9 +1564,11 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of, int ret; int index = of_cft(of)->private; - ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, buf, &ctx); + blkg_conf_init(&ctx, buf); + + ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, &ctx); if (ret) - return ret; + goto out_finish; tg = blkg_to_tg(ctx.blkg); tg_update_carryover(tg); @@ -1663,7 +1667,7 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of, tg->td->limit_valid[LIMIT_LOW]); ret = 0; out_finish: - blkg_conf_finish(&ctx); + blkg_conf_exit(&ctx); return ret ?: nbytes; } From patchwork Thu Jan 5 21:24:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 13090498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 859A0C54EBC for ; Thu, 5 Jan 2023 21:24:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235752AbjAEVYs (ORCPT ); Thu, 5 Jan 2023 16:24:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235738AbjAEVYq (ORCPT ); Thu, 5 Jan 2023 16:24:46 -0500 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B91B631AD; Thu, 5 Jan 2023 13:24:45 -0800 (PST) Received: by mail-pj1-x1029.google.com with SMTP id z4-20020a17090a170400b00226d331390cso2283028pjd.5; Thu, 05 Jan 2023 13:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=3WPbvsPoBO1wdRnnR1ui7BApmCnbJf4jlfv1HBKIBy8=; b=h7Rkd2eoAqOSb3NPQcC429lnxpLGOUQWpdpDfVRumSo56ccDcW26pC/hLGdMgC/Iic QgeeKnII9Fkf8pa3RVL9oEpVLXr8eNHSU4BhUMszxqbHlRyDNUTknRTO8JN+aD0Z7Hh4 ILf01WtrtaEjA9OjrHxoW1rKO1wJzGbv1L8QqO3wAlSfVALhhjT6oW45MkTD1qixXnuC pjc6+ch1UdJD8mjiS0+EJsG9dSeZyJlTmvMRzqFgVHC0xiF0aviW8kOnr9tC/zkdOszn C2AiozuHZoAYbEumaaFxY5clPdUZXF8TeGh+gPTJi8kDsfLVPAQIVnoFN7GyykDGZ6M1 cYRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3WPbvsPoBO1wdRnnR1ui7BApmCnbJf4jlfv1HBKIBy8=; b=DCEo0iGvo6tzKgjM7O1XXvjdLNyB1p25NHjSJUes58Nr8vLmK/aQZEHw26vm+oobtb ML89g+h1dD/iGMmo68p5su39IAHAbiR72mG7OEt4KbF/FVrBLwS9/y/n/67fPz6Ol2HJ 794p3NRoY5Ctpxs2ltYC6tKoxutwb75xN54ZTOiwF1iBKdHNYOzQ4h8/I/XUtXW7y+Ol 9FYMTU7J8Pkc0HJ8dpy8niVlnf1YPmkfbn3kxHy/8CGNnNsHhgV0Z8QnvmtbWLF2BPD8 4eL2wfXOgLpeqDXTZfkByQujRWAtWbt5w+ANOsTL7yp8+o4H0z6N/fgWi/9laC0U19Rp NIgQ== X-Gm-Message-State: AFqh2kqlnMNUKzXUTzyPx99DGuYUPlkVIi2cLf7OJRctbpKAlINhOD8G J5Xue6xMJGxg8dumf21GGmg= X-Google-Smtp-Source: AMrXdXv8TvNCpT/+B43gLCoaPxf6EUgyZ6sV9YGzP9vxvPxJPkB0OQ1g0jouAN2nf8P4uTl8EBor4A== X-Received: by 2002:a17:902:f24c:b0:189:7d30:7623 with SMTP id j12-20020a170902f24c00b001897d307623mr47885881plc.30.1672953884912; Thu, 05 Jan 2023 13:24:44 -0800 (PST) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id z4-20020a1709027e8400b0018685257c0dsm3689527pla.58.2023.01.05.13.24.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 13:24:44 -0800 (PST) Sender: Tejun Heo From: Tejun Heo To: axboe@kernel.dk, josef@toxicpanda.com, hch@lst.de Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCH 3/4] blk-iolatency: s/blkcg_rq_qos/iolat_rq_qos/ Date: Thu, 5 Jan 2023 11:24:31 -1000 Message-Id: <20230105212432.289569-4-tj@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230105212432.289569-1-tj@kernel.org> References: <20230105212432.289569-1-tj@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The name was too generic given that there are multiple blkcg rq-qos policies. Signed-off-by: Tejun Heo Cc: Josef Bacik Reviewed-by: Christoph Hellwig --- block/blk-iolatency.c | 2 +- block/blk-rq-qos.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 3b3667f397a9..3601345808d2 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -976,7 +976,7 @@ static void iolatency_pd_init(struct blkg_policy_data *pd) { struct iolatency_grp *iolat = pd_to_lat(pd); struct blkcg_gq *blkg = lat_to_blkg(iolat); - struct rq_qos *rqos = blkcg_rq_qos(blkg->q); + struct rq_qos *rqos = iolat_rq_qos(blkg->q); struct blk_iolatency *blkiolat = BLKIOLATENCY(rqos); u64 now = ktime_to_ns(ktime_get()); int cpu; diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index 1ef1f7d4bc3c..27f004fae66b 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -74,7 +74,7 @@ static inline struct rq_qos *wbt_rq_qos(struct request_queue *q) return rq_qos_id(q, RQ_QOS_WBT); } -static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q) +static inline struct rq_qos *iolat_rq_qos(struct request_queue *q) { return rq_qos_id(q, RQ_QOS_LATENCY); } From patchwork Thu Jan 5 21:24:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 13090497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C96A4C3DA7D for ; Thu, 5 Jan 2023 21:24:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235754AbjAEVYt (ORCPT ); Thu, 5 Jan 2023 16:24:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235725AbjAEVYs (ORCPT ); Thu, 5 Jan 2023 16:24:48 -0500 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DCE32F795; Thu, 5 Jan 2023 13:24:47 -0800 (PST) Received: by mail-pj1-x1029.google.com with SMTP id z4-20020a17090a170400b00226d331390cso2283135pjd.5; Thu, 05 Jan 2023 13:24:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=ptP+TRS/lGASh4wVWy3dTAfwv7VhGgtUgLBdAb814ZI=; b=V7T0Xn1YdlFcaVEgPKSUl8rZlRg3c67hT1gEvABoMED52yoSZTGLsKP/5fmVcP8GjN nqbdzjRpwoF6vrnnuJcKluCYDHOGqepCifxWkuL6gVqmPGz4ZCkrHTsG8r2+R+oW0K17 8LDDEMfCkh9D5QNAEPexlTwfdi1OlNTwsuc8bnf92AcZgL+SBVU62W9pPy1VKQp0MncR YzeenEu2xyKEmAlPCT88MOb+oJk20xKflj9qbJSLZCULPGgvQSAXYr/Vk23TKgDcbpUJ qaMBMxRLOj1uAlPqv+GcZu26tVU49iiiYjGRyx54DxB8ZlUACWfTBARjjw+IWlOgJ6XS NVXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ptP+TRS/lGASh4wVWy3dTAfwv7VhGgtUgLBdAb814ZI=; b=NpoDgZutX5RJFrB+VigjzRoGeVSjMb3JajH7QFkuIyKS1b6Ys/mmhGPPapyvKKMtM5 5X+B3XXK8h1a+09U/bXsSACeGzudoII9Vw2P9M/ijpGuvuyCHlT7TM5D6Z9p3DRPWwkk 2kT0m8nVn/qgNQHBzqmRomT++FVNLO44BXTg0hATMvIezuQwqq5qcJ6JF5iWXwOAbvL3 BWRxoV57ExQCg9r3sHf2GjU5ImTls1+6+c3yPWNiDZ73Z8l5DVWv5cGu4y4gxjsAhHsE jkvNYOMGU2QeSAkCoRza4O1ZtVBCT0tszLLJNhyJL1mheA1rByFwSe0ntnadsx5slIXe 8KeA== X-Gm-Message-State: AFqh2kqKfYV05gwaf2sTQhcbiIo6yc2pJ5s1WgHCozK2lE+6COl1a3ZY RFjlie61yViuuxsa3GonhDI= X-Google-Smtp-Source: AMrXdXsK3z8WPZqVWjPtPqP3vCNRacedh6LJsHsPm3ZgpPyCKjlF7RTeV3Ef3amQLbetdglMcN2iFA== X-Received: by 2002:a17:90b:215:b0:225:bf26:4928 with SMTP id fy21-20020a17090b021500b00225bf264928mr54101453pjb.38.1672953886879; Thu, 05 Jan 2023 13:24:46 -0800 (PST) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id x5-20020a17090aa38500b00219cf5c3829sm1732645pjp.57.2023.01.05.13.24.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 13:24:46 -0800 (PST) Sender: Tejun Heo From: Tejun Heo To: axboe@kernel.dk, josef@toxicpanda.com, hch@lst.de Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Tejun Heo Subject: [PATCH 4/4] blk-iolatency: Make initialization lazy Date: Thu, 5 Jan 2023 11:24:32 -1000 Message-Id: <20230105212432.289569-5-tj@kernel.org> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230105212432.289569-1-tj@kernel.org> References: <20230105212432.289569-1-tj@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Other rq_qos policies such as wbt and iocost are lazy-initialized when they are configured for the first time for the device but iolatency is initialized unconditionally from blkcg_init_disk() during gendisk init. Lazy init is beneficial because rq_qos policies add runtime overhead when initialized as every IO has to walk all registered rq_qos callbacks. This patch switches iolatency to lazy initialization too so that it only registered its rq_qos policy when it is first configured. Note that there is a known race condition between blkcg config file writes and del_gendisk() and this patch makes iolatency susceptible to it by exposing the init path to race against the deletion path. However, that problem already exists in iocost and is being worked on. Signed-off-by: Tejun Heo Cc: Josef Bacik Cc: Christoph Hellwig Reviewed-by: Christoph Hellwig --- block/blk-cgroup.c | 8 -------- block/blk-iolatency.c | 29 ++++++++++++++++++++++++++++- block/blk.h | 6 ------ 3 files changed, 28 insertions(+), 15 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index d8e0625cd12d..844579aff363 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -33,7 +33,6 @@ #include "blk-cgroup.h" #include "blk-ioprio.h" #include "blk-throttle.h" -#include "blk-rq-qos.h" /* * blkcg_pol_mutex protects blkcg_policy[] and policy [de]activation. @@ -1322,14 +1321,8 @@ int blkcg_init_disk(struct gendisk *disk) if (ret) goto err_ioprio_exit; - ret = blk_iolatency_init(disk); - if (ret) - goto err_throtl_exit; - return 0; -err_throtl_exit: - blk_throtl_exit(disk); err_ioprio_exit: blk_ioprio_exit(disk); err_destroy_all: @@ -1345,7 +1338,6 @@ int blkcg_init_disk(struct gendisk *disk) void blkcg_exit_disk(struct gendisk *disk) { blkg_destroy_all(disk); - rq_qos_exit(disk->queue); blk_throtl_exit(disk); } diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 3601345808d2..3484393dbc4a 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -755,7 +755,7 @@ static void blkiolatency_enable_work_fn(struct work_struct *work) } } -int blk_iolatency_init(struct gendisk *disk) +static int blk_iolatency_init(struct gendisk *disk) { struct request_queue *q = disk->queue; struct blk_iolatency *blkiolat; @@ -830,6 +830,29 @@ static void iolatency_clear_scaling(struct blkcg_gq *blkg) } } +static int blk_iolatency_try_init(struct blkg_conf_ctx *ctx) +{ + static DEFINE_MUTEX(init_mutex); + int ret; + + ret = blkg_conf_open_bdev(ctx); + if (ret) + return ret; + + /* + * blk_iolatency_init() may fail after rq_qos_add() succeeds which can + * confuse iolat_rq_qos() test. Make the test and init atomic. + */ + mutex_lock(&init_mutex); + + if (!iolat_rq_qos(ctx->bdev->bd_queue)) + ret = blk_iolatency_init(ctx->bdev->bd_disk); + + mutex_unlock(&init_mutex); + + return ret; +} + static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, size_t nbytes, loff_t off) { @@ -844,6 +867,10 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, blkg_conf_init(&ctx, buf); + ret = blk_iolatency_try_init(&ctx); + if (ret) + goto out; + ret = blkg_conf_prep(blkcg, &blkcg_policy_iolatency, &ctx); if (ret) goto out; diff --git a/block/blk.h b/block/blk.h index 4c3b3325219a..78f1706cddca 100644 --- a/block/blk.h +++ b/block/blk.h @@ -392,12 +392,6 @@ static inline struct bio *blk_queue_bounce(struct bio *bio, return bio; } -#ifdef CONFIG_BLK_CGROUP_IOLATENCY -int blk_iolatency_init(struct gendisk *disk); -#else -static inline int blk_iolatency_init(struct gendisk *disk) { return 0; }; -#endif - #ifdef CONFIG_BLK_DEV_ZONED void disk_free_zone_bitmaps(struct gendisk *disk); void disk_clear_zone_settings(struct gendisk *disk);