From patchwork Tue May 29 21:17:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10437089 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0FEB5601E9 for ; Tue, 29 May 2018 21:18:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2BD728991 for ; Tue, 29 May 2018 21:17:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E74932899C; Tue, 29 May 2018 21:17:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F177E28991 for ; Tue, 29 May 2018 21:17:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B1A76B026D; Tue, 29 May 2018 17:17:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2CD9B6B026C; Tue, 29 May 2018 17:17:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C0296B026C; Tue, 29 May 2018 17:17:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk0-f197.google.com (mail-qk0-f197.google.com [209.85.220.197]) by kanga.kvack.org (Postfix) with ESMTP id CBEA66B026A for ; Tue, 29 May 2018 17:17:43 -0400 (EDT) Received: by mail-qk0-f197.google.com with SMTP id w203-v6so11928053qkb.16 for ; Tue, 29 May 2018 14:17:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=WmbJJ4mEsBtiePOvVC7Kj6eFrizCpZYgmZRaGhMQo6s=; b=sZ0MxB4XJ3sOLTQY7e4lkxWtSFeqhnvBIdvbDeAboG85qE/NkV64qVLGxQzsknnSzo yQbMcSos4mzAvCRAiB/3rA4p1L0L9rWISWEAyWnF3ahmh27e+4kXECepMpObQFAevNP5 OoUbp5nn+a6K3cuUJzSjrI1VYrmPix8koQsyI76EyQedWeLkS3wUIDM+0pciE6sZhcFx cChXgbxw+3SPJADtQ/JU+o8aX2/vvUbhfw4ya7TGi/saonn3AM0SDIBCVLZT9Nopc3SH frrsW2ALZT0vRrnyM0P9q3lphS4IgJU+ryJfL1U3EI9cKw/Rpyx6Bw/h1eWSLg0TEuPk 0gNg== X-Gm-Message-State: APt69E1FVkDWkHUxhXKIAftD5gH61CS9sAQJh29xvpn6SQSy2jt+MTS4 F/BRaOm2DT5Z1V9lu/WwqCvQnwTVnE0HQdEo+8Q2rGrPLfvPQjTaT1wSjAHIWHe86to2TzKCJSq hz5Cj6m5GYoMpnGQHIyfRE+FMOhbXh7UOdMrKJLrVAo7/uSJXx3/hGVICFYhuf45175TPefF4ID XcAhfrr7CjSN2vNHXRbAUm7vhBAozIFx1jF8aV5EJbkqvBr8rB+doAXs4UPamGeKDCgzL33fO0t Tktz+wZlmLLjKDEh0zWrr1o8kKo4vrnqX8Kwduqk5Aeb5y+eYox5UflnhgkY4O96aDMGYFf2Cs+ 89u5O3E1UDuxQBcCn1pCLKLYvnMQqNnbdADaz4T0ADt1eWDZhSyXgA6mAXK8Ced0hA/9pHGNUpR S X-Received: by 2002:ac8:22bd:: with SMTP id f58-v6mr35703qta.423.1527628663574; Tue, 29 May 2018 14:17:43 -0700 (PDT) X-Received: by 2002:ac8:22bd:: with SMTP id f58-v6mr35663qta.423.1527628662594; Tue, 29 May 2018 14:17:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527628662; cv=none; d=google.com; s=arc-20160816; b=T9dnaQ1nI3ie86geDu0v40raX9+yA0ydtAmz0JwFbvC2SJnn463KEUOz1IfKtpven6 /kpwUhL7AiPnEQhM5hHLBVebnXFgXfCkeELU13UqFxgMCxJNz+OqNyZ8i5TjjCSxw2ZH kKiXjsTEG+XHeQgO3SxyVRKExFOnqgrne3GFuUZrIxBmfwDljAUOd0+pkq/YGx7dLedR pGBvFZ8ImH/WS4C5ZJwRcPUPM83LBkLM/t15P1Up27INb5/4MUoav2CyxfB5oUvearAB 42Te0YBRLN/CjXzpa5pfa3n/gIT3U5u7TLELizwZ80X+FueQYuxka+cWfz0FhPuvav+s oBhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=WmbJJ4mEsBtiePOvVC7Kj6eFrizCpZYgmZRaGhMQo6s=; b=XWwHiM67pSgsoOaK+1Cpg6jgbKjaez5LUbbbMKlnAuS4Ss/RyhDNyc1MbPQgqUzmKl X3jKyCBWfqe2wakLYTMsNmRjcfZX2gegb6e84T72Vda9egt4OaZcvR0WAvRppb715GvG skQrz+lbq+5/Zsox8Hcp6Wys+vjElEYhb+q2r8zLhox847HdP8GiQHVXf7LWZ8bSCBU4 QN9kookZ1SPxc3mGigH+Xi0nTuZPpZgVxmuiXcSnuRq2J9W3x8nu1JihkWCd3rFEKmz7 fJ9VGxX6QC+G/RqzAhm/uJTYLt66sw8edEsc/39/V8eoDHvBxNGhwYB8TafNYD12sXcv cStg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=mlNMnKEY; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id x33-v6sor21139956qte.84.2018.05.29.14.17.42 for (Google Transport Security); Tue, 29 May 2018 14:17:42 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=mlNMnKEY; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WmbJJ4mEsBtiePOvVC7Kj6eFrizCpZYgmZRaGhMQo6s=; b=mlNMnKEYHMk0vNlajwdDdRqWEWlQGy0OfBa4T3TyyNnhOD8yPud8FswdaOvNiWjXT6 MinMK3zDfkv+GuX7fB7nsclgwZQL8thuLUrbFpoVSup7Begnkfy5Hh4pQiw6Ms5NuY7J ZBjvOaURE2fqUYo100MP5hKOok5S41vCI2r8lBQchrWQZUu08rwSKAMshp0hRuvx7vMS v+8rR/Lt3FvHTSdcYS7PVs3mwGEZvu8mlRFL2Jc37OnDxGre2PWFh9i8O0tgXRkQZSSZ QSUoUKW9rldFQTFSdW+t5GqOg6KQJkcmEtEZtAFg1/HSPc2jD6YdLlMznVxKoI0waa0b i05w== X-Google-Smtp-Source: ADUXVKKvzi7tJOJNZSyYU9TE/L5XhAndjmtSUVrSrlB/gvasSuARRgQXhywcCeQOgXYFD8zcpt5EMw== X-Received: by 2002:ac8:2e52:: with SMTP id s18-v6mr43701qta.319.1527628662283; Tue, 29 May 2018 14:17:42 -0700 (PDT) Received: from localhost (cpe-2606-A000-4381-1201-225-22FF-FEB3-E51A.dyn6.twc.com. [2606:a000:4381:1201:225:22ff:feb3:e51a]) by smtp.gmail.com with ESMTPSA id h37-v6sm25259666qtc.68.2018.05.29.14.17.41 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 29 May 2018 14:17:41 -0700 (PDT) From: Josef Bacik To: axboe@kernel.dk, kernel-team@fb.com, linux-block@vger.kernel.org, akpm@linux-foundation.org, linux-mm@kvack.org, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, tj@kernel.org, linux-fsdevel@vger.kernel.org Cc: Josef Bacik Subject: [PATCH 10/13] block: remove external dependency on wbt_flags Date: Tue, 29 May 2018 17:17:21 -0400 Message-Id: <20180529211724.4531-11-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180529211724.4531-1-josef@toxicpanda.com> References: <20180529211724.4531-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Josef Bacik We don't really need to save this stuff in the core block code, we can just pass the bio back into the helpers later on to derive the same flags and update the rq->wbt_flags appropriately. Signed-off-by: Josef Bacik --- block/blk-core.c | 9 ++++----- block/blk-mq.c | 9 ++++----- block/blk-rq-qos.c | 24 +++++++++++++++--------- block/blk-rq-qos.h | 11 ++++++----- block/blk-wbt.c | 52 +++++++++++++++++++++++++++++++++++++++------------- block/blk-wbt.h | 5 ----- 6 files changed, 68 insertions(+), 42 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 07594fb34c0f..af67fa68f635 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -42,7 +42,7 @@ #include "blk.h" #include "blk-mq.h" #include "blk-mq-sched.h" -#include "blk-wbt.h" +#include "blk-rq-qos.h" #ifdef CONFIG_DEBUG_FS struct dentry *blk_debugfs_root; @@ -1994,7 +1994,6 @@ static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio) int where = ELEVATOR_INSERT_SORT; struct request *req, *free; unsigned int request_count = 0; - unsigned int wb_acct; /* * low level driver can indicate that it wants pages above a @@ -2052,7 +2051,7 @@ static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio) } get_rq: - wb_acct = rq_qos_throttle(q, bio, q->queue_lock); + rq_qos_throttle(q, bio, q->queue_lock); /* * Grab a free request. This is might sleep but can not fail. @@ -2062,7 +2061,7 @@ static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio) req = get_request(q, bio->bi_opf, bio, 0, GFP_NOIO); if (IS_ERR(req)) { blk_queue_exit(q); - rq_qos_cleanup(q, wb_acct); + rq_qos_cleanup(q, bio); if (PTR_ERR(req) == -ENOMEM) bio->bi_status = BLK_STS_RESOURCE; else @@ -2071,7 +2070,7 @@ static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio) goto out_unlock; } - wbt_track(req, wb_acct); + rq_qos_track(q, req, bio); /* * After dropping the lock and possibly sleeping here, our request diff --git a/block/blk-mq.c b/block/blk-mq.c index 5308396ee22a..7c55c9d35492 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -34,8 +34,8 @@ #include "blk-mq-debugfs.h" #include "blk-mq-tag.h" #include "blk-stat.h" -#include "blk-wbt.h" #include "blk-mq-sched.h" +#include "blk-rq-qos.h" static bool blk_mq_poll(struct request_queue *q, blk_qc_t cookie); static void blk_mq_poll_stats_start(struct request_queue *q); @@ -1852,7 +1852,6 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) struct blk_plug *plug; struct request *same_queue_rq = NULL; blk_qc_t cookie; - unsigned int wb_acct; blk_queue_bounce(q, &bio); @@ -1868,19 +1867,19 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) if (blk_mq_sched_bio_merge(q, bio)) return BLK_QC_T_NONE; - wb_acct = rq_qos_throttle(q, bio, NULL); + rq_qos_throttle(q, bio, NULL); trace_block_getrq(q, bio, bio->bi_opf); rq = blk_mq_get_request(q, bio, bio->bi_opf, &data); if (unlikely(!rq)) { - rq_qos_cleanup(q, wb_acct); + rq_qos_cleanup(q, bio); if (bio->bi_opf & REQ_NOWAIT) bio_wouldblock_error(bio); return BLK_QC_T_NONE; } - wbt_track(rq, wb_acct); + rq_qos_track(q, rq, bio); cookie = request_to_qc_t(data.hctx, rq); diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c index d2f2af8aa10c..b7b02e04f64f 100644 --- a/block/blk-rq-qos.c +++ b/block/blk-rq-qos.c @@ -1,7 +1,5 @@ #include "blk-rq-qos.h" -#include "blk-wbt.h" - /* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -29,13 +27,13 @@ bool rq_wait_inc_below(struct rq_wait *rq_wait, int limit) return atomic_inc_below(&rq_wait->inflight, limit); } -void rq_qos_cleanup(struct request_queue *q, enum wbt_flags wb_acct) +void rq_qos_cleanup(struct request_queue *q, struct bio *bio) { struct rq_qos *rqos; for (rqos = q->rq_qos; rqos; rqos = rqos->next) { if (rqos->ops->cleanup) - rqos->ops->cleanup(rqos, wb_acct); + rqos->ops->cleanup(rqos, bio); } } @@ -69,17 +67,25 @@ void rq_qos_requeue(struct request_queue *q, struct request *rq) } } -enum wbt_flags rq_qos_throttle(struct request_queue *q, struct bio *bio, - spinlock_t *lock) +void rq_qos_throttle(struct request_queue *q, struct bio *bio, + spinlock_t *lock) { struct rq_qos *rqos; - enum wbt_flags flags = 0; for(rqos = q->rq_qos; rqos; rqos = rqos->next) { if (rqos->ops->throttle) - flags |= rqos->ops->throttle(rqos, bio, lock); + rqos->ops->throttle(rqos, bio, lock); + } +} + +void rq_qos_track(struct request_queue *q, struct request *rq, struct bio *bio) +{ + struct rq_qos *rqos; + + for(rqos = q->rq_qos; rqos; rqos = rqos->next) { + if (rqos->ops->track) + rqos->ops->track(rqos, rq, bio); } - return flags; } /* diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h index a321e9e2ba70..501253676dd8 100644 --- a/block/blk-rq-qos.h +++ b/block/blk-rq-qos.h @@ -25,12 +25,12 @@ struct rq_qos { }; struct rq_qos_ops { - enum wbt_flags (*throttle)(struct rq_qos *, struct bio *, - spinlock_t *); + void (*throttle)(struct rq_qos *, struct bio *, spinlock_t *); + void (*track)(struct rq_qos *, struct request *, struct bio *); void (*issue)(struct rq_qos *, struct request *); void (*requeue)(struct rq_qos *, struct request *); void (*done)(struct rq_qos *, struct request *); - void (*cleanup)(struct rq_qos *, enum wbt_flags); + void (*cleanup)(struct rq_qos *, struct bio *); void (*exit)(struct rq_qos *); }; @@ -82,10 +82,11 @@ void rq_depth_scale_up(struct rq_depth *rqd); void rq_depth_scale_down(struct rq_depth *rqd, bool hard_throttle); bool rq_depth_calc_max_depth(struct rq_depth *rqd); -void rq_qos_cleanup(struct request_queue *, enum wbt_flags); +void rq_qos_cleanup(struct request_queue *, struct bio *); void rq_qos_done(struct request_queue *, struct request *); void rq_qos_issue(struct request_queue *, struct request *); void rq_qos_requeue(struct request_queue *, struct request *); -enum wbt_flags rq_qos_throttle(struct request_queue *, struct bio *, spinlock_t *); +void rq_qos_throttle(struct request_queue *, struct bio *, spinlock_t *); +void rq_qos_track(struct request_queue *q, struct request *, struct bio *); void rq_qos_exit(struct request_queue *); #endif diff --git a/block/blk-wbt.c b/block/blk-wbt.c index 6fe20fb823e4..461a9af11efe 100644 --- a/block/blk-wbt.c +++ b/block/blk-wbt.c @@ -549,41 +549,66 @@ static inline bool wbt_should_throttle(struct rq_wb *rwb, struct bio *bio) } } +static enum wbt_flags bio_to_wbt_flags(struct rq_wb *rwb, struct bio *bio) +{ + enum wbt_flags flags = 0; + + if (bio_op(bio) == REQ_OP_READ) { + flags = WBT_READ; + } else if (wbt_should_throttle(rwb, bio)) { + if (current_is_kswapd()) + flags |= WBT_KSWAPD; + if (bio_op(bio) == REQ_OP_DISCARD) + flags |= WBT_DISCARD; + flags |= WBT_TRACKED; + } + return flags; +} + +static void wbt_cleanup(struct rq_qos *rqos, struct bio *bio) +{ + struct rq_wb *rwb = RQWB(rqos); + enum wbt_flags flags = bio_to_wbt_flags(rwb, bio); + __wbt_done(rqos, flags); +} + /* * Returns true if the IO request should be accounted, false if not. * May sleep, if we have exceeded the writeback limits. Caller can pass * in an irq held spinlock, if it holds one when calling this function. * If we do sleep, we'll release and re-grab it. */ -static enum wbt_flags wbt_wait(struct rq_qos *rqos, struct bio *bio, - spinlock_t *lock) +static void wbt_wait(struct rq_qos *rqos, struct bio *bio, spinlock_t *lock) { struct rq_wb *rwb = RQWB(rqos); - enum wbt_flags ret = 0; + enum wbt_flags flags; if (!rwb_enabled(rwb)) - return 0; + return; - if (bio_op(bio) == REQ_OP_READ) - ret = WBT_READ; + flags = bio_to_wbt_flags(rwb, bio); if (!wbt_should_throttle(rwb, bio)) { - if (ret & WBT_READ) + if (flags & WBT_READ) wb_timestamp(rwb, &rwb->last_issue); - return ret; + return; } if (current_is_kswapd()) - ret |= WBT_KSWAPD; + flags |= WBT_KSWAPD; if (bio_op(bio) == REQ_OP_DISCARD) - ret |= WBT_DISCARD; + flags |= WBT_DISCARD; - __wbt_wait(rwb, ret, bio->bi_opf, lock); + __wbt_wait(rwb, flags, bio->bi_opf, lock); if (!blk_stat_is_active(rwb->cb)) rwb_arm_timer(rwb); +} - return ret | WBT_TRACKED; +static void wbt_track(struct rq_qos *rqos, struct request *rq, struct bio *bio) +{ + struct rq_wb *rwb = RQWB(rqos); + rq->wbt_flags |= bio_to_wbt_flags(rwb, bio); } void wbt_issue(struct rq_qos *rqos, struct request *rq) @@ -707,9 +732,10 @@ EXPORT_SYMBOL_GPL(wbt_disable_default); static struct rq_qos_ops wbt_rqos_ops = { .throttle = wbt_wait, .issue = wbt_issue, + .track = wbt_track, .requeue = wbt_requeue, .done = wbt_done, - .cleanup = __wbt_done, + .cleanup = wbt_cleanup, .exit = wbt_exit, }; diff --git a/block/blk-wbt.h b/block/blk-wbt.h index 53b20a58c0a2..f47218d5b3b2 100644 --- a/block/blk-wbt.h +++ b/block/blk-wbt.h @@ -87,11 +87,6 @@ static inline unsigned int wbt_inflight(struct rq_wb *rwb) #ifdef CONFIG_BLK_WBT -static inline void wbt_track(struct request *rq, enum wbt_flags flags) -{ - rq->wbt_flags |= flags; -} - int wbt_init(struct request_queue *); void wbt_update_limits(struct request_queue *); void wbt_disable_default(struct request_queue *);