From patchwork Wed Oct 26 09:28:02 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 9396465 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8AA0360236 for ; Wed, 26 Oct 2016 09:31:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3AE42994F for ; Wed, 26 Oct 2016 09:31:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D78DA29964; Wed, 26 Oct 2016 09:31:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3E1D29963 for ; Wed, 26 Oct 2016 09:31:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756203AbcJZJ3s (ORCPT ); Wed, 26 Oct 2016 05:29:48 -0400 Received: from mail-wm0-f51.google.com ([74.125.82.51]:37990 "EHLO mail-wm0-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757278AbcJZJ3K (ORCPT ); Wed, 26 Oct 2016 05:29:10 -0400 Received: by mail-wm0-f51.google.com with SMTP id d128so74715239wmf.1 for ; Wed, 26 Oct 2016 02:28:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/F6+Yo42Uews/bvVe8fvwVavpMTERfy9u32cmi0MVRk=; b=j4L1EwG5NJYMESWHu0bLcEzZdOJWsOF5EeHrhFPkfBlZS1Qi198tfSbbuxk9mit2TZ b14ig0SlyFcwIiQlUKHzzrhtLuTjvkiwNF3z9g9pxyKPwuGFyfBL0e4QDVov4PM36n+j Bl2pskrgCWp0hS+bN5P2RPaBOoeQ4WHAj7V4s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/F6+Yo42Uews/bvVe8fvwVavpMTERfy9u32cmi0MVRk=; b=OwNqQEWXhbij8K+PZ/87rBY5HR1nyxV/z0H60LtfU1isfHnR/yFN7jOW+L/pJtg/nM ZqeEmL8RqhIZtLNUsQhAwf92O026DbOVjY3Z/GHhLBWH4NujLK/ldlSy8UH+Seku1fNh cD3IiBKol+1TK3v5ao1TdigiEOtplSz8XDcXQwQHeaq/W/8uM4ZxlRbKNYNWPnMjJggp lC+pf2mzdpLz3fL5gdPhajZ2sY0S50aqaY40l4Elv1MEDL2B7lGHL7JVCfYaL0dwZZlG MYICwn0PeQZ1cej43gq1+tRI3bzkQXq2x6e6+KPQEzKLMzoPX3bDLMLAVpdowIW7PqL4 kV7Q== X-Gm-Message-State: ABUngvcQV1+XwXGDEzYRMIE6xrcUlGPP/J2AvzoVkp8qX+rQ/O7uedZhHQKDy3txg+TW+jMC X-Received: by 10.28.35.82 with SMTP id j79mr1742725wmj.32.1477474106935; Wed, 26 Oct 2016 02:28:26 -0700 (PDT) Received: from paolo-VirtualBox.mat.unimo.it (nb-valente.mat.unimo.it. [155.185.5.181]) by smtp.gmail.com with ESMTPSA id q134sm8529945wme.3.2016.10.26.02.28.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Oct 2016 02:28:26 -0700 (PDT) From: Paolo Valente To: Jens Axboe , Tejun Heo Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, hare@suse.de, arnd@arndb.de, bart.vanassche@sandisk.com, grant.likely@secretlab.ca, jack@suse.cz, James.Bottomley@HansenPartnership.com, Paolo Valente , Arianna Avanzini Subject: [PATCH 09/14] block, bfq: reduce latency during request-pool saturation Date: Wed, 26 Oct 2016 11:28:02 +0200 Message-Id: <1477474082-2846-10-git-send-email-paolo.valente@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1477474082-2846-1-git-send-email-paolo.valente@linaro.org> References: <1477474082-2846-1-git-send-email-paolo.valente@linaro.org> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch introduces an heuristic that reduces latency when the I/O-request pool is saturated. This goal is achieved by disabling device idling, for non-weight-raised queues, when there are weight- raised queues with pending or in-flight requests. In fact, as explained in more detail in the comment on the function bfq_bfqq_may_idle(), this reduces the rate at which processes associated with non-weight-raised queues grab requests from the pool, thereby increasing the probability that processes associated with weight-raised queues get a request immediately (or at least soon) when they need one. Along the same line, if there are weight-raised queues, then this patch halves the service rate of async (write) requests for non-weight-raised queues. Signed-off-by: Paolo Valente Signed-off-by: Arianna Avanzini --- block/bfq-iosched.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 63 insertions(+), 3 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 3b11772..46d6df3 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -378,6 +378,8 @@ struct bfq_data { * queue in service, even if it is idling). */ int busy_queues; + /* number of weight-raised busy @bfq_queues */ + int wr_busy_queues; /* number of queued requests */ int queued; /* number of requests dispatched and waiting for completion */ @@ -2019,6 +2021,9 @@ static void bfq_del_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfqd->busy_queues--; + if (bfqq->wr_coeff > 1) + bfqd->wr_busy_queues--; + bfqg_stats_update_dequeue(bfqq_group(bfqq)); bfq_deactivate_bfqq(bfqd, bfqq, requeue); @@ -2035,6 +2040,9 @@ static void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq) bfq_mark_bfqq_busy(bfqq); bfqd->busy_queues++; + + if (bfqq->wr_coeff > 1) + bfqd->wr_busy_queues++; } #if defined(CONFIG_BFQ_GROUP_IOSCHED) && defined(CONFIG_DEBUG_BLK_CGROUP) @@ -3335,7 +3343,16 @@ static unsigned long bfq_serv_to_charge(struct request *rq, if (bfq_bfqq_sync(bfqq) || bfqq->wr_coeff > 1) return blk_rq_sectors(rq); - return blk_rq_sectors(rq) * bfq_async_charge_factor; + /* + * If there are no weight-raised queues, then amplify service + * by just the async charge factor; otherwise amplify service + * by twice the async charge factor, to further reduce latency + * for weight-raised queues. + */ + if (bfqq->bfqd->wr_busy_queues == 0) + return blk_rq_sectors(rq) * bfq_async_charge_factor; + + return blk_rq_sectors(rq) * 2 * bfq_async_charge_factor; } /** @@ -3791,6 +3808,7 @@ static void bfq_add_request(struct request *rq) bfqq->wr_coeff = bfqd->bfq_wr_coeff; bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); + bfqd->wr_busy_queues++; bfqq->entity.prio_changed = 1; } if (prev != bfqq->next_rq) @@ -4003,6 +4021,8 @@ static void bfq_merged_requests(struct request_queue *q, struct request *rq, /* Must be called with bfqq != NULL */ static void bfq_bfqq_end_wr(struct bfq_queue *bfqq) { + if (bfq_bfqq_busy(bfqq)) + bfqq->bfqd->wr_busy_queues--; bfqq->wr_coeff = 1; bfqq->wr_cur_max_time = 0; bfqq->last_wr_start_finish = jiffies; @@ -5049,7 +5069,8 @@ static bool bfq_may_expire_for_budg_timeout(struct bfq_queue *bfqq) static bool bfq_bfqq_may_idle(struct bfq_queue *bfqq) { struct bfq_data *bfqd = bfqq->bfqd; - bool idling_boosts_thr, asymmetric_scenario; + bool idling_boosts_thr, idling_boosts_thr_without_issues, + asymmetric_scenario; if (bfqd->strict_guarantees) return true; @@ -5072,6 +5093,44 @@ static bool bfq_bfqq_may_idle(struct bfq_queue *bfqq) idling_boosts_thr = !bfqd->hw_tag || bfq_bfqq_IO_bound(bfqq); /* + * The value of the next variable, + * idling_boosts_thr_without_issues, is equal to that of + * idling_boosts_thr, unless a special case holds. In this + * special case, described below, idling may cause problems to + * weight-raised queues. + * + * When the request pool is saturated (e.g., in the presence + * of write hogs), if the processes associated with + * non-weight-raised queues ask for requests at a lower rate, + * then processes associated with weight-raised queues have a + * higher probability to get a request from the pool + * immediately (or at least soon) when they need one. Thus + * they have a higher probability to actually get a fraction + * of the device throughput proportional to their high + * weight. This is especially true with NCQ-capable drives, + * which enqueue several requests in advance, and further + * reorder internally-queued requests. + * + * For this reason, we force to false the value of + * idling_boosts_thr_without_issues if there are weight-raised + * busy queues. In this case, and if bfqq is not weight-raised, + * this guarantees that the device is not idled for bfqq (if, + * instead, bfqq is weight-raised, then idling will be + * guaranteed by another variable, see below). Combined with + * the timestamping rules of BFQ (see [1] for details), this + * behavior causes bfqq, and hence any sync non-weight-raised + * queue, to get a lower number of requests served, and thus + * to ask for a lower number of requests from the request + * pool, before the busy weight-raised queues get served + * again. This often mitigates starvation problems in the + * presence of heavy write workloads and NCQ, thereby + * guaranteeing a higher application and system responsiveness + * in these hostile scenarios. + */ + idling_boosts_thr_without_issues = idling_boosts_thr && + bfqd->wr_busy_queues == 0; + + /* * There is then a case where idling must be performed not for * throughput concerns, but to preserve service guarantees. To * introduce it, we can note that allowing the drive to @@ -5145,7 +5204,7 @@ static bool bfq_bfqq_may_idle(struct bfq_queue *bfqq) * is necessary to preserve service guarantees. */ return bfq_bfqq_sync(bfqq) && - (idling_boosts_thr || asymmetric_scenario); + (idling_boosts_thr_without_issues || asymmetric_scenario); } /* @@ -6315,6 +6374,7 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) * high-definition compressed * video. */ + bfqd->wr_busy_queues = 0; /* * Begin by assuming, optimistically, that the device is a