From patchwork Thu Jul 15 13:30:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 12380029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20F0CC47E4B for ; Thu, 15 Jul 2021 13:30:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E9C7F613CC for ; Thu, 15 Jul 2021 13:30:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231713AbhGONdW (ORCPT ); Thu, 15 Jul 2021 09:33:22 -0400 Received: from smtp-out1.suse.de ([195.135.220.28]:53896 "EHLO smtp-out1.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237618AbhGONdV (ORCPT ); Thu, 15 Jul 2021 09:33:21 -0400 Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 6350122782; Thu, 15 Jul 2021 13:30:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1626355827; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3/tUVLi/1ehnmWuHgEgGk245sp/nozJ2MMa9O6HKxS0=; b=ul2dK4DKcpZLhcFUP2HRySeglNc4qyY3ksdiEPVRo+eTgE5IdzMa5osPKzrq9O20CdLgvd 95Viv4GBkgO+yLTk5roYJ6hzr5+1ol5OS54b2y8SQOd21Oj44eu8wWHwtbXtHgRCfg7ra/ bZdAw/kVwLoWJ8EdhbMfNRyV0KvAS5E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1626355827; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3/tUVLi/1ehnmWuHgEgGk245sp/nozJ2MMa9O6HKxS0=; b=uVsPdV7AgHUYF6MZq5iXOgM/UEtAQtXoNblD88l42qi1Tv8WjpxqMttQPcVGNiZgINNmWI Op57UoykyKTg9BDg== Received: from quack2.suse.cz (unknown [10.100.200.198]) by relay2.suse.de (Postfix) with ESMTP id 52157A3B99; Thu, 15 Jul 2021 13:30:27 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 31E581E02EE; Thu, 15 Jul 2021 15:30:27 +0200 (CEST) From: Jan Kara To: Jens Axboe Cc: , Paolo Valente , =?utf-8?q?Michal_Koutn=C3=BD?= , Jan Kara Subject: [PATCH 1/3] block: Provide icq in request allocation data Date: Thu, 15 Jul 2021 15:30:17 +0200 Message-Id: <20210715133027.23975-1-jack@suse.cz> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210715132047.20874-1-jack@suse.cz> References: <20210715132047.20874-1-jack@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3457; h=from:subject; bh=C9/rcXsJhybvwsy9P2BOMgpKKdd8LFIsH6x/qoBn5qE=; b=owEBbQGS/pANAwAIAZydqgc/ZEDZAcsmYgBg8DhpEv78bjRoD4TE+LDq/t91D9dk+MIUVgVsCKR3 N93t4vSJATMEAAEIAB0WIQSrWdEr1p4yirVVKBycnaoHP2RA2QUCYPA4aQAKCRCcnaoHP2RA2TxECA DcX7Qgl4D97JSv89Pnq7LeTewSeU+rbF2P2UNgAnUm9WJWrrKxrxtUb+qbO1OkiayCJXW7rxwf9/97 qpPFKpLLMAhdWfBJS7mei7/yfTP+BePMrWiZTqiyrjUCGghR8aov7PboVCgbVojJ1gENk2utD6b/ZZ cy8uBndICmvQ92IRNqE9h/ytMWTSVJ6QUowg1nXxg/0LOc29+3SVM1qTIWDuZYCoZkQddJXLb/Skzk rNV3xG2sxzr6GE5CPI1jLww5YC/qq87PhmgiwfPWS1RNOKAEH1eGn4Eh11KdZXlnwZKYo5L7YsZcAh Njcf4o5tg+RLW4LLC5srdvzNQbc6q2 X-Developer-Key: i=jack@suse.cz; a=openpgp; fpr=93C6099A142276A28BBE35D815BC833443038D8C Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Currently we lookup ICQ only after the request is allocated. However BFQ will want to decide how many scheduler tags it allows a given bfq queue (effectively a process) to consume based on cgroup weight. So lookup ICQ earlier and provide it in struct blk_mq_alloc_data so that BFQ can use it. Signed-off-by: Jan Kara --- block/blk-mq-sched.c | 18 ++++++++++-------- block/blk-mq-sched.h | 3 ++- block/blk-mq.c | 7 ++++--- block/blk-mq.h | 1 + 4 files changed, 17 insertions(+), 12 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index c838d81ac058..3e34f5bb24ae 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -18,9 +18,8 @@ #include "blk-mq-tag.h" #include "blk-wbt.h" -void blk_mq_sched_assign_ioc(struct request *rq) +struct io_cq *blk_mq_sched_lookup_icq(struct request_queue *q) { - struct request_queue *q = rq->q; struct io_context *ioc; struct io_cq *icq; @@ -29,17 +28,20 @@ void blk_mq_sched_assign_ioc(struct request *rq) */ ioc = current->io_context; if (!ioc) - return; + return NULL; spin_lock_irq(&q->queue_lock); icq = ioc_lookup_icq(ioc, q); spin_unlock_irq(&q->queue_lock); + if (icq) + return icq; + return ioc_create_icq(ioc, q, GFP_ATOMIC); +} - if (!icq) { - icq = ioc_create_icq(ioc, q, GFP_ATOMIC); - if (!icq) - return; - } +void blk_mq_sched_assign_ioc(struct request *rq, struct io_cq *icq) +{ + if (!icq) + return; get_io_context(icq->ioc); rq->elv.icq = icq; } diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h index 5246ae040704..4529991e55e6 100644 --- a/block/blk-mq-sched.h +++ b/block/blk-mq-sched.h @@ -7,7 +7,8 @@ #define MAX_SCHED_RQ (16 * BLKDEV_MAX_RQ) -void blk_mq_sched_assign_ioc(struct request *rq); +struct io_cq *blk_mq_sched_lookup_icq(struct request_queue *q); +void blk_mq_sched_assign_ioc(struct request *rq, struct io_cq *icq); bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs, struct request **merged_request); diff --git a/block/blk-mq.c b/block/blk-mq.c index 2c4ac51e54eb..b9d83644158f 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -333,9 +333,7 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data, rq->elv.icq = NULL; if (e && e->type->ops.prepare_request) { - if (e->type->icq_cache) - blk_mq_sched_assign_ioc(rq); - + blk_mq_sched_assign_ioc(rq, data->icq); e->type->ops.prepare_request(rq); rq->rq_flags |= RQF_ELVPRIV; } @@ -360,6 +358,9 @@ static struct request *__blk_mq_alloc_request(struct blk_mq_alloc_data *data) data->flags |= BLK_MQ_REQ_NOWAIT; if (e) { + if (!op_is_flush(data->cmd_flags) && e->type->icq_cache && + e->type->ops.prepare_request) + data->icq = blk_mq_sched_lookup_icq(q); /* * Flush/passthrough requests are special and go directly to the * dispatch list. Don't include reserved tags in the diff --git a/block/blk-mq.h b/block/blk-mq.h index d08779f77a26..c502232384c6 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -151,6 +151,7 @@ static inline struct blk_mq_ctx *blk_mq_get_ctx(struct request_queue *q) struct blk_mq_alloc_data { /* input parameter */ struct request_queue *q; + struct io_cq *icq; blk_mq_req_flags_t flags; unsigned int shallow_depth; unsigned int cmd_flags; From patchwork Thu Jul 15 13:30:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 12380027 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD5F8C1B08C for ; Thu, 15 Jul 2021 13:30:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 94B67608FE for ; Thu, 15 Jul 2021 13:30:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237525AbhGONdV (ORCPT ); Thu, 15 Jul 2021 09:33:21 -0400 Received: from smtp-out1.suse.de ([195.135.220.28]:53872 "EHLO smtp-out1.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237360AbhGONdV (ORCPT ); Thu, 15 Jul 2021 09:33:21 -0400 Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 635AA2278A; Thu, 15 Jul 2021 13:30:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1626355827; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TtT6HXecD3OMF37cBv9XBf+I51CDX6Q833fzzz8oFNc=; b=QebGYxTz432yuNkRwkqd0D2PGidZnGRhULotIohvElTpd6ke1bo6N0MlAfzIL7QT71nwQV +g9gvpPv+bmzOVSd800vwHuludWF4ii1O3M1w0HuXEMqrzPpfRn1k4sEHcAW2FMf8MhtpU 8nF0cMk/CqL+OK2srP+MhMrwRws3VNI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1626355827; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TtT6HXecD3OMF37cBv9XBf+I51CDX6Q833fzzz8oFNc=; b=7gBOm4E1GP13jS9dv5lAcAyvhOnwwQ7+URK3KmRWyPKDHwc+tU1qSn9thTzPl/tJJ5GWyE 3kjbl6pOLKmD4zAQ== Received: from quack2.suse.cz (unknown [10.100.200.198]) by relay2.suse.de (Postfix) with ESMTP id 52DB0A3B9A; Thu, 15 Jul 2021 13:30:27 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 359C71E0BF6; Thu, 15 Jul 2021 15:30:27 +0200 (CEST) From: Jan Kara To: Jens Axboe Cc: , Paolo Valente , =?utf-8?q?Michal_Koutn=C3=BD?= , Jan Kara Subject: [PATCH 2/3] bfq: Track number of allocated requests in bfq_entity Date: Thu, 15 Jul 2021 15:30:18 +0200 Message-Id: <20210715133027.23975-2-jack@suse.cz> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210715132047.20874-1-jack@suse.cz> References: <20210715132047.20874-1-jack@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3252; h=from:subject; bh=rlIMrQse0DymWFIikMEs/b6gQ7juS+LiMlbECXmQMQ8=; b=owEBbQGS/pANAwAIAZydqgc/ZEDZAcsmYgBg8DhqVsHyFnKZ3R1UWjiH1GHmy1+zJqsnN/ousr94 UWUqP1aJATMEAAEIAB0WIQSrWdEr1p4yirVVKBycnaoHP2RA2QUCYPA4agAKCRCcnaoHP2RA2VtkCA CyFG0nj5ukrYcTYBZoOYfjF1kMxGDeUa/0+Xv2A1NLNDrxC1sVvyciEdtQx4Z7jFjWJBlXaciEcmV2 E6Gb6H/yk6e54bJ6wnJwr/XXQMFIoPL+q8t77xY2hLjgxDAs51rG3mkDqb8OOUNcyURuOhWB/kX/eX e+8VN9ruQBNHNqamNMhyDyrLxZZAWCCNncLzu7hgscY3oL0xLmFOiQzv0qIdLxGde3p8ixosrILMOh vzM4ZnJ698KUzKcQ/g6wEBdC4YZZwMzPhmh/m98L+Xe3yNhqSX+GSuvMV19Y6T1uK+cyQFvpl4tG+I PZcHccK5uAivlLpW71abVZorXf7PYv X-Developer-Key: i=jack@suse.cz; a=openpgp; fpr=93C6099A142276A28BBE35D815BC833443038D8C Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org When we want to limit number of requests used by each bfqq and also cgroup, we need to track also number of requests used by each cgroup. So track number of allocated requests for each bfq_entity. Signed-off-by: Jan Kara --- block/bfq-iosched.c | 28 ++++++++++++++++++++++------ block/bfq-iosched.h | 5 +++-- 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 727955918563..9ef057dc0028 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1113,7 +1113,8 @@ bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, static int bfqq_process_refs(struct bfq_queue *bfqq) { - return bfqq->ref - bfqq->allocated - bfqq->entity.on_st_or_in_serv - + return bfqq->ref - bfqq->entity.allocated - + bfqq->entity.on_st_or_in_serv - (bfqq->weight_counter != NULL) - bfqq->stable_ref; } @@ -5875,6 +5876,22 @@ static void bfq_rq_enqueued(struct bfq_data *bfqd, struct bfq_queue *bfqq, } } +static void bfqq_request_allocated(struct bfq_queue *bfqq) +{ + struct bfq_entity *entity = &bfqq->entity; + + for_each_entity(entity) + entity->allocated++; +} + +static void bfqq_request_freed(struct bfq_queue *bfqq) +{ + struct bfq_entity *entity = &bfqq->entity; + + for_each_entity(entity) + entity->allocated--; +} + /* returns true if it causes the idle timer to be disabled */ static bool __bfq_insert_request(struct bfq_data *bfqd, struct request *rq) { @@ -5888,8 +5905,8 @@ static bool __bfq_insert_request(struct bfq_data *bfqd, struct request *rq) * Release the request's reference to the old bfqq * and make sure one is taken to the shared queue. */ - new_bfqq->allocated++; - bfqq->allocated--; + bfqq_request_allocated(new_bfqq); + bfqq_request_freed(bfqq); new_bfqq->ref++; /* * If the bic associated with the process @@ -6248,8 +6265,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) static void bfq_finish_requeue_request_body(struct bfq_queue *bfqq) { - bfqq->allocated--; - + bfqq_request_freed(bfqq); bfq_put_queue(bfqq); } @@ -6669,7 +6685,7 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) } } - bfqq->allocated++; + bfqq_request_allocated(bfqq); bfqq->ref++; bfq_log_bfqq(bfqd, bfqq, "get_request %p: bfqq %p, %d", rq, bfqq, bfqq->ref); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 99c2a3cb081e..70d4a9b54613 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -170,6 +170,9 @@ struct bfq_entity { /* budget, used also to calculate F_i: F_i = S_i + @budget / @weight */ int budget; + /* Number of requests allocated in the subtree of this entity */ + int allocated; + /* device weight, if non-zero, it overrides the default weight of * bfq_group_data */ int dev_weight; @@ -266,8 +269,6 @@ struct bfq_queue { struct request *next_rq; /* number of sync and async requests queued */ int queued[2]; - /* number of requests currently allocated */ - int allocated; /* number of pending metadata requests */ int meta_pending; /* fifo list of requests in sort_list */ From patchwork Thu Jul 15 13:30:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 12380025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67ACBC07E96 for ; Thu, 15 Jul 2021 13:30:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4156B61360 for ; Thu, 15 Jul 2021 13:30:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237619AbhGONdV (ORCPT ); Thu, 15 Jul 2021 09:33:21 -0400 Received: from smtp-out2.suse.de ([195.135.220.29]:57256 "EHLO smtp-out2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237525AbhGONdV (ORCPT ); Thu, 15 Jul 2021 09:33:21 -0400 Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 66AE71FE27; Thu, 15 Jul 2021 13:30:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1626355827; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hwo33Naj/CPFidpZeosCdaEDnv3XgY9VCBciCRXHwWw=; b=FCJrhGxVLppxZASb7CGalncGBtH0S3Jb1RvaYdc/t+c8U1mPJIAhruAFzBm+v5Kmqg5Sai i1Gm65iRv0RtKHkL6haH9neK/fuk6C84w45vWt+ofLgl/Qwkd2JLBLrfCttPfGgG921lKG jg/O3GQ2aYKitr/QaL69kADVDhuPixM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1626355827; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hwo33Naj/CPFidpZeosCdaEDnv3XgY9VCBciCRXHwWw=; b=eWQCKqD94NZUmtFrPl3D+MmBleg/RT5GAS5xVU+cW5u1mB0Ru1hpSBeyV4V7EZxhsBNUKY OYU/jTi42HyAoCCg== Received: from quack2.suse.cz (unknown [10.100.200.198]) by relay2.suse.de (Postfix) with ESMTP id 58E2EA3B9B; Thu, 15 Jul 2021 13:30:27 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 39ECF1E0BF8; Thu, 15 Jul 2021 15:30:27 +0200 (CEST) From: Jan Kara To: Jens Axboe Cc: , Paolo Valente , =?utf-8?q?Michal_Koutn=C3=BD?= , Jan Kara Subject: [PATCH 3/3] bfq: Limit number of requests consumed by each cgroup Date: Thu, 15 Jul 2021 15:30:19 +0200 Message-Id: <20210715133027.23975-3-jack@suse.cz> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210715132047.20874-1-jack@suse.cz> References: <20210715132047.20874-1-jack@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6369; h=from:subject; bh=Bp/OSDW5Rjr0UeO2reqvBpkYjy5sYrS7zC7p9jDV+Nc=; b=owEBbQGS/pANAwAIAZydqgc/ZEDZAcsmYgBg8Dhrq6VWQjZY4UnX53DHpzTqIe8h6p2MzOlj4Zap TTN89HKJATMEAAEIAB0WIQSrWdEr1p4yirVVKBycnaoHP2RA2QUCYPA4awAKCRCcnaoHP2RA2akCB/ 9mUxSBzvtUGjlpomqcV67PaLox+msu0iERsheXwT7Vsr/lukmy2JEO4q7L+Uijh4n78YET1UsJigdl Cm10Sn+EcrUDMzFV1hvDQuhWLe8fgF9CzX6vKSt1lgYZvsFU620OY1kU5yNxjOeysg+dARnGgDH19f 2NsG1G/Az7hqLDRs56JPwGbiENxZfdoUTHG1IXWl8mwrDkQM4B6nTlNAO1AUQ2RPl11m+F+l+KrsT4 JBKfH05G+iSXQgmMpb9sm52aWqiNjFnLf34I0biH65gu02Wae+q6jNUgQiCSiyRVTOD+kB7aHm8vBW 4MLmxwXD5kIHlfV7FEwuAgIqffvr+i X-Developer-Key: i=jack@suse.cz; a=openpgp; fpr=93C6099A142276A28BBE35D815BC833443038D8C Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org When cgroup IO scheduling is used with BFQ it does not really provide service differentiation if the cgroup drives a big IO depth. That for example happens with writeback which asynchronously submits lots of IO but it can happen with AIO as well. The problem is that if we have two cgroups that submit IO with different weights, the cgroup with higher weight properly gets more IO time and is able to dispatch more IO. However this causes lower weight cgroup to accumulate more requests inside BFQ and eventually lower weight cgroup consumes most of IO scheduler tags. At that point higher weight cgroup stops getting better service as it is mostly blocked waiting for a scheduler tag while its queues inside BFQ are empty and thus lower weight cgroup gets served. Check how many requests submitting cgroup has allocated in bfq_limit_depth() and if it consumes more requests than what would correspond to its weight limit available depth to 1 so that the cgroup cannot consume many more requests. With this limitation the higher weight cgroup gets proper service even with writeback. Signed-off-by: Jan Kara --- block/bfq-iosched.c | 103 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 85 insertions(+), 18 deletions(-) diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 9ef057dc0028..8f9b4904934b 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -565,6 +565,71 @@ static struct request *bfq_choose_req(struct bfq_data *bfqd, } } +#define BFQ_LIMIT_INLINE_DEPTH 16 + +#ifdef CONFIG_BFQ_GROUP_IOSCHED +static bool bfqq_request_over_limit(struct bfq_queue *bfqq, int limit) +{ + struct bfq_data *bfqd = bfqq->bfqd; + struct bfq_entity *entity = &bfqq->entity; + struct bfq_entity *inline_entities[BFQ_LIMIT_INLINE_DEPTH]; + struct bfq_entity **entities = inline_entities; + int depth, level; + bool ret = false; + + if (!entity->on_st_or_in_serv) + return false; + + /* +1 for bfqq entity, root cgroup not included */ + depth = bfqg_to_blkg(bfqq_group(bfqq))->blkcg->css.cgroup->level + 1; + if (depth > BFQ_LIMIT_INLINE_DEPTH) { + entities = kmalloc_array(depth, sizeof(*entities), GFP_NOIO); + if (!entities) + return false; + } + + spin_lock_irq(&bfqd->lock); + if (!entity->on_st_or_in_serv) + goto out; + /* Gather our ancestors as we need to traverse them in reverse order */ + level = 0; + for_each_entity(entity) { + /* Uh, more parents than cgroup subsystem thinks? */ + if (WARN_ON_ONCE(level >= depth)) + break; + entities[level++] = entity; + } + WARN_ON_ONCE(level != depth); + for (level--; level >= 0; level--) { + entity = entities[level]; + /* + * If the leaf entity has work to do, parents should be tracked + * as well. + */ + WARN_ON_ONCE(!entity->on_st_or_in_serv); + limit = DIV_ROUND_CLOSEST(limit * entity->weight, + bfq_entity_service_tree(entity)->wsum); + if (entity->allocated >= limit) { + bfq_log_bfqq(bfqq->bfqd, bfqq, + "too many requests: allocated %d limit %d level %d", + entity->allocated, limit, level); + ret = true; + break; + } + } +out: + spin_unlock_irq(&bfqd->lock); + if (entities != inline_entities) + kfree(entities); + return ret; +} +#else +static bool bfqq_request_over_limit(struct bfq_queue *bfqq, int limit) +{ + return false; +} +#endif + /* * Async I/O can easily starve sync I/O (both sync reads and sync * writes), by consuming all tags. Similarly, storms of sync writes, @@ -575,16 +640,28 @@ static struct request *bfq_choose_req(struct bfq_data *bfqd, static void bfq_limit_depth(unsigned int op, struct blk_mq_alloc_data *data) { struct bfq_data *bfqd = data->q->elevator->elevator_data; + struct bfq_io_cq *bic = data->icq ? icq_to_bic(data->icq) : NULL; + struct bfq_queue *bfqq = bic ? bic_to_bfqq(bic, op_is_sync(op)) : NULL; + int depth; + /* Sync reads have full depth available */ if (op_is_sync(op) && !op_is_write(op)) - return; + depth = 0; + else + depth = bfqd->word_depths[!!bfqd->wr_busy_queues][op_is_sync(op)]; - data->shallow_depth = - bfqd->word_depths[!!bfqd->wr_busy_queues][op_is_sync(op)]; + /* + * Does queue (or any parent entity) exceed number of requests that + * should be available to it? Heavily limit depth so that it cannot + * consume more available requests and thus starve other entities. + */ + if (bfqq && bfqq_request_over_limit(bfqq, data->q->nr_requests)) + depth = 1; bfq_log(bfqd, "[%s] wr_busy %d sync %d depth %u", - __func__, bfqd->wr_busy_queues, op_is_sync(op), - data->shallow_depth); + __func__, bfqd->wr_busy_queues, op_is_sync(op), depth); + if (depth) + data->shallow_depth = depth; } static struct bfq_queue * @@ -6848,11 +6925,8 @@ void bfq_put_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg) * See the comments on bfq_limit_depth for the purpose of * the depths set in the function. Return minimum shallow depth we'll use. */ -static unsigned int bfq_update_depths(struct bfq_data *bfqd, - struct sbitmap_queue *bt) +static void bfq_update_depths(struct bfq_data *bfqd, struct sbitmap_queue *bt) { - unsigned int i, j, min_shallow = UINT_MAX; - /* * In-word depths if no bfq_queue is being weight-raised: * leaving 25% of tags only for sync reads. @@ -6883,22 +6957,15 @@ static unsigned int bfq_update_depths(struct bfq_data *bfqd, bfqd->word_depths[1][0] = max(((1U << bt->sb.shift) * 3) >> 4, 1U); /* no more than ~37% of tags for sync writes (~20% extra tags) */ bfqd->word_depths[1][1] = max(((1U << bt->sb.shift) * 6) >> 4, 1U); - - for (i = 0; i < 2; i++) - for (j = 0; j < 2; j++) - min_shallow = min(min_shallow, bfqd->word_depths[i][j]); - - return min_shallow; } static void bfq_depth_updated(struct blk_mq_hw_ctx *hctx) { struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; struct blk_mq_tags *tags = hctx->sched_tags; - unsigned int min_shallow; - min_shallow = bfq_update_depths(bfqd, tags->bitmap_tags); - sbitmap_queue_min_shallow_depth(tags->bitmap_tags, min_shallow); + bfq_update_depths(bfqd, tags->bitmap_tags); + sbitmap_queue_min_shallow_depth(tags->bitmap_tags, 1); } static int bfq_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int index)