From patchwork Tue Aug 7 22:51:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 10559303 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 248A613AC for ; Tue, 7 Aug 2018 22:51:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 157BA2A768 for ; Tue, 7 Aug 2018 22:51:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1374C2A81F; Tue, 7 Aug 2018 22:51:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A7872A768 for ; Tue, 7 Aug 2018 22:51:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726628AbeHHBIM (ORCPT ); Tue, 7 Aug 2018 21:08:12 -0400 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:64040 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726625AbeHHBIM (ORCPT ); Tue, 7 Aug 2018 21:08:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1533682295; x=1565218295; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=H/bJjHp0+Q0xGRScT+gv0ecVP1w6Ja9KVXTXhlmPYbA=; b=cnszKTy+YsP6SvqS11HFvaeNyzLTp9KoZ1/Ss6m0HCECW7vFbw5gU8vR D/8Y1Zhgs+J6zQ8u7sRWdomzggfhmDeTSin+eUClRrZZcm8FuxC3O1kz7 y8bB7fw2scq4ikArWx//p9qWU4+YUljrh5vJRdHI4sCJ4rLN1UHV7Pr9L mU4rDjz+r/ArDghQbZq60se4CuAOjUN0+wmY1mxExaLl/g6lVDRqs8SiT Jwzup7jKlAwr4H06haWXMig+WDlEeyIqvKYX85DA7zmKnm0gYCNnCuGP3 TonPCe6MFaLYJahE7IxuKcMrrynqbCrXuJEkIGLGmJoMnP/aa28do4iPI A==; X-IronPort-AV: E=Sophos;i="5.51,456,1526313600"; d="scan'208";a="190872938" Received: from uls-op-cesaip02.wdc.com (HELO uls-op-cesaep02.wdc.com) ([199.255.45.15]) by ob1.hgst.iphmx.com with ESMTP; 08 Aug 2018 06:51:35 +0800 Received: from uls-op-cesaip01.wdc.com ([10.248.3.36]) by uls-op-cesaep02.wdc.com with ESMTP; 07 Aug 2018 15:39:14 -0700 Received: from thinkpad-bart.sdcorp.global.sandisk.com ([10.111.67.248]) by uls-op-cesaip01.wdc.com with ESMTP; 07 Aug 2018 15:51:36 -0700 From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , Bart Van Assche , Ming Lei , Jianchao Wang , Hannes Reinecke , Johannes Thumshirn , Alan Stern Subject: [PATCH v5 6/9] block: Change the runtime power management approach (2/2) Date: Tue, 7 Aug 2018 15:51:30 -0700 Message-Id: <20180807225133.27221-7-bart.vanassche@wdc.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180807225133.27221-1-bart.vanassche@wdc.com> References: <20180807225133.27221-1-bart.vanassche@wdc.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Instead of allowing requests that are not power management requests to enter the queue in runtime suspended status (RPM_SUSPENDED), make the blk_get_request() caller block. This change fixes a starvation issue: it is now guaranteed that power management requests will be executed no matter how many blk_get_request() callers are waiting. Instead of maintaining the q->nr_pending counter, rely on q->q_usage_counter. Call pm_runtime_mark_last_busy() every time a request finishes instead of only if the queue depth drops to zero. Signed-off-by: Bart Van Assche Cc: Christoph Hellwig Cc: Ming Lei Cc: Jianchao Wang Cc: Hannes Reinecke Cc: Johannes Thumshirn Cc: Alan Stern --- block/blk-core.c | 37 ++++++++----------------------------- block/blk-mq-debugfs.c | 1 - block/blk-pm.c | 40 +++++++++++++++++++++++++++++++++++----- block/blk-pm.h | 6 ++---- include/linux/blkdev.h | 1 - 5 files changed, 45 insertions(+), 40 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 179a13be0fca..a2ef253edfbd 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -2737,30 +2737,6 @@ void blk_account_io_done(struct request *req, u64 now) } } -#ifdef CONFIG_PM -/* - * Don't process normal requests when queue is suspended - * or in the process of suspending/resuming - */ -static bool blk_pm_allow_request(struct request *rq) -{ - switch (rq->q->rpm_status) { - case RPM_RESUMING: - case RPM_SUSPENDING: - return rq->rq_flags & RQF_PM; - case RPM_SUSPENDED: - return false; - default: - return true; - } -} -#else -static bool blk_pm_allow_request(struct request *rq) -{ - return true; -} -#endif - void blk_account_io_start(struct request *rq, bool new_io) { struct hd_struct *part; @@ -2806,11 +2782,14 @@ static struct request *elv_next_request(struct request_queue *q) while (1) { list_for_each_entry(rq, &q->queue_head, queuelist) { - if (blk_pm_allow_request(rq)) - return rq; - - if (rq->rq_flags & RQF_SOFTBARRIER) - break; +#ifdef CONFIG_PM + /* + * If a request gets queued in state RPM_SUSPENDED + * then that's a kernel bug. + */ + WARN_ON_ONCE(q->rpm_status == RPM_SUSPENDED); +#endif + return rq; } /* diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index a5ea86835fcb..7d74d53dc098 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -332,7 +332,6 @@ static const char *const rqf_name[] = { RQF_NAME(ELVPRIV), RQF_NAME(IO_STAT), RQF_NAME(ALLOCED), - RQF_NAME(PM), RQF_NAME(HASHED), RQF_NAME(STATS), RQF_NAME(SPECIAL_PAYLOAD), diff --git a/block/blk-pm.c b/block/blk-pm.c index bf8532da952d..d6b65cef9764 100644 --- a/block/blk-pm.c +++ b/block/blk-pm.c @@ -86,14 +86,39 @@ int blk_pre_runtime_suspend(struct request_queue *q) if (!q->dev) return ret; + WARN_ON_ONCE(q->rpm_status != RPM_ACTIVE); + + blk_set_pm_only(q); + /* + * This function only gets called if the most recent + * pm_request_resume() call occurred at least autosuspend_delay_ms + * ago. Since blk_queue_enter() is called by the request allocation + * code before pm_request_resume(), if q_usage_counter indicates that + * no requests are in flight it is safe to suspend the device. + */ + ret = -EBUSY; + if (!percpu_ref_is_in_use(&q->q_usage_counter)) { + /* + * Switch to preempt-only mode before calling + * synchronize_rcu() such that later blk_queue_enter() calls + * see the preempt-only state. See also + * http://lwn.net/Articles/573497/. + */ + synchronize_rcu(); + if (!percpu_ref_is_in_use(&q->q_usage_counter)) + ret = 0; + } + spin_lock_irq(q->queue_lock); - if (q->nr_pending) { - ret = -EBUSY; + if (ret < 0) pm_runtime_mark_last_busy(q->dev); - } else { + else q->rpm_status = RPM_SUSPENDING; - } spin_unlock_irq(q->queue_lock); + + if (ret) + blk_clear_pm_only(q); + return ret; } EXPORT_SYMBOL(blk_pre_runtime_suspend); @@ -124,6 +149,9 @@ void blk_post_runtime_suspend(struct request_queue *q, int err) pm_runtime_mark_last_busy(q->dev); } spin_unlock_irq(q->queue_lock); + + if (err) + blk_clear_pm_only(q); } EXPORT_SYMBOL(blk_post_runtime_suspend); @@ -171,13 +199,15 @@ void blk_post_runtime_resume(struct request_queue *q, int err) spin_lock_irq(q->queue_lock); if (!err) { q->rpm_status = RPM_ACTIVE; - __blk_run_queue(q); pm_runtime_mark_last_busy(q->dev); pm_request_autosuspend(q->dev); } else { q->rpm_status = RPM_SUSPENDED; } spin_unlock_irq(q->queue_lock); + + if (!err) + blk_clear_pm_only(q); } EXPORT_SYMBOL(blk_post_runtime_resume); diff --git a/block/blk-pm.h b/block/blk-pm.h index 1ffc8ef203ec..fcb507a29e99 100644 --- a/block/blk-pm.h +++ b/block/blk-pm.h @@ -8,21 +8,19 @@ #ifdef CONFIG_PM static inline void blk_pm_requeue_request(struct request *rq) { - if (rq->q->dev && !(rq->rq_flags & RQF_PM)) - rq->q->nr_pending--; } static inline void blk_pm_add_request(struct request_queue *q, struct request *rq) { - if (q->dev && !(rq->rq_flags & RQF_PM) && q->nr_pending++ == 0 && + if (q->dev && !(rq->rq_flags & RQF_PM) && (q->rpm_status == RPM_SUSPENDED || q->rpm_status == RPM_SUSPENDING)) pm_request_resume(q->dev); } static inline void blk_pm_put_request(struct request *rq) { - if (rq->q->dev && !(rq->rq_flags & RQF_PM) && !--rq->q->nr_pending) + if (rq->q->dev && !(rq->rq_flags & RQF_PM)) pm_runtime_mark_last_busy(rq->q->dev); } #else diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index cc5ef316eb39..b10787fdc69e 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -549,7 +549,6 @@ struct request_queue { #ifdef CONFIG_PM struct device *dev; int rpm_status; - unsigned int nr_pending; #endif /*