From patchwork Fri Nov 2 07:08:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10665103 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 73E5E17DF for ; Fri, 2 Nov 2018 07:10:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D2832B704 for ; Fri, 2 Nov 2018 07:10:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 41C712B75B; Fri, 2 Nov 2018 07:10:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD6862B704 for ; Fri, 2 Nov 2018 07:10:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728258AbeKBQQ2 (ORCPT ); Fri, 2 Nov 2018 12:16:28 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:46322 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728289AbeKBQQ2 (ORCPT ); Fri, 2 Nov 2018 12:16:28 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wA278p4E052023; Fri, 2 Nov 2018 07:10:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=T+bmX7Zqj753umvIWRaK2lmXBxtIoXAE0V0dNmNRtGI=; b=qWu450Eruwv8BZo/T8nmY4yZjJQodLbGReLNUtygBiTqHyvkxF2KM0q9QTbJqc+tlyTN ir8Q6xKfjoF9/Jb/+CJEa+lEy518G4GAgoW/gY2HOZ4Di93SDXluiV3k6Z9P0xO7WDeA EHeLitW4M7JlRoJwM6rDLmMkzb4ptH73GRmHq8DvoJG6ERoCRgVHdcX9w7LAXNXfVvZp bJfi3NT8T8gnDEA3Z+aLWljiH2Olhi9CJWRu2GgAjpZqZFXSKRfqcbOqyHVSF67DUL+9 uHzvtZPMOa7TcarqM8/6JtFZMWETSIHsFZ1JFAxzYMo4KqZDp68m82iakMZ25CeqQXlV 7Q== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2nducmhgbk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 02 Nov 2018 07:10:14 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wA27AECf019620 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Nov 2018 07:10:14 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wA27ADBs005689; Fri, 2 Nov 2018 07:10:13 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Nov 2018 00:10:13 -0700 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V4 1/5] blk-mq: make __blk_mq_issue_directly be able to accept NULL cookie pointer Date: Fri, 2 Nov 2018 15:08:52 +0800 Message-Id: <1541142536-1431-2-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> References: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9064 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=943 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811020067 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Make __blk_mq_issue_directly be able to accept a NULL cookie pointer and remove the dummy unused_cookie in blk_mq_request_issue_directly. Signed-off-by: Jianchao Wang --- block/blk-mq.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index dcf10e3..af5b591 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1700,8 +1700,6 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx, blk_qc_t new_cookie; blk_status_t ret; - new_cookie = request_to_qc_t(hctx, rq); - /* * For OK queue, we are done. For error, caller may kill it. * Any other error (busy), just add it to our list as we @@ -1711,19 +1709,23 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx, switch (ret) { case BLK_STS_OK: blk_mq_update_dispatch_busy(hctx, false); - *cookie = new_cookie; + new_cookie = request_to_qc_t(hctx, rq); break; case BLK_STS_RESOURCE: case BLK_STS_DEV_RESOURCE: blk_mq_update_dispatch_busy(hctx, true); __blk_mq_requeue_request(rq); + new_cookie = BLK_QC_T_NONE; break; default: blk_mq_update_dispatch_busy(hctx, false); - *cookie = BLK_QC_T_NONE; + new_cookie = BLK_QC_T_NONE; break; } + if (cookie) + *cookie = new_cookie; + return ret; } @@ -1791,12 +1793,11 @@ blk_status_t blk_mq_request_issue_directly(struct request *rq) { blk_status_t ret; int srcu_idx; - blk_qc_t unused_cookie; struct blk_mq_ctx *ctx = rq->mq_ctx; struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(rq->q, ctx->cpu); hctx_lock(hctx, &srcu_idx); - ret = __blk_mq_try_issue_directly(hctx, rq, &unused_cookie, true); + ret = __blk_mq_try_issue_directly(hctx, rq, NULL, true); hctx_unlock(hctx, srcu_idx); return ret; From patchwork Fri Nov 2 07:08:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10665101 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2BC017DF for ; Fri, 2 Nov 2018 07:10:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CBDD62B704 for ; Fri, 2 Nov 2018 07:10:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C029F2B75B; Fri, 2 Nov 2018 07:10:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A9FAD2B704 for ; Fri, 2 Nov 2018 07:10:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728492AbeKBQQb (ORCPT ); Fri, 2 Nov 2018 12:16:31 -0400 Received: from aserp2120.oracle.com ([141.146.126.78]:49894 "EHLO aserp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728223AbeKBQQa (ORCPT ); Fri, 2 Nov 2018 12:16:30 -0400 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wA279JpJ054289; Fri, 2 Nov 2018 07:10:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=vP9NZTogfxTR9G3rvMIz+J5V3m/qpmKyl4aDVJu3Rhg=; b=TjIm469nK6xaJBacIOHBd2Op4nI2dNcGiZtsln/30mE5UNW1hxLT4v5MrEVCo84VvrUZ 4NOrbNHCeR/QAcQOAmc5xwvvEHU0+1DC0QZCI6WmRBtmt7qUXshDbzJHmjmJyh/deVa4 3LV7/NunkfKlBNTVY49mFOdA2FxShpOxK7FD93kPi/hmFBGklMjwD2cwgRxloWVyXZmf tMczZUnHshAVH34bpVFX5sPM8paSoxJ9Z0rgHDKC8PY1eZJr0W7CyGfOtkAtyJfeXbfe lCJyzz/T556TOsKx+VTXIwWyn7Ptlyy89f8S4sy9ZMLjJ3nJOg21QbXCoklRMJcTJCwY TQ== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2120.oracle.com with ESMTP id 2ncfyqctvr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 02 Nov 2018 07:10:17 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wA27AGtm019751 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Nov 2018 07:10:16 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wA27AFN9003253; Fri, 2 Nov 2018 07:10:16 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Nov 2018 00:10:15 -0700 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V4 2/5] blk-mq: refactor the code of issue request directly Date: Fri, 2 Nov 2018 15:08:53 +0800 Message-Id: <1541142536-1431-3-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> References: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9064 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811020067 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Merge blk_mq_try_issue_directly and __blk_mq_try_issue_directly into one interface which is able to handle the return value from .queue_rq callback. To make the code clearer, introduce new helpers enum mq_issue_decision and blk_mq_make_decision to decide how to handle the non-issued requests. Signed-off-by: Jianchao Wang --- block/blk-mq.c | 104 +++++++++++++++++++++++++++++++++------------------------ 1 file changed, 61 insertions(+), 43 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index af5b591..962fdfc 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1729,78 +1729,96 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx, return ret; } -static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, +enum mq_issue_decision { + MQ_ISSUE_INSERT_QUEUE, + MQ_ISSUE_END_REQUEST, + MQ_ISSUE_DO_NOTHING, +}; + +static inline enum mq_issue_decision + blk_mq_make_dicision(blk_status_t ret, bool bypass) +{ + enum mq_issue_decision dec; + + switch(ret) { + case BLK_STS_OK: + dec = MQ_ISSUE_DO_NOTHING; + break; + case BLK_STS_DEV_RESOURCE: + case BLK_STS_RESOURCE: + dec = bypass ? MQ_ISSUE_DO_NOTHING : MQ_ISSUE_INSERT_QUEUE; + break; + default: + dec = bypass ? MQ_ISSUE_DO_NOTHING : MQ_ISSUE_END_REQUEST; + break; + } + + return dec; +} + +static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, struct request *rq, blk_qc_t *cookie, - bool bypass_insert) + bool bypass) { struct request_queue *q = rq->q; bool run_queue = true; + blk_status_t ret = BLK_STS_RESOURCE; + enum mq_issue_decision dec; + int srcu_idx; + + hctx_lock(hctx, &srcu_idx); /* - * RCU or SRCU read lock is needed before checking quiesced flag. + * hctx_lock is needed before checking quiesced flag. * - * When queue is stopped or quiesced, ignore 'bypass_insert' from - * blk_mq_request_issue_directly(), and return BLK_STS_OK to caller, - * and avoid driver to try to dispatch again. + * When queue is stopped or quiesced, ignore 'bypass', insert + * and return BLK_STS_OK to caller, and avoid driver to try to + * dispatch again. */ if (blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)) { run_queue = false; - bypass_insert = false; - goto insert; + bypass = false; + goto out_unlock; } - if (q->elevator && !bypass_insert) - goto insert; + if (q->elevator && !bypass) + goto out_unlock; if (!blk_mq_get_dispatch_budget(hctx)) - goto insert; + goto out_unlock; if (!blk_mq_get_driver_tag(rq)) { blk_mq_put_dispatch_budget(hctx); - goto insert; + goto out_unlock; } - return __blk_mq_issue_directly(hctx, rq, cookie); -insert: - if (bypass_insert) - return BLK_STS_RESOURCE; + ret = __blk_mq_issue_directly(hctx, rq, cookie); - blk_mq_sched_insert_request(rq, false, run_queue, false); - return BLK_STS_OK; -} - -static void blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, - struct request *rq, blk_qc_t *cookie) -{ - blk_status_t ret; - int srcu_idx; - - might_sleep_if(hctx->flags & BLK_MQ_F_BLOCKING); - - hctx_lock(hctx, &srcu_idx); +out_unlock: + hctx_unlock(hctx, srcu_idx); - ret = __blk_mq_try_issue_directly(hctx, rq, cookie, false); - if (ret == BLK_STS_RESOURCE || ret == BLK_STS_DEV_RESOURCE) - blk_mq_sched_insert_request(rq, false, true, false); - else if (ret != BLK_STS_OK) + dec = blk_mq_make_dicision(ret, bypass); + switch(dec) { + case MQ_ISSUE_INSERT_QUEUE: + blk_mq_sched_insert_request(rq, false, run_queue, false); + break; + case MQ_ISSUE_END_REQUEST: blk_mq_end_request(rq, ret); + break; + default: + return ret; + } - hctx_unlock(hctx, srcu_idx); + return BLK_STS_OK; } blk_status_t blk_mq_request_issue_directly(struct request *rq) { - blk_status_t ret; - int srcu_idx; struct blk_mq_ctx *ctx = rq->mq_ctx; struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(rq->q, ctx->cpu); - hctx_lock(hctx, &srcu_idx); - ret = __blk_mq_try_issue_directly(hctx, rq, NULL, true); - hctx_unlock(hctx, srcu_idx); - - return ret; + return blk_mq_try_issue_directly(hctx, rq, NULL, true); } void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, @@ -1922,13 +1940,13 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) data.hctx = blk_mq_map_queue(q, same_queue_rq->mq_ctx->cpu); blk_mq_try_issue_directly(data.hctx, same_queue_rq, - &cookie); + &cookie, false); } } else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator && !data.hctx->dispatch_busy)) { blk_mq_put_ctx(data.ctx); blk_mq_bio_to_request(rq, bio); - blk_mq_try_issue_directly(data.hctx, rq, &cookie); + blk_mq_try_issue_directly(data.hctx, rq, &cookie, false); } else { blk_mq_put_ctx(data.ctx); blk_mq_bio_to_request(rq, bio); From patchwork Fri Nov 2 07:08:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10665095 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E31B17DF for ; Fri, 2 Nov 2018 07:10:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 25C472B72C for ; Fri, 2 Nov 2018 07:10:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 19B472B75B; Fri, 2 Nov 2018 07:10:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE4662B6F7 for ; Fri, 2 Nov 2018 07:10:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728219AbeKBQQc (ORCPT ); Fri, 2 Nov 2018 12:16:32 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:46382 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727630AbeKBQQc (ORCPT ); Fri, 2 Nov 2018 12:16:32 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wA278qwF052039; Fri, 2 Nov 2018 07:10:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=TcDT+stwGD3sRN+O6m4lBwRAKKPXFhnXDlpIbDb0E7A=; b=Pkrg9UdQpBEQDZIVjMo9a6dndXq8JoF6FGhE9V5KPC9imF9HrWy1qAtMBUrww3Fwf8xC g/ZKgIJ4F2uxFKu7mnoLEVx/mFs1J8/pIel4ooVxpe8SgdAGVDY5Ndnsh8fRcEwIwXwf PfKdDFgFoZhQq18xtZx8LhuGgOvU5DDXCmDd8Lv1RKxC7sdZGXUj5Dm9Oz3KNY7CHbjo WsBCpLNKNHrtePzss6kbvs2j3FpPwKAoblJkOF0Hd6BsMsopjUopxZ+wNl/+0Us8Zn27 oT8V3gG7ME3XRND4HUEiytATDSxbF2CtTva6GS5GXywfxZnMDKUkakqoWkbkfGCGAJu3 ig== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2nducmhgbq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 02 Nov 2018 07:10:19 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wA27AHH3003385 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Nov 2018 07:10:18 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wA27AH7n004851; Fri, 2 Nov 2018 07:10:17 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Nov 2018 00:10:17 -0700 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V4 3/5] blk-mq: fix issue directly case when q is stopped or quiesced Date: Fri, 2 Nov 2018 15:08:54 +0800 Message-Id: <1541142536-1431-4-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> References: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9064 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811020067 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When try to issue request directly, if the queue is stopped or quiesced, 'bypass' will be ignored and return BLK_STS_OK to caller to avoid it dispatch request again. Then the request will be inserted with blk_mq_sched_insert_request. This is not correct for dm-rq case where we should avoid to pass through the underlying path's io scheduler. To fix it, add new mq_issue_decision entry MQ_ISSUE_INSERT_DISPATCH for above case where the request need to be inserted forcibly. And use blk_mq_request_bypass_insert to insert the request into hctx->dispatch directly. Signed-off-by: Jianchao Wang --- block/blk-mq.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 962fdfc..a0b9b6c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1731,12 +1731,13 @@ static blk_status_t __blk_mq_issue_directly(struct blk_mq_hw_ctx *hctx, enum mq_issue_decision { MQ_ISSUE_INSERT_QUEUE, + MQ_ISSUE_INSERT_DISPATCH, MQ_ISSUE_END_REQUEST, MQ_ISSUE_DO_NOTHING, }; static inline enum mq_issue_decision - blk_mq_make_dicision(blk_status_t ret, bool bypass) + blk_mq_make_dicision(blk_status_t ret, bool bypass, bool force) { enum mq_issue_decision dec; @@ -1746,7 +1747,10 @@ static inline enum mq_issue_decision break; case BLK_STS_DEV_RESOURCE: case BLK_STS_RESOURCE: - dec = bypass ? MQ_ISSUE_DO_NOTHING : MQ_ISSUE_INSERT_QUEUE; + if (force) + dec = bypass ? MQ_ISSUE_INSERT_DISPATCH : MQ_ISSUE_INSERT_QUEUE; + else + dec = bypass ? MQ_ISSUE_DO_NOTHING : MQ_ISSUE_INSERT_QUEUE; break; default: dec = bypass ? MQ_ISSUE_DO_NOTHING : MQ_ISSUE_END_REQUEST; @@ -1762,7 +1766,7 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, bool bypass) { struct request_queue *q = rq->q; - bool run_queue = true; + bool run_queue = true, force = false; blk_status_t ret = BLK_STS_RESOURCE; enum mq_issue_decision dec; int srcu_idx; @@ -1778,7 +1782,7 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, */ if (blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)) { run_queue = false; - bypass = false; + force = true; goto out_unlock; } @@ -1798,11 +1802,14 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, out_unlock: hctx_unlock(hctx, srcu_idx); - dec = blk_mq_make_dicision(ret, bypass); + dec = blk_mq_make_dicision(ret, bypass, force); switch(dec) { case MQ_ISSUE_INSERT_QUEUE: blk_mq_sched_insert_request(rq, false, run_queue, false); break; + case MQ_ISSUE_INSERT_DISPATCH: + blk_mq_request_bypass_insert(rq, run_queue); + break; case MQ_ISSUE_END_REQUEST: blk_mq_end_request(rq, ret); break; From patchwork Fri Nov 2 07:08:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10665099 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A4AE017DB for ; Fri, 2 Nov 2018 07:10:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D78F2B704 for ; Fri, 2 Nov 2018 07:10:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 70E742B774; Fri, 2 Nov 2018 07:10:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1AE422B704 for ; Fri, 2 Nov 2018 07:10:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727630AbeKBQQo (ORCPT ); Fri, 2 Nov 2018 12:16:44 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:46390 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728223AbeKBQQc (ORCPT ); Fri, 2 Nov 2018 12:16:32 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wA278nc8051958; Fri, 2 Nov 2018 07:10:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=L0zUsoRmbfMSOpgHNUToDzdZ2/wRayqRCeqFXxhg6gY=; b=05s1UuGkZUXVIQHoi5YMEvF9oS4Wxi/iWc5sQx37tvT6C6Z0C0d7Ot9Y8yuhhmDfpt3l zURW+dUjkrSsY7tXKneyf7IC96SkIKAm9V4FP8+0cHgIjqI6q6/F3AYm2/XUq/Y/fyOk 9UwWlvn5QXpqEN520agBSNTp84DHe/M09U0uB/g/TarxB0kfHkA85VMptcaxzt3n1Rcs rzK18uyZcb0MMrsgs7KY63OZQ0CbdCaiY4kemDcAFyczXhoDueTVkiJInoNCseBZ1T/e qjF0WvUHMoQLNCjA/+A+WNaBW8WBIVr4EVxgKtBbLf8o9Sdd2GYk3HdjsFqQyBMXV60I mA== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2130.oracle.com with ESMTP id 2nducmhgbt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 02 Nov 2018 07:10:19 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id wA27AJgO019932 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Nov 2018 07:10:19 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wA27AJJA005808; Fri, 2 Nov 2018 07:10:19 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Nov 2018 00:10:18 -0700 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V4 4/5] blk-mq: issue directly with bypass 'false' in blk_mq_sched_insert_requests Date: Fri, 2 Nov 2018 15:08:55 +0800 Message-Id: <1541142536-1431-5-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> References: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9064 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811020067 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It is not necessary to issue request directly with bypass 'true' in blk_mq_sched_insert_requests and insert the non-issued requests itself. Just set bypass to 'false' and let blk_mq_try_issue_directly handle them totally. Signed-off-by: Jianchao Wang --- block/blk-mq-sched.c | 8 +++----- block/blk-mq.c | 11 +---------- 2 files changed, 4 insertions(+), 15 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 29bfe80..23cd97e 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -411,12 +411,10 @@ void blk_mq_sched_insert_requests(struct request_queue *q, * busy in case of 'none' scheduler, and this way may save * us one extra enqueue & dequeue to sw queue. */ - if (!hctx->dispatch_busy && !e && !run_queue_async) { + if (!hctx->dispatch_busy && !e && !run_queue_async) blk_mq_try_issue_list_directly(hctx, list); - if (list_empty(list)) - return; - } - blk_mq_insert_requests(hctx, ctx, list); + else + blk_mq_insert_requests(hctx, ctx, list); } blk_mq_run_hw_queue(hctx, run_queue_async); diff --git a/block/blk-mq.c b/block/blk-mq.c index a0b9b6c..bf8b144 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1832,20 +1832,11 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, struct list_head *list) { while (!list_empty(list)) { - blk_status_t ret; struct request *rq = list_first_entry(list, struct request, queuelist); list_del_init(&rq->queuelist); - ret = blk_mq_request_issue_directly(rq); - if (ret != BLK_STS_OK) { - if (ret == BLK_STS_RESOURCE || - ret == BLK_STS_DEV_RESOURCE) { - list_add(&rq->queuelist, list); - break; - } - blk_mq_end_request(rq, ret); - } + blk_mq_try_issue_directly(hctx, rq, NULL, false); } } From patchwork Fri Nov 2 07:08:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jianchao.wang" X-Patchwork-Id: 10665097 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 668DF3E9D for ; Fri, 2 Nov 2018 07:10:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3F7C32B6F7 for ; Fri, 2 Nov 2018 07:10:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2FCE32B77F; Fri, 2 Nov 2018 07:10:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D187B2B704 for ; Fri, 2 Nov 2018 07:10:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728516AbeKBQQe (ORCPT ); Fri, 2 Nov 2018 12:16:34 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:34490 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727630AbeKBQQd (ORCPT ); Fri, 2 Nov 2018 12:16:33 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wA278qWY007466; Fri, 2 Nov 2018 07:10:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=lvSdNG6bIhLErnPr8+jj42nmVaiIk285oPHnP5XQBsY=; b=bJVaLWCRdCfQKm0y2K2eYsfhZwVq2pqr+TTJVi82gklwtJp2A1iGclE/3AXqQA/w+f6u hB+77/jZkjGGDjsxD2ydWTsQIJHWJO3ETai75xvS1FBD4wYfy/zZ+G+WaX6SZiECC0ft 15VA6Zwk+2nRQW0nitdMGf95Z76grAB1K7DZvuEfwanOFDQPpSoPWIsydjKNxe21o234 CXk/pI9GjN3DAsHF29eWKd6ROppTYj4QY0hzeTiMMTmUCPTaxuD9nrx/8RTqRJNWRKaE cq4Z9JO1Fiv6BqpZS0OgCxXVV9k7WqP84nLvcf67GSrOpXX4/3W6meu+FfvOtoLfqBtv VQ== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2ncgnrcujb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 02 Nov 2018 07:10:21 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wA27AL8E031401 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Nov 2018 07:10:21 GMT Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wA27AKZq004892; Fri, 2 Nov 2018 07:10:20 GMT Received: from will-ThinkCentre-M93p.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Nov 2018 00:10:20 -0700 From: Jianchao Wang To: axboe@kernel.dk Cc: ming.lei@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH V4 5/5] blk-mq: ensure hctx to be ran on mapped cpu when issue directly Date: Fri, 2 Nov 2018 15:08:56 +0800 Message-Id: <1541142536-1431-6-git-send-email-jianchao.w.wang@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> References: <1541142536-1431-1-git-send-email-jianchao.w.wang@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9064 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=948 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1811020067 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When issue request directly and the task is migrated out of the original cpu where it allocates request, hctx could be ran on the cpu where it is not mapped. To fix this, insert the request if BLK_MQ_F_BLOCKING is set, check whether the current is mapped to the hctx and invoke __blk_mq_issue_directly under preemption disabled. Signed-off-by: Jianchao Wang --- block/blk-mq.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index bf8b144..4450eb6 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1771,6 +1771,17 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, enum mq_issue_decision dec; int srcu_idx; + if (hctx->flags & BLK_MQ_F_BLOCKING) { + force = true; + goto out; + } + + if (!cpumask_test_cpu(get_cpu(), hctx->cpumask)) { + put_cpu(); + force = true; + goto out; + } + hctx_lock(hctx, &srcu_idx); /* @@ -1801,7 +1812,8 @@ static blk_status_t blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, out_unlock: hctx_unlock(hctx, srcu_idx); - + put_cpu(); +out: dec = blk_mq_make_dicision(ret, bypass, force); switch(dec) { case MQ_ISSUE_INSERT_QUEUE: