From patchwork Thu May 25 17:24:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9748771 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B383E60209 for ; Thu, 25 May 2017 17:24:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9680526247 for ; Thu, 25 May 2017 17:24:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8AE2A274D0; Thu, 25 May 2017 17:24:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4AF5326247 for ; Thu, 25 May 2017 17:24:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S942950AbdEYRYZ (ORCPT ); Thu, 25 May 2017 13:24:25 -0400 Received: from esa3.hgst.iphmx.com ([216.71.153.141]:33641 "EHLO esa3.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S936582AbdEYRYM (ORCPT ); Thu, 25 May 2017 13:24:12 -0400 X-IronPort-AV: E=Sophos;i="5.38,392,1491235200"; d="scan'208";a="20357410" Received: from mail-dm3nam03lp0020.outbound.protection.outlook.com (HELO NAM03-DM3-obe.outbound.protection.outlook.com) ([207.46.163.20]) by ob1.hgst.iphmx.com with ESMTP; 26 May 2017 01:24:11 +0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sharedspace.onmicrosoft.com; s=selector1-sharedspace-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=tRT6Jl8+8w0o8hd+nvynBlRVrTg0uRooRD1b5ZZQUKA=; b=tNPg0vdAVIYOwfhIzA1n6GlIRS7kTiG89HJmY2w90v7j7s39XNhAnLrpG8JhKO1vVMtxXqAxshi1un9Sl08Z+ib7P7dxfZRXR65UkOEEdXiq6yJHFNI+1kE8NrDE/ooil/WonBz9asvK0jozkF0MnnZHFqv6TFPWR+WPXl0aTeQ= Received: from CY1PR0401MB1536.namprd04.prod.outlook.com (10.163.19.154) by CY1PR0401MB1534.namprd04.prod.outlook.com (10.163.19.152) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1124.9; Thu, 25 May 2017 17:24:08 +0000 Received: from CY1PR0401MB1536.namprd04.prod.outlook.com ([10.163.19.154]) by CY1PR0401MB1536.namprd04.prod.outlook.com ([10.163.19.154]) with mapi id 15.01.1124.011; Thu, 25 May 2017 17:24:08 +0000 From: Bart Van Assche To: "ming.lei@redhat.com" CC: "hch@infradead.org" , "linux-block@vger.kernel.org" , "axboe@fb.com" Subject: Re: [PATCH 0/7] blk-mq: fix queue quiescing Thread-Topic: [PATCH 0/7] blk-mq: fix queue quiescing Thread-Index: AQHS1Q5pQ6I59WlVm0WeSFAcNQmOeaIEhFsAgAA+tQCAAIpAgA== Date: Thu, 25 May 2017 17:24:08 +0000 Message-ID: <1495733047.2615.1.camel@sandisk.com> References: <20170525042131.13172-1-ming.lei@redhat.com> <1495689892.3045.2.camel@sandisk.com> <20170525090917.GB15737@ming.t460p> In-Reply-To: <20170525090917.GB15737@ming.t460p> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: redhat.com; dkim=none (message not signed) header.d=none; redhat.com; dmarc=none action=none header.from=sandisk.com; x-originating-ip: [63.163.107.100] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; CY1PR0401MB1534; 7:0atlIViTgeNNY+h9Bmrw2h8QaQjTTChmO5w1WQfOj/RbtYquMHOHvxBTpi7wNPAf0+zy2WBNc59tm+E+GOOAaPzvOpXMQbM7/HVodjj3c8KNRZvaNkOYCzD3ZRqeG5gpPTzpBQRbQARCHNjuHgPhgbIhC4VP8mJxORXZqFWPKvsDL2v3Go3xlTVWszQmMHWPlAWEAsDGObIuuTp7TPrQ9Z7pT0+Y1Ogwqkqc8LSLa+MCeKmQ+LM/nT/aAk/Epn7hxCqx643aCTroEUBhB0uDEphSD4FEPQKjB9eGWIHEuomBzjST4soc5WUJfPx/x948TwCnOT/hZtziTSyNEwJ6Bg==; 20:t6B5KB3JU6eUj3L8TrWf4TS+gtpRc+8MImWNlD7ZgV/StBEh2w5Qo+fyMG6TXuK8iOQPN+ZVXmMV4ZiVm0y56gM7bi7vqgig2ts9w1VzNBhKlRLU/k7zBV2zTsxzRzxud9Ayabktu5flVMJ9UxtYkbV0A9AP2q2O42b9ZBJQOlU= x-ms-traffictypediagnostic: CY1PR0401MB1534: x-ms-office365-filtering-correlation-id: 4d33212a-9db8-4fa6-82ad-08d4a392da3a x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(48565401081)(201703131423075)(201703031133081)(201702281549075); SRVR:CY1PR0401MB1534; wdcipoutbound: EOP-TRUE x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123558100)(20161123555025)(20161123564025)(20161123562025)(6072148); SRVR:CY1PR0401MB1534; BCL:0; PCL:0; RULEID:; SRVR:CY1PR0401MB1534; x-forefront-prvs: 0318501FAE x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(39400400002)(39850400002)(39860400002)(39840400002)(39450400003)(39410400002)(24454002)(377424004)(99286003)(36756003)(54906002)(6512007)(2351001)(76176999)(50986999)(54356999)(2906002)(478600001)(25786009)(5640700003)(8676002)(38730400002)(110136004)(81166006)(3660700001)(4326008)(6246003)(6916009)(53936002)(2950100002)(5660300001)(102836003)(3846002)(8936002)(33646002)(3280700002)(66066001)(305945005)(2900100001)(2501003)(103116003)(86362001)(122556002)(575784001)(7736002)(6486002)(77096006)(189998001)(72206003)(6506006); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0401MB1534; H:CY1PR0401MB1536.namprd04.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-ID: <6167B1667169464986AEF95DFE7AD3A7@namprd04.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-originalarrivaltime: 25 May 2017 17:24:08.7121 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0401MB1534 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Thu, 2017-05-25 at 17:09 +0800, Ming Lei wrote: > Another big issue is that 'srcu_struct' is very big, which shouldn't > be embedded into hctx, since we only have one real user of > BLK_MQ_F_BLOCKING. > > So I will fix that too. Hello Ming, Is something like the (untested) patch below perhaps what you had in mind? Subject: [PATCH] blk-mq: Reduce blk_mq_hw_ctx size Since the srcu structure is rather large (184 bytes on an x86-64 system), only allocate it if needed. Reported-by: Ming Lei ---  block/blk-mq.c         | 13 ++++++++++++-  include/linux/blk-mq.h |  5 +++--  2 files changed, 15 insertions(+), 3 deletions(-) --  2.12.2 diff --git a/block/blk-mq.c b/block/blk-mq.c index 1e330de4e3c5..15b7d4077638 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2233,6 +2233,17 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set)  }  EXPORT_SYMBOL(blk_mq_init_queue);   +static int blk_mq_hw_ctx_size(struct request_queue *q) +{ + BUILD_BUG_ON(ALIGN(offsetof(struct blk_mq_hw_ctx, queue_rq_srcu) + + sizeof(((struct blk_mq_hw_ctx *)NULL)->queue_rq_srcu), 64) != + sizeof(struct blk_mq_hw_ctx)); + + return q->tag_set->flags & BLK_MQ_F_BLOCKING ? + sizeof(struct blk_mq_hw_ctx) : + offsetof(struct blk_mq_hw_ctx, queue_rq_srcu); +} +  static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,   struct request_queue *q)  { @@ -2247,7 +2258,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,   continue;     node = blk_mq_hw_queue_to_node(q->mq_map, i); - hctxs[i] = kzalloc_node(sizeof(struct blk_mq_hw_ctx), + hctxs[i] = kzalloc_node(blk_mq_hw_ctx_size(q),   GFP_KERNEL, node);   if (!hctxs[i])   break; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index c0d59330b5e0..8467e1f83524 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -39,8 +39,6 @@ struct blk_mq_hw_ctx {   struct blk_mq_tags *tags;   struct blk_mq_tags *sched_tags;   - struct srcu_struct queue_rq_srcu; -   unsigned long queued;   unsigned long run;  #define BLK_MQ_MAX_DISPATCH_ORDER 7 @@ -62,6 +60,9 @@ struct blk_mq_hw_ctx {   struct dentry *debugfs_dir;   struct dentry *sched_debugfs_dir;  #endif + + /* Must be the last member - see also blk_mq_hw_ctx_size(). */ + struct srcu_struct queue_rq_srcu;  };    struct blk_mq_tag_set {