From patchwork Wed Dec 5 20:24:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 10714859 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B08A313BB for ; Wed, 5 Dec 2018 20:24:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9330D2DA38 for ; Wed, 5 Dec 2018 20:24:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 878F02E43D; Wed, 5 Dec 2018 20:24:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A2602DA38 for ; Wed, 5 Dec 2018 20:24:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727436AbeLEUYi (ORCPT ); Wed, 5 Dec 2018 15:24:38 -0500 Received: from mail-qk1-f195.google.com ([209.85.222.195]:42286 "EHLO mail-qk1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbeLEUYi (ORCPT ); Wed, 5 Dec 2018 15:24:38 -0500 Received: by mail-qk1-f195.google.com with SMTP id w12so1908098qkb.9 for ; Wed, 05 Dec 2018 12:24:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=IPsDdKQlx38t2S9AR2ga4G4Dr1vJK59J1pvT/EwZZ1I=; b=iy4GUKPDxuT1gV7T2f4F2eL93pW0UkFoVY3DrQQE0WXJV2GBk6VAQ52lHMCtyLMk4C Xcw2mqDdVEbnHsF2ZjBpvRTq3ESbWET6kN7jG3RAnk308kwWIdzKN+mtnIkGx3XoV4RB swhGvozoUoipnoJqFY5/nlc1hUg/Oqt2jnkWobE7NIHH0qq1Bkd45UUsBZhAezUgoD8x Jrm8jyWabmAn65iJHTUxROVpNGAV1trTKotcFlHhfJAGjWynIw9/DYvsEPeJnAQ4dNHl e9sZWGnfRfuG5x8liyrpXDRJLpLCSTPmySLxXD6ihbN5PFxUUB2BybuPfbloN/lTlHWJ fEVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=IPsDdKQlx38t2S9AR2ga4G4Dr1vJK59J1pvT/EwZZ1I=; b=J8C7HsALyhAJnWZeNDjh8EqSBYVWqOP8B0muAVu6wWZHabpcA7huQbniyuSlC3u5ll cNfJ1hkjhsArBBiyHgjgYPyRw6vIlfYiBIG47ZqrOhkP5aZy9Bwiw5NuqbThkYOpE+ha ItgUQkJ1oDiELFzrPWteUN6GBatk8oKvTPxDHuazKOBKu1+XU9JTxau6Y0YPkfN5c5MZ PwQ3DKfeZpR96J7FQ6hI6Y06Qo6OsxyrOQ3a1H2D2EyzxH1+AED73E9Bj7UMeP63JgOp siaRz7fYJfiwUETuLvPZDl53SZ0DOF5npPUCesUmglWtONbUqMKzD9ESo0g3FINhRbTS KLkg== X-Gm-Message-State: AA+aEWa6NJILv5CDJ6NcL5gtBPgOj6SLXDupnoiAV6GhpTUqs7KaV+HK Ftf8GJQLyFwMKOKFzJCC738= X-Google-Smtp-Source: AFSGD/V5F0whJ55efXgBHr1ccvJU7JBgy0XfZh/SUfic2blW8KYM5VFVP1mLlbKa0hLLdv/uQnSP8Q== X-Received: by 2002:a37:3008:: with SMTP id w8mr23660228qkw.75.1544041477424; Wed, 05 Dec 2018 12:24:37 -0800 (PST) Received: from localhost (pool-68-160-144-192.bstnma.fios.verizon.net. [68.160.144.192]) by smtp.gmail.com with ESMTPSA id z23sm18445488qkg.50.2018.12.05.12.24.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 12:24:36 -0800 (PST) From: Mike Snitzer To: Jens Axboe Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Mikulas Patocka Subject: [PATCH v3 1/7] dm: dont rewrite dm_disk(md)->part0.in_flight Date: Wed, 5 Dec 2018 15:24:27 -0500 Message-Id: <20181205202433.95823-2-snitzer@redhat.com> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20181205202433.95823-1-snitzer@redhat.com> References: <20181205202433.95823-1-snitzer@redhat.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Mikulas Patocka generic_start_io_acct and generic_end_io_acct already update the variable in_flight using atomic operations, so we don't have to overwrite them again. Signed-off-by: Mikulas Patocka Signed-off-by: Mike Snitzer --- drivers/md/dm.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index a733e4c920af..a8ae7931bce7 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -663,8 +663,7 @@ static void start_io_acct(struct dm_io *io) generic_start_io_acct(md->queue, bio_op(bio), bio_sectors(bio), &dm_disk(md)->part0); - atomic_set(&dm_disk(md)->part0.in_flight[rw], - atomic_inc_return(&md->pending[rw])); + atomic_inc(&md->pending[rw]); if (unlikely(dm_stats_used(&md->stats))) dm_stats_account_io(&md->stats, bio_data_dir(bio), @@ -693,7 +692,6 @@ static void end_io_acct(struct dm_io *io) * a flush. */ pending = atomic_dec_return(&md->pending[rw]); - atomic_set(&dm_disk(md)->part0.in_flight[rw], pending); pending += atomic_read(&md->pending[rw^0x1]); /* nudge anyone waiting on suspend queue */ From patchwork Wed Dec 5 20:24:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 10714861 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7B00E13BB for ; Wed, 5 Dec 2018 20:24:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A05C2E396 for ; Wed, 5 Dec 2018 20:24:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E7C32E43D; Wed, 5 Dec 2018 20:24:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 121DF2E396 for ; Wed, 5 Dec 2018 20:24:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728331AbeLEUYk (ORCPT ); Wed, 5 Dec 2018 15:24:40 -0500 Received: from mail-qt1-f196.google.com ([209.85.160.196]:40118 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbeLEUYk (ORCPT ); Wed, 5 Dec 2018 15:24:40 -0500 Received: by mail-qt1-f196.google.com with SMTP id k12so23824745qtf.7 for ; Wed, 05 Dec 2018 12:24:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=icB+G85GZTRRdeXIVpg1MXe93D03U8a/VLdB+3cEmQw=; b=MKxW1lAM9NIpVnC9mpzbiT/XgEhB10AJ8M22AalW6ejgsSQPy645Q88JaO23kECVvs odXaeFxWLaTj92MuCOnUiqbQzHT123EmzWl6vd7DqgYpDE7R4UzVygNRWp/lqIOZsLUz bJPxFdYLPHu7qYkIr81N2ZS9td+otsSQfAtMrRtE6yEtYkWUPEnpmisNubV7kPH38TrH s+I1G5EVAMCaQtqpMgbBjDtLWhvGYu+0gnYL8KZSyn7xjtuLaPL8SryWsdOc4XRVuDqV QTNeNllowSdEt8jw2qbOSxDJ8vdxEJCAvBN7+M1tGzdJ4me5WmdbgSrVZn8ZKdrc1qDB G+mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=icB+G85GZTRRdeXIVpg1MXe93D03U8a/VLdB+3cEmQw=; b=tTmkWYQxEH2mHxFMzKb78QiKFn/mtP1TxRsFuFB5mdPKAX9eYkKjaIP4kCbGscOCYX Z9Ua09PnRnyc5tQCHXfEFB8/3tWDpfyOd+myicz81SRl/BsGYiz7xk9wMoSAD/rKIv7P H1QHpHrTWQmazcQ54TRlkp4/f0uTNwSsm1UCbLma38eo6mq0dUEeNZiTN0ojR5KCuuGG DgqDZj4TOSEIvGezcSStlGaHR6DIilrY2gh5KFdCNCHtfFMP/aBpKUQHERYUReJ1+Kh1 F1nxj//UjUWol7tdOpFyEKk4zkjwXszlsrLkXQCYPPh1Qs0BNirODPGv3yHD7+r+j7rf S4xA== X-Gm-Message-State: AA+aEWaIS2zx1yRkHm0kn5I9WqabtucdjFwNKNtFXlQHV4iaI4nciYGg T57ZhJJSyCqVJDZcPsAKW70= X-Google-Smtp-Source: AFSGD/XSqS3NBGMeF90Gj8HHohgV61AoBK6P5YGtrjoR7bTr6Yd5NwoJRCc8rI3xie6T5TptgB2hBg== X-Received: by 2002:a0c:c16b:: with SMTP id i40mr25240037qvh.28.1544041479128; Wed, 05 Dec 2018 12:24:39 -0800 (PST) Received: from localhost (pool-68-160-144-192.bstnma.fios.verizon.net. [68.160.144.192]) by smtp.gmail.com with ESMTPSA id c12sm15512621qka.42.2018.12.05.12.24.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 12:24:38 -0800 (PST) From: Mike Snitzer To: Jens Axboe Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Mikulas Patocka Subject: [PATCH v3 2/7] dm rq: leverage blk_mq_queue_busy() to check for outstanding IO Date: Wed, 5 Dec 2018 15:24:28 -0500 Message-Id: <20181205202433.95823-3-snitzer@redhat.com> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20181205202433.95823-1-snitzer@redhat.com> References: <20181205202433.95823-1-snitzer@redhat.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Now that request-based dm-multipath only supports blk-mq, make use of the newly introduced blk_mq_queue_busy() to check for outstanding IO -- rather than (ab)using the block core's in_flight counters. Signed-off-by: Mike Snitzer --- drivers/md/dm-rq.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index 1f1fe9a618ea..d2397d8fcbd1 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -130,11 +130,11 @@ static void rq_end_stats(struct mapped_device *md, struct request *orig) */ static void rq_completed(struct mapped_device *md, int rw, bool run_queue) { - atomic_dec(&md->pending[rw]); - /* nudge anyone waiting on suspend queue */ - if (!md_in_flight(md)) - wake_up(&md->wait); + if (unlikely(waitqueue_active(&md->wait))) { + if (!blk_mq_queue_busy(md->queue)) + wake_up(&md->wait); + } /* * dm_put() must be at the end of this function. See the comment above @@ -436,7 +436,6 @@ ssize_t dm_attr_rq_based_seq_io_merge_deadline_store(struct mapped_device *md, static void dm_start_request(struct mapped_device *md, struct request *orig) { blk_mq_start_request(orig); - atomic_inc(&md->pending[rq_data_dir(orig)]); if (unlikely(dm_stats_used(&md->stats))) { struct dm_rq_target_io *tio = tio_from_request(orig); From patchwork Wed Dec 5 20:24:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 10714863 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7244C13BB for ; Wed, 5 Dec 2018 20:24:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F89F2D73C for ; Wed, 5 Dec 2018 20:24:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 03FDE2DB77; Wed, 5 Dec 2018 20:24:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22A642E3DF for ; Wed, 5 Dec 2018 20:24:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728341AbeLEUYm (ORCPT ); Wed, 5 Dec 2018 15:24:42 -0500 Received: from mail-qk1-f195.google.com ([209.85.222.195]:37806 "EHLO mail-qk1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbeLEUYm (ORCPT ); Wed, 5 Dec 2018 15:24:42 -0500 Received: by mail-qk1-f195.google.com with SMTP id 131so12688274qkd.4 for ; Wed, 05 Dec 2018 12:24:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZZOnCWJEPZ+YGCX4qCMP4Op0EYJkiQR64SaGiEx1yO0=; b=MlGIh9/QveemJCAqOCMgeT+aB84xkfv25F0PVYTAy6GRjRHrp3pbSPIENfATkbL7Ny OT2a0QeTU1rK8hV0pMUsRPtNtoeLeFPK26wx+WEWiummg5OF3yX1wuU/k880tzzNUjDk hzNe5rMfZqn41HLAMxDxSDhtqeWIecS1RSusIX1FnisHun2IKV39dx+fDzfK+QMf3UoF xI/+vmbV9TtQZNfsJqUESwcq42fYLmVvC8339rLALWEFu6f24UPWUZr5XoUZN7rMPeVf DpH+QcRsQ3el+ojdlWrfwiRm4NHVMyGxmOXgAMkRa/p19K9StrXQNOP6YyRDIDuaONeB mRUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=ZZOnCWJEPZ+YGCX4qCMP4Op0EYJkiQR64SaGiEx1yO0=; b=kyu8ygJrGKVfdcXkOM6d/MNfqSIIdEjHh9803znLZfmW9MJ9NYIoD+WKYeWOLuAaMb 4KnQQpCa85Lif3aXRpMlY50M5txoYTA3BsSwsFZ68UuAdIkUNCf8QCEhqDTn6NutVGaV 0vq+Y0gjrLgbR+ZBd/qPFaou+ddDfrb8lssgfDjWo61Z0CDb+ccK7HyDqUPPE6ZzvzD9 pkXQU9++V8bOPJ4+0Nhe2p3JSX1D+ND1PcK5F/hrHIgRq7t8gStcPc26+v441ikbhloG 321/7D009tfcruJjxY06t8oePQO6CcVucewd4FbcIi9FIqbGjcJnmdyNgVax2aZXNG4V JeTA== X-Gm-Message-State: AA+aEWbIPtRJcUkk6lDuGR258T7cbhaGVeo+IQm5+ZRklEwZZ0wlNmlb jPzcsnjw1UR2b7f2mXDlKN8= X-Google-Smtp-Source: AFSGD/X0twp+gAYrMRSMJJQd9Tk0mi0pb8hiA5uDxpSXXcOihvYk2VJnw3nURdsiFdqDMu5454hOrw== X-Received: by 2002:a37:9281:: with SMTP id u123mr24585438qkd.0.1544041480987; Wed, 05 Dec 2018 12:24:40 -0800 (PST) Received: from localhost (pool-68-160-144-192.bstnma.fios.verizon.net. [68.160.144.192]) by smtp.gmail.com with ESMTPSA id 46sm14008923qtv.22.2018.12.05.12.24.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 12:24:40 -0800 (PST) From: Mike Snitzer To: Jens Axboe Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Mikulas Patocka Subject: [PATCH v3 3/7] block: stop passing 'cpu' to all percpu stats methods Date: Wed, 5 Dec 2018 15:24:29 -0500 Message-Id: <20181205202433.95823-4-snitzer@redhat.com> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20181205202433.95823-1-snitzer@redhat.com> References: <20181205202433.95823-1-snitzer@redhat.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP All of part_stat_* and related methods are used with preempt disabled, so there is no need to pass cpu around to allow of them. Just call smp_processor_id() as needed. Suggested-by: Jens Axboe Signed-off-by: Mike Snitzer --- block/bio.c | 16 +++++++++------- block/blk-core.c | 34 +++++++++++++++------------------- block/blk-merge.c | 5 ++--- block/genhd.c | 5 ++--- block/partition-generic.c | 5 ++--- drivers/md/md.c | 7 +++---- include/linux/genhd.h | 26 +++++++++++++------------- 7 files changed, 46 insertions(+), 52 deletions(-) diff --git a/block/bio.c b/block/bio.c index 03895cc0d74a..91e398ba57f1 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1667,11 +1667,12 @@ void generic_start_io_acct(struct request_queue *q, int op, unsigned long sectors, struct hd_struct *part) { const int sgrp = op_stat_group(op); - int cpu = part_stat_lock(); - part_round_stats(q, cpu, part); - part_stat_inc(cpu, part, ios[sgrp]); - part_stat_add(cpu, part, sectors[sgrp], sectors); + part_stat_lock(); + + part_round_stats(q, part); + part_stat_inc(part, ios[sgrp]); + part_stat_add(part, sectors[sgrp], sectors); part_inc_in_flight(q, part, op_is_write(op)); part_stat_unlock(); @@ -1683,10 +1684,11 @@ void generic_end_io_acct(struct request_queue *q, int req_op, { unsigned long duration = jiffies - start_time; const int sgrp = op_stat_group(req_op); - int cpu = part_stat_lock(); - part_stat_add(cpu, part, nsecs[sgrp], jiffies_to_nsecs(duration)); - part_round_stats(q, cpu, part); + part_stat_lock(); + + part_stat_add(part, nsecs[sgrp], jiffies_to_nsecs(duration)); + part_round_stats(q, part); part_dec_in_flight(q, part, op_is_write(req_op)); part_stat_unlock(); diff --git a/block/blk-core.c b/block/blk-core.c index ad59102ee30a..734b768c9d9d 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -584,14 +584,14 @@ struct request *blk_get_request(struct request_queue *q, unsigned int op, } EXPORT_SYMBOL(blk_get_request); -static void part_round_stats_single(struct request_queue *q, int cpu, +static void part_round_stats_single(struct request_queue *q, struct hd_struct *part, unsigned long now, unsigned int inflight) { if (inflight) { - __part_stat_add(cpu, part, time_in_queue, + __part_stat_add(part, time_in_queue, inflight * (now - part->stamp)); - __part_stat_add(cpu, part, io_ticks, (now - part->stamp)); + __part_stat_add(part, io_ticks, (now - part->stamp)); } part->stamp = now; } @@ -599,7 +599,6 @@ static void part_round_stats_single(struct request_queue *q, int cpu, /** * part_round_stats() - Round off the performance stats on a struct disk_stats. * @q: target block queue - * @cpu: cpu number for stats access * @part: target partition * * The average IO queue length and utilisation statistics are maintained @@ -613,7 +612,7 @@ static void part_round_stats_single(struct request_queue *q, int cpu, * /proc/diskstats. This accounts immediately for all queue usage up to * the current jiffies and restarts the counters again. */ -void part_round_stats(struct request_queue *q, int cpu, struct hd_struct *part) +void part_round_stats(struct request_queue *q, struct hd_struct *part) { struct hd_struct *part2 = NULL; unsigned long now = jiffies; @@ -635,9 +634,9 @@ void part_round_stats(struct request_queue *q, int cpu, struct hd_struct *part) part_in_flight(q, part, inflight); if (stats & 2) - part_round_stats_single(q, cpu, part2, now, inflight[1]); + part_round_stats_single(q, part2, now, inflight[1]); if (stats & 1) - part_round_stats_single(q, cpu, part, now, inflight[0]); + part_round_stats_single(q, part, now, inflight[0]); } EXPORT_SYMBOL_GPL(part_round_stats); @@ -1362,11 +1361,10 @@ void blk_account_io_completion(struct request *req, unsigned int bytes) if (blk_do_io_stat(req)) { const int sgrp = op_stat_group(req_op(req)); struct hd_struct *part; - int cpu; - cpu = part_stat_lock(); + part_stat_lock(); part = req->part; - part_stat_add(cpu, part, sectors[sgrp], bytes >> 9); + part_stat_add(part, sectors[sgrp], bytes >> 9); part_stat_unlock(); } } @@ -1381,14 +1379,13 @@ void blk_account_io_done(struct request *req, u64 now) if (blk_do_io_stat(req) && !(req->rq_flags & RQF_FLUSH_SEQ)) { const int sgrp = op_stat_group(req_op(req)); struct hd_struct *part; - int cpu; - cpu = part_stat_lock(); + part_stat_lock(); part = req->part; - part_stat_inc(cpu, part, ios[sgrp]); - part_stat_add(cpu, part, nsecs[sgrp], now - req->start_time_ns); - part_round_stats(req->q, cpu, part); + part_stat_inc(part, ios[sgrp]); + part_stat_add(part, nsecs[sgrp], now - req->start_time_ns); + part_round_stats(req->q, part); part_dec_in_flight(req->q, part, rq_data_dir(req)); hd_struct_put(part); @@ -1400,16 +1397,15 @@ void blk_account_io_start(struct request *rq, bool new_io) { struct hd_struct *part; int rw = rq_data_dir(rq); - int cpu; if (!blk_do_io_stat(rq)) return; - cpu = part_stat_lock(); + part_stat_lock(); if (!new_io) { part = rq->part; - part_stat_inc(cpu, part, merges[rw]); + part_stat_inc(part, merges[rw]); } else { part = disk_map_sector_rcu(rq->rq_disk, blk_rq_pos(rq)); if (!hd_struct_try_get(part)) { @@ -1424,7 +1420,7 @@ void blk_account_io_start(struct request *rq, bool new_io) part = &rq->rq_disk->part0; hd_struct_get(part); } - part_round_stats(rq->q, cpu, part); + part_round_stats(rq->q, part); part_inc_in_flight(rq->q, part, rw); rq->part = part; } diff --git a/block/blk-merge.c b/block/blk-merge.c index 4431da69a5cf..a120d59b9705 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -685,12 +685,11 @@ static void blk_account_io_merge(struct request *req) { if (blk_do_io_stat(req)) { struct hd_struct *part; - int cpu; - cpu = part_stat_lock(); + part_stat_lock(); part = req->part; - part_round_stats(req->q, cpu, part); + part_round_stats(req->q, part); part_dec_in_flight(req->q, part, rq_data_dir(req)); hd_struct_put(part); diff --git a/block/genhd.c b/block/genhd.c index 0145bcb0cc76..2fe00cf32b93 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -1326,7 +1326,6 @@ static int diskstats_show(struct seq_file *seqf, void *v) struct hd_struct *hd; char buf[BDEVNAME_SIZE]; unsigned int inflight[2]; - int cpu; /* if (&disk_to_dev(gp)->kobj.entry == block_class.devices.next) @@ -1338,8 +1337,8 @@ static int diskstats_show(struct seq_file *seqf, void *v) disk_part_iter_init(&piter, gp, DISK_PITER_INCL_EMPTY_PART0); while ((hd = disk_part_iter_next(&piter))) { - cpu = part_stat_lock(); - part_round_stats(gp->queue, cpu, hd); + part_stat_lock(); + part_round_stats(gp->queue, hd); part_stat_unlock(); part_in_flight(gp->queue, hd, inflight); seq_printf(seqf, "%4d %7d %s " diff --git a/block/partition-generic.c b/block/partition-generic.c index 5f8db5c5140f..7e663cfb1487 100644 --- a/block/partition-generic.c +++ b/block/partition-generic.c @@ -121,10 +121,9 @@ ssize_t part_stat_show(struct device *dev, struct hd_struct *p = dev_to_part(dev); struct request_queue *q = part_to_disk(p)->queue; unsigned int inflight[2]; - int cpu; - cpu = part_stat_lock(); - part_round_stats(q, cpu, p); + part_stat_lock(); + part_round_stats(q, p); part_stat_unlock(); part_in_flight(q, p, inflight); return sprintf(buf, diff --git a/drivers/md/md.c b/drivers/md/md.c index fc488cb30a94..9a0a1e0934d5 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -334,7 +334,6 @@ static blk_qc_t md_make_request(struct request_queue *q, struct bio *bio) const int sgrp = op_stat_group(bio_op(bio)); struct mddev *mddev = q->queuedata; unsigned int sectors; - int cpu; blk_queue_split(q, &bio); @@ -359,9 +358,9 @@ static blk_qc_t md_make_request(struct request_queue *q, struct bio *bio) md_handle_request(mddev, bio); - cpu = part_stat_lock(); - part_stat_inc(cpu, &mddev->gendisk->part0, ios[sgrp]); - part_stat_add(cpu, &mddev->gendisk->part0, sectors[sgrp], sectors); + part_stat_lock(); + part_stat_inc(&mddev->gendisk->part0, ios[sgrp]); + part_stat_add(&mddev->gendisk->part0, sectors[sgrp], sectors); part_stat_unlock(); return BLK_QC_T_NONE; diff --git a/include/linux/genhd.h b/include/linux/genhd.h index 0c5ee17b4d88..1677cd2a4c4e 100644 --- a/include/linux/genhd.h +++ b/include/linux/genhd.h @@ -295,8 +295,8 @@ extern struct hd_struct *disk_map_sector_rcu(struct gendisk *disk, #define part_stat_lock() ({ rcu_read_lock(); get_cpu(); }) #define part_stat_unlock() do { put_cpu(); rcu_read_unlock(); } while (0) -#define __part_stat_add(cpu, part, field, addnd) \ - (per_cpu_ptr((part)->dkstats, (cpu))->field += (addnd)) +#define __part_stat_add(part, field, addnd) \ + (per_cpu_ptr((part)->dkstats, smp_processor_id())->field += (addnd)) #define part_stat_read(part, field) \ ({ \ @@ -333,7 +333,7 @@ static inline void free_part_stats(struct hd_struct *part) #define part_stat_lock() ({ rcu_read_lock(); 0; }) #define part_stat_unlock() rcu_read_unlock() -#define __part_stat_add(cpu, part, field, addnd) \ +#define __part_stat_add(part, field, addnd) \ ((part)->dkstats.field += addnd) #define part_stat_read(part, field) ((part)->dkstats.field) @@ -362,19 +362,19 @@ static inline void free_part_stats(struct hd_struct *part) part_stat_read(part, field[STAT_WRITE]) + \ part_stat_read(part, field[STAT_DISCARD])) -#define part_stat_add(cpu, part, field, addnd) do { \ - __part_stat_add((cpu), (part), field, addnd); \ +#define part_stat_add(part, field, addnd) do { \ + __part_stat_add((part), field, addnd); \ if ((part)->partno) \ - __part_stat_add((cpu), &part_to_disk((part))->part0, \ + __part_stat_add(&part_to_disk((part))->part0, \ field, addnd); \ } while (0) -#define part_stat_dec(cpu, gendiskp, field) \ - part_stat_add(cpu, gendiskp, field, -1) -#define part_stat_inc(cpu, gendiskp, field) \ - part_stat_add(cpu, gendiskp, field, 1) -#define part_stat_sub(cpu, gendiskp, field, subnd) \ - part_stat_add(cpu, gendiskp, field, -subnd) +#define part_stat_dec(gendiskp, field) \ + part_stat_add(gendiskp, field, -1) +#define part_stat_inc(gendiskp, field) \ + part_stat_add(gendiskp, field, 1) +#define part_stat_sub(gendiskp, field, subnd) \ + part_stat_add(gendiskp, field, -subnd) void part_in_flight(struct request_queue *q, struct hd_struct *part, unsigned int inflight[2]); @@ -399,7 +399,7 @@ static inline void free_part_info(struct hd_struct *part) } /* block/blk-core.c */ -extern void part_round_stats(struct request_queue *q, int cpu, struct hd_struct *part); +extern void part_round_stats(struct request_queue *q, struct hd_struct *part); /* block/genhd.c */ extern void device_add_disk(struct device *parent, struct gendisk *disk, From patchwork Wed Dec 5 20:24:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 10714865 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CC6E81731 for ; Wed, 5 Dec 2018 20:24:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 84C2A2E365 for ; Wed, 5 Dec 2018 20:24:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 78B2C2E433; Wed, 5 Dec 2018 20:24:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8EE652E387 for ; Wed, 5 Dec 2018 20:24:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728356AbeLEUYo (ORCPT ); Wed, 5 Dec 2018 15:24:44 -0500 Received: from mail-qt1-f176.google.com ([209.85.160.176]:34496 "EHLO mail-qt1-f176.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbeLEUYn (ORCPT ); Wed, 5 Dec 2018 15:24:43 -0500 Received: by mail-qt1-f176.google.com with SMTP id r14so23898156qtp.1 for ; Wed, 05 Dec 2018 12:24:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=xGJsbW2rgZ9JSRuIPaddEeemuWEVWwJK5ZTanjIVHww=; b=XR7Wg5z5SUyhvYr50wl9qtusSh0KIoHuhmJlMpluKuMj0Cxb/3jWrbufSShQQe7O4g ZSIrA8NzNqWM/ukRt0FDc95jsYA5wS6gBWs49PFp6rM7EkckidSPbZE16dqTZ4jvAO9x T5+lyEpgh6j/LIkXGlrPmqrHOffZwrbqxdCkv4wE9mWPz0brw94uqH4exyeddL5vXLu/ uNpiXodXvUsyepsBOWXMYJUev6hzhp5uOXSFbewasCOMxZ4EMsAhUppGlvYrFjqMLmB5 BLXt+CRpUhDzNOqGmDKvCMUxSoyyL4wElW8PAR85oN66bWH3pZeM8gvlSTFb5PQjFQIf Whng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=xGJsbW2rgZ9JSRuIPaddEeemuWEVWwJK5ZTanjIVHww=; b=FsB0Lb/zBYqbT88Wedh37J4GJd8Z7B6UZI9ss1bmnA67PlIU6hUvj4L2y2LApWPy2H MEWGfGmsW8rbqY73M3CYcAUfxWWTE9JfLWb/ODJEJc0ZMsZPmcCsxPTRFw+b4Dyq87IU s5wJrqEv8e36JKNinIyXbwA3tMrhFPO3+OJdnpg0y3vIU/d1YxxxWQ167Gn/Q+bZyXmd uNHLW752YxQ1e4vxCPu88giRI+pX2s+qd7eDLhqGku8lU+CyuqWlY38dHx0tC9o2MG6o EyVYL17afkMIY7K/37Cefds7WEYYWrMXQMMgLEjy6U7uriZ/umvXuO6D1m1U0/eQEj9w UWQA== X-Gm-Message-State: AA+aEWZf0CtDrxmKHrTylvvTpUR3wvXTKZKiaMN/NyqZiZV706Um50oY 7K7Ii7kF4HyBea4qcPtX4uM= X-Google-Smtp-Source: AFSGD/XflAFca5sugGRkfpY5BHVLQAU4ySrI+nkfkOhRqbzA0od+9K6YO9cJocUVzKxTGDRS1oryHw== X-Received: by 2002:ac8:31ed:: with SMTP id i42mr24238783qte.323.1544041482823; Wed, 05 Dec 2018 12:24:42 -0800 (PST) Received: from localhost (pool-68-160-144-192.bstnma.fios.verizon.net. [68.160.144.192]) by smtp.gmail.com with ESMTPSA id 190sm10720651qkh.25.2018.12.05.12.24.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 12:24:41 -0800 (PST) From: Mike Snitzer To: Jens Axboe Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Mikulas Patocka Subject: [PATCH v3 4/7] block: delete part_round_stats and switch to less precise counting Date: Wed, 5 Dec 2018 15:24:30 -0500 Message-Id: <20181205202433.95823-5-snitzer@redhat.com> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20181205202433.95823-1-snitzer@redhat.com> References: <20181205202433.95823-1-snitzer@redhat.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Mikulas Patocka We want to convert to per-cpu in_flight counters. The function part_round_stats needs the in_flight counter every jiffy, it would be too costly to sum all the percpu variables every jiffy, so it must be deleted. part_round_stats is used to calculate two counters - time_in_queue and io_ticks. time_in_queue can be calculated without part_round_stats, by adding the duration of the I/O when the I/O ends (the value is almost as exact as the previously calculated value, except that time for in-progress I/Os is not counted). io_ticks can be approximated by increasing the value when I/O is started or ended and the jiffies value has changed. If the I/Os take less than a jiffy, the value is as exact as the previously calculated value. If the I/Os take more than a jiffy, io_ticks can drift behind the previously calculated value. Signed-off-by: Mikulas Patocka Signed-off-by: Mike Snitzer --- block/bio.c | 24 +++++++++++++-- block/blk-core.c | 62 +++------------------------------------ block/blk-merge.c | 1 - block/genhd.c | 3 -- block/partition-generic.c | 3 -- include/linux/genhd.h | 3 +- 6 files changed, 26 insertions(+), 70 deletions(-) diff --git a/block/bio.c b/block/bio.c index 91e398ba57f1..0c2208a5446d 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1663,6 +1663,22 @@ void bio_check_pages_dirty(struct bio *bio) } EXPORT_SYMBOL_GPL(bio_check_pages_dirty); +void update_io_ticks(struct hd_struct *part, unsigned long now) +{ + unsigned long stamp; +again: + stamp = READ_ONCE(part->stamp); + if (unlikely(stamp != now)) { + if (likely(cmpxchg(&part->stamp, stamp, now) == stamp)) { + __part_stat_add(part, io_ticks, 1); + } + } + if (part->partno) { + part = &part_to_disk(part)->part0; + goto again; + } +} + void generic_start_io_acct(struct request_queue *q, int op, unsigned long sectors, struct hd_struct *part) { @@ -1670,7 +1686,7 @@ void generic_start_io_acct(struct request_queue *q, int op, part_stat_lock(); - part_round_stats(q, part); + update_io_ticks(part, jiffies); part_stat_inc(part, ios[sgrp]); part_stat_add(part, sectors[sgrp], sectors); part_inc_in_flight(q, part, op_is_write(op)); @@ -1682,13 +1698,15 @@ EXPORT_SYMBOL(generic_start_io_acct); void generic_end_io_acct(struct request_queue *q, int req_op, struct hd_struct *part, unsigned long start_time) { - unsigned long duration = jiffies - start_time; + unsigned long now = jiffies; + unsigned long duration = now - start_time; const int sgrp = op_stat_group(req_op); part_stat_lock(); + update_io_ticks(part, now); part_stat_add(part, nsecs[sgrp], jiffies_to_nsecs(duration)); - part_round_stats(q, part); + part_stat_add(part, time_in_queue, duration); part_dec_in_flight(q, part, op_is_write(req_op)); part_stat_unlock(); diff --git a/block/blk-core.c b/block/blk-core.c index 734b768c9d9d..268d2b8e9843 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -584,62 +584,6 @@ struct request *blk_get_request(struct request_queue *q, unsigned int op, } EXPORT_SYMBOL(blk_get_request); -static void part_round_stats_single(struct request_queue *q, - struct hd_struct *part, unsigned long now, - unsigned int inflight) -{ - if (inflight) { - __part_stat_add(part, time_in_queue, - inflight * (now - part->stamp)); - __part_stat_add(part, io_ticks, (now - part->stamp)); - } - part->stamp = now; -} - -/** - * part_round_stats() - Round off the performance stats on a struct disk_stats. - * @q: target block queue - * @part: target partition - * - * The average IO queue length and utilisation statistics are maintained - * by observing the current state of the queue length and the amount of - * time it has been in this state for. - * - * Normally, that accounting is done on IO completion, but that can result - * in more than a second's worth of IO being accounted for within any one - * second, leading to >100% utilisation. To deal with that, we call this - * function to do a round-off before returning the results when reading - * /proc/diskstats. This accounts immediately for all queue usage up to - * the current jiffies and restarts the counters again. - */ -void part_round_stats(struct request_queue *q, struct hd_struct *part) -{ - struct hd_struct *part2 = NULL; - unsigned long now = jiffies; - unsigned int inflight[2]; - int stats = 0; - - if (part->stamp != now) - stats |= 1; - - if (part->partno) { - part2 = &part_to_disk(part)->part0; - if (part2->stamp != now) - stats |= 2; - } - - if (!stats) - return; - - part_in_flight(q, part, inflight); - - if (stats & 2) - part_round_stats_single(q, part2, now, inflight[1]); - if (stats & 1) - part_round_stats_single(q, part, now, inflight[0]); -} -EXPORT_SYMBOL_GPL(part_round_stats); - void blk_put_request(struct request *req) { blk_mq_free_request(req); @@ -1383,9 +1327,10 @@ void blk_account_io_done(struct request *req, u64 now) part_stat_lock(); part = req->part; + update_io_ticks(part, jiffies); part_stat_inc(part, ios[sgrp]); part_stat_add(part, nsecs[sgrp], now - req->start_time_ns); - part_round_stats(req->q, part); + part_stat_add(part, time_in_queue, nsecs_to_jiffies64(now - req->start_time_ns)); part_dec_in_flight(req->q, part, rq_data_dir(req)); hd_struct_put(part); @@ -1420,11 +1365,12 @@ void blk_account_io_start(struct request *rq, bool new_io) part = &rq->rq_disk->part0; hd_struct_get(part); } - part_round_stats(rq->q, part); part_inc_in_flight(rq->q, part, rw); rq->part = part; } + update_io_ticks(part, jiffies); + part_stat_unlock(); } diff --git a/block/blk-merge.c b/block/blk-merge.c index a120d59b9705..9da5629d0887 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -689,7 +689,6 @@ static void blk_account_io_merge(struct request *req) part_stat_lock(); part = req->part; - part_round_stats(req->q, part); part_dec_in_flight(req->q, part, rq_data_dir(req)); hd_struct_put(part); diff --git a/block/genhd.c b/block/genhd.c index 2fe00cf32b93..cdf174d7d329 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -1337,9 +1337,6 @@ static int diskstats_show(struct seq_file *seqf, void *v) disk_part_iter_init(&piter, gp, DISK_PITER_INCL_EMPTY_PART0); while ((hd = disk_part_iter_next(&piter))) { - part_stat_lock(); - part_round_stats(gp->queue, hd); - part_stat_unlock(); part_in_flight(gp->queue, hd, inflight); seq_printf(seqf, "%4d %7d %s " "%lu %lu %lu %u " diff --git a/block/partition-generic.c b/block/partition-generic.c index 7e663cfb1487..42d6138ac876 100644 --- a/block/partition-generic.c +++ b/block/partition-generic.c @@ -122,9 +122,6 @@ ssize_t part_stat_show(struct device *dev, struct request_queue *q = part_to_disk(p)->queue; unsigned int inflight[2]; - part_stat_lock(); - part_round_stats(q, p); - part_stat_unlock(); part_in_flight(q, p, inflight); return sprintf(buf, "%8lu %8lu %8llu %8u " diff --git a/include/linux/genhd.h b/include/linux/genhd.h index 1677cd2a4c4e..838c2a7a40c5 100644 --- a/include/linux/genhd.h +++ b/include/linux/genhd.h @@ -398,8 +398,7 @@ static inline void free_part_info(struct hd_struct *part) kfree(part->info); } -/* block/blk-core.c */ -extern void part_round_stats(struct request_queue *q, struct hd_struct *part); +void update_io_ticks(struct hd_struct *part, unsigned long now); /* block/genhd.c */ extern void device_add_disk(struct device *parent, struct gendisk *disk, From patchwork Wed Dec 5 20:24:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 10714867 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 51AC718A7 for ; Wed, 5 Dec 2018 20:24:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E2BB2E415 for ; Wed, 5 Dec 2018 20:24:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 22A352E433; Wed, 5 Dec 2018 20:24:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACEB02D363 for ; Wed, 5 Dec 2018 20:24:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728360AbeLEUYq (ORCPT ); Wed, 5 Dec 2018 15:24:46 -0500 Received: from mail-qt1-f195.google.com ([209.85.160.195]:45487 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbeLEUYq (ORCPT ); Wed, 5 Dec 2018 15:24:46 -0500 Received: by mail-qt1-f195.google.com with SMTP id e5so23809658qtr.12 for ; Wed, 05 Dec 2018 12:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=5qg6a7Y6go7cyLi7i49o60iw5DUZNQ7KPvqRiUxLRyU=; b=vgoMiXIjJTwS80/Ru5Bd4OWuvZwGMxZfGnu9zF2hLsK5+lMUKXIHIyIFgsMPpMRfyn XrLEae4+YoEmbQ5tAexB+EP6oXUkNXDsXDLCTLxAHFQaUB/kD/YinW+K8rdQofenZ+N4 MVVSsmXuit8jpJCybWpIEZnmyLK/d9LHxtm8ekrJ/W1P4aiKFHsq16eYutIyxla35kbM zhRMYsUrR4McY79xtK/slQX//yijX3yO71mFQs1z+6Mbrq3ncOtHh0m9S2rE+ddhXek6 pJDJtVH8jvlzhj1os1ci3+QHRRUV8LBecKuGYsgkFMy3/uhEQmNkL1Ef6LJD4sfolvIu Th4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=5qg6a7Y6go7cyLi7i49o60iw5DUZNQ7KPvqRiUxLRyU=; b=JnO2bPtWlxNaBeLRRLLT8d6aKRXiS3QDlteaWq1WpQFgVnT0qekTWeS6FxBGlvG0y4 uII4bnD2IOTOr1W1t4cpke9sgGhyO5uJjXeg7hluLsdH4sH8iwb0LUcf3dLHtKYVh+Pe IHc751j3IbSKOmek5zY+RdTGIh20XjkTy7jnCUhVoC+Bw549GEEzPyT0iXutqrFb5MR0 BWbfw2PfiEc+I5bYlrT9DxOkh8zH2eJMC6nQsH6xPqMHXR8xdLryenn1XWSb5K5EqyAX widbiDRe9NAFjv3AGHfPm/E6yrTxnMUvUWx17n9Od8LYC7WQ6esBlWY6O3JzvSUngVuF fINQ== X-Gm-Message-State: AA+aEWbR34HgtrmBzZ+OhPHvA+W3E2yTqlEPmvqr2/Hci2T9LPiTH8U6 7oy7ZPj+5cgYO09rCgdOXu0= X-Google-Smtp-Source: AFSGD/W0m9KSqMEEPdonxFDoyQR3WEOtXMM8b8DoFtLBqUaEnLcudoJozTpMcRfBYGYtHBBS/4ySJg== X-Received: by 2002:a0c:a045:: with SMTP id b63mr25424831qva.164.1544041484655; Wed, 05 Dec 2018 12:24:44 -0800 (PST) Received: from localhost (pool-68-160-144-192.bstnma.fios.verizon.net. [68.160.144.192]) by smtp.gmail.com with ESMTPSA id o42sm14380641qtc.90.2018.12.05.12.24.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 12:24:43 -0800 (PST) From: Mike Snitzer To: Jens Axboe Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Mikulas Patocka Subject: [PATCH v3 5/7] block: switch to per-cpu in-flight counters Date: Wed, 5 Dec 2018 15:24:31 -0500 Message-Id: <20181205202433.95823-6-snitzer@redhat.com> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20181205202433.95823-1-snitzer@redhat.com> References: <20181205202433.95823-1-snitzer@redhat.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Mikulas Patocka Now when part_round_stats is gone, we can switch to per-cpu in-flight counters. We use the local-atomic type local_t, so that if part_inc_in_flight or part_dec_in_flight is reentrantly called from an interrupt, the value will be correct. The other counters could be corrupted due to reentrant interrupt, but the corruption only results in slight counter skew - the in_flight counter must be exact, so it needs local_t. Signed-off-by: Mikulas Patocka Signed-off-by: Mike Snitzer --- block/genhd.c | 49 ++++++++++++++++++++++++++++++++++--------- include/linux/genhd.h | 3 ++- 2 files changed, 41 insertions(+), 11 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index cdf174d7d329..ffb9d416db64 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -47,51 +47,80 @@ static void disk_release_events(struct gendisk *disk); void part_inc_in_flight(struct request_queue *q, struct hd_struct *part, int rw) { + int cpu; + if (queue_is_mq(q)) return; - atomic_inc(&part->in_flight[rw]); + cpu = smp_processor_id(); + local_inc(&per_cpu_ptr(part->dkstats, cpu)->in_flight[rw]); if (part->partno) - atomic_inc(&part_to_disk(part)->part0.in_flight[rw]); + local_inc(&per_cpu_ptr(part_to_disk(part)->part0.dkstats, cpu)->in_flight[rw]); } void part_dec_in_flight(struct request_queue *q, struct hd_struct *part, int rw) { + int cpu; + if (queue_is_mq(q)) return; - atomic_dec(&part->in_flight[rw]); + cpu = smp_processor_id(); + local_dec(&per_cpu_ptr(part->dkstats, cpu)->in_flight[rw]); if (part->partno) - atomic_dec(&part_to_disk(part)->part0.in_flight[rw]); + local_dec(&per_cpu_ptr(part_to_disk(part)->part0.dkstats, cpu)->in_flight[rw]); } void part_in_flight(struct request_queue *q, struct hd_struct *part, unsigned int inflight[2]) { + int cpu; + if (queue_is_mq(q)) { blk_mq_in_flight(q, part, inflight); return; } - inflight[0] = atomic_read(&part->in_flight[0]) + - atomic_read(&part->in_flight[1]); + inflight[0] = 0; + for_each_possible_cpu(cpu) { + inflight[0] += local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[0]) + + local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[1]); + } + if ((int)inflight[0] < 0) + inflight[0] = 0; + if (part->partno) { part = &part_to_disk(part)->part0; - inflight[1] = atomic_read(&part->in_flight[0]) + - atomic_read(&part->in_flight[1]); + inflight[1] = 0; + for_each_possible_cpu(cpu) { + inflight[1] += local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[0]) + + local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[1]); + } + if ((int)inflight[1] < 0) + inflight[1] = 0; } } void part_in_flight_rw(struct request_queue *q, struct hd_struct *part, unsigned int inflight[2]) { + int cpu; + if (queue_is_mq(q)) { blk_mq_in_flight_rw(q, part, inflight); return; } - inflight[0] = atomic_read(&part->in_flight[0]); - inflight[1] = atomic_read(&part->in_flight[1]); + inflight[0] = 0; + inflight[1] = 0; + for_each_possible_cpu(cpu) { + inflight[0] += local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[0]); + inflight[1] += local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[1]); + } + if ((int)inflight[0] < 0) + inflight[0] = 0; + if ((int)inflight[1] < 0) + inflight[1] = 0; } struct hd_struct *__disk_get_part(struct gendisk *disk, int partno) diff --git a/include/linux/genhd.h b/include/linux/genhd.h index 838c2a7a40c5..9489dcb7cc27 100644 --- a/include/linux/genhd.h +++ b/include/linux/genhd.h @@ -17,6 +17,7 @@ #include #include #include +#include #ifdef CONFIG_BLOCK @@ -89,6 +90,7 @@ struct disk_stats { unsigned long merges[NR_STAT_GROUPS]; unsigned long io_ticks; unsigned long time_in_queue; + local_t in_flight[2]; }; #define PARTITION_META_INFO_VOLNAMELTH 64 @@ -122,7 +124,6 @@ struct hd_struct { int make_it_fail; #endif unsigned long stamp; - atomic_t in_flight[2]; #ifdef CONFIG_SMP struct disk_stats __percpu *dkstats; #else From patchwork Wed Dec 5 20:24:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 10714869 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8688313BB for ; Wed, 5 Dec 2018 20:24:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57D402E433 for ; Wed, 5 Dec 2018 20:24:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4BAE62E594; Wed, 5 Dec 2018 20:24:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C8E6A2E433 for ; Wed, 5 Dec 2018 20:24:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728391AbeLEUYs (ORCPT ); Wed, 5 Dec 2018 15:24:48 -0500 Received: from mail-qt1-f195.google.com ([209.85.160.195]:34949 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbeLEUYs (ORCPT ); Wed, 5 Dec 2018 15:24:48 -0500 Received: by mail-qt1-f195.google.com with SMTP id v11so23855381qtc.2 for ; Wed, 05 Dec 2018 12:24:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=uDXGZWmWGlMBhtUHpFWeBKtFEXOqKhmCzRaLRoOIvZk=; b=maq5ei4I8Q+znxuXL0sUOoUljfHDWyHIbosxRQsiXx0xGCBnRu8ZyzsRNOSfb4ph9l twZzG85xR7jGgdn+PPIV5z8JYa6EWtYFV2G2UNioOBwFeIT1ikOwJGFCLzcu9q0ppsQJ wMc4ZZepU6SUaBrqqsaekmhcZlhnQAwDoyOLvt5BxlL3k88bMr7+o5HqPcjledrfrAWd CntO73h5KMBfbWpBlLgmo11lq3n6qD8nZ+NivB4MAtfQGeZzDfOyhcOPJZaG5EIs+7Nw tGxOFA4rH7EhAPUwxB8oIx2DDhBURxf7rzVq7las9//3kHbE8FeNgjRqW5eL24VZIeVb P+ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=uDXGZWmWGlMBhtUHpFWeBKtFEXOqKhmCzRaLRoOIvZk=; b=WZ+XWUHnGuYxSIjUuR3jEnCQLb0XfRUBJ/taBLZIjnxHRaK7GwRA1qxu05JxsKEE/K KXR9DJO4w42j3ob6t1XxofF8c6K5w5sHdYR6hEyHj6cew07AoxcbSed2HovecskNmy0Y jylHwXEaljXPMthlPRE4HeU85K4ElHrep3m4gcW9hB065va2GMZaFv+gAzddxnTDGrkN 6fYTQooVrq9D+FIesyUvycv/T0zqvWjxGmZmoOk/Q5WrDKzGVjguZ8nkTc7Bx/dtvh3s HHxLmU9Q9iemycIF/mr2YPHSDlq1bZZSZ9GuxTEqKxcljtl9W9FleLsoEqfFDeG5qSPF akhw== X-Gm-Message-State: AA+aEWb+0Q5XFXrpy2egpCFJFQZNibjXBlk85a4WbZZW51cXLwPkq62i dixzM7L5e9C18D8iBF9X0OI= X-Google-Smtp-Source: AFSGD/Wv2ljeBsmMBPVdR5RsdKZdRXueaQX8heV/YxYz4H3V+l06mf8TQx8ClYLs61bU5qJVHX4AsA== X-Received: by 2002:aed:2aa3:: with SMTP id t32mr24558973qtd.25.1544041486532; Wed, 05 Dec 2018 12:24:46 -0800 (PST) Received: from localhost (pool-68-160-144-192.bstnma.fios.verizon.net. [68.160.144.192]) by smtp.gmail.com with ESMTPSA id q38sm12426825qtj.65.2018.12.05.12.24.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 12:24:45 -0800 (PST) From: Mike Snitzer To: Jens Axboe Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Mikulas Patocka Subject: [PATCH v3 6/7] block: return just one value from part_in_flight Date: Wed, 5 Dec 2018 15:24:32 -0500 Message-Id: <20181205202433.95823-7-snitzer@redhat.com> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20181205202433.95823-1-snitzer@redhat.com> References: <20181205202433.95823-1-snitzer@redhat.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Mikulas Patocka The previous patches deleted all the code that needed the second value returned from part_in_flight - now the kernel only uses the first value. Consequently, part_in_flight (and blk_mq_in_flight) may be changed so that it only returns one value. This patch just refactors the code, there's no functional change. Signed-off-by: Mikulas Patocka Signed-off-by: Mike Snitzer --- block/blk-mq.c | 12 +++++------- block/blk-mq.h | 3 +-- block/genhd.c | 32 +++++++++++--------------------- block/partition-generic.c | 6 +++--- include/linux/genhd.h | 3 +-- 5 files changed, 21 insertions(+), 35 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 900550594651..c6d3101352f4 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -100,25 +100,23 @@ static bool blk_mq_check_inflight(struct blk_mq_hw_ctx *hctx, struct mq_inflight *mi = priv; /* - * index[0] counts the specific partition that was asked for. index[1] - * counts the ones that are active on the whole device, so increment - * that if mi->part is indeed a partition, and not a whole device. + * index[0] counts the specific partition that was asked for. */ if (rq->part == mi->part) mi->inflight[0]++; - if (mi->part->partno) - mi->inflight[1]++; return true; } -void blk_mq_in_flight(struct request_queue *q, struct hd_struct *part, - unsigned int inflight[2]) +unsigned int blk_mq_in_flight(struct request_queue *q, struct hd_struct *part) { + unsigned inflight[2]; struct mq_inflight mi = { .part = part, .inflight = inflight, }; inflight[0] = inflight[1] = 0; blk_mq_queue_tag_busy_iter(q, blk_mq_check_inflight, &mi); + + return inflight[0]; } static bool blk_mq_check_inflight_rw(struct blk_mq_hw_ctx *hctx, diff --git a/block/blk-mq.h b/block/blk-mq.h index a664ea44ffd4..0c9c9ea2fefe 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -187,8 +187,7 @@ static inline bool blk_mq_hw_queue_mapped(struct blk_mq_hw_ctx *hctx) return hctx->nr_ctx && hctx->tags; } -void blk_mq_in_flight(struct request_queue *q, struct hd_struct *part, - unsigned int inflight[2]); +unsigned int blk_mq_in_flight(struct request_queue *q, struct hd_struct *part); void blk_mq_in_flight_rw(struct request_queue *q, struct hd_struct *part, unsigned int inflight[2]); diff --git a/block/genhd.c b/block/genhd.c index ffb9d416db64..dd84922192b6 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -71,34 +71,24 @@ void part_dec_in_flight(struct request_queue *q, struct hd_struct *part, int rw) local_dec(&per_cpu_ptr(part_to_disk(part)->part0.dkstats, cpu)->in_flight[rw]); } -void part_in_flight(struct request_queue *q, struct hd_struct *part, - unsigned int inflight[2]) +unsigned int part_in_flight(struct request_queue *q, struct hd_struct *part) { int cpu; + int inflight; if (queue_is_mq(q)) { - blk_mq_in_flight(q, part, inflight); - return; + return blk_mq_in_flight(q, part); } - inflight[0] = 0; + inflight = 0; for_each_possible_cpu(cpu) { - inflight[0] += local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[0]) + + inflight += local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[0]) + local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[1]); } - if ((int)inflight[0] < 0) - inflight[0] = 0; + if (inflight < 0) + inflight = 0; - if (part->partno) { - part = &part_to_disk(part)->part0; - inflight[1] = 0; - for_each_possible_cpu(cpu) { - inflight[1] += local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[0]) + - local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[1]); - } - if ((int)inflight[1] < 0) - inflight[1] = 0; - } + return (unsigned int)inflight; } void part_in_flight_rw(struct request_queue *q, struct hd_struct *part, @@ -1354,7 +1344,7 @@ static int diskstats_show(struct seq_file *seqf, void *v) struct disk_part_iter piter; struct hd_struct *hd; char buf[BDEVNAME_SIZE]; - unsigned int inflight[2]; + unsigned int inflight; /* if (&disk_to_dev(gp)->kobj.entry == block_class.devices.next) @@ -1366,7 +1356,7 @@ static int diskstats_show(struct seq_file *seqf, void *v) disk_part_iter_init(&piter, gp, DISK_PITER_INCL_EMPTY_PART0); while ((hd = disk_part_iter_next(&piter))) { - part_in_flight(gp->queue, hd, inflight); + inflight = part_in_flight(gp->queue, hd); seq_printf(seqf, "%4d %7d %s " "%lu %lu %lu %u " "%lu %lu %lu %u " @@ -1382,7 +1372,7 @@ static int diskstats_show(struct seq_file *seqf, void *v) part_stat_read(hd, merges[STAT_WRITE]), part_stat_read(hd, sectors[STAT_WRITE]), (unsigned int)part_stat_read_msecs(hd, STAT_WRITE), - inflight[0], + inflight, jiffies_to_msecs(part_stat_read(hd, io_ticks)), jiffies_to_msecs(part_stat_read(hd, time_in_queue)), part_stat_read(hd, ios[STAT_DISCARD]), diff --git a/block/partition-generic.c b/block/partition-generic.c index 42d6138ac876..8e596a8dff32 100644 --- a/block/partition-generic.c +++ b/block/partition-generic.c @@ -120,9 +120,9 @@ ssize_t part_stat_show(struct device *dev, { struct hd_struct *p = dev_to_part(dev); struct request_queue *q = part_to_disk(p)->queue; - unsigned int inflight[2]; + unsigned int inflight; - part_in_flight(q, p, inflight); + inflight = part_in_flight(q, p); return sprintf(buf, "%8lu %8lu %8llu %8u " "%8lu %8lu %8llu %8u " @@ -137,7 +137,7 @@ ssize_t part_stat_show(struct device *dev, part_stat_read(p, merges[STAT_WRITE]), (unsigned long long)part_stat_read(p, sectors[STAT_WRITE]), (unsigned int)part_stat_read_msecs(p, STAT_WRITE), - inflight[0], + inflight, jiffies_to_msecs(part_stat_read(p, io_ticks)), jiffies_to_msecs(part_stat_read(p, time_in_queue)), part_stat_read(p, ios[STAT_DISCARD]), diff --git a/include/linux/genhd.h b/include/linux/genhd.h index 9489dcb7cc27..669b7fe400d7 100644 --- a/include/linux/genhd.h +++ b/include/linux/genhd.h @@ -377,8 +377,7 @@ static inline void free_part_stats(struct hd_struct *part) #define part_stat_sub(gendiskp, field, subnd) \ part_stat_add(gendiskp, field, -subnd) -void part_in_flight(struct request_queue *q, struct hd_struct *part, - unsigned int inflight[2]); +unsigned int part_in_flight(struct request_queue *q, struct hd_struct *part); void part_in_flight_rw(struct request_queue *q, struct hd_struct *part, unsigned int inflight[2]); void part_dec_in_flight(struct request_queue *q, struct hd_struct *part, From patchwork Wed Dec 5 20:24:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 10714871 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A3B6E18A7 for ; Wed, 5 Dec 2018 20:24:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8CB2E2DA3B for ; Wed, 5 Dec 2018 20:24:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B3D62E415; Wed, 5 Dec 2018 20:24:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 18BA72DA3B for ; Wed, 5 Dec 2018 20:24:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728411AbeLEUYt (ORCPT ); Wed, 5 Dec 2018 15:24:49 -0500 Received: from mail-qk1-f196.google.com ([209.85.222.196]:38375 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727388AbeLEUYt (ORCPT ); Wed, 5 Dec 2018 15:24:49 -0500 Received: by mail-qk1-f196.google.com with SMTP id d19so12696542qkg.5 for ; Wed, 05 Dec 2018 12:24:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=/MmpQWfVqjIcIBzc1IhAmuNIPGv+IYi6voBAs2gkkkI=; b=oZMLDyEixH85J4sIUD1wmhel7oo/NXb5Ir16aUhweca/LwChRGYdLzEGPapBcHRtn5 /XOWqU+SY/qvRPa3BjhjnF3+XwPMW0thuYiuEe+u+KQy/57e/cna61l6HMs0Xx5ZB2pX Ju9B/ScvVwZq3ekMo6ASTFwrysQrAS93pp5LTET60sLJqTf3Pl/qzlxDlbYGGnoQPs76 W0iiGiSAfRcZtO3bRdBGCbfmXQFqIh7CCJU3RAVuHbI44XZ9FxIFZRZK0Mr332VVJbHy kItlSOk0SY7syd1hBjoXJzXMvfHnxGy6ikulGUvHrlkvSPzlF0sHlVJVYID6jTGi2mNN Rh5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=/MmpQWfVqjIcIBzc1IhAmuNIPGv+IYi6voBAs2gkkkI=; b=BhYMOkwV0F9fXtYPnuqke29QxZj8zL6ltKW2wakq09dxCimpsj0/TdhWThJhSOUmUT ZlYosCjYgupvnkCvyk/S4JgMwFw6KhmwST+vH1PpFlDIhxdxSQ9lgzhf4frRR9NFHXVv bskIm4MrOGiMLSXkXQ6mibPJ/7s5KmjnAFzZ/c4XBC409SRz1aLF+Ehgq9Ey3xwoPqWM I86nSrAfgBs5EFxRarui8Ke/+6sg6ExlXYLmJEINFJS0OztUtLE0Igia02WzE/G+SetN mai5lJcdjwB3EhzDVpt19V0saVvIoUUmjy/XTQngY7FrP2nQOybNPdWfptFU1FRC9Syh xCDw== X-Gm-Message-State: AA+aEWaRRQadKLXQVx1pwfblc9qc3Qe8AJ94KxJTC/2cfTSxfXyUgSyw XZIM6+UsB2nB/KcXMhwkUck= X-Google-Smtp-Source: AFSGD/XDRY6zz++uNJeWsIYIZGuAPXV86dEHzvxrI+U7lqyKaGr0GePKcLIMFRNxdhluTlRyUTbhfA== X-Received: by 2002:a37:9cd4:: with SMTP id f203mr24203387qke.155.1544041488238; Wed, 05 Dec 2018 12:24:48 -0800 (PST) Received: from localhost (pool-68-160-144-192.bstnma.fios.verizon.net. [68.160.144.192]) by smtp.gmail.com with ESMTPSA id n68sm7914741qte.66.2018.12.05.12.24.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 12:24:47 -0800 (PST) From: Mike Snitzer To: Jens Axboe Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Mikulas Patocka Subject: [PATCH v3 7/7] dm: remove the pending IO accounting Date: Wed, 5 Dec 2018 15:24:33 -0500 Message-Id: <20181205202433.95823-8-snitzer@redhat.com> X-Mailer: git-send-email 2.15.0 In-Reply-To: <20181205202433.95823-1-snitzer@redhat.com> References: <20181205202433.95823-1-snitzer@redhat.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Mikulas Patocka Remove the "pending" atomic counters, that duplicate block-core's in_flight counters, and update md_in_flight() to look at percpu in_flight counters. Signed-off-by: Mikulas Patocka Signed-off-by: Mike Snitzer --- drivers/md/dm-core.h | 2 -- drivers/md/dm.c | 34 +++++++++++++++------------------- 2 files changed, 15 insertions(+), 21 deletions(-) diff --git a/drivers/md/dm-core.h b/drivers/md/dm-core.h index 224d44503a06..6fe883fac471 100644 --- a/drivers/md/dm-core.h +++ b/drivers/md/dm-core.h @@ -65,7 +65,6 @@ struct mapped_device { */ struct work_struct work; wait_queue_head_t wait; - atomic_t pending[2]; spinlock_t deferred_lock; struct bio_list deferred; @@ -119,7 +118,6 @@ struct mapped_device { struct srcu_struct io_barrier; }; -int md_in_flight(struct mapped_device *md); void disable_write_same(struct mapped_device *md); void disable_write_zeroes(struct mapped_device *md); diff --git a/drivers/md/dm.c b/drivers/md/dm.c index a8ae7931bce7..ff6e5a5902f2 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -646,25 +646,30 @@ static void free_tio(struct dm_target_io *tio) bio_put(&tio->clone); } -int md_in_flight(struct mapped_device *md) +static bool md_in_flight(struct mapped_device *md) { - return atomic_read(&md->pending[READ]) + - atomic_read(&md->pending[WRITE]); + int cpu; + struct hd_struct *part = &dm_disk(md)->part0; + + for_each_possible_cpu(cpu) { + if (local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[0]) || + local_read(&per_cpu_ptr(part->dkstats, cpu)->in_flight[1])) + return true; + } + + return false; } static void start_io_acct(struct dm_io *io) { struct mapped_device *md = io->md; struct bio *bio = io->orig_bio; - int rw = bio_data_dir(bio); io->start_time = jiffies; generic_start_io_acct(md->queue, bio_op(bio), bio_sectors(bio), &dm_disk(md)->part0); - atomic_inc(&md->pending[rw]); - if (unlikely(dm_stats_used(&md->stats))) dm_stats_account_io(&md->stats, bio_data_dir(bio), bio->bi_iter.bi_sector, bio_sectors(bio), @@ -676,8 +681,6 @@ static void end_io_acct(struct dm_io *io) struct mapped_device *md = io->md; struct bio *bio = io->orig_bio; unsigned long duration = jiffies - io->start_time; - int pending; - int rw = bio_data_dir(bio); generic_end_io_acct(md->queue, bio_op(bio), &dm_disk(md)->part0, io->start_time); @@ -687,16 +690,11 @@ static void end_io_acct(struct dm_io *io) bio->bi_iter.bi_sector, bio_sectors(bio), true, duration, &io->stats_aux); - /* - * After this is decremented the bio must not be touched if it is - * a flush. - */ - pending = atomic_dec_return(&md->pending[rw]); - pending += atomic_read(&md->pending[rw^0x1]); - /* nudge anyone waiting on suspend queue */ - if (!pending) - wake_up(&md->wait); + if (unlikely(waitqueue_active(&md->wait))) { + if (!md_in_flight(md)) + wake_up(&md->wait); + } } /* @@ -1904,8 +1902,6 @@ static struct mapped_device *alloc_dev(int minor) if (!md->disk) goto bad; - atomic_set(&md->pending[0], 0); - atomic_set(&md->pending[1], 0); init_waitqueue_head(&md->wait); INIT_WORK(&md->work, dm_wq_work); init_waitqueue_head(&md->eventq);