From patchwork Wed May 2 17:34:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Omar Sandoval X-Patchwork-Id: 10376127 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B44FA603B4 for ; Wed, 2 May 2018 17:34:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A5A5728BA7 for ; Wed, 2 May 2018 17:34:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9A5C828F29; Wed, 2 May 2018 17:34:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 01BA528BA7 for ; Wed, 2 May 2018 17:34:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751121AbeEBRep (ORCPT ); Wed, 2 May 2018 13:34:45 -0400 Received: from mail-pf0-f193.google.com ([209.85.192.193]:40869 "EHLO mail-pf0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751076AbeEBReo (ORCPT ); Wed, 2 May 2018 13:34:44 -0400 Received: by mail-pf0-f193.google.com with SMTP id f189so12374772pfa.7 for ; Wed, 02 May 2018 10:34:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=0FWhUYDXlBydstHcRGgGr/Pzt7eyuHenILh6RMoe2Ng=; b=QLG+cam6REfCuCwzoMnHAP9uHgoo4GiDl43J/9UXPDAoHcRBZ9Dgls3UL3WsSIiAHa /YpuZ8cKdb4dBgcZBfM2z9I/pbjbByEEHm9j9B4FW6OSsynLCiuSi0vOUtP0AdYv2HtU RA+g1l+FR1PMgsBcOnLlvNdIFvQncvx/N2eMInKyiKLbVIGj6jNi4Gh6ZwN3OExBZ4FF rNBi09XSZZ0DkNJr2PNDfmaZKixIwA/q/EPXJOsNQvI2xOW8+6hdjMqf7yuw6QYfEKIi x0dOlrNNhyH6RCkTGBuT7Ur1W8+HtgUXznL89cF8UJ9O2fqy9wySJD5EySnVKduakOSy CPJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=0FWhUYDXlBydstHcRGgGr/Pzt7eyuHenILh6RMoe2Ng=; b=una6NG/jhTK6n7sfZKM+yGcT1rA2y6fITNWzgP7dvwWOPZcGDRdgji2fqX5fqkkgvk 8KdYegLphx/dYOjjxW4pi0AlwLOPtk2S4IT0+wiKYmB83wxD4OKUokjgu8E1e4Bk6Itm ognGVUifunIZRMfmGf24CvWHM6jREc+N1LNhEUGuh2S0l4d7kXDjoSoKSMjXW9t3LsA5 rVvf913qgDho+NsDhO3i8PD5Dj5+oGDK/LwARcuwiGVXFSUsxIfmaP+YskWZXgi+PWHP jdNAKHH/OdCmgjAYyMvnQZZlxqhBh/vbtISu310RmhDosjOHNph4c5wZRQFBO+0PL8kC cc7A== X-Gm-Message-State: ALQs6tAUVv4OXd8kitTmMmkaJG4u7vqG2LP9zihZqjFqo9glfpqMie0f BkJLB13dlAjF4pmUpdTCMjzgwDhXsgw= X-Google-Smtp-Source: AB8JxZrSp02RdgKw+GioKCFZYObsl2fADv6ZgNC/wnvBwOq0r+GAI5J6UzgmilND1Vve3Ekr2DTWQA== X-Received: by 10.98.217.5 with SMTP id s5mr20420949pfg.20.1525282483221; Wed, 02 May 2018 10:34:43 -0700 (PDT) Received: from vader.thefacebook.com ([2620:10d:c090:180::1:24e1]) by smtp.gmail.com with ESMTPSA id w184-v6sm17757388pgb.20.2018.05.02.10.34.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 May 2018 10:34:42 -0700 (PDT) From: Omar Sandoval To: linux-block@vger.kernel.org Cc: Jens Axboe , kernel-team@fb.com Subject: [PATCH 3/7] block: replace bio->bi_issue_stat with u64 Date: Wed, 2 May 2018 10:34:22 -0700 Message-Id: X-Mailer: git-send-email 2.17.0 In-Reply-To: References: In-Reply-To: References: Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Omar Sandoval struct blk_issue_stat is going away, and bio->bi_issue_stat doesn't even use the blk-stats interface, so we can provide a separate implementation private to blk-throtl. Signed-off-by: Omar Sandoval --- block/blk-throttle.c | 50 ++++++++++++++++++++++++++++++--------- include/linux/blk_types.h | 2 +- 2 files changed, 40 insertions(+), 12 deletions(-) diff --git a/block/blk-throttle.c b/block/blk-throttle.c index c5a131673733..e2957a93ed11 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -36,8 +36,6 @@ static int throtl_quantum = 32; */ #define LATENCY_FILTERED_HD (1000L) /* 1ms */ -#define SKIP_LATENCY (((u64)1) << BLK_STAT_RES_SHIFT) - static struct blkcg_policy blkcg_policy_throtl; /* A workqueue to queue throttle related work */ @@ -2048,6 +2046,37 @@ static void blk_throtl_update_idletime(struct throtl_grp *tg) } #ifdef CONFIG_BLK_DEV_THROTTLING_LOW +/* + * bio->bi_throtl: + * high bit is SKIP_LATENCY flag + * next 12 bits are original size, in sectors + * rest of the bits (51) are issue time, in nanoseconds + */ +#define BI_THROTL_SKIP_LATENCY (1ULL << 63) +#define BI_THROTL_SIZE_BITS 12 +#define BI_THROTL_SIZE_MASK ((1ULL << BI_THROTL_SIZE_BITS) - 1) +#define BI_THROTL_SIZE_SHIFT (63 - BI_THROTL_SIZE_BITS) +#define BI_THROTL_TIME_MASK ((1ULL << BI_THROTL_SIZE_SHIFT) - 1) + +static void throtl_set_issue(struct bio *bio) +{ + u64 time, size; + + time = ktime_get_ns() & BI_THROTL_TIME_MASK; + size = bio_sectors(bio) & BI_THROTL_SIZE_MASK; + bio->bi_throtl = time | (size << BI_THROTL_SIZE_SHIFT); +} + +static u64 throtl_issue_time(struct bio *bio) +{ + return bio->bi_throtl & BI_THROTL_TIME_MASK; +} + +static sector_t throtl_issue_size(struct bio *bio) +{ + return (bio->bi_throtl >> BI_THROTL_SIZE_SHIFT) & BI_THROTL_SIZE_MASK; +} + static void throtl_update_latency_buckets(struct throtl_data *td) { struct avg_latency_bucket avg_latency[2][LATENCY_BUCKET_SIZE]; @@ -2139,7 +2168,7 @@ static void blk_throtl_assoc_bio(struct throtl_grp *tg, struct bio *bio) bio->bi_cg_private = tg; blkg_get(tg_to_blkg(tg)); } - blk_stat_set_issue(&bio->bi_issue_stat, bio_sectors(bio)); + throtl_set_issue(bio); #endif } @@ -2251,7 +2280,7 @@ bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg, #ifdef CONFIG_BLK_DEV_THROTTLING_LOW if (throttled || !td->track_bio_latency) - bio->bi_issue_stat.stat |= SKIP_LATENCY; + bio->bi_throtl |= BI_THROTL_SKIP_LATENCY; #endif return throttled; } @@ -2302,8 +2331,8 @@ void blk_throtl_bio_endio(struct bio *bio) finish_time_ns = ktime_get_ns(); tg->last_finish_time = finish_time_ns >> 10; - start_time = blk_stat_time(&bio->bi_issue_stat) >> 10; - finish_time = __blk_stat_time(finish_time_ns) >> 10; + start_time = throtl_issue_time(bio) >> 10; + finish_time = (finish_time_ns & BI_THROTL_TIME_MASK) >> 10; if (!start_time || finish_time <= start_time) { blkg_put(tg_to_blkg(tg)); return; @@ -2311,16 +2340,15 @@ void blk_throtl_bio_endio(struct bio *bio) lat = finish_time - start_time; /* this is only for bio based driver */ - if (!(bio->bi_issue_stat.stat & SKIP_LATENCY)) - throtl_track_latency(tg->td, blk_stat_size(&bio->bi_issue_stat), - bio_op(bio), lat); + if (!(bio->bi_throtl & BI_THROTL_SKIP_LATENCY)) + throtl_track_latency(tg->td, throtl_issue_size(bio), + bio_op(bio), lat); if (tg->latency_target && lat >= tg->td->filtered_latency) { int bucket; unsigned int threshold; - bucket = request_bucket_index( - blk_stat_size(&bio->bi_issue_stat)); + bucket = request_bucket_index(throtl_issue_size(bio)); threshold = tg->td->avg_buckets[rw][bucket].latency + tg->latency_target; if (lat > threshold) diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 17b18b91ebac..3a263538d315 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -138,7 +138,7 @@ struct bio { struct cgroup_subsys_state *bi_css; #ifdef CONFIG_BLK_DEV_THROTTLING_LOW void *bi_cg_private; - struct blk_issue_stat bi_issue_stat; + u64 bi_throtl; #endif #endif union {