From patchwork Mon Jul 5 16:23:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 12359423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 773C4C07E99 for ; Mon, 5 Jul 2021 16:23:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 266DA6135D for ; Mon, 5 Jul 2021 16:23:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 266DA6135D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4D22D6B0089; Mon, 5 Jul 2021 12:23:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43E9F6B0096; Mon, 5 Jul 2021 12:23:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D66886B0099; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0042.hostedemail.com [216.40.44.42]) by kanga.kvack.org (Postfix) with ESMTP id 906F66B008A for ; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1B2B821963 for ; Mon, 5 Jul 2021 16:23:32 +0000 (UTC) X-FDA: 78329054664.31.A789484 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf04.hostedemail.com (Postfix) with ESMTP id 8436C50000AC for ; Mon, 5 Jul 2021 16:23:31 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 8152022618; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0OFyCHiVncRr2KeW7uXE977wDdgye+jCGc9PldO8qwQ=; b=ft78QLARSsp7Vmy3WCkXXEDpERjxsN8yW5PyXMpLx7+BDuQ2fSNNQVBG4t9RNSHcp9shn0 N6c2X3vsV2T5BgVodcXuO4Ua/5qKUmM8Dj8OsYMwgebiMQQpEWx/NQlgyKQ+CCWZJVT5Ca 7dFO1+o/LdmE090/diP/W+e4qgrAVTU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0OFyCHiVncRr2KeW7uXE977wDdgye+jCGc9PldO8qwQ=; b=2iUOKLeEB8kRjozv0hedR3Waoyb5TAGl2/zIZi1/nMXM9mkOrUL+29l9klD0ByPFUBYUA9 W/6vl62jYF0pOBAA== Received: from quack2.suse.cz (unknown [10.163.43.118]) by relay2.suse.de (Postfix) with ESMTP id 6EE76A3BA5; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 3FB091E053D; Mon, 5 Jul 2021 18:23:28 +0200 (CEST) From: Jan Kara To: Andrew Morton Cc: , Michael Stapelberg , , Jan Kara Subject: [PATCH 1/5] writeback: Track number of inodes under writeback Date: Mon, 5 Jul 2021 18:23:15 +0200 Message-Id: <20210705162328.28366-1-jack@suse.cz> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210705161610.19406-1-jack@suse.cz> References: <20210705161610.19406-1-jack@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3595; h=from:subject; bh=k6CGYCogJu3IEkHdSZkpTAfQ0gTbB+GaRTJkEGSCe18=; b=owEBbQGS/pANAwAIAZydqgc/ZEDZAcsmYgBg4zHzvi2Nt0j8x6Oj7z8i0T3FfqmJBTHlLuvSgFVp H5zPdcuJATMEAAEIAB0WIQSrWdEr1p4yirVVKBycnaoHP2RA2QUCYOMx8wAKCRCcnaoHP2RA2cGQCA CFQMAkHrC6pHpDME2hNtzfbzEklvZFsN0MmKdtsaIEyCccojUNYmjXKLdb+Hc640mU56Uw2AOh48Da yopo1mp8bbdxwl1s2q7ZEtplNZUfhnGrvCAS92Y69JBKhoj8ywvV32CcROUJKJtBKmJTB4HCVz5y8o hZGQ0NmH4Wy0mFma+b1K5uXQLphPdujMBUAsqoYvMS97n+c8ZDA1y5SxrTNPZLDk/lMpW1cBsH+7f6 2EcoL4AONrUpeI4qR0fVIreKmFORB58Tg/MHWQILXZq6GuR65aNE7iLo2IFf2Zlz+t5n5qeDGWHjxM iPodotr/q2LpgMIMepAZZLQ26urJaO X-Developer-Key: i=jack@suse.cz; a=openpgp; fpr=93C6099A142276A28BBE35D815BC833443038D8C Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ft78QLAR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=2iUOKLeE; spf=pass (imf04.hostedemail.com: domain of jack@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none X-Stat-Signature: z3yq71dybe8t8hyxtsuzyufurgw4fjmt X-Rspamd-Queue-Id: 8436C50000AC X-Rspamd-Server: rspam06 X-HE-Tag: 1625502211-434408 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Track number of inodes under writeback for each bdi_writeback structure. We will use this to decide whether wb does any IO and so we can estimate its writeback throughput. In principle we could use number of pages under writeback (WB_WRITEBACK counter) for this however normal percpu counter reads are too inaccurate for our purposes and summing the counter is too expensive. Signed-off-by: Jan Kara --- fs/fs-writeback.c | 5 +++++ include/linux/backing-dev-defs.h | 1 + mm/backing-dev.c | 1 + mm/page-writeback.c | 22 ++++++++++++++++++++-- 4 files changed, 27 insertions(+), 2 deletions(-) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index e91980f49388..475681362b1c 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -416,6 +416,11 @@ static void inode_switch_wbs_work_fn(struct work_struct *work) inc_wb_stat(new_wb, WB_WRITEBACK); } + if (mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK)) { + atomic_dec(&old_wb->writeback_inodes); + atomic_inc(&new_wb->writeback_inodes); + } + wb_get(new_wb); /* diff --git a/include/linux/backing-dev-defs.h b/include/linux/backing-dev-defs.h index fff9367a6348..148d889f2f7f 100644 --- a/include/linux/backing-dev-defs.h +++ b/include/linux/backing-dev-defs.h @@ -116,6 +116,7 @@ struct bdi_writeback { struct list_head b_dirty_time; /* time stamps are dirty */ spinlock_t list_lock; /* protects the b_* lists */ + atomic_t writeback_inodes; /* number of inodes under writeback */ struct percpu_counter stat[NR_WB_STAT_ITEMS]; unsigned long congested; /* WB_[a]sync_congested flags */ diff --git a/mm/backing-dev.c b/mm/backing-dev.c index 576220acd686..342394ef1e02 100644 --- a/mm/backing-dev.c +++ b/mm/backing-dev.c @@ -293,6 +293,7 @@ static int wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi, INIT_LIST_HEAD(&wb->b_dirty_time); spin_lock_init(&wb->list_lock); + atomic_set(&wb->writeback_inodes, 0); wb->bw_time_stamp = jiffies; wb->balanced_dirty_ratelimit = INIT_BW; wb->dirty_ratelimit = INIT_BW; diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 0062d5c57d41..1560f6626a3b 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2719,6 +2719,16 @@ int clear_page_dirty_for_io(struct page *page) } EXPORT_SYMBOL(clear_page_dirty_for_io); +static void wb_inode_writeback_start(struct bdi_writeback *wb) +{ + atomic_inc(&wb->writeback_inodes); +} + +static void wb_inode_writeback_end(struct bdi_writeback *wb) +{ + atomic_dec(&wb->writeback_inodes); +} + int test_clear_page_writeback(struct page *page) { struct address_space *mapping = page_mapping(page); @@ -2740,6 +2750,9 @@ int test_clear_page_writeback(struct page *page) dec_wb_stat(wb, WB_WRITEBACK); __wb_writeout_inc(wb); + if (!mapping_tagged(mapping, + PAGECACHE_TAG_WRITEBACK)) + wb_inode_writeback_end(wb); } } @@ -2782,8 +2795,13 @@ int __test_set_page_writeback(struct page *page, bool keep_write) PAGECACHE_TAG_WRITEBACK); xas_set_mark(&xas, PAGECACHE_TAG_WRITEBACK); - if (bdi->capabilities & BDI_CAP_WRITEBACK_ACCT) - inc_wb_stat(inode_to_wb(inode), WB_WRITEBACK); + if (bdi->capabilities & BDI_CAP_WRITEBACK_ACCT) { + struct bdi_writeback *wb = inode_to_wb(inode); + + inc_wb_stat(wb, WB_WRITEBACK); + if (!on_wblist) + wb_inode_writeback_start(wb); + } /* * We can come through here when swapping anonymous From patchwork Mon Jul 5 16:23:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 12359421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B62B3C07E9E for ; Mon, 5 Jul 2021 16:23:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 47767613C1 for ; Mon, 5 Jul 2021 16:23:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 47767613C1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 224706B0095; Mon, 5 Jul 2021 12:23:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 10A9E6B008A; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2E676B0096; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id 974596B0093 for ; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1DA50180178B4 for ; Mon, 5 Jul 2021 16:23:32 +0000 (UTC) X-FDA: 78329054664.31.A943613 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf06.hostedemail.com (Postfix) with ESMTP id 7DB6F801AB3B for ; Mon, 5 Jul 2021 16:23:31 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 8C0FE1FEC8; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mvs7mf+Do9fq5KKu7Y4VKMmqNqBaOkFIvFYcZ+dyeTw=; b=t5nh1y4PdZdOgEES+RC0K9YHrTKnzLguAA8brybwdizcapGx3eypjUSpi3b7Daxl2VrHq7 YjrbXzlY6KxJIGgVsEszclGIIoAYDZylv1a7N21S0rIH2dES2etwx9JWVReJt1HRu5YQ3y fmWnMxhbnBfJg/IzttAjKcjWh+IyT1U= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mvs7mf+Do9fq5KKu7Y4VKMmqNqBaOkFIvFYcZ+dyeTw=; b=zy94W6yj2cCz189yAa7Kwciyb7jyFLKKD1CbBO7w/GCXAZcTdtLfjgmuuDWicOhrpfO34/ haXXvhRXLw5xbiAA== Received: from quack2.suse.cz (unknown [10.163.43.118]) by relay2.suse.de (Postfix) with ESMTP id 76F87A3BA9; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 425AD1F2CAF; Mon, 5 Jul 2021 18:23:28 +0200 (CEST) From: Jan Kara To: Andrew Morton Cc: , Michael Stapelberg , , Jan Kara Subject: [PATCH 2/5] writeback: Reliably update bandwidth estimation Date: Mon, 5 Jul 2021 18:23:16 +0200 Message-Id: <20210705162328.28366-2-jack@suse.cz> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210705161610.19406-1-jack@suse.cz> References: <20210705161610.19406-1-jack@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6441; h=from:subject; bh=MAqbSRQHTYqrFQe1Bt2BYpSkOFmuyuvyfoYLeKQXwFE=; b=owEBbQGS/pANAwAIAZydqgc/ZEDZAcsmYgBg4zH0CtLlTonpKlFvZqtvlceodXUXDwY/FxUaRrF3 CxHXElGJATMEAAEIAB0WIQSrWdEr1p4yirVVKBycnaoHP2RA2QUCYOMx9AAKCRCcnaoHP2RA2cMJB/ 0ZKSRZme8f55e3Slb0tFjoZ+bB+/vR7VAcq7UVoQ+arRPiCEWqMrswaoZmyNi6aOC3uKspNkLXyBD7 vlAMbeb6L15yiNSFRFVZmxPQDYK7IjD2OkodPBmfIli6PYUIh67G6X3NWx0xPRRLLnoVhlvgFEq+Mz gbpwd3nxil+0DmSuf2iK9T2SpQTwFby4oM3XR3FkxTn6bJ4pSDv6uYRcEtczpBDE73vdHq9Ykp4knd /b8+GNs4IGipiugg19qPanaBTOSZ8pJbWFNkTkzsVtNvM5us8WKEa6Q43r4/2MUAvje+GZTfIFER+g Ney2M63cJCRGSwx4lfgEWRx+CSe6pg X-Developer-Key: i=jack@suse.cz; a=openpgp; fpr=93C6099A142276A28BBE35D815BC833443038D8C Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=t5nh1y4P; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=zy94W6yj; dmarc=none; spf=pass (imf06.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: bjjx8aepqncpym18w3zymgi1bp6tpo6t X-Rspamd-Queue-Id: 7DB6F801AB3B X-HE-Tag: 1625502211-37173 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently we trigger writeback bandwidth estimation from balance_dirty_pages() and from wb_writeback(). However neither of these need to trigger when the system is relatively idle and writeback is triggered e.g. from fsync(2). Make sure writeback estimates happen reliably by triggering them from do_writepages(). Signed-off-by: Jan Kara --- fs/fs-writeback.c | 3 --- include/linux/backing-dev.h | 19 ++++++++++++++++++ include/linux/writeback.h | 1 - mm/page-writeback.c | 39 +++++++++++++++++++++++++------------ 4 files changed, 46 insertions(+), 16 deletions(-) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 475681362b1c..b53d1513e510 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -1858,7 +1858,6 @@ static long writeback_inodes_wb(struct bdi_writeback *wb, long nr_pages, static long wb_writeback(struct bdi_writeback *wb, struct wb_writeback_work *work) { - unsigned long wb_start = jiffies; long nr_pages = work->nr_pages; unsigned long dirtied_before = jiffies; struct inode *inode; @@ -1912,8 +1911,6 @@ static long wb_writeback(struct bdi_writeback *wb, progress = __writeback_inodes_wb(wb, work); trace_writeback_written(wb, work); - wb_update_bandwidth(wb, wb_start); - /* * Did we write something? Try for more * diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h index 44df4fcef65c..a5d7d625dcc6 100644 --- a/include/linux/backing-dev.h +++ b/include/linux/backing-dev.h @@ -288,6 +288,17 @@ static inline struct bdi_writeback *inode_to_wb(const struct inode *inode) return inode->i_wb; } +static inline struct bdi_writeback *inode_to_wb_wbc( + struct inode *inode, + struct writeback_control *wbc) +{ + /* + * If wbc does not have inode attached, it means cgroup writeback was + * disabled when wbc started. Just use the default wb in that case. + */ + return wbc->wb ? wbc->wb : &inode_to_bdi(inode)->wb; +} + /** * unlocked_inode_to_wb_begin - begin unlocked inode wb access transaction * @inode: target inode @@ -366,6 +377,14 @@ static inline struct bdi_writeback *inode_to_wb(struct inode *inode) return &inode_to_bdi(inode)->wb; } +static inline struct bdi_writeback *inode_to_wb_wbc( + struct inode *inode, + struct writeback_control *wbc) +{ + return inode_to_wb(inode); +} + + static inline struct bdi_writeback * unlocked_inode_to_wb_begin(struct inode *inode, struct wb_lock_cookie *cookie) { diff --git a/include/linux/writeback.h b/include/linux/writeback.h index 8e5c5bb16e2d..47cd732e012e 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -379,7 +379,6 @@ int dirty_writeback_centisecs_handler(struct ctl_table *table, int write, void global_dirty_limits(unsigned long *pbackground, unsigned long *pdirty); unsigned long wb_calc_thresh(struct bdi_writeback *wb, unsigned long thresh); -void wb_update_bandwidth(struct bdi_writeback *wb, unsigned long start_time); void balance_dirty_pages_ratelimited(struct address_space *mapping); bool wb_over_bg_thresh(struct bdi_writeback *wb); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 1560f6626a3b..1fecf8ebadb0 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -1338,7 +1338,6 @@ static void wb_update_dirty_ratelimit(struct dirty_throttle_control *dtc, static void __wb_update_bandwidth(struct dirty_throttle_control *gdtc, struct dirty_throttle_control *mdtc, - unsigned long start_time, bool update_ratelimit) { struct bdi_writeback *wb = gdtc->wb; @@ -1358,13 +1357,6 @@ static void __wb_update_bandwidth(struct dirty_throttle_control *gdtc, dirtied = percpu_counter_read(&wb->stat[WB_DIRTIED]); written = percpu_counter_read(&wb->stat[WB_WRITTEN]); - /* - * Skip quiet periods when disk bandwidth is under-utilized. - * (at least 1s idle time between two flusher runs) - */ - if (elapsed > HZ && time_before(wb->bw_time_stamp, start_time)) - goto snapshot; - if (update_ratelimit) { domain_update_bandwidth(gdtc, now); wb_update_dirty_ratelimit(gdtc, dirtied, elapsed); @@ -1380,17 +1372,36 @@ static void __wb_update_bandwidth(struct dirty_throttle_control *gdtc, } wb_update_write_bandwidth(wb, elapsed, written); -snapshot: wb->dirtied_stamp = dirtied; wb->written_stamp = written; wb->bw_time_stamp = now; } -void wb_update_bandwidth(struct bdi_writeback *wb, unsigned long start_time) +static void wb_update_bandwidth(struct bdi_writeback *wb) { struct dirty_throttle_control gdtc = { GDTC_INIT(wb) }; - __wb_update_bandwidth(&gdtc, NULL, start_time, false); + spin_lock(&wb->list_lock); + __wb_update_bandwidth(&gdtc, NULL, false); + spin_unlock(&wb->list_lock); +} + +/* Interval after which we consider wb idle and don't estimate bandwidth */ +#define WB_BANDWIDTH_IDLE_JIF (HZ) + +static void wb_bandwidth_estimate_start(struct bdi_writeback *wb) +{ + unsigned long now = jiffies; + unsigned long elapsed = now - READ_ONCE(wb->bw_time_stamp); + + if (elapsed > WB_BANDWIDTH_IDLE_JIF && + !atomic_read(&wb->writeback_inodes)) { + spin_lock(&wb->list_lock); + wb->dirtied_stamp = wb_stat(wb, WB_DIRTIED); + wb->written_stamp = wb_stat(wb, WB_WRITTEN); + wb->bw_time_stamp = now; + spin_unlock(&wb->list_lock); + } } /* @@ -1719,7 +1730,7 @@ static void balance_dirty_pages(struct bdi_writeback *wb, if (time_is_before_jiffies(wb->bw_time_stamp + BANDWIDTH_INTERVAL)) { spin_lock(&wb->list_lock); - __wb_update_bandwidth(gdtc, mdtc, start_time, true); + __wb_update_bandwidth(gdtc, mdtc, true); spin_unlock(&wb->list_lock); } @@ -2344,9 +2355,12 @@ EXPORT_SYMBOL(generic_writepages); int do_writepages(struct address_space *mapping, struct writeback_control *wbc) { int ret; + struct bdi_writeback *wb; if (wbc->nr_to_write <= 0) return 0; + wb = inode_to_wb_wbc(mapping->host, wbc); + wb_bandwidth_estimate_start(wb); while (1) { if (mapping->a_ops->writepages) ret = mapping->a_ops->writepages(mapping, wbc); @@ -2357,6 +2371,7 @@ int do_writepages(struct address_space *mapping, struct writeback_control *wbc) cond_resched(); congestion_wait(BLK_RW_ASYNC, HZ/50); } + wb_update_bandwidth(wb); return ret; } From patchwork Mon Jul 5 16:23:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 12359427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ABF5C07E99 for ; Mon, 5 Jul 2021 16:23:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C08C3613B1 for ; Mon, 5 Jul 2021 16:23:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C08C3613B1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A0D9C6B008C; Mon, 5 Jul 2021 12:23:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96F1C6B0096; Mon, 5 Jul 2021 12:23:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 158E96B008C; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0105.hostedemail.com [216.40.44.105]) by kanga.kvack.org (Postfix) with ESMTP id 95DB76B0092 for ; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 197F521941 for ; Mon, 5 Jul 2021 16:23:32 +0000 (UTC) X-FDA: 78329054664.27.4B4ABF2 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf03.hostedemail.com (Postfix) with ESMTP id 7BBE930000BC for ; Mon, 5 Jul 2021 16:23:31 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 813B61FEB7; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+b98YqRK+9Ac0JbTnwMPdtBA76rUD2t6b5cSzamRWx8=; b=zH/nFVVXUlozatZfDdmeXi2zntrIN1hqZd2kHvDHkjBOuYNlo5tZI0kXN1DTtgneW8rrvj Hfhv5uhO0IzoRJSdS+LfTdb9xQhTuCpUmvomtD/rQ0foZdrQ8IPPxmodTpiWRoyM+3b54F SH+bh3Jc4MIT5Ec1xLc5IdeAeYY7TYc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+b98YqRK+9Ac0JbTnwMPdtBA76rUD2t6b5cSzamRWx8=; b=2YmTTMuo6f9Otcc71BzxdOFpJ55jzExThVuD7pahRzdJV26lkcDRo9M9wqxjfb5v4GZJd2 WEGlYJiQySv/7aDA== Received: from quack2.suse.cz (unknown [10.163.43.118]) by relay2.suse.de (Postfix) with ESMTP id 74298A3BA8; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 4688C1F2CBE; Mon, 5 Jul 2021 18:23:28 +0200 (CEST) From: Jan Kara To: Andrew Morton Cc: , Michael Stapelberg , , Jan Kara Subject: [PATCH 3/5] writeback: Fix bandwidth estimate for spiky workload Date: Mon, 5 Jul 2021 18:23:17 +0200 Message-Id: <20210705162328.28366-3-jack@suse.cz> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210705161610.19406-1-jack@suse.cz> References: <20210705161610.19406-1-jack@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6394; h=from:subject; bh=cwzknGok4CQVfoRflzeLzyEd6Wpkt+t4xXEuAWf5TYU=; b=owEBbQGS/pANAwAIAZydqgc/ZEDZAcsmYgBg4zH1HdOWNBXDsim6a4WRVqlLbnqToymTmQcbpjdi REUOKhGJATMEAAEIAB0WIQSrWdEr1p4yirVVKBycnaoHP2RA2QUCYOMx9QAKCRCcnaoHP2RA2WpfCA CnQwiCTudal0/H7QmEavro3MRix5eU/3KiupMX6ItgTJZTnK3vsWKUFlql0YoxC0hzyvYvn37wsdkS 9QC36pXnctkor1BYDIZ5d+KcrCUgSsGj8/tFtGRtR/sozpNsEvFGPXX9luTQfaCoMG9KNvj+XTdVFb uw5GqnsqK76lV2+IDq2kmrQLTtijYYLcF/VYHcVk2UN2K0yCClakm6yIoeQYfIV1y7alQl1GLVVfhC NjO+wDessJiW7pmBh+CRjBeIlIGHbM+vtCYrKdcaOw0yq9sAzqEr2t6pVnn12lDmhGqBuMy44PcKJJ Dn1VKQb4x8hdp8xov9n1Coade+LvH7 X-Developer-Key: i=jack@suse.cz; a=openpgp; fpr=93C6099A142276A28BBE35D815BC833443038D8C X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 7BBE930000BC Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="zH/nFVVX"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=2YmTTMuo; spf=pass (imf03.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none X-Stat-Signature: gqtxiwnwiaoz36hnaxs8rfw4u8q1eatb X-HE-Tag: 1625502211-716791 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Michael Stapelberg has reported that for workload with short big spikes of writes (GCC linker seem to trigger this frequently) the write throughput is heavily underestimated and tends to steadily sink until it reaches zero. This has rather bad impact on writeback throttling (causing stalls). The problem is that writeback throughput estimate gets updated at most once per 200 ms. One update happens early after we submit pages for writeback (at that point writeout of only small fraction of pages is completed and thus observed throughput is tiny). Next update happens only during the next write spike (updates happen only from inode writeback and dirty throttling code) and if that is more than 1s after previous spike, we decide system was idle and just ignore whatever was written until this moment. Fix the problem by making sure writeback throughput estimate is also updated shortly after writeback completes to get reasonable estimate of throughput for spiky workloads. Link: https://lore.kernel.org/lkml/20210617095309.3542373-1-stapelberg+linux@google.com Reported-by: Michael Stapelberg Signed-off-by: Jan Kara --- include/linux/backing-dev-defs.h | 1 + include/linux/writeback.h | 1 + mm/backing-dev.c | 10 ++++++++++ mm/page-writeback.c | 32 ++++++++++++++++---------------- 4 files changed, 28 insertions(+), 16 deletions(-) diff --git a/include/linux/backing-dev-defs.h b/include/linux/backing-dev-defs.h index 148d889f2f7f..57395f7bb192 100644 --- a/include/linux/backing-dev-defs.h +++ b/include/linux/backing-dev-defs.h @@ -143,6 +143,7 @@ struct bdi_writeback { spinlock_t work_lock; /* protects work_list & dwork scheduling */ struct list_head work_list; struct delayed_work dwork; /* work item used for writeback */ + struct delayed_work bw_dwork; /* work item used for bandwidth estimate */ unsigned long dirty_sleep; /* last wait */ diff --git a/include/linux/writeback.h b/include/linux/writeback.h index 47cd732e012e..a45e09ed0711 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -379,6 +379,7 @@ int dirty_writeback_centisecs_handler(struct ctl_table *table, int write, void global_dirty_limits(unsigned long *pbackground, unsigned long *pdirty); unsigned long wb_calc_thresh(struct bdi_writeback *wb, unsigned long thresh); +void wb_update_bandwidth(struct bdi_writeback *wb); void balance_dirty_pages_ratelimited(struct address_space *mapping); bool wb_over_bg_thresh(struct bdi_writeback *wb); diff --git a/mm/backing-dev.c b/mm/backing-dev.c index 342394ef1e02..9baa59d68110 100644 --- a/mm/backing-dev.c +++ b/mm/backing-dev.c @@ -271,6 +271,14 @@ void wb_wakeup_delayed(struct bdi_writeback *wb) spin_unlock_bh(&wb->work_lock); } +static void wb_update_bandwidth_workfn(struct work_struct *work) +{ + struct bdi_writeback *wb = container_of(to_delayed_work(work), + struct bdi_writeback, bw_dwork); + + wb_update_bandwidth(wb); +} + /* * Initial write bandwidth: 100 MB/s */ @@ -303,6 +311,7 @@ static int wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi, spin_lock_init(&wb->work_lock); INIT_LIST_HEAD(&wb->work_list); INIT_DELAYED_WORK(&wb->dwork, wb_workfn); + INIT_DELAYED_WORK(&wb->bw_dwork, wb_update_bandwidth_workfn); wb->dirty_sleep = jiffies; err = fprop_local_init_percpu(&wb->completions, gfp); @@ -351,6 +360,7 @@ static void wb_shutdown(struct bdi_writeback *wb) mod_delayed_work(bdi_wq, &wb->dwork, 0); flush_delayed_work(&wb->dwork); WARN_ON(!list_empty(&wb->work_list)); + flush_delayed_work(&wb->bw_dwork); } static void wb_exit(struct bdi_writeback *wb) diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 1fecf8ebadb0..6a99ddca95c0 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -1346,14 +1346,7 @@ static void __wb_update_bandwidth(struct dirty_throttle_control *gdtc, unsigned long dirtied; unsigned long written; - lockdep_assert_held(&wb->list_lock); - - /* - * rate-limit, only update once every 200ms. - */ - if (elapsed < BANDWIDTH_INTERVAL) - return; - + spin_lock(&wb->list_lock); dirtied = percpu_counter_read(&wb->stat[WB_DIRTIED]); written = percpu_counter_read(&wb->stat[WB_WRITTEN]); @@ -1375,15 +1368,14 @@ static void __wb_update_bandwidth(struct dirty_throttle_control *gdtc, wb->dirtied_stamp = dirtied; wb->written_stamp = written; wb->bw_time_stamp = now; + spin_unlock(&wb->list_lock); } -static void wb_update_bandwidth(struct bdi_writeback *wb) +void wb_update_bandwidth(struct bdi_writeback *wb) { struct dirty_throttle_control gdtc = { GDTC_INIT(wb) }; - spin_lock(&wb->list_lock); __wb_update_bandwidth(&gdtc, NULL, false); - spin_unlock(&wb->list_lock); } /* Interval after which we consider wb idle and don't estimate bandwidth */ @@ -1728,11 +1720,8 @@ static void balance_dirty_pages(struct bdi_writeback *wb, wb->dirty_exceeded = 1; if (time_is_before_jiffies(wb->bw_time_stamp + - BANDWIDTH_INTERVAL)) { - spin_lock(&wb->list_lock); + BANDWIDTH_INTERVAL)) __wb_update_bandwidth(gdtc, mdtc, true); - spin_unlock(&wb->list_lock); - } /* throttle according to the chosen dtc */ dirty_ratelimit = wb->dirty_ratelimit; @@ -2371,7 +2360,13 @@ int do_writepages(struct address_space *mapping, struct writeback_control *wbc) cond_resched(); congestion_wait(BLK_RW_ASYNC, HZ/50); } - wb_update_bandwidth(wb); + /* + * Usually few pages are written by now from those we've just submitted + * but if there's constant writeback being submitted, this makes sure + * writeback bandwidth is updated once in a while. + */ + if (time_is_before_jiffies(wb->bw_time_stamp + BANDWIDTH_INTERVAL)) + wb_update_bandwidth(wb); return ret; } @@ -2742,6 +2737,11 @@ static void wb_inode_writeback_start(struct bdi_writeback *wb) static void wb_inode_writeback_end(struct bdi_writeback *wb) { atomic_dec(&wb->writeback_inodes); + /* + * Make sure estimate of writeback throughput gets + * updated after writeback completed. + */ + queue_delayed_work(bdi_wq, &wb->bw_dwork, BANDWIDTH_INTERVAL); } int test_clear_page_writeback(struct page *page) From patchwork Mon Jul 5 16:23:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 12359419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9C65C07E9B for ; Mon, 5 Jul 2021 16:23:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 72925613D1 for ; Mon, 5 Jul 2021 16:23:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72925613D1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E55476B0098; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDBEB6B0089; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B454D6B0098; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0086.hostedemail.com [216.40.44.86]) by kanga.kvack.org (Postfix) with ESMTP id 7C1BE6B0089 for ; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 00E22180339D4 for ; Mon, 5 Jul 2021 16:23:32 +0000 (UTC) X-FDA: 78329054622.01.BA15D4F Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf08.hostedemail.com (Postfix) with ESMTP id 8002530000A0 for ; Mon, 5 Jul 2021 16:23:31 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 89F591FEBC; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yv+4kF1EjnZ4ZkFyTsjZqCItVDiJyv1dyIx2QGj7x5A=; b=Jk+uZMZRyuO5E7XmUGr7nmAFa+HOJ2vclWY+0tu4zpn2/nsrVHtEHN0Qeas3qR6he0EHG+ iabXTkXR0u7IbOO6gqt0IicSg2H8Y8OvSwtkpbA9nkrtYjnVoEJunqHV89D6ICC4wjyVUI 4DgUjNLULekvub5isS6/HZ9iBJksSv8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yv+4kF1EjnZ4ZkFyTsjZqCItVDiJyv1dyIx2QGj7x5A=; b=yWKp67yLcI+A8Sd93PRE85SZhHaz49f3cw8jTQpMJGrYc0jjzDOzDm83IMNtLh7FS35Swd QbnQ21TU5Ud0r4BQ== Received: from quack2.suse.cz (unknown [10.163.43.118]) by relay2.suse.de (Postfix) with ESMTP id 6EF7DA3BA6; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 4B00D1F2CC9; Mon, 5 Jul 2021 18:23:28 +0200 (CEST) From: Jan Kara To: Andrew Morton Cc: , Michael Stapelberg , , Jan Kara Subject: [PATCH 4/5] writeback: Rename domain_update_bandwidth() Date: Mon, 5 Jul 2021 18:23:18 +0200 Message-Id: <20210705162328.28366-4-jack@suse.cz> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210705161610.19406-1-jack@suse.cz> References: <20210705161610.19406-1-jack@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1550; h=from:subject; bh=0nSUpWwlLNaV2zXsteeC9gDIgF5JqOxmX+5IXpCSxWI=; b=owEBbQGS/pANAwAIAZydqgc/ZEDZAcsmYgBg4zH2pgx1MM1pQm+exJQgkQO4+6tQ+oGbjgmG64Hw yrA+ioiJATMEAAEIAB0WIQSrWdEr1p4yirVVKBycnaoHP2RA2QUCYOMx9gAKCRCcnaoHP2RA2YktCA DEiA0n8tcFO9117fPR/3n+TwR7sGjnx37RTamWH3+UhLZ/KFneYWcFp+rETHWRYl0d+vMPvZRPkDhk VbFmMgKOB5jmg/7TmWKyIatgv3Cej8fuKQAPPydBD5JV0tR6bt2+uTS/TXJMOaWDroaWcKemVde9+w tBvmC+NaTFIeXmfBPv5Yhn8aJzUU8WT1WrppaYzCDgQfz8qKEJoO5vgCSib2HSCA5VPne1R/qn4WXn Sph6WrEy7XxUl1BOjh0nw+CToK8k8EKkmL+FIqv/LuvFOG+68vwzfVlXMU0FyKX9NZxcUj1nNKibxN nuP0KMfHEyGcspGeG6Yo04g+P3d9ww X-Developer-Key: i=jack@suse.cz; a=openpgp; fpr=93C6099A142276A28BBE35D815BC833443038D8C X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8002530000A0 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=Jk+uZMZR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yWKp67yL; spf=pass (imf08.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none X-Stat-Signature: oxcmm1h4yqqyjaq6c59eiftj3anuzzho X-HE-Tag: 1625502211-426958 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rename domain_update_bandwidth() to domain_update_dirty_limit(). The original name is a misnomer. The function has nothing to do with a bandwidth, it updates dirty limits. Signed-off-by: Jan Kara --- mm/page-writeback.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 6a99ddca95c0..95abae9eecaf 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -1153,8 +1153,8 @@ static void update_dirty_limit(struct dirty_throttle_control *dtc) dom->dirty_limit = limit; } -static void domain_update_bandwidth(struct dirty_throttle_control *dtc, - unsigned long now) +static void domain_update_dirty_limit(struct dirty_throttle_control *dtc, + unsigned long now) { struct wb_domain *dom = dtc_dom(dtc); @@ -1351,7 +1351,7 @@ static void __wb_update_bandwidth(struct dirty_throttle_control *gdtc, written = percpu_counter_read(&wb->stat[WB_WRITTEN]); if (update_ratelimit) { - domain_update_bandwidth(gdtc, now); + domain_update_dirty_limit(gdtc, now); wb_update_dirty_ratelimit(gdtc, dirtied, elapsed); /* @@ -1359,7 +1359,7 @@ static void __wb_update_bandwidth(struct dirty_throttle_control *gdtc, * compiler has no way to figure that out. Help it. */ if (IS_ENABLED(CONFIG_CGROUP_WRITEBACK) && mdtc) { - domain_update_bandwidth(mdtc, now); + domain_update_dirty_limit(mdtc, now); wb_update_dirty_ratelimit(mdtc, dirtied, elapsed); } } From patchwork Mon Jul 5 16:23:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 12359425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23690C07E9B for ; Mon, 5 Jul 2021 16:23:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CC7C46135D for ; Mon, 5 Jul 2021 16:23:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC7C46135D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7714F6B008A; Mon, 5 Jul 2021 12:23:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D7266B0099; Mon, 5 Jul 2021 12:23:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E51426B0093; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id 91CAD6B008C for ; Mon, 5 Jul 2021 12:23:32 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0C6D321256 for ; Mon, 5 Jul 2021 16:23:32 +0000 (UTC) X-FDA: 78329054664.11.443B59E Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf17.hostedemail.com (Postfix) with ESMTP id 3F90EF000136 for ; Mon, 5 Jul 2021 16:23:31 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id A6B06203A4; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9B+bgNZjQ9mvr7D5XuowXLwKzGqWg6Ae4fG3Ijnk9bw=; b=j5ktUEHBJESw3YjHB5x3BLocw6tftKpTOuqXLtMSy/4lOJAdlymgKUR9aDjfauwPAXCMDz kGIC2XVyDgt6rTD3LtYZZcm5UnpgRmC3qKZE3DGyuNm2mvcSVHSGyErNBYbHI7HPDkUmHm h4I5uq2iYWgfndOw6oIInwBaRS5Fbu0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1625502208; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9B+bgNZjQ9mvr7D5XuowXLwKzGqWg6Ae4fG3Ijnk9bw=; b=dkem4QSlkxhlOcBwoEoZtDBX73LeVFKgFbZGLeedy6RM3UMatmnCJNksQYYNBZ+G1CT5as gPhMErHbLSce1cAw== Received: from quack2.suse.cz (unknown [10.163.43.118]) by relay2.suse.de (Postfix) with ESMTP id 97468A3BAB; Mon, 5 Jul 2021 16:23:28 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 4E3FA1F2CCA; Mon, 5 Jul 2021 18:23:28 +0200 (CEST) From: Jan Kara To: Andrew Morton Cc: , Michael Stapelberg , , Jan Kara Subject: [PATCH 5/5] writeback: Use READ_ONCE for unlocked reads of writeback stats Date: Mon, 5 Jul 2021 18:23:19 +0200 Message-Id: <20210705162328.28366-5-jack@suse.cz> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210705161610.19406-1-jack@suse.cz> References: <20210705161610.19406-1-jack@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4091; h=from:subject; bh=Y/DQCwhnMO5kfzvgrvnfAdARRzMVl5z0DoIWDYazekc=; b=owEBbQGS/pANAwAIAZydqgc/ZEDZAcsmYgBg4zH3EMQbDh1mvMvpxuRuIsTSbPfutv/A8hcuf62c DxCBt6eJATMEAAEIAB0WIQSrWdEr1p4yirVVKBycnaoHP2RA2QUCYOMx9wAKCRCcnaoHP2RA2cTSB/ 4urNVq5zYQxslNsCJ3vf1hIEFCLtpSXihiz2jSSfjVZ22lI0pRJZu3MdYKDKwy9ys0o1to4aS4wQ+U DETSRej5aZQI7wF6RMJUD2pNiSAHQMx2276BS/OsxkB442F7MCV/j+gwvmtAqWRl4Kqn0YvcnQIrgE MzW4HObTGfI3WUE154Qm2UY6wDTSMfq9HsrSB3WUdpA61Hmtfb315VaRtiZWk6cxdBuV0bvuCxRnKA yLxFBVODagpadfdfF5J0ETyFzgsGRMmuFTINkT4IRMjwTYXwqbjuITbPSQcaqMiRA3iGulOpiHAOXU dAVcfl0QhXHvLEjQMXQOqGDpZ5nrCE X-Developer-Key: i=jack@suse.cz; a=openpgp; fpr=93C6099A142276A28BBE35D815BC833443038D8C Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=j5ktUEHB; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=dkem4QSl; spf=pass (imf17.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz; dmarc=none X-Stat-Signature: b6obto9fwcoic7je9fh63izjorznhy8f X-Rspamd-Queue-Id: 3F90EF000136 X-Rspamd-Server: rspam06 X-HE-Tag: 1625502211-117059 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We do some unlocked reads of writeback statistics like avg_write_bandwidth, dirty_ratelimit, or bw_time_stamp. Generally we are fine with getting somewhat out-of-date values but actually getting different values in various parts of the functions because the compiler decided to reload value from original memory location could confuse calculations. Use READ_ONCE for these unlocked accesses to be on the safe side. Signed-off-by: Jan Kara --- mm/page-writeback.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 95abae9eecaf..736d9e996191 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -189,7 +189,7 @@ static struct fprop_local_percpu *wb_memcg_completions(struct bdi_writeback *wb) static void wb_min_max_ratio(struct bdi_writeback *wb, unsigned long *minp, unsigned long *maxp) { - unsigned long this_bw = wb->avg_write_bandwidth; + unsigned long this_bw = READ_ONCE(wb->avg_write_bandwidth); unsigned long tot_bw = atomic_long_read(&wb->bdi->tot_write_bandwidth); unsigned long long min = wb->bdi->min_ratio; unsigned long long max = wb->bdi->max_ratio; @@ -898,7 +898,7 @@ static long long pos_ratio_polynom(unsigned long setpoint, static void wb_position_ratio(struct dirty_throttle_control *dtc) { struct bdi_writeback *wb = dtc->wb; - unsigned long write_bw = wb->avg_write_bandwidth; + unsigned long write_bw = READ_ONCE(wb->avg_write_bandwidth); unsigned long freerun = dirty_freerun_ceiling(dtc->thresh, dtc->bg_thresh); unsigned long limit = hard_dirty_limit(dtc_dom(dtc), dtc->thresh); unsigned long wb_thresh = dtc->wb_thresh; @@ -1342,11 +1342,12 @@ static void __wb_update_bandwidth(struct dirty_throttle_control *gdtc, { struct bdi_writeback *wb = gdtc->wb; unsigned long now = jiffies; - unsigned long elapsed = now - wb->bw_time_stamp; + unsigned long elapsed; unsigned long dirtied; unsigned long written; spin_lock(&wb->list_lock); + elapsed = now - wb->bw_time_stamp; dirtied = percpu_counter_read(&wb->stat[WB_DIRTIED]); written = percpu_counter_read(&wb->stat[WB_WRITTEN]); @@ -1416,7 +1417,7 @@ static unsigned long dirty_poll_interval(unsigned long dirty, static unsigned long wb_max_pause(struct bdi_writeback *wb, unsigned long wb_dirty) { - unsigned long bw = wb->avg_write_bandwidth; + unsigned long bw = READ_ONCE(wb->avg_write_bandwidth); unsigned long t; /* @@ -1438,8 +1439,8 @@ static long wb_min_pause(struct bdi_writeback *wb, unsigned long dirty_ratelimit, int *nr_dirtied_pause) { - long hi = ilog2(wb->avg_write_bandwidth); - long lo = ilog2(wb->dirty_ratelimit); + long hi = ilog2(READ_ONCE(wb->avg_write_bandwidth)); + long lo = ilog2(READ_ONCE(wb->dirty_ratelimit)); long t; /* target pause */ long pause; /* estimated next pause */ int pages; /* target nr_dirtied_pause */ @@ -1719,12 +1720,12 @@ static void balance_dirty_pages(struct bdi_writeback *wb, if (dirty_exceeded && !wb->dirty_exceeded) wb->dirty_exceeded = 1; - if (time_is_before_jiffies(wb->bw_time_stamp + + if (time_is_before_jiffies(READ_ONCE(wb->bw_time_stamp) + BANDWIDTH_INTERVAL)) __wb_update_bandwidth(gdtc, mdtc, true); /* throttle according to the chosen dtc */ - dirty_ratelimit = wb->dirty_ratelimit; + dirty_ratelimit = READ_ONCE(wb->dirty_ratelimit); task_ratelimit = ((u64)dirty_ratelimit * sdtc->pos_ratio) >> RATELIMIT_CALC_SHIFT; max_pause = wb_max_pause(wb, sdtc->wb_dirty); @@ -2365,7 +2366,8 @@ int do_writepages(struct address_space *mapping, struct writeback_control *wbc) * but if there's constant writeback being submitted, this makes sure * writeback bandwidth is updated once in a while. */ - if (time_is_before_jiffies(wb->bw_time_stamp + BANDWIDTH_INTERVAL)) + if (time_is_before_jiffies(READ_ONCE(wb->bw_time_stamp) + + BANDWIDTH_INTERVAL)) wb_update_bandwidth(wb); return ret; }