From patchwork Wed Sep 25 22:47:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 13812502 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 843ADCCFA07 for ; Wed, 25 Sep 2024 22:47:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19F9C6B009F; Wed, 25 Sep 2024 18:47:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 152EB6B00B6; Wed, 25 Sep 2024 18:47:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBC886B00BF; Wed, 25 Sep 2024 18:47:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C7EDB6B00BA for ; Wed, 25 Sep 2024 18:47:41 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7329C1A069C for ; Wed, 25 Sep 2024 22:47:41 +0000 (UTC) X-FDA: 82604749122.08.411BD36 Received: from out-179.mta0.migadu.com (out-179.mta0.migadu.com [91.218.175.179]) by imf29.hostedemail.com (Postfix) with ESMTP id C46E4120012 for ; Wed, 25 Sep 2024 22:47:39 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=b96mvYb6; spf=pass (imf29.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.179 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727304340; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=slbZ9FKSwH6OAIBuSfcvPtxMsbwgB1Nos7HczeUcRQE=; b=r9L8PyV6YshG69wwOywWkOZG+uBSXEWZVKMBCEigqPB4Rfma4lcr1Nd1RoyOh46xQ/x60X AGD53f6i+ERJcGEX7I1CdeJYeT0ksKePYoc8rgvnvBUer2Ru+fy1SrW1SHyKTTXuq/cw3d tuvbRGba2uzemwHtQzwMZdi46tQCp7A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727304340; a=rsa-sha256; cv=none; b=TqJByeX7M3OPG4g/Scet4174eoqBsZExAFsbAeo2hOeWXQjStkiWjyPCdw6E6gCjF1nbCP E8FMdVUcrCFlBT7oEl6mY6kdze+RorWg4FWVXG2cWBvLNDOonLUoAGNSB53TseZ87gYz5U zYfiFL3/1F77c2nsq1Gxy1etXHu3Te4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=b96mvYb6; spf=pass (imf29.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.179 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1727304458; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=slbZ9FKSwH6OAIBuSfcvPtxMsbwgB1Nos7HczeUcRQE=; b=b96mvYb6xAxPtPPb3lkOg3MmF6z+lRx20ztLlq+lEb6GvoxLsb5txOTLu2fQ0nNRYskp6j MXlHRTFvlL2g/67ij5+TC0d0ktcdzvREN/Ogrrl0x2xzD70paFu5oXoOQ5D0zDYq+9XLgl ySe24QdKOYsZ0zMIOw2wwx9WjVcHvd4= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Matthew Wilcox , Omar Sandoval , Chris Mason , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2 2/2] mm: optimize invalidation of shadow entries Date: Wed, 25 Sep 2024 15:47:16 -0700 Message-ID: <20240925224716.2904498-3-shakeel.butt@linux.dev> In-Reply-To: <20240925224716.2904498-1-shakeel.butt@linux.dev> References: <20240925224716.2904498-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: C46E4120012 X-Stat-Signature: 74c8ht5kucco98hmfmx7g5emb9qd3e9t X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1727304459-820815 X-HE-Meta: U2FsdGVkX1+ukzfPbSNkYrjDa2rXIII53JlojvtZ2IkAywiNBVX/A+V1os5PRFxKBVXskSFF3u32LRk6uNHEApcQL2WDbsGLQxv8ioEeiqugjgmmJlMt0IPuUTrF3TdKWxV0dSO0e4Bu0xvPQddSO7+eV9ZNCmApnGuCJyV7wPff0JnVUWBXG+KHM/B2hHX7k53YWTDRQUxKPYUDrwGBpGnYKfjqdtBgioapNXFxqRwH5thaycNJ1QyV06Mh+z6I//AtmyWH4B/6bcdS/E3vToHSn4sVulo6kBzfoOGlOu6aoJSgfxr+FeXYfwvcn+GSs+9Ubmmt372QxYwBVWoUaoydJTWDobche1AE278FdUNrRe5k3hibjdUuUd09t/3Gb5arWCyOIgOyYFtC2J2ckBbK6UMFdFGKOSv5Rm0Qttfexpqmophq0MRebCmQIfg1afwKEjHwzu/YEBmuiCGK31B/JU4sPnBeT1dxegs2HkMO2/AWzoznI5L7IJsxLKToTg+r0bpqSoQLeW5PHdLFVfmSX3wxeR17JgSY1rw16NGmfcgr+gTxKNioZQtnAUZY63YbBg2pjIuMsvjhsMYQRe9PvwZIAoRioF0rSfkiv4/DmpOWQbmZXYp0/DnhFyNdsAM1Vdka/aYN73rpULDNZ0Ova3yBlMlZ0iS2gKSis8dhit6m9z070ZOaD9CAf9ufmzLOnHdUXBc5hgolqxII0R+BYwoAXgHZ5jPT0YtF7QTM9k9jLeSY4RUFQQnb2+bz7x94HKV8/tqcnMdlPiSDZHgEMDeYuZ+oZTDZSqZpA1jKKimj9IqoqGpXogMILyDCon9nDViMF99/XMErTiqkPfVwNbBixRy1WWBxf5DzTVobMihch2DLnlztrAMLCHlVGXJ7vNX9zqYSg+qwnuzP1OacSxcWl4XYjCqN22wZbmyR5Ljl0uuB/ZtktFjwFLwjhEgZ+GUor5N9aMgjyXo e3KtbVd5 VnuREsH9Jt8YyYfWCj+IeS1PR/8QqGepMeDg0/RpFvn4tv2z1NXr57O31WOf5adUOt/UVjno0/r6LhD4Kr16RTJH4YidYhwbry3VWpIxytzD4cB9vtxKo6UliFJBbdYQ654lyXB2ZaM9oaCI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The kernel invalidates the page cache in batches of PAGEVEC_SIZE. For each batch, it traverses the page cache tree and collects the entries (folio and shadow entries) in the struct folio_batch. For the shadow entries present in the folio_batch, it has to traverse the page cache tree for each individual entry to remove them. This patch optimize this by removing them in a single tree traversal. To evaluate the changes, we created 200GiB file on a fuse fs and in a memcg. We created the shadow entries by triggering reclaim through memory.reclaim in that specific memcg and measure the simple fadvise(DONTNEED) operation. # time xfs_io -c 'fadvise -d 0 ${file_size}' file time (sec) Without 5.12 +- 0.061 With-patch 4.19 +- 0.086 (18.16% decrease) Signed-off-by: Shakeel Butt --- Changes since v1: - N/A mm/truncate.c | 46 ++++++++++++++++++---------------------------- 1 file changed, 18 insertions(+), 28 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 1d51c023d9c5..520c8cf8f58f 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -23,42 +23,28 @@ #include #include "internal.h" -/* - * Regular page slots are stabilized by the page lock even without the tree - * itself locked. These unlocked entries need verification under the tree - * lock. - */ -static inline void __clear_shadow_entry(struct address_space *mapping, - pgoff_t index, void *entry) -{ - XA_STATE(xas, &mapping->i_pages, index); - - xas_set_update(&xas, workingset_update_node); - if (xas_load(&xas) != entry) - return; - xas_store(&xas, NULL); -} - static void clear_shadow_entries(struct address_space *mapping, - struct folio_batch *fbatch, pgoff_t *indices) + unsigned long start, unsigned long max) { - int i; + XA_STATE(xas, &mapping->i_pages, start); + struct folio *folio; /* Handled by shmem itself, or for DAX we do nothing. */ if (shmem_mapping(mapping) || dax_mapping(mapping)) return; - spin_lock(&mapping->host->i_lock); - xa_lock_irq(&mapping->i_pages); + xas_set_update(&xas, workingset_update_node); - for (i = 0; i < folio_batch_count(fbatch); i++) { - struct folio *folio = fbatch->folios[i]; + spin_lock(&mapping->host->i_lock); + xas_lock_irq(&xas); + /* Clear all shadow entries from start to max */ + xas_for_each(&xas, folio, max) { if (xa_is_value(folio)) - __clear_shadow_entry(mapping, indices[i], folio); + xas_store(&xas, NULL); } - xa_unlock_irq(&mapping->i_pages); + xas_unlock_irq(&xas); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock); @@ -481,7 +467,9 @@ unsigned long mapping_try_invalidate(struct address_space *mapping, folio_batch_init(&fbatch); while (find_lock_entries(mapping, &index, end, &fbatch, indices)) { - for (i = 0; i < folio_batch_count(&fbatch); i++) { + int nr = folio_batch_count(&fbatch); + + for (i = 0; i < nr; i++) { struct folio *folio = fbatch.folios[i]; /* We rely upon deletion not changing folio->index */ @@ -508,7 +496,7 @@ unsigned long mapping_try_invalidate(struct address_space *mapping, } if (xa_has_values) - clear_shadow_entries(mapping, &fbatch, indices); + clear_shadow_entries(mapping, indices[0], indices[nr-1]); folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); @@ -612,7 +600,9 @@ int invalidate_inode_pages2_range(struct address_space *mapping, folio_batch_init(&fbatch); index = start; while (find_get_entries(mapping, &index, end, &fbatch, indices)) { - for (i = 0; i < folio_batch_count(&fbatch); i++) { + int nr = folio_batch_count(&fbatch); + + for (i = 0; i < nr; i++) { struct folio *folio = fbatch.folios[i]; /* We rely upon deletion not changing folio->index */ @@ -658,7 +648,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping, } if (xa_has_values) - clear_shadow_entries(mapping, &fbatch, indices); + clear_shadow_entries(mapping, indices[0], indices[nr-1]); folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch);