From patchwork Fri Jan 10 15:46:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Kalyazin X-Patchwork-Id: 13935059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CE95E77188 for ; Fri, 10 Jan 2025 15:47:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CAAC76B00C6; Fri, 10 Jan 2025 10:47:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5A046B00C7; Fri, 10 Jan 2025 10:47:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD4B56B00C8; Fri, 10 Jan 2025 10:47:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8CA046B00C6 for ; Fri, 10 Jan 2025 10:47:20 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 08D5EC0BEF for ; Fri, 10 Jan 2025 15:47:20 +0000 (UTC) X-FDA: 82991971440.01.FB9C19E Received: from smtp-fw-80008.amazon.com (smtp-fw-80008.amazon.com [99.78.197.219]) by imf12.hostedemail.com (Postfix) with ESMTP id E4BF64000D for ; Fri, 10 Jan 2025 15:47:17 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=amazon.com header.s=amazon201209 header.b=kyfxonJm; dmarc=pass (policy=quarantine) header.from=amazon.com; spf=pass (imf12.hostedemail.com: domain of "prvs=09809d163=kalyazin@amazon.co.uk" designates 99.78.197.219 as permitted sender) smtp.mailfrom="prvs=09809d163=kalyazin@amazon.co.uk" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736524038; a=rsa-sha256; cv=none; b=DTv/D6yDuWARFgAEWHySBTV04eUn0ORsqavZkjOsrSczg04niOfHT7ASwaWcZv0+Y/fzCY DvB+fVXoqwttUZv1KaUXGYD+lHVAG5j8LjJj37+oP3kMSmmTIykNfYCI/ut3MlmKCoq7hq lON4PCYNQR8mNzjjR5q4GAqpC0LGCRM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=amazon.com header.s=amazon201209 header.b=kyfxonJm; dmarc=pass (policy=quarantine) header.from=amazon.com; spf=pass (imf12.hostedemail.com: domain of "prvs=09809d163=kalyazin@amazon.co.uk" designates 99.78.197.219 as permitted sender) smtp.mailfrom="prvs=09809d163=kalyazin@amazon.co.uk" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736524038; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OXivU7M5CXSyTYrNpqb9X3JfzB/3lICobLVQuajiBEU=; b=FdiFA+oxAihSoO3BuDEa9zC5pfpRAzjFXI27Snxfae/c7b9oT5S0Ju67fjadYm06Cr0BY+ mY1Dg2SXYfIYWPqYRZskf0sZc3JycDyvWtlSDS0Dg+1ZGHOlE3B9EqE+/u4vTfbIvbdeBZ Yz9MALk9Yvt59nsdU2gGCpdqMfiAE5U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1736524038; x=1768060038; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OXivU7M5CXSyTYrNpqb9X3JfzB/3lICobLVQuajiBEU=; b=kyfxonJmJBsXeGrerHIxEO8L3GClwbPamFkvdhRoG6J8V5coxzRgh6yz fPxE+PV0lUMc73V4Na/v2OAmq7+1/f0pz/Qby6/oqBw22hZWv9hEbAFsp PRk/M2g4zttLkoTjL5LUkF9l/tjJTQyH8pEzFXPGz4C5nPLTnw2CUwlPJ o=; X-IronPort-AV: E=Sophos;i="6.12,303,1728950400"; d="scan'208";a="160494933" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.25.36.214]) by smtp-border-fw-80008.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jan 2025 15:47:15 +0000 Received: from EX19MTAUWC001.ant.amazon.com [10.0.7.35:57625] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.29.65:2525] with esmtp (Farcaster) id 31d70eba-0d84-47e0-86c1-973ac7aa0042; Fri, 10 Jan 2025 15:47:15 +0000 (UTC) X-Farcaster-Flow-ID: 31d70eba-0d84-47e0-86c1-973ac7aa0042 Received: from EX19D020UWC002.ant.amazon.com (10.13.138.147) by EX19MTAUWC001.ant.amazon.com (10.250.64.174) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.39; Fri, 10 Jan 2025 15:47:15 +0000 Received: from EX19MTAUWC001.ant.amazon.com (10.250.64.145) by EX19D020UWC002.ant.amazon.com (10.13.138.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.39; Fri, 10 Jan 2025 15:47:14 +0000 Received: from email-imr-corp-prod-pdx-all-2c-8a67eb17.us-west-2.amazon.com (10.25.36.210) by mail-relay.amazon.com (10.250.64.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.39 via Frontend Transport; Fri, 10 Jan 2025 15:47:14 +0000 Received: from dev-dsk-kalyazin-1a-a12e27e2.eu-west-1.amazon.com (dev-dsk-kalyazin-1a-a12e27e2.eu-west-1.amazon.com [172.19.103.116]) by email-imr-corp-prod-pdx-all-2c-8a67eb17.us-west-2.amazon.com (Postfix) with ESMTPS id 5990140258; Fri, 10 Jan 2025 15:47:12 +0000 (UTC) From: Nikita Kalyazin To: , , , , , CC: , , , , , , , , , , , Subject: [RFC PATCH 1/2] mm: filemap: add filemap_grab_folios Date: Fri, 10 Jan 2025 15:46:58 +0000 Message-ID: <20250110154659.95464-2-kalyazin@amazon.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20250110154659.95464-1-kalyazin@amazon.com> References: <20250110154659.95464-1-kalyazin@amazon.com> MIME-Version: 1.0 X-Stat-Signature: ists76bc4mzdkst47496tc99h4ygjst7 X-Rspam-User: X-Rspamd-Queue-Id: E4BF64000D X-Rspamd-Server: rspam08 X-HE-Tag: 1736524037-816375 X-HE-Meta: U2FsdGVkX1/mGvWB08tLDRz7uDiecbMuFfVajmC1wMBhSUGQENBcop4omTQgwawg5gqLsNhZidN7gKm9wHKnGQIhA9JacZ/YDhUhLl8GKUSY63zj5Xnn8ol0n8LWa7A/35gbTrGkGCYubeX/rlQv8WpU/Jp5gZpcFq1OnBHeuGl5sfod1e4uysON6ryZ9oIKQYpK0Svtb92jUtYXRkRPBrgYucOot3c/Lqgpw6i4dr1lrVcY9d5QgBlCt8IRQTkCDFIAx8vq05WqBYcdTdFvAdV+zJahxvyzdW/OLApWBOzpVDGk8Yha1YAs6aq9CCjdXTpN+cVK2ePKHgEo+CgcFF4+qvgg3Vd+gql3hzYG/aG/DE8UZLhikYph4t0+XcgH1DScJNEt63O//O1OeApgkC1i4RovKPcBqyFnQ0C0sayOPkD8ZO2E60wwTJ+2o+Dg1OgEgHEvlRpL3uWUGcqKtcf4ikxcasO75xIvRnzKHKOYCGC59AIy+Dee/LvGeHFvDYHLhf6JCLiwUCsGzlbQZqiUSj6k4s6x69p0CQlPKMCeUTwwToDMvbCJcCkvDhMGBGnLl2RZs8iV0Wus6AuYL/oXY2xLAH+5EjKc9AhlfKYwVAwgyJ+cOrSt2OoUxENkof037Ftq2x6YZaDFaz83aeX7sSKEpcMmbly1EliQyvyU2lUPT/EnbosYMY9aiYze9QGM1rJS7nKLBYNE2DAqopJ6Y8y639iqTYWiEL1DH0cihOf7hrdf24lJHaiGkCHWUyTtHzh4YzhPXW7ntao9y99OFisqk9RofAom+7p7u9AeAyvk/1eFDSiWzq07a1Hn782tphDwWkjt67lXqfOMALP9Sm/PeLfWBwYVk0MdP4lGNDp5zboe9BF4yHPLBRxqEsHhpdKnEwSr5ASs+q3vZStzoi5nlZnS+LxRc4eJIBIgDllcJzRFJMkyHiBb8cVgLEtIoh/qGu2Fb3fs2Rl +18RU9vp j2Tt8yeEUTwf0x8xQmRzum/PrYtSMipXXEVcLt+RCkdbJMDZ8Bk/Z8lHfvzZaLvpSMaTbSAi4d7CiFdu1k4ZoLcVDVaz9MMR6przLfsD7Ls2tVPSCUOaHnDZ2jz9AQus1GCePs4LnGmAmke6h7Rr9e89a9jnZiU7vqfjdM2sQcEnb+dvWZt5km5f33k1CZh4Ma+Z5edQTPKhVkNZYNRGwUjC8F9w0FMEeDFxmQM1aRw5sAiLfgHtWT8JLIZ2ovHdFk6JJmktC/hxwPEWuGG0YvItkNzBP8Rr7P3SErCxKbcd+cYQF/EmC4bjyCNbMA1/ryVmb+IWx/PgVrZAmKwUuh0aMA8R5dVj2SS56LSg8fqJ7OhHYzZEVQofgqjPWsi7aCrgBDsmiSrofXCuzVIgYiMBvsS36GO5O8AlFid3keY/u2nvb+SVdsjjjcVhyE9V30QvzC3M4aHHBfZR/QbZQ79Em5h1p5mWQFVujPzKW/eeKcRkILu2g7Jb/fKThPTl4DIIj4zGzKsRNtTfcJBinjZHsenDn48zF6wawTQBzs+BE+JA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similar to filemap_grab_folio, but grabs multiple folios at a time. filemap_grab_folios attempts to get 128 adjacent folios in the pagecache starting at the specified index. Whenever a folio is not found, it allocates a new one and adds it to the pagecache. The following is not currently supported: - large folios - reclaim effects in the pagecache (shadows) An equivalent of the following callstack is implemented to work on multiple folios: - filemap_grab_folio - __filemap_get_folio - filemap_add_folio - __filemap_add_folio Signed-off-by: Nikita Kalyazin --- include/linux/pagemap.h | 31 +++++ mm/filemap.c | 263 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 294 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 68a5f1ff3301..fd10d77c07c1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -747,9 +747,16 @@ static inline fgf_t fgf_set_order(size_t size) return (__force fgf_t)((shift - PAGE_SHIFT) << 26); } +/** + * Folio batch size used by __filemap_get_folios. + */ +#define FILEMAP_GET_FOLIOS_BATCH_SIZE 128 + void *filemap_get_entry(struct address_space *mapping, pgoff_t index); struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, fgf_t fgp_flags, gfp_t gfp); +int __filemap_get_folios(struct address_space *mapping, pgoff_t index, + fgf_t fgp_flags, gfp_t gfp, struct folio **folios, int num); struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index, fgf_t fgp_flags, gfp_t gfp); @@ -808,6 +815,30 @@ static inline struct folio *filemap_grab_folio(struct address_space *mapping, mapping_gfp_mask(mapping)); } +/** + * filemap_grab_folios - grab folios from the page cache + * @mapping: The address space to search + * @index: The page index to start with + * @folios: Output buffer for found or created folios + * @num: Number of folios to grab + * + * Looks up @num page cache entries at @mapping starting from @index. If no + * folio is found at the index, a new folio is created. The folios are locked, + * and marked as accessed. + * + * Return: The total number of found and created folios. Returned folios will + * always have adjacent indexes starting from @index. If no folios are found + * and created, -ENOMEM is returned. + */ +static inline int filemap_grab_folios(struct address_space *mapping, + pgoff_t index, struct folio **folios, + int num) +{ + return __filemap_get_folios(mapping, index, + FGP_LOCK | FGP_ACCESSED | FGP_CREAT, + mapping_gfp_mask(mapping), folios, num); +} + /** * find_get_page - find and get a page reference * @mapping: the address_space to search diff --git a/mm/filemap.c b/mm/filemap.c index 56fa431c52af..b5bc203e3350 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -958,6 +958,60 @@ noinline int __filemap_add_folio(struct address_space *mapping, } ALLOW_ERROR_INJECTION(__filemap_add_folio, ERRNO); +static int __filemap_add_folios(struct address_space *mapping, + struct folio **folios, pgoff_t index, + int num, unsigned long *exclude_bm, + bool *conflict) +{ + XA_STATE(xas, &mapping->i_pages, index); + int i; + + mapping_set_update(&xas, mapping); + xas_lock_irq(&xas); + + for (i = 0; i < num; i++) { + struct folio *folio = folios[i]; + + if (test_bit(i, exclude_bm)) { + xas_next(&xas); + if (i == 0) + xas_set(&xas, index + i + 1); + continue; + } + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); + VM_BUG_ON_FOLIO(folio_order(folio) != 0, folio); + VM_BUG_ON_FOLIO(folio_nr_pages(folio) != 1, folio); + + if (xas_find_conflict(&xas)) { + xas_set_err(&xas, -EEXIST); + *conflict = true; + break; + } + + folio_ref_inc(folio); + folio->mapping = mapping; + folio->index = xas.xa_index; + + xas_store(&xas, folio); + if (xas_error(&xas)) { + folio->mapping = NULL; + folio_put(folio); + break; + } + + __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, 1); + trace_mm_filemap_add_to_page_cache(folio); + xas_next(&xas); + mapping->nrpages++; + } + + xas_unlock_irq(&xas); + + return i ?: xas_error(&xas); +} + int filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp) { @@ -991,6 +1045,45 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, } EXPORT_SYMBOL_GPL(filemap_add_folio); +static int filemap_add_folios(struct address_space *mapping, + struct folio **folios, + pgoff_t index, gfp_t gfp, int num, + unsigned long *exclude_bm, bool *conflict) +{ + int ret, i, num_charged, num_added; + + for (i = 0; i < num; i++) { + if (test_bit(i, exclude_bm)) + continue; + ret = mem_cgroup_charge(folios[i], NULL, gfp); + if (unlikely(ret)) + break; + __folio_set_locked(folios[i]); + } + + num_charged = i; + if (!num_charged) + return ret; + + num_added = __filemap_add_folios(mapping, folios, index, num_charged, + exclude_bm, conflict); + + for (i = 0; i < num_added; i++) { + if (test_bit(i, exclude_bm)) + continue; + folio_add_lru(folios[i]); + } + + for (i = num_added; i < num_charged; i++) { + if (test_bit(i, exclude_bm)) + continue; + mem_cgroup_uncharge(folios[i]); + __folio_clear_locked(folios[i]); + } + + return num_added; +} + #ifdef CONFIG_NUMA struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { @@ -1982,6 +2075,176 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, } EXPORT_SYMBOL(__filemap_get_folio); +static int __filemap_get_folios_inner(struct address_space *mapping, + pgoff_t index, fgf_t fgp_flags, + gfp_t gfp, struct folio **folios, + int num, bool *conflict) +{ + DECLARE_BITMAP(present_bm, FILEMAP_GET_FOLIOS_BATCH_SIZE); + int i, err, num_present, num_alloced = 0, num_added = 0; + struct folio *folio; + + bitmap_zero(present_bm, FILEMAP_GET_FOLIOS_BATCH_SIZE); + + for (i = 0; i < num; i++) { + folio = filemap_get_entry(mapping, index + i); + if (xa_is_value(folio)) + folio = NULL; + + if (!folio) { + if (!(fgp_flags & FGP_CREAT)) { + err = -ENOENT; + break; + } + continue; + } + + if (fgp_flags & FGP_LOCK) { + if (fgp_flags & FGP_NOWAIT) { + if (!folio_trylock(folio)) { + folio_put(folio); + err = -EAGAIN; + break; + } + } else { + folio_lock(folio); + } + + /* Has the page been truncated? */ + if (unlikely(folio->mapping != mapping)) { + folio_unlock(folio); + folio_put(folio); + i--; + continue; + } + VM_BUG_ON_FOLIO(!folio_contains(folio, index + i), folio); + } + + if (fgp_flags & FGP_ACCESSED) + folio_mark_accessed(folio); + else if (fgp_flags & FGP_WRITE) { + /* Clear idle flag for buffer write */ + if (folio_test_idle(folio)) + folio_clear_idle(folio); + } + + if (fgp_flags & FGP_STABLE) + folio_wait_stable(folio); + + folios[i] = folio; + set_bit(i, present_bm); + } + + num_present = i ?: err; + + if ((fgp_flags & FGP_CREAT)) { + if ((fgp_flags & FGP_WRITE) && mapping_can_writeback(mapping)) + gfp |= __GFP_WRITE; + if (fgp_flags & FGP_NOFS) + gfp &= ~__GFP_FS; + if (fgp_flags & FGP_NOWAIT) { + gfp &= ~GFP_KERNEL; + gfp |= GFP_NOWAIT | __GFP_NOWARN; + } + if (WARN_ON_ONCE(!(fgp_flags & (FGP_LOCK | FGP_FOR_MMAP)))) + fgp_flags |= FGP_LOCK; + + for (i = 0; i < num; i++) { + if (test_bit(i, present_bm)) + continue; + + folios[i] = filemap_alloc_folio(gfp, 0); + if (!folios[i]) + break; + + /* Init accessed so avoid atomic mark_page_accessed later */ + if (fgp_flags & FGP_ACCESSED) + __folio_set_referenced(folios[i]); + } + + num_alloced = i; + + if (num_alloced > 0) { + num_added = filemap_add_folios(mapping, folios, index, gfp, num_alloced, present_bm, conflict); + + /* + * filemap_add_folios locks the page, and for mmap + * we expect an unlocked page. + */ + if ((fgp_flags & FGP_FOR_MMAP)) + for (i = 0; i < num_added; i++) { + if (!test_bit(i, present_bm)) + folio_unlock(folios[i]); + } + + /* + * Clean up folios that failed to get added. + */ + for (i = num_added; i < num_alloced; i++) { + if (!test_bit(i, present_bm)) { + folio_unlock(folios[i]); + folio_put(folios[i]); + } + } + } + + if (fgp_flags & FGP_LOCK) + /* + * Clean up folios that failed to get allocated. + */ + for (i = num_alloced; i < num; i++) { + if (test_bit(i, present_bm)) + folio_unlock(folios[i]); + } + } + + if (fgp_flags & FGP_CREAT) + return num_added ?: (num_alloced ?: num_present); + else + return num_present; +} + +/** + * __filemap_get_folios - Find and get references to folios. + * @mapping: The address_space to search. + * @index: The page index to start with. + * @fgp_flags: %FGP flags modify how the folio is returned. + * @gfp: Memory allocation flags to use if %FGP_CREAT is specified. + * @folios: Output buffer for found folios. + * @num: Number of folios to find. + * + * Looks up @num page cache entries at @mapping starting at @index. + * + * If %FGP_LOCK or %FGP_CREAT are specified then the function may sleep even + * if the %GFP flags specified for %FGP_CREAT are atomic. + * + * If this function returns @folios, they are returned with an increased + * refcount. + * + * Return: The number of found folios or an error otherwise. + */ +int __filemap_get_folios(struct address_space *mapping, pgoff_t index, + fgf_t fgp_flags, gfp_t gfp, struct folio **folios, + int num) +{ + int ret, processed = 0; + bool conflict; + + do { + conflict = false; + ret = __filemap_get_folios_inner(mapping, index, fgp_flags, + gfp, folios, num, &conflict); + if (ret > 0) { + index += ret; + folios += ret; + num -= ret; + processed += ret; + } + } while (ret > 0 && conflict && num); + + return processed ?: ret; +} + static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max, xa_mark_t mark) {