From patchwork Fri May 3 09:53:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luis Chamberlain X-Patchwork-Id: 13652678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77FD0C4345F for ; Fri, 3 May 2024 11:34:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E99A56B008A; Fri, 3 May 2024 07:34:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E4A2B6B008C; Fri, 3 May 2024 07:34:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D38236B0092; Fri, 3 May 2024 07:34:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B1CDE6B008A for ; Fri, 3 May 2024 07:34:05 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 273E61A03E6 for ; Fri, 3 May 2024 11:34:05 +0000 (UTC) X-FDA: 82076875650.02.A9ED210 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf02.hostedemail.com (Postfix) with ESMTP id 0451A80012 for ; Fri, 3 May 2024 11:34:02 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=jtLUsFwW; spf=none (imf02.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714736043; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g6CL/9c5ofMtPDZ9Wti9OxAUYSiFqve/TjymVkUUxuo=; b=tUYMLaK6dS8WgscpIXxaiVpOek30NbT0ON7MDBM3jriJHLkYzQ/SlemwTEFY+SZ5060NTN vGdfX7jkeDUfDePO2bWE/109xGvfmvNgJNpWhXDgIAB5QsKP1/pseec/l+K9XioYeQdNIL SwG1HvPGkhl26iQAB08sTONgk27ePC0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=jtLUsFwW; spf=none (imf02.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714736043; a=rsa-sha256; cv=none; b=D68N+XudCIr9NOP1eMRNmcU+kjGfB9JAmhV5rxkYLvimVPkJmdtOAIPfg+uEOdATSc3fv/ TafHAxcF9tvgru1jhS8QmE0JNc78sGcerlxS4QyYFJyewEq6h1BEhl6ER1lKZWU1TEYZKu ZjlvVGN2ZQi4oWOAhmaKgpNQVEBL5bY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=g6CL/9c5ofMtPDZ9Wti9OxAUYSiFqve/TjymVkUUxuo=; b=jtLUsFwWhJ04pWkYCpJn82Fe8W avcYDbc8xUij+l8vWmeQIgfeW2n2XvWdqhDFcF+v/InrZrWvHqRK0Ny6c0VMwRN1JqtPlf0268gXC atDNrpV6OFyo35d7rEn/IxBM2onVjnBkIMnVsvXoWwcAuk3qbcev743pOlQ++UpKXfebOThRYii8B 9Hna3EhvTVZ2yuoRXZSKBT6opZTx4iu2KZhAOiGdimvNpB/svB3f5V/sH78oJsNpO5eCmvvjkDNT0 0TRsodsxxMLkf4UQUQWNWl2/7BJ0GoGovST28sN2icgXhlozpqq+hlOdpdccENrFbN7eJjlQ+fJAe INBeTEyw==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1s2pc3-0000000Fw3S-0CmL; Fri, 03 May 2024 09:53:55 +0000 From: Luis Chamberlain To: akpm@linux-foundation.org, willy@infradead.org, djwong@kernel.org, brauner@kernel.org, david@fromorbit.com, chandan.babu@oracle.com Cc: hare@suse.de, ritesh.list@gmail.com, john.g.garry@oracle.com, ziy@nvidia.com, linux-fsdevel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, gost.dev@samsung.com, p.raghav@samsung.com, kernel@pankajraghav.com, mcgrof@kernel.org Subject: [PATCH v5 02/11] fs: Allow fine-grained control of folio sizes Date: Fri, 3 May 2024 02:53:44 -0700 Message-ID: <20240503095353.3798063-3-mcgrof@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240503095353.3798063-1-mcgrof@kernel.org> References: <20240503095353.3798063-1-mcgrof@kernel.org> MIME-Version: 1.0 X-Stat-Signature: tpuqhdfe3eoa3h4p4oqkimuoudbbaoiy X-Rspamd-Queue-Id: 0451A80012 X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1714736042-382155 X-HE-Meta: U2FsdGVkX19/REh+2oSIxzDKjN9rYJ3aAYzpJse/kIWvNc/bqWHqB0h49mnangENI3MrWEpidehW0dZSQDjvdj7PwH0VxDMk0ie72QoF/oAU8U37GUQoYMXCVfOexF4fNttxbMA2Ww3jrcnp1lW2D8/Hqpio2qKwahIUxVI2gGqXQjEzaTSmVfHMQKkzSiiX7kzw+xTtbiUPd4WY3K9WoHsysAdwADTR64fvwJ0/bcCWbG4G6r540RAWVy9yJw9nQomEy7aQTOepiggZRMnuX2ww3Dato2513XJ0Omv+IOxXiii7unNA6X6pScu14uAEmY3v4q76JJnEjPzhqSHzZqaGpvrm7pYkwWCbWkinKkEef80ptyiK/EnCEh0u08N0AuhnzyPdJBFMQzbYTD5/IrhtJpGTtAuEZVI1pgRX4ytVNqK+iZvecYwG/zQKaQImkrsrnrsAprVSFBRYcc0w76Fin1lhSteguZ7aiaUfXxtjlXGpVrCbPTMEXVLOj3/pqKH+5TDrbLMFKuoEW0dNFCb78HAzMz6oNDfsHMhoX+L0DF6FL3ijfkx5ey8zIE2v5F3pXgnItbauKdHgF5JfxvSWEuv85mCGmZvaJuWHYoaQMowqB05kJEkVo4ZlDZb7gI7nvH/B1kFm14W6MdSCM+Gcsb49W/qAHDtFo0xlkL9wPIFjdv9ZqSG+qAq/NQMIWBtFjsJV1X5Bp6bFkuC7iSHSzmVxwpvXMdVfsdAnH7CDJcIy5pqX5gDegMeg5cwh5ru3v5PZnBCsDlg5oGic6dDMdPQpeQFQUtoc4IWuUlJL940MSzDnmcQZl8z0J13mWV6LRLCayX201NdCoxvMMGyF2oSoSMZopAt6cX2t2r1s8AqRfQ0P5yOcgIoPTzbINYdYoTqhZFAy33RLk+dZk1emYIv0XluWDcesjEHcSeUNu86v937bfjItwlu4iQqcytpLyY1VSuhxvTXRTdA PNX4tSiQ 42D+bjK2ll8+mWqohP5ckQn8s6F6ms0igIsWcrUM3bb50fQJsDWddkgyvJistnFgQb725pL+oyJUliwV8l+bLRaMPoKIopWq4G2TroXQa6r81F7IB6CpRX87WP7zQEFTx6mrKdg3vR+muvi5GsCdwxMhHG98+96DYeJVsKc43Uclo1WpMm1rw42h3HfWGgKNnf5U9ShSe/nAOcabmOwigZrD3F/Sz+6ABp/nSKMzSR7k4ypkzA4rCvo9Nhj637p0uxrF2EX6errBxOX2+AgCXG+ggjEgQyKb9TdG/pnqLgUqoi+ObLSpUUz2k2wtIR3V7nax2UzM7HXF2KZdGIamSvQjZRDWy6RMwLec85QWdreKzhcQwCg7EQT10yNS5FOzTenk7W0otVHp/D8DnViAX1jKCkLfaVrOr7KXEJzYDnZgu8/HrJuTr7WJFme6oeQoOkKQ6VWR2yKXVXd0kJkVsl78MTrXKdQSRS4Cl1gWFACSbfzC84sNUjJyeZ14/ooP+4prmuNCwrTNbL7RXtI4tEV6Qhy0naKeEgA5uxrxmzleztzk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: "Matthew Wilcox (Oracle)" Some filesystems want to be able to ensure that folios that are added to the page cache are at least a certain size. Add mapping_set_folio_min_order() to allow this level of control. Signed-off-by: Matthew Wilcox (Oracle) Co-developed-by: Pankaj Raghav Signed-off-by: Pankaj Raghav Reviewed-by: Darrick J. Wong Reviewed-by: Hannes Reinecke Signed-off-by: Luis Chamberlain --- include/linux/pagemap.h | 116 +++++++++++++++++++++++++++++++++------- 1 file changed, 96 insertions(+), 20 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2df35e65557d..2e5612de1749 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -202,13 +202,18 @@ enum mapping_flags { AS_EXITING = 4, /* final truncate in progress */ /* writeback related tags are not used */ AS_NO_WRITEBACK_TAGS = 5, - AS_LARGE_FOLIO_SUPPORT = 6, - AS_RELEASE_ALWAYS, /* Call ->release_folio(), even if no private data */ - AS_STABLE_WRITES, /* must wait for writeback before modifying + AS_RELEASE_ALWAYS = 6, /* Call ->release_folio(), even if no private data */ + AS_STABLE_WRITES = 7, /* must wait for writeback before modifying folio contents */ - AS_UNMOVABLE, /* The mapping cannot be moved, ever */ + AS_FOLIO_ORDER_MIN = 8, + AS_FOLIO_ORDER_MAX = 13, /* Bit 8-17 are used for FOLIO_ORDER */ + AS_UNMOVABLE = 18, /* The mapping cannot be moved, ever */ }; +#define AS_FOLIO_ORDER_MIN_MASK 0x00001f00 +#define AS_FOLIO_ORDER_MAX_MASK 0x0003e000 +#define AS_FOLIO_ORDER_MASK (AS_FOLIO_ORDER_MIN_MASK | AS_FOLIO_ORDER_MAX_MASK) + /** * mapping_set_error - record a writeback error in the address_space * @mapping: the mapping in which an error should be set @@ -344,9 +349,63 @@ static inline void mapping_set_gfp_mask(struct address_space *m, gfp_t mask) m->gfp_mask = mask; } +/* + * There are some parts of the kernel which assume that PMD entries + * are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then, + * limit the maximum allocation order to PMD size. I'm not aware of any + * assumptions about maximum order if THP are disabled, but 8 seems like + * a good order (that's 1MB if you're using 4kB pages) + */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER +#else +#define MAX_PAGECACHE_ORDER 8 +#endif + +/* + * mapping_set_folio_order_range() - Set the folio order range + * @mapping: The address_space. + * @min: Minimum folio order (between 0-MAX_PAGECACHE_ORDER inclusive). + * @max: Maximum folio order (between @min-MAX_PAGECACHE_ORDER inclusive). + * + * The filesystem should call this function in its inode constructor to + * indicate which base size (min) and maximum size (max) of folio the VFS + * can use to cache the contents of the file. This should only be used + * if the filesystem needs special handling of folio sizes (ie there is + * something the core cannot know). + * Do not tune it based on, eg, i_size. + * + * Context: This should not be called while the inode is active as it + * is non-atomic. + */ +static inline void mapping_set_folio_order_range(struct address_space *mapping, + unsigned int min_order, + unsigned int max_order) +{ + if (min_order > MAX_PAGECACHE_ORDER) + min_order = MAX_PAGECACHE_ORDER; + + if (max_order > MAX_PAGECACHE_ORDER) + max_order = MAX_PAGECACHE_ORDER; + + max_order = max(max_order, min_order); + /* + * TODO: max_order is not yet supported in filemap. + */ + mapping->flags = (mapping->flags & ~AS_FOLIO_ORDER_MASK) | + (min_order << AS_FOLIO_ORDER_MIN) | + (max_order << AS_FOLIO_ORDER_MAX); +} + +static inline void mapping_set_folio_min_order(struct address_space *mapping, + unsigned int min) +{ + mapping_set_folio_order_range(mapping, min, MAX_PAGECACHE_ORDER); +} + /** * mapping_set_large_folios() - Indicate the file supports large folios. - * @mapping: The file. + * @mapping: The address_space. * * The filesystem should call this function in its inode constructor to * indicate that the VFS can use large folios to cache the contents of @@ -357,7 +416,37 @@ static inline void mapping_set_gfp_mask(struct address_space *m, gfp_t mask) */ static inline void mapping_set_large_folios(struct address_space *mapping) { - __set_bit(AS_LARGE_FOLIO_SUPPORT, &mapping->flags); + mapping_set_folio_order_range(mapping, 0, MAX_PAGECACHE_ORDER); +} + +static inline unsigned int mapping_max_folio_order(struct address_space *mapping) +{ + return (mapping->flags & AS_FOLIO_ORDER_MAX_MASK) >> AS_FOLIO_ORDER_MAX; +} + +static inline unsigned int mapping_min_folio_order(struct address_space *mapping) +{ + return (mapping->flags & AS_FOLIO_ORDER_MIN_MASK) >> AS_FOLIO_ORDER_MIN; +} + +static inline unsigned long mapping_min_folio_nrpages(struct address_space *mapping) +{ + return 1UL << mapping_min_folio_order(mapping); +} + +/** + * mapping_align_start_index() - Align starting index based on the min + * folio order of the page cache. + * @mapping: The address_space. + * + * Ensure the index used is aligned to the minimum folio order when adding + * new folios to the page cache by rounding down to the nearest minimum + * folio number of pages. + */ +static inline pgoff_t mapping_align_start_index(struct address_space *mapping, + pgoff_t index) +{ + return round_down(index, mapping_min_folio_nrpages(mapping)); } /* @@ -367,7 +456,7 @@ static inline void mapping_set_large_folios(struct address_space *mapping) static inline bool mapping_large_folio_support(struct address_space *mapping) { return IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && - test_bit(AS_LARGE_FOLIO_SUPPORT, &mapping->flags); + (mapping_max_folio_order(mapping) > 0); } static inline int filemap_nr_thps(struct address_space *mapping) @@ -528,19 +617,6 @@ static inline void *detach_page_private(struct page *page) return folio_detach_private(page_folio(page)); } -/* - * There are some parts of the kernel which assume that PMD entries - * are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then, - * limit the maximum allocation order to PMD size. I'm not aware of any - * assumptions about maximum order if THP are disabled, but 8 seems like - * a good order (that's 1MB if you're using 4kB pages) - */ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER -#else -#define MAX_PAGECACHE_ORDER 8 -#endif - #ifdef CONFIG_NUMA struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order); #else