From patchwork Fri Feb 9 14:29:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Gomez X-Patchwork-Id: 13551413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F7E9C48297 for ; Fri, 9 Feb 2024 14:29:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DAAF6B0096; Fri, 9 Feb 2024 09:29:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 062426B0095; Fri, 9 Feb 2024 09:29:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5E066B0088; Fri, 9 Feb 2024 09:29:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9B3A96B0085 for ; Fri, 9 Feb 2024 09:29:11 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6E588C0905 for ; Fri, 9 Feb 2024 14:29:11 +0000 (UTC) X-FDA: 81772497702.05.A53FDB3 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by imf12.hostedemail.com (Postfix) with ESMTP id DD99340023 for ; Fri, 9 Feb 2024 14:29:06 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=Pb7wORK5; spf=pass (imf12.hostedemail.com: domain of da.gomez@samsung.com designates 210.118.77.11 as permitted sender) smtp.mailfrom=da.gomez@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707488947; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CVr+BSsSvYqyUxoK+Z2ICaYEZFi6sZ1CmkC1b8DH1ws=; b=SGDkOrNVnK1Che6NU/d+du5GWwYwXRCKog4FDC8mjhxLf028xNq6hxUJ3UaO0tLxx2TAij 9iIZjBbijbSvN/OapXvMHPMplx+sce4QTtR7uJMWilyNqUsKpYVZGvRZXO6AY/gWFmdqHY Bdj6TLyVP0WBOLefO4mHNoyzolpzqzs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707488947; a=rsa-sha256; cv=none; b=coJWkBrGGeMAgi330BcfBGENurf7Cetf3VZ+jf7yThfNYe9DuLsuXidULRRMuVSFtXA20i IGtGOhv+xgqgkBh/h9bqOrMOoUyNSbWftukKh7PC7Wv9ucREkAezu9copkNRBpkuudykns 1Q1xdS+vWIN2a/IFYeoMwmofnBYt444= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=Pb7wORK5; spf=pass (imf12.hostedemail.com: domain of da.gomez@samsung.com designates 210.118.77.11 as permitted sender) smtp.mailfrom=da.gomez@samsung.com; dmarc=pass (policy=none) header.from=samsung.com Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20240209142905euoutp01ef5ee83ca9d06d23b1847668e7dacb99~yOCYvslgc2699226992euoutp01t for ; Fri, 9 Feb 2024 14:29:05 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20240209142905euoutp01ef5ee83ca9d06d23b1847668e7dacb99~yOCYvslgc2699226992euoutp01t DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1707488945; bh=CVr+BSsSvYqyUxoK+Z2ICaYEZFi6sZ1CmkC1b8DH1ws=; h=From:To:CC:Subject:Date:In-Reply-To:References:From; b=Pb7wORK50Qp+0MacIA0fkdKvzFt9bF4rsGSIhRPNRlT0NZm2ww6wtDMVP4Pd5chC7 I5mWfcqo2MrXuaJZKPqJG5CxmvzRturYToAAG4Q1z+8C9hwqNzrnL7oFcLbjPd0/fC Y8ygeYk3rQ5RfOmWXecirDYjuqC317lWLAdiHFgA= Received: from eusmges2new.samsung.com (unknown [203.254.199.244]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20240209142905eucas1p2574b31587e71bdc31a2993449e51ba90~yOCYaW4Yr0060200602eucas1p26; Fri, 9 Feb 2024 14:29:05 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges2new.samsung.com (EUCPMTA) with SMTP id B9.F5.09814.0B636C56; Fri, 9 Feb 2024 14:29:04 +0000 (GMT) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p2.samsung.com (KnoxPortal) with ESMTPA id 20240209142904eucas1p20a388be8e43b756b84b5a586d5a88f18~yOCX8K7xm2346823468eucas1p2R; Fri, 9 Feb 2024 14:29:04 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20240209142904eusmtrp22b43289d14a97adda5bf8a1bb0345841~yOCX7eDvn2106521065eusmtrp2p; Fri, 9 Feb 2024 14:29:04 +0000 (GMT) X-AuditID: cbfec7f4-711ff70000002656-34-65c636b071cb Received: from eusmtip2.samsung.com ( [203.254.199.222]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id 67.6F.09146.0B636C56; Fri, 9 Feb 2024 14:29:04 +0000 (GMT) Received: from CAMSVWEXC01.scsc.local (unknown [106.1.227.71]) by eusmtip2.samsung.com (KnoxPortal) with ESMTPA id 20240209142904eusmtip20cf00b0cf7880a505516226c14ba4db4~yOCXvA7qt0292502925eusmtip2c; Fri, 9 Feb 2024 14:29:04 +0000 (GMT) Received: from CAMSVWEXC02.scsc.local (2002:6a01:e348::6a01:e348) by CAMSVWEXC01.scsc.local (2002:6a01:e347::6a01:e347) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 9 Feb 2024 14:29:03 +0000 Received: from CAMSVWEXC02.scsc.local ([::1]) by CAMSVWEXC02.scsc.local ([fe80::3c08:6c51:fa0a:6384%13]) with mapi id 15.00.1497.012; Fri, 9 Feb 2024 14:29:03 +0000 From: Daniel Gomez To: "viro@zeniv.linux.org.uk" , "brauner@kernel.org" , "jack@suse.cz" , "hughd@google.com" , "akpm@linux-foundation.org" CC: "dagmcr@gmail.com" , "linux-fsdevel@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , "willy@infradead.org" , "hch@infradead.org" , "mcgrof@kernel.org" , Pankaj Raghav , "gost.dev@samsung.com" , "Daniel Gomez" Subject: [RFC PATCH 2/9] shmem: add per-block uptodate tracking for hugepages Thread-Topic: [RFC PATCH 2/9] shmem: add per-block uptodate tracking for hugepages Thread-Index: AQHaW2RT4T3ZrPeWjkOR05yD13efEQ== Date: Fri, 9 Feb 2024 14:29:02 +0000 Message-ID: <20240209142901.126894-3-da.gomez@samsung.com> In-Reply-To: <20240209142901.126894-1-da.gomez@samsung.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [106.110.32.67] MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDKsWRmVeSWpSXmKPExsWy7djP87obzI6lGjx/ZW4xZ/0aNovXhz8x Wpzt+81mcXrCIiaLp5/6WCxmT29mstiz9ySLxeVdc9gs7q35z2pxY8JTRovzf4+zWvz+MYfN gcdj56y77B4LNpV6bF6h5bFpVSebx6ZPk9g9Tsz4zeJxZsERdo/Pm+Q8Nj15yxTAGcVlk5Ka k1mWWqRvl8CV0br+N3vBB+eKC0/nMzUwbjfvYuTgkBAwkfjclNTFyMUhJLCCUaJv7VZ2COcL o8SLhtlQzmdGiZ+nrzLDdPStsYaIL2eU2HtuNyNc0ceXv5ghnNOMErs/vmHpYuSEGHzojgiI zSagKbHv5CawsSICzxklWnd/BHOYBW4zS8xpn8UIUiUs4C8xf8UlJhBbRCBEovP2eShbT2Lf z3XsIDaLgIrEgqX3mUFsXgEriedr74LZnALWEvMerWcDsRkFZCUerfwFVs8sIC5x68l8sDkS AoISi2bvYYawxST+7XrIBmHrSJy9/oQRwjaQ2Lp0HwuErSjRcewmG8QcPYkbU6dA2doSyxa+ hrpBUOLkzCcsIM9ICPRxSVx7eowNEmAuEuc6OCDmCEu8Or6FHcKWkfi/cz7TBEbtWUjOm4Vk xSwkK2YhWbGAkWUVo3hqaXFuemqxUV5quV5xYm5xaV66XnJ+7iZGYII7/e/4lx2My1991DvE yMTBeIhRgoNZSYQ3ZMmRVCHelMTKqtSi/Pii0pzU4kOM0hwsSuK8qinyqUIC6YklqdmpqQWp RTBZJg5OqQamlZrvfb687S/omfZp+oWYROanaW4f3BVuP62cNpHzc9QjxV0qB+LepUo18JQE yS8Q3eziu+n30Q9l7tGSNhmisXZpqicXTl+zqSz0+UdGkR3cTvP2rsnbyJ8RprqtfnWEXzbr WxWF9ONHZvbPEVk0m2vP5I5L/RduC/xZXMrFyGX8Yft79tN+3/b/09p55GLj8ob4GU7rmbqO 1X8WfNT2vfv8xW+v334J/l+xRuRrrtu291K9d5g+nFnBfP3/7RKXe+kRMUpqTFGt4Ztsshav rYt5dXpb7nm7w4uOrgx692p/zZb+sq4lLw4m2GfeuPvihHVddmtx6h271RUqb6QvfHblvpRS LbDDLEu8JOztGSWW4oxEQy3mouJEAL2AJszfAwAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrJKsWRmVeSWpSXmKPExsVy+t/xe7obzI6lGmy+LmExZ/0aNovXhz8x Wpzt+81mcXrCIiaLp5/6WCxmT29mstiz9ySLxeVdc9gs7q35z2pxY8JTRovzf4+zWvz+MYfN gcdj56y77B4LNpV6bF6h5bFpVSebx6ZPk9g9Tsz4zeJxZsERdo/Pm+Q8Nj15yxTAGaVnU5Rf WpKqkJFfXGKrFG1oYaRnaGmhZ2RiqWdobB5rZWSqpG9nk5Kak1mWWqRvl6CX0br+N3vBB+eK C0/nMzUwbjfvYuTgkBAwkehbY93FyMkhJLCUUeLqfH4QW0JARmLjl6usELawxJ9rXWxdjFxA NR8ZJWZObWSEcE4zSpzecpkVwlnBKPF/9wRmkBY2AU2JfSc3sYMkRASeMkpM/32IBcRhFrjN LDGnfRYjSJWwgK/EtAn3wGwRgRCJ5TOvsELYehL7fq5jB7FZBFQkFiy9DzaVV8BK4vnau8wQ x1pJTNt+mg3E5hSwlpj3aD2YzSggK/Fo5S+wXmYBcYlbT+YzQTwhILFkz3lmCFtU4uXjf1DP 6Uicvf6EEcI2kNi6dB8LhK0o0XHsJhvEHD2JG1OnQNnaEssWvoa6R1Di5MwnLBMYpWchWTcL ScssJC2zkLQsYGRZxSiSWlqcm55bbKhXnJhbXJqXrpecn7uJEZimth37uXkH47xXH/UOMTJx MB5ilOBgVhLhDVlyJFWINyWxsiq1KD++qDQntfgQoykwjCYyS4km5wMTZV5JvKGZgamhiZml gamlmbGSOK9nQUeikEB6YklqdmpqQWoRTB8TB6dUA9PO7ucnjn7xaNVfPk3bZgd/r0FMobJT 7+XuhjneZ1ZdUTZ4WvTtzaL0FTtE2i6eEFu2tEPxQlXulwWLPl19evPln+2RK+oCIkLlDK+x M64uSAj9V/7n4vyqaE5elaQik6u2Cjx7k+f5NkT9/Lbu0LH+2U0OXLcn/OBfxXzCptRTiTl9 27ybP6K76wwtX6v13L1kKR4kdPqOyMHvct+8nl9caPH1rZfoYZ+KpeI8yQ8nJCxYeEhg0SXW jltWfVHhGf5LiyUst04WnrS2dqq8iNWh/A9Pnr2XO62jfcTUrs/UOZ/pwmnZmBszm07ZRTwJ dtOR49/2+mliieDyU5c9l0ZOeqym/8ndMv/S3Dzu1FtKLMUZiYZazEXFiQBXdoA03AMAAA== X-CMS-MailID: 20240209142904eucas1p20a388be8e43b756b84b5a586d5a88f18 X-Msg-Generator: CA X-RootMTR: 20240209142904eucas1p20a388be8e43b756b84b5a586d5a88f18 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20240209142904eucas1p20a388be8e43b756b84b5a586d5a88f18 References: <20240209142901.126894-1-da.gomez@samsung.com> X-Rspamd-Queue-Id: DD99340023 X-Rspam-User: X-Stat-Signature: oinxntmi7k9onw5gpxhab81i96yz8hgi X-Rspamd-Server: rspam03 X-HE-Tag: 1707488946-100296 X-HE-Meta: U2FsdGVkX1879PVFtKyaQNBIQGzUagcCaiQqZtzLCYVUXmUJwNKW7408MnM64aXmW9zmnHo3g/5meDx1oyAqfe6/L5bagIVA0jV54RT+ZsFywW6WBif1TZP8xrRvYS/YVGpHaZP/BMTTQwiAUQufJuWZZJ7g9jwPvFom64bEHgOfx9Gx/HfJcWwa+InjtkBwRqnsESFf2iiAodu9Q84yeQkpgJ9ZNAs4PyzThK+W0DL6M/cM09iNTTxH0cO0i8UxuBAu2Ga+prcrC7J5w1yAntb43jfvsGAw+zCMH50GA87ZObxE5iToj47edZvvbbJUeo12T46qFPtezOF4k67v6dz3JrIbuGa6xHTr4zzXm1YFGXEJV9aP+/leEU3PmdmnuudCQVWzuRJNWlC6WI3dbGAhvPb7aChzu9CPgjMPcEVCnbfs5gnMg3y86+zc8T5rnqYtbtVCtkfEc6jI7Z+hGIjm2Nv/kF940Rxac8xIsDwTRxw0DN0GCPQWBvE1dZ9nkHsUvl4u0+2jIYOlY1kiB1a0sWphQZZseH3z6aTqtMI5sLDuFvOvaB+EThryX6l4irjbCbL1po0nw6cjyBCIzIqkgydem1zw6jXb0CwvcgoL5u6wsBWNrIxAiqEaofbJWhhJtx+/mrSAk5u6Vo8tTWZDEdkbDXa4K+GdFjkiOtEYVzcfLoFQU2RktaOOMOaMp9CRx3JlgPRto7mtxzOHnToje8+Db33dcxL0SE1XjHL1LJe+3/9fHiBsAvHp0kQ1u9vaMI74LZeJ5nNNw3tbKI5lMew6RY/HbH83Pdi4h67vqzUGuzpuzs3uvgubPfFuXPwrAn5x0r0LlanE/1B1+QKOE7Q1vKLmvSCWnatDqisXrzCJpI5unP7cqmSMvyii3a4EHWyRPd71aryoTC6NP9yUw5ZxD6ty5U2Y8MJkXXEGS+ugN5aaTRwwik2FfriyTriA7p/PbTTzbtsPRLz 6uQ8CmxD Zt07qV6JFNIGEs1QG5U+gn8yBG+V0lINckHYtf/WJL8wQi4wAVkTtkDROrmEnadn28faxU8le3fo6rW9wxeBbxokLAkR01kExjOFoS7FulCBZNt2OC2u1nCE2T0lGAcbZqgOey+WHbtviRmV25iO2CEeKwKcP6S0QOpex X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Based on iomap per-block dirty and uptodate state track, add support for shmem_folio_state struct to track the uptodate state per-block for large folios (hugepages). Signed-off-by: Daniel Gomez --- mm/shmem.c | 195 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 189 insertions(+), 6 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index d7c84ff62186..5980d7b94f65 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -131,6 +131,124 @@ struct shmem_options { #define SHMEM_SEEN_QUOTA 32 }; +/* + * Structure allocated for each folio to track per-block uptodate state. + * + * Like buffered-io iomap_folio_state struct but only for uptodate. + */ +struct shmem_folio_state { + spinlock_t state_lock; + unsigned long state[]; +}; + +static inline bool sfs_is_fully_uptodate(struct folio *folio) +{ + struct inode *inode = folio->mapping->host; + struct shmem_folio_state *sfs = folio->private; + + return bitmap_full(sfs->state, i_blocks_per_folio(inode, folio)); +} + +static inline bool sfs_is_block_uptodate(struct shmem_folio_state *sfs, + unsigned int block) +{ + return test_bit(block, sfs->state); +} + +/** + * sfs_get_last_block_uptodate - find the index of the last uptodate block + * within a specified range + * @folio: The folio + * @first: The starting block of the range to search + * @last: The ending block of the range to search + * + * Returns the index of the last uptodate block within the specified range. If + * a non uptodate block is found at the start, it returns UINT_MAX. + */ +static unsigned int sfs_get_last_block_uptodate(struct folio *folio, + unsigned int first, + unsigned int last) +{ + struct inode *inode = folio->mapping->host; + struct shmem_folio_state *sfs = folio->private; + unsigned int nr_blocks = i_blocks_per_folio(inode, folio); + unsigned int aux = find_next_zero_bit(sfs->state, nr_blocks, first); + + /* + * Exceed the range of possible last block and return UINT_MAX if a non + * uptodate block is found at the beginning of the scan. + */ + if (aux == first) + return UINT_MAX; + + return min_t(unsigned int, aux - 1, last); +} + +static void sfs_set_range_uptodate(struct folio *folio, + struct shmem_folio_state *sfs, size_t off, + size_t len) +{ + struct inode *inode = folio->mapping->host; + unsigned int first_blk = off >> inode->i_blkbits; + unsigned int last_blk = (off + len - 1) >> inode->i_blkbits; + unsigned int nr_blks = last_blk - first_blk + 1; + unsigned long flags; + + spin_lock_irqsave(&sfs->state_lock, flags); + bitmap_set(sfs->state, first_blk, nr_blks); + if (sfs_is_fully_uptodate(folio)) + folio_mark_uptodate(folio); + spin_unlock_irqrestore(&sfs->state_lock, flags); +} + +static struct shmem_folio_state *sfs_alloc(struct inode *inode, + struct folio *folio) +{ + struct shmem_folio_state *sfs = folio->private; + unsigned int nr_blocks = i_blocks_per_folio(inode, folio); + gfp_t gfp = GFP_KERNEL; + + if (sfs || nr_blocks <= 1) + return sfs; + + /* + * sfs->state tracks uptodate flag when the block size is smaller + * than the folio size. + */ + sfs = kzalloc(struct_size(sfs, state, BITS_TO_LONGS(nr_blocks)), gfp); + if (!sfs) + return sfs; + + spin_lock_init(&sfs->state_lock); + if (folio_test_uptodate(folio)) + bitmap_set(sfs->state, 0, nr_blocks); + folio_attach_private(folio, sfs); + + return sfs; +} + +static void sfs_free(struct folio *folio, bool force) +{ + if (!folio_test_private(folio)) + return; + + if (!force) + WARN_ON_ONCE(sfs_is_fully_uptodate(folio) != + folio_test_uptodate(folio)); + + kfree(folio_detach_private(folio)); +} + +static void shmem_set_range_uptodate(struct folio *folio, size_t off, + size_t len) +{ + struct shmem_folio_state *sfs = folio->private; + + if (sfs) + sfs_set_range_uptodate(folio, sfs, off, len); + else + folio_mark_uptodate(folio); +} #ifdef CONFIG_TMPFS static unsigned long shmem_default_max_blocks(void) { @@ -1487,7 +1605,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) } folio_zero_range(folio, 0, folio_size(folio)); flush_dcache_folio(folio); - folio_mark_uptodate(folio); + shmem_set_range_uptodate(folio, 0, folio_size(folio)); } swap = folio_alloc_swap(folio); @@ -1769,13 +1887,16 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, if (!new) return -ENOMEM; + if (folio_get_private(old)) + folio_attach_private(new, folio_detach_private(old)); + folio_get(new); folio_copy(new, old); flush_dcache_folio(new); __folio_set_locked(new); __folio_set_swapbacked(new); - folio_mark_uptodate(new); + shmem_set_range_uptodate(new, 0, folio_size(new)); new->swap = entry; folio_set_swapcache(new); @@ -2060,6 +2181,12 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, alloced: alloced = true; + + if (!sfs_alloc(inode, folio) && folio_test_large(folio)) { + error = -ENOMEM; + goto unlock; + } + if (folio_test_pmd_mappable(folio) && DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE) < folio_next_index(folio) - 1) { @@ -2101,7 +2228,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, for (i = 0; i < n; i++) clear_highpage(folio_page(folio, i)); flush_dcache_folio(folio); - folio_mark_uptodate(folio); + shmem_set_range_uptodate(folio, 0, folio_size(folio)); } /* Perhaps the file has been truncated since we checked */ @@ -2746,8 +2873,8 @@ shmem_write_end(struct file *file, struct address_space *mapping, folio_zero_segments(folio, 0, from, from + copied, folio_size(folio)); } - folio_mark_uptodate(folio); } + shmem_set_range_uptodate(folio, 0, folio_size(folio)); folio_mark_dirty(folio); folio_unlock(folio); folio_put(folio); @@ -2755,6 +2882,59 @@ shmem_write_end(struct file *file, struct address_space *mapping, return copied; } +static void shmem_invalidate_folio(struct folio *folio, size_t offset, + size_t len) +{ + /* + * If we're invalidating the entire folio, clear the dirty state + * from it and release it to avoid unnecessary buildup of the LRU. + */ + if (offset == 0 && len == folio_size(folio)) { + WARN_ON_ONCE(folio_test_writeback(folio)); + folio_cancel_dirty(folio); + sfs_free(folio, true); + } +} + +static bool shmem_release_folio(struct folio *folio, gfp_t gfp_flags) +{ + if (folio_test_dirty(folio) && !sfs_is_fully_uptodate(folio)) + return false; + + sfs_free(folio, false); + return true; +} + +/* + * shmem_is_partially_uptodate checks whether blocks within a folio are + * uptodate or not. + * + * Returns true if all blocks which correspond to the specified part + * of the folio are uptodate. + */ +static bool shmem_is_partially_uptodate(struct folio *folio, size_t from, + size_t count) +{ + struct shmem_folio_state *sfs = folio->private; + struct inode *inode = folio->mapping->host; + unsigned int first, last; + + if (!sfs) + return false; + + /* Caller's range may extend past the end of this folio */ + count = min(folio_size(folio) - from, count); + + /* First and last blocks in range within folio */ + first = from >> inode->i_blkbits; + last = (from + count - 1) >> inode->i_blkbits; + + if (sfs_get_last_block_uptodate(folio, first, last) != last) + return false; + + return true; +} + static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) { struct file *file = iocb->ki_filp; @@ -3506,7 +3686,7 @@ static int shmem_symlink(struct mnt_idmap *idmap, struct inode *dir, inode->i_mapping->a_ops = &shmem_aops; inode->i_op = &shmem_symlink_inode_operations; memcpy(folio_address(folio), symname, len); - folio_mark_uptodate(folio); + shmem_set_range_uptodate(folio, 0, folio_size(folio)); folio_mark_dirty(folio); folio_unlock(folio); folio_put(folio); @@ -4476,7 +4656,10 @@ const struct address_space_operations shmem_aops = { #ifdef CONFIG_MIGRATION .migrate_folio = migrate_folio, #endif - .error_remove_folio = shmem_error_remove_folio, + .error_remove_folio = shmem_error_remove_folio, + .invalidate_folio = shmem_invalidate_folio, + .release_folio = shmem_release_folio, + .is_partially_uptodate = shmem_is_partially_uptodate, }; EXPORT_SYMBOL(shmem_aops);