From patchwork Fri Oct 18 03:00:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13841045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8992D3C545 for ; Fri, 18 Oct 2024 03:00:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BC976B008A; Thu, 17 Oct 2024 23:00:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 54AD06B0095; Thu, 17 Oct 2024 23:00:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 370766B0088; Thu, 17 Oct 2024 23:00:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1264D6B008C for ; Thu, 17 Oct 2024 23:00:43 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BE4F01C750E for ; Fri, 18 Oct 2024 03:00:29 +0000 (UTC) X-FDA: 82685220030.12.24074FD Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) by imf27.hostedemail.com (Postfix) with ESMTP id 5CB6940008 for ; Fri, 18 Oct 2024 03:00:29 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=fSbvC9HG; spf=pass (imf27.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729220367; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ynn20XvNY9Nu0PNOg+bwufz+Xr8j8cPFzYmz1b4byuY=; b=Hf7H0hnIMUCUb8iVnKapwvPtjaIQWAXnFNCTWudqj2eqnFj95LUqDpe9JcAiVXz6lBWYcQ w4WwXLRYOg5cVAGvh5N3YKZnji6HNLJ8qZZOho7as5+ShKoLAOK3wEh/sn0czuvgU5UXGs Lux4kWwsIpAN7r6On7WMEgsRt1hLytE= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=fSbvC9HG; spf=pass (imf27.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.118 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729220367; a=rsa-sha256; cv=none; b=GzcGLi80njHT6cz8rPqYuw9q4J35Xf2JOPu29cgf4ic73OR/OIuzXX22shfMgOJ01Uoo0r 76BXUbQ0EpW8BzU2vQaKxLKP5iE7T4TgRUEuUapcyV4l7yuxxZCU8kGogD7Eh+iM+McpeC C5RUAy8UcsQ6cgjvJ7inSR0eBe7TRFg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1729220437; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=Ynn20XvNY9Nu0PNOg+bwufz+Xr8j8cPFzYmz1b4byuY=; b=fSbvC9HG5suhD6RfCJyVwKm6oI611x8f60IpX/UgLJCht54xt8d0EGR+MXXr6FIWjkVeeVIe2eG97RtW37v10/A7iRvYQffqj0M4LRXIj7jjaEIYklbS5UUfNttlEjBbyxe61TegTswvyA4JqaJqQDUIYZXYHQI9R4Zm6RACfag= Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0WHMZWVd_1729220435 cluster:ay36) by smtp.aliyun-inc.com; Fri, 18 Oct 2024 11:00:36 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, shy828301@gmail.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/2] mm: shmem: update iocb->ki_pos directly to simplify tmpfs read logic Date: Fri, 18 Oct 2024 11:00:27 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 5CB6940008 X-Stat-Signature: kb65ht5tzshm7yp3c6y989nemd8p11m1 X-HE-Tag: 1729220429-927461 X-HE-Meta: U2FsdGVkX1+0gutO3meIAUHWJoyQRD+BCw09I8/yUnWbCvdFc25wRTV9Si1qU0zCMxMpOBIx8gyvBTW/ns0uPzn+LYzbyJ4dDGREhwwmq0lWVUj1YpVi7Fn7bhehTCGUYqjPeH16m5mNGNdjp0pArk1K0vtl2RnJryOeT1zxE63+OdtC2vQjAZeselwXF5yzVm5sJJzAjik0BWIIxcEQbe/Io1CfAfEPeyhn95l9IaLv6PR9Qg1RYhlxbKZrCSlU2OnP8iDpsbuXz/KAIIWhp76CtVVbomLvXAR5stHPhM/HbtbzIMEFFQShXFXrjpvrTiiJdKo6+JMLSZPbFkGKcRCT4VsMFscy+5XcKW1RBoorJFzmJntTngZLgz5LZjLLP8whrSh6a4/dGH4D8fA8GFnoaZBB+4CcT2zq/ExGSoWmgVj0v9tONOyYt2jh3Dv7XB+hsqlgVnt/CSe8y22xCJiLDjAOaKUTOFxxXjhgJ+VTVwiZj/ilcxHXRJb6NMQHRW+ibcZLkW3OtQhNZpxvt1GKYjy0Ymue4PYeYb+LOdUjusqTQ0ifAGszcRbzzBUu9GqlkdHI3Iy0Kt1+pgMWCWRgvSa16T6oElzjMukl0jSCidSMOLnpKJISDqeuWFDxA82d5eA7gn078atjnPOKaSvnSWdZLrJFZX1PkugZF6AQbhKyPh/nhMFHmA1WjFado7yAbBMN0TRAmC+n6XaY2XLowawXkrB+HwP1YZeEkjVPdK2qpqtl/W3d3BY+Ssjd+b0wwQtvteDoTkg9tDlv5uARbGJKy9Xw/7RN5GZAJTPWKw6lBeiIbpbFkbC1NhlvG2arBapb3vH/PTzFkp/ZTS8OgeLiHulVkaYcyjPWMOPts/Euqqi5kC8ylAgaAcw/uIbBmW4r9iqZtC56qNIUXsXnvKw0meAPe0oIb+5+2r0mAEntBR9xvYuByvv3G6xO6E7UrARnBOlrTVc1S2o 5dmcj3uG W5J+IfcHDmKI04zSEHstmNNAOhFrGpxrIe8WR6Bg8udyhxxByelqGZzSDgeqaev5ItreAN/h1yQVJQK/9QU5qCOGO4gmhhPfsBBG6SeT+BDrcpR2rhByzYsc1QkUURR+PCoD25ngdyVMxpDpQN+tBhNr9gByirN/771SOwBrXriRaLldl1Pt/votXN3F4YKa6kgeUDSTLJ/AYdiiYa4LRbzoqNR4Uy8n9GsFb80WaJ657ZvYRgw+PvRjhjVX23WryY9QXZIxis03R7Z5HIM/jJPd6KA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use iocb->ki_pos to check if the read bytes exceeds the file size and to calculate the bytes to be read can help simplify the code logic. Meanwhile, this is also a preparation for improving tmpfs large folios read performace in the following patch. Signed-off-by: Baolin Wang --- mm/shmem.c | 35 +++++++++++------------------------ 1 file changed, 11 insertions(+), 24 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 66eae800ffab..93642aa8d1aa 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3106,27 +3106,19 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) unsigned long offset; int error = 0; ssize_t retval = 0; - loff_t *ppos = &iocb->ki_pos; - index = *ppos >> PAGE_SHIFT; - offset = *ppos & ~PAGE_MASK; + offset = iocb->ki_pos & ~PAGE_MASK; for (;;) { struct folio *folio = NULL; struct page *page = NULL; - pgoff_t end_index; unsigned long nr, ret; - loff_t i_size = i_size_read(inode); + loff_t end_offset, i_size = i_size_read(inode); - end_index = i_size >> PAGE_SHIFT; - if (index > end_index) + if (unlikely(iocb->ki_pos >= i_size)) break; - if (index == end_index) { - nr = i_size & ~PAGE_MASK; - if (nr <= offset) - break; - } + index = iocb->ki_pos >> PAGE_SHIFT; error = shmem_get_folio(inode, index, 0, &folio, SGP_READ); if (error) { if (error == -EINVAL) @@ -3148,18 +3140,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) * We must evaluate after, since reads (unlike writes) * are called without i_rwsem protection against truncate */ - nr = PAGE_SIZE; i_size = i_size_read(inode); - end_index = i_size >> PAGE_SHIFT; - if (index == end_index) { - nr = i_size & ~PAGE_MASK; - if (nr <= offset) { - if (folio) - folio_put(folio); - break; - } + if (unlikely(iocb->ki_pos >= i_size)) { + if (folio) + folio_put(folio); + break; } - nr -= offset; + end_offset = min_t(loff_t, i_size, iocb->ki_pos + to->count); + nr = min_t(loff_t, end_offset - iocb->ki_pos, PAGE_SIZE - offset); if (folio) { /* @@ -3199,8 +3187,8 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) retval += ret; offset += ret; - index += offset >> PAGE_SHIFT; offset &= ~PAGE_MASK; + iocb->ki_pos += ret; if (!iov_iter_count(to)) break; @@ -3211,7 +3199,6 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) cond_resched(); } - *ppos = ((loff_t) index << PAGE_SHIFT) + offset; file_accessed(file); return retval ? retval : error; } From patchwork Fri Oct 18 03:00:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13841044 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D20ED3C544 for ; Fri, 18 Oct 2024 03:00:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3740D6B0093; Thu, 17 Oct 2024 23:00:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AC5E6B0092; Thu, 17 Oct 2024 23:00:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DDEC6B008A; Thu, 17 Oct 2024 23:00:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D3A7B6B0085 for ; Thu, 17 Oct 2024 23:00:42 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0758B120521 for ; Fri, 18 Oct 2024 03:00:32 +0000 (UTC) X-FDA: 82685219694.20.7868612 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) by imf10.hostedemail.com (Postfix) with ESMTP id D1263C0012 for ; Fri, 18 Oct 2024 03:00:35 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=e3T2xx1h; spf=pass (imf10.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729220294; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZL+DuJ+RP/aBC8VbvHjJ2i2qhBXaPyZwPwmHC2i5Ozk=; b=XjenpQu3nCm0stBY6cbUoWpX99y08fiBs7Jgx1oUsoHUZjk++o5oy4nLVYraSvF+4QlzAF vQVWb0RMB0wRDrx5hMGclVoAK/J85fr5qj/MsPRZhJLllpkCZ/saZdo8pARRpjeY5Zvhwp 7EDWqIQUyJPAFYu3wo1sSlhJUIPYCQ8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729220294; a=rsa-sha256; cv=none; b=mHsXtYQpmNoeKTwqjzj6SO5BlLYdxZdOI0HalzYj1wstY/9zm/3x26IFTvr7rXqgcdhVnp AILhGwz9rc0O6oGJUHGfgixnOi/zrTpQ28JvM+qBq9N6I7JMarLVfgFj95mpOiHfknY0iG DHAAB2t6U/GiR3XWZPx8LOgofzBFyzg= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=e3T2xx1h; spf=pass (imf10.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1729220437; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=ZL+DuJ+RP/aBC8VbvHjJ2i2qhBXaPyZwPwmHC2i5Ozk=; b=e3T2xx1hFmunyeuiIHLvsNZipxS3GpW7M+65XqNmcSNxcNmN4ib6zKrDOP/zVC/cM7vOctKe2N0eu/GHozWprOPyScw3necBXGl3rhbrqLe6FVndSymYpWJMprRyrs7jIE3K5WPPlDIgnB4I6QtnTKyFbT9KrPWboNIBdfdRG9M= Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0WHMdEgo_1729220436 cluster:ay36) by smtp.aliyun-inc.com; Fri, 18 Oct 2024 11:00:36 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, shy828301@gmail.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/2] mm: shmem: improve the tmpfs large folio read performance Date: Fri, 18 Oct 2024 11:00:28 +0800 Message-Id: <2129a21a5b9f77d3bb7ddec152c009ce7c5653c4.1729218573.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D1263C0012 X-Stat-Signature: ynis1nkummhnytteburunhczs3pjup8z X-HE-Tag: 1729220435-356148 X-HE-Meta: U2FsdGVkX1+b8olvfdsT0Db6hlz8Uke35HivaBko93OuJYdktk++M6t8lmwEBWxYQOXdsajtlXnQrMpGGP7z3xOe2/8VtReZ2+N6sPkwpFHgZg1S1lRHGmNLTkE+0q75/nTKGYdexZT/I/qQHJwOF/JItU3B5QfdnzWfvbyG3+p35WYEkcHm9K6qes0WCpT+RElIb8NPcqXCzcGj5LkXSwO3LaP/2RYnNHQq7dXa3J5HxNulFmXrdPXletQj+Ek0DEKwBN+SPECyTB3pM1Xsusq2vg/tIRtHMz22dNr43ag03MttFtGWbSwWeGfN2Bc5fGdhm9XvPhkQ+vN8+V5Z+2cy7Fhkqqkeqokg1MZUaJ3KIYbBN+q0Wa5WPi8WPwEZezbL9h/koFXJ+Vxceo5Cl5AAhH2hjCNTU0TlTVrvU3dkEsniS1CPvWfHQyGb4Z1FrtpVSoA1KQGvo9LT9lJ+FwVMzVtnZnLvfZ5MRX1fe9TxYjHRXRb1MvkYX83mCKsxVsBIvlvFFlQ2Ef1DNPcOANDH+UB2DTCbqCdctIB0Pvk3j3aROLEJ1BGToV1X6iYTrGK+ZnOPCns3qSM99LowXh7PDnFHxoU0TF3jYjNM+l/BS2UdKM+Lc9WVaBSKQqOFTs6WKFYuuNYACYNI0F51NY1YR9CdeU8pjzEVaCrkRc22QiccZdsBnS3KPOECU66x25n+26kggSjZC1uNKp5EnDYsWQw+9vjKFcgVD45vc0aLdKfowdGlbYsZB/aMwmIXweZGBInqlzrd3pdlE2J7IcFdm0OFpaSZti+pHgT/9lkyoE8ou5q3B4M8vsxRm43RpRhLFGm92XJvBs6+8ZDn0/gVYtYxIQdpI2xn0+86wcNEk4p4k/RsMWxHVe2a2RzbGtszsj81y3C/NGfQd+PdviG/tQzLndZaMiw8FkMdbrzUIXUcp35MW4Ny3RzNrq50GAonw/4xa/EgJRV9gsj 8O6KsKs5 35qsanVfqn4xYFNUcXKqn8LIF+2j3xZl+KWgfR00/GaRe/sOCTdyyOjN81AGm9d5sSDWuiXlvWlAl7DESy36E6mqo7MMiTezySnCM3j4Mxd0kfQkU9LF3z3+LHA1Rv/rQtGOUQdBCutlfUNnddx1qRt0kmrePZI5EDbtqHA4HAAkjcDQTr0kEslx84zXMQFtOXiR86XxAWESlMrORUaULsVEKpQMVnt60/ulJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The tmpfs has already supported the PMD-sized large folios, but the tmpfs read operation still performs copying at the PAGE SIZE granularity, which is unreasonable. This patch changes to copy data at the folio granularity, which can improve the read performance, as well as changing to use folio related functions. Moreoever, if a large folio has a subpage that is hwpoisoned, it will still fallback to page granularity copying. Use 'fio bs=64k' to read a 1G tmpfs file populated with 2M THPs, and I can see about 20% performance improvement, and no regression with bs=4k. Before the patch: READ: bw=10.0GiB/s After the patch: READ: bw=12.0GiB/s Signed-off-by: Baolin Wang --- mm/shmem.c | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 93642aa8d1aa..cbefd9801f6b 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3107,13 +3107,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) int error = 0; ssize_t retval = 0; - offset = iocb->ki_pos & ~PAGE_MASK; - for (;;) { struct folio *folio = NULL; struct page *page = NULL; unsigned long nr, ret; loff_t end_offset, i_size = i_size_read(inode); + bool fallback_page_copy = false; + size_t fsize; if (unlikely(iocb->ki_pos >= i_size)) break; @@ -3134,6 +3134,10 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) error = -EIO; break; } + + if (folio_test_large(folio) && + folio_test_has_hwpoisoned(folio)) + fallback_page_copy = true; } /* @@ -3147,7 +3151,12 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) break; } end_offset = min_t(loff_t, i_size, iocb->ki_pos + to->count); - nr = min_t(loff_t, end_offset - iocb->ki_pos, PAGE_SIZE - offset); + if (folio && likely(!fallback_page_copy)) + fsize = folio_size(folio); + else + fsize = PAGE_SIZE; + offset = iocb->ki_pos & (fsize - 1); + nr = min_t(loff_t, end_offset - iocb->ki_pos, fsize - offset); if (folio) { /* @@ -3155,10 +3164,15 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) * virtual addresses, take care about potential aliasing * before reading the page on the kernel side. */ - if (mapping_writably_mapped(mapping)) - flush_dcache_page(page); + if (mapping_writably_mapped(mapping)) { + if (likely(!fallback_page_copy)) + flush_dcache_folio(folio); + else + flush_dcache_page(page); + } + /* - * Mark the page accessed if we read the beginning. + * Mark the folio accessed if we read the beginning. */ if (!offset) folio_mark_accessed(folio); @@ -3166,9 +3180,11 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) * Ok, we have the page, and it's up-to-date, so * now we can copy it to user space... */ - ret = copy_page_to_iter(page, offset, nr, to); + if (likely(!fallback_page_copy)) + ret = copy_folio_to_iter(folio, offset, nr, to); + else + ret = copy_page_to_iter(page, offset, nr, to); folio_put(folio); - } else if (user_backed_iter(to)) { /* * Copy to user tends to be so well optimized, but @@ -3186,8 +3202,6 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) } retval += ret; - offset += ret; - offset &= ~PAGE_MASK; iocb->ki_pos += ret; if (!iov_iter_count(to))