From patchwork Sun Sep 8 23:21:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13795689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F916ECE577 for ; Sun, 8 Sep 2024 23:21:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5DC36B00EA; Sun, 8 Sep 2024 19:21:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE7016B00EC; Sun, 8 Sep 2024 19:21:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A393E6B00ED; Sun, 8 Sep 2024 19:21:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 827806B00EA for ; Sun, 8 Sep 2024 19:21:46 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 10C451A0A33 for ; Sun, 8 Sep 2024 23:21:46 +0000 (UTC) X-FDA: 82543145412.26.19BB777 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf24.hostedemail.com (Postfix) with ESMTP id 3906218001B for ; Sun, 8 Sep 2024 23:21:44 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SBF1obHn; spf=pass (imf24.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725837570; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R2xC+tIjHn2wWvMR2T1OflcDy+oEzjuKFqq/trb+O1U=; b=RutU+ai8P6zwOtR9hCoiN3YM7og21ZgsbCbSjh/hWuQp7L9MgEY2Ai3f8S9Fryhn5lR/SX frk2oaWC3pIs54B/WmYuvfrKsccPzWMiou/eqeRdTzyeahWsn9c/X4c9cNfWjWH37EC3ZM iFTEfLtbtXn/vPHmaZ1nziUx5aNgpFM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SBF1obHn; spf=pass (imf24.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725837570; a=rsa-sha256; cv=none; b=3tuM6N7Md+RZLzh33+cWBiu8QinvreRoPPoFqYJuWVZvPa6t8s/1ZlA9pOqVu4aYQGqJuA 46by8oZQ0s9lkaDApLbWWA31Qb1hfy8nmnZrvGRZqSVWi8aX7vK8pCz3kGZ6b4S3wk3h9W lAqtJOkyR6rDqXUnodZr7A3myyEsfis= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-20696938f86so31433465ad.3 for ; Sun, 08 Sep 2024 16:21:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725837703; x=1726442503; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R2xC+tIjHn2wWvMR2T1OflcDy+oEzjuKFqq/trb+O1U=; b=SBF1obHnsAhKR1FQy8TRWjwyLoN+nGB0vgq9oNZKVaVQEWr5KWHStjCyGBEnqs5XLn pspFqky5GYT/w27OOWn3YDo0ccYP1aVw7rge8H0+cKFWOzV78RN6tAWJuvoScYQvkyI2 Q9aA4HuZTpjU6eX+JtO75Ej0oDZ2qHKDTSwt/C4VpuNESP9kjTBByxIL5wVah7KC4s/m uAqvGu1WKC5k3Dl3X/Va4baU9lNBme0JxzCqcHFGr0ORxja9/1WVEhU505T0Ank+treG I5QWWhXIWadf2lVyk8gA1Pdlswt4t3lcrffZ/ZpZ1T7JNl7togGhdwNe41yWA8PHEey4 ubjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725837703; x=1726442503; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R2xC+tIjHn2wWvMR2T1OflcDy+oEzjuKFqq/trb+O1U=; b=DlguhM2aZ8yYpo5Fp9D1kJUpGt/FuRW5KHAwupP3ZfLyMAs90uyYlahQS4fGZHEqaQ UuSPqmoWYI1DFR/QaIQUV2C9Q7Dn6Br9YGycfm3s9fLeFANj4cA9O2K1CIMNVHLdUMx2 d4q9fYbtB386NtcNgHskkxGeX7IWPpm6VdvMM21+Oo/1pCADdmqPH6sn9EUdkdK7/Spg 2aSiOQxWB4oU6/t7HtoTKvvb/T+a4wmxsO3g+a2PWYQsvTdenvE645TM0iiVTP6RIXw0 wZ9lCiHOp2JC86cWNdWVI+YaDQYz/0lLa+QFG/qRoaT9cyPpS3h5iFyOZxeJJz+FSPjx Z3BA== X-Forwarded-Encrypted: i=1; AJvYcCXv03WmD/nBrtvuvfcHvhlTZRtXQShnYjjt36qibPybZEdQA38bV11fNqz9FLTJkmQmzVd2bRl1VA==@kvack.org X-Gm-Message-State: AOJu0Yz545NyOfmhQHcIFdv3Ej4q+G5YkeJ/O8WZT/E0Jh2WFy1e4ycv zcAehXnfrx4EYoF+gfCQZr4953uUaN0YUsGwdKSdQOp4P7KUqWgE X-Google-Smtp-Source: AGHT+IFgSDJb5dN2/2g631ifqwqWtDRQhZpgCd9etl+bEZOuXVcAb2DQHY5prsg3HDa8T4Pw/lnH/g== X-Received: by 2002:a05:6a21:4581:b0:1cf:3838:1ec5 with SMTP id adf61e73a8af0-1cf38381efdmr4465870637.5.1725837702701; Sun, 08 Sep 2024 16:21:42 -0700 (PDT) Received: from Barrys-MBP.hub ([2407:7000:8942:5500:ecbd:b95f:f7b2:8158]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-718e58c703asm2447643b3a.82.2024.09.08.16.21.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 08 Sep 2024 16:21:42 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hch@infradead.org, hughd@google.com, kaleshsingh@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, mhocko@suse.com, minchan@kernel.org, nphamcs@gmail.com, ryan.roberts@arm.com, ryncsn@gmail.com, senozhatsky@chromium.org, shakeel.butt@linux.dev, shy828301@gmail.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, Usama Arif Subject: [PATCH v9 1/3] mm: Fix swap_read_folio_zeromap() for large folios with partial zeromap Date: Mon, 9 Sep 2024 11:21:17 +1200 Message-Id: <20240908232119.2157-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240908232119.2157-1-21cnbao@gmail.com> References: <20240908232119.2157-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3906218001B X-Stat-Signature: 1nmk6bso33ndt58qr9fx4oypz436qp39 X-Rspam-User: X-HE-Tag: 1725837704-856591 X-HE-Meta: U2FsdGVkX18p9O4IBnSb+JW3LwbCDgLGFwVt6gOSHarF3ZgLJetUeFfTstgDNAAPpPoqJZvDsFc7F4U0H9dyekwMMayININDtqDLkfU4Cdo8UqJ0m1w6QX0gjiwp4sk3HXKRKET7thmRejAvHcv3llsicSoYZC00gPNZnQSuknhdgGkrpNAAiKYJBiMvt91QfKgBbEBCYkZQXMdQ9nKaL44eWRNi6nkyuxDlMq29B+xENRuDd0RutbkPRoJ7JASVbJ5v0rN1UxTR0OKZ9CXlx6oUcuDuYiQNXOK8nphPGwQAItxkcEoAzl62w/oyCxGXRnerVFUhiUEgy1aTqVkiFhkOPnlonwLndCEq9jC07wAhcYOjwcVMkk34RyWsa8++Svhys6qrvMNa8joTzXqBkB+nsMF9/nVwutU3ngzcA9MPi9nu/ctnG6WwDCMXjEsvYkw0tm8fdqca/AJkC04YZTN+b8Mw3ahDeBDl0AyTXfzrVf+pqbcoYp4th8+D6PlbDH7t18ka2xzptS7FTAl5leQg1xmjrerjQg0t0xc3IWOI7BDJXDwUeaT+aEcvn/yQZkC4/yGGM7sd/8Q4jOrqgLI4/SjV+Qg31iTWtARmSK701m/VItqTkLZoCExPuqEAUKotlDM+WRsuQOhsGH834Hhozma91Nlj4rmUSrZ7i6cH5U2j+vB/lEiZLdCX6FkHZxrd/RTkvDOjPlpznyhFSfbst/tOSize5Zu9V//kxQwNA9Jp8/JWHXppeYK20DjdXJvXgijJ8yS7G4t9mPsCJaxaFoX7H+3nw+SMlWi6p8MqOEkWHClbt0v3JURrgc+lVnn8nLnlse3i8WoSlFXMoewLVAUeD5+9tBWxYl/OGN33cYcDXHYn3Sr5ILcFvui6tiTbuKupa/zGTBknbN6HjKikqqyKBlSvoPwp0z1rrUN7rjYyqgWPdfkIIDYDLcj1+uLdUu+44GCJkzAKSg9 FwGhQHML hZAsJRe0pEiZIVejONiLJ5mthH1rn7e6ESuiuqumUeeRyFIKzMBNphO2hKCZ3QkwvQzzzF8yIi6lLg7xFn90x0Y0yZ4r7CKLnHeWhrPODl4sqFfDEGc3Vq2Kp32jlKyXNmn8WFOO/c1sXxZBNkkGKMpjbhxpZVvEgrbhr/RBaXXlOXOvgK0cp+KTsxuGNVMulCYcxQ1QgzAYnvqPs1AcK1gTw8V/EaVdnbJL5qZrqeWl1r77zTdz8dU4ce7ylR1HXtiWqFS8DYmFaDu7iTRNS7yzwyXpkESQTjgIJNGIHIAxaCqfGaBAvvgqoANq4Xa1RXVL70bfgCtgV0ok1b0Pyu4JEzE/nsuW7/0MxgguOI3ze1sOQPK8G+GeQt9KEEs47t159AhK7J7HMUsiUPxamKEKKEbO2k+r3zILByL8t2y02UL5fBwHYiikol5kGxdl4AxKXnfJh7x2yAmacZ/hMjGSPudidSFpowvjy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song There could be a corner case where the first entry is non-zeromap, but a subsequent entry is zeromap. In this case, we should not let swap_read_folio_zeromap() return false since we will still read corrupted data. Additionally, the iteration of test_bit() is unnecessary and can be replaced with bitmap operations, which are more efficient. We can adopt the style of swap_pte_batch() and folio_pte_batch() to introduce swap_zeromap_batch() which seems to provide the greatest flexibility for the caller. This approach allows the caller to either check if the zeromap status of all entries is consistent or determine the number of contiguous entries with the same status. Since swap_read_folio() can't handle reading a large folio that's partially zeromap and partially non-zeromap, we've moved the code to mm/swap.h so that others, like those working on swap-in, can access it. Fixes: 0ca0c24e3211 ("mm: store zero pages to be swapped out in a bitmap") Cc: Usama Arif Reviewed-by: Yosry Ahmed Signed-off-by: Barry Song Reviewed-by: Usama Arif --- mm/page_io.c | 32 +++++++------------------------- mm/swap.h | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 25 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index 4bc77d1c6bfa..bc1183299a7d 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -226,26 +226,6 @@ static void swap_zeromap_folio_clear(struct folio *folio) } } -/* - * Return the index of the first subpage which is not zero-filled - * according to swap_info_struct->zeromap. - * If all pages are zero-filled according to zeromap, it will return - * folio_nr_pages(folio). - */ -static unsigned int swap_zeromap_folio_test(struct folio *folio) -{ - struct swap_info_struct *sis = swp_swap_info(folio->swap); - swp_entry_t entry; - unsigned int i; - - for (i = 0; i < folio_nr_pages(folio); i++) { - entry = page_swap_entry(folio_page(folio, i)); - if (!test_bit(swp_offset(entry), sis->zeromap)) - return i; - } - return i; -} - /* * We may have stale swap cache pages in memory: notice * them here and get rid of the unnecessary final write. @@ -524,19 +504,21 @@ static void sio_read_complete(struct kiocb *iocb, long ret) static bool swap_read_folio_zeromap(struct folio *folio) { - unsigned int idx = swap_zeromap_folio_test(folio); - - if (idx == 0) - return false; + int nr_pages = folio_nr_pages(folio); + bool is_zeromap; /* * Swapping in a large folio that is partially in the zeromap is not * currently handled. Return true without marking the folio uptodate so * that an IO error is emitted (e.g. do_swap_page() will sigbus). */ - if (WARN_ON_ONCE(idx < folio_nr_pages(folio))) + if (WARN_ON_ONCE(swap_zeromap_batch(folio->swap, nr_pages, + &is_zeromap) != nr_pages)) return true; + if (!is_zeromap) + return false; + folio_zero_range(folio, 0, folio_size(folio)); folio_mark_uptodate(folio); return true; diff --git a/mm/swap.h b/mm/swap.h index f8711ff82f84..ad2f121de970 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -80,6 +80,32 @@ static inline unsigned int folio_swap_flags(struct folio *folio) { return swp_swap_info(folio->swap)->flags; } + +/* + * Return the count of contiguous swap entries that share the same + * zeromap status as the starting entry. If is_zeromap is not NULL, + * it will return the zeromap status of the starting entry. + */ +static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr, + bool *is_zeromap) +{ + struct swap_info_struct *sis = swp_swap_info(entry); + unsigned long start = swp_offset(entry); + unsigned long end = start + max_nr; + bool first_bit; + + first_bit = test_bit(start, sis->zeromap); + if (is_zeromap) + *is_zeromap = first_bit; + + if (max_nr <= 1) + return max_nr; + if (first_bit) + return find_next_zero_bit(sis->zeromap, end, start) - start; + else + return find_next_bit(sis->zeromap, end, start) - start; +} + #else /* CONFIG_SWAP */ struct swap_iocb; static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug) @@ -171,6 +197,13 @@ static inline unsigned int folio_swap_flags(struct folio *folio) { return 0; } + +static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr, + bool *has_zeromap) +{ + return 0; +} + #endif /* CONFIG_SWAP */ #endif /* _MM_SWAP_H */