From patchwork Wed May 8 22:40:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13659211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 749FEC04FFE for ; Wed, 8 May 2024 22:41:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03C5A6B008C; Wed, 8 May 2024 18:41:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F2EA96B0093; Wed, 8 May 2024 18:41:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA80D6B0095; Wed, 8 May 2024 18:41:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B929B6B008C for ; Wed, 8 May 2024 18:41:53 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 43B371A0CFD for ; Wed, 8 May 2024 22:41:53 +0000 (UTC) X-FDA: 82096702506.04.5BECF39 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf13.hostedemail.com (Postfix) with ESMTP id 61B782000F for ; Wed, 8 May 2024 22:41:51 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=TLo3C6sa; spf=pass (imf13.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715208111; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DOx+XLJ6NG229/wS5wOB5lVEjUmMQDyjSrW3ljEjCDo=; b=sktrh8FCKDpSGvVSDAqYEkuQatF6mY+wM5T5fUDx1VndgBazZo4hRMzeBa28llVSMzrfaF Kyw8rOB2yNTEbn4pc2JLBk8CkD4GgBjqRMqUX/I4/SScnhQTtzTjWdXBsKHF0PX7r7bC0y EvCwpa5INgz4EDB6g+olTYnfYhOCkf0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=TLo3C6sa; spf=pass (imf13.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715208111; a=rsa-sha256; cv=none; b=1FVlHtcsJMMuOSdjBBGdaI4zhSDgG+ckxd7VQTMunVWxLuTrVR9yTMItsFZXE61W1k2BIC g+udscXSn3dc/bBXh0cDPEyga/1j6hXuLJyDO9/9VwStJNEAGTmLh6JLufin+21ec4LKq9 +mcr69+9T1+lFGuXn/LR9Zdmh2NLC3Q= Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2b3646494a8so241120a91.2 for ; Wed, 08 May 2024 15:41:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208110; x=1715812910; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DOx+XLJ6NG229/wS5wOB5lVEjUmMQDyjSrW3ljEjCDo=; b=TLo3C6saTOLrnosYOt/1xBmXgWNgMZkLzPAyvenoqq0oAjga8kWqVhD7tpZnTU4KCb Q4+3O1ULQzKWfu0eMJqp6lsLNWVjpICuxUtoISr+nQAVzxFRwdWbZ46tWOH7q+DgzfgZ ni7xzG4uDwSX501nTb6we4dkh4V0rw124ruCxnRVGlOyKrzeN37Lif+4tsjDL7uEI9Cp Ardikaw/YU+drnc4hWf1wcP3sYOu5Tyg9JCI1SaGbPpPnLQWbYpUYn9Dy/+BN9v5RDhU jzlyyu5S9M/5xenRanQwDXLrQ6lavVW9hUkgRTOF9GnKxC+mNYZTWDqZWKIeMohEjtAr 2GAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208110; x=1715812910; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DOx+XLJ6NG229/wS5wOB5lVEjUmMQDyjSrW3ljEjCDo=; b=vVxQE2wkXORLN9MCHaCL+zcjJNueUSNk/V0yuC7vYdUhvC6xTYjjhAfVnnbZ95RrGO ZTYESpgR2RNREpDsIsst0tdZxUVKAsRGrZcGWwFi8oxb9P8GwkerDh3szXDbKg0HV3Rs 7qJ4l7L7HCjHvcGub/5y/tNhgzXxGRhFXzR+HThyUAdgWpkr3jOqH4rIb9clM5wzd9+O PuukpAzFO5D1euDFvdxoPpytiT1kPyRkYP0imT8Tp/doPiEdwk/pyxA+KkHgRl2gsD01 a/96EjRS5MMWpFEofsc+WsHHa/TyZ/ydMuZg1Ir2oA7fDSuDcVOJUfvyCAlfpCysP5lN 5Rjw== X-Forwarded-Encrypted: i=1; AJvYcCX7PmjhNZh4yaj4ZfyEXO4FieBOcRT0TneO7Hvfr20pJPZU7sGnn4XhlIf+HvNEgAa24A6E4/UgMcNbYUjMqA9m8Io= X-Gm-Message-State: AOJu0YxZVjhsAuqeQvxGF4ZNHISae2Jbw+wSgA1jH6sJxSoQ4bb5Ts/n sphbWmzH+qIiNBWtINEK9Hdk/REOv+AOZT1o03QSDJCPI29k+aHP X-Google-Smtp-Source: AGHT+IEFp0bWFb71EA7kiwqcgB9bekLg3MgzbqqZFw9IyUlHL4NxSW9RrXf+dnY9e3fmsofVV/uA3A== X-Received: by 2002:a17:90a:fa8c:b0:2b5:1f7d:7eae with SMTP id 98e67ed59e1d1-2b6165c4fbbmr3664457a91.24.1715208110215; Wed, 08 May 2024 15:41:50 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.41.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:41:49 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 1/6] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Thu, 9 May 2024 10:40:35 +1200 Message-Id: <20240508224040.190469-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 61B782000F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: i8cnm6mxx99wfeogwbaznqt9m9ztdg51 X-HE-Tag: 1715208111-739799 X-HE-Meta: U2FsdGVkX18E6z+l4SB2gjFiM3wO2sXA7GW+NbD9WsPtB19N9IgJFFqQrRtNlcPgf9XV2OXREDAjseq8cp84CHSm4IC9uzFEwl8cRswPJycJ9N95OPPpSabjSRZZVCBWekk8tWh91ny7zWHcItsXHxMJkm4gLunl0viOO6Xv6FO61wN0drs0mtzEy6NaUEC/CM/vQR6dxQLpo/qMjspvTlagU58my1mCIxtMDqVdy9KH8Z0p9zPZrc/njLpQKdHWmkV+spAJR0s6bfl46zlfkr/uXabECyQnpXxEGZ4pV3tdTiiql/wpCeuiFTE1zvMBoULLkwvb6J/Bh55WGB+AHFoEoBM4CyJtTfFSQIUg8Z5TY44txUh/1OEFQ09suxVXNW74YGt9e2U6DlTG3idcA5ah8OjGXZGy9Dw5WJ/NpE4fc3KKUMnyPdConTOsuBDzOowVW/zW8o6S1eV9e7pYKm3gB+aa+X7eJZScL7RC3p1MbSPz7TgajRb13NcC30iyzv9z6PzwYK7j56eGzsObHL6Shp7FjMwx+J4SgxEEUTugkne7biSDJws/svjhHNy9PSFNTgpauW66nnZyRpJjQ1UUtUIM/QEwsrzZGOq9c06EMEXJHAk4mKYfxGc2qnoa4YJsFX0Z9uyAbZJrdZ9x9PbtRiERDqlohM6Wv2SNzExUlZYwZcIAbVWL7pxJCxvjPqWAygLjG/bv980p/Z1cVUkPJUfPi+w7kNP7KAlNScxaZUeSKiiCs9rNSGtcphN6pdixUSz7FvAoL4dQgNpzJaVyEp555ieru2kGT/1KInuOnwSdLOa4o2p/tHTeLVS6rmyAIE+jg9MoQhQ0f7rqpWnG06pN3FoNzIkHsjRl2KroTIZGDTg6AN0XcWZ1YtuabBB7nyu9aEJwwXfnYKGVjEuuTEtNn2tnUDh2FTwe43IzomwQSWKBZgyPydQdxbfYS/UNvqEeJ7TRW8g8/qN OA+3taJh 1MGwrct+AYuPJC+3JnFJp7x8iRmHn8ZzoOpzmJXIm22rKB61s+vEpog/LcEjWw42cleRrd3VEm16wbD0Fb2wBFzadGsK+THDnI4xMfgHOgr4A+YaxGvbpgJdG49ZCLYTEwjvFNGe3IdNCxVX/kwsAsjRqvR6mAsU4QabDDnepMnZJAq3onribj5KUR1m6EDD3qkbyOuyGLV4Y1JeU7evvA+KNKTsZ6XNGSQU4ROSugd8TgcjKUvNvHsvLk9BLDGdepN20dvmEJOiWt1IO5Tj8U50+9IKaZ9ZfXCsS5/7P3sykYJaDZAiL2lwo52UsDADFENOtlONAFXFh4FwELhxNrzz1oqUrG0u6cQ2W+tuqxDsHunfJiiQsHu5w0hSiIyBFhBwhbTJ9YVGOr9l+zbk57D2lf+vcKAvr03l84LcGQsihtppwbTVzgam/hhEkabdmSVE/2AxJ8YzgkYnWfJn6WB/oHem+bAFFft1eeGqzWlsjfipU+ELQ8AU5fy+Mtai2dLaoc3W4ZDqkWZG/dFCcQWfhAUodXKwlv4lvnJM1IFVFBSTC9oqVKUfBkVgTDZxiPM1c+hypw89iPubuAK0M5Tf5YAeQ28/SX/S5m+2D77Snck4JgiWT5emS1JWoPYSi22c4z3YAZZZpdJe5r2vPYI1kdL+dOFi3fNb9AN3eDURnQu/RBI15hCloq3CYDTW8g3eU3qkrAPlrr3UNzb0rqDJ5ig== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Furthermore, this new function, swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Acked-by: Chris Li Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..d1d35e92d7e9 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +static inline void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index f6ca215fb92f..ec12f2b9d229 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } +static void cluster_swap_free_nr(struct swap_info_struct *sis, + unsigned long offset, int nr_pages) +{ + struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 }; + int i, nr; + + ci = lock_cluster_or_swap_info(sis, offset); + while (nr_pages) { + nr = min(BITS_PER_LONG, nr_pages); + for (i = 0; i < nr; i++) { + if (!__swap_entry_free_locked(sis, offset + i, 1)) + bitmap_set(to_free, i, 1); + } + if (!bitmap_empty(to_free, BITS_PER_LONG)) { + unlock_cluster_or_swap_info(sis, ci); + for_each_set_bit(i, to_free, BITS_PER_LONG) + free_swap_slot(swp_entry(sis->type, offset + i)); + if (nr == nr_pages) + return; + bitmap_clear(to_free, 0, BITS_PER_LONG); + ci = lock_cluster_or_swap_info(sis, offset); + } + offset += nr; + nr_pages -= nr; + } + unlock_cluster_or_swap_info(sis, ci); +} + +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int nr; + struct swap_info_struct *sis; + unsigned long offset = swp_offset(entry); + + sis = _swap_info_get(entry); + if (!sis) + return; + + while (nr_pages) { + nr = min_t(int, nr_pages, SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER); + cluster_swap_free_nr(sis, offset, nr); + offset += nr; + nr_pages -= nr; + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ From patchwork Wed May 8 22:40:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13659212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C6DEC04FFE for ; Wed, 8 May 2024 22:42:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D82B26B0095; Wed, 8 May 2024 18:42:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D322F6B0096; Wed, 8 May 2024 18:42:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFA9C6B0098; Wed, 8 May 2024 18:42:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9F1046B0095 for ; Wed, 8 May 2024 18:42:02 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2BF83120BD9 for ; Wed, 8 May 2024 22:42:02 +0000 (UTC) X-FDA: 82096702884.14.CDCF4EC Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf03.hostedemail.com (Postfix) with ESMTP id 4487220011 for ; Wed, 8 May 2024 22:41:59 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LiGAJxcj; spf=pass (imf03.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715208119; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BKP4frf3SJsYj3bpFkLEvp3HW+vS2YimBRDSFS3kDwg=; b=5s/OnU7tDeWPS9IDHudC2X9j+V99K8tS/IdYsLi7TfZJg9UhBgKlVKT70O22/E40BC9KtW uwHjHSz33SPuyqyB8dN5MQZao/DUg3gJ+k8yJ0DPYQ5T7myp4t805zT23uGQhDaCjeZ7Jj HN8cwzmGgi6q/MVCO21YX6VUQIiAWS4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LiGAJxcj; spf=pass (imf03.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715208119; a=rsa-sha256; cv=none; b=5zUprCjZ/TfehDDGgYAeeIkn5nKgAwbBnUgHj22qaNZgaxVu7EArzWTQioEMz6OqGnV15L Aq/0ivKZe6lzcYblLY0K5nKTTIsX0X3L9+3V9q9LkhtZhTNJNhXQgRJ3vfFwrgDtfIYU+f J34C9PFQGXuKdyX5l5P9u7xRlE/UTyg= Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2b432be5cc9so242230a91.3 for ; Wed, 08 May 2024 15:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208118; x=1715812918; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BKP4frf3SJsYj3bpFkLEvp3HW+vS2YimBRDSFS3kDwg=; b=LiGAJxcjd00xHXnCLVxpPq1VVL9FwE9rcIo1IFQvwtr2pvlP1SPCaPtatkRMJcKCzK yDl7PNn8ujEnw3ZgzrzE/lMgEpAldMlPmllJ1C1xqHD5USrSwPzUEtNuqBit9KNcQZAU PFJjPP+3ik4fjRMeF8CTZ1IBW7JtCL4SUa/u6kJ32jgJXRw1qp43/oguSvCko8H2vlvE qCIQEzgPW/e4tjFO0A77lqRmrFDewv5KEXaqXBMxcoXTsNmNbnlR3/fnftYqu+UXxiCn fvkaCUm7GiffwHIlYVk2Fal+wqeZMHWQiGQTJMIKZsz98mUiRfVyeIGrJwa486jdQpXb mpzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208118; x=1715812918; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BKP4frf3SJsYj3bpFkLEvp3HW+vS2YimBRDSFS3kDwg=; b=AGGbI34SpYKLWbxjxlvBjahSbWy2yqY2cNPugaAxt2uM+KQpHAHvSZ9/Ls3WKrUlsp V/vE4gCV71cos3kGsl5N+38KGQutA85gW8g48g6uvvcwQQVjSDZue9XUESJOu4491W6T 9SXqOIDGyWnf1m4VAz40tesUYBjlG0442jzjnyrL2TRZ242RNC+vTz9jQiPT2Cga6rbX ouxO8avn7km2fipWvN8dLDMvyWzIJFMwx6ZtKVxGtSvDmQ4gMgb9xQHkto6qh+mHJJf9 hDWiqvZJys8XXrRusVmVYzc106eFlgVH0Diiqu0kZrmMjJ3RS9tqFQ/O7ffwNjVWw+7J VyyQ== X-Forwarded-Encrypted: i=1; AJvYcCUBWbQS2CGEbWyauLPTfovjo6BJxPDrW2sDBghChsgP41pGOA57vw/i2cUoTEAD79lm5gfQIOfQqY5myS51IER9uAk= X-Gm-Message-State: AOJu0Yz8QECP1GYh355lXf9TF3wDWXdg89MHipyNippsceLWW4Au5a72 Z39KDm2lRRvGGuvYOMRJwjbU32AkICypZ70X1G//lSGD0SfZXc7Q X-Google-Smtp-Source: AGHT+IFi9srqYFLAGkCKh6k9BgPnyvD1B3mYyYQyZ3x2gIfLtku4oUuO4QjlFLxaj5GaZF8qFdK07w== X-Received: by 2002:a17:90b:894:b0:2b6:2138:e407 with SMTP id 98e67ed59e1d1-2b62138e4c9mr3786419a91.9.1715208118171; Wed, 08 May 2024 15:41:58 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:41:57 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, "Rafael J. Wysocki" , Pavel Machek , Len Brown , Christoph Hellwig Subject: [PATCH v4 2/6] mm: remove the implementation of swap_free() and always use swap_free_nr() Date: Thu, 9 May 2024 10:40:36 +1200 Message-Id: <20240508224040.190469-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: nm16ie7cz8yt37eamd8req35iqgu36w4 X-Rspamd-Queue-Id: 4487220011 X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1715208119-206963 X-HE-Meta: U2FsdGVkX1803TjMUwWq1Sf/xqRW1NWp5zT+frXgaR3RVM00PuIef/gG3uBKf5tnArFid7JyFYvsrlHKGm/7iq9RBsMLM0TUnPfBn40Q/FFEQVevMUNv+45HA8A3CZgQawkUnokIcHXAmt01QPAFlPxLp7q5MCkxmXhslVcovZmtapdcOUj1pvNTPvy3fmA0C4tfj0qTr9gcvYZclRca3S68RK8zgdXuGtNSw9H1AxGHXYdsBnbY/HsTd5ljk3Cax2Gg8LxKVHO3aBS/EpvguZ81QlRBuzvntWFYpxkuOmtVG2RmOjvyvEcCgidwzV/oBacKJk1HdvSdKZGrcmhEWt9vgd3oeA5kIUx0SWLBk56C8d4zxgbLr123/gwx1e+HUEwrgCGopavwq8yBmkeYSIph3I10Ri5iQx4dQ4sZ7FWTbPuiwRst5vmIhNTbZhX5ncv06Gi2LFTc8p9D5tPL2f1i3OvnUDEFpkJ7lqR4T9+uA4W2gT/J0WzVm3lv5u8bG9W4mr2hluccyfBR03vtpxHDsMEMlc8yqIvueXRfTcKntkCw+VUt1ergVIC1ug3f4CT7wH3ZIdTCeL8Nhae439CkAlNlgYYdwxoJ3kQQMUBtb49V6kv5sLVKVmXDHp4dz+19TLP4f2/bW5TlR/ZBYFQdCWYDrDPP8lmcyXp9BuQzjrPh6ZCmfhG17COJYmKbW2NTcKlgOswKc6SXfylST7+nU5AMIb6vsc/W+Lbzm0O57hcs1EITVo2Yg87+abyJwGXidKUHwAKGlwsNxkzG32G+wRP1vbQX92UGtJazZurCX8vFsWrWvtNCWfhR39zWi3vygeYSk4LXrKBB/UUYi2UzsG2TCMyO7xY+ig3YYU9/F7muBnRygfzJHvHtZD4fY/P7nfNB/1Zva6wMUvlpd1hgZAPzDualEm6HoGvQJ0cyhbPOgyZXIBJ6fZZ0x0iwKamHsfZ/qTsB/wuqyUj QADnvCul 96SWuHYx00HviL22PYFOYAViW0m/VRat3sw8WqzzZ1O+tKhZPeaVfyqBLSIm4qEqq/TsSkzCOQhC/Ta/doejyoJ1nnuQaTcZW+b3fT+skv/t0v0wLUv8Y4CCZtB0JHt9KrIaJ6p38yXDe1uzbS/jr8JjVbCJ1RpI5z1KMnD53+nFNdIkTj4hZHC8p6SRSO15rwK7Efir887Ipj5mgLH+Q93Ky5Em2uU9UMP7aRZjcusoEuGsUY3EaWRK9ZEMSXItMN4Z/2cdvSca5byKnpVFXKr17ICRFdmY3dNqH9k9e3+yn3N3TilX+G7uz32z+bkyiM1PQYEFTuioPJu1WONcyu6REfcTEQ14LzMvUV77tNzP2JDka4na0B1VIPktA9+tF9+f7QnfW8QL4wcwfCFcCeiGmDb3uVKS7WAiCWGLZU7UB5leDKzUVS3YYXu9T4X48KcZc2llYDvemaDSQMa34CCOic8grRE8zvLlfym/alpUAzG6RtGly/fTZ7Z9hithlRaW2zrhrxrDGvbYIgOJaR2BvnLqIkEGOb1wjdg9JdfVro77MvMwu3Zn3BX4BIa68XolTqeZzrt0qnqb3N4PO0INyvwEaJebqz3QQGJducqBxhPP7zvhoowrPDdubHzGl8sUQy8P3fSvuYEz8+GKqXZuhkiRDKNetRo2ltIDMDKAq+NlwyZllgCTY4JJ2ry0HjL/wYAs5jhrAfYLWkMPRF1gRtiaJmthwPhkNVocQL+kk8HA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song To streamline maintenance efforts, we propose removing the implementation of swap_free(). Instead, we can simply invoke swap_free_nr() with nr set to 1. swap_free_nr() is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1. A prime candidate for leveraging swap_free_nr() lies within kernel/power/swap.c. Implementing this change facilitates the adoption of batch processing for hibernation. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: "Huang, Ying" Acked-by: Chris Li Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Len Brown Cc: Hugh Dickins Cc: Christoph Hellwig Reviewed-by: Ryan Roberts --- include/linux/swap.h | 10 +++++----- kernel/power/swap.c | 5 ++--- mm/swapfile.c | 17 ++++------------- 3 files changed, 11 insertions(+), 21 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index d1d35e92d7e9..48131b869a4d 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t); extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); -extern void swap_free(swp_entry_t); extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); @@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp) return 0; } -static inline void swap_free(swp_entry_t swp) -{ -} - static inline void swap_free_nr(swp_entry_t entry, int nr_pages) { } @@ -613,6 +608,11 @@ static inline void free_swap_and_cache(swp_entry_t entry) free_swap_and_cache_nr(entry, 1); } +static inline void swap_free(swp_entry_t entry) +{ + swap_free_nr(entry, 1); +} + #ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) { diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 5bc04bfe2db1..75bc9e3f9d59 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -200,12 +200,11 @@ void free_all_swap_pages(int swap) while ((node = swsusp_extents.rb_node)) { struct swsusp_extent *ext; - unsigned long offset; ext = rb_entry(node, struct swsusp_extent, node); rb_erase(node, &swsusp_extents); - for (offset = ext->start; offset <= ext->end; offset++) - swap_free(swp_entry(swap, offset)); + swap_free_nr(swp_entry(swap, ext->start), + ext->end - ext->start + 1); kfree(ext); } diff --git a/mm/swapfile.c b/mm/swapfile.c index ec12f2b9d229..99e701620562 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry) swap_range_free(p, offset, 1); } -/* - * Caller has made sure that the swap device corresponding to entry - * is still around or has not been recycled. - */ -void swap_free(swp_entry_t entry) -{ - struct swap_info_struct *p; - - p = _swap_info_get(entry); - if (p) - __swap_entry_free(p, entry); -} - static void cluster_swap_free_nr(struct swap_info_struct *sis, unsigned long offset, int nr_pages) { @@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_struct *sis, unlock_cluster_or_swap_info(sis, ci); } +/* + * Caller has made sure that the swap device corresponding to entry + * is still around or has not been recycled. + */ void swap_free_nr(swp_entry_t entry, int nr_pages) { int nr; From patchwork Wed May 8 22:40:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13659213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D35A8C25B4F for ; Wed, 8 May 2024 22:42:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 657A86B0098; Wed, 8 May 2024 18:42:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6070F6B0099; Wed, 8 May 2024 18:42:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 481086B009A; Wed, 8 May 2024 18:42:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 26E746B0098 for ; Wed, 8 May 2024 18:42:08 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DFFD8A205E for ; Wed, 8 May 2024 22:42:07 +0000 (UTC) X-FDA: 82096703094.27.6985861 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf07.hostedemail.com (Postfix) with ESMTP id 1309240007 for ; Wed, 8 May 2024 22:42:05 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DkJWcL2w; spf=pass (imf07.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715208126; a=rsa-sha256; cv=none; b=YxDv4rjGNq+nwobLwjYDj808MAiser73yYEqrOPLEENTkEhS5l91gDtE0wH490YSQ0C6Or /pofCD97uxItrlysawAyYcUSDx3Cc30soz72gFMYvi4yAZ3bf3UxqiErHRMlS9135aLD7M 80XXzaDklFR2eA8qZ4uT24QaSCq9skI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DkJWcL2w; spf=pass (imf07.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715208126; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UYGLEQ5dG8GZxiWnoQtIB+ho0dBRJLc2akdsz5aCuO8=; b=UvYrf5tWBF30LDXe9c5zJ9JHCWo1TBJbopq11FHQcoIkIio0iW4SF3FGzyk92DimXzPVkr KmAwwiXM92IdKGl5pLaCmmtBQ/riKC9P1rg4NpS1UWpZ1/be4gDrt83WHGSSHH4/q9Z/c2 hyQXf56WCRDUIYnbZCvOLXnGyF5pwms= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1ee38966529so10195765ad.1 for ; Wed, 08 May 2024 15:42:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208125; x=1715812925; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UYGLEQ5dG8GZxiWnoQtIB+ho0dBRJLc2akdsz5aCuO8=; b=DkJWcL2w63M8vKKwBwR71Ok2PXZRjLQcZ6UviZ/ObiLZ1r6GRmc/usOcteSb4ms7YF PvHxKmQYd12NqWwqc1ESPyDS2p3nejL/3TgmzIVwREjmPEpp5+khIK+nkpm87d2zriwT Dz0JvrV9CtUV37Q1JAqGmIy/P+WLlvqmwWQmAyu89QJnaSOolLVEGTYvDfZ5x8VrqxA9 6ev8a3DwBCA4EodrIp4akU0vWcumJVTIv5fp9nFZrPfgZBUVFLgK0sj1dIubr/bT+E5P Ih6uRcLOJUS1ysISn5ILKywDI8/k84UGt98HaZ9Mtjq+RPyj/qGWJZd7snkxo4DUqyeD ts1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208125; x=1715812925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UYGLEQ5dG8GZxiWnoQtIB+ho0dBRJLc2akdsz5aCuO8=; b=Ra1mY3zew8nUWz9zlmMNHuciR8Z8DeVXF9iauSO0mkBIky7qQBzXbgy34sHD6yslMI BiQYvjFfJvZPcb4mINTRzUNilpoKBOx3D6hePUOSQdRPIa/+K64dDWQEZt8EE3mnNPnz PR9OcmucGyHiRNKrTSuqoBubbH+cua1a6NXvLNen6pIVQZWKEbTCj5DGLd6RaIfMg8Ta w2idtW89husvVqAHVXG4Q0mbb8h1SqaeKGBE3y42/ht9AnoNpV1I6iWwZgURWux+2ujM K/0zFlMEASte5KhmVSXvsSNtDFIviEpCwdoMDYSAYOwXoVAu8eC/T1yCnt6h1zltFKry n8dQ== X-Forwarded-Encrypted: i=1; AJvYcCW4luIUyV2cBOU6Se0uvfUIs8GRSOr2xKKsEoq82Nv3HxpGNLfTr9vDd172XxEoMNbJqv/idrUpJUvPNkSP0Lq+9Yw= X-Gm-Message-State: AOJu0YwnNi1CXobdVJM/JB6xjmioA2JCruzmfMog2/fP2CPmMKBjm5ye MN/9XbxfnTC5CWVAfeNBMMyy8er9QlkNoMzrcszBoh/cSorDHO2l X-Google-Smtp-Source: AGHT+IFAtnVi6xLor2XHGazgK0Nm3nnCaOAaL5JyBksp2vgFvFWPvYOXqRq2Ix3zEh88ZA/XgC19yw== X-Received: by 2002:a17:90a:db4c:b0:2b4:32be:4442 with SMTP id 98e67ed59e1d1-2b65fa1c4a4mr1504710a91.9.1715208124905; Wed, 08 May 2024 15:42:04 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.41.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:04 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally Date: Thu, 9 May 2024 10:40:37 +1200 Message-Id: <20240508224040.190469-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: teftnpz7if9dff37hzjobtqzutban4tq X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1309240007 X-HE-Tag: 1715208125-875723 X-HE-Meta: U2FsdGVkX1+6KAGQ1tTdDbrBEj+mX5YJLns+37NYgL4P59pnssK/BfNwnEDBW5viLmvYCT2ygYooZ5obODdAg1F70CEutwZ05Y3O2PeNLVZt1puX8xXJ9eQWEF35BgNoYqn/3bI0ojqrz+4r/V1kDMg71Blu/ixBJ1xMDR4zUvRt5mWCcC3U1z9oZkbkDAuoh5Ts3OkPXXuB4JHyAjkMICRv7PI0t062go87DAiGwWe98G+aOOfNQU4wWEmq4TIrfZ/Ilo2uIbU3jjN7NPv+7DtOF8k0mdFrnlX/LxtKyAwcoEqFPJPkMmDXJYImRyUWHpweCVt15Ns8O8TtZrk7WHtcJAX4AfdNXe+oqHq9B8g1qsdyKla3zQZxBz/OAINyXH5o3Nwq9OS0WyLADkuKOKMEzUR8W7vojaugtPEKS1drtvOZt04edTayIdmxmo8IPfWpKgNJ+34FF/geD4b6ShuKAj0iuXf/z2F0OnCpX7vDk/v8Muh6cv62MvFtUPMiU/EjeWSkPiKUktOw6+RUXMNPxJUAF8K/174kHV3RTncO0OztTpzs9wb7ScxBWkTj3INjbQJASQ/c1Urj/1WMcVy6DzPudxoDV8iOPi4ZXav0XI/5udFv1gYUO43WMT2PFgB+2IFlS5TWOrSOC48XNZu5X60LkGZSLSitxlB57Okp6QGKZrKbNLhEuDqmKKgKuRVruXOWXruLzZYzU9uNM6aWt+S6aPqdkw7cfMIEmzDMavpMn7N437a3vG+XWnY5apU5TZXPnFdDtw3ms0G2yuhkwtxf0zNvtHYpsPobQK00hMYFpwAfbLXAbuad7nnDy1Ozo4SAI4bmq3in2ViD8NlyD57UiLdXwe+c3e5WuOx+25qcYiuW0vl69moeQei1RaxEOpgXtAexzUJlRStFVzYyRnQF/WRG9WhLYD6HujVbFvMLKCHtI/dWChWWo9f7aSStM9rLBWCUjIGt1rE SCroAXhM doYt0BMFPd2Fnt4oW5+wmXqzC/6roFBisOS24dbA6QpCCERbIU/n3q304ZKIf3vQMUjPqziH6wgzp5a9NG8mn3z09CIeVcDiafMqS8t20MxgzryawzUV48P9KN2XPLSwzgURsGTG7OexJtJEYTPF5Qpp8k4PojFcpFn9PB83G4YsZIK6jYQl/FdDTFZNf9SVFcdaDhXjPKatiB14BIy3hgQyia1G1Q56TDa/mpqh2P1ZYSnZe3SlK62DqhIc0s2r+qTxDGyZNyF4k7IYIT+0795nPEDc1pGAwgnWnk8yUI3aZtZUE3vJ2ED0bzwdAlcksf/OIEN56pG2tBiOCLGFVxz3TzMLvMZ104HQ2ToFDTRr247RwwgbZ7m9LlDODgmLcu5yN7Ldq93nCF0xz6JfXn15bx+eenMyJVi+m9ZCQc7jskEzNUuZYPmGyT3+4TfOdezOgEgCoouSYU1vZkAHk8EnJAyXNHLtlpQhEvYfuNUkVAnQbWB5lgUY3gmSEzNV2CUaVqVWmp6mdtzzagG4VB9Auwa7/NnkuqxXr8tQrpGBYipnq3XjWXDFcaDVgBi++cL8w2X8RoH+ZjeRO0xIMNCCHy/MzIm44VpJ3gnOg2p6W6cNlFjgRITtQ8/McJhumdZUkCPStjfgPn+aD4QbwbysS+Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song There could arise a necessity to obtain the first pte_t from a swap pte_t located in the middle. For instance, this may occur within the context of do_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introduces pte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently, pte_next_swp_offset() will directly invoke it with delta = 1. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b2c75b12014e..17b0a1824948 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, } /** - * pte_next_swp_offset - Increment the swap entry offset field of a swap pte. + * pte_move_swp_offset - Move the swap entry offset field of a swap pte + * forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. + * @delta: The direction and the offset we are moving; forward if delta + * is positive; backward if delta is negative * - * Increments the swap offset, while maintaining all other fields, including + * Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset(pte_t pte) +static inline pte_t pte_move_swp_offset(pte_t pte, long delta) { swp_entry_t entry = pte_to_swp_entry(pte); pte_t new = __swp_entry_to_pte(__swp_entry(swp_type(entry), - (swp_offset(entry) + 1))); + (swp_offset(entry) + delta))); if (pte_swp_soft_dirty(pte)) new = pte_swp_mksoft_dirty(new); @@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte) return new; } + +/** + * pte_next_swp_offset - Increment the swap entry offset field of a swap pte. + * @pte: The initial pte state; is_swap_pte(pte) must be true and + * non_swap_entry() must be false. + * + * Increments the swap offset, while maintaining all other fields, including + * swap type, and any swp pte bits. The resulting pte is returned. + */ +static inline pte_t pte_next_swp_offset(pte_t pte) +{ + return pte_move_swp_offset(pte, 1); +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. From patchwork Wed May 8 22:40:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13659214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89A0BC04FFE for ; Wed, 8 May 2024 22:42:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E96E6B009A; Wed, 8 May 2024 18:42:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19A626B009B; Wed, 8 May 2024 18:42:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01FC16B009C; Wed, 8 May 2024 18:42:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DE5906B009A for ; Wed, 8 May 2024 18:42:15 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A04391C165D for ; Wed, 8 May 2024 22:42:15 +0000 (UTC) X-FDA: 82096703430.22.A6F54A4 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf17.hostedemail.com (Postfix) with ESMTP id C89E240011 for ; Wed, 8 May 2024 22:42:13 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FfrVoinK; spf=pass (imf17.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715208133; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7DcpNxuaHKQ2XGPRcrgwYPClc6OYNtCFnxYQ34hBDF0=; b=pEjaI5PdV7V4CFRBKVGnFmXJmdVaE3kxOV3awAqE4cAckkaFRkoIEIQx1p0Ygvfyh5syvT ZwzUI/4hC4AuN6Fkkbs1gDCNdw73GvyhI1TWPANI9BoqXWcgb4bLZhiaWhjdgeUez+T8rd o25WfoTeLvuYDmVcfUYoIqq3V3myhAE= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FfrVoinK; spf=pass (imf17.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715208133; a=rsa-sha256; cv=none; b=kbWR7FYkhAJPWSbsTeDoEUJhf1iaCfn170VVz3OIvhPkC5DGrz/VLg2vXMnE85pUHzdpxV E4s5ceK5m/hQwnjo69EuqPGokB07ENnrBHl3DuJIKpY9NGpj+AunjMXbzoHnMrhW+sBzg2 iya4KAei3ybBZmsJCst8er2WqjM8+rA= Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2b5f3e20615so276145a91.1 for ; Wed, 08 May 2024 15:42:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208132; x=1715812932; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7DcpNxuaHKQ2XGPRcrgwYPClc6OYNtCFnxYQ34hBDF0=; b=FfrVoinK6z7/0GLtD5B2z7yb7UMEyAkM/2yKa8NE5IphASI5mfB85d/Zj7P0bi8xVr 1aCBfPQYYKPmcJF0yxcvs5qEB0tiuzeAP98LKfGKRa+fxwSQQDDLOlLxfFIcDyxFYkzb OCptXJzwsNiCydU27dLyE+c/nCrm+9etOdN4ZFqUZWEQz1j3itlXI7lB9QOgPvT5Ever +1MqQ/IpTz4rRR3T/+o/UstH1SOb2qLnqA30dhTR08F6PMvNJCn3Bcuqg/4cBdKTNq1+ 3+qHhlZS6nb1fnORiBDjimLeRdLaI46IYyjSmGaJQ+pT1sD56FHUnB9/DDrCpGYXcent LWdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208132; x=1715812932; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7DcpNxuaHKQ2XGPRcrgwYPClc6OYNtCFnxYQ34hBDF0=; b=wZ+/QcpQxr0/4PnUGmzajKWUtExtvskMPVrbVh7mYupvQwyDaJSA05dhFS8NGuwf2a 835PzRVoLiOoWSeczY5kABNtdQmFKJ6/IK7/G7+h2R7APWAkHRLKSL//bK1Q766UX/kr 2GY649p623Gk0T0AOXsmLZuJ6JZmu+6W0UutldZ9jC4Ap17xVFOrEtTwN3xa7S9tb+qf Z16NaMBz6ANV6DUl0sonLYSl/QxtBshzgsYCiRSwqNohz4EMeln2MHR/QP2lfabNHOA8 LWUd+A1zkgFgVn3olistCPJiBx5kwUi3i7sNiGki5EPfBBJGqfw6hQvKiuaKtkl/3dWi Z5Kg== X-Forwarded-Encrypted: i=1; AJvYcCVms1uUnGk+ZXvYy9O4FG8Dg3q8w+G8LhKS6kml2bPIsLHy8ReAyUIojzg7pE0Y2C5V3U2mVeWuNyjoLHRE+Flf6Rw= X-Gm-Message-State: AOJu0YyE5IFyF5ubbPJa5U/4pIECGgzKKjvhO5kF9mkQWmSEde8+pETN N7AmpsxumSm4aDx1j94OCnwMVattjWvKHuiJ81pD5AfnX9dbeA+r X-Google-Smtp-Source: AGHT+IGkFdAUAzrMlbUlBhREM/CEvwatOEIMRBL45vN2bg1LTAUKPSPifp6IeAAwfIL/STXKxZU/xg== X-Received: by 2002:a17:90a:8b06:b0:2a2:981b:2c9e with SMTP id 98e67ed59e1d1-2b6169e1a86mr4021869a91.36.1715208132570; Wed, 08 May 2024 15:42:12 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.42.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:12 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, Khalid Aziz , "David S. Miller" , Andreas Larsson Subject: [PATCH v4 4/6] mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages Date: Thu, 9 May 2024 10:40:38 +1200 Message-Id: <20240508224040.190469-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: C89E240011 X-Rspamd-Server: rspam06 X-Stat-Signature: q83ao8y758pwybwgcx8q34hswa8nemhj X-HE-Tag: 1715208133-970747 X-HE-Meta: U2FsdGVkX1/Wnbtnp2GrJlQH0CWX6haCZ3SWcKN8iHhhI9PtxQiOYJixEJGIMYGw6lgu6afFtjIfQwm0IHplg38SQSdqts+T/AT42hcyUMz8umKZ3iWJGugDo07hf6ctsJlikq2RwnpVtbUDAi0xcC9kJ2jjRHPSFiCdC1lO6X4QjL8atWOKStwZAyjbCEig64bLTqujEQQRgi13KmV4Ynmo/r9V+p5zLqxNWLzxSu6wjc04q9BtlZE1gMiIr8WJwGacCqDFn3MtA+Ux16ImApORUn5uV/6nFO41P+nNy738yHsfqzR2oOkgfpdCEDbDfyNyvp+enI4OXTF2JZ0Nl4r+fu1m7ADjNks2jt3Ug2/zDcQmaY70cJUYn9C4gp1cij+nZWsRMuBswny05IIh0SVBMpGEwt+todZOgCDSvEaNZX85YY9+zkyLrsLT34ly/ObcA3u3xAiakXmEZMoSxzd4IqiMYinRAbHD+1BrbnO36sbSxKI3pqImqEFWGyp73i+BLXlCy1u2kTdHzXLh5QNh+xD+7PdFtaoIOyw6aZ3k8u4pvpK9qH46FgVwfDbmlNVlgfoiOb+YvOCoamGbIPyBeFWqL6SFREkeVGNGIuu9aewFDbFEhT1eK7NZZfTXp/Qo3gnJZSod2si1q0pmQl3Cd+ygOtHYBj+mrAr15pRj3F+3uUestbtETV0HRIhmtYOCFivAvXWjeCKMLKLnwlQAF5D5UABTOaaBafFGVjvPPiQFRLO5vzUkMR6ovjBO8V3rgIMGYXej3dXG1IB6lABzGLkilEFc+gROf+tYHlIbXFtzVAdCRQ/BCR8Bz7xvH4GBKE9ZrImrCN0tH76tyGKI5Dycf65nscKh8IrDmHoL3SJQ6STTE5T2HC0AYduBI5cpryFn2AI9nMHJg+kBkTVG2cRcMZbYfDKCynDOGCAB6xq34ebNGHqgiS1O7wQZpKDGrFx35cXxrh4Zt+S GlgKSclD PAb6Put9gVXKTfM8ArbA8qGtnUiZuyWXp4aWKU8m/hOMgelpEsfo+auAhslk2rwulKZSzenX9wld7RzvOG4mrEhuAxLL7t9WyZ8XA72brjBvMW2f2DfezRLfRpUGB5odjfU52/fs/typIoXuJxGtm2MPJt4MZTos60OwF7LLmTVJ9i/4Ter6pTSnadRQ2WVsbOHZ4KJupQnTc6pTwGxHHZi+nKmGUAHPZsFB7GqUMdIsBSkPj7ScDCCZm6X0qEMP/LBtLiwBrYWtV0Jm8aRfJ4RMsE/cL4k/81gJXsTn/+SGWYYuF3QRlN3qG/Gox39ZZYXrgg34E7ixnnAH3TeZZrceISe8+E+zm7drG+HZuUzRb3JkkCB/sFkQ1KNwUQ6rZLH+b0zEZUTI98bGARDkgrpmH0YNtOOfpjhpjzPiSbMcBwwKBFGsqKCSHz5j5tMrxwhdHxbqeuD6eP3bS4hYXV49gKbJm8f9MDWbZ8i58ua4y+TVDrM/4fiDRMV7BFib/fUQdq/UQuSvF5Xy4LxL1Y7WJbtlEmq9gaeHKdMh65XveefCdSKMSVMrwTfU4a6WXs0fftZ+wGF6tpSEy9qPq4edoyorNWpPC5elqNy5m4XrsU0pLmS0QWYh6WrJa9dvfAc8TF7T3RuwtOlMik0ZQWxJK7TIrbuHTSt//ljPrS8pROfsNu00XvmbyHLyoELpgeTQHyL5yJfs4Uzy9S1cjke5nYedii4FczRKzRnj6VsUx29/dycrrtujOuFB9Bl1bUXD2hnfvo3w7Uy0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Should do_swap_page() have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enable THP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementing THP_SWAP might invoke this function with nr values exceeding 1, subsequent to do_swap_page() successfully mapping an entire large folio. Nonetheless, their arch_do_swap_page_nr() functions remain empty. Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: Khalid Aziz Cc: "David S. Miller" Cc: Andreas Larsson --- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 18019f037bae..463e84c3de26 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1084,6 +1084,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) }) #ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be @@ -1092,12 +1101,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) * page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page(struct mm_struct *mm, - struct vm_area_struct *vma, - unsigned long addr, - pte_t pte, pte_t oldpte) -{ - +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + for (int i = 0; i < nr; i++) { + arch_do_swap_page(vma->vm_mm, vma, addr + i * PAGE_SIZE, + pte_advance_pfn(pte, i), + pte_advance_pfn(oldpte, i)); + } } #endif diff --git a/mm/memory.c b/mm/memory.c index eea6e4984eae..b51c059af0b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4266,7 +4266,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte, 1); folio_unlock(folio); if (folio != swapcache && swapcache) { From patchwork Wed May 8 22:40:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13659215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85204C04FFE for ; Wed, 8 May 2024 22:42:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1275F6B009C; Wed, 8 May 2024 18:42:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D7796B009D; Wed, 8 May 2024 18:42:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE2A86B009E; Wed, 8 May 2024 18:42:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CFA6A6B009C for ; Wed, 8 May 2024 18:42:22 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5ACDDC06FB for ; Wed, 8 May 2024 22:42:22 +0000 (UTC) X-FDA: 82096703724.10.0F5C438 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf23.hostedemail.com (Postfix) with ESMTP id 72FF8140014 for ; Wed, 8 May 2024 22:42:20 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=k3lnSQfq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715208140; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0eTHILZ80XouT+1m3vAPlwsDJfF9Pcan8u7a91N/lZo=; b=B9kq8SE35MTHg4X5hzaJjqNxJ6clJbux8q8DMmwZ2u6tD+RCBCHoI73KEnbSXWyUk7z9RN dNg7QKGO7GOU3pS2wWTrUeHNFdBq/lQQhKx5Qxxa+0awhAM+Erx0+OjC6YG8NT13Ftf9XY Af6d5XeaX7KLmKa7A1M00iwEvQITDcw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715208140; a=rsa-sha256; cv=none; b=rEDj0bpyGXPjsXMvU1zAxyE41TwxaO1uIIcYugetXXpcMtsRbqqU9eqs0nY1lA79ulAb28 AWDtTMEoezxiIISvxz+lOWAa2FuV8r+jsJLrOp4piJkEGxDenx/wZsOP/oBm77gw/GpY0e IswtLMBSNWqHlRTJyNqt1gzqdsxzZPs= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=k3lnSQfq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=21cnbao@gmail.com Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2b33e342c03so286915a91.0 for ; Wed, 08 May 2024 15:42:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208139; x=1715812939; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0eTHILZ80XouT+1m3vAPlwsDJfF9Pcan8u7a91N/lZo=; b=k3lnSQfq7sFi4IRtUF9QHAsY5a3Y3wNhEOlMTNdvxsmyOLGDjO+x5b465V6Zeo76ec oIwVTm9nF3DbJnnNDh1lh0xvRLFGUtisd5luWFj0Rvv0YV+zYqMiDw12XuQeEECx73OS Ul1ZGkUxa0+dNtRjXHbkQ5Y17D7gvI26qHb+/twyHACLiMoZR/UGDLfnx9yHSfVW8yeO fTE/vpDEHBa10NXBRd5LktDTbFE0kjmdccitIjsPqf8BqO9IUghfoF9eXNDOs/AfCQI1 kiZzqbUDy75jTvkOikUJO/WONmF5hhCkGl3ZdVa9WGNPy0zZ+ge4/GLeTol2iUaGobDc jPdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208139; x=1715812939; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0eTHILZ80XouT+1m3vAPlwsDJfF9Pcan8u7a91N/lZo=; b=HL6km6QBNKyMN0+IfLtruiX+XBCzj9EJK4EtYv7tdy+zIGCM2SnZZOAvU9lgbvyAyD fJnoKI/ioBpHp3K+BDlvoP6UIotJ5BdfWMtPv8IvZcPFNfPTOaF5+apJTc6b4IaKOmfl Ts6gsrhYKMF00t9o/DlsyknFRQGvwRNuaMI/K+W+2zN/XYtz96PvVdzgcnki279cULy6 hv0bLStOe9ApxKXTecewyy322USbki1lQOlEqhOhtS+GDVB+3vpt5F6gAbZdnOLdNtNC ht25on/qB22600Dqu1UY28stvxa4SEsQyJA2RnNatw6ED9ZZVLSkCkkXesPrn4p5Lf0J 7yMA== X-Forwarded-Encrypted: i=1; AJvYcCXxrw0Irdr447XY6ZHAGl35I0mijLctcIlnX0+Fp4pbcbM+B4Qw0gMmOI+uDYyMr/q8SgNhMfErO64aFj8t2RUt5XM= X-Gm-Message-State: AOJu0YyU7oDWA0+B+taxeIwDeR22Xb9nOoUOYRoUHze3AUSZUIF/wyQR S8A/GqXd7NULgetkm6xPInEZi3qNtAhwKsqJ9/UzZfSthBAAzjNA X-Google-Smtp-Source: AGHT+IHz0/iZAnfRiaUeJgoYtdpHEy5GFxUrXuxrhikbMPcC7Bwle7yCU/X9SrLM3+bghWb9T3SB5w== X-Received: by 2002:a17:90a:ce94:b0:2b1:73f8:a013 with SMTP id 98e67ed59e1d1-2b61639d6b2mr3773903a91.2.1715208139383; Wed, 08 May 2024 15:42:19 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.42.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:18 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 5/6] mm: swap: make should_try_to_free_swap() support large-folio Date: Thu, 9 May 2024 10:40:39 +1200 Message-Id: <20240508224040.190469-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 72FF8140014 X-Stat-Signature: 3o5ahji3cis6k77ou8qfhybrtkqrhwxx X-Rspam-User: X-HE-Tag: 1715208140-320540 X-HE-Meta: U2FsdGVkX1/Dc/1dtwGdn092QFGwGJxaSv7U4Umj/XyE+ntTOgiO/hAw5qHCW55VZkmP+WKqepR4qC5cEsxes5rYgps+W/XBeUGwDbBDNFLU5U6Z4jhOIE7mQb3gdEiwx+OzhmPwga3ODwoO+K/Mb+haRtOlHMwDSMFP/+1I8sBfhzJx7rQSnvzr8cs6KYg/LYIR5qFWCc0n7vMuKEpw/wQIur0krQIXQxx+YQV3BxPKbqSn/lNp8P/g1NsY1RwXDz2sUJfqWXL6P211Xku5zBnw8+h7XJZ6euo6DWBio6ivcDVm03WqrnUahyu4/QzepgvPnODVWtHDsqgQPevmaufsGyMfrRMmuDLyv7VGNsw4h8R4bLgjnjgWZrPTzUS9+3NkkKmh5WANv+NL4i6NhG9epyMjPT6Szgst9O0ujJnP6GQgNqDZYn0CE82QUU/iSJNDWludXxW60g4c4TJm8BaY9bLMjb8hMqElI8zNmQo5ahlCE6AhOpsZ11ZLanzJ12P+xptAFVn825vH64z8JLTSEJh6sHUV9NBoy9nhr5UBEBoNUstZn3wus+5//l2oJZdLDRgnV4oXDwJRB6guWYxQXq9b5uXdH+TqN47Nc9JyyiLfhfIWRHJXk0iX1Db17cTirkP3gd9n9jT9AP3pKSHdkkW+oodgq1uHstJA2wTNvmwYE62iL1TqPXl4i4jQ5IrKrQA2UiWjXoN8cwZEusXaQLKbBjMw9kFceLsoOe9JCH/nXrGLsNGwbIO6ZbsmVziaoBujrsXAM4X7HwNcrrnWTTjMfwLRCh/bpeQytvDMfE6RgA5mDRdeAtrB97SYvRBSP0ikopUos2OSLAq73ZnAE8XB61NUGZSo2FHEBeEPgmJBpMv67bR39Znpf3kRCoH5p/2bmP646BzOp3Uhmse3DI4iO4aNAQCfyjzTY1JJ4577+FB+6xVmtcQaCDf486OuJIkp//lzKMpowhe SwFU1VHT Y05/U2BluRSR45uRj8gUrrvR2D5sZsc30vuqLnXusPx+lFTnoTqX83GpMyQiOMK9M4bayUUX6wJwJ3dgG+qDPm0g1XRqgIoJuPKJk0IUoobM9J0BS5b5w703jqMJqnK0bcqRQfUvaShZo7wAjZoNnJCYzEu3WjsWm1n80iL1d60ZfhnHMrHf3CTtNAJ05DgrNklN/mu4uLu2QLUKpXyJ377ZgAHSTQ9gkf/xKfZZyNyi1XOph+mKKsn8yHZXSHUSatMK7aixVlmoI4KqSiyriG2xGmgSVSOtj/nw0E/Jc4ldJLcYAb5mWk5evicF3drv51Thh6bhJE8qwLmFdUk5Mm3jzMxPjHE9ToB7VhHM04IO7Zyd79mdS8KTIgsWc31qcldW9joLR1coAO6CC9xhwaHyi8/N/PHO4UghxIM2WiZ2Blcru88ibB38Rbj1n4gW9rjNSB8SJK1IkLvxOk1Mk9dBYFZohNhGJ9AKbXCfZY6JXV61kTGzGdRQ3fmJ1XIcaqsn+pBGweO6hSHJGOJ77G677jv26kVm+qHv5FlyHgkWtb9eUEtd4t6F0fvGaPkGW5GXgkS+X8Mz3+LP0/kIaGpdpk5Q4HwUp/9GqejN0qNCzrs34Qa9qoxFdAJmLJwf1sQ0CxVkso0gWKsDQIu4DR0JtszFtP6YRw4C+D9hvgWJkp5Wx7MH9lVeSinjILjgrfLhPUSJON+fpB2+NxrZ90WjMuAMMbFrVPj9LGBs9ZgtNIAg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages() = 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" Reviewed-by: David Hildenbrand --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index b51c059af0b0..d9434df24d62 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3877,7 +3877,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) == 2; + folio_ref_count(folio) == (1 + folio_nr_pages(folio)); } static vm_fault_t pte_marker_clear(struct vm_fault *vmf) From patchwork Wed May 8 22:40:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13659216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C638C04FFE for ; Wed, 8 May 2024 22:42:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B23E6B009E; Wed, 8 May 2024 18:42:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 962FB6B009F; Wed, 8 May 2024 18:42:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8033F6B00A0; Wed, 8 May 2024 18:42:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 624AC6B009E for ; Wed, 8 May 2024 18:42:29 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1922F40E64 for ; Wed, 8 May 2024 22:42:29 +0000 (UTC) X-FDA: 82096704018.24.780892F Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf01.hostedemail.com (Postfix) with ESMTP id 4469040018 for ; Wed, 8 May 2024 22:42:27 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ibHvdWR5; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715208147; a=rsa-sha256; cv=none; b=Tp6yT1ceXclTgTuqvt/3pOaBhqwmTFQ8VsFz92Bjh0A0RUD6MFap2bBmxWimMWQbi0N6wr cXttxXLt20wn6VIeIEkC6I/bNFykFQ9dxUatOVbpEDF9qHHdgg51yiSO3VMCgtHEV8lxj+ kfwPIam+5ulls/DOztNPIjykqgbGRL8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ibHvdWR5; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715208147; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SEA7q2xvrQmRo9zBznhNS6ApXjfQDgq/4gBxPbR9Hqk=; b=OEBFdM+8VloBirws3g92MOu92M4A2Hn4ilJ9n21oL1MaNAiqIaXlKUPukalSO6ILB0B6Dy iCslyC9EZi3JMZA2ct6oAVeVYMkjasYW+eKq6Vd0mZg71Agp6TMp5OTIcSbGP+mpX6uMCY FEdgdQ3FEo1bpwJUc7pORlz18BaBHwQ= Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-5bdbe2de25fso205177a12.3 for ; Wed, 08 May 2024 15:42:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208146; x=1715812946; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SEA7q2xvrQmRo9zBznhNS6ApXjfQDgq/4gBxPbR9Hqk=; b=ibHvdWR5OonSOBvkEPoHVVMlk6KqPBEqP8Fcb9lInCGxRJ6KFxBguqxdGBTVElCX0H dG0PFOENwc73pbOOn5vCzWPlZLhDdL5d1Hjhwccci1qW2SsdYR4kvNQQ3e32bNK76K6q +5tgZHf2qf1IqHxIhVFabPpcpBnGxWQ4Wsd8+oO1kYXClBp4KExAj4o7NeB/TWREJqcp rmP14hTDKKg/odINj10em5ID1yWOLj2GQzDIrPHO6IuVkzBpJ2u4CmQ+DLpuM/FWcxDW ULhS08l4onCNyysz+ClINDXrmVw2ziphV/aHhCg7Ys0/vXZ4bszPlzFz7F+Tm4gpQp/J pp7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208146; x=1715812946; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SEA7q2xvrQmRo9zBznhNS6ApXjfQDgq/4gBxPbR9Hqk=; b=CGgyKr3dtjSVHtWVu540jK9cGQ4McqVLHIFu8ZDJ/UQqNrU+8/XOvMOz99GZMilWcS dIyyxf/IJKs6ekxdeprQZKYIvnL4VyxxxJGDJ81GDjeeYqQP61/5HGnjdB5Tgp9hkHgX kCPOPZ+m4QV2l5U3syDy2mPCFbZkANT5LHP+DiafzZQ5ipNXrsG3LQBtqIkUvFN7XoYi 2a0CDoTi2BIesOf6xbHdPu0rtC95T/247UmpivZEa/T216GJb7r9J9EQg17VKz8p3+Cy gslomtXuAEqZw38jPfvusR2HNzyonDpOjpvXB7ULllB3hIPBpKK2GgdBBFl8kv6gqORb TOdg== X-Forwarded-Encrypted: i=1; AJvYcCVlatOYooyUWeUy7jNMPQs6CfC8IjlUMWimz15c/nGggk0C4mIAcpJEDBRcJJbfoyAuPBNABcGKBA9YmbzG6oCinFQ= X-Gm-Message-State: AOJu0YxV74simNpni2T6+n0Hc5Ffyjyp2AtNM/c2nKoV1zE9oD9fCPHI /Mj2tLghCUr1ax+xyemytd2nEbFX5+6CL2AFgb6ObnsKmMffhoPV X-Google-Smtp-Source: AGHT+IGq29rmz9yqnIyAWg43yTePii4lbT2fQtdkcVHel1ug/Dy/4QuBgQDgX9YahoaNoa9CKKLOjw== X-Received: by 2002:a17:90a:8811:b0:2b3:28df:92bb with SMTP id 98e67ed59e1d1-2b6165a62eemr3793409a91.13.1715208146212; Wed, 08 May 2024 15:42:26 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.42.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:25 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 6/6] mm: swap: entirely map large folios found in swapcache Date: Thu, 9 May 2024 10:40:40 +1200 Message-Id: <20240508224040.190469-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: a1ag9sxaw67rudqk9i1gpq69dymgp5ju X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 4469040018 X-HE-Tag: 1715208147-463164 X-HE-Meta: U2FsdGVkX19juY2ZnubYppaZ2h2HlCMb1wFvFjA9KHcNTmTueLU1IlfKs4jrF9tn5Fc+N1AyQ691qUNZgNuYiNr6FhvZOO+AbcUN68NvcgwMEjPHB24MfCwkGzF2xvXEqoZg3MLat3y5JxeODHx/1FVQpdtzdqF07n5tmi7o77eJnY1gI4d6/fe4uIPWPcc7huXv38oFDeL8UBO6jNtpR8u64hoWrVI5Dv9EEfMLJ6TBFvkYZ1fEY5JGPbVfwPBQr0ZWrFr3EY6YjmUurH56InRL0wPwlBBN3rxHfxVua6+d9tCvSn7T6PO3j7o04p/gNZ6rXy/3vmLpk8H4D13Npo94n8SN1GsG1p8/MbIxCOKHbybACTJvf1Qm1WgSs4wtHOnUqPfM4L+NIbJN6J6VGteDND8/7XxKW700fBLy+5ofJ0JQKSbai0UrC8Z6+Jb6ra2pNnRAZmoUfaVGunBxBkCY9atHvPH93/ResFgJHSlv4a2+qnX6hGFdqUA7IPbKGirWaaTvzqhQhIehY+igWq7i0Ba5DOshA35bIaxIH5+PZYD6CqWPbZCfGmpgdAAPsbQODb3FB/ZvYznrNDYO3s/1BnHn1WAfiziuSjf8mpws751QF2JGAOX0ln3+iejIY6W77yefn338fbD+EFCbmt+nWnKQaejv4pYSghCT5S4clRuNsjQgwLjPqsvI28DOOUJtkj09Drxh8KXQTxi0Lqi5bphFThaM2U1Mhe4sdurD2+t8vu4DvOcKAbXA5DPCVrMa1eJ2xIHr1th0T9HJmMG9Lk8AyJw+tag5M8JTS9nUAqY6qNDK0DDoUNV7MUIGlu96CN/y92ijJLnBzK6j82Jg8iQNSbqlX26wrUff9HUt6vosXegADxUDlDBNr1jY+dp8Gw3FPSXlfuMYvzzpBjFcwIPmRE77KZ/QMRA1nJ6tRYHisy27M5OahfjZlqiPFH/zpHjZwNPkjDNZpxE YEot6hza gdDmIufW2DM9jieDt4XwTpUoPiQjPaXndnLAFl6PfOc0wfiiTS2HieflTWwC125GWzNHMSp8E95HHzYiJ/t50fm9IVK7gl16tADJnk3qhtb0aSyx9r70HiJnIcA9XXSqQDGHnLNtVYTK/aEhkUxP2eYRznqrqfLT+ZKSxWQTY5oTyULUIEOtCXMPeHf2+dRL4H/cv/ANBN7uCJTJ+IJILnfbE4puBgQjWcASUBpxt3PJWreZ6NHOz8MaTfbAhMqQN/xV0eygaxWpaDC38GwIQQ0iivus0FL1Dfpnybn6nhEe5jFf+SPqEUKY2NAQ9/6Aw4J6pdRniAlNhnjao/Yl/IrvPHYze8RVY7Svy2dE42BqcUcAexGWukB3jsg2D9BPHqIz59kbxoXoN7rutRLfTjKNpfhaCF/4kOI9e5i+ysAT5bHeeRspm6238uQs9TPbe+s2H7MfvSYF7QaJnRqQhBcaxDQJfce65G2c8A8JkUMS3TwjNgHO9YA+CFItTMoTZ/eLD67ac/B68NeYZ4SYtTCSg5df2UWSFAtvWWmcuIpULCl2f2n6ERQNf4xYxNwNv+N6CKbItMCkOrjA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 59 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index d9434df24d62..8b9e4cab93ed 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages; + unsigned long page_idx; + unsigned long address; + pte_t *ptep; if (!pte_unmap_same(vmf)) goto out; @@ -4166,6 +4170,38 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + nr_pages = 1; + page_idx = 0; + address = vmf->address; + ptep = vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr = folio_nr_pages(folio); + unsigned long idx = folio_page_idx(folio, page); + unsigned long folio_start = address - idx * PAGE_SIZE; + unsigned long folio_end = folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte = ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4261,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte = mk_pte(page, vma->vm_page_prot); /* @@ -4247,27 +4283,28 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + folio_ref_add(folio, nr_pages - 1); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; + vmf->orig_pte = pte_advance_pfn(pte, page_idx); /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4291,7 +4328,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl);