From patchwork Fri May 3 00:50:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13652246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19FD8C4345F for ; Fri, 3 May 2024 00:50:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A42876B0088; Thu, 2 May 2024 20:50:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F3036B008A; Thu, 2 May 2024 20:50:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 894316B008C; Thu, 2 May 2024 20:50:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 692D46B0088 for ; Thu, 2 May 2024 20:50:54 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D5EEF1A0AF4 for ; Fri, 3 May 2024 00:50:53 +0000 (UTC) X-FDA: 82075254786.06.7F44D37 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf15.hostedemail.com (Postfix) with ESMTP id 1938CA0007 for ; Fri, 3 May 2024 00:50:51 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gg5L8Cr7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714697452; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9ZtaFVcTSDB4ul8nNoaO/PcEtn9Rrjc3ITa59z6anjw=; b=7HOGwnd5uO3lZpO/eUTKE9Ms7CpAMZsGIcDgGKWMqpWTNmzs65HX9T3Ocnj+pj4OEWTh8F wPBWSMDtUlWa5EvjbcDV659BpTTYzQlj4UWop+Eiz6bJWi93t1ZAhvcxZdkwMk/tpLlyAV gp83h56BdZaEVsJt/j5rIywXYjnlzgo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gg5L8Cr7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714697452; a=rsa-sha256; cv=none; b=OR0nsZKFLUvOil7ghx4nsiWvCDCdgOTtpOw2pwX/vaQuiisgPgDcw13KhUbEdmuDj0Kf+R zs34gehDvm9JYsHZSnQEp0adOWBnVHEgp8x45z6LkjCBdNhac2FC42KV9+nwd748kgdGyy 2GKLYLVcN2H/RZBx+wzukmTboEsUDwk= Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1e83a2a4f2cso58180775ad.1 for ; Thu, 02 May 2024 17:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697451; x=1715302251; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9ZtaFVcTSDB4ul8nNoaO/PcEtn9Rrjc3ITa59z6anjw=; b=gg5L8Cr7QkN8i8I6HM0yRBuK7/xe2ryly4OyrHUABKZ21kS5Yv4urGz2cvrNtSuN9f b9sHifPb2eYYAIPdGjdDwOsBMaZ96unIE/QlNTiFWwfifqYJxoCvZ4Mj3KieKAYxaesL 8A4izSFGHzD3DY8wVtkv3TlODhHdkayW4mqQFlapdehuDEpH8dI/bgidpYcMA6rhPBsO qcQhRMq20jSm+o6LN7xfDpqe6MHHLRcSeOW4w8ZisybwDs399xBFO6KdWzgcFJhsxCeh FkC/mNDDG1ILpDK/zYRVWeLmWkfHlzzS7ftDLq0vp+2J90HihQ/9bMqa6d2gu+npTxbc adCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697451; x=1715302251; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9ZtaFVcTSDB4ul8nNoaO/PcEtn9Rrjc3ITa59z6anjw=; b=S19alW3JNSaVH/Hf9SC0XGhOBTyaNkGg4/rEbIM3FDHrT2eff+Qi+PXcA7iu6Tw8te 8Vn484Sl2NqBYtOWi6D9UJ3u7LSw3bfr5P92Z6ot1vwM9s3BcSCqHdimaMV+Kg4dpucJ mnex8lfxhvlwavSWWx0012VVdd4wmyvBi7NksVa9fQYp5zUY5GjZ644d+R3UfoSq3CsF Y9FZ90fgvTT91EQiLZeg75BrQZJ3IqERBVVNJoKKVKJI8ADD3tZVBK36G5etCs+/Qnjq p7/RmAg/1+dppVFFpF6Osry6KqwH0evTQD/LumQ+mCRPdMgqVqFerSsAySCc7NpJaN60 GJ9w== X-Forwarded-Encrypted: i=1; AJvYcCWbyxgZ+2BSMJ4lVD8ZtG9jCtr4TR5FJgyO8zmd9BxptaXytAkEsGgTov+I89ElNns+gWXvqBikrJ0jkeqhZGALuSc= X-Gm-Message-State: AOJu0YxbjIQcjN+xIVvivps/zA8sqfVxIiok0jhvrrSXmuY8WBd9SgrS lqS5O0A9Kmn+f+SIjqNTD0NilE44kPt8i+g/srdqNuv8Qv5og2eF X-Google-Smtp-Source: AGHT+IEefQkadUJjt20GqF9jK1qsxgF1ScWbbqdgQAWnvfCb3u+RiScs5Eeaq3V8oWvBSMJbyG0VfA== X-Received: by 2002:a17:902:bb8b:b0:1e8:b768:2c25 with SMTP id m11-20020a170902bb8b00b001e8b7682c25mr1211101pls.15.1714697450969; Thu, 02 May 2024 17:50:50 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.50.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:50:50 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 1/6] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Fri, 3 May 2024 12:50:18 +1200 Message-Id: <20240503005023.174597-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1938CA0007 X-Stat-Signature: o4dwqd6dzqwaksrypk46dbasamdj81df X-Rspam-User: X-HE-Tag: 1714697451-316649 X-HE-Meta: U2FsdGVkX1/lGDCIq7L7C8K7fJe5KDkkRk5imtVEd+XAJtOqxDuI2seYbghBcYv6rFv9qd/VfEuA39RDNNFNsoAUMfAn0OoM6+TfAsnw8HKFhiHfkVPreRcyGu9iL+a67Na5eF7JjUTncR7XbcuQB2YbsbIBcSfhcxdAgL+FYmuXO7E4Ar4l82Jmu9g6ofLnjUoEXWFGex9as8uwpTmHD+okPfWXEdcszKZIIVmscYAjVqaKniOSkFxrJTo6gifZfZdNJolbS4G4G6LSSz4wjTc8ZvnE8uDiCk9urgwT8XpaJk5dHO6okIlIrsjmQ7jdUuJPeBIvK41DvFdALjsrmk8UeYMUNBRvJ9HTBDUBDXGN6uqeHzo77EcpSETsYmoEwht8JiFSclRNhDJ1GAr4bzkt5brY0n8kWMe0HyoNse7eCB22+rUYdqtwsFl8GGgedrRyETE4wWTu9/jGguPNwdlUPDl8iEa8jS+xKH1xsSs5bam52Q0qi6E3g3wNbcfH8nwCr7CY3pNxLp+pSkBIYAacMeGQ3MR2bGLJq9UA6DSTj47BT2GxyLXCfT9O897nUUns622cjhltYznqRLYdbfM++RqM4Gin3lcu0BEi/A8TtO57nfo9/rUIAZB+fMJHFCzXNRNWouHaw6vbCBYXBEPbZoHWXZaaCaznP6D+wNObHtMslF06JlkZkofkCODGSTU+6NhTR6/4wNM40/ChePKTd/9/orPJzo3CHTnaBJ/qYluYMJb8/wnxYJt4/Ah5oXBcrm2kSdnYQmj+gv0686EDbeJB7sL1BH6o2QE1FXkKtA/WUKHaEDy4AB3hmHCFHAq9hmZ+oHLKStbZruYkwRicIjxNkZrl5rKvqjUg86MP8zZa1zRXx0ypDNP6FCK2imTlFQEMdI7/sXGAPDox4J5682ANZqGE1IaTcQlNGoIdsYO70D1IuAl97cUOXWwnGXwDNQDIJFmF8fmPQ8s c9wYi1nl OAQqPKnkiA3pNOfclwTXR8ZFzXijeXdA7nWX4huDSPdcB/AUamuHihGMqlfPmF0wxUyPzarHXXFJXKgmZlS6MPG6kuijOllgJUvgsMHGDaD8MHhlJB0Tcc6PAC3rVFQPNXca/YD5YBiqcQhrqhMUsPaHcgku42Hp+n2dJl0cIoP/cH7cY3D90pXCUFN0zLzHr7r5fwhwh1eER8nsDDOjck9IzxLCvvBakTnzFcYrvX4Ms0dRf07E+rnXEbV6wte9S7zky+F/0RoU/KPNms37CF5MuTFCX5aZW2BLHuOm2cEDtXDr7ZjRPtkgTbGtXeINmLiNtoc3ofZnWdZpwvoZ6gO5dHqzfnmwNdB69VTlniYb1zlBnk5rWEcVp/Tm/XG+/n/ILYkiUs/zqet1Mlyj3awy4ntf/N33zoadCo0NEyUFE34OtTg4qkuey8mu2rCBuNop+YuSEhMmpujeapvDTuiR0PuSjc1KGmVnsGm4vhQj8yopmEWt3Y8ebZy9x663vvXaw1uJ9xxu39KSch9JIfdqYtLXrTmPEhKWf8DfdU+yh3ipvnqBEnuD2nvZH2IM8LXCc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Furthermore, this new function, swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Acked-by: Chris Li Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..d1d35e92d7e9 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +static inline void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index f6ca215fb92f..ec12f2b9d229 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } +static void cluster_swap_free_nr(struct swap_info_struct *sis, + unsigned long offset, int nr_pages) +{ + struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 }; + int i, nr; + + ci = lock_cluster_or_swap_info(sis, offset); + while (nr_pages) { + nr = min(BITS_PER_LONG, nr_pages); + for (i = 0; i < nr; i++) { + if (!__swap_entry_free_locked(sis, offset + i, 1)) + bitmap_set(to_free, i, 1); + } + if (!bitmap_empty(to_free, BITS_PER_LONG)) { + unlock_cluster_or_swap_info(sis, ci); + for_each_set_bit(i, to_free, BITS_PER_LONG) + free_swap_slot(swp_entry(sis->type, offset + i)); + if (nr == nr_pages) + return; + bitmap_clear(to_free, 0, BITS_PER_LONG); + ci = lock_cluster_or_swap_info(sis, offset); + } + offset += nr; + nr_pages -= nr; + } + unlock_cluster_or_swap_info(sis, ci); +} + +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int nr; + struct swap_info_struct *sis; + unsigned long offset = swp_offset(entry); + + sis = _swap_info_get(entry); + if (!sis) + return; + + while (nr_pages) { + nr = min_t(int, nr_pages, SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER); + cluster_swap_free_nr(sis, offset, nr); + offset += nr; + nr_pages -= nr; + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ From patchwork Fri May 3 00:50:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13652247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0110C4345F for ; Fri, 3 May 2024 00:51:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44F466B008C; Thu, 2 May 2024 20:51:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D81B6B0092; Thu, 2 May 2024 20:51:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 252706B0093; Thu, 2 May 2024 20:51:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 03BBA6B008C for ; Thu, 2 May 2024 20:51:01 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7ACA8A0947 for ; Fri, 3 May 2024 00:51:01 +0000 (UTC) X-FDA: 82075255122.15.15C5C44 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf15.hostedemail.com (Postfix) with ESMTP id 9A983A0008 for ; Fri, 3 May 2024 00:50:59 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gE6XFe7q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714697459; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wFl5l9cUG6TUpWzvnR4XiCKTQilGE1cDMcBQh7vvAdE=; b=5lHtZXp/dFN8pRgw5o7RdfD1QC7+s3z/hy79AT/iEqyrjd4YvNwxy7xJXTddGfEH+Im3gm R4Do9qtcqno2FjGJgHE3FknbAeo6Pv06yNzRDv9JApY5ywyZdtw4yGMOVq2FLoZKg39jhw WANk9Pev3m1ough9PNEQPY/uAQYs6DI= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gE6XFe7q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714697459; a=rsa-sha256; cv=none; b=kutd+Q3KgmG8Y1OypWpleWxo31KPebE7Diu2bq9dpwz0I53edUTInRoHteE8EwDuMAVk1R tIBiKIksAv3D44pSuuildR4OXO8GrGgWX8aTvuJ+Q1I8FEG6xDzmfyhLGnQ4i7bgD7rlet qLI3ryZbGXcZDWY54M0RK7gN7sYZnys= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1ec5387aed9so31516245ad.3 for ; Thu, 02 May 2024 17:50:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697458; x=1715302258; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wFl5l9cUG6TUpWzvnR4XiCKTQilGE1cDMcBQh7vvAdE=; b=gE6XFe7qiit3jn5vV2Qrisl3SaTgx7OnShOlg6hdfDM6nAujZ5yLpGOigYNDHLxf30 KzltE3HtgJevgqETlekd8T08Ej1VdpUDFRKNJDI9xb3Sx+fX8VyC4wabPtrh8aPnIxkN 7YzsqccxHC+tfGc7Y60gw/joNhcbT4vitDeMoeTGRurUagcP0euD1xHoU61f2/02cLM9 nNnhfc2mdBZb7k+PwpirguohwncleuK/kkSV4urdD+0aUQYS6VYOGx5/Y4c9OvVf5IZf eK38wm5DX16ySx5yODed6Tm//FWjGwVFwqy4Fe2p9KXm8ET0a5V6o8IDxaL3GyPBueDy gFrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697458; x=1715302258; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wFl5l9cUG6TUpWzvnR4XiCKTQilGE1cDMcBQh7vvAdE=; b=R+JUPchGEu3i+ZGIuTPO+S5Eeh6x5M8G4WCnlNedsJT5y/h2wRLTuE0AB0jlCZqJ/T 27Sbg0gcNk2FR1K7y78Wk0wtwT0tdYIIY6TvKgKSp5L8xYzHKXDu6SYqLDw7pE7FoFmC b66B3sjrwu9qZ/yHwxFu/C+sFYaa4tZBMmLIOtd7929aUCH/FAeIEu1bT8lTkej8WA9k zwTnjHy8sBmdbAdDby8wjwffDJFboP98WEx4oyMmXBYA2wKQk+ydHA/qttn+mLi3S/EL nvAO0I4T/2K27NJcQ/idWp3tPDXL7ji/QvtjXQmVklB2CdYxs7YQ6OL7ViytrhOGuPsk LXGw== X-Forwarded-Encrypted: i=1; AJvYcCXlk8TKQPrt3t+TgfIYCG/LWy3fkXVgHQ2PV3PvCEJU9JY06UlPTkXFU5Ua5P/3pwMs8jhJSXnlU4JXgYUOXFOlQ6k= X-Gm-Message-State: AOJu0YwET/IR6rC8B5nGlOSHIoTjM13BWSUes446Z0CU1Oh66Qg6PH98 IxhD0Tix8B1feNQARu1SF3x/px20QqQi+ByWFFpYL3cvo5g5k//K X-Google-Smtp-Source: AGHT+IHgGPs6Tx/aHvD+V7N1MdRzTr0fR1RogpKIZp6wEsXbZrG/1C29hAfRf2MxmZtp2ra0jMkL5g== X-Received: by 2002:a17:902:b495:b0:1ea:f7d4:cf2b with SMTP id y21-20020a170902b49500b001eaf7d4cf2bmr1203978plr.17.1714697458583; Thu, 02 May 2024 17:50:58 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.50.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:50:58 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, "Rafael J. Wysocki" , Pavel Machek , Len Brown Subject: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr() Date: Fri, 3 May 2024 12:50:19 +1200 Message-Id: <20240503005023.174597-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9A983A0008 X-Stat-Signature: izszgn8oew8z3us5bopqken61w1edymd X-HE-Tag: 1714697459-867255 X-HE-Meta: U2FsdGVkX18Gx1NVaqwYKKo1lq3W/mQtVVNPxQDzgPtiY6OcXz6tm8XlFihLoutX7nON4CLFCnKJArCwns4ySndot0qBYYgFvb8v5xyBJDQi8qqeVGCAxDxpOjqDTy+GOXajL3Y3cx6Xpw+VmBuIiGX0FOJ9Vn7mzIRn8m0AC3LqcBIZ1bu8DRWyMKKspebtKLt3ZMBhJrGdpDqaRi+RU/3FcaLHdYvphpZOerO2sYBcN7qfYkkQcupL8vscR4P8MrECVk1Zbw7qOwwWYX1tBnrWgmykIemvG7hbGzh7TozwKr9QQIXeRKSm3dVfHK1gI+VYeyaE+Uzd+KYwPrDM7cODDuBz22mxd1wlaamYEZjJ29tP4lIMt4LbpUe9NgyxcxoPSc3lrWpgUroAqATd8mb0aCyTGUplpugt2yFAvhpuTWrx1GXnXR2EGZCJY7DYb83sakJiZpQTC/ZMy+Mg7J1kpxkEfPAu85PLQtmBEXE8jJ3Azkhq4paLBPnmfFRPBbP7nuNEEyRNS29sfxPthZ3RxCH8sNOvuK1DoEIuTu+3snk6f4/ey7JnVJ6DY+96s3/LniCzF7DxwMctwazZJWrTzBVN1qDMdJf+dFJNwIPKRh/aJHpqf7bWv6UfbPpeqSk615KALVfVulaxx9JJ9LUXYw52ilJsq2Gh3lAoZ6UyfCYMug6O4Zd+7hrLsGYQJ1nqXLz0XTLuzocu1yCtGuFJytf6Eb7XnV1ipnL9CgZcKj/lzJOBrqE2DRbHg019gZDM3q0h8uCaXmOwzJHBS0vml84XJ/T1zlAYp8X2BOIsmh6kgYRxK09iwaD0xMF56s9fOsLmqVF/9y7YyeOyw9AjRjpMkSV7LPi5XSMd15BWFHJHXIU9C7ErYmqW1qH9lBH13vCqWVebaqAm4TK7ArvN65rFEQTcRSWlNJVYkaii+ELdDFMJefq8zq+BDmdyR7BvZekj0hnRz5+JEc0 ixsVmJ9T lcDHXKq+xjZCv97VvIdW50E4eEDJWYzBrGQc+OO7+C6sViA/vut8F4+hsd74IElYA070I8MnbJTN1e8/nNTwuLgUId0Avzubd/lOqH8pm7pRxWd1yDVT1nkqP8AtUDReKzf3rRdticna6wiRPO4c4UjiNxaaAHn5uoW7GmN1EX/JHF6a4n32QW0Hby679Ry798x39+VqADAHBjxp5xgFCeMJI+VX4L8c5RIhg6PZuDd+/p9LOAgpS+zaxrMlMmTNFeDj3dWfzEnPDj77ApqJdvv4+uP3DoNiSZe+MHRDmIvAzypzyOzBMlf1kQN/4LnilfQyM+oldHG7GJnyKfs+Q+VFxYUdgMx1ydIlZNxMC4SJdrsP6iAj2gxa9pxWW9eOyXf/nfCw7uvB2KxD+7l2mnP6Jrsn9NdxwDxxkgsY2cz91ma4n+uUSMzNMvUlu6A0FVRW/V0TkqFZNwK9U5IvABOP7GeeRxls3qHJp3i0oSXvCDf6Ki6qwivA+ur2/hoG0cWdts/0MfAPYC0pPsQn+wC+w1WJIPVh398QeDdJeUcrA6txM3oSrLk2PHJekaxYV4ut0YjAKL9Ss9xbMkqyVHNOXkc/lnMMgSpdCSjb9uTmwVSJLJTjU5pp9yLXgPSOv/bS4FJSgYKtxG5TwEH4OR/Lk3px0UmpxJwOPfFMhyMBKRPA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song To streamline maintenance efforts, we propose discontinuing the use of swap_free(). Instead, we can simply invoke swap_free_nr() with nr set to 1. This adjustment offers the advantage of enabling batch processing within kernel/power/swap.c. Furthermore, swap_free_nr() is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Len Brown Cc: Hugh Dickins Acked-by: Chris Li Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 5 ----- kernel/power/swap.c | 7 +++---- mm/memory.c | 2 +- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/swapfile.c | 19 +++++-------------- 6 files changed, 13 insertions(+), 28 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index d1d35e92d7e9..f03cb446124e 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t); extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); -extern void swap_free(swp_entry_t); extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); @@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp) return 0; } -static inline void swap_free(swp_entry_t swp) -{ -} - static inline void swap_free_nr(swp_entry_t entry, int nr_pages) { } diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 5bc04bfe2db1..6befaa88a342 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -181,7 +181,7 @@ sector_t alloc_swapdev_block(int swap) offset = swp_offset(get_swap_page_of_type(swap)); if (offset) { if (swsusp_extents_insert(offset)) - swap_free(swp_entry(swap, offset)); + swap_free_nr(swp_entry(swap, offset), 1); else return swapdev_block(swap, offset); } @@ -200,12 +200,11 @@ void free_all_swap_pages(int swap) while ((node = swsusp_extents.rb_node)) { struct swsusp_extent *ext; - unsigned long offset; ext = rb_entry(node, struct swsusp_extent, node); rb_erase(node, &swsusp_extents); - for (offset = ext->start; offset <= ext->end; offset++) - swap_free(swp_entry(swap, offset)); + swap_free_nr(swp_entry(swap, ext->start), + ext->end - ext->start + 1); kfree(ext); } diff --git a/mm/memory.c b/mm/memory.c index eea6e4984eae..f033eb3528ba 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4225,7 +4225,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, 1); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); diff --git a/mm/rmap.c b/mm/rmap.c index 087a79f1f611..39ec7742acec 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1865,7 +1865,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_done_err; } if (arch_unmap_one(mm, vma, address, pteval) < 0) { - swap_free(entry); + swap_free_nr(entry, 1); set_pte_at(mm, address, pvmw.pte, pteval); goto walk_done_err; } @@ -1873,7 +1873,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && folio_try_share_anon_rmap_pte(folio, subpage)) { - swap_free(entry); + swap_free_nr(entry, 1); set_pte_at(mm, address, pvmw.pte, pteval); goto walk_done_err; } diff --git a/mm/shmem.c b/mm/shmem.c index fa2a0ed97507..bfc8a2beb24f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1836,7 +1836,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index, * in shmem_evict_inode(). */ shmem_recalc_inode(inode, -1, -1); - swap_free(swap); + swap_free_nr(swap, 1); } /* @@ -1927,7 +1927,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, delete_from_swap_cache(folio); folio_mark_dirty(folio); - swap_free(swap); + swap_free_nr(swap, 1); put_swap_device(si); *foliop = folio; diff --git a/mm/swapfile.c b/mm/swapfile.c index ec12f2b9d229..ddcd0f24b9a1 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry) swap_range_free(p, offset, 1); } -/* - * Caller has made sure that the swap device corresponding to entry - * is still around or has not been recycled. - */ -void swap_free(swp_entry_t entry) -{ - struct swap_info_struct *p; - - p = _swap_info_get(entry); - if (p) - __swap_entry_free(p, entry); -} - static void cluster_swap_free_nr(struct swap_info_struct *sis, unsigned long offset, int nr_pages) { @@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_struct *sis, unlock_cluster_or_swap_info(sis, ci); } +/* + * Caller has made sure that the swap device corresponding to entry + * is still around or has not been recycled. + */ void swap_free_nr(swp_entry_t entry, int nr_pages) { int nr; @@ -1930,7 +1921,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, new_pte = pte_mkuffd_wp(new_pte); setpte: set_pte_at(vma->vm_mm, addr, pte, new_pte); - swap_free(entry); + swap_free_nr(entry, 1); out: if (pte) pte_unmap_unlock(pte, ptl); From patchwork Fri May 3 00:50:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13652248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31C94C4345F for ; Fri, 3 May 2024 00:51:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7B436B0093; Thu, 2 May 2024 20:51:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B05206B0095; Thu, 2 May 2024 20:51:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A56C6B0096; Thu, 2 May 2024 20:51:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 785A26B0093 for ; Thu, 2 May 2024 20:51:08 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 39F2B160164 for ; Fri, 3 May 2024 00:51:08 +0000 (UTC) X-FDA: 82075255416.29.C734F2A Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf05.hostedemail.com (Postfix) with ESMTP id 6AB6D100004 for ; Fri, 3 May 2024 00:51:06 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AP+zSDyD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714697466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V8/xoQXqz11CQNwI09Ird7+bH6VcV03qRwDFMsnqJvs=; b=pprtE019vkmfGRGtijrKxPim9+W6YHie77cmTaOBlPurtrMd4RQpQZPYZcJ9Wv/Wvj7EAd kA3I3sEFf1gpWZ899ZtPsrwN4w/zfyRnuA4ROWeUHmJuTklrjcCCaF5m2xZziMslrI6jGZ rxOh7sapa5iXbgFh5Ecm62QqIBv0Av4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714697466; a=rsa-sha256; cv=none; b=ej1UugDRb96npqTEZlJ814h9JXyJX1eGM7MKpiMtosRa3boB6JRogPc5VnQvP54vkS9UIz T6cSNiYDT78UPbv7U4578En3C0LsCacvJVKYWx50JI4l3fxdCuxnqJ8mzSOgFbIXV5L8G+ ErfCkOz3iZND+FAUMCn69Tt4BRM+MvU= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AP+zSDyD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1ecd3867556so16100995ad.0 for ; Thu, 02 May 2024 17:51:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697465; x=1715302265; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V8/xoQXqz11CQNwI09Ird7+bH6VcV03qRwDFMsnqJvs=; b=AP+zSDyD2cbertc81m97nnSYNoeAekWZ943XqPXSMx5fpyvqEvzyrI0Hdob0Z6u6nB xQDGCBd8f/ys+dh1OzBNZxc9ThJ6y2TikE9o+A5ZdeRiKV/1WY06W9wa0qReJ7/Ri9Tl O/znHUkMkDl/Zd5g/gjehfs6ZnNMM8qbduXwXj/js14C+VjUs10+43vYVUo7bzaKzbAX iy9uhR6hzvytWdWQzpJOjRMtCAKWKG6wY9JORnrQv/scI7tth/k9on7SzW0SUn3rp386 JZxUtSwx0xlJadbLu9ThfDSzROKEccPsz3TZYoJebxdFebWN+06hvIHdzRXaTMzHx6c1 znwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697465; x=1715302265; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V8/xoQXqz11CQNwI09Ird7+bH6VcV03qRwDFMsnqJvs=; b=dcxWEVr5jHxu96CWrz1V8cUEYhropgrxfMh5tnEno+FTxR4V3M7aDEMGTawnc2sayV GTpRmTrgnCSNxIA7fHeZmPeIoSSwHu3fZGPipjl69/zFV2TN3Slxo2S6YaxAXM3TSjTi JyJ0vdbzTgHbVf7IRTr8ISs7Y65WXPUNDNbzLP8atED8HOpk1Mu7DHPSF+ymMQY1+5VX kiveNw3YT8XyALWV0dIOOBnZ6/EcFdYfPyJ2YX4aCxam6jjaOgSx30wYQyeZacE1fDPc LWlUTATOaC6sDKp8h/fFSzUYLcgkEb+EzrES+ZbL8I3vts6ZnORJzCbxZOm0a2hZnhbs HmgQ== X-Forwarded-Encrypted: i=1; AJvYcCXbh+r1HC8n10QwmqTQFYaKV6RPbnwjQKFenbw/rYpXMRnoc+GbFZ1LSaCV4lCb+Mrjk31Lfjd2/1DsAXdoBMYs5Ps= X-Gm-Message-State: AOJu0YzawGZelcMhIT15euXwvSOS5RPknOpSKcwSadWlSbAcqxyBMPVC CtmaenOQwlt42KCqASa/vyySuXP5FrE3ph0uczIQGelEBnG1Wl6e X-Google-Smtp-Source: AGHT+IGFQTJEucm2ac55Okxx+HqgTWR7zEJF87WiBhCpMaLEsrJyrBBOIayzx6pibugr3l3cuGjLhA== X-Received: by 2002:a17:903:2384:b0:1ed:36dc:a570 with SMTP id v4-20020a170903238400b001ed36dca570mr620343plh.49.1714697465334; Thu, 02 May 2024 17:51:05 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.50.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:04 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally Date: Fri, 3 May 2024 12:50:20 +1200 Message-Id: <20240503005023.174597-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6AB6D100004 X-Stat-Signature: rmejz7ygs1brrd8gex7ewrdho7jm3rhg X-Rspam-User: X-HE-Tag: 1714697466-685770 X-HE-Meta: U2FsdGVkX1/QvhRCXfNh5zrz/kAxHA/d3pa7StQzpMafLXlPdyBCLLDm/PtFbYsH/VbuQ3xTk4aKSKkW3LK1HaI9XmiZNe2oYcItOJaw9Q3hLGeE/FzTqbxHYULJBrsnPhzM2ciGJ6SuEceHFu8KZEnHOPt+2z31ol7L5VuZJJw/LMMBn8tLp6HMLVyp7FIKB0byrbfdVxfnUdu0KZ/JvJwlmJyjAr1SgU7cYAunEiF9IpkPluT4u/Sj9wUFEKEcgKN7ZLbm0D2gfgXf/2am4uCKuvVb47T2ciWYTBni7EyEt/wuj3PfpZr2QHQGo7ijbJfSiT95RBhp6fLC5gWbyS5XNI6XLr/EGW89+ughDUnrQZZK38Dg6C7+iKE+qaF81OmJLOyaTJa8mB4grbzqs8G+7fWaG5vWnl99PzqlgdYHkn+lPrsSSgkcJtiAQHdbnCIJenpLba4YnigWsIfSIvGwT7q2Tniu/vXZDA9P0YRd5vHjgMTGCuRVcPWXglEGgBFEwe4PlcaFdOsdI44szX60m+dO6tFBwFFO7TwU/kqk1+EXN/R7EYh8Hxh/TQ8QWZHb5G3T/+iw7PPzDeKIyAXdn15GJgMp4YQaJj6N9om1VN5xZyQqGyfU0bVdq+bNxLxgVyGelYFqzXruk9dqOaK8ODZMlobNdMRrm3FzWdI5ruMX+Hf3uH3z+2enYjFFSSkTGNBFBR361fznBTJPyM9UObihuT8ADtcI51X6SLcCQOHV5UXcvwLoU+5HCrNLGWa/+nmd0hU9ooskTbEfXxQPhi/tlZa2bcBQq64OjnlwYY8NQ6pdyh6DpK/igM44w3ByjdiQ/CAQVNN5AnbLi8GdwDXsgAsT/PcN07I9QMlqN1vxU8f7oOF6BxzJKZYjctxoYqoP6WWKbBtbzmeK9uziF/U1by2u8jFJHWv/LWqpYzJyK2mEy+W6ZzHPunxjMxYGHELRqayiTkuVFuA CuAZHltI A0mJNlggQVxruKE5TaYBpdY0UPPNaeMs1hN7ulwE5sE4+MijoQ8WV/jejYe+rpF38Rtqf83QVBb0rqlDC/50adgjeOFP4bPYoj9HHxrCZBpMinJEGOIMw6m39gWfRETfsLJePr6F5cWXrDQ7env8aiXsfkp/lZcmwTiWug0cOPPHVcjk5ds1tdTPt/BgKdZtIDYYaGVnzB2X0S2MxtmQuZ5LKX33+RBmKxUY/znuj1nGzeeSdoo4IonYshteEoH5pMIqk8cHFujKFUzIfEC96P+hr2+E8w0rzuSkUDEqUgFHPTmEYf1P3Bkg1joeZhE4nFltMLlkB5vJ8mxllLhKaHftPiu9+miIilHhEHa0BWampUZHRyRpcF00+NwMOWmxHj0WTyOoHB4+g44IcoqRkB5dak0P7oqjl3tT6LgrvMnkUCc0absJOQvi1iXHlTNQqJS/T1CvJZ/BUhGB0ZPjsN3LHoNe9kW5dtopIPAXwnzZlvyTQPf7sTGaaMF8wBPG6tX92jAASFWsWR8yrQUhB+JdQKwv2EfhN538jImc/FPUDhdmXZc2wVhCLx56tVkA1hkDJL0srn5DEC2ja8uvjqvJqow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song There could arise a necessity to obtain the first pte_t from a swap pte_t located in the middle. For instance, this may occur within the context of do_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introduces pte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently, pte_increment_swp_offset() will directly invoke it with delta = 1. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index c5552d35d995..cfe4aed66a5c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, } /** - * pte_next_swp_offset - Increment the swap entry offset field of a swap pte. + * pte_move_swp_offset - Move the swap entry offset field of a swap pte + * forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. + * @delta: The direction and the offset we are moving; forward if delta + * is positive; backward if delta is negative * - * Increments the swap offset, while maintaining all other fields, including + * Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset(pte_t pte) +static inline pte_t pte_move_swp_offset(pte_t pte, long delta) { swp_entry_t entry = pte_to_swp_entry(pte); pte_t new = __swp_entry_to_pte(__swp_entry(swp_type(entry), - (swp_offset(entry) + 1))); + (swp_offset(entry) + delta))); if (pte_swp_soft_dirty(pte)) new = pte_swp_mksoft_dirty(new); @@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte) return new; } + +/** + * pte_next_swp_offset - Increment the swap entry offset field of a swap pte. + * @pte: The initial pte state; is_swap_pte(pte) must be true and + * non_swap_entry() must be false. + * + * Increments the swap offset, while maintaining all other fields, including + * swap type, and any swp pte bits. The resulting pte is returned. + */ +static inline pte_t pte_next_swp_offset(pte_t pte) +{ + return pte_move_swp_offset(pte, 1); +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. From patchwork Fri May 3 00:50:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13652249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5A46C4345F for ; Fri, 3 May 2024 00:51:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4EC5E6B0096; Thu, 2 May 2024 20:51:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 475CB6B0099; Thu, 2 May 2024 20:51:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33D3D6B009A; Thu, 2 May 2024 20:51:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 13D996B0096 for ; Thu, 2 May 2024 20:51:16 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BCDC2405D1 for ; Fri, 3 May 2024 00:51:15 +0000 (UTC) X-FDA: 82075255710.15.F1C0B9A Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf21.hostedemail.com (Postfix) with ESMTP id EAC251C0004 for ; Fri, 3 May 2024 00:51:13 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bH1Lm60O; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714697474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iTZWC4mSVwFMl/e34a7NVZCzCVLqGdUKErQlq6SYz3Y=; b=X5KY7ViVV6xnCKlJbdO2GtpQr9dpIwr/8frubz2T9Xwe4ax7WmQYeGMmDSJT3cJZe4P5n7 7IzInH4eCZ8yNWQoP50UtIGj33L9kOlIIC1ivlYVHYsNqvR9ymFFUNmYZhXONcbVVW+78s capDts0f+MzSQZFTzJgKu8O+V0KtVzQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bH1Lm60O; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714697474; a=rsa-sha256; cv=none; b=XIKLMOHa4I5Bn0qdAEyhXKGl0yYWfOeRuqFUaEb9ypmjr24bcTHRUfpp+skwO/Sm/hjRKk +I3NHaUBmhxZyodd2zQqOJPlNaPC7inRsAfurTf4qPurzyMewzQOCoef7w94TC3qyHopQ7 5tiONWx6WwldB88IW5U0AVMfYkjGp1Q= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1ecd9dab183so12675905ad.1 for ; Thu, 02 May 2024 17:51:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697473; x=1715302273; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iTZWC4mSVwFMl/e34a7NVZCzCVLqGdUKErQlq6SYz3Y=; b=bH1Lm60O0Lr+9Wacspg9Hw9bKV9d2HwVLFoQapN+DN2PVDPO/QKFjiPthJcY0Pyqd7 t2O2Gc68/ZhMKe2ukrFtXN4bfFyBd2aHP4IGnndKVVLF18TlOayP2xQVPW168SvEMG0D 3aq7Fm/2Wbprq2zoc0tikFFCfGaLAOND4XoJq67AhCqoexbwMPvXcaHWJwF1A26xPjvm DUwK/xNtQIEU9dPsnjGev9lyq/qYgdHlgAqnHscJcaUx2r+fmZMKUJ9sQ/uNyBQSqg9f zDutEw49coRBC0Yfo/z6biZttFbA/NsUUm86xOzORtTHmHajdDeSzFP4+eezxwcTownD adUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697473; x=1715302273; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iTZWC4mSVwFMl/e34a7NVZCzCVLqGdUKErQlq6SYz3Y=; b=xEuADAqon9xdL9d4BuyNqXa1ctlOjd/M3B/xrs8VqMTDKpf6WfUXSfMsARcLTLcHFf Qq6x4b6Nk0BN4XEcC59AReHwb3fxZBgWZ5kfJO8uzLgME1NAOx1e8GFcnkqMQeePV+ub V2zjb5+pIzEnmW5tYhb7z/DUPy4X7GrYtf5sw0z8za/Y63pHFDM3OpLJJm1xLaeNvVfS 2SdwxuTpZeTl52ruRDCzws4MZz+e97qFOo/hNvKBi8rtRFIxDofg0gh8+aFoVxyEufuC tgs74VpWGWQwGqz/OoA3a4o3XW6c1dlcvNGJ2Q0N0j6ex6MFKyhUjpEVa6tnyjLfDqP0 PcFg== X-Forwarded-Encrypted: i=1; AJvYcCUTD4jDM0QbSVqe4DkW1QoxoxGR9dL+TgvMFlQYiBHjgha67Qo0SRdte2a7ITYkAyOnI06zsMJ5+x3hc85xxtvA0GE= X-Gm-Message-State: AOJu0YxghYUVxckgOSyaWSWpwR11kG5rVujJTU0Q2yFhuQa5QHasj8yW U2jn7zFf9UIoBfIMH7sRU5GAxypvZQ0zxwBE1Yoh1ZivI2u/Coza X-Google-Smtp-Source: AGHT+IF6KPbkDG4oPcXF3nnGu2vkHcu3vUbx1MmrFOaaiRUAVvVI/eyC0SeYLYYd7YsLphvtZmA6tw== X-Received: by 2002:a17:902:ecc4:b0:1e4:59a2:d7c1 with SMTP id a4-20020a170902ecc400b001e459a2d7c1mr6562669plh.33.1714697472877; Thu, 02 May 2024 17:51:12 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.51.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:12 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, Khalid Aziz , "David S. Miller" , Andreas Larsson Subject: [PATCH v3 4/6] mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages Date: Fri, 3 May 2024 12:50:21 +1200 Message-Id: <20240503005023.174597-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: x987oxecgoerqwajsbn9wcrzejb3urqr X-Rspamd-Queue-Id: EAC251C0004 X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1714697473-726275 X-HE-Meta: U2FsdGVkX1+d38qhjMU2xcCTnWQd5709B4YjpaiTpKMgdk8YfDmvoaXlxhlFeXkNsBKZR1pdV6fprYvkfnXD9Hli0d9jeIajxnVNeV+LQzmNkvxzVMIVNQP7yKmzb7HjGhO4ZBLv9uQX/xhEyulqno2AQRArQNC/VKV9cIXBecfEm9V4D+yQn0HR9g2ndh8NDrd6C30KQIQwJo0M8fpwXmNJxGIzo6e0bGXnSHhrBYCYuMxxjhUkKPiqSPi4mZ2KuhsiT6sNqLhVpDUQWT5bzv6Fop+hFKhF161XlEVXe1XbNuhiXJ57A8NKRHuXW1UQkUrvfcdXVBX7KYnC2ex4D/Il27xbRVIsWFfgpYidKCTcMa1hmZcsjxk2PZjZz3hg7/I2pAX1czk1DVKuzZ9UGgxbTxaBOykBEth59SflTi/CpqVMU/Z/muCMeGps7IE+SRu5CrUI8/eohH0XX4CdTwr57oatkbLAgw2NURbM6TpxAlK1LaSudxA8aXj1vZEvxsuM+2iasPsR8zqAlly7zdigXQX/ksnYN818cpk/27xNyWpaXIePgb8D2IhC3pdlylgSQ3h15ff44rHY+HBJQhp5ouW0K/mvjIXdpAk1BJqXH6S/+mj+hNQLH7fyBh9Y31d/WqF7F1vlBPEQSlOIb0j0bwJOWCuWrOKLVMcpPqrWM8InjXPy4F83bx/jHMN3A2+8ZoAG6OCNurPFQolLEm2ovbfEyMnJtpmQE+DYQ/Hfi2ADn6JYNpEqPC+v5j3nVYNoMKMTPEAi5ZrXvA2sYk3Vs3AEcq/cdcpOPHBNTDWzC7Dvx2LzdN1xHoakdm1qvTkh/J5XwmRDZYacVdx0RqZ8XxSzvwsYVAQ5n3w94uL92tHazX1w6G93mIaxWtPFc6BW2OCr2+/2uiS+X3w3xVCj2DqDCwxDIIYwzgEv3seaNpp3Ce6YBY1gCUJt4bNgSWRjWDnaNerjTxQFhoH cRehxUP1 Io24k/7eQUPKpW0AVLJCxQanD1VQ8U3V0GOOOj00rCzSkuMuJIIgiPYYaTsJ80lxeNdj+VyjXOSlE5vQY/o/GP6c9p1NTctCwe+bbpYNJc6ynRkOjNxTYwsFlC+r7qqrLT4vo/jaEsjnL3Kila0DabRP7mgvK7EvzVNQV+fskFBMweY7CoHj1H5c+ZzfnBxg5i1Yb6mmUMNJE33qH2DS85wZXcY8KvodOVvFuayC/bWr5DKRRXC3VGpahHf2SlI80DS6KBvuG5tWHIiJNSbGJy5mumCMYVYOpOyVU0co1CUlfrr4dpUJDk7Klqh0giqYYjtqIlZWgZntekJOtzN3K++Pc/uLOtR4jgRQ053jeMiwccZ3A6HrTUoz2SaSJQbhSmWpL8jvS1jZWd333VQsooGPyCgkPF80wifV5A+qjQ0fqzoOHC7NEVZDxocxFE9RMZZYGFHQMDQdKbUDY1wDQSXlRP9DMWVKeOkEIS/28HXtRwMFB+sm1vH9gdYoA2uUW2uhK/95DpeC5Go0RE0WpO/fj3mdy6BfxLid/ljsRxbx4TFMRg6Q4StwEED6X8b6JYHF+WpMnYM0Q899K42y8JJl2i0oXrFBZ5wJ3KqcoALDd50yq9klDn65t5IFnSoUjmDwgtAKY05tj7asWjndzv0W5wlPGDOg7DdFjXl6aHoBqdXJxlMKm2kqPVvEzppatnmZoTIX6PZHhdTgHzlOuWu1uIw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Should do_swap_page() have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enable THP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementing THP_SWAP might invoke this function with nr values exceeding 1, subsequent to do_swap_page() successfully mapping an entire large folio. Nonetheless, their arch_do_swap_page_nr() functions remain empty. Cc: Khalid Aziz Cc: "David S. Miller" Cc: Andreas Larsson Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: Khalid Aziz --- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 18019f037bae..463e84c3de26 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1084,6 +1084,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) }) #ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be @@ -1092,12 +1101,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) * page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page(struct mm_struct *mm, - struct vm_area_struct *vma, - unsigned long addr, - pte_t pte, pte_t oldpte) -{ - +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + for (int i = 0; i < nr; i++) { + arch_do_swap_page(vma->vm_mm, vma, addr + i * PAGE_SIZE, + pte_advance_pfn(pte, i), + pte_advance_pfn(oldpte, i)); + } } #endif diff --git a/mm/memory.c b/mm/memory.c index f033eb3528ba..74cdefd58f5f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4266,7 +4266,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte, 1); folio_unlock(folio); if (folio != swapcache && swapcache) { From patchwork Fri May 3 00:50:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13652250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEED8C4345F for ; Fri, 3 May 2024 00:51:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D7146B009A; Thu, 2 May 2024 20:51:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 586536B009B; Thu, 2 May 2024 20:51:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 427B86B009C; Thu, 2 May 2024 20:51:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1FFDB6B009A for ; Thu, 2 May 2024 20:51:23 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 969CE12084E for ; Fri, 3 May 2024 00:51:22 +0000 (UTC) X-FDA: 82075256004.27.F379510 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf30.hostedemail.com (Postfix) with ESMTP id BFA5080003 for ; Fri, 3 May 2024 00:51:20 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Zww9Db46; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714697480; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y50dQl+upnub8ly1QPHs/qiP8XcDN3yKo0FUp2Qgag8=; b=8k0e/QFh4RF50/PnKn7zdGwtMu1j6YZuiNqcquyxcHAjfS+gmgndbl9bi1wEJ9OzZ+DP26 9WNpobYC60b1qxoLxiNLuAtaRJDhXOgaHAF7r+faktEUUrxD8jNfndAFflRe/9DFHmCZ5E r4RqFTfsiKTn5rtwHRyDvFJ4wnqaUEw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714697480; a=rsa-sha256; cv=none; b=qWjTdOsp/gp+8q8yG2KOrBlLd9grxZ4e/6nAsZXHJB6//udRQRefeI2JWhIs5EDAIIy8i/ iqDUtTfOqV0kzuu2SBgyWaJ826KnZzGL65RLThkaF/v0Xi2vU1Oa1DpXNik2MxVTdNPwii b2Nz5SRhCk/AXgIhPV8tumGnYwULFVE= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Zww9Db46; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=21cnbao@gmail.com Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1ecff927a45so10984815ad.0 for ; Thu, 02 May 2024 17:51:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697480; x=1715302280; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y50dQl+upnub8ly1QPHs/qiP8XcDN3yKo0FUp2Qgag8=; b=Zww9Db46ckUe5YW/Eiu3Eo/Arwewm6tP5cdb/63AWgbOoyMQJXxClaYEUMNXz/qwhR XJjCgikKkw4n5+HcICnSdH1k3iUW/vp95X9T06FWnnG0WeaAV5jTjIOZAG33UC3nZ/Kj 8R8p08G/R4zkKgu4dHnR0Nf/2W76xzSlh8MmejmVCsYVrmCUZDbpXbndOeqaodbZn/7a GQu/bEO9lXg61UAo80fhp9ovoqekEF3AvDdQjabmBW/IMSsiUd6FWddob3Rp502lTVGz rDUL+9mzn0WOGKxrprWoCmGmkSU8nX1c1K/+rVJ81Yt540ymKY8uf80XLatMCUHMQHfC 9EqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697480; x=1715302280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y50dQl+upnub8ly1QPHs/qiP8XcDN3yKo0FUp2Qgag8=; b=o0gw2lK9Vpdu9RbzunFrwcg1lCqptgRzmBrA8SpGGcHO42CXvLpXa0Ep/Bus4WwtwJ ozGs+L2wOxxeZpL5z15rCBKqNXtSxm93Vj/tKWq1t2QlTI0qvjm54bvlxGyo0Ly9bK8E d/lxEzCneFgpeZdYl3EFwYR+Vejt6LANHkyT+EuarttGn6wH11EYe8/7jAfDU0v8bLgq dzD/gf0CKR/Ry8vGhhuDS5tnLucOZPkpLLJtSxyzGWHvHrqI9wi7h8tnRnZGfOqolrgc 0l00QjB9JPtWmPZ8kk3DjIX4Gh952N8zeJIKnAnlk1onyc2NJ5Fol9xnwn1hJkFJyqrp Zb5Q== X-Forwarded-Encrypted: i=1; AJvYcCUkQLb4QieewjPwLdksI+qWiFH3KdIxIy1RT5xvZ2X8icG5U1Vc8cX1QggR6nPl2u+CunBaZ0aAvsjv3+Pfeor5APY= X-Gm-Message-State: AOJu0YxfMGpeG6KxxptC0mk2yxDioawy74EaNZRcXNTso5SCIavs4M39 /HV/QX8yBrI+aamKQ7qlaxcTD0HRGRaBJVSFNybMv1H0YUeygycy X-Google-Smtp-Source: AGHT+IF/YGckt6lCQLRiwu1+5wNzKcMLcZACnvpyDGbcY6XjGmdPdZXMo8MwxWmuAEpW19MZtkC8pA== X-Received: by 2002:a17:902:ce8b:b0:1e4:362b:17d5 with SMTP id f11-20020a170902ce8b00b001e4362b17d5mr1757897plg.4.1714697479694; Thu, 02 May 2024 17:51:19 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.51.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:19 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 5/6] mm: swap: make should_try_to_free_swap() support large-folio Date: Fri, 3 May 2024 12:50:22 +1200 Message-Id: <20240503005023.174597-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: BFA5080003 X-Stat-Signature: 9ju6c7kwgnf3m3hccwrnh1thwta8ihpg X-Rspam-User: X-HE-Tag: 1714697480-68983 X-HE-Meta: U2FsdGVkX19Lz5ssjOrN0AT7fJoN59DCaiR6urmx6Bn5OGIvWCrpQcaf+Fogm88IlNKvDCfXG+4BxEVw+mETt01WOYRsaMan9n5CF7xAR/0tqjy5VnhsRc0xyfUismU/SSYreXR/aEhI/B4jWUFVN5Cbcrb6uiuEEONJ7PkLh+0oNfYFc3U0uJxbafK5L9TEY34Eh5ziwv4P5UUA6gw+ekNv8Q4y+Odc4Zba0WSAyxB8ykRm4ZLk3zmNll0IwCHeRaZB7pcFQlMxsSAzPLEih36THM00GDvsye76GJ9S4YqPu7XO+U2xauj+9FBPY62/LND4bELWLQW77Z/Spc15Gl28Th2/f2bdWr7SE5gPRtOoi0JtJK0YT3bjhLt7EUO5gVQHQCijCZudFPLddBGpJe+uJUBz7iNmj1DsXPsq5TQ80etlu9LiCuNJb6hOn20pcgN5K/gejwJ0N15ZV/O/7FEefgv1J8TZyJRtJ3wr71Ia7EOIIq9sL2qeqkUvU+4H3trUY6fIwYPzE0gUJIMfqISFlCPmOtXOsQU/zawrloZ3bPCDai0PsR9AaHwyMTr9fMlFk5E2yqF72/CqiGJQKv/AuJ998y4KIrMUUWRQ7H0Q6ajev2mWjXsN5FTm0F8Tizviy6DkRa2cWUCbpMF9gjC52lx5/I/LNvtG/ZwhR+/MK5mBBlR/mbsKwyESYjK7HM81mnSC8sz/KJHAjBxkIlZ6ZwuXc+89T4CK4r/UyQjeQ62cY4/w1+W+PiP17b3FmmaF0/tmskbxRfRU9fSfu1BjL2I+3EGZXkvO7XyRk7uZ+If5Ja2bqexulkFUdp4Na+7H27a23wLLGk7M+Kz/WBdb94i8HXgOjZri/2xiEx+U1an/KKhjzuNqRNpZHgaz1jiV8G4S6EixkHGxQGDT0hN6L6KQw0F9eSgumXLYj92z/3r9xgroMmJKtPWP/5sEkpXI+wP5ZzuCH49NVGj tOzLgCqF /2c7xisieDaSHH4Am+qXeFlAepDuDHe6uJ/ygk5MPUF9mjtA8ejRAiCFVY++HNg+2TeyKouKNYwPQ5Tw5Kn5HvdspkzVcgken2sd0p2VX1AfT4jTdGSQXShY2+Gov72DJGT18kWhQLZAWoH/MIGJuZfqwcnKCFEhlvZ3VTZicvZ/VJN5iVrq0CJTl0RUk4ftiaNDlolkeP6tTFuxCPFuZxdadVxYGfl6Y4mQATRD4nQlNTCEAX2eE+BFsYNt+iWz8Fi8X+GQ2if59cgYJ3b+kdW8zLsqmxgc2+zyiEqGP6CtrbBO49CJnLeGeQ2/+ayG1Wi+PfpgC/XnyPRtYarxw6j9yF6LT14IYYk8WPslSXIQpJmtrMwpgRnxAmnltEdvsRgeVMCA+6m4oBoUMitZt9pdUYo0mRmLcfbXBTwYsh8DpH0+/+qp+Y9j91sPAL7+MMkq1ese3GDLdTB9ZGdtVZnl9x0kZgh1m9v5ZzOeY8GK/TCfeIFh6H6Q4w/NruhNhzMqAB2BR/X/udsDR04evEVWvvh5qEtQLPYxdLmqZ+IUcYjhQArk4WgNoZXDGETrlpWJeumBzGlUY7KxuIR/XUreMh4Soodhi+9NgiZWIZkWNkzAmRX8xAkKwhj8DULeTPffsYGnMXuHu3bocsI7a/oJB4WWCgKErcIqfuvduxavJ2IxByyNhQq8zd+EGQ6mOx/dMKR1SQLMBrKjB4eiio4TCwg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages() = 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 74cdefd58f5f..22e7c33cc747 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3877,7 +3877,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) == 2; + folio_ref_count(folio) == (1 + folio_nr_pages(folio)); } static vm_fault_t pte_marker_clear(struct vm_fault *vmf) From patchwork Fri May 3 00:50:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13652251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A514C4345F for ; Fri, 3 May 2024 00:51:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CA6B6B009C; Thu, 2 May 2024 20:51:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2796A6B009D; Thu, 2 May 2024 20:51:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11B186B009E; Thu, 2 May 2024 20:51:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E689C6B009C for ; Thu, 2 May 2024 20:51:29 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6726D1A0DED for ; Fri, 3 May 2024 00:51:29 +0000 (UTC) X-FDA: 82075256298.25.AC2753C Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf09.hostedemail.com (Postfix) with ESMTP id 91D2314001B for ; Fri, 3 May 2024 00:51:27 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="J5bRGjP/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714697487; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pCPlcQiDc4aEgZRUGIFwOS4XTVcepx+utYhjOWkqIL0=; b=4sWuIFZeyA3912pgd7+ZHRc6Yn/7I8b2k9gyqj2zG3HFCtPgj+a4cg0DmQST9YPViOgatf 9mqPKSohiYgL7TXWW7OSFt6ce0J4QZWY4VDfHGZVz/NQJ2NRFt9fkNXgXLg1MVj2dKzMSB H6ZJGhnxjYWjobUn3twbJOupHu8uPPA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714697487; a=rsa-sha256; cv=none; b=edWYrQ/Wjr5Hp97vIGdSbU9/edVK3ZzvCepdP9/Fp+XcqHgquFOPDv/p2ld42fi2v82yH8 Cmw2gMchhe52Cj9EXMtxM+kYV0BSVIbzETXOsf/sY8IRqC9JZSyvgeyH0iVhWx6dObGqv8 wsoUEsg6zN6pt5t1XhFmZmf35Asoq1I= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="J5bRGjP/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=21cnbao@gmail.com Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1ed012c1afbso9547425ad.1 for ; Thu, 02 May 2024 17:51:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697486; x=1715302286; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pCPlcQiDc4aEgZRUGIFwOS4XTVcepx+utYhjOWkqIL0=; b=J5bRGjP/hoSoGrAgIG0BmmlJA7DXN8LdIkFpK+cxklevrrmCfWWmp0wIwV8xM9h84w iPdxZmWqxJN8SsWVi+xR2/6Tj1C/ilk92YukiOMzJbhb0/6+yM7VU5IUpnZSsyGXftqD kal12BMfeJQkuT5lPVN2RKm8Ep4+0xxXQU4Lm9krjLjhFNxt6vV/PGwjBC92wmDPdril aqWP6y1gx86pWYabKH3LwzzlqyaOq14vI4kF+Q0q+crV80GOY/XEBLLxAenQ32rz5VJp tFGHG+h029EhwCwYLn09u2suP56FKo05HO1HZ3spBQcclBN9z/TpdTTr6HpGa0LU1cT9 SDHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697486; x=1715302286; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pCPlcQiDc4aEgZRUGIFwOS4XTVcepx+utYhjOWkqIL0=; b=nrTe3KfeW50n65GLn3GlLfXvEv7R16Bi1TfoE79dcBb9iUqzyMfmHPdk8o0tzc11Sa jmObPkJrK4tvGEkPy8MDyqp/9Lbx6Htv6no4m4C3F+OiySbHPUnuMtvcOnubRAxTPD9r KwBVLSNf0/3Xs/I+rACjL13m60vXEbKCqGiOJCJjAqoMY74AUv9Mh0YFbHCxqbO37SYf vq1hg/o0JnA3YuiNwfkML5M+j1N3k8r03XU/6fVqqFbj5UatiIdzH2KkR1Z6BLMFJiFP d0QwIv/x4O9yATj2HF41Y04n5iUf7aMwgZ7YIt26j0VD5OsUgmWvBAv2zalDfw2eStDZ VyZg== X-Forwarded-Encrypted: i=1; AJvYcCUxiLrRfet9glYbFwmLNsFndigA1GlK3saUPSZOMLMei4i5+YnwLCeqtlRLKnV601KKhEOMHE6THSBG/FTHMg4IWO0= X-Gm-Message-State: AOJu0Ywr04zk0OPEaltcZDAV4dR5gN3Xzu9zcl4gf1fqIMfn0CGFAnF/ 4NwIdNPipOB+xd2lyNWSxx6PNagxaEf8oeltJAXWJcZBVANzgew8 X-Google-Smtp-Source: AGHT+IHl/5rfXtgmTPFxtFxlvOgwDmCzyiDZXrRPvhqN/z1hpeuCBJnRPssnBcPqlIhHPeY6Yn+CcQ== X-Received: by 2002:a17:902:c407:b0:1e5:e676:4b09 with SMTP id k7-20020a170902c40700b001e5e6764b09mr1726427plk.25.1714697486404; Thu, 02 May 2024 17:51:26 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.51.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:26 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache Date: Fri, 3 May 2024 12:50:23 +1200 Message-Id: <20240503005023.174597-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 91D2314001B X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: abhmekcrrx66r7uyebc1tane75qdk5bq X-HE-Tag: 1714697487-151288 X-HE-Meta: U2FsdGVkX1+t+YHZVvS1dFPg8vHFWDxijRJnD0/ane0idqNhMKVYvPam9lTKWKV70Wgy2pmipUL0/9EFa+V7qF2es7BFbhTXDgDiXRPvzcBQ9xE/g6eNVAxnLPAF2DSR/AJL343EcqYSnLrYCah3keu87o/jsjlGir1tWqH0N+AItG11JwHOd/aQMNwwGadWDjFehPHMbNVdX9V3j1bcG6C80bXxHCg3YUHbC1KHU0nWdzPk6NOhCr5kaFDNNm91+/yLznB48CPZCHxqnnrHS3kl1oZeVLA/ku2CDq6uIOBQH1/bJ7OZ7knptUcO+e54tcPoNXkQnvxjgfJ/UuRaF5GjVUMsATOOhDl24brScGaow8zb+6bhCYHpVLvLOaAvdu3qBAKkltNhxd2sYUiNsqSPFjDDakNKqRur092hpkvr5u6F3OlMMrjL6UaLJI90YNmjxFafCs02p5UGxbORXLA9N4z/pYY2aGRRnzEMMirVYCdGrOAftR5cbK4YSxyufh1iAp+l5HPNZctbFzIZKC44IFwdTPcz8lB3BJfxtyvg0nGptYf1iwWgxXBtgKhhjvp3GLNdMNk0GwMil7GLTvUHcvVWhN5qOjJa/CpYJEuyvSwciCaBzRPz7D1l41dtdxYE1KWGWtogeVz2AT4GT2FXaKt0eCxNo9dk5wNpineWuqjgyym8wTBLElb2ppQOVi6IH6kZ67dzhF8aDRYoAtzMJHz6kqsJWZUNHv2JElpuCk8eB0izXsLN3vEjAiFBaeIAzO+pHEBKhqQ/jQ33DRxMp9QQH2HojjeHsZkkjW6fM5fJP0UtXw5eBuJW9Hysnwe95iTaea8blTbRx2o9dOHo1Qrt39rGXVFjXzAGtiaQOOPwa25NeGNLwOFaXwIc3hgeAi+ZQaOtyqSq1wlO7qnzXmXMypFu/SZhL51s+pCeWQ8C4Oqh88efNtyAuVw4Mbqf9pempwOqyvk/BOC uObNGXe4 3I+IfTSG+6Zzyc6KG5q2KS1fFuJ9KzQNx16l6vw8Aio74vOFqAIeDZFASX3oFJtNx+wmiNLEoH1P34hosp0sf3UeeGzj5on/FOA2rgDU6MQn5BtqqNEDggF+/yQ0PtS+jQqDUtwkNYyr0BkQQPcl15gx2rkvFkrCyRKhjIblp/pRFbEMfjXamaJt4VyaFJWkiXGLU+gAasxhSRji0A5yXLI4JAGwsYsE/vvUcyhX8PnFUC1p2yV7jLaLtfXg5hVQakcoB6NpRYe3bjNlSRhjYqZNJBaCoO6apXQAmX+J7x2CkeZrmtyfOz2zu4BBIK6fpn0UhTyJ0GSXjYFwQOALmXAUhBV+63w87/k0TkwbRrf9dqacOQ9ltObzGC5bkuglYWC/Ufx/J/8WXOT0Uq5wMiVdltd+SWaaZzdtH/z/XKQmNliChNOoD8TSpVUBF68ktDc/AOL5RCChWdmoKchCKfG+VGiSj/qQsRcKA2z7PgRh+SzY2MgIpmyqNQQSwekykb1KfxtgIBRoEYU5U41IWAwBufb+Ayl7hzvNEslGj7iKAZK2Mi0HZ4qO2ew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts --- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 22e7c33cc747..940fdbe69fa1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages = 1; + unsigned long page_idx = 0; + unsigned long address = vmf->address; + pte_t *ptep; if (!pte_unmap_same(vmf)) goto out; @@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr = folio_nr_pages(folio); + unsigned long idx = folio_page_idx(folio, page); + unsigned long folio_start = vmf->address - idx * PAGE_SIZE; + unsigned long folio_end = folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte = ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: + /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free_nr(entry, 1); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte = mk_pte(page, vma->vm_page_prot); /* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) == 1)) { + (exclusive || (folio_ref_count(folio) == nr_pages && + folio_nr_pages(folio) == nr_pages))) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; + vmf->orig_pte = pte_advance_pfn(pte, page_idx); /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4291,7 +4327,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl);