From patchwork Tue Apr 2 07:32:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13613506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DA22CD1284 for ; Tue, 2 Apr 2024 07:33:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E569B6B008A; Tue, 2 Apr 2024 03:33:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E06B86B008C; Tue, 2 Apr 2024 03:33:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA73B6B0092; Tue, 2 Apr 2024 03:33:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A713C6B008A for ; Tue, 2 Apr 2024 03:33:06 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 78C12A07A4 for ; Tue, 2 Apr 2024 07:33:06 +0000 (UTC) X-FDA: 81963775572.03.A77B062 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf03.hostedemail.com (Postfix) with ESMTP id 9FB0A2000E for ; Tue, 2 Apr 2024 07:33:04 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ppy3xq+n; spf=pass (imf03.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712043184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HqLi1Li9mPDQzeMhmTcE3iTZttLxYb10NaxNd2/X0Ys=; b=ZVCw2B7KQxpocjK9IGI0Ao8QivgURa3CaChKhLCc9m9EF6GPV3SPP9lgPFuxvM5UI8O0VR 9kke/X6If+kbAcovoTxsIPyoK+6JBFo+WLbGPdzStk1ypZrleTWbtpAIlbnUxkJ6Hhit75 pnwjMne1UE6A4QtXhk2ERJJixKWgjhM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712043184; a=rsa-sha256; cv=none; b=jT6EUIi2IzMgLn8/A/GsEkwSNY+GaVZpqCBp2NNTxP9XpyBHF4fGmu+HdjJarQ7VllAVcj WcRuS2GF7WEVYfIa/3hkTbM2krZ/CU3RxEpiaoJ2WC9eWxSMEtZ11Xi+GiCiPj5R/rDsb4 PNx53RpR/ZdFMvzkMjw5vHdBEuSXd8E= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ppy3xq+n; spf=pass (imf03.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6eae2b57ff2so3689541b3a.2 for ; Tue, 02 Apr 2024 00:33:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712043183; x=1712647983; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HqLi1Li9mPDQzeMhmTcE3iTZttLxYb10NaxNd2/X0Ys=; b=Ppy3xq+nySyif7WjjV1VhXIyv8GeN1b24SQfA4UlPe0U3z/HCwtrmXGxJrWEw2PAMd GjnghLc31enQ2YxeqA0bLG+OBhV8ADiLQ0caqFJ1rSDN7FyUTAB36L4P/y5oCEahnQ4C EWaUOjFBL1EjVwfz8AOqe0MAdQeE1UUSQhzCwxqCw2IJT8heAQV2jmP2gCYfGS6P9bP/ shHxgfnyfAY3a10hSexxyfrGMgvM2vhVyw+gRHCRFr73W7hIs640ZoohUUgy3lv/mDcg 5Bqb5r6vddXsf1IPhoGDkLYMGY8vaIETIGEj+r9lzpAHOzxC/bOYUTsLD1wt9Iwq8m+Q 5sIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712043183; x=1712647983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HqLi1Li9mPDQzeMhmTcE3iTZttLxYb10NaxNd2/X0Ys=; b=V+RMqPsCzCAZ/zIHppJAR/6HZMyxPP0gkW9t5rTWm2TPOxvAuRQDszxsx3V9X4V2BD gdEvLlqQbdtkgbWuKMF7V5N5pU2nuBvVSxZh7CcusFsBg81SlUkZIEza9wAVxtaAQyGH 1gZM7AMoxngPrvBwiT+gxuGdy8lwwXYoGEySMfY6pXv2z5H3ZETNMBByBWfdRUinld6Q Dn+ntpEXVA4chcDNn/6zayb+JqUvBRRYXlKMqwK6+Wq4cgkCJn9539+Npul9urKWw+2m Z5orACavSFEyVAHkXSO5iF9HZZqhLMD+nlJH+BBOPeKb3saQOCsFbwpc9SnD5PhXQzJr aT1g== X-Forwarded-Encrypted: i=1; AJvYcCXe5j0bAiuOvi4byEOQRY9fG2kw0YTNlLnuXdmqxhxsRhAFm1VtgS7BgIZg8EOWgge/HlR09MDhQzn/2BKCkCANOlo= X-Gm-Message-State: AOJu0Yy231Dq2AdSdZAmjMP+8luP3yzLDyYfTa2OfA992I24n6oYAnav uEBkWAlk9xCeFUyLZOHWLxflYORDg65go1RdiStG4aREB76QyIkB X-Google-Smtp-Source: AGHT+IHQS2pMgzGS3ZZ9BOQYMyBw3FyEprxhgztZHlVlDCuWgKVjsoIbCWo8BMdbjCYd4MvwvVp6QQ== X-Received: by 2002:a05:6a20:17a7:b0:1a3:34c4:b184 with SMTP id bl39-20020a056a2017a700b001a334c4b184mr11043769pzb.19.1712043183375; Tue, 02 Apr 2024 00:33:03 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id x15-20020a170902ec8f00b001dc9422891esm10187588plg.30.2024.04.02.00.32.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 00:33:03 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: david@redhat.com, willy@infradead.org, ryan.roberts@arm.com, yosryahmed@google.com, hughd@google.com, hannes@cmpxchg.org, surenb@google.com, xiang@kernel.org, yuzhao@google.com, ying.huang@intel.com, chrisl@kernel.org, kasong@tencent.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, hanchuanhua@oppo.com, Barry Song Subject: [PATCH 1/4] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Tue, 2 Apr 2024 20:32:34 +1300 Message-Id: <20240402073237.240995-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240402073237.240995-1-21cnbao@gmail.com> References: <20240402073237.240995-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9FB0A2000E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 547nmisqf55ob3ebrx76y8tcoinq6onh X-HE-Tag: 1712043184-942191 X-HE-Meta: U2FsdGVkX1//awXXxhsC4RF91qyGYLGH3n16n87Hethj7OpiMWU2VVknvpRf1ulQ9Fhb6rcFwrzFJJn7Xj4+RIshC5YlIaZlalLrB0lqopcKl6/Fxe2z+sE1eR7zGn/aCOPURQ4902XcYFrpq/8Wfc8jOO0FwTTgsCU1j//Wbkytps6iQjXGJd0TguopIJqeiOwRtsnc05xS0BPcmnmLxBRvq+xgYHFAhXTBKjrzMrS/+hU6HL2IQ8+AEpnvYDnUHrSAX6PDGWslz0apU2F2C+B4QHXTy1VQpITYvJxW29knOM0oVtGTxUW1AMpJC4znJfS0+6q7N3Xs+x8dzW1l4Cj7lOdctaiRV1nxAhtBM+clam9zkmPPft+A1KwXS/fhFhFpU/vBgeKYKyOXJ4/OAGYbi7qHgP9dZaAnSdDFJ0L7y1Rr/4u4NdgV/TQ1PZv3vRSPMpbfVnaa97BtPoe1Cvk1cCW/amwpPnb4L1XkK8ApgnvtO+iQwo1aOAbY45a9580j8ezvax03RmEOty5K1DZ+WepCJqqFW3SHVhShurhyNiPVmufXIpVnN7BBaJoj5/1xQcaFtAq1Vj9FPmWmAVEQGqvKY+ylKKcbj57eMcJhZv8IZrBc7BGovoeUuLRC8WAIVHCObKykmxtTOyZ0CpBV7TKc6L/3HQVfMOUQ6BhpG7ABtDbIx0CcqjwcWGeGgGDzSAZUrAcko36fgO8J7tdpFxnM4FX4RSLLYunfmb+VgTCF+QKIBCr5CWC9jkkFnQ6utb+8ZWFxy0HmVpfX2WEe1esAWaLIwNmXOdZ9Kta1r/GDT2XVP/j9sHiqQcZEt+2COfC0lYEF6zZ3hosbv+eYs6zCuw1VKyqdul8WGluGSrES4+P2HMM5qyU3TPZ30+vIRscy8495d9GsO8Dy/xw+2+lSqeAvgrFXL/NZiKs1fXZ2ZUxsxnaXKoDqx4FeC8ZpOYct4rFgVT0x+4K 8aB1zPBI GlMeKkucVxR58qjJHK4m6cIfDba24hwP70Jz1SzMkOaM00a/8qJTLOZC53AQyOy4YZ3oaspCNblY0XxnGjpGOkBhgA7G4mI+ySNqZ8OSbc8hv8T/mGfjAMGixxGwR31Ki9rg+x7GZpRu2XPnpeUB6UBfJM/xgAAXICd5HZE6aDgNpQyYd5/FJwrE5KxafKHDmtapjBS4Oh87F6vsa7AGemIRL3o5cdGtbGhtcXJP/Nsp+hUhgn0JWW6BOq5n8/natlzhdDGX8PbRZL4bFd7hWK2MJuM/qOP8hyyu59UsBpC5CP115RXhlG27z9xh8cNQ9dD8I8qApJsV1YIZEE6MRe9nVVgpG6dNO7HZ7qVXjvjYp0iEChd8AXaRmC2+skRRB3i8cPizZL97yhvB1DUNssW92ezeZvxNFS+Cz5ZLaSd/ws5OULuGgOsS2pW5FyIjiZRw19Mz6iv+dDzdzMJZwWMxlsHRY8bHaRFz0Fmepu4iu7QH1u1b7CDqgf5CiSWtyRYPFgtpahz3+HH4sY+twWzAC4NkANyavktIi X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 51 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..b7a107e983b8 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index d56cdc547a06..b6a63095ae67 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1357,6 +1357,57 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } +/* + * Free up the maximum number of swap entries at once to limit the + * maximum kernel stack usage. + */ +#define SWAP_BATCH_NR (SWAPFILE_CLUSTER > 512 ? 512 : SWAPFILE_CLUSTER) + +/* + * Called after swapping in a large folio, batched free swap entries + * for this large folio, entry should be for the first subpage and + * its offset is aligned with nr_pages + */ +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int i, j; + struct swap_cluster_info *ci; + struct swap_info_struct *p; + unsigned int type = swp_type(entry); + unsigned long offset = swp_offset(entry); + int batch_nr, remain_nr; + DECLARE_BITMAP(usage, SWAP_BATCH_NR) = { 0 }; + + /* all swap entries are within a cluster for mTHP */ + VM_BUG_ON(offset % SWAPFILE_CLUSTER + nr_pages > SWAPFILE_CLUSTER); + + if (nr_pages == 1) { + swap_free(entry); + return; + } + + remain_nr = nr_pages; + p = _swap_info_get(entry); + if (p) { + for (i = 0; i < nr_pages; i += batch_nr) { + batch_nr = min_t(int, SWAP_BATCH_NR, remain_nr); + + ci = lock_cluster_or_swap_info(p, offset); + for (j = 0; j < batch_nr; j++) { + if (__swap_entry_free_locked(p, offset + i * SWAP_BATCH_NR + j, 1)) + __bitmap_set(usage, j, 1); + } + unlock_cluster_or_swap_info(p, ci); + + for_each_clear_bit(j, usage, batch_nr) + free_swap_slot(swp_entry(type, offset + i * SWAP_BATCH_NR + j)); + + bitmap_clear(usage, 0, SWAP_BATCH_NR); + remain_nr -= batch_nr; + } + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ From patchwork Tue Apr 2 07:32:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13613507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 234BFCD1284 for ; Tue, 2 Apr 2024 07:33:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D89F6B0092; Tue, 2 Apr 2024 03:33:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AF8C6B0095; Tue, 2 Apr 2024 03:33:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74FF96B0098; Tue, 2 Apr 2024 03:33:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 56C4B6B0092 for ; Tue, 2 Apr 2024 03:33:13 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E8EB140A19 for ; Tue, 2 Apr 2024 07:33:12 +0000 (UTC) X-FDA: 81963775824.17.7E50CC9 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf11.hostedemail.com (Postfix) with ESMTP id 1333E40004 for ; Tue, 2 Apr 2024 07:33:10 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SE+1BhVI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712043191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O6zOKBip9tcW2yKqv67ogMIPuHDVX+LrEKHPk0SxbO0=; b=xFfyQPC5y7L9rA1zDWy8BzAP0iTESOszAEOOmZMYgdfDtI+u/EFsfmB3I9P7lLm+OOkFgx EEniC8IiFHv4WA0AlLBe+/fMmEPa3YXwzDhkABK6Xce/LAkZVLp6LbIZ4I24atX1ElgJMD FTKG8WtoHqZEgrsma0ElN3RLshjUpjI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SE+1BhVI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712043191; a=rsa-sha256; cv=none; b=rtRWSBW4GfR0IwnLEYyyBSRDjFv6RGzKBmA5TJPu11DQMHVBwCv6qKcLuyJoZTcinsI5hO boKnvDkaFoUpOmCFyyk32/qs6m2glrGdKa4dWrTXNCK1U07q/n3VMiGp2VXNaox//L2CXx IaSZQAYyQgiDYFNhuzAEaLEV1+EPcDw= Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6eae2b57ff2so3689624b3a.2 for ; Tue, 02 Apr 2024 00:33:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712043190; x=1712647990; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O6zOKBip9tcW2yKqv67ogMIPuHDVX+LrEKHPk0SxbO0=; b=SE+1BhVIZGHHKynFwfBGuoumFxDk7BU2qWnVPVuBMdJtKxj1rc4NZnAoyMpnwQg4LM sfUSc9LFVY91h0FgrNV7OZRarf+u4ethw6NYF3BPeak8JDv6CjCVVSE9lzMYsvGkGKlB 2d46cEule/jwKRFWiAzQyhd656pga96fkFDf8ubUWuvt8ndgX3PZA9SuXvzKfrwOq9dT TAgsIJIe3PPVQWBMdVSJF6q2vCWO9r+QAkW91TBWt2iJPVa9x6BgX215p5cejcIJDNkU leEFbwHEqY8zDaIeKb73tWWfJIoU1xvMo0c8BKlZ5bGaJb2gNw329s1WG6waG3wCC2Y/ 7+/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712043190; x=1712647990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O6zOKBip9tcW2yKqv67ogMIPuHDVX+LrEKHPk0SxbO0=; b=Coijp1WKbalI8oBIOENyyluSdANnnyS7eqIZM+xcmpa6RcK7pHJn3boGoeemVNCgCl K5dwhV95hR3aup2dCLyPpyHIFPqY7EZ+mb+6GM9egsmF79C9+uMK7xZXRWO3F9WT/n2q rEsFpWP+5Gyei7AQhTzWSzp+rUpjddVulrXH8J6vsTdrc2KO6Pdi/RaWDrZfNpWkw8eD bRxeF+6IAC528GpF1iWxhuHRNx3i7CUnEump9DZc759EQWQxv/ESj6GIbSIY6XvPWFJy EHo44ct9zryW/9lYLdFu/bq9bL3XAAzpyBKgpfogPBhOrhdGrjaVssFx9FmB1+PGEfbV D7JQ== X-Forwarded-Encrypted: i=1; AJvYcCUbw6XlYi6PfKA3rE9AtlqXA/H2TCEUHYqr07+0KD9XnI0z/EH1gonLzkrnX2UcLqV9rbwfjHlPfR94WggJfOguSyM= X-Gm-Message-State: AOJu0YxBpwgLvlC1fn+Q8A4mvCCakyruZ325O42oGaponLIH9zCS+vhS yDGA33zMSDP6U7eEyrg1yKiLd+QTrarCpku7MIyxx9ORAMXK3fZy X-Google-Smtp-Source: AGHT+IFu8667qqHLR0wMKmYAhROFagzkwq+9TEq9aLVUMSzEu/u8+FjteDvnnmd7ONKMBz9GkxxvAA== X-Received: by 2002:a05:6a20:bf28:b0:1a3:e0b5:418 with SMTP id gc40-20020a056a20bf2800b001a3e0b50418mr9500313pzb.36.1712043189947; Tue, 02 Apr 2024 00:33:09 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id x15-20020a170902ec8f00b001dc9422891esm10187588plg.30.2024.04.02.00.33.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 00:33:09 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: david@redhat.com, willy@infradead.org, ryan.roberts@arm.com, yosryahmed@google.com, hughd@google.com, hannes@cmpxchg.org, surenb@google.com, xiang@kernel.org, yuzhao@google.com, ying.huang@intel.com, chrisl@kernel.org, kasong@tencent.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, hanchuanhua@oppo.com, Barry Song Subject: [PATCH 2/4] mm: swap: make should_try_to_free_swap() support large-folio Date: Tue, 2 Apr 2024 20:32:35 +1300 Message-Id: <20240402073237.240995-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240402073237.240995-1-21cnbao@gmail.com> References: <20240402073237.240995-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1333E40004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: prjjs7ot6qpwejbueu1ch6hkfkdf9zdk X-HE-Tag: 1712043190-243257 X-HE-Meta: U2FsdGVkX18yN7kSi7cCPMpbeq3MPPq7yAheSGNxUTmToLnzgl74t8g3REdeoi6HD9iZqgb2pAxFVwbSJ9SQ2TWBbG8uXW7yLroSM5uWOvXocYU5e22qswZu66t4FnGmzTWTeZ8eL4aNwunv33M4/pxga1e6A5EfNyCcnZXvyvwC0AvZlmSspsm7st7rSWSLHYABht2Wvv3D/gxYKgDhcrYh4IhGtoOu3vAZlVCA0WEDJNBJnlPgriNobWeLzHXFqqZ4lU69K+9P6Q3pbzm1XZsM6dg22yfyfXaMFhkC6muaZO2zySMYhXFNaRHCcpt/il5NLwcItnrtLjbFDH/U8gN/IArfAti0j+EBLtEB+MrwibmqQi1KZXO9pl39/VSQAaxhO8TcypmdIyFdrXNGAkIzzOXqzUQbRNljtxlgWemcAAKhJSNpsEVi6jKMQWxNKPwkAPbxOCSa7POLULzPLxmi0HotiBjA3fp1H1nBhjQMy5aJM7OQHfoh5wOJmoNkIOs2l2ggW8EpaMKxCU5aak8LB0xlQPUlthT8Gcj344UPqOzBBXb667b3LPYwo9zppKm7Mr8cqWthMDDR2bSFx4q9XyHMbz+u80ltwEC1OPEcKsIjYP8gxup5jZ1tQi6v1CFMtOqumdaVLQKYEKF/dFx9kStqF7h1n9aeDZIslWrY9KIBG/pUvGp5Ag/BrLIfNDIlxTEg07RbJRFVb0UPFp6pku9QumIoynmFAASCLEdgMV/1OkCIBHzmZPw9s9CwRbUt8sOAwAUsTTkizYwUhXS4Wd9fsCgUjylbwoH0G/oOq3/hyJ5+rwX4R/Y8oZXsCUIHTP3BgxlVwRk0pUiZzOZ7vPSxslZAnDMkYbCmTo5ns6ZyC311YTLmhPZvmkiHB7e/+GwsGjlE2s+4QvJ24PcVDoF7uEitMy3cdjzmUUD4GV+GMR4a5XwdvXdTJAmyWIUVQwcBbPSm6zXPNi+ 1uoSbL94 2ELMfV5cHs46ZzNXQuUFOq4WDm5gkD65ZUSlidaXDp9k2TWsERMbGfRirI+ZeTl1sLJIzA3oF0KKJQ72UNliioSJJzhHE36qINy4kSGzNBRh6vfb1xCps5avwdjmQvVrdh5CHY3AosYuW6SOG1Mc39IiV/jmjEA/V4P9WI1bRo58GieEPCk+LtqeTukpGdFzUlukyhjlvWq2jEM0cenyxV5TLLszfOxN6Lgi7SH/i40sew7sJsE5SwMfPICLKSAvx0Rrd2NWiFbkCZFh78xKn6Len3gYrY1we2SoJjS7N/qOw62+yHqE7WIARIZqZWuyh6pY8aA06oumAiHS6rqzV6X4afYutBTGvhHxq2+sWLJawz80nWAasr7EQki2rh2+GqohAjg0ipBEKUgJvwKfK7OCJypybilJPp2ur6HCY8dgmC/DVjxh911mrRwXMmlJPjgc6OuMsYs1JzTYOGEzN1vbukYS6t58lIrZM+PPsbJ9GuzW/lv9vouyFauIye0BSHBxafXmEHi7SQbhIc+oAC6k8o3LsIqdKMyyMIlwMo5tFwGeim6fLmUeC//jtIKf3doZ/9o6dQfL24yGjqKPKCbhPe54b/bC399cDCXNMm4/JjJK5R1g0X9Dss11ahWpiYCdsyxedtCIFk92uHhaERQ/XrKimDBjVRcI5AL9ikyMRzfvttEYhaygA+MbD5HaC4ht+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages = 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 010e7bb20d2b..f6377cc4c1ca 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3850,7 +3850,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) == 2; + folio_ref_count(folio) == (1 + folio_nr_pages(folio)); } static vm_fault_t pte_marker_clear(struct vm_fault *vmf) From patchwork Tue Apr 2 07:32:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13613508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3BAECD1293 for ; Tue, 2 Apr 2024 07:33:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 446A46B0098; Tue, 2 Apr 2024 03:33:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F4866B0099; Tue, 2 Apr 2024 03:33:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 295646B009A; Tue, 2 Apr 2024 03:33:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0B6FF6B0098 for ; Tue, 2 Apr 2024 03:33:20 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8EF3AC04C8 for ; Tue, 2 Apr 2024 07:33:19 +0000 (UTC) X-FDA: 81963776118.25.C68D400 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf08.hostedemail.com (Postfix) with ESMTP id AD895160018 for ; Tue, 2 Apr 2024 07:33:17 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="jOdV/n6S"; spf=pass (imf08.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712043197; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VVdqk/euRe/6YNu7J/p5kBNan1T8d3Q5WxOyEpgBot8=; b=UJc9zIm4/Cp+a7iHssqW9Ya4GCjoeGl4fIBinAcXwTiigHW+fYmVDpvexo3EorRqZNtknf Yws6WOX8Hl0SKY5Ylt17no1xKMcRuY9SwM5cds9vzMtnep/6PysnUzQFWXlqrvIp3sQDi8 bsEPNcmUqLQKxUkU82kvw2QOTTzosjc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712043197; a=rsa-sha256; cv=none; b=HWBuBWIdzQW8JSn9hQ/8xe4OjJ0u6crbdQsaCUv0JtCFd0oxzn+CD+bwZNG4zrFH16X1kh RqnyYo7tp1KPLxxqV6C6SL4wQSPjaxwsiWjmX6Mt0KvUgrSnQSQYOAbo4Iwd171VSjSXan gWI71ubL5/8EgFMe/knPAUHOJELuQmM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="jOdV/n6S"; spf=pass (imf08.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6e6b22af648so4122400b3a.0 for ; Tue, 02 Apr 2024 00:33:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712043196; x=1712647996; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VVdqk/euRe/6YNu7J/p5kBNan1T8d3Q5WxOyEpgBot8=; b=jOdV/n6SBnd3YksFI/wnBHcolU7RiDnnBgkroE4FaKctqSczCdx/8kLVbLaqBcUFIj 051bvwnvqBNiVnHCA70JVS0icfKVkx+nyduCbD8WFZUfXUhRfxqWNVEHwKgltmfczNql z5A4v0yB6aZzmeJRMuBouXZ2X0Oh0t7iCnIj6twZEkDV8UczRiUGikB9xSxJnu0zPf9P wtyi6tyA+dlNqJO1MJgbhyqvFeVOR7zcQjei9JQeOF1WAJ9M8UTdGPJ9GJ2PLM1duSiZ 1NqTtlt/KKg1w1SMlNvceq0f5euRqDM1xeEYrkvrcqWYNPVbwhUR8G4cIFpN/ruqksUf IgAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712043196; x=1712647996; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VVdqk/euRe/6YNu7J/p5kBNan1T8d3Q5WxOyEpgBot8=; b=f4lSH7oxxJA5zu+lXwydn8eHQSft++xMgeigwLxr6iSN3fS1MTaxFcGT0UrlwqeUWs owlvOi+VFRhbxdlN42T5gJy1lGg2ZIAuDLgC79Bwyx44bZZsvmO30lxZ4H2WxGM1Sxg+ 1Ov12VPgO0MHZk2Nw66sCWyxWhPthlUZnN70Y3tOVW9xYqXY9E79EYEeq9Q/uF43zeH7 ZAAtWKHesnKiqV0ZhjpsATW/fMTOMpdJBd1c3WvdHVktFgajZDq4mNEF0zKGjBC3NVMB QlyNX3P0Egn1MdE4We/ujlg2T+r+rCdpXgEUFtSkFJGoUxyQ0Uc8KxNbAK6fAn/A6Usm UIEg== X-Forwarded-Encrypted: i=1; AJvYcCVPKn73ThjpDG4Gel3HDYKNsJ0MLcVCzaPiIKC3vFxnVfR8Bze4FyYSDS7qOtMxUp8F9bTAmc/c5aiyJ9IzupdyG0A= X-Gm-Message-State: AOJu0Yy1INiPRAVLKt/F3cPQx4hNJ38WL8WzecL1U+AXkzc/Ou7e5IuF XSlXOVui9aQjVHvmdcWU78Qy2irk45B36Q07L3Us6g+eFY/2RCJb X-Google-Smtp-Source: AGHT+IHjqcwqXBgyea3FcZfcTEPqa/n4oleP8gsVIrG7pfCassAxs0TZG0eP0GNwLl26jpY0q6OzTQ== X-Received: by 2002:a17:903:22cf:b0:1e0:73d:9172 with SMTP id y15-20020a17090322cf00b001e0073d9172mr22662352plg.23.1712043196476; Tue, 02 Apr 2024 00:33:16 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id x15-20020a170902ec8f00b001dc9422891esm10187588plg.30.2024.04.02.00.33.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 00:33:16 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: david@redhat.com, willy@infradead.org, ryan.roberts@arm.com, yosryahmed@google.com, hughd@google.com, hannes@cmpxchg.org, surenb@google.com, xiang@kernel.org, yuzhao@google.com, ying.huang@intel.com, chrisl@kernel.org, kasong@tencent.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, hanchuanhua@oppo.com, Barry Song Subject: [PATCH 3/4] mm: swap_pte_batch: add an output argument to reture if all swap entries are exclusive Date: Tue, 2 Apr 2024 20:32:36 +1300 Message-Id: <20240402073237.240995-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240402073237.240995-1-21cnbao@gmail.com> References: <20240402073237.240995-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AD895160018 X-Rspam-User: X-Stat-Signature: zhiww9p9ywep9bsrjfs1dstqmqtcd7z6 X-Rspamd-Server: rspam03 X-HE-Tag: 1712043197-514031 X-HE-Meta: U2FsdGVkX1/Dk+bH1LVKMRo6RlTbXGezrngRCUeDjJo9u4BRpltTYomBp6xjP60dlL4dkEcnYJ+9Hefe/edEFUGAdii6ZFiuTxAP1IdsPcAUSd5UTxZxavcTEb1yHxF+CQ5r9vJnHSPt8CG89vyXpKGf9IHdyw6CZjIfoxNX4z867SMMqyl1WJA5HVBS9CtLqihqohYPc/9dWSobMHPKo9Ki2Pylb6HOO0V5I5Pf0A+9FOlzJiQkQli8K2kO5mxWW/FhIj0skTekcjkT/Hw8i0ujlSmmjhI2WIbbbFeeqKwrtG9SKqDH5Ona8ymT1jYUFvkXmrjhALiMjPtBo+pxijUs0jQvbtNXC2TdwYQlKLhhQOBEhJlQoxHzgYMPcNHhnWhTNyGv9mixq0c+7yG5HLK96c7wm0hNAkzeh97UN/OEOOqSozYRFLQ91DTLv91EprfoAvOdIijgTPbOgr+ix4r46ffMnnIX2qi6yb890C6fUDOBvhaN6qLZ3Qsx2KhXLJh0aAc7uAR/vb1TRJFj26ab5Mx3R9OFqFpsAm+uqYQV9lyGyu8VDAC+kdkfGgiXbKMVmePUfewKcmIShLp4wEiw+jwAvw0RGqk82hpxRshGLCmXIBZ3C2npTxebm8edCx7ZBezLtWjBI5eH3UXbAjauI/E7xRtnj6dyLG6uDsMI/lkTNmvTb4oBUm0PYVGvaJs764SYL31z5E9pZcCW2eOYbRuYnmBZjZqxTf3H573GZ98UXo/j8YHMr4ylXOLlObAfVxwO5nJw9iG8gdVgdIVSngc4/IeglK97VtwTGnLqgkUOLWn0FZ1BzTykV2vkH4toNAkwPyz5nQbswXtzsAc6HR3GQdtl0hJSbjo3YUSBbxbccGnqVZ+m4n4LNroRtpmqHe4s366J4bxKC/h093w1/joP+qmisvncxZeRUa+nihnLqa+gEEHtPU2AsObIwCYQ53eRk8oQf5T4dWC tlRPJiYz DRITTObTNlqYvNMg21/IbXqbSRBj7AISE4db8g0C+SVUe27REk5hWCkrp4KCdBfqU7oU58I2xjEN1Mim+2wEPAQHlEdR/d9p4/SFyFH3xoPlBC4rP+CQ51ghniBDemfgAjBIKjifLAMez9hrYPETUFeoxv2SQFc5W1jdQqdmNpSRmwP/jUKCpEtrMjAXGpjUDw3N22OFxJ+pGgWvJ8i3IRwckCG/T8K3umzvmplm7xdOUcccWATELfcOf1dF1UDRqWHM7+xl/VfHCSczKgB/AMTWKjHFQ7k1Xajo3/29fK0idymCvMLSNXCr0L7trNKF1EY2OjW5M7/c63Mp5EjHqZEIiZ72WdtP2ilwkj2qjy2x1PZycSzUVdDpus2s9JW2BDhodzk6mMR5IB866FwBMdzYwNmoFJLsUemdOED1HxBfivZCuCkjLHqzG1acGxCAjPZBnfLxcIRPio36Q3RJgOli3oiJhtZR6TQaXPaivQbgn6liZFDUHAjDbUZINOVqcGSeWzFczrqNkNO+hBbPdqPptvG7CXiJYde3OA4WZrcghPOTqQECrp9fWESjDRnmVhwqT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Add a boolean argument named any_shared. If any of the swap entries are non-exclusive, set any_shared to true. The function do_swap_page() can then utilize this information to determine whether the entire large folio can be reused. Signed-off-by: Barry Song --- mm/internal.h | 5 ++++- mm/madvise.c | 2 +- mm/memory.c | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 9512de7398d5..ffdd1b049c77 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -216,7 +216,7 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, * Return: the number of table entries in the batch. */ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, - swp_entry_t entry) + swp_entry_t entry, bool *any_shared) { const pte_t *end_ptep = start_ptep + max_nr; unsigned long expected_offset = swp_offset(entry) + 1; @@ -239,6 +239,9 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, swp_offset(entry) != expected_offset) break; + if (any_shared) + *any_shared |= !pte_swp_exclusive(pte); + expected_offset++; ptep++; } diff --git a/mm/madvise.c b/mm/madvise.c index bd00b83e7c50..d4624fb92665 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -672,7 +672,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, entry = pte_to_swp_entry(ptent); if (!non_swap_entry(entry)) { max_nr = (end - addr) / PAGE_SIZE; - nr = swap_pte_batch(pte, max_nr, entry); + nr = swap_pte_batch(pte, max_nr, entry, NULL); nr_swap -= nr; free_swap_and_cache_nr(entry, nr); clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); diff --git a/mm/memory.c b/mm/memory.c index f6377cc4c1ca..0a80e75af22c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1632,7 +1632,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, folio_put(folio); } else if (!non_swap_entry(entry)) { max_nr = (end - addr) / PAGE_SIZE; - nr = swap_pte_batch(pte, max_nr, entry); + nr = swap_pte_batch(pte, max_nr, entry, NULL); /* Genuine swap entries, hence a private anon pages */ if (!should_zap_cows(details)) continue; From patchwork Tue Apr 2 07:32:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13613509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4D8CC6FD1F for ; Tue, 2 Apr 2024 07:33:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C6A66B009A; Tue, 2 Apr 2024 03:33:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 777476B009B; Tue, 2 Apr 2024 03:33:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 617A06B009C; Tue, 2 Apr 2024 03:33:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4136F6B009A for ; Tue, 2 Apr 2024 03:33:26 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 09477A059D for ; Tue, 2 Apr 2024 07:33:26 +0000 (UTC) X-FDA: 81963776412.30.9363342 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf21.hostedemail.com (Postfix) with ESMTP id 2F6171C000C for ; Tue, 2 Apr 2024 07:33:23 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HDiojobp; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712043204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YC/Kgduuu0t6Vvbwsn8PXX7BTHypb3qdZsKV5RTpqdo=; b=XYMXCZD2eqgHjGrYrDCSS/JGLcXYDgrfuyiHsM42mjs3I9HBN+aL9Ni6JSh3OQyvD+uQwX 66xspGYLLlqrXbKppjeXh9MdYSGxX52kjA0zLCjfnx99J//bj9VpWmsYf8TSDHnX6K6won 152o2FFVbCQPYHjO7XTNN8AQ/sqLOBE= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HDiojobp; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712043204; a=rsa-sha256; cv=none; b=THyadWc6mfanr/bUIwBgTvseXScZANYBH7MCFgFaEq3KkxAkmBhS5ofzKOuw2JT+mcopub LyOFRjOuh3ocuRkyE/lYu1oK+LiPL+jqOyPapymR1jGeEc71BqnwZNp1/RTNqB8CpS7AaL QflJmkqidNrkxaDI4nGLha2qtWTBh2c= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1e0f3052145so45012505ad.2 for ; Tue, 02 Apr 2024 00:33:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712043203; x=1712648003; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YC/Kgduuu0t6Vvbwsn8PXX7BTHypb3qdZsKV5RTpqdo=; b=HDiojobpnLx3AzHXVTzpUR/FNRW6f8B9GOlwZPnEZ80MZkiriCZV/lB7YaVmQryu7m 3Nz42ILk2j+gJlXSBNlYeGxl6xs42PJCIF2hWLhHxDK8xb6/WqwcsGIYIwkaGBwvg+2R KKn7DzCESkWBI8nk7fktCWfbTIL7fj7O/a+Oz8D+GTe+T05ss13ON8TbNJZUJy/OHn2F SyVzE1xVCCDGX36A1zXjX88nfNhGcJRtHdaud0WbPKZeF3LW5wOW9dOgHlphZDI2MaK9 zfnt6kqtRtBvkVTwmCan3scVM6/iBpvnfFQkliSXNoz1RNibNaET541evgYZb2nXYfPU 7Hqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712043203; x=1712648003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YC/Kgduuu0t6Vvbwsn8PXX7BTHypb3qdZsKV5RTpqdo=; b=Nk2GMmT0gamSiT04WJk6/v9eoWvQX5OHOfs6cBy64WZMjbCAKFyLxtX59QS7yMKuwi 38oaUYBGRQtvfOZZAdwpiaAULawiVOlHlzt43kflnQKBY/CfseQYoFuwtPmT948ysN0H cT8wgfg+RpY+LT67UbRvDfoaoEdcfD5sFVSI4QtbZIfynMRVIvi6BurN5zZEKyE5tDmU u37ta8WINi4RMvuuPZMjIGWfN5f5GrBl5IJx+MiwV6cmeI/Tykl3CIsvx2ZsYu2qkm9m cJTFG/6+P4R0eMTm8pUlDX94oaz3uREthWlwchHPWXegRBUVTxhtsj0me0/p/JjBEuTU 7Z6Q== X-Forwarded-Encrypted: i=1; AJvYcCX/38nzwg3WpIgnh7RNGKUSxFmli6nq4SV/Qayw4dweuTXsPZ+ESPGcQeLWAD1g0fXyeGEfE4YiVKkOsMFELgLMjA4= X-Gm-Message-State: AOJu0Yx4dBk94zFePXUMdgtaOzYg1Xg2v/3jbXcE3JnGmwe8LMYfskiN VlE1lMo9VKN7dKzu2weK3u3Hkz21blYnCAB30qkQDEhsTuPISb4K X-Google-Smtp-Source: AGHT+IFTa4UbxkTyNq2/JmkPohFzu+gnIk87mqQPgulDZl37xJQ6GRcUGutEVaqVEIW/xbzp7Sw1FA== X-Received: by 2002:a17:902:b20f:b0:1de:f7cf:471 with SMTP id t15-20020a170902b20f00b001def7cf0471mr11834234plr.69.1712043203122; Tue, 02 Apr 2024 00:33:23 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id x15-20020a170902ec8f00b001dc9422891esm10187588plg.30.2024.04.02.00.33.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 00:33:22 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: david@redhat.com, willy@infradead.org, ryan.roberts@arm.com, yosryahmed@google.com, hughd@google.com, hannes@cmpxchg.org, surenb@google.com, xiang@kernel.org, yuzhao@google.com, ying.huang@intel.com, chrisl@kernel.org, kasong@tencent.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, hanchuanhua@oppo.com, Barry Song Subject: [PATCH 4/4] mm: swap: entirely map large folios found in swapcache Date: Tue, 2 Apr 2024 20:32:37 +1300 Message-Id: <20240402073237.240995-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240402073237.240995-1-21cnbao@gmail.com> References: <20240402073237.240995-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2F6171C000C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ibpfwm7i44y371a71bdkibuub8ahb7ch X-HE-Tag: 1712043203-428739 X-HE-Meta: U2FsdGVkX1+TwqP8AOki6tOKKbbPyVI9zFqqwBj6xYsc2GgWfbQO6dGYD8zAj+41aslGnjeVDO/qcSaBD/Zl+LP0hVbDnT609Uice3h0bPTsQyD7znAO6LVGhJNQlekP4WoIKMkltL9IYOlJ16DpnDKlg2pYTGLJyN7cRk8mv2l59l6+cOkBvBnFScii6rMRK/dn8ux9q7vn0yhqEVnEWwv0Nkl+0ovzQCAb3RZu6ENMphrsvHrd0n1K6dFKkzLNA+YiIy9rHIgmhMlwkrsO07jA98OtFfIYw8zalKWxzkZw+M4YR3zqbN64zpWnc7MfKTDrVkiN1+KjlvP5voJ8zEH8T77/W8Odb4bTdvtKblaf2O24D+MbwRNzv7kYBsyM1cSFLcoHW4+/c/V5GO/8WvHR2y8ualQiBvS0juRO7PzH5DTOlT2+FrEihs7g3xYqIzyceCwuCMRkKREtrA1i+2WRi3K10bwfRFy0x9s2KIlPR2EmpD/I8m3L6tU5lgajph+FnfaEU6aI2srDCOKJRI0bw34pYt6fe1Wb3XTWOjs8cB6YlZXYWKhEJbhL4soh07QJoGG8y9oMM5v7V5xEzuJGacXW23b4ZWYfjym04iQBAV02nhrnHBLi13DN9MtMT/qjpZoYOOkKv9cDDJ9qZ/7e0STKI1aW8dlgBKMrv3LtHOaz9AUDqmuOSGTcfZbj4B49kYigqZAZaNtZ/KMFL8OMOnwaILBWNCxtC/L8OYCW7K4bAVhIDy1oGU5EgB4wCL32xSDfP8Em9jO55JFIWiC5SJJLBegfzUnVo049V8RvqO8oaBK4qerHyD03Z8DmkCBT4SNQwJ3EW02yi3ousDfnDhWeWWCQxmWhgdTZdhepjsc+y0kqU3HCllWj3MsXe7d3CE/hE7rTzqXwzbtD4dMgMMZFq2g7H+Q6ckOOq13t50QeM6HrLQDIv0txlMbvOefgd1H0cFVoMJwKuY4 zan6g7+E C8TBj1vvkchd4HNX43OHvB6LPKG6TfDr+3K7cy+q1vOtIFi8dEMf1mbH9AWybqwSf3Rt21xmKD8NIbOWEyXxfRr3LL66Dl4BATrU6lsShJ1Ugh+07sxrb7UJKYVn2AYVNuI9TBwcZKvT93Aw6778wvtqiORzYkZaQ8Jf1GgNph5zp9rKSamDuW0eyCnmEhj4gmsf/5jwhunZOqrRE2DZH0cgposLPM8AUeyk463ghu/nXCYEMjN8xvHQeiF/jXmVMc1AL/iUXKgLvnc/yNJYAf2OOdofFs9AYuMDwaXb94FW7nWIaSMAq+CggBQ9mCEtyqHEa8ofp776e9GRaI1HxFTtDX9JLpa2R1ACTwwyVmUq8jtQIkqcsFw0x4x8WxBZaj2UR/Qi9YWMv7nMpwsQIbRubij/BUM2PrqnUZ6QkpN0f8dxm+W2oblWE96JU1rwsxFrsrb6gDQ4FJ7C+XZDmXOq+rJ5n9ORpHNi9O1FL1X9BhhZuWJmUtz3z5XsZkxYn91JhjGo5il4Ih8H476awKRj06FHN35zOiszemgU9o7bsCZ58NWNlSfvoew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- mm/memory.c | 61 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 49 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0a80e75af22c..5f52db6eb494 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3941,6 +3941,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages = 1; + unsigned long start_address = vmf->address; + pte_t *start_pte = vmf->pte; + bool any_swap_shared = false; if (!pte_unmap_same(vmf)) goto out; @@ -4131,6 +4135,30 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + /* We hit large folios in swapcache */ + if (start_pte && folio_test_large(folio) && folio_test_swapcache(folio)) { + unsigned long folio_start = vmf->address - folio_page_idx(folio, page) * PAGE_SIZE; + unsigned long folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE; + pte_t *folio_pte = vmf->pte - folio_page_idx(folio, page); + int nr = folio_nr_pages(folio); + + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) + goto check_pte; + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) + goto check_pte; + + if (swap_pte_batch(folio_pte, nr, folio->swap, &any_swap_shared) != nr) + goto check_pte; + + start_address = folio_start; + start_pte = folio_pte; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_pte: if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; @@ -4184,6 +4212,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ exclusive = false; } + + /* Reuse the whole large folio iff all entries are exclusive */ + if (nr_pages > 1 && any_swap_shared) + exclusive = false; } /* @@ -4198,12 +4230,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); + pte = mk_pte(page, vma->vm_page_prot); /* @@ -4213,33 +4247,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) == 1)) { + (exclusive || (folio_ref_count(folio) == nr_pages && + folio_nr_pages(folio) == nr_pages))) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, start_address); folio_add_lru_vma(folio, vma); + } else if (!folio_test_anon(folio)) { + folio_add_new_anon_rmap(folio, vma, start_address); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, - rmap_flags); + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, start_address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); + vmf->orig_pte = ptep_get(vmf->pte); + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, pte); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4263,7 +4300,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl);