From patchwork Tue Jan 2 17:53:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54361C47074 for ; Tue, 2 Jan 2024 17:54:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D942C6B0152; Tue, 2 Jan 2024 12:54:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D427D6B0153; Tue, 2 Jan 2024 12:54:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBC146B0155; Tue, 2 Jan 2024 12:54:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A7C376B0152 for ; Tue, 2 Jan 2024 12:54:20 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 85776A18C6 for ; Tue, 2 Jan 2024 17:54:20 +0000 (UTC) X-FDA: 81635120280.01.766EC53 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf04.hostedemail.com (Postfix) with ESMTP id 9CB0540021 for ; Tue, 2 Jan 2024 17:54:18 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OuRIbcsN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218058; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wKrEvGkZ8GQyBX+qIUqWP+XNbS2tofN+iPdO+E071Zk=; b=LVlOvDi3DqG2U3/fPDEbuCIuwApcanyU42jrbGWePWKix1BD1hRZIfBq/lZIvNVe4xt60c TT8pFuxuc+plwzQGkSmMsVmng+2ZDV57/14/yHuNpi2HIcsCpBvWXgfHRqp7eS6e41cQkv dYm0bA9CRvT2E/CuYW48RIINGlv6w7I= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OuRIbcsN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218058; a=rsa-sha256; cv=none; b=6RgOMl13l77FM2714EpYHLKyDvyhOERwZZG9IEcWhunEPTvHL1Dxtnvp6fAFw2iX24gRG1 k+TjQRmhcM45zUHpmk5d2EQXBLer7u6t5l8jR7WBVBOEYeLTfecpwcUjiyTlxjTHWfCM8z Otv98nqqpdSWBj2SBbhQfdTTbLvzrPw= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1d2e6e14865so44973735ad.0 for ; Tue, 02 Jan 2024 09:54:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218057; x=1704822857; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=wKrEvGkZ8GQyBX+qIUqWP+XNbS2tofN+iPdO+E071Zk=; b=OuRIbcsN0+uyt6Zi+T28b6jyAzTU9qrBLo8OchfCNpsNE95WWWOOfG/crm0y2LONBP 6uSWHnuwFIQzNMTBVv0Ml+BYRM8+3NwoTAqdhl0RT4Zk4I8X6+zenL25tR2M4XeKgy3s ecQ7pyuiw0zcnJ8D21bVijMT9PpzPfESjRGLSbhIpKG9Vkwa5UKU7E/417WjPXWYHjwE +xPSQs/Sq7G/CiO6HGpRZDPX/kwLZ8/y8GRvNDAT/JtIxoKgmNV2Ccg8xmXAqdP4yVqn hIMWkAonV02RgXtEeEqf9imY9tNHOMsUYUk42u9z7BOe30J9WEQj3cBEp/73FFTJBjJl gJxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218057; x=1704822857; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=wKrEvGkZ8GQyBX+qIUqWP+XNbS2tofN+iPdO+E071Zk=; b=d5peSDLxEKJyVa82bQv4FXuagH0V1q0D1A5IIL9Nt0ZmPf+B7vKS/lyBTxV7EUD6f3 UpubyObVCh52cklMvvEHWW9I93zLjXJLlyMo0Mfg+Wciz3wq5+roIAfvFtixAxIc+EeD i12HzKQSHqkzWyRjdPEG/Fdhxw76YHWtS/2CQJnDZR3oFC96kdGh5TKiCBH2dyitmmee Y9dCvKdRtBa1v9QvxIhbMpVMcvXacXNXibjljAtkbe5mt3qkSDcG3eoQRj0VFoGCgT4R LXwGXpc1Qi2FVZIoJzr0S/10BdrlkRyodzneY/DpqiRdd4emQDRI41Waw+ySOdRSnWBz n2fg== X-Gm-Message-State: AOJu0Yz7UZXG7tZeIOggDcsjPOEoIU89BirjkfiytOwtVvoRfV5N5J93 TDQ2nUwbmdbewOqUzxksmZwUqmbwM5F0VBFe X-Google-Smtp-Source: AGHT+IEdREnUFQzLQLXziwcbdTCFJIEnWtbesYR/Yi2j7ybSJZObtayDa/HB7ljxXyH3M9oPbUbcVA== X-Received: by 2002:a17:902:e88e:b0:1d4:cdcf:97e8 with SMTP id w14-20020a170902e88e00b001d4cdcf97e8mr238233plg.126.1704218056928; Tue, 02 Jan 2024 09:54:16 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:16 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 7/9] mm/swap: avoid a duplicated swap cache lookup for SWP_SYNCHRONOUS_IO Date: Wed, 3 Jan 2024 01:53:36 +0800 Message-ID: <20240102175338.62012-8-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9CB0540021 X-Stat-Signature: 4g3mqfmzqkcxbbu99s4gtr9i47d3zcgy X-Rspam-User: X-HE-Tag: 1704218058-804457 X-HE-Meta: U2FsdGVkX1+jUlIBWRVzpOuNHcYi6Mgngb/nKAgKKXggd+vaPLvQtTMP546yu2ver5U0ekPS9v/dh+lToCOmj3hWw/tlZYmzlc2PO2vOz0UCR2g53mSsxMdsMOmv8xByHW016AlYwF78BOQnmn7dM3x90Tjo1smcPm+5tN4IoEtUN2EPgSJISmmLhEKfH/qBpnQc5d8i/oJs9vjgBEm3U/wTLCUcwzj8SpxxOJO+a0C/0v9oA7jDD279Pw70YiiHRfZYCkX2P6pKNUJsKDXCM7av5+mideFT2PLQ3MILr5b7XEyO8/c9ecSUTe/3QsIaCBS3D0Ac8EkW6FfEBju0Xts/CQEZ+NqIf6FUHHOe6cg/eKSuXO3h9PbzN4uuHDM9jAMDLBQ36J32QMQq1Pnp7Q0MG4ffS3G+fw+xhzc+9EaUYLiRjM0EA/Gy17D0kOSzpU2yy68JYo2beC5N1Mqf9jK5dkKz/vglG2AQwatmFFzyZaNxhFYWS18LgPmXo8n7gXKF2AbwcMoKQt8eo2R4jwl8FnYET3k9mos/xqJvqtB7cJIv6ExTqDeSN8KRs5ZMLoeRcSDP/S+mfTSqU3769Bdsby29ol+GpEJ5XX07gKBtrlxJE7G3CiE/eIoxTmYmLM+/8a5TrKhvT9gk1L7bhTZELTwqs4ePFX23j9oJGwVD2xCmX39m9DflOUon6F4T+CFpqJsvjFYd9LnNWBYHDmvhnulkrrzXI+qJoPJUeWv4ucF2+PBAU/+FvxudUsBpV9UmdItPr47YT9aHUwgqlP9OW1PG9V59uXLPwweqPNaKI8hazSxGicAOMWlnlTr33ml1jgCAKPU7YvcFa10s9q6CReVOhDnbXyJ1hnQ9GCPeQi9k7BMHizD/wX0Whd5NspPCXVGGa3wpGuxiQK7RbS3Jzon658y3k2AzgOdkZYpietoD6SvILDb25xjGKZeYRZTshnM4MZoLiTnpxNe 7jm+pDC7 YB643/qCCxBhKVm5JzSdt0+nNv4Qq6/JjEEsLUFcbXhc8Grp7d8E47mZD8yJjiZw+/FuHhoonSoG1O0U5dqE2M1kxtRNdhyI0Dtjhmn3sf608U5GDLMpAZciIgOj9BGCfCksHTQ6tkJKgy1FN6HaocNHieAd7r22yKSQ+oJJ6QQmI0aOrXSsRpmHd0Zd3wuYXAD0VbUg1qIxnh1mT1lUud4WjkM8y3ynXlCc//iic3pJ3vg8WHvP+J1kori4F/ndT3Vw0wxhj6dLLakZqXe89ZE7tQtyLNnv0VRJbi4seLrgDEtgyIQ1xgsGf5tRWTsDcp0VfHKiJ2oEXrPiRvgCSMbjxmKEHOu8hf/5KkEGUfEm5RcxEn87F0m1SjtkpayL8yY/j/dnHOZjqx/xoeA6ydPojMlhAaqzoxsZHN6SYnLPIhfZnzYTZluLKflUZeyNRlV9Z0iyrKAfav7jo27ixJZMhkF4CT5HYL9M6sZiVjcPr6krHcHkEiwUFZTofD8Xul9h8bHM0Hy4E/ecp418qmJTl8HfGMVBaCUquoSB9xU/FrFb8nrSxQiXnkz/TY77TchiwCmODHh58GUb/mXzac9gyDdMB6PNd6QuxdYk5ZdvPYtMKHFlxq+uzKnPZlayJZbU52qGpF1PqBbCH84/BZHOgqgSVS5Vutybt6OekkvgrKOs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When a xa_value is returned by the cache lookup, keep it to be used later for workingset refault check instead of doing the looking up again in swapin_no_readahead. This does have a side effect of making swapoff also triggers workingset check, but should be fine since swapoff does affect the workload in many ways already. After this commit, swappin is about 4% faster for ZRAM, micro benchmark result which use madvise to swap out 10G zero-filled data to ZRAM then read them in: Before: 11143285 us After: 10692644 us (+4.1%) Signed-off-by: Kairui Song --- mm/shmem.c | 2 +- mm/swap.h | 3 ++- mm/swap_state.c | 24 +++++++++++++----------- 3 files changed, 16 insertions(+), 13 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 928aa2304932..9da9f7a0e620 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1872,7 +1872,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, } /* Look it up and read it in.. */ - folio = swap_cache_get_folio(swap, NULL, 0); + folio = swap_cache_get_folio(swap, NULL, 0, NULL); if (!folio) { /* Or update major stats only when swapin succeeds?? */ if (fault_type) { diff --git a/mm/swap.h b/mm/swap.h index 1f4cdb324bf0..9180411afcfe 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -58,7 +58,8 @@ void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr); + struct vm_area_struct *vma, unsigned long addr, + void **shadowp); struct folio *filemap_get_incore_folio(struct address_space *mapping, pgoff_t index); diff --git a/mm/swap_state.c b/mm/swap_state.c index f6f1e6f5d782..21badd4f0fc7 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -335,12 +335,18 @@ static inline bool swap_use_vma_readahead(void) * Caller must lock the swap device or hold a reference to keep it valid. */ struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr) + struct vm_area_struct *vma, unsigned long addr, void **shadowp) { struct folio *folio; - folio = filemap_get_folio(swap_address_space(entry), swp_offset(entry)); - if (!IS_ERR(folio)) { + folio = filemap_get_entry(swap_address_space(entry), swp_offset(entry)); + if (xa_is_value(folio)) { + if (shadowp) + *shadowp = folio; + return NULL; + } + + if (folio) { bool vma_ra = swap_use_vma_readahead(); bool readahead; @@ -370,8 +376,6 @@ struct folio *swap_cache_get_folio(swp_entry_t entry, if (!vma || !vma_ra) atomic_inc(&swapin_readahead_hits); } - } else { - folio = NULL; } return folio; @@ -876,11 +880,10 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * in. */ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) + struct vm_fault *vmf, void *shadow) { struct vm_area_struct *vma = vmf->vma; struct folio *folio; - void *shadow = NULL; /* skip swapcache */ folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, @@ -897,7 +900,6 @@ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, mem_cgroup_swapin_uncharge_swap(entry); - shadow = get_shadow_from_swap_cache(entry); if (shadow) workingset_refault(folio, shadow); @@ -931,17 +933,18 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, { enum swap_cache_result cache_result; struct mempolicy *mpol; + void *shadow = NULL; struct folio *folio; pgoff_t ilx; - folio = swap_cache_get_folio(entry, vmf->vma, vmf->address); + folio = swap_cache_get_folio(entry, vmf->vma, vmf->address, &shadow); if (folio) { cache_result = SWAP_CACHE_HIT; goto done; } if (swap_use_no_readahead(swp_swap_info(entry), entry)) { - folio = swapin_direct(entry, gfp_mask, vmf); + folio = swapin_direct(entry, gfp_mask, vmf, shadow); cache_result = SWAP_CACHE_BYPASS; } else { mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); @@ -952,7 +955,6 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, mpol_cond_put(mpol); cache_result = SWAP_CACHE_MISS; } - done: if (result) *result = cache_result;