From patchwork Mon Nov 15 13:49:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12619541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9071C433EF for ; Mon, 15 Nov 2021 13:50:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 45ECE60FD7 for ; Mon, 15 Nov 2021 13:50:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 45ECE60FD7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id DB1016B007D; Mon, 15 Nov 2021 08:50:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D386D6B007E; Mon, 15 Nov 2021 08:50:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB2B86B0080; Mon, 15 Nov 2021 08:50:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A973A6B007D for ; Mon, 15 Nov 2021 08:50:15 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 5DDBF180373DE for ; Mon, 15 Nov 2021 13:50:15 +0000 (UTC) X-FDA: 78811298790.07.81553AD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id CCC6C509210B for ; Mon, 15 Nov 2021 13:49:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1636984214; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zc9KdahAguXkghVeJqdY3150Du1QUqNhK9U4RYp1RCU=; b=V2+NoBULSGm0HaWk/GjfROc/n5mHsINXCXm925VADcgNtahmI/N3843/MbdWFA+BJWZsms rnz+RIIBc5MrTTMUvtmMJZsE5qfCG+nuOzpwGfXsbHyXRcW4KyHyg7pNyleaYquEI6ss1j Eav1+xvKhG3UkNQ5GlJRpAO7UFiA7uc= Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-392-Thxoi8a-PymbllQ_ytqUCQ-1; Mon, 15 Nov 2021 08:50:13 -0500 X-MC-Unique: Thxoi8a-PymbllQ_ytqUCQ-1 Received: by mail-pf1-f198.google.com with SMTP id y124-20020a623282000000b0047a09271e49so9962813pfy.16 for ; Mon, 15 Nov 2021 05:50:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zc9KdahAguXkghVeJqdY3150Du1QUqNhK9U4RYp1RCU=; b=dW7T5ZqHPl69ba4e15gLdcpdRwFohRiwsbqpUG61DgjujwJIiALLuOeGjx9bRY5aUP 5OstK14N3wrDpc/G5C8EqaVdZwgY+Y+Z/teO41NeRYTJWCOWOjlPzY1jeCd5J9gPEaJk v9LRQRW1kENaB5B9juaqDn/JH87dlUef38gLHPbgFMAcNFzKoGeQiMgQXFbK1V4c6oXy q0AiTmLvrnEptAA6oZlH6sO60PinvzgSPMHN+3frF4p/oIF2LDjEBw9otHXoiblvuh2q /fTfj3mOR24/X6LZULeGAyPU/9L/YS0OdCr2iWdRywnNtlUozRtWeqETsENVWSTeQV12 5PLQ== X-Gm-Message-State: AOAM532efIdnu5SE+aD1JnO7Ckb5rEh3S4KB2iuhJURSBAKq8brdluFu nLsLIRzzTxkK/rWPDTx3G7YUxGfhUayC0SKBBBbCUP3No4/D6MFWmt3JbX9lZ0M2UjslltPQDle zktyDE9yWE44= X-Received: by 2002:a17:902:c7c4:b0:141:deb4:1b2f with SMTP id r4-20020a170902c7c400b00141deb41b2fmr35276048pla.44.1636984212229; Mon, 15 Nov 2021 05:50:12 -0800 (PST) X-Google-Smtp-Source: ABdhPJyVSWk3fByWPePLZoQ7jzLPYtHg8URAdE0VnftMpGgFIdHPUy8j42a0fPsBR8PAdhvEB0sRZw== X-Received: by 2002:a17:902:c7c4:b0:141:deb4:1b2f with SMTP id r4-20020a170902c7c400b00141deb41b2fmr35276002pla.44.1636984211931; Mon, 15 Nov 2021 05:50:11 -0800 (PST) Received: from localhost.localdomain ([191.101.132.72]) by smtp.gmail.com with ESMTPSA id p188sm15499471pfg.102.2021.11.15.05.50.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Nov 2021 05:50:11 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: David Hildenbrand , peterx@redhat.com, Andrea Arcangeli , Yang Shi , Vlastimil Babka , Hugh Dickins , Andrew Morton , Alistair Popple , "Kirill A . Shutemov" Subject: [PATCH RFC v2 1/2] mm: Don't skip swap entry even if zap_details specified Date: Mon, 15 Nov 2021 21:49:50 +0800 Message-Id: <20211115134951.85286-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211115134951.85286-1-peterx@redhat.com> References: <20211115134951.85286-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: CCC6C509210B X-Stat-Signature: 7wmyptob6w7n6nhtfpiyne3x6asub5mh Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V2+NoBUL; spf=none (imf01.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1636984198-90651 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This check existed since the 1st git commit of Linux repository, but at that time there's no page migration yet so I think it's okay. With page migration enabled, it should logically be possible that we zap some shmem pages during migration. When that happens, IIUC the old code could have the RSS counter accounted wrong on MM_SHMEMPAGES because we will zap the ptes without decreasing the counters for the migrating entries. I have no unit test to prove it as I don't know an easy way to trigger this condition, though. Besides, the optimization itself is already confusing IMHO to me in a few points: - The wording "skip swap entries" is confusing, because we're not skipping all swap entries - we handle device private/exclusive pages before that. - The skip behavior is enabled as long as zap_details pointer passed over. It's very hard to figure that out for a new zap caller because it's unclear why we should skip swap entries when we have zap_details specified. - With modern systems, especially performance critical use cases, swap entries should be rare, so I doubt the usefulness of this optimization since it should be on a slow path anyway. - It is not aligned with what we do with huge pmd swap entries, where in zap_huge_pmd() we'll do the accounting unconditionally. This patch drops that trick, so we handle swap ptes coherently. Meanwhile we should do the same mapping check upon migration entries too. Signed-off-by: Peter Xu --- mm/memory.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8f1de811a1dc..e454f3c6aeb9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1382,16 +1382,14 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) - continue; - if (!non_swap_entry(entry)) rss[MM_SWAPENTS]--; else if (is_migration_entry(entry)) { struct page *page; page = pfn_swap_entry_to_page(entry); + if (unlikely(zap_skip_check_mapping(details, page))) + continue; rss[mm_counter(page)]--; } if (unlikely(!free_swap_and_cache(entry))) From patchwork Mon Nov 15 13:49:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12619543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 040AAC433F5 for ; Mon, 15 Nov 2021 13:50:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9643A61C4F for ; Mon, 15 Nov 2021 13:50:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9643A61C4F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 34F226B007E; Mon, 15 Nov 2021 08:50:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D9276B0080; Mon, 15 Nov 2021 08:50:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 152106B0081; Mon, 15 Nov 2021 08:50:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0146.hostedemail.com [216.40.44.146]) by kanga.kvack.org (Postfix) with ESMTP id 03D236B007E for ; Mon, 15 Nov 2021 08:50:25 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C30C68226C for ; Mon, 15 Nov 2021 13:50:24 +0000 (UTC) X-FDA: 78811299168.15.6176795 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf03.hostedemail.com (Postfix) with ESMTP id B058730000B1 for ; Mon, 15 Nov 2021 13:50:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1636984223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/Mopq8eqVCQt3k2LgKCs45drQOHTZZHv/QrpqMo5cqQ=; b=BKGGyAhkYUcPRrZZeIfBY8sNOIWduzkbHTQyeVmq/UV4tZl8ngKylteWFpQpnmuW4DBnox EgCiyVnSdwcJqvwv8L0vp9+wOSu6e1FQdo6krojg8/4Mqagzv2wDq7NI7lVFNQaL9nCGC4 48MIGzs/HsfZzdM1cWe2dZA1RHIZHOE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1636984223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/Mopq8eqVCQt3k2LgKCs45drQOHTZZHv/QrpqMo5cqQ=; b=BKGGyAhkYUcPRrZZeIfBY8sNOIWduzkbHTQyeVmq/UV4tZl8ngKylteWFpQpnmuW4DBnox EgCiyVnSdwcJqvwv8L0vp9+wOSu6e1FQdo6krojg8/4Mqagzv2wDq7NI7lVFNQaL9nCGC4 48MIGzs/HsfZzdM1cWe2dZA1RHIZHOE= Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-503-SoWpPsuiPLGjL2S83_zUuw-1; Mon, 15 Nov 2021 08:50:20 -0500 X-MC-Unique: SoWpPsuiPLGjL2S83_zUuw-1 Received: by mail-pl1-f200.google.com with SMTP id v23-20020a170902bf9700b001421d86afc4so450181pls.9 for ; Mon, 15 Nov 2021 05:50:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/Mopq8eqVCQt3k2LgKCs45drQOHTZZHv/QrpqMo5cqQ=; b=fP23neXdcfzUQQ+I6nFjMTLWqK2Xj8hBiaEF9/w3I2d396vmMUGqSXc3hk769UJTaw nYbi1WhU2//ZMictXC2OAjWfp5DeKLwi/kvexSBKXlJo2iTfVKiVMRQDLdawVXUXvGNf Nbg/mU1GhNxNsfxRo+o5FY+N+pzNEcdvk+DmYmEKnydx9/xGSacAKl9vNgfrocs0pOYN wimr1RQ7wqnaTHKURunQeyYFoq27E3Y4Buceuo2RXq1EfDufZkxl2lGe8oJS4GPdpbFt JIuYgYTedCOIQC5WbQhNsQPiO+04mzI0NASEx5CXi7bK+GYUAap/QI1/gYc29wFPmeaD mF6g== X-Gm-Message-State: AOAM5300lJ9GeR0nV155xZmoMLROhqzM1djPXjLCO559tvIsRQGBthZn Z7293/BwGHArM9aL2TxbZZp77sMpQuh0D8eYCcXb9RHvYIuj22SDhDKEE2lpoPKBz2MbGR3ljUw JgqyNlk+9Y50= X-Received: by 2002:a05:6a00:8c4:b0:44c:9827:16cc with SMTP id s4-20020a056a0008c400b0044c982716ccmr32670989pfu.7.1636984219184; Mon, 15 Nov 2021 05:50:19 -0800 (PST) X-Google-Smtp-Source: ABdhPJy6qrMgzTa1gzoxCkZL1qeOAjLUb/BBgC3H5Gijen4dm8xnmmk9vd5L0ATwGFMxJeM3vHMokQ== X-Received: by 2002:a05:6a00:8c4:b0:44c:9827:16cc with SMTP id s4-20020a056a0008c400b0044c982716ccmr32670970pfu.7.1636984218969; Mon, 15 Nov 2021 05:50:18 -0800 (PST) Received: from localhost.localdomain ([191.101.132.72]) by smtp.gmail.com with ESMTPSA id p188sm15499471pfg.102.2021.11.15.05.50.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Nov 2021 05:50:18 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: David Hildenbrand , peterx@redhat.com, Andrea Arcangeli , Yang Shi , Vlastimil Babka , Hugh Dickins , Andrew Morton , Alistair Popple , "Kirill A . Shutemov" Subject: [PATCH RFC v2 2/2] mm: Rework swap handling of zap_pte_range Date: Mon, 15 Nov 2021 21:49:51 +0800 Message-Id: <20211115134951.85286-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211115134951.85286-1-peterx@redhat.com> References: <20211115134951.85286-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B058730000B1 X-Stat-Signature: rhy6xx88ht5j8ks5539aw4zcpakng6k1 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BKGGyAhk; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BKGGyAhk; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf03.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=peterx@redhat.com X-HE-Tag: 1636984214-378958 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Clean the code up by merging the device private/exclusive swap entry handling with the rest, then we merge the pte clear operation too. struct* page is defined in multiple places in the function, move it upward. free_swap_and_cache() is only useful for !non_swap_entry() case, put it into the condition. No functional change intended. Signed-off-by: Peter Xu --- mm/memory.c | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index e454f3c6aeb9..e5d59a6b6479 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1326,6 +1326,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = *pte; + struct page *page; + if (pte_none(ptent)) continue; @@ -1333,8 +1335,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - struct page *page; - page = vm_normal_page(vma, addr, ptent); if (unlikely(zap_skip_check_mapping(details, page))) continue; @@ -1368,32 +1368,23 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, entry = pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { - struct page *page = pfn_swap_entry_to_page(entry); - + page = pfn_swap_entry_to_page(entry); if (unlikely(zap_skip_check_mapping(details, page))) continue; - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; - if (is_device_private_entry(entry)) page_remove_rmap(page, false); - put_page(page); - continue; - } - - if (!non_swap_entry(entry)) - rss[MM_SWAPENTS]--; - else if (is_migration_entry(entry)) { - struct page *page; - + } else if (is_migration_entry(entry)) { page = pfn_swap_entry_to_page(entry); if (unlikely(zap_skip_check_mapping(details, page))) continue; rss[mm_counter(page)]--; + } else if (!non_swap_entry(entry)) { + rss[MM_SWAPENTS]--; + if (unlikely(!free_swap_and_cache(entry))) + print_bad_pte(vma, addr, ptent, NULL); } - if (unlikely(!free_swap_and_cache(entry))) - print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); } while (pte++, addr += PAGE_SIZE, addr != end);