From patchwork Thu Sep 26 06:46:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 81782CCFA07 for ; Thu, 26 Sep 2024 06:50:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VEMasL/thV9Tj1Tfta4uIol3BYRuzPr3NckO1OF+52k=; b=m/mO5mkMnGmgtRj/ZAcw7aqWJM ciwYs5jNBXrpJDEGToYf3WA7uph66/ap62bgu6Ty0sxYPk/Ag+l8bk84HLEpEZBZfteXHgOqBd48x kFGDpNcvynhc2J6VvjeJiKjrKblWXp9UgYW5PRwDC2SekdIqAWdTtycvEwuTaCj0Ixqsr86XKIe+6 VUArksJz0Je1yotAPlCpK2NBRFbfbIHPi9kFa5wCUhlez+n/sIpFsmjpXW6LnLR8bslQ+Nu7X+Rmn IqIIsOKW7lvjb+dEKB77sE8yNS9/1JnPYKDtLZq5GfEcX8ln0I1mRyxDjIj+MBYM4TcPE5fPTZm6M Ys2FHS3Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiKf-00000007OWQ-3WLO; Thu, 26 Sep 2024 06:50:33 +0000 Received: from mail-qk1-x735.google.com ([2607:f8b0:4864:20::735]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiHB-00000007N8Z-0PMi for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:46:58 +0000 Received: by mail-qk1-x735.google.com with SMTP id af79cd13be357-7a9a3071c6bso69263285a.0 for ; Wed, 25 Sep 2024 23:46:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333216; x=1727938016; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VEMasL/thV9Tj1Tfta4uIol3BYRuzPr3NckO1OF+52k=; b=AyKJKiSKaxYoQqwvKNz8ZpCbtVPKwBgpXpgV/awjogzw0DSCrLEPtkhMXV4uSXxdyi zdluipgiABsXuLeLZLpAVMSAOSWcz25vpKb9si084mPQ1qTAex4vNQBluUDE8yaM+rVW fiz4ra652PX9xQinuiluyPHiyYNg9CaNuWuBWXRDvTpQ52Dh+BnrV2bLu+BgRhu3euh2 3totJoRNdPsNSoZwXdbd3q1lXZDUacWLgfM1zU7WiBLvhfV5SZxa5NkoFJjKBkgEwsos dsFEjgbjIRhUfVnNwAvU/II+nFZsceLPcMqA4gmOF3a4YDZSOi5V9s3I6ThY7saJ01Vn AVzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333216; x=1727938016; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VEMasL/thV9Tj1Tfta4uIol3BYRuzPr3NckO1OF+52k=; b=EhENVrlU/+ciilnfUlQtcOCRjJYiDzJJWGaA7omhMlWsNnMS0u1/1qY0fgTB9vOEQO rIQY/r1TYoDv7Qus8uMMGqIRg3Jc8osJQumqnBAYM93cb173wTkpcTq9RQei3OLPy9Si fhrkZnonrrEUWcbSsBHXxXmaOQVK37KmFp6a27rGgaNgGngWOSDPChJqA1iW+YHZGdhc VnNZ29cEOUQjjPG6mHE89vvsCXF3g/5OAmEDxmLBxgKcB/ct5f3BC8E+LiGaPuO2Rn3w t1yyjNCUSzjqsxlJb0mCmf0AUozv3T125842KSJvuQj1aahK8l0g0DasaCxDP4HEuvfX zqaA== X-Forwarded-Encrypted: i=1; AJvYcCUYT7IGUIvvWSNlv7Kc/uNjcEacwJj3IU79aw3cuclRqXnCW2z6fi5gWSv6dXs9PUXd+MUkyXnJFJk/tX1itPSW@lists.infradead.org X-Gm-Message-State: AOJu0Yy/n48UPLydQA8NNf4tcJSAr4LHrH8tJMp3R2tHCvyWeOc3sket sL3mtLELro56MVABHOhf4uDPs/0D89aUd5ZjWla9DR3Pk57m7BwrlUlW9hTeCn0= X-Google-Smtp-Source: AGHT+IFdBHDg/pcI8fYkiDBKrKz5/bcsNyjF/Vh4ia+kT/ICQCu69/2acL+Eh3ZQbIhYSZdsv978Jw== X-Received: by 2002:a05:622a:20b:b0:458:4b6b:ff16 with SMTP id d75a77b69052e-45b5def4888mr86143491cf.32.1727333215740; Wed, 25 Sep 2024 23:46:55 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.46.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:46:55 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 01/13] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Thu, 26 Sep 2024 14:46:14 +0800 Message-Id: <5aeecfa131600a454b1f3a038a1a54282ca3b856.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234657_178558_A3A1D908 X-CRM114-Status: GOOD ( 24.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. It is applicable for may-write cases where any modification operations to the page table may happen after the corresponding spinlock is held afterwards. But the users should make sure the page table is stable like checking pte_same() or checking pmd_same() by using the output pmdval before performing the write operations. Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will be read-only/read-write protected. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- Documentation/mm/split_page_table_lock.rst | 7 ++++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 48 ++++++++++++++++++++++ 3 files changed, 60 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index e9077ab169723..46828b9a74f2c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3017,6 +3017,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..daa08b91ab6b2 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,31 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, return pte; } +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pte_t *pte; + + VM_WARN_ON_ONCE(!pmdvalp); + pte = __pte_offset_map(pmd, addr, pmdvalp); + if (likely(pte)) + *ptlp = pte_lockptr(mm, pmdvalp); + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementation * __pte_offset_map_lock() below, is usually called with the pmd pointer for @@ -356,6 +381,29 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs that - * either the mmap_lock for write, or pte_same() check on contents, is enough. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); + * but when successful, it also outputs a pointer to the spinlock in ptlp - as + * pte_offset_map_lock() does, but in this case without locking it. This helps + * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time + * act on a changed *pmd: pte_offset_map_ro_nolock() provides the correct spinlock + * pointer for the page table that it returns. Even after grabbing the spinlock, + * we might be looking either at a page table that is still mapped or one that + * was unmapped and is about to get freed. But for R/O access this is sufficient. + * So it is only applicable for read-only cases where any modification operations + * to the page table are not allowed even if the corresponding spinlock is held + * afterwards. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the pdmval. + * It is applicable for may-write cases where any modification operations to the + * page table may happen after the corresponding spinlock is held afterwards. + * But the users should make sure the page table is stable like checking pte_same() + * or checking pmd_same() by using the output pmdval before performing the write + * operations. + * + * Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will + * be read-only/read-write protected. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a page * table, and may not use RCU at all: "outsiders" like khugepaged should avoid From patchwork Thu Sep 26 06:46:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 32E10CCFA07 for ; Thu, 26 Sep 2024 06:51:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=T4mKgUXXYBowibFxHMs4Thtkpj zEZx+gE8weUz+XB6RQkbhHNc6xRhVYJivz6HmYtaKYpXaAQfa4XZIz4+SEgcDgZAaunEQaqTyWpNy NY5+Ua0zqPx/awGhUiSYtzmYagQFFPO69DvOXPcOE88I+d8j9WI1ry1Ix+oEGCT3Gb4sL09L3jNvc 1fLIzNTUunWl9h8EDmcPhc2W1da69ZId3gq85JJ3b/Ct8yrDWsIPmRwiksWajeu7EJwyoy7hR6c/K d5aLSkNdjsImCS27uJxyRWV1YlP/mk1hCPObCssqvUzmP9t5Bixdqhv6U15TlMzhDNAB6OEMkYMno hcX094Cw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiLp-00000007Oh6-2ZaH; Thu, 26 Sep 2024 06:51:45 +0000 Received: from mail-qt1-x835.google.com ([2607:f8b0:4864:20::835]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiHI-00000007NDM-39zF for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:47:05 +0000 Received: by mail-qt1-x835.google.com with SMTP id d75a77b69052e-4581ee65b46so4697041cf.3 for ; Wed, 25 Sep 2024 23:47:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333223; x=1727938023; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=hfRYRoXekCR7hLDVQNGSFhqJqZknhdX2EXeqTGepmFpxnmnhh+zsJSy7DA9F24d79J oqz04dLLrA7FOo4+F7OHylVRFFErHbMk3DNS7WEQVGmXARGrsyOq+o/FIQqjEFVCDjei KIHvLiGZhbSkVEHAZK0ZbwEQm0Ma534XKHu3/x84zfMWydW9SCTtKcwBGdVuAV+AOBiC ReHHM/om53SgY5D34uOtSSQ32/d9yL+O2tr2rSYQk3UEqZzQBJsxZ8YzZOhIStn/Pr27 BJBCyl4YFJSU9triqEG4ueRe2XdohpRmPLrsJSGOMDWr+0cNFJZ/Teg5gyScPBxC+S6F QIiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333223; x=1727938023; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=raOY0RWky5zhFzgGfj++3w/MYLWYu+kTmOfYbUc+5pDOPABePhQvteJoPLpSczKiBd rnY0vRLoc0XRqsf8q3TCGAwxZ9TGGLlBCKSupYvkf7574qwY9T8eaIGfdMwYMnS+V9KK 6T0rkuYb+RZcsHvWAyh+y5wHCfG/PhRkoVKQBNYmYax1xUVhf/QSFLKefXlKXTCy1Sow tQxwvBgvZteWRuJ3MWm3ERexHprTQPPKf+SBUxyUB68uciowuBLUfXg7zac6/GdGfQKA eNgfo0oOajAe3NGg3nEK8VMolIEs/DklkjGk8NANQ+mtnTcCoHuXPJPnXmMsk5hv5nVB ldtg== X-Forwarded-Encrypted: i=1; AJvYcCUXRQpKZyqfaD3TKo7cDDwN2rlbrb4hhB8toZE9xQSRITKtUAaxLudM7mVAexYSLRBU033KVkY3R96cXkXv2djO@lists.infradead.org X-Gm-Message-State: AOJu0YwU8JUxtvhKn/vMZ7fh08KzKBeYE7NHtglO2dOZhvXpkfXStSml pHSW/zg4hUZlNjU2ywBnmstdaM3oYXBG6pMy85oGVMGjuJtIqZh+MsV3wkpelss= X-Google-Smtp-Source: AGHT+IHKcdRFC9q2GD6595fUPe4NgxH7f7bGHM/1jVFUOYjtanMZU2hJPsIx01XdKDMweQg9NQ43+w== X-Received: by 2002:a05:622a:105:b0:458:4c86:4564 with SMTP id d75a77b69052e-45b5e0952b0mr82475131cf.37.1727333223424; Wed, 25 Sep 2024 23:47:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.46.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:03 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 02/13] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Thu, 26 Sep 2024 14:46:15 +0800 Message-Id: <42559e042eb6fc3129a40f710d671712030646b4.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234704_808796_F335E50E X-CRM114-Status: GOOD ( 10.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) */ if (pmd_none(*pmd)) return; - pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte = pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); From patchwork Thu Sep 26 06:46:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CE736CCF9E9 for ; Thu, 26 Sep 2024 06:53:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=Jrq+5duwpj+H43dkkr4u5NL7xf i905m/SuZVn2KSzO+oxGYfEE1DTiYxi5S9ECJ6wer6UyUiBumwcFSg/E+hFYE6QAQIBV0th42ynom leccLApmhOkC3uav67bZIZUeVJAGFh28X6311SSK86BngsFX4zm6JfnCELMGP379de9go0wPUNqgq 2wrwUqK4EWHGpkxdcEhzOLE1K4B/Fw12TfXKfW06ZA+JNVpIdsI2QY85yok4dlax6ZqjeJyQmp0ud oyR7BseQ0bYejxDlOrTkCJUuYvLRLMjA/dVTlGiWGwgZinFUu8Xg9cZG/XLCN9WNpLxL0vQykJ7Jk GNUbpgHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiMx-00000007On1-19so; Thu, 26 Sep 2024 06:52:55 +0000 Received: from mail-qk1-x732.google.com ([2607:f8b0:4864:20::732]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiHQ-00000007NGW-0zhe for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:47:13 +0000 Received: by mail-qk1-x732.google.com with SMTP id af79cd13be357-7a9ab8e5f96so44559785a.1 for ; Wed, 25 Sep 2024 23:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333231; x=1727938031; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=kG+/wloWw/iUnVgXL6okmAIEZP246/hL5FZGvFFNCZw4r/7ex33fGwFZsQLE7/thps aqGa/xisADbU5AKx0zWQ4u5e35V1gBdQSTGCvosmgJXsoQv8RO9xNMT3Vlta3xoe6aTA QpTSIF5z9uO3K56qwjaQKRdsPRc3SmW8h1ZNPHGBwl8QBsnpatlNH6TJlBBZWo+B/NS/ 177RZjMTrk0z+EU3C7Gbwx30OuWs4WHAfai7fQ76mZNnMRuN2qD/YRbEsJyxQKVJMRyR IrO2NWkijiHgUhnuV5cXW6bhI1fwbVgWynpZVO1pN9B8lOeoCxUG2BA3qhJGFD5fbUbf ljKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333231; x=1727938031; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=TaqOrrViKKP1bACbGwo7NdP/MOcGcgWkYisCZVwh09bWAM0fUogaNy1sW7ib1Ei5RN xVvaSn6MtzmByx9wdshIgnh3DieGv+1TG7jCzAAsVbofLxtcDuUTz5QKv5zFp7mYx8Bx cP6ayBev+q7e+oAZCkUtdAv778II6aMLfqmGeko9SeyJukmyUxIzenN6n2duEM2Ka8q7 uwiSrbX37Syr7CZCIu4fGMbEK6KY7luGaZNPhgR4Wno/vpFvg8rfNSEk/aYDf8yzrZ+l BiR9Kbzn5Uom7X+49rqLCMA6eg/9XFNf/GO2tBLV+XPkBQy5QxoR690bccvbUNI9HHWF mMIg== X-Forwarded-Encrypted: i=1; AJvYcCVEkR9GV5IwP/pyHDPoWdZBiCgsuXYESAw2LgO1XaxuBPw2drEILBMBD5iXVgPy0WWIXl6HhqwBzoKR3Mr8pRCI@lists.infradead.org X-Gm-Message-State: AOJu0YzsCNVySHdlEPjwNbyZlmEqmnrebIpx5izf0+2bgu3GdGTI/uEB TXmRTVYP6GV8PDPVP61DhaBxdeZastFel9upbWYC6W1RCtUueSBkbjMScs49PxI= X-Google-Smtp-Source: AGHT+IF3jq8XAx4d8MvdG//ydygdKBfb8RG21eWCHGRQrOC65QRa0E/gKmvca15+75cy4x5ki2/uwA== X-Received: by 2002:ac8:5914:0:b0:458:fb8:9dc3 with SMTP id d75a77b69052e-45b5def1dadmr61649591cf.25.1727333230912; Wed, 25 Sep 2024 23:47:10 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:10 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 03/13] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Thu, 26 Sep 2024 14:46:16 +0800 Message-Id: <9f7cbbaa772385ced1b8931b67a8b9d246c9b82d.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234712_292087_1F60529D X-CRM114-Status: GOOD ( 11.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3e46ca45e13dc..6c6ff8722550f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3271,8 +3271,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; - ptep = pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep = pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; From patchwork Thu Sep 26 06:46:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E50BCCCFA07 for ; Thu, 26 Sep 2024 06:54:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=OPev8hwJDwNp7MqDiubdZo6m3A +P2CNrjC4DzlL4bnRDEM+lTNTwGlU0Xv2mMaXPGoJ3inubCZ4B5n2YBKUcRI/Vd+J+d6+q55dH8xd 33NHxpuoXAuULVjpjN7Jys8Sxkc41MFrXw6k3v6LgvzScggty6xs+9SDm1jaW+2jd6ywEYylhIiG5 z4B387fGndPwj1TFjn3rRZMfoj1FIPERF6ESdx2SMT7yJJvHZBBa8iASt6lTjLZnuAQjsdkytr+Ld oRgm4xzOroD1ZJNfBwfNOoOII0lYez6p0AViiuAGkQ0r1KSLBxDjE6FB0YI/hR7OsV+VzfrAV6lY2 g1MZBkKw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiO4-00000007Oub-3KZd; Thu, 26 Sep 2024 06:54:04 +0000 Received: from mail-vs1-xe2b.google.com ([2607:f8b0:4864:20::e2b]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiHX-00000007NKg-44RG for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:47:22 +0000 Received: by mail-vs1-xe2b.google.com with SMTP id ada2fe7eead31-49bc7387371so271698137.2 for ; Wed, 25 Sep 2024 23:47:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333239; x=1727938039; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=R7qC/HCrYKJGrFuunD5WZDpikoghB1FOIj0VNPgrfQ2AniOLjSeQtvZ5ebivIkSAyq UBm81zoWWlzXkc2lxpX2hf1UI/CzSBtGBRoMR3JjUCgbLYjTVBzeXvsLT5ZX167sXl4M 9ZSZAyPfTNP0qwWD3hW5sYS1S5RECrEbeyNkUUEKIJFD4dFydRCjmzi6ZU6pngWQykn1 l28INegK56+UwjE0cfNVAXScFtNgXWRoEu/ktkVIXVRCp4+vun/7DJGyy7EDt5kmg090 0cFzcg/lvO0vuyvGWJx8zoyXAcTAY4Cwyzs9FAHhC9PVe0cpZFLwiTH0tft7I1SwskVQ uc6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333239; x=1727938039; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=YtT3CgCiWbOJJV39my+UIptixmkv+D1ixGlmPV3Sx+0XrNoWI774x07qBf697stcRS K1WkE5kuFUF1TpnM6tw8BMfsnD8ywBd6ugFmMNnc+X/oWrzwKBg1t+suj6JWhnbsiDGH E+6vktZvg5RWFU3kNX6GkUNMxIuyl2BRXaayde7SnNyOE49XSuGeFdE+BVbYmKPLp2kI /bs34riRnXCZe3dWwRxengAAUJoTEsGDcM9ZbYRTpm7v5QUGZT8qYv4lqxaiH35jllwK vjJaj/p591Tvf+RRQeTUqx5H4d4byrSbVG8fBli/ibIlkLytqR9+XYDsJxoQSVW7PypT 3vfg== X-Forwarded-Encrypted: i=1; AJvYcCVqpCE+btnsJclkLrfVF6sCnjW8KVwEZm9CyEBv0mcGrgTLKIPLCmtijwTvJ96b1lhosk5pXntKSugBzxzw7/Xo@lists.infradead.org X-Gm-Message-State: AOJu0YxcU3fZ9XE2P8BGNnx/Wbp+aTGdlpIVHz7Lnuk0KyUgc57Z4Gyw TpZAEe80chd3lY04SV+9lTiAgy+fpTEkU/5xQ/KVXJcaIhqQJxO/0aanNTrqzpg= X-Google-Smtp-Source: AGHT+IHnBlRBHwKCxjYpHBiGYKwBIoiWGnfAohABSGODP2aU4XU9V+T3qIbiD3kwEf9HDK6sMKBjlg== X-Received: by 2002:a05:6102:6c3:b0:4a0:1cdb:3eb4 with SMTP id ada2fe7eead31-4a15dd1a11amr7011565137.20.1727333238630; Wed, 25 Sep 2024 23:47:18 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:18 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 04/13] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Thu, 26 Sep 2024 14:46:17 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234720_047839_2786B0CB X-CRM114-Status: GOOD ( 11.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f9c39898eaff6..6498721d4783a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte = pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; From patchwork Thu Sep 26 06:46:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 29D1CCCFA07 for ; Thu, 26 Sep 2024 06:55:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VXy51M36oXFNN3kWVEF6T1v76BlvE2cEswNH4MTPbjU=; b=223RvYJRB6uVPe0jEgSNmA4btG Bqexi9ZbuyslJiHSrr3QVy6fSQEVDElue6Vrxbd7EYFGoLSN6MWXzz3JbLRXMrfJPI5uSctxYGE99 PjBlBBtKRo9rZkd90XAAIDqYobNMmTiey5ILD2mWsDJcE02Il6//bFbM7BAFN4gq2SRERy+p1PCop RUVrw8hPsPj9Ju4cXuRzfvKwZkn4X3cwODU5upWfnVuBlIKbYUqQmJls0oEGzp+T9YmkwZNCBwG4w PwX7sobseSGNjg7b9aMD/rVwANYyeNfUkWFA698oyUWyyg8GOKDm8iS9uTTz0MOERGzo045qgKCX1 1QuP5Ssg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiPD-00000007PEB-2VV8; Thu, 26 Sep 2024 06:55:15 +0000 Received: from mail-qt1-x832.google.com ([2607:f8b0:4864:20::832]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiHf-00000007NOH-2BFl for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:47:29 +0000 Received: by mail-qt1-x832.google.com with SMTP id d75a77b69052e-458362e898aso4395551cf.0 for ; Wed, 25 Sep 2024 23:47:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333246; x=1727938046; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VXy51M36oXFNN3kWVEF6T1v76BlvE2cEswNH4MTPbjU=; b=lcCAvSa//xV41NjMFEMrg3qlHU1nzU6Gjj08zCzoRbDwrx/aZYWMSo5f8aj6e9wnN+ Y9r81f19UjpykpDeq7J1uWAj7k3j2OWEGMXEjWHnNkfcUk0yxkZ0JBHk1PgiVuLbYrqt cggpSTQCbwQHOei1D/K4uMluvKtSmVllewrF85nf1hoq5fukeAqsKkO+9OtvcJ9YJam5 27BblynUo/+sFkpXE5QBSeLpyAYAmiFoCOZMBJ+Ye2LMunZMtjDtTgcwNexWkjezabyv 9Gers6we838x3ZHtevI1ovho9//SBFcXuPtO/Aci3ejrGT614k0TznGfuUpv0my29GGQ nuEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333246; x=1727938046; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VXy51M36oXFNN3kWVEF6T1v76BlvE2cEswNH4MTPbjU=; b=u3OzsWsPexXhNpfIoNhxvhDfuxEiDWBdVf1HNPlezvxkyMhvqnx/v44SRIleJ1W6oT McmCUwXL0gV0sZvRQ1K1bAJ+TAmsud16CyMBGKV+XwDbhKikmGPeiJ6E9kFLMaodWYfm FsJAc21PY4qm0lSHS59ruTVyfSFlIk6TK0TZDjjQdx+7rm0gzuXHNvuQSZel5XsSMl0K isCWj8zEPuPV4LmnAE15+LxF8WYDSm+pmt62aKm5XsyWwLBberT5hFDrBxQ9zZKcyOk8 169neejgwXEaJXbTe4EBsbBLOxvUZaI5nl1TDoWisDTAbC1Y0mkewR3VBxk3+okO3IS6 Mh+g== X-Forwarded-Encrypted: i=1; AJvYcCUoZH53dQDd3c7dY/32aEQtX39FUxmVI9g6KYe2jl+q+Q0/T7pzMxMGR1HiiZgEWROfIjNtG7guea2/4ROw3JIe@lists.infradead.org X-Gm-Message-State: AOJu0Ywd4ICtwychWKreoE4C00kCQb1mxDLHd1QNZ9kuclhHQfvJFC2k +dUJ57UJ+We9CX8AvRtQeBv7iBhyrmYJy/kEnBcUHGMbh5E/vrpyp0ibCWS+1PU= X-Google-Smtp-Source: AGHT+IGInGARMP6O0lOtN8I02J26ctBRW9WAU/Ifxw0RvFOYg2EOnEie2juwEDAOYLUuf+d4v6O7qQ== X-Received: by 2002:ac8:5885:0:b0:456:94fe:dcdf with SMTP id d75a77b69052e-45b5def4674mr66807991cf.35.1727333246109; Wed, 25 Sep 2024 23:47:26 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:25 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 05/13] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:18 +0800 Message-Id: <0eaf6b69aeb2fe35092a633fed12537efe645303.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234727_640079_0451D7DD X-CRM114-Status: GOOD ( 18.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In do_adjust_pte(), we may modify the pte entry. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. All callers of update_mmu_cache_range() hold the vmf->ptl, so we can determined whether split PTE locks is being used by doing the following, just as we do elsewhere in the kernel. ptl != vmf->ptl And then we can delete the do_pte_lock() and do_pte_unlock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/arm/mm/fault-armv.c | 53 +++++++++++++++++----------------------- 1 file changed, 22 insertions(+), 31 deletions(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..2bec87c3327d2 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -61,32 +61,8 @@ static int do_adjust_pte(struct vm_area_struct *vma, unsigned long address, return ret; } -#if defined(CONFIG_SPLIT_PTE_PTLOCKS) -/* - * If we are using split PTE locks, then we need to take the page - * lock here. Otherwise we are using shared mm->page_table_lock - * which is already locked, thus cannot take it. - */ -static inline void do_pte_lock(spinlock_t *ptl) -{ - /* - * Use nested version here to indicate that we are already - * holding one similar spinlock. - */ - spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); -} - -static inline void do_pte_unlock(spinlock_t *ptl) -{ - spin_unlock(ptl); -} -#else /* !defined(CONFIG_SPLIT_PTE_PTLOCKS) */ -static inline void do_pte_lock(spinlock_t *ptl) {} -static inline void do_pte_unlock(spinlock_t *ptl) {} -#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ - static int adjust_pte(struct vm_area_struct *vma, unsigned long address, - unsigned long pfn) + unsigned long pfn, struct vm_fault *vmf) { spinlock_t *ptl; pgd_t *pgd; @@ -94,6 +70,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; pgd = pgd_offset(vma->vm_mm, address); @@ -112,20 +89,33 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, if (pmd_none_or_clear_bad(pmd)) return 0; +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte = pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; - do_pte_lock(ptl); + /* + * If we are using split PTE locks, then we need to take the page + * lock here. Otherwise we are using shared mm->page_table_lock + * which is already locked, thus cannot take it. + */ + if (ptl != vmf->ptl) { + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + goto again; + } + } ret = do_adjust_pte(vma, address, pfn, pte); - do_pte_unlock(ptl); + if (ptl != vmf->ptl) + spin_unlock(ptl); pte_unmap(pte); return ret; @@ -133,7 +123,8 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, static void make_coherent(struct address_space *mapping, struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, unsigned long pfn) + unsigned long addr, pte_t *ptep, unsigned long pfn, + struct vm_fault *vmf) { struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *mpnt; @@ -160,7 +151,7 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma, if (!(mpnt->vm_flags & VM_MAYSHARE)) continue; offset = (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT; - aliases += adjust_pte(mpnt, mpnt->vm_start + offset, pfn); + aliases += adjust_pte(mpnt, mpnt->vm_start + offset, pfn, vmf); } flush_dcache_mmap_unlock(mapping); if (aliases) @@ -203,7 +194,7 @@ void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, __flush_dcache_folio(mapping, folio); if (mapping) { if (cache_is_vivt()) - make_coherent(mapping, vma, addr, ptep, pfn); + make_coherent(mapping, vma, addr, ptep, pfn, vmf); else if (vma->vm_flags & VM_EXEC) __flush_icache_all(); } From patchwork Thu Sep 26 06:46:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3CDABCCF9E9 for ; Thu, 26 Sep 2024 06:56:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=nb9n7E67s2XhOAzaJJIvK6ZLpZ mcx9W9U359Zo9IbE+svY5l9rEL8Tc37G7ixA6IVDT2qa2gkmmB3mdJE0d7kOcCMzVBUH6cbb8JV4Y C2KNMLNTJFU3l0tEvQWNmJHfV4HjqKdue+ItN9qHYETqQdGMav4FLrauZhM03E8Pof11HL6S1MJKr kbefHJ9QWgZ6y5l9R4Dp3xV0SQgBbBgf+oZrgcGhCnIT4uS2BPxKQZ1Acr4Hzf+7Nf66uy3VHlE6+ cuO3gb05SRdz8JbMlladKfW1Eax0HIfqJg+Z65MN7MUTCgTNHlP5GHbbOpbmJL7qhZqBaKYmMwVqS VtPaS6BQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiQP-00000007Pfm-42xr; Thu, 26 Sep 2024 06:56:29 +0000 Received: from mail-qt1-x836.google.com ([2607:f8b0:4864:20::836]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiHn-00000007NS7-2PgX for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:47:36 +0000 Received: by mail-qt1-x836.google.com with SMTP id d75a77b69052e-4585e250f9dso3064871cf.1 for ; Wed, 25 Sep 2024 23:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333254; x=1727938054; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=gseWG6ZFpn/oU53572WELno7DYvQ/nYjpuC3dMPwVnkzd9hXXBwUQggSNS4kgpxVQo yeSzui97RdGo5FtLxFQE/ediKpQVzu9i7NCbsARpgDk7nijZoDamHE8v+niJ7RJxFEpb pCjQzWd8rYxX2BLGiRkYUDMkXkWdfLyN3FPS24LkXVv5RRmI2A/K+kcNAm1EMkBkI0xv THYVgsjC9iEt1rj7lupHAgB5CyXTXEo4IUQyHU0hkyx9GGNOlmefn9cspI6hxOzjWxwS WPRDEwb0kFQdMl35d+2116XGhWKA8+BAH+u+3sLumLTjgR5BbuM2ulneRyG7wgLDGgHj AZ0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333254; x=1727938054; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=GageM00VItgBgVa8VEvGDck8HbxLbfyYaS3YjmIRtEG5Qzh0LnSQbdXr8XyJlF9OpQ RBoQ/AjGjT3KbtugCgInEwVM5vW2HrXXkbNU9JAUTeK1G3Yaa1GAwu6LaRtXkvGIxBnv vlZQjbyEdpwtPYJRUyJfYw+rZAQgNBxfIB4fAQOfRCIhihVfXwluW7tVzzDhwlVvVfeO JisT9xRu74OX7OB6nFN4m4qd8l1Ai7sk59PI957kJ0Q7Z2085RSzZQUJaa/xn552MUNG wS1j2ha5BItWz8ERus5/7UnAQWihd/pw04eLQBu90lzVnCi7/fC0QK9io3SU+oQRssvH 3bUw== X-Forwarded-Encrypted: i=1; AJvYcCXQ1TDhKPm0hHJCyWkXUFq6C49WRpbounJ3+mhWvMWA5DwRZr7tUXE0dxVmpbD4Kw/mRK2DqeaspW1sJJBrVQSU@lists.infradead.org X-Gm-Message-State: AOJu0YxmEYQZPwmpJYJ5aWpFL/0YMIPTpDr6v0PqCMvm6u4JD/p5qM8N nhOMQxI8TO7LQL/fqokssTl6hCM8l6XlnuOaVFxdM5GJX42JwJwyKVirg9iGYAY= X-Google-Smtp-Source: AGHT+IEkNfjkqcOSOf8WFqyh7ycpXwLGwDHOuzbRwWfPgE4JYY85w8LbZC4JktZjOBGqNKfXWcA2mg== X-Received: by 2002:ac8:5a43:0:b0:458:3824:a083 with SMTP id d75a77b69052e-45b5def43dcmr69885101cf.32.1727333254117; Wed, 25 Sep 2024 23:47:34 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:33 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 06/13] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:19 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234735_659889_76C6B2AE X-CRM114-Status: GOOD ( 13.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a245d28787034..6432b636d1ba7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5750,14 +5750,24 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->pte = NULL; vmf->flags &= ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte may be + * modified, but since we will use pte_same() to detect the + * change of the !pte_none() entry, there is no need to recheck + * the pmdval. Here we chooes to pass a dummy variable instead + * of NULL, which helps new user think about why this place is + * special. */ - vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte = pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); From patchwork Thu Sep 26 06:46:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DCB1BCCF9E9 for ; Thu, 26 Sep 2024 06:57:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ffdIJrLHJ9HLo1zmy1fROVnOG/sDLfcmik2lSMoQorU=; b=FyN5/g6dN4bNzB6q/5acDOR93V neMz/66BdA48bvfEv4wgcqtuvrynRZ+R1XEjfmggbxw6viaJlvKc56DOe9gVPI5DEUbqxg4rN4Wz8 /TBjU61My0XQSmBILtvPgbfqsJNcA/2AbAaOE3xGfETR1KYPnMSV05auQxF+PhqWUv4lflw1+Uk0A joR7+L6eDYCNDILAYc0haAQ5X2DHpdfsqR7DVloFOPwD5S1ChRlNIcDxWSb0F1PEBq0cBoPxYXwAZ 6LebI9CeXxsZOaVhD7jdiHj1xCJwzWbG/2iE4J77CvpOf7D23vR5BHPix+3FGfiUz4LcyJXKz5hPN ydydlaAw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiRc-00000007Q1y-08lJ; Thu, 26 Sep 2024 06:57:44 +0000 Received: from mail-qt1-x82a.google.com ([2607:f8b0:4864:20::82a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiHv-00000007NWS-0zMP for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:47:44 +0000 Received: by mail-qt1-x82a.google.com with SMTP id d75a77b69052e-45816db2939so4046201cf.3 for ; Wed, 25 Sep 2024 23:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333262; x=1727938062; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ffdIJrLHJ9HLo1zmy1fROVnOG/sDLfcmik2lSMoQorU=; b=BqvzdPRwjq8A64Ab2GnJzuF2l8VEIgPoqVRPVupOH4fZ/f2NIF4iqS++kQZnI7dpak 6BIyPuWoJYWfzfpFzUvMXecGcqcrQosTef7Jz7P/WVV0ECZPCmqV94iGsKiBD5+0O5Ja 3FnTtIcWjlb2qlU6Wcl663i3/IFYGqMUZN1IXjN+hDguvZRQnmjjaGqMAsyQ5HXAGzKD YfmR87U2JxszvTK2q8J9NCzca/FT8RzBrb+F+3lttBfh4KJ4VfVwA1DOCqb4l9Hzs2kz f1c7Ifhe6e/GdhU2x4kxO49zdETiZ0B3KevUk8DTiI9u3qG0th0f6NToAS8DVVj0QwGn FHQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333262; x=1727938062; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ffdIJrLHJ9HLo1zmy1fROVnOG/sDLfcmik2lSMoQorU=; b=ae/fjtJzQoIkgSy9FMvsniFjq6JEFKbSm6Y/ZJxZIpeqERK02KDl09gY0EOFFmlwtk lvaGUE8AWByZLrOR+F3ALYBTWuYGeuznhyiBqYeJRwccA84B00PVpApfIqN39Fepr5lk Mwm3Fpb2KZa/YfTyeMxtbLxAtoh+L/+pRg1EiIQbP9taK8DGjtSnr53vI1SUGBcoBW75 2TxhNs4wboS0RcU9Bh3KwuQluNgjNePaEjkMogmS2ax9l3YVyU77gHYkkyBHT7YV0xG2 udeMpJZ3DROm468Lkmij6zv6Ng51xqF958Rcmp3XMFbu6Pjf2VT9wbK19ebbNBkLmBCF peaQ== X-Forwarded-Encrypted: i=1; AJvYcCUZaOkf0IdEL+jUkIGHoDP8pdQhP8KFAyJfcAwsO46QNCFcQ8EFYalPEP1AQGUfm31IhdijLtlBKuaf2j9u4JMu@lists.infradead.org X-Gm-Message-State: AOJu0Yz57wm2JXMh9bOlAv7aMCiXkxvhWOdBXeTmljpd3ObgJ2+ndhx4 4J6Akkm02Jdsca15DmknYlv2fAev29aMTpJEhKmA/pKIPLvrhHOcbS3huYMhcWY= X-Google-Smtp-Source: AGHT+IFPPE2WMfZ0ye6LGa3/R8zXQ/6KjVtS14uw6772ox/YEy7cH2DCZDAwEci91jHXPd4jVDRFUg== X-Received: by 2002:a05:622a:1493:b0:457:c747:cf3c with SMTP id d75a77b69052e-45b5e0534ebmr99370671cf.57.1727333261763; Wed, 25 Sep 2024 23:47:41 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:41 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 07/13] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:20 +0800 Message-Id: <055e42db68da00ac8ecab94bd2633c7cd965eb1c.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234743_309455_CC0CD18A X-CRM114-Status: GOOD ( 13.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/khugepaged.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6498721d4783a..854577f39957d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1605,7 +1605,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1613,6 +1613,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1645,7 +1648,6 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, nr_ptes++; } - pte_unmap(start_pte); if (!pml) spin_unlock(ptl); @@ -1658,14 +1660,19 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); - if (ptl != pml) + if (ptl != pml) { spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + flush_tlb_mm(mm); + goto unlock; + } + } } pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); pmdp_get_lockless_sync(); + pte_unmap_unlock(start_pte, ptl); if (ptl != pml) - spin_unlock(ptl); - spin_unlock(pml); + spin_unlock(pml); mmu_notifier_invalidate_range_end(&range); @@ -1685,6 +1692,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, folio_ref_sub(folio, nr_ptes); add_mm_counter(mm, mm_counter_file(folio), -nr_ptes); } +unlock: if (start_pte) pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl) From patchwork Thu Sep 26 06:46:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812890 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 794A0CCFA07 for ; Thu, 26 Sep 2024 06:59:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3DwHdqVJkGu7HChE1YzlDZaDZvpQCVqvAB2HVgACkHs=; b=4MfFxI0zXps0ZxZ3EjyHMuNk/3 YwaHILPHri7s6LoqbZYK0QxQprQsHm+J2mS+sObAIFt902ZDykm3FNfQYsC56ShjUpUhqdq03vzB9 2xt4oY45Q5Upihqgqr8AOSHARXhH+8QehOa3L6pVjg5pZXJGjrznIIYaUPzwLDyekvOYm53LAR/Tt Xu2prVwiOseJQQWDVYG5sXSdT1ybJ5ygQFmjtbRcMRDhVrWLjp9Oid81I/W5Ko5wbL9BB7J6da/4m jOqyVpSJk9IJGZPeZjMFMTlEIbERAR7MNDSi3Mof35xyzXPzrzzuc80tA7gqLHt7cRfOLRIWswFBy TdxH5vPg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiSl-00000007QM2-18j8; Thu, 26 Sep 2024 06:58:55 +0000 Received: from mail-qk1-x729.google.com ([2607:f8b0:4864:20::729]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiI2-00000007NaF-1Bh6 for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:47:52 +0000 Received: by mail-qk1-x729.google.com with SMTP id af79cd13be357-7a7f8b0b7d0so34240785a.2 for ; Wed, 25 Sep 2024 23:47:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333269; x=1727938069; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3DwHdqVJkGu7HChE1YzlDZaDZvpQCVqvAB2HVgACkHs=; b=cpboEdH3rBEd4AkuinfKjCoIptNyB3usKxkIVM5vv6pOOdgGGre/npl4cZVqWL/aft O2+sw8H2ooaecGjt24/ttbtSZxtLoHVseuGaUliz3E1XFZIhJTJr66aNJAP6pHHEpOSr dyuJFtu0H30FyqogMkR5CgK2rQ5JSA5PswYk5c+KxR4G7xCFGOBncXZ02ds7y/s5/2b5 3aZ/eVW+6WrP5//OEycvttv3GYEYEDkAGWRaYWKHFKFk0pUiuC58TnOwgfxB1CfV6vlI Bswi3dP18wZLWJJXEncC6gXfBoqKsONQ96IUsAm7Lph4IQ3GXumvNjjT63qijtMc2aGu HJuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333269; x=1727938069; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3DwHdqVJkGu7HChE1YzlDZaDZvpQCVqvAB2HVgACkHs=; b=oMaCw7sqeZMpNSuF+mpnLzjm5RNMm1nj03wgcajg4Y4k9nvVNrG83YFyZ2a8OhtYb/ /kORGUVKVkMqO0iUTbCLuR800n6B2NESP+wTxskOYJMLivFvUIiE3qGGu7DgKgEtVwN3 khULRYky91nA3JxV+AQ2m6aNw93j+quWXPBrYhYDWSP8+xmpGsPp54OD7g5gOmmuV0ml Blil4u/D4CMGyxnhorU8HU3fazxAbkOsYFVs4ZrUisRXr3Pz7liWSIZ0N3dQfwg7iZpB KtEoYFM05pg0CEvsmOFzh27fLBP5uJpjkaAJLCcYlBVWl+CspMh7yOawmsgDbsZuahOj CXqA== X-Forwarded-Encrypted: i=1; AJvYcCXVRNewBt751if9LPWQ0pWV5Q6SVfSV1dh6ANvKfrT0WYGglrTpM613VCeaBJuDoug8iZoUx+i9KpioxP1NYxK/@lists.infradead.org X-Gm-Message-State: AOJu0YwZQ/KFUQpk91AQkmWTqYy2pMP3nl5OlS6/gVf3KDchFpkkFeD2 UNLFMPuq6+cwOSMwUyYVod3p1rmpiAojLbE5RJ7J2k/NAqzInmMDIWHYLbDFSkI= X-Google-Smtp-Source: AGHT+IGf6yhFgM7V9dSkmNMD310kmtT4NpPr6tml7YSLsMbfjuBNhYOZWImI0+hdwHwJZd31DH/Ogg== X-Received: by 2002:a05:620a:4009:b0:7a7:f18a:e46f with SMTP id af79cd13be357-7ace744e486mr712325485a.43.1727333269291; Wed, 25 Sep 2024 23:47:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:48 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 08/13] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:21 +0800 Message-Id: <9166f6fad806efbca72e318ab6f0f8af458056a9.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234750_499960_68F867F3 X-CRM114-Status: GOOD ( 12.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we already hold the exclusive mmap_lock, and the copy_pte_range() and retract_page_tables() are using vma->anon_vma to be exclusive, so the PTE page is stable, there is no need to get pmdval and do pmd_same() check. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/memory.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 6432b636d1ba7..c19cf14e1c565 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1086,6 +1086,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, struct mm_struct *src_mm = src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t dummy_pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret = 0; @@ -1111,7 +1112,15 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, ret = -ENOMEM; goto out; } - src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * We already hold the exclusive mmap_lock, the copy_pte_range() and + * retract_page_tables() are using vma->anon_vma to be exclusive, so + * the PTE page is stable, and there is no need to get pmdval and do + * pmd_same() check. + */ + src_pte = pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &dummy_pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret == 0 */ From patchwork Thu Sep 26 06:46:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1A8E2CCF9E9 for ; Thu, 26 Sep 2024 07:00:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xHtLGGTaDvd+j7nlO+D4nlmKtDmH5wZnqdyAY1Z6B8s=; b=1DwRp1pP1KgPzL0pKS4ByEElAD aSi+7nx9oi/pb3TK6VTYdc4QBbdw4N58jKOHxMc6o7t4cIw7Xgmv+8A0nTxZXeK9tgXGvCgGS+BgC eZfHH4vCbnk7CgdmG3FJ69+BrdA51pZ9bIjVPIe6XURFzRmK7STcohFuKQFUa7Hct8KsicwHLxhvd W+T5tj5YeV5iaErq/G/+BH1FC9qdZ/KAh3RUn/59mm/PHDyb3xR7udkI+g4rvcYD1tmW1ILdPmX73 p+fNFBjbAWPkN+LDsWUv/XnLSFydOCki3pUXx0LRxHCT0ULi7LiAIXqC28snVIq1t8Po7sT78aFZC QAa+ceEw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiTt-00000007QaV-3CMb; Thu, 26 Sep 2024 07:00:05 +0000 Received: from mail-qk1-x734.google.com ([2607:f8b0:4864:20::734]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiIA-00000007NdF-0EBv for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:47:59 +0000 Received: by mail-qk1-x734.google.com with SMTP id af79cd13be357-7acb0499ffeso54904585a.1 for ; Wed, 25 Sep 2024 23:47:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333277; x=1727938077; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xHtLGGTaDvd+j7nlO+D4nlmKtDmH5wZnqdyAY1Z6B8s=; b=BWOFC28jW4CxZFofIUxQI2fQoIS9qlXguFb+AoFYThdY7bITEeQGMZ2qXXmGQndt8F GuNeZyHGUyySm5OwZhj6RmMnMMd8UNPecY3YVq5UTIbhRHQq27Duxm9tuvz7FKAnDydm AaMbGc6mhLx8iu3czb4+VMg9mky7L85lhQVE66HNFOSmJC4nazOCUFoekRbANerIlIGT yOccRopLBMYbkEoLe/SrmDHwhf+YQf9NDvl20p0d1YtqtJZABcb/PyygB4Cmvun+o9q7 UW5pJ7LKzfnjDqD0EPNutlgSr3iISD4kfl2JLeYMSC6lx9ui1x6I5p0liflQJSE1d692 rDAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333277; x=1727938077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xHtLGGTaDvd+j7nlO+D4nlmKtDmH5wZnqdyAY1Z6B8s=; b=fkfnjv6B3yR2zxl+n50LqMqJZrRZsENU2V32TEXAipcBPu1vJ+qfJR5CegUmaUPR4I XcRLwhb7nKcSv1DIr0D0nteGEAJyo4dCi0qFDVL459jG1GCbPjVz/9gRPj6XOzHdgAEP txq7OXLlpJHlKDwOMlEIFDUGTX+gegGBVjS7z+JcG0P/2cGg/3UdvNYlpSlXzoW5/vGg K7rSvoci+Ref6Tbwshd5XgiUWSjeippPPN7ZK2fqQhpyfaFMRb1YX6IegfRum/J1Pfoc mTRQAP/4Q9Z0VqUXZxnuGfz1tdYJSl62D1RsnrdyWqe4Pk4E1CHOMRwc8GAllOQq3t8Q TmzA== X-Forwarded-Encrypted: i=1; AJvYcCWlopTqCcHgCGf55G2ePB3JaatXyCQRdy22wwCL21agCGxh8GTRtLZku3pI4qfD7QrDyUrYBb4u7SP1tiRWSLbo@lists.infradead.org X-Gm-Message-State: AOJu0Yx+0iCzcUpTyxdTlE8KQCC7GyTM+Dr/+qrvHQek134U5PVLgvza +tJDinhWr7nwDS2mVlf6T1Nyi1TXDSEW78rLkxiAgcxUCpZ2hPiTtTteRiZox0g= X-Google-Smtp-Source: AGHT+IGbo2MSjNAvLC3c2ApJAOpwKD09S+YUwBzMfKcY13i9YN/E+PXjNs0XRR1eOHqqnOUdrqUWyg== X-Received: by 2002:ac8:7d42:0:b0:458:2ce1:26f8 with SMTP id d75a77b69052e-45b5e045ea9mr77074431cf.60.1727333276776; Wed, 25 Sep 2024 23:47:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:56 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 09/13] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:22 +0800 Message-Id: <9d582a09dbcf12e562ac5fe0ba05e9248a58f5e0.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234758_109394_5DB6CC9A X-CRM114-Status: GOOD ( 12.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). Now new_pte is none, so hpage_collapse_scan_file() path can not find this by traversing file->f_mapping, so there is no concurrency with retract_page_tables(). In addition, we already hold the exclusive mmap_lock, so this new_pte page is stable, so there is no need to get pmdval and do pmd_same() check. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/mremap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..9dffd4a5b4d18 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long len = old_end - old_addr; + pmd_t dummy_pmdval; int err = 0; /* @@ -175,7 +176,15 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, err = -EAGAIN; goto out; } - new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Now new_pte is none, so hpage_collapse_scan_file() path can not find + * this by traversing file->f_mapping, so there is no concurrency with + * retract_page_tables(). In addition, we already hold the exclusive + * mmap_lock, so this new_pte page is stable, so there is no need to get + * pmdval and do pmd_same() check. + */ + new_pte = pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &dummy_pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err = -EAGAIN; From patchwork Thu Sep 26 06:46:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812896 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24729CCF9E9 for ; Thu, 26 Sep 2024 07:01:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RsEgu/iD0T6JQDFamAgFHiL1Lm0nKVk+wAfYQhZRThE=; b=i69DTgT3X6V2mF3HtzptmV6Zrj S9MTPmRQ029WkvNfGHRl2oVeZA1+yYZN5pCEatchTnO+Byc9h03wTAnIocs4mVpwFVyLhDXwzgd69 5fpZ7uvw3majZX60rYK6hAwXoIaMzyJNPaqDQLkGc/ZKWAfNXvl2uXfvru9w+5nVNO367Zu730+qa TBAp9AUMcXBw9k1eOSI5xmm+0kawL7wvpbWX/ssfzVMsLC5X1Q37NCbRZ46fj8aN9wxkypMMkZo+e q9UFceSJlMJlU+eO70OZW8p62w+BtE+Pm/GC7AJcxOPrlv72EpOpqJndMg9teeUZMRpt+wDoh0XU5 ktC7Q5mw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiV2-00000007Qks-22mA; Thu, 26 Sep 2024 07:01:16 +0000 Received: from mail-qt1-x830.google.com ([2607:f8b0:4864:20::830]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiIH-00000007Ngi-0c4S for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:48:07 +0000 Received: by mail-qt1-x830.google.com with SMTP id d75a77b69052e-457ce5fda1aso4651741cf.1 for ; Wed, 25 Sep 2024 23:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333284; x=1727938084; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RsEgu/iD0T6JQDFamAgFHiL1Lm0nKVk+wAfYQhZRThE=; b=fJsAM0EQ4f8nryWPPM8I7A7qWly5sj5rCT83I3gPFdZ6B8jS4+HhY/Aps7eWK6P+yc /zLkmuz/AilRw5m+wz/8kD35eVDUiL/OsoxwXhSlepuJBt2sY58tl0B6S+PuU05tMOfM MYGWAIpk6kPhgypdohLoUqwAgdgPhaXjh7e//ltDeGzaDEoc7KpLrt6JdwfrsocF/vzu BR6ycCw/HmhQVLpuXKwDUUGZxc9Ah+eqlWhCp60Fe+jMa04x+f3PrU5Ex0vYRPacEaTg TvEYMDAQ8HVSWfcmPo3gUPrTnO75GTlt3ulslPO+XWfdGoEs+JyNfp8AwJmBkhjTxeTR 9dUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333284; x=1727938084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RsEgu/iD0T6JQDFamAgFHiL1Lm0nKVk+wAfYQhZRThE=; b=dUupoODYrxHUnt6CO2hP4b0LPAsHDAvZcA1U1jAgkGykm9eckH1TU33PLZMRsXrUsJ zZui9pY6FgJ1VlqTYA1CzNbAPpWyMlD4B6aCrWONrz0hcUL90tLpaSuLsFdF7GJ7JfB0 GQNbaatatVeGjR6rslQUFus1QpuPEGe+nfIjDWGfaDyhUv64Ms1KtOhbe16UNnkh2SYo /hPnNGJq8CIle1Aesc6q7ZyolghgWa7Rl1MFp/oCOFYYpmhUjIcADEKjddVDXR7pXjPI g/WXiVsW98wIwJjGVNjoInl/b2frNdt49d1TLKiMYQnPFXZAGeHxB542ZuTZRGW7L15L 4pgw== X-Forwarded-Encrypted: i=1; AJvYcCWZjaefyQMgbv+4JpviUyrrFnV5VHDcw7OrAYxrBIxUD4NDLvl5hpkq843PsSIx5SVquspP04cq7Jg9OrWM/kDv@lists.infradead.org X-Gm-Message-State: AOJu0YyyW+pF5gu1UkfKIzUPr30z6rPCJYlVyylxORUyFwNsM7se+Bjg Z6DR8AieHuRa5miIdeFlM+QecPJT8AKoBZw1vXmSuPpfTyCkbWPD3LBH2uZHyn8= X-Google-Smtp-Source: AGHT+IGAlBKd7aKCt6XxIcAXM/Sjs0v2+e+7/droKiEQYGe4LhoKPN6bWAepHInU8bkNuwlluGjQYA== X-Received: by 2002:ac8:7d4c:0:b0:458:4aec:2749 with SMTP id d75a77b69052e-45b5e045e56mr87889061cf.57.1727333284339; Wed, 25 Sep 2024 23:48:04 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:48:03 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 10/13] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:23 +0800 Message-Id: <2620a48f34c9f19864ab0169cdbf253d31a8fcaa.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234805_262097_381560C4 X-CRM114-Status: GOOD ( 13.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/page_vma_mapped.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..ab1671e71cb2d 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,7 +13,8 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; @@ -25,6 +26,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) return !!pvmw->pte; } +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,8 +34,8 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte = pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, pmdvalp, ptlp); if (!pvmw->pte) return false; @@ -67,8 +69,13 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) } else if (!pte_present(ptent)) { return false; } + spin_lock(*ptlp); + if (unlikely(!pmd_same(*pmdvalp, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, *ptlp); + goto again; + } pvmw->ptl = *ptlp; - spin_lock(pvmw->ptl); + return true; } @@ -278,7 +285,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -305,8 +312,13 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) } while (pte_none(ptep_get(pvmw->pte))); if (!pvmw->ptl) { + spin_lock(ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, ptl); + pvmw->pte = NULL; + goto restart; + } pvmw->ptl = ptl; - spin_lock(pvmw->ptl); } goto this_pte; } while (pvmw->address < end); From patchwork Thu Sep 26 06:46:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 82CB0CCF9E9 for ; Thu, 26 Sep 2024 07:02:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=FXXjBipcGMo1ubnmEmrqRtJVUO STe2EsyDZj8olaznZXEDwApfaHOqJJFlDwtcKEg0cjFS49I+n6DeZ37kLSLCMwNQraSnyS3p8meVQ WbAwnL3bsfju26yLCHTV+uzXKHewA03bsRsY9JF2+YL7HHJCr5BBavSjVB+ml3Hvn+mF37kmuKptO zHE2JSSxwg3rNDyKGLheFlDnYKapO99phzHGpNzsl5zYa5d+oyr9zcCI3j8Ea5qxm7HruZdPYPRZr qlYWTCB0A2m08VQdG8tAxcvKsy+ZysAC3CchKJlRmBN/dRsujYm+EUscVa9ObhW8OpVax0AbCTYU3 rH/lQ8Jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiWA-00000007QuQ-1CF7; Thu, 26 Sep 2024 07:02:26 +0000 Received: from mail-vs1-xe34.google.com ([2607:f8b0:4864:20::e34]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiIP-00000007NlO-2HI1 for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:48:15 +0000 Received: by mail-vs1-xe34.google.com with SMTP id ada2fe7eead31-49bdc6e2e2cso272883137.0 for ; Wed, 25 Sep 2024 23:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333292; x=1727938092; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=kLIPbwiuZj1SUrdccigASkz1JMiUy5JdSspYvAKzldnJ+SGSYwm16rPde1CDLfVtWp QEnTj6WvhlPrYbpZKBV9Ap71r4D7Q6HJ8RqGWr0ORRLiI7enjcfpNixuLbk3ALWPZQU+ QXUti00f5bs1eKe1j6LQXwSLuDIg8utfmzP/Bjvx5S8fp5UTMF1iLV2elUIleehiynKY olwddDxYtmj3vlbRkLuX37mXhsyL6dBZvLrBKOHMFd+HGnFaPoBh+an5HAnmAV1SoIML lbv0OsnqeXKXtxkJGdlXFmGK+z8YPETEWSeB+XC/3M6XOFfv9i51LQvrpdXQEwTA8hg/ Z21A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333292; x=1727938092; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=J4Q6dAaEQ6XhBNhGFqQUfO67u3hByNh/4X73ag3lruSuW80WHPKRmmX2rSgcJljTV8 aKpTokrqomHucRhhesoH4rWWxbS3ZEDDxRG+Z3d3DsoEqg5AfXRFHHWYlZkDHKm+iCRL 4xD4s+t7/qgekh5j/TG834x+LzQ1wdDUwynOkXi+gvAr5oOWX9K6d13y6j3Ht8V/pztm KccSWW4PQKoLTqtAOhuLAJGO4aTQMEy/+GAIKxszWT5Sem5KOqC2nSaW18w6SPKenr0g KQmTcai7Sk26ixsUunFUxEC6I3r89qyVHa6jR5ax5nzAqSkJvTkyig+BQ+VcT3FpfmXr BzEw== X-Forwarded-Encrypted: i=1; AJvYcCV3s7SaQaM0kLp9wL39XP1kMAZxcLrvdvi5XfslKKFz01wSd2KucfHwlQP34g39qj/Q/7So/64ltBdaOocYRM5M@lists.infradead.org X-Gm-Message-State: AOJu0Yy+GwHIvrLKAlSqIJetWl6U/6Rvbo4wBnF0/GdregR3BeyTrla/ AvZvtQyixtsb+Tqk0va6E/gpwuxMD4DFYC6Vz3dMAK6/+43dqlF6kpJEfep8U3I= X-Google-Smtp-Source: AGHT+IEjiufoSG+9f7J9DudEHL9YuIq0UaJx7YZ52pOk+qHYx73DXmea6YZQnzcRnEEhuuY+i+vKHw== X-Received: by 2002:a05:6102:e06:b0:493:b055:bf56 with SMTP id ada2fe7eead31-4a15dc39965mr6802341137.2.1727333291940; Wed, 25 Sep 2024 23:48:11 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.48.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:48:11 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 11/13] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:24 +0800 Message-Id: <1530e8fdbfc72eacf3b095babe139ce3d715600a.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234813_606455_C860992B X-CRM114-Status: GOOD ( 12.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will use pte_same() to detect the change of the pte entry, there is no need to get pmdval, so just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index ce13c40626472..48b87c62fc3dd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte = NULL; pte_t *dst_pte = NULL; - + pmd_t dummy_pmdval; struct folio *src_folio = NULL; struct anon_vma *src_anon_vma = NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte = pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte = pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - src_pte = pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte = pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); /* * We held the mmap_lock for reading so MADV_DONTNEED From patchwork Thu Sep 26 06:46:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812898 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B4B6CCF9E9 for ; Thu, 26 Sep 2024 07:03:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=wKWf8ZYo9tlbP3964sEqp7HhdDy32FNbDdGL8iSt+Ok=; b=zc0Lcu90PxMsGh+69POAZg+Yzv DADMmNHMGLteqGfD/v63ny34CJGxBF1eR0XU7lOISYVVekh0alpb/1TP556W/wNwVsRJ5z0QCcXfy F/VLPHvGU+thl8rmDpawXYDbQrlJ+UFwFMPxu61Qzwc49F7bKcxuoNTjI/NzHnvmc12HtiNeleQVo nr0q/L4zk3ODycRnFRb1TJjRx8HwXB6Wf1i0p409XKqPVsAK1ofZ4YcspWryt8BKQqSqam1KXxMeG 87IzHimXPVZAiId0XhC9ux5IXQV282hN1n7vkxkFh+ous9ZJ7I9LIXVYLcUPak1w0sDcxw6IxJW5t nomq3pkw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiXI-00000007R1A-2xsK; Thu, 26 Sep 2024 07:03:36 +0000 Received: from mail-qt1-x831.google.com ([2607:f8b0:4864:20::831]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiIX-00000007NpG-0CQ7 for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:48:22 +0000 Received: by mail-qt1-x831.google.com with SMTP id d75a77b69052e-458362e898aso4398991cf.0 for ; Wed, 25 Sep 2024 23:48:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333300; x=1727938100; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wKWf8ZYo9tlbP3964sEqp7HhdDy32FNbDdGL8iSt+Ok=; b=jMGDMYS/1X8lIOjF1pjE41c5QO8x0m1YVGgoKq3lEE+zGi9ulQoaC0MlEOHSKmTfBW SB5IAlSYJLwBAmD/2D1vfep0HkhbpIl7j5BPXa5oRDBPU0mp0TQslA0sNlbcxDazHdTY ot9hAzZqngdVboh41EgMDgvmHeia8zL8lQlsMfkvXTJVz+JrqzBwwsphEy8Q0avVjqpp QN5JAUXs78hXkuwLKQNtOXtD0tVdR1PDkFApxXoHLuJTfj20o1PRp4MHv84+p14MsceE htMsp+LbQ5mgerLospNUFgsbAoYK69r4luln5GeU135gerCofNTIXLr7RA6K9TS6W8pG wMpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333300; x=1727938100; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wKWf8ZYo9tlbP3964sEqp7HhdDy32FNbDdGL8iSt+Ok=; b=GNcdl1Zf3J3ZQIxeG4WSz+Cg1bkSsvPmfQh/i1ltwr+56vAAVupox6NoszaSKlBuB9 xUNYpr/LxkdEYhURPsywbwuLROyOeabP5bmbFox+/bMx7KdLcZ2+Wmdw194zn3+RXTR1 0D8JbY+O+yJtrumVBJz0DELRKL/xqtOcqlLXMp6yi37Ew8jyc6av6JKIhcfgAmy2d8Vz RRkhObjepSV5g4uioUkMzPqKme0ejjreh2RrLeeGyI34LxflzS/dhC875DbYixcvJAq/ bf6aNRoao+fFM8OY4pbSDHHnjB81J0rHBKwHt5e1hexLEvZmIHcha6xGGn/XUWiVDXSZ oriA== X-Forwarded-Encrypted: i=1; AJvYcCX0oVTZaScGLHD1wGBnf0w5QQV+q17Mz3aoFAv+96SXSGMuWZ5heSkDLk+jEnCcHgoQ1Ik9umxh/8aGxKAu8htE@lists.infradead.org X-Gm-Message-State: AOJu0YyKa4xfrQFJaJx3PtN0yBpSndPAxhrHPCcWinZr3TVVLkYzWAZy dNM1M/4NxOrz6s7xxupAtShCbuseNiFUCZ3DoMyvLnPh5abBgvRGvq8ZjxFc59Y= X-Google-Smtp-Source: AGHT+IH0PykjARYOlhRdYD8Ei00ntSXkZvlp+w/DvL/wK29HFMzX/PK1qWz6iq3HMQhndU3PBLr5iA== X-Received: by 2002:a05:622a:51:b0:458:3e20:65d3 with SMTP id d75a77b69052e-45b5e06627amr81705351cf.60.1727333299714; Wed, 25 Sep 2024 23:48:19 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.48.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:48:19 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 12/13] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:25 +0800 Message-Id: <7e9c194a5efacc9609cfd31abb9c7df88b53b530.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234821_168784_4924DBE4 X-CRM114-Status: GOOD ( 11.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 749cdc110c745..bdca94e663bc5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3375,8 +3375,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + pmd_t pmdval; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3384,6 +3386,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, return false; } + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { From patchwork Thu Sep 26 06:46:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13812904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 20BD6CCF9E9 for ; Thu, 26 Sep 2024 07:05:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XK9xLAd1aeWWLl9nHV9gcamA18VQvE5iTgorpb3Ql3Q=; b=s2T42Qt4JA+sQH+O0Y8tN9uTup vWjvgS0G7VjjWZD0bZBSS81shyc6fNnOj1YQno2eax5RcdlL6T8X3pycDGvru8/SEY67tU1OWHOIP hX3dVr9d3sDKpzMeUTEEn/WP5mRLK6NNpENMeXvHEPcs+LOLYUhZAqQkNWIL8LOx8wvCmwf2JZaIr e+86hwwNP8V8KUdxe1mrM5ni/4nv9qfB8W6iFNIWqw2bT5nR8MYlOxDZ8VIZq4Hb5p0nbXlgrBosa TksszgVRI63NPmbJQbH87B18wB2+9+KBaF70xImvjFrkZZR+rA3y7Ocu2UnA8P/FWerSANM534GtL ZJJuRq8A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1stiYS-00000007RFV-3Gdm; Thu, 26 Sep 2024 07:04:48 +0000 Received: from mail-qt1-x82b.google.com ([2607:f8b0:4864:20::82b]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1stiIe-00000007NuJ-0S1G for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2024 06:48:30 +0000 Received: by mail-qt1-x82b.google.com with SMTP id d75a77b69052e-4583209a17dso5030971cf.1 for ; Wed, 25 Sep 2024 23:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333307; x=1727938107; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XK9xLAd1aeWWLl9nHV9gcamA18VQvE5iTgorpb3Ql3Q=; b=FZt2U1Jb8PxlFuH0vSNSoTVUYxw1I9IqSGlsFLn/Cg3t2MFM7iM0I30qsRXjuXSC9t RzMIgthS35GafcT9kyIyz1mbRq8HNawO4pnLhsNKOmMxpLQ/gS9s47O0hJRsKbG/gu6I B5qVvPrRHDXFI2td/Ph8o1WdgrHr3lGJLE33LbivRWm6pjR543nlfEe0UUdNAAl3jdGq ArJ3fycMDMBgPG0Mv5M8xNsBtiVEhv8gRe+rpzJ81BntHICUz6lIV/dyaN7/WbfPCE5f jSnWn3cLfNh8vN+TsRgvc1DIJzrAwU2KbfVwTY8JqCSFE12agyg0AKklUN+9GkfyPF4T 0meg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333307; x=1727938107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XK9xLAd1aeWWLl9nHV9gcamA18VQvE5iTgorpb3Ql3Q=; b=NuRcrVtGjxO6/rJEZeyuMLc0BGCF1sSCDXPmIE3ATBiTX56a0Z00AV/WXkH3O/u2VR fur3avYNA0YDrc7tvExEMc625MdX2h4jonKnxB+x2tMosSuaUWZ/+/j8z4Kws/HHAbdn Zg8bIr1f1mOT7XZAEr7VgzEVVnKeUULiH3LgaJKq/QsR6Ok3ikMTZVHmtQ7KBHIDLu+N TPSy/0+ngtb+Gilx2pg3hfsB95i7aXqzCA/77W6SUikST7P0T4tIRhGRPNMirY4F05hJ XdrpACMdgMQMreVxFpSbI61bUZyn0c6eHwqXZhNFBmZ+5Z3DDmzyvgC/Fu/ixv2kRWwl frYw== X-Forwarded-Encrypted: i=1; AJvYcCVChiLwlPdYgaHf7hkmTDTBYihXWuNCVz7RvFykC9sGUC4HsAVDpLd7ay1spHoNvYlVciCJfSgwihg1Fe0Bricd@lists.infradead.org X-Gm-Message-State: AOJu0YzbJX39+U6Lb0zh8Of6YJFHja9uerckPAX3BXNrn6qj8teoZeGz GTI8/VoBTI8WyqbdL+RUW/I6ovK4Zb4rgLpRTVseHQ2Rl7oWuqcHdeGPDECMDAA= X-Google-Smtp-Source: AGHT+IHnYRThLjQtuqQ25g++f96or6jGXpOGKBOHjBYtOEzyfC2Zv3m6NDV/6g7eYEpePNjEvQY05w== X-Received: by 2002:a05:622a:2996:b0:458:37c6:46fa with SMTP id d75a77b69052e-45b5def2036mr66535021cf.28.1727333307294; Wed, 25 Sep 2024 23:48:27 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.48.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:48:26 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 13/13] mm: pgtable: remove pte_offset_map_nolock() Date: Thu, 26 Sep 2024 14:46:26 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240925_234828_190406_BDC5264C X-CRM114-Status: GOOD ( 11.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 46828b9a74f2c..60a55b28cf679 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3015,8 +3015,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, return pte; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index daa08b91ab6b2..5297dcc38c37a 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) return NULL; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte = __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp = pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -372,15 +360,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unlock()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); - * but when successful, it also outputs a pointer to the spinlock in ptlp - as - * pte_offset_map_lock() does, but in this case without locking it. This helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spinlock - * pointer for the page table that it returns. In principle, the caller should - * recheck *pmd once the lock is taken; in practice, no callsite needs that - - * either the mmap_lock for write, or pte_same() check on contents, is enough. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); * but when successful, it also outputs a pointer to the spinlock in ptlp - as * pte_offset_map_lock() does, but in this case without locking it. This helps