From patchwork Fri Aug 9 16:08:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2A729C52D7C for ; Fri, 9 Aug 2024 16:10:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fKim6v00HbGyszP90Y55Ixe8BGOuNlK0Kt5OSqhmIXk=; b=tuWGBoeRvlUs9TIFG/N7Dbm9AY Oc+aseI2FSPOw31ZIC8Y06I3nh4CBNy8x0spMLWUdRM5eCCAxjGM4NdPQQ9bMHwNKrLzrGNKjVny2 pB0F75MOmnUjbX+jBX6DJHXJ0GNMGh+Jcjqk3Okc0ssoFAs8wPe3igyqDjxy2z2r1INGFX+v9sVIE KixlerB389o9c5dSrkmG22h6uq7widZVHjpoxxDdpcF7dOqgGta3aL+qzBWQjTHoLim/KnyikHyzv ASMO/lYDkB0eB8CA+MzpvM/LSdGsmq2CsVcS+bB4sxymRe/6Pc1rf+XJ5ssdllvXrk0goq3xvUBZq /08Lq5Jg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSCK-0000000BpuR-1Kgc; Fri, 09 Aug 2024 16:10:36 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSB5-0000000BpIe-1pEi for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219758; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fKim6v00HbGyszP90Y55Ixe8BGOuNlK0Kt5OSqhmIXk=; b=WQmcrt4GzDvpomcCmmo5ToSnVO+INctBuXCOUQ3auzuvhxMYhACHwmP6D2GBDboupjq6HE YuMgs/UHQaUL4xWQ7A+X9qwagzP3gN93+XlfwJxYc29WzvqODbkXjw5/QmLXAhM1NSAqds lu+LyExRnkakYClNc5vUpHYUh4HJCb8= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-384-vHflURxwMHezLJDBrXQVwA-1; Fri, 09 Aug 2024 12:09:17 -0400 X-MC-Unique: vHflURxwMHezLJDBrXQVwA-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-44ff44e2f9bso967811cf.3 for ; Fri, 09 Aug 2024 09:09:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219757; x=1723824557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fKim6v00HbGyszP90Y55Ixe8BGOuNlK0Kt5OSqhmIXk=; b=jdZ+aPu4xngZxtuKtJzPsQRGYvKg335WDHa+6eY6YX9WxbFO100dJiYKWkMrfap98p wmuc396PylSuzdIR5POWe1BP5PEkAjIa3WGe+iaIRrwAEezH+Ysld+7k6NjJEAqGOTm7 qTAlzLs7zEvZ0nzQTxIQU0O05nmdYzADdqQWaMwwCaU+sy1Kli9SQZMpU1f+UqH/lXZO w9jzxq1W8+MbWPf7n3IC1OsTDJRcE0N0Ry3RjPYooYAxu4cT8YJEOqRItDhvNEjxyR8p e7hUYTLxIZPuhBGTf+GsKKvuFNCXF21bdK4CVpYnUqyOK995FuzsVCb36rv0+JKE4/xv hCwA== X-Forwarded-Encrypted: i=1; AJvYcCWqKbSlpQ+jqb9HMV+8PlHocRR9RUGukeLeXGffnL+yi8IctWmbrq1z9NlMqD3cYue/pZWsp5X5I/WaSB7EN/Tarsn8E6CeJRY/WXASyXg+4nXE+R0= X-Gm-Message-State: AOJu0YyjWpVnwHC4W66yFVoIRGUcYruE+86eU36gyZYpg7HjuiQ3hwuC KUZ6txx2FFvWe/HggFYVQ9q+fZKrcYIOVVN+17I9o+rUMz+CWcYSiNbf29x///JobelK61+mLIG 7Bb0mu+Rqk3oUCOoWC1yubMt+DM8ZID0v2xKpWDIrbAvhs3DNJwnQAQSr0kGY6F6myMm+B2eR X-Received: by 2002:ac8:58c4:0:b0:450:1e5e:3f3f with SMTP id d75a77b69052e-4531255bb38mr13611111cf.5.1723219756653; Fri, 09 Aug 2024 09:09:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEAN0G3ozf6o5j8NgDrSR8g+6fB/Zkz8yhBLk8I4VWUl++aRhUduuiwR699bOA3QLxmHpo6kg== X-Received: by 2002:ac8:58c4:0:b0:450:1e5e:3f3f with SMTP id d75a77b69052e-4531255bb38mr13610781cf.5.1723219756198; Fri, 09 Aug 2024 09:09:16 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:15 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 01/19] mm: Introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud Date: Fri, 9 Aug 2024 12:08:51 -0400 Message-ID: <20240809160909.1023470-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090919_666518_9F6D3FB4 X-CRM114-Status: GOOD ( 12.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch introduces the option to introduce special pte bit into pmd/puds. Archs can start to define pmd_special / pud_special when supported by selecting the new option. Per-arch support will be added later. Before that, create fallbacks for these helpers so that they are always available. Signed-off-by: Peter Xu --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/Kconfig | 13 +++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 43b40334e9b2..90ca84200800 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2644,6 +2644,30 @@ static inline pte_t pte_mkspecial(pte_t pte) } #endif +#ifndef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd; +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifndef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return false; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud; +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + #ifndef CONFIG_ARCH_HAS_PTE_DEVMAP static inline int pte_devmap(pte_t pte) { diff --git a/mm/Kconfig b/mm/Kconfig index 3936fe4d26d9..3db0eebb53e2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -881,6 +881,19 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE +# TODO: Allow to be enabled without THP +config ARCH_SUPPORTS_HUGE_PFNMAP + def_bool n + depends on TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PMD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PUD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + # # UP and nommu archs use km based percpu allocator # From patchwork Fri Aug 9 16:08:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758964 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8D6AC52D7C for ; Fri, 9 Aug 2024 16:11:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xVA/urUlESz7em6kHS5QV39/c1UeLmTRXD3YGysY/mU=; b=w/iUgcrn2kBSLpDU4Qmo3rzOHI rxps8J1mNJrOOYuDbNDzCui2RVV/in9KwgksNdQE/TsxK0mMMB1S67oxhU5ILCetRnGlzeur48vzA yfAqiICDfcCv43plR9u9KPsYqWLdy24SsDQyY530pHzviewDy0mHPjGRVIFU/GwgfKedD/IpSkkRW 8k0PWpvXpuIAbMuX1+fUNnKhjwO6kAxq7qFLjZ1vRFRmk4tqUKhBRf3RaXvdGLczmAuQns4sSkdFb aIsE9fnee2pofrvRMgJBxvL13PbPWXPk8pVWcYvA57RL4L0DhhuCtmAmc+zCQ5BytyNhL6/Zwv9It kybTXYeA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSCr-0000000Bq4r-3o6h; Fri, 09 Aug 2024 16:11:09 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSB7-0000000BpJm-3zpV for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219760; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xVA/urUlESz7em6kHS5QV39/c1UeLmTRXD3YGysY/mU=; b=dTTtQIxpp6qNpBzypXaRlAUVD6r+soiTZNbxn969XXlmgjO4hPaM/UUZ4T1fkp91Cq1Fyr QDCsn5DXuncicmQiJ6d2clPFrJPEOFCQu7Y7k5lKitx4BeYx0HeGm4A6opE4dA3raAaYJP BQPGSWIMdBFUQGV4LMid+wDEMo6MqPs= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-237-g4yEv93FOGiAiRtUZAvt7g-1; Fri, 09 Aug 2024 12:09:19 -0400 X-MC-Unique: g4yEv93FOGiAiRtUZAvt7g-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a1e1828321so6878485a.1 for ; Fri, 09 Aug 2024 09:09:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219759; x=1723824559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xVA/urUlESz7em6kHS5QV39/c1UeLmTRXD3YGysY/mU=; b=iyVCnY/WJa3fCVG5d25ZAa7qiA44uA0JkFXVu0HBlN/7Ga0OjfcnhUp0mOcllNO/pG OGXPAy6nF+PIhzTghyg4yolhYkfD7I1IoEbYhxYZH6t0hm71od2FvJjxSfUXK5jnvIrQ 9ksyFlC6lR6C2dz2fA8wObdncRIiq5zi7kwHlDhBTDAnS0IE9+iTaXnDtVMn7c7Vjp+X TdaAohePKQckztbcq4e0ZDCAHk3joT6/WLEGKQ/5kIO7bsFDKaPGtSLrZgEGRCepe/Mc k5xvmPxmEApvkM/kSuSiDjVA2NLk6SgHRlYadcnm8zkQ/OrfKdZdsXenYDAz4XKWxEJJ X58Q== X-Forwarded-Encrypted: i=1; AJvYcCViYsOvP6HOX3sPLPuK9moO9L43WB8bRhFDVi1/dHVAx9sT4nBXWOArmHMDg8b1rWLGx1QuewuWQqwkMWWK/DgCqDMjx2wJrwUE7KDZBu/7l/i0SIc= X-Gm-Message-State: AOJu0YyBOTE43vhRmHKgGNabT0iR8R6qkSxzYt5VZMw19ewzdb4OKtUP y4Gf/f7dgHpi0Ah/Rjo0n9sLaNxcwnX4bPP59284tIFrnRZ/xxHUboMOlfIwwb1zA/NLo3+yzS5 /acdBJuHbJBFHJelijAbNkTWV7JIqVA/e9we9I/d+SIDmRIDwounckdYoWmFvQvUQWCCgWS1y X-Received: by 2002:a05:620a:2906:b0:7a1:3ff9:1e1 with SMTP id af79cd13be357-7a4c16af689mr138617885a.0.1723219759082; Fri, 09 Aug 2024 09:09:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHesXKAH7m1pKPVMriR5+O5cSg1VUsP7Em6MavFjtwcHLSKYNYbUt0UMVXsf+PUo1OYJsvsTg== X-Received: by 2002:a05:620a:2906:b0:7a1:3ff9:1e1 with SMTP id af79cd13be357-7a4c16af689mr138614585a.0.1723219758659; Fri, 09 Aug 2024 09:09:18 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:18 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Matthew Wilcox , "Aneesh Kumar K . V" Subject: [PATCH 02/19] mm: Drop is_huge_zero_pud() Date: Fri, 9 Aug 2024 12:08:52 -0400 Message-ID: <20240809160909.1023470-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090922_175744_D6694BB7 X-CRM114-Status: GOOD ( 16.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org It constantly returns false since 2017. One assertion is added in 2019 but it should never have triggered, IOW it means what is checked should be asserted instead. If it didn't exist for 7 years maybe it's good idea to remove it and only add it when it comes. Cc: Matthew Wilcox Cc: Aneesh Kumar K.V Signed-off-by: Peter Xu Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe --- include/linux/huge_mm.h | 10 ---------- mm/huge_memory.c | 13 +------------ 2 files changed, 1 insertion(+), 22 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 6370026689e0..2121060232ce 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -421,11 +421,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd); } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - struct folio *mm_get_huge_zero_folio(struct mm_struct *mm); void mm_put_huge_zero_folio(struct mm_struct *mm); @@ -566,11 +561,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return false; } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - static inline void mm_put_huge_zero_folio(struct mm_struct *mm) { return; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0aafd26d7a53..39c401a62e87 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1245,10 +1245,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, ptl = pud_lock(mm, pud); if (!pud_none(*pud)) { if (write) { - if (pud_pfn(*pud) != pfn_t_to_pfn(pfn)) { - WARN_ON_ONCE(!is_huge_zero_pud(*pud)); + if (WARN_ON_ONCE(pud_pfn(*pud) != pfn_t_to_pfn(pfn))) goto out_unlock; - } entry = pud_mkyoung(*pud); entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) @@ -1496,15 +1494,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pud_trans_huge(pud) && !pud_devmap(pud))) goto out_unlock; - /* - * When page table lock is held, the huge zero pud should not be - * under splitting since we don't split the page itself, only pud to - * a page table. - */ - if (is_huge_zero_pud(pud)) { - /* No huge zero pud yet */ - } - /* * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. From patchwork Fri Aug 9 16:08:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B38FBC52D7C for ; Fri, 9 Aug 2024 16:11:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pOevS25kl1+p5C9bnayjnL4IWCIfturozNfzs94vCz8=; b=JGItot10WKM3tMowxCntAUaO9O nnDFtbfeXRTW5bhrk+8ijEYoUMJYxaxPKYihMrfBbX2xkGO8x8RMg3BRnlMq2PddH7IRTSSw7GX69 7mdHxRK8bD+yLe812LOArOJFnJ6OrqH4hc9O7WpWFZ6AWp28kn64M3f8XMsYhxpo+nl29igMaprot LGlQdfLesukJL6k5r1EuvZqYb5fm5J7Q+VYwBML+UCvKEKiyYCrMiDKBYBEZTafHx2uoqblJHAdYh goVRFP2/S0naxsLIL6QlpP8hm1chsXsP5B1NhirrFgLU6v1cizEr8S1D6njfsFl1OquyOF/7wPJag dAPhBJqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSDP-0000000BqFf-2qlo; Fri, 09 Aug 2024 16:11:43 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBA-0000000BpLk-1GNk for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219763; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pOevS25kl1+p5C9bnayjnL4IWCIfturozNfzs94vCz8=; b=AIkLRe78Zxz80FdBVVF94Ok0iKX6PlppauloyrnCDhoRbT+1guSOkX9X42aP94brCJe12a NoqdEgbelGqZckfmJHl7mqnvKHmIQJjnd3miBCiJfR1Az3sUNGJUYHjyQuZRJrblqsWaBC 2AXDbRkHHI9NIA3Z6gxYJaHz4YaPMDQ= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-404-Zw-T__HONVuPJ8Ap6LJMrg-1; Fri, 09 Aug 2024 12:09:22 -0400 X-MC-Unique: Zw-T__HONVuPJ8Ap6LJMrg-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a1d3c02fcfso7767185a.0 for ; Fri, 09 Aug 2024 09:09:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219761; x=1723824561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pOevS25kl1+p5C9bnayjnL4IWCIfturozNfzs94vCz8=; b=tdDPl3Yb+YbH5EMnQAxpgdCKn3O0Fdd9+DqWRCZ0SFQ5JhOBM5N6lj4wM3Nr03EI1W thDKyUdAPkrOxoTkaTUrHN+cTNQMsx86SJWTU87+AauWuPfbJ7XfHvO+PSQrQ0LIw5Go 12W89+vJgPPiwNu8VZzvgQdYbBvIoc34sgAhIsLSoE7era+ZlVVGYM3H6e6chEXhzm1s ao8kc6nUXTWKvaZpzRaIz5d4X3vblWZMMqAbRGxhYkbKHVKrJAFz1kqyI9gyzE4S29ut oiGH1khawWhj39S1dVzNB73eRPTXS9xzkFGud+bdBhUMbroUDalnBNiS+ouCwkUczLFj OAFg== X-Forwarded-Encrypted: i=1; AJvYcCX3jjOqSpZHmlCZcrkHfTUxKd8iT1SQoC/T293cc5Mv9gJBqCUM1/DDisPRe6a+gMzNBIsCw8nutYbDdsli8Bqe@lists.infradead.org X-Gm-Message-State: AOJu0YxBmb7encSRyEBfzFGpEBg+b2Rxa6uaIiOvLKnTD/4GKlxVOHry 0VvpQkz9yBmG1JWC2e6iH6sMr7jxe5JTKOUKH4JUH2qSUrc+1C1OXv2K8RSlAEw5Aa/c0SXCPnM plDpXsia1WD7jiTySady17n0JNg+v/H9YuVsXBiGTzkrFrFNZQOMzayDJrC6Ty+m1F2Q6UmMU X-Received: by 2002:a05:622a:1a8b:b0:44f:89e3:d119 with SMTP id d75a77b69052e-453126babcemr11885661cf.12.1723219761643; Fri, 09 Aug 2024 09:09:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGHUujWoHeQHvUCs2OAJSjyQdbxlWd9FX1yi8fb3RZKwv0O5abV6g6G64GeuNGe2uyjoJD48A== X-Received: by 2002:a05:622a:1a8b:b0:44f:89e3:d119 with SMTP id d75a77b69052e-453126babcemr11885331cf.12.1723219761262; Fri, 09 Aug 2024 09:09:21 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:20 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 03/19] mm: Mark special bits for huge pfn mappings when inject Date: Fri, 9 Aug 2024 12:08:53 -0400 Message-ID: <20240809160909.1023470-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090924_464670_9266B81B X-CRM114-Status: GOOD ( 11.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We need these special bits to be around to enable gup-fast on pfnmaps. Mark properly for !devmap case, reflecting that there's no page struct backing the entry. Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 39c401a62e87..e95b3a468aee 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1162,6 +1162,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, entry = pmd_mkhuge(pfn_t_pmd(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pmd_mkdevmap(entry); + else + entry = pmd_mkspecial(entry); if (write) { entry = pmd_mkyoung(pmd_mkdirty(entry)); entry = maybe_pmd_mkwrite(entry, vma); @@ -1258,6 +1260,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, entry = pud_mkhuge(pfn_t_pud(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pud_mkdevmap(entry); + else + entry = pud_mkspecial(entry); if (write) { entry = pud_mkyoung(pud_mkdirty(entry)); entry = maybe_pud_mkwrite(entry, vma); From patchwork Fri Aug 9 16:08:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758966 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BC468C52D7C for ; Fri, 9 Aug 2024 16:12:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ip3F/lMuK2on6ujO8FbOHDG+DYlKv6bK2Y5nLIm/Zzk=; b=w2CVn/dYhglqqp+LMtAu7l7IYm HwBbxBv+el1m+fqnjWcOJYcRIn7mOs2PozRSFnkQMqdZpOKIENEAUiC1q6h+PpdaPUlPSN+HbfQqw XsQvBiovbily5NB41RMOr3y5YCG7z4p8I5YTuJ2NPBXujCUPeH5tBqGTrG5+UrcLlPNw7eakpb/wE D/NxOiS//FmrvqrqGeabrI9eMZ26zgRCkV7cgBbg4+u77cVZ+l939OKcY3Z+S3HELRlG5p2k5HlxX 2KvJ2jC6GbjBMyUW5AVr2NqNcifDJZ0QmFxI8THTuVmKrj5o9yVe6JL89HbLz9dO/8IskTy82fJGK 1Sl3EGKw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSDx-0000000BqPv-1MDB; Fri, 09 Aug 2024 16:12:17 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBD-0000000BpNh-1Hyn for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ip3F/lMuK2on6ujO8FbOHDG+DYlKv6bK2Y5nLIm/Zzk=; b=LpGFTbVtTohrLvzxpIXLTw/68JaA/e2bvqtH9OdcyDr4M3dL7GL0wehytF2O3k0zkI27Z1 qy43mt1rF+mGCAqaH+810FzBaH6C7OI0f6hRhxo11qmogUg3RIrYODPxQONNBEmkmxN+Xn cUBhgL/Hf6g25S/f4zRD93+En9e0/ZE= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-558-U8ruFUJbNVSNJ16jIhSujQ-1; Fri, 09 Aug 2024 12:09:24 -0400 X-MC-Unique: U8ruFUJbNVSNJ16jIhSujQ-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4531adc3c76so35711cf.2 for ; Fri, 09 Aug 2024 09:09:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219764; x=1723824564; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ip3F/lMuK2on6ujO8FbOHDG+DYlKv6bK2Y5nLIm/Zzk=; b=Cn/6TcdSwfE0Q0duxpf6hwjVf+aEetP1bD1WaCubxYByzUVi/0iXQMqe+qdKnIundF 1xZk1LltRBF6xnqxHwFXa2gXQASUfmgTh4ycLEH2fFhg1WEuXIN1o2aKNKsudQSyKQMI oijXuUcKf6bFfwTA+fHSAvCxRAebtGKavlurrLPzapvp1XEJy8BM4QEAhV/CeqsaD2Lu Xw/tTVCv+1DHwvmxxH1KdRkMj1fcO2iROxiuFsZkXL6C+AHSsoebl9WEvh1iMLuo3y4z BUX4+A+QVsAiYnx73hHGj8z/DHFr5aH145U/A3CERjGx8oBbyRG7eMadPTcmV6Y00hHa YhSA== X-Forwarded-Encrypted: i=1; AJvYcCV0NnR3jvZB2+kHRZU1IwgmRuTwJrzJsG9BP2KuRXGlvdSCwVKIlb0m9VpN4M4p0CijDA99kpH183kLPxoPNm/J@lists.infradead.org X-Gm-Message-State: AOJu0Yy/EQ/a4FVAP0xcaIk7Vwt5nPezJWlOtWTdur1XPb8HOT2lHgqg FqkPx2x1hexvzI/IIHIWyL4/9jGwmC0t4IWYbVOXrMHy4QyeEOWEAoJ2OTJ0H1bCvabaFYgeeAv Hh9UZn7+FDQyCLrkeYDmb2Bg2MfNr4Tz1d6ppCo9FH5PUYCCToX2CsBOs6IHPmy2x9QR+tpVB X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14423201cf.3.1723219764198; Fri, 09 Aug 2024 09:09:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHps50LqsfKl8qXevdYdePG8tewmjaKqGlRovYXRf84yciplQ1RIErbjQjT+BAlToj6EpXuWQ== X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14422831cf.3.1723219763777; Fri, 09 Aug 2024 09:09:23 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:22 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Matthew Wilcox , Ryan Roberts Subject: [PATCH 04/19] mm: Allow THP orders for PFNMAPs Date: Fri, 9 Aug 2024 12:08:54 -0400 Message-ID: <20240809160909.1023470-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090927_518727_914D96B9 X-CRM114-Status: GOOD ( 17.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This enables PFNMAPs to be mapped at either pmd/pud layers. Generalize the dax case into vma_is_special_huge() so as to cover both. Meanwhile, rename the macro to THP_ORDERS_ALL_SPECIAL. Cc: Matthew Wilcox Cc: Gavin Shan Cc: Ryan Roberts Cc: David Hildenbrand Cc: Zi Yan Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- include/linux/huge_mm.h | 6 +++--- mm/huge_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 2121060232ce..984cbc960d8b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -76,9 +76,9 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; /* * Mask of all large folio orders supported for file THP. Folios in a DAX * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to - * it. + * it. Same to PFNMAPs where there's neither page* nor pagecache. */ -#define THP_ORDERS_ALL_FILE_DAX \ +#define THP_ORDERS_ALL_SPECIAL \ (BIT(PMD_ORDER) | BIT(PUD_ORDER)) #define THP_ORDERS_ALL_FILE_DEFAULT \ ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) @@ -87,7 +87,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * Mask of all large folio orders supported for THP. */ #define THP_ORDERS_ALL \ - (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DAX | THP_ORDERS_ALL_FILE_DEFAULT) + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL | THP_ORDERS_ALL_FILE_DEFAULT) #define TVA_SMAPS (1 << 0) /* Will be used for procfs */ #define TVA_IN_PF (1 << 1) /* Page fault handler */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e95b3a468aee..6568586b21ab 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -95,8 +95,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders = THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma)) - supported_orders = THP_ORDERS_ALL_FILE_DAX; + else if (vma_is_special_huge(vma)) + supported_orders = THP_ORDERS_ALL_SPECIAL; else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; From patchwork Fri Aug 9 16:08:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 34761C3DA4A for ; Fri, 9 Aug 2024 16:13:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bfMsI243pXvbDIYk3LIHdKnZKRp0h0WHhTX8nwPWLTI=; b=QKieir05PtdoGH/mQZG1V4b1Ti 3jFgG5wWgqbqhLIZSCbHGtSMUjlGPF6ES2Hqnv7+J/L2cXpyJ3O51TWTm3b62jSkiBRkHl4qyd3Bo QRgc4Te1m+nlgJ+BwQzwH/MXpRUZpOf11S08eql7AvGn7Q2L2ESShSGjDjZqbbQqH3XjGgo/fPhIy 2bkubnqj3F63ZPn5pkL2qpXYeDeeEuRItun9KEQxebxCUgGlTFqR1iZ7U/R+T0r02R8XdFEF0QsyA ywpINfHnYZMeD05Y4FmENMvkQdmJKRANafspwF5ed6bdIINi3V8PL99OJi2MweU3cw8MIT0q4gQ6e 9PwN/BSA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSEU-0000000Bqba-3uyb; Fri, 09 Aug 2024 16:12:50 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBF-0000000BpP3-3pBu for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:31 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bfMsI243pXvbDIYk3LIHdKnZKRp0h0WHhTX8nwPWLTI=; b=gBd4TJUhLvNQDYh1PUDUgel3bAtJU85/efPnUWHoaHH9sH1Agh0YmjeA2xolzfHxVEWxaj jXi8Qent0c1uqpyE4/KJasL8B1HC+HLtctnOFbrSP5O8JGu3QD6iC6fDKjolgG0ZX4XpW9 5PbJfmW8O0jT8Ttx8EEUJCWm3MeZzOs= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-661-Qlvm6XNnOGGU2XnCd5syeA-1; Fri, 09 Aug 2024 12:09:27 -0400 X-MC-Unique: Qlvm6XNnOGGU2XnCd5syeA-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-44fe325cd56so2983921cf.1 for ; Fri, 09 Aug 2024 09:09:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219767; x=1723824567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bfMsI243pXvbDIYk3LIHdKnZKRp0h0WHhTX8nwPWLTI=; b=sZz/U4Xal3znuwFWlOZes5GwCFR2AtHrHqnFcJzupgYAywnaGKeMAt5RaG7kOfU92l 94YHIbs7wOcJYLhBjiE/JImsrBuxagOoCs5PrtTUPSyJwWgaIvD5hbSMpMjyhFQHbRS2 nJEoeuKG2DdYaanF6Vg3qvWGwxMIbMzyvJmsswM/im5GfvejQzCBS9b2ZDL78aTn7GDX 5yxQQVyxSphaViQyXziRasQuZv/l/Ac/QPvGa8vxwB/zCjNV1Y14hinFgWLt4eWRQYJA YMOuHUSgSPm5cSDuDH8QARXnoShuhdbu68cK+Au/9h7liBRtKi7CoSWCG+f6VPJr2nsF Nukg== X-Forwarded-Encrypted: i=1; AJvYcCVjWecKWIX8nWxDV/pCj2887t2G1qWHciifdVY2QcxYLH5yo9n/bxUquWRk4JP16QCRpWk7R7Ee9tc4ipDPJnzg@lists.infradead.org X-Gm-Message-State: AOJu0YyOaNaE30hriPnr7IszWygkbSFEP+aMefyd00DBOwjgxQvhmnsd zzfa5soQYRED5H2hJSQk7yFTiFfB9qAYSbbdWbBpJ6qS8apzuzppjYIjfeUMTIO5xF297lmiPeT 4XrrvzKexzQMNGIaQHujDYSSjIDk84cs8ePwgUZmsGz/ZQTheQQyA6Vjyz+2VfG2AYt47rxxW X-Received: by 2002:ac8:59cc:0:b0:44f:c953:290d with SMTP id d75a77b69052e-4531251d17bmr12886111cf.2.1723219766851; Fri, 09 Aug 2024 09:09:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IElkbLJ8q6bD88BtBGZOYKdaksYNPrmA3JWwjtOEQ9+GQ5ZX+OpUCG/V8WuVaLDdrJFvk6CQQ== X-Received: by 2002:ac8:59cc:0:b0:44f:c953:290d with SMTP id d75a77b69052e-4531251d17bmr12885751cf.2.1723219766412; Fri, 09 Aug 2024 09:09:26 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:25 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 05/19] mm/gup: Detect huge pfnmap entries in gup-fast Date: Fri, 9 Aug 2024 12:08:55 -0400 Message-ID: <20240809160909.1023470-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090930_094216_1DBBB0B3 X-CRM114-Status: GOOD ( 12.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Since gup-fast doesn't have the vma reference, teach it to detect such huge pfnmaps by checking the special bit for pmd/pud too, just like ptes. Signed-off-by: Peter Xu Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe --- mm/gup.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d19884e097fd..a49f67a512ee 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3038,6 +3038,9 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, if (!pmd_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pmd_special(orig)) + return 0; + if (pmd_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; @@ -3082,6 +3085,9 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, if (!pud_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pud_special(orig)) + return 0; + if (pud_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; From patchwork Fri Aug 9 16:08:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DC133C52D7C for ; Fri, 9 Aug 2024 16:14:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=nP4ocM4VA1bHw0KaG7AmUa9593dVgU8QzxEKB9c5vMc=; b=KT00qZEq6Wat93kCUAXE0EXbKE LNCIm8m81cZKDKtXeo0VPfhNZWFvsqeiFZMUMWKbay0HOmFUA9iX5OaNSZNLrVlmLhETDsiP22NPk qk/nDCxF1IE4LaJ+u1CCCTdc01UHFSGHr7ctgkttvF6Hhz8PtiGT4++UvdfKoRnhNVeMuGLNj+qw3 mWvJYrfGwrKAFG0nNQrAd87ydBFYtu4w8YUxO7faMyndnXidJakeKrq3PTCM+0MEMeC3/IY5saMAF vKuil61TGNm87EDeyEc8H0frgLWDJceDDO/sk2jVdC222FcNdBiLQgi6tIbUzs6gJ1XVV5lWJnJ1V Ow3hshcw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSGE-0000000BrCf-1FAK; Fri, 09 Aug 2024 16:14:38 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBM-0000000BpTT-2y8O for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nP4ocM4VA1bHw0KaG7AmUa9593dVgU8QzxEKB9c5vMc=; b=hf6v7sOQnpnPlfpmSoBxTI/cz6PUk47DibmfuJg/P/12SD/BvcKDCXVJqPjppZ5cVJpdRd 8jU2Nb+6e/Ipth0XL8wREobIYA0Eaqu3q8yLxvRQbNoEYj1dd6EoEtxlAxMFo0yk762IHj fJZTZ6keNu+O9g0sn8ncAWb2x4QyU6c= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-318-BverxwdkNY2S0nS0_wSSgg-1; Fri, 09 Aug 2024 12:09:34 -0400 X-MC-Unique: BverxwdkNY2S0nS0_wSSgg-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44fe05a4b61so1652531cf.0 for ; Fri, 09 Aug 2024 09:09:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219774; x=1723824574; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nP4ocM4VA1bHw0KaG7AmUa9593dVgU8QzxEKB9c5vMc=; b=L0KKnXoVakC4NM2459dHJkasah/dN1Tz5OZKDNa4Yo87OwWTSXzo7RrfwkW5XwZpdO s9sgWvw95Ba2OgdY9pSLcPZugyRZqXq4DEIl37SYQOgR7jiDSrX5Us9gTuExKNNB4d58 vnhvRabG21k1PWugXbGAF1Owac76WRbU9UWvlWO6r4SC/NBjTb2BfDJupEGw6XwWAvvf yKaXSqsW49jbozd1iqSylIcNqGDJKOS21qFfEbhgDUrNiKjvLz5vbE0GatXtkAg3ycZP cj+z7hnRiukg7kvs8bzA0h51hSC7jJSuKyx8k1wIbwqWBSYcQbHKeJJLP4DW0kEMM66D 7BpA== X-Forwarded-Encrypted: i=1; AJvYcCUhomPUdWbo++3m1nDva1pamBmJalOeDr69JrjaPZPUarGAoVrb71tMgn/rnG0rkBA9qnREu85+GM04tA4S7qs9@lists.infradead.org X-Gm-Message-State: AOJu0YyxwkB2JKIUpiWEl/0XG78jQKoYUamku9cwkeOeTANPvCMuy/b5 T/X0z2Bur8998fp3VCyuXLOmSx588uDU+RAThEhttnlGvbV1qK1//W2erQJ8ggAjsuWttDNznXL 0yd57HcekUzYbuKvPTIrJSsHLL3edANUXGeDOoHZz5Ku5o2BYM51K4VKbiDrmOXrIhcS7Rzqn X-Received: by 2002:a05:622a:386:b0:450:21b2:22e5 with SMTP id d75a77b69052e-453125745b1mr14927781cf.4.1723219774004; Fri, 09 Aug 2024 09:09:34 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGLMln6bbvPWjvF9TKd48hi2B1jMtNjQrsP4cRdycFMZCE1oQ0WYqhx5h2X//7ZLsXT7ckFXA== X-Received: by 2002:a05:622a:386:b0:450:21b2:22e5 with SMTP id d75a77b69052e-453125745b1mr14924951cf.4.1723219768577; Fri, 09 Aug 2024 09:09:28 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:27 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 06/19] mm/pagewalk: Check pfnmap early for folio_walk_start() Date: Fri, 9 Aug 2024 12:08:56 -0400 Message-ID: <20240809160909.1023470-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090936_874122_D99B18FF X-CRM114-Status: GOOD ( 12.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Pfnmaps can always be identified with special bits in the ptes/pmds/puds. However that's unnecessary if the vma is stable, and when it's mapped under VM_PFNMAP | VM_IO. Instead of adding similar checks in all the levels for huge pfnmaps, let folio_walk_start() fail even earlier for these mappings. It's also something gup-slow already does, so make them match. Cc: David Hildenbrand Signed-off-by: Peter Xu --- mm/pagewalk.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cd79fb3b89e5..fd3965efe773 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -727,6 +727,11 @@ struct folio *folio_walk_start(struct folio_walk *fw, p4d_t *p4dp; mmap_assert_locked(vma->vm_mm); + + /* It has no folio backing the mappings at all.. */ + if (vma->vm_flags & (VM_IO | VM_PFNMAP)) + return NULL; + vma_pgtable_walk_begin(vma); if (WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end)) From patchwork Fri Aug 9 16:08:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B0C0FC3DA4A for ; Fri, 9 Aug 2024 16:13:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GAUMBfy0kdSy8bea6Q2/adQQTYYkrWuiUWfctkd/rW8=; b=1GMtTBgF2eQPftdWrkSdTG31II uvPbea4ovExMLPpGWW9omWni6qEmv7L7Eloo/oDH25asdm1Quyewxmu86pfPrbu50KiI9rXbALcL3 ZDeHxjXSgaSjWNLcgCg0XFBlYB0u1bPWdNdcqhdIl4X+uwuz/VavY8ig6Ci4wlIhb9sZTXmWYGITL vGJ56EZeQOG6Ue3vtfLgI4z0rDbasHBHBLG4rSzMp39PyBoyPI5ADIEVX4fBe2tyEQMT9e0BijRVd /bJs56JLsIlAIk1Ahpg1A2q39nXdlJihBV47KbD2oKu8qKz8ANOzC6FbkM1KDyWc8FppNhz4posu+ oT12/NVw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSF4-0000000BqpR-2qxu; Fri, 09 Aug 2024 16:13:26 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBJ-0000000BpRF-3DK7 for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GAUMBfy0kdSy8bea6Q2/adQQTYYkrWuiUWfctkd/rW8=; b=dCxLousm2hinMnNUx3LAx1ozMlXoSTC9LBWZRxkdsnwfyVnxFIY+WIrGqEok1dw2qixhWC n4odeIo4PSDKA7NXx73edl1PyVzSP1lR4z4/JhobVpVx5Cei4Vdawu0/Mjz+ghz6cTtY8B NvSebNv7QqH/KQhFBE0ud/AVphS1prc= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-154-4GnaSwWpMkGjEhHTT_6Jpg-1; Fri, 09 Aug 2024 12:09:31 -0400 X-MC-Unique: 4GnaSwWpMkGjEhHTT_6Jpg-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4502562fc7eso657021cf.0 for ; Fri, 09 Aug 2024 09:09:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219771; x=1723824571; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GAUMBfy0kdSy8bea6Q2/adQQTYYkrWuiUWfctkd/rW8=; b=E9vlnvrm1j+i12ngv9alfSrW1hgxIJ9t4h6G6IZMisBAhQekotK450iXP8jXCXBUbP gKLgi06kJeKIbj6MSuxCDLLHtCVEtCjpmVt7WbKz0V7VUs1FpArQIYBNEgRd3SHD6OkA Ta1xkdjluBy97LRm8/idHPY3Rkhwqx5KLLV71UuzqhMv/KIPfZquol2D5cwRXElAkOOI reSbqg0PbwFgVWYE8IP/PIGvTYl9GDsidKoMWXhvAyMQBDsBggqQDQHWLVd2A3htIiv8 1Etod8ORPpy0tHVUfD0MB2Y5aDOUqzTI/fTddhX9nJvx70Q3j3V6k4J35pnM6Hl8Ers4 sIMA== X-Forwarded-Encrypted: i=1; AJvYcCWOYa8DMdv06XtWbXcZt8aAEhKsDZA5pEBZE6wnJaODHe4NBl3X2imWlyXr4JGtNXbHrne1kNrxorY6t2qRacLOzAGe3XxCoZLN7q79ooKC6TAZfMo= X-Gm-Message-State: AOJu0YwOdxMevkqH2bm4wIlwBSOVa8fCTawRfya9UExxWOh09naF+sOc bQOwurb4Gl5ANSJ2xT3T+sJFggWF24R68798iPrgh5doCRo8AMLiak0VcImExguuz2JTNzMZr3q qGWWQCci7FRPI93RVwXAf4/iGuAL/GOhLQY57kaecLP9YGgXo0hPQ56YiRJRv20IyCGtfCzQF X-Received: by 2002:a05:622a:180e:b0:44e:d016:ef7 with SMTP id d75a77b69052e-45312652444mr12909261cf.7.1723219771015; Fri, 09 Aug 2024 09:09:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2AJ498flXqsf/RjsZizkWR5q4LBdwGhozxnqN6XMiL4rwNH8ESKXDSym68P6o8cpdHObfXw== X-Received: by 2002:a05:622a:180e:b0:44e:d016:ef7 with SMTP id d75a77b69052e-45312652444mr12908941cf.7.1723219770588; Fri, 09 Aug 2024 09:09:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:30 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 07/19] mm/fork: Accept huge pfnmap entries Date: Fri, 9 Aug 2024 12:08:57 -0400 Message-ID: <20240809160909.1023470-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090933_898085_CB66E8BB X-CRM114-Status: GOOD ( 15.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Teach the fork code to properly copy pfnmaps for pmd/pud levels. Pud is much easier, the write bit needs to be persisted though for writable and shared pud mappings like PFNMAP ones, otherwise a follow up write in either parent or child process will trigger a write fault. Do the same for pmd level. Signed-off-by: Peter Xu --- mm/huge_memory.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6568586b21ab..015c9468eed5 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1375,6 +1375,22 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pgtable_t pgtable = NULL; int ret = -ENOMEM; + pmd = pmdp_get_lockless(src_pmd); + if (unlikely(pmd_special(pmd))) { + dst_ptl = pmd_lock(dst_mm, dst_pmd); + src_ptl = pmd_lockptr(src_mm, src_pmd); + spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + /* + * No need to recheck the pmd, it can't change with write + * mmap lock held here. + */ + if (is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)) { + pmdp_set_wrprotect(src_mm, addr, src_pmd); + pmd = pmd_wrprotect(pmd); + } + goto set_pmd; + } + /* Skip if can be re-fill on fault */ if (!vma_is_anonymous(dst_vma)) return 0; @@ -1456,7 +1472,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); if (!userfaultfd_wp(dst_vma)) pmd = pmd_clear_uffd_wp(pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + pmd = pmd_wrprotect(pmd); +set_pmd: + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1502,8 +1520,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; From patchwork Fri Aug 9 16:08:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 257ADC3DA4A for ; Fri, 9 Aug 2024 16:14:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3aM3oVPeXmFPFf4Q68bjufzjsUq2ESmscQPnLEkaBB8=; b=o2tH0xkwBtcwVbGyTGSBUfXMKk xKj6rsShZ+8WW3zzcilSM1hk0NnljcPbrAwrquVHvcP8/Xr4XcFJXWl3G1dLsNsTCJf925gnL73iS G7ihFZvbrBASNCQyE/9oz6CYc9xcdvcriRgkLFaiEqjrH4wLJzJGXcta31OcBtvGLNkKq78oWiVu2 gHsg4CU9WtCj1e5x070JrxlKJuJ02TsPwHg9DbPSS2WClNVHDkvTyrhQuLU+H/KC7NZMuKM0hb5dT M3cY3tFOmkoME0o81C5O6ScancZDXMyKb1i6ugPW9ecrTN8JPEPUfx4Mb1TitKwvl4yfg/oLDJ+pd 6Sfxntdg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSFg-0000000Br3L-29bE; Fri, 09 Aug 2024 16:14:04 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBL-0000000BpSQ-3NE1 for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3aM3oVPeXmFPFf4Q68bjufzjsUq2ESmscQPnLEkaBB8=; b=eORBY1uRaUO9lSbRYizP2nsz7FI9IlQrkJVOFYiRn8Ej41YCR3+VKULbsFE08ThpeQK+4f 1epGQN9dUIOzsHfUOZWIJC3eK31PgNRZXsh/JB2tu5rf8oYwOW6Sd7pKIdPw70fhhlH3AN 0xo4ty2JOz1AT0BPj3mDFbbHyHQ/3qM= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-216-6nw2c6XlPJSqVAeZpjTy-Q-1; Fri, 09 Aug 2024 12:09:33 -0400 X-MC-Unique: 6nw2c6XlPJSqVAeZpjTy-Q-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-44ff585970bso4756161cf.2 for ; Fri, 09 Aug 2024 09:09:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219773; x=1723824573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3aM3oVPeXmFPFf4Q68bjufzjsUq2ESmscQPnLEkaBB8=; b=qYHuYnsMrOMN7i7v0VHJ6J2VWpdz1A4ij3sufGyskQvUv9ea+GmUcoRyvgw3RrLn+s HayAOf7lPV+uyzMEB4bMINLJHzXalDwNZvoCzKID8st+7vbQd5XShEHIx9hZTqnGYEIG 1JuZhvZYK4xRpLEtEaJYoT5kn6mGaxg+jnZtgzGIaQ2U3Q/c1a5LKoMPMmMXzSl2oS5M DERE19DLFz9zjtLGhKAqvPDI1Dz3feKEl6/yLd6fiPB0OqYTwgAnrLupHVkqH9uFpOgq ID2KG4Q9QOYWUUL3yabgRq1BnlqK8ZcfzkcI7SLDBxyc0njYOAhfdalJXcyMbIv14dVH RkWQ== X-Forwarded-Encrypted: i=1; AJvYcCX6UNZTpa8ZQ9ZljEb6oFmwzXpyPFW1HHPkoIl60qHgYHkyJxZOhH1jMcSSfmsraQErOzloAcLsqMSZPgpT2uZUCswcq/Zx0n/3OtlrbotPP8DfVbA= X-Gm-Message-State: AOJu0Yxfe8j3afvf832MdJHSZzqY81LQvcLHYiSJZgr1etfwNmaz1Y9W qzqK6/PlNPmYEx1eSqla7d1AJQhLkBQKKTCkDhpzKzO8shTqN4UeMS3l3WbOH3FmsDnh8Ng42Gx xnaRGNi5xstaz8a+qnQHAsUNUQ4g/UKkC+elJ0/fY2ZQdkaVi5mPfTCSiEmDnMmfl49NYrPZk X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14739611cf.6.1723219773074; Fri, 09 Aug 2024 09:09:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE6GyuXgyukqToIMYwmYCyyJms6T0RBSwsqvRpSKO6DWSQsB926zH9/mqcqwcpfSE6Nqd5Tsw== X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14739431cf.6.1723219772696; Fri, 09 Aug 2024 09:09:32 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:32 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 08/19] mm: Always define pxx_pgprot() Date: Fri, 9 Aug 2024 12:08:58 -0400 Message-ID: <20240809160909.1023470-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090935_991250_C91A85AC X-CRM114-Status: GOOD ( 13.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There're: - 8 archs (arc, arm64, include, mips, powerpc, s390, sh, x86) that support pte_pgprot(). - 2 archs (x86, sparc) that support pmd_pgprot(). - 1 arch (x86) that support pud_pgprot(). Always define them to be used in generic code, and then we don't need to fiddle with "#ifdef"s when doing so. Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- arch/arm64/include/asm/pgtable.h | 1 + arch/powerpc/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + include/linux/pgtable.h | 12 ++++++++++++ 5 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7a4f5604be3f..b78cc4a6758b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -384,6 +384,7 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pfn = pte_pfn(pte); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 264a6c09517a..2f72ad885332 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -65,6 +65,7 @@ static inline unsigned long pte_pfn(pte_t pte) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags; diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 3fa280d0672a..0ffbaf741955 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -955,6 +955,7 @@ static inline int pte_unused(pte_t pte) * young/old accounting is not supported, i.e _PAGE_PROTECT and _PAGE_INVALID * must not be set. */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags = pte_val(pte) & _PAGE_CHG_MASK; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 3fe429d73a65..2b7f358762c1 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -783,6 +783,7 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) return __pmd(pte_val(pte)); } +#define pmd_pgprot pmd_pgprot static inline pgprot_t pmd_pgprot(pmd_t entry) { unsigned long val = pmd_val(entry); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 780f3b439d98..e8b2ac6bd2ae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1956,6 +1956,18 @@ typedef unsigned int pgtbl_mod_mask; #define MAX_PTRS_PER_P4D PTRS_PER_P4D #endif +#ifndef pte_pgprot +#define pte_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pmd_pgprot +#define pmd_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pud_pgprot +#define pud_pgprot(x) ((pgprot_t) {0}) +#endif + /* description of effects of mapping type and prot in current implementation. * this is due to the limited x86 page protection hardware. The expected * behavior is in parens: From patchwork Fri Aug 9 16:08:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 710FAC3DA4A for ; Fri, 9 Aug 2024 16:15:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=IXGL3lb2RETUVA/g/GpjABQk9U9hrAJ/lFeyUGjNHTs=; b=tv/99fYHB43DyFlh2SLw5UYH5D kjCI71fmwsI0kGg+1Gpbc/Zua9ZUJ64ahtER8aPXKBaux+tqRUC6nQvelovxpn+uscJBo2RlgQjtm tsBFE0fhUyw9na8NAd4GbkTJWQ4eQ5+ZgLGV89Sbdn1IppAOvmw1WpxPA8ozsQ6Q6wR5QsquX0/DS D0WmJZofVe9jJdMkdr2/hitYo4fw/G9x6btf/lST9zQ3ayxYRjVJidRHBk6h0AuGyp7vEbETSHmYR liHwICaI6u2RWfk28rsg+tOba253sa4bZ0vZ7hRlI3CqIa2uv3wH4ukHbAc47522zXZeTNqIcqVx+ +sSFv4/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSGl-0000000BrMR-3DaI; Fri, 09 Aug 2024 16:15:11 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBO-0000000BpUS-2SKQ for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219777; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IXGL3lb2RETUVA/g/GpjABQk9U9hrAJ/lFeyUGjNHTs=; b=WzztwzzW/V4BSMv+BJT2qbbP4QvWjUpg8Z6IEyTXUVCW0hUkjQxiphQEeZmsHsVGVSMOyz AkCsGSGK6RmmEI9l1STLsWbF81uZIbRuRP9Iyew/tvpbFTC+C0pYB43lwzzA0lYlNbn7An KqsaAjRWmnpUIAc/IrpX2uCXZOP6FTs= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-529-_0TbAr9FMripk5-ajpoA_A-1; Fri, 09 Aug 2024 12:09:36 -0400 X-MC-Unique: _0TbAr9FMripk5-ajpoA_A-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-44fe325cd56so2984101cf.1 for ; Fri, 09 Aug 2024 09:09:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219776; x=1723824576; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IXGL3lb2RETUVA/g/GpjABQk9U9hrAJ/lFeyUGjNHTs=; b=qV1ko5ytX49TkA0JtwWGnLgA9XjKlTcRwqKX9ZrY+lVXgBAuRjO/1HTnyJLchDHkZ2 dOOsXPgac6kKgC1ibhJRRk5ylxeK5HKTc11B/bVpJ4lUOhQhYfjDJZ6qzu3cKW1lflFn zC4MMbuTXNphxLuiO+jIq51atZDzX84UoOpFUQYox8/cCsx4/m6XPreRst63GCYRwJfr fJOfs+uqbRO+05fboHFrRbn6ZQpl/hCW6PAhkNtuA5Pd1zeCS1N8z9x/bjM0WyWNeL40 fkwenR6CFKaM4CvDuXLk8mX496OAnHbbgt2Z3AnVChN/DulSMLTWiFLwlq7cgKaB9n2J XtUA== X-Forwarded-Encrypted: i=1; AJvYcCUtQ0LUOvgKre4hBjkeeB4f9YU4m4JvADGYKNBGquiBqD/Kcdn5/jY0CNJN97mHt999wjHn3KFfqfCNAZGCWG30@lists.infradead.org X-Gm-Message-State: AOJu0YwZP4XSr4GNrszalx407PBzkAqzuKI7arDzvdLM807nZvpXRszJ EedSj9uDpMlGfnnKa1wlQEJIx0GzWnjnUULTd2tL79oKPGOyL21o+0e9GESf2IBa34LPDcunxZL XvoHuj3wllgZQ5MqRah0elXfDC62KWhECuOSVH8O3jwm8HQGALvdbXVbz7x1CL6apVBb0/0i6 X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14428751cf.3.1723219775687; Fri, 09 Aug 2024 09:09:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFUrJX3vSa/2Mr5PpFf2V5KDcJEixKltLA8oaEJqLbElyFRMCegYAzgvcdg3qK/Zl0Af+cLBA== X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14428311cf.3.1723219775158; Fri, 09 Aug 2024 09:09:35 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:34 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 09/19] mm: New follow_pfnmap API Date: Fri, 9 Aug 2024 12:08:59 -0400 Message-ID: <20240809160909.1023470-10-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090938_732455_B6A7562F X-CRM114-Status: GOOD ( 21.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a pair of APIs to follow pfn mappings to get entry information. It's very similar to what follow_pte() does before, but different in that it recognizes huge pfn mappings. Signed-off-by: Peter Xu --- include/linux/mm.h | 31 ++++++++++ mm/memory.c | 147 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 90ca84200800..7471302658af 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2374,6 +2374,37 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); +struct follow_pfnmap_args { + /** + * Inputs: + * @vma: Pointer to @vm_area_struct struct + * @address: the virtual address to walk + */ + struct vm_area_struct *vma; + unsigned long address; + /** + * Internals: + * + * The caller shouldn't touch any of these. + */ + spinlock_t *lock; + pte_t *ptep; + /** + * Outputs: + * + * @pfn: the PFN of the address + * @pgprot: the pgprot_t of the mapping + * @writable: whether the mapping is writable + * @special: whether the mapping is a special mapping (real PFN maps) + */ + unsigned long pfn; + pgprot_t pgprot; + bool writable; + bool special; +}; +int follow_pfnmap_start(struct follow_pfnmap_args *args); +void follow_pfnmap_end(struct follow_pfnmap_args *args); + extern void truncate_pagecache(struct inode *inode, loff_t new); extern void truncate_setsize(struct inode *inode, loff_t newsize); void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to); diff --git a/mm/memory.c b/mm/memory.c index 67496dc5064f..2194e0f9f541 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6338,6 +6338,153 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(follow_pte); +static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, + spinlock_t *lock, pte_t *ptep, + pgprot_t pgprot, unsigned long pfn_base, + unsigned long addr_mask, bool writable, + bool special) +{ + args->lock = lock; + args->ptep = ptep; + args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pgprot = pgprot; + args->writable = writable; + args->special = special; +} + +static inline void pfnmap_lockdep_assert(struct vm_area_struct *vma) +{ +#ifdef CONFIG_LOCKDEP + struct address_space *mapping = vma->vm_file->f_mapping; + + if (mapping) + lockdep_assert(lockdep_is_held(&vma->vm_file->f_mapping->i_mmap_rwsem) || + lockdep_is_held(&vma->vm_mm->mmap_lock)); + else + lockdep_assert(lockdep_is_held(&vma->vm_mm->mmap_lock)); +#endif +} + +/** + * follow_pfnmap_start() - Look up a pfn mapping at a user virtual address + * @args: Pointer to struct @follow_pfnmap_args + * + * The caller needs to setup args->vma and args->address to point to the + * virtual address as the target of such lookup. On a successful return, + * the results will be put into other output fields. + * + * After the caller finished using the fields, the caller must invoke + * another follow_pfnmap_end() to proper releases the locks and resources + * of such look up request. + * + * During the start() and end() calls, the results in @args will be valid + * as proper locks will be held. After the end() is called, all the fields + * in @follow_pfnmap_args will be invalid to be further accessed. + * + * If the PTE maps a refcounted page, callers are responsible to protect + * against invalidation with MMU notifiers; otherwise access to the PFN at + * a later point in time can trigger use-after-free. + * + * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore + * should be taken for read, and the mmap semaphore cannot be released + * before the end() is invoked. + * + * This function must not be used to modify PTE content. + * + * Return: zero on success, -ve otherwise. + */ +int follow_pfnmap_start(struct follow_pfnmap_args *args) +{ + struct vm_area_struct *vma = args->vma; + unsigned long address = args->address; + struct mm_struct *mm = vma->vm_mm; + spinlock_t *lock; + pgd_t *pgdp; + p4d_t *p4dp, p4d; + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pfnmap_lockdep_assert(vma); + + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + goto out; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + goto out; +retry: + pgdp = pgd_offset(mm, address); + if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) + goto out; + + p4dp = p4d_offset(pgdp, address); + p4d = READ_ONCE(*p4dp); + if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) + goto out; + + pudp = pud_offset(p4dp, address); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + goto out; + if (pud_leaf(pud)) { + lock = pud_lock(mm, pudp); + if (!unlikely(pud_leaf(pud))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pud_pgprot(pud), + pud_pfn(pud), PUD_MASK, pud_write(pud), + pud_special(pud)); + return 0; + } + + pmdp = pmd_offset(pudp, address); + pmd = pmdp_get_lockless(pmdp); + if (pmd_leaf(pmd)) { + lock = pmd_lock(mm, pmdp); + if (!unlikely(pmd_leaf(pmd))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pmd_pgprot(pmd), + pmd_pfn(pmd), PMD_MASK, pmd_write(pmd), + pmd_special(pmd)); + return 0; + } + + ptep = pte_offset_map_lock(mm, pmdp, address, &lock); + if (!ptep) + goto out; + pte = ptep_get(ptep); + if (!pte_present(pte)) + goto unlock; + pfnmap_args_setup(args, lock, ptep, pte_pgprot(pte), + pte_pfn(pte), PAGE_MASK, pte_write(pte), + pte_special(pte)); + return 0; +unlock: + pte_unmap_unlock(ptep, lock); +out: + return -EINVAL; +} +EXPORT_SYMBOL_GPL(follow_pfnmap_start); + +/** + * follow_pfnmap_end(): End a follow_pfnmap_start() process + * @args: Pointer to struct @follow_pfnmap_args + * + * Must be used in pair of follow_pfnmap_start(). See the start() function + * above for more information. + */ +void follow_pfnmap_end(struct follow_pfnmap_args *args) +{ + if (args->lock) + spin_unlock(args->lock); + if (args->ptep) + pte_unmap(args->ptep); +} +EXPORT_SYMBOL_GPL(follow_pfnmap_end); + #ifdef CONFIG_HAVE_IOREMAP_PROT /** * generic_access_phys - generic implementation for iomem mmap access From patchwork Fri Aug 9 16:09:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0438CC3DA4A for ; Fri, 9 Aug 2024 16:15:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XgMG3z25w/FU4W74BEAmfGyy8QhkBSj3TNUPa/vYriM=; b=QscG85F89KPmc5IKkbNaYETqRm 5jl+Ccutkj/8WjLMiUdn1PazsVPVpS+GiWVCj/2RUqOlNyxxLPsUmjqZbW4/74v4Keel5PAMchKQp m+qxQAkKdHNL50X3qPQAmnAW+9HzV4NMiZLfOk4M0+Whklp43PqWyMoGinrQCJUl7ZlmPGY4aJziJ AKDl/N17iepUJoSAJ4btu3BWJZTwU7xgPG6RY4r10PrgEQTj2GYc1MHWh+CWiAkoEnom20cDfcaw6 2SYt+lnxrSJzXSpQUpNwFaHfd5IVfByQRTD5xBwBcsuBknDDJgE2GNpOxVPD2qt0EMOYn5gSaLNKs k8tMW2VA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSHJ-0000000BrXE-1xsw; Fri, 09 Aug 2024 16:15:45 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBR-0000000BpVR-15eD for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XgMG3z25w/FU4W74BEAmfGyy8QhkBSj3TNUPa/vYriM=; b=YPhU4q7vBmE+shzeD+qcwbh8e578X+hGr3vuwNeBm5qYqMxsMBnME+zKlxZN7JnCarnqL2 yTKc1TIdFJl2CKURUSHheXT3gTeIMavlJlMCO1vMsS7VtZKDZ4ZLtL6g1VrU9fF135a2D6 YhTZ1Ot52w7rd8XGzCds6VOcOi3H04U= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-232-yF-pRAuMNYKq1fjWMX8bUw-1; Fri, 09 Aug 2024 12:09:38 -0400 X-MC-Unique: yF-pRAuMNYKq1fjWMX8bUw-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1e41eda5aso5457885a.3 for ; Fri, 09 Aug 2024 09:09:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219778; x=1723824578; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XgMG3z25w/FU4W74BEAmfGyy8QhkBSj3TNUPa/vYriM=; b=D13J1l4pWMgGFOyNSxJghdJKLzmbVYZwhjaCu/Ub8HtX8RaYlrfAuFtP/6Na4khJmj ZeWV4qVDSHf6fw6mKdOxgXjDJbwh1VQ9LcIRfLd9YeA4ap5Vi83pO3eztyJMd61wS6G3 d+H4b2+p5JIkljBbdIu4QUgdPXwJf0DYVGTd6X8/H2LyYluKWSiveeV+z1NICP1ZjNbj Mg4074v0K2OEXGFs2RjH1vLVer3h9CjDmS3XdLK9jCpVlQ2nOBTqgHmgpGpQlVLd7nSE iP5+IWoNj0sALMOJFOKYS7JR5y8aNxDH/sKrfMUT+NyX+G9juT3F+1HqsZou0TXWlR5B XmCg== X-Forwarded-Encrypted: i=1; AJvYcCUtzqs3d8sfd0DoZXOa3eNyuf2m3/mlQBD+7eus3Ye37AmSoR7peyy+8ngI6V7JZt2Bkw+HS3fsZraciGVZpXvBlLZcndWv8itcO4+VYG2lIvfgYQQ= X-Gm-Message-State: AOJu0YxE7izvKiC8Ym1olwpBjSdLeIvWwDAUBxTMthvZMTH97Ybfgf5d jdQ42xN0iVssnAXcOtz0JD5OXl3CT7zgZg3e8sgwRiPMxJR0Hvdy1OHWOqI3BmdKaTB4HSYTQEw FP8tB4FBna25hBoQ9rDBhQKibX9LCS5aIIisWm4JQ0tfrnXGpWOAoQbLKfc2AyPpUEe1OYYGF X-Received: by 2002:ac8:5dd3:0:b0:447:dfa0:283e with SMTP id d75a77b69052e-4531251fe34mr11225391cf.1.1723219777984; Fri, 09 Aug 2024 09:09:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE6MoN+Dd8Hdt5Y2cXuQIqSY6AztxI8VpmGorOFfK4YC2KlFlbKhz5m1DVHT1NSvONycoblfw== X-Received: by 2002:ac8:5dd3:0:b0:447:dfa0:283e with SMTP id d75a77b69052e-4531251fe34mr11225161cf.1.1723219777518; Fri, 09 Aug 2024 09:09:37 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:36 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 10/19] KVM: Use follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:00 -0400 Message-ID: <20240809160909.1023470-11-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090941_451637_01A553E4 X-CRM114-Status: GOOD ( 15.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new pfnmap API to allow huge MMIO mappings for VMs. The rest work is done perfectly on the other side (host_pfn_mapping_level()). Cc: Paolo Bonzini Cc: Sean Christopherson Signed-off-by: Peter Xu --- virt/kvm/kvm_main.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0788d0a72cc..9fb1c527a8e1 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2862,13 +2862,11 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, unsigned long addr, bool write_fault, bool *writable, kvm_pfn_t *p_pfn) { + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; kvm_pfn_t pfn; - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; int r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) { /* * get_user_pages fails for VM_IO and VM_PFNMAP vmas and does @@ -2883,21 +2881,19 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, if (r) return r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) return r; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) { + if (write_fault && !args.writable) { pfn = KVM_PFN_ERR_RO_FAULT; goto out; } if (writable) - *writable = pte_write(pte); - pfn = pte_pfn(pte); + *writable = args.writable; + pfn = args.pfn; /* * Get a reference here because callers of *hva_to_pfn* and @@ -2918,9 +2914,8 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, */ if (!kvm_try_get_pfn(pfn)) r = -EFAULT; - out: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); *p_pfn = pfn; return r; From patchwork Fri Aug 9 16:09:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D02C0C52D7C for ; Fri, 9 Aug 2024 16:16:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=XcC2LmZsGAebGsLG2VfPCJQbFC er8aOrqQAm2XaO2zhBAQ7XfZqYivQCoaDi6o5HrZZ6ScGVOdSix0r6weKGJq1KTmwQ2/6qBFgB2P2 09SCoUWh/HLd5FXyi3bDlnmduKEUsdtB3EyjzU5z2uzRoxhe9EoWK4PccY8F6h4R7jbtw6cJYMPTc YC0W7XMbU31EXiL9r4acoyilFWmZQncybs+lJCzV3tA3icVXNFRZc0DBdNF6E4zI2oioxBgD7NAWH bmNmOoUQDDfPmg50GX+Wbl4z3ppRXUXbZKEVPw5UzRZn+C4O+G7rBW3Wkga87vJ8uwaKNqjkThgFM PYXLB8gw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSHr-0000000Brgt-20LD; Fri, 09 Aug 2024 16:16:19 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBT-0000000BpWo-0s9o for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219782; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=esEb7RxG1OyNGuXULMwbkHisXkJiTy4f7Gr1RZYOWuxPxZLfmXbbewqKJq1MGTnfMa5sEe nD4zRgC6zv0aiU9gHZ2y4KRvfU5cWwGs6wIr5pIUSotNGQZruG6lLVwmgIehOMnlncZX5y FeSpNhDEXQ2v5g5I49a2jONl8vGP9L8= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-384-JGqBU97POEeYkLLRvxE--Q-1; Fri, 09 Aug 2024 12:09:41 -0400 X-MC-Unique: JGqBU97POEeYkLLRvxE--Q-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-4503ee1b5d6so5446931cf.3 for ; Fri, 09 Aug 2024 09:09:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219780; x=1723824580; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=s6HmnU//i1VPu3nlSYTWMu5fE+FVRmwykXG5hBk70Gf4aowqaquRM7DHTZS0wc65i1 RhR9GJj879NHkdysI6Le9jafuGb3ehniVfCW1gY5tS0mOVcs0pqB8BJxlFx0O7+VicGR qub/q7eMYq7z+BpGhw0HQ0NKpY3/9csxSZD3YfOqCfgwI8c7MWee26o38AP48MrKK/JZ +lG4IMTqScxmumEx4Gzo0gaXtXGQLmQEhgOoKcJH8RClu5oWYZSr+OPi6NTkZqftsPTe 1idkyfez665OWm9Aqcs8CPubn9n23AI5sAguG56vpBx+KnArFm3E76RCRGFns6nUnG1U YlCg== X-Forwarded-Encrypted: i=1; AJvYcCUSrmBXMUyBBItTgLLxNwrt6M8Hnq4sOCj/tjnx3eJyVBSc7B+OZlhq9Rh0yOV1F/Pge633GxIbOoQSOeHbZZhWezKW8n5w4omPNADwYrY+BmKga3s= X-Gm-Message-State: AOJu0YxYD5QN/SO+Mw/wZaMDMaCneo5C0km9VcWrbOClyhrHby+d43Ad ajwhdfNZgvunOGcenZ8KLxFrjccB5Ufqy19UcnBc6Oh1qqt6llOQkeOVrBEudd1QUa4BGNN73bj dYRMDCmz6+KCvd/MYT6rV50HZFvaDtUHvGeKTXjIDpvenD1x71SAmyZndT6Mk20tR1Qm+4b8hIE 95CfpUEBU= X-Received: by 2002:ac8:5912:0:b0:450:efd:723 with SMTP id d75a77b69052e-453125552fbmr13163441cf.4.1723219780569; Fri, 09 Aug 2024 09:09:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFtbIkutydCKQHHfrahTRflQkyMYGSh3zYrObmthmw14T659Ni251Xo72HIqIFQZU+hooLp2A== X-Received: by 2002:ac8:5912:0:b0:450:efd:723 with SMTP id d75a77b69052e-453125552fbmr13162951cf.4.1723219780083; Fri, 09 Aug 2024 09:09:40 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:39 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Niklas Schnelle , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-s390@vger.kernel.org Subject: [PATCH 11/19] s390/pci_mmio: Use follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:01 -0400 Message-ID: <20240809160909.1023470-12-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090943_360261_2F3155E1 X-CRM114-Status: GOOD ( 13.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Cc: Niklas Schnelle Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: linux-s390@vger.kernel.org Signed-off-by: Peter Xu --- arch/s390/pci/pci_mmio.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/s390/pci/pci_mmio.c b/arch/s390/pci/pci_mmio.c index 5398729bfe1b..de5c0b389a3e 100644 --- a/arch/s390/pci/pci_mmio.c +++ b/arch/s390/pci/pci_mmio.c @@ -118,12 +118,11 @@ static inline int __memcpy_toio_inuser(void __iomem *dst, SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, const void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -169,11 +168,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.address = mmio_addr; + args.vma = vma; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) @@ -181,7 +182,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, ret = zpci_memcpy_toio(io_addr, buf, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); out_free: @@ -260,12 +261,11 @@ static inline int __memcpy_fromio_inuser(void __user *dst, SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -308,11 +308,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.vma = vma; + args.address = mmio_addr; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) { @@ -322,7 +324,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, ret = zpci_memcpy_fromio(buf, io_addr, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); From patchwork Fri Aug 9 16:09:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758974 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6FCC9C52D7C for ; Fri, 9 Aug 2024 16:17:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=dHUdq0ZI2fvSzmSbu56kn/TFI4mY4GVPslnR6ZzQETQ=; b=wyKwNWNCmwE/jj3y/U1xM2A3gj tnNW8UbPzGDcojJc+ScSnm1nYC02li3Exu8CDObJdT4gxyyUY2Xp89zHnlA8ThqYPdRThMXQJ8WGJ m7reARjWlZUjIsavibZrNrBVEJKLmbWNLXW5wBLaZM2iTXiRtOXFMLv2wPQKVs5RnXH00Of5Nr07y OAwb/NMcZ97AMC/ypR6NDTNw+cq6lcvmcNIeaY9GK8i76NKxw6qOC+75ERSWR0vgGfQ0q57HNXyG2 +grYe1m0GWE3irQupaCS4sllRemRmYc2YiRx60PKdSS01RoFI7qCjcit1p+Y1UDeZzk06FIZdWgz3 GlC4u9Rw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSIW-0000000Bs0e-09C8; Fri, 09 Aug 2024 16:17:00 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBX-0000000BpZC-1DNG for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219786; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dHUdq0ZI2fvSzmSbu56kn/TFI4mY4GVPslnR6ZzQETQ=; b=ZJTKl7xfAN3j+FhuK467ve2KlUvmsJVpgwBxssJHnf1L4dJcrNfmLlLqaRR5cFfMIXFIBI LHuWD0eUOuG9JT9IazO3biEBb5MjV/HjsgjirNavXqQU0HywfB6G8IHZvF/u/ovLp1Psng Rhsh51rGVNR4rCyin+I97emBPAcOsZk= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-487-q76ciPmXPI-VHRTt8G8vUg-1; Fri, 09 Aug 2024 12:09:43 -0400 X-MC-Unique: q76ciPmXPI-VHRTt8G8vUg-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-6b7a17dea12so2075346d6.2 for ; Fri, 09 Aug 2024 09:09:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219783; x=1723824583; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dHUdq0ZI2fvSzmSbu56kn/TFI4mY4GVPslnR6ZzQETQ=; b=sLqB0FLHXuZ/Pyokhrie/Nqn6n7UJgoglQAWc4BlbCz1Fq3n+c1iQn7YizM0iqyvLC okNLbPkMhdu5M8TZ6kKv5rpl1VJok9Nz8UAyD8J2LyaMK4VC21exegCd71kCbTM0HXkF nLFEUabPAObe9KnfTS+1wmpKZRzJ1U5dkCDTVUJdlcfScrIGx7Fj5UEvBfscknGmpIJH XH820MBN6qLq3eMECa3J82o1EmQa2Yz7xwuG+TlOzeOoTYIhAVSabdXmBurut4qywGDH zVy3t3NHNerXM8FGLy6JjaDVT2XUi0rVTimWNlJ9weVyTiUEbnIFZEwkozivSyjKjV63 lHTg== X-Forwarded-Encrypted: i=1; AJvYcCVbm+n0lcR3ABliL/4nhlnh0qxJG7FuycOO26X3Hvs1cTmECZ9VglhIcvD3+FqfX6Tcs4GlkeZqndogZZLG4hwNRPKE+xkfHpQd8k2zmAZ2TOX6URQ= X-Gm-Message-State: AOJu0YwCgm9I1xGIiOA3wkHf1Y3K0JK3PgcU5fQLoj6Gf9NssEMCcSMS CW5BrcCJR6qOq+z8rcK5gSQv+B7zPWQprazf9fFEo98TZi15NfiJ4HfnJzsFAwlkIUaXE/SSrj/ qAiE909g3fz9UgJdZ3ikDij/fqZT7y2mBhJdsz8r3UIb1DMb8qdGiNzd01anBJixaLSgdholQ X-Received: by 2002:a05:620a:460d:b0:7a3:5e3f:cbf4 with SMTP id af79cd13be357-7a4c18467b6mr127416385a.8.1723219782745; Fri, 09 Aug 2024 09:09:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFqMMArTR19FgmAq+7tz35tycS6tSaD4MMjthIpuW8WNcff+L3Yj5M2c5Huoz5dZxDipeszqw== X-Received: by 2002:a05:620a:460d:b0:7a3:5e3f:cbf4 with SMTP id af79cd13be357-7a4c18467b6mr127412385a.8.1723219782335; Fri, 09 Aug 2024 09:09:42 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:41 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 12/19] mm/x86/pat: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:02 -0400 Message-ID: <20240809160909.1023470-13-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090947_432839_C7FC1A51 X-CRM114-Status: GOOD ( 14.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/mm/pat/memtype.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index bdc2a240c2aa..fd210b362a04 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -951,23 +951,20 @@ static void free_pfn_range(u64 paddr, unsigned long size) static int follow_phys(struct vm_area_struct *vma, unsigned long *prot, resource_size_t *phys) { - pte_t *ptep, pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vma->vm_start }; - if (follow_pte(vma, vma->vm_start, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - /* Never return PFNs of anon folios in COW mappings. */ - if (vm_normal_folio(vma, vma->vm_start, pte)) { - pte_unmap_unlock(ptep, ptl); + if (!args.special) { + follow_pfnmap_end(&args); return -EINVAL; } - *prot = pgprot_val(pte_pgprot(pte)); - *phys = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - pte_unmap_unlock(ptep, ptl); + *prot = pgprot_val(args.pgprot); + *phys = (resource_size_t)args.pfn << PAGE_SHIFT; + follow_pfnmap_end(&args); return 0; } From patchwork Fri Aug 9 16:09:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E401C52D7C for ; Fri, 9 Aug 2024 16:17:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=kgdRDY3FxKIZdGztTYnhSNYfE/ eXKwMzrcU71mskepsZnBI0TMN5WX3vINIgBGLUlDPLAFDSAxVp0O0nsiIU8tzCVrRkzCBRGBmmXlL SpbJSbkfU6uXIbE4mX4sypxX680xwpJ1on0Xs6fKLKz9O8qCBt1AbzChroRLyR9yNSRQugnpheJ9y V00LxkB3bu2N9p0Wkn0vjPw8jNPhyupoZGRHftTIlVhD8F5gw9dasZxT3O6yI5PIeLl7T1FEO8csf fq8lc02hvudmYj3cqpWfe65SFMOmukRxhj0nZ5Qra402CeAa/fotGx9EO6LP1FvwkCnqBw6LHhJ6a vZiAhAwA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSJ3-0000000BsAc-2vq4; Fri, 09 Aug 2024 16:17:33 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBY-0000000BpaE-45C4 for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219788; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=VYVATtrlnw5KmuwEBRbTmIJvpSCUa1nZGITHn/Vwkhc9bhxQ0/WNP2pIZnPcGgbRX9v78j qoMrw3TCF3S5G3J6UB6B6jSYJuUw0S39shv3VZEw2uNQp+nR0W1UCL8RObJJTDhCa/a+Ph o6N/qJUiDRNbxcOFDAPkRT9MMyjWsV0= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-247-C_Bn9SnBPvS0RRNCl5wO_w-1; Fri, 09 Aug 2024 12:09:45 -0400 X-MC-Unique: C_Bn9SnBPvS0RRNCl5wO_w-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-44fea369811so4530381cf.0 for ; Fri, 09 Aug 2024 09:09:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219784; x=1723824584; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=sDB5hnllDmc1l2mMeGcjsna8jxBCmZmrBCjd3O0oDfNIoMWu0U05ydB0cM6wIoc7FG edwp0ZmNwWIswDs55W7FP1CUhRzCPqaX5EmNI4gNdTTbBFyvO2JVqL7OjTDzUvR6xvca 5Jh4VcIAY+IfJEoRVdixPLud6c66lOoExAQIkEkrhZJd3Y7gEuxL7Sfb1Ne1naU0nxSn JsveMh6qB5iCfnYkBfFTJabFHOxVP8aUghXawbxHtgaxJFhUl+tikpJNUwkB8SBOdpl4 /byY/Nsufo1p2sJ+K8EWPdzuHQVN3pO00w75lOAkr0fU8alfDaNXfoHyB1X6qCNPYpy9 g7GQ== X-Forwarded-Encrypted: i=1; AJvYcCXdH7G2vyNRF8/L4WqdVOP8UxO3Tnljzc96QuLlPpDPyzOznI2c/KE4usRPIff6QVZzFw4tFanPi5V7O9ycTj7NYNlDhivLZGPhcFJ4fiy6ugwcmhQ= X-Gm-Message-State: AOJu0Yxsb2k91fHSMsXpyOGCguKtjFKIshS6fzbCBNAuBcwVjRdhwZgz N6NIn9xutKDUX5GgmWdT4zrTmWz9pLTkNrkR2cs3Dpzh20ay5HuFJOwil4cHwjC+TZCchSYFcKE 4TiUzdaDJOeMMKF+1kkk61c+EBwGyNg4iwVa79Krt4kRaHvbEe2PsXD41isl+cMgWhVRs3dHH X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14745471cf.6.1723219784599; Fri, 09 Aug 2024 09:09:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEFbC3kPrc0fRPufwvcIMbfwNKbDcEyDmf8f6qiGpsAJzwvtAb4AIbgx0osGWWLWwEBOAE+Pw== X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14745171cf.6.1723219784295; Fri, 09 Aug 2024 09:09:44 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:43 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 13/19] vfio: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:03 -0400 Message-ID: <20240809160909.1023470-14-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090949_123983_18C75A50 X-CRM114-Status: GOOD ( 14.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Cc: Alex Williamson Cc: Jason Gunthorpe Signed-off-by: Peter Xu --- drivers/vfio/vfio_iommu_type1.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 0960699e7554..bf391b40e576 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -513,12 +513,10 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, bool write_fault) { - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) { bool unlocked = false; @@ -532,19 +530,17 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, if (ret) return ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) return ret; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) + if (write_fault && !args.writable) ret = -EFAULT; else - *pfn = pte_pfn(pte); + *pfn = args.pfn; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); return ret; } From patchwork Fri Aug 9 16:09:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758976 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5471CC3DA4A for ; Fri, 9 Aug 2024 16:18:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=edSHF5/yxiIyn4DvQ9nHTrcoIS YSTUDY05yThFrUFnMV/Ti2y0iUZq7ZpauohLZjsEOZvkuhyDsnNs0b3llemzDYHS0aJPk7tb2guKP ngXmXbgSwiakaXY17mxmLDO/d8QNB2FTLQefjWMHL+ZGyqGcNEh6PkfYwAZNgxr/MgOwRGHLAd3Wb NRxLrSOX9YAIyVc3buyoeZ4LH534bJZ9v5WwGejjzAVYHnIx5CodTdLhCtlZfwOONyTe5+uItJlLf KY2QAAenoi7nEgWk+OvKt9mNQ61mpjTiDc2EE7Uyi7JVGpMwrKcTry5fX5oP97v9nOMkYwRRYak6A pBrufecA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSJb-0000000BsKB-1ypQ; Fri, 09 Aug 2024 16:18:07 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBZ-0000000Bpb1-1oTG for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219788; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=YFVhFEyZlAk3sG/F6MSx+BQ114LOjPsaCSOl7OHhHXCThWgGLRRHDIyyjKEWdSEo3kK9rO L9e8RrQcqJUquiDmUsWcMulNkwph/ZkRGgQTIHFdyRL3Q73hWx6n3QI1MBpXqfxTWgFKRY Ef5Ff1pbUq5KsDmbvd1fH/NUquY1MF8= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-552--mRK31t7NimjYx1S1w1OYg-1; Fri, 09 Aug 2024 12:09:47 -0400 X-MC-Unique: -mRK31t7NimjYx1S1w1OYg-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-44fe28cd027so3758421cf.0 for ; Fri, 09 Aug 2024 09:09:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219787; x=1723824587; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=bD8SkjnV0ZzXNBsIG/Zu+4azz7VrcWM6KiTp+yuKuUokm4gU/J7ElmeHe2r78e7a54 HF/EGKD7dKQz+h7wrLClLd9msadd7xQ6UBGG3LV4IOmF6TgBJxRBS43+jI1+ePJI0Ocz Qp7tCPPzk2vAeMbu3n6eiD2rqgjuQ51QXoY3sYKamOKhbPRZY8NY4hPXrrkp1XwZrvDo Ubh9NIi0kGqfDMfszQzBnfFWXo2m+AU8LKl4/U6CYLF8LyjIThsqR8qE5TY1jf/7YWk/ 1FdlNr6IMXZg76s974iG2nONCFsChf00on7cg5PxCIhkIssS7+d9DkolQRqowWKRgAiR QEzA== X-Forwarded-Encrypted: i=1; AJvYcCXwP0R2fmND7Y+SUP+94qcI4ZezhMir+jhyIGTFsards3K4anBaRhMai3eU8WlsVIWUbvoU/bRgEbMzk7Dp7QYW@lists.infradead.org X-Gm-Message-State: AOJu0YyNKHsFqmBn3ssfKOEHIZTbFdQUqVcRvtJrT/5odu7R4rU5txUY AuK7gJ8HfKovodOgcdOBSx6gYsI2HW9h4nlQAlwyWiM6sQiHPIDA0JF9TcXTrc9YCDZ8AA0VCq/ VaFwXEJlHmqzAKDOAri+nktwKoX6kVnVE6LXW/f1Mjz/E9Wd6za7PdieteqpZEcVLRd4iqxTv X-Received: by 2002:ac8:5acc:0:b0:450:26ca:9170 with SMTP id d75a77b69052e-4531255b48dmr13596521cf.5.1723219787020; Fri, 09 Aug 2024 09:09:47 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGxpG5xPFbbw3Y9PVyV1kJ47S21qHA1+Fquj+FmgsVP6Zo0aX1jKS3YWr/9l77XMEl3bpPFxw== X-Received: by 2002:ac8:5acc:0:b0:450:26ca:9170 with SMTP id d75a77b69052e-4531255b48dmr13596351cf.5.1723219786668; Fri, 09 Aug 2024 09:09:46 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:45 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 14/19] acrn: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:04 -0400 Message-ID: <20240809160909.1023470-15-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090949_566088_5AF6794A X-CRM114-Status: GOOD ( 14.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- drivers/virt/acrn/mm.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index db8ff1d0ac23..4c2f28715b70 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -177,9 +177,7 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) vma = vma_lookup(current->mm, memmap->vma_base); if (vma && ((vma->vm_flags & VM_PFNMAP) != 0)) { unsigned long start_pfn, cur_pfn; - spinlock_t *ptl; bool writable; - pte_t *ptep; if ((memmap->vma_base + memmap->len) > vma->vm_end) { mmap_read_unlock(current->mm); @@ -187,16 +185,20 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) } for (i = 0; i < nr_pages; i++) { - ret = follow_pte(vma, memmap->vma_base + i * PAGE_SIZE, - &ptep, &ptl); + struct follow_pfnmap_args args = { + .vma = vma, + .address = memmap->vma_base + i * PAGE_SIZE, + }; + + ret = follow_pfnmap_start(&args); if (ret) break; - cur_pfn = pte_pfn(ptep_get(ptep)); + cur_pfn = args.pfn; if (i == 0) start_pfn = cur_pfn; - writable = !!pte_write(ptep_get(ptep)); - pte_unmap_unlock(ptep, ptl); + writable = args.writable; + follow_pfnmap_end(&args); /* Disallow write access if the PTE is not writable. */ if (!writable && From patchwork Fri Aug 9 16:09:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE886C3DA4A for ; Fri, 9 Aug 2024 16:18:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9FTs1XtnGv67Q34nCFPbcUqDR1Dyk8Tfq0P0nZ4zw7U=; b=RLSYegR5ZA3esvrDCU/0s/LXPi YL8PuuANuBkWv1VXbiWu7nUFY1/dinuZu47Stb3w2XjSdcFXHkmTuFTMEYRgZgt9BAAdTz4MBQ3td peDgrxZo7YwYFJD8X0t+HnJwz5AVt0sOPitKXUGq42yhEcyz88D56wJqYfpemIO+lREG4ClXlbdij qoYt952SEXll05feLVu9+3yNpmVJv2Nr2pFIP0xWLPjXiUTWBbduInQlHE4rkj7UaFw5Qw36ZPXYv NX93LH4Q73zO6nLgLFX6Typgih+771z5Mcp04K/AoCHlzQwebccojAEDgpI5Isi42gT3d5k+9qrW1 XeFp9ZLA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSK9-0000000BsUn-2FXn; Fri, 09 Aug 2024 16:18:41 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBd-0000000Bpd9-21Ty for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9FTs1XtnGv67Q34nCFPbcUqDR1Dyk8Tfq0P0nZ4zw7U=; b=Tw4zTLDvHzKryH+MYhj9J69fOLbbw9t92l8KBZJZgQMHHitTAU7Am5mzZVWK4YE4qHcNtF cvK93I1RVm1LyAtyFnjizqoLTJn/kZiPcOmYxBLFUZttLOnUyq2Qy1e6p6kCHo9azn2mlr +yvWmokTQOV612Sae2+f1IL7TssxPQU= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-627-cPm4y48jPSeCZ7bhfDnpzg-1; Fri, 09 Aug 2024 12:09:49 -0400 X-MC-Unique: cPm4y48jPSeCZ7bhfDnpzg-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a1d3c02fcfso7774485a.0 for ; Fri, 09 Aug 2024 09:09:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219789; x=1723824589; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9FTs1XtnGv67Q34nCFPbcUqDR1Dyk8Tfq0P0nZ4zw7U=; b=ljIHQkGAEMdBRWzrIkLjlkHfFGWVc6M1E43SN9UhFzglBWN3iBfSUfSz4+Q2GRKvcv DOlQ5QGUouTkklMhZA3cJC32fGWXVB0OS5BaUhrqPrjRRva/MCIxynZ/iPW9h+n/CPB6 XfxZOfrfEk8NdD0nMudhCiqSNdVC5bK2XcTBVTwSXxlrM4GU2cBQEs2OvczykbvPRCbJ 5Nlpp++HuXg/2a+Bi1cmdXBuRWWOe6gJpxDPYq2fH8Uz15s+yQVQ7C4dyopyGTD/PwR0 Cyii12qe318R0XkBSw5+JitdlBqpsQyNTZUMzooY6nUXwdhDEQyJHIXY3PXLRU9Ch/z8 NXbw== X-Forwarded-Encrypted: i=1; AJvYcCUpfqQUdN5Rvp3N6SdlpUhq2lydpfhTzydl3bkw2u/iyo1khn97pz9AmwFu+uMyOfGtdlKzMAsge5y+W7pdSYVqZ7u/ZXCBQkiMkPnCSWyijcVHcIE= X-Gm-Message-State: AOJu0YyAjm5c048VknE2PcTKScVJdKSUGDpZzbscKysIzkDx2XRbHVMT Ntb9wSP1BTWojQ39zsS9yXmTqsFIBgYvAIPlWsY4Y1SlLj1urFf5dE0v+TtG9xRAzomZSmjkHpx LXKGdheUaR0ASt7uNTesh7KvJRjD/wF5ci4fotUGedKeUKlm79f42L0KO+1kkDdHHa8nY351p X-Received: by 2002:a05:622a:11d1:b0:446:5b0a:e778 with SMTP id d75a77b69052e-4531235a44cmr11675181cf.0.1723219788793; Fri, 09 Aug 2024 09:09:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH3kKOCpVIG6mSA6dMC4Xb4vB8Z6SmUroM26OhSq4nP1v0KE/gOqkuWVbJLnRl3s3aUYCeIEQ== X-Received: by 2002:a05:622a:11d1:b0:446:5b0a:e778 with SMTP id d75a77b69052e-4531235a44cmr11674811cf.0.1723219788421; Fri, 09 Aug 2024 09:09:48 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:48 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 15/19] mm/access_process_vm: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:05 -0400 Message-ID: <20240809160909.1023470-16-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090953_688600_A4F64873 X-CRM114-Status: GOOD ( 13.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- mm/memory.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2194e0f9f541..313c17eedf56 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6504,34 +6504,34 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; unsigned long prot = 0; void __iomem *maddr; - pte_t *ptep, pte; - spinlock_t *ptl; int offset = offset_in_page(addr); int ret = -EINVAL; + bool writable; + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; retry: - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - pte_unmap_unlock(ptep, ptl); + prot = pgprot_val(args.pgprot); + phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; + writable = args.writable; + follow_pfnmap_end(&args); - prot = pgprot_val(pte_pgprot(pte)); - phys_addr = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - - if ((write & FOLL_WRITE) && !pte_write(pte)) + if ((write & FOLL_WRITE) && !writable) return -EINVAL; maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) goto out_unmap; - if (!pte_same(pte, ptep_get(ptep))) { - pte_unmap_unlock(ptep, ptl); + if ((prot != pgprot_val(args.pgprot)) || + (phys_addr != (args.pfn << PAGE_SHIFT)) || + (writable != args.writable)) { + follow_pfnmap_end(&args); iounmap(maddr); - goto retry; } @@ -6540,7 +6540,7 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, else memcpy_fromio(buf, maddr + offset, len); ret = len; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unmap: iounmap(maddr); From patchwork Fri Aug 9 16:09:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7D246C52D7C for ; Fri, 9 Aug 2024 16:19:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=rQ5uhZ2uiWdH1/UlOzJ8LVGX9FzFBDg5i1FarWNBfgY=; b=xz5UiRIwi2zEzDIGR8v4CbiLx3 /foSdEfFC6a7m1wMMRRDCL5p0vsf8FOH5vzMAB2G0U9/r5+jHghPgiAxnpr+Ktx9gDPoUvZJ9Yakd BKu2aftrEeMJGrHTDxZRiopaP0winK+iOJUaoeOlijyxqUlYoMa7pTPoxJ2j4S/cehaG3KgPHt1d6 oSq5JGOrkq3Xs6GmQLk9tGNfW42m0V9RfW3i9Sqfg19W61KliP1kZw1vfSH1F+jT3lMRmQVt5hEn5 NxRBnj1NuHa6Uo4+KPBh8HgO8wMgjpZ+oT1mLejctWLSP3hRbf7S/r6/xfUG4tnl6F/QGfxc056ZT wgAZYaUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSKh-0000000Bsey-1laQ; Fri, 09 Aug 2024 16:19:15 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBd-0000000BpdH-386j for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rQ5uhZ2uiWdH1/UlOzJ8LVGX9FzFBDg5i1FarWNBfgY=; b=aElG6i26o3qTujyZaeRIgxw7WlSKDdiVol10gkPV3eDjoBF9R+EaolFsAWLz1y7hkkkj8d uZMYhKsqAruT6ekX8Cc+XwHEGGXZnHLmVpbUS9BkTkQiAdplLkJOxwtLseSin/hfSP4V5p 72WLHZOxDWXLorKBdA2o7RJZbBErt0Y= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-80-5PgEOGzeMkygKkKAWgW9SA-1; Fri, 09 Aug 2024 12:09:51 -0400 X-MC-Unique: 5PgEOGzeMkygKkKAWgW9SA-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-44ff25bbfe1so3070561cf.3 for ; Fri, 09 Aug 2024 09:09:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219791; x=1723824591; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rQ5uhZ2uiWdH1/UlOzJ8LVGX9FzFBDg5i1FarWNBfgY=; b=AqYgNf+LWz+7O56EC4vnHaoel6q7ifrEU0ZGDW4J4TAhnx9uBPgJy/mNwWyFqqYUYP OHN3lMOkOJHzRcT0ufQvy6C1fWcWh3Y8yegOsQx8HJ+IZ0SCEWXPDvQRf5F5Syvc26ae DskdbgGMgBHc9VRMGVA2CSDn1c6l+nkLXp5Tt+2uBfe7n5psFMi1ojxDmt/Qn44x/jb+ 1C284tDRawF8NwQpelHIO5DT1bJNc61YShPwT30QXGvo3YoqQaIWULaN+CbfoFc+9H/S YhsDWnj8Lf+Gn28bLpy95qaEzbjL9rQZxJcAmtRkF/xOfFX6tNrlDufH/YWwaWEYuQGP dCqw== X-Forwarded-Encrypted: i=1; AJvYcCWBPg8pnydGyJ5N09Bt131r8dt9wAv8HEyUQjLJs6EVizg8rziOKpWcUfnpkSMF4Pn0jKJ7fg6kYKCBtmxwXqyt@lists.infradead.org X-Gm-Message-State: AOJu0Yxg3bkp7EHTIGR2muh6BcSh+iIv8lJrfA/ihICfhrCAmCuJQEx3 hitF/jD2vNHH0pI3ASdCaNW1eoliEd+t8rMiT7TEaOCYyJEALBKx7kLzkFUGzJ9P52G/DsAvzws qhNRAZedkWzdUc83cTk0Kd9XZzqInI/BzBVHAFVBMuDz8/2vX4C1XSOYBXfgGeEBVHYuWSELq X-Received: by 2002:a05:622a:412:b0:444:b935:ebb5 with SMTP id d75a77b69052e-45312359fbcmr13373541cf.0.1723219790800; Fri, 09 Aug 2024 09:09:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2UCzm8Bb1pveediLGLwWSJm3e36K8ZWtfh9bSQqsAnCTBAfRwieQENSqzRtCTpHEz3lk41Q== X-Received: by 2002:a05:622a:412:b0:444:b935:ebb5 with SMTP id d75a77b69052e-45312359fbcmr13373301cf.0.1723219790408; Fri, 09 Aug 2024 09:09:50 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:49 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 16/19] mm: Remove follow_pte() Date: Fri, 9 Aug 2024 12:09:06 -0400 Message-ID: <20240809160909.1023470-17-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090953_999408_FEEDA4DF X-CRM114-Status: GOOD ( 18.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org follow_pte() users have been converted to follow_pfnmap*(). Remove the API. Signed-off-by: Peter Xu --- include/linux/mm.h | 2 -- mm/memory.c | 73 ---------------------------------------------- 2 files changed, 75 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7471302658af..c5949b8052c6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2369,8 +2369,6 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp); int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index 313c17eedf56..72f61fffdda2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6265,79 +6265,6 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) } #endif /* __PAGETABLE_PMD_FOLDED */ -/** - * follow_pte - look up PTE at a user virtual address - * @vma: the memory mapping - * @address: user virtual address - * @ptepp: location to store found PTE - * @ptlp: location to store the lock for the PTE - * - * On a successful return, the pointer to the PTE is stored in @ptepp; - * the corresponding lock is taken and its location is stored in @ptlp. - * - * The contents of the PTE are only stable until @ptlp is released using - * pte_unmap_unlock(). This function will fail if the PTE is non-present. - * Present PTEs may include PTEs that map refcounted pages, such as - * anonymous folios in COW mappings. - * - * Callers must be careful when relying on PTE content after - * pte_unmap_unlock(). Especially if the PTE maps a refcounted page, - * callers must protect against invalidation with MMU notifiers; otherwise - * access to the PFN at a later point in time can trigger use-after-free. - * - * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore - * should be taken for read. - * - * This function must not be used to modify PTE content. - * - * Return: zero on success, -ve otherwise. - */ -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp) -{ - struct mm_struct *mm = vma->vm_mm; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - - mmap_assert_locked(mm); - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) - goto out; - - if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) - goto out; - - pgd = pgd_offset(mm, address); - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - goto out; - - p4d = p4d_offset(pgd, address); - if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d))) - goto out; - - pud = pud_offset(p4d, address); - if (pud_none(*pud) || unlikely(pud_bad(*pud))) - goto out; - - pmd = pmd_offset(pud, address); - VM_BUG_ON(pmd_trans_huge(*pmd)); - - ptep = pte_offset_map_lock(mm, pmd, address, ptlp); - if (!ptep) - goto out; - if (!pte_present(ptep_get(ptep))) - goto unlock; - *ptepp = ptep; - return 0; -unlock: - pte_unmap_unlock(ptep, *ptlp); -out: - return -EINVAL; -} -EXPORT_SYMBOL_GPL(follow_pte); - static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, spinlock_t *lock, pte_t *ptep, pgprot_t pgprot, unsigned long pfn_base, From patchwork Fri Aug 9 16:09:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13759001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B007DC3DA4A for ; Fri, 9 Aug 2024 16:19:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YoGoBxg1RIhRUWN8551lXVuosUhfDUfYXVK/Rz8tB4Q=; b=v1f0twkBczIKvGCazafyHcqE6T t1oJj62C9ewbRmzl637z250CugpP5ogjt3rNXueqV99RFDqYTdz/QhSyjbTrB5mkEY+NJZFANbVpi h04NqMMiE+MdeJw2D92losyMc7QjMJAxOYRbKoyyGD2biOhVaKXUCtm0kVJRNcOv/PNt6rokEBsoU IQ1CKEVnGdVgStKmF6IH0cBGJQsFSUHY/xTUeTtp/vUJpopugR99XV4baH7J/Ihiz4MOP4AXgf/A6 YVbYtRl7JHphAqK5dzt9BW6qrq3TdXkIKogt0rUzS0nu5FJw8+f91wS3ZMxM3nYwCAf5QQAQiy5Xd x0731c5w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSLF-0000000Bsnx-0f0X; Fri, 09 Aug 2024 16:19:49 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBf-0000000Bpeu-2w5p for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219794; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YoGoBxg1RIhRUWN8551lXVuosUhfDUfYXVK/Rz8tB4Q=; b=YLFBFgxFqOjvnP9gVdywNLyR/NfA2e7MRz0havdBRWuD7YRvw5u9AwZG6PX1jDJvp9PzVe kUiylYybluMpABAnqMdVsjfhAySJE044Aqr5/Sf1yrqy9GCXCE25u61ezo8WlogH2cV8gt k5Esc5s/0+xyjyfO93s5aG8jKvuDbds= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-678-Eep06HT7O9mpa8WGEy_LuQ-1; Fri, 09 Aug 2024 12:09:53 -0400 X-MC-Unique: Eep06HT7O9mpa8WGEy_LuQ-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-45009e27b83so4906291cf.1 for ; Fri, 09 Aug 2024 09:09:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219793; x=1723824593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YoGoBxg1RIhRUWN8551lXVuosUhfDUfYXVK/Rz8tB4Q=; b=Cv8R5kXRNPxLYITaGTt7K2c1M1wz5sR9Ll8g4WG6ySM7e2Bg7lYBQFF6HwZVTz6WY6 yeFw7YX7uAXMvXJ1v2n2K6QXQ0g40phH7gQjSaa50FWgRfi5mGpnaOc3r7XsNepZQD+8 fCyKKpugX/gSzgBtgneScyTOm23oqRiav9r/0WpUUSySbrqLXP9fAqenYJ+oyxWA9PeD cQS5SoBA3F4YgEdy8cKj02xyAgjLTCvqnhrzsyQv/hE8U/arDExjAevjENozl6Z0f8xK Yh+qRhzKEmNEbiXj97wx/Vl8zODpLgp+br9M5pzvdPVLachWYu17AffGgjUIKJ58/6MM nYWQ== X-Forwarded-Encrypted: i=1; AJvYcCXKJpjOgqdDJNLUXaB29ZgKq7v5jTsbrJPRR2Pt5Zm8JSvt7ePlgoV5+3VGPIV6KUOzidfkDdS0EZRaoReg+PpnwO/QPLVxH9L1cMXOlCmZhXAcLgA= X-Gm-Message-State: AOJu0YwhuHQWYHFZ2Zwo+upVyFj5uI8YsfajuzGEKplLMKwawIAIQCOZ uheHaWI7Rz5k3iYmAqJgW+coqAJZyz69KPbfe0qpZtJEEP6R7xDrsulpxRnYfkuzJlUy1z1w23E 7b6ryl+ICPujEPyq463H8Bom1J0o2oYsXA1D2tXa9TT0Vj5T4L5WDj7qERE4Mu0XR6VHxJCzT X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14005691cf.7.1723219793235; Fri, 09 Aug 2024 09:09:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHyC9JbjVCeQC7LVvckqh3sKVh2QUlZ0IzLJIVdarr6jR5vj53aOO53iHwQIsrAsjgq+0MAjA== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14005321cf.7.1723219792685; Fri, 09 Aug 2024 09:09:52 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:51 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 17/19] mm/x86: Support large pfn mappings Date: Fri, 9 Aug 2024 12:09:07 -0400 Message-ID: <20240809160909.1023470-18-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090955_921649_E8CA4838 X-CRM114-Status: GOOD ( 17.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Helpers to install and detect special pmd/pud entries. In short, bit 9 on x86 is not used for pmd/pud, so we can directly define them the same as the pte level. One note is that it's also used in _PAGE_BIT_CPA_TEST but that is only used in the debug test, and shouldn't conflict in this case. One note is that pxx_set|clear_flags() for pmd/pud will need to be moved upper so that they can be referenced by the new special bit helpers. There's no change in the code that was moved. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 80 ++++++++++++++++++++++------------ 2 files changed, 53 insertions(+), 28 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index acd9745bf2ae..7a3fb2ff3e72 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index a7c1e9cfea41..1e463c9a650f 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,6 +120,34 @@ extern pmdval_t early_pmd_flags; #define arch_end_context_switch(prev) do {} while(0) #endif /* CONFIG_PARAVIRT_XXL */ +static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v | set); +} + +static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v & ~clear); +} + +static inline pud_t pud_set_flags(pud_t pud, pudval_t set) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v | set); +} + +static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v & ~clear); +} + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -317,6 +345,30 @@ static inline int pud_devmap(pud_t pud) } #endif +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SPECIAL; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SPECIAL; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud_set_flags(pud, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + static inline int pgd_devmap(pgd_t pgd) { return 0; @@ -487,20 +539,6 @@ static inline pte_t pte_mkdevmap(pte_t pte) return pte_set_flags(pte, _PAGE_SPECIAL|_PAGE_DEVMAP); } -static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v | set); -} - -static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pmd_t pmd_mksaveddirty(pmd_t pmd) { @@ -595,20 +633,6 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); #define pmd_mkwrite pmd_mkwrite -static inline pud_t pud_set_flags(pud_t pud, pudval_t set) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v | set); -} - -static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pud_t pud_mksaveddirty(pud_t pud) { From patchwork Fri Aug 9 16:09:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13759002 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3059C52D7C for ; Fri, 9 Aug 2024 16:20:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=vcVNn0/to5tbioyAEpFiwB7dHN 930/G86iAQuEC54u6cdWEtSPEqw/PKi99h/7q6H8QVqspjkdhAMopbUvqFv8ddLARfLfiQi22nkR+ eXkdA3+0+1//I3l8xJl4oUxgMUToe+TVmYD8grqPnrgRT+qO/BqreiJvE0+RpTIhYLkfRBa4GxGB/ JgQ887Xep5fYbxYuGk8re2OYLfqgeF81cvBTkZrwQXlK7CfU6K5dSwvi9JX4DPg5a2EJXR08i7BYT XbMTLog21rS2xkALbuaEd5DFdCjGjKcAAX/u8YabOkwMTZmtM+uB512sqnixoll9WFHGXQFldO87q /gHd2L0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSLm-0000000Bsvh-2sOz; Fri, 09 Aug 2024 16:20:22 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBh-0000000BpgK-31N7 for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:09:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219796; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=awElHe3ixtN9NVlv03OfaUPbqTJSdnQxIG2NwfT5iCUNwWrDouXuLtg/EMvqBfNSLtCKgG IoDJmFN310WvvGM6p6lGhY2G8THLBZSXm1aUjgAeO1pSsQXOrkaUTBYLOeJjGqH/ThdE8e hnNcFrCSfb9ww1dO+ZpqnwvyPtEp5zk= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-37-m5om8dLsO9-hJDc6ocr2vw-1; Fri, 09 Aug 2024 12:09:55 -0400 X-MC-Unique: m5om8dLsO9-hJDc6ocr2vw-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-45009e27b83so4906341cf.1 for ; Fri, 09 Aug 2024 09:09:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219795; x=1723824595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=s/QV9uj76cBR9bQrbpnkhih3ir+6qnOgEmeDr9+qB6nOqYLfC0nSW36WWGl0nHIql0 FsVEt71X3ojiHLlEqZpObRubmmX9V5kK7WCfdN+Dl8NmFunvLrDilArVLo5M64QBV9OD Dod/UHBRxbvOaKAxtdKQL9Bbg0G/p8DjKlwAf6hK1l2FJ9XHcSqOWkCUffj6CnJ+QOYm NSmIowQeV9jlvnu+RAFtzvhJdzfcqeo3lx4YmNK2jpBFUbFBvoCczzZQPH4gHnbDEjaJ sIc3VJ4WCMiITykXOowhyBvaNqhjrfj9Ggg8LR7UKBqcnoSdcoG01w8UwfVcCcx4XaYm EFmg== X-Forwarded-Encrypted: i=1; AJvYcCXEhn3qqv01zS63BWXjEZSD8V8tm9cKDE14drjgZm54ENL9Heo/utk/qftkgMnUhkEQubRkzVmj1teVG/tWyISEfiNsjEcpRUgWB8A56EBDtHcxoCE= X-Gm-Message-State: AOJu0YwfxnQHvbJnghuN61gOVgbK98MNyNgPncMkC/DxkEzbw8WbPIg7 E+rsFa640CmC0TpWZTf8/w7LcmSzXHNa4YYtg5f99OtqsYC8djMNhpyS9RqowHK1by2pDzYMWhv xXn2GxthuWKm413pOEDxoTleFwZ89rhjhia9rAHAqCgsnLaSxDptkmioU6RaBlErc3rt/JQr3 X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14006691cf.7.1723219795212; Fri, 09 Aug 2024 09:09:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEzDYjFq+v9SYAdrw2G1PaR/i1nnMK5UGwVdyzJwk9mK3TcvaabWRsZT43a6L+rPfA/Jc6OAg== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14006381cf.7.1723219794745; Fri, 09 Aug 2024 09:09:54 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:54 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 18/19] mm/arm64: Support large pfn mappings Date: Fri, 9 Aug 2024 12:09:08 -0400 Message-ID: <20240809160909.1023470-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_090957_903398_C4802558 X-CRM114-Status: GOOD ( 13.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b3fc891f1544..5f026b95f309 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn = pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr) From patchwork Fri Aug 9 16:09:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13759003 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5A5BEC3DA4A for ; Fri, 9 Aug 2024 16:21:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=adY9dxsnyZll1ULB8Vgr1o/yxf /Lg1lS6zYXnOvIZRG6XECN3K33/Yw000nF1yR65lORlfymuqKW9lher3XW800OwKeJh193Ip6vArw NnlJIpIA31C8Vs2Y/dTq8K/kRDCeBtgjiyXnE2e/PA9Ex9PXFz+XGO3S3r71RSauxWXPSCWYtxQE0 rZ1V4XToIljmHLaaM145zjvTmR5Di1Yrh3EVp+8FeMJyQAilY/uxoThlnQ42StFbaQLvIayD4IDhw ZUArcWj6ygZqsuD0iQC3mCobjWhj5uPWwphoIKwlq2kfUw9LMyw7docJfX+bdfOaBUFm5NK5rayPi AVWJYTEg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSMO-0000000BtBi-3mgm; Fri, 09 Aug 2024 16:21:00 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1scSBk-0000000BpiA-2aZz for linux-arm-kernel@lists.infradead.org; Fri, 09 Aug 2024 16:10:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219799; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=S2sOimJa8p6RctdbGnPR9rKWFA3WMrsMtuRW93puUjg/BZ1bjg/Ieomfbu85nRs8cfMdxI WL9slw0xyWKb271NAndbMq/N7WY04kn9CiulQ43Qn4igLmPnZs66xHPn0oEn3Ymu5NzpQb hzWeZUaYyLyjVi28K186ryNohoMb8QI= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-156-zI1QCG9FNqSYlVF7AKiGFg-1; Fri, 09 Aug 2024 12:09:58 -0400 X-MC-Unique: zI1QCG9FNqSYlVF7AKiGFg-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7a1d3c02fcfso7776685a.0 for ; Fri, 09 Aug 2024 09:09:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219797; x=1723824597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=jJex9pbmUIHkQ+dO9e2MMAmEU/Ph/qdv8Tu3Xi17uAZu7PSD8KCOkBzyYDwKKSqqkq zk54PQpIjnhQQbatTwnTaNjDoTD2kmA6NOyVknPRb3v/Rz77AVWcntjLD91HBRzAx/eO FQ4GDD5uCIUVtydZYJmYrGSf7wV3Ek7zUgzJSIOtE+/7ApwlW1cgujPAQTBixcUmleRm ou8DZUztGQc+QRWsxcKGKsBEppDJNGOsGER7vIzGrpjijxJsnE9wuT3sLm6Xv4ea32kR zOwOSJ6L8u6bVpGVWrN0/5zUJ0iEHt1pa7+nYAslBjBkV1rGKuhWlOhHx5c9n040yeCb NT8Q== X-Forwarded-Encrypted: i=1; AJvYcCUgS4ZtcGMV4CV9m0Oyih9ZCAUMk1n6H0ZrpbU2wHkfpGFtE1hbe49EwlHZWMZU2GyB/Q/DXKEIbE2f3xv/76IG7DRxMuXFFjoOuUwmueK2jzcxhOM= X-Gm-Message-State: AOJu0YxtonqpEhzsEfN6LjcyjbL+lLKu0tnWnW/qh0REUxrvcu3GGe2D p2J4jq2zvW/Xp0QeatQLgLG/O7AzBoUlwEatN8R8x+Lx71mHCZbkWmt3noujQ4VxVi1ahA99Sz0 rIdfyNzYVhrgoZV0Fbj9p/yaLghJ9m59IhXXlgd7YFNGofTJrr3Z4vsIrEE9Pwbugom8oe2pd X-Received: by 2002:a05:622a:1b8d:b0:446:58f5:e6ff with SMTP id d75a77b69052e-4531251d52cmr13545091cf.2.1723219797250; Fri, 09 Aug 2024 09:09:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHJY3GNDsLRixrwL3Yd/1vgrL/iS2SbMHaD3fo6KrtERFLLt1PJ4ncszNtj/1mVMrjvOJRHSw== X-Received: by 2002:a05:622a:1b8d:b0:446:58f5:e6ff with SMTP id d75a77b69052e-4531251d52cmr13544641cf.2.1723219796830; Fri, 09 Aug 2024 09:09:56 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:56 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 19/19] vfio/pci: Implement huge_fault support Date: Fri, 9 Aug 2024 12:09:09 -0400 Message-ID: <20240809160909.1023470-20-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240809_091000_759402_B9A7196D X-CRM114-Status: GOOD ( 18.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Alex Williamson With the addition of pfnmap support in vmf_insert_pfn_{pmd,pud}() we can take advantage of PMD and PUD faults to PCI BAR mmaps and create more efficient mappings. PCI BARs are always a power of two and will typically get at least PMD alignment without userspace even trying. Userspace alignment for PUD mappings is also not too difficult. Consolidate faults through a single handler with a new wrapper for standard single page faults. The pre-faulting behavior of commit d71a989cf5d9 ("vfio/pci: Insert full vma on mmap'd MMIO fault") is removed in this refactoring since huge_fault will cover the bulk of the faults and results in more efficient page table usage. We also want to avoid that pre-faulted single page mappings preempt huge page mappings. Cc: kvm@vger.kernel.org Signed-off-by: Alex Williamson Signed-off-by: Peter Xu --- drivers/vfio/pci/vfio_pci_core.c | 60 +++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index ba0ce0075b2f..2d7478e9a62d 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1657,14 +1658,20 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma) return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff; } -static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) +static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, + unsigned int order) { struct vm_area_struct *vma = vmf->vma; struct vfio_pci_core_device *vdev = vma->vm_private_data; unsigned long pfn, pgoff = vmf->pgoff - vma->vm_pgoff; - unsigned long addr = vma->vm_start; vm_fault_t ret = VM_FAULT_SIGBUS; + if (order && (vmf->address & ((PAGE_SIZE << order) - 1) || + vmf->address + (PAGE_SIZE << order) > vma->vm_end)) { + ret = VM_FAULT_FALLBACK; + goto out; + } + pfn = vma_to_pfn(vma); down_read(&vdev->memory_lock); @@ -1672,30 +1679,49 @@ static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) goto out_unlock; - ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); - if (ret & VM_FAULT_ERROR) - goto out_unlock; - - /* - * Pre-fault the remainder of the vma, abort further insertions and - * supress error if fault is encountered during pre-fault. - */ - for (; addr < vma->vm_end; addr += PAGE_SIZE, pfn++) { - if (addr == vmf->address) - continue; - - if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) - break; + switch (order) { + case 0: + ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); + break; +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + case PMD_ORDER: + ret = vmf_insert_pfn_pmd(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP + case PUD_ORDER: + ret = vmf_insert_pfn_pud(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif + default: + ret = VM_FAULT_FALLBACK; } out_unlock: up_read(&vdev->memory_lock); +out: + dev_dbg_ratelimited(&vdev->pdev->dev, + "%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n", + __func__, order, + vma->vm_pgoff >> + (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT), + pgoff, (unsigned int)ret); return ret; } +static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf) +{ + return vfio_pci_mmap_huge_fault(vmf, 0); +} + static const struct vm_operations_struct vfio_pci_mmap_ops = { - .fault = vfio_pci_mmap_fault, + .fault = vfio_pci_mmap_page_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault = vfio_pci_mmap_huge_fault, +#endif }; int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma)