From patchwork Fri Aug 9 16:08:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758921 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F1AF1990C7 for ; Fri, 9 Aug 2024 16:09:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219760; cv=none; b=nQyHW/1lLNvNVCwUhX7+Z4QkF6S8ljXAyhGKZ6LSJevBapOIRMX7mwr7hZox2s8LiNcz/CtURZdAFCS6jkw3d59j/DuSsteSLILbJO56Ow8TeCekXkNIu2av1g+tBQPyGBXmUqAcVdCafNGCagSo9mKMiZEwNz0x3I3YZC20DiE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219760; c=relaxed/simple; bh=6BzQDInfC57JnWWffubPvoRSjDM7Wzn3qARHvi6r3og=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EezvkcEPAVc4oJvS8buOWuhIfQOe9RzcUIqzQ4H2fmumJaTt9nwv/zccU8JJhSi02s1QFQrJ5iVcznZUdyScBo+SFllaiYrR6mY5RpzNwMDrzQ0SLPv1riLEAW655xv+Wqq8HPg8UXajVnxRhJNEuoIOliheFK5Gf10wKv8MECo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=IHSZAnXD; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IHSZAnXD" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219758; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fKim6v00HbGyszP90Y55Ixe8BGOuNlK0Kt5OSqhmIXk=; b=IHSZAnXDEI1RUl3SEJ44WYgpa3FsRsc8rxYXz+UIFlk/+4CB/LjZl2vejxzr3e9Vx9dyxH z866pgNJjZTXdXiSE/0fVeQolByIN3/mKfkH1Xpd7B+h3KCVFvmJuw5qtupNiXwXeS55BF 1XgYUhBic42a7qZWKpuUrxRl7dh7PGk= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-486-x4MpwDEqPOWAipCExx2hqw-1; Fri, 09 Aug 2024 12:09:17 -0400 X-MC-Unique: x4MpwDEqPOWAipCExx2hqw-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44ff65342daso843661cf.2 for ; Fri, 09 Aug 2024 09:09:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219756; x=1723824556; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fKim6v00HbGyszP90Y55Ixe8BGOuNlK0Kt5OSqhmIXk=; b=T50poobiSmcxQjH5/LDtmsMKLV69TbvJDaJVQnNhnFCpdgcvi5JEFx6YCHhpIjXG1V fFY8NSAYdVVr6ukoHh1iwKpBOQnpVrYGR7t+40U6BLyIlJjAq3iTBj3FqXdvPIMt8ort u07AYrEajsjoTkDonbEEcGfcuy6pal+/YvoRcvDFfmBWngUZx7a4oEcCIgV2//hzu9oR 5Z3k+2zG8BHpQpRHEbTJ8/oER9FvYO01HECYstgAcBN8bRaUFyaTd718CPzSwDDfkHeg vMuf98e51Beovrc+ZFWdFVBTnpvK2Xdvb67Yj281ompb9B7cuw4aexBd9h2Yup72ftc7 vkcw== X-Forwarded-Encrypted: i=1; AJvYcCVBEQ1/Czz0NgVMH5iX2Esst8kW1vAbg3bmYB9/qzbwsTUfBBM0I7MPjKuwgbZvGz5Axt3ZMirw+shDmxJtQouxjMle X-Gm-Message-State: AOJu0YyDqO4TlmWGX1KQUAQjEmk7YSmkypUYddLA6IPux/2g63k60WlI BjzhRxIqCIT4FVPtT7evVAkVFjWhngDZIHNZyqj7Y5Z4HG8yS8saOdfSw9vGcBVBBHkMDWjD0Ex 6sRjrQeLn3ueLsSs3BY7XVuZLFzqEIurQPTF3f8YAaFon8cc/Cg== X-Received: by 2002:ac8:58c4:0:b0:450:1e5e:3f3f with SMTP id d75a77b69052e-4531255bb38mr13611051cf.5.1723219756646; Fri, 09 Aug 2024 09:09:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEAN0G3ozf6o5j8NgDrSR8g+6fB/Zkz8yhBLk8I4VWUl++aRhUduuiwR699bOA3QLxmHpo6kg== X-Received: by 2002:ac8:58c4:0:b0:450:1e5e:3f3f with SMTP id d75a77b69052e-4531255bb38mr13610781cf.5.1723219756198; Fri, 09 Aug 2024 09:09:16 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:15 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 01/19] mm: Introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud Date: Fri, 9 Aug 2024 12:08:51 -0400 Message-ID: <20240809160909.1023470-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This patch introduces the option to introduce special pte bit into pmd/puds. Archs can start to define pmd_special / pud_special when supported by selecting the new option. Per-arch support will be added later. Before that, create fallbacks for these helpers so that they are always available. Signed-off-by: Peter Xu --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/Kconfig | 13 +++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 43b40334e9b2..90ca84200800 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2644,6 +2644,30 @@ static inline pte_t pte_mkspecial(pte_t pte) } #endif +#ifndef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return false; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd; +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifndef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return false; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud; +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + #ifndef CONFIG_ARCH_HAS_PTE_DEVMAP static inline int pte_devmap(pte_t pte) { diff --git a/mm/Kconfig b/mm/Kconfig index 3936fe4d26d9..3db0eebb53e2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -881,6 +881,19 @@ endif # TRANSPARENT_HUGEPAGE config PGTABLE_HAS_HUGE_LEAVES def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE +# TODO: Allow to be enabled without THP +config ARCH_SUPPORTS_HUGE_PFNMAP + def_bool n + depends on TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PMD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE + +config ARCH_SUPPORTS_PUD_PFNMAP + def_bool y + depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + # # UP and nommu archs use km based percpu allocator # From patchwork Fri Aug 9 16:08:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758922 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2B841991CA for ; Fri, 9 Aug 2024 16:09:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219763; cv=none; b=Zs7rVONKL/BFUaO8awJp9jCL7EVBKvsMJtqGth5eD+XzhGdFFsi7Loor3WZjT2GeKkHHjEwip4ekNtFfYfOmA7w6ZZLl2WjzJZROLd8tlPYQsPlhFlaVT0l0CFyEBQ16Odr66GHxFJf1MOmoaK3d5g2v1IwxK7iAeTJ5UxAoqyc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219763; c=relaxed/simple; bh=AwZNgy93B60KruQhwAcu4Flier/Awee4T8kw3wLwJnY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gqb7c8TTMw4mQaOK7qdHGxLW8vq1VdBUPSlbCSAdmtHJG3yF0Z+zIlDgdzxkCxL9OjEuar+2CKgZfolNWZ5QgJbfMknfygFEhPicxINDLlQh66Ysy5Cd9bffKV2ijizRtFk58ZH4riF8Rtw4OmgUrhS9Sg1qs9sCOuLXDievY1o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=b8bDZtRJ; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="b8bDZtRJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219760; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xVA/urUlESz7em6kHS5QV39/c1UeLmTRXD3YGysY/mU=; b=b8bDZtRJmVC16S7/XV4ED8qVjJnw0txmyQ3nepAa26xTuo9GxpTMaQoNIsnAHQw4LXtOpF twPyvccHe930hStJAnRi4fhR/b6LeLVYf4hBzWK66ZyRWf9bIhyD0k5Qgq0VaD6umoy0kR sbuApSN9XIhiHKJfkzfOwfLB0Su2wQ0= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-237-QQ046IouP9W-uFHx12f9_A-1; Fri, 09 Aug 2024 12:09:19 -0400 X-MC-Unique: QQ046IouP9W-uFHx12f9_A-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7a1dd7e9a72so5250585a.2 for ; Fri, 09 Aug 2024 09:09:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219759; x=1723824559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xVA/urUlESz7em6kHS5QV39/c1UeLmTRXD3YGysY/mU=; b=OSDzbg4IHK2krwUwrXxtQBcKHmwOPBCWYAdBS90N8GtO0wQ78jtzOAyfbg3LqaA/Dn x4tgLkiwmGVGajJi4BVUKuyxdPbz8izBBbXZ+jOGNGwCbeRVFhw47WxA89ZvjBXhqzzm AtLeFf93wsHkm0zxuavuIMs0BWJVdpe+HdR1PkZA6VD37OQRcffkMcfmAxm6nmUWr7um lc5+Y7Zj8EfpnsBuGzb3G77jjDZwju3sN++i2PRmdCr0exrMUTgoJ6KK1LPWbdAQaYNb ppj0pKCISOEeNoTxzxY3dSSz4guyn8ebmADlNNCjX5pmSY7Q8bf2Me/qWHct4uG0JecQ u2Jw== X-Forwarded-Encrypted: i=1; AJvYcCVH6YWAlksaO49I98fiJXkFW+BqvR887WP2how0sVAhfwymFRP50T+8FL04oAbiAVa7bYMyM+T3OJko5rtoWsMQR5As X-Gm-Message-State: AOJu0YzNe/e4vgnqeMhhq7lSEnDbo80Nf8CFRiQYlm/kGVk/tdx+rrxd JgaZT3J9DeM0bu1Q6t5GJQKjoL6HhGzi1Glt3nFstzwAvd+zMIjWLpd6i/Sr++W7BlbsqmahbVt ZthnqefqR65W0eQB4UAVZgdGAFrkXP4w4pT4zufDr+Xl9dxvAMw== X-Received: by 2002:a05:620a:2906:b0:7a1:3ff9:1e1 with SMTP id af79cd13be357-7a4c16af689mr138618585a.0.1723219759086; Fri, 09 Aug 2024 09:09:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHesXKAH7m1pKPVMriR5+O5cSg1VUsP7Em6MavFjtwcHLSKYNYbUt0UMVXsf+PUo1OYJsvsTg== X-Received: by 2002:a05:620a:2906:b0:7a1:3ff9:1e1 with SMTP id af79cd13be357-7a4c16af689mr138614585a.0.1723219758659; Fri, 09 Aug 2024 09:09:18 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:18 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Matthew Wilcox , "Aneesh Kumar K . V" Subject: [PATCH 02/19] mm: Drop is_huge_zero_pud() Date: Fri, 9 Aug 2024 12:08:52 -0400 Message-ID: <20240809160909.1023470-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It constantly returns false since 2017. One assertion is added in 2019 but it should never have triggered, IOW it means what is checked should be asserted instead. If it didn't exist for 7 years maybe it's good idea to remove it and only add it when it comes. Cc: Matthew Wilcox Cc: Aneesh Kumar K.V Signed-off-by: Peter Xu Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe --- include/linux/huge_mm.h | 10 ---------- mm/huge_memory.c | 13 +------------ 2 files changed, 1 insertion(+), 22 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 6370026689e0..2121060232ce 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -421,11 +421,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd); } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - struct folio *mm_get_huge_zero_folio(struct mm_struct *mm); void mm_put_huge_zero_folio(struct mm_struct *mm); @@ -566,11 +561,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return false; } -static inline bool is_huge_zero_pud(pud_t pud) -{ - return false; -} - static inline void mm_put_huge_zero_folio(struct mm_struct *mm) { return; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0aafd26d7a53..39c401a62e87 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1245,10 +1245,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, ptl = pud_lock(mm, pud); if (!pud_none(*pud)) { if (write) { - if (pud_pfn(*pud) != pfn_t_to_pfn(pfn)) { - WARN_ON_ONCE(!is_huge_zero_pud(*pud)); + if (WARN_ON_ONCE(pud_pfn(*pud) != pfn_t_to_pfn(pfn))) goto out_unlock; - } entry = pud_mkyoung(*pud); entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma); if (pudp_set_access_flags(vma, addr, pud, entry, 1)) @@ -1496,15 +1494,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (unlikely(!pud_trans_huge(pud) && !pud_devmap(pud))) goto out_unlock; - /* - * When page table lock is held, the huge zero pud should not be - * under splitting since we don't split the page itself, only pud to - * a page table. - */ - if (is_huge_zero_pud(pud)) { - /* No huge zero pud yet */ - } - /* * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. From patchwork Fri Aug 9 16:08:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758945 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D33919924D for ; Fri, 9 Aug 2024 16:09:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219766; cv=none; b=QI84c55+4vU1pNE9ZRIowCo4P28F00Q02iko/1HPfWvshdruoT/qIRmIl5zsWz2okKFVPn6Aymja+myp6z4XbEOn1ebvmourjIjjHuIePB0VCFXD2ilmVCxrkO6jNgKG6upO+oxucHhRwaBxn/co77w9SGr+WjgnQH1xAKWpOlA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219766; c=relaxed/simple; bh=c/Kt4iwzwH+bi+XcBBAmrHfE490XylwYWMNetRWl84M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dPc1ERyC9BWXyPBEIoKCULvwxWDhNGVzBHHNrTL996keTzzMjBqXGBm7erpnunLSP461It1fXStzcbdq6L+zG8W1YF7mn4QYrOozl9p0WRBZdpZzBX/4PinzJe/jxqJf18yA4hrFyndPesolSHe1k9HcHoYNwSViFOi4pSGkuiw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=dzy9cGwo; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="dzy9cGwo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219763; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pOevS25kl1+p5C9bnayjnL4IWCIfturozNfzs94vCz8=; b=dzy9cGwoDlnVfld0DZTrbmzpJXb+FM0sY3FC/cq2XjzLFzwEckd5B9ZzKPmcPINDjADbBL iLWZKpoKE9Uc0VX+HHVMXIGhTo8i6qbzf3E0tcfNxfpNoRehoxZhVj2e6K+9a2h82NqiL1 p7SfX32IH2m6jppX2gpUfr8jjd9LgaQ= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-588-q59OqY5BPIusNpvmo4OleA-1; Fri, 09 Aug 2024 12:09:22 -0400 X-MC-Unique: q59OqY5BPIusNpvmo4OleA-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1d3c02fcfso7767285a.0 for ; Fri, 09 Aug 2024 09:09:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219762; x=1723824562; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pOevS25kl1+p5C9bnayjnL4IWCIfturozNfzs94vCz8=; b=HSZBQlXP4x86UvumcYA03fk5FMmQLZn/ZZcpFp3ozyAPDwmvFT23YHA95osOGzUbin BZh/f8N7CdcIFlFHGcHKuk4P8LAAauVJk0KDrlFUy8Y2xsjfeG1zYrgyTI8PPK5mQ2Ek vTzZ96wHvmeC43hnZFGOEpF+r76l2um5cKAMIn5tgH40UyAde8GD8+XDcfmPBIRmW0NI FTz8rpL1hUoo0X833PTFlHiJzcDPL4cGz1SprYIs2gk4V624EKABUMNRgO0fH4yYX6ua utLr/9lmCZbKAVRs0VKQkoQMz/wra6SHPQ9YbvhOXOtQnDwsYg20xe5M2+yTZn3d5zsR FP7Q== X-Forwarded-Encrypted: i=1; AJvYcCUJisVCMWPCq6BijdHMUr7wWQw6h5LfzqvA3SiBEBtnAipzkBKIbyiAzOeGvQ5mzTIiqFw=@vger.kernel.org X-Gm-Message-State: AOJu0YxQSAywKknlHMXA5f6wvKNKwJqykAvMeCcY8P7URV9dpWWKPtlZ O1P1tFZIAtDQOFQselKDF4HzXD4eEx3jlerC6D5+2usYbvUdJ/kd7fFYrUHK1P5RVSxt9JupWvI a1V61q3XW8rgQtAYua6ipLPB0oXW4iR2nonDELRh797IBHI9ygA== X-Received: by 2002:a05:622a:1a8b:b0:44f:89e3:d119 with SMTP id d75a77b69052e-453126babcemr11885571cf.12.1723219761635; Fri, 09 Aug 2024 09:09:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGHUujWoHeQHvUCs2OAJSjyQdbxlWd9FX1yi8fb3RZKwv0O5abV6g6G64GeuNGe2uyjoJD48A== X-Received: by 2002:a05:622a:1a8b:b0:44f:89e3:d119 with SMTP id d75a77b69052e-453126babcemr11885331cf.12.1723219761262; Fri, 09 Aug 2024 09:09:21 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:20 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 03/19] mm: Mark special bits for huge pfn mappings when inject Date: Fri, 9 Aug 2024 12:08:53 -0400 Message-ID: <20240809160909.1023470-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We need these special bits to be around to enable gup-fast on pfnmaps. Mark properly for !devmap case, reflecting that there's no page struct backing the entry. Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- mm/huge_memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 39c401a62e87..e95b3a468aee 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1162,6 +1162,8 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, entry = pmd_mkhuge(pfn_t_pmd(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pmd_mkdevmap(entry); + else + entry = pmd_mkspecial(entry); if (write) { entry = pmd_mkyoung(pmd_mkdirty(entry)); entry = maybe_pmd_mkwrite(entry, vma); @@ -1258,6 +1260,8 @@ static void insert_pfn_pud(struct vm_area_struct *vma, unsigned long addr, entry = pud_mkhuge(pfn_t_pud(pfn, prot)); if (pfn_t_devmap(pfn)) entry = pud_mkdevmap(entry); + else + entry = pud_mkspecial(entry); if (write) { entry = pud_mkyoung(pud_mkdirty(entry)); entry = maybe_pud_mkwrite(entry, vma); From patchwork Fri Aug 9 16:08:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758946 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5A89199397 for ; Fri, 9 Aug 2024 16:09:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219768; cv=none; b=nPyNa+nTNes+9CfqVc3z0wEh90msENOLiL5N0eziizGu29yNkSGJA4uruOW+cI6Rkw0NDYSiqnuqzBa3JItcN5Ay6/ZkbXX/mNj1HRGb8fOOADwafjZrdiko1oi790J6wud6QvQppkgjqzde59WebnaNLGaEdPqOF5kH6zUdQ7I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219768; c=relaxed/simple; bh=3ePjgkaqs/lSLe5ciTxCy/YgQAh2UAoTQVxwiusU8PA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mntP30snWVHHO0I26eBcSpShYbbGlOn4gc2uOduVUYa5qlVQ98LfDR5ZD/sZ/zZKCtqnR9azkaEyUVzUnepylp0+ubXOuBwodnAjcP0gh6kndXFWLRav+pcoAHlQs+ne7ald+kUmfW2lRw8GGKLIxv8CFnrAzFzhlRO7vQ1DYWc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=T494QrgN; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="T494QrgN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ip3F/lMuK2on6ujO8FbOHDG+DYlKv6bK2Y5nLIm/Zzk=; b=T494QrgNX9xLP4oSjPuFYZLMnvD0G4s9Y60bVuFDDybJC0tc6vw4yCIpomehURCWDiGBgG p9MO2k+h4Clz290aq7HHtkiUNy7jWzgFxKaLxQa6oa8B0x+3iT30QIB98g4itGL+LDZc2w oHGJb+9zTbGwEWFo84SjC9L1Rz6Ju8E= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-613-kys3osrRNyeIDY5iDu-mqA-1; Fri, 09 Aug 2024 12:09:24 -0400 X-MC-Unique: kys3osrRNyeIDY5iDu-mqA-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-44fe325cd56so2983781cf.1 for ; Fri, 09 Aug 2024 09:09:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219764; x=1723824564; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ip3F/lMuK2on6ujO8FbOHDG+DYlKv6bK2Y5nLIm/Zzk=; b=NMrrUclKBvcOTGmQxIsqzmnkgMcx5pNl8Qf9IIZ5W3oG8aNovxBqkYC1x6ni19uTjP 7H3zF46/rO7a5L5MdkRbfv74y/iaIFiHELFQQn/2FLSRpOfsGJ/rfJHUj3kJoYue5BSS 36oOoHK4UlgIO/Qr6bD2DUHmuyd04ZkpmaxXVNitCQYFLzYY6L4W5o+SZ7yHFncQ3Co1 kFDpitKu9Xh26/vzJU/sK1DyxFSYvuPQzKrw5E2AVkND9rZjT26u1/SgsJo31T6ilahF 8HZNGLLKwh95oD9MXmWrNo6C/wLQr8KAWI1tE/n005/WvVVkyTWJTL/kmPeWn7OVqHPb gJ/w== X-Forwarded-Encrypted: i=1; AJvYcCXjzST6hgdHgLne/ietqgznqnphfAlM1foNZ/T3S1rvxa2/Pog5OpwCWwP8MZwNB5P2XGk=@vger.kernel.org X-Gm-Message-State: AOJu0YxM1Qh+mlgSNTVWsTFoGJVFrtA3Bl0GuN8w+9PULomQPXA2wE/P pPt3lJ4tl11ZAJ9Oqlt6RQ0n6xImB79tfzNlFXdwBxDYZv04vCmOlfuIIC62wKZzyww7LW8AZo3 extsW6VYCTec5BiCHsZUrmQ8qS28AgJwpHFuOETd85LRN518nyA== X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14423261cf.3.1723219764201; Fri, 09 Aug 2024 09:09:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHps50LqsfKl8qXevdYdePG8tewmjaKqGlRovYXRf84yciplQ1RIErbjQjT+BAlToj6EpXuWQ== X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14422831cf.3.1723219763777; Fri, 09 Aug 2024 09:09:23 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:22 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Matthew Wilcox , Ryan Roberts Subject: [PATCH 04/19] mm: Allow THP orders for PFNMAPs Date: Fri, 9 Aug 2024 12:08:54 -0400 Message-ID: <20240809160909.1023470-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This enables PFNMAPs to be mapped at either pmd/pud layers. Generalize the dax case into vma_is_special_huge() so as to cover both. Meanwhile, rename the macro to THP_ORDERS_ALL_SPECIAL. Cc: Matthew Wilcox Cc: Gavin Shan Cc: Ryan Roberts Cc: David Hildenbrand Cc: Zi Yan Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- include/linux/huge_mm.h | 6 +++--- mm/huge_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 2121060232ce..984cbc960d8b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -76,9 +76,9 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; /* * Mask of all large folio orders supported for file THP. Folios in a DAX * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to - * it. + * it. Same to PFNMAPs where there's neither page* nor pagecache. */ -#define THP_ORDERS_ALL_FILE_DAX \ +#define THP_ORDERS_ALL_SPECIAL \ (BIT(PMD_ORDER) | BIT(PUD_ORDER)) #define THP_ORDERS_ALL_FILE_DEFAULT \ ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) @@ -87,7 +87,7 @@ extern struct kobj_attribute thpsize_shmem_enabled_attr; * Mask of all large folio orders supported for THP. */ #define THP_ORDERS_ALL \ - (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DAX | THP_ORDERS_ALL_FILE_DEFAULT) + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_SPECIAL | THP_ORDERS_ALL_FILE_DEFAULT) #define TVA_SMAPS (1 << 0) /* Will be used for procfs */ #define TVA_IN_PF (1 << 1) /* Page fault handler */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e95b3a468aee..6568586b21ab 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -95,8 +95,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, /* Check the intersection of requested and supported orders. */ if (vma_is_anonymous(vma)) supported_orders = THP_ORDERS_ALL_ANON; - else if (vma_is_dax(vma)) - supported_orders = THP_ORDERS_ALL_FILE_DAX; + else if (vma_is_special_huge(vma)) + supported_orders = THP_ORDERS_ALL_SPECIAL; else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; From patchwork Fri Aug 9 16:08:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758947 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B646D199397 for ; Fri, 9 Aug 2024 16:09:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219771; cv=none; b=MDyfA06RBztWh5zTkEaRNzAUAWd6Ndip6cl6NBOC96Ok28ndI42xeu0RlGS2h9mDWfKAeTR8MYjD8IqEXyr1ndahP0vbDvEbJviyqb9pR8I2VrEl/NNp3obcU2NmWrNOTAqMIa/AZZpNucYAP4zx9vFI6qZ4AY47W9UUO9Q5SAE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219771; c=relaxed/simple; bh=C1EqeOtQk0O3VuTVuRQD9+bF0J9qW0lfyAutjUQ5RI4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KknRFRTnsIKKt/4HPKm+0vfhPxS368mORNZOs7YN3Xij7jY/ZZjLbJyowyJdT/3pY1Tpf0tENy1TogcSkegZDDoEYFix85hCD4nP9QD+boimMWVGHAUVYcFUPTr7aY2+O52vMEyVs6Xq7qzRirHm8eg0xfCRY0QnSoRRsBYiZX4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ezHs7ZOz; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ezHs7ZOz" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219768; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bfMsI243pXvbDIYk3LIHdKnZKRp0h0WHhTX8nwPWLTI=; b=ezHs7ZOzlrsJ1+uPUfu+/Kur2CwuLGgYAIIB5Vig/qzGZQqkB5tx9BLzBY7+OokfCdnKIc rBWMX5UZQ3Hs98PeSB1xqWqDJB8IwJUQKneQpgYwjr0JHqUBTZMUrotd4bBZaBqKpPabA0 G+LCFTSf3Mn7Ukl0sJCUMvcM/uMDJiU= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-644-fXLsxlGEN36qijpgpyqcXg-1; Fri, 09 Aug 2024 12:09:27 -0400 X-MC-Unique: fXLsxlGEN36qijpgpyqcXg-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-44fe28cd027so3757821cf.0 for ; Fri, 09 Aug 2024 09:09:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219767; x=1723824567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bfMsI243pXvbDIYk3LIHdKnZKRp0h0WHhTX8nwPWLTI=; b=klxhpXo0BfzR4yK79mI7NU46DSh9hCajeJPuaNi91zV7wasLNnMUHwA79h62BZRThx NhDPLGNsw2ZoSFdLOskxs92UE4HMiZilr8jgwfVkRIoTn6RL6P7L7hCgHWrPR8O7HqXc th0ikQzEa6KkMDjvF1cc3CHkNPTcuKTfr41wbvFWRGU9uFZs9j8FqyosPN++A5VzrbOe Xlwzd/0c24UmalSAH192GFyIyi6W1Wv+vdzo4oj4FQMPePnj5+HnxcaB0Gahvhv9WbJL 0/HFOtjxr23HOJChz9Q9bfHsr3VBrY30F+JOGnUWGhLy3F+enk8PbCCyEBC9WJi5GXD8 L+fw== X-Forwarded-Encrypted: i=1; AJvYcCX/g5faVvR9wm9FuEE+UFHMBPd3RsMY1ptVWr3FGyCzavnuGW28G+/iU4bAuMvACngE+3c=@vger.kernel.org X-Gm-Message-State: AOJu0YyuyrQ3DT9TPFihW2Y1Q1nJAVSTFmDwWMxdX+JuyBwXetX3KrXi phGYY3vTIWpiwCsg0SKetCFwTyGMifevmBF/PCHY/Yb981U7Hjq9JY8+3vWGAYtHXOJpYKr1o7i bf4azrdlfipElplAWys73SWe0GffY4cD5n0Hc6qpTy14ceDDIIw== X-Received: by 2002:ac8:59cc:0:b0:44f:c953:290d with SMTP id d75a77b69052e-4531251d17bmr12886141cf.2.1723219766852; Fri, 09 Aug 2024 09:09:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IElkbLJ8q6bD88BtBGZOYKdaksYNPrmA3JWwjtOEQ9+GQ5ZX+OpUCG/V8WuVaLDdrJFvk6CQQ== X-Received: by 2002:ac8:59cc:0:b0:44f:c953:290d with SMTP id d75a77b69052e-4531251d17bmr12885751cf.2.1723219766412; Fri, 09 Aug 2024 09:09:26 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:25 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 05/19] mm/gup: Detect huge pfnmap entries in gup-fast Date: Fri, 9 Aug 2024 12:08:55 -0400 Message-ID: <20240809160909.1023470-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Since gup-fast doesn't have the vma reference, teach it to detect such huge pfnmaps by checking the special bit for pmd/pud too, just like ptes. Signed-off-by: Peter Xu Acked-by: David Hildenbrand Reviewed-by: Jason Gunthorpe --- mm/gup.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d19884e097fd..a49f67a512ee 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3038,6 +3038,9 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr, if (!pmd_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pmd_special(orig)) + return 0; + if (pmd_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; @@ -3082,6 +3085,9 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, if (!pud_access_permitted(orig, flags & FOLL_WRITE)) return 0; + if (pud_special(orig)) + return 0; + if (pud_devmap(orig)) { if (unlikely(flags & FOLL_LONGTERM)) return 0; From patchwork Fri Aug 9 16:08:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758950 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9294719AD71 for ; Fri, 9 Aug 2024 16:09:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219778; cv=none; b=s4CjLoIIBXWWOqWX5Kg/8d9jxQjIC0SgN44t/Izg23fkIyDkrvbzLtzZpjOSA/ipM6NQxU31pfKTeW437MnMCq2OzArcX7utjJz6ICDeUhMBRaHhfWKI9rCiA9gQStmqi7ff5nG3JnonOv/yiV8rhLZnwOcvxdLdBx2MpZ/vUJs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219778; c=relaxed/simple; bh=HoGYsL9LSVsYg/VMQEnaKHTD8IFeuEK3k/nOrqKlNJA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TIodtesacBibJgXHef8LPYkBIKymBG+UkITQXIiFzKs0wd5pRvyVGJXxYb/NzvbP5md0G+3r0GfgfRowSqMTiJ6WZKR9LBbLmEp/mC0nhMbVw9ebF56zbaVwNuh59GATpquKAMAG7Tm1wh1bKyOXitShXGHWi4UX7Ml6CFjOP68= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=aG5sFqPh; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aG5sFqPh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nP4ocM4VA1bHw0KaG7AmUa9593dVgU8QzxEKB9c5vMc=; b=aG5sFqPh9l8eF/bWDsFjKfp5b+EfhPGstMj5jJ1J9DP/at5l6YFnTAzK/fYzRkgGbTtL3u OrIN47hQJAIkjR+xypvZJdZU5uKIALROc2eITmZNTPY5UWxtI9kHlR2EDyYmHSNrRBrezQ BU/akbUa3rWuqtGfCImRdnxm97pfNCU= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-81-DCRf7DI4PXuZ9-cIrvCOiQ-1; Fri, 09 Aug 2024 12:09:34 -0400 X-MC-Unique: DCRf7DI4PXuZ9-cIrvCOiQ-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44fee12c33bso1777521cf.1 for ; Fri, 09 Aug 2024 09:09:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219774; x=1723824574; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nP4ocM4VA1bHw0KaG7AmUa9593dVgU8QzxEKB9c5vMc=; b=mMGoqNRBS6fkNgzjzu2o/pVTINKey8h9wz53owVEgMHh5y97/97lr9ax+qdL2II4+Q 3ONTKcrYXoB1zaVwfmKMdBI8a0ZJAP7j7mnSM9YDkQwnDd8U5/enoN4Usp4J3wgi6VVM C6ONHteyiDR07kuiWk5cVWzYz9XCUjKOCUSNRQwDefjnjIsQeIaAo+p/sf81qMBgqWMf LvdEtn1xIrWjcLZelMNdTLBlMsGZMA4WA8VL+ezFz+4eBBam3mp1HeXNdTPcMeO7wBlL r18tO0Md5qx5ujS5ZAKZ1IbRorcckf7cP3aNOQOG+mmz5hT996SWGFdP7VJtgjFrtebM m2Rg== X-Forwarded-Encrypted: i=1; AJvYcCU+Zujrq7aV3IZwjnIcdFOeuSTweXIEDAr8afef0TWA+D/uqeWAT2oYIhfd9YpTNz0uZVw=@vger.kernel.org X-Gm-Message-State: AOJu0YwtiHn+jIdS6dJUYBsKrjOsdPHjNc9820xeUDiBPaHy66a6xYzY miLISfta1ZAFfLtsIvqt4K8vFl3VI1mMx3uUcwZZvVw28/YJz1Gkd9OiO55oHl1hK38FyWWskdq 9ypfuMyO7tii+u+XlBm8OSz8GfZgflPNIvVN3k7QS7oT2MyJhiQ== X-Received: by 2002:a05:622a:386:b0:450:21b2:22e5 with SMTP id d75a77b69052e-453125745b1mr14927911cf.4.1723219774010; Fri, 09 Aug 2024 09:09:34 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGLMln6bbvPWjvF9TKd48hi2B1jMtNjQrsP4cRdycFMZCE1oQ0WYqhx5h2X//7ZLsXT7ckFXA== X-Received: by 2002:a05:622a:386:b0:450:21b2:22e5 with SMTP id d75a77b69052e-453125745b1mr14924951cf.4.1723219768577; Fri, 09 Aug 2024 09:09:28 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:27 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 06/19] mm/pagewalk: Check pfnmap early for folio_walk_start() Date: Fri, 9 Aug 2024 12:08:56 -0400 Message-ID: <20240809160909.1023470-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Pfnmaps can always be identified with special bits in the ptes/pmds/puds. However that's unnecessary if the vma is stable, and when it's mapped under VM_PFNMAP | VM_IO. Instead of adding similar checks in all the levels for huge pfnmaps, let folio_walk_start() fail even earlier for these mappings. It's also something gup-slow already does, so make them match. Cc: David Hildenbrand Signed-off-by: Peter Xu --- mm/pagewalk.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cd79fb3b89e5..fd3965efe773 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -727,6 +727,11 @@ struct folio *folio_walk_start(struct folio_walk *fw, p4d_t *p4dp; mmap_assert_locked(vma->vm_mm); + + /* It has no folio backing the mappings at all.. */ + if (vma->vm_flags & (VM_IO | VM_PFNMAP)) + return NULL; + vma_pgtable_walk_begin(vma); if (WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end)) From patchwork Fri Aug 9 16:08:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758948 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E50441990C3 for ; Fri, 9 Aug 2024 16:09:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219775; cv=none; b=tUZH0obe+/SXTm9pGK9nRhfvrJg6kWHbWavnQYOoeGlHRvNQZLEdJUx8lk2niaccv/YJWSXMXCv6LkETOo5RlTnkqL3B0aNBnFDh7/iz1j1F70HQGrM6ldpQhr4su8Kcystg5i0pq93gnLGedRaOj+hqA2n0P7FDJd8HtcBmLwY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219775; c=relaxed/simple; bh=NRE6RSln8yk8W4KIRztpuy77GMwMRMzE4sT6CQJcyFY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=r3/pndqueJyD0WpV3hw8XaV7/q2ST1HS7+s10aN/MKV6z7pYlisz1/Vk9uPoos/rPHmWA5bYloa5KHyNKafvOWZRQ0oYwzpH7SkAX2aOpyvbnzUxaApPUIvfCWEXwFJZk4/g87cewflMH+34KRg+XwPtA/g5NXIArV/FQbgL6Tk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=c45B0Sr3; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="c45B0Sr3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GAUMBfy0kdSy8bea6Q2/adQQTYYkrWuiUWfctkd/rW8=; b=c45B0Sr3yazNTZNhbIwtCuU7Ramq6snK4CcOpNVf1lQ5CTjMIK5jQWCW6ZJV3AxHh+dpT8 210JM1/828YOAGynPsiCJ22reBHXB9VKdQyMG4KuqkTk278gUwQhm4g8LtbBkivUY9lNDV M9RL4suMguN+XyCweQsVBwbTDd5J//Y= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-94-8_PQV3YGOneoMhEncp45Pg-1; Fri, 09 Aug 2024 12:09:31 -0400 X-MC-Unique: 8_PQV3YGOneoMhEncp45Pg-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-4519ec19cb6so810431cf.1 for ; Fri, 09 Aug 2024 09:09:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219771; x=1723824571; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GAUMBfy0kdSy8bea6Q2/adQQTYYkrWuiUWfctkd/rW8=; b=rq/ijOqFj2E5ZO95GTbednU/WVEeMLqe5sniv4sooULu9SBRk3dc/cfkg5SWB02mYb wrbncWwc82bb9Yq7P0z9xIFMDrt6kd+/vHHzYej3ER4Sswjy+q+FgMjvDLFyGKb2gz4w l1Bp0a10bVfgDCfoPyYpuZYg01qREO40Q1NBT+FBzR8kIuz3IQItWl+3rbVfKnA/ivKt SWSv+W6rQF+xa/x8Nmdhz5XKsbn+fnF8G5KHmc0X1o1SJ01yeOOiYmUlM63vh9iYdMMh ZrK3AJJRUozwUIQiT9dpMIzDTBZXWcwcKUU6n75Z+CznVjx43UcxfN/KBGdF5NVa2vQ3 TrgA== X-Forwarded-Encrypted: i=1; AJvYcCWfupfv+uCahPhYQX243fPWECdz+qDKDVcYhGlfzgPl1vIw8Ortpp653P8AzABXyg4rDK2J8oUGUltyQN3yGRml43v4 X-Gm-Message-State: AOJu0YxjB4An2r+Oufytyy/B8GoCo4J48jI8aRNr0AUQJjoQ4jVXx1QK 4uD2YjDghv6f8CbcDGSAb2HUfHhYIrTwIuhTHM4ZTzG82hwcJFHFLJP5TaO1WYyRLTTs/d9pyXz fJrUfjekBRUjRy7nCIevQvH93wO1jAABg/yUdzdFny1T7NxaaBA== X-Received: by 2002:a05:622a:180e:b0:44e:d016:ef7 with SMTP id d75a77b69052e-45312652444mr12909311cf.7.1723219771015; Fri, 09 Aug 2024 09:09:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2AJ498flXqsf/RjsZizkWR5q4LBdwGhozxnqN6XMiL4rwNH8ESKXDSym68P6o8cpdHObfXw== X-Received: by 2002:a05:622a:180e:b0:44e:d016:ef7 with SMTP id d75a77b69052e-45312652444mr12908941cf.7.1723219770588; Fri, 09 Aug 2024 09:09:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:30 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 07/19] mm/fork: Accept huge pfnmap entries Date: Fri, 9 Aug 2024 12:08:57 -0400 Message-ID: <20240809160909.1023470-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Teach the fork code to properly copy pfnmaps for pmd/pud levels. Pud is much easier, the write bit needs to be persisted though for writable and shared pud mappings like PFNMAP ones, otherwise a follow up write in either parent or child process will trigger a write fault. Do the same for pmd level. Signed-off-by: Peter Xu --- mm/huge_memory.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6568586b21ab..015c9468eed5 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1375,6 +1375,22 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pgtable_t pgtable = NULL; int ret = -ENOMEM; + pmd = pmdp_get_lockless(src_pmd); + if (unlikely(pmd_special(pmd))) { + dst_ptl = pmd_lock(dst_mm, dst_pmd); + src_ptl = pmd_lockptr(src_mm, src_pmd); + spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + /* + * No need to recheck the pmd, it can't change with write + * mmap lock held here. + */ + if (is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)) { + pmdp_set_wrprotect(src_mm, addr, src_pmd); + pmd = pmd_wrprotect(pmd); + } + goto set_pmd; + } + /* Skip if can be re-fill on fault */ if (!vma_is_anonymous(dst_vma)) return 0; @@ -1456,7 +1472,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); if (!userfaultfd_wp(dst_vma)) pmd = pmd_clear_uffd_wp(pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + pmd = pmd_wrprotect(pmd); +set_pmd: + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1502,8 +1520,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; From patchwork Fri Aug 9 16:08:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758949 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07DDA19AA7A for ; Fri, 9 Aug 2024 16:09:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219777; cv=none; b=Nf7M8ZRfwsduYq7CnyUD40UPsGvuMWu6ndzPg2zZaIF1KyFiI45dKia0ggJzSoa1aV78BbeDc4gAA95pMQY9B3mZ1AWJt0wvVE1Tqr67lsxR4Vo2J0R1UX6X43C5cr6Kgu8bjA2UTG0+CvO6vuazXTcWKZS9n5JSbLC+ykoSmaU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219777; c=relaxed/simple; bh=KGvOFcjjnMZbGILhL/il8UWmtwN07FBEuaEKlUcuKek=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nKJSIG6haHhLtah1wV/8qrh2FmhlcrEZ4UcYRDdp9xiWMuf5Kj2ZNYrFPeFxDXHEJkJ4t/gUTMP5yyd8QSgmwr+CTtOI2gG2MGjMkiN4v+6d1UTjpIHrJLpYODDRkTpem6qIk/7VgT2Fi2QkpCVWhW8oHhGnSELUN5xLqFU0v0Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=EzKgwvCp; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EzKgwvCp" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3aM3oVPeXmFPFf4Q68bjufzjsUq2ESmscQPnLEkaBB8=; b=EzKgwvCpMXxU9YgyJ7NmKvLgz6ByhsBcszjh8Z3ilu4mFQOtFV5B+Ps2fZ9GG1C8v8yBun Uadfrw257Flm8DUa4bRV04py/anO36iu9EIkEuHHPyfLrkyf0J2ObsAkbV/lbxoBuxWxtI NcLdEdg4uSduAMfp4IW2Ra5SsmfFr10= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-218-sLBCqYkvPVayT_rwR-P1WQ-1; Fri, 09 Aug 2024 12:09:33 -0400 X-MC-Unique: sLBCqYkvPVayT_rwR-P1WQ-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-45009e27b83so4905401cf.1 for ; Fri, 09 Aug 2024 09:09:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219773; x=1723824573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3aM3oVPeXmFPFf4Q68bjufzjsUq2ESmscQPnLEkaBB8=; b=hPq6V2Hx+mzsf4kbDZ68msyzooAzYAV8o4NYdxLpkTV81e8nAToxk0BnrGWyL0xL0F y1MrF9WEMQJBMLEze8WGrnm4PPVQew/gViC3DP3/x1patbnMAIgR20lkhLHteaqqg65T tDGkkSvs5iSQYj/BS7/I9/K2IYdIuQSm3j/ewEM4wrULLAq/6HQeWrz2jMH0LZCnXbjb Qv2gO4a28MvGjVNwpVD6kLSQJNknY8GesUyJxV/OnmCGS8O1VM+gtnhQZgsc0fqlvP0M mv/UisIMg3K0bhnGYyZPXGV+eSE27UkZMoEVSyKOMIeWgbo10aG8RR3S6GdNHqIa1a17 3mlQ== X-Forwarded-Encrypted: i=1; AJvYcCVuVv8V06y3uww067ea27Ws7eaYz0NE/i3Oysam0OSDWukbPjJBGkSmEPuNEkVEfBfhTgL/HP6zuvkYT0FiJlBWWkHD X-Gm-Message-State: AOJu0Yz6ClATZR94YdP7xawnpJSth9AaFdvgTm025dfL2BUccE0Riage t02wXBPcPWW6hWNxOscAnxICDj1ZRBsrFwZUtMb48V4TKlG0GZPumS7ZBaH/b619OvJJ07S9BIk AkIui+cMzYOnVziNhWra3+iCO0RO3jAcGHqFxF217GbOphGS05Q== X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14739801cf.6.1723219773082; Fri, 09 Aug 2024 09:09:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE6GyuXgyukqToIMYwmYCyyJms6T0RBSwsqvRpSKO6DWSQsB926zH9/mqcqwcpfSE6Nqd5Tsw== X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14739431cf.6.1723219772696; Fri, 09 Aug 2024 09:09:32 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:32 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 08/19] mm: Always define pxx_pgprot() Date: Fri, 9 Aug 2024 12:08:58 -0400 Message-ID: <20240809160909.1023470-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 There're: - 8 archs (arc, arm64, include, mips, powerpc, s390, sh, x86) that support pte_pgprot(). - 2 archs (x86, sparc) that support pmd_pgprot(). - 1 arch (x86) that support pud_pgprot(). Always define them to be used in generic code, and then we don't need to fiddle with "#ifdef"s when doing so. Signed-off-by: Peter Xu Reviewed-by: Jason Gunthorpe --- arch/arm64/include/asm/pgtable.h | 1 + arch/powerpc/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + include/linux/pgtable.h | 12 ++++++++++++ 5 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7a4f5604be3f..b78cc4a6758b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -384,6 +384,7 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pfn = pte_pfn(pte); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 264a6c09517a..2f72ad885332 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -65,6 +65,7 @@ static inline unsigned long pte_pfn(pte_t pte) /* * Select all bits except the pfn */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags; diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 3fa280d0672a..0ffbaf741955 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -955,6 +955,7 @@ static inline int pte_unused(pte_t pte) * young/old accounting is not supported, i.e _PAGE_PROTECT and _PAGE_INVALID * must not be set. */ +#define pte_pgprot pte_pgprot static inline pgprot_t pte_pgprot(pte_t pte) { unsigned long pte_flags = pte_val(pte) & _PAGE_CHG_MASK; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 3fe429d73a65..2b7f358762c1 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -783,6 +783,7 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) return __pmd(pte_val(pte)); } +#define pmd_pgprot pmd_pgprot static inline pgprot_t pmd_pgprot(pmd_t entry) { unsigned long val = pmd_val(entry); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 780f3b439d98..e8b2ac6bd2ae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1956,6 +1956,18 @@ typedef unsigned int pgtbl_mod_mask; #define MAX_PTRS_PER_P4D PTRS_PER_P4D #endif +#ifndef pte_pgprot +#define pte_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pmd_pgprot +#define pmd_pgprot(x) ((pgprot_t) {0}) +#endif + +#ifndef pud_pgprot +#define pud_pgprot(x) ((pgprot_t) {0}) +#endif + /* description of effects of mapping type and prot in current implementation. * this is due to the limited x86 page protection hardware. The expected * behavior is in parens: From patchwork Fri Aug 9 16:08:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758951 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69C0319ADBA for ; Fri, 9 Aug 2024 16:09:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219780; cv=none; b=DxWiIbThAU/0ziehFp846TzK9+AxjeKUeAx6fpnArKsx0/np0erSvDCOAIf1yCD6zfqeJI77cOJYCN/MEkv8TVxw43LfL8sFpOoz/3P5hHld6SU2bfxIjPfmo43b9NqEzBi5H/0nBqtA3hkT5BvShFxP8yfOrTYYJSkxyHQEVHE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219780; c=relaxed/simple; bh=yfBU6y6kSdOFyZjcdoL0YKRvT9xfK+pyg+o4OfDl314=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nr/3k+l+IdbGz6kyP3/n6sm5Ucm/vCVKrr19gMYlTFBmbnYRANNA77rkELH1rcJk8mBHtONsAAG0jnrob2JfNx7oKEoFrVqce7Szn9xEZyLgSY0MgkKiOx4dXTJMq+P2tT03NnLg0reKiPAuK+A0eoUYYTwkAlW5EBsNHRkuUT0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=UCFXZ4vh; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="UCFXZ4vh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219777; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IXGL3lb2RETUVA/g/GpjABQk9U9hrAJ/lFeyUGjNHTs=; b=UCFXZ4vhMbtvOf7JuxO23tzqGl/ML7sXTCRlID+n7fKZ6GG9HgMHYxVk2m5A4N1OwebuHS 2UfvU5Au06T5dG8873orcFME9Fh0jFbVQSScz2yRpITDkPeQyaXi9U8j00BtI5xWHoJngH 0TVAzxwFqL9ZOEM4o2Pwz5oGw78drOU= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-497-zcwEZ0K2PTycbgwZyLk3zA-1; Fri, 09 Aug 2024 12:09:36 -0400 X-MC-Unique: zcwEZ0K2PTycbgwZyLk3zA-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-44ff25bbfe1so3070091cf.3 for ; Fri, 09 Aug 2024 09:09:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219776; x=1723824576; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IXGL3lb2RETUVA/g/GpjABQk9U9hrAJ/lFeyUGjNHTs=; b=YCieJEUZ4UV/6AGQKa/LyYoJJzZUfjqpBJVlgT00HhFm2td+nPSVblv7/frgl3M2TZ U845omhESQNpWTNyqQdTn1IiD0Ya5W/VveDdtuYDECsE9lag54L6+t/lN8CipA70pQSP ok2NwCDsz1JP4vzZW7RmeUo3/cwIkQ/VrUtRNT69pWr1TjJb+IWqEuYCusNZcpvG6esZ tnZ846fKJod0gqdTEX7IgtIzQ1XqrvyDgU4gwrACrbpjBl5+gFmOjoSGXcSjJMo1Mj3u rhQtaD2ljDELMvby1ESIYOEXUJQSnsjpOHWE0z9+xNHUEKsYzO1v39RXMQWlUIwr4FPo XEfA== X-Forwarded-Encrypted: i=1; AJvYcCWqArgp/vdPVVmXuBBcu5LWlSam3OrlUgvFbBH3nEDkSus+sVQ7Y+pCl0OEwPnSSY4r6Ts=@vger.kernel.org X-Gm-Message-State: AOJu0YzopVctfc2Ale4HUYXwjO1LI5k1rzbZtVIdILEMW+S5KbEfFqC6 jX5M0tUyDJbRHaaPq4wd+K73KKXjeBdEckvBb9656s1vHYTRoclOHB3NWUPqUSEFKO3tQljRX3S KJ1WOqZvjEcpU3D5wnrpCmP5Xat3MmH3ZFKgZVmNUvTBI8pQUSQ== X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14428781cf.3.1723219775689; Fri, 09 Aug 2024 09:09:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFUrJX3vSa/2Mr5PpFf2V5KDcJEixKltLA8oaEJqLbElyFRMCegYAzgvcdg3qK/Zl0Af+cLBA== X-Received: by 2002:a05:622a:1b8e:b0:447:e636:9ea9 with SMTP id d75a77b69052e-4531251c724mr14428311cf.3.1723219775158; Fri, 09 Aug 2024 09:09:35 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:34 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 09/19] mm: New follow_pfnmap API Date: Fri, 9 Aug 2024 12:08:59 -0400 Message-ID: <20240809160909.1023470-10-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Introduce a pair of APIs to follow pfn mappings to get entry information. It's very similar to what follow_pte() does before, but different in that it recognizes huge pfn mappings. Signed-off-by: Peter Xu --- include/linux/mm.h | 31 ++++++++++ mm/memory.c | 147 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 90ca84200800..7471302658af 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2374,6 +2374,37 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); +struct follow_pfnmap_args { + /** + * Inputs: + * @vma: Pointer to @vm_area_struct struct + * @address: the virtual address to walk + */ + struct vm_area_struct *vma; + unsigned long address; + /** + * Internals: + * + * The caller shouldn't touch any of these. + */ + spinlock_t *lock; + pte_t *ptep; + /** + * Outputs: + * + * @pfn: the PFN of the address + * @pgprot: the pgprot_t of the mapping + * @writable: whether the mapping is writable + * @special: whether the mapping is a special mapping (real PFN maps) + */ + unsigned long pfn; + pgprot_t pgprot; + bool writable; + bool special; +}; +int follow_pfnmap_start(struct follow_pfnmap_args *args); +void follow_pfnmap_end(struct follow_pfnmap_args *args); + extern void truncate_pagecache(struct inode *inode, loff_t new); extern void truncate_setsize(struct inode *inode, loff_t newsize); void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to); diff --git a/mm/memory.c b/mm/memory.c index 67496dc5064f..2194e0f9f541 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6338,6 +6338,153 @@ int follow_pte(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(follow_pte); +static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, + spinlock_t *lock, pte_t *ptep, + pgprot_t pgprot, unsigned long pfn_base, + unsigned long addr_mask, bool writable, + bool special) +{ + args->lock = lock; + args->ptep = ptep; + args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pgprot = pgprot; + args->writable = writable; + args->special = special; +} + +static inline void pfnmap_lockdep_assert(struct vm_area_struct *vma) +{ +#ifdef CONFIG_LOCKDEP + struct address_space *mapping = vma->vm_file->f_mapping; + + if (mapping) + lockdep_assert(lockdep_is_held(&vma->vm_file->f_mapping->i_mmap_rwsem) || + lockdep_is_held(&vma->vm_mm->mmap_lock)); + else + lockdep_assert(lockdep_is_held(&vma->vm_mm->mmap_lock)); +#endif +} + +/** + * follow_pfnmap_start() - Look up a pfn mapping at a user virtual address + * @args: Pointer to struct @follow_pfnmap_args + * + * The caller needs to setup args->vma and args->address to point to the + * virtual address as the target of such lookup. On a successful return, + * the results will be put into other output fields. + * + * After the caller finished using the fields, the caller must invoke + * another follow_pfnmap_end() to proper releases the locks and resources + * of such look up request. + * + * During the start() and end() calls, the results in @args will be valid + * as proper locks will be held. After the end() is called, all the fields + * in @follow_pfnmap_args will be invalid to be further accessed. + * + * If the PTE maps a refcounted page, callers are responsible to protect + * against invalidation with MMU notifiers; otherwise access to the PFN at + * a later point in time can trigger use-after-free. + * + * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore + * should be taken for read, and the mmap semaphore cannot be released + * before the end() is invoked. + * + * This function must not be used to modify PTE content. + * + * Return: zero on success, -ve otherwise. + */ +int follow_pfnmap_start(struct follow_pfnmap_args *args) +{ + struct vm_area_struct *vma = args->vma; + unsigned long address = args->address; + struct mm_struct *mm = vma->vm_mm; + spinlock_t *lock; + pgd_t *pgdp; + p4d_t *p4dp, p4d; + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pfnmap_lockdep_assert(vma); + + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + goto out; + + if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) + goto out; +retry: + pgdp = pgd_offset(mm, address); + if (pgd_none(*pgdp) || unlikely(pgd_bad(*pgdp))) + goto out; + + p4dp = p4d_offset(pgdp, address); + p4d = READ_ONCE(*p4dp); + if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) + goto out; + + pudp = pud_offset(p4dp, address); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + goto out; + if (pud_leaf(pud)) { + lock = pud_lock(mm, pudp); + if (!unlikely(pud_leaf(pud))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pud_pgprot(pud), + pud_pfn(pud), PUD_MASK, pud_write(pud), + pud_special(pud)); + return 0; + } + + pmdp = pmd_offset(pudp, address); + pmd = pmdp_get_lockless(pmdp); + if (pmd_leaf(pmd)) { + lock = pmd_lock(mm, pmdp); + if (!unlikely(pmd_leaf(pmd))) { + spin_unlock(lock); + goto retry; + } + pfnmap_args_setup(args, lock, NULL, pmd_pgprot(pmd), + pmd_pfn(pmd), PMD_MASK, pmd_write(pmd), + pmd_special(pmd)); + return 0; + } + + ptep = pte_offset_map_lock(mm, pmdp, address, &lock); + if (!ptep) + goto out; + pte = ptep_get(ptep); + if (!pte_present(pte)) + goto unlock; + pfnmap_args_setup(args, lock, ptep, pte_pgprot(pte), + pte_pfn(pte), PAGE_MASK, pte_write(pte), + pte_special(pte)); + return 0; +unlock: + pte_unmap_unlock(ptep, lock); +out: + return -EINVAL; +} +EXPORT_SYMBOL_GPL(follow_pfnmap_start); + +/** + * follow_pfnmap_end(): End a follow_pfnmap_start() process + * @args: Pointer to struct @follow_pfnmap_args + * + * Must be used in pair of follow_pfnmap_start(). See the start() function + * above for more information. + */ +void follow_pfnmap_end(struct follow_pfnmap_args *args) +{ + if (args->lock) + spin_unlock(args->lock); + if (args->ptep) + pte_unmap(args->ptep); +} +EXPORT_SYMBOL_GPL(follow_pfnmap_end); + #ifdef CONFIG_HAVE_IOREMAP_PROT /** * generic_access_phys - generic implementation for iomem mmap access From patchwork Fri Aug 9 16:09:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758952 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0536219B3DD for ; Fri, 9 Aug 2024 16:09:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219782; cv=none; b=Ea9sAyiVhSbqtu+jt9kcL7RgSX9F8eXTNrVLXZO8FELIMYHfp24jNJZlPePD/mbxlZYDVPHQUFiA9yQvAgGsMeg+c0Pj4JVt7jYuGSdwcGm1aQgBtOL9y6mLnx9HuNWogNKbf8yIvXSwD3pNYLjYwp6fEh24z+unqul+0ZmXoT8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219782; c=relaxed/simple; bh=KfYE5tIQA4wIRG7So5bdir7+S/1uf3KsBeoybnD12MQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=okGw8q/Cj5OPGcCx7WTb2+QfGm8hVpeuA7QPV3b8HhOXcYSUEDbfD6SyUycodaA0qfdojfn5dsOi0l7GpMC1iiBQNO3pDSK8/PjhfJstUWgR2TXar4bG33lg0sZZ/Z9MMcWVNhVcdB1tql5p4d9fZUsgkzs9hQuI2WIa1sMubHk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=amiC2tTi; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="amiC2tTi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XgMG3z25w/FU4W74BEAmfGyy8QhkBSj3TNUPa/vYriM=; b=amiC2tTiiJWK+7NM6JHHnTQqJXVJstQiiLHVgPfnlhcWoYFspVa+3MiamqftqPpMZh/O70 KfU/DwV8l959mx3LcKyoBb7bo7wlmVXJ/GypS87u1Ug/iOqgAEoCY1egmRGUflyuPxdDdE vA+XjF88W4k8EB1uKJneWfMi1AjY1qc= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-600-vzpY8QQnPy6H62q7XPSglQ-1; Fri, 09 Aug 2024 12:09:38 -0400 X-MC-Unique: vzpY8QQnPy6H62q7XPSglQ-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-6b7678caf7dso6679376d6.1 for ; Fri, 09 Aug 2024 09:09:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219778; x=1723824578; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XgMG3z25w/FU4W74BEAmfGyy8QhkBSj3TNUPa/vYriM=; b=CzJu3KJY671FbG9EDQXhfnaYEjkKVpT0trMelTw/Iwzxfk0loDvPs7zdafHXY/FynC drQWgqtKjuhq2LwOb7d9vpIJCuDdhfKCdg9FUE061JA4SLzgC1QL7IseuoXTQEdaynB/ 205kEOCTJ937fYI2d2UDtPu8hqWUo24fQ4xHtg+VH7NDGCJMMWGujqurCVrQMhI684d8 Cgy+0Z2IFku0qybFgWj+1v/Wm85Xf0UUBdm79Wp4aqDGD1N/868VX+JrAk9C7Y9we0Nr 74aTmuwG6DjZKJvsYvdYT/Cci8Dt/hA2MgNaQ3fZwZVamGe89hM6pVYnXkyBKJIDPfoQ 83Eg== X-Forwarded-Encrypted: i=1; AJvYcCW+ht088hTzdu3ef0SGxLORTGl5UCm1PbKFF6uW4r+aZI+vfvdi28HTlhPoVVX/+7svWdqmY51tzDpRRXc2q/LgI4bj X-Gm-Message-State: AOJu0YzBIhKbDfYrA9HgQP2Uy8Ljca1hREs2xWdZ2loj5lF89Mh8iaS2 Jc07J4eUy9waE5KbxBCl3a3DoS2gdQQp3rOdN7RJNN9wgJ2vvS4Um6WFCrR1YYWqlV7w6qCH4sD tkEaJt6eLh7WadeiSk6WVKMdbS2sCcCO8fbb+RzkqfWnhFq5WiA== X-Received: by 2002:ac8:5dd3:0:b0:447:dfa0:283e with SMTP id d75a77b69052e-4531251fe34mr11225551cf.1.1723219778002; Fri, 09 Aug 2024 09:09:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE6MoN+Dd8Hdt5Y2cXuQIqSY6AztxI8VpmGorOFfK4YC2KlFlbKhz5m1DVHT1NSvONycoblfw== X-Received: by 2002:ac8:5dd3:0:b0:447:dfa0:283e with SMTP id d75a77b69052e-4531251fe34mr11225161cf.1.1723219777518; Fri, 09 Aug 2024 09:09:37 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:36 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 10/19] KVM: Use follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:00 -0400 Message-ID: <20240809160909.1023470-11-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new pfnmap API to allow huge MMIO mappings for VMs. The rest work is done perfectly on the other side (host_pfn_mapping_level()). Cc: Paolo Bonzini Cc: Sean Christopherson Signed-off-by: Peter Xu --- virt/kvm/kvm_main.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0788d0a72cc..9fb1c527a8e1 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2862,13 +2862,11 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, unsigned long addr, bool write_fault, bool *writable, kvm_pfn_t *p_pfn) { + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; kvm_pfn_t pfn; - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; int r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) { /* * get_user_pages fails for VM_IO and VM_PFNMAP vmas and does @@ -2883,21 +2881,19 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, if (r) return r; - r = follow_pte(vma, addr, &ptep, &ptl); + r = follow_pfnmap_start(&args); if (r) return r; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) { + if (write_fault && !args.writable) { pfn = KVM_PFN_ERR_RO_FAULT; goto out; } if (writable) - *writable = pte_write(pte); - pfn = pte_pfn(pte); + *writable = args.writable; + pfn = args.pfn; /* * Get a reference here because callers of *hva_to_pfn* and @@ -2918,9 +2914,8 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma, */ if (!kvm_try_get_pfn(pfn)) r = -EFAULT; - out: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); *p_pfn = pfn; return r; From patchwork Fri Aug 9 16:09:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758953 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7CD819D071 for ; Fri, 9 Aug 2024 16:09:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219786; cv=none; b=EetaEi5ayarMEFgGgbqrJRPKQyJCkiHxA2ZAgW3h4GQHD1oQkfEOTi2cgbIEEZAGww788D3aBl4WGI+fz3KUc76Qq5wtd7XGKSHk65XYq0izXThNleEWCYhYFjECzzd4U9fAH6qIYpCzusvuYQWdHy35S7XmoKKqnNzTVU9bosM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219786; c=relaxed/simple; bh=F3zrw7GW4L3DDhmifn0KqTitiZnPFDtSbk8TCndoYpY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MBbOTnIVqlGRNAdZml5aAhgw9HcURyIXPeibvwTOR6wGRnxI5Xshq/tJxNm89R6Ibm1USIxuxpOTjWkq/6wWktmOIj8tQBT7Ub7HmsKga4KGBTmFS1b3C+S4i1LGgBeAfETqhrFb7hz6JrVi0YAc/egjanA4uZ1qPkWNp4aDdyg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=DzLn74dv; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="DzLn74dv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219783; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=DzLn74dvuR9SXv0dLM+3DF359zb3zRK69RemrXXjuSh0+28EwxbG9tvOE4GEnCgDRVdz8P SbLOjlpJkDt03IOafLHUxYl4wDgIhSI7eN875jUGHJJiGl3+Q/KMImDOXSgf99fJ7KJ8Do tsu9XwqlhrWWB3r/fbadlNaPut/YmsY= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-352-ml8DzDuIMm667dfu79W5xA-1; Fri, 09 Aug 2024 12:09:41 -0400 X-MC-Unique: ml8DzDuIMm667dfu79W5xA-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-44fea369811so4530231cf.0 for ; Fri, 09 Aug 2024 09:09:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219780; x=1723824580; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kaY3EfeA+k09YWgS+MfGM8qkjlv58srLyvVau/9jXnU=; b=Butxo7gH07NpbVfD0J6Ww3y4PZUpXZQxPv5SYWC73PzbVbuJCRrcTCJfsjfA9UMOqN IgLoAU27xbCORv6Gjsn8BVNsIPv+bir29Uz9Mw7MT7PTWbaY98yPTxrJrFSJkgBdv4g9 4zZFvX5qH25W9v8AW6u4d1b/6RKeMdDO3wNqXy7tK7MCzNwuzyMYffOegMFerkTtGSnp Fd6Nrppz2WB2oHOpk1bifvlnzOu8UClhohhkqyhk+2OhO7/5eOi0jOJBokfH5NSXDtEu 9+GSmBW5+lGiCtlJMAsBzUQdEMuL/6LU51sURHID7+Sk9DAhSyrOy1Ft314T4BmLLoa0 rQjQ== X-Forwarded-Encrypted: i=1; AJvYcCVJ+kQcBwYTLQwCd4upu0alZIxtvTA93tahDzLTKBWPratCvhKS1U0978dgzpO5ti9/Cwot3a8iurOl4TDUZlowRcRM X-Gm-Message-State: AOJu0YzJEQI1yaSP24w+4o6pUmp+3QOoTeveRPE0LFi9L8DBHuQHQ9il JMPHKhdUzYLcfmQacJ17zDvvzqEJT6VTU9x/xEd8NxQacjf08k7Wr+8+ChuK0P/NebF3RvAlUjr RC0PS+lppjb4IXzAOpbIpN07xZiC3um3/d+0fVM2U3LPNY4eQPw== X-Received: by 2002:ac8:5912:0:b0:450:efd:723 with SMTP id d75a77b69052e-453125552fbmr13163141cf.4.1723219780554; Fri, 09 Aug 2024 09:09:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFtbIkutydCKQHHfrahTRflQkyMYGSh3zYrObmthmw14T659Ni251Xo72HIqIFQZU+hooLp2A== X-Received: by 2002:ac8:5912:0:b0:450:efd:723 with SMTP id d75a77b69052e-453125552fbmr13162951cf.4.1723219780083; Fri, 09 Aug 2024 09:09:40 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:39 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Niklas Schnelle , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , linux-s390@vger.kernel.org Subject: [PATCH 11/19] s390/pci_mmio: Use follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:01 -0400 Message-ID: <20240809160909.1023470-12-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Cc: Niklas Schnelle Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: linux-s390@vger.kernel.org Signed-off-by: Peter Xu --- arch/s390/pci/pci_mmio.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/s390/pci/pci_mmio.c b/arch/s390/pci/pci_mmio.c index 5398729bfe1b..de5c0b389a3e 100644 --- a/arch/s390/pci/pci_mmio.c +++ b/arch/s390/pci/pci_mmio.c @@ -118,12 +118,11 @@ static inline int __memcpy_toio_inuser(void __iomem *dst, SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, const void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -169,11 +168,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.address = mmio_addr; + args.vma = vma; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) @@ -181,7 +182,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_write, unsigned long, mmio_addr, ret = zpci_memcpy_toio(io_addr, buf, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); out_free: @@ -260,12 +261,11 @@ static inline int __memcpy_fromio_inuser(void __user *dst, SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, void __user *, user_buffer, size_t, length) { + struct follow_pfnmap_args args = { }; u8 local_buf[64]; void __iomem *io_addr; void *buf; struct vm_area_struct *vma; - pte_t *ptep; - spinlock_t *ptl; long ret; if (!zpci_is_enabled()) @@ -308,11 +308,13 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, if (!(vma->vm_flags & VM_WRITE)) goto out_unlock_mmap; - ret = follow_pte(vma, mmio_addr, &ptep, &ptl); + args.vma = vma; + args.address = mmio_addr; + ret = follow_pfnmap_start(&args); if (ret) goto out_unlock_mmap; - io_addr = (void __iomem *)((pte_pfn(*ptep) << PAGE_SHIFT) | + io_addr = (void __iomem *)((args.pfn << PAGE_SHIFT) | (mmio_addr & ~PAGE_MASK)); if ((unsigned long) io_addr < ZPCI_IOMAP_ADDR_BASE) { @@ -322,7 +324,7 @@ SYSCALL_DEFINE3(s390_pci_mmio_read, unsigned long, mmio_addr, ret = zpci_memcpy_fromio(buf, io_addr, length); out_unlock_pt: - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unlock_mmap: mmap_read_unlock(current->mm); From patchwork Fri Aug 9 16:09:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758954 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3ACC319B3C6 for ; Fri, 9 Aug 2024 16:09:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219786; cv=none; b=aynlJnUOUy9s+2amdi1Bl0Hb7Yyas4DBmjH2URvMD4Gh+nRCQFl5qX1ne0rrzte4S1TeHxnZo77zNvMy6B8zZF9fAyUEGd5fqiayFwZuE20wTwa+yU4Ymcspp14zxnYJ3XYQgoD8hZRP4wmb+nnZWfQU9pJAINHZ6IBmIk3ZdNs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219786; c=relaxed/simple; bh=cEBXN2JPB5h1OhPiVAJoVLWyvN7G/G3nl780c90YYQ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Kc3gA+srep9hOAke50W9eQfGFZ/lPqgo1LkKSZENj0CbatY7vEVH/129FQ3upvZnVVBh9ZHn0J3KoM/Ae1akQEXvVdJ1AXfu268J2TJuvPE9LwJFngm6HzVaE6tTIo5nIMAQmX5tC1JkhAehtIyERTFb/n1QNrrpDLQ4QCWMK6E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Pwy5jr0F; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Pwy5jr0F" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219784; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dHUdq0ZI2fvSzmSbu56kn/TFI4mY4GVPslnR6ZzQETQ=; b=Pwy5jr0FWp9f9lKMhTPNW66YCIQpcq7yB16eGBtHYi0Yz1mpYI/WG1e7dtaZC8N/Ik1H3B A9R6jrD2AmYijGzdH3kk5y9Ue3DZzoMj7kWtQRQkET0Ag4kfZRosAeixyTWusG0h16eNlH Qu2Ls129nThiUeO0Slrj93TkAn/BbfU= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-224-Q4MpPjTNM3ecUiZ0cqc-tg-1; Fri, 09 Aug 2024 12:09:43 -0400 X-MC-Unique: Q4MpPjTNM3ecUiZ0cqc-tg-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6b79c5c972eso2605446d6.1 for ; Fri, 09 Aug 2024 09:09:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219783; x=1723824583; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dHUdq0ZI2fvSzmSbu56kn/TFI4mY4GVPslnR6ZzQETQ=; b=gnUdN4PAKcuagv0h6G9NVTkxzJ4KHOhqBqueYksiG63nrG/mhhezSEYD7VDTli3J+i nUWUE9LHisxA5qzK0MGZkow9G706mCPV4sVhcGIrhjf/3ZuF6DCKZ70TnQ0JBkNGUtlb nPcTdXDlBVbcLlJxCSWm2l9/PbKG1wF3z0DhGtsKPvTj4jDVWETsWUKyGQKzFiXHioU/ fZtzWEL7x1fWUAR3NcF4VGGj6zKaqIGZq4DFJwQ9fFLmian3yfT/Q0zkkjFy/v03y7Gp mPYifrUCRU+/nJ27RUmqTXo+PfEkAWwICUNfYpKQR1kiyS2j8PoWFXUZwoa4fmhLUXgg lptg== X-Forwarded-Encrypted: i=1; AJvYcCWN5ZhUbLKH1BAiuYC2+1DO76hzCrrep+m6s8w2CDs8pJlnlVTn6UWt/xZBMttR3J+b7Zw3VYTo397PY98mH+7tpXtX X-Gm-Message-State: AOJu0YwZd3wzKYB4Un+/LM6xfFy9h1g98jtwY1WR/GuczTviHWrQJNsw /4FafGwrl29j20XXfBl5KVb7iJqCyReFpq3G1R67OlgoEdpmNG2iIMwquoZAtyQX9cRiEvF8fSD sRtf3Pf1eV91E89IdLoYjyWoMvlyd+cxdvSHsukG7gPGKGJZe5A== X-Received: by 2002:a05:620a:460d:b0:7a3:5e3f:cbf4 with SMTP id af79cd13be357-7a4c18467b6mr127416085a.8.1723219782741; Fri, 09 Aug 2024 09:09:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFqMMArTR19FgmAq+7tz35tycS6tSaD4MMjthIpuW8WNcff+L3Yj5M2c5Huoz5dZxDipeszqw== X-Received: by 2002:a05:620a:460d:b0:7a3:5e3f:cbf4 with SMTP id af79cd13be357-7a4c18467b6mr127412385a.8.1723219782335; Fri, 09 Aug 2024 09:09:42 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:41 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 12/19] mm/x86/pat: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:02 -0400 Message-ID: <20240809160909.1023470-13-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/mm/pat/memtype.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index bdc2a240c2aa..fd210b362a04 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -951,23 +951,20 @@ static void free_pfn_range(u64 paddr, unsigned long size) static int follow_phys(struct vm_area_struct *vma, unsigned long *prot, resource_size_t *phys) { - pte_t *ptep, pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vma->vm_start }; - if (follow_pte(vma, vma->vm_start, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - /* Never return PFNs of anon folios in COW mappings. */ - if (vm_normal_folio(vma, vma->vm_start, pte)) { - pte_unmap_unlock(ptep, ptl); + if (!args.special) { + follow_pfnmap_end(&args); return -EINVAL; } - *prot = pgprot_val(pte_pgprot(pte)); - *phys = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - pte_unmap_unlock(ptep, ptl); + *prot = pgprot_val(args.pgprot); + *phys = (resource_size_t)args.pfn << PAGE_SHIFT; + follow_pfnmap_end(&args); return 0; } From patchwork Fri Aug 9 16:09:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758955 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CA1D19D880 for ; Fri, 9 Aug 2024 16:09:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219788; cv=none; b=ZkQu70LmM/t6PI59sc0Yts/E3W3AH43UO3pa9+9e6aK5Sc1w4SMpU6E7Ez28JYjm76N4jCZew/+wG1qkO4dVriupABXTdb9GCp17/8Ay3/dBVq+jGOnj7BygJODMsMEJdRAl0yXKckxVBfcLlU21whHx8NcaX98HfTlS8cpKl50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219788; c=relaxed/simple; bh=BQuf5gTrb+LFvvLLmklTnMaR8Ti7tRzNRxtQXaWJ5ko=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DZRE6DsHEYJskHjYPG18L1pMkql0nWdlHypmpFzg0DBRb+fdNGJ85dPfB72RDKX3eXIKC/DkZ47DGHoQ132CtKUBSx6ajIZXIgYUel6qnWE3fafdJniJy6gb13tQB8DMkdrjL97NqjStZQgkfv9OXrtzK/acsqbQ1RlMrJJFsnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=TutlbcGV; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TutlbcGV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219786; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=TutlbcGVSJNNlcvN9XGBcssDqQCAvuGKSnOUBaHjCtsHktXvs5aGKYIliG1sEOojv/5vYI 1X0Z8Grqev/KiSG8m2MnpNs5yJKbe/o+1A90VrmmhOkYqDdGVwFgeq1X0ULlbndHcOkm5y NSgy7JTqdlzveTIX/SuGDxEwg5Q5h1U= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-617-IHLM6ygLOaeuxJSKqUGobQ-1; Fri, 09 Aug 2024 12:09:45 -0400 X-MC-Unique: IHLM6ygLOaeuxJSKqUGobQ-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-4503ee1b5d6so5447001cf.3 for ; Fri, 09 Aug 2024 09:09:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219784; x=1723824584; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLBqit68L/U2mdwYaFkydFK3txHa+LkmL8n2YTQXnnw=; b=J91QfHg5Xwwc9luhOjiQrz/ZutZDyuvfQ0HJn/ALXP5i+lzLXPJmWshXAv2VfOf/Yn iKllfkNW8Ho97terwxZyZ1PGCOMkvTUElw9OLui7yVoso5S7MOQqcxsW8nFy1UNJJ9SK Th5sIMyKv1qWMs4ycwSHByxZnY5PcS6oYp8gwgtWMhOh2/QYtbZT957cMgtPqYFoknul 2lr0lYnJK7Ns9kQdQNpnaKmEkxJbF5xr/AG7AbzvYVzb9pgRTamV1p+O8wjupkN4D/DL eHr6lnM1Rjt/KvTbAEZiQqYbLY6l5zh1/ZbHuBq5guTNN9UgykCV4YSg7734BkJsimsL kgRg== X-Forwarded-Encrypted: i=1; AJvYcCWSbOLTlz2EFB3lw1Q2KV5M+VMmOy+su03pqYG2+igRckD2JOQ09jBydo9QaZj/XVmjttTd3DjHpkcNxh7SBZugCKTB X-Gm-Message-State: AOJu0YzH8WGrDoqDI1AiA3xM46gtWBwZyimz3jp76dceNVoCytfUhjhr y88SoapZB+R44gwtOYqkYQg/QnWmhZaHSXjXQLIz3kJ0FPbM03aHy86qa4pMQ0zrzqN3oKq9nrI 80Kq6BNcKLNExtpIWddNA2p1j+FsC23xMp/Cwr3mzY8wgtELNdQ== X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14745281cf.6.1723219784587; Fri, 09 Aug 2024 09:09:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEFbC3kPrc0fRPufwvcIMbfwNKbDcEyDmf8f6qiGpsAJzwvtAb4AIbgx0osGWWLWwEBOAE+Pw== X-Received: by 2002:ac8:5f10:0:b0:450:349:1170 with SMTP id d75a77b69052e-4531255c62dmr14745171cf.6.1723219784295; Fri, 09 Aug 2024 09:09:44 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:43 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 13/19] vfio: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:03 -0400 Message-ID: <20240809160909.1023470-14-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Cc: Alex Williamson Cc: Jason Gunthorpe Signed-off-by: Peter Xu --- drivers/vfio/vfio_iommu_type1.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 0960699e7554..bf391b40e576 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -513,12 +513,10 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, bool write_fault) { - pte_t *ptep; - pte_t pte; - spinlock_t *ptl; + struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) { bool unlocked = false; @@ -532,19 +530,17 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, if (ret) return ret; - ret = follow_pte(vma, vaddr, &ptep, &ptl); + ret = follow_pfnmap_start(&args); if (ret) return ret; } - pte = ptep_get(ptep); - - if (write_fault && !pte_write(pte)) + if (write_fault && !args.writable) ret = -EFAULT; else - *pfn = pte_pfn(pte); + *pfn = args.pfn; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); return ret; } From patchwork Fri Aug 9 16:09:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758956 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3EDAB19DF45 for ; Fri, 9 Aug 2024 16:09:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219791; cv=none; b=eTyEigwBofKmCRcIZ751DBOAfZHxa73Nv6IZDvJtx3jyZokq3pKxhIRF3IStv0NPlHFV89SVI1EJ+zivwA5vNES8bWT085Mpyq4XuZ8/64LWttjR5LgnhgvPjDPiUo9g2K414+7olJJmELlurq41T+sl4L4CFeCXTujH/SIomZo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219791; c=relaxed/simple; bh=RxaKGf99KcIXWtv8SyMqO8KfjLx2VTZrfelSTDMLhuM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G2tPd6dxSWXBXc/aYAn1IBnxRir/JGSfy0MrqQhPjg6PQwGISE4qYbT4p7VBhIv5wqtBq6+TBFk5tBamLanNoOmL4pD6pfsdqhuzfMBGiTuGOib5heZMxGaA2FiUc2lvJ2n8LLcDSWus+ffOt+pPm50ucF6CiRps4XCGr5k6joY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=FxGnCoP8; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FxGnCoP8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219789; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=FxGnCoP8SUfK8tA5RcVz8ty+i+dCCHyLzapYZu+lYunDTQifcu8gEcZpFnSBUbAz6sLbP6 sMJeP0OgMdZYek3O8slkrNCFwMn7b2a5EFLF3aFudhwkGT78D7kxshvg8zZ+6hCcCoBlW3 OtZsIw/UVoDnwfczGVeXUXnuP6GwZLM= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-569-JLVXzBLkM42347RHlZGVAA-1; Fri, 09 Aug 2024 12:09:48 -0400 X-MC-Unique: JLVXzBLkM42347RHlZGVAA-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-44ff25bbfe1so3070441cf.3 for ; Fri, 09 Aug 2024 09:09:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219787; x=1723824587; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d0kI5he4gLOJUpsF87CpEeiS1FOCa69CaYI81su9df0=; b=RF4tVWAqmNVW3OYaOUfB8/JN3BKoykBGRyZZHS57Vn5uIn/ZJphByaT91eGJz9zTnl cmt5hwsdlBeFTD9fhRbnMR6u3MO8lOGmMuKa/X3F0igpEI4dTsRSPtQ6fzGxqeDCtKYd IvjXJl7Dw94g69udeQITQPE6ZlSwdS6zk+Y1Q7J4mi3hWCbnF57FN0iJW3ovRIgYT8FJ ag0EUJ+sY1tZMfaLEF5rzgrnpyCR5wuNUUe/K3uTejwPf6obhyQJbB1uWZKg4K1boBcL c6EejvmtIcab4ObmpBsGJ+Iqv1kB4NiAdc1UdwQ/mxW1Q3eiKGlgmwX6JNq/GiVd3Qaj 33tg== X-Forwarded-Encrypted: i=1; AJvYcCWlx9l4QfMspgPGWALnZL9a6+UOWHsdnD5QcBNReWf3aFGDY0iUR0w0R8/yL9WmybsUoH0=@vger.kernel.org X-Gm-Message-State: AOJu0Yx3oJAZD0Cb7zvwzWLQUVa0xp3bp4AvAF3xj/AiZs48wU65qgif i4R4DlcrBqhwkh35p2vG78ZbR0hZsMdlSYZ6V6qiip7tX1VyiDNhb90eBCWHXS0W1zXUS4YgOvl 8fX+mCOQWngOjTz0GJcQTfw2F6zSACQxomihIYJomIY+YVfSJMQ== X-Received: by 2002:ac8:5acc:0:b0:450:26ca:9170 with SMTP id d75a77b69052e-4531255b48dmr13596631cf.5.1723219787023; Fri, 09 Aug 2024 09:09:47 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGxpG5xPFbbw3Y9PVyV1kJ47S21qHA1+Fquj+FmgsVP6Zo0aX1jKS3YWr/9l77XMEl3bpPFxw== X-Received: by 2002:ac8:5acc:0:b0:450:26ca:9170 with SMTP id d75a77b69052e-4531255b48dmr13596351cf.5.1723219786668; Fri, 09 Aug 2024 09:09:46 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:45 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 14/19] acrn: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:04 -0400 Message-ID: <20240809160909.1023470-15-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- drivers/virt/acrn/mm.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index db8ff1d0ac23..4c2f28715b70 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -177,9 +177,7 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) vma = vma_lookup(current->mm, memmap->vma_base); if (vma && ((vma->vm_flags & VM_PFNMAP) != 0)) { unsigned long start_pfn, cur_pfn; - spinlock_t *ptl; bool writable; - pte_t *ptep; if ((memmap->vma_base + memmap->len) > vma->vm_end) { mmap_read_unlock(current->mm); @@ -187,16 +185,20 @@ int acrn_vm_ram_map(struct acrn_vm *vm, struct acrn_vm_memmap *memmap) } for (i = 0; i < nr_pages; i++) { - ret = follow_pte(vma, memmap->vma_base + i * PAGE_SIZE, - &ptep, &ptl); + struct follow_pfnmap_args args = { + .vma = vma, + .address = memmap->vma_base + i * PAGE_SIZE, + }; + + ret = follow_pfnmap_start(&args); if (ret) break; - cur_pfn = pte_pfn(ptep_get(ptep)); + cur_pfn = args.pfn; if (i == 0) start_pfn = cur_pfn; - writable = !!pte_write(ptep_get(ptep)); - pte_unmap_unlock(ptep, ptl); + writable = args.writable; + follow_pfnmap_end(&args); /* Disallow write access if the PTE is not writable. */ if (!writable && From patchwork Fri Aug 9 16:09:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758957 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D826119DF5F for ; Fri, 9 Aug 2024 16:09:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219793; cv=none; b=Da3ziQiz7y0sHtx8MjozEo301eSKXSdj0EItOqC0z5Qjxyv4pu7NWi9U6VupK9kK4hJadw37CGzGFNGNoYnTv6fAcdrbfPkeNMURpJlk0GuIxDXonhEchWjRx6woCfuqF0WUAJaA/hFKCsWYDTSrvnySd2btx7PBqinxa4I2VBc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219793; c=relaxed/simple; bh=oaBnSvpJERjRBU4mG2zRAc+u205z76QvJz7BlHn3mFk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ub5zuwndqBuoZNA7zuEc/rbRuJ0zpWTL8+RNggOvc/sn1pHlODwlQFqzHWmW940Da3R8PJ+qa2oNCWYTS5NXdTbh8iB6+Xy4N/IFmlIGKGhyX4ftw9xV0uuUvOjdO2ij3c7TxZA4e4D/IZ/4pTPsRSHsbougUn7dS+UEH7QjH40= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=EWkpGJB9; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EWkpGJB9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219790; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9FTs1XtnGv67Q34nCFPbcUqDR1Dyk8Tfq0P0nZ4zw7U=; b=EWkpGJB9yukEdJFlqJlsDn0v09vGPmQHACYE792f040Ts870fkR/OmRP45y9m53MjPvMBt nI8YZg8Ed1u8evzruu7rxZ18p0P6izczQD3/YulugyF2Iu5nvgPKM4k0EeTmtQSKWxvcF6 Od/mFbvn0Kg1Ot6DCHbJhPzn3LmXDnA= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-634-GOfxvW5hPrKLpt3uKCltMQ-1; Fri, 09 Aug 2024 12:09:49 -0400 X-MC-Unique: GOfxvW5hPrKLpt3uKCltMQ-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-6b7973ab9cbso5685826d6.3 for ; Fri, 09 Aug 2024 09:09:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219789; x=1723824589; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9FTs1XtnGv67Q34nCFPbcUqDR1Dyk8Tfq0P0nZ4zw7U=; b=R3qzg+Ex1dVsY/YtWyUttYTkL+xXHuLl4EVtfsFr0l/K1JgwsczdySVjF9S9o45Wca Lk2LbfkdAnMNNQ9Pucs1up0mU9pFsB0spoYYuPR2wu7CCt4auwiJcnleXnr3vVkWJy56 CGXl8e5yXDdDU780lBgp8wyebddkV1YZCC9C+dszcnvehsSZI71j/U5wP7PDOp2Uv5D2 61u8O9nfyAbS+5czBCqFf6E9O4bBMA1qjI1MmL9ntQvAehDhvAVzyr+FCcMBBA3RLoQH zzpphU9yKn9Ddg4UNUqtCZdhxyR71DdPGUSJyrWsEBIQrCduwaz9Y2WtKPfKuh7Va/Yj PK3g== X-Forwarded-Encrypted: i=1; AJvYcCUrdW37LvxhYf9Y44CLCdvC/NkHy8dvaq6Qsn7dd/GheRwtDHdiVdpw/Wx7lx3Te+13jbE9/kwMaaDhAVV3o/9wOZfw X-Gm-Message-State: AOJu0Yxrpf3TZd9W8N3sXBHg4E7jaI351WYM/w1qD4lW9g8hOL6BHtLs nw2qx1yQj6dYbKdfIoFO7uec7XE8jsMT7DU8m/FmBC4tYMarE/wGHCcN2GiMQOwp5fW6q+BFedc NxZ9DX3Knh+X/rh0XEniv8ehtcfFnbg1Yb4c5fLp2gnakMvNW3Q== X-Received: by 2002:a05:622a:11d1:b0:446:5b0a:e778 with SMTP id d75a77b69052e-4531235a44cmr11674971cf.0.1723219788788; Fri, 09 Aug 2024 09:09:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH3kKOCpVIG6mSA6dMC4Xb4vB8Z6SmUroM26OhSq4nP1v0KE/gOqkuWVbJLnRl3s3aUYCeIEQ== X-Received: by 2002:a05:622a:11d1:b0:446:5b0a:e778 with SMTP id d75a77b69052e-4531235a44cmr11674811cf.0.1723219788421; Fri, 09 Aug 2024 09:09:48 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:48 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 15/19] mm/access_process_vm: Use the new follow_pfnmap API Date: Fri, 9 Aug 2024 12:09:05 -0400 Message-ID: <20240809160909.1023470-16-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use the new API that can understand huge pfn mappings. Signed-off-by: Peter Xu --- mm/memory.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2194e0f9f541..313c17eedf56 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6504,34 +6504,34 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, resource_size_t phys_addr; unsigned long prot = 0; void __iomem *maddr; - pte_t *ptep, pte; - spinlock_t *ptl; int offset = offset_in_page(addr); int ret = -EINVAL; + bool writable; + struct follow_pfnmap_args args = { .vma = vma, .address = addr }; retry: - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) return -EINVAL; - pte = ptep_get(ptep); - pte_unmap_unlock(ptep, ptl); + prot = pgprot_val(args.pgprot); + phys_addr = (resource_size_t)args.pfn << PAGE_SHIFT; + writable = args.writable; + follow_pfnmap_end(&args); - prot = pgprot_val(pte_pgprot(pte)); - phys_addr = (resource_size_t)pte_pfn(pte) << PAGE_SHIFT; - - if ((write & FOLL_WRITE) && !pte_write(pte)) + if ((write & FOLL_WRITE) && !writable) return -EINVAL; maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot); if (!maddr) return -ENOMEM; - if (follow_pte(vma, addr, &ptep, &ptl)) + if (follow_pfnmap_start(&args)) goto out_unmap; - if (!pte_same(pte, ptep_get(ptep))) { - pte_unmap_unlock(ptep, ptl); + if ((prot != pgprot_val(args.pgprot)) || + (phys_addr != (args.pfn << PAGE_SHIFT)) || + (writable != args.writable)) { + follow_pfnmap_end(&args); iounmap(maddr); - goto retry; } @@ -6540,7 +6540,7 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, else memcpy_fromio(buf, maddr + offset, len); ret = len; - pte_unmap_unlock(ptep, ptl); + follow_pfnmap_end(&args); out_unmap: iounmap(maddr); From patchwork Fri Aug 9 16:09:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758958 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 838BA19DF8D for ; Fri, 9 Aug 2024 16:09:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219795; cv=none; b=sczpQ/YF70Ya//j69s5a5QgNVTNHAKnpJgSThFzOixo/RAQQ49orQKMJ0EocyUlrQmn5Y2ptyj6T4S8Ym+9bhRTLajSDGaBEgeSi9cPOSYbUW66oY3jiKFxoYFsLUYORwxrenTSAA13cL20dx5uig30CdJuShhK7jrIZ8UvKgcg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219795; c=relaxed/simple; bh=jYEp23FOIF/6kH+o3wuOpGOrvsbC/NmaM2/0jhGYsK8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XZcmAUz+GVsM+BibuQchJRnESSUGwl6j1hy4qAFKC+bZGtiJB6GMkp3VWB4ueiFE50uE0ZANVI1/gAx1vSVjRzyXapwv6XOX2VOzlxDZ5fuc67s2AGCLf0Lk9OBREus3ULBBZOwzjrHbdjFeUQ1We1ZN5NZWQoH/yh0h3vL4ysU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=EDJ3sZfv; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EDJ3sZfv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rQ5uhZ2uiWdH1/UlOzJ8LVGX9FzFBDg5i1FarWNBfgY=; b=EDJ3sZfvJOAoL/XjHxfXyGwrwHufTelKwswGuzgdGQNDGqDvEmTEjKkt5yPmvTfZRw0OMA nMRUy+wOJqvy1geBhF/abXLGjZwRAxr7ZWLUgQGN/qt+P9zGa+OOAO+tkiIic05B7BY0DP H8I2wpa7K/VwKVGN0bH6/TRCwlg09lQ= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-80-YcosiXs-OXyHZjt4TVJ9aw-1; Fri, 09 Aug 2024 12:09:51 -0400 X-MC-Unique: YcosiXs-OXyHZjt4TVJ9aw-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-44fe28cd027so3758561cf.0 for ; Fri, 09 Aug 2024 09:09:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219791; x=1723824591; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rQ5uhZ2uiWdH1/UlOzJ8LVGX9FzFBDg5i1FarWNBfgY=; b=OgJrEPLPSBhFwZQlm2EwBEcXvImPKOHuT4c4q7aUjU2LFS2KzR3G7zMd3VIcxfNWFZ PlfwQ3nukIXZZ7c/mEef5jm1Cy/jFpEUZ7PHLuC6Fq4/4ccSAyE44QzIfCFePVJ41Q0W HUkvun32JHmBMRhwSchUzhZLaaXO/lIewtkXY9mAbGoFmW9lV36tR7s9jtKE4jpLPPtz kF2+jt3PI5G+87hWu8gcnPaHSdXGfgHbBx7Rhvb+JbSJw1dxSxyOqEaRixmuCdsOZLmi 7/UUGLq0JVAL9l5/Ejl4q0p+sVcZiSnHsNizr+F6QY8HRb9E/LaYnupG0z4wr6xwDfGA DCLQ== X-Forwarded-Encrypted: i=1; AJvYcCXCiY3VA5HmbcYE/0M52QVsa+NcZMD4F7F0vBUcOHTn7QuSL/7JC+j3H1ZQncv13iE6+9s=@vger.kernel.org X-Gm-Message-State: AOJu0YxgVmdzMrkKUDLDjGzTPHmuP2H8fhk6YOmR/cnEk0Mrdq7mbnm7 sUBtLS5eTXG/ze01JeJ2rmwDGDog0N+AtiGdYdX+dbMncJbA9NxxR/PqTWKVhL7gH3yw2+bEKSd dWNUSzN2XYgAtoYby+RhmmqoCB8jsRa5C3ORYLaZXL5o+7QqPVw== X-Received: by 2002:a05:622a:412:b0:444:b935:ebb5 with SMTP id d75a77b69052e-45312359fbcmr13373571cf.0.1723219790803; Fri, 09 Aug 2024 09:09:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2UCzm8Bb1pveediLGLwWSJm3e36K8ZWtfh9bSQqsAnCTBAfRwieQENSqzRtCTpHEz3lk41Q== X-Received: by 2002:a05:622a:412:b0:444:b935:ebb5 with SMTP id d75a77b69052e-45312359fbcmr13373301cf.0.1723219790408; Fri, 09 Aug 2024 09:09:50 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:49 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 16/19] mm: Remove follow_pte() Date: Fri, 9 Aug 2024 12:09:06 -0400 Message-ID: <20240809160909.1023470-17-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 follow_pte() users have been converted to follow_pfnmap*(). Remove the API. Signed-off-by: Peter Xu --- include/linux/mm.h | 2 -- mm/memory.c | 73 ---------------------------------------------- 2 files changed, 75 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7471302658af..c5949b8052c6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2369,8 +2369,6 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp); int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/memory.c b/mm/memory.c index 313c17eedf56..72f61fffdda2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6265,79 +6265,6 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) } #endif /* __PAGETABLE_PMD_FOLDED */ -/** - * follow_pte - look up PTE at a user virtual address - * @vma: the memory mapping - * @address: user virtual address - * @ptepp: location to store found PTE - * @ptlp: location to store the lock for the PTE - * - * On a successful return, the pointer to the PTE is stored in @ptepp; - * the corresponding lock is taken and its location is stored in @ptlp. - * - * The contents of the PTE are only stable until @ptlp is released using - * pte_unmap_unlock(). This function will fail if the PTE is non-present. - * Present PTEs may include PTEs that map refcounted pages, such as - * anonymous folios in COW mappings. - * - * Callers must be careful when relying on PTE content after - * pte_unmap_unlock(). Especially if the PTE maps a refcounted page, - * callers must protect against invalidation with MMU notifiers; otherwise - * access to the PFN at a later point in time can trigger use-after-free. - * - * Only IO mappings and raw PFN mappings are allowed. The mmap semaphore - * should be taken for read. - * - * This function must not be used to modify PTE content. - * - * Return: zero on success, -ve otherwise. - */ -int follow_pte(struct vm_area_struct *vma, unsigned long address, - pte_t **ptepp, spinlock_t **ptlp) -{ - struct mm_struct *mm = vma->vm_mm; - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - - mmap_assert_locked(mm); - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) - goto out; - - if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) - goto out; - - pgd = pgd_offset(mm, address); - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - goto out; - - p4d = p4d_offset(pgd, address); - if (p4d_none(*p4d) || unlikely(p4d_bad(*p4d))) - goto out; - - pud = pud_offset(p4d, address); - if (pud_none(*pud) || unlikely(pud_bad(*pud))) - goto out; - - pmd = pmd_offset(pud, address); - VM_BUG_ON(pmd_trans_huge(*pmd)); - - ptep = pte_offset_map_lock(mm, pmd, address, ptlp); - if (!ptep) - goto out; - if (!pte_present(ptep_get(ptep))) - goto unlock; - *ptepp = ptep; - return 0; -unlock: - pte_unmap_unlock(ptep, *ptlp); -out: - return -EINVAL; -} -EXPORT_SYMBOL_GPL(follow_pte); - static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, spinlock_t *lock, pte_t *ptep, pgprot_t pgprot, unsigned long pfn_base, From patchwork Fri Aug 9 16:09:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758959 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D702D19E7EF for ; Fri, 9 Aug 2024 16:09:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219797; cv=none; b=ho4IWOIU+XG4EFbRgAPPUYNGu/v8B8FXV5iO/EpRxGHUsvqu8NDbtKNJE3loaVnIzodqSLIMZ8nLw7Ow7fdPk4YZvMMaqWmibFlhZP+PX+1Wz5jdnnZROpdz5p+9gBBwvt45xwixjJyKD0WDLiJkmC7wl6pfKHdrpIPPqksWm4E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219797; c=relaxed/simple; bh=6vd9dKzSoYHnE87pbZ79rfnDAvgUPCtQqMHJIVAxycI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bEHSFeL/OW3vTcZgYhA0p2Y/eEgdtbWmWw6pjOMahrHRvBgGXX5Pw/P4trlHQ77MK18yIUxWwnju35oYV+Fy2lDenXleXjUo7uMVCNfmz5Ig4ob3mmwLD4bk0DPfOSTjRHI0+yrQzEjPlewAnkwOeMro7Qb39Mg4tmoT03fSGAQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=hprFUyNs; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="hprFUyNs" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219795; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YoGoBxg1RIhRUWN8551lXVuosUhfDUfYXVK/Rz8tB4Q=; b=hprFUyNsXu62UOaW2ILvBg7uWsxhM3gv11ZLAxlf4KKM9ghkPhiXGh9IucHRr/f0/doycp htU9oEmYo291aq5zOGkqyMsy22a8ACoBjkEGHDIOaxpZKTkqERnJvvR6cbAdMGkM+lyi/E vpEium1+71XvyZkIMr9s2V93k1wyaNw= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-678-xiAWKRP_Ppu4CKXKvjgz0g-1; Fri, 09 Aug 2024 12:09:53 -0400 X-MC-Unique: xiAWKRP_Ppu4CKXKvjgz0g-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-45009e27b83so4906271cf.1 for ; Fri, 09 Aug 2024 09:09:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219793; x=1723824593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YoGoBxg1RIhRUWN8551lXVuosUhfDUfYXVK/Rz8tB4Q=; b=JC7lOhXKa5evAwD9F3aUS/8RJYi1mGmuAmAZL/jCVbwJQu6/Z8RKCSMRo7WQ1jYbgN ySAigj0JsaAEDuivvEg8E5vXV8l5r35rRmtVFrKvRF9UITysdQR2UjmQaZOHsAvddLAi tR0ZKRy0O7JNwxGfsJ9PYJvOjEQcqTxK/xd2SudWLu8SI+n8tYrC6x3z13BW9nFrajFG 5jED69adsrEywQY6HyIoADaUoXuVG6PUsUmiU21QrZNVxdMY4Zk/xQp6qufDTcV6NilW gSXClqWtVj/ClOEhlRB8j0eCfW02aMD6vNI0gLMJQ2e6EH3bc4uWUxAfcKYpP5syhwr+ tYnw== X-Forwarded-Encrypted: i=1; AJvYcCWx7nJEcS/kWqGTFkyT0+TsIs469tW1Ig/m2Pn4THZC9ePEW7mzf04QhDdZSKL46hqGkOQnbRyvjcQZ4UvwmCmT/b3O X-Gm-Message-State: AOJu0YyzFSyABL62nYDlPcmkWS9sjcltVGdpamT02XJV8BcNgeSDRU16 bbJom05tR98HJOPNMPetaRpoJP4RD1wXImtDWFNQ1cYD5dtPdSiXbk3FT/BUWH/W44CewL2sF97 6wsOpX+sCbkYelfySSinLAFs5HF8bRthrZg8lOTAtPuCHUVX3eQ== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14005731cf.7.1723219793238; Fri, 09 Aug 2024 09:09:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHyC9JbjVCeQC7LVvckqh3sKVh2QUlZ0IzLJIVdarr6jR5vj53aOO53iHwQIsrAsjgq+0MAjA== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14005321cf.7.1723219792685; Fri, 09 Aug 2024 09:09:52 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:51 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 17/19] mm/x86: Support large pfn mappings Date: Fri, 9 Aug 2024 12:09:07 -0400 Message-ID: <20240809160909.1023470-18-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Helpers to install and detect special pmd/pud entries. In short, bit 9 on x86 is not used for pmd/pud, so we can directly define them the same as the pte level. One note is that it's also used in _PAGE_BIT_CPA_TEST but that is only used in the debug test, and shouldn't conflict in this case. One note is that pxx_set|clear_flags() for pmd/pud will need to be moved upper so that they can be referenced by the new special bit helpers. There's no change in the code that was moved. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 80 ++++++++++++++++++++++------------ 2 files changed, 53 insertions(+), 28 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index acd9745bf2ae..7a3fb2ff3e72 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index a7c1e9cfea41..1e463c9a650f 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,6 +120,34 @@ extern pmdval_t early_pmd_flags; #define arch_end_context_switch(prev) do {} while(0) #endif /* CONFIG_PARAVIRT_XXL */ +static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v | set); +} + +static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v & ~clear); +} + +static inline pud_t pud_set_flags(pud_t pud, pudval_t set) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v | set); +} + +static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v & ~clear); +} + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -317,6 +345,30 @@ static inline int pud_devmap(pud_t pud) } #endif +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SPECIAL; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SPECIAL; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud_set_flags(pud, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + static inline int pgd_devmap(pgd_t pgd) { return 0; @@ -487,20 +539,6 @@ static inline pte_t pte_mkdevmap(pte_t pte) return pte_set_flags(pte, _PAGE_SPECIAL|_PAGE_DEVMAP); } -static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v | set); -} - -static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pmd_t pmd_mksaveddirty(pmd_t pmd) { @@ -595,20 +633,6 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); #define pmd_mkwrite pmd_mkwrite -static inline pud_t pud_set_flags(pud_t pud, pudval_t set) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v | set); -} - -static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pud_t pud_mksaveddirty(pud_t pud) { From patchwork Fri Aug 9 16:09:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758960 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9DBA19E828 for ; Fri, 9 Aug 2024 16:09:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219799; cv=none; b=p2enzGrjjrbiKDLMoVxbVyMZu7gFrcL3PzZFRs+B3i81r2MFg83k5Tffjc+KypNwCjuFV9PiV3lm/trwog/53l+nV6+CKQUfWFqFApSU8tJtFRAf4sTdM89eW0PxkIB0oz8Onngrr1up3Ic+weol2pxCNBsgB1iAjRUV/HdTRdY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219799; c=relaxed/simple; bh=Pk1TeOIKWSjAqBlIFkGydJm2rF11Zc/q9MVN9zTRZbQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mxYCQPi1DvTkc0rjulFwLOz/YiPOqMpGy/8TXTa/jA+GjiZsaFt19jFSzqL74YHSM+UUUR0Tbb4MVtP2EyGLEgJbHvCUNAMR7vZz6tj9DGCT1M0Ywa2dDqsPCNdobxkacJHQGhfYQiFZlZiGqrgZldG5aTi1wBcLXfqkUwF1B/U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Lpo0D3yx; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Lpo0D3yx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219796; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=Lpo0D3yxB98V3pjIwIbcRmR0iGbEGA4mNWJ13SnVz9N9BqWg9G6VEm/G581C02CiT3hToL uxvjbKAsyPwjyHH5cHmxh34+glNxbJFRAR2aJmwcCq2sQsKHxS8WzpoPST0CoJvBbr0elp +H2h/9g/RiXcJgWnHyMWcB0xYQXTNlc= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-54-YHPtvVxrN7iRc0NFCikK7A-1; Fri, 09 Aug 2024 12:09:55 -0400 X-MC-Unique: YHPtvVxrN7iRc0NFCikK7A-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44ff585970bso4757171cf.2 for ; Fri, 09 Aug 2024 09:09:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219795; x=1723824595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=XyBoxmrRAqp5Em0FfsaTjEOk+zwlHeWJgCvSPepUGCqXoaFaXLLMClf0BZ24SSsuoy Uq/qwKuSrQSB2yXnCpoYjwJ3cQ81E4+GcAPFbIvkUDRbgzzHecMkP1YYCpNqGMfrSI7j ZzGdyYgH8F4H879fg67SIC3c+R4saCbvVZViWMUMGOgX3FfM2Fg7Uv/WDvr/4EqIZUJ2 LEWpj1FTXZIq2O9Gh73NeKi8iD+iLKHzV490CxfR818XkXGUsqL7pWKkuhyQCryCjypI eaDbo9EIhQS1w7myDOMyN8RorN58spHR4INYVIfw0/URM6GGaY//X3gIhVjzJQ/5GDk8 XbZg== X-Forwarded-Encrypted: i=1; AJvYcCVnFZ9mTJSg5KmbyPBwDDwD327jgMfiTiHgBu9q+Cr9oayKAguLECECrw60aAP3S8eMUS4fK32X9c49G0FU8NMXDjbM X-Gm-Message-State: AOJu0YyIxet0rX2RbpYzo+i/uIqJqZ+mdr0eoVObbv43IvvvdBUllBlG oIaoLO1LBltVVHMPVmr8z4ikM8/K9TRAvnze5P8KzFBQaSmLzvqzadfwgY/AjvTXk4khPNBnzY5 g91t/Fbx61QJhYcIYhY6icCtW+oTgYanxvZlPGasxiq4KaGJDBQ== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14006661cf.7.1723219795211; Fri, 09 Aug 2024 09:09:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEzDYjFq+v9SYAdrw2G1PaR/i1nnMK5UGwVdyzJwk9mK3TcvaabWRsZT43a6L+rPfA/Jc6OAg== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14006381cf.7.1723219794745; Fri, 09 Aug 2024 09:09:54 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:54 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 18/19] mm/arm64: Support large pfn mappings Date: Fri, 9 Aug 2024 12:09:08 -0400 Message-ID: <20240809160909.1023470-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b3fc891f1544..5f026b95f309 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn = pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr) From patchwork Fri Aug 9 16:09:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758961 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31CBF19EECF for ; Fri, 9 Aug 2024 16:10:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219801; cv=none; b=eCY6X0B2ZMduqmztL7CeYOiaE/OEukSZTBNJa6ApbsZlLghrSwDQ+fqUjTAQd17pi9PNCDNMHZOZIKE88UMs0tS2X1xYlsI6NagfuhOGMjejmykp97UoP9/iqXQNT8USRbjMAmq4cdoEznHOPw75oLZopdHcr1wrHtrwqv7i2Jc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723219801; c=relaxed/simple; bh=Dmoew7i8XEW+gHBA0zJyGyEzte9fLwrtktcen/8enVY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HxsWBysv+A6vEkQGmibBKDCgpGHEC61Uu7buTi1+qZXM4jqL+fXonBS2YfqoIGWszEAFWeiEFbY3jZT+r7Oydt0qaf+nF+Ou4ZGrDjVmD+ax5q7BawiyUJ0W8pz2C/JxxUvJK96opCLpFPdvVF3lVU0cx/0E/REJnWO1zsQ+JnU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=cyAa6GFL; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="cyAa6GFL" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219799; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=cyAa6GFLGHtlzcmKnPcY4J3slYxrQLW16267/YAdLhGoMkxswEZlf/p8HEA+JSDpmJeHeX QWaENn6DKc6zsqqZjThCWeb9xuC1e6GfmvLiq/X5UjOaztJZdUm3IAHfSCtHvIuq5DRI+L ZRHzC3pEr6KmjX6OEJmyU8Tj8JDxPTw= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-367-AN3ttUz2OrSLrByOfQ0bMQ-1; Fri, 09 Aug 2024 12:09:58 -0400 X-MC-Unique: AN3ttUz2OrSLrByOfQ0bMQ-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1e1828321so6887885a.1 for ; Fri, 09 Aug 2024 09:09:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219797; x=1723824597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9MPOki3YF/wyEgBnejzItxwjIF1moGifGifGb5OwzY4=; b=PilIKMhYCWj3C+B9YfccGYm1iG9F0UAd4bK0i7HaDuhnHoJTiZEOpqWB1/OjImLvfa BSR+Sj7f607ypVm60rrUoi4RMO0SGXuF4nj/dXLwjYqAEDcmju0N3TLFL9ypF/uWhjnN +f+7sNUMGMB4/3wvhKVjg+X56socZGPHMrewnfIl+XdDWfbrdTQ9n7D5MkI4RoWj8vof nXfF08KFh3/yhrV4t42ofGs89dZfmIcd7fCushJa6fIzZFQJ/NTHa8a2uvmOGXFOxKwj UvwBRV/Ns4t96yEa7NsdBpaMPAEUzUfJlKzhqu0Evfazw0Pzy6JQWWCmZ4QJsBCrarvs szAg== X-Forwarded-Encrypted: i=1; AJvYcCVp3X+/4MRnikYBzplanpacqJHxWCbaRcV7EnJQXq8cPoRL/U+jzrxnpcaEkRkM6o+psXNxpEIa607gtz8DFVVJAMuW X-Gm-Message-State: AOJu0YyFf5Se5q3VaiZMBGkQUS+amptnw60UN0lco4mFiueFHgqMb1JW GplBvOoZVTI/dbjK03O3jcmTyjJdVBniwBHI6qM6rw+k8id7RXPoOmJ5RDxURhN/XgwNVah3yvF E54suH9x6q1lLA73SOXmi8JHR+1TuBluKvAvBSV9Ys1qgKKNyKA== X-Received: by 2002:a05:622a:1b8d:b0:446:58f5:e6ff with SMTP id d75a77b69052e-4531251d52cmr13544951cf.2.1723219797251; Fri, 09 Aug 2024 09:09:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHJY3GNDsLRixrwL3Yd/1vgrL/iS2SbMHaD3fo6KrtERFLLt1PJ4ncszNtj/1mVMrjvOJRHSw== X-Received: by 2002:a05:622a:1b8d:b0:446:58f5:e6ff with SMTP id d75a77b69052e-4531251d52cmr13544641cf.2.1723219796830; Fri, 09 Aug 2024 09:09:56 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:56 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 19/19] vfio/pci: Implement huge_fault support Date: Fri, 9 Aug 2024 12:09:09 -0400 Message-ID: <20240809160909.1023470-20-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Alex Williamson With the addition of pfnmap support in vmf_insert_pfn_{pmd,pud}() we can take advantage of PMD and PUD faults to PCI BAR mmaps and create more efficient mappings. PCI BARs are always a power of two and will typically get at least PMD alignment without userspace even trying. Userspace alignment for PUD mappings is also not too difficult. Consolidate faults through a single handler with a new wrapper for standard single page faults. The pre-faulting behavior of commit d71a989cf5d9 ("vfio/pci: Insert full vma on mmap'd MMIO fault") is removed in this refactoring since huge_fault will cover the bulk of the faults and results in more efficient page table usage. We also want to avoid that pre-faulted single page mappings preempt huge page mappings. Cc: kvm@vger.kernel.org Signed-off-by: Alex Williamson Signed-off-by: Peter Xu --- drivers/vfio/pci/vfio_pci_core.c | 60 +++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 17 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index ba0ce0075b2f..2d7478e9a62d 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1657,14 +1658,20 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma) return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff; } -static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) +static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, + unsigned int order) { struct vm_area_struct *vma = vmf->vma; struct vfio_pci_core_device *vdev = vma->vm_private_data; unsigned long pfn, pgoff = vmf->pgoff - vma->vm_pgoff; - unsigned long addr = vma->vm_start; vm_fault_t ret = VM_FAULT_SIGBUS; + if (order && (vmf->address & ((PAGE_SIZE << order) - 1) || + vmf->address + (PAGE_SIZE << order) > vma->vm_end)) { + ret = VM_FAULT_FALLBACK; + goto out; + } + pfn = vma_to_pfn(vma); down_read(&vdev->memory_lock); @@ -1672,30 +1679,49 @@ static vm_fault_t vfio_pci_mmap_fault(struct vm_fault *vmf) if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) goto out_unlock; - ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); - if (ret & VM_FAULT_ERROR) - goto out_unlock; - - /* - * Pre-fault the remainder of the vma, abort further insertions and - * supress error if fault is encountered during pre-fault. - */ - for (; addr < vma->vm_end; addr += PAGE_SIZE, pfn++) { - if (addr == vmf->address) - continue; - - if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) - break; + switch (order) { + case 0: + ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); + break; +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + case PMD_ORDER: + ret = vmf_insert_pfn_pmd(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP + case PUD_ORDER: + ret = vmf_insert_pfn_pud(vmf, __pfn_to_pfn_t(pfn + pgoff, + PFN_DEV), false); + break; +#endif + default: + ret = VM_FAULT_FALLBACK; } out_unlock: up_read(&vdev->memory_lock); +out: + dev_dbg_ratelimited(&vdev->pdev->dev, + "%s(,order = %d) BAR %ld page offset 0x%lx: 0x%x\n", + __func__, order, + vma->vm_pgoff >> + (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT), + pgoff, (unsigned int)ret); return ret; } +static vm_fault_t vfio_pci_mmap_page_fault(struct vm_fault *vmf) +{ + return vfio_pci_mmap_huge_fault(vmf, 0); +} + static const struct vm_operations_struct vfio_pci_mmap_ops = { - .fault = vfio_pci_mmap_fault, + .fault = vfio_pci_mmap_page_fault, +#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP + .huge_fault = vfio_pci_mmap_huge_fault, +#endif }; int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma)