From patchwork Sun Mar 21 18:45:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Thomas_Hellstr=C3=B6m_=28Intel=29?= X-Patchwork-Id: 12153243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D1C1C433C1 for ; Sun, 21 Mar 2021 18:46:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 14EC36194F for ; Sun, 21 Mar 2021 18:46:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 14EC36194F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shipmail.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1E2DB6B0070; Sun, 21 Mar 2021 14:46:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 118976B0075; Sun, 21 Mar 2021 14:46:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D68FE6B0074; Sun, 21 Mar 2021 14:46:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0213.hostedemail.com [216.40.44.213]) by kanga.kvack.org (Postfix) with ESMTP id AF2216B0071 for ; Sun, 21 Mar 2021 14:46:15 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 69E031E01 for ; Sun, 21 Mar 2021 18:46:15 +0000 (UTC) X-FDA: 77944761510.03.2766213 Received: from ste-pvt-msa1.bahnhof.se (ste-pvt-msa1.bahnhof.se [213.80.101.70]) by imf23.hostedemail.com (Postfix) with ESMTP id 9D1E6A0009D5 for ; Sun, 21 Mar 2021 18:46:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ste-pvt-msa1.bahnhof.se (Postfix) with ESMTP id 60C4042376; Sun, 21 Mar 2021 19:46:11 +0100 (CET) Authentication-Results: ste-pvt-msa1.bahnhof.se; dkim=pass (1024-bit key; unprotected) header.d=shipmail.org header.i=@shipmail.org header.b=PQ83JxXd; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at bahnhof.se Received: from ste-pvt-msa1.bahnhof.se ([127.0.0.1]) by localhost (ste-pvt-msa1.bahnhof.se [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DYYkbKdnSTuh; Sun, 21 Mar 2021 19:46:09 +0100 (CET) Received: by ste-pvt-msa1.bahnhof.se (Postfix) with ESMTPA id 11C0141474; Sun, 21 Mar 2021 19:46:09 +0100 (CET) Received: from localhost.localdomain (2.70.38.73.mobile.tre.se [2.70.38.73]) by mail1.shipmail.org (Postfix) with ESMTPSA id 33797361E5A; Sun, 21 Mar 2021 19:46:08 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=shipmail.org; s=mail; t=1616352368; bh=eIp5oHPDAcmaQRleCtZI+6MjuyOrnVz5ZEJhMg48fNo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PQ83JxXd/yVYagz6sx67TBDGljF2mEwjfTIVFv928dIHgfA4GwaNO+MXLPeRFMWdo hdLKCdJ9AwSB8a8T8cfCBNouOq8LIVNboC9LIv6jdRK4YaCKCXNl13mJBfxiF1SLUA 5fIvG6eW8tTml8wQJbPkv3RRBJ6QpCwV6RCQwnk8= From: =?utf-8?q?Thomas_Hellstr=C3=B6m_=28Intel=29?= To: dri-devel@lists.freedesktop.org Cc: =?utf-8?q?Thomas_Hellstr=C3=B6m_=28Intel=29?= , Christian Koenig , David Airlie , Daniel Vetter , Andrew Morton , Jason Gunthorpe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 1/2] mm,drm/ttm: Block fast GUP to TTM huge pages Date: Sun, 21 Mar 2021 19:45:28 +0100 Message-Id: <20210321184529.59006-2-thomas_os@shipmail.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210321184529.59006-1-thomas_os@shipmail.org> References: <20210321184529.59006-1-thomas_os@shipmail.org> MIME-Version: 1.0 X-Stat-Signature: osobmaukn65r94enwz4rm988rgkapna9 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 9D1E6A0009D5 Received-SPF: none (shipmail.org>: No applicable sender policy available) receiver=imf23; identity=mailfrom; envelope-from=""; helo=ste-pvt-msa1.bahnhof.se; client-ip=213.80.101.70 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616352372-911438 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: TTM sets up huge page-table-entries both to system- and device memory, and we don't want gup to assume there are always valid backing struct pages for these. For PTEs this is handled by setting the pte_special bit, but for the huge PUDs and PMDs, we have neither pmd_special nor pud_special. Normally, huge TTM entries are identified by looking at vma_is_special_huge(), but fast gup can't do that, so as an alternative define _devmap entries for which there are no backing dev_pagemap as special, update documentation and make huge TTM entries _devmap, after verifying that there is no backing dev_pagemap. One other alternative would be to block TTM huge page-table-entries completely, and while currently only vmwgfx use them, they would be beneficial to other graphis drivers moving forward as well. Cc: Christian Koenig Cc: David Airlie Cc: Daniel Vetter Cc: Andrew Morton Cc: Jason Gunthorpe Cc: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Thomas Hellström (Intel) --- drivers/gpu/drm/ttm/ttm_bo_vm.c | 17 ++++++++++++++++- mm/gup.c | 21 +++++++++++---------- mm/memremap.c | 5 +++++ 3 files changed, 32 insertions(+), 11 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 6dc96cf66744..1c34983480e5 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -195,6 +195,7 @@ static vm_fault_t ttm_bo_vm_insert_huge(struct vm_fault *vmf, pfn_t pfnt; struct ttm_tt *ttm = bo->ttm; bool write = vmf->flags & FAULT_FLAG_WRITE; + struct dev_pagemap *pagemap; /* Fault should not cross bo boundary. */ page_offset &= ~(fault_page_size - 1); @@ -210,6 +211,20 @@ static vm_fault_t ttm_bo_vm_insert_huge(struct vm_fault *vmf, if ((pfn & (fault_page_size - 1)) != 0) goto out_fallback; + /* + * Huge entries must be special, that is marking them as devmap + * with no backing device map range. If there is a backing + * range, Don't insert a huge entry. + * If this check turns out to be too much of a performance hit, + * we can instead have drivers indicate whether they may have + * backing device map ranges and if not, skip this lookup. + */ + pagemap = get_dev_pagemap(pfn, NULL); + if (pagemap) { + put_dev_pagemap(pagemap); + goto out_fallback; + } + /* Check that memory is contiguous. */ if (!bo->mem.bus.is_iomem) { for (i = 1; i < fault_page_size; ++i) { @@ -223,7 +238,7 @@ static vm_fault_t ttm_bo_vm_insert_huge(struct vm_fault *vmf, } } - pfnt = __pfn_to_pfn_t(pfn, PFN_DEV); + pfnt = __pfn_to_pfn_t(pfn, PFN_DEV | PFN_MAP); if (fault_page_size == (HPAGE_PMD_SIZE >> PAGE_SHIFT)) ret = vmf_insert_pfn_pmd_prot(vmf, pfnt, pgprot, write); #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD diff --git a/mm/gup.c b/mm/gup.c index e40579624f10..1b6a127f0bdd 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1993,6 +1993,17 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL +/* + * If we can't determine whether or not a pte is special, then fail immediately + * for ptes. Note, we can still pin HugeTLB as it is guaranteed not to be + * special. For THP, special huge entries are indicated by xxx_devmap() + * returning true, but a corresponding call to get_dev_pagemap() will + * return NULL. + * + * For a futex to be placed on a THP tail page, get_futex_key requires a + * get_user_pages_fast_only implementation that can pin pages. Thus it's still + * useful to have gup_huge_pmd even if we can't operate on ptes. + */ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { @@ -2069,16 +2080,6 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, return ret; } #else - -/* - * If we can't determine whether or not a pte is special, then fail immediately - * for ptes. Note, we can still pin HugeTLB and THP as these are guaranteed not - * to be special. - * - * For a futex to be placed on a THP tail page, get_futex_key requires a - * get_user_pages_fast_only implementation that can pin pages. Thus it's still - * useful to have gup_huge_pmd even if we can't operate on ptes. - */ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { diff --git a/mm/memremap.c b/mm/memremap.c index 7aa7d6e80ee5..757551cd2a4d 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -471,6 +471,11 @@ void vmem_altmap_free(struct vmem_altmap *altmap, unsigned long nr_pfns) * * If @pgmap is non-NULL and covers @pfn it will be returned as-is. If @pgmap * is non-NULL but does not cover @pfn the reference to it will be released. + * + * Return: A referenced pointer to a struct dev_pagemap containing @pfn, + * or NULL if there was no such pagemap registered. For interpretion + * of NULL returns for pfns extracted from valid huge page table entries, + * please see gup_pte_range(). */ struct dev_pagemap *get_dev_pagemap(unsigned long pfn, struct dev_pagemap *pgmap) From patchwork Sun Mar 21 18:45:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Thomas_Hellstr=C3=B6m_=28Intel=29?= X-Patchwork-Id: 12153241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B0D8C433E0 for ; Sun, 21 Mar 2021 18:46:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C19B061955 for ; Sun, 21 Mar 2021 18:46:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C19B061955 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shipmail.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E563C6B0072; Sun, 21 Mar 2021 14:46:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDF1D6B0070; Sun, 21 Mar 2021 14:46:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA3F36B0073; Sun, 21 Mar 2021 14:46:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0034.hostedemail.com [216.40.44.34]) by kanga.kvack.org (Postfix) with ESMTP id 974CE6B0070 for ; Sun, 21 Mar 2021 14:46:15 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 496D61DED for ; Sun, 21 Mar 2021 18:46:15 +0000 (UTC) X-FDA: 77944761510.03.2D830E9 Received: from pio-pvt-msa1.bahnhof.se (pio-pvt-msa1.bahnhof.se [79.136.2.40]) by imf26.hostedemail.com (Postfix) with ESMTP id 87F45407F8F6 for ; Sun, 21 Mar 2021 18:46:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by pio-pvt-msa1.bahnhof.se (Postfix) with ESMTP id 470BD40762; Sun, 21 Mar 2021 19:46:12 +0100 (CET) Authentication-Results: pio-pvt-msa1.bahnhof.se; dkim=pass (1024-bit key; unprotected) header.d=shipmail.org header.i=@shipmail.org header.b="QgstfKZg"; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at bahnhof.se Received: from pio-pvt-msa1.bahnhof.se ([127.0.0.1]) by localhost (pio-pvt-msa1.bahnhof.se [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Zz_OBa3Q9uRF; Sun, 21 Mar 2021 19:46:10 +0100 (CET) Received: by pio-pvt-msa1.bahnhof.se (Postfix) with ESMTPA id C1E1F40391; Sun, 21 Mar 2021 19:46:09 +0100 (CET) Received: from localhost.localdomain (2.70.38.73.mobile.tre.se [2.70.38.73]) by mail1.shipmail.org (Postfix) with ESMTPSA id 08C20361E92; Sun, 21 Mar 2021 19:46:08 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=shipmail.org; s=mail; t=1616352369; bh=ZnblYh+CL4ivnDI/E6ELj+Eqjfd3BvnVp2d0Kcr13LY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QgstfKZge72BS3EhP/IMO2NO6UDPjjWw3EOjZQCCAMAZbOhSTAWCUxhA0NuKRdbAb O17orFTuvm1/2qUmi4x93pHCMDBI9J8h4+MyhcJS2AMr2K+PHTWIkcqJyU4cmkLUCi R2rjarNFky6QOoSe+ndSkv2nZu4fJR5yy1qRp8gY= From: =?utf-8?q?Thomas_Hellstr=C3=B6m_=28Intel=29?= To: dri-devel@lists.freedesktop.org Cc: =?utf-8?q?Thomas_Hellstr=C3=B6m_=28Intel=29?= , Christian Koenig , David Airlie , Daniel Vetter , Andrew Morton , Jason Gunthorpe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 2/2] mm,drm/ttm: Use VM_PFNMAP for TTM vmas Date: Sun, 21 Mar 2021 19:45:29 +0100 Message-Id: <20210321184529.59006-3-thomas_os@shipmail.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210321184529.59006-1-thomas_os@shipmail.org> References: <20210321184529.59006-1-thomas_os@shipmail.org> MIME-Version: 1.0 X-Stat-Signature: bxs1ppfz9eof36ycf3eytixfdo34n38u X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 87F45407F8F6 Received-SPF: none (shipmail.org>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=pio-pvt-msa1.bahnhof.se; client-ip=79.136.2.40 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616352373-955593 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To block fast gup we need to make sure TTM ptes are always special. With MIXEDMAP we, on architectures that don't support pte_special, insert normal ptes, but OTOH on those architectures, fast is not supported. At the same time, the function documentation to vm_normal_page() suggests that ptes pointing to system memory pages of MIXEDMAP vmas are always normal, but that doesn't seem consistent with what's implemented in vmf_insert_mixed(). I'm thus not entirely sure this patch is actually needed. But to make sure and to avoid also normal (non-fast) gup, make all TTM vmas PFNMAP. With PFNMAP we can't allow COW mappings anymore so make is_cow_mapping() available and use it to reject COW mappigs at mmap time. There was previously a comment in the code that WC mappings together with x86 PAT + PFNMAP was bad for performance. However from looking at vmf_insert_mixed() it looks like in the current code PFNMAP and MIXEDMAP are handled the same for architectures that support pte_special. This means there should not be a performance difference anymore, but this needs to be verified. Cc: Christian Koenig Cc: David Airlie Cc: Daniel Vetter Cc: Andrew Morton Cc: Jason Gunthorpe Cc: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Thomas Hellström (Intel) Reviewed-by: Daniel Vetter --- drivers/gpu/drm/ttm/ttm_bo_vm.c | 22 ++++++++-------------- include/linux/mm.h | 5 +++++ mm/internal.h | 5 ----- 3 files changed, 13 insertions(+), 19 deletions(-) diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 1c34983480e5..708c6fb9be81 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -372,12 +372,7 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf, * at arbitrary times while the data is mmap'ed. * See vmf_insert_mixed_prot() for a discussion. */ - if (vma->vm_flags & VM_MIXEDMAP) - ret = vmf_insert_mixed_prot(vma, address, - __pfn_to_pfn_t(pfn, PFN_DEV), - prot); - else - ret = vmf_insert_pfn_prot(vma, address, pfn, prot); + ret = vmf_insert_pfn_prot(vma, address, pfn, prot); /* Never error on prefaulted PTEs */ if (unlikely((ret & VM_FAULT_ERROR))) { @@ -555,18 +550,14 @@ static void ttm_bo_mmap_vma_setup(struct ttm_buffer_object *bo, struct vm_area_s * Note: We're transferring the bo reference to * vma->vm_private_data here. */ - vma->vm_private_data = bo; /* - * We'd like to use VM_PFNMAP on shared mappings, where - * (vma->vm_flags & VM_SHARED) != 0, for performance reasons, - * but for some reason VM_PFNMAP + x86 PAT + write-combine is very - * bad for performance. Until that has been sorted out, use - * VM_MIXEDMAP on all mappings. See freedesktop.org bug #75719 + * PFNMAP forces us to block COW mappings in mmap(), + * and with MIXEDMAP we would incorrectly allow fast gup + * on TTM memory on architectures that don't have pte_special. */ - vma->vm_flags |= VM_MIXEDMAP; - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + vma->vm_flags |= VM_PFNMAP | VM_IO | VM_DONTEXPAND | VM_DONTDUMP; } int ttm_bo_mmap(struct file *filp, struct vm_area_struct *vma, @@ -579,6 +570,9 @@ int ttm_bo_mmap(struct file *filp, struct vm_area_struct *vma, if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET_START)) return -EINVAL; + if (unlikely(is_cow_mapping(vma->vm_flags))) + return -EINVAL; + bo = ttm_bo_vm_lookup(bdev, vma->vm_pgoff, vma_pages(vma)); if (unlikely(!bo)) return -EINVAL; diff --git a/include/linux/mm.h b/include/linux/mm.h index 77e64e3eac80..c6ebf7f9ddbb 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -686,6 +686,11 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma) return vma->vm_flags & VM_ACCESS_FLAGS; } +static inline bool is_cow_mapping(vm_flags_t flags) +{ + return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE; +} + #ifdef CONFIG_SHMEM /* * The vma_is_shmem is not inline because it is used only by slow diff --git a/mm/internal.h b/mm/internal.h index 9902648f2206..1432feec62df 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -296,11 +296,6 @@ static inline unsigned int buddy_order(struct page *page) */ #define buddy_order_unsafe(page) READ_ONCE(page_private(page)) -static inline bool is_cow_mapping(vm_flags_t flags) -{ - return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE; -} - /* * These three helpers classifies VMAs for virtual memory accounting. */