From patchwork Thu Dec 12 18:22:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barret Rhoden X-Patchwork-Id: 11289013 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74F2C112B for ; Thu, 12 Dec 2019 18:23:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 538AC2253D for ; Thu, 12 Dec 2019 18:23:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MAdkof8C" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730375AbfLLSW6 (ORCPT ); Thu, 12 Dec 2019 13:22:58 -0500 Received: from mail-pj1-f73.google.com ([209.85.216.73]:55779 "EHLO mail-pj1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730357AbfLLSW6 (ORCPT ); Thu, 12 Dec 2019 13:22:58 -0500 Received: by mail-pj1-f73.google.com with SMTP id e7so1569542pjt.22 for ; Thu, 12 Dec 2019 10:22:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=IvHeRx1oBCh9J6TLvVWdv2Gb1z0rpMzGcWyY+BXVgxo=; b=MAdkof8C2KW30AALfnNJO3vr9VoF0vmBXanMXL/vR51qGyvIZkGihOmBNe+C0or/8I NpO/KCC3oEKNcwp7NIyw7WnD6H+rCsDvTI30S8NWdXPxMl0MDaFljsm9JXiaxGx8tZCO d4KLSyqgiZeCGiJqkepl0hPb47CUKa9ToIY3jibKjBu4kZUmg7uhITO9RyP7k+b0KyoU zEcEs3lkMbzHsbNJ2xA4Mv3lS6SzC0mJbsHZ2qOxVL621b4e0Gt/qvA+mGVTzX9rJKB5 /8rQMuMbnVTaGER/Xvhr4PwqP6nQELiDlJ6qv20w9iVMpmIsQeJffxIkfXKhEXwrX/ah Gvjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=IvHeRx1oBCh9J6TLvVWdv2Gb1z0rpMzGcWyY+BXVgxo=; b=CRlcwjzyuxnYTKBv8V5C5V/Dm50ScEglei+cW1LSsvvpDu4602Xuj9V+JiyEnMRMCT x74OQK20tsd0iEpMiIpZxkL4ki1V3ae71+COeXav9XzLKR49XwpdDHrVbzmy+C9YsrjL +rjGMEd7J5nU+mSmLIV2E3qH384Gt1E3V2uYI4bWoo32lwWk/wyvSB0/OHTbcgTkmfvr rpDfiXG/Aii4sUp4UGTrknCRH5vzmIOi1LpGMm1080Fjo6KHuEb3yaJdkBCTu+uKfhHe XWrbfEGpeGtM9VVNjQ37M++gq7YYEL/RmCVsBBjzyb91VayI99vLARRWu4vc68L3SMV8 rvqw== X-Gm-Message-State: APjAAAWluXfFFUI2C8d8d0imH+XlZ5IEzxjECwTds0aJ/Yzqudgvw35R v6cVWmDkcHkMq+IdPpBR7G4b6orY X-Google-Smtp-Source: APXvYqwVXoFFdqVc4oFKrnkNNIY1YlbI3lRBL8mXjnVcsOoBTjQ2NnV0mup2jeDoJ+6bSYWUQLoZOkJ/ X-Received: by 2002:a63:a508:: with SMTP id n8mr11683668pgf.278.1576174977297; Thu, 12 Dec 2019 10:22:57 -0800 (PST) Date: Thu, 12 Dec 2019 13:22:37 -0500 In-Reply-To: <20191212182238.46535-1-brho@google.com> Message-Id: <20191212182238.46535-2-brho@google.com> Mime-Version: 1.0 References: <20191212182238.46535-1-brho@google.com> X-Mailer: git-send-email 2.24.1.735.g03f4e72817-goog Subject: [PATCH v5 1/2] mm: make dev_pagemap_mapping_shift() externally visible From: Barret Rhoden To: Paolo Bonzini , Dan Williams , David Hildenbrand , Dave Jiang , Alexander Duyck , Sean Christopherson Cc: linux-nvdimm@lists.01.org, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, jason.zeng@intel.com Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org KVM has a use case for determining the size of a dax mapping. The KVM code has easy access to the address and the mm, and dev_pagemap_mapping_shift() needs only those parameters. It was deriving them from page and vma. This commit changes those parameters from (page, vma) to (address, mm). Signed-off-by: Barret Rhoden Reviewed-by: David Hildenbrand Acked-by: Dan Williams --- include/linux/mm.h | 3 +++ mm/memory-failure.c | 38 +++----------------------------------- mm/util.c | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 35 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a2adf95b3f9c..bfd1882dd5c6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1013,6 +1013,9 @@ static inline bool is_pci_p2pdma_page(const struct page *page) #define page_ref_zero_or_close_to_overflow(page) \ ((unsigned int) page_ref_count(page) + 127u <= 127u) +unsigned long dev_pagemap_mapping_shift(unsigned long address, + struct mm_struct *mm); + static inline void get_page(struct page *page) { page = compound_head(page); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 3151c87dff73..bafa464c8290 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -261,40 +261,6 @@ void shake_page(struct page *p, int access) } EXPORT_SYMBOL_GPL(shake_page); -static unsigned long dev_pagemap_mapping_shift(struct page *page, - struct vm_area_struct *vma) -{ - unsigned long address = vma_address(page, vma); - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - - pgd = pgd_offset(vma->vm_mm, address); - if (!pgd_present(*pgd)) - return 0; - p4d = p4d_offset(pgd, address); - if (!p4d_present(*p4d)) - return 0; - pud = pud_offset(p4d, address); - if (!pud_present(*pud)) - return 0; - if (pud_devmap(*pud)) - return PUD_SHIFT; - pmd = pmd_offset(pud, address); - if (!pmd_present(*pmd)) - return 0; - if (pmd_devmap(*pmd)) - return PMD_SHIFT; - pte = pte_offset_map(pmd, address); - if (!pte_present(*pte)) - return 0; - if (pte_devmap(*pte)) - return PAGE_SHIFT; - return 0; -} - /* * Failure handling: if we can't find or can't kill a process there's * not much we can do. We just print a message and ignore otherwise. @@ -324,7 +290,9 @@ static void add_to_kill(struct task_struct *tsk, struct page *p, } tk->addr = page_address_in_vma(p, vma); if (is_zone_device_page(p)) - tk->size_shift = dev_pagemap_mapping_shift(p, vma); + tk->size_shift = + dev_pagemap_mapping_shift(vma_address(page, vma), + vma->vm_mm); else tk->size_shift = compound_order(compound_head(p)) + PAGE_SHIFT; diff --git a/mm/util.c b/mm/util.c index 3ad6db9a722e..59984e6b40ab 100644 --- a/mm/util.c +++ b/mm/util.c @@ -901,3 +901,37 @@ int memcmp_pages(struct page *page1, struct page *page2) kunmap_atomic(addr1); return ret; } + +unsigned long dev_pagemap_mapping_shift(unsigned long address, + struct mm_struct *mm) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pgd = pgd_offset(mm, address); + if (!pgd_present(*pgd)) + return 0; + p4d = p4d_offset(pgd, address); + if (!p4d_present(*p4d)) + return 0; + pud = pud_offset(p4d, address); + if (!pud_present(*pud)) + return 0; + if (pud_devmap(*pud)) + return PUD_SHIFT; + pmd = pmd_offset(pud, address); + if (!pmd_present(*pmd)) + return 0; + if (pmd_devmap(*pmd)) + return PMD_SHIFT; + pte = pte_offset_map(pmd, address); + if (!pte_present(*pte)) + return 0; + if (pte_devmap(*pte)) + return PAGE_SHIFT; + return 0; +} +EXPORT_SYMBOL_GPL(dev_pagemap_mapping_shift); From patchwork Thu Dec 12 18:22:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barret Rhoden X-Patchwork-Id: 11289011 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4317E14B7 for ; Thu, 12 Dec 2019 18:23:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 20FE922527 for ; Thu, 12 Dec 2019 18:23:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ba0ZrxLu" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730168AbfLLSXC (ORCPT ); Thu, 12 Dec 2019 13:23:02 -0500 Received: from mail-pj1-f74.google.com ([209.85.216.74]:43686 "EHLO mail-pj1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730210AbfLLSXB (ORCPT ); Thu, 12 Dec 2019 13:23:01 -0500 Received: by mail-pj1-f74.google.com with SMTP id b23so1581356pjz.10 for ; Thu, 12 Dec 2019 10:23:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=QJzS5cf1va+nrqwiPMy+1/CZ1shjgAqUeBFnOuHAc2s=; b=Ba0ZrxLur6jY5/yOkxJ3TztRuqj4f0wSiJ88PrOZrz6TKvyfeg1L/B0CGclVQMSKMS fKntKcN+MxF41ny/fjJJQzA/TxZnRfVwqE4Abw4zWB1eEXysAMU83K8rIwXbt5IfGWXL rTV7A36Z09p7odDbSD00a4mbTY0Lvu0IyNj4tT49mjw31Q5L9IhFdG8cXyX4r40FjVEo hjq3LR16hnJ3NKy94C2uwvo9okl0CTJT0ZPXdKG5OjVbxKPM8y7IX0CWAy14neVt1hO7 4Os1UMSWnkdTVnbXe37U4lvvoG1ujSbhU8yfBrGa1meSA02y74bsiSjr7iBssE0ojlWA XmTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=QJzS5cf1va+nrqwiPMy+1/CZ1shjgAqUeBFnOuHAc2s=; b=Z3dUoHwY35ohFoQ6bbxPx6sZZ6yPOna6lAKc8GmW3ZPd2uWx3UkeThRBHx4p6Tp8DP kd8CC3VH2qt+/HloW0TBaqc1pW5EANYS+br3v3Vo56qIMmkrSbgrt+tB13kGOv4WafuQ tytvOOVV7HT6Q51rT4uLSOQE6rsvnOTkTxHOyy6JHpGI4xXnJAw5izgmggghTAl+46mH 2O4DryzwMbO67bPP/38kswas0bfAKTYeDDGudsWNH0FfbCvEdDQPfrH1uT5yPnR5+uHz 7VLkH0/b6Aqw+n0dvjcyZWBdN8496CY6BukaXsComI5SJrQTaA5Zm5hr6namcy/8Wmm2 0wZg== X-Gm-Message-State: APjAAAVvuSzsPJKbHwLz1aVLvWiWMnWAP4CdQERVjMbjrujWATeJ8tAo V+QqfTRtz2qU3VbAZr7An/w1jQj8 X-Google-Smtp-Source: APXvYqx3OSYyAEpeU1hpMzeLCwi0nWCUbjiyweJI73Oot2AiHdC93XItct57lFn8p5M9ukNOGXwyBoTV X-Received: by 2002:a65:5a4d:: with SMTP id z13mr11986210pgs.21.1576174981140; Thu, 12 Dec 2019 10:23:01 -0800 (PST) Date: Thu, 12 Dec 2019 13:22:38 -0500 In-Reply-To: <20191212182238.46535-1-brho@google.com> Message-Id: <20191212182238.46535-3-brho@google.com> Mime-Version: 1.0 References: <20191212182238.46535-1-brho@google.com> X-Mailer: git-send-email 2.24.1.735.g03f4e72817-goog Subject: [PATCH v5 2/2] kvm: Use huge pages for DAX-backed files From: Barret Rhoden To: Paolo Bonzini , Dan Williams , David Hildenbrand , Dave Jiang , Alexander Duyck , Sean Christopherson Cc: linux-nvdimm@lists.01.org, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, jason.zeng@intel.com Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This change allows KVM to map DAX-backed files made of huge pages with huge mappings in the EPT/TDP. DAX pages are not PageTransCompound. The existing check is trying to determine if the mapping for the pfn is a huge mapping or not. For non-DAX maps, e.g. hugetlbfs, that means checking PageTransCompound. For DAX, we can check the page table itself. Note that KVM already faulted in the page (or huge page) in the host's page table, and we hold the KVM mmu spinlock. We grabbed that lock in kvm_mmu_notifier_invalidate_range_end, before checking the mmu seq. Signed-off-by: Barret Rhoden --- arch/x86/kvm/mmu/mmu.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 7269130ea5e2..ea8f6951398b 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3328,6 +3328,30 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) __direct_pte_prefetch(vcpu, sp, sptep); } +static bool pfn_is_huge_mapped(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn) +{ + struct page *page = pfn_to_page(pfn); + unsigned long hva; + + if (!is_zone_device_page(page)) + return PageTransCompoundMap(page); + + /* + * DAX pages do not use compound pages. The page should have already + * been mapped into the host-side page table during try_async_pf(), so + * we can check the page tables directly. + */ + hva = gfn_to_hva(kvm, gfn); + if (kvm_is_error_hva(hva)) + return false; + + /* + * Our caller grabbed the KVM mmu_lock with a successful + * mmu_notifier_retry, so we're safe to walk the page table. + */ + return dev_pagemap_mapping_shift(hva, current->mm) > PAGE_SHIFT; +} + static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t *pfnp, int *levelp) @@ -3342,8 +3366,8 @@ static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu, * here. */ if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) && - !kvm_is_zone_device_pfn(pfn) && level == PT_PAGE_TABLE_LEVEL && - PageTransCompoundMap(pfn_to_page(pfn))) { + level == PT_PAGE_TABLE_LEVEL && + pfn_is_huge_mapped(vcpu->kvm, gfn, pfn)) { unsigned long mask; /* @@ -5957,8 +5981,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, * mapping if the indirect sp has level = 1. */ if (sp->role.direct && !kvm_is_reserved_pfn(pfn) && - !kvm_is_zone_device_pfn(pfn) && - PageTransCompoundMap(pfn_to_page(pfn))) { + pfn_is_huge_mapped(kvm, sp->gfn, pfn)) { pte_list_remove(rmap_head, sptep); if (kvm_available_flush_tlb_with_range())