From patchwork Wed Jan 8 20:24:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 11324645 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7757B1398 for ; Wed, 8 Jan 2020 20:27:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40D7F20643 for ; Wed, 8 Jan 2020 20:27:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40D7F20643 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E92DD8E000B; Wed, 8 Jan 2020 15:27:11 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B7BF78E0012; Wed, 8 Jan 2020 15:27:11 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 904748E0011; Wed, 8 Jan 2020 15:27:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0207.hostedemail.com [216.40.44.207]) by kanga.kvack.org (Postfix) with ESMTP id 4F48F8E000A for ; Wed, 8 Jan 2020 15:27:11 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id F2C4A180AD80F for ; Wed, 8 Jan 2020 20:27:10 +0000 (UTC) X-FDA: 76355601420.15.dolls94_218501121f454 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,sean.j.christopherson@intel.com,:pbonzini@redhat.com:paulus@ozlabs.org:sean.j.christopherson@intel.com:vkuznets@redhat.com:wanpengli@tencent.com:jmattson@google.com:joro@8bytes.org:dave.hansen@linux.intel.com:luto@kernel.org:peterz@infradead.org:akpm@linux-foundation.org:maz@kernel.org:james.morse@arm.com:julien.thierry.kdev@gmail.com:suzuki.poulose@arm.com:kvm-ppc@vger.kernel.org:kvm@vger.kernel.org:linux-kernel@vger.kernel.org::linux-arm-kernel@lists.infradead.org:kvmarm@lists.cs.columbia.edu:syzbot+c9d1fb51ac9d0d10c39d@syzkaller.appspotmail.com:aarcange@redhat.com:dan.j.williams@intel.com:brho@google.com:david@redhat.com:jason.zeng@intel.com:dave.jiang@intel.com:liran.alon@oracle.com:linux-nvdimm@lists.01.org,RULES_HIT:30012:30054:30064:30070,0,RBL:192.55.52.93:@intel.com:.lbl8.mailshell.net-62.50.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0 :0,LFtim X-HE-Tag: dolls94_218501121f454 X-Filterd-Recvd-Size: 7955 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Wed, 8 Jan 2020 20:27:10 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Jan 2020 12:27:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,411,1571727600"; d="scan'208";a="211658395" Received: from sjchrist-coffee.jf.intel.com ([10.54.74.202]) by orsmga007.jf.intel.com with ESMTP; 08 Jan 2020 12:27:07 -0800 From: Sean Christopherson To: Paolo Bonzini Cc: Paul Mackerras , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , kvm-ppc@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, syzbot+c9d1fb51ac9d0d10c39d@syzkaller.appspotmail.com, Andrea Arcangeli , Dan Williams , Barret Rhoden , David Hildenbrand , Jason Zeng , Dave Jiang , Liran Alon , linux-nvdimm Subject: [PATCH 12/14] KVM: x86/mmu: Fold max_mapping_level() into kvm_mmu_hugepage_adjust() Date: Wed, 8 Jan 2020 12:24:46 -0800 Message-Id: <20200108202448.9669-13-sean.j.christopherson@intel.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200108202448.9669-1-sean.j.christopherson@intel.com> References: <20200108202448.9669-1-sean.j.christopherson@intel.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fold max_mapping_level() into kvm_mmu_hugepage_adjust() now that HugeTLB mappings are handled in kvm_mmu_hugepage_adjust(), i.e. there isn't a need to pre-calculate the max mapping level. Co-locating all hugepage checks eliminates a memslot lookup, at the cost of performing the __mmu_gfn_lpage_is_disallowed() checks while holding mmu_lock. The latency of lpage_is_disallowed() is likely negligible relative to the rest of the code run while holding mmu_lock, and can be offset to some extent by eliminating the mmu_gfn_lpage_is_disallowed() check in set_spte() in a future patch. Eliminating the check in set_spte() is made possible by performing the initial lpage_is_disallowed() checks while holding mmu_lock. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 60 ++++++++++++++-------------------- arch/x86/kvm/mmu/paging_tmpl.h | 2 -- 2 files changed, 24 insertions(+), 38 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f93b0c5e4170..f2667fe0dc75 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1310,27 +1310,6 @@ gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu, gfn_t gfn, return slot; } -static int max_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn, - int max_level) -{ - struct kvm_memory_slot *slot; - - if (unlikely(max_level == PT_PAGE_TABLE_LEVEL)) - return PT_PAGE_TABLE_LEVEL; - - slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); - if (!memslot_valid_for_gpte(slot, true)) - return PT_PAGE_TABLE_LEVEL; - - max_level = min(max_level, kvm_x86_ops->get_lpage_level()); - for ( ; max_level > PT_PAGE_TABLE_LEVEL; max_level--) { - if (!__mmu_gfn_lpage_is_disallowed(gfn, max_level, slot)) - break; - } - - return max_level; -} - /* * About rmap_head encoding: * @@ -3101,10 +3080,11 @@ static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep, if (pte_access & ACC_WRITE_MASK) { /* - * Other vcpu creates new sp in the window between - * max_mapping_level() and acquiring mmu-lock. We can - * allow guest to retry the access, the mapping can - * be fixed if guest refault. + * Legacy code to handle an obsolete scenario where a different + * vcpu creates new sp in the window between this vcpu's query + * of lpage_is_disallowed() and acquiring mmu_lock. No longer + * necessary now that lpage_is_disallowed() is called after + * acquiring mmu_lock. */ if (level > PT_PAGE_TABLE_LEVEL && mmu_gfn_lpage_is_disallowed(vcpu, gfn, level)) @@ -3295,9 +3275,8 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) } static int host_pfn_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn, - kvm_pfn_t pfn) + kvm_pfn_t pfn, struct kvm_memory_slot *slot) { - struct kvm_memory_slot *slot; unsigned long hva; pte_t *pte; int level; @@ -3310,16 +3289,13 @@ static int host_pfn_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn, return PT_PAGE_TABLE_LEVEL; /* - * Manually do the equivalent of kvm_vcpu_gfn_to_hva() to avoid the + * Note, using the already-retrieved memslot and __gfn_to_hva_memslot() + * is not solely for performance, it's also necessary to avoid the * "writable" check in __gfn_to_hva_many(), which will always fail on * read-only memslots due to gfn_to_hva() assuming writes. Earlier * page fault steps have already verified the guest isn't writing a * read-only memslot. */ - slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); - if (!memslot_valid_for_gpte(slot, true)) - return PT_PAGE_TABLE_LEVEL; - hva = __gfn_to_hva_memslot(slot, gfn); pte = lookup_address_in_mm(vcpu->kvm->mm, hva, &level); @@ -3332,18 +3308,32 @@ static int host_pfn_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn, static int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn, int max_level, kvm_pfn_t *pfnp) { + struct kvm_memory_slot *slot; kvm_pfn_t pfn = *pfnp; kvm_pfn_t mask; int level; - if (max_level == PT_PAGE_TABLE_LEVEL) + if (unlikely(max_level == PT_PAGE_TABLE_LEVEL)) return PT_PAGE_TABLE_LEVEL; if (is_error_noslot_pfn(pfn) || kvm_is_reserved_pfn(pfn) || kvm_is_zone_device_pfn(pfn)) return PT_PAGE_TABLE_LEVEL; - level = host_pfn_mapping_level(vcpu, gfn, pfn); + slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); + if (!memslot_valid_for_gpte(slot, true)) + return PT_PAGE_TABLE_LEVEL; + + max_level = min(max_level, kvm_x86_ops->get_lpage_level()); + for ( ; max_level > PT_PAGE_TABLE_LEVEL; max_level--) { + if (!__mmu_gfn_lpage_is_disallowed(gfn, max_level, slot)) + break; + } + + if (max_level == PT_PAGE_TABLE_LEVEL) + return PT_PAGE_TABLE_LEVEL; + + level = host_pfn_mapping_level(vcpu, gfn, pfn, slot); if (level == PT_PAGE_TABLE_LEVEL) return level; @@ -4182,8 +4172,6 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, if (lpage_disallowed) max_level = PT_PAGE_TABLE_LEVEL; - max_level = max_mapping_level(vcpu, gfn, max_level); - if (fast_page_fault(vcpu, gpa, error_code)) return RET_PF_RETRY; diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 0560982eda8b..ea174d85700a 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -817,8 +817,6 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gpa_t addr, u32 error_code, else max_level = walker.level; - max_level = max_mapping_level(vcpu, walker.gfn, max_level); - mmu_seq = vcpu->kvm->mmu_notifier_seq; smp_rmb();