From patchwork Thu Oct 10 18:23:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13830906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30280CFC5F9 for ; Thu, 10 Oct 2024 19:13:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Reply-To:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Content-Transfer-Encoding:Content-Type:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bLMllFEXz5NbCB79T5EDeXxXCOdT0E/lPN9LlBgR2FM=; b=d/bjsRn5vX4P6C 1UvOkTJhTGHRYaVHLsUAL1X1Y448+0ftlgpnaKXeJDpi2C9lkvBArXkf3Kd3qd7H/Yi7TlCdGCiTW KahyqMegSu0jjhMZy6puQsvZeHYAgnVV8g6ITH7hJIj5MzuP/TTUdYcufWgnIdAS7Z53NVEgTJFaH eMVtWBmLrpwmxJgMkXf6CkI53OUq4hW26CUJ967LJ2RytybWBSbTX5NBJfFPTmc7De3vE594uJloN DENljuuYNnwS3c38qAkDz5Hc4De3/ynI/yeLVXo7tOVQq9Nw1NoAScrmf+1CQ0rw0TKno9yb5Mbb+ gQiiQIlA6Sl0pQq9dQow==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1syyb4-0000000E4RQ-2Vqw; Thu, 10 Oct 2024 19:13:14 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1syxra-0000000Dqfc-3Xul for linux-arm-kernel@lists.infradead.org; Thu, 10 Oct 2024 18:26:16 +0000 Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-20c61e4b890so14347665ad.1 for ; Thu, 10 Oct 2024 11:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728584774; x=1729189574; darn=lists.infradead.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:from:to:cc:subject:date :message-id:reply-to; bh=bLMllFEXz5NbCB79T5EDeXxXCOdT0E/lPN9LlBgR2FM=; b=iqLFJk08lU4sXCqW4q3PDDrFFs5ZbTmdrSdsuuvf3BYW0ezc3qn0B+PT1xdp95wJKP v8vCmQJzR9uGRWWX3xM5EFUHYKi2EyUNEbk9tyfb71n60zeMppgYPZ/Vj4gdSD5N6RvJ FkgKdr3Z6z8/NYvrrP30gumqXo23lNuT93fQVxGcHiqCRnlW4E/mnK7E1UGmNlyGu+b8 Y5W97QXCMKKOXRoadOfj7pRQ5An6esrBRxEqAAv4FVktbDLfZXuA1uVEw7KCqLLpPIOx JbdwK5ygGHHjecSj53zztRzMuX8uHhsU/M+w90oMCOX27vE9+USr78CsT5jhtEz3h4c7 2qkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728584774; x=1729189574; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=bLMllFEXz5NbCB79T5EDeXxXCOdT0E/lPN9LlBgR2FM=; b=pOIJ2HSQASoW8Dc408VZLp5r8REvXvdyQyb7MT9ZevRtBAy69n3vi3KXWb8ehbsbLs qjRWI3CZXUzGMGO+/p96Tv/uGRoeNGj7fOZxibixyv4hP/tgpY6BnifQ/myCNV4sjFoo 4uXHn45BzeadOtKSJkn657Hx1Oi7OsdB00Aw5HebZZ1c2QyS0BA+kSkD6FxIaS616HHy fNQAnLlhLgTGa8b6j8v5+aNhGBSKQ5AB92pSdFZ6Cz5b6aXKLLuq0vuCGr+jPxjX9JGc vpWfU3iQM8Hz5dmoGlu+cHuxIQSuD+79jduPZnQBM2lNf0SCmFDLvO5cFdWTJCR9sZh5 jTUw== X-Forwarded-Encrypted: i=1; AJvYcCVkERDGP9v87mglppXZc6Wg9ljYVyvV93xZyN/BPYq4NTWdmJ89QRWS9kIQT5V6XG7tF8t6Bs5IKVphfR2ALYKA@lists.infradead.org X-Gm-Message-State: AOJu0YzE3QZbMqB+otAJAiJ3pbcyXPx/QRwp5orhTQmYKuIeJVk2ILDG YeQg0UA+2Rwypikf9BD6MUiykubtcuZlYVg8+t75nJ1sNBMiNIsAdPgwv7pDEwJnvk5FG1+566E OFg== X-Google-Smtp-Source: AGHT+IHTeV3dFyfdACUyzit3PmgjtoD4THwZ77SaAB8Cv7KtK6sEvfc4IjBdPT+nDHGdhMcMXcKC7R1qaBw= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a17:903:1c8:b0:20c:716c:5af with SMTP id d9443c01a7336-20c716c078dmr946605ad.3.1728584773496; Thu, 10 Oct 2024 11:26:13 -0700 (PDT) Date: Thu, 10 Oct 2024 11:23:41 -0700 In-Reply-To: <20241010182427.1434605-1-seanjc@google.com> Mime-Version: 1.0 References: <20241010182427.1434605-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241010182427.1434605-40-seanjc@google.com> Subject: [PATCH v13 39/85] KVM: x86/mmu: Add common helper to handle prefetching SPTEs From: Sean Christopherson To: Paolo Bonzini , Marc Zyngier , Oliver Upton , Tianrui Zhao , Bibo Mao , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Sean Christopherson Cc: kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, " =?utf-8?q?Alex_Benn=C3=A9e?= " , Yan Zhao , David Matlack , David Stevens , Andrew Jones X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241010_112614_917732_7126D7F5 X-CRM114-Status: GOOD ( 15.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Sean Christopherson Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Deduplicate the prefetching code for indirect and direct MMUs. The core logic is the same, the only difference is that indirect MMUs need to prefetch SPTEs one-at-a-time, as contiguous guest virtual addresses aren't guaranteed to yield contiguous guest physical addresses. Tested-by: Alex Bennée Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 40 +++++++++++++++++++++------------- arch/x86/kvm/mmu/paging_tmpl.h | 13 +---------- 2 files changed, 26 insertions(+), 27 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 62924f95a398..65d3a602eb2c 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2943,32 +2943,41 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, return ret; } -static int direct_pte_prefetch_many(struct kvm_vcpu *vcpu, - struct kvm_mmu_page *sp, - u64 *start, u64 *end) +static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep, + int nr_pages, unsigned int access) { struct page *pages[PTE_PREFETCH_NUM]; struct kvm_memory_slot *slot; - unsigned int access = sp->role.access; - int i, ret; - gfn_t gfn; + int i; + + if (WARN_ON_ONCE(nr_pages > PTE_PREFETCH_NUM)) + return false; - gfn = kvm_mmu_page_get_gfn(sp, spte_index(start)); slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, access & ACC_WRITE_MASK); if (!slot) - return -1; + return false; - ret = kvm_prefetch_pages(slot, gfn, pages, end - start); - if (ret <= 0) - return -1; + nr_pages = kvm_prefetch_pages(slot, gfn, pages, nr_pages); + if (nr_pages <= 0) + return false; - for (i = 0; i < ret; i++, gfn++, start++) { - mmu_set_spte(vcpu, slot, start, access, gfn, + for (i = 0; i < nr_pages; i++, gfn++, sptep++) { + mmu_set_spte(vcpu, slot, sptep, access, gfn, page_to_pfn(pages[i]), NULL); kvm_release_page_clean(pages[i]); } - return 0; + return true; +} + +static bool direct_pte_prefetch_many(struct kvm_vcpu *vcpu, + struct kvm_mmu_page *sp, + u64 *start, u64 *end) +{ + gfn_t gfn = kvm_mmu_page_get_gfn(sp, spte_index(start)); + unsigned int access = sp->role.access; + + return kvm_mmu_prefetch_sptes(vcpu, gfn, start, end - start, access); } static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, @@ -2986,8 +2995,9 @@ static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, if (is_shadow_present_pte(*spte) || spte == sptep) { if (!start) continue; - if (direct_pte_prefetch_many(vcpu, sp, start, spte) < 0) + if (!direct_pte_prefetch_many(vcpu, sp, start, spte)) return; + start = NULL; } else if (!start) start = spte; diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 9bd3d6f5db91..a476a5428017 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -533,9 +533,7 @@ static bool FNAME(prefetch_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, u64 *spte, pt_element_t gpte) { - struct kvm_memory_slot *slot; unsigned pte_access; - struct page *page; gfn_t gfn; if (FNAME(prefetch_invalid_gpte)(vcpu, sp, spte, gpte)) @@ -545,16 +543,7 @@ FNAME(prefetch_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, pte_access = sp->role.access & FNAME(gpte_access)(gpte); FNAME(protect_clean_gpte)(vcpu->arch.mmu, &pte_access, gpte); - slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, pte_access & ACC_WRITE_MASK); - if (!slot) - return false; - - if (kvm_prefetch_pages(slot, gfn, &page, 1) != 1) - return false; - - mmu_set_spte(vcpu, slot, spte, pte_access, gfn, page_to_pfn(page), NULL); - kvm_release_page_clean(page); - return true; + return kvm_mmu_prefetch_sptes(vcpu, gfn, spte, 1, pte_access); } static bool FNAME(gpte_changed)(struct kvm_vcpu *vcpu,