From patchwork Wed May 25 23:09:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 12861821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 244E7C433EF for ; Wed, 25 May 2022 23:09:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345214AbiEYXJM (ORCPT ); Wed, 25 May 2022 19:09:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230474AbiEYXJL (ORCPT ); Wed, 25 May 2022 19:09:11 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0208D91546 for ; Wed, 25 May 2022 16:09:09 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id u8-20020a656708000000b003f70a43fe1eso7494107pgf.3 for ; Wed, 25 May 2022 16:09:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=YrjJHRFdS5VonxkVsAHBuBVkWew0ibSghSiVOFylo/0=; b=gFfiWRVGugXdj5kNECaC63se48jXOxl37x3pQVu/GVdoyQAVtuM/EK0DWrxIz5eUqb HBRcZZA/xAqyUbMfWJ13wlYg4NU2bBktbl2Oh4qqoJSR/gC6M6lpnAQhO84Z7ftzZVXF RjE2gdia/87jx/S7lxbRW3lGpFw4BL/jqQ6NmyogTRAZpLfx26cza5BdYkGwzDcfmBBv CWt2X3uKH3NZChj1746knGWBVJetjBjJ1hcqzxXAxRs3B0V00ZqLdeuaWczFVTmYf6vc 00RAmX0eOL3LR8LuFFB9GunMA6l/yWmQetsIsSxvOaKO0SxnO22FxhYc1GuihdOKPkof poqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=YrjJHRFdS5VonxkVsAHBuBVkWew0ibSghSiVOFylo/0=; b=Gtsly2ByCi5CW8t1cCgEEoUNUfFjdZETviwsH76Wbg9FfAIxQzSkPYE2MWMUFUj671 KAHNUsY0M/2fNfLJVPDZJB+j1mmGQL1qf1lKH1mpeW3S/M4r7skYEvtWDLf68Evfd4MX ReaYB6yx8fY/kKxjwhUcnWkxKL9nA7RWwLgVMkquHfkl2sNrITsj10icq735GWL5CgYg ocRyJSGSKgzgch5bUulz78V+0R5N/LHV4G/co/aVRqPEPGVaUmeZcYQbMt2oy0Q1z43l YMvWEQbth2OdGswVlh3Wju1rf+8P9E8oa/HY2GHZtAtRDw4/LSXRzxP73zWgS1gavDFV c6SQ== X-Gm-Message-State: AOAM533ELncES/anjCTRJHmwmx0aeOyRvVAwv6CGoZxcIySbjyYY7XmQ lkWPHCDgcXZsrEPImTTgAnb1qQiPWAMXQR/Z9ZX6Hz9RO6X4cz8Tz3qO1HZgJNCFkEXmacKyoZZ uBhEeNIyxevOSZdXlgXPkePPmafku8zE0TMLd4IbFNYJq6AVokmRzGiLCnYea X-Google-Smtp-Source: ABdhPJx37yu8A24ZMkoDUIBUi5tSrrE6OPryVyWVtHXumbOflxgrCjmDatYIZ+WQA1RdSDVx2Gn3EKpW5B/n X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:aa7:8081:0:b0:518:26c4:ea42 with SMTP id v1-20020aa78081000000b0051826c4ea42mr29589288pff.7.1653520149143; Wed, 25 May 2022 16:09:09 -0700 (PDT) Date: Wed, 25 May 2022 23:09:04 +0000 Message-Id: <20220525230904.1584480-1-bgardon@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.36.1.124.g0e6072fb45-goog Subject: [PATCH] KVM: x86/MMU: Zap non-leaf SPTEs when disabling dirty logging From: Ben Gardon To: kvm@vger.kernel.org, Paolo Bonzini Cc: linux-kernel@vger.kernel.org, Peter Xu , Sean Christopherson , David Matlack , Jim Mattson , David Dunn , Jing Zhang , Junaid Shahid , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When disabling dirty logging, zap non-leaf parent entries to allow replacement with huge pages instead of recursing and zapping all of the child, leaf entries. This reduces the number of TLB flushes required. Currently disabling dirty logging with the TDP MMU is extremely slow. On a 96 vCPU / 96G VM backed with gigabyte pages, it takes ~200 seconds to disable dirty logging with the TDP MMU, as opposed to ~4 seconds with the shadow MMU. This patch reduces the disable dirty log time with the TDP MMU to ~3 seconds. Testing: Ran KVM selftests and kvm-unit-tests on an Intel Haswell. This patch introduced no new failures. Signed-off-by: Ben Gardon Reviewed-by: David Matlack --- arch/x86/kvm/mmu/tdp_iter.c | 9 +++++++++ arch/x86/kvm/mmu/tdp_iter.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 38 +++++++++++++++++++++++++++++++------ 3 files changed, 42 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c index 6d3b3e5a5533..ee4802d7b36c 100644 --- a/arch/x86/kvm/mmu/tdp_iter.c +++ b/arch/x86/kvm/mmu/tdp_iter.c @@ -145,6 +145,15 @@ static bool try_step_up(struct tdp_iter *iter) return true; } +/* + * Step the iterator back up a level in the paging structure. Should only be + * used when the iterator is below the root level. + */ +void tdp_iter_step_up(struct tdp_iter *iter) +{ + WARN_ON(!try_step_up(iter)); +} + /* * Step to the next SPTE in a pre-order traversal of the paging structure. * To get to the next SPTE, the iterator either steps down towards the goal diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index f0af385c56e0..adfca0cf94d3 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -114,5 +114,6 @@ void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn); void tdp_iter_next(struct tdp_iter *iter); void tdp_iter_restart(struct tdp_iter *iter); +void tdp_iter_step_up(struct tdp_iter *iter); #endif /* __KVM_X86_MMU_TDP_ITER_H */ diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 841feaa48be5..7b9265d67131 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1742,12 +1742,12 @@ static void zap_collapsible_spte_range(struct kvm *kvm, gfn_t start = slot->base_gfn; gfn_t end = start + slot->npages; struct tdp_iter iter; + int max_mapping_level; kvm_pfn_t pfn; rcu_read_lock(); tdp_root_for_each_pte(iter, root, start, end) { -retry: if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true)) continue; @@ -1755,15 +1755,41 @@ static void zap_collapsible_spte_range(struct kvm *kvm, !is_last_spte(iter.old_spte, iter.level)) continue; + /* + * This is a leaf SPTE. Check if the PFN it maps can + * be mapped at a higher level. + */ pfn = spte_to_pfn(iter.old_spte); - if (kvm_is_reserved_pfn(pfn) || - iter.level >= kvm_mmu_max_mapping_level(kvm, slot, iter.gfn, - pfn, PG_LEVEL_NUM)) + + if (kvm_is_reserved_pfn(pfn)) continue; + max_mapping_level = kvm_mmu_max_mapping_level(kvm, slot, + iter.gfn, pfn, PG_LEVEL_NUM); + + WARN_ON(max_mapping_level < iter.level); + + /* + * If this page is already mapped at the highest + * viable level, there's nothing more to do. + */ + if (max_mapping_level == iter.level) + continue; + + /* + * The page can be remapped at a higher level, so step + * up to zap the parent SPTE. + */ + while (max_mapping_level > iter.level) + tdp_iter_step_up(&iter); + /* Note, a successful atomic zap also does a remote TLB flush. */ - if (tdp_mmu_zap_spte_atomic(kvm, &iter)) - goto retry; + tdp_mmu_zap_spte_atomic(kvm, &iter); + + /* + * If the atomic zap fails, the iter will recurse back into + * the same subtree to retry. + */ } rcu_read_unlock();