From patchwork Sat Feb 13 00:50:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A18FEC433E6 for ; Sat, 13 Feb 2021 00:51:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7B53E64E02 for ; Sat, 13 Feb 2021 00:51:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229650AbhBMAvW (ORCPT ); Fri, 12 Feb 2021 19:51:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232005AbhBMAvE (ORCPT ); Fri, 12 Feb 2021 19:51:04 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70F13C061786 for ; Fri, 12 Feb 2021 16:50:24 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id 203so1567722ybz.2 for ; Fri, 12 Feb 2021 16:50:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=4X8Xawmkqq6JrMI8EQ6Ez131d+8bBQcFXdbpvRYDUz4=; b=GnlMk7vhQFA5V4yJOVvifabbC2ElQwDOtmcUG44rof+fgdTrwGAFwFX4p3LG9fLNL1 WaxcclGNJu64urykhUWGvJ++jy5bkggdIO7iB0nOvezt1DQxBx70ZyzB109ZGWQ1KRko xZsW64uIrhXxCvXyhiJx0h7L9Uk00AEvRY166UfiWKCjhiiylBRnm1R2ZpOX5tDzX5zp NGGTV/eaLw78lGrN8y9luYglA6il47mudOSGjs0bmXxZE8aqUeEyjJucAdH9kbulQwob TdW9q/RIhC7ZZxuXKxHk127puPNTRLP8Xmmyd5SNT83kU/GrswKAKKeqb5+lePK25SbG d+JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=4X8Xawmkqq6JrMI8EQ6Ez131d+8bBQcFXdbpvRYDUz4=; b=SOUt0c6V58Ma9zlSrVBgQuO2eGsNypU5om49zYCBAYceCwUMLGWgXh5LuchR8z2G/Q LDW5YZCUd93Mgic4Hwaxu9omFysxPOe8eFqCCiwl7+wDotmgT+Qcx5VQH94cPKMdnQ0m kGhvJW42xWX+5rgAbVPHu9zZUwPpQQrz+bzXUMl5Y/LiX7mYXpOdIpVBFYmzekOj0JYQ Q+ZW1baSbvg+Z7g1ed1IDjIEb4A6HMi3CYvQGZSbqE0XNKynq4yfrDbRvco9CB7nqIj9 UzXf0oIev1ej28jHAhS5PLtUA4cw+83QRW7kllcntsDXiH4zQimqzJga1GlCvCQ6GXDv OmHw== X-Gm-Message-State: AOAM533B9x1/KKXXO2Qj3uVj1tbQGDzmiSfwG7OVNkdh86SI/nx9LLg2 buBqynFbVICBiufRuH47kct0C2yIQh8= X-Google-Smtp-Source: ABdhPJwisJLVUO82g7ZpXF02safHacPg3WhZ69zQ2JPkhSdW2WAFwkbFqEizlPxxQ/9Fc1CQ+oCBmi5Py7E= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a25:e010:: with SMTP id x16mr7990153ybg.17.1613177423616; Fri, 12 Feb 2021 16:50:23 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:02 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-2-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 01/14] KVM: x86/mmu: Expand collapsible SPTE zap for TDP MMU to ZONE_DEVICE pages From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Zap SPTEs that are backed by ZONE_DEVICE pages when zappings SPTEs to rebuild them as huge pages in the TDP MMU. ZONE_DEVICE huge pages are managed differently than "regular" pages and are not compound pages. Cc: Ben Gardon Fixes: 14881998566d ("kvm: x86/mmu: Support disabling dirty logging for the tdp MMU") Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 71e100a5670f..3cc332ed099d 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1348,7 +1348,8 @@ static void zap_collapsible_spte_range(struct kvm *kvm, pfn = spte_to_pfn(iter.old_spte); if (kvm_is_reserved_pfn(pfn) || - !PageTransCompoundMap(pfn_to_page(pfn))) + (!PageTransCompoundMap(pfn_to_page(pfn)) && + !kvm_is_zone_device_pfn(pfn))) continue; tdp_mmu_set_spte(kvm, &iter, 0); From patchwork Sat Feb 13 00:50:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C953C433DB for ; Sat, 13 Feb 2021 00:52:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F38CD64E12 for ; Sat, 13 Feb 2021 00:52:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232210AbhBMAvt (ORCPT ); Fri, 12 Feb 2021 19:51:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232035AbhBMAvl (ORCPT ); Fri, 12 Feb 2021 19:51:41 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B815C06178A for ; Fri, 12 Feb 2021 16:50:26 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id 6so1554687ybq.7 for ; Fri, 12 Feb 2021 16:50:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=i+ZGhrnKuPWM2EUTyjRTEqFvmRgkb+1SD/2rklmgcTk=; b=KqaYODfcV7nggQfaJ4scLLVmaifGfME/RMcIJIdgowvyGEWXW9E8lqApJYsKmfDrlF rjLe25VECS2mTpwRLbDq1JHOKHjlQUvkCf0nGaanW2J8XmTGoTHrQGgc1pu19DeG8QdM i4EQaOC2+MzIa22bqqVekL/UoA6S6HGSPFnYfPgxaeVK4ZATE/ItsocqH3HJdT9kOlF9 XsRxpBvtjQunUw4OW98zBktnPUK4Ml4EW+cgBF3AMsaRxVkY9KcVdjSIV6sOEfnkCEjC Z0D2mcMiOSjglDyPduEYnkyBwib9myX0ATDli2qavK69zmg/dpL29Y0u/JcAX/gx2z0p qgrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=i+ZGhrnKuPWM2EUTyjRTEqFvmRgkb+1SD/2rklmgcTk=; b=merExqAe8jTZWJdiwu19BeySBoW63xWBx0JrZgQNjTcGbnUiDI4ATdeOIv51pH0y0z Ncm2P5oNsmnRF4elDG+4AxiUIbk0hacaSvLcfXI3Y3UdFNtPjRPCiOd42bVvKnx04Hpm WMUftLey2/+zspqkUwyMLKCJ8uCemh/kFd5qltzm3rJykSxCVgtc0XNG6mdCJfO3Fdju II6dcChL+SoQ55LpDnLXaCm51PuS5PPnMEoBsokz7OBvMFkdNAiLdq5RpdPZoNCzAhrr Pd3DqOYKq24IfsbnTVqQzaihXg5s9vpirrQK65Hf34+nkGFoXcoLzRGlE+zzwnkid4Bj QtZA== X-Gm-Message-State: AOAM533eT/BQtDjt97l5cMp6gkcxzowlvsibZc+HgsmjcOd5Bw3+5RBv LPCTwRf7SQFK1AqzbhHCEHmzvMcdVlI= X-Google-Smtp-Source: ABdhPJydadXiqwiePPV/u2nRZxbP9oNG8qal8CxU4l+Y3inZUDXFtfyb9NoV3cZws6bGobKUJ1RK+HH9dPk= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a5b:44f:: with SMTP id s15mr7978526ybp.85.1613177425898; Fri, 12 Feb 2021 16:50:25 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:03 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-3-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 02/14] KVM: x86/mmu: Don't unnecessarily write-protect small pages in TDP MMU From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Respect start_level when write-protect pages in the TDP MMU for dirty logging. When the dirty bitmaps are initialized with all bits set, small pages don't need to be write-protected as they've already been marked dirty. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index e507568cd55d..24325bdcd387 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5500,7 +5500,7 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, flush = slot_handle_level(kvm, memslot, slot_rmap_write_protect, start_level, KVM_MAX_HUGEPAGE_LEVEL, false); if (is_tdp_mmu_enabled(kvm)) - flush |= kvm_tdp_mmu_wrprot_slot(kvm, memslot, PG_LEVEL_4K); + flush |= kvm_tdp_mmu_wrprot_slot(kvm, memslot, start_level); write_unlock(&kvm->mmu_lock); /* From patchwork Sat Feb 13 00:50:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67B75C433E9 for ; Sat, 13 Feb 2021 00:52:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 335F46186A for ; Sat, 13 Feb 2021 00:52:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232292AbhBMAwG (ORCPT ); Fri, 12 Feb 2021 19:52:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232266AbhBMAv6 (ORCPT ); Fri, 12 Feb 2021 19:51:58 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4873C06178C for ; Fri, 12 Feb 2021 16:50:28 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id 11so1515984ybl.21 for ; Fri, 12 Feb 2021 16:50:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=Xy8pCiIvuQawQkMclMW43HMKCXEyM/TQwcT6KDk/2UM=; b=wAQXnTKYjLxM1KYKu3MpoFcyTfkTRoWBI/P4nnUbtn5uFdGazTlZ/tQmAEDPSN5e4f 2Qw8cmxwdgv1EX2K7PAHEJKBwFzo7EquKs8O3k4PnRz3TZlsDFntv0rsaUobpVkVQymQ Muir3noWpX10lSYdLXkmj0GB5z2n4VcE80nwlnPhWZiI7mXNEzEIT2IDP+CRytEUxg4f +k0OUV480QFbzGGqHkod6rO364qj6ok5PwjyQMBvoYQXXkCna5kgqS+JjttwZdMRwNgK BGinmZ64cdSJXMHL8um8Y4vQSRdmykd2r7NexzT94OH48N96aCEec9ddZ6fWqeEMaeju Fi3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Xy8pCiIvuQawQkMclMW43HMKCXEyM/TQwcT6KDk/2UM=; b=q5f2vMchwwYo35HFyIO5ONHppsgrGtlH5OM/OTqAc4a7VrJew5e+vohaOQD0kK4A3M SGd0xo4FeEbI95D4scCc5RYYXVv2bkfHDEqu+NlbqqJimecEFWxjfyZs5hjNciEmWpMg ZyVWeXdmip33Oql9vdnGZEEQmxvkMG/Qyv1pnwM+m1903tWM9JqKKZLesTbZ17rc4HmN k455RD4WLaPXGNCfcRJPb1ZBUHfHUfEidDhZqT/abJjUhQxJiOpS25HUXKDs8NmoKbdy PKmHRKU5M6Ay6bapn2CDUYULd/hxa932cK0Qj2Vn4yrADluHeqs4dpmGjmdDcvJz19qM bdyA== X-Gm-Message-State: AOAM531FSsfURisFXfIpp7113a/GY8TiDLhycNuC7aZqkapKLP2a7fNE ux1zfCyUKeAUDZVVCKhbCSvB9TMg9Fc= X-Google-Smtp-Source: ABdhPJwipfwSvALM02NbIZU9hDEBvvc702cNbC3lBfKrkGF50eyli8r9WWdXlfcLpzEkVEMNgm685llCU3c= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a5b:410:: with SMTP id m16mr8127952ybp.451.1613177428230; Fri, 12 Feb 2021 16:50:28 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:04 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-4-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 03/14] KVM: x86/mmu: Split out max mapping level calculation to helper From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Factor out the logic for determining the maximum mapping level given a memslot and a gpa. The helper will be used when zapping collapsible SPTEs when disabling dirty logging, e.g. to avoid zapping SPTEs that can't possibly be rebuilt as hugepages. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 37 ++++++++++++++++++++------------- arch/x86/kvm/mmu/mmu_internal.h | 2 ++ 2 files changed, 24 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 24325bdcd387..9be7fd474b2d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2756,8 +2756,8 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) __direct_pte_prefetch(vcpu, sp, sptep); } -static int host_pfn_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn, - kvm_pfn_t pfn, struct kvm_memory_slot *slot) +static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, + struct kvm_memory_slot *slot) { unsigned long hva; pte_t *pte; @@ -2776,19 +2776,36 @@ static int host_pfn_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn, */ hva = __gfn_to_hva_memslot(slot, gfn); - pte = lookup_address_in_mm(vcpu->kvm->mm, hva, &level); + pte = lookup_address_in_mm(kvm->mm, hva, &level); if (unlikely(!pte)) return PG_LEVEL_4K; return level; } +int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_memory_slot *slot, + gfn_t gfn, kvm_pfn_t pfn, int max_level) +{ + struct kvm_lpage_info *linfo; + + max_level = min(max_level, max_huge_page_level); + for ( ; max_level > PG_LEVEL_4K; max_level--) { + linfo = lpage_info_slot(gfn, slot, max_level); + if (!linfo->disallow_lpage) + break; + } + + if (max_level == PG_LEVEL_4K) + return PG_LEVEL_4K; + + return host_pfn_mapping_level(kvm, gfn, pfn, slot); +} + int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn, int max_level, kvm_pfn_t *pfnp, bool huge_page_disallowed, int *req_level) { struct kvm_memory_slot *slot; - struct kvm_lpage_info *linfo; kvm_pfn_t pfn = *pfnp; kvm_pfn_t mask; int level; @@ -2805,17 +2822,7 @@ int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn, if (!slot) return PG_LEVEL_4K; - max_level = min(max_level, max_huge_page_level); - for ( ; max_level > PG_LEVEL_4K; max_level--) { - linfo = lpage_info_slot(gfn, slot, max_level); - if (!linfo->disallow_lpage) - break; - } - - if (max_level == PG_LEVEL_4K) - return PG_LEVEL_4K; - - level = host_pfn_mapping_level(vcpu, gfn, pfn, slot); + level = kvm_mmu_max_mapping_level(vcpu->kvm, slot, gfn, pfn, max_level); if (level == PG_LEVEL_4K) return level; diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index 9e38d3c5daad..0b55aa561ec8 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -138,6 +138,8 @@ enum { #define SET_SPTE_NEED_REMOTE_TLB_FLUSH BIT(1) #define SET_SPTE_SPURIOUS BIT(2) +int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_memory_slot *slot, + gfn_t gfn, kvm_pfn_t pfn, int max_level); int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn, int max_level, kvm_pfn_t *pfnp, bool huge_page_disallowed, int *req_level); From patchwork Sat Feb 13 00:50:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7528FC43381 for ; Sat, 13 Feb 2021 00:52:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5AA9E64E12 for ; Sat, 13 Feb 2021 00:52:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229648AbhBMAwN (ORCPT ); Fri, 12 Feb 2021 19:52:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232274AbhBMAwA (ORCPT ); Fri, 12 Feb 2021 19:52:00 -0500 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64BEEC061797 for ; Fri, 12 Feb 2021 16:50:31 -0800 (PST) Received: by mail-qv1-xf49.google.com with SMTP id k14so849256qvw.17 for ; Fri, 12 Feb 2021 16:50:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=/pgObbPbQ4Sej93TFL9Cnuu26Kb3Ki6jPLia3nFiTXA=; b=AIjpzv6+8htZQRayxywmziLjdvR/NzxH2IuQBza8yuJn07/odsujV31NpiUWeXOUTR TWWVdISqcXa5wxiCREFw94UIWTxqqN2Mw+LFesqMB4trjM0A1MRQ/ytXfmUOlg7PCjUs rZ1pVuSQWUaXIVj6k15m0/KxMQIJiVTLV26bPf7q/+qsbvAQMT+j4xsSA95gRnUeKkUM tzoXrFFDFRYy/oqZ5OuDDW7BdWQ0BQj3DMZJ7DewX6yfnuzQcuU1Bwg1bGGJHx9SDNt/ NAbTAqW1brL5l30WBhFjQgOWD1AdFU9C5MUiM4/WzDDQ5l+hWQ2iFAdp/p+l4U6avkh1 jxnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=/pgObbPbQ4Sej93TFL9Cnuu26Kb3Ki6jPLia3nFiTXA=; b=lYncmJxZqXcX2jAO+aD1n8m6T07aK8mMKSWVi/4m6SLCA8PgkdhXIRuUiBXpXmbWsf CR3jNxOPBSkLlkCt4xgxK5UnYAKi1JJwxZv8TZiYVpHeGKhPaMFfmbDfOH1b3I5+laU4 k1SdslhPP/2PTRFmOWwdUsuxRjOJT/HOT+BGYLwxiSnFuoUbpLnvwiErVoCoqVAqqpMq RiXxLfcasylNFOmvKTgzxPIhIG2RbDDC/Z6CeTFEc+UrvcYXHQti8dFAYoYMFzxhU8a9 AgeB5VztTYGFgySW1EBYaXiioN9AJHWCBC3n95pM0xk2lN21JRGsDmH5XYPh9hdN0IIr aCoA== X-Gm-Message-State: AOAM530cD15/YxC6VvmW5T4Ll4ZWDgDYAUp/WjLJU1bsHqQ6BBFZtQQG V8i5clCDrLbJhYmSc3BzxkVSZLkOBV8= X-Google-Smtp-Source: ABdhPJzLCM67nkU5MoKMf58mtdWJl/IRwyhumZ+dwwpFdhfFlqR/KeLI9I3s95s5+hJtmi7nEYAAtpjwqTA= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:ad4:55aa:: with SMTP id f10mr5145395qvx.46.1613177430549; Fri, 12 Feb 2021 16:50:30 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:05 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-5-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 04/14] KVM: x86/mmu: Pass the memslot to the rmap callbacks From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Pass the memslot to the rmap callbacks, it will be used when zapping collapsible SPTEs to verify the memslot is compatible with hugepages before zapping its SPTEs. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 9be7fd474b2d..fb719e7a0cbb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1165,7 +1165,8 @@ static bool spte_wrprot_for_clear_dirty(u64 *sptep) * - W bit on ad-disabled SPTEs. * Returns true iff any D or W bits were cleared. */ -static bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head) +static bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot) { u64 *sptep; struct rmap_iterator iter; @@ -1196,7 +1197,8 @@ static bool spte_set_dirty(u64 *sptep) return mmu_spte_update(sptep, spte); } -static bool __rmap_set_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head) +static bool __rmap_set_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot) { u64 *sptep; struct rmap_iterator iter; @@ -1260,7 +1262,7 @@ void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, while (mask) { rmap_head = __gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), PG_LEVEL_4K, slot); - __rmap_clear_dirty(kvm, rmap_head); + __rmap_clear_dirty(kvm, rmap_head, slot); /* clear the first set bit */ mask &= mask - 1; @@ -1325,7 +1327,8 @@ static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn) return kvm_mmu_slot_gfn_write_protect(vcpu->kvm, slot, gfn); } -static bool kvm_zap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head) +static bool kvm_zap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot) { u64 *sptep; struct rmap_iterator iter; @@ -1345,7 +1348,7 @@ static int kvm_unmap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct kvm_memory_slot *slot, gfn_t gfn, int level, unsigned long data) { - return kvm_zap_rmapp(kvm, rmap_head); + return kvm_zap_rmapp(kvm, rmap_head, slot); } static int kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, @@ -5189,7 +5192,8 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_max_root_level, EXPORT_SYMBOL_GPL(kvm_configure_mmu); /* The return value indicates if tlb flush on all vcpus is needed. */ -typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head); +typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot); /* The caller should hold mmu-lock before calling this function. */ static __always_inline bool @@ -5203,7 +5207,7 @@ slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, for_each_slot_rmap_range(memslot, start_level, end_level, start_gfn, end_gfn, &iterator) { if (iterator.rmap) - flush |= fn(kvm, iterator.rmap); + flush |= fn(kvm, iterator.rmap, memslot); if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { if (flush && lock_flush_tlb) { @@ -5492,7 +5496,8 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) } static bool slot_rmap_write_protect(struct kvm *kvm, - struct kvm_rmap_head *rmap_head) + struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot) { return __rmap_write_protect(kvm, rmap_head, false); } @@ -5526,7 +5531,8 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, } static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, - struct kvm_rmap_head *rmap_head) + struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot) { u64 *sptep; struct rmap_iterator iter; From patchwork Sat Feb 13 00:50:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A960C433E0 for ; Sat, 13 Feb 2021 00:52:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 531506186A for ; Sat, 13 Feb 2021 00:52:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232302AbhBMAwS (ORCPT ); Fri, 12 Feb 2021 19:52:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232283AbhBMAwA (ORCPT ); Fri, 12 Feb 2021 19:52:00 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98778C0617A9 for ; Fri, 12 Feb 2021 16:50:33 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id v76so1506873ybi.20 for ; Fri, 12 Feb 2021 16:50:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=+te5vGqQ5mfKNQSufhmqpYAsxPiqcN/tujlZYmMYYVw=; b=VjSddAkEPcauk0zGfQvzxdrdIOG7KG4YcmZKKPvj0nqqVta/CYkoAnlYAVP7NGhDkm VdZBTa51S4/hpyxd0ThLgJG8f/dEZ0AlJ1CVOPolFsNR0OyUFNvc91z9e1NRtqqgw5sr /gf/SrL62hwp+M62aEA9Et5BWfTJb6BO7DKeaqZ3mk4dohhHJsioMM4q2xFhiRsJyQWd K1p7uqg/r1oLyvNHeQFTE6hz8gNo7buXmcu9K0UC+9uM+DbZfGTVyUtveDcU0HYaFaB7 7T1xNQuzSzf3R/90eBeZf4uTBH5HOsvIZT0fNO7TU9GLNZ0vLeA72aJs4zeY8urzInzv x9dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=+te5vGqQ5mfKNQSufhmqpYAsxPiqcN/tujlZYmMYYVw=; b=Waj7n7b4ibk1l5ZgIVCvQWPC4r0/oyWBUiwK0DXJeFA/tSQQ8PFcga7BPkrzBcTWqz FoIFZd/WF+5ETvl483qnISAz6DRelN5iHakybLWApM6c5dwmGWgkiZ8dEnIG2YdR3t/c 2C1LDo5OWbcigVqzLFDnDxh46m1WmcyMyugBYvCuP8Yv3MMcbY2BJBjhl+RgWpWNnjZ6 s0AO5zZN7E/Jprhb4YwRr3e7fQaDyKUvV8jBzFIEmpGVb4qXhL0yKKAXkkwWJi0hbK6Y hv1TWACzCHMEbLwQmohDdn6cqchGCDBRfoObF0kD0kEV3OzOw1HEO7Ig4hzXc/Z5m6UD /F7Q== X-Gm-Message-State: AOAM533mZi3W4YCr0pBtgQzwbtOPMqSAVPVXgvbRAWZUXw4LxPLzIYYs spObGHzJtm/FOy+jwBq751MsAuU0u3Y= X-Google-Smtp-Source: ABdhPJxhavZBuKuUkKtaiyn3oAiDTSshrMq+lKxR+qNEpsbs/3dJju2omzfy+anN6gfXHZXppW95DqorLbE= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a25:cbc8:: with SMTP id b191mr7818312ybg.61.1613177432884; Fri, 12 Feb 2021 16:50:32 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:06 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-6-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 05/14] KVM: x86/mmu: Consult max mapping level when zapping collapsible SPTEs From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When zapping SPTEs in order to rebuild them as huge pages, use the new helper that computes the max mapping level to detect whether or not a SPTE should be zapped. Doing so avoids zapping SPTEs that can't possibly be rebuilt as huge pages, e.g. due to hardware constraints, memslot alignment, etc... This also avoids zapping SPTEs that are still large, e.g. if migration was canceled before write-protected huge pages were shattered to enable dirty logging. Note, such pages are still write-protected at this time, i.e. a page fault VM-Exit will still occur. This will hopefully be addressed in a future patch. Sadly, TDP MMU loses its const on the memslot, but that's a pervasive problem that's been around for quite some time. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 11 ++++++----- arch/x86/kvm/mmu/tdp_mmu.c | 13 +++++++------ arch/x86/kvm/mmu/tdp_mmu.h | 2 +- 3 files changed, 14 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index fb719e7a0cbb..d5849a0e3de1 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5553,8 +5553,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, * mapping if the indirect sp has level = 1. */ if (sp->role.direct && !kvm_is_reserved_pfn(pfn) && - (kvm_is_zone_device_pfn(pfn) || - PageCompound(pfn_to_page(pfn)))) { + sp->role.level < kvm_mmu_max_mapping_level(kvm, slot, sp->gfn, + pfn, PG_LEVEL_NUM)) { pte_list_remove(rmap_head, sptep); if (kvm_available_flush_tlb_with_range()) @@ -5574,12 +5574,13 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *memslot) { /* FIXME: const-ify all uses of struct kvm_memory_slot. */ + struct kvm_memory_slot *slot = (struct kvm_memory_slot *)memslot; + write_lock(&kvm->mmu_lock); - slot_handle_leaf(kvm, (struct kvm_memory_slot *)memslot, - kvm_mmu_zap_collapsible_spte, true); + slot_handle_leaf(kvm, slot, kvm_mmu_zap_collapsible_spte, true); if (is_tdp_mmu_enabled(kvm)) - kvm_tdp_mmu_zap_collapsible_sptes(kvm, memslot); + kvm_tdp_mmu_zap_collapsible_sptes(kvm, slot); write_unlock(&kvm->mmu_lock); } diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 3cc332ed099d..f8fa1f64e10d 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1328,8 +1328,10 @@ bool kvm_tdp_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *slot) */ static void zap_collapsible_spte_range(struct kvm *kvm, struct kvm_mmu_page *root, - gfn_t start, gfn_t end) + struct kvm_memory_slot *slot) { + gfn_t start = slot->base_gfn; + gfn_t end = start + slot->npages; struct tdp_iter iter; kvm_pfn_t pfn; bool spte_set = false; @@ -1348,8 +1350,8 @@ static void zap_collapsible_spte_range(struct kvm *kvm, pfn = spte_to_pfn(iter.old_spte); if (kvm_is_reserved_pfn(pfn) || - (!PageTransCompoundMap(pfn_to_page(pfn)) && - !kvm_is_zone_device_pfn(pfn))) + iter.level >= kvm_mmu_max_mapping_level(kvm, slot, iter.gfn, + pfn, PG_LEVEL_NUM)) continue; tdp_mmu_set_spte(kvm, &iter, 0); @@ -1367,7 +1369,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm, * be replaced by large mappings, for GFNs within the slot. */ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm, - const struct kvm_memory_slot *slot) + struct kvm_memory_slot *slot) { struct kvm_mmu_page *root; int root_as_id; @@ -1377,8 +1379,7 @@ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm, if (root_as_id != slot->as_id) continue; - zap_collapsible_spte_range(kvm, root, slot->base_gfn, - slot->base_gfn + slot->npages); + zap_collapsible_spte_range(kvm, root, slot); } } diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index b4b65e3699b3..d31c5ed81a18 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -35,7 +35,7 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm, bool wrprot); bool kvm_tdp_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *slot); void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm, - const struct kvm_memory_slot *slot); + struct kvm_memory_slot *slot); bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn); From patchwork Sat Feb 13 00:50:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9766DC433DB for ; Sat, 13 Feb 2021 00:52:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6A03164E9A for ; Sat, 13 Feb 2021 00:52:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232319AbhBMAwX (ORCPT ); Fri, 12 Feb 2021 19:52:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232287AbhBMAwA (ORCPT ); Fri, 12 Feb 2021 19:52:00 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 157A2C0617AA for ; Fri, 12 Feb 2021 16:50:36 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id f3so1519870ybg.15 for ; Fri, 12 Feb 2021 16:50:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=4/zuTH5SAwBEhAeoyaVLOks5C/8P9wHBnzUFavIOLk8=; b=s+oXipZ6yecLqPMU3rX0QhB2XzhuAwb9QNI+3AJlokHkJ2/o5GxKndtdVsWBmxzpcU DdCQBi7Mc3hoo6OMLBtG2maAgaqMd45ko82uPhyFuCYTuoyMp7Zj4Mm2LvnmNVeecGBU yVUmijWN3hCJrxoeehLsVyD70jJvbkti4Os2glK28almIjplJB3l/ejBPn4/7GCBtjd8 0pFxzgkBv54a0kH/1Zc27OY16fuJFIRqWutualLHtnTCkbunJ6HqmaiZC0JtqZC8UIeg gYcZtdlCXu2KZqdJw7jxIPkd1FDjfGKYUJPIq49iHRUU+t8QABroZ6vcVSwQWDucGAsK 4RMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=4/zuTH5SAwBEhAeoyaVLOks5C/8P9wHBnzUFavIOLk8=; b=QT54LnxbBRLu54/JmfQpYqqTnvjqb7pfWRdcuF9HPPxfftXoZOCawREBwF0M3D0nF4 o28GIdiAo4PWx4reO8GFEV6+JSg2/T8k/NESE6obtmkso0HHhdpzSQAJG3Oj3/o8/fO/ mY/h3FRQ007lFofftYOwEt49en6MKwLMvNv/dIup8lcM1gyY4za7XjyyFXhoWb3ca9Kg VUcEvBQrhbHSga0n8TYF9c3bNEOvpAYWA3He34TgPOyyggGMpqwbyYpFkc/5Mf2xqBPa uTlyV5NiuSKmNH5ALLBm/syr79c0j8CFJ0gVH3ylhNS/LO0k/AMM86yD1dWglVOyjCpR ITIA== X-Gm-Message-State: AOAM532mETo2+vkA6NBWW7LzA8RD6dk6NBux72Djvpig9XSo4SMEfc+Z 1oiIhp7iAzgBxq0Ncwichy8wFMX4U1E= X-Google-Smtp-Source: ABdhPJwRpIOAwKGTBkrSNtQ2peg5RJBH7kNNlsuhvGsFoqI0qZsXjtEjWgj3M8V9YbgMlEom8Vs70Zqe06s= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a25:d94b:: with SMTP id q72mr7215415ybg.135.1613177435305; Fri, 12 Feb 2021 16:50:35 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:07 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-7-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 06/14] KVM: nVMX: Disable PML in hardware when running L2 From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Unconditionally disable PML in vmcs02, KVM emulates PML purely in the MMU, e.g. vmx_flush_pml_buffer() doesn't even try to copy the L2 GPAs from vmcs02's buffer to vmcs12. At best, enabling PML is a nop. At worst, it will cause vmx_flush_pml_buffer() to record bogus GFNs in the dirty logs. Initialize vmcs02.GUEST_PML_INDEX such that PML writes would trigger VM-Exit if PML was somehow enabled, skip flushing the buffer for guest mode since the index is bogus, and freak out if a PML full exit occurs when L2 is active. Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/nested.c | 29 +++++++++++++++-------------- arch/x86/kvm/vmx/vmx.c | 12 ++++++++++-- 2 files changed, 25 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index b2f0b5e9cd63..0c6dda9980a6 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -2167,15 +2167,13 @@ static void prepare_vmcs02_constant_state(struct vcpu_vmx *vmx) vmcs_write64(MSR_BITMAP, __pa(vmx->nested.vmcs02.msr_bitmap)); /* - * The PML address never changes, so it is constant in vmcs02. - * Conceptually we want to copy the PML index from vmcs01 here, - * and then back to vmcs01 on nested vmexit. But since we flush - * the log and reset GUEST_PML_INDEX on each vmexit, the PML - * index is also effectively constant in vmcs02. + * PML is emulated for L2, but never enabled in hardware as the MMU + * handles A/D emulation. Disabling PML for L2 also avoids having to + * deal with filtering out L2 GPAs from the buffer. */ if (enable_pml) { - vmcs_write64(PML_ADDRESS, page_to_phys(vmx->pml_pg)); - vmcs_write16(GUEST_PML_INDEX, PML_ENTITY_NUM - 1); + vmcs_write64(PML_ADDRESS, 0); + vmcs_write16(GUEST_PML_INDEX, -1); } if (cpu_has_vmx_encls_vmexit()) @@ -2210,7 +2208,7 @@ static void prepare_vmcs02_early_rare(struct vcpu_vmx *vmx, static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12) { - u32 exec_control, vmcs12_exec_ctrl; + u32 exec_control; u64 guest_efer = nested_vmx_calc_efer(vmx, vmcs12); if (vmx->nested.dirty_vmcs12 || vmx->nested.hv_evmcs) @@ -2284,11 +2282,11 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12) SECONDARY_EXEC_APIC_REGISTER_VIRT | SECONDARY_EXEC_ENABLE_VMFUNC); if (nested_cpu_has(vmcs12, - CPU_BASED_ACTIVATE_SECONDARY_CONTROLS)) { - vmcs12_exec_ctrl = vmcs12->secondary_vm_exec_control & - ~SECONDARY_EXEC_ENABLE_PML; - exec_control |= vmcs12_exec_ctrl; - } + CPU_BASED_ACTIVATE_SECONDARY_CONTROLS)) + exec_control |= vmcs12->secondary_vm_exec_control; + + /* PML is emulated and never enabled in hardware for L2. */ + exec_control &= ~SECONDARY_EXEC_ENABLE_PML; /* VMCS shadowing for L2 is emulated for now */ exec_control &= ~SECONDARY_EXEC_SHADOW_VMCS; @@ -5793,7 +5791,10 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu *vcpu, case EXIT_REASON_PREEMPTION_TIMER: return true; case EXIT_REASON_PML_FULL: - /* We emulate PML support to L1. */ + /* + * PML is emulated for an L1 VMM and should never be enabled in + * vmcs02, always "handle" PML_FULL by exiting to userspace. + */ return true; case EXIT_REASON_VMFUNC: /* VM functions are emulated through L2->L0 vmexits. */ diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index e0a3a9be654b..b47ed3f412ef 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5976,9 +5976,10 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) * updated. Another good is, in kvm_vm_ioctl_get_dirty_log, before * querying dirty_bitmap, we only need to kick all vcpus out of guest * mode as if vcpus is in root mode, the PML buffer must has been - * flushed already. + * flushed already. Note, PML is never enabled in hardware while + * running L2. */ - if (enable_pml) + if (enable_pml && !is_guest_mode(vcpu)) vmx_flush_pml_buffer(vcpu); /* @@ -5994,6 +5995,13 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) return handle_invalid_guest_state(vcpu); if (is_guest_mode(vcpu)) { + /* + * PML is never enabled when running L2, bail immediately if a + * PML full exit occurs as something is horribly wrong. + */ + if (exit_reason.basic == EXIT_REASON_PML_FULL) + goto unexpected_vmexit; + /* * The host physical addresses of some pages of guest memory * are loaded into the vmcs02 (e.g. vmcs12's Virtual APIC From patchwork Sat Feb 13 00:50:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B16B3C433E6 for ; Sat, 13 Feb 2021 00:53:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 802A164EAA for ; Sat, 13 Feb 2021 00:53:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232360AbhBMAxG (ORCPT ); Fri, 12 Feb 2021 19:53:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232338AbhBMAw2 (ORCPT ); Fri, 12 Feb 2021 19:52:28 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A07CCC06121C for ; Fri, 12 Feb 2021 16:50:38 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id u1so1523879ybu.14 for ; Fri, 12 Feb 2021 16:50:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=qgHHy11/Gmg7Kjm2oa98mzp3zSXXJwJ1MHurK3QurWI=; b=JzER6/A4WcR1Yxht+KX+/Ie2lNSUt4QDexn+mTaBXkA3mUuSs4eh+qma7hs8FCrhX9 a9I98dy32xEYCSrwTkDbadFnZmBsAvDRXPreGzmTPDzI+orEI3qKKMoQCqZrjCZYaFzQ ZX9qU1Qc3D3y3vWrxxbTo0GMSfTBQDPJkm3Xg4rrVmifiiu3rpbeYzAGZfUx/wZFYAOt pGsIfZQFJe+yFQzjKJCfQNMnBMNNVaxP/iY4lWZNItq6y10vYPBYxA2U28WcfVOnchbv hCL6a1w9X8EeRIMsP5yVhf6s+o0YxVuA3x45+RgbRo2wmMaETBELL+eM/OJB0ixMaYw5 CPzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=qgHHy11/Gmg7Kjm2oa98mzp3zSXXJwJ1MHurK3QurWI=; b=BPijwuW2mJtvRoZue42IEH3xezpTnr6AHScr5dIkE9WBQXEZ4JD/mlQAUpZWbhZwf3 t3s9Jt7UXmVHvA0ugo3tmWlkMUESOD369B9XFkq0giiAYEZlM65HmMdD5Cv31FhP5VJh 8Co8bgfC8Bdq0WpzEVNtTGx42erARAudsV0lTWtGE32Z3V3vifW3c/T/H6+ZCs4+0sB8 IVavyznDEpOIFhhwUyyKXL98a4gywzSNK74GdJ+iz6rX5OL3zNzHyizAnvxvCblD2B/t tdMkvUjorKKpad9L459LBgLXPUzSgZUTncDMFDLV3SAGQdNkMZ7aKDgNRnOWZnGLxLYN ibJw== X-Gm-Message-State: AOAM531DSirDbfLIowiniEWz7osqXVmWMxEo+lTVBL2H6kJqB3mpHKId ZFc4c851dtal6B3TgdiF7edflQQ7XkI= X-Google-Smtp-Source: ABdhPJz1CJdQJZRt8FQ7rL9sa+SP7EuA2BCPuYyVPIm2ReE/REHoc291v+SKPS1vg/Ta3l+cxKlaFs0Ntx4= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a05:6902:1d2:: with SMTP id u18mr7581074ybh.103.1613177437917; Fri, 12 Feb 2021 16:50:37 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:08 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-8-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 07/14] KVM: x86/mmu: Expand on the comment in kvm_vcpu_ad_need_write_protect() From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Expand the comment about need to use write-protection for nested EPT when PML is enabled to clarify that the tagging is a nop when PML is _not_ enabled. Without the clarification, omitting the PML check looks wrong at first^Wfifth glance. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu_internal.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index 0b55aa561ec8..72b0928f2b2d 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -84,7 +84,10 @@ static inline bool kvm_vcpu_ad_need_write_protect(struct kvm_vcpu *vcpu) * When using the EPT page-modification log, the GPAs in the log * would come from L2 rather than L1. Therefore, we need to rely * on write protection to record dirty pages. This also bypasses - * PML, since writes now result in a vmexit. + * PML, since writes now result in a vmexit. Note, this helper will + * tag SPTEs as needing write-protection even if PML is disabled or + * unsupported, but that's ok because the tag is consumed if and only + * if PML is enabled. Omit the PML check to save a few uops. */ return vcpu->arch.mmu == &vcpu->arch.guest_mmu; } From patchwork Sat Feb 13 00:50:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.5 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A727EC433E0 for ; Sat, 13 Feb 2021 00:52:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7F7EA64E12 for ; Sat, 13 Feb 2021 00:52:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232336AbhBMAw2 (ORCPT ); Fri, 12 Feb 2021 19:52:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232291AbhBMAwD (ORCPT ); Fri, 12 Feb 2021 19:52:03 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED095C06121D for ; Fri, 12 Feb 2021 16:50:40 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id x4so1505142ybj.22 for ; Fri, 12 Feb 2021 16:50:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=Ieh3m7EQmvHAenB/lO365hnHz2Mjo7nHpxo7FvHHoBw=; b=mNuXGR4d6txwMt3SlFenFMnZv3aIX6yvtjONUJnLPMPhv1wOi6LCn4DDstnc+BgAfR zcoEDg6VUgK7a6RWWMDSJgC1CH/BdKj7XF+q57uWZPURSrqJpdkKCwtDX6Q7vO7MpwkW FUne44zNf0ElX+1aXYr4XBGIkfi3emdxvP/3THped4mZElx6G7cKm3+JIFrnYUp8IOlU 6ZP785nI6R2m4TCfFvKdFw76c/bD3d7ZDtJb2d8qy3Mns/EGPOtI2ZKM0O5N4P2iWp6q mC/ZnEMcEqXG7Bitfwp0mIf875iMUA4yyjATr5fzqlcMwVY8DUsXalYYfhcIVzu0xvD/ jR3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Ieh3m7EQmvHAenB/lO365hnHz2Mjo7nHpxo7FvHHoBw=; b=P3yKp7AjsUJJqG25AjFvM7MPVQubzWbDNL63iejVoGQQetZuhGV2XwNY4yX/4oASLA AdtkcCUYLh18b0u71iavnL0sis/Twvl7D1d4kAyn/wdb0M5cOiRf8ihQ2n0NfbWka8V/ BV+pxJ0hsPHJFNgwz6ogBWyTumsvLZ0zb4hZXvdlFt8Enq9QHBugq2un7vj0tWycTQPY w7istRNSUril/xb1npgBJJHpAr9LDFA/C4/s6B+UsMy7RosrS1UMoBbikPJj8P0Wic1H +aajEAeiHzIG9Kb3ZgNlfYD0gMvJ+2xR+pG8Bu4iqhs1XbLRIymm/rlvIh0PkoydKGuw Osyg== X-Gm-Message-State: AOAM532+hJindXrsBQPFnNALufC9A0cW+4Hyze/JUHLjKj6wOW3h8f3M HZ+s4d9qqP8OVmy1kHbdSXrv3+OKyCM= X-Google-Smtp-Source: ABdhPJwTkYoQFr39i7NoXmnOU+dg9Dn93Bld3ArJYdgab0PwxQiWSWu+mp0scIpzemMdicfiwZLRRV4zHEc= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a25:db48:: with SMTP id g69mr7712789ybf.109.1613177440250; Fri, 12 Feb 2021 16:50:40 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:09 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-9-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 08/14] KVM: x86/mmu: Make dirty log size hook (PML) a value, not a function From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Store the vendor-specific dirty log size in a variable, there's no need to wrap it in a function since the value is constant after hardware_setup() runs. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm-x86-ops.h | 1 - arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/mmu/mmu.c | 5 +---- arch/x86/kvm/vmx/vmx.c | 9 ++------- 4 files changed, 4 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index 355a2ab8fc09..28c07cc01474 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -97,7 +97,6 @@ KVM_X86_OP_NULL(slot_enable_log_dirty) KVM_X86_OP_NULL(slot_disable_log_dirty) KVM_X86_OP_NULL(flush_log_dirty) KVM_X86_OP_NULL(enable_log_dirty_pt_masked) -KVM_X86_OP_NULL(cpu_dirty_log_size) KVM_X86_OP_NULL(pre_block) KVM_X86_OP_NULL(post_block) KVM_X86_OP_NULL(vcpu_blocking) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 84499aad01a4..fb59933610d9 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1294,7 +1294,7 @@ struct kvm_x86_ops { void (*enable_log_dirty_pt_masked)(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t offset, unsigned long mask); - int (*cpu_dirty_log_size)(void); + int cpu_dirty_log_size; /* pmu operations of sub-arch */ const struct kvm_pmu_ops *pmu_ops; diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index d5849a0e3de1..6c32e8e0f720 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1294,10 +1294,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, int kvm_cpu_dirty_log_size(void) { - if (kvm_x86_ops.cpu_dirty_log_size) - return static_call(kvm_x86_cpu_dirty_log_size)(); - - return 0; + return kvm_x86_ops.cpu_dirty_log_size; } bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index b47ed3f412ef..f843707dd7df 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7650,11 +7650,6 @@ static bool vmx_check_apicv_inhibit_reasons(ulong bit) return supported & BIT(bit); } -static int vmx_cpu_dirty_log_size(void) -{ - return enable_pml ? PML_ENTITY_NUM : 0; -} - static struct kvm_x86_ops vmx_x86_ops __initdata = { .hardware_unsetup = hardware_unsetup, @@ -7758,6 +7753,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .slot_disable_log_dirty = vmx_slot_disable_log_dirty, .flush_log_dirty = vmx_flush_log_dirty, .enable_log_dirty_pt_masked = vmx_enable_log_dirty_pt_masked, + .cpu_dirty_log_size = PML_ENTITY_NUM, .pre_block = vmx_pre_block, .post_block = vmx_post_block, @@ -7785,7 +7781,6 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .msr_filter_changed = vmx_msr_filter_changed, .complete_emulated_msr = kvm_complete_insn_gp, - .cpu_dirty_log_size = vmx_cpu_dirty_log_size, .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, }; @@ -7907,7 +7902,7 @@ static __init int hardware_setup(void) vmx_x86_ops.slot_disable_log_dirty = NULL; vmx_x86_ops.flush_log_dirty = NULL; vmx_x86_ops.enable_log_dirty_pt_masked = NULL; - vmx_x86_ops.cpu_dirty_log_size = NULL; + vmx_x86_ops.cpu_dirty_log_size = 0; } if (!cpu_has_vmx_preemption_timer()) From patchwork Sat Feb 13 00:50:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9F5BC433E9 for ; Sat, 13 Feb 2021 00:53:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B752164EB6 for ; Sat, 13 Feb 2021 00:53:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232418AbhBMAxO (ORCPT ); Fri, 12 Feb 2021 19:53:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232367AbhBMAwk (ORCPT ); Fri, 12 Feb 2021 19:52:40 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 485BCC061221 for ; Fri, 12 Feb 2021 16:50:43 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id s7so1574832ybj.0 for ; Fri, 12 Feb 2021 16:50:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=eB7aZXTG/r9znzgI5ELRRkyUfRHeewkpT855MZu37AE=; b=OVopWNaHtFPNCOpDQ1/tCi/GhbNIPMIFCXIxIk4h2rZQU17WGXCmdsamNVRvmid/Gl 38AMLiyl6bBcg3RMgSzWBEQMsjvoYeQB9uc86BaQg/WBSo5CabOUHS+IPJpcuAM3LrH5 CXbKdC4Iz6On9OGkJDk0bYRz42c/zFNtqxzhCafqhHMR1ttU/wgbpaiCkkeSoaepnnap ZeGIAP5HxLzbhWaMzAmteFCV3Ln+LysJ+94LNf1b+0MvknxBMqI8BteX2eq/kDGlLNj6 T3sDrIeksAdHdldmrn7Ug7m5FmvQIblcIwRbV36CqDmgyXWh1r3VfU81xyuj+k5cDd3h lsew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=eB7aZXTG/r9znzgI5ELRRkyUfRHeewkpT855MZu37AE=; b=fz4PiSKxKBwtMYHbcm7XdyNl/XktQpEjB+LWRGc/rI9vBRFRTrBfJrKzf/6OErRhBv NOlvVX7OWgRCdt+bl6SG7p8ViGLyIsNC4uxfEWJTTmOk1Pn/mM6M4mKtP67+rTZ3VceS m2wx5LlCBuAUyCtLKjYej2rLoX9FUVqRVvpxwDEiZ8CmGI7m0RxLk6tCbSTzmweicPMi kntjL4a06wIWLXFyfM+gESX/8f0/pdrwItt85X7ugDjCiXret+c5QD4y2ztcZ40TKdFY QhWZpKpIRDbG3jYY7bysXjJXk+R1mG3lNDbGLyWymbeX55R986bnIh02ahylmvBMndrJ eIFQ== X-Gm-Message-State: AOAM530/dqGf8uNZxYMF9RDgmuqRw6YgzrTOW8+aXot0hILrA2IYHwOb GLQJihFnai0TQF4gHW7UGWA/AjR44TA= X-Google-Smtp-Source: ABdhPJz4tMH1EqczGDDlpnTKl59BDhmPNmaAEMWZFfS0wgHvCUC6pMK7TO8gbZZ33TZZ4xCW1FhHc0s5mXQ= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a25:20c2:: with SMTP id g185mr8171042ybg.31.1613177442563; Fri, 12 Feb 2021 16:50:42 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:10 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-10-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 09/14] KVM: x86: Move MMU's PML logic to common code From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Drop the facade of KVM's PML logic being vendor specific and move the bits that aren't truly VMX specific into common x86 code. The MMU logic for dealing with PML is tightly coupled to the feature and to VMX's implementation, bouncing through kvm_x86_ops obfuscates the code without providing any meaningful separation of concerns or encapsulation. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm-x86-ops.h | 4 --- arch/x86/include/asm/kvm_host.h | 27 ++------------- arch/x86/kvm/mmu/mmu.c | 16 +++------ arch/x86/kvm/vmx/vmx.c | 55 +----------------------------- arch/x86/kvm/x86.c | 22 ++++++++---- 5 files changed, 24 insertions(+), 100 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index 28c07cc01474..90affdb2cbbc 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -93,10 +93,6 @@ KVM_X86_OP(check_intercept) KVM_X86_OP(handle_exit_irqoff) KVM_X86_OP_NULL(request_immediate_exit) KVM_X86_OP(sched_in) -KVM_X86_OP_NULL(slot_enable_log_dirty) -KVM_X86_OP_NULL(slot_disable_log_dirty) -KVM_X86_OP_NULL(flush_log_dirty) -KVM_X86_OP_NULL(enable_log_dirty_pt_masked) KVM_X86_OP_NULL(pre_block) KVM_X86_OP_NULL(post_block) KVM_X86_OP_NULL(vcpu_blocking) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index fb59933610d9..5cf382ec48b0 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1271,29 +1271,9 @@ struct kvm_x86_ops { void (*sched_in)(struct kvm_vcpu *kvm, int cpu); /* - * Arch-specific dirty logging hooks. These hooks are only supposed to - * be valid if the specific arch has hardware-accelerated dirty logging - * mechanism. Currently only for PML on VMX. - * - * - slot_enable_log_dirty: - * called when enabling log dirty mode for the slot. - * - slot_disable_log_dirty: - * called when disabling log dirty mode for the slot. - * also called when slot is created with log dirty disabled. - * - flush_log_dirty: - * called before reporting dirty_bitmap to userspace. - * - enable_log_dirty_pt_masked: - * called when reenabling log dirty for the GFNs in the mask after - * corresponding bits are cleared in slot->dirty_bitmap. + * Size of the CPU's dirty log buffer, i.e. VMX's PML buffer. A zero + * value indicates CPU dirty logging is unsupported or disabled. */ - void (*slot_enable_log_dirty)(struct kvm *kvm, - struct kvm_memory_slot *slot); - void (*slot_disable_log_dirty)(struct kvm *kvm, - struct kvm_memory_slot *slot); - void (*flush_log_dirty)(struct kvm *kvm); - void (*enable_log_dirty_pt_masked)(struct kvm *kvm, - struct kvm_memory_slot *slot, - gfn_t offset, unsigned long mask); int cpu_dirty_log_size; /* pmu operations of sub-arch */ @@ -1439,9 +1419,6 @@ void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm, struct kvm_memory_slot *memslot); void kvm_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *memslot); -void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, - struct kvm_memory_slot *slot, - gfn_t gfn_offset, unsigned long mask); void kvm_mmu_zap_all(struct kvm *kvm); void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen); unsigned long kvm_mmu_calculate_default_mmu_pages(struct kvm *kvm); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 6c32e8e0f720..86182e79beaf 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1250,9 +1250,9 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, * * Used for PML to re-log the dirty GPAs after userspace querying dirty_bitmap. */ -void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, - struct kvm_memory_slot *slot, - gfn_t gfn_offset, unsigned long mask) +static void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, + struct kvm_memory_slot *slot, + gfn_t gfn_offset, unsigned long mask) { struct kvm_rmap_head *rmap_head; @@ -1268,7 +1268,6 @@ void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, mask &= mask - 1; } } -EXPORT_SYMBOL_GPL(kvm_mmu_clear_dirty_pt_masked); /** * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected @@ -1284,10 +1283,8 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn_offset, unsigned long mask) { - if (kvm_x86_ops.enable_log_dirty_pt_masked) - static_call(kvm_x86_enable_log_dirty_pt_masked)(kvm, slot, - gfn_offset, - mask); + if (kvm_x86_ops.cpu_dirty_log_size) + kvm_mmu_clear_dirty_pt_masked(kvm, slot, gfn_offset, mask); else kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); } @@ -5616,7 +5613,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, if (flush) kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } -EXPORT_SYMBOL_GPL(kvm_mmu_slot_leaf_clear_dirty); void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm, struct kvm_memory_slot *memslot) @@ -5633,7 +5629,6 @@ void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm, if (flush) kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } -EXPORT_SYMBOL_GPL(kvm_mmu_slot_largepage_remove_write_access); void kvm_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *memslot) @@ -5649,7 +5644,6 @@ void kvm_mmu_slot_set_dirty(struct kvm *kvm, if (flush) kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } -EXPORT_SYMBOL_GPL(kvm_mmu_slot_set_dirty); void kvm_mmu_zap_all(struct kvm *kvm) { diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index f843707dd7df..862d1f5627e7 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5776,24 +5776,6 @@ static void vmx_flush_pml_buffer(struct kvm_vcpu *vcpu) vmcs_write16(GUEST_PML_INDEX, PML_ENTITY_NUM - 1); } -/* - * Flush all vcpus' PML buffer and update logged GPAs to dirty_bitmap. - * Called before reporting dirty_bitmap to userspace. - */ -static void kvm_flush_pml_buffers(struct kvm *kvm) -{ - int i; - struct kvm_vcpu *vcpu; - /* - * We only need to kick vcpu out of guest mode here, as PML buffer - * is flushed at beginning of all VMEXITs, and it's obvious that only - * vcpus running in guest are possible to have unflushed GPAs in PML - * buffer. - */ - kvm_for_each_vcpu(i, vcpu, kvm) - kvm_vcpu_kick(vcpu); -} - static void vmx_dump_sel(char *name, uint32_t sel) { pr_err("%s sel=0x%04x, attr=0x%05x, limit=0x%08x, base=0x%016lx\n", @@ -7517,32 +7499,6 @@ static void vmx_sched_in(struct kvm_vcpu *vcpu, int cpu) shrink_ple_window(vcpu); } -static void vmx_slot_enable_log_dirty(struct kvm *kvm, - struct kvm_memory_slot *slot) -{ - if (!kvm_dirty_log_manual_protect_and_init_set(kvm)) - kvm_mmu_slot_leaf_clear_dirty(kvm, slot); - kvm_mmu_slot_largepage_remove_write_access(kvm, slot); -} - -static void vmx_slot_disable_log_dirty(struct kvm *kvm, - struct kvm_memory_slot *slot) -{ - kvm_mmu_slot_set_dirty(kvm, slot); -} - -static void vmx_flush_log_dirty(struct kvm *kvm) -{ - kvm_flush_pml_buffers(kvm); -} - -static void vmx_enable_log_dirty_pt_masked(struct kvm *kvm, - struct kvm_memory_slot *memslot, - gfn_t offset, unsigned long mask) -{ - kvm_mmu_clear_dirty_pt_masked(kvm, memslot, offset, mask); -} - static int vmx_pre_block(struct kvm_vcpu *vcpu) { if (pi_pre_block(vcpu)) @@ -7749,10 +7705,6 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .sched_in = vmx_sched_in, - .slot_enable_log_dirty = vmx_slot_enable_log_dirty, - .slot_disable_log_dirty = vmx_slot_disable_log_dirty, - .flush_log_dirty = vmx_flush_log_dirty, - .enable_log_dirty_pt_masked = vmx_enable_log_dirty_pt_masked, .cpu_dirty_log_size = PML_ENTITY_NUM, .pre_block = vmx_pre_block, @@ -7897,13 +7849,8 @@ static __init int hardware_setup(void) if (!enable_ept || !enable_ept_ad_bits || !cpu_has_vmx_pml()) enable_pml = 0; - if (!enable_pml) { - vmx_x86_ops.slot_enable_log_dirty = NULL; - vmx_x86_ops.slot_disable_log_dirty = NULL; - vmx_x86_ops.flush_log_dirty = NULL; - vmx_x86_ops.enable_log_dirty_pt_masked = NULL; + if (!enable_pml) vmx_x86_ops.cpu_dirty_log_size = 0; - } if (!cpu_has_vmx_preemption_timer()) enable_preemption_timer = false; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 3fa140383f5d..e89fe98a0099 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5214,10 +5214,18 @@ static int kvm_vm_ioctl_reinject(struct kvm *kvm, void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) { + /* - * Flush potentially hardware-cached dirty pages to dirty_bitmap. + * Flush all CPUs' dirty log buffers to the dirty_bitmap. Called + * before reporting dirty_bitmap to userspace. KVM flushes the buffers + * on all VM-Exits, thus we only need to kick running vCPUs to force a + * VM-Exit. */ - static_call_cond(kvm_x86_flush_log_dirty)(kvm); + struct kvm_vcpu *vcpu; + int i; + + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_vcpu_kick(vcpu); } int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_event, @@ -10809,8 +10817,10 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, * is enabled the D-bit or the W-bit will be cleared. */ if (new->flags & KVM_MEM_LOG_DIRTY_PAGES) { - if (kvm_x86_ops.slot_enable_log_dirty) { - static_call(kvm_x86_slot_enable_log_dirty)(kvm, new); + if (kvm_x86_ops.cpu_dirty_log_size) { + if (!kvm_dirty_log_manual_protect_and_init_set(kvm)) + kvm_mmu_slot_leaf_clear_dirty(kvm, new); + kvm_mmu_slot_largepage_remove_write_access(kvm, new); } else { int level = kvm_dirty_log_manual_protect_and_init_set(kvm) ? @@ -10826,8 +10836,8 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, */ kvm_mmu_slot_remove_write_access(kvm, new, level); } - } else { - static_call_cond(kvm_x86_slot_disable_log_dirty)(kvm, new); + } else if (kvm_x86_ops.cpu_dirty_log_size) { + kvm_mmu_slot_set_dirty(kvm, new); } } From patchwork Sat Feb 13 00:50:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1DB1C43381 for ; Sat, 13 Feb 2021 00:52:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B1F4664E12 for ; Sat, 13 Feb 2021 00:52:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232346AbhBMAwd (ORCPT ); Fri, 12 Feb 2021 19:52:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232307AbhBMAwO (ORCPT ); Fri, 12 Feb 2021 19:52:14 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBA63C061222 for ; Fri, 12 Feb 2021 16:50:45 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id u17so1553132ybi.10 for ; Fri, 12 Feb 2021 16:50:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=5iYKiaJj9dDMogv04/M2zPHef+mxHxjSzdzarzVZBMA=; b=KhzN85JTB4LyOGgVmIAOxAsftbf/3k1ZB1SASfN+AvgZoI/b1RjJJFqg0IbhJ0/cFW 7UJeBWp+VvM8iczpzTL3dyqZNM6+ZudU1tEG86mnIw3xcDzxIqNWcatHxE143iXzgOIP WndlzEM7NTni7SVoomPCsq2498tc7s62Z1svZSLgyJiHyOu0rapwvPXgScNQgwYCrbI8 iLKVI2Ss8sHMdU6N+lVosYXD0AQbnHLjuUQOMLL1ZFo71o55zWdlRgwpfsk2NMlHN+4s 9MvkMl2nTc8Hvi4DkIHAaXrDO2KLGiQ6opbvPZ3493dnNxOjwHZ/gnX2JHHzSZYxzATA t3qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=5iYKiaJj9dDMogv04/M2zPHef+mxHxjSzdzarzVZBMA=; b=p/UhEzghakQorqP7RlMmn+i3mI+Gzm02nV7o5jV4PyO8pZcjV/+zlNnC8VbAI7B4oN isP67I0RMnekt7Tmtt0sEG1ofPbFgTA3sxYKq2ySZMcuPI7+NhAXt0PoeNT3oUYHbGp+ KrWM/tUXE1kSLG3XaoEp6FraY2sHA2q1SrYRRnAz5/TkW+j5AcOdow3YXAyrX+pzflBw V2SI5FeQqoomejbkwigTTT93GQgnmTVk0QzKHoiyWYlbgA4M/j6llXu/kArgerDJNMZ3 t+sHIiD1pqdqieubdpdmhWqZ4M/6XBdWbZIfWDETzbgj3TxZl2N/HVcJ+jm0lhQftQcT rraQ== X-Gm-Message-State: AOAM532NG+pKT5FoMMN8MJJALjuRnXaRx5F0/A/0rxqc4pxmoq/yL+jc eSXUOGcnvEqLJDpbk9/O1/zlCDu4gbU= X-Google-Smtp-Source: ABdhPJyF5LeQQ9Cl9wea1QgnVNkw9hArbt+rlKxJm5d2zLnjlmnkopI6K8sIbrRa4Bb4y0OkwUZemlDZgpQ= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a25:4252:: with SMTP id p79mr7480566yba.492.1613177445027; Fri, 12 Feb 2021 16:50:45 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:11 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-11-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 10/14] KVM: x86: Further clarify the logic and comments for toggling log dirty From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add a sanity check in kvm_mmu_slot_apply_flags to assert that the LOG_DIRTY_PAGES flag is indeed being toggled, and explicitly rely on that holding true when zapping collapsible SPTEs. Manipulating the CPU dirty log (PML) and write-protection also relies on this assertion, but that's not obvious in the current code. Signed-off-by: Sean Christopherson --- arch/x86/kvm/x86.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e89fe98a0099..c0d22f19aed0 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10761,12 +10761,20 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, enum kvm_mr_change change) { /* - * Nothing to do for RO slots or CREATE/MOVE/DELETE of a slot. - * See comments below. + * Nothing to do for RO slots (which can't be dirtied and can't be made + * writable) or CREATE/MOVE/DELETE of a slot. See comments below. */ if ((change != KVM_MR_FLAGS_ONLY) || (new->flags & KVM_MEM_READONLY)) return; + /* + * READONLY and non-flags changes were filtered out above, and the only + * other flag is LOG_DIRTY_PAGES, i.e. something is wrong if dirty + * logging isn't being toggled on or off. + */ + if (WARN_ON_ONCE(!((old->flags ^ new->flags) & KVM_MEM_LOG_DIRTY_PAGES))) + return; + /* * Dirty logging tracks sptes in 4k granularity, meaning that large * sptes have to be split. If live migration is successful, the guest @@ -10784,8 +10792,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, * MOVE/DELETE: The old mappings will already have been cleaned up by * kvm_arch_flush_shadow_memslot() */ - if ((old->flags & KVM_MEM_LOG_DIRTY_PAGES) && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) + if (!(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) kvm_mmu_zap_collapsible_sptes(kvm, new); /* From patchwork Sat Feb 13 00:50:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4B50C433E9 for ; Sat, 13 Feb 2021 00:52:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C971964E9D for ; Sat, 13 Feb 2021 00:52:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232395AbhBMAwt (ORCPT ); Fri, 12 Feb 2021 19:52:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232283AbhBMAwV (ORCPT ); Fri, 12 Feb 2021 19:52:21 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31D19C061224 for ; Fri, 12 Feb 2021 16:50:48 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id u1so1524289ybu.14 for ; Fri, 12 Feb 2021 16:50:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=vAUj1QEzZ3IxK3/yVqmKPFgLci8u89T6WcbRl7o0ykk=; b=v38vgtVex5oPOQ/p2ju/i7R4VJ05Z3u4THIpVVuw3QoUoGddVSK9XylTZWnIX9NmBs c9sVuGNoEdF5IL6IK1z+luuyOYQThJD0LEa45+yjFtpq55453hF1Q7LNj6/6NCGl9aeM AEiy6Gr2Irjydjw8m9IVuJSfIADSSTziGYj7trFlDUXgqxuAU/+Zz17Ix+4V13YaQWpJ 8JjXmQ2s+CYTltmHthWSU2bLn8bCxbzcg4L/5Q2ljHB7A6UuWBYMwi5f+chm87MAKd9E 2swIPO+agfjeEL/Ls1uqim5leNDQLNGVR7hmVME4jXmGm8BofGNXpACQtYakKLYSiJB6 5IQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=vAUj1QEzZ3IxK3/yVqmKPFgLci8u89T6WcbRl7o0ykk=; b=EqOHVEoSfEJBj4cTSqDnn2zDsUgOILg7fT0xqFgA/l+lKLlLWBkCqBMVP3ybUBS1Ug xR00jvQvNfOyqGWlsxcw51zwkI1Xlyk3799V2wstVEzLJKLFowxHZ5IIYVTeOHNv0bBm oBEOggp1RiqZS5GxRxzxNkHhrU1uBglv4nFdXwdfAimaD/axz3rzf9XAui4qDywtHJfc ico5EM6DXx4RiUuDI0zR66BHeH061eK3UnGCReE0eq4g+TGxEPFsHknPjuA6DmUXoxk+ OiVxpCsvnJjM3hkCYKaDYI9pnS7nscyy+jfVtIPpXYnLICP1Uf9XgXPtvqKqUMJSX0so DCLQ== X-Gm-Message-State: AOAM531adGxTOitBgwNeXw1l/H8zH+hJ3vt6jGiV/LJlBb5OhT6r9xab twT5poG7qdBmMrvJe54iAOgmgM/BCKI= X-Google-Smtp-Source: ABdhPJyVoTF6JdPZf5atU9FYPKpJaVfFucGsmKAXdWcBu9raL7IJ+KWWsssySoXqhCD6QceFfYMXuid8GD0= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a25:e752:: with SMTP id e79mr8095690ybh.373.1613177447390; Fri, 12 Feb 2021 16:50:47 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:12 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-12-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 11/14] KVM: VMX: Dynamically enable/disable PML based on memslot dirty logging From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Makarand Sonare Currently, if enable_pml=1 PML remains enabled for the entire lifetime of the VM irrespective of whether dirty logging is enable or disabled. When dirty logging is disabled, all the pages of the VM are manually marked dirty, so that PML is effectively non-operational. Setting the dirty bits is an expensive operation which can cause severe MMU lock contention in a performance sensitive path when dirty logging is disabled after a failed or canceled live migration. Manually setting dirty bits also fails to prevent PML activity if some code path clears dirty bits, which can incur unnecessary VM-Exits. In order to avoid this extra overhead, dynamically enable/disable PML when dirty logging gets turned on/off for the first/last memslot. Signed-off-by: Makarand Sonare Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/vmx/nested.c | 5 +++++ arch/x86/kvm/vmx/vmx.c | 28 +++++++++++++++++++++++- arch/x86/kvm/vmx/vmx.h | 2 ++ arch/x86/kvm/x86.c | 35 ++++++++++++++++++++++++++---- 6 files changed, 70 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index 90affdb2cbbc..323641097f63 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -93,6 +93,7 @@ KVM_X86_OP(check_intercept) KVM_X86_OP(handle_exit_irqoff) KVM_X86_OP_NULL(request_immediate_exit) KVM_X86_OP(sched_in) +KVM_X86_OP_NULL(update_cpu_dirty_logging) KVM_X86_OP_NULL(pre_block) KVM_X86_OP_NULL(post_block) KVM_X86_OP_NULL(vcpu_blocking) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 5cf382ec48b0..ffcfa84c969d 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -89,6 +89,8 @@ KVM_ARCH_REQ_FLAGS(27, KVM_REQUEST_NO_WAKEUP) #define KVM_REQ_APF_READY KVM_ARCH_REQ(28) #define KVM_REQ_MSR_FILTER_CHANGED KVM_ARCH_REQ(29) +#define KVM_REQ_UPDATE_CPU_DIRTY_LOGGING \ + KVM_ARCH_REQ_FLAGS(30, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) #define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \ @@ -1007,6 +1009,7 @@ struct kvm_arch { u32 bsp_vcpu_id; u64 disabled_quirks; + int cpu_dirty_logging_count; enum kvm_irqchip_mode irqchip_mode; u8 nr_reserved_ioapic_pins; @@ -1275,6 +1278,7 @@ struct kvm_x86_ops { * value indicates CPU dirty logging is unsupported or disabled. */ int cpu_dirty_log_size; + void (*update_cpu_dirty_logging)(struct kvm_vcpu *vcpu); /* pmu operations of sub-arch */ const struct kvm_pmu_ops *pmu_ops; diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 0c6dda9980a6..a63da447ede9 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -4493,6 +4493,11 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason, vmx_set_virtual_apic_mode(vcpu); } + if (vmx->nested.update_vmcs01_cpu_dirty_logging) { + vmx->nested.update_vmcs01_cpu_dirty_logging = false; + vmx_update_cpu_dirty_logging(vcpu); + } + /* Unpin physical memory we referred to in vmcs02 */ if (vmx->nested.apic_access_page) { kvm_release_page_clean(vmx->nested.apic_access_page); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 862d1f5627e7..1204e5f0fe67 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -4277,7 +4277,12 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx) */ exec_control &= ~SECONDARY_EXEC_SHADOW_VMCS; - if (!enable_pml) + /* + * PML is enabled/disabled when dirty logging of memsmlots changes, but + * it needs to be set here when dirty logging is already active, e.g. + * if this vCPU was created after dirty logging was enabled. + */ + if (!vcpu->kvm->arch.cpu_dirty_logging_count) exec_control &= ~SECONDARY_EXEC_ENABLE_PML; if (cpu_has_vmx_xsaves()) { @@ -7499,6 +7504,26 @@ static void vmx_sched_in(struct kvm_vcpu *vcpu, int cpu) shrink_ple_window(vcpu); } +void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vcpu) +{ + struct vcpu_vmx *vmx = to_vmx(vcpu); + + if (is_guest_mode(vcpu)) { + vmx->nested.update_vmcs01_cpu_dirty_logging = true; + return; + } + + /* + * Note, cpu_dirty_logging_count can be changed concurrent with this + * code, but in that case another update request will be made and so + * the guest will never run with a stale PML value. + */ + if (vcpu->kvm->arch.cpu_dirty_logging_count) + secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_ENABLE_PML); + else + secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_ENABLE_PML); +} + static int vmx_pre_block(struct kvm_vcpu *vcpu) { if (pi_pre_block(vcpu)) @@ -7706,6 +7731,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .sched_in = vmx_sched_in, .cpu_dirty_log_size = PML_ENTITY_NUM, + .update_cpu_dirty_logging = vmx_update_cpu_dirty_logging, .pre_block = vmx_pre_block, .post_block = vmx_post_block, diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 12c53d05a902..89da5e1251f1 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -165,6 +165,7 @@ struct nested_vmx { bool change_vmcs01_virtual_apic_mode; bool reload_vmcs01_apic_access_page; + bool update_vmcs01_cpu_dirty_logging; /* * Enlightened VMCS has been enabled. It does not mean that L1 has to @@ -393,6 +394,7 @@ int vmx_find_loadstore_msr_slot(struct vmx_msrs *m, u32 msr); void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu); void vmx_set_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type, bool value); +void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vcpu); static inline u8 vmx_get_rvi(void) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index c0d22f19aed0..b9a8c8af9713 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8987,6 +8987,9 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_check_async_pf_completion(vcpu); if (kvm_check_request(KVM_REQ_MSR_FILTER_CHANGED, vcpu)) static_call(kvm_x86_msr_filter_changed)(vcpu); + + if (kvm_check_request(KVM_REQ_UPDATE_CPU_DIRTY_LOGGING, vcpu)) + static_call(kvm_x86_update_cpu_dirty_logging)(vcpu); } if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win || @@ -10755,14 +10758,38 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, return 0; } + +static void kvm_mmu_update_cpu_dirty_logging(struct kvm *kvm, bool enable) +{ + struct kvm_arch *ka = &kvm->arch; + + if (!kvm_x86_ops.cpu_dirty_log_size) + return; + + if ((enable && ++ka->cpu_dirty_logging_count == 1) || + (!enable && --ka->cpu_dirty_logging_count == 0)) + kvm_make_all_cpus_request(kvm, KVM_REQ_UPDATE_CPU_DIRTY_LOGGING); + + WARN_ON_ONCE(ka->cpu_dirty_logging_count < 0); +} + static void kvm_mmu_slot_apply_flags(struct kvm *kvm, struct kvm_memory_slot *old, struct kvm_memory_slot *new, enum kvm_mr_change change) { + bool log_dirty_pages = new->flags & KVM_MEM_LOG_DIRTY_PAGES; + /* - * Nothing to do for RO slots (which can't be dirtied and can't be made - * writable) or CREATE/MOVE/DELETE of a slot. See comments below. + * Update CPU dirty logging if dirty logging is being toggled. This + * applies to all operations. + */ + if ((old->flags ^ new->flags) & KVM_MEM_LOG_DIRTY_PAGES) + kvm_mmu_update_cpu_dirty_logging(kvm, log_dirty_pages); + + /* + * Nothing more to do for RO slots (which can't be dirtied and can't be + * made writable) or CREATE/MOVE/DELETE of a slot. See comments below. */ if ((change != KVM_MR_FLAGS_ONLY) || (new->flags & KVM_MEM_READONLY)) return; @@ -10792,7 +10819,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, * MOVE/DELETE: The old mappings will already have been cleaned up by * kvm_arch_flush_shadow_memslot() */ - if (!(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) + if (!log_dirty_pages) kvm_mmu_zap_collapsible_sptes(kvm, new); /* @@ -10823,7 +10850,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, * initial-all-set state. Otherwise, depending on whether pml * is enabled the D-bit or the W-bit will be cleared. */ - if (new->flags & KVM_MEM_LOG_DIRTY_PAGES) { + if (log_dirty_pages) { if (kvm_x86_ops.cpu_dirty_log_size) { if (!kvm_dirty_log_manual_protect_and_init_set(kvm)) kvm_mmu_slot_leaf_clear_dirty(kvm, new); From patchwork Sat Feb 13 00:50:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51E51C433DB for ; Sat, 13 Feb 2021 00:53:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1197E64E12 for ; Sat, 13 Feb 2021 00:53:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232435AbhBMAxb (ORCPT ); Fri, 12 Feb 2021 19:53:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232388AbhBMAws (ORCPT ); Fri, 12 Feb 2021 19:52:48 -0500 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 898E1C0611C0 for ; Fri, 12 Feb 2021 16:50:50 -0800 (PST) Received: by mail-qk1-x749.google.com with SMTP id u14so985974qke.14 for ; Fri, 12 Feb 2021 16:50:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=bF14ErUVX+UBo5e1DmjEeRuMTBgvOOGaRVvf7ECwhSc=; b=F9S+sMjcAv2NeqcY+KxShiklpZpZOdR2GNLaMP/mYCzYQ6Ev13GHJkLjnqRycXrcbf i1cLnn6qwQj2lVE0QHCq19vcSSKaE9EgvpC6RvjdicYpF6H2qiCllKf2uwdpE8dktV0P ADRLh0aF429Z5/WSzJCijNmU6ImxWOBPy8efSyqKiaOjy+4NBq43HJQfu2hUgdnMykLL D/clAPgE5Ld82ovRlpXMJN/C6TL2Pj2bohzM4gJxXWPvFFrC/HLA12qlhrIZHgZteOUO hy1kdL6lS26ZWDC97pqgcD0BzpJ8EudlJE7mFu731IsddarwECQOcgV3iKkVLzjiUD4Y 310Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=bF14ErUVX+UBo5e1DmjEeRuMTBgvOOGaRVvf7ECwhSc=; b=RiBSCC7kijefGddxJnv4unt5pWcEOWNPLjShe4uGqZpWFk59zh7BQ1xCvbebii/mZN JdWq/5B1fCJwLv2tkxU2ir20jyBUHgD3Kp3kaMsE01kXMNQSX0Z7m8h+Bi2oSPZ16eWX qXiMuMDks8Ib1H/Tb9nYXTW/RldBzieYa24W1yoG/CYc9mSusRjmMTTkv7wJR2OfbG8n qVemhZBIPQ+TuwqguSkvt8veXae4wte/KR9OxxVrBkLthpPGyyVDdBryY+bB/Eg561F/ +0XX2FSJJYhMg6y9/MaLaoyzhAzYOHDoCmzJDByubFmFoKeOTPYCxfFqH3FvZ3e7soYl 0SWw== X-Gm-Message-State: AOAM530IhtWFACFIKCCqkzMOWbbu0dHxeNzgorWWaWIgNCp8V+MKpXtl I7UfCaGLb7gWEZX1ap0XXqOllKBENMc= X-Google-Smtp-Source: ABdhPJwHfC5qiRrwILSKinBtYfACaPZ6484V39/wwESfEmROgSNzj8uE29BmfLnJEPJ7XEBdVg6jWzNkXrg= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a05:6214:7cd:: with SMTP id bb13mr5161078qvb.7.1613177449760; Fri, 12 Feb 2021 16:50:49 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:13 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-13-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 12/14] KVM: x86/mmu: Don't set dirty bits when disabling dirty logging w/ PML From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Stop setting dirty bits for MMU pages when dirty logging is disabled for a memslot, as PML is now completely disabled when there are no memslots with dirty logging enabled. This means that spurious PML entries will be created for memslots with dirty logging disabled if at least one other memslot has dirty logging enabled, but for all known use cases, dirty logging is a global VMM control. Furthermore, spurious PML entries are already possible since dirty bits are set only when a dirty logging is turned off, i.e. memslots that are never dirty logged will have dirty bits cleared. In the end, it's faster overall to eat a few spurious PML entries in the window where dirty logging is being disabled across all memslots. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 2 - arch/x86/kvm/mmu/mmu.c | 45 ----------------- arch/x86/kvm/mmu/tdp_mmu.c | 54 -------------------- arch/x86/kvm/mmu/tdp_mmu.h | 1 - arch/x86/kvm/x86.c | 87 ++++++++++++++------------------- 5 files changed, 36 insertions(+), 153 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index ffcfa84c969d..c15d6de8c457 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1421,8 +1421,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, struct kvm_memory_slot *memslot); void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm, struct kvm_memory_slot *memslot); -void kvm_mmu_slot_set_dirty(struct kvm *kvm, - struct kvm_memory_slot *memslot); void kvm_mmu_zap_all(struct kvm *kvm); void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen); unsigned long kvm_mmu_calculate_default_mmu_pages(struct kvm *kvm); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 86182e79beaf..44ee55b26c3d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1181,36 +1181,6 @@ static bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, return flush; } -static bool spte_set_dirty(u64 *sptep) -{ - u64 spte = *sptep; - - rmap_printk("spte %p %llx\n", sptep, *sptep); - - /* - * Similar to the !kvm_x86_ops.slot_disable_log_dirty case, - * do not bother adding back write access to pages marked - * SPTE_AD_WRPROT_ONLY_MASK. - */ - spte |= shadow_dirty_mask; - - return mmu_spte_update(sptep, spte); -} - -static bool __rmap_set_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot) -{ - u64 *sptep; - struct rmap_iterator iter; - bool flush = false; - - for_each_rmap_spte(rmap_head, &iter, sptep) - if (spte_ad_enabled(*sptep)) - flush |= spte_set_dirty(sptep); - - return flush; -} - /** * kvm_mmu_write_protect_pt_masked - write protect selected PT level pages * @kvm: kvm instance @@ -5630,21 +5600,6 @@ void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm, kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } -void kvm_mmu_slot_set_dirty(struct kvm *kvm, - struct kvm_memory_slot *memslot) -{ - bool flush; - - write_lock(&kvm->mmu_lock); - flush = slot_handle_all_level(kvm, memslot, __rmap_set_dirty, false); - if (is_tdp_mmu_enabled(kvm)) - flush |= kvm_tdp_mmu_slot_set_dirty(kvm, memslot); - write_unlock(&kvm->mmu_lock); - - if (flush) - kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); -} - void kvm_mmu_zap_all(struct kvm *kvm) { struct kvm_mmu_page *sp, *node; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index f8fa1f64e10d..c926c6b899a1 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1268,60 +1268,6 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm, } } -/* - * Set the dirty status of all the SPTEs mapping GFNs in the memslot. This is - * only used for PML, and so will involve setting the dirty bit on each SPTE. - * Returns true if an SPTE has been changed and the TLBs need to be flushed. - */ -static bool set_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, - gfn_t start, gfn_t end) -{ - struct tdp_iter iter; - u64 new_spte; - bool spte_set = false; - - rcu_read_lock(); - - tdp_root_for_each_pte(iter, root, start, end) { - if (tdp_mmu_iter_cond_resched(kvm, &iter, false)) - continue; - - if (!is_shadow_present_pte(iter.old_spte) || - iter.old_spte & shadow_dirty_mask) - continue; - - new_spte = iter.old_spte | shadow_dirty_mask; - - tdp_mmu_set_spte(kvm, &iter, new_spte); - spte_set = true; - } - - rcu_read_unlock(); - return spte_set; -} - -/* - * Set the dirty status of all the SPTEs mapping GFNs in the memslot. This is - * only used for PML, and so will involve setting the dirty bit on each SPTE. - * Returns true if an SPTE has been changed and the TLBs need to be flushed. - */ -bool kvm_tdp_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *slot) -{ - struct kvm_mmu_page *root; - int root_as_id; - bool spte_set = false; - - for_each_tdp_mmu_root_yield_safe(kvm, root) { - root_as_id = kvm_mmu_page_as_id(root); - if (root_as_id != slot->as_id) - continue; - - spte_set |= set_dirty_gfn_range(kvm, root, slot->base_gfn, - slot->base_gfn + slot->npages); - } - return spte_set; -} - /* * Clear leaf entries which could be replaced by large mappings, for * GFNs within the slot. diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index d31c5ed81a18..3b761c111bff 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -33,7 +33,6 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, unsigned long mask, bool wrprot); -bool kvm_tdp_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *slot); void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm, struct kvm_memory_slot *slot); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b9a8c8af9713..dca2c3333ef2 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10789,7 +10789,18 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, /* * Nothing more to do for RO slots (which can't be dirtied and can't be - * made writable) or CREATE/MOVE/DELETE of a slot. See comments below. + * made writable) or CREATE/MOVE/DELETE of a slot. + * + * For a memslot with dirty logging disabled: + * CREATE: No dirty mappings will already exist. + * MOVE/DELETE: The old mappings will already have been cleaned up by + * kvm_arch_flush_shadow_memslot() + * + * For a memslot with dirty logging enabled: + * CREATE: No shadow pages exist, thus nothing to write-protect + * and no dirty bits to clear. + * MOVE/DELETE: The old mappings will already have been cleaned up by + * kvm_arch_flush_shadow_memslot(). */ if ((change != KVM_MR_FLAGS_ONLY) || (new->flags & KVM_MEM_READONLY)) return; @@ -10802,55 +10813,31 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, if (WARN_ON_ONCE(!((old->flags ^ new->flags) & KVM_MEM_LOG_DIRTY_PAGES))) return; - /* - * Dirty logging tracks sptes in 4k granularity, meaning that large - * sptes have to be split. If live migration is successful, the guest - * in the source machine will be destroyed and large sptes will be - * created in the destination. However, if the guest continues to run - * in the source machine (for example if live migration fails), small - * sptes will remain around and cause bad performance. - * - * Scan sptes if dirty logging has been stopped, dropping those - * which can be collapsed into a single large-page spte. Later - * page faults will create the large-page sptes. - * - * There is no need to do this in any of the following cases: - * CREATE: No dirty mappings will already exist. - * MOVE/DELETE: The old mappings will already have been cleaned up by - * kvm_arch_flush_shadow_memslot() - */ - if (!log_dirty_pages) + if (!log_dirty_pages) { + /* + * Dirty logging tracks sptes in 4k granularity, meaning that + * large sptes have to be split. If live migration succeeds, + * the guest in the source machine will be destroyed and large + * sptes will be created in the destination. However, if the + * guest continues to run in the source machine (for example if + * live migration fails), small sptes will remain around and + * cause bad performance. + * + * Scan sptes if dirty logging has been stopped, dropping those + * which can be collapsed into a single large-page spte. Later + * page faults will create the large-page sptes. + */ kvm_mmu_zap_collapsible_sptes(kvm, new); - - /* - * Enable or disable dirty logging for the slot. - * - * For KVM_MR_DELETE and KVM_MR_MOVE, the shadow pages of the old - * slot have been zapped so no dirty logging updates are needed for - * the old slot. - * For KVM_MR_CREATE and KVM_MR_MOVE, once the new slot is visible - * any mappings that might be created in it will consume the - * properties of the new slot and do not need to be updated here. - * - * When PML is enabled, the kvm_x86_ops dirty logging hooks are - * called to enable/disable dirty logging. - * - * When disabling dirty logging with PML enabled, the D-bit is set - * for sptes in the slot in order to prevent unnecessary GPA - * logging in the PML buffer (and potential PML buffer full VMEXIT). - * This guarantees leaving PML enabled for the guest's lifetime - * won't have any additional overhead from PML when the guest is - * running with dirty logging disabled. - * - * When enabling dirty logging, large sptes are write-protected - * so they can be split on first write. New large sptes cannot - * be created for this slot until the end of the logging. - * See the comments in fast_page_fault(). - * For small sptes, nothing is done if the dirty log is in the - * initial-all-set state. Otherwise, depending on whether pml - * is enabled the D-bit or the W-bit will be cleared. - */ - if (log_dirty_pages) { + } else { + /* + * Large sptes are write-protected so they can be split on first + * write. New large sptes cannot be created for this slot until + * the end of the logging. See the comments in fast_page_fault(). + * + * For small sptes, nothing is done if the dirty log is in the + * initial-all-set state. Otherwise, depending on whether pml + * is enabled the D-bit or the W-bit will be cleared. + */ if (kvm_x86_ops.cpu_dirty_log_size) { if (!kvm_dirty_log_manual_protect_and_init_set(kvm)) kvm_mmu_slot_leaf_clear_dirty(kvm, new); @@ -10870,8 +10857,6 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, */ kvm_mmu_slot_remove_write_access(kvm, new, level); } - } else if (kvm_x86_ops.cpu_dirty_log_size) { - kvm_mmu_slot_set_dirty(kvm, new); } } From patchwork Sat Feb 13 00:50:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A2A7C433DB for ; Sat, 13 Feb 2021 00:53:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6210064E9D for ; Sat, 13 Feb 2021 00:53:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232350AbhBMAxB (ORCPT ); Fri, 12 Feb 2021 19:53:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232323AbhBMAwX (ORCPT ); Fri, 12 Feb 2021 19:52:23 -0500 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1D2DC0611C1 for ; Fri, 12 Feb 2021 16:50:52 -0800 (PST) Received: by mail-qt1-x849.google.com with SMTP id p20so1298538qtn.23 for ; Fri, 12 Feb 2021 16:50:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=iY4gwnK7J48jZMcuGf9v6sNJyAAi1bPowIi7TdXtMyw=; b=H4WA+PJ9Fq4cpg1iosv8ji5OuoQUgpxkxqxHNFPQaq8e+qlNU0GVOWeOEA9/9dbj2O d8kcqOJbPAHYHCFu6btXnCfeg0vv9SV0YDgm69ig2j6LbOueFb1cpJ5TU272r+loQ0z+ CBWAYcVt3HPw7sQYzd8PIjgT/Kzm6Mct5wGEInUEpQ5vKPdOgzIPhgkpFPol28va4F26 8+bHm2yF9pWkV55xWL2fV8uSNO3EJlNvMYEAIXEcd1TWizNxjM8TNvhb0FlUXB4CKnva UN6mgfA2m+9XBts/tww2psDUZmKwjj+dfb+E3tTQQI7PmetqtccMDbS6ThEit3xhwON/ duvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=iY4gwnK7J48jZMcuGf9v6sNJyAAi1bPowIi7TdXtMyw=; b=ZOh70s0ckPxrmZ2yBIxnK/aDEqaeShuKAvnGkmFTr060AHQgxmJ26CBwA4fX7PMgj6 tXxGUY8/N805lHNQc3d9oURjiWtXav+Si81LHD2FYhl3dHPEjeLoeoeiBC185JOB/s8g Zd49UGDiyStkQKCj8tt6Ermja6Fa/t5VpuS0whkrxkSHcW7USih4W4qDbovblIrBJZUm OJu08h6V0u7nxHoAKWwep2FaDFyaARQMdtt1vjM6z0UTqpe3rN+MSxDPJJqbyqqFCIO7 kZ+zYEAxZ1N7v6hj/1ubMCGiI9jWPN/ywfg4tqxpQhJhgjYsnPxXTx+fAzYEkwyKO3oM TcqA== X-Gm-Message-State: AOAM533NvzDNyVVXhFeuMgKaXAXOlD0EGAtU8gfHaqofy0KggkyKPRlZ m9NxNKsFIIEQ1LBMNSd4bmd/JCFuLac= X-Google-Smtp-Source: ABdhPJzt5cZeVVu5oJFqAzLPuL/5SxpbmnCGTACPKpZBlgHirSFx1mmjdE3vTqILZOSyBX4Yp8GQzUAkcNw= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a05:6214:b11:: with SMTP id u17mr1060498qvj.50.1613177451984; Fri, 12 Feb 2021 16:50:51 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:14 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-14-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 13/14] KVM: x86: Fold "write-protect large" use case into generic write-protect From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Drop kvm_mmu_slot_largepage_remove_write_access() and refactor its sole caller to use kvm_mmu_slot_remove_write_access(). Remove the now-unused slot_handle_large_level() and slot_handle_all_level() helpers. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 32 -------------------------------- arch/x86/kvm/x86.c | 32 +++++++++++++++++--------------- 2 files changed, 17 insertions(+), 47 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 44ee55b26c3d..6ad0fb1913c6 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5204,22 +5204,6 @@ slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, lock_flush_tlb); } -static __always_inline bool -slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, - slot_level_handler fn, bool lock_flush_tlb) -{ - return slot_handle_level(kvm, memslot, fn, PG_LEVEL_4K, - KVM_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); -} - -static __always_inline bool -slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, - slot_level_handler fn, bool lock_flush_tlb) -{ - return slot_handle_level(kvm, memslot, fn, PG_LEVEL_4K + 1, - KVM_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); -} - static __always_inline bool slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot, slot_level_handler fn, bool lock_flush_tlb) @@ -5584,22 +5568,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } -void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm, - struct kvm_memory_slot *memslot) -{ - bool flush; - - write_lock(&kvm->mmu_lock); - flush = slot_handle_large_level(kvm, memslot, slot_rmap_write_protect, - false); - if (is_tdp_mmu_enabled(kvm)) - flush |= kvm_tdp_mmu_wrprot_slot(kvm, memslot, PG_LEVEL_2M); - write_unlock(&kvm->mmu_lock); - - if (flush) - kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); -} - void kvm_mmu_zap_all(struct kvm *kvm) { struct kvm_mmu_page *sp, *node; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index dca2c3333ef2..1d2bc89431a2 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10829,24 +10829,25 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, */ kvm_mmu_zap_collapsible_sptes(kvm, new); } else { - /* - * Large sptes are write-protected so they can be split on first - * write. New large sptes cannot be created for this slot until - * the end of the logging. See the comments in fast_page_fault(). - * - * For small sptes, nothing is done if the dirty log is in the - * initial-all-set state. Otherwise, depending on whether pml - * is enabled the D-bit or the W-bit will be cleared. - */ + /* By default, write-protect everything to log writes. */ + int level = PG_LEVEL_4K; + if (kvm_x86_ops.cpu_dirty_log_size) { + /* + * Clear all dirty bits, unless pages are treated as + * dirty from the get-go. + */ if (!kvm_dirty_log_manual_protect_and_init_set(kvm)) kvm_mmu_slot_leaf_clear_dirty(kvm, new); - kvm_mmu_slot_largepage_remove_write_access(kvm, new); - } else { - int level = - kvm_dirty_log_manual_protect_and_init_set(kvm) ? - PG_LEVEL_2M : PG_LEVEL_4K; + /* + * Write-protect large pages on write so that dirty + * logging happens at 4k granularity. No need to + * write-protect small SPTEs since write accesses are + * logged by the CPU via dirty bits. + */ + level = PG_LEVEL_2M; + } else if (kvm_dirty_log_manual_protect_and_init_set(kvm)) { /* * If we're with initial-all-set, we don't need * to write protect any small page because @@ -10855,8 +10856,9 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, * so that the page split can happen lazily on * the first write to the huge page. */ - kvm_mmu_slot_remove_write_access(kvm, new, level); + level = PG_LEVEL_2M; } + kvm_mmu_slot_remove_write_access(kvm, new, level); } } From patchwork Sat Feb 13 00:50:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12086461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDF59C433E9 for ; Sat, 13 Feb 2021 00:54:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9B0FB64EA6 for ; Sat, 13 Feb 2021 00:54:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232367AbhBMAxp (ORCPT ); Fri, 12 Feb 2021 19:53:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232392AbhBMAws (ORCPT ); Fri, 12 Feb 2021 19:52:48 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5F11C0611BC for ; Fri, 12 Feb 2021 16:50:54 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id v17so1539718ybq.9 for ; Fri, 12 Feb 2021 16:50:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=RgRHrMHpc3mAabRpLEPGgNZ/Y1mqc6HXCjmCsxUVhP0=; b=LGW4AnCqosd83PWIGeb5tY9knVOICydIEuP7XSipaBkqC6BBGDxe0SLoOcNH8NU4TL hEcYwRMQyFOtUTkY891A/JRUcbcc2EqqODo+4P5Mk4ByApDC7bewFyyAoBsTwgI1ARZY dNaAqq1bsK6RotAueD+dNa0+k/2rzzCpH/vs2GTszN+X9ROF7/b/55ea14sQs58Latty j/0Ozkwr1okWzg8v2qy5QwUFNAJzOEsm7ue1FU/+L1JtmitKPzfHJNl1QXIxikbzUqIv OSXDyQ11ELOCM5l8qHvhv8zdKHpGQCzgq1RAbSrmdRncMCsfA6hjbUDsEl/m8THfF01k FtEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=RgRHrMHpc3mAabRpLEPGgNZ/Y1mqc6HXCjmCsxUVhP0=; b=eTOWBfdaezun6plfpkn092zmm4T+i6leFPn388Tea4S/umVwjjzL1yHKqz6pcHLPnJ UVPb/f2NVJNJNWBEu5oYGU415wqxoYxtkX2C5cSKBjWwkLw6Dzo3K6g2lTfyYLoIxNle RsanE7YQgiRYlfybBpPT8iKb+HW5n+iZ1MajHb2JQygfiVgNQl9UmW1zY6Yr8gRnq1j4 JafmfeuRJ8lxBKDB4mFC+9pZvFSIKj6ZW/7N5XT8ZJzPLi/3RCVAx8X1EZ/QW8uv8Ttv eDD0wHWxHWgOydQlud8yZjZcMTJTHS1tP0JaABXiWhSQNvOnfPlfjY2EOOrYNjHVrOPX MuBQ== X-Gm-Message-State: AOAM530wGJFQCSqZU06ieVTlclckFBA9sIVwevkxT3WyKwmMPodnKmOb 7BjgFzYln9FKFaY+98+ZiSfrZMr1K0E= X-Google-Smtp-Source: ABdhPJxLOkH5Z7C2Gw9z1cCsrM4sF2G2OQMItCYQ3W6+OEd8alqx1gK/2aBeWdVihs3Gm++kp3akl3FutLY= Sender: "seanjc via sendgmr" X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:b407:1780:13d2:b27]) (user=seanjc job=sendgmr) by 2002:a25:c407:: with SMTP id u7mr7528207ybf.449.1613177454236; Fri, 12 Feb 2021 16:50:54 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 12 Feb 2021 16:50:15 -0800 In-Reply-To: <20210213005015.1651772-1-seanjc@google.com> Message-Id: <20210213005015.1651772-15-seanjc@google.com> Mime-Version: 1.0 References: <20210213005015.1651772-1-seanjc@google.com> X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH 14/14] KVM: x86/mmu: Remove a variety of unnecessary exports From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon , Makarand Sonare Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Remove several exports from the MMU that are no longer necessary. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 1 - arch/x86/kvm/mmu/mmu.c | 35 ++++++++++++++------------------- 2 files changed, 15 insertions(+), 21 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index c15d6de8c457..0cf71ff2b2e5 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1592,7 +1592,6 @@ void kvm_inject_nmi(struct kvm_vcpu *vcpu); void kvm_update_dr7(struct kvm_vcpu *vcpu); int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn); -int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva); void __kvm_mmu_free_some_pages(struct kvm_vcpu *vcpu); int kvm_mmu_load(struct kvm_vcpu *vcpu); void kvm_mmu_unload(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 6ad0fb1913c6..30e9b0cb9abd 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2466,7 +2466,21 @@ int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn) return r; } -EXPORT_SYMBOL_GPL(kvm_mmu_unprotect_page); + +static int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva) +{ + gpa_t gpa; + int r; + + if (vcpu->arch.mmu->direct_map) + return 0; + + gpa = kvm_mmu_gva_to_gpa_read(vcpu, gva, NULL); + + r = kvm_mmu_unprotect_page(vcpu->kvm, gpa >> PAGE_SHIFT); + + return r; +} static void kvm_unsync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) { @@ -3411,7 +3425,6 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) kvm_mmu_audit(vcpu, AUDIT_POST_SYNC); write_unlock(&vcpu->kvm->mmu_lock); } -EXPORT_SYMBOL_GPL(kvm_mmu_sync_roots); static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, gpa_t vaddr, u32 access, struct x86_exception *exception) @@ -4977,22 +4990,6 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, write_unlock(&vcpu->kvm->mmu_lock); } -int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva) -{ - gpa_t gpa; - int r; - - if (vcpu->arch.mmu->direct_map) - return 0; - - gpa = kvm_mmu_gva_to_gpa_read(vcpu, gva, NULL); - - r = kvm_mmu_unprotect_page(vcpu->kvm, gpa >> PAGE_SHIFT); - - return r; -} -EXPORT_SYMBOL_GPL(kvm_mmu_unprotect_page_virt); - int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code, void *insn, int insn_len) { @@ -5091,7 +5088,6 @@ void kvm_mmu_invalidate_gva(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, mmu->invlpg(vcpu, gva, root_hpa); } } -EXPORT_SYMBOL_GPL(kvm_mmu_invalidate_gva); void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva) { @@ -5131,7 +5127,6 @@ void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid) * for them. */ } -EXPORT_SYMBOL_GPL(kvm_mmu_invpcid_gva); void kvm_configure_mmu(bool enable_tdp, int tdp_max_root_level, int tdp_huge_page_level)