From patchwork Fri Apr 2 00:56:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12180147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F126C433ED for ; Fri, 2 Apr 2021 00:59:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3F873610E9 for ; Fri, 2 Apr 2021 00:59:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F873610E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:Reply-To:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version:Message-Id:Date: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=MozDaXi+HJyXfXQgpv24omNLQ/7CdTF/ingnmFhASJ8=; b=j+rtvU/xicm6xu5nT8K3B7aen9 649TAFsts7P9oEDCj5m/XSVNpjuDSVi6dIZcdPvj2b4ffI32pJ4yRZ6BDDIU5tYBFjqB8gnEUtMRJ MJDjAw+f0Yx7etVKf2h0yYnX3DtCH1biH11PitNfh/ieNwQFr3cchgSEEVLHP5O9JV1aXEKMkp5ns l1D7trwZHppPXza5ljxQQrdH/1+qB+SHzhfHOwtSixRRw7O/g09Y2qHWR0eOBO6MCagZtlqk/esxg QchQSsio9N87AoheFxCc7KMe3bYti+EkoZ0GMz4dBkAiYTd+wXWMHY6N00MhzMLLpqlbmGaucZFGf p6Sjukzg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lS87Z-00BYoW-OK; Fri, 02 Apr 2021 00:57:09 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lS87U-00BYm7-3i for linux-arm-kernel@lists.infradead.org; Fri, 02 Apr 2021 00:57:06 +0000 Received: by mail-yb1-xb49.google.com with SMTP id k15so1100427ybh.6 for ; Thu, 01 Apr 2021 17:57:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:message-id:mime-version:subject:from:to:cc; bh=FViJNMuAQXVG92IcJ9ASyfw+gqYDqmookinTy4APMz8=; b=tM5HW6/xk4O5sMn2fgKiWu76aEhiiETLruJpg/eDqJDR4WtSvS9+XoKICGLJgo/aR9 PHUygm3nt6Iyd9H+hRkE+aN2uWXSniHsrGAukhueCHEWKNiGLvDdvm/C35c51KL7qq/J fgt3utJuyw2N+OiFjKc36bNdeTZMCG2OXJjHEkQxFIqyXipTvjAOsKRQR3H93WpjXiTJ zZlKgCHu1/48AstcZ0TNb2v8J6l74NmyyCG68O3osE84/6JpVuFsdO2EgEGawRt1qn/R MTRQMS94V9rHBAev+i0CHpGE75U1N9vbPoIhstygJ3ud1d//b6oFND6DleWTsUfEpKQo REdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:message-id:mime-version:subject :from:to:cc; bh=FViJNMuAQXVG92IcJ9ASyfw+gqYDqmookinTy4APMz8=; b=Z0V6teOTAWqDXk+SB2//7usItHL79B9+2i6ux4JkVppO3UDpOGqv63uNkn7cHa7k/3 s0pTnj8BgdgWn1Y8Z9LCEcPpHLrgEj9HVR+cItDXMLbkVwy3EUMGbmqbX1jg9Su5AQpN bKV2FqAJIKkZ8ZLYBdDYrUPLvsOU1lxHf1q02QX4+rMKQM0jmwpaLk+hz4ASElofL4Tt fW56tedDmKNCeu6Snik/txLJr9ZI/8b40OR6/oEtDXtsx4WLjD5YOnITnm8CiZOnS/qZ X1/dsq7jdeX0TEtQHtxKA43QV9LFcV1kSn5Qw7beEOMTi5xopiuEXdl2qCKJIrbwTXOn IXSQ== X-Gm-Message-State: AOAM531JGC+6Nz9obDPTuLA6fG/BZvjYLwPocLIeUrsaZ2APht4YbWdj F1Wfw58mH47qmNkdUW15Z6G6dfV7hDE= X-Google-Smtp-Source: ABdhPJzX9kzo85wBh2DvMcR2Y9UxX/gVfIA/wY1qXyxNt7lm9J3jjA4HAyWrYbcDrQf6yDOm390BaT3pn8k= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:c0b4:8b8:bb34:6a56]) (user=seanjc job=sendgmr) by 2002:a25:a187:: with SMTP id a7mr8464267ybi.377.1617325020964; Thu, 01 Apr 2021 17:57:00 -0700 (PDT) Date: Thu, 1 Apr 2021 17:56:48 -0700 Message-Id: <20210402005658.3024832-1-seanjc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.31.0.208.g409f899ff0-goog Subject: [PATCH v2 00/10] KVM: Consolidate and optimize MMU notifiers From: Sean Christopherson To: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Paolo Bonzini Cc: James Morse , Julien Thierry , Suzuki K Poulose , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210402_015704_249614_9ABE4E13 X-CRM114-Status: GOOD ( 13.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Sean Christopherson Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The end goal of this series is to optimize the MMU notifiers to take mmu_lock if and only if the notification is relevant to KVM, i.e. the hva range overlaps a memslot. Large VMs (hundreds of vCPUs) are very sensitive to mmu_lock being taken for write at inopportune times, and such VMs also tend to be "static", e.g. backed by HugeTLB with minimal page shenanigans. The vast majority of notifications for these VMs will be spurious (for KVM), and eliding mmu_lock for spurious notifications avoids an otherwise unacceptable disruption to the guest. To get there without potentially degrading performance, e.g. due to multiple memslot lookups, especially on non-x86 where the use cases are largely unknown (from my perspective), first consolidate the MMU notifier logic by moving the hva->gfn lookups into common KVM. Based on kvm/queue, commit 5f986f748438 ("KVM: x86: dump_vmcs should include the autoload/autostore MSR lists"). Well tested on Intel and AMD. Compile tested for arm64, MIPS, PPC, PPC e500, and s390. Absolutely needs to be tested for real on non-x86, I give it even odds that I introduced an off-by-one bug somewhere. v2: - Drop the patches that have already been pushed to kvm/queue. - Drop two selftest changes that had snuck in via "git commit -a". - Add a patch to assert that mmu_notifier_count is elevated when .change_pte() runs. [Paolo] - Split out moving KVM_MMU_(UN)LOCK() to __kvm_handle_hva_range() to a separate patch. Opted not to squash it with the introduction of the common hva walkers (patch 02), as that prevented sharing code between the old and new APIs. [Paolo] - Tweak the comment in kvm_vm_destroy() above the smashing of the new slots lock. [Paolo] - Make mmu_notifier_slots_lock unconditional to avoid #ifdefs. [Paolo] v1: - https://lkml.kernel.org/r/20210326021957.1424875-1-seanjc@google.com Sean Christopherson (10): KVM: Assert that notifier count is elevated in .change_pte() KVM: Move x86's MMU notifier memslot walkers to generic code KVM: arm64: Convert to the gfn-based MMU notifier callbacks KVM: MIPS/MMU: Convert to the gfn-based MMU notifier callbacks KVM: PPC: Convert to the gfn-based MMU notifier callbacks KVM: Kill off the old hva-based MMU notifier callbacks KVM: Move MMU notifier's mmu_lock acquisition into common helper KVM: Take mmu_lock when handling MMU notifier iff the hva hits a memslot KVM: Don't take mmu_lock for range invalidation unless necessary KVM: x86/mmu: Allow yielding during MMU notifier unmap/zap, if possible arch/arm64/kvm/mmu.c | 117 +++------ arch/mips/kvm/mmu.c | 97 ++------ arch/powerpc/include/asm/kvm_book3s.h | 12 +- arch/powerpc/include/asm/kvm_ppc.h | 9 +- arch/powerpc/kvm/book3s.c | 18 +- arch/powerpc/kvm/book3s.h | 10 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 98 ++------ arch/powerpc/kvm/book3s_64_mmu_radix.c | 25 +- arch/powerpc/kvm/book3s_hv.c | 12 +- arch/powerpc/kvm/book3s_pr.c | 56 ++--- arch/powerpc/kvm/e500_mmu_host.c | 27 +- arch/x86/kvm/mmu/mmu.c | 127 ++++------ arch/x86/kvm/mmu/tdp_mmu.c | 245 +++++++------------ arch/x86/kvm/mmu/tdp_mmu.h | 14 +- include/linux/kvm_host.h | 22 +- virt/kvm/kvm_main.c | 325 +++++++++++++++++++------ 16 files changed, 552 insertions(+), 662 deletions(-) Tested-by: Marc Zyngier