From patchwork Sat Feb 10 23:39:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Woodhouse, David" X-Patchwork-Id: 10210617 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DF9E760236 for ; Sat, 10 Feb 2018 23:40:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4EB228841 for ; Sat, 10 Feb 2018 23:40:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C952D28935; Sat, 10 Feb 2018 23:40:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 697E428841 for ; Sat, 10 Feb 2018 23:40:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752947AbeBJXjt (ORCPT ); Sat, 10 Feb 2018 18:39:49 -0500 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:12571 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752896AbeBJXjr (ORCPT ); Sat, 10 Feb 2018 18:39:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.co.uk; i=@amazon.co.uk; q=dns/txt; s=amazon201209; t=1518305987; x=1549841987; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=oDZcgkx/CPhCYrDzSEVJ5sYO7s1ZWuclLYBOLrcoKOc=; b=MyXqzBYqSsrarBOePOz9Y/UIn7rKfmNUmMUDLqF1IDp+3Ne7LZmPDhYg 0AD0Fbqqhw9nbOOoBLvEmYibONHrLXJzvmiP+ZTOmJWs9PFSmEIRp4zh9 NPAB6pz3ZhJ0rkecv9NrmPXWx0uCI21984sYM2sHNj0IYc5sAW9vDIKrt Y=; X-IronPort-AV: E=Sophos;i="5.46,492,1511827200"; d="scan'208";a="722034696" Received: from sea3-co-svc-lb6-vlan3.sea.amazon.com (HELO email-inbound-relay-2b-1171e197.us-west-2.amazon.com) ([10.47.22.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 10 Feb 2018 23:39:46 +0000 Received: from uc8d3ff76b9bc5848a9cc.ant.amazon.com (pdx2-ws-svc-lb17-vlan2.amazon.com [10.247.140.66]) by email-inbound-relay-2b-1171e197.us-west-2.amazon.com (8.14.7/8.14.7) with ESMTP id w1ANdgp1048557 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 10 Feb 2018 23:39:44 GMT Received: from uc8d3ff76b9bc5848a9cc.ant.amazon.com (localhost [127.0.0.1]) by uc8d3ff76b9bc5848a9cc.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTP id w1ANdfmt031605; Sat, 10 Feb 2018 23:39:41 GMT Received: (from dwmw@localhost) by uc8d3ff76b9bc5848a9cc.ant.amazon.com (8.15.2/8.15.2/Submit) id w1ANdfFL031604; Sat, 10 Feb 2018 23:39:41 GMT From: David Woodhouse To: tglx@linutronix.de, karahmed@amazon.de, sironi@amazon.de, x86@kernel.org, kvm@vger.kernel.org, torvalds@linux-foundation.org, pbonzini@redhat.com, linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, jmattson@google.com, rkrcmar@redhat.com, arjan.van.de.ven@intel.com, dave.hansen@intel.com Subject: [PATCH v2 3/6] KVM: x86: Reduce retpoline performance impact in slot_handle_level_range() Date: Sat, 10 Feb 2018 23:39:24 +0000 Message-Id: <1518305967-31356-4-git-send-email-dwmw@amazon.co.uk> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1518305967-31356-1-git-send-email-dwmw@amazon.co.uk> References: <1518305967-31356-1-git-send-email-dwmw@amazon.co.uk> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP With retpoline, tight loops of "call this function for every XXX" are very much pessimised by taking a prediction miss *every* time. This one is by far the biggest contributor to the guest launch time with retpoline. By marking the iterator slot_handle_…() functions always_inline, we can ensure that the indirect function call can be optimised away into a direct call and it actually generates slightly smaller code because some of the other conditionals can get optimised away too. Performance is now pretty close to what we see with nospectre_v2 on the command line. Suggested-by: Linus Torvalds Signed-off-by: David Woodhouse Reviewed-by: Filippo Sironi Tested-by: Filippo Sironi --- arch/x86/kvm/mmu.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index 2b8eb4d..cc83bdc 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -5058,7 +5058,7 @@ void kvm_mmu_uninit_vm(struct kvm *kvm) typedef bool (*slot_level_handler) (struct kvm *kvm, struct kvm_rmap_head *rmap_head); /* The caller should hold mmu-lock before calling this function. */ -static bool +static __always_inline bool slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, slot_level_handler fn, int start_level, int end_level, gfn_t start_gfn, gfn_t end_gfn, bool lock_flush_tlb) @@ -5088,7 +5088,7 @@ slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot, return flush; } -static bool +static __always_inline bool slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, slot_level_handler fn, int start_level, int end_level, bool lock_flush_tlb) @@ -5099,7 +5099,7 @@ slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, lock_flush_tlb); } -static bool +static __always_inline bool slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, slot_level_handler fn, bool lock_flush_tlb) { @@ -5107,7 +5107,7 @@ slot_handle_all_level(struct kvm *kvm, struct kvm_memory_slot *memslot, PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); } -static bool +static __always_inline bool slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, slot_level_handler fn, bool lock_flush_tlb) { @@ -5115,7 +5115,7 @@ slot_handle_large_level(struct kvm *kvm, struct kvm_memory_slot *memslot, PT_MAX_HUGEPAGE_LEVEL, lock_flush_tlb); } -static bool +static __always_inline bool slot_handle_leaf(struct kvm *kvm, struct kvm_memory_slot *memslot, slot_level_handler fn, bool lock_flush_tlb) {