From patchwork Fri Apr 1 23:37:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12798853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E553FC433FE for ; Fri, 1 Apr 2022 23:37:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353408AbiDAXjo (ORCPT ); Fri, 1 Apr 2022 19:39:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235236AbiDAXjo (ORCPT ); Fri, 1 Apr 2022 19:39:44 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE62F39BBD for ; Fri, 1 Apr 2022 16:37:53 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id d6-20020a655886000000b00398b858cdd3so2319236pgu.7 for ; Fri, 01 Apr 2022 16:37:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=OVnrY2Z/BepLYHj2NLDSF8lUsNNOgxbKn23EFBIjPZQ=; b=iplM/xFcUsxZRgXikSbWHtSQ52UL0RsOsLJ6gpuAaIZvyF5DFDeKl+IqWUneubWMq7 mW7W1bkhqZ5x1vCZ3I5wXaBNxrDgZVWr1SL3b5wQnWV0DoPicVmoBx9a+5hhrM0sHv9U mH1zXlRxl7KA5MQ2LFx4aOSS3kbeVmR+C08Y9I4gdfDH872paQo+kBe1+8+2m5MTphqL jqi+0zKBr8UrHwXvciZ1NMeNHiWCIbZSi7sdcPRdquxEvtOy40quDJ+XwsD26xGr5cmy W6i5YpUey5sQgNI6NLznNJbUs1S213gRnS8IvwnNgZNoEOTJb5HgSImgInu/bB36+C1p AEyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=OVnrY2Z/BepLYHj2NLDSF8lUsNNOgxbKn23EFBIjPZQ=; b=FRgNbpBJy3H56d+WE7O/bYmFBceomDP83Q/UjoqQUeK60mDUij9iLzk09fv8VPHosr cwiTchIh71Ae1lBDFxwCJqgd/70v3KlCecQwMw7Uw2MNq3BwpXVHy5IMX3Ua5SvzgmDi rYV48iwB/yNAHUvA+tALNRVs+A+Db+pZnQm46Dxb2W2G3FHIjKeTQRSEoQmzS+G3tdF/ CFNNuwI+lDHeLqGbSoGTXBNt96u2ThWLLFS8C/2kCFBxfsEaYPWNLpi0hLmBdLeAO2w9 5oyb2LjLUyh9jHasZbv04j1OO/nZJMzjVLQFEHHG7u5Mr7G3i9q7/z2HdsMmnL1ZnYDn 6M3g== X-Gm-Message-State: AOAM533OtOenT/c83/H2h1lvaqVjJaT43owJS0Cy4fC/doEo5Y5HTmNc WfzUfdlqt8iGYem/YllGbj23BKqH/1deMA== X-Google-Smtp-Source: ABdhPJyaL8KicpcF2UlvW2DCewVHjdYfwlQbVdzqW9ZBHn8NnGuM0oUKo8XyTqnKVf9HZLPSUh0CN48+f9DJwA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:1791:b0:4fb:2796:83a1 with SMTP id s17-20020a056a00179100b004fb279683a1mr36726050pfg.36.1648856273268; Fri, 01 Apr 2022 16:37:53 -0700 (PDT) Date: Fri, 1 Apr 2022 23:37:35 +0000 In-Reply-To: <20220401233737.3021889-1-dmatlack@google.com> Message-Id: <20220401233737.3021889-2-dmatlack@google.com> Mime-Version: 1.0 References: <20220401233737.3021889-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.1.1094.g7c7d902a7c-goog Subject: [PATCH 1/3] KVM: selftests: Introduce a selftest to measure execution performance From: David Matlack To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Matlack , Ben Gardon , Zhenzhong Duan , "open list:KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)" , Peter Xu Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduce a new selftest, execute_perf_test, that uses the perf_test_util framework to measure the performance of executing code within a VM. This test is similar to the other perf_test_util-based tests in that it spins up a variable number of vCPUs and runs them concurrently, accessing memory. In order to support executiong, extend perf_test_util to populate guest memory with return instructions rather than random garbage. This way memory can be execute simply by calling it. Currently only x86-64 supports execution, but other architectures can be easily added by providing their return code instruction. Signed-off-by: David Matlack --- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../testing/selftests/kvm/execute_perf_test.c | 188 ++++++++++++++++++ .../selftests/kvm/include/perf_test_util.h | 2 + .../selftests/kvm/lib/perf_test_util.c | 25 ++- 5 files changed, 215 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/kvm/execute_perf_test.c base-commit: d1fb6a1ca3e535f89628193ab94203533b264c8c diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index 1f1b6c978bf7..3647ddacb103 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -56,6 +56,7 @@ /demand_paging_test /dirty_log_test /dirty_log_perf_test +/execute_perf_test /hardware_disable_test /kvm_create_max_vcpus /kvm_page_table_test diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile index c9cdbd248727..3c67346b0766 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -92,6 +92,7 @@ TEST_GEN_PROGS_x86_64 += access_tracking_perf_test TEST_GEN_PROGS_x86_64 += demand_paging_test TEST_GEN_PROGS_x86_64 += dirty_log_test TEST_GEN_PROGS_x86_64 += dirty_log_perf_test +TEST_GEN_PROGS_x86_64 += execute_perf_test TEST_GEN_PROGS_x86_64 += hardware_disable_test TEST_GEN_PROGS_x86_64 += kvm_create_max_vcpus TEST_GEN_PROGS_x86_64 += kvm_page_table_test diff --git a/tools/testing/selftests/kvm/execute_perf_test.c b/tools/testing/selftests/kvm/execute_perf_test.c new file mode 100644 index 000000000000..fa78facf44e7 --- /dev/null +++ b/tools/testing/selftests/kvm/execute_perf_test.c @@ -0,0 +1,188 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include + +#include "kvm_util.h" +#include "test_util.h" +#include "perf_test_util.h" +#include "guest_modes.h" + +/* Global variable used to synchronize all of the vCPU threads. */ +static int iteration; + +/* Set to true when vCPU threads should exit. */ +static bool done; + +/* The iteration that was last completed by each vCPU. */ +static int vcpu_last_completed_iteration[KVM_MAX_VCPUS]; + +/* Whether to overlap the regions of memory vCPUs access. */ +static bool overlap_memory_access; + +struct test_params { + /* The backing source for the region of memory. */ + enum vm_mem_backing_src_type backing_src; + + /* The amount of memory to allocate for each vCPU. */ + uint64_t vcpu_memory_bytes; + + /* The number of vCPUs to create in the VM. */ + int vcpus; +}; + +static void assert_ucall(struct kvm_vm *vm, uint32_t vcpu_id, + uint64_t expected_ucall) +{ + struct ucall uc; + uint64_t actual_ucall = get_ucall(vm, vcpu_id, &uc); + + TEST_ASSERT(expected_ucall == actual_ucall, + "Guest exited unexpectedly (expected ucall %" PRIu64 + ", got %" PRIu64 ")", + expected_ucall, actual_ucall); +} + +static bool spin_wait_for_next_iteration(int *current_iteration) +{ + int last_iteration = *current_iteration; + + do { + if (READ_ONCE(done)) + return false; + + *current_iteration = READ_ONCE(iteration); + } while (last_iteration == *current_iteration); + + return true; +} + +static void vcpu_thread_main(struct perf_test_vcpu_args *vcpu_args) +{ + struct kvm_vm *vm = perf_test_args.vm; + int vcpu_id = vcpu_args->vcpu_id; + int current_iteration = 0; + + while (spin_wait_for_next_iteration(¤t_iteration)) { + vcpu_run(vm, vcpu_id); + assert_ucall(vm, vcpu_id, UCALL_SYNC); + vcpu_last_completed_iteration[vcpu_id] = current_iteration; + } +} + +static void spin_wait_for_vcpu(int vcpu_id, int target_iteration) +{ + while (READ_ONCE(vcpu_last_completed_iteration[vcpu_id]) != + target_iteration) { + continue; + } +} + +static void run_iteration(struct kvm_vm *vm, int vcpus, const char *description) +{ + struct timespec ts_start; + struct timespec ts_elapsed; + int next_iteration; + int vcpu_id; + + /* Kick off the vCPUs by incrementing iteration. */ + next_iteration = ++iteration; + + clock_gettime(CLOCK_MONOTONIC, &ts_start); + + /* Wait for all vCPUs to finish the iteration. */ + for (vcpu_id = 0; vcpu_id < vcpus; vcpu_id++) + spin_wait_for_vcpu(vcpu_id, next_iteration); + + ts_elapsed = timespec_elapsed(ts_start); + pr_info("%-30s: %ld.%09lds\n", + description, ts_elapsed.tv_sec, ts_elapsed.tv_nsec); +} + +static void run_test(enum vm_guest_mode mode, void *arg) +{ + struct test_params *params = arg; + struct kvm_vm *vm; + int vcpus = params->vcpus; + + vm = perf_test_create_vm(mode, vcpus, params->vcpu_memory_bytes, 1, + params->backing_src, !overlap_memory_access); + + perf_test_start_vcpu_threads(vcpus, vcpu_thread_main); + + pr_info("\n"); + + perf_test_set_wr_fract(vm, 1); + run_iteration(vm, vcpus, "Populating memory"); + + perf_test_set_execute(vm, true); + run_iteration(vm, vcpus, "Executing from memory"); + + /* Set done to signal the vCPU threads to exit */ + done = true; + + perf_test_join_vcpu_threads(vcpus); + perf_test_destroy_vm(vm); +} + +static void help(char *name) +{ + puts(""); + printf("usage: %s [-h] [-m mode] [-b vcpu_bytes] [-v vcpus] [-o] [-s mem_type]\n", + name); + puts(""); + printf(" -h: Display this help message."); + guest_modes_help(); + printf(" -b: specify the size of the memory region which should be\n" + " dirtied by each vCPU. e.g. 10M or 3G.\n" + " (default: 1G)\n"); + printf(" -v: specify the number of vCPUs to run.\n"); + printf(" -o: Overlap guest memory accesses instead of partitioning\n" + " them into a separate region of memory for each vCPU.\n"); + backing_src_help("-s"); + puts(""); + exit(0); +} + +int main(int argc, char *argv[]) +{ + struct test_params params = { + .backing_src = DEFAULT_VM_MEM_SRC, + .vcpu_memory_bytes = DEFAULT_PER_VCPU_MEM_SIZE, + .vcpus = 1, + }; + int opt; + + guest_modes_append_default(); + + while ((opt = getopt(argc, argv, "hm:b:v:os:")) != -1) { + switch (opt) { + case 'm': + guest_modes_cmdline(optarg); + break; + case 'b': + params.vcpu_memory_bytes = parse_size(optarg); + break; + case 'v': + params.vcpus = atoi(optarg); + break; + case 'o': + overlap_memory_access = true; + break; + case 's': + params.backing_src = parse_backing_src_type(optarg); + break; + case 'h': + default: + help(argv[0]); + break; + } + } + + for_each_guest_mode(run_test, ¶ms); + + return 0; +} diff --git a/tools/testing/selftests/kvm/include/perf_test_util.h b/tools/testing/selftests/kvm/include/perf_test_util.h index a86f953d8d36..0a5a56539aff 100644 --- a/tools/testing/selftests/kvm/include/perf_test_util.h +++ b/tools/testing/selftests/kvm/include/perf_test_util.h @@ -33,6 +33,7 @@ struct perf_test_args { uint64_t gpa; uint64_t guest_page_size; int wr_fract; + bool execute; struct perf_test_vcpu_args vcpu_args[KVM_MAX_VCPUS]; }; @@ -46,6 +47,7 @@ struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus, void perf_test_destroy_vm(struct kvm_vm *vm); void perf_test_set_wr_fract(struct kvm_vm *vm, int wr_fract); +void perf_test_set_execute(struct kvm_vm *vm, bool execute); void perf_test_start_vcpu_threads(int vcpus, void (*vcpu_fn)(struct perf_test_vcpu_args *)); void perf_test_join_vcpu_threads(int vcpus); diff --git a/tools/testing/selftests/kvm/lib/perf_test_util.c b/tools/testing/selftests/kvm/lib/perf_test_util.c index 722df3a28791..1a5eb60b59da 100644 --- a/tools/testing/selftests/kvm/lib/perf_test_util.c +++ b/tools/testing/selftests/kvm/lib/perf_test_util.c @@ -36,6 +36,16 @@ static void (*vcpu_thread_fn)(struct perf_test_vcpu_args *); /* Set to true once all vCPU threads are up and running. */ static bool all_vcpu_threads_running; +/* + * When writing to guest memory, write the opcode for the `ret` instruction so + * that subsequent iteractions can exercise instruction fetch by calling the + * memory. + * + * NOTE: Non-x86 architectures would to use different values here to support + * execute. + */ +#define RETURN_OPCODE 0xC3 + /* * Continuously write to the first 8 bytes of each page in the * specified region. @@ -58,8 +68,10 @@ static void guest_code(uint32_t vcpu_id) for (i = 0; i < pages; i++) { uint64_t addr = gva + (i * pta->guest_page_size); - if (i % pta->wr_fract == 0) - *(uint64_t *)addr = 0x0123456789ABCDEF; + if (pta->execute) + ((void (*)(void)) addr)(); + else if (i % pta->wr_fract == 0) + *(uint64_t *)addr = RETURN_OPCODE; else READ_ONCE(*(uint64_t *)addr); } @@ -198,6 +210,15 @@ void perf_test_set_wr_fract(struct kvm_vm *vm, int wr_fract) sync_global_to_guest(vm, perf_test_args); } +void perf_test_set_execute(struct kvm_vm *vm, bool execute) +{ +#ifndef __x86_64__ + TEST_ASSERT(false, "Execute not supported on this architure; see RETURN_OPCODE."); +#endif + perf_test_args.execute = execute; + sync_global_to_guest(vm, perf_test_args); +} + static void *vcpu_thread_main(void *data) { struct vcpu_thread *vcpu = data; From patchwork Fri Apr 1 23:37:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12798854 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1198DC433F5 for ; Fri, 1 Apr 2022 23:37:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353411AbiDAXjq (ORCPT ); Fri, 1 Apr 2022 19:39:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235236AbiDAXjp (ORCPT ); Fri, 1 Apr 2022 19:39:45 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51BE939BBD for ; Fri, 1 Apr 2022 16:37:55 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id e15-20020a17090ab38f00b001c9989ae56cso2296467pjr.9 for ; Fri, 01 Apr 2022 16:37:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=/07WefWYm1wjvVb2w15UWKePj9kXzDuj1TWXAOE2EVo=; b=eYTTX5lfE6p222+w72/5uNt0nD792l5cxW4Ex9/heJZNpEjSUtlCXjCSasxn9jCEWR 1wOuc/q2/lxKJ6SO5wKU6mPF+WxOg2GvP+yO8L7bx31A/dWKRtYHsIvyzUjZthAeUmMj ch5PKnX41GoMkVLROrjTRW8B+yGgDbDw6tcnY/J9XaVoPQ1nWhYY6dN3d3roQVyzjqzZ MHL7FDykPKPFkcCIuzCwyQJC3JibB/uBcW+XjMMeOtmStAvo1yfxIYaWGq/lbfqpc+gk 6JKT45aqu3ELMNAQLjDQDV0T81WhPdQA77Dig0b6TtNAVk0CK0BZQ80lSVOnbW2rCnin KPaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/07WefWYm1wjvVb2w15UWKePj9kXzDuj1TWXAOE2EVo=; b=JJJVnjvwQbdloNYbDGrcJkbm8cp2uXp4zVJFLwZ1fOu5vcxXZu51Z3W+LIcf4JjCbj XYqzHb1S1uRo6eg56Y9PK1E4sv0OpBW/6qoPvRysroysThRTk+qvuHwocA6ZuEGPxHbc OZ+ZF9OViJQAgEW+8iQxnOw9WBALDF+g8rSvFFtc6Y2IyrBgwuUMn49STK0GxoFopH3k AIJQkEmxKORzEdfYIwZg78+c5qDzC0q/Ul7pZ3Sl2RRzd7i+NK0jOly8Pk43FiE2qlet kSv64RrvTpk0xy5NZ3C8t04QJAZ1V1fGWUIEXhMM8Ikz48hu7H3V/6fG9H6JB+yfbWF8 jqZg== X-Gm-Message-State: AOAM531CzvaInX95kXa7CJdLuTm9J1hSsJL1wmb7ZNPtEQdVeR19fEBG 6z6Nr9N3pu8KJZPaRpY+8yuAF+qmSNY1yw== X-Google-Smtp-Source: ABdhPJxHQibJi37gdIS76x91BNH9kBJAg7rh3O7oMRDnpRvtI6uIWHoKF9/gODtt/fBp29LCwEOHm6G4yFajaw== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:3316:b0:4fa:80fd:f3f6 with SMTP id cq22-20020a056a00331600b004fa80fdf3f6mr13133642pfb.65.1648856274826; Fri, 01 Apr 2022 16:37:54 -0700 (PDT) Date: Fri, 1 Apr 2022 23:37:36 +0000 In-Reply-To: <20220401233737.3021889-1-dmatlack@google.com> Message-Id: <20220401233737.3021889-3-dmatlack@google.com> Mime-Version: 1.0 References: <20220401233737.3021889-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.1.1094.g7c7d902a7c-goog Subject: [PATCH 2/3] KVM: x86/mmu: Pass account_nx to tdp_mmu_split_huge_page() From: David Matlack To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Matlack , Ben Gardon , Zhenzhong Duan , "open list:KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)" , Peter Xu Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In preparation for splitting huge pages during fault, pass account_nx to tdp_mmu_split_huge_page(). Eager page splitting hard-codes account_nx to false because the splitting is being done for dirty-logging rather than vCPU execution faults. No functional change intended. Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index d71d177ae6b8..9263765c8068 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1456,7 +1456,8 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm, } static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_mmu_page *sp, bool shared) + struct kvm_mmu_page *sp, bool shared, + bool account_nx) { const u64 huge_spte = iter->old_spte; const int level = iter->level; @@ -1479,7 +1480,7 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, * correctness standpoint since the translation will be the same either * way. */ - ret = tdp_mmu_link_sp(kvm, iter, sp, false, shared); + ret = tdp_mmu_link_sp(kvm, iter, sp, account_nx, shared); if (ret) goto out; @@ -1539,7 +1540,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, continue; } - if (tdp_mmu_split_huge_page(kvm, &iter, sp, shared)) + if (tdp_mmu_split_huge_page(kvm, &iter, sp, shared, false)) goto retry; sp = NULL; From patchwork Fri Apr 1 23:37:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12798855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDE7CC433EF for ; Fri, 1 Apr 2022 23:37:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353412AbiDAXjr (ORCPT ); Fri, 1 Apr 2022 19:39:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235236AbiDAXjr (ORCPT ); Fri, 1 Apr 2022 19:39:47 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE1965D658 for ; Fri, 1 Apr 2022 16:37:56 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id n17-20020a17090ac69100b001c77ebd900fso2299996pjt.8 for ; Fri, 01 Apr 2022 16:37:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=6MhzfPPt8Lt29HW2soehUAhsaF23NR3MiZKU9lPVNiE=; b=JiGiVMwwkkhZXltJIgFMjp6avFgKy8hlW9bb4ck60qGHGRLGswJHKL4ortZYinzLjy /2qHkjPtIRhaBcunooYefX1iSWF4OUaF+YLrJc7ZsnHvBMHqONipoGz1SWchu0WG8err 96sXT0tWOX8v9PuMQoSlTgq+m6OAO4uBuADV8H9Y2K6qbXAQ/aiTxKC4Bnbni0G4y1H8 r0NztMXfhqF3Q+MDX+Xp8tjlAKVlnco1RDZSjUMJoiw+Xls0+8uOSC9GgEngeFj/uVSt u68qbs+8/JbubO7tzDnasy0fCSz+GM996W3B3rnKs38uGxkJlP5ue4s81reJXHXDQd47 OpgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=6MhzfPPt8Lt29HW2soehUAhsaF23NR3MiZKU9lPVNiE=; b=eWguVnn9tflc3Avid4gUQ9kOxrHX2q2kEpPPOxxmbbO4Rt4N2W+XZyCB6gPvnf5Dwc lbUtbuW5tBM3dQq1hz8kWYAqSCJQcvmiI7XMJlan8H9zqn2XMSdgMSUX2cuPtMfgbLp1 DI6S4Ua252Gy0ZKyVbLrTQB9vFhal5McWeTg+6p5/TWHIEFcJn3MUorVdsSwhTh0162D I3NCFA7u/L2yo7fLmuoo6UGWCBFCjAoAsRxjjZDcofrsCN2gqOX9Gbm3TpNKEnPsaEWL 6HmACA3bk3oixlN9kGWQzSEePPMx865QlNCJcsYDmpfigTIw7leJztYRK2oMAFmzvhjw dLiA== X-Gm-Message-State: AOAM531bjL/ZVrVX9GRpWF+y6nRJQ9YwYhKru4+ZJpotUhYisp+zBKgC hTAu6rz+qKo7fQjVSqbwXBI8tByU32PkFg== X-Google-Smtp-Source: ABdhPJy2ANvpOoAUmT4XdfoUfQaWDjFLt4xjfKMH+bV8eDlkwuh0sE6FuM+8GgAJMP+6f8DmwCBkJekAZjmNhw== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:ce87:b0:156:5c6e:b6e4 with SMTP id f7-20020a170902ce8700b001565c6eb6e4mr9077706plg.12.1648856276308; Fri, 01 Apr 2022 16:37:56 -0700 (PDT) Date: Fri, 1 Apr 2022 23:37:37 +0000 In-Reply-To: <20220401233737.3021889-1-dmatlack@google.com> Message-Id: <20220401233737.3021889-4-dmatlack@google.com> Mime-Version: 1.0 References: <20220401233737.3021889-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.1.1094.g7c7d902a7c-goog Subject: [PATCH 3/3] KVM: x86/mmu: Split huge pages mapped by the TDP MMU on fault From: David Matlack To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , David Matlack , Ben Gardon , Zhenzhong Duan , "open list:KERNEL VIRTUAL MACHINE FOR X86 (KVM/x86)" , Peter Xu Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that the TDP MMU has a mechanism to split huge pages, use it in the fault path when a huge page needs to be replaced with a mapping at a lower level. This change reduces the negative performance impact of NX HugePages. Prior to this change if a vCPU executed from a huge page and NX HugePages was enabled, the vCPU would take a fault, zap the huge page, and mapping the faulting address at 4KiB with execute permissions enabled. The rest of the memory would be left *unmapped* and have to be faulted back in by the guest upon access (read, write, or execute). If guest is backed by 1GiB, a single execute instruction can zap an entire GiB of its physical address space. For example, it can take a VM longer to execute from its memory than to populate that memory in the first place: $ ./execute_perf_test -s anonymous_hugetlb_1gb -v96 Populating memory : 2.748378795s Executing from memory : 2.899670885s With this change, such faults split the huge page instead of zapping it, which avoids the non-present faults on the rest of the huge page: $ ./execute_perf_test -s anonymous_hugetlb_1gb -v96 Populating memory : 2.729544474s Executing from memory : 0.111965688s <--- This change also reduces the performance impact of dirty logging when eager_page_split=N for the same reasons as above but write faults. eager_page_split=N (abbreviated "eps=N" below) can be desirable for read-heavy workloads, as it avoids allocating memory to split huge pages that are never written and avoids increasing the TLB miss cost on reads of those pages. | Config: ept=Y, tdp_mmu=Y, 5% writes | | Iteration 1 dirty memory time | | --------------------------------------------- | vCPU Count | eps=N (Before) | eps=N (After) | eps=Y | ------------ | -------------- | ------------- | ------------ | 2 | 0.332305091s | 0.019615027s | 0.006108211s | 4 | 0.353096020s | 0.019452131s | 0.006214670s | 8 | 0.453938562s | 0.019748246s | 0.006610997s | 16 | 0.719095024s | 0.019972171s | 0.007757889s | 32 | 1.698727124s | 0.021361615s | 0.012274432s | 64 | 2.630673582s | 0.031122014s | 0.016994683s | 96 | 3.016535213s | 0.062608739s | 0.044760838s | Eager page splitting remains beneficial for write-heavy workloads, but the gap is now reduced. | Config: ept=Y, tdp_mmu=Y, 100% writes | | Iteration 1 dirty memory time | | --------------------------------------------- | vCPU Count | eps=N (Before) | eps=N (After) | eps=Y | ------------ | -------------- | ------------- | ------------ | 2 | 0.317710329s | 0.296204596s | 0.058689782s | 4 | 0.337102375s | 0.299841017s | 0.060343076s | 8 | 0.386025681s | 0.297274460s | 0.060399702s | 16 | 0.791462524s | 0.298942578s | 0.062508699s | 32 | 1.719646014s | 0.313101996s | 0.075984855s | 64 | 2.527973150s | 0.455779206s | 0.079789363s | 96 | 2.681123208s | 0.673778787s | 0.165386739s | Further study is needed to determine if the remaining gap is acceptable for customer workloads or if eager_page_split=N still requires a-priori knowledge of the VM workload, especially when considering these costs extrapolated out to large VMs with e.g. 416 vCPUs and 12TB RAM. Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 9263765c8068..5a2120d85347 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1131,6 +1131,10 @@ static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, return 0; } +static int tdp_mmu_split_huge_page_atomic(struct kvm_vcpu *vcpu, + struct tdp_iter *iter, + bool account_nx); + /* * Handle a TDP page fault (NPT/EPT violation/misconfiguration) by installing * page tables and SPTEs to translate the faulting guest physical address. @@ -1140,6 +1144,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) struct kvm_mmu *mmu = vcpu->arch.mmu; struct tdp_iter iter; struct kvm_mmu_page *sp; + bool account_nx; int ret; kvm_mmu_hugepage_adjust(vcpu, fault); @@ -1155,28 +1160,22 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (iter.level == fault->goal_level) break; + account_nx = fault->huge_page_disallowed && + fault->req_level >= iter.level; + /* * If there is an SPTE mapping a large page at a higher level - * than the target, that SPTE must be cleared and replaced - * with a non-leaf SPTE. + * than the target, split it down one level. */ if (is_shadow_present_pte(iter.old_spte) && is_large_pte(iter.old_spte)) { - if (tdp_mmu_zap_spte_atomic(vcpu->kvm, &iter)) + if (tdp_mmu_split_huge_page_atomic(vcpu, &iter, account_nx)) break; - /* - * The iter must explicitly re-read the spte here - * because the new value informs the !present - * path below. - */ - iter.old_spte = kvm_tdp_mmu_read_spte(iter.sptep); + continue; } if (!is_shadow_present_pte(iter.old_spte)) { - bool account_nx = fault->huge_page_disallowed && - fault->req_level >= iter.level; - /* * If SPTE has been frozen by another thread, just * give up and retry, avoiding unnecessary page table @@ -1496,6 +1495,20 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, return ret; } +static int tdp_mmu_split_huge_page_atomic(struct kvm_vcpu *vcpu, + struct tdp_iter *iter, + bool account_nx) +{ + struct kvm_mmu_page *sp = tdp_mmu_alloc_sp(vcpu); + int r; + + r = tdp_mmu_split_huge_page(vcpu->kvm, iter, sp, true, account_nx); + if (r) + tdp_mmu_free_sp(sp); + + return r; +} + static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t start, gfn_t end,