From patchwork Mon Apr 1 23:29:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613105 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 851FE56B91 for ; Mon, 1 Apr 2024 23:29:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712014199; cv=none; b=lVX7ZAZoYZVeTjz/bjbK+0lWl7jB1WKaos6JYPjPCKFhnGPz2zd6sR9djAbhG3tkNCmmfLXXIq4+cEZzLdWP393jNS0qKNXLS3t2MxwQTDqqWZMTynTyCTFYGqi0NAC8GRymC+sBO4kMeEeFXIWapzP3K5BkrydeXBOPW3ls8yU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712014199; c=relaxed/simple; bh=LIU3srPkuAU2Il0Ayu4Wz3ZF0a3r6V03GyttaxoyE9Q=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=WAKSLWHGjbbCcaL2cgaznW14qdGZEAcmhLGU8lyhwn2NxFa/p7M1WllSRGHlDnAQprARgEuTGFzcwITD2t7q8KXJY0+aeavmfTbEHFbAXXkVq5qXXRGhD4N8c0bbPfHHpcX3sH8g3XlsNpJ4iybeL5QkUGvOWMKgWyUkDfYthM8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rcUf3/3H; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rcUf3/3H" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a61b31993so81552277b3.1 for ; Mon, 01 Apr 2024 16:29:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014196; x=1712618996; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=3wg9CCGpIdLpfDRqh60QLQofCQe0biAtvSj1Gdo5bk8=; b=rcUf3/3Hx79SQpAY7DJI+A7c4PJxcLpRh9bjArXZWq5RmH8UM59RZPPelNnD82Zs2v UYJAGIA2LNKy5g0kCM+Tdp+TrzmMGXUIfhOkF4TC95s0ATZG3/vMyIwARciwcG/O/gmF 6+X9vHutCcV5cnN+YoSUptFqxNDv85n87KX4kICeyLKyMYwtkQRGJLxOClKGqzneIe0H qUlvjbHRsiBu8lP4wG+NsScN5NednQGAv8Yk1N7Qe+Ml6hd7+M0z4KxtuvJlAZPied4s GwOQQd1hW1LyUQqaybsLrbpsRpdhUUjWidmwDeflXEnhFB+OpHvNA55KUD966PLllaKH g2UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014196; x=1712618996; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=3wg9CCGpIdLpfDRqh60QLQofCQe0biAtvSj1Gdo5bk8=; b=Ae4eafFkth165AAEFLY3GxTiPuq/JPP8jCDpcr69CklnkwKrZHIvQUOFsDY5Ebl8l0 zdUSsQjox/KK/RbRW/JXzMG9uXmKy//fLDSft9mNXkHu9ekwgcFAFBlmZw3rHd78byPK 64f1FHpnQs0LevKy+0me+hCMmd5phEM4V600f9aqPcQTTlHFFGjCaBFN34z91427unDg 87HVF92DucB7GPrDw9vtBEwEFm7sTk/RxNHmvl9cilgJpRCxV/GdZLiVBz//l94u7o8f gZX0FyJMsL5qqAAubQHanu8VG7jBpIYQP/7o3Tah0UWuoFaW89xsqZGp+1vlHcbTDLQi aEgA== X-Forwarded-Encrypted: i=1; AJvYcCVjQKz+o6U2AYkCCUPcp6Ph92j1kJOKDKeEtXjI5fVIOK7jW5BcdGCyWKwYRipFbyQsPPErnVD3sYamT3PE/CfsOzb+XX21cW9RGASDrdiOqN1B X-Gm-Message-State: AOJu0YwJSDOnTli+DjVjXxDy0T7YlGChqStmILUR1cnhLuSrVtsto1dx PTC3KRAPszd+cw/YmhaIRQrE9fZFQBqeXm2YQ75PPAlT7bvnJJzclL9fIzwSns5IscH/Vjj+ebj 8yfBo6ok6LyBGVonQoQ== X-Google-Smtp-Source: AGHT+IGQKV963bGOmZkg1qJdb4wkJ3JL7Okh/SQUhgB3vOnWcX/gauefdkact4kTuSTRZOisDp+cn6EF/9Mo2eKk X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:f83:b0:614:e20c:d423 with SMTP id df3-20020a05690c0f8300b00614e20cd423mr823185ywb.10.1712014196503; Mon, 01 Apr 2024 16:29:56 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:39 +0000 Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-1-jthoughton@google.com> Subject: [PATCH v3 0/7] mm/kvm: Improve parallelism for access bit harvesting From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton This patchset adds a fast path in KVM to test and clear access bits on sptes without taking the mmu_lock. It also adds support for using a bitmap to (1) test the access bits for many sptes in a single call to mmu_notifier_test_young, and to (2) clear the access bits for many ptes in a single call to mmu_notifier_clear_young. With Yu's permission, I'm now working on getting this series into a mergeable state. I'm posting this as an RFC because I'm not sure if the arm64 bits are correct, and I haven't done complete performance testing. I want to do broader experimentation to see how much this improves VM performance in a cloud environment, but I want to be sure that the code is mergeable first. Yu has posted other performance results[1], [2]. This v3 shouldn't significantly change the x86 results, but the arm64 results may have changed. The most important changes since v2[3]: - Split the test_clear_young MMU notifier back into test_young and clear_young. I did this because the bitmap passed in has a distinct meaning for each of them, and I felt that this was cleaner. - The return value of test_young / clear_young now indicates if the bitmap was used. - Removed the custom spte walker to implement the lockless path. This was important for arm64 to be functionally correct (thanks Oliver), and it avoids a lot of problems brought up in review of v2 (for example[4]). - Add kvm_arch_prepare_bitmap_age and kvm_arch_finish_bitmap_age to allow for arm64 to implement its bitmap-based aging to grab the MMU lock for reading while allowing x86 to be lockless. - The powerpc changes have been dropped. - The logic to inform architectures how to use the bitmap has been cleaned up (kvm_should_clear_young has been split into kvm_gfn_should_age and kvm_gfn_record_young) (thanks Nicolas). There were some smaller changes too: - Added test_clear_young_metadata (thanks Sean). - MMU_NOTIFIER_RANGE_LOCKLESS has been renamed to MMU_NOTIFIER_YOUNG_FAST, to indicate to the caller that passing a bitmap for MGLRU look-around is likely to be beneficial. - Cleaned up comments that describe the changes to mmu_notifier_test_young / mmu_notifier_clear_young (thanks Nicolas). [1]: https://lore.kernel.org/all/20230609005943.43041-1-yuzhao@google.com/ [2]: https://lore.kernel.org/all/20230609005935.42390-1-yuzhao@google.com/ [3]: https://lore.kernel.org/kvmarm/20230526234435.662652-1-yuzhao@google.com/ [4]: https://lore.kernel.org/all/ZItX64Bbx5vdjo9M@google.com/ James Houghton (5): mm: Add a bitmap into mmu_notifier_{clear,test}_young KVM: Move MMU notifier function declarations KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young KVM: x86: Participate in bitmap-based PTE aging KVM: arm64: Participate in bitmap-based PTE aging Yu Zhao (2): KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask mm: multi-gen LRU: use mmu_notifier_test_clear_young() Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/include/asm/kvm_host.h | 5 + arch/arm64/include/asm/kvm_pgtable.h | 4 +- arch/arm64/kvm/hyp/pgtable.c | 21 +- arch/arm64/kvm/mmu.c | 23 ++- arch/x86/include/asm/kvm_host.h | 20 ++ arch/x86/kvm/mmu.h | 6 - arch/x86/kvm/mmu/mmu.c | 16 +- arch/x86/kvm/mmu/spte.h | 1 - arch/x86/kvm/mmu/tdp_mmu.c | 10 +- include/linux/kvm_host.h | 101 ++++++++-- include/linux/mmu_notifier.h | 93 ++++++++- include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 13 +- mm/mmu_notifier.c | 20 +- mm/rmap.c | 9 +- mm/vmscan.c | 183 ++++++++++++++---- virt/kvm/kvm_main.c | 100 +++++++--- 18 files changed, 509 insertions(+), 128 deletions(-) base-commit: 0cef2c0a2a356137b170c3cb46cb9c1dd2ca3e6b