From patchwork Tue Jun 11 00:21:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13692704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CAD19C27C4F for ; Tue, 11 Jun 2024 00:22:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: Mime-Version:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=nw7u3MUPcG18uClirnXW2UEhm1DI0k1qY7lBX4O6wws=; b=NOC OcdobMOesKQKg7upLg4sYIHEpjig7WsYx4dDy6E8SLJYK+CNHsqz+96izWJ/are6H/MTE3r6OV0wy jUW6ejORVgh+y+e2V2U0SdI1QUvDwLEw4O4KD8vODXx+SIqY+ZynW08R4/NxS06LolODRSFJ4/MkX IkJ7YX4dsVI00UnHpYVXEPWua+IOvGUOSihBGZ6TZp9u8kt8LSWtvKyGAn3dmfmhbiwFOHPd9Ccfi hg7ZASK21MpleN1AWVlrlCX0QY1kHmmcltlInYvffPi6m8dwfSBWxgYKWJzxBfY6JkkcD7HaZi+kl 66Vky3rpuYXRo5kLSg8bQCGPrD4F36w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sGpH4-00000006vmE-05Sp; Tue, 11 Jun 2024 00:22:06 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sGpH1-00000006vkU-0c4H for linux-arm-kernel@lists.infradead.org; Tue, 11 Jun 2024 00:22:04 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-62a0eb1c809so93689597b3.2 for ; Mon, 10 Jun 2024 17:21:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718065319; x=1718670119; darn=lists.infradead.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=znnebkeVSvsFTLm00akjOWCV2leg9gyqu/Y0G491TkM=; b=NvPKbviPYkivoT4ByaHHlsq93M56Tw29jFdo0f2mWGJBhCuqY/L+LpRs9+KzgJdHi6 AaOYTNvWvmgaTD2R9xwWvSsupBbtgnmJy9+hpJzW4RBSXAjguxYO5qNKxFNbC2IOQwBc bxOmdWbmcehZq1lzyn+MBoZvFEaK30zJ/okQ9wdwQchQ1BcTZ8PdIrrK9nI5mKVk56mp EBwOWsOern9QTU8vzmfAwCALYJqPqD0iHQZCwRvJVR51W5a98QB32zscqN3uBl/dkDHG ri6cZXzuQ4nOU/gq1qwR/0JObsOAGuxKTZrONfwvH88BXBWGfB3/+hIIO3RFpFo1XygJ ff7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718065319; x=1718670119; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=znnebkeVSvsFTLm00akjOWCV2leg9gyqu/Y0G491TkM=; b=YqeE13KQqVcjajimCyK53EjyAJ4s1J81RI4U8zd7Z1Wo4bP7sPlwLyzGxwyAK+NBWD RvZXr8wo5y+vT+Snf8sH4ZNxMHznxb+JOP5E6z4Y3TxY9RaDxPaDwD11GPXohirVmsvi NEdombNqw1rCUlsgZZSqRsWXdm8opBEEueiSGq2a6yQbieZGveOJoAtNqJGTWfbrl2nm TWO5Wc3cM1rzuxIOcHKAwrkBjPSsz9Nh4dD+bKNx5rRubOB/VDgz0es518sSiYQbeRR+ eFfJ276Acpp+fYyuwPnnfLT965w7k95v7nhJ5Zu7CBocgLQrEYwxP9pUEXX/wcOG7r/I H8Hw== X-Forwarded-Encrypted: i=1; AJvYcCVOozKNQ4FArQslU9PGrbfzw9Yz4tJ+HNBA52M6go1kN0ElD+em/1WTwOQIruqSwIAMGbJ5ukLzWbwGR57GuDqZnbSXhOIdPzR7kozc6BO7tNmt1Y4= X-Gm-Message-State: AOJu0YzwtoTYermRKCIm03XgiUKWREJmJCMH/8Z55nSFmnDubcZWwZYP mIWCaDf2kI1Z2GKhwjdZivD6EHBz3FCjfaMcfDoeBSMrr32MHpbCNobrlgOddAh1YuosdJBubfo 3lN00BURKoesV4ZpJCA== X-Google-Smtp-Source: AGHT+IHAQpzXE/JP1O7Hg4+Kq/h9j2tVw1U1SGU5I2M7dDYM3/uT/d6pwcmk2+n0+ICjilMbsMpRbG4g7tWO2ExY X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:6612:b0:62d:21:3f66 with SMTP id 00721157ae682-62d0021471dmr22261127b3.1.1718065319010; Mon, 10 Jun 2024 17:21:59 -0700 (PDT) Date: Tue, 11 Jun 2024 00:21:36 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611002145.2078921-1-jthoughton@google.com> Subject: [PATCH v5 0/9] mm: multi-gen LRU: Walk secondary MMU page tables while aging From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240610_172203_221875_B8D66FEA X-CRM114-Status: GOOD ( 20.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim. This series does the following: 1. Improve locking for the existing test/clear_young notifiers for x86 and arm64. 2. Add a new notifier test_clear_young_fast_only(), implemented only by KVM/x86. 3. Incorporate test_clear_young_fast_only() into MGLRU aging. To make aging work for more than just x86, the test_clear_young_fast_only() notifier must be implemented by those other architectures. access_tracking_perf_test now has a mode (-p) to check performance of MGLRU aging while the VM is faulting memory in. See the v4 cover letter[1] for performance data collected with this test. Previous versions of this series included logic in MGLRU and KVM to support batching the updates to secondary page tables. This version removes this logic, as it was complex and not necessary to enable proactive reclaim. This optimization, as well as the additional optimizations for arm64 and powerpc, can be done in a later series. === Previous Versions === This v5 re-adds a lot of logic that was present in v3 and earlier versions of the series. There is an important difference I want to call out: - should_look_around() can sometimes require two notifiers instead of one. This is necessary if I forbid myself from modifying mmu_notifier_clear_young(). It may simply be better to do what v2/v3 did have and not have a fast-only notifier, and merge them all. This makes the API slightly more complex. I'm not sure which is better. Change log: Since v4[1]: - Removed Kconfig that controlled when aging was enabled. Aging will be done whenever the architecture supports it (thanks Yu). - Added a new MMU notifier, test_clear_young_fast_only(), specifically for MGLRU to use. - Add kvm_fast_{test_,}age_gfn, implemented by x86. - Fix locking for clear_flush_young(). - Added KVM_MMU_NOTIFIER_YOUNG_LOCKLESS to clean up locking changes (thanks Sean). - Fix WARN_ON and other cleanup for the arm64 locking changes (thanks Oliver). Since v3[2]: - Vastly simplified the series (thanks David). Removed mmu notifier batching logic entirely. - Cleaned up how locking is done for mmu_notifier_test/clear_young (thanks David). - Look-around is now only done when there are no secondary MMUs subscribed to MMU notifiers. - CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added. - Fixed the lockless implementation of kvm_{test,}age_gfn for x86 (thanks David). - Added MGLRU functional and performance tests to access_tracking_perf_test (thanks Axel). - In v3, an mm would be completely ignored (for aging) if there was a secondary MMU but support for secondary MMU walking was missing. Now, missing secondary MMU walking support simply skips the notifier calls (except for eviction). - Added a sanity check for that range->lockless and range->on_lock are never both provided for the memslot walk. For the changes since v2[3], see v3. Based on 6.10-rc3. [1]: https://lore.kernel.org/linux-mm/20240529180510.2295118-1-jthoughton@google.com/ [2]: https://lore.kernel.org/linux-mm/20240401232946.1837665-1-jthoughton@google.com/ [3]: https://lore.kernel.org/kvmarm/20230526234435.662652-1-yuzhao@google.com/ James Houghton (8): KVM: Add lockless memslot walk to KVM KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn mm: Add test_clear_young_fast_only MMU notifier KVM: Add kvm_fast_age_gfn and kvm_fast_test_age_gfn KVM: x86: Implement kvm_fast_test_age_gfn and kvm_fast_age_gfn mm: multi-gen LRU: Have secondary MMUs participate in aging KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test Yu Zhao (1): KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +- arch/arm64/kvm/mmu.c | 26 +- arch/x86/include/asm/kvm_host.h | 14 + arch/x86/kvm/Kconfig | 2 + arch/x86/kvm/mmu.h | 6 - arch/x86/kvm/mmu/mmu.c | 60 ++- arch/x86/kvm/mmu/spte.h | 1 - arch/x86/kvm/mmu/tdp_iter.h | 27 +- arch/x86/kvm/mmu/tdp_mmu.c | 67 ++- include/linux/kvm_host.h | 8 + include/linux/mmu_notifier.h | 50 +++ include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 22 + mm/mmu_notifier.c | 26 ++ mm/rmap.c | 9 +- mm/vmscan.c | 185 +++++++-- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 365 ++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ virt/kvm/Kconfig | 7 + virt/kvm/kvm_main.c | 73 +++- 24 files changed, 1283 insertions(+), 140 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c base-commit: 83a7eefedc9b56fe7bfeff13b6c7356688ffa670