From patchwork Thu Sep 14 01:55:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13384015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B730EEE0212 for ; Thu, 14 Sep 2023 01:56:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09A086B0299; Wed, 13 Sep 2023 21:55:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 04ECF6B029D; Wed, 13 Sep 2023 21:55:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDFD46B02A0; Wed, 13 Sep 2023 21:55:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CB6636B0299 for ; Wed, 13 Sep 2023 21:55:54 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A7F8D40E86 for ; Thu, 14 Sep 2023 01:55:54 +0000 (UTC) X-FDA: 81233537028.17.724F8F3 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf09.hostedemail.com (Postfix) with ESMTP id DE8BE140003 for ; Thu, 14 Sep 2023 01:55:52 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0JbeIQyc; spf=pass (imf09.hostedemail.com: domain of 3KGgCZQYKCDEfRNaWPTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--seanjc.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3KGgCZQYKCDEfRNaWPTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--seanjc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694656552; a=rsa-sha256; cv=none; b=7zcNNRM0XEqgT3FWwgnHETvDjxoMcQ60x3q+HAQRu1FztlLxGaV/+LfJHSxUytYl43OtRR DJo4wtTfaNKYByegXCwaxoriNd9w3QmKvX1pPWBRwaohJD0e1+WtiOHWd9VVhG21BI9+XM N6Cky9ljwKkmz2RxGFkjxfCvq84f+BQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0JbeIQyc; spf=pass (imf09.hostedemail.com: domain of 3KGgCZQYKCDEfRNaWPTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--seanjc.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3KGgCZQYKCDEfRNaWPTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--seanjc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694656552; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c0gbqnW9KUovCg+coU7ldlYhYVFXBRbmaH2jYlXJSe8=; b=0B7wVgW0HV9chiIy4/5Eidms4YCUylXnTiU9hARoJqEJx6INWz6UgEtSVhSPmGsEWAoxEf yFyMJCrZtyDrhbb3JwS2qN9C8tgnh6M1WcZ4yFOv38mnCp9A3x5otmvXgatjDxa4FNnvIU a0YbFZwlw1hDUtUhM6vub8S3bYDPZiw= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-59b56dab74bso6570877b3.2 for ; Wed, 13 Sep 2023 18:55:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694656552; x=1695261352; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=c0gbqnW9KUovCg+coU7ldlYhYVFXBRbmaH2jYlXJSe8=; b=0JbeIQycHs7BZ2aCSIcNnvfrqMKRnixwKzzFQdOfw0tlcEtJS3gTqFDrmeiI4daCfG Zi03vQ+WaDZDOBb9k9kmUdaXiFli0QkmfHTRYYlhkjB/+9Hcc2Wifx+yd+OawzIieLEZ m6g3dryGNLHGKL6e57O0+70PG4pgYlCrR7Dba21c8f7BK9itG870O5b1ojujcAs1qgOE H2l3ZQiBhMtUu5agu9TVkkMhx6rF8Fkeb8YFu3lx6twwWzE0wmxkRbGVXywWX6vaVKsQ JBHTyemjHw7K1cSgCpMWTYOU4FBLIhMVi5/N7byl1i4rqL18aLtj4aAPaZeNebKB+R3G bPPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694656552; x=1695261352; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=c0gbqnW9KUovCg+coU7ldlYhYVFXBRbmaH2jYlXJSe8=; b=YmQ7/6XGcQO9CMlLfFIxk5oCdmJKJ7mTaquAqfTbZedPn146kGZd8ySXquQg7CCWnW QyaTRRJ9dumUanXPeKrK6SjER+h2YADjE50T3IwDL1KGD35Gr3jbSdB3T43BC/t0aBH2 oXo4WJrmW8lGUzN00v9ST01+i3+ym2/1Jcu8hZg+0pg+Qqfu87zyjud+4WoUpoCcAzz1 I63FAs4h7nrJ+4oS6dK8oXxpVe4bQSo6BqIl6c431OCnatLXTF75EeeyZDChtDvVJEAV EApm/TWFgcPqfkt0RUF0a8M2CaIR1mdYjWRTtbliCda2LHpJEvtyoaUGRlZl3K4dCzB3 Pp3A== X-Gm-Message-State: AOJu0Yxv9hb6wqA6V1ut40Rc8XCzIEUTr9a+bFM8+yeAKnfyw/b7/7ze n+0WphV59Z85W0WneBeWDiquZCRLKWk= X-Google-Smtp-Source: AGHT+IG9pkaa+NfBfBN3tv0PMD61sScfAedWpdRl9BNgXYxxGW/3XiB249I/ScBV6f/2zKuHHJhj18tIK2Y= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:83d0:0:b0:d78:3c2e:b186 with SMTP id v16-20020a2583d0000000b00d783c2eb186mr79839ybm.5.1694656552108; Wed, 13 Sep 2023 18:55:52 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 13 Sep 2023 18:55:06 -0700 In-Reply-To: <20230914015531.1419405-1-seanjc@google.com> Mime-Version: 1.0 References: <20230914015531.1419405-1-seanjc@google.com> X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230914015531.1419405-9-seanjc@google.com> Subject: [RFC PATCH v12 08/33] KVM: Add a dedicated mmu_notifier flag for reclaiming freed memory From: Sean Christopherson To: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Sean Christopherson , "Matthew Wilcox (Oracle)" , Andrew Morton , Paul Moore , James Morris , "Serge E. Hallyn" Cc: kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, linux-kernel@vger.kernel.org, Chao Peng , Fuad Tabba , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , Xu Yilun , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DE8BE140003 X-Stat-Signature: dewzyf7fopfa8bkdfnhb61hrtn3ukn1j X-Rspam-User: X-HE-Tag: 1694656552-427043 X-HE-Meta: U2FsdGVkX19xJ+VOgsvFOR+Dqz4X2GdLdFGHE3yMOawC30YnO6lXynS6upbzYSDKrMjqXFNLHjuZ90GYnhk5fCYREedwDHimjdlun1J34XDHYsijG5s+5uuvmjJqqCnmNpY8j9rYduuhVuGAsQ1GRQAEWaHzwlCWS2kb9PSl2XlStUmzWrEkeuOKXoJimJT8EGVoSA+Df3VOeDpWIWpUqhhEZTYTr+39yurYjikUrlQtZhZiyuvi5Yb3FiUafr+1Yp1MMhvE3OmyboPqJ4FxH6qAtVHOINeeWCfkehuKHA+IfElr7Kyeedl0ImZUA+MQfGo2ORnc+8oZntg5UGEdMUIPGQINzt6dEWpYzMaMdJ6WAL4BGDDQ7xVgrhnNe/nDTHis+02ypt0o7zez6VQTS9lSlWFkQX6Ruuz5DI6ClbM236vjZO0uXuW0vaW1uP9TQDLiE8852722H4E7js1P1WawKjZyNEMOJjg2OrLfJPBetaMr2UgPjVa8YALPQSJclcisnRKAJsZ18ZQsm6Ww9T5fHzQRroHZYLHGzx047nntI9LJaXomLskkP4KsN6p/TcfZvpLyOplAuCxutlY70a5CevjpXdNZmDXoAK0+frsOiTMD9C/v+FtvCRxr7jKK70Ewyc1PKvfkLEZ+Ft5c4OCTfh+lzh8JERdU1HOXvc/9hN7YpiK3dhBntqPUO6AzIBd92HrIaru/YHKKjqgB+STqpz7yU71IOfft+yH0dy/FohmTAa/9ugGdDvyayG3PZDDx+t7vw8ZBuVPPPm+n8/2Bygb7RRvO8fPbAQxH3Zv9qARBbw8NJfovKT+9CJLVqQcQ8z5Kqm3Dlt4iu+riLKRX1ubK2KTtVJATQh1E6RP1kAydQm+9XTNnMdorQHrSfLCgaFYIko+Vowkc81fsFENhFOhL6qE0AUXTqyb99NzDMfjH/w8z5uEk2bdURaGQ84cDXrkZpzkAReKNH8i 8wLWXmBg mDNosOtmatwfDTskUFHd1FaXuO2WYvIgAD4+yzf047s6+B5qy4n7YHQCebQ46uc8CFVLOZzbMyZKRamf/2GhmAO1GCHTuda2WDfa5tOp2eVN8DHhRo5EugnakCzues9cwJ+QH6NFLXyW3MaxgMq97cCcCYElFBNi2ZLvvXnaVLavD+6YX+/cFE6H7+jXtjN8zw0S1on4xwjvTgOSWmG5B7CnpP3i1Scw0gfCIZ6oPUsCO0FwpRiC3Wlm2nml2FjCkFWYESZAntVPThCOu25hbRXNXnnch6SICLGb2hvB6PRzRy3MDuMM/3AtqgHhRwC6cgcDuspt4K7nyM2Hl0I7mYzmJEXsWmUWmJvtH7XCZwuUrnRMvkbc+3TmEa3NEPlW8Uhog7vMSINoFMMCP/7hB+J+p+/pWtVj6WZ5HOp8Hz7zjx1WBygxCg2eGXlr/K1AZCauEsWACNfOVikXUC3Xvufa80+h1WDwrIznp9Piwjp62SilNba/AnrcRJMV8SXSD7AqZRJDph3Z6XisypBy7U6Cui5wSl4msnuFT+9tTUbuYkYvbCtTShTkxgBpkUARegJ26MpObN7SChQOMFsLE6ZhcNW+KaxvSTYd3IibJ+xDNa+kXHpCw/1DdFBP6VYDvmI1mNmeDElfoJmqLMoEzwN+7zybocsbjo5/euk4ynBv5xENuJqIud3p3rFbemMzsXmLODArTfjO+eVWsU7Aoe7SCnCsx3snjG1/dT0OO1zmQVt+H+PcZJ0+5MjEViUWuhMq2MDbA7m/5v+IaSprzZL8ZyCY6dPGo6i9RwkgvIyWHGQHI+pcLX6z2DA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Handle AMD SEV's kvm_arch_guest_memory_reclaimed() hook by having __kvm_handle_hva_range() return whether or not an overlapping memslot was found, i.e. mmu_lock was acquired. Using the .on_unlock() hook works, but kvm_arch_guest_memory_reclaimed() needs to run after dropping mmu_lock, which makes .on_lock() and .on_unlock() asymmetrical. Use a small struct to return the tuple of the notifier-specific return, plus whether or not overlap was found. Because the iteration helpers are __always_inlined, practically speaking, the struct will never actually be returned from a function call (not to mention the size of the struct will be two bytes in practice). Signed-off-by: Sean Christopherson --- virt/kvm/kvm_main.c | 53 +++++++++++++++++++++++++++++++-------------- 1 file changed, 37 insertions(+), 16 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 7c0e38752526..76d01de7838f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -561,6 +561,19 @@ struct kvm_mmu_notifier_range { bool may_block; }; +/* + * The inner-most helper returns a tuple containing the return value from the + * arch- and action-specific handler, plus a flag indicating whether or not at + * least one memslot was found, i.e. if the handler found guest memory. + * + * Note, most notifiers are averse to booleans, so even though KVM tracks the + * return from arch code as a bool, outer helpers will cast it to an int. :-( + */ +typedef struct kvm_mmu_notifier_return { + bool ret; + bool found_memslot; +} kvm_mn_ret_t; + /* * Use a dedicated stub instead of NULL to indicate that there is no callback * function/handler. The compiler technically can't guarantee that a real @@ -582,22 +595,25 @@ static const union kvm_mmu_notifier_arg KVM_MMU_NOTIFIER_NO_ARG; node; \ node = interval_tree_iter_next(node, start, last)) \ -static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, - const struct kvm_mmu_notifier_range *range) +static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, + const struct kvm_mmu_notifier_range *range) { - bool ret = false, locked = false; + struct kvm_mmu_notifier_return r = { + .ret = false, + .found_memslot = false, + }; struct kvm_gfn_range gfn_range; struct kvm_memory_slot *slot; struct kvm_memslots *slots; int i, idx; if (WARN_ON_ONCE(range->end <= range->start)) - return 0; + return r; /* A null handler is allowed if and only if on_lock() is provided. */ if (WARN_ON_ONCE(IS_KVM_NULL_FN(range->on_lock) && IS_KVM_NULL_FN(range->handler))) - return 0; + return r; idx = srcu_read_lock(&kvm->srcu); @@ -631,8 +647,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; - if (!locked) { - locked = true; + if (!r.found_memslot) { + r.found_memslot = true; KVM_MMU_LOCK(kvm); if (!IS_KVM_NULL_FN(range->on_lock)) range->on_lock(kvm); @@ -640,14 +656,14 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, if (IS_KVM_NULL_FN(range->handler)) break; } - ret |= range->handler(kvm, &gfn_range); + r.ret |= range->handler(kvm, &gfn_range); } } - if (range->flush_on_ret && ret) + if (range->flush_on_ret && r.ret) kvm_flush_remote_tlbs(kvm); - if (locked) { + if (r.found_memslot) { KVM_MMU_UNLOCK(kvm); if (!IS_KVM_NULL_FN(range->on_unlock)) range->on_unlock(kvm); @@ -655,8 +671,7 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, srcu_read_unlock(&kvm->srcu, idx); - /* The notifiers are averse to booleans. :-( */ - return (int)ret; + return r; } static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, @@ -677,7 +692,7 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, .may_block = false, }; - return __kvm_handle_hva_range(kvm, &range); + return __kvm_handle_hva_range(kvm, &range).ret; } static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, @@ -696,7 +711,7 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .may_block = false, }; - return __kvm_handle_hva_range(kvm, &range); + return __kvm_handle_hva_range(kvm, &range).ret; } static bool kvm_change_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) @@ -796,7 +811,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, .end = range->end, .handler = kvm_mmu_unmap_gfn_range, .on_lock = kvm_mmu_invalidate_begin, - .on_unlock = kvm_arch_guest_memory_reclaimed, + .on_unlock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = mmu_notifier_range_blockable(range), }; @@ -828,7 +843,13 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, gfn_to_pfn_cache_invalidate_start(kvm, range->start, range->end, hva_range.may_block); - __kvm_handle_hva_range(kvm, &hva_range); + /* + * If one or more memslots were found and thus zapped, notify arch code + * that guest memory has been reclaimed. This needs to be done *after* + * dropping mmu_lock, as x86's reclaim path is slooooow. + */ + if (__kvm_handle_hva_range(kvm, &hva_range).found_memslot) + kvm_arch_guest_memory_reclaimed(kvm); return 0; }