From patchwork Tue Jun 11 22:05:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 13694229 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EE51153823 for ; Tue, 11 Jun 2024 22:05:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718143523; cv=none; b=mOmF2hxh1f2z4LK6S4h0F6TH2sCozJb71khyqboQePA7nnAfhx8Yxy5Xre4lg8olL29MNlaNEgn9OGxcN0y+ihtFB0fRxOhKo3tx7lAqYfW6TCGSq8j5+KFdVHYwjxjloMEj2N89SnyURW47yAGntONmpB2lF5Q4HmVSoDbqPrs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718143523; c=relaxed/simple; bh=tBTT0YzYZmIQ08jDKrRrM52oI3cCt7LuvA+9gttFOMg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=JSdwvjMV1rVx6SgAroYnXsHXRKvTxC6Sb5vviZ/evIeLHlASsAtLYi5IdqYImkXlpXL+NnDKbW+GsL4BfysvfKeohEvF1vwTXwJqaA0Ap2YxiAlKMjeAYRajTcjNCfADYKRo5zRxv45RsSkXxoTIONZMdjE6BBRdFqerKdImysw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=maX5ke7g; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="maX5ke7g" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-df771b5e942so9846226276.2 for ; Tue, 11 Jun 2024 15:05:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718143521; x=1718748321; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6V/i1CuMRW7vCXDnaQc6ILDmfsdZJdA9Ksq5F+/kAHk=; b=maX5ke7g1WuWJLVBrEyZa+zPAlITGcZXG3QCA/ikwHgMQYqHj68qrJZwSxo/P5pGtk gIxicu+wXJmbk+MeKrys+BvZE50lmp1KS+48u1DuPy7UNUQxgV7lVQhW+tin5ibkwnuC WvWrKSUIV7S9ZjfvBNnP+5RhSmXrrZbd+b1UFJtGeM2dj/oGDAKITtFQsIQUrTDbOk/L tRb2OMeXHn+R9aHCwrU/1PwmyurzS6kXgL6JAXyyo2cd9MGQkW/DK51sEIjKkYQrDFKy Aw9m3Q8+HteJppAi4Rbbp0YyThQ8PlbZKD8dc8K9xXS8b/jvC3zZRgmLciAk5+lgqZHj z/iA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718143521; x=1718748321; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6V/i1CuMRW7vCXDnaQc6ILDmfsdZJdA9Ksq5F+/kAHk=; b=YDvFBXNSFH886l/HXiap2Sc4LOaHbnLT1bGNhLUuRfOr438H687+YzDGPGL4T3cH3M amVztQfbQJOMHIoXxNFDp+c29mawmH2vH6a5+WXtxuQ44uWTFJ5jcGGC30h+0ELYLBg0 DTI6GJFrv7dHJwELufTHMp+y6oERTOCdFUay/2332fo5MPrLR/kMhxjhAyyZMg4tDyVM nu4yJPUAD3HSZHyiQZLHW8h1G+w6IZPHftCimfSMTy3mTJyRtYjlyPIeUkfelt3xlK2v klBdVpIMz6t0sFRKy5nk9KqdrB7I3qY/Mi/UstntgsnHddscQqdU+NH4BzlXZiAP7Riu NZvw== X-Gm-Message-State: AOJu0YxoTQTY5KtusyVvPBP5ZV2zq0yXYq1vE/BdiKtONH4CMMe7HXe3 mewDC7v02kQqXEVPog/lXEwmH2x8TYPxA9B/KZKhrahoVCgi+prwkeEFeU+c2f2pypsRnc7NLIh KqK2o1/i9SA== X-Google-Smtp-Source: AGHT+IHbQPJ57HBK4CfqZWqkrKv+avZJS1Ml7m7jCeBhUs+6SkgJuz1wnVp4bG+9bZDmV0O7yAWEH64b6Gq3dw== X-Received: from dmatlack-n2d-128.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1309]) (user=dmatlack job=sendgmr) by 2002:a05:6902:2b8a:b0:df1:d00c:130c with SMTP id 3f1490d57ef6-dfe66673483mr7031276.5.1718143521361; Tue, 11 Jun 2024 15:05:21 -0700 (PDT) Date: Tue, 11 Jun 2024 15:05:09 -0700 In-Reply-To: <20240611220512.2426439-1-dmatlack@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240611220512.2426439-1-dmatlack@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611220512.2426439-2-dmatlack@google.com> Subject: [PATCH v4 1/4] KVM: x86/mmu: Always drop mmu_lock to allocate TDP MMU SPs for eager splitting From: David Matlack To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, David Matlack , Bibo Mao Always drop mmu_lock to allocate shadow pages in the TDP MMU when doing eager page splitting. Dropping mmu_lock during eager page splitting is cheap since KVM does not have to flush remote TLBs, and avoids stalling vCPU threads that are taking page faults while KVM is eager splitting under mmu_lock held for write. This change reduces 20%+ dips in MySQL throughput during live migration in a 160 vCPU VM while userspace is issuing CLEAR_DIRTY_LOG ioctls (tested with 1GiB and 8GiB CLEARs). Userspace could issue finer-grained CLEARs, which would also reduce contention on mmu_lock, but doing so will increase the rate of remote TLB flushing, since KVM must flush TLBs before returning from CLEAR_DITY_LOG. When there isn't contention on mmu_lock[1], this change does not regress the time it takes to perform eager page splitting. [1] Tested with dirty_log_perf_test, which does not run vCPUs during eager page splitting, and with a 16 vCPU VM Live Migration with manual-protect disabled (where mmu_lock is held for read). Cc: Bibo Mao Cc: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 36539c1b36cd..c1f3b3798764 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1366,19 +1366,6 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm, kvm_lockdep_assert_mmu_lock_held(kvm, shared); - /* - * Since we are allocating while under the MMU lock we have to be - * careful about GFP flags. Use GFP_NOWAIT to avoid blocking on direct - * reclaim and to avoid making any filesystem callbacks (which can end - * up invoking KVM MMU notifiers, resulting in a deadlock). - * - * If this allocation fails we drop the lock and retry with reclaim - * allowed. - */ - sp = __tdp_mmu_alloc_sp_for_split(GFP_NOWAIT | __GFP_ACCOUNT); - if (sp) - return sp; - rcu_read_unlock(); if (shared) @@ -1478,8 +1465,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, break; } - if (iter.yielded) - continue; + continue; } tdp_mmu_init_child_sp(sp, &iter); From patchwork Tue Jun 11 22:05:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 13694230 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3423F1552E3 for ; Tue, 11 Jun 2024 22:05:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718143525; cv=none; b=CCySOFwh8cUPgI4pTKKBUMtAi002YiZy+5rpjQ0XG4EHUbGlDpkQaY7rXE41eGUeW3XJ9t/cUMAfXnA2N8e9J1eaXPsAwINGL3+nRvCnaCW7HNwEEaMF09MByttHi6G1xrToTfxN6otXEq+uYUHDipZnWcCxYpDgNCaj18TjSC0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718143525; c=relaxed/simple; bh=kMarZ5D/P8bfyIYmQRXdOL1MNPG84ZeZ1Mp5KuBlhhA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TDRPfYFmuxszUtZumVEMi+KsUBWG7eTvb00DBKr61CXn9Ca3iVY7kHOVbMJGI1IGcqF0X+BWHGbyCQz+crIJNM9jCv3upTpQp/bd2QDkvnZZ8y4MOSX3SSpZgEPld9WXvlQD89oJwjxCXwzKjA4vrhDA7jEl9XKvu0HyiThaYak= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TbYe9r6L; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TbYe9r6L" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-62a088faef7so3244827b3.0 for ; Tue, 11 Jun 2024 15:05:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718143523; x=1718748323; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5Siu/KDRE6dW5TN6w90WfnqseJSDZUhC+eUsxP9R36Q=; b=TbYe9r6LqYYkfyNNi8tPI0050k6RAgRYMV28+RVXFpForqFwL1hks/ztWpvyvO+gQM DWAh/PC4YRJ9cl3PLRqqiX+8HhLg+3Gjy5MBhbAS5LZCfF0iLcGa74Wq53oH6Z7aIkey Zk89oik38AdVdFEZncvNVJlc7jpqVDbgyHDLXf1tF7O+UaTidAWkQsuOgFw6P6oGrYst UIpC94iiMQ3gdOBj1GFH0uKvLkvbaJwV2HzuxdF5HkHsbgK/+ryuV6UI7oIyi0AETZjp uDI33pjFf0pFFrXRHLKc3y1wwlH0JJYjHtZ3mQRpMDcUiqO6/FG8zG0VPy8MkBmjsEKa Q+3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718143523; x=1718748323; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5Siu/KDRE6dW5TN6w90WfnqseJSDZUhC+eUsxP9R36Q=; b=msgC2xmxT8+h+IpPGUwgFUjDTxyO9yHPZAkY+FlrQpQMD5QFNN8t+8FHZMJoY+ccsw tkrp1khr283/WvKn6D0vkx4J4pCaklTOI4FS7gOWvVwhbDnObvJ9Ax9ziMqnitgL0Qw2 QtoNCELru59xqJRWA/XPFmu0CeENavg8kIfSysT4k2gea7+RR9tbn++FSjhaQwMKXQgq +8xSSJy3weGjfLTFe1Hh/poi/Ir85zLBPrMtscc/rziOP+ek+dSDMJeS62hGtruxHbyv oJY15RRcrgmT3dP/cRIKxolm9yGqmXOca0txa3IaambeUGMaMKvEC/OYT0+J3TRbaXaJ 1u0A== X-Gm-Message-State: AOJu0YyzIVUwP05DRm2sxl1/Nuc5s90eumkVTaDY1s/PDA/hAt7frWDM W0w0R8TLjIGj6TyrXUYpPM23CpSHc7GsE5599XAThWadyXy3sdyTSNhtMZabTqrX2Lj/KwOWF+k soMdS3SxqCw== X-Google-Smtp-Source: AGHT+IEeJOGaERGrbjAgR+SjU4vJym2g+HYlDduBmzN0PvflaZq6Mt3HRl6LoKLhf3j1/gUuS/7bx+ndZQN6rw== X-Received: from dmatlack-n2d-128.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1309]) (user=dmatlack job=sendgmr) by 2002:a05:690c:fc3:b0:620:32ea:e1d4 with SMTP id 00721157ae682-62fa26ff1c1mr566577b3.0.1718143523145; Tue, 11 Jun 2024 15:05:23 -0700 (PDT) Date: Tue, 11 Jun 2024 15:05:10 -0700 In-Reply-To: <20240611220512.2426439-1-dmatlack@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240611220512.2426439-1-dmatlack@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611220512.2426439-3-dmatlack@google.com> Subject: [PATCH v4 2/4] KVM: x86/mmu: Hard code GFP flags for TDP MMU eager split allocations From: David Matlack To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, David Matlack , Bibo Mao Hard code GFP_KERNEL_ACCOUNT when allocating shadow pages during eager page splitting in the TDP MMU. Opportunistically replace use of __GFP_ZERO with allocations that zero to improve readability. No functional change intended. Suggested-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index c1f3b3798764..09c6b16630ac 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1339,17 +1339,15 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, return spte_set; } -static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(gfp_t gfp) +static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(void) { struct kvm_mmu_page *sp; - gfp |= __GFP_ZERO; - - sp = kmem_cache_alloc(mmu_page_header_cache, gfp); + sp = kmem_cache_zalloc(mmu_page_header_cache, GFP_KERNEL_ACCOUNT); if (!sp) return NULL; - sp->spt = (void *)__get_free_page(gfp); + sp->spt = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); if (!sp->spt) { kmem_cache_free(mmu_page_header_cache, sp); return NULL; @@ -1374,7 +1372,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm, write_unlock(&kvm->mmu_lock); iter->yielded = true; - sp = __tdp_mmu_alloc_sp_for_split(GFP_KERNEL_ACCOUNT); + sp = __tdp_mmu_alloc_sp_for_split(); if (shared) read_lock(&kvm->mmu_lock); From patchwork Tue Jun 11 22:05:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 13694231 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B41F215531A for ; Tue, 11 Jun 2024 22:05:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718143527; cv=none; b=Ef1nBZkaVCajUrRG9fMDoUayH+/cf7dOB8rGBaWgAX/y7mIR8axcz17Usk+QSFEe6e/d7ElNnzGb240szBTv0CBusGaiigJ28qIqaBArHTEgQ9/hCty2yhg7Wrk+L/mBUyDNKJbLymfN92oQwLYqx3utSGbY24Ckeonk+qKSE+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718143527; c=relaxed/simple; bh=eJ//Sfy8cAge3Wfm8BzQCUM7ZYsp+rvNBG0LK8AgQRc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lrG4BXFAi8wxOLXBx/LaMlLVSO37CUO2hkJjkjjJPPOCyWIeUdwQfweugiyAkCAUlr7r09HvMuSC3OXjdwYUu+TKN8CTFCQOXW6rnxlLXbIYA6iF2U1chzRsvGoCvHeNpdJw85oY6Ahf5paEe/YwG+wfQBreMpaUgUIDcEeQ5CE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cqv7ATU1; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cqv7ATU1" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-62d054b1ceeso53039257b3.2 for ; Tue, 11 Jun 2024 15:05:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718143525; x=1718748325; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DUWBsEFIwb4pTI1yX8yUcgSMBcKADICY2lFC0KxNmpc=; b=cqv7ATU194uiBlQjQZZd+xOps2lxSHy2we/c59zI+TP+koqfDuk7FgUXtNU5IM7W4z xLAtEejGYqgzs94XtwsUVnnvbWGcv3F1heZPI8KFkrmc8vPEKrqoj5r588F/0yeZrqn4 XXBupuVKs6r4dRqQCN0/44mKg62yvwVGL0mb/fDUZDfcmeo75n+iitPyzXh2Ifcm97Us QMiq1IpZ8+cL+rVhphaghLuv1S2tCAb86RsTQlLSQ4T3ihWt1dh0fsjakUuwjB+IDwuH dfnbaF1sL99pIeutXOHgIFnefE2CUGxZkAE8dyVBGYZ7LAutQW6dvCptCFfOHpzcZKUd 3dqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718143525; x=1718748325; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DUWBsEFIwb4pTI1yX8yUcgSMBcKADICY2lFC0KxNmpc=; b=U/AdHsPnDiZ5B6WQuJRE5/UPjSbw06kt7teWCfl8NVgYEpPtlEnt5zS5USTAkHH2LW SeVWvmqZzOzz1yBCP9t6qEYiN5aJ6DaRiyKRKTXUC91Igo2c8324t5jbudXj9Xp8n6pu B7E7yD74p5OUCZP6egXmoQDSbvAKayTf6Ga+jeEuxcONXSEXkgPGjalLW6lpjrf/W+xq rwUNefZe1keC/Bl74ElEsF39FFxgqNH8HULO9fExH5ekYjc1RcqhGHDjGNnMMRefYKzW +r/VgJB0pliUrWPSbJ22DCyYWC7q5Rx2mTly+QHWWoSijrUhlgT8lFIy8w9Ad49LW7ep jCgw== X-Gm-Message-State: AOJu0YyECNzeREW89BfqxNiXbDOYLQ7uoIjHqIE3iJ3IxC/jmdrAYTdb UtUKyJiE+g9JPMJSj1jXOQIML7SaUryY0Di2PnS42em8r8TcRs5uOXadtHOeZp2FlHLTgMu/BL6 zqKdqEgsnjQ== X-Google-Smtp-Source: AGHT+IEtw+dbgGqLK+TqohKs4ltrJ5dY3k8g7GlcbbT36CmgWjab+PiCCe+vf0qxrWOERhHkhXBIgaF9FhWagw== X-Received: from dmatlack-n2d-128.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1309]) (user=dmatlack job=sendgmr) by 2002:a05:690c:3810:b0:61b:ec22:8666 with SMTP id 00721157ae682-62fb6c1f4damr216507b3.0.1718143524510; Tue, 11 Jun 2024 15:05:24 -0700 (PDT) Date: Tue, 11 Jun 2024 15:05:11 -0700 In-Reply-To: <20240611220512.2426439-1-dmatlack@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240611220512.2426439-1-dmatlack@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611220512.2426439-4-dmatlack@google.com> Subject: [PATCH v4 3/4] KVM: x86/mmu: Unnest TDP MMU helpers to allocate SPs for eager splitting From: David Matlack To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, David Matlack , Bibo Mao Move the implementation of tdp_mmu_alloc_sp_for_split() to its one and only caller to reduce unnecessary nesting and make it more clear why the eager split loop continues after allocating a new SP. Opportunistically drop the double-underscores from __tdp_mmu_alloc_sp_for_split() now that its parent is gone. No functional change intended. Suggested-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 48 ++++++++++++++------------------------ 1 file changed, 18 insertions(+), 30 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 09c6b16630ac..7e89a0dc7df7 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1339,7 +1339,7 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, return spte_set; } -static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(void) +static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(void) { struct kvm_mmu_page *sp; @@ -1356,34 +1356,6 @@ static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(void) return sp; } -static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm, - struct tdp_iter *iter, - bool shared) -{ - struct kvm_mmu_page *sp; - - kvm_lockdep_assert_mmu_lock_held(kvm, shared); - - rcu_read_unlock(); - - if (shared) - read_unlock(&kvm->mmu_lock); - else - write_unlock(&kvm->mmu_lock); - - iter->yielded = true; - sp = __tdp_mmu_alloc_sp_for_split(); - - if (shared) - read_lock(&kvm->mmu_lock); - else - write_lock(&kvm->mmu_lock); - - rcu_read_lock(); - - return sp; -} - /* Note, the caller is responsible for initializing @sp. */ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, struct kvm_mmu_page *sp, bool shared) @@ -1454,7 +1426,22 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, continue; if (!sp) { - sp = tdp_mmu_alloc_sp_for_split(kvm, &iter, shared); + rcu_read_unlock(); + + if (shared) + read_unlock(&kvm->mmu_lock); + else + write_unlock(&kvm->mmu_lock); + + sp = tdp_mmu_alloc_sp_for_split(); + + if (shared) + read_lock(&kvm->mmu_lock); + else + write_lock(&kvm->mmu_lock); + + rcu_read_lock(); + if (!sp) { ret = -ENOMEM; trace_kvm_mmu_split_huge_page(iter.gfn, @@ -1463,6 +1450,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, break; } + iter.yielded = true; continue; } From patchwork Tue Jun 11 22:05:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 13694232 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69468155350 for ; Tue, 11 Jun 2024 22:05:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718143528; cv=none; b=VVqd5eyGdr/qA+bmzYsTzHwSVl8POZy1UK35E9zuyh6ngj2FxHFAXb7UM5UP6mcbGVkt4Y8V7zgpBj+KYjS161d9KKwPja7TPPBc/lYqukLt9MBQbE6tVvEydvqJLAJJ68dXHC9zic0LKzMu9r89sXcS0EsP1VcOA+SsG0zOSUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718143528; c=relaxed/simple; bh=gFu+Sw6FfHFhBGeL0KWbFRGu8jwMbi8iDkm+qP8tqvY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=P5sfgk7ab3dxPvDyhpf+acMpWJqZogJSBem4QyiEMU143QdQSUZBSSM4ivqT4n5IjZTfEi5HteQh/MMphYfHb0QXoASfn98HPjtUo4+I/k24ceOSLA75baM7Ubu/xecqfjguXhgYgAoZrMG0z4wmHvV045jV8cX6z2rwI1X41Rg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=wHIQK/kg; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="wHIQK/kg" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dfa84f6a603so10276434276.3 for ; Tue, 11 Jun 2024 15:05:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718143526; x=1718748326; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y0tWpVro+PTLkdjwwJJw7gcPsAniIuq8D/3VEiAryJ0=; b=wHIQK/kgPlsyRRR/7P8YlB6Ze8xbmHB2YSWs9Vo/a3maivGci0fyga7Mi81rz8B/ib tyGPGBz5O02JwDh7pPkmDHXcnccfQp7ptqZE16hpvFM9aAcRnHQHCBDd4uSyfmBPsjD+ xGE5vMVNDJPpreZdVRqIe/my0utL1oNlGKhZ5/77wJ9L7PKqQE3xsZgRjeXVf8Wue0G2 37MnQ8/iFQNlyeE8FOevRxsxW/EQlciLNO0f1zVDA6wiB3bLPRp27isCCRjX6zy9Km4q 0BnneHB4f3pXYcG1jS4CE1+MgroZ30tUEqJpIyrq2wm8Jz6C0V9Rv6Z5nwluBqJbVk3+ /tDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718143526; x=1718748326; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y0tWpVro+PTLkdjwwJJw7gcPsAniIuq8D/3VEiAryJ0=; b=vRFRPcFvMysi7EFrKTZQnaAB5PR/h81MNZWYpBUUHk4GqVNaMQ1xgGOwduEVq4S5KL WE94AU8b8ynXey975hNewCcf1nII3eUwnaSRArmikFC7ReAhOiXiMa35iuoKFn5fqpu2 e0BIN7UxmAoVAUndAl9ltjMCpGdEOfQmNUY0jMU3kCfijVVTxQCNegkMlon3b2ESk9+8 t4Imr4MFfzu0gj8pwWrdrXPAjXC6tjlecTlYxEhK/Um3LNGoS3BqHpWs0DiJPiYlb31V VrK6d56OBGyHT2dREUs0cHxhfitent74y6JB7IE4hDfjnITKqXGu2/eFH6am0VRX3rA6 J5xQ== X-Gm-Message-State: AOJu0Yy7FxTf/qwS+ESZT9aNlKPVtWk0jsu6jYCIuT40qXLXrABf9UER B0bMYnukiKlR7awWre2LjAWbWZ1cD36d28laF/KK98Vp4TK5TyoH24FXIq/poeYEJz5fPZOmYtU J3tCWRnDNUQ== X-Google-Smtp-Source: AGHT+IFf99FyBHSmkELQndmLkR1qklKV6sa+0sTgV1cL8/trlo9d4ArkqHnv66FB/zN34RJqqxrF0EDgBsUzqw== X-Received: from dmatlack-n2d-128.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:1309]) (user=dmatlack job=sendgmr) by 2002:a05:6902:709:b0:dfa:ff27:db1 with SMTP id 3f1490d57ef6-dfe66559ef6mr332276.4.1718143526381; Tue, 11 Jun 2024 15:05:26 -0700 (PDT) Date: Tue, 11 Jun 2024 15:05:12 -0700 In-Reply-To: <20240611220512.2426439-1-dmatlack@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240611220512.2426439-1-dmatlack@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611220512.2426439-5-dmatlack@google.com> Subject: [PATCH v4 4/4] KVM: x86/mmu: Avoid reacquiring RCU if TDP MMU fails to allocate an SP From: David Matlack To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, David Matlack , Bibo Mao Avoid needlessly reacquiring the RCU read lock if the TDP MMU fails to allocate a shadow page during eager page splitting. Opportunistically drop the local variable ret as well now that it's no longer necessary. Suggested-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 7e89a0dc7df7..142a36d91e05 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1402,7 +1402,6 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, { struct kvm_mmu_page *sp = NULL; struct tdp_iter iter; - int ret = 0; rcu_read_lock(); @@ -1440,16 +1439,15 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, else write_lock(&kvm->mmu_lock); - rcu_read_lock(); - if (!sp) { - ret = -ENOMEM; trace_kvm_mmu_split_huge_page(iter.gfn, iter.old_spte, - iter.level, ret); - break; + iter.level, -ENOMEM); + return -ENOMEM; } + rcu_read_lock(); + iter.yielded = true; continue; } @@ -1472,7 +1470,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, if (sp) tdp_mmu_free_sp(sp); - return ret; + return 0; }