From patchwork Tue Apr 1 16:34:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14035166 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E82E220D4F4 for ; Tue, 1 Apr 2025 16:43:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525797; cv=none; b=pK3YRFu1qHUupc8IioiybcWNwK3z0MMNvneuVU4Ax4GEqMTn76AhYiCbB6fl/w4fWAGeZG9bvWP3EHeplrrsPOufCK5VZtA+TncGwQCxrpYpu0vcKMPBQd4Bwivltk7azWcPlcu7qHJtb1G0DNl9Mks0Q32ISi0EUSSbrfT28eA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525797; c=relaxed/simple; bh=IX2hIEFDZxr1s7mPt549rL+2MP5J40qc7gWfaob7z2I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SIeDx+HmqCNe6jOQXA8WeXRXbDEltBULpKqRM6l2Y+Ode6SBrJ3KW11bKxHQzNFrWZIErYWP3PTdvd3grl/XRfTbmHuKD7waaikz8ZyXUH810bXCs/1ZuHnl95t82bE934OVZRssdVPQ+lTw/xhjFITj2oMTPHwh3Br667dBsas= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=2RV4GpZl; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="2RV4GpZl" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-229170fbe74so86380615ad.2 for ; Tue, 01 Apr 2025 09:43:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1743525795; x=1744130595; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=tjqTIWn86XObYZ0dlBWL1fKISTH8kVDuFqpN5liSeqY=; b=2RV4GpZlnot4SCQ5wABLNoj1h5/geg50gJV/aZ0WrLzGYB/VV3xhNfojzwmHO0uDYz g31k7XvWqBamA1iirpWdFwlIerGnukt8vXTN16dbap04aiQd38ioMrYWgU3kFFc5odti G3p58wvcGb0TWKOKcZh4W9l5+3syRSR1wYrrRMdvuji2Yn4VG1pIqMWInZnmw4M6Sezs 7l2ZkZNNoe0TUoaH4AB6Ite60SMnnBmvXQgUhl2So3HEUiHXZv9jlhMmxdb3We6fpkHa 8wqwkYjn53FsRSZfxI7C7rZRuTeCyd7KO4GJrlszzn9sFKa8uVGmvZKcdJaFMIDs46Ln Ejhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743525795; x=1744130595; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tjqTIWn86XObYZ0dlBWL1fKISTH8kVDuFqpN5liSeqY=; b=oG9vclUCeRivhTXWswkzzJanjONHapf6vmfCYv14MBnHf+AhlEDf5HZFlX/AVcNf2E lOGu/Q0pehCn05iUEycBUJ19dQ+4V962w2hY31Od3R+IdzrQxOOUJv8gsdwsTdZDEL8I LM0wNooAittXTNNVTbQ9W4mXdTZ3pt5FPeO5gVFA1Bf0ifVonW95w4W3XtXQ+HF8qCcd Y6KOoZVMSCcMBqzP0kvHRdFdXU4Hsmeh9zoer7HHlZ+an2B978mVBhxKy9rz/2K1BBzi 98dBgEZjnn4inA2avJ8HqYvDCuinjzOt3158Uu+HTvS88lCugllUpcEDy+flxv2m+/J3 O+ug== X-Forwarded-Encrypted: i=1; AJvYcCVTG02mqyJJqUITqL+qColtxEzu+IxGZa/XxawIR3w1QDYPoswmUBOOnQS+gpqWllIgXfA=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2dXe4aVKF4zJL6DzELtaYz86msenWKB2NE3V95xytRca8VSG1 b15T5ZxWFIgg/6l9CXuVjCseYgcLhvLsmsRqNKUjjfev7rYgCKEbC5W3u7EGWrdKbO7PL8UbR/Y nBw== X-Google-Smtp-Source: AGHT+IF7k6V4hI9NSmqeQR7Ldpinq9z+9LeYTtWSOW3n7weH2d7MasMbpC1/xTtwiRZrQxL5X02TqjREFfU= X-Received: from pfbhw8.prod.google.com ([2002:a05:6a00:8908:b0:730:7e2d:df69]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:1309:b0:736:450c:fa54 with SMTP id d2e1a72fcca58-7398036acbemr20074041b3a.6.1743525795261; Tue, 01 Apr 2025 09:43:15 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 1 Apr 2025 09:34:40 -0700 In-Reply-To: <20250401163447.846608-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250401163447.846608-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.472.ge94155a9ec-goog Message-ID: <20250401163447.846608-2-seanjc@google.com> Subject: [PATCH v2 1/8] x86/irq: Ensure initial PIR loads are performed exactly once From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jim Mattson Ensure the PIR is read exactly once at the start of handle_pending_pir(), to guarantee that checking for an outstanding posted interrupt in a given chuck doesn't reload the chunk from the "real" PIR. Functionally, a reload is benign, but it would defeat the purpose of pre-loading into a copy. Fixes: 1b03d82ba15e ("x86/irq: Install posted MSI notification handler") Reviewed-by: Thomas Gleixner Signed-off-by: Sean Christopherson --- arch/x86/kernel/irq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index feca4f20b06a..85fa2db38dc4 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -414,7 +414,7 @@ static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) bool handled = false; for (i = 0; i < 4; i++) - pir_copy[i] = pir[i]; + pir_copy[i] = READ_ONCE(pir[i]); for (i = 0; i < 4; i++) { if (!pir_copy[i]) From patchwork Tue Apr 1 16:34:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14035167 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B22C920E309 for ; Tue, 1 Apr 2025 16:43:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525799; cv=none; b=fo+WYRLk04yX2XDMG1bQaMMVQKgan4GQxNWYLonNrWwLmYizqGZQhZ23J0vJwvfUViL7L6b5+1C/XVRQ1SRm6FLGZfKK9c6cmHDTKgQ6u/AbYrmAs2p9sjUo+oQBcx2SVBRgjq8AcuTI9Z0E3cacpnnusUrAkHAjaPWwtxkgqn4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525799; c=relaxed/simple; bh=jGuwQyi4PUoNb9tVJY2auvrWnk5ldsHm2gyBlrKR40I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Gp63UGaJ5MwwJk1P76eI9tnLsJjDSqX5EMs06LfZrhBnRc1IrSYWeX4y0jezzPYn3+anHNRAE/pA21qVe27q1Ws8B5E/0Zp72Ttvj3TUkE30+BPC1i7XBLiY7EgF+H/PPUTsTar8g6FpZDanPyLqBfV7EhR6NPsN5iI9+LhXhBo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hVa7N9El; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hVa7N9El" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2254e500a73so77471365ad.0 for ; Tue, 01 Apr 2025 09:43:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1743525797; x=1744130597; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=cDvP5BZpEDyKBmPstvDIZN80Syur3P8OaOWHbakcInk=; b=hVa7N9ElrbNfcAg1jLOLP5EPxthvxMGVYFreS7KIRdo6rOKEzm52+yLMNQleULnE4g MiJm9t67idfdGe0PgNVfr/uVIZg4AWNy9IWk7aAVquEypW1Jx6ZonnVmYiNGtutctN/g op7VCaUpr3vLPtm0tnN3wMDWeBh2fMqjkC8V+n5cITDfoWrfTnTBWqhk87x8hzRaJvX9 k6dpgkiJ0Q2Hie7qI9zcEl856CIUem2YIUqC49m5hdknnNoWXTxR2IkTZ7TyzJhJmG8w 3XTdpIsuMe8McNsA1BAaWAW4yxcQOPY/RGyMZI6NmX3XAiFcczj/K7PtJJJk0rAPEu8k 6uaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743525797; x=1744130597; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cDvP5BZpEDyKBmPstvDIZN80Syur3P8OaOWHbakcInk=; b=AeQNPpUBXIUONS3NJnKxoLF+7SWLXFm7Vi80xegOjbdScsVPW8AehO7tK1I/2IQW3i TwoE8zyQfLuBWNeLYObHDibLtZofQb3ofS0fmG8NmlNKx6j0c8nQcNoO0oHU6UaJsrR1 oGDlhEzDyHdcT1BjRAY1VnqB1kXhmjmZQzpB1IYRRkTSbgddCf1w9GfNA7c+h0yjj27g r2qPqSwSnVWpGemHWcfzSuTzmqecwoujbFT2bz3JYYvLJHJc0euCSigu8zDU/YMhLZ02 nfsCq/facUZzW8QEmYnQZ68d9B5rPHeYE1L3bVN6ZPQNt7MUBToFC9vFN7dTpbZIjIwn m4zA== X-Forwarded-Encrypted: i=1; AJvYcCVHUUNz63ssmnDRhxvNGACZd8GwwwW0DVKLIPzJUYledbSXS5qnmU4IvaGQwKqKEo7EWSc=@vger.kernel.org X-Gm-Message-State: AOJu0Yx0YKRrmfrncUTjEQ7s321SoFw/QPhjLYOQBkFIPw7JyqqmsXCx m0njVl9gfIe2JXQa9/G6XYDI/7rvdZZiXqeAIoSzqneeh1B84lsA9dbFAx/UEQlS5u+mVmuBwqz a4Q== X-Google-Smtp-Source: AGHT+IGxxWnUBIlHU15BzVG9W/GeQ6sCwXTyMrI61xhzsb0eCK4R7c40Z6JqSwtcB6fShFN5RLpLo6Zov7E= X-Received: from pfbfo6.prod.google.com ([2002:a05:6a00:6006:b0:736:3cd5:ba36]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:1901:b0:739:48f2:4374 with SMTP id d2e1a72fcca58-739b5ff7f6cmr5338777b3a.10.1743525797032; Tue, 01 Apr 2025 09:43:17 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 1 Apr 2025 09:34:41 -0700 In-Reply-To: <20250401163447.846608-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250401163447.846608-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.472.ge94155a9ec-goog Message-ID: <20250401163447.846608-3-seanjc@google.com> Subject: [PATCH v2 2/8] x86/irq: Track if IRQ was found in PIR during initial loop (to load PIR vals) From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jim Mattson Track whether or not at least one IRQ was found in PIR during the initial loop to load PIR chunks from memory. Doing so generates slightly better code (arguably) for processing the for-loop of XCHGs, especially for the case where there are no pending IRQs. Note, while PIR can be modified between the initial load and the XCHG, it can only _gain_ new IRQs, i.e. there is no danger of a false positive due to the final version of pir_copy[] being empty. Opportunistically convert the boolean to an "unsigned long" and compute the effective boolean result via bitwise-OR. Some compilers, e.g. clang-14, need the extra "hint" to elide conditional branches. Opportunistically rename the variable in anticipation of moving the PIR accesses to a common helper that can be shared by posted MSIs and KVM. Old: <+74>: test %rdx,%rdx <+77>: je 0xffffffff812bbeb0 <+88>: mov $0x1,%dl> <+90>: test %rsi,%rsi <+93>: je 0xffffffff812bbe8c <+106>: mov $0x1,%dl <+108>: test %rcx,%rcx <+111>: je 0xffffffff812bbe9e <+124>: mov $0x1,%dl <+126>: test %rax,%rax <+129>: je 0xffffffff812bbeb9 <+142>: jmp 0xffffffff812bbec1 <+144>: xor %edx,%edx <+146>: test %rsi,%rsi <+149>: jne 0xffffffff812bbe7f <+151>: jmp 0xffffffff812bbe8c <+153>: test %dl,%dl <+155>: je 0xffffffff812bbf8e New: <+74>: mov %rax,%r8 <+77>: or %rcx,%r8 <+80>: or %rdx,%r8 <+83>: or %rsi,%r8 <+86>: setne %bl <+89>: je 0xffffffff812bbf88 <+95>: test %rsi,%rsi <+98>: je 0xffffffff812bbe8d <+109>: test %rdx,%rdx <+112>: je 0xffffffff812bbe9d <+125>: test %rcx,%rcx <+128>: je 0xffffffff812bbead <+141>: test %rax,%rax <+144>: je 0xffffffff812bbebd Signed-off-by: Sean Christopherson --- arch/x86/kernel/irq.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 85fa2db38dc4..5d732ff357ef 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -409,27 +409,28 @@ void intel_posted_msi_init(void) */ static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) { + unsigned long pir_copy[4], pending = 0; int i, vec = FIRST_EXTERNAL_VECTOR; - unsigned long pir_copy[4]; - bool handled = false; - for (i = 0; i < 4; i++) + for (i = 0; i < 4; i++) { pir_copy[i] = READ_ONCE(pir[i]); + pending |= pir_copy[i]; + } + + if (!pending) + return false; for (i = 0; i < 4; i++) { if (!pir_copy[i]) continue; pir_copy[i] = arch_xchg(&pir[i], 0); - handled = true; } - if (handled) { - for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) - call_irq_handler(vec, regs); - } + for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) + call_irq_handler(vec, regs); - return handled; + return true; } /* From patchwork Tue Apr 1 16:34:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14035168 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F25E20F060 for ; Tue, 1 Apr 2025 16:43:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525800; cv=none; b=KqG7HAZ5FUWQ3a4mWKXPctWHguywxOzpLMqA0gfudHiftg/QZgvL0/Q94TA00T9ROzamxCu4unTZTZ2OOAkG0v6hvB8MvWwxvQgFHOu+GYsL1u3phQ0Ye9fdQChFWsZTzdYK4FBl7CfYT1Jb6qvA6PI9B1BWvGuMxV82ZlRZNpw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525800; c=relaxed/simple; bh=68XUd7R1fyxPr0p6LhY6RzrECfMFspwxL2oY+XxMEXg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ir48YSfUqEgprWTwVpvy7/lcktLIi476a3lRad8evuvoq9auiHg2Bw2XnOmzMT/Cx5mfhK7TB+oEHhKerQRmSTAVHuQ2bIW+gw6f0CKxgYnhH2vg/UFd1yRzjW2iYxKLbveYf5TWojNro29HtBbu22m1smkXuswd1udKyiZbdb4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=myq5/FO5; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="myq5/FO5" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-22647ff3cf5so93725665ad.0 for ; Tue, 01 Apr 2025 09:43:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1743525799; x=1744130599; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=ENuRHHo4buatmc9xHBzqgn6lL1J28WBga1j/Fc+SdOU=; b=myq5/FO5y+daJ8IlBfsRl2hkMwC9QXPl6TRhHGULZ6f3dgUcLpSRWKIqyWp7vaRV6O cE3mSpbyXhn7oWMlcZ2EbxrxGxzPFj5VywLL7VPbCYvVJs8wE983pHbHmRjguXjOqXGw EjJdozdFdi8MbTqH8AQ9CvBwcM9/rPfvMfBHyQXgvRhNO4SMd5UP1Q97THSGb7KMrCMW IeGlfQyImZ4DWpWM1Na4wvU9ghcWDH6XqZpckVcCaY6JOeHNRm7ys5iUKQnRlXl93Rc3 kn9lpocnTNnHEOp465RkPMCi8k/j8E4aiwERj+uPnVxOV465aKEePy2YuPFgrveSlO20 VttA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743525799; x=1744130599; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ENuRHHo4buatmc9xHBzqgn6lL1J28WBga1j/Fc+SdOU=; b=lEFO3w/tBwKoDDvO2RzPKnGSlqDSQKnsnDaA6s1sYz8hOR/4EHYMhRG7E22MVdtGcE CjJAliJrl1H2G86c5TiGM8hLXI64vNw/+7Pa7EoYMe41pL8b6QwPYRN689O/5bP5AoqV oyQQ+imI5jHslJ3uCfQ/YbxCZgisH0G0DX/+eSXjsJ6Ajyj5RyOyPz6yR4u+ZMcU/pgB Z1yfQIPKZieIKPK7+1n85kk42PWKMeGU0wnOqoU5Ark8icHLIQm+jE9uxackGYW6/9Mu rdRs+U/acZUuQ3GffJO7UsBhT93tJdH6+mNQzaTNbynDvaBXAN/Ejr0zdEeCrNW0fxpj YzAA== X-Forwarded-Encrypted: i=1; AJvYcCWwbrNgebhLgmlTieJMg63/iEZg9KaI7k9wge/dTzx5nuJiGYNsGqNlJaGBJv9pOX30o6g=@vger.kernel.org X-Gm-Message-State: AOJu0YzywBcp2O4sTlaU7/MWndoDBNRvpyO4KlpxXYnvhQQXc+NcnHU2 MFyhaLm8JBVoad7sIuURsYLmiYc/dg1vTvlilwsvA1PujualnvLfdVoX4Gc3JZ8e7L3ETzbJsLO vWQ== X-Google-Smtp-Source: AGHT+IG7uPldPi4KrDzTQHRm25erUUAd10L5t443gFis5z6W+qmvlaB54rM8wSKwHj0ezPMBh17NInSp4ik= X-Received: from plbm1.prod.google.com ([2002:a17:902:d181:b0:223:2747:3d22]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1c7:b0:224:78e:4ebe with SMTP id d9443c01a7336-2292f9e62cbmr194250575ad.33.1743525798682; Tue, 01 Apr 2025 09:43:18 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 1 Apr 2025 09:34:42 -0700 In-Reply-To: <20250401163447.846608-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250401163447.846608-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.472.ge94155a9ec-goog Message-ID: <20250401163447.846608-4-seanjc@google.com> Subject: [PATCH v2 3/8] KVM: VMX: Ensure vIRR isn't reloaded at odd times when sync'ing PIR From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jim Mattson Read each vIRR exactly once when shuffling IRQs from the PIR to the vAPIC to ensure getting the highest priority IRQ from the chunk doesn't reload from the vIRR. In practice, a reload is functionally benign as vcpu->mutex is held and so IRQs can be consumed, i.e. new IRQs can appear, but existing IRQs can't disappear. Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 9dbc0f5d9865..cb4aeab914eb 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -667,7 +667,7 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) for (i = vec = 0; i <= 7; i++, vec += 32) { u32 *p_irr = (u32 *)(regs + APIC_IRR + i * 0x10); - irr_val = *p_irr; + irr_val = READ_ONCE(*p_irr); pir_val = READ_ONCE(pir[i]); if (pir_val) { From patchwork Tue Apr 1 16:34:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14035169 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD0E220FA9C for ; Tue, 1 Apr 2025 16:43:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525802; cv=none; b=aw3XqLTud9TlEwIsR+IP1oFd4yflmeHrMmUhJ3Z5de/zF60l+5gyGCbrItPauCFVxRAC+f3Teabm087J0JzNcF9tia+/K8A1+12GTMR/pCrmavfGXgE2kGOm59zDq68oAtM3X4luX5mvUT2xGKrHW1pxzxr8CY9NPW1nZ+R1YCw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525802; c=relaxed/simple; bh=/GjbkA6Kd7hsHZq/P/3sF3zT4FudF94Hv2Bxy1AVK8U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ATt4WbkohnM1/L8SA/fq5JcStIroDvZVrwRkBUUdBBjxgL9QZPFdnP4NcFo+EWtJifYoctyz4kMiAK7v6QtBlETldk+5EcYEBD2x42QxBvzdYP6C+NyqROMvMUeadX1QdPBtwcqYNqw4lVTLuBGM0lZNselsoqYNy2iX2kAjTok= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XDct9ytc; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XDct9ytc" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2265a09dbfcso7595965ad.0 for ; Tue, 01 Apr 2025 09:43:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1743525800; x=1744130600; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Zwrym7xxEAtgq4x4qvWpfuf9S76CNXdDvpgAdSnAM4Y=; b=XDct9ytcdGerG8zJTTQxiKoEpO8dmphUu3EL8CFqbq15SUkZCDb8YTza4FcCi5VICZ PfM7nJn7bHWVkj4NPA4WxIQ5Dqwoejd2H1q6gAjs0O4iqOF7uHWeSu5gH68xII52PX9t MMNz1U/R4VSgAt2mDP2vOEuwDN3A1/IMnP+cq8wfAc5JxHqbh9Ttm2CVks7J3Ge92+o3 Ly/R5mjckJCXdqS2GBi/D+7nMfrUf8F201xsK7Ladn/UMb4MByif9r5TMnx90cx4wBUt 5Uu1qWYirvbdNLjJdx4mnesAOxIZ2N4BBEDWnzj1NFGi8kr+zJxEs24nuh1MJWlGz/56 NLkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743525800; x=1744130600; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Zwrym7xxEAtgq4x4qvWpfuf9S76CNXdDvpgAdSnAM4Y=; b=xFSOzWfP9iiCCrgbjkRI9IInB/qZh+dgF63lzlywFgUby11jOVfYJfQ/KiPpdvJdNq rIWTPKEL/6rb0JmJ1P5OY8e5Igt5n+OX75I5zVekMi45UtCBpzj4KduS/kUFdAYX6JUw nLL7uWF4fmE1DoKNquK0WJFjzr7VcHAHL/lml1br+MM/MP8cqRFX/87err6RftOwWPrO W8dbmo1Ib/2Xl8drSPoOV2HZrJT6qzYLVG3Oz7QU0exB4y8plNGla8EzNJWDgl3w9PH5 CZsn5UCxfzAN4djaWX6kK/26Ep7Tk98w5B87zPivfucH/XlzcDSj0iRXRK+DKIoD/qwE ELJA== X-Forwarded-Encrypted: i=1; AJvYcCUvNR10PTzCaQDYmk3TYdgqYw+61/gNO8a7lkd01CGwmcOulrcAq/DnEAkHN8pupZcqc68=@vger.kernel.org X-Gm-Message-State: AOJu0Yx/8qV/HRqVeYUW65Z14ut4Cak+3ujvmClqzpqbvUeSgtKMjvH2 GoGJQzvsajXH1rtNAlEpxrHntdpSUrWL6pR3e9k/zktFpd0orJVGWRT//Mk2ZXX89lUndNHfxZO vQg== X-Google-Smtp-Source: AGHT+IFbVo2sDfRTB5QdrlauwzOsHEHDj5i2PQ/L+ccHkR2b9gPNztMxgVK194+kKMnHLMvn4E7liI+Holc= X-Received: from pfbln19.prod.google.com ([2002:a05:6a00:3cd3:b0:736:a1eb:1520]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e74e:b0:220:f59b:6e6 with SMTP id d9443c01a7336-2292f9494e1mr156226415ad.8.1743525800423; Tue, 01 Apr 2025 09:43:20 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 1 Apr 2025 09:34:43 -0700 In-Reply-To: <20250401163447.846608-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250401163447.846608-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.472.ge94155a9ec-goog Message-ID: <20250401163447.846608-5-seanjc@google.com> Subject: [PATCH v2 4/8] x86/irq: KVM: Track PIR bitmap as an "unsigned long" array From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jim Mattson Track the PIR bitmap in posted interrupt descriptor structures as an array of unsigned longs instead of using unionized arrays for KVM (u32s) versus IRQ management (u64s). In practice, because the non-KVM usage is (sanely) restricted to 64-bit kernels, all existing usage of the u64 variant is already working with unsigned longs. Using "unsigned long" for the array will allow reworking KVM's processing of the bitmap to read/write in 64-bit chunks on 64-bit kernels, i.e. will allow optimizing KVM by reducing the number of atomic accesses to PIR. Opportunstically replace the open coded literals in the posted MSIs code with the appropriate macro. Deliberately don't use ARRAY_SIZE() in the for-loops, even though it would be cleaner from a certain perspective, in anticipation of decoupling the processing from the array declaration. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/posted_intr.h | 14 +++++++------- arch/x86/kernel/irq.c | 12 ++++++------ arch/x86/kvm/lapic.c | 9 +++++---- arch/x86/kvm/lapic.h | 4 ++-- arch/x86/kvm/vmx/posted_intr.h | 2 +- 5 files changed, 21 insertions(+), 20 deletions(-) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h index de788b400fba..c3e6e4221a5b 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -8,12 +8,12 @@ #define PID_TABLE_ENTRY_VALID 1 +#define NR_PIR_VECTORS 256 +#define NR_PIR_WORDS (NR_PIR_VECTORS / BITS_PER_LONG) + /* Posted-Interrupt Descriptor */ struct pi_desc { - union { - u32 pir[8]; /* Posted interrupt requested */ - u64 pir64[4]; - }; + unsigned long pir[NR_PIR_WORDS]; /* Posted interrupt requested */ union { struct { u16 notifications; /* Suppress and outstanding bits */ @@ -43,12 +43,12 @@ static inline bool pi_test_and_clear_sn(struct pi_desc *pi_desc) static inline bool pi_test_and_set_pir(int vector, struct pi_desc *pi_desc) { - return test_and_set_bit(vector, (unsigned long *)pi_desc->pir); + return test_and_set_bit(vector, pi_desc->pir); } static inline bool pi_is_pir_empty(struct pi_desc *pi_desc) { - return bitmap_empty((unsigned long *)pi_desc->pir, NR_VECTORS); + return bitmap_empty(pi_desc->pir, NR_VECTORS); } static inline void pi_set_sn(struct pi_desc *pi_desc) @@ -105,7 +105,7 @@ static inline bool pi_pending_this_cpu(unsigned int vector) if (WARN_ON_ONCE(vector > NR_VECTORS || vector < FIRST_EXTERNAL_VECTOR)) return false; - return test_bit(vector, (unsigned long *)pid->pir); + return test_bit(vector, pid->pir); } extern void intel_posted_msi_init(void); diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 5d732ff357ef..3753a8acd009 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -407,12 +407,12 @@ void intel_posted_msi_init(void) * instead of: * read, xchg, read, xchg, read, xchg, read, xchg */ -static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) +static __always_inline bool handle_pending_pir(unsigned long *pir, struct pt_regs *regs) { - unsigned long pir_copy[4], pending = 0; + unsigned long pir_copy[NR_PIR_WORDS], pending = 0; int i, vec = FIRST_EXTERNAL_VECTOR; - for (i = 0; i < 4; i++) { + for (i = 0; i < NR_PIR_WORDS; i++) { pir_copy[i] = READ_ONCE(pir[i]); pending |= pir_copy[i]; } @@ -420,7 +420,7 @@ static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) if (!pending) return false; - for (i = 0; i < 4; i++) { + for (i = 0; i < NR_PIR_WORDS; i++) { if (!pir_copy[i]) continue; @@ -460,7 +460,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification) * MAX_POSTED_MSI_COALESCING_LOOP - 1 loops are executed here. */ while (++i < MAX_POSTED_MSI_COALESCING_LOOP) { - if (!handle_pending_pir(pid->pir64, regs)) + if (!handle_pending_pir(pid->pir, regs)) break; } @@ -475,7 +475,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification) * process PIR bits one last time such that handling the new interrupts * are not delayed until the next IRQ. */ - handle_pending_pir(pid->pir64, regs); + handle_pending_pir(pid->pir, regs); apic_eoi(); irq_exit(); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index cb4aeab914eb..893e7d06e0e6 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -655,8 +655,9 @@ static u8 count_vectors(void *bitmap) return count; } -bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) +bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { + u32 *__pir = (void *)pir; u32 i, vec; u32 pir_val, irr_val, prev_irr_val; int max_updated_irr; @@ -668,10 +669,10 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) u32 *p_irr = (u32 *)(regs + APIC_IRR + i * 0x10); irr_val = READ_ONCE(*p_irr); - pir_val = READ_ONCE(pir[i]); + pir_val = READ_ONCE(__pir[i]); if (pir_val) { - pir_val = xchg(&pir[i], 0); + pir_val = xchg(&__pir[i], 0); prev_irr_val = irr_val; do { @@ -691,7 +692,7 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) } EXPORT_SYMBOL_GPL(__kvm_apic_update_irr); -bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, u32 *pir, int *max_irr) +bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, unsigned long *pir, int *max_irr) { struct kvm_lapic *apic = vcpu->arch.apic; bool irr_updated = __kvm_apic_update_irr(pir, apic->regs, max_irr); diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index 1a8553ebdb42..0d41780852e4 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -101,8 +101,8 @@ bool kvm_apic_match_dest(struct kvm_vcpu *vcpu, struct kvm_lapic *source, int shorthand, unsigned int dest, int dest_mode); int kvm_apic_compare_prio(struct kvm_vcpu *vcpu1, struct kvm_vcpu *vcpu2); void kvm_apic_clear_irr(struct kvm_vcpu *vcpu, int vec); -bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr); -bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, u32 *pir, int *max_irr); +bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr); +bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, unsigned long *pir, int *max_irr); void kvm_apic_update_ppr(struct kvm_vcpu *vcpu); int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_lapic_irq *irq, struct dest_map *dest_map); diff --git a/arch/x86/kvm/vmx/posted_intr.h b/arch/x86/kvm/vmx/posted_intr.h index ad9116a99bcc..4ff9d720dec0 100644 --- a/arch/x86/kvm/vmx/posted_intr.h +++ b/arch/x86/kvm/vmx/posted_intr.h @@ -18,7 +18,7 @@ static inline int pi_find_highest_vector(struct pi_desc *pi_desc) { int vec; - vec = find_last_bit((unsigned long *)pi_desc->pir, 256); + vec = find_last_bit(pi_desc->pir, 256); return vec < 256 ? vec : -1; } From patchwork Tue Apr 1 16:34:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14035170 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F1C9213240 for ; Tue, 1 Apr 2025 16:43:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525804; cv=none; b=acASuH3f8vTJxCSC2RmDONFtSXTN4MJpS/9k1yda5YhtxaP1eHfOVEMcNIa1Rj8M4cDjfz9KBMam7SNTrAmhpXUVPU7vjvZaEKl+yP8kRnXyHJTR07LCYsqjweD7PEybo0TDJagJHqA8m8pn6myT98FA+zkyOr7WQC+7XN7fDus= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525804; c=relaxed/simple; bh=5iq6Ej0JukMZOt4v5A++8d64xSPyqQ5v5er7/1iuZW4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MlHc9kPO/2rrtXvUKUAs6MjEhLqPojKSYKgaFbC4FqNM+o/GiEyQNKY9ryMnl9XV/w5QVHESQupr23sJEypre/dBb3kU873SnygG68yQgDgklZJgSAyVklfZziG2XQ3TcTtiszI2g2TCu5fhSjVsAQnCDJPgh50JP6J2jWad43g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=IB82jXoQ; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="IB82jXoQ" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-223fd6e9408so20635ad.1 for ; Tue, 01 Apr 2025 09:43:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1743525802; x=1744130602; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=/XYYrhooXHNQcqBoobC7D5bK0XS2jFQGb7UC9bcI9h8=; b=IB82jXoQ/BkQnpCN7Rnwso7yq0sh9Olsv/ISZOjrxCNXY38HN21itGuR8GTyHGf5QT elKTjYZKc0bXe7hovNqFTEKchRz5UlBPzDilS+eSSz4Ek6pGZ+j8pDbPHD7rU/xm2IOp FfFptLjn4aizDgbDhMESnlL8jF+KDt0ecpZnxYvi9AzUzt1XMMX/5oSc/UHa2owTPg/g 9jVjaJFdqd1DB7wZJHIg3/bY5u1INOUwXBLOlX+j7yuXxFNWOfK8uj53110xYeHxDJ+w iyv0WPoY+M0BnIZRVGN1LSrtD9FctfBHdKZXyOhcomhuWmiWK6wC7DhjuD5O94ELMmIY qjyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743525802; x=1744130602; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/XYYrhooXHNQcqBoobC7D5bK0XS2jFQGb7UC9bcI9h8=; b=U+61i11yK9yRJBHZolMyHwsH6sAw2wafLQ8zfOESyvXf3DPp0L45/zbg59PRhlOEQZ SnMURS/1at9KOMZfWsDSNfPWgV3p4r9K8cYIErFPwTRHhVQBNC6USFjjM9RSTXhO8KNv 8o9IAW66roVBuo8s6LlvurZwO7eKSmkU1f/a3o6oqxGvhFS7GBc2ia5atWZwO7hrbgnv ETfisNv22T2kyMQAqY1x0EsiV9jE+5Oz/tEKsB9/0ytqbr+Bu3b0pZ4PxYmwMrg7FD9U ymJRE57GZnJWDnFLdaNdi6tF/nYldD6UQwYIt8dkh6OmtNSshNcG6hCFbW7KZ2EDp/Ua GY9Q== X-Forwarded-Encrypted: i=1; AJvYcCVBAJ65ATALeiG8KIwrudmE9RpXxEje5enqLZ7EjscmMP7xawPb01QIJextcgn3hONLQKI=@vger.kernel.org X-Gm-Message-State: AOJu0YyoyiIUvSkNAIkZ5X97HsbOoptLDubwumCCdN1956UKRu9/66fT vfx7kN5bG3uJxUiDWBcgpWEragdhuTOFccG/22KdbuXq9NWpn8XtEuwzOpBtJY8f4IW9dphMiwu Qfg== X-Google-Smtp-Source: AGHT+IFy0IhgEaHcg00977nQlGMuEOMnb3oYwKlOFTc/u3M4G7iMNsSXATt5UTOpWjp44DoW3ZbtcVNrt4s= X-Received: from pfbfo6.prod.google.com ([2002:a05:6a00:6006:b0:736:3cd5:ba36]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:8186:b0:736:b400:b58f with SMTP id d2e1a72fcca58-739c3f771bcmr873231b3a.0.1743525801845; Tue, 01 Apr 2025 09:43:21 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 1 Apr 2025 09:34:44 -0700 In-Reply-To: <20250401163447.846608-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250401163447.846608-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.472.ge94155a9ec-goog Message-ID: <20250401163447.846608-6-seanjc@google.com> Subject: [PATCH v2 5/8] KVM: VMX: Process PIR using 64-bit accesses on 64-bit kernels From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jim Mattson Process the PIR at the natural kernel width, i.e. in 64-bit chunks on 64-bit kernels, so that the worst case of having a posted IRQ in each chunk of the vIRR only requires 4 loads and xchgs from/to the PIR, not 8. Deliberately use a "continue" to skip empty entries so that the code is a carbon copy of handle_pending_pir(), in anticipation of deduplicating KVM and posted MSI logic. Suggested-by: Jim Mattson Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 893e7d06e0e6..e4f182ee9340 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -657,26 +657,32 @@ static u8 count_vectors(void *bitmap) bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { - u32 *__pir = (void *)pir; + unsigned long pir_vals[NR_PIR_WORDS]; + u32 *__pir = (void *)pir_vals; u32 i, vec; - u32 pir_val, irr_val, prev_irr_val; + u32 irr_val, prev_irr_val; int max_updated_irr; max_updated_irr = -1; *max_irr = -1; + for (i = 0; i < NR_PIR_WORDS; i++) { + pir_vals[i] = READ_ONCE(pir[i]); + if (!pir_vals[i]) + continue; + + pir_vals[i] = xchg(&pir[i], 0); + } + for (i = vec = 0; i <= 7; i++, vec += 32) { u32 *p_irr = (u32 *)(regs + APIC_IRR + i * 0x10); irr_val = READ_ONCE(*p_irr); - pir_val = READ_ONCE(__pir[i]); - - if (pir_val) { - pir_val = xchg(&__pir[i], 0); + if (__pir[i]) { prev_irr_val = irr_val; do { - irr_val = prev_irr_val | pir_val; + irr_val = prev_irr_val | __pir[i]; } while (prev_irr_val != irr_val && !try_cmpxchg(p_irr, &prev_irr_val, irr_val)); From patchwork Tue Apr 1 16:34:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14035171 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A1AC2144C3 for ; Tue, 1 Apr 2025 16:43:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525805; cv=none; b=i2uDjxLNIzdUOK9No9+MNRNAXZnSv4byEClt1wibAD5P0bXEjVNabOxQQv+4Vahwt7QdONWmsDncNWUjc0cFE7eXOk8pHyRNbmmBC+pzMHyJnXk4GAeUlZj2dFfz0vXEJh+gzfR4DTisDQOvefaCPtm7QXAmDme0j7WIzGYI6H4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525805; c=relaxed/simple; bh=0mb7W2KJR+X1+iO3XRU17z9B6gV64MNgKjj7DrBL3rQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Zq53z4uowIdV/kDf39b86wDUxyMrHNcKXGUuy0LabEKw824G0R5JyZt8kmovbbZVN7KMWccyGahcmuaU1S07sIBVhviznIH2b3FFog4K685yasCv9TW3j5cF18PoowZNS1F0/J/Gk41ywQdhbb7a/aB2ebFAJcz0MdFXzJFaNSQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yKTYDGFw; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yKTYDGFw" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ff58318acaso826537a91.0 for ; Tue, 01 Apr 2025 09:43:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1743525803; x=1744130603; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=JvMZfz//CfX9sBYOfMJB+q+wsvNCB9UZGeb1vhxbwqc=; b=yKTYDGFw7VHBl+U+Jw+Xp1PqBR+1bhJZkrct3n/fRHAeZAqn8xA30ig0dm58liStjJ 0HI0d0hdELhixyO8n+di9f2jPcc8HHu/PCNKJ3k+aZFWZDapAnmOhEiOb1nj/YAgTdEs KLBbjsgK0HauVh0/vERpOcvBgPeNChm7TPvHrrjPAdJKtpZgGFiD+KvP4QJhKsMkzGLZ N6qRoNYFjVc36dLSirXnEVPYfV8m8lW48gFarwsJB1BLqzFS8NwEke1vK2whTWooWi7J TKKDOH1clXCd/1FTmkQWNvRoCykjVeYGBr+9gV1eH5B8zcWwhSKWkSgGkexkgrkdxKU0 rmGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743525803; x=1744130603; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JvMZfz//CfX9sBYOfMJB+q+wsvNCB9UZGeb1vhxbwqc=; b=VR8IZLzDQTU5CVgobCtzHT4e6Uv/NWaHsPtEuTof2rHI1cVlYvUmOo+2iwq525sZCy zC9a4jSYXvpdrk+2ZT+0oEvZysIIW0+V/UcgJ4t3lFl5b95VNLwBy3cGlDM72bkhwRQB QArvqjpXdk1XqBqMjRFWbI4fiyc2zGzuHA9jn6HRXqfHO2C7Izr+YotkPfkf5oPXNldu AxgSA24cnaa2Ex9+PArh4HWM+7mxBW3eeV7R98CEZWIJrrBu41UISMDJzfkkB53QD02k jAdSBj+be8gJsphoo2rAZMgybAD/K2ozPDxXLBqx+6GJf082b0wFnm3EzyfFeQDqcFuj 4zFQ== X-Forwarded-Encrypted: i=1; AJvYcCVC2BZtUQo3SIAZTqNuL+zXlrPUJG2O0N3fd03g3I+YSdmjGS3Dfl0fUixI1fdLrErsXKE=@vger.kernel.org X-Gm-Message-State: AOJu0YyISA7aL/oxfE8foSjDpuCmd2+5S83nE/Snnu4YNNcHmOY/rtOT YH7mWlmaszRt4s+1x4/Y0h8SkDXkyyaBS434YQRuZw4O5eTdXaNk+PAl6Kb3GOJINj072pn9yCt bHQ== X-Google-Smtp-Source: AGHT+IFappA+693mpg8zPDSbFvdj433IwkFt3wteT5TPuG94UFq5oeeszFM3n1+NCGeGHcm5eLfjft4ZDV0= X-Received: from pfbhh11.prod.google.com ([2002:a05:6a00:868b:b0:739:485f:c33e]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:9f47:b0:1f5:67e2:7790 with SMTP id adf61e73a8af0-2009f606767mr20650362637.17.1743525803622; Tue, 01 Apr 2025 09:43:23 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 1 Apr 2025 09:34:45 -0700 In-Reply-To: <20250401163447.846608-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250401163447.846608-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.472.ge94155a9ec-goog Message-ID: <20250401163447.846608-7-seanjc@google.com> Subject: [PATCH v2 6/8] KVM: VMX: Isolate pure loads from atomic XCHG when processing PIR From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jim Mattson Rework KVM's processing of the PIR to use the same algorithm as posted MSIs, i.e. to do READ(x4) => XCHG(x4) instead of (READ+XCHG)(x4). Given KVM's long-standing, sub-optimal use of 32-bit accesses to the PIR, it's safe to say far more thought and investigation was put into handling the PIR for posted MSIs, i.e. there's no reason to assume KVM's existing logic is meaningful, let alone superior. Matching the processing done by posted MSIs will also allow deduplicating the code between KVM and posted MSIs. See the comment for handle_pending_pir() added by commit 1b03d82ba15e ("x86/irq: Install posted MSI notification handler") for details on why isolating loads from XCHG is desirable. Suggested-by: Jim Mattson Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index e4f182ee9340..0463e89376fb 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -657,7 +657,7 @@ static u8 count_vectors(void *bitmap) bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { - unsigned long pir_vals[NR_PIR_WORDS]; + unsigned long pir_vals[NR_PIR_WORDS], pending = 0; u32 *__pir = (void *)pir_vals; u32 i, vec; u32 irr_val, prev_irr_val; @@ -668,6 +668,13 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) for (i = 0; i < NR_PIR_WORDS; i++) { pir_vals[i] = READ_ONCE(pir[i]); + pending |= pir_vals[i]; + } + + if (!pending) + return false; + + for (i = 0; i < NR_PIR_WORDS; i++) { if (!pir_vals[i]) continue; From patchwork Tue Apr 1 16:34:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14035172 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CF65215F76 for ; Tue, 1 Apr 2025 16:43:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525808; cv=none; b=YjHuJuYDGs1+g5dB9PG8gJfzTJ7kj6rqcOM1hikAwEI4BU7ZxPXfwDn82zdrN0DRbMPvW1TMSgS7u+Dc6qBm8Tyt2RUjA78XIeTmRPW/EmBAwne/DrRveuMTIK6bzbtKd6TJnNHGy+xZ4JL2GTOp65egD1C9HdxLEMRwljAfPjY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525808; c=relaxed/simple; bh=+39fVy6ji7kbX1Wk79JiqH4KHEpEfCwzFOyM5dkWHVQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g46m/4mgtghqxWsUHSSXF5comIuy12Q9Gj9fwAyT1DwhJZG0iAPRm2lSKeDmvIvFsmjJAuTEQIfghEAc9krU6s4IFG+CJtJ/KONkKG/FcT+tn9gy7EY++Z1ZQjtcrjRJAZFaxHmQPmCaD7AMnlXotP9hqq8+ZeT3peU4i9Ho8k4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=M/3oiPhQ; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="M/3oiPhQ" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ff6aaa18e8so10160650a91.1 for ; Tue, 01 Apr 2025 09:43:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1743525807; x=1744130607; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=7VeOo+YDl1ZaLAjoRUcMX3QDQbr2P1YpT08wmcSjNJ0=; b=M/3oiPhQKMvRclF7Tg7m0qq5LBKoMhM5lQXQYVVYetodLxjm7HFHekmU6KfYUOM36f 6QA++ysoaTl/vfPcusN4EB1P8auXIoelQoGQl6r8C7fIpw3tgsLHLp9JrcmyocaIjdVo r355KQ3l7fzZQXZuMSS3o3WP+VDZfvblbqyAypDSnWnhqnwM3bk5upeTqKy+eiU/ODa3 hinQXevTgYEO+nwmGwM4pX49ZVeKfz7ZLutwdQIUf0XP/3i3MXatUXCTbYCWcyp3A8QO QHyKWvC9NIMg80R0yWffLxwzO1ujR3JOus2PHw09k+tZACEh2T52rMkVdpZppejy0yqO ZRvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743525807; x=1744130607; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7VeOo+YDl1ZaLAjoRUcMX3QDQbr2P1YpT08wmcSjNJ0=; b=XTgF7IUkf2BEk9u39yUYUoKAIxxDXLUegefADi6tsRSoRzJfYsz9dPoQ58xubCxcD0 D7xj0WbruhmDTf3iUfBS7K8SCZEpNHqhto9znt62BJ5RUHB0Nvo7P00PsxFPpgd9mfLh IF7SS5sj4pZss68Ao+zQZ1Pcc2pQq3QbysdPC63FnOm1TZQ8WwE9J6Dk7qZk6p3LVNSz nc2jKv+PVL6wD0vfiGzNxf3xCi+4FDy75rU4v5rEY6gWNHjmuBP1Vn0VLkn7sHczQVJm UcBpTX1Hv3K5vQGdRwU9ZRaeOpgJQuctxn7kc47X2efthqc2J5x2VcYA4kGyUVNBxuQr cLXA== X-Forwarded-Encrypted: i=1; AJvYcCXmAISapEPaSd5rEZvKwZSk0ev0eqSWNXmuj3VBPgR8705Zj07EdAWLK4MxaGTjv4UipJY=@vger.kernel.org X-Gm-Message-State: AOJu0YwdqoIo53cRLv0JwjiFw/lHmQKnES/5ARmKwbkg8c1mkDMFwQ0P oO4VpGdo5vVBTsYgZbIy8gnxd4P9ywR8ZMGqzEJaUpmACb1RgzElz/HXBbU/ecCj5O8c+NQkU72 gCw== X-Google-Smtp-Source: AGHT+IEFM34tlug3Ergg9oOZ5ia6DKaSJuVy+JKjeggDch7iurgtwMeC43/CRt3s5ueDrR+9rtMhvbS0aMM= X-Received: from pjvf6.prod.google.com ([2002:a17:90a:da86:b0:2fc:2ee0:d38a]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3d45:b0:2fe:e9c6:689e with SMTP id 98e67ed59e1d1-305608771admr5940492a91.8.1743525806882; Tue, 01 Apr 2025 09:43:26 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 1 Apr 2025 09:34:46 -0700 In-Reply-To: <20250401163447.846608-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250401163447.846608-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.472.ge94155a9ec-goog Message-ID: <20250401163447.846608-8-seanjc@google.com> Subject: [PATCH v2 7/8] KVM: VMX: Use arch_xchg() when processing PIR to avoid instrumentation From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jim Mattson Use arch_xchg() when moving IRQs from the PIR to the vIRR, purely to avoid instrumentation so that KVM is compatible with the needs of posted MSI. This will allow extracting the core PIR logic to common code and sharing it between KVM and posted MSI handling. Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 0463e89376fb..13363327ad87 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -678,7 +678,7 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) if (!pir_vals[i]) continue; - pir_vals[i] = xchg(&pir[i], 0); + pir_vals[i] = arch_xchg(&pir[i], 0); } for (i = vec = 0; i <= 7; i++, vec += 32) { From patchwork Tue Apr 1 16:34:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14035173 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D12BB21A437 for ; Tue, 1 Apr 2025 16:43:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525811; cv=none; b=RuFwzC5f+IppPvPWwtESmMq4CoTL2WpFbpn3dIOldXqvoMZY2AYpgVTVBNsDh0m1sfHX+H8WqUmDKtPIMQrCKLrQYvDMx/z4SIGM5kApLRvyYvfjsC8+2jH+sqMRyWQxLsGAKoqBY/bZf6Tx8+QQILryU/6tHMgMoY4sj0vRVas= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1743525811; c=relaxed/simple; bh=OLOwb1ooeYE8kEqHbAiwmAqVs2RQsyUhuURfPkAdAQg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=F4ogm55VWYAU9/LuvIZJhDyT4reu5wshjwDNczJYXq4n7gXS3SftbKNcf8cvwV6mgmUTAOUlMcwbGO+sZYh9O0Xng9HqnHDiWS7FMCLf0D66KMlyA4OCNAR3wWe/1s/ci2bQ6k1Q99DuPFwL09fO1U8U6DJxSakszlyVjHQ5NHU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=lnEeISpW; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lnEeISpW" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2241ae15dcbso134245805ad.0 for ; Tue, 01 Apr 2025 09:43:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1743525808; x=1744130608; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=J4EaWThVax9a+rwMhF6yZ1iQunAjXVQBx8Hy6/tixQ4=; b=lnEeISpW1dK+cEevA+wWDVeAveNKwsnIhg43db/iYyyEQatGqQXlIvSynAg3xPnT7a 1JxZ1GbD79QXmvPJJgfmceLJUhcLCFcsG1jbQmOoo9aIkqMVAEis7LlWwRtizBuduCwm 8mBuxE10G11ZepnKnNf3RcAPqyNOrqXcnA2KaYQFDmT4aQLUjYDlzP13BPuh+xaL6kak rs+we2AFZdYVwURJJ/HO7oFmL16W3fqTs7KFWeDgw/1/km1X1ddvDWUkaG3qJEreGfct TEdw+t+PRA3ed3hWqK04MqDQAPeS6bLnr938k7tLw2aU8wuVxm+AO0A2mhT8/LJ9A9dV 91bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743525808; x=1744130608; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=J4EaWThVax9a+rwMhF6yZ1iQunAjXVQBx8Hy6/tixQ4=; b=sKxWl/KhUNSpyJJ6OMj4NdQowV5uoqmpUNY6EXmGqzzR1CZ5yUeW7V45qOQxEDFhE9 MY3BbRATdXJeYnFr/2s05UAqchYYYOGzdRzyAXvWXjlpZIKOsuvF/BOMs8P4ARTuHC+B s2YFJ3liECE212skn3Mdbzs1DSEVyImPKOG/T3RUZJxG9R/gMPsDPNaHr/gyASiT+055 vyS7VaZxXTOVZagU7cqEGtDCI4wshixzislYp+5X8VZ6SN1Vq4ZoyQC3IyN77BRp1Xui 1e4PGsKXzxa/WFl+RXvB5H5fdvXdPTiP87oSupEznwTZF+q7bOWEYz7qOF2VoFfucaSc feoA== X-Forwarded-Encrypted: i=1; AJvYcCXHZTxaJ1Lmu6zoj1sanj/GfiAgtBAOFu52BvdcxC2jtAyNhMS+bmghl+xdhHSyfNi+Yw4=@vger.kernel.org X-Gm-Message-State: AOJu0YwCfzItz/BpF1SCGVZLzQNGYpp25uGGViNqyFzJKzU4rIRZU6Ro etXZSHp9vuKlGiyIN/MkmnRk7tRm8MLMEe7WMhwez6oNV8JR9ccxddAwd6PT/rmxwCdiZa6Z9ZJ JZQ== X-Google-Smtp-Source: AGHT+IG+549uJ+p5h+jsssZ8JRnj/shC1446VEr9mloO2RqS2MvoVJe5+Bo4jdQs31e7cn/R3826qBseTE8= X-Received: from pfbde10.prod.google.com ([2002:a05:6a00:468a:b0:730:7b0c:592c]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2383:b0:736:34a2:8a18 with SMTP id d2e1a72fcca58-7398044db06mr17829997b3a.24.1743525808635; Tue, 01 Apr 2025 09:43:28 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 1 Apr 2025 09:34:47 -0700 In-Reply-To: <20250401163447.846608-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250401163447.846608-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.472.ge94155a9ec-goog Message-ID: <20250401163447.846608-9-seanjc@google.com> Subject: [PATCH v2 8/8] x86/irq: KVM: Add helper for harvesting PIR to deduplicate KVM and posted MSIs From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jim Mattson Now that posted MSI and KVM harvesting of PIR is identical, extract the code (and posted MSI's wonderful comment) to a common helper. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/posted_intr.h | 64 ++++++++++++++++++++++++++++++ arch/x86/kernel/irq.c | 50 ++--------------------- arch/x86/kvm/lapic.c | 16 +------- 3 files changed, 69 insertions(+), 61 deletions(-) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h index c3e6e4221a5b..3198e4f217dc 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -1,8 +1,13 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef _X86_POSTED_INTR_H #define _X86_POSTED_INTR_H + +#include +#include #include +#include + #define POSTED_INTR_ON 0 #define POSTED_INTR_SN 1 @@ -26,6 +31,65 @@ struct pi_desc { u32 rsvd[6]; } __aligned(64); +/* + * De-multiplexing posted interrupts is on the performance path, the code + * below is written to optimize the cache performance based on the following + * considerations: + * 1.Posted interrupt descriptor (PID) fits in a cache line that is frequently + * accessed by both CPU and IOMMU. + * 2.During software processing of posted interrupts, the CPU needs to do + * natural width read and xchg for checking and clearing posted interrupt + * request (PIR), a 256 bit field within the PID. + * 3.On the other side, the IOMMU does atomic swaps of the entire PID cache + * line when posting interrupts and setting control bits. + * 4.The CPU can access the cache line a magnitude faster than the IOMMU. + * 5.Each time the IOMMU does interrupt posting to the PIR will evict the PID + * cache line. The cache line states after each operation are as follows, + * assuming a 64-bit kernel: + * CPU IOMMU PID Cache line state + * --------------------------------------------------------------- + *...read64 exclusive + *...lock xchg64 modified + *... post/atomic swap invalid + *...------------------------------------------------------------- + * + * To reduce L1 data cache miss, it is important to avoid contention with + * IOMMU's interrupt posting/atomic swap. Therefore, a copy of PIR is used + * when processing posted interrupts in software, e.g. to dispatch interrupt + * handlers for posted MSIs, or to move interrupts from the PIR to the vIRR + * in KVM. + * + * In addition, the code is trying to keep the cache line state consistent + * as much as possible. e.g. when making a copy and clearing the PIR + * (assuming non-zero PIR bits are present in the entire PIR), it does: + * read, read, read, read, xchg, xchg, xchg, xchg + * instead of: + * read, xchg, read, xchg, read, xchg, read, xchg + */ +static __always_inline bool pi_harvest_pir(unsigned long *pir, + unsigned long *pir_vals) +{ + unsigned long pending = 0; + int i; + + for (i = 0; i < NR_PIR_WORDS; i++) { + pir_vals[i] = READ_ONCE(pir[i]); + pending |= pir_vals[i]; + } + + if (!pending) + return false; + + for (i = 0; i < NR_PIR_WORDS; i++) { + if (!pir_vals[i]) + continue; + + pir_vals[i] = arch_xchg(&pir[i], 0); + } + + return true; +} + static inline bool pi_test_and_set_on(struct pi_desc *pi_desc) { return test_and_set_bit(POSTED_INTR_ON, (unsigned long *)&pi_desc->control); diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 3753a8acd009..f7c6d94cf070 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -375,58 +375,14 @@ void intel_posted_msi_init(void) this_cpu_write(posted_msi_pi_desc.ndst, destination); } -/* - * De-multiplexing posted interrupts is on the performance path, the code - * below is written to optimize the cache performance based on the following - * considerations: - * 1.Posted interrupt descriptor (PID) fits in a cache line that is frequently - * accessed by both CPU and IOMMU. - * 2.During posted MSI processing, the CPU needs to do 64-bit read and xchg - * for checking and clearing posted interrupt request (PIR), a 256 bit field - * within the PID. - * 3.On the other side, the IOMMU does atomic swaps of the entire PID cache - * line when posting interrupts and setting control bits. - * 4.The CPU can access the cache line a magnitude faster than the IOMMU. - * 5.Each time the IOMMU does interrupt posting to the PIR will evict the PID - * cache line. The cache line states after each operation are as follows: - * CPU IOMMU PID Cache line state - * --------------------------------------------------------------- - *...read64 exclusive - *...lock xchg64 modified - *... post/atomic swap invalid - *...------------------------------------------------------------- - * - * To reduce L1 data cache miss, it is important to avoid contention with - * IOMMU's interrupt posting/atomic swap. Therefore, a copy of PIR is used - * to dispatch interrupt handlers. - * - * In addition, the code is trying to keep the cache line state consistent - * as much as possible. e.g. when making a copy and clearing the PIR - * (assuming non-zero PIR bits are present in the entire PIR), it does: - * read, read, read, read, xchg, xchg, xchg, xchg - * instead of: - * read, xchg, read, xchg, read, xchg, read, xchg - */ static __always_inline bool handle_pending_pir(unsigned long *pir, struct pt_regs *regs) { - unsigned long pir_copy[NR_PIR_WORDS], pending = 0; - int i, vec = FIRST_EXTERNAL_VECTOR; + unsigned long pir_copy[NR_PIR_WORDS]; + int vec = FIRST_EXTERNAL_VECTOR; - for (i = 0; i < NR_PIR_WORDS; i++) { - pir_copy[i] = READ_ONCE(pir[i]); - pending |= pir_copy[i]; - } - - if (!pending) + if (!pi_harvest_pir(pir, pir_copy)) return false; - for (i = 0; i < NR_PIR_WORDS; i++) { - if (!pir_copy[i]) - continue; - - pir_copy[i] = arch_xchg(&pir[i], 0); - } - for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) call_irq_handler(vec, regs); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 13363327ad87..1c611b84b8ab 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -657,7 +657,7 @@ static u8 count_vectors(void *bitmap) bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { - unsigned long pir_vals[NR_PIR_WORDS], pending = 0; + unsigned long pir_vals[NR_PIR_WORDS]; u32 *__pir = (void *)pir_vals; u32 i, vec; u32 irr_val, prev_irr_val; @@ -666,21 +666,9 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) max_updated_irr = -1; *max_irr = -1; - for (i = 0; i < NR_PIR_WORDS; i++) { - pir_vals[i] = READ_ONCE(pir[i]); - pending |= pir_vals[i]; - } - - if (!pending) + if (!pi_harvest_pir(pir, pir_vals)) return false; - for (i = 0; i < NR_PIR_WORDS; i++) { - if (!pir_vals[i]) - continue; - - pir_vals[i] = arch_xchg(&pir[i], 0); - } - for (i = vec = 0; i <= 7; i++, vec += 32) { u32 *p_irr = (u32 *)(regs + APIC_IRR + i * 0x10);