From patchwork Tue Jun 4 05:09:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684640 Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F0D977A1E; Tue, 4 Jun 2024 05:10:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477840; cv=none; b=VyvdtlMHdYbmbKG49SicS5Uh9BLTF/B6AoUMDQnwAGNW6UukQQli+AXvEqW+fwCm7klTd5YXTsX7q1qsIuSLymWU6a7GyqP41pxitDxV+7phiO66kYTVEljykmZRzMXb+LbgQnHmyYH6kVaqGVNeg7+O3UKCXSlpzV7gvAmKWII= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477840; c=relaxed/simple; bh=VBR98JGVknh0QIXU1LbDkiwLPZeprLJur5lg0RvecfE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dXzi8chAOHKg5RPzMD14fmsjxbYOtYi3aYp9aWvZdC1I6IQ+fUWbfb0utLbYR0NcTZaWgq0JCk/59qXI77860zOSs/s0f6R8LT1Cqo4jIEgGfgFTL6wgYHN54pefALlLCp9QY7DDkkvplgxIwYbBxd4AGbEqxA1mqjb7jef6da0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=d1EaCc/x; arc=none smtp.client-ip=209.85.167.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="d1EaCc/x" Received: by mail-oi1-f180.google.com with SMTP id 5614622812f47-3d1facdf12bso224547b6e.0; Mon, 03 Jun 2024 22:10:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477837; x=1718082637; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=gOw8AgrUJDkpfPA0Lzrm6o/e4yMy7flaTek+BD0V1x4=; b=d1EaCc/xzVsuRc/jIMk93NC6Rd6JYq4x6wAQ/2xKJGnTESMLLInq/rxVQauFaUmav/ 3xZdm+/uIBg8su5/+0O8Z62p8N4MlMnzB/e8SA5xo/kffL0WHwKSt7WWjA/2bftmECUg yb880VnAD4J8fVqa6qK+RmHbtrRBJC/kdtyCP8WO8VdsOXlsIdB+5K/q4WxxbDX/lJY/ 9DkFqjY8NOQ2vMOZZ07DgJCdBdUrLxj2rbU1TdDNWIyG/zRDAwUFk7p2Zq7EO+wUXoLn mgZn7rBGJhm31ouvZWoGXFFJk4uES6TR8/ObRocQVcZ72W8ol+T3GLgDWUqv1OwgaeAS VuMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477837; x=1718082637; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gOw8AgrUJDkpfPA0Lzrm6o/e4yMy7flaTek+BD0V1x4=; b=SqeXS0MK57/bEj9MsENLYM7Ju+k+/LfEtWSFQqjBziSw376gktb7jPT/PF3y9MuHUr x+jyM5DrkG3A1RaHrCW2Xz+BJSxFKd6la3PSbTFnK10uo0VC3mYFP1ZEDWHg/3fecJ1Q 6kayjbY3CvbipB1xGkDPZXnTL0gK8eRI4r7Xd+dyX9rV/eLPSggVtb5jZ615JIpUywki dZI1HzVbvZs8E8cQ6oXxwcR5dYow1AdEcVCWH4rU7lphUON3ZxOJCudLPm7nh5zqgI3a kezgyrLrSGusOqUTn3ZmAS4eYt2sb2wOtpPt89HwQ5RvB26OzJySY6hTpzbj67IIO1dN H6HA== X-Forwarded-Encrypted: i=1; AJvYcCVcQo7+yDJzz5X0FPrSDUUUSnsP3KB/cN/MCebfzUJysqCYcYKsYEL37EEpB3guQb+xK2UMhGjEoz+Iur7/QshjE4n23ltDbP3R+hxwo7WBIhjbW/jcroqQ9c+4w7opxDaQ+naGKT4p2ClPY2Z1MhdtLF2kQKilugW637nUupQtoo8q81KDG/80we+rt3cenSKtaC+HMmAqIVhaCYuwnKOuTApxF+Z1jkKXVLJYyFKRMUA7PcJWHRFiE7CSYBM= X-Gm-Message-State: AOJu0Yyw4csjZmAHEZvO/SxyOgLbWF38UGqIbPp0WDvkl3SD3z+s9UaC 1THVUQqjIHAfJ/LyhHPij2QjyWlVah1+ljJHukeE5EkqEnod9Evv X-Google-Smtp-Source: AGHT+IEyA+H/NUVYyadV6QS+sJdPE+MGaP8srmU6cCY3RydFGapT11Ew5xGOL60RZNIYPFbiiOl3Sw== X-Received: by 2002:a05:6808:602:b0:3d2:5b:df2d with SMTP id 5614622812f47-3d2005be0acmr41704b6e.35.1717477837405; Mon, 03 Jun 2024 22:10:37 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:37 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 01/12] Drivers: hv: vmbus: Drop unsupported VMBus devices earlier Date: Mon, 3 Jun 2024 22:09:29 -0700 Message-Id: <20240604050940.859909-2-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley Because Hyper-V doesn't know ahead-of-time what operating system will be running in a guest VM, it may offer to Linux guests VMBus devices that are intended for use only in Windows guests. Currently in Linux, VMBus channels are created for these devices, and they are processed by vmbus_device_register(). While nothing further happens because no matching driver is found, the channel continues to exist. To avoid having the spurious channel, drop such devices immediately in vmbus_onoffer(), based on identifying the device in the vmbus_unsupported_devs table. If Hyper-V should issue a rescind request for the device, no matching channel will be found and the rescind will also be ignored. Since unsupported devices are dropped early, the check for unsupported devices in hv_get_dev_type() is redundant. Remove it. Signed-off-by: Michael Kelley Acked-by: Wei Liu --- drivers/hv/channel_mgmt.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c index 3c6011a48dab..a216a0aede73 100644 --- a/drivers/hv/channel_mgmt.c +++ b/drivers/hv/channel_mgmt.c @@ -203,7 +203,7 @@ static u16 hv_get_dev_type(const struct vmbus_channel *channel) const guid_t *guid = &channel->offermsg.offer.if_type; u16 i; - if (is_hvsock_channel(channel) || is_unsupported_vmbus_devs(guid)) + if (is_hvsock_channel(channel)) return HV_UNKNOWN; for (i = HV_IDE; i < HV_UNKNOWN; i++) { @@ -1036,6 +1036,16 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr) trace_vmbus_onoffer(offer); + /* + * Hyper-V may offer pseudo-devices with functionality intended for + * Windows guests. Silently ignore them. There's no value in + * cluttering dmesg with error messages. + */ + if (is_unsupported_vmbus_devs(&offer->offer.if_type)) { + atomic_dec(&vmbus_connection.offer_in_progress); + return; + } + if (!vmbus_is_valid_offer(offer)) { pr_err_ratelimited("Invalid offer %d from the host supporting isolation\n", offer->child_relid); From patchwork Tue Jun 4 05:09:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684641 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65F3B13D28B; Tue, 4 Jun 2024 05:10:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477840; cv=none; b=oZzszEqsUvGHhPemDDthC7hmX/2TP8TrHnIG/f3bVu60oj70ORBC/4uCcs7CZtIRBixMyd5tZsvc+/Q2TPuy6RysSggRwwZn71+VFtsbTzewyzp7uSsjXa59CVk6w0T7F489XOL8pV/pQw1EmHbEWhEGiPcPPIqspMdXsi4VIS4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477840; c=relaxed/simple; bh=4Bvo+apk2N2nn6cjDyl5CQhxEltWUBZFYmNRgBJhxgI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fOs1qTJGLHigeADaGDOTFCfcXa/u48g4+jpD0dG5XX2hnJsP4/+X4Ze832lMT75YxNOp+A7IEI6mLolPk0qcu48uwWBrYMzJVJMykQ9ri3s3x0+T+7uu6N/cno7AVX9k+uaAZ+q/tqN5aQa+TOlLN37K1Y0xhFUjGZh0MU5ioeI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U01oW3qS; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U01oW3qS" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-6c7bf648207so2258002a12.0; Mon, 03 Jun 2024 22:10:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477839; x=1718082639; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=5YKAeu5Z5tm9aQZs9F3XxaISr/2oY38/ITUqBxWif4w=; b=U01oW3qSe827FcEeFRmM9nlmpBduxt7BIH7qq+7CH+XDFDJz23Xc0/Lm9RqSherobX IEoaPgOTQ2V5ZBJFFln5H74MazOv/LB1kYpueZpkiUu5vsZertJpS6wE8UHiABbsm3RR PzDNrZKMJhGohS9d5C4Clg//Em1mhcQIqVTVqSMe7CwvTV5OlAgNTU2sJem7NNwUbabF K0oWt12jR2AvSnqfsk7NfB8CjJVX/JZGjYJLQSycGzE7ANwL1EM4BUZdmyBMlrP7fTXD k9r7cXBV+XlFTcxtejfQ6hP3aoJzRVrhOsHFd1fpA57GWoLZlcv3/dFurzX4ZG7I8035 a+sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477839; x=1718082639; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=5YKAeu5Z5tm9aQZs9F3XxaISr/2oY38/ITUqBxWif4w=; b=m/04jISTOAy3AAf3LLsbCpD7MfL7toh1fIxp8cSAlqVR6Hb+YemdQNgc8i5O8FmngF A3ez8Y3Kwc5t2RIXJ5jhj0IBdHW7g9yx+ebF3fQOlzMqyzYXkgUYRaO7/OduJrFdno5N 7LABe8RJH9RV5iDtwbaJYFDh4sUW7ocMdxfEaVgVX5bRe0LoQJuxSOtQLp0fsLTRY6VQ 5s67tdOuJq8lBJOy1+yUhxKEMkL9us6WmVt12BXNHnG7xKO4hby3uwXjWdYkglgVgZZd Qgr4/x9d4x5XflRlfy7f3ebvywZlNQV0M/0Tcks4iGMi1Ey0MPbWZPnizFjmyl8AChdd e66w== X-Forwarded-Encrypted: i=1; AJvYcCXu6eubyj3b19jh/XfmYGuO6MIifWSvyiTcNEc/PnMUwk1RCEJiuf+7aljI/xj3Mk4Pm3aQ/KVR+89LoIZ05NpFzdPC2SghmOpSsanbBA5os535I2rkWrRI8NZnvBUkfvfFW6YV1ngATIpec1siMolI6lu57iqJFgtkTyr+cMMkucVEJ/r/FPL7d6JWpD18zuSS+g3mFxy+13BmO53YCzkfihoRJhkNcxrOfAoKT7UjhU7ZFjTSoCkSUTdpaPY= X-Gm-Message-State: AOJu0YzohobAEKnfBJh6uZFJIdmRI36ikaGvdwYwhAAWuUNoblaHDHTV OjF3Gwfws3LSJ37xnLslhN1juCueL4KD1DRPZkFgaZ04eVIk+3FD X-Google-Smtp-Source: AGHT+IHKnRboB5tCmxwmDAJRCzrZ1L4NYqoQuhFMq5JMjnWnuknQC44+KcCK4lbhV5B4umVkIn2zaQ== X-Received: by 2002:a05:6a20:9184:b0:1a3:b642:5fc3 with SMTP id adf61e73a8af0-1b26f253adamr14486371637.41.1717477838675; Mon, 03 Jun 2024 22:10:38 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:38 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 02/12] Drivers: hv: vmbus: Fix error path that deletes non-existent sysfs group Date: Mon, 3 Jun 2024 22:09:30 -0700 Message-Id: <20240604050940.859909-3-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley If vmbus_device_create() returns an error to vmbus_add_channel_work(), the cleanup path calls free_channel(), which in turn calls vmbus_remove_channel_attr_group(). But the channel attr group hasn't been created yet, causing sysfs_remove_group() to generate multiple WARNs about non-existent entries. Fix the WARNs by adding a flag to struct vmbus_channel to indicate whether the sysfs group for the channel has been created. Use the flag to determine if the sysfs group should be removed. Signed-off-by: Michael Kelley --- drivers/hv/vmbus_drv.c | 5 ++++- include/linux/hyperv.h | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index 12a707ab73f8..291a8358370b 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -1842,6 +1842,7 @@ int vmbus_add_channel_kobj(struct hv_device *dev, struct vmbus_channel *channel) dev_err(device, "Unable to set up channel sysfs files\n"); return ret; } + channel->channel_attr_set = true; kobject_uevent(kobj, KOBJ_ADD); @@ -1853,7 +1854,9 @@ int vmbus_add_channel_kobj(struct hv_device *dev, struct vmbus_channel *channel) */ void vmbus_remove_channel_attr_group(struct vmbus_channel *channel) { - sysfs_remove_group(&channel->kobj, &vmbus_chan_group); + if (channel->channel_attr_set) + sysfs_remove_group(&channel->kobj, &vmbus_chan_group); + channel->channel_attr_set = false; } /* diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h index 5e39baa7f6cb..d52c916cc492 100644 --- a/include/linux/hyperv.h +++ b/include/linux/hyperv.h @@ -980,6 +980,7 @@ struct vmbus_channel { * For sysfs per-channel properties. */ struct kobject kobj; + bool channel_attr_set; /* * For performance critical channels (storage, networking From patchwork Tue Jun 4 05:09:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684642 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F47F13E05B; Tue, 4 Jun 2024 05:10:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477842; cv=none; b=e0ldZiF/PTOsxaz/oiwlDz3fdml5x3YNdzpWlnN4VgO/UEyEVbRekclONmJenAS79ZKJ2zUIm9D0mtXTcwD3spKbwujIwXn0/uiAz8IDn4tsQ0KwRAetCzGYZn2ctOdIm9o2vqGk/n9YPcpl9mOLMG4SRf0TphUJXvXtdRM5T4s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477842; c=relaxed/simple; bh=zSe+eD0qV+e71QCOWMNlBwLmbidtc2MRHQzXAFw6Uj0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tG1B+NU8M0rftQfv03G2JtiCImCDAKZi2dBfa7EZPoozF+bWMDiZAHDHfulcey+b5tTHowR1EOmVMdxsjTYMDMql6qcTpoN3A6lGNnA3HoULNyCYRAoOSTTBW5l3RP1uMcDbrfQDO1+9P0E6ft68QGj23/uFp4Qr0zmJdyodtHU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KaHnNqYa; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KaHnNqYa" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-70276322ad8so649008b3a.1; Mon, 03 Jun 2024 22:10:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477840; x=1718082640; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Tyt2LCARZb9G8yUFB+4R0wf0B3JqyvqSEeHJTztg/6g=; b=KaHnNqYa6zww4Gs0ITsJICh+2jdTpNEqbSaT8vj8aXA1FzDPxx0wyvfFOoJz6vsnAI bIP9NtGE/rhQimPSFyeV+X6eyltXiKTxcZyt7oI7Z2q2Euagsrchtkv6UQV9kV6UWk5w Sslp39Zbar4B23I3PX9j0Ax1uQVFlq/CaB4+LW/xuWWDP9Pzysi3lLi7/CozHikUjXma BgLo6CbQsBvKCzizKJbVb02DZKSn0zd/7Bx4mKKK/SVStHC+9g553hAfvMZAOZ91v1J2 xQsg1cwLXVdwErhza6mEcDJEsk4Iu8J4idlDi3jrOmlmWSTmnKSJWzlvRKeO71PXA6P8 wVrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477840; x=1718082640; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Tyt2LCARZb9G8yUFB+4R0wf0B3JqyvqSEeHJTztg/6g=; b=OOBUnWtI2Vz7+DH5fajOqm/oxadjVX3BLRtvDhSLz57jsSj5DdY+isssM6f+EkhuFg M0Jp5nek+oM+Ikn1RhXZFkENyrZzmTLUDeAAEjVgkutqUXGafQ/Iocj3m6JISwTCCVcR SCX83i2L0hhJ/QCrHDJ5Rl15/ams+FgdjqylLErDS58ns61H3PPJQWtg0NflQJxEMlh1 WkwfmdFTRJxJoP3M7ysKjqoSRugVU1uloH5ccw8npeqKjzYJ5y1dBpm8get/6EvUOpBV 1OdIf0u3K3VPcDZR07g82K8V5lWhXWgV+pdXZOd2vJhwID0JEG03xEY+mBHAGUJVVqf+ 4yzA== X-Forwarded-Encrypted: i=1; AJvYcCV+6Yn5P8tDv/GNAGGh/DhpOewAWjfQmXyWr/2XiFJym8rkXiw7pdPg5f/OtVID5+B4oTl++hKpiAsGBVA/NUpEsPjd7A9MKdEOE3wtYuxgWS1HY2pwiE3OUTPJlVvbWffjW1Zc9LYAeN8MbrImzFXpoYzJ8lgWT8n+JQXszbazVgqsFOd42elSZnbe/+iRyewxL9rVZzew8BwE3zdjJaYmzGIT691VC33L6v+OAfPUZUXfLhL3Hkc5ic30aRs= X-Gm-Message-State: AOJu0YyJpdOyGVO6dMe3B1x8mgDO+K88apQyuBA+g7TsfY3ZFw3G6jip FosOxmFpgacuDJkIQTUu+A7kDXuTp6AFLrXGfws1PoSUasKjAsfj X-Google-Smtp-Source: AGHT+IFhTA5PSRFrRUUXGMk5zEUWP+kSdnmr7igsWz9177BNVWX0y0FgoWXqzbK+Nj7MFqlG3jitow== X-Received: by 2002:a05:6a00:230a:b0:6f3:f970:9f2a with SMTP id d2e1a72fcca58-702477e69femr12966122b3a.10.1717477839971; Mon, 03 Jun 2024 22:10:39 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:39 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 03/12] Drivers: hv: vmbus: Add an IRQ name to VMBus channels Date: Mon, 3 Jun 2024 22:09:31 -0700 Message-Id: <20240604050940.859909-4-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley In preparation for assigning Linux IRQs to VMBus channels, assign a string name to each channel. The name is used when the IRQ is requested, and it will appear in the output of /proc/interrupts to make it easier to identify the device the IRQ is associated with. Many VMBus devices are single-instance, with a single channel. For such devices, a default string name can be determined based on the GUID identifying the device. So add default names to the vmbus_devs table that lists recognized devices. When a channel is created, set the channel name to that default so a reasonable name is displayed without requiring VMBus driver modifications. However, individual VMBus device drivers may be optionally modified to override the default name to provide additional information. In cases where a driver supports multiple instances of a device, or a device has multiple channels, the additional information may be helpful to display in /proc/interrupts. Signed-off-by: Michael Kelley --- drivers/hv/channel_mgmt.c | 45 +++++++++++++++++++++++++++++++-------- include/linux/hyperv.h | 5 +++++ 2 files changed, 41 insertions(+), 9 deletions(-) diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c index a216a0aede73..adbe184e5197 100644 --- a/drivers/hv/channel_mgmt.c +++ b/drivers/hv/channel_mgmt.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -33,6 +34,7 @@ const struct vmbus_device vmbus_devs[] = { HV_IDE_GUID, .perf_device = true, .allowed_in_isolated = false, + .irq_name = "storvsc", }, /* SCSI */ @@ -40,6 +42,7 @@ const struct vmbus_device vmbus_devs[] = { HV_SCSI_GUID, .perf_device = true, .allowed_in_isolated = true, + .irq_name = "storvsc", }, /* Fibre Channel */ @@ -47,6 +50,7 @@ const struct vmbus_device vmbus_devs[] = { HV_SYNTHFC_GUID, .perf_device = true, .allowed_in_isolated = false, + .irq_name = "storvsc", }, /* Synthetic NIC */ @@ -54,6 +58,7 @@ const struct vmbus_device vmbus_devs[] = { HV_NIC_GUID, .perf_device = true, .allowed_in_isolated = true, + .irq_name = "netvsc", }, /* Network Direct */ @@ -61,6 +66,7 @@ const struct vmbus_device vmbus_devs[] = { HV_ND_GUID, .perf_device = true, .allowed_in_isolated = false, + .irq_name = "netdirect", }, /* PCIE */ @@ -68,6 +74,7 @@ const struct vmbus_device vmbus_devs[] = { HV_PCIE_GUID, .perf_device = false, .allowed_in_isolated = true, + .irq_name = "vpci", }, /* Synthetic Frame Buffer */ @@ -75,6 +82,7 @@ const struct vmbus_device vmbus_devs[] = { HV_SYNTHVID_GUID, .perf_device = false, .allowed_in_isolated = false, + .irq_name = "framebuffer", }, /* Synthetic Keyboard */ @@ -82,6 +90,7 @@ const struct vmbus_device vmbus_devs[] = { HV_KBD_GUID, .perf_device = false, .allowed_in_isolated = false, + .irq_name = "keyboard", }, /* Synthetic MOUSE */ @@ -89,6 +98,7 @@ const struct vmbus_device vmbus_devs[] = { HV_MOUSE_GUID, .perf_device = false, .allowed_in_isolated = false, + .irq_name = "mouse", }, /* KVP */ @@ -96,6 +106,7 @@ const struct vmbus_device vmbus_devs[] = { HV_KVP_GUID, .perf_device = false, .allowed_in_isolated = false, + .irq_name = "kvp", }, /* Time Synch */ @@ -103,6 +114,7 @@ const struct vmbus_device vmbus_devs[] = { HV_TS_GUID, .perf_device = false, .allowed_in_isolated = true, + .irq_name = "timesync", }, /* Heartbeat */ @@ -110,6 +122,7 @@ const struct vmbus_device vmbus_devs[] = { HV_HEART_BEAT_GUID, .perf_device = false, .allowed_in_isolated = true, + .irq_name = "heartbeat", }, /* Shutdown */ @@ -117,6 +130,7 @@ const struct vmbus_device vmbus_devs[] = { HV_SHUTDOWN_GUID, .perf_device = false, .allowed_in_isolated = true, + .irq_name = "shutdown", }, /* File copy */ @@ -126,6 +140,7 @@ const struct vmbus_device vmbus_devs[] = { HV_FCOPY_GUID, .perf_device = false, .allowed_in_isolated = false, + .irq_name = "fcopy", }, /* Backup */ @@ -133,6 +148,7 @@ const struct vmbus_device vmbus_devs[] = { HV_VSS_GUID, .perf_device = false, .allowed_in_isolated = false, + .irq_name = "vss", }, /* Dynamic Memory */ @@ -140,6 +156,7 @@ const struct vmbus_device vmbus_devs[] = { HV_DM_GUID, .perf_device = false, .allowed_in_isolated = false, + .irq_name = "balloon", }, /* @@ -198,20 +215,30 @@ static bool is_unsupported_vmbus_devs(const guid_t *guid) return false; } -static u16 hv_get_dev_type(const struct vmbus_channel *channel) +static void hv_set_dev_type_and_irq_name(struct vmbus_channel *channel) { const guid_t *guid = &channel->offermsg.offer.if_type; + char *name = NULL; u16 i; - if (is_hvsock_channel(channel)) - return HV_UNKNOWN; - - for (i = HV_IDE; i < HV_UNKNOWN; i++) { - if (guid_equal(guid, &vmbus_devs[i].guid)) - return i; + if (is_hvsock_channel(channel)) { + i = HV_UNKNOWN; + name = "hv_sock"; + goto found; } + + for (i = HV_IDE; i < HV_UNKNOWN; i++) + if (guid_equal(guid, &vmbus_devs[i].guid)) { + name = vmbus_devs[i].irq_name; + goto found; + } + pr_info("Unknown GUID: %pUl\n", guid); - return i; + +found: + channel->device_id = i; + if (name) + strscpy(channel->irq_name, name, VMBUS_CHAN_IRQ_NAME_MAX); } /** @@ -970,7 +997,7 @@ static void vmbus_setup_channel_state(struct vmbus_channel *channel, sizeof(struct vmbus_channel_offer_channel)); channel->monitor_grp = (u8)offer->monitorid / 32; channel->monitor_bit = (u8)offer->monitorid % 32; - channel->device_id = hv_get_dev_type(channel); + hv_set_dev_type_and_irq_name(channel); } /* diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h index d52c916cc492..897d96fa19d4 100644 --- a/include/linux/hyperv.h +++ b/include/linux/hyperv.h @@ -826,6 +826,7 @@ struct vmbus_device { guid_t guid; bool perf_device; bool allowed_in_isolated; + char *irq_name; }; #define VMBUS_DEFAULT_MAX_PKT_SIZE 4096 @@ -837,6 +838,8 @@ struct vmbus_gpadl { bool decrypted; }; +#define VMBUS_CHAN_IRQ_NAME_MAX 32 + struct vmbus_channel { struct list_head listentry; @@ -1068,6 +1071,8 @@ struct vmbus_channel { /* The max size of a packet on this channel */ u32 max_pkt_size; + + char irq_name[VMBUS_CHAN_IRQ_NAME_MAX]; }; #define lock_requestor(channel, flags) \ From patchwork Tue Jun 4 05:09:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684643 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D40FF13EFE5; Tue, 4 Jun 2024 05:10:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477843; cv=none; b=S/ouQervdfAEVovCTjOKiPJDUpHq7kqtnDxWF3JnNGvllS8valdiFaH/0TQkzuFHGnZYXSC+LevJEts6xecPFc53pQm6racSKfQG4O6ICynUyFs9T0lmZqsXCr3OK+9r7BX0opVZJc2279d0Bbm9JBGWm5ysSxSCKRyuIJnmgm8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477843; c=relaxed/simple; bh=NDQ/4bdVCsdeoYBuw6VFx+6yBgoNk62i9zXSX9enGps=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rpzfPwslRIklIZsXaoHXR7Ii3Zndtm6w42QSH8PzgFd3cXZwVqL+2dHhFwsjFU8noy919H2THwg4DAzm2WYs0TrjUGXWil6eT7r2vAK8UwufIK2ZmMvtaIR2OWTobbWS/52Bhk5q0JN9lJtIEgQn82OvrCxNqGylHj/qJIBx/KE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SMaep3ek; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SMaep3ek" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-7025e7a42dcso667422b3a.1; Mon, 03 Jun 2024 22:10:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477841; x=1718082641; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=BAofxEUtzPzqe88fnpeV5QdjFS9qkBRDgafLpMrolX4=; b=SMaep3ekQV5KBQ+VYoqJRXj5dgVtAoXpABTo4SV0ERCy3XSOFXPfJ56luqOtj5c61d OP35ZnjoywgYSlc68n9VMTnhhm2+t/DjnHHdBTX3oAvGb8bSGWHUabTqOmvUdUqokm/o dfKfw8k5KDdT4LoBfG98aJS+YtB5Y+9IvynNPHeDT5KOSE+LL/I+1S3kSjgwYnpwk6Mh epDAcn8uSTEkqxUcmcUt2ZQINp4C+j2oAmYdFFPtJwLEYhkFE61buNJiX7RA8Ah5hBZL aw9rpOpAfSOSbt+F2T2MnWwEsE+HfC9cxtyVolJLYcx4lmiFNKN8rl+TKHgU1Y0Y6yvd slcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477841; x=1718082641; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=BAofxEUtzPzqe88fnpeV5QdjFS9qkBRDgafLpMrolX4=; b=Qn/M6xNdmeU0OycO9o4Px+gFsujtYYA4PSxp7kb5641yZ3wIRutLGhUFSzgiuelhFl PAkY6svQ8rAtdbDYxTRagurPiOMtxWHH1pJwn2atCvJTrjn+DlOvxqRtTvlmPPls1IbQ ZhzwMoEIChxIklqNG3PhhS7UEsH6OVxKtNz/f1cSo1U4Hpp+apmUhPnwGTYzImXXLLPq 5f8FtLqvDGrOcOW2/TbwLgUqnHsCGN4OKxvhDiSGo4YNsgUCNbfjBT7WtMpF3cdbVNuS qaWFe8bcL9KgAuouiJRSH9e1AKgWHKGPeB+jPORH4bgwmcssTdsQ9NGxSRzyEnUCZ4ik 5Jcw== X-Forwarded-Encrypted: i=1; AJvYcCV3Cp5BjfAJf+/eYeAb8MfEyQcYVuWuEYpd+Jli6KKnlEBdi/RSoG65k72I/P8eTgtqRmIYcOMEmp6tdkiJ4GDWMN7imlQUlBLUH86oYfx4d9eBzUafnCw3nlwBY28LYHbZmz0darye5NCLGDJqMzmCAhM7D/2TvHrZ+CumOU6oXnp+eR4aP7kqXs8vlnKdeXmYtdvmWe5ZM3ODGc9JbsH87ttL97dtawvZEtaRldWE25ZRjWzGzIdCqqEJDRo= X-Gm-Message-State: AOJu0YwihcUTZvovuBLkVx04v/xUQXhuS5lJLFLXcGTVMWKtmw15l3rR EvmjTcYM7LhAH9AzMMB6i3HEqMRm1lMyccjyv5fIy1RQR6ypENv5 X-Google-Smtp-Source: AGHT+IHYq1EjkLSNHV65k7JemFghR4rO9Oxzcpw2btvtEflNJd9CODY0LBqkqVQGqpF9hi37VEPnCg== X-Received: by 2002:aa7:8893:0:b0:6ed:cd4c:cc11 with SMTP id d2e1a72fcca58-7024789b0abmr12433927b3a.25.1717477841230; Mon, 03 Jun 2024 22:10:41 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:40 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 04/12] PCI: hv: Annotate the VMBus channel IRQ name Date: Mon, 3 Jun 2024 22:09:32 -0700 Message-Id: <20240604050940.859909-5-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley In preparation for assigning Linux IRQs to VMBus channels, annotate the IRQ name in the single VMBus channel used for setup and teardown of a virtual PCI device in a Hyper-V guest. The annotation adds the 16-bit PCI domain ID that the Hyper-V vPCI driver assigns to the virtual PCI bus for the device. Signed-off-by: Michael Kelley Acked-by: Bjorn Helgaas --- drivers/pci/controller/pci-hyperv.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c index 5992280e8110..4f70cddb61dc 100644 --- a/drivers/pci/controller/pci-hyperv.c +++ b/drivers/pci/controller/pci-hyperv.c @@ -3705,6 +3705,9 @@ static int hv_pci_probe(struct hv_device *hdev, hdev->channel->request_addr_callback = vmbus_request_addr; hdev->channel->rqstor_size = HV_PCI_RQSTOR_SIZE; + snprintf(hdev->channel->irq_name, VMBUS_CHAN_IRQ_NAME_MAX, + "vpci:%04x", dom); + ret = vmbus_open(hdev->channel, pci_ring_size, pci_ring_size, NULL, 0, hv_pci_onchannelcallback, hbus); if (ret) @@ -4018,6 +4021,8 @@ static int hv_pci_resume(struct hv_device *hdev) hdev->channel->next_request_id_callback = vmbus_next_request_id; hdev->channel->request_addr_callback = vmbus_request_addr; hdev->channel->rqstor_size = HV_PCI_RQSTOR_SIZE; + snprintf(hdev->channel->irq_name, VMBUS_CHAN_IRQ_NAME_MAX, + "vpci:%04x", hbus->bridge->domain_nr); ret = vmbus_open(hdev->channel, pci_ring_size, pci_ring_size, NULL, 0, hv_pci_onchannelcallback, hbus); From patchwork Tue Jun 4 05:09:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684644 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41F8B13FD9D; Tue, 4 Jun 2024 05:10:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477844; cv=none; b=qigPTb1BXbFM4oz4WRQ29ddn5x3480uxvXjZgM8SeP+5RxLUKouRBE06XIqEuM7YzgzJtX3j0AuwT0Oe1IsmypxImsNXSEL1H0teNljGPWu8J6ATthDoQ8B6WmtZvnTYJl92+uPdDK6zrlB1/w9uLclTEqG+m4XMbbbZVvwMvaU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477844; c=relaxed/simple; bh=IqYFKETOx4ZEABhYk+CFGp7F4wFOKDydc/vQs12EN78=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cnBQAuzecfmB9VvcgU8aw6p1Ghvcm+s0cersmwRlhCmuFU2cdGNJSosj3RFpROtpOK8Zem2QZ+/BIuIFRA0Pw/Q6Zy+rTr3JcHaiPFIYxh0vDY6ng87ugR9JNSZzdVHfr1dZiUWhlB4gZwz3U6UEQOqeyqHTGQlslVrN9GlerIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TCadiSsd; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TCadiSsd" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-7025b84c0daso1946939b3a.2; Mon, 03 Jun 2024 22:10:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477842; x=1718082642; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=EJqMDDGIN+lFQ1M8fhUyjmsJ5wMGfKMbSehKHoi0sAs=; b=TCadiSsdC2MLNKUp7ZPmtYmHtvvyJ8E2H4WwpErN5fiY6MW+2+TTpAWnl1jMNi5OqR r6qBJYgU+eYQf2Ieg3IklH92yfCxeL1QKwoQkipso6SElEtUjwgorcXf9xN1xnujArRR HrTPQD+vdcYbOrkZr/8Ar68hG7EhdrWe0kzREKXu/MXuWvc1g84oAWlmE/rvaA4wf/8O GN5cL38HY4Z7w0RCDmJ3KacqH4XFXb/kqSyiFF9XSbHn8QXE08+8t3EX0UTgQHb45EY/ w8BJK3Rg75T0eyCUGXz5W1JX8rg0iu+JTJYEBx4OMRanr/nt267Mo33ZrzPLMULOUlRH HRpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477842; x=1718082642; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=EJqMDDGIN+lFQ1M8fhUyjmsJ5wMGfKMbSehKHoi0sAs=; b=uN17L4saVxFptCgphL1jcyF5sdnR770kAMwLVcc/OlGNGBDmde+LeeQcHD0aIkZGJA Z1gnAJ2t7dkud6Knp5fK2gA9vMhPnxZ6aE8Ed3Azc9T1zalnQSqyHKLZ+edddYDMDXk2 bOxl6nrdCn1/uyGHsfOCBAVMn3aMLOQj6zuyw3lCfd10L6dc4N9VrbowTXs/5qECDUgR TwRCrMXB8RM8Dw7JdQFKWiAHlcyOaLA6FF4z71dY9f2d7p8enWozNAAR1LwBK1mx5FlH OhAnKdVzJ9XZOUWjydHzyKxt2PskpvWRTbbpR3fUAN9hvvGf3WUb/a2nNCg7b6dLaNCL 0BcQ== X-Forwarded-Encrypted: i=1; AJvYcCWPrt8PfzC+3uKj8aCDZ9czcetS4tntlUrhk90GQjbH+hcFdphhGRPF5klvOXrk4QY9z+3R63mZoK/p8/V+oqmn4XnkI29BE3ojdcQVGkLjsa40ToX0+8FWofVsahO5Mv22C/z3dP/datF1jkIovecm3rafNZsAqbUa90cl0V0mxgyGh1ejpM1vymVPDk+B40HqkETvfUdF5fHXA67mxhkso64ZG9jtLgtwHuvYwEgxtRaO5B15uHzYl9ENicQ= X-Gm-Message-State: AOJu0Yz43P0ID3eqIXaMORUadL0PVriTtEmPVPRQBPGJVtSRiNJMelge QphaxW1RahZPQh2UFtP14T86GUG+uifAyra/oL4br7L+0x5RWJgs X-Google-Smtp-Source: AGHT+IHtRsaw2+6nASXeqSpTfk7BQZcffCsouxI2iBmgROU4XGXj+c/Mw8gWcFSMTNAGumyngsyiDA== X-Received: by 2002:a05:6a00:2e90:b0:702:5950:178b with SMTP id d2e1a72fcca58-702595017e1mr8916905b3a.30.1717477842482; Mon, 03 Jun 2024 22:10:42 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:42 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 05/12] scsi: storvsc: Annotate the VMBus channel IRQ name Date: Mon, 3 Jun 2024 22:09:33 -0700 Message-Id: <20240604050940.859909-6-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley In preparation for assigning Linux IRQs to VMBus channels, annotate the IRQ name in the VMBus channels used by the virtual IDE, SCSI, and Fibre Channel controllers in a Hyper-V guest. Distinguish the primary channel and sub-channels with "pri" and "sub" prefixes. Identify controllers with a numeric suffix that matches the numeric part of the "host" name displayed by "lsscsi" and SCSI-related entries in sysfs. Signed-off-by: Michael Kelley --- drivers/scsi/storvsc_drv.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c index 7ceb982040a5..11b3fc3b24c9 100644 --- a/drivers/scsi/storvsc_drv.c +++ b/drivers/scsi/storvsc_drv.c @@ -686,6 +686,8 @@ static void handle_sc_creation(struct vmbus_channel *new_sc) new_sc->max_pkt_size = STORVSC_MAX_PKT_SIZE; new_sc->next_request_id_callback = storvsc_next_request_id; + snprintf(new_sc->irq_name, VMBUS_CHAN_IRQ_NAME_MAX, "sub@storvsc%d", + stor_device->host->host_no); ret = vmbus_open(new_sc, aligned_ringbuffer_size, @@ -1322,6 +1324,7 @@ static int storvsc_connect_to_vsp(struct hv_device *device, u32 ring_size, bool is_fc) { struct vmstorage_channel_properties props; + struct storvsc_device *stor_device; int ret; memset(&props, 0, sizeof(struct vmstorage_channel_properties)); @@ -1329,6 +1332,12 @@ static int storvsc_connect_to_vsp(struct hv_device *device, u32 ring_size, device->channel->max_pkt_size = STORVSC_MAX_PKT_SIZE; device->channel->next_request_id_callback = storvsc_next_request_id; + stor_device = get_out_stor_device(device); + if (!stor_device) + return -EINVAL; + snprintf(device->channel->irq_name, VMBUS_CHAN_IRQ_NAME_MAX, "pri@storvsc%d", + stor_device->host->host_no); + ret = vmbus_open(device->channel, ring_size, ring_size, From patchwork Tue Jun 4 05:09:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684645 Received: from mail-oa1-f53.google.com (mail-oa1-f53.google.com [209.85.160.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCB791411D3; Tue, 4 Jun 2024 05:10:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477846; cv=none; b=KVPCJk01vyZ/v+XERGbc/S3nIwSESzfbr1KLfn8D2K30xMultZPXf5ggDR0m6k+N9Ff7vMN79w7bHjdehoTARCSXd6HvL0OlfjXGvY7nRzHpNaW/gVMj4sVV6nPD6yJqMvMm6CwScxTpsDxXGhuy3unk2m4VoME6p7LC8zgUbKE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477846; c=relaxed/simple; bh=8vTFceAsLpgln0QaJDr9UyqQci5nGMvow5UnG1jar74=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VvzWN10bIqWwEbtQkGeC4RIbngO4SEdmCwfBLAIeHTLOHZHThxgxHb+fWB9V4Oy9zJR/ECI2xiVMAzvKUMhb6+l+vkjNh1/4WbZvDsfThWH9bW6PTEcMWJbrFWFPLSXT9Qg+V6NfI/WMSZLFPqhKM85MU7xAu4cQ5iGNlFYSdWw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ps3vhxnn; arc=none smtp.client-ip=209.85.160.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ps3vhxnn" Received: by mail-oa1-f53.google.com with SMTP id 586e51a60fabf-250928a8580so1789045fac.3; Mon, 03 Jun 2024 22:10:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477844; x=1718082644; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=4rqdND7i4OjEglrNB2gz2qk2VRnqZQkQh3OgdrDACL0=; b=Ps3vhxnnB6Mzkc6z5PlgzSzjc8xrBBm30l8Krf6cM6CMGWUwGQ7KsSAZHSIgjusci8 II2B5+jWoSmMr/NTEdtBlBmymUMWa0Khg+HEnR3mWzKdUEF5RE01OvnRNoo4LFTvUZvV ZrldILl2/IKN4uxmYGACUmTNMQOli4yF9ngsH6wFJyMUlwnFTW6aiEvPfRoolGKFOBmK 7R+4tI5fwdtXTCrKECOjqgisnaM8sQPGPzcJtTKgEuwEadzhT+mnRx3gjZj9XjNzMLXi RBeUz8GD3nlqQT+dvsvtzgjQ5K3ptZHBjveUiG0uwxoKOAHJ7uXQIUIEGaWDTJVOvZbC Yytw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477844; x=1718082644; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=4rqdND7i4OjEglrNB2gz2qk2VRnqZQkQh3OgdrDACL0=; b=IcJZ10gut1WIAzxC6UgNKi7L8GW6w8yHxJZL6ldqyi1pf0yNn4m+0MG7kc7M7aZPfV BpymUMizf+40hsd0BJgdgFlAmblEIo9oblrsxwFCK+wtuNEAbqJqRteCJIFoCEvqAhP1 +Rli56umprjOFy78ip9ISYCN67ORCwEhkSDNYSx8VqTcUsZZPr1SK3+uMONcR9PDIcNR V1odRfk21Xg8ommS2Z27HWkwfanXU7jCQDvy0aN+evs7p0AuzY7lbXL5w54kyY1DPAbb MCVaA8fxo7MudthfW3aubxJuUxUjJwMb+QH9Lg07bMwFn9c6ZbrQhq3w5GwDHlPJNTW8 vWrA== X-Forwarded-Encrypted: i=1; AJvYcCVd1qWSZC/cuQmFORrMRJ4+olxyRxGjqbl0Rsp1E48sUgxCD6hngVLgnp5z0tuJDGr1W7wf/+++eroYHZC4ZNrdwhWXYAg9XbPvmOPRddjIMmXxWMZn7fAJnYpo0qk6n4+ou3J+9/1NNu0mjk8a0apDqteu1ezwA2K7BOP6NSnWYP0BJ9HEFphdA3Ar+sr49iqNe2+d4wIC188AabQCw8HBjRzsgBajXpoVMD1y6gIgV2jJxRNPKpAZsZHLvUg= X-Gm-Message-State: AOJu0YwUfB176z1En4w/tCC6/QTeZk37+VJQEvS98viyHtsfqKMwvPOH uSuf3nNNxHe1Ouj0eSnOGcq/UTMD/uyEATP4eSsyBlawZuFIU0gh X-Google-Smtp-Source: AGHT+IEyK5gUfY9/lSLdd30TiSbxTGT5bgkCQfCQCTSfZcQeNghhepHgsH5TsmMYS1KZXrNP4YZsNQ== X-Received: by 2002:a05:6870:1702:b0:24f:e3a3:4431 with SMTP id 586e51a60fabf-2508b9efb87mr10561586fac.13.1717477843743; Mon, 03 Jun 2024 22:10:43 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:43 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 06/12] genirq: Add per-cpu flow handler with conditional IRQ stats Date: Mon, 3 Jun 2024 22:09:34 -0700 Message-Id: <20240604050940.859909-7-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley Hyper-V VMBus devices generate interrupts that are multiplexed onto a single per-CPU architectural interrupt. The top-level VMBus driver ISR demultiplexes these interrupts and invokes per-device handlers. Currently, these per-device handlers are not modeled as Linux IRQs, so /proc/interrupts shows all VMBus interrupts as accounted to the top level architectural interrupt. Visibility into per-device interrupt stats requires accessing VMBus-specific entries in sysfs. The top-level VMBus driver ISR also handles management-related interrupts that are not attributable to a particular VMBus device. As part of changing VMBus to model VMBus per-device handlers as normal Linux IRQs, the top-level VMBus driver needs to conditionally account for interrupts. If it passes the interrupt off to a device-specific IRQ, the interrupt stats are done by that IRQ handler, and accounting for the interrupt at the top level is duplicative. But if it handles a management-related interrupt itself, then it should account for the interrupt itself. Introduce a new flow handler that provides this functionality. The new handler parallels handle_percpu_irq(), but does stats only if the ISR returns other than IRQ_NONE. The existing handle_untracked_irq() can't be used because it doesn't work for per-cpu IRQs, and it doesn't provide conditional stats. Signed-off-by: Michael Kelley --- include/linux/irq.h | 1 + kernel/irq/chip.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/include/linux/irq.h b/include/linux/irq.h index a217e1029c1d..8825b95cefe5 100644 --- a/include/linux/irq.h +++ b/include/linux/irq.h @@ -656,6 +656,7 @@ extern void handle_edge_eoi_irq(struct irq_desc *desc); extern void handle_simple_irq(struct irq_desc *desc); extern void handle_untracked_irq(struct irq_desc *desc); extern void handle_percpu_irq(struct irq_desc *desc); +extern void handle_percpu_demux_irq(struct irq_desc *desc); extern void handle_percpu_devid_irq(struct irq_desc *desc); extern void handle_bad_irq(struct irq_desc *desc); extern void handle_nested_irq(unsigned int irq); diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c index dc94e0bf2c94..1f37a9db4a4d 100644 --- a/kernel/irq/chip.c +++ b/kernel/irq/chip.c @@ -910,6 +910,35 @@ void handle_percpu_irq(struct irq_desc *desc) chip->irq_eoi(&desc->irq_data); } +/** + * handle_percpu_demux_irq - Per CPU local irq handler for IRQs + * that may demultiplex to other IRQs + * @desc: the interrupt description structure for this irq + * + * For per CPU interrupts on SMP machines without locking requirements. + * Used for IRQs that may demultiplex to other IRQs that will do the + * stats tracking and randomness. If the handler result indicates no + * demultiplexing is done, account for the interrupt on this IRQ. + */ +void handle_percpu_demux_irq(struct irq_desc *desc) +{ + struct irq_chip *chip = irq_desc_get_chip(desc); + + if (chip->irq_ack) + chip->irq_ack(&desc->irq_data); + + if (__handle_irq_event_percpu(desc)) + /* + * PER CPU interrupts are not serialized. Do not touch + * desc->tot_count. + */ + __kstat_incr_irqs_this_cpu(desc); + + if (chip->irq_eoi) + chip->irq_eoi(&desc->irq_data); +} +EXPORT_SYMBOL_GPL(handle_percpu_demux_irq); + /** * handle_percpu_devid_irq - Per CPU local irq handler with per cpu dev ids * @desc: the interrupt description structure for this irq From patchwork Tue Jun 4 05:09:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684646 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06AAB1419B1; Tue, 4 Jun 2024 05:10:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477847; cv=none; b=uxX2k1GjZmD1i37HzEJU1qFC14Zc4wYTYrGkJQRbmsnvofAglfnsN9t9U1gPOgsgmmmVEpwABtyPxknSZSzcD4d1F36IBvRGfeDe5F5hLvaH8nqnLfyLKy/B4ta/rUcYiIw+n2wnXbpUQeVTjw7mZ1haTsrSSF8RZL8EPKFJaco= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477847; c=relaxed/simple; bh=vVRI2TfY68wado4CSOgxUZSaUZKTk/D7LMeKecnuSBk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=G5y1bdCkGoNWGM9E/z3nirrKsZ0gpns4xKj7wU8gvj0w+6FFUDQQbtDCgrwspeaw9VDLFh0ga2U2wBO0IUtLReG+ilPGVhCKPq6GMzfETb2tWar5Bs87HvY63CweuJEWSXnN7YI20/3y76o6WbM7N+5I4BM13hNKxB+A3CCmlpw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XYYwQnoN; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XYYwQnoN" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-70245b22365so2957212b3a.1; Mon, 03 Jun 2024 22:10:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477845; x=1718082645; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=uI7hPio5EOSCOqhNP+fmg82P5jjcL6y/nOk7OMudZ2w=; b=XYYwQnoN+NFssVXY7qDW7r1aRkkg25tQnfiARNrXfqKp6Zsvk5sobOSsPPFYcE8mpM v8mCplCymvf4hlJt4C01bt2gLCWIABmUsbKXd24drl2NAB6d6kcbb12xcCn8cTpYVtoR zaRRxz9XeT3eINwS9t5gOT2Gw08qOjXD1WmPRKrg+YYQqG624rI4Lp9eLRUukspNMyiL 9QYmI5js6ys1N4D4an/f5lQUqh0K4FzvUcC7C3nM2cKSZcutTZwsfSTk4uKAU/MMqOnm 08hvToLUcHWAvF6LSDcr6XSGEyeeG3OCiBYlno+sFpOvt2p06E6BYexboMKNIKd2Ua/b YaWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477845; x=1718082645; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=uI7hPio5EOSCOqhNP+fmg82P5jjcL6y/nOk7OMudZ2w=; b=Mf156JGjEHOWPPbXaryMgVD3HzkIa8iKf33umrJxo+bzmNJsLIdLqgTKW5l/8OCJ7W LJLwSuSkXHlqmEqCjyTI8L9lrMOE+BKI/q53aw3ptMrOByUGIv/xhAkE2SLnKx3gneNK nb0mGAKnZjmAPWhFjDAiXlqdoCM7vEFNUbJxY64Ox7TvFJu7M17bYBiU52JNVhoHVO4m l9eYFenOqaCejSmE/Q7bWQFMsFxa6Elu6e/YJVUm1Gt1AoQu5aMU7kmxYMWkHzBzQr4e Ecid4sBOTI1E3d/if9MCzxD6Tbup9BxqNws30YXq77B3JF/4Xz0AudPNVobu6ISjI4x8 Eypg== X-Forwarded-Encrypted: i=1; AJvYcCXeDUJyACJ7fPm0hDHWH9B7FN80vY2Bdwd2sIOxDwzFqVtkI7hZdFQW+BeoDtQNJ3OF/f+X5fIGR0UUYeAONT9FM409sRmFh+ZD4vKozPOaN8X1L1G3/u27gyGkbEMRhKiNSTeRpjc75JS1G3eFHYfWRXKIVgzXF9s6WfECnJXp9IWYOhgNZ/ZtV6NMHFAQ1IiZywKFFtNr0o4rwKAecWLF95mKbR6Sm0ZwHmoMCnc1yn0G8PwbuTU6eH2MBiE= X-Gm-Message-State: AOJu0YxK9/EaoG2BQKloxegYdzGlXPzeX6fAqe5osPTsyV4/8fOKRJ8M Fj8ulnRvo8VPiEVTJJJjeOrE3nOQRxSEnT10fPOs5Ir1J5T9+qcd X-Google-Smtp-Source: AGHT+IEmnCRnuRNZeIpORES3MssYPt/+L5A69/3IuhurrqBWUQ1ji+4Umw+ygAq+nk7kFKVPAk9Pog== X-Received: by 2002:a05:6a20:8415:b0:1af:7646:fc14 with SMTP id adf61e73a8af0-1b26ef54e19mr11855148637.0.1717477845049; Mon, 03 Jun 2024 22:10:45 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:44 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 07/12] Drivers: hv: vmbus: Set up irqdomain and irqchip for the VMBus connection Date: Mon, 3 Jun 2024 22:09:35 -0700 Message-Id: <20240604050940.859909-8-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley In preparation for assigning Linux IRQs to VMBus channels, set up an irqdomain and irqchip for the VMBus connection. The irqdomain is linear, with the VMBus relid used as the "hwirq" value. A relid is a unique index assigned by Hyper-V to each VMBus channel, with values ranging from 1 to 2047. Because these hwirqs don't map to anything in the architectural hardware, the domain is not part of the domain hierarchy. VMBus channel interrupts provide minimal management functionality, so provide only a minimal set of irqchip functions. The set_affinity function is a place-holder that is populated in a subsequent patch. Signed-off-by: Michael Kelley --- drivers/hv/connection.c | 24 +++++++++----- drivers/hv/hyperv_vmbus.h | 9 +++++ drivers/hv/vmbus_drv.c | 60 +++++++++++++++++++++++++++++++++- include/asm-generic/mshyperv.h | 6 ++++ 4 files changed, 90 insertions(+), 9 deletions(-) diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c index f001ae880e1d..cb01784e5c3b 100644 --- a/drivers/hv/connection.c +++ b/drivers/hv/connection.c @@ -21,21 +21,29 @@ #include #include #include +#include +#include +#include #include #include "hyperv_vmbus.h" struct vmbus_connection vmbus_connection = { - .conn_state = DISCONNECTED, - .unload_event = COMPLETION_INITIALIZER( - vmbus_connection.unload_event), - .next_gpadl_handle = ATOMIC_INIT(0xE1E10), - - .ready_for_suspend_event = COMPLETION_INITIALIZER( - vmbus_connection.ready_for_suspend_event), + .conn_state = DISCONNECTED, + .unload_event = COMPLETION_INITIALIZER( + vmbus_connection.unload_event), + .next_gpadl_handle = ATOMIC_INIT(0xE1E10), + + .vmbus_irq_chip.name = "VMBus", + .vmbus_irq_chip.irq_set_affinity = vmbus_irq_set_affinity, + .vmbus_irq_chip.irq_mask = vmbus_irq_mask, + .vmbus_irq_chip.irq_unmask = vmbus_irq_unmask, + + .ready_for_suspend_event = COMPLETION_INITIALIZER( + vmbus_connection.ready_for_suspend_event), .ready_for_resume_event = COMPLETION_INITIALIZER( - vmbus_connection.ready_for_resume_event), + vmbus_connection.ready_for_resume_event), }; EXPORT_SYMBOL_GPL(vmbus_connection); diff --git a/drivers/hv/hyperv_vmbus.h b/drivers/hv/hyperv_vmbus.h index 76ac5185a01a..95d4d47d34f7 100644 --- a/drivers/hv/hyperv_vmbus.h +++ b/drivers/hv/hyperv_vmbus.h @@ -18,7 +18,11 @@ #include #include #include +#include #include +#include +#include +#include #include "hv_trace.h" @@ -258,6 +262,11 @@ struct vmbus_connection { /* Array of channels */ struct vmbus_channel **channels; + /* IRQ domain data */ + struct fwnode_handle *vmbus_fwnode; + struct irq_domain *vmbus_irq_domain; + struct irq_chip vmbus_irq_chip; + /* * An offer message is handled first on the work_queue, and then * is further handled on handle_primary_chan_wq or diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index 291a8358370b..cbccdfad49a2 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -36,6 +36,9 @@ #include #include #include +#include +#include +#include #include #include #include "hyperv_vmbus.h" @@ -1306,6 +1309,40 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id) return IRQ_HANDLED; } +int vmbus_irq_set_affinity(struct irq_data *data, + const struct cpumask *dest, bool force) +{ + return 0; +} + +/* + * VMBus channel interrupts do not need to be masked or unmasked, and the + * Hyper-V synic doesn't provide any masking functionality anyway. But in the + * absence of these irqchip functions, the IRQ subsystem keeps the IRQ marked + * as "masked". To prevent any problems associated with staying the "masked" + * state, and so that IRQ status shown in debugfs doesn't indicate "masked", + * provide null implementations. + */ +void vmbus_irq_unmask(struct irq_data *data) +{ +} + +void vmbus_irq_mask(struct irq_data *data) +{ +} + +static int vmbus_irq_map(struct irq_domain *d, unsigned int irq, + irq_hw_number_t hw) +{ + irq_set_chip_and_handler(irq, + &vmbus_connection.vmbus_irq_chip, handle_simple_irq); + return 0; +} + +static const struct irq_domain_ops vmbus_domain_ops = { + .map = vmbus_irq_map, +}; + /* * vmbus_bus_init -Main vmbus driver initialization routine. * @@ -1340,6 +1377,7 @@ static int vmbus_bus_init(void) if (vmbus_irq == -1) { hv_setup_vmbus_handler(vmbus_isr); } else { + irq_set_handler(vmbus_irq, handle_percpu_demux_irq); vmbus_evt = alloc_percpu(long); ret = request_percpu_irq(vmbus_irq, vmbus_percpu_isr, "Hyper-V VMbus", vmbus_evt); @@ -1355,6 +1393,20 @@ static int vmbus_bus_init(void) if (ret) goto err_alloc; + /* Create IRQ domain for VMBus devices */ + vmbus_connection.vmbus_fwnode = irq_domain_alloc_named_fwnode("hv-vmbus"); + if (!vmbus_connection.vmbus_fwnode) { + ret = -ENOMEM; + goto err_alloc; + } + vmbus_connection.vmbus_irq_domain = irq_domain_create_linear( + vmbus_connection.vmbus_fwnode, MAX_CHANNEL_RELIDS, + &vmbus_domain_ops, NULL); + if (!vmbus_connection.vmbus_irq_domain) { + ret = -ENOMEM; + goto err_fwnode; + } + /* * Initialize the per-cpu interrupt state and stimer state. * Then connect to the host. @@ -1362,7 +1414,7 @@ static int vmbus_bus_init(void) ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "hyperv/vmbus:online", hv_synic_init, hv_synic_cleanup); if (ret < 0) - goto err_alloc; + goto err_domain; hyperv_cpuhp_online = ret; ret = vmbus_connect(); @@ -1382,6 +1434,10 @@ static int vmbus_bus_init(void) err_connect: cpuhp_remove_state(hyperv_cpuhp_online); +err_domain: + irq_domain_remove(vmbus_connection.vmbus_irq_domain); +err_fwnode: + irq_domain_free_fwnode(vmbus_connection.vmbus_fwnode); err_alloc: hv_synic_free(); if (vmbus_irq == -1) { @@ -2690,6 +2746,8 @@ static void __exit vmbus_exit(void) hv_debug_rm_all_dir(); vmbus_free_channels(); + irq_domain_remove(vmbus_connection.vmbus_irq_domain); + irq_domain_free_fwnode(vmbus_connection.vmbus_fwnode); kfree(vmbus_connection.channels); /* diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h index 8fe7aaab2599..0488ff8b511f 100644 --- a/include/asm-generic/mshyperv.h +++ b/include/asm-generic/mshyperv.h @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -187,6 +188,11 @@ void hv_remove_kexec_handler(void); void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs)); void hv_remove_crash_handler(void); +extern void vmbus_irq_mask(struct irq_data *data); +extern void vmbus_irq_unmask(struct irq_data *data); +extern int vmbus_irq_set_affinity(struct irq_data *data, + const struct cpumask *dest, bool force); + extern int vmbus_interrupt; extern int vmbus_irq; From patchwork Tue Jun 4 05:09:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684647 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51EAD1422A4; Tue, 4 Jun 2024 05:10:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477849; cv=none; b=mNLb4SFJxhTwXFfp3q+SoQNd7ANITME+4kqLiR2fJVrHOwzywPOsyjbfCQgTWuIhB9kgtixC+EM/f697+cyJuc8UQ8/msPN+FF4mgAtIhdEypeGeRIiqKsGmpDutB1goy9lg0k7gqLABYVUtc7elWhzJO/usx6m1zUVUc46Cx74= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477849; c=relaxed/simple; bh=E+4d26O9KX6W8V3kSd+4KV+3CKb0TWbekzCJh1k17NQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PAv6RgETfpKYTjKofV2NiYsqJbMs3T8jqFw2JTQiHNkBc2wE2AQV4vPPXKeMhZdL4n+O8lqsVM/05ijlubn4BfRVH0iY2IjHMbQEdPbuYVHRwR+k5iCdPDYB8do3BTQ/2z+3OQEhi1BmPUbZM2TxH6qT1BZ8Sgeb04CScBIPcf4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UdyYF/Yv; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UdyYF/Yv" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3cb353c73ccso2791596b6e.0; Mon, 03 Jun 2024 22:10:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477846; x=1718082646; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Gzkdr7jopxvyXBA+TIdPtuUwB5jzrxlcbv3cbqedfWg=; b=UdyYF/YvBpgG4j/fPyBb4GGtY2M6fwGXUhHI2rSbEynifPSdrV09z9jDmRNRezjcwe HhOjUSqCSJ4kQyaLpzCrQx+5vUWb6voaNdYq5Ck9NsG+8ZWne2B4SrKKIGcx7BCtbg9a 33CgVl7EWZfwj93HW3ijQ8STgz7P2am8qpws4gkwzDVB5hVSL+gax/mqvljoWOa7Qkal TJyrTPD91q3Hh46yplik0FCWdxIHtBqPuXnNNr0u5UjNuWgywP2EWVCj6j2Bwj658ZZg cfZGvSmtHAhx6Hw7bRVXSwR0q6FDSsZEExFEqo2JFrnsKwjm3UcZrjkO0gV32io8yXQg /4AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477846; x=1718082646; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Gzkdr7jopxvyXBA+TIdPtuUwB5jzrxlcbv3cbqedfWg=; b=jUlaGX2/wTdx7CJYg1g48B0P53xEP3/truZGj5pIMLN5kJ+dhAorvw1QRKdCNW2kp9 HJwGYERyLysWkB3s214GxdjA6BFqgGK0Sm+c6DXBiIvjW5OWIhVB4k9MQ1XRR3tdbXl7 4YTMY7D9ZIRUba665hmONjIMtHybia4hLggeqSYCvvEqfE0WcbnI+HRPAfSEtMwbMbkL i8QDDo6yAVYFRL4F3Ts1KkquarisxhoazK2SdGL2bOuyVyNmros5VANyOYfJMevMQW/L XSF3M0BW6cqtZMKnjiPkK98YMxhoUyC/+okp5sk0zwyyclhd6h94OGVGbAiPyS4Nzyw1 u5AA== X-Forwarded-Encrypted: i=1; AJvYcCXShpcAoHz30GIifiAbxbGPZHEnlj3aHQScS08PHC8AvRcW3RIiW8LoIQ4vfgOFZBjegZyJl7QeEzOh+VgBRLhtdQJv2Jzj7zJRzKmx9bU6RSm4L/Bf7w0zn4NSgl8nYb5U5dPBRARC2dS9/DrV7hvtDUu8LSEcfCunpeEwk1khuX+VQXoDqrPfODJz5JGCyU7hAKL1cTwKbTqzMhhuqPn3ESNk9+ly6jYvRiSaF4tTN+VcKMQjs75BvjnrZbA= X-Gm-Message-State: AOJu0YxQNMa2Km1K8i++bbGV/MmixZ4y1PevQhfr6bpOc6x0KQGHlYyu /b9Jc74T7YEnDFnAdzfc4fBe4Ly6wf1r+Ur19LUAlAmanRpKpVvp X-Google-Smtp-Source: AGHT+IGHyfvHrRKqIbyeaMIfOflT83+zkAgXQUpgUOT1FSxBpckU8L78mwisaPyDCpXnNI4EaDnmQw== X-Received: by 2002:a05:6808:1925:b0:3d1:ff8e:a5b6 with SMTP id 5614622812f47-3d1ff8ea68dmr275682b6e.15.1717477846375; Mon, 03 Jun 2024 22:10:46 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:46 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 08/12] Drivers: hv: vmbus: Allocate an IRQ per channel and use for relid mapping Date: Mon, 3 Jun 2024 22:09:36 -0700 Message-Id: <20240604050940.859909-9-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley Current code uses the "channels" array in the VMBus connection to map from relid to VMBus channel. Functions exist to establish and remove mappings, and to map from a relid to its channel. To implement assigning Linux IRQs to VMBus channels, repurpose the existing relid functions. Change the "map" function to create an IRQ mapping in the VMBus IRQ domain, and set the handler data for the IRQ to be the VMBus channel. Change the "unmap" function to remove the IRQ mapping after freeing the IRQ if it has been requested. Change the relid2channel() function to look up the mapping and return both the channel and the corresponding irqdesc. While Linux IRQs are allocated for each channel, they are used only for relid mapping. A subsequent patch changes the interrupt handling path to use the Linux IRQ. With these changes, the "channels" array in the VMBus connection is no longer used. Remove it. Also fixup comments that refer to the channels array and make them refer to the IRQ mapping instead. Signed-off-by: Michael Kelley --- drivers/hv/channel_mgmt.c | 52 ++++++++++++++++++++++++++++----------- drivers/hv/connection.c | 23 ++++++----------- drivers/hv/hyperv_vmbus.h | 5 +--- drivers/hv/vmbus_drv.c | 8 +++--- include/linux/hyperv.h | 3 +++ 5 files changed, 54 insertions(+), 37 deletions(-) diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c index adbe184e5197..da42aaae994e 100644 --- a/drivers/hv/channel_mgmt.c +++ b/drivers/hv/channel_mgmt.c @@ -10,6 +10,9 @@ #include #include +#include +#include +#include #include #include #include @@ -410,7 +413,10 @@ static void free_channel(struct vmbus_channel *channel) void vmbus_channel_map_relid(struct vmbus_channel *channel) { - if (WARN_ON(channel->offermsg.child_relid >= MAX_CHANNEL_RELIDS)) + int res; + u32 relid = channel->offermsg.child_relid; + + if (WARN_ON(relid >= MAX_CHANNEL_RELIDS)) return; /* * The mapping of the channel's relid is visible from the CPUs that @@ -437,18 +443,33 @@ void vmbus_channel_map_relid(struct vmbus_channel *channel) * of the VMBus driver and vmbus_chan_sched() can not run before * vmbus_bus_resume() has completed execution (cf. resume_noirq). */ - virt_store_mb( - vmbus_connection.channels[channel->offermsg.child_relid], - channel); + + channel->irq = irq_create_mapping(vmbus_connection.vmbus_irq_domain, relid); + if (!channel->irq) { + pr_err("irq_create_mapping failed for relid %d\n", relid); + return; + } + + res = irq_set_handler_data(channel->irq, channel); + if (res) { + irq_dispose_mapping(channel->irq); + channel->irq = 0; + pr_err("irq_set_handler_data failed with %d for relid %d\n", + res, relid); + return; + } + + irq_set_status_flags(channel->irq, IRQ_MOVE_PCNTXT); } void vmbus_channel_unmap_relid(struct vmbus_channel *channel) { - if (WARN_ON(channel->offermsg.child_relid >= MAX_CHANNEL_RELIDS)) - return; - WRITE_ONCE( - vmbus_connection.channels[channel->offermsg.child_relid], - NULL); + if (channel->irq_requested) { + irq_update_affinity_hint(channel->irq, NULL); + free_irq(channel->irq, channel); + } + channel->irq_requested = false; + irq_dispose_mapping(channel->irq); } static void vmbus_release_relid(u32 relid) @@ -478,10 +499,10 @@ void hv_process_channel_removal(struct vmbus_channel *channel) !is_hvsock_channel(channel)); /* - * Upon suspend, an in-use hv_sock channel is removed from the array of - * channels and the relid is invalidated. After hibernation, when the + * Upon suspend, an in-use hv_sock channel is removed from the IRQ + * map and the relid is invalidated. After hibernation, when the * user-space application destroys the channel, it's unnecessary and - * unsafe to remove the channel from the array of channels. See also + * unsafe to remove the channel from the IRQ map. See also * the inline comments before the call of vmbus_release_relid() below. */ if (channel->offermsg.child_relid != INVALID_RELID) @@ -533,6 +554,9 @@ static void vmbus_add_channel_work(struct work_struct *work) struct vmbus_channel *primary_channel = newchannel->primary_channel; int ret; + if (!newchannel->irq) + goto err_deq_chan; + /* * This state is used to indicate a successful open * so that when we do close the channel normally, we @@ -1144,7 +1168,7 @@ static void vmbus_onoffer(struct vmbus_channel_message_header *hdr) vmbus_setup_channel_state(oldchannel, offer); } - /* Add the channel back to the array of channels. */ + /* Re-establish the channel's IRQ mapping using the new relid */ vmbus_channel_map_relid(oldchannel); check_ready_for_resume_event(); @@ -1225,7 +1249,7 @@ static void vmbus_onoffer_rescind(struct vmbus_channel_message_header *hdr) } mutex_lock(&vmbus_connection.channel_mutex); - channel = relid2channel(rescind->child_relid); + channel = relid2channel(rescind->child_relid, NULL); if (channel != NULL) { /* * Guarantee that no other instance of vmbus_onoffer_rescind() diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c index cb01784e5c3b..a105eecdeec2 100644 --- a/drivers/hv/connection.c +++ b/drivers/hv/connection.c @@ -308,14 +308,6 @@ int vmbus_connect(void) pr_info("Vmbus version:%d.%d\n", version >> 16, version & 0xFFFF); - vmbus_connection.channels = kcalloc(MAX_CHANNEL_RELIDS, - sizeof(struct vmbus_channel *), - GFP_KERNEL); - if (vmbus_connection.channels == NULL) { - ret = -ENOMEM; - goto cleanup; - } - kfree(msginfo); return 0; @@ -373,15 +365,16 @@ void vmbus_disconnect(void) * relid2channel - Get the channel object given its * child relative id (ie channel id) */ -struct vmbus_channel *relid2channel(u32 relid) +struct vmbus_channel *relid2channel(u32 relid, struct irq_desc **desc_ptr) { - if (vmbus_connection.channels == NULL) { - pr_warn_once("relid2channel: relid=%d: No channels mapped!\n", relid); - return NULL; - } - if (WARN_ON(relid >= MAX_CHANNEL_RELIDS)) + struct irq_desc *desc; + + desc = irq_resolve_mapping(vmbus_connection.vmbus_irq_domain, relid); + if (!desc) return NULL; - return READ_ONCE(vmbus_connection.channels[relid]); + if (desc_ptr) + *desc_ptr = desc; + return irq_desc_get_handler_data(desc); } /* diff --git a/drivers/hv/hyperv_vmbus.h b/drivers/hv/hyperv_vmbus.h index 95d4d47d34f7..bf35bb40c55e 100644 --- a/drivers/hv/hyperv_vmbus.h +++ b/drivers/hv/hyperv_vmbus.h @@ -259,9 +259,6 @@ struct vmbus_connection { struct list_head chn_list; struct mutex channel_mutex; - /* Array of channels */ - struct vmbus_channel **channels; - /* IRQ domain data */ struct fwnode_handle *vmbus_fwnode; struct irq_domain *vmbus_irq_domain; @@ -364,7 +361,7 @@ void vmbus_remove_channel_attr_group(struct vmbus_channel *channel); void vmbus_channel_map_relid(struct vmbus_channel *channel); void vmbus_channel_unmap_relid(struct vmbus_channel *channel); -struct vmbus_channel *relid2channel(u32 relid); +struct vmbus_channel *relid2channel(u32 relid, struct irq_desc **desc); void vmbus_free_channels(void); diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index cbccdfad49a2..8fd03d41e71a 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -1219,6 +1219,7 @@ static void vmbus_chan_sched(struct hv_per_cpu_context *hv_cpu) for_each_set_bit(relid, recv_int_page, maxbits) { void (*callback_fn)(void *context); struct vmbus_channel *channel; + struct irq_desc *desc; if (!sync_test_and_clear_bit(relid, recv_int_page)) continue; @@ -1236,7 +1237,7 @@ static void vmbus_chan_sched(struct hv_per_cpu_context *hv_cpu) rcu_read_lock(); /* Find channel based on relid */ - channel = relid2channel(relid); + channel = relid2channel(relid, &desc); if (channel == NULL) goto sched_unlock_rcu; @@ -2466,10 +2467,10 @@ static int vmbus_bus_suspend(struct device *dev) list_for_each_entry(channel, &vmbus_connection.chn_list, listentry) { /* - * Remove the channel from the array of channels and invalidate + * Remove the channel from the IRQ map and invalidate * the channel's relid. Upon resume, vmbus_onoffer() will fix * up the relid (and other fields, if necessary) and add the - * channel back to the array. + * channel back to the IRQ map. */ vmbus_channel_unmap_relid(channel); channel->offermsg.child_relid = INVALID_RELID; @@ -2748,7 +2749,6 @@ static void __exit vmbus_exit(void) vmbus_free_channels(); irq_domain_remove(vmbus_connection.vmbus_irq_domain); irq_domain_free_fwnode(vmbus_connection.vmbus_fwnode); - kfree(vmbus_connection.channels); /* * The vmbus panic notifier is always registered, hence we should diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h index 897d96fa19d4..d545b1b635e5 100644 --- a/include/linux/hyperv.h +++ b/include/linux/hyperv.h @@ -1072,6 +1072,9 @@ struct vmbus_channel { /* The max size of a packet on this channel */ u32 max_pkt_size; + /* The Linux IRQ for the channel in the "hv-vmbus" IRQ domain */ + u32 irq; + bool irq_requested; char irq_name[VMBUS_CHAN_IRQ_NAME_MAX]; }; From patchwork Tue Jun 4 05:09:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684648 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B763142629; Tue, 4 Jun 2024 05:10:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477850; cv=none; b=EqXGxC0gXq2SiXvje+nGXKbOdLB7E7eXZUHOFns9qipm55m4hys7vdKWUZC0sJIVSRN27EdklySPWZAtOpnQtuMqD/FdM3E6EaB7zHErmjAUodzejRO34Q6Ux+u2KpJTTpUYFIC+nLlIDKEayTb2THvTRq0ZrdfzkHNNDCmzuYE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477850; c=relaxed/simple; bh=Dtv+8BHewlDtrscRUC9E1bIoWTmplqu0ARBJT2pimYM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iD/qWg58Pmzgjy73tgZ5jHGdcStZ21Iud2TyUwt0iH6kEyWlHP0ehzT3BtHtuOOYyXzJDP5wCkvAU1Pm9qD+Fqt2rD/LCmFt9xKWyw1CiQtNYMPodMxM2kWIhNqAZiXwoFaDRQd6WWuaMjfc1XVQS+1sDOaFAGTV+CVN6yvRuA8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cGxsXeiW; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cGxsXeiW" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-7025cb09553so2191891b3a.3; Mon, 03 Jun 2024 22:10:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477848; x=1718082648; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=X+SPYtiC+eKOOluq0xEXtbfDYX7+i/A8vs71JSRv2/w=; b=cGxsXeiWR9IUNANgy10u9SpNbdv7smq9YxQpMQlD2ws4Gs+doIyxU2vCqS5WN1qzEn u5v1mzM2Li3Jks+3HC/Uy1pphm9Z5kk6Af0f1a2jxiya8VPPcv1+TL+TwzibdQy0FRiE gM3XvNEOoN3aazBEClnbCwvUFSfVLs2wZ82NeS56SwVbWJd13xh9yd9G9xalGe8uxDKj 8LsZMJ8AIDYU6kKyZOM9PJsF551AhCnQIZdIogsGE4aA6hugr6cM/7U3Lqo93BTEL/K7 ivSQPTi8AD2+3F53vcj8fYb1FdaG5xJxBvxrvnJIGzHG/59UI+aAVHzFFurZM0AIsDxI M1GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477848; x=1718082648; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=X+SPYtiC+eKOOluq0xEXtbfDYX7+i/A8vs71JSRv2/w=; b=iLHFT9Z92Qu0EII8dClG4m3siZMVQOXytfboIWpp5lHVTOjgSt0FTlnLs/H/KUfYgR XlkdXtL2mMl3dW2RFMc6c3oIfCvUbLIf5Op7MejMed6x3xW7xkgUy64TnhuVMVidmyDD IKL9BVVJfGgpXOgdPLfNlmzxoSQ8G5qxsWQsM0J2EHYjwQbJ0ieruH5rlNmTTo8k2kur 9sTb0v4T4Bj5bepjltU8mEryuPXBeYO2gIN2C7lx+ffvT4JKaEcOi0+KL4GNCCslEU+Y SzlYslcVQ1trT/DkENgEH9xHnN13/yQkAarnwQC35FBRnSEoNsIVP5Q6eeW9TrPPFLiw kTZQ== X-Forwarded-Encrypted: i=1; AJvYcCUKexbd54Am8PgtEQf1mzk21yOU/wjwr3QIJYBHmF7Dic4RL9NqL3ifStGhmHqiLV2nPqQE1P4B/aW/sAebInF0bGsD6BibVlStB+Fl7Qn6nXNLPuawO9PMaSmbcWQg7uRnP1BHswrbYo/QrG9LdtB3ZVVpmnFYLlV8GWv2ou7uWw6CP+/y6iACMEVPWsR0hJo8Ws24o5GSSStGX38wm6kBbi1lHz88D/6vYgeZ6dATYZZuQVprKUBDin3kVH8= X-Gm-Message-State: AOJu0YxVu9QtuMNGLq7XqhHmTYwZWDom2M3W/xVJuPWC4QAdmhaC/P8+ oqk/tguDNYaHfTywvsLpSm7zsN9emnCEZzpB/UptfyJ3I02w7JvV X-Google-Smtp-Source: AGHT+IF/QjqqduNmy9R3nfgGphf2kfowTvEQN0egF5f68NiQrvZUbfJu9Gb0OBfN+Uknx2sDslo5gg== X-Received: by 2002:a05:6a00:39a6:b0:6f4:4319:4cea with SMTP id d2e1a72fcca58-702478da697mr15516163b3a.33.1717477847750; Mon, 03 Jun 2024 22:10:47 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:47 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 09/12] Drivers: hv: vmbus: Use Linux IRQs to handle VMBus channel interrupts Date: Mon, 3 Jun 2024 22:09:37 -0700 Message-Id: <20240604050940.859909-10-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley Do the following: 1) Create an interrupt handler for VMBus channel interrupts by pulling out portions of vmbus_chan_sched() into vmbus_chan_handler(). The outer part of vmbus_chan_sched() that loops through the synic event page bitmap remains unchanged. But when a pending VMBus channel interrupt is found, call generic_handle_irq_desc() to invoke handle_simple_irq() and then vmbus_chan_handler() for the channel's IRQ. handle_simple_irq() does the IRQ stats for that channel's IRQ, so that per-channel interrupt counts appear in /proc/interrupts. The overall processing of VMBus channel interrupts is unchanged except for the intervening handle_simple_irq() that does the stats. No acks or EOIs are required for VMBus channel IRQs. 2) Update __vmbus_open() to call request_irq(), specifying the previously setup channel IRQ name and vmbus_chan_handler() as the interrupt handler. Set the IRQ affinity to the target_cpu assigned when the channel was created. 3) Update vmbus_isr() to return "false" if it only handles VMBus interrupts, which were passed to the channel IRQ handler. If vmbus_isr() handles one or more control message interrupts, then return "true". Update the related definitions to specify a boolean return value. 4) The callers of vmbus_isr() increment IRQ stats for the top-level IRQ only if "true" is returned. On x86, the caller is sysvec_hyperv_callback(), which manages the stats directly. On arm64, the caller is vmbus_percpu_isr(), which maps the boolean return value to IRQ_NONE ("false") or IRQ_HANDLED ("true"). Then handle_percpu_demux_irq() conditionally updates the stats based on the return value from vmbus_percpu_isr(). With these changes, interrupts from VMBus channels are now processed as Linux IRQs that are demultiplexed from the main VMBus interrupt. Signed-off-by: Michael Kelley --- arch/x86/kernel/cpu/mshyperv.c | 9 ++-- drivers/hv/channel.c | 25 +++++++++- drivers/hv/hv_common.c | 2 +- drivers/hv/vmbus_drv.c | 84 +++++++++++++++++++--------------- include/asm-generic/mshyperv.h | 3 +- 5 files changed, 79 insertions(+), 44 deletions(-) diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c index e0fd57a8ba84..18bc282a99db 100644 --- a/arch/x86/kernel/cpu/mshyperv.c +++ b/arch/x86/kernel/cpu/mshyperv.c @@ -110,7 +110,7 @@ void hv_set_msr(unsigned int reg, u64 value) } EXPORT_SYMBOL_GPL(hv_set_msr); -static void (*vmbus_handler)(void); +static bool (*vmbus_handler)(void); static void (*hv_stimer0_handler)(void); static void (*hv_kexec_handler)(void); static void (*hv_crash_handler)(struct pt_regs *regs); @@ -119,9 +119,8 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback) { struct pt_regs *old_regs = set_irq_regs(regs); - inc_irq_stat(irq_hv_callback_count); - if (vmbus_handler) - vmbus_handler(); + if (vmbus_handler && vmbus_handler()) + inc_irq_stat(irq_hv_callback_count); if (ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED) apic_eoi(); @@ -129,7 +128,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback) set_irq_regs(old_regs); } -void hv_setup_vmbus_handler(void (*handler)(void)) +void hv_setup_vmbus_handler(bool (*handler)(void)) { vmbus_handler = handler; } diff --git a/drivers/hv/channel.c b/drivers/hv/channel.c index fb8cd8469328..1aa020b538f1 100644 --- a/drivers/hv/channel.c +++ b/drivers/hv/channel.c @@ -638,6 +638,7 @@ static int __vmbus_open(struct vmbus_channel *newchannel, struct vmbus_channel_open_channel *open_msg; struct vmbus_channel_msginfo *open_info = NULL; struct page *page = newchannel->ringbuffer_page; + u32 relid = newchannel->offermsg.child_relid; u32 send_pages, recv_pages; unsigned long flags; int err; @@ -685,13 +686,31 @@ static int __vmbus_open(struct vmbus_channel *newchannel, if (err) goto error_free_gpadl; + /* Request the IRQ and assign to target_cpu */ + err = request_irq(newchannel->irq, vmbus_chan_handler, 0, + newchannel->irq_name, newchannel); + if (err) { + pr_err("request_irq failed with %d for relid %d irq %d\n", + err, relid, newchannel->irq); + goto error_free_gpadl; + } + err = irq_set_affinity_and_hint(newchannel->irq, + cpumask_of(newchannel->target_cpu)); + if (err) { + pr_err("irq_set_affinity_and_hint failed with %d for relid %d irq %d\n", + err, relid, newchannel->irq); + free_irq(newchannel->irq, newchannel); + goto error_free_gpadl; + } + newchannel->irq_requested = true; + /* Create and init the channel open message */ open_info = kzalloc(sizeof(*open_info) + sizeof(struct vmbus_channel_open_channel), GFP_KERNEL); if (!open_info) { err = -ENOMEM; - goto error_free_gpadl; + goto error_free_irq; } init_completion(&open_info->waitevent); @@ -759,6 +778,10 @@ static int __vmbus_open(struct vmbus_channel *newchannel, spin_unlock_irqrestore(&vmbus_connection.channelmsg_lock, flags); error_free_info: kfree(open_info); +error_free_irq: + irq_update_affinity_hint(newchannel->irq, NULL); + free_irq(newchannel->irq, newchannel); + newchannel->irq_requested = false; error_free_gpadl: vmbus_teardown_gpadl(newchannel, &newchannel->ringbuffer_gpadlhandle); error_clean_ring: diff --git a/drivers/hv/hv_common.c b/drivers/hv/hv_common.c index 9c452bfbd571..38a23add721c 100644 --- a/drivers/hv/hv_common.c +++ b/drivers/hv/hv_common.c @@ -610,7 +610,7 @@ bool __weak hv_isolation_type_tdx(void) } EXPORT_SYMBOL_GPL(hv_isolation_type_tdx); -void __weak hv_setup_vmbus_handler(void (*handler)(void)) +void __weak hv_setup_vmbus_handler(bool (*handler)(void)) { } EXPORT_SYMBOL_GPL(hv_setup_vmbus_handler); diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index 8fd03d41e71a..b73be7c02d37 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -1193,6 +1193,45 @@ static void vmbus_force_channel_rescinded(struct vmbus_channel *channel) } #endif /* CONFIG_PM_SLEEP */ +irqreturn_t vmbus_chan_handler(int irq, void *dev_id) +{ + void (*callback_fn)(void *context); + struct vmbus_channel *channel = dev_id; + + /* + * Make sure that the ring buffer data structure doesn't get + * freed while we dereference the ring buffer pointer. Test + * for the channel's onchannel_callback being NULL within a + * sched_lock critical section. See also the inline comments + * in vmbus_reset_channel_cb(). + */ + spin_lock(&channel->sched_lock); + + callback_fn = channel->onchannel_callback; + if (unlikely(callback_fn == NULL)) + goto spin_unlock; + + trace_vmbus_chan_sched(channel); + + ++channel->interrupts; + + switch (channel->callback_mode) { + case HV_CALL_ISR: + (*callback_fn)(channel->channel_callback_context); + break; + + case HV_CALL_BATCHED: + hv_begin_read(&channel->inbound); + fallthrough; + case HV_CALL_DIRECT: + tasklet_schedule(&channel->callback_event); + } + +spin_unlock: + spin_unlock(&channel->sched_lock); + return IRQ_HANDLED; +} + /* * Schedule all channels with events pending */ @@ -1217,7 +1256,6 @@ static void vmbus_chan_sched(struct hv_per_cpu_context *hv_cpu) return; for_each_set_bit(relid, recv_int_page, maxbits) { - void (*callback_fn)(void *context); struct vmbus_channel *channel; struct irq_desc *desc; @@ -1244,43 +1282,14 @@ static void vmbus_chan_sched(struct hv_per_cpu_context *hv_cpu) if (channel->rescind) goto sched_unlock_rcu; - /* - * Make sure that the ring buffer data structure doesn't get - * freed while we dereference the ring buffer pointer. Test - * for the channel's onchannel_callback being NULL within a - * sched_lock critical section. See also the inline comments - * in vmbus_reset_channel_cb(). - */ - spin_lock(&channel->sched_lock); - - callback_fn = channel->onchannel_callback; - if (unlikely(callback_fn == NULL)) - goto sched_unlock; - - trace_vmbus_chan_sched(channel); - - ++channel->interrupts; - - switch (channel->callback_mode) { - case HV_CALL_ISR: - (*callback_fn)(channel->channel_callback_context); - break; - - case HV_CALL_BATCHED: - hv_begin_read(&channel->inbound); - fallthrough; - case HV_CALL_DIRECT: - tasklet_schedule(&channel->callback_event); - } + generic_handle_irq_desc(desc); -sched_unlock: - spin_unlock(&channel->sched_lock); sched_unlock_rcu: rcu_read_unlock(); } } -static void vmbus_isr(void) +static bool vmbus_isr(void) { struct hv_per_cpu_context *hv_cpu = this_cpu_ptr(hv_context.cpu_context); @@ -1299,15 +1308,18 @@ static void vmbus_isr(void) vmbus_signal_eom(msg, HVMSG_TIMER_EXPIRED); } else tasklet_schedule(&hv_cpu->msg_dpc); - } - add_interrupt_randomness(vmbus_interrupt); + add_interrupt_randomness(vmbus_interrupt); + return true; + } + return false; } static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id) { - vmbus_isr(); - return IRQ_HANDLED; + if (vmbus_isr()) + return IRQ_HANDLED; + return IRQ_NONE; } int vmbus_irq_set_affinity(struct irq_data *data, diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h index 0488ff8b511f..0a5559b9d5f7 100644 --- a/include/asm-generic/mshyperv.h +++ b/include/asm-generic/mshyperv.h @@ -178,7 +178,7 @@ static inline void vmbus_signal_eom(struct hv_message *msg, u32 old_msg_type) int hv_get_hypervisor_version(union hv_hypervisor_version_info *info); -void hv_setup_vmbus_handler(void (*handler)(void)); +void hv_setup_vmbus_handler(bool (*handler)(void)); void hv_remove_vmbus_handler(void); void hv_setup_stimer0_handler(void (*handler)(void)); void hv_remove_stimer0_handler(void); @@ -188,6 +188,7 @@ void hv_remove_kexec_handler(void); void hv_setup_crash_handler(void (*handler)(struct pt_regs *regs)); void hv_remove_crash_handler(void); +extern irqreturn_t vmbus_chan_handler(int irq, void *dev_id); extern void vmbus_irq_mask(struct irq_data *data); extern void vmbus_irq_unmask(struct irq_data *data); extern int vmbus_irq_set_affinity(struct irq_data *data, From patchwork Tue Jun 4 05:09:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684649 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE12B1428EE; Tue, 4 Jun 2024 05:10:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477851; cv=none; b=Qhn2dV5LPKaCOCvH4P/KlCLA1mzCVVT5AKzWlCLxwO0CoxMkAGY41nOQnfw1RrmpWKOAALr4nz3tRp76ONG30K3QmCEUg/+fKznhJQKehLn99KGRI+8H7L9ygJJOkkvZNVckRroI6Ao5nb1miQX0LI4bNc1L03tWwoEZDRD16aA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477851; c=relaxed/simple; bh=XhjKDttjU9NrAh+Qpdc7TFdMquWNr+zfZHtc1ueQnWs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=M3i7/u3TgHEfBH1FMPqv3IfhYYWMbBI7/2FXj5SSRIRQDlpA/r6uN/FPhfNKGsnmblYpvYrW6X9mfK+c1qw869qdYxUliHWrUF2qNZLdRtbJnp3KPHOjwmKHCzYAcT5DG7P+Jjp6QZgTWxIJ/KZj1zOp9CH+soX/nBoli/dQkmw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Dvo2xr0W; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Dvo2xr0W" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-7024791a950so2443584b3a.0; Mon, 03 Jun 2024 22:10:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477849; x=1718082649; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=EwYlTiV6Rt4HKzKI2WcesBOG37oKFHVkVyKg4Lxmmlw=; b=Dvo2xr0WiYXxz25sKGb4Zj1hYJQDudlvVceFETSzX99/OMZhQGYh9aOAkg+sUFrNIi rlOOeyBfkD07cXdMvqNFE1Ucno2Hv5Mu8KSrqSkyK+T5HkTLbMHyTImoTjKun6QpdcNn +nHQwj75dmSNQIq16MOParA6BtLfDVeatvoR6J7efa7zvmvLvVJ97lpY8ap7GAwKXF6K dT7cemYfXc6U9S/ICQOUgOPbkCNF7A44FpHCBFLWKW7JWWtfzM8sYIU21V9cCpaGUqx6 bZ1XZ30r33ucXDUrwmuEaczZ1JmgRowQocYVNAtQEUCIixWC6xempw8ITFppy6Ex+cru Ympw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477849; x=1718082649; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=EwYlTiV6Rt4HKzKI2WcesBOG37oKFHVkVyKg4Lxmmlw=; b=oA5KHMP6DtG6nvMOWHoybHDcrRUUMxHbMoD9k1mS6uDFNUhloaqpdGGrvnaBtV257o t9vlBVV0/B/FrZRFhrMb0oie+DQzEPZ1MwanuP2LL0SxjEdkHFD6+NVjDi76zBv5ru0k T8IbfiI2R+a68EffGnOrMe4U6Kn+v27D6HLGG5mFoAmlr8iLbHcoVhlmIahyWLA7956l TeerAZhVpAmQwhsVyrHr7oPV/ufCatW3z+M0GUAxGn8s9Zk63+mAzvkCGrHTj44xcepl eG6mhWk6O5l1XQuYbi0FqgOUdGpCbXzOLmyW0vDI0AedNi1qMH6poHNhvSdljVyizgMD E6Dg== X-Forwarded-Encrypted: i=1; AJvYcCXvMa2F+VB2HS3ErxGaudLXAKTLoPJ+lmlzRNa0BQ4yY/XSbR0389/4RZ+FBtmuFSPAitmKM4kZaplrs8SIHYfJp73ckzmJCrHJG4pUIL9hz6fkVJ0eed9uvIQW39Nm59gw2+/Bw04U76h/HebSat3F1BjZjizGF6I9lofLsJlB4WGTbxDAPbbpLFKvPmkuvJN++2wP00gvicAtT6mwkl0hkg4SVjaxaFtTOTH49IQZtLRvO3M2+75owWTQ62o= X-Gm-Message-State: AOJu0YwWBrYE4cwphF1JamjRgSGHIa8n4W/D5lII/+fb3HNKYijGz9Dk PYfxl7Weye8eO0XM3OB/l3hF+T6UoJdvx9Of/IqqakCjl6WCWDbN X-Google-Smtp-Source: AGHT+IEfT2hnwHPbvFJIeRLoZD+/0+fscvURnMxyUITt/1Y3MBXO14CtptRiEclz06LVJZSxOY5bzQ== X-Received: by 2002:a05:6a20:2452:b0:1a8:2cd1:e493 with SMTP id adf61e73a8af0-1b2a345db4emr2209302637.29.1717477849084; Mon, 03 Jun 2024 22:10:49 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:48 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 10/12] Drivers: hv: vmbus: Implement vmbus_irq_set_affinity Date: Mon, 3 Jun 2024 22:09:38 -0700 Message-Id: <20240604050940.859909-11-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley Pull out core code from target_cpu_store() to implement vmbus_irq_set_affinity() so the affinity of VMBus channel interrupts can be updated from user space via /proc/irq. Since vmbus_irq_set_affinity() runs with interrupts disabled, vmbus_send_modifychannel() can't wait for an ACK from Hyper-V. As such, remove the "wait for ack" version of vmbus_send_modifychannel(). Not waiting isn't a problem unless the old CPU is quickly taken offline before Hyper-V makes the change, which is dealt with in a subsequent patch. Also change target_cpu_store() to call irq_set_affinity() so that changes made via /sys/bus/vmbus/devices//channels//cpu are in sync with the /proc/irq interface. The cpus_read_lock() is no longer needed in target_cpu_store() because irq_set_affinity() ensures that the interrupt affinity is not set to an offline CPU. Signed-off-by: Michael Kelley --- drivers/hv/channel.c | 97 ++++++------------------- drivers/hv/vmbus_drv.c | 161 +++++++++++++++++++++++++---------------- 2 files changed, 121 insertions(+), 137 deletions(-) diff --git a/drivers/hv/channel.c b/drivers/hv/channel.c index 1aa020b538f1..b7920072e243 100644 --- a/drivers/hv/channel.c +++ b/drivers/hv/channel.c @@ -212,79 +212,6 @@ int vmbus_send_tl_connect_request(const guid_t *shv_guest_servie_id, } EXPORT_SYMBOL_GPL(vmbus_send_tl_connect_request); -static int send_modifychannel_without_ack(struct vmbus_channel *channel, u32 target_vp) -{ - struct vmbus_channel_modifychannel msg; - int ret; - - memset(&msg, 0, sizeof(msg)); - msg.header.msgtype = CHANNELMSG_MODIFYCHANNEL; - msg.child_relid = channel->offermsg.child_relid; - msg.target_vp = target_vp; - - ret = vmbus_post_msg(&msg, sizeof(msg), true); - trace_vmbus_send_modifychannel(&msg, ret); - - return ret; -} - -static int send_modifychannel_with_ack(struct vmbus_channel *channel, u32 target_vp) -{ - struct vmbus_channel_modifychannel *msg; - struct vmbus_channel_msginfo *info; - unsigned long flags; - int ret; - - info = kzalloc(sizeof(struct vmbus_channel_msginfo) + - sizeof(struct vmbus_channel_modifychannel), - GFP_KERNEL); - if (!info) - return -ENOMEM; - - init_completion(&info->waitevent); - info->waiting_channel = channel; - - msg = (struct vmbus_channel_modifychannel *)info->msg; - msg->header.msgtype = CHANNELMSG_MODIFYCHANNEL; - msg->child_relid = channel->offermsg.child_relid; - msg->target_vp = target_vp; - - spin_lock_irqsave(&vmbus_connection.channelmsg_lock, flags); - list_add_tail(&info->msglistentry, &vmbus_connection.chn_msg_list); - spin_unlock_irqrestore(&vmbus_connection.channelmsg_lock, flags); - - ret = vmbus_post_msg(msg, sizeof(*msg), true); - trace_vmbus_send_modifychannel(msg, ret); - if (ret != 0) { - spin_lock_irqsave(&vmbus_connection.channelmsg_lock, flags); - list_del(&info->msglistentry); - spin_unlock_irqrestore(&vmbus_connection.channelmsg_lock, flags); - goto free_info; - } - - /* - * Release channel_mutex; otherwise, vmbus_onoffer_rescind() could block on - * the mutex and be unable to signal the completion. - * - * See the caller target_cpu_store() for information about the usage of the - * mutex. - */ - mutex_unlock(&vmbus_connection.channel_mutex); - wait_for_completion(&info->waitevent); - mutex_lock(&vmbus_connection.channel_mutex); - - spin_lock_irqsave(&vmbus_connection.channelmsg_lock, flags); - list_del(&info->msglistentry); - spin_unlock_irqrestore(&vmbus_connection.channelmsg_lock, flags); - - if (info->response.modify_response.status) - ret = -EAGAIN; - -free_info: - kfree(info); - return ret; -} - /* * Set/change the vCPU (@target_vp) the channel (@child_relid) will interrupt. * @@ -294,14 +221,32 @@ static int send_modifychannel_with_ack(struct vmbus_channel *channel, u32 target * out an ACK, we can not know when the host will stop interrupting the "old" * vCPU and start interrupting the "new" vCPU for the given channel. * + * But even if Hyper-V provides the ACK, we don't wait for it because the + * caller, vmbus_irq_set_affinity(), is running with a spin lock held. The + * unknown delay in when the host will start interrupting the new vCPU is not + * a problem unless the old vCPU is taken offline, and that situation is dealt + * with separately in the CPU offlining path. + * * The CHANNELMSG_MODIFYCHANNEL message type is supported since VMBus version * VERSION_WIN10_V4_1. */ int vmbus_send_modifychannel(struct vmbus_channel *channel, u32 target_vp) { - if (vmbus_proto_version >= VERSION_WIN10_V5_3) - return send_modifychannel_with_ack(channel, target_vp); - return send_modifychannel_without_ack(channel, target_vp); + struct vmbus_channel_modifychannel msg; + int ret; + + if (vmbus_proto_version < VERSION_WIN10_V4_1) + return -EINVAL; + + memset(&msg, 0, sizeof(msg)); + msg.header.msgtype = CHANNELMSG_MODIFYCHANNEL; + msg.child_relid = channel->offermsg.child_relid; + msg.target_vp = target_vp; + + ret = vmbus_post_msg(&msg, sizeof(msg), false); + trace_vmbus_send_modifychannel(&msg, ret); + + return ret; } EXPORT_SYMBOL_GPL(vmbus_send_modifychannel); diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index b73be7c02d37..87f2f3436136 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -22,7 +22,6 @@ #include #include #include -#include #include #include @@ -1322,10 +1321,107 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id) return IRQ_NONE; } +/* + * This function is invoked by user space affinity changes initiated + * from /proc/irq/ or from the legacy VMBus-specific interface at + * /sys/bus/vmbus/devices//channels//cpu. + * + * In the former case, the /proc implementation ensures that unmapping + * (i.e., deleting) the IRQ will pend while this function is in progress. + * Since deleting the channel unmaps the IRQ first, the channel can't go + * away either. + * + * In the latter case, the VMBus connection channel_mutex is held, which + * prevents channel deltion, and therefore IRQ unampping as well. + * + * So in both cases, accessing the channel and IRQ data structures is safe. + */ int vmbus_irq_set_affinity(struct irq_data *data, const struct cpumask *dest, bool force) { - return 0; + static int next_cpu; + static cpumask_t tempmask; + int origin_cpu, target_cpu; + struct vmbus_channel *channel = irq_data_get_irq_handler_data(data); + int ret; + + if (!channel) { + pr_err("Bad channel in vmbus_irq_set_affinity for relid %ld\n", + data->hwirq); + return -EINVAL; + } + + /* Don't consider CPUs that are isolated */ + if (housekeeping_enabled(HK_TYPE_MANAGED_IRQ)) + cpumask_and(&tempmask, dest, + housekeeping_cpumask(HK_TYPE_MANAGED_IRQ)); + else + cpumask_copy(&tempmask, dest); + + /* + * If Hyper-V is already targeting a CPU in the new affinity mask, + * keep that targeting and Hyper-V doesn't need to be updated. But + * still set effective affinity as it may be unset when the IRQ is + * first created. + */ + origin_cpu = channel->target_cpu; + if (cpumask_test_cpu(origin_cpu, &tempmask)) { + target_cpu = origin_cpu; + goto update_effective; + } + + /* + * Pick a CPU from the new affinity mask. As a simple heuristic to + * spread out the selection when the mask contains multiple CPUs, + * start with whatever CPU was last selected. + */ + target_cpu = cpumask_next_wrap(next_cpu, &tempmask, nr_cpu_ids, false); + if (target_cpu >= nr_cpu_ids) + return -EINVAL; + next_cpu = target_cpu; + + /* + * Hyper-V will ignore MODIFYCHANNEL messages for "non-open" channels; + * avoid sending the message and fail here for such channels. + */ + if (channel->state != CHANNEL_OPENED_STATE) + return -EIO; + + ret = vmbus_send_modifychannel(channel, + hv_cpu_number_to_vp_number(target_cpu)); + if (ret) + return ret; + + /* + * Warning. At this point, there is *no* guarantee that the host will + * have successfully processed the vmbus_send_modifychannel() request. + * See the header comment of vmbus_send_modifychannel() for more info. + * + * Lags in the processing of the above vmbus_send_modifychannel() can + * result in missed interrupts if the "old" target CPU is taken offline + * before Hyper-V starts sending interrupts to the "new" target CPU. + * hv_synic_cleanup() will ensure no interrupts are missed. + * + * But apart from this offlining scenario, the code tolerates such + * lags. It will function correctly even if a channel interrupt comes + * in on a CPU that is different from the channel target_cpu value. + */ + + channel->target_cpu = target_cpu; + + /* See init_vp_index(). */ + if (hv_is_perf_channel(channel)) + hv_update_allocated_cpus(origin_cpu, target_cpu); + + /* Currently set only for storvsc channels. */ + if (channel->change_target_cpu_callback) { + (*channel->change_target_cpu_callback)(channel, + origin_cpu, target_cpu); + } + +update_effective: + irq_data_update_effective_affinity(data, cpumask_of(target_cpu)); + return IRQ_SET_MASK_OK; } /* @@ -1655,7 +1751,7 @@ static ssize_t target_cpu_show(struct vmbus_channel *channel, char *buf) static ssize_t target_cpu_store(struct vmbus_channel *channel, const char *buf, size_t count) { - u32 target_cpu, origin_cpu; + u32 target_cpu; ssize_t ret = count; if (vmbus_proto_version < VERSION_WIN10_V4_1) @@ -1668,17 +1764,6 @@ static ssize_t target_cpu_store(struct vmbus_channel *channel, if (target_cpu >= nr_cpumask_bits) return -EINVAL; - if (!cpumask_test_cpu(target_cpu, housekeeping_cpumask(HK_TYPE_MANAGED_IRQ))) - return -EINVAL; - - /* No CPUs should come up or down during this. */ - cpus_read_lock(); - - if (!cpu_online(target_cpu)) { - cpus_read_unlock(); - return -EINVAL; - } - /* * Synchronizes target_cpu_store() and channel closure: * @@ -1703,55 +1788,9 @@ static ssize_t target_cpu_store(struct vmbus_channel *channel, */ mutex_lock(&vmbus_connection.channel_mutex); - /* - * Hyper-V will ignore MODIFYCHANNEL messages for "non-open" channels; - * avoid sending the message and fail here for such channels. - */ - if (channel->state != CHANNEL_OPENED_STATE) { - ret = -EIO; - goto cpu_store_unlock; - } - - origin_cpu = channel->target_cpu; - if (target_cpu == origin_cpu) - goto cpu_store_unlock; - - if (vmbus_send_modifychannel(channel, - hv_cpu_number_to_vp_number(target_cpu))) { - ret = -EIO; - goto cpu_store_unlock; - } - - /* - * For version before VERSION_WIN10_V5_3, the following warning holds: - * - * Warning. At this point, there is *no* guarantee that the host will - * have successfully processed the vmbus_send_modifychannel() request. - * See the header comment of vmbus_send_modifychannel() for more info. - * - * Lags in the processing of the above vmbus_send_modifychannel() can - * result in missed interrupts if the "old" target CPU is taken offline - * before Hyper-V starts sending interrupts to the "new" target CPU. - * But apart from this offlining scenario, the code tolerates such - * lags. It will function correctly even if a channel interrupt comes - * in on a CPU that is different from the channel target_cpu value. - */ - - channel->target_cpu = target_cpu; - - /* See init_vp_index(). */ - if (hv_is_perf_channel(channel)) - hv_update_allocated_cpus(origin_cpu, target_cpu); - - /* Currently set only for storvsc channels. */ - if (channel->change_target_cpu_callback) { - (*channel->change_target_cpu_callback)(channel, - origin_cpu, target_cpu); - } + ret = irq_set_affinity(channel->irq, cpumask_of(target_cpu)); -cpu_store_unlock: mutex_unlock(&vmbus_connection.channel_mutex); - cpus_read_unlock(); return ret; } static VMBUS_CHAN_ATTR(cpu, 0644, target_cpu_show, target_cpu_store); From patchwork Tue Jun 4 05:09:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684650 Received: from mail-oo1-f50.google.com (mail-oo1-f50.google.com [209.85.161.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A5F714291B; Tue, 4 Jun 2024 05:10:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477853; cv=none; b=ZNcHb10xtJjNs4NB4vOeqqUC4Z2BMRqEqrJUA7hPcDH6Ly3qhizI/J4RDOpWG5p5zD/fOt2UjOfz+jFGWqqo7+s0CUxf9frPePTQYGS0EzzStd5Pm0ALbrcdsTXuT6R2VKj8KgZjkoQ7Qp+2/HA4Z0eyIwnZ0UzXvDG3QkDcE8k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477853; c=relaxed/simple; bh=1jVA1hbkuJFk2X1Wq2r6QErsU8Q1jsab7e/GaD/Uy9A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=u6eU6d/srp+MUTTnikiNGGfoDkv7FXUZ7/H4/IbWG8iWF431PTf7bO0MjF5FHKw2TOmUpAzKys9lUnTmTAE2WC3v6SePBjOf0QH+vK71tlSMdRUsxPD3bMyTSRtbGft2naVv+UjWcvrGYK05INV9t8He6uNdvNJt99xj0QNC7ds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kGuexhTD; arc=none smtp.client-ip=209.85.161.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kGuexhTD" Received: by mail-oo1-f50.google.com with SMTP id 006d021491bc7-5b970a97e8eso364474eaf.1; Mon, 03 Jun 2024 22:10:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477850; x=1718082650; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=pVV6Kv0tGk5nmsV3DBWEtw/NPqCxYSXN7XTZ22q11dg=; b=kGuexhTDqFeWTDafOqg1qOhkm5DYBi+TIpnFpS9UgUyFz3jEt7tgd7RhDKpngD35+7 B5OOqs7BsKkCgj24e4ZcwRLeaQPBbP8zvbJB53/mFirRvbYq5Ah05rNgAb7UlNAVC0h/ cZbagt/2bcZ69K4ezI1b9vtQuwLewtfKoP5BA2JGKq4Bq4SXUbQb62w/ENAos3uUV3Cu Z7uh3egA0tokhelxHjEbgFfi9Vh1vtVlsVTJNkLw5xl6jOv9aF8d+XQ5cJVOXf/1LwEc dNMR3OxsX7TJXlj/GFGFR8b1U/zd+1ZVOYBfV7ZaxQwSjsXnJHzdETuNZIdmXyKNd3cZ Oo2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477850; x=1718082650; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=pVV6Kv0tGk5nmsV3DBWEtw/NPqCxYSXN7XTZ22q11dg=; b=jLd1sBne9UX1vW7mJeE5JWnraJPw2lVexNVC8GwnUiOiSraOkx3taz8qVxaKaxrFvz gVhFm7F7hCehOx1P5oHie4GRkNxRrwA7IfTkU1/FLiqqT3Jvx3oEPdDzHW3EhtnMi4E8 sLaWPLPHlwErjT1Gb1kkF00U4Q5Z46XFeg0yHB0HmfoX1C9BGLstGwBHKJX3vs3eKnqc kZ2pRKIy02EOiwALvX1n6nbzUpFEmGlyPc9/E3B5eO02t+pawD29ys06yc1HThmHrd5w DkscbXQrPG19zSRS7WU6NlwkyQXN8ooRoa6ZZH0WXn/m5XvJwLdLXaNQygNkRiPoFzWr 2ETw== X-Forwarded-Encrypted: i=1; AJvYcCX3Wl+oyj1DD0852BwA32pLa/rMem68QB5xMnGVSRnkch+iu3hnNORjVI0WS0lDNwZXOghB2VDjRWLMepzmIBrZFPAu5c6TB3QdbYBmr0+3PqMMdWflO54LWJnwVWq0ID65xHEh2VPMnfTeyr7nexWP/+qUjRAMjvd5pJqdz+Lx6PYJdHfSAtPawF+r9Lbb0aH2MA6I4/TAREqsHRKab6xqOk63DtA7wyqTp3BGJUOPN2t5XLP1mA4rcNDHolQ= X-Gm-Message-State: AOJu0YwrIUcKciZisCei0bTuYpBxDNJ0IdN/MZd/LTOt8lMd4i6kGq3N tkGGBNX10bqHO6RcjpkV0JsVS2zZ1u4VmAmtwcA0pUI7G8JBcIf2 X-Google-Smtp-Source: AGHT+IFF4pZiPOoszBF2RPe9oAaHaW1lz4eLkIWpJh01f0tjODzp1ogdUGgeuXzlAkHajFkd9k+xLA== X-Received: by 2002:a05:6820:1c91:b0:5b5:1f77:9603 with SMTP id 006d021491bc7-5ba05dc205cmr11327808eaf.8.1717477850379; Mon, 03 Jun 2024 22:10:50 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:50 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 11/12] Drivers: hv: vmbus: Wait for MODIFYCHANNEL to finish when offlining CPUs Date: Mon, 3 Jun 2024 22:09:39 -0700 Message-Id: <20240604050940.859909-12-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley vmbus_irq_set_affinity() may issue a MODIFYCHANNEL request to Hyper-V to target a VMBus channel interrupt to a different CPU. While newer versions of Hyper-V send a response to the guest when the change is complete, vmbus_irq_set_affinity() does not wait for the response because it is running with interrupts disabled. So Hyper-V may continue to direct interrupts to the old CPU for a short window after vmbus_irq_set_affinity() completes. This lag is not a problem during normal operation. But if the old CPU is taken offline during that window, Hyper-V may drop the interrupt because the synic in the target CPU is disabled. Dropping the interrupt may cause the VMBus channel to hang. To prevent this, wait for in-process MODIFYCHANNEL requests when taking a CPU offline. On newer versions of Hyper-V, completion can be confirmed by waiting for the response sent by Hyper-V. But on older versions of Hyper-V that don't send a response, wait a fixed interval of time that empirically should be "long enough", as that's the best that can be done. Signed-off-by: Michael Kelley --- drivers/hv/channel.c | 3 ++ drivers/hv/channel_mgmt.c | 32 ++++-------------- drivers/hv/hv.c | 70 +++++++++++++++++++++++++++++++++++---- drivers/hv/hyperv_vmbus.h | 8 +++++ 4 files changed, 81 insertions(+), 32 deletions(-) diff --git a/drivers/hv/channel.c b/drivers/hv/channel.c index b7920072e243..b7ee95373049 100644 --- a/drivers/hv/channel.c +++ b/drivers/hv/channel.c @@ -246,6 +246,9 @@ int vmbus_send_modifychannel(struct vmbus_channel *channel, u32 target_vp) ret = vmbus_post_msg(&msg, sizeof(msg), false); trace_vmbus_send_modifychannel(&msg, ret); + if (!ret) + vmbus_connection.modchan_sent++; + return ret; } EXPORT_SYMBOL_GPL(vmbus_send_modifychannel); diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c index da42aaae994e..960a2f0367d8 100644 --- a/drivers/hv/channel_mgmt.c +++ b/drivers/hv/channel_mgmt.c @@ -1465,40 +1465,20 @@ static void vmbus_ongpadl_created(struct vmbus_channel_message_header *hdr) * vmbus_onmodifychannel_response - Modify Channel response handler. * * This is invoked when we received a response to our channel modify request. - * Find the matching request, copy the response and signal the requesting thread. + * Increment the count of responses received. No locking is needed because + * responses are always received on the VMBUS_CONNECT_CPU. */ static void vmbus_onmodifychannel_response(struct vmbus_channel_message_header *hdr) { struct vmbus_channel_modifychannel_response *response; - struct vmbus_channel_msginfo *msginfo; - unsigned long flags; response = (struct vmbus_channel_modifychannel_response *)hdr; + if (response->status) + pr_err("Error status %x in MODIFYCHANNEL response for relid %d\n", + response->status, response->child_relid); + vmbus_connection.modchan_completed++; trace_vmbus_onmodifychannel_response(response); - - /* - * Find the modify msg, copy the response and signal/unblock the wait event. - */ - spin_lock_irqsave(&vmbus_connection.channelmsg_lock, flags); - - list_for_each_entry(msginfo, &vmbus_connection.chn_msg_list, msglistentry) { - struct vmbus_channel_message_header *responseheader = - (struct vmbus_channel_message_header *)msginfo->msg; - - if (responseheader->msgtype == CHANNELMSG_MODIFYCHANNEL) { - struct vmbus_channel_modifychannel *modifymsg; - - modifymsg = (struct vmbus_channel_modifychannel *)msginfo->msg; - if (modifymsg->child_relid == response->child_relid) { - memcpy(&msginfo->response.modify_response, response, - sizeof(*response)); - complete(&msginfo->waitevent); - break; - } - } - } - spin_unlock_irqrestore(&vmbus_connection.channelmsg_lock, flags); } /* diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c index a8ad728354cb..76658dfc5008 100644 --- a/drivers/hv/hv.c +++ b/drivers/hv/hv.c @@ -401,6 +401,56 @@ void hv_synic_disable_regs(unsigned int cpu) disable_percpu_irq(vmbus_irq); } +static void hv_synic_wait_for_modifychannel(int cpu) +{ + int i = 5; + u64 base; + + /* + * If we're on a VMBus version where MODIFYCHANNEL doesn't send acks, + * just sleep for 20 milliseconds and hope that gives Hyper-V enough + * time to process them. Empirical data on recent server-class CPUs + * (both x86 and arm64) shows that the Hyper-V response is typically + * received and processed in the guest within a few hundred + * microseconds. The 20 millisecond wait is somewhat arbitrary and + * intended to give plenty to time in case there are multiple + * MODIFYCHANNEL requests in progress and the host is busy. It's + * the best we can do. + */ + if (vmbus_proto_version < VERSION_WIN10_V5_3) { + usleep_range(20000, 25000); + return; + } + + /* + * Otherwise compare the current value of modchan_completed against + * modchan_sent. If some MODIFYCHANNEL requests have been sent that + * haven't completed, sleep 5 milliseconds and check again. If the + * requests still haven't completed after 5 attempts, output a + * message and proceed anyway. + * + * Hyper-V guarantees to process MODIFYCHANNEL requests in the order + * they are received from the guest, so simply comparing the counts + * is sufficient. + * + * Note that this check may encompass MODIFYCHANNEL requests that are + * unrelated to the CPU that is going offline. But the only effect is + * to potentially wait a little bit longer than necessary. CPUs going + * offline and affinity changes that result in MODIFYCHANNEL are + * relatively rare and it's not worth the complexity to track them more + * precisely. + */ + base = READ_ONCE(vmbus_connection.modchan_sent); + while (READ_ONCE(vmbus_connection.modchan_completed) < base && i) { + usleep_range(5000, 10000); + i--; + } + + if (i == 0) + pr_err("Timed out waiting for MODIFYCHANNEL. CPU %d sent %lld completed %lld\n", + cpu, base, vmbus_connection.modchan_completed); +} + #define HV_MAX_TRIES 3 /* * Scan the event flags page of 'this' CPU looking for any bit that is set. If we find one @@ -485,13 +535,21 @@ int hv_synic_cleanup(unsigned int cpu) /* * channel_found == false means that any channels that were previously * assigned to the CPU have been reassigned elsewhere with a call of - * vmbus_send_modifychannel(). Scan the event flags page looking for - * bits that are set and waiting with a timeout for vmbus_chan_sched() - * to process such bits. If bits are still set after this operation - * and VMBus is connected, fail the CPU offlining operation. + * vmbus_send_modifychannel(). First wait until any MODIFYCHANNEL + * requests have been completed by Hyper-V, after which we know that + * no new bits in the event flags will be set. Then scan the event flags + * page looking for bits that are set and waiting with a timeout for + * vmbus_chan_sched() to process such bits. If bits are still set + * after this operation, fail the CPU offlining operation. */ - if (vmbus_proto_version >= VERSION_WIN10_V4_1 && hv_synic_event_pending()) - return -EBUSY; + if (vmbus_proto_version >= VERSION_WIN10_V4_1) { + hv_synic_wait_for_modifychannel(cpu); + if (hv_synic_event_pending()) { + pr_err("Events pending when trying to offline CPU %d\n", + cpu); + return -EBUSY; + } + } always_cleanup: hv_stimer_legacy_cleanup(cpu); diff --git a/drivers/hv/hyperv_vmbus.h b/drivers/hv/hyperv_vmbus.h index bf35bb40c55e..571b2955b38e 100644 --- a/drivers/hv/hyperv_vmbus.h +++ b/drivers/hv/hyperv_vmbus.h @@ -264,6 +264,14 @@ struct vmbus_connection { struct irq_domain *vmbus_irq_domain; struct irq_chip vmbus_irq_chip; + /* + * VM-wide counts of MODIFYCHANNEL messages sent and completed. + * Used when taking a CPU offline to make sure the relevant + * MODIFYCHANNEL messages have been completed. + */ + u64 modchan_sent; + u64 modchan_completed; + /* * An offer message is handled first on the work_queue, and then * is further handled on handle_primary_chan_wq or From patchwork Tue Jun 4 05:09:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Kelley X-Patchwork-Id: 13684651 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8859513DBAA; Tue, 4 Jun 2024 05:10:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477854; cv=none; b=NKYiN2BvM9dYR9OxO+vrZx/p9gpXhRFMXP7tUJk10fmpiSyYJvyf1TC8pOITXj8oJDmFyb0Jxi7gCr4ETRp3a94EsjXnpXLmoMa2QkCvnnpliJujloWHR8XSEFPR92vUGI9kI3ifHnNZkypScBrMfuN9CGjFAHcg8A4zzPfraAI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717477854; c=relaxed/simple; bh=yD6QlTva6OVI3xK1DDtiQ1txjxXIzA1gAMGkk74myI4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IZJWxbiEBDOEPOHZXPrrDAm/SRE2jhBrm390K7YGaNkAAa7bdqoS+au6LLmhxhHCkh2qyN223IJJdpgb7X0ttYD44nbDd2+/GmzlJG1qC+bjXu32wFBFVXUR5tUJOFTjtRJLYEAkWZemyTwFrHGOeKw7dkRFR8FndIycjB8Zobw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NgWKX8v6; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NgWKX8v6" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-7025ca8bebcso1997746b3a.3; Mon, 03 Jun 2024 22:10:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717477852; x=1718082652; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=UqE8PqDSTScf2ssH6F4PbWUoO9g+WHgsI4Glih1KQPg=; b=NgWKX8v61be7LwLmyuKYqzvXSqjJnLUVlS0uiVB0qYhd5KQnTRShItJOXGREtEZh4L kqMMezuE9b1uyVntAnmfONuWXxpZiiAcNyIHGS9gsgOgPvYPVuAY7s8md8bTsCGpBcY2 gtCf+9Ey7ccFqN7XLBFHNhpbD9clOBVH4dOB2cVlAe2e5fUAVh57PkJXXANdqrySF6nh gC1FLafi1NYNxMe7lBK2ggghZ6jrxi/b7Sff1cHnyNwYXZ9x37/2RmQKsFSfUmADAy6x TTGSQCQJ7WnHGJpftDEIep30EhrlmtaFdwuRLPyMjN6C+eNamVV+C+MSBlyEwV5r0urj kNuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717477852; x=1718082652; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=UqE8PqDSTScf2ssH6F4PbWUoO9g+WHgsI4Glih1KQPg=; b=wLfKc4tRGd1dbBQl7ZpncSVBeTW17sZ2GKvvbzjRX9C2DhOFBkpI8vBItmqtHTYhXS BskRt4MaGauTfo9HMolZ97nZbmZj8Z8OJuCTM9g3sffv6qdbocs9v2t8C/dlQdQRhvPN vaaaKK3JISkTNlytkAZH2qfbLyJxnCVucPeJEdIoR+WZcfrm/eJpgfk7M2wkp4zNyjBJ qMXPLJi7FsHwgl/sXI78P1ySc9qztZ6J9IGNHvbIXkU3dhUgxYTLi1gYz0b6MewBjM3M YUHhVsMHD9GznMWASY3zUxMnVDUhlQ0ToW60qIgqvi9CaaHmuawcFWQbMl0Pl4u2th8S aznA== X-Forwarded-Encrypted: i=1; AJvYcCUndx/mfGrforncp5/nJzVvf6kW1bgxbszSrvd7vxDiye6X7s8sJbcw48G9Pv5YrGFuKbaEdfTNrn6adQKyhjl/uxTlbjlE0Dg7jBoDprv6zj3D+sNROXf8WrCB6+i103YjKMJF9LMpfZ1dD1plipiGySVWqBmvZWrcz3e7rQLDStML025fGynR3GwMcyYTpSbiB/REvgcPp2Ic/TM9TDURey2hyuwvJ00ox8d2wiNHNx45TABbNX/mBf3D2Aw= X-Gm-Message-State: AOJu0YzkbCR6PFflAlnxIh3oLdP/3OoNjoN8qtxokRtd/4Ji6JLHCKfL Q2Vt7UkwAYvS90R3LjGPmV+2w6q19TzyHpuVxcvE8D2Z/TRLJ3xu X-Google-Smtp-Source: AGHT+IHQeTJUGPqk9fJQ3bYqVezOkQwad17MvaE3G/i0Wg6Qa5p2Z3v+Yzuh/jC062uvo4bGBB7jUw== X-Received: by 2002:a05:6a00:847:b0:6e9:38d0:5019 with SMTP id d2e1a72fcca58-70247664028mr13199794b3a.0.1717477851685; Mon, 03 Jun 2024 22:10:51 -0700 (PDT) Received: from localhost.localdomain (c-67-161-114-176.hsd1.wa.comcast.net. [67.161.114.176]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70242c270c7sm6298153b3a.220.2024.06.03.22.10.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 22:10:51 -0700 (PDT) From: mhkelley58@gmail.com X-Google-Original-From: mhklinux@outlook.com To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, lpieralisi@kernel.org, kw@linux.com, robh@kernel.org, bhelgaas@google.com, James.Bottomley@HansenPartnership.com, martin.petersen@oracle.com, arnd@arndb.de, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-arch@vger.kernel.org Cc: maz@kernel.org, den@valinux.co.jp, jgowans@amazon.com, dawei.li@shingroup.cn Subject: [RFC 12/12] Drivers: hv: vmbus: Ensure IRQ affinity isn't set to a CPU going offline Date: Mon, 3 Jun 2024 22:09:40 -0700 Message-Id: <20240604050940.859909-13-mhklinux@outlook.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240604050940.859909-1-mhklinux@outlook.com> References: <20240604050940.859909-1-mhklinux@outlook.com> Reply-To: mhklinux@outlook.com Precedence: bulk X-Mailing-List: linux-scsi@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Michael Kelley hv_synic_cleanup() currently prevents a CPU from going offline if any VMBus channel IRQs are targeted at that CPU. However, current code has a race in that an IRQ could be affinitized to the CPU after the check in hv_synic_cleanup() and before the CPU is removed from cpu_online_mask. Any channel interrupts could be lost and the channel would hang. Fix this by adding a flag for each CPU indicating if the synic is online. Filter the new affinity with these flags so that vmbus_irq_set_affinity() doesn't pick a CPU where the synic is already offline. Also add a spin lock so that vmbus_irq_set_affinity() changing the channel target_cpu and sending the MODIFYCHANNEL message to Hyper-V are atomic with respect to the checks made in hv_synic_cleanup(). hv_synic_cleanup() needs these operations to be atomic so that it can correctly count the MODIFYCHANNEL messages that need to be ack'ed by Hyper-V. Signed-off-by: Michael Kelley --- drivers/hv/connection.c | 1 + drivers/hv/hv.c | 22 ++++++++++++++++++++-- drivers/hv/hyperv_vmbus.h | 2 ++ drivers/hv/vmbus_drv.c | 34 ++++++++++++++++++++++++++++------ 4 files changed, 51 insertions(+), 8 deletions(-) diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c index a105eecdeec2..b44ce3d39135 100644 --- a/drivers/hv/connection.c +++ b/drivers/hv/connection.c @@ -213,6 +213,7 @@ int vmbus_connect(void) INIT_LIST_HEAD(&vmbus_connection.chn_list); mutex_init(&vmbus_connection.channel_mutex); + spin_lock_init(&vmbus_connection.set_affinity_lock); /* * Setup the vmbus event connection for channel interrupt diff --git a/drivers/hv/hv.c b/drivers/hv/hv.c index 76658dfc5008..89e491219129 100644 --- a/drivers/hv/hv.c +++ b/drivers/hv/hv.c @@ -338,6 +338,8 @@ int hv_synic_init(unsigned int cpu) { hv_synic_enable_regs(cpu); + cpumask_set_cpu(cpu, &vmbus_connection.synic_online); + hv_stimer_legacy_init(cpu, VMBUS_MESSAGE_SINT); return 0; @@ -513,6 +515,17 @@ int hv_synic_cleanup(unsigned int cpu) * TODO: Re-bind the channels to different CPUs. */ mutex_lock(&vmbus_connection.channel_mutex); + spin_lock(&vmbus_connection.set_affinity_lock); + + /* + * Once the check for channels assigned to this CPU is complete, we + * must not allow a channel to be assigned to this CPU. So mark + * the synic as no longer online. This cpumask is checked in + * vmbus_irq_set_affinity() to prevent setting the affinity of + * an IRQ to such a CPU. + */ + cpumask_clear_cpu(cpu, &vmbus_connection.synic_online); + list_for_each_entry(channel, &vmbus_connection.chn_list, listentry) { if (channel->target_cpu == cpu) { channel_found = true; @@ -527,10 +540,11 @@ int hv_synic_cleanup(unsigned int cpu) if (channel_found) break; } + spin_unlock(&vmbus_connection.set_affinity_lock); mutex_unlock(&vmbus_connection.channel_mutex); if (channel_found) - return -EBUSY; + goto set_online; /* * channel_found == false means that any channels that were previously @@ -547,7 +561,7 @@ int hv_synic_cleanup(unsigned int cpu) if (hv_synic_event_pending()) { pr_err("Events pending when trying to offline CPU %d\n", cpu); - return -EBUSY; + goto set_online; } } @@ -557,4 +571,8 @@ int hv_synic_cleanup(unsigned int cpu) hv_synic_disable_regs(cpu); return 0; + +set_online: + cpumask_set_cpu(cpu, &vmbus_connection.synic_online); + return -EBUSY; } diff --git a/drivers/hv/hyperv_vmbus.h b/drivers/hv/hyperv_vmbus.h index 571b2955b38e..92ae5af10778 100644 --- a/drivers/hv/hyperv_vmbus.h +++ b/drivers/hv/hyperv_vmbus.h @@ -263,6 +263,8 @@ struct vmbus_connection { struct fwnode_handle *vmbus_fwnode; struct irq_domain *vmbus_irq_domain; struct irq_chip vmbus_irq_chip; + cpumask_t synic_online; + spinlock_t set_affinity_lock; /* * VM-wide counts of MODIFYCHANNEL messages sent and completed. diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index 87f2f3436136..3430ad42d7ba 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -1351,6 +1351,14 @@ int vmbus_irq_set_affinity(struct irq_data *data, return -EINVAL; } + /* + * The spin lock must be held so that checking synic_online, sending + * the MODIFYCHANNEL message, and setting channel->target_cpu are + * atomic with respect to hv_synic_cleanup() clearing the CPU in + * synic_online and doing the search. + */ + spin_lock(&vmbus_connection.set_affinity_lock); + /* Don't consider CPUs that are isolated */ if (housekeeping_enabled(HK_TYPE_MANAGED_IRQ)) cpumask_and(&tempmask, dest, @@ -1367,30 +1375,39 @@ int vmbus_irq_set_affinity(struct irq_data *data, origin_cpu = channel->target_cpu; if (cpumask_test_cpu(origin_cpu, &tempmask)) { target_cpu = origin_cpu; + spin_unlock(&vmbus_connection.set_affinity_lock); goto update_effective; } /* * Pick a CPU from the new affinity mask. As a simple heuristic to * spread out the selection when the mask contains multiple CPUs, - * start with whatever CPU was last selected. + * start with whatever CPU was last selected. Also filter out any + * CPUs where synic_online isn't set -- these CPUs are in the process + * of going offline and must not have channel interrupts assigned + * to them. */ + cpumask_and(&tempmask, &tempmask, &vmbus_connection.synic_online); target_cpu = cpumask_next_wrap(next_cpu, &tempmask, nr_cpu_ids, false); - if (target_cpu >= nr_cpu_ids) - return -EINVAL; + if (target_cpu >= nr_cpu_ids) { + ret = -EINVAL; + goto unlock; + } next_cpu = target_cpu; /* * Hyper-V will ignore MODIFYCHANNEL messages for "non-open" channels; * avoid sending the message and fail here for such channels. */ - if (channel->state != CHANNEL_OPENED_STATE) - return -EIO; + if (channel->state != CHANNEL_OPENED_STATE) { + ret = -EIO; + goto unlock; + } ret = vmbus_send_modifychannel(channel, hv_cpu_number_to_vp_number(target_cpu)); if (ret) - return ret; + goto unlock; /* * Warning. At this point, there is *no* guarantee that the host will @@ -1408,6 +1425,7 @@ int vmbus_irq_set_affinity(struct irq_data *data, */ channel->target_cpu = target_cpu; + spin_unlock(&vmbus_connection.set_affinity_lock); /* See init_vp_index(). */ if (hv_is_perf_channel(channel)) @@ -1422,6 +1440,10 @@ int vmbus_irq_set_affinity(struct irq_data *data, update_effective: irq_data_update_effective_affinity(data, cpumask_of(target_cpu)); return IRQ_SET_MASK_OK; + +unlock: + spin_unlock(&vmbus_connection.set_affinity_lock); + return ret; } /*