From patchwork Fri Dec 1 10:48:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13475650 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="CrMptfkL" Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2069.outbound.protection.outlook.com [40.107.243.69]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 974EC10DF; Fri, 1 Dec 2023 02:49:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=L5adrPuLaMD2fPz/ToNnO23Hpg1439ZWtLXFaEQy1pwsDRQtq8gDt5CWNdBgZJWNQyOc4GRH6wRW3gmShE2OUBYBcRoGl7hml6KKlecxtdT4yH9Ujz98a7Jdr3TLf72vN6Rt01Rk3DTq6vu30dlSGWrqteA6+jZxCdSnWDZOlvgSiIMmdBUEeJTg3anYeEkOrm3YseuOhRFrr2YYlX+bAwJDiJShphi0YxK73K+PthK18QxKwlVTOlH9okMmEpHvEyvJShGoWpmVIiv68iq/Q9piltiXFRBYAuIfBrXygBHEWma37kfRCU6fv+SGXLlUvfXj+3FcI8CzWZWPw3s+6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PLndcNIvbHaL6btQSx+smp5PpDwR7TaSywJapj3aEO8=; b=SuEeg15CurT8PI361zSeSiVWVrMbP9h40Tx/LEriwxGzG18uyRL0OaVcuIIDiDyVP3niusiuleSjslqC0E9C2aFytc4zQQsl4d6Xaqi7haxij8IZWBwEMGPm8E4vSwU60StbKeykUWFiC2ZZ+lhE14uMU2YZ6JoR0FzgBcZ1WVGgbiOHJ4BjAWGztBzvY/Fi3sK2dhDJEeO2fOro+Jvd2Xze8VwjnaZrfLif/LW0JtXmP7dLX94M5bUIP+gyFeHDbX79JbAAEttSkpGV6WZRVRSOcoezkldINzwZZ4AnXOYo0w+aQFKSqr/6JyMrR8Norz4EO1WVdOalKtB3gdIl/A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PLndcNIvbHaL6btQSx+smp5PpDwR7TaSywJapj3aEO8=; b=CrMptfkLOfF0i46iHFH1yBzyAXDe5sKzq5aU7CxkbQdLlySllZ+JInZxyL972URmiaWdd+APo7jVP8ZVVdCFV84obkbWSAa3Ssq+MrEVmZtnOTuuK5Ai7I96avI/LruO/df128/bXDD+csvqE2nu4MUgTnRCqTqP6I892q7/KN3qVNeiS4nmKNBBKILC8vwBkLSN36RKXSWuRqmqstnePgarjZ/M955wck+PKxE1JaV/eAwRQH1etTM/zQNqRJpeBM/v4RAFwiifs3A0AVsNwr1rZ1UwHOfYcIOh/p7QPQKWI14OttRnzc+yQWTUdIuFItYRxj5mcl0Bbb32qV3Qmg== Received: from MN2PR20CA0063.namprd20.prod.outlook.com (2603:10b6:208:235::32) by SJ2PR12MB8831.namprd12.prod.outlook.com (2603:10b6:a03:4d0::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.23; Fri, 1 Dec 2023 10:49:40 +0000 Received: from MN1PEPF0000ECDA.namprd02.prod.outlook.com (2603:10b6:208:235::4) by MN2PR20CA0063.outlook.office365.com (2603:10b6:208:235::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.28 via Frontend Transport; Fri, 1 Dec 2023 10:49:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECDA.mail.protection.outlook.com (10.167.242.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.17 via Frontend Transport; Fri, 1 Dec 2023 10:49:39 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:22 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:22 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Fri, 1 Dec 2023 02:49:18 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Gal Pressman , Parav Pandit , Xuan Zhuo Subject: [PATCH mlx5-vhost 1/7] vdpa/mlx5: Expose resumable vq capability Date: Fri, 1 Dec 2023 12:48:51 +0200 Message-ID: <20231201104857.665737-2-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231201104857.665737-1-dtatulea@nvidia.com> References: <20231201104857.665737-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECDA:EE_|SJ2PR12MB8831:EE_ X-MS-Office365-Filtering-Correlation-Id: 5d504498-a8bc-48a5-7653-08dbf25b37fd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: O7Z1MNOFn+dnL8jzlmONFP++umlm3SVksvhoychYLULUWaDc6HOCaXevpVjgJ7OvPLChbtyHGBlAjWOjJKuP41lm8UuIJmFS3WS3tyNg5k9BqmAzZF5cwxDjLiGWat33vaLalzrsjWaN+CDAhKoVyKAFVfhJbsTvfmz6GCriJLCyDB2rkpF4STq9vEVTiiJ2DBbuoQXnaZQYDpRfwDV5+vE/qHyEH5C7a+9+2hYVf8B2eUBc1t3tV0phMseFvNSouYiTxA6J6fTMYRlJlOM+cOxUo7cxNx+m7eCJQvZxUpHqOul4l9XBExv09D3rlwa4UWisVEPfZtcjne9dMijCeJEhlCLhLko5kO0nrkCb13r+9HpTRXFum9DAgN1P7LxrFoXksIOe8FkYAiRaeC/YlwAbtpfyEYG5Ti15IuwO0KqmLqJVi4mAs/kTMJtsbcoyb95iKMhshnjinuAleI91tkYLD9j4WVv6khcB7e7N/2InWz3UGLVzt49026G6Cs7T39cp+VwId3uxSw6DpOsR08fIgmGb9XLE34WtPd9T/XEWYIZZKIG2TVqBAM5zyp3BmB/+heb79ck6o2Duz8uTRMuTroF7yDutRpZ/jqoRrexdG03QqHQbZTuwMK40UryVcHO7KNIA64jvAY8ks15FIBYxtv5z6MtdpgatTW6riS8A5BERIEw4L6GoQctlo8LlenvDuTWDb8OPz4RkbBmUrA5YK+7wpiqoebAKGz+7CNIwgipRq+/ca6Yh7miMTfvm X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(346002)(376002)(396003)(136003)(230922051799003)(186009)(451199024)(82310400011)(64100799003)(1800799012)(46966006)(40470700004)(36840700001)(40460700003)(356005)(7636003)(82740400003)(86362001)(36756003)(478600001)(2906002)(6666004)(4744005)(4326008)(5660300002)(336012)(41300700001)(426003)(2616005)(40480700001)(26005)(1076003)(316002)(8936002)(8676002)(110136005)(70586007)(54906003)(47076005)(36860700001)(70206006)(83380400001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2023 10:49:39.6329 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5d504498-a8bc-48a5-7653-08dbf25b37fd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECDA.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8831 Necessary for checking if resumable vqs are supported by the hardware. Actual support will be added in a downstream patch. Signed-off-by: Dragos Tatulea --- include/linux/mlx5/mlx5_ifc.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 3388007c645f..21dcfa034b7b 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -1235,7 +1235,8 @@ struct mlx5_ifc_virtio_emulation_cap_bits { u8 reserved_at_c0[0x13]; u8 desc_group_mkey_supported[0x1]; - u8 reserved_at_d4[0xc]; + u8 freeze_to_rdy_supported[0x1]; + u8 reserved_at_d5[0xb]; u8 reserved_at_e0[0x20]; From patchwork Fri Dec 1 10:48:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13475651 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Qjwzzcxd" Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2053.outbound.protection.outlook.com [40.107.94.53]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4E7B10F1; Fri, 1 Dec 2023 02:49:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M1cjpykQXSsdTKf/PZUA6ZuRNGqVOyN6XIsXc8SRktZPPf3y4NpO6EK6LvDLWH5lqTqwJKmhjTmzORP0YKk5WQHy9byeaAeUMC5hNaPQrabEZzhk9PsrcR/sY3l+VZUGLnTXhy9q75bBG5V3DiUzxUg+ArZApTJfkJlbnYErIQ2sFPO1/3CwutmGafZIIWRSsOPQXZzOc1KzUf/TtxmAYy6l6Hd9gzalSsgWDiPRqfCcfM2x91GD71ncZD99WgRAAiZu5to0yaKQku5VkuKP2kW5A1u4vclLlfZvWt7kF0h4Eas5gwbuSSCVTeOQk2PyZeSpI+Qg49IfO2f4lgx5xg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=b6hDxocxpj5OzTimObjmwJdf213sHeAV07ZmX51gJo0=; b=Pp28i5A3wWb+90VhQZV4AWWVwtTJ9iyW0ESY4aF3b8FG9TPlHwZhs9Y1SIfc1t3o23yyxIkbHhMxhxNc7XT2bxmUsdtv/w7AvlyrPCQIv6OWkAqN8LYC2PJC3+tq9etBRg25u6QGpHJQZGgZ5/oh+18tUtfxXHtQ0a4MUM4K322e9VwQGw7b/BktjvzaMJNbilC06Q1oe5QrV3GC92vg6cSDdGZGkcoQewE3RpZmLforQhKkYylPf2EspWoHCUPOz0k5a5ycG/iuMTqFvXDXwyIKn3U4rGZw4Xv7ETrhJnbtV9jXf6z4UrwHX0pzqWgJEBDYSjmWv/kK4VkpQJTz3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=b6hDxocxpj5OzTimObjmwJdf213sHeAV07ZmX51gJo0=; b=QjwzzcxdEbxoDa1DWwjep5QTohhbK3bS0rPeWcKa1HABcGIytiK/C+CjJ9Hf4vxlxAhVaiz5lM6aDQH/kT1dxDaxj+Uv31HliDjahUKEKPGwhA1/eqipqqMA68Llys3Ple9fxeb31ng5O/NqRh38pjsqZcZ+qNWB2NcNxU8VCC8PM46RVurS92L7AqrOSktnD/5fYSztdOPqbuH51k5yUeSdyyGdqCSyJNJ/v8LmEZ394XpFb0lsiBKgKzEkrzed7+wRqyC+JQkOvaxbJ3zXSs1BOnndwjgftIpSXdkaZrwrvNCOG1DSmyLRPl+mQ7xZsU1XhnO+j1WI15DuDGMyKQ== Received: from BL0PR02CA0079.namprd02.prod.outlook.com (2603:10b6:208:51::20) by PH0PR12MB8052.namprd12.prod.outlook.com (2603:10b6:510:28b::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.24; Fri, 1 Dec 2023 10:49:43 +0000 Received: from BL02EPF0001A0FA.namprd03.prod.outlook.com (2603:10b6:208:51:cafe::a7) by BL0PR02CA0079.outlook.office365.com (2603:10b6:208:51::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.23 via Frontend Transport; Fri, 1 Dec 2023 10:49:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL02EPF0001A0FA.mail.protection.outlook.com (10.167.242.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.17 via Frontend Transport; Fri, 1 Dec 2023 10:49:42 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:26 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:26 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Fri, 1 Dec 2023 02:49:23 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Gal Pressman , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost 2/7] vdpa/mlx5: Split function into locked and unlocked variants Date: Fri, 1 Dec 2023 12:48:52 +0200 Message-ID: <20231201104857.665737-3-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231201104857.665737-1-dtatulea@nvidia.com> References: <20231201104857.665737-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FA:EE_|PH0PR12MB8052:EE_ X-MS-Office365-Filtering-Correlation-Id: 10bf0976-c178-43af-b9f4-08dbf25b39eb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OVHDmkUXYoslfIzi8AZ0BIwVHWpuw9cJXTB44HU739z5aA92R7LNE0gGSrdLXEpS0Ts3SEUoWAwCjIfpJdzjDfPhHXFP6XLQHVfuIJ3eVXYgMged7uFLgVzL7xAuWxjnFl6avcyfxTM2sc39oLR9y5nKSzJybv0EFfHwmZ7mkmztZMcaNf+myo6Ei4z8MDNjQYGvtwwAuPxWgdMC6MVQK6sItfl9ZPlaWuJDbWoHaXP9eVdn99V7hCN0QqOF7OT7YlYnlRJnEo9rs6sWtOT8RKgXy60JpKXwVu5KUBOTccIfqGTXbeukQwBXJSslnTroXt2q2RrF0CTSrgoYkoz7yTMjnpBXNw9NDaCzcbN5Oau0VH8p80HI1tYVnv9bLzSb7P0CKm1zqwr7oB1bKnWqFfbauE2H0rbFfIJmXyhor8edIyoq9Ld9S7Qkj0A7Lk7AXeLvNkVbgHIY0BKAHwKiB/UaRP+CmSFXwbbqZqi36JSMPkcT/0LvSLo6WuUhgPBgdkRszHZtp36HhrQRxhN99CWo8QrAkm9hmNCTFRFudTVxtwI+L411YiZzGFAyBrlEr++fEsWCx0iooE3bHy+8okuvSxzzn7ZAC/MsEMolqtM/oClHj4oyfLPblpDEA8Pd+dGXLlgKOoeH3CME+glMZyd7PR8kDk9x07+mwI+votry0C6MvmZlebCDPTc7jcPOnZCb4FRdkIliZkihCHObgV1Y4aR/rymw4Gg/YZXwCvfmFgirsh+RGD/zzt7BnxFm X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(346002)(39860400002)(136003)(230922051799003)(186009)(64100799003)(82310400011)(451199024)(1800799012)(40470700004)(36840700001)(46966006)(40480700001)(86362001)(40460700003)(70586007)(70206006)(7636003)(356005)(82740400003)(36756003)(83380400001)(54906003)(47076005)(426003)(36860700001)(1076003)(26005)(2616005)(6666004)(110136005)(2906002)(316002)(4326008)(8676002)(5660300002)(336012)(8936002)(478600001)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2023 10:49:42.8877 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 10bf0976-c178-43af-b9f4-08dbf25b39eb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FA.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8052 mlx5_vdpa_destroy_mr contains more logic than _mlx5_vdpa_destroy_mr. There is no reason for this to be the case. All the logic can go into the unlocked variant. Using the unlocked version is needed in a follow-up patch. And it also makes it more consistent with mlx5_vdpa_create_mr. Signed-off-by: Dragos Tatulea Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/core/mr.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 2197c46e563a..8c80d9e77935 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -498,32 +498,32 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { + if (!mr) + return; + if (mr->user_mr) destroy_user_mr(mvdev, mr); else destroy_dma_mr(mvdev, mr); + for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) { + if (mvdev->mr[i] == mr) + mvdev->mr[i] = NULL; + } + vhost_iotlb_free(mr->iotlb); + + kfree(mr); } void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - if (!mr) - return; - mutex_lock(&mvdev->mr_mtx); _mlx5_vdpa_destroy_mr(mvdev, mr); - for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) { - if (mvdev->mr[i] == mr) - mvdev->mr[i] = NULL; - } - mutex_unlock(&mvdev->mr_mtx); - - kfree(mr); } void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, @@ -535,10 +535,7 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, mutex_lock(&mvdev->mr_mtx); mvdev->mr[asid] = new_mr; - if (old_mr) { - _mlx5_vdpa_destroy_mr(mvdev, old_mr); - kfree(old_mr); - } + _mlx5_vdpa_destroy_mr(mvdev, old_mr); mutex_unlock(&mvdev->mr_mtx); @@ -546,8 +543,12 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { + mutex_lock(&mvdev->mr_mtx); + for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) - mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[i]); + _mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[i]); + + mutex_unlock(&mvdev->mr_mtx); prune_iotlb(mvdev->cvq.iotlb); } From patchwork Fri Dec 1 10:48:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13475652 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="G1dsSuB8" Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2089.outbound.protection.outlook.com [40.107.95.89]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 793D2170E; Fri, 1 Dec 2023 02:49:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=D4v5PMou7X0UrRJFP3JtnRIa4YEALvESf0HNWNWkOD7VGNEle8gjf7s3tw7L9ndN/l5LM/ahG3ffPOTLjMEjt5vxgtkuy2WrDRVE9NtwjDj8X8iaVRBMW9M2IZq9xPaH6xxTLJh63HChabvvYmtvIKhshtfXKpb+cuidwiiojidz2nI2Mx70XqaJbt6ldDjHpetZ2Col41a4+xN91Q44+wJhVD7rzwVDGbR+in+NOoHNosrT0DtcEYRDG5bxyoHOE7auRDFBAdyrvcWA0b5WqhQklVyaGLpIgMV1mRRBKW+UoFAjkaG0MApzOo2tGYPYlCriXhNySiTN7OfTj9cHQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AbkkSRuhAnoXxmmmaWrUCRkEl/cD3hqskadOAIqvb/g=; b=SQLmG5IkCBIiUo5TNv6evMZhohvOPNdqI7LHEcsB1X9jZ/1OuYD3j4QztJDUX0R9EOmraqf5ReFaitq2qFqdENjpGsk4x8RKihGW4i2QY5IvP2OuSIzbVNEs95G8BOYm9yOzT7FV/mdumIBOT5UTuy+eRh6CyiEKgYkeB5TXqUWxSaKQmQbxTlX/0mQyBmpYzXuH3eK0T3fAh2t9ZPSp0pnKnATANw0rkx/WWr6E1yCx/nQYeX/FvnqBLftl//r6ZTyZQ7v1ZbsvZGXBt4dh2oAX3gkDdHh1YtzI8AwaEG4pleqB/Hi1U3ng5jf+fVQRvjKmusUrJgxjAzUCFnFzPg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AbkkSRuhAnoXxmmmaWrUCRkEl/cD3hqskadOAIqvb/g=; b=G1dsSuB8rtBLNM3Aa4T1+ml7mYl+Rw+J5chgMtFHjg/Kk0sYbSNaj4LSX5LJ8JGec5mjnpdVJ1Pp/ZvnqSFa0JEUtwvK2fZZ+S7TTBw9wG/eprZ6tOqD9IW2avgefe+Y4hNeW/v5bCpp/3Ebqkv22eFhDe4ka08xh1MLHxRJzj9IrOG2YddracIFYrFRk5BMGNt10TnYQZvZEqd6tgNA/nx9E5PwmF7QPRoAxIRPOhZyRba7N0VaAD1XW4se7eMtBMr9R2r/goIDfcHMKebkG3q+cBhcvSUrEnaZ/Ul+IWz6DjAfYQVA9oO5ZPq46H9YcgyXRtwPJy/eJArKUB2Igg== Received: from BL0PR02CA0103.namprd02.prod.outlook.com (2603:10b6:208:51::44) by DM4PR12MB5723.namprd12.prod.outlook.com (2603:10b6:8:5e::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.27; Fri, 1 Dec 2023 10:49:47 +0000 Received: from BL02EPF0001A0FA.namprd03.prod.outlook.com (2603:10b6:208:51:cafe::19) by BL0PR02CA0103.outlook.office365.com (2603:10b6:208:51::44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.22 via Frontend Transport; Fri, 1 Dec 2023 10:49:47 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL02EPF0001A0FA.mail.protection.outlook.com (10.167.242.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.17 via Frontend Transport; Fri, 1 Dec 2023 10:49:47 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:31 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:30 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Fri, 1 Dec 2023 02:49:27 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Gal Pressman , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost 3/7] vdpa/mlx5: Allow modifying multiple vq fields in one modify command Date: Fri, 1 Dec 2023 12:48:53 +0200 Message-ID: <20231201104857.665737-4-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231201104857.665737-1-dtatulea@nvidia.com> References: <20231201104857.665737-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FA:EE_|DM4PR12MB5723:EE_ X-MS-Office365-Filtering-Correlation-Id: 6179766c-a5ec-41c3-c62f-08dbf25b3c9c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5kJWC9zPolktG+YSeSwWTnGFVcE+I04TD1wn941VTluQLB9kPtVGetdsAUq2uTLOHmoK3cYlu4WH9I2hr2uAkb2/Pyin0rPCjLWDUtqhMBP9Ip/QzfRqmXzlkJkjZSb8r6cFvMYOpl7nHstwWpwvMH5LfQSoCOlzJEn/PlVYLbBWdwc/GW3x6ixBpOolZPrXkjVe0Zrh21F9VYJK95o6SM/h2IqLZZdIDgThSdRRgkThwgLuTS/xroZqysvFwPtopXOyZAJ8QiD6XBOPJ+Pad/ELF/t8gRUHw8j20G8PrvQmAnvAjGW2md72T8AYLa7VCekir+M1Z8lsy9A+hLr2GwXsUjW7cNP3Z3AHjfis/CEFUqPT8MLdDfXYRTRH1+W12sN7gAE5HAQ1u3u4WeBF+AoflZ5rGQ9pZ36r4BN1wprcDdvWAozLlQoPAe/JJhXv3YAtHgD/yER3XWkQubMwWkxca3895jpWmBI+HYdfNrPHu/4dzISJ7v1I6dT4L1S58z9AHAikWvwouj0FM6KjQRTdD/3VLI5Hh3FeUvCrduXfbwRkpdDXT7aBR5mTK0Sincz8dsEVcXrWft4Nqeaf9qDf26cO0/EWjZ697BMFs7JVeps6ZrKrHFcd0zMWNk+DmmxR7bKv94IScFZmFVIxrEEBhQzSqA8A3zScCGvn9b9sHYLKXdFFQsgGug/ihg4HRyjagbe6l/7pxvwEekds/WM/PnzlNK89qbwGo68vqk4Jb5dJbu93FIKSNE/jw8I+ X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(346002)(39860400002)(136003)(230922051799003)(186009)(82310400011)(64100799003)(451199024)(1800799012)(40470700004)(36840700001)(46966006)(316002)(66899024)(54906003)(70206006)(110136005)(70586007)(478600001)(40460700003)(6666004)(5660300002)(41300700001)(36756003)(2906002)(8676002)(8936002)(86362001)(336012)(426003)(2616005)(26005)(4326008)(1076003)(40480700001)(356005)(47076005)(7636003)(36860700001)(82740400003)(83380400001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2023 10:49:47.4034 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6179766c-a5ec-41c3-c62f-08dbf25b3c9c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FA.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5723 Add a bitmask variable that tracks hw vq field changes that are supposed to be modified on next hw vq change command. This will be useful to set multiple vq fields when resuming the vq. The state needs to remain as a parameter as it doesn't make sense to make it part of the vq struct: fw_state is updated only after a successful command. Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 48 +++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 12ac3397f39b..d06285e46fe2 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -120,6 +120,9 @@ struct mlx5_vdpa_virtqueue { u16 avail_idx; u16 used_idx; int fw_state; + + u64 modified_fields; + struct msi_map map; /* keep last in the struct */ @@ -1181,7 +1184,19 @@ static bool is_valid_state_change(int oldstate, int newstate) } } -static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq, int state) +static bool modifiable_virtqueue_fields(struct mlx5_vdpa_virtqueue *mvq) +{ + /* Only state is always modifiable */ + if (mvq->modified_fields & ~MLX5_VIRTQ_MODIFY_MASK_STATE) + return mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_STATE_INIT || + mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND; + + return true; +} + +static int modify_virtqueue(struct mlx5_vdpa_net *ndev, + struct mlx5_vdpa_virtqueue *mvq, + int state) { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; @@ -1193,6 +1208,9 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque if (mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_NONE) return 0; + if (!modifiable_virtqueue_fields(mvq)) + return -EINVAL; + if (!is_valid_state_change(mvq->fw_state, state)) return -EINVAL; @@ -1208,17 +1226,28 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); - MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, - MLX5_VIRTQ_MODIFY_MASK_STATE); - MLX5_SET(virtio_net_q_object, obj_context, state, state); + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) + MLX5_SET(virtio_net_q_object, obj_context, state, state); + + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); kfree(in); if (!err) mvq->fw_state = state; + mvq->modified_fields = 0; + return err; } +static int modify_virtqueue_state(struct mlx5_vdpa_net *ndev, + struct mlx5_vdpa_virtqueue *mvq, + unsigned int state) +{ + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_STATE; + return modify_virtqueue(ndev, mvq, state); +} + static int counter_set_alloc(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) { u32 in[MLX5_ST_SZ_DW(create_virtio_q_counters_in)] = {}; @@ -1347,7 +1376,7 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) goto err_vq; if (mvq->ready) { - err = modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); + err = modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); if (err) { mlx5_vdpa_warn(&ndev->mvdev, "failed to modify to ready vq idx %d(%d)\n", idx, err); @@ -1382,7 +1411,7 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *m if (mvq->fw_state != MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY) return; - if (modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND)) + if (modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND)) mlx5_vdpa_warn(&ndev->mvdev, "modify to suspend failed\n"); if (query_virtqueue(ndev, mvq, &attr)) { @@ -1407,6 +1436,7 @@ static void teardown_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue * return; suspend_vq(ndev, mvq); + mvq->modified_fields = 0; destroy_virtqueue(ndev, mvq); dealloc_vector(ndev, mvq); counter_set_dealloc(ndev, mvq); @@ -2207,7 +2237,7 @@ static void mlx5_vdpa_set_vq_ready(struct vdpa_device *vdev, u16 idx, bool ready if (!ready) { suspend_vq(ndev, mvq); } else { - err = modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); + err = modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); if (err) { mlx5_vdpa_warn(mvdev, "modify VQ %d to ready failed (%d)\n", idx, err); ready = false; @@ -2804,8 +2834,10 @@ static void clear_vqs_ready(struct mlx5_vdpa_net *ndev) { int i; - for (i = 0; i < ndev->mvdev.max_vqs; i++) + for (i = 0; i < ndev->mvdev.max_vqs; i++) { ndev->vqs[i].ready = false; + ndev->vqs[i].modified_fields = 0; + } ndev->mvdev.cvq.ready = false; } From patchwork Fri Dec 1 10:48:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13475653 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="OysHD+s5" Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2040.outbound.protection.outlook.com [40.107.237.40]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 464E41704; Fri, 1 Dec 2023 02:49:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Co24YRV2t+8sYdjvGE9HfiCMdhV8uHCUbLqu/wwXMDZ4Urmdv07Ykv9axzuFnAWgy3OTLYL0hGAVwKd4KcgWY9Pfc3qNzgNvV1bPMQGwssjaePV2Er7irgu3pHxRHL+I4FoCEPyXh01tprQFv9wnwgX2cRbUl0O6+gnM1As2a+yQgWVdjilXGvNTCzSO2//WHQG9LFs2UyPDZcRWyj0KgjB43A87nERuhc/AuEkg0nuhmJH44F4FjdpoIWNU5usc/+sRVZZQ0cybK91IIY6Bgiz1zpk6TDeEmF6LIoBU/w37dyujaC3pN27EG9fkPUXPBUA3r0LSEl9F8mn1TZYJ8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ept1Tf2SnzBMD4BLVt8Odf9sFEtXbZW1+WTxNC48gAI=; b=lx8wYttaPgSp9wKXya8KGxCVOhxk9gnTu3m29+/DsrKGAymMq1XvgdMaQzgjUxFfaYKDTMbOgjKxeTSMIWso7gDrniPvkN9Y3c8dYWxveRPl2f9qASVu9QJI91P6XFB7GzeRk6+TM2VoQOFN3cI5yOMBLxgxgedfxurAiSzMOASLQ5yulP9Xlear7bB4dVOjFJsJwbdcCJSQMqU0I1jAntx7eEs92hjtxHNqVS9mFJ+/nrG6qA22hcO21ngDmO2tosmtrlzMuns86J3KJoE19lFG55i5LFsaZdiEp1/EZ1KaGIqbC9LH2KEajA5sZhBgravdafskedJ9R5FCVbz6oA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ept1Tf2SnzBMD4BLVt8Odf9sFEtXbZW1+WTxNC48gAI=; b=OysHD+s5UJknLkE/Oi7VzC9Mgp2BW56FtV9XSpe35wkyxKYe7CYHJR4iJwgZ4CfW07++mPugZae8nl7mbWLtUU1blWTneX/d5LWqjBIMk6FQ9sQkKrecYTXTvI9fQ6zCUos74QfpTpU4QLu0Trkx1uukFErZ6X2PDdIExXycw3bxRNQ6uf5fLIKp+jiYPrWQ+kM4PG9LYVVWkQ/DvrzTiroNOjz4T6i+yVmQ6gE4dwBn+PAmDmiqic1ezczRxh7zkLXBxgTSlzEQBJX3n3KE7SQNrZUPfP4aWP/OseGABvdcCtBioHOysS9qfz/YXjaIR2wld9F2/t/UnnzvUHijOA== Received: from MN2PR20CA0060.namprd20.prod.outlook.com (2603:10b6:208:235::29) by IA1PR12MB9031.namprd12.prod.outlook.com (2603:10b6:208:3f9::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.24; Fri, 1 Dec 2023 10:49:52 +0000 Received: from MN1PEPF0000ECDA.namprd02.prod.outlook.com (2603:10b6:208:235:cafe::42) by MN2PR20CA0060.outlook.office365.com (2603:10b6:208:235::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.29 via Frontend Transport; Fri, 1 Dec 2023 10:49:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECDA.mail.protection.outlook.com (10.167.242.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.17 via Frontend Transport; Fri, 1 Dec 2023 10:49:52 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:34 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:34 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Fri, 1 Dec 2023 02:49:31 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Gal Pressman , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost 4/7] vdpa/mlx5: Introduce per vq and device resume Date: Fri, 1 Dec 2023 12:48:54 +0200 Message-ID: <20231201104857.665737-5-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231201104857.665737-1-dtatulea@nvidia.com> References: <20231201104857.665737-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECDA:EE_|IA1PR12MB9031:EE_ X-MS-Office365-Filtering-Correlation-Id: 5d62fbca-2648-4b26-2e1f-08dbf25b3f92 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5v0hwFR4APWRLGbjDBeNs3gzx7ZIUg6F21gHalWJRNgFd5xzG3a5M21Tv9SXtLI+zI93liIwqAng1DX82Sz1Y/iksr5Xpg2ic7fqdzl5S36GnyIfvgyiuCQnR1XeEDOV2SUi6QY+zk8/j1U1PDBVO7fL4/9WZWum+a8HHerVqZEqXE1plBumtFUfRppVJ/0qQgReFMOULssEQzTc00vM/hisyf/9o68r5TS58mu4D4Q9bFWEgcX/OF4D0O/Q5/cJcnsNsCLZavp0dROlhLLA2uaqHi5t48lTnwfCnY3bK4Ivc6FS3xzFQCf9B9Hz+0RaYuErfhPRmkwDl5rw/DL+1Mh3cnNNTYM3wnAptXLDV9X5tFXjRkFWrE5D9ypjcx5IjizSjiqSNc6AfDq6i5oR3Vuqn1P+uqXbK/anDtqll8bnHDIq2XQM/+7vdFjQ8MZlA9kcIjxfEynonZTCj6xDsU+j1MN9i2wJ9dD+/aPMZnVJiGl5dotuJs+ZoyA6z5HiOGNkhSzRMKCuVthu//AQiFbmm7zgOpay0HfetG9qwWVn92tG130eWrkOr+5BurfPO4KLu1+7RzJEqKhPQwnKs4/8BpbnfmQQQHYZ/4Ypt+2f5O/i8w1H9v9BC5OAPYLIdH2oND7v+uTarnxvbnq8V4DDIRtodSutP6fL3SQx+xcDb5dMo1WSnn1o5Lwyv5qIpapb5wfh3OEb2CcQ0KZDhQ7/QqrMQ5jFQG424JQb7Rj7pmzJHPwg5e2F3/1dsB/D X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(346002)(39860400002)(136003)(230922051799003)(186009)(64100799003)(82310400011)(451199024)(1800799012)(40470700004)(36840700001)(46966006)(40480700001)(86362001)(40460700003)(70586007)(70206006)(7636003)(356005)(82740400003)(36756003)(83380400001)(54906003)(47076005)(426003)(36860700001)(1076003)(26005)(2616005)(6666004)(110136005)(2906002)(316002)(4326008)(8676002)(5660300002)(336012)(8936002)(478600001)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2023 10:49:52.3672 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5d62fbca-2648-4b26-2e1f-08dbf25b3f92 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECDA.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB9031 Implement vdpa vq and device resume if capability detected. Add support for suspend -> ready state change. Signed-off-by: Dragos Tatulea Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 67 +++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index d06285e46fe2..68e534cb57e2 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1170,7 +1170,12 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueu return err; } -static bool is_valid_state_change(int oldstate, int newstate) +static bool is_resumable(struct mlx5_vdpa_net *ndev) +{ + return ndev->mvdev.vdev.config->resume; +} + +static bool is_valid_state_change(int oldstate, int newstate, bool resumable) { switch (oldstate) { case MLX5_VIRTIO_NET_Q_OBJECT_STATE_INIT: @@ -1178,6 +1183,7 @@ static bool is_valid_state_change(int oldstate, int newstate) case MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY: return newstate == MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND; case MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND: + return resumable ? newstate == MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY : false; case MLX5_VIRTIO_NET_Q_OBJECT_STATE_ERR: default: return false; @@ -1200,6 +1206,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; + bool state_change = false; void *obj_context; void *cmd_hdr; void *in; @@ -1211,9 +1218,6 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, if (!modifiable_virtqueue_fields(mvq)) return -EINVAL; - if (!is_valid_state_change(mvq->fw_state, state)) - return -EINVAL; - in = kzalloc(inlen, GFP_KERNEL); if (!in) return -ENOMEM; @@ -1226,17 +1230,29 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); - if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) { + if (!is_valid_state_change(mvq->fw_state, state, is_resumable(ndev))) { + err = -EINVAL; + goto done; + } + MLX5_SET(virtio_net_q_object, obj_context, state, state); + state_change = true; + } MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); - kfree(in); - if (!err) + if (err) + goto done; + + if (state_change) mvq->fw_state = state; mvq->modified_fields = 0; +done: + kfree(in); return err; } @@ -1430,6 +1446,24 @@ static void suspend_vqs(struct mlx5_vdpa_net *ndev) suspend_vq(ndev, &ndev->vqs[i]); } +static void resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) +{ + if (!mvq->initialized || !is_resumable(ndev)) + return; + + if (mvq->fw_state != MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND) + return; + + if (modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY)) + mlx5_vdpa_warn(&ndev->mvdev, "modify to resume failed\n"); +} + +static void resume_vqs(struct mlx5_vdpa_net *ndev) +{ + for (int i = 0; i < ndev->mvdev.max_vqs; i++) + resume_vq(ndev, &ndev->vqs[i]); +} + static void teardown_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) { if (!mvq->initialized) @@ -3256,6 +3290,21 @@ static int mlx5_vdpa_suspend(struct vdpa_device *vdev) return 0; } +static int mlx5_vdpa_resume(struct vdpa_device *vdev) +{ + struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); + struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); + + mlx5_vdpa_info(mvdev, "resuming device\n"); + + down_write(&ndev->reslock); + mvdev->suspended = false; + resume_vqs(ndev); + register_link_notifier(ndev); + up_write(&ndev->reslock); + return 0; +} + static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, unsigned int asid) { @@ -3312,6 +3361,7 @@ static const struct vdpa_config_ops mlx5_vdpa_ops = { .get_vq_dma_dev = mlx5_get_vq_dma_dev, .free = mlx5_vdpa_free, .suspend = mlx5_vdpa_suspend, + .resume = mlx5_vdpa_resume, /* Op disabled if not supported. */ }; static int query_mtu(struct mlx5_core_dev *mdev, u16 *mtu) @@ -3683,6 +3733,9 @@ static int mlx5v_probe(struct auxiliary_device *adev, if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, desc_group_mkey_supported)) mgtdev->vdpa_ops.get_vq_desc_group = NULL; + if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, freeze_to_rdy_supported)) + mgtdev->vdpa_ops.resume = NULL; + err = vdpa_mgmtdev_register(&mgtdev->mgtdev); if (err) goto reg_err; From patchwork Fri Dec 1 10:48:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13475654 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="hEiawE8z" Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2056.outbound.protection.outlook.com [40.107.102.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 985EA1992; Fri, 1 Dec 2023 02:49:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DCSjNbBdIWSL63lPSQGllvT9KP4SDnkUVlEpMcHyYtBt1ufKgUox+e2XzKHh85yc8DzSHTuJymQzSwGkhkfLMyRNf3SB+HBpUO+EIhufZSzKXz6wcrR6FUw/a5N4x/zgyh1D8UT5b9GQz71XPDrrjhzMg5VeQOCTOq5I468CkMJWhqvKu5nKYitP108ZYq63HN7kxCYv4qFHiu4KHgJ/miJ0pyW1RTN31otO/BYFK2gR9kHxS7MEifg/yggvI0ndTuL11fDMnM9VNR15cbfLvItvC6UVOiBNc/JsbRik5eS0VZlLSFeAfH9/fKaUMWOM58Hw+BvuaoNd4zzwrmCZkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=51vWU5QOyBdAV9Vx5J79GCDO5Bn3tO/JvvgMCz/mhCc=; b=JbDbfd/ezK68w0YPJPwG0lbRPeHaEOJhBLIY74J+9cogE9t9rHkrfO24ep6Rt58Vb68G1BAfwNMTl5ABErMHVIHReR+I1U9sM0mjR3Unjwt7sbq096K0e2aHd1uG6j5GrL8PxJovdzmhc6S/mNtPo9ckak1QAwKpgksyu7T0eEx33ZAIhW8eZ16Ja+NelZbA0ldeSo/rmwYkmcpIdnoaNIoWh7nw2orEL+M9pClHSrS9aryvy8+vvM7Reh1iNGLdk4oE4uui6W/TkHkMkpkiBEKoc6orqO6wXE8CDBM2u3DNRojqXLTO5OCGvbaer647Y/tHq+1HJ7LBipXxrdD1cw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=51vWU5QOyBdAV9Vx5J79GCDO5Bn3tO/JvvgMCz/mhCc=; b=hEiawE8zzVq3/HYVYsLJX2IX0d1q0Xsj+5Q5L7nfpKgxv33MeYMNajF0LvbHkbIi/BIMoJ77BqBWR3err2ya1yPyGkYX7yqvo6AsXILAwvhyR1zG6EA3s21otFdd35zgWF0ugvuxgdse/x+V8SV5/me3fA6mjSurZvt7poyCG/1/EM0qemwL4UUjiSv8Q8NFF0j5zY1uBlfgemvzE2iujm2rcEEdRg9uFDXigYl3JXa7LkCDclPcH2vZGTC3xRabN1y3GyhKdTxC59GII9DRULdAANf62XDHfeqw7TF3/WrdKda5PPxlKMk/v48UhBiVkoDZntTFCbND0cGKfTaZvw== Received: from MN2PR20CA0051.namprd20.prod.outlook.com (2603:10b6:208:235::20) by IA1PR12MB6530.namprd12.prod.outlook.com (2603:10b6:208:3a5::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.24; Fri, 1 Dec 2023 10:49:55 +0000 Received: from BL02EPF0001A0FC.namprd03.prod.outlook.com (2603:10b6:208:235:cafe::5b) by MN2PR20CA0051.outlook.office365.com (2603:10b6:208:235::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.29 via Frontend Transport; Fri, 1 Dec 2023 10:49:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL02EPF0001A0FC.mail.protection.outlook.com (10.167.242.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.17 via Frontend Transport; Fri, 1 Dec 2023 10:49:54 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:38 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:38 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Fri, 1 Dec 2023 02:49:35 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Gal Pressman , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost 5/7] vdpa/mlx5: Mark vq addrs for modification in hw vq Date: Fri, 1 Dec 2023 12:48:55 +0200 Message-ID: <20231201104857.665737-6-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231201104857.665737-1-dtatulea@nvidia.com> References: <20231201104857.665737-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FC:EE_|IA1PR12MB6530:EE_ X-MS-Office365-Filtering-Correlation-Id: 59916c1e-55c0-48a5-a3e6-08dbf25b40fe X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZY0Mty1JnBPkuhhNRBmn4igHXCBvA1Q13wMSRhmZGpx1b1vZ2iYNgESd1tdBcRMX961UWO1C5+QncQQq+SowmI+Jl2z9Nwe3qG+89G1kb05gKarInk44GRaaYLxRRaY7jHXTCxThi28mXp0lZCuukXGBXzsEDVsOPdxvKCSiZHzbt6iEhLlpns92hfsZ09EW7lK8OA1CMptKvRZAyfO+fFTIvIs6foGSzzgJeyoBfeBUc52srzaNUopeO+GOpkuDc8b1fos09qmNFGeSj2ASLB6MD6g0+6n4gdro2IppYeP3BPa7/Rt2CnlVtbxC5973RAnipJHLVbI5sNv8I9ATHicP8Sq8J1HLGLEWY8ChkdKb+3HPAe3RBTQoFBmP6rLLq9IfnVmg7y/uonYRUhJb9gqkDaYStk8d5Fb51/95ow9Hxfl08nQiKUndAx9/0csYmzeQ4ZteahU3HoFV5o+l1BhtH1UCSJ9Ey6YqyXgZ0hO7DIEDHGrgDzNmSZIXr06ioi97TLSpv6G1Ymsf2lJRg7ovUz126scT8PwDDYP/oNeqIqMa2ImZQMyXBajsZI3bTwESHB7SjE+yZYaOU6zevbHK4fe3rYTpuXgnB3LchI1P69KnrN4GQk4xOFWs417rSVIwcZmrZHGqk9VVlot5XlNt6CNabz554+eBlLPN3EYgYAcw1TTg0ll0Nl/X+6+4wZLk8snQCBQq7h85MAkqsPl6a/lUrQPaVvHDFp0eHG/mUZQfa6uKgdj2/NUU6cV9 X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(39860400002)(346002)(376002)(396003)(230922051799003)(1800799012)(186009)(64100799003)(82310400011)(451199024)(46966006)(40470700004)(36840700001)(40460700003)(86362001)(26005)(1076003)(6666004)(2616005)(47076005)(36860700001)(426003)(336012)(5660300002)(8936002)(41300700001)(478600001)(8676002)(2906002)(110136005)(4326008)(316002)(70586007)(70206006)(54906003)(36756003)(356005)(7636003)(82740400003)(83380400001)(40480700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2023 10:49:54.7401 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 59916c1e-55c0-48a5-a3e6-08dbf25b40fe X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FC.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6530 Addresses get set by .set_vq_address. hw vq addresses will be updated on next modify_virtqueue. Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 9 +++++++++ include/linux/mlx5/mlx5_ifc_vdpa.h | 1 + 2 files changed, 10 insertions(+) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 68e534cb57e2..2277daf4814f 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1209,6 +1209,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, bool state_change = false; void *obj_context; void *cmd_hdr; + void *vq_ctx; void *in; int err; @@ -1230,6 +1231,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); + vq_ctx = MLX5_ADDR_OF(virtio_net_q_object, obj_context, virtio_q_context); if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) { if (!is_valid_state_change(mvq->fw_state, state, is_resumable(ndev))) { @@ -1241,6 +1243,12 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, state_change = true; } + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS) { + MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr); + MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); + MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); + } + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2202,6 +2210,7 @@ static int mlx5_vdpa_set_vq_address(struct vdpa_device *vdev, u16 idx, u64 desc_ mvq->desc_addr = desc_area; mvq->device_addr = device_area; mvq->driver_addr = driver_area; + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS; return 0; } diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index b86d51a855f6..9594ac405740 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -145,6 +145,7 @@ enum { MLX5_VIRTQ_MODIFY_MASK_STATE = (u64)1 << 0, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS = (u64)1 << 3, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; From patchwork Fri Dec 1 10:48:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13475655 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="hrU2cjWN" Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2056.outbound.protection.outlook.com [40.107.94.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B15FD19B9; Fri, 1 Dec 2023 02:50:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kVgVl/8bxgtTRh/JNNMtm860s3UpzYQncEPNEJ5yogX6z3hfK7D8oVrc89BYGb0KNMuQaPHvVEUT4KuWc9N5d0QWn3c5ubZY0qpbm8I6QKSdJbF5KiE6yheCBphQaRUj22dWXfI57klViMiGpdrpQiPQQ5vp3yqzUIWjNUvK0Isd/ykh9mPh1Ur0g/b6GqEfS0cdsrwJFA9cm1voOpvIytZ45V6jQXc7nF5Pby5Bn52gZ5JlBT0VqEN7gv8t0F2F3vJMGc3o44MLbRjh3/Pa+/970RpnbQEQqr/zfrkcp+2fcbrTQR0IjlpAAA4SWUWhRtD2XFtNf/ONByNteOWXYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=C9Jspx+SLwQ9ATSuVOd0UT8zDB4SlGNgjSwZ95Ao924=; b=LIQUsMWMdq7XRqdEcMJ4qac9psotUy6TT+BSGys4zCwLeXtJcQ9GyMbn7bjNLjCjHC6QlGChkLL+WyPlEV75ZvOIs3i6HQAGnMTv5Zg4HAC3Nza9Zv2d3dpKa/D4gklFdi+5Rns8kps9e3vu5EImWbMH+E5KUlMBOgZ62vX3mB2Ot3OG1k8XD5kcwO11XHF5g1wOWZFI6Ug94wjSbxORR4KHTiQtoyJaezWspU/SyJuuAChLY+lW9XvetYUXfxd/aWVelQGmCAasLd7msUp7L7UqDQP3yono5Vw717vf4Lqa8Gr5xfki2jCOmYqIIoq80S1U+vgzzYhpS9oMoviNaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=C9Jspx+SLwQ9ATSuVOd0UT8zDB4SlGNgjSwZ95Ao924=; b=hrU2cjWNT+MmvzkBYduT0vd4+p29uT49FvTfmpEx2db6avlh2epeNf4L+VlUcWrAbtsbPL9IqWKXHeEQf5ujc5ZqT4VT3onZvU1ml89O7a7zsGNk8UHqonF+aSRswNR3wV3Ud1c+xwLyqRU2xcnnqY1ky/KKosxVIHe84VadRivXIrnfIsMQkPlbmATIJL2MJD8HxGf/IgU1qQ76zxzGsgyDBNT2vHZUQFd8uOMr2filzIP3faMH5Scu+D5rL8/5+HD2xQoTTq+6jLTV0Nhq+MWdbIZPV1hoJIoMUnQIMQ7bexaqjZ2A85UCXzm0pa7P4tuBaTNFZppdU6XPO06+2A== Received: from BL0PR05CA0008.namprd05.prod.outlook.com (2603:10b6:208:91::18) by IA1PR12MB8405.namprd12.prod.outlook.com (2603:10b6:208:3d8::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.29; Fri, 1 Dec 2023 10:49:58 +0000 Received: from BL02EPF0001A0FC.namprd03.prod.outlook.com (2603:10b6:208:91:cafe::ad) by BL0PR05CA0008.outlook.office365.com (2603:10b6:208:91::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.17 via Frontend Transport; Fri, 1 Dec 2023 10:49:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL02EPF0001A0FC.mail.protection.outlook.com (10.167.242.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.17 via Frontend Transport; Fri, 1 Dec 2023 10:49:58 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:42 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:41 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Fri, 1 Dec 2023 02:49:38 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Gal Pressman , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost 6/7] vdpa/mlx5: Mark vq state for modification in hw vq Date: Fri, 1 Dec 2023 12:48:56 +0200 Message-ID: <20231201104857.665737-7-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231201104857.665737-1-dtatulea@nvidia.com> References: <20231201104857.665737-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FC:EE_|IA1PR12MB8405:EE_ X-MS-Office365-Filtering-Correlation-Id: 9b5aea7f-6dc4-4869-02e8-08dbf25b4323 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8/ZMmQWXbsByNyNUq/tb5s3+MlTT5VWcOvWiN7bbKrun9WZTH7jJUyZ02hxdCR7AIly8fETyWjuC7P1LvbMi1IEMGoj+ZdNouqkEHEN31wWB5XksN7nJUHCL43P1uy1h1hI3KlEiwCc15JVnadOQ4hSvurI1WfHYuSIzP2rrl3oFrdJ7otTkxI+DixCucgTiUH2Yd5nUG6w+QWVIh3KWpzeZbzFUuAhZ9SQ4aK8W9nbddf0d8bDytKhwq9Yct9VIbqgSWVVXrigH/frK92Hpe1jThKRGJjuHHh8jKSVm7NICRYc8QTHC/orqIht3SJOmjYHIj1T112s6enHg3ErPNxbfK23Df0Fu+u92tliCHSFPFDfRH80c78bU7f/ZDEvA1IurMU1C9lLbbWkCOmG6nj6c9OSLOCQZmlKfuK8SReep7Ok5e5wWOm5JTTyvu+1mWsf5fvu8uGRRQyPIKxC4no//Vy7RpYZhXxHk8zEBrx9S8+dfKBY+piSRx8m41ta3USevdJPCsypNHGha48diLzZ3POpHk2Of+7xac8xg/RBK9o03GrWGNw/oTxs1c0tRHSfvsvXFMAWTtIaN5yuamwe7+rnCPOxcIRrh6mSjlAsyB7wPOmONRr7J2Jc3FB1KyRDNNFOeMWfQM7+XiluuNVUnOWVVdif7aNYC2ZNS8bAxhvW7o/7wMVOcUeOHJhZzqpeXqo+qlj5C0RpbuXU2yeSV6nalRw79uVY5u2HGRw36+rEzZZpHnGu1tgO35owm X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(136003)(39860400002)(346002)(230922051799003)(1800799012)(451199024)(64100799003)(82310400011)(186009)(36840700001)(40470700004)(46966006)(40460700003)(426003)(336012)(82740400003)(36756003)(86362001)(7636003)(356005)(47076005)(36860700001)(110136005)(70586007)(70206006)(4326008)(8676002)(8936002)(54906003)(316002)(40480700001)(5660300002)(2906002)(2616005)(1076003)(478600001)(26005)(6666004)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2023 10:49:58.3495 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9b5aea7f-6dc4-4869-02e8-08dbf25b4323 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FC.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8405 .set_vq_state will set the indices and mark the fields to be modified in the hw vq. Signed-off-by: Dragos Tatulea Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 ++++++++ include/linux/mlx5/mlx5_ifc_vdpa.h | 2 ++ 2 files changed, 10 insertions(+) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 2277daf4814f..6325aef045e2 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1249,6 +1249,12 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); } + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX) + MLX5_SET(virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX) + MLX5_SET(virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx); + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2328,6 +2334,8 @@ static int mlx5_vdpa_set_vq_state(struct vdpa_device *vdev, u16 idx, mvq->used_idx = state->split.avail_index; mvq->avail_idx = state->split.avail_index; + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX | + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX; return 0; } diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index 9594ac405740..32e712106e68 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -146,6 +146,8 @@ enum { MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS = (u64)1 << 3, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX = (u64)1 << 7, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX = (u64)1 << 8, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; From patchwork Fri Dec 1 10:48:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13475656 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="qBGTlKRS" Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2069.outbound.protection.outlook.com [40.107.223.69]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B961170F; Fri, 1 Dec 2023 02:50:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g1Uy7AtnTX6+PsciGjfrnHsJYljcQ5zMsampAL/IEZRbsLh0i1WlmZj1TxaG7awjDrTF91Fx8QiaIA6XLKHOaCj4a5bkx7sLnIuoF5qY5rNL38ghbZgNTO+yOqaM/YvXMicOhLXMWBsctdccHJtPQt2IwiQHGSGh7LIciy9cVSey5jTci93zkgVkmPW1E2OHKtCSHVHQ8wfPugqJzqJh/tLtWnFww6OcLE4A8puCBxKWDHI96W7EGd5OILmuZoAWCoaCSDaL+phIvfra2h1ovtnca668bFmtrK2+/erxx5QLq1R53Qd2ChjvDSxmTT3jk9z6QjO+SD+V/mMSsZ/enA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rgPfObaWb0RRVtn7uMUlOSz5sZ2GrOBEfteGvXPSkDE=; b=X4fbsM+xtkZiImMcyyP9s2EpszpiGXpEctzk1RMWSSMfzlpjBTeCBwnlOjAR9ay4VgQnUp0tTf+oIrE4k4XFM4HG1Tk3HTNLbqiDJL//YsC7XxE2Qc1dHGXxhCYVi0njGCWzxRk9+Blxn6NXumiuwlBNbqxFjjhC/XoZjwvG1RHWoq9iqzHfDvAiI+ZsXT/NaqFriVV2+Mv/u1HpDqJJw0qCJRGDqPjVxhZ8aY7qpTGr6PvaRF9R/E7rXCBUJlnl16+kQjACIpjzEvqNCQ8kHYCTDseapF2Ga+Mv/2qj8Ue1L/KKboDQ64UWb3+bQdpSfWKW9I87MMche8C7raWOqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rgPfObaWb0RRVtn7uMUlOSz5sZ2GrOBEfteGvXPSkDE=; b=qBGTlKRS5E0coY0PgEelMwcPLj3AjMVFcM+r+eet38YWBSQqkOoBecZezl3BHcD3SnMk8He46F+t5hNb0nqrqnM8TUAdzb3zZUQ7rswXMpcTFnJLL0xJlgTxoD2IQ349X2cEOsPMf31vu6UkVbK9Cb/2voqH5ZiYnBblIBCUwmY9IvAFlo0jbxztyOmVz37LV7e4H8cIyMAfdtm2YHhCls2r9BH3euhQM+uAHtBZoXXtep8gArvdrW8LAoBYEoFC7j/E4zqc1gUB55DN9h+42ydRAfnBgZHLnrYQLmBQjPzIT9pgTrsPYew2ELo425cHg03Cni/pU+HMhyLF9p5TVg== Received: from MN2PR02CA0001.namprd02.prod.outlook.com (2603:10b6:208:fc::14) by PH7PR12MB8122.namprd12.prod.outlook.com (2603:10b6:510:2b7::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.27; Fri, 1 Dec 2023 10:50:02 +0000 Received: from BL02EPF0001A0FF.namprd03.prod.outlook.com (2603:10b6:208:fc:cafe::a8) by MN2PR02CA0001.outlook.office365.com (2603:10b6:208:fc::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.24 via Frontend Transport; Fri, 1 Dec 2023 10:50:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL02EPF0001A0FF.mail.protection.outlook.com (10.167.242.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.17 via Frontend Transport; Fri, 1 Dec 2023 10:50:01 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:45 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Fri, 1 Dec 2023 02:49:45 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Fri, 1 Dec 2023 02:49:42 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Gal Pressman , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost 7/7] vdpa/mlx5: Use vq suspend/resume during .set_map Date: Fri, 1 Dec 2023 12:48:57 +0200 Message-ID: <20231201104857.665737-8-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231201104857.665737-1-dtatulea@nvidia.com> References: <20231201104857.665737-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FF:EE_|PH7PR12MB8122:EE_ X-MS-Office365-Filtering-Correlation-Id: eb842aa8-2c05-4d55-cd2c-08dbf25b4524 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lP7yC7mm+hhX2Ew99I4UXE2wLk15YRdAZrewYfe3DLydqEO6VgPeWXFcVn+9wbg1pSpW0REvWWRav/5KjR5HQyipDEXFc1166caUBUu9LZBb/8ZqCnlStQI6KYuaHdhSRFeF+Uqg1RoTbG+sQesyBJ5iApTLU5LVzMCDd3RAf4C38arbbELHsVbmZuNG9OQgNPZyV2h9+3ZW1kPIQSYv3IXZI2uvm2IKFk054ZD+bZJk3jKZyaB74/NhH4eJmLqvC9CH6BmXJKRQ1vkhVV7MNNzHT51TzpJVmYmGxjcah394OmSWtYDFZyZmgf9fMy8/AVBrGPrWsQnAAQxVAMJPcdMW7L5/g4YmjiJazN8q6UMpopAtkyxFmWfgSxqgl14j9D/4rx7KJcPraBXol6Le1Kw8T1UV6HoEbTTNCsMKar7NGvopWwU/adexJDli1P8aLOx7ogEsWQcEudufCemc1rAS4/b0E1lQzWsJdwZJNfQFJQ2vlt4ET31pSkdBq42D8rGeG46R62EyvtCUrhnOVKdFzMFDnw/z9MpXRacjo/kjc7VdXroD36rAynOD7pJhvLVfAfZq/UieLeu6MkLFBHJ252hzbTvb/jt+22+zz2en3oF9GS2DEjc+4VpU7Qq+PawSxd58oQ9+zDxefX1N/g0bfnh8HJLB2O3RuxBQmMBYMAHTaDuTj00gUCy9fMHdIF5vXBdo+d0RnoM4fv7eAQOP9sifU+QjsqGp9vJe5/XKIF/TSdro15q6ti3ohIRagD5sM1iUO2OIt0UvxrQNUg== X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(376002)(346002)(136003)(396003)(230922051799003)(1800799012)(186009)(82310400011)(64100799003)(451199024)(36840700001)(46966006)(40470700004)(1076003)(26005)(2616005)(478600001)(6666004)(47076005)(336012)(426003)(83380400001)(15650500001)(5660300002)(41300700001)(2906002)(110136005)(70586007)(36860700001)(54906003)(70206006)(4326008)(8676002)(8936002)(316002)(7636003)(82740400003)(356005)(86362001)(36756003)(40480700001)(40460700003)(142923001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2023 10:50:01.7011 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: eb842aa8-2c05-4d55-cd2c-08dbf25b4524 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FF.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8122 Instead of tearing down and setting up vq resources, use vq suspend/resume during .set_map to speed things up a bit. The vq mr is updated with the new mapping while the vqs are suspended. If the device doesn't support resumable vqs, do the old teardown and setup dance. Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 46 ++++++++++++++++++++++++------ include/linux/mlx5/mlx5_ifc_vdpa.h | 1 + 2 files changed, 39 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 6325aef045e2..eb0ac3d9c9d2 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1206,6 +1206,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; + struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; bool state_change = false; void *obj_context; void *cmd_hdr; @@ -1255,6 +1256,24 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX) MLX5_SET(virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx); + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { + struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + + if (mr) + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, mr->mkey); + else + mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY; + } + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { + struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + + if (mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) + MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, mr->mkey); + else + mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; + } + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2784,24 +2803,35 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); + bool teardown = !is_resumable(ndev); int err; suspend_vqs(ndev); - err = save_channels_info(ndev); - if (err) - return err; + if (teardown) { + err = save_channels_info(ndev); + if (err) + return err; - teardown_driver(ndev); + teardown_driver(ndev); + } mlx5_vdpa_update_mr(mvdev, new_mr, asid); + for (int i = 0; i < ndev->cur_num_vqs; i++) + ndev->vqs[i].modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY | + MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; + if (!(mvdev->status & VIRTIO_CONFIG_S_DRIVER_OK) || mvdev->suspended) return 0; - restore_channels_info(ndev); - err = setup_driver(mvdev); - if (err) - return err; + if (teardown) { + restore_channels_info(ndev); + err = setup_driver(mvdev); + if (err) + return err; + } + + resume_vqs(ndev); return 0; } diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index 32e712106e68..40371c916cf9 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -148,6 +148,7 @@ enum { MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX = (u64)1 << 7, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX = (u64)1 << 8, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY = (u64)1 << 11, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, };