From patchwork Tue Dec 19 18:08:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498807 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2058.outbound.protection.outlook.com [40.107.220.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E977439854; Tue, 19 Dec 2023 18:09:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="BJ0ppR6i" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Oq5LLEuAeRq3w8+CyCdQQ7IVzBqUIpL0H8GEzwNQSuRk68GYMFhKtsBp/XpAG0PSUWiw4ujjo9UGY1VW8Ud31jQRb9RgRe3nqIhD8E/VS0FA89dnhWlOkB0oYPTOTnrkgSWU3KCZSk6ehhd/lDUYoKG38tsuHu+Cs0RxSup3nIQU54yGFwrQxxw7O2f1yAQmt8BaVy81KVp6f8ihcnvcdl4fkAopnnmv/l9IKmxehV7osrVK9p1ecLsUmvJ+HVQoJR501Lu5DFe8sF19nco8Sh1HfLxmcG/tg7xdN6KJT6xK/E7843zqb/QIWoOePIB91AXQuvRfOWLkYCiakpIo/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yfHOQCLFJHnJBlBhyduPWB5HBqN4yk6hN4g/bB/sNgw=; b=VLHyWypmAD0iyyrHjAAn4BZjku9U33QD9eWWgt4Xjl/PbY0T4QVsTnREE5iXDzK1kZZYwcs37BDOeADjR8KSDCgPhgyAkWSN50CIAu1gFH6kuRhGtv6tks2ifcOWEQrHBtPeX2qLd8r3aXrjNHsr5bHOsm56nFxxYsuptNMkh6th05cHqAy+9erFYQKNZXZPRXiy2uDFORDqSdbiF0c43OEl0BlVZHNHkGmad3MhU6S6PI1vR1tcBqfMExfewre7D6aK+SiGNdOQbOntiZSWrP97nphRffgsba2lHvhy1dFNfeLWPW69U49JOUTBrhuu+u9rbS/eCToPG7BcTDBkeg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yfHOQCLFJHnJBlBhyduPWB5HBqN4yk6hN4g/bB/sNgw=; b=BJ0ppR6i7QUSld338BqP6rvFYB5PVrMNUJVn+BlM1waZxDNP4P2CvJJ+JAY2Md1QbNJ2vkSPraX21JOQ4iWK5ICbymJ4gQI/fxu+grusZF2zuu1WTyuqNKl5oYjGP+/XbiK0uSG66N5eO+PQeleGqp6TY8cwRRMgYA4azaRXWt/HqMIK7V9yA6IKx6uUqPqgAs2ij9hlEzjAOzzq8waNUCjzHJiA7qXU9pNPZcpjYrthqpK+qs7auHRu62Nblacp2YUg3FLmPZd5OyZmd26PJtpYmfsOOrxfZ7FvO5GaoK1e9G8sSeq/1Hzo/UvpMGGaTWX76HPvFxfRuPDa6nrDNw== Received: from DS7PR03CA0269.namprd03.prod.outlook.com (2603:10b6:5:3b3::34) by CH3PR12MB8305.namprd12.prod.outlook.com (2603:10b6:610:12e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38; Tue, 19 Dec 2023 18:09:20 +0000 Received: from DS3PEPF000099D7.namprd04.prod.outlook.com (2603:10b6:5:3b3:cafe::1a) by DS7PR03CA0269.outlook.office365.com (2603:10b6:5:3b3::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38 via Frontend Transport; Tue, 19 Dec 2023 18:09:20 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS3PEPF000099D7.mail.protection.outlook.com (10.167.17.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:20 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:09 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:09 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:06 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH mlx5-vhost v4 01/15] vdpa/mlx5: Expose resumable vq capability Date: Tue, 19 Dec 2023 20:08:44 +0200 Message-ID: <20231219180858.120898-2-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D7:EE_|CH3PR12MB8305:EE_ X-MS-Office365-Filtering-Correlation-Id: e189d0be-b292-4287-c46f-08dc00bd9fc7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BiX/i7bsBjL0t8T3XDEl3fJmvZdUcfKuCZj0nSd32cljh7ZLLV7GScW446Hgb2bG2HMttvGbeMM5ApfsE3Jg2106kYukuhmGM8fizGAATvzirFQJ9vwzqPdPOhAP1tNqv0+vlcTHTeOWhFEYuq4gmoyvOmKWcu06EfMgmWKQx/QtGRne31gltTavsJs23KZy1dPa2cJL8lnZAcNAUf5jmoFGLppAnpMYCTzHXUQ2+NBx/6DkVwCDi5Ft2qLFF6cNtuFsU9lletf6DNahkEY0vZtwNHblv695AV/SDIod6Voon/kF/GBVUD5ZIeBQJf+HR3gz4nswtkdvTPkBNQM19EUxhhw1qE72U8NEJADVfIabCJ7UZ9A11hAi+DOc21FVt7TZndPvrlj/JzgSbomd31ZsfYENzB9ER/OZCxRLanJQPcZeENxaTZ/wi/uuuG2gAFJan/gRdVFAl78ema2qMA38kJHpYNScv+LF+IhXRvMijIi8WIvzyhNt2g+OpH2dJZr37/3uvsgOMzojevRJMeNHDjyqhLnMLWTipY1UfukxwpszYnbKWEFzKaNa+Yx0MLL1CVEgr/7kEiyPrDqRrLBx25eFEt3gjlvDiVWiDWK3sKDeEV8p48wgvYKU/z/9x3VmBv7QRJWihJ7TrCw0YD71ipiYoBTpwh6EN0flDYu5oALwIfPgsFOwkn4nAK0elSUja0g+mx5AbvKqVZB5WsEZmkL8ikxp24sMOBtbd0XXdCj1Bw6i8HIAqEWmSsyh X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(39860400002)(376002)(346002)(396003)(230922051799003)(64100799003)(451199024)(82310400011)(1800799012)(186009)(46966006)(36840700001)(40470700004)(36756003)(316002)(83380400001)(40480700001)(70206006)(426003)(336012)(66574015)(70586007)(54906003)(110136005)(4744005)(6636002)(2906002)(47076005)(4326008)(8936002)(5660300002)(36860700001)(82740400003)(6666004)(8676002)(1076003)(86362001)(2616005)(26005)(41300700001)(40460700003)(7636003)(478600001)(356005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:20.7849 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e189d0be-b292-4287-c46f-08dc00bd9fc7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D7.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8305 Necessary for checking if resumable vqs are supported by the hardware. Actual support will be added in a downstream patch. Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- include/linux/mlx5/mlx5_ifc.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 6f3631425f38..9eaceaf6bcb0 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -1236,7 +1236,8 @@ struct mlx5_ifc_virtio_emulation_cap_bits { u8 reserved_at_c0[0x13]; u8 desc_group_mkey_supported[0x1]; - u8 reserved_at_d4[0xc]; + u8 freeze_to_rdy_supported[0x1]; + u8 reserved_at_d5[0xb]; u8 reserved_at_e0[0x20]; From patchwork Tue Dec 19 18:08:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498808 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2089.outbound.protection.outlook.com [40.107.223.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DDD139ADF; Tue, 19 Dec 2023 18:09:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="tkKuo4CE" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CXfc8hchL0+v6aNC9EUj8K/UCNAjztvhVN5wJ20zSc3x7wK6i5Ks+CE7SJy0QqXWjVgklI2zMeiaWnpY/vBFLFek8kkCaPa+MzM68Brf/oLhaasOkol+umFmDadK1jouGYnO45ftEn6qdLhDPwN78iX32EII7504rrUMDbruPqBKmDPkekOhAv3HlmwaZ8J/OOCXcyOFf0CO5BjvOwE5jzKIBpkGr4gjIX9LnkRzqatXx/XfC96L/hgKGWAr5nij2xMSZZxkTT2hNisdSGGTsH1pWuJoq9wXD5sV7ehm6FV7V1HsupRmlIor3AAsbZrrPiBgR0BbKWWRPO/+vJDfDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hFTHaY9PtUATO1cmPdLknCzEq/FUKk+/vuKSvCcAzAk=; b=C2YGIuDbIx/6hwdFS+bXK43xm3uK7bZn/1Mbm5JtyNG0Ku6sNpsC+rrvshp98eyhvBHre7TeSnGqUV9dLpfltsXGrgmLZpREWZ5JSwKhwYK7bhpyoMnRB8NrtOomoQNGLDAz6K7+CA3vUGQvOt1GnJGQfj4v55vdQxtp/A1oVOfMv6boNT1/hSCPC+ggHXe2FgTmHWI6ohfllj1f7P9sOflXCO4XMaWnkpp9hLEppNRUcWPHkH5SEBXRXkfJMyiicwZGJNOUpNyVh6oLu0werI6D7QsbQcNlTZyHZsfFpNd6vgZ6rjVPyw5+S9jRfOJx2o/ELRn6Hlut5BpaIRdovw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hFTHaY9PtUATO1cmPdLknCzEq/FUKk+/vuKSvCcAzAk=; b=tkKuo4CEWQoCKg7aXdBG+kGYgXaWi1y9r4KzEkKg45WO08k4tcS8DbR7DtNEAQjk/BlsvCDlAeqCHEkUvHQ3x3a3Y1qvR7aoszCeIriqWCJEdjyD/t9Lq1ron9U+jWGPVmTRQf9+CH3OXzVbo3I5KqE2iX7Ov5S3jExcf6rLUM0cmtN24MyZ9XmYRjM5YghA52bJHmYTd0aBYFOPYasomBliHj7hqrEt1g1Y4p6UXJVWaIu2Ef6z2Mt9v1sc9AKnhWOOFggavl9/6FmgOQXksQNuxLBDVNJgVxRSxAp/f0pa3uj2uLK2W8M5RrwCe6NK46WaDZ4jxJoHliuiMdmSPA== Received: from BL1P221CA0003.NAMP221.PROD.OUTLOOK.COM (2603:10b6:208:2c5::9) by DM6PR12MB4546.namprd12.prod.outlook.com (2603:10b6:5:2ae::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.39; Tue, 19 Dec 2023 18:09:26 +0000 Received: from BL6PEPF0001AB72.namprd02.prod.outlook.com (2603:10b6:208:2c5:cafe::4c) by BL1P221CA0003.outlook.office365.com (2603:10b6:208:2c5::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38 via Frontend Transport; Tue, 19 Dec 2023 18:09:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB72.mail.protection.outlook.com (10.167.242.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:26 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:13 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:13 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:10 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 02/15] vdpa: Add VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND flag Date: Tue, 19 Dec 2023 20:08:45 +0200 Message-ID: <20231219180858.120898-3-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB72:EE_|DM6PR12MB4546:EE_ X-MS-Office365-Filtering-Correlation-Id: d4e7cc82-c982-4f55-7be0-08dc00bda317 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: onUa2fx0UvWyG6ApRLkSf4/fc/VAGqo959+wNRV0SvK/E9InpGxx+wDU8RT7/IykFPW7/GBBf1kvWIzqolrAnuLw/9SG01FyfDt1SDEsVFesjRPy++uB7nnosU7nKOgzWKGT/34ez8zMdTT29cvpAtFesPlkSid7D4T7Dh+RkDwrtWz4B99OE1YlOi6H801PKcK0Z9acUW+QhN8glp0z6PdWMNUbOtM2I+528Ev1DoswsmuWDZ5ay9XAJo9+qegqsjHNvO0AXz+ucbInZx9c7Cx5F6nU+kYDnfs8Ru9He03bJPdWEyYnCQEEvW1Ss9cFBWQ7wLq1gO8VJXSUT/pqj08IKt3PsXyf8utWg/f5pJTRXO4v1RrzaJ1SPXsB6guwSgkOhrUHAsPb7aFU27AM19MPAEvzM1mitlDhPrjjT58GwOn100vy+RUojX6ktlPLDaWFHVXQoOrIVovNmyU8m/fl6CtfRk+xQyB5fW30VRFqbrgkWEUsaRMhs1WJlG5XuNBYNQXtkb3u8o1pSKmDndLO60MszRrawmQH7Cxe1t5LToyoqLJqSvcdq5HGSNe6KXMXE7/qX3nX0lS5lIEkBW+XhNBJMFPp705mfv1oSe2TYtFTnSXMN9fhKCT2Y/UbMdM9Cc28dfs6vYstT+QGG3vwwTIE7u1XJ66vAU96gOFSVaw3KTfqqSc0NFq6BJQLV1Lif9YUErRz7VKFjexErwYVwwyZhnrkTXIljAtfirSJ1suLIScdUfvSyo1icmWa X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(396003)(136003)(346002)(39860400002)(230922051799003)(64100799003)(186009)(451199024)(1800799012)(82310400011)(40470700004)(36840700001)(46966006)(36860700001)(7636003)(356005)(40480700001)(47076005)(40460700003)(1076003)(336012)(426003)(83380400001)(2616005)(66574015)(26005)(36756003)(86362001)(82740400003)(6666004)(478600001)(6636002)(316002)(54906003)(110136005)(70586007)(70206006)(4744005)(4326008)(8676002)(8936002)(2906002)(5660300002)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:26.2936 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d4e7cc82-c982-4f55-7be0-08dc00bda317 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB72.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4546 The virtio spec doesn't allow changing virtqueue addresses after DRIVER_OK. Some devices do support this operation when the device is suspended. The VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND flag advertises this support as a backend features. Signed-off-by: Dragos Tatulea Suggested-by: Eugenio Pérez Acked-by: Eugenio Pérez --- include/uapi/linux/vhost_types.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h index d7656908f730..aacd067afc89 100644 --- a/include/uapi/linux/vhost_types.h +++ b/include/uapi/linux/vhost_types.h @@ -192,5 +192,9 @@ struct vhost_vdpa_iova_range { #define VHOST_BACKEND_F_DESC_ASID 0x7 /* IOTLB don't flush memory mapping across device reset */ #define VHOST_BACKEND_F_IOTLB_PERSIST 0x8 +/* Device supports changing virtqueue addresses when device is suspended + * and is in state DRIVER_OK. + */ +#define VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND 0x9 #endif From patchwork Tue Dec 19 18:08:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498809 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2089.outbound.protection.outlook.com [40.107.93.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26A723B194; Tue, 19 Dec 2023 18:09:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="dpW7youp" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ebQZEu+WspDP1QYSfz0NnSQ6Kmlyx2BIOznSNVTJzG1L9ffwdGWpk4thDfgz3kOtDTpms9pgAv3rucU5kKuooYc1XNmiyrm0dT2booetnuDHg9F3FJH8GwvJjl38peRrSFsMFe+naaUcZPKRM5iyHXwB/dajMqLk1vJt6me6eu9mLDWbElXUVYp4QY9eD8svzmxnSzTCXNtgMQpFAYagmVhsXvAMLtmOn+E+5zCI5e7z3lKB6XhqXMyshBmCNVoGAgWZJxXe44TNX4ggFasT4ap1grbL94msJSwQjd7tudUTmtpcppsTXE23x4naSvJLrVFDD5qbGfMzeV9g/YN3nA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vu78OW1bcKc+D9aGItF8zsUN18eG6wRMeld+vpa5LW0=; b=lMbqt7V6fzUUWDy4La4epikpUNAuDR6p8T34EuZcd1WIaKgUyWtwUifzp42p95a/CwoFkAD4sO0uANuSFyZP6tcNwb90d4l7tRlTzZif+b+MS1z048jA46wSz6pKpSv5EEMJ+6y3xUFAslSJ5/8MXxzSkWA/X3er/LbJvQsZkqTrT5m+FD7di2pYTb+s2j8FtSBX+esJ0mlN6niFAnRJiYgkTMtiQYEwWHhO2RloX8QBLjTZNVX7PRP5i1IZZuJhjl00ts0wRr6rt+FTSfELGEhcrH3D7QK+fuLYiCj50e5Afilcvb1QvY/bwYpBBjcRfwvIKw4L1YKGclSGW2/Bkw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vu78OW1bcKc+D9aGItF8zsUN18eG6wRMeld+vpa5LW0=; b=dpW7youp2vJFWKKGbVx2/ggJ6/JJC9xSTXgX3rGmN0UncB8HReT/IjhQ8kJ3l5CD9anZlDO2mozIFgOG5nkb9DSYfyuYXXW18lkwhSr/HvNSAxYL4J9Hket43AHjJBfzFM/QEw/MfcL6fCJUtHhKTGeX7ZpDGR0dIAf5R5Lwr4dFYGrHpoqv/qWh2l2bOkpTOCe5YVvt9ahv0ivTausimWI1LU3GTs9/5chptyzp5xuSDH7gl7sQIcIaW1WLPZDGvDbbVyayfM+tXn2OPBWX7NfviE2dOUUJ3mYribUAGNZBsnSW5hPe6NP5b1V/bWUIVKk6RZRaNmY1sPfkpwsVjQ== Received: from BL1P221CA0017.NAMP221.PROD.OUTLOOK.COM (2603:10b6:208:2c5::35) by SA3PR12MB7922.namprd12.prod.outlook.com (2603:10b6:806:314::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38; Tue, 19 Dec 2023 18:09:32 +0000 Received: from BL6PEPF0001AB72.namprd02.prod.outlook.com (2603:10b6:208:2c5:cafe::a4) by BL1P221CA0017.outlook.office365.com (2603:10b6:208:2c5::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18 via Frontend Transport; Tue, 19 Dec 2023 18:09:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB72.mail.protection.outlook.com (10.167.242.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:31 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:17 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:17 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:13 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 03/15] vdpa: Add VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND flag Date: Tue, 19 Dec 2023 20:08:46 +0200 Message-ID: <20231219180858.120898-4-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB72:EE_|SA3PR12MB7922:EE_ X-MS-Office365-Filtering-Correlation-Id: 32a3734e-99f7-4866-031d-08dc00bda66d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6kjW57HZAiO05GgMrFppji681V5qQF0QV7RzqGlK6IbdsnqbsHmnOoSMWtFp3xqZK2MnI1KY+FPoAG2O8W5q5ACo1XpeL51gFB92XDAtdIF3EimFeuebINUmbd/Rwmnh7gagCCakO3j57QcxXOXthVFr0lB+cLw8cMjAd/fLN7LfXg8CXzROLkXVazdKUujip/Mst7NMhjKl/JoUny2rOVpHPXkT1QgZcqG1paaCjRcTz3Q7mr7Fs8Wwx2iOSPboIxmUnok6xN1j7jmKul4BeIduiJsKO1Qd4dE2UFb7lDbmbsjaKEc+08gPMzkL0quQtxa+w8Dlp2JSXReEZ/my0tYrIsKVe1iEDqKhD/gR3RBL4HXCG7ogv7/K0KmNDX2sOmWcRDiF1dXpQcqpUeCwyYbXj/gHIb4+cJFm4w8TKX9Lp2ngU9KTqi+81r7noLAdydjZr9OdQjrKkgTPNy/8ztWSRtq/P5tTIqk3TTk7a9dZYUd2oc6GoWRT2tu3wxFLjnf2roptamN0PYKma7x9L9eqJUswnW6Ossk5/BBB4NVIL0zEyTj3ooo+TxgdOk6w4XHiik0XYw9jLysvo+HLqiU25XBHOrcVoykVIGS5BOO7UvV7XqnzCs2y7zGtDUfmtd2LhYgpJUzQLapY2y0DzK5n5IQliGEWoBew7g/lem2SaYUv1mc6eCoEbH64xukxM5ovcZGCc4lQinBYvvQSTlVvxJ2jY8qzLgm4mKD7SgiqNLVi7W4bwYzlxt8t6hBB X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(396003)(136003)(376002)(39860400002)(230922051799003)(1800799012)(451199024)(82310400011)(186009)(64100799003)(46966006)(36840700001)(40470700004)(40460700003)(40480700001)(6666004)(426003)(1076003)(86362001)(2906002)(4744005)(2616005)(5660300002)(66574015)(336012)(83380400001)(7636003)(356005)(26005)(47076005)(36860700001)(82740400003)(316002)(110136005)(4326008)(36756003)(54906003)(41300700001)(6636002)(478600001)(8936002)(70206006)(8676002)(70586007);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:31.8873 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32a3734e-99f7-4866-031d-08dc00bda66d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB72.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7922 The virtio spec doesn't allow changing virtqueue state after DRIVER_OK. Some devices do support this operation when the device is suspended. The VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND flag advertises this support as a backend features. Signed-off-by: Dragos Tatulea Suggested-by: Eugenio Pérez Acked-by: Eugenio Pérez --- include/uapi/linux/vhost_types.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h index aacd067afc89..848dadc95ca1 100644 --- a/include/uapi/linux/vhost_types.h +++ b/include/uapi/linux/vhost_types.h @@ -196,5 +196,9 @@ struct vhost_vdpa_iova_range { * and is in state DRIVER_OK. */ #define VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND 0x9 +/* Device supports changing virtqueue state when device is suspended + * and is in state DRIVER_OK. + */ +#define VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND 0x10 #endif From patchwork Tue Dec 19 18:08:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498810 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2067.outbound.protection.outlook.com [40.107.223.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 061333B18E; Tue, 19 Dec 2023 18:09:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="rTPm1bmK" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ie0LEafI4GZYwVdYKfM5bYtFZxBsDlMrLwj28MEBzAKJVOwFvdH8meexzQzPP0Yc1kuVVpBWVp9WGB9VFVnsuVWlftmI42g6gF6vvRv8NzG+3a1G6Ld8ApcmSeDwXO1JojFz5acqGfWOvms9e7b52rk/TV5E7UmiC5jlyRIkqNR1Dlv3da4Sz/Pa00TM1DTIu+fWIzYUOCz3TZoU6WIJwsSLDrAoK8LWz40mh4YIaSNi58poxk+Sogd7i6hgq1IKBvv0p/W5h+1DS66EM7AMn7hprw64i7EQv/Ol4aYH1+3q6qs2SdGCo0aEFjX/P+lsI0OkPiZed/TgaWprys2/1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nguvkgd7vI+uB21EEtUqXSqoijHWYfY5mS4xdXcQ8Fc=; b=LAYP2moOZcS/jkfKiFFcDV7fdrsSbB0DWniWAQo2xE7P2uZIbLzcsR8xSztLk2VJvoe+CVxzqADwrmCWdO6oC8XeQVdbBhQlvN6Is2JNrrGr62Fcx/gaNfrcRYM42ZrMjoV1Zh7gJ3rQaFS3sRXovra+A5LFIf04iK999S6L7ZXrcqK9386JPwXkWKN/tZfJRWXIQcHqKOqhzOyQIVm2ZsCbn6zu3lwXsWt689flneYIp84NU4E1KvuSPCfpjdOrq420dovqvY5BSr544ELofHdWvzKTLE26X5Fs/mS5zHcsCNM42ALy9InO873SuFPfJ48qzX1/2HRqAVcke8Ixqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nguvkgd7vI+uB21EEtUqXSqoijHWYfY5mS4xdXcQ8Fc=; b=rTPm1bmKWBb51vKhnwsTVTBCt8bCmwKzU35CrFmmfhGvywpSBO8KEeZc6cYLJe201k9tMLLacXnXoq/OVJFotqgL+So2d2EkRhngboNpQc8d/t+prZpXZbtSXhuU5lh7p1LVTFeMFXvmGKHAvV27KE8mVQxhDEUKl+8iuRnWH03uQeCQtu/prpHmruYD2yKLhdkhltTLGU+0nj3GDziFo/dLzoZvVIWvUYhxDDK3HFXQ4Y0+BhRd51jFpBsws1UNxlah+SP2Yg3T8+mnyoz9jwRxMaKpIGEdhZtBVcpqPJkg0aCOZBPCzCMbFAch7pXy6lLnIBVoqT2mQk1EPjkTuw== Received: from CH0P221CA0021.NAMP221.PROD.OUTLOOK.COM (2603:10b6:610:11c::17) by SA3PR12MB7903.namprd12.prod.outlook.com (2603:10b6:806:307::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.37; Tue, 19 Dec 2023 18:09:32 +0000 Received: from DS3PEPF000099D3.namprd04.prod.outlook.com (2603:10b6:610:11c:cafe::21) by CH0P221CA0021.outlook.office365.com (2603:10b6:610:11c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18 via Frontend Transport; Tue, 19 Dec 2023 18:09:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS3PEPF000099D3.mail.protection.outlook.com (10.167.17.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:32 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:21 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:20 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:17 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 04/15] vdpa: Accept VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND backend feature Date: Tue, 19 Dec 2023 20:08:47 +0200 Message-ID: <20231219180858.120898-5-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D3:EE_|SA3PR12MB7903:EE_ X-MS-Office365-Filtering-Correlation-Id: 0c54d6cd-7dfe-477f-02b5-08dc00bda6b4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vMfKGckqlLqS2RkH3YC7d6KERtZs/a5w6cL6V3QPLHk/W9H7Tpb+pPwOdJgL8JUvjk8JKeKj8BPgfp5TLFGOhCKNHaGEsTtj31ZAL5hWirok6Zo0wMEBcQsNMEZ+KeoCkh2+4xg81+qp8ROUrugWq0VdMWDeDQ1M7LyPc5hY47DJbrwAjN2atIoTxxofXFwmPQArd/o2Ddnrx2IC03GNHISxDW/IUPWYTugmU4tDdYYoNRur+hySQlz6pWHN0GBGpkV4j23U8RKW+HMoDO43QZTKrzpOMfyNMgbB/wx2W8VFXhLlFn+i+uRSxMSjx2nK7MVQcu/yKhMGW1lkKz4dBJDp88swT22ZcT6r7VSAY9c7mlcWEYg3lElw677PSbiSdSZ7R43v4kUqxTJSpFJ9GovhPz4eBKDthS/6CvCBRx7huBAxFSGB3uw++DlpXqe/Xgf8BenCcWLSdpcemgNJscHPw/xH5H35pVkOlUoSJUk0bw/s6wdciowcfenaFAZRgjYoH/GtPfE6kWvG6BymHTXE31WwcFAiSAzG3fc3IRnMov+/mnDCdOJQFDvY5kH825Dh2UdrHCgFiajfU6ugfqsmrxBPMXHZCzmG3yU0AUTUc7xdotW+NIZMjpDoh5ygs2Zcg0eSVCe5aLfB2E2sKiWvZoG3JNC/jCQuQAGBH0NqSDuWS3Wc+wQfUy9HkWVhr+WXc9v8ht/5QQ3p2GTQBlwUcN5gt1b2AxJE+jXjqnQwZWKjPVagNycFpOdviCqz X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(346002)(396003)(376002)(136003)(230922051799003)(82310400011)(1800799012)(451199024)(186009)(64100799003)(40470700004)(36840700001)(46966006)(54906003)(316002)(6666004)(478600001)(26005)(47076005)(336012)(36860700001)(66574015)(426003)(2616005)(1076003)(4744005)(2906002)(5660300002)(41300700001)(8936002)(70586007)(70206006)(110136005)(4326008)(8676002)(83380400001)(6636002)(7636003)(40460700003)(356005)(82740400003)(36756003)(40480700001)(86362001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:32.3839 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0c54d6cd-7dfe-477f-02b5-08dc00bda6b4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D3.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7903 If userland sets this feature, allow it. Signed-off-by: Dragos Tatulea Suggested-by: Eugenio Pérez Acked-by: Eugenio Pérez --- drivers/vhost/vdpa.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index da7ec77cdaff..2250fcd90e5b 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -749,6 +749,7 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST) | BIT_ULL(VHOST_BACKEND_F_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_RESUME) | + BIT_ULL(VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK))) return -EOPNOTSUPP; if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) && From patchwork Tue Dec 19 18:08:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498812 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2051.outbound.protection.outlook.com [40.107.243.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AB5B3EA75; Tue, 19 Dec 2023 18:09:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="hzwOkIcj" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EK5sm12K7f/IcSyPuklNVwdmfe+GSDzHhYYoIqAxRC4/3hJf7OHhjuhCMZCpjpWmcczxRnCXMoPlDbTlHmKqCg77Wj2MlXqVv0gLVmSPMBzmhb5K5JQChSuUGK1xMYpVNVNLMkuuM4HAvZGM+44cpwDp5aBXznISL1qD/hlrWSLE5DmJQtpfd/xCbQlgOw+r7Z6D4oyXmYJR4BjTZCWsLAIKI4QWzmhEs51Sr6GDoPXOLdjdqhTCnD/r6JIcmL+EWLgxUat3aFqiyrKSWl8DJtZbKgfWHPAVsth0M0PzxmhsB6OcR3tXRZPbOvpC6z2MX4OfloPpkk6BwhnKTcpL5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Aq1IejAJZi+jwn+fRnf3zq6axHOwHpgeCeFJajDdu4g=; b=JlkZ4IwZyxpotkF0vacYtmuJdo26E70/fqn5qnIL7l8ktSeHGeBe3FwXtt5gjOftVqU3tS3S1ozKJTJ/Vgp3BaCvouf8XuFakc7ozBRXvw8ENobNmqM46hqErFmyHYnsTJmvHrD/jlbHgN7y1j0IoXTOldlJlakNLQjQU3rTyHd4AYJniLlfZ0hVgL1H/5pqUAA/q/qzaOssU6g55JgX7E1udasWUYtMFCWLiCZuj+AK0ryvQ/FV6yyNPECIGlGdCwEuO3fzdLEJOqlNbqGIGGscXPbth3Y3okPEy1YIwiBF6FnQdOIiDQOuBiQ6pMOiI86KYHguS4VIRn5LPjroFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Aq1IejAJZi+jwn+fRnf3zq6axHOwHpgeCeFJajDdu4g=; b=hzwOkIcjNC21aWC65UI4KmDjvY1CvVmNj3+4gtb1oXD8KSJZ3WjnLX8IvZnMvWuLVzcR7YOcVFwvNE8bWMOj/ceUX0pSPC8VYnZzo9GTiw88KDv0EnrbPhGPCdKepBuEnKYn40XPGENDh2e8MSL8b1octp5iZ6TPKLElQLzzEvz3ZBOzJal9zWuLeDi8q1L0FCAn9MEzIyny8Mf2t2xOtc4zmyyb0cByCNPGSGp/r76PavQDgYCIlOddaC7j4tQ7mnk7HfNtWkvuHPDxrvWi992TY9iQerUVu26ym2B5PqzLDvgWWrYflb1LklSYkJo8n5DSHZpOaBEjTpb+EtyF1w== Received: from MN2PR15CA0065.namprd15.prod.outlook.com (2603:10b6:208:237::34) by CH3PR12MB8849.namprd12.prod.outlook.com (2603:10b6:610:178::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38; Tue, 19 Dec 2023 18:09:41 +0000 Received: from BL6PEPF0001AB74.namprd02.prod.outlook.com (2603:10b6:208:237:cafe::a9) by MN2PR15CA0065.outlook.office365.com (2603:10b6:208:237::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.39 via Frontend Transport; Tue, 19 Dec 2023 18:09:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB74.mail.protection.outlook.com (10.167.242.167) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:40 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:25 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:24 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:21 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 05/15] vdpa: Accept VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND backend feature Date: Tue, 19 Dec 2023 20:08:48 +0200 Message-ID: <20231219180858.120898-6-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB74:EE_|CH3PR12MB8849:EE_ X-MS-Office365-Filtering-Correlation-Id: d935b6aa-7f37-4a78-cebd-08dc00bdaba9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JeJls1S91VzzW8knT2nQkJWAAzEa7vABhbulvFDQvEOHfKAgFK4t6iRYzyViFOuvW6Jlv8FVXBuIbA+KrlXR0UUliJoHOQ6FfCipLuILhllkLExtnQZJzKb2vngM/RYy/5Pe4wCyoEfRizil6xU9onkETzPJeQMO43U5fqLh5VYbuHm3PJ7j4fu/PbkW2uWax4H8s4+jB0rUgysPRSspQZGLG7NshwkFrdIpsbz88FvqIjFC+E9VUGJhcZutCAhp0Pb+XD3PtNWa8Zele7DxdLIVSLXPeuwVLX6CmhQMew1TLXSEFQGyxLKjTyEwWY1+jxphcgtN2lMRVsynddspU5O5Q2DiPt5qSNOomlxY07aVnO/5GB2eMSzpwpwMWKpdZskKMMmNYLbamoekwYdxs6lxK6sEdqoJlEOtfrm4Bl+WdL809N9/RvCAUtpFiBCo6uvC15zrUTqlSPpuAZEjRYI/mLRcW1JFkIaQ812LwFIbd8Yj77QtTdD0QOCazKmze4pD6aGDoDN9Q28xNNK1ViVjDEMTPrZIb71MfJLvqMrCpX++4IRA4KyymaW8SD112PmNYIzcRs8aMhwi9MhZQMwvZ7k9QeCRAIgz8wiHoVunLijJv95evnNSHmfFD4xUKzeq0zvG6vRYP1VT+bYEiynozFDmN1LOW5u5I9J2quxuLY9hfh6Py7WyLppiQ8ShLzf+Sa6y3yvWAbJaAy2bzJygb4KXFhMEJ3CKiFB8aB/gzTO2wLiVbF+p9m2yzSSS X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(39860400002)(376002)(396003)(346002)(230922051799003)(186009)(82310400011)(64100799003)(451199024)(1800799012)(36840700001)(46966006)(40470700004)(40480700001)(40460700003)(70206006)(36756003)(82740400003)(7636003)(356005)(70586007)(86362001)(4326008)(66574015)(1076003)(426003)(336012)(83380400001)(6636002)(2616005)(110136005)(6666004)(36860700001)(54906003)(8676002)(8936002)(316002)(2906002)(478600001)(5660300002)(4744005)(26005)(41300700001)(47076005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:40.6744 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d935b6aa-7f37-4a78-cebd-08dc00bdaba9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB74.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8849 If userland sets this feature, allow it. Signed-off-by: Dragos Tatulea Suggested-by: Eugenio Pérez Acked-by: Eugenio Pérez --- drivers/vhost/vdpa.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 2250fcd90e5b..b4e8ddf86485 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -750,6 +750,7 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, BIT_ULL(VHOST_BACKEND_F_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_RESUME) | BIT_ULL(VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND) | + BIT_ULL(VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK))) return -EOPNOTSUPP; if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) && From patchwork Tue Dec 19 18:08:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498811 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2063.outbound.protection.outlook.com [40.107.244.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 896153D3B7; Tue, 19 Dec 2023 18:09:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="c2DRyu8c" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fS3n2aoMkbtfiLertH9ft6Vo/HVHxQhOWjMDv/ZwV+qtIRi8eYIfziCFL61QMycWGF+YDKA2eavXTTRU2xHXCUQHeeWKWU6FBSbHCYpmTFYlN1jAKo2MjtPxILj4PffkOp+ibgmlQFr6h30gsG9mE2C02tOodCCViR/n/LTzaQoV43FWBlo+foi3HUbA/TGFWBC6jIpA7iYmr/5B5PwnkzLOqnPozLwbWAMZuPq3PhJ8g9yNsALnp0Lyh6Aex/33j/HMU1JfOUXA0SVJZOL3dBVJNtvr3M5XSxlwtsVHQ9WGgZurUhZ5so5uD5OJhZ4kmvGWWU5Mk5hHsQq6EfanoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QIfSBnXmuVIqu1lAUk6iMnr++Wfajo5Pv3OFCXdv0GI=; b=alBkkNUNelnnXkFgBpFxFaQwz8hELS3GX2gRlo/yeekb1LoWwH6z+r1DEFqGPOEzz2/dZOwRQg5F2GrZgDq8E/6LIdJLb0xXTESgvsAY4DX2nJOHpufd5ngJN4XS3QwAxkR5zO4t/puQ1zu+oE+QKBifgJxee12qg10quzrDXLPBcjX4oo2DQ3URZodW3wYT1TRTgSrfkLDKb2/ll4uaTQmvx9RRIgn6pv+LE6u3F3/BHEYf1B5ffhPiZX2EXpHgzWsG9YO4XQ1T8bHlGBAvX+12InpaxweaW9O2D4rDXG8ZGQbrhT5d8HHqlMqY6lNcgs67LnDiZ+T0Nz0PjgI5dw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QIfSBnXmuVIqu1lAUk6iMnr++Wfajo5Pv3OFCXdv0GI=; b=c2DRyu8c6iBgEkskiy+nug+zc1MFX5QjBLRqXMiwpKF/cTT1tqgy8j9T+ALQs1bhhICb0Pjy8GWIh7adYkCpUM7v8+1SWgm79WURy8omqVPeaQiiHQaIgq7ApBwyBhKXcyuaGfQfMF5zMQsdg3jZkEcfK5uiwUHE5o5FyN/b02vhPCwooNFwnPVV2KeN+fGT+nAB6C+b6/fE2QOLIt//8IKj/JAXUHgrjqcfk0QC/7uE/sm4q2xy9E7eS0ffbyREjp0EEcG/j9KQkxDvBaOBVp9irJOWMv2xbfNeFkdgejThNdXzeZP1AiekWcuHGrF07AnsbzwgVPQSvuU110EZ7w== Received: from DM6PR03CA0084.namprd03.prod.outlook.com (2603:10b6:5:333::17) by PH7PR12MB9222.namprd12.prod.outlook.com (2603:10b6:510:2ef::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18; Tue, 19 Dec 2023 18:09:39 +0000 Received: from DS3PEPF000099D5.namprd04.prod.outlook.com (2603:10b6:5:333:cafe::81) by DM6PR03CA0084.outlook.office365.com (2603:10b6:5:333::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38 via Frontend Transport; Tue, 19 Dec 2023 18:09:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS3PEPF000099D5.mail.protection.outlook.com (10.167.17.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:39 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:29 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:28 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:25 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 06/15] vdpa: Track device suspended state Date: Tue, 19 Dec 2023 20:08:49 +0200 Message-ID: <20231219180858.120898-7-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D5:EE_|PH7PR12MB9222:EE_ X-MS-Office365-Filtering-Correlation-Id: 940319ce-a940-4cfd-2d59-08dc00bdab00 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tnOMGzmrt/bHF/+Ug1DF6OYO9Gc6NnX+7juJ9uw0dMVcqLzfLx7ZdYVxZkVlC3zlqyXomP99QdwJxcf1a/p7/CUzQKaoWR6+ozU+VY3Su7b2Jydqhq9lj5Y+9rbFwZ6pLqHwbF6q2TNcnB8A+Vz6hlQE7tHUK4oXBcVBXIDtfmD9AG+YwUT1oRrREoTczBmixy8FJadpNLkxT/8BZeeIlgl3hFjYeJh+UgLs4a67Q6X3b2X6qRTER6DyzKKcBH8G9DmznW2XaRIKn80WN4HyI9OYSUJv1dZHDdubzHZp1sDULqbSTuaot7+hN60wTiJ/lQFvW9zB5DQ34uS2BH9mfgIq2s6FmP8Q5N6pUWJrIYM9nAGsJrab/mxLC2JSLuVhtZJiBNMnq2MXAzg/ZSWv4YZOXUYmaVeyODH8w4Yb8U06b1cNJ0TYpssIhq/rtQJGv8/oWeMbSJ3RHZnAAnaRsHw2Msaro0hSj2lgm+nrnsFDDDMvDwDCOTOUmknqq4TK4QPgUCRy0jcp8IrLPsV0KiRVjMnu2e+drnODWL8LvsW3892LxyRjeay65oW3ClAzFcn3kj1uWhGA6Aknx56lkxALRSJkRKyXWh3gwzazxEPUhTtuTXXa6GmzgtW+u7JRz+hkW3+scTYIgWUD5h4Ddv6IVzP7NxRVX+I1WQeV9ixXSj5WF1cWiidH/xT2AtRP1uniJ/uK64dE+txdv4DFauh0Zz9tRA5iF1d/oKDbMPkQnCwM/7waJ16z0QrxJOYr X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(136003)(346002)(39860400002)(230922051799003)(64100799003)(186009)(82310400011)(451199024)(1800799012)(46966006)(36840700001)(40470700004)(316002)(8676002)(8936002)(4326008)(6636002)(26005)(478600001)(2906002)(40480700001)(40460700003)(70586007)(54906003)(110136005)(70206006)(6666004)(5660300002)(86362001)(83380400001)(36860700001)(1076003)(47076005)(356005)(7636003)(82740400003)(36756003)(2616005)(15650500001)(336012)(66574015)(426003)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:39.5939 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 940319ce-a940-4cfd-2d59-08dc00bdab00 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D5.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB9222 Set vdpa device suspended state on successful suspend. Clear it on successful resume and reset. The state will be locked by the vhost_vdpa mutex. The mutex is taken during suspend, resume and reset in vhost_vdpa_unlocked_ioctl. The exception is vhost_vdpa_open which does a device reset but that should be safe because it can only happen before the other ops. Signed-off-by: Dragos Tatulea Suggested-by: Eugenio Pérez --- drivers/vhost/vdpa.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index b4e8ddf86485..00b4fa8e89f2 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -59,6 +59,7 @@ struct vhost_vdpa { int in_batch; struct vdpa_iova_range range; u32 batch_asid; + bool suspended; }; static DEFINE_IDA(vhost_vdpa_ida); @@ -232,6 +233,8 @@ static int _compat_vdpa_reset(struct vhost_vdpa *v) struct vdpa_device *vdpa = v->vdpa; u32 flags = 0; + v->suspended = false; + if (v->vdev.vqs) { flags |= !vhost_backend_has_feature(v->vdev.vqs[0], VHOST_BACKEND_F_IOTLB_PERSIST) ? @@ -590,11 +593,16 @@ static long vhost_vdpa_suspend(struct vhost_vdpa *v) { struct vdpa_device *vdpa = v->vdpa; const struct vdpa_config_ops *ops = vdpa->config; + int ret; if (!ops->suspend) return -EOPNOTSUPP; - return ops->suspend(vdpa); + ret = ops->suspend(vdpa); + if (!ret) + v->suspended = true; + + return ret; } /* After a successful return of this ioctl the device resumes processing @@ -605,11 +613,16 @@ static long vhost_vdpa_resume(struct vhost_vdpa *v) { struct vdpa_device *vdpa = v->vdpa; const struct vdpa_config_ops *ops = vdpa->config; + int ret; if (!ops->resume) return -EOPNOTSUPP; - return ops->resume(vdpa); + ret = ops->resume(vdpa); + if (!ret) + v->suspended = false; + + return ret; } static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd, From patchwork Tue Dec 19 18:08:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498813 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2060.outbound.protection.outlook.com [40.107.92.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78540405CB; Tue, 19 Dec 2023 18:09:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="mKUth2pU" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FTpCiLdYLfAJ4k6LGSX9tqDSepAMwTYCF8aabYHwMzdwyRLz7UEmVl+7SvE08Ia0b7xPZcZ3ikDtcP/vxSr3MLQJRPun77zWe0ledi2p4mZoJ0W6A/Efq/MCLtOmrJrUjNTEZz3YnyK/5gC/+Wlhmp7gwpKBY+XetjRYPuQqjBATLCTeRV634yUZ56mrBVxgd+R5tTT06uoDzv/n+hglRc07ulCvKN5+B1MrAcTgssAqB1r4+QoFehx0k6apIvfLRF6WaIdIGvjRT74EEGTOtMcn3dae3mEAnUUK6BiRdSFqSlUB6BIyQynCzJ0Rj9XQi759XEVILHiXVNHqm6RrcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ah/KgQ3doi5dp5nhrYCt9Wh2CB6FV9/WRtkHhh0JYIA=; b=D865j/7qjkjlzl6wf5veCa5vrCwHpZVFgEYDKF8O2VB/BekDdO85ZHVJ5EkbbdMI/yY8N2EHoevirTqdBj8y1dCh93yAeHrmLZs5lWR528pEMmOiho00Y2Fd31rMXPPZTRXRTU4Oq1fLYAa37qFm+GSEGIBV3GIOlep6keURIN5cbgMEuAf9J1bsLBVfmOV1uYW6CJQipIz+I2FsO5IunHeyIW2G1n3EdvtSeZVXWtUtSqVnsIbgFpWxNOrjlmhOhRkcOY6pF6+OLEVnH7SxaaQbMLFBDyKIlgoYKfmb/eLL41B5rN+YlcaQ+vH1HOW5xrDtD6fu1DPa/DimomzCyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ah/KgQ3doi5dp5nhrYCt9Wh2CB6FV9/WRtkHhh0JYIA=; b=mKUth2pUzI5SrNV02nwdSirHq2S2ncIQk8HFfKo/nafnyKMzqK7IiZN0EzjjBJl5dPSYVP1+5fo9+VJQrl1xynWSFuxJUGRH7zYmOipo81Epmu45oaA8AqCl8ZOh5281ssaq3BzrBaWAXW5LQRNHDCAq3IuHum9KSprmzzZa67/lCa+vegd7p4U2jfGaCOr4Qu7dI49vVnnOR4zPULIvB2wuV1zbPCrCtef6sZgdG53S5lqX289IW9TeiXlLcL4hftBJU5J2ugrO8FzBpOsln4BlHKA30kCfq9ovkmIiy+Sk9nW6wQdZUekPKjBnN+tLAZ1KiX6jTiaPcuPFWNclvA== Received: from DM6PR03CA0075.namprd03.prod.outlook.com (2603:10b6:5:333::8) by DS0PR12MB7605.namprd12.prod.outlook.com (2603:10b6:8:13d::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18; Tue, 19 Dec 2023 18:09:45 +0000 Received: from DS3PEPF000099D5.namprd04.prod.outlook.com (2603:10b6:5:333:cafe::f4) by DM6PR03CA0075.outlook.office365.com (2603:10b6:5:333::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18 via Frontend Transport; Tue, 19 Dec 2023 18:09:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS3PEPF000099D5.mail.protection.outlook.com (10.167.17.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:45 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:32 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:32 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:29 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 07/15] vdpa: Block vq address change in DRIVER_OK unless device supports it Date: Tue, 19 Dec 2023 20:08:50 +0200 Message-ID: <20231219180858.120898-8-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D5:EE_|DS0PR12MB7605:EE_ X-MS-Office365-Filtering-Correlation-Id: af5971da-9b31-48dd-c75f-08dc00bdae87 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3CiTZoBNujGltv99325bCAMjhNAsNXP5YRnnSZveR8vBDgYC8aiEnnA2VYbG82/smDRCzoRO4jpyrW9pJAdedzFrH6/+2Z80w9teqYXyFrysU7WFU9qjOsC3riXl23xdF2FhINt9YDmA9pFCp3BjqP8jrAbtmUfl0XeT+DDAclwPJsbuua2g4TUkz1QUzLT0wQmr8HUTphnymg7Xwxqvv1/m2jh0DCDoy20TfkdMC5bq2IPXT0fQMLOk0GpGDkfUmud8iMJ74oEQH0GN7wsLpZSt1x3yAT4iLHKrqj/fxwwDt0EfuvHWroppv0EgtYlA3JOo7ho6BnUOwx2NLiBsWclX1EOQftnijHy4vYdkLx9c+8HAZrw67Une+8Lo1s9yhSFyZqwABx8ECYXhiz8blix69G9ov4eR8rZqHzH4jznDZwrjMFWn9c050+bqeEgHhZbM1IwKqyg0oVN6pKhlP0fMwjEeUL5WE2hzusheK/Kq6xtKVGgsdZxJhOgUmU2YM8hvArQg3I22EDpohhLYKnwkkGaYgZw11kw3OprqQmaMplCywQ8ry/vjO+8o/I1bq1RLvV7y+oAz0scwAVvBYeLzDZxFoNZqkydL7Rr+vDG4TyjCWFHUc6T4/e29ICdWWYqOgEtMqzkDg5jG2ucobYQuC36Uyr/rNyjUQDu7G+DG8LuEhEH1qOu/RIabzY6K5L0jT8P9iIWEB4VaQJuzRch1F2mONCLHtNQ9QvTd0h1X+bcrDfsvsDO4OsTcCpR7 X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(136003)(376002)(346002)(39860400002)(230922051799003)(1800799012)(82310400011)(64100799003)(451199024)(186009)(46966006)(40470700004)(36840700001)(36860700001)(2906002)(4326008)(8936002)(47076005)(8676002)(6666004)(6636002)(83380400001)(110136005)(40480700001)(5660300002)(316002)(336012)(66574015)(70206006)(54906003)(70586007)(426003)(40460700003)(7636003)(356005)(82740400003)(2616005)(1076003)(41300700001)(478600001)(26005)(36756003)(86362001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:45.5158 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: af5971da-9b31-48dd-c75f-08dc00bdae87 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D5.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7605 Virtqueue address change during DRIVE_OK is not supported by the virtio standard. Allow this op in DRIVER_OK only for devices that support changing the address during DRIVER_OK if the device is suspended. Signed-off-by: Dragos Tatulea Suggested-by: Eugenio Pérez Acked-by: Eugenio Pérez --- drivers/vhost/vdpa.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 00b4fa8e89f2..6bfa3391935a 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -625,6 +625,29 @@ static long vhost_vdpa_resume(struct vhost_vdpa *v) return ret; } +static bool vhost_vdpa_vq_config_allowed(struct vhost_vdpa *v, unsigned int cmd) +{ + struct vdpa_device *vdpa = v->vdpa; + const struct vdpa_config_ops *ops = vdpa->config; + int feature; + + if (!(ops->get_status(vdpa) & VIRTIO_CONFIG_S_DRIVER_OK)) + return true; + + if (!v->suspended) + return false; + + switch (cmd) { + case VHOST_SET_VRING_ADDR: + feature = VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND; + break; + default: + return false; + } + + return v->vdev.vqs && vhost_backend_has_feature(v->vdev.vqs[0], feature); +} + static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd, void __user *argp) { @@ -703,6 +726,9 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd, switch (cmd) { case VHOST_SET_VRING_ADDR: + if (!vhost_vdpa_vq_config_allowed(v, cmd)) + return -EOPNOTSUPP; + if (ops->set_vq_address(vdpa, idx, (u64)(uintptr_t)vq->desc, (u64)(uintptr_t)vq->avail, From patchwork Tue Dec 19 18:08:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498815 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2082.outbound.protection.outlook.com [40.107.92.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA74641854; Tue, 19 Dec 2023 18:09:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="gTFJds4H" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Q1q1rJlCjCk5Hk3BSbMq4xXPc0pT7EiwiJM0IKPj3FfWjeozSzjmZ7KDYwf3eS16bHp1S7ArTcW5PBljNZUCCssAiRgolkpwWyLXB0B/c+UdZom/eg3veD7g4+/O1iwo2tZsPqIsnqijE8HjkJ+I9aCrN8VOKUZM013CPnXi3Szd/bHzagYEFkLY9nK4JlE5fOX2Fs5O5M9AqbyhXadW+SGYv8DJ7seiFwlTS8rkcnDNYifCG2SwNr3mVjQjdd1RNpP0NGku85QJ3Bst8hYPdBKr7WScU3XrrOgWYIsjEXqR8zM5fkgU8HhJtKefvHv1H7dnbI22K64SAhv1rXLSmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KZRtPKqdWEMRTC6mFKrmvUf27QCWsQ5pVGAeS4l4Y3o=; b=ivmn0niEPfWelRK+upkNqSMNkIZ1AoP11wZ3NiuLrrMa5kp9Pop4cfsHr/susj4GfWUd6vYNgU312oAiCdXNpRuCD6tXJxx94RXHq76S9yvybAH9/IoscVRachPeBNhrTeyZRT+gMyQrIgXDKN2+Unh/lnYtlE84iUy3g+AQj2yM6vV1wI6NtxoTPRMrET6u9cOWuj54ri33/+NJE3hs9ZO1O3auTxVaLPEV+5XqQhXyw/4JUQ9tEopG/1nc+HoVLYqcNsWoH7LsCZt80HGUffUGqmWsze3/eXpR/rUp1/KtQdp18jU/NsLZuViLhRk7foenP3lMoAaMxRk1RNsluw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KZRtPKqdWEMRTC6mFKrmvUf27QCWsQ5pVGAeS4l4Y3o=; b=gTFJds4Hx+V3vNt8c8c9ZVOlzo2CupnBdmFKdo+LWmh/ZrQtq1lvF+EA7QLKpyVB2/mLFklSyxo56seFxx0UhEE6JnSQW3215uyi5XS7fiABLMpNLQHL+jKLu1b20aJn9KJYbOZGVZq2RJ3r3+x4O20EyT6hs/XQ5Ca0YGfWUSXlgush0HVnN4+XQuULJDWC01SAs8sGUDu7OsHdTYIC3eZpp+GxQArg0XGwF2on3fGxVQ/Nub5eh+ozI7r9V8sonZeX0+krF3BKL84PsocydW5C64IcO5js+9pY+Xz0ApyoFgABhdH2/V85optxIPIvm37ZUyeH1KZjT9RPPiChBw== Received: from MN2PR03CA0028.namprd03.prod.outlook.com (2603:10b6:208:23a::33) by PH7PR12MB9152.namprd12.prod.outlook.com (2603:10b6:510:2ec::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.37; Tue, 19 Dec 2023 18:09:54 +0000 Received: from BL6PEPF0001AB73.namprd02.prod.outlook.com (2603:10b6:208:23a:cafe::38) by MN2PR03CA0028.outlook.office365.com (2603:10b6:208:23a::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38 via Frontend Transport; Tue, 19 Dec 2023 18:09:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB73.mail.protection.outlook.com (10.167.242.166) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:53 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:36 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:36 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:32 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 08/15] vdpa: Block vq state change in DRIVER_OK unless device supports it Date: Tue, 19 Dec 2023 20:08:51 +0200 Message-ID: <20231219180858.120898-9-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB73:EE_|PH7PR12MB9152:EE_ X-MS-Office365-Filtering-Correlation-Id: 6aca3a01-01a8-4607-8bdb-08dc00bdb39c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g3pOcKTKa9ycqZnWDiUkVCsFcANALcJeAcFJR/i7hpN8oWGS8qK3boMHr5O/vQFXfdOhr+QutMraSWqJH05sJMFbi+dQXStOon/e7hRifi6y2vYZCfzCaunjtPQ/t34N7T633mjDJkWlojsj5TSWrOUxqWp79GBbjJtm2wlN7Bn1FzgvYzFYKGpXkeLlGxfm2bWu33U9RU4dosnlqlsHfufrtYLWEL29g0ZuUjnnPFTAAzBD4DUd1ZwNrMNcZQGOorQ038YeYRZ/leGGq+fUZL6LkYsbl+Z77OYFYqI26Doww5orANXBZuzRdEJa+UJOGGIsd61MPSdHiHmK5BX4+hN0kg7sgOeISMU9Z4s554vZOyv2EWSr5f4V032l7efyEScdbhVS0qdxmkVLfnBxUjCQZqXosVeQQPdc5mpgrdBZ3l2CAKGsJ7gYFBVBd2vA2VKp6LE2ivK4LcxykGFp14i7pZLnf+uBWpyt06//5kJMzZnXg5V3FNUlpAqZ8Cu0Jv5rX7Kocxvr1y6VdCWQQRy7dy/jnpQatkVb4uSDcDu4z/ydjTKMNTZ01+tlHRhp0NOJXmnFOTdEuguz+3kJ/DA3ZCb5lgn7ggqI10P1YDkGhMe7PlnMZJv9YYNTWJF+owxwQ4bcA25RyaAOl6xubQel4zgUoTD9D+4V556uDQFb8yQO/yTkM9p7xLQcet2y3WUgrYadiOi58IM2TAmITSpvoF4B1pso61SNJcywkBHAYfKnlSMKgO+FrHdLHyqt X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(346002)(136003)(376002)(396003)(230922051799003)(186009)(1800799012)(451199024)(64100799003)(82310400011)(46966006)(40470700004)(36840700001)(40480700001)(478600001)(1076003)(2616005)(6636002)(6666004)(83380400001)(426003)(316002)(66574015)(336012)(110136005)(70586007)(70206006)(54906003)(5660300002)(4326008)(47076005)(8936002)(8676002)(36860700001)(26005)(2906002)(40460700003)(356005)(7636003)(82740400003)(86362001)(41300700001)(36756003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:53.9936 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6aca3a01-01a8-4607-8bdb-08dc00bdb39c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB73.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB9152 Virtqueue state change during DRIVE_OK is not supported by the virtio standard. Allow this op in DRIVER_OK only for devices that support changing the state during DRIVER_OK if the device is suspended. Signed-off-by: Dragos Tatulea Suggested-by: Eugenio Pérez Acked-by: Eugenio Pérez --- drivers/vhost/vdpa.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 6bfa3391935a..77509440c723 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -641,6 +641,9 @@ static bool vhost_vdpa_vq_config_allowed(struct vhost_vdpa *v, unsigned int cmd) case VHOST_SET_VRING_ADDR: feature = VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND; break; + case VHOST_SET_VRING_BASE: + feature = VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND; + break; default: return false; } @@ -737,6 +740,9 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd, break; case VHOST_SET_VRING_BASE: + if (!vhost_vdpa_vq_config_allowed(v, cmd)) + return -EOPNOTSUPP; + if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) { vq_state.packed.last_avail_idx = vq->last_avail_idx & 0x7fff; vq_state.packed.last_avail_counter = !!(vq->last_avail_idx & 0x8000); From patchwork Tue Dec 19 18:08:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498814 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 976534185D; Tue, 19 Dec 2023 18:09:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="nTt1YB6c" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ceA3Erkev07AWpWJZArd/1i2ONQyTjSCrrdkCN2FVbQifZIGRkzvgtsHiU05R9E+s1OZDsPALKWTLX9CbXFj1+oKLJ3Jclc8Ac5EOHDjYjjjC2xSuVaUXQUVBj4DSnKgt7zmlCsxxv6Y5jMJSRqkfW/UwgStwRf9W5CGgCqndOJ2FteSFVyiwPysWW1aVJ6QexKLvy1CUZf9UpQp1/4LVR4td6yB+YUmxqa9LqCJJyzdP799AN4xEswC3+aX6gt9QsCRc8BTuJlgk544fkJteYWZTK0tTC6pif/uQom54cQcerwJ+HrTI4D6DoJmc5wMBhtKSW873dXDjThwf2x26A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZPdT9vtITGHn3MYb+1Dsvi1MMhbs1EU5ai/kuf3a03Q=; b=d7tbeYUDmmapNOhKHf7p3uu2rdGMWCdWs2TQuhM0B7rqxYFxhj+qvuz12H9VdET5eSTity41jVbKK1YbtxcwP71Xb48seCwhJCTSuo7TFlE8ORG8uNo7nGfJZ0qnrGRc5RFOIUABCxwGTWTcOZ5quM/PV1Nc8AnAaYU7kKis78aD8532pvhRlKHRZnQelZJV2Qdd8ZD3/FRAE+BhuZ8i7Qlx7lpcUNSym7F/Xe+5XCGsC0CDwNFOF90FidwLbZ2XZUDvJDn8Lu0PiLl0/8BIycCsj6RiwAdEqUIwJxwiGF8tz47b3HdLXmafIbzNXjW7kD5O8xglH1ytGC7Jw6aM1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZPdT9vtITGHn3MYb+1Dsvi1MMhbs1EU5ai/kuf3a03Q=; b=nTt1YB6ch51WgxpNPbRlgFteiAPqyzmX53klER0HM+C3AgIbtkCguf3Ill5mjRDGtKvGMdQeqJ18pabqcR7khuBlLJuxJU4LFupNnFwEjNWi/o0Ez/vmnWB1WOYbHAdfLeodiDra9i8T2Q28rT+1mZbR8d/FGtAG5O8iiW/rDxkGU1YF7iRopZL2dBkJbjxHPh2UN1gIw4PhtSRUFj4HhTxpmvWpcG5W6SUNkneq3PmDkzfA5bu/GCJEut78Ni+1brHJBpGkNbxPKBmRZAmiw5AmOl2VwQfnA2dCM1PLz5JTCVFoqZnp8B5IWP3XP1pJ42JQ7rUyBkeXXEcHAMU2mQ== Received: from CH0PR13CA0053.namprd13.prod.outlook.com (2603:10b6:610:b2::28) by IA0PR12MB8424.namprd12.prod.outlook.com (2603:10b6:208:40c::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18; Tue, 19 Dec 2023 18:09:53 +0000 Received: from DS3PEPF000099D8.namprd04.prod.outlook.com (2603:10b6:610:b2:cafe::5c) by CH0PR13CA0053.outlook.office365.com (2603:10b6:610:b2::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.17 via Frontend Transport; Tue, 19 Dec 2023 18:09:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS3PEPF000099D8.mail.protection.outlook.com (10.167.17.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:53 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:40 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:39 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:36 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 09/15] vdpa/mlx5: Allow modifying multiple vq fields in one modify command Date: Tue, 19 Dec 2023 20:08:52 +0200 Message-ID: <20231219180858.120898-10-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D8:EE_|IA0PR12MB8424:EE_ X-MS-Office365-Filtering-Correlation-Id: 591a0248-5841-4061-9987-08dc00bdb340 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RT7UreJ5nnXh9NCEk6grvdxtov1WZh4Nh1/s7zWGmNfqcHPHKha8Fcq4ZM65jPA2t8bNSPPWp4fCe9+NmbmRswmnvFjW59upOBXXnLjTQxquVNL3wln5/hF4JAk1usDJuSrvisB1afvweRFgIoBjj+kT8Pmp+JMdkpKf/KEBkOO7ekjxcCzPhi4uhXQ8XKTDmLX7A3DuiIZy8dGMpFiDXUYmn4lBlOoV3+xezSYquyvJkCyq4n6TfGfDy9Dp5Vwe+XP3zRqD6zbeSQzLvn1j7SbZpRjPAKJSldrNUqJnbR4SOU/7uofiu5+R6W/Ywkh4oKmqMAQsXvo/4NhvLuUgtNGn61LL/lVU1sBFqm2eO2Ky2IueIyjI1dp/NUDCNW++c1z3t+0KsXsz7UAbjN47yH3SuQQJFtYmjFrtkiQp949wVtZnHOgExw45ttkrH0TOvMz/1ziZQWpsghae0SXj9QiYVPihdje25Pp0gmspGCSM+La3kQodOVZ/UFxKVWir5s4wNqhhYLCCBGNdIcokUvFbPW3El6r+0e4o3KTb/u0LrYGo+/bRXHgJuS4e4NO8tICLvQLbvEYf3gea641Hk3gPa8qEFo1HWabJbRbFIVPt9vcJF/zSdv9cfYToukdlwv5is33GSu8mZBvpacYMz5yzgHi3tjpTg8cbyHyt5FPDq+fisva6xJN1UKA/zPB0dNDptHlHs20F44P/qjY6j+Py4QJnuQr7VKi2fDCDSuPm2wVnbm4k4+ZdNYK9y8Gg X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(376002)(396003)(136003)(39860400002)(230922051799003)(82310400011)(1800799012)(64100799003)(451199024)(186009)(40470700004)(36840700001)(46966006)(86362001)(356005)(40480700001)(26005)(5660300002)(6636002)(478600001)(40460700003)(110136005)(54906003)(70586007)(70206006)(82740400003)(6666004)(7636003)(8676002)(1076003)(4326008)(316002)(2616005)(66899024)(41300700001)(66574015)(47076005)(2906002)(83380400001)(36756003)(426003)(36860700001)(8936002)(336012);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:53.4379 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 591a0248-5841-4061-9987-08dc00bdb340 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D8.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8424 Add a bitmask variable that tracks hw vq field changes that are supposed to be modified on next hw vq change command. This will be useful to set multiple vq fields when resuming the vq. Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 48 +++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 26ba7da6b410..1e08a8805640 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -120,6 +120,9 @@ struct mlx5_vdpa_virtqueue { u16 avail_idx; u16 used_idx; int fw_state; + + u64 modified_fields; + struct msi_map map; /* keep last in the struct */ @@ -1181,7 +1184,19 @@ static bool is_valid_state_change(int oldstate, int newstate) } } -static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq, int state) +static bool modifiable_virtqueue_fields(struct mlx5_vdpa_virtqueue *mvq) +{ + /* Only state is always modifiable */ + if (mvq->modified_fields & ~MLX5_VIRTQ_MODIFY_MASK_STATE) + return mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_STATE_INIT || + mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND; + + return true; +} + +static int modify_virtqueue(struct mlx5_vdpa_net *ndev, + struct mlx5_vdpa_virtqueue *mvq, + int state) { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; @@ -1193,6 +1208,9 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque if (mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_NONE) return 0; + if (!modifiable_virtqueue_fields(mvq)) + return -EINVAL; + if (!is_valid_state_change(mvq->fw_state, state)) return -EINVAL; @@ -1208,17 +1226,28 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); - MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, - MLX5_VIRTQ_MODIFY_MASK_STATE); - MLX5_SET(virtio_net_q_object, obj_context, state, state); + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) + MLX5_SET(virtio_net_q_object, obj_context, state, state); + + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); kfree(in); if (!err) mvq->fw_state = state; + mvq->modified_fields = 0; + return err; } +static int modify_virtqueue_state(struct mlx5_vdpa_net *ndev, + struct mlx5_vdpa_virtqueue *mvq, + unsigned int state) +{ + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_STATE; + return modify_virtqueue(ndev, mvq, state); +} + static int counter_set_alloc(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) { u32 in[MLX5_ST_SZ_DW(create_virtio_q_counters_in)] = {}; @@ -1347,7 +1376,7 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) goto err_vq; if (mvq->ready) { - err = modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); + err = modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); if (err) { mlx5_vdpa_warn(&ndev->mvdev, "failed to modify to ready vq idx %d(%d)\n", idx, err); @@ -1382,7 +1411,7 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *m if (mvq->fw_state != MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY) return; - if (modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND)) + if (modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND)) mlx5_vdpa_warn(&ndev->mvdev, "modify to suspend failed\n"); if (query_virtqueue(ndev, mvq, &attr)) { @@ -1407,6 +1436,7 @@ static void teardown_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue * return; suspend_vq(ndev, mvq); + mvq->modified_fields = 0; destroy_virtqueue(ndev, mvq); dealloc_vector(ndev, mvq); counter_set_dealloc(ndev, mvq); @@ -2207,7 +2237,7 @@ static void mlx5_vdpa_set_vq_ready(struct vdpa_device *vdev, u16 idx, bool ready if (!ready) { suspend_vq(ndev, mvq); } else { - err = modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); + err = modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); if (err) { mlx5_vdpa_warn(mvdev, "modify VQ %d to ready failed (%d)\n", idx, err); ready = false; @@ -2804,8 +2834,10 @@ static void clear_vqs_ready(struct mlx5_vdpa_net *ndev) { int i; - for (i = 0; i < ndev->mvdev.max_vqs; i++) + for (i = 0; i < ndev->mvdev.max_vqs; i++) { ndev->vqs[i].ready = false; + ndev->vqs[i].modified_fields = 0; + } ndev->mvdev.cvq.ready = false; } From patchwork Tue Dec 19 18:08:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498816 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2040.outbound.protection.outlook.com [40.107.93.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1127A41C8F; Tue, 19 Dec 2023 18:09:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="MbuDuNi2" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RQXz++aF/+Bv0tt0QKW9AKpsHN7lEwRWXOpAa5tbeGbSseK1OrongwickP9JmtUZvLAHehYpqOynmdLn7UD7+P5WaZ8VYKa4cjSP9jdV8RpuP3iFnWEb7Wak7IZaxDSPUl6iNi/pB190UG8MjWK+Qxi7EKr4/MjzRQtVhlvgbAbH5l9u+A09geBfGyMYEIGB8jY6Iv0Uv1pujZ7ecPVbXuwXihK4sx3ad82cW1b/dRNt5qKXcqjgO+qldNhhnIJY3vt5rIFoi7Ls9r7lD6YtTWD5MV0TnEaiVJ8nTQB9te37R4shiqdZb9EObNft7lvIrXL6OjvxKsrj0s7xWp1grQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pWpOeWI+R1cb4FK6LjTB81MteAujPvP3wVhkPCOcOb0=; b=nOc1gI+VXx5xa0iG0jGYtj+Yo9s46kC+GmA0LVdzUjSFSfENzhWhedbcPLkwfJ77X2NE98NCJDhc2Xp2oqG2t9NgtLj7A3gRK7sHbpXXuZn7typevSgWPFR+flRklHkZq+C9gSodw/eZDBiSRQgkYf3NEu4vDz+eaxGEJfw8O6Vl1mSryJkZ829oi/EV31f/dg3WjnBmhC3k+EFR3X5URdUhWXLoVGR6i4zeeiOzP2gnLz80Ya/g8SNESzoRHhDjq2Q4dGxALiA8BqCh2V0GBTHH2qAETzxPu9F/hrLR7++SFZ5aChQg0SDwxwLdi/K+h2sasVi6MaxEFU2UNH7IPg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pWpOeWI+R1cb4FK6LjTB81MteAujPvP3wVhkPCOcOb0=; b=MbuDuNi2jAIGRXjcsxnWnQuqHmPSBiqxU4UhDRu83xbgLCJZJBqSV1Qx/wX/xLNpyjNPHeL9TIOQ5mdAumL4VeJrKuivLBY8cPYQCrdsHZ41oSunp/P8NkT5tWdU58GhO0VRw7iF/wZef0A5ir4hRfPgMt9QamTAPhXmxMZqVNPPhqDRO4CAUKm00sqat2kvUDBbGA38DQKEDtZ9X9G/yMQQjeey/3RNV0JEfvxZsqv+oDwGYNAFmx/BS2enfEUndNvqz39P2F0nxVvmybm6BQ3c4dVzdIcRNCzx9R2swikgEpRC6WGi+wfk8uZENAXQ5amuZa0Sq1kou7zHyMMp3g== Received: from CH0P221CA0014.NAMP221.PROD.OUTLOOK.COM (2603:10b6:610:11c::12) by SN7PR12MB8101.namprd12.prod.outlook.com (2603:10b6:806:321::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18; Tue, 19 Dec 2023 18:09:57 +0000 Received: from DS3PEPF000099D3.namprd04.prod.outlook.com (2603:10b6:610:11c:cafe::24) by CH0P221CA0014.outlook.office365.com (2603:10b6:610:11c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18 via Frontend Transport; Tue, 19 Dec 2023 18:09:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS3PEPF000099D3.mail.protection.outlook.com (10.167.17.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:09:56 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:44 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:43 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:40 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 10/15] vdpa/mlx5: Introduce per vq and device resume Date: Tue, 19 Dec 2023 20:08:53 +0200 Message-ID: <20231219180858.120898-11-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D3:EE_|SN7PR12MB8101:EE_ X-MS-Office365-Filtering-Correlation-Id: 03a1d461-438f-4908-5a43-08dc00bdb54e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NNqX+JG2vcYilVMZ2GMv09CeRLzI+XFL4hyMykRyXa3jJiBBGgntE1K6dOg2VVI06NgCBYCiK4mZsapwq1eUfoQb4aTfWxJaqvUbDYsVcFCd/6QT7x+TxeEpOaLW+B6TH1ZTPF6C77lGuACrXax+3i84ZGdANWE0bc3ef0uxI8v8gAmvDbxyvPCpO49gqBZSSUqMpHiiqGtPD4XWLHb87XMUtOcMQt9NJZ/fK/827UnmcvNvzKOmJOzcaJGKLYS12HHLpsj8eMD2+K8A2/C2LnVjZymKkrwcY2oSRaC7dZ5RWyHTIr5GPjjScPcq4qVB6rl2LUbRHqFEcQ2QT3ZljBSJAWKuJbQ7HUcIR0gfTrJ+bSZOmHawy6QRKDDvBzuwNtf7v2FWJAgXLIHTm6VQEHkUl2SCKM8YHjcjLyG7Aj3CiXwNP4UyoCXT2JzqdNWRXQdL00wZstbs3pKnFv50LnHHkUB2oTKvxfWG0RU19ps0/bsJN1+9VLq1LINXFCiXKbgnNL7qWj8CfrL8+Ud0C6HUrrgt3nBdE5ut9KXC9elGMZ10Aes0jbWIpHE+gtf6X2t4LIGTY+7MKv8xhWh658Xzbpn6hRx/mSt7H3gbdhB8hcGCRIMvnd2WrlaEyGROSXvcmSIFTtYIqR32YJl+GV29uLYiSNJ1SZ/P67UDaQQqfDtYvZlEXnce6Owr939RyQ1DGq20Z+6tePpzvwE4qIJFDvGIV+6KNVho2j0wAcTXbEzan5FLNBUfhKv1rsBs X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(376002)(39860400002)(396003)(346002)(230922051799003)(64100799003)(186009)(451199024)(1800799012)(82310400011)(36840700001)(40470700004)(46966006)(40480700001)(36860700001)(356005)(7636003)(47076005)(1076003)(26005)(40460700003)(66574015)(336012)(426003)(83380400001)(2616005)(86362001)(36756003)(82740400003)(478600001)(6666004)(6636002)(316002)(54906003)(110136005)(70586007)(70206006)(4326008)(8936002)(8676002)(2906002)(5660300002)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:09:56.8835 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 03a1d461-438f-4908-5a43-08dc00bdb54e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D3.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8101 Implement vdpa vq and device resume if capability detected. Add support for suspend -> ready state change. Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 69 +++++++++++++++++++++++++++---- 1 file changed, 62 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 1e08a8805640..f8f088cced50 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1170,7 +1170,12 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueu return err; } -static bool is_valid_state_change(int oldstate, int newstate) +static bool is_resumable(struct mlx5_vdpa_net *ndev) +{ + return ndev->mvdev.vdev.config->resume; +} + +static bool is_valid_state_change(int oldstate, int newstate, bool resumable) { switch (oldstate) { case MLX5_VIRTIO_NET_Q_OBJECT_STATE_INIT: @@ -1178,6 +1183,7 @@ static bool is_valid_state_change(int oldstate, int newstate) case MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY: return newstate == MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND; case MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND: + return resumable ? newstate == MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY : false; case MLX5_VIRTIO_NET_Q_OBJECT_STATE_ERR: default: return false; @@ -1200,6 +1206,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; + bool state_change = false; void *obj_context; void *cmd_hdr; void *in; @@ -1211,9 +1218,6 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, if (!modifiable_virtqueue_fields(mvq)) return -EINVAL; - if (!is_valid_state_change(mvq->fw_state, state)) - return -EINVAL; - in = kzalloc(inlen, GFP_KERNEL); if (!in) return -ENOMEM; @@ -1226,17 +1230,29 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); - if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) { + if (!is_valid_state_change(mvq->fw_state, state, is_resumable(ndev))) { + err = -EINVAL; + goto done; + } + MLX5_SET(virtio_net_q_object, obj_context, state, state); + state_change = true; + } MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); - kfree(in); - if (!err) + if (err) + goto done; + + if (state_change) mvq->fw_state = state; mvq->modified_fields = 0; +done: + kfree(in); return err; } @@ -1430,6 +1446,24 @@ static void suspend_vqs(struct mlx5_vdpa_net *ndev) suspend_vq(ndev, &ndev->vqs[i]); } +static void resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) +{ + if (!mvq->initialized || !is_resumable(ndev)) + return; + + if (mvq->fw_state != MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND) + return; + + if (modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY)) + mlx5_vdpa_warn(&ndev->mvdev, "modify to resume failed for vq %u\n", mvq->index); +} + +static void resume_vqs(struct mlx5_vdpa_net *ndev) +{ + for (int i = 0; i < ndev->mvdev.max_vqs; i++) + resume_vq(ndev, &ndev->vqs[i]); +} + static void teardown_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) { if (!mvq->initialized) @@ -3261,6 +3295,23 @@ static int mlx5_vdpa_suspend(struct vdpa_device *vdev) return 0; } +static int mlx5_vdpa_resume(struct vdpa_device *vdev) +{ + struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); + struct mlx5_vdpa_net *ndev; + + ndev = to_mlx5_vdpa_ndev(mvdev); + + mlx5_vdpa_info(mvdev, "resuming device\n"); + + down_write(&ndev->reslock); + mvdev->suspended = false; + resume_vqs(ndev); + register_link_notifier(ndev); + up_write(&ndev->reslock); + return 0; +} + static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, unsigned int asid) { @@ -3317,6 +3368,7 @@ static const struct vdpa_config_ops mlx5_vdpa_ops = { .get_vq_dma_dev = mlx5_get_vq_dma_dev, .free = mlx5_vdpa_free, .suspend = mlx5_vdpa_suspend, + .resume = mlx5_vdpa_resume, /* Op disabled if not supported. */ }; static int query_mtu(struct mlx5_core_dev *mdev, u16 *mtu) @@ -3688,6 +3740,9 @@ static int mlx5v_probe(struct auxiliary_device *adev, if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, desc_group_mkey_supported)) mgtdev->vdpa_ops.get_vq_desc_group = NULL; + if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, freeze_to_rdy_supported)) + mgtdev->vdpa_ops.resume = NULL; + err = vdpa_mgmtdev_register(&mgtdev->mgtdev); if (err) goto reg_err; From patchwork Tue Dec 19 18:08:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498817 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2073.outbound.protection.outlook.com [40.107.223.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8455547F5F; Tue, 19 Dec 2023 18:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="tH50IN93" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bdw+EI1c+lT/xR2jEst/Qj85XAPGqRptvyDbMR5gE9VQshSXs+b4T9tM6Jvf/eQXJnFNcTgjyddGhSB1q3XXVpJFXNIdL2DD3NZ/ZtZwphoAaoK/xg783aEPdlxzHjf+Iehbv+bRz8GCcfNmKX8EA3XMskxaJiNKn5i+e8xbpip7DpTcBoUEZ14CoKXEJZSa6Oc9WOzBoKDlKXqzpD0+C01Dh0LzZPDxEkPEB4X1Z++Y3ArTBmPTKTfGaXXzk0wQYtyiSed1i1RSi/1g7HfqwtA/axcq94HTyf6RWpNePghXqxMsovHiQdO+1S3yYLS42KSrEpLAAwoj/3QLhJstHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bUG4piYXpvfVFnQYkmolDKFtW4yjyp7QBkHdKBaJgEI=; b=ALRr/4+ele59JgnJVP7By0sffJHrL7UuwWLAwLIWy6OPMqvJuDM07tgXhKaSglLwptpTl1f7XCGqNIspbjvUZbkAK+sB6A9SaWZOlCo1HUj9i416oVd7urOY/0UNHNglZWmgJiicUroA+5tfQezKknU4Z6HfUofzg7Legl8eOmIh2El04nHcQzMTydH5mMxzpV5RxuEvIXL58BHjhFwYMLLw47aIWXXhQLkRityxJOkR4s1qvRMmzVzhcALkDiv1sgEhgwFQBOpkr8Aau6CRFB0z8T9Ecjyml1VJXGYg9CZBLojkrH5APQz6u/Z7YtYJSK73MqEk/qWjgs+G2PvAqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bUG4piYXpvfVFnQYkmolDKFtW4yjyp7QBkHdKBaJgEI=; b=tH50IN93fo2ERl9GRNzotnr4Tbx+E35bf2TmuUZMqKyt5jwSz4Gle2nmSIGXBRSXfOJbrA8AoHkwhqwfzt1XgnVTCNA4t/RRUO4XuoRTZKeGLTL2XkksRLFWJ5Gz5C/r6lq9gVKMar2a4cU4hs7fL2CgZcNQ4Mw691arrrejDZoLNFh5MA0g0PoVxgWUX0Dx6p+D81kCCmgRdZ4ZPwErpsbbIcdkXMfrk64+hjQyStgOAXx7LnILcSCS+j2dJunTxXvGjmFYgR8WgOBXdyFAxws59gh5AzvHdMe0rjbdxXwncp+HgFx8Tj2cEtTqqdZ6IvIyw3U/BCAeHTMI9EYaWw== Received: from BLAPR05CA0038.namprd05.prod.outlook.com (2603:10b6:208:335::19) by PH7PR12MB7210.namprd12.prod.outlook.com (2603:10b6:510:205::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38; Tue, 19 Dec 2023 18:10:02 +0000 Received: from BL6PEPF0001AB76.namprd02.prod.outlook.com (2603:10b6:208:335:cafe::e) by BLAPR05CA0038.outlook.office365.com (2603:10b6:208:335::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18 via Frontend Transport; Tue, 19 Dec 2023 18:10:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB76.mail.protection.outlook.com (10.167.242.169) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:10:01 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:47 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:47 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:44 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 11/15] vdpa/mlx5: Mark vq addrs for modification in hw vq Date: Tue, 19 Dec 2023 20:08:54 +0200 Message-ID: <20231219180858.120898-12-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB76:EE_|PH7PR12MB7210:EE_ X-MS-Office365-Filtering-Correlation-Id: d4d39210-0d25-407a-8f66-08dc00bdb84b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7SVdkiy75Oe/OiyYr0v2HQEdJORfVlrthecj0cwS5aoE60+d07YAa09g7h5ke17i80uOwQDfprT6DAY99ZbAwwcFILhtZDod4FB0DOFj2WGyY4XPNk+23cbo+PtOwsGUMlm4p0loZDoNrfmv8vOdVaobBtSpTak5AGAypq5avZuo2fX3ZOQH+py/+74tyUalOdl9pXSPQVh+78/In7shov9NCR+jca6wvvZpK9Bs/RD+J2O13LzR49j8xkZB5R9Jmel/ns5hRDfslXC8BR6IdYYIe3ip8gErtHISMT7j7yDpaicTmfaTcfILYwzD5/fsXuMjW9jHtAaz7+GbQx6IoNTVM35andkT07xu3Seox1I/yZK1axYacAJXRR3suACJRQfMgTdLN6GjrrPxD21TOoexGKvq+dt4UO2pGF+FJXSy5uOdbwrNDlMhrFbKLhIBbnFmWqpzPyojtuNa3BJz7Eb/PxfBN1OPRjel+psr9BxfRdbRnEYuDE0CV2GvesabrU6vAi3ktkd35mk2gDROLGpgxUC3G4rSh6yVWUzwoElj32sHOm6DDXN3WOWSYiX0e+huJarEJ+FU/Z8DToU5c30UktU8lnfzFlVwKtuxXcgiAthaBqW2pFa5692XIS75Xv3puS+A+IiyymEFjVlZpgG6W6/+3cwLWJtTHPQYfj/Fv5IGOxFaW+tSktuduYstiRqjwm0HSL/2jd974tJp7wustrMepLrpjfE0mURBbFQ5GFANTzZobLFGb3/45lCW X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(346002)(39860400002)(136003)(230922051799003)(186009)(1800799012)(451199024)(64100799003)(82310400011)(36840700001)(46966006)(40470700004)(36860700001)(110136005)(2616005)(70586007)(6636002)(478600001)(54906003)(70206006)(316002)(40480700001)(83380400001)(336012)(26005)(1076003)(426003)(40460700003)(47076005)(8676002)(8936002)(4326008)(6666004)(5660300002)(2906002)(86362001)(356005)(82740400003)(7636003)(36756003)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:10:01.8671 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d4d39210-0d25-407a-8f66-08dc00bdb84b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB76.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7210 Addresses get set by .set_vq_address. hw vq addresses will be updated on next modify_virtqueue. Advertise that the device supports changing the vq addresses when device is in DRIVER_OK state and suspended. Reviewed-by: Gal Pressman Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 17 ++++++++++++++++- include/linux/mlx5/mlx5_ifc_vdpa.h | 1 + 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index f8f088cced50..93812683c88c 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1209,6 +1209,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, bool state_change = false; void *obj_context; void *cmd_hdr; + void *vq_ctx; void *in; int err; @@ -1230,6 +1231,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); + vq_ctx = MLX5_ADDR_OF(virtio_net_q_object, obj_context, virtio_q_context); if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) { if (!is_valid_state_change(mvq->fw_state, state, is_resumable(ndev))) { @@ -1241,6 +1243,12 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, state_change = true; } + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS) { + MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr); + MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); + MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); + } + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2202,6 +2210,7 @@ static int mlx5_vdpa_set_vq_address(struct vdpa_device *vdev, u16 idx, u64 desc_ mvq->desc_addr = desc_area; mvq->device_addr = device_area; mvq->driver_addr = driver_area; + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS; return 0; } @@ -2626,7 +2635,13 @@ static void unregister_link_notifier(struct mlx5_vdpa_net *ndev) static u64 mlx5_vdpa_get_backend_features(const struct vdpa_device *vdpa) { - return BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK); + u64 features = BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK); + struct mlx5_vdpa_dev *mvdev = to_mvdev(vdpa); + + if (MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, freeze_to_rdy_supported)) + features |= BIT_ULL(VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND); + + return features; } static int mlx5_vdpa_set_driver_features(struct vdpa_device *vdev, u64 features) diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index b86d51a855f6..9594ac405740 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -145,6 +145,7 @@ enum { MLX5_VIRTQ_MODIFY_MASK_STATE = (u64)1 << 0, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS = (u64)1 << 3, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; From patchwork Tue Dec 19 18:08:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498818 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2069.outbound.protection.outlook.com [40.107.94.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 175584B144; Tue, 19 Dec 2023 18:10:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="NH/fne/4" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fCCK32o300sc3ZoJRATCagHIXzCXwWmzZjl5ctvA1Vc5HcnYEoU6LlJkPuD2EAc5q5EH5D1o5FGs5LlUdZ09RcK9IbsBydD9xwfxggVcx6Gr+wVrR20O8DpJhRro5NpuMYoJt+UloDLbt0x02wbJDsrTjcQ4bSr4Kom65U2Sa4BBbK5RuGaI8jc9FuUthY886wL1OSgwl69pX1Jonej+fyeWJK5CFAkLHROiJ6443HzYSFH4LOCUVZmFWSyFB7ZYhz9dE5RdXTkVzig6+O9ELMAYrzsgWpCCFQR3R39TBgo/xKKeOlkRUsF0SSyUdcXwQm6ihqcFbZFCIW2m3MUpSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DIOWw8hu9FilT3R8SChRoV6sn+1Gou8Ky7fnsBKdc4g=; b=Efj4nPJnZdx+E+xx28ZFR43AHSo2LY5ZFhJmNnyxbtfHtjubMznuQ/CpjNlTq75JTxQN5aA8KT5syBmhlo2PA4FAmT36EzlXswFmfE32oVSewLQVpySsGqfg0WsfhfWM7VOb9wOAQ4sFydd0HU+44XQ8nkA4Mt0nkmya+ABKcx5jjZQ6f9w75Pfg9YN0o2DyBXJ/N4bFEsRj0DFjjiZQt34aKKSnsU81xYa6JjFpPJ/mbepu4NLwj+Rs5MzLSzYr9ezTz4kLi+MIXc5O8Jb1rkaEf3AszFInx8Sz/lRTxlsor2vl+LkrnChDRMIc/jDOduDHAqQbEXkLJsZfaif6lg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DIOWw8hu9FilT3R8SChRoV6sn+1Gou8Ky7fnsBKdc4g=; b=NH/fne/4TmugSJAYxqgyUAIwaWePYxb9IKJLRGhyFJIgnNWWTh/PRXJ3fYQXFg9oLC0Kx8XBqZzcxwrg3IiGAzu9Qre4unI3L0YJaCUPyd3ZIeLqBcb2wwjyJkeHI5sxQXtjYpFfl4xfHtfVIs/xsnB391uMI8s4KTZZuc0R5i02LSteOb42F12PLiTLsNhItU96eoS8oG0Gt2P0dCmN81EkWxZRpTG/CLdG48S2n9FTIyQ1/Jdss6RUdmgOALJq30gS7iNaBvBRSnmycFPh4NRFASVTg8x+F9pg/NXluhmKTGgN7NmYCuRXLqkfhUwHRiiBWSRL52sysK7ZAdXaDg== Received: from MN2PR03CA0015.namprd03.prod.outlook.com (2603:10b6:208:23a::20) by DM6PR12MB4330.namprd12.prod.outlook.com (2603:10b6:5:21d::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38; Tue, 19 Dec 2023 18:10:06 +0000 Received: from BL6PEPF0001AB73.namprd02.prod.outlook.com (2603:10b6:208:23a:cafe::95) by MN2PR03CA0015.outlook.office365.com (2603:10b6:208:23a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18 via Frontend Transport; Tue, 19 Dec 2023 18:10:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB73.mail.protection.outlook.com (10.167.242.166) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:10:05 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:51 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:51 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:48 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 12/15] vdpa/mlx5: Mark vq state for modification in hw vq Date: Tue, 19 Dec 2023 20:08:55 +0200 Message-ID: <20231219180858.120898-13-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB73:EE_|DM6PR12MB4330:EE_ X-MS-Office365-Filtering-Correlation-Id: 0d155b3b-ea96-4e67-9c25-08dc00bdbaa4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PAlMPqAUnOKZWNvSUaABNZO48i6Qh0aq5hM+vQrw7pvq53ZSaY9WZRPWyk2PDPu6PLXrBYswHR+HkSKZF4OzFwvWwRTKVvN6fpy5NlqyhkWd4qD2KOzrQOiSLvraodCkfydBuSiuQpTmdF/gHctXW2jLwDbKi9dlg5aqGBzK/RCTGk3ymBT4emyVyqokreG6eUvxwW/u0NngoK/dvGvJNmilegaHSnZlBMPXeNOteGEwjCIbDjOxw+CksO4CePImEXM8JVHEPa1+ix+4zeee4q19F6NOYUJAephLHstlFjeY2D4kJou09QxVv6C2w6zhcjhPZOt5S9Rjfb5z7cDRZnjQPYoG5iSPUAzuxZdENtGbqYexvfqX4GUx6dqQK2hbmgzhpsQTEfCR4eRf75AmmNRvIHBNyidCncGqyr1YeHs1ExcCedHPIiQzh98RiObN3kFRt7zddgSLxsHKfjgFmVOhN5g0bH3wtKb0RHzMjrR7MIVjUwa7EZBmRVepc6YK/jyG200ldyuWvpTZg2hEG2U8xGYyBWlmc8teN2L5NjxLSshhjUvl7QJfum1/SdGg3hlDXHIrUlxR+kU6/y8005Uzdz7KpHOjaK0FYjkI7bZOXdwgUT+Oot+84nXfu92vRS9/IaDeHLJ+9zm/qYUxpLsI/51ekjCTX9Eya2W6+OJLQZFVllPQRUN1fqFiY3WG/aV8o5q6ORqo/BTSUffwIDy2DmeiNgQ15NQSKVqoVbjjy+fU+vUzpn/yxeOKeQQA X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(346002)(39860400002)(396003)(376002)(230922051799003)(1800799012)(186009)(82310400011)(451199024)(64100799003)(46966006)(40470700004)(36840700001)(7636003)(40460700003)(356005)(40480700001)(82740400003)(86362001)(6636002)(54906003)(36860700001)(70206006)(110136005)(70586007)(316002)(83380400001)(2906002)(336012)(2616005)(426003)(8676002)(8936002)(1076003)(4326008)(478600001)(6666004)(26005)(5660300002)(36756003)(41300700001)(47076005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:10:05.8061 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0d155b3b-ea96-4e67-9c25-08dc00bdbaa4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB73.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4330 .set_vq_state will set the indices and mark the fields to be modified in the hw vq. Advertise that the device supports changing the vq state when the device is in DRIVER_OK state and suspended. Reviewed-by: Gal Pressman Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++++++- include/linux/mlx5/mlx5_ifc_vdpa.h | 2 ++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 93812683c88c..b760005e2920 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1249,6 +1249,12 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); } + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX) + MLX5_SET(virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX) + MLX5_SET(virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx); + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2328,6 +2334,8 @@ static int mlx5_vdpa_set_vq_state(struct vdpa_device *vdev, u16 idx, mvq->used_idx = state->split.avail_index; mvq->avail_idx = state->split.avail_index; + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX | + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX; return 0; } @@ -2639,7 +2647,8 @@ static u64 mlx5_vdpa_get_backend_features(const struct vdpa_device *vdpa) struct mlx5_vdpa_dev *mvdev = to_mvdev(vdpa); if (MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, freeze_to_rdy_supported)) - features |= BIT_ULL(VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND); + features |= BIT_ULL(VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND) | + BIT_ULL(VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND); return features; } diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index 9594ac405740..32e712106e68 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -146,6 +146,8 @@ enum { MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS = (u64)1 << 3, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX = (u64)1 << 7, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX = (u64)1 << 8, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; From patchwork Tue Dec 19 18:08:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498819 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2065.outbound.protection.outlook.com [40.107.237.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADBC84B5BB; Tue, 19 Dec 2023 18:10:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="FEIkv8lr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=a0b89nPKlF51vUMBgEAwHtnl9iPM8huZ61z4ZUv/JRut0suuZoa4GPRUW5hfPwVnAJgjNQAZx2yQlLPZdqb8Z189pXHicBv6Fhc8pOqtPU2Yc7HGMp7nIc6x2pj0M6U55jL06u6WHHFx2c/WnLXwSYeCL+UGLcfmtQSuMbIMy9cx2j+ie0KqVu3eSozQVLZJEb4Z6nILflE9Nlcr/AxiJ/TdgPqDxso4GrFmhRGnKL2ZLQnAoNnRWkXLPiG7VxWGKy1UZ9IKKD/ZjNDbBTSYKPFVdRmT7z4lNt0VF5qSrIvHC6Y3KUtvSbEdbTfw8DA2HBmZh/XI8bJ1isHs+NGntQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qWsw/BvKJBmDtMxN1GM40Dr5OaaudI4NDI0vMxWYfSI=; b=U4ldRSMv0gCE0zKrL+dB9Stf9oon59eZBxgYOayClupvhtE3/2dxPJ3xB82vMaYCZupOPR1vv7Y3/7mUvXlAZ/A1TCy2uxlxJP8CtVA42jhX+cr4dSGcfHipoG0WaJVhegIQBeGzwPFZcp7b78NDYPDCqEo/fWe8kcW5R613FVZs4JnjB+r3dwywI8KRuy3CLiVGuZbI/owC+Df3dgBsAs213AGo8zLdyfHkb5FTO5dC7yf18Bhl5pCUXBeOelW9ISOzFEaHsxKjB407mtUjUETLdez2QLNqPiWImN+gN9IkonX4nBpNjw/Ur4U67SMuV2rfiSZhQU9F6PF57S9ESQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qWsw/BvKJBmDtMxN1GM40Dr5OaaudI4NDI0vMxWYfSI=; b=FEIkv8lrkRGU86QwIxWcjl0vubBEwT7gOVX7LcU7zuO8OH+E93+Nin8Wn3GxXHKrk9s0Nt9vf/rQ/quGvXVSpHVdICW531E8TC+IYlqx7krHCFj32VSWvguNhIpzgh9JLdxW4auFf4bqkg9oba9qjcmdOGo98KUK59pwH9X9hGHPjLob7Qzs/E661CFPaHt/6likRqB9uxbaZdkAnYt3eIWwVNcvwjod1sfAAn2PsvCKhb7bykBJgua7CaUJV/QhFVI0BhgxQyBGOUROjsE0j0V77Cb5Jhtb0cbYSI2sdr2x/TNmS0Z32cUPhLJB1UrNoCd48ailYnzw1ZZgSSencA== Received: from MN2PR15CA0044.namprd15.prod.outlook.com (2603:10b6:208:237::13) by PH7PR12MB5781.namprd12.prod.outlook.com (2603:10b6:510:1d0::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18; Tue, 19 Dec 2023 18:10:09 +0000 Received: from BL6PEPF0001AB74.namprd02.prod.outlook.com (2603:10b6:208:237:cafe::b7) by MN2PR15CA0044.outlook.office365.com (2603:10b6:208:237::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.18 via Frontend Transport; Tue, 19 Dec 2023 18:10:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB74.mail.protection.outlook.com (10.167.242.167) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:10:08 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:55 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:55 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:51 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 13/15] vdpa/mlx5: Use vq suspend/resume during .set_map Date: Tue, 19 Dec 2023 20:08:56 +0200 Message-ID: <20231219180858.120898-14-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB74:EE_|PH7PR12MB5781:EE_ X-MS-Office365-Filtering-Correlation-Id: f4c411a6-cedc-457b-ae01-08dc00bdbc3d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9B5Jwi0A+bWa/ycD2FRPnAI+ZjJDGbYQFPinA/a5ZQiaBico6Lt+5Kl1ZfxHuTL5E2OsPPQtiYPKt6Tl6PTOioYp2wLGNwre+6Xk8YmDfrJMZJioemKTC5Qb6FLVGB/ZzvTQ2GHEOHa8HYnP1IX+elLcF6JoYnbJ5NkUyJrPl/DGz+SMFZAgaYTSHHTq9fbijBa293VM0R1DYBRUk72dFOmcm63RMO2vHMAR29l5RsQJyC1ufNjzbnpCJv4odlUhTrm5ZgW/prn9VofnLBtd2wZy/SIvfopkojv45KI85z2zJhvU2PFyYdudW3FmIS15xYhTS4lIa79VyTsHfKjXyeZfuZ4/OkygEtltwhqfUjkyLmNwaPAOzTaWb7U7zhZE7E+R8IZHZOESFYf68MAsH7m06bHDLCa0s/3UPC2pdwwssWvnBKSrznkkWW7omdfc0+w2dmR4COaiY8pGaJIvS4S02vcgar3kRx1UUZTO4UoHyl9Dt5Djqt1dYnws6qftUCp6vd/QHjArecp+xw5GIdoYbFN178oWySDJkFPd3AtDoShTVH5OYBbUfQwJr/AFbub5c5roccspExfsHhcbArjtQyw8Zk2FmIgl/7Qf6A9o4b4U22nxA1h//Hl2A+U63eIAeoLAYf/2PUgrdd59hweQlokXwFByEeTDSNTEuAmp+vYsx4hDx1VgdguDGiGITLqRNCRgIOtqivai3eC0QlEcWOA1o30QTjRDhrFGzDwOfbiZNLu03Hbh/nR23mR8t7yX6Yaqt+QgQLnem9MBsA== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(39860400002)(346002)(136003)(396003)(230922051799003)(451199024)(1800799012)(82310400011)(186009)(64100799003)(46966006)(36840700001)(40470700004)(5660300002)(66574015)(426003)(336012)(41300700001)(82740400003)(7636003)(36756003)(1076003)(86362001)(83380400001)(36860700001)(47076005)(356005)(15650500001)(2616005)(26005)(2906002)(40480700001)(478600001)(4326008)(6636002)(316002)(8676002)(8936002)(54906003)(70586007)(70206006)(110136005)(40460700003)(142923001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:10:08.4715 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f4c411a6-cedc-457b-ae01-08dc00bdbc3d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB74.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5781 Instead of tearing down and setting up vq resources, use vq suspend/resume during .set_map to speed things up a bit. The vq mr is updated with the new mapping while the vqs are suspended. If the device doesn't support resumable vqs, do the old teardown and setup dance. Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 46 ++++++++++++++++++++++++------ include/linux/mlx5/mlx5_ifc_vdpa.h | 1 + 2 files changed, 39 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index b760005e2920..fcadbeede3e1 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1206,6 +1206,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; + struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; bool state_change = false; void *obj_context; void *cmd_hdr; @@ -1255,6 +1256,24 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX) MLX5_SET(virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx); + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { + struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + + if (mr) + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, mr->mkey); + else + mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY; + } + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { + struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + + if (mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) + MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, mr->mkey); + else + mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; + } + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2791,24 +2810,35 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); + bool teardown = !is_resumable(ndev); int err; suspend_vqs(ndev); - err = save_channels_info(ndev); - if (err) - return err; + if (teardown) { + err = save_channels_info(ndev); + if (err) + return err; - teardown_driver(ndev); + teardown_driver(ndev); + } mlx5_vdpa_update_mr(mvdev, new_mr, asid); + for (int i = 0; i < ndev->cur_num_vqs; i++) + ndev->vqs[i].modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY | + MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; + if (!(mvdev->status & VIRTIO_CONFIG_S_DRIVER_OK) || mvdev->suspended) return 0; - restore_channels_info(ndev); - err = setup_driver(mvdev); - if (err) - return err; + if (teardown) { + restore_channels_info(ndev); + err = setup_driver(mvdev); + if (err) + return err; + } + + resume_vqs(ndev); return 0; } diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index 32e712106e68..40371c916cf9 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -148,6 +148,7 @@ enum { MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX = (u64)1 << 7, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX = (u64)1 << 8, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY = (u64)1 << 11, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; From patchwork Tue Dec 19 18:08:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498820 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2069.outbound.protection.outlook.com [40.107.93.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E807138FA7; Tue, 19 Dec 2023 18:10:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="VpJ9wUa+" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JgEGuFS6vv7nLwUrUrlzHPVn3lgd2nYejuiPL85d4mPSOmQkin0wCgkeFyjmUsfnx13jL+tT4pWbrRV7yyyIFEmA762e7ayuPCO/OyYvPm7lX6I+HTjZBGjqrKlHRoQhLqSIo274Q5xqWN4rVvVRD0TxUZ0AriUgeTW6UzbCtfmDtAKrQuhxSIEFmFGpvljYdKKZWIyQuSrDHOBwA7YvE6dnDRaBf+IaJGRCO1AtzPHhI3ymiys99gkIWNla32zDIDLzMrHE46f++p2Q63bWZKIcKFLzI3JqnPrp+7J3XX57e1yCyJaUopF7SGccymcBl9ZofNK8wzCliTZXOG7N/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YDVmZqcRyoa1ZdteQQ5jmdeApTj3yX+tPraOMFSZZ/A=; b=aFS6rzkK6lVQMOXpTTHVedghKELeWvPBCacxRUWqyWwJrp32jrnZ4tKhHxAfdTJwT0iCDn1xYWFxXupqBpJ7lg4kgqQl+s0RMt2to5CSbY0xNxRBsANX0JhOUixmroDKdxGyOXRkL4+7bUH6ctmRQ1sRcIr7pKuuAbrv9dMwZelNPpnBGd7NrecOFWGtjWugukqbk0FvSYSwRARYS1/Gh8QrihE+x45o4Ho6kFUms+jd18jFdEYrpyEC0rrLPX+BhDVaiGsoZsxmjQSyXvZspnFBZyxiTXxF4tn3It9OtEcDe3xQac9pC6uAxIQlBzFeV9GEjN/R3jOOImEiKxfROQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YDVmZqcRyoa1ZdteQQ5jmdeApTj3yX+tPraOMFSZZ/A=; b=VpJ9wUa+ub/94cOKsX3wXRT1tfKxDIfEgoINYziVUjw//oEA6rANdQ+RXqhLUdf2e/8dlN5t8qqCyOBnL5MA9WKTagPZTqw9LPWsqITq0p9N8b2vNvDtSsdHqERXiQsdBewyV2HrKUeFsCTZKUZGzanQzeBB4vNW9V4X46vtaDMkh3CXiy7cT9Obq1/WSXkh+I5GwGFIMjZ4Ln9RLHBM2rOvwnFGIc2EJS/KQezYk97JFtyFxmCbM90tGZyVic95BBJ+6argOBtUoQOa/jVFkYtpeHkGCsZL8nU665aQko06qwjyXuMthbQgwFHOvbIkDjZs0YK27uBITqbmpATIOw== Received: from BLAPR03CA0141.namprd03.prod.outlook.com (2603:10b6:208:32e::26) by DM6PR12MB4329.namprd12.prod.outlook.com (2603:10b6:5:211::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38; Tue, 19 Dec 2023 18:10:14 +0000 Received: from BL6PEPF0001AB75.namprd02.prod.outlook.com (2603:10b6:208:32e:cafe::a9) by BLAPR03CA0141.outlook.office365.com (2603:10b6:208:32e::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38 via Frontend Transport; Tue, 19 Dec 2023 18:10:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB75.mail.protection.outlook.com (10.167.242.168) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:10:13 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:59 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:09:58 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:55 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 14/15] vdpa/mlx5: Introduce reference counting to mrs Date: Tue, 19 Dec 2023 20:08:57 +0200 Message-ID: <20231219180858.120898-15-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB75:EE_|DM6PR12MB4329:EE_ X-MS-Office365-Filtering-Correlation-Id: 0b2c8442-81c3-4837-1705-08dc00bdbf5c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: z47rwzkk3IykgzDNbqaRDHswtIx6j6qD0eHGo1Cw2DVfGzX27gG/ga/c/SmKOuZVl/si5lRI8dWDpDKz2MGyJdz2ggdbCYGFR34IY1Qq4v7uIN7G9beScaRlzJd7vpx5+VZbp0n7xr/uK/oxLSlspd/1jsN5dc1fvmuDVnbOb3flCu7dok8WtCh+X0SzF/nrRckYqLUkzJctg8kB3EZxACfDsQg0BafqG3BUMAZ3rQr5Nc9sLh22VlYgi5bPf1l78pRDjOdSOVABrlgUj95zNMYEaiTlMJRiDFbPnn37sTNskE999WHmHW803LNaiN9Y/O0LsCTqm9cTFjSfNZaH9v3gojHEtBnSDbSYqfbpYfMJRin0a3qhQIpaoy3qfZHWfEbgmAQ6UzJEJumm/y8QCbGj3yamLG1/5AxnSq3j4eDvXOKN3VEVniaqEFGFet1sxSL9wJzmABQCE7jiaV86w4ZCW0A4gDo6f83AyT/riSYb+bEimfTqP2HRn+JSl4AVq8t6NTydJCuCW1sBod2FdvJDxvSOn4SDNSYRYfBFgDJ3kcBi2InJqmiBxckJONLsyff796SHS6NX3CfTapzqYPmVcMZIF930PHc2V99s+vskEiO7AurI89Mn8h4shrONJrGY0zNFBa3cUTTqpFUngHjw6YqLjpfjYGXIwhLEhJTKISR6pDGM+6mtucRVbWW0EbXLpnmkcjLZ6beHcKMU7dzzpDnneN9mJfP1k38rsgDVtWXGTxCCFTkLPQ8ReWdC X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(1800799012)(451199024)(64100799003)(186009)(82310400011)(40470700004)(36840700001)(46966006)(47076005)(2906002)(8936002)(8676002)(4326008)(40480700001)(54906003)(6636002)(83380400001)(316002)(5660300002)(36860700001)(70206006)(426003)(70586007)(66574015)(336012)(110136005)(82740400003)(356005)(41300700001)(40460700003)(36756003)(7636003)(1076003)(478600001)(2616005)(26005)(86362001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:10:13.7038 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0b2c8442-81c3-4837-1705-08dc00bdbf5c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB75.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4329 Deleting the old mr during mr update (.set_map) and then modifying the vqs with the new mr is not a good flow for firmware. The firmware expects that mkeys are deleted after there are no more vqs referencing them. Introduce reference counting for mrs to fix this. It is the only way to make sure that mkeys are not in use by vqs. An mr reference is taken when the mr is associated to the mr asid table and when the mr is linked to the vq on create/modify. The reference is released when the mkey is unlinked from the vq (trough modify/destroy) and from the mr asid table. To make things consistent, get rid of mlx5_vdpa_destroy_mr and use get/put semantics everywhere. Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 8 +++-- drivers/vdpa/mlx5/core/mr.c | 50 ++++++++++++++++++++---------- drivers/vdpa/mlx5/net/mlx5_vnet.c | 45 ++++++++++++++++++++++----- 3 files changed, 78 insertions(+), 25 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 84547d998bcf..1a0d27b6e09a 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -35,6 +35,8 @@ struct mlx5_vdpa_mr { struct vhost_iotlb *iotlb; bool user_mr; + + refcount_t refcount; }; struct mlx5_vdpa_resources { @@ -118,8 +120,10 @@ int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey); struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb); void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *mr); +void mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr); +void mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr); void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr, unsigned int asid); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 2197c46e563a..c7dc8914354a 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -498,32 +498,52 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { + if (WARN_ON(!mr)) + return; + if (mr->user_mr) destroy_user_mr(mvdev, mr); else destroy_dma_mr(mvdev, mr); vhost_iotlb_free(mr->iotlb); + + kfree(mr); } -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *mr) +static void _mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) { if (!mr) return; + if (refcount_dec_and_test(&mr->refcount)) + _mlx5_vdpa_destroy_mr(mvdev, mr); +} + +void mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) +{ mutex_lock(&mvdev->mr_mtx); + _mlx5_vdpa_put_mr(mvdev, mr); + mutex_unlock(&mvdev->mr_mtx); +} - _mlx5_vdpa_destroy_mr(mvdev, mr); +static void _mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) +{ + if (!mr) + return; - for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) { - if (mvdev->mr[i] == mr) - mvdev->mr[i] = NULL; - } + refcount_inc(&mr->refcount); +} +void mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) +{ + mutex_lock(&mvdev->mr_mtx); + _mlx5_vdpa_get_mr(mvdev, mr); mutex_unlock(&mvdev->mr_mtx); - - kfree(mr); } void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, @@ -534,20 +554,16 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, mutex_lock(&mvdev->mr_mtx); + _mlx5_vdpa_put_mr(mvdev, old_mr); mvdev->mr[asid] = new_mr; - if (old_mr) { - _mlx5_vdpa_destroy_mr(mvdev, old_mr); - kfree(old_mr); - } mutex_unlock(&mvdev->mr_mtx); - } void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) - mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[i]); + mlx5_vdpa_update_mr(mvdev, NULL, i); prune_iotlb(mvdev->cvq.iotlb); } @@ -607,6 +623,8 @@ struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, if (err) goto out_err; + refcount_set(&mr->refcount, 1); + return mr; out_err: @@ -651,7 +669,7 @@ int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) if (asid >= MLX5_VDPA_NUM_AS) return -EINVAL; - mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[asid]); + mlx5_vdpa_update_mr(mvdev, NULL, asid); if (asid == 0 && MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { if (mlx5_vdpa_create_dma_mr(mvdev)) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index fcadbeede3e1..f81968b3f9cf 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -123,6 +123,9 @@ struct mlx5_vdpa_virtqueue { u64 modified_fields; + struct mlx5_vdpa_mr *vq_mr; + struct mlx5_vdpa_mr *desc_mr; + struct msi_map map; /* keep last in the struct */ @@ -946,6 +949,14 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque kfree(in); mvq->virtq_id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id); + mlx5_vdpa_get_mr(mvdev, vq_mr); + mvq->vq_mr = vq_mr; + + if (vq_desc_mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) { + mlx5_vdpa_get_mr(mvdev, vq_desc_mr); + mvq->desc_mr = vq_desc_mr; + } + return 0; err_cmd: @@ -972,6 +983,12 @@ static void destroy_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtq } mvq->fw_state = MLX5_VIRTIO_NET_Q_OBJECT_NONE; umems_destroy(ndev, mvq); + + mlx5_vdpa_put_mr(&ndev->mvdev, mvq->vq_mr); + mvq->vq_mr = NULL; + + mlx5_vdpa_put_mr(&ndev->mvdev, mvq->desc_mr); + mvq->desc_mr = NULL; } static u32 get_rqpn(struct mlx5_vdpa_virtqueue *mvq, bool fw) @@ -1207,6 +1224,8 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; + struct mlx5_vdpa_mr *desc_mr = NULL; + struct mlx5_vdpa_mr *vq_mr = NULL; bool state_change = false; void *obj_context; void *cmd_hdr; @@ -1257,19 +1276,19 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET(virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx); if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { - struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; - if (mr) - MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, mr->mkey); + if (vq_mr) + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, vq_mr->mkey); else mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY; } if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { - struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + desc_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; - if (mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) - MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, mr->mkey); + if (desc_mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) + MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, desc_mr->mkey); else mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; } @@ -1282,6 +1301,18 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, if (state_change) mvq->fw_state = state; + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { + mlx5_vdpa_put_mr(mvdev, mvq->vq_mr); + mlx5_vdpa_get_mr(mvdev, vq_mr); + mvq->vq_mr = vq_mr; + } + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { + mlx5_vdpa_put_mr(mvdev, mvq->desc_mr); + mlx5_vdpa_get_mr(mvdev, desc_mr); + mvq->desc_mr = desc_mr; + } + mvq->modified_fields = 0; done: @@ -3102,7 +3133,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); out_err: - mlx5_vdpa_destroy_mr(mvdev, new_mr); + mlx5_vdpa_put_mr(mvdev, new_mr); return err; } From patchwork Tue Dec 19 18:08:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 13498821 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2048.outbound.protection.outlook.com [40.107.244.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D7163EA7A; Tue, 19 Dec 2023 18:10:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="dwKZsxvI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CYDzmx52GbsOqF8a+KVSazDhdsyjEIDc2d/5D+pGi9Bxu7nFIs7fVpLgvPgdby6YGl9m5us8/vmwrm2pGF/Rd6ujr6o647RKiDFUbf3b1Vt22Cht6GKFlhIRIpnldQZd/wKURjiEu5+WaLlwAfE8/rOGLFysfnGkNouc34sIXt/bbD3D0NXzBIQBEjsh6CCH03OdkfDdj8wQiQ9WbIit8jh/pCqBj42a06TyKQBaOD5BmOHcSnR4amFqy4VkLWE0xteNuYZRb1RZDljloFwIbJ/U+f653ymAlZD8BZjeljasus5D67bwJ/Pm7QEoxKIvQv4AzLltIIpljib3xU/PZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MKO46lgwXxoc9p1PhLlI2slnpBU+0n8UQO1kpc1S1rY=; b=REQnaZ5+5f7rSFoalnvhmNY9V+SD36M2n3ZgMeQrCZA3b8u9L7S10IOn9HvT46D74hzA67zMfAVxyqQHhcDzz+glmRLRXt86cpZKWKLWLSpsdr/3swuD8cSIQWzg36hYdlXHZVT03I/UpCkVsQ6RxjU67nTjUg4+HyxWmR7hFn500q8xTPzGgHZCyvkhIK9eAXNocRfht/BFZO7y+Er6NA+CKh9ZJ9erEMP6hvgFo3NSzic3LhXSwx4ZC17qEKl6DWbqAmBqjECs/0dseyin+M2SVHKBk29dOknp0apOTzLwqOljDIkKYdpb6pqiu0i8rB/lQ3Oi6YianwULITeolA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MKO46lgwXxoc9p1PhLlI2slnpBU+0n8UQO1kpc1S1rY=; b=dwKZsxvIf45R6vg6rhgRJhIBlL+JMp+3qErlkXy+k9D+O6+l3vhNK+KB1uORd9Dz8vagfVW6C7S7rpM1UZlFBtY5cmvrTkK6i6vP1F5Bv7pa2RG8Gmvi/p+pc60MiGEHNWN1+Hgq8ZQJQQoRIMennmsz6qyRyGqzLeFAQMTV/FT2zvykFzdj9CuShIN8no4btHMhtYiLJi9f4IvutY/KHqdCwM3ZV3XdgQ0ZB3hEP7hF/uF1wuiF+R7t4T6d7OAtaYU+h2oCDe6pCUGC2su8OnKiQWrqdmRv5tLEp3G3DiEV8TsZOkkYErsL5LRhxTbr/k5ne4Xn839aR9l2Gxn3TQ== Received: from BLAPR03CA0129.namprd03.prod.outlook.com (2603:10b6:208:32e::14) by DS0PR12MB7996.namprd12.prod.outlook.com (2603:10b6:8:14f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38; Tue, 19 Dec 2023 18:10:16 +0000 Received: from BL6PEPF0001AB75.namprd02.prod.outlook.com (2603:10b6:208:32e:cafe::b8) by BLAPR03CA0129.outlook.office365.com (2603:10b6:208:32e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7091.38 via Frontend Transport; Tue, 19 Dec 2023 18:10:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL6PEPF0001AB75.mail.protection.outlook.com (10.167.242.168) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7113.14 via Frontend Transport; Tue, 19 Dec 2023 18:10:16 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:10:03 -0800 Received: from rnnvmail205.nvidia.com (10.129.68.10) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 19 Dec 2023 10:10:02 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.10) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 19 Dec 2023 10:09:59 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 15/15] vdpa/mlx5: Add mkey leak detection Date: Tue, 19 Dec 2023 20:08:58 +0200 Message-ID: <20231219180858.120898-16-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231219180858.120898-1-dtatulea@nvidia.com> References: <20231219180858.120898-1-dtatulea@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB75:EE_|DS0PR12MB7996:EE_ X-MS-Office365-Filtering-Correlation-Id: a8c7a0a5-8d40-424c-4fe3-08dc00bdc11c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fD3pj2K2/23gktmNY3Dcxb1UftwqDM1kCHqkbjdv7cjLER9zcsgHprv3dp49p/m6RUzGqFa5/p4/hLGCq7MPgknUPz4vFr7UNP0B9TnwKmS2MUg5vQ3BzJLZtQXeY+dw2whzpgwNod50seX39zbdiG5rlaRFp8CixefEoqqZ4S32qvHOFM0JtRVD9dnFviYq+ghjxI6TYXEyZqxpGDOsLZi5ExDs+cDCC0ypK8SUw3wrEoAiuoPvdsz1FYlw3YqrXWPJdcoMAYtDhOM5eBdTHB8gb7EuzxDvlsYBUwfpaY9nT46ILqz4uHr/sst5pTr40Hn7r+UGytL3jFNJlLVpGylQCVOOCz4YCuUqittqdxGDtQd9d14c1m4NRb+GBmpKBTvh2q6JCGjQPetZnunem7ZDpTuovyUwQRPfFkWDwQCTqasz+c41eL1CS7TBFPv22nIRzRTVPuDrsd7lN7+d2lnLG8OPHNho/lRgCqRXIhbJANwrdR7wYkzqEZsTQ2CPehedWThTE7Ye+4cHiMmnXfQGY7i+KB8X3dSPRw5Qh1bysvUczR4wxV2kn1xX/nauJmjNZDFxBoXaFs3YzqipE5YNshTTfpb+WD55ztaWuYXEPPoGVCC7M3nFcPxSf+jjoJpwNrPbD8HXM0caeoQXQIj01OCS/gi83ANw5BmCAoGAC2qTVnWNpzdn3w1fvORejI2l7eajjiE+5Js6ydnuVe56AjS2oEsn1uxDS8BBgu0qxw+weWKRlegSlyi8284muIEp6Q46oQmsmodllLNCQJUZ9YmtCUNm7KJbM59yB5U= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(136003)(346002)(376002)(396003)(230273577357003)(230922051799003)(230173577357003)(1800799012)(82310400011)(451199024)(64100799003)(186009)(36840700001)(40470700004)(46966006)(336012)(26005)(82740400003)(426003)(2616005)(66574015)(1076003)(2906002)(47076005)(36860700001)(5660300002)(83380400001)(4326008)(110136005)(8936002)(8676002)(478600001)(41300700001)(316002)(6636002)(70206006)(54906003)(70586007)(7636003)(356005)(86362001)(36756003)(40480700001)(40460700003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 18:10:16.6413 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a8c7a0a5-8d40-424c-4fe3-08dc00bdc11c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB75.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7996 Track allocated mrs in a list and show warning when leaks are detected on device free or reset. Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 ++ drivers/vdpa/mlx5/core/mr.c | 23 +++++++++++++++++++++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 ++ 3 files changed, 27 insertions(+) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 1a0d27b6e09a..50aac8fe57ef 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -37,6 +37,7 @@ struct mlx5_vdpa_mr { bool user_mr; refcount_t refcount; + struct list_head mr_list; }; struct mlx5_vdpa_resources { @@ -95,6 +96,7 @@ struct mlx5_vdpa_dev { u32 generation; struct mlx5_vdpa_mr *mr[MLX5_VDPA_NUM_AS]; + struct list_head mr_list_head; /* serialize mr access */ struct mutex mr_mtx; struct mlx5_control_vq cvq; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index c7dc8914354a..4758914ccf86 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -508,6 +508,8 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_ vhost_iotlb_free(mr->iotlb); + list_del(&mr->mr_list); + kfree(mr); } @@ -560,12 +562,31 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, mutex_unlock(&mvdev->mr_mtx); } +static void mlx5_vdpa_show_mr_leaks(struct mlx5_vdpa_dev *mvdev) +{ + struct mlx5_vdpa_mr *mr; + + mutex_lock(&mvdev->mr_mtx); + + list_for_each_entry(mr, &mvdev->mr_list_head, mr_list) { + + mlx5_vdpa_warn(mvdev, "mkey still alive after resource delete: " + "mr: %p, mkey: 0x%x, refcount: %u\n", + mr, mr->mkey, refcount_read(&mr->refcount)); + } + + mutex_unlock(&mvdev->mr_mtx); + +} + void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) mlx5_vdpa_update_mr(mvdev, NULL, i); prune_iotlb(mvdev->cvq.iotlb); + + mlx5_vdpa_show_mr_leaks(mvdev); } static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, @@ -592,6 +613,8 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, if (err) goto err_iotlb; + list_add_tail(&mr->mr_list, &mvdev->mr_list_head); + return 0; err_iotlb: diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index f81968b3f9cf..a783e8bd784d 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -3729,6 +3729,8 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, if (err) goto err_mpfs; + INIT_LIST_HEAD(&mvdev->mr_list_head); + if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { err = mlx5_vdpa_create_dma_mr(mvdev); if (err)