From patchwork Sun Sep 22 12:49:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809105 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2058.outbound.protection.outlook.com [40.107.93.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C4812CCC2 for ; Sun, 22 Sep 2024 12:50:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009453; cv=fail; b=Ad4KOqgGKbvqTk2jpjfCYbcHebTsTD17H182mUZZ+HjUroeGtq2i/ve6HywZjyxMwGKJY4Dwg+Pm+uwKxbcUBfgCEywyvzX41q62SoY1V3BY7Y6Tx4DF1+0kpdZpufRseqm9evAsvDNPOIVrD/GAWG7HaqDtjcsPfswOhH+ZyWo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009453; c=relaxed/simple; bh=aOWt9elKSOoR1bJCvJ5C3PYz1TkBxBXofeR5Yvrl+Ww=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eC3lpIqDWjDiSc4S51bKAWWgixQLSiQSjnKIBvwqDWvL6zXXv48+shodcCjn9w28CLj90udxvnL3iN1e14ymwsqKAdxAggPXiaAZLlDd1YPVykRP43UcrQDfgc1jd4tEDwXU1YPC9Jeu0Z8ysaS9tBj8rHGvoqkGG7svl4PvaW4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=CgIHs6xD; arc=fail smtp.client-ip=40.107.93.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="CgIHs6xD" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YASe/G7rmiAp8cI5yxT2t5UIHRGIpYiClx/ZV2R5wBRqnNYzf8Wh1mHMbNYZpcLn2rH/qEEeAD9ZWMAQ62celTK4OqkRJJjY9IHwPvk2p1+ObjUfJEc5/McL7RyZdOCSYI5AneNZZaIpossySSJqyiR0F0AzTb4VORJ0JjHeP3ma8lm1Mx1boNNtXVpopk0qr1iWuqVcwCpuD1QAdX/Qnb21w3BOjSPuP2932Pt4xAMy1kTjpx4kIggaeF/0hpFPaZ/dqu0u1B2fP761BgrwkxTc9naYe1SKUeqDOCuJcCw9D+g3Wdb81qndNzf6otO7oF7Gd7qDAzPGS6y376PClw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ov5LC7GbiOMPGtzA4WRPVETEec9xsRePWztyrkfU+jo=; b=dhe6VlXn2zLRmAVTnFOdTWTu/dfDKgYryqFW8JqM//QtymnRm3ejPQaaLWdxNoWmvjfdLIYg+ssRfvJJJ6Qh6Pz/GkIESdEKo8CN+0/iY1aAeVwe3AEjF4u/fhsfoMS5Bud1t/iE/iW8q+X8Tvk9mRtQIPw1R4R04rAQrOF1OQtB8ZLlKQrYQl4u9JDtgm2snHtxWu1mo+Xgy14Jqnap7VGDzHJ6Bms+VFrwrg6usbSFPKqC6pgHETjeif7DYA08U87QjdllUeEjIJdHKOsvbTjaw+N/Y2mgubLF5HbdnasC71I3zoKdEQu8TTBTzJWcOEWfe+4VJMQ7amEW8M/zuQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ov5LC7GbiOMPGtzA4WRPVETEec9xsRePWztyrkfU+jo=; b=CgIHs6xDtCix6XNcH4x2RrbxQ/T1oDyfIUg+msy7Ly3BJ91v0VZGkf0mXTnkm1X3H3gl+qyTiSng7ZsGVomrbvbjATbNbzkf+3H/WJAHX+oQcdN2KXxmPID2Oqjqb+1VpgbIWCybby1vxskxg2d/O1p5Og/2jOmyV+Fg/pvPgHQZz55rIOiN9yjQJDA90C0Z7czwS1+TxeYOQvtEF5hJniwV/lOjyRhQD5WGq+d9GTJpjux55szFLaurgRsPxGNILJhAQCIXGmx8LQY0DCvzSuzfax02g15e042FvRFmWuneWQU9tW1m2n7og8s7B9zSV2beu5+/D0Y7SiXcFsFNbQ== Received: from BY1P220CA0013.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::17) by CY5PR12MB6203.namprd12.prod.outlook.com (2603:10b6:930:24::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Sun, 22 Sep 2024 12:50:46 +0000 Received: from CO1PEPF000075F2.namprd03.prod.outlook.com (2603:10b6:a03:59d:cafe::9) by BY1P220CA0013.outlook.office365.com (2603:10b6:a03:59d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23 via Frontend Transport; Sun, 22 Sep 2024 12:50:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F2.mail.protection.outlook.com (10.167.249.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:46 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:34 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:33 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:33 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 22/29] vfio/vgpu_mgr: allocate vGPU FB memory when creating vGPUs Date: Sun, 22 Sep 2024 05:49:44 -0700 Message-ID: <20240922124951.1946072-23-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F2:EE_|CY5PR12MB6203:EE_ X-MS-Office365-Filtering-Correlation-Id: 8608cda4-3fce-412e-2d1a-08dcdb052d89 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?kDP5HGo8XFqmr+rLex/pxjwmVbQqA5c?= =?utf-8?q?ej7HUgkhkJvK3bvOY5RHIls6RID7EnZWGDtnMwbutz9dWQwJuEG7+n3/yXByY1/+0?= =?utf-8?q?u4JAXbCoppuEgePNQr7g7KY7wL2z9/9vxq/pSqWSPsiQCVTk1yIbPNTYqbPknbP8z?= =?utf-8?q?UdyKvYOvQYxn4IZ0v5eROfTmrtiN6EdQbDU2fOc2HZhKKAWTaO+1l2ltADUGVXVnX?= =?utf-8?q?+YwG6tdzeyQyQJIq4o/otUdWgHxQxOKgShxcHQjfSqpU7/3Kc6U0zdGEHQecPrq1P?= =?utf-8?q?To59pUVnhKEDrWFs6G+alWVR4auxsw9xRlGSVFmYLJJ5VgZWKz9lvDkHsG8uGp9Gw?= =?utf-8?q?36dyYRuczfE8uP935EXcennyu2lvx5x7+qcfdTsRigwdKd4Zgdk/dYRkwGXVGJorp?= =?utf-8?q?GPolz2nl0cZhEo89v2HZ5EIDbz2M6eIoFj0rdarS1Fe6wEmGKKigYi52VHwqEU8rH?= =?utf-8?q?AlidDFadDc5cWMBiuV1jtMZ+k0uwnHyRs8jgWpAHB4snjHeSaYuG7Ipx4LOpvT5Ha?= =?utf-8?q?mgrfjrWgD9aMcER90k40z/Dd2Cu21RFFO36btq7IWpJKQSAbSmgaon1OoY5/j8Nij?= =?utf-8?q?530wwq8C9HK4Wk0BnfPYdR8zsWbvrQHBFXUaLzDFUbnIzbRta7et92LSXzk1OWBJC?= =?utf-8?q?uU8tTklV0gdoT5eZyuxf4k8+rebWZPNfcOkDyr2ufppM30xN/uR3/UULKOZmhd27x?= =?utf-8?q?fuFqmnN+UAsSV8AHLZzncQ1DR9Jnz8cd0yUGwPXMTe9BQCsCNGQOIJhXa8glsYRJm?= =?utf-8?q?R0uYRnoAxTOZ/2fXGBG8hQsbJn3WuoaTFBqEVbrFztQLlqljKsNKGk3vYpWO/ei+A?= =?utf-8?q?28xuxSyYfcSf5/c6LjiJs2k2LX12QofpciLQG9lxDcc1NuWW74B2aFoB8m6byEmir?= =?utf-8?q?eNVZwiqFY4CQ9y0ybUokAZRZXy6jsCF0KNtDqnJTN7UXHbYLz20VzkBhpVIvQh9/d?= =?utf-8?q?Kzw2ZqOrwpcacbTwU7Ii0ePNgbFcZWNzP178ybRzlGhrDu6tBcK7/oXVkl40vi0Hi?= =?utf-8?q?cILJ01ZQ1+azNfTdEsM6dIDeLLNlwjcDSpoHgabKl9KB/VzqdBe+OXeOa/+kzvRBM?= =?utf-8?q?g5C7m6PNDGKCb7soBA/shnq2I89mIXZEpDf3ULL5hIExdZR8/T7nM4laiXg5PjcQP?= =?utf-8?q?NP8GrGUl6EkoYhZIsj+++sFrQqMTLNRMqzK164HrrBlZ/sKKKnZa3lrHKBZp4ko3z?= =?utf-8?q?wP08Ev+YdyfN2If0ukH3eEz+vqFokC5Ue8zvXCf1p24prUNJWUzsCEfXAO2j0T+xU?= =?utf-8?q?4opYW9s2aC1tUud+PbXs17ooawRqET/bMriQZJFnT1M95p+rxoss5vcbwYBAHLoZA?= =?utf-8?q?HW3MtEo+YXi5s3xVR6ZfoURxVwm4fftLdlPx5UisaU9vwmJTTpIXqSZso2bOwSofJ?= =?utf-8?q?zsgXfe35qrz?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:46.4270 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8608cda4-3fce-412e-2d1a-08dcdb052d89 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6203 Creating a vGPU requires allocating a portion of the FB memory from the NVKM. The size of the FB memory that a vGPU requires is from the vGPU type. Acquire the size of the required FB memory from the vGPU type. Allocate the FB memory from NVKM when creating a vGPU. Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/nvkm.h | 6 ++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | 38 +++++++++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 2 ++ 3 files changed, 46 insertions(+) diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h index 065bb7aa55f8..d3c77d26c734 100644 --- a/drivers/vfio/pci/nvidia-vgpu/nvkm.h +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -64,4 +64,10 @@ static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, #define nvidia_vgpu_mgr_rm_ctrl_done(m, g, c) \ m->handle.ops->rm_ctrl_done(, c) +#define nvidia_vgpu_mgr_alloc_fbmem_heap(m, s) \ + m->handle.ops->alloc_fbmem(m->handle.pf_drvdata, s, true) + +#define nvidia_vgpu_mgr_free_fbmem_heap(m, h) \ + m->handle.ops->free_fbmem(h) + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index 34f6adb9dfe4..54e27823820e 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -3,6 +3,11 @@ * Copyright © 2024 NVIDIA Corporation */ +#include + +#include +#include + #include "vgpu_mgr.h" static void unregister_vgpu(struct nvidia_vgpu *vgpu) @@ -34,6 +39,29 @@ static int register_vgpu(struct nvidia_vgpu *vgpu) return 0; } +static void clean_fbmem_heap(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + + nvidia_vgpu_mgr_free_fbmem_heap(vgpu_mgr, vgpu->fbmem_heap); + vgpu->fbmem_heap = NULL; +} + +static int setup_fbmem_heap(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + NVA081_CTRL_VGPU_INFO *info = + (NVA081_CTRL_VGPU_INFO *)vgpu->vgpu_type; + struct nvidia_vgpu_mem *mem; + + mem = nvidia_vgpu_mgr_alloc_fbmem_heap(vgpu_mgr, info->fbLength); + if (IS_ERR(mem)) + return PTR_ERR(mem); + + vgpu->fbmem_heap = mem; + return 0; +} + /** * nvidia_vgpu_mgr_destroy_vgpu - destroy a vGPU instance * @vgpu: the vGPU instance going to be destroyed. @@ -45,6 +73,7 @@ int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu) if (!atomic_cmpxchg(&vgpu->status, 1, 0)) return -ENODEV; + clean_fbmem_heap(vgpu); unregister_vgpu(vgpu); return 0; } @@ -76,8 +105,17 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) if (ret) return ret; + ret = setup_fbmem_heap(vgpu); + if (ret) + goto err_setup_fbmem_heap; + atomic_set(&vgpu->status, 1); return 0; + +err_setup_fbmem_heap: + unregister_vgpu(vgpu); + + return ret; } EXPORT_SYMBOL(nvidia_vgpu_mgr_create_vgpu); diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index eb2df9f0fe07..35312d814996 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -23,6 +23,8 @@ struct nvidia_vgpu { u8 *vgpu_type; struct nvidia_vgpu_info info; struct nvidia_vgpu_mgr *vgpu_mgr; + + struct nvidia_vgpu_mem *fbmem_heap; }; struct nvidia_vgpu_mgr {