| Message ID | 20240922124951.1946072-5-zhiw@nvidia.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Introduce NVIDIA GPU Virtualization (vGPU) Support | expand |
On Sun, Sep 22, 2024 at 05:49:26AM -0700, Zhi Wang wrote: > GSP firmware needs to know the number of max-supported vGPUs when > initialization. > > The field of VF partition count in the GSP WPR2 is required to be set > according to the number of max-supported vGPUs. > > Set the VF partition count in the GSP WPR2 when NVKM is loading the GSP > firmware and initializes the GSP WPR2, if vGPU is enabled. How/why is this different from the SRIOV num_vfs concept? The way the SRIOV flow should work is you boot the PF, startup the device, then userspace sets num_vfs and you get the SRIOV VFs. Why would you want less/more partitions than VFs? Is there some way to consume more than one partition per VF? At least based on the commit message this seems like a very poor FW interface. Jason
On 27/09/2024 1.51, Jason Gunthorpe wrote: > On Sun, Sep 22, 2024 at 05:49:26AM -0700, Zhi Wang wrote: >> GSP firmware needs to know the number of max-supported vGPUs when >> initialization. >> >> The field of VF partition count in the GSP WPR2 is required to be set >> according to the number of max-supported vGPUs. >> >> Set the VF partition count in the GSP WPR2 when NVKM is loading the GSP >> firmware and initializes the GSP WPR2, if vGPU is enabled. > > How/why is this different from the SRIOV num_vfs concept? > 1) The VF is considered as an HW interface of vGPU exposed to the VMM/VM. 2) Number of VF is not always equal to number of max vGPU supported, which depends on a) the size of metadata of video memory space allocated for FW to manage the vGPUs. b) how user divide the resources. E.g. if a card has 48GB video memory, and user creates two vGPUs each has 24GB video memory. Only two VFs are usable even SRIOV num_vfs can be large than that. > The way the SRIOV flow should work is you boot the PF, startup the > device, then userspace sets num_vfs and you get the SRIOV VFs. > > Why would you want less/more partitions than VFs? Is there some way to > consume more than one partition per VF? No. > > At least based on the commit message this seems like a very poor FW > interface. > > Jason
On Sun, Oct 13, 2024 at 06:54:32PM +0000, Zhi Wang wrote: > On 27/09/2024 1.51, Jason Gunthorpe wrote: > > On Sun, Sep 22, 2024 at 05:49:26AM -0700, Zhi Wang wrote: > >> GSP firmware needs to know the number of max-supported vGPUs when > >> initialization. > >> > >> The field of VF partition count in the GSP WPR2 is required to be set > >> according to the number of max-supported vGPUs. > >> > >> Set the VF partition count in the GSP WPR2 when NVKM is loading the GSP > >> firmware and initializes the GSP WPR2, if vGPU is enabled. > > > > How/why is this different from the SRIOV num_vfs concept? > > > > 1) The VF is considered as an HW interface of vGPU exposed to the VMM/VM. > > 2) Number of VF is not always equal to number of max vGPU supported, > which depends on a) the size of metadata of video memory space allocated > for FW to manage the vGPUs. b) how user divide the resources. E.g. if a > card has 48GB video memory, and user creates two vGPUs each has 24GB > video memory. Only two VFs are usable even SRIOV num_vfs can be large > than that. But that can't be determine at driver load time, the profiling of the VFs must happen at run time when the orchestation determins what kind of VM instance type to run. Which again gets back to the question of why do you need to specify the number of VFs at FW boot time? Why isn't it just fully dynamic and driven on the SRIOV enable? Jason
On 15/10/2024 15.20, Jason Gunthorpe wrote: > On Sun, Oct 13, 2024 at 06:54:32PM +0000, Zhi Wang wrote: >> On 27/09/2024 1.51, Jason Gunthorpe wrote: >>> On Sun, Sep 22, 2024 at 05:49:26AM -0700, Zhi Wang wrote: >>>> GSP firmware needs to know the number of max-supported vGPUs when >>>> initialization. >>>> >>>> The field of VF partition count in the GSP WPR2 is required to be set >>>> according to the number of max-supported vGPUs. >>>> >>>> Set the VF partition count in the GSP WPR2 when NVKM is loading the GSP >>>> firmware and initializes the GSP WPR2, if vGPU is enabled. >>> >>> How/why is this different from the SRIOV num_vfs concept? >>> >> >> 1) The VF is considered as an HW interface of vGPU exposed to the VMM/VM. >> >> 2) Number of VF is not always equal to number of max vGPU supported, >> which depends on a) the size of metadata of video memory space allocated >> for FW to manage the vGPUs. b) how user divide the resources. E.g. if a >> card has 48GB video memory, and user creates two vGPUs each has 24GB >> video memory. Only two VFs are usable even SRIOV num_vfs can be large >> than that. > > But that can't be determine at driver load time, the profiling of the > VFs must happen at run time when the orchestation determins what kind > of VM instance type to run. > > Which again gets back to the question of why do you need to specify > the number of VFs at FW boot time? Why isn't it just fully dynamic and > driven on the SRIOV enable? > The FW needs to pre-calculate the reserved video memory for its own use, which includes the size of metadata of max-supported vGPUs. It needs to be decided at the FW loading time. We can always set it to the max number and the trade-off is we lose some usable video memory, at around (549-256)MB so far. > Jason
On Tue, Oct 15, 2024 at 03:19:33PM +0000, Zhi Wang wrote: > The FW needs to pre-calculate the reserved video memory for its own use, > which includes the size of metadata of max-supported vGPUs. It needs to > be decided at the FW loading time. We can always set it to the max > number and the trade-off is we lose some usable video memory, at around > (549-256)MB so far. I think that is where you have to end up as we don't really want probe-time configurables, and consider later updating how the FW works to make this more optimal. Jason
diff --git a/drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h b/drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h index 3fbc57b16a05..f52143df45c1 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h @@ -61,6 +61,7 @@ struct nvkm_gsp { } frts, boot, elf, heap; u64 addr; u64 size; + u8 vf_partition_count; } wpr2; struct { u64 addr; diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c index a38a6abcac6f..14fc152d6859 100644 --- a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c @@ -2037,6 +2037,7 @@ r535_gsp_wpr_meta_init(struct nvkm_gsp *gsp) meta->vgaWorkspaceOffset = gsp->fb.bios.vga_workspace.addr; meta->vgaWorkspaceSize = gsp->fb.bios.vga_workspace.size; meta->bootCount = 0; + meta->gspFwHeapVfPartitionCount = gsp->fb.wpr2.vf_partition_count; meta->partitionRpcAddr = 0; meta->partitionRpcRequestOffset = 0; meta->partitionRpcReplyOffset = 0; @@ -2640,6 +2641,7 @@ r535_gsp_oneinit(struct nvkm_gsp *gsp) if (nvkm_vgpu_mgr_is_supported(device)) { gsp->fb.wpr2.heap.size = SZ_256M; + gsp->fb.wpr2.vf_partition_count = NVIDIA_MAX_VGPUS; } else { u32 fb_size_gb = DIV_ROUND_UP_ULL(gsp->fb.size, 1 << 30);
GSP firmware needs to know the number of max-supported vGPUs when initialization. The field of VF partition count in the GSP WPR2 is required to be set according to the number of max-supported vGPUs. Set the VF partition count in the GSP WPR2 when NVKM is loading the GSP firmware and initializes the GSP WPR2, if vGPU is enabled. Cc: Neo Jia <cjia@nvidia.com> Cc: Surath Mitra <smitra@nvidia.com> Signed-off-by: Zhi Wang <zhiw@nvidia.com> --- drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h | 1 + drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c | 2 ++ 2 files changed, 3 insertions(+)