| Message ID | 20210916234100.122368-12-logang@deltatee.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | Userspace P2PDMA with O_DIRECT NVMe devices | expand |
On Thu, Sep 16, 2021 at 05:40:51PM -0600, Logan Gunthorpe wrote: > Introduce the helper function ib_dma_pci_p2p_dma_supported() to check > if a given ib_device can be used in P2PDMA transfers. This ensures > the ib_device is not using virt_dma and also that the underlying > dma_device supports P2PDMA. > > Use the new helper in nvme-rdma to replace the existing check for > ib_uses_virt_dma(). Adding the dma_pci_p2pdma_supported() check allows > switching away from pci_p2pdma_[un]map_sg(). > > Signed-off-by: Logan Gunthorpe <logang@deltatee.com> > --- > drivers/nvme/target/rdma.c | 2 +- > include/rdma/ib_verbs.h | 11 +++++++++++ > 2 files changed, 12 insertions(+), 1 deletion(-) Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> > +/* > + * Check if a IB device's underlying DMA mapping supports P2PDMA transfers. > + */ > +static inline bool ib_dma_pci_p2p_dma_supported(struct ib_device *dev) > +{ > + if (ib_uses_virt_dma(dev)) > + return false; If someone wants to make rxe/hfi/qib use this stuff then they will have to teach the the driver to do all the p2p checks and add some struct ib_device flag Jason
Logan, On 9/17/2021 2:40 AM, Logan Gunthorpe wrote: > Introduce the helper function ib_dma_pci_p2p_dma_supported() to check > if a given ib_device can be used in P2PDMA transfers. This ensures > the ib_device is not using virt_dma and also that the underlying > dma_device supports P2PDMA. > > Use the new helper in nvme-rdma to replace the existing check for > ib_uses_virt_dma(). Adding the dma_pci_p2pdma_supported() check allows > switching away from pci_p2pdma_[un]map_sg(). > > Signed-off-by: Logan Gunthorpe <logang@deltatee.com> > --- > drivers/nvme/target/rdma.c | 2 +- > include/rdma/ib_verbs.h | 11 +++++++++++ > 2 files changed, 12 insertions(+), 1 deletion(-) Looks good, Reviewed-by: Max Gurtovoy <mgurtovoy@nvidia.com>
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index 891174ccd44b..9ea212c187f2 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -414,7 +414,7 @@ static int nvmet_rdma_alloc_rsp(struct nvmet_rdma_device *ndev, if (ib_dma_mapping_error(ndev->device, r->send_sge.addr)) goto out_free_rsp; - if (!ib_uses_virt_dma(ndev->device)) + if (ib_dma_pci_p2p_dma_supported(ndev->device)) r->req.p2p_client = &ndev->device->dev; r->send_sge.length = sizeof(*r->req.cqe); r->send_sge.lkey = ndev->pd->local_dma_lkey; diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 4b50d9a3018a..2b71c9ca2186 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -3986,6 +3986,17 @@ static inline bool ib_uses_virt_dma(struct ib_device *dev) return IS_ENABLED(CONFIG_INFINIBAND_VIRT_DMA) && !dev->dma_device; } +/* + * Check if a IB device's underlying DMA mapping supports P2PDMA transfers. + */ +static inline bool ib_dma_pci_p2p_dma_supported(struct ib_device *dev) +{ + if (ib_uses_virt_dma(dev)) + return false; + + return dma_pci_p2pdma_supported(dev->dma_device); +} + /** * ib_dma_mapping_error - check a DMA addr for error * @dev: The device for which the dma_addr was created
Introduce the helper function ib_dma_pci_p2p_dma_supported() to check if a given ib_device can be used in P2PDMA transfers. This ensures the ib_device is not using virt_dma and also that the underlying dma_device supports P2PDMA. Use the new helper in nvme-rdma to replace the existing check for ib_uses_virt_dma(). Adding the dma_pci_p2pdma_supported() check allows switching away from pci_p2pdma_[un]map_sg(). Signed-off-by: Logan Gunthorpe <logang@deltatee.com> --- drivers/nvme/target/rdma.c | 2 +- include/rdma/ib_verbs.h | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-)