| Message ID | 20230423141330.40437-1-mgurtovoy@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Fix failover to non integrity NVMe path | expand |
On Sun, Apr 23, 2023 at 05:13:28PM +0300, Max Gurtovoy wrote: > Hi Christoph/Sagi/Martin, > > We're encountered a crash while testing failover between NVMeF/RDMA > paths to a target that expose a namespace with metadata. The scenario is > the following: > Configure one initiator/host path on PI offload capable port (e.g ConnectX-5 > device) and configure second initiator/host path on non PI offload capable > port (e.g ConnectX-3). Hmm. I suspect the right thing to do here is to just fail the second connect.
On 4/24/23 07:11, Christoph Hellwig wrote: > On Sun, Apr 23, 2023 at 05:13:28PM +0300, Max Gurtovoy wrote: >> Hi Christoph/Sagi/Martin, >> >> We're encountered a crash while testing failover between NVMeF/RDMA >> paths to a target that expose a namespace with metadata. The scenario is >> the following: >> Configure one initiator/host path on PI offload capable port (e.g ConnectX-5 >> device) and configure second initiator/host path on non PI offload capable >> port (e.g ConnectX-3). > > Hmm. I suspect the right thing to do here is to just fail the second > connect. Yeah, I'm slightly unhappy with this whole setup. If we were just doing DIF I guess the setup could work, but then we have to disable DIX (as we cannot support integrity data on the non-PI path). But we would need an additional patch to disable DIX functionality in those cases. Cheers, Hannes
On Mon, Apr 24, 2023 at 08:10:59AM +0200, Hannes Reinecke wrote: > Yeah, I'm slightly unhappy with this whole setup. > If we were just doing DIF I guess the setup could work, but then we have to > disable DIX (as we cannot support integrity data on the non-PI path). > But we would need an additional patch to disable DIX functionality in those > cases. NVMeoF only supports inline integrity data, the remapping from out of line integrity data is always done by the HCA for NVMe over RDMA, and integrity data is not supported without that. Because of that I can't see how we could sensibly support one path with integrity offload and one without. And yes, it might make sense to offer a way to explicitly disable integrity support to allow forming such a multipath setup.
>> Yeah, I'm slightly unhappy with this whole setup. >> If we were just doing DIF I guess the setup could work, but then we have to >> disable DIX (as we cannot support integrity data on the non-PI path). >> But we would need an additional patch to disable DIX functionality in those >> cases. > > NVMeoF only supports inline integrity data, the remapping from out of > line integrity data is always done by the HCA for NVMe over RDMA, > and integrity data is not supported without that. > > Because of that I can't see how we could sensibly support one path with > integrity offload and one without. And yes, it might make sense to > offer a way to explicitly disable integrity support to allow forming such > a multipath setup. I agree. I didn't read through the change log well enough, I thought that one path is DIF and the other is DIX. I agree that we should not permit such a configuration.
On 24/04/2023 11:53, Sagi Grimberg wrote: > >>> Yeah, I'm slightly unhappy with this whole setup. >>> If we were just doing DIF I guess the setup could work, but then we >>> have to >>> disable DIX (as we cannot support integrity data on the non-PI path). >>> But we would need an additional patch to disable DIX functionality in >>> those >>> cases. >> >> NVMeoF only supports inline integrity data, the remapping from out of >> line integrity data is always done by the HCA for NVMe over RDMA, >> and integrity data is not supported without that. >> >> Because of that I can't see how we could sensibly support one path with >> integrity offload and one without. And yes, it might make sense to >> offer a way to explicitly disable integrity support to allow forming such >> a multipath setup. > > I agree. I didn't read through the change log well enough, I thought > that one path is DIF and the other is DIX. > > I agree that we should not permit such a configuration. I'm not yet convinced why not to permit it. The spec allows this to happen and I can think about scenarios that users will want this kind of configuration. We also support it today (with the exception on this bug). There is a special PRACT bit in the spec that asks the controller to take action upon each R/W IO request. The Multipath is not related to md IMO. One path can generate/verify md and other can raise PRACT bit. You can also create 2 paths from different hosts to same target and one will have ConnectX-5 and other ConnectX-3. The fact that these path are from the same host is not so important IMO.