| Message ID | 1437548143-24893-40-git-send-email-sagig@mellanox.com (mailing list archive) |
|---|---|
| State | RFC |
| Headers | show |
> + IB_DEVICE_MAP_ARB_SG = (1ULL<<32), > +enum ib_mr_flags { > + IB_MR_MAP_ARB_SG = 1, > +}; > + s/ARB_SG/SG_GAPS/? Also please try to document new flags. I know the IB code currently doesn't do it, but starting a trend there would be very useful. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jul 22, 2015 at 09:55:39AM +0300, Sagi Grimberg wrote: > +enum ib_mr_flags { > + IB_MR_MAP_ARB_SG = 1, > +}; Something about this just seems ugly. We are back to what we were trying to avoid: Adding more types of MRs.. Is this really necessary? Do you really need to know the MR type when the MR is created, or can the adaptor change types on the fly during registration? iSER for example has a rarely used corner case where it needs this, but it just turns on the feature unconditionally right away. This incures 2x the overhead in the MR allocations and who knows what performance impact on the adaptor side. It would be so much better if it could switch to this mode on a SG by SG list basis. Same for signature. In other words: It would be so much cleaner if ib_map_mr_sg set the MR type based on the need. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 7/22/2015 8:22 PM, Jason Gunthorpe wrote: > On Wed, Jul 22, 2015 at 09:55:39AM +0300, Sagi Grimberg wrote: >> +enum ib_mr_flags { >> + IB_MR_MAP_ARB_SG = 1, >> +}; > > Something about this just seems ugly. We are back to what we were > trying to avoid: Adding more types of MRs.. > > Is this really necessary? Do you really need to know the MR type when > the MR is created, or can the adaptor change types on the fly during > registration? > > iSER for example has a rarely used corner case where it needs this, I can tell you that its anything but a corner case. direct-io, bio merges, FS operations and PI are examples where most of the sg lists *will* be "gappy". Trust me, it's fairly common to see those... > but it just turns on the feature unconditionally right away. This > incures 2x the overhead in the MR allocations and who knows what > performance impact on the adaptor side. I ran various workloads with this, and performance seems to sustain. > > It would be so much better if it could switch to this mode on a SG by > SG list basis. It would, but unfortunately it can't. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index d543fee..cc83c39 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -133,6 +133,7 @@ enum ib_device_cap_flags { IB_DEVICE_MANAGED_FLOW_STEERING = (1<<29), IB_DEVICE_SIGNATURE_HANDOVER = (1<<30), IB_DEVICE_ON_DEMAND_PAGING = (1<<31), + IB_DEVICE_MAP_ARB_SG = (1ULL<<32), }; enum ib_signature_prot_cap { @@ -193,7 +194,7 @@ struct ib_device_attr { u32 hw_ver; int max_qp; int max_qp_wr; - int device_cap_flags; + u64 device_cap_flags; int max_sge; int max_sge_rd; int max_cq; @@ -556,6 +557,11 @@ __attribute_const__ int ib_rate_to_mult(enum ib_rate rate); */ __attribute_const__ int ib_rate_to_mbps(enum ib_rate rate); +enum ib_mr_flags { + IB_MR_MAP_ARB_SG = 1, +}; + + enum ib_mr_type { IB_MR_TYPE_FAST_REG, IB_MR_TYPE_SIGNATURE,
Signed-off-by: Sagi Grimberg <sagig@mellanox.com> --- include/rdma/ib_verbs.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)