| Message ID | 20221125010026.755-1-lengchao@huawei.com (mailing list archive) |
|---|---|
| State | Accepted |
| Delegated to: | Jason Gunthorpe |
| Headers | show |
| Series | infiniband:cma: change iboe packet life time from 18 to 16 | expand |
Hi, jason
Gently ping, thank you.
On 2022/11/25 9:00, Chao Leng wrote:
> The ack timeout retransmission time is affected by the following two
> factors: one is packet life time, another is the HCA processing time.
> Now the default packet lifetime(CMA_IBOE_PACKET_LIFETIME) is 18.
> That means the minimum ack timeout is 2 seconds(2^(18+1)*4us=2.097seconds).
> The packet lifetime means the maximum transmission time of packets
> on the network, 2 seconds is too long.
> Assume the network is a clos topology with three layers, every packet
> will pass through five hops of switches. Assume the buffer of every
> switch is 128MB and the port transmission rate is 25 Gbit/s,
> the maximum transmission time of the packet is 200ms(128MB*5/25Gbit/s).
> Add double redundancy, it is less than 500ms.
> So change the CMA_IBOE_PACKET_LIFETIME to 16,
> the maximum transmission time of the packet will be about 500+ms,
> it is long enough.
>
> Signed-off-by: Chao Leng <lengchao@huawei.com>
> ---
> drivers/infiniband/core/cma.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index cc2222b85c88..2f5b5e6f3d11 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -47,7 +47,7 @@ MODULE_LICENSE("Dual BSD/GPL");
> #define CMA_CM_RESPONSE_TIMEOUT 20
> #define CMA_MAX_CM_RETRIES 15
> #define CMA_CM_MRA_SETTING (IB_CM_MRA_FLAG_DELAY | 24)
> -#define CMA_IBOE_PACKET_LIFETIME 18
> +#define CMA_IBOE_PACKET_LIFETIME 16
> #define CMA_PREFERRED_ROCE_GID_TYPE IB_GID_TYPE_ROCE_UDP_ENCAP
>
> static const char * const cma_events[] = {
>
On Fri, Nov 25, 2022 at 09:00:26AM +0800, Chao Leng wrote: > The ack timeout retransmission time is affected by the following two > factors: one is packet life time, another is the HCA processing time. > Now the default packet lifetime(CMA_IBOE_PACKET_LIFETIME) is 18. > That means the minimum ack timeout is 2 seconds(2^(18+1)*4us=2.097seconds). > The packet lifetime means the maximum transmission time of packets > on the network, 2 seconds is too long. > Assume the network is a clos topology with three layers, every packet > will pass through five hops of switches. Assume the buffer of every > switch is 128MB and the port transmission rate is 25 Gbit/s, > the maximum transmission time of the packet is 200ms(128MB*5/25Gbit/s). > Add double redundancy, it is less than 500ms. > So change the CMA_IBOE_PACKET_LIFETIME to 16, > the maximum transmission time of the packet will be about 500+ms, > it is long enough. > > Signed-off-by: Chao Leng <lengchao@huawei.com> > --- > drivers/infiniband/core/cma.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) Applied to for-next, thanks Jason
Is it possible to make this parameter tunable? In pure IB fabrics I believe you can.... -----Original Message----- From: Jason Gunthorpe <jgg@nvidia.com> Sent: Wednesday, December 7, 2022 6:49 PM To: Chao Leng <lengchao@huawei.com> Cc: leon@kernel.org; linux-rdma@vger.kernel.org Subject: Re: [PATCH] infiniband:cma: change iboe packet life time from 18 to 16 On Fri, Nov 25, 2022 at 09:00:26AM +0800, Chao Leng wrote: > The ack timeout retransmission time is affected by the following two > factors: one is packet life time, another is the HCA processing time. > Now the default packet lifetime(CMA_IBOE_PACKET_LIFETIME) is 18. > That means the minimum ack timeout is 2 seconds(2^(18+1)*4us=2.097seconds). > The packet lifetime means the maximum transmission time of packets on > the network, 2 seconds is too long. > Assume the network is a clos topology with three layers, every packet > will pass through five hops of switches. Assume the buffer of every > switch is 128MB and the port transmission rate is 25 Gbit/s, the > maximum transmission time of the packet is 200ms(128MB*5/25Gbit/s). > Add double redundancy, it is less than 500ms. > So change the CMA_IBOE_PACKET_LIFETIME to 16, the maximum transmission > time of the packet will be about 500+ms, it is long enough. > > Signed-off-by: Chao Leng <lengchao@huawei.com> > --- > drivers/infiniband/core/cma.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) Applied to for-next, thanks Jason This correspondence, and any attachments or files transmitted with this correspondence, contains information which may be confidential and privileged and is intended solely for the use of the addressee. Unless you are the addressee or are authorized to receive messages for the addressee, you may not use, copy, disseminate, or disclose this correspondence or any information contained in this correspondence to any third party. If you have received this correspondence in error, please notify the sender immediately and delete this correspondence and any attachments or files transmitted with this correspondence from your system, and destroy any and all copies thereof, electronic or otherwise. Your cooperation and understanding are greatly appreciated.
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index cc2222b85c88..2f5b5e6f3d11 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -47,7 +47,7 @@ MODULE_LICENSE("Dual BSD/GPL"); #define CMA_CM_RESPONSE_TIMEOUT 20 #define CMA_MAX_CM_RETRIES 15 #define CMA_CM_MRA_SETTING (IB_CM_MRA_FLAG_DELAY | 24) -#define CMA_IBOE_PACKET_LIFETIME 18 +#define CMA_IBOE_PACKET_LIFETIME 16 #define CMA_PREFERRED_ROCE_GID_TYPE IB_GID_TYPE_ROCE_UDP_ENCAP static const char * const cma_events[] = {
The ack timeout retransmission time is affected by the following two factors: one is packet life time, another is the HCA processing time. Now the default packet lifetime(CMA_IBOE_PACKET_LIFETIME) is 18. That means the minimum ack timeout is 2 seconds(2^(18+1)*4us=2.097seconds). The packet lifetime means the maximum transmission time of packets on the network, 2 seconds is too long. Assume the network is a clos topology with three layers, every packet will pass through five hops of switches. Assume the buffer of every switch is 128MB and the port transmission rate is 25 Gbit/s, the maximum transmission time of the packet is 200ms(128MB*5/25Gbit/s). Add double redundancy, it is less than 500ms. So change the CMA_IBOE_PACKET_LIFETIME to 16, the maximum transmission time of the packet will be about 500+ms, it is long enough. Signed-off-by: Chao Leng <lengchao@huawei.com> --- drivers/infiniband/core/cma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)