| Message ID | 1515388465-9307-1-git-send-email-yanjun.zhu@oracle.com (mailing list archive) |
|---|---|
| State | Accepted |
| Headers | show |
On Mon, 2018-01-08 at 00:14 -0500, Zhu Yanjun wrote: > In xmit, there is a skb_clone. This function copies the struct sk_buff. > And some parameters are changed to the new skb. Then the new skb is sent > while the old skb is freed. > > While the function skb_clone is removed, the parameter changes are made on > the old skb, then the old skb is sent. It can also work well. > > The following tests are made. > > server client > --------- --------- > > 1.1.1.1|<----rxe-channel--->|1.1.1.2| > > --------- --------- > > On server: rping -s -a 1.1.1.1 -v -C 1000 -S 512 > On client: rping -c -a 1.1.1.1 -v -C 1000 -S 512 > > The kernel config CONFIG_DEBUG_KMEMLEAK is enabled on both server > and client. > > This test runs for several hours. There is no memory leak and the whole > system can work well. > > As the above network, the following tests are made. > > Server: ibv_rc_pingpong -d rxe0 -g 1 > Client: ibv_rc_pingpong -d rxe0 -g 1 1.1.1.1 > > The result on Server. > Before: > 8192000 bytes in 0.88 seconds = 74.36 Mbit/sec > 1000 iters in 0.88 seconds = 881.30 usec/iter > > After: > 8192000 bytes in 0.81 seconds = 81.15 Mbit/sec > 1000 iters in 0.81 seconds = 807.62 usec/iter > > The throughput is enhanced and the latency is reduced. > > CC: Srinivas Eeda <srinivas.eeda@oracle.com> > CC: Joe Jin <joe.jin@oracle.com> > CC: Junxiao Bi <junxiao.bi@oracle.com> > Signed-off-by: Zhu Yanjun <yanjun.zhu@oracle.com> Thanks, applied.
diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index 59dee10..5fc6bb5 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -452,31 +452,26 @@ static void rxe_skb_tx_dtor(struct sk_buff *skb) int rxe_send(struct rxe_dev *rxe, struct rxe_pkt_info *pkt, struct sk_buff *skb) { - struct sk_buff *nskb; struct rxe_av *av; int err; av = rxe_get_av(pkt); - nskb = skb_clone(skb, GFP_ATOMIC); - if (!nskb) - return -ENOMEM; - - nskb->destructor = rxe_skb_tx_dtor; - nskb->sk = pkt->qp->sk->sk; + skb->destructor = rxe_skb_tx_dtor; + skb->sk = pkt->qp->sk->sk; rxe_add_ref(pkt->qp); atomic_inc(&pkt->qp->skb_out); if (av->network_type == RDMA_NETWORK_IPV4) { - err = ip_local_out(dev_net(skb_dst(skb)->dev), nskb->sk, nskb); + err = ip_local_out(dev_net(skb_dst(skb)->dev), skb->sk, skb); } else if (av->network_type == RDMA_NETWORK_IPV6) { - err = ip6_local_out(dev_net(skb_dst(skb)->dev), nskb->sk, nskb); + err = ip6_local_out(dev_net(skb_dst(skb)->dev), skb->sk, skb); } else { pr_err("Unknown layer 3 protocol: %d\n", av->network_type); atomic_dec(&pkt->qp->skb_out); rxe_drop_ref(pkt->qp); - kfree_skb(nskb); + kfree_skb(skb); return -EINVAL; } @@ -485,7 +480,6 @@ int rxe_send(struct rxe_dev *rxe, struct rxe_pkt_info *pkt, struct sk_buff *skb) return -EAGAIN; } - kfree_skb(skb); return 0; }
In xmit, there is a skb_clone. This function copies the struct sk_buff. And some parameters are changed to the new skb. Then the new skb is sent while the old skb is freed. While the function skb_clone is removed, the parameter changes are made on the old skb, then the old skb is sent. It can also work well. The following tests are made. server client --------- --------- |1.1.1.1|<----rxe-channel--->|1.1.1.2| --------- --------- On server: rping -s -a 1.1.1.1 -v -C 1000 -S 512 On client: rping -c -a 1.1.1.1 -v -C 1000 -S 512 The kernel config CONFIG_DEBUG_KMEMLEAK is enabled on both server and client. This test runs for several hours. There is no memory leak and the whole system can work well. As the above network, the following tests are made. Server: ibv_rc_pingpong -d rxe0 -g 1 Client: ibv_rc_pingpong -d rxe0 -g 1 1.1.1.1 The result on Server. Before: 8192000 bytes in 0.88 seconds = 74.36 Mbit/sec 1000 iters in 0.88 seconds = 881.30 usec/iter After: 8192000 bytes in 0.81 seconds = 81.15 Mbit/sec 1000 iters in 0.81 seconds = 807.62 usec/iter The throughput is enhanced and the latency is reduced. CC: Srinivas Eeda <srinivas.eeda@oracle.com> CC: Joe Jin <joe.jin@oracle.com> CC: Junxiao Bi <junxiao.bi@oracle.com> Signed-off-by: Zhu Yanjun <yanjun.zhu@oracle.com> --- drivers/infiniband/sw/rxe/rxe_net.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) --- V1->V2: Add the test results. ---