From patchwork Wed May 1 14:44:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4AA761515 for ; Wed, 1 May 2019 14:45:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B04228AD0 for ; Wed, 1 May 2019 14:45:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3942C28B56; Wed, 1 May 2019 14:45:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 250F328B8C for ; Wed, 1 May 2019 14:45:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727068AbfEAOo7 (ORCPT ); Wed, 1 May 2019 10:44:59 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40075 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727053AbfEAOo4 (ORCPT ); Wed, 1 May 2019 10:44:56 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:50 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41Eiogd019831; Wed, 1 May 2019 17:44:50 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41Eiong036038; Wed, 1 May 2019 17:44:50 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41EioKV036037; Wed, 1 May 2019 17:44:50 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 1/9] linux/dim: Move logic to dim.h Date: Wed, 1 May 2019 17:44:31 +0300 Message-Id: <1556721879-35987-2-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In preparation for supporting more implementations of the DIM algorithm, I'm moving what would become common logic to a common library. Downstream DIM implementations will use the common lib for their implementation. Signed-off-by: Tal Gilboa --- include/linux/dim.h | 182 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/net_dim.h | 148 +-------------------------------------- 2 files changed, 184 insertions(+), 146 deletions(-) create mode 100644 include/linux/dim.h diff --git a/include/linux/dim.h b/include/linux/dim.h new file mode 100644 index 0000000..d06f6e4 --- /dev/null +++ b/include/linux/dim.h @@ -0,0 +1,182 @@ +/* + * Copyright (c) 2016, Mellanox Technologies. All rights reserved. + * Copyright (c) 2017-2018, Broadcom Limited. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef DIM_H +#define DIM_H + +#include + +#define NET_DIM_NEVENTS 64 +#define IS_SIGNIFICANT_DIFF(val, ref) \ + (((100UL * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */ +#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1)) + +struct net_dim_cq_moder { + u16 usec; + u16 pkts; + u8 cq_period_mode; +}; + +struct net_dim_sample { + ktime_t time; + u32 pkt_ctr; + u32 byte_ctr; + u16 event_ctr; +}; + +struct net_dim_stats { + int ppms; /* packets per msec */ + int bpms; /* bytes per msec */ + int epms; /* events per msec */ +}; + +struct net_dim { /* Dynamic Interrupt Moderation */ + u8 state; + struct net_dim_stats prev_stats; + struct net_dim_sample start_sample; + struct work_struct work; + u8 profile_ix; + u8 mode; + u8 tune_state; + u8 steps_right; + u8 steps_left; + u8 tired; +}; + +enum { + NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE = 0x0, + NET_DIM_CQ_PERIOD_MODE_START_FROM_CQE = 0x1, + NET_DIM_CQ_PERIOD_NUM_MODES +}; + +enum { + NET_DIM_START_MEASURE, + NET_DIM_MEASURE_IN_PROGRESS, + NET_DIM_APPLY_NEW_PROFILE, +}; + +enum { + NET_DIM_PARKING_ON_TOP, + NET_DIM_PARKING_TIRED, + NET_DIM_GOING_RIGHT, + NET_DIM_GOING_LEFT, +}; + +enum { + NET_DIM_STATS_WORSE, + NET_DIM_STATS_SAME, + NET_DIM_STATS_BETTER, +}; + +enum { + NET_DIM_STEPPED, + NET_DIM_TOO_TIRED, + NET_DIM_ON_EDGE, +}; + +static inline bool net_dim_on_top(struct net_dim *net_dim) +{ + switch (net_dim->tune_state) { + case NET_DIM_PARKING_ON_TOP: + case NET_DIM_PARKING_TIRED: + return true; + case NET_DIM_GOING_RIGHT: + return (net_dim->steps_left > 1) && (net_dim->steps_right == 1); + default: /* NET_DIM_GOING_LEFT */ + return (net_dim->steps_right > 1) && (net_dim->steps_left == 1); + } +} + +static inline void net_dim_turn(struct net_dim *net_dim) +{ + switch (net_dim->tune_state) { + case NET_DIM_PARKING_ON_TOP: + case NET_DIM_PARKING_TIRED: + break; + case NET_DIM_GOING_RIGHT: + net_dim->tune_state = NET_DIM_GOING_LEFT; + net_dim->steps_left = 0; + break; + case NET_DIM_GOING_LEFT: + net_dim->tune_state = NET_DIM_GOING_RIGHT; + net_dim->steps_right = 0; + break; + } +} + +static inline void net_dim_park_on_top(struct net_dim *net_dim) +{ + net_dim->steps_right = 0; + net_dim->steps_left = 0; + net_dim->tired = 0; + net_dim->tune_state = NET_DIM_PARKING_ON_TOP; +} + +static inline void net_dim_park_tired(struct net_dim *net_dim) +{ + net_dim->steps_right = 0; + net_dim->steps_left = 0; + net_dim->tune_state = NET_DIM_PARKING_TIRED; +} + +static inline void net_dim_sample(u16 event_ctr, + u64 packets, + u64 bytes, + struct net_dim_sample *s) +{ + s->time = ktime_get(); + s->pkt_ctr = packets; + s->byte_ctr = bytes; + s->event_ctr = event_ctr; +} + +static inline void net_dim_calc_stats(struct net_dim_sample *start, + struct net_dim_sample *end, + struct net_dim_stats *curr_stats) +{ + /* u32 holds up to 71 minutes, should be enough */ + u32 delta_us = ktime_us_delta(end->time, start->time); + u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr); + u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr, + start->byte_ctr); + + if (!delta_us) + return; + + curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us); + curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us); + curr_stats->epms = DIV_ROUND_UP(NET_DIM_NEVENTS * USEC_PER_MSEC, + delta_us); +} + +#endif /* DIM_H */ diff --git a/include/linux/net_dim.h b/include/linux/net_dim.h index fd45838..373cda7 100644 --- a/include/linux/net_dim.h +++ b/include/linux/net_dim.h @@ -35,73 +35,10 @@ #define NET_DIM_H #include - -struct net_dim_cq_moder { - u16 usec; - u16 pkts; - u8 cq_period_mode; -}; - -struct net_dim_sample { - ktime_t time; - u32 pkt_ctr; - u32 byte_ctr; - u16 event_ctr; -}; - -struct net_dim_stats { - int ppms; /* packets per msec */ - int bpms; /* bytes per msec */ - int epms; /* events per msec */ -}; - -struct net_dim { /* Adaptive Moderation */ - u8 state; - struct net_dim_stats prev_stats; - struct net_dim_sample start_sample; - struct work_struct work; - u8 profile_ix; - u8 mode; - u8 tune_state; - u8 steps_right; - u8 steps_left; - u8 tired; -}; - -enum { - NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE = 0x0, - NET_DIM_CQ_PERIOD_MODE_START_FROM_CQE = 0x1, - NET_DIM_CQ_PERIOD_NUM_MODES -}; - -/* Adaptive moderation logic */ -enum { - NET_DIM_START_MEASURE, - NET_DIM_MEASURE_IN_PROGRESS, - NET_DIM_APPLY_NEW_PROFILE, -}; - -enum { - NET_DIM_PARKING_ON_TOP, - NET_DIM_PARKING_TIRED, - NET_DIM_GOING_RIGHT, - NET_DIM_GOING_LEFT, -}; - -enum { - NET_DIM_STATS_WORSE, - NET_DIM_STATS_SAME, - NET_DIM_STATS_BETTER, -}; - -enum { - NET_DIM_STEPPED, - NET_DIM_TOO_TIRED, - NET_DIM_ON_EDGE, -}; +#include #define NET_DIM_PARAMS_NUM_PROFILES 5 -/* Adaptive moderation profiles */ +/* Netdev dynamic interrupt moderation profiles */ #define NET_DIM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256 #define NET_DIM_DEFAULT_TX_CQ_MODERATION_PKTS_FROM_EQE 128 #define NET_DIM_DEF_PROFILE_CQE 1 @@ -188,36 +125,6 @@ enum { return net_dim_get_tx_moderation(cq_period_mode, profile_ix); } -static inline bool net_dim_on_top(struct net_dim *dim) -{ - switch (dim->tune_state) { - case NET_DIM_PARKING_ON_TOP: - case NET_DIM_PARKING_TIRED: - return true; - case NET_DIM_GOING_RIGHT: - return (dim->steps_left > 1) && (dim->steps_right == 1); - default: /* NET_DIM_GOING_LEFT */ - return (dim->steps_right > 1) && (dim->steps_left == 1); - } -} - -static inline void net_dim_turn(struct net_dim *dim) -{ - switch (dim->tune_state) { - case NET_DIM_PARKING_ON_TOP: - case NET_DIM_PARKING_TIRED: - break; - case NET_DIM_GOING_RIGHT: - dim->tune_state = NET_DIM_GOING_LEFT; - dim->steps_left = 0; - break; - case NET_DIM_GOING_LEFT: - dim->tune_state = NET_DIM_GOING_RIGHT; - dim->steps_right = 0; - break; - } -} - static inline int net_dim_step(struct net_dim *dim) { if (dim->tired == (NET_DIM_PARAMS_NUM_PROFILES * 2)) @@ -245,21 +152,6 @@ static inline int net_dim_step(struct net_dim *dim) return NET_DIM_STEPPED; } -static inline void net_dim_park_on_top(struct net_dim *dim) -{ - dim->steps_right = 0; - dim->steps_left = 0; - dim->tired = 0; - dim->tune_state = NET_DIM_PARKING_ON_TOP; -} - -static inline void net_dim_park_tired(struct net_dim *dim) -{ - dim->steps_right = 0; - dim->steps_left = 0; - dim->tune_state = NET_DIM_PARKING_TIRED; -} - static inline void net_dim_exit_parking(struct net_dim *dim) { dim->tune_state = dim->profile_ix ? NET_DIM_GOING_LEFT : @@ -267,9 +159,6 @@ static inline void net_dim_exit_parking(struct net_dim *dim) net_dim_step(dim); } -#define IS_SIGNIFICANT_DIFF(val, ref) \ - (((100UL * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */ - static inline int net_dim_stats_compare(struct net_dim_stats *curr, struct net_dim_stats *prev) { @@ -351,39 +240,6 @@ static inline bool net_dim_decision(struct net_dim_stats *curr_stats, return dim->profile_ix != prev_ix; } -static inline void net_dim_sample(u16 event_ctr, - u64 packets, - u64 bytes, - struct net_dim_sample *s) -{ - s->time = ktime_get(); - s->pkt_ctr = packets; - s->byte_ctr = bytes; - s->event_ctr = event_ctr; -} - -#define NET_DIM_NEVENTS 64 -#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1)) - -static inline void net_dim_calc_stats(struct net_dim_sample *start, - struct net_dim_sample *end, - struct net_dim_stats *curr_stats) -{ - /* u32 holds up to 71 minutes, should be enough */ - u32 delta_us = ktime_us_delta(end->time, start->time); - u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr); - u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr, - start->byte_ctr); - - if (!delta_us) - return; - - curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us); - curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us); - curr_stats->epms = DIV_ROUND_UP(NET_DIM_NEVENTS * USEC_PER_MSEC, - delta_us); -} - static inline void net_dim(struct net_dim *dim, struct net_dim_sample end_sample) { From patchwork Wed May 1 14:44:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925187 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6986917E6 for ; Wed, 1 May 2019 14:45:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A67428BA6 for ; Wed, 1 May 2019 14:45:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4EE0A28D53; Wed, 1 May 2019 14:45:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 363DD28BA6 for ; Wed, 1 May 2019 14:45:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727051AbfEAOpX (ORCPT ); Wed, 1 May 2019 10:45:23 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40070 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727048AbfEAOo4 (ORCPT ); Wed, 1 May 2019 10:44:56 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:51 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41Eiphk019837; Wed, 1 May 2019 17:44:51 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41Eip1p036040; Wed, 1 May 2019 17:44:51 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41Eip1H036039; Wed, 1 May 2019 17:44:51 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 2/9] linux/dim: Remove "net" prefix from internal DIM members Date: Wed, 1 May 2019 17:44:32 +0300 Message-Id: <1556721879-35987-3-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Only renaming functions and structs which aren't used by an external code. Signed-off-by: Tal Gilboa --- include/linux/dim.h | 88 ++++++++++++++++++++++++------------------------- include/linux/net_dim.h | 86 +++++++++++++++++++++++------------------------ 2 files changed, 86 insertions(+), 88 deletions(-) diff --git a/include/linux/dim.h b/include/linux/dim.h index d06f6e4..e5e3d0cf 100644 --- a/include/linux/dim.h +++ b/include/linux/dim.h @@ -36,7 +36,7 @@ #include -#define NET_DIM_NEVENTS 64 +#define DIM_NEVENTS 64 #define IS_SIGNIFICANT_DIFF(val, ref) \ (((100UL * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */ #define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1)) @@ -54,7 +54,7 @@ struct net_dim_sample { u16 event_ctr; }; -struct net_dim_stats { +struct dim_stats { int ppms; /* packets per msec */ int bpms; /* bytes per msec */ int epms; /* events per msec */ @@ -62,7 +62,7 @@ struct net_dim_stats { struct net_dim { /* Dynamic Interrupt Moderation */ u8 state; - struct net_dim_stats prev_stats; + struct dim_stats prev_stats; struct net_dim_sample start_sample; struct work_struct work; u8 profile_ix; @@ -86,67 +86,67 @@ enum { }; enum { - NET_DIM_PARKING_ON_TOP, - NET_DIM_PARKING_TIRED, - NET_DIM_GOING_RIGHT, - NET_DIM_GOING_LEFT, + DIM_PARKING_ON_TOP, + DIM_PARKING_TIRED, + DIM_GOING_RIGHT, + DIM_GOING_LEFT, }; enum { - NET_DIM_STATS_WORSE, - NET_DIM_STATS_SAME, - NET_DIM_STATS_BETTER, + DIM_STATS_WORSE, + DIM_STATS_SAME, + DIM_STATS_BETTER, }; enum { - NET_DIM_STEPPED, - NET_DIM_TOO_TIRED, - NET_DIM_ON_EDGE, + DIM_STEPPED, + DIM_TOO_TIRED, + DIM_ON_EDGE, }; -static inline bool net_dim_on_top(struct net_dim *net_dim) +static inline bool dim_on_top(struct net_dim *dim) { - switch (net_dim->tune_state) { - case NET_DIM_PARKING_ON_TOP: - case NET_DIM_PARKING_TIRED: + switch (dim->tune_state) { + case DIM_PARKING_ON_TOP: + case DIM_PARKING_TIRED: return true; - case NET_DIM_GOING_RIGHT: - return (net_dim->steps_left > 1) && (net_dim->steps_right == 1); - default: /* NET_DIM_GOING_LEFT */ - return (net_dim->steps_right > 1) && (net_dim->steps_left == 1); + case DIM_GOING_RIGHT: + return (dim->steps_left > 1) && (dim->steps_right == 1); + default: /* DIM_GOING_LEFT */ + return (dim->steps_right > 1) && (dim->steps_left == 1); } } -static inline void net_dim_turn(struct net_dim *net_dim) +static inline void dim_turn(struct net_dim *dim) { - switch (net_dim->tune_state) { - case NET_DIM_PARKING_ON_TOP: - case NET_DIM_PARKING_TIRED: + switch (dim->tune_state) { + case DIM_PARKING_ON_TOP: + case DIM_PARKING_TIRED: break; - case NET_DIM_GOING_RIGHT: - net_dim->tune_state = NET_DIM_GOING_LEFT; - net_dim->steps_left = 0; + case DIM_GOING_RIGHT: + dim->tune_state = DIM_GOING_LEFT; + dim->steps_left = 0; break; - case NET_DIM_GOING_LEFT: - net_dim->tune_state = NET_DIM_GOING_RIGHT; - net_dim->steps_right = 0; + case DIM_GOING_LEFT: + dim->tune_state = DIM_GOING_RIGHT; + dim->steps_right = 0; break; } } -static inline void net_dim_park_on_top(struct net_dim *net_dim) +static inline void dim_park_on_top(struct net_dim *dim) { - net_dim->steps_right = 0; - net_dim->steps_left = 0; - net_dim->tired = 0; - net_dim->tune_state = NET_DIM_PARKING_ON_TOP; + dim->steps_right = 0; + dim->steps_left = 0; + dim->tired = 0; + dim->tune_state = DIM_PARKING_ON_TOP; } -static inline void net_dim_park_tired(struct net_dim *net_dim) +static inline void dim_park_tired(struct net_dim *dim) { - net_dim->steps_right = 0; - net_dim->steps_left = 0; - net_dim->tune_state = NET_DIM_PARKING_TIRED; + dim->steps_right = 0; + dim->steps_left = 0; + dim->tune_state = DIM_PARKING_TIRED; } static inline void net_dim_sample(u16 event_ctr, @@ -160,9 +160,9 @@ static inline void net_dim_sample(u16 event_ctr, s->event_ctr = event_ctr; } -static inline void net_dim_calc_stats(struct net_dim_sample *start, - struct net_dim_sample *end, - struct net_dim_stats *curr_stats) +static inline void dim_calc_stats(struct net_dim_sample *start, + struct net_dim_sample *end, + struct dim_stats *curr_stats) { /* u32 holds up to 71 minutes, should be enough */ u32 delta_us = ktime_us_delta(end->time, start->time); @@ -175,7 +175,7 @@ static inline void net_dim_calc_stats(struct net_dim_sample *start, curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us); curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us); - curr_stats->epms = DIV_ROUND_UP(NET_DIM_NEVENTS * USEC_PER_MSEC, + curr_stats->epms = DIV_ROUND_UP(DIM_NEVENTS * USEC_PER_MSEC, delta_us); } diff --git a/include/linux/net_dim.h b/include/linux/net_dim.h index 373cda7..3d9a2ff 100644 --- a/include/linux/net_dim.h +++ b/include/linux/net_dim.h @@ -128,67 +128,67 @@ static inline int net_dim_step(struct net_dim *dim) { if (dim->tired == (NET_DIM_PARAMS_NUM_PROFILES * 2)) - return NET_DIM_TOO_TIRED; + return DIM_TOO_TIRED; switch (dim->tune_state) { - case NET_DIM_PARKING_ON_TOP: - case NET_DIM_PARKING_TIRED: + case DIM_PARKING_ON_TOP: + case DIM_PARKING_TIRED: break; - case NET_DIM_GOING_RIGHT: + case DIM_GOING_RIGHT: if (dim->profile_ix == (NET_DIM_PARAMS_NUM_PROFILES - 1)) - return NET_DIM_ON_EDGE; + return DIM_ON_EDGE; dim->profile_ix++; dim->steps_right++; break; - case NET_DIM_GOING_LEFT: + case DIM_GOING_LEFT: if (dim->profile_ix == 0) - return NET_DIM_ON_EDGE; + return DIM_ON_EDGE; dim->profile_ix--; dim->steps_left++; break; } dim->tired++; - return NET_DIM_STEPPED; + return DIM_STEPPED; } static inline void net_dim_exit_parking(struct net_dim *dim) { - dim->tune_state = dim->profile_ix ? NET_DIM_GOING_LEFT : - NET_DIM_GOING_RIGHT; + dim->tune_state = dim->profile_ix ? DIM_GOING_LEFT : + DIM_GOING_RIGHT; net_dim_step(dim); } -static inline int net_dim_stats_compare(struct net_dim_stats *curr, - struct net_dim_stats *prev) +static inline int net_dim_stats_compare(struct dim_stats *curr, + struct dim_stats *prev) { if (!prev->bpms) - return curr->bpms ? NET_DIM_STATS_BETTER : - NET_DIM_STATS_SAME; + return curr->bpms ? DIM_STATS_BETTER : + DIM_STATS_SAME; if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms)) - return (curr->bpms > prev->bpms) ? NET_DIM_STATS_BETTER : - NET_DIM_STATS_WORSE; + return (curr->bpms > prev->bpms) ? DIM_STATS_BETTER : + DIM_STATS_WORSE; if (!prev->ppms) - return curr->ppms ? NET_DIM_STATS_BETTER : - NET_DIM_STATS_SAME; + return curr->ppms ? DIM_STATS_BETTER : + DIM_STATS_SAME; if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms)) - return (curr->ppms > prev->ppms) ? NET_DIM_STATS_BETTER : - NET_DIM_STATS_WORSE; + return (curr->ppms > prev->ppms) ? DIM_STATS_BETTER : + DIM_STATS_WORSE; if (!prev->epms) - return NET_DIM_STATS_SAME; + return DIM_STATS_SAME; if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms)) - return (curr->epms < prev->epms) ? NET_DIM_STATS_BETTER : - NET_DIM_STATS_WORSE; + return (curr->epms < prev->epms) ? DIM_STATS_BETTER : + DIM_STATS_WORSE; - return NET_DIM_STATS_SAME; + return DIM_STATS_SAME; } -static inline bool net_dim_decision(struct net_dim_stats *curr_stats, +static inline bool net_dim_decision(struct dim_stats *curr_stats, struct net_dim *dim) { int prev_state = dim->tune_state; @@ -197,44 +197,43 @@ static inline bool net_dim_decision(struct net_dim_stats *curr_stats, int step_res; switch (dim->tune_state) { - case NET_DIM_PARKING_ON_TOP: + case DIM_PARKING_ON_TOP: stats_res = net_dim_stats_compare(curr_stats, &dim->prev_stats); - if (stats_res != NET_DIM_STATS_SAME) + if (stats_res != DIM_STATS_SAME) net_dim_exit_parking(dim); break; - case NET_DIM_PARKING_TIRED: + case DIM_PARKING_TIRED: dim->tired--; if (!dim->tired) net_dim_exit_parking(dim); break; - case NET_DIM_GOING_RIGHT: - case NET_DIM_GOING_LEFT: + case DIM_GOING_RIGHT: + case DIM_GOING_LEFT: stats_res = net_dim_stats_compare(curr_stats, &dim->prev_stats); - if (stats_res != NET_DIM_STATS_BETTER) - net_dim_turn(dim); + if (stats_res != DIM_STATS_BETTER) + dim_turn(dim); - if (net_dim_on_top(dim)) { - net_dim_park_on_top(dim); + if (dim_on_top(dim)) { + dim_park_on_top(dim); break; } step_res = net_dim_step(dim); switch (step_res) { - case NET_DIM_ON_EDGE: - net_dim_park_on_top(dim); + case DIM_ON_EDGE: + dim_park_on_top(dim); break; - case NET_DIM_TOO_TIRED: - net_dim_park_tired(dim); + case DIM_TOO_TIRED: + dim_park_tired(dim); break; } break; } - if ((prev_state != NET_DIM_PARKING_ON_TOP) || - (dim->tune_state != NET_DIM_PARKING_ON_TOP)) + if (prev_state != DIM_PARKING_ON_TOP || dim->tune_state != DIM_PARKING_ON_TOP) dim->prev_stats = *curr_stats; return dim->profile_ix != prev_ix; @@ -243,7 +242,7 @@ static inline bool net_dim_decision(struct net_dim_stats *curr_stats, static inline void net_dim(struct net_dim *dim, struct net_dim_sample end_sample) { - struct net_dim_stats curr_stats; + struct dim_stats curr_stats; u16 nevents; switch (dim->state) { @@ -251,10 +250,9 @@ static inline void net_dim(struct net_dim *dim, nevents = BIT_GAP(BITS_PER_TYPE(u16), end_sample.event_ctr, dim->start_sample.event_ctr); - if (nevents < NET_DIM_NEVENTS) + if (nevents < DIM_NEVENTS) break; - net_dim_calc_stats(&dim->start_sample, &end_sample, - &curr_stats); + dim_calc_stats(&dim->start_sample, &end_sample, &curr_stats); if (net_dim_decision(&curr_stats, dim)) { dim->state = NET_DIM_APPLY_NEW_PROFILE; schedule_work(&dim->work); From patchwork Wed May 1 14:44:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925163 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF5861398 for ; Wed, 1 May 2019 14:45:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BE39028AFC for ; Wed, 1 May 2019 14:45:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BC87E28B70; Wed, 1 May 2019 14:45:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D99BE28B85 for ; Wed, 1 May 2019 14:45:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727010AbfEAOo7 (ORCPT ); Wed, 1 May 2019 10:44:59 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40069 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727055AbfEAOo4 (ORCPT ); Wed, 1 May 2019 10:44:56 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:51 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41EipEY019840; Wed, 1 May 2019 17:44:51 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41EipQa036044; Wed, 1 May 2019 17:44:51 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41EipeW036043; Wed, 1 May 2019 17:44:51 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 3/9] linux/dim: Rename externally exposed macros Date: Wed, 1 May 2019 17:44:33 +0300 Message-Id: <1556721879-35987-4-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Renamed macros in use by external drivers. Signed-off-by: Tal Gilboa --- drivers/net/ethernet/broadcom/bcmsysport.c | 4 ++-- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c | 2 +- drivers/net/ethernet/broadcom/genet/bcmgenet.c | 4 ++-- drivers/net/ethernet/mellanox/mlx5/core/en_dim.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 10 +++++----- include/linux/dim.h | 12 ++++++------ include/linux/net_dim.h | 18 +++++++++--------- 8 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c index bc3ac36..480089f 100644 --- a/drivers/net/ethernet/broadcom/bcmsysport.c +++ b/drivers/net/ethernet/broadcom/bcmsysport.c @@ -1108,7 +1108,7 @@ static void bcm_sysport_dim_work(struct work_struct *work) net_dim_get_rx_moderation(dim->mode, dim->profile_ix); bcm_sysport_set_rx_coalesce(priv, cur_profile.usec, cur_profile.pkts); - dim->state = NET_DIM_START_MEASURE; + dim->state = DIM_START_MEASURE; } /* RX and misc interrupt routine */ @@ -1460,7 +1460,7 @@ static void bcm_sysport_init_dim(struct bcm_sysport_priv *priv, struct bcm_sysport_net_dim *dim = &priv->dim; INIT_WORK(&dim->dim.work, cb); - dim->dim.mode = NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE; + dim->dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE; dim->event_ctr = 0; dim->packets = 0; dim->bytes = 0; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 4c586ba..21db503 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -7703,7 +7703,7 @@ static void bnxt_enable_napi(struct bnxt *bp) if (bp->bnapi[i]->rx_ring) { INIT_WORK(&cpr->dim.work, bnxt_dim_work); - cpr->dim.mode = NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE; + cpr->dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE; } napi_enable(&bp->bnapi[i]->napi); } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c index afa97c8..16a4588 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c @@ -28,5 +28,5 @@ void bnxt_dim_work(struct work_struct *work) cpr->rx_ring_coal.coal_bufs = cur_moder.pkts; bnxt_hwrm_set_ring_coal(bnapi->bp, bnapi); - dim->state = NET_DIM_START_MEASURE; + dim->state = DIM_START_MEASURE; } diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c index 983245c..eee48da 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c @@ -1928,7 +1928,7 @@ static void bcmgenet_dim_work(struct work_struct *work) net_dim_get_rx_moderation(dim->mode, dim->profile_ix); bcmgenet_set_rx_coalesce(ring, cur_profile.usec, cur_profile.pkts); - dim->state = NET_DIM_START_MEASURE; + dim->state = DIM_START_MEASURE; } /* Assign skb to RX DMA descriptor. */ @@ -2085,7 +2085,7 @@ static void bcmgenet_init_dim(struct bcmgenet_rx_ring *ring, struct bcmgenet_net_dim *dim = &ring->dim; INIT_WORK(&dim->dim.work, cb); - dim->dim.mode = NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE; + dim->dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE; dim->event_ctr = 0; dim->packets = 0; dim->bytes = 0; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_dim.c b/drivers/net/ethernet/mellanox/mlx5/core/en_dim.c index d67adf7..a80303a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_dim.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_dim.c @@ -38,7 +38,7 @@ struct mlx5_core_dev *mdev, struct mlx5_core_cq *mcq) { mlx5_core_modify_cq_moderation(mdev, mcq, moder.usec, moder.pkts); - dim->state = NET_DIM_START_MEASURE; + dim->state = DIM_START_MEASURE; } void mlx5e_rx_dim_work(struct work_struct *work) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index b5fdbd3..589ecef 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -681,11 +681,11 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, switch (params->rx_cq_moderation.cq_period_mode) { case MLX5_CQ_PERIOD_MODE_START_FROM_CQE: - rq->dim.mode = NET_DIM_CQ_PERIOD_MODE_START_FROM_CQE; + rq->dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_CQE; break; case MLX5_CQ_PERIOD_MODE_START_FROM_EQE: default: - rq->dim.mode = NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE; + rq->dim.mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE; } rq->page_cache.head = 0; @@ -2238,7 +2238,7 @@ static void mlx5e_build_ico_cq_param(struct mlx5e_priv *priv, mlx5e_build_common_cq_param(priv, param); - param->cq_period_mode = NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE; + param->cq_period_mode = DIM_CQ_PERIOD_MODE_START_FROM_EQE; } static void mlx5e_build_icosq_param(struct mlx5e_priv *priv, @@ -4485,8 +4485,8 @@ static struct net_dim_cq_moder mlx5e_get_def_rx_moderation(u8 cq_period_mode) static u8 mlx5_to_net_dim_cq_period_mode(u8 cq_period_mode) { return cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE ? - NET_DIM_CQ_PERIOD_MODE_START_FROM_CQE : - NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE; + DIM_CQ_PERIOD_MODE_START_FROM_CQE : + DIM_CQ_PERIOD_MODE_START_FROM_EQE; } void mlx5e_set_tx_cq_mode_params(struct mlx5e_params *params, u8 cq_period_mode) diff --git a/include/linux/dim.h b/include/linux/dim.h index e5e3d0cf..67b80e9 100644 --- a/include/linux/dim.h +++ b/include/linux/dim.h @@ -74,15 +74,15 @@ struct net_dim { /* Dynamic Interrupt Moderation */ }; enum { - NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE = 0x0, - NET_DIM_CQ_PERIOD_MODE_START_FROM_CQE = 0x1, - NET_DIM_CQ_PERIOD_NUM_MODES + DIM_CQ_PERIOD_MODE_START_FROM_EQE = 0x0, + DIM_CQ_PERIOD_MODE_START_FROM_CQE = 0x1, + DIM_CQ_PERIOD_NUM_MODES }; enum { - NET_DIM_START_MEASURE, - NET_DIM_MEASURE_IN_PROGRESS, - NET_DIM_APPLY_NEW_PROFILE, + DIM_START_MEASURE, + DIM_MEASURE_IN_PROGRESS, + DIM_APPLY_NEW_PROFILE, }; enum { diff --git a/include/linux/net_dim.h b/include/linux/net_dim.h index 3d9a2ff..d90f96f 100644 --- a/include/linux/net_dim.h +++ b/include/linux/net_dim.h @@ -78,13 +78,13 @@ } static const struct net_dim_cq_moder -rx_profile[NET_DIM_CQ_PERIOD_NUM_MODES][NET_DIM_PARAMS_NUM_PROFILES] = { +rx_profile[DIM_CQ_PERIOD_NUM_MODES][NET_DIM_PARAMS_NUM_PROFILES] = { NET_DIM_RX_EQE_PROFILES, NET_DIM_RX_CQE_PROFILES, }; static const struct net_dim_cq_moder -tx_profile[NET_DIM_CQ_PERIOD_NUM_MODES][NET_DIM_PARAMS_NUM_PROFILES] = { +tx_profile[DIM_CQ_PERIOD_NUM_MODES][NET_DIM_PARAMS_NUM_PROFILES] = { NET_DIM_TX_EQE_PROFILES, NET_DIM_TX_CQE_PROFILES, }; @@ -101,7 +101,7 @@ static inline struct net_dim_cq_moder net_dim_get_def_rx_moderation(u8 cq_period_mode) { - u8 profile_ix = cq_period_mode == NET_DIM_CQ_PERIOD_MODE_START_FROM_CQE ? + u8 profile_ix = cq_period_mode == DIM_CQ_PERIOD_MODE_START_FROM_CQE ? NET_DIM_DEF_PROFILE_CQE : NET_DIM_DEF_PROFILE_EQE; return net_dim_get_rx_moderation(cq_period_mode, profile_ix); @@ -119,7 +119,7 @@ static inline struct net_dim_cq_moder net_dim_get_def_tx_moderation(u8 cq_period_mode) { - u8 profile_ix = cq_period_mode == NET_DIM_CQ_PERIOD_MODE_START_FROM_CQE ? + u8 profile_ix = cq_period_mode == DIM_CQ_PERIOD_MODE_START_FROM_CQE ? NET_DIM_DEF_PROFILE_CQE : NET_DIM_DEF_PROFILE_EQE; return net_dim_get_tx_moderation(cq_period_mode, profile_ix); @@ -246,7 +246,7 @@ static inline void net_dim(struct net_dim *dim, u16 nevents; switch (dim->state) { - case NET_DIM_MEASURE_IN_PROGRESS: + case DIM_MEASURE_IN_PROGRESS: nevents = BIT_GAP(BITS_PER_TYPE(u16), end_sample.event_ctr, dim->start_sample.event_ctr); @@ -254,17 +254,17 @@ static inline void net_dim(struct net_dim *dim, break; dim_calc_stats(&dim->start_sample, &end_sample, &curr_stats); if (net_dim_decision(&curr_stats, dim)) { - dim->state = NET_DIM_APPLY_NEW_PROFILE; + dim->state = DIM_APPLY_NEW_PROFILE; schedule_work(&dim->work); break; } /* fall through */ - case NET_DIM_START_MEASURE: + case DIM_START_MEASURE: net_dim_sample(end_sample.event_ctr, end_sample.pkt_ctr, end_sample.byte_ctr, &dim->start_sample); - dim->state = NET_DIM_MEASURE_IN_PROGRESS; + dim->state = DIM_MEASURE_IN_PROGRESS; break; - case NET_DIM_APPLY_NEW_PROFILE: + case DIM_APPLY_NEW_PROFILE: break; } } From patchwork Wed May 1 14:44:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925189 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC83D1398 for ; Wed, 1 May 2019 14:45:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A72828BC5 for ; Wed, 1 May 2019 14:45:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 96FA928D53; Wed, 1 May 2019 14:45:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2459028BC5 for ; Wed, 1 May 2019 14:45:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727060AbfEAOpY (ORCPT ); Wed, 1 May 2019 10:45:24 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40067 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726949AbfEAOoz (ORCPT ); Wed, 1 May 2019 10:44:55 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:52 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41EiqNR019843; Wed, 1 May 2019 17:44:52 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41EipUF036046; Wed, 1 May 2019 17:44:52 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41EipXr036045; Wed, 1 May 2019 17:44:51 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 4/9] linux/dim: Rename net_dim_sample() to net_dim_create_sample() Date: Wed, 1 May 2019 17:44:34 +0300 Message-Id: <1556721879-35987-5-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In order to avoid confusion between the function and the similarly named struct. In preparation for removing the 'net' prefix from dim members. Signed-off-by: Tal Gilboa --- drivers/net/ethernet/broadcom/bcmsysport.c | 4 ++-- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 ++++---- drivers/net/ethernet/broadcom/genet/bcmgenet.c | 4 ++-- drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 6 ++---- include/linux/dim.h | 8 ++++---- include/linux/net_dim.h | 4 ++-- 6 files changed, 16 insertions(+), 18 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c index 480089f..7a41ec7 100644 --- a/drivers/net/ethernet/broadcom/bcmsysport.c +++ b/drivers/net/ethernet/broadcom/bcmsysport.c @@ -1028,8 +1028,8 @@ static int bcm_sysport_poll(struct napi_struct *napi, int budget) } if (priv->dim.use_dim) { - net_dim_sample(priv->dim.event_ctr, priv->dim.packets, - priv->dim.bytes, &dim_sample); + net_dim_create_sample(priv->dim.event_ctr, priv->dim.packets, + priv->dim.bytes, &dim_sample); net_dim(&priv->dim.dim, dim_sample); } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 21db503..1528620 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -2127,10 +2127,10 @@ static int bnxt_poll(struct napi_struct *napi, int budget) if (bp->flags & BNXT_FLAG_DIM) { struct net_dim_sample dim_sample; - net_dim_sample(cpr->event_ctr, - cpr->rx_packets, - cpr->rx_bytes, - &dim_sample); + net_dim_create_sample(cpr->event_ctr, + cpr->rx_packets, + cpr->rx_bytes, + &dim_sample); net_dim(&cpr->dim, dim_sample); } mmiowb(); diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c index eee48da..8ab2d67 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c @@ -1909,8 +1909,8 @@ static int bcmgenet_rx_poll(struct napi_struct *napi, int budget) } if (ring->dim.use_dim) { - net_dim_sample(ring->dim.event_ctr, ring->dim.packets, - ring->dim.bytes, &dim_sample); + net_dim_create_sample(ring->dim.event_ctr, ring->dim.packets, + ring->dim.bytes, &dim_sample); net_dim(&ring->dim.dim, dim_sample); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c index b4af5e1..6dd8202 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c @@ -53,8 +53,7 @@ static void mlx5e_handle_tx_dim(struct mlx5e_txqsq *sq) if (unlikely(!test_bit(MLX5E_SQ_STATE_AM, &sq->state))) return; - net_dim_sample(sq->cq.event_ctr, stats->packets, stats->bytes, - &dim_sample); + net_dim_create_sample(sq->cq.event_ctr, stats->packets, stats->bytes, &dim_sample); net_dim(&sq->dim, dim_sample); } @@ -66,8 +65,7 @@ static void mlx5e_handle_rx_dim(struct mlx5e_rq *rq) if (unlikely(!test_bit(MLX5E_RQ_STATE_AM, &rq->state))) return; - net_dim_sample(rq->cq.event_ctr, stats->packets, stats->bytes, - &dim_sample); + net_dim_create_sample(rq->cq.event_ctr, stats->packets, stats->bytes, &dim_sample); net_dim(&rq->dim, dim_sample); } diff --git a/include/linux/dim.h b/include/linux/dim.h index 67b80e9..2413021 100644 --- a/include/linux/dim.h +++ b/include/linux/dim.h @@ -149,10 +149,10 @@ static inline void dim_park_tired(struct net_dim *dim) dim->tune_state = DIM_PARKING_TIRED; } -static inline void net_dim_sample(u16 event_ctr, - u64 packets, - u64 bytes, - struct net_dim_sample *s) +static inline void net_dim_create_sample(u16 event_ctr, + u64 packets, + u64 bytes, + struct net_dim_sample *s) { s->time = ktime_get(); s->pkt_ctr = packets; diff --git a/include/linux/net_dim.h b/include/linux/net_dim.h index d90f96f..93c4e21 100644 --- a/include/linux/net_dim.h +++ b/include/linux/net_dim.h @@ -260,8 +260,8 @@ static inline void net_dim(struct net_dim *dim, } /* fall through */ case DIM_START_MEASURE: - net_dim_sample(end_sample.event_ctr, end_sample.pkt_ctr, end_sample.byte_ctr, - &dim->start_sample); + net_dim_create_sample(end_sample.event_ctr, end_sample.pkt_ctr, + end_sample.byte_ctr, &dim->start_sample); dim->state = DIM_MEASURE_IN_PROGRESS; break; case DIM_APPLY_NEW_PROFILE: From patchwork Wed May 1 14:44:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925185 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E94641398 for ; Wed, 1 May 2019 14:45:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D91C528B56 for ; Wed, 1 May 2019 14:45:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D71AD28B7A; Wed, 1 May 2019 14:45:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9356728D43 for ; Wed, 1 May 2019 14:45:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727083AbfEAOpW (ORCPT ); Wed, 1 May 2019 10:45:22 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40073 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726730AbfEAOo4 (ORCPT ); Wed, 1 May 2019 10:44:56 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:52 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41Eiqdf019846; Wed, 1 May 2019 17:44:52 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41Eiq08036048; Wed, 1 May 2019 17:44:52 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41EiqIe036047; Wed, 1 May 2019 17:44:52 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 5/9] linux/dim: Rename externally used net_dim members Date: Wed, 1 May 2019 17:44:35 +0300 Message-Id: <1556721879-35987-6-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Removed 'net' prefix from functions and structs used by external drivers. Signed-off-by: Tal Gilboa --- drivers/net/ethernet/broadcom/bcmsysport.c | 16 ++++++------ drivers/net/ethernet/broadcom/bcmsysport.h | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 10 ++++---- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt_debugfs.c | 4 +-- drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c | 5 ++-- drivers/net/ethernet/broadcom/genet/bcmgenet.c | 14 +++++----- drivers/net/ethernet/broadcom/genet/bcmgenet.h | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en.h | 8 +++--- drivers/net/ethernet/mellanox/mlx5/core/en_dim.c | 10 ++++---- .../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 4 +-- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 12 ++++----- drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 8 +++--- include/linux/dim.h | 28 ++++++++++---------- include/linux/net_dim.h | 30 +++++++++++----------- 15 files changed, 77 insertions(+), 78 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c index 7a41ec7..b2c5b1a 100644 --- a/drivers/net/ethernet/broadcom/bcmsysport.c +++ b/drivers/net/ethernet/broadcom/bcmsysport.c @@ -621,7 +621,7 @@ static int bcm_sysport_set_coalesce(struct net_device *dev, struct ethtool_coalesce *ec) { struct bcm_sysport_priv *priv = netdev_priv(dev); - struct net_dim_cq_moder moder; + struct dim_cq_moder moder; u32 usecs, pkts; unsigned int i; @@ -1004,7 +1004,7 @@ static int bcm_sysport_poll(struct napi_struct *napi, int budget) { struct bcm_sysport_priv *priv = container_of(napi, struct bcm_sysport_priv, napi); - struct net_dim_sample dim_sample; + struct dim_sample dim_sample; unsigned int work_done = 0; work_done = bcm_sysport_desc_rx(priv, budget); @@ -1028,8 +1028,8 @@ static int bcm_sysport_poll(struct napi_struct *napi, int budget) } if (priv->dim.use_dim) { - net_dim_create_sample(priv->dim.event_ctr, priv->dim.packets, - priv->dim.bytes, &dim_sample); + dim_create_sample(priv->dim.event_ctr, priv->dim.packets, + priv->dim.bytes, &dim_sample); net_dim(&priv->dim.dim, dim_sample); } @@ -1099,13 +1099,13 @@ static void bcm_sysport_resume_from_wol(struct bcm_sysport_priv *priv) static void bcm_sysport_dim_work(struct work_struct *work) { - struct net_dim *dim = container_of(work, struct net_dim, work); + struct dim *dim = container_of(work, struct dim, work); struct bcm_sysport_net_dim *ndim = container_of(dim, struct bcm_sysport_net_dim, dim); struct bcm_sysport_priv *priv = container_of(ndim, struct bcm_sysport_priv, dim); - struct net_dim_cq_moder cur_profile = - net_dim_get_rx_moderation(dim->mode, dim->profile_ix); + struct dim_cq_moder cur_profile = net_dim_get_rx_moderation(dim->mode, + dim->profile_ix); bcm_sysport_set_rx_coalesce(priv, cur_profile.usec, cur_profile.pkts); dim->state = DIM_START_MEASURE; @@ -1469,7 +1469,7 @@ static void bcm_sysport_init_dim(struct bcm_sysport_priv *priv, static void bcm_sysport_init_rx_coalesce(struct bcm_sysport_priv *priv) { struct bcm_sysport_net_dim *dim = &priv->dim; - struct net_dim_cq_moder moder; + struct dim_cq_moder moder; u32 usecs, pkts; usecs = priv->rx_coalesce_usecs; diff --git a/drivers/net/ethernet/broadcom/bcmsysport.h b/drivers/net/ethernet/broadcom/bcmsysport.h index 0b192fe..a24fdc1 100644 --- a/drivers/net/ethernet/broadcom/bcmsysport.h +++ b/drivers/net/ethernet/broadcom/bcmsysport.h @@ -711,7 +711,7 @@ struct bcm_sysport_net_dim { u16 event_ctr; unsigned long packets; unsigned long bytes; - struct net_dim dim; + struct dim dim; }; /* Software view of the TX ring */ diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 1528620..9e62e3f 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -2125,12 +2125,12 @@ static int bnxt_poll(struct napi_struct *napi, int budget) } } if (bp->flags & BNXT_FLAG_DIM) { - struct net_dim_sample dim_sample; + struct dim_sample dim_sample; - net_dim_create_sample(cpr->event_ctr, - cpr->rx_packets, - cpr->rx_bytes, - &dim_sample); + dim_create_sample(cpr->event_ctr, + cpr->rx_packets, + cpr->rx_bytes, + &dim_sample); net_dim(&cpr->dim, dim_sample); } mmiowb(); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h index cf81ace..12a53b9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -809,7 +809,7 @@ struct bnxt_cp_ring_info { u64 rx_bytes; u64 event_ctr; - struct net_dim dim; + struct dim dim; union { struct tx_cmp *cp_desc_ring[MAX_CP_PAGES]; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_debugfs.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_debugfs.c index 94e208e..3d1d53f 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_debugfs.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_debugfs.c @@ -21,7 +21,7 @@ static ssize_t debugfs_dim_read(struct file *filep, char __user *buffer, size_t count, loff_t *ppos) { - struct net_dim *dim = filep->private_data; + struct dim *dim = filep->private_data; int len; char *buf; @@ -61,7 +61,7 @@ static ssize_t debugfs_dim_read(struct file *filep, .read = debugfs_dim_read, }; -static struct dentry *debugfs_dim_ring_init(struct net_dim *dim, int ring_idx, +static struct dentry *debugfs_dim_ring_init(struct dim *dim, int ring_idx, struct dentry *dd) { static char qname[16]; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c index 16a4588..11605f9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_dim.c @@ -13,15 +13,14 @@ void bnxt_dim_work(struct work_struct *work) { - struct net_dim *dim = container_of(work, struct net_dim, - work); + struct dim *dim = container_of(work, struct dim, work); struct bnxt_cp_ring_info *cpr = container_of(dim, struct bnxt_cp_ring_info, dim); struct bnxt_napi *bnapi = container_of(cpr, struct bnxt_napi, cp_ring); - struct net_dim_cq_moder cur_moder = + struct dim_cq_moder cur_moder = net_dim_get_rx_moderation(dim->mode, dim->profile_ix); cpr->rx_ring_coal.coal_ticks = cur_moder.usec; diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c index 8ab2d67..68d96e3 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c @@ -643,7 +643,7 @@ static void bcmgenet_set_rx_coalesce(struct bcmgenet_rx_ring *ring, static void bcmgenet_set_ring_rx_coalesce(struct bcmgenet_rx_ring *ring, struct ethtool_coalesce *ec) { - struct net_dim_cq_moder moder; + struct dim_cq_moder moder; u32 usecs, pkts; ring->rx_coalesce_usecs = ec->rx_coalesce_usecs; @@ -1898,7 +1898,7 @@ static int bcmgenet_rx_poll(struct napi_struct *napi, int budget) { struct bcmgenet_rx_ring *ring = container_of(napi, struct bcmgenet_rx_ring, napi); - struct net_dim_sample dim_sample; + struct dim_sample dim_sample; unsigned int work_done; work_done = bcmgenet_desc_rx(ring, budget); @@ -1909,8 +1909,8 @@ static int bcmgenet_rx_poll(struct napi_struct *napi, int budget) } if (ring->dim.use_dim) { - net_dim_create_sample(ring->dim.event_ctr, ring->dim.packets, - ring->dim.bytes, &dim_sample); + dim_create_sample(ring->dim.event_ctr, ring->dim.packets, + ring->dim.bytes, &dim_sample); net_dim(&ring->dim.dim, dim_sample); } @@ -1919,12 +1919,12 @@ static int bcmgenet_rx_poll(struct napi_struct *napi, int budget) static void bcmgenet_dim_work(struct work_struct *work) { - struct net_dim *dim = container_of(work, struct net_dim, work); + struct dim *dim = container_of(work, struct dim, work); struct bcmgenet_net_dim *ndim = container_of(dim, struct bcmgenet_net_dim, dim); struct bcmgenet_rx_ring *ring = container_of(ndim, struct bcmgenet_rx_ring, dim); - struct net_dim_cq_moder cur_profile = + struct dim_cq_moder cur_profile = net_dim_get_rx_moderation(dim->mode, dim->profile_ix); bcmgenet_set_rx_coalesce(ring, cur_profile.usec, cur_profile.pkts); @@ -2094,7 +2094,7 @@ static void bcmgenet_init_dim(struct bcmgenet_rx_ring *ring, static void bcmgenet_init_rx_coalesce(struct bcmgenet_rx_ring *ring) { struct bcmgenet_net_dim *dim = &ring->dim; - struct net_dim_cq_moder moder; + struct dim_cq_moder moder; u32 usecs, pkts; usecs = ring->rx_coalesce_usecs; diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.h b/drivers/net/ethernet/broadcom/genet/bcmgenet.h index 14b49612a..6e418d9 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet.h +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.h @@ -581,7 +581,7 @@ struct bcmgenet_net_dim { u16 event_ctr; unsigned long packets; unsigned long bytes; - struct net_dim dim; + struct dim dim; }; struct bcmgenet_rx_ring { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index 8ce9b51..e2ffb6c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -238,8 +238,8 @@ struct mlx5e_params { u16 num_channels; u8 num_tc; bool rx_cqe_compress_def; - struct net_dim_cq_moder rx_cq_moderation; - struct net_dim_cq_moder tx_cq_moderation; + struct dim_cq_moder rx_cq_moderation; + struct dim_cq_moder tx_cq_moderation; bool lro_en; u32 lro_wqe_sz; u8 tx_min_inline_mode; @@ -356,7 +356,7 @@ struct mlx5e_txqsq { /* dirtied @completion */ u16 cc; u32 dma_fifo_cc; - struct net_dim dim; /* Adaptive Moderation */ + struct dim dim; /* Adaptive Moderation */ /* dirtied @xmit */ u16 pc ____cacheline_aligned_in_smp; @@ -590,7 +590,7 @@ struct mlx5e_rq { int ix; unsigned int hw_mtu; - struct net_dim dim; /* Dynamic Interrupt Moderation */ + struct dim dim; /* Dynamic Interrupt Moderation */ /* XDP */ struct bpf_prog *xdp_prog; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_dim.c b/drivers/net/ethernet/mellanox/mlx5/core/en_dim.c index a80303a..ba3c1be 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_dim.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_dim.c @@ -34,7 +34,7 @@ #include "en.h" static void -mlx5e_complete_dim_work(struct net_dim *dim, struct net_dim_cq_moder moder, +mlx5e_complete_dim_work(struct dim *dim, struct dim_cq_moder moder, struct mlx5_core_dev *mdev, struct mlx5_core_cq *mcq) { mlx5_core_modify_cq_moderation(mdev, mcq, moder.usec, moder.pkts); @@ -43,9 +43,9 @@ void mlx5e_rx_dim_work(struct work_struct *work) { - struct net_dim *dim = container_of(work, struct net_dim, work); + struct dim *dim = container_of(work, struct dim, work); struct mlx5e_rq *rq = container_of(dim, struct mlx5e_rq, dim); - struct net_dim_cq_moder cur_moder = + struct dim_cq_moder cur_moder = net_dim_get_rx_moderation(dim->mode, dim->profile_ix); mlx5e_complete_dim_work(dim, cur_moder, rq->mdev, &rq->cq.mcq); @@ -53,9 +53,9 @@ void mlx5e_rx_dim_work(struct work_struct *work) void mlx5e_tx_dim_work(struct work_struct *work) { - struct net_dim *dim = container_of(work, struct net_dim, work); + struct dim *dim = container_of(work, struct dim, work); struct mlx5e_txqsq *sq = container_of(dim, struct mlx5e_txqsq, dim); - struct net_dim_cq_moder cur_moder = + struct dim_cq_moder cur_moder = net_dim_get_tx_moderation(dim->mode, dim->profile_ix); mlx5e_complete_dim_work(dim, cur_moder, sq->cq.mdev, &sq->cq.mcq); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c index 5efce4a..c346b59 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c @@ -466,7 +466,7 @@ static int mlx5e_set_channels(struct net_device *dev, int mlx5e_ethtool_get_coalesce(struct mlx5e_priv *priv, struct ethtool_coalesce *coal) { - struct net_dim_cq_moder *rx_moder, *tx_moder; + struct dim_cq_moder *rx_moder, *tx_moder; if (!MLX5_CAP_GEN(priv->mdev, cq_moderation)) return -EOPNOTSUPP; @@ -521,7 +521,7 @@ static int mlx5e_get_coalesce(struct net_device *netdev, int mlx5e_ethtool_set_coalesce(struct mlx5e_priv *priv, struct ethtool_coalesce *coal) { - struct net_dim_cq_moder *rx_moder, *tx_moder; + struct dim_cq_moder *rx_moder, *tx_moder; struct mlx5_core_dev *mdev = priv->mdev; struct mlx5e_channels new_channels = {}; int err = 0; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 589ecef..95ed0cb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -1666,7 +1666,7 @@ static void mlx5e_destroy_cq(struct mlx5e_cq *cq) } static int mlx5e_open_cq(struct mlx5e_channel *c, - struct net_dim_cq_moder moder, + struct dim_cq_moder moder, struct mlx5e_cq_param *param, struct mlx5e_cq *cq) { @@ -1871,7 +1871,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix, struct mlx5e_channel **cp) { int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix)); - struct net_dim_cq_moder icocq_moder = {0, 0}; + struct dim_cq_moder icocq_moder = {0, 0}; struct net_device *netdev = priv->netdev; struct mlx5e_channel *c; unsigned int irq; @@ -4456,9 +4456,9 @@ static bool slow_pci_heuristic(struct mlx5_core_dev *mdev) link_speed > MLX5E_SLOW_PCI_RATIO * pci_bw; } -static struct net_dim_cq_moder mlx5e_get_def_tx_moderation(u8 cq_period_mode) +static struct dim_cq_moder mlx5e_get_def_tx_moderation(u8 cq_period_mode) { - struct net_dim_cq_moder moder; + struct dim_cq_moder moder; moder.cq_period_mode = cq_period_mode; moder.pkts = MLX5E_PARAMS_DEFAULT_TX_CQ_MODERATION_PKTS; @@ -4469,9 +4469,9 @@ static struct net_dim_cq_moder mlx5e_get_def_tx_moderation(u8 cq_period_mode) return moder; } -static struct net_dim_cq_moder mlx5e_get_def_rx_moderation(u8 cq_period_mode) +static struct dim_cq_moder mlx5e_get_def_rx_moderation(u8 cq_period_mode) { - struct net_dim_cq_moder moder; + struct dim_cq_moder moder; moder.cq_period_mode = cq_period_mode; moder.pkts = MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_PKTS; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c index 6dd8202..4324747 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c @@ -48,24 +48,24 @@ static inline bool mlx5e_channel_no_affinity_change(struct mlx5e_channel *c) static void mlx5e_handle_tx_dim(struct mlx5e_txqsq *sq) { struct mlx5e_sq_stats *stats = sq->stats; - struct net_dim_sample dim_sample; + struct dim_sample dim_sample; if (unlikely(!test_bit(MLX5E_SQ_STATE_AM, &sq->state))) return; - net_dim_create_sample(sq->cq.event_ctr, stats->packets, stats->bytes, &dim_sample); + dim_create_sample(sq->cq.event_ctr, stats->packets, stats->bytes, &dim_sample); net_dim(&sq->dim, dim_sample); } static void mlx5e_handle_rx_dim(struct mlx5e_rq *rq) { struct mlx5e_rq_stats *stats = rq->stats; - struct net_dim_sample dim_sample; + struct dim_sample dim_sample; if (unlikely(!test_bit(MLX5E_RQ_STATE_AM, &rq->state))) return; - net_dim_create_sample(rq->cq.event_ctr, stats->packets, stats->bytes, &dim_sample); + dim_create_sample(rq->cq.event_ctr, stats->packets, stats->bytes, &dim_sample); net_dim(&rq->dim, dim_sample); } diff --git a/include/linux/dim.h b/include/linux/dim.h index 2413021..4fa55eb 100644 --- a/include/linux/dim.h +++ b/include/linux/dim.h @@ -41,13 +41,13 @@ (((100UL * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */ #define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1)) -struct net_dim_cq_moder { +struct dim_cq_moder { u16 usec; u16 pkts; u8 cq_period_mode; }; -struct net_dim_sample { +struct dim_sample { ktime_t time; u32 pkt_ctr; u32 byte_ctr; @@ -60,10 +60,10 @@ struct dim_stats { int epms; /* events per msec */ }; -struct net_dim { /* Dynamic Interrupt Moderation */ +struct dim { /* Dynamic Interrupt Moderation */ u8 state; struct dim_stats prev_stats; - struct net_dim_sample start_sample; + struct dim_sample start_sample; struct work_struct work; u8 profile_ix; u8 mode; @@ -104,7 +104,7 @@ enum { DIM_ON_EDGE, }; -static inline bool dim_on_top(struct net_dim *dim) +static inline bool dim_on_top(struct dim *dim) { switch (dim->tune_state) { case DIM_PARKING_ON_TOP: @@ -117,7 +117,7 @@ static inline bool dim_on_top(struct net_dim *dim) } } -static inline void dim_turn(struct net_dim *dim) +static inline void dim_turn(struct dim *dim) { switch (dim->tune_state) { case DIM_PARKING_ON_TOP: @@ -134,7 +134,7 @@ static inline void dim_turn(struct net_dim *dim) } } -static inline void dim_park_on_top(struct net_dim *dim) +static inline void dim_park_on_top(struct dim *dim) { dim->steps_right = 0; dim->steps_left = 0; @@ -142,17 +142,17 @@ static inline void dim_park_on_top(struct net_dim *dim) dim->tune_state = DIM_PARKING_ON_TOP; } -static inline void dim_park_tired(struct net_dim *dim) +static inline void dim_park_tired(struct dim *dim) { dim->steps_right = 0; dim->steps_left = 0; dim->tune_state = DIM_PARKING_TIRED; } -static inline void net_dim_create_sample(u16 event_ctr, - u64 packets, - u64 bytes, - struct net_dim_sample *s) +static inline void dim_create_sample(u16 event_ctr, + u64 packets, + u64 bytes, + struct dim_sample *s) { s->time = ktime_get(); s->pkt_ctr = packets; @@ -160,8 +160,8 @@ static inline void net_dim_create_sample(u16 event_ctr, s->event_ctr = event_ctr; } -static inline void dim_calc_stats(struct net_dim_sample *start, - struct net_dim_sample *end, +static inline void dim_calc_stats(struct dim_sample *start, + struct dim_sample *end, struct dim_stats *curr_stats) { /* u32 holds up to 71 minutes, should be enough */ diff --git a/include/linux/net_dim.h b/include/linux/net_dim.h index 93c4e21..1be7592 100644 --- a/include/linux/net_dim.h +++ b/include/linux/net_dim.h @@ -77,28 +77,28 @@ {64, 32} \ } -static const struct net_dim_cq_moder +static const struct dim_cq_moder rx_profile[DIM_CQ_PERIOD_NUM_MODES][NET_DIM_PARAMS_NUM_PROFILES] = { NET_DIM_RX_EQE_PROFILES, NET_DIM_RX_CQE_PROFILES, }; -static const struct net_dim_cq_moder +static const struct dim_cq_moder tx_profile[DIM_CQ_PERIOD_NUM_MODES][NET_DIM_PARAMS_NUM_PROFILES] = { NET_DIM_TX_EQE_PROFILES, NET_DIM_TX_CQE_PROFILES, }; -static inline struct net_dim_cq_moder +static inline struct dim_cq_moder net_dim_get_rx_moderation(u8 cq_period_mode, int ix) { - struct net_dim_cq_moder cq_moder = rx_profile[cq_period_mode][ix]; + struct dim_cq_moder cq_moder = rx_profile[cq_period_mode][ix]; cq_moder.cq_period_mode = cq_period_mode; return cq_moder; } -static inline struct net_dim_cq_moder +static inline struct dim_cq_moder net_dim_get_def_rx_moderation(u8 cq_period_mode) { u8 profile_ix = cq_period_mode == DIM_CQ_PERIOD_MODE_START_FROM_CQE ? @@ -107,16 +107,16 @@ return net_dim_get_rx_moderation(cq_period_mode, profile_ix); } -static inline struct net_dim_cq_moder +static inline struct dim_cq_moder net_dim_get_tx_moderation(u8 cq_period_mode, int ix) { - struct net_dim_cq_moder cq_moder = tx_profile[cq_period_mode][ix]; + struct dim_cq_moder cq_moder = tx_profile[cq_period_mode][ix]; cq_moder.cq_period_mode = cq_period_mode; return cq_moder; } -static inline struct net_dim_cq_moder +static inline struct dim_cq_moder net_dim_get_def_tx_moderation(u8 cq_period_mode) { u8 profile_ix = cq_period_mode == DIM_CQ_PERIOD_MODE_START_FROM_CQE ? @@ -125,7 +125,7 @@ return net_dim_get_tx_moderation(cq_period_mode, profile_ix); } -static inline int net_dim_step(struct net_dim *dim) +static inline int net_dim_step(struct dim *dim) { if (dim->tired == (NET_DIM_PARAMS_NUM_PROFILES * 2)) return DIM_TOO_TIRED; @@ -152,7 +152,7 @@ static inline int net_dim_step(struct net_dim *dim) return DIM_STEPPED; } -static inline void net_dim_exit_parking(struct net_dim *dim) +static inline void net_dim_exit_parking(struct dim *dim) { dim->tune_state = dim->profile_ix ? DIM_GOING_LEFT : DIM_GOING_RIGHT; @@ -189,7 +189,7 @@ static inline int net_dim_stats_compare(struct dim_stats *curr, } static inline bool net_dim_decision(struct dim_stats *curr_stats, - struct net_dim *dim) + struct dim *dim) { int prev_state = dim->tune_state; int prev_ix = dim->profile_ix; @@ -239,8 +239,8 @@ static inline bool net_dim_decision(struct dim_stats *curr_stats, return dim->profile_ix != prev_ix; } -static inline void net_dim(struct net_dim *dim, - struct net_dim_sample end_sample) +static inline void net_dim(struct dim *dim, + struct dim_sample end_sample) { struct dim_stats curr_stats; u16 nevents; @@ -260,8 +260,8 @@ static inline void net_dim(struct net_dim *dim, } /* fall through */ case DIM_START_MEASURE: - net_dim_create_sample(end_sample.event_ctr, end_sample.pkt_ctr, - end_sample.byte_ctr, &dim->start_sample); + dim_create_sample(end_sample.event_ctr, end_sample.pkt_ctr, + end_sample.byte_ctr, &dim->start_sample); dim->state = DIM_MEASURE_IN_PROGRESS; break; case DIM_APPLY_NEW_PROFILE: From patchwork Wed May 1 14:44:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925177 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D21E11398 for ; Wed, 1 May 2019 14:45:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C127F28B5D for ; Wed, 1 May 2019 14:45:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B540028B35; Wed, 1 May 2019 14:45:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A769528B5A for ; Wed, 1 May 2019 14:45:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727077AbfEAOpR (ORCPT ); Wed, 1 May 2019 10:45:17 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40119 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727066AbfEAOpB (ORCPT ); Wed, 1 May 2019 10:45:01 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:53 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41EirWW019849; Wed, 1 May 2019 17:44:53 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41EiqB4036050; Wed, 1 May 2019 17:44:52 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41EiqYY036049; Wed, 1 May 2019 17:44:52 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 6/9] linux/dim: Move implementation to .c files Date: Wed, 1 May 2019 17:44:36 +0300 Message-Id: <1556721879-35987-7-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Moved all logic from dim.h and net_dim.h to dim.c and net_dim.c. Signed-off-by: Tal Gilboa --- MAINTAINERS | 2 + include/linux/dim.h | 86 +++------------------- include/linux/net_dim.h | 180 ++------------------------------------------- lib/Kconfig | 7 ++ lib/Makefile | 1 + lib/dim/Makefile | 9 +++ lib/dim/dim.c | 83 +++++++++++++++++++++ lib/dim/net_dim.c | 191 ++++++++++++++++++++++++++++++++++++++++++++++++ 8 files changed, 310 insertions(+), 249 deletions(-) create mode 100644 lib/dim/Makefile create mode 100644 lib/dim/dim.c create mode 100644 lib/dim/net_dim.c diff --git a/MAINTAINERS b/MAINTAINERS index da2cd72..c405029 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5472,6 +5472,8 @@ DYNAMIC INTERRUPT MODERATION M: Tal Gilboa S: Maintained F: include/linux/net_dim.h +F: include/linux/dim.h +F: lib/dim/ DZ DECSTATION DZ11 SERIAL DRIVER M: "Maciej W. Rozycki" diff --git a/include/linux/dim.h b/include/linux/dim.h index 4fa55eb..ffaea87 100644 --- a/include/linux/dim.h +++ b/include/linux/dim.h @@ -104,79 +104,17 @@ enum { DIM_ON_EDGE, }; -static inline bool dim_on_top(struct dim *dim) -{ - switch (dim->tune_state) { - case DIM_PARKING_ON_TOP: - case DIM_PARKING_TIRED: - return true; - case DIM_GOING_RIGHT: - return (dim->steps_left > 1) && (dim->steps_right == 1); - default: /* DIM_GOING_LEFT */ - return (dim->steps_right > 1) && (dim->steps_left == 1); - } -} - -static inline void dim_turn(struct dim *dim) -{ - switch (dim->tune_state) { - case DIM_PARKING_ON_TOP: - case DIM_PARKING_TIRED: - break; - case DIM_GOING_RIGHT: - dim->tune_state = DIM_GOING_LEFT; - dim->steps_left = 0; - break; - case DIM_GOING_LEFT: - dim->tune_state = DIM_GOING_RIGHT; - dim->steps_right = 0; - break; - } -} - -static inline void dim_park_on_top(struct dim *dim) -{ - dim->steps_right = 0; - dim->steps_left = 0; - dim->tired = 0; - dim->tune_state = DIM_PARKING_ON_TOP; -} - -static inline void dim_park_tired(struct dim *dim) -{ - dim->steps_right = 0; - dim->steps_left = 0; - dim->tune_state = DIM_PARKING_TIRED; -} - -static inline void dim_create_sample(u16 event_ctr, - u64 packets, - u64 bytes, - struct dim_sample *s) -{ - s->time = ktime_get(); - s->pkt_ctr = packets; - s->byte_ctr = bytes; - s->event_ctr = event_ctr; -} - -static inline void dim_calc_stats(struct dim_sample *start, - struct dim_sample *end, - struct dim_stats *curr_stats) -{ - /* u32 holds up to 71 minutes, should be enough */ - u32 delta_us = ktime_us_delta(end->time, start->time); - u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr); - u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr, - start->byte_ctr); - - if (!delta_us) - return; - - curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us); - curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us); - curr_stats->epms = DIV_ROUND_UP(DIM_NEVENTS * USEC_PER_MSEC, - delta_us); -} +bool dim_on_top(struct dim *dim); + +void dim_turn(struct dim *dim); + +void dim_park_on_top(struct dim *dim); + +void dim_park_tired(struct dim *dim); + +void dim_create_sample(u16 event_ctr, u64 packets, u64 bytes, struct dim_sample *s); + +void dim_calc_stats(struct dim_sample *start, struct dim_sample *end, + struct dim_stats *curr_stats); #endif /* DIM_H */ diff --git a/include/linux/net_dim.h b/include/linux/net_dim.h index 1be7592..debf548 100644 --- a/include/linux/net_dim.h +++ b/include/linux/net_dim.h @@ -89,184 +89,14 @@ NET_DIM_TX_CQE_PROFILES, }; -static inline struct dim_cq_moder -net_dim_get_rx_moderation(u8 cq_period_mode, int ix) -{ - struct dim_cq_moder cq_moder = rx_profile[cq_period_mode][ix]; +struct dim_cq_moder net_dim_get_rx_moderation(u8 cq_period_mode, int ix); - cq_moder.cq_period_mode = cq_period_mode; - return cq_moder; -} - -static inline struct dim_cq_moder -net_dim_get_def_rx_moderation(u8 cq_period_mode) -{ - u8 profile_ix = cq_period_mode == DIM_CQ_PERIOD_MODE_START_FROM_CQE ? - NET_DIM_DEF_PROFILE_CQE : NET_DIM_DEF_PROFILE_EQE; - - return net_dim_get_rx_moderation(cq_period_mode, profile_ix); -} - -static inline struct dim_cq_moder -net_dim_get_tx_moderation(u8 cq_period_mode, int ix) -{ - struct dim_cq_moder cq_moder = tx_profile[cq_period_mode][ix]; - - cq_moder.cq_period_mode = cq_period_mode; - return cq_moder; -} - -static inline struct dim_cq_moder -net_dim_get_def_tx_moderation(u8 cq_period_mode) -{ - u8 profile_ix = cq_period_mode == DIM_CQ_PERIOD_MODE_START_FROM_CQE ? - NET_DIM_DEF_PROFILE_CQE : NET_DIM_DEF_PROFILE_EQE; - - return net_dim_get_tx_moderation(cq_period_mode, profile_ix); -} - -static inline int net_dim_step(struct dim *dim) -{ - if (dim->tired == (NET_DIM_PARAMS_NUM_PROFILES * 2)) - return DIM_TOO_TIRED; - - switch (dim->tune_state) { - case DIM_PARKING_ON_TOP: - case DIM_PARKING_TIRED: - break; - case DIM_GOING_RIGHT: - if (dim->profile_ix == (NET_DIM_PARAMS_NUM_PROFILES - 1)) - return DIM_ON_EDGE; - dim->profile_ix++; - dim->steps_right++; - break; - case DIM_GOING_LEFT: - if (dim->profile_ix == 0) - return DIM_ON_EDGE; - dim->profile_ix--; - dim->steps_left++; - break; - } - - dim->tired++; - return DIM_STEPPED; -} - -static inline void net_dim_exit_parking(struct dim *dim) -{ - dim->tune_state = dim->profile_ix ? DIM_GOING_LEFT : - DIM_GOING_RIGHT; - net_dim_step(dim); -} - -static inline int net_dim_stats_compare(struct dim_stats *curr, - struct dim_stats *prev) -{ - if (!prev->bpms) - return curr->bpms ? DIM_STATS_BETTER : - DIM_STATS_SAME; +struct dim_cq_moder net_dim_get_def_rx_moderation(u8 cq_period_mode); - if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms)) - return (curr->bpms > prev->bpms) ? DIM_STATS_BETTER : - DIM_STATS_WORSE; +struct dim_cq_moder net_dim_get_tx_moderation(u8 cq_period_mode, int ix); - if (!prev->ppms) - return curr->ppms ? DIM_STATS_BETTER : - DIM_STATS_SAME; +struct dim_cq_moder net_dim_get_def_tx_moderation(u8 cq_period_mode); - if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms)) - return (curr->ppms > prev->ppms) ? DIM_STATS_BETTER : - DIM_STATS_WORSE; - - if (!prev->epms) - return DIM_STATS_SAME; - - if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms)) - return (curr->epms < prev->epms) ? DIM_STATS_BETTER : - DIM_STATS_WORSE; - - return DIM_STATS_SAME; -} - -static inline bool net_dim_decision(struct dim_stats *curr_stats, - struct dim *dim) -{ - int prev_state = dim->tune_state; - int prev_ix = dim->profile_ix; - int stats_res; - int step_res; - - switch (dim->tune_state) { - case DIM_PARKING_ON_TOP: - stats_res = net_dim_stats_compare(curr_stats, &dim->prev_stats); - if (stats_res != DIM_STATS_SAME) - net_dim_exit_parking(dim); - break; - - case DIM_PARKING_TIRED: - dim->tired--; - if (!dim->tired) - net_dim_exit_parking(dim); - break; - - case DIM_GOING_RIGHT: - case DIM_GOING_LEFT: - stats_res = net_dim_stats_compare(curr_stats, &dim->prev_stats); - if (stats_res != DIM_STATS_BETTER) - dim_turn(dim); - - if (dim_on_top(dim)) { - dim_park_on_top(dim); - break; - } - - step_res = net_dim_step(dim); - switch (step_res) { - case DIM_ON_EDGE: - dim_park_on_top(dim); - break; - case DIM_TOO_TIRED: - dim_park_tired(dim); - break; - } - - break; - } - - if (prev_state != DIM_PARKING_ON_TOP || dim->tune_state != DIM_PARKING_ON_TOP) - dim->prev_stats = *curr_stats; - - return dim->profile_ix != prev_ix; -} - -static inline void net_dim(struct dim *dim, - struct dim_sample end_sample) -{ - struct dim_stats curr_stats; - u16 nevents; - - switch (dim->state) { - case DIM_MEASURE_IN_PROGRESS: - nevents = BIT_GAP(BITS_PER_TYPE(u16), - end_sample.event_ctr, - dim->start_sample.event_ctr); - if (nevents < DIM_NEVENTS) - break; - dim_calc_stats(&dim->start_sample, &end_sample, &curr_stats); - if (net_dim_decision(&curr_stats, dim)) { - dim->state = DIM_APPLY_NEW_PROFILE; - schedule_work(&dim->work); - break; - } - /* fall through */ - case DIM_START_MEASURE: - dim_create_sample(end_sample.event_ctr, end_sample.pkt_ctr, - end_sample.byte_ctr, &dim->start_sample); - dim->state = DIM_MEASURE_IN_PROGRESS; - break; - case DIM_APPLY_NEW_PROFILE: - break; - } -} +void net_dim(struct dim *dim, struct dim_sample end_sample); #endif /* NET_DIM_H */ diff --git a/lib/Kconfig b/lib/Kconfig index a9e5653..153465c 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -551,6 +551,13 @@ config SIGNATURE Digital signature verification. Currently only RSA is supported. Implementation is done using GnuPG MPI library +config DIMLIB + bool "DIM library" + default y + help + Dynamic Interrupt Moderation library. + It is used to implement DIM logic. + # # libfdt files, only selected if needed. # diff --git a/lib/Makefile b/lib/Makefile index 3b08673..179c081 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -190,6 +190,7 @@ obj-$(CONFIG_GLOB) += glob.o obj-$(CONFIG_GLOB_SELFTEST) += globtest.o obj-$(CONFIG_MPILIB) += mpi/ +obj-$(CONFIG_DIMLIB) += dim/ obj-$(CONFIG_SIGNATURE) += digsig.o lib-$(CONFIG_CLZ_TAB) += clz_tab.o diff --git a/lib/dim/Makefile b/lib/dim/Makefile new file mode 100644 index 0000000..160afe2 --- /dev/null +++ b/lib/dim/Makefile @@ -0,0 +1,9 @@ +# +# DIM Dynamic Interrupt Moderation library +# + +obj-$(CONFIG_DIMLIB) = net_dim.o + +net_dim-y = \ + dim.o \ + net_dim.o diff --git a/lib/dim/dim.c b/lib/dim/dim.c new file mode 100644 index 0000000..93e1ddd --- /dev/null +++ b/lib/dim/dim.c @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* + * Copyright (c) 2018, Mellanox Technologies inc. All rights reserved. + */ + +#include + +bool dim_on_top(struct dim *dim) +{ + switch (dim->tune_state) { + case DIM_PARKING_ON_TOP: + case DIM_PARKING_TIRED: + return true; + case DIM_GOING_RIGHT: + return (dim->steps_left > 1) && (dim->steps_right == 1); + default: /* DIM_GOING_LEFT */ + return (dim->steps_right > 1) && (dim->steps_left == 1); + } +} +EXPORT_SYMBOL(dim_on_top); + +void dim_turn(struct dim *dim) +{ + switch (dim->tune_state) { + case DIM_PARKING_ON_TOP: + case DIM_PARKING_TIRED: + break; + case DIM_GOING_RIGHT: + dim->tune_state = DIM_GOING_LEFT; + dim->steps_left = 0; + break; + case DIM_GOING_LEFT: + dim->tune_state = DIM_GOING_RIGHT; + dim->steps_right = 0; + break; + } +} +EXPORT_SYMBOL(dim_turn); + +void dim_park_on_top(struct dim *dim) +{ + dim->steps_right = 0; + dim->steps_left = 0; + dim->tired = 0; + dim->tune_state = DIM_PARKING_ON_TOP; +} +EXPORT_SYMBOL(dim_park_on_top); + +void dim_park_tired(struct dim *dim) +{ + dim->steps_right = 0; + dim->steps_left = 0; + dim->tune_state = DIM_PARKING_TIRED; +} +EXPORT_SYMBOL(dim_park_tired); + +void dim_create_sample(u16 event_ctr, u64 packets, u64 bytes, struct dim_sample *s) +{ + s->time = ktime_get(); + s->pkt_ctr = packets; + s->byte_ctr = bytes; + s->event_ctr = event_ctr; +} +EXPORT_SYMBOL(dim_create_sample); + +void dim_calc_stats(struct dim_sample *start, struct dim_sample *end, + struct dim_stats *curr_stats) +{ + /* u32 holds up to 71 minutes, should be enough */ + u32 delta_us = ktime_us_delta(end->time, start->time); + u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr); + u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr, + start->byte_ctr); + + if (!delta_us) + return; + + curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us); + curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us); + curr_stats->epms = DIV_ROUND_UP(DIM_NEVENTS * USEC_PER_MSEC, + delta_us); +} +EXPORT_SYMBOL(dim_calc_stats); diff --git a/lib/dim/net_dim.c b/lib/dim/net_dim.c new file mode 100644 index 0000000..05d259e --- /dev/null +++ b/lib/dim/net_dim.c @@ -0,0 +1,191 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* + * Copyright (c) 2018, Mellanox Technologies inc. All rights reserved. + */ + +#include +#include + +struct dim_cq_moder +net_dim_get_rx_moderation(u8 cq_period_mode, int ix) +{ + struct dim_cq_moder cq_moder = rx_profile[cq_period_mode][ix]; + + cq_moder.cq_period_mode = cq_period_mode; + return cq_moder; +} +EXPORT_SYMBOL(net_dim_get_rx_moderation); + +struct dim_cq_moder +net_dim_get_def_rx_moderation(u8 cq_period_mode) +{ + u8 profile_ix = cq_period_mode == DIM_CQ_PERIOD_MODE_START_FROM_CQE ? + NET_DIM_DEF_PROFILE_CQE : NET_DIM_DEF_PROFILE_EQE; + + return net_dim_get_rx_moderation(cq_period_mode, profile_ix); +} +EXPORT_SYMBOL(net_dim_get_def_rx_moderation); + +struct dim_cq_moder +net_dim_get_tx_moderation(u8 cq_period_mode, int ix) +{ + struct dim_cq_moder cq_moder = tx_profile[cq_period_mode][ix]; + + cq_moder.cq_period_mode = cq_period_mode; + return cq_moder; +} +EXPORT_SYMBOL(net_dim_get_tx_moderation); + +struct dim_cq_moder +net_dim_get_def_tx_moderation(u8 cq_period_mode) +{ + u8 profile_ix = cq_period_mode == DIM_CQ_PERIOD_MODE_START_FROM_CQE ? + NET_DIM_DEF_PROFILE_CQE : NET_DIM_DEF_PROFILE_EQE; + + return net_dim_get_tx_moderation(cq_period_mode, profile_ix); +} +EXPORT_SYMBOL(net_dim_get_def_tx_moderation); + +static inline int net_dim_step(struct dim *dim) +{ + if (dim->tired == (NET_DIM_PARAMS_NUM_PROFILES * 2)) + return DIM_TOO_TIRED; + + switch (dim->tune_state) { + case DIM_PARKING_ON_TOP: + case DIM_PARKING_TIRED: + break; + case DIM_GOING_RIGHT: + if (dim->profile_ix == (NET_DIM_PARAMS_NUM_PROFILES - 1)) + return DIM_ON_EDGE; + dim->profile_ix++; + dim->steps_right++; + break; + case DIM_GOING_LEFT: + if (dim->profile_ix == 0) + return DIM_ON_EDGE; + dim->profile_ix--; + dim->steps_left++; + break; + } + + dim->tired++; + return DIM_STEPPED; +} + +static inline void net_dim_exit_parking(struct dim *dim) +{ + dim->tune_state = dim->profile_ix ? DIM_GOING_LEFT : + DIM_GOING_RIGHT; + net_dim_step(dim); +} + +static inline int net_dim_stats_compare(struct dim_stats *curr, + struct dim_stats *prev) +{ + if (!prev->bpms) + return curr->bpms ? DIM_STATS_BETTER : + DIM_STATS_SAME; + + if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms)) + return (curr->bpms > prev->bpms) ? DIM_STATS_BETTER : + DIM_STATS_WORSE; + + if (!prev->ppms) + return curr->ppms ? DIM_STATS_BETTER : + DIM_STATS_SAME; + + if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms)) + return (curr->ppms > prev->ppms) ? DIM_STATS_BETTER : + DIM_STATS_WORSE; + + if (!prev->epms) + return DIM_STATS_SAME; + + if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms)) + return (curr->epms < prev->epms) ? DIM_STATS_BETTER : + DIM_STATS_WORSE; + + return DIM_STATS_SAME; +} + +static inline bool net_dim_decision(struct dim_stats *curr_stats, + struct dim *dim) +{ + int prev_state = dim->tune_state; + int prev_ix = dim->profile_ix; + int stats_res; + int step_res; + + switch (dim->tune_state) { + case DIM_PARKING_ON_TOP: + stats_res = net_dim_stats_compare(curr_stats, &dim->prev_stats); + if (stats_res != DIM_STATS_SAME) + net_dim_exit_parking(dim); + break; + + case DIM_PARKING_TIRED: + dim->tired--; + if (!dim->tired) + net_dim_exit_parking(dim); + break; + + case DIM_GOING_RIGHT: + case DIM_GOING_LEFT: + stats_res = net_dim_stats_compare(curr_stats, &dim->prev_stats); + if (stats_res != DIM_STATS_BETTER) + dim_turn(dim); + + if (dim_on_top(dim)) { + dim_park_on_top(dim); + break; + } + + step_res = net_dim_step(dim); + switch (step_res) { + case DIM_ON_EDGE: + dim_park_on_top(dim); + break; + case DIM_TOO_TIRED: + dim_park_tired(dim); + break; + } + + break; + } + + if (prev_state != DIM_PARKING_ON_TOP || dim->tune_state != DIM_PARKING_ON_TOP) + dim->prev_stats = *curr_stats; + + return dim->profile_ix != prev_ix; +} + +void net_dim(struct dim *dim, struct dim_sample end_sample) +{ + struct dim_stats curr_stats; + u16 nevents; + + switch (dim->state) { + case DIM_MEASURE_IN_PROGRESS: + nevents = BIT_GAP(BITS_PER_TYPE(u16), + end_sample.event_ctr, + dim->start_sample.event_ctr); + if (nevents < DIM_NEVENTS) + break; + dim_calc_stats(&dim->start_sample, &end_sample, &curr_stats); + if (net_dim_decision(&curr_stats, dim)) { + dim->state = DIM_APPLY_NEW_PROFILE; + schedule_work(&dim->work); + break; + } + /* fall through */ + case DIM_START_MEASURE: + dim_create_sample(end_sample.event_ctr, end_sample.pkt_ctr, + end_sample.byte_ctr, &dim->start_sample); + dim->state = DIM_MEASURE_IN_PROGRESS; + break; + case DIM_APPLY_NEW_PROFILE: + break; + } +} +EXPORT_SYMBOL(net_dim); From patchwork Wed May 1 14:44:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925179 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E4131398 for ; Wed, 1 May 2019 14:45:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F81A28B85 for ; Wed, 1 May 2019 14:45:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1DB9E28D53; Wed, 1 May 2019 14:45:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98CC028B85 for ; Wed, 1 May 2019 14:45:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727073AbfEAOpA (ORCPT ); Wed, 1 May 2019 10:45:00 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40131 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727069AbfEAOo7 (ORCPT ); Wed, 1 May 2019 10:44:59 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:53 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41Eirqf019853; Wed, 1 May 2019 17:44:53 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41Eirwt036052; Wed, 1 May 2019 17:44:53 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41Eir40036051; Wed, 1 May 2019 17:44:53 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 7/9] linux/dim: Add completions count to dim_sample Date: Wed, 1 May 2019 17:44:37 +0300 Message-Id: <1556721879-35987-8-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Yamin Friedman Added a measurement of completions per/msec to allow for completion based dim algorithms. Signed-off-by: Yamin Friedman Reviewed-by: Max Gurtovoy Signed-off-by: Tal Gilboa --- drivers/net/ethernet/broadcom/bcmsysport.c | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 1 + drivers/net/ethernet/broadcom/genet/bcmgenet.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 6 ++++-- include/linux/dim.h | 8 +++++++- lib/dim/dim.c | 13 ++++++++++++- lib/dim/net_dim.c | 2 +- 7 files changed, 27 insertions(+), 7 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c index b2c5b1a..fc21432 100644 --- a/drivers/net/ethernet/broadcom/bcmsysport.c +++ b/drivers/net/ethernet/broadcom/bcmsysport.c @@ -1029,7 +1029,7 @@ static int bcm_sysport_poll(struct napi_struct *napi, int budget) if (priv->dim.use_dim) { dim_create_sample(priv->dim.event_ctr, priv->dim.packets, - priv->dim.bytes, &dim_sample); + priv->dim.bytes, 0, &dim_sample); net_dim(&priv->dim.dim, dim_sample); } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 9e62e3f..91607ef 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -2130,6 +2130,7 @@ static int bnxt_poll(struct napi_struct *napi, int budget) dim_create_sample(cpr->event_ctr, cpr->rx_packets, cpr->rx_bytes, + 0, &dim_sample); net_dim(&cpr->dim, dim_sample); } diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c index 68d96e3..aca82ef 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c @@ -1910,7 +1910,7 @@ static int bcmgenet_rx_poll(struct napi_struct *napi, int budget) if (ring->dim.use_dim) { dim_create_sample(ring->dim.event_ctr, ring->dim.packets, - ring->dim.bytes, &dim_sample); + ring->dim.bytes, 0, &dim_sample); net_dim(&ring->dim.dim, dim_sample); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c index 4324747..ab68d7d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c @@ -53,7 +53,8 @@ static void mlx5e_handle_tx_dim(struct mlx5e_txqsq *sq) if (unlikely(!test_bit(MLX5E_SQ_STATE_AM, &sq->state))) return; - dim_create_sample(sq->cq.event_ctr, stats->packets, stats->bytes, &dim_sample); + dim_create_sample(sq->cq.event_ctr, stats->packets, stats->bytes, 0, + &dim_sample); net_dim(&sq->dim, dim_sample); } @@ -65,7 +66,8 @@ static void mlx5e_handle_rx_dim(struct mlx5e_rq *rq) if (unlikely(!test_bit(MLX5E_RQ_STATE_AM, &rq->state))) return; - dim_create_sample(rq->cq.event_ctr, stats->packets, stats->bytes, &dim_sample); + dim_create_sample(rq->cq.event_ctr, stats->packets, stats->bytes, 0, + &dim_sample); net_dim(&rq->dim, dim_sample); } diff --git a/include/linux/dim.h b/include/linux/dim.h index ffaea87..2bc4c40 100644 --- a/include/linux/dim.h +++ b/include/linux/dim.h @@ -44,6 +44,7 @@ struct dim_cq_moder { u16 usec; u16 pkts; + u16 comps; u8 cq_period_mode; }; @@ -52,18 +53,22 @@ struct dim_sample { u32 pkt_ctr; u32 byte_ctr; u16 event_ctr; + u32 comp_ctr; }; struct dim_stats { int ppms; /* packets per msec */ int bpms; /* bytes per msec */ int epms; /* events per msec */ + int cpms; /* completions per msec */ + int cpe_ratio; /* ratio of completions to events */ }; struct dim { /* Dynamic Interrupt Moderation */ u8 state; struct dim_stats prev_stats; struct dim_sample start_sample; + struct dim_sample measuring_sample; struct work_struct work; u8 profile_ix; u8 mode; @@ -112,7 +117,8 @@ enum { void dim_park_tired(struct dim *dim); -void dim_create_sample(u16 event_ctr, u64 packets, u64 bytes, struct dim_sample *s); +void dim_create_sample(u16 event_ctr, u64 packets, u64 bytes, u64 comps, + struct dim_sample *s); void dim_calc_stats(struct dim_sample *start, struct dim_sample *end, struct dim_stats *curr_stats); diff --git a/lib/dim/dim.c b/lib/dim/dim.c index 93e1ddd..4f25c0f 100644 --- a/lib/dim/dim.c +++ b/lib/dim/dim.c @@ -54,12 +54,14 @@ void dim_park_tired(struct dim *dim) } EXPORT_SYMBOL(dim_park_tired); -void dim_create_sample(u16 event_ctr, u64 packets, u64 bytes, struct dim_sample *s) +void dim_create_sample(u16 event_ctr, u64 packets, u64 bytes, u64 comps, + struct dim_sample *s) { s->time = ktime_get(); s->pkt_ctr = packets; s->byte_ctr = bytes; s->event_ctr = event_ctr; + s->comp_ctr = comps; } EXPORT_SYMBOL(dim_create_sample); @@ -71,6 +73,8 @@ void dim_calc_stats(struct dim_sample *start, struct dim_sample *end, u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr); u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr, start->byte_ctr); + u32 ncomps = BIT_GAP(BITS_PER_TYPE(u32), end->comp_ctr, + start->comp_ctr); if (!delta_us) return; @@ -79,5 +83,12 @@ void dim_calc_stats(struct dim_sample *start, struct dim_sample *end, curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us); curr_stats->epms = DIV_ROUND_UP(DIM_NEVENTS * USEC_PER_MSEC, delta_us); + curr_stats->cpms = DIV_ROUND_UP(ncomps * USEC_PER_MSEC, delta_us); + if (curr_stats->epms != 0) + curr_stats->cpe_ratio = + (curr_stats->cpms * 100) / curr_stats->epms; + else + curr_stats->cpe_ratio = 0; + } EXPORT_SYMBOL(dim_calc_stats); diff --git a/lib/dim/net_dim.c b/lib/dim/net_dim.c index 05d259e..67a8f82 100644 --- a/lib/dim/net_dim.c +++ b/lib/dim/net_dim.c @@ -181,7 +181,7 @@ void net_dim(struct dim *dim, struct dim_sample end_sample) /* fall through */ case DIM_START_MEASURE: dim_create_sample(end_sample.event_ctr, end_sample.pkt_ctr, - end_sample.byte_ctr, &dim->start_sample); + end_sample.byte_ctr, 0, &dim->start_sample); dim->state = DIM_MEASURE_IN_PROGRESS; break; case DIM_APPLY_NEW_PROFILE: From patchwork Wed May 1 14:44:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925171 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3E6321398 for ; Wed, 1 May 2019 14:45:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2B2FA28B76 for ; Wed, 1 May 2019 14:45:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2991528BB7; Wed, 1 May 2019 14:45:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A46928B76 for ; Wed, 1 May 2019 14:45:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727085AbfEAOpC (ORCPT ); Wed, 1 May 2019 10:45:02 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40147 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727064AbfEAOpB (ORCPT ); Wed, 1 May 2019 10:45:01 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:53 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41Eirhk019856; Wed, 1 May 2019 17:44:53 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41EiriV036054; Wed, 1 May 2019 17:44:53 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41EirAt036053; Wed, 1 May 2019 17:44:53 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 8/9] linux/dim: Implement rdma_dim Date: Wed, 1 May 2019 17:44:38 +0300 Message-Id: <1556721879-35987-9-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Yamin Friedman rdma_dim implements a different algorithm than net_dim and is based on completions which is how we can implement interrupt moderation in RDMA. The algorithm optimizes for number of completions and ratio between completions and events. It also has a feature for fast reduction of moderation level when the traffic changes in such a way as to no longer require high moderation in order to avoid long latencies. rdma_dim.h will be called from the ib_core module. Signed-off-by: Yamin Friedman Reviewed-by: Max Gurtovoy Signed-off-by: Tal Gilboa --- MAINTAINERS | 1 + include/linux/rdma_dim.h | 57 +++++++++++++++++ lib/dim/Makefile | 7 ++- lib/dim/rdma_dim.c | 161 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 225 insertions(+), 1 deletion(-) create mode 100644 include/linux/rdma_dim.h create mode 100644 lib/dim/rdma_dim.c diff --git a/MAINTAINERS b/MAINTAINERS index c405029..c7ebd38 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5472,6 +5472,7 @@ DYNAMIC INTERRUPT MODERATION M: Tal Gilboa S: Maintained F: include/linux/net_dim.h +F: include/linux/blk_dim.h F: include/linux/dim.h F: lib/dim/ diff --git a/include/linux/rdma_dim.h b/include/linux/rdma_dim.h new file mode 100644 index 0000000..c0bed9c --- /dev/null +++ b/include/linux/rdma_dim.h @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) */ +/* Copyright (c) 2018-2019 Mellanox Technologies. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef RDMA_DIM_H +#define RDMA_DIM_H + +#include +#include + +#define RDMA_DIM_PARAMS_NUM_PROFILES 9 +#define RDMA_DIM_START_PROFILE 0 + +static const struct dim_cq_moder +rdma_dim_prof[RDMA_DIM_PARAMS_NUM_PROFILES] = { + {1, 0, 1, 0}, + {1, 0, 4, 0}, + {2, 0, 4, 0}, + {2, 0, 8, 0}, + {4, 0, 8, 0}, + {16, 0, 8, 0}, + {16, 0, 16, 0}, + {32, 0, 16, 0}, + {32, 0, 32, 0}, +}; + +void rdma_dim(struct dim *dim, struct dim_sample *end_sample); + +#endif /* RDMA_DIM_H */ diff --git a/lib/dim/Makefile b/lib/dim/Makefile index 160afe2..73ddd0c 100644 --- a/lib/dim/Makefile +++ b/lib/dim/Makefile @@ -2,8 +2,13 @@ # DIM Dynamic Interrupt Moderation library # -obj-$(CONFIG_DIMLIB) = net_dim.o +obj-$(CONFIG_DIMLIB) += net_dim.o +obj-$(CONFIG_DIMLIB) += rdma_dim.o net_dim-y = \ dim.o \ net_dim.o + +rdma_dim-y = \ + dim.o \ + rdma_dim.o diff --git a/lib/dim/rdma_dim.c b/lib/dim/rdma_dim.c new file mode 100644 index 0000000..c126209 --- /dev/null +++ b/lib/dim/rdma_dim.c @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* + * Copyright (c) 2019, Mellanox Technologies inc. All rights reserved. + */ + +#include + +/** + ** rdma_dim_step: - Moves the moderation profile one step. + ** @dim: The moderation struct. + ** + ** Description: Moves the moderation profile of @dim by one step. If we + ** are at the edge of the profile range returns DIM_ON_EDGE without + ** moving. + **/ +int rdma_dim_step(struct dim *dim) +{ + switch (dim->tune_state) { + case DIM_PARKING_ON_TOP: + /* fall through */ + case DIM_PARKING_TIRED: + break; + case DIM_GOING_RIGHT: + if (dim->profile_ix == (RDMA_DIM_PARAMS_NUM_PROFILES - 1)) + return DIM_ON_EDGE; + dim->profile_ix++; + dim->steps_right++; + break; + case DIM_GOING_LEFT: + if (dim->profile_ix == 0) + return DIM_ON_EDGE; + dim->profile_ix--; + dim->steps_left++; + break; + } + + return DIM_STEPPED; +} + +/** + ** rdma_dim_stats_compare - Compares the current stats to the previous stats. + ** @curr: The current dim stats. + ** @prev: The previous dim stats. + ** + ** Description: Checks to see if we have significantly more or less + ** completions. + ** If the completions are not greatly changed checks if the completion to + ** event ratio has significantly changed. + **/ +int rdma_dim_stats_compare(struct dim_stats *curr, struct dim_stats *prev) +{ + /* first stat */ + if (!prev->cpms) + return DIM_STATS_SAME; + + if (IS_SIGNIFICANT_DIFF(curr->cpms, prev->cpms)) + return (curr->cpms > prev->cpms) ? DIM_STATS_BETTER : + DIM_STATS_WORSE; + + if (IS_SIGNIFICANT_DIFF(curr->cpe_ratio, prev->cpe_ratio)) + return (curr->cpe_ratio > prev->cpe_ratio) ? DIM_STATS_BETTER : + DIM_STATS_WORSE; + + return DIM_STATS_SAME; +} + +/** + ** rdma_dim_decision - Decides the next moderation level. + ** @curr_stats: The current dim stats. + ** @dim: The moderation struct. + ** + ** Description: Uses rdma_dim_stats_compare to decide what the next moderation + ** level should be. If the completion to event ratio is low compared to the + ** current level we reset the moderation to keep latency low. + **/ +bool rdma_dim_decision(struct dim_stats *curr_stats, struct dim *dim) +{ + int prev_ix = dim->profile_ix; + int stats_res; + int step_res; + + switch (dim->tune_state) { + case DIM_PARKING_ON_TOP: + /* fall through */ + case DIM_PARKING_TIRED: + break; + case DIM_GOING_RIGHT: + /* fall through */ + case DIM_GOING_LEFT: + stats_res = rdma_dim_stats_compare(curr_stats, + &dim->prev_stats); + + switch (stats_res) { + case DIM_STATS_SAME: + if (curr_stats->cpe_ratio <= 50 * prev_ix) + dim->profile_ix = 0; + break; + case DIM_STATS_WORSE: + dim_turn(dim); + /* fall through */ + case DIM_STATS_BETTER: + step_res = rdma_dim_step(dim); + if (step_res == DIM_ON_EDGE) + dim_turn(dim); + break; + } + break; + } + + dim->prev_stats = *curr_stats; + + return dim->profile_ix != prev_ix; +} + +/** + ** rdma_dim - Runs the adaptive moderation. + ** @dim: The moderation struct. + ** @end_sample: The sampled stats for the end of this profiling period. + ** + ** Description: Each call to rdma_dim takes the latest stats that have been + ** sampled and uses them to decide on the next moderation level once enough + ** events have been collected. + **/ +void rdma_dim(struct dim *dim, struct dim_sample *end_sample) +{ + struct dim_stats curr_stats; + u32 nevents; + + switch (dim->state) { + case DIM_MEASURE_IN_PROGRESS: + nevents = end_sample->event_ctr - dim->start_sample.event_ctr; + if (nevents < DIM_NEVENTS) { + dim_create_sample(end_sample->event_ctr, + end_sample->pkt_ctr, + end_sample->byte_ctr, + end_sample->comp_ctr, + &dim->measuring_sample); + break; + } + dim_calc_stats(&dim->start_sample, end_sample, + &curr_stats); + if (rdma_dim_decision(&curr_stats, dim)) { + dim->state = DIM_APPLY_NEW_PROFILE; + schedule_work(&dim->work); + break; + } + /* fall through */ + case DIM_START_MEASURE: + dim->state = DIM_MEASURE_IN_PROGRESS; + dim_create_sample(end_sample->event_ctr, end_sample->pkt_ctr, + end_sample->byte_ctr, end_sample->comp_ctr, + &dim->start_sample); + dim_create_sample(end_sample->event_ctr, end_sample->pkt_ctr, + end_sample->byte_ctr, end_sample->comp_ctr, + &dim->measuring_sample); + break; + case DIM_APPLY_NEW_PROFILE: + break; + } +} +EXPORT_SYMBOL(rdma_dim); From patchwork Wed May 1 14:44:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tal Gilboa X-Patchwork-Id: 10925167 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C71F614DB for ; Wed, 1 May 2019 14:45:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B4FD028B26 for ; Wed, 1 May 2019 14:45:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B290128BAB; Wed, 1 May 2019 14:45:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B20EB28B56 for ; Wed, 1 May 2019 14:45:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726683AbfEAOpF (ORCPT ); Wed, 1 May 2019 10:45:05 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40165 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727084AbfEAOpD (ORCPT ); Wed, 1 May 2019 10:45:03 -0400 Received: from Internal Mail-Server by MTLPINE2 (envelope-from talgi@mellanox.com) with ESMTPS (AES256-SHA encrypted); 1 May 2019 17:44:54 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (gen-l-vrt-692.mtl.labs.mlnx [10.141.69.20]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x41EisrF019859; Wed, 1 May 2019 17:44:54 +0300 Received: from gen-l-vrt-692.mtl.labs.mlnx (localhost [127.0.0.1]) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7) with ESMTP id x41Eis9u036056; Wed, 1 May 2019 17:44:54 +0300 Received: (from talgi@localhost) by gen-l-vrt-692.mtl.labs.mlnx (8.14.7/8.14.7/Submit) id x41EirwW036055; Wed, 1 May 2019 17:44:53 +0300 From: Tal Gilboa To: linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Cc: Yishai Hadas , Leon Romanovsky , Jason Gunthorpe , Doug Ledford , Tariq Toukan , Tal Gilboa , Saeed Mahameed , Idan Burstein , Yamin Friedman , Max Gurtovoy Subject: [PATCH rdma-for-next 9/9] drivers/infiniband: Use rdma_dim in infiniband driver Date: Wed, 1 May 2019 17:44:39 +0300 Message-Id: <1556721879-35987-10-git-send-email-talgi@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1556721879-35987-1-git-send-email-talgi@mellanox.com> References: <1556721879-35987-1-git-send-email-talgi@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Yamin Friedman Added the interface in the infiniband driver that applies the rdma_dim adaptive moderation. There is now a special function for allocating an ib_cq that uses rdma_dim. Performance improvement (ConnectX-5 100GbE, x86) running FIO benchmark over NVMf between two equal end-hosts with 56 cores across a Mellanox switch using null_blk device: READS without DIM: blk size | BW | IOPS | 99th percentile latency | 99.99th latency 512B | 3.8GiB/s | 7.7M | 1401 usec | 2442 usec 4k | 7.0GiB/s | 1.8M | 4817 usec | 6587 usec 64k | 10.7GiB/s| 175k | 9896 usec | 10028 usec IO WRITES without DIM: blk size | BW | IOPS | 99th percentile latency | 99.99th latency 512B | 3.6GiB/s | 7.5M | 1434 usec | 2474 usec 4k | 6.3GiB/s | 1.6M | 938 usec | 1221 usec 64k | 10.7GiB/s| 175k | 8979 usec | 12780 usec IO READS with DIM: blk size | BW | IOPS | 99th percentile latency | 99.99th latency 512B | 4GiB/s | 8.2M | 816 usec | 889 usec 4k | 10.1GiB/s| 2.65M| 3359 usec | 5080 usec 64k | 10.7GiB/s| 175k | 9896 usec | 10028 usec IO WRITES with DIM: blk size | BW | IOPS | 99th percentile latency | 99.99th latency 512B | 3.9GiB/s | 8.1M | 799 usec | 922 usec 4k | 9.6GiB/s | 2.5M | 717 usec | 1004 usec 64k | 10.7GiB/s| 176k | 8586 usec | 12256 usec The rdma_dim algorithm was designed to measure the effectiveness of moderation on the flow in a general way and thus should be appropriate for all RDMA storage protocols. Signed-off-by: Yamin Friedman Reviewed-by: Max Gurtovoy Signed-off-by: Tal Gilboa --- drivers/infiniband/core/cq.c | 79 +++++++++++++++++++++++++++++++++++++---- drivers/infiniband/hw/mlx4/qp.c | 2 +- drivers/infiniband/hw/mlx5/qp.c | 2 +- include/linux/irq_poll.h | 5 +++ include/rdma/ib_verbs.h | 54 +++++++++++++++++++++++++--- lib/irq_poll.c | 15 +++++++- 6 files changed, 144 insertions(+), 13 deletions(-) diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c index a4c8199..6fb3270 100644 --- a/drivers/infiniband/core/cq.c +++ b/drivers/infiniband/core/cq.c @@ -14,6 +14,7 @@ #include #include #include +#include /* # of WCs to poll for with a single call to ib_poll_cq */ #define IB_POLL_BATCH 16 @@ -26,6 +27,47 @@ #define IB_POLL_FLAGS \ (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS) +static int ib_cq_dim_modify_cq(struct ib_cq *cq, unsigned short level) +{ + u16 usec = rdma_dim_prof[level].usec; + u16 comps = rdma_dim_prof[level].comps; + + return cq->device->ops.modify_cq(cq, comps, usec); +} + +static void update_cq_moderation(struct dim *dim, struct ib_cq *cq) +{ + dim->state = DIM_START_MEASURE; + + ib_cq_dim_modify_cq(cq, dim->profile_ix); +} + +static void ib_cq_rdma_dim_workqueue_work(struct work_struct *w) +{ + struct dim *dim = container_of(w, struct dim, work); + struct ib_cq *cq = container_of(dim, struct ib_cq, workqueue_poll.dim); + + update_cq_moderation(dim, cq); +} + +static void ib_cq_rdma_dim_irqpoll_work(struct work_struct *w) +{ + struct dim *dim = container_of(w, struct dim, work); + struct irq_poll *iop = container_of(dim, struct irq_poll, dim); + struct ib_cq *cq = container_of(iop, struct ib_cq, iop); + + update_cq_moderation(dim, cq); +} + +void rdma_dim_init(struct dim *dim, work_func_t func) +{ + memset(dim, 0, sizeof(*dim)); + dim->state = DIM_START_MEASURE; + dim->tune_state = DIM_GOING_RIGHT; + dim->profile_ix = RDMA_DIM_START_PROFILE; + INIT_WORK(&dim->work, func); +} + static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *wcs, int batch) { @@ -105,19 +147,30 @@ static void ib_cq_completion_softirq(struct ib_cq *cq, void *private) static void ib_cq_poll_work(struct work_struct *work) { - struct ib_cq *cq = container_of(work, struct ib_cq, work); + struct ib_cq *cq = container_of(work, struct ib_cq, + workqueue_poll.work); int completed; + struct dim_sample e_sample; + struct dim_sample *m_sample = &cq->workqueue_poll.dim.measuring_sample; completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE, cq->wc, IB_POLL_BATCH); + + if (cq->workqueue_poll.dim_used) + dim_create_sample(m_sample->event_ctr + 1, m_sample->pkt_ctr, + m_sample->byte_ctr, + m_sample->comp_ctr + completed, &e_sample); + if (completed >= IB_POLL_BUDGET_WORKQUEUE || ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) - queue_work(cq->comp_wq, &cq->work); + queue_work(cq->comp_wq, &cq->workqueue_poll.work); + else if (cq->workqueue_poll.dim_used) + rdma_dim(&cq->workqueue_poll.dim, &e_sample); } static void ib_cq_completion_workqueue(struct ib_cq *cq, void *private) { - queue_work(cq->comp_wq, &cq->work); + queue_work(cq->comp_wq, &cq->workqueue_poll.work); } /** @@ -129,6 +182,7 @@ static void ib_cq_completion_workqueue(struct ib_cq *cq, void *private) * @poll_ctx: context to poll the CQ from. * @caller: module owner name. * @udata: Valid user data or NULL for kernel object + * @use_dim: use dynamic interrupt moderation * * This is the proper interface to allocate a CQ for in-kernel users. A * CQ allocated with this interface will automatically be polled from the @@ -138,7 +192,8 @@ static void ib_cq_completion_workqueue(struct ib_cq *cq, void *private) struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private, int nr_cqe, int comp_vector, enum ib_poll_context poll_ctx, - const char *caller, struct ib_udata *udata) + const char *caller, struct ib_udata *udata, + bool use_dim) { struct ib_cq_init_attr cq_attr = { .cqe = nr_cqe, @@ -174,12 +229,22 @@ struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private, cq->comp_handler = ib_cq_completion_softirq; irq_poll_init(&cq->iop, IB_POLL_BUDGET_IRQ, ib_poll_handler); + if (cq->device->ops.modify_cq && use_dim) { + rdma_dim_init(&cq->iop.dim, + ib_cq_rdma_dim_irqpoll_work); + cq->iop.dim_used = true; + } ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); break; case IB_POLL_WORKQUEUE: case IB_POLL_UNBOUND_WORKQUEUE: cq->comp_handler = ib_cq_completion_workqueue; - INIT_WORK(&cq->work, ib_cq_poll_work); + INIT_WORK(&cq->workqueue_poll.work, ib_cq_poll_work); + if (cq->device->ops.modify_cq && use_dim) { + rdma_dim_init(&cq->workqueue_poll.dim, + ib_cq_rdma_dim_workqueue_work); + cq->workqueue_poll.dim_used = true; + } ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); cq->comp_wq = (cq->poll_ctx == IB_POLL_WORKQUEUE) ? ib_comp_wq : ib_comp_unbound_wq; @@ -220,7 +285,9 @@ void ib_free_cq_user(struct ib_cq *cq, struct ib_udata *udata) break; case IB_POLL_WORKQUEUE: case IB_POLL_UNBOUND_WORKQUEUE: - cancel_work_sync(&cq->work); + cancel_work_sync(&cq->workqueue_poll.work); + if (cq->workqueue_poll.dim_used) + flush_work(&cq->iop.dim.work); break; default: WARN_ON_ONCE(1); diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index 364e16b..b9b550b 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -4385,7 +4385,7 @@ static void handle_drain_completion(struct ib_cq *cq, irq_poll_enable(&cq->iop); break; case IB_POLL_WORKQUEUE: - cancel_work_sync(&cq->work); + cancel_work_sync(&cq->workqueue_poll.work); break; default: WARN_ON_ONCE(1); diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index efe1f6f..ccee41a 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -6267,7 +6267,7 @@ static void handle_drain_completion(struct ib_cq *cq, irq_poll_enable(&cq->iop); break; case IB_POLL_WORKQUEUE: - cancel_work_sync(&cq->work); + cancel_work_sync(&cq->workqueue_poll.work); break; default: WARN_ON_ONCE(1); diff --git a/include/linux/irq_poll.h b/include/linux/irq_poll.h index 16aaecc..3601e75 100644 --- a/include/linux/irq_poll.h +++ b/include/linux/irq_poll.h @@ -2,6 +2,8 @@ #ifndef IRQ_POLL_H #define IRQ_POLL_H +#include + struct irq_poll; typedef int (irq_poll_fn)(struct irq_poll *, int); @@ -10,6 +12,9 @@ struct irq_poll { unsigned long state; int weight; irq_poll_fn *poll; + + bool dim_used; + struct dim dim; }; enum { diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 737ef5e..0a92549 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -1587,6 +1587,12 @@ enum ib_poll_context { IB_POLL_UNBOUND_WORKQUEUE, /* poll from unbound workqueue */ }; +struct ib_cq_workqueue_poll { + struct dim dim; + struct work_struct work; + bool dim_used; +}; + struct ib_cq { struct ib_device *device; struct ib_uobject *uobject; @@ -1598,8 +1604,8 @@ struct ib_cq { enum ib_poll_context poll_ctx; struct ib_wc *wc; union { - struct irq_poll iop; - struct work_struct work; + struct irq_poll iop; + struct ib_cq_workqueue_poll workqueue_poll; }; struct workqueue_struct *comp_wq; /* @@ -3628,7 +3634,8 @@ static inline int ib_post_recv(struct ib_qp *qp, struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private, int nr_cqe, int comp_vector, enum ib_poll_context poll_ctx, - const char *caller, struct ib_udata *udata); + const char *caller, struct ib_udata *udata, + bool use_dim); /** * ib_alloc_cq_user: Allocate kernel/user CQ @@ -3646,7 +3653,27 @@ static inline struct ib_cq *ib_alloc_cq_user(struct ib_device *dev, struct ib_udata *udata) { return __ib_alloc_cq_user(dev, private, nr_cqe, comp_vector, poll_ctx, - KBUILD_MODNAME, udata); + KBUILD_MODNAME, udata, false); +} + +/** + * ib_alloc_cq_user_dim: Allocate kernel/user CQ with dynamic interrupt + * moderation + * @dev: The IB device + * @private: Private data attached to the CQE + * @nr_cqe: Number of CQEs in the CQ + * @comp_vector: Completion vector used for the IRQs + * @poll_ctx: Context used for polling the CQ + * @udata: Valid user data or NULL for kernel objects + */ +static inline struct ib_cq *ib_alloc_cq_user_dim(struct ib_device *dev, + void *private, int nr_cqe, + int comp_vector, + enum ib_poll_context poll_ctx, + struct ib_udata *udata) +{ + return __ib_alloc_cq_user(dev, private, nr_cqe, comp_vector, poll_ctx, + KBUILD_MODNAME, udata, true); } /** @@ -3668,6 +3695,25 @@ static inline struct ib_cq *ib_alloc_cq(struct ib_device *dev, void *private, } /** + * ib_alloc_cq_dim: Allocate kernel CQ with dynamic interrupt moderation + * @dev: The IB device + * @private: Private data attached to the CQE + * @nr_cqe: Number of CQEs in the CQ + * @comp_vector: Completion vector used for the IRQs + * @poll_ctx: Context used for polling the CQ + * + * NOTE: for user cq use ib_alloc_cq_user with valid udata! + */ +static inline struct ib_cq *ib_alloc_cq_dim(struct ib_device *dev, + void *private, int nr_cqe, + int comp_vector, + enum ib_poll_context poll_ctx) +{ + return ib_alloc_cq_user_dim(dev, private, nr_cqe, comp_vector, + poll_ctx, NULL); +} + +/** * ib_free_cq_user - Free kernel/user CQ * @cq: The CQ to free * @udata: Valid user data or NULL for kernel objects diff --git a/lib/irq_poll.c b/lib/irq_poll.c index 2f17b48..c63bc9b 100644 --- a/lib/irq_poll.c +++ b/lib/irq_poll.c @@ -50,6 +50,8 @@ void irq_poll_sched(struct irq_poll *iop) **/ static void __irq_poll_complete(struct irq_poll *iop) { + if (iop->dim_used) + rdma_dim(&iop->dim, &iop->dim.measuring_sample); list_del(&iop->list); smp_mb__before_atomic(); clear_bit_unlock(IRQ_POLL_F_SCHED, &iop->state); @@ -86,6 +88,7 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) while (!list_empty(list)) { struct irq_poll *iop; int work, weight; + struct dim_sample *m_sample; /* * If softirq window is exhausted then punt. @@ -104,10 +107,18 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) */ iop = list_entry(list->next, struct irq_poll, list); + m_sample = &iop->dim.measuring_sample; weight = iop->weight; work = 0; - if (test_bit(IRQ_POLL_F_SCHED, &iop->state)) + if (test_bit(IRQ_POLL_F_SCHED, &iop->state)) { work = iop->poll(iop, weight); + if (iop->dim_used) + dim_create_sample(m_sample->event_ctr + 1, + m_sample->pkt_ctr, + m_sample->byte_ctr, + m_sample->comp_ctr + work, + &iop->dim.measuring_sample); + } budget -= work; @@ -144,6 +155,8 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) **/ void irq_poll_disable(struct irq_poll *iop) { + if (iop->dim_used) + flush_work(&iop->dim.work); set_bit(IRQ_POLL_F_DISABLE, &iop->state); while (test_and_set_bit(IRQ_POLL_F_SCHED, &iop->state)) msleep(1);