| Message ID | 1717070038-18381-1-git-send-email-sbhatta@marvell.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net-next] octeontx2: Improve mailbox tracepoints for debugging | expand |
On 5/30/2024 4:53 AM, Subbaraya Sundeep wrote: > The tracepoints present currently wrt mailbox do not > provide enough information to debug mailbox activity. > For a VF to send a message to AF, VF sends message to PF > and PF forwards it to AF. This involves stages of > PF receiving interrupt from VF, forwarding to AF, AF > processing and sending response back to PF, PF sending back > the response to VF. This patch adds pcifunc which represents > PF and VF device to the tracepoints otx2_msg_alloc, > otx2_msg_send, otx2_msg_process so that it is easier > to correlate which device allocated the message, which > device forwarded it and which device processed that message. > Also add message id in otx2_msg_send tracepoint and new > tracepoint otx2_msg_status to display the status at each > stage. > > Below is the trace log when a VF sends a message to AF with > this patch in place: > > ifconfig-523 [001] .... 146.134718: otx2_msg_alloc: [0002:05:00.1] > msg:(NIX_RSS_FLOWKEY_CFG) size:28 pcifunc:0x1001 > > ifconfig-523 [001] ...1 146.134719: otx2_msg_send: [0002:05:00.1] > sent 1 msg(s) of size:32 msg:(NIX_RSS_FLOWKEY_CFG) pcifunc:0x1001 > > <idle>-0 [000] d.h1 146.134722: otx2_msg_interrupt: [0002:05:00.0] > mbox interrupt VF(s) to PF (0x1) > > kworker/u49:2-238 [002] .... 146.134723: otx2_msg_status: [0002:05:00.0] > PF-VF down queue handler(forwarding) num_msgs:1 > > kworker/u49:2-238 [002] ...1 146.134724: otx2_msg_send: [0002:05:00.0] > sent 1 msg(s) of size:32 msg:(NIX_RSS_FLOWKEY_CFG) pcifunc:0x1001 > > <idle>-0 [000] d.h1 146.134726: otx2_msg_interrupt: [0002:01:00.0] > mbox interrupt PF(s) to AF (0x10) > > kworker/u49:1-184 [000] .... 146.134739: otx2_msg_process: [0002:01:00.0] > msg:(NIX_RSS_FLOWKEY_CFG) error:0 pcifunc:0x1001 > > kworker/u49:1-184 [000] ...1 146.134740: otx2_msg_send: [0002:01:00.0] > sent 1 msg(s) of size:32 msg:(NIX_RSS_FLOWKEY_CFG) pcifunc:0x1001 > > <idle>-0 [000] dNh2 146.134742: otx2_msg_interrupt: [0002:05:00.0] > mbox interrupt DOWN reply from AF to PF (0x1) > > <idle>-0 [000] dNh2 146.134742: otx2_msg_status: [0002:05:00.0] > PF-AF down work queued(interrupt) num_msgs:1 > > kworker/u49:1-184 [000] .... 146.134743: otx2_msg_status: [0002:05:00.0] > PF-AF down queue handler(response) num_msgs:1 > > <idle>-0 [000] d.h1 146.135730: otx2_msg_interrupt: [0002:05:00.1] > mbox interrupt DOWN reply from PF to VF (0x1) > > Signed-off-by: Subbaraya Sundeep <sbhatta@marvell.com> Makes sense. Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
Hi Subbaraya, kernel test robot noticed the following build errors: [auto build test ERROR on net-next/main] url: https://github.com/intel-lab-lkp/linux/commits/Subbaraya-Sundeep/octeontx2-Improve-mailbox-tracepoints-for-debugging/20240530-195537 base: net-next/main patch link: https://lore.kernel.org/r/1717070038-18381-1-git-send-email-sbhatta%40marvell.com patch subject: [net-next PATCH] octeontx2: Improve mailbox tracepoints for debugging config: alpha-allyesconfig (https://download.01.org/0day-ci/archive/20240531/202405310351.9HtVnVJ5-lkp@intel.com/config) compiler: alpha-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240531/202405310351.9HtVnVJ5-lkp@intel.com/reproduce) If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags | Reported-by: kernel test robot <lkp@intel.com> | Closes: https://lore.kernel.org/oe-kbuild-all/202405310351.9HtVnVJ5-lkp@intel.com/ All error/warnings (new ones prefixed by >>): drivers/net/ethernet/marvell/octeontx2/af/mbox.c: In function '__otx2_mbox_reset': >> drivers/net/ethernet/marvell/octeontx2/af/mbox.c:23:29: warning: unused variable 'msg' [-Wunused-variable] 23 | struct mbox_msghdr *msg; | ^~~ drivers/net/ethernet/marvell/octeontx2/af/mbox.c: In function 'otx2_mbox_msg_send_data': >> drivers/net/ethernet/marvell/octeontx2/af/mbox.c:254:9: error: 'msg' undeclared (first use in this function); did you mean 'ndmsg'? 254 | msg = (struct mbox_msghdr *)(hw_mbase + mbox->tx_start + msgs_offset); | ^~~ | ndmsg drivers/net/ethernet/marvell/octeontx2/af/mbox.c:254:9: note: each undeclared identifier is reported only once for each function it appears in vim +254 drivers/net/ethernet/marvell/octeontx2/af/mbox.c 216 217 static void otx2_mbox_msg_send_data(struct otx2_mbox *mbox, int devid, u64 data) 218 { 219 struct otx2_mbox_dev *mdev = &mbox->dev[devid]; 220 struct mbox_hdr *tx_hdr, *rx_hdr; 221 void *hw_mbase = mdev->hwbase; 222 u64 intr_val; 223 224 tx_hdr = hw_mbase + mbox->tx_start; 225 rx_hdr = hw_mbase + mbox->rx_start; 226 227 /* If bounce buffer is implemented copy mbox messages from 228 * bounce buffer to hw mbox memory. 229 */ 230 if (mdev->mbase != hw_mbase) 231 memcpy(hw_mbase + mbox->tx_start + msgs_offset, 232 mdev->mbase + mbox->tx_start + msgs_offset, 233 mdev->msg_size); 234 235 spin_lock(&mdev->mbox_lock); 236 237 tx_hdr->msg_size = mdev->msg_size; 238 239 /* Reset header for next messages */ 240 mdev->msg_size = 0; 241 mdev->rsp_size = 0; 242 mdev->msgs_acked = 0; 243 244 /* Sync mbox data into memory */ 245 smp_wmb(); 246 247 /* num_msgs != 0 signals to the peer that the buffer has a number of 248 * messages. So this should be written after writing all the messages 249 * to the shared memory. 250 */ 251 tx_hdr->num_msgs = mdev->num_msgs; 252 rx_hdr->num_msgs = 0; 253 > 254 msg = (struct mbox_msghdr *)(hw_mbase + mbox->tx_start + msgs_offset); 255 256 trace_otx2_msg_send(mbox->pdev, tx_hdr->num_msgs, tx_hdr->msg_size, 257 msg->id, msg->pcifunc); 258 259 spin_unlock(&mdev->mbox_lock); 260 261 /* Check if interrupt pending */ 262 intr_val = readq((void __iomem *)mbox->reg_base + 263 (mbox->trigger | (devid << mbox->tr_shift))); 264 265 intr_val |= data; 266 /* The interrupt should be fired after num_msgs is written 267 * to the shared memory 268 */ 269 writeq(intr_val, (void __iomem *)mbox->reg_base + 270 (mbox->trigger | (devid << mbox->tr_shift))); 271 } 272
Hi Subbaraya,
kernel test robot noticed the following build errors:
[auto build test ERROR on net-next/main]
url: https://github.com/intel-lab-lkp/linux/commits/Subbaraya-Sundeep/octeontx2-Improve-mailbox-tracepoints-for-debugging/20240530-195537
base: net-next/main
patch link: https://lore.kernel.org/r/1717070038-18381-1-git-send-email-sbhatta%40marvell.com
patch subject: [net-next PATCH] octeontx2: Improve mailbox tracepoints for debugging
config: um-allyesconfig (https://download.01.org/0day-ci/archive/20240531/202405310425.kxMtnCmV-lkp@intel.com/config)
compiler: gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240531/202405310425.kxMtnCmV-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202405310425.kxMtnCmV-lkp@intel.com/
All errors (new ones prefixed by >>):
In file included from include/trace/trace_events.h:419,
from include/trace/define_trace.h:102,
from drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.h:144,
from drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.c:9:
>> drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:119:1: error: macro "__assign_str" passed 2 arguments, but takes just 1
119 | );
| ^~
In file included from include/trace/trace_events.h:375:
include/trace/stages/stage6_event_callback.h:34: note: macro "__assign_str" defined here
34 | #define __assign_str(dst) \
|
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h: In function 'trace_event_raw_event_otx2_msg_wait_rsp':
>> drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:115:28: error: '__assign_str' undeclared (first use in this function)
115 | TP_fast_assign(__assign_str(dev, pci_name(pdev))
| ^~~~~~~~~~~~
include/trace/trace_events.h:402:11: note: in definition of macro 'DECLARE_EVENT_CLASS'
402 | { assign; } \
| ^~~~~~
include/trace/trace_events.h:44:30: note: in expansion of macro 'PARAMS'
44 | PARAMS(assign), \
| ^~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:110:1: note: in expansion of macro 'TRACE_EVENT'
110 | TRACE_EVENT(otx2_msg_wait_rsp,
| ^~~~~~~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:115:13: note: in expansion of macro 'TP_fast_assign'
115 | TP_fast_assign(__assign_str(dev, pci_name(pdev))
| ^~~~~~~~~~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:115:28: note: each undeclared identifier is reported only once for each function it appears in
115 | TP_fast_assign(__assign_str(dev, pci_name(pdev))
| ^~~~~~~~~~~~
include/trace/trace_events.h:402:11: note: in definition of macro 'DECLARE_EVENT_CLASS'
402 | { assign; } \
| ^~~~~~
include/trace/trace_events.h:44:30: note: in expansion of macro 'PARAMS'
44 | PARAMS(assign), \
| ^~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:110:1: note: in expansion of macro 'TRACE_EVENT'
110 | TRACE_EVENT(otx2_msg_wait_rsp,
| ^~~~~~~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:115:13: note: in expansion of macro 'TP_fast_assign'
115 | TP_fast_assign(__assign_str(dev, pci_name(pdev))
| ^~~~~~~~~~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h: At top level:
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:134:1: error: macro "__assign_str" passed 2 arguments, but takes just 1
134 | );
| ^~
include/trace/stages/stage6_event_callback.h:34: note: macro "__assign_str" defined here
34 | #define __assign_str(dst) \
|
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:134:1: error: macro "__assign_str" passed 2 arguments, but takes just 1
134 | );
| ^~
include/trace/stages/stage6_event_callback.h:34: note: macro "__assign_str" defined here
34 | #define __assign_str(dst) \
|
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h: In function 'trace_event_raw_event_otx2_msg_status':
>> drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:128:28: error: unknown type name '__assign_str'
128 | TP_fast_assign(__assign_str(dev, pci_name(pdev))
| ^~~~~~~~~~~~
include/trace/trace_events.h:402:11: note: in definition of macro 'DECLARE_EVENT_CLASS'
402 | { assign; } \
| ^~~~~~
include/trace/trace_events.h:44:30: note: in expansion of macro 'PARAMS'
44 | PARAMS(assign), \
| ^~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:121:1: note: in expansion of macro 'TRACE_EVENT'
121 | TRACE_EVENT(otx2_msg_status,
| ^~~~~~~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:128:13: note: in expansion of macro 'TP_fast_assign'
128 | TP_fast_assign(__assign_str(dev, pci_name(pdev))
| ^~~~~~~~~~~~~~
>> include/trace/stages/stage6_event_callback.h:9:17: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'entry'
9 | #define __entry entry
| ^~~~~
include/trace/trace_events.h:402:11: note: in definition of macro 'DECLARE_EVENT_CLASS'
402 | { assign; } \
| ^~~~~~
include/trace/trace_events.h:44:30: note: in expansion of macro 'PARAMS'
44 | PARAMS(assign), \
| ^~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:121:1: note: in expansion of macro 'TRACE_EVENT'
121 | TRACE_EVENT(otx2_msg_status,
| ^~~~~~~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:128:13: note: in expansion of macro 'TP_fast_assign'
128 | TP_fast_assign(__assign_str(dev, pci_name(pdev))
| ^~~~~~~~~~~~~~
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:130:28: note: in expansion of macro '__entry'
130 | __entry->num_msgs = num_msgs;
| ^~~~~~~
In file included from include/trace/trace_events.h:469:
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h: At top level:
>> drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:119:1: error: macro "__assign_str" passed 2 arguments, but takes just 1
119 | );
| ^~
include/trace/stages/stage6_event_callback.h:34: note: macro "__assign_str" defined here
34 | #define __assign_str(dst) \
|
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:134:1: error: macro "__assign_str" passed 2 arguments, but takes just 1
134 | );
| ^~
include/trace/stages/stage6_event_callback.h:34: note: macro "__assign_str" defined here
34 | #define __assign_str(dst) \
|
drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h:134:1: error: macro "__assign_str" passed 2 arguments, but takes just 1
134 | );
| ^~
include/trace/stages/stage6_event_callback.h:34: note: macro "__assign_str" defined here
34 | #define __assign_str(dst) \
|
vim +/__assign_str +119 drivers/net/ethernet/marvell/octeontx2/af/./rvu_trace.h
109
110 TRACE_EVENT(otx2_msg_wait_rsp,
111 TP_PROTO(const struct pci_dev *pdev),
112 TP_ARGS(pdev),
113 TP_STRUCT__entry(__string(dev, pci_name(pdev))
114 ),
> 115 TP_fast_assign(__assign_str(dev, pci_name(pdev))
116 ),
117 TP_printk("[%s] timed out while waiting for response\n",
118 __get_str(dev))
> 119 );
120
121 TRACE_EVENT(otx2_msg_status,
122 TP_PROTO(const struct pci_dev *pdev, const char *msg, u16 num_msgs),
123 TP_ARGS(pdev, msg, num_msgs),
124 TP_STRUCT__entry(__string(dev, pci_name(pdev))
125 __string(str, msg)
126 __field(u16, num_msgs)
127 ),
> 128 TP_fast_assign(__assign_str(dev, pci_name(pdev))
129 __assign_str(str, msg)
130 __entry->num_msgs = num_msgs;
131 ),
132 TP_printk("[%s] %s num_msgs:%d\n", __get_str(dev),
133 __get_str(str), __entry->num_msgs)
> 134 );
135
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/mbox.c b/drivers/net/ethernet/marvell/octeontx2/af/mbox.c index 1e5aa53..905da00 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/mbox.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/mbox.c @@ -20,6 +20,7 @@ void __otx2_mbox_reset(struct otx2_mbox *mbox, int devid) struct otx2_mbox_dev *mdev = &mbox->dev[devid]; struct mbox_hdr *tx_hdr, *rx_hdr; void *hw_mbase = mdev->hwbase; + struct mbox_msghdr *msg; tx_hdr = hw_mbase + mbox->tx_start; rx_hdr = hw_mbase + mbox->rx_start; @@ -188,14 +189,13 @@ int otx2_mbox_wait_for_rsp(struct otx2_mbox *mbox, int devid) { unsigned long timeout = jiffies + msecs_to_jiffies(MBOX_RSP_TIMEOUT); struct otx2_mbox_dev *mdev = &mbox->dev[devid]; - struct device *sender = &mbox->pdev->dev; while (!time_after(jiffies, timeout)) { if (mdev->num_msgs == mdev->msgs_acked) return 0; usleep_range(800, 1000); } - dev_dbg(sender, "timed out while waiting for rsp\n"); + trace_otx2_msg_wait_rsp(mbox->pdev); return -EIO; } EXPORT_SYMBOL(otx2_mbox_wait_for_rsp); @@ -251,7 +251,10 @@ static void otx2_mbox_msg_send_data(struct otx2_mbox *mbox, int devid, u64 data) tx_hdr->num_msgs = mdev->num_msgs; rx_hdr->num_msgs = 0; - trace_otx2_msg_send(mbox->pdev, tx_hdr->num_msgs, tx_hdr->msg_size); + msg = (struct mbox_msghdr *)(hw_mbase + mbox->tx_start + msgs_offset); + + trace_otx2_msg_send(mbox->pdev, tx_hdr->num_msgs, tx_hdr->msg_size, + msg->id, msg->pcifunc); spin_unlock(&mdev->mbox_lock); @@ -445,6 +448,14 @@ const char *otx2_mbox_id2name(u16 id) #define M(_name, _id, _1, _2, _3) case _id: return # _name; MBOX_MESSAGES #undef M + +#define M(_name, _id, _1, _2, _3) case _id: return # _name; + MBOX_UP_CGX_MESSAGES +#undef M + +#define M(_name, _id, _1, _2, _3) case _id: return # _name; + MBOX_UP_CPT_MESSAGES +#undef M default: return "INVALID ID"; } diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu.c index ff78251..a5e4888 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu.c @@ -2106,7 +2106,7 @@ static int rvu_process_mbox_msg(struct otx2_mbox *mbox, int devid, if (rsp && err) \ rsp->hdr.rc = err; \ \ - trace_otx2_msg_process(mbox->pdev, _id, err); \ + trace_otx2_msg_process(mbox->pdev, _id, err, req->pcifunc); \ return rsp ? err : -ENOMEM; \ } MBOX_MESSAGES diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c index 266ecbc..a335404 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c @@ -34,7 +34,7 @@ static struct _req_type __maybe_unused \ return NULL; \ req->hdr.sig = OTX2_MBOX_REQ_SIG; \ req->hdr.id = _id; \ - trace_otx2_msg_alloc(rvu->pdev, _id, sizeof(*req)); \ + trace_otx2_msg_alloc(rvu->pdev, _id, sizeof(*req), 0); \ return req; \ } diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.c index 775fd4c..5f69380 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.c @@ -11,3 +11,4 @@ EXPORT_TRACEPOINT_SYMBOL(otx2_msg_alloc); EXPORT_TRACEPOINT_SYMBOL(otx2_msg_interrupt); EXPORT_TRACEPOINT_SYMBOL(otx2_msg_process); +EXPORT_TRACEPOINT_SYMBOL(otx2_msg_status); diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.h b/drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.h index 5704520f..e7c2160 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.h +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_trace.h @@ -18,33 +18,42 @@ #include "mbox.h" TRACE_EVENT(otx2_msg_alloc, - TP_PROTO(const struct pci_dev *pdev, u16 id, u64 size), - TP_ARGS(pdev, id, size), + TP_PROTO(const struct pci_dev *pdev, u16 id, u64 size, u16 pcifunc), + TP_ARGS(pdev, id, size, pcifunc), TP_STRUCT__entry(__string(dev, pci_name(pdev)) __field(u16, id) __field(u64, size) + __field(u16, pcifunc) ), TP_fast_assign(__assign_str(dev); __entry->id = id; __entry->size = size; + __entry->pcifunc = pcifunc; ), - TP_printk("[%s] msg:(%s) size:%lld\n", __get_str(dev), - otx2_mbox_id2name(__entry->id), __entry->size) + TP_printk("[%s] msg:(%s) size:%lld pcifunc:0x%x\n", __get_str(dev), + otx2_mbox_id2name(__entry->id), __entry->size, + __entry->pcifunc) ); TRACE_EVENT(otx2_msg_send, - TP_PROTO(const struct pci_dev *pdev, u16 num_msgs, u64 msg_size), - TP_ARGS(pdev, num_msgs, msg_size), + TP_PROTO(const struct pci_dev *pdev, u16 num_msgs, u64 msg_size, + u16 id, u16 pcifunc), + TP_ARGS(pdev, num_msgs, msg_size, id, pcifunc), TP_STRUCT__entry(__string(dev, pci_name(pdev)) __field(u16, num_msgs) __field(u64, msg_size) + __field(u16, id) + __field(u16, pcifunc) ), TP_fast_assign(__assign_str(dev); __entry->num_msgs = num_msgs; __entry->msg_size = msg_size; + __entry->id = id; + __entry->pcifunc = pcifunc; ), - TP_printk("[%s] sent %d msg(s) of size:%lld\n", __get_str(dev), - __entry->num_msgs, __entry->msg_size) + TP_printk("[%s] sent %d msg(s) of size:%lld msg:(%s) pcifunc:0x%x\n", + __get_str(dev), __entry->num_msgs, __entry->msg_size, + otx2_mbox_id2name(__entry->id), __entry->pcifunc) ); TRACE_EVENT(otx2_msg_check, @@ -81,18 +90,47 @@ TRACE_EVENT(otx2_msg_interrupt, ); TRACE_EVENT(otx2_msg_process, - TP_PROTO(const struct pci_dev *pdev, u16 id, int err), - TP_ARGS(pdev, id, err), + TP_PROTO(const struct pci_dev *pdev, u16 id, int err, u16 pcifunc), + TP_ARGS(pdev, id, err, pcifunc), TP_STRUCT__entry(__string(dev, pci_name(pdev)) __field(u16, id) __field(int, err) + __field(u16, pcifunc) ), TP_fast_assign(__assign_str(dev); __entry->id = id; __entry->err = err; + __entry->pcifunc = pcifunc; + ), + TP_printk("[%s] msg:(%s) error:%d pcifunc:0x%x\n", __get_str(dev), + otx2_mbox_id2name(__entry->id), + __entry->err, __entry->pcifunc) +); + +TRACE_EVENT(otx2_msg_wait_rsp, + TP_PROTO(const struct pci_dev *pdev), + TP_ARGS(pdev), + TP_STRUCT__entry(__string(dev, pci_name(pdev)) + ), + TP_fast_assign(__assign_str(dev, pci_name(pdev)) + ), + TP_printk("[%s] timed out while waiting for response\n", + __get_str(dev)) +); + +TRACE_EVENT(otx2_msg_status, + TP_PROTO(const struct pci_dev *pdev, const char *msg, u16 num_msgs), + TP_ARGS(pdev, msg, num_msgs), + TP_STRUCT__entry(__string(dev, pci_name(pdev)) + __string(str, msg) + __field(u16, num_msgs) + ), + TP_fast_assign(__assign_str(dev, pci_name(pdev)) + __assign_str(str, msg) + __entry->num_msgs = num_msgs; ), - TP_printk("[%s] msg:(%s) error:%d\n", __get_str(dev), - otx2_mbox_id2name(__entry->id), __entry->err) + TP_printk("[%s] %s num_msgs:%d\n", __get_str(dev), + __get_str(str), __entry->num_msgs) ); #endif /* __RVU_TRACE_H */ diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h index 24fbbef..f441103 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h @@ -847,6 +847,7 @@ static struct _req_type __maybe_unused \ *otx2_mbox_alloc_msg_ ## _fn_name(struct mbox *mbox) \ { \ struct _req_type *req; \ + u16 pcifunc = mbox->pfvf->pcifunc; \ \ req = (struct _req_type *)otx2_mbox_alloc_msg_rsp( \ &mbox->mbox, 0, sizeof(struct _req_type), \ @@ -855,7 +856,8 @@ static struct _req_type __maybe_unused \ return NULL; \ req->hdr.sig = OTX2_MBOX_REQ_SIG; \ req->hdr.id = _id; \ - trace_otx2_msg_alloc(mbox->mbox.pdev, _id, sizeof(*req)); \ + req->hdr.pcifunc = pcifunc; \ + trace_otx2_msg_alloc(mbox->mbox.pdev, _id, sizeof(*req), pcifunc); \ return req; \ } diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c index f5bce3e..77156f4 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c @@ -463,6 +463,9 @@ static void otx2_pfvf_mbox_handler(struct work_struct *work) offset = ALIGN(sizeof(struct mbox_hdr), MBOX_MSG_ALIGN); + trace_otx2_msg_status(pf->pdev, "PF-VF down queue handler(forwarding)", + vf_mbox->num_msgs); + for (id = 0; id < vf_mbox->num_msgs; id++) { msg = (struct mbox_msghdr *)(mdev->mbase + mbox->rx_start + offset); @@ -471,7 +474,7 @@ static void otx2_pfvf_mbox_handler(struct work_struct *work) goto inval_msg; /* Set VF's number in each of the msg */ - msg->pcifunc &= RVU_PFVF_FUNC_MASK; + msg->pcifunc &= ~RVU_PFVF_FUNC_MASK; msg->pcifunc |= (vf_idx + 1) & RVU_PFVF_FUNC_MASK; offset = msg->next_msgoff; } @@ -501,6 +504,9 @@ static void otx2_pfvf_mbox_up_handler(struct work_struct *work) offset = mbox->rx_start + ALIGN(sizeof(struct mbox_hdr), MBOX_MSG_ALIGN); + trace_otx2_msg_status(pf->pdev, "PF-VF up queue handler(response)", + vf_mbox->up_num_msgs); + for (id = 0; id < vf_mbox->up_num_msgs; id++) { msg = mdev->mbase + offset; @@ -816,6 +822,9 @@ static void otx2_pfaf_mbox_handler(struct work_struct *work) offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); pf = af_mbox->pfvf; + trace_otx2_msg_status(pf->pdev, "PF-AF down queue handler(response)", + num_msgs); + for (id = 0; id < num_msgs; id++) { msg = (struct mbox_msghdr *)(mdev->mbase + offset); otx2_process_pfaf_mbox_msg(pf, msg); @@ -938,6 +947,9 @@ static void otx2_pfaf_mbox_up_handler(struct work_struct *work) offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); + trace_otx2_msg_status(pf->pdev, "PF-AF up queue handler(notification)", + num_msgs); + for (id = 0; id < num_msgs; id++) { msg = (struct mbox_msghdr *)(mdev->mbase + offset); @@ -987,6 +999,9 @@ static irqreturn_t otx2_pfaf_mbox_intr_handler(int irq, void *pf_irq) trace_otx2_msg_interrupt(pf->pdev, "UP message from AF to PF", BIT_ULL(0)); + + trace_otx2_msg_status(pf->pdev, "PF-AF up work queued(interrupt)", + hdr->num_msgs); } if (mbox_data & MBOX_DOWN_MSG) { @@ -1003,6 +1018,9 @@ static irqreturn_t otx2_pfaf_mbox_intr_handler(int irq, void *pf_irq) trace_otx2_msg_interrupt(pf->pdev, "DOWN reply from AF to PF", BIT_ULL(0)); + + trace_otx2_msg_status(pf->pdev, "PF-AF down work queued(interrupt)", + hdr->num_msgs); } return IRQ_HANDLED; @@ -3170,6 +3188,7 @@ static void otx2_vf_link_event_task(struct work_struct *work) req = (struct cgx_link_info_msg *)msghdr; req->hdr.id = MBOX_MSG_CGX_LINK_EVENT; req->hdr.sig = OTX2_MBOX_REQ_SIG; + req->hdr.pcifunc = pf->pcifunc; memcpy(&req->link_info, &pf->linfo, sizeof(req->link_info)); otx2_mbox_wait_for_zero(&pf->mbox_pfvf[0].mbox_up, vf_idx); diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c index 99fcc56..cb94dce 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c @@ -134,7 +134,7 @@ static int otx2vf_process_mbox_msg_up(struct otx2_nic *vf, rsp->hdr.id = MBOX_MSG_CGX_LINK_EVENT; rsp->hdr.sig = OTX2_MBOX_RSP_SIG; - rsp->hdr.pcifunc = 0; + rsp->hdr.pcifunc = req->pcifunc; rsp->hdr.rc = 0; err = otx2_mbox_up_handler_cgx_link_event( vf, (struct cgx_link_info_msg *)req, rsp);
The tracepoints present currently wrt mailbox do not provide enough information to debug mailbox activity. For a VF to send a message to AF, VF sends message to PF and PF forwards it to AF. This involves stages of PF receiving interrupt from VF, forwarding to AF, AF processing and sending response back to PF, PF sending back the response to VF. This patch adds pcifunc which represents PF and VF device to the tracepoints otx2_msg_alloc, otx2_msg_send, otx2_msg_process so that it is easier to correlate which device allocated the message, which device forwarded it and which device processed that message. Also add message id in otx2_msg_send tracepoint and new tracepoint otx2_msg_status to display the status at each stage. Below is the trace log when a VF sends a message to AF with this patch in place: ifconfig-523 [001] .... 146.134718: otx2_msg_alloc: [0002:05:00.1] msg:(NIX_RSS_FLOWKEY_CFG) size:28 pcifunc:0x1001 ifconfig-523 [001] ...1 146.134719: otx2_msg_send: [0002:05:00.1] sent 1 msg(s) of size:32 msg:(NIX_RSS_FLOWKEY_CFG) pcifunc:0x1001 <idle>-0 [000] d.h1 146.134722: otx2_msg_interrupt: [0002:05:00.0] mbox interrupt VF(s) to PF (0x1) kworker/u49:2-238 [002] .... 146.134723: otx2_msg_status: [0002:05:00.0] PF-VF down queue handler(forwarding) num_msgs:1 kworker/u49:2-238 [002] ...1 146.134724: otx2_msg_send: [0002:05:00.0] sent 1 msg(s) of size:32 msg:(NIX_RSS_FLOWKEY_CFG) pcifunc:0x1001 <idle>-0 [000] d.h1 146.134726: otx2_msg_interrupt: [0002:01:00.0] mbox interrupt PF(s) to AF (0x10) kworker/u49:1-184 [000] .... 146.134739: otx2_msg_process: [0002:01:00.0] msg:(NIX_RSS_FLOWKEY_CFG) error:0 pcifunc:0x1001 kworker/u49:1-184 [000] ...1 146.134740: otx2_msg_send: [0002:01:00.0] sent 1 msg(s) of size:32 msg:(NIX_RSS_FLOWKEY_CFG) pcifunc:0x1001 <idle>-0 [000] dNh2 146.134742: otx2_msg_interrupt: [0002:05:00.0] mbox interrupt DOWN reply from AF to PF (0x1) <idle>-0 [000] dNh2 146.134742: otx2_msg_status: [0002:05:00.0] PF-AF down work queued(interrupt) num_msgs:1 kworker/u49:1-184 [000] .... 146.134743: otx2_msg_status: [0002:05:00.0] PF-AF down queue handler(response) num_msgs:1 <idle>-0 [000] d.h1 146.135730: otx2_msg_interrupt: [0002:05:00.1] mbox interrupt DOWN reply from PF to VF (0x1) Signed-off-by: Subbaraya Sundeep <sbhatta@marvell.com> --- drivers/net/ethernet/marvell/octeontx2/af/mbox.c | 17 ++++-- drivers/net/ethernet/marvell/octeontx2/af/rvu.c | 2 +- .../net/ethernet/marvell/octeontx2/af/rvu_cgx.c | 2 +- .../net/ethernet/marvell/octeontx2/af/rvu_trace.c | 1 + .../net/ethernet/marvell/octeontx2/af/rvu_trace.h | 62 +++++++++++++++++----- .../ethernet/marvell/octeontx2/nic/otx2_common.h | 4 +- .../net/ethernet/marvell/octeontx2/nic/otx2_pf.c | 21 +++++++- .../net/ethernet/marvell/octeontx2/nic/otx2_vf.c | 2 +- 8 files changed, 91 insertions(+), 20 deletions(-)