Message ID | 20240805131815.7588-1-gakula@marvell.com (mailing list archive) |
---|---|
Headers | show |
Series | Introduce RVU representors | expand |
On Mon, 5 Aug 2024 18:48:04 +0530 Geetha sowjanya wrote: > This series adds representor support for each rvu devices. > When switchdev mode is enabled, representor netdev is registered > for each rvu device. In implementation of representor model, > one NIX HW LF with multiple SQ and RQ is reserved, where each > RQ and SQ of the LF are mapped to a representor. A loopback channel > is reserved to support packet path between representors and VFs. > CN10K silicon supports 2 types of MACs, RPM and SDP. This > patch set adds representor support for both RPM and SDP MAC > interfaces. > > - Patch 1: Refactors and exports the shared service functions. > - Patch 2: Implements basic representor driver. > - Patch 3: Add devlink support to create representor netdevs that > can be used to manage VFs. > - Patch 4: Implements basec netdev_ndo_ops. > - Patch 5: Installs tcam rules to route packets between representor and > VFs. > - Patch 6: Enables fetching VF stats via representor interface > - Patch 7: Adds support to sync link state between representors and VFs . > - Patch 8: Enables configuring VF MTU via representor netdevs. > - Patch 9: Add representors for sdp MAC. > - Patch 10: Add devlink port support. I can't bring myself to apply this. Jiri do you have an opinion? The device is a NPU/SmartNIC/DPU/IPU, it should be very flexible. Yet, instead of just implementing the representors like everyone else you do your own thing and create separate bus devices. Not sure if this breaks anything today, but it certainly subverts the model where representors represent bus devices. You can't represent all bus devices, because of the obvious cycle. Also your documentation is full of typos.
- Patch 10: Add devlink port support. > >I can't bring myself to apply this. >Jiri do you have an opinion? >The device is a NPU/SmartNIC/DPU/IPU, it should be very flexible. >Yet, instead of just implementing the representors like everyone else you do your >own thing and create separate bus devices. Can you please elaborate what you mean by "create separate bus devices" Just to clarify we are not creating separate bus devices for representors perse. On our HW, there are multiple SRIOV PCI devices. We are using one of those PCI devices (pci/0002:1c:00.0), to attach required hardware resources to it, for doing packet IO Between representors and the representees. Once the HW resources (Rx, Tx queues) are attached to it we are registering multiple netdevs from this ie one rx/tx queue for one representor. Beyond that there are no other bus devices being created/used. >Not sure if this breaks anything today, but it certainly subverts the model where >representors represent bus devices. >You can't represent all bus devices, because of the obvious cycle. >
On Thu, 8 Aug 2024 11:30:29 +0000 Sunil Kovvuri Goutham wrote: > >I can't bring myself to apply this. > >Jiri do you have an opinion? > >The device is a NPU/SmartNIC/DPU/IPU, it should be very flexible. > >Yet, instead of just implementing the representors like everyone else you do your > >own thing and create separate bus devices. > > Can you please elaborate what you mean by "create separate bus devices" You have a separate PF (on top of the existing "AF" PF) which doesn't have any datapath port of its own. > Just to clarify we are not creating separate bus devices for representors perse. > On our HW, there are multiple SRIOV PCI devices. As in SR-IOV-capable PFs? > We are using one of those PCI devices (pci/0002:1c:00.0), to attach required hardware resources to it, for doing packet IO > Between representors and the representees. Once the HW resources (Rx, Tx queues) are attached to it > we are registering multiple netdevs from this ie one rx/tx queue for one representor. > Beyond that there are no other bus devices being created/used. What port is used for communication to the host and the uplink? We're on v10 and I keep asking for documentation showing the system topology.
Mon, Aug 05, 2024 at 03:18:04PM CEST, gakula@marvell.com wrote: >This series adds representor support for each rvu devices. >When switchdev mode is enabled, representor netdev is registered >for each rvu device. In implementation of representor model, >one NIX HW LF with multiple SQ and RQ is reserved, where each >RQ and SQ of the LF are mapped to a representor. A loopback channel >is reserved to support packet path between representors and VFs. >CN10K silicon supports 2 types of MACs, RPM and SDP. This >patch set adds representor support for both RPM and SDP MAC >interfaces. > >- Patch 1: Refactors and exports the shared service functions. >- Patch 2: Implements basic representor driver. >- Patch 3: Add devlink support to create representor netdevs that > can be used to manage VFs. >- Patch 4: Implements basec netdev_ndo_ops. >- Patch 5: Installs tcam rules to route packets between representor and > VFs. >- Patch 6: Enables fetching VF stats via representor interface >- Patch 7: Adds support to sync link state between representors and VFs . >- Patch 8: Enables configuring VF MTU via representor netdevs. >- Patch 9: Add representors for sdp MAC. >- Patch 10: Add devlink port support. > > >Command to create PF/VF representor >#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev >VF representors are created for each VF when switch mode is set switchdev on representor PCI device > >#devlink dev >pci/0002:01:00.0 >pci/0002:02:00.0 >pci/0002:1c:00.0 What are these 3 instances representing? How many PFs do you have? 3? How many physical ports you have? > >#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev > ># ip link show > eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 7e:58:2d:b6:97:51 brd ff:ff:ff:ff:ff:ff What is this eth0? Why isn't it connected to any devlink port? > r0p1v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 7e:5a:66:ea:fe:d6 brd ff:ff:ff:ff:ff:ff > r1p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether de:29:be:10:9e:bf brd ff:ff:ff:ff:ff:ff > r2p1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 4a:12:c7:a2:66:ad brd ff:ff:ff:ff:ff:ff > r3p1v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether c2:b8:a8:0e:73:fd brd ff:ff:ff:ff:ff:ff > > >~# devlink port >pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 pfnum 1 vfnum 0 external false splittable false >pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 pfnum 1 vfnum 1 external false splittable false >pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 pfnum 1 vfnum 2 external false splittable false >pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 pfnum 1 vfnum 3 external false splittable false You are missing physical port devlink instance here? Where is it? > > >----------- >v1-v2: > -Fixed build warnings. > -Address review comments provided by "Kalesh Anakkur Purayil". > >v2-v3: > - Used extack for error messages. > - As suggested reworked commit messages. > - Fixed sparse warning. > >v3-v4: > - Patch 2 & 3: Fixed coccinelle reported warnings. > - Patch 10: Added devlink port support. When someone reviews your patchset at some version, you put him to cc list from that point. Why didn't you put me to cc list? > >v4-v5: > - Patch 3: Removed devm_* usage in rvu_rep_create() > - Patch 3: Fixed build warnings. > >v5-v6: > - Addressed review comments provided by "Simon Horman". > - Added review tag. > >v6-v7: > - Rebased on top net-next branch. > >v7-v8: > - Implmented offload stats ndo. > - Added documentation. > >v8-v9: > - Updated the documentation. > >v9-v10: > - Fixed build warning w.r.t documentation. > >Geetha sowjanya (11): > octeontx2-pf: Refactoring RVU driver > octeontx2-pf: RVU representor driver > octeontx2-pf: Create representor netdev > octeontx2-pf: Add basic net_device_ops > octeontx2-af: Add packet path between representor and VF > octeontx2-pf: Get VF stats via representor > octeontx2-pf: Add support to sync link state between representor and > VFs > octeontx2-pf: Configure VF mtu via representor > octeontx2-pf: Add representors for sdp MAC > octeontx2-pf: Add devlink port support > octeontx2-pf: Implement offload stats ndo for representors > > .../ethernet/marvell/octeontx2.rst | 85 ++ > .../net/ethernet/marvell/octeontx2/Kconfig | 8 + > .../ethernet/marvell/octeontx2/af/Makefile | 3 +- > .../ethernet/marvell/octeontx2/af/common.h | 2 + > .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ > .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + > .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + > .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- > .../marvell/octeontx2/af/rvu_debugfs.c | 27 - > .../marvell/octeontx2/af/rvu_devlink.c | 6 + > .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 +- > .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + > .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + > .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 +++++++++++ > .../marvell/octeontx2/af/rvu_struct.h | 26 + > .../marvell/octeontx2/af/rvu_switch.c | 20 +- > .../ethernet/marvell/octeontx2/nic/Makefile | 2 + > .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- > .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- > .../marvell/octeontx2/nic/otx2_common.c | 58 +- > .../marvell/octeontx2/nic/otx2_common.h | 84 +- > .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ > .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- > .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- > .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- > .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- > .../net/ethernet/marvell/octeontx2/nic/rep.c | 725 ++++++++++++++++++ > .../net/ethernet/marvell/octeontx2/nic/rep.h | 53 ++ > 28 files changed, 1962 insertions(+), 227 deletions(-) > create mode 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c > create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.c > create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.h > >-- >2.25.1 > >
>-----Original Message----- >From: Jiri Pirko <jiri@resnulli.us> >Sent: Thursday, August 8, 2024 9:17 PM >To: Geethasowjanya Akula <gakula@marvell.com> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org; >davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil >Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta ><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> >Subject: [EXTERNAL] Re: [net-next PATCH v10 00/11] Introduce RVU >representors > >Mon, Aug 05, 2024 at 03: 18: 04PM CEST, gakula@ marvell. com wrote: >This >series adds representor support for each rvu devices. >When switchdev mode >is enabled, representor netdev is registered >for each rvu device. In >implementation of >Mon, Aug 05, 2024 at 03:18:04PM CEST, gakula@marvell.com wrote: >>This series adds representor support for each rvu devices. >>When switchdev mode is enabled, representor netdev is registered for >>each rvu device. In implementation of representor model, one NIX HW LF >>with multiple SQ and RQ is reserved, where each RQ and SQ of the LF are >>mapped to a representor. A loopback channel is reserved to support >>packet path between representors and VFs. >>CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set >>adds representor support for both RPM and SDP MAC interfaces. >> >>- Patch 1: Refactors and exports the shared service functions. >>- Patch 2: Implements basic representor driver. >>- Patch 3: Add devlink support to create representor netdevs that >> can be used to manage VFs. >>- Patch 4: Implements basec netdev_ndo_ops. >>- Patch 5: Installs tcam rules to route packets between representor and >> VFs. >>- Patch 6: Enables fetching VF stats via representor interface >>- Patch 7: Adds support to sync link state between representors and VFs . >>- Patch 8: Enables configuring VF MTU via representor netdevs. >>- Patch 9: Add representors for sdp MAC. >>- Patch 10: Add devlink port support. >> >> >>Command to create PF/VF representor >>#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF >>representors are created for each VF when switch mode is set switchdev >>on representor PCI device >> >>#devlink dev >>pci/0002:01:00.0 >>pci/0002:02:00.0 >>pci/0002:1c:00.0 > >What are these 3 instances representing? How many PFs do you have? 3? >How many physical ports you have? The test setup has 3 PFs one for each physical port. Below example is for the device pci/0002:1c:00.0. > > >> >>#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev >> >># ip link show >> eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode >>DEFAULT group default qlen 1000 link/ether 7e:58:2d:b6:97:51 brd >>ff:ff:ff:ff:ff:ff > >What is this eth0? Why isn't it connected to any devlink port? > >> r0p1v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode DEFAULT group default qlen 1000 link/ether 7e:5a:66:ea:fe:d6 brd >ff:ff:ff:ff:ff:ff >> r1p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode DEFAULT group default qlen 1000 link/ether de:29:be:10:9e:bf brd >ff:ff:ff:ff:ff:ff >> r2p1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode DEFAULT group default qlen 1000 link/ether 4a:12:c7:a2:66:ad brd >ff:ff:ff:ff:ff:ff >> r3p1v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode >>DEFAULT group default qlen 1000 link/ether c2:b8:a8:0e:73:fd brd >>ff:ff:ff:ff:ff:ff >> >> >>~# devlink port >>pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 >>pfnum 1 vfnum 0 external false splittable false >>pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 >>pfnum 1 vfnum 1 external false splittable false >>pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 >>pfnum 1 vfnum 2 external false splittable false >>pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 >>pfnum 1 vfnum 3 external false splittable false > >You are missing physical port devlink instance here? Where is it? pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 pfnum 1 vfnum 0 external false splittable false This is for the PF. Below is the example on a setup with one PF before 3VFs are created. #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev # ip link show eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 7e:58:2d:b6:97:51 brd ff:ff:ff:ff:ff:ff r0p1v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 7e:5a:66:ea:fe:d6 brd ff:ff:ff:ff:ff:ff r1p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether de:29:be:10:9e:bf brd ff:ff:ff:ff:ff:ff r2p1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 4a:12:c7:a2:66:ad brd ff:ff:ff:ff:ff:ff r3p1v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether c2:b8:a8:0e:73:fd brd ff:ff:ff:ff:ff:ff Above shows the PF physical port and 4 representors(1 for PF and 3 for VFs). # devlink port pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 pfnum 1 vfnum 0 external false splittable false pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 pfnum 1 vfnum 1 external false splittable false pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 pfnum 1 vfnum 2 external false splittable false pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 pfnum 1 vfnum 3 external false splittable false > >> >> >>----------- >>v1-v2: >> -Fixed build warnings. >> -Address review comments provided by "Kalesh Anakkur Purayil". >> >>v2-v3: >> - Used extack for error messages. >> - As suggested reworked commit messages. >> - Fixed sparse warning. >> >>v3-v4: >> - Patch 2 & 3: Fixed coccinelle reported warnings. >> - Patch 10: Added devlink port support. > > >When someone reviews your patchset at some version, you put him to cc list >from that point. Why didn't you put me to cc list? > > >> >>v4-v5: >> - Patch 3: Removed devm_* usage in rvu_rep_create() >> - Patch 3: Fixed build warnings. >> >>v5-v6: >> - Addressed review comments provided by "Simon Horman". >> - Added review tag. >> >>v6-v7: >> - Rebased on top net-next branch. >> >>v7-v8: >> - Implmented offload stats ndo. >> - Added documentation. >> >>v8-v9: >> - Updated the documentation. >> >>v9-v10: >> - Fixed build warning w.r.t documentation. >> >>Geetha sowjanya (11): >> octeontx2-pf: Refactoring RVU driver >> octeontx2-pf: RVU representor driver >> octeontx2-pf: Create representor netdev >> octeontx2-pf: Add basic net_device_ops >> octeontx2-af: Add packet path between representor and VF >> octeontx2-pf: Get VF stats via representor >> octeontx2-pf: Add support to sync link state between representor and >> VFs >> octeontx2-pf: Configure VF mtu via representor >> octeontx2-pf: Add representors for sdp MAC >> octeontx2-pf: Add devlink port support >> octeontx2-pf: Implement offload stats ndo for representors >> >> .../ethernet/marvell/octeontx2.rst | 85 ++ >> .../net/ethernet/marvell/octeontx2/Kconfig | 8 + >> .../ethernet/marvell/octeontx2/af/Makefile | 3 +- >> .../ethernet/marvell/octeontx2/af/common.h | 2 + >> .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ >> .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + >> .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + >> .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- >> .../marvell/octeontx2/af/rvu_debugfs.c | 27 - >> .../marvell/octeontx2/af/rvu_devlink.c | 6 + >> .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 +- >> .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + >> .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + >> .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 +++++++++++ >> .../marvell/octeontx2/af/rvu_struct.h | 26 + >> .../marvell/octeontx2/af/rvu_switch.c | 20 +- >> .../ethernet/marvell/octeontx2/nic/Makefile | 2 + >> .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- >> .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- >> .../marvell/octeontx2/nic/otx2_common.c | 58 +- >> .../marvell/octeontx2/nic/otx2_common.h | 84 +- >> .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ >> .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- >> .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- >> .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- >> .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- >> .../net/ethernet/marvell/octeontx2/nic/rep.c | 725 ++++++++++++++++++ >> .../net/ethernet/marvell/octeontx2/nic/rep.h | 53 ++ >> 28 files changed, 1962 insertions(+), 227 deletions(-) create mode >> 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c >> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.c >> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.h >> >>-- >>2.25.1 >> >>
Fri, Aug 16, 2024 at 03:36:25PM CEST, gakula@marvell.com wrote: > > >>-----Original Message----- >>From: Jiri Pirko <jiri@resnulli.us> >>Sent: Thursday, August 8, 2024 9:17 PM >>To: Geethasowjanya Akula <gakula@marvell.com> >>Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org; >>davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil >>Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta >><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> >>Subject: [EXTERNAL] Re: [net-next PATCH v10 00/11] Introduce RVU >>representors >> >>Mon, Aug 05, 2024 at 03: 18: 04PM CEST, gakula@ marvell. com wrote: >This >>series adds representor support for each rvu devices. >When switchdev mode >>is enabled, representor netdev is registered >for each rvu device. In >>implementation of >>Mon, Aug 05, 2024 at 03:18:04PM CEST, gakula@marvell.com wrote: >>>This series adds representor support for each rvu devices. >>>When switchdev mode is enabled, representor netdev is registered for >>>each rvu device. In implementation of representor model, one NIX HW LF >>>with multiple SQ and RQ is reserved, where each RQ and SQ of the LF are >>>mapped to a representor. A loopback channel is reserved to support >>>packet path between representors and VFs. >>>CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set >>>adds representor support for both RPM and SDP MAC interfaces. >>> >>>- Patch 1: Refactors and exports the shared service functions. >>>- Patch 2: Implements basic representor driver. >>>- Patch 3: Add devlink support to create representor netdevs that >>> can be used to manage VFs. >>>- Patch 4: Implements basec netdev_ndo_ops. >>>- Patch 5: Installs tcam rules to route packets between representor and >>> VFs. >>>- Patch 6: Enables fetching VF stats via representor interface >>>- Patch 7: Adds support to sync link state between representors and VFs . >>>- Patch 8: Enables configuring VF MTU via representor netdevs. >>>- Patch 9: Add representors for sdp MAC. >>>- Patch 10: Add devlink port support. >>> >>> >>>Command to create PF/VF representor >>>#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF >>>representors are created for each VF when switch mode is set switchdev >>>on representor PCI device >>> >>>#devlink dev >>>pci/0002:01:00.0 >>>pci/0002:02:00.0 >>>pci/0002:1c:00.0 >> >>What are these 3 instances representing? How many PFs do you have? 3? >>How many physical ports you have? >The test setup has 3 PFs one for each physical port. > >Below example is for the device pci/0002:1c:00.0. 3 port nic, that sounds odd. Is this something shipped already? Care to paste a link? >> >> >>> >>>#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev >>> >>># ip link show >>> eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>mode >>>DEFAULT group default qlen 1000 link/ether 7e:58:2d:b6:97:51 brd >>>ff:ff:ff:ff:ff:ff >> >>What is this eth0? Why isn't it connected to any devlink port? >> >>> r0p1v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>mode DEFAULT group default qlen 1000 link/ether 7e:5a:66:ea:fe:d6 brd >>ff:ff:ff:ff:ff:ff >>> r1p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>mode DEFAULT group default qlen 1000 link/ether de:29:be:10:9e:bf brd >>ff:ff:ff:ff:ff:ff >>> r2p1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>mode DEFAULT group default qlen 1000 link/ether 4a:12:c7:a2:66:ad brd >>ff:ff:ff:ff:ff:ff >>> r3p1v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>mode >>>DEFAULT group default qlen 1000 link/ether c2:b8:a8:0e:73:fd brd >>>ff:ff:ff:ff:ff:ff >>> >>> >>>~# devlink port >>>pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 >>>pfnum 1 vfnum 0 external false splittable false >>>pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 >>>pfnum 1 vfnum 1 external false splittable false >>>pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 >>>pfnum 1 vfnum 2 external false splittable false >>>pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 >>>pfnum 1 vfnum 3 external false splittable false >> >>You are missing physical port devlink instance here? Where is it? >pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 pfnum 1 vfnum 0 external false splittable false >This is for the PF. That is not a physical port. The flavour should be "physical" for them. > >Below is the example on a setup with one PF before 3VFs are created. > >#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev > ># ip link show > eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode >DEFAULT group default qlen 1000 link/ether 7e:58:2d:b6:97:51 brd ff:ff:ff:ff:ff:ff >r0p1v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode DEFAULT group default qlen 1000 link/ether 7e:5a:66:ea:fe:d6 brd ff:ff:ff:ff:ff:ff >r1p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode DEFAULT group default qlen 1000 link/ether de:29:be:10:9e:bf brd >ff:ff:ff:ff:ff:ff >r2p1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode DEFAULT group default qlen 1000 link/ether 4a:12:c7:a2:66:ad brd ff:ff:ff:ff:ff:ff >r3p1v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode How you setup the netdevice names? Do you have custom udev rule? Would be great to give here the out-of-box udev names instead. >DEFAULT group default qlen 1000 link/ether c2:b8:a8:0e:73:fd brd ff:ff:ff:ff:ff:ff > >Above shows the PF physical port and 4 representors(1 for PF and 3 for VFs). ># devlink port >pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 pfnum 1 vfnum 0 external false splittable false >pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 pfnum 1 vfnum 1 external false splittable false >pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 pfnum 1 vfnum 2 external false splittable false >pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 pfnum 1 vfnum 3 external false splittable false > >> >>> >>> >>>----------- >>>v1-v2: >>> -Fixed build warnings. >>> -Address review comments provided by "Kalesh Anakkur Purayil". >>> >>>v2-v3: >>> - Used extack for error messages. >>> - As suggested reworked commit messages. >>> - Fixed sparse warning. >>> >>>v3-v4: >>> - Patch 2 & 3: Fixed coccinelle reported warnings. >>> - Patch 10: Added devlink port support. >> >> >>When someone reviews your patchset at some version, you put him to cc list >>from that point. Why didn't you put me to cc list? >> >> >>> >>>v4-v5: >>> - Patch 3: Removed devm_* usage in rvu_rep_create() >>> - Patch 3: Fixed build warnings. >>> >>>v5-v6: >>> - Addressed review comments provided by "Simon Horman". >>> - Added review tag. >>> >>>v6-v7: >>> - Rebased on top net-next branch. >>> >>>v7-v8: >>> - Implmented offload stats ndo. >>> - Added documentation. >>> >>>v8-v9: >>> - Updated the documentation. >>> >>>v9-v10: >>> - Fixed build warning w.r.t documentation. >>> >>>Geetha sowjanya (11): >>> octeontx2-pf: Refactoring RVU driver >>> octeontx2-pf: RVU representor driver >>> octeontx2-pf: Create representor netdev >>> octeontx2-pf: Add basic net_device_ops >>> octeontx2-af: Add packet path between representor and VF >>> octeontx2-pf: Get VF stats via representor >>> octeontx2-pf: Add support to sync link state between representor and >>> VFs >>> octeontx2-pf: Configure VF mtu via representor >>> octeontx2-pf: Add representors for sdp MAC >>> octeontx2-pf: Add devlink port support >>> octeontx2-pf: Implement offload stats ndo for representors >>> >>> .../ethernet/marvell/octeontx2.rst | 85 ++ >>> .../net/ethernet/marvell/octeontx2/Kconfig | 8 + >>> .../ethernet/marvell/octeontx2/af/Makefile | 3 +- >>> .../ethernet/marvell/octeontx2/af/common.h | 2 + >>> .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ >>> .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + >>> .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + >>> .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- >>> .../marvell/octeontx2/af/rvu_debugfs.c | 27 - >>> .../marvell/octeontx2/af/rvu_devlink.c | 6 + >>> .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 +- >>> .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + >>> .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + >>> .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 +++++++++++ >>> .../marvell/octeontx2/af/rvu_struct.h | 26 + >>> .../marvell/octeontx2/af/rvu_switch.c | 20 +- >>> .../ethernet/marvell/octeontx2/nic/Makefile | 2 + >>> .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- >>> .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- >>> .../marvell/octeontx2/nic/otx2_common.c | 58 +- >>> .../marvell/octeontx2/nic/otx2_common.h | 84 +- >>> .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ >>> .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- >>> .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- >>> .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- >>> .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- >>> .../net/ethernet/marvell/octeontx2/nic/rep.c | 725 ++++++++++++++++++ >>> .../net/ethernet/marvell/octeontx2/nic/rep.h | 53 ++ >>> 28 files changed, 1962 insertions(+), 227 deletions(-) create mode >>> 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c >>> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.c >>> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.h >>> >>>-- >>>2.25.1 >>> >>>
>-----Original Message----- >From: Jiri Pirko <jiri@resnulli.us> >Sent: Friday, August 16, 2024 7:47 PM >To: Geethasowjanya Akula <gakula@marvell.com> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org; >davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil >Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta ><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> >Subject: Re: [EXTERNAL] Re: [net-next PATCH v10 00/11] Introduce RVU >representors > >Fri, Aug 16, 2024 at 03: 36: 25PM CEST, gakula@ marvell. com wrote: > > >>----- >Original Message----- >>From: Jiri Pirko <jiri@ resnulli. us> >>Sent: Thursday, >August 8, 2024 9: 17 PM >>To: Geethasowjanya Akula <gakula@ marvell. com> >> >>>-----Original Message----- >>>From: Jiri Pirko <jiri@resnulli.us> >>>Sent: Thursday, August 8, 2024 9:17 PM >>>To: Geethasowjanya Akula <gakula@marvell.com> >>>Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; >>>kuba@kernel.org; davem@davemloft.net; pabeni@redhat.com; >>>edumazet@google.com; Sunil Kovvuri Goutham <sgoutham@marvell.com>; >>>Subbaraya Sundeep Bhatta <sbhatta@marvell.com>; Hariprasad Kelam >>><hkelam@marvell.com> >>>Subject: [EXTERNAL] Re: [net-next PATCH v10 00/11] Introduce RVU >>>representors >>> >>>Mon, Aug 05, 2024 at 03: 18: 04PM CEST, gakula@ marvell. com wrote: >>>>This series adds representor support for each rvu devices. >When >>>switchdev mode is enabled, representor netdev is registered >for each >>>rvu device. In implementation of Mon, Aug 05, 2024 at 03:18:04PM CEST, >>>gakula@marvell.com wrote: >>>>This series adds representor support for each rvu devices. >>>>When switchdev mode is enabled, representor netdev is registered for >>>>each rvu device. In implementation of representor model, one NIX HW >>>>LF with multiple SQ and RQ is reserved, where each RQ and SQ of the >>>>LF are mapped to a representor. A loopback channel is reserved to >>>>support packet path between representors and VFs. >>>>CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set >>>>adds representor support for both RPM and SDP MAC interfaces. >>>> >>>>- Patch 1: Refactors and exports the shared service functions. >>>>- Patch 2: Implements basic representor driver. >>>>- Patch 3: Add devlink support to create representor netdevs that >>>> can be used to manage VFs. >>>>- Patch 4: Implements basec netdev_ndo_ops. >>>>- Patch 5: Installs tcam rules to route packets between representor and >>>> VFs. >>>>- Patch 6: Enables fetching VF stats via representor interface >>>>- Patch 7: Adds support to sync link state between representors and VFs . >>>>- Patch 8: Enables configuring VF MTU via representor netdevs. >>>>- Patch 9: Add representors for sdp MAC. >>>>- Patch 10: Add devlink port support. >>>> >>>> >>>>Command to create PF/VF representor >>>>#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF >>>>representors are created for each VF when switch mode is set >>>>switchdev on representor PCI device >>>> >>>>#devlink dev >>>>pci/0002:01:00.0 >>>>pci/0002:02:00.0 >>>>pci/0002:1c:00.0 >>> >>>What are these 3 instances representing? How many PFs do you have? 3? >>>How many physical ports you have? >>The test setup has 3 PFs one for each physical port. >> >>Below example is for the device pci/0002:1c:00.0. > >3 port nic, that sounds odd. Is this something shipped already? Care to paste a >link? > https://www.marvell.com/content/dam/marvell/en/public-collateral/embedded-processors/marvell-infrastructure-processors-octeon-tx2-cn913x-product-brief.pdf Please refer to page2, Key features -> Ethernet Ports. Based on the silicon version number of physical ports varies. > >>> >>> >>>> >>>>#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev >>>> >>>># ip link show >>>> eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>>mode >>>>DEFAULT group default qlen 1000 link/ether 7e:58:2d:b6:97:51 brd >>>>ff:ff:ff:ff:ff:ff >>> >>>What is this eth0? Why isn't it connected to any devlink port? >>> >>>> r0p1v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>>mode DEFAULT group default qlen 1000 link/ether 7e:5a:66:ea:fe:d6 brd >>>ff:ff:ff:ff:ff:ff >>>> r1p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>>mode DEFAULT group default qlen 1000 link/ether de:29:be:10:9e:bf brd >>>ff:ff:ff:ff:ff:ff >>>> r2p1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>>mode DEFAULT group default qlen 1000 link/ether 4a:12:c7:a2:66:ad brd >>>ff:ff:ff:ff:ff:ff >>>> r3p1v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >>>mode >>>>DEFAULT group default qlen 1000 link/ether c2:b8:a8:0e:73:fd brd >>>>ff:ff:ff:ff:ff:ff >>>> >>>> >>>>~# devlink port >>>>pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 >>>>pfnum 1 vfnum 0 external false splittable false >>>>pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 >>>>pfnum 1 vfnum 1 external false splittable false >>>>pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 >>>>pfnum 1 vfnum 2 external false splittable false >>>>pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 >>>>pfnum 1 vfnum 3 external false splittable false >>> >>>You are missing physical port devlink instance here? Where is it? >>pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 >>pfnum 1 vfnum 0 external false splittable false This is for the PF. > >That is not a physical port. The flavour should be "physical" for them. I was using PCI_PF flavour. Will fix it in the next version. > > >> >>Below is the example on a setup with one PF before 3VFs are created. >> >>#devlink dev eswitch set pci/0002:1c:00.0 mode switchdev >> >># ip link show >> eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode >>DEFAULT group default qlen 1000 link/ether 7e:58:2d:b6:97:51 brd >>ff:ff:ff:ff:ff:ff >>r0p1v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode >>DEFAULT group default qlen 1000 link/ether 7e:5a:66:ea:fe:d6 brd >>ff:ff:ff:ff:ff:ff >>r1p1v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode >>DEFAULT group default qlen 1000 link/ether de:29:be:10:9e:bf brd >>ff:ff:ff:ff:ff:ff >>r2p1v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode >>DEFAULT group default qlen 1000 link/ether 4a:12:c7:a2:66:ad brd >>ff:ff:ff:ff:ff:ff >>r3p1v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode > >How you setup the netdevice names? Do you have custom udev rule? Would >be great to give here the out-of-box udev names instead. Ok will fix it in the next version. > > > >>DEFAULT group default qlen 1000 link/ether c2:b8:a8:0e:73:fd brd >>ff:ff:ff:ff:ff:ff >> >>Above shows the PF physical port and 4 representors(1 for PF and 3 for VFs). >># devlink port >>pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 >>pfnum 1 vfnum 0 external false splittable false >>pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 >>pfnum 1 vfnum 1 external false splittable false >>pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 >>pfnum 1 vfnum 2 external false splittable false >>pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 >>pfnum 1 vfnum 3 external false splittable false >> >>> >>>> >>>> >>>>----------- >>>>v1-v2: >>>> -Fixed build warnings. >>>> -Address review comments provided by "Kalesh Anakkur Purayil". >>>> >>>>v2-v3: >>>> - Used extack for error messages. >>>> - As suggested reworked commit messages. >>>> - Fixed sparse warning. >>>> >>>>v3-v4: >>>> - Patch 2 & 3: Fixed coccinelle reported warnings. >>>> - Patch 10: Added devlink port support. >>> >>> >>>When someone reviews your patchset at some version, you put him to cc >>>list from that point. Why didn't you put me to cc list? >>> >>> >>>> >>>>v4-v5: >>>> - Patch 3: Removed devm_* usage in rvu_rep_create() >>>> - Patch 3: Fixed build warnings. >>>> >>>>v5-v6: >>>> - Addressed review comments provided by "Simon Horman". >>>> - Added review tag. >>>> >>>>v6-v7: >>>> - Rebased on top net-next branch. >>>> >>>>v7-v8: >>>> - Implmented offload stats ndo. >>>> - Added documentation. >>>> >>>>v8-v9: >>>> - Updated the documentation. >>>> >>>>v9-v10: >>>> - Fixed build warning w.r.t documentation. >>>> >>>>Geetha sowjanya (11): >>>> octeontx2-pf: Refactoring RVU driver >>>> octeontx2-pf: RVU representor driver >>>> octeontx2-pf: Create representor netdev >>>> octeontx2-pf: Add basic net_device_ops >>>> octeontx2-af: Add packet path between representor and VF >>>> octeontx2-pf: Get VF stats via representor >>>> octeontx2-pf: Add support to sync link state between representor and >>>> VFs >>>> octeontx2-pf: Configure VF mtu via representor >>>> octeontx2-pf: Add representors for sdp MAC >>>> octeontx2-pf: Add devlink port support >>>> octeontx2-pf: Implement offload stats ndo for representors >>>> >>>> .../ethernet/marvell/octeontx2.rst | 85 ++ >>>> .../net/ethernet/marvell/octeontx2/Kconfig | 8 + >>>> .../ethernet/marvell/octeontx2/af/Makefile | 3 +- >>>> .../ethernet/marvell/octeontx2/af/common.h | 2 + >>>> .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ >>>> .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + >>>> .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + >>>> .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- >>>> .../marvell/octeontx2/af/rvu_debugfs.c | 27 - >>>> .../marvell/octeontx2/af/rvu_devlink.c | 6 + >>>> .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 +- >>>> .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + >>>> .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + >>>> .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 +++++++++++ >>>> .../marvell/octeontx2/af/rvu_struct.h | 26 + >>>> .../marvell/octeontx2/af/rvu_switch.c | 20 +- >>>> .../ethernet/marvell/octeontx2/nic/Makefile | 2 + >>>> .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- >>>> .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- >>>> .../marvell/octeontx2/nic/otx2_common.c | 58 +- >>>> .../marvell/octeontx2/nic/otx2_common.h | 84 +- >>>> .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ >>>> .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- >>>> .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- >>>> .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- >>>> .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- >>>> .../net/ethernet/marvell/octeontx2/nic/rep.c | 725 >>>> ++++++++++++++++++ .../net/ethernet/marvell/octeontx2/nic/rep.h | >>>> 53 ++ >>>> 28 files changed, 1962 insertions(+), 227 deletions(-) create mode >>>> 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c >>>> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.c >>>> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.h >>>> >>>>-- >>>>2.25.1 >>>> >>>>