| Message ID | 1628084828-119542-8-git-send-email-liudongdong3@huawei.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | PCI: Enable 10-Bit tag support for PCIe devices | expand |
On 2021-08-04 7:47 a.m., Dongdong Liu wrote: > PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports > sending Requests to other Endpoints (as opposed to host memory), the > Endpoint must not send 10-Bit Tag Requests to another given Endpoint > unless an implementation-specific mechanism determines that the Endpoint > supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, > write 0 to disable 10-Bit Tag Requester when the driver does not bind > the device if the peer device does not support the 10-Bit Tag Completer. > This will make P2P traffic safe. the 10bit_tag file content indicate > current 10-Bit Tag Requester Enable status. Can we not have both the sysfs file and the command line parameter? If the user wants to disable it always for a specific device this sysfs parameter is fairly awkward. A script at boot to unbind the driver, set the sysfs file and rebind the driver is not trivial and the command line parameter offers additional options for users. Logan
On Wed, Aug 04, 2021 at 09:47:06PM +0800, Dongdong Liu wrote: > PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports > sending Requests to other Endpoints (as opposed to host memory), the > Endpoint must not send 10-Bit Tag Requests to another given Endpoint > unless an implementation-specific mechanism determines that the Endpoint > supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, > write 0 to disable 10-Bit Tag Requester when the driver does not bind > the device if the peer device does not support the 10-Bit Tag Completer. > This will make P2P traffic safe. the 10bit_tag file content indicate > current 10-Bit Tag Requester Enable status. > > Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> > --- > Documentation/ABI/testing/sysfs-bus-pci | 16 +++++++- > drivers/pci/pci-sysfs.c | 69 +++++++++++++++++++++++++++++++++ > 2 files changed, 84 insertions(+), 1 deletion(-) > > diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci > index 793cbb7..0e0c97d 100644 > --- a/Documentation/ABI/testing/sysfs-bus-pci > +++ b/Documentation/ABI/testing/sysfs-bus-pci > @@ -139,7 +139,7 @@ Description: > binary file containing the Vital Product Data for the > device. It should follow the VPD format defined in > PCI Specification 2.1 or 2.2, but users should consider > - that some devices may have incorrectly formatted data. > + that some devices may have incorrectly formatted data. > If the underlying VPD has a writable section then the > corresponding section of this file will be writable. > > @@ -407,3 +407,17 @@ Description: > > The file is writable if the PF is bound to a driver that > implements ->sriov_set_msix_vec_count(). > + > +What: /sys/bus/pci/devices/.../10bit_tag > +Date: August 2021 > +Contact: Dongdong Liu <liudongdong3@huawei.com> > +Description: > + If a PCI device support 10-Bit Tag Requester, will create the > + 10bit_tag sysfs file. The file is readable, the value > + indicate current 10-Bit Tag Requester Enable. > + 1 - enabled, 0 - disabled. > + > + The file is also writeable, the value only accept by write 0 > + to disable 10-Bit Tag Requester when the driver does not bind > + the deivce. The typical use case is for p2pdma when the peer > + device does not support 10-BIT Tag Completer. s/writeable/writable/ s/deivce/device/ The first sentence does not parse. > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c > index 5d63df7..e93ce8b 100644 > --- a/drivers/pci/pci-sysfs.c > +++ b/drivers/pci/pci-sysfs.c > @@ -306,6 +306,49 @@ static ssize_t enable_show(struct device *dev, struct device_attribute *attr, > } > static DEVICE_ATTR_RW(enable); > > +static ssize_t pci_10bit_tag_store(struct device *dev, > + struct device_attribute *attr, > + const char *buf, size_t count) > +{ > + struct pci_dev *pdev = to_pci_dev(dev); > + bool enable; > + > + if (kstrtobool(buf, &enable) < 0) > + return -EINVAL; > + > + if (enable != false ) > + return -EINVAL; Is this the same as "if (enable)"? > + if (pdev->driver) > + return -EBUSY; > + > + pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL2, > + PCI_EXP_DEVCTL2_10BIT_TAG_REQ_EN); > + pci_info(pdev, "disabled 10-Bit Tag Requester\n"); > + > + return count; > +} > + > +static ssize_t pci_10bit_tag_show(struct device *dev, > + struct device_attribute *attr, > + char *buf) > +{ > + struct pci_dev *pdev = to_pci_dev(dev); > + u16 ctl; > + int ret; > + > + ret = pcie_capability_read_word(pdev, PCI_EXP_DEVCTL2, &ctl); > + if (ret) > + return -EINVAL; > + > + return sysfs_emit(buf, "%u\n", > + !!(ctl & PCI_EXP_DEVCTL2_10BIT_TAG_REQ_EN)); > +} > + > +static struct device_attribute dev_attr_10bit_tag = __ATTR(10bit_tag, 0600, > + pci_10bit_tag_show, > + pci_10bit_tag_store); Is this DEVICE_ATTR_ADMIN_RW()? Why is it 0600? Everything else in this file looks like DEVICE_ATTR_RO or DEVICE_ATTR_RW. This should be the same unless there's a reason to be different. > #ifdef CONFIG_NUMA > static ssize_t numa_node_store(struct device *dev, > struct device_attribute *attr, const char *buf, > @@ -635,6 +678,11 @@ static struct attribute *pcie_dev_attrs[] = { > NULL, > }; > > +static struct attribute *pcie_dev_10bit_tag_attrs[] = { > + &dev_attr_10bit_tag.attr, > + NULL, > +}; > + > static struct attribute *pcibus_attrs[] = { > &dev_attr_bus_rescan.attr, > &dev_attr_cpuaffinity.attr, > @@ -1482,6 +1530,21 @@ static umode_t pcie_dev_attrs_are_visible(struct kobject *kobj, > return 0; > } > > +static umode_t pcie_dev_10bit_tag_attrs_are_visible(struct kobject *kobj, > + struct attribute *a, int n) > +{ > + struct device *dev = kobj_to_dev(kobj); > + struct pci_dev *pdev = to_pci_dev(dev); > + > + if (pdev->is_virtfn) > + return 0; > + > + if (!(pdev->pcie_devcap2 & PCI_EXP_DEVCAP2_10BIT_TAG_REQ)) > + return 0; > + > + return a->mode; > +} > + > static const struct attribute_group pci_dev_group = { > .attrs = pci_dev_attrs, > }; > @@ -1521,6 +1584,11 @@ static const struct attribute_group pcie_dev_attr_group = { > .is_visible = pcie_dev_attrs_are_visible, > }; > > +static const struct attribute_group pcie_dev_10bit_tag_attr_group = { > + .attrs = pcie_dev_10bit_tag_attrs, > + .is_visible = pcie_dev_10bit_tag_attrs_are_visible, > +}; > + > static const struct attribute_group *pci_dev_attr_groups[] = { > &pci_dev_attr_group, > &pci_dev_hp_attr_group, > @@ -1530,6 +1598,7 @@ static const struct attribute_group *pci_dev_attr_groups[] = { > #endif > &pci_bridge_attr_group, > &pcie_dev_attr_group, > + &pcie_dev_10bit_tag_attr_group, > #ifdef CONFIG_PCIEAER > &aer_stats_attr_group, > #endif > -- > 2.7.4 >
On Wed, Aug 04, 2021 at 09:47:06PM +0800, Dongdong Liu wrote: > PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports > sending Requests to other Endpoints (as opposed to host memory), the > Endpoint must not send 10-Bit Tag Requests to another given Endpoint > unless an implementation-specific mechanism determines that the Endpoint > supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, > write 0 to disable 10-Bit Tag Requester when the driver does not bind > the device if the peer device does not support the 10-Bit Tag Completer. > This will make P2P traffic safe. the 10bit_tag file content indicate > current 10-Bit Tag Requester Enable status. > > Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> > + The file is also writeable, the value only accept by write 0 > + to disable 10-Bit Tag Requester when the driver does not bind > + the deivce. The typical use case is for p2pdma when the peer > + device does not support 10-BIT Tag Completer. s/10-BIT/10-Bit/
On 2021/8/5 7:49, Bjorn Helgaas wrote: > On Wed, Aug 04, 2021 at 09:47:06PM +0800, Dongdong Liu wrote: >> PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports >> sending Requests to other Endpoints (as opposed to host memory), the >> Endpoint must not send 10-Bit Tag Requests to another given Endpoint >> unless an implementation-specific mechanism determines that the Endpoint >> supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, >> write 0 to disable 10-Bit Tag Requester when the driver does not bind >> the device if the peer device does not support the 10-Bit Tag Completer. >> This will make P2P traffic safe. the 10bit_tag file content indicate >> current 10-Bit Tag Requester Enable status. >> >> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> >> --- >> Documentation/ABI/testing/sysfs-bus-pci | 16 +++++++- >> drivers/pci/pci-sysfs.c | 69 +++++++++++++++++++++++++++++++++ >> 2 files changed, 84 insertions(+), 1 deletion(-) >> >> diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci >> index 793cbb7..0e0c97d 100644 >> --- a/Documentation/ABI/testing/sysfs-bus-pci >> +++ b/Documentation/ABI/testing/sysfs-bus-pci >> @@ -139,7 +139,7 @@ Description: >> binary file containing the Vital Product Data for the >> device. It should follow the VPD format defined in >> PCI Specification 2.1 or 2.2, but users should consider >> - that some devices may have incorrectly formatted data. >> + that some devices may have incorrectly formatted data. >> If the underlying VPD has a writable section then the >> corresponding section of this file will be writable. >> >> @@ -407,3 +407,17 @@ Description: >> >> The file is writable if the PF is bound to a driver that >> implements ->sriov_set_msix_vec_count(). >> + >> +What: /sys/bus/pci/devices/.../10bit_tag >> +Date: August 2021 >> +Contact: Dongdong Liu <liudongdong3@huawei.com> >> +Description: >> + If a PCI device support 10-Bit Tag Requester, will create the >> + 10bit_tag sysfs file. The file is readable, the value >> + indicate current 10-Bit Tag Requester Enable. >> + 1 - enabled, 0 - disabled. >> + >> + The file is also writeable, the value only accept by write 0 >> + to disable 10-Bit Tag Requester when the driver does not bind >> + the deivce. The typical use case is for p2pdma when the peer >> + device does not support 10-BIT Tag Completer. > > s/writeable/writable/ > s/deivce/device/ Will fix. > > The first sentence does not parse. 10bit_tag sysfs file will be visible only when the device have 10-Bit Tag Requester Supported capability. Will fix. > >> diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c >> index 5d63df7..e93ce8b 100644 >> --- a/drivers/pci/pci-sysfs.c >> +++ b/drivers/pci/pci-sysfs.c >> @@ -306,6 +306,49 @@ static ssize_t enable_show(struct device *dev, struct device_attribute *attr, >> } >> static DEVICE_ATTR_RW(enable); >> >> +static ssize_t pci_10bit_tag_store(struct device *dev, >> + struct device_attribute *attr, >> + const char *buf, size_t count) >> +{ >> + struct pci_dev *pdev = to_pci_dev(dev); >> + bool enable; >> + >> + if (kstrtobool(buf, &enable) < 0) >> + return -EINVAL; >> + >> + if (enable != false ) >> + return -EINVAL; > > Is this the same as "if (enable)"? Yes, Will fix. > >> + if (pdev->driver) >> + return -EBUSY; >> + >> + pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL2, >> + PCI_EXP_DEVCTL2_10BIT_TAG_REQ_EN); >> + pci_info(pdev, "disabled 10-Bit Tag Requester\n"); >> + >> + return count; >> +} >> + >> +static ssize_t pci_10bit_tag_show(struct device *dev, >> + struct device_attribute *attr, >> + char *buf) >> +{ >> + struct pci_dev *pdev = to_pci_dev(dev); >> + u16 ctl; >> + int ret; >> + >> + ret = pcie_capability_read_word(pdev, PCI_EXP_DEVCTL2, &ctl); >> + if (ret) >> + return -EINVAL; >> + >> + return sysfs_emit(buf, "%u\n", >> + !!(ctl & PCI_EXP_DEVCTL2_10BIT_TAG_REQ_EN)); >> +} >> + >> +static struct device_attribute dev_attr_10bit_tag = __ATTR(10bit_tag, 0600, >> + pci_10bit_tag_show, >> + pci_10bit_tag_store); > > Is this DEVICE_ATTR_ADMIN_RW()? > > Why is it 0600? Everything else in this file looks like > DEVICE_ATTR_RO or DEVICE_ATTR_RW. This should be the same unless > there's a reason to be different. Should be 0644, I do not use DEVICE_ATTR_RW, just want to use "10bit_tag" file name. Will fix. Thanks, Dongdong > >> #ifdef CONFIG_NUMA >> static ssize_t numa_node_store(struct device *dev, >> struct device_attribute *attr, const char *buf, >> @@ -635,6 +678,11 @@ static struct attribute *pcie_dev_attrs[] = { >> NULL, >> }; >> >> +static struct attribute *pcie_dev_10bit_tag_attrs[] = { >> + &dev_attr_10bit_tag.attr, >> + NULL, >> +}; >> + >> static struct attribute *pcibus_attrs[] = { >> &dev_attr_bus_rescan.attr, >> &dev_attr_cpuaffinity.attr, >> @@ -1482,6 +1530,21 @@ static umode_t pcie_dev_attrs_are_visible(struct kobject *kobj, >> return 0; >> } >> >> +static umode_t pcie_dev_10bit_tag_attrs_are_visible(struct kobject *kobj, >> + struct attribute *a, int n) >> +{ >> + struct device *dev = kobj_to_dev(kobj); >> + struct pci_dev *pdev = to_pci_dev(dev); >> + >> + if (pdev->is_virtfn) >> + return 0; >> + >> + if (!(pdev->pcie_devcap2 & PCI_EXP_DEVCAP2_10BIT_TAG_REQ)) >> + return 0; >> + >> + return a->mode; >> +} >> + >> static const struct attribute_group pci_dev_group = { >> .attrs = pci_dev_attrs, >> }; >> @@ -1521,6 +1584,11 @@ static const struct attribute_group pcie_dev_attr_group = { >> .is_visible = pcie_dev_attrs_are_visible, >> }; >> >> +static const struct attribute_group pcie_dev_10bit_tag_attr_group = { >> + .attrs = pcie_dev_10bit_tag_attrs, >> + .is_visible = pcie_dev_10bit_tag_attrs_are_visible, >> +}; >> + >> static const struct attribute_group *pci_dev_attr_groups[] = { >> &pci_dev_attr_group, >> &pci_dev_hp_attr_group, >> @@ -1530,6 +1598,7 @@ static const struct attribute_group *pci_dev_attr_groups[] = { >> #endif >> &pci_bridge_attr_group, >> &pcie_dev_attr_group, >> + &pcie_dev_10bit_tag_attr_group, >> #ifdef CONFIG_PCIEAER >> &aer_stats_attr_group, >> #endif >> -- >> 2.7.4 >> > . >
On 2021/8/5 7:52, Bjorn Helgaas wrote: > On Wed, Aug 04, 2021 at 09:47:06PM +0800, Dongdong Liu wrote: >> PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports >> sending Requests to other Endpoints (as opposed to host memory), the >> Endpoint must not send 10-Bit Tag Requests to another given Endpoint >> unless an implementation-specific mechanism determines that the Endpoint >> supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, >> write 0 to disable 10-Bit Tag Requester when the driver does not bind >> the device if the peer device does not support the 10-Bit Tag Completer. >> This will make P2P traffic safe. the 10bit_tag file content indicate >> current 10-Bit Tag Requester Enable status. >> >> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> > >> + The file is also writeable, the value only accept by write 0 >> + to disable 10-Bit Tag Requester when the driver does not bind >> + the deivce. The typical use case is for p2pdma when the peer >> + device does not support 10-BIT Tag Completer. > > s/10-BIT/10-Bit/ Will fix and will check other place. Thank, Dongdong > . >
On 2021/8/4 23:51, Logan Gunthorpe wrote: > > > > On 2021-08-04 7:47 a.m., Dongdong Liu wrote: >> PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports >> sending Requests to other Endpoints (as opposed to host memory), the >> Endpoint must not send 10-Bit Tag Requests to another given Endpoint >> unless an implementation-specific mechanism determines that the Endpoint >> supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, >> write 0 to disable 10-Bit Tag Requester when the driver does not bind >> the device if the peer device does not support the 10-Bit Tag Completer. >> This will make P2P traffic safe. the 10bit_tag file content indicate >> current 10-Bit Tag Requester Enable status. > > Can we not have both the sysfs file and the command line parameter? If > the user wants to disable it always for a specific device this sysfs > parameter is fairly awkward. A script at boot to unbind the driver, set > the sysfs file and rebind the driver is not trivial and the command line > parameter offers additional options for users. Does the command line parameter as "[PATCH V6 7/8] PCI: Add "pci=disable_10bit_tag=" parameter for peer-to-peer support" does? Do we also need such command line if we already had sysfs file? I think we may not need. Thanks, Dongdong > > Logan > . >
On Thu, Aug 05, 2021 at 09:14:50PM +0800, Dongdong Liu wrote: > On 2021/8/4 23:51, Logan Gunthorpe wrote: > > > > > > > > On 2021-08-04 7:47 a.m., Dongdong Liu wrote: > > > PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports > > > sending Requests to other Endpoints (as opposed to host memory), the > > > Endpoint must not send 10-Bit Tag Requests to another given Endpoint > > > unless an implementation-specific mechanism determines that the Endpoint > > > supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, > > > write 0 to disable 10-Bit Tag Requester when the driver does not bind > > > the device if the peer device does not support the 10-Bit Tag Completer. > > > This will make P2P traffic safe. the 10bit_tag file content indicate > > > current 10-Bit Tag Requester Enable status. > > > > Can we not have both the sysfs file and the command line parameter? If > > the user wants to disable it always for a specific device this sysfs > > parameter is fairly awkward. A script at boot to unbind the driver, set > > the sysfs file and rebind the driver is not trivial and the command line > > parameter offers additional options for users. > Does the command line parameter as "[PATCH V6 7/8] PCI: Add > "pci=disable_10bit_tag=" parameter for peer-to-peer support" does? > > Do we also need such command line if we already had sysfs file? > I think we may not need. I think the same. > > Thanks, > Dongdong > > > > Logan > > . > >
On Thu, Aug 05, 2021 at 04:37:39PM +0800, Dongdong Liu wrote: > > > On 2021/8/5 7:49, Bjorn Helgaas wrote: > > On Wed, Aug 04, 2021 at 09:47:06PM +0800, Dongdong Liu wrote: > > > PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports > > > sending Requests to other Endpoints (as opposed to host memory), the > > > Endpoint must not send 10-Bit Tag Requests to another given Endpoint > > > unless an implementation-specific mechanism determines that the Endpoint > > > supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, > > > write 0 to disable 10-Bit Tag Requester when the driver does not bind > > > the device if the peer device does not support the 10-Bit Tag Completer. > > > This will make P2P traffic safe. the 10bit_tag file content indicate > > > current 10-Bit Tag Requester Enable status. > > > > > > Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> > > > --- > > > Documentation/ABI/testing/sysfs-bus-pci | 16 +++++++- > > > drivers/pci/pci-sysfs.c | 69 +++++++++++++++++++++++++++++++++ > > > 2 files changed, 84 insertions(+), 1 deletion(-) > > > > > > diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci > > > index 793cbb7..0e0c97d 100644 > > > --- a/Documentation/ABI/testing/sysfs-bus-pci > > > +++ b/Documentation/ABI/testing/sysfs-bus-pci > > > @@ -139,7 +139,7 @@ Description: > > > binary file containing the Vital Product Data for the > > > device. It should follow the VPD format defined in > > > PCI Specification 2.1 or 2.2, but users should consider > > > - that some devices may have incorrectly formatted data. > > > + that some devices may have incorrectly formatted data. > > > If the underlying VPD has a writable section then the > > > corresponding section of this file will be writable. > > > > > > @@ -407,3 +407,17 @@ Description: > > > > > > The file is writable if the PF is bound to a driver that > > > implements ->sriov_set_msix_vec_count(). > > > + > > > +What: /sys/bus/pci/devices/.../10bit_tag > > > +Date: August 2021 > > > +Contact: Dongdong Liu <liudongdong3@huawei.com> > > > +Description: > > > + If a PCI device support 10-Bit Tag Requester, will create the > > > + 10bit_tag sysfs file. The file is readable, the value > > > + indicate current 10-Bit Tag Requester Enable. > > > + 1 - enabled, 0 - disabled. > > > + > > > + The file is also writeable, the value only accept by write 0 > > > + to disable 10-Bit Tag Requester when the driver does not bind > > > + the deivce. The typical use case is for p2pdma when the peer > > > + device does not support 10-BIT Tag Completer. > > > +static ssize_t pci_10bit_tag_store(struct device *dev, > > > + struct device_attribute *attr, > > > + const char *buf, size_t count) > > > +{ > > > + struct pci_dev *pdev = to_pci_dev(dev); > > > + bool enable; > > > + > > > + if (kstrtobool(buf, &enable) < 0) > > > + return -EINVAL; > > > + > > > + if (enable != false ) > > > + return -EINVAL; > > > > Is this the same as "if (enable)"? > Yes, Will fix. I actually don't like the one-way nature of this. When the hierarchy supports 10-bit tags, we automatically enable them during enumeration. Then we provide this sysfs file, but it can only *disable* 10-bit tags. There's no way to re-enable them except by rebooting (or using setpci, I guess). Why can't we allow *enabling* them here if they're supported in this hierarchy? > > > + if (pdev->driver) > > > + return -EBUSY; > > > + > > > + pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL2, > > > + PCI_EXP_DEVCTL2_10BIT_TAG_REQ_EN); > > > + pci_info(pdev, "disabled 10-Bit Tag Requester\n"); > > > + > > > + return count; > > > +}
On 2021-08-05 7:14 a.m., Dongdong Liu wrote: > On 2021/8/4 23:51, Logan Gunthorpe wrote: >> >> >> >> On 2021-08-04 7:47 a.m., Dongdong Liu wrote: >>> PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports >>> sending Requests to other Endpoints (as opposed to host memory), the >>> Endpoint must not send 10-Bit Tag Requests to another given Endpoint >>> unless an implementation-specific mechanism determines that the Endpoint >>> supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, >>> write 0 to disable 10-Bit Tag Requester when the driver does not bind >>> the device if the peer device does not support the 10-Bit Tag Completer. >>> This will make P2P traffic safe. the 10bit_tag file content indicate >>> current 10-Bit Tag Requester Enable status. >> >> Can we not have both the sysfs file and the command line parameter? If >> the user wants to disable it always for a specific device this sysfs >> parameter is fairly awkward. A script at boot to unbind the driver, set >> the sysfs file and rebind the driver is not trivial and the command line >> parameter offers additional options for users. > Does the command line parameter as "[PATCH V6 7/8] PCI: Add > "pci=disable_10bit_tag=" parameter for peer-to-peer support" does? > > Do we also need such command line if we already had sysfs file? > I think we may not need. In my opinion, for reasons stated above, the command line parameter is way more convenient. Logan
On 2021/8/5 23:31, Bjorn Helgaas wrote: > On Thu, Aug 05, 2021 at 04:37:39PM +0800, Dongdong Liu wrote: >> >> >> On 2021/8/5 7:49, Bjorn Helgaas wrote: >>> On Wed, Aug 04, 2021 at 09:47:06PM +0800, Dongdong Liu wrote: >>>> PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports >>>> sending Requests to other Endpoints (as opposed to host memory), the >>>> Endpoint must not send 10-Bit Tag Requests to another given Endpoint >>>> unless an implementation-specific mechanism determines that the Endpoint >>>> supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, >>>> write 0 to disable 10-Bit Tag Requester when the driver does not bind >>>> the device if the peer device does not support the 10-Bit Tag Completer. >>>> This will make P2P traffic safe. the 10bit_tag file content indicate >>>> current 10-Bit Tag Requester Enable status. >>>> >>>> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> >>>> --- >>>> Documentation/ABI/testing/sysfs-bus-pci | 16 +++++++- >>>> drivers/pci/pci-sysfs.c | 69 +++++++++++++++++++++++++++++++++ >>>> 2 files changed, 84 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci >>>> index 793cbb7..0e0c97d 100644 >>>> --- a/Documentation/ABI/testing/sysfs-bus-pci >>>> +++ b/Documentation/ABI/testing/sysfs-bus-pci >>>> @@ -139,7 +139,7 @@ Description: >>>> binary file containing the Vital Product Data for the >>>> device. It should follow the VPD format defined in >>>> PCI Specification 2.1 or 2.2, but users should consider >>>> - that some devices may have incorrectly formatted data. >>>> + that some devices may have incorrectly formatted data. >>>> If the underlying VPD has a writable section then the >>>> corresponding section of this file will be writable. >>>> >>>> @@ -407,3 +407,17 @@ Description: >>>> >>>> The file is writable if the PF is bound to a driver that >>>> implements ->sriov_set_msix_vec_count(). >>>> + >>>> +What: /sys/bus/pci/devices/.../10bit_tag >>>> +Date: August 2021 >>>> +Contact: Dongdong Liu <liudongdong3@huawei.com> >>>> +Description: >>>> + If a PCI device support 10-Bit Tag Requester, will create the >>>> + 10bit_tag sysfs file. The file is readable, the value >>>> + indicate current 10-Bit Tag Requester Enable. >>>> + 1 - enabled, 0 - disabled. >>>> + >>>> + The file is also writeable, the value only accept by write 0 >>>> + to disable 10-Bit Tag Requester when the driver does not bind >>>> + the deivce. The typical use case is for p2pdma when the peer >>>> + device does not support 10-BIT Tag Completer. > >>>> +static ssize_t pci_10bit_tag_store(struct device *dev, >>>> + struct device_attribute *attr, >>>> + const char *buf, size_t count) >>>> +{ >>>> + struct pci_dev *pdev = to_pci_dev(dev); >>>> + bool enable; >>>> + >>>> + if (kstrtobool(buf, &enable) < 0) >>>> + return -EINVAL; >>>> + >>>> + if (enable != false ) >>>> + return -EINVAL; >>> >>> Is this the same as "if (enable)"? >> Yes, Will fix. > > I actually don't like the one-way nature of this. When the hierarchy > supports 10-bit tags, we automatically enable them during enumeration. > > Then we provide this sysfs file, but it can only *disable* 10-bit > tags. There's no way to re-enable them except by rebooting (or using > setpci, I guess). > > Why can't we allow *enabling* them here if they're supported in this > hierarchy? Yes, we can also provide this sysfs to enable 10-bit tag for EP devices when the hierarchy supports 10-bit tags. I do not want to provide sysfs to enable/disable 10-bit tag for RP devices as I can not tell current if the the Function has outstanding Non-Posted Requests, may need to unbind all the EP drivers under the RP, and current seems no scenario need to do this. This will make things more complex. Thanks, Dongdong > >>>> + if (pdev->driver) >>>> + return -EBUSY; >>>> + >>>> + pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL2, >>>> + PCI_EXP_DEVCTL2_10BIT_TAG_REQ_EN); >>>> + pci_info(pdev, "disabled 10-Bit Tag Requester\n"); >>>> + >>>> + return count; >>>> +} > . >
On Sat, Aug 07, 2021 at 03:01:56PM +0800, Dongdong Liu wrote: > > On 2021/8/5 23:31, Bjorn Helgaas wrote: > > On Thu, Aug 05, 2021 at 04:37:39PM +0800, Dongdong Liu wrote: > > > > > > > > > On 2021/8/5 7:49, Bjorn Helgaas wrote: > > > > On Wed, Aug 04, 2021 at 09:47:06PM +0800, Dongdong Liu wrote: > > > > > PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports > > > > > sending Requests to other Endpoints (as opposed to host memory), the > > > > > Endpoint must not send 10-Bit Tag Requests to another given Endpoint > > > > > unless an implementation-specific mechanism determines that the Endpoint > > > > > supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, > > > > > write 0 to disable 10-Bit Tag Requester when the driver does not bind > > > > > the device if the peer device does not support the 10-Bit Tag Completer. > > > > > This will make P2P traffic safe. the 10bit_tag file content indicate > > > > > current 10-Bit Tag Requester Enable status. > > > > > > > > > > Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> > > > > > --- > > > > > Documentation/ABI/testing/sysfs-bus-pci | 16 +++++++- > > > > > drivers/pci/pci-sysfs.c | 69 +++++++++++++++++++++++++++++++++ > > > > > 2 files changed, 84 insertions(+), 1 deletion(-) > > > > > > > > > > diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci > > > > > index 793cbb7..0e0c97d 100644 > > > > > --- a/Documentation/ABI/testing/sysfs-bus-pci > > > > > +++ b/Documentation/ABI/testing/sysfs-bus-pci > > > > > @@ -139,7 +139,7 @@ Description: > > > > > binary file containing the Vital Product Data for the > > > > > device. It should follow the VPD format defined in > > > > > PCI Specification 2.1 or 2.2, but users should consider > > > > > - that some devices may have incorrectly formatted data. > > > > > + that some devices may have incorrectly formatted data. > > > > > If the underlying VPD has a writable section then the > > > > > corresponding section of this file will be writable. > > > > > > > > > > @@ -407,3 +407,17 @@ Description: > > > > > > > > > > The file is writable if the PF is bound to a driver that > > > > > implements ->sriov_set_msix_vec_count(). > > > > > + > > > > > +What: /sys/bus/pci/devices/.../10bit_tag > > > > > +Date: August 2021 > > > > > +Contact: Dongdong Liu <liudongdong3@huawei.com> > > > > > +Description: > > > > > + If a PCI device support 10-Bit Tag Requester, will create the > > > > > + 10bit_tag sysfs file. The file is readable, the value > > > > > + indicate current 10-Bit Tag Requester Enable. > > > > > + 1 - enabled, 0 - disabled. > > > > > + > > > > > + The file is also writeable, the value only accept by write 0 > > > > > + to disable 10-Bit Tag Requester when the driver does not bind > > > > > + the deivce. The typical use case is for p2pdma when the peer > > > > > + device does not support 10-BIT Tag Completer. > > > > > > > +static ssize_t pci_10bit_tag_store(struct device *dev, > > > > > + struct device_attribute *attr, > > > > > + const char *buf, size_t count) > > > > > +{ > > > > > + struct pci_dev *pdev = to_pci_dev(dev); > > > > > + bool enable; > > > > > + > > > > > + if (kstrtobool(buf, &enable) < 0) > > > > > + return -EINVAL; > > > > > + > > > > > + if (enable != false ) > > > > > + return -EINVAL; > > > > > > > > Is this the same as "if (enable)"? > > > Yes, Will fix. > > > > I actually don't like the one-way nature of this. When the hierarchy > > supports 10-bit tags, we automatically enable them during enumeration. > > > > Then we provide this sysfs file, but it can only *disable* 10-bit > > tags. There's no way to re-enable them except by rebooting (or using > > setpci, I guess). > > > > Why can't we allow *enabling* them here if they're supported in this > > hierarchy? > Yes, we can also provide this sysfs to enable 10-bit tag for EP devices > when the hierarchy supports 10-bit tags. > > I do not want to provide sysfs to enable/disable 10-bit tag for RP > devices as I can not tell current if the the Function has outstanding > Non-Posted Requests, may need to unbind all the EP drivers under the > RP, and current seems no scenario need to do this. This will make things > more complex. You mean "no scenario requires disabling 10-bit tags and then re-enabling them"? That may be true, but I'm still hesitant to provide a switch than can only be reversed by rebooting. This is similar to the issue Leon raised that it's not practical to reboot machines. Maybe we accept a one-way switch if the sole purpose is to work around a hardware defect. Or maybe a kernel parameter that disables 10-bit tags completely is the defect mitigation. I think we probably need such a parameter in case a defect prevents us from booting far enough to use the sysfs switch.
On 2021/8/10 1:37, Bjorn Helgaas wrote: > On Sat, Aug 07, 2021 at 03:01:56PM +0800, Dongdong Liu wrote: >> >> On 2021/8/5 23:31, Bjorn Helgaas wrote: >>> On Thu, Aug 05, 2021 at 04:37:39PM +0800, Dongdong Liu wrote: >>>> >>>> >>>> On 2021/8/5 7:49, Bjorn Helgaas wrote: >>>>> On Wed, Aug 04, 2021 at 09:47:06PM +0800, Dongdong Liu wrote: >>>>>> PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports >>>>>> sending Requests to other Endpoints (as opposed to host memory), the >>>>>> Endpoint must not send 10-Bit Tag Requests to another given Endpoint >>>>>> unless an implementation-specific mechanism determines that the Endpoint >>>>>> supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, >>>>>> write 0 to disable 10-Bit Tag Requester when the driver does not bind >>>>>> the device if the peer device does not support the 10-Bit Tag Completer. >>>>>> This will make P2P traffic safe. the 10bit_tag file content indicate >>>>>> current 10-Bit Tag Requester Enable status. >>>>>> >>>>>> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> >>>>>> --- >>>>>> Documentation/ABI/testing/sysfs-bus-pci | 16 +++++++- >>>>>> drivers/pci/pci-sysfs.c | 69 +++++++++++++++++++++++++++++++++ >>>>>> 2 files changed, 84 insertions(+), 1 deletion(-) >>>>>> >>>>>> diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci >>>>>> index 793cbb7..0e0c97d 100644 >>>>>> --- a/Documentation/ABI/testing/sysfs-bus-pci >>>>>> +++ b/Documentation/ABI/testing/sysfs-bus-pci >>>>>> @@ -139,7 +139,7 @@ Description: >>>>>> binary file containing the Vital Product Data for the >>>>>> device. It should follow the VPD format defined in >>>>>> PCI Specification 2.1 or 2.2, but users should consider >>>>>> - that some devices may have incorrectly formatted data. >>>>>> + that some devices may have incorrectly formatted data. >>>>>> If the underlying VPD has a writable section then the >>>>>> corresponding section of this file will be writable. >>>>>> >>>>>> @@ -407,3 +407,17 @@ Description: >>>>>> >>>>>> The file is writable if the PF is bound to a driver that >>>>>> implements ->sriov_set_msix_vec_count(). >>>>>> + >>>>>> +What: /sys/bus/pci/devices/.../10bit_tag >>>>>> +Date: August 2021 >>>>>> +Contact: Dongdong Liu <liudongdong3@huawei.com> >>>>>> +Description: >>>>>> + If a PCI device support 10-Bit Tag Requester, will create the >>>>>> + 10bit_tag sysfs file. The file is readable, the value >>>>>> + indicate current 10-Bit Tag Requester Enable. >>>>>> + 1 - enabled, 0 - disabled. >>>>>> + >>>>>> + The file is also writeable, the value only accept by write 0 >>>>>> + to disable 10-Bit Tag Requester when the driver does not bind >>>>>> + the deivce. The typical use case is for p2pdma when the peer >>>>>> + device does not support 10-BIT Tag Completer. >>> >>>>>> +static ssize_t pci_10bit_tag_store(struct device *dev, >>>>>> + struct device_attribute *attr, >>>>>> + const char *buf, size_t count) >>>>>> +{ >>>>>> + struct pci_dev *pdev = to_pci_dev(dev); >>>>>> + bool enable; >>>>>> + >>>>>> + if (kstrtobool(buf, &enable) < 0) >>>>>> + return -EINVAL; >>>>>> + >>>>>> + if (enable != false ) >>>>>> + return -EINVAL; >>>>> >>>>> Is this the same as "if (enable)"? >>>> Yes, Will fix. >>> >>> I actually don't like the one-way nature of this. When the hierarchy >>> supports 10-bit tags, we automatically enable them during enumeration. >>> >>> Then we provide this sysfs file, but it can only *disable* 10-bit >>> tags. There's no way to re-enable them except by rebooting (or using >>> setpci, I guess). >>> >>> Why can't we allow *enabling* them here if they're supported in this >>> hierarchy? >> Yes, we can also provide this sysfs to enable 10-bit tag for EP devices >> when the hierarchy supports 10-bit tags. >> >> I do not want to provide sysfs to enable/disable 10-bit tag for RP >> devices as I can not tell current if the the Function has outstanding >> Non-Posted Requests, may need to unbind all the EP drivers under the >> RP, and current seems no scenario need to do this. This will make things >> more complex. > > You mean "no scenario requires disabling 10-bit tags and then > re-enabling them"? Just for Root Port devices. > That may be true, but I'm still hesitant to > provide a switch than can only be reversed by rebooting. > > This is similar to the issue Leon raised that it's not practical to > reboot machines. Maybe we accept a one-way switch if the sole purpose > is to work around a hardware defect. Or maybe a kernel parameter that > disables 10-bit tags completely is the defect mitigation. I think we > probably need such a parameter in case a defect prevents us from > booting far enough to use the sysfs switch. Make sense, will provide sysfs to enable and disable 10-bit tag. Thanks, Dongdong > . >
diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci index 793cbb7..0e0c97d 100644 --- a/Documentation/ABI/testing/sysfs-bus-pci +++ b/Documentation/ABI/testing/sysfs-bus-pci @@ -139,7 +139,7 @@ Description: binary file containing the Vital Product Data for the device. It should follow the VPD format defined in PCI Specification 2.1 or 2.2, but users should consider - that some devices may have incorrectly formatted data. + that some devices may have incorrectly formatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. @@ -407,3 +407,17 @@ Description: The file is writable if the PF is bound to a driver that implements ->sriov_set_msix_vec_count(). + +What: /sys/bus/pci/devices/.../10bit_tag +Date: August 2021 +Contact: Dongdong Liu <liudongdong3@huawei.com> +Description: + If a PCI device support 10-Bit Tag Requester, will create the + 10bit_tag sysfs file. The file is readable, the value + indicate current 10-Bit Tag Requester Enable. + 1 - enabled, 0 - disabled. + + The file is also writeable, the value only accept by write 0 + to disable 10-Bit Tag Requester when the driver does not bind + the deivce. The typical use case is for p2pdma when the peer + device does not support 10-BIT Tag Completer. diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c index 5d63df7..e93ce8b 100644 --- a/drivers/pci/pci-sysfs.c +++ b/drivers/pci/pci-sysfs.c @@ -306,6 +306,49 @@ static ssize_t enable_show(struct device *dev, struct device_attribute *attr, } static DEVICE_ATTR_RW(enable); +static ssize_t pci_10bit_tag_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct pci_dev *pdev = to_pci_dev(dev); + bool enable; + + if (kstrtobool(buf, &enable) < 0) + return -EINVAL; + + if (enable != false ) + return -EINVAL; + + if (pdev->driver) + return -EBUSY; + + pcie_capability_clear_word(pdev, PCI_EXP_DEVCTL2, + PCI_EXP_DEVCTL2_10BIT_TAG_REQ_EN); + pci_info(pdev, "disabled 10-Bit Tag Requester\n"); + + return count; +} + +static ssize_t pci_10bit_tag_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct pci_dev *pdev = to_pci_dev(dev); + u16 ctl; + int ret; + + ret = pcie_capability_read_word(pdev, PCI_EXP_DEVCTL2, &ctl); + if (ret) + return -EINVAL; + + return sysfs_emit(buf, "%u\n", + !!(ctl & PCI_EXP_DEVCTL2_10BIT_TAG_REQ_EN)); +} + +static struct device_attribute dev_attr_10bit_tag = __ATTR(10bit_tag, 0600, + pci_10bit_tag_show, + pci_10bit_tag_store); + #ifdef CONFIG_NUMA static ssize_t numa_node_store(struct device *dev, struct device_attribute *attr, const char *buf, @@ -635,6 +678,11 @@ static struct attribute *pcie_dev_attrs[] = { NULL, }; +static struct attribute *pcie_dev_10bit_tag_attrs[] = { + &dev_attr_10bit_tag.attr, + NULL, +}; + static struct attribute *pcibus_attrs[] = { &dev_attr_bus_rescan.attr, &dev_attr_cpuaffinity.attr, @@ -1482,6 +1530,21 @@ static umode_t pcie_dev_attrs_are_visible(struct kobject *kobj, return 0; } +static umode_t pcie_dev_10bit_tag_attrs_are_visible(struct kobject *kobj, + struct attribute *a, int n) +{ + struct device *dev = kobj_to_dev(kobj); + struct pci_dev *pdev = to_pci_dev(dev); + + if (pdev->is_virtfn) + return 0; + + if (!(pdev->pcie_devcap2 & PCI_EXP_DEVCAP2_10BIT_TAG_REQ)) + return 0; + + return a->mode; +} + static const struct attribute_group pci_dev_group = { .attrs = pci_dev_attrs, }; @@ -1521,6 +1584,11 @@ static const struct attribute_group pcie_dev_attr_group = { .is_visible = pcie_dev_attrs_are_visible, }; +static const struct attribute_group pcie_dev_10bit_tag_attr_group = { + .attrs = pcie_dev_10bit_tag_attrs, + .is_visible = pcie_dev_10bit_tag_attrs_are_visible, +}; + static const struct attribute_group *pci_dev_attr_groups[] = { &pci_dev_attr_group, &pci_dev_hp_attr_group, @@ -1530,6 +1598,7 @@ static const struct attribute_group *pci_dev_attr_groups[] = { #endif &pci_bridge_attr_group, &pcie_dev_attr_group, + &pcie_dev_10bit_tag_attr_group, #ifdef CONFIG_PCIEAER &aer_stats_attr_group, #endif
PCIe spec 5.0 r1.0 section 2.2.6.2 says that if an Endpoint supports sending Requests to other Endpoints (as opposed to host memory), the Endpoint must not send 10-Bit Tag Requests to another given Endpoint unless an implementation-specific mechanism determines that the Endpoint supports 10-Bit Tag Completer capability. Add a 10bit_tag sysfs file, write 0 to disable 10-Bit Tag Requester when the driver does not bind the device if the peer device does not support the 10-Bit Tag Completer. This will make P2P traffic safe. the 10bit_tag file content indicate current 10-Bit Tag Requester Enable status. Signed-off-by: Dongdong Liu <liudongdong3@huawei.com> --- Documentation/ABI/testing/sysfs-bus-pci | 16 +++++++- drivers/pci/pci-sysfs.c | 69 +++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 1 deletion(-)