diff mbox series

[V7,5/9] PCI/IOV: Enable 10-Bit tag support for PCIe VF devices

Message ID 1628084828-119542-6-git-send-email-liudongdong3@huawei.com (mailing list archive)
State Superseded
Delegated to: Bjorn Helgaas
Headers show
Series PCI: Enable 10-Bit tag support for PCIe devices | expand

Commit Message

Dongdong Liu Aug. 4, 2021, 1:47 p.m. UTC 
Enable VF 10-Bit Tag Requester when it's upstream component support
10-bit Tag Completer.

Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 drivers/pci/iov.c | 8 ++++++++
 1 file changed, 8 insertions(+)

Comments

Bjorn Helgaas Aug. 4, 2021, 11:29 p.m. UTC | #1 
On Wed, Aug 04, 2021 at 09:47:04PM +0800, Dongdong Liu wrote:
> Enable VF 10-Bit Tag Requester when it's upstream component support
> 10-bit Tag Completer.

s/it's/its/
s/support/supports/

I think "upstream component" here means the PF, doesn't it?  I don't
think the PF is really an *upstream* component; there's no routing
like with a switch.

> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> ---
>  drivers/pci/iov.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index dafdc65..0d0bed1 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -634,6 +634,10 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>  
>  	pci_iov_set_numvfs(dev, nr_virtfn);
>  	iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
> +	if ((iov->cap & PCI_SRIOV_CAP_VF_10BIT_TAG_REQ) &&
> +	    dev->ext_10bit_tag)
> +		iov->ctrl |= PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;
> +
>  	pci_cfg_access_lock(dev);
>  	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>  	msleep(100);
> @@ -650,6 +654,8 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>  
>  err_pcibios:
>  	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
> +	if (iov->ctrl & PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN)
> +		iov->ctrl &= ~PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;
>  	pci_cfg_access_lock(dev);
>  	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>  	ssleep(1);
> @@ -682,6 +688,8 @@ static void sriov_disable(struct pci_dev *dev)
>  
>  	sriov_del_vfs(dev);
>  	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
> +	if (iov->ctrl & PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN)
> +		iov->ctrl &= ~PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;

You can just clear PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN unconditionally,
can't you?  I know it wouldn't change anything, but removing the "if"
makes the code prettier.  You could just add it in the existing
PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE mask.

>  	pci_cfg_access_lock(dev);
>  	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>  	ssleep(1);
> -- 
> 2.7.4
>
Dongdong Liu Aug. 5, 2021, 8:03 a.m. UTC | #2 
On 2021/8/5 7:29, Bjorn Helgaas wrote:
> On Wed, Aug 04, 2021 at 09:47:04PM +0800, Dongdong Liu wrote:
>> Enable VF 10-Bit Tag Requester when it's upstream component support
>> 10-bit Tag Completer.
>
> s/it's/its/
> s/support/supports/
Will fix.
>
> I think "upstream component" here means the PF, doesn't it?  I don't
> think the PF is really an *upstream* component; there's no routing
> like with a switch.
I want to say the switch and root port devices that support 10-Bit
Tag Completer. Sure, VF also needs to have 10-bit Tag Requester
Supported capability.
>
>> Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>> ---
>>  drivers/pci/iov.c | 8 ++++++++
>>  1 file changed, 8 insertions(+)
>>
>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>> index dafdc65..0d0bed1 100644
>> --- a/drivers/pci/iov.c
>> +++ b/drivers/pci/iov.c
>> @@ -634,6 +634,10 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>>
>>  	pci_iov_set_numvfs(dev, nr_virtfn);
>>  	iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
>> +	if ((iov->cap & PCI_SRIOV_CAP_VF_10BIT_TAG_REQ) &&
>> +	    dev->ext_10bit_tag)
>> +		iov->ctrl |= PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;
>> +
>>  	pci_cfg_access_lock(dev);
>>  	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>>  	msleep(100);
>> @@ -650,6 +654,8 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>>
>>  err_pcibios:
>>  	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
>> +	if (iov->ctrl & PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN)
>> +		iov->ctrl &= ~PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;
>>  	pci_cfg_access_lock(dev);
>>  	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>>  	ssleep(1);
>> @@ -682,6 +688,8 @@ static void sriov_disable(struct pci_dev *dev)
>>
>>  	sriov_del_vfs(dev);
>>  	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
>> +	if (iov->ctrl & PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN)
>> +		iov->ctrl &= ~PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;
>
> You can just clear PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN unconditionally,
> can't you?  I know it wouldn't change anything, but removing the "if"
> makes the code prettier.  You could just add it in the existing
> PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE mask.
Will do.

Thanks,
Dongdong
>
>>  	pci_cfg_access_lock(dev);
>>  	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>>  	ssleep(1);
>> --
>> 2.7.4
>>
> .
>
Bjorn Helgaas Aug. 6, 2021, 10:59 p.m. UTC | #3 
On Thu, Aug 05, 2021 at 04:03:58PM +0800, Dongdong Liu wrote:
> 
> On 2021/8/5 7:29, Bjorn Helgaas wrote:
> > On Wed, Aug 04, 2021 at 09:47:04PM +0800, Dongdong Liu wrote:
> > > Enable VF 10-Bit Tag Requester when it's upstream component support
> > > 10-bit Tag Completer.
> > 
> > I think "upstream component" here means the PF, doesn't it?  I don't
> > think the PF is really an *upstream* component; there's no routing
> > like with a switch.
>
> I want to say the switch and root port devices that support 10-Bit
> Tag Completer. Sure, VF also needs to have 10-bit Tag Requester
> Supported capability.

OK.  IIUC we're not talking about P2PDMA here; we're talking about
regular DMA to host memory, which means I *think* only the Root Port
is important, since it is the completer for DMA to host memory.  We're
not talking about P2PDMA to a switch BAR, where the switch would be
the completer.
Dongdong Liu Aug. 7, 2021, 7:46 a.m. UTC | #4 
On 2021/8/7 6:59, Bjorn Helgaas wrote:
> On Thu, Aug 05, 2021 at 04:03:58PM +0800, Dongdong Liu wrote:
>>
>> On 2021/8/5 7:29, Bjorn Helgaas wrote:
>>> On Wed, Aug 04, 2021 at 09:47:04PM +0800, Dongdong Liu wrote:
>>>> Enable VF 10-Bit Tag Requester when it's upstream component support
>>>> 10-bit Tag Completer.
>>>
>>> I think "upstream component" here means the PF, doesn't it?  I don't
>>> think the PF is really an *upstream* component; there's no routing
>>> like with a switch.
>>
>> I want to say the switch and root port devices that support 10-Bit
>> Tag Completer. Sure, VF also needs to have 10-bit Tag Requester
>> Supported capability.
>
> OK.  IIUC we're not talking about P2PDMA here; we're talking about
> regular DMA to host memory, which means I *think* only the Root Port
> is important, since it is the completer for DMA to host memory.  We're
> not talking about P2PDMA to a switch BAR, where the switch would be
> the completer.

Yes, only the Root Port is important, this is also PCIe spec
recommended.

Only when switch detects an error with an NPR carrying a 10-Bit Tag,
and that Switch handles the error by acting as the Completer for the
NPR, the resulting Completion will have an invalid 10-Bit Tag.

Enable 10-bit for EP devices depend on the hierarchy(include switch)
supports 10-bit tags in "[PATCH V7 4/9] PCI: Enable 10-Bit Tag support
for PCIe Endpoint devices". This seems complex.
I will fix this. Enable 10-bit tag for EP devices only depend on Root
Port 10-bit tag completer about regular DMA to host memory.

The below is the PCIe spec describe:
PCIe spec 5.0 r1.0 section 2.2.6.2 "Considerations for Implementing
10-Bit Tag Capabilities" Implementation Note.

   For platforms where the RC supports 10-Bit Tag Completer capability,
   it is highly recommended for platform firmware or operating software
   that configures PCIe hierarchies to Set the 10-Bit Tag Requester
   Enable bit automatically in Endpoints with 10-Bit Tag Requester
   capability. This enables the important class of 10-Bit Tag capable
   adapters that send Memory Read Requests only to host memory.

...
   Switches that lack 10-Bit Tag Completer capability are still able to
   forward NPRs and Completions carrying 10-Bit Tags correctly, since the
   two new Tag bits are in TLP Header bits that were formerly Reserved,
   and Switches are required to forward Reserved TLP Header bits without
   modification. However, if such a Switch detects an error with an NPR
   carrying a 10-Bit Tag, and that Switch handles the error by acting as
   the Completer for the NPR, the resulting Completion will have an
   invalid 10-Bit Tag. Thus, it is strongly recommended that Switches
   between any components using 10-Bit Tags support 10-Bit Tag Completer
   capability.  Note that Switches supporting 16.0 GT/s data rates or
   greater must support 10-Bit Tag Completer capability.

Thanks,
Dongdong

> .
>
diff mbox series

Patch

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index dafdc65..0d0bed1 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -634,6 +634,10 @@  static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
 
 	pci_iov_set_numvfs(dev, nr_virtfn);
 	iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
+	if ((iov->cap & PCI_SRIOV_CAP_VF_10BIT_TAG_REQ) &&
+	    dev->ext_10bit_tag)
+		iov->ctrl |= PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;
+
 	pci_cfg_access_lock(dev);
 	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
 	msleep(100);
@@ -650,6 +654,8 @@  static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
 
 err_pcibios:
 	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
+	if (iov->ctrl & PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN)
+		iov->ctrl &= ~PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;
 	pci_cfg_access_lock(dev);
 	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
 	ssleep(1);
@@ -682,6 +688,8 @@  static void sriov_disable(struct pci_dev *dev)
 
 	sriov_del_vfs(dev);
 	iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
+	if (iov->ctrl & PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN)
+		iov->ctrl &= ~PCI_SRIOV_CTRL_VF_10BIT_TAG_REQ_EN;
 	pci_cfg_access_lock(dev);
 	pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
 	ssleep(1);