From patchwork Thu May 14 02:32:07 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 23677 Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by demeter.kernel.org (8.14.2/8.14.2) with ESMTP id n4E2W3oG006601 for ; Thu, 14 May 2009 02:32:06 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1762576AbZENCcB (ORCPT ); Wed, 13 May 2009 22:32:01 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1762559AbZENCcB (ORCPT ); Wed, 13 May 2009 22:32:01 -0400 Received: from mga11.intel.com ([192.55.52.93]:51109 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1761896AbZENCby (ORCPT ); Wed, 13 May 2009 22:31:54 -0400 Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP; 13 May 2009 19:26:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.41,192,1241420400"; d="scan'208";a="690256904" Received: from yzhao-otc.sh.intel.com ([10.239.48.165]) by fmsmga001.fm.intel.com with ESMTP; 13 May 2009 19:35:05 -0700 From: Yu Zhao To: dwmw2@infradead.org, jbarnes@virtuousgeek.org Cc: linux-pci@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, Yu Zhao Subject: [PATCH v4 resend 2/6] PCI: handle Virtual Function ATS enabling Date: Thu, 14 May 2009 10:32:07 +0800 Message-Id: <1242268331-1401-3-git-send-email-yu.zhao@intel.com> X-Mailer: git-send-email 1.6.1 In-Reply-To: <1242268331-1401-1-git-send-email-yu.zhao@intel.com> References: <1242268331-1401-1-git-send-email-yu.zhao@intel.com> Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org The SR-IOV spec requires that the Smallest Translation Unit and the Invalidate Queue Depth fields in the Virtual Function ATS capability are hardwired to 0. If a function is a Virtual Function, then and set its Physical Function's STU before enabling the ATS. Signed-off-by: Yu Zhao --- drivers/pci/iov.c | 66 +++++++++++++++++++++++++++++++++++++++++----------- drivers/pci/pci.h | 4 ++- 2 files changed, 55 insertions(+), 15 deletions(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 0a7a1b4..4151404 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -491,10 +491,10 @@ found: if (pdev) iov->dev = pci_dev_get(pdev); - else { + else iov->dev = dev; - mutex_init(&iov->lock); - } + + mutex_init(&iov->lock); dev->sriov = iov; dev->is_physfn = 1; @@ -514,11 +514,11 @@ static void sriov_release(struct pci_dev *dev) { BUG_ON(dev->sriov->nr_virtfn); - if (dev == dev->sriov->dev) - mutex_destroy(&dev->sriov->lock); - else + if (dev != dev->sriov->dev) pci_dev_put(dev->sriov->dev); + mutex_destroy(&dev->sriov->lock); + kfree(dev->sriov); dev->sriov = NULL; } @@ -723,19 +723,40 @@ int pci_enable_ats(struct pci_dev *dev, int ps) int rc; u16 ctrl; - BUG_ON(dev->ats); + BUG_ON(dev->ats && dev->ats->is_enabled); if (ps < PCI_ATS_MIN_STU) return -EINVAL; - rc = ats_alloc_one(dev, ps); - if (rc) - return rc; + if (dev->is_physfn || dev->is_virtfn) { + struct pci_dev *pdev = dev->is_physfn ? dev : dev->physfn; + + mutex_lock(&pdev->sriov->lock); + if (pdev->ats) + rc = pdev->ats->stu == ps ? 0 : -EINVAL; + else + rc = ats_alloc_one(pdev, ps); + + if (!rc) + pdev->ats->ref_cnt++; + mutex_unlock(&pdev->sriov->lock); + if (rc) + return rc; + } + + if (!dev->is_physfn) { + rc = ats_alloc_one(dev, ps); + if (rc) + return rc; + } ctrl = PCI_ATS_CTRL_ENABLE; - ctrl |= PCI_ATS_CTRL_STU(ps - PCI_ATS_MIN_STU); + if (!dev->is_virtfn) + ctrl |= PCI_ATS_CTRL_STU(ps - PCI_ATS_MIN_STU); pci_write_config_word(dev, dev->ats->pos + PCI_ATS_CTRL, ctrl); + dev->ats->is_enabled = 1; + return 0; } @@ -747,13 +768,26 @@ void pci_disable_ats(struct pci_dev *dev) { u16 ctrl; - BUG_ON(!dev->ats); + BUG_ON(!dev->ats || !dev->ats->is_enabled); pci_read_config_word(dev, dev->ats->pos + PCI_ATS_CTRL, &ctrl); ctrl &= ~PCI_ATS_CTRL_ENABLE; pci_write_config_word(dev, dev->ats->pos + PCI_ATS_CTRL, ctrl); - ats_free_one(dev); + dev->ats->is_enabled = 0; + + if (dev->is_physfn || dev->is_virtfn) { + struct pci_dev *pdev = dev->is_physfn ? dev : dev->physfn; + + mutex_lock(&pdev->sriov->lock); + pdev->ats->ref_cnt--; + if (!pdev->ats->ref_cnt) + ats_free_one(pdev); + mutex_unlock(&pdev->sriov->lock); + } + + if (!dev->is_physfn) + ats_free_one(dev); } /** @@ -765,13 +799,17 @@ void pci_disable_ats(struct pci_dev *dev) * The ATS spec uses 0 in the Invalidate Queue Depth field to * indicate that the function can accept 32 Invalidate Request. * But here we use the `real' values (i.e. 1~32) for the Queue - * Depth. + * Depth; and 0 indicates the function shares the Queue with + * other functions (doesn't exclusively own a Queue). */ int pci_ats_queue_depth(struct pci_dev *dev) { int pos; u16 cap; + if (dev->is_virtfn) + return 0; + if (dev->ats) return dev->ats->qdep; diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 3c2ec64..f73bcbe 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -234,6 +234,8 @@ struct pci_ats { int pos; /* capability position */ int stu; /* Smallest Translation Unit */ int qdep; /* Invalidate Queue Depth */ + int ref_cnt; /* Physical Function reference count */ + int is_enabled:1; /* Enable bit is set */ }; #ifdef CONFIG_PCI_IOV @@ -255,7 +257,7 @@ extern int pci_ats_queue_depth(struct pci_dev *dev); */ static inline int pci_ats_enabled(struct pci_dev *dev) { - return !!dev->ats; + return dev->ats && dev->ats->is_enabled; } #else static inline int pci_iov_init(struct pci_dev *dev)