From patchwork Mon Apr 2 22:46:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 10320445 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id F417060116 for ; Mon, 2 Apr 2018 22:47:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA878288B4 for ; Mon, 2 Apr 2018 22:47:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CF0DA288DE; Mon, 2 Apr 2018 22:47:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 230F9288B4 for ; Mon, 2 Apr 2018 22:47:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754556AbeDBWrB (ORCPT ); Mon, 2 Apr 2018 18:47:01 -0400 Received: from mail-pl0-f66.google.com ([209.85.160.66]:37812 "EHLO mail-pl0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754548AbeDBWrA (ORCPT ); Mon, 2 Apr 2018 18:47:00 -0400 Received: by mail-pl0-f66.google.com with SMTP id v5-v6so4139704plo.4 for ; Mon, 02 Apr 2018 15:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id; bh=5aFsDUotAO+jRJPCTP0szVcSePYaLtaDtYTWf8ZGFAk=; b=nCTF6rvfu/WZ+DTdnYxoeZUn4C3/T//WWK/G9Iwihh5zC06mrayi73UGbifXFq18Kp nvQ82inqfGpsKJGllC+UrwpM4NmVhinbOvomYP4sEnYI8h7LgE3Dn5osspOkiU5SBPsI 9IyFEPm2qnUIFWC1UvHtTt0XbJBVt3yKBaqZFXvc0fgPlLdg7PLDLFKdQIcFoYr8n08U EVoDlEk1UhLrZidz9KagBMvoPkPWAqFscVzcnxiggceUFfS8bzQG6D4aOgSGq6mhsJ2o DEVk398VNfB8IR0aGuklSRKJTZ22u1a+AqcV1w8Vy0BN5FQNoH6dMj6nuRNMwb8Yu2+9 YJXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=5aFsDUotAO+jRJPCTP0szVcSePYaLtaDtYTWf8ZGFAk=; b=OgxBg4IbY1EqtAnsFb9AiBDzj9dy7n4Mb0SxYSzNY1ZwX3QviAJb9WqwdqQLoLLI8l gixJjrlvQCddnnfsXSrPUYSFYUwUnHqD6KT510O28XeauSS8EJ4wE4w7wHSOPlhQyWJ+ kFx75zTjXkXWPIcbd/ouFY8uPno2mHakT+IazFC1UO7AQNhws7i74bwQ06Q8l1FxXLdt rB3c7TzDUs8wID9a+QJYvCCouQQ6Dc2JTSK9Zqs8MmEr4yAHsoOPOyHZbDDqg6wx3Vs1 wqLuNf8W5AWxmO4xSosDSbRFPphQFeTw2gHSYQAbxKLZT/B7OtI012XDiEO2O2N69wnl 6/ZA== X-Gm-Message-State: AElRT7F1ld+udtyhq765mYhQ4eGUFgy2hnlz8zNNoFnaQoftySH0mFEQ nViEQ1kO4I9DoxxgfMYAnKXgJQ== X-Google-Smtp-Source: AIpwx49zujv3NPZC2fLHbADB1+3M0+IxADIzRsmBzViq761iPAH0kg8FHe9i16cm0THMYd2bkzWC4Q== X-Received: by 10.99.122.22 with SMTP id v22mr7165561pgc.300.1522709220258; Mon, 02 Apr 2018 15:47:00 -0700 (PDT) Received: from jkicinski-Precision-T1700.netronome.com ([75.53.12.129]) by smtp.gmail.com with ESMTPSA id p1sm2047714pgr.67.2018.04.02.15.46.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 15:46:59 -0700 (PDT) From: Jakub Kicinski To: Bjorn Helgaas Cc: linux-pci@vger.kernel.org, netdev@vger.kernel.org, Sathya Perla , Felix Manlunas , alexander.duyck@gmail.com, john.fastabend@gmail.com, Jacob Keller , Donald Dutile , oss-drivers@netronome.com, Christoph Hellwig , Jakub Kicinski Subject: [PATCH] PCI: allow drivers to limit the number of VFs to 0 Date: Mon, 2 Apr 2018 15:46:52 -0700 Message-Id: <20180402224652.4058-1-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.16.2 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Some user space depends on enabling sriov_totalvfs number of VFs to not fail, e.g.: $ cat .../sriov_totalvfs > .../sriov_numvfs For devices which VF support depends on loaded FW we have the pci_sriov_{g,s}et_totalvfs() API. However, this API uses 0 as a special "unset" value, meaning drivers can't limit sriov_totalvfs to 0. Remove the special values completely and simply initialize driver_max_VFs to total_VFs. Then always use driver_max_VFs. Add a helper for drivers to reset the VF limit back to total. Signed-off-by: Jakub Kicinski --- drivers/net/ethernet/netronome/nfp/nfp_main.c | 6 +++--- drivers/pci/iov.c | 27 +++++++++++++++++++++------ include/linux/pci.h | 2 ++ 3 files changed, 26 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c index c4b1f344b4da..a76d177e40dd 100644 --- a/drivers/net/ethernet/netronome/nfp/nfp_main.c +++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c @@ -123,7 +123,7 @@ static int nfp_pcie_sriov_read_nfd_limit(struct nfp_pf *pf) return pci_sriov_set_totalvfs(pf->pdev, pf->limit_vfs); pf->limit_vfs = ~0; - pci_sriov_set_totalvfs(pf->pdev, 0); /* 0 is unset */ + pci_sriov_reset_totalvfs(pf->pdev); /* Allow any setting for backwards compatibility if symbol not found */ if (err == -ENOENT) return 0; @@ -537,7 +537,7 @@ static int nfp_pci_probe(struct pci_dev *pdev, err_net_remove: nfp_net_pci_remove(pf); err_sriov_unlimit: - pci_sriov_set_totalvfs(pf->pdev, 0); + pci_sriov_reset_totalvfs(pf->pdev); err_fw_unload: kfree(pf->rtbl); nfp_mip_close(pf->mip); @@ -570,7 +570,7 @@ static void nfp_pci_remove(struct pci_dev *pdev) nfp_hwmon_unregister(pf); nfp_pcie_sriov_disable(pdev); - pci_sriov_set_totalvfs(pf->pdev, 0); + pci_sriov_reset_totalvfs(pf->pdev); nfp_net_pci_remove(pf); diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 677924ae0350..c63ea870d8be 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -443,6 +443,7 @@ static int sriov_init(struct pci_dev *dev, int pos) iov->nres = nres; iov->ctrl = ctrl; iov->total_VFs = total; + iov->driver_max_VFs = total; pci_read_config_word(dev, pos + PCI_SRIOV_VF_DID, &iov->vf_device); iov->pgsz = pgsz; iov->self = dev; @@ -788,12 +789,29 @@ int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs) } EXPORT_SYMBOL_GPL(pci_sriov_set_totalvfs); +/** + * pci_sriov_reset_totalvfs -- return the TotalVFs value to the default + * @dev: the PCI PF device + * + * Should be called from PF driver's remove routine with + * device's mutex held. + */ +void pci_sriov_reset_totalvfs(struct pci_dev *dev) +{ + /* Shouldn't change if VFs already enabled */ + if (!dev->is_physfn || dev->sriov->ctrl & PCI_SRIOV_CTRL_VFE) + return; + + dev->sriov->driver_max_VFs = dev->sriov->total_VFs; +} +EXPORT_SYMBOL_GPL(pci_sriov_reset_totalvfs); + /** * pci_sriov_get_totalvfs -- get total VFs supported on this device * @dev: the PCI PF device * - * For a PCIe device with SRIOV support, return the PCIe - * SRIOV capability value of TotalVFs or the value of driver_max_VFs + * For a PCIe device with SRIOV support, return the value of driver_max_VFs + * which can be equal to the PCIe SRIOV capability value of TotalVFs or lower * if the driver reduced it. Otherwise 0. */ int pci_sriov_get_totalvfs(struct pci_dev *dev) @@ -801,9 +819,6 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev) if (!dev->is_physfn) return 0; - if (dev->sriov->driver_max_VFs) - return dev->sriov->driver_max_VFs; - - return dev->sriov->total_VFs; + return dev->sriov->driver_max_VFs; } EXPORT_SYMBOL_GPL(pci_sriov_get_totalvfs); diff --git a/include/linux/pci.h b/include/linux/pci.h index 024a1beda008..95fde8850393 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1952,6 +1952,7 @@ void pci_iov_remove_virtfn(struct pci_dev *dev, int id); int pci_num_vf(struct pci_dev *dev); int pci_vfs_assigned(struct pci_dev *dev); int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs); +void pci_sriov_reset_totalvfs(struct pci_dev *dev); int pci_sriov_get_totalvfs(struct pci_dev *dev); resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno); void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe); @@ -1978,6 +1979,7 @@ static inline int pci_vfs_assigned(struct pci_dev *dev) { return 0; } static inline int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs) { return 0; } +static inline void pci_sriov_reset_totalvfs(struct pci_dev *dev) { } static inline int pci_sriov_get_totalvfs(struct pci_dev *dev) { return 0; } static inline resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno)