From patchwork Tue Feb 27 19:06:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10245979 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 171C960384 for ; Tue, 27 Feb 2018 19:07:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0862628877 for ; Tue, 27 Feb 2018 19:07:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F010728A21; Tue, 27 Feb 2018 19:07:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4283628877 for ; Tue, 27 Feb 2018 19:07:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751747AbeB0THA (ORCPT ); Tue, 27 Feb 2018 14:07:00 -0500 Received: from mail-io0-f196.google.com ([209.85.223.196]:36442 "EHLO mail-io0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751126AbeB0TG6 (ORCPT ); Tue, 27 Feb 2018 14:06:58 -0500 Received: by mail-io0-f196.google.com with SMTP id e30so427109ioc.3; Tue, 27 Feb 2018 11:06:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:user-agent:mime-version :content-transfer-encoding; bh=SUgJxVH88TNv/9lDzHbd0OcM6tmUoC6oGuun2x5d6uU=; b=KznCRpec0dQxdQSudDbs944xX9l3JfDTH78OrIkfgEPYwg9AsfsKnZ2Oc+7NUDUVYR QtWvjii2n0q8bHvBDD2p1zO3hFGafX6zZdHGW2JQX4j7crFeo8UM3x8DAThMtV7FLv4x mQRBOZ/DHo2RMGeq/mKzxK6knBN2z1v0XgYjLsECoBYtRNFFekbigD4qv0zOT3yQKTwc rzkWkyshQimL4UH2pnZGvD2A2iu9NQxKx/UPVP/+XALChEY/11OA2y4BC3aU/6AM/+kv tD/5l7sUcBFtCFaGsaqgokFglYpDnE1t2eDj5JONYlNiia8cyniOLgwZBo65tVHS108p eVUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:user-agent :mime-version:content-transfer-encoding; bh=SUgJxVH88TNv/9lDzHbd0OcM6tmUoC6oGuun2x5d6uU=; b=K/iX3Mcn6gP4lTaU7IKYbQ0oVAOvxF/7ji2eiYti6Wz6IKV85NoumU09xJ/UdQ/XUu lTZV3JjaQaF5MLK1MpmHYrLDfk0NLFjisKMU2dm1Vai7ZM797HT/87k16HY8knkUWhnj ys2IVsZg48IIP6J8Hs0Fkh55rmZcZCVsyNa+sdF/lEiXSiFK8BImk7kgsq9HFiRBqXRx OYuO5uYfsQ9ulwCnsHYk+93dxK/1KID1hXFF0hTsNOR/egVCjc17EKos7ubwvtTFmfWS Q1pZnvc7UFDdxtWhK3eS4rU9XM57gxQkTmUOZAckaGyNgB8oUKXumXByJrZPAlYkGEAS O41A== X-Gm-Message-State: APf1xPAKrX2QpzRTTqKqCgWEzhJoLmmiCqCL5KZ6qcAgV9O48RPKMO3l z2w0/NzWIgpOUnLawggjEP4= X-Google-Smtp-Source: AG47ELs3tSu3StE6D90cbdlzX3eCLEZTuAhfteoKKY/7WvIKW9/v2tnAgKQo0tfy1RpLz2pKXTn2Aw== X-Received: by 10.107.111.6 with SMTP id k6mr6399602ioc.86.1519758416980; Tue, 27 Feb 2018 11:06:56 -0800 (PST) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id v140sm110432itb.42.2018.02.27.11.06.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Feb 2018 11:06:56 -0800 (PST) Subject: [PATCH] pci-iov: Add support for unmanaged SR-IOV From: Alexander Duyck To: bhelgaas@google.com, linux-pci@vger.kernel.org Cc: Alexander Duyck , virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org, netdev@vger.kernel.org, dan.daly@intel.com, linux-kernel@vger.kernel.org, mheyne@amazon.de, liang-min.wang@intel.com, mark.d.rustad@intel.com, dwmw2@infradead.org, dwmw@amazon.co.uk Date: Tue, 27 Feb 2018 11:06:54 -0800 Message-ID: <20180227190520.3273.79728.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck This patch is meant to add support for SR-IOV on devices when the VFs are not managed by the kernel. Examples of recent patches attempting to do this include: virto - https://patchwork.kernel.org/patch/10241225/ pci-stub - https://patchwork.kernel.org/patch/10109935/ vfio - https://patchwork.kernel.org/patch/10103353/ uio - https://patchwork.kernel.org/patch/9974031/ Since this is quickly blowing up into a multi-driver problem it is probably best to implement this solution in one spot. This patch is an attempt to do that. What we do with this patch is provide a generic call to enable SR-IOV in the case that the PF driver is either not present, or the PF driver doesn't support configuring SR-IOV. A new sysfs value called sriov_unmanaged_autoprobe has been added. This value is used as the drivers_autoprobe setting of the VFs when they are being managed by an external entity such as userspace or device firmware instead of being managed by the kernel. One side effect of this change is that the sriov_drivers_autoprobe and sriov_unmanaged_autoprobe will only apply their updates when SR-IOV is disabled. Attempts to update them when SR-IOV is in use will only update the local value and will not update sriov->autoprobe. I based my patch set originally on the patch by Mark Rustad but there isn't much left after going through and cleaning out the bits that were no longer needed, and after incorporating the feedback from David Miller. I have included the authors of the original 4 patches above in the Cc here. My hope is to get feedback and/or review on if this works for their use cases. Cc: Mark Rustad Cc: Maximilian Heyne Cc: Liang-Min Wang Cc: David Woodhouse Signed-off-by: Alexander Duyck --- drivers/pci/iov.c | 27 +++++++++++++++++++- drivers/pci/pci-driver.c | 2 + drivers/pci/pci-sysfs.c | 62 +++++++++++++++++++++++++++++++++++++++++----- drivers/pci/pci.h | 4 ++- include/linux/pci.h | 1 + 5 files changed, 86 insertions(+), 10 deletions(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 677924ae0350..7b8858bd8d03 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -446,6 +446,7 @@ static int sriov_init(struct pci_dev *dev, int pos) pci_read_config_word(dev, pos + PCI_SRIOV_VF_DID, &iov->vf_device); iov->pgsz = pgsz; iov->self = dev; + iov->autoprobe = true; iov->drivers_autoprobe = true; pci_read_config_dword(dev, pos + PCI_SRIOV_CAP, &iov->cap); pci_read_config_byte(dev, pos + PCI_SRIOV_FUNC_LINK, &iov->link); @@ -643,8 +644,11 @@ void pci_restore_iov_state(struct pci_dev *dev) */ void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool auto_probe) { - if (dev->is_physfn) + if (dev->is_physfn) { dev->sriov->drivers_autoprobe = auto_probe; + if (!dev->sriov->num_VFs) + dev->sriov->autoprobe = auto_probe; + } } /** @@ -703,6 +707,27 @@ void pci_disable_sriov(struct pci_dev *dev) EXPORT_SYMBOL_GPL(pci_disable_sriov); /** + * pci_sriov_configure_unmanaged - helper to configure unmanaged SR-IOV + * @dev: the PCI device + * @nr_virtfn: number of virtual functions to enable, 0 to disable + * + * Used to provide generic enable/disable SR-IOV option for devices + * that do not manage the VFs generated by their driver, or have no + * driver present. + */ +int pci_sriov_configure_unmanaged(struct pci_dev *dev, int nr_virtfn) +{ + int rc = 0; + + if (!nr_virtfn) + pci_disable_sriov(dev); + else + rc = pci_enable_sriov(dev, nr_virtfn); + + return rc ? rc : nr_virtfn; +} + +/** * pci_num_vf - return number of VFs associated with a PF device_release_driver * @dev: the PCI device * diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index 3bed6beda051..2cc68dff6130 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -398,7 +398,7 @@ void __weak pcibios_free_irq(struct pci_dev *dev) #ifdef CONFIG_PCI_IOV static inline bool pci_device_can_probe(struct pci_dev *pdev) { - return (!pdev->is_virtfn || pdev->physfn->sriov->drivers_autoprobe); + return (!pdev->is_virtfn || pdev->physfn->sriov->autoprobe); } #else static inline bool pci_device_can_probe(struct pci_dev *pdev) diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c index eb6bee8724cc..e701b6dbc267 100644 --- a/drivers/pci/pci-sysfs.c +++ b/drivers/pci/pci-sysfs.c @@ -605,6 +605,7 @@ static ssize_t sriov_numvfs_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { + int (*sriov_configure)(struct pci_dev *dev, int num_vfs); struct pci_dev *pdev = to_pci_dev(dev); int ret; u16 num_vfs; @@ -622,15 +623,20 @@ static ssize_t sriov_numvfs_store(struct device *dev, goto exit; /* is PF driver loaded w/callback */ - if (!pdev->driver || !pdev->driver->sriov_configure) { - pci_info(pdev, "Driver doesn't support SRIOV configuration via sysfs\n"); - ret = -ENOENT; - goto exit; - } + if (pdev->driver && pdev->driver->sriov_configure) + sriov_configure = pdev->driver->sriov_configure; + else + sriov_configure = pci_sriov_configure_unmanaged; if (num_vfs == 0) { /* disable VFs */ - ret = pdev->driver->sriov_configure(pdev, 0); + ret = sriov_configure(pdev, 0); + + /* + * Fall back to drivers_autoprobe in case legacy driver + * decides to enable SR-IOV on load. + */ + pdev->sriov->autoprobe = pdev->sriov->drivers_autoprobe; goto exit; } @@ -642,7 +648,14 @@ static ssize_t sriov_numvfs_store(struct device *dev, goto exit; } - ret = pdev->driver->sriov_configure(pdev, num_vfs); + /* + * Update autoprobe based on unmanaged_autoprobe settings if PF + * driver is not managing the SR-IOV configuration for this device. + */ + if (!pdev->driver || !pdev->driver->sriov_configure) + pdev->sriov->autoprobe = pdev->sriov->unmanaged_autoprobe; + + ret = sriov_configure(pdev, num_vfs); if (ret < 0) goto exit; @@ -705,7 +718,37 @@ static ssize_t sriov_drivers_autoprobe_store(struct device *dev, if (kstrtobool(buf, &drivers_autoprobe) < 0) return -EINVAL; + device_lock(&pdev->dev); + pdev->sriov->drivers_autoprobe = drivers_autoprobe; + if (!pdev->sriov->num_VFs) + pdev->sriov->autoprobe = drivers_autoprobe; + + device_unlock(&pdev->dev); + + return count; +} + +static ssize_t sriov_unmanaged_autoprobe_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct pci_dev *pdev = to_pci_dev(dev); + + return sprintf(buf, "%u\n", pdev->sriov->unmanaged_autoprobe); +} + +static ssize_t sriov_unmanaged_autoprobe_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct pci_dev *pdev = to_pci_dev(dev); + bool unmanaged_autoprobe; + + if (kstrtobool(buf, &unmanaged_autoprobe) < 0) + return -EINVAL; + + pdev->sriov->unmanaged_autoprobe = unmanaged_autoprobe; return count; } @@ -720,6 +763,10 @@ static ssize_t sriov_drivers_autoprobe_store(struct device *dev, static struct device_attribute sriov_drivers_autoprobe_attr = __ATTR(sriov_drivers_autoprobe, (S_IRUGO|S_IWUSR|S_IWGRP), sriov_drivers_autoprobe_show, sriov_drivers_autoprobe_store); +static struct device_attribute sriov_unmanaged_autoprobe_attr = + __ATTR(sriov_unmanaged_autoprobe, (S_IRUGO|S_IWUSR|S_IWGRP), + sriov_unmanaged_autoprobe_show, + sriov_unmanaged_autoprobe_store); #endif /* CONFIG_PCI_IOV */ static ssize_t driver_override_store(struct device *dev, @@ -1789,6 +1836,7 @@ static umode_t pcie_dev_attrs_are_visible(struct kobject *kobj, &sriov_stride_attr.attr, &sriov_vf_device_attr.attr, &sriov_drivers_autoprobe_attr.attr, + &sriov_unmanaged_autoprobe_attr.attr, NULL, }; diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index fcd81911b127..b5f8b034f02d 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -272,7 +272,9 @@ struct pci_sriov { struct pci_dev *dev; /* Lowest numbered PF */ struct pci_dev *self; /* This PF */ resource_size_t barsz[PCI_SRIOV_NUM_BARS]; /* VF BAR size */ - bool drivers_autoprobe; /* Auto probing of VFs by driver */ + bool autoprobe; /* Auto probing of VFs by VF driver */ + bool drivers_autoprobe; /* "" managed by PF driver */ + bool unmanaged_autoprobe; /* "" unmanaged by kernel */ }; /* pci_dev priv_flags */ diff --git a/include/linux/pci.h b/include/linux/pci.h index 024a1beda008..0d2b10aeadfa 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1947,6 +1947,7 @@ static inline void pci_mmcfg_late_init(void) { } int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn); void pci_disable_sriov(struct pci_dev *dev); +int pci_sriov_configure_unmanaged(struct pci_dev *dev, int num_vfs); int pci_iov_add_virtfn(struct pci_dev *dev, int id); void pci_iov_remove_virtfn(struct pci_dev *dev, int id); int pci_num_vf(struct pci_dev *dev);