From patchwork Tue Mar 6 19:29:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10262547 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9456360211 for ; Tue, 6 Mar 2018 19:30:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 853CD29160 for ; Tue, 6 Mar 2018 19:30:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 798D52916C; Tue, 6 Mar 2018 19:30:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 18C302916C for ; Tue, 6 Mar 2018 19:30:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754052AbeCFT35 (ORCPT ); Tue, 6 Mar 2018 14:29:57 -0500 Received: from mail-io0-f196.google.com ([209.85.223.196]:45494 "EHLO mail-io0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754030AbeCFT3x (ORCPT ); Tue, 6 Mar 2018 14:29:53 -0500 Received: by mail-io0-f196.google.com with SMTP id m22so206318iob.12; Tue, 06 Mar 2018 11:29:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=3OGGZ1C4FnkyldqOvS6sTBiiwnMOR617LQEXPowyiC4=; b=Vmfje/RYcoKG6KAJA5NJIVU1rb4R1/J+Y3qpP9iRgXnRB4rMB8fPEOkLDhZ1sHFvvH JinogLlCm0/+dJL2J/SCwlzmehdcoT8EvMckgm1lGfWrm2yYQiw9aZCl5nnB5TIKjUXw 56E3kHD3e8HPamu93WZ5GyLFaB/baylZqzHrF3++FWQIm+gMpX+i4B+GHH3HnpHg0zOL CTJrJQH5fBv63TKL3LOPb7G2yJGrZJvNSMbjhyzCzJwUJklMNwaRMlz1KVGGGvTLfe3C lK9CJr3y5gG3QmrDsFG9lYURAAysElvH0NG/nFljA5PTO3W38931cR0iDR8mVsFiD5xA aXlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=3OGGZ1C4FnkyldqOvS6sTBiiwnMOR617LQEXPowyiC4=; b=a14m2DOGQCebrSDPX2pcjrUX9iMk6gvlsPQAQEsjRA5nSyBXs4SY5ccKDn6OAmL/Jy hYs4BcgyyEtlCHaggBHpBggqgNvEuk/UmnNBD4ksJUv980OVsUDj78MuvtbR/bRKf6uw jzsRdyiFr/QW3B/9X/s4cFvkM1C2pdj5Pqxq2T05T+ofpryLNalE31vDOy/U1FNEfKL/ CesNhd8WoKhlpwtKRETtCb47MvEBmxYc/VpVLLIzJMuRbZ4ILRl9TGD1WZNj/uS7KeRe WdxCxk2wb0hybljbtZxRfDzkkYdPetIhgHsGPz//ci3jHYZxB26lE+1u95mpDCNQNAv3 sm0A== X-Gm-Message-State: APf1xPBVSuInV75ejMCjqFUaARtNSr6hl8YlUc1ScI6uJYn5HKmf7l2d 47fvNc9N3Hlo5ycECMHrvtE= X-Google-Smtp-Source: AG47ELt10uiDrKYg7H4yjwiFkDLzvntpOT/jJRuzJH+hNP6805MYi6HB+PKKl6rZKCTl4Ea0T+HaZA== X-Received: by 10.107.181.197 with SMTP id e188mr23643799iof.243.1520364592231; Tue, 06 Mar 2018 11:29:52 -0800 (PST) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id r6sm6769049ioe.85.2018.03.06.11.29.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Mar 2018 11:29:51 -0800 (PST) Subject: [pci PATCH v3 2/3] vfio: Add support for unmanaged or userspace managed SR-IOV From: Alexander Duyck To: bhelgaas@google.com, alexander.h.duyck@intel.com, linux-pci@vger.kernel.org Cc: virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org, netdev@vger.kernel.org, dan.daly@intel.com, linux-kernel@vger.kernel.org, mheyne@amazon.de, liang-min.wang@intel.com, mark.d.rustad@intel.com, dwmw2@infradead.org, dwmw@amazon.co.uk Date: Tue, 06 Mar 2018 11:29:50 -0800 Message-ID: <20180306192947.3153.20106.stgit@localhost.localdomain> In-Reply-To: <20180306192423.3153.42741.stgit@localhost.localdomain> References: <20180306192423.3153.42741.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck This patch is meant to allow assignment of an SR-IOV enabled PF, as in VFs have been generated, with vfio-pci. My understanding is the primary use case for this is something like DPDK running the PF while the VFs are all assigned to guests. A secondary effect of this is that it provides an interface through which it would be possible to enable SR-IOV on drivers that may not have a physical function that actually manages the device. Enabling SR-IOV should be pretty straight forward. As long as there are no userspace processes currently controlling the interface the number of VFs can be changed, and VFs will be generated without drivers being loaded on the host. Once the userspace process begins controlling the interface the number of VFs cannot be updated via the sysfs until the control is released. Note the VFs will have drivers load on them in the host if the sriov_unmanaged_autoprobe is updated to a value of 1. However the behavior of the VFs in such a setup cannot be guaranteed as the PF will not be available until the userspace process starts and begins to manage the device. For now I am leaving the value as locked when the PF is being controlled from userspace as a form of synchronization. Basically this way we cannot have the number of VFs change out from under the process so it should not require any notification framework, and the configuration can just be read out via configuration space accesses. Signed-off-by: Alexander Duyck --- drivers/vfio/pci/vfio_pci.c | 59 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index b0f759476900..8025d7336071 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -1224,6 +1224,8 @@ static void vfio_pci_remove(struct pci_dev *pdev) VGA_RSRC_LEGACY_IO | VGA_RSRC_LEGACY_MEM); } + pci_disable_sriov(pdev); + if (!disable_idle_d3) pci_set_power_state(pdev, PCI_D0); } @@ -1260,12 +1262,69 @@ static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, .error_detected = vfio_pci_aer_err_detected, }; +#ifdef CONFIG_PCI_IOV +static int vfio_pci_sriov_configure(struct pci_dev *pdev, int nr_virtfn) +{ + struct vfio_pci_device *vdev; + struct vfio_device *device; + int err; + + device = vfio_device_get_from_dev(&pdev->dev); + if (device == NULL) + return -ENODEV; + + vdev = vfio_device_data(device); + if (vdev == NULL) { + vfio_device_put(device); + return -ENODEV; + } + + /* + * If a userspace process is already using this device just return + * busy and don't allow for any changes. + */ + if (vdev->refcnt) { + pci_warn(pdev, + "PF is currently in use, blocked until released by user\n"); + return -EBUSY; + } + + err = pci_sriov_configure_unmanaged(pdev, nr_virtfn); + if (err <= 0) + return err; + + /* + * We are now leaving VFs in the control of some unknown PF entity. + * + * Best case is a well behaved userspace PF is expected and any VMs + * that the VFs will be assigned to are dependent on the userspace + * entity anyway. An example being NFV where maybe the PF is acting + * as an accelerated interface for a firewall or switch. + * + * Worst case is somebody really messed up and just enabled SR-IOV + * on a device they were planning to assign to a VM somwhere. + * + * In either case it is probably best for us to set the taint flag + * and warn the user since this could get really ugly really quick + * if this wasn't what they were planning to do. + */ + add_taint(TAINT_USER, LOCKDEP_STILL_OK); + pci_warn(pdev, + "Adding kernel taint for vfio-pci now managing SR-IOV PF device\n"); + + return nr_virtfn; +} +#endif /* CONFIG_PCI_IOV */ + static struct pci_driver vfio_pci_driver = { .name = "vfio-pci", .id_table = NULL, /* only dynamic ids */ .probe = vfio_pci_probe, .remove = vfio_pci_remove, .err_handler = &vfio_err_handlers, +#ifdef CONFIG_PCI_IOV + .sriov_configure = vfio_pci_sriov_configure, +#endif }; struct vfio_devices {