From patchwork Sat May 26 03:00:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 10428829 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7E50060225 for ; Sat, 26 May 2018 03:00:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B9CC29417 for ; Sat, 26 May 2018 03:00:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5C62A2944E; Sat, 26 May 2018 03:00:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C7E4729417 for ; Sat, 26 May 2018 03:00:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1031031AbeEZDAf (ORCPT ); Fri, 25 May 2018 23:00:35 -0400 Received: from mail-qt0-f193.google.com ([209.85.216.193]:35857 "EHLO mail-qt0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1030996AbeEZDAe (ORCPT ); Fri, 25 May 2018 23:00:34 -0400 Received: by mail-qt0-f193.google.com with SMTP id q6-v6so8969954qtn.3 for ; Fri, 25 May 2018 20:00:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ep+GCRsP69bpIKB9OdagxJYPPm+N9aS3PaoJF1b45z4=; b=DK8v+xgBNubiZf6zhwRLG+/gfGnbkq2+2vLEzIm9E/WQBwRpItQfzhfHG4v4URrCbt 5nLBFQtLakqHjcQsWDyqHIfYFko0JIEoHV9j5tiQuqbxIJ7eX4XkCKJFWhN2F8L6MfRl nqAFdMZTLSsZRgW3y9W5QnBuXOiFl+ZibhKZ1hLSoJgDz9SXkjsL/wyqETvuLng4Oi6X AU7KgBE7Y256I2sPp4kCv05OmnS9ADSX4gVdLNDtLC0cUwUeXi9OB1xGW2fepi1eYKOt SHdfCz/UUw2Au3lTASpTIY5JTwLH8mgGBV+Yo0s9d4hkdrxJFgHsgNgqHDyZoXhLycMa qO5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ep+GCRsP69bpIKB9OdagxJYPPm+N9aS3PaoJF1b45z4=; b=TZuWaTISSX8y2l4Vbo8YIvhGq57TJ6Zpt7oMpst4E7t3ZvjqzAa8SuFL5Suvec5o8o Dg6Wk3VrNqIJlYCWInHs7sszXaWbSGCJ5xxWNop/eizfE/iMcJ7dUGq+qW+L2juxBEUe QWACaH2ryyxKZG+2c587vrHyRJ1/kKx/V+XiL0hnNdRMnyJZs33vMXFf5dncVSJPb/fS 3P66ezUpeg0SstY7hLjOxRkhRcRLIlQUE7Exgr7W4gpFg8lpVPztq/N6oGRAZEDkoRD8 RgYnsJSznyF5k5CbDj0EpBBMtKKypFdSJ3rMo82lhHRqCmTofkvwqI+kT5zbl/s0E1eT e83Q== X-Gm-Message-State: ALKqPwfMqYNkAg+G6gVigan+iGUJieoYstAAIdOf8xto5PuCjdajUg6J x1qGKgIznWaQLKNEJNngnSgnXQ== X-Google-Smtp-Source: ADUXVKJVAZsRUgYZR0kboo2+j6rUW94sSMJVj1jTtfyAg9E0Pa8sLyoC00x/uxjPHYlPdyW7CKv9iw== X-Received: by 2002:a0c:b586:: with SMTP id g6-v6mr4688015qve.94.1527303634091; Fri, 25 May 2018 20:00:34 -0700 (PDT) Received: from jkicinski-Precision-T1700.netronome.com ([75.53.12.129]) by smtp.gmail.com with ESMTPSA id f26-v6sm18240350qtg.50.2018.05.25.20.00.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 25 May 2018 20:00:33 -0700 (PDT) From: Jakub Kicinski To: Bjorn Helgaas Cc: linux-pci@vger.kernel.org, netdev@vger.kernel.org, Sathya Perla , Felix Manlunas , alexander.duyck@gmail.com, Jacob Keller , Donald Dutile , oss-drivers@netronome.com, Christoph Hellwig , Jakub Kicinski Subject: [PATCH] PCI: reset driver SR-IOV state after remove Date: Fri, 25 May 2018 20:00:24 -0700 Message-Id: <20180526030024.14866-1-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180525214525.GB92995@bhelgaas-glaptop.roam.corp.google.com> References: <20180525214525.GB92995@bhelgaas-glaptop.roam.corp.google.com> Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Bjorn points out that currently core and most of the drivers don't clean up dev->sriov->driver_max_VFs settings on .remove(). This means that if a different driver is bound afterwards it will inherit the old setting: - load PF driver 1 - driver calls pci_sriov_set_totalvfs() to reduce driver_max_VFs - unload PF driver 1 - load PF driver 2 Reset driver_max_VFs back to total_VFs after device remove. Signed-off-by: Jakub Kicinski --- I gave into the temptation and also added a warning about SR-IOV being on after remove :) Please let me know if this is anywhere close to what you had in mind! drivers/pci/iov.c | 16 ++++++++++++++++ drivers/pci/pci-driver.c | 1 + drivers/pci/pci.h | 4 ++++ 3 files changed, 21 insertions(+) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index db86fd26f8e1..5d0f560a1e28 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -574,6 +574,22 @@ void pci_iov_release(struct pci_dev *dev) sriov_release(dev); } +/** + * pci_sriov_drv_cleanup - clean up SR-IOV state after PF driver is detached + * @dev: the PCI device + */ +void pci_sriov_drv_cleanup(struct pci_dev *dev) +{ + struct pci_sriov *iov = dev->sriov; + + if (!dev->is_physfn) + return; + iov->driver_max_VFs = iov->total_VFs; + if (iov->num_VFs) + dev_warn(&dev->dev, + "driver left SR-IOV enabled after remove\n"); +} + /** * pci_iov_update_resource - update a VF BAR * @dev: the PCI device diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index b9a131137e64..932a1acf7b1b 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -443,6 +443,7 @@ static int pci_device_remove(struct device *dev) } pcibios_free_irq(pci_dev); pci_dev->driver = NULL; + pci_sriov_drv_cleanup(pci_dev); } /* Undo the runtime PM settings in local_pci_probe() */ diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 023f7cf25bff..5fa6d19762bd 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -311,6 +311,7 @@ static inline void pci_restore_ats_state(struct pci_dev *dev) #ifdef CONFIG_PCI_IOV int pci_iov_init(struct pci_dev *dev); void pci_iov_release(struct pci_dev *dev); +void pci_sriov_drv_cleanup(struct pci_dev *dev); void pci_iov_update_resource(struct pci_dev *dev, int resno); resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno); void pci_restore_iov_state(struct pci_dev *dev); @@ -323,6 +324,9 @@ static inline int pci_iov_init(struct pci_dev *dev) } static inline void pci_iov_release(struct pci_dev *dev) +{ +} +static inline void pci_sriov_drv_cleanup(struct pci_dev *dev) { } static inline void pci_restore_iov_state(struct pci_dev *dev)