From patchwork Thu Sep 23 12:54:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4D08C4332F for ; Thu, 23 Sep 2021 12:58:07 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B733161090 for ; Thu, 23 Sep 2021 12:58:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B733161090 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194150.345947 (Exim 4.92) (envelope-from ) id 1mTOIZ-0006fo-2k; Thu, 23 Sep 2021 12:57:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194150.345947; Thu, 23 Sep 2021 12:57:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOIY-0006fC-RQ; Thu, 23 Sep 2021 12:57:58 +0000 Received: by outflank-mailman (input) for mailman id 194150; Thu, 23 Sep 2021 12:57:57 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOGr-0004it-5p for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:56:13 +0000 Received: from mail-lf1-x130.google.com (unknown [2a00:1450:4864:20::130]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id b46d3d5b-4489-4e5a-941c-d26fc3fb88f9; Thu, 23 Sep 2021 12:55:06 +0000 (UTC) Received: by mail-lf1-x130.google.com with SMTP id p29so25910679lfa.11 for ; Thu, 23 Sep 2021 05:55:06 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:04 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: b46d3d5b-4489-4e5a-941c-d26fc3fb88f9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FKgxJK/5NyOFFpbY0xtHY00JQINZqmrpnwuzOqh/Ukk=; b=ea9E6tTANi/jveiexkEVCENbTHTScpXopjrod/3psE9GkasQWaKTmKmNiQdbaWuDC/ yH7u43xdaWw6JGuAmS7VdZIbZs2VK2GJpvAX1gsdiwuVDd+TbekUS79BlpSs/bsPGdKQ wP7eOlSKkJfw0G+fTQSHuDwBcZKnZidY09Ob43s+XqlFLMymD65CN10T5mkq4cPwxDdv F82q1IJ6In4oWymTxGjH3GeMN5quVUG8HgV9f9dHpD81lh977Lm95mcWCzHZfGcvaIYS 3ceMEDf10lcNmeqfesDK5HQo4ZBjVI5XVdBj4QNsDHEcVBEJwNF9cuMOkmMvRJYGDj2x XeDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FKgxJK/5NyOFFpbY0xtHY00JQINZqmrpnwuzOqh/Ukk=; b=tlGEHPq9WAcQiTQPWEZ4UJctEr7SyGlw8PBcgg3Bw5L0iBcvlrmrNhP6zXbibKbXss Qh6f3eAo2/hbg5xBc3cKX46ugR9thksH66fPINvqjpJk3pN88BoPDVhKWUwvsH7vrqlJ m9y9CjKNWXYOP+3yuuFgeeVYYmUiQdEEfyxy7nsbYK92i4JLDcoptmrjUTWc7fHbM5tW GAAKUuqMGkUZJbITl8UAAF8N9TK2bh4vFSZjfFU6tcZcN3HGWUDy9KcFuunExCOjPEoQ 6LMcYgRKIzXUOGZVNUJdKi3VMQA8lq8ira9Fo7q2G5kR7tE43v+GrEok3VcFTA/pKI1M mzCg== X-Gm-Message-State: AOAM532d/NTwju1TkcHaV8WSpYdFEMhgQr4slKU4Z7oDVpBsvoXZuVBR 7wldFMiFV6TUF1ULxIq1N5aYU6srfofs2g== X-Google-Smtp-Source: ABdhPJxISqOOWmLgWDAAOw3kzzTRX2xcHM4R9vZqpv+EYE/4AAOVpEOzHCAWnBLNdTRXDedIXQbwPg== X-Received: by 2002:a05:6512:2244:: with SMTP id i4mr4065599lfu.219.1632401704895; Thu, 23 Sep 2021 05:55:04 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 01/11] vpci: Make vpci registers removal a dedicated function Date: Thu, 23 Sep 2021 15:54:51 +0300 Message-Id: <20210923125501.234252-2-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko This is in preparation for dynamic assignment of the vpci register handlers depending on the domain: hwdom or guest. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- Since v1: - constify struct pci_dev where possible --- xen/drivers/vpci/vpci.c | 7 ++++++- xen/include/xen/vpci.h | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index cbd1bac7fc33..1666402d55b8 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -35,7 +35,7 @@ extern vpci_register_init_t *const __start_vpci_array[]; extern vpci_register_init_t *const __end_vpci_array[]; #define NUM_VPCI_INIT (__end_vpci_array - __start_vpci_array) -void vpci_remove_device(struct pci_dev *pdev) +void vpci_remove_device_registers(const struct pci_dev *pdev) { spin_lock(&pdev->vpci->lock); while ( !list_empty(&pdev->vpci->handlers) ) @@ -48,6 +48,11 @@ void vpci_remove_device(struct pci_dev *pdev) xfree(r); } spin_unlock(&pdev->vpci->lock); +} + +void vpci_remove_device(struct pci_dev *pdev) +{ + vpci_remove_device_registers(pdev); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 9f5b5d52e159..2e910d0b1f90 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -28,6 +28,8 @@ int __must_check vpci_add_handlers(struct pci_dev *dev); /* Remove all handlers and free vpci related structures. */ void vpci_remove_device(struct pci_dev *pdev); +/* Remove all handlers for the device given. */ +void vpci_remove_device_registers(const struct pci_dev *pdev); /* Add/remove a register handler. */ int __must_check vpci_add_register(struct vpci *vpci, From patchwork Thu Sep 23 12:54:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F20AEC433FE for ; Thu, 23 Sep 2021 12:57:51 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AE79961152 for ; Thu, 23 Sep 2021 12:57:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AE79961152 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194144.345924 (Exim 4.92) (envelope-from ) id 1mTOIJ-000572-3S; Thu, 23 Sep 2021 12:57:43 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194144.345924; Thu, 23 Sep 2021 12:57:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOII-000556-RB; Thu, 23 Sep 2021 12:57:42 +0000 Received: by outflank-mailman (input) for mailman id 194144; Thu, 23 Sep 2021 12:57:41 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOH1-0004it-6A for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:56:23 +0000 Received: from mail-lf1-x12d.google.com (unknown [2a00:1450:4864:20::12d]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 56e2b585-9c23-4014-bc02-da210cf5b44a; Thu, 23 Sep 2021 12:55:07 +0000 (UTC) Received: by mail-lf1-x12d.google.com with SMTP id i4so26250580lfv.4 for ; Thu, 23 Sep 2021 05:55:07 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:05 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 56e2b585-9c23-4014-bc02-da210cf5b44a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9Y+DU0DSrlSJm3RBHsKqMThf7inWBIT9pPD3eSNkVAE=; b=hYxKWCjYG91LckucdXz7kiTTL5AB09I7V3lbQ4uZfD/q/MYCdyjE2/BPyX2RGkGkdD LfzE8+Dq+LMIIayzkHPshcRPxLaXsNy1adj8t3KB0xIEB7dvCLotKdf5i3SbpzPtedxp izc40arDT+3hOiCrJkkto5MPJztcsDKNoskt7sClcJASUMvw7ijT/tgxdo2IeKV32+ku EHFUoUbQLhE8/6dSUTfwfUKIDtfWs3YfQIptdnsV2te7IokWTGwuBHgN30IWV3kbcbbJ TFLgalge/nTmIig4YTtpi5Be+40ZTTJZ/wbm4FINugBjRCfV+W/vTLDcKfhE0U7wNW5J Fgzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9Y+DU0DSrlSJm3RBHsKqMThf7inWBIT9pPD3eSNkVAE=; b=dFdbrSulI0lh5wU1CMbGz/KoUiC86HHuc+ShB7P56GBiwaswyR0ZcruFRXt/BpwetH C8p1PHajJMfNaOo9bk/j/dNdnhqm02/ZKtOd62EVNnquRynUg+gcveBtNk0ZnSIeAMZk ID3iR8JX3YBgejMyjzQrgtNOk8XIOWHxwe0d53hV+lkaFapkYSdmHHeXfL0f7spSEAnf BWlXe5VfABOGsDv2q85U2gdZQ06osozRUxYZ4aJr+dK6zhLAapyMPiaErro782SnbN9v DVnc9KUX3hW7ClwM40faGe4VsJxtIfs1pJsejST7xphlXXShzXRYv6PHopFqmO7/jl+n 0FZw== X-Gm-Message-State: AOAM532dvFyUfJbVzxEVfhnwPHJTapvH488VP/CqErlzTv0QDTk4EhEA 1r+jhQx6h0IZxQ/+ra7zxoDgvKOhX5QErw== X-Google-Smtp-Source: ABdhPJws+VUTSrWiIzEEm+gWdSYS1iA4cTrQf4OFeA2lEZRm4C2x+KWOsrumrELjauDBZ1A2tBy7QA== X-Received: by 2002:ac2:5f83:: with SMTP id r3mr4179435lfe.686.1632401706071; Thu, 23 Sep 2021 05:55:06 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 02/11] vpci: Add hooks for PCI device assign/de-assign Date: Thu, 23 Sep 2021 15:54:52 +0300 Message-Id: <20210923125501.234252-3-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko When a PCI device gets assigned/de-assigned some work on vPCI side needs to be done for that device. Introduce a pair of hooks so vPCI can handle that. Please note, that in the current design the error path is handled by the toolstack via XEN_DOMCTL_assign_device/XEN_DOMCTL_deassign_device, so this is why it is acceptable not to de-assign devices if vPCI's assign fails, e.g. the roll back will be handled on deassign_device when it is called by the toolstack. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - extended the commit message --- xen/drivers/passthrough/pci.c | 9 +++++++++ xen/drivers/vpci/vpci.c | 21 +++++++++++++++++++++ xen/include/xen/vpci.h | 18 ++++++++++++++++++ 3 files changed, 48 insertions(+) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index fc3469bc12dc..e1da283d73ad 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -872,6 +872,10 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, if ( ret ) goto out; + ret = vpci_deassign_device(d, pdev); + if ( ret ) + goto out; + if ( pdev->domain == hardware_domain ) pdev->quarantine = false; @@ -1431,6 +1435,11 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) rc = hd->platform_ops->assign_device(d, devfn, pci_to_dev(pdev), flag); } + if ( rc ) + goto done; + + rc = vpci_assign_device(d, pdev); + done: if ( rc ) printk(XENLOG_G_WARNING "%pd: assign (%pp) failed (%d)\n", diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 1666402d55b8..a8fed3d2c42e 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -86,6 +86,27 @@ int __hwdom_init vpci_add_handlers(struct pci_dev *pdev) return rc; } + +/* Notify vPCI that device is assigned to guest. */ +int vpci_assign_device(struct domain *d, const struct pci_dev *dev) +{ + /* It only makes sense to assign for hwdom or guest domain. */ + if ( is_system_domain(d) || !has_vpci(d) ) + return 0; + + return 0; +} + +/* Notify vPCI that device is de-assigned from guest. */ +int vpci_deassign_device(struct domain *d, const struct pci_dev *dev) +{ + /* It only makes sense to de-assign from hwdom or guest domain. */ + if ( is_system_domain(d) || !has_vpci(d) ) + return 0; + + return 0; +} + #endif /* __XEN__ */ static int vpci_register_cmp(const struct vpci_register *r1, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 2e910d0b1f90..b9485b2aea1b 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -26,6 +26,12 @@ typedef int vpci_register_init_t(struct pci_dev *dev); /* Add vPCI handlers to device. */ int __must_check vpci_add_handlers(struct pci_dev *dev); +/* Notify vPCI that device is assigned/de-assigned to/from guest. */ +int __must_check vpci_assign_device(struct domain *d, + const struct pci_dev *dev); +int __must_check vpci_deassign_device(struct domain *d, + const struct pci_dev *dev); + /* Remove all handlers and free vpci related structures. */ void vpci_remove_device(struct pci_dev *pdev); /* Remove all handlers for the device given. */ @@ -220,6 +226,18 @@ static inline int vpci_add_handlers(struct pci_dev *pdev) return 0; } +static inline int vpci_assign_device(struct domain *d, + const struct pci_dev *dev) +{ + return 0; +}; + +static inline int vpci_deassign_device(struct domain *d, + const struct pci_dev *dev) +{ + return 0; +}; + static inline void vpci_dump_msi(void) { } static inline uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, From patchwork Thu Sep 23 12:54:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CC26C433FE for ; Thu, 23 Sep 2021 12:57:34 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 030B96103D for ; Thu, 23 Sep 2021 12:57:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 030B96103D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194129.345876 (Exim 4.92) (envelope-from ) id 1mTOI3-00037S-2K; Thu, 23 Sep 2021 12:57:27 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194129.345876; Thu, 23 Sep 2021 12:57:27 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOI2-000377-TO; Thu, 23 Sep 2021 12:57:26 +0000 Received: by outflank-mailman (input) for mailman id 194129; Thu, 23 Sep 2021 12:57:25 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHL-0004it-6g for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:56:43 +0000 Received: from mail-lf1-x132.google.com (unknown [2a00:1450:4864:20::132]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 96030892-7a61-4732-967d-19f6ffea7ce8; Thu, 23 Sep 2021 12:55:09 +0000 (UTC) Received: by mail-lf1-x132.google.com with SMTP id b15so25737297lfe.7 for ; Thu, 23 Sep 2021 05:55:09 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:06 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 96030892-7a61-4732-967d-19f6ffea7ce8 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TDiVVp/85Hu7qvLwikbqJ56SNZPH692g3qboO1XLKAU=; b=DH6sRara2lIzT7fiwygzxRyOEAe577X/U+ypPVBFF3U4XBHY3okF7NHXhAa5CI51Wv Fw/7ho2wGWStYpkbJMigRCdA+zzNU2ENw6FgfszkkxD9GKyiH27CqGyMG/KCrbP2FLDs oTfO8tIjkCLZ58BkutfSpTkZYIHxJMyLxumi5oe85jSGgg3N9yOqUjoc+QmgCWY8kxNV ZB8EGKcddtd4yuCpos2HcQM64WWqhJWGxWhp7FyxFSOFjVzHwpjwRw1RUkrak7yZBWxN zTkTxwCRoDZH2Cemj7kYlRKwv6wuP/qBCODwtaxVsC5+dedlFvIzSEZiMRSOA/u4r/Fo SHNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TDiVVp/85Hu7qvLwikbqJ56SNZPH692g3qboO1XLKAU=; b=FlZSmYqkQf2Tyl3See2tex9xXOdO5oHHltCsviCHS5fIWWC18VidiU9SOPJSMXnkAq qOvXuCxiaOPMXWpsOAoZ2ldPD7AcyqalmZhzd5Zx+aG+PkNCi4EYoyxVlBgq4nkEZm9T d0CvmFrPhGJHELV2YCS5DHAZu1Ab80+B4CWv3mKZuUG/wQoIz09WNrBP/HUuNtkKdf8s BYT5CP/RU9elfhQDEBW6ov1VaqQMNgCdGnsh2TaQG91ZO6y/nDUOYJJq3Q7JTGwBp7X4 zbiDgIVT/pB5LCjHsMtTeOoEozBDvAXsaXcie87z+wYLaa4ZXc/Orog2nDxLORwsy4F+ g86Q== X-Gm-Message-State: AOAM53178bYDJmQzvewRufKUY2XifSr8uQ1IwGg2UzuSE8UfZi9V24VX QlXIlHKkUUSjETERgLfoItjltJ2fJFa1Dg== X-Google-Smtp-Source: ABdhPJwtuLnpa/+icsyqakDioUzcXtFi1z998oW4N/1k6zq5qBg75ioZqd9T0FC0ynDimZ2biRkQeQ== X-Received: by 2002:a05:6512:2e8:: with SMTP id m8mr4278108lfq.22.1632401707216; Thu, 23 Sep 2021 05:55:07 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 03/11] vpci/header: Move register assignments from init_bars Date: Thu, 23 Sep 2021 15:54:53 +0300 Message-Id: <20210923125501.234252-4-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko This is in preparation for dynamic assignment of the vPCI register handlers depending on the domain: hwdom or guest. The need for this step is that it is easier to have all related functionality put at one place. When the subsequent patches add decisions on which handlers to install, e.g. hwdom or guest handlers, then this is easily achievable. Signed-off-by: Oleksandr Andrushchenko --- Since v1: - constify struct pci_dev where possible - extend patch description --- xen/drivers/vpci/header.c | 83 ++++++++++++++++++++++++++------------- 1 file changed, 56 insertions(+), 27 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index f8cd55e7c024..3d571356397a 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -445,6 +445,55 @@ static void rom_write(const struct pci_dev *pdev, unsigned int reg, rom->addr = val & PCI_ROM_ADDRESS_MASK; } +static int add_bar_handlers(const struct pci_dev *pdev) +{ + unsigned int i; + struct vpci_header *header = &pdev->vpci->header; + struct vpci_bar *bars = header->bars; + int rc; + + /* Setup a handler for the command register. */ + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, + 2, header); + if ( rc ) + return rc; + + if ( pdev->ignore_bars ) + return 0; + + for ( i = 0; i < PCI_HEADER_NORMAL_NR_BARS + 1; i++ ) + { + if ( (bars[i].type == VPCI_BAR_IO) || (bars[i].type == VPCI_BAR_EMPTY) ) + continue; + + if ( bars[i].type == VPCI_BAR_ROM ) + { + unsigned int rom_reg; + uint8_t header_type = pci_conf_read8(pdev->sbdf, + PCI_HEADER_TYPE) & 0x7f; + if ( header_type == PCI_HEADER_TYPE_NORMAL ) + rom_reg = PCI_ROM_ADDRESS; + else + rom_reg = PCI_ROM_ADDRESS1; + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, + rom_reg, 4, &bars[i]); + if ( rc ) + return rc; + } + else + { + uint8_t reg = PCI_BASE_ADDRESS_0 + i * 4; + + /* This is either VPCI_BAR_MEM32 or VPCI_BAR_MEM64_{LO|HI}. */ + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, + 4, &bars[i]); + if ( rc ) + return rc; + } + } + return 0; +} + static int init_bars(struct pci_dev *pdev) { uint16_t cmd; @@ -470,14 +519,8 @@ static int init_bars(struct pci_dev *pdev) return -EOPNOTSUPP; } - /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); - if ( rc ) - return rc; - if ( pdev->ignore_bars ) - return 0; + return add_bar_handlers(pdev); /* Disable memory decoding before sizing. */ cmd = pci_conf_read16(pdev->sbdf, PCI_COMMAND); @@ -492,14 +535,6 @@ static int init_bars(struct pci_dev *pdev) if ( i && bars[i - 1].type == VPCI_BAR_MEM64_LO ) { bars[i].type = VPCI_BAR_MEM64_HI; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); - if ( rc ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } - continue; } @@ -532,14 +567,6 @@ static int init_bars(struct pci_dev *pdev) bars[i].addr = addr; bars[i].size = size; bars[i].prefetchable = val & PCI_BASE_ADDRESS_MEM_PREFETCH; - - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, 4, - &bars[i]); - if ( rc ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } } /* Check expansion ROM. */ @@ -553,11 +580,13 @@ static int init_bars(struct pci_dev *pdev) rom->addr = addr; header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & PCI_ROM_ADDRESS_ENABLE; + } - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, rom_reg, - 4, rom); - if ( rc ) - rom->type = VPCI_BAR_EMPTY; + rc = add_bar_handlers(pdev); + if ( rc ) + { + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + return rc; } return (cmd & PCI_COMMAND_MEMORY) ? modify_bars(pdev, cmd, false) : 0; From patchwork Thu Sep 23 12:54:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19893C433F5 for ; Thu, 23 Sep 2021 12:57:22 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C7A8661019 for ; Thu, 23 Sep 2021 12:57:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C7A8661019 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194125.345858 (Exim 4.92) (envelope-from ) id 1mTOHn-00027K-DR; Thu, 23 Sep 2021 12:57:11 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194125.345858; Thu, 23 Sep 2021 12:57:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHn-000266-8l; Thu, 23 Sep 2021 12:57:11 +0000 Received: by outflank-mailman (input) for mailman id 194125; Thu, 23 Sep 2021 12:57:10 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHG-0004it-6L for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:56:38 +0000 Received: from mail-lf1-x136.google.com (unknown [2a00:1450:4864:20::136]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 9d0baee5-c887-4f26-b64c-1231e4c064d0; Thu, 23 Sep 2021 12:55:09 +0000 (UTC) Received: by mail-lf1-x136.google.com with SMTP id t10so26352813lfd.8 for ; Thu, 23 Sep 2021 05:55:09 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:07 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9d0baee5-c887-4f26-b64c-1231e4c064d0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8BU9mTDeAzJlCIShSCG3B8dpXtNyqwk/IzMfbTSVhQk=; b=EMo4gYUd9HuWyuM57Ppq0ISKssPDVmxqd/a2Lx8YxHs+mmxcN3gEsd/EPXpnRYNWZp xUlW26eHqyQlqEpQH2chnUCMJNrSGZO3Olbx+hOgheIIXNim7uQ4qz1WphFudbVt50hd BCE6CDc6s2BR9ov02TwJG7FxwD0KVKAB8inrPSF6WAz2svtSEwAjq2zndY8KtQBCs2Bt HImAJ9sxfgDV5pyymroSzP67w/57I5BESX9bfWnOdMX54tMd6Ku2IEzbJ+9gjj97OMYH lbjW5qmWvr4ZF3gWxm6X3XoGpR5GKQmxcH2VeFEYmbXfqZJOLOnSZqJuBjHlzRFybmot cppw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8BU9mTDeAzJlCIShSCG3B8dpXtNyqwk/IzMfbTSVhQk=; b=4a0G+kWshZWoBxi7Xqd6K1O0pb4xD9vF9QBvwoI+2kpIT30DmiXFGhsxycY2KiQi3S Lw5b9ssgWW9GSy9SiRzo2bRAcaAo4Yuex4wY0zslgEyQqhd2hDbJd6+O/p2UKgcuGLuf 2Ry2oZ4bRY0rAOXt+RcG5gOSCDuJKmDzDmguPcS+Mgtippe3FZPtYI/ov8jTS8WHfkx0 V2GrjXDjxMLpF0RHBkqAM/uyZOow8uCx3gTXnA+PUaAL3kND54isqo/kA6OMlDRfm5xO V0mB5IqAo6aerG/eQBCwGEITkgGZ+X4tCiZkuZOpvxuBhzUMlxsIyo5C1q14PH+n6Ju3 wz5w== X-Gm-Message-State: AOAM530be766ngxLmP3nQ1ryXOXn/iHx3aoT3hHskcql+DL0REAxtDYC uIiuKsby47M63fe6c8Zz40S0P+MwCL9U9A== X-Google-Smtp-Source: ABdhPJyHMtn4B5EwqiLHMWf9G5BqvVikcJTSRJI2KpV/yKtKmYNsU7V2hYr/2ZAL2nqe2p+lvspzlQ== X-Received: by 2002:a05:6512:10d0:: with SMTP id k16mr4161319lfg.530.1632401708273; Thu, 23 Sep 2021 05:55:08 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 04/11] vpci/header: Add and remove register handlers dynamically Date: Thu, 23 Sep 2021 15:54:54 +0300 Message-Id: <20210923125501.234252-5-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add relevant vpci register handlers when assigning PCI device to a domain and remove those when de-assigning. This allows having different handlers for different domains, e.g. hwdom and other guests. Use stubs for guest domains for now. Signed-off-by: Oleksandr Andrushchenko --- Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - simplify some code3. simplify - use gdprintk + error code instead of gprintk - gate vpci_bar_{add|remove}_handlers with CONFIG_HAS_VPCI_GUEST_SUPPORT, so these do not get compiled for x86 - removed unneeded is_system_domain check --- xen/drivers/Kconfig | 4 +++ xen/drivers/vpci/header.c | 72 ++++++++++++++++++++++++++++++++++----- xen/drivers/vpci/vpci.c | 8 +++++ xen/include/xen/vpci.h | 8 +++++ 4 files changed, 84 insertions(+), 8 deletions(-) diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig index db94393f47a6..f3e3a05a4544 100644 --- a/xen/drivers/Kconfig +++ b/xen/drivers/Kconfig @@ -15,4 +15,8 @@ source "drivers/video/Kconfig" config HAS_VPCI bool +config HAS_VPCI_GUEST_SUPPORT + bool + depends on HAS_PCI + endmenu diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 3d571356397a..1ce98795fcca 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -397,6 +397,17 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write32(pdev->sbdf, reg, val); } +static void guest_bar_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ +} + +static uint32_t guest_bar_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + return 0xffffffff; +} + static void rom_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -445,14 +456,25 @@ static void rom_write(const struct pci_dev *pdev, unsigned int reg, rom->addr = val & PCI_ROM_ADDRESS_MASK; } -static int add_bar_handlers(const struct pci_dev *pdev) +static void guest_rom_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ +} + +static uint32_t guest_rom_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + return 0xffffffff; +} + +static int add_bar_handlers(const struct pci_dev *pdev, bool is_hwdom) { unsigned int i; struct vpci_header *header = &pdev->vpci->header; struct vpci_bar *bars = header->bars; int rc; - /* Setup a handler for the command register. */ + /* Setup a handler for the command register: same for hwdom and guests. */ rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, 2, header); if ( rc ) @@ -475,8 +497,13 @@ static int add_bar_handlers(const struct pci_dev *pdev) rom_reg = PCI_ROM_ADDRESS; else rom_reg = PCI_ROM_ADDRESS1; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, - rom_reg, 4, &bars[i]); + if ( is_hwdom ) + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, + rom_reg, 4, &bars[i]); + else + rc = vpci_add_register(pdev->vpci, + guest_rom_read, guest_rom_write, + rom_reg, 4, &bars[i]); if ( rc ) return rc; } @@ -485,8 +512,13 @@ static int add_bar_handlers(const struct pci_dev *pdev) uint8_t reg = PCI_BASE_ADDRESS_0 + i * 4; /* This is either VPCI_BAR_MEM32 or VPCI_BAR_MEM64_{LO|HI}. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); + if ( is_hwdom ) + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, + reg, 4, &bars[i]); + else + rc = vpci_add_register(pdev->vpci, + guest_bar_read, guest_bar_write, + reg, 4, &bars[i]); if ( rc ) return rc; } @@ -520,7 +552,7 @@ static int init_bars(struct pci_dev *pdev) } if ( pdev->ignore_bars ) - return add_bar_handlers(pdev); + return add_bar_handlers(pdev, true); /* Disable memory decoding before sizing. */ cmd = pci_conf_read16(pdev->sbdf, PCI_COMMAND); @@ -582,7 +614,7 @@ static int init_bars(struct pci_dev *pdev) PCI_ROM_ADDRESS_ENABLE; } - rc = add_bar_handlers(pdev); + rc = add_bar_handlers(pdev, true); if ( rc ) { pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); @@ -593,6 +625,30 @@ static int init_bars(struct pci_dev *pdev) } REGISTER_VPCI_INIT(init_bars, VPCI_PRIORITY_MIDDLE); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +int vpci_bar_add_handlers(const struct domain *d, const struct pci_dev *pdev) +{ + int rc; + + /* Remove previously added registers. */ + vpci_remove_device_registers(pdev); + + rc = add_bar_handlers(pdev, is_hardware_domain(d)); + if ( rc ) + gdprintk(XENLOG_ERR, + "%pp: failed to add BAR handlers for dom%pd: %d\n", + &pdev->sbdf, d, rc); + return rc; +} + +int vpci_bar_remove_handlers(const struct domain *d, const struct pci_dev *pdev) +{ + /* Remove previously added registers. */ + vpci_remove_device_registers(pdev); + return 0; +} +#endif + /* * Local variables: * mode: C diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index a8fed3d2c42e..0cdc0c3f75c4 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -94,7 +94,11 @@ int vpci_assign_device(struct domain *d, const struct pci_dev *dev) if ( is_system_domain(d) || !has_vpci(d) ) return 0; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + return vpci_bar_add_handlers(d, dev); +#else return 0; +#endif } /* Notify vPCI that device is de-assigned from guest. */ @@ -104,7 +108,11 @@ int vpci_deassign_device(struct domain *d, const struct pci_dev *dev) if ( is_system_domain(d) || !has_vpci(d) ) return 0; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + return vpci_bar_remove_handlers(d, dev); +#else return 0; +#endif } #endif /* __XEN__ */ diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index b9485b2aea1b..912cbc6aa7ad 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -63,6 +63,14 @@ uint32_t vpci_hw_read32(const struct pci_dev *pdev, unsigned int reg, */ bool __must_check vpci_process_pending(struct vcpu *v); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Add/remove BAR handlers for a domain. */ +int vpci_bar_add_handlers(const struct domain *d, + const struct pci_dev *pdev); +int vpci_bar_remove_handlers(const struct domain *d, + const struct pci_dev *pdev); +#endif + struct vpci { /* List of vPCI handlers for a device. */ struct list_head handlers; From patchwork Thu Sep 23 12:54:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3730C433F5 for ; Thu, 23 Sep 2021 12:58:06 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8E8C861090 for ; Thu, 23 Sep 2021 12:58:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8E8C861090 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194149.345942 (Exim 4.92) (envelope-from ) id 1mTOIY-0006bt-M3; Thu, 23 Sep 2021 12:57:58 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194149.345942; Thu, 23 Sep 2021 12:57:58 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOIY-0006bf-Hc; Thu, 23 Sep 2021 12:57:58 +0000 Received: by outflank-mailman (input) for mailman id 194149; Thu, 23 Sep 2021 12:57:57 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHQ-0004it-6b for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:56:48 +0000 Received: from mail-lf1-x12d.google.com (unknown [2a00:1450:4864:20::12d]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e7f5a879-b865-434e-b8f1-e906b434769c; Thu, 23 Sep 2021 12:55:10 +0000 (UTC) Received: by mail-lf1-x12d.google.com with SMTP id g41so26148306lfv.1 for ; Thu, 23 Sep 2021 05:55:10 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:08 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e7f5a879-b865-434e-b8f1-e906b434769c DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cIPXz2mA5EJS9da7dICK94JpwcF7nJh6i2ljGMdw31k=; b=HeIp6l9OjTuGNCf2NE4aerZ8ypyKK4xUPQnFXvFUXP077VWYVQl01B1SqpfXR79jpp lumKKt3PtFC1SzIMYHOJ1Xg2K84l1AcTFIOxYWPqFqWJjzHGP9TzgWH0/yE4ulu9RSYJ DtMCkXFpokEB7iVvHSiyvLsZau03miegPDqQoXi+dbxgJAyzoJxd0pYptWTney0odSQR 8f5mzPtX2eYXntjJtjpAMmS1a04xUPR9dNMOfpfvvRtPfzI6cFMmvzoeotsNMzl4sCbb TEyTdPUBXZLPe/XpOXC2Gjo/wkpueSH5afjhVEKZRVifB+EMxHeZ5bYVukc7cEAOJaAd kkzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cIPXz2mA5EJS9da7dICK94JpwcF7nJh6i2ljGMdw31k=; b=MlSlWj8Qf8sXRqgSEoaa9ld7qLAzYnja+fM0PiQlP1/mUEgWv6gVeBqoMRZBYQVbKP WqVl9jT0F22Sqwj/1a+8CBW2BEi06XNyFTkMocSOnYPl2l93vys7HWgDO8BCzsDNqqkr 0xasLNJ/LtrRLbtzKkRGiucNfui78K6Ny9p+NRXPXj7XPer5HUpaIQdNcXgszQBc3mry gF8kprtG4tLn3n8vVkB0+057G5QBhyM9BtVVoyMK1pTlLNzfHcqSB5QNaFIFWnNJs4y0 hf7R5ch9YNKoKTuapIe7aVJ+7iw+ZbAW1SbGMcRYHhki79Be0ITrUSP6fNiHYWrLK+Sl /7Qg== X-Gm-Message-State: AOAM532MICJdvEHOJNwSS0Xo+w9fgG2Ie0Ud83DnSMNQz9FSOmZD+rRw 0yoaKXubjGG6oxkKJgH1p4kisz1JEnlYJQ== X-Google-Smtp-Source: ABdhPJzNsCE1gRB70SpWfN2PFcPnO8jYHLcnJzCh/r0yGox61/c0i3oggOazkuEWskoC7iatqYj6qw== X-Received: by 2002:a2e:1508:: with SMTP id s8mr5206646ljd.240.1632401709286; Thu, 23 Sep 2021 05:55:09 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 05/11] vpci/header: Implement guest BAR register handlers Date: Thu, 23 Sep 2021 15:54:55 +0300 Message-Id: <20210923125501.234252-6-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Emulate guest BAR register values: this allows creating a guest view of the registers and emulates size and properties probe as it is done during PCI device enumeration by the guest. ROM BAR is only handled for the hardware domain and for guest domains there is a stub: at the moment PCI expansion ROM is x86 only, so it might not be used by other architectures without emulating x86. Other use-cases may include using that expansion ROM before Xen boots, hence no emulation is needed in Xen itself. Or when a guest wants to use the ROM code which seems to be rare. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- Since v1: - re-work guest read/write to be much simpler and do more work on write than read which is expected to be called more frequently - removed one too obvious comment Signed-off-by: Oleksandr Andrushchenko --- xen/drivers/vpci/header.c | 30 +++++++++++++++++++++++++++++- xen/include/xen/vpci.h | 3 +++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 1ce98795fcca..ec4d215f36ff 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -400,12 +400,38 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg, static void guest_bar_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { + struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + else + { + val &= PCI_BASE_ADDRESS_MEM_MASK; + val |= bar->type == VPCI_BAR_MEM32 ? PCI_BASE_ADDRESS_MEM_TYPE_32 + : PCI_BASE_ADDRESS_MEM_TYPE_64; + val |= bar->prefetchable ? PCI_BASE_ADDRESS_MEM_PREFETCH : 0; + } + + bar->guest_addr &= ~(0xffffffffull << (hi ? 32 : 0)); + bar->guest_addr |= (uint64_t)val << (hi ? 32 : 0); + + bar->guest_addr &= ~(bar->size - 1) | ~PCI_BASE_ADDRESS_MEM_MASK; } static uint32_t guest_bar_read(const struct pci_dev *pdev, unsigned int reg, void *data) { - return 0xffffffff; + const struct vpci_bar *bar = data; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + return bar->guest_addr >> 32; + + return bar->guest_addr; } static void rom_write(const struct pci_dev *pdev, unsigned int reg, @@ -522,6 +548,8 @@ static int add_bar_handlers(const struct pci_dev *pdev, bool is_hwdom) if ( rc ) return rc; } + + bars[i].guest_addr = 0; } return 0; } diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 912cbc6aa7ad..9eaf99f356fe 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -81,7 +81,10 @@ struct vpci { struct vpci_header { /* Information about the PCI BARs of this device. */ struct vpci_bar { + /* Physical view of the BAR. */ uint64_t addr; + /* Guest view of the BAR. */ + uint64_t guest_addr; uint64_t size; enum { VPCI_BAR_EMPTY, From patchwork Thu Sep 23 12:54:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D618C433FE for ; Thu, 23 Sep 2021 12:57:36 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3EC956103D for ; Thu, 23 Sep 2021 12:57:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3EC956103D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194134.345895 (Exim 4.92) (envelope-from ) id 1mTOI4-0003he-SI; Thu, 23 Sep 2021 12:57:28 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194134.345895; Thu, 23 Sep 2021 12:57:28 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOI4-0003g3-Nj; Thu, 23 Sep 2021 12:57:28 +0000 Received: by outflank-mailman (input) for mailman id 194134; Thu, 23 Sep 2021 12:57:26 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHa-0004it-6y for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:56:58 +0000 Received: from mail-lf1-x12c.google.com (unknown [2a00:1450:4864:20::12c]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 26526be8-ef0e-4cb2-b1b7-0fa4db338ccd; Thu, 23 Sep 2021 12:55:13 +0000 (UTC) Received: by mail-lf1-x12c.google.com with SMTP id y28so26182005lfb.0 for ; Thu, 23 Sep 2021 05:55:13 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:10 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 26526be8-ef0e-4cb2-b1b7-0fa4db338ccd DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=omkszF+qjH4yhR2r1KBhAVTg1e2rgb4747WsTYEwl9M=; b=Diu59nZbLtVeysxoliiZuuEQkQgFilWmqQls8NsOhdkmZ5wxSw/zo1goR/KwhzT/xi Lt9GtjdUG3sKql9xvyV4XvyruHnLRqrfEuzQp6dU2uZ4VkJbmbkNJpih1Dxvq/u6PpQM iVnsRf+UWQAcCMeTMPTztOWeW38uk7TY0W0uhva/PJyLmo/PClGeXFc4w+W8ehVd7X0T qvaaE3u7GlSml5X/idcIN0e5cB3GvkHhkB8BzlnWuyvNg7WYz7Ldz1JTPM3cXD9EYmk2 ZpQoQCm6hW2ptx6ahvUKLP2F+S2bhg30587oxYZ+a0qb3Wl0uuARXEpBvVujgCQyQBXm +rjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=omkszF+qjH4yhR2r1KBhAVTg1e2rgb4747WsTYEwl9M=; b=qX6lfOxIjirjQ0SNreYBcXKqyCAURSwEpM1nbqFtcdgqSGHLlb4dsQfRg5YBggKdUe 4ASecVe7YiTt5gt3SJFI6ppG/EdN/30zOhHIGIv1xnluT5n0shUZ9RkgmVJfoIniM8ri Fyz85beMT9udHcQO6J9KMMpz9fcgl8jeQMppAiBnR4s718KmZg6YXbkFa2IKXuV8Nzkp 88TYh1yYoVVNCQE6onSkEq8ouUdtYxumjYrGwdiowFxLk+rbSjZn3lgiL1OBV2kZhlkf VpIWwFIHDRSUwsvx5i1LADwG6IDs5d6Bj/v9txMRdLv08AlXrFoUTHQHPRVJow1IkRFZ A9NA== X-Gm-Message-State: AOAM533M+wKw+RIwrjlac3iyi2yi0A/GX2b5DGh/aOoZeUKjaOX4S4WB zu9/4ruYDoWXG2gFMFgujYFJIoPyGPlt2g== X-Google-Smtp-Source: ABdhPJzXcmkYQUmHLW0CXz3pHHxFBTBXyogaOKmeC1zkP4o3nmiDDwSXr7uy1B0TdVvjqkto2D8sUA== X-Received: by 2002:a05:6512:6cb:: with SMTP id u11mr4034394lff.517.1632401710794; Thu, 23 Sep 2021 05:55:10 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 06/11] vpci/header: Handle p2m range sets per BAR Date: Thu, 23 Sep 2021 15:54:56 +0300 Message-Id: <20210923125501.234252-7-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Instead of handling a single range set, that contains all the memory regions of all the BARs and ROM, have them per BAR. This is in preparation of making non-identity mappings in p2m for the MMIOs/ROM. Signed-off-by: Oleksandr Andrushchenko --- xen/drivers/vpci/header.c | 172 ++++++++++++++++++++++++++------------ xen/include/xen/vpci.h | 3 +- 2 files changed, 122 insertions(+), 53 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index ec4d215f36ff..9c603d26d302 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -131,49 +131,75 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, bool vpci_process_pending(struct vcpu *v) { - if ( v->vpci.mem ) + if ( v->vpci.num_mem_ranges ) { struct map_data data = { .d = v->domain, .map = v->vpci.cmd & PCI_COMMAND_MEMORY, }; - int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data); + struct pci_dev *pdev = v->vpci.pdev; + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; - if ( rc == -ERESTART ) - return true; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + int rc; - spin_lock(&v->vpci.pdev->vpci->lock); - /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, - !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci->lock); + if ( !bar->mem ) + continue; - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; - if ( rc ) - /* - * FIXME: in case of failure remove the device from the domain. - * Note that there might still be leftover mappings. While this is - * safe for Dom0, for DomUs the domain will likely need to be - * killed in order to avoid leaking stale p2m mappings on - * failure. - */ - vpci_remove_device(v->vpci.pdev); + rc = rangeset_consume_ranges(bar->mem, map_range, &data); + + if ( rc == -ERESTART ) + return true; + + spin_lock(&pdev->vpci->lock); + /* Disable memory decoding unconditionally on failure. */ + modify_decoding(pdev, + rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, + !rc && v->vpci.rom_only); + spin_unlock(&pdev->vpci->lock); + + rangeset_destroy(bar->mem); + bar->mem = NULL; + v->vpci.num_mem_ranges--; + if ( rc ) + /* + * FIXME: in case of failure remove the device from the domain. + * Note that there might still be leftover mappings. While this is + * safe for Dom0, for DomUs the domain will likely need to be + * killed in order to avoid leaking stale p2m mappings on + * failure. + */ + vpci_remove_device(pdev); + } } return false; } static int __init apply_map(struct domain *d, const struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd) + uint16_t cmd) { struct map_data data = { .d = d, .map = true }; - int rc; + struct vpci_header *header = &pdev->vpci->header; + int rc = 0; + unsigned int i; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; - while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART ) - process_pending_softirqs(); - rangeset_destroy(mem); + if ( !bar->mem ) + continue; + + while ( (rc = rangeset_consume_ranges(bar->mem, map_range, + &data)) == -ERESTART ) + process_pending_softirqs(); + rangeset_destroy(bar->mem); + bar->mem = NULL; + } if ( !rc ) modify_decoding(pdev, cmd, false); @@ -181,7 +207,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, } static void defer_map(struct domain *d, struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd, bool rom_only) + uint16_t cmd, bool rom_only, uint8_t num_mem_ranges) { struct vcpu *curr = current; @@ -192,9 +218,9 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, * started for the same device if the domain is not well-behaved. */ curr->vpci.pdev = pdev; - curr->vpci.mem = mem; curr->vpci.cmd = cmd; curr->vpci.rom_only = rom_only; + curr->vpci.num_mem_ranges = num_mem_ranges; /* * Raise a scheduler softirq in order to prevent the guest from resuming * execution with pending mapping operations, to trigger the invocation @@ -206,42 +232,47 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) { struct vpci_header *header = &pdev->vpci->header; - struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; const struct vpci_msix *msix = pdev->vpci->msix; - unsigned int i; + unsigned int i, j; int rc; - - if ( !mem ) - return -ENOMEM; + uint8_t num_mem_ranges; /* - * Create a rangeset that represents the current device BARs memory region + * Create a rangeset per BAR that represents the current device memory region * and compare it against all the currently active BAR memory regions. If * an overlap is found, subtract it from the region to be mapped/unmapped. * - * First fill the rangeset with all the BARs of this device or with the ROM + * First fill the rangesets with all the BARs of this device or with the ROM * BAR only, depending on whether the guest is toggling the memory decode * bit of the command register, or the enable bit of the ROM BAR register. */ for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - const struct vpci_bar *bar = &header->bars[i]; + struct vpci_bar *bar = &header->bars[i]; unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + bar->mem = NULL; + if ( !MAPPABLE_BAR(bar) || (rom_only ? bar->type != VPCI_BAR_ROM : (bar->type == VPCI_BAR_ROM && !header->rom_enabled)) ) continue; - rc = rangeset_add_range(mem, start, end); + bar->mem = rangeset_new(NULL, NULL, 0); + if ( !bar->mem ) + { + rc = -ENOMEM; + goto fail; + } + + rc = rangeset_add_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to add [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } @@ -252,14 +283,21 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long end = PFN_DOWN(vmsix_table_addr(pdev->vpci, i) + vmsix_table_size(pdev->vpci, i) - 1); - rc = rangeset_remove_range(mem, start, end); - if ( rc ) + for ( j = 0; j < ARRAY_SIZE(header->bars); j++ ) { - printk(XENLOG_G_WARNING - "Failed to remove MSIX table [%lx, %lx]: %d\n", - start, end, rc); - rangeset_destroy(mem); - return rc; + const struct vpci_bar *bar = &header->bars[j]; + + if ( !bar->mem ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove MSIX table [%lx, %lx]: %d\n", + start, end, rc); + goto fail; + } } } @@ -291,7 +329,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); - if ( !bar->enabled || !rangeset_overlaps_range(mem, start, end) || + if ( !bar->enabled || + !rangeset_overlaps_range(bar->mem, start, end) || /* * If only the ROM enable bit is toggled check against other * BARs in the same device for overlaps, but not against the @@ -300,13 +339,12 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) continue; - rc = rangeset_remove_range(mem, start, end); + rc = rangeset_remove_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } } @@ -324,12 +362,42 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) * will always be to establish mappings and process all the BARs. */ ASSERT((cmd & PCI_COMMAND_MEMORY) && !rom_only); - return apply_map(pdev->domain, pdev, mem, cmd); + return apply_map(pdev->domain, pdev, cmd); } - defer_map(dev->domain, dev, mem, cmd, rom_only); + /* Find out how many memory ranges has left after MSI and overlaps. */ + num_mem_ranges = 0; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + + if ( !rangeset_is_empty(bar->mem) ) + num_mem_ranges++; + } + + /* + * There are cases when PCI device, root port for example, has neither + * memory space nor IO. In this case PCI command register write is + * missed resulting in the underlying PCI device not functional, so: + * - if there are no regions write the command register now + * - if there are regions then defer work and write later on + */ + if ( !num_mem_ranges ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + else + defer_map(dev->domain, dev, cmd, rom_only, num_mem_ranges); return 0; + +fail: + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + + rangeset_destroy(bar->mem); + bar->mem = NULL; + } + return rc; } static void cmd_write(const struct pci_dev *pdev, unsigned int reg, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 9eaf99f356fe..3696c73a4237 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -86,6 +86,7 @@ struct vpci { /* Guest view of the BAR. */ uint64_t guest_addr; uint64_t size; + struct rangeset *mem; enum { VPCI_BAR_EMPTY, VPCI_BAR_IO, @@ -160,9 +161,9 @@ struct vpci { struct vpci_vcpu { /* Per-vcpu structure to store state while {un}mapping of PCI BARs. */ - struct rangeset *mem; struct pci_dev *pdev; uint16_t cmd; + uint8_t num_mem_ranges; bool rom_only : 1; }; From patchwork Thu Sep 23 12:54:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0568C433F5 for ; Thu, 23 Sep 2021 12:57:09 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 59AAA60F43 for ; Thu, 23 Sep 2021 12:57:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 59AAA60F43 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194122.345843 (Exim 4.92) (envelope-from ) id 1mTOHd-0001ez-Qh; Thu, 23 Sep 2021 12:57:01 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194122.345843; Thu, 23 Sep 2021 12:57:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHd-0001es-ND; Thu, 23 Sep 2021 12:57:01 +0000 Received: by outflank-mailman (input) for mailman id 194122; Thu, 23 Sep 2021 12:57:00 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHV-0004it-6j for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:56:53 +0000 Received: from mail-lf1-x12d.google.com (unknown [2a00:1450:4864:20::12d]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5945c790-8869-4b99-afdc-a6c7733b4bcc; Thu, 23 Sep 2021 12:55:13 +0000 (UTC) Received: by mail-lf1-x12d.google.com with SMTP id z24so26712439lfu.13 for ; Thu, 23 Sep 2021 05:55:13 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:11 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 5945c790-8869-4b99-afdc-a6c7733b4bcc DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/VZhQBHA/qWsZnQaFLYfN7XFH1H4UA39toP1GBs37TM=; b=P8Hj9IuBQQDngliolaPP7ZmvBgMYQvbZ3Apn/ouQ8J5yPra/sFgxauc0LD/tzbp8wo rQVHv5c8j9RWxG5p+ChF6f6tuID8Q5zvstZjZXEgtkX07RIVDTs+GA7+lz++OaiMcLhi IFjpzJZaoO3dkvtRJMTIhAmVdMP2OA+9A/SwKLgTT56y7vVw7+JpwsebBeCIWEfcb1hD M/puTGIUrPojnOjh1Pv8I8JEqW926nKjvKg3bpiEvoid8ZqJ+/nD/+SNZGC+xtHx2zhb PUk7NxVks2afQC+Z3eZ0AeqP4bhyZ5n3O66dDvY+S8vQsxsiz6I0/wUTjrG+70nJjuNw ZCyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/VZhQBHA/qWsZnQaFLYfN7XFH1H4UA39toP1GBs37TM=; b=JmCcsFNyCR1+yeqn4tUT48yvyQ6/eojjMBv74n7WWxb2axj3GbIw8LTWCrqQT7rBz2 kY3IUSr0IblsBKKuvSkkxs/bpdmr7WRNt7ii9X5dS9vn1pIidADsMTZxOFp+Yq+Jb2Ja sC/n7WSm0ucGzH/pvs4dNZ+X5SUfG8MSZYAaboGRED3lUdW8UdHyj9PxjBWfZuOzNBK2 9duN9CI5AgiZoU9/ZBInGoM3EnusrmEYLKVhkU6Ob8NODqkFayMmIw/4/xF4bF7Px12f E6wcacS3lwvD3tBduD39I/sk8X8fp3XhBOofJFOPKQYxJQtWgXiY30/AHTtfV1RYBAcj +/og== X-Gm-Message-State: AOAM533QoIhod2VbBZrkh0F8oAk/l1evunRrcJbxJTN7rjtqnVj1IhYv BXNJLWZea9MKEJDIEw2WWPaUhjvxW5snYw== X-Google-Smtp-Source: ABdhPJxVZKYeKANBbtQt4yOeYPS1abKifRIc3+NhdRNVbgxW2Y/9Ry0MALLSfk/9SB4lbTL6En8Icg== X-Received: by 2002:a05:6512:110a:: with SMTP id l10mr4093541lfg.550.1632401711887; Thu, 23 Sep 2021 05:55:11 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 07/11] vpci/header: program p2m with guest BAR view Date: Thu, 23 Sep 2021 15:54:57 +0300 Message-Id: <20210923125501.234252-8-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value as set up by the host bridge in the hardware domain. This way hardware doamin sees physical BAR values and guest sees emulated ones. Signed-off-by: Oleksandr Andrushchenko --- Since v1: - s/MSI/MSI-X in comments --- xen/drivers/vpci/header.c | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 9c603d26d302..bdd18599b205 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -30,6 +30,10 @@ struct map_data { struct domain *d; + /* Start address of the BAR as seen by the guest. */ + gfn_t start_gfn; + /* Physical start address of the BAR. */ + mfn_t start_mfn; bool map; }; @@ -37,12 +41,28 @@ static int map_range(unsigned long s, unsigned long e, void *data, unsigned long *c) { const struct map_data *map = data; + gfn_t start_gfn; int rc; for ( ; ; ) { unsigned long size = e - s + 1; + /* + * Any BAR may have holes in its memory we want to map, e.g. + * we don't want to map MSI-X regions which may be a part of that BAR, + * e.g. when a single BAR is used for both MMIO and MSI-X. + * In this case MSI-X regions are subtracted from the mapping, but + * map->start_gfn still points to the very beginning of the BAR. + * So if there is a hole present then we need to adjust start_gfn + * to reflect the fact of that substraction. + */ + start_gfn = gfn_add(map->start_gfn, s - mfn_x(map->start_mfn)); + + printk(XENLOG_G_DEBUG + "%smap [%lx, %lx] -> %#"PRI_gfn" for d%d\n", + map->map ? "" : "un", s, e, gfn_x(start_gfn), + map->d->domain_id); /* * ARM TODOs: * - On ARM whether the memory is prefetchable or not should be passed @@ -52,8 +72,10 @@ static int map_range(unsigned long s, unsigned long e, void *data, * - {un}map_mmio_regions doesn't support preemption. */ - rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, _mfn(s)) - : unmap_mmio_regions(map->d, _gfn(s), size, _mfn(s)); + rc = map->map ? map_mmio_regions(map->d, start_gfn, + size, _mfn(s)) + : unmap_mmio_regions(map->d, start_gfn, + size, _mfn(s)); if ( rc == 0 ) { *c += size; @@ -69,6 +91,7 @@ static int map_range(unsigned long s, unsigned long e, void *data, ASSERT(rc < size); *c += rc; s += rc; + gfn_add(map->start_gfn, rc); if ( general_preempt_check() ) return -ERESTART; } @@ -149,6 +172,10 @@ bool vpci_process_pending(struct vcpu *v) if ( !bar->mem ) continue; + data.start_gfn = is_hardware_domain(v->vpci.pdev->domain) ? + _gfn(PFN_DOWN(bar->addr)) : + _gfn(PFN_DOWN(bar->guest_addr)); + data.start_mfn = _mfn(PFN_DOWN(bar->addr)); rc = rangeset_consume_ranges(bar->mem, map_range, &data); if ( rc == -ERESTART ) @@ -194,6 +221,10 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, if ( !bar->mem ) continue; + data.start_gfn = is_hardware_domain(d) ? + _gfn(PFN_DOWN(bar->addr)) : + _gfn(PFN_DOWN(bar->guest_addr)); + data.start_mfn = _mfn(PFN_DOWN(bar->addr)); while ( (rc = rangeset_consume_ranges(bar->mem, map_range, &data)) == -ERESTART ) process_pending_softirqs(); From patchwork Thu Sep 23 12:54:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53621C433F5 for ; Thu, 23 Sep 2021 13:07:35 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1C3BB61090 for ; Thu, 23 Sep 2021 13:07:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1C3BB61090 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194181.345986 (Exim 4.92) (envelope-from ) id 1mTORj-0002YV-IA; Thu, 23 Sep 2021 13:07:27 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194181.345986; Thu, 23 Sep 2021 13:07:27 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTORj-0002YO-ED; Thu, 23 Sep 2021 13:07:27 +0000 Received: by outflank-mailman (input) for mailman id 194181; Thu, 23 Sep 2021 13:07:26 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHp-0004it-7d for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:57:13 +0000 Received: from mail-lf1-x136.google.com (unknown [2a00:1450:4864:20::136]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id af9177de-3a8d-49c1-bd4a-675a79d72820; Thu, 23 Sep 2021 12:55:14 +0000 (UTC) Received: by mail-lf1-x136.google.com with SMTP id g41so26148978lfv.1 for ; Thu, 23 Sep 2021 05:55:14 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:12 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: af9177de-3a8d-49c1-bd4a-675a79d72820 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fpEmkp363zGTNG/sREpOTD2+RNFlDcJmlTvIa8Vah1w=; b=bpnFyr4vGY6stFAtsMo9qX7ZkqJW7256vkP2otZyANDs5R1Qyh53a03oaQJiIO9Tfi VADZFcegMz3d43HptXQRRBCE9Gxx0vMyS8AniNFVc6xLaJS/vx7Kr/+07sfBFJGV6Xpy LWMfQ+PUwmyqY1X4GEDMUqma1EjVbZWsOyffYXaXWsC3eFY2Q3Tjq5sZYjCYc7l44EbK +BEtyp5dq7r4QF+GRr9FVsKAJNxYFta58KVMQ46xAP7fQOZS6a7hNPN2rK1zMl8tkN0i jQnXs/gd5/oBSDCzcsb/stq18qcDugzMPAdsbGVksToFDkWLkYEExgyQFAkWHJhxMcTk 0iYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fpEmkp363zGTNG/sREpOTD2+RNFlDcJmlTvIa8Vah1w=; b=XNnh9CH91uvB4PJSWypfv+a04zGMlc4T0gpFr0UW/wldx8mHS8w+AfqnBoNeEld63T FkMAWn85cOS+DWTIt/qDxow0sTpgNhiNo1ISDEyDaugaJIDn5iQQYEZhr55tR9ASs6X0 YHYKxYgiIky21a+m90TD+uwgD0ZP/fcn5c/94Ff+ipXGvsiTasr+p15aq+E+A7AbaNtd WQ5OjQE37XOAyDcKzvvCjCbHzhFshhHDfsw+IAoWOpiaH3H2GWo+SGm4rX92idwe+fEn Ad60xkX8w5NC4lrVrVUwgmgecjVWElRnVd2BzQRFjDEbKyM5kzvaXrn9oM8KKPgIz2R9 Z0vw== X-Gm-Message-State: AOAM533t8YI7tTjNk9C3SUk3nz/tlb3MMoGhftBhiAuFhRKfAYti2fex DzEdRyooIXWUDc8uKlfJWcbMqWmt9HcHiQ== X-Google-Smtp-Source: ABdhPJzdYDxTei6IVbz6wkAAT9SIcXwlU/2hhWG9paV9ZKyig0hoON0I5dQvzIu3rLdWmGK1j+rN+w== X-Received: by 2002:a05:6512:32c6:: with SMTP id f6mr3953415lfg.503.1632401713136; Thu, 23 Sep 2021 05:55:13 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 08/11] vpci/header: Emulate PCI_COMMAND register for guests Date: Thu, 23 Sep 2021 15:54:58 +0300 Message-Id: <20210923125501.234252-9-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add basic emulation support for guests. At the moment only emulate PCI_COMMAND_INTX_DISABLE bit, the rest is not emulated yet and left as TODO. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- New in v2 --- xen/drivers/vpci/header.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index bdd18599b205..99f9c37dfb00 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -452,6 +452,32 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + /* TODO: Add proper emulation for all bits of the command register. */ + + if ( (cmd & PCI_COMMAND_INTX_DISABLE) == 0 ) + { + /* + * Guest wants to enable INTx. It can't be enabled if: + * - host has INTx disabled + * - MSI/MSI-X enabled + */ + if ( pdev->vpci->msi->enabled ) + cmd |= PCI_COMMAND_INTX_DISABLE; + else + { + uint16_t current_cmd = pci_conf_read16(pdev->sbdf, reg); + + if ( current_cmd & PCI_COMMAND_INTX_DISABLE ) + cmd |= PCI_COMMAND_INTX_DISABLE; + } + } + + cmd_write(pdev, reg, cmd, data); +} + static void bar_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -599,9 +625,12 @@ static int add_bar_handlers(const struct pci_dev *pdev, bool is_hwdom) struct vpci_bar *bars = header->bars; int rc; - /* Setup a handler for the command register: same for hwdom and guests. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); + if ( is_hwdom ) + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, + PCI_COMMAND, 2, header); + else + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, guest_cmd_write, + PCI_COMMAND, 2, header); if ( rc ) return rc; From patchwork Thu Sep 23 12:54:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B754C433EF for ; Thu, 23 Sep 2021 13:08:42 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E6D396103D for ; Thu, 23 Sep 2021 13:08:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E6D396103D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194208.346008 (Exim 4.92) (envelope-from ) id 1mTOSm-0003pP-3E; Thu, 23 Sep 2021 13:08:32 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194208.346008; Thu, 23 Sep 2021 13:08:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOSm-0003pI-00; Thu, 23 Sep 2021 13:08:32 +0000 Received: by outflank-mailman (input) for mailman id 194208; Thu, 23 Sep 2021 13:08:30 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHz-0004it-87 for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:57:23 +0000 Received: from mail-lf1-x131.google.com (unknown [2a00:1450:4864:20::131]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id dde5d022-867a-48eb-9e7e-8b40d99586ad; Thu, 23 Sep 2021 12:55:17 +0000 (UTC) Received: by mail-lf1-x131.google.com with SMTP id b15so25738434lfe.7 for ; Thu, 23 Sep 2021 05:55:17 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:13 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: dde5d022-867a-48eb-9e7e-8b40d99586ad DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t+TG3ZsjniyYOn1TjIrSDa/ZTsIEIfPrt6I6dPYbhyk=; b=eV8YVH5NC4DmSL4x9/RAgitTwlfCmkQPMgFjPMCQK0n3AeZxwnSn+Udn3ubor53chc UEnrd9Smm1404yKVVdaLjy3hOJiWBs7T/rVlalmkXg8d5F1EoxTvtMek+bFqTzq7ufkJ Ydn5qa0Q1Oxs7w9E3XPmEQi5d5dBjhmPZfK3TsooOGEDddIb3G3she0IOltWH9l+WSHp BoZAPk6yFF9zp3nTpBJacx+xWWVLPo3Lt6e8BYc8qB1gr7SsEU4ljwiOFgqvyHmb9Leo VFFmKbTyuoA/iNCivDgXGHSYRXDzRaeZbBxNiOBI6RvCi+2vKVJbQWuifzkPHZXRqUEv jw/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t+TG3ZsjniyYOn1TjIrSDa/ZTsIEIfPrt6I6dPYbhyk=; b=a4/HE7eoZgMpj0aWHKgbDfO6ZtTPItnWKwsbJHvbMH45Q6R0cJlyjXAt+x1R+40x0b x59R7FZAfaSfjNbw7QCO5xsCIkOoBBNmei47TleChP1Kvtye2uOy/QykAmKzaKb1DMK5 JZWdVDJQIb4a/tr6bmRmC/Zg4mOichC8cKiZIpuwKwfQComQbzeP4vq4o7Ax3Fn3oOIu IdUHAxe4LO0wl4MV0F+HXFDQT5WeNEntgUlAQ7l2jygd6KkoTYtsE0oDoJ7TFgQzKl+y myz5c63ww7C4SgbDcifSbDVNkUJJUdTc4JZ175psdAXDt0rJigyBp9HNwcVAwJ94p3vx IDdw== X-Gm-Message-State: AOAM530xdypUlPNi81Zzfcc264FQKMSX2UEoFBFTVHR/y2k++2zUxPSW GNSakt2H/UpebwyaZIRsXK4MeGIOKAVgfA== X-Google-Smtp-Source: ABdhPJyeEpTK6x6nOoWHlX/qYeM4XHS0Q00/ZB2U8VKbmN/QGEVH32sZonpdDV4deZk47lwatxUBhA== X-Received: by 2002:a2e:bd03:: with SMTP id n3mr5073432ljq.359.1632401714161; Thu, 23 Sep 2021 05:55:14 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 09/11] vpci/header: Reset the command register when adding devices Date: Thu, 23 Sep 2021 15:54:59 +0300 Message-Id: <20210923125501.234252-10-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Reset the command register when passing through a PCI device: it is possible that when passing through a PCI device its memory decoding bits in the command register are already set. Thus, a guest OS may not write to the command register to update memory decoding, so guest mappings (guest's view of the BARs) are left not updated. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- Since v1: - do not write 0 to the command register, but respect host settings. --- xen/drivers/vpci/header.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 99f9c37dfb00..b2829b9d206b 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -452,8 +452,7 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } -static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, - uint32_t cmd, void *data) +static uint32_t emulate_cmd_reg(const struct pci_dev *pdev, uint32_t cmd) { /* TODO: Add proper emulation for all bits of the command register. */ @@ -468,14 +467,20 @@ static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, cmd |= PCI_COMMAND_INTX_DISABLE; else { - uint16_t current_cmd = pci_conf_read16(pdev->sbdf, reg); + uint16_t current_cmd = pci_conf_read16(pdev->sbdf, PCI_COMMAND); if ( current_cmd & PCI_COMMAND_INTX_DISABLE ) cmd |= PCI_COMMAND_INTX_DISABLE; } } - cmd_write(pdev, reg, cmd, data); + return cmd; +} + +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + cmd_write(pdev, reg, emulate_cmd_reg(pdev, cmd), data); } static void bar_write(const struct pci_dev *pdev, unsigned int reg, @@ -794,6 +799,10 @@ int vpci_bar_add_handlers(const struct domain *d, const struct pci_dev *pdev) gdprintk(XENLOG_ERR, "%pp: failed to add BAR handlers for dom%pd: %d\n", &pdev->sbdf, d, rc); + + /* Reset the command register with respect to host settings. */ + pci_conf_write16(pdev->sbdf, PCI_COMMAND, emulate_cmd_reg(pdev, 0)); + return rc; } From patchwork Thu Sep 23 12:55:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5039C433F5 for ; Thu, 23 Sep 2021 13:08:46 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 66C8E60F39 for ; Thu, 23 Sep 2021 13:08:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 66C8E60F39 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194212.346030 (Exim 4.92) (envelope-from ) id 1mTOSr-0004VK-Rs; Thu, 23 Sep 2021 13:08:37 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194212.346030; Thu, 23 Sep 2021 13:08:37 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOSr-0004VD-NJ; Thu, 23 Sep 2021 13:08:37 +0000 Received: by outflank-mailman (input) for mailman id 194212; Thu, 23 Sep 2021 13:08:36 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOHu-0004it-7y for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:57:18 +0000 Received: from mail-lf1-x12e.google.com (unknown [2a00:1450:4864:20::12e]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id bf96aaab-a758-4d13-82cb-0f9b9f285ee2; Thu, 23 Sep 2021 12:55:16 +0000 (UTC) Received: by mail-lf1-x12e.google.com with SMTP id b20so26425942lfv.3 for ; Thu, 23 Sep 2021 05:55:16 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:14 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: bf96aaab-a758-4d13-82cb-0f9b9f285ee2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=01d6SaTi/ep3sxri+AlZsSZXy8XnKOJzH2RZEoLRhnE=; b=EWpEK98V3gn+BXUIniS4aeCA9MHYJ+uraGzPRHWI6TQV1B/vOoZeupB+37cr+IGN5l ecgShTHEtU2cFr68IVOvI7ZjrPH0mELx7q5M2RfFA8RxYf62iGZGZgB4d9huCMUKwJfI rGF75wr4M7zdy8Vi/3omiimpqMmoiPB++DcA182+C+cCOS5SNWsaGte1VMc4ISIdMYeU rvNSAtt5UvAYcglcHlgkOy6E/D+dEtx4joYE3SnnrHq3wzNfOPDbQQ5ep1vQAXvSRdWv ZwRm1UayVuL8hQeFccvHM3BPFjlSdpDw6ZPFeeWU6FplECWsRyCoua3q5zY1abpy29BS TyoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=01d6SaTi/ep3sxri+AlZsSZXy8XnKOJzH2RZEoLRhnE=; b=nlcDXV1u1cnc+4YFn2ba3tN+MGMyo7eUUp1AuyM5rohwKnRgY+7oW0X6RtD5eUJ7+e AqYMP9BBVaOlTKaGkRt4R54QFKzSbbNwPwpicQTodRd7BiSTQXxLV5kzi8lpDOm6qwkQ ye4PgQrSTQHy5gaIOcl/x+3T3bQi75+LN/sx/Ug7N1k8u6vrZ/mH1FbfnKjfcOx3Xfzx Fb5FwOgfOUnzqqAtUPZTyHdzcWB3sWimRMyKFLrSSLZrk/sAeVS1mm0kTRfM2kG4D+hx EQtoqiFas0VSEZSQqJBoWg5wSrk/+sAqHLBv5mWR5pZMJoQtcyDXc43gDsYdbIAPJnt4 K88w== X-Gm-Message-State: AOAM530ziUhSgaeuaJ6B9MN3eXhcEwn5DSVjiDSnr4jS0ZrYc0ctve/1 zZt8XsdKY4miS6JmE6ymVZBk41ACHtlJ2g== X-Google-Smtp-Source: ABdhPJx3puHxJXs6dd2LzzWCWWYJzbY4YRnuivwUaxA1/mtGOwJpGs6ZnDTtx7CE1RfoyWW9sphKRw== X-Received: by 2002:a2e:6e08:: with SMTP id j8mr4786466ljc.247.1632401715286; Thu, 23 Sep 2021 05:55:15 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 10/11] vpci: Add initial support for virtual PCI bus topology Date: Thu, 23 Sep 2021 15:55:00 +0300 Message-Id: <20210923125501.234252-11-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Michal Orzel --- New in v2 --- xen/common/domain.c | 1 + xen/drivers/passthrough/pci.c | 57 +++++++++++++++++++++++++++++++++++ xen/drivers/vpci/vpci.c | 18 +++++++++-- xen/include/xen/pci.h | 18 +++++++++++ xen/include/xen/sched.h | 6 ++++ 5 files changed, 97 insertions(+), 3 deletions(-) diff --git a/xen/common/domain.c b/xen/common/domain.c index 40d67ec34232..b80ff2e5e2e6 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -601,6 +601,7 @@ struct domain *domain_create(domid_t domid, #ifdef CONFIG_HAS_PCI INIT_LIST_HEAD(&d->pdev_list); + INIT_LIST_HEAD(&d->vdev_list); #endif /* All error paths can depend on the above setup. */ diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index e1da283d73ad..4552ace855e0 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -833,6 +833,63 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) return ret; } +static struct vpci_dev *pci_find_virtual_device(const struct domain *d, + const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + list_for_each_entry ( vdev, &d->vdev_list, list ) + if ( vdev->pdev == pdev ) + return vdev; + return NULL; +} + +int pci_add_virtual_device(struct domain *d, const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + ASSERT(!pci_find_virtual_device(d, pdev)); + + /* Each PCI bus supports 32 devices/slots at max. */ + if ( d->vpci_dev_next > 31 ) + return -ENOSPC; + + vdev = xzalloc(struct vpci_dev); + if ( !vdev ) + return -ENOMEM; + + /* We emulate a single host bridge for the guest, so segment is always 0. */ + *(u16*) &vdev->seg = 0; + /* + * The bus number is set to 0, so virtual devices are seen + * as embedded endpoints behind the root complex. + */ + *((u8*) &vdev->bus) = 0; + *((u8*) &vdev->devfn) = PCI_DEVFN(d->vpci_dev_next++, 0); + + vdev->pdev = pdev; + vdev->domain = d; + + pcidevs_lock(); + list_add_tail(&vdev->list, &d->vdev_list); + pcidevs_unlock(); + + return 0; +} + +int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev) +{ + struct vpci_dev *vdev; + + pcidevs_lock(); + vdev = pci_find_virtual_device(d, pdev); + if ( vdev ) + list_del(&vdev->list); + pcidevs_unlock(); + xfree(vdev); + return 0; +} + /* Caller should hold the pcidevs_lock */ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, uint8_t devfn) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 0cdc0c3f75c4..a40a138a14f7 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -90,24 +90,36 @@ int __hwdom_init vpci_add_handlers(struct pci_dev *pdev) /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, const struct pci_dev *dev) { +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + int rc; +#endif + /* It only makes sense to assign for hwdom or guest domain. */ if ( is_system_domain(d) || !has_vpci(d) ) return 0; #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT - return vpci_bar_add_handlers(d, dev); -#else - return 0; + rc = vpci_bar_add_handlers(d, dev); + if ( rc ) + return rc; #endif + + return pci_add_virtual_device(d, dev); } /* Notify vPCI that device is de-assigned from guest. */ int vpci_deassign_device(struct domain *d, const struct pci_dev *dev) { + int rc; + /* It only makes sense to de-assign from hwdom or guest domain. */ if ( is_system_domain(d) || !has_vpci(d) ) return 0; + rc = pci_remove_virtual_device(d, dev); + if ( rc ) + return rc; + #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT return vpci_bar_remove_handlers(d, dev); #else diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 2b2dfb6f1b49..35ae1d093921 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -136,6 +136,22 @@ struct pci_dev { struct vpci *vpci; }; +struct vpci_dev { + struct list_head list; + /* Physical PCI device this virtual device is connected to. */ + const struct pci_dev *pdev; + /* Virtual SBDF of the device. */ + const union { + struct { + uint8_t devfn; + uint8_t bus; + uint16_t seg; + }; + pci_sbdf_t sbdf; + }; + struct domain *domain; +}; + #define for_each_pdev(domain, pdev) \ list_for_each_entry(pdev, &(domain)->pdev_list, domain_list) @@ -165,7 +181,9 @@ int pci_add_segment(u16 seg); const unsigned long *pci_get_ro_map(u16 seg); int pci_add_device(u16 seg, u8 bus, u8 devfn, const struct pci_dev_info *, nodeid_t node); +int pci_add_virtual_device(struct domain *d, const struct pci_dev *pdev); int pci_remove_device(u16 seg, u8 bus, u8 devfn); +int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev); int pci_ro_device(int seg, int bus, int devfn); int pci_hide_device(unsigned int seg, unsigned int bus, unsigned int devfn); struct pci_dev *pci_get_pdev(int seg, int bus, int devfn); diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 28146ee404e6..d304c7ebe766 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -444,6 +444,12 @@ struct domain #ifdef CONFIG_HAS_PCI struct list_head pdev_list; + struct list_head vdev_list; + /* + * Current device number used by the virtual PCI bus topology + * to assign a unique SBDF to a passed through virtual PCI device. + */ + int vpci_dev_next; #endif #ifdef CONFIG_HAS_PASSTHROUGH From patchwork Thu Sep 23 12:55:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12512851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 306B8C433FE for ; Thu, 23 Sep 2021 13:08:43 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D38A36103D for ; Thu, 23 Sep 2021 13:08:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D38A36103D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.194209.346019 (Exim 4.92) (envelope-from ) id 1mTOSn-00047j-CB; Thu, 23 Sep 2021 13:08:33 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 194209.346019; Thu, 23 Sep 2021 13:08:33 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOSn-00047Z-7h; Thu, 23 Sep 2021 13:08:33 +0000 Received: by outflank-mailman (input) for mailman id 194209; Thu, 23 Sep 2021 13:08:31 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mTOI4-0004it-89 for xen-devel@lists.xenproject.org; Thu, 23 Sep 2021 12:57:28 +0000 Received: from mail-lf1-x132.google.com (unknown [2a00:1450:4864:20::132]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id c1ab318b-7c38-43f1-8007-8693397c9763; Thu, 23 Sep 2021 12:55:17 +0000 (UTC) Received: by mail-lf1-x132.google.com with SMTP id i25so26496283lfg.6 for ; Thu, 23 Sep 2021 05:55:17 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id o12sm453010lft.254.2021.09.23.05.55.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Sep 2021 05:55:15 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: c1ab318b-7c38-43f1-8007-8693397c9763 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XlpFRu8hRrvPpjMFenpSAJbBdeavkULfWEH3e9kj/3o=; b=pn/1WK9cTRCVMT8S8N+kHcScv8SVfp4WAmQl55y/V20VbgiCkjI6Mk421HuVf6HD5d gDj6lasHPMcFqrunqkhsT24D1oJD0PfKOWmQLaRUOyNed5XAR7hMxyJZ/puHpCBQKzlq GeA2LA9d28oc5bmqWILcBA1L3xPM08dTwe5eVd/3dhpR6mLVE9XbtOzCp18JDAg19kK1 MSTJNsM0HppMoi6zI5afgpVAVQlaOc97sFuV7yj5jvHo+w6X6PoZrUhXvz6jZn30jTAq PrhFFr4qfbF33EQv2Xff6GNBP7lYEX2WiZq1luKuv6c9ezife0E9U3tQnXqFF0k7uy+G dwZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XlpFRu8hRrvPpjMFenpSAJbBdeavkULfWEH3e9kj/3o=; b=yNttErMUSD8q+Gz2bHDy7TpdMhaYjVpufRncBVD7FfZN9qlUsfLeOAUg46bYOb3Jcw Mh2CyJwL2mAVahntilEtfs/oMrkjbC/iKBU+ymTFgO7oznIEGwuUEropsA7qFkow/dnF 4mrYIEsAzQpKK2hjaDFroPbqT1K2yWI2MlQEDtnazlTpioLDuYAnvUm5QT6w0Og8i3DR 9vAGuQgU9ZU6QrKhTlIYFjpJ/GUroR7vBFUGNH/pFLN3k7cCNj7aOhk+jj/jh49qyMVc +am2ZBN7MG6Ku08+BZb1gSF+4INDd5QX8lfz/Qo/4dXt4KBWzhpgd5PIH7nr49E9MRxe /u2w== X-Gm-Message-State: AOAM530VvNIpAoQJabylMuoyRopAMu5aZ0RoegxAQDLf9p2EyhGoiD1k ThFQ7IZWx25pyeHg3rUsqlTl+ixkQl0mpg== X-Google-Smtp-Source: ABdhPJx2iIC0KBtVCRSDDJlP12bf3jCvWEtBjbBIXUe+n2z2VKx+YKSykuDGyFp3BrRnSXWUp8dwoA== X-Received: by 2002:a05:6512:1151:: with SMTP id m17mr4030772lfg.144.1632401716421; Thu, 23 Sep 2021 05:55:16 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v2 11/11] xen/arm: Translate virtual PCI bus topology for guests Date: Thu, 23 Sep 2021 15:55:01 +0300 Message-Id: <20210923125501.234252-12-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210923125501.234252-1-andr2000@gmail.com> References: <20210923125501.234252-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are three originators for the PCI configuration space access: 1. The domain that owns physical host bridge: MMIO handlers are there so we can update vPCI register handlers with the values written by the hardware domain, e.g. physical view of the registers vs guest's view on the configuration space. 2. Guest access to the passed through PCI devices: we need to properly map virtual bus topology to the physical one, e.g. pass the configuration space access to the corresponding physical devices. 3. Emulated host PCI bridge access. It doesn't exist in the physical topology, e.g. it can't be mapped to some physical host bridge. So, all access to the host bridge itself needs to be trapped and emulated. Signed-off-by: Oleksandr Andrushchenko --- New in v2 --- xen/arch/arm/domain.c | 1 + xen/arch/arm/vpci.c | 87 +++++++++++++++++++++++++++++++---- xen/arch/arm/vpci.h | 3 ++ xen/drivers/passthrough/pci.c | 25 ++++++++++ xen/include/asm-arm/pci.h | 1 + xen/include/xen/pci.h | 1 + xen/include/xen/sched.h | 2 + 7 files changed, 111 insertions(+), 9 deletions(-) diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index c7b25bc70439..c0ad6ad682d2 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -797,6 +797,7 @@ void arch_domain_destroy(struct domain *d) get_order_from_bytes(d->arch.efi_acpi_len)); #endif domain_io_free(d); + domain_vpci_free(d); } void arch_domain_shutdown(struct domain *d) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 14947e975d69..012f958960d1 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -17,6 +17,14 @@ #define REGISTER_OFFSET(addr) ( (addr) & 0x00000fff) +struct vpci_mmio_priv { + /* + * Set to true if the MMIO handlers were set up for the emulated + * ECAM host PCI bridge. + */ + bool is_virt_ecam; +}; + /* Do some sanity checks. */ static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len) { @@ -38,6 +46,7 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, pci_sbdf_t sbdf; unsigned long data = ~0UL; unsigned int size = 1U << info->dabt.size; + struct vpci_mmio_priv *priv = (struct vpci_mmio_priv *)p; sbdf.sbdf = MMCFG_BDF(info->gpa); reg = REGISTER_OFFSET(info->gpa); @@ -45,6 +54,13 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, if ( !vpci_mmio_access_allowed(reg, size) ) return 0; + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( priv->is_virt_ecam && !pci_translate_virtual_device(v, &sbdf) ) + return 1; + data = vpci_read(sbdf, reg, min(4u, size)); if ( size == 8 ) data |= (uint64_t)vpci_read(sbdf, reg + 4, 4) << 32; @@ -61,6 +77,7 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, pci_sbdf_t sbdf; unsigned long data = r; unsigned int size = 1U << info->dabt.size; + struct vpci_mmio_priv *priv = (struct vpci_mmio_priv *)p; sbdf.sbdf = MMCFG_BDF(info->gpa); reg = REGISTER_OFFSET(info->gpa); @@ -68,6 +85,13 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, if ( !vpci_mmio_access_allowed(reg, size) ) return 0; + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( priv->is_virt_ecam && !pci_translate_virtual_device(v, &sbdf) ) + return 1; + vpci_write(sbdf, reg, min(4u, size), data); if ( size == 8 ) vpci_write(sbdf, reg + 4, 4, data >> 32); @@ -80,13 +104,48 @@ static const struct mmio_handler_ops vpci_mmio_handler = { .write = vpci_mmio_write, }; +/* + * There are three originators for the PCI configuration space access: + * 1. The domain that owns physical host bridge: MMIO handlers are + * there so we can update vPCI register handlers with the values + * written by the hardware domain, e.g. physical view of the registers/ + * configuration space. + * 2. Guest access to the passed through PCI devices: we need to properly + * map virtual bus topology to the physical one, e.g. pass the configuration + * space access to the corresponding physical devices. + * 3. Emulated host PCI bridge access. It doesn't exist in the physical + * topology, e.g. it can't be mapped to some physical host bridge. + * So, all access to the host bridge itself needs to be trapped and + * emulated. + */ static int vpci_setup_mmio_handler(struct domain *d, struct pci_host_bridge *bridge) { - struct pci_config_window *cfg = bridge->cfg; + struct vpci_mmio_priv *priv; + + priv = xzalloc(struct vpci_mmio_priv); + if ( !priv ) + return -ENOMEM; + + priv->is_virt_ecam = !is_hardware_domain(d); - register_mmio_handler(d, &vpci_mmio_handler, - cfg->phys_addr, cfg->size, NULL); + if ( is_hardware_domain(d) ) + { + struct pci_config_window *cfg = bridge->cfg; + + bridge->mmio_priv = priv; + register_mmio_handler(d, &vpci_mmio_handler, + cfg->phys_addr, cfg->size, + priv); + } + else + { + d->vpci_mmio_priv = priv; + /* Guest domains use what is programmed in their device tree. */ + register_mmio_handler(d, &vpci_mmio_handler, + GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, + priv); + } return 0; } @@ -95,13 +154,16 @@ int domain_vpci_init(struct domain *d) if ( !has_vpci(d) ) return 0; - if ( is_hardware_domain(d) ) - return pci_host_iterate_bridges(d, vpci_setup_mmio_handler); - - /* Guest domains use what is programmed in their device tree. */ - register_mmio_handler(d, &vpci_mmio_handler, - GUEST_VPCI_ECAM_BASE, GUEST_VPCI_ECAM_SIZE, NULL); + return pci_host_iterate_bridges(d, vpci_setup_mmio_handler); +} +static int domain_vpci_free_cb(struct domain *d, + struct pci_host_bridge *bridge) +{ + if ( is_hardware_domain(d) ) + XFREE(bridge->mmio_priv); + else + XFREE(d->vpci_mmio_priv); return 0; } @@ -124,6 +186,13 @@ int domain_vpci_get_num_mmio_handlers(struct domain *d) return count; } +void domain_vpci_free(struct domain *d) +{ + if ( !has_vpci(d) ) + return; + + pci_host_iterate_bridges(d, domain_vpci_free_cb); +} /* * Local variables: * mode: C diff --git a/xen/arch/arm/vpci.h b/xen/arch/arm/vpci.h index 27a2b069abd2..38e5a28c0d95 100644 --- a/xen/arch/arm/vpci.h +++ b/xen/arch/arm/vpci.h @@ -18,6 +18,7 @@ #ifdef CONFIG_HAS_VPCI int domain_vpci_init(struct domain *d); int domain_vpci_get_num_mmio_handlers(struct domain *d); +void domain_vpci_free(struct domain *d); #else static inline int domain_vpci_init(struct domain *d) { @@ -28,6 +29,8 @@ static inline int domain_vpci_get_num_mmio_handlers(struct domain *d) { return 0; } + +static inline void domain_vpci_free(struct domain *d) { } #endif #endif /* __ARCH_ARM_VPCI_H__ */ diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 4552ace855e0..579c6947cc35 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -890,6 +890,31 @@ int pci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev) return 0; } +/* + * Find the physical device which is mapped to the virtual device + * and translate virtual SBDF to the physical one. + */ +bool pci_translate_virtual_device(struct vcpu *v, pci_sbdf_t *sbdf) +{ + struct domain *d = v->domain; + struct vpci_dev *vdev; + bool found = false; + + pcidevs_lock(); + list_for_each_entry ( vdev, &d->vdev_list, list ) + { + if ( vdev->sbdf.sbdf == sbdf->sbdf ) + { + /* Replace virtual SBDF with the physical one. */ + *sbdf = vdev->pdev->sbdf; + found = true; + break; + } + } + pcidevs_unlock(); + return found; +} + /* Caller should hold the pcidevs_lock */ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, uint8_t devfn) diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index b81f66e813ef..983a63be7572 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -72,6 +72,7 @@ struct pci_host_bridge { struct pci_config_window* cfg; /* Pointer to the bridge config window */ void *sysdata; /* Pointer to the config space window*/ const struct pci_ops *ops; + void *mmio_priv; /* MMIO handler's private data. */ }; struct pci_ops { diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 35ae1d093921..0017de19e7c8 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -191,6 +191,7 @@ struct pci_dev *pci_get_real_pdev(int seg, int bus, int devfn); struct pci_dev *pci_get_pdev_by_domain(const struct domain *, int seg, int bus, int devfn); void pci_check_disable_device(u16 seg, u8 bus, u8 devfn); +bool pci_translate_virtual_device(struct vcpu *v, pci_sbdf_t *sbdf); uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg); uint16_t pci_conf_read16(pci_sbdf_t sbdf, unsigned int reg); diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index d304c7ebe766..0626820545cc 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -450,6 +450,8 @@ struct domain * to assign a unique SBDF to a passed through virtual PCI device. */ int vpci_dev_next; + /* Virtual PCI MMIO handler's private data. */ + void *vpci_mmio_priv; #endif #ifdef CONFIG_HAS_PASSTHROUGH