From patchwork Thu Jul 20 00:32:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0361C001E0 for ; Thu, 20 Jul 2023 00:33:10 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566256.884849 (Exim 4.92) (envelope-from ) id 1qMHb3-0001gC-GK; Thu, 20 Jul 2023 00:32:45 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566256.884849; Thu, 20 Jul 2023 00:32:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb3-0001ds-9a; Thu, 20 Jul 2023 00:32:45 +0000 Received: by outflank-mailman (input) for mailman id 566256; Thu, 20 Jul 2023 00:32:43 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb0-0001JN-Vo for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:43 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id ee9a29ce-2694-11ee-8611-37d641c3527e; Thu, 20 Jul 2023 02:32:40 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiB5017190; Thu, 20 Jul 2023 00:32:37 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 20 Jul 2023 00:32:37 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:32 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:32 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ee9a29ce-2694-11ee-8611-37d641c3527e ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jwMuXJV4hDcgX76yzhHxDh+sOKtQ+0l0jRTb4FOfd4dDit1OA239Nnfnp/Xzf+uqHI/eQMq6UghXSpj6lDxhfqPuPPxF7VvODdudWWTq76HxmjGrOvhKQezbZpI8UEzcgQ3oF5lKoxTE9Tjb6AUvmJiZc2vrOgh5RDVGjeLofur903FeLWDD0y7dzbx8Cfeqv16H6eHqdap127mDHH8d6cA9c/4zMlLLD3UW2Vv1DCNWRPv1tLDVD4yMyhPEudv8+LhAYFSSttwHbEcq0P9evv7h4Bm8tZdQzu09cKkPMCrwbRc/UvYUDP+QwBUUI660cAm1A9PNdHi+4qbynshD/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iKqHtyDUDA8wFxvQepqDFxbxA7jnJx0SzIFiYOphyxM=; b=eCqwV3mcykF2KIDkepRT2dKaSU+GWKBjiRkSDG8eX/QZbputTmrhKjDG1L3zeOUeMku97chKWeZSuSF7yCoKuMsGFT5RiGLqI2GUrvQfqNaNsjjCiwJXp5/lZ9sboB468v8Nu/NqyOkggViNMFpLKncuZ1JMKI3kHuSLLcuXBNBQT2V3mATXVLFFHTCO0lTWtXsMIC4kfBd2L7DKddfQUYe9yBk1kz8cJPCZ4lDA4Z68OtB6mxBj5VDkKn5DaztGx0xL6MZj6ocPh8jzcNCxoJjo1rlkO2+q7LY4uLnaX0thVUNBHGmvxjjJ/GXoze1EGI9QMTFK4EmTnIvmP00Bew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iKqHtyDUDA8wFxvQepqDFxbxA7jnJx0SzIFiYOphyxM=; b=ZQBpaN6niSuGrXcawTKRpgYx6qZz/TsfszX99XpaxDmU1xbZMIlCyQEIJ14ER76wiW6amh7zwSyYazPFt1LrjOYp68TcSuLO3M3oGQRyNHZEddd13NAPpeFIl38Qb14bgNXvNcZr0dwlUk7HeQahgVd0mUym+t29V3oNlZBvLszt2jqEw+r6CPMDkihu4aHpQSCm1CLW6UGPXcGX7QLpO9H5FEGUXG6QH1OdDrbsJdCCPevfTlkwKRHXpq70KtXU0nKEuPGOXnABh3VjPfNu2YJBh/TwfxsIw7W9+kHk/5GHYq6ZFFOCkZmu7eIhYGUNe4+4+muQ7pyiL8nHgxF9LA== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Volodymyr Babchuk , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= , Jan Beulich Subject: [PATCH v8 01/13] pci: introduce per-domain PCI rwlock Thread-Topic: [PATCH v8 01/13] pci: introduce per-domain PCI rwlock Thread-Index: AQHZuqGr3g/gocvkCU2kdSqq613AoQ== Date: Thu, 20 Jul 2023 00:32:31 +0000 Message-ID: <20230720003205.1828537-2-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: a41c2575-ed4d-4f4b-0c2a-08db88b8ce60 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Tza7cq3IoswCttZHApdAfEwDOPMqbHTQnybXZNR+0LOP1FwzPl1Fp7Xmq++1sjFnIX+LohpBHtBhxqtoungcRyyZrc96M+k2ilMpED6Ws6u5PBVoVjSEY+GVkM8ZnNj2yWfOlZZtRX6OvS7SKYFtDpPHdEciGK6BzlfZxLV5pNeTOK2SNuUjaOHQvQb7PZsYWNBJiaazlp9GwCLeZLx28E6Mmybd+M9oIMTSWVwEtpUtbvdqe5odJsp9j/i/H9roOVPZ8OF8j1Jq9OlVOohzpb+PuUMyYzOvobzGRa7YV8DVGsVX9h/UodE2NL2osxYu0Fy+JM7IDM8wGS8ym/bOSAkw8iHmYnH9OYdym++LgsSruy0r42b7NVcyYLiR47++SFujq5QCRL3ZHv9CgHcrvuf0RJgDkehTU2pukiKtYDJ/Xr5vTZrWzIM0NSKjAARxGwq8hdUWFOR+txUgWIDxEYjTc2vgJAmGZm9iqnZgleUOEab9MtTp8pRrqPD4FlJK6Vag4g0EzK9sNxeecAaEPgz/Iwg3GiLnGnzOyxC/Tl8pFo9tBr8mb7NhlgZZowExF/H+m11FAFMTajNUZGQq9OLvJ8tqkTbPY7VVLXO7QD8f9r7DazSGbb6B1UDftIGR x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(54906003)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?EShXOCN3BVN8OW2QPu5AYfeoxNAN?= =?utf-8?q?w+xX3FRMMVrcQna0QY8Nw5R+Ej1fIaE4Np1fi/lXOE/AzuVXEsFxMlX4/LnIIFMht?= =?utf-8?q?amORsB19x6SZpNcqHw7hB/u5GRbbzYGSgAU6Gwdqs2BxjZy/752wsak14SSADNZWC?= =?utf-8?q?ObIO3k6AxTrqEo+YutxJQ2EhjRqYdYrYGmIj2KHftSEbEmgXfEViNBha5k6HPd1Ge?= =?utf-8?q?OoxeSBV7SxTra45zEMXafgrPmnk1dp8Z9vfuVLC+ZinT5OubnVwuG7m/ci4cRM7F7?= =?utf-8?q?ksk6dSCyCjON+Zog06HIURvO1Vj5YfQK5zfX++V/tztCixcq42/+I6nLQJvI0+JrT?= =?utf-8?q?owu1OMoXecG+H4xgqdKfjTilV1jea+E8gYtswVSEc76bB4h25J/L2Y42sOJvzDFLI?= =?utf-8?q?MEXtWw+IQ40pdL23dF6rlOuEGKEhIjFNNW3bEbeIMVHNj0le1My316Qa9JLH77vqo?= =?utf-8?q?+pHhC5/xhLajTWlb5/k88KRtKM1Rn8iUFqs/nMkhN7cmhrEuY5KNsFVfGe0Vr7A8l?= =?utf-8?q?ccLzKLEuqdknvarohLaCokk+MT/4338Z50zS8fNFDqimzL9KI8/4lL41BIwsCes8g?= =?utf-8?q?ZA2e2ezCB2vNol6kUkAGBghR+73Sme3Eakh6614FSGnL+hK6VNLxPbonKI+qKfVO3?= =?utf-8?q?MFiUhxSvLcjjZBnzZkZFdpYNfXHk9aVVlafhEYVJd1mRJCighrNVXIQWiy1El3TyK?= =?utf-8?q?WLcM0zz+yDoSnSrxgwZKhJ4WVpow64kOO6gHqgScAKYsBmTcejaZvq/8/5JnZ89jx?= =?utf-8?q?nYxajSRR+aPDya8/e7UhwLnJrCfWsj8efp6tUBvFu9z0RSyayKCCAkLUPbMAJlTda?= =?utf-8?q?dzIEQuu9J6riQpz0XUgxzJb6w9sVNe/tKBNkBZw3PbWm/CmdFddFZ8xY1UeZDYkEQ?= =?utf-8?q?NpeHRj3XWjnMTdVMBEkbWMgBgn9m0/5rU3MswHaI/wo3V2p+ikCvPrlOOGTd65RTI?= =?utf-8?q?OJ761Jf+Fi6mqhZzQ6CTj7TG1Uqot9VRnUDWJmi7B7/f7zooV/XzQV49ipn4q1dPq?= =?utf-8?q?SScXEUkTxiq6ZuCCWqQ3CH6HZaUFNgPJFBhufo3NM2zyg5DCjPHcrINtE5V2weqZZ?= =?utf-8?q?iLAyA8fjNyBIdKHXxS4KuWJipsASlVL8Dcb0m3Z2+5wSiOB5IaqwG0jifR/geOc3d?= =?utf-8?q?KbqXgyrSXu8mWyMe2O1GILzikw2nK6YTbiDMmFznsSY/IKYreVCkr6bsUeqIP9dYt?= =?utf-8?q?fqPOGJwaO6il4EOowzVtxicnDngYUDGkHgM7PfT+ysT7MZQ2uDv+Cw4Tn0lgV1Ej8?= =?utf-8?q?CcXiF4zV+Gp0bhq9B9I3tUn7Ji6rt67GCuNUS06xrEdvWucl+EiUm5tMBjz+u6PMe?= =?utf-8?q?XKFEpmEif3Ei5JgnX9Ehw5kKm07bfck3tgsV/ZoU2X9BGpctMHP2WueZvAfrC8xjO?= =?utf-8?q?2u1L4RbwvBDIhDIgj3Wm9mPIWMfWXd7K7lh+Hy2X6SKiSq249y/Vy6L3Te0qUuoIN?= =?utf-8?q?lCKplCWWaVbRUStcdEiLVZluf6jWZUTPxGuoVP3MT9/Yw+L9KXm1udkRXHQa1kOE6?= =?utf-8?q?+7EtXCGNpgDFf1KVTwy32MJUEv00EGsuGw=3D=3D?= Content-ID: MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a41c2575-ed4d-4f4b-0c2a-08db88b8ce60 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:31.0563 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ON8KfDYwRzy+4rxSN6hp6WFYMVg/DtrqaVKH64tyOp4zqQkL7xpnRGkD4U/zgClCb2ANvCoOtqJl1aKJuxztblui1KYBPBl1jYAKMyZwTjc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: GVDTlzfBD9UJ1hjtXI1V19gWUeRZeGKf X-Proofpoint-GUID: GVDTlzfBD9UJ1hjtXI1V19gWUeRZeGKf X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 Add per-domain d->pci_lock that protects access to d->pdev_list. Purpose of this lock is to give guarantees to VPCI code that underlying pdev will not disappear under feet. This is a rw-lock, but this patch adds only write_lock()s. There will be read_lock() users in the next patches. This lock should be taken in write mode every time d->pdev_list is altered. This covers both accesses to d->pdev_list and accesses to pdev->domain_list fields. All write accesses also should be protected by pcidevs_lock() as well. Idea is that any user that wants read access to the list or to the devices stored in the list should use either this new d->pci_lock or old pcidevs_lock(). Usage of any of this two locks will ensure only that pdev of interest will not disappear from under feet and that the pdev still will be assigned to the same domain. Of course, any new users should use pcidevs_lock() when it is appropriate (e.g. when accessing any other state that is protected by the said lock). Any write access to pdev->domain_list should be protected by both pcidevs_lock() and d->pci_lock in the write mode. Suggested-by: Roger Pau Monné Suggested-by: Jan Beulich Signed-off-by: Volodymyr Babchuk --- Changes in v8: - New patch Changes in v8 vs RFC: - Removed all read_locks after discussion with Roger in #xendevel - pci_release_devices() now returns the first error code - extended commit message - added missing lock in pci_remove_device() - extended locked region in pci_add_device() to protect list_del() calls --- xen/common/domain.c | 1 + xen/drivers/passthrough/amd/pci_amd_iommu.c | 9 ++- xen/drivers/passthrough/pci.c | 68 +++++++++++++++++---- xen/drivers/passthrough/vtd/iommu.c | 9 ++- xen/include/xen/sched.h | 1 + 5 files changed, 74 insertions(+), 14 deletions(-) diff --git a/xen/common/domain.c b/xen/common/domain.c index caaa402637..5d8a8836da 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -645,6 +645,7 @@ struct domain *domain_create(domid_t domid, #ifdef CONFIG_HAS_PCI INIT_LIST_HEAD(&d->pdev_list); + rwlock_init(&d->pci_lock); #endif /* All error paths can depend on the above setup. */ diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c b/xen/drivers/passthrough/amd/pci_amd_iommu.c index 94e3775506..e2f2e2e950 100644 --- a/xen/drivers/passthrough/amd/pci_amd_iommu.c +++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c @@ -476,8 +476,13 @@ static int cf_check reassign_device( if ( devfn == pdev->devfn && pdev->domain != target ) { - list_move(&pdev->domain_list, &target->pdev_list); - pdev->domain = target; + write_lock(&pdev->domain->pci_lock); + list_del(&pdev->domain_list); + write_unlock(&pdev->domain->pci_lock); + + write_lock(&target->pci_lock); + list_add(&pdev->domain_list, &target->pdev_list); + write_unlock(&target->pci_lock); } /* diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 95846e84f2..5b4632ead2 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -454,7 +454,9 @@ static void __init _pci_hide_device(struct pci_dev *pdev) if ( pdev->domain ) return; pdev->domain = dom_xen; + write_lock(&dom_xen->pci_lock); list_add(&pdev->domain_list, &dom_xen->pdev_list); + write_unlock(&dom_xen->pci_lock); } int __init pci_hide_device(unsigned int seg, unsigned int bus, @@ -747,6 +749,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, ret = 0; if ( !pdev->domain ) { + write_lock(&hardware_domain->pci_lock); pdev->domain = hardware_domain; list_add(&pdev->domain_list, &hardware_domain->pdev_list); @@ -760,6 +763,7 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret); list_del(&pdev->domain_list); pdev->domain = NULL; + write_unlock(&hardware_domain->pci_lock); goto out; } ret = iommu_add_device(pdev); @@ -768,8 +772,10 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn, vpci_remove_device(pdev); list_del(&pdev->domain_list); pdev->domain = NULL; + write_unlock(&hardware_domain->pci_lock); goto out; } + write_unlock(&hardware_domain->pci_lock); } else iommu_enable_device(pdev); @@ -812,11 +818,13 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) if ( pdev->bus == bus && pdev->devfn == devfn ) { + write_lock(&pdev->domain->pci_lock); vpci_remove_device(pdev); pci_cleanup_msi(pdev); ret = iommu_remove_device(pdev); if ( pdev->domain ) list_del(&pdev->domain_list); + write_unlock(&pdev->domain->pci_lock); printk(XENLOG_DEBUG "PCI remove device %pp\n", &pdev->sbdf); free_pdev(pseg, pdev); break; @@ -887,26 +895,62 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, int pci_release_devices(struct domain *d) { - struct pci_dev *pdev, *tmp; - u8 bus, devfn; - int ret; + int combined_ret; + LIST_HEAD(failed_pdevs); pcidevs_lock(); - ret = arch_pci_clean_pirqs(d); - if ( ret ) + write_lock(&d->pci_lock); + combined_ret = arch_pci_clean_pirqs(d); + if ( combined_ret ) { pcidevs_unlock(); - return ret; + write_unlock(&d->pci_lock); + return combined_ret; } - list_for_each_entry_safe ( pdev, tmp, &d->pdev_list, domain_list ) + + while ( !list_empty(&d->pdev_list) ) { - bus = pdev->bus; - devfn = pdev->devfn; - ret = deassign_device(d, pdev->seg, bus, devfn) ?: ret; + struct pci_dev *pdev = list_first_entry(&d->pdev_list, + struct pci_dev, + domain_list); + uint16_t seg = pdev->seg; + uint8_t bus = pdev->bus; + uint8_t devfn = pdev->devfn; + int ret; + + write_unlock(&d->pci_lock); + ret = deassign_device(d, seg, bus, devfn); + write_lock(&d->pci_lock); + if ( ret ) + { + bool still_present = false; + const struct pci_dev *tmp; + + /* + * We need to check if deassign_device() left our pdev in + * domain's list. As we dropped the lock, we can't be sure + * that list wasn't permutated in some random way, so we + * need to traverse the whole list. + */ + for_each_pdev ( d, tmp ) + { + if ( tmp == pdev ) + { + still_present = true; + break; + } + } + if ( still_present ) + list_move(&pdev->domain_list, &failed_pdevs); + combined_ret = combined_ret?:ret; + } } + + list_splice(&failed_pdevs, &d->pdev_list); + write_unlock(&d->pci_lock); pcidevs_unlock(); - return ret; + return combined_ret; } #define PCI_CLASS_BRIDGE_HOST 0x0600 @@ -1125,7 +1169,9 @@ static int __hwdom_init cf_check _setup_hwdom_pci_devices( if ( !pdev->domain ) { pdev->domain = ctxt->d; + write_lock(&ctxt->d->pci_lock); list_add(&pdev->domain_list, &ctxt->d->pdev_list); + write_unlock(&ctxt->d->pci_lock); setup_one_hwdom_device(ctxt, pdev); } else if ( pdev->domain == dom_xen ) diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c index 0e3062c820..55ee3f110d 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -2806,7 +2806,14 @@ static int cf_check reassign_device_ownership( if ( devfn == pdev->devfn && pdev->domain != target ) { - list_move(&pdev->domain_list, &target->pdev_list); + write_lock(&pdev->domain->pci_lock); + list_del(&pdev->domain_list); + write_unlock(&pdev->domain->pci_lock); + + write_lock(&target->pci_lock); + list_add(&pdev->domain_list, &target->pdev_list); + write_unlock(&target->pci_lock); + pdev->domain = target; } diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 85242a73d3..80dd150bbf 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -460,6 +460,7 @@ struct domain #ifdef CONFIG_HAS_PCI struct list_head pdev_list; + rwlock_t pci_lock; #endif #ifdef CONFIG_HAS_PASSTHROUGH From patchwork Thu Jul 20 00:32:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319658 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30E3CC04E69 for ; Thu, 20 Jul 2023 00:33:03 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566255.884843 (Exim 4.92) (envelope-from ) id 1qMHb3-0001ad-6C; Thu, 20 Jul 2023 00:32:45 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566255.884843; Thu, 20 Jul 2023 00:32:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb2-0001Zq-Va; Thu, 20 Jul 2023 00:32:44 +0000 Received: by outflank-mailman (input) for mailman id 566255; Thu, 20 Jul 2023 00:32:42 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb0-0001JI-NT for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:42 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id ef2c5212-2694-11ee-b23a-6b7b168915f2; Thu, 20 Jul 2023 02:32:41 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiB6017190; Thu, 20 Jul 2023 00:32:38 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 20 Jul 2023 00:32:38 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:32 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:32 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ef2c5212-2694-11ee-b23a-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TxH1RSkGK+hRVuJxMTWcF/mwPOL8tedwFBXn1EVCfoz8eQYQ2veAjpLGgiftfacCcm6ymLNn+gtHTD9Buk4/hhD8zdcN8CDWv3sh4uIGZtxDzO92zGH8CQditvGpwQkkne7UywWjaW2dfFDBTKo6BJaFoMLDGVaZsefNqYMqJpLm9AMjYuWVUjwWeh/sWmAi0hXo6AtZxT0herAEd7snRPKnoaE9/MfPZraQAFYZ35JeryOJgVGBdXlrQIQnFq9nrkcM7R9cIGdqjFa7l49q/92CoF3TtyUwic3y0EOj+xNpsS48ixcFRqc3HIEWOowiubtXLpYtk0D05g2KU+XV7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=abxFPIgNa+QJErFwy718sKh+mXFlflc8O/+7z9aj10M=; b=XlLL/AbwfjZ3YhwkQfAWXJF5vTdB8dcOJEjgbNS+tKfJ3BwjaXdkfp1gNHo/So9O30J96gZJYTKzB+9kYx6Euj4O02R5C9c3Y3BWIMSk9SdTLd5M983c0J47GLciuKXjdSXAse9+ODudlLTuKe4ts/6Y8HK6sr8zAF2met39raZBpTM3Zsqlr/Wq15EYstpgEh1iWFlfUGEL3n9FN/cqS+NsJoYDYKgElCBPj8J93MG5pBieg5Ne0fQGKbFQhshVWR6z0H8be3Cs8RaiKbiz0wO/FPQ9Zz31TaUpdLQTufEOE64exGihU5b7Pp/K66NyW56TxETSs5MWuByQ7wjLlg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=abxFPIgNa+QJErFwy718sKh+mXFlflc8O/+7z9aj10M=; b=TyyKgI2SR5Uc4Kkx937N+OftwGWHvc5rC7erAlz8hx7+8OKuSqnR6x/lEroi0WRE6MgnRtdW3/x4XSIhHJ3s5JJBOZQTvdOcQ6aqViZnpQBLH4TIaWx3y2DWgr5boQMEtdE1sLZitllESNzSWTQNr0iCHRZuiwWVo0JufN5X13bOQ0b9tpjTIbBmIzXM92jzOpwEqvO9/tPK+S/Ij6ZPN8fQMtLGVVMozUhrTwMQ05C7upDUy/PhHdIwf78sh9K8l91c4bs37TBNwbhXZmRFUgbZNzUq3+lW3EkEGACMaxovHgHQSnNE79CS8DCmshQG010loZ3tM+0bMn64bMnOhQ== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= , Jan Beulich , Volodymyr Babchuk Subject: [PATCH v8 02/13] vpci: use per-domain PCI lock to protect vpci structure Thread-Topic: [PATCH v8 02/13] vpci: use per-domain PCI lock to protect vpci structure Thread-Index: AQHZuqGrLMJPz1ipBEu4urExXp8yiw== Date: Thu, 20 Jul 2023 00:32:31 +0000 Message-ID: <20230720003205.1828537-3-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 581027c8-a6bc-453c-1cb2-08db88b8ce92 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 8XLhK+cI/Y9/osOHRZyqh+db/Z5my1ZRXPzSGCw7gyTueyTcI6v+cs7vPCHs6a8Bte9fqou3G8uPYHWhcXtagPsIGQT/vcDEPirj31Bi23z6+B89gh4cPNvfVQaVeGSVQZYpTOibaQJatablAGot6FtGj2GtloDDlzMH8lhG1RtIdERw+rta5Sj4433dTEwr9+1wpCuBRPN6ft9DI4lYnea+PtSogtXQshYivv+eluSu4anKjGxmKA8QVLlEvZNa/Q1Pp4eQQjwfcWMoLZxwZH0MkdPB3v5mo9SScecv6THFWL9PFNEWj4KKcvbc4Yt+Hw3sE8LIRXyEk+2Hya+isMZFsb1M5YGpYwFo+DP/hWD5kq/87kESs6JNe0IpIlsMK7ykVCLNzAnj9vxEyRJ06B6hP3MCTNB4/bodePEJuWH8solmaOqkNTSkHkkBDqij6y04XAbFI9cbS1GrBD8ieqUw9qrySgbKV/D8mrxLeXGWgRporTI8YJ3y1lXbyAcQCQXLxGlAgH1bW0iBnPeeQiDKMWCUP/LD//airW2f+Pl8odR+7nVxXpI0xKh+HYftXHkqQdb8swtuEQwhZUaI1gAhMzVpxS2yO/S+WjyEuia0lYubAc7APMIrDr4O2ibgkjgLnWLOHky1+boroXcElw== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(54906003)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(30864003)(2906002)(309714004);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?vUfqebxAuyOkAC3xYlQDxkLQ5/NQ?= =?utf-8?q?j00d+6oDa4w8IBEc8bOLVidLNx0F+bOvGewUH0OkVnJuuJC9XlQXr//YMe2SaWM3e?= =?utf-8?q?qaerebhMsSkL75LLxdCqszqB9/9naHW766xGoS45zqcnKUzkdyvpAcfPoQ42sWYPh?= =?utf-8?q?4SO8m31Wt5qIzYld3N2LcfJmAcyRFAJpSz7xxVhpxiOIJegwlQTdmGuQ2iOkwHMS/?= =?utf-8?q?YtU+ocjculYYO5jPMVPSEjSJANFwUz/4EKCz1lYv73KnbjqToW9iG0artqU2iH89o?= =?utf-8?q?CKFS+eLHgWyla9F4j4tGnoPrdej444/lbmjwnCPDlunVxV/2jGbquopc2qUykBc3+?= =?utf-8?q?Rzox1y6q5E+kovLAUbKDb5dVS3cj95HEBZP0s5tu6QYSVGwQEjVm925puMKP+0YlI?= =?utf-8?q?kjz2DYOwJRpcY89WhrFWkro4f1JteEBl1dfpbksT2Fs58CX6qJlmtRgzkCfEuZ9qv?= =?utf-8?q?RqzEahYbLTozfMm8385+4agBjYTPYXn5eBmod8oELkuX9QO+avZT7IB4zr3uD1nyd?= =?utf-8?q?WZmTEjfK1wdEfpvHRvY3pvoNUb5QqiH3/H0QQa2zn+mI5RTLEht7NsSdEmbI5Zk1g?= =?utf-8?q?MFWBbHbDr4L4FBAwi0SAad1Vt+wBi7Xcl9f1ubDkk6SUX9UY3E/tVbSMaFsBWitXI?= =?utf-8?q?DcGlcEU49FYZn0OsFtxLDo3OcYRII+8DZx6Z0YhJl9JMEMPRpUWouKNoOYQUinFgK?= =?utf-8?q?3bUNSniySVtNCXZ/elb47v99yS49OxhzyocCCxi+eXEd1griYCjpvdifDrgGslfb6?= =?utf-8?q?naHIiDXsNu3UvaN+PdD3SUeWp9ktEfbv42DQzsHInBoKZcn78oh4b8Zd5wCOxz8RL?= =?utf-8?q?M/3Wj0N8v7pq1kcjpk2c7gsfwokMDGWQtrQa2Xq9LNxse94NB+p8PJrecbnJD+8sw?= =?utf-8?q?fgf9xdvWSb/bqw+P4MslUKTg5I1wGplvcjZ/I8CBELWumN852yK97aMNPNRoBG1fU?= =?utf-8?q?NNVveBAAc14ulLy4ul4ABrkncRAUDlPoRqp/ksH3mOVbFK/27BYBznXA3olLuAUhK?= =?utf-8?q?xIhK1NwCRAcsyGppUmkD8JylVqNskHjF7dEXQgXYbh4iBKcyKOfLVdRpFKsSym9zW?= =?utf-8?q?6Kde6T1zJgRv0pYnVF0Qmm1Ns34ExHjTix5dwnrcU4n4xNqV0tl5QFYZmCfpP/auY?= =?utf-8?q?W4LtEdsJLoP8BaFPU7L6PxCCNJ5yo6Jwev0xkSowmOiYmLzncSgL2ATTv9XQsOXI8?= =?utf-8?q?0ZPZzM8FjQNXYb7Jm6ZslbNUdULin9thbQDyflXyF85Q4xf5rwqxb5OoPkGGx2Alk?= =?utf-8?q?Q8EGD9ze2TjkC746DZ9cji3w8XgLdeWjb/qUbCZSE3xP+9E0/x/6QHNciBOmSb5i+?= =?utf-8?q?h62b9PKhN6/7zdioEQaHcqd7d3ymUSgt9HYXeGr713XmZiYfvG0lCD2tDPpsSHuuT?= =?utf-8?q?Tsc/IArhtvT/3mkU3IIhJ8ALxmRSID487iCDswqL2Tot9bQWio4N+bxyR1Yiz5b9Q?= =?utf-8?q?UxpnYfTauLEzMLB8hM+9a09CTi/px0lVGYWIW8Tv0GfWVyw4girwr5Q+mnkusP6Wb?= =?utf-8?q?AyfcPbWTiJ/bVCPAkuSK6ETegsDEFHYp0A=3D=3D?= Content-ID: MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 581027c8-a6bc-453c-1cb2-08db88b8ce92 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:31.3755 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: BkrJQVEi0+vUpEXg8coxK4v4webME8P0jKt7mpCq1pDwZMW3s3L/0XE2lry/IS9VL37hbKLR8fCDjGh30BJFJEqJWpQCVUZoCx5rhhalqQk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: wS0UN25VQYO8CBIh5be9I2KXNyp6YnQE X-Proofpoint-GUID: wS0UN25VQYO8CBIh5be9I2KXNyp6YnQE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko Use a previously introduced per-domain read/write lock to check whether vpci is present, so we are sure there are no accesses to the contents of the vpci struct if not. This lock can be used (and in a few cases is used right away) so that vpci removal can be performed while holding the lock in write mode. Previously such removal could race with vpci_read for example. 1. Per-domain's pci_rwlock is used to protect pdev->vpci structure from being removed. 2. Writing the command register and ROM BAR register may trigger modify_bars to run, which in turn may access multiple pdevs while checking for the existing BAR's overlap. The overlapping check, if done under the read lock, requires vpci->lock to be acquired on both devices being compared, which may produce a deadlock. It is not possible to upgrade read lock to write lock in such a case. So, in order to prevent the deadlock, use d->pci_lock instead. To prevent deadlock while locking both hwdom->pci_lock and dom_xen->pci_lock, always lock hwdom first. All other code, which doesn't lead to pdev->vpci destruction and does not access multiple pdevs at the same time, can still use a combination of the read lock and pdev->vpci->lock. 3. Drop const qualifier where the new rwlock is used and this is appropriate. 4. Do not call process_pending_softirqs with any locks held. For that unlock prior the call and re-acquire the locks after. After re-acquiring the lock there is no need to check if pdev->vpci exists: - in apply_map because of the context it is called (no race condition possible) - for MSI/MSI-X debug code because it is called at the end of pdev->vpci access and no further access to pdev->vpci is made 5. Introduce pcidevs_trylock, so there is a possibility to try locking the pcidev's lock. 6. Use d->pci_lock around for_each_pdev and pci_get_pdev_by_domain while accessing pdevs in vpci code. Suggested-by: Roger Pau Monné Suggested-by: Jan Beulich Signed-off-by: Oleksandr Andrushchenko Signed-off-by: Volodymyr Babchuk --- Changes in v8: - changed d->vpci_lock to d->pci_lock - introducing d->pci_lock in a separate patch - extended locked region in vpci_process_pending - removed pcidevs_lockis vpci_dump_msi() - removed some changes as they are not needed with the new locking scheme - added handling for hwdom && dom_xen case --- xen/arch/x86/hvm/vmsi.c | 4 +++ xen/drivers/passthrough/pci.c | 7 +++++ xen/drivers/vpci/header.c | 18 ++++++++++++ xen/drivers/vpci/msi.c | 14 ++++++++-- xen/drivers/vpci/msix.c | 52 ++++++++++++++++++++++++++++++----- xen/drivers/vpci/vpci.c | 46 +++++++++++++++++++++++++++++-- xen/include/xen/pci.h | 1 + 7 files changed, 129 insertions(+), 13 deletions(-) diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c index 3cd4923060..8c1bd66b9c 100644 --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -895,6 +895,8 @@ int vpci_msix_arch_print(const struct vpci_msix *msix) { unsigned int i; + ASSERT(rw_is_locked(&msix->pdev->domain->pci_lock)); + for ( i = 0; i < msix->max_entries; i++ ) { const struct vpci_msix_entry *entry = &msix->entries[i]; @@ -913,7 +915,9 @@ int vpci_msix_arch_print(const struct vpci_msix *msix) struct pci_dev *pdev = msix->pdev; spin_unlock(&msix->pdev->vpci->lock); + read_unlock(&pdev->domain->pci_lock); process_pending_softirqs(); + read_lock(&pdev->domain->pci_lock); /* NB: we assume that pdev cannot go away for an alive domain. */ if ( !pdev->vpci || !spin_trylock(&pdev->vpci->lock) ) return -EBUSY; diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 5b4632ead2..6f8692cd9c 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -57,6 +57,11 @@ void pcidevs_lock(void) spin_lock_recursive(&_pcidevs_lock); } +int pcidevs_trylock(void) +{ + return spin_trylock_recursive(&_pcidevs_lock); +} + void pcidevs_unlock(void) { spin_unlock_recursive(&_pcidevs_lock); @@ -1144,7 +1149,9 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, } while ( devfn != pdev->devfn && PCI_SLOT(devfn) == PCI_SLOT(pdev->devfn) ); + write_lock(&ctxt->d->pci_lock); err = vpci_add_handlers(pdev); + write_unlock(&ctxt->d->pci_lock); if ( err ) printk(XENLOG_ERR "setup of vPCI for d%d failed: %d\n", ctxt->d->domain_id, err); diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index b41556d007..2780fcae72 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -152,6 +152,7 @@ bool vpci_process_pending(struct vcpu *v) if ( rc == -ERESTART ) return true; + write_lock(&v->domain->pci_lock); spin_lock(&v->vpci.pdev->vpci->lock); /* Disable memory decoding unconditionally on failure. */ modify_decoding(v->vpci.pdev, @@ -170,6 +171,7 @@ bool vpci_process_pending(struct vcpu *v) * failure. */ vpci_remove_device(v->vpci.pdev); + write_unlock(&v->domain->pci_lock); } return false; @@ -181,8 +183,20 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, struct map_data data = { .d = d, .map = true }; int rc; + ASSERT(rw_is_locked(&d->pci_lock)); + while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART ) + { + /* + * It's safe to drop and reacquire the lock in this context + * without risking pdev disappearing because devices cannot be + * removed until the initial domain has been started. + */ + read_unlock(&d->pci_lock); process_pending_softirqs(); + read_lock(&d->pci_lock); + } + rangeset_destroy(mem); if ( !rc ) modify_decoding(pdev, cmd, false); @@ -223,6 +237,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned int i; int rc; + ASSERT(rw_is_locked(&pdev->domain->pci_lock)); + if ( !mem ) return -ENOMEM; @@ -502,6 +518,8 @@ static int cf_check init_bars(struct pci_dev *pdev) struct vpci_bar *bars = header->bars; int rc; + ASSERT(rw_is_locked(&pdev->domain->pci_lock)); + switch ( pci_conf_read8(pdev->sbdf, PCI_HEADER_TYPE) & 0x7f ) { case PCI_HEADER_TYPE_NORMAL: diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index 8f2b59e61a..e63152c224 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -190,6 +190,8 @@ static int cf_check init_msi(struct pci_dev *pdev) uint16_t control; int ret; + ASSERT(rw_is_write_locked(&pdev->domain->pci_lock)); + if ( !pos ) return 0; @@ -265,7 +267,7 @@ REGISTER_VPCI_INIT(init_msi, VPCI_PRIORITY_LOW); void vpci_dump_msi(void) { - const struct domain *d; + struct domain *d; rcu_read_lock(&domlist_read_lock); for_each_domain ( d ) @@ -277,6 +279,9 @@ void vpci_dump_msi(void) printk("vPCI MSI/MSI-X d%d\n", d->domain_id); + if ( !read_trylock(&d->pci_lock) ) + continue; + for_each_pdev ( d, pdev ) { const struct vpci_msi *msi; @@ -318,14 +323,17 @@ void vpci_dump_msi(void) * holding the lock. */ printk("unable to print all MSI-X entries: %d\n", rc); - process_pending_softirqs(); - continue; + goto pdev_done; } } spin_unlock(&pdev->vpci->lock); + pdev_done: + read_unlock(&d->pci_lock); process_pending_softirqs(); + read_lock(&d->pci_lock); } + read_unlock(&d->pci_lock); } rcu_read_unlock(&domlist_read_lock); } diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 25bde77586..9481274579 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -147,6 +147,8 @@ static struct vpci_msix *msix_find(const struct domain *d, unsigned long addr) { struct vpci_msix *msix; + ASSERT(rw_is_locked(&d->pci_lock)); + list_for_each_entry ( msix, &d->arch.hvm.msix_tables, next ) { const struct vpci_bar *bars = msix->pdev->vpci->header.bars; @@ -163,7 +165,13 @@ static struct vpci_msix *msix_find(const struct domain *d, unsigned long addr) static int cf_check msix_accept(struct vcpu *v, unsigned long addr) { - return !!msix_find(v->domain, addr); + int rc; + + read_lock(&v->domain->pci_lock); + rc = !!msix_find(v->domain, addr); + read_unlock(&v->domain->pci_lock); + + return rc; } static bool access_allowed(const struct pci_dev *pdev, unsigned long addr, @@ -358,21 +366,34 @@ static int adjacent_read(const struct domain *d, const struct vpci_msix *msix, static int cf_check msix_read( struct vcpu *v, unsigned long addr, unsigned int len, unsigned long *data) { - const struct domain *d = v->domain; - struct vpci_msix *msix = msix_find(d, addr); + struct domain *d = v->domain; + struct vpci_msix *msix; const struct vpci_msix_entry *entry; unsigned int offset; *data = ~0ul; + read_lock(&d->pci_lock); + + msix = msix_find(d, addr); if ( !msix ) + { + read_unlock(&d->pci_lock); return X86EMUL_RETRY; + } if ( adjacent_handle(msix, addr) ) - return adjacent_read(d, msix, addr, len, data); + { + int rc = adjacent_read(d, msix, addr, len, data); + read_unlock(&d->pci_lock); + return rc; + } if ( !access_allowed(msix->pdev, addr, len) ) + { + read_unlock(&d->pci_lock); return X86EMUL_OKAY; + } spin_lock(&msix->pdev->vpci->lock); entry = get_entry(msix, addr); @@ -404,6 +425,7 @@ static int cf_check msix_read( break; } spin_unlock(&msix->pdev->vpci->lock); + read_unlock(&d->pci_lock); return X86EMUL_OKAY; } @@ -491,19 +513,32 @@ static int adjacent_write(const struct domain *d, const struct vpci_msix *msix, static int cf_check msix_write( struct vcpu *v, unsigned long addr, unsigned int len, unsigned long data) { - const struct domain *d = v->domain; - struct vpci_msix *msix = msix_find(d, addr); + struct domain *d = v->domain; + struct vpci_msix *msix; struct vpci_msix_entry *entry; unsigned int offset; + read_lock(&d->pci_lock); + + msix = msix_find(d, addr); if ( !msix ) + { + read_unlock(&d->pci_lock); return X86EMUL_RETRY; + } if ( adjacent_handle(msix, addr) ) - return adjacent_write(d, msix, addr, len, data); + { + int rc = adjacent_write(d, msix, addr, len, data); + read_unlock(&d->pci_lock); + return rc; + } if ( !access_allowed(msix->pdev, addr, len) ) + { + read_unlock(&d->pci_lock); return X86EMUL_OKAY; + } spin_lock(&msix->pdev->vpci->lock); entry = get_entry(msix, addr); @@ -579,6 +614,7 @@ static int cf_check msix_write( break; } spin_unlock(&msix->pdev->vpci->lock); + read_unlock(&d->pci_lock); return X86EMUL_OKAY; } @@ -665,6 +701,8 @@ static int cf_check init_msix(struct pci_dev *pdev) struct vpci_msix *msix; int rc; + ASSERT(rw_is_write_locked(&pdev->domain->pci_lock)); + msix_offset = pci_find_cap_offset(pdev->seg, pdev->bus, slot, func, PCI_CAP_ID_MSIX); if ( !msix_offset ) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index d73fa76302..f22cbf2112 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -38,6 +38,8 @@ extern vpci_register_init_t *const __end_vpci_array[]; void vpci_remove_device(struct pci_dev *pdev) { + ASSERT(rw_is_write_locked(&pdev->domain->pci_lock)); + if ( !has_vpci(pdev->domain) || !pdev->vpci ) return; @@ -73,6 +75,8 @@ int vpci_add_handlers(struct pci_dev *pdev) const unsigned long *ro_map; int rc = 0; + ASSERT(rw_is_write_locked(&pdev->domain->pci_lock)); + if ( !has_vpci(pdev->domain) ) return 0; @@ -326,11 +330,12 @@ static uint32_t merge_result(uint32_t data, uint32_t new, unsigned int size, uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) { - const struct domain *d = current->domain; + struct domain *d = current->domain; const struct pci_dev *pdev; const struct vpci_register *r; unsigned int data_offset = 0; uint32_t data = ~(uint32_t)0; + rwlock_t *lock; if ( !size ) { @@ -342,11 +347,21 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) * Find the PCI dev matching the address, which for hwdom also requires * consulting DomXEN. Passthrough everything that's not trapped. */ + lock = &d->pci_lock; + read_lock(lock); pdev = pci_get_pdev(d, sbdf); if ( !pdev && is_hardware_domain(d) ) + { + read_unlock(lock); + lock = &dom_xen->pci_lock; + read_lock(lock); pdev = pci_get_pdev(dom_xen, sbdf); + } if ( !pdev || !pdev->vpci ) + { + read_unlock(lock); return vpci_read_hw(sbdf, reg, size); + } spin_lock(&pdev->vpci->lock); @@ -392,6 +407,7 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) ASSERT(data_offset < size); } spin_unlock(&pdev->vpci->lock); + read_unlock(lock); if ( data_offset < size ) { @@ -431,10 +447,23 @@ static void vpci_write_helper(const struct pci_dev *pdev, r->private); } +/* Helper function to unlock locks taken by vpci_write in proper order */ +static void unlock_locks(struct domain *d) +{ + ASSERT(rw_is_locked(&d->pci_lock)); + + if ( is_hardware_domain(d) ) + { + ASSERT(rw_is_locked(&d->pci_lock)); + read_unlock(&dom_xen->pci_lock); + } + read_unlock(&d->pci_lock); +} + void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, uint32_t data) { - const struct domain *d = current->domain; + struct domain *d = current->domain; const struct pci_dev *pdev; const struct vpci_register *r; unsigned int data_offset = 0; @@ -447,8 +476,16 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, /* * Find the PCI dev matching the address, which for hwdom also requires - * consulting DomXEN. Passthrough everything that's not trapped. + * consulting DomXEN. Passthrough everything that's not trapped. + * If this is hwdom, we need to hold locks for both domain in case if + * modify_bars is called() */ + read_lock(&d->pci_lock); + + /* dom_xen->pci_lock always should be taken second to prevent deadlock */ + if ( is_hardware_domain(d) ) + read_lock(&dom_xen->pci_lock); + pdev = pci_get_pdev(d, sbdf); if ( !pdev && is_hardware_domain(d) ) pdev = pci_get_pdev(dom_xen, sbdf); @@ -459,6 +496,8 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, if ( !ro_map || !test_bit(sbdf.bdf, ro_map) ) vpci_write_hw(sbdf, reg, size, data); + + unlock_locks(d); return; } @@ -498,6 +537,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, ASSERT(data_offset < size); } spin_unlock(&pdev->vpci->lock); + unlock_locks(d); if ( data_offset < size ) /* Tailing gap, write the remaining. */ diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 5975ca2f30..4512910dca 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -157,6 +157,7 @@ struct pci_dev { */ void pcidevs_lock(void); +int pcidevs_trylock(void); void pcidevs_unlock(void); bool_t __must_check pcidevs_locked(void); From patchwork Thu Jul 20 00:32:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319662 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2FB8AC3DA40 for ; Thu, 20 Jul 2023 00:33:10 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566254.884838 (Exim 4.92) (envelope-from ) id 1qMHb2-0001Yj-R3; Thu, 20 Jul 2023 00:32:44 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566254.884838; Thu, 20 Jul 2023 00:32:44 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb2-0001Yc-NX; Thu, 20 Jul 2023 00:32:44 +0000 Received: by outflank-mailman (input) for mailman id 566254; Thu, 20 Jul 2023 00:32:42 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb0-0001JN-Ok for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:42 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id ed2a2834-2694-11ee-8611-37d641c3527e; Thu, 20 Jul 2023 02:32:38 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiB3017190; Thu, 20 Jul 2023 00:32:36 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 20 Jul 2023 00:32:36 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:32 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:32 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ed2a2834-2694-11ee-8611-37d641c3527e ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=geA3nPDtFETcaj7QB62wA2L8PvdYFtir95WHhhHCjM2J8p1+pIz7CEx4pAHjtCP2X1O1eXAyMevTYCAk2iA+05YMXCKfMKbDQK5gTExYv4+xw79J4Lx22UdlUx3cjy6AKUqzEQ+hC5ByBkLKAFSXv9OmmIE4PjnWsZL3ZE7Iwy0Dk1bpE1Td8X6o13pE8y3YCK7s6LsumslZhvwmXtN6HbKYz5eGSR+3t5IAY1XeZn2NjtisBj85dOGZbXIjMigYiNXfidY6BhNRvCKkaMImeD/iUCwiGxY4pNzRzByitZARRdo/2aSfxLLK4Ub5laqqfOdeidQRn69ma493hpPU8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Unt8egmRvwPUu0glZoccRX4BB5aszaqAV25MRw9Jz1w=; b=RFFhrRn6a0MspLZ82RZVpuMhd3S+pTc/hUCtnOqx3xLsMExRhWIcS7AekH6O/djArhQodEMx7GEs84A1EN8Z/sYQP0XmVjU2exzsoaZu2+alXB2DYboZ66kPFiKbeU+QgvxqhSDtFDlp749HF+mY1OBHNWeYhs/VdiiNAE5l+E1w+Ttz2hPluvn61J3ISVW76WOFSYXQmnaYvvoMRPbnbrogCg5eCPI908sgzEa8J62BQaG0YB4vpRvvGq43nqg0BO0GsTJRsBzNRh+XMB0AjGezTjKGOzIIaHPPycdhuKAvp/taEh9MqriuluwvZwSqZY/iZox/cbZhvcecagU/Yw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Unt8egmRvwPUu0glZoccRX4BB5aszaqAV25MRw9Jz1w=; b=TmJVbu0/5GJJnLw579OMEjN4V8+nQAmdstF96gdQFjdLHX/U4XQu8wBviwSZoWFx3BDARylSDcokkIXbr9ohQZ/VVBgzeaZdv29I5xUHBPw2KbXorQ63kp6geKoSdbULs0lZSzWAvOgZJx09DDH0pcazW7PHcYeDmstt52VeoDCQjVpCEgbR1DYOPa6nIPexiGaoZWIeaNA5GfMULb7LbeGg/3YHhsTBtnJ7VEFlQ0RZy87t7UpLbjOod6BGmGCAFJN0/LiT+FRrCTdpNTLlvMDoq+gijiP3S8T3kjbM+VuOz+Rfys0ceKl9GK2IEK5qNAaDHuIt/2rKqVDQp37eeA== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= Subject: [PATCH v8 03/13] vpci: restrict unhandled read/write operations for guests Thread-Topic: [PATCH v8 03/13] vpci: restrict unhandled read/write operations for guests Thread-Index: AQHZuqGrmAJWEwjbUkyUjUzRXQkJ0A== Date: Thu, 20 Jul 2023 00:32:31 +0000 Message-ID: <20230720003205.1828537-4-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 079fc536-4bd1-462a-cae4-08db88b8cec1 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: YliDazfML/9kzzPYPHo1POb0eObPftuP504X69jE80TnIcyZBqPsLmGon9O1mf+8ZX6q8luDZqtKuQzNfTSRmqbGnnkQvaoUqC87UBfr0ZBfoLYavFPJJao4curMIVYnwJs8xpVkmZ/RzU8LXzhyrXhyklq1LVqpN5il7cwIs2BOcwVASlCIybe6LXzd8xynvhOWFZmF58lThOHiKZZJV8gqj1UoiSJJaIjSv7cIukpX/GqEqZQYAR8V9UoDpsQts3bDNwbayWVINnIfIUHXzquuKBFN6dcYWdrFC5hWHYr0idhNFrWqIhbZsnqWM8LlenhZfcdWh58DORMMiXOMw2BGs5t/sguk2AkRGNZZxAE8c9Ta0AJjuFwd2ipnvxeeAauv1U8Q4ZOUyUXN/GvDvksmfcJv1xJVsFHIdpTSQEXX5LQ5828U6KrEfyh8Mnq1R3uAg0sNCGavWsgkD6zCTk6pqc1xWag/ESubdhNy3UjnE1xVtULlRYN6rZJuufSvfMKfDFbtM0LOg7lOYApXLH25IN5wuHwu0XsAHexnzjS0TZHzKMv2pPln17VM4N9coE/bQX+ess93q+LHXglTUnfRPYbrl2D4sejR3Ajnx/zWfvOVyjnKIS7Zjm9HmE7q x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(54906003)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?PYGyBQogR38dbDsjRIQk2fBpZN5C?= =?utf-8?q?B3bmc3l7JPU6xr9Z5bu/QNx+Oy4NQUPtwVMxzODRXU5Wq+aG12uhHY+xAbnWnIA/X?= =?utf-8?q?Yg+iO/TKBx5+tl+T05LHbEwfncxaWscx9XVxS1Fdx6Ti87yrHLAzOUTMAhzAyd2YD?= =?utf-8?q?VJXTvm7tTYY9pHqwyyiS7Z47Rfz0Rv9qDt35RkiB9QOER9xdNX0jhz1GlWnHIigvc?= =?utf-8?q?FtISNe6iGZ0aGR9UVAR9bTZiEXHln3zpH/1TMyLNxUm78no07R37FB4q0hQA+NiDG?= =?utf-8?q?mx4Hy90vJUahfab+lYInr820ltTMcwtrZyWlEHZ4yYsGg66diqzZ5p7ndj8pEkFqq?= =?utf-8?q?uKmz+til3j8+nwB/0KTAZhdzJY1ogcFROVgX/kVrh2UTD1Hunwbn3UuUxE+kz779y?= =?utf-8?q?0eCWqDcCkh7NwuarrpZSw9C7powIPgcl4roRz/VkVwSYvE5wRfNKjP+EKbkTRjrZU?= =?utf-8?q?AGOqiXHtISQJKPhqTH2ceNrPl5XV/qc6R0NutjfMpiyIKXsqYOvOMoEgdLkg5S8at?= =?utf-8?q?bShaduAmlPSMz1P+AhGgFPNpMywGkc8pXin8IeM7euGwGJgmYvdZgEYY30pIsEstf?= =?utf-8?q?7bZSRrLbMX7+24OO1/HMjjZeYE3Qr6wOK5TM5OqaI2pYMOYpkaCG1rT8KuMUoxbeH?= =?utf-8?q?VSkHDEdtM/kqsXdjZ7KblN7RGMIFrdyNflbgemOEg9iaOU437U2rRLlaa/177c0w1?= =?utf-8?q?I7D6VZ4vFcHNmsW+Al/kQDCKTQqJI+AmvjxtDpB18HqaEYReQEzZqBzeUFQXBK7BN?= =?utf-8?q?qVoi8qlzElUuFxWXK68MHkWzjCo5OIuIyb8LYFpdFBhulQ3QlZRZcZMB5aQL14iqi?= =?utf-8?q?pdr+q+ZN+cIkgv8O8RMweJWK1LT2NyU0VA4x61l6vjcpzxkKiJHVeu9n8vdbilZPX?= =?utf-8?q?kewqE5P2lGqZEpX+sBHNWUvojIE0g3N8gMortJD8nMTG6XOF9AgTKgwjqMrOmK/bp?= =?utf-8?q?yjo6igz5MftJ/CJqC4wDRsxwHEaq4m3CyP80TmoC1JxO6p/l7sGTYJVYnzmHeylsV?= =?utf-8?q?23LC19S0RgtELT9o3nnGkWYb19ITFA21RPCrN82nCofcyiWj1Apmdj6uptKr9hxkF?= =?utf-8?q?fEAcESEM53xdtCfrjzYlWOr14xMUf4JkUWb2uLIcwIFD+OummTJxAcGwzDD1TpfWn?= =?utf-8?q?2qpDNJ9WW3qbBKhYbh1wVS6pEiPpEz1Y7bJMoFx+EkzN0eWJojYFEb+sClClHp787?= =?utf-8?q?dPUk6/G8D1eRXG3ALWvYMC8oViKfpDZigemleb+sHtRnPxa4ffrzb5X5zFcFs2w+/?= =?utf-8?q?rvYCgKHC1L6X45yd3ofMXbrcLVfmhAobp2jjrcc+VDD1Q7yAE7PnAZDgvC1E3X/A7?= =?utf-8?q?A9LyNyaGeeDDMXOoghUMMsE1mBUKHwcV21xoHDBw6ZDhcyTxfurl6nXyoXX2sf085?= =?utf-8?q?qzrLxptVh5xjCd1keOLxtMsb/TNeRLmByIRCgd1Dtggl8huF8BUbiT8R3NKVxJnzt?= =?utf-8?q?M27hXmz8fhdLkyrkIhSCDZFWsncbgDRTc90ZoyLbyKYiPqgkC51tvfoI4Jl3nTz7n?= =?utf-8?q?iHNK53ULBi8bOQ2OhM1vAaBI3W+OsWItUg=3D=3D?= Content-ID: <513FB179A8D41C4180D34D81C8EA2D34@eurprd03.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 079fc536-4bd1-462a-cae4-08db88b8cec1 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:31.6418 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: PMfP3jj2zbhu2X3oFr26LYJtxUYAs6EFDgJAURJWY2iAk93qvuH8CrFJU99K8IQAExkucqwW+DjgOAhjQLyPCABVWZg7KRFTSUGHJ2Cykc0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: 1Ip4MEnJV-84fShS-NmeYKZyp_1jFr8g X-Proofpoint-GUID: 1Ip4MEnJV-84fShS-NmeYKZyp_1jFr8g X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko A guest would be able to read and write those registers which are not emulated and have no respective vPCI handlers, so it will be possible for it to access the hardware directly. In order to prevent a guest from reads and writes from/to the unhandled registers make sure only hardware domain can access the hardware directly and restrict guests from doing so. Suggested-by: Roger Pau Monné Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Roger Pau Monné --- Since v6: - do not use is_hwdom parameter for vpci_{read|write}_hw and use current->domain internally - update commit message New in v6 --- xen/drivers/vpci/vpci.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index f22cbf2112..a6d2cf8660 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -233,6 +233,10 @@ static uint32_t vpci_read_hw(pci_sbdf_t sbdf, unsigned int reg, { uint32_t data; + /* Guest domains are not allowed to read real hardware. */ + if ( !is_hardware_domain(current->domain) ) + return ~(uint32_t)0; + switch ( size ) { case 4: @@ -273,9 +277,13 @@ static uint32_t vpci_read_hw(pci_sbdf_t sbdf, unsigned int reg, return data; } -static void vpci_write_hw(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, - uint32_t data) +static void vpci_write_hw(pci_sbdf_t sbdf, unsigned int reg, + unsigned int size, uint32_t data) { + /* Guest domains are not allowed to write real hardware. */ + if ( !is_hardware_domain(current->domain) ) + return; + switch ( size ) { case 4: From patchwork Thu Jul 20 00:32:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E6E2C001E0 for ; Thu, 20 Jul 2023 00:33:02 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566259.884879 (Exim 4.92) (envelope-from ) id 1qMHb5-0002Q5-Uv; Thu, 20 Jul 2023 00:32:47 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566259.884879; Thu, 20 Jul 2023 00:32:47 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb5-0002Nt-IA; Thu, 20 Jul 2023 00:32:47 +0000 Received: by outflank-mailman (input) for mailman id 566259; Thu, 20 Jul 2023 00:32:44 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb2-0001JI-10 for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:44 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id f01d57f2-2694-11ee-b23a-6b7b168915f2; Thu, 20 Jul 2023 02:32:43 +0200 (CEST) Received: from pps.filterd (m0174676.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JJCjt1031890 for ; Thu, 20 Jul 2023 00:32:41 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2059.outbound.protection.outlook.com [104.47.13.59]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgcw1sme-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jul 2023 00:32:41 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:33 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:33 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f01d57f2-2694-11ee-b23a-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cXJRK6rwzW5HMe0sgsb43DAcB33at01X+Mr9WQROcdlhjpnDWjCXP9rmSLP9IfKPQxmP8gbU+g4OwmpUrgfXPnOewx/YS5qc6kHlNOE6cviAR/RTNAK3+Fxj5RUDw4j5u4yt+gKoDN6eHCp4EJGenWFP0TAYswajCnU7t/heLY1aXuMP8DO/Rl2FCT04c6+Tr6TNBpsCmZD1lmHzBuCM8ayQoByIajPXiwJAaHTEyR8CPqSbH5sTRrhV+wR1EmBcRWWypGslwahaprGUcnOExUGmPLzOEDf3ckQegFUrHtDy3OXLG2Jnhf7T1qMSI06pjLmMs4QRCpYb18oMd2JEWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MizkqGgd+iTF6D6kSkQzhVLSHfvXaFyO/O67Z15pDNc=; b=noR6Odkjrb8yHjw59rTKh3q9h3eJIPW5H99jRtID8J/VsPt/SJ1/2F9l93B9gyd0KHi+cL8YQULWCXGLpuPIohNEb8K20HKr0+C3WAkHlyGU81D/so6oFVxqeTk5MtRgo201FeQKmjwOpeA0rtzXa6mS1pCbA7gm/Bd5Wllj+akDGkqCv23HHGa2u4ZPnUMmDT3nRm7PvHN4hCklPvQB4gSkriOBNJEXTX7pmHypmJnxNPlQU80fcYu2xJpsGwNdH7fxp8iX2z2Zv0A0fxSSGCaG/BscYu8GaK0AVNj2MuG88omCTxyKwStbIOEtzWV+OMZQ/6C6onX6XSd/kfYx9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MizkqGgd+iTF6D6kSkQzhVLSHfvXaFyO/O67Z15pDNc=; b=T+8RkjQGZ8TlgG3f0HqxwkRRnsDXAageIIj5iOjPe1JJFwhGvLNyJ5Cd5C4NdSqp6XtXzXBbxP5aOHOOxyMgFRSKfLTqHxsc04r2ALCfyxDZwudCezltZKZAcW8F3QkSSof7r3vnogI/jYmj+ncIALs3L2mW8ObNLeRoxQs5yfruoFG68oAapF8cB3XUmHREQvD7xDJksmwVVX8MZ0xnT9mttjlHwWp/VXMwixmKVjlaDztisd7TW7oH/sUKX+p0NbIHEhsIPG4r82J0Wqv814WBzMQR+if1VfzKbKabO+RRLA1ba+2aF5dzEn6xY+4UEtk/EoUZmbShctlwXlfu1Q== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko Subject: [PATCH v8 04/13] vpci: add hooks for PCI device assign/de-assign Thread-Topic: [PATCH v8 04/13] vpci: add hooks for PCI device assign/de-assign Thread-Index: AQHZuqGrBGIo4QxjlkuKy7Wb3D/bwQ== Date: Thu, 20 Jul 2023 00:32:31 +0000 Message-ID: <20230720003205.1828537-5-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 80f4905c-4e6b-424e-0264-08db88b8cefb x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: CbkOUNSSwFztXpVcFoBsryppOmW6mMhJUjydlAhR2KvG51i2nh8KYn999frhttOQD33AXPUEG2IGMKDhTLRg4hlJPWB853VC6IZB0yVK5b/G9xMHouKFFnZGwH3ZiJ57jYik1peZTnNB+vEYNXUnHkm4EPHIzzzZYSy31OwLxOmFIrA2Y+LUxNOlNj2sgo6BDsr5pNRtv6flHE3NUzYXqjLrOChVv8T0XyrMuYeOGh2Nn3b3XOdQXnfG2+6A68kYEeOVy+oq/K++Os8K9OTHrUIELBlXZATTg8RCeFrBGDbbW7t+3HvugoAGzmUHLNlP088kTEPJvYFVNwj6ameLI12Kax6CO9KnZD8ZEtZOW4bPM7xylGrCDdeOyZSngiMCnAfETJJ6ZhKUhSD+E3zF/soVZEabCS73UqJDJ5MSYuAKBIGhQ6+XgbGLtk/h/Mus3CiW3q25N9GBIiQNVK98q0k7IdfjOeRwyVqgtKyV84xgaQPUPHPf/p0DByFlMDUa9I1LDiS46DYFgxEk3HraVr8zF0+T9jKlO+G6VNxt+xK1T1FBGXOzAhA4C71vKIUF6Gx8PyaIMyOPu4XoCwTooeSAM6+IMZ/4rYoU29Z6SFDbJW0D+hI5r0KdhMk8mDGE x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?FnsETNGgYgV3EtAqO7rg0TL?= =?iso-8859-1?q?K184/cbwvgu+zIkBnJkdAqZXCNPANihWeXSas/PqVY3aHBGOJhRESpwL6a2A?= =?iso-8859-1?q?ZT7EocJ7SphHI+IyaypUwX4CLJKdwC7EPTAofAJmicrrlzRm1cJGeAS+O+Kd?= =?iso-8859-1?q?/da5hgB5lB0KLklEmOIniklS1VULM/MF1CSNsYMNUekyWHmQScZMsDn5U8L0?= =?iso-8859-1?q?ykFiLYsoayBO+F8ZcdPtIaQ8CsxPeZcaab95EV8CVxmtH2fHXA6KDNRCpQRw?= =?iso-8859-1?q?EI5JvTgKmQBRL5ZM5P9q4MLg361DStrxf7eumf0CPtKdzQNYbVzd8Oa2W00K?= =?iso-8859-1?q?lVPssMcsaIRuU6GtAFvWApCH2K9O1EQEchcTr7zm+Rtt/sX2su2bs+G+33CT?= =?iso-8859-1?q?w/GeWBTOjX/4oP7I22QHkLHKBi4sY9pbGEUwCVaFG+AW1Gdo4uSpaFQLtqFi?= =?iso-8859-1?q?tKfDYKQi+9LgyB/kgVfmlVT6XwC1SXjjBVDh5m96A71Yk32P9m6pla91zoFy?= =?iso-8859-1?q?+KL6s4PGMFr3EI1td45fEQs4T6UsN4bAHCYisGyv5KbQniRKpcw0Gm61epAY?= =?iso-8859-1?q?LCknmA3ov3Cq140feIFW7FX14sFnXvjyKXHZyaccER1Q4tTOygqEcIfHwKkb?= =?iso-8859-1?q?wvEYcyeMaUy18ibMsVFSEcQXzClslsI3X6ExZKG4JRU3z0cejKSnd6PtCnK0?= =?iso-8859-1?q?3gn3yFquofHfk0oHLZ3vrTNbQMLaqj+UDpd5GqhhddzUVKzntyxEomfWka+j?= =?iso-8859-1?q?WTjw+iYYqe3+ALeCpXOS9WoZioEtpoMmq+c3IHzxCp60bWsazM84cSFp0iSZ?= =?iso-8859-1?q?iIODqhoquqc9pLaYtpXWmKNcqj1JH/kSAVVyx0g7vPLaXsH2ovIIuvkk4V+T?= =?iso-8859-1?q?er8VHHRfFJ4YA3Y8jzwAuMTS4kXqb8Y0SGdb6yUZ617oJxpLYOlfjM7n7cCa?= =?iso-8859-1?q?/oURI1EkUOSuFfKynxXOLENnSuCY+L5pKwckuBJP5rlF3JKWTnR9Syu0L+6E?= =?iso-8859-1?q?/i4MIg8sgWzaJQRo3M9ywDiIsRYQ6Or1PGVXAH/B9UHymBjoea9+MvHkT3Kr?= =?iso-8859-1?q?DiAOFLkzviWiJVsabk54vEHf45/hpZOcFRp1DoXCJrJO39xymIadmMi156tl?= =?iso-8859-1?q?g27xFr929XdrE3bVVwzMxAHMXbTMr9AXz3XMm5y0pjQsk5RQ1GlnjC6ti609?= =?iso-8859-1?q?JY54HMuhSl/FBY8Z1IMQoTvGSwg8WaHf6OqO9gH0J6o7RlimDFFmw3exq6Ny?= =?iso-8859-1?q?ftmcFlFIRDyvYG2CgnBDu+TT0+thxN9ZKt/VwnoE7emY4IuzL9zcqbxItBJU?= =?iso-8859-1?q?17tLGs6yIEacA7/GHXg70tmnwrw7rDhSjqk9OlH/mxB25eC4EMi5aekpfUf8?= =?iso-8859-1?q?w9RgNKBTrln1g/YvLEhZe7NSvDrgWzokbOiumt2CeASBPxrLE8DMkVkONhCr?= =?iso-8859-1?q?TTP4CNBWqtUKPvt2qtp/GUpVdlIiQb+DjhyPH7UTHEunQ6eoLoHh3/FGDPeR?= =?iso-8859-1?q?lwIike6lmbr83Rx+wfVsOzcyCTeuGon77Mu3eDk0WMbXETypX6UhaxJSwRpB?= =?iso-8859-1?q?qa9/nS/Qt7Uo2fR7jtNrSc6yf1rgVG0N0MxUcCcchUHkaZUfPUDvly1dakUl?= =?iso-8859-1?q?ITVf+JAVjIVnbWXs6F+Pla6itCj5F9uskikocVQ=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 80f4905c-4e6b-424e-0264-08db88b8cefb X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:32.0060 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: PW7Av36w37QxXiylXQkqnmXRHwqPHay44pjY8rUnvXiAWKZtbuKkTEsOC9cNqc170y+GKtcwosZND05F2FT3c2Od02Lkf8u5wBeNYzi81mM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: yiqGwBQ8nkusslf8AHeYd76_d-OHqROd X-Proofpoint-GUID: yiqGwBQ8nkusslf8AHeYd76_d-OHqROd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 phishscore=0 mlxscore=0 spamscore=0 bulkscore=0 clxscore=1015 impostorscore=0 adultscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko When a PCI device gets assigned/de-assigned some work on vPCI side needs to be done for that device. Introduce a pair of hooks so vPCI can handle that. Signed-off-by: Oleksandr Andrushchenko --- Since v8: - removed vpci_deassign_device Since v6: - do not pass struct domain to vpci_{assign|deassign}_device as pdev->domain can be used - do not leave the device assigned (pdev->domain == new domain) in case vpci_assign_device fails: try to de-assign and if this also fails, then crash the domain Since v5: - do not split code into run_vpci_init - do not check for is_system_domain in vpci_{de}assign_device - do not use vpci_remove_device_handlers_locked and re-allocate pdev->vpci completely - make vpci_deassign_device void Since v4: - de-assign vPCI from the previous domain on device assignment - do not remove handlers in vpci_assign_device as those must not exist at that point Since v3: - remove toolstack roll-back description from the commit message as error are to be handled with proper cleanup in Xen itself - remove __must_check - remove redundant rc check while assigning devices - fix redundant CONFIG_HAS_VPCI check for CONFIG_HAS_VPCI_GUEST_SUPPORT - use REGISTER_VPCI_INIT machinery to run required steps on device init/assign: add run_vpci_init helper Since v2: - define CONFIG_HAS_VPCI_GUEST_SUPPORT so dead code is not compiled for x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - extended the commit message --- xen/drivers/Kconfig | 4 ++++ xen/drivers/passthrough/pci.c | 21 +++++++++++++++++++++ xen/drivers/vpci/vpci.c | 18 ++++++++++++++++++ xen/include/xen/vpci.h | 15 +++++++++++++++ 4 files changed, 58 insertions(+) diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig index db94393f47..780490cf8e 100644 --- a/xen/drivers/Kconfig +++ b/xen/drivers/Kconfig @@ -15,4 +15,8 @@ source "drivers/video/Kconfig" config HAS_VPCI bool +config HAS_VPCI_GUEST_SUPPORT + bool + depends on HAS_VPCI + endmenu diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 6f8692cd9c..265d359704 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -885,6 +885,10 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, if ( ret ) goto out; + write_lock(&pdev->domain->pci_lock); + vpci_deassign_device(pdev); + write_unlock(&pdev->domain->pci_lock); + if ( pdev->domain == hardware_domain ) pdev->quarantine = false; @@ -1484,6 +1488,10 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) if ( pdev->broken && d != hardware_domain && d != dom_io ) goto done; + write_lock(&pdev->domain->pci_lock); + vpci_deassign_device(pdev); + write_unlock(&pdev->domain->pci_lock); + rc = pdev_msix_assign(d, pdev); if ( rc ) goto done; @@ -1509,6 +1517,19 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) rc = iommu_call(hd->platform_ops, assign_device, d, devfn, pci_to_dev(pdev), flag); } + if ( rc ) + goto done; + + devfn = pdev->devfn; + write_lock(&pdev->domain->pci_lock); + rc = vpci_assign_device(pdev); + write_unlock(&pdev->domain->pci_lock); + if ( rc && deassign_device(d, seg, bus, devfn) ) + { + printk(XENLOG_ERR "%pd: %pp was left partially assigned\n", + d, &PCI_SBDF(seg, bus, devfn)); + domain_crash(d); + } done: if ( rc ) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index a6d2cf8660..a97710a806 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -107,6 +107,24 @@ int vpci_add_handlers(struct pci_dev *pdev) return rc; } + +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned to guest. */ +int vpci_assign_device(struct pci_dev *pdev) +{ + int rc; + + if ( !has_vpci(pdev->domain) ) + return 0; + + rc = vpci_add_handlers(pdev); + if ( rc ) + vpci_deassign_device(pdev); + + return rc; +} +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ + #endif /* __XEN__ */ static int vpci_register_cmp(const struct vpci_register *r1, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 0b8a2a3c74..44296623e1 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -264,6 +264,21 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) } #endif +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned/de-assigned to/from guest. */ +int vpci_assign_device(struct pci_dev *pdev); +#define vpci_deassign_device vpci_remove_device +#else +static inline int vpci_assign_device(struct pci_dev *pdev) +{ + return 0; +}; + +static inline void vpci_deassign_device(struct pci_dev *pdev) +{ +}; +#endif + #endif /* From patchwork Thu Jul 20 00:32:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92537C001B0 for ; Thu, 20 Jul 2023 00:33:09 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566258.884864 (Exim 4.92) (envelope-from ) id 1qMHb4-0001yr-Jb; Thu, 20 Jul 2023 00:32:46 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566258.884864; Thu, 20 Jul 2023 00:32:46 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb4-0001xR-7d; Thu, 20 Jul 2023 00:32:46 +0000 Received: by outflank-mailman (input) for mailman id 566258; Thu, 20 Jul 2023 00:32:43 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb1-0001JN-CE for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:43 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id ef07d056-2694-11ee-8611-37d641c3527e; Thu, 20 Jul 2023 02:32:41 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiB7017190 for ; Thu, 20 Jul 2023 00:32:39 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jul 2023 00:32:39 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:33 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:33 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ef07d056-2694-11ee-8611-37d641c3527e ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m4IAAEfkzQsbvRlDDa17C5fdwQkIDR8WAyiLrE65raG1Vt67H8W/ArPD72U+7YVRq0vc9FOb3Z8IwXJJdtD5sWxdIzntPVCr/h7PhZq+yxmVeCyZ7qa4IxCipj70HJTsRUNYNfxAz3Dr3ovAVemxPk47AS3XRbxNcyKxoZReQBe6CTTLcN11O/j7FoILajeKJAf/PV6n0MY0vc1ASGoN/EZ32COFCB86FzC6xgcSe413Yg+YqFGSfbmlkFHTnwnuSdn8aHPp4wOIkkTe72xoOP54xSyc56TGV4pWDgKp5Xr32m8AEeJQYvApfn2nnci2HcPSIeONX3fjD78vuRZflQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=47oYdY92/xdhz0lVCilDwn1lYzCxtQyRzltahBREZd4=; b=LtCfVutNjdvZIKZOEP4ydcqXJvrobo0WcZfED8ssJCDkpj9yTTTwwrJt512udvDqNCf8TT3YGqj/bW0u25VMF70PyhHYhV/h5xjuXaUSI+naKrjaYG/m/3rsLp9okzXuUp4T2SkoNvdBymsOG6WmC5GLSmdm/YVaRwB4quvg7Jx55EzTLWlf9PLByeE1nhACABjQPnckK1Q8bmDv9bS3FvHS5fS7r3oh5qD3E5EFBv9GcK3WSfCI/cAziEwH0I3lLSKvONiTp/Z/eLyV2HFAKE/rcnZllUV+KMgT2PLsOh7VRbz+lhlr57qZpA1YSYEpCjkx74ZYQ+v1sflo9ylQvw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=47oYdY92/xdhz0lVCilDwn1lYzCxtQyRzltahBREZd4=; b=Avez4eDF3dpAmhUCNnbHldRA9tNFpHgdsFjvLvuZqKih3Y2WPKBlG5Xf/AhIPAB/D3GyVgBZU/nP0OgfrQ+UyCrOAhDQYSns0+aTBKLKYW2GoWAEQkDE5tWUI2C8QtvrA4AGWE4swCuZsAcTKphZdDQ3Q/pWzRsAknkz1buTigAE7kHulJrss0cR51wnZX0kApTeNRnl+YFZJ6Y4LWQa2virAMDwOYC0oFMxF4Rfs+iZFMU/Nc7ZDFOEIn0YJinnjjW+4PXWeTbA2GTyBxCQkIEjgOqA3KB/lmKnhLooUjHkaDcVHmaAIg+qBB2CxHxTvN87snyESeX+R6UMvty9vw== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko Subject: [PATCH v8 05/13] vpci/header: implement guest BAR register handlers Thread-Topic: [PATCH v8 05/13] vpci/header: implement guest BAR register handlers Thread-Index: AQHZuqGsyq2xz4eSA0OJuZWqMM1tiQ== Date: Thu, 20 Jul 2023 00:32:32 +0000 Message-ID: <20230720003205.1828537-6-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 14347ab2-45a8-4005-b09b-08db88b8cf2a x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Nf5JftkPigHZ3pofGPDnsayT2K7vwxoRLIA7KiuSWTOoi2nMxKEnmJ6SuZrVQgjYHi09Vbz2aocIS4r6bViEZPlCUkbAv+jASBRjGGFS8CMmxSKhGVvN40SPSiAJ9Auuajt9GOCWFMO+L95JVjpX55dBGiA+t+lsZgM/2R4B+5micq0P6ho+tEEXutNBFZtWHVOCXYImCORi0cE7EtqOK8o1KlgruEO/puV5vbig+S7ypVBAORKChyIlOKa35n8NqDjg8G90LIV2LhQnskXgBv7OCbjYajXccNsrvLISACGI2mk+HapHU4a6Y3ZEMa9y2Khaxmx1K8crdB6BV7dFVxwm+yIJXf5Q6j15ISON0t1CQIPVaS+wMUv2Trm+zITVqTrs2eHvshbRIWI81M30qLa3KnVSR/ksagK7dHeyFexG+jFR7+AsU/EW3bZxV/yNS1p3B9CnoF9rvV4ACPxHJYxPNWvgUw9VFLuk1E3w1j3WrTrfv1m1dKsrrj7HEF77Bc7mChzQFdWrTNAslJGeLzlWKAgX7DY/Nx7LVd5e7Dw4lkpVttt1qq8gUi+dBsfb8erJkMgs+AGSV4y22f5LYhEqhE1tlqH/xU14OobKSeTEwOzud2TJ1VsT4XZJTsVQ x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?fwnCYpu+qHtMGGkhWcbpZkb?= =?iso-8859-1?q?GH7mufrjWPqgc7GzAYpzYazsvyZ5LaseZhuGPo6NRCFyQYCJ8toHfuJVvmiT?= =?iso-8859-1?q?xiEeOugLsNTeYMjxyaPbiB6tn6n58DmKFyq9Gf+dRMKNqGV46DimWYC4jLZw?= =?iso-8859-1?q?SHwRjqCQP9zVHMkOM5UdkKSurbbDgkxU9NydCuWtUMRru0j0EJFNuJTSuFnz?= =?iso-8859-1?q?+jH2FLb+dA+NiBNgb5FIfMxgPGhspBwM/KmpzDtmawzC5JEYNjo28aFUVdWl?= =?iso-8859-1?q?l63D0RY6u1CzizBwK9iTdGiRP4jRto/W9zgaGFu+gbnqH+PJFh8Y/xlg+Q7Q?= =?iso-8859-1?q?jhOZF6ekBiDeqk7ppRftsk84iVzK22bnnDlH+KIUWRnb/DVvU/ucQwoYteeC?= =?iso-8859-1?q?EgVdAHtFpLJOdcxmzigqyKQs8A77GuLksarUsAh31vcOzFUV5x/84u4azmoq?= =?iso-8859-1?q?n6OLFE5OvSIb1CthPTNph6AE6HXLxODXuq9MU8L4wl+dYPsmYfZKSo8Yz8gp?= =?iso-8859-1?q?BfFKAK8QcQdPySdwoJ3beKcLZ5I4KM1k81p6nhEYssoHNUl0xX7EDLSdFm2E?= =?iso-8859-1?q?FX3sDXLlmFt3oboIa49M5nLIjSVCOeDT2zS8XILPvWIGFmoTPQhbt1T/fqiR?= =?iso-8859-1?q?8x1Nj5aum0M65Hjxsoi/3jCe0QPepATf0/GRzagMM6YxjLvExP5OSQ0Tfn2J?= =?iso-8859-1?q?qaNNJHpBM56eiiCFdEBurT4AAoG2kvp9GqHKrNBytmbo96Kb7GknTpFJVD1/?= =?iso-8859-1?q?+NQ7sUmLqA9HexP/gKG0pCa1vbLHZJwrfcPlRB+2RiaH5CNoQkwT0LvP7pT4?= =?iso-8859-1?q?PHux0hRAAn9q/F7gOLgVcCz08MwRi8BvRX4gmLj2j2/HHGYVaTZM2yjWQHOw?= =?iso-8859-1?q?mRK/3oDsPXTXAEaQ22s07C2+Afu72NdYT4KFrP+wm9wspHVZ0/otxjnI7khF?= =?iso-8859-1?q?/NdPecJfe0MmwRQqrn3F6Srdw+OHfZrAEnrIq8n4B6FX9/KXRadOEW5MY8+U?= =?iso-8859-1?q?bzn0xhMHqQty1H/yFKYYRrHNM+NkXuq1BGK3F2coqT+oFmPDrpxcWCjeFY2p?= =?iso-8859-1?q?wTmAJlnSML1k2XsZT0tCVQt3Yb3CYr7as/Qmyp01oXTp2YR3OR7pBDtx86dp?= =?iso-8859-1?q?jNNiWWDu80XKc2HUMsjro0ovuJZlT3oT0U0hwVmaRirkTcUkRVCD/JX43+sB?= =?iso-8859-1?q?GFruNThNS6NJSX0m227eEmg+sg1FQel4+fhYe9DWqbzQQsJgWM8QYsSF/TfA?= =?iso-8859-1?q?IDzmocRUPBY7AaOvUAy1z6F4IiaVUM4Zavk5G8gzaMx0/tbjfz2gbCoDpIdH?= =?iso-8859-1?q?OXYON7cGJ2gL+rDY721+RP9bywz4OMURYFFkTTP6DoRKeuhbLtx0hDoD86mH?= =?iso-8859-1?q?LXj2RvwCwTDQbBmzVNX7BzY/pEsMcQF1/235uyS6x46LjQZZwowfG/8Wl7V/?= =?iso-8859-1?q?8es5/6KpQh6hZHVTcTbMdYhW5wgfOeaX4m4hxVLaNLzvfJVnWVdjU7qayZS7?= =?iso-8859-1?q?e12A4z8IArH71/gWmOcuGsxeC2GD1hqjt8WFwWUxu4mVO4AtsPl3tNgOBZnU?= =?iso-8859-1?q?UQcGaZpnCeukzoyTfmWjQdtJNZNGZbGPdQBYtNMshELuBJ8DgDbhBsfCLmSM?= =?iso-8859-1?q?xwSOep9mLA3Ie4o0K7Gbain+O3fI8k5mP3164Ig=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 14347ab2-45a8-4005-b09b-08db88b8cf2a X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:32.3931 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: w8RWG2bOpi2oQU4HCv4RrlKof3RxWcLf98L+G+4Qk3EGlOHuK81EY1P876O7V8Wb8sbWWlKyZS11XTzRxZy8dpaEkdYMlCnxU5fMX/jBRMA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: KmXizPNq9VrH_mBgkg5SXfcwwxWWp4S5 X-Proofpoint-GUID: KmXizPNq9VrH_mBgkg5SXfcwwxWWp4S5 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko Add relevant vpci register handlers when assigning PCI device to a domain and remove those when de-assigning. This allows having different handlers for different domains, e.g. hwdom and other guests. Emulate guest BAR register values: this allows creating a guest view of the registers and emulates size and properties probe as it is done during PCI device enumeration by the guest. All empty, IO and ROM BARs for guests are emulated by returning 0 on reads and ignoring writes: this BARs are special with this respect as their lower bits have special meaning, so returning default ~0 on read may confuse guest OS. Memory decoding is initially disabled when used by guests in order to prevent the BAR being placed on top of a RAM region. Signed-off-by: Oleksandr Andrushchenko --- Since v6: - unify the writing of the PCI_COMMAND register on the error path into a label - do not introduce bar_ignore_access helper and open code - s/guest_bar_ignore_read/empty_bar_read - update error message in guest_bar_write - only setup empty_bar_read for IO if !x86 Since v5: - make sure that the guest set address has the same page offset as the physical address on the host - remove guest_rom_{read|write} as those just implement the default behaviour of the registers not being handled - adjusted comment for struct vpci.addr field - add guest handlers for BARs which are not handled and will otherwise return ~0 on read and ignore writes. The BARs are special with this respect as their lower bits have special meaning, so returning ~0 doesn't seem to be right Since v4: - updated commit message - s/guest_addr/guest_reg Since v3: - squashed two patches: dynamic add/remove handlers and guest BAR handler implementation - fix guest BAR read of the high part of a 64bit BAR (Roger) - add error handling to vpci_assign_device - s/dom%pd/%pd - blank line before return Since v2: - remove unneeded ifdefs for CONFIG_HAS_VPCI_GUEST_SUPPORT as more code has been eliminated from being built on x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - simplify some code3. simplify - use gdprintk + error code instead of gprintk - gate vpci_bar_{add|remove}_handlers with CONFIG_HAS_VPCI_GUEST_SUPPORT, so these do not get compiled for x86 - removed unneeded is_system_domain check - re-work guest read/write to be much simpler and do more work on write than read which is expected to be called more frequently - removed one too obvious comment --- xen/drivers/vpci/header.c | 156 +++++++++++++++++++++++++++++++------- xen/include/xen/vpci.h | 3 + 2 files changed, 130 insertions(+), 29 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 2780fcae72..5dc9b5338b 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -457,6 +457,71 @@ static void cf_check bar_write( pci_conf_write32(pdev->sbdf, reg, val); } +static void cf_check guest_bar_write(const struct pci_dev *pdev, + unsigned int reg, uint32_t val, void *data) +{ + struct vpci_bar *bar = data; + bool hi = false; + uint64_t guest_reg = bar->guest_reg; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + else + { + val &= PCI_BASE_ADDRESS_MEM_MASK; + val |= bar->type == VPCI_BAR_MEM32 ? PCI_BASE_ADDRESS_MEM_TYPE_32 + : PCI_BASE_ADDRESS_MEM_TYPE_64; + val |= bar->prefetchable ? PCI_BASE_ADDRESS_MEM_PREFETCH : 0; + } + + guest_reg &= ~(0xffffffffull << (hi ? 32 : 0)); + guest_reg |= (uint64_t)val << (hi ? 32 : 0); + + guest_reg &= ~(bar->size - 1) | ~PCI_BASE_ADDRESS_MEM_MASK; + + /* + * Make sure that the guest set address has the same page offset + * as the physical address on the host or otherwise things won't work as + * expected. + */ + if ( (guest_reg & (~PAGE_MASK & PCI_BASE_ADDRESS_MEM_MASK)) != + (bar->addr & ~PAGE_MASK) ) + { + gprintk(XENLOG_WARNING, + "%pp: ignored BAR %zu write attempting to change page offset\n", + &pdev->sbdf, bar - pdev->vpci->header.bars + hi); + return; + } + + bar->guest_reg = guest_reg; +} + +static uint32_t cf_check guest_bar_read(const struct pci_dev *pdev, + unsigned int reg, void *data) +{ + const struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + + return bar->guest_reg >> (hi ? 32 : 0); +} + +static uint32_t cf_check empty_bar_read(const struct pci_dev *pdev, + unsigned int reg, void *data) +{ + return 0; +} + static void cf_check rom_write( const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -517,6 +582,7 @@ static int cf_check init_bars(struct pci_dev *pdev) struct vpci_header *header = &pdev->vpci->header; struct vpci_bar *bars = header->bars; int rc; + bool is_hwdom = is_hardware_domain(pdev->domain); ASSERT(rw_is_locked(&pdev->domain->pci_lock)); @@ -558,13 +624,12 @@ static int cf_check init_bars(struct pci_dev *pdev) if ( i && bars[i - 1].type == VPCI_BAR_MEM64_LO ) { bars[i].type = VPCI_BAR_MEM64_HI; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } + goto fail; continue; } @@ -573,6 +638,17 @@ static int cf_check init_bars(struct pci_dev *pdev) if ( (val & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_IO ) { bars[i].type = VPCI_BAR_IO; + +#ifndef CONFIG_X86 + if ( !is_hwdom ) + { + rc = vpci_add_register(pdev->vpci, empty_bar_read, NULL, + reg, 4, &bars[i]); + if ( rc ) + goto fail; + } +#endif + continue; } if ( (val & PCI_BASE_ADDRESS_MEM_TYPE_MASK) == @@ -584,14 +660,20 @@ static int cf_check init_bars(struct pci_dev *pdev) rc = pci_size_mem_bar(pdev->sbdf, reg, &addr, &size, (i == num_bars - 1) ? PCI_BAR_LAST : 0); if ( rc < 0 ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } + goto fail; if ( size == 0 ) { bars[i].type = VPCI_BAR_EMPTY; + + if ( !is_hwdom ) + { + rc = vpci_add_register(pdev->vpci, empty_bar_read, NULL, + reg, 4, &bars[i]); + if ( rc ) + goto fail; + } + continue; } @@ -599,34 +681,50 @@ static int cf_check init_bars(struct pci_dev *pdev) bars[i].size = size; bars[i].prefetchable = val & PCI_BASE_ADDRESS_MEM_PREFETCH; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, 4, - &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } + goto fail; } - /* Check expansion ROM. */ - rc = pci_size_mem_bar(pdev->sbdf, rom_reg, &addr, &size, PCI_BAR_ROM); - if ( rc > 0 && size ) + /* Check expansion ROM: we do not handle ROM for guests. */ + if ( is_hwdom ) { - struct vpci_bar *rom = &header->bars[num_bars]; + rc = pci_size_mem_bar(pdev->sbdf, rom_reg, &addr, &size, PCI_BAR_ROM); + if ( rc > 0 && size ) + { + struct vpci_bar *rom = &header->bars[num_bars]; - rom->type = VPCI_BAR_ROM; - rom->size = size; - rom->addr = addr; - header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & - PCI_ROM_ADDRESS_ENABLE; + rom->type = VPCI_BAR_ROM; + rom->size = size; + rom->addr = addr; + header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & + PCI_ROM_ADDRESS_ENABLE; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, rom_reg, - 4, rom); - if ( rc ) - rom->type = VPCI_BAR_EMPTY; + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, + rom_reg, 4, rom); + if ( rc ) + rom->type = VPCI_BAR_EMPTY; + } + } + else + { + if ( !is_hwdom ) + { + rc = vpci_add_register(pdev->vpci, empty_bar_read, NULL, + rom_reg, 4, &header->bars[num_bars]); + if ( rc ) + goto fail; + } } return (cmd & PCI_COMMAND_MEMORY) ? modify_bars(pdev, cmd, false) : 0; + + fail: + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + return rc; } REGISTER_VPCI_INIT(init_bars, VPCI_PRIORITY_MIDDLE); diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 44296623e1..486a655e8d 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -67,7 +67,10 @@ struct vpci { struct vpci_header { /* Information about the PCI BARs of this device. */ struct vpci_bar { + /* Physical (host) address. */ uint64_t addr; + /* Guest view of the BAR: address and lower bits. */ + uint64_t guest_reg; uint64_t size; enum { VPCI_BAR_EMPTY, From patchwork Thu Jul 20 00:32:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D3FCC001B0 for ; Thu, 20 Jul 2023 00:33:01 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566257.884855 (Exim 4.92) (envelope-from ) id 1qMHb3-0001nH-Sv; Thu, 20 Jul 2023 00:32:45 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566257.884855; Thu, 20 Jul 2023 00:32:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb3-0001lG-KK; Thu, 20 Jul 2023 00:32:45 +0000 Received: by outflank-mailman (input) for mailman id 566257; Thu, 20 Jul 2023 00:32:43 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb1-0001JI-5H for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:43 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id ef7930b0-2694-11ee-b23a-6b7b168915f2; Thu, 20 Jul 2023 02:32:42 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiB8017190; Thu, 20 Jul 2023 00:32:40 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 20 Jul 2023 00:32:40 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:33 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:33 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ef7930b0-2694-11ee-b23a-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YSEuixspl4E2XEB56BGXbHZXRo4kb6jXMQAXgyLmNUE9vkTINhe7m6luyKjgijja+aiHWfjuFtNQ1BByFjlRrVicA2+7p9SqmkRDtrgO3+uxiq5WeghW3SpZwKE5XwASXhwXckWO14+ZgtFyWe+l9yNxup1ZkHYiBYYWCkoE+b801hBZlf2kybXPh1hWDoYwD2So0zudO5turdYPXdG7pCA19aOOYnYANU1f2dz51g/kr77hvFmzjFOcmCmDvq7mkxeNgW2zTLBcvQTF84xIuDxH1Lfx8NQKWkuEwNT1J2++5p36L/fmbAJwPlly3HM019fsYwJCfCKtSHtw2L2LUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pnyVf/Quc1OthOHa0YYbOyyYyHBFi9n/oxj2Krygg6Y=; b=mM4C+h2U1tqNWIRHFXzDEpV2J6p5+d21YvbuSmYPKDICPIYj7XkmXnQziVtsQeAVFgJRM+Jdd1s1M4WPtccgXKyLCHRssaeuhIt2BgTz/p/KgDMMAp4OTqJSx+Ls74VlsbaC3QVRBPkQJ6jNBNdCZf8bZooUlEp9gVVvKX15XFHL5g7uFbcKabGVSos0eC3xOjFQ0u8roxvIPMUo9gtworAOF8g4C/S0xBNMO3T4+F5rUPmlU9Pttl5JxON2yILcI0tgs7voajnJp3oXAcljcCMq5tQNpELSKuveKkkYSLgNeqkAY8C3zyjJSJ55GgBP4bZzNjUaAmrKG7hhWe5y9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pnyVf/Quc1OthOHa0YYbOyyYyHBFi9n/oxj2Krygg6Y=; b=BxrKeQvrH7h/FwkxuK8ToDD09fsF6psZJw7cOl+3dU/PrHWNPOCFmTeszWRjBPaXXK1H4pmF8yUsfxrxdMoJ4vHIp7oWfBOjRCUdBzSggYAaWvC9V2NNcVsSOWFPan3hdZZJrSRstmXAniMuQsROjEr53RjQJWhIczFSgV408ynpwfvTQz97crpMZs6LAIkCqYQdFEL3462x0d8prEfuDkazABjsh+b18TCo8ldP8loIynOHkJqv/r9QtCVs9/JMXsxDjlndfCrQBY3EAjuPwGAe2oTh9wsjShmG9f+ggLDoQJLfGhOqZebJANLnSa8EwWgUkbH9In4VlHYJuroI5g== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko , Jan Beulich Subject: [PATCH v8 06/13] rangeset: add RANGESETF_no_print flag Thread-Topic: [PATCH v8 06/13] rangeset: add RANGESETF_no_print flag Thread-Index: AQHZuqGsAISBZpl7eEC6l+liOUz9ag== Date: Thu, 20 Jul 2023 00:32:32 +0000 Message-ID: <20230720003205.1828537-7-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: d09c8808-874f-4f42-e819-08db88b8cf63 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: NPmnq0ySwCmiCn7Px5F95uf8HA3AJITLpfPTnSGVBO26nPxxOzay2MCT2SWtYfkBQOD7BzMAMczY/BjFZ72NHL8anyOjW5wnpYZBlUdfyTejiiNwXiABfvYEBBWP3kQFLtb2AJ8WfBn7LpMWIZUgYC1CoZsU7Pz93R17zUYXeoTDvaBLb5aUtKDbnaXKgL5aMnx8au+KT88IGlLXaMwCSsbAhzszOCMFO55VoGok8TKAYBr7UVvSf6ZpFCgrqibenOBkBO/LeIUdCzAKYxTCoDuLs5JZPkhmxmWtOX//39m9UECwcBF/8z4rSBj1b8GcjJVrCQHNpbtSo5MJ39VXt2IwvQY/mQNfu12LdUSt0EjJe5ccagkB3yO7ezzxNsq0al89WZWzgFeem36k3qpZZf/fQBVJbx7G7ZfWtEw7kLPd3fDRviaPWieCbZ+VhbSF2PwjHgXzjlrXyh5fFA2SCiFODhJ/FRA/fmQhMHvi2PN8EXeGGCJG1UxAvh8CaC57PHoB5lR5QsOdZk8MYlxaVovgaN01JFciFHCiCWxJUD3jA7cXTs50mw6FlYMWemOEOz3IxDERpFuelUNOsHqkm64aMMoeN4C3kZHw4ZBJO7nrcpDAnJ+1ZPTKM3bQ3uOZ x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(54906003)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?UY5sNfrWZwe8mPRTL1fI8l3?= =?iso-8859-1?q?/3uBmZ4viGlzjrgH5NMfGnEpq/p3sYzz+EPoJ2vsXDa1RgsLwNYEZd+O7emI?= =?iso-8859-1?q?HPQXxqM76Ko/vJZ7JNWE538QtVklVN/qM/wm7KqEoaFrbGqczMzak6YQFhJ6?= =?iso-8859-1?q?BBimqtS3VaKpZKCHHR6tmMQ/QdZqNZNuU6u44O7W08rrg0p0/r3fNGYd9n44?= =?iso-8859-1?q?/Ji241/u2clPl+a6mMcCOgrHhzKJwqGVlPRx4nnIVf2QggYKaKxNK2uu0xyj?= =?iso-8859-1?q?oACvwnp00hwbUPFC2WknERWkuAK5btVqHl2fSC8XY4mn7n9T+v9fygTflQ/V?= =?iso-8859-1?q?6ElqEGcH98a2kIKPQhU1I6xW2BIyV4LmeOqET+9VK91KF58dNJPJquZgzdjo?= =?iso-8859-1?q?fpM1VfLhpNGIW9L64i2judoLmQbRf+4k3PT9/4od0z6P1ZkkQeMm67R0t2wc?= =?iso-8859-1?q?zy/QeAOJwaoEduVqZlXQWiPW2k6iMRDAEAnfz9VGelZwlDUDgnH5iaI6Isx4?= =?iso-8859-1?q?WJsvhrvad5tviQodczz2wONMLC5ukPw9O7qByMYlj3mrXpPBHA7PMp0tzYiN?= =?iso-8859-1?q?mXTcRiTBdXfsYVJrUcdmfBm7ENPbQ0A5ZBl8CijKFibdsF39ZF0fre3RKvNI?= =?iso-8859-1?q?7xHn2rcHOT4Kd/kT1yNE/iD7g6plKgw3q3liFIAE45CH8bGSnz98vgYeqQLK?= =?iso-8859-1?q?o7NxiHtVAU5t8Dg4wzieEmYBqSRwAf8nghSR3QB0686sI4UPWvG7LhLjXQgn?= =?iso-8859-1?q?bNVr5GbEqXlc9efXY/3eL/U20WaREkO+1GO4h06m/2XOc/TjDRiulnqQQp1p?= =?iso-8859-1?q?8fM+n4lqRMhQcJHzmbCIA7q3v8mPwmn+bokzi5UsbPIokiJfTemZuExrrQkz?= =?iso-8859-1?q?o8uRLXiw1V9ORvvhsOBk/cMFDn1nY2hYxaRriuACt1cbMr0I3j6ih8R7mOEa?= =?iso-8859-1?q?4E1J28/dMKRgSeLr21iFJpSt65Nr/buwPsvgXp0Rw6RCq/Iu8oVnXr/+0arp?= =?iso-8859-1?q?OKVClwE8Wr4YI1DiwWHpa2yTaONYPn2Db0EY1xb3gGG96i4nUR71a1bblhVD?= =?iso-8859-1?q?FCYPIbfnx8mo6daprYObQ6cPlylS9ZJWL21ad6HisCHTs20yKHpLaCgWIM/P?= =?iso-8859-1?q?3vAxmUWVwObgOh25vNDB6F/miZ5coY8h3FWup9xUEzASxMAwLgHCiUjdzUI5?= =?iso-8859-1?q?91k7GflDpl3dWwwsG8YHzXrp2NOAAeiDwo9opnjLkY62tq9bCSpXmRhrUeaG?= =?iso-8859-1?q?P1AerdPS8QSGdU0h9hr/84hcfpeUpcQBtCuWgAfb0J/CPe8AjoirkDxiZCaO?= =?iso-8859-1?q?uMJRf/upYEFoKNXoiTb8sPi++aDJZRyt5o6FgtICH6MFo8yOITz4rXxGoIJW?= =?iso-8859-1?q?Aa+366XDhtQpbdIWaocIMKODyJtNFhVqpIdVN4pUmg5CwyGWamaMEMzqjODf?= =?iso-8859-1?q?IdcnCiX97BvnzeWvpT7iIKdyzKwxBtXOwd2VZ6R8PkBTyGb2/yMlJ3EfTW74?= =?iso-8859-1?q?u1kYmz8JJc2gPFWuIYsEbqXm7DqC1R2qF+w9osUbj7eNdylq7Pc1OSL//SRE?= =?iso-8859-1?q?CjCh8w3aX2mRXNWm3gOMPa2tPAOt4pdhPxHY8r/MLPYQ8Jj5F0arJ5r0SY3l?= =?iso-8859-1?q?7FmhEhi89NBgV6IUTslhtOxedlXEErCc/VqLz5A=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: d09c8808-874f-4f42-e819-08db88b8cf63 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:32.6433 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Uz0QoHWLPj7IXhUmPUqWMrtT2H0W8ncZN1vSFhbVOWoAurBpxD4Tl1v9gva1xOD+yCfQ1mSNy/Yl3ma4f4Czf/DDvyuW1wbs9svYXSBVfc0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: wSC_5BsEzDIb_Nbf8daHlVd-fT0giIVi X-Proofpoint-GUID: wSC_5BsEzDIb_Nbf8daHlVd-fT0giIVi X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko There are range sets which should not be printed, so introduce a flag which allows marking those as such. Implement relevant logic to skip such entries while printing. While at it also simplify the definition of the flags by directly defining those without helpers. Suggested-by: Jan Beulich Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Jan Beulich --- Since v5: - comment indentation (Jan) Since v1: - update BUG_ON with new flag - simplify the definition of the flags --- xen/common/rangeset.c | 5 ++++- xen/include/xen/rangeset.h | 5 +++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/xen/common/rangeset.c b/xen/common/rangeset.c index a6ef264046..f8b909d016 100644 --- a/xen/common/rangeset.c +++ b/xen/common/rangeset.c @@ -433,7 +433,7 @@ struct rangeset *rangeset_new( INIT_LIST_HEAD(&r->range_list); r->nr_ranges = -1; - BUG_ON(flags & ~RANGESETF_prettyprint_hex); + BUG_ON(flags & ~(RANGESETF_prettyprint_hex | RANGESETF_no_print)); r->flags = flags; safe_strcpy(r->name, name ?: "(no name)"); @@ -575,6 +575,9 @@ void rangeset_domain_printk( list_for_each_entry ( r, &d->rangesets, rangeset_list ) { + if ( r->flags & RANGESETF_no_print ) + continue; + printk(" "); rangeset_printk(r); printk("\n"); diff --git a/xen/include/xen/rangeset.h b/xen/include/xen/rangeset.h index 135f33f606..f7c69394d6 100644 --- a/xen/include/xen/rangeset.h +++ b/xen/include/xen/rangeset.h @@ -49,8 +49,9 @@ void rangeset_limit( /* Flags for passing to rangeset_new(). */ /* Pretty-print range limits in hexadecimal. */ -#define _RANGESETF_prettyprint_hex 0 -#define RANGESETF_prettyprint_hex (1U << _RANGESETF_prettyprint_hex) +#define RANGESETF_prettyprint_hex (1U << 0) + /* Do not print entries marked with this flag. */ +#define RANGESETF_no_print (1U << 1) bool_t __must_check rangeset_is_empty( const struct rangeset *r); From patchwork Thu Jul 20 00:32:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F0075C001DC for ; Thu, 20 Jul 2023 00:33:09 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566260.884883 (Exim 4.92) (envelope-from ) id 1qMHb6-0002ZN-FW; Thu, 20 Jul 2023 00:32:48 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566260.884883; Thu, 20 Jul 2023 00:32:48 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb6-0002Wl-2w; Thu, 20 Jul 2023 00:32:48 +0000 Received: by outflank-mailman (input) for mailman id 566260; Thu, 20 Jul 2023 00:32:44 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb2-0001JI-Nb for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:44 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id f00b8948-2694-11ee-b23a-6b7b168915f2; Thu, 20 Jul 2023 02:32:43 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiB9017190 for ; Thu, 20 Jul 2023 00:32:41 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jul 2023 00:32:41 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:34 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:34 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f00b8948-2694-11ee-b23a-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nYqEJpvxPZBlTonCwr9TjNiNyZ+rpTnSAEfBNglKZtr188G7UwvqgDBw1xMytIqehwuY+b3W4OOY1nGN/GFOl9ZojPxhKoAnPEbUwHHjTiu5WQmOmZvdfQVFUGpXkthaC6yHvupJx9E3pHF1IQHZD23pj++M43EtNg1xQPeER6/3C/C463rchSDIAAKkd72PKj2l/EEQvnfIB2F0oh6Jm/gkmfkxOZ0YEUCO5OW3nd9SzPcbu51JO5Yp2Rk4ZEpEU+9a4yWIYP4yofLDH1dgDFhq1dn3bJHG1ar5FtLNSRimGM+XpGPK+hwCxpZSFLcpdNnlQFAKvlaRw6TbsgeFAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bfSYFj3qCWK47xmolGi8XD+FB4uZs6VedMBJamb2Ugc=; b=YW1GaM+YsJ5rRQo1Gi6tsw4/8E3YqmuEcUaRqIud8EgGuVEe14aPx7Oe178L0dWT9/AE0SonYbGpOttEYOUYaUy5GqSlBBgoLhS0T4pGmm+JX8/PT4yrQbUq27fDV44Agei49v2gufcNR40J1jGv1+ykgrUBmkyPDceQyGamCpWVz9rfWyQCNB+++mGieWaJ44k9uvjxLRPVoITKssXhwvJisCJKOyw9m2RXBwma/6P1XnhGkNG/0HnrFFVlvIGXTGYN/rTqHT31uMdqUunBqH3p75XsuGJHlUOGO7VlwAUh+/LZqSq27VyZ0iUeCKP6lajSjBXlrJLbfyFSe/ZpfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bfSYFj3qCWK47xmolGi8XD+FB4uZs6VedMBJamb2Ugc=; b=NPR0A6/NeRFbFCMCF/wH/kDzC2fdmn+ICsaLvWry6oOyoSP3AzmO2/nB0wu2zmCoPre3rGCo+AxlMkAqC9ZM24AAcTi+CHySrtU/gK6hHefL+SRVBSTD3MRw0rrCHC+890fIA1s5zIFKTysu9pu5q+UncorEPSNwxy14QWJpEU+gXiqS4478aOUqglqpkPIGRdLZDwiwzfDjFMTgD4oGXxgqe0Y8lE3XRlMDU1iwpFj7X6Cj34xZBaCIVpXEiZodY4B/9cO1KGFwJMNQ3qZ5UQwzop28N3Us6RrjxyH/vTFwgZgztC3BhYfN9iC4ovOyCY0a7iz0RMXRulLpOUBGmQ== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko Subject: [PATCH v8 07/13] vpci/header: handle p2m range sets per BAR Thread-Topic: [PATCH v8 07/13] vpci/header: handle p2m range sets per BAR Thread-Index: AQHZuqGsZnHxruBgBUyW03RT0neDNA== Date: Thu, 20 Jul 2023 00:32:32 +0000 Message-ID: <20230720003205.1828537-8-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 3e92be7c-f784-46f7-965c-08db88b8cf93 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: BK1LdzmU+hqekDAeNPKL4IDn/fEeCCv9qazQv56iSKag8/YWoRUwunYPSdwxh9jK+ajt2XX6FJrLMgtxy2BF7l2JCIlw2K69OER88KEDEZddVsXZR7Jn0kx9VN1WWNwrp1+/jJ9Km4WGEyLOHAykZXaXvdQqxTOy+HWnrtTZEZT/JDg7uZg9kFoaC8wyHkUlte9WAOg7BM1d9yU4twOXv/U8zjIS4k2z7fc1AyP4qGJFAFbd8QcPcWrHo91zAEVlZiMBKxKeCR2CkvXhQtcOnWATpX9S2cccpdQFJs6oirjyB4DEqbBYy+LDAnsG/pS6R/tHWOZSzdqarbji9IIQQJUuGFWEmvrl8sOlR9TUdd3ypWbnfVHCta9X/NEKFvH5uQUc7rFZffkzIzY/Pi8cP6jK7ko4Hzk0owMm6zSjREC4vuUguLnuUyxyqVRhs+cA7/pVxKrVoHFOqyzH6eiaJCEGp+XBGuqXUPxeUFUYPy7CbyFdPp4AEVd2jtI847pwP1s2eFdpmdE4mU9/d8Vg2+pdy6cAjgPZ34tKwaeBdpcnhgDJa9fMUEorsT4EzAf8AKp1jH1eXwOoe195a84ozqciZWZu6Z999STFlcwIT1pchDHXICFtYlwwEtRYvSbo x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(30864003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?2CqbqWqG/524K9F4nPw4O4w?= =?iso-8859-1?q?GKQmJmSlw8DyE0zB0mTfVEwo4PxM2m3GAxZ1Q3/3mqsh+95o75DmgM3bFH1H?= =?iso-8859-1?q?O3QbQD6zMb0Qgj7aFxyWGGUcNvGOI6CEMjyVsOW7Hh47ihHcjV5Qo5CZSAlX?= =?iso-8859-1?q?hovur8+Y4bXZhGbAhg5eNyDoR3fSyo8aw8LlJJvvwDmq6agYhxLsl1ds7EuQ?= =?iso-8859-1?q?ZxBzmcfGe2RAo9ZGY027sllv6SMx1my0cb87I6qege6rKD8+s8H8/nnw4N6v?= =?iso-8859-1?q?tjB6/ha2N8XKC2m7BJGA8hOiLtfJczi3GAbuMhSbNUQDEoJsJVE55/Izeis0?= =?iso-8859-1?q?qsX80uY4qhfwYy9Lbq+OCC6RhfRQ9M9DcIzGUywpV+4cB5QRxIYGA/vVBJRQ?= =?iso-8859-1?q?okCFdji+V53OrUBiggLJ18Ip0tAewYF9Wjg9rAtaftbefJnsDLZugBz+MNFk?= =?iso-8859-1?q?hEqTCdEAh9My+UF1Ya0cSGZ92Os5yowwG8RLn79Dd6B95zo/1IU7UgwhGqMh?= =?iso-8859-1?q?mPBYVRv12jPBosDhwIvUcr4bngASup5VWg1rFQUwvqU/SeAAq3F/ah7PoA6O?= =?iso-8859-1?q?aaOt45hGNZPQ1jInAhEx//Bt8uxy5e9mktAccSWUjQGYZeRpVTcDWwP8eF5l?= =?iso-8859-1?q?8lEOkrqzLEYdrTbwTgJ7uQsQtJHCzzLcE1IlwXaL12w6BUH4niXJ/CpybG/E?= =?iso-8859-1?q?18xNRQbAsjf10DpcD9j+IEDlauaenlwJHjN/gzQ8zHA+WQ9GfLWDDJwQqHwj?= =?iso-8859-1?q?rRZBSNZq4GW+zsHlLIB3C3aqNM+UzrP/5r9JtwxVbusAVnhDhaaaLQPnzGpw?= =?iso-8859-1?q?ItmPJAubJ2j8B9X0ILhh6M8uyCbXWajNRE14Ieje2vZnMtTgcjX9hrNoNtyv?= =?iso-8859-1?q?W+c65nBYYw3fuTI2/k4lRNw6p/gSknEG6gzoZmhbDJqzSO5uvAp1niU49iY5?= =?iso-8859-1?q?12kv1m7uEfNAPpuJgv8HyHeZjcmAimLQqHYw+xktg/zsMFVtx4HgO17E+swd?= =?iso-8859-1?q?1SSA0pNMbwCkgPkf90k7X7BjiVhL1jZbrNhz/d7XFBNFFVHdrvHFju+vK6ST?= =?iso-8859-1?q?MVWNH25fGpc+yD1ljHOQyg3puSpUhCOqKgM9Raejn3a9OhMDNpVJbKRbEPLp?= =?iso-8859-1?q?vSCrnmgmk+fYZWLE0yGiGpUixwrlrEeXundIS+Ce2Z44RsZ6K45QwC1T9Nt/?= =?iso-8859-1?q?5WReQtsohZMJwX4toPZkt5qcuKTAhGK8GQudwy6Jz1JE19wGTDPYsVxjJGk3?= =?iso-8859-1?q?uGaBEnD4eKEY4XEFk0THrsAAaLgK3aAsIpMBiLI1BN2PVSlp2a7oE8oOg9qd?= =?iso-8859-1?q?YMzaMZMuvyM+KqYAbPNjth3kztJvaiVWRXKWLFyMdYlRiEu0QwPUjvpgUBwG?= =?iso-8859-1?q?fYUsDw8eiJCcxKLEQ+Jva2bqoEu4iLCPaVg5vt3a0WBgzx6Hqf8gqNiFqCUU?= =?iso-8859-1?q?k9/3/RQQ3ABFfH6JXhDq+3baVA6j4/nE6a/SoAjLEsw+x06mJ6HVH278Sspp?= =?iso-8859-1?q?4xdyCriQSL6Lt3cmHOkwm6Ues82PqNkT/Yj6w7zTaMhcp4YZggnxi2Vtbvzb?= =?iso-8859-1?q?H0ktncvOP2671zweZdI58iyMAuTAJJEiZgCXkl3YF3bo/8o3jEzQ5FILRUYF?= =?iso-8859-1?q?Fka/1sdh/ZO7pTVBNYLnJ9QdmzQ9n2g9fYSr/Vg=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3e92be7c-f784-46f7-965c-08db88b8cf93 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:32.9924 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Oez4NfckEs57RyrUo/1qQTogW9Yip7DfAQWbsDZ2gNJj9eiMpm5G6Nea+VyT5xExOY6Y7NSQMFhCdtJ2zJYv36/VlZP0djs3cbC7gX+YZaU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: v7awCBYqKbbJt-sdSLCk-QjslHKggHmC X-Proofpoint-GUID: v7awCBYqKbbJt-sdSLCk-QjslHKggHmC X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko Instead of handling a single range set, that contains all the memory regions of all the BARs and ROM, have them per BAR. As the range sets are now created when a PCI device is added and destroyed when it is removed so make them named and accounted. Note that rangesets were chosen here despite there being only up to 3 separate ranges in each set (typically just 1). But rangeset per BAR was chosen for the ease of implementation and existing code re-usability. This is in preparation of making non-identity mappings in p2m for the MMIOs. Signed-off-by: Oleksandr Andrushchenko --- Since v6: - update according to the new locking scheme - remove odd fail label in modify_bars Since v5: - fix comments - move rangeset allocation to init_bars and only allocate for MAPPABLE BARs - check for overlap with the already setup BAR ranges Since v4: - use named range sets for BARs (Jan) - changes required by the new locking scheme - updated commit message (Jan) Since v3: - re-work vpci_cancel_pending accordingly to the per-BAR handling - s/num_mem_ranges/map_pending and s/uint8_t/bool - ASSERT(bar->mem) in modify_bars - create and destroy the rangesets on add/remove --- xen/drivers/vpci/header.c | 235 ++++++++++++++++++++++++++++---------- xen/drivers/vpci/vpci.c | 6 + xen/include/xen/vpci.h | 3 +- 3 files changed, 181 insertions(+), 63 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 5dc9b5338b..eb07fa0bb2 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -141,63 +141,106 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, bool vpci_process_pending(struct vcpu *v) { - if ( v->vpci.mem ) + struct pci_dev *pdev = v->vpci.pdev; + + if ( !pdev ) + return false; + + if ( v->vpci.map_pending ) { struct map_data data = { .d = v->domain, .map = v->vpci.cmd & PCI_COMMAND_MEMORY, }; - int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data); - - if ( rc == -ERESTART ) - return true; + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; write_lock(&v->domain->pci_lock); - spin_lock(&v->vpci.pdev->vpci->lock); - /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, - !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci->lock); - - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; - if ( rc ) - /* - * FIXME: in case of failure remove the device from the domain. - * Note that there might still be leftover mappings. While this is - * safe for Dom0, for DomUs the domain will likely need to be - * killed in order to avoid leaking stale p2m mappings on - * failure. - */ - vpci_remove_device(v->vpci.pdev); + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + int rc; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_consume_ranges(bar->mem, map_range, &data); + + if ( rc == -ERESTART ) + { + write_unlock(&v->domain->pci_lock); + return true; + } + + spin_lock(&pdev->vpci->lock); + /* Disable memory decoding unconditionally on failure. */ + modify_decoding(pdev, rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : + v->vpci.cmd, !rc && v->vpci.rom_only); + spin_unlock(&pdev->vpci->lock); + + if ( rc ) + { + /* + * FIXME: in case of failure remove the device from the domain. + * Note that there might still be leftover mappings. While this + * is safe for Dom0, for DomUs the domain needs to be killed in + * order to avoid leaking stale p2m mappings on failure. + */ + v->vpci.map_pending = false; + + if ( is_hardware_domain(v->domain) ) + { + vpci_remove_device(pdev); + write_unlock(&v->domain->pci_lock); + } + else + { + write_unlock(&v->domain->pci_lock); + domain_crash(v->domain); + } + return false; + } + } write_unlock(&v->domain->pci_lock); + + v->vpci.map_pending = false; } + return false; } static int __init apply_map(struct domain *d, const struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd) + uint16_t cmd) { struct map_data data = { .d = d, .map = true }; - int rc; + struct vpci_header *header = &pdev->vpci->header; + int rc = 0; + unsigned int i; ASSERT(rw_is_locked(&d->pci_lock)); - while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART ) + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - /* - * It's safe to drop and reacquire the lock in this context - * without risking pdev disappearing because devices cannot be - * removed until the initial domain has been started. - */ - read_unlock(&d->pci_lock); - process_pending_softirqs(); - read_lock(&d->pci_lock); - } + struct vpci_bar *bar = &header->bars[i]; - rangeset_destroy(mem); + if ( rangeset_is_empty(bar->mem) ) + continue; + + while ( (rc = rangeset_consume_ranges(bar->mem, map_range, + &data)) == -ERESTART ) + { + /* + * It's safe to drop and reacquire the lock in this context + * without risking pdev disappearing because devices cannot be + * removed until the initial domain has been started. + */ + write_unlock(&d->pci_lock); + process_pending_softirqs(); + write_lock(&d->pci_lock); + } + } if ( !rc ) modify_decoding(pdev, cmd, false); @@ -205,10 +248,12 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, } static void defer_map(struct domain *d, struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd, bool rom_only) + uint16_t cmd, bool rom_only) { struct vcpu *curr = current; + ASSERT(!!rw_is_write_locked(&pdev->domain->pci_lock)); + /* * FIXME: when deferring the {un}map the state of the device should not * be trusted. For example the enable bit is toggled after the device @@ -216,7 +261,7 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, * started for the same device if the domain is not well-behaved. */ curr->vpci.pdev = pdev; - curr->vpci.mem = mem; + curr->vpci.map_pending = true; curr->vpci.cmd = cmd; curr->vpci.rom_only = rom_only; /* @@ -230,33 +275,34 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) { struct vpci_header *header = &pdev->vpci->header; - struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; const struct domain *d; const struct vpci_msix *msix = pdev->vpci->msix; - unsigned int i; + unsigned int i, j; int rc; + bool map_pending; ASSERT(rw_is_locked(&pdev->domain->pci_lock)); - if ( !mem ) - return -ENOMEM; - /* - * Create a rangeset that represents the current device BARs memory region - * and compare it against all the currently active BAR memory regions. If - * an overlap is found, subtract it from the region to be mapped/unmapped. + * Create a rangeset per BAR that represents the current device memory + * region and compare it against all the currently active BAR memory + * regions. If an overlap is found, subtract it from the region to be + * mapped/unmapped. * - * First fill the rangeset with all the BARs of this device or with the ROM + * First fill the rangesets with the BARs of this device or with the ROM * BAR only, depending on whether the guest is toggling the memory decode * bit of the command register, or the enable bit of the ROM BAR register. */ for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - const struct vpci_bar *bar = &header->bars[i]; + struct vpci_bar *bar = &header->bars[i]; unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + if ( !bar->mem ) + continue; + if ( !MAPPABLE_BAR(bar) || (rom_only ? bar->type != VPCI_BAR_ROM : (bar->type == VPCI_BAR_ROM && !header->rom_enabled)) || @@ -272,14 +318,31 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) continue; } - rc = rangeset_add_range(mem, start, end); + rc = rangeset_add_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to add [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); return rc; } + + /* Check for overlap with the already setup BAR ranges. */ + for ( j = 0; j < i; j++ ) + { + struct vpci_bar *bar = &header->bars[j]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove overlapping range [%lx, %lx]: %d\n", + start, end, rc); + return rc; + } + } } /* Remove any MSIX regions if present. */ @@ -289,14 +352,21 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long end = PFN_DOWN(vmsix_table_addr(pdev->vpci, i) + vmsix_table_size(pdev->vpci, i) - 1); - rc = rangeset_remove_range(mem, start, end); - if ( rc ) + for ( j = 0; j < ARRAY_SIZE(header->bars); j++ ) { - printk(XENLOG_G_WARNING - "Failed to remove MSIX table [%lx, %lx]: %d\n", - start, end, rc); - rangeset_destroy(mem); - return rc; + const struct vpci_bar *bar = &header->bars[j]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove MSIX table [%lx, %lx]: %d\n", + start, end, rc); + return rc; + } } } @@ -341,7 +411,7 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); if ( !bar->enabled || - !rangeset_overlaps_range(mem, start, end) || + !rangeset_overlaps_range(bar->mem, start, end) || /* * If only the ROM enable bit is toggled check against * other BARs in the same device for overlaps, but not @@ -350,12 +420,11 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) continue; - rc = rangeset_remove_range(mem, start, end); + rc = rangeset_remove_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); return rc; } } @@ -380,10 +449,23 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) * will always be to establish mappings and process all the BARs. */ ASSERT((cmd & PCI_COMMAND_MEMORY) && !rom_only); - return apply_map(pdev->domain, pdev, mem, cmd); + return apply_map(pdev->domain, pdev, cmd); } - defer_map(dev->domain, dev, mem, cmd, rom_only); + /* Find out how many memory ranges has left after MSI and overlaps. */ + map_pending = false; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + if ( !rangeset_is_empty(header->bars[i].mem) ) + { + map_pending = true; + break; + } + + /* If there's no mapping work write the command register now. */ + if ( !map_pending ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + else + defer_map(dev->domain, dev, cmd, rom_only); return 0; } @@ -574,6 +656,19 @@ static void cf_check rom_write( rom->addr = val & PCI_ROM_ADDRESS_MASK; } +static int bar_add_rangeset(struct pci_dev *pdev, struct vpci_bar *bar, int i) +{ + char str[32]; + + snprintf(str, sizeof(str), "%pp:BAR%d", &pdev->sbdf, i); + + bar->mem = rangeset_new(pdev->domain, str, RANGESETF_no_print); + if ( !bar->mem ) + return -ENOMEM; + + return 0; +} + static int cf_check init_bars(struct pci_dev *pdev) { uint16_t cmd; @@ -657,6 +752,13 @@ static int cf_check init_bars(struct pci_dev *pdev) else bars[i].type = VPCI_BAR_MEM32; + rc = bar_add_rangeset(pdev, &bars[i], i); + if ( rc ) + { + bars[i].type = VPCI_BAR_EMPTY; + return rc; + } + rc = pci_size_mem_bar(pdev->sbdf, reg, &addr, &size, (i == num_bars - 1) ? PCI_BAR_LAST : 0); if ( rc < 0 ) @@ -707,6 +809,15 @@ static int cf_check init_bars(struct pci_dev *pdev) rom_reg, 4, rom); if ( rc ) rom->type = VPCI_BAR_EMPTY; + else + { + rc = bar_add_rangeset(pdev, rom, i); + if ( rc ) + { + rom->type = VPCI_BAR_EMPTY; + return rc; + } + } } } else diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index a97710a806..ca3505ecb7 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -38,6 +38,8 @@ extern vpci_register_init_t *const __end_vpci_array[]; void vpci_remove_device(struct pci_dev *pdev) { + unsigned int i; + ASSERT(rw_is_write_locked(&pdev->domain->pci_lock)); if ( !has_vpci(pdev->domain) || !pdev->vpci ) @@ -63,6 +65,10 @@ void vpci_remove_device(struct pci_dev *pdev) if ( pdev->vpci->msix->table[i] ) iounmap(pdev->vpci->msix->table[i]); } + + for ( i = 0; i < ARRAY_SIZE(pdev->vpci->header.bars); i++ ) + rangeset_destroy(pdev->vpci->header.bars[i].mem); + xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 486a655e8d..b78dd6512b 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -72,6 +72,7 @@ struct vpci { /* Guest view of the BAR: address and lower bits. */ uint64_t guest_reg; uint64_t size; + struct rangeset *mem; enum { VPCI_BAR_EMPTY, VPCI_BAR_IO, @@ -156,9 +157,9 @@ struct vpci { struct vpci_vcpu { /* Per-vcpu structure to store state while {un}mapping of PCI BARs. */ - struct rangeset *mem; struct pci_dev *pdev; uint16_t cmd; + bool map_pending : 1; bool rom_only : 1; }; From patchwork Thu Jul 20 00:32:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 695C4C001DF for ; Thu, 20 Jul 2023 00:33:01 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566261.884896 (Exim 4.92) (envelope-from ) id 1qMHb7-0002sw-F2; Thu, 20 Jul 2023 00:32:49 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566261.884896; Thu, 20 Jul 2023 00:32:49 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb7-0002qw-63; Thu, 20 Jul 2023 00:32:49 +0000 Received: by outflank-mailman (input) for mailman id 566261; Thu, 20 Jul 2023 00:32:45 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb3-0001JI-Nt for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:45 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id f0ca61cf-2694-11ee-b23a-6b7b168915f2; Thu, 20 Jul 2023 02:32:44 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiBA017190 for ; Thu, 20 Jul 2023 00:32:42 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jul 2023 00:32:42 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:34 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:34 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f0ca61cf-2694-11ee-b23a-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LrLH6BhKTdUsOIh5Cn8RGZLmKJzxnMZJ2Y1k3U7xK0hR/+6EhVcpd5+/b7S/0/hoSuxkCFpKGNya1eyESmKv5D0QtGAsDzWELY9bEz5JS1vj53tJojDCna8hPoYXo0ENIEMehej7N17OV+RZGeG/GKFrbeiHH54EQCzLtmvLNJMcNvYcc1wWiXRJHtVskJvHUXinbCVffT8ImAYfbLfQKvw6CYGecqSmNPksficjpHGGkrh9g5GS5c4ci7y6pi7RPVH+1q/z1FbuvjLT2vlpUOJU3MNPC4vGsYqdB5hqzGwgFCV2pYTBNbzMFKjd9FzwGcc/ejK1YV4ujbh/jd4iCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=o4qMyvBtfNRc4aBSfSbMUWMhpLQYprUCPRNweR/SKGA=; b=SmWOL6VyRR6aisYAoFC4foFiR1bUD5S00WuF63OLs1XBAdGSM7O98Q7S3nL+p5xkfQ9+kvHwZiCBmejOZa7sWJ5HckEgNPtYDYy7q9ZzzeO/ioBJJFOqIBROy4Hjo0cx/ICbd/lYLUqtkSmjLB9iKyN4qqQOy7lXX+sU8aMEJuDLf2FN2jA2Cw+lmQ6e893+/INYlJRvTCm1nV2vKVOz7EUvo2hZ5/LU8ZWy0eH4PPTwgTpFAs73j3IbyMN2mY4+0LJFhqDp1G1SDr8rv8TJ1AjZzmRU6TDvBD1/W+PhUTNhb2c3aYJkJz3sw4B6ld3mJ7CASx4iFUqiusyQ2QNz6A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o4qMyvBtfNRc4aBSfSbMUWMhpLQYprUCPRNweR/SKGA=; b=N8QDd1XQw+ZnMRbWvaozaOmrAo3GbDrL1WldcZPU9USh79lRBR/7zoMbzR+rDa62/DTeW/d2eO8j8GAMgOZnr0mq+Dom27Ccnx8gDuSnx1YJpv99Y25pAHzBxpUFEAcVOCMUOM6LTOGzVThqvqh49/5ApgDliwaXmrho7QMgPUvZh7Mo5I+RudroUmR9VKAr46sBYDU555ZYTLHGY64RlsH464dTkqTVVal6FvIdejWLHH7z1bFsHDZa90CuqDhejy+qgz/khlX4lh9eWENJbDwZd4MGJ9Y008retZjeGX8+ee0fLQFb0RJ1Z+UjSFskc4lfhiVHBEcSit13Us+viw== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko Subject: [PATCH v8 08/13] vpci/header: program p2m with guest BAR view Thread-Topic: [PATCH v8 08/13] vpci/header: program p2m with guest BAR view Thread-Index: AQHZuqGsvPRRGO0LlUKscysi74kMlA== Date: Thu, 20 Jul 2023 00:32:33 +0000 Message-ID: <20230720003205.1828537-9-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: e473c0ba-1993-432a-40b3-08db88b8d001 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: jK/sLJLxamYos2oW3q92xo8y9iSq8/9sIruG+eYYFPU6eyu+SdVCq3Mt4RIXr7kypkzimXa5k6iCtv6/TzXiy/U6XHQoePAUmBly+pyxMBRawbZjaxMT/vyWnieebXV09nc9GcEYOz81598fUEDFzpVtCuOu05I70+v1C6+88Bg+xUlPK9zc2+Un2e57Lpt+hf22hRX+ofCL+MPYYt1ejHWFThDr7UIl+5XnZLwOOhcN6yq5zakB9x13ah2d0X5/e7qA1zqxIFilnp4IytR0MHO3QGwptJRLoijs97y1oZf8+SW2IEZoHKGnKqPHZoX81JEeblkugFk10bGN1XMFFW36n17bjaDR1YbPSH79m5Rv8wpu0MFHBVTAZIsb7KIt0mwlkSTkiHbMRmAz1w1osQ+YV/PPk3OeKpJlxZyXPV6hQcA/2hrZ53g9XaWa8ILC6RC9IPWYR0+VQLn97Mq6ANWjFDDV2rckfkHLTjFlwRwYysJMr74MgLncJqjgqGF6dXWNlsPoFT02PBwpWm0W919s+VMi9/Nks+oJ0iY+UEuii2AQhcsF1tkSDEvVycq15wy95xXqUOewi8lakadZ+qWRF+N9urjhoRcispiIq0re3Hejv5fq6BuWVe5401a0 x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?ITM6ytIAruRUjzvQlxuOsyh?= =?iso-8859-1?q?DyFE+eaUuACsPV3X8bHVaen/Gg/3mIicHjEwGoQU2HATLd37nk4ZDC89r5w3?= =?iso-8859-1?q?E1EkU+khYqu0Jvoxrbpn/5GSfF1hkXq95UBVrNmb8zTHnH6OeNgV5FvzbPGC?= =?iso-8859-1?q?Efm/MNEfvKGRlfkLtZbH2irud2ZmTUrHdkbDpjLFS/1afZRWEiw+/dsts2hY?= =?iso-8859-1?q?Pw5qtD1VRRw7SjHrMKGvH7NoEM2EMy1otqVb264P3I6XLgCNGRPOpwBpc+ob?= =?iso-8859-1?q?Eif8QvLrP/HXtuxIPY4RdrNka8Ymc3SMBzYQxz2ciN3b64UkkbXuXkF3QUph?= =?iso-8859-1?q?LzvcJEYrnC/zfnTUDTNUK9KO5XI9Gt4e9/Jz/QvCoVcwg8N8lgbTbBkOAdwM?= =?iso-8859-1?q?+wBLoRXtA4V1sSA1hVcr0z9MoDX3OMyw9HhKqaZ38hSgJJUF4RI8gTt2B0MF?= =?iso-8859-1?q?3HH/on33KvXDSV/o7OfymD4Cp5ERjJxa3vuVtGDqYbMJOUUtP3mNLYUUfzjY?= =?iso-8859-1?q?wRedNQk/1fJb4UhGFw1DjfrxZ237SQLnftI88h1sgDy0z2BGRhsQmU7poxwZ?= =?iso-8859-1?q?3hQDIxuOKWvmUK5v575LzHsqunlSYdQLjDAkn9j466D27yDRx9XXR2DC+x71?= =?iso-8859-1?q?xaFHkA22MdDnPVXrsefBtfLsapthiDQavSzjFDpK3htVFsn/WuhG5vw4qgkj?= =?iso-8859-1?q?tKuiMyJPEU4W2LGaBdZCY/IuGzHjRH+6VAe1MzBDKaUEChKyF3vMqNVO3Hcm?= =?iso-8859-1?q?OlP31fBeT3DqwJcd9Vf3ufQkPFQI7DLrzA88WlE+uYVFNSP9QE0qfmctqLqf?= =?iso-8859-1?q?EZXQ+tQNa0WnBcwU4ZiOh+oMJXiWa50jrbNxRTY1zCJIvgaLyZxEg5jLbTa1?= =?iso-8859-1?q?1pTsegb6RzOM22B18ER6ysa3otCXej+2FcpAoPOAhFK9XLhuV9/y3zGfKLQH?= =?iso-8859-1?q?2V9mWRwiT8cZ5YVI2eyCbsyvedn9kvw8bgY6G6AUlU1lU9ms2UuT4FE+dODd?= =?iso-8859-1?q?Pe2OM1p4WM3VsSOeJ0yhlIOjvjCACDJYsKpQxtV+k9wqk0biLdUc+4XkLVm+?= =?iso-8859-1?q?iogkWVAOxAOLKxOijeE/rOMwR+YnXC6cbMc/QbZWOq3izWWlfbBRSqOLtQvA?= =?iso-8859-1?q?MrExLbb0BPzC7o3w+44t5N/1EibNX3u9fES7U8268AJHMoWROOrNY++3vx2r?= =?iso-8859-1?q?1x4ljuYM1FDSmxPH5CB+HHy5ek4CFHJ6Ok1ND2hrdcUMFEhbp0kgoKQJBPH7?= =?iso-8859-1?q?WG1fjFGAvIi3dIGYB1XrfL0fWVCNFIYhuZuGNQkisVjqjKtcwRIH2hXxBIvP?= =?iso-8859-1?q?tOiFigfG+jMhiNkBjw0nl/K1p7Ro3Y9arseRuvSPLbX6gKJobxgJVSeQQoI4?= =?iso-8859-1?q?Vrt+ynQvh1oVgflw3u+dCn5fcnYt5gap6axfAkS1NEheVP3eMu1HfHieKKW/?= =?iso-8859-1?q?da6mFlUA89/7P2HwtnKTalzJFjPu6zoQGFfqQ2zr7y1MrWCq7WVZvYJKiR4h?= =?iso-8859-1?q?r4fnX1AYj1Jh00ZdA/sUDcWknqmJnK8Kl79lanevyM6pFjTuByJWCtXqMNTh?= =?iso-8859-1?q?X2mW06QPptUXEc3s6AWJrM4t6XZ7V/yhiENAxr5qPCVl4W9ZkJSn40KQMXcd?= =?iso-8859-1?q?pgCZ5Qq5uqdRL0Pr9qY3mw0UunEnOyJt5gMe6eQ=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e473c0ba-1993-432a-40b3-08db88b8d001 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:33.2421 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: yeyPkEV1rHLodU8l+XJTj1Fhts+L6tlnUYju8BkhNVvqcL56MvK/COAhv9+OjINEliTl2ZQmAULxtLFuLeBzvSiPThR2XIMtZP73LxL7ETs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: 0FRUiznFJ4kRDzCYwm57OU7wA6uC2nrp X-Proofpoint-GUID: 0FRUiznFJ4kRDzCYwm57OU7wA6uC2nrp X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value as set up by the PCI bus driver in the hardware domain. This way hardware domain sees physical BAR values and guest sees emulated ones. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - remove debug print in map_range callback - remove "identity" from the debug print Since v4: - moved start_{gfn|mfn} calculation into map_range - pass vpci_bar in the map_data instead of start_{gfn|mfn} - s/guest_addr/guest_reg Since v3: - updated comment (Roger) - removed gfn_add(map->start_gfn, rc); which is wrong - use v->domain instead of v->vpci.pdev->domain - removed odd e.g. in comment - s/d%d/%pd in altered code - use gdprintk for map/unmap logs Since v2: - improve readability for data.start_gfn and restructure ?: construct Since v1: - s/MSI/MSI-X in comments --- xen/drivers/vpci/header.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index eb07fa0bb2..e1a448b674 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -30,6 +30,7 @@ struct map_data { struct domain *d; + const struct vpci_bar *bar; bool map; }; @@ -41,8 +42,21 @@ static int cf_check map_range( for ( ; ; ) { + /* Start address of the BAR as seen by the guest. */ + gfn_t start_gfn = _gfn(PFN_DOWN(is_hardware_domain(map->d) + ? map->bar->addr + : map->bar->guest_reg)); + /* Physical start address of the BAR. */ + mfn_t start_mfn = _mfn(PFN_DOWN(map->bar->addr)); unsigned long size = e - s + 1; + /* + * Ranges to be mapped don't always start at the BAR start address, as + * there can be holes or partially consumed ranges. Account for the + * offset of the current address from the BAR start. + */ + start_gfn = gfn_add(start_gfn, s - mfn_x(start_mfn)); + /* * ARM TODOs: * - On ARM whether the memory is prefetchable or not should be passed @@ -52,8 +66,8 @@ static int cf_check map_range( * - {un}map_mmio_regions doesn't support preemption. */ - rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, _mfn(s)) - : unmap_mmio_regions(map->d, _gfn(s), size, _mfn(s)); + rc = map->map ? map_mmio_regions(map->d, start_gfn, size, _mfn(s)) + : unmap_mmio_regions(map->d, start_gfn, size, _mfn(s)); if ( rc == 0 ) { *c += size; @@ -62,8 +76,8 @@ static int cf_check map_range( if ( rc < 0 ) { printk(XENLOG_G_WARNING - "Failed to identity %smap [%lx, %lx] for d%d: %d\n", - map->map ? "" : "un", s, e, map->d->domain_id, rc); + "Failed to %smap [%lx, %lx] for %pd: %d\n", + map->map ? "" : "un", s, e, map->d, rc); break; } ASSERT(rc < size); @@ -165,6 +179,7 @@ bool vpci_process_pending(struct vcpu *v) if ( rangeset_is_empty(bar->mem) ) continue; + data.bar = bar; rc = rangeset_consume_ranges(bar->mem, map_range, &data); if ( rc == -ERESTART ) @@ -228,6 +243,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, if ( rangeset_is_empty(bar->mem) ) continue; + data.bar = bar; while ( (rc = rangeset_consume_ranges(bar->mem, map_range, &data)) == -ERESTART ) { From patchwork Thu Jul 20 00:32:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319660 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1688BC001DF for ; Thu, 20 Jul 2023 00:33:10 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566262.884906 (Exim 4.92) (envelope-from ) id 1qMHb8-00038c-8Y; Thu, 20 Jul 2023 00:32:50 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566262.884906; Thu, 20 Jul 2023 00:32:50 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb7-00036L-TT; Thu, 20 Jul 2023 00:32:49 +0000 Received: by outflank-mailman (input) for mailman id 566262; Thu, 20 Jul 2023 00:32:46 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb3-0001JN-UN for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:46 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id f099c095-2694-11ee-8611-37d641c3527e; Thu, 20 Jul 2023 02:32:43 +0200 (CEST) Received: from pps.filterd (m0174676.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JJCjt2031890 for ; Thu, 20 Jul 2023 00:32:42 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2059.outbound.protection.outlook.com [104.47.13.59]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgcw1sme-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jul 2023 00:32:42 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:35 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:35 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f099c095-2694-11ee-8611-37d641c3527e ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OlEGG5D81kgg0nBGcdBSz1ogsAqZsjDWFf4cz6H77k8C3wI54OBE2EgV7Ln3l1c2Nl7EzOY5HrfHKhgpRurjHYB143QtGgm+ECQaHtqRpPznax3uX9BDPtpYhAJ18OZD1bYMs2dB/tT5WNVXTEcVb9YpbDnOrapL/T64sOmaJaO+83pChlTQJZh47ETyoSNokeSq8xcWd8pDW6hM+4la0Pj+gaWzZOFwM02+KNwA5YsdpVNcf2JAVCilb6QJCoV+pYs4gYQauqRY2jSo9rRpVOtEXs1nDRKM/73VQgEam8J4HecHjO6nbtnSL9MZgKGNOflb6nI4VBqNpDm1qEKObQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VWkvYfFfLZLgGm2PrjN1dlq8IatJg4jM7gGw7jvqor4=; b=WCMng+mW0cs98fl/wbyl+v2M8xp9tPU2Hn4E++SIImsmFXOC8UGPt1hKAz8tZ3qOEAcWVBYRs4aYhiZF1IzqnT0OypdD0KUcmEsip6TiBwV/m1ld0QTnmreddKDl2mVGNsX3o6hkdDqw86bWKM9nQC64g12KgCLGloDNQ5kXwl6I1TwL1ao4/ElL/88frYrJYsCq8k2igWLJQratdzvIZPeABtNT1Q+Au04uOsjsPN7buEXgvkRpz7L4bbVXB4Lgjg4jn6PvjK3PpOBGCeBQP4vPRAGexFk8FapWBLFgO/uADHNPCIQzZRfDKcRYDx4vHGfWaB+eb8C8652316+DBw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VWkvYfFfLZLgGm2PrjN1dlq8IatJg4jM7gGw7jvqor4=; b=k1uBJbL3DJaxjlNCWWpan75impnFIbVKv51FuRW9eq90xY8ehP/o0Bc80fkOq0z7BsQdaE1LDD/PCSHxVdgtIICK07htJbzN0IgsNsAR9tw6rmAG1pPwL7SLXmtMPKhEhtP/ElcMO+3r5cUozVNyLqZHKMVhdJ56TzBA/2vXJucig0dBz658zWJS7r4AV/Vg0ieL2youU46HkNPdi4sj1qo31pa7WqbQxF01U4H5ti1xa5x+tKXioRzAkc8kE7naJq8+MbK1QWyAEgZPUvviFLn7FEOWhPBtt50YlJdeiG/mU32gUoJVWOmq5NI5RZ+vPagAZ7qV5ipHMT0CFocQoQ== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko Subject: [PATCH v8 09/13] vpci/header: emulate PCI_COMMAND register for guests Thread-Topic: [PATCH v8 09/13] vpci/header: emulate PCI_COMMAND register for guests Thread-Index: AQHZuqGsgAu2PWCwAEG/SX7tDNW7DQ== Date: Thu, 20 Jul 2023 00:32:33 +0000 Message-ID: <20230720003205.1828537-10-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 15423d10-3364-4e45-eaf8-08db88b8d06e x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ldben+njrd6tuRPGDxq0qY2XeS/kQmntw+CyPZ76p+N1wNnbdXIlgKp1VQiJjqDsnKsfij2/glQmBRuxaUal4FSiWsodCGt5Hyj3v4IThkoB6S8A9igj5NtpU39Q4NvBCIKJOkWFyYN5znr2RUDOteB2yt+AUgEU8hrfcrnj362eRaQ5eEfpS3HbZxbHfAMMG2cPAXlxz+vYFnw6vThbMUfYpxh3fzsiwqi04mztJrREb1ZJh/nFbE7BElH/bTsLdlOSZiiDDsJC2iys3ZjmYstZoA1idUTy5dtsIGOOhdGWfxZ7rIBH4uoT1l1cUxvujWc3WsaztQQblxYeBT70jCqyxMZAacAkZYIfR8eHHRrhvRhQIEloOy5Ui+INYeddWL0u2I0x5pGsdfA0tsXkkUdzu/6vtTwQmOISUEv1DOBJh/zi36aDUysVvjbsZ4hB3QCYq4EvIMbBOOYRs2vihhSWpQWqgh051GaTroufuB2L8xGVpEaeBPGOoWSCgspBvdiIt5CXlqLtWx/52hh4v66jl9rZ8JPXbz5Qmarop+TNP9XiGmO74Pnr9KCo1yM584SRh8doee0/XF+U/4ARx7FqMjbAoMCjbOvXqhdOLG4Ld9IC+f956OjCsPaz1iqde5j2JsNWZKbnUd+s+a9GgQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(966005)(6512007)(6486002)(66899021)(122000001)(478600001)(66946007)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?/4FzHm0tyoHJlaWPnSdgyHY?= =?iso-8859-1?q?rg4RR6maXsURkRz5PstfeO5E8bZ4n2WkNtlVOfhz48p3cv5aG+Wiuk1mlEJq?= =?iso-8859-1?q?gHg0/MFj1jDqAchKGxFnEjlFRGquY/tf63ZaxSOe1ND9U3RuHsmaqTYLFQmQ?= =?iso-8859-1?q?EymIk5GcHFWTDoGm4zbEhh15EuJ9kKRZcHOLoCRPJ6CLCTQJaKreRq/u2C93?= =?iso-8859-1?q?bdUKmkupHlXHBm7p1hIHi6kqSlLTYAlJzM6mt2a8DLPSIJ1mGYDiUKiXGHNI?= =?iso-8859-1?q?50563B4rP7TeP0WrYjmVBPFrKFQehXyKHJCi2pIYnwwOp1EfpW8P36EmT+zL?= =?iso-8859-1?q?nxqh398NmXV6ZwZsKpR8bV3+moS92qq4L8DjeJ4LselZQHzHXMG4/6hQ6fEy?= =?iso-8859-1?q?kubzz/kDIZ3jwtLwmUKRFDIDOzLlD93FcsvK9iucXnHyFFmiA5dAGlO34H7y?= =?iso-8859-1?q?AzJsct7Br0qrqGSLOUzxalRo9t8gpkCkjEpzkvR1JD2GxDFGW3oXxHrnxUik?= =?iso-8859-1?q?IEbfUtPVz+1VlqYORW/v4TP8XeRoWst6zvTVydxgi3kKFQ3ChwOupxfwkeUg?= =?iso-8859-1?q?u0LHSCnPzkeFZoU7dCuScsnSgXX2mDIYXTiVR7d0MAFIZ8UIIFiFgqUXEOFa?= =?iso-8859-1?q?v6Zy7jt/YCSnQJxOUMwv+32FuIAFQP+o98s492BJt2YPvT05uIYSxPEXgN/B?= =?iso-8859-1?q?+2TIe/GVm7xEUXPNflZBTwqb4P0VkcZe5dZ3/lkexgRHaBn4Jp1IrepOYAWw?= =?iso-8859-1?q?/s3EOryTLaYDzophRmDloEbTTa7SqCmWbD0YPpfz/EzOHMhVF4jByhoJfvsP?= =?iso-8859-1?q?9PiHzBjPSN0SyGEhQCjMeN8m4daQYEwwpfa4Udqa/Jr5lPIz36jWVxPY0gCP?= =?iso-8859-1?q?Jo3Fg8f7fhai2KbUhAgF2VeZsXIycBH473XykH6YKW6DxE8a7ZF6OfJBWmug?= =?iso-8859-1?q?ROQULzIUxbCQJT5CfGUlDra3vnmKh/Xr23BISo5hykgHsG/zDlRCht2YFE3U?= =?iso-8859-1?q?oTmRzwqtjqjJ/mphqvlmj/ZCEwCdSdzNOdNkRL43cZgsuJSVbDPGzpnJltmM?= =?iso-8859-1?q?N3Ib1lFSwWAXXQssvq/Nbkbpa5n8ivrrsQZLPdmCCJGQl5D78CFjOALsAF6l?= =?iso-8859-1?q?LFmxpF2Ih0OwZo+twbXAVykP3TyaRmpjhsObQ994cStKGlp02X41ivKdDQ1A?= =?iso-8859-1?q?Gl5/oXdOVj0pPAnhpcU54jequptEf1IYSJtx8u/8GZAw72TbAWA4kjg0tvi9?= =?iso-8859-1?q?uVm7RhXGwRA1pRM+K1EtIy1+y/7XG1E6uC9K0IwGXxKVe0eQ3LAmgF/6eiSq?= =?iso-8859-1?q?+2qfaC3pgJp6gCsTWCz6oEPpMs/rCiAAXyPm+chWFS5x2fYqoW5YnXQkgDwm?= =?iso-8859-1?q?2MXotgGmf6/GBYAsNTq3498NdEhJ9yXEWcHHKypCb62l+u0TGczwNYZrFOUW?= =?iso-8859-1?q?ydLdI9a0mzjJcffUQX3574JKtsWSpW3YAB9qWZBdsRiDpy3dUW7K9CLweYei?= =?iso-8859-1?q?+1lz+3+e4xMFWPigT3pOyn1Lcnpd5+Bam8gcwwVyH0Nx74SebIG/dUe77dgH?= =?iso-8859-1?q?oNKDRHDYAYpyQ9jm0l4v2/bKNEjAguqQ77I5eWIbOYR/oc+P9BMteXZwNyS1?= =?iso-8859-1?q?voXVVo3ttLxFOzmg7Uvm6LuGUfxVgPAEaTnmx7A=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 15423d10-3364-4e45-eaf8-08db88b8d06e X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:33.5184 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: nJTWxVu0YK8N81H0kRyPPM/NVmLU6RAwol1YnJg4PH/Dz19fHGrixwwfiGMx4TvIr8BronvI+RwWW6YgmmfEMI5FGiOjgNYY17G6uFcRQ00= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: CBU4TaoMcSbmhCGN8vhvLzoC3hWNTWx9 X-Proofpoint-GUID: CBU4TaoMcSbmhCGN8vhvLzoC3hWNTWx9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 phishscore=0 mlxscore=0 spamscore=0 bulkscore=0 clxscore=1015 impostorscore=0 adultscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko Xen and/or Dom0 may have put values in PCI_COMMAND which they expect to remain unaltered. PCI_COMMAND_SERR bit is a good example: while the guest's view of this will want to be zero initially, the host having set it to 1 may not easily be overwritten with 0, or else we'd effectively imply giving the guest control of the bit. Thus, PCI_COMMAND register needs proper emulation in order to honor host's settings. There are examples of emulators [1], [2] which already deal with PCI_COMMAND register emulation and it seems that at most they care about is the only INTx bit (besides IO/memory enable and bus master which are write through). It could be because in order to properly emulate the PCI_COMMAND register we need to know about the whole PCI topology, e.g. if any setting in device's command register is aligned with the upstream port etc. This makes me think that because of this complexity others just ignore that. Neither I think this can easily be done in Xen case. According to "PCI LOCAL BUS SPECIFICATION, REV. 3.0", section "6.2.2 Device Control" the reset state of the command register is typically 0, so when assigning a PCI device use 0 as the initial state for the guest's view of the command register. For now our emulation only makes sure INTx is set according to the host requirements, i.e. depending on MSI/MSI-X enabled state. This implementation and the decision to only emulate INTx bit for now is based on the previous discussion at [3]. [1] https://github.com/qemu/qemu/blob/master/hw/xen/xen_pt_config_init.c#L310 [2] https://github.com/projectacrn/acrn-hypervisor/blob/master/hypervisor/hw/pci.c#L336 [3] https://patchwork.kernel.org/project/xen-devel/patch/20210903100831.177748-9-andr2000@gmail.com/ Signed-off-by: Oleksandr Andrushchenko --- Since v6: - fold guest's logic into cmd_write - implement cmd_read, so we can report emulated INTx state to guests - introduce header->guest_cmd to hold the emulated state of the PCI_COMMAND register for guests Since v5: - add additional check for MSI-X enabled while altering INTX bit - make sure INTx disabled while guests enable MSI/MSI-X Since v3: - gate more code on CONFIG_HAS_MSI - removed logic for the case when MSI/MSI-X not enabled --- xen/drivers/vpci/header.c | 38 +++++++++++++++++++++++++++++++++++++- xen/drivers/vpci/msi.c | 4 ++++ xen/drivers/vpci/msix.c | 4 ++++ xen/include/xen/vpci.h | 3 +++ 4 files changed, 48 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index e1a448b674..ae05d242a5 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -486,11 +486,27 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) return 0; } +/* TODO: Add proper emulation for all bits of the command register. */ static void cf_check cmd_write( const struct pci_dev *pdev, unsigned int reg, uint32_t cmd, void *data) { struct vpci_header *header = data; + if ( !is_hardware_domain(pdev->domain) ) + { + struct vpci_header *header = data; + + header->guest_cmd = cmd; +#ifdef CONFIG_HAS_PCI_MSI + if ( pdev->vpci->msi->enabled || pdev->vpci->msix->enabled ) + /* + * Guest wants to enable INTx, but it can't be enabled + * if MSI/MSI-X enabled. + */ + cmd |= PCI_COMMAND_INTX_DISABLE; +#endif + } + /* * Let Dom0 play with all the bits directly except for the memory * decoding one. @@ -507,6 +523,19 @@ static void cf_check cmd_write( pci_conf_write16(pdev->sbdf, reg, cmd); } +static uint32_t cmd_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + if ( !is_hardware_domain(pdev->domain) ) + { + struct vpci_header *header = data; + + return header->guest_cmd; + } + + return pci_conf_read16(pdev->sbdf, reg); +} + static void cf_check bar_write( const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -713,8 +742,15 @@ static int cf_check init_bars(struct pci_dev *pdev) return -EOPNOTSUPP; } + /* + * According to "PCI LOCAL BUS SPECIFICATION, REV. 3.0", section "6.2.2 + * Device Control" the reset state of the command register is + * typically all 0's, so this is used as initial value for the guests. + */ + ASSERT(header->guest_cmd == 0); + /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, + rc = vpci_add_register(pdev->vpci, cmd_read, cmd_write, PCI_COMMAND, 2, header); if ( rc ) return rc; diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index e63152c224..c37845a949 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -70,6 +70,10 @@ static void cf_check control_write( if ( vpci_msi_arch_enable(msi, pdev, vectors) ) return; + + /* Make sure guest doesn't enable INTx while enabling MSI. */ + if ( !is_hardware_domain(pdev->domain) ) + pci_intx(pdev, false); } else vpci_msi_arch_disable(msi, pdev); diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 9481274579..eab1661b87 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -97,6 +97,10 @@ static void cf_check control_write( for ( i = 0; i < msix->max_entries; i++ ) if ( !msix->entries[i].masked && msix->entries[i].updated ) update_entry(&msix->entries[i], pdev, i); + + /* Make sure guest doesn't enable INTx while enabling MSI-X. */ + if ( !is_hardware_domain(pdev->domain) ) + pci_intx(pdev, false); } else if ( !new_enabled && msix->enabled ) { diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index b78dd6512b..6099d2141d 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -87,6 +87,9 @@ struct vpci { } bars[PCI_HEADER_NORMAL_NR_BARS + 1]; /* At most 6 BARS + 1 expansion ROM BAR. */ + /* Guest view of the PCI_COMMAND register. */ + uint16_t guest_cmd; + /* * Store whether the ROM enable bit is set (doesn't imply ROM BAR * is mapped into guest p2m) if there's a ROM BAR on the device. From patchwork Thu Jul 20 00:32:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319656 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 86C79C04A94 for ; Thu, 20 Jul 2023 00:33:02 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566263.884918 (Exim 4.92) (envelope-from ) id 1qMHb9-0003T7-H4; Thu, 20 Jul 2023 00:32:51 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566263.884918; Thu, 20 Jul 2023 00:32:51 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb8-0003OM-Sw; Thu, 20 Jul 2023 00:32:50 +0000 Received: by outflank-mailman (input) for mailman id 566263; Thu, 20 Jul 2023 00:32:46 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb4-0001JI-Ns for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:46 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id f0ffb643-2694-11ee-b23a-6b7b168915f2; Thu, 20 Jul 2023 02:32:44 +0200 (CEST) Received: from pps.filterd (m0174676.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JJCjt3031890 for ; Thu, 20 Jul 2023 00:32:43 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2059.outbound.protection.outlook.com [104.47.13.59]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgcw1sme-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jul 2023 00:32:43 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:35 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:35 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f0ffb643-2694-11ee-b23a-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TmM4Ni51K9+lPW5CTwQ3iRc6+CBnX1wRg4Cz9ZuCUNG0mesjkS7AFqOniuH5vWgKHFGhX8uGBfX6uRxPI+tUq113lzXDIx6o7aGSKNszlYm/7FQyoKSOkDlQkDSnzveGogU1xcA1uyIghyi89x/PmdPrkwUQB/osELWDFNAWnHMNLsyHQSya0eFSPbl9V+aYLvmzVq+snxGL1gjG3cWP7ahMgHEz6XKD190CoLDxrIR4bxoym8OqV0C9L+TztVr1ttOBaGXY8r7hnjJjt0tQMoxcgiTWQ0s/z9IiIQWNs3/bgXo12ioCgH8n23wgBlvOpb0qzI3CQZkt1tIn929IrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nFkYMNZ/YSdMjIjhyT2BLIGgyk/+pe/n0+TIrOCbGnw=; b=dWUqfBegTH1Pmxrns1UUxQnJQYXDvFMhvxQyz1qXY3CDMa1fgzt1yK98MugOZ+FeBLIqlPDq2s6LfdJcKCm6vJwQ5F+U1vouTw6HCsBCRsvi6iFbjOKDVMwBg5vIXMJaiQQIYucCIF9QsPk3acl+Gw0yEIS9LA1G/ckbWDCsh5sPwLqWHNEXqAeguc3253RKi5utUt86T70hS4m1h2VReuw0/F4hiVjaQPkZfPcH7yf31qp10qL3EEN3rlkL4Yl8PYztuoCPNYXfIs8JCiA4ZIgmMIxNHIOn+dVD5gDMoYFA/hLJzN6sYyr9UletxZ4PwZofe8HYk2vCEq2STAl0yQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nFkYMNZ/YSdMjIjhyT2BLIGgyk/+pe/n0+TIrOCbGnw=; b=rBMwZrAyw/RiXMDkWz+4L0lEW4IvPoG2jEgCFYzHh+e+jbksxL6rsUqk+ospORvrACCe6O0bl7v6A3sReEjZbAInV8Rh5PBRgl5MAGO+RsloM40rGpTczZnZ76r2U1w0bPrlPylhKHQNegLZMLMq5BKt8gqUJq0vh6uYdRAj3BxxriflVeR89PJL+hhIARSmfqMlz1ZGkTkg/PzT+bgBJW7amXnU5P3TsQlj2X49BkM3cScbCUmzKe5q86/doCZLLAC+/muTR+flxgvlKQAxcBT687WMuMzgM7NkCHLwTiqfF+BltbreJVkTQ1RcoWPbiYg1dsVSWg3Fsh4P9canRQ== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko Subject: [PATCH v8 10/13] vpci/header: reset the command register when adding devices Thread-Topic: [PATCH v8 10/13] vpci/header: reset the command register when adding devices Thread-Index: AQHZuqGsniCLpx4fUkKP62aGMajP0g== Date: Thu, 20 Jul 2023 00:32:33 +0000 Message-ID: <20230720003205.1828537-11-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 8d9b9ea6-8c06-4c54-993f-08db88b8d09e x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 3nzE4mzoivnSuO+5z+hvlH4XcbbJTSxzQcGi/0oBuZE/ZtX0uiQzB80I6ojQxM2/T0zfufWs3qYBsURI86j4FFaT7T/8gyI9I0Lv9fyUv1PabAeHLbXgzohJI7CU/lzKcOTlfsBL6Yi7VCLMJCA4rJOcZCSCAfO4Yj/Knzbkz2IZv25Dymt3hLvCM2O+Le8pC3UrBMVwqClKK3NAPSnEsiNdL8s5BojczF0VlzNVM+lVacKDvyIWcaQTSWUhTDyL98RZdzFT0/UMpE8GTsDCCg6jow9YLcStFKln/9ros41ilo59EfpDy0ud2Y55rprdHMBkO8Wn3FzgFbooQjoGL92r8NKm/Szwi411xIRekasZ+lV9dZ8EiOkRd8mje2tRHek83ORsEFP0tvmThfL1HTIcYXlbaMneZj8U/1YgVMJ6F+T8884RM7sVaVkgifgyniQQudaGOV4TumB7HejKkIAbZXtc/WBJElpz5muDTd8F2fJUCKMmgCOKmAwpv/geTym0B00ZE8gwF/Bsxuw9VQ917u2ma5S1OPvOVBqmy80J9mfWKgc9oB2O05VApMSG2XATOx+EErszbU2ziT6BuctYZo1v5XSWohZdB9GnPiMALqBO9vgx9lVKVajVAuoX x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?ewHhk/zGIZ08Be8V9upuaLY?= =?iso-8859-1?q?FTiZDW2rx2zmulvCSzHmVwP1n5RLGZG7eNoirKnXRA6SvJyBYpdXaBk+awdR?= =?iso-8859-1?q?/Xb9dmAIdYk4X0+cpg4rXhhb6qF0c44jnFqpAUl0QlAA5uG1CelZUTwj4RGj?= =?iso-8859-1?q?XgVNeePaPhc4NZJU5xroUSOFAdq93htZFhVMHkSCpSuBQHivInP1YAndm3Bw?= =?iso-8859-1?q?rnaS6/hREsuzxv7WT9aSq7apqnVuV/hyTIsttMXNW+/zzKR7egbaHmOVQn+K?= =?iso-8859-1?q?xyzf5NJ8PdlWPznGbIkLUkLHhGAWPgn/GqwBFr4p1BsRc7zwjeZHKqJLqHV2?= =?iso-8859-1?q?jJhNIUhYhcd8/42Ef/UnvUA9ooF5cTitk7d/EETxtykqjd7rdCwZXL4VywHH?= =?iso-8859-1?q?VEmpYa8bKNTKxUvuxNAlhtXIoQ8zl1MTAxRFZmuecBEIVg3rc39j/yRd3rQP?= =?iso-8859-1?q?M62Fgs0rXCcYI9FfQgkOTTBTItiPL2jf6CVXk4uAFVrH5dqL/Q6cZ3nnuTB9?= =?iso-8859-1?q?7e+ZHT0kXH1p/rs7xK5uKZWQNMbVMvllwoLz44jpu09jF2uWxcV6NLlX+ZiH?= =?iso-8859-1?q?Qyl8KoR3rEHZo586VmAspX83Ihp7L5m8ei5tF4L5Fn0rcTegwZ0F4pUFv8gN?= =?iso-8859-1?q?r9OH+qnOehWtt1VFgrQDKT2xAGDNciT9achmGgs1vtPyzLeumKGby+FLPR3p?= =?iso-8859-1?q?Ehgm8Dxsm5TPzVPeCtnjCPlz0pMiD3H3aOLz6GA333Tq7qpbO5Czk59OiiRP?= =?iso-8859-1?q?iZ+p0vgymEMnnEo4XQ07r9fV5qzrbVxw8yYerp55cheat0ULFuvIeVDexkp1?= =?iso-8859-1?q?DgqHtSPPHeGhpANjW94rMPIQtPD38jA8k7mI2DJnszAfM46riFTuvnO3eMHS?= =?iso-8859-1?q?JFl8Ps13i3dNB/7u4L1BiGtEXo1mLSZh9LKgpK6EEJe86Joqv1E/2LQg0KOG?= =?iso-8859-1?q?kgTusQhBsQ+8Pjpjd8aFFPt/W/vEPOkNNPg5nqS/xIDmjD9xorArVETBweb7?= =?iso-8859-1?q?numQXfgH0SVJT8M+RyvKQSY5ceCdxHUZIFcZr8S15D30trWxGylto/hmNrM7?= =?iso-8859-1?q?wGJFWEyUobBlWPwvQ8be87l/ku/8bR7Pco0O2oLJk6UyhXXLeZYydjGQGucU?= =?iso-8859-1?q?e84oFC/2kvufxHCQW/5EDm1Pp0PJyoVjVb3QrETyvm/6+kbJllAoA3LmmG+/?= =?iso-8859-1?q?JX0kkR8DpmFiU/EESykHLYXb92s8QEiCTZgh0wDCBF9gHtpbZSaVoFoC7jVo?= =?iso-8859-1?q?1HrhfC3+P3/B493iLEFCUotXeZlUzFBTRNtDAZUOyOG86lzCO5tiB6fNB9yQ?= =?iso-8859-1?q?vJFrWhUvDWQO6H0EgOzoL4aey1KPfZ82sKhRcZIa9WrQKoBiZsWrxwSQARNP?= =?iso-8859-1?q?TV0gd56MRuJqIgUuAbhSfQ7lgRfRM7yjoEVy7OzufUgf8B71mgCn58sAlfEl?= =?iso-8859-1?q?PtC3yerquTVSdnHLuP/GvnlxAk0A2l9drYObVW4O54EgNdZxTNZjAkmZ4kcc?= =?iso-8859-1?q?yRZkvn8izsZVsSwgR7KjzQkHDj2z8sN0efXarcvWEFTLmpjit8foO5zfYqMh?= =?iso-8859-1?q?1iNBvGIg/dY7qRWi/WiotuF8/mpckMIldGzN6rhR2Db0fYrdrvyXyMj21b1G?= =?iso-8859-1?q?jct9WpdEY7qJ+o43nlF6lwMD50c1WGLV5I64S6g=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8d9b9ea6-8c06-4c54-993f-08db88b8d09e X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:33.7654 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: wfDbTiZneEZiF7NSFGM1VA+CihzfriT78UIupWekvD1JyjNHFjuRqUOSOD6nO2R3WjDRzYmSR3EplkVheqrOqrRIVmT68tnBoHboBBmabS4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: lMPdmvqFV_0bNMCojSfKJrzaUh83neHB X-Proofpoint-GUID: lMPdmvqFV_0bNMCojSfKJrzaUh83neHB X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 phishscore=0 mlxscore=0 spamscore=0 bulkscore=0 clxscore=1015 impostorscore=0 adultscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko Reset the command register when assigning a PCI device to a guest: according to the PCI spec the PCI_COMMAND register is typically all 0's after reset, but this might not be true for the guest as it needs to respect host's settings. For that reason, do not write 0 to the PCI_COMMAND register directly, but go through the corresponding emulation layer (cmd_write), which will take care about the actual bits written. Signed-off-by: Oleksandr Andrushchenko --- Since v6: - use cmd_write directly without introducing emulate_cmd_reg - update commit message with more description on all 0's in PCI_COMMAND Since v5: - updated commit message Since v1: - do not write 0 to the command register, but respect host settings. --- xen/drivers/vpci/header.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index ae05d242a5..44a9940fb9 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -749,6 +749,10 @@ static int cf_check init_bars(struct pci_dev *pdev) */ ASSERT(header->guest_cmd == 0); + /* Reset the command register for guests. */ + if ( !is_hwdom ) + cmd_write(pdev, PCI_COMMAND, 0, header); + /* Setup a handler for the command register. */ rc = vpci_add_register(pdev->vpci, cmd_read, cmd_write, PCI_COMMAND, 2, header); From patchwork Thu Jul 20 00:32:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319664 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B495AC001DE for ; Thu, 20 Jul 2023 00:33:11 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566264.884925 (Exim 4.92) (envelope-from ) id 1qMHbA-0003j7-Af; Thu, 20 Jul 2023 00:32:52 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566264.884925; Thu, 20 Jul 2023 00:32:52 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb9-0003gc-Th; Thu, 20 Jul 2023 00:32:51 +0000 Received: by outflank-mailman (input) for mailman id 566264; Thu, 20 Jul 2023 00:32:47 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb5-0001JN-1B for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:47 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id f145304a-2694-11ee-8611-37d641c3527e; Thu, 20 Jul 2023 02:32:45 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiBB017190 for ; Thu, 20 Jul 2023 00:32:43 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jul 2023 00:32:43 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:36 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:36 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f145304a-2694-11ee-8611-37d641c3527e ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EbmXx/22LKfQRQwLVu7oAp+Y09LuzEaKSN1INqGdI2J44hndpKNfevWbKi3DuHAyyQkXeBGvpJpY4RFOhEiEk55p3yVgwvUJZmeToiIrooS9HVYHQQIKcwiLbmZyclS4ihQgo22PAv3DbpNSajymtQvPEkHVzl7MnIyPafKzuiNEAzIV2/2+NxHUhem1AQfllvSVxfMLORxXsRiQMBqeF9ATu+ywKVJ//t383zISHbBIGshlxc3zlxuW0I+hIeHqzJfL+jbuCRBP8x2R6wZxL/OZKolihWYXmZLev2RwZJpEW8v2wvXdl51KUweKpLou3/Dvt0TShb6Uhetl9MvizA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yKI7dy86L8X5wiqLrmIlzaIxxL726p8ahnVAZh89ZZg=; b=WRll1E6h0KeiUgyUnJmi0g8d2h+oEznsETceUb6RfVNZjRrQQxKdXO/7z2PTfX7BxSLo3YZC48vPq5tKyeox5BIoiCuiAm3zSQnh1VDsxSl6BHhb5bmtK48Fx4dMAtmrTpP4dln1zUgXaCm9oysT9HF8lsLSwqYkCZBc3F/3KMj+AwpaR/C7seM1JmKDs0CwpT2R7tacW1PuWICA6L1g2IBTSFFHn5Dy5NykmAMGV9beiADuxwsr2IRdIV5EyLCXYVEJ13OpJXdX0e9yiuGbZMtp8HG0yi0B7yaAF85b2X9Qfxtc05cs4ua8t5meVdvEWCkTrEhbXJmH7gOxk1ciUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yKI7dy86L8X5wiqLrmIlzaIxxL726p8ahnVAZh89ZZg=; b=P0D1qmX0OevASwK/CWkJ+A2BOFfgJpF6nIO5lrhNFxUdy3k0xoLilE+33Fe13qcP8IJA66dUktDFkd8enhonXJGWr0GzbgyzU/PZrMNSZcNkorGOo0nkgVAoLCEcBHxmweATXAj9oa8fOX+Uib1HgQdQ5hmAbnFyBerJzos6PsZPhB923WMTJyA0yWG3l8kO3tvX7Yo2uu+YRaQ2QIQQ1aGH5EVnQwxGL0LSJ181mbQmlFdRPCaXpZ48nvzvCUp1rahivn9cOvBychIamG4hLDTx2wCC0ZxqdWg6pJD7GDMOLQYetyjZY+gqx/CPA9hP4pC/dgDAPIosxuk4U3QP0A== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko Subject: [PATCH v8 11/13] vpci: add initial support for virtual PCI bus topology Thread-Topic: [PATCH v8 11/13] vpci: add initial support for virtual PCI bus topology Thread-Index: AQHZuqGtdYewPbeaYUqE4k9SPFwhog== Date: Thu, 20 Jul 2023 00:32:33 +0000 Message-ID: <20230720003205.1828537-12-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 40e488b3-4a91-452a-6e5e-08db88b8d0d5 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 6PGJdDLYhXwsyvAPjswMBSvoufFZdqvQxN87xdbKiF+H5LDXpZbdP5fY8GoTtRLUNzDtbPLS1DCUz+BnAXeEjGsFhmm3xzYtO7IaqZ5cfeQYTZx3I9UroLTLwwPBgz2l9IV6P5tXXDUWN7FTRE0BvPYCWlER22Xs4RBdUmCjI/ubivr8ioEd+AN04giOXfEojChV9Z1VvumOQokyTAi3vh7eZfwxYni9ltpimeXM5tBQ9o0gPhw578Ewf+4VqSRNsEQ5j9dToaM1XYhx9EoUmmpIiMB9vzBtw2XXjeAcePCVoahs6LEXlTGp54CucpRO8k8LE0vUbjpMua0ItaqmVPvMszCLImpJh12pvSuJBFig601Pxfu6/EI1/ob3eYxYU/mj1Fo46ckbZgzLZuiAEGSgbqOo8RWsF4fY5Ss7Zh99K5tGW1cjb5uqpNrlDQN4NwGlMQj6apGtJj3eTWCT4q/v9G3a3C003taXWpZx0wrF5COKFvJrIiiN1rQzHbqdixEA345GiI1XGTYheWjPm0IZuVO9yKbIrX23pkEhVrkeoQh4E95PCp8vl1qdphKeeukQrXAfxkrNnZgb+tHL9lfOiEUEuUmNPGGFkCAbk9Q= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(6666004)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?36oUKNGtNs106p3sGC1Mt3q?= =?iso-8859-1?q?chr1vTDTf914cJKzkCaXH3M8CoDnK2k/yx3eIQW+41w8IVJ45wQXB9ANB6MY?= =?iso-8859-1?q?V4t29rU18cz36BKytQR5G9SjKz+5BgHpX3ZeyT2JPq4JkZfw6EyfZdVpDZna?= =?iso-8859-1?q?cJXhY5xcCZVBt9VjP8wrlL8FyDk0YcwcRL5vqcTOe/vM9TNLzLa981GdKd7K?= =?iso-8859-1?q?21twFV4ywFprFKMNMZajMqfvXOfvh4xyR2p0msktZVnYJTrg83f6f9Z+IUqv?= =?iso-8859-1?q?yrTB0/hIhRxDVZ1p46VdIuFI/Lgawwg0ADh8fspP6leo74Yybq4bEnqseAXA?= =?iso-8859-1?q?WWU6JPNt1qFmuTOUoC3z7ouZnYcUVoU+R+zy/k6gfFqZUu2KDPQZtGOFo4dD?= =?iso-8859-1?q?/MvxEcie1HeOOvrbmpc2TmpzKDVajTqhpBoGE2w3VOL6soo7mNZxmsiytYsl?= =?iso-8859-1?q?pLwAjl5Sq3rP0/wOPf9s6FAUw47CPBfNnI4O14NH6KAomaourvgmIDN2+2wZ?= =?iso-8859-1?q?pphN9soNKtRHe5rXlce+/YJ5fHF5XexcmAAALo9vCoUOGesnPEHRFcqXoco9?= =?iso-8859-1?q?UqeNdq1erxNoe+k/x5o63sjtzVMpxfRrJbfBBEj/DMD18MbPhilM/HZPrpcG?= =?iso-8859-1?q?UxCFbUsle+JLfFFQLTN6botHSDlM4TK8f3ziiqiCihknqeB2xvN1bDfxnxz6?= =?iso-8859-1?q?vZl+HD+0TAkLoo8gdQ0GR8bG1MbIbpz1qttfCY1VgGizEO6MUSg2QUQGN8p9?= =?iso-8859-1?q?QRfpqECWDJK0MyTsUXwSbpLPvjqe9ZHrop1z2w8LrN1oO046PrJLJShJyprO?= =?iso-8859-1?q?Ya4maLlUy8f6mCp4upjV6L/rBppqJ4OcvGaTKISGC825gIpwHElnM69PF3Ce?= =?iso-8859-1?q?AexJRQUmeF1q+YL/wNyOxQT+csUmN2iFTUErkr1DUj8ojRdQ0bDgjeTd27y4?= =?iso-8859-1?q?o4l/FzhrKT+vBjaRS5O5MPYwnelaOzGMTvOtmfgtbwsEFNuFAiLtEeoupcop?= =?iso-8859-1?q?maHVOFi5RvM+zGM0D+gaV95MZeOSzh5cVXAKhF4vRDBLvraJ+OtsTfobFkrU?= =?iso-8859-1?q?N5fpe0Ty4RGXXqutfLveH86xRCmJinHwIld+ACbSWw+ZjtBCsxH5uz0IjfTo?= =?iso-8859-1?q?tMBLWRMG/Fr8vVlV11bVosHZq7skcysJ4ifcRWw9RZXrCQmffLWrrPNusYd6?= =?iso-8859-1?q?TgpN2/lYuHNJPirFjN1tDKv3p4neNTSCMTSY4ELP1IFNQnNVPXDOq9NIA3Ln?= =?iso-8859-1?q?eR7H2lq4RXAG9QpVTN9bIqFbdEa0whHVkXEugSYmz8N/5H32GuZDvcxOE5Sk?= =?iso-8859-1?q?9MflUbK4y8Km2JgYs8VuVtWxV/rz08RuXa/Jf42jL2SZ5cWdY8sDagDvGuK9?= =?iso-8859-1?q?rpWw9bHFqHQNyv0Wg9pCKv2fgGmw1Hv3iE0NekMdDd0XLjhWyBZpRaulHSss?= =?iso-8859-1?q?EPHwkfp2KweMt24kXXHBChKZCOq/IoetmFv+3z9FlbbeoH/AZvn0vJsjL9RB?= =?iso-8859-1?q?HPjMsVghWjiOAKwluC7krf5LmOBFLlv8MQ4KcyBLo2GRVl46q6301n22XkIq?= =?iso-8859-1?q?LuXp/Nw01SZx2fwGT41t5FuXWbpzmkbiLyKwJbJAq+VKZh4FgqlcpdBZBRbk?= =?iso-8859-1?q?PU6AjPHlavlRwlqAB7fJzM+t/SbUYQMmDZZP/mA=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 40e488b3-4a91-452a-6e5e-08db88b8d0d5 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:34.0150 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: k84+8ahfRZfxLDIqeXyM/qpkacYmv3X1KEXBSPF3Lj8xewp0T4Uc/22arBwdoywXsURyeQsYFBDdpPIl3aI0NbjIFv/0HB4nAACfIZtVxH4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: q9JJ6ULmBvRj2LdFzvGmWJYkebN7q3t- X-Proofpoint-GUID: q9JJ6ULmBvRj2LdFzvGmWJYkebN7q3t- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Please note, that at the moment only function 0 of a multifunction device can be passed through. Signed-off-by: Oleksandr Andrushchenko --- Since v8: - Added write lock in add_virtual_device Since v6: - re-work wrt new locking scheme - OT: add ASSERT(pcidevs_write_locked()); to add_virtual_device() Since v5: - s/vpci_add_virtual_device/add_virtual_device and make it static - call add_virtual_device from vpci_assign_device and do not use REGISTER_VPCI_INIT machinery - add pcidevs_locked ASSERT - use DECLARE_BITMAP for vpci_dev_assigned_map Since v4: - moved and re-worked guest sbdf initializers - s/set_bit/__set_bit - s/clear_bit/__clear_bit - minor comment fix s/Virtual/Guest/ - added VPCI_MAX_VIRT_DEV constant (PCI_SLOT(~0) + 1) which will be used later for counting the number of MMIO handlers required for a guest (Julien) Since v3: - make use of VPCI_INIT - moved all new code to vpci.c which belongs to it - changed open-coded 31 to PCI_SLOT(~0) - added comments and code to reject multifunction devices with functions other than 0 - updated comment about vpci_dev_next and made it unsigned int - implement roll back in case of error while assigning/deassigning devices - s/dom%pd/%pd Since v2: - remove casts that are (a) malformed and (b) unnecessary - add new line for better readability - remove CONFIG_HAS_VPCI_GUEST_SUPPORT ifdef's as the relevant vPCI functions are now completely gated with this config - gate common code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/drivers/vpci/vpci.c | 72 ++++++++++++++++++++++++++++++++++++++++- xen/include/xen/sched.h | 8 +++++ xen/include/xen/vpci.h | 11 +++++++ 3 files changed, 90 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index ca3505ecb7..baaafe4a2a 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -46,6 +46,16 @@ void vpci_remove_device(struct pci_dev *pdev) return; spin_lock(&pdev->vpci->lock); + +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + if ( pdev->vpci->guest_sbdf.sbdf != ~0 ) + { + __clear_bit(pdev->vpci->guest_sbdf.dev, + &pdev->domain->vpci_dev_assigned_map); + pdev->vpci->guest_sbdf.sbdf = ~0; + } +#endif + while ( !list_empty(&pdev->vpci->handlers) ) { struct vpci_register *r = list_first_entry(&pdev->vpci->handlers, @@ -101,6 +111,10 @@ int vpci_add_handlers(struct pci_dev *pdev) INIT_LIST_HEAD(&pdev->vpci->handlers); spin_lock_init(&pdev->vpci->lock); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + pdev->vpci->guest_sbdf.sbdf = ~0; +#endif + for ( i = 0; i < NUM_VPCI_INIT; i++ ) { rc = __start_vpci_array[i](pdev); @@ -115,6 +129,54 @@ int vpci_add_handlers(struct pci_dev *pdev) } #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +static int add_virtual_device(struct pci_dev *pdev) +{ + struct domain *d = pdev->domain; + pci_sbdf_t sbdf = { 0 }; + unsigned long new_dev_number; + + if ( is_hardware_domain(d) ) + return 0; + + ASSERT(pcidevs_locked()); + + /* + * Each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ + if ( pdev->info.is_extfn ) + { + gdprintk(XENLOG_ERR, "%pp: only function 0 passthrough supported\n", + &pdev->sbdf); + return -EOPNOTSUPP; + } + + write_lock(&pdev->domain->pci_lock); + new_dev_number = find_first_zero_bit(d->vpci_dev_assigned_map, + VPCI_MAX_VIRT_DEV); + if ( new_dev_number >= VPCI_MAX_VIRT_DEV ) + { + write_unlock(&pdev->domain->pci_lock); + return -ENOSPC; + } + + __set_bit(new_dev_number, &d->vpci_dev_assigned_map); + + /* + * Both segment and bus number are 0: + * - we emulate a single host bridge for the guest, e.g. segment 0 + * - with bus 0 the virtual devices are seen as embedded + * endpoints behind the root complex + * + * TODO: add support for multi-function devices. + */ + sbdf.devfn = PCI_DEVFN(new_dev_number, 0); + pdev->vpci->guest_sbdf = sbdf; + write_unlock(&pdev->domain->pci_lock); + + return 0; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct pci_dev *pdev) { @@ -125,8 +187,16 @@ int vpci_assign_device(struct pci_dev *pdev) rc = vpci_add_handlers(pdev); if ( rc ) - vpci_deassign_device(pdev); + goto fail; + + rc = add_virtual_device(pdev); + if ( rc ) + goto fail; + + return 0; + fail: + vpci_deassign_device(pdev); return rc; } #endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 80dd150bbf..478bd21f3e 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -461,6 +461,14 @@ struct domain #ifdef CONFIG_HAS_PCI struct list_head pdev_list; rwlock_t pci_lock; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * The bitmap which shows which device numbers are already used by the + * virtual PCI bus topology and is used to assign a unique SBDF to the + * next passed through virtual PCI device. + */ + DECLARE_BITMAP(vpci_dev_assigned_map, VPCI_MAX_VIRT_DEV); +#endif #endif #ifdef CONFIG_HAS_PASSTHROUGH diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 6099d2141d..c55c45f7a1 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -21,6 +21,13 @@ typedef int vpci_register_init_t(struct pci_dev *dev); #define VPCI_ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12) +/* + * Maximum number of devices supported by the virtual bus topology: + * each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ +#define VPCI_MAX_VIRT_DEV (PCI_SLOT(~0) + 1) + #define REGISTER_VPCI_INIT(x, p) \ static vpci_register_init_t *const x##_entry \ __used_section(".data.vpci." p) = x @@ -155,6 +162,10 @@ struct vpci { struct vpci_arch_msix_entry arch; } entries[]; } *msix; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* Guest SBDF of the device. */ + pci_sbdf_t guest_sbdf; +#endif #endif }; From patchwork Thu Jul 20 00:32:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6795AC001DE for ; Thu, 20 Jul 2023 00:33:02 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566265.884935 (Exim 4.92) (envelope-from ) id 1qMHbB-000430-Aa; Thu, 20 Jul 2023 00:32:53 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566265.884935; Thu, 20 Jul 2023 00:32:53 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHbB-00040Y-0P; Thu, 20 Jul 2023 00:32:53 +0000 Received: by outflank-mailman (input) for mailman id 566265; Thu, 20 Jul 2023 00:32:47 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHb5-0001JI-Nu for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:47 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id f17084d7-2694-11ee-b23a-6b7b168915f2; Thu, 20 Jul 2023 02:32:45 +0200 (CEST) Received: from pps.filterd (m0174676.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JJCjt4031890 for ; Thu, 20 Jul 2023 00:32:43 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2059.outbound.protection.outlook.com [104.47.13.59]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgcw1sme-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jul 2023 00:32:43 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:36 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:36 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f17084d7-2694-11ee-b23a-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Hq9NQ4T9GFEyCjOepuzmQUYB2AOapblkNLljB0ubBBmeeNcth9BzZqzzS8CqFlDtiuV+aoLebFeawm/kMNPPCJ6nxv2kFP1PiPegwGu9MO7GPL5FvBhkS2uQXE8j/O5UAe+7XuNwUQFmsXcQwc67WM+cLXs6QnF1ZpxwhpOfzE/A0UlkUxKI1CnUC/FER0UG9rCjp3zeEis+et37Lny+tFmZo/hrIw46kj2uUc/GCD7rljEO9Oqf94rXMOm/fv1S8vXPBCKzi2L2EdUr3JaTUM5W0DIv6oVvlGNfNnhnBlVH61i+5FrG480AeQ7J07i8XfT+qSFu10sEIzUo33NoFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oJF4BGYeq1qshKG9OZht+I75cGYA32L43+SEUEeNDk8=; b=MKLqHiqVUweJuEplcHL7CtXNA4Cu7CpARupbUdk3AhVrg30wQqi4mFPhmIwqtYC/66qEY6leSZAIiASxtR3TNgzE5yOyYSFXHuipk4dZkKlCTFVt0j8ykMDUV21yCfXVQYgwVvL6QCtYht50ctTtuocPy5xlwQr1SKRE/o01/UmG7kHEJ8P7QB1PhEBuhO35sCFFR3LO5rzwjaFK6PEmsB5KJCc3XHk63ssmillaFTs0LwF7vkyriSxaZZmHLtyEQRhl/tQrEW6Zz5XRT6ZMhC6NKTFiqaR1Dyo13iaQFlnf+U1YMRIRCqBJc2vM5PLCfwzfuh3fhE2FK7LGKtFk9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oJF4BGYeq1qshKG9OZht+I75cGYA32L43+SEUEeNDk8=; b=cAqDNUWcf9qr1dkrr+F5JTg8jqEPmB2+AvXdzepu66WHSTHzBzV6xNi116bDrkvw/QKoD38JGiFEdmiLwvIXXxMghcopUMEDE57zPfX/wY/Zy7+Qp2f+g5xFVdwL5uQY6p4oWfsDlAXifJ5tm52cpysSXQhMhjgt3jyhSCdqT3d+jzii78/KHOezI+Ifc4UML3wxTbPGKdWTVVeedjtRj70JrLgyl1IrVQ6bprYMqC/OZbcLHV3hkMhTzqr0+5YnsKuxx0KgS/uSNMup/XOSgTGJcglRcGX514Wg/EhemyLhav0w3WAlHkJba6mKaDfRJeIcUb93yWh7PG15CdYhsQ== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko Subject: [PATCH v8 12/13] xen/arm: translate virtual PCI bus topology for guests Thread-Topic: [PATCH v8 12/13] xen/arm: translate virtual PCI bus topology for guests Thread-Index: AQHZuqGt3D08el+/O0KSh2o4KcGPqA== Date: Thu, 20 Jul 2023 00:32:34 +0000 Message-ID: <20230720003205.1828537-13-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 77a64a71-2bb4-4a14-4390-08db88b8d10c x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Oqgb7QyutVeqAbEUZpCtx0QVejZDeJmyBF1IbYrUlDQCIOWpAMflv84pXv4rHFStmWV+Wh/GXDhxHOtD9wjDBDtVU99/FP/M1ZTv+zIvHGJbvdgnXDuohaHg6pkPAKmEK3qUo1a9wW/Kpp0T4GLmDtix2CP6bO7siWZNywROmANIOVqYaB+EbIAMSwGYV31VfcxI7lh/GovBgmmnhiRSKQemcRDsMt+KL3PpjXu2DDW4Bulrm5vEHTdOVhQuvGZxeZ/xTm+UO9L97kN4IiJKiY3bhnH2bon67G+Xp7/EwxhwqsoYTpEl2exA4DIbjnMFQYHNkByBc8qudCcgIGPXbvDGfWMk4asHAl/eQFEbq6kItVGztcAbohN+u5eDlk1rTrFyDp0EVU8vpz7uDsGxV0JbzcA5EnoTTIDf3JOj9lRJS83Yb9nBJ8nvh45ayJ43yKCy5ajxaSFo0OoOQ52kiMcOMfQfujfHfqh8hvqLU4VJcwVl0wcf7l7Ph4PEvWuRFQRPHhaW5uVDpC6XM5C90xzXVm04V/9ntOSf1ZeXMaBGt8eSYJhhwUjF04lNInZ/JLCeLZP/u9o4eJ2KlIkmE4Uu5EFOdrNCxYKitEJ7I6RRY49UBvbQQ86bM3y0oKZt x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(107886003)(186003)(6512007)(6486002)(122000001)(478600001)(66946007)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?XUkEb/Bx0A+PVp0RtUNVkn5?= =?iso-8859-1?q?3Sgv323b/A8GWV1gvBOLywzIwRhmA094H49dEJD+jTw8tjpe6gSHKxdCfjtq?= =?iso-8859-1?q?blm+fBGR6w7BsCfsuczlXjHUAR5MVpLCHSzzFK3MOUwcOo/HUi02izrS3ILU?= =?iso-8859-1?q?Si7OtkYmE44ZOGGzBwn32unb6LrqrQAIR5UlP+YaYGIWI311kvmnAuSxu+fa?= =?iso-8859-1?q?VQsMF42UIe5wf9pXxgc1jc3qkrD8gfV37M79kRwIbxa0kWN+eTg77eJUclCJ?= =?iso-8859-1?q?hqK2Xea64eJJ4bpkHMKawunnbWc+alEziYR5sUcFstd0F/r0tEndRK4HaC7u?= =?iso-8859-1?q?3P6eHQRnyCnv8R0qm31gCaN34PgiOnodTvRRnCY95cyN/Ca0j6LAiy5hKufx?= =?iso-8859-1?q?TdT/BmMShA1E886D0L1jnshnLNB4xwJgTr2vkcby4do3m1bXogafojUMIDIq?= =?iso-8859-1?q?01c3y8EY4MaoXm+Ui2Za8Ymp/j5oXMt2kWAxp70/k0ZZmQWuQ/qZmE+wDgrH?= =?iso-8859-1?q?MX6JslH69IfcOjVtmjBPfWD4/izuAueAnM/buIw6WJX04N8H9++1/R1BfxZp?= =?iso-8859-1?q?WBaFeyZ+LgbOeYkAKs+S9h3tXO47YYGGaCKQfRgxBjM4IEdLAoWDRk/zx9Ff?= =?iso-8859-1?q?26HNsNP7Vk5kkoWVDUT68gUAxA42LTsYvdZAqprRbR91uUrIP+Frltfv0Kqh?= =?iso-8859-1?q?JCY10iQTkqXV8CgQ9d1psMcj6db+GLgtGfc6Wrar1oaapyXPqgLu8VdYct94?= =?iso-8859-1?q?0OPPyOJL2S1pTQObzEQj08jMci3taLay5vk+JumBqKTvKJZTciVR8xqmX/Cm?= =?iso-8859-1?q?nBiifxZYlxeerhPqsPavHhhfEHkOg211DGM/5SNB2EEvA9vd3H9QgRtMPt7o?= =?iso-8859-1?q?lRB3jV7A4v6gIlKJu17+fU1ocEwzgp/z9iZxP+wi6RHSN2nNSZFjyMgCzXjh?= =?iso-8859-1?q?AWcjvU4JXhCgYCRx1PokhrcF7qzuiBEY3TYbWK+ZRlXEvR0DQAd+ZIJJBGOt?= =?iso-8859-1?q?0FQGEev4O0AFe1f5ijqdi8oLqlRrdMUxw1KYM06eMabIopnXcXrWnGqwoC7A?= =?iso-8859-1?q?z/mQhKvcmYQqKakfaY6y3CF4IMAz1cnUbchhqGLzvSAltE+1J5bunSOijkYa?= =?iso-8859-1?q?RB6HLgNzDbPk7CidDwPRIQUZHEhlB3nqs8PRy+dFM+0KgsleUx5dsAxJyyKw?= =?iso-8859-1?q?MrfEE5LhlCp+ryi4TDjz/a+EjkL/tWKTZZKn5T3Vi5ZBgNuUfHalX7pjF9yM?= =?iso-8859-1?q?28exOGPsfLaLnIUu9UH5NLWtjb/2pxyyg+l6Pr8GBbJ8vdccvexryVPeVqj1?= =?iso-8859-1?q?wx9fGev5hwdca1DIzUq+YyWCqiBR/lYvqJYnXhaJpZLzymULjz4Ki8OiJkoz?= =?iso-8859-1?q?SRgP7gokHJQd45zQJP7psx8W2zuyduWaibpCEFBsSP4hhd3vdC6/YPkAib2h?= =?iso-8859-1?q?m2c7bwiJ8iBaglORC8L1hy1o9evmgX/DEeqWGVo/VQD2OG1iTIbJ/4pbtSlS?= =?iso-8859-1?q?78Y+bVPEOYTFILzEtsk7ZMOo9i53S538J+zYZqk1MF2vNAA/wHVoGxjqC7uu?= =?iso-8859-1?q?KW9Y7SVyn2P7klCXB0wLDbfQuWqtTOy/s1EDjbKow+5Xx98kMzNTJFFiYbpD?= =?iso-8859-1?q?HkbcZ0Pm4ZRot6LtY9NejlZehxockna1wkJop3w=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 77a64a71-2bb4-4a14-4390-08db88b8d10c X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:34.8480 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: lSMPc/r6SKKbIIxYjoKQ3VF4smbwheeA3VKxw9l5/4ep8qOJhZ7ShyhEpDEYYq7KimGzXYIrYhPcj7JBK/b8DEXDuj44/MoQUwhOnL+Oz38= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: 99B5JOBEbaenm1Hfi0uSRfKSpYSUhGpZ X-Proofpoint-GUID: 99B5JOBEbaenm1Hfi0uSRfKSpYSUhGpZ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 phishscore=0 mlxscore=0 spamscore=0 bulkscore=0 clxscore=1015 impostorscore=0 adultscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko There are three originators for the PCI configuration space access: 1. The domain that owns physical host bridge: MMIO handlers are there so we can update vPCI register handlers with the values written by the hardware domain, e.g. physical view of the registers vs guest's view on the configuration space. 2. Guest access to the passed through PCI devices: we need to properly map virtual bus topology to the physical one, e.g. pass the configuration space access to the corresponding physical devices. 3. Emulated host PCI bridge access. It doesn't exist in the physical topology, e.g. it can't be mapped to some physical host bridge. So, all access to the host bridge itself needs to be trapped and emulated. Signed-off-by: Oleksandr Andrushchenko --- Since v8: - locks moved out of vpci_translate_virtual_device() Since v6: - add pcidevs locking to vpci_translate_virtual_device - update wrt to the new locking scheme Since v5: - add vpci_translate_virtual_device for #ifndef CONFIG_HAS_VPCI_GUEST_SUPPORT case to simplify ifdefery - add ASSERT(!is_hardware_domain(d)); to vpci_translate_virtual_device - reset output register on failed virtual SBDF translation Since v4: - indentation fixes - constify struct domain - updated commit message - updates to the new locking scheme (pdev->vpci_lock) Since v3: - revisit locking - move code to vpci.c Since v2: - pass struct domain instead of struct vcpu - constify arguments where possible - gate relevant code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/arch/arm/vpci.c | 47 +++++++++++++++++++++++++++++++++++++++-- xen/drivers/vpci/vpci.c | 24 +++++++++++++++++++++ xen/include/xen/vpci.h | 7 ++++++ 3 files changed, 76 insertions(+), 2 deletions(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 3bc4bb5508..66701465af 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -28,10 +28,33 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, register_t *r, void *p) { struct pci_host_bridge *bridge = p; - pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + pci_sbdf_t sbdf; /* data is needed to prevent a pointer cast on 32bit */ unsigned long data; + ASSERT(!bridge == !is_hardware_domain(v->domain)); + + sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge ) + { + bool translated; + + read_lock(&v->domain->pci_lock); + translated = vpci_translate_virtual_device(v->domain, &sbdf); + read_unlock(&v->domain->pci_lock); + + if ( !translated ) + { + *r = ~0ul; + return 1; + } + } + if ( vpci_ecam_read(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, &data) ) { @@ -48,7 +71,27 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, register_t r, void *p) { struct pci_host_bridge *bridge = p; - pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + pci_sbdf_t sbdf; + + ASSERT(!bridge == !is_hardware_domain(v->domain)); + + sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge ) + { + bool translated; + + read_lock(&v->domain->pci_lock); + translated = vpci_translate_virtual_device(v->domain, &sbdf); + read_unlock(&v->domain->pci_lock); + + if ( !translated ) + return 1; + } return vpci_ecam_write(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, r); diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index baaafe4a2a..2ce36e811d 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -177,6 +177,30 @@ static int add_virtual_device(struct pci_dev *pdev) return 0; } +/* + * Find the physical device which is mapped to the virtual device + * and translate virtual SBDF to the physical one. + * This must hold domain's pci_lock in read mode. + */ +bool vpci_translate_virtual_device(struct domain *d, pci_sbdf_t *sbdf) +{ + struct pci_dev *pdev; + + ASSERT(!is_hardware_domain(d)); + + for_each_pdev( d, pdev ) + { + if ( pdev->vpci && (pdev->vpci->guest_sbdf.sbdf == sbdf->sbdf) ) + { + /* Replace guest SBDF with the physical one. */ + *sbdf = pdev->sbdf; + return true; + } + } + + return false; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct pci_dev *pdev) { diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index c55c45f7a1..7d30fbdd28 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -286,6 +286,7 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) /* Notify vPCI that device is assigned/de-assigned to/from guest. */ int vpci_assign_device(struct pci_dev *pdev); #define vpci_deassign_device vpci_remove_device +bool vpci_translate_virtual_device(struct domain *d, pci_sbdf_t *sbdf); #else static inline int vpci_assign_device(struct pci_dev *pdev) { @@ -295,6 +296,12 @@ static inline int vpci_assign_device(struct pci_dev *pdev) static inline void vpci_deassign_device(struct pci_dev *pdev) { }; + +static inline bool vpci_translate_virtual_device(struct domain *d, + pci_sbdf_t *sbdf) +{ + return false; +} #endif #endif From patchwork Thu Jul 20 00:32:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13319665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA910C001B0 for ; Thu, 20 Jul 2023 00:33:17 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.566266.884958 (Exim 4.92) (envelope-from ) id 1qMHbH-0005Eu-8f; Thu, 20 Jul 2023 00:32:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 566266.884958; Thu, 20 Jul 2023 00:32:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHbH-0005E7-1d; Thu, 20 Jul 2023 00:32:59 +0000 Received: by outflank-mailman (input) for mailman id 566266; Thu, 20 Jul 2023 00:32:57 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qMHbF-0001JN-7d for xen-devel@lists.xenproject.org; Thu, 20 Jul 2023 00:32:57 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id f775df9f-2694-11ee-8611-37d641c3527e; Thu, 20 Jul 2023 02:32:55 +0200 (CEST) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36JMeiBC017190; Thu, 20 Jul 2023 00:32:44 GMT Received: from eur04-he1-obe.outbound.protection.outlook.com (mail-he1eur04lp2057.outbound.protection.outlook.com [104.47.13.57]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3rxgyx9k43-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 20 Jul 2023 00:32:44 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by AS8PR03MB7783.eurprd03.prod.outlook.com (2603:10a6:20b:407::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.33; Thu, 20 Jul 2023 00:32:37 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::68d2:d90f:ac32:7c85%3]) with mapi id 15.20.6588.035; Thu, 20 Jul 2023 00:32:37 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f775df9f-2694-11ee-8611-37d641c3527e ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MuNNQsPzZyGbm3cQ4i6q7pkUtW05fdJNEW4n3hl5Q1Bz3DAtWbLTjHUXtfPm6yigIm0GrMDcDbyv0FFn/AZu6eHTRaso/eAReMLGAD7RcqfeoSs2VEwRKwdza5PP5S2pVCJkKnezPXLXzVV3jhTlWMdqflnZBG71g5fZYdHCq/7iY0rbXa9SVWVI83HwQdOTSuGELGqS7nmYaxCObSGJ3KM9wtrX3NDXaRQmZyeJ8s2tXQYTZpfE8AJHPpV1LzgsKoTbZiepLbO8mdjrI5p7Nlh15ELwZZD9Wy5dqg5StRneCh8bNFE+e21Mvz2MnI8mCryCTFcGQIMO6tfeRFCQGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zOeSMPEmX5pm3X2mdny5gRhNwdhBLZKMFqHNsWU7Urc=; b=Zy5UCkwd1PL+YvSbZoS9AprGbpEYwmnF5X48sWWe4KttoPrAcpp2v9UDDUqfOeF7B2O33Eygf/tmSRzWwBmywlOZ/1qJjPUPLcnIm1LtrCa6fEj24Ga8ddra4grxxAalhp0RzchLyxrJksbGOlt0nX8aecwRseCvPI8qiplH6Qg9LJIwieupzDAs9fQljEiDESmWOGFS6wGmukEimw2s9FsmFr81/WkpO+9QcwQUaEoRnE984F1oNitEHUziwXKGANBxhwhUY4GVNAUQAjIxuCnYao64IaKeJAsceaiy8mTbvbWougJahYrJbDMaPGSePywGPOGnWcsRgm3Xp04EKA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zOeSMPEmX5pm3X2mdny5gRhNwdhBLZKMFqHNsWU7Urc=; b=izNF63YXVb0+slPPZDorbJg1hNAt0cHS2Cv5dBZxIXhuEjNmqJ4694OMETo0ixqmMc9+2MVtGPhgpUeQfk7H6VSkI2dzdsVBQlrj2SlZBOr+eQa02TWkxNGWBztZDae/mR88YZDp9HOZSaQYpcjXFr8/fP36x3oJJwpGHfu5YUdEwZsm1+3wHDMOAzujENz/SNTV6JTT2rqjUP1ahornRaRWjHcVJfLLv7DBZWCf8lcycKXhxKL7NkmgsA7/X/k43IaHHKsH6FJyLt7H085GHa+02+G72tkh2cu/kOqmEoBNFXkdNL1uzWI5mXrMRt18gIzSVZFU+c2fnc37AIo/fQ== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Oleksandr Andrushchenko , Julien Grall , Julien Grall , Stefano Stabellini Subject: [PATCH v8 13/13] xen/arm: account IO handlers for emulated PCI MSI-X Thread-Topic: [PATCH v8 13/13] xen/arm: account IO handlers for emulated PCI MSI-X Thread-Index: AQHZuqGuy4TImO+CVUmVm6UGc30HEQ== Date: Thu, 20 Jul 2023 00:32:35 +0000 Message-ID: <20230720003205.1828537-14-volodymyr_babchuk@epam.com> References: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> In-Reply-To: <20230720003205.1828537-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.41.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|AS8PR03MB7783:EE_ x-ms-office365-filtering-correlation-id: 6ac755eb-59f8-4879-2b37-08db88b8d156 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: oelM7Z1bEuOVBBZ3IYujx3sAcD7DroaJvScaCRnLkjNCsnB0uk2KRPuKFsHcRktevtdSdkEhp9SNUTZbwjRt9IZsbOOHRUkdnO/15J0bvm2AKRtkzlMsOiICScjUQpkFHDbCjWRw/BygXxttNCGbdbMvBLhR2c7G4sTRtS3WwjB0/B2Y/7ygQJC3VnGNCE+l2ZbiJX7Wjt6gffyyVOVmTiJcBC3mugvttRmmqWKuqrTEc3i9DuseRnpZdreqYIv21SETcqtgznjhrQJ/yJvilTJLHBVLoEUG3aUydlRjfLMZxgQTqw/K7ApS6YAkdvO7Z6UcGM1IMRWWVihEB+v1ESHZj2YG59fWFmKjYPDHgsoE1qXqC6m/Fc5gBCAlt8/4bNOJcH705Jt97DGmw5UQsFk3tWuTnj/Qty/OXdJyZ7Dc9S2ktZEaIchsZ9E/VIvXhcBoXB50HORsiBOODPQGeSS/FaPCHipSUMwS6WPwNqFuRDAoUpW9rUfRn1cZv7t6nd3DEcbfcZygYQv/EOe+N3MvIDE4P+wbB7+NJWt3zpfMqc8IDa41nRSk81tAkLTTSSDK/ij1Vhqkr09MP5peASV9Cn7sDNRMnSVpOafJiRPUjNrEQQT4u7qsRZBf792q x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(366004)(396003)(376002)(451199021)(55236004)(26005)(1076003)(6506007)(41300700001)(316002)(83380400001)(2616005)(186003)(6512007)(6486002)(66899021)(122000001)(478600001)(66946007)(54906003)(71200400001)(4326008)(64756008)(38070700005)(66556008)(76116006)(66476007)(91956017)(6916009)(38100700002)(86362001)(15650500001)(5660300002)(66446008)(8936002)(8676002)(36756003)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?t+zdmAPD7M+3QdUvGvlMfXm?= =?iso-8859-1?q?E6GDWI09QK8YTLmkWKKP99bQ9QdfMmSygN2cCUT9Sd9/CWkwf1yVwpJdPUyB?= =?iso-8859-1?q?j4dFmwrQ+NZ1hVMdSZe04C0o4shwXmvUxWRGR5slUbKMr8UpkKlWw51uJn84?= =?iso-8859-1?q?dAXDuH58OBWkSVO6YR66KddJLfGAlSJ+fvopaf7eoDEh7P1rlP+E0nZDq0MW?= =?iso-8859-1?q?baZzZ0Swik/qv93eP6Shc801/5ZP+O+WP8Th1le8qiGPYHJt3o84FmZaEeI4?= =?iso-8859-1?q?7rYfsSQu0hI7QOZSPhn5KkAKaJlEcbChHgkDEK/sHIQkG1yUPmQQ+Wzdei6W?= =?iso-8859-1?q?XFuY3shTFezlkzZhJl+6VE7SbG29PTDwh5xAzChNW3C7DBdPB9MVuEPn95RS?= =?iso-8859-1?q?xnPKMoYlfR/m0+nu9xpqKfhppow95kQ39jpCI1/baGmSwvqf3EUrF0KQJCq8?= =?iso-8859-1?q?BluO6wAQWCh+jGIUwHi9PdDl3AYJXKjfpFo98UNMAQP3h240QhGPJExLL30n?= =?iso-8859-1?q?zg9pRVdu6nC3WmYHYkzPxbatzsf7G1Y17W02Y9j6BeV6LWLVdhvEumr1H+Py?= =?iso-8859-1?q?xZsp/PpvxOold1jw/gPOahBP093SuErXKA5GXhf92VGGH7SykdkSjnD9vbDF?= =?iso-8859-1?q?kcrTDIK4bFJIWUwH+j13RWkl0MCs8dIns5frAhjmd5ONxxS+LhkCYuY1Qank?= =?iso-8859-1?q?5tAmQ40BHx1luxZlJmdthcTNMB1L+Q66XYYTYydrh4eQ8sWIe3gfBLdexqmj?= =?iso-8859-1?q?kNq1KP7YAZ3Vewb6yYxGb5+vMd48PIqWo8P6rUCYy7HCFSeQEZSRi/GYK843?= =?iso-8859-1?q?rQvkV6jCT1q63Ttc30sk6r9xt5JdmF97ORlwHTIhVQSobLmcAAhw5OMhSrDs?= =?iso-8859-1?q?Vag3j+libG09lqeLKS6U3XUYfIQsl9EBKI2tpJCPCXG0tSHRKYwvBvyyGqAC?= =?iso-8859-1?q?0uQpVmzVOGFuzN6vwB/GUT/wAbdpW7CQH6MDE5VoEKfDTrQPO5p5vsvpQSSk?= =?iso-8859-1?q?aociDGE3fDGw0HsXXsvGIH/vBedOyV8UEcuQLizFRZcasZBH1Msu9XjFhUpi?= =?iso-8859-1?q?h+R1wT7Oq8k8juvCZMC3ePCH5AW/WySvYKsM/n8my/cpLekFkEOPOD1YgF5A?= =?iso-8859-1?q?3tV+4CFDQJNdUXzC4nSko76WfyW7vaHEB1zEM28H9by4o8AaY8/CmO3bM0BX?= =?iso-8859-1?q?Nvk8cyqoOQeludQXso7wg+p08ltYqqPDt1totypZfNelB5wSqfDlW2nwJQeQ?= =?iso-8859-1?q?ZsmIesXVn0s14LoO7WgwvAY6Vb0d+fkFiBaIguTUOwZ+vUShqyjkQjZMnJmj?= =?iso-8859-1?q?9d6+6uq1kz6m8d0GeMp+yzWE4FkQCzc1aArYrx4qhKGAY3HM8RMOaom71dx2?= =?iso-8859-1?q?XlwNSbSX6oyu2VcepO0FFtiGAvQGN2jER8Pd9FpAa81VAWMMyLLvEk1CYiUM?= =?iso-8859-1?q?J8G361wPhRFZMPF30cNc7X4czsnzlsZfAfQq/itf/5g/JvvSB3nTmuQXQbff?= =?iso-8859-1?q?CFtaWZoigtTy9KZbW8mTNro2POYwNiyDw8T9TYIXiog2byZkk6Qz1lAbCPen?= =?iso-8859-1?q?65t2W/HA+5EL4K0izg58od+niESNmml39PvdDLt4CHN1ns6gkkEy7AjbWHgV?= =?iso-8859-1?q?4iP69/qW8D8jX0ZwiFrjoiby0P11ElTAEy+IP8g=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6ac755eb-59f8-4879-2b37-08db88b8d156 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jul 2023 00:32:35.8123 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: aUbo/Bc74a1N2QfKguoHLKPITtpKvZDZiD5pvtJdbGXHOzLxfZnsuGGCzQiTiPXbP4rbYqwlHcOVqWtPO4xWGnwjs9Vs7WMp0918AaTT1pM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR03MB7783 X-Proofpoint-ORIG-GUID: dy_s8m5gAloB9sJl1xY2va40dAXzmnBl X-Proofpoint-GUID: dy_s8m5gAloB9sJl1xY2va40dAXzmnBl X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-19_16,2023-07-19_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 spamscore=0 adultscore=0 clxscore=1011 suspectscore=0 phishscore=0 mlxlogscore=659 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307200002 From: Oleksandr Andrushchenko At the moment, we always allocate an extra 16 slots for IO handlers (see MAX_IO_HANDLER). So while adding IO trap handlers for the emulated MSI-X registers we need to explicitly tell that we have additional IO handlers, so those are accounted. Signed-off-by: Oleksandr Andrushchenko Acked-by: Julien Grall --- Cc: Julien Grall Cc: Stefano Stabellini --- This actually moved here from the part 2 of the prep work for PCI passthrough on Arm as it seems to be the proper place for it. Since v5: - optimize with IS_ENABLED(CONFIG_HAS_PCI_MSI) since VPCI_MAX_VIRT_DEV is defined unconditionally New in v5 --- xen/arch/arm/vpci.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 66701465af..cd9f5d0757 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -148,6 +148,8 @@ static int vpci_get_num_handlers_cb(struct domain *d, unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) { + unsigned int count; + if ( !has_vpci(d) ) return 0; @@ -168,7 +170,17 @@ unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) * For guests each host bridge requires one region to cover the * configuration space. At the moment, we only expose a single host bridge. */ - return 1; + count = 1; + + /* + * There's a single MSI-X MMIO handler that deals with both PBA + * and MSI-X tables per each PCI device being passed through. + * Maximum number of emulated virtual devices is VPCI_MAX_VIRT_DEV. + */ + if ( IS_ENABLED(CONFIG_HAS_PCI_MSI) ) + count += VPCI_MAX_VIRT_DEV; + + return count; } /*