From patchwork Wed Sep 30 14:56:27 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bharat Bhushan X-Patchwork-Id: 7294031 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id ED1ACBEEA4 for ; Wed, 30 Sep 2015 09:38:03 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id BE9F420641 for ; Wed, 30 Sep 2015 09:38:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8001320640 for ; Wed, 30 Sep 2015 09:38:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755338AbbI3Jh5 (ORCPT ); Wed, 30 Sep 2015 05:37:57 -0400 Received: from mail-bn1bbn0108.outbound.protection.outlook.com ([157.56.111.108]:8416 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754972AbbI3Jhg (ORCPT ); Wed, 30 Sep 2015 05:37:36 -0400 Received: from BLUPR0301CA0013.namprd03.prod.outlook.com (10.162.113.151) by BY1PR0301MB1269.namprd03.prod.outlook.com (10.161.205.143) with Microsoft SMTP Server (TLS) id 15.1.280.20; Wed, 30 Sep 2015 09:23:30 +0000 Received: from BY2FFO11FD032.protection.gbl (2a01:111:f400:7c0c::112) by BLUPR0301CA0013.outlook.office365.com (2a01:111:e400:5259::23) with Microsoft SMTP Server (TLS) id 15.1.280.20 via Frontend Transport; Wed, 30 Sep 2015 09:23:29 +0000 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=freescale.com; freescale.mail.onmicrosoft.com; dkim=none (message not signed) header.d=none; freescale.mail.onmicrosoft.com; dmarc=none action=none header.from=freescale.com; Received-SPF: Fail (protection.outlook.com: domain of freescale.com does not designate 192.88.168.50 as permitted sender) receiver=protection.outlook.com; client-ip=192.88.168.50; helo=tx30smr01.am.freescale.net; Received: from tx30smr01.am.freescale.net (192.88.168.50) by BY2FFO11FD032.mail.protection.outlook.com (10.1.14.210) with Microsoft SMTP Server (TLS) id 15.1.274.4 via Frontend Transport; Wed, 30 Sep 2015 09:23:28 +0000 Received: from localhost.ap.freescale.net ([10.232.132.243]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id t8U9NCDU000590; Wed, 30 Sep 2015 02:23:25 -0700 From: Bharat Bhushan To: , , CC: , , , , , Bharat Bhushan Subject: [RFC PATCH 4/6] vfio: Add interface to iommu-map/unmap MSI pages Date: Wed, 30 Sep 2015 20:26:27 +0530 Message-ID: <1443624989-24346-4-git-send-email-Bharat.Bhushan@freescale.com> X-Mailer: git-send-email 1.9.3 In-Reply-To: <1443624989-24346-1-git-send-email-Bharat.Bhushan@freescale.com> References: <1443624989-24346-1-git-send-email-Bharat.Bhushan@freescale.com> X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD032; 1:mLsmHiASx2bL8eKPUCWKuApe/4D21BwN5f21yBqoUT5Vj3JpkuYGEvhXaFLwX2bo8k842Ovp7J+2QZvyjvTTO5F6dKO4F0jND1cK7cqrTZWYorCNw9Wgd71qpv5QJUZ+qEfqm13RWX/Ju3olCe0d+ywIRBtqy5FjJM7A6Hz36uTDlntii+yDFTFnggNIvDd3zYYvesejvt5j/GG/5RRU8nDBWZAW0Cd7uFqzZXWCmD4bnQ5iwveg2ztvSVcxiSqbwhs7sFYBDNgLRgpIrKGKuSnqO+Q3Ct2hQTQb8D5mZaXJsresQf9PtOAF6C/0aUXIu6x4MTQrSA2c1S15cr6UmuCb8ezJTSLBxAnpWdHzt5r/9N1fhKg2ITR0Gg3dP57+Ruy22vfkAmjE3fj8e2GKVkW/I1lIqCBq2GavM63AxOK5zmIz4jpsSfN67BrwZ5dv X-Forefront-Antispam-Report: CIP:192.88.168.50; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(1110001)(1109001)(339900001)(3190300001)(199003)(189002)(107886002)(189998001)(50466002)(48376002)(106466001)(4001540100001)(76176999)(5001830100001)(77156002)(36756003)(97736004)(46102003)(229853001)(5001960100002)(68736005)(5001770100001)(81156007)(92566002)(5007970100001)(50986999)(104016004)(62966003)(19580395003)(19580405001)(105606002)(5001860100001)(2201001)(2950100001)(6806005)(50226001)(5003940100001)(64706001)(77096005)(2171001)(85426001)(87936001)(86362001)(5008740100001)(76506005)(5890100001)(47776003)(4001430100001)(217873001)(2101003); DIR:OUT; SFP:1102; SCL:1; SRVR:BY1PR0301MB1269; H:tx30smr01.am.freescale.net; FPR:; SPF:Fail; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; MIME-Version: 1.0 X-Microsoft-Exchange-Diagnostics: 1; BY1PR0301MB1269; 2:h6oSOXZTMOIrFbh4nZSQSvRo7rfPBu3DLDpgTmWhUiEcSUOnNHQ4RQN02UTQAC4LyUAA36U4w6dBhD/G75J5/fxgI/Urq5gW2EO1NPuUvG0gX0hiHevBb3BfCTa7W+zt05bHQt9hOAmt7R4xbQjzwRbnbR4E84B9YgzkBxNV5ug=; 3:a0NaYR34DiXyn+DFY6oq1BwxdRkTbe712tc+Nf3oJr8EY5HYaPunHQV9dUxuijf/DPr37/86tw1fW5eaQhUD5OmyY3N118buNBZLppFKX+ig7/pNP6AMcRyA0g1V45l0UQIFiDYeS92wc2YyGaGyeB/NCL4iq9Hl1Iz2CJ4fiQGoSdSKcdhUFBDRKMcC+kE4VHjJlWQdUCtxVp9ZV4/qzmUPpJfRwIFUIg3TKu9nfGQ=; 25:4fGEpSNBXh8g3YnXLnjBuv8mfwA5F7ttkxiRAYcRSFmGU7Qm5TXvZnmxs2k+2APRovqV8NwFSHNbxAmDdnBEdNCu5YOAH/Zc3h1bPFX/z6qzFDGb0WFfNWC4XNsoHSjmjWhl3lVhx7QGXcEK4VvBaumWmdbY6KqdF3+6YcO2IKPXpOJnsVLxZ/9ydXX8anAADMyMxy4cAyrBe2RWtRxdyNr6QlX0HsWuty6I0oTLbiuC2B4wB8pxuXr0QCW1saWE1sdRCtQ/4qywMjH4q8QTow== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY1PR0301MB1269; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0301MB1269; 20:JKjisbgF0d/aeb1D2hSIToIDyQlZ9uSAI+4SwojgRsnTb2pGOmGejDoyWmQ+m2ve14BLvLzBwXZ34rxN80AE7kZBLCUhBAxbdgaUwvTkcujArIzyFgCNRR5r3H70sDAghf96MvziytC503dLwba38D62ylYplGxlaUCFHevmkngnflmDyy+m9EAfM12NYMXPvcKJgw7SZYw7S2ZK/xsYR6kT1cePkWIrEp6YpZMO5aj/JxNrpMNUzX7QgpEUzbjdejYUn0BbG9KZEQ1hEmlj1PkaEBsc434nt1m0wlNKt2yC4hIfvW2jnjYaDmMyeDXUxoCV9fp5/jIiJ39vaDemwme64TRFQJnbS5BQOCyI/V4=; 4:9qaB+einqEKk0vsHOfwK4zap3x+UT3nwchQKDq6lrxPzlyxo3IDlWCsoOmFDnLn/eCccgsH1K+sXlDf0himtwYYZXlo9HbCI7b78hxkhYPvIF4qXbkTLVa8i1F/Dy3vfY+aSGRwllwY/w3XwAEJBOMm8xuaqjfjczf3zs31BuEEoz2uEbtt69zVkurWEOQnpWKp5ka9C6U7T5VGgAH6rnvb/qtYVl/SaYIG1leRECE/aIrJ1DNaZA5NUUVv7fs89yPEw/Ih+x95//bHDEbv15zKpZ+GC9emkn952gTZbf6hW3iriKkdVuNS7PGFJalZULO/3yiBWXMozo23MlS0t9cKHl2jJH5vfH3YZ32rZaEA= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001); SRVR:BY1PR0301MB1269; BCL:0; PCL:0; RULEID:; SRVR:BY1PR0301MB1269; X-Forefront-PRVS: 071518EF63 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY1PR0301MB1269; 23:bNyQCd0umW5GQtcPAuZ0+1FhunmbhfI73ecMYOZ?= =?us-ascii?Q?HeE4aUwNCsl5VlkF/0tMuvGAIrIYu2rDzqK+xOR84nQflnha6Y92KJ3ndoVQ?= =?us-ascii?Q?PlmOjGzcl5e7sVfgWvpx6BV2qUrUqPaCeHBlhnH7eU5L4o7Z5PFx6mDVYyS/?= =?us-ascii?Q?tqkW6zuUe9upa8mKeFdHZFZyrt57Jmgr3xzjgeoAiJNVPbOuWYgT2gtqzV6h?= =?us-ascii?Q?Fpx15ZMM9p8vvNDZxItp602dbtsVOfo7wgTuLokZAI26E43CtXvcChCq9VFz?= =?us-ascii?Q?7gy/VRNdAn6Euo3HfZfYDrBTjQ7lZina7UQXH+CEsY8acyf47EVvemSACHtc?= =?us-ascii?Q?CLMzw2sWURWDf0F5tBHrlmIXW4Pd+enNk92aOD95qxGbhQkrBMcFyobaM2RY?= =?us-ascii?Q?1uF6x22VeQEMhlIE9ruV2g9VSraTmYIZIcBDEk6oJ/fbBagONGH1EcDWpb2I?= =?us-ascii?Q?pRegNFbj8bB0i2gZfzg2xIn0W9zg6/L460zqoH+eNndWVyey07ylpi4wnSEf?= =?us-ascii?Q?3gV9ZnIbzV4Eow6elh5JhGCslWR+6U8wDG5eYW/PdZWOYDp3CHdxRDe9dTUF?= =?us-ascii?Q?+o7tcP+9mhdi3WmP408ncfEce96S2PqaElKmKwRvtnHJPaxS0OTCaX3oqNK2?= =?us-ascii?Q?vw+fZM5moD1MlS3DXdOi+NGd+jGtMPTZKuyJnkJPO60f4nl72sunJlsvjg1i?= =?us-ascii?Q?KQsRTsbdtS4PVqFHwaBBMYPsZlO6c/UIpt1XUdXodJ8/Vp3TpYVc6PwzeZyL?= =?us-ascii?Q?K91pqgeb2awcBSlHhUh2iP8hn0fe8H0QabXqbrkqQFSIFXv7L4FmejSNe151?= =?us-ascii?Q?Iewr99AR3MuNGXhNsn2j8c+zzLumhhRIyMo8XrUC58U+fmvep8f+0P39LXhk?= =?us-ascii?Q?11MHLJIwiV+5B2XhQu+/ivUhW4kOPUBoC/12FjPBSa/rbWOWN6vCt+xPEsqJ?= =?us-ascii?Q?FrjcvI9pCbMVo/lMZpShHvscJH6yf4zO0I7Hu+W/CBXwzMKPuQN1yIicObmS?= =?us-ascii?Q?Ys0nm/yQY18pjsGeA1fVqZIU9J2grnyTIHdOzacckHhD5r59A2EgCDq0vv3p?= =?us-ascii?Q?/7XPeZcmnIHRrCjM2WZomImITKFtl7T8fmJPoUOGb0dJE3wXqOfIO+lU1ryZ?= =?us-ascii?Q?bXoCTL26zS492U+gpP9ajRQJJNBeJDH6fX2l2yNtAw6BrOxYVGmh5+T+RNOY?= =?us-ascii?Q?rQl5lPGZOSYG1/ohqXaQVaU8hk3pFRUDLV3kdAQEcoagRGIqSJWpliQagMSN?= =?us-ascii?Q?lE5amubg61W6I88A1aOGueuD/QC+1F8P6UI2OYhywJhiPqR7NZWGwd85lEDD?= =?us-ascii?Q?X5wL45SrSIIIEwO94597QLV6Y/YYE0NgXwohJyhLyBDmqMHc9FQkt9yxGD7A?= =?us-ascii?Q?21JFpb+claMQsSipkLbX93eGJ+MQ=3D?= X-Microsoft-Exchange-Diagnostics: 1; BY1PR0301MB1269; 5:3S/bS2gZnBQrRmA6TO+sgWEVFRtWv6RfdhBK+SjtWpoN2vq7ZJifrdcyKEvexqzGZ4W78xwzxT3PdnfbDqLzPeyyV2C+liGFAxActIsWSaz3PG0uCAhqYalzlj4Wh4u4OrkDZxiIgrf631eLPQQKyA==; 24:1dx18ryc+Een9zGAXO4d+fZp29Kf15kO66477qbJ8LaCTiwRts4HimAUARaSA1/UmuH9wCb15caMFOzxwT+QKTjDre7Vex/21wUkhJ2Mt9s=; 20:vPYc9MZetJZ31fyc7/dYBUksDtR22eOLWOOkVnjMQFpq0nfUc13fOD65EVC/Pqgss+yxRDn2vMWUrXxks9Y29w== X-OriginatorOrg: freescale.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Sep 2015 09:23:28.9745 (UTC) X-MS-Exchange-CrossTenant-Id: 710a03f5-10f6-4d38-9ff4-a80b81da590d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=710a03f5-10f6-4d38-9ff4-a80b81da590d; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0301MB1269 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-3.9 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_03_06, RCVD_IN_DNSWL_HI,T_RP_MATCHES_RCVD,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP For MSI interrupts to work for a pass-through devices we need to have mapping of msi-pages in iommu. Now on some platforms (like x86) does this msi-pages mapping happens magically and in these case they chooses an iova which they somehow know that it will never overlap with guest memory. But this magic iova selection may not be always true for all platform (like PowerPC and ARM64). Also on x86 platform, there is no problem as long as running a x86-guest on x86-host but there can be issues when running a non-x86 guest on x86 host or other userspace applications like (I think ODP/DPDK). As in these cases there can be chances that it overlaps with guest memory mapping. This patch add interface to iommu-map and iommu-unmap msi-pages at reserved iova chosen by userspace. Signed-off-by: Bharat Bhushan --- drivers/vfio/vfio.c | 52 +++++++++++++++++++ drivers/vfio/vfio_iommu_type1.c | 111 ++++++++++++++++++++++++++++++++++++++++ include/linux/vfio.h | 9 +++- 3 files changed, 171 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c index 2fb29df..a817d2d 100644 --- a/drivers/vfio/vfio.c +++ b/drivers/vfio/vfio.c @@ -605,6 +605,58 @@ static int vfio_iommu_group_notifier(struct notifier_block *nb, return NOTIFY_OK; } +int vfio_device_map_msi(struct vfio_device *device, uint64_t msi_addr, + uint32_t size, uint64_t *msi_iova) +{ + struct vfio_container *container = device->group->container; + struct vfio_iommu_driver *driver; + int ret; + + /* Validate address and size */ + if (!msi_addr || !size || !msi_iova) + return -EINVAL; + + down_read(&container->group_lock); + + driver = container->iommu_driver; + if (!driver || !driver->ops || !driver->ops->msi_map) { + up_read(&container->group_lock); + return -EINVAL; + } + + ret = driver->ops->msi_map(container->iommu_data, + msi_addr, size, msi_iova); + + up_read(&container->group_lock); + return ret; +} + +int vfio_device_unmap_msi(struct vfio_device *device, uint64_t msi_iova, + uint64_t size) +{ + struct vfio_container *container = device->group->container; + struct vfio_iommu_driver *driver; + int ret; + + /* Validate address and size */ + if (!msi_iova || !size) + return -EINVAL; + + down_read(&container->group_lock); + + driver = container->iommu_driver; + if (!driver || !driver->ops || !driver->ops->msi_unmap) { + up_read(&container->group_lock); + return -EINVAL; + } + + ret = driver->ops->msi_unmap(container->iommu_data, + msi_iova, size); + + up_read(&container->group_lock); + return ret; +} + /** * VFIO driver API */ diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 3315fb6..ab376c2 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -1003,12 +1003,34 @@ out_free: return ret; } +static void vfio_iommu_unmap_all_reserved_regions(struct vfio_iommu *iommu) +{ + struct vfio_resvd_region *region; + struct vfio_domain *d; + + list_for_each_entry(region, &iommu->reserved_iova_list, next) { + list_for_each_entry(d, &iommu->domain_list, next) { + if (!region->map_paddr) + continue; + + if (!iommu_iova_to_phys(d->domain, region->iova)) + continue; + + iommu_unmap(d->domain, region->iova, PAGE_SIZE); + region->map_paddr = 0; + cond_resched(); + } + } +} + static void vfio_iommu_unmap_unpin_all(struct vfio_iommu *iommu) { struct rb_node *node; while ((node = rb_first(&iommu->dma_list))) vfio_remove_dma(iommu, rb_entry(node, struct vfio_dma, node)); + + vfio_iommu_unmap_all_reserved_regions(iommu); } static void vfio_iommu_type1_detach_group(void *iommu_data, @@ -1048,6 +1070,93 @@ done: mutex_unlock(&iommu->lock); } +static int vfio_iommu_type1_msi_map(void *iommu_data, uint64_t msi_addr, + uint64_t size, uint64_t *msi_iova) +{ + struct vfio_iommu *iommu = iommu_data; + struct vfio_resvd_region *region; + int ret; + + mutex_lock(&iommu->lock); + + /* Do not try ceate iommu-mapping if msi reconfig not allowed */ + if (!iommu->allow_msi_reconfig) { + mutex_unlock(&iommu->lock); + return 0; + } + + /* Check if there is already region mapping the msi page */ + list_for_each_entry(region, &iommu->reserved_iova_list, next) { + if (region->map_paddr == msi_addr) { + *msi_iova = region->iova; + region->refcount++; + mutex_unlock(&iommu->lock); + return 0; + } + } + + /* Get a unmapped reserved region */ + list_for_each_entry(region, &iommu->reserved_iova_list, next) { + if (!region->map_paddr) + break; + } + + if (region == NULL) { + mutex_unlock(&iommu->lock); + return -ENODEV; + } + + ret = vfio_iommu_map(iommu, region->iova, msi_addr >> PAGE_SHIFT, + size >> PAGE_SHIFT, region->prot); + if (ret) { + mutex_unlock(&iommu->lock); + return ret; + } + + region->map_paddr = msi_addr; + *msi_iova = region->iova; + region->refcount++; + + mutex_unlock(&iommu->lock); + + return 0; +} + +static int vfio_iommu_type1_msi_unmap(void *iommu_data, uint64_t iova, + uint64_t size) +{ + struct vfio_iommu *iommu = iommu_data; + struct vfio_resvd_region *region; + struct vfio_domain *d; + + mutex_lock(&iommu->lock); + + /* find the region mapping the msi page */ + list_for_each_entry(region, &iommu->reserved_iova_list, next) + if (region->iova == iova) + break; + + if (region == NULL || region->refcount <= 0) { + mutex_unlock(&iommu->lock); + return -EINVAL; + } + + region->refcount--; + if (!region->refcount) { + list_for_each_entry(d, &iommu->domain_list, next) { + if (!iommu_iova_to_phys(d->domain, iova)) + continue; + + iommu_unmap(d->domain, iova, size); + cond_resched(); + } + } + region->map_paddr = 0; + + mutex_unlock(&iommu->lock); + return 0; +} + static void *vfio_iommu_type1_open(unsigned long arg) { struct vfio_iommu *iommu; @@ -1264,6 +1373,8 @@ static const struct vfio_iommu_driver_ops vfio_iommu_driver_ops_type1 = { .ioctl = vfio_iommu_type1_ioctl, .attach_group = vfio_iommu_type1_attach_group, .detach_group = vfio_iommu_type1_detach_group, + .msi_map = vfio_iommu_type1_msi_map, + .msi_unmap = vfio_iommu_type1_msi_unmap, }; static int __init vfio_iommu_type1_init(void) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index ddb4409..b91085d 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -52,6 +52,10 @@ extern void *vfio_del_group_dev(struct device *dev); extern struct vfio_device *vfio_device_get_from_dev(struct device *dev); extern void vfio_device_put(struct vfio_device *device); extern void *vfio_device_data(struct vfio_device *device); +extern int vfio_device_map_msi(struct vfio_device *device, uint64_t msi_addr, + uint32_t size, uint64_t *msi_iova); +int vfio_device_unmap_msi(struct vfio_device *device, uint64_t msi_iova, + uint64_t size); /** * struct vfio_iommu_driver_ops - VFIO IOMMU driver callbacks @@ -72,7 +76,10 @@ struct vfio_iommu_driver_ops { struct iommu_group *group); void (*detach_group)(void *iommu_data, struct iommu_group *group); - + int (*msi_map)(void *iommu_data, uint64_t msi_addr, + uint64_t size, uint64_t *msi_iova); + int (*msi_unmap)(void *iommu_data, uint64_t msi_iova, + uint64_t size); }; extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);