From patchwork Tue Oct 25 18:17:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13019686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7E2BFC38A2D for ; Tue, 25 Oct 2022 18:17:57 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DC5F810E42B; Tue, 25 Oct 2022 18:17:29 +0000 (UTC) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2056.outbound.protection.outlook.com [40.107.93.56]) by gabe.freedesktop.org (Postfix) with ESMTPS id 152A110E252; Tue, 25 Oct 2022 18:17:20 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f3woTcQRrNto8kajJOsPrxce+8jIO9L8g/kw4C3VGrpAHykIouEXZF/dVeJSgBn5rC6VjPlgYATP8QQTPC9L13Zxrfslo2bGE2mtnQ+GcZgSL/9mgMbxK99lVs7No7MOZMw4dbc1LjE2VIxU3Uyp+9AR8VdeGZ07NBoZ5podRVTV70IQLH/r4DWMtMRzT760j0UpU5P6sReqtTXUP/44CGKN4ObNlT3Cx7QW/GzNoqDIqheb1+F8CSbdX93GgkPwGhJu6d6cIVn7CDBPo5m0m+RwwvmLbvyVQaTkVBr0rIHlt1y6CR+ZrqrQG4BPMKjHelezdGqL/kOjL96Yw4Yggw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BTogf53A0OaVKvNg0Go21GP+KUpGF4/2xivK801AMJ8=; b=QTIw2uB50C62qqWUnSCdnxdhp68oM/ys/+H+9YkjmlGujPjMOIupC86npWgLVJh+JHaiOBVU3e1co96o/P07T5KLLIG2S7AjRbsr1FnXAiNt6I5zhMafiDKqxnkBSNm2u1/XUD9dCKOOSXTV0nofdn9rI769ko7GJ76kE7n5bh+cndBABlgqsX5x6jDLhLMZ0yC5y3PHpI4Ra1tqP/f6ymCUQ7KtfK4GUoQoo2I8LXiZNqDHiXp1zRCPvvs+N6WilrMZli9x9XQG7QBk5H8okSvvY4020IS0LITB0JYDs5tgcyAJ1Pt3rxh8AnVSZEZH+7RiDYoNOrcLymgcrrQDsA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BTogf53A0OaVKvNg0Go21GP+KUpGF4/2xivK801AMJ8=; b=my6GRdgfSreqmi78u7ObklLcNbKdbDLgPBNi6iYBGs0KPqpnNaxSgz+4ghonHQ1GoREaB7PKZuvQZEHi/4w4T8JMvc5ljg+VanfR24yCIVkSd9bH+Nxg+iM0QJi0XXuM1vSQGRjeWbDDzSKaLhwU2+S6zZJbYzIoifp9dKkX81980dF1GQxMYQsDoecxhjDkS/A1X7APjcprhw9lep07S+ylLhc3UHTrWRc4Q4R2KEb8BtwPQk+vKNeu4JXcbkNXkHr5oLi8G670Qj/iLhHe4hdEp9Nfu9/f/yRGeue3D7PT95xtycOZB8915NC4ojBJm3T0OFwiYsKI0uzAR0qQfQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by SN7PR12MB7201.namprd12.prod.outlook.com (2603:10b6:806:2a8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5746.28; Tue, 25 Oct 2022 18:17:18 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::7a81:a4e4:bb9c:d1de]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::7a81:a4e4:bb9c:d1de%6]) with mapi id 15.20.5746.021; Tue, 25 Oct 2022 18:17:18 +0000 From: Jason Gunthorpe To: Alexander Gordeev , David Airlie , Tony Krowiak , Alex Williamson , Christian Borntraeger , Cornelia Huck , Daniel Vetter , Diana Craciun , dri-devel@lists.freedesktop.org, Eric Auger , Eric Farman , Harald Freudenberger , Vasily Gorbik , Heiko Carstens , intel-gfx@lists.freedesktop.org, intel-gvt-dev@lists.freedesktop.org, iommu@lists.linux.dev, Jani Nikula , Jason Herne , Joonas Lahtinen , Joerg Roedel , Kevin Tian , kvm@vger.kernel.org, linux-s390@vger.kernel.org, Longfang Liu , Matthew Rosato , Peter Oberparleiter , Halil Pasic , Robin Murphy , Rodrigo Vivi , Shameer Kolothum , Sven Schnelle , Tvrtko Ursulin , Vineeth Vijayan , Will Deacon , Yishai Hadas , Zhenyu Wang , Zhi Wang Subject: [PATCH 02/10] vfio: Move vfio_device_assign_container() into vfio_device_first_open() Date: Tue, 25 Oct 2022 15:17:08 -0300 Message-Id: <2-v1-4991695894d8+211-vfio_iommufd_jgg@nvidia.com> In-Reply-To: <0-v1-4991695894d8+211-vfio_iommufd_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0399.namprd13.prod.outlook.com (2603:10b6:208:2c2::14) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|SN7PR12MB7201:EE_ X-MS-Office365-Filtering-Correlation-Id: cefe6abf-86f8-4666-8d02-08dab6b52657 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hNujF4XPYr2HW0n8Zywj3mN6QWMmZioSgDkkRXe3vRkfhKbmPVCKROG/kG1WqZs1uEVOd1ymronjPLvUTwxV6zp+oJLiG05MNnIPa3CnFlI6LQNMYAbDO73HTkf+0rDFQz0mCop2ZmNS878HB2PKZonBvR6303tBPEjvXyUaRkto54WzMSUpBDMaKBDTzqNELDup3sYh0PAYGu3GgB5A+2Ml6V8wwsosDP6eBtfbUpdC+LnNogobeUIHmbhpEW2G/6OyBOEo/M8K/NvXeCnNyavjzzcuxS91EFNU93KckTKqXdRE0VwWlU4eBrlWY+ymGTv3LoQQEMJn79yQD7jrkROXlFCTCi5c5srDUvw/FbdOaH6AV7bJZOUAHln5vobA2oSkvUCRsL8AfxxQm77zQyosV03pkNGUgCftSS647a69wni2tfSUf0bwFyCA/34OiN1pmdhmL2Rx5qJTK6v088LSS//bq/LrcqjjYq3BOlksXXHa2xLfAgVcVU9KgbhXiHwA5gKaI7o9+kwh71dyBGRUsyzYLRnX+GefL8fktUGW5cuC/zf3exj/B9F9K/8Zczd/1vvG7jGHt+uUWxUxGRTQkd7xXeByTTmE6iY+83Z4BcWacWjh+bQzZP7xKmoQOELEtUvKbj90aB3XYXkGVcz5P1cd04oHV4B+ni4AgmGP9YYRsLJN4WI70FSxY49jkAjbQC8aK3P7D3455xb3LbMtJHk55n8CtkVboOvlcT23Ue8lIQy33k9tdN+12k+WDSVdmy5ZaAXEIFUKJvqy9A== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:LV2PR12MB5869.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(346002)(366004)(39860400002)(376002)(396003)(136003)(451199015)(110136005)(8936002)(54906003)(41300700001)(6666004)(6506007)(921005)(86362001)(38100700002)(5660300002)(7406005)(7416002)(66476007)(66556008)(66946007)(4326008)(8676002)(316002)(2906002)(478600001)(36756003)(2616005)(6486002)(83380400001)(186003)(26005)(6512007)(4216001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: g2uP8hPmstq8J3/teevv1caadii/7W+FXkwGG8MvX2x0oGWHeLWyohrqQAJZkh8ZTd5UK9oNyB8UWDsfR/2cXAes7q1MfWq1r2lCeS1I+Z2KrQv6wNpEIw9pfcxt8eGX5zl8oKV5JHBo3ZS9uHN5XiqpYOIoRjvxZkp2YmistHlB0VGlDx9gl+b/z+nzIM8Xk+KPgngffpCvIIp8TmmHr9xNRMrfcMw3D9VEo782wXjGgbOFKbxMv8cPZdo/G2FvsrmHd0/amQ9DlxX9zhkWi6uY2mdIZWsXYCFK1ZuhYiRnREWKf20dJOXRZH0nI9ILCht19iBi5ZWC6ysGk3QT6dMdoQsqhFoeEugQj+tTBM3mN4einTnyo4YNnIEIunk9bTw9MDHbomBS9F/GdUKfZVCZDFM7aV825ZSF9R5WGZhrkxiJWttdu6mpyTC3B3vCBgpCT1AeElyE/kv5rxXMq2kxbewVAo2X22xoJR51HTpt2jlQb2noZW0XXAgYXus8KyynG+Lri++qoDGhaYlgxpP1qpipYXAa7WISAzXqB3GjxqXh5AoKZwYSQ9N3JwEcW9XSgKCdX/dwYpqCXkq8mb+nqr+Fk1IVTxel1GdnBwOqxu2Mr3h7G4k7P8OYDqvsYekfmfQR5fGyZIeBVhq6LEtk4TOhAttGWfJ+mmw3yWHonx2AYMeDT0jzD/osV4AckdNBjCEvVpEqOv2tK8ZGlpF/IEwEo6LhTb6BjrEioUYGrVhP+ydBAvaC12F6c3Si7upQ9QTuUGbHO9yz2ZNQBWl5AkQ/NAY9hcgrJBkjIPeC0jw28HdrrCSAX1h2i3sor6n8JE1Xl1c0s7SHIx1OMNxyrY6U3e70FXGTR0fLmUCbHaD5MVhEog2e0KS3uwaCYk6jj1tfgUVwjphOoLr5v5TWt0JizQnSCEX3TJAEKlKP4bR1MZ5971VtwVuLyGbAErndL0JZJyrchW2cuCoUeFxnLE292f8cQSn+/MHwJ4Mf673jaM4XkrnVyuxRKg844aWSnbzECOEmWAjR+EZ6+FVJD4mVNvpBSUIBhyWA9rV+lxgoaDRp7Dfob6tOEx6TzAbXRjrL/bvOBH4cUFgqCC9GI0/W8/F1yfYDoP5GqlJDxYXeR/QdniGgrYDEeQvBNRHToP23qFcq7DeN2oaBWnVcqwug8ZEYBsczkA9IoZFxODH6DL+MhETY0hWAmSuuLILCLeqVY19EZael1dcyHPWwyLJGgLTzFBTCqkF6IWPZKQb/Q2pgziUBNLAB9AtZWTZ2qpvxJ1pu/IAeuIP2Cg4MqTCfet8JU0qF89zPLfQUBeaLTgP+4mqLoUsdMAUJiN5Byr9HtjOxlCQDS/KXPnpWUO1S9eykfPYAm1ay3B5unIyxQkb+GnO6MJMuKNiTKP4kuw5/Ge0zMMQBy3j2k7CLdMjnHtnHKF6hQgjrxaZXGY4Z5AsXrJjGCtuN3E2cDG0HTMIolFD9si3bClo+AUUEmrN4bLAfQKGD6+ux9zIN/k2Pq1/9lqhNUa8zkflOd69DqcjCg/IdGpFz/F/+qzIRIOpSLuSjmX87tcmWsYc= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: cefe6abf-86f8-4666-8d02-08dab6b52657 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Oct 2022 18:17:17.5904 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: q+3B2HeyrqZXCoXAuZm8EW+kNMz1711MDvwNZVg5nS/+QBQAdNOC7HCTiqYRUfbu X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7201 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yi Liu , Nicolin Chen , Lu Baolu Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" The only thing this function does is assert the group has an assigned container and incrs refcounts. The overall model we have is that once a conatiner_users refcount is incremented it cannot be de-assigned from the group - vfio_group_ioctl_unset_container() will fail and the group FD cannot be closed. Thus we do not need to check this on evey device FD open, just the first. Reorganize the code so that only the first open and last close manages the container. Signed-off-by: Jason Gunthorpe Reviewed-by: Kevin Tian Reviewed-by: Yi Liu --- drivers/vfio/container.c | 4 ++-- drivers/vfio/vfio_main.c | 18 ++++++++---------- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/drivers/vfio/container.c b/drivers/vfio/container.c index d74164abbf401d..dd79a66ec62cad 100644 --- a/drivers/vfio/container.c +++ b/drivers/vfio/container.c @@ -531,11 +531,11 @@ int vfio_device_assign_container(struct vfio_device *device) void vfio_device_unassign_container(struct vfio_device *device) { - mutex_lock(&device->group->group_lock); + lockdep_assert_held_write(&device->group->group_lock); + WARN_ON(device->group->container_users <= 1); device->group->container_users--; fput(device->group->opened_file); - mutex_unlock(&device->group->group_lock); } /* diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index d043383fc3ba2b..204443ba3b3cd9 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -749,16 +749,22 @@ static int vfio_device_first_open(struct vfio_device *device) * it during close_device. */ mutex_lock(&device->group->group_lock); + ret = vfio_device_assign_container(device); + if (ret) + goto err_module_put; + device->kvm = device->group->kvm; if (device->ops->open_device) { ret = device->ops->open_device(device); if (ret) - goto err_module_put; + goto err_container; } vfio_device_container_register(device); mutex_unlock(&device->group->group_lock); return 0; +err_container: + vfio_device_unassign_container(device); err_module_put: device->kvm = NULL; mutex_unlock(&device->group->group_lock); @@ -775,6 +781,7 @@ static void vfio_device_last_close(struct vfio_device *device) if (device->ops->close_device) device->ops->close_device(device); device->kvm = NULL; + vfio_device_unassign_container(device); mutex_unlock(&device->group->group_lock); module_put(device->dev->driver->owner); } @@ -784,12 +791,6 @@ static struct file *vfio_device_open(struct vfio_device *device) struct file *filep; int ret; - mutex_lock(&device->group->group_lock); - ret = vfio_device_assign_container(device); - mutex_unlock(&device->group->group_lock); - if (ret) - return ERR_PTR(ret); - mutex_lock(&device->dev_set->lock); device->open_count++; if (device->open_count == 1) { @@ -833,7 +834,6 @@ static struct file *vfio_device_open(struct vfio_device *device) err_unassign_container: device->open_count--; mutex_unlock(&device->dev_set->lock); - vfio_device_unassign_container(device); return ERR_PTR(ret); } @@ -1040,8 +1040,6 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep) device->open_count--; mutex_unlock(&device->dev_set->lock); - vfio_device_unassign_container(device); - vfio_device_put_registration(device); return 0;