From patchwork Tue Mar 28 02:14:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13190304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7EFCC77B60 for ; Tue, 28 Mar 2023 02:14:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 442786B0072; Mon, 27 Mar 2023 22:14:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F1866B0074; Mon, 27 Mar 2023 22:14:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2925A900002; Mon, 27 Mar 2023 22:14:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1B6C46B0072 for ; Mon, 27 Mar 2023 22:14:50 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E34E51C6108 for ; Tue, 28 Mar 2023 02:14:49 +0000 (UTC) X-FDA: 80616688698.02.ACDFD50 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2057.outbound.protection.outlook.com [40.107.243.57]) by imf14.hostedemail.com (Postfix) with ESMTP id 251F910000D for ; Tue, 28 Mar 2023 02:14:46 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=aCZVxjju; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf14.hostedemail.com: domain of apopple@nvidia.com designates 40.107.243.57 as permitted sender) smtp.mailfrom=apopple@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679969687; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=PB/pMG0Ct6b5YIoLZCDIjS6jkfqnJOjUc51ozxc71TE=; b=04BWebQdwSfQ3pkEB/kHY766cUXb9kxy4UDAiEcZlU8Qm6yCQihGJFaU28UW9H7M7y0jJY VyzM17c3zngjRY4oHrYTkCIrtWjunktXnatkAVAdXQIVjXVOvlxZ6J7zflFXxS2YWfdhAe bR48jqh0MqwigQSIf7MLNXIKtYqBUYg= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=aCZVxjju; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf14.hostedemail.com: domain of apopple@nvidia.com designates 40.107.243.57 as permitted sender) smtp.mailfrom=apopple@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1679969687; a=rsa-sha256; cv=pass; b=btff6ErZ6LDbMnj6XfnzlQ+Njw7ChuL2l2uubNkf/Osgd/DeqMRWsMjIcTItaH5k0KG6M2 LKBzqYo/wrLzdDpKomH1e4HZFPtpN8WxJF11foWBSshyQXTIVQ3A/5SUZWHI8F+rG0djo9 vX7T/Pic4mkbzxRBYzFFxlk2Zif5Pag= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K2gpUF00smSnjDM/fWptovg7mtNnegIEqbjhxhf91Hps2RbMwc2A1h44leW8ywGysrADeOemw2I2f3NyBU7t21Losex1vPoubE/HlIfNlvMnUMQFXe6+lpww/b30pjN+/e2Il6GEMQ4+xDByscKfnh8SdwivAz5S8ECbgt712yKCaJZbglZ60LNJvN0dAg9vXIlgZCAg1QrNK7sSHm/X6Ts8lhrilRz8p00U3+OwXcFnqIsu+9/pRikXt55ogvuJv4iWj3DtWaa9iIrITgXfUSmdzRRI7s1wzsPsh7LmKxcGk324oLDfHpt29P5dX0bCON6aHDd8DjIze5jWcGSnBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PB/pMG0Ct6b5YIoLZCDIjS6jkfqnJOjUc51ozxc71TE=; b=U1PJE0orhN/Ucpg0wU9Xk3SSOuK1bRpRIoGHuWFpHFElx2SK1Mz5HT5wHR6Bwi2rjO/n5rRdYBH1296MuGTBBUMJUpoGnJ+rMpU9VZZgSQKcdbjZluvcEFgtxdgDjawzzaCaLfcEnG73T4N+Dmat63boNJErCivXgo4K+ZV8UslBNimUOOCHZrX2rxKR6byPyH73Hxj+3F8PB9/I6RhzEF2iRfE/0etjowFRM288pZnizV1bWfSWLRjmPSaPRN/6tADyCzyjtE40R4Q99AiAuCioZWhrJ6YHXtpOqCluNijCIijitXCcZMlUhmBGk1BI+2XVsXoZlHEI+gdQyYXB1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PB/pMG0Ct6b5YIoLZCDIjS6jkfqnJOjUc51ozxc71TE=; b=aCZVxjjuyhqvHjGVyqBcIhlkLgsuFOkpEf/srUKUUVFdTcCQxg6g9ZjfOlb8AhsqWjtw1D1Rut6abrqwdHVdeOcFej4+ZYDaGUBsXnDsKXAuH6+bIUSLeMDMjBMrU9SwyuOWwA2HfXI5WCCbCebJjSIc7ha9ZScL67HBL/mm7aRmb49V6XuLglTPh05AEzvH/bYv6gplqQ3w1E6aAXcQXT1+3J7zFY43swGt6npmaIHCLY4NgJw1pxH8z5ots6BW6qy8QQgBdxdE59wP8+vfrlfuzli0wW4zxfseuaFEor8i8nzEBXDS9jlOxnjd6VcJqq6UxknigfG+PWG2TAWyjQ== Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by SA1PR12MB8643.namprd12.prod.outlook.com (2603:10b6:806:387::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.37; Tue, 28 Mar 2023 02:14:43 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::84b1:7c9:e9f0:bab5]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::84b1:7c9:e9f0:bab5%6]) with mapi id 15.20.6222.028; Tue, 28 Mar 2023 02:14:43 +0000 From: Alistair Popple To: linux-mm@kvack.org, Andrew Morton Cc: Ralph Campbell , John Hubbard , nouveau@lists.freedesktop.org, Alistair Popple , stable@vger.kernel.org Subject: [PATCH] mm: Take a page reference when removing device exclusive entries Date: Tue, 28 Mar 2023 13:14:34 +1100 Message-Id: <20230328021434.292971-1-apopple@nvidia.com> X-Mailer: git-send-email 2.39.2 X-ClientProxiedBy: SJ0PR05CA0086.namprd05.prod.outlook.com (2603:10b6:a03:332::31) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|SA1PR12MB8643:EE_ X-MS-Office365-Filtering-Correlation-Id: 9cff133f-1cf9-4fba-549d-08db2f3231a1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TK1SUWJWh+VfBH1GnEbMpJF5e24im/o4WfJZGKitARIQSCnukLabNmqVpEpzn9hmA6sQe8s7l8slsSpQoIuPi36ZB4zYAjAqQRC4/k9hHeViKBIA+PjMNF/s2d3Kab1mL+7UHRv+efeBMHyFHbgtf2lN5a56EmX6W1e3W/cZVzx0z0M2ds9POkrSK4HOpZ/BxQFsAB9jddBKsiQzJhJ23fp7TByeugWOMMVgacaIMRjEqJjVsWJSZS1PZlMck30fGR0wQchwhAscjHESU+7YWj/cpjurc6rg1qTEg9yhxGrVZDZ2hGwthLjzeLjUMvwJqbzaDZtGmJFkOBYwpB77ed8aUmEVNz/sC+mQ7T/pZulgXkE1Ek6IrIUEB7FlPWkjyjUAN8cyqQf7v56Puu52GNizTrhMZ6mmOtAXyffqk2K5mKYR5tat5/ptrLJ4nxCe440wbHpGaN5SEUhd1GMBXo8Q4txIttZFVjje/4GTTvi485oqOidwyOXSaA+yGnD6zBlo2BVWuc3G+tNpwexZehelnFKGNjOFFz5UVbPeqgfkNbo00dIvmb2VzHuRKLuX X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(376002)(39860400002)(136003)(346002)(396003)(451199021)(26005)(1076003)(83380400001)(6512007)(8936002)(6506007)(186003)(36756003)(6486002)(6916009)(8676002)(86362001)(478600001)(66946007)(66556008)(2906002)(66476007)(41300700001)(316002)(2616005)(38100700002)(4326008)(54906003)(6666004)(5660300002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: lkRF8mmdPwyRERj5KZSiZzP/yP9TSUlEnC5bksaesQbaqcMeUL40wr7ASCqFUD5oSPB7r3GACx5b+meWynIHjVxSqrBVvmYPED14vN7yz9b+2Lb1FpDc2cou98Aep3TVpTpykPb+pYp8MEDIvNsNgd6HQMOHClYO75M+yKQWxcTHxxHIDBCUkAGefm/dh87vjyJ5/+3rsS8SqtZC4YO5FVXl8oSb2S6thWVjTW1Q9mWu4I/WL5iwfK7C3Rm9/7QBOBT50QIbkqJ8neOptJknLXyNQvsd3wGQT0aLn6QOPdFI4DX5W7Jw/aD9319NwRM7Pn2ILLu7H4JA84ywOUxdeD9Qb41d4sAZMTX9rayCmSe9DQ5ICMV4vpsT7wiUDK+PovgteXQGEx0DXX44uLpWFL8k1qucp/MDOIilOE/w7ItkBdUdkxpJPqPLZikJ1+nAK8B3KQ7Y0+FY1emm5K49fUwFWwpIa9jK+5SwhR1DWQao4czdDKh2E9IJRkiV1nRGofgjGu0ZOoYcBz5jp88nUadMtX93Sz5AHYNK/VsDeVRX0NDzSnMxtyOw5mVAL6yysL7aBsvqmiNkIdXVtDB1ERfmbhLXEMPe9677qQ86GX+GKG+4o9BB5YfP9tJ/dFktau6DbTQPZnqmt6YfAT4qtDMObqrnRrkx6nU5Y3X+QJjxZnBqUa+WAmcVK3hqU0fpCMa78t5L1iDkLIYsRHtvB0Bwl3gButsjBQZJ4TWQt4OEKh9q0/aUOF/tIuSZTXG8K4nmfC+eFFHv9hVrZvaXS3Z5QVkNMpBt8paFd5VQUPvQYRnH17SYDGwRl7wtoQeWVnufRpbbdTwxlTjM8PuJO+jZ9FaJMaBRT7/x2QaYGqG103bz2OMeYuCCSJJiMzvrdp4nY6rWaVLBOMR9S+danYzWOWMEWGyYb3rvOciFEvl1IdjejFDf6Lk+gV/3ie4rlnHQjSG3haekm02fTxoko5D3TgVEIZrjOIwydXgvxAQcOaag0qcIENDYs2+uk41NpwTO4OMiVYBIg9VYNGPirjENM4O+xgdRdpuM9bTkNjJSKxZKHjGHSLeFsN5EO+Vvv9JK/6hJDjCRtCtd0rJIfZchW0tqDTbO1+oo+OuhU8gzO8hEyvxsr7c2f2pxixk85kSCXdLeROhxQxdstXgGzNn7H7ZVzLkj2RTdFVmfYrzxy0qNwhG4wuLFDwy4dVaBM8/Qy+FnMy1WGAwWxJ7S9zbmVIx4gXGd0CE+ZXe77oyVTCppvLRefIO8Dd2cj3vjlmOkp+tusnFZmnoGof1bML+lXsE3JeF3pzOUoTkFdy4H+wj/JTIxCZSbsXaSPyhmn8Qa/70irtSG9wS2brblasX8NKAqQh0jGEziiyOEYC4c5PaupFLqxnQiDfNvT6o96eaeshcOdfGT9/713zqoMxbkKEQ1HvBXY1JF4EdbwRW5V5Zc26lWjOcEm6WMBmv3Mlo+DTm0qG+EqscpozLi7NCf/Z4Dk7Hxl9a0LiSoOfppnV7E1k2vpgxyXKnk3HRvVptpG4sH95+/AmZADuLerL4olRAj+ao6B3ckkU5KKzv4IRI4Ar3B7NCFG7CDxylp X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9cff133f-1cf9-4fba-549d-08db2f3231a1 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Mar 2023 02:14:43.2554 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: oObOjz5MeMe18XyqjRKDaPTHP6YNwHGUAQPh4rlVZwkzYfkmJI0iQOfUBK6NK3cXdc+jfajlOoKcPSvOULtbrg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB8643 X-Rspamd-Queue-Id: 251F910000D X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: jirddxn1zy8tx61ojcednx56mxizgcxd X-HE-Tag: 1679969686-992488 X-HE-Meta: U2FsdGVkX19aTYoQeX9PcwOcl2JIMrZS30lfHyk9RvkBotyspooRHwD2n8SliZjEAuJ/nxe11MlfVT0XaVNBy1lzOG9rjC6W+Rdt4JQbDY8tqyB0RU5tM3bNXsUk0RYZIgMQvx56YAxiPaArLfga7OGWR9VHhX9yyzuA+cL1v0CjsHw6B7Y+XE0QqkiVpsYHqicIjJm6GY9lCocZZrtXrXp7r3OXV8Lb9L38+33Af1GvVC/7NN/yqQqGpic6Od+o/Cs3465F9lhL29U6fGOg4RH0uKPoKXuvLOSmXMZlOOOCGc97EBx5+I14CnasgDvNzo7dZQOFRoef959MBmhUZOXeJkDkWEJMi5J+TxuYKYYcbdCXSvkowbjyjAm/s5n+mnZf3UpngpGn92N2Jq3/qGyHhSmdUM8PJWKU7ucuiSbv+B2vaGIWXxQBwReG3Uw1UsQUgqH5L2GEh7j9JkyX9kxTGxDqNYy64qPK0FQQ5NF9cDvedon4F1cx1VZEEaKIxc4dWRX/mDG5fyAoDS+YfZXHtZThmymz9uup5mgZPK2QmqQseb9Bx84riKgwfr82uTkvdSFITdRA38Z/7zRiV39TU7h438EACPIRRLszFNGVXxDFOoO6o+wpKNIqosQVQ3tD0KdmlnVZG2T7/bBSqF9cRKPfupSEGHlGkqMHtn1jdrwm2h98N/w6lAYlZlRc9FEILF1p4HKzeEfL/wbNg+YkKHj6viPooerMxb6pHE6UOAFGNo3gZLeEEp0JjrKhmaurPt4RS1IZGKT80vhCe2fl3PSEdvQrA955l574ZnjlzVr8pKjyI4PUpJIYabOUMDFHvwGV1C1qULgpxVboG9yGacHP3dfwEgdKe+M5JegSLQQfqjAo9+x96Y+VB3eQLhbxaAsBtjepeFyfa5IQumptI/QTpraF2UPaZkJQy7arcyilNv8Oehi0zfMe338KD4n8ffQ/FE1hToJ8Rlg 8NZo9GdI A0mdOUMptALpF/1SdCWGfvPbMqIHu2PxYaHXaXV4CyY056CDz7CQwvzPefJnxJMnl0xhIZKuGXqccsks0O014q1Imw/HkIbRLkTPSW0UebRzhtLOU5xjkvvvnSAcolm3FR+rSo3Yk0llxYfCQRWLBYKL13WG0B5X8/FwRYK03cGjQDcfqBFKMrLuFvNS+oRSj/DvNH5BK7zbP5bfFfnl/zf8A8OLv4HKPwr/HToBjLWjZpKmj1hdl3Zyj1odAR28Xig3FkfwIDcyxlH1avi5yBYYhDrRrVkrG8+YUKq2AC/+RxBU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Device exclusive page table entries are used to prevent CPU access to a page whilst it is being accessed from a device. Typically this is used to implement atomic operations when the underlying bus does not support atomic access. When a CPU thread encounters a device exclusive entry it locks the page and restores the original entry after calling mmu notifiers to signal drivers that exclusive access is no longer available. The device exclusive entry holds a reference to the page making it safe to access the struct page whilst the entry is present. However the fault handling code does not hold the PTL when taking the page lock. This means if there are multiple threads faulting concurrently on the device exclusive entry one will remove the entry whilst others will wait on the page lock without holding a reference. This can lead to threads locking or waiting on a page with a zero refcount. Whilst mmap_lock prevents the pages getting freed via munmap() they may still be freed by a migration. This leads to warnings such as PAGE_FLAGS_CHECK_AT_FREE due to the page being locked when the refcount drops to zero. Note that during removal of the device exclusive entry the PTE is currently re-checked under the PTL so no futher bad page accesses occur once it is locked. Signed-off-by: Alistair Popple Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Cc: stable@vger.kernel.org Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell --- mm/memory.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 8c8420934d60..b499bd283d8e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3623,8 +3623,19 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct mmu_notifier_range range; - if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) + /* + * We need a page reference to lock the page because we don't + * hold the PTL so a racing thread can remove the + * device-exclusive entry and unmap the page. If the page is + * free the entry must have been removed already. + */ + if (!get_page_unless_zero(vmf->page)) + return 0; + + if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) { + put_page(vmf->page); return VM_FAULT_RETRY; + } mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0, vma, vma->vm_mm, vmf->address & PAGE_MASK, (vmf->address & PAGE_MASK) + PAGE_SIZE, NULL); @@ -3637,6 +3648,7 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); folio_unlock(folio); + put_page(vmf->page); mmu_notifier_invalidate_range_end(&range); return 0;