From patchwork Thu Mar 30 01:25:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13193334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C89BEC761A6 for ; Thu, 30 Mar 2023 01:26:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2D526B007B; Wed, 29 Mar 2023 21:26:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EDD096B007D; Wed, 29 Mar 2023 21:26:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7F2C6B007E; Wed, 29 Mar 2023 21:26:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C9E676B007B for ; Wed, 29 Mar 2023 21:26:02 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9AFAD1C64A3 for ; Thu, 30 Mar 2023 01:26:02 +0000 (UTC) X-FDA: 80623823364.24.3578B32 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2089.outbound.protection.outlook.com [40.107.237.89]) by imf26.hostedemail.com (Postfix) with ESMTP id 6686E140005 for ; Thu, 30 Mar 2023 01:25:58 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=kmqwDId5; spf=pass (imf26.hostedemail.com: domain of apopple@nvidia.com designates 40.107.237.89 as permitted sender) smtp.mailfrom=apopple@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680139560; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=AtXwYhy9cVcI/6nGPOMa1JOGakPLhFQCZ71DWmvEDvc=; b=iFof8CEDTyid5zFPOmGF2xLdeaoA7rbInO8P0GGWVdHi+GknvmbDVY43jUCukhd9dFrokT 3lC515SSfy6PoauVaV6BzjZwGE3C60QvMxJn7mBhTumZWkoOAKEEvaIFSo0Tl3/iK8c5gR rX4A3iUfgbPW6dFwLAuxKmmPuUrwdy0= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=kmqwDId5; spf=pass (imf26.hostedemail.com: domain of apopple@nvidia.com designates 40.107.237.89 as permitted sender) smtp.mailfrom=apopple@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1680139560; a=rsa-sha256; cv=pass; b=vkRtWcfsPrSr7FYxp4cckp/C7UJs1kTI1+99LvDqjwNlMrb7rRN1VJkyZx9IgfNasJrO6d GXV/kGsKVIulZPML/lIQ3FvgHbOH+5UOPpRKKlW0Yel1B1K2m6WW/Ek7zAtjYBDSRakZpJ nmQ9VUr2mTge17unTGcB+Jf8yD7Ct1c= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SaeHwtcJHuwb/5Qo0oDEIdrrBWKgUJj4l3QPYCuNEPpefKjyrVDI8V9UL6rGbXHUbJ9aWf7Izz8+80F4TL68Sg4pblrP7nrDXDftABFTIoNbKSIu/4ECnAjcZb8YnFmaWRmwdfBXbwAS9gyRqRszW8hbFtHPco/262GrZkzzCMzPRXB4QsrSRGbBthSASLHlUo826js6bHsgPp4xWlG58HZtDAP3ELqNDYb1bd7hPG+TShfNHMM0FtJSE5IUKRW5qTZ+JHlKmWVNzVHYWeM6RAEF2RMjjZYMJk/w7xzsAEtDXyuffe9u5NjGqfjid2hhMsqkOt35wtG5Bz76qGuRHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AtXwYhy9cVcI/6nGPOMa1JOGakPLhFQCZ71DWmvEDvc=; b=GwSwtbxc4zbZad79W9KAp/T+YkDGpFUxonsCu+ReXFgRy6C+yHO1AZcE/7/2i0VL/psOtaaqC80Xd3YgNkr1/k6PcqdG5Nb6t0ln50bNVunSpfQZ7l1Q4X8y+KOS/Bki4eAOVXb96dw/2uXKKr2eVqNXDaBU6PbCFHuNoA7fPhUHmAwjLMdzCwLDZKkaBHgM1sCpCG1L+tUjef//VQdPo96h7ZIi2Ro5yZIp/fvTmvJWCFub/W1P66nAmdcq5jbJT0jf3rP5bXVV1yM4qHGprbxGKx7VAO2rijlGGUffhjbIjz/lO20ZrEkQnrwfq6WfSwYxY4nSYdm72G2CAINHPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AtXwYhy9cVcI/6nGPOMa1JOGakPLhFQCZ71DWmvEDvc=; b=kmqwDId5doDe6FVh3/4QZ2GMM/hjtefquo3f5epB/ms38A7X0z5t2HK1AqjgWzGL+WxOUMlwauyTvHnGZnbBmWZN9QcehacnWvAVLsUceYmgHXihqoPdV6NppSz2GPHNmc9AU0WeHw9Fup7RGx5s92VSnR+31oldrEBlKNJIoUUeyJEAUOIHGhYAIQWy1FW6HkBxhUkjmxKAuYe9p5f66lkmWEIhiXE3T/m96TyPa0HdheeDS3bGgGaRjZ8Bd0mK7E+PSMnJl85W5WlHzA710OLIOyHvq14zUU6131mjdvvoPYzaa9DN7umqbbSz/BlgW3m0J1XnVWSrOsrp5F1I1w== Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by SN7PR12MB7882.namprd12.prod.outlook.com (2603:10b6:806:348::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6222.33; Thu, 30 Mar 2023 01:25:55 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::84b1:7c9:e9f0:bab5]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::84b1:7c9:e9f0:bab5%7]) with mapi id 15.20.6222.035; Thu, 30 Mar 2023 01:25:55 +0000 From: Alistair Popple To: linux-mm@kvack.org, Andrew Morton Cc: Ralph Campbell , John Hubbard , nouveau@lists.freedesktop.org, Matthew Wilcox , Alistair Popple , stable@vger.kernel.org Subject: [PATCH v2] mm: Take a page reference when removing device exclusive entries Date: Thu, 30 Mar 2023 12:25:19 +1100 Message-Id: <20230330012519.804116-1-apopple@nvidia.com> X-Mailer: git-send-email 2.39.2 X-ClientProxiedBy: SJ0P220CA0014.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:41b::22) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|SN7PR12MB7882:EE_ X-MS-Office365-Filtering-Correlation-Id: 21e5ff3a-758c-4715-68ad-08db30bdb57d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gYXqahKLeEs22oMrD9d44JSgf8sVQ0Ul2QsT+kb+yEMaYS726qtuBaVLELuWg3lpAy4zH3OnxtvQSR2wbsG9a74s5HZEhUFRW1omMw0aeVcnCr2bfAr1qolzr+xDPKpGK2XItMnyAJbtG+Tp91jnI807W5IcfdvWIBezs5Bj3rb6VOps126FXuPUOeAWvQEufE3tlX8iIDvMlxONUd9UgTG3dn/SWGZH2pETw03pImYOwvDNo7iRyBAXapHQ1xNanm1QNji+E6/wXf6GfeghSmLp7X7VLyPntzvcI7ZHqv6A0tcyLSqL8+9x5S2ZUe9SXVwjwXSTx2t8sNBMJNfpdlWgnwh4TGTo2JZV0KAwHj1tBtzJyeQyBv34nGx+drNxDUp8o5o/FbI2GYaybm0mS/o7NiedXvdZMOSqOe7iHoib1YLoq0vBrC/mQe1T2xeiTQ42N9Xa140vzRL5ZO5PkU6nG3urvesa6niCvj1i7qxDtAjzyVBHzSOQE1jgBQjm1EF6oJLrGtXlenB9KeC/XQxlK5mt8XVdsC/zfet/j1f2wI/GZg3123MtftFK07qI X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(366004)(39860400002)(346002)(396003)(376002)(451199021)(6916009)(4326008)(41300700001)(316002)(8676002)(478600001)(66556008)(66476007)(38100700002)(66946007)(54906003)(186003)(5660300002)(2616005)(86362001)(36756003)(6486002)(83380400001)(6666004)(2906002)(6512007)(6506007)(1076003)(8936002)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8zycUDWL6ptrbSW39Zj6hSHRNQdWwyI0gwrGQ/N9SrXDWUDVAr+VsqzbiQ/xuyGEBWg3GTKQFdj3HpxUgciDuAYTmButEGiZ3gQ+AJYAzB6Nay1LcmXjrCVgvUZov4FUIY2c7JU0Y4IFC9PNYVWnEFbZnHTQVbo9Lsmfuj3FsNJyD4S2twkXF7U0zlKJlQLG/lG+KqEoHLS812A94fdMh6yOfxYYnDp2ieAaP/n8z9Wz1FqIwonxcSVJ+3uu7y8/C8BrJHfIauT9IKRaio+c9rhQzZLZvMzJzQvgXh/ZW//2vFz4aJ+r07rSTfqG5/KeFwHkuOu6alWa16IQ55BO3Wws2/FuMiXlAWfEyEye+zjlR9tbigKgJWdAVP1VmbQW47t3EW2H4w96dkGPDPJrqGBGC1e1NaWgkI9F9LqXc4R8Oraw/Dd/h3uqUtFk8nhT11iSjUJXVY/uGJZnc5FOPqBIfTS6HbHNDJM3FXmtZE4EFzdMpnoFtNPj3yuVY0Kr0Ytfd9NUR4YVcAt53Omabxt4t4hGq7WcLXGSE9nE9shX1YA4gbX58rzY7E+fjF2Q0xCFsfHb4m8tW3dMFmcXg5g82RW6B+R093Sx+uc7Y9SxjhfnEb/d/okKiYTB+yvGNhXS0KGfXA0W2sL/S8Z5K2HPiWKAab46LARPzrx1qK0UCslSHZJGBKLW9tynMmTnbwAma8DVZ/qFtdoIkKZICpuMfwQY481v7fvkpj8lgdtzVRe+6AERRY+PJtDiXZOjxKGGBIoSAbz5NZionxXy8srdu2mIHbCncs8QcEJ9HOu25RoKqyKPYLYxeKdDlLoAiRpMOtWCkIulzHKRV126LuGXWLWhvvG05aMikHZwvzVcX6UIWkKb+T7OX3OP1h8wqa3EGjbqRwPxuJSeEwqdQoOJauFni2eZB05GBjlEiB/5hpg8xBv9QrCE+Fn+f+012szO9LU2jzLJA4+GTTO27yreAL+FXyyvzGs/9lNBwsOVVI00Zd9mKE6MMRosBTIAxnEsfi0kbRyx4dUxMf+u8cr1DVOo5NH6dlGx/yyKcJsaFbSh4IjE/BmXeGWF5FSfDj/tnAMsgKIuH/4broPzvf9E6LW7qZbelSBCP3MYV8nwcNlu00fqXkAPPUqerqThqaAw/jh2ALhiRHV5YW0xEbVcc2iynsy5uOzqgGYvlzsX2OMbOnU8Nh/Uym/20HUkwWDy8df5lCP/rNImikgwr8s6t8uEUN6c6VpR3ebRxZac4TC5pk/Y07i4fmpxFsVOEKu1uRyQy33CE8KW+kfLP8XjhMxB4M+QFR6b44qy+YSzCKHKDSfH8Ox1i610SLK/nlQz+Id585iYLPSMonmj9XxhlFwNExnri0MnPnGyLu/OlKG2FwYOVlyfJxioJP5tJpml5AqcbwG2s1uzoo0wsjK3ySKZBhFDi10kTHr5nYZxeoewT+Bw2DKZ4AJNXF4+Vl0oUe5lNKUTID6fMv4qkbgOkY94+NDmwrFhFD7rsZtOVSuz11hfjRHjaBtnUFPvp3+QPI9hn49HdzeubiH6vljlTy6KOF/Zbzt6yld0HHhwHLvLS3IRG9j4C9f1Q8Ee X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 21e5ff3a-758c-4715-68ad-08db30bdb57d X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Mar 2023 01:25:55.6382 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KMNA96RRZysJkFhOpZh8uscmsAqBf8wCPS+ESQU0CMoJNJSg5nVkBxBjElz5t60rtL8RZB3q9nbb2/LG8xz7aA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7882 X-Stat-Signature: 6fk95sioaz8tiywutkqwrjgsk1safk8t X-Rspam-User: X-Rspamd-Queue-Id: 6686E140005 X-Rspamd-Server: rspam06 X-HE-Tag: 1680139558-482113 X-HE-Meta: U2FsdGVkX1+xG9fiFq4aO7pDVPI0koqSAOYWzCqcfv5anJkE0/nm82/tdb7z3+k4HhXSRWcOW6YcYiZks/CGvay/z/AJ4vDS+miLfy7H2w+FhkMbQTLyGDpXld5FFC/SqZWXykaup9TTpLMeWwVW8maoFY4nGmKaCmWKq29JMTtzUpTf3fHxqCjvcKPAaPoRltmTKDvTfdLTVtPooEvtcErKGzV6n54H3Fn2w5k5IjkzbioEzezC1P3ok7eU/18zwyLcLr0Tvg+M/F8jh6DvSyyZpyYt4fkdtU+ce/pbJNpBrxla2FBZLjvc/BFXNzw/gbFRhf4v9m2tjv1iDRFSU8lhtsZF00kXCB+DAiSKyQuK87O8k6i++FqCV6KmEYsZhUUQdKsEVeWgJuNWnGwICuerJ5XJXQYmtt2G7Iflerh0MhwQTUwgwUzv5QU7Aqp+DLdXyjslutXTXP4umHNVuzhwn5N8FCCIgylFnYERXLsyXXEs8cOT0SikmMYxgN/GuoEPinwaRWbI0irGYxGg06NdiiJx5awe+9CNpbAdLPk+0DDtbh5wmiYML2VKwvy4HovYQxC6hspbrYEV0eFQyJ2ASQQktijXtyco+cIqfM1h4a9cDQThPYR2fzMpi7o/c4+wlhvedu0b4k6tvGJTuZoJ9H/3IM4yCCNDxZeGICWuaVKIcfI4K7Wr63xw+XakFBVDXu/0aCHecP9ywH5KnqT8H3cATnAAxwZCFggyFzgCSAQbplTS0l8iP/ELFAQGYGBMAUb9CslMq0k65j1go80kp6u3+ts0s3Z53Q6fYPpUFQlUP6Ms5N78xpGb/GquzmKeacsiKnPZ60il82xeBAz3mpPHBoRXbHwZ0uYhEX37D4Hu3N5AT5XsK9Ack9iWNv4TJoWH2VLh+6VnAc6mFQ1x6sTXFq98CVgThA7raU79DW91y3t6dnJYnmNYKLivho+YMzDdm76gnFqrSOe z9aP7zcW trYO513HJ8i8mR7MI4x2WrDa19fYMChH5tKb0fnLgwbH+eYFLw7Msc5qxYUBouenYToEb5hU7ul0FQsgKKEddPB+EVpLhORHelyxPcf6f96oIX8+OaDjdrbfMIEVVKZ7CUFzxkWqRFj3fICbTCB8t4tuqnKZPvNaxk3e/DShAE918CEjvVICly7ckiyfQD3Ttr9zw1/HhO9DwJKvLDjMnVqE9yOfBydj3vFzBVo76PlWL4OjptC+XkkKXTsc8CiGDPgxuFAUlduVNSLayEm0ju5hTG+P1JVlc2rjVsORvtaWgyWA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Device exclusive page table entries are used to prevent CPU access to a page whilst it is being accessed from a device. Typically this is used to implement atomic operations when the underlying bus does not support atomic access. When a CPU thread encounters a device exclusive entry it locks the page and restores the original entry after calling mmu notifiers to signal drivers that exclusive access is no longer available. The device exclusive entry holds a reference to the page making it safe to access the struct page whilst the entry is present. However the fault handling code does not hold the PTL when taking the page lock. This means if there are multiple threads faulting concurrently on the device exclusive entry one will remove the entry whilst others will wait on the page lock without holding a reference. This can lead to threads locking or waiting on a folio with a zero refcount. Whilst mmap_lock prevents the pages getting freed via munmap() they may still be freed by a migration. This leads to warnings such as PAGE_FLAGS_CHECK_AT_FREE due to the page being locked when the refcount drops to zero. Fix this by trying to take a reference on the folio before locking it. The code already checks the PTE under the PTL and aborts if the entry is no longer there. It is also possible the folio has been unmapped, freed and re-allocated allowing a reference to be taken on an unrelated folio. This case is also detected by the PTE check and the folio is unlocked without further changes. Signed-off-by: Alistair Popple Reviewed-by: Ralph Campbell Reviewed-by: John Hubbard Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Cc: stable@vger.kernel.org Acked-by: David Hildenbrand --- Changes for v2: - Rebased to Linus master - Reworded commit message - Switched to using folios (thanks Matthew!) - Added Reviewed-by's --- mm/memory.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index f456f3b5049c..01a23ad48a04 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3563,8 +3563,21 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct mmu_notifier_range range; - if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) + /* + * We need a reference to lock the folio because we don't hold + * the PTL so a racing thread can remove the device-exclusive + * entry and unmap it. If the folio is free the entry must + * have been removed already. If it happens to have already + * been re-allocated after being freed all we do is lock and + * unlock it. + */ + if (!folio_try_get(folio)) + return 0; + + if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) { + folio_put(folio); return VM_FAULT_RETRY; + } mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0, vma->vm_mm, vmf->address & PAGE_MASK, (vmf->address & PAGE_MASK) + PAGE_SIZE, NULL); @@ -3577,6 +3590,7 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); folio_unlock(folio); + folio_put(folio); mmu_notifier_invalidate_range_end(&range); return 0;