From patchwork Thu Apr 14 03:18:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 12812867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E689C433EF for ; Thu, 14 Apr 2022 03:18:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C87E76B0071; Wed, 13 Apr 2022 23:18:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C36526B0073; Wed, 13 Apr 2022 23:18:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A89A36B0074; Wed, 13 Apr 2022 23:18:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 96F346B0071 for ; Wed, 13 Apr 2022 23:18:23 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 4AC4A60160 for ; Thu, 14 Apr 2022 03:18:23 +0000 (UTC) X-FDA: 79354026486.18.BCEDF7C Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2062.outbound.protection.outlook.com [40.107.243.62]) by imf25.hostedemail.com (Postfix) with ESMTP id B8A4DA0002 for ; Thu, 14 Apr 2022 03:18:22 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dhbQFO+xiwnMcMxbX5F2a0chmYUcTjHRP8r0r9OoAK8mzYIlWA9DPvBNk2S5xvJgj378o+oPnz7HoVJM73rFWN+Unu/Q4KMFylgESVkWot7N4H88h6LV9Ni57y+LKOAH6dRLPVpzJaTLHK7exF32OCRSHo9T9rfwzCaGjsCCn4Xl7hgHvF0MF9lXYES3xfKaVCxUp/8Hio1HDtX3v4ErzqNTmDu7OCosyXCo3GS5o/O4vUrQ/EHvvOEV72kNnJ1GOzhQe5iV6gj9j3L/+Q0e0mS05/oq35ejpJA1eo0O0ma+m50q4GtSYeRTl6h/el6nWjT+ejCItiNgjAvaclKvtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ykyLcbCpUmTRazmxshBm2wVohiYaxcQog2jO/6aM0sQ=; b=N3zpEJlth/8c+CWcQGnyqKuWt9Rp1s6icoMAlEv+Gdx8yb2vfeq0woTceVKMp1O6329K27JnJf04PWnGJzCTc8vLqRGsTK+z9uh6qB1P9+dXtTZwIdkfkjvc4HGW0FLJlQQU0d+zafo6f3BGt7Ely5QgA0IgwfMCghe8nrCETSRfbAEotOJHypgWrlKXciv0e7qDHlEMeAbSoZbRqB4sye8R8/EqS5xu0P3EE3en+qkCaamizbvFYprhRWj/eVN/W6TmEgzUaXKEXFt6JHW5U1qRcVloWzqUbJjVIz5KWrS1i7JqOKu5Rr7Msm02zAWFdKJMvTUCSYD9agQ6IIejrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ykyLcbCpUmTRazmxshBm2wVohiYaxcQog2jO/6aM0sQ=; b=soyxPMpOMNrKmJG35LJkm/tKOYMDYCoyEmJNt0etxZ75/v/cBWDQW0Gd+jDnBQpSO97nmh6ZWe3LTHWjHr5GJ6terxo+gmyuPzjRmVq03FOp2T+C6MNs0dkPJ8qnOJWT8XmypHV479q6AMg/I4PyUpnfwSbLin/+Fn+9f7Xq6L7Sp9y+kW21FsBK8IqRC+JpVj7HfRn10EOsBt5WTfld4vK+Yy4Y0X373rcp6fCpsjihI9X5854hseg19gntUsG/uP9PMmOEiStalcP+iwPACjzX/zDpntMC46WD/g2Kx5Wv/f3GZCfRawbOcEQvbegjp1cPdaGd34QkR1jXA6/9cA== Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by BY5PR12MB4902.namprd12.prod.outlook.com (2603:10b6:a03:1dd::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5144.29; Thu, 14 Apr 2022 03:18:20 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::2d17:b68a:e101:4c18]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::2d17:b68a:e101:4c18%6]) with mapi id 15.20.5144.030; Thu, 14 Apr 2022 03:18:20 +0000 From: Alistair Popple To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, christian.koenig@amd.com, jhubbard@nvidia.com, jgg@ziepe.ca, rcampbell@nvidia.com, Alistair Popple Subject: [PATCH] mm/mmu_notifier.c: Fix race in mmu_interval_notifier_remove() Date: Thu, 14 Apr 2022 13:18:10 +1000 Message-Id: <20220414031810.1787209-1-apopple@nvidia.com> X-Mailer: git-send-email 2.34.1 X-ClientProxiedBy: SJ0PR13CA0091.namprd13.prod.outlook.com (2603:10b6:a03:2c5::6) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: dc7bb581-3069-4ca5-c8ca-08da1dc56d00 X-MS-TrafficTypeDiagnostic: BY5PR12MB4902:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8O+SL+N2AFLf+R2wxcCjIcdp7Tzb4KTuqs0UdAL4S9K6tX+WdWHApxFpzq2jOfmCsubQetQuJShmDxXXUsZeoVrJxVnogJXjY+/bRqzJJQ8RMpESq2Lc/cJUBjnM2ZyWci2Qm2nuscW/NSlSoKEHw6GZq5FG4HPHs5RkrGS5VjAbxjmJ4+UZ+488RpJnr5WTxVMz57f1km4i8FLSR58KRbJUjOfHe1nZFU7xx2TUOR3bqOVQn17k/ffiH8LnswZ+R7XFnpUJO6CJRi/t3k+LZ1dzsJPwXGjSu9Y6sHd/kBcQcOKoTcoQz0zioFLQ8rzQDHdAAWnUbRekNuFVy03MMwmRMyFx7oRenlQs20aTsuKpNH6wpKgSf5Rd20Zmw3dgcrgblfR/x2VmIGYe3LRCE/i9sOeLdVHKq+tAbz4H0pru3TbAcGzYwGkdWnwS6BZUkkgtxl9h+c14fZeoI/bxIxKwlcvzQQ6Mp8/5Fv9tv+pORl6IeSGOpX/V4uTIBX03hyRanVpvEjAtuHtsAPOiEQ/f/NTKg0LlV5fTwpgreudl/v17ihrxsRcSaf9E7y0o24+Ei0H0dV1eEop937tOm1VhjQHzhQu/bqmlYuxwKp01h3S3UOoCEqtClH0motNiUG9f86eWpKB84OZp+Xc1KA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(6666004)(508600001)(5660300002)(107886003)(6512007)(316002)(8936002)(2906002)(1076003)(38100700002)(36756003)(66946007)(6916009)(6506007)(2616005)(26005)(186003)(66476007)(4326008)(83380400001)(86362001)(8676002)(6486002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: F2yk78zA9bhAntaS1d5tFrG61KoiEt/KhSoz8Xb1H2GNo4qbC3SyHJpbOEIpM7ynxsG7rHFW1WyZfV6Y7/AKMNbhcKQf4tJ67KcHRcYVQeNb5JYUr+uk3+cL1wrXXy4jPJMMeBJ5qkh1Wwe71MX71d3uEJKtmPtwkoAv9lfOBBWbyg6yOSO5vtRyv3TzaQvOj5QObroCSr61SMaRGqYXkq5chChFEzX+6guja8TSot4TB6xcg+kNv8zyDTnF0GqdyyGPE36qmenpuGrsdmfDiU8C/s+5NAgEo2ji6hOpNdD98X080JQxzY5ClZ/hOSnCx3IeAdLl5p4qUX62On6HPVrT2K8vCxfcYfQz+bg07VRXhmF3e7gq0gcV5lA7uDgJoqMGv+wpJUVhVpJrxT697HDmZ6y31Dlou5jHSbOXblMr2uUIrQWmupMTB9ZyA217mVu5kG954vPn5N44sThtl/8P1wYqIXlQzByqb2j5N2BVy1eTAivngZ4XjvRSOykvG9OTTnYjLnqKPCqYAGbx+w4D5oecQzlRv77nNki7vhoHNPpApmE1gx21rIs5aCvPwyA7Z2dRIbgY2xuNRIm+cK/u3xSWUoOqsdihgyVd8d4zK2FzMc4Wa8U6C+rRV8Di4dXZPSZj7BFRtt/Lvkb4dILEO6JzQ+vJb54gKuKX8CaferNtT0dGEbM1WFGp540RgxEiMftJo1OFFp1qLk0Osa7LSz5n1dAGdO0EgavVogcrIyV5WWkdnWMHidm199v9u9mBH074TjO1xoKEbpCEiOV4doRPHT9D0EjulFqmChTzuTFPW7Ag1dKWx24hWqlqf0BO9C567rvLnrsRJEvIGactvjXmtauw+tp80WFz3TnwruYFPt1oUCxmI0xJSL97FfOdAcGaYOfB1yKVb2ZccWX6gcv5cYx43oskSq/nrl37QE4SY5sHmSSHiJKj+p/sQYulGW3qvTFVUOAMHpYpFn6Qp09YVwZg6unjmlvsBeTYACUDqfggcAFGV6ylvGyolM1vtdgAJ/mWOgGWGfSHvbykd7kxjA9DC60BkV6XsHkOVEDcgnlCZWGLHa3vcTRiJ66aLX6mWKebRP6/3Bq6FmK+9RoGgI7MSMG59NkDqLYTavrnSagJLy8qC6s5z2bru822xc67OEW0rkXolxQSdIYbBGjKdPJq06VDsIx/Ojd95yoTnR6dDMmszMR6LzirWnDkNhGdG2TFfrZNJ5Xk3n+w5fI3hmBdbIg55THU9WhFOJetq/GGBiGHZH1reRR6XUKpZYlitByHRPjWuyjGx5f5gzRP2aEdNUAQbct0JzdZv37DW0B8ZZoga/4aO7pjZKyROLnTmAO4rEvri0E2Zr/5DinOYalvriErJ305C1/Wfo04k4FxRkTEhriiuHUCGNvUEDiSLFCNUFmyN04L86TjcsFkgMLFGAqVuVs3TkopVkLlwizTUOeADVfINgY5QAyekjU6NX1bvQtSbVQ3MCkf+J3+s0d0kR/M9pDaLSyIIt15mESzU4Ur0inwWo+8xVOEYyh9tcbz67F1odrqS9VMJWk+a8UdcJJmRW+Hwj3GfJbCIhiZwHZvsvHV/IYjWXqPg5lCUYb6Kan2nBBBsufDlBplkUjfI2ucbh+kMLf2+V57RnzKm23x48lAyWu9JH6OB2lyyDnUUKCnDMbEMRTveCyKCtnE7pDhVRj+iQJRAI5teuaKj6LOcn26z4/o3TimDHr6AtS/2thmz+U2mw== X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: dc7bb581-3069-4ca5-c8ca-08da1dc56d00 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2022 03:18:20.2752 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ssWUPu3wEAozm5cjyvDyrYvaApQVWJsEBA+nfbRfkm/JfEs4elsy+8stvSkf+TWApZKAxD2AmHx4ZUMDpjUcOw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4902 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=soyxPMpO; spf=none (imf25.hostedemail.com: domain of apopple@nvidia.com has no SPF policy when checking 40.107.243.62) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com X-Stat-Signature: f63ouebi7eyknyrxoqwcyqqd1fjtzqa9 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B8A4DA0002 X-HE-Tag: 1649906302-115973 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In some cases it is possible for mmu_interval_notifier_remove() to race with mn_tree_inv_end() allowing it to return while the notifier data structure is still in use. Consider the following sequence: CPU0 - mn_tree_inv_end() CPU1 - mmu_interval_notifier_remove() ----------------------------------- ------------------------------------ spin_lock(subscriptions->lock); seq = subscriptions->invalidate_seq; spin_lock(subscriptions->lock); spin_unlock(subscriptions->lock); subscriptions->invalidate_seq++; wait_event(invalidate_seq != seq); return; interval_tree_remove(interval_sub); kfree(interval_sub); spin_unlock(subscriptions->lock); wake_up_all(); As the wait_event() condition is true it will return immediately. This can lead to use-after-free type errors if the caller frees the data structure containing the interval notifier subscription while it is still on a deferred list. Fix this by changing invalidate_seq to an atomic type as it is read outside of the lock and moving the increment until after deferred lists have been updated. Signed-off-by: Alistair Popple Fixes: 99cb252f5e68 ("mm/mmu_notifier: add an interval tree notifier") Signed-off-by: Jason Gunthorpe --- mm/mmu_notifier.c | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 3f3bbcd298c6..41eb5c2e09f4 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -40,7 +40,7 @@ struct mmu_notifier_subscriptions { bool has_itree; /* to serialize the list modifications and hlist_unhashed */ spinlock_t lock; - unsigned long invalidate_seq; + atomic_long_t invalidate_seq; unsigned long active_invalidate_ranges; struct rb_root_cached itree; wait_queue_head_t wq; @@ -87,7 +87,7 @@ static bool mn_itree_is_invalidating(struct mmu_notifier_subscriptions *subscriptions) { lockdep_assert_held(&subscriptions->lock); - return subscriptions->invalidate_seq & 1; + return atomic_long_read(&subscriptions->invalidate_seq) & 1; } static struct mmu_interval_notifier * @@ -103,12 +103,12 @@ mn_itree_inv_start_range(struct mmu_notifier_subscriptions *subscriptions, node = interval_tree_iter_first(&subscriptions->itree, range->start, range->end - 1); if (node) { - subscriptions->invalidate_seq |= 1; + atomic_long_or(1, &subscriptions->invalidate_seq); res = container_of(node, struct mmu_interval_notifier, interval_tree); } - *seq = subscriptions->invalidate_seq; + *seq = atomic_long_read(&subscriptions->invalidate_seq); spin_unlock(&subscriptions->lock); return res; } @@ -138,9 +138,6 @@ static void mn_itree_inv_end(struct mmu_notifier_subscriptions *subscriptions) return; } - /* Make invalidate_seq even */ - subscriptions->invalidate_seq++; - /* * The inv_end incorporates a deferred mechanism like rtnl_unlock(). * Adds and removes are queued until the final inv_end happens then @@ -158,6 +155,13 @@ static void mn_itree_inv_end(struct mmu_notifier_subscriptions *subscriptions) &subscriptions->itree); hlist_del(&interval_sub->deferred_item); } + + /* Pairs with the atomic_long_read in mmu_interval_notifier_remove(). */ + smp_mb__before_atomic(); + + /* Make invalidate_seq even */ + atomic_long_inc(&subscriptions->invalidate_seq); + spin_unlock(&subscriptions->lock); wake_up_all(&subscriptions->wq); @@ -232,7 +236,7 @@ mmu_interval_read_begin(struct mmu_interval_notifier *interval_sub) spin_lock(&subscriptions->lock); /* Pairs with the WRITE_ONCE in mmu_interval_set_seq() */ seq = READ_ONCE(interval_sub->invalidate_seq); - is_invalidating = seq == subscriptions->invalidate_seq; + is_invalidating = seq == atomic_long_read(&subscriptions->invalidate_seq); spin_unlock(&subscriptions->lock); /* @@ -246,7 +250,7 @@ mmu_interval_read_begin(struct mmu_interval_notifier *interval_sub) lock_map_release(&__mmu_notifier_invalidate_range_start_map); if (is_invalidating) wait_event(subscriptions->wq, - READ_ONCE(subscriptions->invalidate_seq) != seq); + atomic_long_read(&subscriptions->invalidate_seq) != seq); /* * Notice that mmu_interval_read_retry() can already be true at this @@ -648,7 +652,7 @@ int __mmu_notifier_register(struct mmu_notifier *subscription, INIT_HLIST_HEAD(&subscriptions->list); spin_lock_init(&subscriptions->lock); - subscriptions->invalidate_seq = 2; + atomic_long_set(&subscriptions->invalidate_seq, 2); subscriptions->itree = RB_ROOT_CACHED; init_waitqueue_head(&subscriptions->wq); INIT_HLIST_HEAD(&subscriptions->deferred_list); @@ -954,11 +958,11 @@ static int __mmu_interval_notifier_insert( hlist_add_head(&interval_sub->deferred_item, &subscriptions->deferred_list); else { - subscriptions->invalidate_seq |= 1; + atomic_long_or(1, &subscriptions->invalidate_seq); interval_tree_insert(&interval_sub->interval_tree, &subscriptions->itree); } - interval_sub->invalidate_seq = subscriptions->invalidate_seq; + interval_sub->invalidate_seq = atomic_long_read(&subscriptions->invalidate_seq); } else { WARN_ON(mn_itree_is_invalidating(subscriptions)); /* @@ -968,7 +972,7 @@ static int __mmu_interval_notifier_insert( * soon. */ interval_sub->invalidate_seq = - subscriptions->invalidate_seq - 1; + atomic_long_read(&subscriptions->invalidate_seq) - 1; interval_tree_insert(&interval_sub->interval_tree, &subscriptions->itree); } @@ -1066,7 +1070,7 @@ void mmu_interval_notifier_remove(struct mmu_interval_notifier *interval_sub) } else { hlist_add_head(&interval_sub->deferred_item, &subscriptions->deferred_list); - seq = subscriptions->invalidate_seq; + seq = atomic_long_read(&subscriptions->invalidate_seq); } } else { if (WARN_ON(RB_EMPTY_NODE(&interval_sub->interval_tree.rb))) { @@ -1086,7 +1090,7 @@ void mmu_interval_notifier_remove(struct mmu_interval_notifier *interval_sub) lock_map_release(&__mmu_notifier_invalidate_range_start_map); if (seq) wait_event(subscriptions->wq, - READ_ONCE(subscriptions->invalidate_seq) != seq); + atomic_long_read(&subscriptions->invalidate_seq) != seq); /* pairs with mmgrab in mmu_interval_notifier_insert() */ mmdrop(mm);