From patchwork Wed Aug 7 19:16:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11082571 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F25AA1709 for ; Wed, 7 Aug 2019 19:16:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D758328068 for ; Wed, 7 Aug 2019 19:16:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C8B4428A19; Wed, 7 Aug 2019 19:16:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 314AD28068 for ; Wed, 7 Aug 2019 19:16:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F260B6B0003; Wed, 7 Aug 2019 15:16:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ED6236B0006; Wed, 7 Aug 2019 15:16:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D9E436B0007; Wed, 7 Aug 2019 15:16:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 881246B0003 for ; Wed, 7 Aug 2019 15:16:35 -0400 (EDT) Received: by mail-ed1-f72.google.com with SMTP id a15so63369edv.2 for ; Wed, 07 Aug 2019 12:16:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:thread-topic :thread-index:date:message-id:accept-language:content-language :content-id:content-transfer-encoding:mime-version; bh=pMOHoPa07aXfVJpE5Faowks/FP6FWc28xgM4je0v0MM=; b=jghzGG9CoGoScMtH4JOYre0WyOzY6k2SGx6Ns0YKdx4q49ksvgXxaN142Pol3VJlOm qrkaHzRVdndoJrImlG1JaBHGCOJZXSxvMr/CitL2ycnRwlQ1t6TbYnwIF78bRhVbrWjS r9Q0gEPPF962b8ntPBS/bTnTg2/LjpZb89SbrMebtO2w2hpZ2kinI7rZ8vHZlbrq++F4 qSRWMHEQvmeeQxV667794CTo4ZHxu1Z8CdwhRofIiExfA9ofSDq6HsqK0AfA0o4SEYVs h9F4dgz8iw2CLVvDoJi7aMatNpQz/4sFQ7O8nG39yoipR6F3p1ywarzBM2kERmdX50Lj etHw== X-Gm-Message-State: APjAAAURZGDwFjzhZ+tzl071D1ulSOS0cJ+0nBAV/xUG1QCpY0TxjFny 7SQ1IswbXEkQuBMozgYlzskcfvlmldnqEPFsWvnhODevO2otwsRvcWOpjE9b8TjlYinKfOQRIwA KD50j2eYIML+Ee9Q1CA9GfN0Op/1nYLxzzSd9qaqDBYLeniYLOyH1Uk52Fi8beTjovg== X-Received: by 2002:a50:91ae:: with SMTP id g43mr11678320eda.279.1565205395116; Wed, 07 Aug 2019 12:16:35 -0700 (PDT) X-Google-Smtp-Source: APXvYqxP1rC6heHlTDJNIKDC4eP4hlfSSGkPklCmcEk7tSzmjCxT0Uyyq5wkUYKjoyudf6LdJSOV X-Received: by 2002:a50:91ae:: with SMTP id g43mr11678255eda.279.1565205394320; Wed, 07 Aug 2019 12:16:34 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1565205394; cv=pass; d=google.com; s=arc-20160816; b=eK3zfGXPKnt7dcfZMCCr9K8iRX259rNW5VCXT4JA8EpZJIz/yP9I3I+uw5XA52Df6z /WCtyNe0ieZX5vvXHBXdt9Sf2U4zwgX+JlFsdd+t4AYeS5k1RCiftG0rWPmcLbzdoxd2 G/DQ53dTwUdo5zqUJMVgYOcLqUho9jhCWhW3ZTJHCxS1v7nYrbyFcjN3OuDbPxePm2Oz /n9HZaX7E19OKWEV7BCDhWqbD8yRjvFhgxGEOQlSyoO0jOd1fi0ArVcqCy/Ho/c4UY+B mycjbG9UqmgmQud5vUWEOgdf1vb1/cXyciz7BzGvJODWW3gZDWwnFpYn3DIG6k4SCzqi Pziw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:content-id:content-language :accept-language:message-id:date:thread-index:thread-topic:subject :cc:to:from:dkim-signature; bh=pMOHoPa07aXfVJpE5Faowks/FP6FWc28xgM4je0v0MM=; b=jgha4GkxJSxniay+nfmpkJxplbEWrUyNwYmf6VRD2T0CrIQudE0RTrZ416OGNQuVZ5 VhaGoLO94HJMRmfoWboI+TYU8YK1GwGVV+4Nztx+ZQp+KrMWC3rZxRnQdO4Kyjp/Ck8X KEIsQx9kcsjLOllHC9vW2i2LbdfNmnYnf71Mnq8SiU29RZZzADGg+AzcBV8DXXUCkH9R Sc59qwdeXsIBN0EqgkjW0V+77L1q9A1aVPKzd/gxWcn+0dvWZCaBlG4cTvsZhrNYvFad XdH0LeIJmXR/nI02tCG1RxArJpUraxmDuQzhOvd6Jre6oqSKR6s5z2JXoHcJLo7CPIrh nl7A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Mellanox.com header.s=selector2 header.b=nKfaGar1; arc=pass (i=1 spf=pass spfdomain=mellanox.com dkim=pass dkdomain=mellanox.com dmarc=pass fromdomain=mellanox.com); spf=pass (google.com: domain of jgg@mellanox.com designates 40.107.2.78 as permitted sender) smtp.mailfrom=jgg@mellanox.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=mellanox.com Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20078.outbound.protection.outlook.com. [40.107.2.78]) by mx.google.com with ESMTPS id i3si32905668eda.107.2019.08.07.12.16.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Aug 2019 12:16:34 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@mellanox.com designates 40.107.2.78 as permitted sender) client-ip=40.107.2.78; Authentication-Results: mx.google.com; dkim=pass header.i=@Mellanox.com header.s=selector2 header.b=nKfaGar1; arc=pass (i=1 spf=pass spfdomain=mellanox.com dkim=pass dkdomain=mellanox.com dmarc=pass fromdomain=mellanox.com); spf=pass (google.com: domain of jgg@mellanox.com designates 40.107.2.78 as permitted sender) smtp.mailfrom=jgg@mellanox.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=mellanox.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mXd5FNGqs7qd3AZEhl3hyGKFIeHoA39ge7hC81JNHBhE3EjZMIX+SqYs0Pvr6qQTIHB0z6f8KGnYi0Ow30cIsLzqYbMHRkrzObCYNQBkyVKc2zgdyjRSmkO+SS+fztUfiJfXfUrvNJ+WuPKKYKUA0PfnAWucxIwiQcvJNc9TXGcGyops6nLyDYrbbzYND3H1AQyzFc3+G3pHruaPNqsUCjma4WfDqmNyMa3w9ApqrY/zmh/+tJZijn/d0ZVfqkFU5iDGuuC2L/B4GEwRtqw/vI0z0NhP4MlHIdD0/hemG/9ej340+wjxQrw+fTUOtTYM1RoNS7x17asH5CeJ3HElow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pMOHoPa07aXfVJpE5Faowks/FP6FWc28xgM4je0v0MM=; b=h0RMBb9urnAV5LePmx6m8FyIQqmzn6r9/Q1Pq2f0qthpCVH3D3kCP+O3k7NMBD3jbue+HdE/HKExPlvun2NL7QQx3fy+v2j7dYl2yEKSEtfmbwRtybzfQLXNfiQ+ILNGq9ChP1BPfQRFA/rqaMNNRlv42K1cCkYx57lIaoDhQIS0/yKaSKpSSQC713zs1REmhT88lA1YLTZF8tNIgntvCB2m4gdKPhSQsntwIH6UXIxdP9l9+rlkZLYG0aiUoUw+lY2+u3zeBRlyRAcreCMi5vAu/ru4VFnpW3TjnytX7Xp8to9+dLNB92C+L4S3fcumY+asofTvXkNvWpz9Gt+WHg== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=mellanox.com;dmarc=pass action=none header.from=mellanox.com;dkim=pass header.d=mellanox.com;arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pMOHoPa07aXfVJpE5Faowks/FP6FWc28xgM4je0v0MM=; b=nKfaGar1+2N77EAD+X/SKWjo9xRYRyAa5xPuxrQSuhYAvdmQ8Y4YJz47WSY5fQ8sHaqUaT3mlgFTJEKyyOo+MaPv7FOS15HqKS3ASoAepromIHQ2zYpbV7HlBdaPfD+P3yxSBVfyCF5f4D5DdRYqB2gNnIu1/N3p0R5StJ8P22E= Received: from VI1PR05MB4141.eurprd05.prod.outlook.com (10.171.182.144) by VI1PR05MB6064.eurprd05.prod.outlook.com (20.178.204.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2136.14; Wed, 7 Aug 2019 19:16:33 +0000 Received: from VI1PR05MB4141.eurprd05.prod.outlook.com ([fe80::5c6f:6120:45cd:2880]) by VI1PR05MB4141.eurprd05.prod.outlook.com ([fe80::5c6f:6120:45cd:2880%4]) with mapi id 15.20.2136.018; Wed, 7 Aug 2019 19:16:33 +0000 From: Jason Gunthorpe To: "linux-mm@kvack.org" , Michal Hocko CC: =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrea Arcangeli , Christoph Hellwig Subject: [PATCH] mm/mmn: prevent unpaired invalidate_start and invalidate_end with non-blocking Thread-Topic: [PATCH] mm/mmn: prevent unpaired invalidate_start and invalidate_end with non-blocking Thread-Index: AQHVTVSfVNc5Iz/dOkyoKdT7iQpJSw== Date: Wed, 7 Aug 2019 19:16:32 +0000 Message-ID: <20190807191627.GA3008@ziepe.ca> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: YTXPR0101CA0072.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:1::49) To VI1PR05MB4141.eurprd05.prod.outlook.com (2603:10a6:803:4d::16) authentication-results: spf=none (sender IP is ) smtp.mailfrom=jgg@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [156.34.55.100] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a44207ad-0643-4da2-30b9-08d71b6bc218 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020);SRVR:VI1PR05MB6064; x-ms-traffictypediagnostic: VI1PR05MB6064: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 01221E3973 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(4636009)(376002)(136003)(396003)(39860400002)(346002)(366004)(199004)(189003)(305945005)(7736002)(53936002)(36756003)(2501003)(8936002)(256004)(6116002)(66066001)(14444005)(71190400001)(3846002)(14454004)(71200400001)(66476007)(66446008)(66556008)(64756008)(66946007)(6506007)(386003)(25786009)(2906002)(102836004)(476003)(33656002)(486006)(99286004)(86362001)(52116002)(478600001)(8676002)(26005)(186003)(5660300002)(66574012)(6486002)(54906003)(110136005)(6512007)(9686003)(6436002)(316002)(81156014)(4326008)(81166006)(1076003);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR05MB6064;H:VI1PR05MB4141.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: xWmf52uy9DhMeZjJxftnTdQrncXLB9Mrf1xtWptO4rQtfp1rlibG3v6Ei8Lr0t2sAepjolqkY69z/k1+hza99+k19XP9nwvzZtkchpLQxiDuZCGtd+QlSHcL/B95mF19jynCwQNFWWo8//8hBu1DdR/9WQq02rnccMLci5MLrvY66SR4012WaYZa2GvYP/DdtXVj5JnijLgLcXMuP/CD7ZqEgffHqcf/nSw8TpHR7NLicFLjOPtNGyC5JCJdOjqdqrV0/H0mmtXaGP6TlfV/0UMkVh9pPTf2Oot2shD2fQLHynG2bXSLMY9doU/cGfYMT61IVZ5xfphPv0fO5mMLEqKFTgGf7ocncypXK1TwiCn3jxfyh1djeyzPmMQbzqTDF+fZp+HbZmN8MLsfBmeiDJcz9x4hri/1gtNacDXd5Ko= Content-ID: <2C39424E4A67374A9FD32E0888EAEE0A@eurprd05.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: a44207ad-0643-4da2-30b9-08d71b6bc218 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Aug 2019 19:16:32.9519 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: jgg@mellanox.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB6064 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Many users of the mmu_notifier invalidate_range callbacks maintain locking/counters/etc on a paired basis and have long expected that invalidate_range start/end are always paired. The recent change to add non-blocking notifiers breaks this assumption when multiple notifiers are present in the list as an EAGAIN return from a later notifier causes all earlier notifiers to get their invalidate_range_end() skipped. During the development of non-blocking each user was audited to be sure they can skip their invalidate_range_end() if their start returns -EAGAIN, so the only place that has a problem is when there are multiple subscriptions. Due to the RCU locking we can't reliably generate a subset of the linked list representing the notifiers already called, and generate an invalidate_range_end() pairing. Rather than design an elaborate fix, for now, just block non-blocking requests early on if there are multiple subscriptions. Fixes: 93065ac753e4 ("mm, oom: distinguish blockable mode for mmu notifiers") Cc: Michal Hocko Cc: "Jérôme Glisse" Cc: Andrea Arcangeli Cc: Christoph Hellwig Signed-off-by: Jason Gunthorpe Acked-by: Michal Hocko --- include/linux/mmu_notifier.h | 1 + mm/mmu_notifier.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+) HCH suggested to make the locking common so we don't need to have an invalidate_range_end, but that is a longer journey. Here is a simpler stop-gap for this bug. What do you think Michal? I don't have a good way to test this flow .. This lightly clashes with the other mmu notififer series I just sent, so it should go to either -rc or hmm.git Thanks, Jason diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index b6c004bd9f6ad9..170fa2c65d659c 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -53,6 +53,7 @@ struct mmu_notifier_mm { struct hlist_head list; /* to serialize the list modifications and hlist_unhashed */ spinlock_t lock; + bool multiple_subscriptions; }; #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index b5670620aea0fc..4e56f75c560242 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -171,6 +171,19 @@ int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) int ret = 0; int id; + /* + * If there is more than one notififer subscribed to this mm then we + * cannot support the EAGAIN return. invalidate_range_start/end() must + * always be paired unless start returns -EAGAIN. When we return + * -EAGAIN from here the caller will skip all invalidate_range_end() + * calls. However, if there is more than one notififer then some + * notifiers may have had a successful invalidate_range_start() - + * causing imbalance when the end is skipped. + */ + if (!mmu_notifier_range_blockable(range) && + range->mm->mmu_notifier_mm->multiple_subscriptions) + return -EAGAIN; + id = srcu_read_lock(&srcu); hlist_for_each_entry_rcu(mn, &range->mm->mmu_notifier_mm->list, hlist) { if (mn->ops->invalidate_range_start) { @@ -274,6 +287,8 @@ static int do_mmu_notifier_register(struct mmu_notifier *mn, * thanks to mm_take_all_locks(). */ spin_lock(&mm->mmu_notifier_mm->lock); + mm->mmu_notifier_mm->multiple_subscriptions = + !hlist_empty(&mm->mmu_notifier_mm->list); hlist_add_head_rcu(&mn->hlist, &mm->mmu_notifier_mm->list); spin_unlock(&mm->mmu_notifier_mm->lock);