From patchwork Fri Jun 14 00:44:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993953 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 818B318A6 for ; Fri, 14 Jun 2019 00:45:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77BA427968 for ; Fri, 14 Jun 2019 00:45:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6255927B81; Fri, 14 Jun 2019 00:45:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F17CA27BA5 for ; Fri, 14 Jun 2019 00:44:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A79C6B000E; Thu, 13 Jun 2019 20:44:56 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 12A066B026A; Thu, 13 Jun 2019 20:44:56 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6DE76B026B; Thu, 13 Jun 2019 20:44:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id B77EC6B000E for ; Thu, 13 Jun 2019 20:44:55 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id t196so669417qke.0 for ; Thu, 13 Jun 2019 17:44:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=pwIeL+wG8OH0fR+HAJUAqgJL6j2cGj41BLXWfdyXw+8=; b=ABTpH5vIF1JTk0vYntORhW/OK/dFSws597b2F0GrBYV/T93xTlG1dHzkdTgxePRr8f qe1vVwuwpDwEpw12mqgXEs8GQQaRZEPTsz7vem+IHS5FJoDOr9/OPVcbXEmp89NZWcx1 zxq/wSGb7Qh3+67QkPJ5sfWw54EbYD0bzyW9TmeeC5QNz+U7p8mIIIlUfNugGZpDwxfN CVKiXkZ8/tbEe9vAt4ceq9iyVjftKMQdVsjLPUzaIepR3Ew/uGSq3qnGD1jqCQ4sZ/t+ q18vH0Zvndpst0tQrUTP9sBb12DRZLCTGtSdRON2sC1N+bBEeWj8GW3aKD4tZnAUOItx cayg== X-Gm-Message-State: APjAAAVQyt9g9IcV6dbn66mqaNdnrHVPP7t3Lf/s++P0d/xFjL84xWO3 DaNYPyhMGocxiuVHiASY0xxuvDnFOrtBTR4TVztgndYUrA3hFrNbErmQ4EfhWQdMmznWlybP2hU Mrm1048bD86wHuZLM/uxC11fC2rft0ijfuSWTnibJpFMF+S4bgVD94drODavrZOdRxA== X-Received: by 2002:a05:620a:15b3:: with SMTP id f19mr9830224qkk.314.1560473095504; Thu, 13 Jun 2019 17:44:55 -0700 (PDT) X-Received: by 2002:a05:620a:15b3:: with SMTP id f19mr9830200qkk.314.1560473094911; Thu, 13 Jun 2019 17:44:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473094; cv=none; d=google.com; s=arc-20160816; b=unUXESpH2GZIybLfKOVBd9LMEL9PBNSrDj/dXqg3C5zicgfONgQn+cN1GqDmgLATZ0 OooxOW2xoGSNXHEGFk5Js9GNkdQBSBEwy3umlOcuaT60yKiNhw69R6cNmRA/CJcwtZHO F51b1o2CT939TdGYBPRzIqt07US0HccNopakJnyQlqKtajD2RfzUUG8iKjRqp+Tc99RZ PM3R0gG/afGJeFIt28rfLeg0Wbz3zK+L6/RpICIMYXdlNcra3FT4s4n6gzSLLamcOwjU 7chvQy2H9rHBx077iRD0I9QxdSyCbhScShtJ0uA+Kl0WcgWPK+ZVOVn5+/O6XrnQHQ03 10Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=pwIeL+wG8OH0fR+HAJUAqgJL6j2cGj41BLXWfdyXw+8=; b=wMLgmCiMvS94DXYNeN2HL+3Ues+OG/wCmw32olDxXr41MWRiJqR+yZ6bvexYAmu+Yc o8MidWjn6DwK3RtEnwPK53mefasTMDoZEFDbBMg3So3Uxz9IArVWPOrKfd72M3PAvmKZ dzY3msOiXeZ9LUsFtJBTLsc6/M+JLMNPbuH7144saSAP7o4jJq0erCg94wCj/URdLLMq BohYYI5a/Vm8wP1NGusjalPIZ+XL9exkgXbXCJajdAW/wvO1y9htybz/lG4F3mMZz7Dh JpAiqBgi/EVy+GQ0f+ckyRmJSvbY+is3ErQeYQ8/kYVEtHkYtJGi8e40JjjFNcf0cRGR E3Og== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=ceFocxAl; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id o10sor2375783qte.26.2019.06.13.17.44.54 for (Google Transport Security); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=ceFocxAl; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pwIeL+wG8OH0fR+HAJUAqgJL6j2cGj41BLXWfdyXw+8=; b=ceFocxAlhwC8ooDNQRZa8uwa3+dU+B2w/i8m8E2hFOYMSjwgE+QDjK2xkLm6VD72gE WwLJsyaKZvlbLH1T3GBNub1NlDlCK6BubNsnv1WbkKG8RNUlggQq8bBRTYJ5ZD3RlBUe 5Xc4hRx6laHCPEGJWrsK7S9WLrDOWPmFo5Th/r3RbossSgw6mMZGxENAPWZu3X3SfIxe NKWPsMMCkPfwjaA/cZXhWZkA83335+sxM+8H517upoEyeM0OgJPsn7GyvWZ56NeXZYZ/ EkASXeoAhhhAsXq8AEvm382rhtOeMbcMdaynxt6xBN1zXqpJZUfvbUeY29aKi9gpY3TE tkiw== X-Google-Smtp-Source: APXvYqwfWvFSMiMgzDqFFMCY1VR/I1m7UwfYWn0XpXg+QyDkHYMt7hRBdXD727WohxKVcOUw6fUB+w== X-Received: by 2002:ac8:1a3c:: with SMTP id v57mr77400825qtj.339.1560473094617; Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id w55sm902620qtw.11.2019.06.13.17.44.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005JX-Je; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Ira Weiny , Philip Yang Subject: [PATCH v3 hmm 01/12] mm/hmm: fix use after free with struct hmm in the mmu notifiers Date: Thu, 13 Jun 2019 21:44:39 -0300 Message-Id: <20190614004450.20252-2-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe mmu_notifier_unregister_no_release() is not a fence and the mmu_notifier system will continue to reference hmm->mn until the srcu grace period expires. Resulting in use after free races like this: CPU0 CPU1 __mmu_notifier_invalidate_range_start() srcu_read_lock hlist_for_each () // mn == hmm->mn hmm_mirror_unregister() hmm_put() hmm_free() mmu_notifier_unregister_no_release() hlist_del_init_rcu(hmm-mn->list) mn->ops->invalidate_range_start(mn, range); mm_get_hmm() mm->hmm = NULL; kfree(hmm) mutex_lock(&hmm->lock); Use SRCU to kfree the hmm memory so that the notifiers can rely on hmm existing. Get the now-safe hmm struct through container_of and directly check kref_get_unless_zero to lock it against free. Signed-off-by: Jason Gunthorpe Reviewed-by: Ira Weiny Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- v2: - Spell 'free' properly (Jerome/Ralph) v3: - Have only one clearer comment about kref_get_unless_zero (John) --- include/linux/hmm.h | 1 + mm/hmm.c | 23 +++++++++++++++++------ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 7007123842ba76..cb01cf1fa3c08b 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -93,6 +93,7 @@ struct hmm { struct mmu_notifier mmu_notifier; struct rw_semaphore mirrors_sem; wait_queue_head_t wq; + struct rcu_head rcu; long notifiers; bool dead; }; diff --git a/mm/hmm.c b/mm/hmm.c index 826816ab237799..f6956d78e3cb25 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -104,6 +104,11 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) return NULL; } +static void hmm_free_rcu(struct rcu_head *rcu) +{ + kfree(container_of(rcu, struct hmm, rcu)); +} + static void hmm_free(struct kref *kref) { struct hmm *hmm = container_of(kref, struct hmm, kref); @@ -116,7 +121,7 @@ static void hmm_free(struct kref *kref) mm->hmm = NULL; spin_unlock(&mm->page_table_lock); - kfree(hmm); + mmu_notifier_call_srcu(&hmm->rcu, hmm_free_rcu); } static inline void hmm_put(struct hmm *hmm) @@ -144,10 +149,14 @@ void hmm_mm_destroy(struct mm_struct *mm) static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) { - struct hmm *hmm = mm_get_hmm(mm); + struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); struct hmm_mirror *mirror; struct hmm_range *range; + /* Bail out if hmm is in the process of being freed */ + if (!kref_get_unless_zero(&hmm->kref)) + return; + /* Report this HMM as dying. */ hmm->dead = true; @@ -185,13 +194,14 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) static int hmm_invalidate_range_start(struct mmu_notifier *mn, const struct mmu_notifier_range *nrange) { - struct hmm *hmm = mm_get_hmm(nrange->mm); + struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); struct hmm_mirror *mirror; struct hmm_update update; struct hmm_range *range; int ret = 0; - VM_BUG_ON(!hmm); + if (!kref_get_unless_zero(&hmm->kref)) + return 0; update.start = nrange->start; update.end = nrange->end; @@ -236,9 +246,10 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, static void hmm_invalidate_range_end(struct mmu_notifier *mn, const struct mmu_notifier_range *nrange) { - struct hmm *hmm = mm_get_hmm(nrange->mm); + struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); - VM_BUG_ON(!hmm); + if (!kref_get_unless_zero(&hmm->kref)) + return; mutex_lock(&hmm->lock); hmm->notifiers--; From patchwork Fri Jun 14 00:44:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993959 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E52FE14C0 for ; Fri, 14 Jun 2019 00:45:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA0EA27816 for ; Fri, 14 Jun 2019 00:45:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CAF3927B81; Fri, 14 Jun 2019 00:45:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 49C1027816 for ; Fri, 14 Jun 2019 00:45:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1ED9A6B0266; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1794B6B026A; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE6C36B026B; Thu, 13 Jun 2019 20:44:56 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id B5EC16B0266 for ; Thu, 13 Jun 2019 20:44:56 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id g56so729540qte.4 for ; Thu, 13 Jun 2019 17:44:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=op2+qqBk6rWtrVzX85Tdji9xbKocy11Zcttde2t5Nu8=; b=m4g3awBRso8JjcRIVaz6RuSf9bN0V1MdxRnHa6ZyMvfxilVihdZXVJYaoQEE4Ry/nJ pCuzXd48nujpv+C3xVz0Y2JYR/AxPOWmGIvXD4EKSlYTPk8nFEj1XIZsw84LWI/SKSK4 OO/vO15E3XHoJEN2uopTA6ehSkIo8HtDylOlDpI6iQ+wHP4eETQ0hp/Vo8CSr7P3+8pS 2Czz4J2pXqTOxpgECfAX+3mlvsFOPSQbP/1rnWqC2I5a8lrqxpStKGNJq1pu3CzKmB83 K24cogrkXoa1MVayDLmwkv2ysJTufzgQMCqt8BaYZI4l5x1JbbrxNpH7v5FbuiH+Lps8 VvQQ== X-Gm-Message-State: APjAAAULfjGXSKpRIHTlQDwmLuZavqpGwpJlPv+9B+KEcamiECGsJnCF EB23Oos1q2IFM2fZDIDI+AQJbzKxyJg9YiFFMYltVWOmjjXM1trrumbqoWMZX7S21lJGWsRyInl RijtZbRikmx9ZrDsgj4qnf1Wcwyp02fp8SQT4rFX/C7qlvnxfAgk4rvlRsWQob0SQ8g== X-Received: by 2002:a0c:d0ab:: with SMTP id z40mr5978640qvg.216.1560473096483; Thu, 13 Jun 2019 17:44:56 -0700 (PDT) X-Received: by 2002:a0c:d0ab:: with SMTP id z40mr5978598qvg.216.1560473095795; Thu, 13 Jun 2019 17:44:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473095; cv=none; d=google.com; s=arc-20160816; b=a3/UFsTV0FqtNrd/BTCMmNdYqlsY6noN1mGJkOhgDWXuZE8sLGRaD956aO6OoOUnB9 8mxZj7KeoadTRQDKHk9j3c1I8+5ePJl3ZQNoibBEdw+eCvuYSYU+pzrsgmLJb42EBTzW SYtAAqseA87KMlX5OYDoYzqjajHa+VS3Ucj5F1HYph2DHGceBXmVmhjAor3qdzXP2DX0 KsQnMLEnnTSG7tjndih1a4rqZycvGI0A/MXil6fXB9AmyTKnEL4bJr2/6FlR2k0yQyCa jOelwMT4zQDbR/sLTFDBlPW93GgsxPftqCTypinUhD6rih3GArfui+drvURGb6ESj4Bb zkpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=op2+qqBk6rWtrVzX85Tdji9xbKocy11Zcttde2t5Nu8=; b=WbYFbL1Nsnic4CrZ+Zr+HiEE/jF49LQI2P+5y9N3bHFAMAzZfEoYRCi27zr/ag4ryf WRWc62J8YYGzkgT7icJQwkGMdgfomjW/IFexazfBVPy7mLkR26JHLiG1GdnLdJDxIfyh giCl1d1Dnlo/yGfL6E9rWoBFpS5qj3MlKfZ6ByveeiBK4bYfeQj1o7E1SBscjKFF6bdY gHb14dvRnMU2YyaXZynCvcKfuAy6Ys8XSUi4QWZQehlE+O+v/e7aVIviroktL81nqFPp h3KziQrtEQqNNOch0St8pYb0Y0FUTGnMrarBcUuJv6eIZRwS1x0hH6dZEiIx9mG8bdt4 AoJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=D30mf9PG; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id t66sor1066556qka.63.2019.06.13.17.44.55 for (Google Transport Security); Thu, 13 Jun 2019 17:44:55 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=D30mf9PG; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=op2+qqBk6rWtrVzX85Tdji9xbKocy11Zcttde2t5Nu8=; b=D30mf9PGmOmg9I5/9g0MzWxEXedyBaWkkmnNqAl4FA9d0B5THPVVzTtCux6VHVWP76 xPxHkBZrtAGHIjR/J+Se6HOXwAs75X6jrsndrpgNBnZ6e5GoT5vB7IpW5/KkPufHsMig /lKj49pcdbL5F8wiFJWjPLmrYeiEWCPItURysc/SGIc3LsiunwY9uiqrmWElrd7Qwv6Q mtT1IaGjA4RD5yCV1MA+3beQzN1ULDnUdoMUn+fqlk7WTzzxYo8IKqJG0BdnDdhGjwPC zYmW3IXzrRRtAS559siQ/AodNBiqOD7M6e4S7LJU7trsqP+0KuWPmBGt4JAQ6K2dl3hp FojQ== X-Google-Smtp-Source: APXvYqzHD04DoOdk3O8JWoFpyfnlVsPMz9EOodeX9LHTjciArgHYxwJEEoe/sklf3HM5JZVxOOzqsA== X-Received: by 2002:a37:490c:: with SMTP id w12mr71905018qka.327.1560473095502; Thu, 13 Jun 2019 17:44:55 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id s134sm759219qke.51.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005Je-Km; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Ira Weiny , Philip Yang Subject: [PATCH v3 hmm 02/12] mm/hmm: Use hmm_mirror not mm as an argument for hmm_range_register Date: Thu, 13 Jun 2019 21:44:40 -0300 Message-Id: <20190614004450.20252-3-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe Ralph observes that hmm_range_register() can only be called by a driver while a mirror is registered. Make this clear in the API by passing in the mirror structure as a parameter. This also simplifies understanding the lifetime model for struct hmm, as the hmm pointer must be valid as part of a registered mirror so all we need in hmm_register_range() is a simple kref_get. Suggested-by: Ralph Campbell Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- v2 - Include the oneline patch to nouveau_svm.c --- drivers/gpu/drm/nouveau/nouveau_svm.c | 2 +- include/linux/hmm.h | 7 ++++--- mm/hmm.c | 13 ++++--------- 3 files changed, 9 insertions(+), 13 deletions(-) diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c index 93ed43c413f0bb..8c92374afcf227 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.c +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c @@ -649,7 +649,7 @@ nouveau_svm_fault(struct nvif_notify *notify) range.values = nouveau_svm_pfn_values; range.pfn_shift = NVIF_VMM_PFNMAP_V0_ADDR_SHIFT; again: - ret = hmm_vma_fault(&range, true); + ret = hmm_vma_fault(&svmm->mirror, &range, true); if (ret == 0) { mutex_lock(&svmm->mutex); if (!hmm_vma_range_done(&range)) { diff --git a/include/linux/hmm.h b/include/linux/hmm.h index cb01cf1fa3c08b..1fba6979adf460 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -496,7 +496,7 @@ static inline bool hmm_mirror_mm_is_alive(struct hmm_mirror *mirror) * Please see Documentation/vm/hmm.rst for how to use the range API. */ int hmm_range_register(struct hmm_range *range, - struct mm_struct *mm, + struct hmm_mirror *mirror, unsigned long start, unsigned long end, unsigned page_shift); @@ -532,7 +532,8 @@ static inline bool hmm_vma_range_done(struct hmm_range *range) } /* This is a temporary helper to avoid merge conflict between trees. */ -static inline int hmm_vma_fault(struct hmm_range *range, bool block) +static inline int hmm_vma_fault(struct hmm_mirror *mirror, + struct hmm_range *range, bool block) { long ret; @@ -545,7 +546,7 @@ static inline int hmm_vma_fault(struct hmm_range *range, bool block) range->default_flags = 0; range->pfn_flags_mask = -1UL; - ret = hmm_range_register(range, range->vma->vm_mm, + ret = hmm_range_register(range, mirror, range->start, range->end, PAGE_SHIFT); if (ret) diff --git a/mm/hmm.c b/mm/hmm.c index f6956d78e3cb25..22a97ada108b4e 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -914,13 +914,13 @@ static void hmm_pfns_clear(struct hmm_range *range, * Track updates to the CPU page table see include/linux/hmm.h */ int hmm_range_register(struct hmm_range *range, - struct mm_struct *mm, + struct hmm_mirror *mirror, unsigned long start, unsigned long end, unsigned page_shift) { unsigned long mask = ((1UL << page_shift) - 1UL); - struct hmm *hmm; + struct hmm *hmm = mirror->hmm; range->valid = false; range->hmm = NULL; @@ -934,20 +934,15 @@ int hmm_range_register(struct hmm_range *range, range->start = start; range->end = end; - hmm = hmm_get_or_create(mm); - if (!hmm) - return -EFAULT; - /* Check if hmm_mm_destroy() was call. */ - if (hmm->mm == NULL || hmm->dead) { - hmm_put(hmm); + if (hmm->mm == NULL || hmm->dead) return -EFAULT; - } /* Initialize range to track CPU page table updates. */ mutex_lock(&hmm->lock); range->hmm = hmm; + kref_get(&hmm->kref); list_add_rcu(&range->list, &hmm->ranges); /* From patchwork Fri Jun 14 00:44:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993967 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2DA5514BB for ; Fri, 14 Jun 2019 00:45:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2368F27816 for ; Fri, 14 Jun 2019 00:45:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1799D2793B; Fri, 14 Jun 2019 00:45:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9411127816 for ; Fri, 14 Jun 2019 00:45:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 91CF76B026B; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 856186B026D; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65AB76B026C; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 2E30D6B026B for ; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id g30so691577qtm.17 for ; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=OqqjfqwYsPO/UoPKKcnblUG+sfS41bCL5Aovzlt5Fak=; b=gjSL4vCIWnOkzYr+yF0u1cNRdOF+K3arveQPSqZM99/GQPOzywE552JOrjEtlkgw7B b0Op+kSsaVvYgXgquqK6yXWJan+DknaoH76xVsMEif5EtNPcbLNqB4MhAICESIY+Q135 aISGDE+XRam7G3hAmFGdTWCHjfKspXTqXkm2Fju/CSEfwVILn5VyqG+tOAHO5Im7A87K gPfAhrL0Vecttc57sbckipEsGAcdP2ej8ehBa48L/T3zowawb15G1IgkDYpIIZgqX/zQ r6jCDebgwzMoCcT7ZEDRmp6zqZkScViJdYqb2WKmGPqJ2x4zxn6C9viHcbxXsSZ/Tp0U 2pYg== X-Gm-Message-State: APjAAAW9BurrJD8O/jnXgBYswaRHP3h9gngP3WjYlDu0A9ZKQBZ4/y9R JhcNe8uVkyGdeZK1YvnLhAfSNbmnPU/PRX5paqaAR4uUGca0nZWBkCucDY5Y43M4BhKp2TY9/I6 UqGR24TQzaUR6GBPLsI8KI9dx3NxfyjI3aKILtEyMJt5PeWIeX984ZnZNmzj+YX6sGw== X-Received: by 2002:ac8:2bdc:: with SMTP id n28mr40007631qtn.197.1560473096936; Thu, 13 Jun 2019 17:44:56 -0700 (PDT) X-Received: by 2002:ac8:2bdc:: with SMTP id n28mr40007602qtn.197.1560473096276; Thu, 13 Jun 2019 17:44:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473096; cv=none; d=google.com; s=arc-20160816; b=P4t/QZ66ligwcnIIk0kHhL8y+mPiKWdmJO+nhDem8WtMY3Z+OPj4JwVpOmn5p0eFPG 3XaOKRyMZ52205m4FTSXqdDG1KwJ+3tDmpR+CHSC6KOp7t2hWazVO7NuKmsmd5ThOtG/ FU+bnwLWyCF2yD92u1WSO1oYuSVbVCdCNgOKerMUI4IXT0JfEJ5u89odaMqtRm/CmWWh PSv0Or7/PVfRBVPaxFI1kkmXoOyBflziGox/+z9olNHzv5Asxt6Xq7yQ9VsW+u06n4zm l+B+pixEcijAWTE9R0zEsg8nSduG7rT1NGLMoBUyITPKJ+KXtS4xSzc6WLFQzT7F7ZNM 0nZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=OqqjfqwYsPO/UoPKKcnblUG+sfS41bCL5Aovzlt5Fak=; b=sqS6Hx+fdr9RhfmZoNOzQS0WVG/9SoizyChQA9kDeGA0LDI34FT/KArqiLqtFUZB2K RqyBy8JEp32NpeePo0qjfyw1jrIEEbnsXJPjAazOGipIkChwkskaN8LPrQcalkijTufQ xYIzFSwrw2upUhKOH1xEfyyTMBQeQCfb+XH/Fgc8lyg4AA+QwDypoPtS7IHPt85vvi3Y O3pEcUod+bQTvDRFBu7HovVMOkvHdD24PGq4uVq/wg9eCWDSsub26TVxP9vZeaPGOxIB 64clDDCdWvcUUyurgZ8vjramLyD3A+7tRq23U/g7CSMgJdrCGszlM/66mfmNsRh8ouPM aQSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=nqSoFbcl; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z124sor1069641qkd.42.2019.06.13.17.44.56 for (Google Transport Security); Thu, 13 Jun 2019 17:44:56 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=nqSoFbcl; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OqqjfqwYsPO/UoPKKcnblUG+sfS41bCL5Aovzlt5Fak=; b=nqSoFbcl84tRTGM0WJXSa71SkzhvazEOiTJYeTLGJQq74mfUC1lqiEvr2Cco4L/nEy +weOauOt2ZTDcB4618svO7UVbc6fH2ldmdAwFFTugHFlngHnhlI1EIp+5Pb4DZ+cU9DH nCFbS0OWzHmOdR7HoiKIJusLGSKZAhdQlbxUq/KvAcwi16XKsKuEurWf26yqrEIaa3Vl RR10h9G3VhLECokaJc79c/MVxgdr2FpTEz0w1U1aSeyvMQnIU2P999ygE45IVDHnmpyJ uyVIkUfl3punGaCvwfhde1GHKE3NYN65JghNbdZrKyPDAkr38tiYFJFciWM5POtH84K2 Ahcw== X-Google-Smtp-Source: APXvYqw4a4tKkG03NQONcZyCSERYSMaSwwsyoSMJBpDffae71sBOEyhn8TO2i7nXdfVHe4fh2n7xcw== X-Received: by 2002:a37:be41:: with SMTP id o62mr62770902qkf.356.1560473095934; Thu, 13 Jun 2019 17:44:55 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id b203sm657058qkg.29.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005Jk-MM; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Ira Weiny , Philip Yang Subject: [PATCH v3 hmm 03/12] mm/hmm: Hold a mmgrab from hmm to mm Date: Thu, 13 Jun 2019 21:44:41 -0300 Message-Id: <20190614004450.20252-4-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe So long as a struct hmm pointer exists, so should the struct mm it is linked too. Hold the mmgrab() as soon as a hmm is created, and mmdrop() it once the hmm refcount goes to zero. Since mmdrop() (ie a 0 kref on struct mm) is now impossible with a !NULL mm->hmm delete the hmm_hmm_destroy(). Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- v2: - Fix error unwind paths in hmm_get_or_create (Jerome/Jason) --- include/linux/hmm.h | 3 --- kernel/fork.c | 1 - mm/hmm.c | 22 ++++------------------ 3 files changed, 4 insertions(+), 22 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 1fba6979adf460..1d97b6d62c5bcf 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -577,14 +577,11 @@ static inline int hmm_vma_fault(struct hmm_mirror *mirror, } /* Below are for HMM internal use only! Not to be used by device driver! */ -void hmm_mm_destroy(struct mm_struct *mm); - static inline void hmm_mm_init(struct mm_struct *mm) { mm->hmm = NULL; } #else /* IS_ENABLED(CONFIG_HMM_MIRROR) */ -static inline void hmm_mm_destroy(struct mm_struct *mm) {} static inline void hmm_mm_init(struct mm_struct *mm) {} #endif /* IS_ENABLED(CONFIG_HMM_MIRROR) */ diff --git a/kernel/fork.c b/kernel/fork.c index 75675b9bf6dfd3..c704c3cedee78d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -673,7 +673,6 @@ void __mmdrop(struct mm_struct *mm) WARN_ON_ONCE(mm == current->active_mm); mm_free_pgd(mm); destroy_context(mm); - hmm_mm_destroy(mm); mmu_notifier_mm_destroy(mm); check_mm(mm); put_user_ns(mm->user_ns); diff --git a/mm/hmm.c b/mm/hmm.c index 22a97ada108b4e..080b17a2e87e2d 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -73,6 +74,7 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) hmm->notifiers = 0; hmm->dead = false; hmm->mm = mm; + mmgrab(hmm->mm); spin_lock(&mm->page_table_lock); if (!mm->hmm) @@ -100,6 +102,7 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) mm->hmm = NULL; spin_unlock(&mm->page_table_lock); error: + mmdrop(hmm->mm); kfree(hmm); return NULL; } @@ -121,6 +124,7 @@ static void hmm_free(struct kref *kref) mm->hmm = NULL; spin_unlock(&mm->page_table_lock); + mmdrop(hmm->mm); mmu_notifier_call_srcu(&hmm->rcu, hmm_free_rcu); } @@ -129,24 +133,6 @@ static inline void hmm_put(struct hmm *hmm) kref_put(&hmm->kref, hmm_free); } -void hmm_mm_destroy(struct mm_struct *mm) -{ - struct hmm *hmm; - - spin_lock(&mm->page_table_lock); - hmm = mm_get_hmm(mm); - mm->hmm = NULL; - if (hmm) { - hmm->mm = NULL; - hmm->dead = true; - spin_unlock(&mm->page_table_lock); - hmm_put(hmm); - return; - } - - spin_unlock(&mm->page_table_lock); -} - static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); From patchwork Fri Jun 14 00:44:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993969 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 082B6924 for ; Fri, 14 Jun 2019 00:45:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2F3127816 for ; Fri, 14 Jun 2019 00:45:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E75AF2793B; Fri, 14 Jun 2019 00:45:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 62C7727816 for ; Fri, 14 Jun 2019 00:45:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E72E16B026A; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DAD5A6B026C; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8D1D6B026F; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 7022B6B026A for ; Thu, 13 Jun 2019 20:44:57 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id u129so642289qkd.12 for ; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=iQFZUVD3bm8q29tG8rKodDV9niV08KJ4TO1srbmenK4=; b=E/taCVPq+f97Ed9wscAUpd/Fp1rtKG8wLsm9pmlinKogMM29vZThMs6EZjL5l2RR0r 7zM7m4eTJEQnCvugytFYHc2j2oaHvq1YUsmwtXHaXPtOzWygPGYBLtenywJ3vX18l0gH LBYH+G82FtpvVHmC5bm5axEXZO6jNEGV3qc+gj66BSd6UMkTFmSWkv0VmyBr2+L31s8h px8lC50IfxgF7d5hxIwmVcjaNZVeTdu/ZeUy6PR5j9M2ZbRQsE33T5jIMf61rYvSwZQX C10O77jj+Y9WMUYWQHv17LQcs3wqEdGgBPNd8ijByHLLTrhp7qZIZX1koLFD4bbjzGlI VZaQ== X-Gm-Message-State: APjAAAVLXF1Y49yTRCrHI6lMm7tR81GDK0mcZSo/CTRSaVoAo4spB3K/ 8yfNtTcni/Kyj6H6x2zzAB8UVhfqE0OSIivWYex+ngs9Rvfv2pVnoFmXNHzqvh+wInwuBsQ1QMo Kqyb9tcYT4vkQGs+jBx7vNEj5Wae8Ehf69rBFncSe03HAyOTaTX1+bRavpoiQfhYDMA== X-Received: by 2002:a37:de18:: with SMTP id h24mr18765496qkj.147.1560473097240; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) X-Received: by 2002:a37:de18:: with SMTP id h24mr18765470qkj.147.1560473096612; Thu, 13 Jun 2019 17:44:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473096; cv=none; d=google.com; s=arc-20160816; b=z2vwekf0JYqv5CTw+lY+BO0tgfBvfKzvW0GXymUbvqvm/Gj9qi1ZKihUlW6oFeSyVw jKlcqtBI7fxAenVC4qLFBGa6NTR4yZfEDOrJj90xo2mbSg7agVS+485RdfcEBYSvMQw/ /3ALDdT+SB3TO25Lj5dKnW4BNgJT2Gu7VqV8OBO94cr0lWi5Osx6DTpcUnKjjRyGfzlJ 9vNl/8cYEKGvHv2xQb+ikpL8fPfdV1TnGBtgRPLfOYCIBsZsBJI7HPiRAhqvLZ5I5CMT fu0Kuj6ACe704BdyntI3IprUXLQ8tgciOHIPWqoO5paWiiB7ljhCMSPwrZIHyT704Krm qNQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=iQFZUVD3bm8q29tG8rKodDV9niV08KJ4TO1srbmenK4=; b=eyOcvSGUvgcCpAFwsGZxuSuyiRiYhUlXs6OvgKLv5KuxAxc1npm/8jJGX3tjTaInwd DK3bNSDdhrv6WZAnYCtR+vtGszQzyOZSw2tEsQPWjjtJcoMKe1FFo4e+s3INyx26uj83 ttuvsYE8uEdqEzuCCnD/PIcc4rXekrvy9tz3EoeuqKBCt8DbMje9KNymtEDSEfG+2L2/ 2iV5AHA8hyxT7ETLHcKShuq3g/CbkC2NPRHL6oDcGFUKh41OPUod3vWeYHvMfjw9xBnZ VaS3HtsUfze9mcJI3Zhk0pCJn7wI8zwRleNnSLWu3RaSpEzeRso2wfCiCat7CkZRlOdS IK2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=UA1C5NPq; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id o7sor1032381qvf.60.2019.06.13.17.44.56 for (Google Transport Security); Thu, 13 Jun 2019 17:44:56 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=UA1C5NPq; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iQFZUVD3bm8q29tG8rKodDV9niV08KJ4TO1srbmenK4=; b=UA1C5NPqvlm8z++G2laNCnwB/GWT7FWVDsQKpzNw396EnXD5grCbfeRcH4MVcv9Pxt Bw608pa0SQVwTZPesERpsVjDiheiUA0ZxpcaOhSCg3wFfLyCKPSY6m0rcqgkt0F7+Btu VcnvcuEPLZ4+31oNvXwgQ7Mzr5sdaFHqpO9DtB7bD2Mphja5lK5GoftxQttJJRITdM9l ZjsrOqKbZRdEDMQmdCny9EgekTtIr+yc7ps9gI2Z9j+YV7fGRjNJ2Dq2ZT34qRz8bx4L 2H+VNusPv5/abMz/Jh+ee/Y2I8OCp7Q6Wc38Z0vyw+mbGlfjmn34SHlMrcpzNY0sbebl RR8A== X-Google-Smtp-Source: APXvYqw56kQGKLPlj2Zgzkw2ysxgtbRlSboLCdNQH17EQruEZjbScaFJD3FwSz5lL9I3GIRJGcwbsg== X-Received: by 2002:a0c:afa2:: with SMTP id s31mr5894610qvc.186.1560473096316; Thu, 13 Jun 2019 17:44:56 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id l3sm683628qkd.49.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005Jq-Ni; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Ira Weiny , Philip Yang Subject: [PATCH v3 hmm 04/12] mm/hmm: Simplify hmm_get_or_create and make it reliable Date: Thu, 13 Jun 2019 21:44:42 -0300 Message-Id: <20190614004450.20252-5-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe As coded this function can false-fail in various racy situations. Make it reliable and simpler by running under the write side of the mmap_sem and avoiding the false-failing compare/exchange pattern. Due to the mmap_sem this no longer has to avoid racing with a 2nd parallel hmm_get_or_create(). Unfortunately this still has to use the page_table_lock as the non-sleeping lock protecting mm->hmm, since the contexts where we free the hmm are incompatible with mmap_sem. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Tested-by: Philip Yang Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Tested-by: Philip Yang --- v2: - Fix error unwind of mmgrab (Jerome) - Use hmm local instead of 2nd container_of (Jerome) v3: - Can't use mmap_sem in the SRCU callback, keep using the page_table_lock (Philip) --- mm/hmm.c | 84 ++++++++++++++++++++++++-------------------------------- 1 file changed, 36 insertions(+), 48 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 080b17a2e87e2d..4c64d4c32f4825 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -31,16 +31,6 @@ #if IS_ENABLED(CONFIG_HMM_MIRROR) static const struct mmu_notifier_ops hmm_mmu_notifier_ops; -static inline struct hmm *mm_get_hmm(struct mm_struct *mm) -{ - struct hmm *hmm = READ_ONCE(mm->hmm); - - if (hmm && kref_get_unless_zero(&hmm->kref)) - return hmm; - - return NULL; -} - /** * hmm_get_or_create - register HMM against an mm (HMM internal) * @@ -55,11 +45,19 @@ static inline struct hmm *mm_get_hmm(struct mm_struct *mm) */ static struct hmm *hmm_get_or_create(struct mm_struct *mm) { - struct hmm *hmm = mm_get_hmm(mm); - bool cleanup = false; + struct hmm *hmm; - if (hmm) - return hmm; + lockdep_assert_held_exclusive(&mm->mmap_sem); + + /* Abuse the page_table_lock to also protect mm->hmm. */ + spin_lock(&mm->page_table_lock); + if (mm->hmm) { + if (kref_get_unless_zero(&mm->hmm->kref)) { + spin_unlock(&mm->page_table_lock); + return mm->hmm; + } + } + spin_unlock(&mm->page_table_lock); hmm = kmalloc(sizeof(*hmm), GFP_KERNEL); if (!hmm) @@ -74,57 +72,47 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) hmm->notifiers = 0; hmm->dead = false; hmm->mm = mm; - mmgrab(hmm->mm); - spin_lock(&mm->page_table_lock); - if (!mm->hmm) - mm->hmm = hmm; - else - cleanup = true; - spin_unlock(&mm->page_table_lock); + hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops; + if (__mmu_notifier_register(&hmm->mmu_notifier, mm)) { + kfree(hmm); + return NULL; + } - if (cleanup) - goto error; + mmgrab(hmm->mm); /* - * We should only get here if hold the mmap_sem in write mode ie on - * registration of first mirror through hmm_mirror_register() + * We hold the exclusive mmap_sem here so we know that mm->hmm is + * still NULL or 0 kref, and is safe to update. */ - hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops; - if (__mmu_notifier_register(&hmm->mmu_notifier, mm)) - goto error_mm; - - return hmm; - -error_mm: spin_lock(&mm->page_table_lock); - if (mm->hmm == hmm) - mm->hmm = NULL; + mm->hmm = hmm; spin_unlock(&mm->page_table_lock); -error: - mmdrop(hmm->mm); - kfree(hmm); - return NULL; + return hmm; } static void hmm_free_rcu(struct rcu_head *rcu) { - kfree(container_of(rcu, struct hmm, rcu)); + struct hmm *hmm = container_of(rcu, struct hmm, rcu); + + /* + * The mm->hmm pointer is kept valid while notifier ops can be running + * so they don't have to deal with a NULL mm->hmm value + */ + spin_lock(&hmm->mm->page_table_lock); + if (hmm->mm->hmm == hmm) + hmm->mm->hmm = NULL; + spin_unlock(&hmm->mm->page_table_lock); + mmdrop(hmm->mm); + + kfree(hmm); } static void hmm_free(struct kref *kref) { struct hmm *hmm = container_of(kref, struct hmm, kref); - struct mm_struct *mm = hmm->mm; - - mmu_notifier_unregister_no_release(&hmm->mmu_notifier, mm); - spin_lock(&mm->page_table_lock); - if (mm->hmm == hmm) - mm->hmm = NULL; - spin_unlock(&mm->page_table_lock); - - mmdrop(hmm->mm); + mmu_notifier_unregister_no_release(&hmm->mmu_notifier, hmm->mm); mmu_notifier_call_srcu(&hmm->rcu, hmm_free_rcu); } From patchwork Fri Jun 14 00:44:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993973 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E1C6924 for ; Fri, 14 Jun 2019 00:45:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4461427816 for ; Fri, 14 Jun 2019 00:45:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3855C2793B; Fri, 14 Jun 2019 00:45:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BCC4027816 for ; Fri, 14 Jun 2019 00:45:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B51ED6B026C; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A53816B0272; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 747C36B026F; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 39EA36B026D for ; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id n77so636166qke.17 for ; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=4TWMeUG5gYhJoHE/SGg4kXx8gam66v1Nu5XHiuhyFtI=; b=aAGVhIgz+IcWOGL0epYA1EkItPBK52iyXTxof7vEkm/YyCOLuJBIexio2ODmMoF9zW ktRc1lga8sQna/vcGfIsXoOPYSBfoXZFSzz7vQa3A1zMzOV7oPUDXKcqMRf1CbQgBxnl g9oHg/ToVjM4JB4TMdF3RQr6gKjug9nP7WD+bbruUl7qsxJND0t1w910lm0fxaVmM7Ae wTfaudTSkKMNfRT7pq2AIjSEwR0+VXOeSkTkcOXQD+ruR/NDka4Go4LPq71Uzx/uWFuk R3Ch3ssb5zRbXGSXiCMJmZMrvRdc/xiprR1bZ9W12ihwB3amgj+Btn2giZG+tgH/PMFk 2XNw== X-Gm-Message-State: APjAAAX+DPRB/EtdtrjOj+xT+6Kkrh/Mx0GrtakXGorrAd5PzYLkhjnu A7051rJYHKjagIlYDXL7sE6GIKZPIsRSGzauHbpc1aG69YyPgsuHOwZxi4hdbl1JTZvkN/yzMbJ b6xQjSfLjaBR0YIzSZbgKk/+jXfpolOFqQXl7CqSFuq+hMq6w19k28yNXTKqEMMdfVw== X-Received: by 2002:a37:64cb:: with SMTP id y194mr64936778qkb.197.1560473098006; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) X-Received: by 2002:a37:64cb:: with SMTP id y194mr64936750qkb.197.1560473097399; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473097; cv=none; d=google.com; s=arc-20160816; b=l0Ekqo1oK1qcwKEzJjUE399g0kFuYGJVnU7fr2ibbSFKqpSgOE7fiVFjeY3LxmKJx4 tWAb/waPFCeLa4ETt+/s7rZNDI4fbOuujN8oZ/RfAi+Bi6sEaJecNotBTsjZZwlkLDwk 5hl8ldQ839QKHnuP1oR8yngMsVbxbBkY9cg7ggenlSbeewOstDyfwV9sPCGSksKgqhLX Oeref9xHYkobFXPrTHC+S0uUNwTa2UsfJ/TM2fai9/2qo3JVMOi48oBPMzlEuR8noek6 HDfDjAOwzeXMHDT77BNO6+9Y64oMvT0Rl1XydU8Fo/vyRdub8ZzYQKUQotib752hMRUA dcIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=4TWMeUG5gYhJoHE/SGg4kXx8gam66v1Nu5XHiuhyFtI=; b=ydbVizKxLN/HPtCYqVwFJLqlO+LehaqT555B6S0jdyG3eeu6/Li3Rf9SzN+DESGVP3 PnMGPbm/jo3PWVpmMLYWTRDu9DJgsbzK/W3WzyEe81F9eEIFn/kYEWqB5+n47AHZTh0o u9uTfFBOkcDjIZK7KkX9fclfeQk+9UuBwBtZVJqmXMiCLcc0eyjqTMfCGk9bMF4tlPw0 61gBsX2xyOA0IsvWGayfKDg4pyaY/k+Y/9FMVN7Spu0EALRrem83HA3P/0JaR7u3Wb28 B18mvY67dDxska1vCC/Z8E6BvLLVtBqft+DkmM3dkppZps5dIEDc7YaqkYSjBlpP0xFZ 0uiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=Z24AVcLX; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z26sor2385488qta.30.2019.06.13.17.44.57 for (Google Transport Security); Thu, 13 Jun 2019 17:44:57 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=Z24AVcLX; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4TWMeUG5gYhJoHE/SGg4kXx8gam66v1Nu5XHiuhyFtI=; b=Z24AVcLXtS9fjUgk7s4+KXtzjE1aMT2PBcEb07xw67SglUSd3OzuNg62oNcMTOdCCO tuaeQKOzQsp5HbqBLHd8K7YGNauURpRUKgNdhJfNe96/SZel8ZB2rP2x8NzbogFeDlIr +yZSWZUazmveTBDnu7n84ZdABGyatv0J9NLBsDYYL9OXjABLlvao0fEL2KwxuHwW8PZp /+vezcHxEC4oBD4j7w/eFErEFwQXwCtbfqwi4hyrQWWNYY2o77J7tHghIjFCTg3ZdCds UZ2dBkr9KXBNBNjxQciAe4xHvlePPQaKw/4isjgNYaLJ9hRnWkLaIVZIqWtqd6dVzykM qQrQ== X-Google-Smtp-Source: APXvYqwqGlFwoXSFqE9WKQZMugVvrqSYWsVu7EvQI2b+rocz25JrwetUsVc8pOMWgciELT4SKnPuKA== X-Received: by 2002:ac8:3014:: with SMTP id f20mr78276048qte.69.1560473097168; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id g53sm678695qtk.65.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005Jw-Px; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Ira Weiny , Philip Yang Subject: [PATCH v3 hmm 05/12] mm/hmm: Remove duplicate condition test before wait_event_timeout Date: Thu, 13 Jun 2019 21:44:43 -0300 Message-Id: <20190614004450.20252-6-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe The wait_event_timeout macro already tests the condition as its first action, so there is no reason to open code another version of this, all that does is skip the might_sleep() debugging in common cases, which is not helpful. Further, based on prior patches, we can now simplify the required condition test: - If range is valid memory then so is range->hmm - If hmm_release() has run then range->valid is set to false at the same time as dead, so no reason to check both. - A valid hmm has a valid hmm->mm. Allowing the return value of wait_event_timeout() (along with its internal barriers) to compute the result of the function. Signed-off-by: Jason Gunthorpe Reviewed-by: Ralph Campbell Reviewed-by: John Hubbard Reviewed-by: Ira Weiny Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- v3 - Simplify the wait_event_timeout to not check valid --- include/linux/hmm.h | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 1d97b6d62c5bcf..26e7c477490c4e 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -209,17 +209,8 @@ static inline unsigned long hmm_range_page_size(const struct hmm_range *range) static inline bool hmm_range_wait_until_valid(struct hmm_range *range, unsigned long timeout) { - /* Check if mm is dead ? */ - if (range->hmm == NULL || range->hmm->dead || range->hmm->mm == NULL) { - range->valid = false; - return false; - } - if (range->valid) - return true; - wait_event_timeout(range->hmm->wq, range->valid || range->hmm->dead, - msecs_to_jiffies(timeout)); - /* Return current valid status just in case we get lucky */ - return range->valid; + return wait_event_timeout(range->hmm->wq, range->valid, + msecs_to_jiffies(timeout)) != 0; } /* From patchwork Fri Jun 14 00:44:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993971 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38F82924 for ; Fri, 14 Jun 2019 00:45:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E6EE27816 for ; Fri, 14 Jun 2019 00:45:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 211372793B; Fri, 14 Jun 2019 00:45:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 866D027816 for ; Fri, 14 Jun 2019 00:45:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 859056B026D; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7B9D56B026C; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6338C6B0270; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 3F0496B026C for ; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id p43so674182qtk.23 for ; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=vOT/1//lPYO2KnTg7JAYlGe+L6wOGbnqhk1iQ3bfzXA=; b=EpttiUUm26FVUI4Mln1mOXkMZd/lsZE8j8zPy18pFLK01XQi9xzAETW2kM/1arYOJG SemmzzXHFyUUataGohH6ll0+v8Pey9oR8UYj+ypS2oOssK4oyRQd7uEtQP6qc+2VkTqz bWr9qA8Ud52iEe7J/QiYQQDXrSzCvmw8KHXRLOtAk+UFF4w8816eb4rlVqyA4nnlg98E TMCycHmorN7fhVeCLuM3DCGtwVXgE7gLf/Q4vgqjCnl1rrkeC0ICKV+/UF0P7SqazlNH 74fafiJRAUyUqG4SN+M2+xPGTjWBQma5AG0G875VnMY4pHIl9dX0G/88vJbpop03uK1R /B+w== X-Gm-Message-State: APjAAAWTKjfM8tTiBTwxPrcQx2MUgJjG9tkc6nXBjglmcKcw21GYcSsU 0QaRwX0iRVifzjIkbSoUEmOVpH5lIiJ9fJQwVJsFS8swj+WW3fwZnAI4arM+O7psRotyboOWA5c ObEQtAokgUTvzoU55Q4uLRUhTMxJEqZ/4aYBz9rJPE0kNzt5fEBpM4VZBkQfTT/hNzw== X-Received: by 2002:a37:4f51:: with SMTP id d78mr73989732qkb.253.1560473097967; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) X-Received: by 2002:a37:4f51:: with SMTP id d78mr73989702qkb.253.1560473097127; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473097; cv=none; d=google.com; s=arc-20160816; b=AbZk+IrUiPfCBt6QJVgpAXaaI8AxziPRyGRlUB5VXhRc1feBpXfvI9Zy1flwgK2UYj 3EAB7MlL20R9xblPJvAHlDMhZmB2xVtz8uO7c7EeYSlK1lybAq3VQj+CbZ/wFvoj57+e pSafxxbABstNClL1vRxKUpL05hDhIMSLXxuM0nJNr5IoRD+VtEYG69t+KWy+hMnJOJz0 Y0cs59bBcSCJIIvvaDpH5L+S8tHdd2U7Xa9t20tjkqdaJEhRxIekbXFyoLLU0yjWKfuL PsMHvQFflDMiiVqVuzOkrmIyLipHfO00eALmgaBGAoR5SOmuEpkSMqbeaCmwiP9NMeGE Swog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=vOT/1//lPYO2KnTg7JAYlGe+L6wOGbnqhk1iQ3bfzXA=; b=V13zMOuKi4DHrcXBy/3wRsq79wjX1jGg2BhNSt9Bn0OsBjbW38xqeshR/hJYJZqzOz ytm0T/RCJdzefB2axMuvZwD0lCcBSV0Ri6/YlESILn8awcu+K4fQCIg6YOaJ9a8EhH1a fdoeo6Co9x2NKVBmb/wO5XwX3YieVzYRYTazW1XAWDsARJUXwqVeXINVx6mC9IuRrzyp GxwmDQ4qTrg2Xq0oBjft++MHhydQsTvTc4acalg1ivG0biQ4pzbLAqnKNM8/Ksr36SHK zaeITvmNe6eVa6Mg/NjaeYnJjOof6PQJGKXkCXLDUJfLB/3PE2GAMN84pSCg8du6HI6Y nKPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=KUyynqtg; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v83sor1036472qkb.100.2019.06.13.17.44.57 for (Google Transport Security); Thu, 13 Jun 2019 17:44:57 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=KUyynqtg; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vOT/1//lPYO2KnTg7JAYlGe+L6wOGbnqhk1iQ3bfzXA=; b=KUyynqtgRE6EpplNjkgK7xEKW+4i8YtoCdu847R/HKHbJfnzbjofwdI3Ea6k0EuVSe 1CZaEOJYQRtMq3h0sFFCemgzzSgbiIlR01ffFmvPX9pmujQZhT4dybPJ0azLpv2M3X9+ 6Tb39zktfnoe3oIet0A0akwq1fikYli3gOB8Ej+qnnX38PCI4BOZsTugxauTr9grN1NY o/xA6GHrOsOdFk1lM9xdmnpjoyYypBmE2AejoMzAHHaTMtA6L+OhWLo1qhD7nA3VHByF jua1HGswi4DoY/NGhXWwkfny9XYjh5QfCt8aiJgJGRL1dZHZMavFJ9hgwEsQQVXXj1WN tnCg== X-Google-Smtp-Source: APXvYqxEi3V4pWZnBZ6Egs6ifesaB2oNEQOI04AD8yR+MI7rsCDs0etSdyPsSPlsZHy2Q2fiQCFnwA== X-Received: by 2002:a05:620a:1310:: with SMTP id o16mr71613012qkj.196.1560473096852; Thu, 13 Jun 2019 17:44:56 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id z126sm884276qkb.7.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005K2-Rs; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Philip Yang Subject: [PATCH v3 hmm 06/12] mm/hmm: Hold on to the mmget for the lifetime of the range Date: Thu, 13 Jun 2019 21:44:44 -0300 Message-Id: <20190614004450.20252-7-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe Range functions like hmm_range_snapshot() and hmm_range_fault() call find_vma, which requires hodling the mmget() and the mmap_sem for the mm. Make this simpler for the callers by holding the mmget() inside the range for the lifetime of the range. Other functions that accept a range should only be called if the range is registered. This has the side effect of directly preventing hmm_release() from happening while a range is registered. That means range->dead cannot be false during the lifetime of the range, so remove dead and hmm_mirror_mm_is_alive() entirely. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- v2: - Use Jerome's idea of just holding the mmget() for the range lifetime, rework the patch to use that as as simplification to remove dead in one step --- include/linux/hmm.h | 26 -------------------------- mm/hmm.c | 28 ++++++++++------------------ 2 files changed, 10 insertions(+), 44 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 26e7c477490c4e..bf013e96525771 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -82,7 +82,6 @@ * @mirrors_sem: read/write semaphore protecting the mirrors list * @wq: wait queue for user waiting on a range invalidation * @notifiers: count of active mmu notifiers - * @dead: is the mm dead ? */ struct hmm { struct mm_struct *mm; @@ -95,7 +94,6 @@ struct hmm { wait_queue_head_t wq; struct rcu_head rcu; long notifiers; - bool dead; }; /* @@ -459,30 +457,6 @@ struct hmm_mirror { int hmm_mirror_register(struct hmm_mirror *mirror, struct mm_struct *mm); void hmm_mirror_unregister(struct hmm_mirror *mirror); -/* - * hmm_mirror_mm_is_alive() - test if mm is still alive - * @mirror: the HMM mm mirror for which we want to lock the mmap_sem - * Return: false if the mm is dead, true otherwise - * - * This is an optimization, it will not always accurately return false if the - * mm is dead; i.e., there can be false negatives (process is being killed but - * HMM is not yet informed of that). It is only intended to be used to optimize - * out cases where the driver is about to do something time consuming and it - * would be better to skip it if the mm is dead. - */ -static inline bool hmm_mirror_mm_is_alive(struct hmm_mirror *mirror) -{ - struct mm_struct *mm; - - if (!mirror || !mirror->hmm) - return false; - mm = READ_ONCE(mirror->hmm->mm); - if (mirror->hmm->dead || !mm) - return false; - - return true; -} - /* * Please see Documentation/vm/hmm.rst for how to use the range API. */ diff --git a/mm/hmm.c b/mm/hmm.c index 4c64d4c32f4825..58712d74edd585 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -70,7 +70,6 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) mutex_init(&hmm->lock); kref_init(&hmm->kref); hmm->notifiers = 0; - hmm->dead = false; hmm->mm = mm; hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops; @@ -125,20 +124,17 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); struct hmm_mirror *mirror; - struct hmm_range *range; /* Bail out if hmm is in the process of being freed */ if (!kref_get_unless_zero(&hmm->kref)) return; - /* Report this HMM as dying. */ - hmm->dead = true; - - /* Wake-up everyone waiting on any range. */ mutex_lock(&hmm->lock); - list_for_each_entry(range, &hmm->ranges, list) - range->valid = false; - wake_up_all(&hmm->wq); + /* + * Since hmm_range_register() holds the mmget() lock hmm_release() is + * prevented as long as a range exists. + */ + WARN_ON(!list_empty(&hmm->ranges)); mutex_unlock(&hmm->lock); down_write(&hmm->mirrors_sem); @@ -908,8 +904,8 @@ int hmm_range_register(struct hmm_range *range, range->start = start; range->end = end; - /* Check if hmm_mm_destroy() was call. */ - if (hmm->mm == NULL || hmm->dead) + /* Prevent hmm_release() from running while the range is valid */ + if (!mmget_not_zero(hmm->mm)) return -EFAULT; /* Initialize range to track CPU page table updates. */ @@ -952,6 +948,7 @@ void hmm_range_unregister(struct hmm_range *range) /* Drop reference taken by hmm_range_register() */ range->valid = false; + mmput(hmm->mm); hmm_put(hmm); range->hmm = NULL; } @@ -979,10 +976,7 @@ long hmm_range_snapshot(struct hmm_range *range) struct vm_area_struct *vma; struct mm_walk mm_walk; - /* Check if hmm_mm_destroy() was call. */ - if (hmm->mm == NULL || hmm->dead) - return -EFAULT; - + lockdep_assert_held(&hmm->mm->mmap_sem); do { /* If range is no longer valid force retry. */ if (!range->valid) @@ -1077,9 +1071,7 @@ long hmm_range_fault(struct hmm_range *range, bool block) struct mm_walk mm_walk; int ret; - /* Check if hmm_mm_destroy() was call. */ - if (hmm->mm == NULL || hmm->dead) - return -EFAULT; + lockdep_assert_held(&hmm->mm->mmap_sem); do { /* If range is no longer valid force retry. */ From patchwork Fri Jun 14 00:44:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993975 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4EF5D14BB for ; Fri, 14 Jun 2019 00:45:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4462127816 for ; Fri, 14 Jun 2019 00:45:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 380352793B; Fri, 14 Jun 2019 00:45:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D940227816 for ; Fri, 14 Jun 2019 00:45:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DEF46B0270; Thu, 13 Jun 2019 20:44:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 08FBA6B0272; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC4526B0270; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 97BAF6B0271 for ; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id x10so706408qti.11 for ; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=dvkFkBeOW+6z/DYza2/ra7fU7bkF8TW6bEZ+9Ch/ZO4=; b=M9MS6UX3cOW9vrXvXtLfmEmtTEickPNgtPSd2m9dRwCIhvKycetbe8zqSb7qFy1wug vX9ohmvxsHpfr2vd4BMO75UtDyuTf0WqO3bJyiznB+OX+5lvfKEMJubBCh+IVWSaWWXz E2FRYqNMSdeaozR9BzrlgXJYC7ryZjqPW3KqXriBBhGY3WPIpfycjrGiImGRfGIkEmQ7 1cNKaSpQhHcJ/MFbxAfGM/a++alO0wBSexdSl32E8PvkKv8Z9sG2MNCly+LywCgBquCB hZugafhSMuUf5DkPl6mbnppJhYbivwxBEka2FGuZ04JBa6os9vKkkJ+ZKt1oVo6jksSk 95Lw== X-Gm-Message-State: APjAAAXICNtrF/DHS9PXCnjuAYKa9TNTybUpYp0dG66dc1Pzp+1m5AAx EdixC/jvN+/rvo+TOEfV1wsawzgMxZ8PNu/aBWTbHT+EnkGHX7P6+t6HHV438LN/YcSxZADHFLT My/mBL5nAugGGG1Is89TelmHYsHyjulwkNrbsW3LLQGWGhU1zWBMsOJaBLFQpbkXngw== X-Received: by 2002:ac8:224d:: with SMTP id p13mr57455900qtp.154.1560473098395; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) X-Received: by 2002:ac8:224d:: with SMTP id p13mr57455877qtp.154.1560473097878; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473097; cv=none; d=google.com; s=arc-20160816; b=DM9ueje5cRb9guKVmU97/QU5LSGSybIpF5/KGZ2fIHxD1uJ0bY4GqrX/DGvFk37jzE 4yuEBLJqQW7aMabVBVKV6jgzOxNP3OT5Yy9hooeuR+SCzDhmFRTLZH+AlBgp25AiiCdq ULsLsvb4cUPi1Cqm51qXu1bx463uDkAfzpdjDzK4BQFLZPZrOL7KrF9XJ5O8uQBYlgYx 4iBpWVxLJwYaE/g8fWCbIEiEdeqItjox1m7JyUgIOHoO0MSSOLRLN0iPt+aGEMDfZMQN tdg2E1TVf/igh01dRKO5U51NxxOtgixs50rcV9zNpElCfSlGGQ4JCfqhEENIkUi4VTuI LnLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=dvkFkBeOW+6z/DYza2/ra7fU7bkF8TW6bEZ+9Ch/ZO4=; b=G2qkK27fBTlmt1t+4apd7hayw5BW0ydWN7ZKyLMIcOHvyzlxpd0ouITmMmZ+si/8ls GmSHpmeFjqmtZatNM1NLBjBurzetpgwBPa8KK++VTwlTXXUkcPxep26zcHuW5V3Uk68A 11ut/rwlXpUnHHquU1giPy2c2yz8rbs2rkjXN5vmMYimNq05DbsSz1HFIsOzd5IRq7jQ pzzSoZEgWJBHwUgsu+YQaCogArz4R5i7PzkaC+uuiuD/o+5mAGRV3ULURF9UZETBwebF TeuuF9AbuWdNgajEq04GHtQQ/Jwy/runq8UXZRPOuSOOH3cDAxiR2GfJ2yWvJxwbCYDG 5iEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=eYn88Fgu; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id q38sor2362635qtc.33.2019.06.13.17.44.57 for (Google Transport Security); Thu, 13 Jun 2019 17:44:57 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=eYn88Fgu; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dvkFkBeOW+6z/DYza2/ra7fU7bkF8TW6bEZ+9Ch/ZO4=; b=eYn88FgunOZPk+YyYFR2JnEf7f2u6p9sflPJ4sT+jetqG8JYSyAB8le6N712NkSohY Xwi2VyaEkzxPEqC8yi2d5vTAq6YJehvf3luJZZZ4GvsaI+jdZ2gVqX0uxK3KJG8fUqJN 3FvqnY5Sb8nnX+WQ++LovwzB8U/d4qUmUyh+DIkLwGIxK2N5g7LSrT+GUNIoWzDFfVad ehBIjCCC/XRamOtEckjvwtqZSft++bwMX6ZPisONHj3wc7uPh1GqiR5ueIOcNZHjO+1j aWPQ70BMw6QV6bP5mmW+tRl6mOixqkIWqtgj8Ext78+i143gu4SmwIuthYXP1wzIHsDP faQA== X-Google-Smtp-Source: APXvYqwehM/5YZE2rHAarSGtvnt4dD381MX43AFO3OEKYtZwTgDkAYkIX/giHqXAhpmFto1vVdJqMw== X-Received: by 2002:aed:21ca:: with SMTP id m10mr72204568qtc.97.1560473097656; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id o27sm688657qtf.13.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005K8-TQ; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Souptick Joarder , Philip Yang Subject: [PATCH v3 hmm 07/12] mm/hmm: Use lockdep instead of comments Date: Thu, 13 Jun 2019 21:44:45 -0300 Message-Id: <20190614004450.20252-8-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe So we can check locking at runtime. Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Acked-by: Souptick Joarder Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- v2 - Fix missing & in lockdeps (Jason) --- mm/hmm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 58712d74edd585..c0f622f86223c2 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -253,11 +253,11 @@ static const struct mmu_notifier_ops hmm_mmu_notifier_ops = { * * To start mirroring a process address space, the device driver must register * an HMM mirror struct. - * - * THE mm->mmap_sem MUST BE HELD IN WRITE MODE ! */ int hmm_mirror_register(struct hmm_mirror *mirror, struct mm_struct *mm) { + lockdep_assert_held_exclusive(&mm->mmap_sem); + /* Sanity check */ if (!mm || !mirror || !mirror->ops) return -EINVAL; From patchwork Fri Jun 14 00:44:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993977 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 85EC714BB for ; Fri, 14 Jun 2019 00:45:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7C0ED27816 for ; Fri, 14 Jun 2019 00:45:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6FEFC2793B; Fri, 14 Jun 2019 00:45:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 196D827816 for ; Fri, 14 Jun 2019 00:45:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 486536B026F; Thu, 13 Jun 2019 20:44:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 435D86B0271; Thu, 13 Jun 2019 20:44:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28D6A6B0273; Thu, 13 Jun 2019 20:44:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id E45156B026F for ; Thu, 13 Jun 2019 20:44:58 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id c207so645909qkb.11 for ; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=qJP3s5LCTJOa2Bpv3WsF12kBeuFb/g7s2rVIALekTDo=; b=bQnaDIl18YZpYEmrZfK0KhxIyd/TI8FphJc+j2k18CkxnqPBV3gr99PmPgwcYG8yGq ZO4KcnwFyxiSb5kVLxxMs9+Tr66nRXOnagWODasTL9IMc8nRhhIfhL8fIQtbMcqW93za C0G1VPjjL/2QucOEW2pKVg1+8N4nyMWYFchyfWc1AsG1SfrD7wDQtVMb7RViYMeMjbzo f6MZPvcUZ75+bZ3pA6sX3jgfPHwA4zMWM5Pv09ZWl6GRbWjmP4Rl8TUZv9ZJ7hseSEro DvjE5fXrSdd/7x219+QuDHYQ6ueRGOe0ziVgj69J6/jkZBvgFeW8RT+jf3kJ5EMoNyoe gvcg== X-Gm-Message-State: APjAAAVHv9RknFHGRdYQ+nou/r4wGEk+qjuHUSFY9ZlDZprwA/Ngit/y m9AFif9OnY6GR5a4XoMNtWw8bIi57Wi9BH5CMrkyuAgFsajAVwKO7qx9BR+MNXxFPpvyr0oXIuS iTt3R77VEtsRiaOlAjo1Xw4VmGHaRbmr5U9tiKq4GAUo2H/QCEGTLmKdxjeW4to/7xQ== X-Received: by 2002:a05:620a:152:: with SMTP id e18mr71948262qkn.101.1560473098711; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) X-Received: by 2002:a05:620a:152:: with SMTP id e18mr71948239qkn.101.1560473098175; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473098; cv=none; d=google.com; s=arc-20160816; b=jbEvCUpC9GITi4VbaLZxY42BCz3OhBpSVtk0sUTpqpCXLTswC9JAf0x3nlGkuh7rdH rPzYgfNyv76UpQ4pVphJ5d4b2xYNoa8ZGoGJT3C8pIMLS2JXmrPTWD13itdrmjsTvNJ/ 0mv1oDn+b2GyBsZngjjq9UUognnpHojBktVwqTej2D7iZBjpBtjha+ESWxbUnnCf5XBr XRvnefJpSXCjp8Lz/AmJdMC/Jark/feDQoVxETlIRkaMjAdMjN8q4r7L8M/5bOsFMT0H e7UQxfHg32z41aHqlh20u3mEBMN4cccpA6YH/JeWAGhudOajuRFt+DdRKb6eOAZEAEeE 9lpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=qJP3s5LCTJOa2Bpv3WsF12kBeuFb/g7s2rVIALekTDo=; b=a5bQFG4F3m76XRce2wVx/sHADOHzL4rWmmrT9BHcs6lSQTFUqUCPK9OKNbS/1xFhBE 94KF1xaSJqf9hUJUBXQGvO8y72A87jDJL+5/Wv5nS4XdAj3sUoNjwpcJjsy3eMW46RTQ fdLwKKUS9QtsA5X4vhKl7l641x/PuPDS8FCl4vF2Gr8QWTK9AQhUyYpYNP7FSMnYDkwo ngK7mMxw648SvIzNGwWqTDYUxIrHKNEGqhucnkl/jjpKCDBDG8bi13RRJ6f2rl8Gc6Js w6xdLT3bM/uzLA08YzWOoyqpQsZfb526QwbPguAdZb9EFvRYW49p0sZEMQoaHnvuPsuO 1Fgg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=dyouVytY; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 136sor1077907qkm.23.2019.06.13.17.44.58 for (Google Transport Security); Thu, 13 Jun 2019 17:44:58 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=dyouVytY; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qJP3s5LCTJOa2Bpv3WsF12kBeuFb/g7s2rVIALekTDo=; b=dyouVytY+jmXKdkQ6AEcE55kZQEWs6bscsYHu17tMqMjO0ykbKjOyu4AZmogdzcKOO okRY9yuPH0h4p3cAV4QrnZ4n34NPuyuK97/dkGmB3VzQKnqK9s/XAJtvvIAGZkXqI8Jd kp8Bb3YU8e6ArRWr/jcI134rAI1zQOreSW9nSzEGghOTmJk/pJslGcpZKvhe05E8ekCA wMSMERjmVYizkkeqJXXp4Y3n+nU+hNQwdsuP0lS3FbYfoYhnJGG75bzPsyyRZjGeXB24 rhFdNEenuxIK01r0M01cZSie0WszSDebNOB+aFhxWAf9m8Ut+WvZhCrUuTgOing0GJS6 L08w== X-Google-Smtp-Source: APXvYqwYuZdBUIyBE3tnoT2pvasr4EEIwX7YCMbTNFDbznpFFTMobqSAyRGm2KIgVEvCZYbf+NYJww== X-Received: by 2002:a05:620a:144a:: with SMTP id i10mr72376829qkl.130.1560473097962; Thu, 13 Jun 2019 17:44:57 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id m21sm498643qtp.92.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005KE-Ue; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Philip Yang Subject: [PATCH v3 hmm 08/12] mm/hmm: Remove racy protection against double-unregistration Date: Thu, 13 Jun 2019 21:44:46 -0300 Message-Id: <20190614004450.20252-9-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe No other register/unregister kernel API attempts to provide this kind of protection as it is inherently racy, so just drop it. Callers should provide their own protection, it appears nouveau already does, but just in case drop a debugging POISON. Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- mm/hmm.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index c0f622f86223c2..e3e0a811a3a774 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -283,18 +283,13 @@ EXPORT_SYMBOL(hmm_mirror_register); */ void hmm_mirror_unregister(struct hmm_mirror *mirror) { - struct hmm *hmm = READ_ONCE(mirror->hmm); - - if (hmm == NULL) - return; + struct hmm *hmm = mirror->hmm; down_write(&hmm->mirrors_sem); list_del_init(&mirror->list); - /* To protect us against double unregister ... */ - mirror->hmm = NULL; up_write(&hmm->mirrors_sem); - hmm_put(hmm); + memset(&mirror->hmm, POISON_INUSE, sizeof(mirror->hmm)); } EXPORT_SYMBOL(hmm_mirror_unregister); From patchwork Fri Jun 14 00:44:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993979 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C4D6924 for ; Fri, 14 Jun 2019 00:45:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0272927816 for ; Fri, 14 Jun 2019 00:45:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA4A12793B; Fri, 14 Jun 2019 00:45:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 82F1827816 for ; Fri, 14 Jun 2019 00:45:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC57D6B0271; Thu, 13 Jun 2019 20:44:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C72BE6B0272; Thu, 13 Jun 2019 20:44:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF1526B0273; Thu, 13 Jun 2019 20:44:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id 825D66B0271 for ; Thu, 13 Jun 2019 20:44:59 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id 18so643516qkl.13 for ; Thu, 13 Jun 2019 17:44:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=TdUL2Q0xcyDT6JNdu0gecxArNl/zROPizNaMNSjNgKw=; b=C2T+wn0jgBnhaMLPQwY+mW6YyQ9x93IQy13gUoVQ9Zplwph8aYsYfPa6UZLRU/NIM1 BVYyjRFYAVMcaoddRVAEXEvjnoa4kX7XEikFSLzAWUZ9KF6MXnp1LLkZnzdlRyyDgbbA 9f5OCh62aDX4l057680oSdGIXccoc6euCqLk8tF8sZewoCyY7vRAjhUCqalL0PWcoS2m NjZIDBzYgOmGoIHarajSCA27oqEjqRY/iSS41Ajcr0rQ1vhPJar2vD4eMSZ49qcFd3Hh 0vsqKgySwuBBJuYB7vRFgu+9fCsAsuRemyifAcP1+/Cyfjrzvgojz3d+pTosAqYM6wDL 2uXg== X-Gm-Message-State: APjAAAWcVynLSPcV30FQ8DU8He0tjYettOxPEsfxiHW+Ljm/Mz7zmo9y qBinhSbPJQvi4UqB0QEbA+o8wMvOJM/a0vEVQdhFa4Sq1dnLiE8Tw9LW19/jUvrIzJ9gtLi5nfr DqjcR2l8SE6iW9Lx+zU/wMJB4vuSrG06Du1seWDEDQIBU4xQumanK8aI1PtFALMDQkg== X-Received: by 2002:aed:2389:: with SMTP id j9mr51873358qtc.244.1560473099301; Thu, 13 Jun 2019 17:44:59 -0700 (PDT) X-Received: by 2002:aed:2389:: with SMTP id j9mr51873336qtc.244.1560473098742; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473098; cv=none; d=google.com; s=arc-20160816; b=KrPzUQITDQcapdmE9LLc6IC6+2suV84ro5p+6ZY/OKq7hYgE8OHp41Kvlp64Auqcbc LKgMbwg0i85uqcQ6cOOO8dllr3sA4+hFmA+bvSJCuWTWLagLNAZbYQwEAUDpasfimGQx TKHS1s4Z5+eA/owmDQspvhSmaNMf8p3WuPS9VMvlCoWvgWLRfqKhckF85mWoUNdJnwoQ 5OdcQ58g6tV5OaXTjobkdoxEOrTkTSMrA2ESEImt8FFgq0aTWDjHY5NkhECipfqvAEC4 YIblyqZyjZFyEUnQD6Tos+SRAq6cudH4X3lS1E927fZiSx84jPdcmDJUA8iaW1YxobAK DGcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=TdUL2Q0xcyDT6JNdu0gecxArNl/zROPizNaMNSjNgKw=; b=yj2ZI2Emlnn9p01x5LwInpqwqOpKxPrYwnc5uAxuFLhr7KixR5nRI7P8cyNWnEOCAq EGjeOfLKC82xwboiJ6uzcig7fA2vmruK8YzvQj7afdvFTbQ4bNlkWla93xJYz77+HUET kOd/HOTvUjoflW/LPd0oXjDQdVPFD4ytWyszjmv5zZ53WEFOMSe8blDu+F2knlbWFroE eqV6gkLr1cxYBbY11/ld3GQsN0Jfj7okYX6K1jDrtHuHRKElMX2/drG2uhTSmb5avqiB 1QXHJ1Rxd5VChPJqzVQ5pxClJcNtzh6x99TEoGpS+oo9PseTVhI4Ar+tWhKEtG1XMGFx UJnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=FK1BE5Q4; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id q4sor2370998qtj.16.2019.06.13.17.44.58 for (Google Transport Security); Thu, 13 Jun 2019 17:44:58 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=FK1BE5Q4; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TdUL2Q0xcyDT6JNdu0gecxArNl/zROPizNaMNSjNgKw=; b=FK1BE5Q4R1zyEjUAK720jTuxSx/cyRhVSQ3jayrUcNyqoaPlGBfGmGC1CPOx43s3ST 6RpK5IdpVUApoJZo8g1TT+C5zdvZ2TG9uTPr7XHg6jJi/A9cEPbHcheLCuTa0i1x7Z41 Mi3bwHwKr3TrXQAMy8P4kjfKrR78HOCBwiNjC5ZDJZ0bKUEU0G6TbXkgv+rpTyv/xlcc dIOmYy6zgWm5Iw4sumeXpNYtGyd/WNLx1EtYDSkyfOURkacd+3GKhBbELl5JFem6hlqX 2IL4K40Gw3ZaHhwFwAEd4IWGIF1j4HktqVkUa6VK5L2oBj8qFHmE/mdZt+ga7o2HmLyu aP3g== X-Google-Smtp-Source: APXvYqzF52VUT6PLbBdFr+9hAJYSTEw5qk29kEu9lNT9XBNX8ynGL9sSdzpvwppRLNVS4ebk9QrzSQ== X-Received: by 2002:aed:224e:: with SMTP id o14mr78885704qtc.271.1560473098488; Thu, 13 Jun 2019 17:44:58 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id n48sm812748qtc.90.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKr-0005KK-Vm; Thu, 13 Jun 2019 21:44:53 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Souptick Joarder , Ira Weiny , Philip Yang Subject: [PATCH v3 hmm 09/12] mm/hmm: Poison hmm_range during unregister Date: Thu, 13 Jun 2019 21:44:47 -0300 Message-Id: <20190614004450.20252-10-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe Trying to misuse a range outside its lifetime is a kernel bug. Use poison bytes to help detect this condition. Double unregister will reliably crash. Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Acked-by: Souptick Joarder Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Tested-by: Philip Yang --- v2 - Keep range start/end valid after unregistration (Jerome) v3 - Revise some comments (John) - Remove start/end WARN_ON (Souptick) --- mm/hmm.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index e3e0a811a3a774..e214668cba3474 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -933,19 +933,21 @@ void hmm_range_unregister(struct hmm_range *range) { struct hmm *hmm = range->hmm; - /* Sanity check this really should not happen. */ - if (hmm == NULL || range->end <= range->start) - return; - mutex_lock(&hmm->lock); list_del_rcu(&range->list); mutex_unlock(&hmm->lock); /* Drop reference taken by hmm_range_register() */ - range->valid = false; mmput(hmm->mm); hmm_put(hmm); - range->hmm = NULL; + + /* + * The range is now invalid and the ref on the hmm is dropped, so + * poison the pointer. Leave other fields in place, for the caller's + * use. + */ + range->valid = false; + memset(&range->hmm, POISON_INUSE, sizeof(range->hmm)); } EXPORT_SYMBOL(hmm_range_unregister); From patchwork Fri Jun 14 00:44:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA3FA924 for ; Fri, 14 Jun 2019 00:45:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 94F0A1FEBA for ; Fri, 14 Jun 2019 00:45:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 881FA26224; Fri, 14 Jun 2019 00:45:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2665E1FEBA for ; Fri, 14 Jun 2019 00:45:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6844D6B0272; Thu, 13 Jun 2019 20:45:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4FCC06B0273; Thu, 13 Jun 2019 20:45:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D6CF6B0274; Thu, 13 Jun 2019 20:45:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 023466B0272 for ; Thu, 13 Jun 2019 20:45:00 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id v58so732771qta.2 for ; Thu, 13 Jun 2019 17:44:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=x2xcZ0hUzEe5obPQKOQJn2EFwkT6zfHmS/ojlfdj62g=; b=IggjAGeOwnSmGbaPpkQdZWJfEvKlQiCV+sdWcYZYtPsx8c1XuulvcT1D9mPYZsGmza XSG3Vj9q1wsroP//rnaQjto3BEnfZbueCFSJW3HFE7arkM4jHPvifZVibX4mAwI3CpXd s7u0oIGblbcqmcNe1uFCQeLolnTUQvwYqwqk1Kp0SGUJuWyy5MRKB6EoQOjGG/+OPOJg AoV7+l+mpoMmwpy6Avna6ryU5gbp3UfHAl84cyhvKLJ9VUp5p9C4H3AGHix65eqx9Heh xYQQ5LA4pv+4mwg3utrmV0eiX/cqrTCWzuIp9t8Ee1SKRWxyuPnn2osMmdWLrJGf6cRE GVSQ== X-Gm-Message-State: APjAAAVhDeP3Har2+kintKtWVs3tvz2oJyPp5Kpag35GCHuU8WfAXJGv GgIZWLDuWV+CTW/72moK/yGc/3IytwubAX0mUKU+TZ7np3G3AyHzEf9zlxZtX2p5qNT1cCha6O+ PcVR/0NnfbPpnNit13dLn6exmSotO9AOOsusypC5z79T0yGWPWKzRjJVQbOAf3dRs/g== X-Received: by 2002:a05:620a:147:: with SMTP id e7mr72760190qkn.247.1560473099774; Thu, 13 Jun 2019 17:44:59 -0700 (PDT) X-Received: by 2002:a05:620a:147:: with SMTP id e7mr72760171qkn.247.1560473099240; Thu, 13 Jun 2019 17:44:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473099; cv=none; d=google.com; s=arc-20160816; b=CUR0TfA3K3vfWeZIs7st+ut8PCNZ07pJRZFW/V4UQgwAif2KWkvFWo5ngvCqF7VoAt +sDHWA1vkpCR2KDBNCNIRssORnAlBPTw0Z1OkLOuOmfBFJQtCyPKi4FpOkZwrQgd+ZPv pwUEwiN63IB6dqwsIFBvNC7vp8hO7CfEl3YS8N6NZ+Mn5I7JC8iSTFMdEoDRTsz2WF4R eBGvPbZBSxO8zxRByPMXaEUdau2iUmXyXVE7ygzRUuem8zzKvfDlSq1gNe1wlGqK63T7 4yQPKgSiWmraeySPKWoRmadYQItMly+70QzBzK788cA72TT6n5HzBLGxqGlSlippSJq5 XulA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=x2xcZ0hUzEe5obPQKOQJn2EFwkT6zfHmS/ojlfdj62g=; b=ruaOOkho3g1o3z92j4gOLmvzbxwGJETCy1X9AVkpXMU882YcROucZo7L7bBSZzEWMW 9cDXs2DSU2jJ6/cfdhuOdbtRjxP0sYeAXtif91ALq0K2sgyYJVwXConjp0DdgtWpCP5l IAMfIeiNjU3j6bH5CBW9mNW7gAv7Ny+pbCrGfcmlxqRZTzcmxgPZkvk52UlHxllDyS20 2Muc9qCndQLSBg3FMj1TR0+5TmeR3+MX35CiyypwmsAUCRkS4O3dkSLNzC1QDkfiDbQJ j55CDCH7jI07hQVYmCBUsIixkTGshA9Tdtc2QlWTS9Cq8953XUIzlol1NjieIC+HZaA2 yNWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=fmkDVzgS; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id n1sor1033422qkd.102.2019.06.13.17.44.59 for (Google Transport Security); Thu, 13 Jun 2019 17:44:59 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=fmkDVzgS; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x2xcZ0hUzEe5obPQKOQJn2EFwkT6zfHmS/ojlfdj62g=; b=fmkDVzgS2vVrdEiIFYVAnbf6P4BATGmI5DV7vsT+OeMjI4tAox+P6sovJkfdiZ77nO TT+3owVCCkhWvSGoSNguAk+3EcblRkw7UWgxiYOoQUIg3Krv9JVjCiJAQm/Z0vfo/EQO KilEhiFU8QK10vlLGWCYF1bTI+QT+GOn8FgfKnRlLwynY8K2mvFNbldzrI4Lge3bpnfB LwwdBdhl/qwje1Wf18PrfGX26/MhvkXIgT48JtyA/Mh0PWYhgZzhVUdAvgS8HFf+7d0p m4F365LJ+Y/0qG+JXDxYUTDe3aQRit0hcrauKtBkjsXKtLtzRJ2lxMjKyKDLXCmxU6K9 9xfw== X-Google-Smtp-Source: APXvYqzCPZOcBvdDvO5A/+aWYFKA+YngTOxLtxM859wIOaaSHbD/Xj8J7AD4AHMtWoKrqY+ZKezZsg== X-Received: by 2002:a05:620a:1661:: with SMTP id d1mr13812583qko.192.1560473099006; Thu, 13 Jun 2019 17:44:59 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id q56sm978355qtq.64.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKs-0005KQ-1j; Thu, 13 Jun 2019 21:44:54 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Souptick Joarder , Ira Weiny , Philip Yang Subject: [PATCH v3 hmm 10/12] mm/hmm: Do not use list*_rcu() for hmm->ranges Date: Thu, 13 Jun 2019 21:44:48 -0300 Message-Id: <20190614004450.20252-11-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe This list is always read and written while holding hmm->lock so there is no need for the confusing _rcu annotations. Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Acked-by: Souptick Joarder Reviewed-by: Ralph Campbell Acked-by: Souptick Joarder Reviewed-by: Ira Weiny Tested-by: Philip Yang Signed-off-by: Christoph Hellwig --- mm/hmm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index e214668cba3474..26af511cbdd075 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -908,7 +908,7 @@ int hmm_range_register(struct hmm_range *range, range->hmm = hmm; kref_get(&hmm->kref); - list_add_rcu(&range->list, &hmm->ranges); + list_add(&range->list, &hmm->ranges); /* * If there are any concurrent notifiers we have to wait for them for @@ -934,7 +934,7 @@ void hmm_range_unregister(struct hmm_range *range) struct hmm *hmm = range->hmm; mutex_lock(&hmm->lock); - list_del_rcu(&range->list); + list_del(&range->list); mutex_unlock(&hmm->lock); /* Drop reference taken by hmm_range_register() */ From patchwork Fri Jun 14 00:44:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993995 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A6CDA76 for ; Fri, 14 Jun 2019 01:04:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DCE226E90 for ; Fri, 14 Jun 2019 01:04:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 91BB427528; Fri, 14 Jun 2019 01:04:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2700A26E90 for ; Fri, 14 Jun 2019 01:04:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11B098E0002; Thu, 13 Jun 2019 21:04:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0CD4D6B026A; Thu, 13 Jun 2019 21:04:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFBA58E0002; Thu, 13 Jun 2019 21:04:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id CAFC86B0266 for ; Thu, 13 Jun 2019 21:04:04 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id n77so671467qke.17 for ; Thu, 13 Jun 2019 18:04:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=8awC9I57ul4Ehs65t7/5PNP0ULl1lssVMkwIyYI5A4o=; b=kS/l0YsfmLZWtm68oZ6+WMDf5dyng02KlawbUtgxJJug3ckdr1ijynoBFE/VD6mQUT 1PD8+Ng+FZB8aZAT/nh10fvqdtKRIYG1+E+bMYgxOu9rZSPL/komw1WWCv8nEkvC6IN5 jqh++BoRYlILdRMcB36Ryo2R4UtGNljiWpEwIeibPBn2BcXbiGxpwbTA/aoh+NDgUJzn CrDBhJCGsxb/BavjTmZewMGE6q27ietoyzhim7qWeX1NNnzfjTaFmCQJb6jLWpobwd/5 PyBq5n3C3M7ZGjczcvrP+SyjeS+34gUT2C2Kq13jX8QZgSYbOS7rSbDz6pMLhzLeZQtZ 0kvw== X-Gm-Message-State: APjAAAWYgRfknQyF+uqZuP36AHAoqsv/18b3GV9gVqEsoxLfBp793u34 tzC1b5MWTGJuv0pAgsqBBpSrO4hJbYxJrWf35xMvO+1WPOhK57Bl7D+gqR/EPzpU5xgyPsbsaYN UzSD+eVqPoOHtP9/iuvYqrIqA5kXKshxVk+SUpuTuCS88864KQS1Rx/kr5cHpzWwbJQ== X-Received: by 2002:ac8:2cfc:: with SMTP id 57mr76808158qtx.194.1560474244594; Thu, 13 Jun 2019 18:04:04 -0700 (PDT) X-Received: by 2002:ac8:2cfc:: with SMTP id 57mr76808112qtx.194.1560474244015; Thu, 13 Jun 2019 18:04:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560474244; cv=none; d=google.com; s=arc-20160816; b=IIOUgWutTT5df44up0Be/AoeawL+S5ddpxtnhHs4n7q7a4xR5qAUAo2YVv5QxZDMQd vx2i5awW/++vbLUoot+utu9TQQ5JNtrDBKtIwSyjRWg4RdUANnuKR8chbSM29HOZbU7R 2EgIOVnlr7M4tL+rPxy+eRZTUNp4MEKGWDqysO48rfIRzCmWvp6cfc8OZwlVMEJRvqp3 6duHHOPDQ/2BdZbSKpii/CdnyFq02KTr7kTHhOk1HFWkK8nZlQq84CttP9weQTq0kM4k 2QsuS81IRgb6HcbfL9TpU8w7qj2HPdqdoT7ZKOYIWuPpwb0/A6QGYf1PoZRbTcg2/Is3 7G3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=8awC9I57ul4Ehs65t7/5PNP0ULl1lssVMkwIyYI5A4o=; b=S0+2xZ7zQ/HnGKbEaexCdR52RxAYsp4xyvzql+AWECVrTI3lHiWqAh4Aocc1tFG89X QnvDUOzmwGb1uDVfeRqL/Ck9K9hXbbqfdXXQgvS4enq8OKym08jB+6Tim1yyZjqD+HGa 0n0IwwOZsgCyDaMlFJNUi/sws85OxRsJgQ5xUce5Pht242vfr2dDy3ED2n1Q5ZutJ6zS jpDemnBRXjZ1cy7nfAUt5brqGgYWXQysuI+tHN+lfrX6fYkymxFZvg3NiB0EwbfTDhwm TOaeUWQvWDMJ0feDPTlz4t+fHofpLzNrqfWn/eJrbtFJIPCXtkh7+LNgEHFMJJmRHkBr xQ6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=d7bdApJp; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id x3sor1094210qkf.22.2019.06.13.18.04.03 for (Google Transport Security); Thu, 13 Jun 2019 18:04:03 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=d7bdApJp; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8awC9I57ul4Ehs65t7/5PNP0ULl1lssVMkwIyYI5A4o=; b=d7bdApJpYLjcmtg3xYRxybVOfz2KS93AC0fzZ1fW4XyHVqwL3OM+gFQej1Ln9qn0DE xz1SgRMDXyVjAJ/iauLJS09acThOE7zN7n1V63w1pV4dNLuGqG3b8zAuTr/PFa4b30o3 fbxyk9MLPAGnMTt8pryVLuPSPpy6jxJkRiEUy1WphR9OOkegUcdTJ5BDKJnvCM2Hh4yW n+3I8FhRNhZAxdrmmWxLdwqvz0zePJfQiNoLHWaFUtjNaqK8MbBX+kIYlUs31nGnJC6k xIJaEsma2lTn79hWn1/ETECrmvNZum8sX2V88MjB1tGUFrF/ZbWZmNodexZOS8frzOdP 2fhg== X-Google-Smtp-Source: APXvYqxXIo3h4su+d4IRSr2dLrSynk5yL3SLa1bT6tdawPdbOlK3JH39LvD6Y28mBndOcMefOVTFTQ== X-Received: by 2002:a05:620a:5ad:: with SMTP id q13mr19545434qkq.154.1560474243703; Thu, 13 Jun 2019 18:04:03 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id g53sm699466qtk.65.2019.06.13.18.04.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 18:04:03 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKs-0005KW-2y; Thu, 13 Jun 2019 21:44:54 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Philip Yang Subject: [PATCH v3 hmm 11/12] mm/hmm: Remove confusing comment and logic from hmm_release Date: Thu, 13 Jun 2019 21:44:49 -0300 Message-Id: <20190614004450.20252-12-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe hmm_release() is called exactly once per hmm. ops->release() cannot accidentally trigger any action that would recurse back onto hmm->mirrors_sem. This fixes a use after-free race of the form: CPU0 CPU1 hmm_release() up_write(&hmm->mirrors_sem); hmm_mirror_unregister(mirror) down_write(&hmm->mirrors_sem); up_write(&hmm->mirrors_sem); kfree(mirror) mirror->ops->release(mirror) The only user we have today for ops->release is an empty function, so this is unambiguously safe. As a consequence of plugging this race drivers are not allowed to register/unregister mirrors from within a release op. Signed-off-by: Jason Gunthorpe Tested-by: Philip Yang --- mm/hmm.c | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 26af511cbdd075..c0d43302fd6b2f 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -137,26 +137,16 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) WARN_ON(!list_empty(&hmm->ranges)); mutex_unlock(&hmm->lock); - down_write(&hmm->mirrors_sem); - mirror = list_first_entry_or_null(&hmm->mirrors, struct hmm_mirror, - list); - while (mirror) { - list_del_init(&mirror->list); - if (mirror->ops->release) { - /* - * Drop mirrors_sem so the release callback can wait - * on any pending work that might itself trigger a - * mmu_notifier callback and thus would deadlock with - * us. - */ - up_write(&hmm->mirrors_sem); + down_read(&hmm->mirrors_sem); + list_for_each_entry(mirror, &hmm->mirrors, list) { + /* + * Note: The driver is not allowed to trigger + * hmm_mirror_unregister() from this thread. + */ + if (mirror->ops->release) mirror->ops->release(mirror); - down_write(&hmm->mirrors_sem); - } - mirror = list_first_entry_or_null(&hmm->mirrors, - struct hmm_mirror, list); } - up_write(&hmm->mirrors_sem); + up_read(&hmm->mirrors_sem); hmm_put(hmm); } @@ -286,7 +276,7 @@ void hmm_mirror_unregister(struct hmm_mirror *mirror) struct hmm *hmm = mirror->hmm; down_write(&hmm->mirrors_sem); - list_del_init(&mirror->list); + list_del(&mirror->list); up_write(&hmm->mirrors_sem); hmm_put(hmm); memset(&mirror->hmm, POISON_INUSE, sizeof(mirror->hmm)); From patchwork Fri Jun 14 00:44:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10993983 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9898B14BB for ; Fri, 14 Jun 2019 00:45:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E5C91FEBA for ; Fri, 14 Jun 2019 00:45:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 821D726224; Fri, 14 Jun 2019 00:45:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CCE051FEBA for ; Fri, 14 Jun 2019 00:45:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32A756B0273; Thu, 13 Jun 2019 20:45:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1F0BE6B0274; Thu, 13 Jun 2019 20:45:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 065FF6B0275; Thu, 13 Jun 2019 20:45:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id CE5DC6B0273 for ; Thu, 13 Jun 2019 20:45:00 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id c4so639207qkd.16 for ; Thu, 13 Jun 2019 17:45:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=81jT1Hdc80WGYZ0NolRfY9wVErFdmf0CnSOj/bK3nzo=; b=UPq1okH3V47Sy+JwyRhmDq9Rouw1kTtrWII9HyXsN3jM/GL3v6R7TAXLz/9MDcJlUU mfScxiuuGMgsXS9QBTHTLzUUBP/we+wNnv7oMzEm3w0MOvgYTCx3SQMXLF7aeDWW6BXh Y5/0S79EAgn1bZPKExJDsP9bcZEH1n1nQuO5vWZu6e2egRsY0j+NARBebJ1EL7Nmol2R YagTVKxB3Trkjbf0fgd2st7g7AgzZ+hPx0/CK5viA/XHaIBwG4U8KFZ+4xZHEA6yDBO0 MyQuZN+xFj+4WoVl/ZAbHomYJ4/XyyQsjR3lMfAp+I0vqMlS/N5GXGM+YN+ap7y/Yeo9 VYcw== X-Gm-Message-State: APjAAAXBSfLJf/UdkS/ly+n3Qk0+RYra99Sj1o1/anDWktRwa9zcX7T4 i52KT9IOft/IothDWSEJjq8z4CZwbSgPKcPOAK63vCZUWqynJMDKUadZOt0t88Wc0n3jFeMoGRl ScpOYwnrlO/4O02AFQqiQECZgwYGsEL81Yk/pBmAVrO6+Rkyuix7dc9glV8inVpUr/A== X-Received: by 2002:a37:7c05:: with SMTP id x5mr71159457qkc.313.1560473100604; Thu, 13 Jun 2019 17:45:00 -0700 (PDT) X-Received: by 2002:a37:7c05:: with SMTP id x5mr71159424qkc.313.1560473099858; Thu, 13 Jun 2019 17:44:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560473099; cv=none; d=google.com; s=arc-20160816; b=UTJ5c66E2UndTql+guJQ/VtZ1H3vkhLfY2bOXusINFm6nck5gXFsj+i/+5i88NRHlJ CJBbo+gqNyrXpFhR4MsuvlqB52NZZ5zmgYBx5kov9V6jGMYyCMMMguu7fx1ZVHu8NJYc 09hCZEdxjNpYFbrIX/l+2EN2ePuq3joAzbngMQEItNRKqENei6uJ/usqfo+0UU+IkbLr aHgsq87wqJnSrKEfBtM2G/NXH3FeNV4fKCYdBg9phGTHcwgtyw1Qk56/ZLO79/qMOPPx z4ViiF/ui4Vh8dCEbLpoRek9Jo2TwTlt7ApKji+wxnONGuqP/TNg5QlCkBNXHb0bMBxW HZkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=81jT1Hdc80WGYZ0NolRfY9wVErFdmf0CnSOj/bK3nzo=; b=Bty+75oPq22v+9gQASff8/fLwuTUz40eUmkZA8QEYC9OSIsLrsCIfBtXovhh6PNElf gxUEe0VLKhoiPz+0mSBWc9/7ngBpNE9DDmZAi3/7ipHgOWHGxY7qr6GW/j2Zh9C7eb3h QtpXQwyZE7oulf3KB2s5QdEbeSpbHaI2wRO3t0ARI6X2jMcSBmrk/ZfAoB2Wv/j9J9lu hMhhIetldcAur3YGBi5O3n3IXO51eub10QKAp6j+9o1FKm8AiM6my/1MttUCASNcALuQ 0OoUYNjPg9L5f87meElyrwesPDlAHv22i24U5Pamy/QfSxPFo7/2HgJ+JtSiUU6sMgtL 1eVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=kJzXO444; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id w1sor1009267qvf.73.2019.06.13.17.44.59 for (Google Transport Security); Thu, 13 Jun 2019 17:44:59 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=kJzXO444; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=81jT1Hdc80WGYZ0NolRfY9wVErFdmf0CnSOj/bK3nzo=; b=kJzXO444RBh00bh7hl3AxI0tIXExReIacqR5hmZGVB/6O2TxYzqutiFQfsr4MDUAv8 wCBoArV0Ho5b23f6oN/r3cbv5i9u9XYGWevXpsUuafVa4rMh8Li+8TSsMcLyTYxfzKNd iMm+0vRb+qSK6K4qKnf3QAfbRS8WkHLXwRL82O0/+MxT69BHfBer1KbhPSo7UAdtHxtl Quupwe3y5+dEUrrEHpCsVgVJCiyZ35e+8ARRZsziW+azHgQiqcHz2E/7IPNRpVG3unMJ cddZbQk+ZL6uk83JsBLSVP50bQsfaNZatZde6LJ9+w+IwAMgEIIss93Hv8qv8WHSIJoS AKGg== X-Google-Smtp-Source: APXvYqzrXFiUSgzqu3ffxXhDqird+PVtZh0YXGYvXI5ZHYMZM5dV6GcQ4Vg9uZ1+KYqYHu1zRMYt4w== X-Received: by 2002:a0c:bd9a:: with SMTP id n26mr5905854qvg.25.1560473099585; Thu, 13 Jun 2019 17:44:59 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-55-100.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.55.100]) by smtp.gmail.com with ESMTPSA id r5sm756136qkc.42.2019.06.13.17.44.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jun 2019 17:44:54 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hbaKs-0005Kc-4A; Thu, 13 Jun 2019 21:44:54 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Jason Gunthorpe , Philip Yang Subject: [PATCH v3 hmm 12/12] mm/hmm: Fix error flows in hmm_invalidate_range_start Date: Thu, 13 Jun 2019 21:44:50 -0300 Message-Id: <20190614004450.20252-13-jgg@ziepe.ca> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190614004450.20252-1-jgg@ziepe.ca> References: <20190614004450.20252-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe If the trylock on the hmm->mirrors_sem fails the function will return without decrementing the notifiers that were previously incremented. Since the caller will not call invalidate_range_end() on EAGAIN this will result in notifiers becoming permanently incremented and deadlock. If the sync_cpu_device_pagetables() required blocking the function will not return EAGAIN even though the device continues to touch the pages. This is a violation of the mmu notifier contract. Switch, and rename, the ranges_lock to a spin lock so we can reliably obtain it without blocking during error unwind. The error unwind is necessary since the notifiers count must be held incremented across the call to sync_cpu_device_pagetables() as we cannot allow the range to become marked valid by a parallel invalidate_start/end() pair while doing sync_cpu_device_pagetables(). Signed-off-by: Jason Gunthorpe Reviewed-by: Ralph Campbell Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- include/linux/hmm.h | 2 +- mm/hmm.c | 77 +++++++++++++++++++++++++++------------------ 2 files changed, 48 insertions(+), 31 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index bf013e96525771..0fa8ea34ccef6d 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -86,7 +86,7 @@ struct hmm { struct mm_struct *mm; struct kref kref; - struct mutex lock; + spinlock_t ranges_lock; struct list_head ranges; struct list_head mirrors; struct mmu_notifier mmu_notifier; diff --git a/mm/hmm.c b/mm/hmm.c index c0d43302fd6b2f..1172a4f0206963 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -67,7 +67,7 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) init_rwsem(&hmm->mirrors_sem); hmm->mmu_notifier.ops = NULL; INIT_LIST_HEAD(&hmm->ranges); - mutex_init(&hmm->lock); + spin_lock_init(&hmm->ranges_lock); kref_init(&hmm->kref); hmm->notifiers = 0; hmm->mm = mm; @@ -124,18 +124,19 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); struct hmm_mirror *mirror; + unsigned long flags; /* Bail out if hmm is in the process of being freed */ if (!kref_get_unless_zero(&hmm->kref)) return; - mutex_lock(&hmm->lock); + spin_lock_irqsave(&hmm->ranges_lock, flags); /* * Since hmm_range_register() holds the mmget() lock hmm_release() is * prevented as long as a range exists. */ WARN_ON(!list_empty(&hmm->ranges)); - mutex_unlock(&hmm->lock); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); down_read(&hmm->mirrors_sem); list_for_each_entry(mirror, &hmm->mirrors, list) { @@ -151,6 +152,23 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) hmm_put(hmm); } +static void notifiers_decrement(struct hmm *hmm) +{ + lockdep_assert_held(&hmm->ranges_lock); + + hmm->notifiers--; + if (!hmm->notifiers) { + struct hmm_range *range; + + list_for_each_entry(range, &hmm->ranges, list) { + if (range->valid) + continue; + range->valid = true; + } + wake_up_all(&hmm->wq); + } +} + static int hmm_invalidate_range_start(struct mmu_notifier *mn, const struct mmu_notifier_range *nrange) { @@ -158,6 +176,7 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, struct hmm_mirror *mirror; struct hmm_update update; struct hmm_range *range; + unsigned long flags; int ret = 0; if (!kref_get_unless_zero(&hmm->kref)) @@ -168,12 +187,7 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, update.event = HMM_UPDATE_INVALIDATE; update.blockable = mmu_notifier_range_blockable(nrange); - if (mmu_notifier_range_blockable(nrange)) - mutex_lock(&hmm->lock); - else if (!mutex_trylock(&hmm->lock)) { - ret = -EAGAIN; - goto out; - } + spin_lock_irqsave(&hmm->ranges_lock, flags); hmm->notifiers++; list_for_each_entry(range, &hmm->ranges, list) { if (update.end < range->start || update.start >= range->end) @@ -181,7 +195,7 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, range->valid = false; } - mutex_unlock(&hmm->lock); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); if (mmu_notifier_range_blockable(nrange)) down_read(&hmm->mirrors_sem); @@ -189,16 +203,26 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, ret = -EAGAIN; goto out; } + list_for_each_entry(mirror, &hmm->mirrors, list) { - int ret; + int rc; - ret = mirror->ops->sync_cpu_device_pagetables(mirror, &update); - if (!update.blockable && ret == -EAGAIN) + rc = mirror->ops->sync_cpu_device_pagetables(mirror, &update); + if (rc) { + if (WARN_ON(update.blockable || rc != -EAGAIN)) + continue; + ret = -EAGAIN; break; + } } up_read(&hmm->mirrors_sem); out: + if (ret) { + spin_lock_irqsave(&hmm->ranges_lock, flags); + notifiers_decrement(hmm); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); + } hmm_put(hmm); return ret; } @@ -207,23 +231,14 @@ static void hmm_invalidate_range_end(struct mmu_notifier *mn, const struct mmu_notifier_range *nrange) { struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); + unsigned long flags; if (!kref_get_unless_zero(&hmm->kref)) return; - mutex_lock(&hmm->lock); - hmm->notifiers--; - if (!hmm->notifiers) { - struct hmm_range *range; - - list_for_each_entry(range, &hmm->ranges, list) { - if (range->valid) - continue; - range->valid = true; - } - wake_up_all(&hmm->wq); - } - mutex_unlock(&hmm->lock); + spin_lock_irqsave(&hmm->ranges_lock, flags); + notifiers_decrement(hmm); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); hmm_put(hmm); } @@ -876,6 +891,7 @@ int hmm_range_register(struct hmm_range *range, { unsigned long mask = ((1UL << page_shift) - 1UL); struct hmm *hmm = mirror->hmm; + unsigned long flags; range->valid = false; range->hmm = NULL; @@ -894,7 +910,7 @@ int hmm_range_register(struct hmm_range *range, return -EFAULT; /* Initialize range to track CPU page table updates. */ - mutex_lock(&hmm->lock); + spin_lock_irqsave(&hmm->ranges_lock, flags); range->hmm = hmm; kref_get(&hmm->kref); @@ -906,7 +922,7 @@ int hmm_range_register(struct hmm_range *range, */ if (!hmm->notifiers) range->valid = true; - mutex_unlock(&hmm->lock); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); return 0; } @@ -922,10 +938,11 @@ EXPORT_SYMBOL(hmm_range_register); void hmm_range_unregister(struct hmm_range *range) { struct hmm *hmm = range->hmm; + unsigned long flags; - mutex_lock(&hmm->lock); + spin_lock_irqsave(&hmm->ranges_lock, flags); list_del(&range->list); - mutex_unlock(&hmm->lock); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); /* Drop reference taken by hmm_range_register() */ mmput(hmm->mm);