From patchwork Mon Jun 24 21:00:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014141 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A4D3614E5 for ; Mon, 24 Jun 2019 21:02:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 97110288D8 for ; Mon, 24 Jun 2019 21:02:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B07528910; Mon, 24 Jun 2019 21:02:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 08959288D8 for ; Mon, 24 Jun 2019 21:02:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 701B28E000B; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5E8FA8E0009; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43BEC8E0002; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by kanga.kvack.org (Postfix) with ESMTP id EC7B18E0009 for ; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) Received: by mail-wm1-f71.google.com with SMTP id c190so68070wme.8 for ; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=hG9eEbhO9ZvVI+JAJ4Kr5jKBUZBlX9y3IyF9CExVuM8=; b=BBaowxP7AnSF33PF5fvcTZr5iinA4QdWFKfvNwjFsGai2rAoj6702zcUSEERhUIItJ 4VMnusHREgqy9EGGVs/BT9QfSFrHmteJMgbhN9+vavrYPP8uLicH0Qct92dymd7rhV4O rb9PXC3KNgmu37M5Fm8/ptzmpTRg+N3LWBUheH2o4kM9NXrd89KAH4Mt1IOcEUrGqJWM +Da8+ZrKfrlb0fiDuJbZZJXeDrhBd52V3wr9suk+ykLuA/Pt1uEaarhPcxZaIopjXdTv 77eTfkGPC51ymHnsZKBOM4Sjm6mF4M8QlG49paN/5+b5OhOvP4ClNr/s0+XGZz1dVoq0 ONLA== X-Gm-Message-State: APjAAAU9wz8PrU96JRpDkjhK3ANaNEE+swtG3/8xh2R/qN/i4UfOmJeE /xnznvAZmmnPLPQp8Jsq4QT44f+xC17nF7DxQ7UPEnAmXE6kWivm+y41SrQYMOkpaW656OHddfK JesTDZSNWokG2NIGrxBat97tdRO/FgUtrJKM6k6Ey2yIq3Q8MFoAXFUc/PoJGQqWWrg== X-Received: by 2002:a1c:a842:: with SMTP id r63mr17029916wme.117.1561410127456; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) X-Received: by 2002:a1c:a842:: with SMTP id r63mr17029877wme.117.1561410126497; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410126; cv=none; d=google.com; s=arc-20160816; b=EulV0Tu9SmSRIVR/9IMocSWbzn4lLHVODaR3Dlk8yQBkrgg1lLVrdOhO5PJzjXXD5G 2n4EzpknNdEUoLgQZ30O+OM/dDyyo2Wb9POGR/G+3p6qAzS0ZF6KbLStTmOCdKB0Fz67 JkKER71KJWXRLl4c8FXVjTZ9K8BeIeInL4LTePmrfjOs+Fso1jPQWkESsCxVQX001WxD BuYfQWs2RXUfhJwUfDAorhp0QZXMJEZu5riuo6tW0F0QqdEs4UbgzrHMZKnt+KnY5gQR QjZuFAlDoVdL4LGq77YqpzEY23eTP3/5iUWJiceQaQ7nv1l/6ic+U8n1M2yf5whgy+MZ 5MRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=hG9eEbhO9ZvVI+JAJ4Kr5jKBUZBlX9y3IyF9CExVuM8=; b=Ypf58MvyPaN54UdYdL1U6f8FIYjhW9neip4Y6LIw54Skx1qTGHTrKyl0tloFrx+nsZ G8gC0dJvqUso+nmfXKCZ01AKeIsUSehBIsXURmBVvCjNyGJptI8KuYY7MF4IL1jRgFVn sshMPCHVBWneCEnZTdQgNsiwREkIkkj5uj9D2xGYB1mhXdZ4PPIBheUwydwLNjadAtSV eD/eDuk7L0XigtwHGLVXZ5MIwRuRGayGEMe33Ke8mOKs1OwbSGAJXL4glaL/pBrY9tGS fxQr9WhtLIah++BEhjjzg89xYU4QZFNh8IiiuKe/JfXBvS8oBJR1gmvDUeIuAz5xtYvS LkfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=Dfnwj7SR; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id u17sor7241966wrr.4.2019.06.24.14.02.06 for (Google Transport Security); Mon, 24 Jun 2019 14:02:06 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=Dfnwj7SR; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hG9eEbhO9ZvVI+JAJ4Kr5jKBUZBlX9y3IyF9CExVuM8=; b=Dfnwj7SRwV2PMmn2U0d+WODdclNP+8MKrJJpQxHP3JU1uHDlj7Em+GRbnOVlGf37aZ x/iFL81waqsH2R/ZDnogOEsWqssPFpPx6ng2iVzd42Ph8pxRGkGdeHnQFn7WXHxychn7 qhcCnDDKKA2XJlL0t9jgEmXK5iD+LSA9ISspJalIOTaGOUy2P3iEqUcWHAlRfr10A2Zi sVektpaLu8hOSGiRgUg+86mxYZntLzr5JZQnXIkn15UuhOjzFImdCeWJqoAaOquAmGLI hpzcweH8Q2CBVObvgPXPT94/km2oYbyVknkNINwL80SwNwAFFuHpdFRRunzkNVq5yivf 2PaQ== X-Google-Smtp-Source: APXvYqxAEk4qQsJeqyqINse+yHJkRouINtL91JL0sEIFqMM29cMWXWeCvzOgl6HwO/QEFlzvN+y93Q== X-Received: by 2002:adf:f186:: with SMTP id h6mr21080937wro.274.1561410126128; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id b71sm446129wmb.7.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6C-0001M2-OO; Mon, 24 Jun 2019 18:02:00 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 01/12] mm/hmm: fix use after free with struct hmm in the mmu notifiers Date: Mon, 24 Jun 2019 18:00:59 -0300 Message-Id: <20190624210110.5098-2-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe mmu_notifier_unregister_no_release() is not a fence and the mmu_notifier system will continue to reference hmm->mn until the srcu grace period expires. Resulting in use after free races like this: CPU0 CPU1 __mmu_notifier_invalidate_range_start() srcu_read_lock hlist_for_each () // mn == hmm->mn hmm_mirror_unregister() hmm_put() hmm_free() mmu_notifier_unregister_no_release() hlist_del_init_rcu(hmm-mn->list) mn->ops->invalidate_range_start(mn, range); mm_get_hmm() mm->hmm = NULL; kfree(hmm) mutex_lock(&hmm->lock); Use SRCU to kfree the hmm memory so that the notifiers can rely on hmm existing. Get the now-safe hmm struct through container_of and directly check kref_get_unless_zero to lock it against free. Signed-off-by: Jason Gunthorpe Reviewed-by: Ira Weiny Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- v2: - Spell 'free' properly (Jerome/Ralph) v3: - Have only one clearer comment about kref_get_unless_zero (John) --- include/linux/hmm.h | 1 + mm/hmm.c | 23 +++++++++++++++++------ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 7007123842ba76..cb01cf1fa3c08b 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -93,6 +93,7 @@ struct hmm { struct mmu_notifier mmu_notifier; struct rw_semaphore mirrors_sem; wait_queue_head_t wq; + struct rcu_head rcu; long notifiers; bool dead; }; diff --git a/mm/hmm.c b/mm/hmm.c index 826816ab237799..f6956d78e3cb25 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -104,6 +104,11 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) return NULL; } +static void hmm_free_rcu(struct rcu_head *rcu) +{ + kfree(container_of(rcu, struct hmm, rcu)); +} + static void hmm_free(struct kref *kref) { struct hmm *hmm = container_of(kref, struct hmm, kref); @@ -116,7 +121,7 @@ static void hmm_free(struct kref *kref) mm->hmm = NULL; spin_unlock(&mm->page_table_lock); - kfree(hmm); + mmu_notifier_call_srcu(&hmm->rcu, hmm_free_rcu); } static inline void hmm_put(struct hmm *hmm) @@ -144,10 +149,14 @@ void hmm_mm_destroy(struct mm_struct *mm) static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) { - struct hmm *hmm = mm_get_hmm(mm); + struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); struct hmm_mirror *mirror; struct hmm_range *range; + /* Bail out if hmm is in the process of being freed */ + if (!kref_get_unless_zero(&hmm->kref)) + return; + /* Report this HMM as dying. */ hmm->dead = true; @@ -185,13 +194,14 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) static int hmm_invalidate_range_start(struct mmu_notifier *mn, const struct mmu_notifier_range *nrange) { - struct hmm *hmm = mm_get_hmm(nrange->mm); + struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); struct hmm_mirror *mirror; struct hmm_update update; struct hmm_range *range; int ret = 0; - VM_BUG_ON(!hmm); + if (!kref_get_unless_zero(&hmm->kref)) + return 0; update.start = nrange->start; update.end = nrange->end; @@ -236,9 +246,10 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, static void hmm_invalidate_range_end(struct mmu_notifier *mn, const struct mmu_notifier_range *nrange) { - struct hmm *hmm = mm_get_hmm(nrange->mm); + struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); - VM_BUG_ON(!hmm); + if (!kref_get_unless_zero(&hmm->kref)) + return; mutex_lock(&hmm->lock); hmm->notifiers--; From patchwork Mon Jun 24 21:01:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014133 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9FEF014E5 for ; Mon, 24 Jun 2019 21:02:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 92022288D8 for ; Mon, 24 Jun 2019 21:02:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 857A628910; Mon, 24 Jun 2019 21:02:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 08FD7288D8 for ; Mon, 24 Jun 2019 21:02:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47CFD8E0003; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 329548E0008; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06AEB8E0003; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by kanga.kvack.org (Postfix) with ESMTP id 9BE908E0002 for ; Mon, 24 Jun 2019 17:02:06 -0400 (EDT) Received: by mail-wr1-f69.google.com with SMTP id t4so6896938wrs.10 for ; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bvrLwS8goK61NFHHKG+L59ElQQ6a+tjA3TuLNO1hols=; b=Eb6g2yx5+S9E7n/HWlhwAmBpgiG/YdXjyTK0DT8ki8ciFk4bv/6cp06bUktf3Kk49r PMSaAVj8P+ZDN/BBijr1fuUIa/JZUFLLvK4gQ/YVzoNr77ZLHr9kN0C3cQlafl5f6Ppa afnyO161BsOG0iDTwHrkSO3blNGET9j8NZpZSqDf1TBwNF+DkBe+dr7Fr4EjDrJuwTzi h7nV3eqTBnP874ELXcAmY9nX8lkYHoZatoL7dsXZplwMVWpUc/W7r8Zvv74A4FjqgyaH FT1u3Ea15Sn7aGWMjelIWq0Ovvv8II2kWly4xK54bzRJo6jtZ2zHtk5P9wicECFg65+R ekvg== X-Gm-Message-State: APjAAAXUxxPdTZBOql9eBtloP6UX8Duk5/4z8tuolQtKOzaJf7QKMnme BK1QeH2CSkYZOHvsJ0s/hnmaTxlZIxtVonjQiINTNWNLUyTbQwFtVPF6RoovLnDr/42s1+Gg8i8 R+h0WyaHUFmfCFOzkyNofJEyNt5CMA3aQ3GTsBd2RsEYnLasoJBi1YTYH9/0LOWWqeQ== X-Received: by 2002:a1c:305:: with SMTP id 5mr17857927wmd.101.1561410126186; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) X-Received: by 2002:a1c:305:: with SMTP id 5mr17857891wmd.101.1561410125233; Mon, 24 Jun 2019 14:02:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410125; cv=none; d=google.com; s=arc-20160816; b=kUEr9PZ8hHH4KHm1DU1zMxZfykdYaAzB0+gozmjZ3hOEOq3t6MBOC9oSSuoQLXz1iz LA46Kz5NJcgfHHPKiaWj3gmIClWDQf4C5dlx4sJUvrgHhthaBaaVETxH5rrgWtLh3Zi2 s3q8ncleAcu7AzgEUp0sPSD4259Koi3keQCN9HqWv0iWbdJY7gV042vlUx9Y+x9fyF9A FHryaCrJhFMLi7D09sQr4t+/mRNfRMyJBrW5UaenIzV76aWno1+xfe1Ft9DZZF2ffz8z fT4rcU1qb/Gy99xWjgAIqHrj6g60PB6M7WT5MGaWFnUvUJwb9FDuP38rlfu+swKNWwlO 0lSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=bvrLwS8goK61NFHHKG+L59ElQQ6a+tjA3TuLNO1hols=; b=QUEB+T5V/7/sZwu6/v+yDTHRCjlBvqKK9LWS/+kdNmWL4MI61nLLViFV/7xHBc/0Fi bwAZTv29GnIZBJqGujZc5PLRAVH7i8TlnUoNRzS9dcszJSm2PMCqHeyolMRvf5/gqHh7 mDlxXdHfTeRv9sC2ruWLtplf444CPHpRdMyQTdrI+pC7jLywV34LsHF/U7V518l4cTUl 4P7ijFi+03rtoY7aXORrqgh5dq97rgDu0+SxXXpmqNd54OC/2K11n7KHgsYwwlOHN1VQ ODIw/SRnjKp5hd+iShUh9+JLAfTiGdiyM2IwJWpnP5dVvF5PksBYGNk3dB8pL+0CwlIS fBJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=eVFuqGal; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id g8sor7213085wrp.5.2019.06.24.14.02.05 for (Google Transport Security); Mon, 24 Jun 2019 14:02:05 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=eVFuqGal; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bvrLwS8goK61NFHHKG+L59ElQQ6a+tjA3TuLNO1hols=; b=eVFuqGal7MmrKBtKN1rp7PO9U9nA5Z3G5ySp7ACdyjU8KPVJwiMhZOX3ijBDCcRL1S GTWW6OsNpDRwAQ99IpOKbjE4ke8ZJxp+O8FP/MG8xEtB6bNNyhoddDXqx09MksC3ucTO oJ93yVe3SvCZh0B9cLyVmwFyg4NWK+yvtmHIR5Ht19fKXdc1deBlAadM2MfQOo7DpfxL u45+uLrEWPgz5VW+SYVHONBoIIEBy3hHaaX9UXvxqchVA5MxvXU51AT2KJdXKstwNhpK HS8SeD4+RAjEkcCeH/9NHi+z2QOq0pKZI8APHroOaypyHDbREEEL9MVSPuU3viLHbv3C +MFg== X-Google-Smtp-Source: APXvYqzXxflttImXsUfCwAd7r65RK1DVJZBGVN7mipcY4Irf1EG1vjNhQIrTK0Km8IeO1SnCpwVb8g== X-Received: by 2002:adf:b1ca:: with SMTP id r10mr34168985wra.156.1561410124460; Mon, 24 Jun 2019 14:02:04 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id n14sm26883973wra.75.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6C-0001M7-PX; Mon, 24 Jun 2019 18:02:00 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 02/12] mm/hmm: Use hmm_mirror not mm as an argument for hmm_range_register Date: Mon, 24 Jun 2019 18:01:00 -0300 Message-Id: <20190624210110.5098-3-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe Ralph observes that hmm_range_register() can only be called by a driver while a mirror is registered. Make this clear in the API by passing in the mirror structure as a parameter. This also simplifies understanding the lifetime model for struct hmm, as the hmm pointer must be valid as part of a registered mirror so all we need in hmm_register_range() is a simple kref_get. Suggested-by: Ralph Campbell Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- v2 - Include the oneline patch to nouveau_svm.c --- drivers/gpu/drm/nouveau/nouveau_svm.c | 2 +- include/linux/hmm.h | 7 ++++--- mm/hmm.c | 13 ++++--------- 3 files changed, 9 insertions(+), 13 deletions(-) diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c index 93ed43c413f0bb..8c92374afcf227 100644 --- a/drivers/gpu/drm/nouveau/nouveau_svm.c +++ b/drivers/gpu/drm/nouveau/nouveau_svm.c @@ -649,7 +649,7 @@ nouveau_svm_fault(struct nvif_notify *notify) range.values = nouveau_svm_pfn_values; range.pfn_shift = NVIF_VMM_PFNMAP_V0_ADDR_SHIFT; again: - ret = hmm_vma_fault(&range, true); + ret = hmm_vma_fault(&svmm->mirror, &range, true); if (ret == 0) { mutex_lock(&svmm->mutex); if (!hmm_vma_range_done(&range)) { diff --git a/include/linux/hmm.h b/include/linux/hmm.h index cb01cf1fa3c08b..1fba6979adf460 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -496,7 +496,7 @@ static inline bool hmm_mirror_mm_is_alive(struct hmm_mirror *mirror) * Please see Documentation/vm/hmm.rst for how to use the range API. */ int hmm_range_register(struct hmm_range *range, - struct mm_struct *mm, + struct hmm_mirror *mirror, unsigned long start, unsigned long end, unsigned page_shift); @@ -532,7 +532,8 @@ static inline bool hmm_vma_range_done(struct hmm_range *range) } /* This is a temporary helper to avoid merge conflict between trees. */ -static inline int hmm_vma_fault(struct hmm_range *range, bool block) +static inline int hmm_vma_fault(struct hmm_mirror *mirror, + struct hmm_range *range, bool block) { long ret; @@ -545,7 +546,7 @@ static inline int hmm_vma_fault(struct hmm_range *range, bool block) range->default_flags = 0; range->pfn_flags_mask = -1UL; - ret = hmm_range_register(range, range->vma->vm_mm, + ret = hmm_range_register(range, mirror, range->start, range->end, PAGE_SHIFT); if (ret) diff --git a/mm/hmm.c b/mm/hmm.c index f6956d78e3cb25..22a97ada108b4e 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -914,13 +914,13 @@ static void hmm_pfns_clear(struct hmm_range *range, * Track updates to the CPU page table see include/linux/hmm.h */ int hmm_range_register(struct hmm_range *range, - struct mm_struct *mm, + struct hmm_mirror *mirror, unsigned long start, unsigned long end, unsigned page_shift) { unsigned long mask = ((1UL << page_shift) - 1UL); - struct hmm *hmm; + struct hmm *hmm = mirror->hmm; range->valid = false; range->hmm = NULL; @@ -934,20 +934,15 @@ int hmm_range_register(struct hmm_range *range, range->start = start; range->end = end; - hmm = hmm_get_or_create(mm); - if (!hmm) - return -EFAULT; - /* Check if hmm_mm_destroy() was call. */ - if (hmm->mm == NULL || hmm->dead) { - hmm_put(hmm); + if (hmm->mm == NULL || hmm->dead) return -EFAULT; - } /* Initialize range to track CPU page table updates. */ mutex_lock(&hmm->lock); range->hmm = hmm; + kref_get(&hmm->kref); list_add_rcu(&range->list, &hmm->ranges); /* From patchwork Mon Jun 24 21:01:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014131 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DB28E112C for ; Mon, 24 Jun 2019 21:02:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CCE8D288D8 for ; Mon, 24 Jun 2019 21:02:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C04E228910; Mon, 24 Jun 2019 21:02:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 45657288D8 for ; Mon, 24 Jun 2019 21:02:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23EBA8E0005; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 156328E0007; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F135F8E0006; Mon, 24 Jun 2019 17:02:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by kanga.kvack.org (Postfix) with ESMTP id A2E928E0003 for ; Mon, 24 Jun 2019 17:02:06 -0400 (EDT) Received: by mail-wr1-f71.google.com with SMTP id b14so6892206wrn.8 for ; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=MJQSyx5I9WQIuat3hBB7cHn44HQ02+T6fWrOLMHDIpE=; b=hLgcMB0gVuKIEZe/VllO3Qgu6ooGYO/qSNqWS+kTg97tQnkJpzxTS5MWoIvxvkRu9j ER1DDUq1EOvibCym3Guc6qv7jgvIIzkJ8RVDWIf55uu/rnFbUpbH2ReYb+C6A41QT+88 p0lYuGkIoRehrrmKQLmwXvmblPbPoJIqVSUTkDvZi4UtPeiZgEQBPJTJ9SasC6Vf/Ppd d9IeoHRJ35Se0mpDVuJ697RjSd3dKwENDoFHA414i2kMPkCBxqt5jcOsJdXm6rqQ6fqE 74+gWfEvqy7ZL84DZowKdC32OQ226CBbSM0hAdXG0GP7ebe1npp6+s4dMljlH2aKjB6K nvqQ== X-Gm-Message-State: APjAAAU3Dn8p4PX2BRDoDINggDCs8C6z/eivzdueDBhJgnm+K86Mid3Q zc5sL19QNGZrWgoewT5g6CWCb6wLkaKhrd+TV4sqsbpelJ8DM9csilw05sgEDrBHBmx9m/X+Gn0 QP9BsUIBP6wFntCSk3KKLxTpJ9asWA3MyRPwNC42j9MPu0cKGq5BbXWKYAf+kpWzGhQ== X-Received: by 2002:adf:fd11:: with SMTP id e17mr42434333wrr.337.1561410126177; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) X-Received: by 2002:adf:fd11:: with SMTP id e17mr42434297wrr.337.1561410125298; Mon, 24 Jun 2019 14:02:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410125; cv=none; d=google.com; s=arc-20160816; b=g8y+ZeUnTMLHBlzXXgMqhikKXhPShHW/eAta3g7oaoUwX4bs/Xt5nlHd98oXwoQJFZ l6VrM1PzHr7gFYLAP4TTm2nScCVL6/QyffFU+D9xJHHSzEJBRTA+83t8+pijbuxzNkoY x5UzM7bt0y/t0T6WSMpa4MDBtmhJtpCGHnSCVON5zGTvWJoPJBOOmNbVz6MMB69CLWS1 df/dTWfCQnYRjdfiyYwsLl0THDR1mI7ABreeukzO3oao+f5Fu+07F9/LmnMQlJ06Z+PN gBJvk+tNcAcRkXkk9BXsOwhgLuCqnhsCzZekvVV/avTUqiw+jE4vtEnDnl3MdwAuGBRu Ex4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=MJQSyx5I9WQIuat3hBB7cHn44HQ02+T6fWrOLMHDIpE=; b=kdYu74tsSOqYAYRQ9lt1Zpns2Z9BIhhOSeYPCvpJ+aizWdiA0V5hLkpPmpIDGzdd6y hxbsAn0tFQCKUJOIeohWJVzaW5wtIWsyyq7eyyP7EdNe4ppKzcxlYAU8afE9TQcQq8+m tvJFwmWfFTmO5M2EBJx8UWuvc2IThGpdd13cDZMZXv9YtEYyHJLSFk0fDewwW9Wcu7Qn OhJJOtyQ9fgX0f18eP3k8pnNcejYAxGTzTEzgyUS0A2C44OFDtXa5Jsyda96gGV8co1U qbyMVF8E3+qcSJvFqQzDZl8PUz3YmP/1sZNt77yaaeYFfT4IHsGnK3A6P8HoYiwjpBom 5fIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=oMvQEeOW; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c64sor361742wma.17.2019.06.24.14.02.05 for (Google Transport Security); Mon, 24 Jun 2019 14:02:05 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=oMvQEeOW; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MJQSyx5I9WQIuat3hBB7cHn44HQ02+T6fWrOLMHDIpE=; b=oMvQEeOW3jlA/jInbebXvBiNOpipgFRfSGqcRm7zQTy9n0fBZhNWc+HqRdsBGvfoas 9ZagIA6tsA27jF8jJhsuzXhsn1FXmqG0j5xv4shnb4nwTXoKwmM7oJ2k65yF3t5gXBSE xlXz3qHDEErdU0D2pMcseGextk7OdhnOkqInYbmQI1gUO7qrtJE1uX0pyvlz3YlLnknO sqCZVvuYR+lhn6UVq6Z9jO7x/rArDqufw9dwJ5VG1vfgusW9/rg/xQREUYZgy4+U1mqN JQUgexOwqWQrBjbYL4JVl6Alf5y/LhMUNGKdzg0+gQVKP467Kb+hNL89aBTa4xHO/0eq Mg5g== X-Google-Smtp-Source: APXvYqy/o9BJkcVraqWn87goYHtDioMKUZC2UcYj3qjd4Ok68R0RasKDzib4jR1bwJteEjTWcnZR4g== X-Received: by 2002:a05:600c:20c3:: with SMTP id y3mr17777867wmm.3.1561410124814; Mon, 24 Jun 2019 14:02:04 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id l8sm26977546wrg.40.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6C-0001MC-Qe; Mon, 24 Jun 2019 18:02:00 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 03/12] mm/hmm: Hold a mmgrab from hmm to mm Date: Mon, 24 Jun 2019 18:01:01 -0300 Message-Id: <20190624210110.5098-4-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe So long as a struct hmm pointer exists, so should the struct mm it is linked too. Hold the mmgrab() as soon as a hmm is created, and mmdrop() it once the hmm refcount goes to zero. Since mmdrop() (ie a 0 kref on struct mm) is now impossible with a !NULL mm->hmm delete the hmm_hmm_destroy(). Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- v2: - Fix error unwind paths in hmm_get_or_create (Jerome/Jason) --- include/linux/hmm.h | 3 --- kernel/fork.c | 1 - mm/hmm.c | 22 ++++------------------ 3 files changed, 4 insertions(+), 22 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 1fba6979adf460..1d97b6d62c5bcf 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -577,14 +577,11 @@ static inline int hmm_vma_fault(struct hmm_mirror *mirror, } /* Below are for HMM internal use only! Not to be used by device driver! */ -void hmm_mm_destroy(struct mm_struct *mm); - static inline void hmm_mm_init(struct mm_struct *mm) { mm->hmm = NULL; } #else /* IS_ENABLED(CONFIG_HMM_MIRROR) */ -static inline void hmm_mm_destroy(struct mm_struct *mm) {} static inline void hmm_mm_init(struct mm_struct *mm) {} #endif /* IS_ENABLED(CONFIG_HMM_MIRROR) */ diff --git a/kernel/fork.c b/kernel/fork.c index 75675b9bf6dfd3..c704c3cedee78d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -673,7 +673,6 @@ void __mmdrop(struct mm_struct *mm) WARN_ON_ONCE(mm == current->active_mm); mm_free_pgd(mm); destroy_context(mm); - hmm_mm_destroy(mm); mmu_notifier_mm_destroy(mm); check_mm(mm); put_user_ns(mm->user_ns); diff --git a/mm/hmm.c b/mm/hmm.c index 22a97ada108b4e..080b17a2e87e2d 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -73,6 +74,7 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) hmm->notifiers = 0; hmm->dead = false; hmm->mm = mm; + mmgrab(hmm->mm); spin_lock(&mm->page_table_lock); if (!mm->hmm) @@ -100,6 +102,7 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) mm->hmm = NULL; spin_unlock(&mm->page_table_lock); error: + mmdrop(hmm->mm); kfree(hmm); return NULL; } @@ -121,6 +124,7 @@ static void hmm_free(struct kref *kref) mm->hmm = NULL; spin_unlock(&mm->page_table_lock); + mmdrop(hmm->mm); mmu_notifier_call_srcu(&hmm->rcu, hmm_free_rcu); } @@ -129,24 +133,6 @@ static inline void hmm_put(struct hmm *hmm) kref_put(&hmm->kref, hmm_free); } -void hmm_mm_destroy(struct mm_struct *mm) -{ - struct hmm *hmm; - - spin_lock(&mm->page_table_lock); - hmm = mm_get_hmm(mm); - mm->hmm = NULL; - if (hmm) { - hmm->mm = NULL; - hmm->dead = true; - spin_unlock(&mm->page_table_lock); - hmm_put(hmm); - return; - } - - spin_unlock(&mm->page_table_lock); -} - static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); From patchwork Mon Jun 24 21:01:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014139 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A4F7B14E5 for ; Mon, 24 Jun 2019 21:02:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 96109288D8 for ; Mon, 24 Jun 2019 21:02:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8A50A28910; Mon, 24 Jun 2019 21:02:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F753288D8 for ; Mon, 24 Jun 2019 21:02:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3B878E0007; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EC5428E0002; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB1048E0007; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by kanga.kvack.org (Postfix) with ESMTP id 91D608E0002 for ; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) Received: by mail-wr1-f72.google.com with SMTP id b6so6885759wrp.21 for ; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=5QRiWs/DQbjPzNp3Mmcykk+AEBHTNlfm8mEd5w/KsEk=; b=UZwzdgmmtXWmI+7RIsx1ijbxY8slI96URmOwe+N9Geml6Wywtr+s3IhzN3ydELyAha 1Qe8sEXkZduS6SoZoJv10tPudEesIZWFsixvHDnYjw4wdBYaJoeC9hEuwSojtRVeMHoE QH6rMSOzzgy5wShOf20AEhvpAP2wUkYcudHi5/X9AP44idaDGQl6R6k/8W+mZrr6b2iV 2kjK1fBVZ193G6yvlZkxIZLpruaEDnxSbpC+3SaVIUuee0yRlhaWOTI0I4gG/XwEoswm LxMrLEFFVVurQmARgPwzLgqhkJbrFzibTsw1UwjDbA0Ry1C8L65Sq5THgsEaHxQGuyap 27hw== X-Gm-Message-State: APjAAAUMZO/TKhMxT0G0NDGAf3/drZblj3CWM8mtuUh7ZU/Aku75/jtP pK4NgGbQqhduyQKKAVEoAx6a1Nb4jP0iPTU5oCpDromC9y1CKEzaMeArnl+FILI1nK1Gbs0TMTJ UaDFO87FhhZw/vKCeVgf1pUz2Os0VMEOuxY6jbUTCaNJ8yZ4WxN2xErRhw5fZ2bYpyA== X-Received: by 2002:a05:600c:114f:: with SMTP id z15mr16868279wmz.131.1561410127148; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) X-Received: by 2002:a05:600c:114f:: with SMTP id z15mr16868246wmz.131.1561410126202; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410126; cv=none; d=google.com; s=arc-20160816; b=nBtQ2qQhLdQ3jHuLvW3RpYhtZ22Xy2hKYXmbJY1S21/FzD/dOifSwFdNy9ttJX/H8A nEMEW4HFHGmWPemdDJFbS+I+jvCgHO8Z+12tp4YtlhYrMKQoXf3QjCLFRtdRIwWbIs0/ iI0MsCYnsIiJqSMOT/eu5K+GMYHTFzSIEYOPEP9qnjXNfjo9wfnMaq6PWabFgNRSuz+a xfSlcl/T2izZmAYBuzA8DWifgkRJnclgHU5sNpM6Lm+LnG66QP2TNyGuXGtZF7vCijKc LOqWtQ6+f8ZbCAr3QsdtBCMy96zshLVIOpzxI/KTcAByeuo+BTyQgvQXhb/Rh/J778sK r4Qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=5QRiWs/DQbjPzNp3Mmcykk+AEBHTNlfm8mEd5w/KsEk=; b=uBFjwjbef+nYtoXH2zKuK8Di9x9VIwQQXGGb8LZVS9STTGrTb+1Bp2n440+YbE0Kbk DK94wSUpPF+/N0a3/fcBIvhVwCxt47A7JgfQCs7YlPdq1XH5+QkzIQ3yP+gQgV3Sk6TR NidcF98C47VML8cfT+zNsZa3j8oN7rXurs/LxNcH/XDdq2JaGHkSb8MG3Fax5b5DCEMf BqPPDXrYVv0LBDpYWNdCtnvuwicwRlgmJz29xuSra27ITx+YxOUGbTMjissnEy/1kYZU tMIPPUrno43z13PuglUpO2K9Fpn7XAvrkgDncaPkg1LRTUR3k8+zsGhe0kegcsRpq2GS p9iA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=QHyBSbia; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id b202sor368630wmd.20.2019.06.24.14.02.06 for (Google Transport Security); Mon, 24 Jun 2019 14:02:06 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=QHyBSbia; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5QRiWs/DQbjPzNp3Mmcykk+AEBHTNlfm8mEd5w/KsEk=; b=QHyBSbiaOkKGu/MxsSMUiXu2VTSCzVLYMxWwINOB/2Q6vPwGK/3fpuJ8DeqVV/RHuy PlWt0PKGF2DxILefQKepmvvAs9zfo0sJ/4JuHPb0QB5wuqyBLoHp42kHvsETulBVPEc1 nqIqqhYShDyfMHcp9pog0fds+dL+EL98KHWdrqc1jzmg4ngIMYHE16t7uPvX3+VkB4vI bsZkTMXkFYDGzt6JPUksZtjhk0vmx/AJejbnSzHMAh8/EfzJ0wiL46ukX5ePRnoNGE/1 nJ7mvYjHCVInsefIPkoIGcN7Hka7i1yS3Hv0kHKhgEn2UJ3p9gHquSgDjpsddvKgWt95 ZSwg== X-Google-Smtp-Source: APXvYqyUO9q1BVm7bVuWFkSD3xUfLwXJTWFUz+E3q3pTmFAgmdWwYSJRzm7jgl6xIMlY4/wE2Sy8yg== X-Received: by 2002:a1c:2311:: with SMTP id j17mr9036341wmj.84.1561410125783; Mon, 24 Jun 2019 14:02:05 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id d5sm10927937wrc.17.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6C-0001MH-Rm; Mon, 24 Jun 2019 18:02:00 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 04/12] mm/hmm: Simplify hmm_get_or_create and make it reliable Date: Mon, 24 Jun 2019 18:01:02 -0300 Message-Id: <20190624210110.5098-5-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe As coded this function can false-fail in various racy situations. Make it reliable and simpler by running under the write side of the mmap_sem and avoiding the false-failing compare/exchange pattern. Due to the mmap_sem this no longer has to avoid racing with a 2nd parallel hmm_get_or_create(). Unfortunately this still has to use the page_table_lock as the non-sleeping lock protecting mm->hmm, since the contexts where we free the hmm are incompatible with mmap_sem. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- v2: - Fix error unwind of mmgrab (Jerome) - Use hmm local instead of 2nd container_of (Jerome) v3: - Can't use mmap_sem in the SRCU callback, keep using the page_table_lock (Philip) v4: - Put the mm->hmm = NULL in the kref release, reduce LOC in hmm_get_or_create() (Christoph) --- mm/hmm.c | 77 ++++++++++++++++++++++---------------------------------- 1 file changed, 30 insertions(+), 47 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 080b17a2e87e2d..0423f4ca3a7e09 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -31,16 +31,6 @@ #if IS_ENABLED(CONFIG_HMM_MIRROR) static const struct mmu_notifier_ops hmm_mmu_notifier_ops; -static inline struct hmm *mm_get_hmm(struct mm_struct *mm) -{ - struct hmm *hmm = READ_ONCE(mm->hmm); - - if (hmm && kref_get_unless_zero(&hmm->kref)) - return hmm; - - return NULL; -} - /** * hmm_get_or_create - register HMM against an mm (HMM internal) * @@ -55,11 +45,16 @@ static inline struct hmm *mm_get_hmm(struct mm_struct *mm) */ static struct hmm *hmm_get_or_create(struct mm_struct *mm) { - struct hmm *hmm = mm_get_hmm(mm); - bool cleanup = false; + struct hmm *hmm; + + lockdep_assert_held_exclusive(&mm->mmap_sem); - if (hmm) - return hmm; + /* Abuse the page_table_lock to also protect mm->hmm. */ + spin_lock(&mm->page_table_lock); + hmm = mm->hmm; + if (mm->hmm && kref_get_unless_zero(&mm->hmm->kref)) + goto out_unlock; + spin_unlock(&mm->page_table_lock); hmm = kmalloc(sizeof(*hmm), GFP_KERNEL); if (!hmm) @@ -74,57 +69,45 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) hmm->notifiers = 0; hmm->dead = false; hmm->mm = mm; - mmgrab(hmm->mm); - spin_lock(&mm->page_table_lock); - if (!mm->hmm) - mm->hmm = hmm; - else - cleanup = true; - spin_unlock(&mm->page_table_lock); + hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops; + if (__mmu_notifier_register(&hmm->mmu_notifier, mm)) { + kfree(hmm); + return NULL; + } - if (cleanup) - goto error; + mmgrab(hmm->mm); /* - * We should only get here if hold the mmap_sem in write mode ie on - * registration of first mirror through hmm_mirror_register() + * We hold the exclusive mmap_sem here so we know that mm->hmm is + * still NULL or 0 kref, and is safe to update. */ - hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops; - if (__mmu_notifier_register(&hmm->mmu_notifier, mm)) - goto error_mm; - - return hmm; - -error_mm: spin_lock(&mm->page_table_lock); - if (mm->hmm == hmm) - mm->hmm = NULL; + mm->hmm = hmm; + +out_unlock: spin_unlock(&mm->page_table_lock); -error: - mmdrop(hmm->mm); - kfree(hmm); - return NULL; + return hmm; } static void hmm_free_rcu(struct rcu_head *rcu) { - kfree(container_of(rcu, struct hmm, rcu)); + struct hmm *hmm = container_of(rcu, struct hmm, rcu); + + mmdrop(hmm->mm); + kfree(hmm); } static void hmm_free(struct kref *kref) { struct hmm *hmm = container_of(kref, struct hmm, kref); - struct mm_struct *mm = hmm->mm; - mmu_notifier_unregister_no_release(&hmm->mmu_notifier, mm); + spin_lock(&hmm->mm->page_table_lock); + if (hmm->mm->hmm == hmm) + hmm->mm->hmm = NULL; + spin_unlock(&hmm->mm->page_table_lock); - spin_lock(&mm->page_table_lock); - if (mm->hmm == hmm) - mm->hmm = NULL; - spin_unlock(&mm->page_table_lock); - - mmdrop(hmm->mm); + mmu_notifier_unregister_no_release(&hmm->mmu_notifier, hmm->mm); mmu_notifier_call_srcu(&hmm->rcu, hmm_free_rcu); } From patchwork Mon Jun 24 21:01:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014143 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 67C82112C for ; Mon, 24 Jun 2019 21:02:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A1BF288D8 for ; Mon, 24 Jun 2019 21:02:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4DC1228910; Mon, 24 Jun 2019 21:02:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C091C288D8 for ; Mon, 24 Jun 2019 21:02:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A583D8E0002; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A0BB68E0009; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6FFCE8E0002; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by kanga.kvack.org (Postfix) with ESMTP id 120018E000A for ; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) Received: by mail-wr1-f69.google.com with SMTP id l11so6937842wrv.9 for ; Mon, 24 Jun 2019 14:02:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Xvvvfm/iICt6EAx3A7hygYXF1jRJFUeMrawdTwOi4ns=; b=VE9wrLonYLVYhGcNza3Xd82/AOSVblYF41ASbAN+88YceWGY81mY55GRWBaXrYixlA oAoP0qJT01H5svtj22PQxUH29fgatQM91XpEQ5tLV7TMGpCutrwaT5CXKD1qWWKDW2+8 PO3FYeat94KmVvOWDZMpoHzNlMvf7DvNDkeYyqh1vzFaOdd5s8PisgjoKcz4M/DtWtoF Rg7OCd+LLGT2um6e4u58vKyiJ+xnUhGmsubgeiM55dhg0QHlhKh/z3NEPQ4G5w3ASW5L bDcHXXwrSF1II6+mpxmXCCY9Xo7CuKrNZJGQr6UXlTFs6NyfR9EzM6a+JEo32AKTJ3NP nGWA== X-Gm-Message-State: APjAAAV4+fuYwsB7MPMB7m997dxRpO7EYlsPeE/Z4jTDZ3ZnY4Js7t2r PzFFo75ms1zFWmI8nfu/7tVJonUGv8DNAnk/9oO41vGvtZtL371RRvSFbLmmZmM/joYxXFtKn7E L4c0yr1xN6z0RMMJXbvK+/ikNGVP+oFI4r15jlkiP4U0wF0hyzpHFl2jpL81C9NC+ag== X-Received: by 2002:adf:c654:: with SMTP id u20mr26984336wrg.271.1561410127605; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) X-Received: by 2002:adf:c654:: with SMTP id u20mr26984304wrg.271.1561410126747; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410126; cv=none; d=google.com; s=arc-20160816; b=nAUCL1Pi8cWgQGNLVtLXRovrmp8/kuBH4PI6gPUK5G8E0fvLkdHWy4enaMrCSsIHqc tbRSGsfxUtsjwt9Arxo2X/PkWq/ykAfNEECBgqDyyRV2fFKRMXfWbS1HlxTvnrhOMOWn 2BO3q6Eq7f+5itZmj4yYzmvDv/BcHA/hS3PnjHKME4mGupzjP0T770bAmQHweTh38jh3 pv00G1zwFXloIJcfcE+xNnruNIRkwD5j+DW0dbZpz9TeQDiIXSzFXlXqBXUExEPzblLO I6V7Prv/0ggSrCkMiUhpH7FaqipXFOtsgzQo1e4YW39itde88T5800kIhzLoo8NLuCPa YG7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=Xvvvfm/iICt6EAx3A7hygYXF1jRJFUeMrawdTwOi4ns=; b=TrffeMSxZJg+kGWWfQTb1NZBJvb59819VWvQoXG0gxYk2sQrPZ7hactsLskMjb+lS1 5rw7oYcxAi2SXFd97A5d6K0PjwBeVt2Y57FMIblBqtlHyV9RMay9xfTOnX1ZnZ1M7+UI p7glr7xQdWHgaeeBFR5P1jM9bSdxVr4tTn1dyp3SefMPJ8Kt7rlgV+N4LCtFrCzhb7k3 +G4X5Lxm6Ata6zFO6m2gJz09il6k8itFpP1TDrdnUKqImEICtFg4vm4Q2m60TjylhJil vHpELJiSAg6DDsykNQ6YHBmSmSJUx/6BFZW/K1VUY62RKZQkbsCMAqlrKrGKzI6ocHly 3T2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=oRmOk0Zj; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id t16sor7104312wrn.0.2019.06.24.14.02.06 for (Google Transport Security); Mon, 24 Jun 2019 14:02:06 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=oRmOk0Zj; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Xvvvfm/iICt6EAx3A7hygYXF1jRJFUeMrawdTwOi4ns=; b=oRmOk0Zjj6i+pMHvCNnVsb1wjfONs2JBTQnEE3rglcDcUQoG84gXWTDhq9ll76MAqu NbvHm5LXP2Mk44izNVB3slQr8uyWefSSkQB31EfSp+uwq/HLZh/BdNDnY90d603FucOz AC8Ce8Oh4o0dEDF5pq+h9XhIOGQHW9OrQOCgPMR6XEU0ePQEDlF6JmUB9j6v51IImlni 0CRtVjdLLAq8G7uZWcwboxDMUZIuPWUPunZLZVkEDI1o2Eb0mtkYKg08tRTwS/QFWzk2 3v3z9wVq/LNXCzMnbVfsewXyPw+o1BeCYz8bMIx9Jd6EvXRVTZY/u0AkUUg08yGiPfXO xJgA== X-Google-Smtp-Source: APXvYqxEB08pD22xw7z431fz4zoGHVSBNwMRdaiB40BUYEvmuPcR/sYQyqDcLZPR3XganfuFLHNJMQ== X-Received: by 2002:a5d:5283:: with SMTP id c3mr28666603wrv.268.1561410126403; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id j7sm16820277wru.54.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6C-0001MM-Sw; Mon, 24 Jun 2019 18:02:00 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 05/12] mm/hmm: Remove duplicate condition test before wait_event_timeout Date: Mon, 24 Jun 2019 18:01:03 -0300 Message-Id: <20190624210110.5098-6-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe The wait_event_timeout macro already tests the condition as its first action, so there is no reason to open code another version of this, all that does is skip the might_sleep() debugging in common cases, which is not helpful. Further, based on prior patches, we can now simplify the required condition test: - If range is valid memory then so is range->hmm - If hmm_release() has run then range->valid is set to false at the same time as dead, so no reason to check both. - A valid hmm has a valid hmm->mm. Allowing the return value of wait_event_timeout() (along with its internal barriers) to compute the result of the function. Signed-off-by: Jason Gunthorpe Reviewed-by: Ralph Campbell Reviewed-by: John Hubbard Reviewed-by: Ira Weiny Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- v3 - Simplify the wait_event_timeout to not check valid --- include/linux/hmm.h | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 1d97b6d62c5bcf..26e7c477490c4e 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -209,17 +209,8 @@ static inline unsigned long hmm_range_page_size(const struct hmm_range *range) static inline bool hmm_range_wait_until_valid(struct hmm_range *range, unsigned long timeout) { - /* Check if mm is dead ? */ - if (range->hmm == NULL || range->hmm->dead || range->hmm->mm == NULL) { - range->valid = false; - return false; - } - if (range->valid) - return true; - wait_event_timeout(range->hmm->wq, range->valid || range->hmm->dead, - msecs_to_jiffies(timeout)); - /* Return current valid status just in case we get lucky */ - return range->valid; + return wait_event_timeout(range->hmm->wq, range->valid, + msecs_to_jiffies(timeout)) != 0; } /* From patchwork Mon Jun 24 21:01:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014187 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 699DF13AF for ; Mon, 24 Jun 2019 21:33:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 53BA2289CF for ; Mon, 24 Jun 2019 21:33:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4721628AB6; Mon, 24 Jun 2019 21:33:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 97E00289CF for ; Mon, 24 Jun 2019 21:33:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B78846B0007; Mon, 24 Jun 2019 17:33:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B4FA88E0003; Mon, 24 Jun 2019 17:33:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3F8D8E0002; Mon, 24 Jun 2019 17:33:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by kanga.kvack.org (Postfix) with ESMTP id 593C76B0007 for ; Mon, 24 Jun 2019 17:33:07 -0400 (EDT) Received: by mail-wr1-f69.google.com with SMTP id b6so6917187wrp.21 for ; Mon, 24 Jun 2019 14:33:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=LXLRo2PPKRMD0FsdNO4kvH6cJWSg4ysuh5L5UkxtS4c=; b=RL6e357Oo0qdKw2VDAEM40Xk2uEOTy5Kz5oJwwFTPesWT3N9h7vHH3nSoxeWK+yc5X 4DXw6zC1UU7K76Z5kdbajFHhB5T0QFkUuq0C+IxEwXgI4k1qqCX2o8+zbtw7XmotKwS1 JATHrVvXWrHCg+/AqQVLygDSjv1gZYKfEqrEl3CiI6CjZYRdYm4/TtEmD7NsaJUhCiw4 RAUO7hqpEUW9IcyiTHX4X0736PITM9j9p9wFXo2FM66PxW7mMhHT+5d0tjBRDyG1nGmh RA4Ld0sjGGKHlDG26RrGLxd4HXt0jev7uwQqUTiFVYfTKR/6B3WCLvi79HcutZPJL2dx e09Q== X-Gm-Message-State: APjAAAV1Vg5Otqn4NdATDhoX58rhyEAPnKr0KuvI4YS+ZEdrS+K2BtN8 21MNto6WDsrYcmfNfx0GCuQQ+DxnQNXJSAh7gwcR3gH+Yb71uCF1auWIgO2wy6FczkIgBpgQHl1 ArpAmVz8FHQsDoVvBeAG0qiu83o1BHI5ADJObe2xn4h/3eBg81jbPvbNr071K+nyJ1w== X-Received: by 2002:adf:cf0a:: with SMTP id o10mr33570695wrj.37.1561411986784; Mon, 24 Jun 2019 14:33:06 -0700 (PDT) X-Received: by 2002:adf:cf0a:: with SMTP id o10mr33570668wrj.37.1561411985967; Mon, 24 Jun 2019 14:33:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561411985; cv=none; d=google.com; s=arc-20160816; b=hMg6f9rHEo4HnFXNTetWHElhDPWfCH0W6ZUWU+hQIsPIbdS8H7xoNHYM34Eg0CMXcK 2AL5JRuum7RwOErbUoLfUDA+/jf9C5VDO3/MEVaW6+1dTUH5n3BtmdUqrWjEndrSVb6F uSE3QRMpUgjw0gk8qQ3/oJ4FY1DETFndkz90O0rw69x7n5Mn5XmBaPEbokT+ZXYS559Z W2wlt4KP7LUTvCWmwvt/LOrAVVsv2qAn+CLDcOMBVfYIxQwtfxFILeQwlk99tfNUuply U3WSM3vMIHgyHIXDdc0e4O03KJOoXvG+NKIlqN5JJOqiEyaQMtcA22EcAd2/+/rikCHF fEAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=LXLRo2PPKRMD0FsdNO4kvH6cJWSg4ysuh5L5UkxtS4c=; b=lkD1kvhCweQ2WYJWLIG5gtqb/A5MKOxwd/dMKUSI8fYnkDr+PxhbODYqyS4Q9EOuXd MDw/2TxxXwsHl00a83Xle/5EmX6kFq9Dr7O4Y+rgsuPQa5BISA7KQhrFpqR5ernuSGlf 84MCwUC2hssOIBnac7F6wnasFEOvGU6Rvr9pvRcHdWyO0v9eA1Yc7VZ2dWHPrPuxo8Vc u80PwLCNgQls0edUbNXO2sRetMlFy+NsyTJJuX304K76W8mWURyfbIS/hN0Txh/8Ktd3 CPJd4MOLI78OFK78fAQksXIguhjgIkb2LwPD7lHrLOuuSMcPER7fbmO9pBEic10FThsr dfiQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=gMJNqHRI; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id u18sor391242wmm.18.2019.06.24.14.33.05 for (Google Transport Security); Mon, 24 Jun 2019 14:33:05 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=gMJNqHRI; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LXLRo2PPKRMD0FsdNO4kvH6cJWSg4ysuh5L5UkxtS4c=; b=gMJNqHRIngzsRcDJSRqNGhhAFj5qszbwfHp7sWjkREH0cC5Wg+51IUv6Ztnp6y1sQJ 4s8sZl4zFkpY3KIjUkN7OCuNT+5pVhM4Z9P2VYMGh6zz+dgJPGQTFMmyTF8pYefOKxHb RQC3U3vsTjMnbDULYbk6PEP2Dr94o6uyCpK8jOxwldZdOsr7VxS3+nUpYxsCXash5XQk 7PaTVmaPpwPfDGC4tTkJn+amRntW1ha7ej9YjFLte5PbAElDu2HZyg7aDGR1mtYjhEC4 a5es/TsjfPiC+icZkrWdCKcFqlYMm30vlAIazLUZsF/W2Ct6q9aWO4iYeVJJJbe9l0jw 3AYQ== X-Google-Smtp-Source: APXvYqxpgSLdOjxG4AEMzxa4goOMrAbz+ND9ASIrtQwBg7DrjmWk1TBdaulquFyqhwOqbOtHtgBWrg== X-Received: by 2002:a1c:407:: with SMTP id 7mr18250094wme.113.1561411985569; Mon, 24 Jun 2019 14:33:05 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id r4sm18908060wra.96.2019.06.24.14.33.04 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:33:04 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6C-0001MR-U4; Mon, 24 Jun 2019 18:02:00 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe , Souptick Joarder , Ira Weiny Subject: [PATCH v4 hmm 06/12] mm/hmm: Do not use list*_rcu() for hmm->ranges Date: Mon, 24 Jun 2019 18:01:04 -0300 Message-Id: <20190624210110.5098-7-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe This list is always read and written while holding hmm->lock so there is no need for the confusing _rcu annotations. Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Acked-by: Souptick Joarder Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- mm/hmm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 0423f4ca3a7e09..73c8af4827fe87 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -912,7 +912,7 @@ int hmm_range_register(struct hmm_range *range, range->hmm = hmm; kref_get(&hmm->kref); - list_add_rcu(&range->list, &hmm->ranges); + list_add(&range->list, &hmm->ranges); /* * If there are any concurrent notifiers we have to wait for them for @@ -942,7 +942,7 @@ void hmm_range_unregister(struct hmm_range *range) return; mutex_lock(&hmm->lock); - list_del_rcu(&range->list); + list_del(&range->list); mutex_unlock(&hmm->lock); /* Drop reference taken by hmm_range_register() */ From patchwork Mon Jun 24 21:01:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014149 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BB11C112C for ; Mon, 24 Jun 2019 21:02:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC62C288D8 for ; Mon, 24 Jun 2019 21:02:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A07C028910; Mon, 24 Jun 2019 21:02:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07B6B288D8 for ; Mon, 24 Jun 2019 21:02:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1E2C8E000A; Mon, 24 Jun 2019 17:02:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B91378E000C; Mon, 24 Jun 2019 17:02:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BBCA8E000D; Mon, 24 Jun 2019 17:02:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by kanga.kvack.org (Postfix) with ESMTP id 484828E000C for ; Mon, 24 Jun 2019 17:02:09 -0400 (EDT) Received: by mail-wr1-f69.google.com with SMTP id e6so6857353wrv.20 for ; Mon, 24 Jun 2019 14:02:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=tsR/80TrTiUsWjQJpDlrT4mzZAmfUfoSI9FV/A0UaHU=; b=Gi9i/VMmy4CYHQ+v+9t6E0aCIjItxT7LUzL9N3qKh0N/whwn374g8Z0eVLWxlxhkHF pdszpHzF5atiA9OfvX4rwHRhUEhagt2DjSL4OMAf7uhKeNCqIp09rgOkEgDGK3MF8m6l Km5qtCQJpC5BaLjwXJiQ6IQibkfX65w2W3YK2XENsi7AMXz2x0nw26bb7YRAFz+iFNgY JCqDvwtgLjNJK/T1HLmEzAgWNpQXWK8ZIt/e+zLYkXgJlHRR23l0Uq6o2S/v+qtkc5Kl 1QZAfRttCT/goS7goWHcoi67Pp8rFC8BMpKtDzXGQ4iZDCji7PRKthHy8ud5C7DTJTpt kmvw== X-Gm-Message-State: APjAAAUapMYhqqOYFB39xS4ZKABDdyYANAhficwXFqbxVHVJCzUSy/6O EnEMGOAmzZA2Q+nuJqI/MolEq4D3roR/VHL6U0ht077MJrywkw/xS6eAZ1olW3+k3X4j4Lpp1r8 5poF4+RiXg5yTpAxnje7wtTVx2Io3KVcnHQ/lWuQOLT3CWFz45ycHVahG4w+yyVW6XA== X-Received: by 2002:adf:f34b:: with SMTP id e11mr26147353wrp.230.1561410128806; Mon, 24 Jun 2019 14:02:08 -0700 (PDT) X-Received: by 2002:adf:f34b:: with SMTP id e11mr26147308wrp.230.1561410127912; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410127; cv=none; d=google.com; s=arc-20160816; b=QMPSgDpWPbze10RIH7pBgypYtu77YsmvjskqIh0vZIe3t3buLqK4DamYBQ/aNG24WD TPC6eYQvrhy/X3vXG4VhsW3HoRWR8nszj5PU0W5Vml9oS0moWk7K1YpSocFwXM3VaI9y e2SOvzE64kpS9HguWFxG8nxSOSKl9kmEb11Wc1EfREwAfByyDX0wy/fW+7Wsn/FnX7l/ gAmrsEcx+cQxciretPNgBg+2MZCK8mB5i7cGEg9Y6gcy/+6v5hNhP+c1zcYvHyZHNqJJ s3fjIbi/yMSdnBbMOQBGOh1rFJkn4nknKJBNAaGJMhs0KKvopyTr4pWJf0JrAs54C3pG gcBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=tsR/80TrTiUsWjQJpDlrT4mzZAmfUfoSI9FV/A0UaHU=; b=ihClauULuSr2xf6vuSq/pY1YrYqck7Cv9UBI/3BwNepGugBk748RH3lQ4DytqZRmYV Qo2Qb8jZzwRZNRY5n2Q86oJUMKqCisIr/r4IaKVcRyUAMqyFWrC5czbw0n6J7Q5OeEId 8eZfRvUoK++QVu2cB4qvnZfUaJmELq86+qsEZ5C4KhzWeZqeS0izvjgw1bAgPe+iFbvQ 1beOIVAkbciDV630XVRLC/CDpESWqrtD0lq2UgKVDIf1J0Ls1GzCHrY2V9xPFHNLd0WN CQbShHqDerzu7uaMfUik4161Kif0OwwJMrBMpqO6gtlb5Enr1+s3l2XQ7flrEywOexE6 sreA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=eEL2e+Rd; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id e15sor2502495wru.2.2019.06.24.14.02.07 for (Google Transport Security); Mon, 24 Jun 2019 14:02:07 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=eEL2e+Rd; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tsR/80TrTiUsWjQJpDlrT4mzZAmfUfoSI9FV/A0UaHU=; b=eEL2e+Rdsoiw+QWNMAoS5eOmfH9fxVT52iqVy3PrFcraDcKDNUukaiSHGjOkUDbkf4 8WHJYUyZdN7bf/y/Jstfu3A1fXcJitPxheR1/65Jv090zH+Eo5OQi1q+O4dWucKvZtrF Svhl2dcm8mOneehpNFXJVkZ7TYkrWa7CSn4DrBB60/QLpDJEVZeh45IqxTnVZkAWRCWG fnhCq6Ea5ODgJg75hQ/Xpeto43ZyjWXSU53z1FdntKD0LH/Otni2IWokQ2rJk9vV5gII L0dENrfxVOm9si6Xksz7dp0Ey4NfHgch3yvMfxoEZTAShU86NCM8h0UHtjz8kyN+4J6G MbOw== X-Google-Smtp-Source: APXvYqyzrTyoPhYHFWjo48jic83g1/CALuSvPyW3MCerg+RPA+Kf+K4POtxSR/W0sWmHBTVwQ3b5Mw== X-Received: by 2002:adf:a312:: with SMTP id c18mr18493691wrb.332.1561410127528; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id x11sm469692wmg.23.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6C-0001MW-VA; Mon, 24 Jun 2019 18:02:00 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 07/12] mm/hmm: Hold on to the mmget for the lifetime of the range Date: Mon, 24 Jun 2019 18:01:05 -0300 Message-Id: <20190624210110.5098-8-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe Range functions like hmm_range_snapshot() and hmm_range_fault() call find_vma, which requires hodling the mmget() and the mmap_sem for the mm. Make this simpler for the callers by holding the mmget() inside the range for the lifetime of the range. Other functions that accept a range should only be called if the range is registered. This has the side effect of directly preventing hmm_release() from happening while a range is registered. That means range->dead cannot be false during the lifetime of the range, so remove dead and hmm_mirror_mm_is_alive() entirely. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- v2: - Use Jerome's idea of just holding the mmget() for the range lifetime, rework the patch to use that as as simplification to remove dead in one step v3: - Use list_del_careful (Christoph) --- include/linux/hmm.h | 26 -------------------------- mm/hmm.c | 32 +++++++++++--------------------- 2 files changed, 11 insertions(+), 47 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 26e7c477490c4e..bf013e96525771 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -82,7 +82,6 @@ * @mirrors_sem: read/write semaphore protecting the mirrors list * @wq: wait queue for user waiting on a range invalidation * @notifiers: count of active mmu notifiers - * @dead: is the mm dead ? */ struct hmm { struct mm_struct *mm; @@ -95,7 +94,6 @@ struct hmm { wait_queue_head_t wq; struct rcu_head rcu; long notifiers; - bool dead; }; /* @@ -459,30 +457,6 @@ struct hmm_mirror { int hmm_mirror_register(struct hmm_mirror *mirror, struct mm_struct *mm); void hmm_mirror_unregister(struct hmm_mirror *mirror); -/* - * hmm_mirror_mm_is_alive() - test if mm is still alive - * @mirror: the HMM mm mirror for which we want to lock the mmap_sem - * Return: false if the mm is dead, true otherwise - * - * This is an optimization, it will not always accurately return false if the - * mm is dead; i.e., there can be false negatives (process is being killed but - * HMM is not yet informed of that). It is only intended to be used to optimize - * out cases where the driver is about to do something time consuming and it - * would be better to skip it if the mm is dead. - */ -static inline bool hmm_mirror_mm_is_alive(struct hmm_mirror *mirror) -{ - struct mm_struct *mm; - - if (!mirror || !mirror->hmm) - return false; - mm = READ_ONCE(mirror->hmm->mm); - if (mirror->hmm->dead || !mm) - return false; - - return true; -} - /* * Please see Documentation/vm/hmm.rst for how to use the range API. */ diff --git a/mm/hmm.c b/mm/hmm.c index 73c8af4827fe87..1eddda45cefae7 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -67,7 +67,6 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) mutex_init(&hmm->lock); kref_init(&hmm->kref); hmm->notifiers = 0; - hmm->dead = false; hmm->mm = mm; hmm->mmu_notifier.ops = &hmm_mmu_notifier_ops; @@ -120,21 +119,16 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); struct hmm_mirror *mirror; - struct hmm_range *range; /* Bail out if hmm is in the process of being freed */ if (!kref_get_unless_zero(&hmm->kref)) return; - /* Report this HMM as dying. */ - hmm->dead = true; - - /* Wake-up everyone waiting on any range. */ - mutex_lock(&hmm->lock); - list_for_each_entry(range, &hmm->ranges, list) - range->valid = false; - wake_up_all(&hmm->wq); - mutex_unlock(&hmm->lock); + /* + * Since hmm_range_register() holds the mmget() lock hmm_release() is + * prevented as long as a range exists. + */ + WARN_ON(!list_empty_careful(&hmm->ranges)); down_write(&hmm->mirrors_sem); mirror = list_first_entry_or_null(&hmm->mirrors, struct hmm_mirror, @@ -903,8 +897,8 @@ int hmm_range_register(struct hmm_range *range, range->start = start; range->end = end; - /* Check if hmm_mm_destroy() was call. */ - if (hmm->mm == NULL || hmm->dead) + /* Prevent hmm_release() from running while the range is valid */ + if (!mmget_not_zero(hmm->mm)) return -EFAULT; /* Initialize range to track CPU page table updates. */ @@ -942,11 +936,12 @@ void hmm_range_unregister(struct hmm_range *range) return; mutex_lock(&hmm->lock); - list_del(&range->list); + list_del_init(&range->list); mutex_unlock(&hmm->lock); /* Drop reference taken by hmm_range_register() */ range->valid = false; + mmput(hmm->mm); hmm_put(hmm); range->hmm = NULL; } @@ -974,10 +969,7 @@ long hmm_range_snapshot(struct hmm_range *range) struct vm_area_struct *vma; struct mm_walk mm_walk; - /* Check if hmm_mm_destroy() was call. */ - if (hmm->mm == NULL || hmm->dead) - return -EFAULT; - + lockdep_assert_held(&hmm->mm->mmap_sem); do { /* If range is no longer valid force retry. */ if (!range->valid) @@ -1072,9 +1064,7 @@ long hmm_range_fault(struct hmm_range *range, bool block) struct mm_walk mm_walk; int ret; - /* Check if hmm_mm_destroy() was call. */ - if (hmm->mm == NULL || hmm->dead) - return -EFAULT; + lockdep_assert_held(&hmm->mm->mmap_sem); do { /* If range is no longer valid force retry. */ From patchwork Mon Jun 24 21:01:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014117 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8FD65186E for ; Mon, 24 Jun 2019 21:02:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 837F128925 for ; Mon, 24 Jun 2019 21:02:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7830B28949; Mon, 24 Jun 2019 21:02:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 69AFA28925 for ; Mon, 24 Jun 2019 21:02:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E07F06B0008; Mon, 24 Jun 2019 17:02:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DB5D58E0002; Mon, 24 Jun 2019 17:02:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2E3B6B000C; Mon, 24 Jun 2019 17:02:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by kanga.kvack.org (Postfix) with ESMTP id 631B36B0008 for ; Mon, 24 Jun 2019 17:02:05 -0400 (EDT) Received: by mail-wm1-f70.google.com with SMTP id 21so70024wmj.4 for ; Mon, 24 Jun 2019 14:02:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2HyibpLykYkiKTwGwDVDE6Jz0uAzR58LFXlNi/YRy3U=; b=AdmJ7ofz2m7I9PyYmBo29XE77XmQJLSkkrrbbHC0wNVNRXA8RB0uoIykV76D0niWSR pCbgclK2dgeio1G0l2R1ituDUwJ0+lXhca/bPGuo6b+fW/vcxzL50TrSQPODcAVFtSAJ 8YUux5zGMEt+7C0uTqIl6S7Hz+qEBLARpNaJIGmRCsJ60MwjBtLrb3cVECzG3/smmjTZ 73Y+84EDwdgjKp1OH3goapZQ8QJleF1wWrTN2A/ighHBWuqtWoANR8w1yFDQsKAGM6tf B7WHWI7EUFdNB/BzU+F6F4BIrrxi1RVTSLlla9RA/hzbKqQmVf12MZ/jNsLbHL7g5026 nxzg== X-Gm-Message-State: APjAAAXXEsXdqa8xTZX4E222NldEgzhyfbK4qIKWfFU8kjIknWmdutnF CrfuSKF4j0oTlDjiTXopewjYENdt8nnYsziiJfN86J+TYP4zd4E2p72Rsfn38XrblWbXcIPtGvu YSCRJ/Cj2mtnxIE50IgYNzIIUXHh8e4dK/bDlPfRJFU2jcb9fjfvzWcSqmOKnahRmAw== X-Received: by 2002:adf:fbc7:: with SMTP id d7mr4577515wrs.224.1561410124768; Mon, 24 Jun 2019 14:02:04 -0700 (PDT) X-Received: by 2002:adf:fbc7:: with SMTP id d7mr4577491wrs.224.1561410123980; Mon, 24 Jun 2019 14:02:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410123; cv=none; d=google.com; s=arc-20160816; b=YCWelJbfnPOf6CktV+AqjLi1DS+xJ6UtVWYghiOcFXGLKawZfOktE8EY3d09YH5d5P WGMHf/+3YFUOtbBZ+b25cRXDT+wTtrVuPLXXP5jqed2EzJFttIJ3DA/eyVbi4QWNOx/C 1QI3vdJWdmfTyfB0KIOhtogP5OGGPrRNXD+S4L2Ujp6IMievRdBEjNlPderf6rxJWZ07 85CZAXUd97/EExvBfYHw8NNW0vLMD892fT4SCwwNiWjMDRNMxTTAtUACQJcqMHTLfj1p yufbYAtUjDD7Y4ZKJaz7lhf7RnimbgT4dwGRFm4i2RG5PwQSkQC44iYjtXstfHKDBymf YlgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=2HyibpLykYkiKTwGwDVDE6Jz0uAzR58LFXlNi/YRy3U=; b=wcH1LaYNeuMYEUjUvmnsvsYgJatuugyiGsxQ3yDJZLi4E+pYFh4m9NeIPC+729twMj 0VI6UGqvpntvul7C75AWElEagrFrnsfqQZES/n2hrGODjAY2JJ0eL8AQLwwdMiZULQs5 wzRvLcbuWCdoiun+nhAv+BBZo0WtM1uJ/G4U2QaTUv1b2fO6xarna4JrI4aSufdtqunk o4t3nWdE9d8tBVNBmO38uYVBehGv29isxs/+N07s58LawsNvfeYLno5qQm32keQgAPOK S6a8O01vO5LUC8yTr/zsoItuJlcOeBkyrlhZ9wYZdzUxinoAHFU4DjqMRh1H7lnGR+ag 9+7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b="o/tFvhN2"; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v15sor7233665wrp.38.2019.06.24.14.02.03 for (Google Transport Security); Mon, 24 Jun 2019 14:02:03 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b="o/tFvhN2"; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2HyibpLykYkiKTwGwDVDE6Jz0uAzR58LFXlNi/YRy3U=; b=o/tFvhN275Le/5WQ2vu2fc7dpnfllCtch7SunXoc0gfrceioMrV84vJNXj3fcaoXjc qKdfy6+FErMOpWLLywUqcU/CCvqkzz60BPZVTd/0tIIXWv44kB8+wkXkIP3fIfmQuRfU VCtdLewh0B23U6TTG5KtYq7rxUdtI/n6yUmErMCpp/rXPr3h474Gd7GbjYQ+iiDYdpy1 7shV4eVpmV49XoNejllG2+McGSuy2k7ZkrwJwBNEK6NQF3+JDLsqVqku3k4UUdsaGuab CgCCYDDWmuYZ06PXgv5ghP0Ro725vIEY+j0ePamWI4Lrm/GuhZ/Rebt3wxbycRP+uaHI i7Cw== X-Google-Smtp-Source: APXvYqxbykTnskxCEk6vJXnhGhJhL+Hc/YVg3s4yNHZO6Rq1Ef69wTB8s8QzzXVmzALZYfvIEAb5sA== X-Received: by 2002:adf:fa4c:: with SMTP id y12mr96903628wrr.282.1561410123670; Mon, 24 Jun 2019 14:02:03 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id x11sm469693wmg.23.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6D-0001Mb-0S; Mon, 24 Jun 2019 18:02:01 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe , Souptick Joarder Subject: [PATCH v4 hmm 08/12] mm/hmm: Use lockdep instead of comments Date: Mon, 24 Jun 2019 18:01:06 -0300 Message-Id: <20190624210110.5098-9-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe So we can check locking at runtime. Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Acked-by: Souptick Joarder Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- v2 - Fix missing & in lockdeps (Jason) --- mm/hmm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 1eddda45cefae7..6f5dc6d568feb1 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -246,11 +246,11 @@ static const struct mmu_notifier_ops hmm_mmu_notifier_ops = { * * To start mirroring a process address space, the device driver must register * an HMM mirror struct. - * - * THE mm->mmap_sem MUST BE HELD IN WRITE MODE ! */ int hmm_mirror_register(struct hmm_mirror *mirror, struct mm_struct *mm) { + lockdep_assert_held_exclusive(&mm->mmap_sem); + /* Sanity check */ if (!mm || !mirror || !mirror->ops) return -EINVAL; From patchwork Mon Jun 24 21:01:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014105 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D26AC14E5 for ; Mon, 24 Jun 2019 21:02:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C291C288FC for ; Mon, 24 Jun 2019 21:02:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B3AB328949; Mon, 24 Jun 2019 21:02:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BFE082891A for ; Mon, 24 Jun 2019 21:02:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A89526B0007; Mon, 24 Jun 2019 17:02:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A2E468E0003; Mon, 24 Jun 2019 17:02:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91E178E0002; Mon, 24 Jun 2019 17:02:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by kanga.kvack.org (Postfix) with ESMTP id 46F166B0007 for ; Mon, 24 Jun 2019 17:02:05 -0400 (EDT) Received: by mail-wr1-f72.google.com with SMTP id e6so6857271wrv.20 for ; Mon, 24 Jun 2019 14:02:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=rs1sEkzygtR6lbRJ0eUbAFkZlHYnT1CPNgA7awG9NQc=; b=D4BvSBgYpS5IxpPNmX0vv1uWrHU4rNZ+drk95kMvUKqB2zsmMS48OCZbP3bFjL+MUl ywbDZ0RwoxFhiStiI0/R5mWKyEBW2ZkyWMU1NOusNM0C7LXj27pCuC1sugUyxc1LuEZy QPgYT5NO1kx3meTe6CavYnlek661EK+AIqqePCwYMeysYvGdkRk/qpzXYdbrD+Ax25Kp pTE81M3lVeXu45lO/TCgqF6z1e/vFgQkgzulNWz5OFY6OtOkUTh6FOG87tzshpmEhHlR j5CFD9C5mYP0v+yShJOhYnm+F7mrSpZ5KVpt8+MExsgZ+egDm8QLHlyFJZpbmq/SAhs3 PWSA== X-Gm-Message-State: APjAAAWC/fXJfolWipoW9TLsvdGSqeqYBSiGXT6cs4RJr8bGZQ4mmeeQ IkcXqOvklYnvZ1G9Na75VJaaYfjUaxCgTvhmRuUUqkfmzWlPQL5SgIaTcyQ4H0qZlSVMDf+shqr SxphnELMf5MhYlrXW11g0yvVHwOBjEmPqXplyzKKP41PTUvRrcuZ33FhAECcakGtdnQ== X-Received: by 2002:a5d:4302:: with SMTP id h2mr15490214wrq.137.1561410124612; Mon, 24 Jun 2019 14:02:04 -0700 (PDT) X-Received: by 2002:a5d:4302:: with SMTP id h2mr15490190wrq.137.1561410123789; Mon, 24 Jun 2019 14:02:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410123; cv=none; d=google.com; s=arc-20160816; b=PaUMIEuU4X0FRh7dyJalFqecQrJgo8mvhM3XUtAIJV4b6MYxlOxsU1RxxYDepGtYM+ X0XDiGLzV43KUtEfzypIIzejqK4Y56nU9YYY4ofVSDgFl4FvnByOCTIGkyed/LioJV2J O0vq0/myYqHlqaQXdmxx+Y/bYHRxYfTduXHmnTLzMrtc0ba3DcnRzIIgRLvO5IA0v8GR dOZbShGQDPoosoHmeaBo3QUwupjqRBm/ecd3+FHOWpkvXsvrY+AgIoEtSw3yPmWcLcp1 Hrn1NBRfbJQAk9frfmUm43T/MGTD/BVj502CPCy+VHL1hxSMLi7sTTtat4AiufROsU3m 32VA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=rs1sEkzygtR6lbRJ0eUbAFkZlHYnT1CPNgA7awG9NQc=; b=TaY3kQ29TEDZmgNG6GU748XaPWzFbhERT+l132UBwtMdHUndXe4sGfvwEyInbAJwwS i4LTrjog1tyckYYrKA6d4mkXPjVUsVH6Rvro+U80nXwZLveGq9kZkaZdRqkl4worOKNb 8SSFeSZKlCMfSxBWWPKTs8WDUJkRqYr/gGYTrDtfry6kapXQyH5qgpiwZBBIuXZ0wZVD cPGQJqOjAIOv8uQUF6JyGdz6rk8TJP51ajkQt1ZUITQDygdRN1alpsBfqo2vF+cD9JVE UTWjN39IR47kAL0+T8tQMhI5Pwq3pR25mDuDd1bblDJJRrIcS1l6oCIxl0DITlHJHz1c Q+gQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=f0F8mUp3; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id n10sor7223427wrq.21.2019.06.24.14.02.03 for (Google Transport Security); Mon, 24 Jun 2019 14:02:03 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=f0F8mUp3; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rs1sEkzygtR6lbRJ0eUbAFkZlHYnT1CPNgA7awG9NQc=; b=f0F8mUp38+BSw/O1WjANVXVVMw8zhFZORjdRX1RtgtwdBgUWBtS4DuFgknhtRckqjJ fDPYac3PHVynkBRdhou+SH6UptVQIxVUIQifavRFZYv+BeWfES7diE3vVczt6RfpUEOL HKyKRdDLLYkWyPF33zlBvSCxa/KBn2/+Y2Sy1hmyEQzxoSkWZdP4GxRWfGqLBUquGL9C Ly8tJeOzyL9mpaDnKGNVFtycDXcnewWGVJEzIZFjBXuryCCVlKkVubMEGUB9qDKesfui S5zt2wnmoNX0dsm2two3hmXge0XbsLY2LY0GsVP/TYD0CD+hqf8x5J0pMRR0Lz+jvG54 bukQ== X-Google-Smtp-Source: APXvYqxoDaUcVHfte6JF2hIkD4U/Q2PNoEzAwuS3zYv3vRjkVzwDyM9fIRPi2AhLNMqFWzhsn/OFnQ== X-Received: by 2002:a5d:5446:: with SMTP id w6mr102260622wrv.164.1561410123399; Mon, 24 Jun 2019 14:02:03 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id h14sm11086221wrs.66.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6D-0001Mp-1k; Mon, 24 Jun 2019 18:02:01 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 09/12] mm/hmm: Remove racy protection against double-unregistration Date: Mon, 24 Jun 2019 18:01:07 -0300 Message-Id: <20190624210110.5098-10-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe No other register/unregister kernel API attempts to provide this kind of protection as it is inherently racy, so just drop it. Callers should provide their own protection, and it appears nouveau already does. Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Reviewed-by: Ralph Campbell Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- v3 - Drop poison, looks like there are no new patches that will use this wrong (Christoph) --- mm/hmm.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 6f5dc6d568feb1..2ef14b2b5505f6 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -276,17 +276,11 @@ EXPORT_SYMBOL(hmm_mirror_register); */ void hmm_mirror_unregister(struct hmm_mirror *mirror) { - struct hmm *hmm = READ_ONCE(mirror->hmm); - - if (hmm == NULL) - return; + struct hmm *hmm = mirror->hmm; down_write(&hmm->mirrors_sem); list_del_init(&mirror->list); - /* To protect us against double unregister ... */ - mirror->hmm = NULL; up_write(&hmm->mirrors_sem); - hmm_put(hmm); } EXPORT_SYMBOL(hmm_mirror_unregister); From patchwork Mon Jun 24 21:01:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014145 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EE23014E5 for ; Mon, 24 Jun 2019 21:02:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DFA8F288D8 for ; Mon, 24 Jun 2019 21:02:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D385E28910; Mon, 24 Jun 2019 21:02:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F2DE288D8 for ; Mon, 24 Jun 2019 21:02:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E51FB8E0009; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DD8988E000A; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF0978E0009; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by kanga.kvack.org (Postfix) with ESMTP id 71BC28E000A for ; Mon, 24 Jun 2019 17:02:08 -0400 (EDT) Received: by mail-wr1-f71.google.com with SMTP id i2so6891574wrp.12 for ; Mon, 24 Jun 2019 14:02:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Y7+HSfyIPKIyVRlHiPSUTXfMPMA2cBMnzwF3/BJJWoM=; b=BqOmy/kO2Y2p/J/T2qLA+VYJgg2PUcvXM3rwZ6J7724FSm9FohhjG9RoE/Gb26Pf/+ fB0gG4c7SJcUAdjkV6+qhy0u1o07YrH+Jccjm3rmGLMwcCzly/sihTnN+tZOtfIpqeWh CCDXohxqB7kqGvKR5K1He24fBWKjv8kigrslVsB14LVUD/3zZThNJ2p1w4kmDpZqvOKT plURq46XuWaceZO8Feg0fsEih/dKpbhIPaDLlnLEr07dxEGmumhyUUePkzKAzP75oCV2 faZaz2m/KjBiRsSRNY3IR6Qp8o+9lylWjjkG26rvmpANAxmYlBQ9PnmTiwRE/vmCaMl9 B1sQ== X-Gm-Message-State: APjAAAUw3X2/P0/St9CP5pEx0MTB7OsvdnJBWkCN3ZBzqrYwvPjtyY7O JO6LjjJYtrAH3NqL891A5kwv0ZkPiLGZ4W9PvH4ejz1YykeDpJ8sqfSa9uMNYOSCvIIc875lLmE tumySIpbKOrDov4uMv1Km1NWTMqS1ANuZDegv+WItxCB5NjWZATc9Ni11A4is906vqQ== X-Received: by 2002:adf:f542:: with SMTP id j2mr49635972wrp.16.1561410127955; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) X-Received: by 2002:adf:f542:: with SMTP id j2mr49635944wrp.16.1561410127141; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410127; cv=none; d=google.com; s=arc-20160816; b=W/6Xwd8ef2k1kDRE8q7pBcnjqoio+NNsNVYOjnqfuXWaPDA6TeqxoYbBXPncVzoWCt iouHgZ/G6gNEEcf1vWkD1dpUu+DixqvDORXjDnkQrex6HPjGLiXXxGGKbrEtXCi7LZYF puD4AOkk3mvR5ex2KV88aqAKt7zxCoRhIJRjHtAYYMQCL0cbxVfXL6ruXP0lz59PNGjs kKmP5BpZhOS4rcsp9IkEv8rTBOI62BHzfWz4QEiXZn0j2GKT0N+1NhxyO4o2RbnxO5An 1dHicIsxBpuwg/DcHuIiQeqKE1R1ieI5Wl0ZGdWHD5Ix1BSBa3rImvydn4dpM1ljya4f 0S9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=Y7+HSfyIPKIyVRlHiPSUTXfMPMA2cBMnzwF3/BJJWoM=; b=aju9sPxXNd3bPGvytfRbhyWBAOGnz55byW4SDOfH3zs6IrMFKTY2t+7MNtC7+x1mz+ RwQe9R/Ww+2fMAAwHPffWKrfB+0BfYRV2g48rmcNklVQKLLsXLOR9vzWUinoCJy+L/bB lk4ChepYi6F05p85mT8Cs0S3S/GkNHCAYNKbi/SaGgVrqQIFEeMCGHH717gPXCBs9IYx MViFmdQyrS0UhJbsVqmjYptvWCW7hocxVm5bHV1zgY2IiUo/fKIPb63Gom3gUACq2dDK QhOzuF1nZ+DMo9hU/jxtWNfykKh/SAiwi0INNRYPOrHbFuEPZ2hzorsvZguS7pkSkz9S qRUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=i+h5jdKO; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p2sor350098wma.27.2019.06.24.14.02.07 for (Google Transport Security); Mon, 24 Jun 2019 14:02:07 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=i+h5jdKO; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y7+HSfyIPKIyVRlHiPSUTXfMPMA2cBMnzwF3/BJJWoM=; b=i+h5jdKODrJM6HAgihfib7r1p3HTIoXotHgwp0BCzd39jXyo1BGTDGN5CpLNMF70TR V6PVWeRW3fvAO6anYPbAasncAm2gfDig99onMjq8DGbcxmOsbdJoUx+UOv3Ew172buN4 Vpsbiiy/5q+1oRn8UdTlfhMTuBnx4G1ji+wcVIaoYK5Nbt5H8mqxT5v/JzerbsooORD5 HimcJCbD82rblSOa3TsdNlNx3R6AxGDFuxuK+lEyTuOggfGhT5ua5kBPNLimVjJkqu5U RIddsbNK0RH7LIGRspoDR8r0YYISQuOycZZHVH/0xfxFXODjXkabiQ3hMa0QYuximxJ6 o2Mg== X-Google-Smtp-Source: APXvYqxYsEWfPj+ojP/Pdo0/oemDCD+9/xSgl5N/QS6OZzqvHa256HMQOw+3zV/4M5gAiwA2ZCoPBA== X-Received: by 2002:a1c:228b:: with SMTP id i133mr17321325wmi.140.1561410126760; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id k125sm600943wmf.41.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6D-0001Mv-3O; Mon, 24 Jun 2019 18:02:01 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe , Souptick Joarder Subject: [PATCH v4 hmm 10/12] mm/hmm: Poison hmm_range during unregister Date: Mon, 24 Jun 2019 18:01:08 -0300 Message-Id: <20190624210110.5098-11-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe Trying to misuse a range outside its lifetime is a kernel bug. Use poison bytes to help detect this condition. Double unregister will reliably crash. Signed-off-by: Jason Gunthorpe Reviewed-by: Jérôme Glisse Reviewed-by: John Hubbard Acked-by: Souptick Joarder Reviewed-by: Ralph Campbell Reviewed-by: Ira Weiny Tested-by: Philip Yang --- v2 - Keep range start/end valid after unregistration (Jerome) v3 - Revise some comments (John) - Remove start/end WARN_ON (Souptick) v4 - Fix tabs vs spaces in comment (Christoph) --- mm/hmm.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index 2ef14b2b5505f6..c30aa9403dbe4d 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -925,19 +925,21 @@ void hmm_range_unregister(struct hmm_range *range) { struct hmm *hmm = range->hmm; - /* Sanity check this really should not happen. */ - if (hmm == NULL || range->end <= range->start) - return; - mutex_lock(&hmm->lock); list_del_init(&range->list); mutex_unlock(&hmm->lock); /* Drop reference taken by hmm_range_register() */ - range->valid = false; mmput(hmm->mm); hmm_put(hmm); - range->hmm = NULL; + + /* + * The range is now invalid and the ref on the hmm is dropped, so + * poison the pointer. Leave other fields in place, for the caller's + * use. + */ + range->valid = false; + memset(&range->hmm, POISON_INUSE, sizeof(range->hmm)); } EXPORT_SYMBOL(hmm_range_unregister); From patchwork Mon Jun 24 21:01:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014135 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D86C4112C for ; Mon, 24 Jun 2019 21:02:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CA304288D8 for ; Mon, 24 Jun 2019 21:02:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BDDEC28910; Mon, 24 Jun 2019 21:02:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D7238288D8 for ; Mon, 24 Jun 2019 21:02:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A36D8E0006; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6121F8E0007; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22B8F8E0006; Mon, 24 Jun 2019 17:02:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by kanga.kvack.org (Postfix) with ESMTP id C08568E0005 for ; Mon, 24 Jun 2019 17:02:06 -0400 (EDT) Received: by mail-wr1-f70.google.com with SMTP id b14so6892209wrn.8 for ; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=yY5nuWFfvtGdjXx5X0jOTLVoM6UrkzhjnRnf2/dFDmQ=; b=VbbJwV5In/MFRvPpTBzLGAA3T7/XV+za3y/4JQgy4Nub8TJomJq//JdJAsdQgRfRUw VbaopRUD2AGanmkW1VWaWucNbxmqIevJ7DosvRKxMzmniEqDVpyjZ23bbqwX82+WbWOO YTNdOpjG7ExUSlCy3r9JMMpVq8sWuuVxivrC9wwgXrg+FY5xPqGJ+9em7U3Uqslq9rdz OFUUK/scHYPN/Wu9zkYitcfvCnpsfXcaUdxW5YZwiZEpXaD6TwpsU00P60guOwSh25Q3 vptb/QiFISV4y9ze+4+q58ZvJmuOle/Ij6pSedu0rDqxSD3U2LT16W8c1tL1EJ/AjPBT 7aMQ== X-Gm-Message-State: APjAAAV5kqklpxxFM1EVAPkKs2Al2WBg2Rk/i0XYBR0ulXI45I3/gaUd +KRSPmN7+ECTZ22u7jjeHngX0Bg9i4G20CM5cwVYF+xCRQcs6firfkBFgkL4b7612GLr+++Jy1D rd5XSf0oVMvVVpJn3wE2aGs8HMb2o82ANVyULAOnedwu829HThzfBeLrOsoszX4o+mw== X-Received: by 2002:adf:ebcd:: with SMTP id v13mr36750941wrn.263.1561410126329; Mon, 24 Jun 2019 14:02:06 -0700 (PDT) X-Received: by 2002:adf:ebcd:: with SMTP id v13mr36750908wrn.263.1561410125481; Mon, 24 Jun 2019 14:02:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410125; cv=none; d=google.com; s=arc-20160816; b=iG2mhDdgm/0r1N1Yn43w8IO6TKYBz1CdoXgzkm9pw3448vu46VW2uP01FBnfrBSUMB ZuRc2G5AV9BP0VysbK9UGtVRuKQEOoLMPhuYL9XC9BD2ySUprCmCOyAl7Axp32g5SyMy 7jRs02zUa5n1ivGgSsVeVkm9+wNwc/FznzHY/4SbD4A14hgv0Iyp4VPuf0np5nDIi/8p uHuP9ua7tleUi6iqkYj39ZsmABYX1gbEGO0kJpWWlMINgRjz4TWzct0V+Jz1jGQChbaV 5t1NkDZkuCr4PauUkYL5KNLXg/4UyftICYqdE79h7Lm/hzhDdRisglY4iumoJxWJCZS9 l86g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=yY5nuWFfvtGdjXx5X0jOTLVoM6UrkzhjnRnf2/dFDmQ=; b=im8wty/ZCgjPSF6GKYRpEDWw2mDdy89KPiUu2JWGGptMlLTO05tpMgaNgUUSjtIz/O r8pL6JlKq4kOzqwUcJNdaKgnElFonAG2MIEMBhj26QzNSswujuQvX0QFKzU94ieZH2ey FvUQkK1mB8GPLKJY2BzKiXhoFZBc3iRoXbuhmEx+Zvcluqarc3q5N942Wxm0hnjUwW7W w/mle07OOJd3C1CHukCw/auDLEz4NMe8VSiEhPlBmz32xBH3+zNCG2EgYurrcV+ENSbU ZpDIPLbF2EIvLaOm0BPZ0fxA8yrGc0Av+z/pJEeD6RUfxxoqGk1ChMsOSRGTt1obKLzG ivCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=F8IpBc2p; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id y134sor368769wmc.4.2019.06.24.14.02.05 for (Google Transport Security); Mon, 24 Jun 2019 14:02:05 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=F8IpBc2p; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yY5nuWFfvtGdjXx5X0jOTLVoM6UrkzhjnRnf2/dFDmQ=; b=F8IpBc2pCC9TRRcnAQyPopBTGejfFjN4O3manyf4o/rILQiqjMrQn93gZmQT0zfX3x 5Z+FOf3huG56JiHRxImMjpmYUHWLJoUWRxsyiLyvuER8NJ6yXAhiFhmoMeS3xXIFDdo2 S8cp9J0djUNYJQZ5sFZYQ/C2WzaQmeRBmrPgWsubS3c8Ie4OcoFe0BbQ1hX51zSmNWX8 jRx4QvWLtE9d+wLs0Pn2yh3tjD1LsHbOoGDIFKF99fyOS9uFazFhPFuBYcSfa5SLimKH 7oBiYCs7lU7Wqx9ujUyjct2anJPQXnJ7qZm2I66poFSkOkPL9DAIWEUtnwKp1WVc7xDc HnQg== X-Google-Smtp-Source: APXvYqyRMuYsh8UNUKwpWttANdd8lJKeiEYwmSXHHDDyx29FbNSbr+pndqxDZmcjLhM+HZNLBH9CVQ== X-Received: by 2002:a1c:4054:: with SMTP id n81mr17413906wma.78.1561410125081; Mon, 24 Jun 2019 14:02:05 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id f7sm6578766wrv.38.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6D-0001N1-4V; Mon, 24 Jun 2019 18:02:01 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 11/12] mm/hmm: Remove confusing comment and logic from hmm_release Date: Mon, 24 Jun 2019 18:01:09 -0300 Message-Id: <20190624210110.5098-12-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe hmm_release() is called exactly once per hmm. ops->release() cannot accidentally trigger any action that would recurse back onto hmm->mirrors_sem. This fixes a use after-free race of the form: CPU0 CPU1 hmm_release() up_write(&hmm->mirrors_sem); hmm_mirror_unregister(mirror) down_write(&hmm->mirrors_sem); up_write(&hmm->mirrors_sem); kfree(mirror) mirror->ops->release(mirror) The only user we have today for ops->release is an empty function, so this is unambiguously safe. As a consequence of plugging this race drivers are not allowed to register/unregister mirrors from within a release op. Signed-off-by: Jason Gunthorpe Tested-by: Philip Yang Reviewed-by: Christoph Hellwig --- mm/hmm.c | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-) diff --git a/mm/hmm.c b/mm/hmm.c index c30aa9403dbe4d..b224ea635a7716 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -130,26 +130,16 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) */ WARN_ON(!list_empty_careful(&hmm->ranges)); - down_write(&hmm->mirrors_sem); - mirror = list_first_entry_or_null(&hmm->mirrors, struct hmm_mirror, - list); - while (mirror) { - list_del_init(&mirror->list); - if (mirror->ops->release) { - /* - * Drop mirrors_sem so the release callback can wait - * on any pending work that might itself trigger a - * mmu_notifier callback and thus would deadlock with - * us. - */ - up_write(&hmm->mirrors_sem); + down_read(&hmm->mirrors_sem); + list_for_each_entry(mirror, &hmm->mirrors, list) { + /* + * Note: The driver is not allowed to trigger + * hmm_mirror_unregister() from this thread. + */ + if (mirror->ops->release) mirror->ops->release(mirror); - down_write(&hmm->mirrors_sem); - } - mirror = list_first_entry_or_null(&hmm->mirrors, - struct hmm_mirror, list); } - up_write(&hmm->mirrors_sem); + up_read(&hmm->mirrors_sem); hmm_put(hmm); } @@ -279,7 +269,7 @@ void hmm_mirror_unregister(struct hmm_mirror *mirror) struct hmm *hmm = mirror->hmm; down_write(&hmm->mirrors_sem); - list_del_init(&mirror->list); + list_del(&mirror->list); up_write(&hmm->mirrors_sem); hmm_put(hmm); } From patchwork Mon Jun 24 21:01:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11014147 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 67C3B14E5 for ; Mon, 24 Jun 2019 21:02:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 59DD7288D8 for ; Mon, 24 Jun 2019 21:02:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D8E228910; Mon, 24 Jun 2019 21:02:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 980A5288D8 for ; Mon, 24 Jun 2019 21:02:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BE478E000E; Mon, 24 Jun 2019 17:02:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8F65C8E000A; Mon, 24 Jun 2019 17:02:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C2308E000D; Mon, 24 Jun 2019 17:02:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by kanga.kvack.org (Postfix) with ESMTP id 213528E000A for ; Mon, 24 Jun 2019 17:02:09 -0400 (EDT) Received: by mail-wm1-f70.google.com with SMTP id t62so289678wmt.1 for ; Mon, 24 Jun 2019 14:02:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=S3r5edHB129w6+WHP2qce1cv++fCEyzuU0DGZG/FVjU=; b=Q90qIyYHsVGJ5kHDuvm2ZoNyMFVaOfzZqLQeAlECoL5a5brZaoAbDnAZZwOU0KwXb6 JPNpPCThvKv/7YHvtdqZ3/4Kg9RwA7oK49NCM1buaRU45whfdLHJRscTD+ytFfUitdia 0xalviR5VwzlOkrT3gqH2wrzfYJCYQsG99vAkCQAXmg/ucgsRLEC1hG4J3fSSNw2SUZH bEZ+ZQv16N9aX3JCGrh7MtnQrvRy7TL5wClecvQ9i7ya/HxVqd9wF9O4ainIj9eZXfxR P+yjD+6a7/o93MdKcDmqp3OxdzcHpGkkA7RGppZIkNDMSigTddHve7j1b2JQLJ4A8OfZ MYag== X-Gm-Message-State: APjAAAUSsbWcLmgdvzbsMrY9ZPl2YSe1yvbpZ89OgVqG/v4GP04h7zYi i7HPcFrWq8LI9br/X4+UlHv7IqNc91y2EoKkz70GZTQzk25f6l7n/5wHgrYTxwWvWbhXzlUz2TO AvMLLI1km46m9Bt6wK7h7Eg4De0e/IiJBSKlevFEMbMm1uhpAZ6n2EwHr0pmiTvoXDw== X-Received: by 2002:a1c:20cf:: with SMTP id g198mr5317672wmg.88.1561410128700; Mon, 24 Jun 2019 14:02:08 -0700 (PDT) X-Received: by 2002:a1c:20cf:: with SMTP id g198mr5317637wmg.88.1561410127628; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561410127; cv=none; d=google.com; s=arc-20160816; b=u4FwKWXssNZvTXJ6K79A1RqCe3/e7nFAQljqZMal7EomvMkXZAeR9mQLBK+L5UdDoV i75JNGBNmrQoN/Ea1kXZya3aR9lKZ6kfkjIofSjsEIgBAiMko3ZBmHPiqgwadxZvdPUE UJN1Gihj9tjiS6wgOBXoZZgGKJ/Y4aomdgy4u7zP57M/E5pCPuDFW9XeqB8JCpWW968/ TbtfKWptbf6QTTuKVY0MLdpVxdiWjifL1BfUoVIefARkBzXWZVu4C8UexFlvfX3ky3up OevbHTQER+WQghJoZLtZaibaXMvDgB0yi5UsZ5rapw70jTv5WSNEIblL5K+pV/XvlycI D9vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=S3r5edHB129w6+WHP2qce1cv++fCEyzuU0DGZG/FVjU=; b=SO8v/hZBxdXn3o1R0S6MUsTKqBb0F1ERiVdc30+mR0n4ZdsZzqmhpObX0Cbf8s1q2y pcZPCXiPH6zsXEP5xhNAjpGTPBiB/JLf8wKRRtn2nYp+2vbq2BlnOTxfduBM/RZnQ4Wg qUFEJQXUtcPBnk4WyE2cI/fp846NW7J+Dsx3cTQymmzLDPLVvzEejyqINfcvqCYVNW3V 14JL2162b1hn6KlKYoxLxkeH6UIK0ewpTgo72Bwoh/w+6soDKjHln3sqyc4+jo85zJQg 2lh2iHDeasbsUO6Vg5xzBTDnFoA6eKfbUMbC4Sn6K0DCdxXMKcyRoREtGszKEysj9hHm apDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=hxllCZWi; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v187sor374981wme.3.2019.06.24.14.02.07 for (Google Transport Security); Mon, 24 Jun 2019 14:02:07 -0700 (PDT) Received-SPF: pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=hxllCZWi; spf=pass (google.com: domain of jgg@ziepe.ca designates 209.85.220.65 as permitted sender) smtp.mailfrom=jgg@ziepe.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=S3r5edHB129w6+WHP2qce1cv++fCEyzuU0DGZG/FVjU=; b=hxllCZWingbWA/f5vgNASI2pGcmQkq+YMV0+41KqM+6SHez+NGQ1uJO082ws99YMd9 a02GfWbWiybTgkW5XKpp58ScKdfgpRyQkF+lx8bQgY/E0sad83Uy5xKeM+5fDXk5l+dn kREAcw817qhEHGbJzh/JPNg+S+obDw/5JZAk7sJkD1VoJOjy+A9WMNZt4hUg76s0LIwA 7CnwGTdpr/y52OLSiCtTuB/yhZqVQPS2pkaGeOalh4K4i+1u3/L+S9jsgkydPV6oBOsv +C7ZAWxMOYu5gNiQFVoJHSQaBp2JqH18fs4FjehP3vhAnvD2AIJi3c1yJBiWTS4qwHYX +fIw== X-Google-Smtp-Source: APXvYqyInST0pwmwLzJ/uxOPmS6zD6FzKVUhnIxI9FAO5JkPPLVhNSreFOU/LTKh9NST5RUcH7Pi3Q== X-Received: by 2002:a1c:3:: with SMTP id 3mr16798813wma.6.1561410127205; Mon, 24 Jun 2019 14:02:07 -0700 (PDT) Received: from ziepe.ca ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id l4sm411869wmh.18.2019.06.24.14.02.02 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 24 Jun 2019 14:02:02 -0700 (PDT) Received: from jgg by jggl.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1hfW6D-0001N7-5h; Mon, 24 Jun 2019 18:02:01 -0300 From: Jason Gunthorpe To: Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, linux-mm@kvack.org, Andrea Arcangeli , dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Ben Skeggs , Christoph Hellwig , Philip Yang , Ira Weiny , Jason Gunthorpe Subject: [PATCH v4 hmm 12/12] mm/hmm: Fix error flows in hmm_invalidate_range_start Date: Mon, 24 Jun 2019 18:01:10 -0300 Message-Id: <20190624210110.5098-13-jgg@ziepe.ca> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190624210110.5098-1-jgg@ziepe.ca> References: <20190624210110.5098-1-jgg@ziepe.ca> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe If the trylock on the hmm->mirrors_sem fails the function will return without decrementing the notifiers that were previously incremented. Since the caller will not call invalidate_range_end() on EAGAIN this will result in notifiers becoming permanently incremented and deadlock. If the sync_cpu_device_pagetables() required blocking the function will not return EAGAIN even though the device continues to touch the pages. This is a violation of the mmu notifier contract. Switch, and rename, the ranges_lock to a spin lock so we can reliably obtain it without blocking during error unwind. The error unwind is necessary since the notifiers count must be held incremented across the call to sync_cpu_device_pagetables() as we cannot allow the range to become marked valid by a parallel invalidate_start/end() pair while doing sync_cpu_device_pagetables(). Signed-off-by: Jason Gunthorpe Reviewed-by: Ralph Campbell Reviewed-by: Christoph Hellwig Tested-by: Philip Yang Signed-off-by: Jason Gunthorpe Reviewed-by: Ralph Campbell Reviewed-by: Christoph Hellwig Tested-by: Philip Yang --- include/linux/hmm.h | 2 +- mm/hmm.c | 72 +++++++++++++++++++++++++++------------------ 2 files changed, 45 insertions(+), 29 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index bf013e96525771..0fa8ea34ccef6d 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -86,7 +86,7 @@ struct hmm { struct mm_struct *mm; struct kref kref; - struct mutex lock; + spinlock_t ranges_lock; struct list_head ranges; struct list_head mirrors; struct mmu_notifier mmu_notifier; diff --git a/mm/hmm.c b/mm/hmm.c index b224ea635a7716..89549eac03d506 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -64,7 +64,7 @@ static struct hmm *hmm_get_or_create(struct mm_struct *mm) init_rwsem(&hmm->mirrors_sem); hmm->mmu_notifier.ops = NULL; INIT_LIST_HEAD(&hmm->ranges); - mutex_init(&hmm->lock); + spin_lock_init(&hmm->ranges_lock); kref_init(&hmm->kref); hmm->notifiers = 0; hmm->mm = mm; @@ -144,6 +144,23 @@ static void hmm_release(struct mmu_notifier *mn, struct mm_struct *mm) hmm_put(hmm); } +static void notifiers_decrement(struct hmm *hmm) +{ + lockdep_assert_held(&hmm->ranges_lock); + + hmm->notifiers--; + if (!hmm->notifiers) { + struct hmm_range *range; + + list_for_each_entry(range, &hmm->ranges, list) { + if (range->valid) + continue; + range->valid = true; + } + wake_up_all(&hmm->wq); + } +} + static int hmm_invalidate_range_start(struct mmu_notifier *mn, const struct mmu_notifier_range *nrange) { @@ -151,6 +168,7 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, struct hmm_mirror *mirror; struct hmm_update update; struct hmm_range *range; + unsigned long flags; int ret = 0; if (!kref_get_unless_zero(&hmm->kref)) @@ -161,12 +179,7 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, update.event = HMM_UPDATE_INVALIDATE; update.blockable = mmu_notifier_range_blockable(nrange); - if (mmu_notifier_range_blockable(nrange)) - mutex_lock(&hmm->lock); - else if (!mutex_trylock(&hmm->lock)) { - ret = -EAGAIN; - goto out; - } + spin_lock_irqsave(&hmm->ranges_lock, flags); hmm->notifiers++; list_for_each_entry(range, &hmm->ranges, list) { if (update.end < range->start || update.start >= range->end) @@ -174,7 +187,7 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, range->valid = false; } - mutex_unlock(&hmm->lock); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); if (mmu_notifier_range_blockable(nrange)) down_read(&hmm->mirrors_sem); @@ -182,16 +195,26 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, ret = -EAGAIN; goto out; } + list_for_each_entry(mirror, &hmm->mirrors, list) { - int ret; + int rc; - ret = mirror->ops->sync_cpu_device_pagetables(mirror, &update); - if (!update.blockable && ret == -EAGAIN) + rc = mirror->ops->sync_cpu_device_pagetables(mirror, &update); + if (rc) { + if (WARN_ON(update.blockable || rc != -EAGAIN)) + continue; + ret = -EAGAIN; break; + } } up_read(&hmm->mirrors_sem); out: + if (ret) { + spin_lock_irqsave(&hmm->ranges_lock, flags); + notifiers_decrement(hmm); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); + } hmm_put(hmm); return ret; } @@ -200,23 +223,14 @@ static void hmm_invalidate_range_end(struct mmu_notifier *mn, const struct mmu_notifier_range *nrange) { struct hmm *hmm = container_of(mn, struct hmm, mmu_notifier); + unsigned long flags; if (!kref_get_unless_zero(&hmm->kref)) return; - mutex_lock(&hmm->lock); - hmm->notifiers--; - if (!hmm->notifiers) { - struct hmm_range *range; - - list_for_each_entry(range, &hmm->ranges, list) { - if (range->valid) - continue; - range->valid = true; - } - wake_up_all(&hmm->wq); - } - mutex_unlock(&hmm->lock); + spin_lock_irqsave(&hmm->ranges_lock, flags); + notifiers_decrement(hmm); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); hmm_put(hmm); } @@ -868,6 +882,7 @@ int hmm_range_register(struct hmm_range *range, { unsigned long mask = ((1UL << page_shift) - 1UL); struct hmm *hmm = mirror->hmm; + unsigned long flags; range->valid = false; range->hmm = NULL; @@ -886,7 +901,7 @@ int hmm_range_register(struct hmm_range *range, return -EFAULT; /* Initialize range to track CPU page table updates. */ - mutex_lock(&hmm->lock); + spin_lock_irqsave(&hmm->ranges_lock, flags); range->hmm = hmm; kref_get(&hmm->kref); @@ -898,7 +913,7 @@ int hmm_range_register(struct hmm_range *range, */ if (!hmm->notifiers) range->valid = true; - mutex_unlock(&hmm->lock); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); return 0; } @@ -914,10 +929,11 @@ EXPORT_SYMBOL(hmm_range_register); void hmm_range_unregister(struct hmm_range *range) { struct hmm *hmm = range->hmm; + unsigned long flags; - mutex_lock(&hmm->lock); + spin_lock_irqsave(&hmm->ranges_lock, flags); list_del_init(&range->list); - mutex_unlock(&hmm->lock); + spin_unlock_irqrestore(&hmm->ranges_lock, flags); /* Drop reference taken by hmm_range_register() */ mmput(hmm->mm);