From patchwork Mon Jan 9 20:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADF8FC5479D for ; Mon, 9 Jan 2023 20:54:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DCC98E0003; Mon, 9 Jan 2023 15:54:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48CFC8E0001; Mon, 9 Jan 2023 15:54:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32E968E0003; Mon, 9 Jan 2023 15:54:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 23F548E0001 for ; Mon, 9 Jan 2023 15:54:54 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 03F10C0F33 for ; Mon, 9 Jan 2023 20:54:53 +0000 (UTC) X-FDA: 80336464908.14.AA328D0 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 5EC1014000B for ; Mon, 9 Jan 2023 20:54:52 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=T55Sv4TW; spf=pass (imf26.hostedemail.com: domain of 3G3-8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3G3-8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297692; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=aLSKYTjSglZhgpp2YZut65CdLFpYZl5+h1A4PBRiMPu7Lg2aO7gVyQ3Yv4KjlLEyc5wO3V sOMSriOA/UZ8ncj4kQ00nxgq4b1/5wTm5zWC2l1CIibAI3KZpJTGSh2HS82rUpmu5gYNcl J2VfPGMiydr+YPVM6KYCwIppwbXeNJ0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=T55Sv4TW; spf=pass (imf26.hostedemail.com: domain of 3G3-8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3G3-8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297692; a=rsa-sha256; cv=none; b=e8uwJ+eHyDewmbvHPPrHeS1t7g8KuMX76Ma/KId6rILkOrEVUD8wjo1dUA9wCcQgMbVaW3 BGFVrJNAaZIpXCey8FJ40iGxVz+p4JUKbz0jeHBpUKgZyg49mr8b0czQqwdSEnPyQUveHn jC4eNCReAj0/hsvQKxE7+9QCWihCCz8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-349423f04dbso105151677b3.13 for ; Mon, 09 Jan 2023 12:54:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=T55Sv4TWznVzBLEvpmky/jAR0QJoeEeCX90o+hjvZVdyHXxi4X+dLVqm78fwsxY8qM 7mstQqJ2q4Sn5cjstBMo6qt6HReHOpVXqnm5LMKSQ635M0MCSPIfIgxAkOTHc8oB12I/ ai6kXEsVt00A/xCygYMywSMRtpTqMTCmAJNGp6/HS0isXJekWM6FOqzBH2+trv87L4rS u+h7CTRZzB07Ow9ioaR2euNV+QdGF1fgPh0hG8fHh3bLFfxrE2kcC3qaflJTUkk0dQTh 8vb62xpka2Ra5fG5NjEzQJYLTJMKCpwH3JBwVijZjrLZOBQ13qNu1ZZEA2iHn/6uZCd9 oE+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=gtSBro3W5+d/vhM7vwE77iFChU1Q30Kq2/jC4NxWZW1dPQOPfgubeqcghZlHSf5V1v 5pFtheW5qw8O1QrkxEISv1BTQPyf7n/JWovMU5gle1FBJCygHNoYhqCz3SF/CbhKpmgM 2VQD8vsEzUAon1cZK+ONB4jK+c5FrwhKHC9zafyrKA5ZFU9Ius8d6OIS4DLRhM2rvPVp tXaztTDw+2KznuT37BFqSZREsbxwbEuHQ/ICtobM8wgrXyJ6mSElc++aWrq7zOIwlymA V0orjC90cmmo+Fwc7Hvy4Ql+IFHFnAsEF/rCQE+7QkwosxL00INEbBEr+s7nKU2id3me QDrw== X-Gm-Message-State: AFqh2ko/Fuol29KA6OdvV/LiDmAEZyDnRS39lMKDlApPGlgrutHEociZ dBoNXqcCDbyMpUEfWDbB8DMtA3dnqS0= X-Google-Smtp-Source: AMrXdXsg8SKnvt1lG56Fk9uUJaowgWL0vbqBlxMLAmQhblw+uvVMcBKNRkD6OuTHfHWzX2ebIh14gNZU+Cs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a05:6902:920:b0:781:eeed:9abf with SMTP id bu32-20020a056902092000b00781eeed9abfmr7003326ybb.538.1673297691543; Mon, 09 Jan 2023 12:54:51 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:23 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-29-surenb@google.com> Subject: [PATCH 28/41] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5EC1014000B X-Stat-Signature: s731aoyy3z3xzhbnh5r946hr3ptszjzs X-HE-Tag: 1673297692-566057 X-HE-Meta: U2FsdGVkX1/X4m6Vm83b6ozKcdKQCqL5MpVtBjOM2du/ykWsRDGw9KqXDFO/hkf1oBGDLVm2F/xitLpsR/xBAwc+1BT1B9z2mdismQokGrf6c0MtYWejQMhqLw1JpmqJRf0iZ3MrSlnuQTO0rkVfpskPg0A1+7m65UcyVUPblFM3ptFZBKtPOKzNaycKfqHKOEpc33k2x8jfxjP9Hven6uFsIitTfJitzJ60fxSsQqDpWPHzSqNGx8BHm4BXWlBz3qFzMsJxL7emDUAJeD9roPgvw3UUign/Ay2CJcvL3GQvDy0J6drZeZpP3tNPj2Vc6XrpdZNVj1nxxiaJKxscUXj8VeTjevBAhLnP4Jq5/q/OjO/9WIzpj/wCE4nby8JhUkwXxFLhhWcV1s8mXbegjMMv6p78rs5LA02q8qlYWRrERxty5tv438eCB1gD4/jTwdTPH/cT+kpJyOGy2gIyOfob6Fexk/E4FcRg9w3Ta1qmbgtv9HBRn98LMsoHSNnCdc4l/fO8xHczfTYZyCbtYh734E+okiOOyp3/Rbx9+0gl7if45ep513mkn2MAnCUMzeopmB9eJfMo+FES2giWH3OJXdvBDKKxC8uUlD2/jJcUqHA2wwsvc4zN0StHyuysXSJh2iOglURJmtRvuf2rSzFJgc3yGxUu5jcpUBi87tY0xB5pI/xBzjjlGY9rATW1K1NLH0QIydKz+RLA13YcDINsbuRmp+z2ruLbQrnSTREJRXtgiQKoI/Mzhc3PSauag88LQ3sOPdq8Iyi1MYp6jjtbbQq2XFPh6BbPwn29KrGXyWt//gtf9ubecLnIYJo25tzvi9FuBt2Ykr8/UR0b1rBb6twJGhpm4tN0YpeX6LDBZ8PNLWNbqOD7Bvzwp2kQQuR9pEj/GpP0wr33fFwG8H3ubuygE9kDq8jqyLyn/kFB+K9ztGCu83tWcJQXWoLa3sf7WgTBIIj8Gb4F1qF LHMAKBbi jHDeJI0sMdDuQHaV65JvLEe2pNLrxo2kfPUrlw7OMz28mz+TCfOd0wb6Q3Eqc+b+G+Gea/ESCrxfVBH+22wrnY+CtDQQV40HSjJfh5VOvx3wmBpyv+HrwO/T5MZ8i+ftBN/r9lN/cX6m6dtSrxpMopCt32pg2WU75g1Zeaew11QD/anqcIHy7522zZHY+uX4uGez43pgx2JoHPDDZfIzlEafueFhGzS/Q98vVi/OaIaoZoAYV6SiV7s0NSZtVfN/OAdFXFDWhPUIeQK2oxnc6PVON2GsJ5LqelVOhbE6pPpClF3yPlA1hie7iEC3G+2iBbsmUhS5xvj4Atl++iGMKBWJWK5BkQR96GL39t2dDa1bSAW6pYNGKUzxzcoQaFIljM1tKS35CfWHjJgS2PHFK6wk6JcqB3WACcXnF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index c464fc8a514c..d0fddf6a1de9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -687,6 +687,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 9ece18548db1..a658e26d965d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5242,6 +5242,57 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma, *validate; + + rcu_read_lock(); + vma = mas_walk(&mas); +retry: + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_read_trylock(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_read_unlock(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + mas.index = address; + validate = mas_walk(&mas); + if (validate != vma) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one. */ + vma = validate; + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table.