From patchwork Mon Aug 29 21:25:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3EF9ECAAD2 for ; Mon, 29 Aug 2022 21:26:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 665D5940019; Mon, 29 Aug 2022 17:26:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C6A8940008; Mon, 29 Aug 2022 17:26:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A6D5940019; Mon, 29 Aug 2022 17:26:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1960A940008 for ; Mon, 29 Aug 2022 17:26:15 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E8DC9120B0B for ; Mon, 29 Aug 2022 21:26:14 +0000 (UTC) X-FDA: 79853913468.18.7511B87 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf02.hostedemail.com (Postfix) with ESMTP id 9D41080043 for ; Mon, 29 Aug 2022 21:26:14 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id x8-20020a17090a1f8800b001faa9857ef2so3895191pja.0 for ; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=WI+iAkKYdp/dw+q+sE6vVFwD5mLSWBWMhRHboF9Ttbardy/unLndkxHVrzFVqulr1A yPcJLa3Ur0z+PJer1B1IamGoCGgxEpMLeAxgWf9CgasGK7cypr9FW+usqow4Tv90c3k6 hSoUgkPn+GgzurgQcPzRD2rwA/PJHxIAlhRbX6toXLNmtMd1EOWFLhXluJKlLDk1UBbT O0I6rKmCm2uwsJ4JnyJZiGAk3wWQ6Qh6RctA9jkOKMvzLM+0NSmJ3ovBUaPQyVoGm6YI JXf6mvqPKdqWiUAL/+XE2dov1V39lipJcNcZcI/t4fYcyAX1GpBk//UErawM2Q1iWoKr uzug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=LtbT42ENU5gV7KQWm11ZCul51r/6VSIUChhRCl5LtXnxMxiL+PnGFOL4pnW5jI6nYY Z5e3f/Y5n0RJ/9VI0+BQ9ww1T1yCjBTpfXJzjeWSxjh5K4Q0fWkvuenuyvBlqSPELr2G NA7BXPn6jQCbBAfTNCiRTHRUz7q5TlX3qTlw8w67z4eiNC/UTEjwxwHqjKjbOp2rE4uo srGLkCg3zTzFJePUVNHKQFWImzAKtalYaOIkeYxhu7zOn5P17BAwiv9Bny26Urdn74JT Weh9nSEKkxWXDeDkZ9zXUCQqHEAK99/hMR12w+HVCyNsypAlPwD0PGER/atjXjmcchcD aeZg== X-Gm-Message-State: ACgBeo2b+paQJjSxTYCChAyWVs2RY0BqoNJ1MLDdLpN1fr8VCme/u+eb zlCqBjtIzYgniYVz2K5Tt+WPGUN/YFg= X-Google-Smtp-Source: AA6agR5aic644PZ0OuL1YK8O5lnv8GmzPsmAclgLgLZvZKPuRUB75AtHpFEjRVbtgx264YmpiRiioO9wBJc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:249:b0:1e0:a8a3:3c6c with SMTP id t9-20020a17090a024900b001e0a8a33c6cmr21187pje.0.1661808373054; Mon, 29 Aug 2022 14:26:13 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:24 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-22-surenb@google.com> Subject: [RFC PATCH 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WI+iAkKY; spf=pass (imf02.hostedemail.com: domain of 39S4NYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=39S4NYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808374; a=rsa-sha256; cv=none; b=6JV3cnkVvlOdJkUhpgGTkqy6mnSabB9v826vMNmK+gyi9sZJP+X1Wav7g+Etpmg3QmBCH/ Q4fm8SC7Vqe7G7/7eAjjhprpjWAsz/R3SMvbhu+GOzHr50lHS+7DneqeOEjCJoPFqXWcDW ABcicT91Q+Pnb6Zj6ezxcXILFPeSz4g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808374; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=IQSN/GyHADhQl15P3a73DyNcRweZjMfD6/kThi7SFaHrtXArhhm3bqd0J2dEpa66UT5W9M ChGYq8Yx1UPqZ/Mw7Hv2f64S/CMu64hfNd1kIcMu/DD2JqE6QVIHzg6t3iJZStWcyBd+Ij fm0wwaBBUC2b7xF5FPCTPsvY4IkwmTM= X-Stat-Signature: f879kk7qbw5cbonuxhya71bj5ryjazgs X-Rspam-User: Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WI+iAkKY; spf=pass (imf02.hostedemail.com: domain of 39S4NYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=39S4NYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9D41080043 X-HE-Tag: 1661808374-886945 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma = __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be stable + * and not isolated. If the VMA is not found of is being modified the function + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma = find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table.