From patchwork Thu Sep 1 17:34:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51F50ECAAD3 for ; Thu, 1 Sep 2022 17:36:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+XbGUXwBJufe6oWqAP1baH/SsrbvKG5GU97C+xaOyqs=; b=GdKaM28eFW6VncBIGDkQmAlXXy A4o153f8ASG6BNupjjUdZC1zFj65+4ZCmfxBvsUPKWDweZPLNDLBo84ZEP1n4vKud4tIrPMg9YcWb FmIWvMhZx7MQnvWai59/FCNHheRVKNt3exHZy19o/XxXizU1XVb942iVa9uKvnY2QgYCbtyPGaQhA DHPcY5IC8YUqv/zng/L75c4x+TgHmqEcb9+LPVxE2XWFzzrshtVyqRlQNGBEx01rB8D7TnWF51biB y88cIUUyl3EsoT01XmOZ21cr3JYm89RCH3o2x5aD8qv5hZ/0CnZQyKflr43Ty1zWT9FbnfFCjsqM9 nJUINQXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6M-00DbPv-T7; Thu, 01 Sep 2022 17:35:39 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo69-00DbGn-Cd for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:26 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-343a77f2129so37847597b3.15 for ; Thu, 01 Sep 2022 10:35:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=SI5DHC7dpiRJVHhhWYV+lwBh2iwV9A0Ou3FEYYZKTIo=; b=ouDtFH7Jsoj5d0amZ6w++vn1JIcEBsAg0+Jc3jSu1CI/YwtCjIjWCXa3h1daFKHXKc cShsEzXCG6pl7Qo8JUwfzrGY3LljVOQU7WP+kQRk9jvRvgn5CYGLdVxDx9mvrs6u6z8K /gtKC5pqelI7Ja6YPAH2266yNgoWGngIuBTy9e4Ky4U5csa2GQgfjUrkug/tPpsGFles yx5i2EqiywQtBnfaWAJzlm/RHBLABr6GNuXXMOHeNq811kfvPtT71YKWoN7/bLzrrmPy z0tKwDEKWW16qZuqOrYHGA9KvA/VUBzIFMBe0AdL5oZh7HL1RXGViqdJIuZcrmP9S3mv xPFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=SI5DHC7dpiRJVHhhWYV+lwBh2iwV9A0Ou3FEYYZKTIo=; b=NAwhXIqnUa3t+zzVimXSQaYx7vI10up3dx1HK1B+u+ZcG3W1rhwYbZLvFWfkiA0Ly9 4QvjlKlee85z3DcqT9MWHPB2u6s3C5vC+4EWnGz6m2tQkahzjZNXtTQXcq6bkxxDQJ5q 4OOauuF5V9YUS0Q4f98eVXmrC83QKjUrCVPF/4y97glbPbbcVfPiyCGWr5J9/d7VvjfH WSXAKpS83+x92i9rKLOBqH+Vdx/UHWOuKTJR54uXqfbRm26Qp7hkd+p1HiA0PrICmVC2 BqhXzv0c3YgxprQmKdHCHFMooEJMN44+aW1HEYclaSvN+UV4PJVTk90DSrQfQrNCLLF3 XBwA== X-Gm-Message-State: ACgBeo3V0duTb+oEUuiAGp+AfZtiJ7xOne+enMtGr3PQs8PRhwsmsVIL 4xFjp6lE3L5c4bT0rLRQc+J8Lq+mZmU= X-Google-Smtp-Source: AA6agR4TWv71WCFs9L88lUEt8sy6dDZVYBPPgP4/10a2Bx/Ool+t8WQWT1a/Z5ySDpeCnHjOLWwcwNW6snI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:8e8f:0:b0:691:9579:d3c5 with SMTP id q15-20020a258e8f000000b006919579d3c5mr19748036ybl.249.1662053723940; Thu, 01 Sep 2022 10:35:23 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:49 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-2-surenb@google.com> Subject: [RFC PATCH RESEND 01/28] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103525_446726_8960A47F X-CRM114-Status: GOOD ( 12.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 0331f1461f81..58c20fad9cf9 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1124,6 +1124,19 @@ config PTE_MARKER_UFFD_WP purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + bool "Per-vma locking support" + default y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Thu Sep 1 17:34:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6BB99ECAAD5 for ; Thu, 1 Sep 2022 17:36:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=B0nBj8EMVlg5mxVgSbI9PEM02AQrHhW5D1WC8laZ98s=; b=bHtXSFC2mDajebgiK2stPszBD9 nOGrxQqoZLEzdAihDx0LQ2YbUAooG5pGb7N9QpfX56QMi0YRitDmOlLBQQRDh1yNqh06HMvRuI9re lbUQ91kaQ+xZdL2BhIf6XY2c0hP+5b1EcJkBSUCvyZ57vS0QkgwrW732iFe3ILG3Q0brmqNayFTRa fcl74XmDfqtw2G1cAmgVsgMWFkNcGG9ZlapX90vRNe/44z2K63IdvK85aNU8q3pWbOCgeu8nHd2zl r/UEHctIUj+eUrFe3w62nzKS4QfsAcKFm+Ai/vPxVGvWntLZFpObhVmZtT2oxhTy1PGRS7YycZw4H h8r89pAg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6W-00DbW5-U1; Thu, 01 Sep 2022 17:35:49 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6B-00DbIp-Ul for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:29 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id v5-20020a2583c5000000b006964324be8cso4905211ybm.14 for ; Thu, 01 Sep 2022 10:35:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=p6utfGSGU2ZWrUa6c0eUAGSz6ILgySPKVkHsaiIDl8M=; b=LbkT/AkX1WcT/2fyJlqSXegH2trsBNyd6QrbYfZ/YY10o0bOfV+Ix+uJLegE6lSF35 ne85BxCuyx7lBpnSiZ42TY86RYtZR3AAc5vYrFsx4gBxgTwamIONmdTxtR/0iKqdUJEh gCGpqjCaOQV7LzbqDjbUJKDXfAR5/j45YmmGmhxu+HXk+Hwd00xyXauy+PEzZWtLyw4U rqIn+41hVfJNUiIovZy6pL0FSV03X1G3rIrczW6PRcfgw8lXvG9gGaQmr8gUZrra+vXa CCuSzq9IAYSPtY4Wqdt70146omIRNbSeU/rl39Xw+3e0A38BgKuGXtDi12EMKnqTbGy3 VnZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=p6utfGSGU2ZWrUa6c0eUAGSz6ILgySPKVkHsaiIDl8M=; b=gxBW5TP0fc7fB/YJqzNcEa7P4Ed/qQ/aQ1pxQ9rLcn+wBUFJMFNS5+wdmo2JAVRlWd FSzGZ3jdxbsi6I3Y3UU55hSCet0WEHgOzSvK60vZZeAGbc+kbHj4Yr7aU1Xc3w6xi8z+ dgthjmYEGa4qchPpEdahIy2fYCM4kV4qR/UQhMzrd29oN03IGdFNLQtLsqlGPpvKYh2a VgkU9yFllQDw2+wlQX8rUFr3EGB6TsF7PCNwO9UvMxPW1P0NledZfJge0bgwhXhEwUxL s2XAYy7iZZBcqYLOJcmsOZ9Zvc+8pxuwoIywPv3SEa/3CyJ7m5OiJnoaULDTDHXZoq4U FYGw== X-Gm-Message-State: ACgBeo2W7JL1PYryJFcqmtkJ2OmizKo8s+bKUB218A4kFkUCLZWbugct 20/Q8GfwAoMuj4JJolP7RordTWeTh3vaJv04Kc8tCWQd+lObseOwyuDU/6509a5CiWtfnToOMWf gezX7G+bn3BTp7M6mctYD3d34yS1T0kc= X-Google-Smtp-Source: AA6agR4JzlVLX4tUxHHNIbaNOZW1dz2wXadaTbKwNizSq+Zw9LDNKeWUkMtSnrbieYwXe2AgTTBR2su3VSU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a81:5cc4:0:b0:327:59f2:d673 with SMTP id q187-20020a815cc4000000b0032759f2d673mr22886370ywb.272.1662053726626; Thu, 01 Sep 2022 10:35:26 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:50 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-3-surenb@google.com> Subject: [RFC PATCH RESEND 02/28] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103528_023664_2A039B49 X-CRM114-Status: GOOD ( 14.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 16 +++++++++++----- kernel/fork.c | 13 +++++++++++++ 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index cf97f3884fda..bed25ef7c994 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -403,12 +403,18 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start, vm_end; - /* linked list of VM areas per task, sorted by address */ - struct vm_area_struct *vm_next, *vm_prev; + /* linked list of VMAs per task, sorted by address */ + struct vm_area_struct *vm_next, *vm_prev; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct rb_node vm_rb; diff --git a/kernel/fork.c b/kernel/fork.c index 90c85b17bf69..614872438393 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -481,10 +481,23 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +#ifdef CONFIG_PER_VMA_LOCK +static void __vm_area_free(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + kmem_cache_free(vm_area_cachep, vma); +} +#endif + void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free(vm_area_cachep, vma); +#endif } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Thu Sep 1 17:34:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69A88ECAAD1 for ; Thu, 1 Sep 2022 17:36:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kdz8PODI2i7C8I7+ggsuEHH2RWlzJUT6B1XhwGUJFFU=; b=SUrNgK+QsmJIdHdkIwvvV/BbEx u6mJdQFp8jVoTRhHbKtSOU67RnvHa/mLuJL+6u5xc1GETxlC5qzdcK13IxWxVMZkIkvANU4rwGTPb 1kZ5KrOlbml8yE7zphkbSG5cWYfBENHdVRa0rh9EIqaQdp14VQHzt6opCjsNcHBqgQPrJPz1Z5gZU c1VYYtsR35B2Dfmqmit68sUHZG1zNp7XWRFf/VsYF0jquNJxq/0sDlQu1rd0l2/AlES0VMe7wYb5G +0tV99tQ2R9g/1SvudWh4FYSrhjeSlf45+bUuRkMek5L4NWSlktvX00S9GgT4Jp3VkyoF2GLfv/fb HuWwb75A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6g-00Dbaj-Jz; Thu, 01 Sep 2022 17:35:58 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6H-00DbKD-2M for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:35 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-340ae84fb7dso202781687b3.17 for ; Thu, 01 Sep 2022 10:35:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=LVrV0vnZTvQ/veouCVIJ2VKTA1Jw28r4MCl6DiRcaDY=; b=l1V4yVONtNOUfesG20Bnq72dRtkCeRcz+SXGtADVXwnLsewU/bV2Wj+5K0ek8rbJL5 /VnIb4yjvzQAA64ppyeiv2tN7SgqNypFCNDrFPFTIYKrt4K6WTB+yC3yHFvMB+ausBHt SSGrmorbtqBk+v9sXJa+cknHYK7SdsAdZfIlHKdNs08dvIfwUr5y5G+bCOuk9GDx3jZe o3Fb422GusB/mLwU2fWqp5J3gMNLaE1B/UaEPdikTdaz6ywBZryiSPL/0bdT3F8UE3iA 1RBsfylmTB6McidRxeHwpl546i9VITPOIoqZ/5tnno4M6ww922Mvm4adk4fXxTHJ359z mtIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=LVrV0vnZTvQ/veouCVIJ2VKTA1Jw28r4MCl6DiRcaDY=; b=A2DX3cELwkTUSVK4ldTW+jkenz+0CEc0PDjI+1jRen3O0qMDAY7i/d6DaWHoflGEdU p+97Hay5NDsYNbITCVgIBF1vqAM0Z/oEbQLV1L3mhyvQLqq1qG1iAcBiCBIN4FM8mo6S pPjjSXDtYRhgAECxGNDTOrHP51bkMzrT1bvCBBQ76La/fpIi2KTIp2A8Avip8uW2EfSJ bjFiBgGkNpGDln1AR2VXIN+QQHJC/ztZGdqrqy5EfEsskOG2Jnt5XHOaczoPVPXDWZjV RAJrlo9XP3YZSOxf0xAWRtNW+bbppW9xZpj3GbMCcrgtxahkCG7nva4ZMURpEvQYWuf2 khnw== X-Gm-Message-State: ACgBeo0OK5Py1B+riFJxgXv2YqtMuGCVSUqse7AtGEAvSuGMk3tVKEQi zNroBOMnNKzGsxdZ5OA4lJeqjbx4GGg= X-Google-Smtp-Source: AA6agR5xL9cSC69c+1y/uWKTqLFokWxQZenGw3Ln10dS2nyve2acvx9iJCiS8dOE4h1muByL940MlpUFLCY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:25cf:0:b0:6a2:fe22:657 with SMTP id l198-20020a2525cf000000b006a2fe220657mr784117ybl.481.1662053729233; Thu, 01 Sep 2022 10:35:29 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:51 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-4-surenb@google.com> Subject: [RFC PATCH RESEND 03/28] mm: introduce __find_vma to be used without mmap_lock protection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103533_120945_E016263D X-CRM114-Status: GOOD ( 14.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add __find_vma function to be used for VMA lookup under rcu protection. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kent Overstreet --- drivers/gpu/drm/i915/i915_gpu_error.c | 4 ++-- include/linux/mm.h | 9 ++++++++- mm/mmap.c | 6 ++---- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c index 32e92651ef7c..fc94985c95c8 100644 --- a/drivers/gpu/drm/i915/i915_gpu_error.c +++ b/drivers/gpu/drm/i915/i915_gpu_error.c @@ -507,7 +507,7 @@ static void error_print_context(struct drm_i915_error_state_buf *m, } static struct i915_vma_coredump * -__find_vma(struct i915_vma_coredump *vma, const char *name) +__i915_find_vma(struct i915_vma_coredump *vma, const char *name) { while (vma) { if (strcmp(vma->name, name) == 0) @@ -521,7 +521,7 @@ __find_vma(struct i915_vma_coredump *vma, const char *name) struct i915_vma_coredump * intel_gpu_error_find_batch(const struct intel_engine_coredump *ee) { - return __find_vma(ee->vma, "batch"); + return __i915_find_vma(ee->vma, "batch"); } static void error_print_engine(struct drm_i915_error_state_buf *m, diff --git a/include/linux/mm.h b/include/linux/mm.h index 21f8b27bd9fd..7d322a979455 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2712,7 +2712,14 @@ extern int expand_upwards(struct vm_area_struct *vma, unsigned long address); #endif /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long addr); +extern struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr); +static inline +struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +{ + mmap_assert_locked(mm); + return __find_vma(mm, addr); +} + extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr, struct vm_area_struct **pprev); diff --git a/mm/mmap.c b/mm/mmap.c index 9d780f415be3..693e6776be39 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2250,12 +2250,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, EXPORT_SYMBOL(get_unmapped_area); /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr) { struct rb_node *rb_node; struct vm_area_struct *vma; - mmap_assert_locked(mm); /* Check the cache first. */ vma = vmacache_find(mm, addr); if (likely(vma)) @@ -2281,8 +2280,7 @@ struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) vmacache_update(addr, vma); return vma; } - -EXPORT_SYMBOL(find_vma); +EXPORT_SYMBOL(__find_vma); /* * Same as find_vma, but also return a pointer to the previous VMA in *pprev. From patchwork Thu Sep 1 17:34:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12962999 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A8B43ECAAD3 for ; Thu, 1 Sep 2022 17:37:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ipXSV+v03npSB1uLTu43+oTA4ME3hw5EsSRmZEb138k=; b=0sBakXkHxjhEW87h68m9PYntFR b6nn94FTt3T/RoFJZlqdCBker/0Zd4S7Kk1sNxoTllrG0su63F6cwI+fz5JNMnoSyoOEeet93CcuJ OOFsry7/vTWR/q7ZJwT1N86x0crkjxoJjDVKLiRzP5s6xSqFqv5nhQn3uZ0hKfmwEOzeEoDP+reMX WpCic5DuCccud8s6N28V/HCxw/Udq2uFaMF2UTBcRTtQNvl66dXdhQRMHI66SuJMvvughg5ppj0DD 7hDPeLpwhVY1Wi/Gs/8mvwW5AXNwaASJs+7TzyyJnInlnh3Xnn93VlwMDFL2I8kucnxbKf/f08jvP WIHu+gyQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6z-00Dbqy-Mi; Thu, 01 Sep 2022 17:36:17 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6M-00DbOw-Ch for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 17:35:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=SmLlv5S95s1SYD/hq46yY1KSJ7CCwWbvNekA+AzNinI=; b=INos90YTuX6AoIiZyRrPYeqLGB rjabHJ7/CuWjO44Na1i67BIQMVY96hiyN8rGf+BUkV0Y2ZBYk+KWLE8kWLGA+mjFc84QjGv0a7pHB +tzcDazi5oPvf7TYzRYDBiSG1Ti6AYol9LoWbITII6UqlUhK/l4iF4oVerawLkT8UmVLmh77bHO8Z 5/IEFSRPbRm2sIE6zEBj2HkSbCW2vJG/sQlFaRp6vQQ0rZMtnW3UFR2gyn2eWD+DF5gs0la4jgNcW xSxA1/y0MZo4+CnPJXCO7t/17ic4B8xf4HjzeJU4uCK6QZxxLynSCwZcjIbOYPHh7w+JeDOOKCK4+ tNWon64w==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6I-008RcA-HP for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:36 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33f8988daecso226840387b3.12 for ; Thu, 01 Sep 2022 10:35:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=SmLlv5S95s1SYD/hq46yY1KSJ7CCwWbvNekA+AzNinI=; b=IEnDpG3YZTPCaLyxObjk7amCPTURBOnqU7bJ+HMdi+ub5B49wgcvI5S5rtmD90rv2Q AZvqZR25+q4mnRP5jUchvAffVc/FzpkmhNDkv4FRnlMM4OlY6itcjxw3UZXpfpnem++C EJWIMKaiqyYuCH4R4X9b15IgDB9xODniSPpGeOsiXfRmLSll0Bi1jUHU15lpXlXRSc5J +CLYoEdfwTQrLCZejly8sqYsr7A+RsOmJlAwK6fhbCZlwscLc4WTaIF87eXTXBY8BoJB i633PKqUr/jul5ZT9UiyTCl07dJq2r9+I6Xxo/FEUAli95MNIWyATvVTZ7NKsjxM0gu/ 3dZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=SmLlv5S95s1SYD/hq46yY1KSJ7CCwWbvNekA+AzNinI=; b=TWe/N7nk4tPBvdovYbBE43Jh5eWJlDRTgpz3VCpCkkK1HZ5Yo9pHJIhiypSDaefO9Y /mApINIel7aMlOhZfGcl3OG6VGuQfYi6uBTDASuAOmAqL8wJNqlYQN2SuXJyY13Wu4Vl WYCeh/x88t5yb9KTEqZRF9l7w6sSMhhUJBiHHkwguPAbku7iHulExeJpvRrl5Vosk1ta ZguRR9xJgI/GmRicdfpke58Ywf/+lnYhUuOdXorzSx6xAnkzUY7MppmLL7pc7BAdlG12 eFdtuXP8E2iTTGGhX6URSq2RN+tWqcmSDT0Ab/7FiA0iVKoFwZeH+8HrW2lwuMDaf4P7 k08w== X-Gm-Message-State: ACgBeo2YzPlyIek9jX8JFOGl9HGaHvLBBT6KeUDfAepAYd1ILO7GOqEZ BKVICQV2TVpLxt3W8YN55MoEBFN5pJw= X-Google-Smtp-Source: AA6agR7LHtl/RnUFMM8Ba/UDLwcr+KKZqP3s3F3o0L7ZSX4nHIA8V0azyARB9tbw+yLexNRTedxLJOpzAXk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:1366:b0:691:4335:455b with SMTP id bt6-20020a056902136600b006914335455bmr19454643ybb.282.1662053731874; Thu, 01 Sep 2022 10:35:31 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:52 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-5-surenb@google.com> Subject: [RFC PATCH RESEND 04/28] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183534_690546_9E8A041A X-CRM114-Status: GOOD ( 10.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Thu Sep 1 17:34:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963044 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 698CCECAAD5 for ; Thu, 1 Sep 2022 18:10:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZEj4bjQHoR0jrfLEAwN2z8SDBZpl5pAbwShk3AlSPnY=; b=q+mOGtF6FY1Xn1ONrusR/OmkhV yXqesTP8xHpcul3TITkHBsT7yUGbn0fXP0vFhFFUFLtuaEZgC69J04D+aXzERR60D/cS3/vg5537A tm5KSAv5XWjiDui3egEYy9XGFc+EDGUj5hFVNENfg3DJQvvIOf8ka5nayAfv3IMWcA4EyhsrnBCxs +EKQv5QyAlS090/zmghnHrtgQV0sDdZS/XAxUlkKS8VE7YeTAa+6SmEHX5A5KyLVKTPJ7WSo0NNik HAP371CchLSQtGfww05WIz6TbxGnkYmTekLGEBdN8Sjrlgp7nNAnf7lzYf9shu76CsDoaDhd6WFaa mA1VCRvw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTodK-00Dp2y-Hi; Thu, 01 Sep 2022 18:09:44 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXX-00Dm5L-T6 for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=eIMt0K4Dq/+E4lj9SQ/fnMIDQQYvNY/8MPu6+maRPr0=; b=Tt8kXCMfZWo9Bn9LYB2mHpMbvP zHvU8JHVLuIitZGu7oBJqC1OBFejQ5BBJ3vkyHwSkDs0DvP7NFt23pmRCbZ4IYIaPX9irSiJ0jmSQ hv4a0VWBzVff1jh005m6NBWhHp3t+OecOGaTtziYSYhWPWsj4P0g3FyF8uodr5kPykhZXs3ZWtxsn cdxWOD+1fcsMOOHaoOI8ginzbuPPRIwLRSJ3y9K7rKpL8tE+y3eAXuPzS/xbhAtxvk+5C/B/8fYxR G5w2CRxTfn2MIHXhoYTmj+OCiSIttYwQ7q+V5PSdichYA0mLDUVPxGCeYX5+zZqDjdpSANVuwr6m1 ujcG0x5g==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6K-008RcV-NK for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:38 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id x27-20020a25ac9b000000b0069140cfbbd9so4902017ybi.8 for ; Thu, 01 Sep 2022 10:35:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=eIMt0K4Dq/+E4lj9SQ/fnMIDQQYvNY/8MPu6+maRPr0=; b=OULzLK//lObTRMGAKryDz/AAItdyiK5yqhZqIIxu+sL1Dieokt2jidmnz3C7/IkvFZ m9XlR/LP/dWBo2UxnXyYn7qdVCzxgLbSFD2CG9EOYY1gruRvre30gd8M27oZrFHurKvO KqLi6QjNq7f2EcLrXhJZo+HJMeLkP9RVRxlBnBeo9kXpz7L5PtGPdJWGZy8ptDnDibsD BU++Zg9dqZVio0ctwQpkNxICzBLOlMkeBzfVJBfO5ajkBVunOWr6F+myD0p5P9YWiss6 hO75UDowCTjrDDdzDEi22CFFlp+R1gx4mTcBiehvp3OGDbyQbHz6/QfMRzJJk1Rc6B1v GPRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=eIMt0K4Dq/+E4lj9SQ/fnMIDQQYvNY/8MPu6+maRPr0=; b=N9Q7CLmqd9p+6fjG8oUQRFCLoflWlbtIZo7AQODixPoq5vr3tX7elotYaZ5rLUzc3/ TSo7qICJT+7AY1lIkOYVWX8+Aa+ZRKfY9q7gHrdmOFaLy/T44HxKlgTmEXrPHk352uuY nbmEgwjsVm6POIG5Jndzzb19R4D4FZPq7S1Zx3EafXAA+ZmtLvED0kG4az4p47jx+3Qr guC7myxZgytSOSN8SbhCYpRmQ6UuB0uOqSxpPeyoZ5CsZgOabjrJN/qEAZFDeNfa5EpP rc0yc9TraR+nCo+TL4qBI3lTB//tJbF70+YLJLXGlFxZ25sWDz11WC47gN/f2yta1sFo rdKw== X-Gm-Message-State: ACgBeo1v5QPLPlUWopPLMg4LYBcwMuuGo32vIAFUe4YrNu/A1chbVN/p kUxP3W7j5xLCi6qbR3ITMLTcNDS894T1i16QtYhNTe+P5bYCMn4TTWwwrESYKKAIpCt+snGfeiB IOvUb9QE8r4XMT2369CL0MFfwuRSeUkU= X-Google-Smtp-Source: AA6agR57KkNIgUFHWPdFiQWyoY7oyjG1ZvVgnOFqYQ9kJCo94JBJEo43qz2wa5ev7XB45jgLE1aOSNe2TR4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a0d:ff86:0:b0:341:5844:5527 with SMTP id p128-20020a0dff86000000b0034158445527mr14770867ywf.504.1662053734514; Thu, 01 Sep 2022 10:35:34 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:53 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-6-surenb@google.com> Subject: [RFC PATCH RESEND 05/28] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183536_820348_BE8F712E X-CRM114-Status: GOOD ( 19.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a per-VMA rw_semaphore to be used during page fault handling instead of mmap_lock. Because there are cases when multiple VMAs need to be exclusively locked during VMA tree modifications, instead of the usual lock/unlock patter we mark a VMA as locked by taking per-VMA lock exclusively and setting vma->lock_seq to the current mm->lock_seq. When mmap_write_lock holder is done with all modifications and drops mmap_lock, it will increment mm->lock_seq, effectively unlocking all VMAs marked as locked. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 78 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 7 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 105 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7d322a979455..476bf936c5f0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -611,6 +611,83 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +static inline void vma_mark_locked(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline bool vma_read_trylock(struct vm_area_struct *vma) +{ + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result but it's not critical. + * False unlocked result is critical but is impossible because we + * modify and check vma->vm_lock_seq under vma->lock protection and + * mm->mm_lock_seq modification invalidates all existing locks. + */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_read_unlock(struct vm_area_struct *vma) +{ + up_read(&vma->lock); +} + +static inline void vma_assert_locked(struct vm_area_struct *vma) +{ + lockdep_assert_held(&vma->lock); + VM_BUG_ON_VMA(!rwsem_is_locked(&vma->lock), vma); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline void vma_mark_locked(struct vm_area_struct *vma) {} +static inline bool vma_read_trylock(struct vm_area_struct *vma) + { return false; } +static inline void vma_read_unlock(struct vm_area_struct *vma) {} +static inline void vma_assert_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -619,6 +696,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bed25ef7c994..6a03f59c1e78 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -486,6 +486,10 @@ struct vm_area_struct { struct mempolicy *vm_policy; /* NUMA policy for the VMA */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + struct rw_semaphore lock; + int vm_lock_seq; +#endif } __randomize_layout; struct kioctx_table; @@ -567,6 +571,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..a391ae226564 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_mark_unlocked_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_mark_unlocked_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_mark_unlocked_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_mark_unlocked_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 614872438393..bfab31ecd11e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -475,6 +475,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); new->vm_next = new->vm_prev = NULL; dup_anon_vma_name(orig, new); } @@ -1130,6 +1131,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + WRITE_ONCE(mm->mm_lock_seq, 0); +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index fbe7844d0912..8399f90d631c 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Thu Sep 1 17:34:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA57AECAAD3 for ; Thu, 1 Sep 2022 18:10:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kuz/HQtWQCB+j51Wv0DeMDCHUNh/j3v546RzcNGmPwg=; b=xIvhj2WswCauSW13mVyy/KsoX9 OHRaBnCNkOkIHWzK0YOjeu50DLOLn/gUIaZP4J0Difc20qbxL1vTq3tBqH5AYsXLskb2Ixv5MHMMY 9sTZvtiTuuIPnjtLfJpkZAIdIy68l5KcyqmTA/prfiXZWHXj8+CR1z90wEM/PtfbmqOJfTlPXTrIz vMsjAHCxhQpw15BlQKYk0PVyZUV5fbLdpbzObP9CsxEm2nRTh+oPaM+Fd6WOz0Dit/l/9QP98K4XV B9paX4Ulvmi79fctbkADHj9z32eoOi4y1VQLTg6609FHxUL5MrcRKpEu+e6NOQRz/cRUJ3bhlcMR6 i1Lh4cnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTocg-00DomZ-PW; Thu, 01 Sep 2022 18:09:02 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXX-00Dm5L-6u for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=OSuAy3hdf+ZsQNxs3kxuMbhy9QBzPPm6GtoQo3aB3NA=; b=XzEFOvhsVW5/+f8Tup3Yv/92ZH oFlFiViypo96vuUTaKOASj9CpxDXjKhJGDkXPrvXRI93jcazwHakOZFeoUaC7yS2qWgtAnlwAXAhF hyW5iehugfhccnYTc9jJXzrbHT/cd1RbEv+LYqKDr1vbhpiYI418oI7LGFnQcGMqcX4nH2ATIrDH3 CR7QL4239dt1E7h3SFrxh8eJHS1PM7m8Cq4ea0f+6NyQWHXNNB621HF0hvZZvzdX6Iix0S+SBk7cW AI/+MeIAa5KJf7+0V6Hy/6pFpFR+EpmVhzHs7oJ52neOVMUXYwgcM5D4tnnp11YgA1c0rUX14H9mI a1I1OBYw==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6N-008Rcy-RW for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:41 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-3449870f0e4so11666907b3.6 for ; Thu, 01 Sep 2022 10:35:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=OSuAy3hdf+ZsQNxs3kxuMbhy9QBzPPm6GtoQo3aB3NA=; b=KO0CCoeJjfr6uQrGAGyvNgiswHns8WtW+vNCGbmb+v3lgrTdGqCfv4Q0htUhCqi7L8 NjudqvxWt3vJ1U4f4TaUrpiWDGDkYHTslXKwJR5AqzCzKMPyrwODeBVbDXy2W3OJ+0AJ 32abPt60ZQu7g2dOidue0bSh6Xh6XjE19T0FxDLe4AuFdEJaS9z9AXmzzaZHqpBfL4WP CC3bKhkCnU66eINFmY1rA50iR0WfFedon1RB2Q6DxPGeP1j32wzMpXKoaqT2OAwWSdUJ 2Dr+2H3hwJpx9s6tPPyXHohy9BSbHgL+aey4lgZu/S9fAtQBjK/jZiqIlpp1Ch7Jebxy Blpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=OSuAy3hdf+ZsQNxs3kxuMbhy9QBzPPm6GtoQo3aB3NA=; b=ia0II2gcZP42UUmdXz/l12AkptP1rA7hwalD+9u2+swLSx1okdh3VDHPggcnbMdzdT cFQl7hdHuBclojFEPFSH1D11tEmKwRKxE5bSuBN48v6K3Si5drJqf+QSauk7FCJZuFee jYi9O3e9asIOI37uA4CDwNv3k/fnOvDtDJ88uwwHl2cpS9XJI+xLbafk3Z59apkhyXDM l7Z3zPOdC0809gdjXT0k51JYHHf8xSspLCw39u+HnQ297QnufZ2qTkrP+D4IJkqKhk25 77RnhQLvUV2Sk64oa/qXWdrRt8TlKP002vIpDpub555mCP/wOnhDtmr3uipgZefpRFxd POVQ== X-Gm-Message-State: ACgBeo3tciWXuA6RSGe+ToPEoBLRZ2DVo+Ce+FI4kMFAjyeHJ6RI85qk pgaJ4mw4qf4NrfLVwYJxrMj0i8O6+Lw= X-Google-Smtp-Source: AA6agR5mOhnOZ5TXGA7QbMyhBcTFpfMYAPy+7Q1nZkdORl1Ru4vF2e+A26u5Wht4tIYfxlBv6SIkkjkdNGw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a0d:edc2:0:b0:338:957:a719 with SMTP id w185-20020a0dedc2000000b003380957a719mr24916063ywe.132.1662053737185; Thu, 01 Sep 2022 10:35:37 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:54 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-7-surenb@google.com> Subject: [RFC PATCH RESEND 06/28] mm: mark VMA as locked whenever vma->vm_flags are modified From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183539_984445_C40E466F X-CRM114-Status: GOOD ( 15.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org VMA flag modifications should be done under VMA lock to prevent concurrent page fault handling in that area. Signed-off-by: Suren Baghdasaryan --- fs/proc/task_mmu.c | 1 + fs/userfaultfd.c | 6 ++++++ mm/madvise.c | 1 + mm/mlock.c | 2 ++ mm/mmap.c | 1 + mm/mprotect.c | 1 + 6 files changed, 12 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 4e0023643f8b..ceffa5c2c650 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1285,6 +1285,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, for (vma = mm->mmap; vma; vma = vma->vm_next) { if (!(vma->vm_flags & VM_SOFTDIRTY)) continue; + vma_mark_locked(vma); vma->vm_flags &= ~VM_SOFTDIRTY; vma_set_page_prot(vma); } diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 175de70e3adf..fe557b3d1c07 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -620,6 +620,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, mmap_write_lock(mm); for (vma = mm->mmap; vma; vma = vma->vm_next) if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; } @@ -653,6 +654,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; return 0; @@ -734,6 +736,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, atomic_inc(&ctx->mmap_changing); } else { /* Drop uffd context if remap feature not enabled */ + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; } @@ -891,6 +894,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) vma = prev; else prev = vma; + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } @@ -1449,6 +1453,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx.ctx = ctx; @@ -1630,6 +1635,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; diff --git a/mm/madvise.c b/mm/madvise.c index 5f0f0948a50e..a173f0025abd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -181,6 +181,7 @@ static int madvise_update_vma(struct vm_area_struct *vma, /* * vm_flags is protected by the mmap_lock held in write mode. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; if (!vma->vm_file) { error = replace_anon_vma_name(vma, anon_name); diff --git a/mm/mlock.c b/mm/mlock.c index b14e929084cc..f62e1a4d05f2 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -380,6 +380,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; + vma_mark_locked(vma); WRITE_ONCE(vma->vm_flags, newflags); lru_add_drain(); @@ -456,6 +457,7 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ + vma_mark_locked(vma); vma->vm_flags = newflags; } else { mlock_vma_pages_range(vma, start, end, newflags); diff --git a/mm/mmap.c b/mm/mmap.c index 693e6776be39..f89c9b058105 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1818,6 +1818,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, out: perf_event_mmap(vma); + vma_mark_locked(vma); vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT); if (vm_flags & VM_LOCKED) { if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) || diff --git a/mm/mprotect.c b/mm/mprotect.c index bc6bddd156ca..df47fc21b0e4 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -621,6 +621,7 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ + vma_mark_locked(vma); vma->vm_flags = newflags; /* * We want to check manually if we can change individual PTEs writable From patchwork Thu Sep 1 17:34:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 257DDECAAD1 for ; Thu, 1 Sep 2022 18:09:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=NV/arcZXtYqcG9zB1qTbDwCef2oLRVc0gUn5h1JYhHY=; b=eNSRgLdTQG27guBs3Htrf9S8OY F8mqGYNKRLiyIjq0EpH1SGGKoxebEvMN0lcmGBkLlIUj9uASvLkp5bEh3rTjd+TYE/7PnKJ1o8qY7 S6Zy571a9nxfqRTGluEes8k3d8amO4ACFcQucRBv4X2fM5UwjOw19OvSAQFXUKAW09Uy16s0id1DC BGBlF/FB6SM7kzDeUzk2AJWTckeUr0ZsYD0jhh9y0GyqMAci9SNK8OCkqF3aYxBxDDU+1z3loXNgk kg+yfwxFFPAmevsl0T9Seu9qUcb2oaedWAzwfMKZxAG+LYHm/c7loT2FtHfNWqReNoI54pSnDBa2M CBShqcRQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToc0-00DoS8-52; Thu, 01 Sep 2022 18:08:21 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXW-00Dm5L-NT for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=byYdzgQ2nb1BwMzAK8Db5EPtebwswE17va5vsTNHDvI=; b=ZMyHx8zbdoP8//i/FAEmPU+doR Tr/DMseEjJ2nCsweACVJBo/exy1i6qqZhqmdfmg/bDZdTxWcBbr5NtZXT91MjKVV/pSAAyAOmWqSy W7LUjfJxwwVXa7oLWjhnoI6rmnzrsfk17HFzKwvLgOkRzKDlUXyWzw6kTj1DjwsvxLgBA4qJjd36P LOBwqO3SBDe+REUILdkq2IrVC35A+DXiy2uCHIkA44uboi5cOUM9+adzRz2byqmC/cvl63bvSieSy /rXNWWoiPZRQkw3+/Lfql07u9ZJsfzBiB84+57R9a5j+7j1PhZx/FBPOnIl8iPqaxxqnrNq4b8/uI 1i3iwV7w==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6P-008RdZ-Qt for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:43 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33dce8cae71so234413807b3.8 for ; Thu, 01 Sep 2022 10:35:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=byYdzgQ2nb1BwMzAK8Db5EPtebwswE17va5vsTNHDvI=; b=U5hxVmDFnvg3Zh8mYi2DwtWs5BxWMTUQESfFDYoN6r6izeiWA5yaRxl1nHFUTbwk// Dnd8HHqghiMd97Zw1zIzmeB4ars9SXmdp/qLpi9efptTbpKgnXouiU6jw117QN+WP9qs 3UWSwrzjzVFJ688YmwTVx+XNrjpewHxiYcGpgCH/HREdk2crWCVRYVVfKgVwMdBm3Bmp 0cmyfTq+1Is86HcJ8rQc8vT+aBrCdpHLLyCE4j/1ArHGf12SJ516YZIyRc8DbZTCjB20 ojn6KPPAcVM9EzDOR735/i5/Hcf8Q/BBi2k53VcG0tY24s/K8Lqz5/8Ca9tGedcRR97p PE0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=byYdzgQ2nb1BwMzAK8Db5EPtebwswE17va5vsTNHDvI=; b=UghLPfbO5Ki7K0StUQcCaNAaL4TZBITVFv3ELKXkHXX6WFrqyKX/lbTjmJhfs3QN7r sBEC0Eh6mkuTKCW31q5fBLzgxmSZ8iDUZ6xBldvEF1PadtYhTWJAVi3smWrJORa+oS+G A5OHOm4ENlmnGb29R/t+Sn+UNiL+Tc7JXG0ZBIvfrGGmM+UmcfMu5WU1+iLkdUr3/gaU b5+tHv8VZEC9dVsO5RVEwRSjnlHAb84Nco7tOHYcb0MNblgOpFzzgQd+r57nJONJEtZK zFVonZykikI5jNfh0Wyvf/8wx3A5jpKSxB16oRgFGSuQRMyRrNXCNOuDGifuzadcpIgc pCLA== X-Gm-Message-State: ACgBeo2FKWXc0FGK2rGrNWHhL6GwCEqJLRWtiATv2SpEIPuCq0cPNo8x kJGYg8BDi+xAJeoHQ8zF2cn6WXvyQBQ= X-Google-Smtp-Source: AA6agR7fnwkRjroGVHJxi1RhhEsQJMH0R75zEmK00qUWtCXpP69RbgB6GRXVWsVMb26fPpccLe4aPhca22w= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:40c6:0:b0:67a:5c8a:21d2 with SMTP id n189-20020a2540c6000000b0067a5c8a21d2mr20567843yba.158.1662053739927; Thu, 01 Sep 2022 10:35:39 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:55 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-8-surenb@google.com> Subject: [RFC PATCH RESEND 07/28] kernel/fork: mark VMAs as locked before copying pages during fork From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183542_001229_07920C39 X-CRM114-Status: GOOD ( 11.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Protect VMAs from concurrent page fault handler while performing copy_page_range for VMAs having VM_WIPEONFORK flag set. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/fork.c b/kernel/fork.c index bfab31ecd11e..1872ad549fed 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -709,8 +709,10 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, rb_parent = &tmp->vm_rb; mm->map_count++; - if (!(tmp->vm_flags & VM_WIPEONFORK)) + if (!(tmp->vm_flags & VM_WIPEONFORK)) { + vma_mark_locked(mpnt); retval = copy_page_range(tmp, mpnt); + } if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open(tmp); From patchwork Thu Sep 1 17:34:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 05C49ECAAD1 for ; Thu, 1 Sep 2022 18:08:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=kA8yonofoeEgggH6vO1aojdxFtif9V2mkf+dqHBN6oc=; b=pV89gabCMz4wm/HUIufDOFg797 fvwGIn5gOWFlhQMjqg+8UE67R8ZEsPFYHl6svMHo1HJGQ9bq/9oV5DEXwJd4r9A7VMq8oKfyJLpBR 5ZOh7AflI35lTLmkTDmWGVhqNQwgHOVFOOj69AZpoJjXiz+QcffGgz9UiSPSDmxZJIIrG4JpHO/RF li1A8VtCE4CdlKHd7eQdug4Wc9iyGlbB6DX4pqhoPOxlP3AruoDswEvyegOsY4jXMn+e0yvuaEUv9 JvWwMQnppMzcsa+cubQfK/4lo15M2gNXIjT4nyy6DHDO4Aqp+BidtH0DQNKb9cZydhuT0orgkSSDO M0AhJndA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTobT-00DoDF-JG; Thu, 01 Sep 2022 18:07:47 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXW-00Dm5L-7y for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=sXPRs4un8Hv1KgYmIkiOTpDZHlwje1UN18vDRt/znXA=; b=T7+AuhrvjDZ49Aqn8CGG4uujrf xVZTC5iaRT2i1AnhNXwHb+hHyI0veMjPTCwkA3gXP87PqEXGpMPYDdsgGX9NchamulHSkFIkST/a4 ycFzBU2J5k65O9dYBbAISgbj/pbnr2mbMw4F7eIhDCn/qWTBRHmGGTOQHCWmjgsBQWllDTZwB9tfQ BLD/iyTOuedJbXW6xALkH2gFt7jYlgeJovuQf9J66fmElhRgBmgm+qy+ldPp/bl6cB/93vVVdW1H5 IGXo+DpxuFOyb+d22ZFGrkhC+9nM/x7FPpJl8pDIYRO43PE6D8eXn++ZBn6Zu09d3HRNAvvmklLEg RFZCOOrQ==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6S-008Req-3c for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:45 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33dbfb6d2a3so234904027b3.11 for ; Thu, 01 Sep 2022 10:35:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=sXPRs4un8Hv1KgYmIkiOTpDZHlwje1UN18vDRt/znXA=; b=LBJTSzoCXGb4iiWKyO0OEdlY4M2wMr4mJ5fdr9ss+SWylmfoHLBZ3bDcjUEYCuu5eS feBNbsez+RUGQT+LSEIL3jhNfGQ3mzrdU70gDdYtaUSlqG4c3zRWe4jgoZRdVjvHQEvZ SPCTeif98/8TgsTaO2lykBXQO/Ydnr0/XtxYChsH6bMYMfoCTEgHM9QVPOJS0Mu1k2UR t6jxn7uXNz4Ix/YY107csqUZ1FzpL1YUPLT0wcSHa5tJ7jOlfkB6DfkgR6ONkj0xZI4Q CdDjSBazMRaOu6i4p9xpWVSdGu4/KGHF+Yxu3Dy4GMEd28lc2vp55tWuelmW724hNsiA vxOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=sXPRs4un8Hv1KgYmIkiOTpDZHlwje1UN18vDRt/znXA=; b=OSd5rhLdW/zmgI6oWahUjnu/T5jLhN4zQ0pLlSEdSQQ7Lh9+5o2iRzHat/vVR0NEot 8JFB3qU3Ayqmjcm1Sd9CC1ZvCd62aYQMBzX5vIHQRZ2WNUXQdlKGiBn8whz/VyckQvnZ 7HHvNfCGPB3upJUT0h8jPNXfzbHtc5LiqwtycGgrbOUoTRimgsbIcB99WpXPGd4ruf0g JnXmlMp2aNs4b60JBIxh3oDeOuNebnvbnYXmMzHRMhEZQfOexGiiyNuqT19xW3STyzpR tJxguFuVrUN7fcG3XM2KihLQVscc62hDU+Wz4N5Is/M/AbJH7dLpt63dRu6MvUQcO9mr cyyA== X-Gm-Message-State: ACgBeo0z023p7Xk14T/itcqEr1Tjj9eO3IyOpyo+bwYpX+CQbQcUGnrW aPgiPEna4mntlLl4bT4CS47oEzbJicE= X-Google-Smtp-Source: AA6agR4Ppy4TlAx5Lt/45OnUgJ5yxX2999apq2x6zNaCId6+AMBNoDmW6LxkNrA1Ax704mp9y4OtGqUy/dg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:120f:b0:676:aaa4:8434 with SMTP id s15-20020a056902120f00b00676aaa48434mr20939193ybu.218.1662053742565; Thu, 01 Sep 2022 10:35:42 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:56 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-9-surenb@google.com> Subject: [RFC PATCH RESEND 08/28] mm/khugepaged: mark VMA as locked while collapsing a hugepage From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183544_254793_C993E301 X-CRM114-Status: GOOD ( 10.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Protect VMA from concurrent page fault handler while modifying it in collapse_huge_page. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..030680633989 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1072,6 +1072,7 @@ static void collapse_huge_page(struct mm_struct *mm, if (mm_find_pmd(mm, address) != pmd) goto out_up_write; + vma_mark_locked(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, From patchwork Thu Sep 1 17:34:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3534DECAAD3 for ; Thu, 1 Sep 2022 18:08:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Tgp0PnzG+5BYMx2humBISfI3f3BMI5lpEgiFaRxTOXw=; b=uABM9f30kICbE+ZBixTNxRuthE +pMUICTKprk6bhB7SbcVqa/LDZI9PEWEnVKelDVamLxWxSj7+u9XaLGb5yBO1KrbKbiXGXoiIGDpb RR+pPU5VZKR/2QIfEd/DC7clLwYSrpmYK5rQBuMgOEjO6YgZ2hcX4ztnaEQDL00qbO40gFtEtmeat Ztq0nGlBgEYfPl3Oidpcel207dIiuBBMZxrS23Hdb3ph6zLGOG2Xl64ECb1zMQi/Zv3txAkHi8yeA 7y2Ts/shlfOZi8WwB0eHSIvHa6ZUhgF7xuscRpSSOrHi7hdg68plOfuNdF4EPoDuq3wmeRKVrbLJW SZsehFfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToay-00Dntn-4P; Thu, 01 Sep 2022 18:07:16 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXV-00Dm5L-P5 for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:41 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=f3HBd//8MTEjGtUKjIJ6VlUao66B8ATA87dRitXKN6o=; b=UJl8ez6p/uoYuHa5bvfKiIRxEZ B/cfPLR3E1GXweGY7UnbMaynSZWBYOIqq5Qu1knOGtueiBlb8AIdIrfofIlwZOubNvIll0axklp2x 0MFeub3T6mC9NhNaDHLUNe90rLSbxicGAPsmZQpGnIuHbBJXd7HJ2iPU/QdAutcb6Nxb8FNyu5b6D XWrIjTFhVrhp27E/8ntUfwj6D9I6X8yE4139xMqoB420c1xWO7FxNbkydptSNpKQTC+t9ZZX3qWMt 4r7A0K+e5tGLzE1x/ha2CRSaMexKD4kTrOMGb47r1+AcOdlQvddqpN8aIzJP9SJC90uj85wiSrZkn 9vEV9HbQ==; Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6W-008RfZ-Fe for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:51 +0000 Received: by mail-pl1-x64a.google.com with SMTP id f1-20020a170902ce8100b001731029cd6bso12053858plg.1 for ; Thu, 01 Sep 2022 10:35:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=f3HBd//8MTEjGtUKjIJ6VlUao66B8ATA87dRitXKN6o=; b=osN5P5u2gNPNqCUBvTG+Zi66behXZjVMeiTYran173d3bK7x/C4pNezlQHCFvkYQk7 qOQqzQeCFbXQDX9e+QRzeVBdZKr22OFya12CKF/BNucIJ15gJeM6CJ6xJtVhkJ86JMIW z0BrEcSFGRuuGzZ7AIa/XsKdDh+JkQT9DWUmeCCWJ5VsR1VGSIBZmgZx0ZCJYBHl+XPj GoCI/RFibFsonWSqgiZwplWva8pXUgT5urz+18AucAe/seeiQYlfr5LWvUC2qg96tyno slrVqaBJlR1Fm38ZmASWvylf1J3rbhYM8n7OsiK4uazMHDS7+qFeKWZULOPVqfmGOKH5 n3ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=f3HBd//8MTEjGtUKjIJ6VlUao66B8ATA87dRitXKN6o=; b=yBtzGSKqZWMdM+tUjTNJU2OzmmVBGKqSsOZYSEuP0b41LIxA0w5csmgVOAxjlYXb1L 2IPd5x9PSqqeNCk2SH+kdolV66UhhVTVkTUgiMC7fQuzg7RWShb7ZpvL1/S6cTyTklyI AKQkqQae2XH0B3qlZ7a2yWZ6WqvzTZ+zhploR2dyzFnkuvEfFAjlibvHApTATtSk8xY2 Btx+wgyPoscNYUQjkrmx6xLFn48bzJzJ3Obv6PR/bJs1YXIYFxC318Caobuob2OAGgS5 IV6bYXxcQU2faOD6Us8heiZ3Vt2rcT8rVF9z/dYJA2/ZowtjnX4hIZ+x1TsN21Gtzruh uJbQ== X-Gm-Message-State: ACgBeo2Eo8WaFJuDOQolDyAdhGkN1ZoaBYw7v6/ySxoCXcWUCcSuC0fF 0eMlbzN35HV6qt2I4HGX+Le39wYGskw= X-Google-Smtp-Source: AA6agR6Q3wPydZYjoco/mRjMkhmNWxJo3Xdi/ebZ3ovzfa+ebF5iWZWFhqHdfMvubIqW+qGVzpwNqTYAd9U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a17:903:2351:b0:175:58b8:cdef with SMTP id c17-20020a170903235100b0017558b8cdefmr4890777plh.30.1662053745409; Thu, 01 Sep 2022 10:35:45 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:57 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-10-surenb@google.com> Subject: [RFC PATCH RESEND 09/28] mm/mempolicy: mark VMA as locked when changing protection policy From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183549_865768_A4CC59CB X-CRM114-Status: GOOD ( 11.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Protect VMA from concurrent page fault handler while performing VMA protection policy changes. Signed-off-by: Suren Baghdasaryan --- mm/mempolicy.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index b73d3248d976..6be1e5c75556 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -383,8 +383,10 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) struct vm_area_struct *vma; mmap_write_lock(mm); - for (vma = mm->mmap; vma; vma = vma->vm_next) + for (vma = mm->mmap; vma; vma = vma->vm_next) { + vma_mark_locked(vma); mpol_rebind_policy(vma->vm_policy, new); + } mmap_write_unlock(mm); } @@ -632,6 +634,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, struct mmu_gather tlb; int nr_updated; + vma_mark_locked(vma); tlb_gather_mmu(&tlb, vma->vm_mm); nr_updated = change_protection(&tlb, vma, addr, end, PAGE_NONE, @@ -765,6 +768,7 @@ static int vma_replace_policy(struct vm_area_struct *vma, if (IS_ERR(new)) return PTR_ERR(new); + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->set_policy) { err = vma->vm_ops->set_policy(vma, new); if (err) From patchwork Thu Sep 1 17:34:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963000 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 27395ECAAD1 for ; Thu, 1 Sep 2022 17:37:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=cP3bOnWvDiO45pgC4cwY1LVywkgDo5ZwwmUkez/cVLo=; b=KORt2scME10u8dB8nxKONuPKzB IRSTiEqQRb69fqsklvuLZj+w+huU73BZb4DE3c6KZQS96KUNX9QvpvZb3WYJ8XxTJKwqvFohFe+l1 ix2r4Buiyb8FAY8h9rEYqIt5u+l6PdWZnsPz6U6aAlKLj1SZ+2AgZRU4nbhMwLXNsqWhhymwdmAMg 8V107LQytWRZK+ExP3DT4KB7zX/qSycG19HyG+s15vSlDtqGLIYz6JLwufp7zozkkojHLakjHQTaO obe4Mwcwq8st5Y8XLKg4HDrO4f/QTs1+/eBrn0gISZTc8+6nlogpcN26eF49wxcTqCbzLftUGxgWT WnypP9qw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo7N-00DcCo-Q5; Thu, 01 Sep 2022 17:36:42 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6X-00DbWE-JE for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:50 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id i6-20020a259786000000b0069648392979so4915132ybo.11 for ; Thu, 01 Sep 2022 10:35:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=DiVLSVAVO410fRJcvWWuX+r6LjGy2i2DyacaHp6fzuc=; b=kNx38Jhqjx9XlmKxOVsRKukj17KeQAoUkvOgezSjJcI7qDa8s6siy6+ZCSn18LnVHi n4QLcTPTYnWQedOp2NOA065emr+I7/FhDV/20btA2UJprNGZCfJvY2fzU6JS3rffkIEf 2FurYAnF0+QmVLyL1XlOy/6eS9/B08Rzb+EHT4ihXLuAM0YGViLuLR9Pls5xzIb0RA++ peqVg5K5+yJOyniQSXXLxLw8N9e7jk4WJ8bD01w4ADM9IUXJZsw/A2NPWpCJiNE61osh 1/9xE4X1j0QNkz0GiHjK4YBUmosqgJkbBEuUlP/cGGRjR6eOLhrxcNhT+yoJZX8wCM9W ZvhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=DiVLSVAVO410fRJcvWWuX+r6LjGy2i2DyacaHp6fzuc=; b=I3jx2c3mwQ8yJS8RhnU54sM7Bo9btaXBEhokVM/gTO1bUDaRkoLMDX8nMGhjYAWV9c slobOF/JiB//mIxhqbWfyCu9SM/L1ZrBo/pLkOMc8W9IUnhn4zenRpGmWlpZOhcJR1sC 22zIq2Hy8vtvZyL+YQgQlPDBzpmuSUIqmFvPadxDtn6BL7VDqeyx3WHxYn9XjQtg/lcu /VDlqEoxVX65x1fW5HtkSnJYTEBUrBlhDu1TbYpizzl2uuvRh3/RxENXRyAvzTZtoeao z6qBXZVChPZn1haHhcl/sHzuHYEtcUq8WefL3+b4/M9xTW7FdDjH/QS3YZ7ZatV9G/Xc bkYg== X-Gm-Message-State: ACgBeo1pp/9kAaQDVTNn/0uuWSZlzvGdPXJf9Amco5rHfHC6SpN6AJ2Y gP3JdER1Y4bY41aYrdv3sEUA1q5RuVA= X-Google-Smtp-Source: AA6agR4qMG/7UuuiqhHhvohWNal3nFjDJKXC1X/Q7rlLCDCvl4ZyXaMMKRMmizmDLQF7LDsGfEt+/G6N4D4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:bf92:0:b0:695:8d8f:2af4 with SMTP id l18-20020a25bf92000000b006958d8f2af4mr20427159ybk.200.1662053748060; Thu, 01 Sep 2022 10:35:48 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:58 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-11-surenb@google.com> Subject: [RFC PATCH RESEND 10/28] mm/mmap: mark VMAs as locked in vma_adjust From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103549_655463_3EF0C493 X-CRM114-Status: GOOD ( 15.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vma_adjust modifies a VMA and possibly its neighbors. Mark them as locked before making the modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index f89c9b058105..ed58cf0689b2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -710,6 +710,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, long adjust_next = 0; int remove_next = 0; + vma_mark_locked(vma); + if (next) + vma_mark_locked(next); + if (next && !insert) { struct vm_area_struct *exporter = NULL, *importer = NULL; @@ -754,8 +758,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * If next doesn't have anon_vma, import from vma after * next, if the vma overlaps with it. */ - if (remove_next == 2 && !next->anon_vma) + if (remove_next == 2 && !next->anon_vma) { exporter = next->vm_next; + if (exporter) + vma_mark_locked(exporter); + } } else if (end > next->vm_start) { /* @@ -931,6 +938,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * "vma->vm_next" gap must be updated. */ next = vma->vm_next; + if (next) + vma_mark_locked(next); } else { /* * For the scope of the comment "next" and From patchwork Thu Sep 1 17:34:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62FB5ECAAD1 for ; Thu, 1 Sep 2022 18:07:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bNTGF2k1Hk3s6woWfNpqxwa76HeC72g9z8y1D4sjWoE=; b=IM8xWcUV3zf2X1hKYnxzObMDbE ZuHoU+P9ZVd0SAw4LoWLHisl2iuX4WG+jCjVte5gVwq1CO7Xt11OBdWF3NJ90miF0YAIoKek/ULZM Ft71zOeEX83ADIT8knpfS8x1x+9BgyTwtyRa1hJVVxIL9cDItmATFj3m3bO1xfj90+YOuYHEmNsik tzFB0cJck6F50xxSOkuZaD0Us9EG2llgA5Z6F5YghmGH8auFPJ3KCA66wVOMa6o0V4G79wfDVMynV OCtarq6JCH7Tqpe2BqDFktWKm7TwGtQnkIjdBL0Qbb5EWFpr+KiNFQ7XqDbfN+hPsjkFWwWpRVXR/ yc+InTxA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToaQ-00DngB-4H; Thu, 01 Sep 2022 18:06:42 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXT-00Dm5L-Vn for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=IUrOZBQLzZPUHaSf75EcaCRmzwc/1Uln8Bhr3CCRW40=; b=eaLQCSL5U1Y43yS4QZelMBtpTX 9/UsjpShi9VTIbOl43Lwm6bENN5gtlABXMcRe5C1YOW61NPLJMGB2zUYeAKiXW1wp2gbhgYYxfRYr ZQI64PIvYEdqts20rFUWgZVZ5DC808Qva7BQHqghbcu+Tpw0JIPRKymmBps3ht9miXRZ5dA6tWkbP ZG0DceRSka2fe1b5NCuI1OhWaebOR+bfl44tPthdBbyQ95xXNqtoXW8ylL11yRbMDFdnbtYdvLN5k Wo8mis/facWGlGi4TLGQv6sx/znUC/AHoetUgAym5f6A6YwTcQaH0JhEblrzDrtGdvxfY/dW4inAV mPwXIf+Q==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6Z-008RdZ-B8 for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:53 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33dce8cae71so234417047b3.8 for ; Thu, 01 Sep 2022 10:35:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=IUrOZBQLzZPUHaSf75EcaCRmzwc/1Uln8Bhr3CCRW40=; b=RqTxEDKmKz8lQ9e4qoEc4uYxV2l6ZaYMJyNDEM2PccdKcjIk6CnRCwBsEF2cj3JFvG j1vM9+I5MFJ8Y28kW9pfnzwXL/sLYGBlEyABKHcApM/RmMkSzapOCNpQMv7qth5lMe9Z r4q/GOZemPIxyE1pZ9TWp0QmyjLs2nHHEkPOAyS+5ktrUUhx4EO8hL8MjHB9ieBIjUIm Yg7XSf+rAL+v2NSP5NDXERvFs20Phv4/bhur7fMLyFA6L2AN5XBQFKbc4pQ3AuRWkJ+n hhBQmROSc08GaGFWWbM/WsNha74+z1T1INvujKYoiSdnjlDI7QzZLxl1fThOa82IPdUF 4lpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=IUrOZBQLzZPUHaSf75EcaCRmzwc/1Uln8Bhr3CCRW40=; b=Q0Sf7K2RvcX0vGHprC3QHTx0dSudOjGrNo3TdLYnOvWL+aNk/2ADC1CAjfPm3abftf 0Nw3NpBrF7eKu+OC+Pdrb2QDEafAOAAm0yEZt7kp5XcXTWkGrlwXdU5EA3+y35mzE3GB 2oCVt7nB5wk4btJz7OFzpkPK1WprSU80u7LnhSdiRWcDJCYkLf/YlUsC0gAJ4xmGHLfc lKgxon135yl1CSMcFwhPnDep6sj8EpcIaFRkBlYXZ8RZn49ufpKmrEEVF9hVdM9QGwUL tFebMkCYhdzTcZ2VzPZ0FkJbu5UhY8P1JVsiY5SM7Q9MMeHbsJmMIElqX3JzWPgxOecd 53PQ== X-Gm-Message-State: ACgBeo1Wk7pGfSbp5o+O+4PZ+GZDebXmhOBxu6Z3APIFkMKkF7AyUg9s jklBvtnttFcEn3pI4E7XS4Ts9HcZ3OM= X-Google-Smtp-Source: AA6agR7hyHeb8khg+/3ZoJp3oFVauMFyBSatbGnIGl1CpSfGDYZylUwLzf/LZXvggze+7Aght5uln8kJbB0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a0d:e083:0:b0:343:2928:867f with SMTP id j125-20020a0de083000000b003432928867fmr6902464ywe.467.1662053750534; Thu, 01 Sep 2022 10:35:50 -0700 (PDT) Date: Thu, 1 Sep 2022 10:34:59 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-12-surenb@google.com> Subject: [RFC PATCH RESEND 11/28] mm/mmap: mark VMAs as locked before merging or splitting them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183551_488912_54EEDD2D X-CRM114-Status: GOOD ( 13.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Decisions about whether VMAs can be merged or split must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Mark all VMAs which might be affected by a merge or split operation as locked before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index ed58cf0689b2..ade3909c89b4 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1147,10 +1147,17 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_mark_locked(prev); next = vma_next(mm, prev); area = next; - if (area && area->vm_end == end) /* cases 6, 7, 8 */ + if (area) + vma_mark_locked(area); + if (area && area->vm_end == end) { /* cases 6, 7, 8 */ next = next->vm_next; + if (next) + vma_mark_locked(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2687,6 +2694,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *new; int err; + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) From patchwork Thu Sep 1 17:35:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 93841ECAAD3 for ; Thu, 1 Sep 2022 17:59:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZHvCljM1hUBVuJYOMpFa0NTfCkSqnJ9GJz7aLbs8pd4=; b=4uWspi4ZMojGNbieQ9KjvgG/8Y Hf+Jwlq9D9am2dMtvTWW0L/SiislM59qn26yB5ex+K1Vq2kccsmTS+HXlMZac1f2ItVMhYoOPSpKR tBndkLYc6/SNW2AU4FPzGPbmVHVHXnBNMxwYZU+IynD6pyUhIV1+Xx0kXxpwqg3OzBFiULJxDGF1J LTemrL48zqm6Yk+AO8RusFiAco2tMl2/LrLZ/eNjd0MpgK2Fja18V++S0LrdQeHHnRIchZIWcLRIe obBrbOIDxXvq7k8EqmXteED049UoZNqACJSV/lNrLFz3ydF944dRV9ybuINagHp3w6KRI0lto9sbS 3sB47T3A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToSG-00DkPG-5H; Thu, 01 Sep 2022 17:58:16 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToRz-00DkKI-5L for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 17:57:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=SOzt98PYJdog9tZtODUdaF02HJMaRj5G7nUIc+/7MKg=; b=r7MTpVjeKgcsIVevhQLVl9GPmt p+KwlQrqlWrFLbpZDtA4usakok3VDOKsG/7guyNu8UaI6L4NDHwv/t5XJtS3LYhjFSFftIWmiOz+z g+CdZ10AFWcMT8CpTbYlz/Xi9YK9ecuj6WosymYnwlBLqKw7Vp/4orz4kEJdeBme7r6x/DUR3P1di nACrUQ887m7hBvzlTaoFD5Mwb5ekR0StQ+zyiV4PaECsB0BwDDGT8KRfcz4Tz3IDtwjWXBipJeqYB OLxXl9cA4CYu2WG5ly+F+WzJFQx9kGA7YkmECShu+/oq0MSkzkR6EsRnJ9mc/YkkepqXDKXPM+hVN YS7T3yDg==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6f-006DfI-5v for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:59 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-34489a1a2d2so20324427b3.23 for ; Thu, 01 Sep 2022 10:35:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=SOzt98PYJdog9tZtODUdaF02HJMaRj5G7nUIc+/7MKg=; b=kTPvKd0Pj/2ICi2JaJbj4mvw83rGSI8D/7SpSALxOaZxZxYULx54dkMykzJllj/oUS acc8cAwohzpv9TG/117+M7kDarZ4pbcDRJw9kbqISJMHllJVQJpNPTnmGZHaFtdhW7jV te5Y1eTVCGFOLeMRtGhakQSv/C95dQTatvR3jNPFepOpi9BM4A2IDo4sHrk/Ld/IusAA C91XyLoa1Kj543iPIZk7cC0y8cEUyAun26iVaV/jukaSDh0nNNy3USNMwmiwPC/u6rrR uYxoBw75emQZr0TAylDwoEOSkfchxb9rhKIYoM9w58cv+ILGWIFbRnVoqe89UuzvgGW7 /7mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=SOzt98PYJdog9tZtODUdaF02HJMaRj5G7nUIc+/7MKg=; b=6VrhxuxS9C1tNmRlXhiaTTPwpnxuqhFQvfj2bWnfUmqX50u0LtR0v4jI0C+hzIhvjA N54gSHqx6LGNpqG/3dzkYwV2vLl0z9mLTTqI6sWdMe42jF+DXRv7rPvQdEQFIyNK8NT9 zie2ULwIelZhftqfBz7KyLOkDBV4buKWjJ9e49pgGroj2CqD+w/9PmqdUVHqeqGDoj/0 KlWYLbZbs7UKdIfwoOdYsETD9jtR1TNoBDSCqi2t6PRPjCz+Qhfqf8ayFRHi0H/rdfKl ZRO3xZ5TTxVTIomrSg+I4fzWHQ4n7uc3cfSkoRTS2R8b8ubdJBJK0R56UmFQzzY4xMMa DO3g== X-Gm-Message-State: ACgBeo2nn/c4NA0MPvOrbMZilI+FxxV00za8TUZ85zJLusPwPeq2Lp7Y qPBVy5getwYv04b1DE9EXZqOtuXdeiM= X-Google-Smtp-Source: AA6agR6YY4L9oqUniRiEloavxF9b7QDLSzaw7mgfeu4Gq//fIM/XBjFxGsDu3HVa5YMmDCpCvrS70nu4RUs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:3746:0:b0:69b:a100:521f with SMTP id e67-20020a253746000000b0069ba100521fmr16650279yba.135.1662053753176; Thu, 01 Sep 2022 10:35:53 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:00 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-13-surenb@google.com> Subject: [RFC PATCH RESEND 12/28] mm/mremap: mark VMA as locked while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183557_458198_068FBB40 X-CRM114-Status: GOOD ( 10.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Mark VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ade3909c89b4..121544fd90de 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3248,6 +3248,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_mark_locked(new_vma); vma_link(mm, new_vma, prev, rb_link, rb_parent); *need_rmap_locks = false; } diff --git a/mm/mremap.c b/mm/mremap.c index b522cd0259a0..bdbf96254e43 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -620,6 +620,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_mark_locked(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Thu Sep 1 17:35:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 52602ECAAD3 for ; Thu, 1 Sep 2022 18:07:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=yd/W78/L/8Yafmn93Jggs0IDF1BolvZ79/Fw0fv3k1Y=; b=dX0ju5Ca1wQkRSeAA0b+bBHvH1 WRSdOdhiOgKe3TyJZriKVyanenCvvYYSBq2Kr1g36l/9qPZj5tTiWqPaNpGrrnB2csctVoolBvZOz NZn8ohWycGONqBPr9MKrbm3rxfyMy/SkTM1nOcpYaZ09szIsgr9eM+yb7NvFqn2fjb07NTgAQwy6B OoW5Ao36wb3Yx8wfnwCPjTlWJ4CSQMJqF0bcLHiF9SoI5XSYn2Pz8NK8n6NLHhRdPJMSAAWum4oTd ieMFykxRIB6v0yt9MY7/u/fIyyFxp7qyNYqIPR4RV21ZhxGml1Nmau3wA5NA3rWpiHOr3iY3o8FCq 77DGvh5Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToZm-00DnOG-Vg; Thu, 01 Sep 2022 18:06:04 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXS-00Dm5L-TP for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=r773C9YRjCBNvMT4CzLpPbmrmyQxRNwpck2TtfCck1k=; b=Jz/EgR46kmXoChR/R6mTWeExUp 8dFoyKZ0VGQA3HzDRj+0ZCMZKkk+HssKyQOeEMQnHmcGu3ytZgR0bBVmWuWB4XjV7Gm0mC6UjPI5u /vlJaPLtO+8nRdQpTRKuV9vvP47H4z5wr67sCKBMVDL7E+Nh2UBAyneXFFZXMpxwtyK8t6MoheO11 GKbk6viOMuELBGPPL57Dm3XtOi46/OeDvoYI3UfRFQycc6KgJMtKqnfA/2GpYk28x2p98pxJrv1Y0 swJ3XM9Xxw9HsB4SwVkTd4gKXbnqu0945h02e2ef4uXYjAki/HF7VgucJ1oYtoJcsIkfpCTvZK/3X Td4QqWqA==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6f-008RiZ-9s for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:35:59 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33dc390f26cso233044717b3.9 for ; Thu, 01 Sep 2022 10:35:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=r773C9YRjCBNvMT4CzLpPbmrmyQxRNwpck2TtfCck1k=; b=ObA20LITrip87VziaHHJxhsQdt90Sartw38u0QjInvigVBBYp9UQt6FfSRSeXBXEZU PLrSrOLMVOwI3/sz3dUO/NCHhkXbIW3GF8M34MWk3xttvYsGitw637zJhtS5J3KbjVTq FLk9jwJNLpWEznbWA1xdOdydu+93KP79dvrPgyK7xujYPxVhRBYXk7sgB7YC6n2PoUnC kIiWKC2Oq5VXot4jMCKtmju4UEkl0k8b2etD7U9K2bdu4kKMwZb5udL9M4YbR2lJrZYA TRfi2Vr1zk+DfsN0D7bvgFUbxdpPUkljp0W3vF7qidlJ9uVGD7Pq+2IeCcnvt21qYUIk bD3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=r773C9YRjCBNvMT4CzLpPbmrmyQxRNwpck2TtfCck1k=; b=oYvAvCyK3T6yzKSx1ubxubCU+3IOMLINk20f56J4CHo81Lg+MuSKlVn/xcCX4zRpw1 L7Jm0W0R0fkCaDeh8qYHZDan+KpphC+YrVQ0JfRwXVooBepK+AHbIMPNxft99EXhYDpv xUktGBnUbwtlNZ2qSy2DDgkH9dlMIuWDEfE7R+HPJQ89xOr+o2l4R6n4J7qdDTMT/tAs EdoCZWhDI/PzCd8wKXNbR+qnSzeV4mzB9RSdRJNPnD2rMi6vmQhlP0rWlD93dMKdtmRU mY1ZyWyApVk6cHxGYSw5QItPsoSeHS9QlpdyobLqE5CuiuazNm8UUnOl1b0xbpKRnt+5 Oc3A== X-Gm-Message-State: ACgBeo3IoWMP/y+xhAvDtBv2YFcNnt06TlTIdaZNBvRxjul/9LlbAR2h 7x6w9DDF3pF6Tm3D/JRGG5zVegng9OI= X-Google-Smtp-Source: AA6agR5CArj0XA3uZUba700jvv45FjGtdD6CkgMKCuV3RkIgq+Sm8gt1hyIVkRx9I0S41UWPyjD3FlSg3pw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:dd05:0:b0:69c:97aa:f08e with SMTP id u5-20020a25dd05000000b0069c97aaf08emr9069882ybg.583.1662053755821; Thu, 01 Sep 2022 10:35:55 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:01 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-14-surenb@google.com> Subject: [RFC PATCH RESEND 13/28] mm: conditionally mark VMA as locked in free_pgtables and unmap_page_range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183557_500360_5990771A X-CRM114-Status: GOOD ( 19.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org free_pgtables and unmap_page_range functions can be called with mmap_lock held for write (e.g. in mmap_region), held for read (e.g in madvise_pageout) or not held at all (e.g in madvise_remove might drop mmap_lock before calling vfs_fallocate, which ends up calling unmap_page_range). Provide free_pgtables and unmap_page_range with additional argument indicating whether to mark the VMA as locked or not based on the usage. The parameter is set based on whether mmap_lock is held in write mode during the call. This ensures no change in behavior between mmap_lock and per-vma locks. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 +- mm/internal.h | 4 ++-- mm/memory.c | 32 +++++++++++++++++++++----------- mm/mmap.c | 17 +++++++++-------- mm/oom_kill.c | 3 ++- 5 files changed, 35 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 476bf936c5f0..dc72be923e5b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1874,7 +1874,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, void zap_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size); void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); struct mmu_notifier_range; diff --git a/mm/internal.h b/mm/internal.h index 785409805ed7..e6c0f999e0cb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -85,14 +85,14 @@ bool __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long floor, unsigned long ceiling); + unsigned long floor, unsigned long ceiling, bool lock_vma); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details); + struct zap_details *details, bool lock_vma); void page_cache_ra_order(struct readahead_control *, struct file_ra_state *, unsigned int order); diff --git a/mm/memory.c b/mm/memory.c index 4ba73f5aa8bb..9ac9944e8c62 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -403,7 +403,7 @@ void free_pgd_range(struct mmu_gather *tlb, } void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, - unsigned long floor, unsigned long ceiling) + unsigned long floor, unsigned long ceiling, bool lock_vma) { while (vma) { struct vm_area_struct *next = vma->vm_next; @@ -413,6 +413,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -427,6 +429,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, && !is_vm_hugetlb_page(next)) { vma = next; next = vma->vm_next; + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } @@ -1631,12 +1635,16 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb, void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details) + struct zap_details *details, + bool lock_vma) { pgd_t *pgd; unsigned long next; BUG_ON(addr >= end); + if (lock_vma) + vma_mark_locked(vma); + tlb_start_vma(tlb, vma); pgd = pgd_offset(vma->vm_mm, addr); do { @@ -1652,7 +1660,7 @@ void unmap_page_range(struct mmu_gather *tlb, static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, - struct zap_details *details) + struct zap_details *details, bool lock_vma) { unsigned long start = max(vma->vm_start, start_addr); unsigned long end; @@ -1691,7 +1699,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, i_mmap_unlock_write(vma->vm_file->f_mapping); } } else - unmap_page_range(tlb, vma, start, end, details); + unmap_page_range(tlb, vma, start, end, details, lock_vma); } } @@ -1715,7 +1723,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, - unsigned long end_addr) + unsigned long end_addr, bool lock_vma) { struct mmu_notifier_range range; struct zap_details details = { @@ -1728,7 +1736,8 @@ void unmap_vmas(struct mmu_gather *tlb, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) - unmap_single_vma(tlb, vma, start_addr, end_addr, &details); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details, + lock_vma); mmu_notifier_invalidate_range_end(&range); } @@ -1753,7 +1762,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < range.end; vma = vma->vm_next) - unmap_single_vma(&tlb, vma, start, range.end, NULL); + unmap_single_vma(&tlb, vma, start, range.end, NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1768,7 +1777,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, * The range must fit into one VMA. */ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, - unsigned long size, struct zap_details *details) + unsigned long size, struct zap_details *details, bool lock_vma) { struct mmu_notifier_range range; struct mmu_gather tlb; @@ -1779,7 +1788,7 @@ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long addr tlb_gather_mmu(&tlb, vma->vm_mm); update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); - unmap_single_vma(&tlb, vma, address, range.end, details); + unmap_single_vma(&tlb, vma, address, range.end, details, lock_vma); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1802,7 +1811,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, !(vma->vm_flags & VM_PFNMAP)) return; - zap_page_range_single(vma, address, size, NULL); + zap_page_range_single(vma, address, size, NULL, true); } EXPORT_SYMBOL_GPL(zap_vma_ptes); @@ -3483,7 +3492,8 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct zap_details *details) { - zap_page_range_single(vma, start_addr, end_addr - start_addr, details); + zap_page_range_single(vma, start_addr, end_addr - start_addr, details, + false); } static inline void unmap_mapping_range_tree(struct rb_root_cached *root, diff --git a/mm/mmap.c b/mm/mmap.c index 121544fd90de..094678b4434b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -79,7 +79,7 @@ core_param(ignore_rlimit_data, ignore_rlimit_data, bool, 0644); static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); static pgprot_t vm_pgprot_modify(pgprot_t oldprot, unsigned long vm_flags) { @@ -1866,7 +1866,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_file = NULL; /* Undo any partial mapping done by a device driver. */ - unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end); + unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end, true); if (vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); free_vma: @@ -2626,7 +2626,7 @@ static void remove_vma_list(struct mm_struct *mm, struct vm_area_struct *vma) */ static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { struct vm_area_struct *next = vma_next(mm, prev); struct mmu_gather tlb; @@ -2634,9 +2634,10 @@ static void unmap_region(struct mm_struct *mm, lru_add_drain(); tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); - unmap_vmas(&tlb, vma, start, end); + unmap_vmas(&tlb, vma, start, end, lock_vma); free_pgtables(&tlb, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + lock_vma); tlb_finish_mmu(&tlb); } @@ -2849,7 +2850,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len, if (downgrade) mmap_write_downgrade(mm); - unmap_region(mm, vma, prev, start, end); + unmap_region(mm, vma, prev, start, end, !downgrade); /* Fix up all other VM information */ remove_vma_list(mm, vma); @@ -3129,8 +3130,8 @@ void exit_mmap(struct mm_struct *mm) tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use -1 here to ensure all VMAs in the mm are unmapped */ - unmap_vmas(&tlb, vma, 0, -1); - free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); + unmap_vmas(&tlb, vma, 0, -1, true); + free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* Walk the list again, actually closing and freeing it. */ diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 3c6cf9e3cd66..6ffa7c511aa3 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -549,7 +549,8 @@ bool __oom_reap_task_mm(struct mm_struct *mm) ret = false; continue; } - unmap_page_range(&tlb, vma, range.start, range.end, NULL); + unmap_page_range(&tlb, vma, range.start, range.end, + NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } From patchwork Thu Sep 1 17:35:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 986AFECAAD5 for ; Thu, 1 Sep 2022 17:59:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Ia/+i9D2sDm6s7rGM8RNg0Mieqj+mMq+HQ2bO206k68=; b=IPtLxsfvypqBqJugV1EWs/iG1F CPmYDY7EG4znbVMTjIxM13VPe54Id28o2EUjioxMWQg1Ad1YS45C8Y/rfmHfWRPpS7ElK5hC3/ulB VHgfZoc403T1ty/zOFT9tToZcVO0WAvw67oVziKXYre2x+hN26Gs8pTYyg0Gj6HI5E+KEkiosTYxA 2QKOuDf6KRp/rS5eqp8EgsUNVzGvHzkgENVCM/5M26oX5pqmMoBX2gV7lofCerGZ//oceTFuJHvVA Y9M7Cn65NV5d4QgGjftfDPj9dtPtIn1dleEKffUMIXUK7P4QyIk9lHx+B/xDio/CqPTDgouGq7fzc QQxS54xw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToRz-00DkL8-2o; Thu, 01 Sep 2022 17:57:59 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToRx-00DkKI-0Z for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 17:57:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=CVmSMjmLYwB9FBtpYSNLJ0EV++HpteWYtmAikuEnyj8=; b=XeSo9FvEvGzyggMjN2BObfInaE TyCCd5+hM3G3KVZg6h6JBnPOGv41RYvYNqmCmwSwg9oaS9JcpLp5Xs5sjd1o3xECR2WYiThd0gswf oT524/rSH+6eIXzu9B1NwZS33gMcRXl0pfUL2AYkB+riuu2TFTK8LyIOOmBeetAeNmiQ2akMwr8kZ I1w1MAwbeJT8xo50onNtelIoy9WW+eX2uKK3kphLHm7fFJyztvJCMhIowYQ5FsXcNBko1u1qoF9E3 Rv+3U576eavBwldybLo5y9IF9mKbCDFic5p3gQtiT4JptSrRVTnfr+AG5VI0A5ikX+FIDnNRNdp9T bEMsQPsg==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6k-006Dhf-Ok for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:04 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id w63-20020a25c742000000b006960ac89fedso4937220ybe.18 for ; Thu, 01 Sep 2022 10:36:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=CVmSMjmLYwB9FBtpYSNLJ0EV++HpteWYtmAikuEnyj8=; b=RQU4Av3GuaGnYpE9oeVZ2Qc2D9oyD+ip4sYivAtlCkyS2/hNn9rv0v8p2M0NIGYJND Yf8x5cX18al4RtSO4Q1PhkVmfw1aU1Ab/Cll55dcKrRpQKauqrB7xyNC80pmZ6bW9bAC e95UH0jr+QtsvW6E0Yi+7Q29f7LTJtJy2OM2cQAlA/mMmKe3iC7olC3Rgqr7kYyQ1CJQ Co86J6JNei7V+ykfhNyFAPymL77cyA2ZQroFaHENAEs99CXDu6wa2X/Gpfr/HBQQdv4H QNCI3+GHYcOaSG+P0vp/Qcr7DGv7/3qUpIL4nQqbN3ksWTquAWAHtimY4d22gaOesNyT C0Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=CVmSMjmLYwB9FBtpYSNLJ0EV++HpteWYtmAikuEnyj8=; b=0Qo4TLBzpY+gxYktd1cCBBkWoEn/Krwk37n6LxkCGAuWra/07ZR1/SH2mMoVOCsAsF /Y7WWp8EOAH3iOFeeNc7p2VUyDXxhIpLt9ADAOH42uqqnfAMgO8MjqCzQ8V6+pDmN0uG mTHrceAwwnEl513fItKVW2eqt1po6qfGF8z78TO4Hx9l4vCMQ/mPrA26KR2Y9g101JC+ fmcp7spub6NhvryvCY0Ue3KC4V+nxKaDW/BEdK9XMBvf7cZEnN6Ph9V90lqQa7RtxoIs WUgjEvcnR6ItdKbYs0P8bAm3uuspN6N8ZPoSeofORNoRRTbDUc0txuMqGAiED/bmj+WN 2tRQ== X-Gm-Message-State: ACgBeo3QWk1J8pToPSUFAxRyiFo8twCyoKRaOpFwV6lWP3riPpyZgnwn 8cdKmQW9+QOFvVOQtZEJnzK6FdA06Xw= X-Google-Smtp-Source: AA6agR571otBJ73SV6fz6quBhFyvEDhG9tsbvxbf3EGsugmeaIr00wfh/qpDUZrPhQkoYyC6wjg4lFM6a0U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:1c3:0:b0:6a2:2d4e:6bf8 with SMTP id 186-20020a2501c3000000b006a22d4e6bf8mr1787307ybb.564.1662053758377; Thu, 01 Sep 2022 10:35:58 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:02 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-15-surenb@google.com> Subject: [RFC PATCH RESEND 14/28] mm: mark VMAs as locked before isolating them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183602_821918_3841EC4A X-CRM114-Status: GOOD ( 11.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Mark VMAs as locked before isolating them and clear their tree node so that isolated VMAs are easily identifiable. In the later patches page fault handlers will try locking the found VMA and will check whether the VMA was isolated. Locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ mm/nommu.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 094678b4434b..b0d78bdc0de0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -421,12 +421,14 @@ static inline void vma_rb_insert(struct vm_area_struct *vma, static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *root) { + vma_mark_locked(vma); /* * Note rb_erase_augmented is a fairly large inline function, * so make sure we instantiate it only once with our desired * augmented rbtree callbacks. */ rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks); + RB_CLEAR_NODE(&vma->vm_rb); } static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma, diff --git a/mm/nommu.c b/mm/nommu.c index e819cbc21b39..ff9933e57501 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -622,6 +622,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma) struct mm_struct *mm = vma->vm_mm; struct task_struct *curr = current; + vma_mark_locked(vma); mm->map_count--; for (i = 0; i < VMACACHE_SIZE; i++) { /* if the vma is cached, invalidate the entire cache */ @@ -644,6 +645,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma) /* remove from the MM's tree and list */ rb_erase(&vma->vm_rb, &mm->mm_rb); + RB_CLEAR_NODE(&vma->vm_rb); __vma_unlink_list(mm, vma); } From patchwork Thu Sep 1 17:35:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 75895ECAAD1 for ; Thu, 1 Sep 2022 18:06:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=CDorcDB4SUeztmn2bW2unt4tZv4Tgq/+Fb54BI4jzqo=; b=ZZ/y4Un7VC2fWL3TFO3ZDAmcyL cPhRIEZbqRz6qyKVLvg2mFV+wCj7m6Bw8VoPRkYbAqcPoLF/X/Wzm9Kgj03ygUBRStBFzRh5pyMfi GowfSwln/2VbJ+zVl22R35WqCRouL+tRuAcvwedDb6CL7iMcxdUg6gtYfU5P+OKYsyTKGM4IxkNa/ Pvt/afbqKJg7cgeWJ4tAx+I6ho4GDyRnhRL+CAsWdWntpWh4f8adNFXyqXcZi2orBMi69vJhV8BAK /fmIEi+3CVYc2HB8/qCk8ShOpK6QpQX/hFdS8AdjBlkBwBP6PRqjAKPTPc8abMltlmQnwvucgkPQm HuUpQw9w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToZI-00DnBV-Fx; Thu, 01 Sep 2022 18:05:32 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXQ-00Dm5L-OR for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=jyHdLFPHMbjXGLnwUzIxKnvlA/GbLLDhxKEqR7mLfqg=; b=fJhIE9mA69Kkmi5IHp6nIqVFz7 fQEaVKslIHZ0Q2lwqdcayHMP+juOTaXgfuxXKLzhX9iqMzo3gl/Nw5QdX9zdPsoVzjFE2EPlBF+Qm Cfi58X0hvXEvgenqHHwjYzlXkFUi6dssVrBFV4KzeKhcb+/JyHJgw2ya0rGZ3qjSyLjYIW9S8BasH xxYCmUHO0JCLlt/SBKivgQdUbbTbM5+G9XwbPdp1BmfnxmOVTna2GzWDftz3+arCJTyMNqdSX0pEW yW2z4a0D5HFzN8RrMVRANoAHU0N+CiJ9/H6kvjK4hrq0QGE96ewuS1d5umTuEeZFsBMBE77Kp+Bdt ja+YEX+w==; Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6l-008Rjn-7Z for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:05 +0000 Received: by mail-pl1-x64a.google.com with SMTP id c8-20020a170902d48800b0017545868987so3390084plg.15 for ; Thu, 01 Sep 2022 10:36:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=jyHdLFPHMbjXGLnwUzIxKnvlA/GbLLDhxKEqR7mLfqg=; b=TCOmQuZ0ealmr2rj6cwh5S5dr2tHdE6kJbl5yIAByz1pqanqNTRiefLGtYBY7Bfwis HCiC64ttZOFWEgwG30yQXn+Yd1NgQ1aDGCJSNFZhazZq9lt7mplm3W+aibX2RLK/lLN0 uAX/ZCVxAoOvxpbxeS81kOPFHzixGFCJRTo1WJXrfNWNtHfKZEdnfDSt0EcuBcHbmrUl KJFv/JkMarGz2fGZyZ6GmZqZcaDNiv8ls/HHTwoDoMD28NmijZHaa+6d5o2vJqrlXSFO lKUsBOqEBkv1O0j454aqLOLj6gi+Hg3Qlj1nIC3dhAGDngGkgQrOdGW8W2dVGnwocfag do3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=jyHdLFPHMbjXGLnwUzIxKnvlA/GbLLDhxKEqR7mLfqg=; b=by54GsJbyGjc9EiJdZwcQnBrTDA0xKCxXp3PoU9R0TpQWH0HDSuul+SN7dlHzk7dpA EWlnyhhEIiaJkuaXxmM9av+mq7WIcnvVLEmTmTOYARnkSOyLWm9MP2qvmLea0/cYEK1k pTBfHYeHFKV36ps5D5P9I1EPB34mx//aoRX/bA6WpXXlXh9l5f/rwTMya3lDVh4MA3yZ 7Tj72+njWkp2PfuAi4Kh77iyH8mNmqg40bawQoq28IIPbw2Ngbh6ZJnD4Jx+1X72ULTf o9PdIwCNu9V2RnEMbu4e7BHhwDfE2xrkp7qsGAN4qI9Lw/xESWIiiHGMXkgxVMMwxXKA zR7g== X-Gm-Message-State: ACgBeo2d59B41+F0YwH8zKxC5cLNj3NnaapUhxASdGkkPriaoWDxVEps sxZlyDBnfVs2sZDJO9HGJge2pG+XSmk= X-Google-Smtp-Source: AA6agR4mIrBI+VqdmK9XxquJrzUivZu8ctMQy2Mh5DPiMdzbr+5VMUvdiKSOqW1XRdAe5w+IQuXOVrbXRFY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a63:2cc2:0:b0:41c:681d:60d2 with SMTP id s185-20020a632cc2000000b0041c681d60d2mr26204273pgs.502.1662053761168; Thu, 01 Sep 2022 10:36:01 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:03 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-16-surenb@google.com> Subject: [RFC PATCH RESEND 15/28] mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183603_397801_9F6A5C1A X-CRM114-Status: GOOD ( 12.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases mark adjacent VMAs as locked to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index b0d78bdc0de0..b31cc97c2803 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2680,10 +2680,14 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma, * VM_GROWSUP VMA. Such VMAs can change their size under * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ - if (vma && (vma->vm_flags & VM_GROWSDOWN)) + if (vma && (vma->vm_flags & VM_GROWSDOWN)) { + vma_mark_locked(vma); return false; - if (prev && (prev->vm_flags & VM_GROWSUP)) + } + if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_mark_locked(prev); return false; + } return true; } From patchwork Thu Sep 1 17:35:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47030ECAAD1 for ; Thu, 1 Sep 2022 17:59:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UxyqG8JKFb3J6Qlgx18TqqOJ74oIqWnkiaf2U8OeJjE=; b=z90IMZAFO/Le52dl/iwJyKTbb3 RjLfhQd6W9dmQvo03fByDX/A+vSyBrvxnd119Y/LngLloLZ1H/zePXSQWvxnFWnaOPjzLhLI80aE9 PFn009SahlntlXTffglcSiAZ9Hyq0rXzZuIWdllmoDRQenhUbc6Upll0Jnz7xJIxyxX9i1GYY+fKc YLRxjFHF0gFyWqf4UoQyGEHW5ruB8sNZ9Uoeqm0Tah2rSbI/t1EVEOvV/3q1zWD2b8OYlLal1//TX ZJB7Q8jQLr2JIhiEhD1XGjfIdIzlYU4efOmyf5uI2Esq57yIPMC0B2+9f0qpXn+qaXf1oRZRMSoFz 5Wm33Piw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToS7-00DkMo-Hu; Thu, 01 Sep 2022 17:58:07 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToRx-00DkKI-VC for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 17:57:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VpPGH26B3JaNkIfQAA8lk0M2gmgFVNCTIjkTHyeGjq0=; b=i3nLalwwAXeY9IA1zufUiYGCbG zaN6zz6+i39zXGL85of5pJqm9D/7lzG/OC6Ld10zHB+kqHlBKjBekRi+hLfYf8Zd55E5SUXCXVYDQ RIz5Jrei1zU8YxsfuOGOa0QBWZlQHJ9cLO//vo6KcJNeAsQMCkiALDGMEszzfX2h2b8qLTKaTZpYa uq6ovmE3EJtL7CNvdhdugHHF+EPP8t2Insv9W2WgEUt5WQXYAQlRrBzP0hLgYssXiOmQtI42FFA6a npEAVjft2C/g+JcWYxERCB+gJb0s2IZY3850StNoY+k/5CHYu78teJV9pbrso+GG7bSdwQszl3Yni x9xGzHKw==; Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6p-006Dis-CP for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:09 +0000 Received: by mail-pj1-x104a.google.com with SMTP id e22-20020a17090a9a9600b001fdaec2c73aso5769752pjp.7 for ; Thu, 01 Sep 2022 10:36:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=VpPGH26B3JaNkIfQAA8lk0M2gmgFVNCTIjkTHyeGjq0=; b=pbMRwoSFwQraZzGGZr7x5dEEGoBSSezJwWCkFmGpQHOK/gJ9jZl7heInRLvxfZubEx Wj7mEqLcXRU9xxWHf3t+7qSl6CnhQuuJSVxfC2zOp1biHU7QYf19J34qbstjhGHRGexS HGkcltAazUrmNYZt5EcV0prllxXaFrscwVZ4MeMfR5YBnXhADmsl05WrPEXFggi7Gjas iHFv8bsKjSzXGdd4TTCY9kFoj2NTn7nIEkBcfDuEl9YbNXUKLbiljFvk9VYo3diziCwb KLzO8tBeRwrd5fwSMEAfN7AuoAZXQe3a89hFtIYEbzmUJ/R7GnSkDta2dkqtwJ/Rto29 WT6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=VpPGH26B3JaNkIfQAA8lk0M2gmgFVNCTIjkTHyeGjq0=; b=xzgID533BXKhxrCSIgJ2gdFGsg9duHAoUwpFg3TyQ2HuHVHFiPE/gUtc0MUOC4aSwV XE8/+PGq8ETVrqtxgYUplPARhOgfmHbpQA8EMS8xm5TeaGboxEbn5smkLf5+fgYsMr7R BuZRvfxNzC+eVnOwYL7IvioLLn/pjOF4iYcstKE6YpwB9MocvsLcXBORfdxutn5zTzt0 5JkwnhQPLlHoaqmMjMmZ6t+oSLSfTl0afcqSTxS0pMi89sVIm5lJUu15bH3LOW3brCsl tHChPLdmcYvWqm7ui4CTIRKjjuOyUdFLu5NYYv9DYADmKQmA1lLO+WlATN6OJn8k1NWf 0pPg== X-Gm-Message-State: ACgBeo1sb5NXr+XWsgs7DGwZmk4VOIWcpFN/+6xzo+RDDdlvkQ+kdWO6 aHRDklDhjEPUjUqkf0IX5LcgmZ+XssQ= X-Google-Smtp-Source: AA6agR5wOS2QTAVmPmwy5O8cxRkpTpms/e+UYL0OyAuyLY0hN8kJNMUzvy3oLdSFlDJRhvwngVhT3DguUUM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a63:b250:0:b0:422:e3a6:47f2 with SMTP id t16-20020a63b250000000b00422e3a647f2mr4539554pgo.478.1662053763882; Thu, 01 Sep 2022 10:36:03 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:04 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-17-surenb@google.com> Subject: [RFC PATCH RESEND 16/28] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183607_492642_577DE138 X-CRM114-Status: GOOD ( 11.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dc72be923e5b..0d9c1563c354 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -676,6 +676,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_assert_no_reader(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), + vma); +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -685,6 +692,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) {} static inline void vma_assert_locked(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) {} +static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/kernel/fork.c b/kernel/fork.c index 1872ad549fed..b443ba3a247a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -487,6 +487,8 @@ static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + /* The vma should either have no lock holders or be write-locked. */ + vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } #endif From patchwork Thu Sep 1 17:35:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7953CECAAD3 for ; Thu, 1 Sep 2022 17:38:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=gYC4VZe8EfGG0tJgDYd3tqpE0atjqUeQmomgrcrb7KQ=; b=XhJkz6L7xVJIMNQhw5BvGXrPHV j6jTGh3wGAgelibebDSxYPKWZhY4SBQqEegL9i/nql2xkH9y22CAH4Z499RqfDsP8EBc4lKPf32Ts Ekjk29tg5zxy8CMR09tqpeN/mFloJYAwxy+rArmEwSQ/B1MnKzgg6S0+5QLehl6iZm/L52RCQMSgW Fi4wmFRwnyRvVCuOO3WI/I6HEOrnJ//hW14qavFQzA28qap9GDtG13Tv8cO9ZlOokHUBJFn7fHI8+ N8fvu6eR48pVs7X7PmFIdgzGsJf/lhIOv+i1BnnUBBVdWC03OQzSooWxanj3EoSu3ANwKeZ6nW3Jp IsAbRmbw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo7i-00DcNu-Az; Thu, 01 Sep 2022 17:37:02 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6q-00Dbiv-1D for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:09 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-3328a211611so233216807b3.5 for ; Thu, 01 Sep 2022 10:36:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=b/hqij6YMtUKn07CMMZGWuY0xy+IpIQzVd6D2LweKqw=; b=KRAo4osvkXfti83dlr7pDy0khew3B+/GxQjWEAl6TnacUwl+mtLybx0YLCFQUFV0lg sqmNuGzCNSRuL2SViXVJ2r0wrXOlXyxs3u9yAmqqg3AvKB9F3JJSO7rq3ZQJpqtxodOF i7G/Yua6+zv1pQayP4MuTvdM1yGvpgFJMwTj6dIMUAQKUS2ErYsxnukeEQ3tr0wanncl yMd9bQqlR0P+NMVZ/l+rGeUj3MMbqQRN0a5ORe8qDMpkAqtMCeKLfNSS6669H06w2E3e HhGcmv1HGm3kJoo9dZc+gF0PSVEl7OpLVCRJfkfc6aws3/7f1ZLX+WRY47T74JB1rzMb smKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=b/hqij6YMtUKn07CMMZGWuY0xy+IpIQzVd6D2LweKqw=; b=0u6YRs0/UFGUBgVO0zX5WOOnPexqQu6661brN1M43je+pDTrJTirJgZoqwM5VJiOtp 7Ge3hv4Ygu/YrFTA2ctD6XX0XgYddScmxt4wnayysEPyudijsikSbufxSugEFvqlKkBj JgpyzEjYmuPgrPL38RbA2+JCTWsG2IGYU6z2OEIs1S2LdritaLv68t8+p0QOnj8eWHrG UKhMWqHkeEMRB/lxP5pEGaPgtUUXafBjBcnQ0tzkvI+jfSiNd3GAgJyAN6gxWp+TTiz1 uRLwMGIXD/9FDO1P0wo8uOCr/DvmW79hoyTFgQrxs7QZN6ORcsSw5VnB0YL5AH1v33Lq no5g== X-Gm-Message-State: ACgBeo3oeybGNLcSg2c854V/bgHRp1XJMhRMzCoPdXM7XNIORbMaOqGW VIqVLOQARrqH2cqb6qNV+ae51uTg0o8= X-Google-Smtp-Source: AA6agR5jEvG/zgKOeD87JHuYvOxB9Vp8GPj5NQSbzUKBEagOp3ZLZ7YArxKySukyQ3HnfRTZ0d7o3HLJ11s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:91d:b0:672:e583:9ed9 with SMTP id bu29-20020a056902091d00b00672e5839ed9mr20010197ybb.225.1662053766633; Thu, 01 Sep 2022 10:36:06 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:05 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-18-surenb@google.com> Subject: [RFC PATCH RESEND 17/28] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103608_094907_57051E9B X-CRM114-Status: GOOD ( 11.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Pagefault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to mark all VMAs as locked and prevent this race with fault handlers that would hold VMA locks. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b31cc97c2803..1edfcd384f5e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3538,6 +3538,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * hugetlb mapping); * - all i_mmap_rwsem locks; * - all anon_vma->rwseml + * - all vmas marked locked * * We can take all locks within these types randomly because the VM code * doesn't nest them and we protected from parallel mm_take_all_locks() by @@ -3579,6 +3580,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_lock_anon_vma(mm, avc->anon_vma); + vma_mark_locked(vma); } return 0; @@ -3636,6 +3638,7 @@ void mm_drop_all_locks(struct mm_struct *mm) mmap_assert_write_locked(mm); BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); + vma_mark_unlocked_all(mm); for (vma = mm->mmap; vma; vma = vma->vm_next) { if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) From patchwork Thu Sep 1 17:35:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9E96ECAAD1 for ; Thu, 1 Sep 2022 18:06:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=qZUgfF1ntECMoXqVBYpr//G+UvkoYUWgF9I0UAmKW+8=; b=A8gyTdkFAr7DyYPz4bOYYz0F/o o1CDZ+wqkO3i1YO1lg4xxpGy0jLy5EjN+FCQG9uvkmGngctZXM6mkkUP/18N3KsopW7N8TG3Hh2Yh 9Ka02iARj9K5kVWk8o99n8O55PukUawl8Oxxg6ubpi8yOo1uRHtd1quiNisqNwN+McqlduOsqENVD fWyKRqQHCgNyurtDbPbCOsysmjoNZ35ykYQCyX/yheJBNtRBJfuvOMAXJKPNImA2pSSw/iMAqUm5h s71Z/03MfGbNIyS97X+lG25iq5FF0LZV16kw9ZuhZ2qy405DWEsCoiVsRyl3pz4iPrfS/pWLcAp2K 0W0If3wQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToYg-00Dmrf-PV; Thu, 01 Sep 2022 18:04:55 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXP-00Dm5L-Kl for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=CCIJQkAZ/MO9EmT200FCEP3Pgl0ISbihE0u87lNvTtM=; b=kN+HW8Hb1i9B1uy5oQ+d0ikD3f JAm1dA3L/Dhlsw9kuraGDmpQw1Yg8a5sJOgV5taS1fJqBppx7+iZEPzwHdHh2uKkPHHN/jUiHI0Jp 5YoIxGSMhe90w52WsX3nvWOJ600J44Ws5irW+8RQSbj6zL0TILJADURw2KZDAoq0mWQur7EkhvPPs HoqvZ4pBB67GpL+AhKP2IGf2kPH0OiYkWZFNIeLncj/XlDDdbVk7S9NSEx4+GpTrtP+xysje9aKuy SXZ9wL3q7Ey52mcMoWxN+doHpdcAVvrC9FrvraAfXX5eMNgP3Q/xJsKxv/0STibtqZCxgRmCau6+k 1OU+skgw==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6t-008RlM-1K for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:12 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 184-20020a2507c1000000b00696056767cfso4956431ybh.22 for ; Thu, 01 Sep 2022 10:36:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=CCIJQkAZ/MO9EmT200FCEP3Pgl0ISbihE0u87lNvTtM=; b=TDQRkBpQdXfWEYFjLGi7wwliNZ+aP0KfPsImVIZlXY8UjoGX+0qAFXnFV7XGW8IS6l y3VXrf+I2v9M7dPJOYegMdSPcyR+aiyAikKkPyRgRG+NjsroPpw9/4EU6cvNjVuOFM2u OAT4KFsNKXknM9gpAanaNp2m/7ee5CO2zFw2akG6RhT79MTZBIfrWOjgSBLIXLwhTrek 2rRbzNpBpA/qCfbKpI1w2r4GCLJvz7pk5sgu4I7rxGww2tvEjHalildILdcL14mqDmo6 PZNZoe3Kcvw46z9n2HY12LGtvHVlCAmPyMAI6EOC1YQSkgyXm1pQjGEvvxRKe2fezN+L aw6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=CCIJQkAZ/MO9EmT200FCEP3Pgl0ISbihE0u87lNvTtM=; b=pUb57eZ4wJ+5Of1fkGzXPkZGR/ekTxcdLzIQupJXzL7YpeHNEiFyLFJGyzhZ+DhUaS lJ7AiId29v1ifeg6h/e0EFKRoxelhwyUfgeOJTAkjVtVofhZEFxKd5WMY0YjxPH+Msil wHzJZiSnbDqAuDGpIlcSlXmdv67coPgI3YlcUhOYjpymoHhr3unvzq/yDYVggquN1ZLb ZBytftPvLPEUiGVtsKYLFerO9dxKxP+nAXNOHJSJVeJuBe4is9YKeJM6fK4q7IqXTUb1 WaWEbEmaOVDFAGVdABrWbpCr5dJBtWZZAqx3tSBxWjK4rbU/ToOutHBOxhK3U/A708Fy VAbQ== X-Gm-Message-State: ACgBeo1SwZhMwhG5PCITYwjF1ZDBu7SRXaZ7kKIeH9CXrHAUTaPPT0Si qgtQAoXPxIs05Vszbfx/ftvYbAhb71I= X-Google-Smtp-Source: AA6agR4qr6moVIHZv2YyIMkU4NbM3gllPWtVCICGpkk6fJrsf7LoBe2Q5GLKhj1E2SNlSJehbfJ330C873A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a81:94e:0:b0:33d:bfb0:ff55 with SMTP id 75-20020a81094e000000b0033dbfb0ff55mr23236112ywj.322.1662053769056; Thu, 01 Sep 2022 10:36:09 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:06 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-19-surenb@google.com> Subject: [RFC PATCH RESEND 18/28] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183611_226182_B9C903D4 X-CRM114-Status: GOOD ( 12.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0d9c1563c354..7c3190eaabd7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -466,7 +466,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6a03f59c1e78..36562e702baf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -886,6 +886,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Thu Sep 1 17:35:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8ABB8ECAAD3 for ; Thu, 1 Sep 2022 17:38:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=6JR5hEq8X8TGZtG6VHZO/dOijSS4S0aaWRjUnTE8Q2A=; b=BTitrKqU7aKMjd0fVj/9nIkIgT 2El4pYALxfXYW2Mfy+GlSf+ADklc7n0S78l3q5QiIGkYc0rCTUCupTTeXMOW8Iupace8NYmD3sgu0 6j0idFZxgq+y8f+p8GStKFwY3+UPwQ4V42u9QksylLqK4XG46Fkvk4bUKRj2mWYkfgtUHm4C/s9fF jSFAP6GWMr9TrgsHM5wUSFphm/WuJnQrAAgwaxQSyS4Tshg9uEAZtniWDHjrjBX3hBUVT1joABKlW 854egTm1CfTGynQT1t3Y9lXU44n7waD6DeOua+80zLqp9Qkd4WP0J0s1yl8qyCl0BM3QMyb+WHJTa n1kL21LA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo82-00DcZH-PN; Thu, 01 Sep 2022 17:37:23 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6v-00DbnF-5h for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:14 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-340ae84fb7dso202794437b3.17 for ; Thu, 01 Sep 2022 10:36:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=OeaHSvnl3LkKr4l31U2Volg5xaVk+RMFLZSi6dgjePI=; b=YWMUacgHyaw45xClfelut3B8D4opaNKe1Jn1EvQGqzRnq9IAzY1SbxxDOw3yyYq+BA clLIFH+vCJyTvZqkcuk+nPAAVvk+cIcjjJaBgcqIDZI820XVBHBVS4HV09O0+jx02gz/ GF82/kOz5NOA2SL5LGwa69qRF4uUvf5PcNaP24Dp6cprq2xiDwA1y9+n/C6LME/0C0u+ LXVRqnwXAQFfakRcmYHnZMbYxKQr/xy2mJHdmdiugDAKPQMxfZlI1xF8c8WSAaPgs+Sf 8cODiZAHBJrDioo32O0YL9A9GYzXrzP7oLTKiCYFarAedegbhpTrR5084qGw0ahgqNnD Ah8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=OeaHSvnl3LkKr4l31U2Volg5xaVk+RMFLZSi6dgjePI=; b=Api08ekaNMq9Ojoa5ujQn+OkQO9iPFx6MKHJ3XokcB+3f3Zo3zaz+5hsO6kCiDRFOH WXrg+mK2hwuzV1vj5O5NTyid3mxLnRajbKWGsIy2B/AzmTfzwspnmEuyTzU5dTzbUfe0 u7c8m+DAah0uNJLXmQrH7M3Wj6Bbm1/MKf+E8D46NKri3hp+YJNpZ0MBZlbn93kwafze eyQXf1QHm+2/5NLwnzOzLKptDBtvijmEMOMMf+RxpVZtqpKo0VXpBoz6t2Zwa8Mum+Dc tjiE/jXhtAiPYAFg7DW3Au8d7kvVEGgKioCcWT5ImSMksFF5vDPUA9azkz/eVRi027qw I7hg== X-Gm-Message-State: ACgBeo33ZYJevRCjAluUwv+8qEqKcK0+s6aSSEF8rtKaa2upGzeujpOS mSsUpny+FugC5XOdOPq1pcAhi9R+ano= X-Google-Smtp-Source: AA6agR5WRy5fZ7FOeYwPAz+MoQjsxloN5BfCZpV2SaYdV4girws1lZ0WYC4kcT46D4arwsdXcJqkq+Ln5bw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:6744:0:b0:696:52a9:2d84 with SMTP id b65-20020a256744000000b0069652a92d84mr20473184ybc.256.1662053771488; Thu, 01 Sep 2022 10:36:11 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:07 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-20-surenb@google.com> Subject: [RFC PATCH RESEND 19/28] mm: disallow do_swap_page to handle page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103613_241799_DC832784 X-CRM114-Status: GOOD ( 11.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 9ac9944e8c62..29d2f49f922a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3738,6 +3738,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + if (!pte_unmap_same(vmf)) goto out; From patchwork Thu Sep 1 17:35:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6DBBFECAAD1 for ; Thu, 1 Sep 2022 17:38:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=KSwIG26qd68y8obYvIOAdOvDH17wvL0TbgsZ4elYA4o=; b=uN0mFcfCsK43EZ8nbuXe/dumCC DvvWJsQdbOfjjvnV0dJKyKTyy1Om+HSRvGI/hB2OUvfRl9QD5AH3VsyZQOnr9aFfyFbpt8Tg6JkJa pAydhy2XS9tlG/Fi3vXzhGkIJgPvEuEButXwlKJwI4WPQHn0tdnPtISkVueaGV2PTeBrwkHsmmo8A LOI/PHG9jBiFV+M0GLuKAAIm/q2OMm9k0lhZ70Ou+KbeBAn2t6Ia9JMoEjOToxe2kBkE9SaAcVwhk ZiOP06GJ1sC592PKa8e8omfrmrQuU15PFUHdIyEsSLfqrTsOnNkTRxZNZLR7tnqtQovSwybDCvFSd xTo6UuZA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo8S-00Dclv-NI; Thu, 01 Sep 2022 17:37:49 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo6w-00Dbiv-TT for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:16 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-3328a211611so233219357b3.5 for ; Thu, 01 Sep 2022 10:36:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=r+vF5Lo2i1OjVhetiEIJ/9q6C9lfd71c3zrmx7NgTrs=; b=H4jFJ8Dt7ZH5cKrXQpN0z8PEygipTnxhDURDwP2r/L6ST1buj0wC6b62XAFXTWTuHE TRectVq93AO6tGT7OX1/aI7LFkcRGjFnJwRUud+1aazBY2txe7GqBy/qCAKx7DzAzI1F 08tbsOZGFue1vB/xCd7zDPTLBCjfhCGGnp6kgmw7nw7+o4krMVIGwx+wiyWeS1/K9dS3 MY2rAdmwbAo0HfcpKpUrJZgpdcga0Ob80OnRbMBFu5K9+oe2e9P/nHO4ZkzgpYkHj1ef kx644FtyUEEJInyDO+nYaup+HokzUzXjSdh7on7VQkqKFKRY8gzUfUlLXLq38fi2PNgi LbOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=r+vF5Lo2i1OjVhetiEIJ/9q6C9lfd71c3zrmx7NgTrs=; b=zg7wdHq9J/GMzHdMWDBo4pASLkLIGm070yBb93s57IA3tXIh/EKTkfpaJ+U8Y1G3ci 7vhB3pMk6lj2gBCAnmBkDf68+tWYqJeczoZh+4OzvtkJa/Ck+qU47PUWqpF/JHLtKfUF KbCy1W6DvzetoeWeyyXvYnHXGMa4NraBFkH9eUcoCuv2bb/WDQH7RrGj5lZ1k/fa5AL7 QtOxwG0ZDDEX2HP4CaBpBsEgGz3gGQw3WCeNnYVQ7Is0Cns0+3exiUoqTI9aWgJU3zS8 gMOahLsFACICfwT1/GIeFGx+QTKQzH39w7Hjc07LzG9lRESJyF8itHVidSB1BqqXfNUM xPhQ== X-Gm-Message-State: ACgBeo081+4snW2yDzSfIAH1ud/aOxqG9lCe8vL86bQlQjOAigR+o+Fm a0dZbHQLlhcqSdR6YW28aYG5mv3yPQ8= X-Google-Smtp-Source: AA6agR4BDUj9pfWqCFb+xsqUI8F9TUy99Q1kXWNSZ/lAe4Y/IzT7ZfeAPxwUhMVzFJvC8KeIfmdI79dhOFA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:8d05:0:b0:68e:c838:c24a with SMTP id n5-20020a258d05000000b0068ec838c24amr19294531ybl.45.1662053773853; Thu, 01 Sep 2022 10:36:13 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:08 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-21-surenb@google.com> Subject: [RFC PATCH RESEND 20/28] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103615_022970_791B5EA7 X-CRM114-Status: GOOD ( 11.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 8 ++++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f3fc36cd2276..a325783ed05d 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -150,6 +150,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index bfe38869498d..0c2611899cfc 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -131,6 +131,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_vmacache_event(x) do {} while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index ce8dded36de9..075642763a03 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -207,3 +207,11 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + help + Statistics for per-vma locks. + If in doubt, say N. diff --git a/mm/vmstat.c b/mm/vmstat.c index 90af9a8572f5..3f3804c846a6 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1411,6 +1411,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Thu Sep 1 17:35:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47B66ECAAD3 for ; Thu, 1 Sep 2022 18:05:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=hoXnndhZXLo5aIgaycyCxO4b5nSxCgy9dzkYspZxDgo=; b=NzikOf7m9mRXF/IQSyIEzC2zYQ RnIhF0wzE1vp93LqfJlynB7V51cuGMzDZU4votLQWoQeLUP/WiMmeIkEIzsqsHwoefzibBkDUmqDR 2388GwYUvEcuKJ08oEiFg//SIfH77U8IygiiZ+1OhTV1PmG9fvTdsdnHjd2FHfzG01jZSGa1ogvVF IjRsPOmZkuAnrjI9RyEDmGP8dOJYKQU15jw+Txp9yv87kI8duw0POhHQ44Yq1lBLTVxo1sgUPZqV6 B5sBQfOhXonz/2Kmidk55wIl6x9MmlpMHP6ZL2iybK2s4tfKPl7arAoRX0YfieN4C4CLS32WUevAo Ofed+BNA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToY3-00DmZ3-Db; Thu, 01 Sep 2022 18:04:16 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXM-00Dm5L-Ne for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=U17mw/lvD8Omk8eo/K7cBBouOy DF8X3ISxhWY+q2LrktzRLii7q0m2IUTzJ4Uzwv5J3maVdkItQS+Dvo0E/57y2xMoV31M/eCaYEa/0 rVgR1veAjIiSJTcUfQuZbWIoBuvMggpUwy0QGHpnTrn/3kmP0CTLpwYLWY5XpS+TlyECjUmfkN0iK Xm+tKSMgpQ8K4iEOJ7wSGfHskVzcCHIOu+XsE8rpj5vdK7wjLSRLxhjudj1kCe2NeU4koA9+KClWQ ianpQHd4RwGupDrwNYn9RBj0Cc5JA62PEe9uaapsaiu6v9Mv1YHuveod1w33+9+0Z+3jajw76t0zR Cr8UbaHQ==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo72-008Rn1-2T for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:22 +0000 Received: by mail-yb1-xb49.google.com with SMTP id y81-20020a253254000000b0069339437485so4864878yby.5 for ; Thu, 01 Sep 2022 10:36:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=eEZ4BqXkAmPCIUFHBPa2tHlcApvv3HMidc3eCYWOXuAzHrpAN0HS0lhYeZxby9RzBR 0w6QoRpV+iKQz37VUGml6Y4ETbNA7OPStJ8QApy5HYv7GzeLDO7+EvEq5dUOIcaM6UVC 7rLgM8Qnok81T8RLLnPTDqNW3fzyiOCIgOk9erIOBMCvX1CoI/RbBp7yoBH88B/Pqkwt H8ojE/1F974lnPVAUpGISzyJjuOBuorgNT9VfkfOyqVqOEVMNKxpaqT01PsyXu4rsbpt stRow1K+x3Cpkks4NKMTKIZbGi9T2v1db4rNZMcbp4CaDTkAzPc4JsqrDHDLe5K5oO1C a6Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=2Fz0JyGUCaaQ8O4rHOkTIk9V/hZXoJRCCDqiYpYQhbwx+JYeX2jklHOKIpMatgWF4a kcWG+vsKDHNbiICxUrZAeJXjmgF+YYXQbOOKjkf01Hd0LgUk1Y+HmfSRjmU5YvVv3E/C JpSaQYIkapjq4QlKiCnoF2m/49HYAFG9FmkEKjnmXLWEun3SaLBPtSC/bhOzXdQnRAoX UoUtGyybs5+3Q143iTsYZBNIilPTV5kxuzWwz+yuVt28rcYCpWcRuSN+A2dLrbWmKWPG 9ktVHUKufPmd/vwN3dYRHpSeJhXxSOPT4v1C3XblkhISNnMUQc5fma+3ttwgxZ5lFE34 5Xog== X-Gm-Message-State: ACgBeo096JrzQ9fCsjvuVBywZsrDEfPeSrhT1lfvS5PAQVwznaXmAVK+ 5l7qtfhVxO6p2LsxVxo+jveVVL8msAg= X-Google-Smtp-Source: AA6agR7n//3fxGZjvfyML08bNSpeGfau+aTgkZu5QxUJzLJc9oUWAiawZA1UWbNFrnWa2xxdQVaE06YdPF8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:18c:b0:695:9d04:c3e with SMTP id t12-20020a056902018c00b006959d040c3emr19907526ybh.58.1662053776523; Thu, 01 Sep 2022 10:36:16 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:09 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-22-surenb@google.com> Subject: [RFC PATCH RESEND 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183620_864639_04897B54 X-CRM114-Status: GOOD ( 16.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma = __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be stable + * and not isolated. If the VMA is not found of is being modified the function + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma = find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Thu Sep 1 17:35:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0ED35ECAAD3 for ; Thu, 1 Sep 2022 18:05:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=4SlolyXYHS8oFio2c/KswZ/uHOLyuAa1OlKeJoOpd/8=; b=DB53mf7dDYCUkSkTdHt3AK/E6z KZtUKzktRqXYyM5SnejWUA8ueHItkZbfvCUt+7FEBJ/X0a381zN+y4punGBzIBCtPgc9x7P7T/+jA b8mW/JfyPcdIDjIoxJh8mvXPVnTPy3mG82yDZ/3P4FGay9lNW3SgG/Xb8hpse+r8f+MOpHH9ugWPb YHHwkGC4NfGHF0nZgsu2yhlRGRdwidR2tyzNVTLikrtJo3SQ2jfuU69azHINsz/pQp0OYzjI/JQhs 29f6r681cKov7C3E0ToCxiiKX0U8d1Mj5KiqwyjfZKfLSEjlxX8fVoVe3vlxhS9Ws/vNTp8QGxTEt vESMQPgg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXp-00DmQy-JF; Thu, 01 Sep 2022 18:04:01 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXM-00Dm5L-9w for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=TUk3GnQXnzjX+dOXwqqLb3j5iouD0aWKTRx7OYILWmo=; b=QY3WWEbSXQDmPTuMo5wSpGEVj5 Rtdp2J6esEIrHKbMZpDoOx5OkeyHFfVH1nFW0iX2zSfudGBnhx20zkocCeGvip9mmRXS6lJx5xbrz 4Br9KKwzyuYPBSxryRM4yRjAHtuRCZGDds3A4iMGpqQJlvNtDrhu5PbVLs2GzV9gq5g6X6AbMF98P 4CbctgfTBg/LQYbf6U6RjcTZTTVwuJd81X7wbmvAWwM7y9LQN4OoddDpr/5jBtYlsudY2dTYLCvQr VSyV3ZTEJbSUfEvcxpEc0fj1DuOSCYrXYg9hsdvI2pdLwvLvPpOPgPeiflMclgh+37x18dOb08S7J /l9/gfJQ==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo72-008RnC-Ua for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:24 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id z4-20020a5b0a44000000b006a1c47c36beso1426823ybq.15 for ; Thu, 01 Sep 2022 10:36:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=TUk3GnQXnzjX+dOXwqqLb3j5iouD0aWKTRx7OYILWmo=; b=YQqRrP1cOBMmwt/q5otvkRLq5UlRi5nXamV0kPAgq/Llb+F9rrwouZsmTwkjFMW1zs mWLW2S110jPh4eYPUbnJWz9aEFHyDND9nIDA90HkjT5HQ91YMufxWCmYs0WU2ooUk8Du wzdtu/YDYEQeIpF7CzLVtbR6weBauuj1l4a8qgmBNwRm70UISBCLpd93jjiqNIdE0vgs 06VcH4KM5paek23OhAbPGd46WTDb6+CqL4Pyf7E1NmlU70RTzjmiFKgXWz0xeqjkJr9W NG4J3tGGklDETuONG3YWLZrzk72gi/1u5qstFzrzMP1bg8GyXe5bAXne0/Fv+EFg0xsd ouLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=TUk3GnQXnzjX+dOXwqqLb3j5iouD0aWKTRx7OYILWmo=; b=nDMBUMAImgHxo3MwQ5PHiUlDjbTjlrp4yZVDmFR8GrALt2cG8AHbHu5aoD5BbG0UGL PpLG5VXw2szDvv8xlHABX5eP0B1lFIGL8h3HqNvbXD4bAX8B0AU9XXxhx/hNBDIjeReu ExZEwD2oIWiVpgD0Fl5wqBM/iHmAe+kZR0MWqwQFpnzqKF7PRRAkAxJydb7KqZUwafIH DeclS18LtROi/abPryRCJr2t6AYmhQREGtHf7fzsJaj+RSDs44+VSjAPpZwtQC5G03Cm kdwMWN7TsgTs8AWG23IhsW85/bjvLdX3l6sog74kdoD0GG4Zmtfrr3q8EUAIWIPG3heV D17Q== X-Gm-Message-State: ACgBeo3fIBfrmGbTPuC0UX7lbgw1FX9cpgNn8RARVDXFlhk3lyWMy3jk mV3++cmQviO6q3aAZ8DGa0Q2bvf1pIM= X-Google-Smtp-Source: AA6agR5HFIn9x2Nfihxx+X7rvP9mQrh+0fzYESFNay7KqUAMxXqLyyvr7HgjmPtUD/1VpOxvT+Ik33q1JvQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:5f10:0:b0:6a3:7eb:762 with SMTP id t16-20020a255f10000000b006a307eb0762mr718810ybb.85.1662053778908; Thu, 01 Sep 2022 10:36:18 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:10 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-23-surenb@google.com> Subject: [RFC PATCH RESEND 22/28] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183621_161194_E57700FF X-CRM114-Status: GOOD ( 14.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index fa71a5d12e87..35e74e3dc2c1 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1323,6 +1324,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1423,6 +1456,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Thu Sep 1 17:35:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41E99ECAAD3 for ; Thu, 1 Sep 2022 17:39:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Vm6YkkxKXK6PIsneJyc6khrS6UB7FvOZqsv06gp1IZU=; b=bz/0FG8hF/1FNtXO/gURRz3dOd g/nahv1ezHl//OvQG2ynaiSSNUkFIOE9tn30Lmn9Fvzpa/dgvgySfYVYTGg3ezzplUnnTwZYs9ML4 MnYBCR3I+oXRCS5T9mEjzpRrYCOhWs1kQ8K3RLA4Lsh/S0yAoDkHxudSiKhc7SZxXMb46eBjNbrjm HLMjkRKaR3vO1l/8gEM35Cv8S//xtIwY8kxnrWYZApnK9YJ5JbfSRJF91nUGUPtlrzblqZMof91T+ /fPtYfYFLm0rVBY+oJ9rrZWrIQHF/jmx2iK4cQSn70TmE/euGYdVaHyKIKiU7CFwig7y5cmuZOoHH cTmaadwg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo8x-00Dd13-20; Thu, 01 Sep 2022 17:38:19 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo75-00Dbvc-2i for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:24 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-3449870f0e4so11680087b3.6 for ; Thu, 01 Sep 2022 10:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=OXbMCMaGP4Ta1BmgGlOZJjCZdxKE/BRmpYL5lrPVZHE=; b=pWHztwJetxx0qowvevLvkxkCmwM1DSAQ7wjfJnXzpjijYQSBqPMfI0rG6nxi/Vrs1X CRVPAVdxEFSJIvF4wmbUwj7aCZVm5fVfcrhjUQbBf8n+p0SayMBlb1is6WVkqvLBgMjP AxVJ3sKIAz8Kn28I4Sanz2jyBLjhA5Sgmh51jOnVumpasjSH/FjDxpS14KsDzSHBeJ/z Yb37UvWIKHVN5xc1I6JjnIWO9Obyji3HzdX43E3p6Zl4/DUSGGEIR7W2aNpSc1VwX28m zHCfb00UkmpquMPwx5yKTUZ6c6Sedty+81FiCoVN909m08MOGYjyTskfX4dvSl+gNSYm xxfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=OXbMCMaGP4Ta1BmgGlOZJjCZdxKE/BRmpYL5lrPVZHE=; b=CE6fAwxz+Uvv6+ekEzXYslNMOfHWpqxjFqVLlN1U79cky55mIb9ADrOsUzM/szKGXC KAjOrpLes1wQiaCL2zdhcSggoN0rA+rYEJ84N+9ZIuLUx2McBYTVgaf8FL9iXXS+TCjj egIHVR10zS78YCbHRpxLYs332Ia1HBYypGMfWPM1mbEDRfRtUwTcmwy9lAM75zex7E9n rrqtYEFupog2QNv/Yl/v27htF7KCnja8KLbUSi5DRwVsHIh5XzhmdcxgtfEgmGGb70sk 17IW+ty5U1aT4MBXwiCvJOD3cLJr7zkAUZ1i6FOweNlR3ViXVypIgeWQqQu2YHAhOoAQ G9dA== X-Gm-Message-State: ACgBeo1ax8JWAYeFbQXoDLuF4WticBYaMsTDyZget/RGczblnMdgupzl iNQya1izYGQb00Bw0jkNXpw2LC55dKA= X-Google-Smtp-Source: AA6agR61BxBFn9qdHU9nkuJxxLaKayaG/YdyWtcsF9HYohujUkmqtvR1t3mL9v3jybeuPbaW7lxRg4vyZUk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:340d:0:b0:696:4dbf:614a with SMTP id b13-20020a25340d000000b006964dbf614amr21859755yba.470.1662053781470; Thu, 01 Sep 2022 10:36:21 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:11 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-24-surenb@google.com> Subject: [RFC PATCH RESEND 23/28] x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103623_156270_21386241 X-CRM114-Status: UNSURE ( 9.74 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f9920f1341c8..ee19de020b27 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA From patchwork Thu Sep 1 17:35:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 426AFECAAD1 for ; Thu, 1 Sep 2022 18:05:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=6NUtZJzQH5TNdeBx/n9DFcxwlvNIDSGNHsSj9h3lftA=; b=augp6TAEpmDfefCO9jF5WRkL/r RXzrFGPJnp1fvmgwKuH7oPLV/2h+iTrACVaHxkBCBeVxnbaaHWJRE+IUOPje5CMFFuOsFXKLRqKBQ RKrHZHH6Mfu/CJroL5SjZtcXH9Saj16BJiF/A7X2COtUEZOeaKM7N6Gk5QpWV/gF6T+0X4SLFNVWQ sYkQN4af3O449fFe7glYviX7rj/4Jm9U4vj8HcjUHaoYwxhSYHMTWAsCAL5d8MPVES56P+cpXu1L9 A5xqU1mg7eNaoYaUAsKsxhs9gN/KvNOw1BfMjZMzYiXHuQjcYp3CL5NnPi/fA+ggff5WBD2dTQxOD Vi8g0Dnw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXg-00DmOB-0e; Thu, 01 Sep 2022 18:03:52 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXL-00Dm5L-Ri for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=2BdbpYN+cfrtXte3gt5P0NOm/DHEryE/xIPhm7Rz1xg=; b=gsluqkSwcikGu9sxgUUTAFS2Gu mSawk3tz9hxCYwEeZeZvx0Jt695deqauN1Orn8idOImXYMKYeJrU5NBmGfR/4yVlHDjAIDikp5DRD KBS1F6cKPvzCShLhYgIG0tg9t3FZf25A/nYXED79fSo+C4Y/87q54ZVUu4dt/1etnv5+duWCLaqKl hddQQ13br5gWeLqK/WZMHBEVOGhds2/1soTPv/hkpMDlo9wkR0HgbTIuVzs3opy1rz9YEGilEjYvl IUlsYEXDOwgBnbqFnnKIGAJgnz33oCmgSfrJ3gj5j4/tWeV4rpab/P6Ig2OUhOTzWwJw+bRgFNG9k JtX74zjA==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo77-008RpB-N4 for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:27 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id f12-20020a25b6cc000000b0069a9e36de26so4921408ybm.16 for ; Thu, 01 Sep 2022 10:36:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=2BdbpYN+cfrtXte3gt5P0NOm/DHEryE/xIPhm7Rz1xg=; b=VLuZCO0l2heZzbazUktWTHDEraaU4YdHq4KUbShg6HpER9jaDaB5f4y1Bzq/1FTL0k Hef4re0Ge4vhRYArZVPVcwZDWlez9FLwBw1PsG7bcW/Kbgpuv9Pp9OJGNf6JnrGh3HSa qO0dYIzHK0WtoIREDnu1aZZK8mbeTv1bHWQjDQ3XJuqrg6Mb+eT6a9R2PACl9oYgExkp BYTnrJsd86ZN2nyjWVo4xWQ85kSe7EiDolkrvd8eM4uZtObKMsI19IUrWn7JY9Y8YYi6 3keq2T13Rmwv4h2gxcKkEIlsATye3QMfjNN6OckMSJWRV6GvnHlPAwoTd+EEwFOjiqHw hSeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=2BdbpYN+cfrtXte3gt5P0NOm/DHEryE/xIPhm7Rz1xg=; b=3ccMO3ORpPDbtr2UTpUxFw/mcszizWD6FaVDJJGSQuqtfv6fcKAIzuGz1wprtKf8AY m41rSpVtHuFUx9QJX72dHCV89mCXnucGjbR/cLHRQ/iEj0kDBwXgf2+CuAN6i3v9dMUO vUsQK7wF/bDIwh3PpS0uS5H4i1nGxq5asXNk0vVqrPdiNGx5x9INToRnU4iaV02eaH0n 24JF68iulzb+ogOfS6eBvvljE8S4R23y/ISwDb+rDBTCnROIlnLZuYov1wMpge2piLWZ k0woDMiXsNc8YYzsAoCH+4X2/1LUEtKAPLMnszPH3eAidDnmzfBD3VDEGjdwm0ZY8UX6 Ttjg== X-Gm-Message-State: ACgBeo1STqx5fa8wZPaGDLyYrRyo3J6zANDeTZppMwFtbESXmTYyCJ1j ULEZNiAHJDT2cw/9DNXG6Y1/mxFiCSk= X-Google-Smtp-Source: AA6agR4CvVkCfuV2stbCR2feIvgwEhmkbLBkcVFIjpWyBvtco7HBu4vzJWRMfUlZ+fh/HitarNYnWQuKjvQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a81:5a56:0:b0:33b:52a8:c360 with SMTP id o83-20020a815a56000000b0033b52a8c360mr22687651ywb.329.1662053784110; Thu, 01 Sep 2022 10:36:24 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:12 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-25-surenb@google.com> Subject: [RFC PATCH RESEND 24/28] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183625_833245_5AC3D290 X-CRM114-Status: GOOD ( 15.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index c33f1fad2745..f05ce40ff32b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -525,6 +525,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -575,6 +578,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -618,6 +651,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Thu Sep 1 17:35:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63DAFECAAD3 for ; Thu, 1 Sep 2022 18:04:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=FRsMXKtX3k0nvHi7IqS5XXpjuzGV2dbSEnpRbyVt+V0=; b=ztTsMjeGdS1Q6G3latM+uL6JGG PYRZbf3/u6V9qWIDImWk+afr5hEcVEZiar3Jb1KJN5oBf6aEutiCnKNOi9yod7+6tQb292dP+ufw7 WWxkBq9703QRonyuxZ4B5PnXfPUacCN9E71UyexRIUBDP7LoSynQjoVTTyhyv9Bwj5+AvpZBqTVHd 44YyLCWiBewjufMZEQNd0e/cvnwxtwugrQWgwf9bprPp79vDnsNwud6qB/ZN5JB645CELD4HswDfs Ie47EHBeT4jopxXZTlPfo626q5EI6o06GJZkPjY5oqTw9mDTVWd3wsG08vfqQy5JFwQdJQ4SukV3K V06mF85g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXU-00DmJa-6M; Thu, 01 Sep 2022 18:03:40 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXJ-00Dm5L-2k for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=e91/L3Bkt1w9DT3T0Co0rdM+lRiTKLDE/Mz2x0qV8w8=; b=OgpellBG4/jEPkmI432Dwkmp5H oANkklHP8+CncM9O9GySObbrbv6hGUkXaLt2p3P1z1uxURFyd5LrSkEC/A20BArrLkGxNIdSZhTcj 2porZVROVzT528TQzWBNKWm032Y+vUkXYDW9mhMouErSzq6TIozrgBY1JUThcN52YutAkbhR2BFRB MMdEG7YIh+BA25Wz+RHkd6e5otCE1gIEN+gjWphcHyxccvVmLVXnz4lZ4EfKIERGm2c2T0yZY3SeX iu7I1a7HMk7NTaMtNpI8ngANog1I+d1AoLHFgbZ35ZebfGA6/FzpixCzL7ZXu4VAJLl/dVFZiiuw9 MV3yjbNg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo7A-008RpT-FP for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:30 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-335ff2ef600so235547647b3.18 for ; Thu, 01 Sep 2022 10:36:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=e91/L3Bkt1w9DT3T0Co0rdM+lRiTKLDE/Mz2x0qV8w8=; b=O6gzuYJNv3OY1tULNg+rSFd+sWkIjQSaV6yE8xAfnZPN1YTVFfR+wvj2NeC2hBY0+k GIVWJsEk2y8yIAo13lsMU4IUJIgKbBiJ1vv+1z7dNT8xT+lTOjC6URMO5pLiEVmaefML PewWj9g+Vcp5HQ91HsRQ365b/kw+OZnJbC/u2QqpxScDN6oJLilMCO5pf8M9kEkOXKy8 kK1hF6rXLpQy+lNimDC91Nhhjc7ZyM02kAhq+89cKu94J6/Mxp9HnTyvO1IKLxhNY5UV lVIZFG9WwP01TdBwwmdFuCPUOkErkzSY1WgEGRWny4u3kPuzbkYMq5itJGF6CAoS3las sKWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=e91/L3Bkt1w9DT3T0Co0rdM+lRiTKLDE/Mz2x0qV8w8=; b=KYzko3pU8wQWqnhbJ0fNvWB6EUx3Ipd806WElX6NnBLpPtXIeqAuZuAaxZDU1KASLO 5uRz1r5f8g1CZPXEqUIA2toPrbWF0PjN9iAvpqLgsOXcHc1WF322fI8fpEDNPscAslk/ aMhBTL0Kgja2VzUvnNEXlSIj+X+HmE9GExq2D5e/q1Zb7/BBN+o5gPH07PmL8pktDu7r YCtDvfzbt+tKJZBUsENGpq+g1aU16zRVZTe1jx6tNA+SK5InmnfP5IqUyBe2aA0slXve i5tzgx1W1nnJlEVVmfonsBds7JRma/pdPBUhhZLMn0aRpUVyWyTxD/+NwwnIlSv5YtB1 +crA== X-Gm-Message-State: ACgBeo0+cr8VOPiRTxM8YhWBX2DEGNlTUYpCu8VfraMoTDWppNsuWNhU ktILh2hKXEKHMSyZqZDgSNluuuuvwNY= X-Google-Smtp-Source: AA6agR6JzhhY8hWMnjFDm/o9DPxBMt9dAFxOiR6uxCkipERPD2P+sojVgosImEy5Q2VS3ZGfPJeoFsIcTII= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:154d:b0:695:d8b4:36b2 with SMTP id r13-20020a056902154d00b00695d8b436b2mr21142474ybu.599.1662053786750; Thu, 01 Sep 2022 10:36:26 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:13 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-26-surenb@google.com> Subject: [RFC PATCH RESEND 25/28] arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183628_636531_E780744D X-CRM114-Status: UNSURE ( 9.63 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9fb9fff08c94..0747ae1f3b39 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -93,6 +93,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT From patchwork Thu Sep 1 17:35:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BA6B1ECAAD1 for ; Thu, 1 Sep 2022 18:04:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=LTH49qTy8AaI9jMYlXtt42LZZTaI3Oc/WNx+aThf5C8=; b=DeEKDIt3XBhTIXt5Rqcgas99eS Au2DbbVdFMJnYXW7CF1KZH8fbCRkEUcWLa8Vn64+xyU2U2XXqwrb56qfntVg/dUFtmC6U/N4GdIeF RNaUlr7wT11a3XiasCm8LMTxJx+dgFIec7nueFD7VclEaq45z1anRmW53asHamr57ic8p9CwCL7Lr a9s+VZxocd+V6Q2PIjm40g46v4twUm9AJt6rdI0vgO6si05Tah4DtSDnsx4ftAfgXxIvoaEFBx7Ip G5wY/FSvO9kaD9iqh88y/wPg8RviYZ5obWPtm5RXYBjtTzA6E1Qv/Id3ektsS91wa+PdsXr/ZAf3A lgrY2eyQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXJ-00DmFP-Tb; Thu, 01 Sep 2022 18:03:30 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXI-00Dm5L-16 for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=GxTYNSy2f61J0vFNTG2PuheznwcSGYj4QW8UtE2cKCs=; b=WDpNDyORnEEzMtLUStlf53e239 A+/aD+3Nh7KCEOSeYqEIB7eDa+YIBB5iGu3KApJGqhx72YrmMewYquGsW1rgFEfNnv3+M3BS8zEKD SmKrrnFxC90Zz8C4cGBxJDjxmYzGovGGYQEQ7qKM5SAR7mdzOUPT7p7OKdVLVlksmkc8hxQ3Vm9Oz kugQVmGF4krIo1h40ywLff/wDDKJdGq0gro3qRu2uUhA0KZwcfxqaZq6X5aNSXNhpUTpYXVBdgvJy qLLvJxEPmULfOCwvAgROP0NUVn7zcZr287bYsSj9UTWaOna+IvP0tOo9Bdd9L+VgzBT0IBTJakNYz QYJbgkeg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo7C-008RiZ-JQ for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:32 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33dc390f26cso233054867b3.9 for ; Thu, 01 Sep 2022 10:36:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=GxTYNSy2f61J0vFNTG2PuheznwcSGYj4QW8UtE2cKCs=; b=kyDYAAajD8Ict5/thoW7CTIWmHJF9O2tLJRCa3mFAvG6icZFINT6cR9wp1PVCpk34d ZW6ubLRV3shjslvaFAJXlbBGTzPKjvgVDQeukMqdqtwYsjBpj2zHvD901sTZUMPhJtXT 2ltqU4ShN/OuGiFl+1aMyLRwy+OC1Eq/JzpPgbsdEFfqg/fH8TS912PoVzzOgyXwQY4c Neb0lZJutieMXKVcpmbRLxmWRs14AnetVBjrLGSATrWD8Sx1+eyAMSMAZjNqX2Pwkr4/ vSA7Eawe2WnSKfYYTXdrBr34gwdU8C4wAuI9C4koFQPreF/LzpOWpEs87ZTSq2ciWD6Q PcMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=GxTYNSy2f61J0vFNTG2PuheznwcSGYj4QW8UtE2cKCs=; b=f+Haq0tsrPRVSgM7+Ed3SAgYfe3Z2teZfDSqKRQazfp0wEw+8kyyIP8eDOte7ZdkOy CFEBlQu1f6+LdvBSuM87WK6lLEXNZyF+Lc7SYGvVINUVShQ2uT5dP+FDgqm/+evwzCUb +KjnpT3TVnIwZ8TCNx8cIAdMHKhmMQYZPoMk0ytEO4gIpKLRC+3p9sLPuS/nskyeZe4B //4KJdLNcu/NMW2ba6sUtV/GaZ08LfIYtvyWMERfP2hc2VlZD4xVmyV6KZobl5XQrJfD GnybLP1QIR7W4YI6nkh1PjaMCYw6IugJ9XTmKJn0BCKvZUfIkhH0D2ZHw36o70qj9qS3 aHoQ== X-Gm-Message-State: ACgBeo0roBkii0oSO0ybjjwh1j4xbqJ7wV7HUx/X1ODf5FGxn3xxB7Zu B3ZDHD4LwBijruBFyIEconhRSYHlgMA= X-Google-Smtp-Source: AA6agR7h/HXbYJenfJy6WhKZaSvOltpWgwSfG3A/YQRl84q3AZw84By7flMTVszUCEhzYjvcGJ4xQ/dizwg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a81:650a:0:b0:33c:f030:7e9f with SMTP id z10-20020a81650a000000b0033cf0307e9fmr23813234ywb.16.1662053789340; Thu, 01 Sep 2022 10:36:29 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:14 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-27-surenb@google.com> Subject: [RFC PATCH RESEND 26/28] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183630_759673_8D2CEDC0 X-CRM114-Status: GOOD ( 15.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 014005428687..c92bdfcd1796 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -450,6 +450,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_read_unlock(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_read_unlock(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -526,6 +564,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); From patchwork Thu Sep 1 17:35:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C7DF0ECAAD1 for ; Thu, 1 Sep 2022 17:39:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8CYfkb6nYRv0nJ2JOGsV1s32OTSq+gFXpIJ5JaUnn+0=; b=QlxNeDBFZnn8KH+wmUAaJdwYjJ oe/I+yyL0bwnSOIWgzN8G8277RPYhs24PExKZqxqrxbEcnp1IqYBjQNQ92g1CBZvK93Xi7l0YsecB Z/MbZDmLtVllzzAEMY8nTj1gABh73oK9TTWyP9gbjssTVSjhg6Q41g9rAtJ+OG4KDzyAsNTG2iqP3 FrgcHY/fsy+x13WpVdb4RwdXBWBzkPBwXczACMcOCr+8lpZAWkMXSaM84iu+sCUdtXx/1JnBZEgYd O40XKNbCtwUNFBI2bxIiARTtte83d8QT6+Or7Ilf8YQEwjR+M4al58RAi5FTULu+CmTfzkklSYQMZ ovuqZ70g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo9N-00DdEx-Sz; Thu, 01 Sep 2022 17:38:46 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo7F-00Dc5X-KT for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:34 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id j11-20020a05690212cb00b006454988d225so4967780ybu.10 for ; Thu, 01 Sep 2022 10:36:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=vKtrq3HIq0qVh4puVrk/99Ll6PM2V5XVbSUsmvDjj9I=; b=tivdYiD8ZAJK8kY23bdnMjSf6GqsaVRSwINiFtGtPiGjoWq5MFwTdXcs6F+jY7ZJCK /S9q6olR7R7v5bTdyGGJAkaj1esNAWRQw+6z/ncs3LyR7k1VQNGfhn+1/vYhCXSOHiL4 6opl/Yux1TzXEQKGzwy0i4oX47fV8vpc6BEr00D/YdbkLyZ9SHmRwW4kBhG1xNo5Bgh3 mtUhM/EXxGiBQ9J+E4RQUvuZwU4rl5n4anzWcV73TPe35S1Oe6kIZC4EGb3KKcT8isc0 o4+RnjpTFgDSTn6r4q7bAZvWDnDcTYKIxFWZmBACSU/yC2EIvY26FAMGgxLd2T0siGu4 jeUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=vKtrq3HIq0qVh4puVrk/99Ll6PM2V5XVbSUsmvDjj9I=; b=6lg5tXcLDz00ZwWWBV0EDels4A6YsH2Q4Lq1sUtbpMLRwewjgTqgpunqGD2xJ+XtY8 TYqUGNizV2rfPvjonmFCuVOtptc4UwDtPg59DyuICTWer3kdVaO/6jIEa2vsFYEICUBt vfA3MWtB9Mr6zbEwHaaMzefP5ceC9NsmqUIDWEqO+w7eGUAn6h4ygZ4TQ6WyWDF3Vwmg HHBmA/TJNu3S064eZaieHkhJ2VWk4yUWqMycFWIFmRrdJ+6R4FNZ6Anlo+UbuCwozhX5 eKFDOZp5rY5MGewE0GhnF070KDDRiPMCa666vpagOFECtE+d152P15kl/4Dz7DB5EtLg MHOw== X-Gm-Message-State: ACgBeo0aj9YRLDx68epaFDSpuZFAVoW6mLlL3TMfoqLDtPXxYRIdo6vz EfWxshA+Eo8HkBjaaAd93nhx5e+9KVc= X-Google-Smtp-Source: AA6agR6ETpVykmR2l2EiORnO/7/6aTPnOWA5nUbQjzShI0yNekZbMOoTb0Et/3IYHZBh1H/HSgpOo7WMY1M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a25:9a85:0:b0:69b:79d2:7f4e with SMTP id s5-20020a259a85000000b0069b79d27f4emr15568357ybo.379.1662053792100; Thu, 01 Sep 2022 10:36:32 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:15 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-28-surenb@google.com> Subject: [RFC PATCH RESEND 27/28] powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103633_738547_33886A6D X-CRM114-Status: GOOD ( 11.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Laurent Dufour Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on powernv and pseries. It may be use on the other platforms but I can't test that seriously. Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index fb6499977f99..7d13a2de3475 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT_SPINLOCKS From patchwork Thu Sep 1 17:35:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12963018 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4424DECAAD3 for ; Thu, 1 Sep 2022 17:40:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=jVyJGrsE/JMj0U3CiLQaAZYzr1Ls5uP1gPE9B6Ek/SM=; b=Qprjimmb1K0qx6OzTY+1zCIYzq BECTx9dpgRJKF8Y1woxCrWm9suWVXXlJBz6IG/Nzz7zN16be8IH/EiLD7HmQDXOeBzLyf7fxsx5/O hJwNz+SEK14Ri7JyqVbzuiPyc/U8G9jlOInrtKgXUmxeKjP6qXYAuag1JICh1rDDe6eGiK1qLPu94 XgNARMUAcm0u7MWu9tEhxIPGc3lU09MIH10f1vItKma39myBojLnYdBgtaPRof7jvjrI4yAyvvr90 5pFpjcDb2sSF44NCXbvz/pUqzSX/wV8UA0EyUEd7BSzaST+9Ghq61cL724ByNenEoijkAocvzCA1p smAJZ+lQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo9s-00DdSd-Fq; Thu, 01 Sep 2022 17:39:16 +0000 Received: from mail-pj1-f74.google.com ([209.85.216.74]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo7I-00Dc7r-US for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:38 +0000 Received: by mail-pj1-f74.google.com with SMTP id nl16-20020a17090b385000b001fff2445007so9540pjb.7 for ; Thu, 01 Sep 2022 10:36:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=UrxjF4stvOlBFlzO5LCraBZUw6+ypd0ZdM4ea92X40o=; b=bRQh8Am0cRUFMzevPI1RLDrAVB56xe5Pyl618ML1xG5UbYShnlVJ+JUC6EbLQY52Qs CIQg2N84xHgD40zt1+Fs+lh7e1EeSPRAe2LH3ariXyDtYJfOUFwmaa2nyKSPAMv4C2LL ixfdMT3qEKwsQzU0zWx46k2Kgwbo6qL2ZMjk9kwAzxSkMtv3E+2VRRPySs58VX6WIHgc UmMCPxZ+/fBN36M7ZZRa1nwJWHDtOFVZGjkXHsv+iTsKxj9Rn4GNhqh8P+D/9RlG7Da5 L4TwcjQTaJ8DlcHg4rJEt6EYpCMy8N2APz0jZiSGuITg1YyaGh6hU+QCGH5FDg92dBPv Wubw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=UrxjF4stvOlBFlzO5LCraBZUw6+ypd0ZdM4ea92X40o=; b=dnPdi47btGB7TSgLtE8GpU/rzsgz4t5pBwFRNQtlDeBTC8GaaxoUSVoh2PkFXpa1yW bKywemAtGS18zGN73u+vatJG857R4xk2VmQW56LvXGs3wpBL30X12keKh4KFIs37qn8A Wk3ljkCx6mbPundqgbCqfikdc6oCmdbI0DWMM4Do3j/QHuPWGOYOXneBcWlRFlH0rOfc L+ALnnpyukMkONAZ0Is6EtlBydUekLwS3pkdUbfP8vZfbgUtNCaUK7b1tuPdK4hFgZG7 rrLtB9DQoNezvxT+sDkhLDAUIoh0MRGbRUsdaoenHemccEOtnVz1VhsQmQbxw+uNZHOM hqRQ== X-Gm-Message-State: ACgBeo13HWj7yT8VZZypeOz6jo79pi+3vLrXWn40CKhY2NO26VefEHL1 LhoOE0y5/pCKNji/fGQlnulxueTdtPQ= X-Google-Smtp-Source: AA6agR7mDmCeuzlvD1IqXjWBDc5GDtYGg6tZms9fC4JNMJ+6ZNkDA7fZXZfEW+TFSrB8Qt+rbpDI3NajKAM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:aa7:804c:0:b0:536:80bb:8270 with SMTP id y12-20020aa7804c000000b0053680bb8270mr31869167pfm.13.1662053794990; Thu, 01 Sep 2022 10:36:34 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:16 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-29-surenb@google.com> Subject: [RFC PATCH RESEND 28/28] kernel/fork: throttle call_rcu() calls in vm_area_free From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_103637_019481_E54903B7 X-CRM114-Status: GOOD ( 20.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. To minimize that impact, place VMAs into a list and free them in groups using one call_rcu() call per group. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 1 + include/linux/mm_types.h | 11 ++++++- kernel/fork.c | 68 +++++++++++++++++++++++++++++++++++----- mm/init-mm.c | 3 ++ mm/mmap.c | 1 + 5 files changed, 75 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a3cbaa7b9119..81dff694ac14 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -249,6 +249,7 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +void drain_free_vmas(struct mm_struct *mm); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 36562e702baf..6f3effc493b1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -412,7 +412,11 @@ struct vm_area_struct { struct vm_area_struct *vm_next, *vm_prev; }; #ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ + struct { + struct list_head vm_free_list; + /* Used for deferred freeing. */ + struct rcu_head vm_rcu; + }; #endif }; @@ -573,6 +577,11 @@ struct mm_struct { */ #ifdef CONFIG_PER_VMA_LOCK int mm_lock_seq; + struct { + struct list_head head; + spinlock_t lock; + int size; + } vma_free_list; #endif diff --git a/kernel/fork.c b/kernel/fork.c index b443ba3a247a..7c88710aed72 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -483,26 +483,75 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) } #ifdef CONFIG_PER_VMA_LOCK -static void __vm_area_free(struct rcu_head *head) +static inline void __vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } -#endif + +static void vma_free_rcu_callback(struct rcu_head *head) +{ + struct vm_area_struct *first_vma; + struct vm_area_struct *vma, *vma2; + + first_vma = container_of(head, struct vm_area_struct, vm_rcu); + list_for_each_entry_safe(vma, vma2, &first_vma->vm_free_list, vm_free_list) + __vm_area_free(vma); + __vm_area_free(first_vma); +} + +void drain_free_vmas(struct mm_struct *mm) +{ + struct vm_area_struct *first_vma; + LIST_HEAD(to_destroy); + + spin_lock(&mm->vma_free_list.lock); + list_splice_init(&mm->vma_free_list.head, &to_destroy); + mm->vma_free_list.size = 0; + spin_unlock(&mm->vma_free_list.lock); + + if (list_empty(&to_destroy)) + return; + + first_vma = list_first_entry(&to_destroy, struct vm_area_struct, vm_free_list); + /* Remove the head which is allocated on the stack */ + list_del(&to_destroy); + + call_rcu(&first_vma->vm_rcu, vma_free_rcu_callback); +} + +#define VM_AREA_FREE_LIST_MAX 32 + +void vm_area_free(struct vm_area_struct *vma) +{ + struct mm_struct *mm = vma->vm_mm; + bool drain; + + free_anon_vma_name(vma); + + spin_lock(&mm->vma_free_list.lock); + list_add(&vma->vm_free_list, &mm->vma_free_list.head); + mm->vma_free_list.size++; + drain = mm->vma_free_list.size > VM_AREA_FREE_LIST_MAX; + spin_unlock(&mm->vma_free_list.lock); + + if (drain) + drain_free_vmas(mm); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +void drain_free_vmas(struct mm_struct *mm) {} void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, __vm_area_free); -#else kmem_cache_free(vm_area_cachep, vma); -#endif } +#endif /* CONFIG_PER_VMA_LOCK */ + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -1137,6 +1186,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK WRITE_ONCE(mm->mm_lock_seq, 0); + INIT_LIST_HEAD(&mm->vma_free_list.head); + spin_lock_init(&mm->vma_free_list.lock); + mm->vma_free_list.size = 0; #endif mm_pgtables_bytes_init(mm); mm->map_count = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index 8399f90d631c..7b6d2460545f 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -39,6 +39,9 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK .mm_lock_seq = 0, + .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), + .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock), + .vma_free_list.size = 0, #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, diff --git a/mm/mmap.c b/mm/mmap.c index 1edfcd384f5e..d61b7ef84ba6 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3149,6 +3149,7 @@ void exit_mmap(struct mm_struct *mm) } mm->mmap = NULL; mmap_write_unlock(mm); + drain_free_vmas(mm); vm_unacct_memory(nr_accounted); }