From patchwork Mon Aug 29 21:25:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24604ECAAD8 for ; Mon, 29 Aug 2022 21:26:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=dMGs18Nm4kd0hc5Ux3vSqnca0NUpVeeeJyi3d6Ps4Xg=; b=QjII/h8qBDV+hynX4wYOw+/CIE wSg08W+7ktUL6Ii66a68Fjxcx0rViJx84xJCzepxnUOYJ5hKtyiEu6cqpHM7G4Y34SXUfqdBj1YgC A4cZMl0EBIZKuRApGPQbpU0UclrFEm2UJM8f3iEC9X96J6l0y65Cv1OSHXFOu5YjaA7VoI+DeyEr9 xN6vl0GUVXIZVn5B0uR0reVozgPR9nDMwRXtCWqwNqGqKGYySdLm3WkzPrMjVyvMb+KoLOQFARigh VUSfXmwHZiMYnnAm2gQkOlCUHgt3qV0B+REwx65exsn1eC3L3PdmSXv79/u5byzWh4r2z+ngCU/+0 HPjERi2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGX-00CnEM-62; Mon, 29 Aug 2022 21:25:53 +0000 Received: from mail-pg1-x54a.google.com ([2607:f8b0:4864:20::54a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGJ-00Cn4z-NM for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:41 +0000 Received: by mail-pg1-x54a.google.com with SMTP id v135-20020a63618d000000b0042db630eeb7so119110pgb.20 for ; Mon, 29 Aug 2022 14:25:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=g4qOnyYThXS8aPbz4q0EiUiPOz/BrsimeRGT1yiKfMQ=; b=mWzgq+g1hUcWORjEKpHNvgndKK5fYzq5Kwo71ogdoOHT+8RKeEQm1dWhj+jH+fzB93 eAtnxNDhkrvgMQV0NbZW6cX2N1R1+vx9KkVCTcRNm2sUroxHvRgBw6l3WRg5Zt3VpzAp Gtf7jyf0iomGZE2cusptU53XV2fFMuQkz5kSqUL16OlTWKpS5jWEYB7QcGrOHtGU2amu R0EpT4v+6Ubf6b7FtWlOEkEYwXOj5Qkd+op5DjD/MSrHyHjvI8tucin/+y2TJJ3NqFBm d0kZdCDqO99cL/X5NSt2crz3dZBhVMVoGOKJ9zBhEnqaTkK0+up1kisxNJ5uiuFsJR5L UYkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=g4qOnyYThXS8aPbz4q0EiUiPOz/BrsimeRGT1yiKfMQ=; b=jH6kEdD4+VPInyfL5w14v1vRv9ANUp0+4l+DdJ/wXQgH4tlUUsb/flU+T4q7H0EVIC 29K3MNIjLmGlmnone2XG9by0quYQ/uS/IAhH3+Su8zzn+tOB7SGJoKph+3VDH/NvDRlZ o2DHP1q1X7GP1r4zVSwIE58PccrLWKhvbjNOMt1vFHjW2q5kLMeWlOmeL7OssU9UZduB sgdIZ5HgM9RTsPbUdCTJo1bRX2achgtz6k4sWRqLSVxDHkaI3+UDwbcC5K4vFudB0mNh 4YQ079z3k4gmIvU1c+LrWUD5DdbAMqZQIEE8ExYVEgaXUJM3Z2oHKVkIocc+bQ8zQWnn DYNA== X-Gm-Message-State: ACgBeo2Z+ZfGzADbbZcwqJWNpqp8Gw8PRPCum3cup+0Q0Ev3/JsPG6No bptayujtOPZFsiPukrpv42Nv4UCBVtM= X-Google-Smtp-Source: AA6agR70MCvwbcIgBEgSXgn9wLeLPhFpHFOZXigYtzK5DEim0KxhWXMFn5l7ORAicxSiaBREEUpeMlkp2eE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a62:5ac6:0:b0:537:f0fa:4ae1 with SMTP id o189-20020a625ac6000000b00537f0fa4ae1mr12574833pfb.70.1661808337643; Mon, 29 Aug 2022 14:25:37 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:04 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-2-surenb@google.com> Subject: [RFC PATCH 01/28] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_142539_777034_3B34F782 X-CRM114-Status: GOOD ( 11.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 0331f1461f81..58c20fad9cf9 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1124,6 +1124,19 @@ config PTE_MARKER_UFFD_WP purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + bool "Per-vma locking support" + default y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Mon Aug 29 21:25:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2F9BECAAD2 for ; Mon, 29 Aug 2022 22:35:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/cctgxXx5p+wLt2fVrlHloOXlXKWDtJLB0a8oPyJdOQ=; b=xMf2r5Qz5viPMuYOAN2T+ZsTpa 4GV1aJA1gJ8UpO+vo99cHEARRFX8GkNXb4P5HRfpGznHs6gGZR1p1ZM9b8duDtsgLuOAreTtQjStD cYDPAeFXsa0qmIZ5cozpMNlTPkgNjOaxO4HAKhNOiE2DifmaMTLv30qyR2ykVDLNYtLLs6M7d7siT cc7ATH4+ep0mxpj8mGgXuwD9Xc8Th3cwwtwrH1I0OUa68SvI56QLbJJKGnNwITeFi9Jn7tWnvYO8C 0GCX4xrcHG151WUwkmtBgKsScBcNNPbyQfs0J1YbMBodvMqQyAggTr1BT4K3DGVaS0uF+dULloUX+ 6ef4lotg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSnLR-00D4Qs-S8; Mon, 29 Aug 2022 22:35:02 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGK-00Cn5k-JR for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:42 +0000 Received: by mail-pl1-x649.google.com with SMTP id q6-20020a17090311c600b0017266460b8fso6784160plh.4 for ; Mon, 29 Aug 2022 14:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=cPhW2Jsm5VYfGL+tNkfF3HbuaaNlLZ4rflaa6thsEZ0=; b=Ef83aGWTdm10niN8twSSpG3ds5UuJPZXIlmBxHFg2pzrVEdcovVnDh3P/RYOinFzyL 56g3s4OBiiWq4sYS4zyOn+I3sJdcQzzrlMjpsAwGpe/6DNb56LumMLV46gfcGkdOoroQ Xn2Ez1+n3LVumJ9ul/e3Fo4g/eC812QXSiR/GOAlZ/X37IBJZ8GhOEEF+zxhXhrBOhrA wyQ3RxsgNeUblE3TddZPmpCUfJPQ1JoHkMsqbRXnrreQjskaAOAntkxj5MpJLWS1ppBQ s3y364HBN4NsSi7BBvtTGWNjoBJVZ6VaguwCOOzVv7z+u5EVx5HkEViGicNoS6cS9car 5ukQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=cPhW2Jsm5VYfGL+tNkfF3HbuaaNlLZ4rflaa6thsEZ0=; b=MZtAsLQ6MdPzqpCGxnQipftuCmPfq4J0M4AoLA4YcQ5QbsBs49bAFxFyfNjzijDkGt sWF5ofciOg5xtcw3LQJGwiZQoMiJVSBY6jhf5TfWCisnf0qJCPqI/L2y0sPayI9mKbdI QiL4OQGLoPbfZYEEJ9mG+kje58MFueHTK53QUBiIMoNhurxg7ZcXFVLRL/U9rcCdcaD2 gJDYL9E9AB1HrMiOcW0SkjleElN2vEG8YYUBBEkq+8kCNmphUOR6RFPopTJkbPBXTF96 dL/l0iE/37YuXpllHYTp/G6SkfsEJ6vG6yCdTDS5ZRv9VVYdFtZPgv+fSF/Im0rXFnfp 2l3w== X-Gm-Message-State: ACgBeo0zE8WDhCWY6VfE7pQ8ZvNWetBESdMYQTWXrKx5IBHKoA0JsSNu HMGR8CkCmQEU67idBr6C4zakrGioXz5rNqAb0wyLDg6vtPiFY3ViP8D2RiBp4f8ocW0k+eYmhLE OyAhyRiZtTlQPC8I/aThKKHqpCHcGNoU= X-Google-Smtp-Source: AA6agR4h8d6XU8wf7Rba89JuzvO6/M5QQD9xRgO2erAMFyngZx2nEIGpax2LUa1YM3lFKVFmr9HgbHhBg9U= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:694a:b0:16e:e270:1f84 with SMTP id k10-20020a170902694a00b0016ee2701f84mr17988146plt.89.1661808339339; Mon, 29 Aug 2022 14:25:39 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:05 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-3-surenb@google.com> Subject: [RFC PATCH 02/28] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_142540_676244_482E0566 X-CRM114-Status: GOOD ( 14.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 16 +++++++++++----- kernel/fork.c | 13 +++++++++++++ 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index cf97f3884fda..bed25ef7c994 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -403,12 +403,18 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start, vm_end; - /* linked list of VM areas per task, sorted by address */ - struct vm_area_struct *vm_next, *vm_prev; + /* linked list of VMAs per task, sorted by address */ + struct vm_area_struct *vm_next, *vm_prev; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct rb_node vm_rb; diff --git a/kernel/fork.c b/kernel/fork.c index 90c85b17bf69..614872438393 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -481,10 +481,23 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +#ifdef CONFIG_PER_VMA_LOCK +static void __vm_area_free(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + kmem_cache_free(vm_area_cachep, vma); +} +#endif + void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free(vm_area_cachep, vma); +#endif } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Mon Aug 29 21:25:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8726CECAAD4 for ; Mon, 29 Aug 2022 22:35:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=2di83HamPUv27yt8b8F46ySNvW16o/6zD4Eqgor1cnI=; b=O2Wy50pep1jeXxtoK8WCpn+/3U rfdgEclB1rRyoaXKXdGPwMOYvbgSnE1ZJHLekgMA47UB3jIOav8xOApBQydgNdm8iWYOd+9eva06q A0Dh8J9GXftCIpEG/OifkUBb+MY9amukK18xheSGjcDQy1f22l7/kxgnvillGw1pPgfeWtadxbNlc 3SyqrKfrMoZvyZcoQr2sqaDagBPLbA0CwU6A+Rlz06+zJxNij4NUy7GAllvQ6M2nMSSSCuwdDdh7s VWEVy4ViMVg9QsbXRDFCJU0rW0XFG6tBAdXFwC397olgeGfUjYn9/oxPr630hrXU54TeoDV72hc/0 GaCMu+aw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSnLa-00D4Ru-Qq; Mon, 29 Aug 2022 22:35:11 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGM-00Cn7d-UM for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:45 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-340862314d9so131710827b3.3 for ; Mon, 29 Aug 2022 14:25:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=r66X5QU6Hd1zKKrQM4VdH4G/hUbDUrS2PjBFocCJ7kE=; b=f0ZITQpEExaLaSKDQb+h3/jYVFf5BjWfu0A0p9NMJ80xDBwkhscojqDzgDQ6W2gLIU JqyvlNc6N0TYQxhXpPiBzGuyzj1xr5cqDIbNzDR0v00qVutFwkDUQtyKnSj2Dn+NYRmF XbYPUhr7ZlEh9zlxoVr+7UyC5/R2ipvFX7ptRmhGZgjobP0J3ITaBrEOKND2CAY9iU2J fnwalrszPulqwRWfSp/VwWSeSLynEBR581yqV4QALN8e0aPzId5HIRtAjN2fmV4usJpw ghPowcSLA4CLZg9Bm0wFQDYbe3eb9NsWWsLPsurv1jkHcSRK8KYhRIFhxtRmb7OcOQkd NeDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=r66X5QU6Hd1zKKrQM4VdH4G/hUbDUrS2PjBFocCJ7kE=; b=VrXYnRaRt9NUnevkTQjvjb3RyB+5Ozg5vFV16AuR/CGqdhiMMpVkMnbqeuFBByiqqC pQAc49aXEk/rmj+we0nqBEOda+3DunlWVfC3mA3H9sOnLTG1RDBujgd+tk5u66gkvRdG cdaMQ2CBZ6mxz3CmFiTArIr/EpCAmCvQl6+M2H0XFrhZArarJNWEpDsSMq4JKn5pOc5M 4NSMvAcIymLFAFMebFcCXMTompObrWqRtKNWX4cQlDuMy5z/tKk0OedVkGdhbVwtpmC0 mF6YE7QLua4h+hrIPSSPCU6Iw9jz74Z5Kmvyh74zI+r7p0DWYr4wxQ4+NSEbGvTeZ0tF nuvg== X-Gm-Message-State: ACgBeo1iKtU4MWKbBlF0UebHbZPjTIkqTbcVvaHNWU+PY4QjZd16gtX5 C82FVOUSh8gQR8ThLRart7Lg5aiPyPU= X-Google-Smtp-Source: AA6agR7RORyYEvgHp7XQu1QQYhj/biZijMsx0xRWZsAvcqgghjoomkWy7B4TiExDCzWJq5q+aaEXQCYL4WE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a25:254f:0:b0:695:7c37:9202 with SMTP id l76-20020a25254f000000b006957c379202mr10232936ybl.79.1661808341087; Mon, 29 Aug 2022 14:25:41 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:06 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-4-surenb@google.com> Subject: [RFC PATCH 03/28] mm: introduce __find_vma to be used without mmap_lock protection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_142543_008042_B520155F X-CRM114-Status: GOOD ( 13.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add __find_vma function to be used for VMA lookup under rcu protection. Signed-off-by: Suren Baghdasaryan --- drivers/gpu/drm/i915/i915_gpu_error.c | 4 ++-- include/linux/mm.h | 9 ++++++++- mm/mmap.c | 6 ++---- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c index 32e92651ef7c..fc94985c95c8 100644 --- a/drivers/gpu/drm/i915/i915_gpu_error.c +++ b/drivers/gpu/drm/i915/i915_gpu_error.c @@ -507,7 +507,7 @@ static void error_print_context(struct drm_i915_error_state_buf *m, } static struct i915_vma_coredump * -__find_vma(struct i915_vma_coredump *vma, const char *name) +__i915_find_vma(struct i915_vma_coredump *vma, const char *name) { while (vma) { if (strcmp(vma->name, name) == 0) @@ -521,7 +521,7 @@ __find_vma(struct i915_vma_coredump *vma, const char *name) struct i915_vma_coredump * intel_gpu_error_find_batch(const struct intel_engine_coredump *ee) { - return __find_vma(ee->vma, "batch"); + return __i915_find_vma(ee->vma, "batch"); } static void error_print_engine(struct drm_i915_error_state_buf *m, diff --git a/include/linux/mm.h b/include/linux/mm.h index 21f8b27bd9fd..7d322a979455 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2712,7 +2712,14 @@ extern int expand_upwards(struct vm_area_struct *vma, unsigned long address); #endif /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long addr); +extern struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr); +static inline +struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +{ + mmap_assert_locked(mm); + return __find_vma(mm, addr); +} + extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr, struct vm_area_struct **pprev); diff --git a/mm/mmap.c b/mm/mmap.c index 9d780f415be3..693e6776be39 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2250,12 +2250,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, EXPORT_SYMBOL(get_unmapped_area); /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr) { struct rb_node *rb_node; struct vm_area_struct *vma; - mmap_assert_locked(mm); /* Check the cache first. */ vma = vmacache_find(mm, addr); if (likely(vma)) @@ -2281,8 +2280,7 @@ struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) vmacache_update(addr, vma); return vma; } - -EXPORT_SYMBOL(find_vma); +EXPORT_SYMBOL(__find_vma); /* * Same as find_vma, but also return a pointer to the previous VMA in *pprev. From patchwork Mon Aug 29 21:25:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0D98EECAAD2 for ; Mon, 29 Aug 2022 22:00:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=vWKRFxRZynPRLJkZ91RbA4bnWoIqbTbMme1pUvF4gaQ=; b=SeB4AdHojMM9MRNBaRwc7Qnbww ZnMpJC6tqp2mSITm8cGLh3V4VO4U+0QkdLvj7ZDOoeTblAfjQ4sTQQQxfk8QOU4pPe4Nu2QW8rzRG AhTqP01wjJMpeyMP3llyc3TaEdiarNcyBIegvc6Ig38nZ2xaCPcT2aq3vE4/FJW+1McqmtRiKUzFJ yBk8DpaMgIRxXeE0LqXrDQeeNwpT6d3McGcIeafyTa6z0yhuz2iZlN5MuQqCEX7BGGoJK87xDHra1 ULENCnmoGlbJFiDM3MTq11GSpRTxNr6A5LKphMsyMBQmpv2V36cPKoy6QysbOiXyVIBQxSw4RG6Fi N3z/wkWQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmnB-00Cy5l-PV; Mon, 29 Aug 2022 21:59:38 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmU-00Cxu8-Li for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:58:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=VHFOj4kfUa3xGA3HxCxfqSH2fyPnkF//iB8ZL7PxHTQ=; b=Dk+sqOqN/3arJgQexHXOpetUne GiV6wFZZ+Lg9o1v1Im9xTS/x0VlGlv+z5NKIJq526eCq5M/h+ah/tl8SQA6decvQ9gQD0K/Cr+44E Def+MrM+UpZFHG4WgPBS4CyWctEYGlkH/e245jmGwytM0XnraRybcAZx/KpYHz575aCiDvrrxMIT+ Zdt+uYREXTHM2HjQUvncJSE85qHDkvEpPRC1WW+41vK/J6KVdehz2WRo55HUFlptSlUpMxksXoual omUe4lP+jIoK1dtbmNL3KIOTU2tdeFHotzkAYLZZ5xWY9nWQD6PSpUOhu8eNaxIhF0luouemNhR/0 CJaOAC8A==; Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGV-007bZn-Jo for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:53 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-340c6cfc388so96776617b3.20 for ; Mon, 29 Aug 2022 14:25:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=VHFOj4kfUa3xGA3HxCxfqSH2fyPnkF//iB8ZL7PxHTQ=; b=XClgxH8bXaTLY5kgySHmNtvDCCVAB8mjOROUY69yCIkvhLsUcaeiTHEG390LZTJ0UJ f+bTq7v3mEXk1WZIYhpvHYfgX0MddKEDnFBtUXG7eI4mzGho1uyo+BRigftRuUZpe5ed yTJv9FwpqBMqwvSi+XVWkQi2sCAbfOGrPpu1lkrGFWjYvxB7buAqdmGdNV7OSfwNKssU SvorUO1mGFzFR5b/hO2rDfP9WSX1xgC98rCcdyQIaEqfPxTvpPYobfXt4iE+GIx2WNMy khAj4t26F+rdvp0rGETv3PRP0ykKrNZPVAZGmSPMWGLG1X6s1pkbIB5IUW2TBK/ZgFFb KwQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=VHFOj4kfUa3xGA3HxCxfqSH2fyPnkF//iB8ZL7PxHTQ=; b=erALhzML2DaSWUKAJO3mUqs6oldI79BXp+wgsdu3D28JdekXH4K/ibyxLTEtF9CLrV JxCxpcozf53gigjXdfIkTrRqep1qhjaluSHluqBPWYVblDTdLN+XMiXESF8n6QdDgAOp 2/x61bM5Ixy8u78WiWmEcx1RNT+q7q6qk6Cgzmb7A6bdGL1oFim8HBkFImsuYX65Z1zK 7nHjpj6tU6u3UOLsUvRpf8dnZW4+HRysmUxFSG1VNyICvFVwosu14uQZn0Tt0tJgeUEK 1r8ZhcLFkaE0sEVLowI2Nnr63lkWDHwfe4LIXykm7uk/yHN11TQ2BjTqpTQII2vmGIPY uIcQ== X-Gm-Message-State: ACgBeo2SfYOcyr70yRLFExnEYSaBWYKsnHbFAJfC4XZFtVySiK/iBvlU A1ZKPUaeafG4udLt5rmZ3VlI2xUzE70= X-Google-Smtp-Source: AA6agR6wyKOqZ82JQBWMvaTOKlqh8vry2Onq3V9mMTZsNJ+QDBtzKC1cVl90fnxR++4fsXM2QhKkAh66mSQ= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a0d:eac9:0:b0:33d:c025:8a74 with SMTP id t192-20020a0deac9000000b0033dc0258a74mr11722581ywe.430.1661808342911; Mon, 29 Aug 2022 14:25:42 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:07 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-5-surenb@google.com> Subject: [RFC PATCH 04/28] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222551_891201_C9B8145E X-CRM114-Status: GOOD ( 10.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Mon Aug 29 21:25:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 76A43ECAAD4 for ; Mon, 29 Aug 2022 22:01:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=jPuPKoiLpWRkyD+0Oo5IombXm6Hsf7jrvafD7jW2Lmg=; b=CM+U03WVsJx1Lgb1/fJogjzMsm RPHpUkJy5R3QnPfD1TneM9bJ/zUCFcBfO3IouIv9p2YkHJEwvqrwRD/tTbI9YXNAdp1jyagTT3gSh BhwQLZ7WKoikAGQlYh0SH1lBRPRNJJDY1SAInrV/FFHdujy6sQBW5hQzzGSEWj12v/z5OzU1y12lm 1fU6uzQ/rPAjCv25ph7uA3LRrObp+pm+HZARtHTH2/89x7sAS1xDaXMxLgI2XjD0oSIBmSfeDwGnP 7uiwjwNxeakHsMRqrLrySf3Vs+P6fPOcVMl/y7NOjxEarXKPS8SSe+jeOuMYewnteMNXw33EQJYsI hAgtlS/g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmnf-00CyJg-MN; Mon, 29 Aug 2022 22:00:08 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmV-00Cxu8-8d for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:58:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=N9bIms4Zgtt1w5mfZvOohRmDSgEYydYD4bEwHw/V5ow=; b=ENEk06kmk/1AroEMxXGbX3o9hJ ao+Url8E/cdqMOdfp9GbZjA06njEKwkeoOZu8eCmmfrDRrfe/L8kR+lKQOoakIeBhs0wv2ZtHAUix c1dus6iOKFlbuN+tlyqxWgyPsMsMchjqMyd2Pct/cPH7cjEHyCCV8aMHbqQLh9w393jnenyxd8jZ9 JuyejhZ4tMZO541+PwJu0MRl0K/JcAGcoM1MJ7owNLNK5ktU77iXjbjFYfS7I1A5tMPqRtRfioGlE LXZ4YEr3rDKg54e7ktDPlgJf0OJXxcahtrneF2AyVRyQ1zEJdckBnEWyFRtprIwQqs3uqOdhQwVuu M14AObTg==; Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGS-007bZj-04 for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:51 +0000 Received: by mail-pj1-x104a.google.com with SMTP id lw13-20020a17090b180d00b001fb2be14a3dso10088772pjb.8 for ; Mon, 29 Aug 2022 14:25:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=N9bIms4Zgtt1w5mfZvOohRmDSgEYydYD4bEwHw/V5ow=; b=CDYqj2724k+j3UZuEqBII6d0mYlErNKF+DONYkTH+E04Ngvdyx6CFcIzGI3JpE+Ocv IfFz52ELd/oHC5JbA4KJiqR41o93UU1oNs91Ejb3C2WUYMd8rJbvn4wGpH2TVPFpMmJT SgaAVJGTguR7jHY2klbLNlMZG17tv+P143dcZzCLTWGQf5RYnpB7E01bhfNNtgZSKJd4 8iRdR4KuZii8bgu+gGfhXIXfMfDbZfPU8zkTguml14FGrqbs//qWRwWzX5xJRRQfxW3M XOIZ22KxRY8zlSKN1mF3msZnGKAdWpIuRmlb/EMY5nhFhrrOHs7u2uNNMJtGec19XB/4 ScVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=N9bIms4Zgtt1w5mfZvOohRmDSgEYydYD4bEwHw/V5ow=; b=gdOxSp+BtKGnYgkVsJtUuA6q4piYOpitHulHLsx4+j4GRz/eBYgiP6ffsBiW6Zdpsq ykAcwU8KkMf7voKVP9GSBTgtP5JRtmXViQ16NLBYSuNdnL5wEMfdSGnae7VyLupPmtLK vSmizbCJLQQ8KVfeRpanfYLHCgoOHLBmCgH69E3W6wqvDO/4kkEsboKOuKfCNXNcShiz GEILpue/RSgRgcffqCuqgKlnDrwVBU1tsJeTuThN3oqc+GEKsjCPK/817tXfINksx7mT Bwysi98r2h1SpVQDb/ok2XYNMK1vD9sX5zciYD1L3DxVl2bJIwrl5CwK9SKAMDsosNdw TCEA== X-Gm-Message-State: ACgBeo3uK2jbxkeKZnOmMnqCiAcOW7BQx32WjI7tPgAT/6QkydIetkyp NP9RGrGJAMa3QTdzt6LjcLpwnTbB9FqlbTVBj4KXFKlSta1U1jcwnFGHo732rS/OTdmQ5hqHkWS UmFa5PHyhdhwdL4Xr+jZGiSo+Ks/YIMo= X-Google-Smtp-Source: AA6agR7o7Ix1gbIIukXoy+30trWr4dJQ0Q9jpP73SljSWj/tMusjSLFwSN2Bk21+x1fzs5P+BkoH3vuJaG0= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8a0b:0:b0:537:f5f1:fd91 with SMTP id m11-20020aa78a0b000000b00537f5f1fd91mr12449109pfa.41.1661808344609; Mon, 29 Aug 2022 14:25:44 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:08 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-6-surenb@google.com> Subject: [RFC PATCH 05/28] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222548_384792_C8B7FCA6 X-CRM114-Status: GOOD ( 18.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce a per-VMA rw_semaphore to be used during page fault handling instead of mmap_lock. Because there are cases when multiple VMAs need to be exclusively locked during VMA tree modifications, instead of the usual lock/unlock patter we mark a VMA as locked by taking per-VMA lock exclusively and setting vma->lock_seq to the current mm->lock_seq. When mmap_write_lock holder is done with all modifications and drops mmap_lock, it will increment mm->lock_seq, effectively unlocking all VMAs marked as locked. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 78 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 7 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 105 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7d322a979455..476bf936c5f0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -611,6 +611,83 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +static inline void vma_mark_locked(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline bool vma_read_trylock(struct vm_area_struct *vma) +{ + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result but it's not critical. + * False unlocked result is critical but is impossible because we + * modify and check vma->vm_lock_seq under vma->lock protection and + * mm->mm_lock_seq modification invalidates all existing locks. + */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_read_unlock(struct vm_area_struct *vma) +{ + up_read(&vma->lock); +} + +static inline void vma_assert_locked(struct vm_area_struct *vma) +{ + lockdep_assert_held(&vma->lock); + VM_BUG_ON_VMA(!rwsem_is_locked(&vma->lock), vma); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline void vma_mark_locked(struct vm_area_struct *vma) {} +static inline bool vma_read_trylock(struct vm_area_struct *vma) + { return false; } +static inline void vma_read_unlock(struct vm_area_struct *vma) {} +static inline void vma_assert_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -619,6 +696,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bed25ef7c994..6a03f59c1e78 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -486,6 +486,10 @@ struct vm_area_struct { struct mempolicy *vm_policy; /* NUMA policy for the VMA */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + struct rw_semaphore lock; + int vm_lock_seq; +#endif } __randomize_layout; struct kioctx_table; @@ -567,6 +571,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..a391ae226564 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_mark_unlocked_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_mark_unlocked_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_mark_unlocked_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_mark_unlocked_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 614872438393..bfab31ecd11e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -475,6 +475,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); new->vm_next = new->vm_prev = NULL; dup_anon_vma_name(orig, new); } @@ -1130,6 +1131,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + WRITE_ONCE(mm->mm_lock_seq, 0); +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index fbe7844d0912..8399f90d631c 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Mon Aug 29 21:25:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958504 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B884ECAAD4 for ; Mon, 29 Aug 2022 22:00:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=wTot8hiznpwhDSsm9psqS3PCNX7n8hbPIqVhWuS+lXY=; b=GPZf+X4w3FOTCj5WqRkSZp9s4i 7WTtxnA8ZOrQffvf2TG9DjT8qArkXIiguem9L7SDf5HFpFfrDWQhYn8M3f5yKqUpWj+FObVKWY/DY MxwDB9trxw96Kw7VCRs9LY+qfU2zBH+JYK3eVoplgDXggbSW8GeRxluxipTWaxLaDoWydSFYggZOL q87a+shv2jMk/1YAPIwF4+iouN/Xyy9ar+pwF1wdPIFqYzNTPYatIwJazn+ZrbTpTexOx/72xWNNS G7xWZjlyFZphVNE82VF9bMchJ/qRJACLgeSoL16gfhtVruYmsETe/vtbJxkcC9YS1XwYUQtPnYSOx TPLLSE8w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmk-00Cxzs-P8; Mon, 29 Aug 2022 21:59:10 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmU-00Cxu8-3r for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:58:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Ymgoo1txtx46eok3IPOZQ+7oQJBGfBlI8J01hhP7hOA=; b=YEH/blVE7aaaF39ywpEIxgMWlV HjIlfH0clj7tPAwHEc1Itj6JpxV4mxVk5I7v8IlL2fPja6LfSaIRuuVIGwW7eNSe4vGfi/v/2fxHE 5CoNS76TEZhUnGyMx/p44W2kTkvYrRy0AbWBpOejM7gUDnCCSFnNuHk5N1u7k/VRCitDQ0beuOCwB Cxosc9Y9850xc31skyeNEPATCYVkXhzvy+iQmfCsUqQE7HwR9RYMGZ9mlnqjX0mYlzHsozL4ux2J1 dh7LHfIdQezRQupFMm0Wa0W+Y/DT+Nq/JG43JvEGKYLtxnnJaJ2CozfZKmGyTfOVhVunzO/U7dSPS YLUglC5Q==; Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGW-007bZr-UK for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:55 +0000 Received: by mail-pf1-x449.google.com with SMTP id a21-20020a62bd15000000b005360da6b25aso3615895pff.23 for ; Mon, 29 Aug 2022 14:25:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=Ymgoo1txtx46eok3IPOZQ+7oQJBGfBlI8J01hhP7hOA=; b=V7GGgx774wbENy2sL3aG99hkTB/qGZUI2IMkW5Z7Ox/AY4k1IvJaJqWcS2+hJrQTXz qJiaho12WNhOPkKsDNI4kFEEE0MJklisQUNbnL2MLe3SXi7RR3OSC49CxUHm5rWNBWOA wHnBj0rB97rbw4NvbtXfeTLWG+Vyzon45DGcHKdD67SN4QP16ppc4Mfgkavx0/TZazKT jRNtTwjBlBNTzgM29WASwCLg6OtowkNwGUZwoCyX8zuUMvza2TeNlSBp6kFeN5K2wkE2 aGwm4Y8asgOKWmz3tiq46nHGojR+kREyp0h1B2grUgz1rUK/O35dZpO1+I8qz72cYmP5 /MNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=Ymgoo1txtx46eok3IPOZQ+7oQJBGfBlI8J01hhP7hOA=; b=187wsMuotuTgBH3J/zMsktAC0Dhl70EeJ58pkZuOQC56H7OV7tNCRcnw+d7Vpw6WYG tMigXCZELkpnS3TSDlGMjug5qe1VDVvZDNaCrn3ymYmvNC/7eWr6hNjYBkCOuK73Ayqc cXbm2a8/eW0NqJOM2y5iHVwh/pvqKpPl+kWV50cqvPuymUaV/OLWOcyJZTYUWtYKDaDI 8MwhU+ci+jvaFLN+SZZAQiEudH9gRvix2OzplSwpB7AHHN90FNz6NIRCMt7WHduWC6Pv yKE5B4iCZaoHskIsKNxKyMNC8S2oZu1zehN0ipPDcoMHQjcv47uitc5GhLH6EbJgWYnh R5Hw== X-Gm-Message-State: ACgBeo1zaCf8piy7yDwOn8JXhIFKZmuRmTzCnudk8W1pQUUE8vUSwVsY LrZuMU5ly0EKmVbNijjEtHFyzOgElSg= X-Google-Smtp-Source: AA6agR5xVu1GUsjULFDqf0en50jiRHsHcqxte0InGO36KTqnyV9YWBjqbQ+B0gx1566Kx/T+OnpxLJSWOq4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:d48a:b0:16f:c31:7005 with SMTP id c10-20020a170902d48a00b0016f0c317005mr18369268plg.173.1661808346364; Mon, 29 Aug 2022 14:25:46 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:09 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-7-surenb@google.com> Subject: [RFC PATCH 06/28] mm: mark VMA as locked whenever vma->vm_flags are modified From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222553_204570_41544254 X-CRM114-Status: GOOD ( 15.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org VMA flag modifications should be done under VMA lock to prevent concurrent page fault handling in that area. Signed-off-by: Suren Baghdasaryan --- fs/proc/task_mmu.c | 1 + fs/userfaultfd.c | 6 ++++++ mm/madvise.c | 1 + mm/mlock.c | 2 ++ mm/mmap.c | 1 + mm/mprotect.c | 1 + 6 files changed, 12 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 4e0023643f8b..ceffa5c2c650 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1285,6 +1285,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, for (vma = mm->mmap; vma; vma = vma->vm_next) { if (!(vma->vm_flags & VM_SOFTDIRTY)) continue; + vma_mark_locked(vma); vma->vm_flags &= ~VM_SOFTDIRTY; vma_set_page_prot(vma); } diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 175de70e3adf..fe557b3d1c07 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -620,6 +620,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, mmap_write_lock(mm); for (vma = mm->mmap; vma; vma = vma->vm_next) if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; } @@ -653,6 +654,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; return 0; @@ -734,6 +736,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, atomic_inc(&ctx->mmap_changing); } else { /* Drop uffd context if remap feature not enabled */ + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; } @@ -891,6 +894,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) vma = prev; else prev = vma; + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } @@ -1449,6 +1453,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx.ctx = ctx; @@ -1630,6 +1635,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; diff --git a/mm/madvise.c b/mm/madvise.c index 5f0f0948a50e..a173f0025abd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -181,6 +181,7 @@ static int madvise_update_vma(struct vm_area_struct *vma, /* * vm_flags is protected by the mmap_lock held in write mode. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; if (!vma->vm_file) { error = replace_anon_vma_name(vma, anon_name); diff --git a/mm/mlock.c b/mm/mlock.c index b14e929084cc..f62e1a4d05f2 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -380,6 +380,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; + vma_mark_locked(vma); WRITE_ONCE(vma->vm_flags, newflags); lru_add_drain(); @@ -456,6 +457,7 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ + vma_mark_locked(vma); vma->vm_flags = newflags; } else { mlock_vma_pages_range(vma, start, end, newflags); diff --git a/mm/mmap.c b/mm/mmap.c index 693e6776be39..f89c9b058105 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1818,6 +1818,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, out: perf_event_mmap(vma); + vma_mark_locked(vma); vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT); if (vm_flags & VM_LOCKED) { if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) || diff --git a/mm/mprotect.c b/mm/mprotect.c index bc6bddd156ca..df47fc21b0e4 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -621,6 +621,7 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ + vma_mark_locked(vma); vma->vm_flags = newflags; /* * We want to check manually if we can change individual PTEs writable From patchwork Mon Aug 29 21:25:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D037ECAAD4 for ; Mon, 29 Aug 2022 22:00:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=pBGwTwKIO/IkR5b/NLaGSuB7aYr+KQ5QqWGO8Xny1L0=; b=TMUenfntARkIjzR9a0sck9jkeL sqzyyZq/1fuVw/ch8p0LOrAsVFpt/VBxZhIOr9JtqPI88AAYMDoshb7TjQzKqMQ8P0GJ+CxBwrhy/ MVC48o9I1Hxi3BWsIHxXi4qcvS1RZ2HX7Cl8S8GM7e/D4g6ZyCVpUfVcYxE/9kmgGN599+Uj+Fids 2aAam84/NAB8ScOThsy0N2nQO7GHUFZaZFPNkZsw4qpknKWwvaXRIE8yYL4gl7RRgUJzyIJaFhHGs p4KET6QMEvhHEyupBHTdHtmGEPKIUbdALlKna/nHzO68C6q3ZpJzJTpkG7TAWjHnGZnIjk/lv6Y7q KtAZF/DA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmnR-00CyDK-HX; Mon, 29 Aug 2022 21:59:53 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmU-00CxtJ-Mr for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:58:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=knlkft8CSs4ctk+Wf5EGoFcT9IapAOL4+B6Ai/+T1Ew=; b=YorHznBi2Y1lOp3VsItYwavR2F +E4L73N+CennuH0f2+QvHRtit3+X+sbhn+Yow5zkkElM9fAiKH5+zlldea8g0VrhEJIdllOZwyEBQ d5UZiZPMCwOwjfWRLEdlgzpUQKwpMiHLLkpZc2oBrE+xfz2KjW1M6epGXeiXVhYPJVL2mkoYDv8BF fwgE/ep5k+BFWAP2LQKyZYHYQ9OvCpj1wTukRng7y5dB1/2D1yPB7PCuPVFZG1/0PgcI1nRq3UtKN QzzDQ+g7Eei1NOOb5U8XN9apnoEZLheDlL3h2pZV9HPx4SDM47iYcNFp+B0B+TPBjKHTIdZUk2ea/ 0d0NzJSA==; Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGV-007bad-FK for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:53 +0000 Received: by mail-pl1-x649.google.com with SMTP id t13-20020a170902e84d00b00174b03be629so3066065plg.16 for ; Mon, 29 Aug 2022 14:25:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=knlkft8CSs4ctk+Wf5EGoFcT9IapAOL4+B6Ai/+T1Ew=; b=P37RAGhPa0fj1eu5oWQ75AvAaLtugohQsYRWKzKsYPy82nTGbYS2huH/UiZNVvoE/s 88nyZ5hMCHsleX96EzwxSRzBq/7CUKc/bHbJ31Ts1GLCvmDOU9CmcQD2WQXDTk1c+h7j 3oIUBNbln4Bxomo0AKLbatKVQNVrzgPz+A4aQoGdDS5LbHfkkFmAcE6pV6uk/ymeBjbJ 1nc4oAeG1urj8/jUSameTDeH8ziLxBTZVHP9LQuagPlJAxjluMekWoC6NIdC1YJBWEbt nn2+QyaYYhDKNwmBnD8p146GgfwAnKteHIAkGMz5qwPGMxUQt1Cd0qMSAU4DAqNB0mVB aVgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=knlkft8CSs4ctk+Wf5EGoFcT9IapAOL4+B6Ai/+T1Ew=; b=sCX4oBUaEZfCqpRDDBH2r5R2U4ySBiZuNLI/N2VfmyzRAOQR8Y+qHEnfNaYH/CWgWc zwqLy3BELt2slfwAIRrwRCux05Ixpyty4G6zjlkXn5KVqsmt14mawPWsKP7QGtQCyy0u SA50WpfuGQbu8DQzEZZZk/U8FiYIbBFA8aQJwElCd8epF6Ui3tp3r6A/yXAGuoxrUBTN RpzkTEvWdpiJavQc0EufpE2G2juk9lZRDwCsnCxjYgUzI1vs29HQivxBuo637Z3HixTs Cre+XvCyOvQ714PHqoLsDZ81PnBDFQUkm8KoZXTTkUh7mEPbgOZCR8819S5WqRpuel/8 rwqw== X-Gm-Message-State: ACgBeo2Kr6x9ExIexf31ZvNiGUYAHEftoLI7ltV/vYNt8Kvi9XApJCUE q1Ub4WpUwJIkCfVdT8seVhdvxiAshAc= X-Google-Smtp-Source: AA6agR7ucJtBhSAnuB+PqV9qzppZFKeEkxBUzS6um9UDd2r/XYBs5DAVo1iLh4VL9E0CKz9mlSvmLd+h5RQ= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8d08:0:b0:538:105a:eb6a with SMTP id j8-20020aa78d08000000b00538105aeb6amr10112054pfe.42.1661808348114; Mon, 29 Aug 2022 14:25:48 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:10 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-8-surenb@google.com> Subject: [RFC PATCH 07/28] kernel/fork: mark VMAs as locked before copying pages during fork From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222551_689063_C34224DB X-CRM114-Status: GOOD ( 10.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Protect VMAs from concurrent page fault handler while performing copy_page_range for VMAs having VM_WIPEONFORK flag set. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/fork.c b/kernel/fork.c index bfab31ecd11e..1872ad549fed 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -709,8 +709,10 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, rb_parent = &tmp->vm_rb; mm->map_count++; - if (!(tmp->vm_flags & VM_WIPEONFORK)) + if (!(tmp->vm_flags & VM_WIPEONFORK)) { + vma_mark_locked(mpnt); retval = copy_page_range(tmp, mpnt); + } if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open(tmp); From patchwork Mon Aug 29 21:25:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CAE57ECAAD2 for ; Mon, 29 Aug 2022 22:00:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Uh2lQQ1tkDjh/3G8MTqTJSXgUrs57YrZXxNl8dsOT7s=; b=bTVdSdgf63fGf47xgpzPYAJoQ6 dQSr7t93lIpct+p124GZwGYZKz6m3qfeH+U7taW3Qw7pnOQij3q8+b8ZaHnj1stRP7a32G83mu/8c n95JsCJ5jDwtaKF14KKTES4ny2egqTesdUVUN1owv1iUE70T/CCuIONluyV+fuGuE1csiXW6BCHRt 0ceBud0InV+Sg5leesccBIWJ/U0MEvPT9eBqOeuQlXIBXXaDdpV6CiCGg8SCdVTftjW0p/FpAs+Pn BBPqvIWbiYOazHaQTQMLNPuYQiZFd+J5/QuNrbRvyrqZXHgbWIivk/eMlgwxGeJd2K/w/fuKWjH5z Uy17r1Bg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmz-00Cy24-Ph; Mon, 29 Aug 2022 21:59:25 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmU-00Cxud-Ei for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:58:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ooNZAin4dExMhxnzVFEqRWXW+87JR2sLE0fevcXKZcM=; b=C6Rm8LMXvxT3/opGXC4l10pLJY 9lKJ2e1q9Ma3eCcQkLKiuWFkVR6ya+0WhWnxVMu5m40h3qy2wgyyMhuXdZHQUKlJIiF4wOlUkNlTJ V6ohE/4WPrXfC2dfCCn5wVht9Qjbon66AZZZ5TSfRQ7RID4eppZ7guzD024EJhtij/fMopdk3ekf9 rT5JMMkHuaETJhIL50lJGLvlZn3v8q9RF7zM9fLTqDQovGnF99r+ELK+x8HxvQ+dLAvh6RfsSdqww +T1TNY/UYSfwjdWJw7QoDJrHxVRoC/q96XyYjC84VPv5Wfx/wGDonRy0z7Yyn5TAgML8sxfcNTOQ4 UWAVy9KQ==; Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGX-007bbD-Ji for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:55 +0000 Received: by mail-pl1-x64a.google.com with SMTP id m5-20020a170902f64500b0016d313f3ce7so6864493plg.23 for ; Mon, 29 Aug 2022 14:25:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=ooNZAin4dExMhxnzVFEqRWXW+87JR2sLE0fevcXKZcM=; b=I4AROHhc91xv1qcH3nCi9ZpRLfIZ0YYOknUuCx0UCVegrJDEOhp2xXx6sXe9aCLn1C FzO21jqONtvqpljlp8Pb4Y+CdSx9zRBCbsYcOU1CMA6tT/zAnvmRorbY0bmK2T02uMkI Atv+IFZLAVnvs+tVzbF09RqlClFEsd0HUKtTSkQ+YRbpf3d4jB4ZfuXvxVp5TLyaLLyx AxdLnxV0cUJcHwzUd+Lxky4P92oXj4Xhl4or9fJrlsG5GoIvzarsCtdrQ2b29IJKR80y NIBWQdZUPTZNVXZAtYrsTzYsNpAhtyZg3hqM9pNuOBs9nY5UdbnMfXBZJ7yLdA5TAp1d MxBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=ooNZAin4dExMhxnzVFEqRWXW+87JR2sLE0fevcXKZcM=; b=8B1LZ05rXWAt3OsZwwwG9Wsz6NPLPLzmHms3p22LlTFYzse/wNC4J7TjjThDnQERGl 1745S6Qe4d3JG/vQ5pE/KhcN9XTheKmzBNvpRZ+NRb541JU0gTQ27bOERRNGRW7vpMd6 cObbo6SEKXqrzIad4B0ruv+vLLLvXaJmpEWdMQFwlfgZSIHvBkUd2RqumV9prlbJpkED F4ONR95krPFCQeryuilxsMMCcAzZsmjadnLUM0Qyyac52YKLfB70eRjCgpNw1tS58K+d 4jMEeUukWI63Oe3qac7Ush/8HQpCk2oZ/gla5Jl5YeNM4EDZ1qEBYjOHSEKIzKoHqXIw OalQ== X-Gm-Message-State: ACgBeo2nzntEnk9UDeU1eF0WSSDeneLCOwY0+vqOsdX05/1NxK40ZQVa 6ZhcR6ujg1D+iuMrgIaQn3FEzQZZQfQ= X-Google-Smtp-Source: AA6agR6Ch+8H2P4ZiI/nyVX9BprKXIg4wSUP+mENwsodRVHrGORNiLr4bLOGgyZYxzbEnnQphTeE0ezRPcY= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:4607:b0:1fa:f794:2c0d with SMTP id w7-20020a17090a460700b001faf7942c0dmr20939608pjg.24.1661808349897; Mon, 29 Aug 2022 14:25:49 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:11 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-9-surenb@google.com> Subject: [RFC PATCH 08/28] mm/khugepaged: mark VMA as locked while collapsing a hugepage From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222553_913095_4C3C97CA X-CRM114-Status: UNSURE ( 9.20 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Protect VMA from concurrent page fault handler while modifying it in collapse_huge_page. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..030680633989 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1072,6 +1072,7 @@ static void collapse_huge_page(struct mm_struct *mm, if (mm_find_pmd(mm, address) != pmd) goto out_up_write; + vma_mark_locked(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, From patchwork Mon Aug 29 21:25:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958494 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4C8F5ECAAD2 for ; Mon, 29 Aug 2022 21:47:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=3fng3CEwhjgCwQhVW1zKVzj9Ex4cgXlsClWCDlLeAsk=; b=W+tD8ANP7PPUbxh2WH4tfzIhGR qE5VIdwP1lhpXnzTdQhg8Vrj8LmNOOOE+TDxpZFdg4I7qsPNaUgF1wPB8/HjAEY6zGJxf5rDokxEN pzA4gjSas5x/VhXhrdMstGqXnAJZhprmcvEtO1xITP2ys1gVqZxHqg9WIArRqLnNhMpgms4jSs0k2 85ervIFhXuw71axKuoV1Y+rjVcmRtl2NaxZTBnBtzgKKMK80hsePq4R54KizxyCsL1Jvvn3aM0oeL s7MdxNumbPI7uTqo50LPspwcpLA0M8kgXOmaEnG5qlqTaaVa7235eF9cvrXD+FEgr3jqGiP9MR51A 1iqS2YyA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSma3-00CuSC-Hj; Mon, 29 Aug 2022 21:46:03 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXr-00CtVm-3p for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=tMRmbcwOZDIFaTPxLY27tBFqvxpOSXiDeb2XTcL9iTc=; b=qCBK3X51VPxq3Pc14w/oAQI8Y9 A6s5CNKKAy+a0cJEPoY33V+wcbtRZARGJCqHe9vt5wfVOdfoQFl72+cS0UF0Lzs0S7rYLcnNnYYB/ InjaG/n/9BVvrOBc0T/dqze9DkTmDUAPjDs68ROkEeBYi3HSpZ8NRMv+Vw+p364ffixE7EzM1njfA yw+oi6oMhW1Cs7BlGB71DPnDfiVef1zbn4eP4HshAsRNlqQKJ7V0Eam5/wVxvxLwl77C79gu45HkY TfL6rWbOcLtR2hwYvttptGvIyHRR77G4cpnS52HuaCFaT/r4a72rNcOuLujgkqYkCnUB/3LQaly/L zUGD8wqA==; Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGb-003Prz-FP for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:59 +0000 Received: by mail-pl1-x64a.google.com with SMTP id z14-20020a170903018e00b00174fff57d17so978592plg.14 for ; Mon, 29 Aug 2022 14:25:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=tMRmbcwOZDIFaTPxLY27tBFqvxpOSXiDeb2XTcL9iTc=; b=c5Ge838GLG1Fn9WCiifxSW0yY2J286Z3nb7CHVsgQeKl819oavnaneDjs0pHM1pBgc 6/VXxA27srsSSYsH3q/5JQwqbG7gvhz/dxlqpfYYnChp/oSAOK1144eGZRxL5H4xfU4+ gXjPirwbsBHls6abdkbd/Ru5neoY+5POLJLE67tmQyRoTq3+UExyJwB4xZoCc8lFIPHT 1JLKkVoSr3hMYFs+x98zjjRcyw2xrS1rGfAXHEq+RyZfpGyhDpyGAbYsGbpBS3qHA7OY gUezPbUinXA9UrR8RhJyAbOsAhf2VqCoI/O3T8n1+HeqEuYCNoxrQ8z3jvR4TT/rnWyA atPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=tMRmbcwOZDIFaTPxLY27tBFqvxpOSXiDeb2XTcL9iTc=; b=TmDBLy1kgesTGFJopyo8N5YUa9Kvw5zWCnilXmprOgKLbtDW89djETo0CFbrLySNho ni2MW93k26/fwme970RxZZT9tvHzkmoJb+9WhfeLAdyy34ArRy3BYJj97ZW8Yzu4ChWK S2GhK2buNyc3nm9ooWogGGzDor3NBDCRmgxXEOheA/X+YObuWLrvC6FmyrmY5al6yuVr LqnY7sZYMdF4ZxFb16xN0yozJUXt8PRuaBVq8SsO+/0drrbKS2xqPsHs8d3XHmqomaPo gMa9yJFlqdtsh53rgAktf0wzb9nReT+VQ8nADGyvkZevZ6GmfSCzOQUFJo/9yywDcB/f 1fmA== X-Gm-Message-State: ACgBeo0oiapH6pmTrbDYK64ZR2hqBkOltfNmDXcZwzm2RxEVVilGo/3q OOJzEJBmKnFsMkTvZfDJrRbonpHovbk= X-Google-Smtp-Source: AA6agR6WFXWG6lBERn8fKgbng4pdY7+dFZ275Tti+4gL6MGpsob4z29fC0owAPSMM8vQ0mXhvHK4dQ+aJ1U= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:ab08:b0:173:320e:3316 with SMTP id ik8-20020a170902ab0800b00173320e3316mr17541232plb.38.1661808351535; Mon, 29 Aug 2022 14:25:51 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:12 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-10-surenb@google.com> Subject: [RFC PATCH 09/28] mm/mempolicy: mark VMA as locked when changing protection policy From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222557_537733_726894EA X-CRM114-Status: UNSURE ( 9.14 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Protect VMA from concurrent page fault handler while performing VMA protection policy changes. Signed-off-by: Suren Baghdasaryan --- mm/mempolicy.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index b73d3248d976..6be1e5c75556 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -383,8 +383,10 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) struct vm_area_struct *vma; mmap_write_lock(mm); - for (vma = mm->mmap; vma; vma = vma->vm_next) + for (vma = mm->mmap; vma; vma = vma->vm_next) { + vma_mark_locked(vma); mpol_rebind_policy(vma->vm_policy, new); + } mmap_write_unlock(mm); } @@ -632,6 +634,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, struct mmu_gather tlb; int nr_updated; + vma_mark_locked(vma); tlb_gather_mmu(&tlb, vma->vm_mm); nr_updated = change_protection(&tlb, vma, addr, end, PAGE_NONE, @@ -765,6 +768,7 @@ static int vma_replace_policy(struct vm_area_struct *vma, if (IS_ERR(new)) return PTR_ERR(new); + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->set_policy) { err = vma->vm_ops->set_policy(vma, new); if (err) From patchwork Mon Aug 29 21:25:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 904E2ECAAD2 for ; Mon, 29 Aug 2022 21:27:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=S5nEUqOqOrYnMWFI9rN7YiwR9Nvo2zfhcUmaS9Lq/R8=; b=WtsAFksn1NdCVFmSSLT1WG6yeV hbbM4qf7ZMmJQ5zYaev3chcr56O77vr1QCCK2jhNgI2R1Na5gN2xDXMsVYTJoq7x1wJca24efODw/ hUD7nYT3+JrbBeM8PmSDPay0vTtawHe3pqAyvdg9j9YXWE5Mz1ey5Db1C+W10Zde0znSMaTTa6tLT fDi0YOLge5ChuqUyRMJkYFvTA2eORDpFl/70x/rioi95VjHxY456nlM0EKNjFr2tXI/HQ8+vn8HYg 52jp/OfJZFqwEyRgq7YUlx0KAdVBtZAmsjFPwHZDLVVrEG9Bc8IlQl2yUFkOK9r7iaShgD7EpvaPD NMlsdjUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGr-00CnNT-2Z; Mon, 29 Aug 2022 21:26:13 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGa-00CnG4-As for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:25:59 +0000 Received: by mail-pj1-x1049.google.com with SMTP id rj3-20020a17090b3e8300b001fae0be4dc4so10090869pjb.7 for ; Mon, 29 Aug 2022 14:25:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=nWFFKiz2JrcFJbZNC6h3WaeWKSGRlWYuGKA4mg+JUNY=; b=PCjrqm9NhycPwB3kYWPhGmirug2EAINieO21R1tkNllFgsambVeodQV2SWE+SPKnLk B35opb3MQlODsxNphVo3gt2yyCr1QpWXQKLcx1/vKzcN/4STqp6cRFJdHuKeBk+NsfTT tDOTylgaSdZ5nEY1ymyrW4L9evpwJBVsw8WgcZFUheAteEM8kYWXWY1xc4VUjrsXgS7n TvpEAgLiNbXgMlGNPVj+MFOM++dtX+BoFzGnROjLhFZB70sHOrsgtuWSCJIJNO8R1S+2 S3QUIvJ5IyCicdMqNDVbeJ892T5I6yDVEmvFVjdkZe2DnjLJIY2H8E2Qt303ElEJo4Tc HSMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=nWFFKiz2JrcFJbZNC6h3WaeWKSGRlWYuGKA4mg+JUNY=; b=del2atK07FILrbnxtV+jOGY0+kyTUCPIaPFMM/WTYY2CXI7Roi7lbTbbZYM24BTG1i qcu/LzmmRMaLac8kD47GjHLW7mQ48N6wbscT3r3JU6YDxF15rOVo/XDtcQHxirvfUMPU PuNB+WyTo/JeHIcAlOjpe73Gf9hFA70IHznuKLferBEUm/VxsSO1GoAR8w4r04sviZdI ld6WXT/Ltz5Z2xcgfp1+XqJ+497+ph0JYAlHuvdsy7Ep+I9lGw/766Vn3sCH9OkoMIHA YBlThRDXrrCc/MkfAG+zHVMS+EMOS3QOQ4nwNc46S2wjOPHN4PxnIlMh2fiY9PLTLRc8 tB0A== X-Gm-Message-State: ACgBeo1ffDJZJEmf+QZywZQnSZV8N3kjy135a6sK/aj4x/du6HHO8mEv BwRNzUZP4y38Uq9dZqh7JI7AVZD22y4= X-Google-Smtp-Source: AA6agR4dglPLGzQSnoqHwxEZwo0Lk0EwUGiOTYsdKQuDkqIVlV54iwLYmhw7nxi8GDICS/TsbTVUTOyC/V4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:249:b0:1e0:a8a3:3c6c with SMTP id t9-20020a17090a024900b001e0a8a33c6cmr21107pje.0.1661808353487; Mon, 29 Aug 2022 14:25:53 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:13 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-11-surenb@google.com> Subject: [RFC PATCH 10/28] mm/mmap: mark VMAs as locked in vma_adjust From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_142556_428887_9AD37A38 X-CRM114-Status: GOOD ( 14.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org vma_adjust modifies a VMA and possibly its neighbors. Mark them as locked before making the modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index f89c9b058105..ed58cf0689b2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -710,6 +710,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, long adjust_next = 0; int remove_next = 0; + vma_mark_locked(vma); + if (next) + vma_mark_locked(next); + if (next && !insert) { struct vm_area_struct *exporter = NULL, *importer = NULL; @@ -754,8 +758,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * If next doesn't have anon_vma, import from vma after * next, if the vma overlaps with it. */ - if (remove_next == 2 && !next->anon_vma) + if (remove_next == 2 && !next->anon_vma) { exporter = next->vm_next; + if (exporter) + vma_mark_locked(exporter); + } } else if (end > next->vm_start) { /* @@ -931,6 +938,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * "vma->vm_next" gap must be updated. */ next = vma->vm_next; + if (next) + vma_mark_locked(next); } else { /* * For the scope of the comment "next" and From patchwork Mon Aug 29 21:25:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24422ECAAD2 for ; Mon, 29 Aug 2022 21:46:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=yfqGS6wuxG+DJPrAnwUCGNEYKuRAeLd4i7TKGb3QdrA=; b=YfCS+8oOlX3pZX6fEZrZqAj4A2 3CygefYA9tznmAuPsBzxOyAxxezPC3zmVa6o+HawfcEyZygKbpMXrv9uirE5vw1IG/tSVOIo4EsvZ GfuuWUBlgM8FO1B9Ec5FvhjiDFLHrkvSHKhX04tU96IEYZPyzJAf7EdmaOcvmE5+aJX4cqK2iYqgc I7hN+8vFN3d8lvW80UlnYIH53jDTgJI1+sMviImX4N1HqD7EDPMSLQAZImXFbULqe0tzeAu/LX/M6 Tbd9IZ2AK3joXkFpG6NxjIBvbdRoN7oe9vaX7uR44kvjqy2nZaNqo/eDfwzR476G1niicG9RFB5rc UyVXAvfg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmZe-00CuDC-TL; Mon, 29 Aug 2022 21:45:39 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXq-00CtVm-Ka for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=iuFA+HN3YQAhGF4lKe/pszwBtaIFpTgsToRqXaNjG7k=; b=Z/NZCzjOfU3wW31D7x4J55HQiV R/mOjcg+ODEz1H5nh7zY8bh0QQCAz/zmCnZxyY7/2BRF51v0m2m/BolfKyTXuPMKirfHoXxEAKGAm FuWmd297FbdidPX9bCHfrHMVp6u6nFtgDN5w2+QxdFrbWf7pQV26f/0E/BjTjfBVvkDzl3eRFlRKy 2O3DJ6FzB72vCuKycyh10nfMJdbnmTOUxsPYBHA5US66QNPI3e9OouCt/mqMobU1Lv+m7HPNNInUK BF7BPU0+rFy9TMu7eHmOVG/xYyO37ZQELCyAIbrEBGBbxeMkNlwEDRLOwfM/naK9Q+Trh1k4RSH7p F7j2KoGg==; Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGe-003Psg-B5 for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:02 +0000 Received: by mail-pj1-x1049.google.com with SMTP id k1-20020a17090a658100b001fb35f86ccdso10084838pjj.9 for ; Mon, 29 Aug 2022 14:25:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=iuFA+HN3YQAhGF4lKe/pszwBtaIFpTgsToRqXaNjG7k=; b=OQFmY5p6Pe2rtWh8HoOtibHP8P5GG7l2q0OSc7HNXnd2wSmQX9hqgXeo/WRhgSbde1 3GOMVRznNWUQpWpM6oRymYxlQgfSmhokZzxHMaOud6hAHMI7hRSvRQgnpTP2D9VCRijV tvYOdRbm6MoDu4UgOmxt7sejA6CQywedGjOTRBl30ijbrDgQETYn5oZOqnGCtaAYcPZu MjsJRXAG04ApiH5Z6mU8Nc2C+hXCaIEB6zH7hUa5Up7nSwAgqcd3I/44d6H2f140ISuN zqINoaSTwBSBF/Nj+3hF7XpzbCbRiSR9QvteChaGaDG9beNmWGZas127F5+VFT8rM/v5 R54A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=iuFA+HN3YQAhGF4lKe/pszwBtaIFpTgsToRqXaNjG7k=; b=6QXu8h8GTBKu6SjPPzSrUIoWYK16WjheJvs5klc96LClJIiDshjFpTv2z2LEekFAlZ tdxlHEXjd5ZoSwMztio8wkxbTBm+SnmrxqIaRkVRwy5TixSK4KSO6FN0uAgRh1D+cvqk HvRi4Amo13Dl/1YOHeQX5vIjU/bAhV9wBYCgq9kOahO/PsRScwq82BU2/S+HnUcEhf0w SyJ8zExXjWhPOkdTzmfxCQA6FxaqqriDUFXux/0qQomZx5Qb+XSJOJb5/Ed5/QvApdmZ 16wABBNZ4DsACqA0jlRR3qKfpPjtyvRvGHw4fQNkbQyWXKtH8ObWXYNzqrC0dzuhvBvk pdWw== X-Gm-Message-State: ACgBeo3eojrJyTsOy4WBGXbNmYN/bMkimpZ7GaLOQa5oxbdB9ploQBTr pGGxxcM8CFCy1x+2V+2aHdttepMOP0E= X-Google-Smtp-Source: AA6agR67llFT6eQ0+3Cyyc4Z2yCAV5KwAp4HusV+mlw9qyUeJ6m9H2aBfcEzix7gu6JpzsX42cg/TQ27QPw= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:e558:b0:1fb:c4b7:1a24 with SMTP id ei24-20020a17090ae55800b001fbc4b71a24mr21023pjb.1.1661808355887; Mon, 29 Aug 2022 14:25:55 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:14 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-12-surenb@google.com> Subject: [RFC PATCH 11/28] mm/mmap: mark VMAs as locked before merging or splitting them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222600_421282_A3E8D47D X-CRM114-Status: GOOD ( 11.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Decisions about whether VMAs can be merged or split must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Mark all VMAs which might be affected by a merge or split operation as locked before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index ed58cf0689b2..ade3909c89b4 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1147,10 +1147,17 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_mark_locked(prev); next = vma_next(mm, prev); area = next; - if (area && area->vm_end == end) /* cases 6, 7, 8 */ + if (area) + vma_mark_locked(area); + if (area && area->vm_end == end) { /* cases 6, 7, 8 */ next = next->vm_next; + if (next) + vma_mark_locked(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2687,6 +2694,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *new; int err; + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) From patchwork Mon Aug 29 21:25:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958482 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40B9CECAAD4 for ; Mon, 29 Aug 2022 21:44:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Me5CU4igeb0jqZ3WYf+NV2idBJPvBrUywBU5AnaWqiw=; b=C6AHv2995HYL3wien4AqUEpxGO bsQPAb75haEauXyfu9DW6WuVsLL0OBEm0KN2dfTqZXyILkUG/XDyE61MYvhiUYHB2JIBaK0t4/5aZ //r0AlblxnklhXh7RP7q2BV0VM3rXkQtcpZDBPRwPDS/xpH2loLe9g3i4WggvRtKYesPc8/YR+GK1 623Qsap65ajiRYzY9y6YeT3hZRUAIuRmiedo/cErtVDqKt+K918WGKsgPzflQHTVxLwPR2l4NJYAA HbBa7uwsN/gf3FdhpeL+kHUynczR7kOg6spX+qnvn3F4/doWCAK3Yj+3enSoRoueD5pM+Vh+bMu4Y FRwJ6AQg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXn-00CtWD-84; Mon, 29 Aug 2022 21:43:43 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXi-00CtVm-Vo for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:41 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=TWnM2nl+IWV2Xsj5xnZFMNMq0JvYeydCw4P9njL7O34=; b=AfxHNbTX/w7s96ZzlKd+jtyrzJ aKWebnETCStp3gJz2zxA/QxZoNb6vGfZVHqaPMs/lkHXRwAnnv7ICaKkUccUQHyB5qrvmMVkN2/Xt YcCFr8RaHmCiNpShgwIXLD4gSZr4ZaElQycBrY30aXNXvaNFX03W7oeeIA1FDhuIy2uETnMy3zBOh W1Sl32FAsKgPB/4sE2BOXeu3m35ne/BUkkxNvN7CgSrtii6R4OOIDnRiv0f90clkb8vbhL0gOPl59 JV93Ao+LM2kb5KVfrPmRZsKrbSFDYfZC/xbqCi52jk7EgIbxV3vFs8zeFlHRhNo/kSAAJSRcEF3U9 q0H0Gu1g==; Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGg-003Pss-Um for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:04 +0000 Received: by mail-pf1-x449.google.com with SMTP id o5-20020a056a001bc500b005382747a4e2so1503301pfw.19 for ; Mon, 29 Aug 2022 14:26:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=TWnM2nl+IWV2Xsj5xnZFMNMq0JvYeydCw4P9njL7O34=; b=QxyTYU3FERVCw2VziTt0AktgHLl//+1LB6ej8pZ2s1TWfQT3cvsAWfnuU7caTHeT9p cUZ3ty459UM3cwcXwNVqpzu7fZxDdvEiQ7oppDRkd/R2+Ilj86/8iy+0q3ZJXoZ4hnye zLhHEgIO/fbjIHEiSfoXxsUyX6O+Z8dic5XVruk2KEpor5fjgGmf4uNln1+WOsbYbTO8 1++i5QRJ3Dh2yxyVyOmY5Dm8Y1A3ze3Xt4uhd4NBeKcnHbK+ENDvs3AO4SaOBTp9Wi+f EGkkrDdrwisqruuTTVNnHMbKcZefZFlW3jnAC7mX9edrYOzqBxLpV3pgyWig/2c0mbAr sZlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=TWnM2nl+IWV2Xsj5xnZFMNMq0JvYeydCw4P9njL7O34=; b=h2m80EpFC75/t+4Bob2dnkSk+++whNS2CahpUXVXusoT+dIPuyI02XWJx6RWdsX//k bNOKwzmpq0/3hMzyML6LofbFPz3uNDzqQf4g07qXBfoAr+7io5lHHz6z9rgFWFFW+gHf yNL5HTAFTMcT5aTwg1yyfRIWIEmBUBQMmL55APlU+EE/64fZtgH5jXvkT0RYWybzbI+O 2gyGZqhziNEiq/rPdYh9E01UhVQWB/D7FTXS3+dtomuQdtLGMG/ZyLYaEx1DDKg99OlE TE39nRsH/dKLp3WOZW3ZhkWarYOq4KitcKGmHUcmW86c38QMjuMu2aFKyJ25TzDCvCSm IRrA== X-Gm-Message-State: ACgBeo14/T1k3FoXeECgSlqthJewF71WNYGra4TVM9bXidhA9l3Wibb4 02Dd6fTfhmcGpky0oIs3Yn0tyyU/ctE= X-Google-Smtp-Source: AA6agR76cA5Bw6pPdfILXZxKwAftwtVJj++YaWYUCk6VRMlDSa2HJ+OTsxxi3Jretb4HksKEr9YwTk9TbU4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:18b:b0:16e:f09c:919b with SMTP id z11-20020a170903018b00b0016ef09c919bmr18604991plg.135.1661808358173; Mon, 29 Aug 2022 14:25:58 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:15 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-13-surenb@google.com> Subject: [RFC PATCH 12/28] mm/mremap: mark VMA as locked while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222603_002061_6FAB1DCB X-CRM114-Status: UNSURE ( 9.58 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Mark VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ade3909c89b4..121544fd90de 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3248,6 +3248,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_mark_locked(new_vma); vma_link(mm, new_vma, prev, rb_link, rb_parent); *need_rmap_locks = false; } diff --git a/mm/mremap.c b/mm/mremap.c index b522cd0259a0..bdbf96254e43 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -620,6 +620,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_mark_locked(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Mon Aug 29 21:25:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958488 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8C908ECAAD2 for ; Mon, 29 Aug 2022 21:46:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=I/xsoq74vvoI/1PiMBvKycQmWmxgt91WH8EQoovswb0=; b=j6ssA/ke56WrW5dqAOZ4WlU433 lBuAUQAKNNANGIh9GSXh7/HvgLWPB4aL9bowrgH5luKVxGiM6WDOTpk3k8oLGlf86VaID9woLlech //dQaQPZKS/GRp94ofdWSuwsX8mKI4jrZeVOY3nUWsfett/RagNGK2q2Q/PKuLuUgubGyAoefVBbk cWoNMe4js7QpJyX/YdUzTyQA5IBcF8x2T3yPmAaL59/QiK2AsGc9av3sKqyU/KL63/gU3LEW3L3// /PZSlHWIZ6/KFjm8o1LKs66fRHKeUiv7fMb5jECHwFPXj8BEEK+ge4hMYMv6TWi9dyc+ITFEtFbRa vzrW2iIw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmZI-00Cu1T-E8; Mon, 29 Aug 2022 21:45:16 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXq-00CtVm-73 for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=i5dTvRyWQzPS3ycEVnnedNfB+iHCFx9RUaZKzBMQ1fM=; b=T4qZpRT3gwlLAlgR5C74M1solf jBdWfxUSXJytUZWPpVc3TlNVnwzvx2U6dJenB8HNtCKCxwFY/Ou/FtPkR4IiITh0855UYQ4lGqfLH V6AYUG7GNunLnr+ZrLFCBqalxLn/fUpge63DCI3pIJs3AvcEo+gPdEVoWbG+gcEie+/cQC32dvOXJ sxUNzEoXe1A2wMPZVOuJnl4cXEPsbeK42SFwNZW1GoafQU8iuR1yuuyNZSDSNkKmBQNCM6pKU4aKN qIooViI3Gtt6OeIEm2616xLp6uLTWXQwRZY+ubcLzFT8Y4bFiu1179yFr/9b0VOQSipUmb/pSy+Pb A3cEP80A==; Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGh-003Pti-EI for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:05 +0000 Received: by mail-pl1-x649.google.com with SMTP id i1-20020a170902cf0100b001730caeec78so6750287plg.7 for ; Mon, 29 Aug 2022 14:26:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=i5dTvRyWQzPS3ycEVnnedNfB+iHCFx9RUaZKzBMQ1fM=; b=OIPy78dmVL2e9cDtVRNWLABLgvKsSbZL8TpHXpGFOXD1iGGPPAnyGMcNCguJqgzcNF vE+9cNpICzTMp48ey6JmMRyIlEctkJFDoL9MrqQuHHLDpAc2aXZC9NHJQw5sThX00eJ4 cjI0k3jAy5gFEX5VEHv4cVfUWBvRtfOC3iZWstqSQy3ibplhtgaSEpWxPcs9EVcm88qg LMXuGuQRDlQwRuQ7YNFSCH+8XULwWBK9BYSC5nbMXIDpOIovOb1rbCvinLUjcMpntl6K B3Pgew8zey/TtmyxMcXnTuT0BjzA6VIn4rP9dbnKnx4PgoHjYUplxPVEMwf3o1trOT0a 21qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=i5dTvRyWQzPS3ycEVnnedNfB+iHCFx9RUaZKzBMQ1fM=; b=iJ0JdJV73VATLsgB1L0k6Im3sRPy7FaLXFGAGqkfPHwMWRdS7Tv5YYARCZqkNDAjp7 aaewklg+3FM4q+VGkFMCQWGk3NRO85Mnxta2gGqH7dl/0DygFVMkHCBnbT/6g2unWMLE VHO6sQYI6OySO7Z69s9o7ZYTS2zM4OWIEEsGRGOncMcuGubbcmwS+FVWC2pPNzUecA9N PRRvaDemqtG2MEBsXFuAJ+6GQuEJgQcvx0CNgeaVaCXt3Ey/YZk/ukOZlelTqQtninJk 39ngkGMmD/785K81BH/r/AHkj6p5mLzSFfdzE23Cge5Om0UCW96DSqyV9+CtlP7q0nWr 93Tg== X-Gm-Message-State: ACgBeo2pGrEXHJuu0nyAarkE3O+J3p/yN7efuJrkmYbE8bam9d+vtssL ohLm2CTU6IIbveZvzm2M45EVQHNVHv0= X-Google-Smtp-Source: AA6agR4q+ezvBgIOmovwk/KulEVJwqDq2XJXfw+za6wVj0A/SjlqYDTJGsWkQl5JJSuDpwxpjjn7kfBBqpA= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:76c5:b0:174:e586:f6e2 with SMTP id j5-20020a17090276c500b00174e586f6e2mr4373889plt.37.1661808359839; Mon, 29 Aug 2022 14:25:59 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:16 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-14-surenb@google.com> Subject: [RFC PATCH 13/28] mm: conditionally mark VMA as locked in free_pgtables and unmap_page_range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222603_526744_E1DDA3AB X-CRM114-Status: GOOD ( 15.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org free_pgtables and unmap_page_range functions can be called with mmap_lock held for write (e.g. in mmap_region), held for read (e.g in madvise_pageout) or not held at all (e.g in madvise_remove might drop mmap_lock before calling vfs_fallocate, which ends up calling unmap_page_range). Provide free_pgtables and unmap_page_range with additional argument indicating whether to mark the VMA as locked or not based on the usage. The parameter is set based on whether mmap_lock is held in write mode during the call. This ensures no change in behavior between mmap_lock and per-vma locks. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 +- mm/internal.h | 4 ++-- mm/memory.c | 32 +++++++++++++++++++++----------- mm/mmap.c | 17 +++++++++-------- mm/oom_kill.c | 3 ++- 5 files changed, 35 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 476bf936c5f0..dc72be923e5b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1874,7 +1874,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, void zap_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size); void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); struct mmu_notifier_range; diff --git a/mm/internal.h b/mm/internal.h index 785409805ed7..e6c0f999e0cb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -85,14 +85,14 @@ bool __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long floor, unsigned long ceiling); + unsigned long floor, unsigned long ceiling, bool lock_vma); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details); + struct zap_details *details, bool lock_vma); void page_cache_ra_order(struct readahead_control *, struct file_ra_state *, unsigned int order); diff --git a/mm/memory.c b/mm/memory.c index 4ba73f5aa8bb..9ac9944e8c62 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -403,7 +403,7 @@ void free_pgd_range(struct mmu_gather *tlb, } void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, - unsigned long floor, unsigned long ceiling) + unsigned long floor, unsigned long ceiling, bool lock_vma) { while (vma) { struct vm_area_struct *next = vma->vm_next; @@ -413,6 +413,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -427,6 +429,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, && !is_vm_hugetlb_page(next)) { vma = next; next = vma->vm_next; + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } @@ -1631,12 +1635,16 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb, void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details) + struct zap_details *details, + bool lock_vma) { pgd_t *pgd; unsigned long next; BUG_ON(addr >= end); + if (lock_vma) + vma_mark_locked(vma); + tlb_start_vma(tlb, vma); pgd = pgd_offset(vma->vm_mm, addr); do { @@ -1652,7 +1660,7 @@ void unmap_page_range(struct mmu_gather *tlb, static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, - struct zap_details *details) + struct zap_details *details, bool lock_vma) { unsigned long start = max(vma->vm_start, start_addr); unsigned long end; @@ -1691,7 +1699,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, i_mmap_unlock_write(vma->vm_file->f_mapping); } } else - unmap_page_range(tlb, vma, start, end, details); + unmap_page_range(tlb, vma, start, end, details, lock_vma); } } @@ -1715,7 +1723,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, - unsigned long end_addr) + unsigned long end_addr, bool lock_vma) { struct mmu_notifier_range range; struct zap_details details = { @@ -1728,7 +1736,8 @@ void unmap_vmas(struct mmu_gather *tlb, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) - unmap_single_vma(tlb, vma, start_addr, end_addr, &details); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details, + lock_vma); mmu_notifier_invalidate_range_end(&range); } @@ -1753,7 +1762,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < range.end; vma = vma->vm_next) - unmap_single_vma(&tlb, vma, start, range.end, NULL); + unmap_single_vma(&tlb, vma, start, range.end, NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1768,7 +1777,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, * The range must fit into one VMA. */ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, - unsigned long size, struct zap_details *details) + unsigned long size, struct zap_details *details, bool lock_vma) { struct mmu_notifier_range range; struct mmu_gather tlb; @@ -1779,7 +1788,7 @@ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long addr tlb_gather_mmu(&tlb, vma->vm_mm); update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); - unmap_single_vma(&tlb, vma, address, range.end, details); + unmap_single_vma(&tlb, vma, address, range.end, details, lock_vma); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1802,7 +1811,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, !(vma->vm_flags & VM_PFNMAP)) return; - zap_page_range_single(vma, address, size, NULL); + zap_page_range_single(vma, address, size, NULL, true); } EXPORT_SYMBOL_GPL(zap_vma_ptes); @@ -3483,7 +3492,8 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct zap_details *details) { - zap_page_range_single(vma, start_addr, end_addr - start_addr, details); + zap_page_range_single(vma, start_addr, end_addr - start_addr, details, + false); } static inline void unmap_mapping_range_tree(struct rb_root_cached *root, diff --git a/mm/mmap.c b/mm/mmap.c index 121544fd90de..094678b4434b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -79,7 +79,7 @@ core_param(ignore_rlimit_data, ignore_rlimit_data, bool, 0644); static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); static pgprot_t vm_pgprot_modify(pgprot_t oldprot, unsigned long vm_flags) { @@ -1866,7 +1866,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_file = NULL; /* Undo any partial mapping done by a device driver. */ - unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end); + unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end, true); if (vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); free_vma: @@ -2626,7 +2626,7 @@ static void remove_vma_list(struct mm_struct *mm, struct vm_area_struct *vma) */ static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { struct vm_area_struct *next = vma_next(mm, prev); struct mmu_gather tlb; @@ -2634,9 +2634,10 @@ static void unmap_region(struct mm_struct *mm, lru_add_drain(); tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); - unmap_vmas(&tlb, vma, start, end); + unmap_vmas(&tlb, vma, start, end, lock_vma); free_pgtables(&tlb, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + lock_vma); tlb_finish_mmu(&tlb); } @@ -2849,7 +2850,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len, if (downgrade) mmap_write_downgrade(mm); - unmap_region(mm, vma, prev, start, end); + unmap_region(mm, vma, prev, start, end, !downgrade); /* Fix up all other VM information */ remove_vma_list(mm, vma); @@ -3129,8 +3130,8 @@ void exit_mmap(struct mm_struct *mm) tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use -1 here to ensure all VMAs in the mm are unmapped */ - unmap_vmas(&tlb, vma, 0, -1); - free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); + unmap_vmas(&tlb, vma, 0, -1, true); + free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* Walk the list again, actually closing and freeing it. */ diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 3c6cf9e3cd66..6ffa7c511aa3 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -549,7 +549,8 @@ bool __oom_reap_task_mm(struct mm_struct *mm) ret = false; continue; } - unmap_page_range(&tlb, vma, range.start, range.end, NULL); + unmap_page_range(&tlb, vma, range.start, range.end, + NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } From patchwork Mon Aug 29 21:25:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6FC6EECAAD4 for ; Mon, 29 Aug 2022 21:46:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZHs+0gKmDstNvOxrddnRf0cyzmBFLmqqfI9FgJ96PoE=; b=e09zGfjiciYbfUqZgnpnKmP9px S9hLTrCry5lkI3ThrxrX/EST54+3jTf0En5EADybaGD72ywVdTpkiirRewITnalZp2zhTy3ClKAmw DP5TJszvm+dQdn+JUZh4e2gjUpwJi73GGQimzMYeEcpKrfcA9aZqTt7KDvH0RcO7AWC1PkfrkpFgM toUD6+t6WiT4hBhxAXwa+SKA/yvuLSqKmonV0ylwTUW1p63tIsZIKpR1xH/a42IV934+2vjZfnQ93 IRcYJfqXlMizuqYEldUaaaL+g3/+T7GZZUBcOPAYilo8Lmv523Awya5A5AzIcIbquO9cm7sAXyim/ OdWAGp8w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmZ0-00CttJ-Oh; Mon, 29 Aug 2022 21:44:59 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXp-00CtVm-Pm for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=J1u/YkZ7r4U6qP3j3HXey3MGQlv66sCtBUAUQwGtkmo=; b=HPFC5Yt9n1RsAQXWP6b13VWk7q g8PTDS+VI/cqopeEgAuEIwyE07vVOUuFqbbdzut+NQr2SH4Z1XGnGWqsPFVGToGLukD6cfrPaR0Pv 5pJZ2gjJjNVlYE87P6JgcbkOzS3gDJ5bx19IJ1mxpqQbtlgft3pWqCZbObiSg0pGimLxp216d0fc1 UGPhhI3St4hD/d/0yWRV5tda2HKMCM7FV17O9+Swol/nDY3lhQQJTzCPR3NH0AUDlQHW/A0pHrO3o BghRZuFjgll/a2Fesz4N6ztRinlnzd7o8r1rYce0eIsJBg3a8fOdrzf384bxZPuFdEqjfpSS4Yc5g b1/5lk4A==; Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGi-003Pud-Pe for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:06 +0000 Received: by mail-pj1-x1049.google.com with SMTP id a17-20020a17090abe1100b001fda49516e2so3659559pjs.2 for ; Mon, 29 Aug 2022 14:26:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=J1u/YkZ7r4U6qP3j3HXey3MGQlv66sCtBUAUQwGtkmo=; b=bssUgHF5w47sbr7l7KTrCUB9BNCgN6M5weE9wkVgXyWLQuTUzyGpg0OJvcyz0sW5UD jBcbuP4XXvP/J4pS6AqtYRXplk3ioZfRWOizdHfYubwqdaFV7iqu+wCAfN30wOXuSJR5 rOeSjLAnMTV/UMrEb+PillTsGibQBZHi92r8Zd7PRmmwTLk5upCdMt9/ApQU+Kr/zUNy g3HC4+2oFiondWWpmL/CQDHhjiEfoJFEyDpo5hDjW6J9yE1A4zjv6LFdAo5+gh9e4xsA 3B8ihRdAo9AsX2EZ1zty6Setu8LqB98WdsMEL3dnRqIhFnd/9nqnY8Jcva4fprwPuNko 5pyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=J1u/YkZ7r4U6qP3j3HXey3MGQlv66sCtBUAUQwGtkmo=; b=iIBUU4VvugZbnPLM8FMJv8E6euTNiFICMebgEPPXC8MYZKud1s7UyzEckUkwpJaKWN fIIItm4HCeSGOAPKd0rdryMm6Od81Hp5Xi7+xzMatPuCL3hgd4GKaCq1btEtzADBDzeZ nPtjQPeMJsEj3Dxmt3kooeRzdV91iQ/pmad32OSDDFzjN8+8dyR596/EUi9WKsrwrEYl 1BFckwrhfylOWhcPBtDoBFVUvwkqYN+63FV2DdX8EqhD4I+PQ01xL8GgMZPVL6eaZ2Nn z25UulnBNsUWJefEdJi+s7Y5k6cgtU9Tv2f2u5tEsDnTQIX76vAB/AT5052pIcT7xYka V4Sw== X-Gm-Message-State: ACgBeo0ac0peQ61jp/cb7cdAuxd21ixR2U9ksiLH1+eVgI9+D1N1nM74 twf5CVAeub9LWGh827wuVC1Tn0B1tm8= X-Google-Smtp-Source: AA6agR7FrxHuf3ebSGip6Dsrc2j7pCNXLJVmg+GJ9+F6P4LGjbGDMIiC+XxX5XFr/bvFDyLyzfk2y9GLnRc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:248:b0:172:7520:db07 with SMTP id j8-20020a170903024800b001727520db07mr18052271plh.76.1661808361566; Mon, 29 Aug 2022 14:26:01 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:17 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-15-surenb@google.com> Subject: [RFC PATCH 14/28] mm: mark VMAs as locked before isolating them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222604_884013_7B49F202 X-CRM114-Status: GOOD ( 11.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Mark VMAs as locked before isolating them and clear their tree node so that isolated VMAs are easily identifiable. In the later patches page fault handlers will try locking the found VMA and will check whether the VMA was isolated. Locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ mm/nommu.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 094678b4434b..b0d78bdc0de0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -421,12 +421,14 @@ static inline void vma_rb_insert(struct vm_area_struct *vma, static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *root) { + vma_mark_locked(vma); /* * Note rb_erase_augmented is a fairly large inline function, * so make sure we instantiate it only once with our desired * augmented rbtree callbacks. */ rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks); + RB_CLEAR_NODE(&vma->vm_rb); } static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma, diff --git a/mm/nommu.c b/mm/nommu.c index e819cbc21b39..ff9933e57501 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -622,6 +622,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma) struct mm_struct *mm = vma->vm_mm; struct task_struct *curr = current; + vma_mark_locked(vma); mm->map_count--; for (i = 0; i < VMACACHE_SIZE; i++) { /* if the vma is cached, invalidate the entire cache */ @@ -644,6 +645,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma) /* remove from the MM's tree and list */ rb_erase(&vma->vm_rb, &mm->mm_rb); + RB_CLEAR_NODE(&vma->vm_rb); __vma_unlink_list(mm, vma); } From patchwork Mon Aug 29 21:25:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958486 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 97ABFECAAD8 for ; Mon, 29 Aug 2022 21:45:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=fkO/GIwj7fb46uh2XEDSZqeGOrl3vR+tuHKWNCwxQFo=; b=PIQ5RBtGmag12e434cIo5Ps4qX p2RLqW8hR7LigH9Y8JlU8yYiJURDXdewc6G5WU9iCJHdrF50mqcLEZbvYnEK9njrKbXq/PdlPziTo YEqqMcmPepcVdLAPRlXpqMz0W5oT8iFH/8PBRpHCA8EZcfe/xAW1c1YZ7bLrSAGZ5bJVh3AwX16uP GwEKRg0w75fh7LNXyNwkUdKMYh1CeVk5HXR10FW9HOaLHQfjU2+rfBXKAiq2hcYXX+cl4uT4ovTi/ ZRWSh61Q8DnJ2CoomruzCatkj4tmIFBorxucPfqoJryn/LN5vEDgsTnESmpcsGWFdgEhQIxUAlydy E18xQUrg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmYj-00CtlM-4B; Mon, 29 Aug 2022 21:44:41 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXo-00CtVm-Kl for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=KQbM9RENeAKpMamE0B+V4MUsqhzbhGXDlXXjmQQUA6A=; b=d7KijWM/lMaZs+HUsAr+uaQ+oi NUqxuTVeZe8DWCK1u7/6iH3lHTdZz0162tvB+vnZly6W+mENuYr+jmmxICEpI8fqvpd2kN7XlwuLa mAMx1Pdy4DmsAlVF0rHrO9x3ccO8eI+TrhnJlhbVZZMERTzOAOKQ/feHGyRcir4fQqTnA8aia1VM0 Vopd+QlwuRv5TTd+2RvPOU8yLfqieytUqLxZRCYe8AqhMSfESZDCf3PkP0wmn2/tTl49E5fo7fC+7 H3Vc8ms4WBGbPjhKVXYnB/8jmMqLlQB9zUg8sTMLYzA/N6YuI4+d53ASYfcLSmOtYqdbrGpg0Q3Bh 9iRj3M7Q==; Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGm-003Pw6-3J for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:11 +0000 Received: by mail-pg1-x549.google.com with SMTP id k16-20020a635a50000000b0042986056df6so4543342pgm.2 for ; Mon, 29 Aug 2022 14:26:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=KQbM9RENeAKpMamE0B+V4MUsqhzbhGXDlXXjmQQUA6A=; b=B9JYMkX5xydBJOgcGzuYw+brQarCWmRWS+0/Aqinb/OY26MbR1xT5axtOfpYrKlmA6 qGVi7LP6LaJMDcQJmEECw3J6IqaSMiez5M7VUbq2lZEZKkVSQHjZtuqy9g7s10YpHljB L9LJY/vXV43u8aIw7/j+LcG51+Bdx5jgIt7f2hLczg1JgYzlwTyYBe/lmm+nmwMuVZWc thodE+2BI084oTXFC0MD73T0ggmHuAL+ik6FuitgxrIkPY+yM+/axTrbJYGoWGabXb1/ atpqGKZe5bOiVGcNY/fR8gQx2/9qiwRjzQjgpRwLojxFdTcGDVT+d94c/I/T/XPhcGEF rEbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=KQbM9RENeAKpMamE0B+V4MUsqhzbhGXDlXXjmQQUA6A=; b=vcIUCs8o0G1hqbunCxL98uFg2DB6ZBsyWYg5OYtV6K4KyTG3fY2d3jIDbGrX/2QvCT RcehT2B88ciwSUAOjtvOR5EnRK9FyihjocRDWkvusqA/MUxCWzaODWvud3PRvWnH6Rv1 7YdFvcOba+vFX0f8W9wrmYA9mhy7UmI4hs6kZKIu0oVjX9QmwgyyF4CyEaWzVAdSaD6M 6kSOs2cClnjz2VidSsWLMr6L+bzJ/hYeaIhNJXDKD0oMm/a+Rr9EwADh7HwUyK4oHQiv E/NHtcc3yh1Ru6Esq6jFgaaE8SsxscSHkVjwvC9Jtt7UGIH1h7s36hsFcLFnfjXqLVK/ xD1g== X-Gm-Message-State: ACgBeo3NlOeJLmPCFkxYwOCFM26adxs5b5RynwHfYt8s3kYTTo3GFVNy c0XGzgluN8ZYQeJ7J9JbhVLH+D7yphM= X-Google-Smtp-Source: AA6agR7f4s+E2TUUdzlInwBIWNCTTjaS2dHpmImL6O3kfzXJ6rCAm5H+xn8nFf0AEemGeSJLYPdhv1d/+n8= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a05:6a00:b8a:b0:537:f81:203a with SMTP id g10-20020a056a000b8a00b005370f81203amr18341111pfj.80.1661808363176; Mon, 29 Aug 2022 14:26:03 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:18 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-16-surenb@google.com> Subject: [RFC PATCH 15/28] mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222608_156158_C708421E X-CRM114-Status: UNSURE ( 9.35 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases mark adjacent VMAs as locked to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index b0d78bdc0de0..b31cc97c2803 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2680,10 +2680,14 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma, * VM_GROWSUP VMA. Such VMAs can change their size under * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ - if (vma && (vma->vm_flags & VM_GROWSDOWN)) + if (vma && (vma->vm_flags & VM_GROWSDOWN)) { + vma_mark_locked(vma); return false; - if (prev && (prev->vm_flags & VM_GROWSUP)) + } + if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_mark_locked(prev); return false; + } return true; } From patchwork Mon Aug 29 21:25:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 975BEECAAD2 for ; Mon, 29 Aug 2022 21:45:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=4KotQx0+/6p1aeN0sXJ54jMLkR45817Xx3OhPwKjxRQ=; b=FV/kcufsShr/Pnr7BFmpK0ujHG 1c20i5y1uJrUtlr3+7mEPfMjOA37eDgkTor9z8h09bfWqHUx0R9kOHNqciZqtloV4LlWzN2tiFqMO fMVX4FjmpVooxuXLg+02Kiw/Ye6D1+rWWTG7PTgM9TRKqanop2pYL36DtSqmN7T4WeIEczJMIBlNS tNl16CyKdo6PD1diCkmoirKtKb9O/m5AgG2ooZ2SV3G6FOTmVj0XZUm9W+Q1GTE/RtJ1kNmcpvIT7 ei1JbqtTJYayXrbsJAJvBu9gWX/yfeWMT/YBiNFyFBrXxrhIC+qCVU/Zyec09tn/rYJYoW1ZwdIq0 DGlZYPfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmYQ-00CtdY-44; Mon, 29 Aug 2022 21:44:22 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXo-00CtVm-99 for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=NYe8oOJNfY11QXfRhGHbzq3PtP5qYF5OBWlNRgI/ICw=; b=Oh/vxzBHEf+tanbTnfwve01QGL YRjqXwaHRvhEvEcDwNyoxPWkWzMngm4ZzINBsXVjZzLY93rnuEr+JTo+u1ZAn7PFggfgD/6hlwUJU 1UE5IVVcNXEtLwRQ6fEN6H8Psj4NfRpoQzQ7JEBBAokn+DA+VAAwyJPmzm4yT49UKx5QBf/emvKQJ gVYnDum9FETBb6GX4c+2DUbKlnbk7q9BrKPQ7HdBjyI40I8w2UR64+IY382XqB4VleNdSSGbDpfzy u34zhv8CArG+nrBzPOXTwWEWha2ze0l85uaMGr9AQ+xJPuo+midxAzDqCGvduOF87cnfto12uatkC vYW3PhXg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGn-003PwO-Bl for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:11 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33dc888dc62so147197017b3.4 for ; Mon, 29 Aug 2022 14:26:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=NYe8oOJNfY11QXfRhGHbzq3PtP5qYF5OBWlNRgI/ICw=; b=gO3BAu2tPnt5ajc2wonv12EyC/cky1iEagPrWs8TIsmrfOPEdnZICXVlt39nBHt3WW tTV55lawhiZGMRBmA7gdT3YGawg8JjnJFEe5+7PrxS+L7PuBHZcRqu/eGZS5+7HkD+/v WBe3l/+DgKFH8adTgdOoThkCyU3oUkPdhTm9bGePyXaQUn5Kf04YxKMvEOmZ7WCBWmLN BXoUJ7D1rnsd47mFF47xSWH/cS8u1UFuDHVuHAEZYgUQ7/tjgR1JcWJgnN0k47FfJT+A /f4moE4fkIQHN9Nb1qOXqxym1C/DCmqaA6CVw+YBif80n7h7hRdtT5oL02KCkdJe7ned iBKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=NYe8oOJNfY11QXfRhGHbzq3PtP5qYF5OBWlNRgI/ICw=; b=5BIJcI39kHK9UhYU8Cg+O5BjzPnuU+1VRcfh6rzG+HO6fm/1fGUQq9O/NmSpMB1zM4 +xAPxRS5O5huJEkRgfz740yBk3K+EahNSNktIdv/ZH21cgYz5pnAfhRRaGRUp8OnXCsg jIOoWkiUmNtqwJ8fm6ztqMm45JHjExBCimJTTJYFOGI+UIvHKgkNV8xv8nRmWDG1ytzH b6dHzo7Kl4vfK/c4SL+8lZ3YR7V+/RJvAidJcR+Ce6SceEvnycUSBYDxqPwqEBgNhrWE FDBwovU5hp/b1Ru9dd3O/xr3PZ4qUqGQncd+ohi27lBSsRbLTox6jyUDzkXCJ998kSIs UsCA== X-Gm-Message-State: ACgBeo0VaTnfiGlKIJ0i/UdTzn8nnNQGco/3Z+8PfJZnVMxJp3vOQO8R xy7Zb5HKElxPK+Bt7arMKSMnOjaMq4c= X-Google-Smtp-Source: AA6agR7nmkBRROpS79vTx3reOak4mrTZffxlmMbexsEub45N/wO5CnqJNjif+IJpyxwlCbKERnK2QUttv8k= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a25:d944:0:b0:695:6056:7dda with SMTP id q65-20020a25d944000000b0069560567ddamr9840482ybg.277.1661808364958; Mon, 29 Aug 2022 14:26:04 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:19 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-17-surenb@google.com> Subject: [RFC PATCH 16/28] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222609_481698_EB02C319 X-CRM114-Status: GOOD ( 10.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dc72be923e5b..0d9c1563c354 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -676,6 +676,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_assert_no_reader(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), + vma); +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -685,6 +692,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) {} static inline void vma_assert_locked(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) {} +static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/kernel/fork.c b/kernel/fork.c index 1872ad549fed..b443ba3a247a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -487,6 +487,8 @@ static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + /* The vma should either have no lock holders or be write-locked. */ + vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } #endif From patchwork Mon Aug 29 21:25:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E80C6ECAAD2 for ; Mon, 29 Aug 2022 21:34:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+6U1SleA2rbKqXXTgATHEsnB5RjryxIVtQ6WgFNcV54=; b=GFnNShRedJpqGC/nt0v22NrvXI pAC42jvuV/tlif4sgzz8nT3du5L4YveyBuABhGUdoLxdCihXPqooearBgBZbVl0yIrcMePyrMMkaM Rb8d+tITCbOWf9IRISVKWTcmQeiAyMU7exR7y5dGwY2QnrOJ3TdMEXq3SvEHFLaVQ2k7bOruRlytK inVzQ53z4DdLAJDRXe7dfm3EWGqQZmLFQ910QPMIcAz8qbdY9P7mCptn734PF98zQtfSr1uLO7Bhn Z5m4wFWg7vRQGthvYegITjPhJB2qBhLy+1Vx/0nHa88qByjtaAgqPzbSpxhdOGVNOmulwXuWcGl6V INxh9A1g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmNf-00Cq1T-U4; Mon, 29 Aug 2022 21:33:16 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmMy-00CplG-9j for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:32:33 +0000 Received: by mail-pj1-x1049.google.com with SMTP id a17-20020a17090abe1100b001fda49516e2so3666484pjs.2 for ; Mon, 29 Aug 2022 14:32:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=+ZeFgC+onLi5H2Gk/r+wfB2lUIAEpTgiMaISz2j8Yh8=; b=sfdQ+2NyYwxA7WE4W+U26tjnRsGpTwH+W4yCMsULIhDPYOMjHM3COZzPg/S6KTqrLL mCgPnd6W8rrgwvL37XiGeYHG8ArgrFQG0UGf2gLsolv/ME8hJmTyRg7Fg1LBPlsPaNI3 te4whtYEHF8c2GeRvQHa/rDrJjk62KSTUlVEPYqGvjKgQGKn8f4Dp8C4KiASRefP1Qh1 bQmVBjd8KI32N8+vGT1M5VRcf57cOjajmNaLyJUUpin1ccvlQLSGjkUAJOTaDxWOJrJ5 Pk3cN7LOZgsgUwKnowGjucplCuekHbu//K343iXFhb3F2AjszKhuURdvTUr7Fsy7vLjJ fKug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=+ZeFgC+onLi5H2Gk/r+wfB2lUIAEpTgiMaISz2j8Yh8=; b=pFjwUlww4F7maIXd1EDyyftHSjE72l/FXisdy8Hc6yK6LSwk4UyLlp7h8Xq6gBiPma wR6n2UqD7l8Y//5lIb1mopWbbJIzB8C7zM9Rajn+su2Gk+8gT3hjhq8JhDrlOkG2lyg+ A4lr6Gygf5gKG/EkT90oG1F7V7zBzmkVMllbqhoG6/E3GAhZEX0D0IVhHHnNqknwcRO0 OSgnjGFr19bFCiW6udvrS4wCDlSyd4jqVn6Su6feCByzEyIZtIgTJJUAQtHNa+BJJE5n eVafQXF5LVoErwtAh5g/R9mF2BewTSsLTvRvmf4V1czXGT4SEN+HNuwxStUvBI4/kxzd E1hw== X-Gm-Message-State: ACgBeo2V0Z9M4HOVmjL8UsI8SfbZtgNVmy/PIRd4TM7XJFestcv2i+yc HmhgPn4HZ7X3hO51cNwIzvdOwM0N3N8= X-Google-Smtp-Source: AA6agR60gqv1XJF9S7yhKpnItHkmCdUjQh3CpTVXyWE3xiDHKhNiTBjmbY9F8YGX1B9vpUO9txTmTcIisVc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:6b88:b0:173:c1:691b with SMTP id p8-20020a1709026b8800b0017300c1691bmr18372367plk.18.1661808366708; Mon, 29 Aug 2022 14:26:06 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:20 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-18-surenb@google.com> Subject: [RFC PATCH 17/28] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_143232_373876_1434FD37 X-CRM114-Status: GOOD ( 10.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Pagefault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to mark all VMAs as locked and prevent this race with fault handlers that would hold VMA locks. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b31cc97c2803..1edfcd384f5e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3538,6 +3538,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * hugetlb mapping); * - all i_mmap_rwsem locks; * - all anon_vma->rwseml + * - all vmas marked locked * * We can take all locks within these types randomly because the VM code * doesn't nest them and we protected from parallel mm_take_all_locks() by @@ -3579,6 +3580,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_lock_anon_vma(mm, avc->anon_vma); + vma_mark_locked(vma); } return 0; @@ -3636,6 +3638,7 @@ void mm_drop_all_locks(struct mm_struct *mm) mmap_assert_write_locked(mm); BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); + vma_mark_unlocked_all(mm); for (vma = mm->mmap; vma; vma = vma->vm_next) { if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) From patchwork Mon Aug 29 21:25:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EBEC8ECAAD2 for ; Mon, 29 Aug 2022 21:45:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=BujtATRk2caWB8j0EXc5yoJvFGA1RJAdla+qEUe5fUE=; b=TNGwhg/KaLHNynW/+VN6ScQKsp NgiLQUCoeBahT4DxP91Nuyw/YAl76swwer7oaGczXyktj04sE3jyfHVWl255PlO3a7Sd+t8JBQ858 lJUU7gbwODIMUUD2Mt671QqZDSkdH7F8PIsq082QUuRaflMmwysDln5qXmw2+3bYvQ5RX8Vm9eqNR 2J8TGcdCwl/qHCPRsg/kS/6gXvnt4neQjqsw/gca2KnMSty4pmI4oi8Axlm2LCt06H5olPvENC9Bt eYpNCQ5d+nPYR278utEt3uPzxzzIjsvirInM/WkTmQdH0p44zc80UAxIZzDAdAjw8CfXyD7ddm7+Z xMTsYq3w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmYF-00Ctas-7f; Mon, 29 Aug 2022 21:44:11 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXn-00CtVm-TP for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=MrRcKlTR7zCSrJMQg2Z93mfggQWMfDwWeAi691n34gQ=; b=cFP3oaGPnS7KOOrI/ZjVS0C+hx iJvDyLgx/mN7pTMXRwcj3Yotw3AjhYY7QiF8WHOqqSEYYTXwJem90mYSinNIbc65/hC61+/97orYY 2qGBwY4sleyq8l2MrKDQDgTzNg0JXaOnz/6fhu4Psufoi52KzZP65CeqmbN7dd64306M3eK6EFP1A S1VlJkOkBN1KqitKnDaZviF/BcWKAtQ7i6UT3/AuYxZGlouu6wPqE5rIfTx9h8EDLf8oykcdqXWIh tK1E8NPQUySJFii5tKYPtC9NorYtLPeoiesS/IoDovVI+kFwN5ij+hkCsSwRLW0onG9eW6mspSCre hgeLKr4A==; Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGo-003Pud-1s for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:13 +0000 Received: by mail-pj1-x1049.google.com with SMTP id a17-20020a17090abe1100b001fda49516e2so3659683pjs.2 for ; Mon, 29 Aug 2022 14:26:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=MrRcKlTR7zCSrJMQg2Z93mfggQWMfDwWeAi691n34gQ=; b=cVg/tOxX8sxsAXVi/E7ByEF0x14KVPhZxGLvuX25A7I6HfsUpP9Yw5fhBx8ozeM60j nuDrEi09QA9+LTKC8GajK7CQiyi7gN9srLPx+SGJoGK9+OesuSLiR+9oEBujQJz23p/X P3t9ZdRyZDRuaF1wtpM+xaufjpbccFTUx81s4naxMNNB/ygFAADiWbtKwBK4coZj6Khw l+lkYHWl+iss5qkPjBxcVIZuDHxj8lggM4wnCJgOdoiaVpSSSGOBOp9823YRpnW7UBz5 1Wlgpr7ZzU0IbMK3CpPADXeW2X4v/4Fep7gHMya9NlwRNw5kCRaApBpvKmaw4X2L8jbG 4C6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=MrRcKlTR7zCSrJMQg2Z93mfggQWMfDwWeAi691n34gQ=; b=aLgyq27aAqqMq8mWosXiVspeMHeOQcgEhFXX3XCZOAyDWCxwo03TSuUx9pUUM70wPp v23IExlhJDic2XIpJ6f/0BUmpH/1avuB8SWd/X09evFL2FPfMre2x//3BqArYcQ+Enjs 6o8u0cQ8oDV4hYFBag2GfKlstIh1RXrMSKvS6OnCvzMZpSOsf+p6McOtNNZWDHgGiECJ +VpemyPvXUjch2o0XTevRHtQNPnseQf0NwL36QaX66SrRhRDmvmrx5bqfqJWz9EKGjeN qCo8SY4n23IAaD/MXL7wc5FhD1GcpxyrrXv2oC0gmZ4MZEp4klOgBTw7k6r/h37Nd6V/ C3hA== X-Gm-Message-State: ACgBeo0if6agPfz6ymU8ZaPBqrMJz+PN5bH4n9i9nC+F7GOmXrjqjqoL Lgo+tBF1cJLqllhdOswNlomrEH7i1JM= X-Google-Smtp-Source: AA6agR6U/5CcVqFo3qhw78HCsOiNZ4QUVwqc0LtUdoOd4ifUOM6ozQmoAsAgpNZOBFg7UV9Y6fDIpWBGaXE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8742:0:b0:537:ee75:601a with SMTP id g2-20020aa78742000000b00537ee75601amr12578569pfo.37.1661808368234; Mon, 29 Aug 2022 14:26:08 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:21 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-19-surenb@google.com> Subject: [RFC PATCH 18/28] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222610_113904_311AB66D X-CRM114-Status: GOOD ( 10.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0d9c1563c354..7c3190eaabd7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -466,7 +466,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6a03f59c1e78..36562e702baf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -886,6 +886,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Mon Aug 29 21:25:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958502 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F534ECAAD4 for ; Mon, 29 Aug 2022 22:00:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=n+KhZoE1F3wLhd3EGzfCHLgEf2F4w7+Ts2+u7n/57ws=; b=Xj+RW2jrwj+GKI61GdT+0cnWND nODtjevrZd58NQnDlAAQZ31WeBfuNv2LC75czefM293x+phebDnidK2leuLzzdY5FL2cqOP8h8d4n fly/9vL1jDJZN9ctjBLUdPDT7RImuCyI7hvzTZiLpyyiNAVWd2T0CGz3wlyDd3mgtiA/LSMT8LWmf Trkbbij/FAYPW/1D8Pe7ZaYj9Q0Av5VVrV8RKUHp3ojTTu9J0+Pv/5NSMyol0T9MLQ9C8h+btW7QC PV4A2UTdzzlUavYxhKqk82bbAVFL9QPz7Uz/kZ4J/BG7psK1hHa9x4ZtUobI6ZyDGhvMf3Ii2U4o5 K4igSVtw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmc-00Cxyh-1E; Mon, 29 Aug 2022 21:59:02 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmQ-00Cxru-RM for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:58:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=K93TFWAb7JwQdWCvuFbBF/NwLgqzppA/TxAstG/QPvU=; b=eD7o2Aa+4DDmcGXyhMvQsStfCf qWoFgUtYZDQYh8o9vZE7WuMgs0XHuzH0gl0+rbeI+cWlx5pOtNFdj1N2NY6cXqeNuMbdUNlrCaNmh k1VAzeewQXe6j1HJVCdgyd+Ekw/aheUe4cvQIjwhVn4/pUR108m8FSTOs/RBx4dmWMoKq+w/qA0P+ JxgeYlsu9rKP14YaKOa1jbZhFmoHlEgA0yDMMpkiKNVoC86RscJMGjgeznuhqP1Mi18PS/zkn7Ktk 9XnGViJ6UGoKyyBlr5s1ZBKb6f63HlJeCF8Y4Okm2fX8U50l3p/rXt/5N0yVXJNLMTDyqW1uiS4MT tWygzIFw==; Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGp-007bk6-RD for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:14 +0000 Received: by mail-pl1-x649.google.com with SMTP id s8-20020a170902ea0800b00172e456031eso6760165plg.3 for ; Mon, 29 Aug 2022 14:26:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=K93TFWAb7JwQdWCvuFbBF/NwLgqzppA/TxAstG/QPvU=; b=f8ZHy6CiTcDXROb3pFMLfetBSV+kFVTeZws7OjDiiohYptMrJjGynWZ4jN+J/j3SJj dCEDPAmPkmnX5B3X46hqeY2mb8ZvMpLB7j64gw3uDHSYMAUM2B6ucp8KKPug6MxLnEu7 BmebG6DC6czjsVXnb7ENc9u7p9QGYJElvc7qRNsIMqXvgjewnR2ulhT9EyPiZ2zz1B9D 56/cpf1MfCjBxFK5vFsjvAIH2zjjfVoYxcmJWgmvbc3eCKVdp1qhJM+tuigQjNUJJMgZ jrHBPRDF7lp+rxYzH8zskIPXprMRBcWxeh9qo4POOeyeUnThUPILDVIwum49CxVeKLEs T9zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=K93TFWAb7JwQdWCvuFbBF/NwLgqzppA/TxAstG/QPvU=; b=J5VJrtDZt1UgNIKeLjmkFJvt/MWTAwavhD61GepAY9S2a1y90T0dwg7MlS1t20gOJc 2XLlvaXnadTTRWVsy8SoElQQ++ftFYMmt084uAPVxD9l6zYl9w2EfTvUenCCB44PrlEb noy9CsDkehVsBH2rejmILNcj/DFvoWY5nOZmQcCW6dtZJT5IbL3uz//kH3giLOEsMFRE uZAgfwkUuBSOsTv6rNGpRhiKHmRTODu8oOrtMnP+ZNNwUKNB0OoIGpiOvpsNTjkvSWDD pWfGS227Qq30lzG1oY1IaWCjlVpur7blDNjt2AxzDtAG9oj2UMxNADizwGs9ODzSby6p EPow== X-Gm-Message-State: ACgBeo2aiQh5F72lhd/zvhwOPm2IZp/6FKkVBqZfhR30yVxfD4WwrN2L H6b+uU2/4R3M3TfiAZ46qEH7ODfa6fY= X-Google-Smtp-Source: AA6agR7nf9+ItGZZ0+7hqrcNY7xNhTop6ny3nOph/OxIOFh5WmAm4Kp1l2Jp+ca/L6EEvcmkimt0snMwAs0= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:b094:b0:173:12d1:4ece with SMTP id p20-20020a170902b09400b0017312d14ecemr18383730plr.117.1661808370030; Mon, 29 Aug 2022 14:26:10 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:22 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-20-surenb@google.com> Subject: [RFC PATCH 19/28] mm: disallow do_swap_page to handle page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222612_146425_BAE03CE3 X-CRM114-Status: GOOD ( 10.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 9ac9944e8c62..29d2f49f922a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3738,6 +3738,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + if (!pte_unmap_same(vmf)) goto out; From patchwork Mon Aug 29 21:25:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958460 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C6A4AECAAD2 for ; Mon, 29 Aug 2022 21:33:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=BxCcXCp1pMRUtsC3mjoXCZ5GzXrvtlRQdnA/jIWrdSA=; b=lHHa4s9z/YhBI3j6+dCCufcvF9 v1D1OwV780Xoh0cGUixTEmAnSO6gmA2zXlNyLMKRB0wLl4bPWktS3RdTupfAg4IgMJAXjWdC/mUK4 evzCBst6kTcCgvmIOf5vme+FEHvI8Ufs6rtEHIuX3YS7OSw+6raQeHYJ5F53m+ddmMRB8z8CQE0FU lDKe7hcYwWKqRU79sK9Qznyt/uZLFusGUsy8wxhsYGNdPBFFlzwaobSldGtg0u7OgJFTyIdDln1dR V+mtSYlR+RoUfaGiuVE6xjHC2SnUllSGmqUByBwgMpa2q7iDw2AOMOhqoz65jggii54KRtzhSaxgT MhP7dgsg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmN9-00CppG-Ft; Mon, 29 Aug 2022 21:32:43 +0000 Received: from mail-oo1-xc49.google.com ([2607:f8b0:4864:20::c49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmMv-00CpkC-RJ for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:32:31 +0000 Received: by mail-oo1-xc49.google.com with SMTP id l14-20020a4a350e000000b0044b6be0c06cso3835274ooa.16 for ; Mon, 29 Aug 2022 14:32:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=bUksEMWzQ2sQtj6CAseSDP7Wnd3Y9YWuXlHuoAiHS/w=; b=R4f3AmhYQvfrJmCukv/e3Nij13PuVUa8k8O646df0PoObwBnO/xDcemk6+SqKXvVGr hJkm2649shhdYcclI4xYPbiKHXEe7XEvjq6ebwkp6YZFsZtQX3MH5xTV757BPS7Zeyib IALyiqYSG4f0kCJDHQ3+fMXa5F0OzkQKlh2EgahzIDAp6HMSPtfd8qN17COdyNthUkyK vH/w1n6XgnYHxRHIzAjyfQH31twkRddq+jpJViGP9dMWFd2ba2cwGPVTVH8mdAvdjCY4 84OD9xZ4r8HlzIfM65gveSjGxaILhsxn9vJXDz/VODVoOMzJt9RI4cwlcbKnFfX3PjSN BY7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=bUksEMWzQ2sQtj6CAseSDP7Wnd3Y9YWuXlHuoAiHS/w=; b=BBB2f6S/rDemqeRX0pufQlzbt+fEliJRaRziwo9ryiRxgpXf/aAO0af4f4fQyKmUL7 Qvf8maLcX83ImNbCMuiaTznAMTAvOwpwitCwnGNP9euBop/L4lIHzRLnt8DH4VSRov45 A25RsEF1o0SBwZTtiRb2HPpKVsHLdFzYVRr8BygcHIyUtsQ8r6tRNkeQ4V+CGSVAi2Ed 9L7prujP9P40tzG1C1Vuy1TrzzGkuyk9v97meQrEw5hfxHyg4ox9aoEV97O5OxXLPfVX SHMoWZYzDR7e4k+fotcKjTLxXsQWNVf3dutDl7JATYxhPnfSYG5xR6WrKl+XzpLhfHhe SdVA== X-Gm-Message-State: ACgBeo32Ozt2qhmoM0S+Kq+zlAl9pD9IFC+6YSjmo2XKdQ/W9Yi0hoPy vMvp8qi3y2Da6tx8rWtk3AtMc0ZIwwM= X-Google-Smtp-Source: AA6agR4CSgZfYughTw1oP2TGtJ/Gx4qpWPKVumSGbjZh4UG9V5skHPeijnTVviS5CargVPsvloUs2cN/SRk= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90b:2c11:b0:1fd:e56c:79de with SMTP id rv17-20020a17090b2c1100b001fde56c79demr4481996pjb.201.1661808371494; Mon, 29 Aug 2022 14:26:11 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:23 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-21-surenb@google.com> Subject: [RFC PATCH 20/28] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_143229_953887_11A3B42B X-CRM114-Status: GOOD ( 10.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 8 ++++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f3fc36cd2276..a325783ed05d 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -150,6 +150,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index bfe38869498d..0c2611899cfc 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -131,6 +131,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_vmacache_event(x) do {} while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index ce8dded36de9..075642763a03 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -207,3 +207,11 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + help + Statistics for per-vma locks. + If in doubt, say N. diff --git a/mm/vmstat.c b/mm/vmstat.c index 90af9a8572f5..3f3804c846a6 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1411,6 +1411,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Mon Aug 29 21:25:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 482ABECAAD2 for ; Mon, 29 Aug 2022 21:45:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=k3tCB8BiF7mF+zsdo+fSG0spDiWS830Fe33YiqFbguQ=; b=1pxTMnVRAhV313wFxhg7NJVzLx zVAmvflY5J6kz6UEH+4XdHB5n5NtT0cgtsEUm7gghZVezroAYXU2+cZOYrEvNSJjC4nkYaceBUlgQ 8FmTTUAyAgknqsYmQtCj+D7BfnMTdZ/RYLDIdRo2Y6O165HJmMK3rRz+4tTZKGnVKp99sb93cKZxF 0Tblsid/5H86yihY1LIbPIpZc28cJSPyP6jxvB9y3Pw4yH9QuBniJ7+ePYTneurHbR2cNGptPgNbE 4j/xZrOXPFXC/9be4gsns5iHKxQvdrEg0wHrPaK56NmX+mv7INr/8MA7lmqg/vYEJLxbmV8a4geki cJMKMWkg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmY5-00CtZ4-Ki; Mon, 29 Aug 2022 21:44:01 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXm-00CtVm-8N for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=cc92GYW6R7Bceda0fRDa3YFMWm qlCjLkfFdYoj1pyuUs6w693iI4NatYMmMbaKr8CKnv0AE/dqWSefNZcVm+2McCKn6LhUDk7ZkhndZ acEV4aFYn9xJpHtzJ44ehwRU+SswkklYIqygtqegxNCQ087utvD7PauMc1TITo1wJ9LXXydF8Hl92 zCJTNXin2QRddejhz5P8ctfOPfTYKnVKSFf2XJtCIShHt9bo/FtmsB4yxPvL9ef65fSsYV9i2L9PM bSvyu95OKIKHvbgztdL/vwfpnjmJpp6JTdD+eq9k890C3QebCyIaDHiF5UPyWRlDTepnLa/7KyiUH OKIX+ong==; Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGu-003Pss-9I for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:19 +0000 Received: by mail-pf1-x449.google.com with SMTP id o5-20020a056a001bc500b005382747a4e2so1503552pfw.19 for ; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=WI+iAkKYdp/dw+q+sE6vVFwD5mLSWBWMhRHboF9Ttbardy/unLndkxHVrzFVqulr1A yPcJLa3Ur0z+PJer1B1IamGoCGgxEpMLeAxgWf9CgasGK7cypr9FW+usqow4Tv90c3k6 hSoUgkPn+GgzurgQcPzRD2rwA/PJHxIAlhRbX6toXLNmtMd1EOWFLhXluJKlLDk1UBbT O0I6rKmCm2uwsJ4JnyJZiGAk3wWQ6Qh6RctA9jkOKMvzLM+0NSmJ3ovBUaPQyVoGm6YI JXf6mvqPKdqWiUAL/+XE2dov1V39lipJcNcZcI/t4fYcyAX1GpBk//UErawM2Q1iWoKr uzug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=v3sLSmvbuTuCOq5nF3wM/Oau627JJxCPao+XnfUhVLPe6E3kvsIYHKVrsXlhA7mZ1Y gir97B5DDyinqurXyTdRqM9/kWHcxBiheAw5cU/ryXfQGTJ0vB/R8AqBZ7UOZvZUxxHa DuNPofyxKyOLsZeveh6UNWYSFn42K+MQyMXyjDXrGnkwZ/18oiWGHevjxwNdaF9NAhon //sH9NuJbP4FY1yL/iNLiHfZ5d37IRgH0dSTajiNgt/LaV/nlEKNATthu52UOOmxSD2P hZ9mH5rX2Xzg+pjoG3Ovc6Y35AZuOzKv2xPumHhODSjUZY7FyDjlYwBomdmhPGHdcNQr +uZA== X-Gm-Message-State: ACgBeo3hhGmQ5C9n7LQZSOejpbSJOcallutJZn3phUW7wvHEpmMtuZMO Y0mf9i2fw8q/Gbw/83u87/Veujs0OeI= X-Google-Smtp-Source: AA6agR5aic644PZ0OuL1YK8O5lnv8GmzPsmAclgLgLZvZKPuRUB75AtHpFEjRVbtgx264YmpiRiioO9wBJc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:249:b0:1e0:a8a3:3c6c with SMTP id t9-20020a17090a024900b001e0a8a33c6cmr21187pje.0.1661808373054; Mon, 29 Aug 2022 14:26:13 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:24 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-22-surenb@google.com> Subject: [RFC PATCH 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222616_378231_498E6DA4 X-CRM114-Status: GOOD ( 13.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma = __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be stable + * and not isolated. If the VMA is not found of is being modified the function + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma = find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Mon Aug 29 21:25:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 991E2ECAAD4 for ; Mon, 29 Aug 2022 21:33:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=fUUVkOv8y3oR+AhGS0x8eCveBy/7nK/a422PPtOOb2A=; b=Fw2LUpqJsmRpMv7nq40xVXWnwj vJytgd3QhM68MQgLRdP1s2VkYxVOJ9wwKVBBXksemE2wTXjJqUDVBpw51FcN0CxIUw7TOFEXhg1UZ WfQDThUrOAh3EO/pTE3xXwFJRiqcIMsAxR/amWYeTK1JZX2GNkJzr7+Fl0cRz0D7qNQ+QFqmmJrWx qUsJcuVeVqlpN/VC6QW3eoCF9lQNbVCE2/cAUEkqKn6JVvKvEWgmcukMaFcrbm8OoWfkTm2FXl3E1 OLKElmttVNLe7b1vgzvVVD8aZbkLv76a0w8/iOk+711yZHtUnBk9EQv0SbpRU5+sr0TVokzTliD6V IRSDQ9ig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmMe-00Cpfh-Hc; Mon, 29 Aug 2022 21:32:12 +0000 Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmMU-00Cpcc-JJ for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:32:04 +0000 Received: by mail-pg1-x549.google.com with SMTP id h5-20020a636c05000000b00429fa12cb65so4615568pgc.21 for ; Mon, 29 Aug 2022 14:32:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=plnLK4otWaGxR+W8sd3fBxFm06LW87+WFj6YX7JmKJI=; b=O7Fq7HWZ7z8n2j/HcxIFYSMFkVCbPksq++2+VIS66vXRQwbI7dU8JTklRI8MPiby2e edzsUySG6IImo6mrh8ENCQ2+wxA6BfMFPfkVn5AwXOsZupGcnzhGGSKXttVqFe7CqMCi 1h3acQOuI10/ig3PP8xkKsm4NtzNbHGpaQpLwoGjKN2BSoc/HFLvkEbHUIkKXG5J3g9z 9q6Jpb0Ph5QlVq/4hF27NyIuM3/V2/KAkuHoouSJodEv2dWsfVfE58VeJPPY/+QOUQfz r+QwUf2vpfx2TV1s2fbWWB8eIKopRLMSEdWWEDiBiVoTpxt/dnwqSCw7tnlmHUsJj3fT 5A2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=plnLK4otWaGxR+W8sd3fBxFm06LW87+WFj6YX7JmKJI=; b=eBsk295neDzacglOwGgfciaC39WU7OW0h7jwEwShAYg/KByo5JfDncha592o/t6UPd BOhWtF6XOy7uG8R77S/IiX6inXZUgs5MTVepgPLzeqT7R2tFWTbygJh2qAG6gDDCRFdW iR3R+lJezSXKPOSZBWCQPE2aLtqjtxy6ByRTfwRcP03Pkb5kiA3+eenK8I/YeYdcrJhh JzxOIksXO7sYcSR9+Eo2slgb4hFni/gWHeJ0T/oRut/KpA23t7hPbmr6KNNRrXtpLTPE QXs/4MMjYPRSnk9mN9deXRNCK+LXQQvSm/23EXQf/13YCYxI32Zc/A0mO08SytzZCUOo W/nA== X-Gm-Message-State: ACgBeo0jbhU6nMhZiRjxDkMxDsGe9F/mLWaFqlOqCrjXAa9Z3d7ZQjZw dAN1kKhA1awGtH8haYF8ZGR2GPKmf4E= X-Google-Smtp-Source: AA6agR6koDuvaTUSC7OU/WlAja75jS99jSayBh5syt6ErVM0PHA+biyxUvs5Ar838YXdhWnvkXm6gonES4M= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a05:6a00:14c7:b0:52e:efb7:bd05 with SMTP id w7-20020a056a0014c700b0052eefb7bd05mr17908102pfu.24.1661808374985; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:25 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-23-surenb@google.com> Subject: [RFC PATCH 22/28] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_143202_700554_977CCF59 X-CRM114-Status: GOOD ( 13.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index fa71a5d12e87..35e74e3dc2c1 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1323,6 +1324,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1423,6 +1456,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Mon Aug 29 21:25:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22D7FECAAD2 for ; Mon, 29 Aug 2022 21:27:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=J99iK2TBrMjgSxcYyN0qIDRu1Jb2hszpqpWpRPBupJ8=; b=ja+jSnD+3NiV92Q514DlC/w080 xh+QcB5IhqhAex4na+sx3GRWCnixA+nYBIYb0ggmwU+qo93ANie6Q4euO2fYPSNWMHWphCS7hBO+4 gjmsdwE1FtPtF0cisdCdaUNZITmXtzi/Qzgc1AKl5O+teYE2qP17ycwNCPGnzgsL5ESRHAAPqMpSC WOMX6iLrGzlGkEYzEesChrpyK1LJ8Ft8SSHx2HB2VMYJp5R10vlcdqH917S6BHwSgvppY75Nw7Qwm eZ61Ee+vfbAA1kF2LGiZ4WXbJ+ojqmJ/KDucEMUQ9kly/7fZdpTJ9z4VTQS66eFn5VZqlWjDqByDt kOjeupUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmH6-00CnX5-Ke; Mon, 29 Aug 2022 21:26:29 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGv-00CnPX-Q9 for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:19 +0000 Received: by mail-pf1-x44a.google.com with SMTP id a19-20020aa780d3000000b0052bccd363f8so3605945pfn.22 for ; Mon, 29 Aug 2022 14:26:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=2fd/wGll+ATU6BJRTn7wtpRUT69Ps19K4L9NuJFwTCo=; b=CptXAEbTEIrjcI1hpAhqMGzwstwkcEJJxmfb7we2Tkw33BFQo0d25Vl71IxE1hgwCE +/zdjUxBz/kfwlykJ2mpRLD2ASHh0Jx/8NPF+GvmOj+OIPDuJ9h8tJ0qgl8jIPnA0gyY HL/K18g+JnPGw8ocd38eG5KHL97qZgAE2vHMSq8EmM7kYFzwJqKpJAdVhwHK8rQ0CEUk zlwtiJI41bKlNxlGW7NWWuuvJOTrejKhLXTonY1HBOreH+6SF0a8TMKsBr5d7H138Lnw MYC4Krs42a4eoKPM77oYkjvTzwtNbqixe2Jty6RY0Tg1xXxoDrCaGfCoHKiNRbMG4ac9 6NkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=2fd/wGll+ATU6BJRTn7wtpRUT69Ps19K4L9NuJFwTCo=; b=cA2r8pHY5DzQB/hvMyG1vx05ZuCVm09PX55CZrV18nNtxpnEUxpNJWl5H7ASrJ43OL fJiZ1qPlaWXkK5Z8P+fkPnawVL4gV0oZ4W28PrnOBJkpl9KbQ0RqUpvIfQl6zR4iQHch 1SRB5xuA6uJzgowITkGoPdg+VviEUl+O0BiXPUi2LPkGY+ZXUJdTi425kxKrH1iRsFyP py5ihmcgDeo8xyV4XLAvC1URjLKza2yTV4al9LTMQrRAQOCRsGYZY9p6uqoTWYCIHwN0 5iX/3XW1z4jdfUkOnBbnYe1TUHZc1qIbnty296OcMS8KU3p36ZGqrAPFZvfThyh2Rn+e V2GQ== X-Gm-Message-State: ACgBeo2YhSftRgVb7LqojFTnllDZ21ysaVVm2RZummVSmnY9f0x99dcs C7AlXT3aEUIwdZ41FQndQqggWpoyPqM= X-Google-Smtp-Source: AA6agR5QjqhJOQoDiATaLtKSN9vwads3L2SWpCSYlmwHfKycrJZLy9F13OZO0MNqEBq8/z8yofCbHCaZE68= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:d54b:b0:174:af35:4ba0 with SMTP id z11-20020a170902d54b00b00174af354ba0mr7860599plf.74.1661808376452; Mon, 29 Aug 2022 14:26:16 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:26 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-24-surenb@google.com> Subject: [RFC PATCH 23/28] x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_142617_894217_D075210F X-CRM114-Status: UNSURE ( 8.87 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f9920f1341c8..ee19de020b27 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA From patchwork Mon Aug 29 21:25:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CA072ECAAD4 for ; Mon, 29 Aug 2022 21:44:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=VVRN+th6uYVZGtaI7pMGpLi03ZXgDk8z5zsqaajWk5M=; b=odETb3u+qdl7Skeq90PhX0xUS4 OaDu6+zYfSvm1ySyqmWdIxKWntFMWsPps2jk1vEWiOUe99KknewHoP1j6GA1EBrtB3UQvVuvInCAA 6fv2o8u2/amXB1d7AI61d8ttjmTdxYHOYqAtA4mV76YqzgMh4iuB0L2GUbByR+AYD6JO8XpXEIE5f LFjmWbD4nY0/9rsMXkKx1zgdl9Ybb81+9g8ZB/BsaiSsbcZxRgn+cdIt778Ya1RIoaqO+HTEVrr/L 7lMgfrXKLSFknvJHK0OcYXWOyFdoU2n+6B71WSeScAJLSDzmGFMF2iflx0824fiQf3/tecqPP6nm1 noXyQfcQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXw-00CtYN-Ud; Mon, 29 Aug 2022 21:43:53 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmXl-00CtVm-R9 for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:43:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=3FeDd/uESQnEyJqRWNnPK+iT++ObIkcBX2+m7Yxyetk=; b=ts6Wpd5XTNmHV3CNaKO10c0l89 I6cmDAMFxGCVB2hXq2IAkGI8ZaWXpQqYHT3Ss5E388gNe/CydZZ4E2NxlnpCU07YEArQVInhlIqw+ zl7zBjbsGuq/l7MRPTjhYNHwravpJ6c14yonYD8d2c2NzO1rAevxW5U1UumEVym9uRYrTB5HQWkhx /R2XvErU3Z5oosPS+//vXAF7VeiErpbfkggbBahDXQWoMFDY1oNDKGiWn/GMiS3A8kT50B2IQZknY DYLZm+ORDTywnYqqO1Z1zNLLUeS9Bn2Dna/2gd+lkX1MtdxzzO4AcXJLzA8yI5Raw52KTsUYYrrVA wykvdBAA==; Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by casper.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmGz-003Q1f-I0 for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:23 +0000 Received: by mail-pf1-x449.google.com with SMTP id b25-20020aa78119000000b00536a929d8e4so3565640pfi.1 for ; Mon, 29 Aug 2022 14:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=3FeDd/uESQnEyJqRWNnPK+iT++ObIkcBX2+m7Yxyetk=; b=WuXmy7spYby6FKlYZxVkneAX6CYLeedEW25AaCdm5U/xwDtXPaTsukbnW+BREr3wmK Yp+DdB6epKs/SB7JvycD5yPQfOWs3SBJlayjVM1bzBPF4xDdscUUlmQ491HoasCmixWz EDQApyvlKJ0m9j6IZ5Bl5eAvYAMUvQ5PzTt+MmJPi2OMUuj7wlttfCVr1jtd4buEfno/ mHPpjk9OptKM0QuTB8OPa5JPYvPxCHU8q8PlG/PC9yFoVmLZ3eeWZBp1XntU6iG84/rA BgcQG1UC8Z471LrYLx/b3SNnNzdfHV9f4k2uE0RZvNOL2F/Wod5mOLtISQc7G5F1GR1Y 7dZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=3FeDd/uESQnEyJqRWNnPK+iT++ObIkcBX2+m7Yxyetk=; b=l4dxTiX0gtD7JzJXt4u4MCv+yVBGHS0HaVrUx+YalfUqSbcEBYPSuD2o+JKOAm+LP0 aTrPl/kZHT3QqQihV/23cYGlsgcYly8rdFD9CFKOMU0YW1F+cEIBTd401aCeylDh3qTL +6ljliWpVKCjVIPfF0/8n6+98qVsUozBrOkl6G0q5mTaL8xGMq1LargpFmZMD1wERiSa 9+n3weIyCh5HFgQV2SsQafz0i5ouUWEA45JNPrIAFHeegJv8yb4Z3sXunGhupHkat4Ur Swm/YScq5vSpD9sJwHNfcmJ+LEVaEl9zALrYtPBn6EowqsFTEk+RRGGfaG2ttyFoQWoP BSRQ== X-Gm-Message-State: ACgBeo3eKMTjsm4r/e5MtUTcnmZeMpXu/ZYzoMAaIz7QXY+ckoYgJbiH Rkyj3om5gv9weGz163ePdrK920iUU9s= X-Google-Smtp-Source: AA6agR65SMIHezE5eUhR7FLJy4sKjKwDRErJ8crUMropZEJrhJg/maBEVhsE6eunhKdLtoV+/pEsLbtUb5g= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a62:b519:0:b0:537:9723:5cf2 with SMTP id y25-20020a62b519000000b0053797235cf2mr18436343pfe.15.1661808377917; Mon, 29 Aug 2022 14:26:17 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:27 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-25-surenb@google.com> Subject: [RFC PATCH 24/28] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222621_614218_CA0BB349 X-CRM114-Status: GOOD ( 11.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index c33f1fad2745..f05ce40ff32b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -525,6 +525,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -575,6 +578,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -618,6 +651,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Mon Aug 29 21:25:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E48C3ECAAD2 for ; Mon, 29 Aug 2022 21:27:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Nd/uLJlFFU8ijP6/E354t1ekmdY6r2t55C5Wqre6tL4=; b=W87IkEQA9jVkU2nVJmFP3y71qb nhzldtkQ1OWeiiZCcMRQZxqfg6q0dRXTJ6EftDNkLNW5CnCbKOZQav97kJs3uXOK/HG+hsHiN9WW5 4jTOcxtEI6gAdgnQfDF/a9VnctmKjoGXubuaJiaieLXxUbGtqGw1WEodtyL0cgAd1niHM1+5cf/LY i1t1ttfWv2+0sLVWm6jRZVdLDbQXStX2VAXUkM8vkr/3BtFhuVpi+pjQFnE8agPh7wO9uGlP4RJqQ a7/tXIaLDRpo93aTq6ccvea4aEuA62pC30BlMCdeOddzyYV3iHygyRoEFUHfGRniBGYbXkl51tvKT q8Obs9eA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmHO-00Cng7-Bo; Mon, 29 Aug 2022 21:26:46 +0000 Received: from mail-pl1-f202.google.com ([209.85.214.202]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmH0-00CnSl-2v for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:23 +0000 Received: by mail-pl1-f202.google.com with SMTP id y9-20020a17090322c900b00174c881abaeso2531741plg.6 for ; Mon, 29 Aug 2022 14:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=M/5Y14x1SSOe03/jfupiRT6TG6dq1ngDrWV9hskObuI=; b=keg1i0nk+izk1klWl7kacG2dqLVFCBBFr7DTBR9Vkh347LyX6fgOptbA6FX9RaUao+ vJTGW+XEi+UTAE6ZSui/ACoLYvw+CzBRdshi7UOcb6GKyXUkGWluK1e9ViX0u2MOFUby n+VIqiL1KgbaHmBILZRqDKCT7yq9OcaG3eWuWfyFbeDAc3tNJSfOayUDVB763p8OZugP N6WyMveSXRyt1x9RwdvL650Tufs+g4oBOJJ5iU1MJBZM9QMXfDComRZRSTq/8D/ye7ef ZSEe/ZXHQtclALcTUMn5apyFMWCE/g3tgTBxLG+Y3u9vQ4Ps1pm7jxw7nOrCfEth2Ax0 A9EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=M/5Y14x1SSOe03/jfupiRT6TG6dq1ngDrWV9hskObuI=; b=VKkJb8hXfMxqusjpEFEj+2r/ctAV2ZBmM9MPWfeCTOJvVo73UWavZ6rsDWo5ezx2JN qkee7CkF2hVF2eouaj7ufsksqkYaUpbrxHqaiTLSEbbR7y51LlnfQoRk5ZVzZkZS4G8F MieT8t4zDOWBfHSgJt5U/hJ7VtE+BGDaikg9w1ykPGJd4ysLc4bsf5OFL7PjMek4yTcW M9fKsCNmDO/4uKIKhVnqyqUmhN/SrCb8Ty741uwSbkPXAtOBUD/tld8RYX8VLvwjOyca uLqLvPHUI2cN18ivYU9lr7L+XW1hGnzgOoTyOqfEIB06vg8Sek2fSbkcJUMVvL1xC6Tf H4Bw== X-Gm-Message-State: ACgBeo0LrIFQ0TTi1/J94S5nEENW1wkpwX4vQEF76teD2hk7s//TGOuB /xfOrFZ5OpY3GtiI3xCcvRWheJbdaSc= X-Google-Smtp-Source: AA6agR4Lx1uzJZ7T/fCqu0c+3ou3bFPxSP7qIc7HtsIZjbU9yCjHgv7Z+4FP1+5TPgAu6qrR46XQccXnjfE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:e558:b0:1fb:c4b7:1a24 with SMTP id ei24-20020a17090ae55800b001fbc4b71a24mr21113pjb.1.1661808379380; Mon, 29 Aug 2022 14:26:19 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:28 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-26-surenb@google.com> Subject: [RFC PATCH 25/28] arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_142622_147695_36CC484E X-CRM114-Status: UNSURE ( 8.81 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9fb9fff08c94..0747ae1f3b39 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -93,6 +93,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT From patchwork Mon Aug 29 21:25:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958462 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B90F5ECAAD8 for ; Mon, 29 Aug 2022 21:36:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=HqocX36YM4FoZGLLKuvEmevaRPkkgbQXj9c6DuC1sP0=; b=sAQFKKhJBgpRvToxpWLYWwFxMt tct9wKd4xjNZ3cJD4bskVmaaJ1Ema/CiRAIhkoeYiZhLAqgoDaDAK032uLVd2x7DjzyW1CjdOekqv I1+2Kl2SzqJd99gsev2Lm40MRSq6w3UjcSKfT7ZNhXQwyIAoE0cSk+yWy7n/0ZxwWMivdKE/3HkqQ ztISM663dWy62RMSEbfo/XtSZ704onUc6eIejDaCfTZLlq4fEBX9pho86iGwCbHsrdkHsvM1ekibQ pG23QZpIVfmZLi3iNkIQMHqpnQFPXcCtjs9peHzAci7kY+BU30dyBRDeUmzY/NXnPC0xWm0jvsz+3 An8eVCdw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmPp-00Cqzl-2a; Mon, 29 Aug 2022 21:35:29 +0000 Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmNh-00Cq1t-Ck for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:33:18 +0000 Received: by mail-pj1-x104a.google.com with SMTP id e11-20020a17090a630b00b001f8b2deb88dso10122552pjj.1 for ; Mon, 29 Aug 2022 14:33:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=o1aoXsAZRKWKp3E60vO3aTfa7LhhJu+wloCiW42SL3o=; b=hVadOQBIf7hIcCCPF4k/V8ishbMycFmV23IYK2/L+oJFUn4snhEXCB8nhiJ0XvBmGq IPVx7m/qpumur19aTHX425elTvPhBwnAWLq2eu14tCxMYLfyVDPtnXMD7eRtBYdE8ib6 g6FY2QFmIprsXVeiDN0/TJA6z8yPl6Ga6H+AU+5BAz5wvK9q2eG8K6M5/G5IT26iqvDR sF5KQOom085C4/LO0v/IjmuRTEiouanpvkUJ/WwC7Eqe/f3fgCYq7VPZ9r9nrS+kR+CY ydqosByt6O340qUe8wv/4Yn0djwvN5x2tceHdnG95lN+D/X9zA9TkMcItH0K4FkMtDqG cLTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=o1aoXsAZRKWKp3E60vO3aTfa7LhhJu+wloCiW42SL3o=; b=eJXSAvBi/rqiEyVAtkAnlQCZUJguQ76T1yck6JSdRIJBGtmvP9cgWyBSmk39WkqDI8 UjCo3OALehhTbmI8M4CKZ4t8MAenuXkgX7/mD5tizV5Jwzvnl+XiR+GHWqX6qzl5jHN+ tkHVofw+8ZFQReV9PfVdrlI2rFghjgEcneVpk0UjqRpCIP74ESX4RKtnMPpTP4pd4Q0a q+1FtRjXdC20MVm5jGrtMz++V6QkzzR5zes0xQvmil/yPZLcxvovrgN5P1gXKPzxUBv8 mWjqNmwW2XSyPFpPbYWr9O+j42ufWF/3LsH8oDTAOpm3B99O9HMIHC7yNIf++c9syO5z Y3Zg== X-Gm-Message-State: ACgBeo2+49r9fFsSk7RrYLYfgr+FDvuipGLbxNr7Z1mhh8wEbiH3TsHI gP2W2kCCQwDOn6zp4e4fOPZSP1q33dg= X-Google-Smtp-Source: AA6agR5CaUs6YJBbyF56NgpgIgB6cgapfFaHPRJdugast+FKzlkUBYXObPd4n8zhAA8UFAnqt1i7b7mpRsg= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:41c6:b0:174:3acf:8294 with SMTP id u6-20020a17090341c600b001743acf8294mr17683670ple.118.1661808381376; Mon, 29 Aug 2022 14:26:21 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:29 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-27-surenb@google.com> Subject: [RFC PATCH 26/28] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_143317_454328_799CCC06 X-CRM114-Status: GOOD ( 14.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour --- arch/powerpc/mm/fault.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 014005428687..c92bdfcd1796 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -450,6 +450,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_read_unlock(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_read_unlock(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -526,6 +564,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); From patchwork Mon Aug 29 21:25:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 528B4ECAAD2 for ; Mon, 29 Aug 2022 21:33:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=lprXYnmWmG0lsryVE4IAUe3btrbALr7KKYK/+RF7FQM=; b=vn8WPUkkQkOLyfga9O14OAAdRU L2jN1kR3ae1LuBIyOrljsLhypa3lRMAYcYGTm2coft+IN+zqbFYg9b+BSZ7/nKjvdzzj6mNGPztdS TKEpRJTP24G5nz1Olm7YqD6pXvPvtfoDxKI//vX0FOLRF0/Xz9zjo6P8QEOXwd7a0n3xaCEyFM3vj qmFPUVD2E1P3S4j2bABVLKQBq5v44y0wkClIGEFHhryzkLlHCoz6ddoQOY68InBBB33Bj6QQ49Onz hTEWRpWn/Xr/a1DdmMxuCKRtP4GJ9JQu6ZV6S4pxvQaqVz/W9iGoxqmX4sWCWyfEi4g/4uF9DvKl4 3bB3LrQw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmMU-00Cpca-4L; Mon, 29 Aug 2022 21:32:02 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmMP-00CpaJ-WB for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:31:59 +0000 Received: by mail-pj1-x1049.google.com with SMTP id n30-20020a17090a5aa100b001fb0c492d5eso3882875pji.3 for ; Mon, 29 Aug 2022 14:31:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=WpdYqLbUAWRRR7NpWeIwifObGgR3/WEcKlSas1duxWs=; b=A8iXlM7kWh/+WLxmr7Wxu633pJ33WbnreWCJkhO2mN51OTMuCRIiZbXSJTMwU7nnLp 8HrWKhSa16qOE9IHNHTIDFaumIghV/X3vZxzBwxJyopuUJchdMXN0oUY2EJvc97qw23/ 002SGSF6e/GfvGXcSJZJ9OH52bwQ+17bBv1qwUEkh+/hxiA3XGJZS3QMw2bsNQo6K8n5 ASVJIXUmfSCyDSBXlWu7+9C2NTyQxO6WdEpOrWiiN0G0XOZ0xYY++9szDB3PW4z/iVNr kXUF0efo+Jl4A1NDPbFsv2i2F8cuEbcHbyZjyx1JQrC6JX3bDooGkyCEfiAtjL3LX2kX L4Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=WpdYqLbUAWRRR7NpWeIwifObGgR3/WEcKlSas1duxWs=; b=GNz+7bxmV+ErsmqZnrdocYJd3/8RoWnAxgmNJmPD64Widd2shyfczDP6j7WGAOnzX9 Jd3RXEMM5OVxMWb5yn4Q3ZOpilUsbmEQHTNeO2bIUdIc9qf3TmqrsHetnD/xzyQzpMiV +3Dby5TdQlhMKaoTPHssRQWGq5WW2pn6xN1MZb+hQ/iPCOBWLuBKMWj++wLQ5wljk317 c67zAdOn61fHur3DVhKC4zAm+fjWwjSX4TEaxPeIg2hVaCnCclNCcgBr+EkBbdXR3K9e rDbRUlyIxOyZ/8+STs6nnMqWOEGaspCyOG5N8NBm7wJ+5aOxeBFuESaCYM/ICaHVfM1o S0LA== X-Gm-Message-State: ACgBeo0wzooKu6dtVKUUqn4ULz3AW9v0xh/sJgVHueIJv5WMKFQywMiI tYaOm9VrkLZmqZzf9JOq2xfrL5+c7EI= X-Google-Smtp-Source: AA6agR7BIF7zDBFRMtaQFIX+dYsK3ioA4U218I6Y8n5CHeAbnavsVuqAEK5kMo+KsvAOcZae2RDMSpS8XIE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:2d1:b0:171:3773:b95 with SMTP id s17-20020a17090302d100b0017137730b95mr17499271plk.173.1661808383315; Mon, 29 Aug 2022 14:26:23 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:30 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-28-surenb@google.com> Subject: [RFC PATCH 27/28] powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_143158_071181_6DC8524F X-CRM114-Status: GOOD ( 10.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Laurent Dufour Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on powernv and pseries. It may be use on the other platforms but I can't test that seriously. Signed-off-by: Laurent Dufour --- arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index fb6499977f99..7d13a2de3475 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT_SPINLOCKS From patchwork Mon Aug 29 21:25:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31E25ECAAD8 for ; Mon, 29 Aug 2022 22:00:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=FVs83IAvyXkulG4DEsS5YG3XtWN7RSnn+paIMKbmijo=; b=r/XUPAwIXUj+/aRHmlpKBi3bGr ZfqiW0m+mJeeOoy8GzwwPVmfIUVxTm2afKKHfzxLaGAI50zfaFOJe4Nj8YcC9amK0f+3LXOvXwUpR xn7BZ7z4y8kJdOvTZKOzwRy/t3zoNVTMDdgn2XHy51+XNBLsbMCraK1Kxgc6rsHRJuJ4AM4b7Nl2p iR9WiaP8vnPlgJGdgmlF0DKEvVPLIGAydcVsY2rvyvUveGf5OKohcjG+yqhHsHL4q4p0uYQlukOwJ 6snL7RYQMcY9GwCvCT2vcDdA8kd8qooFSMxIhh8zm19AgnFeDOT1V3q4Ju57dJ1U3bKMfYKjTFfXV 6Wwhw4jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmS-00CxtX-AP; Mon, 29 Aug 2022 21:58:52 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmmQ-00Cxru-9w for linux-arm-kernel@bombadil.infradead.org; Mon, 29 Aug 2022 21:58:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=T8fB8hXi5puWeYGLUKktDcvprMtt+Dm3Upe4kr4atwA=; b=VMTQp5jlhp/GQ6T32LI669EsfU Q9ozWjTqYiA3IqxwvC4Pw/v2I0oD43G+iZ8aOllj4TvS4khqzLlHHKuBgSxUT7Ib+pvU02RhPRbhL G3YzQKVFAjIrJj11Gv2stRwNTaOFQ1IeQUb6w6O5zbNeKht9VQXXutJSoM+k822+V0AJ6NuLJg8AX bQREu1NR4Sc5Ift5QkKTplzgXlevwzUXJ8FoPbPnDgInUoBTIOcNxFSZJEFy4Uatw+BTOJQoPVRNY +mJmaIVINqTYPcMdm6XKWAwbr2kH+Cf/6Z+ahuFMjo4+6Z8c4NwYtcEvzTQevfcXI/HGVaoal91tK zU3zlVLg==; Received: from mail-pg1-f202.google.com ([209.85.215.202]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oSmH6-007boD-Be for linux-arm-kernel@lists.infradead.org; Mon, 29 Aug 2022 21:26:30 +0000 Received: by mail-pg1-f202.google.com with SMTP id a33-20020a630b61000000b00429d91cc649so4542986pgl.8 for ; Mon, 29 Aug 2022 14:26:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=T8fB8hXi5puWeYGLUKktDcvprMtt+Dm3Upe4kr4atwA=; b=YPMK3AJpw0lWwzYZfDLzh3K7/2W07auDNNnlJ3qDkZJloq+AWVG8rnBi3EjSTi8Sdc 7WJv0veKl0lZAyzfelPpHbd7cV4CAy5dCoMcdYFQ7CqbidAfsv5F6022/J5YgvQf67DO L1YTRMppSCyvCaIew0iiAysCdKjsWOFUDjcZZGzRH+sdcxnSvug7vs9F56KVsee2c0Zo YLbMsX94AWYL+h4w9bE7JCdOjOkowcodvSgMnxQ2NHaXBQhv7Fitxu5qAmsw46U72GSa SpD2ueQh2UQYjoAvP3IZCnzfNryUU6OHaypVe7QLBtTEl+UaqGMhHtnVjouLDn/cSCG8 82Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=T8fB8hXi5puWeYGLUKktDcvprMtt+Dm3Upe4kr4atwA=; b=1uLhTbn81/qa2fp8TP275AEqPrImkVtc+6+uLnuMMLaTz4oz4ZClqC1L4Z2jqkFEd+ 38PZM72tengv2eJlZCcB6XESV0mRpLsE6rqG6KhKgo17t0yopo6DJ+R0lB+GtdNSx5wx RJfAixHfUB1ziIuPk3jDfHISREWA4ud+DvEramGfIlb/Yp1ZmXFKOF2HBRiBb1dKrqOS A1uTtBVqPGcQrYpiV9mPkA3uHHPqIGGKj4zguTTkKL0+dJzYY7PwN7ejyfvPJepJtgG0 NUnTV5MARupMsVzRcY1neefzAu7pJSUwZaNAkaHNDDl31NngeibxKW9IL/V4dunc5vYT x02Q== X-Gm-Message-State: ACgBeo1RNH7zuEJCid5XSrE/N/iwJ7/20zc3ObcDc9GzxlMvrSa6NQSe YHZgzudCeNSNhNLrZ2esjLeiDmEjC9A= X-Google-Smtp-Source: AA6agR5dzmEiiCM1ZfmS4IP8jubPYXR49V7KmmeKiHJ1f+/Bhg9nkxqem44jNkystynSPzi1e7aYTBBx0e8= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8b44:0:b0:537:a35d:3c11 with SMTP id i4-20020aa78b44000000b00537a35d3c11mr18188023pfd.76.1661808384779; Mon, 29 Aug 2022 14:26:24 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:31 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-29-surenb@google.com> Subject: [RFC PATCH 28/28] kernel/fork: throttle call_rcu() calls in vm_area_free From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220829_222628_664629_A69D925B X-CRM114-Status: GOOD ( 19.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. To minimize that impact, place VMAs into a list and free them in groups using one call_rcu() call per group. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 1 + include/linux/mm_types.h | 11 ++++++- kernel/fork.c | 68 +++++++++++++++++++++++++++++++++++----- mm/init-mm.c | 3 ++ mm/mmap.c | 1 + 5 files changed, 75 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a3cbaa7b9119..81dff694ac14 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -249,6 +249,7 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +void drain_free_vmas(struct mm_struct *mm); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 36562e702baf..6f3effc493b1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -412,7 +412,11 @@ struct vm_area_struct { struct vm_area_struct *vm_next, *vm_prev; }; #ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ + struct { + struct list_head vm_free_list; + /* Used for deferred freeing. */ + struct rcu_head vm_rcu; + }; #endif }; @@ -573,6 +577,11 @@ struct mm_struct { */ #ifdef CONFIG_PER_VMA_LOCK int mm_lock_seq; + struct { + struct list_head head; + spinlock_t lock; + int size; + } vma_free_list; #endif diff --git a/kernel/fork.c b/kernel/fork.c index b443ba3a247a..7c88710aed72 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -483,26 +483,75 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) } #ifdef CONFIG_PER_VMA_LOCK -static void __vm_area_free(struct rcu_head *head) +static inline void __vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } -#endif + +static void vma_free_rcu_callback(struct rcu_head *head) +{ + struct vm_area_struct *first_vma; + struct vm_area_struct *vma, *vma2; + + first_vma = container_of(head, struct vm_area_struct, vm_rcu); + list_for_each_entry_safe(vma, vma2, &first_vma->vm_free_list, vm_free_list) + __vm_area_free(vma); + __vm_area_free(first_vma); +} + +void drain_free_vmas(struct mm_struct *mm) +{ + struct vm_area_struct *first_vma; + LIST_HEAD(to_destroy); + + spin_lock(&mm->vma_free_list.lock); + list_splice_init(&mm->vma_free_list.head, &to_destroy); + mm->vma_free_list.size = 0; + spin_unlock(&mm->vma_free_list.lock); + + if (list_empty(&to_destroy)) + return; + + first_vma = list_first_entry(&to_destroy, struct vm_area_struct, vm_free_list); + /* Remove the head which is allocated on the stack */ + list_del(&to_destroy); + + call_rcu(&first_vma->vm_rcu, vma_free_rcu_callback); +} + +#define VM_AREA_FREE_LIST_MAX 32 + +void vm_area_free(struct vm_area_struct *vma) +{ + struct mm_struct *mm = vma->vm_mm; + bool drain; + + free_anon_vma_name(vma); + + spin_lock(&mm->vma_free_list.lock); + list_add(&vma->vm_free_list, &mm->vma_free_list.head); + mm->vma_free_list.size++; + drain = mm->vma_free_list.size > VM_AREA_FREE_LIST_MAX; + spin_unlock(&mm->vma_free_list.lock); + + if (drain) + drain_free_vmas(mm); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +void drain_free_vmas(struct mm_struct *mm) {} void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, __vm_area_free); -#else kmem_cache_free(vm_area_cachep, vma); -#endif } +#endif /* CONFIG_PER_VMA_LOCK */ + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -1137,6 +1186,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK WRITE_ONCE(mm->mm_lock_seq, 0); + INIT_LIST_HEAD(&mm->vma_free_list.head); + spin_lock_init(&mm->vma_free_list.lock); + mm->vma_free_list.size = 0; #endif mm_pgtables_bytes_init(mm); mm->map_count = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index 8399f90d631c..7b6d2460545f 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -39,6 +39,9 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK .mm_lock_seq = 0, + .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), + .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock), + .vma_free_list.size = 0, #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, diff --git a/mm/mmap.c b/mm/mmap.c index 1edfcd384f5e..d61b7ef84ba6 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3149,6 +3149,7 @@ void exit_mmap(struct mm_struct *mm) } mm->mmap = NULL; mmap_write_unlock(mm); + drain_free_vmas(mm); vm_unacct_memory(nr_accounted); }