From patchwork Mon Aug 29 21:25:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 049D2ECAAD2 for ; Mon, 29 Aug 2022 21:25:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74FA46B0074; Mon, 29 Aug 2022 17:25:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B107940008; Mon, 29 Aug 2022 17:25:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43FDC940007; Mon, 29 Aug 2022 17:25:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3027B6B0074 for ; Mon, 29 Aug 2022 17:25:39 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 06FA740196 for ; Mon, 29 Aug 2022 21:25:39 +0000 (UTC) X-FDA: 79853911998.16.5FD40C5 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf20.hostedemail.com (Postfix) with ESMTP id ADFA11C0042 for ; Mon, 29 Aug 2022 21:25:38 +0000 (UTC) Received: by mail-pg1-f201.google.com with SMTP id a33-20020a630b61000000b00429d91cc649so4542334pgl.8 for ; Mon, 29 Aug 2022 14:25:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=g4qOnyYThXS8aPbz4q0EiUiPOz/BrsimeRGT1yiKfMQ=; b=mWzgq+g1hUcWORjEKpHNvgndKK5fYzq5Kwo71ogdoOHT+8RKeEQm1dWhj+jH+fzB93 eAtnxNDhkrvgMQV0NbZW6cX2N1R1+vx9KkVCTcRNm2sUroxHvRgBw6l3WRg5Zt3VpzAp Gtf7jyf0iomGZE2cusptU53XV2fFMuQkz5kSqUL16OlTWKpS5jWEYB7QcGrOHtGU2amu R0EpT4v+6Ubf6b7FtWlOEkEYwXOj5Qkd+op5DjD/MSrHyHjvI8tucin/+y2TJJ3NqFBm d0kZdCDqO99cL/X5NSt2crz3dZBhVMVoGOKJ9zBhEnqaTkK0+up1kisxNJ5uiuFsJR5L UYkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=g4qOnyYThXS8aPbz4q0EiUiPOz/BrsimeRGT1yiKfMQ=; b=SvVAhkWBunJaVJ6iKH0uqRFZOAurgyHcBKVJYuzD8t9AfStwImwB8/tMo3ga7YERcw BAEeZPQ5kE6cALJ07mj/eAzBoqyKXem8TPSYAPMJyBLALfGvWiI3wByOA6EFph1/XVdR hOFtAjPUqv/a16OEaVhCwe7YHgBi7VIStI1YbiovW935KIX2Cu1m+0Jdydx+7U/GuOr4 ODS3o/PdsDZHNuTB+Anl6nseomuUu0LDl+u3rRrEkUo7/ernufUKXdNMmKSpztl0NGGK tPJYmcKj6qBTeGMV4f25Wa1SJlno1j9TNe0BM6+ZVGAfnyWFFqqmG+TStg5FRxjBLI7z cicQ== X-Gm-Message-State: ACgBeo3eEdy2FdnzMBxMHJ9o8ly8MrPi+A6CDCC7vQro8bruZB5RgR6f DJE9tlHanG6Y+Onun1fj2o9bULnml90= X-Google-Smtp-Source: AA6agR70MCvwbcIgBEgSXgn9wLeLPhFpHFOZXigYtzK5DEim0KxhWXMFn5l7ORAicxSiaBREEUpeMlkp2eE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a62:5ac6:0:b0:537:f0fa:4ae1 with SMTP id o189-20020a625ac6000000b00537f0fa4ae1mr12574833pfb.70.1661808337643; Mon, 29 Aug 2022 14:25:37 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:04 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-2-surenb@google.com> Subject: [RFC PATCH 01/28] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mWzgq+g1; spf=pass (imf20.hostedemail.com: domain of 30S4NYwYKCIIy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=30S4NYwYKCIIy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808338; a=rsa-sha256; cv=none; b=ATRtMkI5bd7VqHl4GgubA1XkndhJ37496AI1xe3t+TRzPB6HRU4kXeZUzEfFwH2q0myGcD o339aQIYL4ZEG0AxYpNAFOiCfpjz7e+T6YpzCieFuhOvVu1Oa0pidfdtke6+olnzyhBxdk zsAxd5w7TDNU/dSiuBR/BnMfAmQzN0I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g4qOnyYThXS8aPbz4q0EiUiPOz/BrsimeRGT1yiKfMQ=; b=pDffCsfbhStktU3pEYHtJ5mrtCpq/+PUWEZPbhAOyOuCo5yZXvWRH5UIxpSvybH7xIcj88 BNAa0VJAGhtwEkca15L8rGdBcYOKl9bjt4ohu/UdugNKRAHIeKJi33ooMzc8hc0RnXhHIC 780AgCqp8niIOjrajdqSs/BTK2fi8UI= X-Stat-Signature: rtkuxtdrjq1kcqyo315pw957zm7z66t9 X-Rspam-User: Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=mWzgq+g1; spf=pass (imf20.hostedemail.com: domain of 30S4NYwYKCIIy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=30S4NYwYKCIIy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: ADFA11C0042 X-HE-Tag: 1661808338-379509 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 0331f1461f81..58c20fad9cf9 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1124,6 +1124,19 @@ config PTE_MARKER_UFFD_WP purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + bool "Per-vma locking support" + default y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Mon Aug 29 21:25:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67FD0C0502E for ; Mon, 29 Aug 2022 21:25:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D91186B0075; Mon, 29 Aug 2022 17:25:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCDF3940008; Mon, 29 Aug 2022 17:25:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6D26940007; Mon, 29 Aug 2022 17:25:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9BBB26B0075 for ; Mon, 29 Aug 2022 17:25:40 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6B63C80AB0 for ; Mon, 29 Aug 2022 21:25:40 +0000 (UTC) X-FDA: 79853912040.12.F294252 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 2C0BBC0057 for ; Mon, 29 Aug 2022 21:25:40 +0000 (UTC) Received: by mail-pl1-f202.google.com with SMTP id t13-20020a170902e84d00b00174b03be629so3065914plg.16 for ; Mon, 29 Aug 2022 14:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=cPhW2Jsm5VYfGL+tNkfF3HbuaaNlLZ4rflaa6thsEZ0=; b=Ef83aGWTdm10niN8twSSpG3ds5UuJPZXIlmBxHFg2pzrVEdcovVnDh3P/RYOinFzyL 56g3s4OBiiWq4sYS4zyOn+I3sJdcQzzrlMjpsAwGpe/6DNb56LumMLV46gfcGkdOoroQ Xn2Ez1+n3LVumJ9ul/e3Fo4g/eC812QXSiR/GOAlZ/X37IBJZ8GhOEEF+zxhXhrBOhrA wyQ3RxsgNeUblE3TddZPmpCUfJPQ1JoHkMsqbRXnrreQjskaAOAntkxj5MpJLWS1ppBQ s3y364HBN4NsSi7BBvtTGWNjoBJVZ6VaguwCOOzVv7z+u5EVx5HkEViGicNoS6cS9car 5ukQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=cPhW2Jsm5VYfGL+tNkfF3HbuaaNlLZ4rflaa6thsEZ0=; b=vYlNP7HB5vyz262M1rgjnZk+931LL5PD7NbqbunJg32TcyFL50t33Lcoi4G7S5p84f YvbucZxmSSeZPbhgzFoeycfr+fLix72nzkflOOsGB4DDNTup911gpJBKAzFkl/QYikqY aRx9aVnZaGwOjw6RkNCB1FrhMwNbc+YYSZ6roZPw0yCnuIiofYDNaFoVhceX1rh1G5RM 1iqECVZjTlFeJyW9tYH0BrWZBFhU5ei9zU/RZiQvkcai3CgfdHwxmU3zXAd12sDYpcBW 73MtcT2bM77cdS/mdw4/bTM+dNkgsh0JZio6dLm1CLUdoXkvXWg4smptNycY40XNILxH 1gjA== X-Gm-Message-State: ACgBeo1JNUy3GDD4Gfv9ZRf2BR8kHfpMZzgkDj/4VHQhB+QyGRokKq64 /lZY6ekzvuczcxNVmotmOtaGOH68rqzJB2MFZAAbmXxyfcESGpCKToRem2HCa/edoX9cER6Q9GP sibXBMx0= X-Google-Smtp-Source: AA6agR4h8d6XU8wf7Rba89JuzvO6/M5QQD9xRgO2erAMFyngZx2nEIGpax2LUa1YM3lFKVFmr9HgbHhBg9U= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:694a:b0:16e:e270:1f84 with SMTP id k10-20020a170902694a00b0016ee2701f84mr17988146plt.89.1661808339339; Mon, 29 Aug 2022 14:25:39 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:05 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-3-surenb@google.com> Subject: [RFC PATCH 02/28] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Ef83aGWT; spf=pass (imf22.hostedemail.com: domain of 30y4NYwYKCIQ02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=30y4NYwYKCIQ02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808340; a=rsa-sha256; cv=none; b=I2Z+/RnIqOctwSFwqtXI9UYrmO3SGiL40hdJ7QBvqaz6MIXwaGyEolX0Dc5S5bnRDFCdG4 MaGV783lBjDvNQ90/E3ySf1hMODwJ81FAh/m5uLwH+Hlj+RXD+7RvFDMxnMRDeORLI54fi AgNep79wVTl2VVaW9xIDv7A/OpIt7JQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808340; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cPhW2Jsm5VYfGL+tNkfF3HbuaaNlLZ4rflaa6thsEZ0=; b=LNbh6M4QSJ+E6X9kqqvNpo2WVSRXBf0UsrYHn8vcVkUs+fqOaoeljvnfzXeQlDz8oSQl4m 36R21AJoWyyU1JdyyYPx30umb7RtmUPyZnfGRjOId+TWrL6cvNQcQyegL2eA/kx6w7g3cA YDqie0OyFojyaxi5tbYRi+amoXprGuM= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Ef83aGWT; spf=pass (imf22.hostedemail.com: domain of 30y4NYwYKCIQ02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=30y4NYwYKCIQ02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: fnen5yhah7c1h3acorayn6h99u61zd4z X-Rspamd-Queue-Id: 2C0BBC0057 X-HE-Tag: 1661808340-924395 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 16 +++++++++++----- kernel/fork.c | 13 +++++++++++++ 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index cf97f3884fda..bed25ef7c994 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -403,12 +403,18 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start, vm_end; - /* linked list of VM areas per task, sorted by address */ - struct vm_area_struct *vm_next, *vm_prev; + /* linked list of VMAs per task, sorted by address */ + struct vm_area_struct *vm_next, *vm_prev; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct rb_node vm_rb; diff --git a/kernel/fork.c b/kernel/fork.c index 90c85b17bf69..614872438393 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -481,10 +481,23 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +#ifdef CONFIG_PER_VMA_LOCK +static void __vm_area_free(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + kmem_cache_free(vm_area_cachep, vma); +} +#endif + void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free(vm_area_cachep, vma); +#endif } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Mon Aug 29 21:25:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 409D0ECAAD4 for ; Mon, 29 Aug 2022 21:25:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9706940007; Mon, 29 Aug 2022 17:25:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B48B36B007B; Mon, 29 Aug 2022 17:25:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 99837940007; Mon, 29 Aug 2022 17:25:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7B9506B0078 for ; Mon, 29 Aug 2022 17:25:42 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 56942A046E for ; Mon, 29 Aug 2022 21:25:42 +0000 (UTC) X-FDA: 79853912124.02.4D35D0C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 0BE3D140068 for ; Mon, 29 Aug 2022 21:25:41 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-337ed9110c2so147590717b3.15 for ; Mon, 29 Aug 2022 14:25:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=r66X5QU6Hd1zKKrQM4VdH4G/hUbDUrS2PjBFocCJ7kE=; b=f0ZITQpEExaLaSKDQb+h3/jYVFf5BjWfu0A0p9NMJ80xDBwkhscojqDzgDQ6W2gLIU JqyvlNc6N0TYQxhXpPiBzGuyzj1xr5cqDIbNzDR0v00qVutFwkDUQtyKnSj2Dn+NYRmF XbYPUhr7ZlEh9zlxoVr+7UyC5/R2ipvFX7ptRmhGZgjobP0J3ITaBrEOKND2CAY9iU2J fnwalrszPulqwRWfSp/VwWSeSLynEBR581yqV4QALN8e0aPzId5HIRtAjN2fmV4usJpw ghPowcSLA4CLZg9Bm0wFQDYbe3eb9NsWWsLPsurv1jkHcSRK8KYhRIFhxtRmb7OcOQkd NeDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=r66X5QU6Hd1zKKrQM4VdH4G/hUbDUrS2PjBFocCJ7kE=; b=zJN2XbHzRTGmohInCkILpD6RuvLvz0/B6OwnaaYr4GOm1NlS48pa2VSe05UdvRo61G EElhQY+ItzgcMoCMb9Npz/VXhl/JLHaDt8b+pwtUVf3xJJUK1d1jwbSHAVBe3CU6bTIq VFANe1u1/uBh1pi/CpgAXfzQTi+qlLsgnhd5SKmn+ArbprAHITsLqmu9ZDAmHN0Dc/fu H7UkDcdRsLA1KtuSoEgwGEYsXlmMOFnszT4hDNcgB5BifofpPGLeE7rNnQbpmc4SIDgW ANMIaFcHFlfSC7hNjnyySZI6SQS30Gu1sUDUxwpPpLn+/thppK/U68VSJL8zzg9ByWHP aRdg== X-Gm-Message-State: ACgBeo1bpjRJ7u0AwbnubykD4JspG3zjMmL+2bL3vqTsBcJOGKKisi/5 XihxfYTIzOjWArOStoZjgjmkfQA9DcY= X-Google-Smtp-Source: AA6agR7RORyYEvgHp7XQu1QQYhj/biZijMsx0xRWZsAvcqgghjoomkWy7B4TiExDCzWJq5q+aaEXQCYL4WE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a25:254f:0:b0:695:7c37:9202 with SMTP id l76-20020a25254f000000b006957c379202mr10232936ybl.79.1661808341087; Mon, 29 Aug 2022 14:25:41 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:06 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-4-surenb@google.com> Subject: [RFC PATCH 03/28] mm: introduce __find_vma to be used without mmap_lock protection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=f0ZITQpE; spf=pass (imf26.hostedemail.com: domain of 31S4NYwYKCIY241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=31S4NYwYKCIY241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808342; a=rsa-sha256; cv=none; b=eZjavHV7/u0dOkTudE3gpQ/9ASvnDMl3/kREnSAGn5eAtPMPbvgt+tr1gAXGdPUTCBPHEg RwttMZJVdo+fuhK16KAysxGWHmhvD5QZq+7yayrZFSc/Njfl50i6d6Xxkc0M21azF2hoxu RIQjrj1NNoZx8FeUvMpyHq17q9mX3lc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808342; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r66X5QU6Hd1zKKrQM4VdH4G/hUbDUrS2PjBFocCJ7kE=; b=ehWTGDRUSU+1bjCvsBUu7LZI/cVWijVO1yQ6YQ+CEdN/btNtszQjrp9P81NaYPBcsbPh+2 i6bUkwAUDTiQY4xDZv+JdNQRthtAXqRbnwAWnjmDdUte7iGFAxDsdoeonIjOlomxR/gab1 KPr5Gullouxt8LA8/XjP9XL+ObbcXnM= Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=f0ZITQpE; spf=pass (imf26.hostedemail.com: domain of 31S4NYwYKCIY241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=31S4NYwYKCIY241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: x17j7r3e5uj7994dp6hgtuet7ju437fj X-Rspamd-Queue-Id: 0BE3D140068 X-HE-Tag: 1661808341-855086 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add __find_vma function to be used for VMA lookup under rcu protection. Signed-off-by: Suren Baghdasaryan --- drivers/gpu/drm/i915/i915_gpu_error.c | 4 ++-- include/linux/mm.h | 9 ++++++++- mm/mmap.c | 6 ++---- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c index 32e92651ef7c..fc94985c95c8 100644 --- a/drivers/gpu/drm/i915/i915_gpu_error.c +++ b/drivers/gpu/drm/i915/i915_gpu_error.c @@ -507,7 +507,7 @@ static void error_print_context(struct drm_i915_error_state_buf *m, } static struct i915_vma_coredump * -__find_vma(struct i915_vma_coredump *vma, const char *name) +__i915_find_vma(struct i915_vma_coredump *vma, const char *name) { while (vma) { if (strcmp(vma->name, name) == 0) @@ -521,7 +521,7 @@ __find_vma(struct i915_vma_coredump *vma, const char *name) struct i915_vma_coredump * intel_gpu_error_find_batch(const struct intel_engine_coredump *ee) { - return __find_vma(ee->vma, "batch"); + return __i915_find_vma(ee->vma, "batch"); } static void error_print_engine(struct drm_i915_error_state_buf *m, diff --git a/include/linux/mm.h b/include/linux/mm.h index 21f8b27bd9fd..7d322a979455 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2712,7 +2712,14 @@ extern int expand_upwards(struct vm_area_struct *vma, unsigned long address); #endif /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long addr); +extern struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr); +static inline +struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +{ + mmap_assert_locked(mm); + return __find_vma(mm, addr); +} + extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr, struct vm_area_struct **pprev); diff --git a/mm/mmap.c b/mm/mmap.c index 9d780f415be3..693e6776be39 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2250,12 +2250,11 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, EXPORT_SYMBOL(get_unmapped_area); /* Look up the first VMA which satisfies addr < vm_end, NULL if none. */ -struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) +struct vm_area_struct *__find_vma(struct mm_struct *mm, unsigned long addr) { struct rb_node *rb_node; struct vm_area_struct *vma; - mmap_assert_locked(mm); /* Check the cache first. */ vma = vmacache_find(mm, addr); if (likely(vma)) @@ -2281,8 +2280,7 @@ struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) vmacache_update(addr, vma); return vma; } - -EXPORT_SYMBOL(find_vma); +EXPORT_SYMBOL(__find_vma); /* * Same as find_vma, but also return a pointer to the previous VMA in *pprev. From patchwork Mon Aug 29 21:25:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29860ECAAD2 for ; Mon, 29 Aug 2022 21:25:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79E786B0078; Mon, 29 Aug 2022 17:25:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 72949940008; Mon, 29 Aug 2022 17:25:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5056B6B007D; Mon, 29 Aug 2022 17:25:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 34D296B0078 for ; Mon, 29 Aug 2022 17:25:44 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EE519A258C for ; Mon, 29 Aug 2022 21:25:43 +0000 (UTC) X-FDA: 79853912166.24.8197EB9 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 9F96218005B for ; Mon, 29 Aug 2022 21:25:43 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-33ef3e5faeeso141504607b3.0 for ; Mon, 29 Aug 2022 14:25:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=VHFOj4kfUa3xGA3HxCxfqSH2fyPnkF//iB8ZL7PxHTQ=; b=XClgxH8bXaTLY5kgySHmNtvDCCVAB8mjOROUY69yCIkvhLsUcaeiTHEG390LZTJ0UJ f+bTq7v3mEXk1WZIYhpvHYfgX0MddKEDnFBtUXG7eI4mzGho1uyo+BRigftRuUZpe5ed yTJv9FwpqBMqwvSi+XVWkQi2sCAbfOGrPpu1lkrGFWjYvxB7buAqdmGdNV7OSfwNKssU SvorUO1mGFzFR5b/hO2rDfP9WSX1xgC98rCcdyQIaEqfPxTvpPYobfXt4iE+GIx2WNMy khAj4t26F+rdvp0rGETv3PRP0ykKrNZPVAZGmSPMWGLG1X6s1pkbIB5IUW2TBK/ZgFFb KwQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=VHFOj4kfUa3xGA3HxCxfqSH2fyPnkF//iB8ZL7PxHTQ=; b=SxHOn55hCWXWdY0SvlW+ilVTjmxARCw+bMCVyckdAalNkXbMALJkXrqIhhlGPuBBfp 6vmtv44vus+e8OxvAwgxo2Mlm/20XRelw8w11qv2Xuk/aNGyWofXobwaoLKu5qOtXrix 3nVI4MIC16dxvHHO18zgDnHJZud+ZEyyFzMu1fMvycD63fBabbLKAa4AsjSrFS0Xra6S DSv8DxhKss9fanazDEJ8T7Q+Zx9UO4h75kZk6ILwO7w4o6idF72clXZ4cSKbHeaHk9xZ mWRviXiZFFXrJtppNafmDUUZB11zJZ6ZSj+Zaj/hh2Dt1QiqXtJLIj6DTYBF2H/lCETJ xh2g== X-Gm-Message-State: ACgBeo1epwplxcQ+aUc17MSoSdDNoQxrOIIXJk7F3i2Y5hQsrz2XvoEF kLFS7t9nkf3yRCe2drhRv5ufOi7qyVg= X-Google-Smtp-Source: AA6agR6wyKOqZ82JQBWMvaTOKlqh8vry2Onq3V9mMTZsNJ+QDBtzKC1cVl90fnxR++4fsXM2QhKkAh66mSQ= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a0d:eac9:0:b0:33d:c025:8a74 with SMTP id t192-20020a0deac9000000b0033dc0258a74mr11722581ywe.430.1661808342911; Mon, 29 Aug 2022 14:25:42 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:07 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-5-surenb@google.com> Subject: [RFC PATCH 04/28] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808343; a=rsa-sha256; cv=none; b=I0RyavnIal3G3+tGI89Pv26Yk7rFN0BOPYScmek+WLjZc+WNwgguZCxszUAPdQyBC6d+ts 1v2w+dGEL/FvpOz9XyFFDT8lRObT1tiM5GY7Z8Lkkghw8MYlMDenYKsTQiXysLRrRPc9xJ fKaBSU7S22oTdN16PGe1EMAP/5cXRZk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XClgxH8b; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 31i4NYwYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31i4NYwYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VHFOj4kfUa3xGA3HxCxfqSH2fyPnkF//iB8ZL7PxHTQ=; b=R739L6oi7oMWr3BMVhGyuiAZJyACg17ej8AktPiqTsSSqyF/VzAo74lRs8caMuPi/fqml1 djMuSLf0tLRl9Bb1U/UsFXVGXP4UP4rRDXFgpvmpvatEkaa+e+vN8sjg/ypSqP3iWU+Je+ e/p5VUvvQBtUjfdu/sWz6e49izQK/ts= X-Stat-Signature: brt3bx8bedzc6m4g3e98gcxnqb9nk9rk X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9F96218005B Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XClgxH8b; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 31i4NYwYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31i4NYwYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com X-Rspam-User: X-HE-Tag: 1661808343-278917 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000084, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Mon Aug 29 21:25:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFAEDECAAD8 for ; Mon, 29 Aug 2022 21:25:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C515940009; Mon, 29 Aug 2022 17:25:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 775CD940008; Mon, 29 Aug 2022 17:25:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A1F7940009; Mon, 29 Aug 2022 17:25:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 430DB940008 for ; Mon, 29 Aug 2022 17:25:46 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2179CC034F for ; Mon, 29 Aug 2022 21:25:46 +0000 (UTC) X-FDA: 79853912292.12.1A3F2C4 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf06.hostedemail.com (Postfix) with ESMTP id AFCE6180037 for ; Mon, 29 Aug 2022 21:25:45 +0000 (UTC) Received: by mail-pf1-f202.google.com with SMTP id e9-20020aa78249000000b00537a62a6175so3620561pfn.20 for ; Mon, 29 Aug 2022 14:25:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=N9bIms4Zgtt1w5mfZvOohRmDSgEYydYD4bEwHw/V5ow=; b=CDYqj2724k+j3UZuEqBII6d0mYlErNKF+DONYkTH+E04Ngvdyx6CFcIzGI3JpE+Ocv IfFz52ELd/oHC5JbA4KJiqR41o93UU1oNs91Ejb3C2WUYMd8rJbvn4wGpH2TVPFpMmJT SgaAVJGTguR7jHY2klbLNlMZG17tv+P143dcZzCLTWGQf5RYnpB7E01bhfNNtgZSKJd4 8iRdR4KuZii8bgu+gGfhXIXfMfDbZfPU8zkTguml14FGrqbs//qWRwWzX5xJRRQfxW3M XOIZ22KxRY8zlSKN1mF3msZnGKAdWpIuRmlb/EMY5nhFhrrOHs7u2uNNMJtGec19XB/4 ScVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=N9bIms4Zgtt1w5mfZvOohRmDSgEYydYD4bEwHw/V5ow=; b=gQ6oA1vKR9Benqar3MXCEPAHZ2ZWw3jpRbBUptiDT8uf6GIQ7Brng4CfbzhEjUrSVE W3m2aFofJ7S+fGYY+1cK6cFtFb1TWbvlNTKa9pbqxXwcP0RmdSR9FKa2L+j9N/wphCXt eLE5R4xApETNNxCr9Hgohprtl1fbM3aX6KmmGEIFvrExabOwgEW9i6vmxNPaMIzt367M lxS7kYoeDHG4nIf4eP0nmlQfY3ySboeLZbE7CqYnO3nSjDHkXHCKw4h5U2bQOGFT/hK8 bIdK5otAmRzTjQ70/vprkdBBHxMj3ZXrCB3sw5Z9EUzktK6wgfI1lUFgsaLpfiUaNUPn 8CzQ== X-Gm-Message-State: ACgBeo1kp0Q/MekXrvHbw2G+/mPq9WkWj8iXaCYnY/NkcxXDZ9sVihHk H+VjlER0OWul3+KBI9tVxvtaaPH/CS1AVYWVwvX3Xgz76HDpJiO1MvuPAgf5BJZy8jjlhZsYid1 mOVEfFxk= X-Google-Smtp-Source: AA6agR7o7Ix1gbIIukXoy+30trWr4dJQ0Q9jpP73SljSWj/tMusjSLFwSN2Bk21+x1fzs5P+BkoH3vuJaG0= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8a0b:0:b0:537:f5f1:fd91 with SMTP id m11-20020aa78a0b000000b00537f5f1fd91mr12449109pfa.41.1661808344609; Mon, 29 Aug 2022 14:25:44 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:08 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-6-surenb@google.com> Subject: [RFC PATCH 05/28] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-ccpol: medium ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808345; a=rsa-sha256; cv=none; b=5sueAoVqxKbYjy+LTv8ctUaq3ocaVTkxuYE+yCcLAPIKHnT6AWYMT8AxXdQT359zf8UW6A qT0yTD1MPZhfS/mH/23bMjIp004LXCB564Tnow0I177xnEKUYwSZId9wlNU7fIXuZJ9LK0 7wrbtbUIJYB9F82FH5TDpUp8ZLiX2uI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CDYqj272; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 32C4NYwYKCIk574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=32C4NYwYKCIk574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808345; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=N9bIms4Zgtt1w5mfZvOohRmDSgEYydYD4bEwHw/V5ow=; b=2UUInXrxI3PHUZCWV2Zf/EgJPgFUXC9Xd2CKr8icfo/XssoAGocDST3L2gHGM1O+dfg58i Evn6vyKyck03hl4jYuej5sj7/NwBNJc6x3loxVlotldhcYsmIwZ0SDt4OYMFDNYzYPwYO1 VLKMmkejpgVpM0OQPbbSyqEsU6ErRE8= X-Rspam-User: X-Rspamd-Queue-Id: AFCE6180037 X-Rspamd-Server: rspam03 X-Stat-Signature: q5sq3hnj8hi7zbr67krs1jexsznnxd18 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CDYqj272; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 32C4NYwYKCIk574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=32C4NYwYKCIk574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com X-HE-Tag: 1661808345-254616 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a per-VMA rw_semaphore to be used during page fault handling instead of mmap_lock. Because there are cases when multiple VMAs need to be exclusively locked during VMA tree modifications, instead of the usual lock/unlock patter we mark a VMA as locked by taking per-VMA lock exclusively and setting vma->lock_seq to the current mm->lock_seq. When mmap_write_lock holder is done with all modifications and drops mmap_lock, it will increment mm->lock_seq, effectively unlocking all VMAs marked as locked. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 78 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 7 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 105 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7d322a979455..476bf936c5f0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -611,6 +611,83 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +static inline void vma_mark_locked(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +static inline bool vma_read_trylock(struct vm_area_struct *vma) +{ + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result but it's not critical. + * False unlocked result is critical but is impossible because we + * modify and check vma->vm_lock_seq under vma->lock protection and + * mm->mm_lock_seq modification invalidates all existing locks. + */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_read_unlock(struct vm_area_struct *vma) +{ + up_read(&vma->lock); +} + +static inline void vma_assert_locked(struct vm_area_struct *vma) +{ + lockdep_assert_held(&vma->lock); + VM_BUG_ON_VMA(!rwsem_is_locked(&vma->lock), vma); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline void vma_mark_locked(struct vm_area_struct *vma) {} +static inline bool vma_read_trylock(struct vm_area_struct *vma) + { return false; } +static inline void vma_read_unlock(struct vm_area_struct *vma) {} +static inline void vma_assert_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -619,6 +696,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bed25ef7c994..6a03f59c1e78 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -486,6 +486,10 @@ struct vm_area_struct { struct mempolicy *vm_policy; /* NUMA policy for the VMA */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + struct rw_semaphore lock; + int vm_lock_seq; +#endif } __randomize_layout; struct kioctx_table; @@ -567,6 +571,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..a391ae226564 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_mark_unlocked_all(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_mark_unlocked_all(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_mark_unlocked_all(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_mark_unlocked_all(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 614872438393..bfab31ecd11e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -475,6 +475,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); new->vm_next = new->vm_prev = NULL; dup_anon_vma_name(orig, new); } @@ -1130,6 +1131,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + WRITE_ONCE(mm->mm_lock_seq, 0); +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index fbe7844d0912..8399f90d631c 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Mon Aug 29 21:25:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0DFCECAAD4 for ; Mon, 29 Aug 2022 21:25:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3325594000A; Mon, 29 Aug 2022 17:25:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26C24940008; Mon, 29 Aug 2022 17:25:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F158594000A; Mon, 29 Aug 2022 17:25:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D7FBE940008 for ; Mon, 29 Aug 2022 17:25:47 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B85B4120B78 for ; Mon, 29 Aug 2022 21:25:47 +0000 (UTC) X-FDA: 79853912334.24.0E7D7DF Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 62881A0052 for ; Mon, 29 Aug 2022 21:25:47 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id q193-20020a632aca000000b0041d95d7ee81so4543182pgq.3 for ; Mon, 29 Aug 2022 14:25:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=Ymgoo1txtx46eok3IPOZQ+7oQJBGfBlI8J01hhP7hOA=; b=V7GGgx774wbENy2sL3aG99hkTB/qGZUI2IMkW5Z7Ox/AY4k1IvJaJqWcS2+hJrQTXz qJiaho12WNhOPkKsDNI4kFEEE0MJklisQUNbnL2MLe3SXi7RR3OSC49CxUHm5rWNBWOA wHnBj0rB97rbw4NvbtXfeTLWG+Vyzon45DGcHKdD67SN4QP16ppc4Mfgkavx0/TZazKT jRNtTwjBlBNTzgM29WASwCLg6OtowkNwGUZwoCyX8zuUMvza2TeNlSBp6kFeN5K2wkE2 aGwm4Y8asgOKWmz3tiq46nHGojR+kREyp0h1B2grUgz1rUK/O35dZpO1+I8qz72cYmP5 /MNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=Ymgoo1txtx46eok3IPOZQ+7oQJBGfBlI8J01hhP7hOA=; b=hrELugyDTKp+oXUcCpEOP791TbSnGjISko316hCMKnkGfpCVbMnPniOQSYPud1WwA/ PmZ7Zd+EDmithCBl+8lBWebkoseh9LULITdYO6PKJjVI9I4y+MXHw/9byli80YrfgJYG t2rnnRv5zQMIabShAq5+jOdaMVQNhPXBRT0jR5Ap87Qi4oMWt7gPcC2/YpYsJVuibZf6 ojJ2J55mtFniQ7aGd5ViQuylMA23tFsG4W3jraxdtENy9uzBwD2d/ajcXqIjKyXaA8MW /CMz4KFIJ+eABZB7pftFBHkGJBNplCaEJXdDo2QX9+5iBm5EAVXk7qnjSNl5OTaholMb opFw== X-Gm-Message-State: ACgBeo3mr5K4T3Y635QEr4t3qrk/emx5NPNsDDKqV4Z9QdFPr4zFERl7 pfZirDL54Emc+tK/1qLCrFd8wcJhILo= X-Google-Smtp-Source: AA6agR5xVu1GUsjULFDqf0en50jiRHsHcqxte0InGO36KTqnyV9YWBjqbQ+B0gx1566Kx/T+OnpxLJSWOq4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:d48a:b0:16f:c31:7005 with SMTP id c10-20020a170902d48a00b0016f0c317005mr18369268plg.173.1661808346364; Mon, 29 Aug 2022 14:25:46 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:09 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-7-surenb@google.com> Subject: [RFC PATCH 06/28] mm: mark VMA as locked whenever vma->vm_flags are modified From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=V7GGgx77; spf=pass (imf25.hostedemail.com: domain of 32i4NYwYKCIs796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=32i4NYwYKCIs796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808347; a=rsa-sha256; cv=none; b=PyWLrYM6rblFwIIKN/snjgEsQJ26EM7nJVlbI0SBdmNfBYkqxdNjI44ZvXoavF2GJp04iz 1i1QqSkzNtUGeCCVJUStZA26eOdvktniDWI+Jz7gZ03IGOZsIp1k0JLR6GJ9XfFpOBGN14 tETnxKlmzjO81alMUSSbcN7z6seGUhw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ymgoo1txtx46eok3IPOZQ+7oQJBGfBlI8J01hhP7hOA=; b=xSULwt4ZolWSeiZWteR6gns0ITlWOEjXbQp0y9hKr4Z5Gy4Z12NoDjNJ46H3RsZ1RS5zr6 vbB6zk8DAA5GWvjjEDyKNtljzZppXnAxupitsTVPpMEDhsrcdDjlh750QYgWu/4BkHZzhw wa+esSG3dolPGFz1OGpX2NwZ5156tEw= X-Rspam-User: X-Stat-Signature: gh7bmm3ch11ahfijwnc94xb1a3bnnmea X-Rspamd-Queue-Id: 62881A0052 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=V7GGgx77; spf=pass (imf25.hostedemail.com: domain of 32i4NYwYKCIs796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=32i4NYwYKCIs796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam07 X-HE-Tag: 1661808347-372267 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: VMA flag modifications should be done under VMA lock to prevent concurrent page fault handling in that area. Signed-off-by: Suren Baghdasaryan --- fs/proc/task_mmu.c | 1 + fs/userfaultfd.c | 6 ++++++ mm/madvise.c | 1 + mm/mlock.c | 2 ++ mm/mmap.c | 1 + mm/mprotect.c | 1 + 6 files changed, 12 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 4e0023643f8b..ceffa5c2c650 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1285,6 +1285,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, for (vma = mm->mmap; vma; vma = vma->vm_next) { if (!(vma->vm_flags & VM_SOFTDIRTY)) continue; + vma_mark_locked(vma); vma->vm_flags &= ~VM_SOFTDIRTY; vma_set_page_prot(vma); } diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 175de70e3adf..fe557b3d1c07 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -620,6 +620,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, mmap_write_lock(mm); for (vma = mm->mmap; vma; vma = vma->vm_next) if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; } @@ -653,6 +654,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; return 0; @@ -734,6 +736,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, atomic_inc(&ctx->mmap_changing); } else { /* Drop uffd context if remap feature not enabled */ + vma_mark_locked(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; vma->vm_flags &= ~__VM_UFFD_FLAGS; } @@ -891,6 +894,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) vma = prev; else prev = vma; + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } @@ -1449,6 +1453,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx.ctx = ctx; @@ -1630,6 +1635,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; diff --git a/mm/madvise.c b/mm/madvise.c index 5f0f0948a50e..a173f0025abd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -181,6 +181,7 @@ static int madvise_update_vma(struct vm_area_struct *vma, /* * vm_flags is protected by the mmap_lock held in write mode. */ + vma_mark_locked(vma); vma->vm_flags = new_flags; if (!vma->vm_file) { error = replace_anon_vma_name(vma, anon_name); diff --git a/mm/mlock.c b/mm/mlock.c index b14e929084cc..f62e1a4d05f2 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -380,6 +380,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; + vma_mark_locked(vma); WRITE_ONCE(vma->vm_flags, newflags); lru_add_drain(); @@ -456,6 +457,7 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ + vma_mark_locked(vma); vma->vm_flags = newflags; } else { mlock_vma_pages_range(vma, start, end, newflags); diff --git a/mm/mmap.c b/mm/mmap.c index 693e6776be39..f89c9b058105 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1818,6 +1818,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, out: perf_event_mmap(vma); + vma_mark_locked(vma); vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT); if (vm_flags & VM_LOCKED) { if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) || diff --git a/mm/mprotect.c b/mm/mprotect.c index bc6bddd156ca..df47fc21b0e4 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -621,6 +621,7 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ + vma_mark_locked(vma); vma->vm_flags = newflags; /* * We want to check manually if we can change individual PTEs writable From patchwork Mon Aug 29 21:25:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B74AECAAD8 for ; Mon, 29 Aug 2022 21:25:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDF5094000B; Mon, 29 Aug 2022 17:25:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D4117940008; Mon, 29 Aug 2022 17:25:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6EEF94000B; Mon, 29 Aug 2022 17:25:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9FD18940008 for ; Mon, 29 Aug 2022 17:25:49 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5C7E71606E1 for ; Mon, 29 Aug 2022 21:25:49 +0000 (UTC) X-FDA: 79853912418.18.7822891 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf15.hostedemail.com (Postfix) with ESMTP id 00C7AA0040 for ; Mon, 29 Aug 2022 21:25:48 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id f186-20020a636ac3000000b0042af745d56cso4590773pgc.17 for ; Mon, 29 Aug 2022 14:25:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=knlkft8CSs4ctk+Wf5EGoFcT9IapAOL4+B6Ai/+T1Ew=; b=P37RAGhPa0fj1eu5oWQ75AvAaLtugohQsYRWKzKsYPy82nTGbYS2huH/UiZNVvoE/s 88nyZ5hMCHsleX96EzwxSRzBq/7CUKc/bHbJ31Ts1GLCvmDOU9CmcQD2WQXDTk1c+h7j 3oIUBNbln4Bxomo0AKLbatKVQNVrzgPz+A4aQoGdDS5LbHfkkFmAcE6pV6uk/ymeBjbJ 1nc4oAeG1urj8/jUSameTDeH8ziLxBTZVHP9LQuagPlJAxjluMekWoC6NIdC1YJBWEbt nn2+QyaYYhDKNwmBnD8p146GgfwAnKteHIAkGMz5qwPGMxUQt1Cd0qMSAU4DAqNB0mVB aVgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=knlkft8CSs4ctk+Wf5EGoFcT9IapAOL4+B6Ai/+T1Ew=; b=tEEAwzouXbLRt+/oeivhrhSdHYtItZqhzDIzSXPj8v+pdG+tHrU5kGJgBPyUT/bM1Z u9nWTnk4e2ZZsHZsGt3FcauSNUMkwpm+ZBcrnZAXsRp7FdK3dY2/P053wk4uW/MnlZmF mkL7u/c/rd63cg/oxDG2JDcMItdjGiOjaSFKtXAT6mWG2t+Sr9zxbgWZifS2c3qZWMGV wojWnQgzzBis+HwhGC16J5c2YnSlOhzZRMUSR4tLcWzYrBrvbyobG8wEhj+jRSa8AAhQ uTED80vQFZZhvW0+yh9eH3JXJQw2ckuSGTS+KqWDqtxU33FMZVtAeK1MvEgh/FJhkZyL V3Fg== X-Gm-Message-State: ACgBeo01zVr8oFV2XMNUTFPGWkyLVYOBBLygbBdfoEzNYwWd42uiWZun h/Z5XeKjH8Ny9n/LkSUXVMU0NVGBlqc= X-Google-Smtp-Source: AA6agR7ucJtBhSAnuB+PqV9qzppZFKeEkxBUzS6um9UDd2r/XYBs5DAVo1iLh4VL9E0CKz9mlSvmLd+h5RQ= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8d08:0:b0:538:105a:eb6a with SMTP id j8-20020aa78d08000000b00538105aeb6amr10112054pfe.42.1661808348114; Mon, 29 Aug 2022 14:25:48 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:10 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-8-surenb@google.com> Subject: [RFC PATCH 07/28] kernel/fork: mark VMAs as locked before copying pages during fork From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808349; a=rsa-sha256; cv=none; b=lR6JhBknyZyN4B/3mtNyvG9Ekk1chR+L3wzWRV4SCWfPiFsddMiSpFkQd6M6ETCxvPqjkN WfjJlwolA7V8iziYNJsoCEiupj4Wi8r5My4y+hsmhnrrMIbMemIjydDSsr5tAmKmAVKHr5 YJISeRUaPBmdcLeCj3N3SC+NlDYdQ7M= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=P37RAGhP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 33C4NYwYKCI09B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=33C4NYwYKCI09B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=knlkft8CSs4ctk+Wf5EGoFcT9IapAOL4+B6Ai/+T1Ew=; b=TUsKqgIBBZ5nUH6Cha+aC/fBmsz0kjalqfE9zeVW1PZ7tOAml10fREIBvJoO7XpElOeR18 ePgCK+z6TWQWniAIF/g4bYkD+7jA1Glx7pxizIT5j0nCnS0UWKDj9MonTO8SIEmCdYilrU QCX621/+0ni1gsgnsy7Tbi3TY2aQU7U= X-Stat-Signature: yr81d4czkywh5uqf98yiiqpkia8wsz4d X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 00C7AA0040 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=P37RAGhP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 33C4NYwYKCI09B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=33C4NYwYKCI09B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com X-Rspam-User: X-HE-Tag: 1661808348-599932 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000320, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMAs from concurrent page fault handler while performing copy_page_range for VMAs having VM_WIPEONFORK flag set. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/fork.c b/kernel/fork.c index bfab31ecd11e..1872ad549fed 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -709,8 +709,10 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, rb_parent = &tmp->vm_rb; mm->map_count++; - if (!(tmp->vm_flags & VM_WIPEONFORK)) + if (!(tmp->vm_flags & VM_WIPEONFORK)) { + vma_mark_locked(mpnt); retval = copy_page_range(tmp, mpnt); + } if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open(tmp); From patchwork Mon Aug 29 21:25:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2627EECAAD2 for ; Mon, 29 Aug 2022 21:25:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94B1094000C; Mon, 29 Aug 2022 17:25:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AD7F940008; Mon, 29 Aug 2022 17:25:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6633594000C; Mon, 29 Aug 2022 17:25:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4E294940008 for ; Mon, 29 Aug 2022 17:25:51 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 220F61C64E3 for ; Mon, 29 Aug 2022 21:25:51 +0000 (UTC) X-FDA: 79853912502.27.2EA2A85 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf01.hostedemail.com (Postfix) with ESMTP id BB5FC40048 for ; Mon, 29 Aug 2022 21:25:50 +0000 (UTC) Received: by mail-pf1-f201.google.com with SMTP id dc10-20020a056a0035ca00b0053870674be9so756750pfb.12 for ; Mon, 29 Aug 2022 14:25:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=ooNZAin4dExMhxnzVFEqRWXW+87JR2sLE0fevcXKZcM=; b=I4AROHhc91xv1qcH3nCi9ZpRLfIZ0YYOknUuCx0UCVegrJDEOhp2xXx6sXe9aCLn1C FzO21jqONtvqpljlp8Pb4Y+CdSx9zRBCbsYcOU1CMA6tT/zAnvmRorbY0bmK2T02uMkI Atv+IFZLAVnvs+tVzbF09RqlClFEsd0HUKtTSkQ+YRbpf3d4jB4ZfuXvxVp5TLyaLLyx AxdLnxV0cUJcHwzUd+Lxky4P92oXj4Xhl4or9fJrlsG5GoIvzarsCtdrQ2b29IJKR80y NIBWQdZUPTZNVXZAtYrsTzYsNpAhtyZg3hqM9pNuOBs9nY5UdbnMfXBZJ7yLdA5TAp1d MxBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=ooNZAin4dExMhxnzVFEqRWXW+87JR2sLE0fevcXKZcM=; b=eOE+m1Ag9k4hA6RPiVoiq64a10DPLakK9sa/DvvVwAX6H96+kq70YhPL18gGgvYtQQ QzisQko2ZYht1LSominrNuROdeAFNiiyWWtKVaDotqSxz9io/5X8s9exZ+B8CFobIfP+ J3YDsT22/T5zuQhXrydU6oL2CAx11BlBezopFsCyEiELcOGdoDZ95Kl179mUvBpPaezN dEl93Pj67VY66FDdSn9L5NmOtrSo9so7s0YMnvPxHyfpFFeuYGgYE1OEaUKthcLF8DG1 y4ZTiMGV55IlSrBWs5MP/QvrCKyGnUzdqTR4Oi7M3wGU6X31AWaqEC2A46nvGLCPmQiU Viwg== X-Gm-Message-State: ACgBeo1b/rd8RD5Yo8c8c8TcAtCoSaPCdmrdiRrRTE8TtPNpkJg2JYJl d2PDyZAgVjZjjzOGdtDQCdjC71AMxQ0= X-Google-Smtp-Source: AA6agR6Ch+8H2P4ZiI/nyVX9BprKXIg4wSUP+mENwsodRVHrGORNiLr4bLOGgyZYxzbEnnQphTeE0ezRPcY= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:4607:b0:1fa:f794:2c0d with SMTP id w7-20020a17090a460700b001faf7942c0dmr20939608pjg.24.1661808349897; Mon, 29 Aug 2022 14:25:49 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:11 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-9-surenb@google.com> Subject: [RFC PATCH 08/28] mm/khugepaged: mark VMA as locked while collapsing a hugepage From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808350; a=rsa-sha256; cv=none; b=fnYe/FTb0AdFzQpTEN7O6L51g4Pwi/YDM/jILiIBr+JCQ8VODJudwrrSPYUJNVErSMYUk8 q/Cyk41HPiLqra49QlRmUqFs+atktzHOZimq+hdx8sdSdlym6dzUdIXLTmTR8F6oY5h2cz NK54nrnROo51UFer8zR6ez8sRjlx8HI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=I4AROHhc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 33S4NYwYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=33S4NYwYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808350; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ooNZAin4dExMhxnzVFEqRWXW+87JR2sLE0fevcXKZcM=; b=cTeZNgNuK6+bD64hK7qFeZ8CwHPyxn+OfQDHlSwJv6HrrlG2hLZnl8161jMdtPmIK8DAth 76xRykJMu9Rsk1VXUM+Qku4+tDs1S13DrIX5ojXgLckA8YmR/wsQGAR98inShBTHpzfBtJ t2KWEBAu163cKMjYoMHVJGMUTNnc5HY= Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=I4AROHhc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 33S4NYwYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=33S4NYwYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Stat-Signature: pqyzyinbz4ssnw5etqssk4mim59h59k1 X-Rspamd-Queue-Id: BB5FC40048 X-HE-Tag: 1661808350-684803 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000189, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMA from concurrent page fault handler while modifying it in collapse_huge_page. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..030680633989 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1072,6 +1072,7 @@ static void collapse_huge_page(struct mm_struct *mm, if (mm_find_pmd(mm, address) != pmd) goto out_up_write; + vma_mark_locked(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, From patchwork Mon Aug 29 21:25:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDE6BECAAD8 for ; Mon, 29 Aug 2022 21:25:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F01E94000D; Mon, 29 Aug 2022 17:25:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 552B0940008; Mon, 29 Aug 2022 17:25:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32FD794000D; Mon, 29 Aug 2022 17:25:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 16EB2940008 for ; Mon, 29 Aug 2022 17:25:53 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EC90E14030D for ; Mon, 29 Aug 2022 21:25:52 +0000 (UTC) X-FDA: 79853912544.19.D6C9F56 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf29.hostedemail.com (Postfix) with ESMTP id 981F812003D for ; Mon, 29 Aug 2022 21:25:52 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id q60-20020a17090a17c200b001fbc6ba91bbso3715361pja.4 for ; Mon, 29 Aug 2022 14:25:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=tMRmbcwOZDIFaTPxLY27tBFqvxpOSXiDeb2XTcL9iTc=; b=c5Ge838GLG1Fn9WCiifxSW0yY2J286Z3nb7CHVsgQeKl819oavnaneDjs0pHM1pBgc 6/VXxA27srsSSYsH3q/5JQwqbG7gvhz/dxlqpfYYnChp/oSAOK1144eGZRxL5H4xfU4+ gXjPirwbsBHls6abdkbd/Ru5neoY+5POLJLE67tmQyRoTq3+UExyJwB4xZoCc8lFIPHT 1JLKkVoSr3hMYFs+x98zjjRcyw2xrS1rGfAXHEq+RyZfpGyhDpyGAbYsGbpBS3qHA7OY gUezPbUinXA9UrR8RhJyAbOsAhf2VqCoI/O3T8n1+HeqEuYCNoxrQ8z3jvR4TT/rnWyA atPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=tMRmbcwOZDIFaTPxLY27tBFqvxpOSXiDeb2XTcL9iTc=; b=U8hByuy8hpHPPJizWPP5I69oDnKwGIU4nISMiwduOjMdttnzwrWFE8gnh915YKn6xh vIVMZXCINzGg1gwVy6521er0Z7i/APWnbO89Wncx3WXPgAi75otR2bjquUjBYL/o7hX1 TA+1mtuzI48BcDoKoW3HMsS5/JPtNW8QmkD+Wn1u0RbfIGXkxaZ0EQGog9+rOWY+STxJ yNTxhew577K3hgDoq1baDdTzabIUwTWE4CggC9TvJGSd9t8K48Ent4k2rTaZqR036IyK XRqHBQ06G0cikWtesN9OA7Aaz9VunooMUkq5kJLgt9rUxfLUBlhy/aTIOhwzcLvog377 b72A== X-Gm-Message-State: ACgBeo2pcfim7oMtdpLnxyZSKQe7fZ5yWNTiwM9gRf0SHY3xzsC22gLr aRFS+xrtgiUTb86zUuZLBvaZbnxduRg= X-Google-Smtp-Source: AA6agR6WFXWG6lBERn8fKgbng4pdY7+dFZ275Tti+4gL6MGpsob4z29fC0owAPSMM8vQ0mXhvHK4dQ+aJ1U= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:ab08:b0:173:320e:3316 with SMTP id ik8-20020a170902ab0800b00173320e3316mr17541232plb.38.1661808351535; Mon, 29 Aug 2022 14:25:51 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:12 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-10-surenb@google.com> Subject: [RFC PATCH 09/28] mm/mempolicy: mark VMA as locked when changing protection policy From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808352; a=rsa-sha256; cv=none; b=e0zvIzlGi3f23E1Cl++bOsaPrea5E6Xt0WwWlGwMyf2uqj9AVGkkE9xi4YsA9Ojx1ISGVN NV3nKXhfvXc/PHAWFTVM3ZHtHqx5UaczxqMAGIHBfF8kvsqcZBbFpOxQIHH2nzNIlHCtNi AvRmQZJs7WcgE7x3LzU2/fyWgInM0Zg= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=c5Ge838G; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 33y4NYwYKCJACEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=33y4NYwYKCJACEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808352; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tMRmbcwOZDIFaTPxLY27tBFqvxpOSXiDeb2XTcL9iTc=; b=KNcUPPymQljKyhX3RNOK1g3lsjJf/4bLg+IYMG0zjlvGj68s3XuZxUNBok/hJ/iB8VMFfX jHhidwGJiyrRTWuO4vGn/Daop7n1WZkon/TNxrnOWLPpJuuFQ817oGNzNMcveLVsuBB55B vj/WfrePCjarZh3BblSkvm06osbF0ho= X-Rspam-User: X-Rspamd-Queue-Id: 981F812003D X-Rspamd-Server: rspam12 X-Stat-Signature: t58apet73313hmzbs1fnsjxtbuxa7qn1 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=c5Ge838G; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 33y4NYwYKCJACEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=33y4NYwYKCJACEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com X-HE-Tag: 1661808352-407584 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000082, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMA from concurrent page fault handler while performing VMA protection policy changes. Signed-off-by: Suren Baghdasaryan --- mm/mempolicy.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index b73d3248d976..6be1e5c75556 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -383,8 +383,10 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) struct vm_area_struct *vma; mmap_write_lock(mm); - for (vma = mm->mmap; vma; vma = vma->vm_next) + for (vma = mm->mmap; vma; vma = vma->vm_next) { + vma_mark_locked(vma); mpol_rebind_policy(vma->vm_policy, new); + } mmap_write_unlock(mm); } @@ -632,6 +634,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, struct mmu_gather tlb; int nr_updated; + vma_mark_locked(vma); tlb_gather_mmu(&tlb, vma->vm_mm); nr_updated = change_protection(&tlb, vma, addr, end, PAGE_NONE, @@ -765,6 +768,7 @@ static int vma_replace_policy(struct vm_area_struct *vma, if (IS_ERR(new)) return PTR_ERR(new); + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->set_policy) { err = vma->vm_ops->set_policy(vma, new); if (err) From patchwork Mon Aug 29 21:25:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB0A0ECAAD2 for ; Mon, 29 Aug 2022 21:25:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 63F1094000E; Mon, 29 Aug 2022 17:25:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C677940008; Mon, 29 Aug 2022 17:25:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4422394000E; Mon, 29 Aug 2022 17:25:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 30ED7940008 for ; Mon, 29 Aug 2022 17:25:56 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1A0FBA046E for ; Mon, 29 Aug 2022 21:25:56 +0000 (UTC) X-FDA: 79853912712.31.AC92017 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf10.hostedemail.com (Postfix) with ESMTP id CB42BC0036 for ; Mon, 29 Aug 2022 21:25:55 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id g9-20020a17090a290900b001fd59cc2c14so3862554pjd.7 for ; Mon, 29 Aug 2022 14:25:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=nWFFKiz2JrcFJbZNC6h3WaeWKSGRlWYuGKA4mg+JUNY=; b=PCjrqm9NhycPwB3kYWPhGmirug2EAINieO21R1tkNllFgsambVeodQV2SWE+SPKnLk B35opb3MQlODsxNphVo3gt2yyCr1QpWXQKLcx1/vKzcN/4STqp6cRFJdHuKeBk+NsfTT tDOTylgaSdZ5nEY1ymyrW4L9evpwJBVsw8WgcZFUheAteEM8kYWXWY1xc4VUjrsXgS7n TvpEAgLiNbXgMlGNPVj+MFOM++dtX+BoFzGnROjLhFZB70sHOrsgtuWSCJIJNO8R1S+2 S3QUIvJ5IyCicdMqNDVbeJ892T5I6yDVEmvFVjdkZe2DnjLJIY2H8E2Qt303ElEJo4Tc HSMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=nWFFKiz2JrcFJbZNC6h3WaeWKSGRlWYuGKA4mg+JUNY=; b=G12cInsR06LBnySAJn7F7/eqZ0rqEEQxX0V+IVVHGBpxJxKTRn8x+O4ZjO8/trIIe7 iTY1A4Z1ziXSEVeBFvDlWzzZNnQq4leUsqAt0eCS8tNp2mvrclpG1BCzZt5e5mxPdQe7 AWSgDAh/83LlT0ExP9dXAIjt3hTyrbE9f6VooBIbC5PWdaVn3KNb78II6GHxEuYDDTfL 4mbc+hvorAHTTQWGEVJbLA5q7oxnMqXOr0BxtKm1XCZE88px7JWyz6qibjee1FB/1KvC 1w0NvZ7mtJBfWwKfMp4kmgDDX85F7JDbApmmMEmlLX6gMVbDwRzUcV0QJKVINzElWdVt DdEA== X-Gm-Message-State: ACgBeo3B8aBLftzWDylqr75lc3Wg1/+s5NOqRbYb6AYrpVL7ODXrpDl9 GBEoGeMKkleNzqzT84T8Mtr99hSWoNE= X-Google-Smtp-Source: AA6agR4dglPLGzQSnoqHwxEZwo0Lk0EwUGiOTYsdKQuDkqIVlV54iwLYmhw7nxi8GDICS/TsbTVUTOyC/V4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:249:b0:1e0:a8a3:3c6c with SMTP id t9-20020a17090a024900b001e0a8a33c6cmr21107pje.0.1661808353487; Mon, 29 Aug 2022 14:25:53 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:13 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-11-surenb@google.com> Subject: [RFC PATCH 10/28] mm/mmap: mark VMAs as locked in vma_adjust From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808355; a=rsa-sha256; cv=none; b=MIxY3+9oqr1YlHXgXhIvTYrk9p/RajEMVmhuFqPf4KVj6CbIRo4s4eF0DV6vfl280dCQxH t/VnmJ3hUvh5d8yHAzxFoomqIPnGfoRxaOGNIDZwTyEEdLjndhyn3SaYsuG0UZezCB5Gm1 99OKqljvUR9ctJpA0Cqzu+G5AyUETks= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=PCjrqm9N; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 34S4NYwYKCJIEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=34S4NYwYKCJIEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nWFFKiz2JrcFJbZNC6h3WaeWKSGRlWYuGKA4mg+JUNY=; b=X6+y9MSeenMGUgBcKguErg8wO4zO30m1u/VkbG9XXauc5Qi6CYYI1tZfRZ0VtCJ664ERZ3 +bnD6if1pvk94Wdp45GCow8NjS1r9lDwhjiqJQ8CXfa03Svxc9x8na8aSfxGZS0FW7m6n5 id3eeCk4y4xG2aXrBPjEq6lNjeAdATw= X-Rspam-User: X-Rspamd-Queue-Id: CB42BC0036 X-Rspamd-Server: rspam12 X-Stat-Signature: qiaoyw7i8415c6xtd8m1p7jqzc6t5649 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=PCjrqm9N; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 34S4NYwYKCJIEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=34S4NYwYKCJIEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com X-HE-Tag: 1661808355-738806 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000280, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_adjust modifies a VMA and possibly its neighbors. Mark them as locked before making the modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index f89c9b058105..ed58cf0689b2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -710,6 +710,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, long adjust_next = 0; int remove_next = 0; + vma_mark_locked(vma); + if (next) + vma_mark_locked(next); + if (next && !insert) { struct vm_area_struct *exporter = NULL, *importer = NULL; @@ -754,8 +758,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * If next doesn't have anon_vma, import from vma after * next, if the vma overlaps with it. */ - if (remove_next == 2 && !next->anon_vma) + if (remove_next == 2 && !next->anon_vma) { exporter = next->vm_next; + if (exporter) + vma_mark_locked(exporter); + } } else if (end > next->vm_start) { /* @@ -931,6 +938,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * "vma->vm_next" gap must be updated. */ next = vma->vm_next; + if (next) + vma_mark_locked(next); } else { /* * For the scope of the comment "next" and From patchwork Mon Aug 29 21:25:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D61CC0502C for ; Mon, 29 Aug 2022 21:25:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6E2B94000F; Mon, 29 Aug 2022 17:25:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF665940008; Mon, 29 Aug 2022 17:25:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9980694000F; Mon, 29 Aug 2022 17:25:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7D48D940008 for ; Mon, 29 Aug 2022 17:25:58 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 535BE1602EB for ; Mon, 29 Aug 2022 21:25:58 +0000 (UTC) X-FDA: 79853912796.16.5FD35B0 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf31.hostedemail.com (Postfix) with ESMTP id 0C07D20047 for ; Mon, 29 Aug 2022 21:25:57 +0000 (UTC) Received: by mail-pf1-f202.google.com with SMTP id o17-20020a056a0015d100b00536fc93b990so3576121pfu.14 for ; Mon, 29 Aug 2022 14:25:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=iuFA+HN3YQAhGF4lKe/pszwBtaIFpTgsToRqXaNjG7k=; b=OQFmY5p6Pe2rtWh8HoOtibHP8P5GG7l2q0OSc7HNXnd2wSmQX9hqgXeo/WRhgSbde1 3GOMVRznNWUQpWpM6oRymYxlQgfSmhokZzxHMaOud6hAHMI7hRSvRQgnpTP2D9VCRijV tvYOdRbm6MoDu4UgOmxt7sejA6CQywedGjOTRBl30ijbrDgQETYn5oZOqnGCtaAYcPZu MjsJRXAG04ApiH5Z6mU8Nc2C+hXCaIEB6zH7hUa5Up7nSwAgqcd3I/44d6H2f140ISuN zqINoaSTwBSBF/Nj+3hF7XpzbCbRiSR9QvteChaGaDG9beNmWGZas127F5+VFT8rM/v5 R54A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=iuFA+HN3YQAhGF4lKe/pszwBtaIFpTgsToRqXaNjG7k=; b=mYMPlBZ2//GskHxIR72o/QsoxVHN4bcP2oAVLIEJx+7M6Tl3xbYJG85Ndhomt7L64Z 7yxUNQS0QQ9pAfhWHzGsGnXxLs+5S9CCQG/h324pjDEb4G1EeGA7Q2UffMQpgI7rBpWh y5aLaqBzXDGepP+Ai4K+jeBdUE6/97LPHrAl+suYndMj2A9CO/kderTHoT18iBUnaWxr CBvaBkO8Y66mhUK7p3J/d9uX97v4pvN1q+i27fzRh+Q3SNDgvefbR1YKeYk0tn6tCNpM gA6jpwhP63VgSsJJC64n4AnFws/qvSZoos2mjZ3qnuxZrAL7Dwi7dmOvh2aedDiqACmL EJoA== X-Gm-Message-State: ACgBeo2OBpKhyVTOKiMFazFW1nlPD635tAhhz/64eM4SzO56ZGZPqZay lRX0q0edFvo3oEl2QiiJ0v0QQIETIqQ= X-Google-Smtp-Source: AA6agR67llFT6eQ0+3Cyyc4Z2yCAV5KwAp4HusV+mlw9qyUeJ6m9H2aBfcEzix7gu6JpzsX42cg/TQ27QPw= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:e558:b0:1fb:c4b7:1a24 with SMTP id ei24-20020a17090ae55800b001fbc4b71a24mr21023pjb.1.1661808355887; Mon, 29 Aug 2022 14:25:55 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:14 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-12-surenb@google.com> Subject: [RFC PATCH 11/28] mm/mmap: mark VMAs as locked before merging or splitting them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808358; a=rsa-sha256; cv=none; b=CrKgnAIfoopB3KqCahsY00Uy/iQfBJhevB7dU7lk0N4notR9pdk0UtknjTVuablqn/p5Ll 0WxUPJulaVF2RcVvBXyo5WFi4+/6LdqGrHn3v1ZYs99ECDWA3kAnd4bxiTVwlsp8/aEjTD i0f1j1g6Gk+V5NXq33SgxYkV/GDmhvg= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=OQFmY5p6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf31.hostedemail.com: domain of 34y4NYwYKCJQGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=34y4NYwYKCJQGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808358; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iuFA+HN3YQAhGF4lKe/pszwBtaIFpTgsToRqXaNjG7k=; b=APBvCaQHdRcQ+6+i21/Iip71geaM5Cq2Ga0BzOUnvt+K7jJxS5SVM5/0yIuJPl4t58SiY8 +Hymz3XNZVB6hCXyuH/w17MTMl5pwlWC6Z/l+kcTKF80a+lUXSNx5qmnccLuZqfWSDyfN4 eAZSfI9dommpCIEIa4ibkJRn8R7ucDY= X-Rspam-User: X-Rspamd-Queue-Id: 0C07D20047 X-Rspamd-Server: rspam12 X-Stat-Signature: gozryitnyxah88r6cfgx9dc9y1zs6ak9 Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=OQFmY5p6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf31.hostedemail.com: domain of 34y4NYwYKCJQGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=34y4NYwYKCJQGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com X-HE-Tag: 1661808357-248361 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Decisions about whether VMAs can be merged or split must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Mark all VMAs which might be affected by a merge or split operation as locked before making decision how such operations should be performed. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index ed58cf0689b2..ade3909c89b4 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1147,10 +1147,17 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_mark_locked(prev); next = vma_next(mm, prev); area = next; - if (area && area->vm_end == end) /* cases 6, 7, 8 */ + if (area) + vma_mark_locked(area); + if (area && area->vm_end == end) { /* cases 6, 7, 8 */ next = next->vm_next; + if (next) + vma_mark_locked(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2687,6 +2694,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *new; int err; + vma_mark_locked(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) From patchwork Mon Aug 29 21:25:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8677FECAAD2 for ; Mon, 29 Aug 2022 21:26:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C314E940010; Mon, 29 Aug 2022 17:25:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB9FA940008; Mon, 29 Aug 2022 17:25:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 922F4940010; Mon, 29 Aug 2022 17:25:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 732FB940008 for ; Mon, 29 Aug 2022 17:25:59 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4C84614030D for ; Mon, 29 Aug 2022 21:25:59 +0000 (UTC) X-FDA: 79853912838.13.EA2909A Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 0D829C0036 for ; Mon, 29 Aug 2022 21:25:58 +0000 (UTC) Received: by mail-pl1-f202.google.com with SMTP id b9-20020a170902d50900b0016f0342a417so6866713plg.21 for ; Mon, 29 Aug 2022 14:25:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=TWnM2nl+IWV2Xsj5xnZFMNMq0JvYeydCw4P9njL7O34=; b=QxyTYU3FERVCw2VziTt0AktgHLl//+1LB6ej8pZ2s1TWfQT3cvsAWfnuU7caTHeT9p cUZ3ty459UM3cwcXwNVqpzu7fZxDdvEiQ7oppDRkd/R2+Ilj86/8iy+0q3ZJXoZ4hnye zLhHEgIO/fbjIHEiSfoXxsUyX6O+Z8dic5XVruk2KEpor5fjgGmf4uNln1+WOsbYbTO8 1++i5QRJ3Dh2yxyVyOmY5Dm8Y1A3ze3Xt4uhd4NBeKcnHbK+ENDvs3AO4SaOBTp9Wi+f EGkkrDdrwisqruuTTVNnHMbKcZefZFlW3jnAC7mX9edrYOzqBxLpV3pgyWig/2c0mbAr sZlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=TWnM2nl+IWV2Xsj5xnZFMNMq0JvYeydCw4P9njL7O34=; b=g9UeQKViw6kpNOjKgY5SJBdrEesNZfNY0Ik7cPhSNt7zJCQcO7RbnjF84mPSY7dGCj o5TMicqj5x2LdF1vqBkswCMoucjvNJyf1TS0dPzz16R/ypUCRn1zo+CzS0HR94SQtJ1+ pNPv1IoDba3SfBeO2AzzhnnPe99dorAf0IndICgyrzzA3MTG2sF1VfFH2PzNEF65B8Ho nU0mbJrXDv40lVCx9wqNE+/UtOIQ+WjDshfSIUPbj7nEekHuJCZyScLLAHa9Q0o8EmpA xPs8jtZKvVjsPHr+DfgwquSsx031Ce9yl6H5g4hbFskssiqLzy0vzr/0sq7e/KZcT4zc rD1g== X-Gm-Message-State: ACgBeo2KYkDFTRYTVBjIdEv0ebt2KUnRCTV3cSLWfXMvPk4aC+arZb+g wCTcsYvuHyshwh/84IlKzlzLW0TKSV8= X-Google-Smtp-Source: AA6agR76cA5Bw6pPdfILXZxKwAftwtVJj++YaWYUCk6VRMlDSa2HJ+OTsxxi3Jretb4HksKEr9YwTk9TbU4= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:18b:b0:16e:f09c:919b with SMTP id z11-20020a170903018b00b0016ef09c919bmr18604991plg.135.1661808358173; Mon, 29 Aug 2022 14:25:58 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:15 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-13-surenb@google.com> Subject: [RFC PATCH 12/28] mm/mremap: mark VMA as locked while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808359; a=rsa-sha256; cv=none; b=KGUpJ62L7aN0T2j+5BWc5lDHLniO3I5AlzDq6w87rFwEo5O9OsvxP2VJ69rFgeBeSy7p4Q ZCMndvfnz2gxStP3PJhzPtR8u8nXN2YMNOJ4KVyRbW+rS2W5gqVQhbQEjLxJFZTz8Bm4ff sUepGl8HZjd61QOe2usz+BSApPgLW7I= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QxyTYU3F; spf=pass (imf10.hostedemail.com: domain of 35i4NYwYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=35i4NYwYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808359; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TWnM2nl+IWV2Xsj5xnZFMNMq0JvYeydCw4P9njL7O34=; b=stoh82rsFW0cnqTLOaSalm59MD0WorVsHFoZGPELw1RNz/SfcU4ZmgD/H6Xhs8DNlMct2W CerZEBmP/GzSgPs/1zO/WuUzZyPuCI5mWCd69TM7XiMulyayKsUxnB+Tv+CFoM//7G7kmD vsN9hwumNLhx7auVGvz7NVjDlpj5Ej0= Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QxyTYU3F; spf=pass (imf10.hostedemail.com: domain of 35i4NYwYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=35i4NYwYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-Stat-Signature: 1gxmntzgi3477wmcuwpjz8micet7crus X-Rspamd-Queue-Id: 0D829C0036 X-Rspam-User: X-HE-Tag: 1661808358-671932 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000039, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mark VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ade3909c89b4..121544fd90de 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3248,6 +3248,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_mark_locked(new_vma); vma_link(mm, new_vma, prev, rb_link, rb_parent); *need_rmap_locks = false; } diff --git a/mm/mremap.c b/mm/mremap.c index b522cd0259a0..bdbf96254e43 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -620,6 +620,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_mark_locked(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Mon Aug 29 21:25:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5ADB9C0502C for ; Mon, 29 Aug 2022 21:26:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 863A8940011; Mon, 29 Aug 2022 17:26:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C1FC940008; Mon, 29 Aug 2022 17:26:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57903940011; Mon, 29 Aug 2022 17:26:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 40984940008 for ; Mon, 29 Aug 2022 17:26:01 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2287F1C64E3 for ; Mon, 29 Aug 2022 21:26:01 +0000 (UTC) X-FDA: 79853912922.02.9D828E1 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf19.hostedemail.com (Postfix) with ESMTP id C98F11A002E for ; Mon, 29 Aug 2022 21:26:00 +0000 (UTC) Received: by mail-pf1-f201.google.com with SMTP id z36-20020a056a001da400b005386b23cf15so876144pfw.9 for ; Mon, 29 Aug 2022 14:26:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=i5dTvRyWQzPS3ycEVnnedNfB+iHCFx9RUaZKzBMQ1fM=; b=OIPy78dmVL2e9cDtVRNWLABLgvKsSbZL8TpHXpGFOXD1iGGPPAnyGMcNCguJqgzcNF vE+9cNpICzTMp48ey6JmMRyIlEctkJFDoL9MrqQuHHLDpAc2aXZC9NHJQw5sThX00eJ4 cjI0k3jAy5gFEX5VEHv4cVfUWBvRtfOC3iZWstqSQy3ibplhtgaSEpWxPcs9EVcm88qg LMXuGuQRDlQwRuQ7YNFSCH+8XULwWBK9BYSC5nbMXIDpOIovOb1rbCvinLUjcMpntl6K B3Pgew8zey/TtmyxMcXnTuT0BjzA6VIn4rP9dbnKnx4PgoHjYUplxPVEMwf3o1trOT0a 21qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=i5dTvRyWQzPS3ycEVnnedNfB+iHCFx9RUaZKzBMQ1fM=; b=6n2sbNi+EX5N7HeJDdXjmw3W3c/oykcVU0tE+OcmjgalWnFDPR1gooRV4ZF+9cMTQF najGyPLPXRCUJNF+r9a8yPCRPyQmFYf3i8OSxF9bJ8eM0zCL8Ua+DZo0TpkW3JDFSfDQ llSQaUHZ+mc8sTgX6/HKjiTd4+TDfEdvh/nsAkUEnDKJ2x1HgBW9ND+RF3EKGsONx5jV cDx4ToLm+oG5CVj3ssh6A9DK3gqN5rTOcRrZA0VcKhXdRJG/YqySrJBzLi3r82gxwC4r cQZYbvtd+VyGgQKBUtXsd4XsClWloHcWRD7moOBt4W/3+L0yPGOC8LZwKk0jIPdvWfHo CeuA== X-Gm-Message-State: ACgBeo2/jbm//yoQ5HPsd9cA4IcjMmpBgNbY3LvKOhQloBqUKa8hvWtK SJHeGXL7WguyNxj9sf33EMu5KdhbVyU= X-Google-Smtp-Source: AA6agR4q+ezvBgIOmovwk/KulEVJwqDq2XJXfw+za6wVj0A/SjlqYDTJGsWkQl5JJSuDpwxpjjn7kfBBqpA= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:76c5:b0:174:e586:f6e2 with SMTP id j5-20020a17090276c500b00174e586f6e2mr4373889plt.37.1661808359839; Mon, 29 Aug 2022 14:25:59 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:16 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-14-surenb@google.com> Subject: [RFC PATCH 13/28] mm: conditionally mark VMA as locked in free_pgtables and unmap_page_range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808360; a=rsa-sha256; cv=none; b=UBmlnXqLdooDLIu5DcRbheEJQCOMX3DVDE+7pCGiIul6nXBl6vack6/FIL3FOVxbzOSSYy tVdtn94R8lnKPB9fV1ITGiFyLQH4DNFbI+wKZf1YejKpUQfz1y7Na0yKlnYp20/oxK5svX S0L2WjheIA3weLJHF0Bo8NlRYHPpyxQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=OIPy78dm; spf=pass (imf19.hostedemail.com: domain of 35y4NYwYKCJgKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=35y4NYwYKCJgKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808360; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i5dTvRyWQzPS3ycEVnnedNfB+iHCFx9RUaZKzBMQ1fM=; b=pcFjWXbLkl2NK9DuPXvL6Od91/+XjFzlzNNMqCObdR7pVfsTCXD0pSSIN9ayTuFoifQihh eC9CofUrUdkpW2gd+CRRKRGNjFBBWIc58wAiL7CfHuR3PadYbKHQwVsUx4Lntak2W6tRA5 9bYKZoF+fJ13D62h8YZBvlue2r7LWBc= Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=OIPy78dm; spf=pass (imf19.hostedemail.com: domain of 35y4NYwYKCJgKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=35y4NYwYKCJgKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-Stat-Signature: kiru98aai4ixufbyjdkh65i6oahdtaxw X-Rspamd-Queue-Id: C98F11A002E X-Rspam-User: X-HE-Tag: 1661808360-767450 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pgtables and unmap_page_range functions can be called with mmap_lock held for write (e.g. in mmap_region), held for read (e.g in madvise_pageout) or not held at all (e.g in madvise_remove might drop mmap_lock before calling vfs_fallocate, which ends up calling unmap_page_range). Provide free_pgtables and unmap_page_range with additional argument indicating whether to mark the VMA as locked or not based on the usage. The parameter is set based on whether mmap_lock is held in write mode during the call. This ensures no change in behavior between mmap_lock and per-vma locks. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 +- mm/internal.h | 4 ++-- mm/memory.c | 32 +++++++++++++++++++++----------- mm/mmap.c | 17 +++++++++-------- mm/oom_kill.c | 3 ++- 5 files changed, 35 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 476bf936c5f0..dc72be923e5b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1874,7 +1874,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, void zap_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size); void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); struct mmu_notifier_range; diff --git a/mm/internal.h b/mm/internal.h index 785409805ed7..e6c0f999e0cb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -85,14 +85,14 @@ bool __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma, - unsigned long floor, unsigned long ceiling); + unsigned long floor, unsigned long ceiling, bool lock_vma); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details); + struct zap_details *details, bool lock_vma); void page_cache_ra_order(struct readahead_control *, struct file_ra_state *, unsigned int order); diff --git a/mm/memory.c b/mm/memory.c index 4ba73f5aa8bb..9ac9944e8c62 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -403,7 +403,7 @@ void free_pgd_range(struct mmu_gather *tlb, } void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, - unsigned long floor, unsigned long ceiling) + unsigned long floor, unsigned long ceiling, bool lock_vma) { while (vma) { struct vm_area_struct *next = vma->vm_next; @@ -413,6 +413,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -427,6 +429,8 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, && !is_vm_hugetlb_page(next)) { vma = next; next = vma->vm_next; + if (lock_vma) + vma_mark_locked(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } @@ -1631,12 +1635,16 @@ static inline unsigned long zap_p4d_range(struct mmu_gather *tlb, void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long addr, unsigned long end, - struct zap_details *details) + struct zap_details *details, + bool lock_vma) { pgd_t *pgd; unsigned long next; BUG_ON(addr >= end); + if (lock_vma) + vma_mark_locked(vma); + tlb_start_vma(tlb, vma); pgd = pgd_offset(vma->vm_mm, addr); do { @@ -1652,7 +1660,7 @@ void unmap_page_range(struct mmu_gather *tlb, static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, - struct zap_details *details) + struct zap_details *details, bool lock_vma) { unsigned long start = max(vma->vm_start, start_addr); unsigned long end; @@ -1691,7 +1699,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, i_mmap_unlock_write(vma->vm_file->f_mapping); } } else - unmap_page_range(tlb, vma, start, end, details); + unmap_page_range(tlb, vma, start, end, details, lock_vma); } } @@ -1715,7 +1723,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, - unsigned long end_addr) + unsigned long end_addr, bool lock_vma) { struct mmu_notifier_range range; struct zap_details details = { @@ -1728,7 +1736,8 @@ void unmap_vmas(struct mmu_gather *tlb, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) - unmap_single_vma(tlb, vma, start_addr, end_addr, &details); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details, + lock_vma); mmu_notifier_invalidate_range_end(&range); } @@ -1753,7 +1762,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); for ( ; vma && vma->vm_start < range.end; vma = vma->vm_next) - unmap_single_vma(&tlb, vma, start, range.end, NULL); + unmap_single_vma(&tlb, vma, start, range.end, NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1768,7 +1777,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, * The range must fit into one VMA. */ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, - unsigned long size, struct zap_details *details) + unsigned long size, struct zap_details *details, bool lock_vma) { struct mmu_notifier_range range; struct mmu_gather tlb; @@ -1779,7 +1788,7 @@ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long addr tlb_gather_mmu(&tlb, vma->vm_mm); update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); - unmap_single_vma(&tlb, vma, address, range.end, details); + unmap_single_vma(&tlb, vma, address, range.end, details, lock_vma); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -1802,7 +1811,7 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, !(vma->vm_flags & VM_PFNMAP)) return; - zap_page_range_single(vma, address, size, NULL); + zap_page_range_single(vma, address, size, NULL, true); } EXPORT_SYMBOL_GPL(zap_vma_ptes); @@ -3483,7 +3492,8 @@ static void unmap_mapping_range_vma(struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct zap_details *details) { - zap_page_range_single(vma, start_addr, end_addr - start_addr, details); + zap_page_range_single(vma, start_addr, end_addr - start_addr, details, + false); } static inline void unmap_mapping_range_tree(struct rb_root_cached *root, diff --git a/mm/mmap.c b/mm/mmap.c index 121544fd90de..094678b4434b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -79,7 +79,7 @@ core_param(ignore_rlimit_data, ignore_rlimit_data, bool, 0644); static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, bool lock_vma); static pgprot_t vm_pgprot_modify(pgprot_t oldprot, unsigned long vm_flags) { @@ -1866,7 +1866,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_file = NULL; /* Undo any partial mapping done by a device driver. */ - unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end); + unmap_region(mm, vma, prev, vma->vm_start, vma->vm_end, true); if (vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); free_vma: @@ -2626,7 +2626,7 @@ static void remove_vma_list(struct mm_struct *mm, struct vm_area_struct *vma) */ static void unmap_region(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { struct vm_area_struct *next = vma_next(mm, prev); struct mmu_gather tlb; @@ -2634,9 +2634,10 @@ static void unmap_region(struct mm_struct *mm, lru_add_drain(); tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); - unmap_vmas(&tlb, vma, start, end); + unmap_vmas(&tlb, vma, start, end, lock_vma); free_pgtables(&tlb, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + lock_vma); tlb_finish_mmu(&tlb); } @@ -2849,7 +2850,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len, if (downgrade) mmap_write_downgrade(mm); - unmap_region(mm, vma, prev, start, end); + unmap_region(mm, vma, prev, start, end, !downgrade); /* Fix up all other VM information */ remove_vma_list(mm, vma); @@ -3129,8 +3130,8 @@ void exit_mmap(struct mm_struct *mm) tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use -1 here to ensure all VMAs in the mm are unmapped */ - unmap_vmas(&tlb, vma, 0, -1); - free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); + unmap_vmas(&tlb, vma, 0, -1, true); + free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* Walk the list again, actually closing and freeing it. */ diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 3c6cf9e3cd66..6ffa7c511aa3 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -549,7 +549,8 @@ bool __oom_reap_task_mm(struct mm_struct *mm) ret = false; continue; } - unmap_page_range(&tlb, vma, range.start, range.end, NULL); + unmap_page_range(&tlb, vma, range.start, range.end, + NULL, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } From patchwork Mon Aug 29 21:25:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F6C1ECAAD2 for ; Mon, 29 Aug 2022 21:26:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B8B0940012; Mon, 29 Aug 2022 17:26:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CBAB940008; Mon, 29 Aug 2022 17:26:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21D92940012; Mon, 29 Aug 2022 17:26:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EF770940008 for ; Mon, 29 Aug 2022 17:26:02 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C807414030D for ; Mon, 29 Aug 2022 21:26:02 +0000 (UTC) X-FDA: 79853912964.30.45A13DC Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 833884003B for ; Mon, 29 Aug 2022 21:26:02 +0000 (UTC) Received: by mail-pl1-f202.google.com with SMTP id q6-20020a17090311c600b0017266460b8fso6784600plh.4 for ; Mon, 29 Aug 2022 14:26:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=J1u/YkZ7r4U6qP3j3HXey3MGQlv66sCtBUAUQwGtkmo=; b=bssUgHF5w47sbr7l7KTrCUB9BNCgN6M5weE9wkVgXyWLQuTUzyGpg0OJvcyz0sW5UD jBcbuP4XXvP/J4pS6AqtYRXplk3ioZfRWOizdHfYubwqdaFV7iqu+wCAfN30wOXuSJR5 rOeSjLAnMTV/UMrEb+PillTsGibQBZHi92r8Zd7PRmmwTLk5upCdMt9/ApQU+Kr/zUNy g3HC4+2oFiondWWpmL/CQDHhjiEfoJFEyDpo5hDjW6J9yE1A4zjv6LFdAo5+gh9e4xsA 3B8ihRdAo9AsX2EZ1zty6Setu8LqB98WdsMEL3dnRqIhFnd/9nqnY8Jcva4fprwPuNko 5pyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=J1u/YkZ7r4U6qP3j3HXey3MGQlv66sCtBUAUQwGtkmo=; b=yHtQuQuqRcqlJ9J2CHR3OZ7dxeQQ8SB0Oi9aOIQ5eA+vk+N8Z+cJWbtAFoF0EGpLMk UbrdMf0eCu85AvsL8znrkccQZ8u8hMYfLX7opVErbv/UpVOav9XD92BzYy4wNRURK6xx BCC5j3rN35k05p6G6JrggXXlnlAUeA6BEE6oaRts3HRHnHflCNsMHW8xs2jdMZIqufTs J2lvp0rCVrGvEOWfGFx/cxgmmj2ceWY9FUuIlPlc0uS/ZwPpvIY3wVaKj0n5vXmp6g+J WJb4haXjtLX0JbmxskyIucjFO8/SnueIocF1C/Gf2I7mE+pgqjJSxQ1JdsH3YEFIBZwQ gGLg== X-Gm-Message-State: ACgBeo1prNRMtVo6mAQk40BCTcUFAx2TdF5AQ3of3D33ztGOwkOABlF+ oUjaPpQAafHM0bM2SOvcoOcV2QowPSA= X-Google-Smtp-Source: AA6agR7FrxHuf3ebSGip6Dsrc2j7pCNXLJVmg+GJ9+F6P4LGjbGDMIiC+XxX5XFr/bvFDyLyzfk2y9GLnRc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:248:b0:172:7520:db07 with SMTP id j8-20020a170903024800b001727520db07mr18052271plh.76.1661808361566; Mon, 29 Aug 2022 14:26:01 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:17 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-15-surenb@google.com> Subject: [RFC PATCH 14/28] mm: mark VMAs as locked before isolating them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808362; a=rsa-sha256; cv=none; b=YaepYkN2KTgAdfl6sDThOBOZ1ixAG3cvc4+ig5QEXdEn+lE9ZpbpYbFwFgTZiFaWo2nV6Z wjZO4hJtzXlhRMuOV8xIKSyedtopiLYMqjPHEoovTZLF0nFe6trjwGSUA+D53BlfhT7Kl8 Nuv7SvmwdGtAOg+aUnmzELOXe4wfCZg= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=bssUgHF5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 36S4NYwYKCJoMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=36S4NYwYKCJoMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808362; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J1u/YkZ7r4U6qP3j3HXey3MGQlv66sCtBUAUQwGtkmo=; b=mVpEdyfLMU7nNBliCZh4Wvb6TF/uCGkZSypl+QQGMau+VJubD80Up1u9Cb4Cnq9D2E3gQB /4z4iC9wazug//DLt793bmT3Y7rGxoCa5BF3pjBaBvkcGkGZhbKq45vOx+mXspo9OiMokw 4Ba7OWz+Bk2OXyshLucnM5V+0KF2TIs= X-Rspam-User: X-Rspamd-Queue-Id: 833884003B X-Rspamd-Server: rspam12 X-Stat-Signature: mdcgdekuqwtdpp4tpsu5i8ddc8bzuomr Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=bssUgHF5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 36S4NYwYKCJoMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=36S4NYwYKCJoMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com X-HE-Tag: 1661808362-681557 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000008, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mark VMAs as locked before isolating them and clear their tree node so that isolated VMAs are easily identifiable. In the later patches page fault handlers will try locking the found VMA and will check whether the VMA was isolated. Locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ mm/nommu.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 094678b4434b..b0d78bdc0de0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -421,12 +421,14 @@ static inline void vma_rb_insert(struct vm_area_struct *vma, static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *root) { + vma_mark_locked(vma); /* * Note rb_erase_augmented is a fairly large inline function, * so make sure we instantiate it only once with our desired * augmented rbtree callbacks. */ rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks); + RB_CLEAR_NODE(&vma->vm_rb); } static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma, diff --git a/mm/nommu.c b/mm/nommu.c index e819cbc21b39..ff9933e57501 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -622,6 +622,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma) struct mm_struct *mm = vma->vm_mm; struct task_struct *curr = current; + vma_mark_locked(vma); mm->map_count--; for (i = 0; i < VMACACHE_SIZE; i++) { /* if the vma is cached, invalidate the entire cache */ @@ -644,6 +645,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma) /* remove from the MM's tree and list */ rb_erase(&vma->vm_rb, &mm->mm_rb); + RB_CLEAR_NODE(&vma->vm_rb); __vma_unlink_list(mm, vma); } From patchwork Mon Aug 29 21:25:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48213ECAAD4 for ; Mon, 29 Aug 2022 21:26:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5BFC940013; Mon, 29 Aug 2022 17:26:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE296940008; Mon, 29 Aug 2022 17:26:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A833D940013; Mon, 29 Aug 2022 17:26:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 83655940008 for ; Mon, 29 Aug 2022 17:26:04 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 60D0E1A0A64 for ; Mon, 29 Aug 2022 21:26:04 +0000 (UTC) X-FDA: 79853913048.02.9AC6227 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 27707C005C for ; Mon, 29 Aug 2022 21:26:03 +0000 (UTC) Received: by mail-pl1-f201.google.com with SMTP id y9-20020a17090322c900b00174c881abaeso2531392plg.6 for ; Mon, 29 Aug 2022 14:26:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=KQbM9RENeAKpMamE0B+V4MUsqhzbhGXDlXXjmQQUA6A=; b=B9JYMkX5xydBJOgcGzuYw+brQarCWmRWS+0/Aqinb/OY26MbR1xT5axtOfpYrKlmA6 qGVi7LP6LaJMDcQJmEECw3J6IqaSMiez5M7VUbq2lZEZKkVSQHjZtuqy9g7s10YpHljB L9LJY/vXV43u8aIw7/j+LcG51+Bdx5jgIt7f2hLczg1JgYzlwTyYBe/lmm+nmwMuVZWc thodE+2BI084oTXFC0MD73T0ggmHuAL+ik6FuitgxrIkPY+yM+/axTrbJYGoWGabXb1/ atpqGKZe5bOiVGcNY/fR8gQx2/9qiwRjzQjgpRwLojxFdTcGDVT+d94c/I/T/XPhcGEF rEbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=KQbM9RENeAKpMamE0B+V4MUsqhzbhGXDlXXjmQQUA6A=; b=kh8LfgVptghICoWYvyxVLAfGY8m//9z82hy8ItZolDOjq7gj9IIpXbIniBidpbUIW9 gYG2Iw48T1dqZV04HD8rpMa3s2K8b3MnSO5qXmXcOMGqkSTcAgg0bOD7B4Z0Qd7Z2AmS Uc/zwQkktPK9wNhPuukU8RHAB0bmNUIDT+ZalNcADD95xLYunoDP9mOszmGoUQzcjK7U /r+14Ucl2lVn7/hhboNBIT5ZCJmhuJg0nlw6x40SZ8mPvbPhxbcQodWzE0/q9/9VXWkm PV/6hIDC8IEBik4F/Kew6RLYvx8EgQsawrE4ad0WcqbjH22XVkNoJYuNIcHWJNJqZ5gf VQ4A== X-Gm-Message-State: ACgBeo36ioFK/YUn4hFGYMLIQB0gFKh8f+QhjmB/el4BxE0dI8L0MJzI rldExyb0WbtzfKM/qG5zoUG0j6NM42U= X-Google-Smtp-Source: AA6agR7f4s+E2TUUdzlInwBIWNCTTjaS2dHpmImL6O3kfzXJ6rCAm5H+xn8nFf0AEemGeSJLYPdhv1d/+n8= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a05:6a00:b8a:b0:537:f81:203a with SMTP id g10-20020a056a000b8a00b005370f81203amr18341111pfj.80.1661808363176; Mon, 29 Aug 2022 14:26:03 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:18 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-16-surenb@google.com> Subject: [RFC PATCH 15/28] mm/mmap: mark adjacent VMAs as locked if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808364; a=rsa-sha256; cv=none; b=En34NEhR0K12vfiiILmQaAlYl/oXu6PI7Z5iPud8+hBlUXEJWX7BCMP7Gd1qaNgemQkHR8 5MSOsfkU5j6oRus6Inf+7fw1cH8N3tsr4DW6Nr8YPZx3fBfkf0sxieCMxOdVXdJ4MHPWpW 0Hb6Tf7uuWLT87sKi6uYufiIzcAkgbU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=B9JYMkX5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 36y4NYwYKCJwOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=36y4NYwYKCJwOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808364; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KQbM9RENeAKpMamE0B+V4MUsqhzbhGXDlXXjmQQUA6A=; b=gQ6EGj7Qn6Hoeu+1fYAvRIkbJzZV5VAu8GIvBCoe8dZiqPnzG5UkjQjndMOQliCA72CXOO XcKq18Dd4PmUkKznTO3HHkjg9d4L/jGZ7LvHCeWZ7b8Z/m2ySbgyC5oylZWU6gRpDgre7m 77F5gTKKdUYT272EV+eUVWwVVStwRtI= Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=B9JYMkX5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 36y4NYwYKCJwOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=36y4NYwYKCJwOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Stat-Signature: 9m8xqzpqj8c87w815oo46yqw3n74u8dm X-Rspamd-Queue-Id: 27707C005C X-HE-Tag: 1661808363-765975 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000275, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases mark adjacent VMAs as locked to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index b0d78bdc0de0..b31cc97c2803 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2680,10 +2680,14 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma, * VM_GROWSUP VMA. Such VMAs can change their size under * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ - if (vma && (vma->vm_flags & VM_GROWSDOWN)) + if (vma && (vma->vm_flags & VM_GROWSDOWN)) { + vma_mark_locked(vma); return false; - if (prev && (prev->vm_flags & VM_GROWSUP)) + } + if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_mark_locked(prev); return false; + } return true; } From patchwork Mon Aug 29 21:25:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2536FECAAD8 for ; Mon, 29 Aug 2022 21:26:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66833940014; Mon, 29 Aug 2022 17:26:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CD25940008; Mon, 29 Aug 2022 17:26:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F3DA940014; Mon, 29 Aug 2022 17:26:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1C512940008 for ; Mon, 29 Aug 2022 17:26:06 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0108FA04E0 for ; Mon, 29 Aug 2022 21:26:05 +0000 (UTC) X-FDA: 79853913132.13.2B189AE Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf22.hostedemail.com (Postfix) with ESMTP id B0934C004A for ; Mon, 29 Aug 2022 21:26:05 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-337ed9110c2so147604757b3.15 for ; Mon, 29 Aug 2022 14:26:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=NYe8oOJNfY11QXfRhGHbzq3PtP5qYF5OBWlNRgI/ICw=; b=gO3BAu2tPnt5ajc2wonv12EyC/cky1iEagPrWs8TIsmrfOPEdnZICXVlt39nBHt3WW tTV55lawhiZGMRBmA7gdT3YGawg8JjnJFEe5+7PrxS+L7PuBHZcRqu/eGZS5+7HkD+/v WBe3l/+DgKFH8adTgdOoThkCyU3oUkPdhTm9bGePyXaQUn5Kf04YxKMvEOmZ7WCBWmLN BXoUJ7D1rnsd47mFF47xSWH/cS8u1UFuDHVuHAEZYgUQ7/tjgR1JcWJgnN0k47FfJT+A /f4moE4fkIQHN9Nb1qOXqxym1C/DCmqaA6CVw+YBif80n7h7hRdtT5oL02KCkdJe7ned iBKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=NYe8oOJNfY11QXfRhGHbzq3PtP5qYF5OBWlNRgI/ICw=; b=Y6Mmkj57eYL4GAyaRHmZZE37TtVytWwTFS7LNQ0UUZmLj7A61qBMTrK6kmSQtz+4R5 +c/EaWqzhfMm9PQqwWB6pK7IlztOb5tpJFpj/Dt8BXVYDHsVyleaFF8XTHlljdrAMTc0 3RNMR0jmLUiQDdlxA5NZL/sDs7xkCABbPJgidh3wgOtfUqu0Mqa9si6c+Wp9TDpUguhK d9lQ+2L2NkI5IWYuwNu9KEc9UQbHOlnwd+gXDO6gypi3epFnHDO2h6EwvUkLV1eyFzpg 1NYtM6r5oHN3G9Xu/W8oqLx1WJwuLFZ89Kcy6gQLyLpG2fmviV1esQGTgrTvlfnTgbnl fF0Q== X-Gm-Message-State: ACgBeo12i/cVfCMxHZxbZrKbmk1wF7oMZawWiB4QeecwaHfZn4PRjlaI b3GgWwr8cA4Rfx+ABkt7EoL7c4GFI3Q= X-Google-Smtp-Source: AA6agR7nmkBRROpS79vTx3reOak4mrTZffxlmMbexsEub45N/wO5CnqJNjif+IJpyxwlCbKERnK2QUttv8k= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a25:d944:0:b0:695:6056:7dda with SMTP id q65-20020a25d944000000b0069560567ddamr9840482ybg.277.1661808364958; Mon, 29 Aug 2022 14:26:04 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:19 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-17-surenb@google.com> Subject: [RFC PATCH 16/28] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808365; a=rsa-sha256; cv=none; b=rZt2QuII6ivup4muqpkNBMqkXt5Sn/7QhU6YtKgdsw7nPVkb0O9mUY78WsyIP0A8xvSVOD wgeTOCpA/laXrjw+x7ho/DPVEKLoiK7sFlCal1MlWjCQBRA6NQygqosQrA1CkMt3MY3PV8 keoS18mliDJN6vw0bNJ8ogntuY9Oumc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gO3BAu2t; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 37C4NYwYKCJ0PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=37C4NYwYKCJ0PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808365; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NYe8oOJNfY11QXfRhGHbzq3PtP5qYF5OBWlNRgI/ICw=; b=VyJ0QF06z76YKh2B8BTW47PRbIIPd13spOMmxcI/+fvyOMiwnRMPHOGNZATdXPOVash/Qb fpuqmIhqzCmxAEAZfOWSVCdPtlDS2nso5K/vXwnUSXdJjdYIYj1cbIwjxVuQm6XOPRFlGf fiitwnvCu7rlpmEQtiSfxe0EQvoodzs= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gO3BAu2t; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 37C4NYwYKCJ0PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=37C4NYwYKCJ0PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Stat-Signature: thpk6pd1htcj6taycbmd9rrcnhn4ntzq X-Rspamd-Queue-Id: B0934C004A X-HE-Tag: 1661808365-111322 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index dc72be923e5b..0d9c1563c354 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -676,6 +676,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_assert_no_reader(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), + vma); +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -685,6 +692,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) {} static inline void vma_assert_locked(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma, int pos) {} +static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/kernel/fork.c b/kernel/fork.c index 1872ad549fed..b443ba3a247a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -487,6 +487,8 @@ static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + /* The vma should either have no lock holders or be write-locked. */ + vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } #endif From patchwork Mon Aug 29 21:25:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D969ECAAD4 for ; Mon, 29 Aug 2022 21:26:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49A07940015; Mon, 29 Aug 2022 17:26:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 41B3C940008; Mon, 29 Aug 2022 17:26:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26E9F940015; Mon, 29 Aug 2022 17:26:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 02F92940008 for ; Mon, 29 Aug 2022 17:26:08 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CB9CCAAA1B for ; Mon, 29 Aug 2022 21:26:07 +0000 (UTC) X-FDA: 79853913174.31.114DBF2 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf14.hostedemail.com (Postfix) with ESMTP id 881A2100040 for ; Mon, 29 Aug 2022 21:26:07 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id a17-20020a17090abe1100b001fda49516e2so3659650pjs.2 for ; Mon, 29 Aug 2022 14:26:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=+ZeFgC+onLi5H2Gk/r+wfB2lUIAEpTgiMaISz2j8Yh8=; b=sfdQ+2NyYwxA7WE4W+U26tjnRsGpTwH+W4yCMsULIhDPYOMjHM3COZzPg/S6KTqrLL mCgPnd6W8rrgwvL37XiGeYHG8ArgrFQG0UGf2gLsolv/ME8hJmTyRg7Fg1LBPlsPaNI3 te4whtYEHF8c2GeRvQHa/rDrJjk62KSTUlVEPYqGvjKgQGKn8f4Dp8C4KiASRefP1Qh1 bQmVBjd8KI32N8+vGT1M5VRcf57cOjajmNaLyJUUpin1ccvlQLSGjkUAJOTaDxWOJrJ5 Pk3cN7LOZgsgUwKnowGjucplCuekHbu//K343iXFhb3F2AjszKhuURdvTUr7Fsy7vLjJ fKug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=+ZeFgC+onLi5H2Gk/r+wfB2lUIAEpTgiMaISz2j8Yh8=; b=2XJJBIbCjFH9CXpyHrb8Irbio6bX5aLp9OMPYlyOj+XAnY1Jjth+9cpMY15lR/p03M 46w9J5iYu9PSgPDYpb3UItAD9imvm706h8qws2wNhvI+9SmAd8NkX3slUAlPydnqNpXz cnLOzJIioiasvu5M44JhgBswy7Ij7mfWeoQ0PQMzljRzh+TgcH8Igw7ogKBvOCKOqgNU FrBE3ucQ0xMQEA5xww59l8jU0iXQvSrO6uHVrHHosqsVVZqwjOYB+MvDx9RxiGH+iQCP Tv25tx+JadUtvQDUlyXehYVpbpUTTh4eZHNRZ1oXu2NiEMiry0i0Ng4dIG1BgJj2nI1M pLMA== X-Gm-Message-State: ACgBeo0VPh7LydpLNQbxtYAuId5g1EcU+1379Y6WAIxuOcwTcjF9RxaZ ImJgpo3rKiYLQzI+sG3Qk6jptEgfhwE= X-Google-Smtp-Source: AA6agR60gqv1XJF9S7yhKpnItHkmCdUjQh3CpTVXyWE3xiDHKhNiTBjmbY9F8YGX1B9vpUO9txTmTcIisVc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:6b88:b0:173:c1:691b with SMTP id p8-20020a1709026b8800b0017300c1691bmr18372367plk.18.1661808366708; Mon, 29 Aug 2022 14:26:06 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:20 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-18-surenb@google.com> Subject: [RFC PATCH 17/28] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sfdQ+2Ny; spf=pass (imf14.hostedemail.com: domain of 37i4NYwYKCJ8RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=37i4NYwYKCJ8RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808367; a=rsa-sha256; cv=none; b=Jyt3AN2urC0w/EYjAiRHDBrC7ug5D2KVKwVUM/2olO/qrUQIWF4/ONC6m4cISOCkmrTUSp ZGDBVuhF9CIVn3gHhLovwBLFKDdpq0GqA+VdQVNNLaoiBmY/KTQOBF+ijkHb/YJb+JTlRi zNFLd1P/0bina8XDV0B9poeOqWGYpgk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808367; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+ZeFgC+onLi5H2Gk/r+wfB2lUIAEpTgiMaISz2j8Yh8=; b=bzA58Enu9P3klH+pg3cej+KufkTTCK1PNcKGYJBAlDcVzuld0gBcV2dGVVmZXatw60YknI GE2mxssQog5A41eX5MLvaM0ueKsag9bvc0p9j2fpASbjDpMN+pbjsyImY5YDRqSgVFNsUd YtH2HlCndvhzQZA9xXTMWc6VVxxbRWQ= X-Rspamd-Queue-Id: 881A2100040 X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sfdQ+2Ny; spf=pass (imf14.hostedemail.com: domain of 37i4NYwYKCJ8RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=37i4NYwYKCJ8RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam02 X-Stat-Signature: 4tnx7uwmwm4jkdjtoduqsqzufmrm6r4x X-HE-Tag: 1661808367-152644 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pagefault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to mark all VMAs as locked and prevent this race with fault handlers that would hold VMA locks. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b31cc97c2803..1edfcd384f5e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3538,6 +3538,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * hugetlb mapping); * - all i_mmap_rwsem locks; * - all anon_vma->rwseml + * - all vmas marked locked * * We can take all locks within these types randomly because the VM code * doesn't nest them and we protected from parallel mm_take_all_locks() by @@ -3579,6 +3580,7 @@ int mm_take_all_locks(struct mm_struct *mm) if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) vm_lock_anon_vma(mm, avc->anon_vma); + vma_mark_locked(vma); } return 0; @@ -3636,6 +3638,7 @@ void mm_drop_all_locks(struct mm_struct *mm) mmap_assert_write_locked(mm); BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); + vma_mark_unlocked_all(mm); for (vma = mm->mmap; vma; vma = vma->vm_next) { if (vma->anon_vma) list_for_each_entry(avc, &vma->anon_vma_chain, same_vma) From patchwork Mon Aug 29 21:25:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1A46ECAAD2 for ; Mon, 29 Aug 2022 21:26:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F4062940016; Mon, 29 Aug 2022 17:26:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EA1C8940008; Mon, 29 Aug 2022 17:26:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6DDB940016; Mon, 29 Aug 2022 17:26:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 92431940008 for ; Mon, 29 Aug 2022 17:26:09 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6B439402B7 for ; Mon, 29 Aug 2022 21:26:09 +0000 (UTC) X-FDA: 79853913258.13.F662E4B Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 1DCF4C005C for ; Mon, 29 Aug 2022 21:26:08 +0000 (UTC) Received: by mail-pg1-f201.google.com with SMTP id i25-20020a635859000000b0042bbb74be8bso2624682pgm.5 for ; Mon, 29 Aug 2022 14:26:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=MrRcKlTR7zCSrJMQg2Z93mfggQWMfDwWeAi691n34gQ=; b=cVg/tOxX8sxsAXVi/E7ByEF0x14KVPhZxGLvuX25A7I6HfsUpP9Yw5fhBx8ozeM60j nuDrEi09QA9+LTKC8GajK7CQiyi7gN9srLPx+SGJoGK9+OesuSLiR+9oEBujQJz23p/X P3t9ZdRyZDRuaF1wtpM+xaufjpbccFTUx81s4naxMNNB/ygFAADiWbtKwBK4coZj6Khw l+lkYHWl+iss5qkPjBxcVIZuDHxj8lggM4wnCJgOdoiaVpSSSGOBOp9823YRpnW7UBz5 1Wlgpr7ZzU0IbMK3CpPADXeW2X4v/4Fep7gHMya9NlwRNw5kCRaApBpvKmaw4X2L8jbG 4C6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=MrRcKlTR7zCSrJMQg2Z93mfggQWMfDwWeAi691n34gQ=; b=MxA8HV4lZzhCuAyDoZZPrBHyEPoZ9hM4UIC3WD7mBf2qQUu0miCeVISfXsd6XvntVO fPf7vpnQCzuGX8QSG/VRKP9wazW+k5CMZ3wR4mljK/qWPHCUWMgH8i/IiSbLNzWI1wFy qWc6EPJr9072tqo4BZifc7EpI8BuUWNRK14+gAvyqPtrBME9xMzds5vLlYQO7DoCuo3E PcfobCZWMTA2qhNyZtX0uMmeKmMYiPWyGIkLI0+KJvaSPQ82t7L6YEG9Ox74udF8JsPc NAMDuB5FVFQ05klrPVyckGz4Gs0wq3JOSqNl0MJo4dRaQ3DwwJtfu/m8fxdyP5E0oto0 VFLQ== X-Gm-Message-State: ACgBeo3AYCEgdVIbhamBKR1MEJujnk1v78CyGR1qLREgAMkQzmuK3LRB 4VGuGpROKrqcb5Chxb2O39ofGQD9ZDQ= X-Google-Smtp-Source: AA6agR6U/5CcVqFo3qhw78HCsOiNZ4QUVwqc0LtUdoOd4ifUOM6ozQmoAsAgpNZOBFg7UV9Y6fDIpWBGaXE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8742:0:b0:537:ee75:601a with SMTP id g2-20020aa78742000000b00537ee75601amr12578569pfo.37.1661808368234; Mon, 29 Aug 2022 14:26:08 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:21 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-19-surenb@google.com> Subject: [RFC PATCH 18/28] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808369; a=rsa-sha256; cv=none; b=zciObebkD/qRrxrvhERF93y4wC8Os0ugAl5DNIc81Rt+TAEYZRcWt9k6PzidBhDO8rTofU txltpmXPHJt2rDRriKplAfPpSgWSkUeEMdjMCCN/fDTsvhCOeyIyn7oj+OANzg37yESsBR p7u2hVSaNc2P+dwJDbEzJP/8E18qwUM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="cVg/tOxX"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 38C4NYwYKCKETVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=38C4NYwYKCKETVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808369; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MrRcKlTR7zCSrJMQg2Z93mfggQWMfDwWeAi691n34gQ=; b=OReoAfJ3oKBse1o7wIUKlhs+mulc8yMMeQ7MSj1/EZhbIZ8N+wwzxGJZCz76rD8mrRnTlb +ssUaAalVS1lrcuJlbAqZvT8fn/ioO95zN5wRD4DvZEcj+B7+eGLZxGQyouJXSA/25zRNA /Ttp5/cq8B7XKuTl+vvc9MtOH42GwOo= X-Rspam-User: X-Rspamd-Queue-Id: 1DCF4C005C X-Rspamd-Server: rspam12 X-Stat-Signature: urwb19qgjxt3ytg83eq1jty736hqa75e Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="cVg/tOxX"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 38C4NYwYKCKETVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=38C4NYwYKCKETVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com X-HE-Tag: 1661808368-465647 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0d9c1563c354..7c3190eaabd7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -466,7 +466,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6a03f59c1e78..36562e702baf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -886,6 +886,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Mon Aug 29 21:25:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6343CECAAD8 for ; Mon, 29 Aug 2022 21:26:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D472E940017; Mon, 29 Aug 2022 17:26:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BBB42940008; Mon, 29 Aug 2022 17:26:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92318940017; Mon, 29 Aug 2022 17:26:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6A980940008 for ; Mon, 29 Aug 2022 17:26:11 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 435E3120B48 for ; Mon, 29 Aug 2022 21:26:11 +0000 (UTC) X-FDA: 79853913342.29.1B196FB Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf20.hostedemail.com (Postfix) with ESMTP id E99121C004E for ; Mon, 29 Aug 2022 21:26:10 +0000 (UTC) Received: by mail-pg1-f201.google.com with SMTP id m34-20020a634c62000000b0042aff6dff12so4547519pgl.14 for ; Mon, 29 Aug 2022 14:26:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=K93TFWAb7JwQdWCvuFbBF/NwLgqzppA/TxAstG/QPvU=; b=f8ZHy6CiTcDXROb3pFMLfetBSV+kFVTeZws7OjDiiohYptMrJjGynWZ4jN+J/j3SJj dCEDPAmPkmnX5B3X46hqeY2mb8ZvMpLB7j64gw3uDHSYMAUM2B6ucp8KKPug6MxLnEu7 BmebG6DC6czjsVXnb7ENc9u7p9QGYJElvc7qRNsIMqXvgjewnR2ulhT9EyPiZ2zz1B9D 56/cpf1MfCjBxFK5vFsjvAIH2zjjfVoYxcmJWgmvbc3eCKVdp1qhJM+tuigQjNUJJMgZ jrHBPRDF7lp+rxYzH8zskIPXprMRBcWxeh9qo4POOeyeUnThUPILDVIwum49CxVeKLEs T9zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=K93TFWAb7JwQdWCvuFbBF/NwLgqzppA/TxAstG/QPvU=; b=MW5u5/JJRi/7eLs8V9DycfsqKh6A6Oo1ToK9Bk4B0aXVXaWsDliKPhdkllCcKkCW15 4IHMwwV3KWkWvwxq+S9ahxPN13JD484CeHnC1RmX2EsmJJQ0H+H/FyKXuDciq3CLYadK T20W25V3yJnUxV0hvlmSQ79PjJ5RSEJ8MurzVMO+yjj1/mLLvxzk9gWN+rMNfusbneZJ BFu8igjBIouSuj+O0FiLtuQHVsR6kbfs6wrEw9FZVTbvYp1cQn9wTJ2X8DI8aRTDiax7 VdzCs/cVKR+iFDZ5KIKnCUmn6C9UAIt4e7r4nC7pGWN+Ur8oqlDWmHSRaWZkmmtx+my7 M5HQ== X-Gm-Message-State: ACgBeo2ipPkVKSrP0PsUZfDC6zaulBnLvAp6RdAo1riv3OmwqhOVbirz InR51ce2gCFcSVyQewevUa0pCb9LM68= X-Google-Smtp-Source: AA6agR7nf9+ItGZZ0+7hqrcNY7xNhTop6ny3nOph/OxIOFh5WmAm4Kp1l2Jp+ca/L6EEvcmkimt0snMwAs0= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:b094:b0:173:12d1:4ece with SMTP id p20-20020a170902b09400b0017312d14ecemr18383730plr.117.1661808370030; Mon, 29 Aug 2022 14:26:10 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:22 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-20-surenb@google.com> Subject: [RFC PATCH 19/28] mm: disallow do_swap_page to handle page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808370; a=rsa-sha256; cv=none; b=t37jJ/k1k1Qg1AaqfzNsrDMKy2f3RygO7ErC8KM2D8hAP+SuDTUe7Jwwciq1VoWWKRRE1t JFPn0xPu5oVXDEVHEoghq+wWlmlqTTab65OQlrPwteh83R6Qwt5GqXp66to5a0ZzMSYIXP R/uQk+4hl3L5OPBdU+MM9O/WWTFds1o= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=f8ZHy6Ci; spf=pass (imf20.hostedemail.com: domain of 38i4NYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=38i4NYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808370; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K93TFWAb7JwQdWCvuFbBF/NwLgqzppA/TxAstG/QPvU=; b=K+boanrmJrtp+ggLvCVhSTUTevYGPuWOB2S6bAndlT7bIrFPBVi1e5UczVJtt7e33CUZ5T ZA+qSDTrsdb53GPSNn86KCjk4BLIEzccrGER92w/ZbfjR93vJTKAhQrLARp3Q+f9VSkysF 9beBl7PmKsBC9d686Ok4Siyx52VrEQw= Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=f8ZHy6Ci; spf=pass (imf20.hostedemail.com: domain of 38i4NYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=38i4NYwYKCKMVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-Stat-Signature: qkoorhagoidibjmf8abp39mia5ehx3a9 X-Rspamd-Queue-Id: E99121C004E X-Rspam-User: X-HE-Tag: 1661808370-389666 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 9ac9944e8c62..29d2f49f922a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3738,6 +3738,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vm_fault_t ret = 0; void *shadow = NULL; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + if (!pte_unmap_same(vmf)) goto out; From patchwork Mon Aug 29 21:25:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA035ECAAD4 for ; Mon, 29 Aug 2022 21:26:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3591C940018; Mon, 29 Aug 2022 17:26:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BDA3940008; Mon, 29 Aug 2022 17:26:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0BFC7940018; Mon, 29 Aug 2022 17:26:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E6497940008 for ; Mon, 29 Aug 2022 17:26:12 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C9B4816040E for ; Mon, 29 Aug 2022 21:26:12 +0000 (UTC) X-FDA: 79853913384.05.8E3D4FE Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 78C8180004 for ; Mon, 29 Aug 2022 21:26:12 +0000 (UTC) Received: by mail-pg1-f201.google.com with SMTP id 136-20020a63008e000000b0042d707c94fbso212004pga.9 for ; Mon, 29 Aug 2022 14:26:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=bUksEMWzQ2sQtj6CAseSDP7Wnd3Y9YWuXlHuoAiHS/w=; b=R4f3AmhYQvfrJmCukv/e3Nij13PuVUa8k8O646df0PoObwBnO/xDcemk6+SqKXvVGr hJkm2649shhdYcclI4xYPbiKHXEe7XEvjq6ebwkp6YZFsZtQX3MH5xTV757BPS7Zeyib IALyiqYSG4f0kCJDHQ3+fMXa5F0OzkQKlh2EgahzIDAp6HMSPtfd8qN17COdyNthUkyK vH/w1n6XgnYHxRHIzAjyfQH31twkRddq+jpJViGP9dMWFd2ba2cwGPVTVH8mdAvdjCY4 84OD9xZ4r8HlzIfM65gveSjGxaILhsxn9vJXDz/VODVoOMzJt9RI4cwlcbKnFfX3PjSN BY7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=bUksEMWzQ2sQtj6CAseSDP7Wnd3Y9YWuXlHuoAiHS/w=; b=nads1cmHQCUBEzdg2OQyJ8/Ciz+YGC+tMnXM2iRWftC/66LVmdSO/VF6ZhIxwexouQ HX076BtX5VsnRumiMtbGyh6U0cH+flRQduoMemmI5lrX8sNgNyYpnuNUrd07/74nw208 FmZrCI+7dUqQhe/hXiZF9Y2sdc7n5nj/IjLTXrw7xOeK0yJOCVxdlL+ggDy+xg+RLxDF SR8W2dXW/KbWlBPE+orXB0lydjiqLnyqhOGZMVHUD38mzdhHNxCEJaeXJFpAiO6iSgoG hQfaOy/4EMt4GwW/iJhEXvWkQjp4G6Js8JekLr1sK8F03fEEcZWctnk7abMJzl41+zAR TNZw== X-Gm-Message-State: ACgBeo1qtN7lme5z+bEvvODT4SQGZVYkUHpQQxD5CMnFG4EXmXOMkyP9 Qn6fDWk5SybHjvFicEZ2r2gGhrc9690= X-Google-Smtp-Source: AA6agR4CSgZfYughTw1oP2TGtJ/Gx4qpWPKVumSGbjZh4UG9V5skHPeijnTVviS5CargVPsvloUs2cN/SRk= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90b:2c11:b0:1fd:e56c:79de with SMTP id rv17-20020a17090b2c1100b001fde56c79demr4481996pjb.201.1661808371494; Mon, 29 Aug 2022 14:26:11 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:23 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-21-surenb@google.com> Subject: [RFC PATCH 20/28] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808372; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bUksEMWzQ2sQtj6CAseSDP7Wnd3Y9YWuXlHuoAiHS/w=; b=IuE13G+N9+mBXqzu9srpZDtv7OSPTkGFIrNE3xS6Bj5mLZDhKxG8eC23OE/FfJvGLIpkyk nhare7M69or+4nb/6xs4znRMtVJ4ORXNgZIIVjUor9SvkRv1pTUuHs1kWMuI8pkLy5K8R7 KdK/g/9+T6y2PWRpbzaJ0HrqlfcGuwo= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=R4f3AmhY; spf=pass (imf30.hostedemail.com: domain of 38y4NYwYKCKQWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=38y4NYwYKCKQWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808372; a=rsa-sha256; cv=none; b=uW2d3SYFpA6h5mXD2Fb52WHmhlcQHkNvUCq8zBYxw0lOmlvinzvBO9j82TZgJzo3A8b++z BKh8Q8mrDujtupxqjEEDDFzFb+6oL0FlIKzmtq5bFH3D019JvXaw6d1IkVUUMm5nP7OTwi MJb5v9Ve9YDP3bJUHHMxIKdnHeplGYU= X-Rspam-User: X-Rspamd-Queue-Id: 78C8180004 X-Stat-Signature: rscthjcj1cgox8cexx5ktwdtd97nbms3 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=R4f3AmhY; spf=pass (imf30.hostedemail.com: domain of 38y4NYwYKCKQWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=38y4NYwYKCKQWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam08 X-HE-Tag: 1661808372-19677 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 8 ++++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f3fc36cd2276..a325783ed05d 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -150,6 +150,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index bfe38869498d..0c2611899cfc 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -131,6 +131,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_vmacache_event(x) do {} while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index ce8dded36de9..075642763a03 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -207,3 +207,11 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + help + Statistics for per-vma locks. + If in doubt, say N. diff --git a/mm/vmstat.c b/mm/vmstat.c index 90af9a8572f5..3f3804c846a6 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1411,6 +1411,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Mon Aug 29 21:25:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3EF9ECAAD2 for ; Mon, 29 Aug 2022 21:26:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 665D5940019; Mon, 29 Aug 2022 17:26:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C6A8940008; Mon, 29 Aug 2022 17:26:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A6D5940019; Mon, 29 Aug 2022 17:26:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1960A940008 for ; Mon, 29 Aug 2022 17:26:15 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E8DC9120B0B for ; Mon, 29 Aug 2022 21:26:14 +0000 (UTC) X-FDA: 79853913468.18.7511B87 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf02.hostedemail.com (Postfix) with ESMTP id 9D41080043 for ; Mon, 29 Aug 2022 21:26:14 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id x8-20020a17090a1f8800b001faa9857ef2so3895191pja.0 for ; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=WI+iAkKYdp/dw+q+sE6vVFwD5mLSWBWMhRHboF9Ttbardy/unLndkxHVrzFVqulr1A yPcJLa3Ur0z+PJer1B1IamGoCGgxEpMLeAxgWf9CgasGK7cypr9FW+usqow4Tv90c3k6 hSoUgkPn+GgzurgQcPzRD2rwA/PJHxIAlhRbX6toXLNmtMd1EOWFLhXluJKlLDk1UBbT O0I6rKmCm2uwsJ4JnyJZiGAk3wWQ6Qh6RctA9jkOKMvzLM+0NSmJ3ovBUaPQyVoGm6YI JXf6mvqPKdqWiUAL/+XE2dov1V39lipJcNcZcI/t4fYcyAX1GpBk//UErawM2Q1iWoKr uzug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=LtbT42ENU5gV7KQWm11ZCul51r/6VSIUChhRCl5LtXnxMxiL+PnGFOL4pnW5jI6nYY Z5e3f/Y5n0RJ/9VI0+BQ9ww1T1yCjBTpfXJzjeWSxjh5K4Q0fWkvuenuyvBlqSPELr2G NA7BXPn6jQCbBAfTNCiRTHRUz7q5TlX3qTlw8w67z4eiNC/UTEjwxwHqjKjbOp2rE4uo srGLkCg3zTzFJePUVNHKQFWImzAKtalYaOIkeYxhu7zOn5P17BAwiv9Bny26Urdn74JT Weh9nSEKkxWXDeDkZ9zXUCQqHEAK99/hMR12w+HVCyNsypAlPwD0PGER/atjXjmcchcD aeZg== X-Gm-Message-State: ACgBeo2b+paQJjSxTYCChAyWVs2RY0BqoNJ1MLDdLpN1fr8VCme/u+eb zlCqBjtIzYgniYVz2K5Tt+WPGUN/YFg= X-Google-Smtp-Source: AA6agR5aic644PZ0OuL1YK8O5lnv8GmzPsmAclgLgLZvZKPuRUB75AtHpFEjRVbtgx264YmpiRiioO9wBJc= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:249:b0:1e0:a8a3:3c6c with SMTP id t9-20020a17090a024900b001e0a8a33c6cmr21187pje.0.1661808373054; Mon, 29 Aug 2022 14:26:13 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:24 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-22-surenb@google.com> Subject: [RFC PATCH 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WI+iAkKY; spf=pass (imf02.hostedemail.com: domain of 39S4NYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=39S4NYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808374; a=rsa-sha256; cv=none; b=6JV3cnkVvlOdJkUhpgGTkqy6mnSabB9v826vMNmK+gyi9sZJP+X1Wav7g+Etpmg3QmBCH/ Q4fm8SC7Vqe7G7/7eAjjhprpjWAsz/R3SMvbhu+GOzHr50lHS+7DneqeOEjCJoPFqXWcDW ABcicT91Q+Pnb6Zj6ezxcXILFPeSz4g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808374; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TAH5MnJT3oUowhISL8bra7V0NlPN+Gm7GlObU5EM5DA=; b=IQSN/GyHADhQl15P3a73DyNcRweZjMfD6/kThi7SFaHrtXArhhm3bqd0J2dEpa66UT5W9M ChGYq8Yx1UPqZ/Mw7Hv2f64S/CMu64hfNd1kIcMu/DD2JqE6QVIHzg6t3iJZStWcyBd+Ij fm0wwaBBUC2b7xF5FPCTPsvY4IkwmTM= X-Stat-Signature: f879kk7qbw5cbonuxhya71bj5ryjazgs X-Rspam-User: Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WI+iAkKY; spf=pass (imf02.hostedemail.com: domain of 39S4NYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=39S4NYwYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9D41080043 X-HE-Tag: 1661808374-886945 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma = __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be stable + * and not isolated. If the VMA is not found of is being modified the function + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma = find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Mon Aug 29 21:25:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27912ECAAD8 for ; Mon, 29 Aug 2022 21:26:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD3A694001A; Mon, 29 Aug 2022 17:26:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A0D45940008; Mon, 29 Aug 2022 17:26:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74E9994001A; Mon, 29 Aug 2022 17:26:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 59138940008 for ; Mon, 29 Aug 2022 17:26:16 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 314EF1C2F85 for ; Mon, 29 Aug 2022 21:26:16 +0000 (UTC) X-FDA: 79853913552.13.04DC367 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf28.hostedemail.com (Postfix) with ESMTP id D703DC0026 for ; Mon, 29 Aug 2022 21:26:15 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id i191-20020a6387c8000000b0042b44ad723bso4563119pge.19 for ; Mon, 29 Aug 2022 14:26:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=plnLK4otWaGxR+W8sd3fBxFm06LW87+WFj6YX7JmKJI=; b=O7Fq7HWZ7z8n2j/HcxIFYSMFkVCbPksq++2+VIS66vXRQwbI7dU8JTklRI8MPiby2e edzsUySG6IImo6mrh8ENCQ2+wxA6BfMFPfkVn5AwXOsZupGcnzhGGSKXttVqFe7CqMCi 1h3acQOuI10/ig3PP8xkKsm4NtzNbHGpaQpLwoGjKN2BSoc/HFLvkEbHUIkKXG5J3g9z 9q6Jpb0Ph5QlVq/4hF27NyIuM3/V2/KAkuHoouSJodEv2dWsfVfE58VeJPPY/+QOUQfz r+QwUf2vpfx2TV1s2fbWWB8eIKopRLMSEdWWEDiBiVoTpxt/dnwqSCw7tnlmHUsJj3fT 5A2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=plnLK4otWaGxR+W8sd3fBxFm06LW87+WFj6YX7JmKJI=; b=0l5VyXjP07a/LhGvASyKqmJ+oN7a1mmW8ulI27Gu6SK7TuhEWc3iiNpL35Ind5VSkB qhu6suUwSP/CKyKYT6ZQEMhREzRP1O84QMxwiQiA0rRsIvpIO0W9fGVTzdciSeoxYuKp y2mjprFMRWevU87FRJMg8sJara9XkARcNvp/mZJSAZekj2EIl57woqw4Io5PVpeKPhaR g1y+IT6tBT0VMkplZrlF1vG/h095FVAqh7zkHsiRiVADBAMZ+3UxxBgsOEhzl74g3cdO vCggqOC94hXVt47tQXeqXzW3VkmVYsdHWp8FBpzN5WCqOGLwNNlGZcaiz15kiimB0wDY SAJQ== X-Gm-Message-State: ACgBeo2142MP9HqDMf01KwFsNNDLiWzzgGlwupbRVTk5gI/+OnaCJNvG ovtCZ66lZDC/gjJkaM1+lj+NxNWw064= X-Google-Smtp-Source: AA6agR6koDuvaTUSC7OU/WlAja75jS99jSayBh5syt6ErVM0PHA+biyxUvs5Ar838YXdhWnvkXm6gonES4M= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a05:6a00:14c7:b0:52e:efb7:bd05 with SMTP id w7-20020a056a0014c700b0052eefb7bd05mr17908102pfu.24.1661808374985; Mon, 29 Aug 2022 14:26:14 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:25 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-23-surenb@google.com> Subject: [RFC PATCH 22/28] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=O7Fq7HWZ; spf=pass (imf28.hostedemail.com: domain of 39i4NYwYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=39i4NYwYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808375; a=rsa-sha256; cv=none; b=NA+NMet9Xkj0/zWTXnD7iqljZrHtOyRjK7//cZFQ7z/9HYjVpxCXAplu9MKH/TSfL1SJ6A 1fMh2nGoErrcCvmBZGH8jffKR7/IKbxSKY27fXe5CArMo1k7WNbmFcIr/Jvt5i6Hz5xJH6 Z4g2ZcG6EcGZojT7rTJpD3FHN17Xitc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808375; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=plnLK4otWaGxR+W8sd3fBxFm06LW87+WFj6YX7JmKJI=; b=QV1xnQXNH1ugGFw4J0z1bWTOYVLWXcSknzWl9oJZo1ftUbeJmSdFvf03kwmsPKyb1Eelo6 /DqpLkX25QBBM1Wi9KYEfWJEkoqo/Ga/g8eT8PCb91BjvNaStVv7tA4IrFwQwbQvaC+KGX Nzb3Xym4CJuyGZ0zG85sWCjnHG6/f3Q= X-Rspamd-Queue-Id: D703DC0026 X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=O7Fq7HWZ; spf=pass (imf28.hostedemail.com: domain of 39i4NYwYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=39i4NYwYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam02 X-Stat-Signature: n7p146edicd36mz5for8ei3io5a9ar57 X-HE-Tag: 1661808375-300463 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index fa71a5d12e87..35e74e3dc2c1 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1323,6 +1324,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1423,6 +1456,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Mon Aug 29 21:25:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A640FC0502C for ; Mon, 29 Aug 2022 21:26:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0991794001B; Mon, 29 Aug 2022 17:26:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3DA6940008; Mon, 29 Aug 2022 17:26:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D915694001B; Mon, 29 Aug 2022 17:26:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B5EFF940008 for ; Mon, 29 Aug 2022 17:26:17 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8B976AA9CE for ; Mon, 29 Aug 2022 21:26:17 +0000 (UTC) X-FDA: 79853913594.08.75E7BCE Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf31.hostedemail.com (Postfix) with ESMTP id 551DA2004C for ; Mon, 29 Aug 2022 21:26:17 +0000 (UTC) Received: by mail-pl1-f201.google.com with SMTP id l16-20020a170902f69000b00175138bcd25so362818plg.2 for ; Mon, 29 Aug 2022 14:26:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=2fd/wGll+ATU6BJRTn7wtpRUT69Ps19K4L9NuJFwTCo=; b=CptXAEbTEIrjcI1hpAhqMGzwstwkcEJJxmfb7we2Tkw33BFQo0d25Vl71IxE1hgwCE +/zdjUxBz/kfwlykJ2mpRLD2ASHh0Jx/8NPF+GvmOj+OIPDuJ9h8tJ0qgl8jIPnA0gyY HL/K18g+JnPGw8ocd38eG5KHL97qZgAE2vHMSq8EmM7kYFzwJqKpJAdVhwHK8rQ0CEUk zlwtiJI41bKlNxlGW7NWWuuvJOTrejKhLXTonY1HBOreH+6SF0a8TMKsBr5d7H138Lnw MYC4Krs42a4eoKPM77oYkjvTzwtNbqixe2Jty6RY0Tg1xXxoDrCaGfCoHKiNRbMG4ac9 6NkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=2fd/wGll+ATU6BJRTn7wtpRUT69Ps19K4L9NuJFwTCo=; b=LvYkIZLoplKFYGFpkpyWQgwV75REoPlMUUoFeDEO/fuZlsV53A+/aruc5MfK+o7Zc8 6z+xhC4oGW6WPWdeJO4dWXJqnqGPHeMrHZOVF+0XlQqTqUA19ulHRLJgJ44CBv8EZN48 Hw+qESlaOy7PvWfcyvzldJwuc4QfJVmKqzWoIztycpfe/rzBrw8Gc0hjzrMrzxc98vZk 2B399oKAbLLy56RNAb41s+V8jTFaQn11AQwHESPuDrI0BCeYiyhXf4SmLr1YVXi0GrRI pyylGTZ5bhBHqZq4AwZilvVVa/LlfL00TcRIlWwyUEx1jKXu4syHz7/+we78V3UGhLyG RsJA== X-Gm-Message-State: ACgBeo070TnTZQPk4uH5z5p+8E3uKKQS6heDk0oU1cXync051C+aRzrJ Qai2hA9ZMu4T6eWv4I64zSEJQKuj2rM= X-Google-Smtp-Source: AA6agR5QjqhJOQoDiATaLtKSN9vwads3L2SWpCSYlmwHfKycrJZLy9F13OZO0MNqEBq8/z8yofCbHCaZE68= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:902:d54b:b0:174:af35:4ba0 with SMTP id z11-20020a170902d54b00b00174af354ba0mr7860599plf.74.1661808376452; Mon, 29 Aug 2022 14:26:16 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:26 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-24-surenb@google.com> Subject: [RFC PATCH 23/28] x86/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808377; a=rsa-sha256; cv=none; b=DKUg1nd3NRpebzGNlOZGGSwzP6FLZnXY4feDmr7vLp0IJWeGgCOizzakiETVkEdECc3Twr IqTs9QfM8DwI+eeayxFNEr++ywLppFmjB1nxFNwqROPcljE90NHDnmBPQmAQjVDlb2g6XZ OsX99j0dqd/ZgXWNSxQYHAAUL+HMxVg= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CptXAEbT; spf=pass (imf31.hostedemail.com: domain of 3-C4NYwYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3-C4NYwYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2fd/wGll+ATU6BJRTn7wtpRUT69Ps19K4L9NuJFwTCo=; b=XlPQDLGNGTMvxWatye/Z6/vBnhW9ZmV9ZenKeSjKapkAADasGZoyl22c5mL9Y43aw/Y4CR JR/zRox4vj0XqPqF1v1Hnw6qTD0P2BpiMhPdzUJ5Nu64jlEpyB/Fx726IC+rtD3ZuKfQTr rFGVRlRAWGIt9CbVa47rERu1YA9nfAE= Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CptXAEbT; spf=pass (imf31.hostedemail.com: domain of 3-C4NYwYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3-C4NYwYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-Stat-Signature: o7jr347zixuk6zmric517zd6cmsxktg4 X-Rspamd-Queue-Id: 551DA2004C X-Rspam-User: X-HE-Tag: 1661808377-351877 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f9920f1341c8..ee19de020b27 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA From patchwork Mon Aug 29 21:25:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43781ECAAD4 for ; Mon, 29 Aug 2022 21:26:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74FFC94001C; Mon, 29 Aug 2022 17:26:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D976940008; Mon, 29 Aug 2022 17:26:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5068D94001C; Mon, 29 Aug 2022 17:26:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 31875940008 for ; Mon, 29 Aug 2022 17:26:19 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0A974120B90 for ; Mon, 29 Aug 2022 21:26:19 +0000 (UTC) X-FDA: 79853913678.03.FD2CE10 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf01.hostedemail.com (Postfix) with ESMTP id C2D1540046 for ; Mon, 29 Aug 2022 21:26:18 +0000 (UTC) Received: by mail-pf1-f202.google.com with SMTP id v65-20020a626144000000b0052f89472f54so3615992pfb.11 for ; Mon, 29 Aug 2022 14:26:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=3FeDd/uESQnEyJqRWNnPK+iT++ObIkcBX2+m7Yxyetk=; b=WuXmy7spYby6FKlYZxVkneAX6CYLeedEW25AaCdm5U/xwDtXPaTsukbnW+BREr3wmK Yp+DdB6epKs/SB7JvycD5yPQfOWs3SBJlayjVM1bzBPF4xDdscUUlmQ491HoasCmixWz EDQApyvlKJ0m9j6IZ5Bl5eAvYAMUvQ5PzTt+MmJPi2OMUuj7wlttfCVr1jtd4buEfno/ mHPpjk9OptKM0QuTB8OPa5JPYvPxCHU8q8PlG/PC9yFoVmLZ3eeWZBp1XntU6iG84/rA BgcQG1UC8Z471LrYLx/b3SNnNzdfHV9f4k2uE0RZvNOL2F/Wod5mOLtISQc7G5F1GR1Y 7dZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=3FeDd/uESQnEyJqRWNnPK+iT++ObIkcBX2+m7Yxyetk=; b=S5q7DjGaXFdNQvthDZMzedccXWxXpOcTwFnm+Ua3AdoZJIsBPkmt/UTFu9HXKiaf8J UDktUUppT/tD6I9B+1HUCZo0jA1F8mE3GkYC0OSzUPZEe2E84D+sq0eBcCO/ANoAjXDo ALzgfqPWq8JURAUJWYz5UsXzkWuuzC5rMzEk+qLGOV0v1Hy4jkLYyownLy0zdWnRGqTt TNSYqXcr8ggj6gUWLHk6EBc9U0ikC8DXl5q9rxdRUKG0PNVwbDc8Yrstrxc/G3mv34H/ JdnSe6bT8bRM4bwaKEhmcALTBL+xj1MTCWcgtYOCAU+YaX9PlhQyx5qPJ9klzZiy9iNx +TYA== X-Gm-Message-State: ACgBeo1aQ4bXkCmGj9dMJ+r0gPieeXIPsPxkc4Ok6eVucOxAExTz147B YMI0FWzHjGEZqb84mt75ToA/DssqL4E= X-Google-Smtp-Source: AA6agR65SMIHezE5eUhR7FLJy4sKjKwDRErJ8crUMropZEJrhJg/maBEVhsE6eunhKdLtoV+/pEsLbtUb5g= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a62:b519:0:b0:537:9723:5cf2 with SMTP id y25-20020a62b519000000b0053797235cf2mr18436343pfe.15.1661808377917; Mon, 29 Aug 2022 14:26:17 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:27 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-25-surenb@google.com> Subject: [RFC PATCH 24/28] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808378; a=rsa-sha256; cv=none; b=OsgOJqldWdaSe3SvvkqNA7ItX8WgR4N/vuWzE0O5MeiZE485s87rxLraiMnxZFIn7bybIW OHE2hYkkEOgpzvoQo2RdIrcUvniJOnF1RVk0EjL2vZ6zcqwR5/Tttl11Bo6dRaIPnh48DU VDw89ZXEgjK+D/BUu4HPMcqnXnWWdEk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WuXmy7sp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3-S4NYwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3-S4NYwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3FeDd/uESQnEyJqRWNnPK+iT++ObIkcBX2+m7Yxyetk=; b=olT/beM1rTNNcgi5HKRrG7syEKrVs9x8NLwBzk59sJ89dFDB8IlP83Z2vkQ51jkzH1DM6b ybC4SdFDl/BHPgrP9Y1AbV/WLT7PkKAAvInzyRzzdXH6h4C/Sk5/cKNjR0w5LpYy5SnoWx uMwNpXKmXFJZrvTjs+PmeRR8HD7kly0= X-Rspam-User: X-Rspamd-Queue-Id: C2D1540046 X-Rspamd-Server: rspam12 X-Stat-Signature: mkrdg1godb17zork8p33buk5sro1c3qz Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WuXmy7sp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3-S4NYwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3-S4NYwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com X-HE-Tag: 1661808378-68185 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index c33f1fad2745..f05ce40ff32b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -525,6 +525,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -575,6 +578,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -618,6 +651,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Mon Aug 29 21:25:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 423D4ECAAD8 for ; Mon, 29 Aug 2022 21:26:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD0DA94001D; Mon, 29 Aug 2022 17:26:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C58F3940008; Mon, 29 Aug 2022 17:26:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF75F94001D; Mon, 29 Aug 2022 17:26:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 94F48940008 for ; Mon, 29 Aug 2022 17:26:21 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 709A9AAB3C for ; Mon, 29 Aug 2022 21:26:21 +0000 (UTC) X-FDA: 79853913762.30.913B99D Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf20.hostedemail.com (Postfix) with ESMTP id 2B6101C0042 for ; Mon, 29 Aug 2022 21:26:21 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id 2-20020a17090a0b8200b001fdb8fd5f29so1549199pjr.8 for ; Mon, 29 Aug 2022 14:26:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=M/5Y14x1SSOe03/jfupiRT6TG6dq1ngDrWV9hskObuI=; b=keg1i0nk+izk1klWl7kacG2dqLVFCBBFr7DTBR9Vkh347LyX6fgOptbA6FX9RaUao+ vJTGW+XEi+UTAE6ZSui/ACoLYvw+CzBRdshi7UOcb6GKyXUkGWluK1e9ViX0u2MOFUby n+VIqiL1KgbaHmBILZRqDKCT7yq9OcaG3eWuWfyFbeDAc3tNJSfOayUDVB763p8OZugP N6WyMveSXRyt1x9RwdvL650Tufs+g4oBOJJ5iU1MJBZM9QMXfDComRZRSTq/8D/ye7ef ZSEe/ZXHQtclALcTUMn5apyFMWCE/g3tgTBxLG+Y3u9vQ4Ps1pm7jxw7nOrCfEth2Ax0 A9EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=M/5Y14x1SSOe03/jfupiRT6TG6dq1ngDrWV9hskObuI=; b=0qBmizoWUiomqnigApKsNgR7rBmI26Uw3KYTWpxdClBTrh0oRnZfr4+YGK+qrSZUh6 Sa/ATp6Lx0Mcy6azE2BlrVaf1HWbeP+hYjFB+77MY97fgKqSL3ZudGox/Bm1COcv9Vl2 IYU1nFOkG2M/mY6WWBK5DUhiuC/40dfZpSeD1NOjzRZojHZebZJIeGFyFsWHFd9jZtKg 2UiLQkgq9n21CvGP47biAs+nNeaw2OktDsYuin8ihM5wUXnvoOmqguTSGKo9LhQY80U8 ASU9YZqTWzJWX302NbFYdvBvHkhRILJG2aCbmbIixm+lhd7OoNwj48t+PE3GHA1r/Nfl NSAA== X-Gm-Message-State: ACgBeo0hp/hcn+s3NsWr6SXpw4rjcx/dzW6mXeSjvpF8BcEMktaRVSeZ aZiw4ya+4RFIEeH013cfk+9p0jscER4= X-Google-Smtp-Source: AA6agR4Lx1uzJZ7T/fCqu0c+3ou3bFPxSP7qIc7HtsIZjbU9yCjHgv7Z+4FP1+5TPgAu6qrR46XQccXnjfE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:90a:e558:b0:1fb:c4b7:1a24 with SMTP id ei24-20020a17090ae55800b001fbc4b71a24mr21113pjb.1.1661808379380; Mon, 29 Aug 2022 14:26:19 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:28 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-26-surenb@google.com> Subject: [RFC PATCH 25/28] arm64/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808381; a=rsa-sha256; cv=none; b=JpHzQCOkOKm9JbzFCbe1MRSF1NrwRdy3TaOGtrf4WPwqFdt8gtvsBYnoCDnAdjBewNnP3E OOVCEe5pl7vxD8xtWe0tv0zdpxDUReTMExM3DaRX3HM0jmQ958z6zZ4teX/zfShTsg/1Fj q/qcODmH/SKpnI0mVLcyPnCfwBBoPJg= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=keg1i0nk; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3-y4NYwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3-y4NYwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808381; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M/5Y14x1SSOe03/jfupiRT6TG6dq1ngDrWV9hskObuI=; b=UwzOke34TFWqYzEHZk58/vGH7KeTIfJUGT0aj5ZNyqNpg8YT0GnrzKp9MUoiDo1lKW5n1+ lpR+OED6YB3/P+yJHwqj1WTnuZiO/NB9L4GxkxfJ7A4X6SgQ/QXLBqKUADD9sct54QvLvI Mc7vFtn813GM4jx1/UXAwzMviJ+tr6g= X-Rspam-User: X-Rspamd-Queue-Id: 2B6101C0042 X-Rspamd-Server: rspam12 X-Stat-Signature: zhhratcikkstbbb1oe37iq86adywygsi Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=keg1i0nk; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3-y4NYwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3-y4NYwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com X-HE-Tag: 1661808381-298577 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on this architecture. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9fb9fff08c94..0747ae1f3b39 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -93,6 +93,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT From patchwork Mon Aug 29 21:25:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0E4BECAAD2 for ; Mon, 29 Aug 2022 21:26:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7C6394001E; Mon, 29 Aug 2022 17:26:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E043E940008; Mon, 29 Aug 2022 17:26:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C31E194001E; Mon, 29 Aug 2022 17:26:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A9667940008 for ; Mon, 29 Aug 2022 17:26:22 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 865B3160952 for ; Mon, 29 Aug 2022 21:26:22 +0000 (UTC) X-FDA: 79853913804.15.D749F2A Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf15.hostedemail.com (Postfix) with ESMTP id 3F213A0035 for ; Mon, 29 Aug 2022 21:26:22 +0000 (UTC) Received: by mail-pl1-f201.google.com with SMTP id t13-20020a170902e84d00b00174b03be629so3066689plg.16 for ; Mon, 29 Aug 2022 14:26:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=o1aoXsAZRKWKp3E60vO3aTfa7LhhJu+wloCiW42SL3o=; b=hVadOQBIf7hIcCCPF4k/V8ishbMycFmV23IYK2/L+oJFUn4snhEXCB8nhiJ0XvBmGq IPVx7m/qpumur19aTHX425elTvPhBwnAWLq2eu14tCxMYLfyVDPtnXMD7eRtBYdE8ib6 g6FY2QFmIprsXVeiDN0/TJA6z8yPl6Ga6H+AU+5BAz5wvK9q2eG8K6M5/G5IT26iqvDR sF5KQOom085C4/LO0v/IjmuRTEiouanpvkUJ/WwC7Eqe/f3fgCYq7VPZ9r9nrS+kR+CY ydqosByt6O340qUe8wv/4Yn0djwvN5x2tceHdnG95lN+D/X9zA9TkMcItH0K4FkMtDqG cLTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=o1aoXsAZRKWKp3E60vO3aTfa7LhhJu+wloCiW42SL3o=; b=QlKGD7jfDtol8H/PAEYKEXOKsS1Dirxoq+yLocSWItKpAiqQBcZPzh0SqQ9VJ5rgL8 /49U6h58n+PhLS6EPk3ukf0YJsc6YEtGP7KOej1p+dMVR0gjIDHcDmZuf0ZFnzTJYxo8 3YI1Ls2qb+56tdLaUj9224PaJQ5qRYmSl5/WTjgoIVhs7CSePoYfGxUFhzPB626Xckwv To7p/MnUpqjwYCNHY5ts35yEx/8YSaHAhREaGLvqCBe//34GE+M10INu3/UNOlv+k6OY EXTNWYlsUsd7IZFZSKAqOEI7zrjbzi560ogWTQATk+wlBS/Zcd0R+e1O1V3AzrDXBaLg MF2Q== X-Gm-Message-State: ACgBeo09ae3zW7LXXyZXl0AiEH+459pswP/WCU7ENNCXwqXV4aVjAhuk NEaRegjWxSfqWhl0OANw8hUFLekKZNw= X-Google-Smtp-Source: AA6agR5CaUs6YJBbyF56NgpgIgB6cgapfFaHPRJdugast+FKzlkUBYXObPd4n8zhAA8UFAnqt1i7b7mpRsg= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:41c6:b0:174:3acf:8294 with SMTP id u6-20020a17090341c600b001743acf8294mr17683670ple.118.1661808381376; Mon, 29 Aug 2022 14:26:21 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:29 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-27-surenb@google.com> Subject: [RFC PATCH 26/28] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hVadOQBI; spf=pass (imf15.hostedemail.com: domain of 3_S4NYwYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3_S4NYwYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808382; a=rsa-sha256; cv=none; b=iZ4EQKq9a8SrMKhQj/82XFDP0Rg4xtYB+dHsRHGixBD4xmY06WhEmHbkmvEDxWVTys4E+S ZKAQcm+1lxiDgB9hINXlGqEcGcXYPd12DPCbCVePxiOqM98IazrxFfgBSr+YLn0jznj7sR glquyJGiUb9UYshNGQXWIQtg4YYPZEA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808382; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o1aoXsAZRKWKp3E60vO3aTfa7LhhJu+wloCiW42SL3o=; b=nt4doqxYosm8qzcfROLJ7XXVK6WRShrYAQtcfzt/eq6Owoek1uwxPF35bt0nX0ghIMaIZt iFn1cLrYCaEGPS36Jx1Qg0Gq+97Nywa8In9BpNpc/0S0lSiFQYB9dL3LiJ1YtaB0v0zpD/ MwKIOE13G9RWFZarM7TCnK1qISnXqCk= Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hVadOQBI; spf=pass (imf15.hostedemail.com: domain of 3_S4NYwYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3_S4NYwYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: gtfba6xr14nbyossau1x9w7ywauj4dbf X-Rspamd-Queue-Id: 3F213A0035 X-HE-Tag: 1661808382-903233 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour --- arch/powerpc/mm/fault.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 014005428687..c92bdfcd1796 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -450,6 +450,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = find_and_lock_anon_vma(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_read_unlock(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_read_unlock(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -526,6 +564,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); From patchwork Mon Aug 29 21:25:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 469F2ECAAD4 for ; Mon, 29 Aug 2022 21:26:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D44C394001F; Mon, 29 Aug 2022 17:26:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CAB1D940008; Mon, 29 Aug 2022 17:26:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAA3B94001F; Mon, 29 Aug 2022 17:26:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 98D4F940008 for ; Mon, 29 Aug 2022 17:26:24 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6CFF7802D5 for ; Mon, 29 Aug 2022 21:26:24 +0000 (UTC) X-FDA: 79853913888.17.1B084A5 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf03.hostedemail.com (Postfix) with ESMTP id 275FC20041 for ; Mon, 29 Aug 2022 21:26:24 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id n30-20020a17090a5aa100b001fb0c492d5eso3877286pji.3 for ; Mon, 29 Aug 2022 14:26:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=WpdYqLbUAWRRR7NpWeIwifObGgR3/WEcKlSas1duxWs=; b=A8iXlM7kWh/+WLxmr7Wxu633pJ33WbnreWCJkhO2mN51OTMuCRIiZbXSJTMwU7nnLp 8HrWKhSa16qOE9IHNHTIDFaumIghV/X3vZxzBwxJyopuUJchdMXN0oUY2EJvc97qw23/ 002SGSF6e/GfvGXcSJZJ9OH52bwQ+17bBv1qwUEkh+/hxiA3XGJZS3QMw2bsNQo6K8n5 ASVJIXUmfSCyDSBXlWu7+9C2NTyQxO6WdEpOrWiiN0G0XOZ0xYY++9szDB3PW4z/iVNr kXUF0efo+Jl4A1NDPbFsv2i2F8cuEbcHbyZjyx1JQrC6JX3bDooGkyCEfiAtjL3LX2kX L4Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=WpdYqLbUAWRRR7NpWeIwifObGgR3/WEcKlSas1duxWs=; b=szCDYfgk5NL8GoBMOY6Se/8pP7HG7YScZLqcvrEoPMhlpB65VGdRDYPqks/neqXRZQ 5z1QXwPmG9mgz/L4f6NO9Jp3v9eh9PB66C6N4L+YcFIFN9ejbNH6NSiyKwaC4o2ctci1 9UFhKqVtt+YT4n9PxSQlPyl671PHQy/gbRSCnnQZaMdz3YOy6ZJqu2WKCqu320I1wYzv dxqvT1wY4X24bOc+znvWR/RJvVfPYeWfNJcr/VAi5su1RFObS2f1oZQtYfb66a4O0W89 0KeHMsqg4jgEr19Q2glP+QE0a2uKbBg4sSOUlIacV8IdXkaiUJaUgNNDpgj9xgXWCFid R5zg== X-Gm-Message-State: ACgBeo0XW3mccbAKZc8QCFBM5qZgcbFCpu6ka17+pTV7tBAEcUC8lks3 pTnLydA0OQpm/DaiLfB02DZdzJ68Bj0= X-Google-Smtp-Source: AA6agR7BIF7zDBFRMtaQFIX+dYsK3ioA4U218I6Y8n5CHeAbnavsVuqAEK5kMo+KsvAOcZae2RDMSpS8XIE= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:a17:903:2d1:b0:171:3773:b95 with SMTP id s17-20020a17090302d100b0017137730b95mr17499271plk.173.1661808383315; Mon, 29 Aug 2022 14:26:23 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:30 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-28-surenb@google.com> Subject: [RFC PATCH 27/28] powerpc/mm: define ARCH_SUPPORTS_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808384; a=rsa-sha256; cv=none; b=aaMSSoQTi1R69N+mBKswafEcWt68TW/S0MwwV38BPkNRsunMpKUB1Dkyk7Vsj0DuDHp5h2 89alhQjy7jY0kjG3K+LvbqcIcGBZs8k28kmnTKVYIqAz6Hs+UjQloFAIbAYUNsrHd4mtAP F7tQAowIsOjQBFXRGshsVALrtY9OnP8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=A8iXlM7k; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3_y4NYwYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3_y4NYwYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WpdYqLbUAWRRR7NpWeIwifObGgR3/WEcKlSas1duxWs=; b=lSbiTRKsv/DWBUlMVIRAnhfu5xUSRGXBKKevqiFIZbP3zpjnCwloOD7Tt+mXeVQ5nZbymj UD/cXhoRbp0GhY/yfYgPYJNfvL7lQTpr5E3O95rdtJ4WOMLOO4SSTStukMYQwiFVMSPR/o sH9sZ99RTZqEGEIdiYIy/pSWUB2raiU= X-Rspam-User: X-Rspamd-Queue-Id: 275FC20041 X-Rspamd-Server: rspam03 X-Stat-Signature: wtz4wp1b4s7xjyhtbjkwmbekuhru8njk Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=A8iXlM7k; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3_y4NYwYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3_y4NYwYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com X-HE-Tag: 1661808384-952720 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laurent Dufour Set ARCH_SUPPORTS_PER_VMA_LOCK so that the per-VMA lock support can be compiled on powernv and pseries. It may be use on the other platforms but I can't test that seriously. Signed-off-by: Laurent Dufour --- arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index fb6499977f99..7d13a2de3475 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT_SPINLOCKS From patchwork Mon Aug 29 21:25:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 12958452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BDE6C0502C for ; Mon, 29 Aug 2022 21:26:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EABC940020; Mon, 29 Aug 2022 17:26:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0234A940008; Mon, 29 Aug 2022 17:26:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D416E940020; Mon, 29 Aug 2022 17:26:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B92D9940008 for ; Mon, 29 Aug 2022 17:26:25 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A2A02A0A03 for ; Mon, 29 Aug 2022 21:26:25 +0000 (UTC) X-FDA: 79853913930.16.3C7FA2F Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 50A1740041 for ; Mon, 29 Aug 2022 21:26:25 +0000 (UTC) Received: by mail-pf1-f202.google.com with SMTP id v65-20020a626144000000b0052f89472f54so3616090pfb.11 for ; Mon, 29 Aug 2022 14:26:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=T8fB8hXi5puWeYGLUKktDcvprMtt+Dm3Upe4kr4atwA=; b=YPMK3AJpw0lWwzYZfDLzh3K7/2W07auDNNnlJ3qDkZJloq+AWVG8rnBi3EjSTi8Sdc 7WJv0veKl0lZAyzfelPpHbd7cV4CAy5dCoMcdYFQ7CqbidAfsv5F6022/J5YgvQf67DO L1YTRMppSCyvCaIew0iiAysCdKjsWOFUDjcZZGzRH+sdcxnSvug7vs9F56KVsee2c0Zo YLbMsX94AWYL+h4w9bE7JCdOjOkowcodvSgMnxQ2NHaXBQhv7Fitxu5qAmsw46U72GSa SpD2ueQh2UQYjoAvP3IZCnzfNryUU6OHaypVe7QLBtTEl+UaqGMhHtnVjouLDn/cSCG8 82Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=T8fB8hXi5puWeYGLUKktDcvprMtt+Dm3Upe4kr4atwA=; b=rfoKKW3Pr/tj/Uqy7xegi0DpcfMDH/62GcNhsHDBRhxRrqZPD/E0RrvmqNw0K9w70b KIFSNFLQUemJGNLyYy22CyKrEW+5Iqmr3sxti8Dba4PbcxCQpol9vesXeN7Z/erW/on5 x0Wra/AAEe/klknbzBX7dNEXJFHBvEE0Wyw7MgEZHS00pOKbJ2DUGfDIqqHjc9ve8SVf aVtJV6i5UJr8O+doO5M6hIsVTIpjwCPtsGyLAty8qnnleDuAHsf8EW+Hx5HBQqgv2Eur SS0ncSnGpBQGj/cWzIf1ufaQL2SNT1Ec6BCi5iZdy7vQjUWIjybpqMycGag3XJVGdyOD OoMw== X-Gm-Message-State: ACgBeo0FIUAGT+/2MKJJrirTIVb6VZ0mtVKhSwYnEsyFSYfGjuwExAQK VhBPoEu06s+/2eup7+V29WjzKTWBOyc= X-Google-Smtp-Source: AA6agR5dzmEiiCM1ZfmS4IP8jubPYXR49V7KmmeKiHJ1f+/Bhg9nkxqem44jNkystynSPzi1e7aYTBBx0e8= X-Received: from surenb-spec.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3e30]) (user=surenb job=sendgmr) by 2002:aa7:8b44:0:b0:537:a35d:3c11 with SMTP id i4-20020aa78b44000000b00537a35d3c11mr18188023pfd.76.1661808384779; Mon, 29 Aug 2022 14:26:24 -0700 (PDT) Date: Mon, 29 Aug 2022 21:25:31 +0000 In-Reply-To: <20220829212531.3184856-1-surenb@google.com> Mime-Version: 1.0 References: <20220829212531.3184856-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220829212531.3184856-29-surenb@google.com> Subject: [RFC PATCH 28/28] kernel/fork: throttle call_rcu() calls in vm_area_free From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, riel@surriel.com, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661808385; a=rsa-sha256; cv=none; b=EifroCnm50GzxnW9I+rHKnKULNgS58oWsCDv/jT6nyQHIx9qDjI5ON9kSvo5KsJ2vG4ztg 40BrhW9Ni2cJMBEpV7uRrIXyqAi1Cv1rCEg20cRC8bvjvDC43OUHGnAOZeJ4GKrqDqSqvW W/UqU7WY/9IcgkbLn3scrSMJFYgJow0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=YPMK3AJp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3AC8NYwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3AC8NYwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661808385; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T8fB8hXi5puWeYGLUKktDcvprMtt+Dm3Upe4kr4atwA=; b=PkDWapaB1SdhJX4I7kgDEHe5j6klE/OgJcpqHXzub+JC8zu0ZUYzrFN6suSgcPhTNfNuP3 sM+jX7V9MBGGbeIT+3f+w5pbT4jYZuQaVxVhkiuWTgFOEp6JFy6VSn1ukP8HKd0mwdqiGl 8wimUKo49t4xlyriw5wHy9b6h/E6ivQ= X-Rspam-User: X-Rspamd-Queue-Id: 50A1740041 X-Rspamd-Server: rspam12 X-Stat-Signature: yxj3c9mxiqbbu4ruddj5e1q6nigszii3 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=YPMK3AJp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3AC8NYwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3AC8NYwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com X-HE-Tag: 1661808385-750533 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. To minimize that impact, place VMAs into a list and free them in groups using one call_rcu() call per group. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 1 + include/linux/mm_types.h | 11 ++++++- kernel/fork.c | 68 +++++++++++++++++++++++++++++++++++----- mm/init-mm.c | 3 ++ mm/mmap.c | 1 + 5 files changed, 75 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a3cbaa7b9119..81dff694ac14 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -249,6 +249,7 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +void drain_free_vmas(struct mm_struct *mm); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 36562e702baf..6f3effc493b1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -412,7 +412,11 @@ struct vm_area_struct { struct vm_area_struct *vm_next, *vm_prev; }; #ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ + struct { + struct list_head vm_free_list; + /* Used for deferred freeing. */ + struct rcu_head vm_rcu; + }; #endif }; @@ -573,6 +577,11 @@ struct mm_struct { */ #ifdef CONFIG_PER_VMA_LOCK int mm_lock_seq; + struct { + struct list_head head; + spinlock_t lock; + int size; + } vma_free_list; #endif diff --git a/kernel/fork.c b/kernel/fork.c index b443ba3a247a..7c88710aed72 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -483,26 +483,75 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) } #ifdef CONFIG_PER_VMA_LOCK -static void __vm_area_free(struct rcu_head *head) +static inline void __vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } -#endif + +static void vma_free_rcu_callback(struct rcu_head *head) +{ + struct vm_area_struct *first_vma; + struct vm_area_struct *vma, *vma2; + + first_vma = container_of(head, struct vm_area_struct, vm_rcu); + list_for_each_entry_safe(vma, vma2, &first_vma->vm_free_list, vm_free_list) + __vm_area_free(vma); + __vm_area_free(first_vma); +} + +void drain_free_vmas(struct mm_struct *mm) +{ + struct vm_area_struct *first_vma; + LIST_HEAD(to_destroy); + + spin_lock(&mm->vma_free_list.lock); + list_splice_init(&mm->vma_free_list.head, &to_destroy); + mm->vma_free_list.size = 0; + spin_unlock(&mm->vma_free_list.lock); + + if (list_empty(&to_destroy)) + return; + + first_vma = list_first_entry(&to_destroy, struct vm_area_struct, vm_free_list); + /* Remove the head which is allocated on the stack */ + list_del(&to_destroy); + + call_rcu(&first_vma->vm_rcu, vma_free_rcu_callback); +} + +#define VM_AREA_FREE_LIST_MAX 32 + +void vm_area_free(struct vm_area_struct *vma) +{ + struct mm_struct *mm = vma->vm_mm; + bool drain; + + free_anon_vma_name(vma); + + spin_lock(&mm->vma_free_list.lock); + list_add(&vma->vm_free_list, &mm->vma_free_list.head); + mm->vma_free_list.size++; + drain = mm->vma_free_list.size > VM_AREA_FREE_LIST_MAX; + spin_unlock(&mm->vma_free_list.lock); + + if (drain) + drain_free_vmas(mm); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +void drain_free_vmas(struct mm_struct *mm) {} void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, __vm_area_free); -#else kmem_cache_free(vm_area_cachep, vma); -#endif } +#endif /* CONFIG_PER_VMA_LOCK */ + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -1137,6 +1186,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK WRITE_ONCE(mm->mm_lock_seq, 0); + INIT_LIST_HEAD(&mm->vma_free_list.head); + spin_lock_init(&mm->vma_free_list.lock); + mm->vma_free_list.size = 0; #endif mm_pgtables_bytes_init(mm); mm->map_count = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index 8399f90d631c..7b6d2460545f 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -39,6 +39,9 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK .mm_lock_seq = 0, + .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), + .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock), + .vma_free_list.size = 0, #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, diff --git a/mm/mmap.c b/mm/mmap.c index 1edfcd384f5e..d61b7ef84ba6 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3149,6 +3149,7 @@ void exit_mmap(struct mm_struct *mm) } mm->mmap = NULL; mmap_write_unlock(mm); + drain_free_vmas(mm); vm_unacct_memory(nr_accounted); }