From patchwork Mon Jan 22 07:13:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13524877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EF4DC47DAF for ; Mon, 22 Jan 2024 07:13:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82B688D0005; Mon, 22 Jan 2024 02:13:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DAB58D0001; Mon, 22 Jan 2024 02:13:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67BC88D0005; Mon, 22 Jan 2024 02:13:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5331C8D0001 for ; Mon, 22 Jan 2024 02:13:31 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 20A27A1910 for ; Mon, 22 Jan 2024 07:13:31 +0000 (UTC) X-FDA: 81706081422.12.2A11914 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 95C6520007 for ; Mon, 22 Jan 2024 07:13:29 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HUsdT70D; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3mBWuZQYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3mBWuZQYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705907609; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=oXyU4pkvCRYAyA2TUvsUC8dh5sAPzBBH4qagn6tOpF0=; b=tfXN+j3/qnE7tTEs6UPwtVEEOFZ827a2hhNzHkxcmAIriTR1jerBTv5ujqTz6QoQEPIs+4 1uAKtnFf5EHjPZQsa7cTxhnO7kdALnTWZc51amTDg/aG0NDtz1C7YS2KvNFhQlGSRC7U0C 0tRfYPCtBcIMjlFRyaBJvTyNdhMIM4U= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HUsdT70D; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3mBWuZQYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3mBWuZQYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705907609; a=rsa-sha256; cv=none; b=EFI3cD04kJ9+sstYqqZ9v/nuyGxigx9LvvzJ/klB8eIGnJZIrTu0jegIlyKUKV/+3zoRLJ /IQvstPzPQ/TKygZDvx/kgt5oJ1Ngptk3PJiDOrgP+p3eHKOHISMmPLwegZF2zFzp1Lu5V msmVsUM8dDHlsbHuQQrL45SZIvgWpYc= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5f38d676cecso43397357b3.0 for ; Sun, 21 Jan 2024 23:13:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1705907608; x=1706512408; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=oXyU4pkvCRYAyA2TUvsUC8dh5sAPzBBH4qagn6tOpF0=; b=HUsdT70Df6ln/jWGIhnnTHuUkIq3034Zxf+0tmEdXDglW/SLrFMRSnTmHGUUadLlvv THyQZmfEeP2nSOYXHMTclZluEjPa+2tnQxgdXtxz3UUvAHDzWD1N14cHcIM5ARTFUjUr Zf7uu8+eyTBs9PWAkioWji8KTv1JX9ZCboyWLocmbL0a9sxGU+B7x+hafxfhlhKD3lnA kU1YSOycDOCZfypPzjJ6EiIFLV2KThxNLNXalvY59WhQ7jJOVjHgfyOcQDPe3JqQ6gWK e1IDRRxPmhmkkexknLfRJ82lwKfWulnriIxBCSgJ01FTLeW1tshvRMeMAKzMIrhZbEMu kMyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705907608; x=1706512408; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=oXyU4pkvCRYAyA2TUvsUC8dh5sAPzBBH4qagn6tOpF0=; b=vVXxmtF4hhTmFjXyjHjcM7bw02EOJTsQclaTviS8/vLIhg5i0AtlN2DrduvSS2apk3 bvJRFRcqGdxxud/IVJ/RTwM/hBL1ekaKrrimeTonTQgOhZhEfGBWkjuoNJkJnh2uYzWc q0m8BnySFbWpcuplCR40Ijn6zL/AU9yVRF1XHx3PpkDGwh/dtchctvEuOrFcVyragvHp oZepTzUXKZHR92JCmEWaZ4x6dpdbUhZaM57SPcYFjK4Ll/jllBmYTtQCkxGiUu3gbGdK Ki/ZGzFmTjvsMZgBgSS5KdT/I+dN1v06KnJ6axbFNxLtItjxrEJx6Rwm9K+1/oTHgI36 axXQ== X-Gm-Message-State: AOJu0Yxi5pNgvPT4eozf+X6DCKlf0u/JMIgtrs7IWfKoREv+Vd2GmRkE msR4Wrj0g8BV6xQOAWi5DvxE3wI6AiDNhrindNZoaKSKDQEfuVSlE84CRy1zH+GfZUhuwra9SJ+ g/Q== X-Google-Smtp-Source: AGHT+IG8d1Gn9VWPvChN4p2/RHkaCghVVAROZ7VOgDGN0KGJpb0phlIzNaSwlCWzvJeInLF/vdqx3fCBkG0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:4979:1d79:d572:5708]) (user=surenb job=sendgmr) by 2002:a25:e90c:0:b0:dbe:49ca:eb03 with SMTP id n12-20020a25e90c000000b00dbe49caeb03mr1889847ybd.5.1705907608657; Sun, 21 Jan 2024 23:13:28 -0800 (PST) Date: Sun, 21 Jan 2024 23:13:22 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240122071324.2099712-1-surenb@google.com> Subject: [PATCH 1/3] mm: make vm_area_struct anon_name field RCU-safe From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, dchinner@redhat.com, casey@schaufler-ca.com, ben.wolsieffer@hefring.com, paulmck@kernel.org, david@redhat.com, avagin@google.com, usama.anjum@collabora.com, peterx@redhat.com, hughd@google.com, ryan.roberts@arm.com, wangkefeng.wang@huawei.com, Liam.Howlett@Oracle.com, yuzhao@google.com, axelrasmussen@google.com, lstoakes@gmail.com, talumbau@google.com, willy@infradead.org, vbabka@suse.cz, mgorman@techsingularity.net, jhubbard@nvidia.com, vishal.moola@gmail.com, mathieu.desnoyers@efficios.com, dhowells@redhat.com, jgg@ziepe.ca, sidhartha.kumar@oracle.com, andriy.shevchenko@linux.intel.com, yangxingui@huawei.com, keescook@chromium.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 95C6520007 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gg9ii7bbk61q4yosm7zbb75z1yujojdb X-HE-Tag: 1705907609-849559 X-HE-Meta: U2FsdGVkX18RWJeslVuqGFpBhQ24dall8CKIcIWtaQdpeUm15JpgyLLtlca9U1HS/4+0G4CtSZOUjMRmcjFCJ/Ec+QwKtYOjxLsp7N2NcAVS78mOtl0VNvzxm40qf2oAF8VLPmbcD5K20aub+5c9cTCziixbXBrlWMKIdbQgBOzhO1a6kzSz5Tmkl/d0dDdRKGVngJnhIZ0PJWw/hln+xYVWPotrVscXa/EJfmoFQ33eF6QWKwoejQlP+Y6cg/ehw7txGtz6Min7RWEML8cOia/B9nl/NpWFo5Wxtt2boKSBS/fC2fA8EpJiKltQBqNENM0q27IaaQZJuw3ZcKU7H21cNzEX8i4iUxCmzA34p4VbrXgivdvFvcpP6QlhGRQo2MhDqczpF6B9mrhtk0T9RuLjCTaD3Aqx1ko6YYWYgjKDUAUB5pyQnME9LfazK4qJP6t3+yMUdEf5Ig3/TEpzP0PhbcHYcafstihCzRkCBLpa7PYCwP21s/S9nfnmLffNCq/5xuYbxSEh5VViE/kUgCIyEC6TKqHFBWQ3FYAwjiheLBdaVHbUcq9ZqsEvdthPkCD3CKTXktko3rR0IQFDlEvgZ9KaLiSrZ+Ob3tsQzjT/xEu5hZ1HkfGCVKx+/v994eVfXT7//BRuLQNy4vMShUKhR+f13NEb0sAjizYsyJ+LPgFsCyE2Z1lQmRNyCQtGsy3P+w00uV4uNsctRPQGJu2yNdbDgQCePG7TeGUeyel5W27n5uMwMN7S9sJH0oTkVb57kGTz+LqUyKkGI4YT7AliDWBxIIYLLNZmNyHKHBIN2FXUewguPa+VDqsyiHdFKe1loouKhPettNw9cYj8CpMg8lpFvUX+tKTGtTzyYMs/EWL8gdUIbMu1lM8ZuBwk/zDKxnNj70vZRiOPs2H0Z6evOMwZCbxdNgsdnaFcNz0ZlARz8rDwjMCQoadylHgxDd7/NKfv0OrPVGKYJ3L r5ub4QB4 vxz5O2+k9qP65hn7nG84MOd98xZDhP4ICK6U+YyXpzbNRrO2TOI0na838Zpwnjhmn7clY2Gp337ttTWeUvZvftsji+1b8z7cmELlm0Uxuh4VmOCRrhgevzKI8AHtczIAmDONXi31knmU29hE2T1vslHcITHVIO3LFyx8EHF5uA2KqwYdkQsPfovGZrYUIPwGPGTDrDiqrgsWRfWP+5OtF1hjoYVhYUANfPZGbHbEE3cy3ozt+/1xIviGUPmqDN3qTkoqgBz1hMr8zBcBmI+/R8Pz5anguv3fDVEbNGUOu8AfOuQDdp51fIfD8KqZ0VL9mALAldGGo4AxLqCSzxPEJ90l8QcMWWuWYFUiZhARt8IFD8gDuM42YnXOOhkHktwuthWRDB4eALH1mApwFjMugmo5Jw+Li+R9su1FhEELINp44wPhkFoREdK+epkzpGV6gg4tIBVb0KPgl/OSt3++wa6nZMQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For lockless /proc/pid/maps reading we have to ensure all the fields used when generating the output are RCU-safe. The only pointer fields in vm_area_struct which are used to generate that file's output are vm_file and anon_name. vm_file is RCU-safe but anon_name is not. Make anon_name RCU-safe as well. Signed-off-by: Suren Baghdasaryan --- include/linux/mm_inline.h | 10 +++++++++- include/linux/mm_types.h | 3 ++- mm/madvise.c | 30 ++++++++++++++++++++++++++---- 3 files changed, 37 insertions(+), 6 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index f4fe593c1400..bbdb0ca857f1 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -389,7 +389,7 @@ static inline void dup_anon_vma_name(struct vm_area_struct *orig_vma, struct anon_vma_name *anon_name = anon_vma_name(orig_vma); if (anon_name) - new_vma->anon_name = anon_vma_name_reuse(anon_name); + rcu_assign_pointer(new_vma->anon_name, anon_vma_name_reuse(anon_name)); } static inline void free_anon_vma_name(struct vm_area_struct *vma) @@ -411,6 +411,8 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, !strcmp(anon_name1->name, anon_name2->name); } +struct anon_vma_name *anon_vma_name_get_rcu(struct vm_area_struct *vma); + #else /* CONFIG_ANON_VMA_NAME */ static inline void anon_vma_name_get(struct anon_vma_name *anon_name) {} static inline void anon_vma_name_put(struct anon_vma_name *anon_name) {} @@ -424,6 +426,12 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, return true; } +static inline +struct anon_vma_name *anon_vma_name_get_rcu(struct vm_area_struct *vma) +{ + return NULL; +} + #endif /* CONFIG_ANON_VMA_NAME */ static inline void init_tlb_flush_pending(struct mm_struct *mm) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8b611e13153e..bbe1223cd992 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -545,6 +545,7 @@ struct vm_userfaultfd_ctx {}; struct anon_vma_name { struct kref kref; + struct rcu_head rcu; /* The name needs to be at the end because it is dynamically sized. */ char name[]; }; @@ -699,7 +700,7 @@ struct vm_area_struct { * terminated string containing the name given to the vma, or NULL if * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. */ - struct anon_vma_name *anon_name; + struct anon_vma_name __rcu *anon_name; #endif #ifdef CONFIG_SWAP atomic_long_t swap_readahead_info; diff --git a/mm/madvise.c b/mm/madvise.c index 912155a94ed5..0f222d464254 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -88,14 +88,15 @@ void anon_vma_name_free(struct kref *kref) { struct anon_vma_name *anon_name = container_of(kref, struct anon_vma_name, kref); - kfree(anon_name); + kfree_rcu(anon_name, rcu); } struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); - return vma->anon_name; + return rcu_dereference_protected(vma->anon_name, + rwsem_is_locked(&vma->vm_mm->mmap_lock)); } /* mmap_lock should be write-locked */ @@ -105,7 +106,7 @@ static int replace_anon_vma_name(struct vm_area_struct *vma, struct anon_vma_name *orig_name = anon_vma_name(vma); if (!anon_name) { - vma->anon_name = NULL; + rcu_assign_pointer(vma->anon_name, NULL); anon_vma_name_put(orig_name); return 0; } @@ -113,11 +114,32 @@ static int replace_anon_vma_name(struct vm_area_struct *vma, if (anon_vma_name_eq(orig_name, anon_name)) return 0; - vma->anon_name = anon_vma_name_reuse(anon_name); + rcu_assign_pointer(vma->anon_name, anon_vma_name_reuse(anon_name)); anon_vma_name_put(orig_name); return 0; } + +/* + * Returned anon_vma_name is stable due to elevated refcount but not guaranteed + * to be assigned to the original VMA after the call. + */ +struct anon_vma_name *anon_vma_name_get_rcu(struct vm_area_struct *vma) +{ + struct anon_vma_name __rcu *anon_name; + + WARN_ON_ONCE(!rcu_read_lock_held()); + + anon_name = rcu_dereference(vma->anon_name); + if (!anon_name) + return NULL; + + if (unlikely(!kref_get_unless_zero(&anon_name->kref))) + return NULL; + + return anon_name; +} + #else /* CONFIG_ANON_VMA_NAME */ static int replace_anon_vma_name(struct vm_area_struct *vma, struct anon_vma_name *anon_name) From patchwork Mon Jan 22 07:13:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13524878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E1FBC47DDC for ; Mon, 22 Jan 2024 07:13:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E93E16B0071; Mon, 22 Jan 2024 02:13:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E44248D0006; Mon, 22 Jan 2024 02:13:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CBD398D0001; Mon, 22 Jan 2024 02:13:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B72F06B0071 for ; Mon, 22 Jan 2024 02:13:34 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 94D44C08AA for ; Mon, 22 Jan 2024 07:13:33 +0000 (UTC) X-FDA: 81706081506.04.7F05923 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf11.hostedemail.com (Postfix) with ESMTP id DAEF24000F for ; Mon, 22 Jan 2024 07:13:31 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VQoQ2Xpn; spf=pass (imf11.hostedemail.com: domain of 3mxWuZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3mxWuZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705907611; a=rsa-sha256; cv=none; b=T5m8wJlbBJhtGxJHQW/xREHov2pyI3XtZ8SJ2Wxmbp4jPEVIGFqH6dp36LnwcUNDeH0hVE EEh7yJjJAs8GvzAQZVp1TZr1gTurr1ao0HLziZ6PyFXNDzN/NObdzpaPdC89g6mpwNjjvy IZQpX3qPDvj2LuvnQP/mVicQl31ZKes= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VQoQ2Xpn; spf=pass (imf11.hostedemail.com: domain of 3mxWuZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3mxWuZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705907611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=jnVksyGD1H6yO/ncPSGZUKrlV8BoT5X2Q/W0XWs7pZv49R0NPoyqWXa0OUVy8djE/HNzVT 7praoifKvLLq0bRNVPa46cJIh26zsGRPvejGPakeAk6h27gaLlg59g5XHPqZkC+Hlk9R56 WP6iCbh5j4i3/4yOFeWu8AlM4mqnd64= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5ffee6fcdc1so7648777b3.2 for ; Sun, 21 Jan 2024 23:13:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1705907611; x=1706512411; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=VQoQ2XpnZRSpuqclyVNqsYqWDf6z30eA18iyVMcecDXVn1yGhVo/f86QRLImEiAhzr 1JhKDe02dO9Ou5PFD9dXlewi3d6SCGhG+V/1kUbayWYAeIZncAYX3YvsLRdB0XL7sHSf 9LTqSqAPDj9qkDtbFizTymzxN81K15vgAzV+Aa/4STFS6QsZ/hE2s+z01JpgIaw/0yKZ UnB+53KV1Fn9RSHbafqnMP7k4Myvn6fHz8EnxFVHwwRGAzDGTmFMyat5hbz2ttxOPwNI WDJQNRF+D+pNBs/8KFLOqNS765d+SGpjsKf7V0Y+i6VX4AwM7lGf0UG7oYhCW7I23e15 aWRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705907611; x=1706512411; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=Wf2rluJdq/INw63y5TFgYiAQ+zsc+Q6j3uDwH1+XaEaaPT9lF3H13OhixSxwXfqoZA X1XFY1cI1kSj+GdorCpmX2EKbEkKv9sUbIM1GPR8lJvsvjwU9vxVUNq7LxmirMr7pQrL QIbS+ZUTt70KRiEWn7QrFx5OD8Uy1TjH5qemybrDrdLlLJAGHSXzD71bsOOzOf81kU9c qbw/nTqONXfq7T3hy6Ukhc+asvfoEHRtmbYtmv3sqp9n2Nwx6q5XebrzZfkTW30YrQTv 6rIox6AmbP6BHMHCXDxh5PmCs4Ivbs1EaUQhdGxVwJd5pIYyw+wc4v7UtJKUK2EVDv3H +wxQ== X-Gm-Message-State: AOJu0YyQnN68Xl8VyW76kQFAVmhi0Hh9jvNQfhARVQ7tp6T/P6mMj+fS xcGNUxCrOWEJlAQfGJzyY6JBuW8oZPZvgPf9EE9BKsJvpdpGHIFIrMQZmdkLWAcccbUVbPiSa86 hRg== X-Google-Smtp-Source: AGHT+IFDFx+rU5MMV2joyEArNdxSbn9+Uqv0sdgUNYpKz6Fm9lpqOqNw6v9H//7PI5LcIGO1EwFhJ6qHlkY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:4979:1d79:d572:5708]) (user=surenb job=sendgmr) by 2002:a81:a096:0:b0:5e6:27ee:67fb with SMTP id x144-20020a81a096000000b005e627ee67fbmr1310854ywg.4.1705907611085; Sun, 21 Jan 2024 23:13:31 -0800 (PST) Date: Sun, 21 Jan 2024 23:13:23 -0800 In-Reply-To: <20240122071324.2099712-1-surenb@google.com> Mime-Version: 1.0 References: <20240122071324.2099712-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240122071324.2099712-2-surenb@google.com> Subject: [PATCH 2/3] mm: add mm_struct sequence number to detect write locks From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, dchinner@redhat.com, casey@schaufler-ca.com, ben.wolsieffer@hefring.com, paulmck@kernel.org, david@redhat.com, avagin@google.com, usama.anjum@collabora.com, peterx@redhat.com, hughd@google.com, ryan.roberts@arm.com, wangkefeng.wang@huawei.com, Liam.Howlett@Oracle.com, yuzhao@google.com, axelrasmussen@google.com, lstoakes@gmail.com, talumbau@google.com, willy@infradead.org, vbabka@suse.cz, mgorman@techsingularity.net, jhubbard@nvidia.com, vishal.moola@gmail.com, mathieu.desnoyers@efficios.com, dhowells@redhat.com, jgg@ziepe.ca, sidhartha.kumar@oracle.com, andriy.shevchenko@linux.intel.com, yangxingui@huawei.com, keescook@chromium.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DAEF24000F X-Stat-Signature: 9j8d93ik3fpjqfifpttoqnqqz3fnkzka X-Rspam-User: X-HE-Tag: 1705907611-158232 X-HE-Meta: U2FsdGVkX1+baU6+ZGPhc/sLlxJJiAolEN5m9X4DAGzxIPTp7+shdJuODa8UvFeKtNwwVUrMA8P0QfjWOvaGi4p/AaDzZ3OKDT60ISBhs2iaoGUg4slc8icteFT1zocGZtKNWNs2e3k7fO7hjhNNF8qiVrKF/2EqzoNnWyNlYF0venzCSwz5u2Y0xFxlfM6rn+Bjfu1z10Bj96e+9opHHovPyjlxzd9v3e8o2xbHRaZp6fy8h/M0Bp4zTUS+uVkVaZhwdgT4QAjhWYKR5mej7S7mfL/5H4EUgvj1EUE7oJHnqiJrY5hDQNPuiF6ySZc2BOvMPrvyntpLo6IvtDFnC5IzSNuSA0umL+oG33ulTSkYrLmdsW7kXVFUgHkK3l2aurn9h9upu8mCDpwHQpjlqAM4CZ3JL4hd+zUXxhz1s1dveX8+KuovBP0ryPgneUuGFkFJTzZl1jiQIm2zeXqbCqS/xHKOtQZ8uXdM8ymvOOV8vWg0mvYIAMPrMkyrDoT/v9JAoxJ2AWN2NmojJC7520dfkZxzLSGpJftXL5irvgzIJcLkB8d0AefMk1zKsyBdXpGtatuwwcuBdsnx0GRYB3i4uapUaP3Ey7AMeRZiVG8fQjk4MUgci5lrHiPEkyRuc2+kUdR31GGJkt4+smHu/XUiughMeRCic3/iD/A0vedHSNdVHy8zIYVQZ1/2MoE4zMkn5kCOuNUWpy/r2mCK/bFcwjPLtleHHjqTqAQ88PW70EFJb9yCaP3q6XkoKOjxczG5jQPW1vA2Q5TTEONdBT7+ZED56SCxBpUINCKZpFYgp7HOdnBhLYpeIycOpXmGmirlFDaSSTwizi1XDftCzr6azl4odqY8QJmLpBC9uaz31MWy+I2EDzpkOThkUryO1JW/Pt30MYfMijYMU5nXJFkIb7IkDCWHzO692RnzxGxGA9gqWEoTlZvZT8wDaKv0k1/09Z1bQpKhgz3hhOJ qsIBvdPM bbeK1wcQKeaqJGaUzE0g9xiOJQ/CZlfFjZYS/dhHI5M0m2UB4SorsPYKs7Wm1/f8IuNHezDzlUS5SkJrwUru17lfmdtx4/aSLxMwBljF+mCkDqHley6BC4qz/SQzWzntJDXtlr8NmuulQULdnJIy4c2agu5VOsD/qf8X033clyRSkPfzzmdB2uLXac9uKXvfzKquhDNJ57iMlHmOGTJ1afALUlTJPFArDtpWKxxSlFV9TCqsKDoq14t7DAnSXOdqImupIwLpdmKQu3NU12qjxNLv7YF1IblzqaXlvxoCqIGn9NO8aXUkjp9uP5DacofxBPaVbIji0UHLFBLMW4H1Qsm4cNOZR9G7VDBdu6SwqZDH7Na3KJmfgJqqbev+uMeWisb4MAT0qvbT5lc8aJKNWMr4NXx9PtU6lMUNDw8hslvKhDOFQNE9G1QRhdmox2OIgc2iEFO2BQqVvf6GscrmJn6ioxBRD27s0ZP/CNGp2VjjSc7AB1/o1UgFiBXwXLDFc9qE3dPaCXksYWrmCoBannsJ/teYXN46LncIBNoME8YZKPAW9Ef7dgueOobL/lB69v26uv1qWHoXd9yN5IPuhInIC6JTu5utq4r7i2xzt3iGbUeC9SnIq12lgbYjdbKLMCuKOTSmzQUpJi4SPAjfLGyFID/A0ZoR8ofioAelbvaxFoj/iLqsPd2QnGHGW9BdZu/BwspWgmNvy3zrDXjCYSw8cYoke/2jquR/h X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide a way for lockless mm_struct users to detect whether mm might have been changed since some specific point in time. The API provided allows the user to record a counter when it starts using the mm and later use that counter to check if anyone write-locked mmap_lock since the counter was recorded. Recording the counter value should be done while holding mmap_lock at least for reading to prevent the counter from concurrent changes. Every time mmap_lock is write-locked mm_struct updates its mm_wr_seq counter so that checks against counters recorded before that would fail, indicating a possibility of mm being modified. Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 2 ++ include/linux/mmap_lock.h | 22 ++++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bbe1223cd992..e749f7f09314 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -846,6 +846,8 @@ struct mm_struct { */ int mm_lock_seq; #endif + /* Counter incremented each time mm gets write-locked */ + unsigned long mm_wr_seq; unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 8d38dcb6d044..0197079cb6fe 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -106,6 +106,8 @@ static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -113,6 +115,8 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) { __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -122,6 +126,10 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ret = down_write_killable(&mm->mmap_lock); + if (!ret) { + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); + } __mmap_lock_trace_acquire_returned(mm, true, ret == 0); return ret; } @@ -140,6 +148,20 @@ static inline void mmap_write_downgrade(struct mm_struct *mm) downgrade_write(&mm->mmap_lock); } +static inline unsigned long mmap_write_seq_read(struct mm_struct *mm) +{ + /* Pairs with RELEASE semantics in mmap_write_lock */ + return smp_load_acquire(&mm->mm_wr_seq); +} + +static inline void mmap_write_seq_record(struct mm_struct *mm, + unsigned long *mm_wr_seq) +{ + mmap_assert_locked(mm); + /* Nobody can concurrently modify since we hold the mmap_lock */ + *mm_wr_seq = mm->mm_wr_seq; +} + static inline void mmap_read_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, false); From patchwork Mon Jan 22 07:13:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13524879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0A26C4725D for ; Mon, 22 Jan 2024 07:13:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78F4C8D0001; Mon, 22 Jan 2024 02:13:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7188E6B0088; Mon, 22 Jan 2024 02:13:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51E348D0001; Mon, 22 Jan 2024 02:13:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 31FE76B0082 for ; Mon, 22 Jan 2024 02:13:36 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F4209140745 for ; Mon, 22 Jan 2024 07:13:35 +0000 (UTC) X-FDA: 81706081590.17.807F471 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 467B6C0002 for ; Mon, 22 Jan 2024 07:13:34 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mTeY0rEd; spf=pass (imf10.hostedemail.com: domain of 3nRWuZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3nRWuZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705907614; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pm6LGPn0g5uP1lRXkRkuwE07r5wy4EXy2R/IId+4s5Y=; b=OikuZlviHOtVUKbClnwOwc55Tsc7zBWAwJ56OdDK/yzmjz0B4RfeyMaVp7cBnm3opU54nu QD1tpDCWTBNlNaoTfMgkrUfXT4rDpDNm3nDISnQCp+7QIpheQp2/Z5nn14SYsNyepQI4Jw PnubuH7B69dJlyfeCi+wuzkds9b185Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705907614; a=rsa-sha256; cv=none; b=jFVfuNcsMOAGNiYq4RTJenpgJyk0PdIHtxT3LhVjwDiIq4ZcQQG/pL7d6kFIqrsbt1VfZe /qyzWmhLvHJ1A2MT1d+3Bgczc8JIExZnSUIOIEdGVIWqjalnYgU0lrHT5jKo7PV5Z5TSRf x9w1l4E8kdy3U8MiTUfLH33LPXNWk9o= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mTeY0rEd; spf=pass (imf10.hostedemail.com: domain of 3nRWuZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3nRWuZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5f53b4554b6so38600607b3.3 for ; Sun, 21 Jan 2024 23:13:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1705907613; x=1706512413; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pm6LGPn0g5uP1lRXkRkuwE07r5wy4EXy2R/IId+4s5Y=; b=mTeY0rEd0hm5Z5yJYoMDHk0JctTOY1U+2b75wMc+F771IUYprWUSxEdBGy+IgToi1J Jfa9znF+WHYlM4uLribSfy9/nxzCo6FoRA/26gU+9+ESPKq97eKYAL3y/b/TOIHw+Nm/ cVCoRWt+czeYvOXzi4L9+1YoryV+yrn6p4NdHZ6FJE2aNHK2WLAVFyK+ui5iNBsaMDpy fKyHLCXJCXJJDwsEOnltqUh9bNVoq/Lm/0ANGe7vlRqPvGjjR3gv38/iqVwJdSAR3AP5 RssHCOc9PXEUSldZ6TBbYyW3Z2QRpPGhhkEGunFSLJ3cJNic0Ix6UbWiNAcGnqNimn1U upDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705907613; x=1706512413; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pm6LGPn0g5uP1lRXkRkuwE07r5wy4EXy2R/IId+4s5Y=; b=WUS7LgnmckBS9ScZ6rqqAYCG23VjH16HqBPPgTRSfkG9EFUrDhqJR0KDm/L4sIaRNQ 9EbGqyUV6IQ92cGk6LEcFPU/df7diRivkgS2qYrlYAY44n2XmekdzhIAQfv/17bXLyxe 9cQtH42jhRi0XSBbB6ABGdrhME2hKM5tigW9dtMdbBjeO/yhnM+sQD5qWJ9LbCOWlBTc gac9b6uPgaMyf225rY3s0XXero0g+NiyuwG+M5Y+jgZ1rTEVWn7urz06uH6P8/STFDKo RP3gtsT7ACfi5XgklZhPPT0F1DTmdUARKod5v1MtptZCdHv73FZ02+49+KhA7MYnwt0n TBXA== X-Gm-Message-State: AOJu0Yw94KVfwSDFTZ+K0A/rrWcO2euPKaUicwNGK1BVKRnU2dpsKV6w 98VhY0EU40MRYIwebFl5UU6Lt6cYexHiyXjOosIp5E0Nd0pO3rKIJWIJKSVo7rS5/fICKQHMqf2 OJA== X-Google-Smtp-Source: AGHT+IH/iYjVx28BB70L/Axhud1gI/xXkTJNuH9QVTuj4mqjLNYD8CKXEzHkMSgNYwCDkSwAyquSZSHHvqg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:4979:1d79:d572:5708]) (user=surenb job=sendgmr) by 2002:a81:9847:0:b0:5e7:12cc:a60f with SMTP id p68-20020a819847000000b005e712cca60fmr1362943ywg.6.1705907613421; Sun, 21 Jan 2024 23:13:33 -0800 (PST) Date: Sun, 21 Jan 2024 23:13:24 -0800 In-Reply-To: <20240122071324.2099712-1-surenb@google.com> Mime-Version: 1.0 References: <20240122071324.2099712-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240122071324.2099712-3-surenb@google.com> Subject: [PATCH 3/3] mm/maps: read proc/pid/maps under RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, dchinner@redhat.com, casey@schaufler-ca.com, ben.wolsieffer@hefring.com, paulmck@kernel.org, david@redhat.com, avagin@google.com, usama.anjum@collabora.com, peterx@redhat.com, hughd@google.com, ryan.roberts@arm.com, wangkefeng.wang@huawei.com, Liam.Howlett@Oracle.com, yuzhao@google.com, axelrasmussen@google.com, lstoakes@gmail.com, talumbau@google.com, willy@infradead.org, vbabka@suse.cz, mgorman@techsingularity.net, jhubbard@nvidia.com, vishal.moola@gmail.com, mathieu.desnoyers@efficios.com, dhowells@redhat.com, jgg@ziepe.ca, sidhartha.kumar@oracle.com, andriy.shevchenko@linux.intel.com, yangxingui@huawei.com, keescook@chromium.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 467B6C0002 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: jzkwqudmefg9ncy8ho4gm1b7scjstyf7 X-HE-Tag: 1705907614-407999 X-HE-Meta: U2FsdGVkX18ggWg/MUvIi2gEYJucBuzpSS3qmysdKLZwqLAXjHvV2XmeAGwJpfg7BsllDESik0MyVpasEsSrLGedobQkBXQp+BNGBmlST22qf1/wGZkM/Z7mLKAzjTVSsOmHVbAalbk+/NEPBK/5dRJUvSdICU/5r96N0eH/Qss9cwXzJLeLs/HPG8GvtSplXFP/WqxrVFxPW9wI8oms5FXz/VmdHvxgqL8bkZlr+0/x03ucQpANpY8X4ev4khRzJ4L1r84VwUDlHi6wNyR5pq3mx8AhaNQTL2sloulaI17ymWAUUipIXwjBtuRXmJ9xlH8Kh2gP85f9wMHaIglXF8w5UoQkQYBu7NG89LWmWCSxsHaLUVNWlqp1Mxz6+Fsf8bXL+2wDk+S9zwQziAGlR5cyukJSkybXtRlKjh7/l33+EQY5C9grmJ+pXPl09HV0yuKRaJ8CXzGC0V1BT6DfFbXsH/WyBv5QXQO1motSRiG1I3aJ4yQH+vm03hvDo0cxan4PWfKgCk6GExcdUS/D8wv2P9FrHxbW+mvD1H8yfpHilEd0T/JCP6lziWO9ybjBrBGf9THRGTtkNz5MtCkDrfPJ4/zF9R5g/f6iyfjEAqka9PnYjNhSi7pT+X+2CsXPnitFP7tYI6Y3J/eQn7ZAaASfpXL8obEKKPwX5PfX3OOROHCQ6GfrpdDBSLU5c42cHWToQzcZjF4bDaPpoFv7lAbSIevJ+nlOnfvley7ZahBkvqX4jdyOO/LyzERye0MmYYHUEkNpnZVkSXjqllZQKA9pPSZIPGMDiKMbAveGJVbsyCHt08+ujdaJDM4e73fBQf2hfPx0ccJOGN33gjNDCj58cILiAYeYPCfkZCiqfG9aDGkhnThGaa6BPnkKffUURxY1FO1JQtkz4e/rJ+2+3Fm7gss8AK8HOEIBOXxzHLpyB5+ZvjMeJy957Bijagn4ohG0UKARjUJaINbrLH2 W3fsjIjG QVDv5m7hvbVWEKmZxFFm/WUigTy/MbFJehCT3HfMsh8KgJoZQL1eIC/6FrbqcmS40cellqzjl+uFiounhcgkhLBSa+bqy3mgi4PUVdT+vBviraLqgpNjsPmsJEhdeo4Tp0ije6gcPVSrrazsubIz+PW+exbiczX80uXaT0RZwE/2glt6oeelApkd1ZacRHKB0agYGXgm5wA4tv+6KpDN17EqBuJR1fXjY4AzCtp+TxvxJnmOgr5fWErQZF0EikfaV5B9COGuOqGuHQWwz1ct18EFBO5QcFL6xoF3E3Lg6BCm5+UmaU1hnDP4UemAZW2rPKaSHIIFpgOywKkX3qeDDD4vn/7gOH2b/j5H2+VAsJ+eJBbFk3AL/8iDpCVL3N5iz6LBQpYVpLCL2xGkXY78d9Zbov1Wm+r8qEC5j5DmI118sJBQEje4ci3ABKc7bYx5qTLi6IMYZfpvNajb811GN/NYY4+8prVsq//CGRzV1LtgpDFYaFmW82MKkJQliBDblBSMZjQUfjVbJ8vu5IYMBu4yJf0zIaAl1DwUqobkLRC2TGWGUXYenuyfFweYj+s4JP391mckc1i/VVsXynYPhLuKciQ77bh8BqyyJJ8eQvU1iF+bmQ42cJuV5BZaE8zaFHitNX1NNLYyRj/BdBWA9NyK4uwvioFediNy7B2cmZ+EdwoM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With maple_tree supporting vma tree traversal under RCU and per-vma locks making vma access RCU-safe, /proc/pid/maps can be read under RCU and without the need to read-lock mmap_lock. However vma content can change from under us, therefore we make a copy of the vma and we pin pointer fields used when generating the output (currently only vm_file and anon_name). Afterwards we check for concurrent address space modifications, wait for them to end and retry. That last check is needed to avoid possibility of missing a vma during concurrent maple_tree node replacement, which might report a NULL when a vma is replaced with another one. While we take the mmap_lock for reading during such contention, we do that momentarily only to record new mm_wr_seq counter. This change is designed to reduce mmap_lock contention and prevent a process reading /proc/pid/maps files (often a low priority task, such as monitoring/data collection services) from blocking address space updates. Note that this change has a userspace visible disadvantage: it allows for sub-page data tearing as opposed to the previous mechanism where data tearing could happen only between pages of generated output data. Since current userspace considers data tearing between pages to be acceptable, we assume is will be able to handle sub-page data tearing as well. Signed-off-by: Suren Baghdasaryan --- fs/proc/internal.h | 2 + fs/proc/task_mmu.c | 114 ++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 109 insertions(+), 7 deletions(-) diff --git a/fs/proc/internal.h b/fs/proc/internal.h index a71ac5379584..e0247225bb68 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -290,6 +290,8 @@ struct proc_maps_private { struct task_struct *task; struct mm_struct *mm; struct vma_iterator iter; + unsigned long mm_wr_seq; + struct vm_area_struct vma_copy; #ifdef CONFIG_NUMA struct mempolicy *task_mempolicy; #endif diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 3f78ebbb795f..3886d04afc01 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -126,11 +126,96 @@ static void release_task_mempolicy(struct proc_maps_private *priv) } #endif -static struct vm_area_struct *proc_get_vma(struct proc_maps_private *priv, - loff_t *ppos) +#ifdef CONFIG_PER_VMA_LOCK + +static const struct seq_operations proc_pid_maps_op; +/* + * Take VMA snapshot and pin vm_file and anon_name as they are used by + * show_map_vma. + */ +static int get_vma_snapshow(struct proc_maps_private *priv, struct vm_area_struct *vma) { + struct vm_area_struct *copy = &priv->vma_copy; + int ret = -EAGAIN; + + memcpy(copy, vma, sizeof(*vma)); + if (copy->vm_file && !get_file_rcu(©->vm_file)) + goto out; + + if (copy->anon_name && !anon_vma_name_get_rcu(copy)) + goto put_file; + + if (priv->mm_wr_seq == mmap_write_seq_read(priv->mm)) + return 0; + + /* Address space got modified, vma might be stale. Wait and retry. */ + rcu_read_unlock(); + ret = mmap_read_lock_killable(priv->mm); + mmap_write_seq_record(priv->mm, &priv->mm_wr_seq); + mmap_read_unlock(priv->mm); + rcu_read_lock(); + + if (!ret) + ret = -EAGAIN; /* no other errors, ok to retry */ + + if (copy->anon_name) + anon_vma_name_put(copy->anon_name); +put_file: + if (copy->vm_file) + fput(copy->vm_file); +out: + return ret; +} + +static void put_vma_snapshot(struct proc_maps_private *priv) +{ + struct vm_area_struct *vma = &priv->vma_copy; + + if (vma->anon_name) + anon_vma_name_put(vma->anon_name); + if (vma->vm_file) + fput(vma->vm_file); +} + +static inline bool needs_mmap_lock(struct seq_file *m) +{ + /* + * smaps and numa_maps perform page table walk, therefore require + * mmap_lock but maps can be read under RCU. + */ + return m->op != &proc_pid_maps_op; +} + +#else /* CONFIG_PER_VMA_LOCK */ + +/* Without per-vma locks VMA access is not RCU-safe */ +static inline bool needs_mmap_lock(struct seq_file *m) { return true; } + +#endif /* CONFIG_PER_VMA_LOCK */ + +static struct vm_area_struct *proc_get_vma(struct seq_file *m, loff_t *ppos) +{ + struct proc_maps_private *priv = m->private; struct vm_area_struct *vma = vma_next(&priv->iter); +#ifdef CONFIG_PER_VMA_LOCK + if (vma && !needs_mmap_lock(m)) { + int ret; + + put_vma_snapshot(priv); + while ((ret = get_vma_snapshow(priv, vma)) == -EAGAIN) { + /* lookup the vma at the last position again */ + vma_iter_init(&priv->iter, priv->mm, *ppos); + vma = vma_next(&priv->iter); + } + + if (ret) { + put_vma_snapshot(priv); + return NULL; + } + vma = &priv->vma_copy; + } +#endif if (vma) { *ppos = vma->vm_start; } else { @@ -169,12 +254,20 @@ static void *m_start(struct seq_file *m, loff_t *ppos) return ERR_PTR(-EINTR); } + /* Drop mmap_lock if possible */ + if (!needs_mmap_lock(m)) { + mmap_write_seq_record(priv->mm, &priv->mm_wr_seq); + mmap_read_unlock(priv->mm); + rcu_read_lock(); + memset(&priv->vma_copy, 0, sizeof(priv->vma_copy)); + } + vma_iter_init(&priv->iter, mm, last_addr); hold_task_mempolicy(priv); if (last_addr == -2UL) return get_gate_vma(mm); - return proc_get_vma(priv, ppos); + return proc_get_vma(m, ppos); } static void *m_next(struct seq_file *m, void *v, loff_t *ppos) @@ -183,7 +276,7 @@ static void *m_next(struct seq_file *m, void *v, loff_t *ppos) *ppos = -1UL; return NULL; } - return proc_get_vma(m->private, ppos); + return proc_get_vma(m, ppos); } static void m_stop(struct seq_file *m, void *v) @@ -195,7 +288,10 @@ static void m_stop(struct seq_file *m, void *v) return; release_task_mempolicy(priv); - mmap_read_unlock(mm); + if (needs_mmap_lock(m)) + mmap_read_unlock(mm); + else + rcu_read_unlock(); mmput(mm); put_task_struct(priv->task); priv->task = NULL; @@ -283,8 +379,10 @@ show_map_vma(struct seq_file *m, struct vm_area_struct *vma) start = vma->vm_start; end = vma->vm_end; show_vma_header_prefix(m, start, end, flags, pgoff, dev, ino); - if (mm) - anon_name = anon_vma_name(vma); + if (mm) { + anon_name = needs_mmap_lock(m) ? anon_vma_name(vma) : + anon_vma_name_get_rcu(vma); + } /* * Print the dentry name for named mappings, and a @@ -338,6 +436,8 @@ show_map_vma(struct seq_file *m, struct vm_area_struct *vma) seq_puts(m, name); } seq_putc(m, '\n'); + if (anon_name && !needs_mmap_lock(m)) + anon_vma_name_put(anon_name); } static int show_map(struct seq_file *m, void *v)