From patchwork Tue Jan 23 23:10:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13528265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34B78C47DDB for ; Tue, 23 Jan 2024 23:10:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 874216B007B; Tue, 23 Jan 2024 18:10:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FD816B0082; Tue, 23 Jan 2024 18:10:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 677126B0083; Tue, 23 Jan 2024 18:10:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 510336B007B for ; Tue, 23 Jan 2024 18:10:20 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2741E40712 for ; Tue, 23 Jan 2024 23:10:20 +0000 (UTC) X-FDA: 81712121400.20.F1F1258 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf10.hostedemail.com (Postfix) with ESMTP id 4A8E9C0016 for ; Tue, 23 Jan 2024 23:10:18 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="gx6r9V/y"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3WUewZQYKCNwQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3WUewZQYKCNwQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706051418; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=oXyU4pkvCRYAyA2TUvsUC8dh5sAPzBBH4qagn6tOpF0=; b=SSyOG72KhppDXzO86sF8ULsoXxBlRDvFomz9v3HU/HOatDYp5fzPCZx2a+QmcGiLxLq9Dc b1W0sdKmGHKQDOdAFlC6iEdCrDZj+51heLr2YkERUwZSahFBT/7KjCz8W/rLRj308zOpvc rM6LHxaOEhTiVZeQ1Wbv65ewErR5wXE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="gx6r9V/y"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3WUewZQYKCNwQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3WUewZQYKCNwQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706051418; a=rsa-sha256; cv=none; b=S9d0OtC7s+iVBulz4NV0KUetvy3eOhfAmobGRDsNgfOGrFIut8UN7jyp4QCEp37/bSGA7d vhx9CmzgZFdg+OxjPKtqCI4B3F4uByR5881bCM8hg+BN/59N9bfvkF7AraszHbN/z6XRfr 9/SrH5c/ATheIgvJ29zjQM7jIVEePA0= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dbf1c3816a3so5766715276.1 for ; Tue, 23 Jan 2024 15:10:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706051417; x=1706656217; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=oXyU4pkvCRYAyA2TUvsUC8dh5sAPzBBH4qagn6tOpF0=; b=gx6r9V/y2qHq4kUdVnxHvZS/Xl+8rba0Eb5MHbQ4TFXhg3QxUHsAhSxVf9K7uOnAsu LGzfEJ/G3B0whQZJFQFhCqEm/ocuAnqtMKfUYkfQYwITrZWPVlEV+dWJ5to+5MlS09XR rM2fD/D6fL/5VSMIlMokr5Br5/ODf8Z4vV5Zq8ftCl+gS1iZ5DE03wRSAVE6DtMRBQtY ldQUwxi1X9T8DomYsGwWuRG6e+AxYLxAK7xeY45dop5prxsDZQuwuADBqvKHK+NtrTdc ZBq+BKOfhFI4tDkA8Gk0MFtRUyGhGrpa9a1UlSGIqV1C0SOdAuMVPKoYeag1k5KGVwY4 m+dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706051417; x=1706656217; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=oXyU4pkvCRYAyA2TUvsUC8dh5sAPzBBH4qagn6tOpF0=; b=giZPb2My0CwsDaflxdRJ9yOadvUOL3LX3gR65JFGCtwAs3j9dldHw+dtCz1qvelesb VUA1Ra7QHrm/3Gaugy5pwGMA5YE28bNu0lH/o/FJ1KHNnKW2k22zD8RyeZN1yPXIWVGc SpjK+C87g3AezNpzBW0h/pd4zgQc4v3X15+fiCBCysB/Oq4vxRaasL925m57qIH08Lbh ppn165KyYaU5PwfR8blgn4hHwkMwg7sODHzSJoKJ2AbRStHl+e3Hwi1+z1F3TgIT7wrJ GpbYgNGosd2sY0JqmV2Ik+l8AX06+d8XMJQSeODr8363FCKe8rYKDUbj6LvYJ/yMcBUG IkXQ== X-Gm-Message-State: AOJu0Yzl4dsCg8Om6inlLHv4OvGIjcI/n8gy5a0JT18TgPd3XnAguhck Hyh6M3I2ITFIbGrDHApRniSS92JkM40zfEWlq5mLZ096Ew3/bVLm4pzpdbKmqSlEft2HcspsSQM Ong== X-Google-Smtp-Source: AGHT+IEgxI304pcV7zFy8N/84TxapaHhJhR9pXZLWhwJ6Iatmm8yKLxxB6dJA4KRTnvXodtXbWXNbbtsUkg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:8fc3:c34f:d407:388]) (user=surenb job=sendgmr) by 2002:a05:6902:218b:b0:dc2:5456:d9ac with SMTP id dl11-20020a056902218b00b00dc25456d9acmr377765ybb.5.1706051417317; Tue, 23 Jan 2024 15:10:17 -0800 (PST) Date: Tue, 23 Jan 2024 15:10:12 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123231014.3801041-1-surenb@google.com> Subject: [PATCH v2 1/3] mm: make vm_area_struct anon_name field RCU-safe From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, dchinner@redhat.com, casey@schaufler-ca.com, ben.wolsieffer@hefring.com, paulmck@kernel.org, david@redhat.com, avagin@google.com, usama.anjum@collabora.com, peterx@redhat.com, hughd@google.com, ryan.roberts@arm.com, wangkefeng.wang@huawei.com, Liam.Howlett@Oracle.com, yuzhao@google.com, axelrasmussen@google.com, lstoakes@gmail.com, talumbau@google.com, willy@infradead.org, vbabka@suse.cz, mgorman@techsingularity.net, jhubbard@nvidia.com, vishal.moola@gmail.com, mathieu.desnoyers@efficios.com, dhowells@redhat.com, jgg@ziepe.ca, sidhartha.kumar@oracle.com, andriy.shevchenko@linux.intel.com, yangxingui@huawei.com, keescook@chromium.org, sj@kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 4A8E9C0016 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: smz6zsmm8n8ojhiew6trtiu4y5ryz88d X-HE-Tag: 1706051418-747907 X-HE-Meta: U2FsdGVkX1/WI7UP9Lgk2yDJTyzt88/v1qb7LcKIMa6E8b5Fsn68/UYMjLjwaOgUsg8pXG63X4Js98ax0I04B1+uy3cYBiCN9b77NktUlwhIP61eFrLw5c+xUASncm9cmQuFolbyeGq5qnh20P+aYtKvv52PAw5cSgpJSTVmtGgQhmBcIQ+hEsCCnejDAfnpT8wGhWPTfXJTEICSoBcpl9MX1iL41QjZTcDB1zVUK9PFXPzd0R/7aeNjhY1scDJHiJYkMvZaxuNwOjtY5Q+bpzQ4vVJN1InQ3c4rnbbghYvPPDGJX4gRee5rTgfuJUe5Ya984pMxuxG46JYLzwJRPHHT7GADJW3eSOaKYBAMebsaOOOy8rTnpVg2hNhs/5BMrfxCCCNI3ETnCHdqNNmrqjraOfAbZHPn1ewMy/EHwvWkSJIGGJ0VpO9pgY24mLILH9Vpie8xGl4fCbofNoKSatNh952w0t2KcWVQWI02RY7r1sc1IUZTC7cuTYRm1tNZb/UUjk21q0AY3bElAnI5ZnTwqiIfSmiE8JalTE+iN0x8lSkd1TwGuGEpbJ9qJz2L5+EaNcgfyS9ZUwzF2g5O6lsfPxNs2fGzzP41LPpeZZ9iXWRF/zLPIGu8fotzrYXU2ZOYnrIcV+9XLnhUZZSbbpVVmhIzt4uiPm3cVGN8zxTkQHqpAH8Ll+h9qwRxwj6mC40BAISk+eaKuHqT9JH3fl6Ktm7HL4CDIUveiSQToxxZV47JzvADvcSQfIa6AePgwNg63exuoSdY8d/v0PONSLfVNn/QgL6jzkNyd8xjqUhane0E0086ignYEa/AjSFCNizY27ozxWrj+tq/oj8aqshBoG8oDo4CW+pHJb29+0qRbCcU+y2axvhsxpRkzWg2a9vAnbT9JXBv3lt1salzrIX6Ml3jOpiUPVr2bkVmop2pEya3bTzyK7BaA1YljTDRb7/xUtJYDk9opEvR9e4 0DkGrU72 AIubq+cBAd9lvt5ZZfxdCV6iizuDREaopSKENxmXu8a45yTD4HUC1RSsojVDyW+oivupfPQfYwsJr/cg8j/sTgy8GIggMrl1/7y1go9dLFy4JC9J1M/WdW5MgBTLgIbFWQKKu+VQ5xxsA10X1koP3Nzk6s1Xw9iMkUPXCTkGarsTtqteDAwkLjvZGtXytersqHQ5fi9NLD+4V8d4xnZT5wkrpFkKmCVGNDfuVc9OLZLwbZqi47QAs3OGmPzHvO3kmu/iAKsdvMrSsEWJdiyw0+3dn6ZtfF/1LtVA+ushgvm2oa0HT8v4cC8Gjpx+1ucyMkyis/Q7C0wVQ3iP6M6deBogLiP90psOFScMQ5n90Mq2tQKFe+uvUGVQexBQFlXdDW0IGLdrw8MOykOlxwEHxTYF54YQdK14YM9kBzX5y2lMwG9JJS/AqVMo3mT4ZNo274GuwZkhzbXXH7I1ZkxPNCZcYgunG5FSWLh5876yu3XQcUL7ebgeia7ShyT+bzMGfl5xCSErfDIVaQ57oeWk7lGP5QTvuLhXP+LDFts42MG0cYLvkXBrrM571Mkph6usziv9ztu0xsGq9N3LZ2543mobhY/MoFLwDnkP5HMra5YHeP1emEnR17wKEU10Ak1DgAPMypid4x9K/A3sDJFUdCQuIhwYJLNEpjcuH4+KHsekSIIYdO3CfbzhtnoSKXEhQdOxhv3/qhCDUfc2XuH6zpqUxLg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For lockless /proc/pid/maps reading we have to ensure all the fields used when generating the output are RCU-safe. The only pointer fields in vm_area_struct which are used to generate that file's output are vm_file and anon_name. vm_file is RCU-safe but anon_name is not. Make anon_name RCU-safe as well. Signed-off-by: Suren Baghdasaryan --- include/linux/mm_inline.h | 10 +++++++++- include/linux/mm_types.h | 3 ++- mm/madvise.c | 30 ++++++++++++++++++++++++++---- 3 files changed, 37 insertions(+), 6 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index f4fe593c1400..bbdb0ca857f1 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -389,7 +389,7 @@ static inline void dup_anon_vma_name(struct vm_area_struct *orig_vma, struct anon_vma_name *anon_name = anon_vma_name(orig_vma); if (anon_name) - new_vma->anon_name = anon_vma_name_reuse(anon_name); + rcu_assign_pointer(new_vma->anon_name, anon_vma_name_reuse(anon_name)); } static inline void free_anon_vma_name(struct vm_area_struct *vma) @@ -411,6 +411,8 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, !strcmp(anon_name1->name, anon_name2->name); } +struct anon_vma_name *anon_vma_name_get_rcu(struct vm_area_struct *vma); + #else /* CONFIG_ANON_VMA_NAME */ static inline void anon_vma_name_get(struct anon_vma_name *anon_name) {} static inline void anon_vma_name_put(struct anon_vma_name *anon_name) {} @@ -424,6 +426,12 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, return true; } +static inline +struct anon_vma_name *anon_vma_name_get_rcu(struct vm_area_struct *vma) +{ + return NULL; +} + #endif /* CONFIG_ANON_VMA_NAME */ static inline void init_tlb_flush_pending(struct mm_struct *mm) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8b611e13153e..bbe1223cd992 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -545,6 +545,7 @@ struct vm_userfaultfd_ctx {}; struct anon_vma_name { struct kref kref; + struct rcu_head rcu; /* The name needs to be at the end because it is dynamically sized. */ char name[]; }; @@ -699,7 +700,7 @@ struct vm_area_struct { * terminated string containing the name given to the vma, or NULL if * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. */ - struct anon_vma_name *anon_name; + struct anon_vma_name __rcu *anon_name; #endif #ifdef CONFIG_SWAP atomic_long_t swap_readahead_info; diff --git a/mm/madvise.c b/mm/madvise.c index 912155a94ed5..0f222d464254 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -88,14 +88,15 @@ void anon_vma_name_free(struct kref *kref) { struct anon_vma_name *anon_name = container_of(kref, struct anon_vma_name, kref); - kfree(anon_name); + kfree_rcu(anon_name, rcu); } struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); - return vma->anon_name; + return rcu_dereference_protected(vma->anon_name, + rwsem_is_locked(&vma->vm_mm->mmap_lock)); } /* mmap_lock should be write-locked */ @@ -105,7 +106,7 @@ static int replace_anon_vma_name(struct vm_area_struct *vma, struct anon_vma_name *orig_name = anon_vma_name(vma); if (!anon_name) { - vma->anon_name = NULL; + rcu_assign_pointer(vma->anon_name, NULL); anon_vma_name_put(orig_name); return 0; } @@ -113,11 +114,32 @@ static int replace_anon_vma_name(struct vm_area_struct *vma, if (anon_vma_name_eq(orig_name, anon_name)) return 0; - vma->anon_name = anon_vma_name_reuse(anon_name); + rcu_assign_pointer(vma->anon_name, anon_vma_name_reuse(anon_name)); anon_vma_name_put(orig_name); return 0; } + +/* + * Returned anon_vma_name is stable due to elevated refcount but not guaranteed + * to be assigned to the original VMA after the call. + */ +struct anon_vma_name *anon_vma_name_get_rcu(struct vm_area_struct *vma) +{ + struct anon_vma_name __rcu *anon_name; + + WARN_ON_ONCE(!rcu_read_lock_held()); + + anon_name = rcu_dereference(vma->anon_name); + if (!anon_name) + return NULL; + + if (unlikely(!kref_get_unless_zero(&anon_name->kref))) + return NULL; + + return anon_name; +} + #else /* CONFIG_ANON_VMA_NAME */ static int replace_anon_vma_name(struct vm_area_struct *vma, struct anon_vma_name *anon_name) From patchwork Tue Jan 23 23:10:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13528266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3B57C47258 for ; Tue, 23 Jan 2024 23:10:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 678C66B0082; Tue, 23 Jan 2024 18:10:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 601486B0083; Tue, 23 Jan 2024 18:10:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4557A6B0085; Tue, 23 Jan 2024 18:10:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 311256B0082 for ; Tue, 23 Jan 2024 18:10:22 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C901F1C14BA for ; Tue, 23 Jan 2024 23:10:21 +0000 (UTC) X-FDA: 81712121442.05.EE767E5 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 4018A40020 for ; Tue, 23 Jan 2024 23:10:20 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CisczW0i; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3W0ewZQYKCN4SURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W0ewZQYKCN4SURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706051420; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=h2ZXeH/q0v/ShjN+m/aCkV1AkHA7pUnaXY25HPiCfirOwlnDfC+Lh8Uz0bRsA1PVRCgP3Z DXCs/rCKVAgFSuk7q9A7su5I+5xhT/XcvO9Q0f7BHfWGvjWFGAEPTQFjMt7FeCKKFPBMTR pusRDOS8dX+cijqon/B3qJuhVYpVHFI= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CisczW0i; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3W0ewZQYKCN4SURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W0ewZQYKCN4SURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706051420; a=rsa-sha256; cv=none; b=ReXfxV1Jtn/dvFmU+OrufG331VBX8jHS/aYMyD4DQJMnJ/cUJVUZwCD7aZJFExhupTC75q fQq6H4o/ATLjSzxdmNJo7stdOnDXhSGITDPxv7IXYHrsx21KYrnNGkNewdZ2h/bfy/8lJb 9wTelZArGwCh6nlermy1BK5rmuzpAi8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5f874219ff9so66801007b3.0 for ; Tue, 23 Jan 2024 15:10:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706051419; x=1706656219; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=CisczW0ieVEh6tFi6AO+XedbqtsE3bHg+0jmGTz3lshqBKlnIZwkNcw5qUqxbabpr4 lNDM31Ya0d41U3uEXlTLJBqkF2fLbHA3Nfhr7ApbhuKj2/WcVb5pIBIGmxqs+yrGT5vP wqq07ZHd3KPBliguxl8L27JxkJUilrnjqCI0OpSeJSAUU6lvRZIs2Ff4zZ4wUVTrHiBe v5EOZvydgJRJXcnb314e7ssiGJEpCgiKCsSCUJES2XqrH9FvhydRvVab0Accp30XuD7l BfGPcVnc+KQ60PuA5zLQ1Bvo6xEI9Z0iFZ6Cv5T8430ShhOCihFKj+ZzeJd2Rnkwv1Ll he5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706051419; x=1706656219; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OLp6U6EsMSw583KQsA7U2jc3UsvPAv4xgtxefIxudO0=; b=l8MaQ7VPCcpq1Wd1IBrjg365N4fLINBg8MRv5a5VI/jZSfFqlHDfog3NlG1QBeyNuh 1mgXA4p6986JFH+lsRQLnzO4r23uRwB7G8ZQEShyI2cWCUrVyFc0oOqbDaq1EjX3KOW5 fJlOsyLLxhqn30XnSt5j/JTQb2ss2bJ2JzHhosOHxemCFh2yr6VhD4Ep2kyyNLio+zIy u9Vf+aMubiBL36Pp+R4WD8mNBoAsJQzZznGembBC4vVBvP/cl7e2mB0r5YnuEfBWlrri keKGs88UdS4ZxxelVN4+Vst+EBoXQO1FTTWld0w17HVGydMQCaR9QZ8nLpBnJkN9wic5 eNbQ== X-Gm-Message-State: AOJu0YwZrrAichh80WYFmwklXwr+R/P0Oftwu6COk8i7pC1sRU5ded6s 6EyUAwQy6/gKiPS4IMxo11YzJDYMyd8J7lk99S1HLnrLbSP6w13ZFT7A5fqlAHl0S649IJKa01k qlg== X-Google-Smtp-Source: AGHT+IFX5qIebM/ZQiKUEXGTW3oxR6pBTWd2HR+W3VQ1MxVAngHzessR/YRxQzpsjGe8I1sOL4WmImG7jew= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:8fc3:c34f:d407:388]) (user=surenb job=sendgmr) by 2002:a81:9896:0:b0:5fc:7f94:da64 with SMTP id p144-20020a819896000000b005fc7f94da64mr134477ywg.5.1706051419411; Tue, 23 Jan 2024 15:10:19 -0800 (PST) Date: Tue, 23 Jan 2024 15:10:13 -0800 In-Reply-To: <20240123231014.3801041-1-surenb@google.com> Mime-Version: 1.0 References: <20240123231014.3801041-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123231014.3801041-2-surenb@google.com> Subject: [PATCH v2 2/3] mm: add mm_struct sequence number to detect write locks From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, dchinner@redhat.com, casey@schaufler-ca.com, ben.wolsieffer@hefring.com, paulmck@kernel.org, david@redhat.com, avagin@google.com, usama.anjum@collabora.com, peterx@redhat.com, hughd@google.com, ryan.roberts@arm.com, wangkefeng.wang@huawei.com, Liam.Howlett@Oracle.com, yuzhao@google.com, axelrasmussen@google.com, lstoakes@gmail.com, talumbau@google.com, willy@infradead.org, vbabka@suse.cz, mgorman@techsingularity.net, jhubbard@nvidia.com, vishal.moola@gmail.com, mathieu.desnoyers@efficios.com, dhowells@redhat.com, jgg@ziepe.ca, sidhartha.kumar@oracle.com, andriy.shevchenko@linux.intel.com, yangxingui@huawei.com, keescook@chromium.org, sj@kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 4018A40020 X-Stat-Signature: k6nip5srpnjg9h9crax6tnfkm5stepp9 X-Rspam-User: X-HE-Tag: 1706051420-269716 X-HE-Meta: U2FsdGVkX1/UIGVhUZimS0FI9PimWe1SG2KSAnLLF0Jq0HKVAxvyY/DXdD2mO37Y1F7Y1z04DIwk9NgqHnExf1ObDjBGLcvoKssfWsk6OMl2C0OHYAbNFbbByAur87DNAuGG9noVAaju2RS5ONQFqECU4OA3LUxOdzpWea0NyclNAtyloYOLJaHrL47gDPXXLXXGlSkaPldYowD7nlYzveM3UkBf9A3XYhc3H79kKbDmx/h2o3ryqBe3cVSMF6zpOQQYPjBqVB2+WToq2b4elCm3oLKv04p1e3QjsLLrX0YxAaoFMocSih2lxgng2W2tFu96fK0UBg8v6EDKU05YXu49N5jbqLj30GLG1yq37NRA2Yk5eSgk0FPP/BxbXrhynGYnqVTS6SBnD9vP41ngYCYoRRjoB2OuY6M07PNZvo4miIldYAbrd8pfR551wzzjszxqUE0CHxijoeTg9RbO79Csa+7QAsay9mvuh1JGvsRlGA2q2w3dZwroUyHIK44aA+X+zR5PHu2yrFgvAParboZdSmCAzhWb0/NtCKDAGQwVn91fuSDNEnFv97PQaQrJ5HBT7ejNnsNa9FngRcMXCpdAZL0/0gl3r+ryBw8IZnQ0NT66wRXfYf9z6KM60kNfPAuh1eEDEiQfVXyQSL8I0yn0QLVzKoZY7Q+oZyR1kdIlVpayRG8YN7zYMhtM5Jonrgik5G7T+UUomxY8zh6EQPa7/GYwMmf8JFTsqQ67xHCmkTOzSUa4jocn+P3MA4ZCWIifKCm1XUfs5BnlydTnDjGEAI2gKqwIuYH2+Hem0x8XS7NZqsCF44jfVc/HMg3fiX8GQlBBqQVKOMCYjcoMjHIo9Xn2fHFkpWB69UYoVzK2GtBc4D/xqmv9PkVDC81H8UYDVjzJJH6Rbp054DA6aepD7ZBj3MOSiTA8pV/XrErjbQlgvhhGpEKCZ2bKf86MsDwdNSxESVXUtbfjpSM FcsUNses efKGjrhAMIcd3BdmA11jJ5AlzLvrt8SgmRjAFlW9wVhr/tCx3zlm5SSZMalasHDpDLWSdB4WvOhFreqXMutvHXBqkLjuS2rIruTb1apLgTCMGkk08httYzCgdgcmWZ3geYDSEp37PNwxp6GM8XlgEGZDaLdnUyxxOw04AT9/j7yPS+cCzdiDIRMizHNlyB5NGcFihk1deOEjJsPCzrvqIF57jzUJfF3yNH82+mCxiVguqlp4pmDNvlMekm5pX8hPMMSUZYmZcShjqA/EiKtlGZcISPorMX+bBfzaHTvnbvIBLEbT61oFLrIBYqye66vd0Zpml6ROuxCVe8XLw0e20SSIEnE9DC0vaT5eJRgWQcESbKfkFsNtC5jyQPUGsMHeVzz7wRHg4qZnkrVkcW5OJhxMv9VmsYK/7kQYjXwZA1YC7DgmEOSW0rch+KRsNPrPolA9BPwnQAt7g1Yn1iymJUPDV0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide a way for lockless mm_struct users to detect whether mm might have been changed since some specific point in time. The API provided allows the user to record a counter when it starts using the mm and later use that counter to check if anyone write-locked mmap_lock since the counter was recorded. Recording the counter value should be done while holding mmap_lock at least for reading to prevent the counter from concurrent changes. Every time mmap_lock is write-locked mm_struct updates its mm_wr_seq counter so that checks against counters recorded before that would fail, indicating a possibility of mm being modified. Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 2 ++ include/linux/mmap_lock.h | 22 ++++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index bbe1223cd992..e749f7f09314 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -846,6 +846,8 @@ struct mm_struct { */ int mm_lock_seq; #endif + /* Counter incremented each time mm gets write-locked */ + unsigned long mm_wr_seq; unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 8d38dcb6d044..0197079cb6fe 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -106,6 +106,8 @@ static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); down_write(&mm->mmap_lock); + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -113,6 +115,8 @@ static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) { __mmap_lock_trace_start_locking(mm, true); down_write_nested(&mm->mmap_lock, subclass); + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); __mmap_lock_trace_acquire_returned(mm, true, true); } @@ -122,6 +126,10 @@ static inline int mmap_write_lock_killable(struct mm_struct *mm) __mmap_lock_trace_start_locking(mm, true); ret = down_write_killable(&mm->mmap_lock); + if (!ret) { + /* Pairs with ACQUIRE semantics in mmap_write_seq_read */ + smp_store_release(&mm->mm_wr_seq, mm->mm_wr_seq + 1); + } __mmap_lock_trace_acquire_returned(mm, true, ret == 0); return ret; } @@ -140,6 +148,20 @@ static inline void mmap_write_downgrade(struct mm_struct *mm) downgrade_write(&mm->mmap_lock); } +static inline unsigned long mmap_write_seq_read(struct mm_struct *mm) +{ + /* Pairs with RELEASE semantics in mmap_write_lock */ + return smp_load_acquire(&mm->mm_wr_seq); +} + +static inline void mmap_write_seq_record(struct mm_struct *mm, + unsigned long *mm_wr_seq) +{ + mmap_assert_locked(mm); + /* Nobody can concurrently modify since we hold the mmap_lock */ + *mm_wr_seq = mm->mm_wr_seq; +} + static inline void mmap_read_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, false); From patchwork Tue Jan 23 23:10:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13528267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B1EEC47DDC for ; Tue, 23 Jan 2024 23:10:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AE7A6B0083; Tue, 23 Jan 2024 18:10:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 936EC6B0088; Tue, 23 Jan 2024 18:10:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 715D66B0089; Tue, 23 Jan 2024 18:10:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5193B6B0083 for ; Tue, 23 Jan 2024 18:10:24 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2C6D3A1A09 for ; Tue, 23 Jan 2024 23:10:24 +0000 (UTC) X-FDA: 81712121568.15.31A2AC6 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 669E3140009 for ; Tue, 23 Jan 2024 23:10:22 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zE5qGg3A; spf=pass (imf23.hostedemail.com: domain of 3XUewZQYKCOAUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3XUewZQYKCOAUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706051422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iOH4eD/kP+yC8NUPTXlai05Ia1nTXRgcymGMGu85eLQ=; b=wTJy6419seVTlvd0xsnvWzSeJoFDyVNUPaIZWHPLCGOemt/jPHgCVCisbbPcMDMev32eNZ lqFKXdLLT6EqO+xdx+Drtl7ISFX82kMW/nxY2fKSnVKnC/J+vVqT2z0wUvRhb911f7EyAH BlRhxekncPl9Cuu5h1clbyIPzrdlUzs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706051422; a=rsa-sha256; cv=none; b=Q8gNPxk9MMjseRxgdzD7Khbyr9O0qezWGOyLt3i73HCZWeeoX63L40A7JCY4JrRxLkVpAg 0x84+cVLMPoG+NTEKDhTuuUkCH2Y40hxC8mlwFfc+Z4AOjycWF1+EOOdo4alJ4TnC0piCe tPoMmJEJr3B3gWZ0NKldMbaETZFBmjA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zE5qGg3A; spf=pass (imf23.hostedemail.com: domain of 3XUewZQYKCOAUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3XUewZQYKCOAUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5f0fd486b9aso60978667b3.2 for ; Tue, 23 Jan 2024 15:10:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706051421; x=1706656221; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iOH4eD/kP+yC8NUPTXlai05Ia1nTXRgcymGMGu85eLQ=; b=zE5qGg3A52Y55Mc6pBFO5+CvTnZDcbuEU4QDz3LTFBFkY75L+M0XcCueblbK+s6msr OqnR7yGB8n2d1uMEcirzqyuCjOAwczNPE4JLGUf3Sye6WIZqQFWTWWE9Lcjn/L/mnGa/ HihH4OIoSKVLePaHAVuWlvnvScsbezSYmfslTakaxXg3tG2IY0c07MbyMZzp6ho6yA51 Tx0AAeT6Hghwf10roIkd3+mGw/IXIu1RLdEi2VqdwbWkhAte74ivwYJUqup0Opql6Lqr MCpyhrDsPIIrPKJlHQfmjAIADrECTodlkV+XsKAH0kxVjBi+y577qQizy1MVwleSfsit O1rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706051421; x=1706656221; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iOH4eD/kP+yC8NUPTXlai05Ia1nTXRgcymGMGu85eLQ=; b=lfTvGCRYNVCsez8hFa2gyFgHYxk4WELNe+qY0w++VOjTUoSSvt8ME8XhDFk2YeJ31l Gaa9JEIfc7sGCwe9JF864nWO/3EIia5s1+07aH8QbDokuAKQZvLZ7BF4A/8dhEDCxwB5 gKWxrIEngiAVrqO/GX9nWteov1hF3zrbabpWVEd93yvOpuPhrGrOOoaN2ubcOOK2jUKF tC3DruTMcEI5ep5GjhR1NbKxeyR8LnKaXILOwakNBSngZsqqvfjXAb75cJThTE6RIN7g jmlVTZkbgpUkwGe89vuyodw/h0/vTBDoqC00evKVBsVBPJ269g7g4zEtkuCq0K+T941z 0lKg== X-Gm-Message-State: AOJu0YziP70zsh/edoJgLmL6HUetroOrZ3pAgWyBAOOlMlDU6L1GyLlq MTYrLtgxLnvioH9Tf3w4wo5oC4R82q7jAah47dFIuo1y0hvX1D/YNbpWF10n23B2dWxuyTH+dF1 GbQ== X-Google-Smtp-Source: AGHT+IFELlpA6dr+hh0ohU9jI75g0Qo+vt9bgKXouAI4UTf2K5w/MytBcB95Yhp71v6eqOEpu4Ziung4r/A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:8fc3:c34f:d407:388]) (user=surenb job=sendgmr) by 2002:a25:ce11:0:b0:dbd:b909:f090 with SMTP id x17-20020a25ce11000000b00dbdb909f090mr373700ybe.11.1706051421519; Tue, 23 Jan 2024 15:10:21 -0800 (PST) Date: Tue, 23 Jan 2024 15:10:14 -0800 In-Reply-To: <20240123231014.3801041-1-surenb@google.com> Mime-Version: 1.0 References: <20240123231014.3801041-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123231014.3801041-3-surenb@google.com> Subject: [PATCH v2 3/3] mm/maps: read proc/pid/maps under RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, dchinner@redhat.com, casey@schaufler-ca.com, ben.wolsieffer@hefring.com, paulmck@kernel.org, david@redhat.com, avagin@google.com, usama.anjum@collabora.com, peterx@redhat.com, hughd@google.com, ryan.roberts@arm.com, wangkefeng.wang@huawei.com, Liam.Howlett@Oracle.com, yuzhao@google.com, axelrasmussen@google.com, lstoakes@gmail.com, talumbau@google.com, willy@infradead.org, vbabka@suse.cz, mgorman@techsingularity.net, jhubbard@nvidia.com, vishal.moola@gmail.com, mathieu.desnoyers@efficios.com, dhowells@redhat.com, jgg@ziepe.ca, sidhartha.kumar@oracle.com, andriy.shevchenko@linux.intel.com, yangxingui@huawei.com, keescook@chromium.org, sj@kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 669E3140009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 97o86oqiyqskzwe4psxuh39rttd538hp X-HE-Tag: 1706051422-730801 X-HE-Meta: U2FsdGVkX19N5umtX7fmHozo5QMoy9B6XQs6/RcApx67Xm6XIzFD5jMFa17mS1jANgYruio5SG0kQZJ6xZ8RsqQxx8mgqwmfXI3T6fuOwkQdBP2iUdZboY+G65iOfqj082w3iv6vUbasbMqRCdWmrMbQWgHvQskqEf5rwy12yMxcNccQU1NrrwZ54OJ0n91PEKXYYIqxBi6pjBCDFXz8RjmWim56CPeVAeYQUgbWgVyPXPiMky2uDKl3AuhzDwmy4eRGj3dBJZY1DsITo9IDrEAwfQJQ2CriBb3RGnZqVW/CrbWu2ZDu2FaPLsMQVl4vF5ON6f6rFmfYbB8Aede3rCkMRkgq9988oJHyd1cCaZ8bRUxB6jIOFFVT6igvfEeAcvjPuhVeNy5IUTsM5zVTKnebRc4imWQYI2Vu2V4VOOOc3B4eu31w4xzBZo6lOWQqA7YjvTJp/Xxe3ad6SMtU+XYW7hDqJ5ATQdZQfkZ4xN8dS6xX4RHqhTCbTEVDIvy48S4ZsSapZnbK7YtvgqoG7EijGmknJk/FPSPw3VS49Yk4B+5tzbGoL9jbXgqqzBgdIuFtYuzZHor0NGfT1S10Tv90J5dJ8xjqrIntVBnuksAKMpZI+RKXIGSs/ULT8bKALj9VrA8hzjBQj0R48NT93xP0oMIQxXNEsfJ1ZHzFWdRqSsxBSTkLVgfJdOoVqxQUZVw1MUhu44Cmv6Qm+bSBW59010fGqWHTKr31xpPAjR6kcYWgKLZIuXVN34Sx1pDDt+M/nQIJqoqCpodntoqYrXiMuNjTk9H4FF1DyjcR1C742wp2tw2w0YUDR444BXH3DPf0zX0BxGMfkjVEGMC7AYUnqXc7silc2s/5kJWCM8tS7nl7UpzGIp1Cetm2QoiVqKy+JmFkui2HdcteLgDem1424HsgrDFbFcHBO53gDgSq9UMPB/M2aOtckNkdc3BkW/L9oxbDqXSNdPKwIkG FLkeIz1K O/mBscF2/D/r3zDge1Rl0NSDBeQP+DDkzJJdW10vWWkvKYsu/c/P07FCTKW3gosOUUeEeRe1PmTzBTGEV+0wCsBA99kJhKu6pF8xFu+6l+VXMkj+YFADvlhecIIoNAd5Rhj066bGQtYePH7ue2mFHcSZsdgfMKI1/+6w+hXrQ0EfJwEkZKbhKxxi0E1Zp6HWbH8xLqy4yeT6BuvNRteEF455ae23F8Uc7h7zpEmyQymIq+7Njqq0K8r7FYXU41kJjvBNW+N/G99JGmPmzqJQlcXrtjZSM82IWkHV/l7q0Ym7EcR1dmwWnncn7YUAGrwaN2Sy0FbTzDko7S8Bf5YFQmUSWbNOK8FPi1mlPHHaVdL2tCpn3dTV977GwD4w6/izHd+gNa/fSIVAFIY2VvnNSk+ueKBGGIxl50SK/VkF2aMWQIFdC+J2RY1x6bSsENU0Ti8B4jZ7d3i82U1/HNTQSU6xhMPwxevwFY4hFVc1UsUcgEjwbb7P2nYZi/HIkfZKU2gnMyRRClcnwKfsdhkn/O+/OXK7AHdbl1qskKLtA5pMPtD6gtWZ1M+TEI/XYx7u6fCfhuH3KCWH8hk/rJwKk02v+7AUUBaEq3elIZdrRcL95qWci0qWjYM3Dfjc6kIS5yFw+yNkY9U8WgbaNjE1+nDUuGgceWlrsA4doUhc4RfL9mpH+7wPHK5XPGDxNoVkJ7N4aRUPoqjgoAoR15XKM8IoolOwY/H5I7JgODZszSWrOr7o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With maple_tree supporting vma tree traversal under RCU and per-vma locks making vma access RCU-safe, /proc/pid/maps can be read under RCU and without the need to read-lock mmap_lock. However vma content can change from under us, therefore we make a copy of the vma and we pin pointer fields used when generating the output (currently only vm_file and anon_name). Afterwards we check for concurrent address space modifications, wait for them to end and retry. That last check is needed to avoid possibility of missing a vma during concurrent maple_tree node replacement, which might report a NULL when a vma is replaced with another one. While we take the mmap_lock for reading during such contention, we do that momentarily only to record new mm_wr_seq counter. This change is designed to reduce mmap_lock contention and prevent a process reading /proc/pid/maps files (often a low priority task, such as monitoring/data collection services) from blocking address space updates. Note that this change has a userspace visible disadvantage: it allows for sub-page data tearing as opposed to the previous mechanism where data tearing could happen only between pages of generated output data. Since current userspace considers data tearing between pages to be acceptable, we assume is will be able to handle sub-page data tearing as well. Signed-off-by: Suren Baghdasaryan --- Changes since v1 [1]: - Fixed CONFIG_ANON_VMA_NAME=n build by introducing anon_vma_name_{get|put}_if_valid, per SeongJae Park - Fixed misspelling of get_vma_snapshot() [1] https://lore.kernel.org/all/20240122071324.2099712-3-surenb@google.com/ fs/proc/internal.h | 2 + fs/proc/task_mmu.c | 113 +++++++++++++++++++++++++++++++++++--- include/linux/mm_inline.h | 18 ++++++ 3 files changed, 126 insertions(+), 7 deletions(-) diff --git a/fs/proc/internal.h b/fs/proc/internal.h index a71ac5379584..e0247225bb68 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -290,6 +290,8 @@ struct proc_maps_private { struct task_struct *task; struct mm_struct *mm; struct vma_iterator iter; + unsigned long mm_wr_seq; + struct vm_area_struct vma_copy; #ifdef CONFIG_NUMA struct mempolicy *task_mempolicy; #endif diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 3f78ebbb795f..0d5a515156ee 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -126,11 +126,95 @@ static void release_task_mempolicy(struct proc_maps_private *priv) } #endif -static struct vm_area_struct *proc_get_vma(struct proc_maps_private *priv, - loff_t *ppos) +#ifdef CONFIG_PER_VMA_LOCK + +static const struct seq_operations proc_pid_maps_op; + +/* + * Take VMA snapshot and pin vm_file and anon_name as they are used by + * show_map_vma. + */ +static int get_vma_snapshot(struct proc_maps_private *priv, struct vm_area_struct *vma) +{ + struct vm_area_struct *copy = &priv->vma_copy; + int ret = -EAGAIN; + + memcpy(copy, vma, sizeof(*vma)); + if (copy->vm_file && !get_file_rcu(©->vm_file)) + goto out; + + if (!anon_vma_name_get_if_valid(copy)) + goto put_file; + + if (priv->mm_wr_seq == mmap_write_seq_read(priv->mm)) + return 0; + + /* Address space got modified, vma might be stale. Wait and retry. */ + rcu_read_unlock(); + ret = mmap_read_lock_killable(priv->mm); + mmap_write_seq_record(priv->mm, &priv->mm_wr_seq); + mmap_read_unlock(priv->mm); + rcu_read_lock(); + + if (!ret) + ret = -EAGAIN; /* no other errors, ok to retry */ + + anon_vma_name_put_if_valid(copy); +put_file: + if (copy->vm_file) + fput(copy->vm_file); +out: + return ret; +} + +static void put_vma_snapshot(struct proc_maps_private *priv) +{ + struct vm_area_struct *vma = &priv->vma_copy; + + anon_vma_name_put_if_valid(vma); + if (vma->vm_file) + fput(vma->vm_file); +} + +static inline bool needs_mmap_lock(struct seq_file *m) +{ + /* + * smaps and numa_maps perform page table walk, therefore require + * mmap_lock but maps can be read under RCU. + */ + return m->op != &proc_pid_maps_op; +} + +#else /* CONFIG_PER_VMA_LOCK */ + +/* Without per-vma locks VMA access is not RCU-safe */ +static inline bool needs_mmap_lock(struct seq_file *m) { return true; } + +#endif /* CONFIG_PER_VMA_LOCK */ + +static struct vm_area_struct *proc_get_vma(struct seq_file *m, loff_t *ppos) { + struct proc_maps_private *priv = m->private; struct vm_area_struct *vma = vma_next(&priv->iter); +#ifdef CONFIG_PER_VMA_LOCK + if (vma && !needs_mmap_lock(m)) { + int ret; + + put_vma_snapshot(priv); + while ((ret = get_vma_snapshot(priv, vma)) == -EAGAIN) { + /* lookup the vma at the last position again */ + vma_iter_init(&priv->iter, priv->mm, *ppos); + vma = vma_next(&priv->iter); + } + + if (ret) { + put_vma_snapshot(priv); + return NULL; + } + vma = &priv->vma_copy; + } +#endif if (vma) { *ppos = vma->vm_start; } else { @@ -169,12 +253,20 @@ static void *m_start(struct seq_file *m, loff_t *ppos) return ERR_PTR(-EINTR); } + /* Drop mmap_lock if possible */ + if (!needs_mmap_lock(m)) { + mmap_write_seq_record(priv->mm, &priv->mm_wr_seq); + mmap_read_unlock(priv->mm); + rcu_read_lock(); + memset(&priv->vma_copy, 0, sizeof(priv->vma_copy)); + } + vma_iter_init(&priv->iter, mm, last_addr); hold_task_mempolicy(priv); if (last_addr == -2UL) return get_gate_vma(mm); - return proc_get_vma(priv, ppos); + return proc_get_vma(m, ppos); } static void *m_next(struct seq_file *m, void *v, loff_t *ppos) @@ -183,7 +275,7 @@ static void *m_next(struct seq_file *m, void *v, loff_t *ppos) *ppos = -1UL; return NULL; } - return proc_get_vma(m->private, ppos); + return proc_get_vma(m, ppos); } static void m_stop(struct seq_file *m, void *v) @@ -195,7 +287,10 @@ static void m_stop(struct seq_file *m, void *v) return; release_task_mempolicy(priv); - mmap_read_unlock(mm); + if (needs_mmap_lock(m)) + mmap_read_unlock(mm); + else + rcu_read_unlock(); mmput(mm); put_task_struct(priv->task); priv->task = NULL; @@ -283,8 +378,10 @@ show_map_vma(struct seq_file *m, struct vm_area_struct *vma) start = vma->vm_start; end = vma->vm_end; show_vma_header_prefix(m, start, end, flags, pgoff, dev, ino); - if (mm) - anon_name = anon_vma_name(vma); + if (mm) { + anon_name = needs_mmap_lock(m) ? anon_vma_name(vma) : + anon_vma_name_get_rcu(vma); + } /* * Print the dentry name for named mappings, and a @@ -338,6 +435,8 @@ show_map_vma(struct seq_file *m, struct vm_area_struct *vma) seq_puts(m, name); } seq_putc(m, '\n'); + if (anon_name && !needs_mmap_lock(m)) + anon_vma_name_put(anon_name); } static int show_map(struct seq_file *m, void *v) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index bbdb0ca857f1..a4a644fe005e 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -413,6 +413,21 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, struct anon_vma_name *anon_vma_name_get_rcu(struct vm_area_struct *vma); +/* + * Takes a reference if anon_vma is valid and stable (has references). + * Fails only if anon_vma is valid but we failed to get a reference. + */ +static inline bool anon_vma_name_get_if_valid(struct vm_area_struct *vma) +{ + return !vma->anon_name || anon_vma_name_get_rcu(vma); +} + +static inline void anon_vma_name_put_if_valid(struct vm_area_struct *vma) +{ + if (vma->anon_name) + anon_vma_name_put(vma->anon_name); +} + #else /* CONFIG_ANON_VMA_NAME */ static inline void anon_vma_name_get(struct anon_vma_name *anon_name) {} static inline void anon_vma_name_put(struct anon_vma_name *anon_name) {} @@ -432,6 +447,9 @@ struct anon_vma_name *anon_vma_name_get_rcu(struct vm_area_struct *vma) return NULL; } +static inline bool anon_vma_name_get_if_valid(struct vm_area_struct *vma) { return true; } +static inline void anon_vma_name_put_if_valid(struct vm_area_struct *vma) {} + #endif /* CONFIG_ANON_VMA_NAME */ static inline void init_tlb_flush_pending(struct mm_struct *mm)