From patchwork Thu Feb 13 22:46:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83E73C021A6 for ; Thu, 13 Feb 2025 22:47:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00A13280005; Thu, 13 Feb 2025 17:47:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ED6D1280001; Thu, 13 Feb 2025 17:47:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D500B280005; Thu, 13 Feb 2025 17:47:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B905A280001 for ; Thu, 13 Feb 2025 17:47:04 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4A25BA1769 for ; Thu, 13 Feb 2025 22:47:04 +0000 (UTC) X-FDA: 83116408368.03.F0F4F2B Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 822B512000B for ; Thu, 13 Feb 2025 22:47:02 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Dr7wzOhp; spf=pass (imf29.hostedemail.com: domain of 3ZXauZwYKCFAAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3ZXauZwYKCFAAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486822; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZTYk4jMv74ux8pRXqzfOQIPTG5wd2Gv112zjF6xUHQA=; b=SiN212BnoXqQTxvo2HoZS4Efrb3/d9J4ig7MIemfb7YmooXNvVEwe3D6hupiTmK7ywAoy7 FlirKZucTxHsFNlWCP2582WYDS8YnB1gmPnZm87Q3oXZu7Brz8duBQcYr8SVzvpAN98JXc aG/Q5upqjJ3jD777XUjHQm5n2ok0YqY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486822; a=rsa-sha256; cv=none; b=oov0ILU0TxZ27tYqhERo16rAla8Y2RONkcDKAKoJmevKP0H3WWj0EX1cEE7mZhi/LYr0kf iSDGDVVaHdSKGXV/KSko4s6b0qQPgeKWSsnSz1DlwuljsMzXIIq+Z9Ni1KwyeS/cZZIHYZ F1ohpfsoGywAwE0l+lQqcDGZW4VVHjA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Dr7wzOhp; spf=pass (imf29.hostedemail.com: domain of 3ZXauZwYKCFAAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3ZXauZwYKCFAAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-220d8599659so20725535ad.0 for ; Thu, 13 Feb 2025 14:47:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486821; x=1740091621; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZTYk4jMv74ux8pRXqzfOQIPTG5wd2Gv112zjF6xUHQA=; b=Dr7wzOhpoFYZdUauB+i314MhO88fzZXy1zD4mSFM7bA+pGR1uXcXX4oflpEY3VVyMi qFLor5RFmGTWFcnG4Bcr6spfij8QJZGLjqlvCUWaA60FQ8nma8DJfFZrCb9bvh0GxNKu WZQ83JC5xFpdy4SmxSloKJutS5u4bS0QywnqG/E8w2JeT1FeKA1h/eVQL3VsEMxrTBMI nDghEKFIH8ryBwlRgi+/3q6EZFXp37MBtGCtbBtakDnhEGnoIKwrb/IPPVopgMpomPWg luEPAbzDs8GDqd8lOLPOlo50ZsZ76BXveC6MOgvsn90oY/zatRJZIz3ppYCBHY2lbawE FTyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486821; x=1740091621; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZTYk4jMv74ux8pRXqzfOQIPTG5wd2Gv112zjF6xUHQA=; b=qdn/BKdPww6Mxg01Nydt6B2lUjYeGnNZdK1m9tW1TETN6jlj86tvSnc6SJhh2hgcwO Wyg2BsoimIKbV4tnjCTl/MFUvHYAdgF2os2zxeyk0ic9tR0gDSiVB3nzT6iSF1Dp0tIm 67fwrv4eubm5QuY+ThnT+oIeMEBbTzKVrTixtqKNtMvmCKMu49J8F8wWK217AtpfvRV1 cWCijIx+JmWunVQqHGAHZmKIQQTE9asKUlo/Iv8Dnx7HsjO9PyUBrbw9TjfhCxYo1CBW +HdFPbp3PJu5hPy6UHlCPzz1TjKv4R2rqmOYuU7wL8J01n5rHSFpDkvBjMW69sFjw0oT 3c+g== X-Forwarded-Encrypted: i=1; AJvYcCXggspkxbVjIFQh7stbFvNUweUyCRkUk1Usx7ZGMl7z/JUhaLPzb5bFLy4gvi2zoAB8VbTKVhBcwQ==@kvack.org X-Gm-Message-State: AOJu0YwRHgCt32nrz+hyCPYPjPrmav+i35oalVgTlZby0hBTYeoCiMkf T5U0NdVj7XjfMcMt16anRHbpfPAHm2fHJRIvCs8VguTAHP+TxE2chP2jpr7tJTPuEYnVX+mcqFP mMg== X-Google-Smtp-Source: AGHT+IGoxbQEHrRgpcds0LXGn7Wsw+mSSdilT5DRw00yKSF5iCDYNJNuQzJz7chpHTNiW9yQ8A9B90jlKU8= X-Received: from pgcu129.prod.google.com ([2002:a63:7987:0:b0:ad5:53f5:6975]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:112:b0:1ee:7b6c:e2f4 with SMTP id adf61e73a8af0-1ee7b6ce592mr3099897637.26.1739486821322; Thu, 13 Feb 2025 14:47:01 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:38 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-2-surenb@google.com> Subject: [PATCH v10 01/18] mm: introduce vma_start_read_locked{_nested} helpers From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Rspamd-Queue-Id: 822B512000B X-Rspamd-Server: rspam07 X-Stat-Signature: bi4ibrxyhtbarnbbtuzdo65tygamn8ch X-HE-Tag: 1739486822-368749 X-HE-Meta: U2FsdGVkX1880YzgY+JaI/fTOUagqqHJNJjCPqZHje/CNIuq/z15zXr3nEJXtmkTGMSHK1jEtdkpD2Uj6iQ0eVXZzqXBYkWn0KDJs1kvBXYWu5qvebNYlq5Z7yD8NfJySXud7YVZLxjCQ/0YA+haOpwbDyzhFoCp4bDhwxcze2U/X80v8x1t3xyuz4AZTWwAVm7tpmC60ieqv+v68T76ai1xpoj1xtlJghpgcHWiwnw3Y5VIYDdXJ3p2do9JgZR62t4hhzsK2Ou0NizMrztLu+vk4k89PgAQ7C/ukyhz46lEl43bNa+mqoX459q4Tq9UCqEw/NMVv3ZsXCsSfN1Td5e7+b8yyIrlRF2BpIdB/qVrI1WZYmqNLT3sVPko78agIgfgb/rCkQ4LiOw8twr+NRp8rTSJtkXGTOjX5LHoYoiK+/kU/o0XxnGvHVfBMd8IGQeAZouWDJ8c06iwTCEIyCP35RiGblNKpvTV4+Qi5yc7uAi0JNfm8VrAGk2CGOViOeI2lrwo4cE1LsRwAAuVFItnwq1YAmMW2vahbzmVUpY2wjGjKDsb7xbSRCiyN/Q9TYtRm1DNWoLIkhtrqpTO0xBVn080Jx10QDUHVdlKCbJVYZP7BOdl0LCXcpey37JFz3nKrxeRn1Vc+ce6WWs8VPlOckqZvkQsK1bvx3yA5E9090tDo20L7+9Q+PZ/s+2DIydqAoulmASU4rAeMfDcVowadDbl5GEX5eOMV9GmUoaQRGZxNa4rPiXMlKL3j3qmFYjMSV9buk2UOdn2DCe17Vr0ZObtlEaho44/suIHFh5UALMIBIMWit4/aEfDp/IgvIwp7iBhyZo+JWwRM7ZSkMJJ9Y7HyBGtg4vIOz5ohEevnXWF9wobtLFsnEs4w2q/c3JbTtbS9QfARCgPv1cSW1bXic4HVecP00v79HBKTi9nJ73syPDlq5ITbsPm6FFBkMG5qrmVU4ssBJpf6mQ f7zc5UwJ n7QRcJ3OHM72Ho/wlBTh48ioq/3Q/UDMGvyH1ocwFgpbwL57xntuNy+59xJv+/pECo6q2DAH+CIGVI+rqOXLNjQteMQP0eo03eTOGU0yIFF4r1qBByI+pj7mVhajBm2dExY1wQ+K1fUJIfBqqYE5BZrhJLsD1etR/LUNKPSpWS2bJlGXbV61M/WQXQYd8IeEXElTxNeqHGjWG0hvUM0hl8oiVXezsJ6Z3c53M9ff+O2v4E6/p9QtvFlg7wgzWzWqsnTydhv95fjDHLSGHZFfl01zc6FL5q9GROoeHv+CUNrZsUVO/9EA81rxS1hQiGfdFwtf5kKGhsDCe5zXko7MXQljGwLGNyO/tF6aw/4KBnF521PCRbw6QCqapBOqm92xria2rf9yXu3bHsIbJh4v4T8q223KgJxNXxZV2JtUkmIKS8Z32zaxRzJ/G8jCX5uIlWcCq15UuwFCvb7KiQL8xDAVU6YAqTRiX/lCX3A6bWkjO4Meozs7PzUpRTuQiWMRyGhueEHfJlA1A5+Vp1Cl37a6fWbvhVI/Fdn/yHw1Pab/ua4LB6KYQxMNvP6QJ+1hpCY+nyreHIwGnDrBdVA09QHu8L9AnnaBd/GkYKAQm0wSidK2G1MMZ1/MOIt5C5P8GAqT2M9Wwg8JmECgUrlfA7ZQyt3UHd8mmouak6BLEfe3E1GJIdqZc5TWAcDr+KuRGSmZDDBZ9bajFP5v5S6AmxQoK41pDgzyY+xF6iyaMUij+0Qo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions which can be used to read-lock a VMA when holding mmap_lock for read. Replace direct accesses to vma->vm_lock with these new helpers. Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes Reviewed-by: Davidlohr Bueso Reviewed-by: Shakeel Butt Reviewed-by: Vlastimil Babka Reviewed-by: Liam R. Howlett --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/userfaultfd.c | 22 +++++----------------- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 838097542939..16b3cd3de29a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -735,6 +735,30 @@ static inline bool vma_start_read(struct vm_area_struct *vma) return true; } +/* + * Use only while holding mmap read lock which guarantees that locking will not + * fail (nobody can concurrently write-lock the vma). vma_start_read() should + * not be used in such cases because it might fail due to mm_lock_seq overflow. + * This functionality is used to obtain vma read lock and drop the mmap read lock. + */ +static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) +{ + mmap_assert_locked(vma->vm_mm); + down_read_nested(&vma->vm_lock->lock, subclass); +} + +/* + * Use only while holding mmap read lock which guarantees that locking will not + * fail (nobody can concurrently write-lock the vma). vma_start_read() should + * not be used in such cases because it might fail due to mm_lock_seq overflow. + * This functionality is used to obtain vma read lock and drop the mmap read lock. + */ +static inline void vma_start_read_locked(struct vm_area_struct *vma) +{ + mmap_assert_locked(vma->vm_mm); + down_read(&vma->vm_lock->lock); +} + static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index af3dfc3633db..4527c385935b 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -84,16 +84,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, mmap_read_lock(mm); vma = find_vma_and_prepare_anon(mm, address); - if (!IS_ERR(vma)) { - /* - * We cannot use vma_start_read() as it may fail due to - * false locked (see comment in vma_start_read()). We - * can avoid that by directly locking vm_lock under - * mmap_lock, which guarantees that nobody can lock the - * vma for write (vma_start_write()) under us. - */ - down_read(&vma->vm_lock->lock); - } + if (!IS_ERR(vma)) + vma_start_read_locked(vma); mmap_read_unlock(mm); return vma; @@ -1491,14 +1483,10 @@ static int uffd_move_lock(struct mm_struct *mm, mmap_read_lock(mm); err = find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap); if (!err) { - /* - * See comment in uffd_lock_vma() as to why not using - * vma_start_read() here. - */ - down_read(&(*dst_vmap)->vm_lock->lock); + vma_start_read_locked(*dst_vmap); if (*dst_vmap != *src_vmap) - down_read_nested(&(*src_vmap)->vm_lock->lock, - SINGLE_DEPTH_NESTING); + vma_start_read_locked_nested(*src_vmap, + SINGLE_DEPTH_NESTING); } mmap_read_unlock(mm); return err; From patchwork Thu Feb 13 22:46:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F9A7C021A0 for ; Thu, 13 Feb 2025 22:47:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1790280006; Thu, 13 Feb 2025 17:47:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D50F5280001; Thu, 13 Feb 2025 17:47:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5561280006; Thu, 13 Feb 2025 17:47:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8A427280001 for ; Thu, 13 Feb 2025 17:47:07 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0C8EE1A174E for ; Thu, 13 Feb 2025 22:47:07 +0000 (UTC) X-FDA: 83116408494.25.ED6BBA4 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf05.hostedemail.com (Postfix) with ESMTP id 325A910000D for ; Thu, 13 Feb 2025 22:47:05 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=B8z894AU; spf=pass (imf05.hostedemail.com: domain of 3Z3auZwYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3Z3auZwYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nixUczrejX4FKBcz+HGm8JxV/4Rz6UEjTIeiIS536Tg=; b=ypiVGzS0r4U1kW4z1ykw9n7n857/AIMmrmeDEMifDA1+t8JpxLckIc56snRlYCtxaQXMJI gS5hSawTWaNPuJwFHnCne/ZoASjOEsA3QDsa65XeyofQHdkXcpYME26l3VX1ylyCxrBXp8 RvRidxEJcgcB90iiI+RmlD1cTV47yso= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=B8z894AU; spf=pass (imf05.hostedemail.com: domain of 3Z3auZwYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3Z3auZwYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486825; a=rsa-sha256; cv=none; b=qu3q+mFAKAzmFJ6agkQegesFbSSHuv2+q5CPuqiOouu0zU5VMk5os6w3kD9OGqdR2GsQAn xtzcNEpF3quOqKRF0VKocXf/+KuUAHyrzxvna/QjxSAcqmuyGEZPmDfHEM1kIEnlalEwf2 d3C9fXLobtUYy1ENME0rKq3yQeSTsxw= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-21a7cbe3b56so23867445ad.0 for ; Thu, 13 Feb 2025 14:47:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486824; x=1740091624; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nixUczrejX4FKBcz+HGm8JxV/4Rz6UEjTIeiIS536Tg=; b=B8z894AUSbbV61fwvLK0Q3EZqVQqzwoQlbOY94OLrfztJ0mq1uX3IKe0vA4Gw4julA HBWVkpGUPxQnD3TnuWCG8PkmO7MJWm/WBYJCGe+1dyWPNIO5PRwuY283ZsMa7g/m/dkB jOVEcp4b4MIZlX+x6JeLmBGbnkQaHUMqvnoA595eRwKNbnzsN1Nv27HpKD4qDZKzV5xg PLSfE9rz4RtjPU4c5YiFb/BdObCs6Ix3CoNIrQTi775LX9G+WU+DYJhkicx8UHLm3vC6 d3Ji3nDWytKc5Bfsx0BHAOzmmk+aMoklKTGCnN/EhIBkWbF7wmjJS3cZMWX8aZpZML/x B59w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486824; x=1740091624; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nixUczrejX4FKBcz+HGm8JxV/4Rz6UEjTIeiIS536Tg=; b=K0EZ1SOB9/I4QsZOkg3jVJbTG63UbYUONN7T1fYjVbwVGBDMapgBPKzPImtj67ZLi8 65pPWQNffULTI6SJn+ls1VY6ZiMIjVRt5/VeaSi8yxsmt2vS44D5AIz1JJlcSzCqzSyM 7V5IfcFiAJyb1glbOmsCaI4ZyjDflJ5p+ybmTMHE80XkyuZgY2EhN5IrCjvNukWKJq1b J4vZzXhcd1SPJac8J/+TF02Wm36JHKiQCZfKLO3GnJG8Ev7z/SNn9hiUjIU+LT9tbYN/ wh+2KQ5iuLszjnLQKgaDirBPk3nWi7IPfJq4Dnlla5e6TmWDu32bfDx1n7ZcaXG17tYJ jIoA== X-Forwarded-Encrypted: i=1; AJvYcCWG80cFDyO1ZLLsaph5r/cLhqxm+Qk3+sEy7FGJc60P17H/vlnkvUrUfgT4Hg6HxaF3KCUWui+5SQ==@kvack.org X-Gm-Message-State: AOJu0Yx8FQUMAj6dUkA3el7jWIfxQ0DzFB++l5wW3mhffdr1BOdswD8q FeI5Hm89aqrIIesBH/EZI4fc0MJxtoY2YMJhdOTdlaxxMvnRDGIihVyotg83a25XtwzeRFPYqUj uhw== X-Google-Smtp-Source: AGHT+IHexWPTyEo/ZnzJkqYLOzvXPZYi2c9eLV49bS4E26ucZsJK7lHyl46qKggig4vIRb/FsPaNzCiwY30= X-Received: from pgjp13.prod.google.com ([2002:a63:e64d:0:b0:ad8:73cf:4390]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:c998:b0:1ee:1e05:206d with SMTP id adf61e73a8af0-1ee5e5b7c8amr15452640637.21.1739486823384; Thu, 13 Feb 2025 14:47:03 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:39 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-3-surenb@google.com> Subject: [PATCH v10 02/18] mm: move per-vma lock into vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspamd-Queue-Id: 325A910000D X-Stat-Signature: kaeicw6rk3nhjjre5b4cu4x1re5ko8em X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1739486825-639567 X-HE-Meta: U2FsdGVkX1+RM15k5almYJgl+G0G3+SINKehfFdSdufmYphcPdNwmHNW3eYycmp7vrZcvDohROxydMfdCcSZJH/P0q9QGRzTsqgKTaAjWfj30h/mN4DCXYFdT4i1dg8ScmeHABha4IbHYBtbzHykP4Vzp5QUnQILZi8wajxlJmPawBFUGjgbRWTa6LY5Al47k5Fj0Il9zpiqNOdPS503GC0h8fs5AZ3w/+CS5/klaOcNhwB+cWpr6AfyeTRnUWcjssR/7NGK+kwWciyNJasfonciFioVeXDbMRu+AbPzZ/kVM6YOTbwP/Us3iddifYRC0LHF+Xzbd0VGs56eMHxdVFDueIuz3mvEgwc+eF7iTXPsTXsEuGProSgbrJr2x6iwZKQJUMKN8yeKLzQPgEkFslyutQWjyEpi2ZFrUNlkWqgypOhPtP/0A/meAJNJK8CESo7ola5W9KnRfNsyRnLLpggr3d2pYLWk7RBoloqTpXQxJ9FTh5hkspFBVb8kJtS/U7sL0qZO1u6AN/dhIMrXxruLB3IzyMS2RotcIseaMxO+qzMswmmiW4RI5gbC4d5ibFp5G/eqh/00gSM9pBP/a0jEOtSXI4cSdLlLcdYX5Ax6FBSCxbWLCfHi+ExLV9Ni659yg6cCLDNuzXbFcZeUsvEnT3ILWZP1dgIyzoLo5NHrWFTylC/AWsvvnR/S38lVr6U4Mk0QUIuZsOHxXDqJB5AdaWICKg6J3mC5lA9mCSbXwVvMR8MRku3NJeniFlkK4DctMmb4MLv0skHOjFPuDYk13nZctUOX0sSXvs//mJqH3BUk+L4bphPkqTuhDZZza18YB+sHcZUwOGnxQ23eUj55DJGBte5m4tonMIkU+Rvik3T1s2x9fD4BmMcEpxXcH4DLE8LoYA1FFYzrOqWSJlwiVJafGQdOM0oN6052KkYonAu3NKGEUULizJjAXKO+pF9CBifeH5Iw5+0y7B9 h8poJBuS Trxjd14Tj1o17ExLGb6OM3LtVBhgO8/0JoTUpcXrLH5CbfdojmkjAequow8TQ2z4qcR+wDjAc9h80JPUe9gNYs8ZIkL2/m+oCnmWytR5mrGFY+eQ6eGniyDiZGIWgNWR/5NFiuKlJYQ5cLw7VXIBQ1x7xQ1QasaeHfbEXVA8basC6EW1AbTWmqOwO/QEdV3UOk/Ml21Bcr5DcGHrlI71/9BpPrpZDDGARJdXwDtPZhBT85HBuZj7A3LQyQJhsyhpflUQdmLW6SjsGPtxKQ2K45SLW0zfgu7N0jvp4TEcGr4zwSP2DjMICXK8WLt5lAVIbkGcVIecsgLfjftWe+IPGo5tYxn0tTCI0O+92TPzOWCvcDQsOTPhYW4+rjshyLOxDC/oJ1ucfgNp5RFhCtN1zJwh2LDTWSMHK3KyfeA0lw1ol31xk1Fps6pvaEmR6sKB+1QGQ7vr4Az6IKuJvf9Ef/onitKytjNnFDWMCgFvAi9p+a/GSv0iLBKhxiOMiyeU9mdgcZgSB5XV8qhTlvmq8O/PoHTy01UcKzvkCWU1A3LFbYZfx4TTxtwV/68PP5FSF0aTPTT/Utw/WCzJO8SJgv/FZViVpUXjLnMa7TSOrGYS7mE7YgETjeOCpC4FygBnXCgabguggYmGWVA+ZKZPZWnvUYrDzRXA/5Yt5P+OKASxYc1XZcvrQB7uQqK4WK0jBNy3UcaZcmSg7ljSCANm1vC9m+PwGTufJNOaS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. Move vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned as well. With kernel compiled using defconfig, this causes VMA memory consumption to grow from 160 (vm_area_struct) + 40 (vm_lock) bytes to 256 bytes: slabinfo before: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 160 51 2 : ... slabinfo after moving vm_lock: ... : ... vm_area_struct ... 256 32 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 pages, which is 5.5MB per 100000 VMAs. Note that the size of this structure is dependent on the kernel configuration and typically the original size is higher than 160 bytes. Therefore these calculations are close to the worst case scenario. A more realistic vm_area_struct usage before this change is: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 176 46 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 54 to 64 pages, which is 3.9MB per 100000 VMAs. This memory consumption growth can be addressed later by optimizing the vm_lock. [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes Reviewed-by: Shakeel Butt Reviewed-by: Vlastimil Babka Reviewed-by: Liam R. Howlett --- include/linux/mm.h | 28 ++++++++++-------- include/linux/mm_types.h | 6 ++-- kernel/fork.c | 49 ++++---------------------------- tools/testing/vma/vma_internal.h | 33 +++++---------------- 4 files changed, 32 insertions(+), 84 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 16b3cd3de29a..e75fae95b48d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -697,6 +697,12 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {} #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_PER_VMA_LOCK +static inline void vma_lock_init(struct vm_area_struct *vma) +{ + init_rwsem(&vma->vm_lock.lock); + vma->vm_lock_seq = UINT_MAX; +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -714,7 +720,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) return false; - if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) return false; /* @@ -729,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); return false; } return true; @@ -744,7 +750,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { mmap_assert_locked(vma->vm_mm); - down_read_nested(&vma->vm_lock->lock, subclass); + down_read_nested(&vma->vm_lock.lock, subclass); } /* @@ -756,13 +762,13 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int static inline void vma_start_read_locked(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); - down_read(&vma->vm_lock->lock); + down_read(&vma->vm_lock.lock); } static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); rcu_read_unlock(); } @@ -791,7 +797,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (__is_vma_write_locked(vma, &mm_lock_seq)) return; - down_write(&vma->vm_lock->lock); + down_write(&vma->vm_lock.lock); /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -799,7 +805,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock->lock); + up_write(&vma->vm_lock.lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -811,7 +817,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_locked(struct vm_area_struct *vma) { - if (!rwsem_is_locked(&vma->vm_lock->lock)) + if (!rwsem_is_locked(&vma->vm_lock.lock)) vma_assert_write_locked(vma); } @@ -844,6 +850,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ +static inline void vma_lock_init(struct vm_area_struct *vma) {} static inline bool vma_start_read(struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} @@ -878,10 +885,6 @@ static inline void assert_fault_locked(struct vm_fault *vmf) extern const struct vm_operations_struct vma_dummy_vm_ops; -/* - * WARNING: vma_init does not initialize vma->vm_lock. - * Use vm_area_alloc()/vm_area_free() if vma needs locking. - */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { memset(vma, 0, sizeof(*vma)); @@ -890,6 +893,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); vma_numab_state_init(vma); + vma_lock_init(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8efafef4637e..8a645bcb2b31 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -740,8 +740,6 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - /* Unstable RCU readers are allowed to read this. */ - struct vma_lock *vm_lock; #endif /* @@ -794,6 +792,10 @@ struct vm_area_struct { struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + /* Unstable RCU readers are allowed to read this. */ + struct vma_lock vm_lock ____cacheline_aligned_in_smp; +#endif } __randomize_layout; #ifdef CONFIG_NUMA diff --git a/kernel/fork.c b/kernel/fork.c index 735405a9c5f3..bdbabe73fb29 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -436,35 +436,6 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; -#ifdef CONFIG_PER_VMA_LOCK - -/* SLAB cache for vm_area_struct.lock */ -static struct kmem_cache *vma_lock_cachep; - -static bool vma_lock_alloc(struct vm_area_struct *vma) -{ - vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = UINT_MAX; - - return true; -} - -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - kmem_cache_free(vma_lock_cachep, vma->vm_lock); -} - -#else /* CONFIG_PER_VMA_LOCK */ - -static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } -static inline void vma_lock_free(struct vm_area_struct *vma) {} - -#endif /* CONFIG_PER_VMA_LOCK */ - struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; @@ -474,10 +445,6 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - kmem_cache_free(vm_area_cachep, vma); - return NULL; - } return vma; } @@ -496,10 +463,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) * will be reinitialized. */ data_race(memcpy(new, orig, sizeof(*new))); - if (!vma_lock_alloc(new)) { - kmem_cache_free(vm_area_cachep, new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); vma_numab_state_init(new); dup_anon_vma_name(orig, new); @@ -511,7 +475,6 @@ void __vm_area_free(struct vm_area_struct *vma) { vma_numab_state_free(vma); free_anon_vma_name(vma); - vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -522,7 +485,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); __vm_area_free(vma); } #endif @@ -3200,11 +3163,9 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - - vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); -#ifdef CONFIG_PER_VMA_LOCK - vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); -#endif + vm_area_cachep = KMEM_CACHE(vm_area_struct, + SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + SLAB_ACCOUNT); mmap_init(); nsproxy_cache_init(); } diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index bb273927af0f..4506e6fb3c6f 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -275,10 +275,10 @@ struct vm_area_struct { /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) - * - vm_lock->lock (in write mode) + * - vm_lock.lock (in write mode) * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_lock->lock (in read or write mode) + * - vm_lock.lock (in read or write mode) * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * @@ -287,7 +287,7 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - struct vma_lock *vm_lock; + struct vma_lock vm_lock; #endif /* @@ -464,17 +464,10 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) return mas_find(&vmi->mas, ULONG_MAX); } -static inline bool vma_lock_alloc(struct vm_area_struct *vma) +static inline void vma_lock_init(struct vm_area_struct *vma) { - vma->vm_lock = calloc(1, sizeof(struct vma_lock)); - - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); + init_rwsem(&vma->vm_lock.lock); vma->vm_lock_seq = UINT_MAX; - - return true; } static inline void vma_assert_write_locked(struct vm_area_struct *); @@ -497,6 +490,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); + vma_lock_init(vma); } static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) @@ -507,10 +501,6 @@ static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - free(vma); - return NULL; - } return vma; } @@ -523,10 +513,7 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return NULL; memcpy(new, orig, sizeof(*new)); - if (!vma_lock_alloc(new)) { - free(new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); return new; @@ -696,14 +683,8 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - free(vma->vm_lock); -} - static inline void __vm_area_free(struct vm_area_struct *vma) { - vma_lock_free(vma); free(vma); } From patchwork Thu Feb 13 22:46:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7423AC021A6 for ; Thu, 13 Feb 2025 22:47:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AB50280007; Thu, 13 Feb 2025 17:47:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 93184280001; Thu, 13 Feb 2025 17:47:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73613280007; Thu, 13 Feb 2025 17:47:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4D14A280001 for ; Thu, 13 Feb 2025 17:47:09 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E6A031416BF for ; Thu, 13 Feb 2025 22:47:08 +0000 (UTC) X-FDA: 83116408536.11.AD234C3 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 3442F20005 for ; Thu, 13 Feb 2025 22:47:07 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yoG7ZMC3; spf=pass (imf13.hostedemail.com: domain of 3anauZwYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3anauZwYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IpC5RSofcM0ZStaR556vDop/rdew408+n2xS+qUK8/0=; b=zEzcBKh1zbbLqAGL/Zi2xLNDpyB05I3u35WToNYEWl+InXXmtKyqRkfZNfQnIrvDoUEPpF 4DqZlEErRi4oq7HzG+hB8JaB+hiGn/gxZavF9LKxtXLizOkYCwUI7H0AinM9sqnnoOji66 gMnQMyZW1qxPOa3DPuLqwkqk74c58Vg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yoG7ZMC3; spf=pass (imf13.hostedemail.com: domain of 3anauZwYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3anauZwYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486827; a=rsa-sha256; cv=none; b=4ob+u/NjT34m/QfpxD0ZLBF6CaOlq0cJCR1kDhPIXDi8GEheu9YHM7oyYxXqAEBlJJB5HZ sGd9dKg1YZVS5U3k8u7XT8p9jRWJVmM/ExDBnjQU00r2nozZqHQBdc9dbUHhgwx85tAKHg yrV3zstKKsIYaTtN/4pXFDXzK0Qub9c= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-220e1593b85so16022135ad.3 for ; Thu, 13 Feb 2025 14:47:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486826; x=1740091626; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IpC5RSofcM0ZStaR556vDop/rdew408+n2xS+qUK8/0=; b=yoG7ZMC3VuCXW4qRutJG48NrlfjrE0Jm0u/KfRu5fQGdgCSsZUXfvBEyAn1J6e7sb6 U6ESv2uGCCakTucQPjhotLrp7fyAJBJ7fkwIjziE8yLSSDlgUtTYSuwjflks6eUrXHgG CZHTC9uWAdkW8ozryqTj3E0kvCUw9Fy/IwqM72CCsNhfyx1NUtzkVA8rEFV8XiF3Defx 14uKWqPMFzn5Iyy/7obgI5rV7AOf3HM1Dqv6c/hCZIAdgKhRzqCUh44Js+6z2SY2eyJv MZdHDuHiOHo5XfjyxzQrQ7OdZbeS16ufoMEccMlg+BUY8W8kguYDuUijRr7vPVUSIFZo C/kA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486826; x=1740091626; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IpC5RSofcM0ZStaR556vDop/rdew408+n2xS+qUK8/0=; b=UiAFoPbr5VXGImEjMqEfkcDOfi+D+O0P5yBNuWvATm8iG9cIu3lTQISCDNXXz6IyRo hUOuFhUG57aCS47uyrJzWlCCSRwd4VAeBLhvg53hI2qwUJKq+Kj6VGNySOVdq7aOAHRf GeHR/BI42nsJsqf8jFyHnPGRA72Cuba9Mmr9V/jReiPb6aWOoulXz4ZawIBewUqHhWC5 vpoNz2kqwqWvSJXMAkOQC0SaYhtxTRrtR7Cl2pEQsRe3I4qNdbk4CqLDN7x1LDhJEgIn mzkYdvbrn3J2U/o2koF2yTuQLH5Xb9XU5wE7Nr7xJEHTHPZgNXx1f9X0/JNldi45R7Fl /8hg== X-Forwarded-Encrypted: i=1; AJvYcCUUzNCFScIL3jZfudxtQ7DitnQvku5MAeb05w8wc2QzhAMtrtJ+s7gSYo4gANIt5e6Dmb7CrVZVRA==@kvack.org X-Gm-Message-State: AOJu0Yxys+n6g39JXT8uSQ9FAUZG1Qb1GtCZ8QfWyNIDK/xtSBu0QruY qhFj/9GkWEU/Eyt4ctGzfYO4C3sQRYEClD2KTcRstcURbqHJSliiBdK33ETk21OdNK8hTV/TI80 BZA== X-Google-Smtp-Source: AGHT+IFlXfVeKvLwc31ADPhBrs1WWy20ACeukrGHKQSKrbfZQq/alc0hc5JcyIHneAFIaVwEe3JXjLuYCE8= X-Received: from pfbbi29.prod.google.com ([2002:a05:6a00:311d:b0:730:83d2:d6a3]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:50a:b0:1ee:69cb:4b42 with SMTP id adf61e73a8af0-1ee6b399cd4mr12028188637.32.1739486826085; Thu, 13 Feb 2025 14:47:06 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:40 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-4-surenb@google.com> Subject: [PATCH v10 03/18] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Stat-Signature: difyctx5rs11imewmdcms5ddud3on656 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3442F20005 X-HE-Tag: 1739486827-35743 X-HE-Meta: U2FsdGVkX19VGumPUmtzS4pTUqnovoDRPq9L3f4hWlU2wit3aAhsLj1JsLFeI7w8aL8N2PTcSsYaKQXYX1WXLSBprvBI/shOYEm6bvzxNUqVYg+AG3KTG+RlPdhln0ecpymGRKFlDsqYBSBrUfaPfu/3lcIO1M4g3ghHkUT99x+X8ATKqCZI+gRFgdiKi8b+ZtFHUTwbD1iruoWtC78Yz2NztQwUtXopaAR0jP5fCwlACn3VtjKUTkq9Dc5fTth9amNjEQj2snmLo67Eub6h8lIqalw1m8vKJNEDBiDBOElMeNDHbUelsWno+r4KaxCfZF+1iy9FgigQJY2z9qTU9TTM1glYWzUyDy1RhuPBAgH+xg/9E6PnazwgNCk4UWZgm0KF9WqwYa+Tuh5MhsjupkIQyoDbhxSR/ovS37LyEA4WhPEE5TFBKazFaWyNbrxwrDQ4s3eCU5WtA59enkgI33AWgTpIDaURmKWfI29/pPbmXsW8JXtM0ffJ9Lfdxvy2Hb0LlZ6VZhXBo5gdjUuFPXzAMEmsVmvPFZjDNZQbqDwfqlBRR7QgfsjCllx96f2hM+ISGC+4lJzZrSYx8CL7Z1RN5oVRd+vxBQ4EHHAse9lzoFqIC+s+lmtDh6eZGfYt5wTJxnUsen4tTZe5ZfiNdvrUMqeXsmgHyb/tzMRSF3Ysr7karu7rZAXtgOm0/Ag/PJorBuAwnjCW1cz1eCnpjWd6qOQSwuBdC0+WvNlBVic0/KYQZY5JNeF6zkPb1Tg9QqIgFBQm48ijDGADKo4uzmcAqIEwx94buBL0k3fOX+DTIogCP0SJ8RyVtJndHEdUHsPScewH0QlFz++t+S8QQbiBTjM9x3WB8TcU+y5MdQs6xJ+XNkBno65rVdT1+IByxRjHW0Sopsd7fhLJmTlb4LYaSW71XXkqPejmSr6dPwiuI/+0ubsAHAZxfDOtJImn/xUshhFSRNxJszTqLP3 58qB4Ue4 TLZvhugFMAJ9lvmLzCL1h7BomTIN69My54AuD7x9Fap24OOk69Lpmtcin+Yajdy30JuqomdzqNsM+7fEVW58g7wGUsa0vwGbNuRTMTsnxEr1T51MdvLbVq7R3GKm7xPzTnpyPA4Ni2v7DVLrx2QEdS11wFbeMt3znDn+wjhJijz0QlQO0DEjaJWNiYSRKWMus6upSD1HuA4zaGTWIPuLzTHrLMyPdg7Iqe5BW285jZdAhBKZd27jGDnKBCSuEk/uTDQJ0cELtXIDx/ehkjRYTZ9DeoWKRSFua3ghXDOSpZrriDlxo6uH6OqcwgP3iQaeiDoqUzI/lMtXVV2jkb2Lw+uB6hJG1VEsxkb9uNDkbhrIJxJnoqRMobNTZ8ECPsqzIL0fswYczKKnoJwJpEuU0me6FoHuVEvyuhbWnHiY7kv6bUdTJg434/zV7U/VMwh0Ad8IEG9I6HYakHvP3vx9eGgpjOat9y0r8ocOHp+v3fMhUPMprSHmE3qgoON5J9eiJjXMPxC6CLxy81FmoGolhouet/24eDUBVqrwJa/sv1q54B5HWkdcuzmtdUt+R90bVMV/uSkWdFfBirloNJhgTQ3MS05cqPWMvMn7xV9xkruFOwv2wYBNE4ELOgynACreHJ3sqI0vymEZmjsU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka Reviewed-by: Liam R. Howlett --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e75fae95b48d..cd5ee61e98f2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -821,12 +821,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -857,8 +866,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -891,7 +900,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1086,6 +1098,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index bdbabe73fb29..5bf3e407c795 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index a8d6dbd03668..e600a5ff3c7a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6414,7 +6414,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index 39146c19f316..498507d8a262 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -341,7 +341,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1238,7 +1238,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1313,7 +1313,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index e55e68abfbe3..bffb56afce5f 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -205,6 +205,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -437,6 +438,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 4506e6fb3c6f..f93f7f74f97b 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -471,12 +471,16 @@ static inline void vma_lock_init(struct vm_area_struct *vma) } static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -489,7 +493,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -515,6 +520,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; } From patchwork Thu Feb 13 22:46:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65F0CC021A4 for ; Thu, 13 Feb 2025 22:47:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA7A5280008; Thu, 13 Feb 2025 17:47:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B37BF280001; Thu, 13 Feb 2025 17:47:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9312A280008; Thu, 13 Feb 2025 17:47:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6A774280001 for ; Thu, 13 Feb 2025 17:47:11 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 197DA16180F for ; Thu, 13 Feb 2025 22:47:11 +0000 (UTC) X-FDA: 83116408662.12.2D64AB3 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf16.hostedemail.com (Postfix) with ESMTP id 439CD18000E for ; Thu, 13 Feb 2025 22:47:09 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=sEKmKHDu; spf=pass (imf16.hostedemail.com: domain of 3bHauZwYKCFcHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3bHauZwYKCFcHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jH+IjsWoi1CDqDTbyIn4LxmH0nN4gQ/NWbFp+/jF/ZM=; b=ruTwWdILsn5/vSiOYzqumbj+J2UvlCg12QC6vyV+rgWKozr19aTeSUP2BgmbcIeoJZyV/V UlPzKVuewcydYfFAboxvBgPFzyoCEt2o9GXClKGpM0+BBt7CQnzKVrY6FIo4TKTZJSUZH2 Hc3+7Kx5kTwI1JpFCR4Gasx15pLpoT0= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=sEKmKHDu; spf=pass (imf16.hostedemail.com: domain of 3bHauZwYKCFcHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3bHauZwYKCFcHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486829; a=rsa-sha256; cv=none; b=oeXLnRWlsx2kuwrCOlHA2DhrwqQzROyzCXffqVpbNgvt3hQJ+zeUy10f+uLiAIQ80IBU0c Rhq4BmU20eSzErX+hcuW/pyzqD2JSJL1DQ+FdaalX8VvJjgrYX7Rn9vDy6XhawdtbsngIj hYUaeqNpcMm8I+qS4krlFWRXjLypi1A= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f81a0d0a18so2884224a91.3 for ; Thu, 13 Feb 2025 14:47:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486828; x=1740091628; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jH+IjsWoi1CDqDTbyIn4LxmH0nN4gQ/NWbFp+/jF/ZM=; b=sEKmKHDumm2B8zd4nfibDfGGH2xnajyiyLPAfz4S5jUAU/TTleADcdzU/nf7FNW3Vp DfzwEoUQOcVdst2Ca/Ej2dx5wBy4ehU9/x6zNJ6RiKR8Vby0sD4kj+pLihuO/JdKXjjY h+8k4ia+9cKfyQWOR12+d0pLyswj0iLwnSYZaUrkx56+bizFfS8PdhzRfgRzANj8dFuR YO2dJePs8OAMgja0PNy3WWF2GwS/+2v9Ycwg2VAZhn96exhW7e9pmSAWT5TCYdnFO3L6 kXGf/xI6HXSUEoAe1++wmhqV9veL9HZ8LnPKEQNl0cfjr10N6YXtCwr1fygodWFmy3pw eTWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486828; x=1740091628; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jH+IjsWoi1CDqDTbyIn4LxmH0nN4gQ/NWbFp+/jF/ZM=; b=fNEAetmzQUYha2PEuMWzDISqftdld9j0bpoqhHEOaTYL+JRbDy555l0h5wQBLdQLtI GdmIVxEn25sejI/w7eyXWWjb/DqCvJUfbLyzOQmL/oWEqO81LMezLvYVIK4BeUssdWey kRUgdvcGTB4ctMN23T6sVTwlIKX703bQtmBK73uD/PNJ3OJDfU6BdIyd+SZTfu3waPeS hCismtS2Az5GMFs7aU8rmWWej7BidDc8OMkEzV+f9BZLFySjky1IfCCvBJZcztCMX6T/ 1VforBZGIrZ5bdJBIPQayeibkhDcB4GlhLzR8umTpTAtKtXCmY/0vFP4ETyU404nYJc3 3Z7w== X-Forwarded-Encrypted: i=1; AJvYcCUBSUaWAbOkQGghzUfl3mcAN9PSZfT5Aj6dBKWsCnEEIOpQh44FmZQOA2IMtIfkWcCK5HIXeZ82WA==@kvack.org X-Gm-Message-State: AOJu0Yyi6tk1dZui3FR+iCySyFtdnLHF5Fr0jylrMDslvqs1ttR3W6/3 EqPnoh8B319anO4aLky9NfryC+yW8QwSATl8lg0gpNvvo0T0lFNOJkFDMBmGPH3gGuc3q0O5Jen xyA== X-Google-Smtp-Source: AGHT+IFAejlKUMRiGK6lse6cj9gDccmhauhvLNb8QCnLKB76TP7y/Vj9rouXSbsWYDlN1QYS+niVoTXiz0o= X-Received: from pjk14.prod.google.com ([2002:a17:90b:558e:b0:2ef:7352:9e97]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:e7d0:b0:2ef:e0bb:1ef2 with SMTP id 98e67ed59e1d1-2fc0e98dc7dmr6643326a91.19.1739486828102; Thu, 13 Feb 2025 14:47:08 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:41 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-5-surenb@google.com> Subject: [PATCH v10 04/18] mm: introduce vma_iter_store_attached() to use with attached vmas From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 439CD18000E X-Stat-Signature: 74wihc68tcycw5mq9btuc9bmuzcag9mr X-HE-Tag: 1739486829-617264 X-HE-Meta: U2FsdGVkX18tZr+pVOMAPoWItlsUxx8y2ZHXDL1i41onpKSAOTy2+/F+wRg3eR1H9tP8wbnM1fQF5ER7gMUxtfOxB4ekO6VNZwQwD0kij5/FiU3H6FIYYMO2FHtfp1T521LEUJJTlYwNVmKUyUthlONibVvQQvabBV90xx2B332vbi+mPjuHUXaFotsNHlU6wzFIt2yMarYxoXmoLTZ2CPMYWLQtYmzfPNkTo8bZP+d31W/GCG3Hq4KXn7du70qRX7FfKWpHRbqcLcQxEnqzJsuBYciOhjZWsZhGjPjXN9qa+S7WEX/kTM9cwLnY1pNyUsJTZBM14WilQqEWovz3ZpezQ0VRyzFt4eHxeUkC8XD596JYUQCVwOVGPRq7BSobowu+68/wsXDtiGWpDTLK7GyHnAiF9FXwwYc47oEazaSLlH/TbM1MsU59Cz7W6HihAJgw8jvip2b+L0GsJ4lxzHrco8lfpZMUzEPu8I9sgs++pafV8+1u3miaxAPZnga0tjDLLzylW4NZr6WPpBbRdhsf7JnQcPYW0VDtv5ywNse8X9F+eyJfmhl5GqJwYuEzig5GWjk4eEAEWFcLFIeZk+lxG4AoCDWEd1oIt0+310iS03pTPq51c2flJUHHVPwanELOwK/LHRNKBWaq96Wv5Ez31cNWfyn8hJzc6kNg5iOV3VQKCGPSYVEGd8pfM8VJwFAdUOj5jXOnSFkk5FsJ1Ln683JDSAEMXnWNJLUHic8ARK2Na67QY4Qqaxiu429BBh236PjoUROJTGvtmA0vIfy9jxJmsYFb2U8BUjeYkvZ1IppcfU5q7gI79K4xLQELv0CHt6YNOzQf4CZTqGMfEXGyygI0mh1AzcUCFSP4RP3qjJx9MWe/19Ym0gYSW6Q8EScGflabzkX+8AWxMImXek0hVWI0HUMJD6dX/kYVlv/mlZF6Am/dG9RL7XIjwgYwWLbgQ4k3jUUUStQ9WJt kRTPF+Qa IJqmwwZ2dSqjQDdjb8t69PevsPaU5Fk0dlht3A2ZqnasLdG721A+IHIqQxTgDMaOo+WrSmDrlb2m3yCOjFaaZHGVTc1Ld/wY0qjeL3AU3jADOyVRf60XaA2GF7unntkBchbZGOFWiOoL14Y4cHR4P6Mca1fQ31bRrpgf7a/RgOYuAuENUKZFY7Nx6eSzt1HkKLMOl1KYRKj6W3CEzQA5xNhTAmorH4cspCZ8FZ0XoonfNCz7P9xWHsV+5rmOPulCkW5bfTQhCWhwsGU58MhaLEXKYuk1440Ahk4ugP6aYcHi6FqU3nfZciyDdcR41Uw0vF4r8iRF1tzichu2LRiTqolgAE9z0QXRlxyKv7H3TbvksWZwXqqIypxHpxvv8g15gBUpuDmmh0/kyqnGoY+xdCka3o79kl60PQ9H6yD6Z3nzvxKWsi1EHc7On3Sle6fWiYZmOFO9CmA4ldFa1C6DKDAYOm/ZmyCH8xL7XDXqmOmmW/1FudlIfywnul3roUTJmAJ8o0il6a3xpTiH2AtNya+zpI/KuWDRS8u2OJzmh2qTGCb6JxBVCiGQtE1Y8gkX0MhOlsDOLxZYBXDQ+Cq00YRp6kxyGAKhmzjf0hMyUNtlZJsj5KBFGRJs3aTBVazFuJosJoovQpG08wAf6W3bLj+l9oq2z4kGYHefd2e7mBd/rci4ejRgTimaGPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vma_iter_store() functions can be used both when adding a new vma and when updating an existing one. However for existing ones we do not need to mark them attached as they are already marked that way. With vma->detached being a separate flag, double-marking a vmas as attached or detached is not an issue because the flag will simply be overwritten with the same value. However once we fold this flag into the refcount later in this series, re-attaching or re-detaching a vma becomes an issue since these operations will be incrementing/decrementing a refcount. Introduce vma_iter_store_new() and vma_iter_store_overwrite() to replace vma_iter_store() and avoid re-attaching a vma during vma update. Add assertions in vma_mark_attached()/vma_mark_detached() to catch invalid usage. Update vma tests to check for vma detached state correctness. Signed-off-by: Suren Baghdasaryan --- Changes since v9 [1]: - Change VM_BUG_ON_VMA() to WARN_ON_ONCE() in vma_assert_{attached|detached}, per Lorenzo Stoakes - Rename vma_iter_store() into vma_iter_store_new(), per Lorenzo Stoakes - Expand changelog, per Lorenzo Stoakes - Update vma tests to check for vma detached state correctness, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-5-surenb@google.com/ include/linux/mm.h | 14 +++++++++++ mm/nommu.c | 4 +-- mm/vma.c | 12 ++++----- mm/vma.h | 11 +++++++-- tools/testing/vma/vma.c | 42 +++++++++++++++++++++++++------- tools/testing/vma/vma_internal.h | 10 ++++++++ 6 files changed, 74 insertions(+), 19 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index cd5ee61e98f2..1b8e72888124 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -821,8 +821,19 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } +static inline void vma_assert_attached(struct vm_area_struct *vma) +{ + WARN_ON_ONCE(vma->detached); +} + +static inline void vma_assert_detached(struct vm_area_struct *vma) +{ + WARN_ON_ONCE(!vma->detached); +} + static inline void vma_mark_attached(struct vm_area_struct *vma) { + vma_assert_detached(vma); vma->detached = false; } @@ -830,6 +841,7 @@ static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ vma_assert_write_locked(vma); + vma_assert_attached(vma); vma->detached = true; } @@ -866,6 +878,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } +static inline void vma_assert_attached(struct vm_area_struct *vma) {} +static inline void vma_assert_detached(struct vm_area_struct *vma) {} static inline void vma_mark_attached(struct vm_area_struct *vma) {} static inline void vma_mark_detached(struct vm_area_struct *vma) {} diff --git a/mm/nommu.c b/mm/nommu.c index baa79abdaf03..8b31d8396297 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1191,7 +1191,7 @@ unsigned long do_mmap(struct file *file, setup_vma_to_mm(vma, current->mm); current->mm->map_count++; /* add the VMA to the tree */ - vma_iter_store(&vmi, vma); + vma_iter_store_new(&vmi, vma); /* we flush the region from the icache only when the first executable * mapping of it is made */ @@ -1356,7 +1356,7 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, setup_vma_to_mm(vma, mm); setup_vma_to_mm(new, mm); - vma_iter_store(vmi, new); + vma_iter_store_new(vmi, new); mm->map_count++; return 0; diff --git a/mm/vma.c b/mm/vma.c index 498507d8a262..f72b73f57451 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -320,7 +320,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, * us to insert it before dropping the locks * (it may either follow vma or precede it). */ - vma_iter_store(vmi, vp->insert); + vma_iter_store_new(vmi, vp->insert); mm->map_count++; } @@ -700,7 +700,7 @@ static int commit_merge(struct vma_merge_struct *vmg) vmg->__adjust_middle_start ? vmg->middle : NULL); vma_set_range(vma, vmg->start, vmg->end, vmg->pgoff); vmg_adjust_set_range(vmg); - vma_iter_store(vmg->vmi, vmg->target); + vma_iter_store_overwrite(vmg->vmi, vmg->target); vma_complete(&vp, vmg->vmi, vma->vm_mm); @@ -1707,7 +1707,7 @@ int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) return -ENOMEM; vma_start_write(vma); - vma_iter_store(&vmi, vma); + vma_iter_store_new(&vmi, vma); vma_link_file(vma); mm->map_count++; validate_mm(mm); @@ -2386,7 +2386,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap) /* Lock the VMA since it is modified after insertion into VMA tree */ vma_start_write(vma); - vma_iter_store(vmi, vma); + vma_iter_store_new(vmi, vma); map->mm->map_count++; vma_link_file(vma); @@ -2862,7 +2862,7 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) anon_vma_interval_tree_pre_update_vma(vma); vma->vm_end = address; /* Overwrite old entry in mtree. */ - vma_iter_store(&vmi, vma); + vma_iter_store_overwrite(&vmi, vma); anon_vma_interval_tree_post_update_vma(vma); perf_event_mmap(vma); @@ -2942,7 +2942,7 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) vma->vm_start = address; vma->vm_pgoff -= grow; /* Overwrite old entry in mtree. */ - vma_iter_store(&vmi, vma); + vma_iter_store_overwrite(&vmi, vma); anon_vma_interval_tree_post_update_vma(vma); perf_event_mmap(vma); diff --git a/mm/vma.h b/mm/vma.h index bffb56afce5f..55be77ff042f 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -413,9 +413,10 @@ static inline struct vm_area_struct *vma_iter_load(struct vma_iterator *vmi) } /* Store a VMA with preallocated memory */ -static inline void vma_iter_store(struct vma_iterator *vmi, - struct vm_area_struct *vma) +static inline void vma_iter_store_overwrite(struct vma_iterator *vmi, + struct vm_area_struct *vma) { + vma_assert_attached(vma); #if defined(CONFIG_DEBUG_VM_MAPLE_TREE) if (MAS_WARN_ON(&vmi->mas, vmi->mas.status != ma_start && @@ -438,7 +439,13 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); +} + +static inline void vma_iter_store_new(struct vma_iterator *vmi, + struct vm_area_struct *vma) +{ vma_mark_attached(vma); + vma_iter_store_overwrite(vmi, vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma.c b/tools/testing/vma/vma.c index c7ffa71841ca..11f761769b5b 100644 --- a/tools/testing/vma/vma.c +++ b/tools/testing/vma/vma.c @@ -74,10 +74,22 @@ static struct vm_area_struct *alloc_vma(struct mm_struct *mm, ret->vm_end = end; ret->vm_pgoff = pgoff; ret->__vm_flags = flags; + vma_assert_detached(ret); return ret; } +/* Helper function to allocate a VMA and link it to the tree. */ +static int attach_vma(struct mm_struct *mm, struct vm_area_struct *vma) +{ + int res; + + res = vma_link(mm, vma); + if (!res) + vma_assert_attached(vma); + return res; +} + /* Helper function to allocate a VMA and link it to the tree. */ static struct vm_area_struct *alloc_and_link_vma(struct mm_struct *mm, unsigned long start, @@ -90,7 +102,7 @@ static struct vm_area_struct *alloc_and_link_vma(struct mm_struct *mm, if (vma == NULL) return NULL; - if (vma_link(mm, vma)) { + if (attach_vma(mm, vma)) { vm_area_free(vma); return NULL; } @@ -108,6 +120,7 @@ static struct vm_area_struct *alloc_and_link_vma(struct mm_struct *mm, /* Helper function which provides a wrapper around a merge new VMA operation. */ static struct vm_area_struct *merge_new(struct vma_merge_struct *vmg) { + struct vm_area_struct *vma; /* * For convenience, get prev and next VMAs. Which the new VMA operation * requires. @@ -116,7 +129,11 @@ static struct vm_area_struct *merge_new(struct vma_merge_struct *vmg) vmg->prev = vma_prev(vmg->vmi); vma_iter_next_range(vmg->vmi); - return vma_merge_new_range(vmg); + vma = vma_merge_new_range(vmg); + if (vma) + vma_assert_attached(vma); + + return vma; } /* @@ -125,7 +142,12 @@ static struct vm_area_struct *merge_new(struct vma_merge_struct *vmg) */ static struct vm_area_struct *merge_existing(struct vma_merge_struct *vmg) { - return vma_merge_existing_range(vmg); + struct vm_area_struct *vma; + + vma = vma_merge_existing_range(vmg); + if (vma) + vma_assert_attached(vma); + return vma; } /* @@ -260,8 +282,8 @@ static bool test_simple_merge(void) .pgoff = 1, }; - ASSERT_FALSE(vma_link(&mm, vma_left)); - ASSERT_FALSE(vma_link(&mm, vma_right)); + ASSERT_FALSE(attach_vma(&mm, vma_left)); + ASSERT_FALSE(attach_vma(&mm, vma_right)); vma = merge_new(&vmg); ASSERT_NE(vma, NULL); @@ -285,7 +307,7 @@ static bool test_simple_modify(void) struct vm_area_struct *init_vma = alloc_vma(&mm, 0, 0x3000, 0, flags); VMA_ITERATOR(vmi, &mm, 0x1000); - ASSERT_FALSE(vma_link(&mm, init_vma)); + ASSERT_FALSE(attach_vma(&mm, init_vma)); /* * The flags will not be changed, the vma_modify_flags() function @@ -351,7 +373,7 @@ static bool test_simple_expand(void) .pgoff = 0, }; - ASSERT_FALSE(vma_link(&mm, vma)); + ASSERT_FALSE(attach_vma(&mm, vma)); ASSERT_FALSE(expand_existing(&vmg)); @@ -372,7 +394,7 @@ static bool test_simple_shrink(void) struct vm_area_struct *vma = alloc_vma(&mm, 0, 0x3000, 0, flags); VMA_ITERATOR(vmi, &mm, 0); - ASSERT_FALSE(vma_link(&mm, vma)); + ASSERT_FALSE(attach_vma(&mm, vma)); ASSERT_FALSE(vma_shrink(&vmi, vma, 0, 0x1000, 0)); @@ -1522,11 +1544,11 @@ static bool test_copy_vma(void) vma = alloc_and_link_vma(&mm, 0x3000, 0x5000, 3, flags); vma_new = copy_vma(&vma, 0, 0x2000, 0, &need_locks); - ASSERT_NE(vma_new, vma); ASSERT_EQ(vma_new->vm_start, 0); ASSERT_EQ(vma_new->vm_end, 0x2000); ASSERT_EQ(vma_new->vm_pgoff, 0); + vma_assert_attached(vma_new); cleanup_mm(&mm, &vmi); @@ -1535,6 +1557,7 @@ static bool test_copy_vma(void) vma = alloc_and_link_vma(&mm, 0, 0x2000, 0, flags); vma_next = alloc_and_link_vma(&mm, 0x6000, 0x8000, 6, flags); vma_new = copy_vma(&vma, 0x4000, 0x2000, 4, &need_locks); + vma_assert_attached(vma_new); ASSERT_EQ(vma_new, vma_next); @@ -1576,6 +1599,7 @@ static bool test_expand_only_mode(void) ASSERT_EQ(vma->vm_pgoff, 3); ASSERT_TRUE(vma_write_started(vma)); ASSERT_EQ(vma_iter_addr(&vmi), 0x3000); + vma_assert_attached(vma); cleanup_mm(&mm, &vmi); return true; diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index f93f7f74f97b..34277842156c 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -470,6 +470,16 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_assert_attached(struct vm_area_struct *vma) +{ + WARN_ON_ONCE(vma->detached); +} + +static inline void vma_assert_detached(struct vm_area_struct *vma) +{ + WARN_ON_ONCE(!vma->detached); +} + static inline void vma_assert_write_locked(struct vm_area_struct *); static inline void vma_mark_attached(struct vm_area_struct *vma) { From patchwork Thu Feb 13 22:46:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 133BFC021A0 for ; Thu, 13 Feb 2025 22:47:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3D46280009; Thu, 13 Feb 2025 17:47:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEE69280001; Thu, 13 Feb 2025 17:47:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1D1E280009; Thu, 13 Feb 2025 17:47:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9A9B4280001 for ; Thu, 13 Feb 2025 17:47:13 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 25CBE8164D for ; Thu, 13 Feb 2025 22:47:13 +0000 (UTC) X-FDA: 83116408746.19.CBA9E99 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf15.hostedemail.com (Postfix) with ESMTP id 3A6B1A0002 for ; Thu, 13 Feb 2025 22:47:11 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Yz8Awu83; spf=pass (imf15.hostedemail.com: domain of 3bnauZwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3bnauZwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486831; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hOJYwLimtWGBUdGEfM7iYC8bKir+nyDvGIJQLaOGSbI=; b=OsjGlJKtLaWiBrqVG1BlJqjyT8uIyjudbMIqqzWpTD3zwUiuioiH8UWnSGgdTDOm5mtXIS H8+SCbrQy7RBOs1TrQSa0ROgpQPwQPiJfBeZ7Nws6dxIXfYfvZO9C4vkh8Yn8RGwjiNE1P BmTVWhhzycp20tEYfHUUeaHb9nNfy8s= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Yz8Awu83; spf=pass (imf15.hostedemail.com: domain of 3bnauZwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3bnauZwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486831; a=rsa-sha256; cv=none; b=nmziGtq1bHd+JSEv0lrMvWkkLLSL6FVcCKtd84DLIS8FjNjvmDgL/0oB0PxzHPvtyja1Fr yE2TeBZMhdTTGap3eCNzUJnfsCc3bZzzQvSWn9ZVHDlokBkwPrlElFry+xIAzdkXwe52Qv O4fjy9RH91TiMZQhRjcWOuaHeV1FZWg= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2fa34df4995so4972790a91.0 for ; Thu, 13 Feb 2025 14:47:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486830; x=1740091630; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hOJYwLimtWGBUdGEfM7iYC8bKir+nyDvGIJQLaOGSbI=; b=Yz8Awu83thrQcMACo21Jww4dIR3SvGWnbk3qbqaBrxjxUFwF3TCsbyg8Z1T4Okabkc T34FmnhsSR05kg0ux9gs1mfeVECnwkqJQbnOPsXQabVSlTmZb4EMh7Rdbig+pTQdg3Tr gbYrTVBee7d39YD8b8FdFEXKMpIPF8fuSuuFh8q5AHxyE+6y4hKP3HaFM2JV5GUYzKp9 YSf41jhTWfkNegxLBZ6EjaG5sGnG1GUaHxDCSu0M+sczmZeXDNEQ+JJjevlCKcbp1vwp +XCslB9+kAsXK3+ynGVg+JnZDjUcCF+jDgu5VbXDgaXN58FNo6FtT01wgbxs96V2OPgm Czmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486830; x=1740091630; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hOJYwLimtWGBUdGEfM7iYC8bKir+nyDvGIJQLaOGSbI=; b=CZs/+m/RPkg9IIk0937vn3K4HlWAD83ydS5G2KNY/qarO5xz12r5c1RJLTCRHUTqsH c2YllK9b5wHA3FXMDXZWLjBFJP6j2H1pmhqsuAznzRKY1tNdQIwp9KLTBVLh4zAT/gFA 3kvWZYt3UtX+OHkwajfalH/vZCxHZS0tg53na0vkJypvirJEsBqJXZHfun6qus+OLYyT v9l5CFVq9G9PRxZN7HcOqqnbTNhtrFoQgNlQqVCf7/0t+iE0pBh4M+R/uegH/34vnJQ5 mf3gXxFFKi+ZdPWAy5Qs47imsy9SBmdNEPO+C0Yu/loe09tPr4dmo/8/u5xKhDPZ4YoS PW0A== X-Forwarded-Encrypted: i=1; AJvYcCX9GpOoBdSEV8n1Xm77oPTmCOqYi6bRYiyfzM6SWIhFZhhWllVqurNmMWc+q1bUwgM2aQpxQ9qVNw==@kvack.org X-Gm-Message-State: AOJu0YyvgKSeH+3vSgUdR0gxjMDBTn1jBt5wDjh28AbWnJtCyxfUmeom LogLK8VgMW/WfOW1QFBJfmUOS5g8aSufCnepYou14qQjI+ZUatRtaRwZza1hJHIcfij6w6TBJiC kPw== X-Google-Smtp-Source: AGHT+IEOMdpbXGqVMbthd+OaRfeUWGsGbBZNR4yGOFrRj8/MuVDw8EpNm8XS12LJXdTQ6lIEpIij678/S+8= X-Received: from pjur3.prod.google.com ([2002:a17:90a:d403:b0:2fa:27e2:a64d]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5245:b0:2ee:f687:6adb with SMTP id 98e67ed59e1d1-2fbf5bc1df4mr12403072a91.3.1739486830202; Thu, 13 Feb 2025 14:47:10 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:42 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-6-surenb@google.com> Subject: [PATCH v10 05/18] mm: mark vmas detached upon exit From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 3A6B1A0002 X-Stat-Signature: j7fxb4y1s69i71c63fsciwiurbxim8f4 X-Rspamd-Server: rspam03 X-HE-Tag: 1739486831-596630 X-HE-Meta: U2FsdGVkX1+tv27Pcu2jN3GnokOpuNiOslclyC6WR0Hro2tAQnxBPa6PYii165ENKBDQ1Dh12o9VqrUPiTa28fMgDbTYxT4FQs87LiH91jgB3ADsgbKf8/Zpa5ULq+WUptWgjqVOKGwSfx7MF/+D9SS99lJjolvqhzv5Eg4ebWD233XxfC8EP75g4aipTdFJfz+FEgSXKurzU4Y915q+h/wc9v/XBCg65MyHIDSjXN0EC4g2mlBnxA62zGWrRhjp9aZs8JXyNVfzxFmHBsTLgzrhbgSDoTgfW8sjfjLIle2lo9FfsAC+kEiAyY6emclaSWgf9jfEl3yX5NBC3G8cDKWIe01vYXyIkq+wEIvtmCE7Qpp4SQHPHcYxobyk0WojiNoGwNtgfw7fxg/RZz4weAv3kiqDZ9cgIYWqXrVAf01P2Ai+1/q1t3fFu+xltFSupMACDmX/Dg0+QbrpQGqIaNL8JT4TJ0Ya61tUqajV+ZdQeid84u9dptB5juWN8yAC/Guh6GwieMGw2IxZLYCebNWWuWcaMce1Wl2vJ1yTFRK0ZMU2KR75ZnxEetXMxe5Y5u7bSzbKRQPvTCJ5bVTrX7TLG4P51MCuBRAi4Eh1qqCvsNaW+cf6tJFS0hhbGy6FoRoDpyuuJBZicLfddQAFAOq0ETFZI7iQXTDpMwQRlYAefSoC7qqaBMlmX+c4X0S7SENRZffZfzew+ngKGPcKtl+vKHIY21PJYsY3b8XOIR3cFgwS/Ujr3ZCAEPf9yTiwBbAuChAfzVSh/yblQypME2ucjl9k5fpS3VpmiErFqzFVT0u0nEDqcfAGgmeEA4qKdUc9YJxILR81Bwgh3fg2abF7J334nJuSuIOJnw0cl6hrMSzBGyftGGNDifXoQ95eEQcsDlfnciP2gNtxdkhZoESZUvJXq3Y22TWn2PJ77VDJXqZyqqHMFwVU++SDIw3QYkShC21ZZwbC87uQlRg dcbbzZpJ XLPxGlgwJiQf2ZkYfl6LTaaPHQJDKR3F1mmp3ugE9L2LdDCoXzsTZKHBQtWSwsc9mjFVL/KefyERcHyMJTNRybhVtR0P/6AdUlagCLMJFaTLdZZTrLQgz6QgmF3Nyp22IF92dnBKhwSIBKMaFHebWR+HATf8o66ZzEUl+hxrPuLR/Gbvnhxctw2/emV1rC8IbHO9NicuoGqpkcUKQtPpVumIpHQHwnoHF3DlrUc+mKZ3Thzl1FlBOSp4uiiYMuJTpnKy+pjvKS9phzC+I2Nr3COAxrc/ftf4xMeeIUW0v10aL5jbzOWPHH6fB+cLXBUqcljUDJzBxAg6agXGOdjJD+V39Ze1wzU0zazL5rIeYaOmlonOcmKf18FgzR5iOQQjCE1hAmr/1j2wnUt0EZ6ttFCGFgRrVpH90Nx0E/wgay0ZSSlFdw+k8weTxx/sTHm4SuFlawfzkgM9tg7ECjQcfBF/TK348ZJBKDeWCrADrXxQxU2k4dWEp3XHHiHtoOcKGm0jxMpl+6oLmei0uY2aDOPGY6MFWk0FL7G55h8UvvGVKKAhCo6rWkheshuk58g9bD5oJRyRF/hCMRD3KjN3081fV88kikFaJ7tUf3Ok4rM7bK4qRR8BS3nPlvSsKhpry72fMsKnV0jVePWlWE9iEtKDxlKdVRtsdzonc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When exit_mmap() removes vmas belonging to an exiting task, it does not mark them as detached since they can't be reached by other tasks and they will be freed shortly. Once we introduce vma reuse, all vmas will have to be in detached state before they are freed to ensure vma when reused is in a consistent state. Add missing vma_mark_detached() before freeing the vma. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka Reviewed-by: Lorenzo Stoakes --- Changes since v9 [1]: - Add Reviewed-by, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-6-surenb@google.com/ mm/vma.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/vma.c b/mm/vma.c index f72b73f57451..a16a83d0253f 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -427,10 +427,12 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - if (unreachable) + if (unreachable) { + vma_mark_detached(vma); __vm_area_free(vma); - else + } else { vm_area_free(vma); + } } /* From patchwork Thu Feb 13 22:46:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD302C021A0 for ; Thu, 13 Feb 2025 22:47:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BF0128000A; Thu, 13 Feb 2025 17:47:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 846C2280001; Thu, 13 Feb 2025 17:47:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6774E28000A; Thu, 13 Feb 2025 17:47:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3B2E8280001 for ; Thu, 13 Feb 2025 17:47:15 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 007391617F5 for ; Thu, 13 Feb 2025 22:47:14 +0000 (UTC) X-FDA: 83116408830.14.C09A8A1 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf02.hostedemail.com (Postfix) with ESMTP id 3DBC78000E for ; Thu, 13 Feb 2025 22:47:13 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=onYj+ddt; spf=pass (imf02.hostedemail.com: domain of 3cHauZwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3cHauZwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486833; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f967rAL9Gs1nAaqEwqbs3/KxJ+f7SIc9ekPyYd69/e8=; b=M34wNHUAbtbPEqaiaHx5fF08y9d1WvI+GI+ZFOwsro2kH7MrFYzMfRRVpFqYKkRmGKLb4T sobVzG0ISlP2yF3qPmotD5QmLz3Ax8phjuwaObtZMN1pRfd5RfSmLb3fIn3HgNWpA+gsXC N71Tenao6eT3ZYWg5p/zuYxfpyL+YYs= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=onYj+ddt; spf=pass (imf02.hostedemail.com: domain of 3cHauZwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3cHauZwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486833; a=rsa-sha256; cv=none; b=Hm9fTHxvXckaNPDMi9yC7ETxZTMSBpDT6tplSscuoBSCEYzy9WcGFbKmcBPHbqb5pqiLmp yIOPXb1aT/XEeZB4cxqQY5lWVJgsiRNabJv4stoxo4Fz5nCXhg5V11H3Unrs6Az1eCBLzP IsuJgf8IT6KoL4gtWKaSXR36TGXzSyM= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-21f444af89fso20253075ad.1 for ; Thu, 13 Feb 2025 14:47:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486832; x=1740091632; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=f967rAL9Gs1nAaqEwqbs3/KxJ+f7SIc9ekPyYd69/e8=; b=onYj+ddt0CM0Ul/0wpDGvGaQg96UBM1t7eRRjFTb0BsQn+THYWoKdNhTjlARLnrb9K /9JVQqaGUbY4JQzSJqV9vNJ4Qb+LFTkiogJM4F7oGoutmmr8q0nWAkfG4Zi7W+8xbAnF 7Cp0u8rRF04MCK7w+sfWukiGhw627F6mSHkxTfqmpf6FNODsZg8vOmP9rwauDEa9Y7t0 4gAV57inNhp4MZmdD8fs9GLDm9M5z6/BA0fzNNI/uOYBR2NByVWsRPYwqRhQX2JjHCUC HTNbLhCYOBWelaP82MuWF2zka0A7Xt8d4Erp7s0ayPCTAvFRXciPV3TOzQLX7lOSnCSY uvJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486832; x=1740091632; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=f967rAL9Gs1nAaqEwqbs3/KxJ+f7SIc9ekPyYd69/e8=; b=sfjEoVViXOCr8hFK5lnlriWASjPefJZLtdZ8ZGXUvVNM55VmUF9MUqa2zkLTDJomG4 DEadPFrpUFkr2yQ561AQRRKtRRQyURydFS0P9QyYWKgqKMPlK/VaMRtA2OndsXrH2o4w sx2yJewgjsc/6g0Js4Bycfqkhhdc1K+Q1gGUgLrYS8v2tprYoUiEGiAmDUlkvciyv/Zp QjJWdYRHb6g4HLQ1m+1zklDbKOVOfaGbEa/do7Sf2wrK5qB+FNkIcmkOFTPDiYPZKjX9 ODfYb7qkQC5OIHvZRznOrMukdL4Pi6WBTk3Cz9H6AUsLhJu/4O57xPzxv/a+aUKEQIrE y/ng== X-Forwarded-Encrypted: i=1; AJvYcCUCI65YR+zkb8N8MkMWPUTI0FtkjnVKsYv6COb9mMxH+mRBrjJKYwsV+v5QOlOlZ3+VX+skJByW5Q==@kvack.org X-Gm-Message-State: AOJu0YxkQfkH7uSE8jrGC1dpMTXSglNfVHA2MIGs0fNvret7UQHl/Zr0 VBNNYV/TX/JfZ2tXgZo5YZmUQhF71FTKLUr3nhqjRikREZ944mzc3ykZ9T5alOwu18EXesnWNeB UPA== X-Google-Smtp-Source: AGHT+IGSWz7GU9dfI59mEnX3KWjK84PY7XxuIsdno4nvi5XjdZpP5t6KXvINYXrYmtHx9RdlsY/1nSR1Ez8= X-Received: from pgy35.prod.google.com ([2002:a63:1863:0:b0:ad5:5841:9f23]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:3a94:b0:1ee:6b2f:1a2a with SMTP id adf61e73a8af0-1ee6b2f1d52mr9549965637.15.1739486832181; Thu, 13 Feb 2025 14:47:12 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:43 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-7-surenb@google.com> Subject: [PATCH v10 06/18] types: move struct rcuwait into types.h From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 3DBC78000E X-Stat-Signature: 5rr7ysmqfeqf83r1trogxtdpqdszso9i X-HE-Tag: 1739486833-51842 X-HE-Meta: U2FsdGVkX1+9DY2/Nu+7XM/FR0zL+iifPnvcasjz6KTtvL+XVvpd4FjKACdg3xL23F6to128ibqbO929SNNbeb7KHq2Wt+GpGRowGI8D0Zr7ytSfjX9QbG0WU5Df6ZqPiF8PJsfNw3m+3Lm6vjRAOYyBQ4OJgJUQ2OQgxebtf3dYmlN19zLpz0acfkct38VRKCdnQzRx5B0ymH20KQomhJwKaE4NhdhlSFUMoH7t2TRl4ab7XCUn54dAyBsJDUYmpunmttrePrni8wgj5Olgvze1AxuOQlpj295/TL5UxgSrLSHqN/SrMFZimy9Bs44aSZtYv4XOsW49aPVFJaiypI5mLY9HSDfd+FY0/rNqOLsQr2xow4IjXViqjjL9JLyLQS6zBlUIyzH3/trEN5aY4qRGn0uDG+m4Vc7CU/LwWb00qDyxtz02LQf12/LZjDzPeH6pgf40D7OlEk5DIH+L4hRn9UpRg8LgyEbVhR3Gem/tFD56eO0QtOELz8NDD3mOu+7MNBlAC8Ku+PnVWIrEwft+ntHhcGnbcX097APdbGpAhE9Yvw3WS/kuGG8SBZ2YaFYFAW9YYoX+rYBYw8FzmxpkQolt26uOmZfZk7HmyCTYhr6UdaHTToQ2G9rg2GIx5SqGJNfCmdrTiAiHYqSW6Pd4FjcfUlxjoao6aGM7ZmfA6hDU+/VkgeBWt/c4u5zNjbxtUHYnQPcA1Z3gXua7K9kEkCsfYXxeHLZ+Yue6/3l4pM+RHhhce2iJ9UFX2qnhrtf1QF5hkT8IV4h/6uRLxa1xLNosrOAsQSJkNYNHq5TbkS+L/AtC7DiOu3nkN3EbYOEND26UjzaqvejftR6iFN7cQA48IbBaS7Xx3ZpYFZ84YwNhICaDCU7CR+VR8BRsu423O6ZrDwU5/hTKxW+qbmqTjWhSxJfaCkFih6abn6YsOiYqq1HwSqugxTF2NYC7+ENU4DLSt2Yu8eTasqM 7CYyaUaz 2sMks95DvGJ9Qn3iFleEmRwigvJiPFzxjAGJ6mlOtaxxkGHWIDvfioc0x0K8LR2GhYv4VFEG5vN8OLgyuNwX9MxYPulIwcmRudexgbuoGbaC0TWQ76a/uTnz2zxhUUNBtKnszqd/hVQVbP6fX83Lm6C54X9dYYxtB2bUNmvuYlN1gN9v5NFItWWxy9BwiiVSdymQXoRgK0tbCvSsnjb/421Fo9hsGzLFEdd/Y2h5ZHej7WCawUSLwZ7ZflHBBISIW3abPQdwDRN9e6PsE9TBZs7CCCr8Y1bgBtIFVPXpmdrTTv/yM8LqKsMHsdQ0DyiLO1MiFfSfsq/3Z15qixzW4kjXwZbOjRbUG9Y2n9/ZxB9l6IU+z7Z8ByQrbgvkFrkHPbFt/QHfJ+tZaIodukuyd+hJX15FIfujwjhb4YbPlUgNyd2yIhTFcKVYYL8rcHkA6Lqecn7akQa1Bn/odDJOFe/BqQ+Kbyh95Fl7JEHZIf9xh3gdrwLCnZy5YZkxbr1QosQq4lacjdD0f+BeXnyau2+Iq5CyRiyqAARWq/qvRjU+nYejlAxx/LwJ6z9LqzUpOc9reeP7WqKHKLXmZ0r3XSSXAX7UaPe1lC6DaRlhbtJRothW1F6hj+VuwC+sConnXOBvbTJ1IWSq+rQ3+e3UnYnEE1RAHFaPTdG8fZ7xCCrGZRjd8+wRL0aK2UcXDsldGGMieWQ+7cGfiyg/ME/W79725GRcVWz4WBGAKBvOlCblkoC7/US4QlatYsbmjm682YY3mvi3EDhulBklmpNN/xxOJ6w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000436, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move rcuwait struct definition into types.h so that rcuwait can be used without including rcuwait.h which includes other headers. Without this change mm_types.h can't use rcuwait due to a the following circular dependency: mm_types.h -> rcuwait.h -> signal.h -> mm_types.h Suggested-by: Matthew Wilcox Signed-off-by: Suren Baghdasaryan Acked-by: Davidlohr Bueso Acked-by: Liam R. Howlett Acked-by: Lorenzo Stoakes --- Changes since v9 [1]: - Add Acked-by, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-7-surenb@google.com/ include/linux/rcuwait.h | 13 +------------ include/linux/types.h | 12 ++++++++++++ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h index 27343424225c..9ad134a04b41 100644 --- a/include/linux/rcuwait.h +++ b/include/linux/rcuwait.h @@ -4,18 +4,7 @@ #include #include - -/* - * rcuwait provides a way of blocking and waking up a single - * task in an rcu-safe manner. - * - * The only time @task is non-nil is when a user is blocked (or - * checking if it needs to) on a condition, and reset as soon as we - * know that the condition has succeeded and are awoken. - */ -struct rcuwait { - struct task_struct __rcu *task; -}; +#include #define __RCUWAIT_INITIALIZER(name) \ { .task = NULL, } diff --git a/include/linux/types.h b/include/linux/types.h index 1c509ce8f7f6..a3d2182c2686 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -248,5 +248,17 @@ typedef void (*swap_func_t)(void *a, void *b, int size); typedef int (*cmp_r_func_t)(const void *a, const void *b, const void *priv); typedef int (*cmp_func_t)(const void *a, const void *b); +/* + * rcuwait provides a way of blocking and waking up a single + * task in an rcu-safe manner. + * + * The only time @task is non-nil is when a user is blocked (or + * checking if it needs to) on a condition, and reset as soon as we + * know that the condition has succeeded and are awoken. + */ +struct rcuwait { + struct task_struct __rcu *task; +}; + #endif /* __ASSEMBLY__ */ #endif /* _LINUX_TYPES_H */ From patchwork Thu Feb 13 22:46:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F2E9C021A6 for ; Thu, 13 Feb 2025 22:47:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 849AA28000B; Thu, 13 Feb 2025 17:47:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D129280001; Thu, 13 Feb 2025 17:47:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6255928000B; Thu, 13 Feb 2025 17:47:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3B159280001 for ; Thu, 13 Feb 2025 17:47:17 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E3BD51416A4 for ; Thu, 13 Feb 2025 22:47:16 +0000 (UTC) X-FDA: 83116408872.26.C845C70 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf23.hostedemail.com (Postfix) with ESMTP id 27D0D140009 for ; Thu, 13 Feb 2025 22:47:14 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=S8cY4MLR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3cnauZwYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3cnauZwYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486835; a=rsa-sha256; cv=none; b=WH7b0lyPc2//aIXSXDbNvAHprv8VgEwo7XeHMCVzFYSOPHb/mCW7GGjAR4BWe7uRQQ3/qm A2AffoYJqp6QwO3URJchD1Q77B5DSoUKdGdMA4kYJWhQlQjMQRQM4oYZLZSZZcCoaOGcyW CmXPwAkxkfvfzqnCKlTs0jrdBSySmYI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=S8cY4MLR; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3cnauZwYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3cnauZwYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486835; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8lnVYcB8HwzzUuW9qEx+Yoh+VM8+OMN/pADhYBeUDVo=; b=4DKz72uarhPfIbHLTaJ09yzfAH+9PXg3fF+mQk+MFBEIrgYolsSxJJ95RPk27PBwpViVOz B1qkDeiEqwMEvDj8kU3bgEqDCE3KM9O3CEpaHhFtuWpuyDwMIMMTDpL8h5q1+qhZFADU/5 8VXTRjtEEZL++0wJtm6pSZhZjMVmHM0= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-220cad2206eso28993525ad.0 for ; Thu, 13 Feb 2025 14:47:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486834; x=1740091634; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8lnVYcB8HwzzUuW9qEx+Yoh+VM8+OMN/pADhYBeUDVo=; b=S8cY4MLRjp2BtIpqbupJ2/PMIsNaz/xcYFP2b/xUisJnceQc73zYR3xRexHoOQhhNo p30X+3eaVe9zfowOnqDyr3bu0liK5Ah1FVgja1Fo7Gt6fhZooH2kzicL1RuSWUqYMZw6 LZb010RA2V1jM8TRxcFfikHowTC3kqBZGl12293gUF2CKePuVp6LU2yaXCUXrAr51lLF oz63aviXlyOnYEF0/W5dDtwaU4yt8IYEWrtFN6WBR28Le7BqIbrp/Jjp/Z68IORh0llx tutqqBaeVrnwZQgkEdcJMK9v8Hd+N6TNuWMgAsKLpqgsgBEM5DgdVZcn/J4qzAym64PL +c+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486834; x=1740091634; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8lnVYcB8HwzzUuW9qEx+Yoh+VM8+OMN/pADhYBeUDVo=; b=Zf/n7R2JUxy71VDwv6k/ELgr0fBkaLEzAOdZDxX42TtsWzbXWCICiolvPOoGHPcddc 1mkR3pE9sdPVmNvWqbQNgkDoHrJsbX/myAfAEoVqAMkCzGOk4JWFNspCFEdoDAKsMdbs TCfU/y/OjOEOYyW4cAuNrsVbz+Lxf3Llvd+E4DbH+eory9UfW1PX343qlKaxsu+rG/ev mzDBWIF4AmJ+Xo9S9dbhK26PwOs22RA3tszKaBc5FrLjjp1exVRQYjwW9OwKay0wxbr0 vx8vmQdYI9qj2nqbOMmAgoy/jgzo9oh+vL6DsqK4Qxk45lOgsy5DwuUN2TNruPaFqN3c sa0Q== X-Forwarded-Encrypted: i=1; AJvYcCXYLTuq2gIGc8ZlNitJX/+27WX2AZnyr2SXsoRmeKYHHDi2TuoS3n5GdzQodR8Ab+OSxc/loqLxug==@kvack.org X-Gm-Message-State: AOJu0YzSOflwY/RSz1XmPYNruBdJ68S9tAN3zX1PdakR4VqF00p+2nMz l3Sg8WeW7f4XdUZN3pIJjVUuJLn+SwbXQo8o1CAIfSXNKkS/rw0oqbkp/gc4M9c99uY0tQL7zLp wtg== X-Google-Smtp-Source: AGHT+IFvP7Rtv2Mvdra3Yg1NiqesqrI7PbBRDKrXPHHWmHQO1TyYtp4d3ffasZylr6rr8GVtjs5kwYbmVJQ= X-Received: from pfbgd3.prod.google.com ([2002:a05:6a00:8303:b0:730:8c7f:979]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:c997:b0:1ee:3b53:b77f with SMTP id adf61e73a8af0-1ee6b4013abmr9718076637.37.1739486834016; Thu, 13 Feb 2025 14:47:14 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:44 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-8-surenb@google.com> Subject: [PATCH v10 07/18] mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 27D0D140009 X-Rspamd-Server: rspam12 X-Stat-Signature: af66qbw1wm91gc359a3smazuzb7efs6u X-HE-Tag: 1739486834-581325 X-HE-Meta: U2FsdGVkX19TULW82tLpLpv5SJTOy8HjnmgEU5m8mlzKT0sdDQfAPhmHNiFYKiozb8rOZuZHeKtLPCCba7mqn2kyP6+8OgGHCVh+Q794M1jBhMajA4TlZmF16RCtlYeUwuPt3GI48nYZzMtu3DTny9VZPANPY/T52AnMqP/v3j64152w+ao7YpmNR/EEVbNIAg95pk1x0uSugROQzPb9ykFnWt8pzISn3fX3IWtaXzHP8L3GeqdKKFUfb1dwHhst8zyxW7T5G2Ns6u2lPIGKwN+I6pO/Fop8b1vAK5SuZSncdAl5CPGtKAsdAp27rZWbbXbncge6PnqVfkNDhDbUlfZ11PnansmvAMHK7UJ0AtuBVTcAlzTMM51007a0XZ6woefqNhoAawxP0h6V05wbkUj/JxR77LjSp0M5kecfqJs90aE78wh/aLeuBbDc9Ah1+kG/37fYpuZ8X014/Q/2YHHrmLdGjXoG/bd5OynnD9Hozjijv4unM26bL2qpt6xDa0J8v6K9v/tb2PsnkttsCgmsuCR0CfvB4hF5B3kSACgC0yP8ic7bXWw1avUVJZaklWFd++OOjeGPDlJVaPpZu6D8e2MqMaqk0xS5jmxw2kOweHGNAaWtYKVLEJSuswAmJnvjMsqbY4XyfBJFJmeX6i8UHaHid6TW5STDHKdfa/AucoSDq1T/vBH3ZX2M44ZRrHyot8AmQIZfnF8rf24fk7IQfArYiGs9FlN8P+BBN02KKEQytm1gTYoQgjMzDcOU6Kd09sGe6+r/9imKlZnB9aI5vzVijRnOYBDpsKlx8rSxyo0iBPjJaTZECdmO60XHh1psW9wyI1Sfty3AWzZbfHOnq+vmQ5scQiYQMb+zpHgZI4afVngeUFCCz9Jpu3nsOLIf+N/UODyONpNt7Fy+VId1QgrXon7XppNapGf3oh0cO9EFUlY54JgIujwkq8vxNjSSrSr4m6Y/qu7gZ2t TnmenAVu CkfCV0FmoecCrTiAtKlnnDHpVx+bcdufyBbzUBgswnXUpY04BDHXeZUkqkK3/h3GXyBfurMR6LWIIfPcGJgfkffQP/0PCBVM9RJSBH8jjy7kjRRM4985QgldgPQN7XKPPt1gAyDswelQRLUK7Z5xVvu1SWFY94ywD+SJAlj9BqGJJv2aaQjVAMgCKaaFkOOqPCNcBwpSa0yI15tFMa1tU7YaPOMKk4aTQL0ZyD5arhapv3ar0DDGKE45A3OoGC/07hvOMjPb/3O8T49d28l1W7saH1n6QO0hE1v5h+iei252MPdLhiQb73VOPVpWeS7sPzdNvTT6/tLnV5ndoGssum0/jNj2jqE1NztJ/2o3eA8ivyUQC712woA3wJzwNi1A+7VZdlXQ89C9JoG/xKweddWaYfsMDjaFJ2F124fQgctqCYizQQ39VI5hy8ZbGjcq2OTBoj5cbBa0M4yo4DNq6f+RT1e4+OJ4dDA177qU2GFv55dK1NCNebFXuOU26ikkOugVpcHRMYKmurekj1kznl/BKnQqMxRmj5PlgP/59N5ag1C7yCxYGGY8rnaYbuxwZBapADltjiUIXOtgtTbCoD2rtTjSKKoi+UfwtdlYHeVIuFG5lg97mV7URDpR6hutfXq9ppR5liuAXxRSD4ai/1v6avw4vju8rniQSz9x+ChesUmfWWbovkiNDNA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With upcoming replacement of vm_lock with vm_refcnt, we need to handle a possibility of vma_start_read_locked/vma_start_read_locked_nested failing due to refcount overflow. Prepare for such possibility by changing these APIs and adjusting their users. Signed-off-by: Suren Baghdasaryan Cc: Lokesh Gidra --- Changes since v9 [1]: - Refactor the code, per Lorenzo Stoakes - Remove Vlastimil's Acked-by since code is changed [1] https://lore.kernel.org/all/20250111042604.3230628-8-surenb@google.com/ include/linux/mm.h | 6 ++++-- mm/userfaultfd.c | 30 +++++++++++++++++++++++------- 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1b8e72888124..7fa7c39162fd 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -747,10 +747,11 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) +static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { mmap_assert_locked(vma->vm_mm); down_read_nested(&vma->vm_lock.lock, subclass); + return true; } /* @@ -759,10 +760,11 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int * not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked(struct vm_area_struct *vma) +static inline bool vma_start_read_locked(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); down_read(&vma->vm_lock.lock); + return true; } static inline void vma_end_read(struct vm_area_struct *vma) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 4527c385935b..867898c4e30b 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -84,8 +84,12 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, mmap_read_lock(mm); vma = find_vma_and_prepare_anon(mm, address); - if (!IS_ERR(vma)) - vma_start_read_locked(vma); + if (!IS_ERR(vma)) { + bool locked = vma_start_read_locked(vma); + + if (!locked) + vma = ERR_PTR(-EAGAIN); + } mmap_read_unlock(mm); return vma; @@ -1482,12 +1486,24 @@ static int uffd_move_lock(struct mm_struct *mm, mmap_read_lock(mm); err = find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap); - if (!err) { - vma_start_read_locked(*dst_vmap); - if (*dst_vmap != *src_vmap) - vma_start_read_locked_nested(*src_vmap, - SINGLE_DEPTH_NESTING); + if (err) + goto out; + + if (!vma_start_read_locked(*dst_vmap)) { + err = -EAGAIN; + goto out; } + + /* Nothing further to do if both vmas are locked. */ + if (*dst_vmap == *src_vmap) + goto out; + + if (!vma_start_read_locked_nested(*src_vmap, SINGLE_DEPTH_NESTING)) { + /* Undo dst_vmap locking if src_vmap failed to lock */ + vma_end_read(*dst_vmap); + err = -EAGAIN; + } +out: mmap_read_unlock(mm); return err; } From patchwork Thu Feb 13 22:46:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5CA0C021A0 for ; Thu, 13 Feb 2025 22:47:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9336D28000C; Thu, 13 Feb 2025 17:47:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BC62280001; Thu, 13 Feb 2025 17:47:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62A8128000C; Thu, 13 Feb 2025 17:47:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 37E54280001 for ; Thu, 13 Feb 2025 17:47:19 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EC80EC174E for ; Thu, 13 Feb 2025 22:47:18 +0000 (UTC) X-FDA: 83116408956.20.99A980D Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf02.hostedemail.com (Postfix) with ESMTP id 2E6D680004 for ; Thu, 13 Feb 2025 22:47:16 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fgeUSCW4; spf=pass (imf02.hostedemail.com: domain of 3dHauZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3dHauZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486837; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PeBZHGW/4v8XWLim1qpAbGeBbAKuN/isMOjrQKZ6x9o=; b=oOA6D3M05c0jlZprr7v7vPeswbnZC2dVZ7jv8z9oz/Iu468CT1AAFW0aYLP6pkK+bwf7NG Nzcav9Ld5vWf2CLWDKD8+Og82T5QSrjkRj6oTYcizwWYmsF1mpNtT7CTMuZKUMjTmaUOC5 ATKuE19GvisTkfODMpMkvOFpgNrWK64= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fgeUSCW4; spf=pass (imf02.hostedemail.com: domain of 3dHauZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3dHauZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486837; a=rsa-sha256; cv=none; b=sx8TmYbwq3pRfiwZTrKRk6uxrV0L/+l7wV44DnCGq8t2A/Fw+bIiz7aZwIrWPU0opFJN8Q Is67ojmAhYyiLy9T5x9w92QPy6QTqsxrViY/qkyasGFV09gaaL8QfLC68DF59hfc8z67Tq snpwhmLv4o6iBFmUljzzwNWyQdIMEcY= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2fbfa786a1aso4528912a91.3 for ; Thu, 13 Feb 2025 14:47:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486836; x=1740091636; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PeBZHGW/4v8XWLim1qpAbGeBbAKuN/isMOjrQKZ6x9o=; b=fgeUSCW4XcFMPJpujnOD32xT1/RBhTSQ1Qq44AwdReAYLnMkZEniHQZX0vkvsZooqJ p5W5lR+6ac9ZNEmSQ6sxp3Ij59J8zX5VabBdMdC1bph094K3Os35OHO4yFYooEugCfcS z0AiDL6PbAKsKySQ/H9OYQtHA2m19xvqKjMhTspIOwPYCdFjIou5sCmTtTMb4FXWR/Xm 87lPLWFY0DbH6sDHI6y+tGbXjRg1FjC3TCfO0twBBHwVc3CTQRELlpgZgxASs9SGWYdB 5tsv9vjFv/gtTc7dIOeI3V8b77Rc3OAt5gB6fhgCbfgEdOexIEkRQHN/zH4/T4vVJIcd hzew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486836; x=1740091636; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PeBZHGW/4v8XWLim1qpAbGeBbAKuN/isMOjrQKZ6x9o=; b=RuaDTgBHkyYvIprbVhZV+lR1M+eYyO3GacSCeOlc4DXxt6zrNi7+bmdsNqg/XyoRFk 6l7qnvRZ/CbZPR5n1bmJnXo1m0k7L8u7q+PdHq9eQoBzIaNUim5I58ZjptOlMAyjZMdu vW9FeJ4NssIYgHbFDrw+7L6Voba7AFojzCV2Myl9dWZ+/IHxCKCUEbluCuXtRRYaij3n drBUCJR83HD/UG4qmZ8VX05g7meyoKrG6JkFZnBEVAmegwJsdT25aEYXw+TvFaCFcAKE l1mxWiZDjD1AR9jfezi88SeELSh6tlW4utOLfIfGOyARAQX/T4GukEYT/jv7gNUiStGM q6pA== X-Forwarded-Encrypted: i=1; AJvYcCVJReKuedawIFqc3Gpq+g1SUq6AWjNml3gJ7xKVHXxnFXQLcl/5IspJjRE+rHxIfQqXOk5ER0dw9A==@kvack.org X-Gm-Message-State: AOJu0YxJGDtCUEvGZ0NQICm7+4yZSGXeMKGdrqKJAV1V+qzNEglCTh9o nAV1UdCnau/n/5PhKQHCYXixLbXrdE8QiqIfhEZuh0qB/TP212jUGkKTSnsrQxg5xqFYJsMyaTO eAw== X-Google-Smtp-Source: AGHT+IHfEnz0nPhbgU0f7q7Auc5HNsVT5eJa8wXZ0ZtBNf3PSVTzFb4/Oj/O6W2xc+itT5Ikm9K7oZ7OyHo= X-Received: from pgvm22.prod.google.com ([2002:a65:62d6:0:b0:ada:4ec0:a7cd]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:512:b0:1e0:ca33:8ccf with SMTP id adf61e73a8af0-1ee6b416346mr10537954637.34.1739486836048; Thu, 13 Feb 2025 14:47:16 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:45 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-9-surenb@google.com> Subject: [PATCH v10 08/18] mm: move mmap_init_lock() out of the header file From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2E6D680004 X-Stat-Signature: kctngsbpzbpujcikez6mw334atm9jy7b X-HE-Tag: 1739486836-795010 X-HE-Meta: U2FsdGVkX1/S/DjlzJhe4bFeAOOlzCXizLlRNfq+DSFbe+ObZN7ufxjGJ3BRX2FNBTn4WVKQlVgU3Dk6E+L+Pgm0AY81lB/1pcgk3708T7rF5xQyAlUJfHgWPSkQZjhecunGxQMpgQBKSsFMwAcNLkNKRPVw9225L0r84h+WZSYSW580Dgc2OdxAp3E6+P6CKszuo3jaTGTyOHBC2xjO7QyL01TVO48yRQLD9L2YkCRsCf5EIxbhU7SsuleeC+V+MGl29fXAaAMqpO1mltR4gwL4Q11n4l3mMLFgWDvMi89T9w1U4njd+0I/nae9f32bWOV8UkLqQRFUo7OQkr/DP3N/4mt+7GewHa1RzI0mvoUNZ1ydNhKfioc5JZGKsGCKemqo4g2ohqPNE3L20f9o1C3cu/DbIdSyioPisHTbcyp/Fx167OUK4XCw1d5iC0VLsOfnA/yJ4D+qVKOyaTxXPRvWzmK9kkgyn5n8rubfdSM/viG33JqyvvRNeWG5eQvZx0Xq5r/IqMUdH9r7NYZpFWgcr2P/vIG7RB0Jwu6/BgP/cLYliyAvo7xbfBZLQXZY7Bc0StYtFPGe3YINCaMmLhEn1axn3pgnh8dAnfECH0cdDVnb/0h2x/x1u5MEhod4QuViAUj2aIeTk4GxiyMwGPoHe3xVy2U5xdQxGUuyFNp4Zv6t6NbNVX4Vx0d/wN6i1kE5W6a0UEYGDPvLrSdHjG7otNfi6oUNvZgnRFAdhewmoOsts+cO0ELzihP6BK8EZ7vzy8s2hJ2xxu+U6gWjlIy7iTuQWQruiWNt6a6pZv/EHIqZS1cxTLmJmwMeYQNOS0u1plvVPnTojk/eyfbJpBZxN6o57GyzXSJGwWLz7zGCndQuw4tpmMVG+Q6OA2sUpV2EngUBBHOZBUQyXlpzOlhkgs9tK4B2C5kO/BhKOxp293X/hg3kX+4fJO8Ahu0jiAP4k5+fibIFynUx0qH amW0BnHn 61uAQcVxd1ZF2cVeYq4FPpfp6Kv39Y20Ho3pkQ5YocSuh0KU2hk6MZFkvOOKIPwtxniG56lvGE3pJq7FNGZXhKpWced0HC+MDhreZKR+Nmk6B5nvfyKT1Ccdvwe3RHUTJzAXfu4jzWEuOF4uNduIkXszKBqP8f74qlbPi6tkeDaliHTELOMOvZV8qyRHoxgkNmtH1LkJ6SKIzbJbKkAOmLyIpllHm8tMH64pIFFTxSDzCKn/0+791V3pS7dN52HX838stDi7RVx5Bs6VmENVk/n1JQvLlhBnkJYRu+81ZeUxeg9IpjGQVDZAHFzo2//pCVbzbBlvwEYZZLgGmT6vluyhPrpeRlx3hQBBSmnQgS7HibQo6bVkZEwtrnhFBeYbu6lHpFRB4hpIKRiP/eK0mzLLR5k+WsTJZ0NHZf3CHTKB6nv962aR2je2+jDOIjxDYo4nQXS4VVqghyoLcbwFzNy6WVIzMy5Zo3Q4HcfjlehjELks/N9JoLwVdkq6fBJb1ve8F+BNQ+B8R7aBwdj1vpIEiC9qfy7uyrHRhbeLHr1UfPEp227xW+d87O3ShVaN5LI6wtnVX/MsyrpUf2DWqrD7sCLuFs6SNMX2sZWSNDdIyyNxMuigyxFmcAlSStxdcJue0mCkHvLCk09XsXIbn777d8mNupWU4Ch4ncjig/pCgTFzZ7zd8tZpwQNSJ/flpC1fRZ47f8aTOUvY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: mmap_init_lock() is used only from mm_init() in fork.c, therefore it does not have to reside in the header file. This move lets us avoid including additional headers in mmap_lock.h later, when mmap_init_lock() needs to initialize rcuwait object. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka Reviewed-by: Lorenzo Stoakes --- Changes since v9 [1]: - Drop inline for mmap_init_lock(), per Lorenzo Stoakes - Add Reviewed-by, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-9-surenb@google.com/ include/linux/mmap_lock.h | 6 ------ kernel/fork.c | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 45a21faa3ff6..4706c6769902 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -122,12 +122,6 @@ static inline bool mmap_lock_speculate_retry(struct mm_struct *mm, unsigned int #endif /* CONFIG_PER_VMA_LOCK */ -static inline void mmap_init_lock(struct mm_struct *mm) -{ - init_rwsem(&mm->mmap_lock); - mm_lock_seqcount_init(mm); -} - static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); diff --git a/kernel/fork.c b/kernel/fork.c index 5bf3e407c795..f1af413e5aa4 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1230,6 +1230,12 @@ static void mm_init_uprobes_state(struct mm_struct *mm) #endif } +static void mmap_init_lock(struct mm_struct *mm) +{ + init_rwsem(&mm->mmap_lock); + mm_lock_seqcount_init(mm); +} + static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, struct user_namespace *user_ns) { From patchwork Thu Feb 13 22:46:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B788AC021A0 for ; Thu, 13 Feb 2025 22:47:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90C5228000D; Thu, 13 Feb 2025 17:47:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 89560280001; Thu, 13 Feb 2025 17:47:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 699B628000D; Thu, 13 Feb 2025 17:47:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 43546280001 for ; Thu, 13 Feb 2025 17:47:21 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E9B2116180E for ; Thu, 13 Feb 2025 22:47:20 +0000 (UTC) X-FDA: 83116409040.06.FAEAC20 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf12.hostedemail.com (Postfix) with ESMTP id 1877240009 for ; Thu, 13 Feb 2025 22:47:18 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=wT1FZos9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3dnauZwYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3dnauZwYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486839; a=rsa-sha256; cv=none; b=qMnSVwHqZQtn7ZGqMnN8nUSISbVM3mxP0dcJFHyi1AHI9V+C4XoXryhuu14Lf7VRX54LT+ HkkL2SehCLiEGjggNm+cqfGuOi0u3u0lFOa6nNOgcnNt06tCpcpyos6cXcj6tDCacOnlx4 6KO9uJT18bbrYHzaHuPcyk9XBZm/kO4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=wT1FZos9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3dnauZwYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3dnauZwYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486839; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YNoCxF31vK1w52ANaPw+siaL/yNI9CD2wHk/1nIOSYo=; b=IvR6lf1XO8jEkyA1bla+QyjKpJQWeL8XXzJIHnteeBB825+uZ13ktEQlVJVsv8qDZYXJ3X IHPNRx+Udi6ZJg3MpkOT7gnjB//W7cR4223lyYAe0iWcTq3U7TwE/8zfVlwfvfNzB8zoLZ 8OLoIFvLKrlM8qvpywjbLvUty6hgGlM= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2fc17c3eeb5so2141682a91.1 for ; Thu, 13 Feb 2025 14:47:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486838; x=1740091638; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YNoCxF31vK1w52ANaPw+siaL/yNI9CD2wHk/1nIOSYo=; b=wT1FZos9dXWeNj7TPBp7GSXcgOiuREwGgajP3D7KYPFxyNZU263OvoiHAgx3V9jX7D ibS5bciseDddmlyq9siBI39mukNS7f53lHnW7NeS4HIR4lS7JCITjjXzVF2m+Pgzin7E /ZFYP+VAyJqg2QgNbsRPenDisWRyy1TSCraXN1Uh+N8JPY3gS3A9MBTg++9FHjoeTiVx cwJXXbzA1GPUCejen5rIEhpEc2/um4O1Myvd/8Fc+1MLyypoI9KE+94MU4jvRuuTbQTG fGZDK1dBzeAXcro57LRRYC8fsWXzD343WYlryicLgpvbKsgprWTe9um6xYHbAvsxe562 sUBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486838; x=1740091638; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YNoCxF31vK1w52ANaPw+siaL/yNI9CD2wHk/1nIOSYo=; b=mIhdLWV/kM7acPoiyJMSIywNlzyDM3ZpvYkjHUKMmv43uf99vmcOQ37Z/7CoYz3yD8 axYs0EZmq8+CbhIy43iR2Mj2+opvAHKFzbK6vLQ8V9u45QOdbAb6pJC0bQwqCakldE4C AC4bTDrwzIJ4KM1aD3u4DasQc8DTHD7tVwNVNe0VWvmVBNnQCXXRT/Kb6tuiJtUm2FVd N3jZBqjE9ljzLTyy5Xl8TBAL9qEbhAIRymDOqVHMyjaOueUWunht9rESFlkfGWc0UOpA F76lVtSFrx0wl/dfj1jG01AysLYf09noKyAIQpMwRmeIkmNek5dboXbo4vDReI6V4Gtk lbsA== X-Forwarded-Encrypted: i=1; AJvYcCX/0lxwbvtaGFFPQuyHuZBgWtxGnBS53nG675FfrBvVAmuQZJ8xGXPetBVnoYIgUaKdmK/5rpZ8HA==@kvack.org X-Gm-Message-State: AOJu0YxDsImroiGztpEQrQ20VVecsX3BdgV5kQEQDJc9dBJU51DR2TZQ dkKybFlD/RI8q3Praid1Go9zWUhKe8JlwCIZ0frJRrPCAnSCc6fqi0Y1kFp/DTeHYTEZ67c089v L+w== X-Google-Smtp-Source: AGHT+IErQJcr8UIFZH8RFb6dXrkV59J3DVPaTleZ6Gt1R/Rh5Le0xTpOCinKGJEwYZoy9GO+2bPedgwmiK8= X-Received: from pjm4.prod.google.com ([2002:a17:90b:2fc4:b0:2ea:5613:4d5d]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:54c4:b0:2ee:ab29:1a65 with SMTP id 98e67ed59e1d1-2fbf5bc07e4mr13915594a91.4.1739486838053; Thu, 13 Feb 2025 14:47:18 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:46 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-10-surenb@google.com> Subject: [PATCH v10 09/18] mm: uninline the main body of vma_start_write() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1877240009 X-Stat-Signature: y48nerxrgpw3gkuxoy8cd81wf78g6mwu X-Rspam-User: X-HE-Tag: 1739486838-157597 X-HE-Meta: U2FsdGVkX19D6DuvElYbygNSIwxtITKPXUsX5YE2/H1BrM02z0cw9vcZLpdZMGB3seT2NSlB3g7UMtyq/wiVHICZWgPEhr1bRrd5I18gJy6ifQWGBXAXZS551bePysXhaJb+CKdUmOTv3yTIOdpee6HZfJlRlYkmssJoRKdcs+eNKSCz+qxOZCT5qfetDZTpM7Uaep/c2LaZVfmkgDt4RtsKc5ArxKKLcPHtboeOh1qhMrP/MTTDQhJgMOu24g89hUtUwZuXqoY2AglnuluUaHC21bY5EffuJVado9fE/KC93VniSAJHRlvHiFUDmox+trbGnCJ0g5ScHmqp/TmISkIhQt9I3EeuvtRqPwTyuK2iFGLapH3tkfwX8m5KuRZ9Nl+KI8r1NPje7ZSEQIzc4/65q58yOx55jqxVKM5nOcApir9g3aO/z9MN3piN4OUiaBniARXrO0U0VsRvyn/jGvc6fY/tHD0lAJPJtVsuFQSlWf5XcXJO9P9cdIjkRlmW/lRsNkRVTwSl9ApXsFT6PxAnKojW16ZmcJ2HEfEdixES7KzT9GiFX+QB7O8yPz5mqtgiIWWwCMTzCuWoxWx+uiir+w9dsYMqtpSxh12EG/01ampRcBsL4TC1bKJsA6w4sMvjNX4UaIeUCiWICGxYsNedPYJcOgmgP7x6xPuFXze/m7iY45MkfsPzFjnZ65Lrzk2Yntlata2q8QKhpGyk5WCCKpPcErCReeCeSNskPlQ0YQSA6BJPDDqncwcO2ROfNLnI7SvNd0To+DEZcRaUSyAl2rssWZX93HNwOZIOye+gSnFqKrbS7lYi5EI7H6zGXZRvNHbtIvCOV79eDsjcQnnsIR5efj0rQIpq5lq+oGEmJb7QwDlrbOailm6yk9UgGGWAZuYdwnvPiAfth9pJ6eC5rUCNhXIVSC1J6D9xcvF8+9wx6qZqJ+LWC7A1cC6AVEVeeVMlrlobHjI4zXg vOmyrNuT 0t9E4+leNjwYQGjKb2ykNQeK3AkD13G6ZxvHUBgQFx+3rcx1pbt1hFjuedBT3t2FcMyiXxXxAnWmaOZ1mGIvw03jb9ZWHjruq/M5NAL5NyWZHtfBheaiShPM8tw5wxCqnRVaVH+RhmT7QNo0LB14u+/TdYuhbzBpvgNW4ICkGLYHABGAQHmb2rPuIkNTTJwpk+SaSbTvT+6T1jvdcaKMrD/a+d3XdvHkt0hgYZs/eCwbDRGCGfwfBO9qeyV+PVAamtSipH+1ymK90wFLkJfcTXzy/2D/SXsbIP/YB7iMfaBGEZVQAF6Exe86TJdh3YmVmnzK7T71g/cw2d5P4AiCm+Cu1mkWPfAUF+zo8/7DrGRZz1QcaxpKv86yKh1vCfspfD3hFOjfJkF7rVYAGMjwDEFYbO5o+YdJMQGFFcB9pMx9EbrfLT+SM9YC5Iroy42RARH+LKARPTp4y+urK18EE6exSxd6ej/jK2OnVWKo39RWJbloRtfKpyN0vpMDu+prWZWuvwoNS9CVBgmhvDcEtqQqoMHjxPIzdW8t/WALPgkkMxw0jc2Dv+726A6cXguIMYBaOtxepJFoJNjRWKUPQ/7CnJqPJRfkJfJPHjpVPlBUrNqA1yT5/R+yJondi5k/F7baEqMoutbT55pyyw5bjO2fJvW7/Zc9ySCLClakB25+z2h0tDV9CbjLZH1Ef9TZJZ5WETJurguYmbhI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vma_start_write() is used in many places and will grow in size very soon. It is not used in performance critical paths and uninlining it should limit the future code size growth. No functional changes. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka Reviewed-by: Lorenzo Stoakes --- Changes since v9 [1]: - Add Reviewed-by, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-10-surenb@google.com/ include/linux/mm.h | 12 +++--------- mm/memory.c | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7fa7c39162fd..557d66e187ff 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -787,6 +787,8 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, unsigned int *mm_l return (vma->vm_lock_seq == *mm_lock_seq); } +void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq); + /* * Begin writing to a VMA. * Exclude concurrent readers under the per-VMA lock until the currently @@ -799,15 +801,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (__is_vma_write_locked(vma, &mm_lock_seq)) return; - down_write(&vma->vm_lock.lock); - /* - * We should use WRITE_ONCE() here because we can have concurrent reads - * from the early lockless pessimistic check in vma_start_read(). - * We don't really care about the correctness of that early check, but - * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. - */ - WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock.lock); + __vma_start_write(vma, mm_lock_seq); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) diff --git a/mm/memory.c b/mm/memory.c index e600a5ff3c7a..3d9c5141193f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6393,6 +6393,20 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, #endif #ifdef CONFIG_PER_VMA_LOCK +void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) +{ + down_write(&vma->vm_lock.lock); + /* + * We should use WRITE_ONCE() here because we can have concurrent reads + * from the early lockless pessimistic check in vma_start_read(). + * We don't really care about the correctness of that early check, but + * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. + */ + WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); + up_write(&vma->vm_lock.lock); +} +EXPORT_SYMBOL_GPL(__vma_start_write); + /* * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be * stable and not isolated. If the VMA is not found or is being modified the From patchwork Thu Feb 13 22:46:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C911EC021A4 for ; Thu, 13 Feb 2025 22:47:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0ACA28000E; Thu, 13 Feb 2025 17:47:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A911E280001; Thu, 13 Feb 2025 17:47:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86EE228000E; Thu, 13 Feb 2025 17:47:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5C39E280001 for ; Thu, 13 Feb 2025 17:47:23 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 167381416D8 for ; Thu, 13 Feb 2025 22:47:23 +0000 (UTC) X-FDA: 83116409166.27.C45101A Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 3EC062000F for ; Thu, 13 Feb 2025 22:47:21 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DDnPziv8; spf=pass (imf13.hostedemail.com: domain of 3eHauZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3eHauZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486841; a=rsa-sha256; cv=none; b=2HJDUH/sJky9UYIEuH+THlabhAX3CnGzOtzGu7yQNMWoTXIKA3NNRAoMB5yMntxt67Tk/t QAFbwf6j3/wbNgkFn3+Nh1ue/r8uODDt6MGIvWEo9N/JOxgDxIxi8wmj4XhPLuDKYuKkKt JqKaVw4cPqtwsGcn3h0jj+T3R7mF7jQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DDnPziv8; spf=pass (imf13.hostedemail.com: domain of 3eHauZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3eHauZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486841; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3RwEBR7WcDA6C9/htn1wwdAW8BOhyJsqS310jozD7lM=; b=jtoRJKWkPxQPgx5UZ8CpqXIi0xo/665S0MBcFAUXNEDT8P0v8GGcaHgC1T2Yq4K19eXmmb pWWjCYrEX87zn/Ah3xgajbr2Mia0agA2o4WYmoytPA+9UbphpFWs8mxpTbrZtmNavALCLe N7b3uwGtajdA8UrHucWhRC9iLvyd2m4= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-220cd43c75aso55463785ad.3 for ; Thu, 13 Feb 2025 14:47:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486840; x=1740091640; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3RwEBR7WcDA6C9/htn1wwdAW8BOhyJsqS310jozD7lM=; b=DDnPziv8fGNxM89rLyWv39XhlG9JoniaFrYt36gEMn27prKe76wD0dda7BlpqwYbId tCYOAac2SbfHPFGNOHWmkUfIBno5Xur3Ya6gGpe4KgFlFuuLypPLVrdyXarIH/r0c8oI K05cvy/iYhxMIwkCFLhVl6bkL7HKDxcxfDHhSSlCkMdODVunyJM8b5e3d4nSlmnX8oFO VekZpPeJ7YGRcqaSwRcM7iofx24CA+tWbSztHNVn834Bzx2j93gVaOoQctRktBJfLOaW 5M8ZpcgTFZxQMXu8wTqy+agamGuOj1edjwtpjAbyQYX2fE9O5S+18IVZIaJvB2uSYvdM TzNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486840; x=1740091640; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3RwEBR7WcDA6C9/htn1wwdAW8BOhyJsqS310jozD7lM=; b=wwQWVri7dU6X3rxGXe2m89XFmJ2oXQvNwR+Fq+u/tnT2y3T7GPwmqPI7oDe3nO9fyz 2/V4FYbw86A53XWOUazqZBIUdSzTjvTl+19TTuIPYypy04LqBpaJLQx4XGdeNUSaM2OC JoOERjg2ey9rLQZ6+YMOWlO2XWe2eS+FktF5yWof3kh2SJca9REl9wZmE5whhNKY5g4j ZdzHLhevEYVETgZpLgD+0bb63WIzmcFXKbA10Ra7NN4VXD+KnCBahKhZE4/1p3ShZOng rvf7GlUWAUileEZ6LJCJqUQs5Gr9bLYx4h9dCpAHs/qw+nBMaO3SC915sIEWFRIG1Rws 9J/g== X-Forwarded-Encrypted: i=1; AJvYcCU96e8pvBeHoDQZFO3XziDeqoEinaKQ/yTmku6Me7bYHTIKXgmRgRtRyrKdbDh5NU8S9epI9vxDiQ==@kvack.org X-Gm-Message-State: AOJu0YwNo13RF5WfUJjNsEEU2LSE9Zzk4q7Xh3ZndKnFtHckJXTIxawk yJG2UZLypmYKrY0t4EdGUOvK9qMnDYoUMTosat4rz3C7WcZ3HAxwjp1dpzH0mu/ewRPuC0Pe4d9 /1Q== X-Google-Smtp-Source: AGHT+IG5tsVaIrkzimQ3d73aZezGeIWfqvUQe6a1x7KexmMSk23y9v/qQliOxJeTTumAOAHvlPYYt46pBpQ= X-Received: from pgid14.prod.google.com ([2002:a63:ed0e:0:b0:801:e378:a64a]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:2e81:b0:1ee:5cf2:9c07 with SMTP id adf61e73a8af0-1ee5cf29ed0mr11346671637.3.1739486840113; Thu, 13 Feb 2025 14:47:20 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:47 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-11-surenb@google.com> Subject: [PATCH v10 10/18] refcount: provide ops for cases when object's memory can be reused From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Will Deacon X-Rspamd-Queue-Id: 3EC062000F X-Stat-Signature: mpzeg9mx3osj5zrjabgn94x83sr5w1zz X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739486841-930021 X-HE-Meta: U2FsdGVkX1+ZkSrCtgCO8eW5okTo3USlYgLMUW1Q4aw1lyGX6yCo2XCAMMEGxQ3Q4tf/aj3xM/k+CroVTz8SZwUrBVjNMeF0AinaINYEUWTf8GWnUXaqGIuegdBQqWwX2zaK9pBhhHAsaubGUy6dst35wDmCcTbx4+R6QKi6YdeZPfZds+nqbe/FsNcy6yB6Ds/LdLhi0icPM7Yc+wq0t65tlOSZfmZ7/Q4phILdJNqQr4ZkSwptQxH3Ihnj1kIVe+10kPiMFIElj93+1SZNiPs6L1KNKqPDmF469ytjk+3Yw0WqwcLyJdOMKDv7CxG/sRe9rETKwuS3GlM0uu3vLnOZwZfDIXvGZ4825I0VJhP6E4aQRo+J/8p5Iqinpz0SbUqG4lJRbfwjvcfyU36SOtmd96c2O6Nf7cqlkn+xag50f9L/2TLdEgJx2dqWvKUw7PX3xKjTaEUrhhmkMFBiV35E/plheVMZ37aKcLafuZiZDH69o8jHlTz1uABrD8nvwDM7RUDfS067FL8CmLHsrsVfwDX6lVdjPUTkyH8TMvnIroz2L1K4yO2zXlO3+LVPiid5n3LW/NZxyya4sQOmix0qVVWX43iZ/OPLWKaV/0jg2wYmDAhmUf1RghtVaDmmhweunirWsMpUPBhNAx8lmsq7zb2O2RcroEICY3zJVBTfZr3ID5o9XXJQvFuUv/fn8aZA72G/kz0g7oAkahIT/8yQeuKZTg7Wqx130REyinLtKsowgjEzPujIbdpSkHxZx9F154qHOrHvwya/+iD/cOyP2elyreVbtMlSNhMIgseDzmeneR9YNHUAV1pvVwYAqaKAg8rex4XObWFkgXroz1x67Q4bGNIro7SQNmmbzYJIcF+IDGva5ZQAH4gwTucGQXtwqILUEKYb+kEB6IHIu74vBJsZiwAxTaBAkWZUz5ChpxT83zmWDQvmaHf+oG4OTUNCBJ3nQ8rfjWyHfLA K/BtiNw0 zTfGGI+NOi5H8HnpnYY/3g+FarTEpgdPtXIvEAeqfWLSTxGBk9h65EuKDgC3cz512cCMOEjk5vCgcemxhLYktDj+pCkigTNBcUmjqVrg0oVxQ//SM6+985clZrIsTpEfb/Qzt02HNR3DX2EcUuiTb1ZWjCsv9Hu9iYoD0Hb3X0yjqYi6Uk4+/AUFoPfNuvkVfrhrSxVWy7Mhsig3h01iKAtfaUFQAsGjzzILlY7JznU4c7yxdb42hugB1mxfUTIp8JSkgOqR44QEUoG9b+NK0Ek2X2wOEy6+nD6XdFtNqp0VMmQvuFrmuO/MIKmSlGS7CMYuI0Soqt1nFFHOQj/0C4FBivkbgrHhrnflsj6fqnAKJEVuJHxYFlcjZ+hYSdOBwyLbuJrnxPE0r+8WNolccaLo48BzP+XVbD5ZlvDDqxbjW2Lt9WQXtFq+4SKz89Hah/GS4c0Ru6VIgZUNUTfxxWTAjSSlU8669E0RiROymtHf+rQxoWild5lu97OPOXid8H3A36ME/QkrCmi+kUX4Av8t8T2f+5uOwhQNGiOQgFRXkm/c5Z9ExYBC2XGXRSC7+K+UnTFP2F5hhp2xssN9q3l+0GZpGI3py7bxdmSqJ5tm4W4wyFEfV8U+fu6lVySIDAwLH6VcSrHcxwjuvMiN1k2c8rIHvN074cLjYVH18bJeRAjgd9Y62hDrJbulIFeaRI/gqBKcBU/stw/91SqOlXseX+pNOHFjRfuZF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For speculative lookups where a successful inc_not_zero() pins the object, but where we still need to double check if the object acquired is indeed the one we set out to acquire (identity check), needs this validation to happen *after* the increment. Similarly, when a new object is initialized and its memory might have been previously occupied by another object, all stores to initialize the object should happen *before* refcount initialization. Notably SLAB_TYPESAFE_BY_RCU is one such an example when this ordering is required for reference counting. Add refcount_{add|inc}_not_zero_acquire() to guarantee the proper ordering between acquiring a reference count on an object and performing the identity check for that object. Add refcount_set_release() to guarantee proper ordering between stores initializing object attributes and the store initializing the refcount. refcount_set_release() should be done after all other object attributes are initialized. Once refcount_set_release() is called, the object should be considered visible to other tasks even if it was not yet added into an object collection normally used to discover it. This is because other tasks might have discovered the object previously occupying the same memory and after memory reuse they can succeed in taking refcount for the new object and start using it. Object reuse example to consider: consumer: obj = lookup(collection, key); if (!refcount_inc_not_zero_acquire(&obj->ref)) return; if (READ_ONCE(obj->key) != key) { /* identity check */ put_ref(obj); return; } use(obj->value); producer: remove(collection, obj->key); if (!refcount_dec_and_test(&obj->ref)) return; obj->key = KEY_INVALID; free(obj); obj = malloc(); /* obj is reused */ obj->key = new_key; obj->value = new_value; refcount_set_release(obj->ref, 1); add(collection, new_key, obj); refcount_{add|inc}_not_zero_acquire() is required to prevent the following reordering when refcount_inc_not_zero() is used instead: consumer: obj = lookup(collection, key); if (READ_ONCE(obj->key) != key) { /* reordered identity check */ put_ref(obj); return; } producer: remove(collection, obj->key); if (!refcount_dec_and_test(&obj->ref)) return; obj->key = KEY_INVALID; free(obj); obj = malloc(); /* obj is reused */ obj->key = new_key; obj->value = new_value; refcount_set_release(obj->ref, 1); add(collection, new_key, obj); if (!refcount_inc_not_zero(&obj->ref)) return; use(obj->value); /* USING WRONG OBJECT */ refcount_set_release() is required to prevent the following reordering when refcount_set() is used instead: consumer: obj = lookup(collection, key); producer: remove(collection, obj->key); if (!refcount_dec_and_test(&obj->ref)) return; obj->key = KEY_INVALID; free(obj); obj = malloc(); /* obj is reused */ obj->key = new_key; /* new_key == old_key */ refcount_set(obj->ref, 1); if (!refcount_inc_not_zero_acquire(&obj->ref)) return; if (READ_ONCE(obj->key) != key) { /* pass since new_key == old_key */ put_ref(obj); return; } use(obj->value); /* USING STALE obj->value */ obj->value = new_value; /* reordered store */ add(collection, key, obj); Signed-off-by: Suren Baghdasaryan Cc: Peter Zijlstra Cc: Will Deacon Cc: Paul E. McKenney Acked-by: Vlastimil Babka #slab --- Documentation/RCU/whatisRCU.rst | 10 ++ Documentation/core-api/refcount-vs-atomic.rst | 37 +++++- include/linux/refcount.h | 106 ++++++++++++++++++ include/linux/slab.h | 9 ++ 4 files changed, 156 insertions(+), 6 deletions(-) diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst index 1ef5784c1b84..53faeed7c190 100644 --- a/Documentation/RCU/whatisRCU.rst +++ b/Documentation/RCU/whatisRCU.rst @@ -971,6 +971,16 @@ unfortunately any spinlock in a ``SLAB_TYPESAFE_BY_RCU`` object must be initialized after each and every call to kmem_cache_alloc(), which renders reference-free spinlock acquisition completely unsafe. Therefore, when using ``SLAB_TYPESAFE_BY_RCU``, make proper use of a reference counter. +If using refcount_t, the specialized refcount_{add|inc}_not_zero_acquire() +and refcount_set_release() APIs should be used to ensure correct operation +ordering when verifying object identity and when initializing newly +allocated objects. Acquire fence in refcount_{add|inc}_not_zero_acquire() +ensures that identity checks happen *after* reference count is taken. +refcount_set_release() should be called after a newly allocated object is +fully initialized and release fence ensures that new values are visible +*before* refcount can be successfully taken by other users. Once +refcount_set_release() is called, the object should be considered visible +by other tasks. (Those willing to initialize their locks in a kmem_cache constructor may also use locking, including cache-friendly sequence locking.) diff --git a/Documentation/core-api/refcount-vs-atomic.rst b/Documentation/core-api/refcount-vs-atomic.rst index 79a009ce11df..9551a7bbfd38 100644 --- a/Documentation/core-api/refcount-vs-atomic.rst +++ b/Documentation/core-api/refcount-vs-atomic.rst @@ -86,7 +86,19 @@ Memory ordering guarantee changes: * none (both fully unordered) -case 2) - increment-based ops that return no value +case 2) - non-"Read/Modify/Write" (RMW) ops with release ordering +------------------------------------------- + +Function changes: + + * atomic_set_release() --> refcount_set_release() + +Memory ordering guarantee changes: + + * none (both provide RELEASE ordering) + + +case 3) - increment-based ops that return no value -------------------------------------------------- Function changes: @@ -98,7 +110,7 @@ Memory ordering guarantee changes: * none (both fully unordered) -case 3) - decrement-based RMW ops that return no value +case 4) - decrement-based RMW ops that return no value ------------------------------------------------------ Function changes: @@ -110,7 +122,7 @@ Memory ordering guarantee changes: * fully unordered --> RELEASE ordering -case 4) - increment-based RMW ops that return a value +case 5) - increment-based RMW ops that return a value ----------------------------------------------------- Function changes: @@ -126,7 +138,20 @@ Memory ordering guarantees changes: result of obtaining pointer to the object! -case 5) - generic dec/sub decrement-based RMW ops that return a value +case 6) - increment-based RMW ops with acquire ordering that return a value +----------------------------------------------------- + +Function changes: + + * atomic_inc_not_zero() --> refcount_inc_not_zero_acquire() + * no atomic counterpart --> refcount_add_not_zero_acquire() + +Memory ordering guarantees changes: + + * fully ordered --> ACQUIRE ordering on success + + +case 7) - generic dec/sub decrement-based RMW ops that return a value --------------------------------------------------------------------- Function changes: @@ -139,7 +164,7 @@ Memory ordering guarantees changes: * fully ordered --> RELEASE ordering + ACQUIRE ordering on success -case 6) other decrement-based RMW ops that return a value +case 8) other decrement-based RMW ops that return a value --------------------------------------------------------- Function changes: @@ -154,7 +179,7 @@ Memory ordering guarantees changes: .. note:: atomic_add_unless() only provides full order on success. -case 7) - lock-based RMW +case 9) - lock-based RMW ------------------------ Function changes: diff --git a/include/linux/refcount.h b/include/linux/refcount.h index 35f039ecb272..4589d2e7bfea 100644 --- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -87,6 +87,15 @@ * The decrements dec_and_test() and sub_and_test() also provide acquire * ordering on success. * + * refcount_{add|inc}_not_zero_acquire() and refcount_set_release() provide + * acquire and release ordering for cases when the memory occupied by the + * object might be reused to store another object. This is important for the + * cases where secondary validation is required to detect such reuse, e.g. + * SLAB_TYPESAFE_BY_RCU. The secondary validation checks have to happen after + * the refcount is taken, hence acquire order is necessary. Similarly, when the + * object is initialized, all stores to its attributes should be visible before + * the refcount is set, otherwise a stale attribute value might be used by + * another task which succeeds in taking a refcount to the new object. */ #ifndef _LINUX_REFCOUNT_H @@ -125,6 +134,31 @@ static inline void refcount_set(refcount_t *r, int n) atomic_set(&r->refs, n); } +/** + * refcount_set_release - set a refcount's value with release ordering + * @r: the refcount + * @n: value to which the refcount will be set + * + * This function should be used when memory occupied by the object might be + * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU. + * + * Provides release memory ordering which will order previous memory operations + * against this store. This ensures all updates to this object are visible + * once the refcount is set and stale values from the object previously + * occupying this memory are overwritten with new ones. + * + * This function should be called only after new object is fully initialized. + * After this call the object should be considered visible to other tasks even + * if it was not yet added into an object collection normally used to discover + * it. This is because other tasks might have discovered the object previously + * occupying the same memory and after memory reuse they can succeed in taking + * refcount to the new object and start using it. + */ +static inline void refcount_set_release(refcount_t *r, int n) +{ + atomic_set_release(&r->refs, n); +} + /** * refcount_read - get a refcount's value * @r: the refcount @@ -178,6 +212,52 @@ static inline __must_check bool refcount_add_not_zero(int i, refcount_t *r) return __refcount_add_not_zero(i, r, NULL); } +static inline __must_check __signed_wrap +bool __refcount_add_not_zero_acquire(int i, refcount_t *r, int *oldp) +{ + int old = refcount_read(r); + + do { + if (!old) + break; + } while (!atomic_try_cmpxchg_acquire(&r->refs, &old, old + i)); + + if (oldp) + *oldp = old; + + if (unlikely(old < 0 || old + i < 0)) + refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF); + + return old; +} + +/** + * refcount_add_not_zero_acquire - add a value to a refcount with acquire ordering unless it is 0 + * + * @i: the value to add to the refcount + * @r: the refcount + * + * Will saturate at REFCOUNT_SATURATED and WARN. + * + * This function should be used when memory occupied by the object might be + * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU. + * + * Provides acquire memory ordering on success, it is assumed the caller has + * guaranteed the object memory to be stable (RCU, etc.). It does provide a + * control dependency and thereby orders future stores. See the comment on top. + * + * Use of this function is not recommended for the normal reference counting + * use case in which references are taken and released one at a time. In these + * cases, refcount_inc_not_zero_acquire() should instead be used to increment a + * reference count. + * + * Return: false if the passed refcount is 0, true otherwise + */ +static inline __must_check bool refcount_add_not_zero_acquire(int i, refcount_t *r) +{ + return __refcount_add_not_zero_acquire(i, r, NULL); +} + static inline __signed_wrap void __refcount_add(int i, refcount_t *r, int *oldp) { @@ -236,6 +316,32 @@ static inline __must_check bool refcount_inc_not_zero(refcount_t *r) return __refcount_inc_not_zero(r, NULL); } +static inline __must_check bool __refcount_inc_not_zero_acquire(refcount_t *r, int *oldp) +{ + return __refcount_add_not_zero_acquire(1, r, oldp); +} + +/** + * refcount_inc_not_zero_acquire - increment a refcount with acquire ordering unless it is 0 + * @r: the refcount to increment + * + * Similar to refcount_inc_not_zero(), but provides acquire memory ordering on + * success. + * + * This function should be used when memory occupied by the object might be + * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU. + * + * Provides acquire memory ordering on success, it is assumed the caller has + * guaranteed the object memory to be stable (RCU, etc.). It does provide a + * control dependency and thereby orders future stores. See the comment on top. + * + * Return: true if the increment was successful, false otherwise + */ +static inline __must_check bool refcount_inc_not_zero_acquire(refcount_t *r) +{ + return __refcount_inc_not_zero_acquire(r, NULL); +} + static inline void __refcount_inc(refcount_t *r, int *oldp) { __refcount_add(1, r, oldp); diff --git a/include/linux/slab.h b/include/linux/slab.h index 09eedaecf120..ad902a2d692b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -136,6 +136,15 @@ enum _slab_flag_bits { * rcu_read_lock before reading the address, then rcu_read_unlock after * taking the spinlock within the structure expected at that address. * + * Note that object identity check has to be done *after* acquiring a + * reference, therefore user has to ensure proper ordering for loads. + * Similarly, when initializing objects allocated with SLAB_TYPESAFE_BY_RCU, + * the newly allocated object has to be fully initialized *before* its + * refcount gets initialized and proper ordering for stores is required. + * refcount_{add|inc}_not_zero_acquire() and refcount_set_release() are + * designed with the proper fences required for reference counting objects + * allocated with SLAB_TYPESAFE_BY_RCU. + * * Note that it is not possible to acquire a lock within a structure * allocated with SLAB_TYPESAFE_BY_RCU without first acquiring a reference * as described above. The reason is that SLAB_TYPESAFE_BY_RCU pages From patchwork Thu Feb 13 22:46:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38E8DC021A0 for ; Thu, 13 Feb 2025 22:47:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D7FF6B008A; Thu, 13 Feb 2025 17:47:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F7BE6B008C; Thu, 13 Feb 2025 17:47:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F0F5280001; Thu, 13 Feb 2025 17:47:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0A89E6B008A for ; Thu, 13 Feb 2025 17:47:25 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C5F221A174E for ; Thu, 13 Feb 2025 22:47:24 +0000 (UTC) X-FDA: 83116409208.04.8840FB3 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf21.hostedemail.com (Postfix) with ESMTP id 00C341C0012 for ; Thu, 13 Feb 2025 22:47:22 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DRH0EOex; spf=pass (imf21.hostedemail.com: domain of 3eXauZwYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3eXauZwYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486843; a=rsa-sha256; cv=none; b=JmMqdqa6DuDYO3Gi8bMY4dDGSRjwalRgDpuLaUFja9ktMVUckbYajSUVfY8xYdD5NtnZIU oSSel+7D/07bXEQ5KF6HuNpu1wdWZnhbLzz/l1choMx0pcWwf0H9UTjQXa1R7ekViO++Y/ uVGdE9djAqj3YDugcEwnLM93zAS94VE= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DRH0EOex; spf=pass (imf21.hostedemail.com: domain of 3eXauZwYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3eXauZwYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486843; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T2U39AcH2pHA/F+TfYeY5gc/lOd8HOvLuGlgxJNlsXk=; b=YkKknoQhADiLkd4pnSiMvb0yntrw4L4xXLdf7QnFaWQOnGD3kaT3J5i3LFC+mNhelWKrR4 AV3I70ePeF4o3upZHVOJeu24cqwkZ62Cep0ymzGAbkJBOOTfgcHf5GsuiYdXQkmlf+ZYq7 IZWF0a6R3JrU1pYNK49HrJFLV9hiVaY= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-220d1c24b25so31170635ad.0 for ; Thu, 13 Feb 2025 14:47:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486842; x=1740091642; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=T2U39AcH2pHA/F+TfYeY5gc/lOd8HOvLuGlgxJNlsXk=; b=DRH0EOexZzFWnNnPqM1FbQGLgVZcjHTYSym4x1EfaSMmK2DEMH1Tt2RaBHqOLzAUEN pu8REVj9MHcBujQDv3UFhyfSd3TBz2KJPlirPoCMxSj4w36gcR1/SywZ8CSuVThVf++t QoU1tHdoeptKbc4qb1d1HJOFxNaL9ME8n9hscWoEIpGiF/GDLfalRlxPpNelMbiw5zAr GK+jkysJB9apYsYI6Ag4XE1o5bn3/fSokVNp8LKRUkLqvhtEUin2OyDafYqy+LcagQc/ 7kt1ZJm4WlVbUueytV1DRFNK76LeLeXTlQxbGhrKT6l8l/xeuZno7HZNNGhhUMgCO9dM xHNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486842; x=1740091642; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T2U39AcH2pHA/F+TfYeY5gc/lOd8HOvLuGlgxJNlsXk=; b=UF3Ck+GgKWzsty8TA+0CbpwNaO89qNVfsirlraCAPEsL4EOEhN3I2JE/Kzkr+TWbHs 9rhhQQnarGga4yGW3P5QCD7pBvbVB/TRf9FMhzGUAdOG7xfMDMI8dLCAKce+9T9Y96rd eS0Ag0O0SHs6zowMulzFbsKjgpdaiTtnpoX9eCYC17bsTem3cePOEkEWwXredG5QDLMr 5uxprxvrfJAoW1RpOlQGOKNkQa7HRwzuRxYTEQYf1SmatFJTFr6TTi/opaXsOg+VLCnL /Zi0P7IZWo2qn1g27fXO6M8h3Bzu4JToVSnIA63GRihehX0yFvMPQRF7DPbhGWEz3uZ3 kYcA== X-Forwarded-Encrypted: i=1; AJvYcCVIvE4G1OXP1mJU2ljLzpd30FyJshg2Db/ctYnoju3dM05bI++69puYrCRZT7Rv3rhaNzx1uh4G8Q==@kvack.org X-Gm-Message-State: AOJu0YxFmNW4JwtVpgmnSAAQ4IXSxDAYGa9P69SnosL0IOXwbqduMC6O XUB3FoUWWjgM00RumHR36xF8PnyOxwajoyvOFYlwO+TWvpI2c+m4s5FMNW+TD6heFuQAjGSNHze rlg== X-Google-Smtp-Source: AGHT+IH/ab4L3c0kIEwgibMtPWiyLfRxDS8XonLWZO0QHAdpAq0IEkMMWm+kC4zlUCoEzXuh3expkwlJzJM= X-Received: from pji4.prod.google.com ([2002:a17:90b:3fc4:b0:2ea:aa56:49c]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:f706:b0:21f:7964:e989 with SMTP id d9443c01a7336-220d2368b12mr72348425ad.52.1739486841935; Thu, 13 Feb 2025 14:47:21 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:48 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-12-surenb@google.com> Subject: [PATCH v10 11/18] refcount: introduce __refcount_{add|inc}_not_zero_limited_acquire From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 00C341C0012 X-Stat-Signature: 8iifcpi877kp11qtik78wgm9tx5e11zw X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739486842-340342 X-HE-Meta: U2FsdGVkX1+eMBvWlN5Zid9lljephZVM/+ZuNuAHF7mWcXNHZWzgSGQ3xNJF9LA4g4qY5aotpINuik7qiACR0Q1CPBxsKHZ6gpSGDbYXO2CunJ1Y06128m2Wai6AKy57UGez+Q21eI6txRA/bUR2GqQAEJdWy3YeZN23RbcuIlCaqEgjjUbZXIf6inzwG8vR8ncLJwpdlI1KV4Fs0b1T1GiSNq2xNprNZmFeOjDc/1apXMkQyqVXmp9Jd/d/RdZ56W2uqoZzn9VNJgwNbiVXoP4tyUi1/IiRL5mBZWfP98f1UYgXCbg+3aMdyg0o/Ui1d9R96Ikh55gGn+eH2G3GWNfR5/xRMXiGunkBuuGl7ehLsdWO/lRN01OZgBLMB/FJ+Obrc/VFMX7pNHrMR3j3/J/JpM68rBvmfRxygCwKoALorp71doApOdxi6+tdNLTYFdgO9Yocr0yIina7UVVb8na3Gax8z9TLZBGWzg/oB72CEGA7wc2PLLSXfNtc1Ji/Nn67ctllOKoq8n8dVJ4m8KWwSZXLQ/pujL4CHKZS9eAQpiJJkzqIUawMjDFshfdKDN2rLshzjD2Mat6mZIN90BIdUYffXFHg+eC0TQHnJomnS/TlyRvjqfPHhnMf0Gcsv7VYzDXJAoVWqdTZifQSzdq8AzVtJNRvUzrOiolDcl92TfLpWNdstCLl+LVtLs+U14l5HBzUMUP9vf9kr355CGwTcEHZMjUpTImgX8pExF/PABnW2wqXI4H8knNGnJJKyJSH1U7piSfZ9zdEReXf04Oc9hhxEHOKtawD4uLlTZ2fbTWOFXTTbxxkZ4SttElilCUCwIc3SYNuSPylvj48yTN/BkpQlHkSe18WNEHnu5IK46fzz7MG3SSMojDq1LpLVaRHtJsLAGAPGQuzbNGAbOo6mAGif6miUYJLC2hc5ZM8p8C48ZpYzQ9Ef4jgPzPQMjCWMV4cjE1ou6pFDLe tmMEUklh md/afkGiP4cK0GMnE+vLSZ8c/u3NVkRzm8hAuMRZg7YdBvLqSgYoSKqfjfnBMKvrm34gJJ/ZzDOia9iAX14lF0i8Lvivz9335NWKgoFyHqSXDBZBTCVly4Ce+hxNO+k5wOPsotOWox+kQgkdb4Gc40v+8leVzYVOAIMQ6+GBD4FZfxyGSbNMbMZk5JOLCiASnW0VkeumFmIwxkzjNeMGqeEhKuUPvU+3EdiQlvZFdovfRdrLMQae1EbdJUJOEnLJOnvCPbKxMEQmxSfxa7PNhUyoMLv7fir8BtEb3sVIqhe38zlQJBifHijVb0E9RudLa82sr/H1fflSaM0Tm1V/oOHx5YfUbNs4u3ftVpEp3oL0MqwT6BNI+GfDJJdE7zDHXCIg24f93eOV8cDBp+w1QDL6dO2A0LqUsbQPvUKsC4twC4rpSvgpygElWePp3BlecnM+vdBWfEa/ssnB4f1ZeqorekllwvDcyIKwpw7qq/l3xJoYS/JC5W+2QZGbbKullgKIBCF69Y3F6GT2OB9RrGpmHwQI1C4dOipaKlhlAADpaHjTxfpEUyie7c9N9Ns6bVHjEiRaf/ZFky61okYycxGkXYkcS9R8lWYRKheTWquXA6QdtcghoFm8MpL6FlDoaYEHriXYjYUtkSOfECGCXvC3GICZILBwljE/H7l1evQnVfmtkvp1cPOx/zA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce functions to increase refcount but with a top limit above which they will fail to increase (the limit is inclusive). Setting the limit to INT_MAX indicates no limit. Signed-off-by: Suren Baghdasaryan --- Changes since v9 [1]: - Change refcount limit to be used with xxx_acquire functions [1] https://lore.kernel.org/all/20250111042604.3230628-11-surenb@google.com/ include/linux/refcount.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/include/linux/refcount.h b/include/linux/refcount.h index 4589d2e7bfea..80dc023ac2bf 100644 --- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -213,13 +213,20 @@ static inline __must_check bool refcount_add_not_zero(int i, refcount_t *r) } static inline __must_check __signed_wrap -bool __refcount_add_not_zero_acquire(int i, refcount_t *r, int *oldp) +bool __refcount_add_not_zero_limited_acquire(int i, refcount_t *r, int *oldp, + int limit) { int old = refcount_read(r); do { if (!old) break; + + if (i > limit - old) { + if (oldp) + *oldp = old; + return false; + } } while (!atomic_try_cmpxchg_acquire(&r->refs, &old, old + i)); if (oldp) @@ -231,6 +238,18 @@ bool __refcount_add_not_zero_acquire(int i, refcount_t *r, int *oldp) return old; } +static inline __must_check bool +__refcount_inc_not_zero_limited_acquire(refcount_t *r, int *oldp, int limit) +{ + return __refcount_add_not_zero_limited_acquire(1, r, oldp, limit); +} + +static inline __must_check __signed_wrap +bool __refcount_add_not_zero_acquire(int i, refcount_t *r, int *oldp) +{ + return __refcount_add_not_zero_limited_acquire(i, r, oldp, INT_MAX); +} + /** * refcount_add_not_zero_acquire - add a value to a refcount with acquire ordering unless it is 0 * From patchwork Thu Feb 13 22:46:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D6D1C021A0 for ; Thu, 13 Feb 2025 22:47:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 785EB28000F; Thu, 13 Feb 2025 17:47:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 70FC4280001; Thu, 13 Feb 2025 17:47:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4ECD328000F; Thu, 13 Feb 2025 17:47:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 242E8280001 for ; Thu, 13 Feb 2025 17:47:27 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DB596816B2 for ; Thu, 13 Feb 2025 22:47:26 +0000 (UTC) X-FDA: 83116409292.17.0C6C800 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 1848C8000B for ; Thu, 13 Feb 2025 22:47:24 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1suQTkjD; spf=pass (imf30.hostedemail.com: domain of 3e3auZwYKCGYWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3e3auZwYKCGYWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486845; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wFgt1YPKwtEKZHtY8eCCGOT/871l+BokJrFagsVrDsk=; b=UWbzSN5OylUbzQArxE7U9q0hY+fwTw1fp6xQmm+iZufI/ROTdhjErF+AHaNQVHi0u8TQnH XvY3yf+HUKrPJv86fYj6tuaxrJNGS5O9e8RIiEdx4AjD/xV7HvgmXTEe2zYETbIA/ah3GK qQrHSb3ZaT9NgkOVwY3GsM1MCWotXuE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486845; a=rsa-sha256; cv=none; b=ohXQs/tWgm6hYYsKk/U5RzFn5oWACbyvtLtkX4xnTaXxHRNqUCgyNdDGjPiCxbhUeidpkK KV/9It1VbhcqYeAcdw0eSeGBH7sp36TzsZJgZ1zSoDIj6d7wiKTsPaidgbq44vV4hyem0p pVu8ZU/8fRHwGMHy/m2cUClFS1q6Ghs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1suQTkjD; spf=pass (imf30.hostedemail.com: domain of 3e3auZwYKCGYWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3e3auZwYKCGYWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-220caea15ccso50153395ad.2 for ; Thu, 13 Feb 2025 14:47:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486844; x=1740091644; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wFgt1YPKwtEKZHtY8eCCGOT/871l+BokJrFagsVrDsk=; b=1suQTkjDLWyf4PTg7NWIkysGf2E9zxWKY4VPOA7hPLVMT5VK7bWu3UOjZrdSblfPRM p48PaGpriY35w/FObbho90rwJA63fqeMo3CfrnwJYRIrXL/Ip+GKaVsvLZDtNtBP3RjN MdCxz3VSEkIKACvW+QMvFoYbx6FjiuqYXFogT07qg2YJSO2zFEATS+oiovjJIPpGIFMe Mvdc/7W6H9Iuj81P9lcpkIJVVsCOQ0+G9YXK+2y1nd3c4bXeo58aJH+qXxIv4WMIHl+K Tt0lG5PrHOnfZHxSb/zTHNSd1NSME2I3Obuw6mi7SKfLGN8c/+vKo96VfHyUdJp6oXG5 OLQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486844; x=1740091644; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wFgt1YPKwtEKZHtY8eCCGOT/871l+BokJrFagsVrDsk=; b=aJmTCN918KKhJ8eya1ozaSw5oaaAyHx0WhI0kUNwcrqPa+MIkV3zAVVUE3G719Qcgd YwRT38THCoGuKJlaQoAvbREYcZQ9vkKrlr5U0qMnlz/J9EahBAPl7v8W3rbB6QEPfVFP kvlMkeO7b6gje4Jm8/j0z2MCABbtn/mWJyU7hbwZNyMqJvhNtLnTNyDy4nsIsBsuMgrl BZ09xvT6BP1HrYbOMnkNyhEg8iHnXBIrY6466FerxRNIsSfPaXyMlEf2fKlya11HJaNW OKpo5LuH6FTP0tVLDlCO/Gl5O2pFbTonptRMdFn9z/M8gamD1wbVB/FACEbd0tFXGVhZ DFNw== X-Forwarded-Encrypted: i=1; AJvYcCXd+LpN52mSvCBE2H9sV08cIa/douNnFnitiP7pKglQCitHnBnLyNRQoVieirRfhfc6BUQmVkE5wg==@kvack.org X-Gm-Message-State: AOJu0Yxu3e9GIxcAfak+x0nvxSLnIZWvY6qxNN3ApgmW+CjhouZ4Bivx RaivIPS+qFgevdmfNIrBNym/cGtsKCJEGr5cv0gs+5pjenatv8MziECNMX5weKrcua8L92AShTh utw== X-Google-Smtp-Source: AGHT+IFBfclnXX1eyWHaQBadHjNWP/bw7xtO4voK2pMlEan4fNA0aBMg12hKdOHhEFlJAYATtSiZC0Oajhs= X-Received: from plgb14.prod.google.com ([2002:a17:902:d50e:b0:21f:39fb:79d3]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:32c8:b0:220:ea90:1925 with SMTP id d9443c01a7336-220ea9019a9mr22932205ad.35.1739486843944; Thu, 13 Feb 2025 14:47:23 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:49 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-13-surenb@google.com> Subject: [PATCH v10 12/18] mm: replace vm_lock and detached flag with a reference count From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 1848C8000B X-Rspamd-Server: rspam07 X-Stat-Signature: 37cwixckc6ncppsgi9xjh9thrgx7p3ko X-HE-Tag: 1739486844-427498 X-HE-Meta: U2FsdGVkX1+bBLVZTc2e9gqlH8jU7CR+N30qnTLDIecUPNgkpHM4v7OlmUZ+HQQOudHt8NQPEoxLSlJQtSfR4ODXZnv4EwyDNmMAbsrksdbKn4obl6mcQQ/00oXj7BuMIVeU4vRCe/rhnXv9Jii1w5qfUe2TXiCnDH8yiEJmtIkZAwoKCvsDAbbXNxKqswodq30ypqVsvVmWoe1sqkvtqvm5ZcRawY6RNpsgYOPWzVErfMqirpok1Jf+2CmGKKNu4Z7d5lRnKVoq5Rx8yNDAAn51VOyITkcOSCt90CUjeghhMqUmcksB6VvUC6Y4bQSuuiBEJTj6nYKuh1FqbO/sHqp5FRQvm6PMBe6PxEpdgV0y8SiZ1txCTXFBpztKT5B8870KGGg0uF7b3eMJCVXpt4a5SckOAGOvA2UrBNIWvYeNILKpE99QVg7MK+yx513KCCUb4WFGotLhFJu+5hlqEqI+KsVClL1fpG+s0uop8NRcgZzkIPDyPxmyjLzOLAGwdJnAKoz678yu2V4biJ9xrqjpCUWIic1oBXcPHf4KsZVbC7J0hXw/VWimDwAfvrRjUY8vkgSetjcxLQZelCPKRF3HwUrn0v/lR/uooa8HS+0QjGRuFVc9s4En+XtMLs7Y02CEwMGS3pagmfHGabvezGgtnlilPCRpDMRQdaCYI/IaSEyWxPZ0kRxDyJVSywYI6wMQzV+dKdpsEwteqXw6uNNf9xwQNZ21wdT5SLAkbJ8ogikECQ3A4MR2DzydNkeNyhBXrEvonYJ5vgFChVhtnOfXo8toQyE+3XdD4P2qZ2K0M0/aRSCRGmmOBWn84Ax/2Nk5nr9IjYbmYZBAwF2fomO2qYXVeA3cgLZj5OpEHVJ/SlPDen2X5MUHOQL2UgEXrBTqNZOEVo9OGRM0ekAWlwipmflyTLM6iJb9vA6d71e/v+G9cnD/nN2U7kyJqPXEzvBHNP8SF68Lc9dw+5a vzTx2jNd TryvKqwY9LvtewKiwtxMSbeBnAPorY66lCtVNZxazJuWwR+Zs0dhjsrPH9gBgkR4TqmrrHQbFqbgDLsJS9j+JMcyuNbAXE8aanD+41AnBNAuQCoyLHyARAepdrz0y2+fgSGwZd0wvme7Sf0B760N+l0tBj66P9n7jw69BcTJc0S+KSFwZjA3rJsI3zdnxHWeNUokIc31rZm/R6RVejbQCV30Awu60oR94CUKArz1poF523Kl5DjwWo8AsYwi8/ot7eU4CFcrb7uVc5Wugn7fEZ2ysSiHyeKV1yX0q4IYPEImDGO5O0ichi3Uv/xHNwUlmLRqdxYIEM1lRdLqk6BanDQ1/eXCOGXuuYZAXPAS/t3mWI6uz6DlFmAgbioWzgtwu+luI9SSbGRG9vwV/SzV7O2+4g+YCkwsTTXExXmnkOvdELBNqh/EaEhZCEQvcEjCI746AoU1GMHyT+Gc63o5aPEysNIH1aNoA5GQ8xc4fApU6iQHIPoikEusV/UOhSSwD7qsnbnOfxa9sTfYBfKX0RoXLxfrGUhcHQxpp8rE7d86Vj1tQIZ9B1eaxEc1Yn9R75BQTumktug5bL0ldm0bn5NV0iEvZsGVm/T4gT+8yfmDTv92jzWwxNi3jU/lCP2ESJz7i6ZI1x3Ggdk3bbXkXnAZevh6XB8t2S8d9iH7SfjzEw3lF7iAy4jowKaN7Q3xEi93fS7JGbwSapgDI9YCUaHgbF/lACQv3tMoDk8iAtz2ZOao= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: rw_semaphore is a sizable structure of 40 bytes and consumes considerable space for each vm_area_struct. However vma_lock has two important specifics which can be used to replace rw_semaphore with a simpler structure: 1. Readers never wait. They try to take the vma_lock and fall back to mmap_lock if that fails. 2. Only one writer at a time will ever try to write-lock a vma_lock because writers first take mmap_lock in write mode. Because of these requirements, full rw_semaphore functionality is not needed and we can replace rw_semaphore and the vma->detached flag with a refcount (vm_refcnt). When vma is in detached state, vm_refcnt is 0 and only a call to vma_mark_attached() can take it out of this state. Note that unlike before, now we enforce both vma_mark_attached() and vma_mark_detached() to be done only after vma has been write-locked. vma_mark_attached() changes vm_refcnt to 1 to indicate that it has been attached to the vma tree. When a reader takes read lock, it increments vm_refcnt, unless the top usable bit of vm_refcnt (0x40000000) is set, indicating presence of a writer. When writer takes write lock, it sets the top usable bit to indicate its presence. If there are readers, writer will wait using newly introduced mm->vma_writer_wait. Since all writers take mmap_lock in write mode first, there can be only one writer at a time. The last reader to release the lock will signal the writer to wake up. refcount might overflow if there are many competing readers, in which case read-locking will fail. Readers are expected to handle such failures. In summary: 1. all readers increment the vm_refcnt; 2. writer sets top usable (writer) bit of vm_refcnt; 3. readers cannot increment the vm_refcnt if the writer bit is set; 4. in the presence of readers, writer must wait for the vm_refcnt to drop to 1 (plus the VMA_LOCK_OFFSET writer bit), indicating an attached vma with no readers; 5. vm_refcnt overflow is handled by the readers. While this vm_lock replacement does not yet result in a smaller vm_area_struct (it stays at 256 bytes due to cacheline alignment), it allows for further size optimization by structure member regrouping to bring the size of vm_area_struct below 192 bytes. Suggested-by: Peter Zijlstra Suggested-by: Matthew Wilcox Signed-off-by: Suren Baghdasaryan --- Changes since v9 [1]: - Use __refcount_inc_not_zero_limited_acquire() in vma_start_read(), per Hillf Danton - Refactor vma_assert_locked() to avoid vm_refcnt read when CONFIG_DEBUG_VM=n, per Mateusz Guzik - Update changelog, per Wei Yang - Change vma_start_read() to return EAGAIN if vma got isolated and changed lock_vma_under_rcu() back to detect this condition, per Wei Yang - Change VM_BUG_ON_VMA() to WARN_ON_ONCE() when checking vma detached state, per Lorenzo Stoakes - Remove Vlastimil's Reviewed-by since code is changed [1] https://lore.kernel.org/all/20250111042604.3230628-12-surenb@google.com/ include/linux/mm.h | 128 ++++++++++++++++++++----------- include/linux/mm_types.h | 22 +++--- kernel/fork.c | 13 ++-- mm/init-mm.c | 1 + mm/memory.c | 91 +++++++++++++++++++--- tools/testing/vma/linux/atomic.h | 5 ++ tools/testing/vma/vma_internal.h | 63 ++++++++------- 7 files changed, 218 insertions(+), 105 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 557d66e187ff..11a042c27aee 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -32,6 +32,7 @@ #include #include #include +#include struct mempolicy; struct anon_vma; @@ -697,19 +698,54 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {} #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_lock_init(struct vm_area_struct *vma) +static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) { - init_rwsem(&vma->vm_lock.lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + static struct lock_class_key lockdep_key; + + lockdep_init_map(&vma->vmlock_dep_map, "vm_lock", &lockdep_key, 0); +#endif + if (reset_refcnt) + refcount_set(&vma->vm_refcnt, 0); vma->vm_lock_seq = UINT_MAX; } +static inline bool is_vma_writer_only(int refcnt) +{ + /* + * With a writer and no readers, refcnt is VMA_LOCK_OFFSET if the vma + * is detached and (VMA_LOCK_OFFSET + 1) if it is attached. Waiting on + * a detached vma happens only in vma_mark_detached() and is a rare + * case, therefore most of the time there will be no unnecessary wakeup. + */ + return refcnt & VMA_LOCK_OFFSET && refcnt <= VMA_LOCK_OFFSET + 1; +} + +static inline void vma_refcount_put(struct vm_area_struct *vma) +{ + /* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt */ + struct mm_struct *mm = vma->vm_mm; + int oldcnt; + + rwsem_release(&vma->vmlock_dep_map, _RET_IP_); + if (!__refcount_dec_and_test(&vma->vm_refcnt, &oldcnt)) { + + if (is_vma_writer_only(oldcnt - 1)) + rcuwait_wake_up(&mm->vma_writer_wait); + } +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. + * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got + * detached. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline struct vm_area_struct *vma_start_read(struct vm_area_struct *vma) { + int oldcnt; + /* * Check before locking. A race might cause false locked result. * We can use READ_ONCE() for the mm_lock_seq here, and don't need @@ -718,15 +754,25 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * need ordering is below. */ if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) - return false; + return NULL; - if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) - return false; + /* + * If VMA_LOCK_OFFSET is set, __refcount_inc_not_zero_limited_acquire() + * will fail because VMA_REF_LIMIT is less than VMA_LOCK_OFFSET. + * Acquire fence is required here to avoid reordering against later + * vm_lock_seq check and checks inside lock_vma_under_rcu(). + */ + if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt, + VMA_REF_LIMIT))) { + /* return EAGAIN if vma got detached from under us */ + return oldcnt ? NULL : ERR_PTR(-EAGAIN); + } + rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_); /* - * Overflow might produce false locked result. + * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * * We must use ACQUIRE semantics for the mm_lock_seq so that if we are @@ -735,10 +781,11 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock.lock); - return false; + vma_refcount_put(vma); + return NULL; } - return true; + + return vma; } /* @@ -749,8 +796,14 @@ static inline bool vma_start_read(struct vm_area_struct *vma) */ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { + int oldcnt; + mmap_assert_locked(vma->vm_mm); - down_read_nested(&vma->vm_lock.lock, subclass); + if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt, + VMA_REF_LIMIT))) + return false; + + rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_); return true; } @@ -762,16 +815,12 @@ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int */ static inline bool vma_start_read_locked(struct vm_area_struct *vma) { - mmap_assert_locked(vma->vm_mm); - down_read(&vma->vm_lock.lock); - return true; + return vma_start_read_locked_nested(vma, 0); } static inline void vma_end_read(struct vm_area_struct *vma) { - rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->vm_lock.lock); - rcu_read_unlock(); + vma_refcount_put(vma); } /* WARNING! Can only be used if mmap_lock is expected to be write-locked */ @@ -813,38 +862,35 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_locked(struct vm_area_struct *vma) { - if (!rwsem_is_locked(&vma->vm_lock.lock)) - vma_assert_write_locked(vma); + unsigned int mm_lock_seq; + + VM_BUG_ON_VMA(refcount_read(&vma->vm_refcnt) <= 1 && + !__is_vma_write_locked(vma, &mm_lock_seq), vma); } +/* + * WARNING: to avoid racing with vma_mark_attached()/vma_mark_detached(), these + * assertions should be made either under mmap_write_lock or when the object + * has been isolated under mmap_write_lock, ensuring no competing writers. + */ static inline void vma_assert_attached(struct vm_area_struct *vma) { - WARN_ON_ONCE(vma->detached); + WARN_ON_ONCE(!refcount_read(&vma->vm_refcnt)); } static inline void vma_assert_detached(struct vm_area_struct *vma) { - WARN_ON_ONCE(!vma->detached); + WARN_ON_ONCE(refcount_read(&vma->vm_refcnt)); } static inline void vma_mark_attached(struct vm_area_struct *vma) { - vma_assert_detached(vma); - vma->detached = false; -} - -static inline void vma_mark_detached(struct vm_area_struct *vma) -{ - /* When detaching vma should be write-locked */ vma_assert_write_locked(vma); - vma_assert_attached(vma); - vma->detached = true; + vma_assert_detached(vma); + refcount_set(&vma->vm_refcnt, 1); } -static inline bool is_vma_detached(struct vm_area_struct *vma) -{ - return vma->detached; -} +void vma_mark_detached(struct vm_area_struct *vma); static inline void release_fault_lock(struct vm_fault *vmf) { @@ -867,9 +913,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ -static inline void vma_lock_init(struct vm_area_struct *vma) {} -static inline bool vma_start_read(struct vm_area_struct *vma) - { return false; } +static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) {} +static inline struct vm_area_struct *vma_start_read(struct vm_area_struct *vma) + { return NULL; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -910,12 +956,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); -#ifdef CONFIG_PER_VMA_LOCK - /* vma is not locked, can't use vma_mark_detached() */ - vma->detached = true; -#endif vma_numab_state_init(vma); - vma_lock_init(vma); + vma_lock_init(vma, false); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8a645bcb2b31..48ddfedfff83 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -19,6 +19,7 @@ #include #include #include +#include #include @@ -639,9 +640,8 @@ static inline struct anon_vma_name *anon_vma_name_alloc(const char *name) } #endif -struct vma_lock { - struct rw_semaphore lock; -}; +#define VMA_LOCK_OFFSET 0x40000000 +#define VMA_REF_LIMIT (VMA_LOCK_OFFSET - 1) struct vma_numab_state { /* @@ -719,19 +719,13 @@ struct vm_area_struct { }; #ifdef CONFIG_PER_VMA_LOCK - /* - * Flag to indicate areas detached from the mm->mm_mt tree. - * Unstable RCU readers are allowed to read this. - */ - bool detached; - /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) - * - vm_lock->lock (in write mode) + * - vm_refcnt bit at VMA_LOCK_OFFSET is set * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_lock->lock (in read or write mode) + * - vm_refcnt bit at VMA_LOCK_OFFSET is set or vm_refcnt > 1 * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * @@ -794,7 +788,10 @@ struct vm_area_struct { struct vm_userfaultfd_ctx vm_userfaultfd_ctx; #ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ - struct vma_lock vm_lock ____cacheline_aligned_in_smp; + refcount_t vm_refcnt ____cacheline_aligned_in_smp; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map vmlock_dep_map; +#endif #endif } __randomize_layout; @@ -929,6 +926,7 @@ struct mm_struct { * by mmlist_lock */ #ifdef CONFIG_PER_VMA_LOCK + struct rcuwait vma_writer_wait; /* * This field has lock-like semantics, meaning it is sometimes * accessed with ACQUIRE/RELEASE semantics. diff --git a/kernel/fork.c b/kernel/fork.c index f1af413e5aa4..48a0038f606f 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -463,12 +463,8 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) * will be reinitialized. */ data_race(memcpy(new, orig, sizeof(*new))); - vma_lock_init(new); + vma_lock_init(new, true); INIT_LIST_HEAD(&new->anon_vma_chain); -#ifdef CONFIG_PER_VMA_LOCK - /* vma is not locked, can't use vma_mark_detached() */ - new->detached = true; -#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); @@ -477,6 +473,8 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) void __vm_area_free(struct vm_area_struct *vma) { + /* The vma should be detached while being destroyed. */ + vma_assert_detached(vma); vma_numab_state_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); @@ -488,8 +486,6 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); - /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); __vm_area_free(vma); } #endif @@ -1234,6 +1230,9 @@ static void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); mm_lock_seqcount_init(mm); +#ifdef CONFIG_PER_VMA_LOCK + rcuwait_init(&mm->vma_writer_wait); +#endif } static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, diff --git a/mm/init-mm.c b/mm/init-mm.c index 6af3ad675930..4600e7605cab 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -40,6 +40,7 @@ struct mm_struct init_mm = { .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK + .vma_writer_wait = __RCUWAIT_INITIALIZER(init_mm.vma_writer_wait), .mm_lock_seq = SEQCNT_ZERO(init_mm.mm_lock_seq), #endif .user_ns = &init_user_ns, diff --git a/mm/memory.c b/mm/memory.c index 3d9c5141193f..528407c0d7cf 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6393,9 +6393,47 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, #endif #ifdef CONFIG_PER_VMA_LOCK +static inline bool __vma_enter_locked(struct vm_area_struct *vma, bool detaching) +{ + unsigned int tgt_refcnt = VMA_LOCK_OFFSET; + + /* Additional refcnt if the vma is attached. */ + if (!detaching) + tgt_refcnt++; + + /* + * If vma is detached then only vma_mark_attached() can raise the + * vm_refcnt. mmap_write_lock prevents racing with vma_mark_attached(). + */ + if (!refcount_add_not_zero(VMA_LOCK_OFFSET, &vma->vm_refcnt)) + return false; + + rwsem_acquire(&vma->vmlock_dep_map, 0, 0, _RET_IP_); + rcuwait_wait_event(&vma->vm_mm->vma_writer_wait, + refcount_read(&vma->vm_refcnt) == tgt_refcnt, + TASK_UNINTERRUPTIBLE); + lock_acquired(&vma->vmlock_dep_map, _RET_IP_); + + return true; +} + +static inline void __vma_exit_locked(struct vm_area_struct *vma, bool *detached) +{ + *detached = refcount_sub_and_test(VMA_LOCK_OFFSET, &vma->vm_refcnt); + rwsem_release(&vma->vmlock_dep_map, _RET_IP_); +} + void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) { - down_write(&vma->vm_lock.lock); + bool locked; + + /* + * __vma_enter_locked() returns false immediately if the vma is not + * attached, otherwise it waits until refcnt is indicating that vma + * is attached with no readers. + */ + locked = __vma_enter_locked(vma, false); + /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -6403,10 +6441,40 @@ void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock.lock); + + if (locked) { + bool detached; + + __vma_exit_locked(vma, &detached); + WARN_ON_ONCE(detached); /* vma should remain attached */ + } } EXPORT_SYMBOL_GPL(__vma_start_write); +void vma_mark_detached(struct vm_area_struct *vma) +{ + vma_assert_write_locked(vma); + vma_assert_attached(vma); + + /* + * We are the only writer, so no need to use vma_refcount_put(). + * The condition below is unlikely because the vma has been already + * write-locked and readers can increment vm_refcnt only temporarily + * before they check vm_lock_seq, realize the vma is locked and drop + * back the vm_refcnt. That is a narrow window for observing a raised + * vm_refcnt. + */ + if (unlikely(!refcount_dec_and_test(&vma->vm_refcnt))) { + /* Wait until vma is detached with no readers. */ + if (__vma_enter_locked(vma, true)) { + bool detached; + + __vma_exit_locked(vma, &detached); + WARN_ON_ONCE(!detached); + } + } +} + /* * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be * stable and not isolated. If the VMA is not found or is being modified the @@ -6424,15 +6492,18 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) - goto inval; + vma = vma_start_read(vma); + if (IS_ERR_OR_NULL(vma)) { + /* Check if the VMA got isolated after we found it */ + if (PTR_ERR(vma) == -EAGAIN) { + vma_end_read(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one */ + goto retry; + } - /* Check if the VMA got isolated after we found it */ - if (is_vma_detached(vma)) { - vma_end_read(vma); - count_vm_vma_lock_event(VMA_LOCK_MISS); - /* The area was replaced with another one */ - goto retry; + /* Failed to lock the VMA */ + goto inval; } /* * At this point, we have a stable reference to a VMA: The VMA is diff --git a/tools/testing/vma/linux/atomic.h b/tools/testing/vma/linux/atomic.h index 3e1b6adc027b..788c597c4fde 100644 --- a/tools/testing/vma/linux/atomic.h +++ b/tools/testing/vma/linux/atomic.h @@ -9,4 +9,9 @@ #define atomic_set(x, y) uatomic_set(x, y) #define U8_MAX UCHAR_MAX +#ifndef atomic_cmpxchg_relaxed +#define atomic_cmpxchg_relaxed uatomic_cmpxchg +#define atomic_cmpxchg_release uatomic_cmpxchg +#endif /* atomic_cmpxchg_relaxed */ + #endif /* _LINUX_ATOMIC_H */ diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 34277842156c..ba838097d3f6 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -25,7 +25,7 @@ #include #include #include -#include +#include extern unsigned long stack_guard_gap; #ifdef CONFIG_MMU @@ -135,10 +135,6 @@ typedef __bitwise unsigned int vm_fault_t; */ #define pr_warn_once pr_err -typedef struct refcount_struct { - atomic_t refs; -} refcount_t; - struct kref { refcount_t refcount; }; @@ -233,15 +229,12 @@ struct mm_struct { unsigned long flags; /* Must use atomic bitops to access */ }; -struct vma_lock { - struct rw_semaphore lock; -}; - - struct file { struct address_space *f_mapping; }; +#define VMA_LOCK_OFFSET 0x40000000 + struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ @@ -269,16 +262,13 @@ struct vm_area_struct { }; #ifdef CONFIG_PER_VMA_LOCK - /* Flag to indicate areas detached from the mm->mm_mt tree */ - bool detached; - /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) - * - vm_lock.lock (in write mode) + * - vm_refcnt bit at VMA_LOCK_OFFSET is set * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_lock.lock (in read or write mode) + * - vm_refcnt bit at VMA_LOCK_OFFSET is set or vm_refcnt > 1 * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * @@ -287,7 +277,6 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - struct vma_lock vm_lock; #endif /* @@ -340,6 +329,10 @@ struct vm_area_struct { struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + /* Unstable RCU readers are allowed to read this. */ + refcount_t vm_refcnt; +#endif } __randomize_layout; struct vm_fault {}; @@ -464,33 +457,40 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) return mas_find(&vmi->mas, ULONG_MAX); } -static inline void vma_lock_init(struct vm_area_struct *vma) -{ - init_rwsem(&vma->vm_lock.lock); - vma->vm_lock_seq = UINT_MAX; -} - +/* + * WARNING: to avoid racing with vma_mark_attached()/vma_mark_detached(), these + * assertions should be made either under mmap_write_lock or when the object + * has been isolated under mmap_write_lock, ensuring no competing writers. + */ static inline void vma_assert_attached(struct vm_area_struct *vma) { - WARN_ON_ONCE(vma->detached); + WARN_ON_ONCE(!refcount_read(&vma->vm_refcnt)); } static inline void vma_assert_detached(struct vm_area_struct *vma) { - WARN_ON_ONCE(!vma->detached); + WARN_ON_ONCE(refcount_read(&vma->vm_refcnt)); } static inline void vma_assert_write_locked(struct vm_area_struct *); static inline void vma_mark_attached(struct vm_area_struct *vma) { - vma->detached = false; + vma_assert_write_locked(vma); + vma_assert_detached(vma); + refcount_set(&vma->vm_refcnt, 1); } static inline void vma_mark_detached(struct vm_area_struct *vma) { - /* When detaching vma should be write-locked */ vma_assert_write_locked(vma); - vma->detached = true; + vma_assert_attached(vma); + /* We are the only writer, so no need to use vma_refcount_put(). */ + if (unlikely(!refcount_dec_and_test(&vma->vm_refcnt))) { + /* + * Reader must have temporarily raised vm_refcnt but it will + * drop it without using the vma since vma is write-locked. + */ + } } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -503,9 +503,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - /* vma is not locked, can't use vma_mark_detached() */ - vma->detached = true; - vma_lock_init(vma); + vma->vm_lock_seq = UINT_MAX; } static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) @@ -528,10 +526,9 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return NULL; memcpy(new, orig, sizeof(*new)); - vma_lock_init(new); + refcount_set(&new->vm_refcnt, 0); + new->vm_lock_seq = UINT_MAX; INIT_LIST_HEAD(&new->anon_vma_chain); - /* vma is not locked, can't use vma_mark_detached() */ - new->detached = true; return new; } From patchwork Thu Feb 13 22:46:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54862C021A4 for ; Thu, 13 Feb 2025 22:47:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 711C6280010; Thu, 13 Feb 2025 17:47:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 69AF4280001; Thu, 13 Feb 2025 17:47:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49FB7280010; Thu, 13 Feb 2025 17:47:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1DE43280001 for ; Thu, 13 Feb 2025 17:47:29 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CBA99816E0 for ; Thu, 13 Feb 2025 22:47:28 +0000 (UTC) X-FDA: 83116409376.08.F11D673 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf01.hostedemail.com (Postfix) with ESMTP id 0D8AD40015 for ; Thu, 13 Feb 2025 22:47:26 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=J0q3Fqxi; spf=pass (imf01.hostedemail.com: domain of 3fXauZwYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3fXauZwYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486847; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p2XHEc7Lc99pXt/+24Q50pb3G2smdSAP5b2ttcYEcJw=; b=wNwrgUoYIaGJrbz3K5HRJQFV0tYtBucn8Puh4snVnqZNBlSfMwv4E8a7pgqzOjvgJDEauz A85eUp5FAnaceqWwibc/5wImCRPmIWgNi0uocNM0++CVQ2irZkPPZTIBEwf3+oawYR2M5v 7bcCHAi32FKlwlfcqM+kdslgYJox4bk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=J0q3Fqxi; spf=pass (imf01.hostedemail.com: domain of 3fXauZwYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3fXauZwYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486847; a=rsa-sha256; cv=none; b=fhUvBvh4dcWlo18pWoTD8S42olrYDyL/DYRbA0mlPvJ2118DD11Sk99yS2mTI41wrueeri mIQ7N2vEQD9R07YcMYb0W7P4qxJxu8AoqCOqtrxw1XVjVfgH0iMIIA0Yo3SVcabiwhwFIW 2SfrOFUe3td3RVASCFiDXp35QPJ3USg= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2fbfa786a1aso4529230a91.3 for ; Thu, 13 Feb 2025 14:47:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486846; x=1740091646; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=p2XHEc7Lc99pXt/+24Q50pb3G2smdSAP5b2ttcYEcJw=; b=J0q3Fqxi1N4u4ZOkkYwK6KVP4C6sQbR1fahreaBYoZBEcAaAmcVU6h8jiExV8N8p0z CpIBLDCXtreY1lg0hh+pc2c5KUd6FF/dnCUIZhk520zVAxvamhEmobQmmI8ZLOgSjUhZ ZhEmTUYaK4uoM70TIaH9JlXC47RYZvmKP6oddSnHGfEii7t515FxfLpM3jm1NZTkNYy0 0RATUTTNn8M8+KeE/ybx5JgVByTGSItMGy9VWkfm/WInlVmX3oyyGAnvhtdajaF3w4vZ qhtn0+5guHIr+iTUNbTuWRPbY5vM7Ea1ojn7VuO4IITTzAZIQP5HEk6Ek+5prJdZ6WKx vONQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486846; x=1740091646; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=p2XHEc7Lc99pXt/+24Q50pb3G2smdSAP5b2ttcYEcJw=; b=rF9o0tAsXFLpk2iTyxIO3oAPVCpwvCdUXgRKvjPbNsZwzEpYNWojtboHC0nwXDCBV5 287WalYA+Mps+edbw8K4UsI9Jzw7HeHTEx0YqqdH5vrBeBlfXE/Byp+oIZtBRZJ7PzvE +fih1w1iPE3yHOk3jyoLC2/TJ1F+K0rwSBU579h7p6CGORomV//x/nq9qGt5oB5tZ18d 2VnrzB6T7aSoJOg/Y6kJTEtbi6JsQIjA67h7aC6VfT7LoDs3ie+QGx7QvqMDo+Em+Rwo 0BHeCM9qOOo6KcC8cyqVLxr0AyHu5vzNTKD9Y7PW49cyUebX93LxSxA6RwImhHwbkXfR Wlbw== X-Forwarded-Encrypted: i=1; AJvYcCVNJ9RjvZAHxSyG7+RU9CTT5DPEmW5unOQH/YViSEqmfT7+OP3NAifrNKxqP0e5Jpg7YqE+Y0lVsg==@kvack.org X-Gm-Message-State: AOJu0Yz9YteYP8xDxoyTeQTLBB9XyFZKNyg/BmIEwz0/IAQEYxiylyyE 2g2kZbptLPMAMPnVOAxGvvAlqPTN9BvVNlGJo5B6zXqzYCWMUOJQEPuoVVmBZdh3PSD1BimVtrX J8w== X-Google-Smtp-Source: AGHT+IHxvu/76zqwjXOeUtKeapkgzxoZmdl37UyKOvAGkMpiwzZvuuSG6bTHkHJu7fwwbTV32aeg1MBY6BY= X-Received: from pjap1.prod.google.com ([2002:a17:90a:e41:b0:2fc:c98:ea47]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1e47:b0:2ee:74a1:fba2 with SMTP id 98e67ed59e1d1-2fc0e5b92e1mr6734995a91.20.1739486845903; Thu, 13 Feb 2025 14:47:25 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:50 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-14-surenb@google.com> Subject: [PATCH v10 13/18] mm: move lesser used vma_area_struct members into the last cacheline From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0D8AD40015 X-Stat-Signature: 38hsy4samrc7eqzgq8rn5ffgbr7r3nes X-HE-Tag: 1739486846-141601 X-HE-Meta: U2FsdGVkX1+ipvsGxm3TdJuZY8rM9zQHDPPCquXwNpYkPgANz0Acq2vxe3KtmB0fhcNwjJ7Ki3qC9ROMoY4qUFPE/JUELs101HVepIsM2F8maLefSRV+jSsEMeJihZJU4L5ut0neIQPB594ElqkWIAolhAt4wj2Iz6PisiuCXGb8AfAK6lMw8wmEM4iz3ukeK9GjNRFIHkKQSYP/mjhrSFz81sWDOfOs8ofZCLNIq0dEf3txyviG8wNceCCtY6OitYxfHE+1dUMzM6Ri0ogk5JTVNOZbdMHFj8IhPCmFVoJ75GatswAorjMjtfIh5QI4NoqwMUw8oO3hjry4prjkNdJ5ufrucdy0IUeMAeoIvZwiSG49i826P2uNi72CvC9zgl8HJ1iizi4h9JeMtNHtV3fNyQAxDu8d4sxEl1POE9ANSj+UDW8QcV0hB6N/dkSFY0TbVOctwrcwPSViCAP/Y25kRHeALNRg4PJ1OPnJmMRVZfIYw1heh5I3+tb03TlVvAMm5HQ3SiaGPtIgwZTdFo3zLvy0ZV70fSO4WRwH5hm+neNKmCO37QjVyrlgAlmOxi7M/Em5zPGhH6OYuTnu5HD/VdlAWFspD7mSOOcydUoYZWEW1UGPKNSD+bybkzuEdaGYUDnFr8J4x8w0sJO4rNv9nOnwIAcJVE61mQI5FQuqaL6BSTXEs9kQLRbmErhNZEY84RIejFVjLX3uhsrqKj/99rYerW4CUDagyH0+gUG8fN06hQiUYHFQgtnpwXO2V8XmJXJXytHiAicfyjs1C1yDioxtnuQuOfc6nsEb4dr7RnXx6RH4zRKAMwp5ePjWiFLfOAsfiaxTE6BCu2gjbIlGWzcImzo8bfR1PO28v2wdZvu2kJF6IlPOCezSw+jE8/a3KUoGUXAuAgblZUqRdKnNP1IlKhyUdyLfUOZvK1aYnjJ+HvC0ErPmF9RsG3cGGXRKN8dY2XaepfPqpZg MwiuFqwu 0JbyTOquRCfgN8LPAnpzsxkrlR15C7MoWxrFKjcPnX2k34RI1rYHclO5NZ/yPr4AUn00tZa2ahE7Tz7NHlA8YXhaEGe2k7uYMQJbmNlhPgFf/k76+hXD/PnX2JCek4a5fGniHeYN7wrFYsGMQ+3U+lv2M3JuxjhCRnLNJrZ4Kf3zNwiQ9culhsrmr2m/HUsd0ii6vwo4WLmJf5jvPmZHssn/enGKw5mu2dFbk3mFjY091//u1kElLCRpZk9XHRAo1msNeofahVYujKQF9GwTmN3FET7Jb5TsiCft+tyl9ulbnRh5FxadGBX3cdNSFvnUVG8mzklPCzROqHbvXrgtobJ7k+MEKbtfUchBe37ArYnIxu59Y93XRLKFPConEW/nj43361sJenLfRglyv7Xl+4blT2KbgPGic5Tyi3RXfKh56d9K1N7lpRWU/K+Ci2lIPExWT4hE7/9MJhswV3UHZB0SyMPYxS7vCLF5f1jKnNxT6nFpLbYZVxm13h9L+/jBILXmgOZnHnFAjknUPTu4+VVSgA9NJJmMgCiXX8HM1EKBwweI0EmcpzMxThEK3W131uJDbEoqZqzSvmzjPB4rrpfdNeyHWZMAETHHp11+ycRvTPO5MPu1h+w4kpuTApHtWWUlEE9NAZ8EJGgtgxr+qaRFVBDtupU7qmEh4iLxUU46XA3pZVKGyG8P4ytcEEmhDnp2hKLOcfb4SQpM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move several vma_area_struct members which are rarely or never used during page fault handling into the last cacheline to better pack vm_area_struct. As a result vm_area_struct will fit into 3 as opposed to 4 cachelines. New typical vm_area_struct layout: struct vm_area_struct { union { struct { long unsigned int vm_start; /* 0 8 */ long unsigned int vm_end; /* 8 8 */ }; /* 0 16 */ freeptr_t vm_freeptr; /* 0 8 */ }; /* 0 16 */ struct mm_struct * vm_mm; /* 16 8 */ pgprot_t vm_page_prot; /* 24 8 */ union { const vm_flags_t vm_flags; /* 32 8 */ vm_flags_t __vm_flags; /* 32 8 */ }; /* 32 8 */ unsigned int vm_lock_seq; /* 40 4 */ /* XXX 4 bytes hole, try to pack */ struct list_head anon_vma_chain; /* 48 16 */ /* --- cacheline 1 boundary (64 bytes) --- */ struct anon_vma * anon_vma; /* 64 8 */ const struct vm_operations_struct * vm_ops; /* 72 8 */ long unsigned int vm_pgoff; /* 80 8 */ struct file * vm_file; /* 88 8 */ void * vm_private_data; /* 96 8 */ atomic_long_t swap_readahead_info; /* 104 8 */ struct mempolicy * vm_policy; /* 112 8 */ struct vma_numab_state * numab_state; /* 120 8 */ /* --- cacheline 2 boundary (128 bytes) --- */ refcount_t vm_refcnt (__aligned__(64)); /* 128 4 */ /* XXX 4 bytes hole, try to pack */ struct { struct rb_node rb (__aligned__(8)); /* 136 24 */ long unsigned int rb_subtree_last; /* 160 8 */ } __attribute__((__aligned__(8))) shared; /* 136 32 */ struct anon_vma_name * anon_name; /* 168 8 */ struct vm_userfaultfd_ctx vm_userfaultfd_ctx; /* 176 8 */ /* size: 192, cachelines: 3, members: 18 */ /* sum members: 176, holes: 2, sum holes: 8 */ /* padding: 8 */ /* forced alignments: 2, forced holes: 1, sum forced holes: 4 */ } __attribute__((__aligned__(64))); Memory consumption per 1000 VMAs becomes 48 pages: slabinfo after vm_area_struct changes: ... : ... vm_area_struct ... 192 42 2 : ... Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes --- Changes since v9 [1]: - Update vm_area_struct for tests, per Lorenzo Stoakes - Add Reviewed-by, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-13-surenb@google.com/ include/linux/mm_types.h | 38 +++++++++++++++----------------- tools/testing/vma/vma_internal.h | 37 +++++++++++++++---------------- 2 files changed, 36 insertions(+), 39 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 48ddfedfff83..63ab51699120 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -735,17 +735,6 @@ struct vm_area_struct { */ unsigned int vm_lock_seq; #endif - - /* - * For areas with an address space and backing store, - * linkage into the address_space->i_mmap interval tree. - * - */ - struct { - struct rb_node rb; - unsigned long rb_subtree_last; - } shared; - /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma * list, after a COW of one of the file pages. A MAP_SHARED vma @@ -765,14 +754,6 @@ struct vm_area_struct { struct file * vm_file; /* File we map to (can be NULL). */ void * vm_private_data; /* was vm_pte (shared mem) */ -#ifdef CONFIG_ANON_VMA_NAME - /* - * For private and shared anonymous mappings, a pointer to a null - * terminated string containing the name given to the vma, or NULL if - * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. - */ - struct anon_vma_name *anon_name; -#endif #ifdef CONFIG_SWAP atomic_long_t swap_readahead_info; #endif @@ -785,7 +766,6 @@ struct vm_area_struct { #ifdef CONFIG_NUMA_BALANCING struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif - struct vm_userfaultfd_ctx vm_userfaultfd_ctx; #ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ refcount_t vm_refcnt ____cacheline_aligned_in_smp; @@ -793,6 +773,24 @@ struct vm_area_struct { struct lockdep_map vmlock_dep_map; #endif #endif + /* + * For areas with an address space and backing store, + * linkage into the address_space->i_mmap interval tree. + * + */ + struct { + struct rb_node rb; + unsigned long rb_subtree_last; + } shared; +#ifdef CONFIG_ANON_VMA_NAME + /* + * For private and shared anonymous mappings, a pointer to a null + * terminated string containing the name given to the vma, or NULL if + * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. + */ + struct anon_vma_name *anon_name; +#endif + struct vm_userfaultfd_ctx vm_userfaultfd_ctx; } __randomize_layout; #ifdef CONFIG_NUMA diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index ba838097d3f6..b385170fbb8f 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -279,16 +279,6 @@ struct vm_area_struct { unsigned int vm_lock_seq; #endif - /* - * For areas with an address space and backing store, - * linkage into the address_space->i_mmap interval tree. - * - */ - struct { - struct rb_node rb; - unsigned long rb_subtree_last; - } shared; - /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma * list, after a COW of one of the file pages. A MAP_SHARED vma @@ -308,14 +298,6 @@ struct vm_area_struct { struct file * vm_file; /* File we map to (can be NULL). */ void * vm_private_data; /* was vm_pte (shared mem) */ -#ifdef CONFIG_ANON_VMA_NAME - /* - * For private and shared anonymous mappings, a pointer to a null - * terminated string containing the name given to the vma, or NULL if - * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. - */ - struct anon_vma_name *anon_name; -#endif #ifdef CONFIG_SWAP atomic_long_t swap_readahead_info; #endif @@ -328,11 +310,28 @@ struct vm_area_struct { #ifdef CONFIG_NUMA_BALANCING struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif - struct vm_userfaultfd_ctx vm_userfaultfd_ctx; #ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ refcount_t vm_refcnt; #endif + /* + * For areas with an address space and backing store, + * linkage into the address_space->i_mmap interval tree. + * + */ + struct { + struct rb_node rb; + unsigned long rb_subtree_last; + } shared; +#ifdef CONFIG_ANON_VMA_NAME + /* + * For private and shared anonymous mappings, a pointer to a null + * terminated string containing the name given to the vma, or NULL if + * unnamed. Serialized by mmap_lock. Use anon_vma_name to access. + */ + struct anon_vma_name *anon_name; +#endif + struct vm_userfaultfd_ctx vm_userfaultfd_ctx; } __randomize_layout; struct vm_fault {}; From patchwork Thu Feb 13 22:46:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49ACBC021A4 for ; Thu, 13 Feb 2025 22:47:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 403A7280011; Thu, 13 Feb 2025 17:47:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 38B8C280001; Thu, 13 Feb 2025 17:47:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B875280011; Thu, 13 Feb 2025 17:47:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EAB49280001 for ; Thu, 13 Feb 2025 17:47:30 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AD4EBA176E for ; Thu, 13 Feb 2025 22:47:30 +0000 (UTC) X-FDA: 83116409460.09.3340DB0 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf19.hostedemail.com (Postfix) with ESMTP id EFD871A0002 for ; Thu, 13 Feb 2025 22:47:28 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=iNl81Hg1; spf=pass (imf19.hostedemail.com: domain of 3f3auZwYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3f3auZwYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486849; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VVzZ14hy4ikDmyPBi6h+538M/HP0SirBYvfDsPDJpOs=; b=7BCF5WSfUvAYXfugWmVRzNqXDwqyL0HZfxXRwDG81Ze9VSF557+K/tBRrvHh1T1GvWQFzE YpZ7HeKUha3epdcNHko38I8mfMmZSUjJavjpWFVlU3cWn4kbEA2dl7FzPkUCGwsi1FNIWD jfz/sQJybHCw9TZHlpwFpRsrgvKFhD4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=iNl81Hg1; spf=pass (imf19.hostedemail.com: domain of 3f3auZwYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3f3auZwYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486849; a=rsa-sha256; cv=none; b=OI+/ayUOabClx01B8c0x5EmI+CHlUZJ+FdlUDxTJj+gehx2C17DO6fcTU12c3DRD4U+JCA 5ldXppnyfdntFpFdQdGmmk1pU+wfBjBohZaAMNNuIZjrep7kJolb88NkRuaQrDPjaXRS/5 CpaZHLJA2kfKW6Es1VquHk8US0fo46c= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-220d6018858so19090015ad.2 for ; Thu, 13 Feb 2025 14:47:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486848; x=1740091648; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VVzZ14hy4ikDmyPBi6h+538M/HP0SirBYvfDsPDJpOs=; b=iNl81Hg1965xtORwnyyChxiNuvf+SbcPekvW7BXSyMf14rcb6k3Q8xmw7GzVXBdO9B TQCp0Ye7ak/r2DKhzcCdvs833Ynrr8oyMIOTlyCpqBozFS9rQcS4n/nB+lx5Ru5zK+7p gOd8SVD2+HifhA4codC28dC68teAIgNlJwM98k5sENcfJYsUej3uNuMtKEsFAq0JT9a6 sWAfHAujaTbULeA2e+HF4H5bMfcRhcEWyeFV9gmxFstgLGV8XRn+CPYorwR3DHT5NYWt KK3QPPyBbov+mN389FcbeENhKW2N9bmLaoEKatHUHU470VjsbJ5AP3PyXnzmFhIlTV8Z 6QPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486848; x=1740091648; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VVzZ14hy4ikDmyPBi6h+538M/HP0SirBYvfDsPDJpOs=; b=UZUKyzq4gvhN1oUm+CogS02PDTmyCr0AXE1u/+SQf9DedtwYr498/Bf7NH7npMBhum dRMZjUbmS7s77c2pKI1eukMVsgYrKiueB7D7rS02V1ISMh+WyD5w60HDlRAKluxZgws8 YCxUiZxHrIlcpaJ6YLiWu/6GfEIz0lx2HYMaKhWS+82CHF8WIndfjupCgIMsiQ01uwAw X/3w1dFaH+v4nKe6TPGnQKq8RH4HJfOARoHs/OOvgzSEJj/7NLBmPM0N36Mdh5y+NOyy EVvhu7q7ioLgf6+iWU0jULjX2t5OLDLomLQkTeXG0ADpf1MpxBqOrl8k7C1EbKRRcMmF ZbUA== X-Forwarded-Encrypted: i=1; AJvYcCVUCNt6T9eBD/Q3wxSBlgbGc6XlarKga8I/nKP9FBotjQ+txfrntgJ3MHp2ITOJTpmK4H566YOxQw==@kvack.org X-Gm-Message-State: AOJu0YzsDM34g+bsKTO+1n1KoNI4zju1haz4QHsfsjhcUS7KSTQj1QpA mw4fyTBqrxUtVH2Ezl/gDXF09JV45ttZ2a//CjQCemNRKMyrYQlgvf6mswKspAL3l7eeQkTTGdV Hpw== X-Google-Smtp-Source: AGHT+IGwKOeZSc6Bvv2juhw3MUD/Tqd1UKmaxd3x+03s9Ch1BsTNeSistfaqs0trat+JLYOevAMQcp8yyOk= X-Received: from plhj11.prod.google.com ([2002:a17:903:24b:b0:220:d668:ff81]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:32cf:b0:215:a04a:89d5 with SMTP id d9443c01a7336-220d1eb5718mr75972105ad.2.1739486847961; Thu, 13 Feb 2025 14:47:27 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:51 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-15-surenb@google.com> Subject: [PATCH v10 14/18] mm/debug: print vm_refcnt state when dumping the vma From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: EFD871A0002 X-Stat-Signature: ojck6bqty5o7abbyu6kbpfrkyrcqqmoh X-HE-Tag: 1739486848-149574 X-HE-Meta: U2FsdGVkX1+UkHirSbO2rJ7PmRjXRWBgMUAYfj9l6Lo8LJrtr9f/SESmSvUj2DE37L21ekRkik5r242+kj553iKy0XFlb8QjZk8FOMIZgyk9A/dn/9h4R56zpSVj51RVkf+C4vZTeh0qeYABlPe9SzOMd+UY57Mc6eu7T5A75xq6wlQ+/py/Nl46DgFhgbIf15eCKL81LWfAdEEXecwOGX1zyMGIZt9SkGVYXWEcs+/HFO+O293Qq5vIiQxHYjCS6k2p4suz294mMJmrSmOukiHtXCPR/nUptfkDW/8SGk/OnLOBXz6wD3Am+G55q3gmwGcMgV4U+YEoZptpt7DEHmjoAixYJLMVM+E+3K0jZjMrYiux3cmfC1be/mtx36jJOL9Zfiz6hpEeqSoHHAjeDm60z85VIBgc0NFJ9mDXPVpJngQk1/sFzBFT1eVf4z94AVxsrd2DmgDzyHqMIdh2IecIAxPQ+20pkJNTwtu1a84diyYC3P5sVUNEs2/4mT0uD5C/1VvXYAF9u9MhwtN3EGUw8m4l4XGCswXD118LmqW9tkj2aZGfvXLBe9Lf9DaGA7iax0BE3G0VhFuQDEqqy37Am7ei36SAomQCxqHVtex9+x7Mn8PM/s6IULjG/qwG0zxPJTVEcavwydo2VXZklv8ickUmXo9eUOELPA+0Qo3ulgYD9pfCD43IZ7jiebMbw2MguB/HKjGESihQiL4cK+NzgLZkAwwFRij2cQ4Blsgi9t4cfBwIWVEa0wjRi+QYgVLPrh5ODE1y4gAHV/ZqS38po6j28Gn0Y10KmA0uyNEQglUZTs8VaFHUont40ej5OGxdbz64MLq9wMhKl3Rpc9sZR59QZ5c3ZJeNhM8mxnDfQdO2Ziwakjp0w0G/k60TmZ/KiJK50sfkHBVVeJZS8C6DLQusaI4UDoG5QwaIouhd79xliQjU9Kj0C8TmUUAXD9+3ZisklbvmZauoNJn hrX8rM9b jA8Sjgni+N4pWEl+/6PcsVkEE7l4jZn0tKE+EV7c9w9FQ9BCVrxqMhEJjQiiEAkpNhLPyldf7m4zbghPZ7s9zQBLg3DMlx6Ne2nEkfVA7Ja1Jo3JrR7FNkuuGTeHj8NMpw2ZFKRlp0gv6TXfbJRWRRU1zVJhOutr8TXdozbepHNFvy9hokcU3Nfod96mfpWU8xk4ej3norIM2e5soXHUCkjvdeTv9d0ACyQxWxAmnPY0mLtMEyATZDp24XeY41bBSWWcCpZ3hsjT+2qjSEDxlJScdo2i7rugIatdeKZ/rvV1s9yY4AflmG1UxbVJa/kDFDfhIFEtr5mBkmvn1mNkqKEpcyrf26BCsS39guU1jfc2/gyOI55s3kt7dZ5/ezglRytnho7JT+323lWT14eMDsUCB5sPY6nYNvBbEEVUPiMObIrcDZsx9UZRSnzzDBiyLVjPNsqJtgmXUEYnjHCOMW9IIZ306CPxrWSLSgVTbOHisjFtdaSbXz6yThL91s0ertnkuheBTrLjhQmw3Da3sXsUDdUPnveQDi/p4075lNy5H85/YOyIJbv3hooAmlUjZELwy6joTNFcXhwNCzSs5vXOFl/kb5DNB75+vn9+fXaUfY3qt4GIX8JGa1yBKNMER8MmQK9bA4KAIsXsQLVpj+lRjA1+MV1AhOJPr/0Pl50ZfGeDjadCk1tbpOA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vm_refcnt encodes a number of useful states: - whether vma is attached or detached - the number of current vma readers - presence of a vma writer Let's include it in the vma dump. Signed-off-by: Suren Baghdasaryan Acked-by: Vlastimil Babka --- Changes since v9 [1]: - Minimized duplicate code, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-14-surenb@google.com/ mm/debug.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/debug.c b/mm/debug.c index e1282b85a877..2d1bd67d957b 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -181,11 +181,17 @@ void dump_vma(const struct vm_area_struct *vma) pr_emerg("vma %px start %px end %px mm %px\n" "prot %lx anon_vma %px vm_ops %px\n" "pgoff %lx file %px private_data %px\n" +#ifdef CONFIG_PER_VMA_LOCK + "refcnt %x\n" +#endif "flags: %#lx(%pGv)\n", vma, (void *)vma->vm_start, (void *)vma->vm_end, vma->vm_mm, (unsigned long)pgprot_val(vma->vm_page_prot), vma->anon_vma, vma->vm_ops, vma->vm_pgoff, vma->vm_file, vma->vm_private_data, +#ifdef CONFIG_PER_VMA_LOCK + refcount_read(&vma->vm_refcnt), +#endif vma->vm_flags, &vma->vm_flags); } EXPORT_SYMBOL(dump_vma); From patchwork Thu Feb 13 22:46:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EE8FC021A4 for ; Thu, 13 Feb 2025 22:47:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C676280012; Thu, 13 Feb 2025 17:47:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 64D11280001; Thu, 13 Feb 2025 17:47:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45063280012; Thu, 13 Feb 2025 17:47:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1BD06280001 for ; Thu, 13 Feb 2025 17:47:33 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CB2AD1A1713 for ; Thu, 13 Feb 2025 22:47:32 +0000 (UTC) X-FDA: 83116409544.12.3FF2455 Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf14.hostedemail.com (Postfix) with ESMTP id 118EC100002 for ; Thu, 13 Feb 2025 22:47:30 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BPARf+lU; spf=pass (imf14.hostedemail.com: domain of 3gnauZwYKCG0dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3gnauZwYKCG0dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486851; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pXllq5kyB/DC+XX0EOcvwmmu4CtVKAeaHuoDmEpYjhg=; b=VFjIkPgysXAnHR17TzsbBwr8ktu+qy63i+V6Hwu622sPfF1d+BB7DQySRMqFzHJBlHB4Jo IuITTQuztQ5rfGC1/EiT4g66dl+WSYvrVCNwDtYeCKWkXhnBr611Vihy60/1z0rsCJXH30 QMjJcVMvxXWweLSpEu+2yaUjX/c+cJc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BPARf+lU; spf=pass (imf14.hostedemail.com: domain of 3gnauZwYKCG0dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=3gnauZwYKCG0dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486851; a=rsa-sha256; cv=none; b=taDTGuVcTVeEZrFxebXlhnnDAi4R9aWgxfSXHEQXcHRj4kBREnJwqvcomnmSVAYQ/jxQ4y SbrXnfvKwmqw08kJZrpI6p9yfat1RDSXAq5fJeCzNjwbK5+CPghi5yYu1kc/LasevVVP9d dk/ufes55/4n4w21ImbK881zzQkWA3I= Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-29e8124e922so2588690fac.2 for ; Thu, 13 Feb 2025 14:47:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486850; x=1740091650; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pXllq5kyB/DC+XX0EOcvwmmu4CtVKAeaHuoDmEpYjhg=; b=BPARf+lUCpXUqtRf36sbCms6+I2BUb7I1+M8taUqew/rR7Y8vMPrmeTj9SZDp6JPaQ furQIkVPeeNC9+v//xcMHHcOiEaMjmBNi52+TsQLU39E1+Le4esxTonJY6gJ2Xi7iuI4 WQ3xWr2SPwy/+uZ1tDgRNF6IdbBYAaRsX6V16boyiS5ngWvZopFap/nMK76Y+5pgdn00 TnETsahkexKqJ6I/gF70WULTaI8ljDPEQlx+Dju7kqT6g9PQ66H1CvRp8xQC8gUgaK9e T2udPmhKzgfqxOtP3uS3jlcwdKdB1kZ+3UFjwWxE6UGG2gC2bAyScVvm74HEoP5TNmNK kVPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486850; x=1740091650; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pXllq5kyB/DC+XX0EOcvwmmu4CtVKAeaHuoDmEpYjhg=; b=ZCgJ6KbWNRkRV3jUasj+On3/Ey9Ex/EhzKiAxLDfJ9oImZ3C/1x+ObYspjlqXU+c6y 5wqszOTcyOQ96grmw/EavWf9pCYu1Lkw+XIhByG6qsp4U6SYSUfmOZyp0DExRPrsP9Bb 6aN8eD8j1xAnGQsHuQSruFNmUlyeiqjNZsQqddG3KenFXnlBcdH8sevAKTUvdXT2yv8Z iA96XSxYo9wo56RE+cy8dlmpQZvWxBFhhm5lX2KyyGA0syaILgfBWTf52T2mU2G1DIN2 Y2PSEXlTXCxoGj7bTZ7nmac/z84l3k/rb+0jmOhx+KYvJbIXyqNxjJDQa//dyRwmvsvV epFw== X-Forwarded-Encrypted: i=1; AJvYcCXl/1kumHIOWWD53WouQRPO56Vb4r/QFnkAGpozITmPFeqv4xHRifNoHxZw2XC6XQFM0RoObbZzfQ==@kvack.org X-Gm-Message-State: AOJu0YylFgH3iWvRA93q8+kjeXiVf7BKYaqNeVEW6wizKAxt5LTzh6+w eb8CIFjl8z3es8xcHwOWVzG5CwJpnA3F7trgczz1QaZP3c5JuVxKl+50GTawvRnDmgytOeiXnbp XAQ== X-Google-Smtp-Source: AGHT+IE3Y43bKeMlQ9EjFiiS4JaeOx4TU+qb97Ij8D1UDPSI8dMXOEq3dWAIQbnurpdiuJXU+uwYfxTupKQ= X-Received: from oacmp6.prod.google.com ([2002:a05:6871:3286:b0:2b8:49d8:2c77]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6870:2107:b0:2b8:306f:c5ad with SMTP id 586e51a60fabf-2b8d65155a8mr5274258fac.13.1739486850086; Thu, 13 Feb 2025 14:47:30 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:52 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-16-surenb@google.com> Subject: [PATCH v10 15/18] mm: remove extra vma_numab_state_init() call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 118EC100002 X-Stat-Signature: ht8drytatm18ztbwhbi8fpcoo1xgihdp X-HE-Tag: 1739486850-890826 X-HE-Meta: U2FsdGVkX18NwW9oxCKkWp0L2c9jJvQgQyPx2C2K5pWSTRzJsX4QsfceqgS+1VumsCoWm7ODP7HWkvgVuyphwipXyNF/D2QmQ47Hob+O/Zkzdyk+ngr7j6x/TUVD5Bn8hTDQYmUhF+cpGMzQlRR7RRyAFgWBO/7VBOmBEIumu+1e6UzwK3TpNjupvwr72fVxJdFG4r9DuKoECvCEU9awXGM5avFwHjpSmOJwg2kGgYnLeQ61hUKO2JpvQTQFUz1USmAxb+W9zH6TqRX6BnLNRhpbAcOjuhk4Z/Kxf6UN4H+Z/AQwKmaqGuoSV9QveAbIGayjfnxQnWQfoemKlVwfV3QzUScWagiBXjGPtHg8NDlTsDt6sQfxg/Ki/lRFX1Uj9SU75mbZSqpDD0fDfMeOYPb8hG2gNzekVJDywOb4HuGzy22yLqr1iHJn72a3wwDsKfgdP+lqaG89AaTrRa3q0WIk4TPeOMhA3E72ZGZYIAU4AmezwKdbKIiVteDeFdBDf21e2e7GC20o497K0G9PRZOCIaRpX6hq/ZAowkA3Ooe6+k/hw/q4HBLvzngFuJBDKjFwY5ThbtR2XP5qJjPpAD4BLgfhqW+XjRj+bUdYlE9EIjMyaNOrQGdo5LKwG0xocDTZS1Oqut24DnxNgBPrgUafF9j2hMQzwSNGn6T/9/n++wahkxgkeJW4/7otAeOrv8j+ufvUV1YGxefx3FFA/fHnSC9iboYCn0ecXrJlQOE58FnWe7XBMP3iiHyx17FIEIeXgGzpSFSzsuhj+pHcL1MgYVN/YiOazrfcgBLiPveSZwnpOCu+WlTL0+8zv6ZnseCxasN3vX7TjHzRzjq4Ju99XdJWpYYhVHhGmsTZyLnnGRo+kjxXQPu9MB7Oyof6AUc/ce2ETCrEIoU7KUlvxBZLaHMdV0gRNQMswWoAGKa9vrkvZH7ZxwPNE0W4IUhygFMT+Fgkd9eHzfsdBVk W/GDJ4ke NFxwbETP7bFl3tAUMgLRLXuGS0IFwn1pL/osdoRrf66rQwU0aivtn/swPsM7yqWBehtd7S+MBfUnYsqE2dxIVH3qkBiMxq1Wh4nvcHQNQH9FoVpaoKFuXBuj7sHKIbujpePDAeJeXqzsfYXX6Oe+AkQBK8W+z71FCZUuV7YSW1BXb4WPaNEgsoFjAgQMYLVaMTovmETdwkv5Mtab4RrS2NZDSdZm7/NIPwIretxnSKNJyyWQmlqQ4OuvLv89fobZ6CKU8v6KxEYtDXxlpEhfIlKgwNh98c76GmLQ2AldjHLTf1ieKAl6BvnZ2BPbAFMyWu6b5rJhiYIaoXOyjxN/Alwn/DenMqk1PsLXyIUHVOUiPzKh7yzaRbNNFYMsPejWjVCUo+Acg+PIlWJg+sY18graOLW/quw5N5Myo6+GC5cdiLDb2k67rasIx9mGCzIa2yT7/PysdVoaX907i+7twx0GrXaq8vKstPTuHrwj9Q0g3OSlmJELHHL/3mJjJJ7DuiZck2TVKJWJZNtYUKk+pFjk2IxBKdoxaKEHRhzXEUgv8h7JsHvGPg4HZ0OW3kVhznvY5aQmw3b1IOcqZkpY7Henp+nirrjK0qkGx/QZ7Uit5tLfAcpHK4BrFYSTToc4VuWVSJKvL2+zhA1b3HhnCrv2UiNRVvZW1COODauYxiR0wzbmXv3qR7NKUbuyEOJ1pQw4Kj7/a2P6HLR8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.004113, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vma_init() already memset's the whole vm_area_struct to 0, so there is no need to an additional vma_numab_state_init(). Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka Reviewed-by: Lorenzo Stoakes --- Changes since v9 [1]: - Add Reviewed-by, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-15-surenb@google.com/ include/linux/mm.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 11a042c27aee..327cf5944569 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -956,7 +956,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_numab_state_init(vma); vma_lock_init(vma, false); } From patchwork Thu Feb 13 22:46:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1010C021A0 for ; Thu, 13 Feb 2025 22:47:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87045280013; Thu, 13 Feb 2025 17:47:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FA6B280001; Thu, 13 Feb 2025 17:47:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6247E280013; Thu, 13 Feb 2025 17:47:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3D6BA280001 for ; Thu, 13 Feb 2025 17:47:35 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EE6E2C175C for ; Thu, 13 Feb 2025 22:47:34 +0000 (UTC) X-FDA: 83116409628.04.17C8324 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf06.hostedemail.com (Postfix) with ESMTP id 2D1D0180005 for ; Thu, 13 Feb 2025 22:47:32 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=GeWRQZFp; spf=pass (imf06.hostedemail.com: domain of 3hHauZwYKCG8fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3hHauZwYKCG8fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486853; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3PBMMH2moGlibOV+gRj99WpX+TZ+10z7I1DXCC5PdQ8=; b=NdqSD39j0HZnhtaKJRzMaBxnJojUt7LdNuAITpTIVoY2FqQUPJpiwxvR2evLnhga4/evMm qlycCKaATiBZOB3DgkrwGWzekF+YGHoFQfkmCQ0qIbpv7rw9HCWSRY4TwclnFI25n93/B0 Z31F0tiMy8WZkv74Td3DTC26CbmFW08= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486853; a=rsa-sha256; cv=none; b=H8nZDN6VV15Hjjw4JDrsbov5gUTQGjiW4ugSVTh8LTriYJSG3wDlHrhJdKOQ5mIIEyg/oY llCfPFufLYGIfQF3qRfGD2NW8P8/6ohFusVKYOzmDk2zcfLWtrpvwzzZm0mWkYfAC3530n s5gBfMommBwWyooUsIDImv7ZTYTQYLo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=GeWRQZFp; spf=pass (imf06.hostedemail.com: domain of 3hHauZwYKCG8fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3hHauZwYKCG8fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-21f444af89fso20256805ad.1 for ; Thu, 13 Feb 2025 14:47:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486852; x=1740091652; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3PBMMH2moGlibOV+gRj99WpX+TZ+10z7I1DXCC5PdQ8=; b=GeWRQZFpCoK5aS7mEgCQNryM15hsy8FcBc4xoOkWqmdRfNmPlA1x9KjbTBgAAFpBs6 1Bb/awZSk8WmmE83X/3PmIIMfKtn4lO7krod9BB9mEy8lW2o2SquPHP39B6sS1EHrzaI UvUsmZHrNPd3gtTrCOL3GHfmrekmqHqBU4Jrldiqi/pwiqOXUkG8mWSsnMuSxpfThgW7 BD1Fk9szFdd5znMU+yAk9QALKoKaFoDp2CrHiaouakJYrbYpyaxpaJibQTKENAigSwUo fwQwHLGkfoH8ms08P7IY2/U4K5wnkSvrNWPfLZ0iX8i8AfN6VyY8QCEsN6JEmmA4WEjf OP6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486852; x=1740091652; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3PBMMH2moGlibOV+gRj99WpX+TZ+10z7I1DXCC5PdQ8=; b=WX2YFHISPtZY27EejFn1rfkWuvAo6kYuzVUSyuWOgXe2Puw0e1Tv3PXmYJ63gtti6X 3hmEd8vOW7UjiJGwwz/L9TKrzbxLhTqQ5paACPtGxsuCQOWLgJStdQsFbHaNKy8pIS2O gKg68NzmbeNr5VC2oJUlIJ3Wn7rz17PUhYnlDskc1t3yEL4ynADq/PiRcB4d9MI8z+y+ Z2lMT5AOqwk0Vi2bceIrzTqINI+CQlfokkWl//r7q4a8ydkmGhydPdfjcE2aN73I6nKS UEu0vMPMYXw21iDUZWiiaUpMYCXr5hMfc3UsLvzwDihXsg6thMNAK8nt9eodnBPqxnox mbDw== X-Forwarded-Encrypted: i=1; AJvYcCVIdmzjyeXZEQs2NopRchAVSucefUN8foWvIISOC3DcXAk0C6PmZAkjHMSavqBdA3W0qzJYw6WBwA==@kvack.org X-Gm-Message-State: AOJu0YyNdE6idv1xVtO/Yiuh+7OIuVerK3TqCUoWPE0JiTqt3XUaesm/ yX9PwUHRf6iFsppQMl77J9ddravGQuHO27cKNLsaA4RfUxolYOsGbNwu5IIpjLUQl/KxoATn627 PCw== X-Google-Smtp-Source: AGHT+IGMj8N9UltRndmqK4fcmCOY71dV/ZTT0TWqfMv3mL8UIMVUEmmB9CXosWINwjoG+HVa2iESOUWrDew= X-Received: from pfblc21.prod.google.com ([2002:a05:6a00:4f55:b0:730:9378:98c1]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:2585:b0:1ea:e81c:60fa with SMTP id adf61e73a8af0-1ee5c78b20bmr13938165637.20.1739486852080; Thu, 13 Feb 2025 14:47:32 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:53 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-17-surenb@google.com> Subject: [PATCH v10 16/18] mm: prepare lock_vma_under_rcu() for vma reuse possibility From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 2D1D0180005 X-Rspamd-Server: rspam07 X-Stat-Signature: id89q9as4nawkxbjbopmuhdraojrjy14 X-HE-Tag: 1739486852-788487 X-HE-Meta: U2FsdGVkX18FUN1sY/nQPW1GxOwUDAUJb00/AZ7Sski/r9++P4hyLMa0jhwjRP3p0mkaoglODnoAquz+uh6x9NaYAVxXYFIJiHBmsGsjq6fgjCgTdLoB/JrqQ3u8dV27Uq9xn28u9CSEKOmh3/+rfdBnSWdkH2YjA+27+2cP9codsDbSvHXsgq5MOipdtDfiWMpcNZfh0lLxWFNhn69Bw5jE/26Yd8n44/Q+8MA04PHiAOMH/UlRbOS05JV1inWuHBb8gf1+iIDgzFN9P+7YU+tvnSj8Zd0ZFHnQ+YDVbkg+BLxSNv0OsU4qEQNeRQHb9rsOxSIUWeqe+4sTbi9GjOmtHVsTuigteuyvU5xtTw8+hm3n9WAhQAuahI9YynbGdYDZhuSJG8KdpaLDbNEbBP+qWZvJuWu43Vz8uW+j7GAzOSpDZvd9VOdHSdOQxEre3evDXtpLShe11KtZXWMAvkANNwhJEsQfCzh8z25J+mFM1+FEzJpibswVx2Gcc5B8WGwwKXO71A/x/939BKE0fRajwvqrVQ9DShat8dzWtrVeRmU6HASfik52ai7TJQunWt0LW6nHCT5Bt7qZF4yZQoYhwh+INKfDAbhXSojqarXNtsiuMeyA5+i1lctGxYDLXxrdpRucwK9UquYWDNs78DdEeF5Nf+KaBfK5gxW2Fk/0k81Ewcy/+tE351TlHlelhb/uzXmMBm3qBYPResyY1hpelL+r8xpFtZPqcTBgpWj0Yh/0lIcc6ap9FOuTuUzLC+BtiAeFI+be741cotNUvoOVlHP4EGj4FP0mi/ggQ7aTNyHsekjaTNfSEBxbvA+0dteeC/mQmQ1tVYcJzYKuoDGo3NEK9nIwWNH1kK2yXGzYfH/kc5O8tytKwzPi78vwKcqm7Ffk2L+FnZZzrWXyFyx/UYYypHkN1UJK6KMT1yi4VdUvNBJK/uumUjqa42SmbBTS1kfIlkYFm3agrIh KcBetd0i z5Okwq6u1UQNIS116kj8/BAp9yVjAPffGvsXVw8yrbDUnkLJ8Z29x2ytRsUPD15/cMUpXS+I4Rdl0R7N0uZcuHY1V7B3TUjSj2XtZYIK9WCUaMbW8xS4DpRADMO+Xt4nizbbKAiH5jmULkXnszzH7wXQKvW0OCLDKGZ9aAhk7tZPohBHwb6O2jNJSWgH3Zw6M3YoLsCYYxELilI0fkRwT8dl8PIhjRNkWMecaEsbzfSu5/9BU12E5FRWFI/KzMZHViqY40umICFNEYymLpz6byvqRPdYOPU3k6WTaRAgixA3j4E9lYrGLvBsHh51I36tL4Wf8NvoCyaOa8U1Nqx57569rS0O8KQ1FuDtDreY8M8ywJI+2ySA+Wamre2qDXs0eyR+Kuja9iH6H1yYjYzpI6DwaQTHnQ4z0jPFx5gESRKsDVs448jZCWIcMYqs4IVGzT5MEB+kF9jf06PHG1XGTb272mIJu/BsZoUaYNq3BHoInc8AFxcUjaava7WBaH3+FbbhxtTULyB1tEmDIcP69ooeAhYmgdQSmkrERgwaI5Iv8KOqmbo9NKjEl1F0pNJU1VBiBBhjuWfXr2Zl43hFC5MUxhfeyQIfKdIoLkpb4wmSrwmVUlqx7aWDlLIlR16M8BFuCmauzcINDeGFbqmOlRZtocg3vKMGeBYuQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Once we make vma cache SLAB_TYPESAFE_BY_RCU, it will be possible for a vma to be reused and attached to another mm after lock_vma_under_rcu() locks the vma. lock_vma_under_rcu() should ensure that vma_start_read() is using the original mm and after locking the vma it should ensure that vma->vm_mm has not changed from under us. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 12 ++++++++---- mm/memory.c | 7 ++++--- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 327cf5944569..88693568c9ef 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -739,10 +739,13 @@ static inline void vma_refcount_put(struct vm_area_struct *vma) * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. + * False locked result is possible if mm_lock_seq overflows or if vma gets + * reused and attached to a different mm before we lock it. * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got * detached. */ -static inline struct vm_area_struct *vma_start_read(struct vm_area_struct *vma) +static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, + struct vm_area_struct *vma) { int oldcnt; @@ -753,7 +756,7 @@ static inline struct vm_area_struct *vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return NULL; /* @@ -780,7 +783,7 @@ static inline struct vm_area_struct *vma_start_read(struct vm_area_struct *vma) * after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { vma_refcount_put(vma); return NULL; } @@ -914,7 +917,8 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) {} -static inline struct vm_area_struct *vma_start_read(struct vm_area_struct *vma) +static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, + struct vm_area_struct *vma) { return NULL; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 528407c0d7cf..6378a873e7c1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6492,7 +6492,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - vma = vma_start_read(vma); + vma = vma_start_read(mm, vma); if (IS_ERR_OR_NULL(vma)) { /* Check if the VMA got isolated after we found it */ if (PTR_ERR(vma) == -EAGAIN) { @@ -6512,8 +6512,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check if the vma we locked is the right one. */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock(); From patchwork Thu Feb 13 22:46:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D033C021A0 for ; Thu, 13 Feb 2025 22:47:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FBAC280014; Thu, 13 Feb 2025 17:47:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78611280001; Thu, 13 Feb 2025 17:47:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58A42280014; Thu, 13 Feb 2025 17:47:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 305BF280001 for ; Thu, 13 Feb 2025 17:47:37 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D60351C9413 for ; Thu, 13 Feb 2025 22:47:36 +0000 (UTC) X-FDA: 83116409712.05.A241BC2 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 1277340005 for ; Thu, 13 Feb 2025 22:47:34 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lPh74Lo6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3hXauZwYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3hXauZwYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486855; a=rsa-sha256; cv=none; b=fZ0k9tsFlGIlHeq6zKpLgSVFabjWy3zXM0y5MpOSAxEV4c+FRxVCFqdVQkzP/GUSdOHIIq VJ+rtrT++WrnZzKwboxfNXli5eHQRw0gkngIMxhWpc0vuRNnRGNyXlytFthY0OQTE8wSkl +B8Sc5Hd5PbwDMsmf2TypMfq8sgi00E= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lPh74Lo6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3hXauZwYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3hXauZwYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486855; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4MXvj5u5NFB0K8/f5IiPMekR+5md5clULG3qbt/TlOg=; b=qs4a8LcjG4oguWJwMnYDg5zYG7uvTTfOYhcXsdCK/2DZTfCAjQDmfKd6ziVvzMD20zUrZR 8yN03k57vxxCBcKJpnaMmiGnfx/Z2VCdGOzVN2ny/BniUJRH+8hFJvGDGmMCaU8Ji64oZg yAxIqGMUXhQ/fsqObNDRulrL5JyKZec= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-220ee2e7746so7885ad.2 for ; Thu, 13 Feb 2025 14:47:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486854; x=1740091654; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4MXvj5u5NFB0K8/f5IiPMekR+5md5clULG3qbt/TlOg=; b=lPh74Lo6JEYIjSimpgVAHBSqxJm8OBS8MmTbtl+BSimrJ9dMVYfxjHRzBwgdY9Cvag jnLk4+MRMmbmPErk07NRiUxxox+yNuubQjPv8p00fPBGZ+MYlihVLsYVckjpJaH1xM5o jcRvNcqMqLF6SO8z7YW6VVOW9MkXz6Wv0i4eV7AJTWHgr61sorWjkzwjZRwfWDr4tk05 QjOxRqOOM4GBB5+zqeOUgKpviq7yqi343TNkuyfAWCy5uCksRxHGBdFBKg3LA3MHLlEh 8rkRzePRw2GGwDZZKJpsQXlN16wRfwvkyEFTKlMw4L08CZ7nlVxl55APdol3UQSVGzZV Q/Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486854; x=1740091654; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4MXvj5u5NFB0K8/f5IiPMekR+5md5clULG3qbt/TlOg=; b=p6jJZoc/rWkC6EdxWWGo9y33e1lMGHo0QRA56l2EjVRkMykhMjFUgmOtcFV2QNkWuY LBIQOu7uqmqFPBi8LBgurpRzPrENRhYddEqAyxUnRMKLLsnV1SiWzDcI1bgehE0QZlq5 9elH+0yMFJRYx35jm6enUHYA8qzvz6C145Etk0Zn07LeO11X/7JlQCNtWR3lfXvev9zI tT05Z48BapafgzYRlETLxnwJq66C295BLzYZ7J1VhYrjAYW+jNpaFvTFi4D+1V+op2dr vMDtUilM7xinMTdYe/zvv8jjIHWMm+lUUVncCWzp3c1Sg/+ERLK/g3DjneF6U6doMTgX C8kw== X-Forwarded-Encrypted: i=1; AJvYcCUjZm+0MLhgVkAqBV0snFfuVftcuSzO5aKNW1I9RusTi6bClTvxYxqgTsFsla6YFyBs+I8rOW2awQ==@kvack.org X-Gm-Message-State: AOJu0YwQpi66eFBNqlXBTkLKFLqdgpO9tJe6CdDqYPXMuNALVj5kZS+D I1LmLlkmvDtAxw/PTlz42QUZfSJMwWNdtGSZ/CL7f9IYNOTnilvWG+Zf7wwXWAquen6k12lH9cq dnw== X-Google-Smtp-Source: AGHT+IFObHFHIWBZFemkWn8ckufyLtHvFFGrsnemtMs+Zktx9OES6MSNojKQ+ZzlKftfhzzQkT2kzbGTjvc= X-Received: from pgid14.prod.google.com ([2002:a63:ed0e:0:b0:801:e378:a64a]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:d4d2:b0:21f:1ae1:dd26 with SMTP id d9443c01a7336-220bbcd0acbmr133795865ad.52.1739486853879; Thu, 13 Feb 2025 14:47:33 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:54 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-18-surenb@google.com> Subject: [PATCH v10 17/18] mm: make vma cache SLAB_TYPESAFE_BY_RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1277340005 X-Stat-Signature: houfjea5kxohwbx89taamfjpm7ewgu7s X-Rspam-User: X-HE-Tag: 1739486854-343205 X-HE-Meta: U2FsdGVkX18/G0g2BFJdQ/xZ+ZpTU0ag2X+bRekh4+uVskWY00/OWT8gNkx/7vewYwDrb4LO2V1VxqmgMxdk/KcUZv8dAUNqug/+spN4PWnhINVAOG6rAzRbpxM7yuPzL5ISOAGlJKC/X1NagrYd9LvIu5ggCsj6OYah/knB9OEPhnK5MF1RqL/496Ebow2jsI94UL97+3ddQWGmMAwH6keuUXdMT35tCV1S8MdFaYYD7s8Xh82ow0rxwjpirZaMdGYZ7g4PlTw1YY6Gnt5aoyQdx+zvuKM6dPd0ewEu6/m9VXdMntQdCXbrSyha2uou0cP6VmvNaz+j60/KioCGQQZfu5dLRBACog7IMyJaBNSf+4lNBtaFnLV62m96T4Wqb5vtRuejTXUZ/ASwx4KjmLA6GcTVaOEJnJ3siMFMQzj5n5g8ltpkbuI/YnW9j2oNnJhfBwWywiLUaA8ZjSUXwX9sl5iUb7z4Y6Q0XNrkTriPuQGjvXLn/CtvaZ4/ML/KpyFflSVAzh7BqgUFTt3NzwyXae+9qWGRbdrvcKGNz1GUxL7zHBuTiMykbOFzfB8ypjzD9N0G3Op9KoNZSaQFwal71XDSwiE801CMqahQcu/WDaTykrQwAUj+ofF0YrbuGhTRpsCeYTJmME5mp+2+PE1kG04sawW7ueL0xq6bXv+Jzcd8vB6tD81vrkpYcWJ4ErQ/ziReBkjgn+qliJBlSIShOri7gu/ZFgTVO3Px9x6bGsxFd5EbZpfjj7/mD0jkX8m+O2Set0U+QY5iltDbr5D+UdJmTV5pvFCUbvQlpe3pyvtkPivl+NygJLHsF53KFyhPGgmnvuAV4AIKkjMoGwiBW1gd/s/aZ44+s9lQijEVYxtFW+IOVGVb1iMzNpFb97oTbSsQu75qsD1ekS6pyhVYOysO27ZbqBUKTDx6C0guJFCLCQUyP5PNFPR4KBo5WuUxHsIFDwWphrHFI+W OBTv8iYt 7XORvQAGGipDrwO5GYniZQ3eDG1CPKDfsw3jcV/toUfGDABKoEgmOSiKLtXctpF0Mp9SWHcFmoXxNRy8HwRochdEOde4QkCqlPVIRy3/Em4Lkj2yj1tcECw+z3rNAQArq4wqUbxvfTBjq6GylIqxhvDB4Gz+hDRTk2SnQyvHocKqmL5YWE8WtSqkLK1jhiUdYzKsxWUoKUBBiuJh5tXon+9d7d8jAP3f0Ep95kW7zVl660JC6Bq4sofodb+VEIgXud7kvvMiD5b8DxgWbNyc0cLcuMJ4NsJXLIg5DM8doIOIjOZ67h8lAJL15ehcOKe0rtjJTSWTC8JIDYwPFe4nuMarIsS15Qh/aCm8mpdgbz3kwLhU3PxgoFiVT73MbtzsMjJ5Kf1WBmk6MchirK8XSeB5x56UFEbAoBTil3uQDT77CBS+nwOP9Z9rUSVRj+pIZUeZA9O26lBG78OTUAK7srmUGjngoEOVAoWGbWzQi72fyEeZIJk2Oqazw/b1YEF446AYvvMMBo+NexxwpmJzVhaUkIlNjuEsYUPr9kByV3Ikrpgk1R4p5dcv0bj27S8UYhabWAqGtCKrDdqSzbBBGB7BNsxmK+gb3VGzmjv34DlTP3JVBOxUONYBoJL8asFNsHWbTZrs3vzM3BGQGvCTIEPhua8c5gU6Pn6/8CF65tEsRUod023PsU9zZcg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected by lock_vma_under_rcu(). Current checks are sufficient as long as vma is detached before it is freed. The only place this is not currently happening is in exit_mmap(). Add the missing vma_mark_detached() in exit_mmap(). Another issue which might trick lock_vma_under_rcu() during vma reuse is vm_area_dup(), which copies the entire content of the vma into a new one, overriding new vma's vm_refcnt and temporarily making it appear as attached. This might trick a racing lock_vma_under_rcu() to operate on a reused vma if it found the vma before it got reused. To prevent this situation, we should ensure that vm_refcnt stays at detached state (0) when it is copied and advances to attached state only after it is added into the vma tree. Introduce vm_area_init_from() which preserves new vma's vm_refcnt and use it in vm_area_dup(). Since all vmas are in detached state with no current readers when they are freed, lock_vma_under_rcu() will not be able to take vm_refcnt after vma got detached even if vma is reused. vma_mark_attached() in modified to include a release fence to ensure all stores to the vma happen before vm_refcnt gets initialized. Finally, make vm_area_cachep SLAB_TYPESAFE_BY_RCU. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- Changes since v9 [1]: - Use refcount_set_release() in vma_mark_attached(), per Will Deacon [1] https://lore.kernel.org/all/20250111042604.3230628-17-surenb@google.com/ include/linux/mm.h | 4 +- include/linux/mm_types.h | 13 ++++-- include/linux/slab.h | 6 --- kernel/fork.c | 73 ++++++++++++++++++++------------ mm/mmap.c | 3 +- mm/vma.c | 11 ++--- mm/vma.h | 2 +- tools/include/linux/refcount.h | 5 +++ tools/testing/vma/linux/atomic.h | 1 + tools/testing/vma/vma_internal.h | 9 +--- 10 files changed, 71 insertions(+), 56 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 88693568c9ef..7b21b48627b0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -258,8 +258,6 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); -/* Use only if VMA has no other users */ -void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; @@ -890,7 +888,7 @@ static inline void vma_mark_attached(struct vm_area_struct *vma) { vma_assert_write_locked(vma); vma_assert_detached(vma); - refcount_set(&vma->vm_refcnt, 1); + refcount_set_release(&vma->vm_refcnt, 1); } void vma_mark_detached(struct vm_area_struct *vma); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 63ab51699120..689b2a746189 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -584,6 +584,12 @@ static inline void *folio_get_private(struct folio *folio) typedef unsigned long vm_flags_t; +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * A region containing a mapping of a non-memory backed file under NOMMU * conditions. These are held in a global tree and are pinned by the VMAs that @@ -687,6 +693,9 @@ struct vma_numab_state { * * Only explicitly marked struct members may be accessed by RCU readers before * getting a stable reference. + * + * WARNING: when adding new members, please update vm_area_init_from() to copy + * them during vm_area_struct content duplication. */ struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ @@ -697,9 +706,7 @@ struct vm_area_struct { unsigned long vm_start; unsigned long vm_end; }; -#ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ -#endif + freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */ }; /* diff --git a/include/linux/slab.h b/include/linux/slab.h index ad902a2d692b..f8924fd6ea26 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -243,12 +243,6 @@ enum _slab_flag_bits { #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED #endif -/* - * freeptr_t represents a SLUB freelist pointer, which might be encoded - * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. - */ -typedef struct { unsigned long v; } freeptr_t; - /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/kernel/fork.c b/kernel/fork.c index 48a0038f606f..364b2d4fd3ef 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -449,6 +449,42 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return vma; } +static void vm_area_init_from(const struct vm_area_struct *src, + struct vm_area_struct *dest) +{ + dest->vm_mm = src->vm_mm; + dest->vm_ops = src->vm_ops; + dest->vm_start = src->vm_start; + dest->vm_end = src->vm_end; + dest->anon_vma = src->anon_vma; + dest->vm_pgoff = src->vm_pgoff; + dest->vm_file = src->vm_file; + dest->vm_private_data = src->vm_private_data; + vm_flags_init(dest, src->vm_flags); + memcpy(&dest->vm_page_prot, &src->vm_page_prot, + sizeof(dest->vm_page_prot)); + /* + * src->shared.rb may be modified concurrently when called from + * dup_mmap(), but the clone will reinitialize it. + */ + data_race(memcpy(&dest->shared, &src->shared, sizeof(dest->shared))); + memcpy(&dest->vm_userfaultfd_ctx, &src->vm_userfaultfd_ctx, + sizeof(dest->vm_userfaultfd_ctx)); +#ifdef CONFIG_ANON_VMA_NAME + dest->anon_name = src->anon_name; +#endif +#ifdef CONFIG_SWAP + memcpy(&dest->swap_readahead_info, &src->swap_readahead_info, + sizeof(dest->swap_readahead_info)); +#endif +#ifndef CONFIG_MMU + dest->vm_region = src->vm_region; +#endif +#ifdef CONFIG_NUMA + dest->vm_policy = src->vm_policy; +#endif +} + struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) { struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); @@ -458,11 +494,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - data_race(memcpy(new, orig, sizeof(*new))); + vm_area_init_from(orig, new); vma_lock_init(new, true); INIT_LIST_HEAD(&new->anon_vma_chain); vma_numab_state_init(new); @@ -471,7 +503,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void __vm_area_free(struct vm_area_struct *vma) +void vm_area_free(struct vm_area_struct *vma) { /* The vma should be detached while being destroyed. */ vma_assert_detached(vma); @@ -480,25 +512,6 @@ void __vm_area_free(struct vm_area_struct *vma) kmem_cache_free(vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb(struct rcu_head *head) -{ - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); - - __vm_area_free(vma); -} -#endif - -void vm_area_free(struct vm_area_struct *vma) -{ -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); -#else - __vm_area_free(vma); -#endif -} - static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -3156,6 +3169,11 @@ void __init mm_cache_init(void) void __init proc_caches_init(void) { + struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr), + }; + sighand_cachep = kmem_cache_create("sighand_cache", sizeof(struct sighand_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| @@ -3172,8 +3190,9 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - vm_area_cachep = KMEM_CACHE(vm_area_struct, - SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + vm_area_cachep = kmem_cache_create("vm_area_struct", + sizeof(struct vm_area_struct), &args, + SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| SLAB_ACCOUNT); mmap_init(); nsproxy_cache_init(); diff --git a/mm/mmap.c b/mm/mmap.c index 6401a1d73f4a..15d6cd7cc845 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1305,7 +1305,8 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma, /* unreachable = */ true); + vma_mark_detached(vma); + remove_vma(vma); count++; cond_resched(); vma = vma_next(&vmi); diff --git a/mm/vma.c b/mm/vma.c index a16a83d0253f..c7abef5177cc 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -420,19 +420,14 @@ static bool can_vma_merge_right(struct vma_merge_struct *vmg, /* * Close a vm structure and free it. */ -void remove_vma(struct vm_area_struct *vma, bool unreachable) +void remove_vma(struct vm_area_struct *vma) { might_sleep(); vma_close(vma); if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - if (unreachable) { - vma_mark_detached(vma); - __vm_area_free(vma); - } else { - vm_area_free(vma); - } + vm_area_free(vma); } /* @@ -1218,7 +1213,7 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, /* Remove and clean up vmas */ mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - remove_vma(vma, /* unreachable = */ false); + remove_vma(vma); vm_unacct_memory(vms->nr_accounted); validate_mm(mm); diff --git a/mm/vma.h b/mm/vma.h index 55be77ff042f..7356ca5a22d3 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -218,7 +218,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm, unsigned long start, size_t len, struct list_head *uf, bool unlock); -void remove_vma(struct vm_area_struct *vma, bool unreachable); +void remove_vma(struct vm_area_struct *vma); void unmap_region(struct ma_state *mas, struct vm_area_struct *vma, struct vm_area_struct *prev, struct vm_area_struct *next); diff --git a/tools/include/linux/refcount.h b/tools/include/linux/refcount.h index 36cb29bc57c2..1ace03e1a4f8 100644 --- a/tools/include/linux/refcount.h +++ b/tools/include/linux/refcount.h @@ -60,6 +60,11 @@ static inline void refcount_set(refcount_t *r, unsigned int n) atomic_set(&r->refs, n); } +static inline void refcount_set_release(refcount_t *r, unsigned int n) +{ + atomic_set_release(&r->refs, n); +} + static inline unsigned int refcount_read(const refcount_t *r) { return atomic_read(&r->refs); diff --git a/tools/testing/vma/linux/atomic.h b/tools/testing/vma/linux/atomic.h index 788c597c4fde..683383d0f2bf 100644 --- a/tools/testing/vma/linux/atomic.h +++ b/tools/testing/vma/linux/atomic.h @@ -7,6 +7,7 @@ #define atomic_inc(x) uatomic_inc(x) #define atomic_read(x) uatomic_read(x) #define atomic_set(x, y) uatomic_set(x, y) +#define atomic_set_release(x, y) uatomic_set(x, y) #define U8_MAX UCHAR_MAX #ifndef atomic_cmpxchg_relaxed diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index b385170fbb8f..572ab2cea763 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -476,7 +476,7 @@ static inline void vma_mark_attached(struct vm_area_struct *vma) { vma_assert_write_locked(vma); vma_assert_detached(vma); - refcount_set(&vma->vm_refcnt, 1); + refcount_set_release(&vma->vm_refcnt, 1); } static inline void vma_mark_detached(struct vm_area_struct *vma) @@ -696,14 +696,9 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void __vm_area_free(struct vm_area_struct *vma) -{ - free(vma); -} - static inline void vm_area_free(struct vm_area_struct *vma) { - __vm_area_free(vma); + free(vma); } static inline void lru_add_drain(void) From patchwork Thu Feb 13 22:46:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13974146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23A85C021A0 for ; Thu, 13 Feb 2025 22:47:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FCF2280015; Thu, 13 Feb 2025 17:47:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 75C2F280001; Thu, 13 Feb 2025 17:47:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D789280015; Thu, 13 Feb 2025 17:47:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 37ECC280001 for ; Thu, 13 Feb 2025 17:47:39 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DE8D01416FB for ; Thu, 13 Feb 2025 22:47:38 +0000 (UTC) X-FDA: 83116409796.02.6494189 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf11.hostedemail.com (Postfix) with ESMTP id 237FC40005 for ; Thu, 13 Feb 2025 22:47:36 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vgiqGPV4; spf=pass (imf11.hostedemail.com: domain of 3iHauZwYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3iHauZwYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739486857; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HXs3qm65Mw4fWBObt4HCubdoJZ7UfWSFf9bCMyPuZ5Q=; b=fU+a33zzgyM9KICGaXn1EKCjt+PoGiliNuI25/637dGidBmKbj4TY1o14vKUG6osYfSgI3 5VNq2MkJEtn52ShC9eioQTNDRsMb2rX7W+KSAou+LfxOace610pe0LG7gKQQkk1RSYlQtR i8VpywBoh7FwYqoy6mky8AT8SIPg3LU= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vgiqGPV4; spf=pass (imf11.hostedemail.com: domain of 3iHauZwYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3iHauZwYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739486857; a=rsa-sha256; cv=none; b=v1P8JWkpOfm3kbSy8YZeVXGBMt/wMa8aNj7uMPO/m4NKqc4LmoqKZlj39/HvFkXhMzx9c2 I9ZutJ8IQouqz3ZZkaypyhkyXtKa9tpjLzEyjtlOm8HbXkUHKsjqLIn6oQQAkthoTtw/GO AoUfr5cAYrOMR8ony4O0+Cyd24JNe9g= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-21fb94c7fc6so29879095ad.1 for ; Thu, 13 Feb 2025 14:47:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739486856; x=1740091656; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HXs3qm65Mw4fWBObt4HCubdoJZ7UfWSFf9bCMyPuZ5Q=; b=vgiqGPV4zts1KfR6tI83XW8Qwqb9s9E1Ie4ohEat+/rvtXAdPmrndoeiZWhd0i4RLX C6TL9bA/SDHyhesNxWVp2meWdROoeAlAzmy+cYe14QcMsXz4mWxnB5BLHnYeiz9T74ux yo3K0XVMe/nOHqUase7nKHNekSeG1wSUuzOEvVDkuDewhr+0kxeN75ty/3AcCK5xhfcY BGauuh9fSg7XvgEMKYOP5OlSoL8no6Z9Ontc+f269N7Y6kCcmoqmqdIWAHYH/zU/yiqy XJNh76Xc2g3HF2Vh7Bv9L5dN52AwNo0i+5Ecl9pyHfhZWuOfzYL9+VUR7gPQuaKQOhOM 5j1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739486856; x=1740091656; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HXs3qm65Mw4fWBObt4HCubdoJZ7UfWSFf9bCMyPuZ5Q=; b=GWIjJWDFeVUCxhiMyt/cKJpqnOu0oKxaNCpE0CmvKlIL7/Vss1E7zKYVhH8Spge35Y KRP6afjsk9BL6B6Y6DrKe5HXkjQUrqAt90ZkYNDhUuRZMd73872mZMPEAVhPeGNl3RPG FgcY4C3cKCgZ4bagcjG6188zVkwTUhLz9IFiEG5q9R8E+Oxs4hJqa/lC3I/wEx9KZZ/o LuQxU+GdOL0w/LWLyxHtQ9MnK6T7YtrSU6CjSXUQ0FtAFr+FVYx/G9sU7stngoLM5suN xf8aWpJHyVlMaKf84w1O96oAC0UCeghD9bZZhK55fdedkNUX8WVN7ALi8nF07U5Xg1vk iAOg== X-Forwarded-Encrypted: i=1; AJvYcCUeNfn2V9UZvRYuMYArHL8vrvVOsYm9Z1GM1viFwWpsIlTPvCYVZa/ObivWGR4tymLVITfgT0Thpg==@kvack.org X-Gm-Message-State: AOJu0Yx5gtxsK9MRR2eY7dPp1HmkbgjN7rpFiS71uLAixUXwq+vgNGvf b4ehor6qU8w/4Av+hU72o5FsvnyAfCuLdL7qI/Grg5BozeZZ980Z1yqf5ItWH1dBL21BUmNBAyh 0vQ== X-Google-Smtp-Source: AGHT+IFEeHfewJkwAt0Xqt/tC1plnftNTUCWIl49bO9ckcrGnetyW/ghDc1KfOz7C3qqfgR140G3l01rzEw= X-Received: from plblc14.prod.google.com ([2002:a17:902:fa8e:b0:212:48d4:bf16]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:dac4:b0:21f:9107:fca3 with SMTP id d9443c01a7336-220d20e90a3mr71286775ad.30.1739486856011; Thu, 13 Feb 2025 14:47:36 -0800 (PST) Date: Thu, 13 Feb 2025 14:46:55 -0800 In-Reply-To: <20250213224655.1680278-1-surenb@google.com> Mime-Version: 1.0 References: <20250213224655.1680278-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250213224655.1680278-19-surenb@google.com> Subject: [PATCH v10 18/18] docs/mm: document latest changes to vm_lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, david.laight.linux@gmail.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, richard.weiyang@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Stat-Signature: eguwoe371kx5mzjapjags1oyw57n79f7 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 237FC40005 X-HE-Tag: 1739486856-699664 X-HE-Meta: U2FsdGVkX18vlt62iGsLSpdThxJyo/9a//zpOl7BDuVCTY9+H2e5GODvuy+V41O8fmwNJqo6QS1CB8e3LGVUkLlJBxdbmCWizSisNsowJI/KwsWTNK1+M58cPorjYVzhht0xMSKoeGsSlrhUecOoONMLP7swGAuAcMkcZdwmEpHw+vRly5yMnA+REPofIRcpn8GjZ0xbDoAYhp8suEUUztSyXGWyBr2CzW7adxkNKVjm1y0deOQ9zunXWxkrMNkbLo2Zb2eetoYZ2Mx2E4bHLm2okP9Is4F7un479ozCuFRVfHuZrtoqlzZNggBTJm5RMviI2bI2CoJBrpv99b58hpvH8+jQLB1WJU/m0SPsVohB32m9IXDvCpDXT+rRYTsRp/deFL96cjtvzAswaS8ol5ONtyE5V0G3tvfjn3H9BCb2HCOvHL81+XDt6Sh46d0quy8AhmckMCyZ1kgom3d/IEhL4hsKcyMawp3Si5346GBA6hwb44HZDgRsPpAAyz6C6U5aWZpUHwHKu1EOfjJYrdSKpvJWs+Fk5TJ2Zmh36EnDb9gQY3D9v5atkGwfwyB2x5Pd199z4zLGbn9f9m9lSmmeU5V5zKUi9CLhjnWVsaQ5Sj8Q9PgJpZI9HryYfn9UBsvgpYGeG0XOpMoiBr0B5yzX1NV8K/Pvuh6VZDBXVMsB6YP24Fn9SdUOCg5aDq4zEap1K/ugM0K7l0zL1O3s+b8dtknaMxdZ+1NtS8+NoG3T38Of9W4ga1B4XuBPdS66zc1Uu7zcxqqVjl5sD64XKTlSqWYdcNPyZCPIUZr/TYaouq0hz0mkzMcKRUd6E2FQBRqzl32etK+xWcFbpXMolz4GJT+YJFAiNgR0608Ewvx2sxWzoDVhLcJur8lsuTKQBl/yn+XqAns1PmdzbjrneK7fjWzXq+Jt+5XCPx+v5ZOJ+Mm7Zdvwn/U8INxbBCVTj/wHWZfG/l1jL0sYXYf uiuUf22P JPOvVuwisczfpTagDuY0RFPGR7bSHiV2mf8CjLH8RONp975JvNySnuhLGUhSV+FhKaoxNp7kV91/KCz28vn3T2D+zG/p3pWC+ZhM5RYGid680C7adBxQlYaeTC+0MlOWxM3asJJwZuVOiBIosdGLrSY2nQDEGnidMKVHvxAYCGqEMZimaySdMH1EUqtOF5+c3i8QdaF26UwxG0L9LUee6a7UiN+e+cmuKQRMf47G7ulq5E8FDCw92iB1SCIVV0GhJpBWwn4cVZa6UKwI0u3nrYrTOGSI3LaSbyaoD5H7hlLRMvRyVvPXNzqICDUrD6nlJuvJLZ5A2FfOHy+DlZXwfYDyB4DcRvYE89VcgCU2WZ3Mt7d9nRArHFh2hGCiES/B5laH1YeX8mi9drsLzSLqy+O1/Fctvu0mtIjxML6aV19ke99YIWN8qOiY3RBTgGEm9NtxMBRH3+03gq9VgD0ToG66rzpnXnpyedJF83Y+wRuyFPodFdT72F3C/gocWH2cgWckvQNrXLrHXEIF6i+qy0ngjK6bvxN6gOJl5dQ8rJAQUecvMTFAz8fWbvRu/gK9f9gmR50zQYK+eMgjB2HqM8PgU6KBVV7DVrf1ayLXfs3etuwt9Bs61/tG01/BFnNWRGADHoc05tLtIZrtB2RStXWIWw3w3DrGI2tZyy0fGGT39bB17s9xHGcGXT6dqb9AgouYi0PVjaqA3LdE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the documentation to reflect that vm_lock is integrated into vma and replaced with vm_refcnt. Document newly introduced vma_start_read_locked{_nested} functions. Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett Reviewed-by: Lorenzo Stoakes --- Changes since v9 [1]: - Updated documenation, per Lorenzo Stoakes - Add Reviewed-by, per Lorenzo Stoakes [1] https://lore.kernel.org/all/20250111042604.3230628-18-surenb@google.com/ Documentation/mm/process_addrs.rst | 44 ++++++++++++++++++------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/Documentation/mm/process_addrs.rst b/Documentation/mm/process_addrs.rst index 81417fa2ed20..e6756e78b476 100644 --- a/Documentation/mm/process_addrs.rst +++ b/Documentation/mm/process_addrs.rst @@ -716,9 +716,14 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`, before releasing the RCU lock via :c:func:`!rcu_read_unlock`. -VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for -their duration and the caller of :c:func:`!lock_vma_under_rcu` must release it -via :c:func:`!vma_end_read`. +In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked` +and :c:func:`!vma_start_read_locked_nested` can be used. These functions do not +fail due to lock contention but the caller should still check their return values +in case they fail for other reasons. + +VMA read locks increment :c:member:`!vma.vm_refcnt` reference counter for their +duration and the caller of :c:func:`!lock_vma_under_rcu` must drop it via +:c:func:`!vma_end_read`. VMA **write** locks are acquired via :c:func:`!vma_start_write` in instances where a VMA is about to be modified, unlike :c:func:`!vma_start_read` the lock is always @@ -726,9 +731,9 @@ acquired. An mmap write lock **must** be held for the duration of the VMA write lock, releasing or downgrading the mmap write lock also releases the VMA write lock so there is no :c:func:`!vma_end_write` function. -Note that a semaphore write lock is not held across a VMA lock. Rather, a -sequence number is used for serialisation, and the write semaphore is only -acquired at the point of write lock to update this. +Note that when write-locking a VMA lock, the :c:member:`!vma.vm_refcnt` is temporarily +modified so that readers can detect the presense of a writer. The reference counter is +restored once the vma sequence number used for serialisation is updated. This ensures the semantics we require - VMA write locks provide exclusive write access to the VMA. @@ -738,7 +743,7 @@ Implementation details The VMA lock mechanism is designed to be a lightweight means of avoiding the use of the heavily contended mmap lock. It is implemented using a combination of a -read/write semaphore and sequence numbers belonging to the containing +reference counter and sequence numbers belonging to the containing :c:struct:`!struct mm_struct` and the VMA. Read locks are acquired via :c:func:`!vma_start_read`, which is an optimistic @@ -779,28 +784,31 @@ release of any VMA locks on its release makes sense, as you would never want to keep VMAs locked across entirely separate write operations. It also maintains correct lock ordering. -Each time a VMA read lock is acquired, we acquire a read lock on the -:c:member:`!vma->vm_lock` read/write semaphore and hold it, while checking that -the sequence count of the VMA does not match that of the mm. +Each time a VMA read lock is acquired, we increment :c:member:`!vma.vm_refcnt` +reference counter and check that the sequence count of the VMA does not match +that of the mm. -If it does, the read lock fails. If it does not, we hold the lock, excluding -writers, but permitting other readers, who will also obtain this lock under RCU. +If it does, the read lock fails and :c:member:`!vma.vm_refcnt` is dropped. +If it does not, we keep the reference counter raised, excluding writers, but +permitting other readers, who can also obtain this lock under RCU. Importantly, maple tree operations performed in :c:func:`!lock_vma_under_rcu` are also RCU safe, so the whole read lock operation is guaranteed to function correctly. -On the write side, we acquire a write lock on the :c:member:`!vma->vm_lock` -read/write semaphore, before setting the VMA's sequence number under this lock, -also simultaneously holding the mmap write lock. +On the write side, we set a bit in :c:member:`!vma.vm_refcnt` which can't be +modified by readers and wait for all readers to drop their reference count. +Once there are no readers, the VMA's sequence number is set to match that of +the mm. During this entire operation mmap write lock is held. This way, if any read locks are in effect, :c:func:`!vma_start_write` will sleep until these are finished and mutual exclusion is achieved. -After setting the VMA's sequence number, the lock is released, avoiding -complexity with a long-term held write lock. +After setting the VMA's sequence number, the bit in :c:member:`!vma.vm_refcnt` +indicating a writer is cleared. From this point on, VMA's sequence number will +indicate VMA's write-locked state until mmap write lock is dropped or downgraded. -This clever combination of a read/write semaphore and sequence count allows for +This clever combination of a reference counter and sequence count allows for fast RCU-based per-VMA lock acquisition (especially on page fault, though utilised elsewhere) with minimal complexity around lock ordering.