From patchwork Mon Dec 16 19:24:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8154E77183 for ; Mon, 16 Dec 2024 19:24:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AC946B00A9; Mon, 16 Dec 2024 14:24:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 335316B00AA; Mon, 16 Dec 2024 14:24:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D50F6B00AB; Mon, 16 Dec 2024 14:24:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 01E506B00A9 for ; Mon, 16 Dec 2024 14:24:28 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A1D56160914 for ; Mon, 16 Dec 2024 19:24:28 +0000 (UTC) X-FDA: 82901797230.03.3FAD3B3 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 2C90580015 for ; Mon, 16 Dec 2024 19:23:29 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zPjeKIge; spf=pass (imf30.hostedemail.com: domain of 3aX5gZwYKCCoYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3aX5gZwYKCCoYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NdPtaL44NpTjP9eS1K5KVvptksJqAFYIrPxgR/8blQY=; b=h+gNg4bMWQibSnjpkuc87PgwkNuvnaj+M1QqD3/DFMYwol+baRjkEQW8moObWF50BNCxag 6qRXda93fxwsIIjSqhfUEJ3gMoAzhhcTWcZDfpBq4T9AKNTChc0TYD1lCBVt+z/Ltqa0uf bIKPHOjYb9sCfjg4su4tv5wcXhATEZk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zPjeKIge; spf=pass (imf30.hostedemail.com: domain of 3aX5gZwYKCCoYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3aX5gZwYKCCoYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377044; a=rsa-sha256; cv=none; b=uyFwPUuzwJmpRdGK9WPb48cKTe9z4EXOZwW5MrU51OAp+Uh56b+y0bDhbdADZh5WT3FLYU Qy0gtAve4LP5k/Ec1GM68AalSFFxRmP3f1XdblNmm8XO3mQtVlQXo1EZXqyqZfjg8xOdvk JnjywHZcMZiqo6eOofYQ7C7FLODfJRI= Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-725e6fe074eso6613831b3a.0 for ; Mon, 16 Dec 2024 11:24:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377065; x=1734981865; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NdPtaL44NpTjP9eS1K5KVvptksJqAFYIrPxgR/8blQY=; b=zPjeKIge8nqGATBWYUP7XOiwmCpAp0aEnBzXQqzPvkizOFxcVVY0/AuNt02YfQy628 qDyJy+mnGaAi4zB4QyTIdq9bXzpL85buJfNwYr8RVw6BaJRC4iZyeBF3W+0Bf4yBaxvC a4XB8aMURO5XUMu997AzmCyYc7X9xUicPqUTe+RsT5vBcdOr9O8Nb+o55jsettKF+FoU 5eZxUS64OSrbPtL3+fLoE7AkhX/o1hCJE//hoG6tdUdo6vW9FIRmqlPmYATFU6qSqCDN CLwhclP5viZRDoXZWwkCBowmVNbIj5eTxPLN14f4D+Jg9ry+VQ8E4T8HYVpu6e3YUzNU eSvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377065; x=1734981865; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NdPtaL44NpTjP9eS1K5KVvptksJqAFYIrPxgR/8blQY=; b=AOZ1eFTnDILYx/5ZT3Zd0h4YwAJvBU1b+aSgEkEKpu1Sgdgl7J0pKPgAwiY9kB2fJW P8lRx2tHETAztf+9wNev+/CiFeOcrLEIX+5OAKyAl7SqO5gQBFZm/4FtrLOeYKSWpEEi MquuRZpEtc5AfJ53r94bltLi4HDmwKDCu/yqebXpVmTPpUDH2qqxm2VjlQFxIs3Ynbq+ qFUER2tjYS7pKOH1cd0kBpBrZKsQwYbdWlOYhBfpSU4H4XvA02wPNdyckQ4sHPjFPMX0 FYn/6b05I/8LAy4Zp3VfLLvmNw7M0qz8ViJKcoY5zn61vchP3lHhWZ8TxFGJb9CV4Zds xiMw== X-Forwarded-Encrypted: i=1; AJvYcCU/P1knJ7Q7ejFY0a9+DX3am0QKVOKOnIC6z1iAwSp0AZTdQ4rYMEE2Jny75oGPLILzABp1JRayyQ==@kvack.org X-Gm-Message-State: AOJu0YzY0h9fkYUuNL4gTwve3xxgGHSbOqn6XK1rIaoJlKBP2/ftwAm7 ILT0+i1EALqztxvOjKYLmSMpY3vBlHn4Eebrjsjvu3vePSLnqskmtWHCFcO4I/tObHPIBW+XMEs 5aA== X-Google-Smtp-Source: AGHT+IGebkhkuPAK7qZirmEWQkyXcqUHZOWu2H7W9h1+nHtKbr+4oTQtJb1gp01e4aHBI2bniuiuItfkaYg= X-Received: from pfbbs2.prod.google.com ([2002:a05:6a00:4442:b0:725:e2fd:dcf9]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4305:b0:725:973f:9d53 with SMTP id d2e1a72fcca58-7290c19ec16mr16656092b3a.15.1734377065528; Mon, 16 Dec 2024 11:24:25 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:04 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-2-surenb@google.com> Subject: [PATCH v6 01/16] mm: introduce vma_start_read_locked{_nested} helpers From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 2C90580015 X-Rspamd-Server: rspam12 X-Stat-Signature: r8mxii5ogg9m6k5s64dn7obrjw17hqf8 X-Rspam-User: X-HE-Tag: 1734377009-921286 X-HE-Meta: U2FsdGVkX1/KZqYPxhZkQRN8A5rtjWc6vcVFrI/mgXJi8krYd7nZY9BCa24KS61ncW+o2za3IlHJCpYig/38A9K6921nNqCY2wjr+drlUY5kkzF7sufzxmqBqo4Fg69O2lK3baVBG9dZOsi23/hmn7k2V6YuBizFTbkc7sqd15Pndo8Ts9wQCNMTGD3uGCffcij5Wp+sW6Auh1oKxJXAQQ9jGSb++AuPTlANlkCmyG3iQCBwZT6VzkdOIMaAWdY/kqDFHWHf22r2nNvtx4YLOvuKrzVWQI+mpxFeO5fxEi07j/IGtUSmrrTwLxgQbI2QgmbtTDbAhfIaSFNp9dsGJ72xHz55Gi79FJVx5g6KHJuSsaUFC8DB8PKwQy4vqLA0ufes/GjOksLSgSNALx1beLtSfcYC8c5AnU3p7GjqGsXszEOJpe83zWF67fwlTcrr2kozGiDCnzK7D4R4wcHl7OUjoxA1+EiQ7mH62x8v6WxyT+mr8IBfN6ocG9yA3MZ3ZuK7a4SkYl+SeIyLHZuH9BZat6wLpoujRByiQ0m5uwaMPI695H4acBRjYA/3SteIqy1ixGDr/R1Ea1qmyO/Sd7p3SMs5gMGszGvjo3tQSaraVKJrdoCNcqPxCJahcI1d8Qtf9JvZ1FynLx+Qb4ERJfZGWBV6/NtY0EKJQrYmTSw29hhjEROVhMO/WlVfJpG5MNeZ0Hlg0kdjTOo7UfLiwyS7tRp90KCACJar/gToRsmBltgyL4LQICOaNM2nT8SFqKSojbvQfD1nHTOynt0+2YZUyN18iQ1iNClsJbJ2ExrgMHSYV3akbI738pulhoNlQd4vKl98t9xHn0JejeAO6fWlu+q1PdEz1UnumsQohw7LorHn3N1ifCkLJBdQSoEo4RJS6BCjXh1A3j0RmZLNLtbDPR8lpx6nlZ7tlmKwoCl8TupilFclOPOMLlMEUwlHClngSg1q/+rmI7Krqng lsY1v6WY pVgBl2MKgR3SLorfmsY01owgMDn0co8xc89RHbNoQtP9HYSbPLQGjptZTdhigaauk+onNsDSvoX5PYmEA6riu2TfoQowcnc+P+fBCG/mn087BuMwwRMki6dQUwy3jML2oGPZAHdspezYpGudIkbNEpSGkknX6MhgGa1qee7vcm+xyBByrlGAZLIW8X9RragNWJOqw3noIQ8J19t2Z58Urc0gKJ9hUyFiP5pPo8OsMYhcS3Bw3T0MPfvUq4eRMVDiuiUc9pgBOar6AScXX2dnHynG/MgrjK2IXNCMQtRO1sYICprS4VIyaotlAx0pNGscNdptDqpqPHtmnG7ED3eLsUBHOuBzJVdGY4QnqR2XQMOrT/TzGiWovBZSHvTVqDVSgmABSv7Ap1cuPyz1ylRplhJqpJb6ZzI1EyoeAys2+wC1ynHL9YfYvXhMarSocOEgAc1UR3t50dwqbZ80fm+MZcEHJJaezX8R1TN0dewJ6PFB0o16+4l47CJu8LITjW30fpQbPBquboN3ztzMq+yP9lC9VKnFq6PbYIhTIKW9htRYksUD3vTxUVtL9OV0plhNQOkrhZAiCSaIY4gBQ0RiCHy1F/fS9mMLhR07ybzWJBzXjOGM7ZYWpT6r/StPYT0cyLotLzLTS6+vG+cu7wHD19izCjucZ2yxwTDFzRseDAeUIvpg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001380, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions which can be used to read-lock a VMA when holding mmap_lock for read. Replace direct accesses to vma->vm_lock with these new helpers. Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes Reviewed-by: Davidlohr Bueso Reviewed-by: Shakeel Butt Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/userfaultfd.c | 22 +++++----------------- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1352147a2648..3815a43ba504 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -735,6 +735,30 @@ static inline bool vma_start_read(struct vm_area_struct *vma) return true; } +/* + * Use only while holding mmap read lock which guarantees that locking will not + * fail (nobody can concurrently write-lock the vma). vma_start_read() should + * not be used in such cases because it might fail due to mm_lock_seq overflow. + * This functionality is used to obtain vma read lock and drop the mmap read lock. + */ +static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) +{ + mmap_assert_locked(vma->vm_mm); + down_read_nested(&vma->vm_lock->lock, subclass); +} + +/* + * Use only while holding mmap read lock which guarantees that locking will not + * fail (nobody can concurrently write-lock the vma). vma_start_read() should + * not be used in such cases because it might fail due to mm_lock_seq overflow. + * This functionality is used to obtain vma read lock and drop the mmap read lock. + */ +static inline void vma_start_read_locked(struct vm_area_struct *vma) +{ + mmap_assert_locked(vma->vm_mm); + down_read(&vma->vm_lock->lock); +} + static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 8e16dc290ddf..bc9a66ec6a6e 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -84,16 +84,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, mmap_read_lock(mm); vma = find_vma_and_prepare_anon(mm, address); - if (!IS_ERR(vma)) { - /* - * We cannot use vma_start_read() as it may fail due to - * false locked (see comment in vma_start_read()). We - * can avoid that by directly locking vm_lock under - * mmap_lock, which guarantees that nobody can lock the - * vma for write (vma_start_write()) under us. - */ - down_read(&vma->vm_lock->lock); - } + if (!IS_ERR(vma)) + vma_start_read_locked(vma); mmap_read_unlock(mm); return vma; @@ -1491,14 +1483,10 @@ static int uffd_move_lock(struct mm_struct *mm, mmap_read_lock(mm); err = find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap); if (!err) { - /* - * See comment in uffd_lock_vma() as to why not using - * vma_start_read() here. - */ - down_read(&(*dst_vmap)->vm_lock->lock); + vma_start_read_locked(*dst_vmap); if (*dst_vmap != *src_vmap) - down_read_nested(&(*src_vmap)->vm_lock->lock, - SINGLE_DEPTH_NESTING); + vma_start_read_locked_nested(*src_vmap, + SINGLE_DEPTH_NESTING); } mmap_read_unlock(mm); return err; From patchwork Mon Dec 16 19:24:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6924E7717F for ; Mon, 16 Dec 2024 19:24:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AC636B00AB; Mon, 16 Dec 2024 14:24:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 35C916B00AD; Mon, 16 Dec 2024 14:24:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 224616B00AF; Mon, 16 Dec 2024 14:24:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 00E886B00AB for ; Mon, 16 Dec 2024 14:24:30 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 92C5B12098D for ; Mon, 16 Dec 2024 19:24:30 +0000 (UTC) X-FDA: 82901797776.30.7F00D60 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf02.hostedemail.com (Postfix) with ESMTP id 26BFB80014 for ; Mon, 16 Dec 2024 19:23:31 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=A3ZgUdSR; spf=pass (imf02.hostedemail.com: domain of 3a35gZwYKCCwacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3a35gZwYKCCwacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377041; a=rsa-sha256; cv=none; b=mqUjLRu6E9U1DK9LrCEjap59jQGt3kh/Y6Kxc4CWSlqPgg4AmeihpCafVDPwmyDMRWIMWu HWRr/PQvwJem3ajbhwDmX8epBAHRsH1sHPAvFHlcaXS3/V9nAPlqER1xVpzqhMpwp9lil8 6AZaVzEKtV+wACNQP3RESGvkl41MYKA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=A3ZgUdSR; spf=pass (imf02.hostedemail.com: domain of 3a35gZwYKCCwacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3a35gZwYKCCwacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377041; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=igJVFGm55VljdfCWWP1FWPjwRGLgL1kTgn08p8BmWdc=; b=hzwC+Hos7pcMN7X/tQC7YaVTI2JIy6/EaZLMOx4BqZ4/EPJGG3Q75YXOk96JWCVfl2YU6O gTF79jq5NIP4vkAHuODV/B2IwaZIM+C+AEq5AqDVgALljdqnd1AQu+00VaMw2j0ugUjka+ 0KxNzXG3DNjd1PvIY5GM3QZV2btigCc= Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-728e4e30163so3251238b3a.1 for ; Mon, 16 Dec 2024 11:24:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377067; x=1734981867; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=igJVFGm55VljdfCWWP1FWPjwRGLgL1kTgn08p8BmWdc=; b=A3ZgUdSR4vn5hIGa85sFpGFq3zBPRyCmf2eFKdj7jDgN0JgOyazfWcCQkmGFopl2PF cDzykDLF+yqkPE7IMXW5TSO9bhsJejD7nYTO1vCZqZAzi2bp8VDFe2APAFvVN7W2DEkm Gq/jjnMahLC9un6YYoaq7EobHHP84eAoekLLWDJtqZhuCIKEHbNg1tkWD555E3+6zDZ3 7ccl1usv/tPO9qZmsGtZ+CN01C3JuvyflGnxkhJLJt0jc3p2U0Gl+803jvEQZNoNSIBs VPYHFkJhgWjSioi9Wr1rNYZpbF1IPSqnZK7speqI8Zsfn94cpm2VQ8zfk8Rm5nlECm0V OeKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377067; x=1734981867; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=igJVFGm55VljdfCWWP1FWPjwRGLgL1kTgn08p8BmWdc=; b=dWKl118Inyu750yz4XaqpC783pyHdRV8+hwysuWhFGGMG/aFllxAUd32JGi5NvuEc6 ikV+Idn3fDBIJGI4BwLNRXw6rc5mh1Tv2QNyuizfE+IrDqpagOFXUlCpBE53E2e97snD ZdjK4i2P57SsnvSB7CVhQhhNI7/wuGvryWeHvqh2K2FLXXXDTsIc4cCxa5MMcRx01tXl /scp4yTpScgkmOa5VwRgT/lfglMpuKzHDyVozJc0+qDGoBEMWe2e9PqDNYNKSDcnDryQ QPCztsWpbxCxpFXPB31dD2NS9KHkBGk8BDCbVenp6rGFpw7Rc4p2JXGkmQpUf30qj6bk jz5g== X-Forwarded-Encrypted: i=1; AJvYcCVq7q3XVUhrQuNAZ1Ccx4RQZuGoLqfhWHkRLvuVWGLZ4xoxkhZIVLWOPB+D8RC8xfOvujrsVkf1Rw==@kvack.org X-Gm-Message-State: AOJu0YyJ2SO3wJyn+i79ssI8/bnAUh7hCOOI1GS8wzoHB2bCpZXvvFyz 8abO0N03arKLyB+zjou6kaCL6hF4Ho7gvWIXRxKW/LZQ7HG9eL5EX0f/V6CnkcfB/joRWGlz+IM 0ug== X-Google-Smtp-Source: AGHT+IH2UojHUVJ+gBvUhOeXPQ2zGfzEXizGrigN1NQ9dKN5xlrh19HGwEfI3CKJSRYp1fQb42E5oINrzxo= X-Received: from pfbbk15.prod.google.com ([2002:aa7:830f:0:b0:728:e508:8a3e]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:cd5:b0:724:f86e:e3d9 with SMTP id d2e1a72fcca58-7290c181255mr20564216b3a.14.1734377067561; Mon, 16 Dec 2024 11:24:27 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:05 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-3-surenb@google.com> Subject: [PATCH v6 02/16] mm: move per-vma lock into vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 26BFB80014 X-Stat-Signature: byw9rqfmwpip7qyawg1wukyrd1gnk3on X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734377011-387777 X-HE-Meta: U2FsdGVkX1+F4aXQrvONZ367o0Sbyqm3vBTLi7vaPTjh5IUjW+zL0+R+Aqi17m1FD1O2Sb3VtWdsnvRznnXbIB26jM5DbT8W+3ERIdE2QFNJ9jGIfKGYIOX0FQCt393/BmZgZyhNvmP5d+IY7luoVMfbW+BmBIL/3G7oWEVwfco/Me6TrTz9pPZdlWgUQmW317IGxxwPI25BeqiR/6MymhCz3D+diRAOmCcD6l8XPLq2SOc5BNnTnw+iK4N4YHa8Q8M9nhYfHXXT0ZJTWIA4a77muO9HtCbyNdc2bUG0rUsI6wnJzMC5IejbDL72X4UHBocQhiMrasmXsYLpPDnATnBN/DlOaBgdpHXJPR2ORd+kXLZHB4tmIso69Uwwqm53YBDrbdthHcnPz4Yhx1uM9RH+wF+Ii8vdc70lqtSqKgE08Yb5nsWlnJv0uZrYioTlMsxy54mE714DGmNFGiLdLngbl1AQEKMaTtXwVZ3tJIWy0pWb+8SELTkPgt4oYpBs5pwoCFuZ5NLBOrFUEhWx+AItUVEtlrY3cBTsL5gzS/n0lKEWF3b8tKpB23KNp+RO8AFPs1Sc38ekhUlrGzPxAAGq4ZD7EZKrSJF3HwDj1lwY/9aRxhTDOYiC+RexxmEsCzO+GmMQ4UE7eJUoviRo/WmWvagK5or7YBYC6rLIbUQfZqaheZ6sgKADRji8/oFoPjV0v6yBpSoRqcxbdJp6aMjvV2UOQWzPu2RxqCE2bH9L5IZHkXZYySbduQg/gNBOSE3op1/IK1aeH6NJldpOV74klBnUVdsyaUlMcIowAI/zgqnWEglZ15BANKBEPtwtdM7c8h7tVWxQIvSjYO2JdUmyzM9SON+hf758QTVGndymwS+ZuxBT/e7md4OSikcZs4P0qSKGHTYN0DXdSfdLsQ/zqWFjQHslcyJW4/EiJ+Sm3XHGGoNuySmB4TB2cjyV0DaVwgnjH29rbF86Qaj 9xpAB/oF UZYwPZmbAhwajSoDtOKUzvkeAZ7RtqX18ZrQuPo4oMFhA6Nuz8Ch4KY5wrxl5QgFbB44scGNhZgpDMMn+h1wLw6BnK7DflADm8fpesmyd5gM7QO269mRUbSQHDFf4bLVKIvnHYz38gIZtAJnAAfI0KyEQADQ7Mq1XfWyCTT37vcG4K6L6tfrve22XMW/ubemxzHLq7Jlf6An5EePPu6JLVmE3ovZcaCIUn/LdRIu0TuEVBtk/BBV5zyR/DZoA9+Zq+TPxfKbzriCIGK7ZNCB9EauJxQY9bYlbWre9B4xQYaJ7Y/9KlJTmvm4X3XwCTKAL9KRp9C8BFUjl0E7tYWNXNpMDZvcJa1MUdLEwub479E0Pl80gAOvQLalX8llBtQo1JoBO/AZr31nnBkh/M7+dfKVgC+jZ3nD9W9yElkUQ7wBL1yWZifRKBov/vGTKxPEHntIw4/Cu49IC/VxGs4MKMRh7/eKmSlr5hbUbdxKP9MT3tD2i7X7r4k9N9Z2TpuBmPiKjmtIsq+s+3GCP/V+MT+No0Frs1zbpOrWLk2UelueQE8xVO9NVhqKGBPiNb/+EBkcjg5jycZ7/Fo4aA2Dvm4BPMRTel2tSbfDZnMVUF3h+6hRvYk0sb4SHqhscT68fpYuC1AeqXDlvhx100gFvrFC41tPRUdtZM7o2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. Move vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned as well. With kernel compiled using defconfig, this causes VMA memory consumption to grow from 160 (vm_area_struct) + 40 (vm_lock) bytes to 256 bytes: slabinfo before: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 160 51 2 : ... slabinfo after moving vm_lock: ... : ... vm_area_struct ... 256 32 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 pages, which is 5.5MB per 100000 VMAs. Note that the size of this structure is dependent on the kernel configuration and typically the original size is higher than 160 bytes. Therefore these calculations are close to the worst case scenario. A more realistic vm_area_struct usage before this change is: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 176 46 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 54 to 64 pages, which is 3.9MB per 100000 VMAs. This memory consumption growth can be addressed later by optimizing the vm_lock. [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes Reviewed-by: Shakeel Butt Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 28 ++++++++++-------- include/linux/mm_types.h | 6 ++-- kernel/fork.c | 49 ++++---------------------------- tools/testing/vma/vma_internal.h | 33 +++++---------------- 4 files changed, 32 insertions(+), 84 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3815a43ba504..e1768a9395c9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -697,6 +697,12 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {} #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_PER_VMA_LOCK +static inline void vma_lock_init(struct vm_area_struct *vma) +{ + init_rwsem(&vma->vm_lock.lock); + vma->vm_lock_seq = UINT_MAX; +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -714,7 +720,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) return false; - if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) return false; /* @@ -729,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); return false; } return true; @@ -744,7 +750,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { mmap_assert_locked(vma->vm_mm); - down_read_nested(&vma->vm_lock->lock, subclass); + down_read_nested(&vma->vm_lock.lock, subclass); } /* @@ -756,13 +762,13 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int static inline void vma_start_read_locked(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); - down_read(&vma->vm_lock->lock); + down_read(&vma->vm_lock.lock); } static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); rcu_read_unlock(); } @@ -791,7 +797,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (__is_vma_write_locked(vma, &mm_lock_seq)) return; - down_write(&vma->vm_lock->lock); + down_write(&vma->vm_lock.lock); /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -799,7 +805,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock->lock); + up_write(&vma->vm_lock.lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -811,7 +817,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_locked(struct vm_area_struct *vma) { - if (!rwsem_is_locked(&vma->vm_lock->lock)) + if (!rwsem_is_locked(&vma->vm_lock.lock)) vma_assert_write_locked(vma); } @@ -844,6 +850,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ +static inline void vma_lock_init(struct vm_area_struct *vma) {} static inline bool vma_start_read(struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} @@ -878,10 +885,6 @@ static inline void assert_fault_locked(struct vm_fault *vmf) extern const struct vm_operations_struct vma_dummy_vm_ops; -/* - * WARNING: vma_init does not initialize vma->vm_lock. - * Use vm_area_alloc()/vm_area_free() if vma needs locking. - */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { memset(vma, 0, sizeof(*vma)); @@ -890,6 +893,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); vma_numab_state_init(vma); + vma_lock_init(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 266f53b2bb49..825f6328f9e5 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -700,8 +700,6 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - /* Unstable RCU readers are allowed to read this. */ - struct vma_lock *vm_lock; #endif /* @@ -754,6 +752,10 @@ struct vm_area_struct { struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + /* Unstable RCU readers are allowed to read this. */ + struct vma_lock vm_lock ____cacheline_aligned_in_smp; +#endif } __randomize_layout; #ifdef CONFIG_NUMA diff --git a/kernel/fork.c b/kernel/fork.c index 8dc670fe90d4..eb3e35d65e95 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -436,35 +436,6 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; -#ifdef CONFIG_PER_VMA_LOCK - -/* SLAB cache for vm_area_struct.lock */ -static struct kmem_cache *vma_lock_cachep; - -static bool vma_lock_alloc(struct vm_area_struct *vma) -{ - vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = UINT_MAX; - - return true; -} - -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - kmem_cache_free(vma_lock_cachep, vma->vm_lock); -} - -#else /* CONFIG_PER_VMA_LOCK */ - -static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } -static inline void vma_lock_free(struct vm_area_struct *vma) {} - -#endif /* CONFIG_PER_VMA_LOCK */ - struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; @@ -474,10 +445,6 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - kmem_cache_free(vm_area_cachep, vma); - return NULL; - } return vma; } @@ -496,10 +463,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) * will be reinitialized. */ data_race(memcpy(new, orig, sizeof(*new))); - if (!vma_lock_alloc(new)) { - kmem_cache_free(vm_area_cachep, new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); vma_numab_state_init(new); dup_anon_vma_name(orig, new); @@ -511,7 +475,6 @@ void __vm_area_free(struct vm_area_struct *vma) { vma_numab_state_free(vma); free_anon_vma_name(vma); - vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -522,7 +485,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); __vm_area_free(vma); } #endif @@ -3189,11 +3152,9 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - - vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); -#ifdef CONFIG_PER_VMA_LOCK - vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); -#endif + vm_area_cachep = KMEM_CACHE(vm_area_struct, + SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + SLAB_ACCOUNT); mmap_init(); nsproxy_cache_init(); } diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index b973b3e41c83..568c18d24d53 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -270,10 +270,10 @@ struct vm_area_struct { /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) - * - vm_lock->lock (in write mode) + * - vm_lock.lock (in write mode) * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_lock->lock (in read or write mode) + * - vm_lock.lock (in read or write mode) * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * @@ -282,7 +282,7 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - struct vma_lock *vm_lock; + struct vma_lock vm_lock; #endif /* @@ -459,17 +459,10 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) return mas_find(&vmi->mas, ULONG_MAX); } -static inline bool vma_lock_alloc(struct vm_area_struct *vma) +static inline void vma_lock_init(struct vm_area_struct *vma) { - vma->vm_lock = calloc(1, sizeof(struct vma_lock)); - - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); + init_rwsem(&vma->vm_lock.lock); vma->vm_lock_seq = UINT_MAX; - - return true; } static inline void vma_assert_write_locked(struct vm_area_struct *); @@ -492,6 +485,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); + vma_lock_init(vma); } static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) @@ -502,10 +496,6 @@ static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - free(vma); - return NULL; - } return vma; } @@ -518,10 +508,7 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return NULL; memcpy(new, orig, sizeof(*new)); - if (!vma_lock_alloc(new)) { - free(new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); return new; @@ -691,14 +678,8 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - free(vma->vm_lock); -} - static inline void __vm_area_free(struct vm_area_struct *vma) { - vma_lock_free(vma); free(vma); } From patchwork Mon Dec 16 19:24:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A2F2E77180 for ; Mon, 16 Dec 2024 19:24:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 449BA6B00AF; Mon, 16 Dec 2024 14:24:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F9016B00B1; Mon, 16 Dec 2024 14:24:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E6696B00B2; Mon, 16 Dec 2024 14:24:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 120266B00AF for ; Mon, 16 Dec 2024 14:24:33 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AF3471608EE for ; Mon, 16 Dec 2024 19:24:32 +0000 (UTC) X-FDA: 82901797860.30.FF80CA2 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf30.hostedemail.com (Postfix) with ESMTP id 3F16E80009 for ; Mon, 16 Dec 2024 19:23:34 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JRHdqEUA; spf=pass (imf30.hostedemail.com: domain of 3bX5gZwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3bX5gZwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377048; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tPcYammI4P+bTAfQ+4gtRvMOMTfi6gzN2lD/68NUW0U=; b=BJYC5gUtYz2jEKTRGdu03qNGSUkkJf3zOUat4+M6/tQU8bsPiQwoH/d1ho/J75q5ubgzNh mPiXrpmoACMs3xjYrIrKzieOGMwmMiXEJryqaJ02dkY1jXZoulJkEuPTzSzrS9elr4UD1Y hIqEluTaiI8xFNzc4PoPd/dQAy2xvQk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JRHdqEUA; spf=pass (imf30.hostedemail.com: domain of 3bX5gZwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3bX5gZwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377048; a=rsa-sha256; cv=none; b=71fLUg/YlF20MESwKV8lsPGxpb9JeWince49eRgtSnpk4AsUSDSCwGuebz9SYEYwidvF8/ Toem8SnoGbXHYZqyZNH2DUJqlUpK5BSgJwxvJa47VxzCMLkdsYGF3F/KhViVCc77ARfEF3 lOA6Rxe+pyweoUph+ixT2r5pMxH0jSI= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-725c882576aso3306792b3a.3 for ; Mon, 16 Dec 2024 11:24:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377069; x=1734981869; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tPcYammI4P+bTAfQ+4gtRvMOMTfi6gzN2lD/68NUW0U=; b=JRHdqEUAKyiImQJ/sgDagfw1r9Zmw64U/QFhfe6dAZPy0JQqqPrc0jR3fBzO4QOeLL 3Qx8o9i2L3/+vWcgpUZLIBjfHDNxhmM8cuBCpxSJCSAa0d4kQusXmKg8eM1sBwhNYSbr AT0qYm2xp6GJ9mzufnFroqwLMndOQRYQIDvady1SQry+fZ/vdueqF0ImIs7StdqE9kc9 eofH8czmx9L6X+vi2x5DLfhPblcu1fiiKjhqHZEfe0VuH/TtX76YKxVqKo+L5l41hgm2 bRkHM+E/2xAlQBm8yYfHnwnY90dXuO/4IlU7JoMG9BFL5E7to/ih9G7v2V2vhXpBoawd mPXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377069; x=1734981869; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tPcYammI4P+bTAfQ+4gtRvMOMTfi6gzN2lD/68NUW0U=; b=d3+28XPt1AWFU6UURN/ptcro/xxYTyJ+kuKUvLovdIqrjjxOK3+TsnBWS+PopgNkHa rqXZGKK4mYVEtkDfWyk6rrYlwXRt3Kvk26xnaD4ySYR5IZi95AngfBmNlnk/hq/EUbU5 ZrLkdbUck9fpViQBmZDCzC93j6W3lkOjcD3WpdZ5xe1DcUuQQ114dm7rZd+X6e2HwkYl 5VaG8nwLITIYf7lxgqTKbAQeJ8oStmNAAlca93cYM1N57Osms8pj/sFzR0T3yucIFiZS 7B4hX5I8ZQS0WLKZ5w11LPWNtHW3HetTdIy3sdH0RDguyZePhh7eN8mo14B3LUgjUaLs PauA== X-Forwarded-Encrypted: i=1; AJvYcCVk23KvoKlCFixnuUiOefMJY25Mmer/e1JeHx1rExkLZxadIcFhFU2M2F61sMAfieMM+SR/QJA0vA==@kvack.org X-Gm-Message-State: AOJu0YxUkDRnVtnK6fKvGMc66IiXSAZTRSA0v3/hs2RZNmWrAH+Zq3eN xtWQKFXuvY05cqJnoeINRfDkfLTZaNdBBUdADiOsFotH7Pyo4TN1L5kDjH6g7jqff8319CfG1L0 HNw== X-Google-Smtp-Source: AGHT+IHuIz3KcGOEBSL7UPXAppbQsIUkVSUTnN31px7An8YO7VAcBSifJ3+A4XtsuM0O2Sd96qkE9oQILpU= X-Received: from pgbcz14.prod.google.com ([2002:a05:6a02:230e:b0:7fd:4e21:2f5a]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:a120:b0:1e0:d32f:24e2 with SMTP id adf61e73a8af0-1e462d97415mr1129422637.38.1734377069542; Mon, 16 Dec 2024 11:24:29 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:06 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-4-surenb@google.com> Subject: [PATCH v6 03/16] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 3F16E80009 X-Rspamd-Server: rspam12 X-Stat-Signature: c98qb9ip7xhnmq7dm1c9h3xioynobqt5 X-Rspam-User: X-HE-Tag: 1734377014-239697 X-HE-Meta: U2FsdGVkX1822SFB/y09aTPd07rG/wKQkb3DevCTXlxGk0qboU/Qpu4hMi3euDrCWq7+zB+1dLl/7CZ8F5hiRKI+me18lXrB3x6fHzpeSrFX2LtDxkXFyKF6+j3tNqDp3t+tSbWlV5UrjdQnKWzY5IRKbGjN4EG40VBKUKcPoOaNvGnwHPkbYywHWayt25pWrerSdaesYhCGIMLYZrRjrkl+0q4DSmvC7iToU/245QfTGTtY8tUPgC96lZxcNDl/xyxReLCEHoCabe6fjoiw8/qmgn6PvXNRjTMzZh5YGDHLiJYEnXZ/l17ALairzVstP3xXwl43yNB4r0O64mKaf1uNBLXo8Bn8S4kAxgC/UVY2/XUnQ5ncT/4MBD7Y+HlM7rmNSkMHNEsKG1/kVWx4hgacMbBwqN08MqtdDhTBTb6S6q0YOJFr5cHZN14k9fICjN+UsgydMZ5b8OKik8yKGqvie8VXRhwm4tqHYBv8kkgkWF1x/2t2j2nwHQZigOmh/20rg1WvSb3UPFEAxd7P3ssbSUhwz/CHFCGtJQOV9LKOqSSzrkNqUPTDGV2JzP4573pl2/VJu/mrAdBsPJBzQ57TSyG+TGcPJqncs7vY8ZlcvCWw8vU2t/DUSToPruzB55BxFY6RhoAeIYOySPW6Ec+25/2P5oxtLwyLnpkch9UL5szRLaFad5dtGBQn1+Z+c5xFy6xg7wOQ29F5CkJ/c8uxjJaxRT4FnlA5b7E7m70NAtRgn3LScYI72EDEyknu30Qe4uUtxAHDzbbBvw7PlaedIuYUnYM2klyalTO+5hJea86eS4OE8RUqM6iLC0hhETfDKXlYztXWsXd2TVw1KCV+zAWRNGDVXribAk+XlXC6KHW9RjEnWgIULXc3AO5xvSAWAoknL9OhV4gMZBkNkP3+rGiPcC17Mq7Qd+7TGwfsfUdbNWz6gGZAxCh0Y2QasTfCqUkonPde5ee6eDl ovA4MYw1 4UXioxJ6ht93M4o+rQiFXdz0igpxravvixCM/EXBvQ5DmgkRC4wYCDTuMFQnuVlT/z2sfqGGB4y4OlRBJt8/DzAu4KBe/70BwcUDUiYuJ2Zng6kSWoPK/bHLCMF+lbrlVW/vqxKpI6wE4AwKlfkxub9nmqNtwbVaujWmhrZEZg47qbYNs1TjrkRotEB1gUtGKKVI7tT5sahc6pbWg54JT/TL2F1FljscOiJh5OBSuzmiTOFCS6uXIICg+wBZy+FxqWdICSk9w5OPcKbwD61QdH8CkdCqfLEW0ioWE0iydCaXbUYdiBhdRbRw4e89ROSgSokxiIgq5bQDMFLqw2y9A8QZg3VhflE9cl4UZmOHf999ZHpwGnia4E4FSthfkOkDwa/BbgEhSEc66KAADE52v55lBZI371rCw+tLuZ4szxiPI9HgbLcHPFukqBYL8ThoUsQWBzX13aoaor1JmC5tN/EDRmsD0OChc696sy/PN71W5oHOgxcMXv3bFJJntB17P/eMbFXmU/sJgXHri/QGG3804V+WHzhPgI5SrmxZVWBZxW15bqnYo+8Heqvm+X6BRHSGz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e1768a9395c9..689f5a1e2181 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -821,12 +821,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -857,8 +866,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -891,7 +900,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1086,6 +1098,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index eb3e35d65e95..57dc5b935f79 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index 2d97a17dd3ba..cc7159aef918 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6350,7 +6350,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index 6fa240e5b0c5..fbd7254517d6 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -327,7 +327,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1222,7 +1222,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1297,7 +1297,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index 61ed044b6145..24636a2b0acf 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -157,6 +157,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -389,6 +390,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 568c18d24d53..0cdc5f8c3d60 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -465,13 +465,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -484,7 +488,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -510,6 +515,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; } From patchwork Mon Dec 16 19:24:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA317E77183 for ; Mon, 16 Dec 2024 19:24:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4220F6B00B1; Mon, 16 Dec 2024 14:24:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D3886B00B2; Mon, 16 Dec 2024 14:24:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2251E6B00B3; Mon, 16 Dec 2024 14:24:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EAD356B00B1 for ; Mon, 16 Dec 2024 14:24:34 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A8D16B0366 for ; Mon, 16 Dec 2024 19:24:34 +0000 (UTC) X-FDA: 82901798322.04.C43A09B Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 980B0A0009 for ; Mon, 16 Dec 2024 19:24:11 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=h3Jgwu8Y; spf=pass (imf25.hostedemail.com: domain of 3b35gZwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3b35gZwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377058; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/Ia4skJ0dxk8cYc3IKUIPhlZF9heDgJL08xGmdJ5E3s=; b=VcxG63RMdDXoyez90s1RKdq8T8Cd1GdRKv8Eh1bQsdcGBhs6pDoa7JoOgYU5l2QSY/yKoS GRNLJuq8k3i9AvK5ojDy03bqEwfrI4H9R4cj0Ru1v4Xigdks+Ytp8OLjrBAeEhU+2YCGS/ P41inF6ats0yiCtKimzmmaqeksp/wxQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377058; a=rsa-sha256; cv=none; b=OcJT1+TeutToBkILl+Tf3jcl4LLKPPk1bdxjjQPFzFGmBDn2aAtFcn1DenDd5uPTpDrMpQ q23Jnv76Dda3nS2HcNKZN4ENHH5b0PMeOWgWdhyRYs5W6ivkBd4ZPT/g9mR8RIT4j7q889 rEq5aENpmE4R5cWlkRo4s9T2+6B3hRo= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=h3Jgwu8Y; spf=pass (imf25.hostedemail.com: domain of 3b35gZwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3b35gZwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-727c59ecb9fso3958002b3a.1 for ; Mon, 16 Dec 2024 11:24:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377071; x=1734981871; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/Ia4skJ0dxk8cYc3IKUIPhlZF9heDgJL08xGmdJ5E3s=; b=h3Jgwu8YZzhhOmVWKTredBhibKzZ5Ga9efJKRimqagRkvSV6S2rh8Th7SHHNGoFEuC 8/2k2a6XAzfPmMyAvJXKzcbKj0J2/vOtTg73b8fnLLpE+rki0lPduisg5GDo7SH6oDB4 44/3yY3jK3sjO2g3kpdTRnCVFwKNKSGj2uK+BbaHTPpvmY5gHPEI57pOq5q8F+WKnpUV vZu5gl8JiakegMdSFdE03Hhtw5eRnMgrRjexRTfQKxL/8yVchnoYyHk/uT2U1CiaZu1+ 1dJksyTWLN+FDUbtv3mKOuPZjnUdowyv0NdjHQYRG3rIO/xAdqxy1v2y+lOSuSP9oW/H bVTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377071; x=1734981871; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/Ia4skJ0dxk8cYc3IKUIPhlZF9heDgJL08xGmdJ5E3s=; b=OmXQAxWzEJjuyDbDHLKz52sx8pohMapheQ5+1+bNgQ7ehWMPIZG7eq+GaJ9U0jJu1p pg1WH/1ZhUgGvaJIMdkLonG0RKufJeNPhZ4Ia/rZRSPWZLzUdstHz0jaVlhCsiFYBaU0 xfZiniKK0SAskSP2/w1boJqPYERBIqNGklgi4AXEJ/Cgy2PJJfAZV6gPQv7UgPKa3NIm aCSU4NWIuYeKuQcwSFjCKiWEEaQda3oNjtft3KBkqbEv8iDvQa+8bgoQSL0DfqINwqqD oRo/CCuxpENoWP+OCt5iaRMMklk49YMvqVKpl/Q8smhZKMvJcZ2WH6JermS2Up/vZjuK MkeQ== X-Forwarded-Encrypted: i=1; AJvYcCUvjBCSQl3Q8Owsll4pBdaqy+SikuAhIRi4ulAn8vdEx4sdQkhWH5lJJzgELgLb546yIfLHoWWbmg==@kvack.org X-Gm-Message-State: AOJu0YxrG++dA2aQvs85SIxrUpQTOaYBmObaK02Mdul3O5hleXnv8EY+ tFjpujlYoW1Z3b/HLgUi6gY4uiuxCgF6kGoOIzcwZ/e/g0W1zh3MkVN0o5BYVvf5sPbgVv/cd1F NvA== X-Google-Smtp-Source: AGHT+IGblgxI1ekRLHe39odjzDuWa2GhKIvnP9JXJ082wXSRcQoL744G5O8IU0+gMQyAryC3i1beb9Fb4As= X-Received: from pfbci13.prod.google.com ([2002:a05:6a00:28cd:b0:725:ec78:5008]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:6c8c:b0:728:b601:86ee with SMTP id d2e1a72fcca58-7290c24330fmr18435707b3a.16.1734377071613; Mon, 16 Dec 2024 11:24:31 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:07 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-5-surenb@google.com> Subject: [PATCH v6 04/16] mm/nommu: fix the last places where vma is not locked before being attached From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 980B0A0009 X-Stat-Signature: afpcj1p3c85xkgnuzxjrienudt8ewxro X-Rspam-User: X-HE-Tag: 1734377051-505235 X-HE-Meta: U2FsdGVkX18gNIX2VdUp7+O9QCIRZ22uwVzKiGFLaBJlDozZ+W/HZjoeDgD9RRKH8sceI2cf+JaGLLxLi2sAVqr+EEHMHC//UelTeIsZDVW/GEPctK9uHk+R7uJpwmUbP9oITzni7nSml3OQ/OqsjPRU5JUybQ4aYVa2OSpz+CNTgx3lNpgYmbntwcZ26zphrz7gmsmKDHIzCg/5sVU+zsgwTsmu31gYSk0RVVJ/ankDjuIamXnevXx7i3uvuxlvS+er1u8C5mcRbyqyc9J0WTpGZbPWeBvRt8xD0h4PTfRiJpWo/pQglMuZQnUm8xa6JYrVHotTmNul/UqxDwQf8XcGJrKsbQOpQy8HxoBJjEaZpKQrAkrAyYv+UoB5UqH8C3VKr7CY3dbtb/RyqVcW+mfMqRBBFsGFKmzyF42hR0COzQmlut+b83KnGokQXYBal6ZHvhJXRxV2nnWc7//+U8e8cvGg7XndQ7+3GAu2cYSY5NloEuhG73ZYTOUixLlIZOc68CCpx3g5Sqk2plZRs6/sN4eUD63ICAJL8RYaOdUFavRxgjhkm38JNHhFmvzu4+NqApklB92vQALO0HX6uHonhBx3HNEQRF4mvpSeyMYpwjcJE9m/kSJFf8+4NhoBTd3dUGngmWYQ+6GSNLWjLZqkSA07mKl10XF9bd2A6MYl9UDHJkNA+2OdD2eJWLqToRHioPHPuY/zD8mnseVwY8Ou+Jd4ySFFvL+v0leIlqpmXJ08t1n4j+XgqQSbRbdMF5GvdXFre4JyDRGNQbqNoBXKOzwtoaemu07ZUoEjR2jfSFN1/Fp71otYOeqq08lfbpwGrCRjQguIHMOIlC1w4nkOBI04Eiw9jYKKuo78vn+K04YieuWevg5hErAzIZQJKJA0T6Rgph01HpYPrr4T2qHTcyVx8o0euTLfpmt6zZHTo9od//D3w0PyyEVcWbpRopAAQFHrjBxWPTVsjnS nuqNMQhO B0rHu/7521wSxR/4kdjgrP+AcEtdj5JHok1CjRC02jLDGmwskr+A4aJ/4HvaAWt1a7030MyDCtd+UqtOLHCkwvAV/w7iHr6/tVcSfiYjQXyqrDrm5LZ33zjlPN0PyskwGSjQAVrm+C8rkL+wnWxs9y8tqyFylqWQKnw49fYYYh1F+cGqWUNbEuzb9IYNIrpocfzW/2AfINAGU2PSftUkgBt8i4a+rMIgHEjXKZ4PG9Ex3uFQyGVzkFyWgbuNsYj0Y7i1mXiNqiyxfl5qCEzdAUsfSfgQtBdycX0WopBfZpNIlBRtWtOARK+ZHFEWRwb1htbe5sNKNma9fKW/ij8YMmJJ8cT+AI6K9qGqAj7IPfs/RnNJpnOnWSWA10XR1S3Eqywsl+Re/u7D4fZfF+fsWQx5u8EMR/uMMsiThGehojQ0J8fqR2rfrTQb+Nc4+8kIUMdEM1fSZTYDnK4bScscDzxSDiwHYmyYAxISlTxOF00G/9s+UfcKsY6Ilc30NqJQzZgoYyM2RF1pNnDU1E1TDIS04JZwF5+vjmoAJhoqTWVbLlCPFxB9RGS6pwr185fanqC0cm6aWEMejtX3UCey5PkcTTBHwUOvFwfyH X-Bogosity: Ham, tests=bogofilter, spamicity=0.301470, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: nommu configuration has two places where vma gets attached to the vma tree without write-locking it. Add the missing locks to ensure vma is always locked before it's attached. Signed-off-by: Suren Baghdasaryan --- mm/nommu.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/nommu.c b/mm/nommu.c index 9cb6e99215e2..248392ef4048 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1189,6 +1189,7 @@ unsigned long do_mmap(struct file *file, goto error_just_free; setup_vma_to_mm(vma, current->mm); + vma_start_write(vma); current->mm->map_count++; /* add the VMA to the tree */ vma_iter_store(&vmi, vma); @@ -1356,6 +1357,7 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, setup_vma_to_mm(vma, mm); setup_vma_to_mm(new, mm); + vma_start_write(new); vma_iter_store(vmi, new); mm->map_count++; return 0; From patchwork Mon Dec 16 19:24:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08EF5E77180 for ; Mon, 16 Dec 2024 19:24:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B6906B00B3; Mon, 16 Dec 2024 14:24:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 88CAD6B00B4; Mon, 16 Dec 2024 14:24:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BB226B00B7; Mon, 16 Dec 2024 14:24:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3DB1F6B00B3 for ; Mon, 16 Dec 2024 14:24:37 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DB469B036F for ; Mon, 16 Dec 2024 19:24:36 +0000 (UTC) X-FDA: 82901798364.11.343AF1E Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf22.hostedemail.com (Postfix) with ESMTP id CEFB2C0004 for ; Mon, 16 Dec 2024 19:24:03 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="rpLvz/Nh"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3cX5gZwYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3cX5gZwYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377061; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0ykFNL3Q77su3l1hmkCa41V3LHYM0KyunH48W8conCI=; b=OpgfhREQ5tLs4ao1sgkjcbR8joN+XelbmL67jphVYfbgRl0oFTp/a45mNzapp/xBaDEbbS 2t7B97O/dGwNVhlas4zthC3iMvMAUdTAF1f0uiySV6Yyg04+Sfmf15bGkQDV9IpgsjnniT p7eW5wALxeYH5yu1ZwjE39WoZp5d/vg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377061; a=rsa-sha256; cv=none; b=bn5tB2Zv2LKxf9akIg8P03judD/QCrxfCgpkFccXehSPe4PR/zmoQGyuQNH4+KMrJgYL75 SOMbVdGg/rsAVDGVnr1+4LC9ZiMqMbzdJygIkX3Je1pknVQv7e2AVMgLRCIQCORLFac3iS lom1/ElgpGkc0K5eZJmhIXycLUST7ZQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="rpLvz/Nh"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3cX5gZwYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3cX5gZwYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-21661949f23so62981465ad.3 for ; Mon, 16 Dec 2024 11:24:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377074; x=1734981874; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0ykFNL3Q77su3l1hmkCa41V3LHYM0KyunH48W8conCI=; b=rpLvz/NhUhULxn3Jn7RjtiD/DOXmfKwrZvwBDNw0GOKdJlxqVfiINoiAwp00tHvO61 /lbCiacG1m4KEWHcENVF+pn82LZWG1m9OJNTRDvKma5DbPrfdIKFs318Tt0xO1OtWBey vJbuuvGMv/gAv3BmYLsbY9HYEInfgZqBbNSClBMjq+skQ75xsNyus6/+oQzUNHrlsyV3 18gHIZG93znqFnFrweMBSUYWvMqjxSY0oeoGU0+mRHj6EgDCf/Fgmp2CaWUJgZFzMmVk K6Yq+XeIlYk4Air18VMxLq3RR+9dhlwRG/k9ejhmHbjJXtS+7DZwClZEjPI+9hQwyj8w CxRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377074; x=1734981874; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0ykFNL3Q77su3l1hmkCa41V3LHYM0KyunH48W8conCI=; b=awCWqDXXpd8jGKK9yN+d92VqGbWFSQADF5NJVONNWou3NqvCElsPHkrXlNif+ie/cs q31ofqJ/5rZc3TfW/x55ZJQvmH/bNALO8Of3H7jgj//Ch7rMry0jLzGpRQXOZiq2+HzH Jt5ydm0LuYCcAllDI/C6sN6RdjWVejKtYKsiw/aS5DONOuAsSyP/PDprxU4brcdolmBL dhnR3TTYs+4xa7bcWhThYy23HV1FSDS52cpgW4xSjEtlNhrR+nGH8vhAe9g0mwLJ54pG caTa1mCG3/UUpuyeeHfF1Yb35+vAWHn+fI0OFNEwNDvYMxLYuQaKtKOfiImNv9BTdq6h 3RaQ== X-Forwarded-Encrypted: i=1; AJvYcCUzPvfawSszDGX/zrygpzYo4axc+zPzKrwksL1Yq7Jm0HMlex2uD+DIHVDl7eOBBhbg/qTg/8HJ6w==@kvack.org X-Gm-Message-State: AOJu0YzbvAo2WF8T5w/NupUNh0ruxsJ2H+uXVkbYJuSLDlwP+sLfoqBq E7dwL2HNPWorGRcOirlLU3k6jF3n8u1Zo7vvNJUF4488UwEA80PHmrCFLTIPkwjzHYILrGgiYsR s+Q== X-Google-Smtp-Source: AGHT+IEVw0uJl7cLZP7L5C6hpMFvqcbjARbisNWJ3VdxfhR00byT5nuQon+Itb1c7QrUoZSJH5F454tTzFQ= X-Received: from pgbel4.prod.google.com ([2002:a05:6a02:4644:b0:7fd:40dd:86a5]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e80d:b0:215:b5d6:5fa8 with SMTP id d9443c01a7336-21892a441admr219433885ad.22.1734377073676; Mon, 16 Dec 2024 11:24:33 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:08 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-6-surenb@google.com> Subject: [PATCH v6 05/16] types: move struct rcuwait into types.h From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: iczfc4i3yoy8zbz8xx7ck3guek3iejet X-Rspamd-Queue-Id: CEFB2C0004 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734377043-344080 X-HE-Meta: U2FsdGVkX19K4o9SAXAjAtvMKNrS4RmsisxPhN89FLxPl8oinIOfCN4QPVXi9s4Iil61A7gC6/6XSoZxkDaB1C9WyXVb5oT/s3ttFq1YntnftpzGVL7pDp/xfej2PCKF1F/jy4DNyzSHMPC6+IQ1wXO8rCHwWry0Fbi7HJ8fAKOeA711sqJX/XsynYsNS/+rHgj+zIc6Q+jUkWtZ3E96MHw/KcXiaIKnL/AsxR3TDkkZXIVi+q2IN1K5F0zdTtS5+2vLLicoPRtpaxBlT6WbQ5okA5y6678HDii8HuKjZqqiD2NxrHts9zHNWtlsqlBRVgo3YeQ2rk9Z/GobgxIyXV9mYUV7E7O8Huq1qpZeooVUP7OCH4oOYsjVsNg/ZBaFo5FA/vQYj1jwWjJdHoMZ5flJPlyfhgOxuPCkLT5ILeDxH7M+032CMkD+Shg7Cl9qZDgb1ZTBE0LO6g9/15SHo/92ugAlMsXPhIla1eH2vDojpuWznoWz2tuDZ2F4Zt4B0EuwlObaAajjRy7VhGc6Z6LZXErrQd96hX6AilMxY3BYGpSnRg4SK49AzgQxCmHOkCUNLtgyAgWR+NHeCmiPSx5XOexpHpinoNO+xEeuu0EiuYN9g4Fkfj0679HOvjL+vn2BSrWcpC5S2o0Wa+TBza7KS8uC71sdWgnwBnfqUfBQ7RspUUai0iy2GpNklwNKa5qIQcTh63+6zUPHN/Lz4Tzdr4r/1bzw6790IXVxtyYvaQhzd9vS7NjlqqCOzh1CetA0Ffj2sgr0wkq7ZOSm4/GwyO5mV2n2X1bIxZB2NIaUplrSYtrbnyteSqN5sIx5JWCIgWmPoGZYNmVZvpoFaDJOu1PuNTvIs4Zwzee7PNH5GkagGYAXJ8CmxLxFFsUJw2qQVho/uhYb1M3/b41Xyiy8QFUWpacDGCweOvS7HiEnjBuGe5H2Xq3V8/R13VuvZjg75PXFWrY2j12fJ/2 aLohbh/P JGcmFTcmNa46DcC882VM0d7l3aCQwJzIiwtoRoku9dj37Ndr93Lod41LThkNDrCJKRb8abQRkgbu4bHT84fzvxsz1KoPjuxygcd+cIb+oBWTWahBeIqOvcM5es8qlA1vVZRHVeXgc0AtCO0Ols0Zn3Qd6cF2Fga3Ckpj0544tHCm1FHqnonH0DcQPfb7KvvQt17IqcJA23//tAW/eBHZgmyBjD63Yl7T0W1GOou6sRt0HMoNoL+VtpQCikdwjZJbrKqHPS0mJpyIfFdbVxMFENVQnYlTUwlGO1eEQ2moO/1qR4N5149j8rGutmEQTISAv92BRFHFPiucj/8O0a/1xE9gSzgJtz5Vr6gKgd29JU4ZZAEmbkLgcsol1Y3SySxp16DYH+ym1OAfkNv+PXoJI3GX5At7+tHFnXZu0z52HuBuvn+hRgh+O09lDZ2X0yNuPGNxh4v/0AuUU/Rg9+tr6Kk0YMBlInzBQUMqyXlqmocXuVSZYrtRgk+uu39bXjuIFHNW/BV8utgg+1IowdaP6wgYmhLIiMdVTZ9HXE4FwA7Z61a0tO2ekdk7/FYQYiBbrJATKi3TpJF9w+c5YT+0L99t0imoYCvs7HtSL X-Bogosity: Ham, tests=bogofilter, spamicity=0.066840, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move rcuwait struct definition into types.h so that rcuwait can be used without including rcuwait.h which includes other headers. Without this change mm_types.h can't use rcuwait due to a the following circular dependency: mm_types.h -> rcuwait.h -> signal.h -> mm_types.h Suggested-by: Matthew Wilcox Signed-off-by: Suren Baghdasaryan --- include/linux/rcuwait.h | 13 +------------ include/linux/types.h | 12 ++++++++++++ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h index 27343424225c..9ad134a04b41 100644 --- a/include/linux/rcuwait.h +++ b/include/linux/rcuwait.h @@ -4,18 +4,7 @@ #include #include - -/* - * rcuwait provides a way of blocking and waking up a single - * task in an rcu-safe manner. - * - * The only time @task is non-nil is when a user is blocked (or - * checking if it needs to) on a condition, and reset as soon as we - * know that the condition has succeeded and are awoken. - */ -struct rcuwait { - struct task_struct __rcu *task; -}; +#include #define __RCUWAIT_INITIALIZER(name) \ { .task = NULL, } diff --git a/include/linux/types.h b/include/linux/types.h index 2d7b9ae8714c..f1356a9a5730 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -248,5 +248,17 @@ typedef void (*swap_func_t)(void *a, void *b, int size); typedef int (*cmp_r_func_t)(const void *a, const void *b, const void *priv); typedef int (*cmp_func_t)(const void *a, const void *b); +/* + * rcuwait provides a way of blocking and waking up a single + * task in an rcu-safe manner. + * + * The only time @task is non-nil is when a user is blocked (or + * checking if it needs to) on a condition, and reset as soon as we + * know that the condition has succeeded and are awoken. + */ +struct rcuwait { + struct task_struct __rcu *task; +}; + #endif /* __ASSEMBLY__ */ #endif /* _LINUX_TYPES_H */ From patchwork Mon Dec 16 19:24:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 676FCE7717F for ; Mon, 16 Dec 2024 19:24:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CC5D6B00B4; Mon, 16 Dec 2024 14:24:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 72D886B00B8; Mon, 16 Dec 2024 14:24:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CF096B00B9; Mon, 16 Dec 2024 14:24:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 34B486B00B4 for ; Mon, 16 Dec 2024 14:24:39 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E0B66A0A2F for ; Mon, 16 Dec 2024 19:24:38 +0000 (UTC) X-FDA: 82901798196.26.C200486 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 9275740011 for ; Mon, 16 Dec 2024 19:24:14 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zLW224GS; spf=pass (imf01.hostedemail.com: domain of 3c35gZwYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3c35gZwYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377046; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1gkuZWwQBkEGAXVOS6gJMPBIIwpS2AmmvzoNNuLoLbs=; b=CBYhxc/AGfMkqTtufGlwj2uECYnybgQL30pdrlgHeCP+hm0rDOS49yP8ScXDHq3RX7f63C ad4jY3DECSaB/fHzR3ZAHtXi6pqQm5f3rCYn2bMy3pfGn8QD50PS60ds6TIY1tolb7PMCJ aSBHQ8/dnW3xHDq85twQElxX7pZqie0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zLW224GS; spf=pass (imf01.hostedemail.com: domain of 3c35gZwYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3c35gZwYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377046; a=rsa-sha256; cv=none; b=rHmXdix80PL5SZT1yYlO2ovFZJ38ZfO3nY9eMK6u/DkDUjqX3wjrBybrzrWO2W/cnofR1h Jfmz6vnwLCUJPXbBr0jR75ig4OlN5Kp1g2bs/ZdFF/f5ExST5CHO2gRt3SY4buiibsDboR 0upSyVSjrsUgr3SKlggLjvhzeTUAnW4= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2163dc0f5dbso37053375ad.2 for ; Mon, 16 Dec 2024 11:24:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377076; x=1734981876; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1gkuZWwQBkEGAXVOS6gJMPBIIwpS2AmmvzoNNuLoLbs=; b=zLW224GS/hz7LVhQYCzdt8WaKQH8KZGInp9G9CWiZqaz/ha7tbgBvBtZV+80cRJmyJ HahHU8U3yK569DypN74+/cOlo/5/1hgnnDPavjIJkqV1fhdDHEPANDhInLiWKKSEbeS4 GB7pcGGoXsKo3jS1/sV+EaxtKmCxnT2lcC9Teh0wbe6C0Z/NrzdAYJZiW2+THJVRVRVh VkwsBQKRE3Ii6JA3fpjUD499rI3o2zGx2wK/04wjjJJheK1i9xvY+qNxMyIGpWls83OD U9Es/Q1rE2n30pOMx20nagY9Xt6lG1RI7OqWt+b6jfL7VT7hPIxNmR6nLFqFVzjYi93A HYJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377076; x=1734981876; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1gkuZWwQBkEGAXVOS6gJMPBIIwpS2AmmvzoNNuLoLbs=; b=GewsbBNwcRYOTKhdAMlEexKQFrd0+dE3RNslekhAKNSXVbT9/FE5fD5woP9QcesU6l phU+NSb9O9pokRR4Eg17K5rvJtdbdNaVvy4V2OIeQQnFkfFax2CzFAfMrUE/HHQtJlAR UJBbmH8cXoXsuQNzTHXJ0TlBod1UrhYTdODJX4rIQGHT9HQ2hqLvYwpR6OODpWjrQ/H2 NLYcy6V7Z/eWd7L1wR9A7djE8RCxpLdn8NVu+EQwFN/y4nUULi+dZWLPkGt072FWpHS4 LWRTnsiar7YuG/FSNEyV/dg+tPlwawn6epxwuM+YODGMRuFIOLqQMCmO35V7p9e+lKVV IplQ== X-Forwarded-Encrypted: i=1; AJvYcCVNEkBh+m/dvyTpsiaweqsuVENP5JeUADuvLyKapKjZHuZXQjvMrgNZWHF0JRHqIQL7fA5hUoMQKQ==@kvack.org X-Gm-Message-State: AOJu0YyeW08r9ZmdAo5vEQpvvOMVa+OaXZhE8sTqRECaaahVp8LRgMFs NsHWL/S0CUUbIDQCel2WwidMXod0cqV7/JjSzzHplBv+hwx7ENlQ35SRg199uKcRXtwdEBVWbd7 vPw== X-Google-Smtp-Source: AGHT+IFRpgE3svFkQGxxP8t9jtfuJXCqP6DbanV+kjCkw5VL6vSIPipGx+mrpM9/g8/yEqYCZ44EecENvBs= X-Received: from pjbsp4.prod.google.com ([2002:a17:90b:52c4:b0:2ee:4679:4a6b]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2744:b0:2ef:33a4:ae6e with SMTP id 98e67ed59e1d1-2f2d7d9fd46mr1314965a91.12.1734377075865; Mon, 16 Dec 2024 11:24:35 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:09 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-7-surenb@google.com> Subject: [PATCH v6 06/16] mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9275740011 X-Rspam-User: X-Stat-Signature: 3xaxo5kcoinfk9pktzryfwxajoqhdtcw X-HE-Tag: 1734377054-174457 X-HE-Meta: U2FsdGVkX19WSsC8ohjXOXC5U5lklMQkqUI8gVNY0pmBIk8L2LM5ZX5NCBvBFjxprJHFVrFNxzSIgoAXeAmETvWLRZJNxCPl8rZrtL1RvndKKS8i+65LYh9xY59cZo6mIGjdM8S3x1kAHvrMs3+hIo4es5DZUZH9VKM5vrz81wTTWvZoxYNwa5Sz/SOqR+mx27ro8QPPyivfuNSZypjeu33EwgZGdXLUv1IFwwfirpo1qHOjoTaqTb+jNPQ5AofPdIkz7WKfctAdpiEG9se4zeqh1kIj/KL/6iP+Ez1OAO91R9J8pJPKPYwZTJVBJ8lMGxYBkGymF0KegKWT8ja5k9OxSNfUw/S9V5kPkDyw10NXS8jAlH1RQ85Sdx7xrM+1Huf/HX3Z6UnzxGuw/EJFePp/phwBMcpROl3XUs4id9XpKjCxjw3bLLThlBpbbztf6QVFTVAEuNBfLnevZ9LcJ5FW61uYKbqLyuKyndYqcnq96Qq8/mBZZ0Wj4yHMY8ZUnL51KtL7sZRO+QiiLngrTLNREaYcmxFeZC/P85XU2iCXNrL9i8lscW5qs8CooZrPqaHnkZeJk1QawkHbLRFnkc+v1UMNLNBXZm8h1+ApY5HFnWETk6rYjoZbwNl14jVOqGYlZqIIKFmJRerAkvBwWMdIz5BpJ9O8WhHMTjRpDAvLBJyO6CKdAMn2ME9TnbwPmmwHHNoKeJGQUP7mczxfDp0CDEOquQyWi2HYRoOEdwNI0BFBerKPN/pYjsERgF8K26PnmhFrvnuleJnaO+pmL05wRenEN0UZdIvYlChn3d7lMpVREEjRCLL4n1KI33Uz6u8YSZhQq1VMnSLdmMPSscUTVpjClU+eQUMptkwptbm1riYqF/DhNFyUs6v/o2J594g/+RVT/335+gsao/WotAkVKbCGNbmrizE11ILB+luzYXqqxKgfL5VODxeUB0M4tLYAPR160CYDN8QgO9h I6hnp+Zc 02orRpSHRS+czrtrELnuFI6oRrTNjtxW/E8LwbgwvIye8gNI7/y0ECVZZFy7fbm5Gs2cmaErj5m8ccN5tlSOkfgVTwHzE1B2fMwHERDpohOoiy+Lh7nV7tGnLjIgBc3t0bZfDW/m1QHsisY9/l8NrL5mMYJM/SayXDnsLYwfXjiy02l976lepH7DOuv4mgQCplAcMplNmLboPoe13l9e5FufVxIwBNxIZnN1fFpfBilejmZGWf9VA9m+xI7e7fGsul67P62Ayac5055FOfPJ5s3W9kNzTdfIV3XJeoMuOSsEjll8P6RVBkTmqWCJkW7PFmfpPaJcaw2S9wChhF5+s3G0pROM6m5yWThfpTp21FXj0WCfJ8vB+aXwJvxnAJQIC+i5URBACNr8RSw71M0qXDnT+bNARiePKGc7PZVQkk9iACVuE7I1UetYT2GqYazP8srH+UjC2f1elesgF0AyNOfEvasiNezLzTzoyuHtMkTgFfeVR9lp1e/Eg4DLGB6/XNAAckIHOBPFSqnKSri9SNtVjU14SzZF9NnEGAxe+dIbuN8qgqbVN/NiBp9UFmwHe8PKXQqC7pCLU8GuMw7pDAFbhPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.158876, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With upcoming replacement of vm_lock with vm_refcnt, we need to handle a possibility of vma_start_read_locked/vma_start_read_locked_nested failing due to refcount overflow. Prepare for such possibility by changing these APIs and adjusting their users. Signed-off-by: Suren Baghdasaryan Cc: Lokesh Gidra --- include/linux/mm.h | 6 ++++-- mm/userfaultfd.c | 17 ++++++++++++----- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 689f5a1e2181..0ecd321c50b7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -747,10 +747,11 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) +static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { mmap_assert_locked(vma->vm_mm); down_read_nested(&vma->vm_lock.lock, subclass); + return true; } /* @@ -759,10 +760,11 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int * not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked(struct vm_area_struct *vma) +static inline bool vma_start_read_locked(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); down_read(&vma->vm_lock.lock); + return true; } static inline void vma_end_read(struct vm_area_struct *vma) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index bc9a66ec6a6e..79e8ae676f75 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -85,7 +85,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, mmap_read_lock(mm); vma = find_vma_and_prepare_anon(mm, address); if (!IS_ERR(vma)) - vma_start_read_locked(vma); + if (!vma_start_read_locked(vma)) + vma = ERR_PTR(-EAGAIN); mmap_read_unlock(mm); return vma; @@ -1483,10 +1484,16 @@ static int uffd_move_lock(struct mm_struct *mm, mmap_read_lock(mm); err = find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap); if (!err) { - vma_start_read_locked(*dst_vmap); - if (*dst_vmap != *src_vmap) - vma_start_read_locked_nested(*src_vmap, - SINGLE_DEPTH_NESTING); + if (!vma_start_read_locked(*dst_vmap)) { + if (*dst_vmap != *src_vmap) { + if (!vma_start_read_locked_nested(*src_vmap, + SINGLE_DEPTH_NESTING)) { + vma_end_read(*dst_vmap); + err = -EAGAIN; + } + } + } else + err = -EAGAIN; } mmap_read_unlock(mm); return err; From patchwork Mon Dec 16 19:24:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2FF0E77183 for ; Mon, 16 Dec 2024 19:24:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D03E6B00B9; Mon, 16 Dec 2024 14:24:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 77E316B00BA; Mon, 16 Dec 2024 14:24:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F5AA6B00BB; Mon, 16 Dec 2024 14:24:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3D1EB6B00B9 for ; Mon, 16 Dec 2024 14:24:41 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F19FC4506C for ; Mon, 16 Dec 2024 19:24:40 +0000 (UTC) X-FDA: 82901798280.22.5404AFA Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 309141C000E for ; Mon, 16 Dec 2024 19:24:07 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zeq2Hkdp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3dX5gZwYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3dX5gZwYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377051; a=rsa-sha256; cv=none; b=2tzgC1vG854qbhpNKRUjG9Ih6udCGNuWKlirjHikkDU78iKT+SqgYMxXa3T3uctXBy8Cr6 EXzsgwjaXCRzyq2BeM7IF18QZKHSOPmdKEUrfJQOE0bAxS9jloPfsivbSJWXuRcdLfKAG1 /T0Gpf4xdKNY4bc//Ru0u+Y2q1R2AL8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zeq2Hkdp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3dX5gZwYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3dX5gZwYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377051; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V23hsekxf1oBAjhtCt33Ew0gdChPa6B6f0O6pWk1TGI=; b=SqYUMkKN10xYwcyz2fmSuxxX3mqIfoXYmMrFtdpxVMZjakry3oCk14M0sHwRqtuhHbhlKB 6w7oFEcw9FGpWk++QGhIpMCF35uNrAA67uI0Kj0Bhqsi/Bhl6Oj4Pk6LU+6uvGQpbFGBcw bJg8z0K/HBtE8963juBoHyD4cMJWdAM= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-216717543b7so70323155ad.0 for ; Mon, 16 Dec 2024 11:24:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377078; x=1734981878; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=V23hsekxf1oBAjhtCt33Ew0gdChPa6B6f0O6pWk1TGI=; b=zeq2HkdpyB/h9D+mdH25kIwXvcIY7zNDDdkhfbFBH3YSJ0RMrQ57+JoAHZeALL9nqZ q2YU+es/ZYFQfZGxruah4bUCvDOjZt7YIAx5pmMuMGHNyo2R4D4XD6LHLLqdKk1Wlfo5 YS3VKtJvSREFBxfNw2ZntYtLUUW1FaHOhaCqwfGi5m4mSvUDDy4xtJ4tahyaVNvqwDUA kPt19jwnsT5xY0WfOD1j4c9/WIKRW/zIJRg+qfSdoL5unp5+2YwoU/ss9YmIZ0hmI5yA VbvBAt5uYrHwoGoHZQv14o+4ZtzwPpvoLYEO7mCsSuTvcmw8dxRE4xYXTOS8nDN/P3Rf mrVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377078; x=1734981878; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V23hsekxf1oBAjhtCt33Ew0gdChPa6B6f0O6pWk1TGI=; b=pzmkVfKmEUvc9bvAWXBIS2supGbAAiWMEJLSXKiBg107YbYbi81m/O8AACOFw99fuk 7t2CrkM9taidCnV7/qsisNcmFuZ+d4k7Dk9iWLOhIHk5zufsJrvqw2z/OZsMx5VzW7Y/ JAm3gzPLk8KEGdfMk5sBjIgNTUjlL7gLY7+Bh/NHhzuJtJnOR2yEuUUYWjmax2IaviQu 2q14z7+JtU6xMQdRpfyNbC2LjtUWt/YfqQ1u+n26dS628m/blJFvhY7GZGBYUsaplilz U74kROtDkYOebN02aAXK1X0gFMULlPds4SnJYJwlbSGe1XI5OmTP4bvYREk9k1xtaFVA QJ/A== X-Forwarded-Encrypted: i=1; AJvYcCVNb3GGNpSlXzS+rLLi4pyNv70080Va0NPbi3kIc4wvvuhhFUBE+bNOY9A2GHEJ5ve0IL3PfJJ+dQ==@kvack.org X-Gm-Message-State: AOJu0YzWE1lW9woSMdW8WUwI4l4B0yx36sxSswrXHT7bKLlJEUWiuhXf c9SPaYui+l6s6XVRx9lKr4sJHlIfk+h84oeEseiE+tln4szTT4b+kEARYTLE76IYP9hA2tIUZwA KqQ== X-Google-Smtp-Source: AGHT+IGOYBhQK/OMn61IKHhYYRKFLZqMwRmmFqwVG/Z+7upwVvTEn0G50CkD9wYvokG+WIRwbcBuoK+avFE= X-Received: from pjboi5.prod.google.com ([2002:a17:90b:3a05:b0:2ee:53fe:d0fc]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:cec9:b0:215:4450:54fb with SMTP id d9443c01a7336-21892a5c02fmr173665245ad.55.1734377077933; Mon, 16 Dec 2024 11:24:37 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:10 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-8-surenb@google.com> Subject: [PATCH v6 07/16] mm: move mmap_init_lock() out of the header file From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 309141C000E X-Stat-Signature: mpntttsu3f5zc789nxajkt6k6bs66znz X-Rspam-User: X-HE-Tag: 1734377047-369996 X-HE-Meta: U2FsdGVkX1+3/55wrZzDhIufgWqocDUKi+IGvdPBiMfYnyueFWv1GFZfsLS0NTILgySsgu6Glb8VqGJVsiM7ynLZ8Diri+SvayDogGovsmr4PppJOCIoAQ5QxkjYDWJKW0x2BxqAmcjE21bCSDflEYrj5bXPY6sgeHfYS+nwuO6o6xdHzZE1Gj3eVH3r7v2fNa/6Xm7CJsbugDJyjlvj2xIIPVniswdeO7AkKeuzU4GgtQmI/Mksecl8NUL4Z4BW7hDH+nRcI08StCzC5bodzhsHWkH9CAfYjHv7y/HfVVBG+aw7yuRZpa7Uoc6PMuv32eSSZKtd5GHdgIP+FzAEjE79zGy+nhiVtuyo4Z+50rQgmp4b9jhsDVPRD0AAGLByBcYdMpt26GYtXMu9le5t80uchjVVWBdqZ47w0qrjQYsYJuYvyxVllCpa9cRdMMWhCKhUiaOyNUaKmNLDU2MgyDHYa7LGCkmmr3eJyHNo2KfHaEb4X6aaFPZ+PnqYHJhRqlD1Bap1t1h5SRwbrU2qa7qWj4Vsz9qUM8kKCxRwx6oK95DKcmZ2alGAe0ZRnj6BmvuvjHc/RxXreFBsrunKdJbtU6ljyrPk2Ti6Qz0E/BCf4dxAMM0YXq8xeUVEPj7omvOIs+wzuhOo5/hXVlFk/9X/WrSctMkV4/6TmKgY1SwW1U+Gn9+bQ/JM6buGke6q9GWYFtn2ffCxkMwXNGN7EaVYDBdXeuTQV/CAZZJnCOoQEwFIdimMv2kwGIPllIj0MLqwRIwSN1Mj9+3HuLQGvioMhwmg1souJRTii3auLjtUBf8Z6NDYkrcdShDqOpwjgdtiZYKbVE3FypYLMQoJMBN+Jb6ZUwXtylm1nOJKVtQTIEz82V01jBI3Cy9i0xRODOB/0aLEXD+8Eukz+9ZUnlpwGQnX8rdX5tf01EsbBj9nr/SLEkNSL6Bs9Y/9uJvHpMyL9WirUSpcVLDlUiR BVrUFrLt ZVm6dg/lT3ovkGempBjKj+vLbK4Lxk9TBfEJ28iAOGLBUbfKsw2GpUdYn/OfU9ny3wZoi8ouaDSLwTmIdVMJECHZ13Av5Bxnfgaj5o8UCIyrDdBwzOwKY5fTcLvFoxEdh98Y6QbLwqo8XLHpJLieBn49Ismn85l6lzSxGXFfAjuA42nscuctsSDwK4Oiys3ZUYoYa0bHyKjq343fNZYulDp+n7L+chMgQkMli59X7/ci5jDK/FDRTh5iGBOTP6UXdbrM4XT2yCmptHgpoYEgrJ0v1H06TFjNx6sWO5Dj/hVrm99zhnTn8pPie0J9ASLuzvC64ZKEV2rXAN1HcAjWtEg3KJb0saMnBlTNAJkxQALafeI/u9oreYP6xXsgWohPT/Da0zvB+V5oLfUiPPlsd9b1IbGj/pz0VV6TJXNLtxPX7F0im9G9yu8052mtd1WsFkerQqik3CVCjI6TjOmfX7/7yeqwgzgVRazX4Z/9CYMEBxeRE57sQi2DQZTcc1cae/0Ozyf90HxSgbFDKFOX6ndtvV+KU8K5qJMAEnDmNkuLHD50JGGTOGJXnLEJ3wgtI/8DgP/ADyAY/IT2+EDyJSOPt/g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.038414, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: mmap_init_lock() is used only from mm_init() in fork.c, therefore it does not have to reside in the header file. This move lets us avoid including additional headers in mmap_lock.h later, when mmap_init_lock() needs to initialize rcuwait object. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 6 ------ kernel/fork.c | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 45a21faa3ff6..4706c6769902 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -122,12 +122,6 @@ static inline bool mmap_lock_speculate_retry(struct mm_struct *mm, unsigned int #endif /* CONFIG_PER_VMA_LOCK */ -static inline void mmap_init_lock(struct mm_struct *mm) -{ - init_rwsem(&mm->mmap_lock); - mm_lock_seqcount_init(mm); -} - static inline void mmap_write_lock(struct mm_struct *mm) { __mmap_lock_trace_start_locking(mm, true); diff --git a/kernel/fork.c b/kernel/fork.c index 57dc5b935f79..8cb19c23e892 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1224,6 +1224,12 @@ static void mm_init_uprobes_state(struct mm_struct *mm) #endif } +static inline void mmap_init_lock(struct mm_struct *mm) +{ + init_rwsem(&mm->mmap_lock); + mm_lock_seqcount_init(mm); +} + static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, struct user_namespace *user_ns) { From patchwork Mon Dec 16 19:24:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83043E77180 for ; Mon, 16 Dec 2024 19:24:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC7016B00BB; Mon, 16 Dec 2024 14:24:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B77716B00BC; Mon, 16 Dec 2024 14:24:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A13ED6B00BD; Mon, 16 Dec 2024 14:24:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 803E76B00BB for ; Mon, 16 Dec 2024 14:24:43 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 328861608EE for ; Mon, 16 Dec 2024 19:24:43 +0000 (UTC) X-FDA: 82901798028.27.F89FE70 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 41BE4C0004 for ; Mon, 16 Dec 2024 19:24:10 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UUrek2xm; spf=pass (imf22.hostedemail.com: domain of 3d35gZwYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3d35gZwYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377053; a=rsa-sha256; cv=none; b=gdNY69I5fuokhcYiR3vP5gMxu8QBLglUh8noRl0Zj9lyzLqtMM8zIdhb69kEnxQyqHb88R opx+N1ntUrIMF/zpNZK2wCzRfOPswB99n69QUKsQflB/JvfNSzoWIQtN/RZrEVdgbL+rcV OHKNU+Xz0Y2Trl4U0ZJT6QoDqjhlE4U= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UUrek2xm; spf=pass (imf22.hostedemail.com: domain of 3d35gZwYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3d35gZwYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IlSIABx/zxxF1ijBMV6mDoRgzimHB9QssQ8SIetRkVU=; b=vVkebs/XRF9Z76cMmqkHdvHBmVsBcSrulxXsvWKhpCaRwmULXNekzr3A5ENOyqby+1YhGJ Mqdip/vOd0ZO5qSktnFkYnA4L/Km1TRfF4iwTl1kn6DQsxKxRBPLFy8uy0ueDwMnNXWRCg VpYPBRDAyc8GsL95/OgvmuBKLyrK8Lo= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-6c8f99fef10so4980635a12.3 for ; Mon, 16 Dec 2024 11:24:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377080; x=1734981880; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IlSIABx/zxxF1ijBMV6mDoRgzimHB9QssQ8SIetRkVU=; b=UUrek2xmE6eXiAnoSjOt1+8VeqRyJKojXydaCr6I+tvlkZ+aw6fcxAZYgIfqu98BcE ly4I33YloPRanruInoguqiGHiW3A5J2pdI4rRafXW/Ka2C5kfPj6YubGG9Sjx3eKiYhn IHOyVe+kfdmxPexeiRRU0QWHKUb3LYMklKDAbyNJiE10xH1SbzaRR2MAublShY+Y7snI IgJhiDI7kU8bmS9ru1wKwUuPtChfu+sFYAq+SXNBpchOVwyxEwkUYtsP+gx5PMDYdW05 gnVpSidXY1ZRXTTHgWNY5ZAKg4Y6e7TJRc1jeZ7EDKpFviese3vNcNe7jvThChkoXPuG lGwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377080; x=1734981880; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IlSIABx/zxxF1ijBMV6mDoRgzimHB9QssQ8SIetRkVU=; b=VEq5ZaGUnm8cfPLcGTlH1KoxXics0qSbbXR9H8SoJ4r7OOMzHfYxGBe1HwJzWd6uoh BbVZTfl3ydIOjJkek5lrDJLABrrapwQe0tligdiA15N5BuHXVV2LYCLh+yStzGJbu8TR QOQCB7O42mkXErxw1uk3B0Lx6LLqO0d28xOMW1TvpwIiQIGdFgcF9NBnmeiGyouMtHOF 6KBRjFOmIJMV17pKIEkaFJ9fY+QdGHYdYM24puSKZEfeC01lXlzzIwetanSJspPOgNjw 8iLzuJ0GxZflJ1vZhT0AFl7IVDa0wqRRln5mup/m038HKu/2ZN/KJ6pt0IUKemzbfyWp 5wMA== X-Forwarded-Encrypted: i=1; AJvYcCWfOEqo+J56ip25OmJ7QA9IhCJCj4jgDxnWGICRTL12fPiyL27Ban48ZlU1prie/oJ3/F0CS2P+Gg==@kvack.org X-Gm-Message-State: AOJu0Yx5bcMOt+ECfICwFD1FTR3kQpHmCUFYTUHNSqE9w9YBviMeOcAs MSuQn84mO94ogP4RiwM/Z43gVhe+kbLLzuTRpso05I/4j6zWyvoayqIHM83cCtQeA7cS4BgDqfG Krw== X-Google-Smtp-Source: AGHT+IFInU2/c8WUI1v0xjpwtORb6r7NqVkTkq8OC0u0JtieghImrnI2AnpvFTBaqoW2DCB2chLg/yy/KrA= X-Received: from pfblh8.prod.google.com ([2002:a05:6a00:7108:b0:728:f1bf:72ad]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:9c8d:b0:1dc:790e:3bd0 with SMTP id adf61e73a8af0-1e1dfd3dbb6mr22228664637.15.1734377079970; Mon, 16 Dec 2024 11:24:39 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:11 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-9-surenb@google.com> Subject: [PATCH v6 08/16] mm: uninline the main body of vma_start_write() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 41BE4C0004 X-Stat-Signature: ahzjqj379o33gmruoaadumsbjjqry9sk X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734377050-741476 X-HE-Meta: U2FsdGVkX19jPM9HJYaaS9t8OCB2ia1Kp6V3VW0qEMeZ1/NPh37DDA0oQrw3EJVDgpOLyU67dsDyAZUP2enCgwLsRKEnaIIeivdo+7x+8Q2tjWrBxbYqkyuoPX+1Wv7+dTON18jNkcdLH4pmddKOesv5Kw2jOifbGYvMIuQT+4W8isDyAvC6MGFDyinp0xIOvGkynI0JRW672nH/f5rfmvGY7bYMvFedBa651laQjEwxP9YQ2H/DTnZaNz4diYWZNABq1rywMLrZkq2Loh3FLUD7F25sSFn2UJWZUMMti27T2YQUV9YUP8CFkpMIXa0kwL/WUpx+97hKnlBImK8zkT6jJEMTCIGzvmXjTpKE4XpuALrdycXSYwwfw58Y/tmRx2Dr+WI3ulgqNVobPaMFPNX/8UxWPd4veohVXpGu5hhbxyjmguf/tqUqScKyTC7FsCiqLXbZ2/AMmWiES1zROJ8uGIibglbcB5yrRPw98DIi+cUiIjVJNQh3qKT6qDK2rrOzqfWr+uacJ4BfAAv3fees+4pQjUWhfXsw5+/voguYEAnVLyCH+kYG0rf5zYebyAEI9mLzEnXhEwoHrl3a+YezgRiCC58moPoJFy/BmsnI17DFLjxSV4OabzOcIGHJMgqP+FZxUQ+6jECBKyEdlLgM+8/CeuQeT9rwPSd+Aj0nBUe+zcnf29KmYiPSgAEsp4BZguCMILow09LVAaP8ZNNeeFFAY7xDV0CNayV3ZHEd/T4k9py/pqtzpRF9iDyFo7Oz5P5KVFi/4mEvOc9r8FMVkwo/b9/UU/IVT8wyvTCDPl68IOcmbhr4Ye7SlzwYcCllY7/XqfUD0Nn3SB5+yufnlL/UCfQb99bO9CY45Kz5wohkRMKUaoGJdJSI6T7k6+gXUMPAY0MoQA/iwFwE/7Prg9aAq076l0GmhF283S9i+IcDbuoerapgA3knZn1ZDDGxEA9g4nNwBU8/YW4 rS9/fUBq Gg53tlxZxHE/BgUpmq6o94MnvwcGgxWWVOv8Lm/Bd7jcCLZM769kOOwox1VvOSLY3v02LTpZgNPzj25490Z7DKfbo6La7bxVqtDsl5i8Hd9b70wHDO5USabxTDHCOVUkjzMgNaT4hT/XUVKscIIS49GX6trEfzocYFuAAY5pODN4ZglixG9v+4vBeFcEtsQzNUfR/6hxMy+rzoO1BxF686jagczjVdPmcjUMYr7RhWe8Mxk8SJCKs0hsnkh459a6C1BslMPX/V8YsE1h/t4hDDaaYkngrCuJJ1gknGpLJpTkHUwG69rkqmsl1dc5Xj9FZQaVFCw0eWCHYij7Xy0tOl+egE/MmBjZ5Zera1e7bQWEiHDAXgjP2NtrG/8s6SNXlF2vtJKbJkSG8kFQNrALTYBL2s9PHnhKmLbwbj8O/kB2accNOBNL1hLee2GGE3uFxLS5RpxNhW0xqxvGWiQu8GK4AG8nDijoVVvRwPzdEesck5dfZQJ5HeQoZbyYvABpM11B5VPdag7ku0vBRIOk429e2u0n7eK5I+DUIHKt/FIl+7SNbaK7CmpBqt/wPhtwozkCZYjagkefb6y7Y4OKNGaQvXw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.058830, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vma_start_write() is used in many places and will grow in size very soon. It is not used in performance critical paths and uninlining it should limit the future code size growth. No functional changes. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 12 +++--------- mm/memory.c | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0ecd321c50b7..ccb8f2afeca8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -787,6 +787,8 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, unsigned int *mm_l return (vma->vm_lock_seq == *mm_lock_seq); } +void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq); + /* * Begin writing to a VMA. * Exclude concurrent readers under the per-VMA lock until the currently @@ -799,15 +801,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (__is_vma_write_locked(vma, &mm_lock_seq)) return; - down_write(&vma->vm_lock.lock); - /* - * We should use WRITE_ONCE() here because we can have concurrent reads - * from the early lockless pessimistic check in vma_start_read(). - * We don't really care about the correctness of that early check, but - * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. - */ - WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock.lock); + __vma_start_write(vma, mm_lock_seq); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) diff --git a/mm/memory.c b/mm/memory.c index cc7159aef918..c6356ea703d8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6329,6 +6329,20 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, #endif #ifdef CONFIG_PER_VMA_LOCK +void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) +{ + down_write(&vma->vm_lock.lock); + /* + * We should use WRITE_ONCE() here because we can have concurrent reads + * from the early lockless pessimistic check in vma_start_read(). + * We don't really care about the correctness of that early check, but + * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. + */ + WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); + up_write(&vma->vm_lock.lock); +} +EXPORT_SYMBOL_GPL(__vma_start_write); + /* * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be * stable and not isolated. If the VMA is not found or is being modified the From patchwork Mon Dec 16 19:24:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 109C3E77180 for ; Mon, 16 Dec 2024 19:24:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8BBB6B00BD; Mon, 16 Dec 2024 14:24:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3A316B00BE; Mon, 16 Dec 2024 14:24:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B5C56B00BF; Mon, 16 Dec 2024 14:24:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6CA286B00BD for ; Mon, 16 Dec 2024 14:24:45 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2C8CB1A0A0B for ; Mon, 16 Dec 2024 19:24:45 +0000 (UTC) X-FDA: 82901798448.22.0835059 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf25.hostedemail.com (Postfix) with ESMTP id 14241A000D for ; Mon, 16 Dec 2024 19:24:21 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Nunpo0r6; spf=pass (imf25.hostedemail.com: domain of 3en5gZwYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3en5gZwYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377060; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PJyEpzu4DU8rJRmq3ow3hqlNy2LMWwwp3HWnjTLdNPo=; b=AuWxEtdhXZfZoe/TNjDZTWsE9DdUP+9Q+/zK+I5If1IWjCCUq5uj1N8wMMo4oHj45pS/kc rMvfZLD7oHmEr+dyLDknA8OewV8IfalFaei2lWS48Qn2ykoM3lue5ZXODiWclwaMSvKZZx Ln0Rrqu1LJVy+o3db3jd0ruMPzwyxsA= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Nunpo0r6; spf=pass (imf25.hostedemail.com: domain of 3en5gZwYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3en5gZwYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377060; a=rsa-sha256; cv=none; b=HZiTDiDc2jiRjnaFK5D/AhQzDo8KzSoK+uhbqS0pxxiaHNqifQOrVueuep2Kw4CbjfELRZ PELX0vvGg0t3Nj3Usqg6wRs+/6ZtWtKAPdDdJZlt4nY1K+4CcIELORBfKnL7UtpXGOXgNx JKT/UhetwX3K8kvOA+u8vCD/K6gg3Hc= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ee5616e986so6677821a91.2 for ; Mon, 16 Dec 2024 11:24:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377082; x=1734981882; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PJyEpzu4DU8rJRmq3ow3hqlNy2LMWwwp3HWnjTLdNPo=; b=Nunpo0r6TyN3CGynG5ca3+GlNXVf0+HDL/rVWOt6dV6KCwb/hPpnJ5k7LcW9Iv8En6 HC//1NqqT5XmbGqxdkeha25NLwvw6zU+0LF3Sc0hCvcILe5KyPQhnhVeFnlrJYTerzzk 13zzu4ykrvKJrNBs56joIo8kn53yRzjFou3gafdxPSHRzfcZIzyR2sN48KcIhC2cd7EI cxEjpkWDlzJOUZtCShtB/NyMx+5zwwDez3CWq+2yl303iuZG6GKeQycFkdNIKPyRwF57 DheBV3o+6kCrWugA9ViqdeImtI8Tgu16JIpnbCbSra2mbppnPdIx7Q0blKhWQ9gn8RdO MgNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377082; x=1734981882; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PJyEpzu4DU8rJRmq3ow3hqlNy2LMWwwp3HWnjTLdNPo=; b=iGulR6SwKe4llmD6ONcbXz7qH0ZYHSLMjlmWlIyzSENmSVHFbofc4MsNMH3iS1JYEv 8IPFBVSV7WXsBbvzXg9qw6MMsWCfMJesPpWnF9ACVtRVp1KyOohP/wEPo2hhtqPb8pwe iKsV01szQqKYO+eae/FDMamZDGWv6WjZQbV4lBl/LzfMe+eCvzvz1+vGIzafknfWEo3H Cyt7xdNb0Dgok3SyqpAUzlexCaU2WIO5gByi/ry3KPP2daK2c05cgz1NbJmjrLKx/UBJ 5V6EM0ftk3QkYQH6z5yxtQ+w3GkocX1nRl9aaXEOUQ5cTOXOYaxG90HVwqaUgNJsjxIJ oWXA== X-Forwarded-Encrypted: i=1; AJvYcCUSK9ZDn9fRzJ/r9UOK3FyCyL6xjVi9e6KvgOPW4VkSRjlHA5T8aG51dQl8ZlPFzlEGuwzGI3RL0Q==@kvack.org X-Gm-Message-State: AOJu0YwwdoHYbsEWVWWNsQXlChW4B1UY2tZZmHFhT8sUq2qi+CZdYypR ohIZ0z+DcUwZsV48uwcTRcaOuZBjJY6A2XjykPA/KvPU6bRZLe+REIiIKA7OS0epr57vOovPcNs AWw== X-Google-Smtp-Source: AGHT+IGSC/WDrL2DnP38qBt3Yi0DRZx0zLMWebStlpO2MkkwMeYlYALEb8oRG5sXg1v7sD7oiSjMQ3XiMF4= X-Received: from pjboi16.prod.google.com ([2002:a17:90b:3a10:b0:2e2:9021:cf53]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3c4a:b0:2ee:7233:4e8c with SMTP id 98e67ed59e1d1-2f28fb522d7mr20621018a91.8.1734377082197; Mon, 16 Dec 2024 11:24:42 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:12 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-10-surenb@google.com> Subject: [PATCH v6 09/16] refcount: introduce __refcount_{add|inc}_not_zero_limited From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 14241A000D X-Rspamd-Server: rspam12 X-Stat-Signature: r4j6z6bshmeg5s7znkumcukmwx7gnbbx X-Rspam-User: X-HE-Tag: 1734377061-85729 X-HE-Meta: U2FsdGVkX18Yr39uhsNhIbS+akYB2502G7eGiKS+dQiJ7kaiXD7CZ4kaPJtNJecTiSBxF6/CC07Tnw0G9+b3ytauSvgEoHBDDsULLd5KbeM0ntwzeffYf6q32y32mbPb2cIuOOnfoEdT4ncx+gdw+fJLmDa2M7RcEIDud1Fpsl2KgyJJkaHaJJZbqZ6j7pbRidrBjeZ578iT4pMP2Wi0dzuPxynSg4fDkz8FM5a91av5UjiiL6c2m9kvOU4uMAPmuWuc5TnsVlSfGSisis4mWRAbiR4TG/5fGv6kf0QAK9f8zjmT7WI7zWGy8Gb463c61OIsKWpaiIQkKosU8izFTStNNR8+NjgalVkw6vOn5m90K/T8vD/sSU+eZzqTLO8vFHQn2ja3uyBIXdG4o4ss9LjLx67+9KYWRyRk2+RwAj9a2W50LbhT979Ylt+JaIW/988n5cfyUnRug2fGBuT8hZc4qJa/gNx2QW334aOkDZfZKzAJ4q9VL409Tw1AyIWJ4si5Af9D0PCjEErXMO5UgP9zzlBJVL5kN7+DBd9db6KMxb8sgvADp8Smge2QhPx3cgUUzGa1LBGJ1rPxZIg3k66a23Pn4b2/d3OWUtlhXpajsCSsw+K+VXdCzTh2VPAaK/sNBENqhU98LQw4MtXpuvJ0mZUPLzX9tEhx+0zu/fZSGIaidsh+m4fynh/ctkX+lny8Tz1MuYBc3EshxZt19UEjhpnQUH+lFYPsoHSN74hfn0feca/SuAMWusrJy5BdUsLdjILpNWFsvNAp0XJ95FmogofuKXW3p+T4U7kPLG+iyAqPGSVV1X2UjOuQd4rUaIBwyC55j9bPzP9fZ+uVwgDtkKCd7EWjWNax1qqZj1VzQbPu+/70g2m/Us0fGeWirCi7FtH8eiBdo99Gy45bqIjTm3bp7uJurJL8m7hc/QGnj0Cz9N5zT3UdoIc99Hnw5N0a/6/2Ttx8gM+8SPK lQNGWVbU XUgJ7Z6rDqCGXJdTvDR/b7Wc1j2RxwPkB8ORaJDKaXZrwUQGWDpAjwrIREW9oIE5t3zR8WGAIHlNeiTZSrU00i78BSVJEuIvsuZfrjZAbTqtX0IyyO3+ZamRGG90JIVsQcclziABfN1+h9muhxyJGkb5p+OxN3Dnn9wtzvun7BRo1IHIKnTZWe+DAYcVNaMV4Rt5N3umANQazqvgDLIh0XC2YOgKws9qrm+tRiPLANtQRrmecgF2HNp7qLUKeGVUp7jRK2UZlAca9PeQCSRNKnrr+0/aavDU9xQnBf9mgCJgl0ejzDOaWC4DlorKGhaR2lLq+JAtXzRgpqMwacnMHBtVLzUUzC92yR95GnKD2tElRzdHbFbxs79/xN0IYk6KIoRfWnjeypOpskP5CcciarKvuyNQw8v1VmVPI8j10tiE8MpqK8zHY6xbuIPkd9xFe5TULAVeI5y0+pXgVJEqqr3kHFzmzQyIIv9J4YYMCgXKrzR5sqpJvVbPGEINAbfg9XQ5CbwzyN7vLtLKafQhF77wQD02g5in7GVkTcacQAqn/qhczuzWfWgMJnLV2rkbcckii2FbonQTeDZGYNDtCQvdZFQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.225839, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce functions to increase refcount but with a top limit above which they will fail to increase. Setting the limit to 0 indicates no limit. Signed-off-by: Suren Baghdasaryan --- include/linux/refcount.h | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/include/linux/refcount.h b/include/linux/refcount.h index 35f039ecb272..e51a49179307 100644 --- a/include/linux/refcount.h +++ b/include/linux/refcount.h @@ -137,13 +137,19 @@ static inline unsigned int refcount_read(const refcount_t *r) } static inline __must_check __signed_wrap -bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp) +bool __refcount_add_not_zero_limited(int i, refcount_t *r, int *oldp, + int limit) { int old = refcount_read(r); do { if (!old) break; + if (limit && old + i > limit) { + if (oldp) + *oldp = old; + return false; + } } while (!atomic_try_cmpxchg_relaxed(&r->refs, &old, old + i)); if (oldp) @@ -155,6 +161,12 @@ bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp) return old; } +static inline __must_check __signed_wrap +bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp) +{ + return __refcount_add_not_zero_limited(i, r, oldp, 0); +} + /** * refcount_add_not_zero - add a value to a refcount unless it is 0 * @i: the value to add to the refcount @@ -213,6 +225,12 @@ static inline void refcount_add(int i, refcount_t *r) __refcount_add(i, r, NULL); } +static inline __must_check bool __refcount_inc_not_zero_limited(refcount_t *r, + int *oldp, int limit) +{ + return __refcount_add_not_zero_limited(1, r, oldp, limit); +} + static inline __must_check bool __refcount_inc_not_zero(refcount_t *r, int *oldp) { return __refcount_add_not_zero(1, r, oldp); From patchwork Mon Dec 16 19:24:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3E20E77180 for ; Mon, 16 Dec 2024 19:24:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF9736B00BF; Mon, 16 Dec 2024 14:24:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E808A6B00C0; Mon, 16 Dec 2024 14:24:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C864F6B00C1; Mon, 16 Dec 2024 14:24:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8CB576B00BF for ; Mon, 16 Dec 2024 14:24:47 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4034F45057 for ; Mon, 16 Dec 2024 19:24:47 +0000 (UTC) X-FDA: 82901797986.01.09BE53B Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf29.hostedemail.com (Postfix) with ESMTP id CF6F0120003 for ; Mon, 16 Dec 2024 19:24:03 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oWLKoSto; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3e35gZwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3e35gZwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377057; a=rsa-sha256; cv=none; b=hwQPsYE4zjJmgO6Sr3faP4RyrFqhnVUuz4t7GR5VRCkOftBS6zqo/v/kdjyio9ht6HjFZS tIqWOvPEz6kbIckM2ltRrmtIDhkTP9NHwRFH68oqQi+yleJ9oD0er9pbRAGHLBNkTmfcai c4eT4kLo6PqmRTh1pqkVe9MOhq0/BWs= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oWLKoSto; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3e35gZwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3e35gZwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377057; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yWr5wuN/XFwhP0jdgiadjUA9/QQ6j9qWDWV+i5/abRA=; b=tPLdda7LPssgC5owE9XgJzFso9Q2C2vu7ToKVypFyTEXxPgsOVqI2/IFe2uWTq8Bj1ps7I qNyffgCqRmNMsEBUth3//vnnBPZCdS80e0Hi0c1HNoYRva6u+QhTjyC2LmQK9/4kZ77C+r UctFWH9f6OT7aLbBFUXc0Z3gSXq3HwA= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-72739b83336so6146098b3a.1 for ; Mon, 16 Dec 2024 11:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377084; x=1734981884; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yWr5wuN/XFwhP0jdgiadjUA9/QQ6j9qWDWV+i5/abRA=; b=oWLKoStoUQryVJJGIBlQb1IuX/rQcJJkFYvCjRfc8ywCFCr2Z8gBGriBlZDoYAEvMK cyI2+qXOue7byCnRK1UhE+e0//AnYAB1N9mzGMW3sEjzBm1HeVRVLGOXLow/SfJJJxHM 7x2zDokow2BZETYpyVhSXTQx0GiyUc1/bhDkU0K1oMHavvcY166Fs0IwF//FkbosX2qV Ch2j+m1DCsA8z/su72s4+SkRnAvnJkMaYR5Yi+u2DCMYQbtwzaMnE7ouyKvWrnhGaL3Q W5Xr2Y7US1bedcVd1/FiRuNQDLT9GO/qmgSbz+6kF75YYXq4ujQ2Yeu3UlJKYX9jStWU TDQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377084; x=1734981884; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yWr5wuN/XFwhP0jdgiadjUA9/QQ6j9qWDWV+i5/abRA=; b=TxWmn5mN1cVVO4DgTJifDS4N2dFRVJmSkhfZw50+H1yMK5xyDa9LXOicxkaJ4VtEJk V4VnIAlY+nQhtXIeyrh4MoHN6ufpqovr14LTs3QVUb6O96yo8oQr41HRwbmtKoPfWi65 +0uPnsJDs2m5qHerop8CW8mixpkbCGfvJLxfnufEM//y4ABOlRxmV8xJ/eFJQ6Kdv0p4 AVLHNkkm5QYdZ2n6Rf0Z3REJ2IApgdiMD38MXN6866AQ9VZsOuDu17QvSOxEInN4zEOP qQCTy9ngHMvvd2DgtAi3W57fAJnej8Gw5uJrPcyyD1Deb8LT1+NgJJ8n4WISgQiiFCxl saGw== X-Forwarded-Encrypted: i=1; AJvYcCWuSB6U5lO+2/jxEP0rfFFM9Cv1GodNNsabYjI+jzY9TKJbkd6+OyONRXhX4n0YDH1a/GFDmm40Kg==@kvack.org X-Gm-Message-State: AOJu0Yysxn+ytzPw8qWKF6bxNqWqM/ezv6Phb3qxE77YDBofjDxf9ZDy OyVxtniutPlGVhaN6iAiXNSfOeZF6GAZ8YqqDbpZZSP3Uu4zGti0o9dX1vXidFNvqLoR0TR0Jyg /3A== X-Google-Smtp-Source: AGHT+IG2bwhxLQwIlqUIeB8kJPQ7+AcPBBfyUQeVN2TC0lLVCS5WDt3dsxGA1e6kkw4Q6iS5TtnTtrsPmuM= X-Received: from pgbdq26.prod.google.com ([2002:a05:6a02:f9a:b0:7fd:577a:6d1e]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:328d:b0:1e1:9bc1:6d6d with SMTP id adf61e73a8af0-1e1dfde67b3mr25323185637.31.1734377083981; Mon, 16 Dec 2024 11:24:43 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:13 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-11-surenb@google.com> Subject: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: 8op9hutp6souei6cyq7sbohgcd66ska1 X-Rspam-User: X-Rspamd-Queue-Id: CF6F0120003 X-Rspamd-Server: rspam08 X-HE-Tag: 1734377043-728993 X-HE-Meta: U2FsdGVkX1/Rw0uOqkFqRFG+SPaxgJC8hvofxLiB4dBBY2xSNl+XgE+nmjfKn98JVLwtVFg67F31abFIyFIjRzC4SAsZ94cL9BiajIs6VKYx8FjiXBVvhfx9lEYpD8yJe14pl5ySs5SYyDFwt9MM9+EwwH0gl7bskDLrCy2Hu4c8PZnP9OfueVHZqwUOMqu/1jStTmMRCRSfJCsKJmGmG+eqpu79lnGQfCJkJMYG158Lrl2Yg1CkmMlie6MJmU/4Y+WU7Jo/z5itW/KTt1IfA59BZo1ONSTBnl62JFSlmJXaEQjEkg0bHNzFKwjJSenSqHfbSfijqgwitmZgIfOKSMK1IPee266/3rlfNs4pAdfqrbHfprU+acGN5+Fpsk+q/4hEF8c02TN+Y088w+cWqH+Y4/AlFW6saTD6XD9HoAzMP64635CO7LKcjp9j+ArkBQnye/quwCybyyyG3UKwrUWvvJSr3SvxCu1b3ktFI2WAHXKKzFJlgmeekduhJ83ZeIO7JUZ+0WLSg7xvo+LSRKaKKWRByeDbMATQH+91uhfan3aLXwAdfr8ycfawPSrbhjt8Wp5OSsVTTKvpv/jVwuXJYYgorM1LIuPpPt2mmTSJvpkjE4hjNK9Cucf2cMJsVXeGDfZgjApnDYGAxavWeybw1kdXydDkFPbZPO/75e6Yd68HQFTSp0hxCmgwX3Jzjtvxr2hyX5Q+kYzIdhbfMcH26H8tbBQpbcXKuEqFAmSLWIRYrjGYHZUXM56cd0Hl2qJuwQKJt5fkv2oEApwyPmNiVYsi0c6NFnSwcCzyNkThW8JB5LGp/FDORZanj2+d4cnyXkf2lJ+veaCW4bwEfjW3OAsKTAyWxl0yG7cat75MFPWvb7lMT8qbw/K6I5h+nQyT174G4xeM9RbKy2JSUsBRY1eROnCA/cppMtYfPb2c5DK1rzlVyyu8WaJj40oKVgzSTMB7ZNwZfH+A6mq tIuzryMA SQTBCBmcQhM1A6WHMlrp3+o5tT/I4v3UO4+oSayXfRrjgJbEXObpnavjYTD67gxa9HmdG7sKXRooEvO17UZeqwpBYogFvRA/bDolklpwANlEPBgXi8Czv+uwqQbDiXtejfW3nvQq4KYN9fCS8vT+AgBq5khEOHPIqkcy0JmMooIQ2inukMuZCUE5Wx+OFUdVxincwBQFiAOkcJ3rR4mUw3yzWS8cN+x4SGO8DMt5v5WCZisLcjdoEGgNaqzaatYifQr1cb9kgjh869diLukyMKpYruQJIym7oZd4wxyRdPzi/hRgnD8kJF7DD7BXDscIsWLwycikQAhouQocF7CaKBKej4ve4mhlOofaw3LH/mrCJCYPqLjgfoBljJ1MKAFdvqvg+oBQ1QLI/04li7EIoqNhA0fkMt8kcCPfAvFu1LaSqezRF9aLSIAw7BveUi+yyRIhUuBIFAB1tG9HRkdeQKsJZ0I4m3dGB6tH/uMMfE32+VFfPfM6fbMLOeKN3k6TWwzZlKYH74+tq/TAoae+hKPjzMjdZdXEausMYjq+VSXXJD4PFwWulLAJWDAA1mK5sobdTX9FYXlkQhLqiN7nUBIXL/g9E899spYVci9XSPZLIFI5xWdBbDxCfMT04ZAMt0Mnf9T84IeroA2N+S0n5wXlEQA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: rw_semaphore is a sizable structure of 40 bytes and consumes considerable space for each vm_area_struct. However vma_lock has two important specifics which can be used to replace rw_semaphore with a simpler structure: 1. Readers never wait. They try to take the vma_lock and fall back to mmap_lock if that fails. 2. Only one writer at a time will ever try to write-lock a vma_lock because writers first take mmap_lock in write mode. Because of these requirements, full rw_semaphore functionality is not needed and we can replace rw_semaphore and the vma->detached flag with a refcount (vm_refcnt). When vma is in detached state, vm_refcnt is 0 and only a call to vma_mark_attached() can take it out of this state. Note that unlike before, now we enforce both vma_mark_attached() and vma_mark_detached() to be done only after vma has been write-locked. vma_mark_attached() changes vm_refcnt to 1 to indicate that it has been attached to the vma tree. When a reader takes read lock, it increments vm_refcnt, unless the top usable bit of vm_refcnt (0x40000000) is set, indicating presence of a writer. When writer takes write lock, it both increments vm_refcnt and sets the top usable bit to indicate its presence. If there are readers, writer will wait using newly introduced mm->vma_writer_wait. Since all writers take mmap_lock in write mode first, there can be only one writer at a time. The last reader to release the lock will signal the writer to wake up. refcount might overflow if there are many competing readers, in which case read-locking will fail. Readers are expected to handle such failures. Suggested-by: Peter Zijlstra Suggested-by: Matthew Wilcox Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 95 ++++++++++++++++++++++++-------- include/linux/mm_types.h | 23 ++++---- kernel/fork.c | 9 +-- mm/init-mm.c | 1 + mm/memory.c | 33 +++++++---- tools/testing/vma/linux/atomic.h | 5 ++ tools/testing/vma/vma_internal.h | 57 ++++++++++--------- 7 files changed, 147 insertions(+), 76 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ccb8f2afeca8..d9edabc385b3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -32,6 +32,7 @@ #include #include #include +#include struct mempolicy; struct anon_vma; @@ -699,10 +700,27 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {} #ifdef CONFIG_PER_VMA_LOCK static inline void vma_lock_init(struct vm_area_struct *vma) { - init_rwsem(&vma->vm_lock.lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + static struct lock_class_key lockdep_key; + + lockdep_init_map(&vma->vmlock_dep_map, "vm_lock", &lockdep_key, 0); +#endif + refcount_set(&vma->vm_refcnt, VMA_STATE_DETACHED); vma->vm_lock_seq = UINT_MAX; } +static inline void vma_refcount_put(struct vm_area_struct *vma) +{ + int refcnt; + + if (!__refcount_dec_and_test(&vma->vm_refcnt, &refcnt)) { + rwsem_release(&vma->vmlock_dep_map, _RET_IP_); + + if (refcnt & VMA_STATE_LOCKED) + rcuwait_wake_up(&vma->vm_mm->vma_writer_wait); + } +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -710,6 +728,8 @@ static inline void vma_lock_init(struct vm_area_struct *vma) */ static inline bool vma_start_read(struct vm_area_struct *vma) { + int oldcnt; + /* * Check before locking. A race might cause false locked result. * We can use READ_ONCE() for the mm_lock_seq here, and don't need @@ -720,13 +740,20 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) return false; - if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) + + rwsem_acquire_read(&vma->vmlock_dep_map, 0, 0, _RET_IP_); + /* Limit at VMA_STATE_LOCKED - 2 to leave one count for a writer */ + if (unlikely(!__refcount_inc_not_zero_limited(&vma->vm_refcnt, &oldcnt, + VMA_STATE_LOCKED - 2))) { + rwsem_release(&vma->vmlock_dep_map, _RET_IP_); return false; + } + lock_acquired(&vma->vmlock_dep_map, _RET_IP_); /* - * Overflow might produce false locked result. + * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * * We must use ACQUIRE semantics for the mm_lock_seq so that if we are @@ -734,10 +761,12 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock.lock); + if (oldcnt & VMA_STATE_LOCKED || + unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + vma_refcount_put(vma); return false; } + return true; } @@ -749,8 +778,17 @@ static inline bool vma_start_read(struct vm_area_struct *vma) */ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { + int oldcnt; + mmap_assert_locked(vma->vm_mm); - down_read_nested(&vma->vm_lock.lock, subclass); + rwsem_acquire_read(&vma->vmlock_dep_map, subclass, 0, _RET_IP_); + /* Limit at VMA_STATE_LOCKED - 2 to leave one count for a writer */ + if (unlikely(!__refcount_inc_not_zero_limited(&vma->vm_refcnt, &oldcnt, + VMA_STATE_LOCKED - 2))) { + rwsem_release(&vma->vmlock_dep_map, _RET_IP_); + return false; + } + lock_acquired(&vma->vmlock_dep_map, _RET_IP_); return true; } @@ -762,15 +800,13 @@ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int */ static inline bool vma_start_read_locked(struct vm_area_struct *vma) { - mmap_assert_locked(vma->vm_mm); - down_read(&vma->vm_lock.lock); - return true; + return vma_start_read_locked_nested(vma, 0); } static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->vm_lock.lock); + vma_refcount_put(vma); rcu_read_unlock(); } @@ -813,25 +849,42 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_locked(struct vm_area_struct *vma) { - if (!rwsem_is_locked(&vma->vm_lock.lock)) + if (refcount_read(&vma->vm_refcnt) <= VMA_STATE_ATTACHED) vma_assert_write_locked(vma); } -static inline void vma_mark_attached(struct vm_area_struct *vma) +/* + * WARNING: to avoid racing with vma_mark_attached(), should be called either + * under mmap_write_lock or when the object has been isolated under + * mmap_write_lock, ensuring no competing writers. + */ +static inline bool is_vma_detached(struct vm_area_struct *vma) { - vma->detached = false; + return refcount_read(&vma->vm_refcnt) == VMA_STATE_DETACHED; } -static inline void vma_mark_detached(struct vm_area_struct *vma) +static inline void vma_mark_attached(struct vm_area_struct *vma) { - /* When detaching vma should be write-locked */ vma_assert_write_locked(vma); - vma->detached = true; + + if (is_vma_detached(vma)) + refcount_set(&vma->vm_refcnt, VMA_STATE_ATTACHED); } -static inline bool is_vma_detached(struct vm_area_struct *vma) +static inline void vma_mark_detached(struct vm_area_struct *vma) { - return vma->detached; + vma_assert_write_locked(vma); + + if (is_vma_detached(vma)) + return; + + /* We are the only writer, so no need to use vma_refcount_put(). */ + if (!refcount_dec_and_test(&vma->vm_refcnt)) { + /* + * Reader must have temporarily raised vm_refcnt but it will + * drop it without using the vma since vma is write-locked. + */ + } } static inline void release_fault_lock(struct vm_fault *vmf) @@ -896,10 +949,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); -#ifdef CONFIG_PER_VMA_LOCK - /* vma is not locked, can't use vma_mark_detached() */ - vma->detached = true; -#endif vma_numab_state_init(vma); vma_lock_init(vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 825f6328f9e5..803f718c007c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -19,6 +19,7 @@ #include #include #include +#include #include @@ -599,9 +600,9 @@ static inline struct anon_vma_name *anon_vma_name_alloc(const char *name) } #endif -struct vma_lock { - struct rw_semaphore lock; -}; +#define VMA_STATE_DETACHED 0x0 +#define VMA_STATE_ATTACHED 0x1 +#define VMA_STATE_LOCKED 0x40000000 struct vma_numab_state { /* @@ -679,19 +680,13 @@ struct vm_area_struct { }; #ifdef CONFIG_PER_VMA_LOCK - /* - * Flag to indicate areas detached from the mm->mm_mt tree. - * Unstable RCU readers are allowed to read this. - */ - bool detached; - /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) - * - vm_lock->lock (in write mode) + * - vm_refcnt VMA_STATE_LOCKED is set * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_lock->lock (in read or write mode) + * - vm_refcnt VMA_STATE_LOCKED is set or vm_refcnt > VMA_STATE_ATTACHED * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * @@ -754,7 +749,10 @@ struct vm_area_struct { struct vm_userfaultfd_ctx vm_userfaultfd_ctx; #ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ - struct vma_lock vm_lock ____cacheline_aligned_in_smp; + refcount_t vm_refcnt ____cacheline_aligned_in_smp; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map vmlock_dep_map; +#endif #endif } __randomize_layout; @@ -889,6 +887,7 @@ struct mm_struct { * by mmlist_lock */ #ifdef CONFIG_PER_VMA_LOCK + struct rcuwait vma_writer_wait; /* * This field has lock-like semantics, meaning it is sometimes * accessed with ACQUIRE/RELEASE semantics. diff --git a/kernel/fork.c b/kernel/fork.c index 8cb19c23e892..283909d082cb 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,10 +465,6 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); -#ifdef CONFIG_PER_VMA_LOCK - /* vma is not locked, can't use vma_mark_detached() */ - new->detached = true; -#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); @@ -488,8 +484,6 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); - /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); __vm_area_free(vma); } #endif @@ -1228,6 +1222,9 @@ static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); mm_lock_seqcount_init(mm); +#ifdef CONFIG_PER_VMA_LOCK + rcuwait_init(&mm->vma_writer_wait); +#endif } static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, diff --git a/mm/init-mm.c b/mm/init-mm.c index 6af3ad675930..4600e7605cab 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -40,6 +40,7 @@ struct mm_struct init_mm = { .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK + .vma_writer_wait = __RCUWAIT_INITIALIZER(init_mm.vma_writer_wait), .mm_lock_seq = SEQCNT_ZERO(init_mm.mm_lock_seq), #endif .user_ns = &init_user_ns, diff --git a/mm/memory.c b/mm/memory.c index c6356ea703d8..cff132003e24 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6331,7 +6331,25 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, #ifdef CONFIG_PER_VMA_LOCK void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) { - down_write(&vma->vm_lock.lock); + bool detached; + + /* + * If vma is detached then only vma_mark_attached() can raise the + * vm_refcnt. mmap_write_lock prevents racing with vma_mark_attached(). + */ + if (!refcount_inc_not_zero(&vma->vm_refcnt)) { + WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); + return; + } + + rwsem_acquire(&vma->vmlock_dep_map, 0, 0, _RET_IP_); + /* vma is attached, set the writer present bit */ + refcount_add(VMA_STATE_LOCKED, &vma->vm_refcnt); + /* wait until state is VMA_STATE_ATTACHED + (VMA_STATE_LOCKED + 1) */ + rcuwait_wait_event(&vma->vm_mm->vma_writer_wait, + refcount_read(&vma->vm_refcnt) == VMA_STATE_ATTACHED + (VMA_STATE_LOCKED + 1), + TASK_UNINTERRUPTIBLE); + lock_acquired(&vma->vmlock_dep_map, _RET_IP_); /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -6339,7 +6357,10 @@ void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock.lock); + detached = refcount_sub_and_test(VMA_STATE_LOCKED + 1, + &vma->vm_refcnt); + rwsem_release(&vma->vmlock_dep_map, _RET_IP_); + VM_BUG_ON_VMA(detached, vma); /* vma should remain attached */ } EXPORT_SYMBOL_GPL(__vma_start_write); @@ -6355,7 +6376,6 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, struct vm_area_struct *vma; rcu_read_lock(); -retry: vma = mas_walk(&mas); if (!vma) goto inval; @@ -6363,13 +6383,6 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_start_read(vma)) goto inval; - /* Check if the VMA got isolated after we found it */ - if (is_vma_detached(vma)) { - vma_end_read(vma); - count_vm_vma_lock_event(VMA_LOCK_MISS); - /* The area was replaced with another one */ - goto retry; - } /* * At this point, we have a stable reference to a VMA: The VMA is * locked and we know it hasn't already been isolated. diff --git a/tools/testing/vma/linux/atomic.h b/tools/testing/vma/linux/atomic.h index e01f66f98982..2e2021553196 100644 --- a/tools/testing/vma/linux/atomic.h +++ b/tools/testing/vma/linux/atomic.h @@ -9,4 +9,9 @@ #define atomic_set(x, y) do {} while (0) #define U8_MAX UCHAR_MAX +#ifndef atomic_cmpxchg_relaxed +#define atomic_cmpxchg_relaxed uatomic_cmpxchg +#define atomic_cmpxchg_release uatomic_cmpxchg +#endif /* atomic_cmpxchg_relaxed */ + #endif /* _LINUX_ATOMIC_H */ diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 0cdc5f8c3d60..b55556b16060 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -25,7 +25,7 @@ #include #include #include -#include +#include extern unsigned long stack_guard_gap; #ifdef CONFIG_MMU @@ -132,10 +132,6 @@ typedef __bitwise unsigned int vm_fault_t; */ #define pr_warn_once pr_err -typedef struct refcount_struct { - atomic_t refs; -} refcount_t; - struct kref { refcount_t refcount; }; @@ -228,15 +224,14 @@ struct mm_struct { unsigned long def_flags; }; -struct vma_lock { - struct rw_semaphore lock; -}; - - struct file { struct address_space *f_mapping; }; +#define VMA_STATE_DETACHED 0x0 +#define VMA_STATE_ATTACHED 0x1 +#define VMA_STATE_LOCKED 0x40000000 + struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ @@ -264,16 +259,13 @@ struct vm_area_struct { }; #ifdef CONFIG_PER_VMA_LOCK - /* Flag to indicate areas detached from the mm->mm_mt tree */ - bool detached; - /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) - * - vm_lock.lock (in write mode) + * - vm_refcnt VMA_STATE_LOCKED is set * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_lock.lock (in read or write mode) + * - vm_refcnt VMA_STATE_LOCKED is set or vm_refcnt > VMA_STATE_ATTACHED * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * @@ -282,7 +274,6 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - struct vma_lock vm_lock; #endif /* @@ -335,6 +326,10 @@ struct vm_area_struct { struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + /* Unstable RCU readers are allowed to read this. */ + refcount_t vm_refcnt; +#endif } __randomize_layout; struct vm_fault {}; @@ -461,21 +456,37 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) static inline void vma_lock_init(struct vm_area_struct *vma) { - init_rwsem(&vma->vm_lock.lock); + refcount_set(&vma->vm_refcnt, VMA_STATE_DETACHED); vma->vm_lock_seq = UINT_MAX; } -static inline void vma_mark_attached(struct vm_area_struct *vma) +static inline bool is_vma_detached(struct vm_area_struct *vma) { - vma->detached = false; + return refcount_read(&vma->vm_refcnt) == VMA_STATE_DETACHED; } static inline void vma_assert_write_locked(struct vm_area_struct *); +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma_assert_write_locked(vma); + + if (is_vma_detached(vma)) + refcount_set(&vma->vm_refcnt, VMA_STATE_ATTACHED); +} + static inline void vma_mark_detached(struct vm_area_struct *vma) { - /* When detaching vma should be write-locked */ vma_assert_write_locked(vma); - vma->detached = true; + + if (is_vma_detached(vma)) + return; + + if (!refcount_dec_and_test(&vma->vm_refcnt)) { + /* + * Reader must have temporarily raised vm_refcnt but it will + * drop it without using the vma since vma is write-locked. + */ + } } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -488,8 +499,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - /* vma is not locked, can't use vma_mark_detached() */ - vma->detached = true; vma_lock_init(vma); } @@ -515,8 +524,6 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); - /* vma is not locked, can't use vma_mark_detached() */ - new->detached = true; return new; } From patchwork Mon Dec 16 19:24:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C52D4E7717F for ; Mon, 16 Dec 2024 19:24:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BEA7F6B0088; Mon, 16 Dec 2024 14:24:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B994E6B008C; Mon, 16 Dec 2024 14:24:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3B946B0095; Mon, 16 Dec 2024 14:24:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7FA346B0088 for ; Mon, 16 Dec 2024 14:24:49 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2ACD445075 for ; Mon, 16 Dec 2024 19:24:49 +0000 (UTC) X-FDA: 82901798280.27.DE6F3D1 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 2F5A8C0010 for ; Mon, 16 Dec 2024 19:24:15 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pclWiwew; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3fn5gZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3fn5gZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377073; a=rsa-sha256; cv=none; b=2ACky+eLoyoQ/1PM5rBF1RS4Tlo5E8bIcAT8jVsYyYCTJxbDNzD1QlxNSxEo/kFRcNrFZA 86wzTRaylhpK7Ewzo3POiRawrdP3DKPsjbhsUK8+T1uN0TwswcGq0pkRasvvZLEKhfKpqK l2RV96Ac7BlQfjJB/5wJoAwlAsEqyMM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pclWiwew; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3fn5gZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3fn5gZwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377073; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nohm0qlZ+D54asZWR+C7SIZfQAywm7JDUn18lM86DfI=; b=SgabJ9bw43IbF3z9LIuzcNq8Cii4YWazI+k1DfFZG8VlGAjd8oNQCUaC1ZIg51jjRXnJAE zdHhuUrSBU8ArEtSzzqGCzJ73+A4tj2g0P8EbsEjAMZAdWV5wYa/lPwARg5p0soOafmiBz YRUp6ATg4mh9ZDxsIdJtB/VuV0hKFi8= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-725e59e41d2so4106918b3a.0 for ; Mon, 16 Dec 2024 11:24:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377086; x=1734981886; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nohm0qlZ+D54asZWR+C7SIZfQAywm7JDUn18lM86DfI=; b=pclWiwewigFNnGxofi1OPwp763hk138cgy4clilITTG2mBGlgObLEoH3kkuelUjmtC KeLbcVToQXPE/xASUOgu2Yq9IQipYh95gLWNSuh6hTpEKH+KOHfgbctreRHt51pEQ/DP B9p/4CXnCsht8dXp5IqtsMLCvmiP1oiist9JgxEev2QPRTomQvor853m4vuYv0xm2lrF Agz1ibHTlFozZ2tEit7WKJSDGy5C4C1d2SDO780pR84Uso3Kb8rsKu4yqzkp+Basl3gb KhSG0MDqE6SrHyLfDYWu9vfcFcaBBA7NJF6XevaUequhohgO4wpMulqD9JhmmHjm05X9 xBYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377086; x=1734981886; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nohm0qlZ+D54asZWR+C7SIZfQAywm7JDUn18lM86DfI=; b=oSVaDsttCePDFwFETMf7notaWD+S974idad+5i2/LEal2KNye6pShgtBIEn/40ZaEG vC4qybY9QqS2ogp02ccH4Byl1II5F2EPT16rqLHIuVJbcPr9C7ROcJXvvxxEwSwbpK+s t/t4+jhhMW86wExnQHuzyWgsozCjNBxPnuaYABnp8KOQsTHq3Z2KHBcMB6J0KBUIwAAA dAW2OFjA6iqfO+5yMxyN1B5ZN282q37iumgWatN4IDEkZVbMmfmad0NDzUE/82lguJwk mznqQpa8SREIXZmOcnT11tUgEZ6TdqDCp4EySD6/TFgFJ/IInWwr2QE/4lVtg/GFVEI5 CUXQ== X-Forwarded-Encrypted: i=1; AJvYcCX9T3tEr1fy1kWmkdPbbcoxmNoOcadzkx99A65/XzsViov4nTyRkb26mSAhSdMqpoFYSPahFbovVg==@kvack.org X-Gm-Message-State: AOJu0Yz3uVsOz+yCduzr62zpKbuPp5yX9F8DhOXNdIR+Bg0XrB8Oejnu bElcYeyIUzz3O0I/j3W+MaQgb4YpmlDljQa2U8LAMZjE7o8D+XnvylVFUSyAVkONFa6GG/H86oG eog== X-Google-Smtp-Source: AGHT+IF9Nz4fP8YCDWuEvv8HenQUzivi17wIWFTDjOw79pIi9puAuyZi3edh0rN76x0eDFYowMyrfvXvJ/c= X-Received: from pgiz13.prod.google.com ([2002:a63:c04d:0:b0:7fd:585b:babf]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:c88e:b0:1e0:f495:1bd9 with SMTP id adf61e73a8af0-1e45ab17a26mr1379210637.8.1734377086086; Mon, 16 Dec 2024 11:24:46 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:14 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-12-surenb@google.com> Subject: [PATCH v6 11/16] mm: enforce vma to be in detached state before freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2F5A8C0010 X-Stat-Signature: ujqfy9b619ygmdfwoonnfihtfz3eimqb X-HE-Tag: 1734377055-46521 X-HE-Meta: U2FsdGVkX19NYvqPxjiLAHIpQvWDbcLMOorRupgzo9Bp4AfzBvQM5S58z2201Z6x3FwqxYPsVtPu49MfVyRpUsSuoT1mpBAzQBClbb6uM++O90ESQdnZK+AIFy1rVyWtM/0cyDpRy3GajMkB2EqJTylS19fQIs9zHjBV8AgGmnX1G39cDYjIczh6pDS1dU5xPx2JiYkRo+5T4sqilMuxHEiGvkYCUB7eDYKVjOkgzLR7hf9TQ0V3KzuF6TkonWIYqXU0e/h7oXiSbWCG0WDJHgw/9bjbtGLTauEx9OkQ6hqus0FpSQFr12oSlnt9BWQNx19JscWpWql/vz5TsRCg0kR4+0i0hVQFsytS90iwAqTI0B6HDzD9Zr5PqPZDV92zMjiAHrtFuWb3FLoxQxlQ7abosIrKiBr/YAjRnbgR3G0a0m2y1DIHP0RTkeN2N3if1i+WoP2PRNtWgoYZQfeD7422bEi+ipBXIA8bJVTsWvswxWgmIjokb5WE9ktC31N05fAbU9M6Ee93lKQYpn92dhNB2GCf8m9BUcmZZ6NYuT2I7078eGXEB2NWu6tnW7fbaQhrAWAbVeU6ng8F5LccmF8bZiK3nSIy2NpV7s+9AaajJqMdxoA8QZlOcROD2fagcKIigCQm4P2Z3LGLffz/Eqw+1JRbVcnYSXzr3PG0XQjhyIDAGqpqWRBSCDKEvv9GVcxw2NYAp/lFuVUnqQ/sBCdB0OYbu1KCt5Co8jGt73uK6vmBuOxRwbwJxQ/NW5ag8VBdNzJDYeIJu4NeY8qmreVSb3Tig9NWfRA9uUhOUvkzAkse5FEVafvwsVrmB8YMKV0zyYa2Oqb2i09SuGWvzpUvH9VyvHPbMeCHurHCbpVz/GSLzFB/VINn9e6t5pqVMzc+gdkn8IQqo0zVunsPSLdJBKJRrVgwEh6ODsyHunFlX0AOjd88SR05dlws8bW5GsMZOCDwyTOQxAnDL9b hPdAeCcp eBt0F+mPQolAEUUqnfWEFUdS4L+TEJQoE5+6Qg8auXS3Z2/0W1rjT6DE8Dyo54jHPlUK+0TK0rj3FDd12u3Z+h51S0PtlhWIFBvvsKLxn9sPNUSn25DJhJ0TCewKP64Lvp3gRR5Yay4TxM6DsK2nkEf80EWeCroruzI+ivMM65fgd38Z7u6PACkWKTCauscX8CBnq+h6sZW2V0i6wxg/L3i7aXoaYCuc1McH/cy0rTDTUWLRBJBQT4ltV0pB3H99cdI/cDFp/gIO/q2DvhHfu5Qzc5PahJ9ZTSWJs7pIiAHXWBOhUBNyW7GTCET+20WPUfgJxOKcrBy9Gf+n21IuGA5u3Qt5/z9OMUDZ5Jf+7xyDLLSTT014ZEejEnmRv57gviO8HkkBcfhmo2p0aOhbtBcm5d+09g1ReCZA7cuZlGY5UE135sEBczmS/xa3koN+mrbiJy8GVbeXvaNSAuppzHuf3huWnbaPcPiVoiqOzQteqhDp3Px1EGQd1IjjRlpsAIGdMbXW4JEP+l9U3s/8NZZma4fWFM2ICZVyPApzcSTmNhVg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.003732, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: exit_mmap() frees vmas without detaching them. This will become a problem when we introduce vma reuse. Ensure that vmas are always detached before being freed. Signed-off-by: Suren Baghdasaryan --- kernel/fork.c | 4 ++++ mm/vma.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 283909d082cb..f1ddfc7b3b48 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -473,6 +473,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) void __vm_area_free(struct vm_area_struct *vma) { +#ifdef CONFIG_PER_VMA_LOCK + /* The vma should be detached while being destroyed. */ + VM_BUG_ON_VMA(!is_vma_detached(vma), vma); +#endif vma_numab_state_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/vma.c b/mm/vma.c index fbd7254517d6..0436a7d21e01 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -413,9 +413,15 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - if (unreachable) + if (unreachable) { +#ifdef CONFIG_PER_VMA_LOCK + if (!is_vma_detached(vma)) { + vma_start_write(vma); + vma_mark_detached(vma); + } +#endif __vm_area_free(vma); - else + } else vm_area_free(vma); } From patchwork Mon Dec 16 19:24:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F198AE7717F for ; Mon, 16 Dec 2024 19:31:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8453E6B00C6; Mon, 16 Dec 2024 14:31:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F4B96B00C7; Mon, 16 Dec 2024 14:31:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E4076B00CD; Mon, 16 Dec 2024 14:31:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4CE806B00C6 for ; Mon, 16 Dec 2024 14:31:11 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 05BCA8081F for ; Mon, 16 Dec 2024 19:31:11 +0000 (UTC) X-FDA: 82901815164.02.DC2D6BD Received: from mail-io1-f74.google.com (mail-io1-f74.google.com [209.85.166.74]) by imf26.hostedemail.com (Postfix) with ESMTP id A91AB14000F for ; Mon, 16 Dec 2024 19:30:46 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=at2aBXpR; spf=pass (imf26.hostedemail.com: domain of 3f35gZwYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.166.74 as permitted sender) smtp.mailfrom=3f35gZwYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377446; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wSQ3mT4u4m7ofM+stpYMvUJpeXhYChT7fq9/vjKfHLA=; b=f4JBB6a36pR/EMfVy5Tj96UZekWKGMMApkxzGIuUyaIMAbFrHMWZ/zo2BjUWyFNW7onOGV 3OkE8zevoQAVX4fXzJtl+10G74a52MYr0tzLnAA++WvwKoklzxpi/h1YemgYiidnS+0RP3 ERyJVFfF+Z3t98ssYaO+8mStjmMTgmM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=at2aBXpR; spf=pass (imf26.hostedemail.com: domain of 3f35gZwYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.166.74 as permitted sender) smtp.mailfrom=3f35gZwYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377446; a=rsa-sha256; cv=none; b=sgoOIzrLps8i7AXJyExmx0id6OF5a5P/A6G46fR/f7llyuHq2zAiXztnumAJl3PnbXpBic M4vfqb+aU2nSIN8r8p4O4rsniHiGhZOd2metJYaF9rnUUdAUShA1CwR24VtBLFyFwqdVZP ip9sWj28N3BipIpEvKWzDouokrccRj8= Received: by mail-io1-f74.google.com with SMTP id ca18e2360f4ac-844d5c4a27fso866592839f.0 for ; Mon, 16 Dec 2024 11:31:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377468; x=1734982268; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wSQ3mT4u4m7ofM+stpYMvUJpeXhYChT7fq9/vjKfHLA=; b=at2aBXpRHWMyxaTMeFCOEsfQ3EzJQoQ+jjb+tKnbhbAdIKw+nbknwsiqw3rWZQE8th MkN5EfX9vkCmkTQvxX5A0lQnmnaqihPZjlIav2xZjDEW+ywDn0QOxJ1nkLWKtra5lCjM ykr4t3xNWhxWfTUFBYTK0HBOc2SOTXkhARAX56NRMI1pcK8W/GY0wP8WnFM8oS2r8RVQ 6emdMlvQPdnFo10IF/bIPrRS33Ltmz1pr2k9qYwPqtAhkHX7J7s/3g2nD+0aupKa71fp tqg9/rmW3rczfuUE58xEnGaqN5iYmpMNUcYDFesUgZWl128i4h6G1IZ8HlUD8fiq2/M6 le+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377468; x=1734982268; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wSQ3mT4u4m7ofM+stpYMvUJpeXhYChT7fq9/vjKfHLA=; b=g7IiOQu0NNZlRFfDRni0rkXdDLQeAzMDYFV12Fg8AA3nfSuW6bbFaFbripRHEv7Kpb A+p+KSXPbYvsKL385SKlLKo/RjtmJmtKaS+q5Qts1+iY2E8UmNF4Xd6eSwLMoZ0JtwiA OBK5D9xD96v9TbvFFj6NJPWIsUSZ8WbKFpQp833TB8wOY7wG/0akuDJoAZD6QquYzxge 7rrOIZOWFJszkiNppTJzEGmo3oKLidCcvhJ/U1jiXXVrE2LbTuRMMJ8bX19Oj8cLdx7V XTFYMqUyXyHmftE5D/VcIcvYdulnOc3+O4W79tfPriCTvEpdU0lm2zz79MCQGQZHHI2c 4OXg== X-Forwarded-Encrypted: i=1; AJvYcCU9a3/lMbttX4bqpONCVeQmPAtWn3xgygwK0wWV7aHlxF7qqDHe37gUi5xSCn0GrzWPCwCrp4DKUg==@kvack.org X-Gm-Message-State: AOJu0YyEp3ISiaJg2ZOaxHJjWHPYO4ceCnGgRnojjEHB7ovfE4o5sz4n ZTzNd3+ydrpUnL9/JDsP/L7rQ6TAJww9bjxyjCUPEoT+9wJHiEIo5mHM+KHJja6jA3kurL0WTse skg== X-Google-Smtp-Source: AGHT+IE4pawsfAUH9TLKs+lZMgX1XervLrLNUWY+S4QjAFjLPVQLu5/+TUucSp06ogqQfpKVSrSRx/CLyDw= X-Received: from pfop18.prod.google.com ([2002:a05:6a00:b52:b0:727:3c81:f42a]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:6d8a:b0:1e1:b12e:edb8 with SMTP id adf61e73a8af0-1e462d60708mr1104953637.30.1734377087952; Mon, 16 Dec 2024 11:24:47 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:15 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-13-surenb@google.com> Subject: [PATCH v6 12/16] mm: remove extra vma_numab_state_init() call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: A91AB14000F X-Rspamd-Server: rspam12 X-Stat-Signature: g1h5ituoge5kt1staidbbgnkipzi5m4o X-Rspam-User: X-HE-Tag: 1734377446-99439 X-HE-Meta: U2FsdGVkX19W3dnbFMQBTgf5vTL5e1fr97TgzkgGlfOHKX3wYkAdaJ2ez7dQiT/E9FdnhQS7ItOZBKcZpAD3C8gj+x46Wu0rEPZBGdNFle6TmJqelKSBfNiNK5+sXttXtnlHHuptHHB03clDkuDfmt2F0K9RN57cYAjT4zUqUKmLUDAbx4/IaBTdvgc61VDy0epJGebh6KSIggarIlPAIN30N/eEDPnOvbsmBMOQUrxvAbSRviksxhwqVXnL6QHdeQqyrJmkz166gFkfPwpvWiVC1oPEL2775MBJqtqPSn4k6eYiN17o3ZCH3kQSBUfKSaA7YD0+cHOI3kiM/w3EgITODEFyI8ChjeX6iYcd8UF9m4ibVwMx0+SGyu0C9dfzLInEulpmBHrtA89pu5OkbSXKlHah+ARy98UUCLaQmC2meb7SOXFgsXUyXKSWaJYCU+n/RzIlgQcCyyxBG7hA1XviIxlqaw/wICm5P1sErQonqn2hqi3AcSKScOMEr+r+tX4o0OIfVmutQ6tEO+smygGXfHkTC/Y/3wDYtk2Gjt4TcH0GXYUxWCZmT6AtRTx/CB2togh7KpK+Kg6kXUK7vDKmY+9MMTt6/rWcSLt/7zzUgCix5zfqIRz/1pi2AU1Wtn9fkovgN+ZZrhut5XFgNsVQ7XMGWjph01GwWH8lMAbnjbVXrHAe9ELDUYlxuc/QIEMU6QfnbaJZngZMp0TRa06CCvADx1/s0TKuji/3KOaq0VJAzZ10XA4vUpLqDqWte39pRYBmdJ6BWCpIUrSCZ+wA/l6OUaKYR3SdcReqwJ3qyQxMo/bCHA4erzwv9NAVAEMfFvPCdnmOe7YKdKzEEcYcipjWPDClyekQ3haJ23It+/u1Jb3vFxyGzBwF3E3MgI9o6C6fuw5xok8MwWzmrpB5fnCKMxD2o5sC+LCxR3zHvKf3/f9zGG/+hrKpOF9I5Y60XHMpGR2D90CdbRc uxFwbs86 NSMK99Od7RyNQ00EtwbENX7MgSyT1xPq5cdEYYzW9GAi8DazdtOoJKFW7hWqiEfcQjFci+BAd68m12qVb345ev4msniacAXNyLFT5fEa9s1NBcTjYgyxWImltCm8sL+8uf+AmWgjTboOW7PlDx9w+sb0ynjr1IjTZypJ0kM6pGlHA+Qp3LECrLDVZRMZisGOZWXsmRYI0w9unGQiihOd7nSL1moTXkEMjryZlZSQ4rUaCcuo8nazAR3JEHrkbt6FxaZD1WUDvWYHZylhBs8JaqyhVxpBAD4WyO1RjPdENG41kGkQIm5r7BVae3x/lwu248SHVex5KxJ8GT3PPRZmIIDo+TKu9XpK6ubiEbRGedmL+SF5XLnwMr9VfETrW6hCA+x8KRQ6gwgTGDYfcq6C+HBj2PFCzykN5zbtQ5MrDegPI/upPJ7REo0ZA3A382qLx5rAzMmqvjqDABho38jm9JLi77sGK8Pn+0Y52AxRvGCDoKXGkA3TzaQbSEe4zw1dA81pS9oNDp0yW+4PeiUEE6QyU4+Gfn6EHlapNo3A5Pj0pSlN4QK4DfXJYH19DIZB9bTwlnNcXEVA4LV9UlBTTQ8sTaQ== X-Bogosity: Unsure, tests=bogofilter, spamicity=0.468236, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vma_init() already memset's the whole vm_area_struct to 0, so there is no need to an additional vma_numab_state_init(). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d9edabc385b3..b73cf64233a4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -949,7 +949,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_numab_state_init(vma); vma_lock_init(vma); } From patchwork Mon Dec 16 19:24:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C63AE7717F for ; Mon, 16 Dec 2024 19:24:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD8206B00C1; Mon, 16 Dec 2024 14:24:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C603C6B00C3; Mon, 16 Dec 2024 14:24:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD7D16B00C4; Mon, 16 Dec 2024 14:24:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8B1EB6B00C1 for ; Mon, 16 Dec 2024 14:24:53 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 476784508F for ; Mon, 16 Dec 2024 19:24:53 +0000 (UTC) X-FDA: 82901798784.22.F63E864 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 325C12001A for ; Mon, 16 Dec 2024 19:24:22 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=c8i5Dshj; spf=pass (imf13.hostedemail.com: domain of 3gX5gZwYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3gX5gZwYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377077; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eye0eVNbsQT4TepH1/hu+cBmomL+3TKzfNRnkVWyI/4=; b=S4KPLin/D/tMt0POj0mB4yGc2ABzyPTDAnaz5ZWA+cpUSCCSUNtp2PHH52NYAOk8VxIH5g +Ob3nG4BscDdKCGqEKHFwrf81O0YL4Ugvv5RwBLYfDDrOib9R+c6h7EHSxlw0G6kTeqd8T AtAcPSxQp0C6wX0x8uYcxTmwk0Dz+go= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=c8i5Dshj; spf=pass (imf13.hostedemail.com: domain of 3gX5gZwYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3gX5gZwYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377077; a=rsa-sha256; cv=none; b=pj6oDFMg58W93NcOmzTwt9fW9shlW/iDlWyCNSL0R+SVIBgBpIGU980HML8aY/jXshN+bP kg+5+/zcnVaitBwDAde6VBfZB/KG96G7wzezkZ3mrbeVi+CpUuAngysSepZCX/KmxnHZnp vTGQMfj0K2lRvgz67ghQEVJfWOzXUm4= Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7fdde44f4f2so2850292a12.1 for ; Mon, 16 Dec 2024 11:24:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377090; x=1734981890; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eye0eVNbsQT4TepH1/hu+cBmomL+3TKzfNRnkVWyI/4=; b=c8i5DshjvR8XtjHzAxhEmXs3u2TkCgmlj87RYy2sSk8Ty1vy1Ov6nkc+LjrD+izg6P KNRxkhflyrsZ/3wQvHMDKJc0Qt5HLYTDmMQQsM4iG8yZnXUbKySxBGUGYxFcP3P5bnRU hHG0DrswVc+mm9gK9S6HwXp/+R1Tlrt/zQgybaELmHg1gogEet/Xd5WnYm2mtatjRrez XsC9aOwHe2BcNik27wmUXqBTseMsPOMhmIcx4NVguoGH0SCUw6yH7XpyeO/0SGzbzlTa xW84YjrPcMWE/MPy208ZPqNI5BTPXTi9lZQ1+oB+88elEXlWOsJJfvcPiyNZd8K4dm6M Hpbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377090; x=1734981890; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eye0eVNbsQT4TepH1/hu+cBmomL+3TKzfNRnkVWyI/4=; b=hiYHJgsm9H9CrWn/8KWCxnaPtaRouBJPKK2GyPboJugoGNqCoc7+XocaRM+fYovsqP ZDOvAgleWZreC7TkBFTwqrcTnrOOaz2N9kRiF+AL6HzpgBuBQr1OJM6LKBLQfCBWJCF+ fQfGCKcj7GtlDlwUR6MVMrooCzZs4jivGgyj6GcH1+EXA1VAObqgdZOY39KtbfsShpgQ Ja4tw7su2Erq4z80Tw9tQzyNXmSaQEfgaHdgX7E2E9E4oyvR1IbwkauLx+oWirAg8CpC JBrWX4onYN8zIL9JsZ5NgZTMXdwEPlxxIhn7BhG6ckaBkvk6DcabW/ZaLMMKpi6nPQrQ PaMg== X-Forwarded-Encrypted: i=1; AJvYcCWY7dTAeootYBIhsYIhHv1mTOh1OQe/Yz4UmsTdlgKh60CrxjOn0omX4/8r1D8x6ktNGkw/TkWVJg==@kvack.org X-Gm-Message-State: AOJu0YxKCoUFbBIA6W+wEmQJAfCl8ZaT7ipLCWknabWudvKVRWAfSypq wcnTKCQOnih9cXn6euSzLYWChfSNdgiDtipZt/U0PyCtvOXBRR5yx+XQOQu/AgCO0e16VIDjjZ/ Z2w== X-Google-Smtp-Source: AGHT+IFpyvj1Cl0PXtve1a3hZL9RrpgDnkSC5MatBr/H5K08KaVaYCnO8eim/kS1J5eWUMkZBi1DYQsuwp4= X-Received: from pfbdf1.prod.google.com ([2002:a05:6a00:4701:b0:728:ec44:ed90]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:9f04:b0:1e0:b5ae:8fc1 with SMTP id adf61e73a8af0-1e1dfd3da76mr21067006637.13.1734377089941; Mon, 16 Dec 2024 11:24:49 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:16 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-14-surenb@google.com> Subject: [PATCH v6 13/16] mm: introduce vma_ensure_detached() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 325C12001A X-Stat-Signature: krciot9t5khbcdgajp7cxdufummtknqp X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734377062-805715 X-HE-Meta: U2FsdGVkX1+xwgSm6/S1LHEmoUPPtieh8P466p3iS+K6HkO96IpGPX4dW2adtmuR+diujZ/MGKh2wxRisD0+etcFayOP4ptCzkEkkQUddRI5YZw9HFjfohvRd4tiGjH6NT+NoOYhzghODLTYLIREBf73HhErbKDCDwjK7CVL2Y7LkLO0d1//vXCn/8yW+DK+ejoHvHfQOc7N4TmFL0BJCZZjbe/+VsvZsINpMZmgdSFm+K1mPNtkkjz202vGhVSOpKq6yzJGKl3igTO8VCTwTgQffLV9o4lAcgzWFMXzNFq3TP05SHTLt6nLowvCS9LYMXI59g65blUB1egYOtsDRGLvwQSFH2TXvdbEWrxAMN3yvSsgQqXTjG8CyAmtNUTSgRzwKNZQz/lcU9Oxm5nVPETjjvvdC3IlWQXMPJMQ+wkGZaV8I+tgKcgJmYFf/O49ZXIaCbwj7PGEGBbFjNnoIYvpHrtan8DgAYGUAeAO8baOfcyVi3AQPw9BDvSPbaqG/mEoL3Dtm+0qB9klzGVXHUwxEFA3y01ggXi0QDy+iiqbbWhxiMP7/8+5iFMUte2/3qRzxk+D9clp+WWzYPwhwjwLLeOxZByHAueQ6L4BcDh+2pZc6V2JolGirBKLM2HqXj3/dUSFa6HQ4CRwZeIxFbIwud6Q+0vs1pFTr2i8H64KeG8VhUALd2kz8IbqWDDBkn05VpToVwK+kLom3B6SZjX0mzQ6h7YyqhnachQ0eG3h9QngEDAg8X6cwuWw5G5CJkjs63fvHBPfLSBqJJ6QTyLOp6CYoP/pwccPrF1C508dZotT1p+dN8yIEbp1rWgR/ENOozgfnW7uV9tu9+iQn2/Yp6NfuspFofDXhtRwC/aXSILumEYSriydSid0R6FpsB++eWTDyTVlKc3GBsfnETPRuf3/NwUphFG8WnsRULCqYvkm/nSoU6QdiUzPoKp+bDbyiyct0PUJ64xm8eX aQYK7Rij zSyTtgjdggdwDTbAm8x4b1+eWMn5tmnxZ3LCUXy5Q43jhm+bKiDt22oKsnhaeVJ54aZ5RoM/S2of6mFx5OMS/IeE6o9qfc0ctyMwDJgCUy4nrqMVw3B0TOwuvd45lx1am2wlC68NoATGv6wleMb8Pel79IQdjVP7ceKcKERra4ket+4+i3eMsB9uVysdpuBy1vpKaoZVBnJBgxdCNFenU6+azWobdK9PICCJPDiwZEx2JKY6nkevZjtcept983xhGbBDoEZNifO8TV4ZJGmA82b/PKMC/KFsYgX/DPAYINkc3qKqozoJc1S5t6QUrmYphoFciB8oOP5grosXbke8jiMDgeGoHChL2T0UVxRJNZPAdSTIn7o7RCJr6NeYwbINQTF1Rk0eD0NsXrCo3D+i+GL/qoEs+lIym3tUqGDy377vVQayhZRvCja3Anl4bIgmije131FilW0I0CXREh/Cq/wPkUl5Rym3npkW0yS44y6UwWm5H5oGyNYuX/BwJFu61osRO8zODIlPjd85DE4bwiB4OhwtvUjOHr/crptkRBy9UI6EelHnteANYItY86k0iD68lNcSR9e/ghJ2LIeKi0UZluQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.005883, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vma_start_read() can temporarily raise vm_refcnt of a write-locked and detached vma: // vm_refcnt==1 (attached) vma_start_write() vma->vm_lock_seq = mm->mm_lock_seq vma_start_read() vm_refcnt++; // vm_refcnt==2 vma_mark_detached() vm_refcnt--; // vm_refcnt==1 // vma is detached but vm_refcnt!=0 temporarily if (vma->vm_lock_seq == mm->mm_lock_seq) vma_refcount_put() vm_refcnt--; // vm_refcnt==0 This is currently not a problem when freeing the vma because RCU grace period should pass before kmem_cache_free(vma) gets called and by that time vma_start_read() should be done and vm_refcnt is 0. However once we introduce possibility of vma reuse before RCU grace period is over, this will become a problem (reused vma might be in non-detached state). Introduce vma_ensure_detached() for the writer to wait for readers until they exit vma_start_read(). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 9 ++++++ mm/memory.c | 55 +++++++++++++++++++++++--------- tools/testing/vma/vma_internal.h | 8 +++++ 3 files changed, 57 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b73cf64233a4..361f26dedab1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -863,6 +863,15 @@ static inline bool is_vma_detached(struct vm_area_struct *vma) return refcount_read(&vma->vm_refcnt) == VMA_STATE_DETACHED; } +/* + * WARNING: to avoid racing with vma_mark_attached(), should be called either + * under mmap_write_lock or when the object has been isolated under + * mmap_write_lock, ensuring no competing writers. + * Should be called after marking vma as detached to wait for possible + * readers which temporarily raised vm_refcnt to drop it back and exit. + */ +void vma_ensure_detached(struct vm_area_struct *vma); + static inline void vma_mark_attached(struct vm_area_struct *vma) { vma_assert_write_locked(vma); diff --git a/mm/memory.c b/mm/memory.c index cff132003e24..534e279f98c1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6329,18 +6329,10 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, #endif #ifdef CONFIG_PER_VMA_LOCK -void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) +static inline bool __vma_enter_locked(struct vm_area_struct *vma) { - bool detached; - - /* - * If vma is detached then only vma_mark_attached() can raise the - * vm_refcnt. mmap_write_lock prevents racing with vma_mark_attached(). - */ - if (!refcount_inc_not_zero(&vma->vm_refcnt)) { - WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - return; - } + if (!refcount_inc_not_zero(&vma->vm_refcnt)) + return false; rwsem_acquire(&vma->vmlock_dep_map, 0, 0, _RET_IP_); /* vma is attached, set the writer present bit */ @@ -6350,6 +6342,22 @@ void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) refcount_read(&vma->vm_refcnt) == VMA_STATE_ATTACHED + (VMA_STATE_LOCKED + 1), TASK_UNINTERRUPTIBLE); lock_acquired(&vma->vmlock_dep_map, _RET_IP_); + + return true; +} + +static inline void __vma_exit_locked(struct vm_area_struct *vma, bool *is_detached) +{ + *is_detached = refcount_sub_and_test(VMA_STATE_LOCKED + 1, + &vma->vm_refcnt); + rwsem_release(&vma->vmlock_dep_map, _RET_IP_); +} + +void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) +{ + bool locked; + + locked = __vma_enter_locked(vma); /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -6357,13 +6365,30 @@ void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - detached = refcount_sub_and_test(VMA_STATE_LOCKED + 1, - &vma->vm_refcnt); - rwsem_release(&vma->vmlock_dep_map, _RET_IP_); - VM_BUG_ON_VMA(detached, vma); /* vma should remain attached */ + if (locked) { + bool detached; + + __vma_exit_locked(vma, &detached); + /* vma was originally attached and should remain so */ + VM_BUG_ON_VMA(detached, vma); + } } EXPORT_SYMBOL_GPL(__vma_start_write); +void vma_ensure_detached(struct vm_area_struct *vma) +{ + if (is_vma_detached(vma)) + return; + + if (__vma_enter_locked(vma)) { + bool detached; + + /* Wait for temporary readers to drop the vm_refcnt */ + __vma_exit_locked(vma, &detached); + VM_BUG_ON_VMA(!detached, vma); + } +} + /* * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be * stable and not isolated. If the VMA is not found or is being modified the diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index b55556b16060..ac0a59906fea 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -465,6 +465,14 @@ static inline bool is_vma_detached(struct vm_area_struct *vma) return refcount_read(&vma->vm_refcnt) == VMA_STATE_DETACHED; } +static inline void vma_ensure_detached(struct vm_area_struct *vma) +{ + if (is_vma_detached(vma)) + return; + + refcount_set(&vma->vm_refcnt, VMA_STATE_DETACHED); +} + static inline void vma_assert_write_locked(struct vm_area_struct *); static inline void vma_mark_attached(struct vm_area_struct *vma) { From patchwork Mon Dec 16 19:24:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5258CE77180 for ; Mon, 16 Dec 2024 19:25:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9E646B00C3; Mon, 16 Dec 2024 14:24:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD5E16B00C4; Mon, 16 Dec 2024 14:24:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92ABB6B00C5; Mon, 16 Dec 2024 14:24:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7391D6B00C3 for ; Mon, 16 Dec 2024 14:24:56 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0F598C09C8 for ; Mon, 16 Dec 2024 19:24:55 +0000 (UTC) X-FDA: 82901799498.29.4E7460F Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 632B840017 for ; Mon, 16 Dec 2024 19:24:12 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ym6+5y6z; spf=pass (imf07.hostedemail.com: domain of 3g35gZwYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3g35gZwYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377073; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t5fnSRoB5oFCIXa/V8gP+VbtOpDmgl8LzmUTLWBvDvo=; b=h99i5n8MdOuv41ScopfEB6hbaZRy1ZmaEiPONWvCZvZ0NXuVqivBDNylnKVypzaOF9gdpI rIFzVbyTLfyhaxgyPQu6oVCUdBfbZ9M1qZZuxp+UmjcZhGY4kkQPiC0Ooe+3M8Y9nW2r9H dlqU9mf7pMeYzv4Thg+P6kdMyuni+7s= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ym6+5y6z; spf=pass (imf07.hostedemail.com: domain of 3g35gZwYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3g35gZwYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377073; a=rsa-sha256; cv=none; b=iamhA5wHNvR01xBcFpz3KgHyLHZC6PG0T9dgvFmfrck4xrm04cZfHZPmifnBPjSnhbMBpa L/NXPfVmIZ6O6fUuUqIG4y7wS7PM6RBhuFcng7J0vYgqhkN5GhM2oHu1xjK/VUBQna0T8E 53K/abmmffJdH2mvoL6tY9+e5jHtH/4= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7fd482c8de2so2885134a12.0 for ; Mon, 16 Dec 2024 11:24:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377092; x=1734981892; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t5fnSRoB5oFCIXa/V8gP+VbtOpDmgl8LzmUTLWBvDvo=; b=Ym6+5y6zNkwI7h49+NgT3khcD63NCJ0PgPQXugEHvSN+EP2FO6CH0v8NVbGZwUM1wp LEbB1eV6jrl8z/p8ehxNoN1chh2g84hBz2RHiSajiS8HsZ6WDN9hHGwIlbD09BWpnqkJ FJQ3w/3hcoAXv2t3/K7SjB9v0gMbV7Fdp/sjsp1wI79gVJlsqBXuoYZXkJh/doPlF4uY +zU8b4KKzb7fMMICIlgFRG5ygwFDf700N9E9j0mHET6yuGeQIyEOvm6fCGHH4bYlLKnS T2PzXYpIIw0JLSG7Hb2FzvQksr1/0beAskgyVbOW47OGQcqxIIXTuoEm9j0j28BmnM7H pCWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377092; x=1734981892; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t5fnSRoB5oFCIXa/V8gP+VbtOpDmgl8LzmUTLWBvDvo=; b=IO1apYi5bY6nAGirS3B52wv3owLUyIhyImc6WrOcHibrwSAKXwUJLEY5w05M45OZzK V8fvurYGErctoAxZmWbvGhu5dTHj685EZjPEmOvYI0SvJ50J2LF1d9gr9nwoStSBcTme JJt1WOzWqNSE0fjIgdArc+TSIuJICaxCKpodt8QSqLkHVpJiyzffwKyJcQMTkeJ/yhHF gKeW1BQzcAwC33qZJIFpfIP1jepkqwLsjn3ogU74TRKjBIVpPOFaabPbRDsosjBHvO7R cOTSJoKKgW/TfTXbYBi6gCkHiJR6shmBSSpQvExCrpk4EFimST2JozQ2LBh3iQ4zdUO4 huCQ== X-Forwarded-Encrypted: i=1; AJvYcCXnLiThdfuQdEaJr3BeEFlTTCunqy7eqCy/nqvkSmcntrFHtVjXBaMjgVPIQOtIfoZp5kZxTFGzWA==@kvack.org X-Gm-Message-State: AOJu0YzfMuDpm/AXLoaafhgFHU/fcbw4o+hHCweNaKqJEczAal82xH38 D1PMpxGBzqkYWS+O6MD7SFHk0DO++Rv4Y9bn7dfARSx0Np2ESbfHsnhCGSfg9cZ5mNYyA8xyaf7 xWw== X-Google-Smtp-Source: AGHT+IGeGCIjsB3KXvsuHWb/nocj10tzreW4ZVMNCmw0sPpWQ78cfKvuNdThF2i3rD3x89K8u/6hh2RJEJY= X-Received: from pggk15.prod.google.com ([2002:a63:d10f:0:b0:801:d768:c174]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:8885:b0:1e2:5cf:c8d6 with SMTP id adf61e73a8af0-1e205cfcb77mr10415260637.36.1734377091912; Mon, 16 Dec 2024 11:24:51 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:17 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-15-surenb@google.com> Subject: [PATCH v6 14/16] mm: prepare lock_vma_under_rcu() for vma reuse possibility From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam05 X-Stat-Signature: qj51h8wbwzkfopkt6ye6rxah6dmjjjbu X-Rspamd-Queue-Id: 632B840017 X-Rspam-User: X-HE-Tag: 1734377052-157086 X-HE-Meta: U2FsdGVkX18tdrkVYm8KYT3JXiz+kv52cBeuSbA/k5UPnBdc1td5O3v0ynBq5uTqCrpOWzYlhutCb8DiPMvtWZ8sPBi605V7Cq6L9mq1m2WFH1NoMSEqeZ856nCk6bqIAVwGjoG1T+Fpk8JyZ/DhPq8F+X/KnrBYDzmdXzg1H1SuvKR2iFkO4n4CoY1J1GrNld+uk2IYFuLN9TaHax0tmDN1dAZhzrz9MJzw1wByx9ez0e/cVx6EoGPe9+dR4LE7TTRZ98HkHn50leAhYMq+kF6uabmnQN1pkfUARX8hKmbJmW98XOg3Jbp2jIqhjsjQ+/SZPEIMTeBKMfRzH7+w51Fh8ykEjiB5X2BLIzbbBDn9TSzPwfvHZkGN1FhvyFiy24f/2cgcPYRIQzcNqvfQRlUC9ok3GUuy13nkKQ9GY6EHPGFsXBKTAXrJZV2jb3ZUKzT69CKzNCy65MEgPK8/w3p66y0581MX0vMOgI2pvB0LWddzjc3q0OeU4pnZHeqE2ZR3LyqTtl7HStcb5UawjlT0wS0D165WKHOzQOwzMrMBpPPkJSROI8a23Vx7I+oU95A+ExMKV65ejZTA10PQ3Qsr47jHdmh+mr9STkru/tFNqBzlJ8yUvxhQvvESpPNZeyZlg7q73sAZtszY7UD+w0S5f06VgaxaUpw2HLB0rxP1sv6W+JZr1ht8cj9xW0U2PRjzL5WbRl9QwkokNJP63s0lRSpGNps/tRF564pLUZxisMNBd5Tz/x+rNyPofmtH45ZXTHnGDGsLxDddDhOWaX1WIr1EeAlXmZywZse+hBhl9GzGSeTGNHLtpR0iHlPjK03SUYVQLMmRZOIAniy8X8pPEErtwuq2Wp1cadV1QSGfDmHZZbLaOqZw0HMRTLMecl7lO+Aa8kKxIZ6AqJtp+PKoald1PcoQvnu8v0LEC/zKvCfNJ0tahtli5nGhHykAeXMcq99LD2dAelAw4gQ rvuez/Wz 6yOI58JgJayVB2+ie/TliU/TPoimeWfZlc/m+nEhYQqnUhgx23VyZmLevbSqMaZAea2FVdjaJucLQebFmDGpUmuFIDe9V3PZmHNS9TYKJFFbLzYWQn0VQ7UwIEgpye7Fhyuwsi23hbcYYpfURmhN3aY7nCLXLGDXu+FbptML8AE8/BjTbGxvRd6P+c1z8eCe5bH5eIr0LihA+Oz0JTpnniSFJDoDAHCZLuGVdcMQygmQYy1h3ljCVZByPnpUjR5tafzX5EOkGbDAFLid3fJTiJNkxO6/1C0y4pUkAwTpErZU5JHCKQb0/oigwIZZ82FeVz4AX4UIliuckSLe+xnuVlyhNsg2W1mr5wBCsV3mhFqosZMjGOiOk1LPU5Hosgkabx0NI07X4abIKbQyMUBelpANc+ZWySwIa491CtUXV5/Q3IDShiel6pHgKjwvJCGH23RQPl/6JWt05BjhlM23f6GYT3fSVcVv6ABt5XWYFE59LkyN8S9Mvv4Ez8Jmhi5yyawCnw0irOenqcYN2Rfr4TvLHzPqxrOBfMEKZ/dxdn6C2RGAG5Rt5t/VDfftbsqCc/oeFVmi6qR07vGxA5HMC7/mK4g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.016091, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Once we make vma cache SLAB_TYPESAFE_BY_RCU, it will be possible for a vma to be reused and attached to another mm after lock_vma_under_rcu() locks the vma. lock_vma_under_rcu() should ensure that vma_start_read() is using the original mm and after locking the vma it should ensure that vma->vm_mm has not changed from under us. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 10 ++++++---- mm/memory.c | 7 ++++--- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 361f26dedab1..bfd01ae07660 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -725,8 +725,10 @@ static inline void vma_refcount_put(struct vm_area_struct *vma) * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. + * False locked result is possible if mm_lock_seq overflows or if vma gets + * reused and attached to a different mm before we lock it. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { int oldcnt; @@ -737,7 +739,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return false; @@ -762,7 +764,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (oldcnt & VMA_STATE_LOCKED || - unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { vma_refcount_put(vma); return false; } @@ -918,7 +920,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init(struct vm_area_struct *vma) {} -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 534e279f98c1..2131d9769bb4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6405,7 +6405,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) + if (!vma_start_read(mm, vma)) goto inval; /* @@ -6415,8 +6415,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check if the vma we locked is the right one. */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock(); From patchwork Mon Dec 16 19:24:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5566E7717F for ; Mon, 16 Dec 2024 19:25:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8A1B6B00C4; Mon, 16 Dec 2024 14:24:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3B146B00C5; Mon, 16 Dec 2024 14:24:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B5376B00C6; Mon, 16 Dec 2024 14:24:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 68CF16B00C4 for ; Mon, 16 Dec 2024 14:24:57 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 25330C09E8 for ; Mon, 16 Dec 2024 19:24:57 +0000 (UTC) X-FDA: 82901798700.07.A6A30B1 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 36865C0010 for ; Mon, 16 Dec 2024 19:24:41 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OROA+Nrh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3hX5gZwYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3hX5gZwYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377081; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kODX23DYVFRUazWZEBIanW7JFjEXStT07MxX5tftr1k=; b=AZX+QrUKUE6tnrA1/UcvjFI/Y6nXsdBeb4f8G3AnA1TGPBSA23UgNJoMepKQN2G1DAvvdl 4Im65epxTZeFb3/8Fe5p5eBGmjzgpU1xX+iRdUyT6aoyysrrnHY6Gj2yUC4Gvr+mxu/c7l uD9iQAQDuylE4NVjwMwmTlKICNgE1P4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377081; a=rsa-sha256; cv=none; b=4+iYv11iLTHY0axXEXgwkrnFiw2FP8X3jM1E1DKfF4MEzWAzaRHVieBvI+xG/dQOflx3d7 PhzCawLgMmbLx3gW2MZhg75uHkQx3yhL/ldw8ORZFkw5HBtrKhF8w72nnaknX5QECNWWCM O06gyKmFykvu2VcJIJTF0L2bKYV++DI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OROA+Nrh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3hX5gZwYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3hX5gZwYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-7289afa200aso5921859b3a.0 for ; Mon, 16 Dec 2024 11:24:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377094; x=1734981894; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kODX23DYVFRUazWZEBIanW7JFjEXStT07MxX5tftr1k=; b=OROA+Nrhah0ZCRy3lL03hQDhX4HHfbIaURtFj7DUVpzu0uf/McITe3QOs3WfJXMBSG KVWxGHTlG+RnAQjhW8jkjo7lFTFUZFlt7MxGDH1EuecMuuWuTEWtCsL0vu5v07lLVfcK 1Se376A/Hml2QabjSLEgxH3rh2A6G8Qhh+aqKTZw8MUqezgiELPqEB4atBAgDN5JXEjz 83cCZZlH4IlxpEuFns05gMit+qsBVK37HKDp/KhKd4aEqy4O3ml9maAxlQ/iGA/j5ZS+ KlUvUhcY5JDj7PKw6aOYnf1TzDBnCCMsZC7nEv/mXJm3vKmHc7OQjUTx2SLVzXHf46q+ TCeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377094; x=1734981894; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kODX23DYVFRUazWZEBIanW7JFjEXStT07MxX5tftr1k=; b=eao4zYZfZ2vuFL5tdWqpw5+TypGP+HY9mqnz9JioMj2JLp2/6Gl4ACkpXFhb+8jA3X WUkXD/sTsjNXJmMeHV3r7QljfNCuvEwWF7HbtnUQ3/dfDV5vxc7Eh7+JNozMzYxOOsWV p5Nme3WHd1xN/yrlBRGKy8RbSUI3NpBCHaa+ffXR7ii/+GMtVct1YQwB00LXILGlsK4l xVLSgG3lRgIGEwESwm5HS/lSMvMM5FV/vBwNOAiuvAcFJfc55VxG0y4472lWiOFDM7gB F/eSOA89kc2U/gk6GbAopaPeg4uY7fg84ZAeIpb+xS+cAqAiO6GHppQ6jy9wlZX/GI3L QuTw== X-Forwarded-Encrypted: i=1; AJvYcCUN7MdFqmwoRsEmIPygizhjgSCrZWOKz49quhLsikmw37inj8zM7PXaf1gH7TPEnBQxvuT5jhwr0g==@kvack.org X-Gm-Message-State: AOJu0Yy9wnqXbPwHJiAPuCOkmAno8Ho+g/iatk45prrjntySNBT64L2h BeiTzmibH68qeAheq/A3YH05U7TO0M0XhUFV+CKLHXhnbSzzSJqcMs65bHvT46rIBsiVI3hj0np XKA== X-Google-Smtp-Source: AGHT+IFs6nvWjzM1OoknSdPIN0ghPnyqI6YPL5avZzhKMYnwXQrh2jvp6FsUbYnz6o+KIo68RndWt2pA8Qg= X-Received: from pfjg1.prod.google.com ([2002:a05:6a00:b81:b0:725:8ee5:e458]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2995:b0:725:eacf:cfda with SMTP id d2e1a72fcca58-7290c25ae78mr18948341b3a.17.1734377093879; Mon, 16 Dec 2024 11:24:53 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:18 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-16-surenb@google.com> Subject: [PATCH v6 15/16] mm: make vma cache SLAB_TYPESAFE_BY_RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: 38aa64w5b5pob78ciogobw5id5ojdt6a X-Rspamd-Queue-Id: 36865C0010 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734377081-129332 X-HE-Meta: U2FsdGVkX18mn0/0xJvCCM3XGH5OCIqlRyHmLFcvzwOjJ6CvOr4mwBcJ1dsDjPiKQJcvcF1/fa0rY1tC6Qwi4PAaN7QA4K3wIrWphpruzeavK2q6ExK54U+np3FK4I+o7wuelJUP0uDSK4o5rWNtg2GKsTimjnFuwOcBOo8gExGJJ/6cHB3xpOxxpGn/+zJUK1Jl6o6L9XCG7YWcM0am0/mCMIRAHwHO+eB333qijrM4zMNY8iVlDdr3xNk4R26QJQ0bIPXHsz3DLfGGH1NUlHy5UGU3w23yVaRxpLjgm1Db+lVd1ZxCYQflBV7XE1JC36IWu3Te3ER83MXjX12apWRW8tAAZ77P3EY4fVYrnqyP+JBDsL8UO/8BWS+f3KyAdsNFgvwUu+bva6O9nNEGWUVfxEpio+i+NqVG9OQqn40yBv9jvZ+nuWcID2oCWvqAVpXG6kLMwIm9vET0TFqBr2D9nBVf+ZKPFNdJkrDBS0YPUXv9ydyjC9MiQy+lzb4Q1odz5wTVMSuR4fthTvbDDRADb9/3FPy6GbczsdLeC+uplFZW3jDSo3Nl5Fwt8MfzeMlXkDtwXaXE8F6AbS+KeF5MqLdETmE97b9OtYFYk2yPp2tDqKrXNJYCgx2n9QkemeNPkCVNWPclfDT1vERVawvj1tU/eRUG/fNRw5LvlPISRD7B/BqN0+VVfVXYd2rrE8GHNZBqO5WoXR+29mDg986/GX024b83UDnZU6y/S4hvNaz+eBwP6dA05hlqglvkEszUDqY7uhLZ+cN/ou2+uJ0x+KP2ATUFdHMJWKlYoru/VuvxuCn/hz0HqzDMdPmiPF69ihALjh4+TSVhjHScxm082werP1JdJXJV+uE5zZPv9Fv7tNf+WUazJC19TdoRpJ3pZ9VM0xADl/oDPIbo7Hpm1FW9bksgm9jCSVxUVaqsUbEOI0FqfpTSzQ9NLrVXtOuNE51bhwRV0mymir6 eaORteX7 NVQHVNtKccbW3TMPsGFGDPNZw80C+yFV1R9aCLE3qf5XvlKtxTNoDJ5cx3zICc+Vl9l1zD7VpF+yNOcOele40h/BEHxD8hNjS5cRff3MMPfvJB04s3jjaqDAbS+PRtSH7FoaDTNKMCBH4OI96QvbclRe/wSNaRztIXcQHS+G8JPiIBfPjwMvyf0nV24XPMiamWpcE2GuTb9HM3FZosFbOzp0eN3ShdyrCc4lEOF/vRnxADIC5Iyd6DHayTmf/Ceu230BC3ASK4E1basMWzeF2Pqwv/uwfbIzRJH7U5ERCv+7J9bsXEdH+ZhekGIILo9h288wfJ4Eu/HVhJtH+qhIP9MtS/TT8Os3IKhriHZoCK/DQZCFvXr/pVep7/hHHH24FhRruF9hhDPh9lH/usThPxpNTnVJSwaouJLIy9/Jv5U/peT7jdQuh1+dPnFbTyZZxgatmKablLNNmjSWqkNpqAUK+Rh/fJE2cAgmr8b5VW3V175ltoNn4BZAXR8u5ltVriqzlJgE617X7mtJi5YTy+8l3PbUVxhVgEDrybK9B2x71ErX60KOLJOfdb0PoV/flbCyQs1mtRAxniDWJ0WUyn/iGoQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected by lock_vma_under_rcu(). Current checks are sufficient as long as vma is detached before it is freed. Implement this guarantee by calling vma_ensure_detached() before vma is freed and make vm_area_cachep SLAB_TYPESAFE_BY_RCU. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 -- include/linux/mm_types.h | 10 +++++++--- include/linux/slab.h | 6 ------ kernel/fork.c | 34 ++++++++++---------------------- mm/mmap.c | 8 +++++++- mm/vma.c | 15 +++----------- mm/vma.h | 2 +- tools/testing/vma/vma_internal.h | 7 +------ 8 files changed, 29 insertions(+), 55 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bfd01ae07660..da773302af70 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -258,8 +258,6 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); -/* Use only if VMA has no other users */ -void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 803f718c007c..a720f7383dd8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -544,6 +544,12 @@ static inline void *folio_get_private(struct folio *folio) typedef unsigned long vm_flags_t; +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * A region containing a mapping of a non-memory backed file under NOMMU * conditions. These are held in a global tree and are pinned by the VMAs that @@ -658,9 +664,7 @@ struct vm_area_struct { unsigned long vm_start; unsigned long vm_end; }; -#ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ -#endif + freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */ }; /* diff --git a/include/linux/slab.h b/include/linux/slab.h index 10a971c2bde3..681b685b6c4e 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -234,12 +234,6 @@ enum _slab_flag_bits { #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED #endif -/* - * freeptr_t represents a SLUB freelist pointer, which might be encoded - * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. - */ -typedef struct { unsigned long v; } freeptr_t; - /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/kernel/fork.c b/kernel/fork.c index f1ddfc7b3b48..7affb9245f64 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -471,36 +471,16 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void __vm_area_free(struct vm_area_struct *vma) +void vm_area_free(struct vm_area_struct *vma) { #ifdef CONFIG_PER_VMA_LOCK - /* The vma should be detached while being destroyed. */ - VM_BUG_ON_VMA(!is_vma_detached(vma), vma); + vma_ensure_detached(vma); #endif vma_numab_state_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb(struct rcu_head *head) -{ - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); - - __vm_area_free(vma); -} -#endif - -void vm_area_free(struct vm_area_struct *vma) -{ -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); -#else - __vm_area_free(vma); -#endif -} - static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -3147,6 +3127,11 @@ void __init mm_cache_init(void) void __init proc_caches_init(void) { + struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr), + }; + sighand_cachep = kmem_cache_create("sighand_cache", sizeof(struct sighand_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| @@ -3163,8 +3148,9 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - vm_area_cachep = KMEM_CACHE(vm_area_struct, - SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + vm_area_cachep = kmem_cache_create("vm_area_struct", + sizeof(struct vm_area_struct), &args, + SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| SLAB_ACCOUNT); mmap_init(); nsproxy_cache_init(); diff --git a/mm/mmap.c b/mm/mmap.c index df9154b15ef9..c848f6d645e9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1284,7 +1284,13 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma, /* unreachable = */ true); +#ifdef CONFIG_PER_VMA_LOCK + if (!is_vma_detached(vma)) { + vma_start_write(vma); + vma_mark_detached(vma); + } +#endif + remove_vma(vma); count++; cond_resched(); vma = vma_next(&vmi); diff --git a/mm/vma.c b/mm/vma.c index 0436a7d21e01..1b46b92b2d4d 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -406,23 +406,14 @@ static bool can_vma_merge_right(struct vma_merge_struct *vmg, /* * Close a vm structure and free it. */ -void remove_vma(struct vm_area_struct *vma, bool unreachable) +void remove_vma(struct vm_area_struct *vma) { might_sleep(); vma_close(vma); if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - if (unreachable) { -#ifdef CONFIG_PER_VMA_LOCK - if (!is_vma_detached(vma)) { - vma_start_write(vma); - vma_mark_detached(vma); - } -#endif - __vm_area_free(vma); - } else - vm_area_free(vma); + vm_area_free(vma); } /* @@ -1206,7 +1197,7 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, /* Remove and clean up vmas */ mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - remove_vma(vma, /* unreachable = */ false); + remove_vma(vma); vm_unacct_memory(vms->nr_accounted); validate_mm(mm); diff --git a/mm/vma.h b/mm/vma.h index 24636a2b0acf..3e6c14a748c2 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -170,7 +170,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm, unsigned long start, size_t len, struct list_head *uf, bool unlock); -void remove_vma(struct vm_area_struct *vma, bool unreachable); +void remove_vma(struct vm_area_struct *vma); void unmap_region(struct ma_state *mas, struct vm_area_struct *vma, struct vm_area_struct *prev, struct vm_area_struct *next); diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index ac0a59906fea..3342cad87ece 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -700,14 +700,9 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void __vm_area_free(struct vm_area_struct *vma) -{ - free(vma); -} - static inline void vm_area_free(struct vm_area_struct *vma) { - __vm_area_free(vma); + free(vma); } static inline void lru_add_drain(void) From patchwork Mon Dec 16 19:24:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64A06E77183 for ; Mon, 16 Dec 2024 19:25:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DA8C6B00C8; Mon, 16 Dec 2024 14:25:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1992F6B00C9; Mon, 16 Dec 2024 14:25:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1D166B00CA; Mon, 16 Dec 2024 14:24:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CCD016B00C8 for ; Mon, 16 Dec 2024 14:24:59 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 556701C7E3C for ; Mon, 16 Dec 2024 19:24:59 +0000 (UTC) X-FDA: 82901797818.17.2AE60BB Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 0635DA000D for ; Mon, 16 Dec 2024 19:24:35 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3HyDeN3m; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3h35gZwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3h35gZwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4HhxNlm5l3oc+a05Upn5SwGTABo4qTWEjrE4UxpsHII=; b=LA9OuAFpCd2xYL5Jf71qHF9h1zQvXa+teBXIWhV7kUUCgXqWHJvPH57H7WzB4t0+p7IEbn Q7Ha2kNxTTm/5mJ0BsKpIOQR2cc8eZ5BVU0usOFUcVdG5SefUeWPG6KV+kZyi1Mj/jd+60 OFXa2U0pY3T2tu+GjlhPdvvAlkwc7QA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377083; a=rsa-sha256; cv=none; b=lgvVcXfTB9XUcTsjcQl60R6KWUIc1Ar+38wFco+PmXsDBUDprjjlCS4m6v7dUDwqTsqBME TgdXqQsdbElqBWRFy8CJMNQ1yprplGvqBvLlWuQF7TJezYtRdBBRY0v7aO0pdhr9JsVFfC F2/GWQN+bwO//Kob27Qr6XsZxuItrj8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3HyDeN3m; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3h35gZwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3h35gZwYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-725e2413114so3422810b3a.1 for ; Mon, 16 Dec 2024 11:24:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377096; x=1734981896; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4HhxNlm5l3oc+a05Upn5SwGTABo4qTWEjrE4UxpsHII=; b=3HyDeN3mg0bDyGoONICe188+zK3XeFdf14rf8gquL7L7tL4qFia3YHNgfQ/ZYLU/BV qO0+K2MudsleykrLs9f6QgvtsJYOiWcJKe3u6FtD62ipgEppY3ByPXHKuY6ZSfBuI0/1 JYPBy5lJ3INXL/c21xEoDZayunbSgDMU4RnuStJiLN+t5ewMMFhBz4vK4VNbqty+HwXl zhi2N7REuwFpsMnShwtLLxnSwZqqE21/tBwcLJ71QgwExtTs04pDJtDGX8uU+ldpuYRs lPH7Zov0onuKzW5pwh3Ux/Xh9Sepo8BYc4Us4pH/xsoPCgedWXYkwSNjbo6IGjFt6UFb Ws7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377096; x=1734981896; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4HhxNlm5l3oc+a05Upn5SwGTABo4qTWEjrE4UxpsHII=; b=NhhNLVJFnvFbddEkrhoYACmK3KYATEzIfefw2B5VP/NPCh48F472EWlAEms3SDTaO0 kWoZraGG1F+o7kEFSg8wqvUPm7OvJY66KtWvetzjzimDMM4ecXk7++/2c2428xS+0uRs H4hzV0e1GPF1UaIw+tRifye0rIc2MhJMVGDnk5/KqIFfUoxX/MPVoMKm2h6UK8kRVcmt 4x13cPWBGfiuxBJEWCLi3NqsOmHxotblcKoq7C6ynmTF9thFkW0CQNA7XX2XmFQ3xuLg d3C1arP6BiQs6b7wVnDEctTBhQM6fpzQk0Ye6RZ/nW41NtMIgryDUQJyDNieRLVQQhjE 1reg== X-Forwarded-Encrypted: i=1; AJvYcCVjaBkrSM3ufXfW+GHWN7agyjuLwN0HAXSg5+l+T9uc8YlVaOgS340K7HF/8myTYEHWjreXM1OBKA==@kvack.org X-Gm-Message-State: AOJu0YxX+fbYVH40N20hH5dY/Ca5JC1xMOIThyz2b6OhcXwJ97uHEY3i FiCv9AKXm7QQt+Ljfpzla9uMvT3bx9XmBuknVIKpdf9HlCnhSIiG4kchsOqy3tfbutU3y6TxubR EuQ== X-Google-Smtp-Source: AGHT+IFCRcSRmnDxCEXNS618C+s8AbCbrJ3sqm2+B/Vq6ZMmO1in0EJV2noWuvjJbPv4qSgauLdf86FSJXo= X-Received: from pfbfb35.prod.google.com ([2002:a05:6a00:2da3:b0:728:b8e3:9934]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4b54:b0:729:49a:2db0 with SMTP id d2e1a72fcca58-7290c273bfemr20412405b3a.25.1734377095966; Mon, 16 Dec 2024 11:24:55 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:19 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-17-surenb@google.com> Subject: [PATCH v6 16/16] docs/mm: document latest changes to vm_lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: qi8whxntz7w79kgy44fu8j9gcy55brhk X-Rspamd-Queue-Id: 0635DA000D X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734377075-316189 X-HE-Meta: U2FsdGVkX18gRgkC+e+NYxEG+7l5oQn9CX5a1kU+ZjVMwNAMgvJ4no3oNIMKp7k8DXNNBHtKtfDk4bk1is3XrK8JHW8fOnF7d0iHTHEplYc0tjFTY6KU01AXWR57rt6y/T8SEG+J+oPyQH2c2kS6gKblZf2VttZkrH7xc/kzav4Lo1FKKYSuOzAccHrOfrOEHounIz5157IavMjNvLnkJXBQx+TZUdmIOPr61Y6Av/RTbAU3/eTr9UDcU8KR4Tyxt0r7wUJMF3ozw+ZXrwANK9+hxogfAYkFYu+RsR7IlUHpocvB6skHvuph4UkbDFWXsp94ChaiN2CmQ5pHV+ZCkj2B89Sl5flQX9pNF7yeEtvW+2aBZNV/34iStsSnb3TwpFZzxFwS+EJ7DP/7MA9Fx5PVkrSomBSgMqr8PfKDUnIKyspOXS/jRY7/gJai8iPvcdZYE5ZJ1BXY5HpGkTl6aUvq78BRaSzGb0W2i7v4FKl7T3FMuGzn3h+KhMBYCEnGDhshvFQbFspiZeirlAROVvNGMjxLCpO0k5kfNkLE6NbR06WHpYYj1/jQWBxwHbbw2i61XbCxKU3hto2cdRy0lEytXRFNVIm8B58msEGKGf/+biAwTp/lbkourAXZOgMltMe2/vfOPHd+crGdBCPKNZkfPfDgWjm/11FUxIthlexWYj0zhtVFGZ8pYJn8Fc11HDjEhT4WCooXt9zjLhmIHJeOold+ALGBRufvNTxmIePO/nUypfPMg91bQsBdmduHXeImjiYA8VgRQkIN8ji53hl49wLIqu5zW0XPDVKWJyQT8fxgMJjm/w23YkGWHSr1AwNs30GW13r+gtZd8SaY1oRYWotKACIJzAWcMZF9YTSYgXuVNgTH0GjnrsvkrxBrpzOyZAQhMtWFavteHWWQobdLHkNU7ebHL7qwQsUUM/Dsrg9Xb3Yywdwe28ZW/RLWQ1Zi25GrpD2KJ9Ap59K kXNaxbM2 gAZg1fJSdDGDThi4uAkCh+GxPhwHME0MQ7iG53Jyb7IkcFtiB/a2u5b5OkJCKuitoZlL+ZVVVTKV4bj5Kp34WGZzoeLg7mTlQyIhB07h6SAHWdYXA6YpLZ374cS0lH8pzjlAzkD4uuG2Qhp8GexkW86EFO9b7l0lsZW7odrRNmSxOOguz9J5QguslIYKfGb1mTnKAXVdHJyrS5x/gI6P43WX5GulAJL4XDo0iOlS1ChiRlD6wP84hc7JULeUt+yOxJuJG/pg/zoRl5GhL/luyw5BmflDOA0f5FAu6U47iq11CY8j0lN94PJKoyyAPLkhO2PtZjU8mrgf4EZEsTsePiYDgFDsHBsXtou0eb0Al3TUIHnaS9NC4wdbEin3QzW8k8MbDtsn5eNztz/FEZsfwzGzKw1s000ni2j3OqEWZV2mP9MQl++i3Sgojvduy0Ijrdfla5JUoo0AC5ROFDaPowxY6Zw1tHc5betAut/PQoNZrOUIlG6+lM5+AsySxU/6fls6NQSepBpwim02Xa86+ThFMxsvmus/eUWEMCaxGpC+UfAQjyPQufzm2xG8v7Os7mDs+0nf3j8uuM8BojpV17WDoUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.127449, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the documentation to reflect that vm_lock is integrated into vma and replaced with vm_refcnt. Document newly introduced vma_start_read_locked{_nested} functions. Signed-off-by: Suren Baghdasaryan --- Documentation/mm/process_addrs.rst | 44 ++++++++++++++++++------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/Documentation/mm/process_addrs.rst b/Documentation/mm/process_addrs.rst index 81417fa2ed20..f573de936b5d 100644 --- a/Documentation/mm/process_addrs.rst +++ b/Documentation/mm/process_addrs.rst @@ -716,9 +716,14 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`, before releasing the RCU lock via :c:func:`!rcu_read_unlock`. -VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for -their duration and the caller of :c:func:`!lock_vma_under_rcu` must release it -via :c:func:`!vma_end_read`. +In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked` +and :c:func:`!vma_start_read_locked_nested` can be used. These functions do not +fail due to lock contention but the caller should still check their return values +in case they fail for other reasons. + +VMA read locks increment :c:member:`!vma.vm_refcnt` reference counter for their +duration and the caller of :c:func:`!lock_vma_under_rcu` must drop it via +:c:func:`!vma_end_read`. VMA **write** locks are acquired via :c:func:`!vma_start_write` in instances where a VMA is about to be modified, unlike :c:func:`!vma_start_read` the lock is always @@ -726,9 +731,9 @@ acquired. An mmap write lock **must** be held for the duration of the VMA write lock, releasing or downgrading the mmap write lock also releases the VMA write lock so there is no :c:func:`!vma_end_write` function. -Note that a semaphore write lock is not held across a VMA lock. Rather, a -sequence number is used for serialisation, and the write semaphore is only -acquired at the point of write lock to update this. +Note that when write-locking a VMA lock, the :c:member:`!vma.vm_refcnt` is temporarily +modified so that readers can detect the presense of a writer. The reference counter is +restored once the vma sequence number used for serialisation is updated. This ensures the semantics we require - VMA write locks provide exclusive write access to the VMA. @@ -738,7 +743,7 @@ Implementation details The VMA lock mechanism is designed to be a lightweight means of avoiding the use of the heavily contended mmap lock. It is implemented using a combination of a -read/write semaphore and sequence numbers belonging to the containing +reference counter and sequence numbers belonging to the containing :c:struct:`!struct mm_struct` and the VMA. Read locks are acquired via :c:func:`!vma_start_read`, which is an optimistic @@ -779,28 +784,31 @@ release of any VMA locks on its release makes sense, as you would never want to keep VMAs locked across entirely separate write operations. It also maintains correct lock ordering. -Each time a VMA read lock is acquired, we acquire a read lock on the -:c:member:`!vma->vm_lock` read/write semaphore and hold it, while checking that -the sequence count of the VMA does not match that of the mm. +Each time a VMA read lock is acquired, we increment :c:member:`!vma.vm_refcnt` +reference counter and check that the sequence count of the VMA does not match +that of the mm. -If it does, the read lock fails. If it does not, we hold the lock, excluding -writers, but permitting other readers, who will also obtain this lock under RCU. +If it does, the read lock fails and :c:member:`!vma.vm_refcnt` is dropped. +If it does not, we keep the reference counter raised, excluding writers, but +permitting other readers, who can also obtain this lock under RCU. Importantly, maple tree operations performed in :c:func:`!lock_vma_under_rcu` are also RCU safe, so the whole read lock operation is guaranteed to function correctly. -On the write side, we acquire a write lock on the :c:member:`!vma->vm_lock` -read/write semaphore, before setting the VMA's sequence number under this lock, -also simultaneously holding the mmap write lock. +On the write side, we set a bit in :c:member:`!vma.vm_refcnt` which can't be +modified by readers and wait for all readers to drop their reference count. +Once there are no readers, VMA's sequence number is set to match that of the +mm. During this entire operation mmap write lock is held. This way, if any read locks are in effect, :c:func:`!vma_start_write` will sleep until these are finished and mutual exclusion is achieved. -After setting the VMA's sequence number, the lock is released, avoiding -complexity with a long-term held write lock. +After setting the VMA's sequence number, the bit in :c:member:`!vma.vm_refcnt` +indicating a writer is cleared. From this point on, VMA's sequence number will +indicate VMA's write-locked state until mmap write lock is dropped or downgraded. -This clever combination of a read/write semaphore and sequence count allows for +This clever combination of a reference counter and sequence count allows for fast RCU-based per-VMA lock acquisition (especially on page fault, though utilised elsewhere) with minimal complexity around lock ordering.