From patchwork Fri Dec 6 22:51:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13897897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D7C2E7717B for ; Fri, 6 Dec 2024 22:52:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C53F16B0307; Fri, 6 Dec 2024 17:52:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C05706B0309; Fri, 6 Dec 2024 17:52:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA6F76B030A; Fri, 6 Dec 2024 17:52:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8D4846B0307 for ; Fri, 6 Dec 2024 17:52:12 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3B22AA0875 for ; Fri, 6 Dec 2024 22:52:12 +0000 (UTC) X-FDA: 82866033558.23.2962683 Received: from mail-oa1-f73.google.com (mail-oa1-f73.google.com [209.85.160.73]) by imf20.hostedemail.com (Postfix) with ESMTP id A53071C0003 for ; Fri, 6 Dec 2024 22:51:51 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d0HOi8Ik; spf=pass (imf20.hostedemail.com: domain of 3GYBTZwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.160.73 as permitted sender) smtp.mailfrom=3GYBTZwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733525512; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2XVlPHpEZtu9xIw4Qap8alaO3+SGpEYwVQX0iy/v7kQ=; b=nV4L6DZcNUNkVEFUl7uzGhq6HIB8YXNAF3vEtCdQawMNW+7A0fwTDX7Ee2bowQauhVjipN a3G8dHbRj0l5SzajYPwHjW+HOo34xviHqZ/RAXTM8K4VUf7ikvYTgdpt1vGJ9JKfcE1auH rzOuuSuklVEqOJ0WcsNWP+qdCcYhvQs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733525512; a=rsa-sha256; cv=none; b=77QPRSwWRetMHVv99OJCE0t2QJAHLxbWBYOvjruz13+R7Gg23KRLWJX+ZjwFSSkqGMFujD n64kGnoYD3VvLYtqqD/7dS4xdICuJxwKB4iCoWe8cGfX0WyGvTvtWmlSTD6waUTn2JFTE/ R0C/dyRVkS9bXL0q7ob1L4ODdZ94nWg= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d0HOi8Ik; spf=pass (imf20.hostedemail.com: domain of 3GYBTZwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.160.73 as permitted sender) smtp.mailfrom=3GYBTZwYKCKgacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-oa1-f73.google.com with SMTP id 586e51a60fabf-2975e172b3bso2318112fac.1 for ; Fri, 06 Dec 2024 14:52:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733525529; x=1734130329; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2XVlPHpEZtu9xIw4Qap8alaO3+SGpEYwVQX0iy/v7kQ=; b=d0HOi8IkjUGZPQy6Qk3yXbY2AtET9l0uQBbkHXFS3cWkq4JfYec5kQOjrIvFzcTKsR Yic1VMGmJuLVxm7uPtrm9WsMSSbRr1OZrB7qs1mcdl9a+EOlTqXgD6HivlMQEXz/L5S1 2hcEgWagi8huo/yV0tPfF2UPp72Lj0XDX3a2/u/6RKxNLJVZJUsSiuXxamRza3vJSTwW pMbirYIn4fqEc0yH/lsTTz24LOC2w1NAr8laCFysnINNpokLxBo5K+fx/G0vzgfB106Q Wff9AUCOWDj8dBklXaVFDAj5KWBtZsTeExsNtJGnvi3tBcklqIxFuZnJrGoepPl4o8al lA2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525529; x=1734130329; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2XVlPHpEZtu9xIw4Qap8alaO3+SGpEYwVQX0iy/v7kQ=; b=vQCo24uhtj4wKFpvgSiMCXbkepUWTVgOKzUDKddEgxMmgYMwH818WKym0n3VJBg5ly Qg/+14Cscg1N3u2WAvcoN9RwTyK+kxrj94sdy9OiMjSF0Qp8nQH2u/Etc3dGEnMPk6Mv r9aFnGSj/JVGbdINiYQCwmyJwE+XwYAoYj6OWiR0hHNLK/nFbTHOY3tdIAUU91daCdt2 sbH+n1cdXklRw5xXVkZdkGvp/sMdR9ZaZ2rmFbEOL3nec7cFKjZy9sug3iQRo8yfaeMz 0SB5g9mBEL5OvEDuWunEZJqWvar14CYeGjSRPHP3gsgJxBfDHYHtrJoyut+w3FbAjRRt /OKw== X-Forwarded-Encrypted: i=1; AJvYcCWtyX5CyCpraCKJJzE1IgyMWbIDgALSvWQb5zLhI2cCo51+kEu+Tppty1X8LDxfLyHqGttc3SKcWQ==@kvack.org X-Gm-Message-State: AOJu0YxOh2zdXX9K9tiUpXxrCUOpyFVl4mxxjqq7IRK5GaHqMz/MP9+x pO2nuXabkkulbQv4fQbAiIwZJzjor/4SEKP3E+i3oPGIqaBZ7HTRRiBJIo8Q7OKWYwCYCAMDPtO vDw== X-Google-Smtp-Source: AGHT+IE3XhAvDScGGmsUTdbbULESWT7yEme/wNWVflifCM9618rjWWcRcIta7LlU8v32EA39xFZwUHCJlPc= X-Received: from oacpd2.prod.google.com ([2002:a05:6871:7a82:b0:297:2483:5994]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6808:e8c:b0:3ea:5161:f71 with SMTP id 5614622812f47-3eb19ce52a7mr4865569b6e.20.1733525529219; Fri, 06 Dec 2024 14:52:09 -0800 (PST) Date: Fri, 6 Dec 2024 14:51:58 -0800 In-Reply-To: <20241206225204.4008261-1-surenb@google.com> Mime-Version: 1.0 References: <20241206225204.4008261-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206225204.4008261-2-surenb@google.com> Subject: [PATCH v5 1/6] mm: introduce vma_start_read_locked{_nested} helpers From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: A53071C0003 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: sex5jju6oxzfko6d8psasoujp9gjpe1u X-HE-Tag: 1733525511-77854 X-HE-Meta: U2FsdGVkX18JONAntbdCLqvabx/hFJq7Fm8KRawW7ZMQiaqcDMfcQbS9vG1qhiWqWSH1yLoFIdgLF+UbBQxubJWWe74mFLAKQIVRtfbKoF6JjsoALjji0pqAo4QXDMSkkEH4a7IayKR3Mht7XpX7yv4K26wDTEAOd/zSXr5yj0uGFpRfCdQ5Dnw7H6K7oyQbEfCvCtZ4SRMPqM9uoJp9uWnLP8UKgEtag9uxBmSrbOfoghJPe+N8flE6aEgMWJx8m8ZigMB+m2PRFuNp5UgTnHjZ8u0XxvgEUjNeORWlCYSNws2PqMEaC+NMDo+1QrTRH05birz94WMJAEAK6GUC6eivZEHlPqooo1gEXltkLAycvnfympt2r/kagP1r2EyNMgmLENtrOrZ8dZphC2+7DZ4o87eplEDQeGtJc2eBOCf05905qr4FJgaimBNpDj7FFzQz5iH1LkDd0UfWSm7G5apfK/cP1/NvTfSPXGfs9cVlmYpe8Innyle/GzbCV2RHUouTe3VbyZ2eDRx0cY5rccX02onAfIlqTHKo4uPgu03IzPQZUUMs6WZEssfebQcyX19B4KXFvAq2sGBITIDywINJHHDMjog2EIESoroQXpT4mmjRZ/T7zPBJ1sMS86Ia3Y8P7gGTMIWyH3NmDq6GDUyVZnOmWHedDcCoJAzBtmH/ETOZ8MVoLI/w9bKx5qpTHvdXpT6clsfdwHQGI0cJiSqckInrXZ0zXjfMGWjL1+KRvduwN+Bh5R0vlVJLceoAljJUGl/XhfC3fhT0lS60Jbi0AArSD2I4PMxu6Ku1epb3UPA7rDTT3DyBCZTrtT/MqhkU5IYPxgTfsEszc5IcNnCLWZaUxtgjnXHqNqIWhdGImTRTQCMJXQgKhRafDRAeqTFkpvczmhIAlFbzV+YpHbvEPxnW+JWZWe4P1fF2/gDpsgCnqQxYMZkD1AIXzFAKa1OCmN7qgOHbveoq3jm aAZ17BHM wZJIXhbHYMefL+9WQ9CP569PEV2/NwCWqvs7UDtLaye6epQYRs8pp2wDmqnfwRNJORTbou8CKB/YywCX17pqluSD1GV3g2POPuG4XDkxbuBjjOU7nV15lx7d4P3XyZUY+9p3tIrq45pSbq3gNObY6ucUCDMFAUrcEo6Ii1MbVy9wHiTBcMZxRJHL50MKL7h7G7Ke+qZ7Cl4o1pHqo7K1o1uF12dgrb+CYNRnSMoSjP+HgIrKtkyU48nND/yvlWWqJyeJEZTUyQy0TFOD/TFNzgmo6aIcqPq7rIRmXRhBVO/H/dt9bmFZpWNhpK8/q920fVR6IfRcFNBySGzc/riopeKbK1gthxuOB0hx3O7itoprDkn48Q7nuUfirHdNFKiEAUi0kce1s8bKzF4E4usDOLfiAMmVFLGwgBNZF6qZHvbGPgmKKWIHGeWGqK7zDbFG49kLBJ8aQLvRt/0eefUypSf5VztEMm+qMbuDTN/WlPr039RPFi598aoqzOAUkBFT/CyR67eBGJ/ZLHkFuzevaMqdteEJwz7GMpPlBgPyBij1PmJpQuwZWmQUFcfS95s9P8adZHUD8KbIDtphyO7JI8aTDBvgWtkQNEZfMSXwc8qFczp3PtfLWz+ij1WJxGMdV4AEFJvo3ftEXhsjOb5TMmi2Y5fNG8LInIYG1ebqxhyVe4Wo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions which can be used to read-lock a VMA when holding mmap_lock for read. Replace direct accesses to vma->vm_lock with these new helpers. Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes Reviewed-by: Davidlohr Bueso Reviewed-by: Shakeel Butt Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/userfaultfd.c | 22 +++++----------------- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 187e42339d8e..c4a001972223 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -734,6 +734,30 @@ static inline bool vma_start_read(struct vm_area_struct *vma) return true; } +/* + * Use only while holding mmap read lock which guarantees that locking will not + * fail (nobody can concurrently write-lock the vma). vma_start_read() should + * not be used in such cases because it might fail due to mm_lock_seq overflow. + * This functionality is used to obtain vma read lock and drop the mmap read lock. + */ +static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) +{ + mmap_assert_locked(vma->vm_mm); + down_read_nested(&vma->vm_lock->lock, subclass); +} + +/* + * Use only while holding mmap read lock which guarantees that locking will not + * fail (nobody can concurrently write-lock the vma). vma_start_read() should + * not be used in such cases because it might fail due to mm_lock_seq overflow. + * This functionality is used to obtain vma read lock and drop the mmap read lock. + */ +static inline void vma_start_read_locked(struct vm_area_struct *vma) +{ + mmap_assert_locked(vma->vm_mm); + down_read(&vma->vm_lock->lock); +} + static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 8e16dc290ddf..bc9a66ec6a6e 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -84,16 +84,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, mmap_read_lock(mm); vma = find_vma_and_prepare_anon(mm, address); - if (!IS_ERR(vma)) { - /* - * We cannot use vma_start_read() as it may fail due to - * false locked (see comment in vma_start_read()). We - * can avoid that by directly locking vm_lock under - * mmap_lock, which guarantees that nobody can lock the - * vma for write (vma_start_write()) under us. - */ - down_read(&vma->vm_lock->lock); - } + if (!IS_ERR(vma)) + vma_start_read_locked(vma); mmap_read_unlock(mm); return vma; @@ -1491,14 +1483,10 @@ static int uffd_move_lock(struct mm_struct *mm, mmap_read_lock(mm); err = find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap); if (!err) { - /* - * See comment in uffd_lock_vma() as to why not using - * vma_start_read() here. - */ - down_read(&(*dst_vmap)->vm_lock->lock); + vma_start_read_locked(*dst_vmap); if (*dst_vmap != *src_vmap) - down_read_nested(&(*src_vmap)->vm_lock->lock, - SINGLE_DEPTH_NESTING); + vma_start_read_locked_nested(*src_vmap, + SINGLE_DEPTH_NESTING); } mmap_read_unlock(mm); return err; From patchwork Fri Dec 6 22:51:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13897898 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B61FE77179 for ; Fri, 6 Dec 2024 22:52:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C561D6B0309; Fri, 6 Dec 2024 17:52:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C06E16B030B; Fri, 6 Dec 2024 17:52:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0DB36B030C; Fri, 6 Dec 2024 17:52:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 77E446B0309 for ; Fri, 6 Dec 2024 17:52:14 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3C4A0C0B46 for ; Fri, 6 Dec 2024 22:52:14 +0000 (UTC) X-FDA: 82866033642.30.2EF52A6 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf19.hostedemail.com (Postfix) with ESMTP id 8FC5E1A0004 for ; Fri, 6 Dec 2024 22:51:53 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vkDrxlbO; spf=pass (imf19.hostedemail.com: domain of 3G4BTZwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3G4BTZwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733525520; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rT8Mwg3YVngq88fQDqUxdC7bmsa9/fkDJKquyXN45js=; b=OtmKboiLpk4s5lA/bDegYuSPkahL8qeaCXMIzGJFDE0tnzau252MzfadiAWHullZB/L7Hi Pi5YQ7W8surCQevkB6tTSq5GyDw3FDu3VSaTZEIipP06jVA9T512KJk+MZfKTF3b/BOqnf tNOi0TYgDRkhgAjsxwgSPE7/ckpqCDQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733525520; a=rsa-sha256; cv=none; b=mVte9bYH4jmKrOyWDLUzSLjJvPRbcqSLVzTxEDNDYVYH0DdAPa/Mvm85KwjMpJ2VU03QYW uJxLfgphJzr4OhRQPO3RB0FR9YxwflmK/33aD/NRTQvHAY0giodO/mD5Dj4YWgXRbcYnSi fidNWF51cuBfJlLE9b2sjcjbBjznOqQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vkDrxlbO; spf=pass (imf19.hostedemail.com: domain of 3G4BTZwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3G4BTZwYKCKocebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ee86953aeaso2515881a91.2 for ; Fri, 06 Dec 2024 14:52:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733525531; x=1734130331; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rT8Mwg3YVngq88fQDqUxdC7bmsa9/fkDJKquyXN45js=; b=vkDrxlbO6Bt/Xf0MGKer7K5Mt7dLmHG79aqyCX7H1bRx2d8/h6d6sZMDaT1XULdOZu 81EcZ6uSmULy9JCNDccW5Pp/BbDLcE4Hy7wBz86UCho16uv1kEcT7Im6j8Fd6X4bpP0O buhfxvrcNZXIIPiQsvf1FfjvfxgIf0pRmt56KQj1cyi8H9ahiRWIz/QEq7tiLSUZ1gsL Be2s68Zn0wdCC1EviKPSyEfj9yOS9GOn4ACPpyBkqRj5d/psVqTpaxu04fgkCPyI6nAH 3RgXKfTPWuNxh3fWaI3SdQeJ5cbKI0g40RisZ/EuHTNAhZRx8aNe/Fa/hoQJfhODGlcA mwkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525531; x=1734130331; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rT8Mwg3YVngq88fQDqUxdC7bmsa9/fkDJKquyXN45js=; b=H2YSf3ZPs2a/jmcw9TlMtb3K54AYRaFULzXaaQolbBz8h5EuJDjI+D/kVDka3THAh0 Ps9T4C5G8+EMR8Bnu75zKKfdkMao6M78g2XLNhOtyLcmNPS/YbW4Ekcwcawupaq5QA5m EqoszDprukfbMUNOT5M0iTmIyy7d5EMfIKHiFTs4Rkg7BcPavofbxbK9+5q7pxZ96HOR 24e/M0WMjZ5mA2hwxJmMLMX0fZRMzmEt1Ubu4WjpUQqm3Ubdeekk3TAFm/A/A7/8L0z1 a4KLoHD1O3pf3ygByNBPMQXJh5sc0Tm7eicRpKP9HI5aaFeoIuRvxPW0Z92C4DWpo+m9 vJbQ== X-Forwarded-Encrypted: i=1; AJvYcCWolXuL+A2gYLJZ+qS0CsI9PY2/l14K0c33sdkoqLGjjlCz1hxvQ/5MaNnBxgC1698bmbHm7uOwpw==@kvack.org X-Gm-Message-State: AOJu0YyQmcqHrUvDs2QSorov+Ra1VmzJlvf5Att/23jT7GJj/WkWbsKM A+PmUopbNm8msQ0D58f6/JlFZnoqigOr2DAEGxArW4earpWJmRVhA2cMjHXSN4TP+iY2RxFbuL7 WIw== X-Google-Smtp-Source: AGHT+IFlxdPNqWtYnG8mmAY9quNf9jbf25IzCiv9883/eI2U6H6bU+wM6Ogc0koGE2fA7VPMtISD4bXqCdc= X-Received: from pjl13.prod.google.com ([2002:a17:90b:2f8d:b0:2e0:aba3:662a]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:ca97:b0:2ee:6d08:7936 with SMTP id 98e67ed59e1d1-2ef6a6c10cemr7355055a91.20.1733525531141; Fri, 06 Dec 2024 14:52:11 -0800 (PST) Date: Fri, 6 Dec 2024 14:51:59 -0800 In-Reply-To: <20241206225204.4008261-1-surenb@google.com> Mime-Version: 1.0 References: <20241206225204.4008261-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206225204.4008261-3-surenb@google.com> Subject: [PATCH v5 2/6] mm: move per-vma lock into vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8FC5E1A0004 X-Stat-Signature: fuyk5qmm33hcihqk8hshty1i9fb1akwp X-Rspam-User: X-HE-Tag: 1733525513-705356 X-HE-Meta: U2FsdGVkX18fmnJ1a6s577GQ8KCX8clXtAYZYgM2UYT+u9mMYh2gnZs13bQi76m3ukGQGvSYaZCcgSPohsNikmpScYr/84DblDrbmmQs2pcgrHqMTLK2k6Pp6w+WSVj6FldopD1w3+1x+xkXzyzjCXPJAV7vcR6bmFjpBXeYXo1n+5hlAT0GsG/2Ni7g3jQXYE/SE5T/HeNq5YNth0NnlpGZQrr1YybkRUUGIVAOorK64kQ5PR1qxs9n/7Diag9oZ5IfqvwkXc6YX3z0+sE+PFcqwfNT8uzSE6DgmSHMd+PAdAmcxjRiMU/BTMCi5i3JHFtG967N3UkzhUf0Mo9xQYPLIsZbeipakLUYuSQRxJyk2NfELP/GRjziCW9rakTdsEstBnQBNhE+JXceu+0taeRaxpWYVOSpd7jTZYMLqL+hPtKiPiXEX2nSRnQ0Wae7rDPyyLsK2BFj0O2oE+szjqn+kMtclaBLTRc880mCilQpwUkF32WocFejMoupZCJ7+veP3O7Y+OJ0HL7uGwfnDRqJ3eVNCoL4q2hWb20ejV6MP3HjfYw2702xcQ6Q/SrltSjoE9LePPGhAxmuIhSOKCLDyJHVX+eRbwrg1TMdjf9RoMO3aAvbWSJV0QmyvH5CqkQJMaGq3/SMv0YW/Z2FvXzg6pk1PrUTJLoJmx1qNqxU7oojfTs6isOWDWnRam6I5fvvhFXbcgBWimxDYreBc5kVUs12WhxxOWUG53Vq0mcNtPH7QX4OK/IRFd74RUAyXdWhtYxe8bUpMiwqvWlR/AtFXELRLYDmYIcsv3zQ3xr/7Eb+AQ2dsrECysRkTP0s4qOfjZY4j+L1fULrAByvfsZTM8NiGgY9pCAJYL3shzYGXuMUiQpc58n4NCd7VJ6pN1CIcPbo8GqE3rxkpJ4aPGq+oxV+m9LDLFAvBrNyVCAwZwxNaMFu3b4MvVSA+u+kIfC8Td1m4cM8+B7UXUB 7UtvBWir a0fBBXeeuev6+kLj/rOh2e+jr13rh6Sccmpn2XaiRHqnJQFxaU6A5aYnSEeLcp57TfCJfbjz5svMB3z8uuO+d58jWRsPpSpU5f5lkztjumVoaZTWObWJCgbSISeAl0S22dEPZ5xdXPDSKSqD/HydET7VKrGpz0b1mjgJ2OhtAM15KH4srnwLEQ5nRRvL/a66dM8x14uAJxE0PT+UAnsvp+VZxVnD3O+EooU4bDzxXVxD5wCi9DsrrWypIEW/wTjyLrh4zgsbOW3Ejknf0ylThyWWMvRPhadCONztC43vnH6GfNaXh+FoOB9U0D7wZ4jTU+iV2yIyiVN0ss0BNTEUCyc1cVaFYTLbEz4/4phdHiA+/JHUZ1YX0QcVFfkGCDAtItgsxsW2ZI+IM0MP3dgADW6cbropzYMBb4Lwstv3ZpWcecZTFwDu0zvgRma5twnb1eknAJF7UD0QzgVLHLM1QfFhKV7c3A/OqfVmiltEPFMKFEyG7CmkOnCBdqDrf7mTsHpfayR+3Lwtohxi1FNXP6dkNb25moJlDJnZG3+x1x3AC5+BG/3saELKAUBWVgdh6thYV2ywr/nX8xyJNcdDq76Qq2sd0EYWD6vErIe8dNontjcIMep66nwP6ymfplaGu9IBZfPEP/3ml0QaIAzYNmCEp6QCobfY11WVg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. Move vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned as well. With kernel compiled using defconfig, this causes VMA memory consumption to grow from 160 (vm_area_struct) + 40 (vm_lock) bytes to 256 bytes: slabinfo before: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 160 51 2 : ... slabinfo after moving vm_lock: ... : ... vm_area_struct ... 256 32 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 pages, which is 5.5MB per 100000 VMAs. Note that the size of this structure is dependent on the kernel configuration and typically the original size is higher than 160 bytes. Therefore these calculations are close to the worst case scenario. A more realistic vm_area_struct usage before this change is: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 176 46 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 54 to 64 pages, which is 3.9MB per 100000 VMAs. This memory consumption growth can be addressed later by optimizing the vm_lock. [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes Reviewed-by: Shakeel Butt Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 28 ++++++++++-------- include/linux/mm_types.h | 6 ++-- kernel/fork.c | 49 ++++---------------------------- tools/testing/vma/vma_internal.h | 33 +++++---------------- 4 files changed, 32 insertions(+), 84 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c4a001972223..ee71a504ef88 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -696,6 +696,12 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {} #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_PER_VMA_LOCK +static inline void vma_lock_init(struct vm_area_struct *vma) +{ + init_rwsem(&vma->vm_lock.lock); + vma->vm_lock_seq = UINT_MAX; +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -713,7 +719,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) return false; - if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) return false; /* @@ -728,7 +734,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); return false; } return true; @@ -743,7 +749,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { mmap_assert_locked(vma->vm_mm); - down_read_nested(&vma->vm_lock->lock, subclass); + down_read_nested(&vma->vm_lock.lock, subclass); } /* @@ -755,13 +761,13 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int static inline void vma_start_read_locked(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); - down_read(&vma->vm_lock->lock); + down_read(&vma->vm_lock.lock); } static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); rcu_read_unlock(); } @@ -790,7 +796,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (__is_vma_write_locked(vma, &mm_lock_seq)) return; - down_write(&vma->vm_lock->lock); + down_write(&vma->vm_lock.lock); /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -798,7 +804,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock->lock); + up_write(&vma->vm_lock.lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -810,7 +816,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_locked(struct vm_area_struct *vma) { - if (!rwsem_is_locked(&vma->vm_lock->lock)) + if (!rwsem_is_locked(&vma->vm_lock.lock)) vma_assert_write_locked(vma); } @@ -843,6 +849,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ +static inline void vma_lock_init(struct vm_area_struct *vma) {} static inline bool vma_start_read(struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} @@ -877,10 +884,6 @@ static inline void assert_fault_locked(struct vm_fault *vmf) extern const struct vm_operations_struct vma_dummy_vm_ops; -/* - * WARNING: vma_init does not initialize vma->vm_lock. - * Use vm_area_alloc()/vm_area_free() if vma needs locking. - */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { memset(vma, 0, sizeof(*vma)); @@ -889,6 +892,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); vma_numab_state_init(vma); + vma_lock_init(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 706b3c926a08..be3551654325 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -700,8 +700,6 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - /* Unstable RCU readers are allowed to read this. */ - struct vma_lock *vm_lock; #endif /* @@ -754,6 +752,10 @@ struct vm_area_struct { struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + /* Unstable RCU readers are allowed to read this. */ + struct vma_lock vm_lock ____cacheline_aligned_in_smp; +#endif } __randomize_layout; #ifdef CONFIG_NUMA diff --git a/kernel/fork.c b/kernel/fork.c index 37055b4c30fb..21660a9ad97a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -436,35 +436,6 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; -#ifdef CONFIG_PER_VMA_LOCK - -/* SLAB cache for vm_area_struct.lock */ -static struct kmem_cache *vma_lock_cachep; - -static bool vma_lock_alloc(struct vm_area_struct *vma) -{ - vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = UINT_MAX; - - return true; -} - -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - kmem_cache_free(vma_lock_cachep, vma->vm_lock); -} - -#else /* CONFIG_PER_VMA_LOCK */ - -static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } -static inline void vma_lock_free(struct vm_area_struct *vma) {} - -#endif /* CONFIG_PER_VMA_LOCK */ - struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; @@ -474,10 +445,6 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - kmem_cache_free(vm_area_cachep, vma); - return NULL; - } return vma; } @@ -496,10 +463,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) * will be reinitialized. */ data_race(memcpy(new, orig, sizeof(*new))); - if (!vma_lock_alloc(new)) { - kmem_cache_free(vm_area_cachep, new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); vma_numab_state_init(new); dup_anon_vma_name(orig, new); @@ -511,7 +475,6 @@ void __vm_area_free(struct vm_area_struct *vma) { vma_numab_state_free(vma); free_anon_vma_name(vma); - vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -522,7 +485,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); __vm_area_free(vma); } #endif @@ -3190,11 +3153,9 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - - vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); -#ifdef CONFIG_PER_VMA_LOCK - vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); -#endif + vm_area_cachep = KMEM_CACHE(vm_area_struct, + SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + SLAB_ACCOUNT); mmap_init(); nsproxy_cache_init(); } diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index b973b3e41c83..568c18d24d53 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -270,10 +270,10 @@ struct vm_area_struct { /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) - * - vm_lock->lock (in write mode) + * - vm_lock.lock (in write mode) * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) - * - vm_lock->lock (in read or write mode) + * - vm_lock.lock (in read or write mode) * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). * @@ -282,7 +282,7 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - struct vma_lock *vm_lock; + struct vma_lock vm_lock; #endif /* @@ -459,17 +459,10 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) return mas_find(&vmi->mas, ULONG_MAX); } -static inline bool vma_lock_alloc(struct vm_area_struct *vma) +static inline void vma_lock_init(struct vm_area_struct *vma) { - vma->vm_lock = calloc(1, sizeof(struct vma_lock)); - - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); + init_rwsem(&vma->vm_lock.lock); vma->vm_lock_seq = UINT_MAX; - - return true; } static inline void vma_assert_write_locked(struct vm_area_struct *); @@ -492,6 +485,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); + vma_lock_init(vma); } static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) @@ -502,10 +496,6 @@ static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - free(vma); - return NULL; - } return vma; } @@ -518,10 +508,7 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return NULL; memcpy(new, orig, sizeof(*new)); - if (!vma_lock_alloc(new)) { - free(new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); return new; @@ -691,14 +678,8 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - free(vma->vm_lock); -} - static inline void __vm_area_free(struct vm_area_struct *vma) { - vma_lock_free(vma); free(vma); } From patchwork Fri Dec 6 22:52:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13897899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B1FBE77173 for ; Fri, 6 Dec 2024 22:52:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96E866B030B; Fri, 6 Dec 2024 17:52:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 91E8E6B030D; Fri, 6 Dec 2024 17:52:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 749DE6B030E; Fri, 6 Dec 2024 17:52:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 56ED86B030B for ; Fri, 6 Dec 2024 17:52:16 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1581A1C6B8D for ; Fri, 6 Dec 2024 22:52:16 +0000 (UTC) X-FDA: 82866033558.25.1FC64AE Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf07.hostedemail.com (Postfix) with ESMTP id 73B7740014 for ; Fri, 6 Dec 2024 22:51:56 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=akPkumXV; spf=pass (imf07.hostedemail.com: domain of 3HYBTZwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3HYBTZwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733525518; a=rsa-sha256; cv=none; b=kL03I/AISMGJotQQK3JVrhUdsfGVXNmZM7T9EV4kE1VagIbgoO13zbpzxRjfRWN+c0r4Dn 8oAfaMRLfEufUeG+cYZU3vCL+u5GUP1AHB/37lriHBeqw52XyEPGtbXJIyPXFaRncPVdFJ UuE9r/tZcT0Zly0vVCyygS3+iWVbZDk= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=akPkumXV; spf=pass (imf07.hostedemail.com: domain of 3HYBTZwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3HYBTZwYKCKwegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733525518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/Y40baPUGWax4wYRQAtfppaknEh3um9zBwyBL9xV1Y4=; b=Qmp3r9nBJLEZKVBJdfg7Mqfw1L6MNkqwWtfqTyLqoZN5YM4KS9TCT66QJOu6zu0T9tEFZm 3gcb4I0+UnYMWNO3mOyGCOqw35XIMJc6NQd9pfZxFlMdCN2FwTj5qsuqK7FwFLuNC14EvK z5knoxfhOIU3Ttcyjqtt+Km478+5iIQ= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ef7fbd99a6so275887a91.1 for ; Fri, 06 Dec 2024 14:52:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733525533; x=1734130333; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/Y40baPUGWax4wYRQAtfppaknEh3um9zBwyBL9xV1Y4=; b=akPkumXV3REPAU/q/XweZ8BeJPKRkue2SZjQDJEMduUWhIzLDMMGO1WlsUjDo20k47 OsOEXiYsHHYgRBAXChavJhmhTwI3kBvsRejnWuYAlcyMBo8Z5heDgKLDTjxa17M4BPpL fQjnr+hIP1ly9n0ZrEZsyDtGY+ZJntjbA3lflldBXDqOjFKcmmZ4aBsRiXqgUhMzJ20F Fydkus+thW/xWEBcT6zlamosIxJdOfF5trIRlUPdm0fYLHN+x0QFXBbOVzJEnNCXJIRi 2oVC9nO5YLeWV1K45DEfuLvKvVcUpHN/VCA2IkbBCQfr28L/IkJ7KwPsD4R4x/6ZGQ+v ZLJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525533; x=1734130333; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/Y40baPUGWax4wYRQAtfppaknEh3um9zBwyBL9xV1Y4=; b=VkEOq4s2+h8V17rS6HK88IJX5KLIZizA23JaD8fEETao3kiYK11/x1+fpINYnoEKKj p2xnnovUND672FsVwwTgRZ7dkCq5T5YuRembJz3I6EIv1+3iX7t3ZpjY4QnbVoc5s5eO QT6NFwnTqazT4Fbgw9NdV9cNvXuCiXtsJ3n2IcglB5JV6e6Fsnl+9fqr5XL/2E9k8AUZ zLDAexsXGbHaL314vydCNaZovpmxMTdtBrsAT8rBJZozTkyUGssRuHbpKUYAkFEfPqKU YFVi2tsZFMhzumnfsZSXWGwHg5Y2to3TrakwXy8nZv04h65DIH5IFaa2GLTCTOcStOAD EvWQ== X-Forwarded-Encrypted: i=1; AJvYcCXjnyUctx0XxpL9/2XvDsO8kTKzVLru+z12fcpzb2JAC8o17NdSmFDfrYOuoTlgJafTwhKcb583OA==@kvack.org X-Gm-Message-State: AOJu0Yzo12CaQpMUZkKXSaeX4ctGjQvnLszF/cGUnMzq58YaYo2rbfeK rbMWrvbHyjeozHQqgzFppFIt3fARhEZP/VAv+GdzSVrDYoVtjFO5xuxDGV+uQLDxfyXdhq3nBW8 49Q== X-Google-Smtp-Source: AGHT+IHCd2OHlcBdUGfCCxX6+VS5I4FF5b82AKepICjOdPKRrERfe6SXg3caG61pXASigp7AKy6oYeA6ikU= X-Received: from pjbmf7.prod.google.com ([2002:a17:90b:1847:b0:2e1:8750:2b46]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2d43:b0:2ee:ba84:5cac with SMTP id 98e67ed59e1d1-2ef6955f863mr7322434a91.7.1733525533068; Fri, 06 Dec 2024 14:52:13 -0800 (PST) Date: Fri, 6 Dec 2024 14:52:00 -0800 In-Reply-To: <20241206225204.4008261-1-surenb@google.com> Mime-Version: 1.0 References: <20241206225204.4008261-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206225204.4008261-4-surenb@google.com> Subject: [PATCH v5 3/6] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 73B7740014 X-Stat-Signature: bownb4s4wpfobao7xz7iqcjeowwqdmaz X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733525516-350001 X-HE-Meta: U2FsdGVkX18lw5rQGmog/R1FR4ERq9pW9d6pmzglwgz0dfjd0CJ2PhC0kvPehurcwItEXQMYAyGW85TjdkWdBRvr7EOfUpmwbHpKZec0uJ+20rNodsat0oO9MX6UKyWfmW/vG0EtrtGN/+8RjgxbfN5iqWGp/h/oGJFY3TDfY7axnbfYmZ2LNUJ/Fj9HBaVOC4IHi3NIXw378fc7JoztOvS0MmBJesvtxJF8nfqH1wCBB1akuu5HhSfhU4NZa+kzx1CUu4+GQIM0f6kfaCkpHJjMiLN6FSlno9J/sd0ZdcVEOISZHqZ18RdIXPZfO7vzlJMZ5zcEXj8ujSClIAR142Bozx8aUS9nb3+kvpJjy8krITeX8c3TuK/dQa+wUxZ19+72gay888Y/Qsjop65DlxX96TsFa3gCY/DOReAo9Pp6csFnH0NG19YTZ2w3fDrNfxxplaajsUdQGxf61M2e2uksHmqeUb0ibLb7cXGavfoTWTvEhtN7v0N8yeBFFpvMWPe4exV4FwLIVq3CQl0ADvoJ9S7iORpATu57M3zdB2bv6sqQkC+zT+TFaI+qCjYU8PCum50X80g5AwuvlW6pKRsBFSO0y7c8m9mNe+dMMW2XZgFdvSA5jJejtP27lSnFUqn9jRIh43b+ss6teoOInlaHTeHDGzk9pO7c5xxe30GxUGwySBpXrci2FMssV+1CfVkKgCqsgnIFD58fho3X3xTxbSh40tIR7zVa4bXpEzWbY4H23QvcysessSf1crPi+qtzUMp9ceq3pvhRBWkZEV8rqZH4EL5wvKU27Zt2tm4TBZ09h2jt2ucbunkMKcW3gjqZnXWFpYkewgErTEwAmH031OiR5UWq4vkzqMtL4YqV3prmBJzZyVUud+w5BaNT6pLQ2Lz9fwMoS8D+j3AXnrAZQVyfkP+Qvc0FR+DaYyoNKB+Y+xsO4/wwV5hYrH3rePav/3JZk+LO+gA6T44 gYqk+WfB XX50UaPIAARtv2hWPEWF1+n73DP5GBWg/zUDN9fQYz17fcTZSOEOLAJmuu4soKy23afcSsZYhvSTR9Jj+91Lzmoxj3t1OyjZfxN6iv+kB4vtX7NP/iVSK/Moh7e8RvZLGN3ez9VExlR+Qkv/sGoReNNJpz8X4kSXjXyybvEW1aS102UmPl/Gmy7fG8V1CR5I3zUjuAcl+A1I6TiCY3nOPfSwGN66y9tQDImGWTMMmB+T/+vZlR4PXNzWQTMYQcag0+Z8sfMsM301bENz9xx6ROwghBecz/cMZSHtWH+bv1DyQ35brkZuroYcAoWV76/+xQRAvAHUFmbfAFPItZiiNCHMl6b4v9HC19iITC1gkPYEhbZyeIUEBvz+uv8wxL5U+NoNksRCeNgfS0FR9pefBwGguf+8TyJQ1K5RNfo2C8LUFeV0vNMd5wlxCxkn6Do0p/0j0z4OFSHD6yRZp/eGLkHwwAbSKqZWUhboDEPkk8ELFWelWpr2v9UiM11+RmeXnCgA4g/8anc7PRrMIoC6944fSg0e955KfgQEcy4J5FBf45iplThKYlwoq7AWxdku+0FwX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Vlastimil Babka --- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ee71a504ef88..2bf38c1e9cca 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -820,12 +820,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) vma_assert_write_locked(vma); } -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; +} + +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -856,8 +865,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) { mmap_assert_write_locked(vma->vm_mm); } -static inline void vma_mark_detached(struct vm_area_struct *vma, - bool detached) {} +static inline void vma_mark_attached(struct vm_area_struct *vma) {} +static inline void vma_mark_detached(struct vm_area_struct *vma) {} static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, unsigned long address) @@ -890,7 +899,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } @@ -1085,6 +1097,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } diff --git a/kernel/fork.c b/kernel/fork.c index 21660a9ad97a..71990f46aa4e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) data_race(memcpy(new, orig, sizeof(*new))); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init(new); dup_anon_vma_name(orig, new); diff --git a/mm/memory.c b/mm/memory.c index f823906a4a0f..b252f19b28c9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6372,7 +6372,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/vma.c b/mm/vma.c index a06747845cac..cdc63728f47f 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -327,7 +327,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, if (vp->remove) { again: - vma_mark_detached(vp->remove, true); + vma_mark_detached(vp->remove); if (vp->file) { uprobe_munmap(vp->remove, vp->remove->vm_start, vp->remove->vm_end); @@ -1220,7 +1220,7 @@ static void reattach_vmas(struct ma_state *mas_detach) mas_set(mas_detach, 0); mas_for_each(mas_detach, vma, ULONG_MAX) - vma_mark_detached(vma, false); + vma_mark_attached(vma); __mt_destroy(mas_detach->tree); } @@ -1295,7 +1295,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, if (error) goto munmap_gather_failed; - vma_mark_detached(next, true); + vma_mark_detached(next); nrpages = vma_pages(next); vms->nr_pages += nrpages; diff --git a/mm/vma.h b/mm/vma.h index 295d44ea54db..32d99b2963df 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -156,6 +156,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi, if (unlikely(mas_is_err(&vmi->mas))) return -ENOMEM; + vma_mark_attached(vma); return 0; } @@ -385,6 +386,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi, __mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1); mas_store_prealloc(&vmi->mas, vma); + vma_mark_attached(vma); } static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 568c18d24d53..0cdc5f8c3d60 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -465,13 +465,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq = UINT_MAX; } +static inline void vma_mark_attached(struct vm_area_struct *vma) +{ + vma->detached = false; +} + static inline void vma_assert_write_locked(struct vm_area_struct *); -static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) +static inline void vma_mark_detached(struct vm_area_struct *vma) { /* When detaching vma should be write-locked */ - if (detached) - vma_assert_write_locked(vma); - vma->detached = detached; + vma_assert_write_locked(vma); + vma->detached = true; } extern const struct vm_operations_struct vma_dummy_vm_ops; @@ -484,7 +488,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; vma_lock_init(vma); } @@ -510,6 +515,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) memcpy(new, orig, sizeof(*new)); vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); + /* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; return new; } From patchwork Fri Dec 6 22:52:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13897900 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDCB1E77179 for ; Fri, 6 Dec 2024 22:52:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC7606B030D; Fri, 6 Dec 2024 17:52:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E77626B030F; Fri, 6 Dec 2024 17:52:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA4636B0310; Fri, 6 Dec 2024 17:52:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A18746B030D for ; Fri, 6 Dec 2024 17:52:18 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 29B3BAF8AD for ; Fri, 6 Dec 2024 22:52:18 +0000 (UTC) X-FDA: 82866033810.22.6E3C568 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf03.hostedemail.com (Postfix) with ESMTP id 6A0CA2000F for ; Fri, 6 Dec 2024 22:52:08 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Zlv/UxuT"; spf=pass (imf03.hostedemail.com: domain of 3HoBTZwYKCK0fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3HoBTZwYKCK0fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733525524; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7xWkv895t2vkmivZEcKc0VmEj0Py63DFJG/LYdHvb3Y=; b=0xJJZaNiGVsNgEj/vWiiHbiyGfRjQCBI/FmZAj1ycJBCxhmWCpAcuOjzsSzqHtV3wzvA/G Aefo7Zz1IeT657F1KujG+4hRbKM4YRJpLQ5jB3WdgpDds9uUI8bKFmh2G5RfRVb1f7lJ46 og4trtvGuB5Tmvarbz88ausO/n6Evx8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Zlv/UxuT"; spf=pass (imf03.hostedemail.com: domain of 3HoBTZwYKCK0fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3HoBTZwYKCK0fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733525524; a=rsa-sha256; cv=none; b=mNVbS0V3udwWTrX3rAYnjNqr17hmxS0AYDntRqiJr6+QkFy8RfOm416LcfkLqQTf3W3/pp CzRi6caqLt9fV/jRYOtu/EYln4rC3crISOV/GcPt8CU4sI5zMLLTRpiklV/B2ZkziIcGIA yRaGKE4sE/up5F1uhVBJUyqJDHKdQoY= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2154e1cdf72so28045105ad.3 for ; Fri, 06 Dec 2024 14:52:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733525535; x=1734130335; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7xWkv895t2vkmivZEcKc0VmEj0Py63DFJG/LYdHvb3Y=; b=Zlv/UxuTC8ozdSjOi+u5ZkXDrEf0BV5hCNk4T7tNbRUVFe+vGdKpf4Xa+TC0s9zH5S IPxzd7nDudGHkD2bdTjLA1Xp38d7kbVttpV+75iZJXLgmlFimIighyKHq2Q4pV2Ac2ih Qap8t5vaRjOSdvdI6Is4bS/MwSOcXhtQNwHdr0tOIpBwATgCxNqWWdIvUwYUOG2T5J0e gUvigYzGNlpS7MGk1yiYDZts8VN70jcaZ3D0aP8A5X0IB+EQj81yJIMDzb+ddjmSsNer CIwGjvgtWdiYm1dmjweY8st/PloHYWkOf5IfTahgpUoQ2wLSFu3D0AABDSxBZy7+wYIt vO+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525535; x=1734130335; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7xWkv895t2vkmivZEcKc0VmEj0Py63DFJG/LYdHvb3Y=; b=gi5v8cZdws11kYJZSySyjJtGvuIcUpC/zMqrPcfn3dWXixYSkEYRAO+Z/dNrWrRQi6 h//QYf07s0JAfs3eJVNWOE/duLEfOPWVY6si7lUW7AaYTk633jJh4wTqw+MNlYHAaXrp WwDqh4Pk2aiPU4FAQzHPW3iPFjxKkCIjrXtMxy4fLNpqpg85sypHlfR0aW9Z2npOoxP1 na3Y7DNSZSMKwtOLGUhRTlrxI5zrpdNiT8MBARthbawO1dRnz2CRf6Mrt0RLGUdRSpPp eqTbqv0nW5/ivyc8oZVjG5r78HAVAGwdddRHqA+8+9hyrtXniOblbqNu0YJYHTz2oQxg dn8Q== X-Forwarded-Encrypted: i=1; AJvYcCXT06o6iPXGbshnAMKv1iFjSJpI9lj1tJDuTRiIrtcaCHklego85rGmTsCWbkVLVwecRFIJ9ni1rA==@kvack.org X-Gm-Message-State: AOJu0YzZDVCaxRMdZ7rJ//CRFSLb+2sXYr/jGp2OOh78N2Y7L6Vu+USY 1N87DKwFXV8T2kkUS06B9nwMeriQl+l0NV/22e8sMWB14Kj02CIDtyzYyg+TUc5uuD/qbf+cV3B USw== X-Google-Smtp-Source: AGHT+IE5HYCr7W0ZnBekrRI/mwxgnpYPkvPquVys2NZfqfzmdtEe1gkkS7IzDbSltL7UMNDRR9lW/VjGZAs= X-Received: from plbkd6.prod.google.com ([2002:a17:903:13c6:b0:20c:526b:44a8]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:fc86:b0:215:9091:4f57 with SMTP id d9443c01a7336-21614dac170mr62965795ad.43.1733525534992; Fri, 06 Dec 2024 14:52:14 -0800 (PST) Date: Fri, 6 Dec 2024 14:52:01 -0800 In-Reply-To: <20241206225204.4008261-1-surenb@google.com> Mime-Version: 1.0 References: <20241206225204.4008261-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206225204.4008261-5-surenb@google.com> Subject: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam05 X-Stat-Signature: qa93x5a1iggiahjrx8m1y73r3tpadqcu X-Rspamd-Queue-Id: 6A0CA2000F X-Rspam-User: X-HE-Tag: 1733525528-3686 X-HE-Meta: U2FsdGVkX18vjxFzOmm8wmuP4qNQ9t1JLQlLYIJ+IERvu2Z/ZOFXlTnEJZ2OoaMjHPiqAROkM6NLaL8wOp6sXMEGQlhqakSToSQ9CFDYni2+NM1isifvYHLvVal8yRoHdL2ZCeaTK9CN5B0kFR345VLfzWDK2e3JUjBznZUlC0m8I4Ty1NgLaOJ3Zxm9tXkWBzrHqjvvu42cOrENDX1o8DrWEWOL9fBvOg0RXEB4Ck8pNfzkIIJSO8GbPglqms5TXUQLd1nYHHrrsuceBpDI+0y5kGm3xVVHx0c7Ax7e+RYTZr/e7V8pANJ0lutGk2rJdAwFreg77tQQqRCQeyyq1W30veiYCx41iK8hCbULOR15zZGhuX0AW0sSrkNVlNSNXYcXSSkLG03wKfoyYAYne51U/3Roz3Cp+qq+fAAGT0oDfnnW4WpILb1oOnbPedSm9928KMuAZV1oSSJRXBhxY1vLeDjG2rDITJqwq9RrKhxt2HPKETW88d+YaMTtC1BNQOceFBzQopta7RWxbWtDsNkdeWxQsAIz4AUDiYish+/rUO3fwO7VhgryzWV/zK0AViHdZxwWErTZt6RZYAmpARbR5/stiDxtbykY19Hf9s6b9982X9NplohzaNY1ciU0iehrlv4YNxTNP2yAkvkaycTE61EvetGK10qdVUvakuZAW9iC9pA1RVRfMVdBLCM9zwGS97silPfHnPPgoOGZU4iQ4+WVhqyn36Ed7Be4eOaF4NKI5358c/fcwFY403poobuVWsBIoqQ49Y66n22jplKtbMoV5rgSJg8a7O5hsJWNkKeJFUWnqeROvnQcAwQPMNClQIvVq4ebbf5Sap2/FLT3XQX6QK46V3Frib3EO9GIeLjEZy+YLFRruHuXbquMvWg1dhjJ5LavLLq6Bd3P3/FPhos80Sp1jUufpPQwIX1waa8WqCe5KpOgUJZrqq/gbQzeDctnPjoIzpdxiza BhwmRkH7 f/V6cKTrhgNyOmJHc7tRp2dlFTzhfCm4m39ifBZOFJl7ZmQac2F4MGC6TaCHYpz41ePROaqPVn2LtL5HZhXzN3yyKNS9760qyjm1zXNOlSunkNunqm1mLBT97xAAMlFnEC9GDa1o/ek/cEPKZ9yzpOhROX5LgZMJNCLHYrHLDeLkL3OyksVoOP7iePw5oaU198Gm6IvaE+6AQidj9M691r2FNt+VPavvMhrKhAncomseFR1sGRImvPpJIoTXa2IA+epes6IvFQBo7oSNCYCaqJZg3QhEG5nKuLoeXugVKIFSL6poKsPpjL7I2r0rPjJJfLhTyyOJsqLpPDXnGMRvEiHMq1wHEysWPh6C0ce6cxpjFZJ50CmL6H6X7lO8dGbDJh1oo1BnOCqj/5vIQpkqsv6GscOGflrleBAq6VN62mk1hRMmJvvhC7KbHn7lI2CpQxGZuQEx/YRQKec9le4dIYRl+82/iGtuBztZAFakB9c82+EjuF5+PQldINCOzzzmbtrza+ZhqV4n4ZNhXbsoPsny3lzRx71bTdVjfMLFV382BAbzrhfR+Am8WB23ULCiSKAOHnDV5RsHXvrZkMlWURHmDKA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu(). lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it. vma reuse introduces several new possibilities: 1. vma can be reused after it was found but before it is locked; 2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read(); 3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked; For case #1 current checks will help detecting cases when: - vma was reused but not yet added into the tree (detached check) - vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock. For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 36 +++++-- include/linux/mm_types.h | 10 +- include/linux/slab.h | 6 -- kernel/fork.c | 157 +++++++++++++++++++++++++------ mm/memory.c | 15 ++- mm/vma.c | 2 +- tools/testing/vma/vma_internal.h | 7 +- 7 files changed, 179 insertions(+), 54 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2bf38c1e9cca..3568bcbc7c81 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); /* Use only if VMA has no other users */ -void __vm_area_free(struct vm_area_struct *vma); +void vm_area_free_unreachable(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; @@ -706,8 +706,10 @@ static inline void vma_lock_init(struct vm_area_struct *vma) * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. + * False locked result is possible if mm_lock_seq overflows or if vma gets + * reused and attached to a different mm before we lock it. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { /* * Check before locking. A race might cause false locked result. @@ -716,7 +718,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return false; if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) @@ -733,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ - if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { up_read(&vma->vm_lock.lock); return false; } @@ -822,7 +824,15 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) static inline void vma_mark_attached(struct vm_area_struct *vma) { - vma->detached = false; + /* + * This pairs with smp_rmb() inside is_vma_detached(). + * vma is marked attached after all vma modifications are done and it + * got added into the vma tree. All prior vma modifications should be + * made visible before marking the vma attached. + */ + smp_wmb(); + /* This pairs with READ_ONCE() in is_vma_detached(). */ + WRITE_ONCE(vma->detached, false); } static inline void vma_mark_detached(struct vm_area_struct *vma) @@ -834,7 +844,18 @@ static inline void vma_mark_detached(struct vm_area_struct *vma) static inline bool is_vma_detached(struct vm_area_struct *vma) { - return vma->detached; + bool detached; + + /* This pairs with WRITE_ONCE() in vma_mark_attached(). */ + detached = READ_ONCE(vma->detached); + /* + * This pairs with smp_wmb() inside vma_mark_attached() to ensure + * vma->detached is read before vma attributes read later inside + * lock_vma_under_rcu(). + */ + smp_rmb(); + + return detached; } static inline void release_fault_lock(struct vm_fault *vmf) @@ -859,7 +880,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init(struct vm_area_struct *vma) {} -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} @@ -893,6 +914,7 @@ static inline void assert_fault_locked(struct vm_fault *vmf) extern const struct vm_operations_struct vma_dummy_vm_ops; +/* Use on VMAs not created using vm_area_alloc() */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { memset(vma, 0, sizeof(*vma)); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index be3551654325..5d8779997266 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -543,6 +543,12 @@ static inline void *folio_get_private(struct folio *folio) typedef unsigned long vm_flags_t; +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * A region containing a mapping of a non-memory backed file under NOMMU * conditions. These are held in a global tree and are pinned by the VMAs that @@ -657,9 +663,7 @@ struct vm_area_struct { unsigned long vm_start; unsigned long vm_end; }; -#ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ -#endif + freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */ }; /* diff --git a/include/linux/slab.h b/include/linux/slab.h index 10a971c2bde3..681b685b6c4e 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -234,12 +234,6 @@ enum _slab_flag_bits { #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED #endif -/* - * freeptr_t represents a SLUB freelist pointer, which might be encoded - * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. - */ -typedef struct { unsigned long v; } freeptr_t; - /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/kernel/fork.c b/kernel/fork.c index 71990f46aa4e..e7e76a660e4c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -436,6 +436,98 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; +static void vm_area_ctor(void *data) +{ + struct vm_area_struct *vma = (struct vm_area_struct *)data; + +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif + INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_lock_init(vma); +} + +#ifdef CONFIG_PER_VMA_LOCK + +static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm) +{ + vma->vm_mm = mm; + vma->vm_ops = &vma_dummy_vm_ops; + vma->vm_start = 0; + vma->vm_end = 0; + vma->anon_vma = NULL; + vma->vm_pgoff = 0; + vma->vm_file = NULL; + vma->vm_private_data = NULL; + vm_flags_init(vma, 0); + memset(&vma->vm_page_prot, 0, sizeof(vma->vm_page_prot)); + memset(&vma->shared, 0, sizeof(vma->shared)); + memset(&vma->vm_userfaultfd_ctx, 0, sizeof(vma->vm_userfaultfd_ctx)); + vma_numab_state_init(vma); +#ifdef CONFIG_ANON_VMA_NAME + vma->anon_name = NULL; +#endif +#ifdef CONFIG_SWAP + memset(&vma->swap_readahead_info, 0, sizeof(vma->swap_readahead_info)); +#endif +#ifndef CONFIG_MMU + vma->vm_region = NULL; +#endif +#ifdef CONFIG_NUMA + vma->vm_policy = NULL; +#endif +} + +static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest) +{ + dest->vm_mm = src->vm_mm; + dest->vm_ops = src->vm_ops; + dest->vm_start = src->vm_start; + dest->vm_end = src->vm_end; + dest->anon_vma = src->anon_vma; + dest->vm_pgoff = src->vm_pgoff; + dest->vm_file = src->vm_file; + dest->vm_private_data = src->vm_private_data; + vm_flags_init(dest, src->vm_flags); + memcpy(&dest->vm_page_prot, &src->vm_page_prot, + sizeof(dest->vm_page_prot)); + memcpy(&dest->shared, &src->shared, sizeof(dest->shared)); + memcpy(&dest->vm_userfaultfd_ctx, &src->vm_userfaultfd_ctx, + sizeof(dest->vm_userfaultfd_ctx)); +#ifdef CONFIG_ANON_VMA_NAME + dest->anon_name = src->anon_name; +#endif +#ifdef CONFIG_SWAP + memcpy(&dest->swap_readahead_info, &src->swap_readahead_info, + sizeof(dest->swap_readahead_info)); +#endif +#ifndef CONFIG_MMU + dest->vm_region = src->vm_region; +#endif +#ifdef CONFIG_NUMA + dest->vm_policy = src->vm_policy; +#endif +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm) +{ + vma_init(vma, mm); +} + +static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest) +{ + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + data_race(memcpy(dest, src, sizeof(*dest))); +} + +#endif /* CONFIG_PER_VMA_LOCK */ + struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; @@ -444,7 +536,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) if (!vma) return NULL; - vma_init(vma, mm); + vma_clear(vma, mm); return vma; } @@ -458,49 +550,46 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - data_race(memcpy(new, orig, sizeof(*new))); - vma_lock_init(new); - INIT_LIST_HEAD(&new->anon_vma_chain); -#ifdef CONFIG_PER_VMA_LOCK - /* vma is not locked, can't use vma_mark_detached() */ - new->detached = true; -#endif + vma_copy(orig, new); vma_numab_state_init(new); dup_anon_vma_name(orig, new); return new; } -void __vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma, bool unreachable) { +#ifdef CONFIG_PER_VMA_LOCK + /* + * With SLAB_TYPESAFE_BY_RCU, vma can be reused and we need + * vma->detached to be set before vma is returned into the cache. + * This way reused object won't be used by readers until it's + * initialized and reattached. + * If vma is unreachable, there can be no other users and we + * can set vma->detached directly with no risk of a race. + * If vma is reachable, then it should have been already detached + * under vma write-lock or it was never attached. + */ + if (unreachable) + vma->detached = true; + else + VM_BUG_ON_VMA(!is_vma_detached(vma), vma); + vma->vm_lock_seq = UINT_MAX; +#endif + VM_BUG_ON_VMA(!list_empty(&vma->anon_vma_chain), vma); vma_numab_state_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb(struct rcu_head *head) +void vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); - - /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); - __vm_area_free(vma); + __vm_area_free(vma, false); } -#endif -void vm_area_free(struct vm_area_struct *vma) +void vm_area_free_unreachable(struct vm_area_struct *vma) { -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); -#else - __vm_area_free(vma); -#endif + __vm_area_free(vma, true); } static void account_kernel_stack(struct task_struct *tsk, int account) @@ -3141,6 +3230,12 @@ void __init mm_cache_init(void) void __init proc_caches_init(void) { + struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr), + .ctor = vm_area_ctor, + }; + sighand_cachep = kmem_cache_create("sighand_cache", sizeof(struct sighand_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| @@ -3157,9 +3252,11 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - vm_area_cachep = KMEM_CACHE(vm_area_struct, - SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + vm_area_cachep = kmem_cache_create("vm_area_struct", + sizeof(struct vm_area_struct), &args, + SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| SLAB_ACCOUNT); + mmap_init(); nsproxy_cache_init(); } diff --git a/mm/memory.c b/mm/memory.c index b252f19b28c9..6f4d4d423835 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6368,10 +6368,16 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) + if (!vma_start_read(mm, vma)) goto inval; - /* Check if the VMA got isolated after we found it */ + /* + * Check if the VMA got isolated after we found it. + * Note: vma we found could have been recycled and is being reattached. + * It's possible to attach a vma while it is read-locked, however a + * read-locked vma can't be detached (detaching requires write-locking). + * Therefore if this check passes, we have an attached and stable vma. + */ if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); @@ -6385,8 +6391,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check if the vma we locked is the right one. */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock(); diff --git a/mm/vma.c b/mm/vma.c index cdc63728f47f..648784416833 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -414,7 +414,7 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable) fput(vma->vm_file); mpol_put(vma_policy(vma)); if (unreachable) - __vm_area_free(vma); + vm_area_free_unreachable(vma); else vm_area_free(vma); } diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 0cdc5f8c3d60..3eeb1317cc69 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -685,14 +685,15 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void __vm_area_free(struct vm_area_struct *vma) +static inline void vm_area_free(struct vm_area_struct *vma) { free(vma); } -static inline void vm_area_free(struct vm_area_struct *vma) +static inline void vm_area_free_unreachable(struct vm_area_struct *vma) { - __vm_area_free(vma); + vma->detached = true; + vm_area_free(vma); } static inline void lru_add_drain(void) From patchwork Fri Dec 6 22:52:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13897901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C8F5E7717B for ; Fri, 6 Dec 2024 22:52:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5A706B030F; Fri, 6 Dec 2024 17:52:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E1F56B0311; Fri, 6 Dec 2024 17:52:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 837116B0312; Fri, 6 Dec 2024 17:52:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 55CB56B030F for ; Fri, 6 Dec 2024 17:52:20 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1C94C1A08B6 for ; Fri, 6 Dec 2024 22:52:20 +0000 (UTC) X-FDA: 82866033642.01.83458BD Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf20.hostedemail.com (Postfix) with ESMTP id A72C01C0009 for ; Fri, 6 Dec 2024 22:51:59 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nZLqgrAj; spf=pass (imf20.hostedemail.com: domain of 3IYBTZwYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3IYBTZwYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733525530; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VpkME8tKt7xRuIeyUR2rAOpo4R/J9PT1fVyVW9ktqRE=; b=QO9aHzuljn6eoH+MZJ5b8EAHDnETSD+QLtdmeIW1GW1ny5mT+VbAjMBC0g15An4d3gzFjP W9TM15fcZFyBrBrRoOhKoS7NmFqzlCxBWzuibAROmML/08on9F76iWwJL4evQXgmfcoh4M In49ALr9vm6FwTlGzPYTz6N9XZXYcpk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nZLqgrAj; spf=pass (imf20.hostedemail.com: domain of 3IYBTZwYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3IYBTZwYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733525530; a=rsa-sha256; cv=none; b=kDiZJ1woCFMl6VUKnc4PFz5JR+MrB2sI4gBQsb0s6reevwWdEufy9/SW5Y6lN2IJJphVDY qN3EJlvsWd4qaNPpNesvZ9k1mVoVQFktqcDrasHs2GUcWT5eFh3r0+Y1ftzG7GU3iE4zxI nVMtRCEXOdY1VQpV6k5U66pBdepJl9I= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ee86953aeaso2515938a91.2 for ; Fri, 06 Dec 2024 14:52:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733525537; x=1734130337; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VpkME8tKt7xRuIeyUR2rAOpo4R/J9PT1fVyVW9ktqRE=; b=nZLqgrAj8Sak3PGy9pxQdmKdQJ1cWPCLYgiveXI3KprtcgC/WFf/4IE4KuSyr4ziiQ Iep8xqFUjPHWojyIq9kmxTgMccPhy/iSDYfYOEMMJX43BcrrIcq7NQAaItNEwfIoyv+1 1Hdk4IGO1ojpSQPBfmXIm+LMYfQoE5oavisfCL04wr8atoS7h3P2E239STdyXo7IFgwp 5zRbV/e9oUY4w1uA/lz4OA8N+AuRzOeMBM8yDIZuPDRlIRAdo7e/omj8yIF1uyj5kA/y fsWmeNCUm174/RRf8D5cxp5dj0n+RbQ1Brtmgy0uhQskvB0mQQXXKp0+6/ZQk6TLJz30 ucHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525537; x=1734130337; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VpkME8tKt7xRuIeyUR2rAOpo4R/J9PT1fVyVW9ktqRE=; b=nuB0VK5ITGvd11JadX5Sd4yEmLx8zl701D14Se4GFDfVNVTIDNeX4KaILSNcvZU1kQ jkf8paypXrqy/u5U+uHdk758HoevRg6VlIRQ8WrfiprP8rvbsa2VQEr8R8an0i3BG1Z+ VWGXD6kzqMfL3GIv0+PqiJjtR/Le3KNoqQm8dRw639ZSD74s5CjkgldKmRMwHjmaM39o E/H/nmFGbpckH2iPuFMNGSNcbWMj9bTpsHPRS9ZfWNBmoxdWcnVRhY1yhUdApRK0wCi/ hV54WYSfkHiLQkMtPRNJQ+8X1sSeA5OXUwImyWuYx61lNHtlJ5hkPCzl38uKPSUMaEMh kRwA== X-Forwarded-Encrypted: i=1; AJvYcCVpg3+YRltAup7PbM0XnRMEirblQcPRyDGNeTGsjRle2IFrHAwvIa5IZGJUbUufkTiailN6aYvgSw==@kvack.org X-Gm-Message-State: AOJu0Yw3sHFSSqNUYCUN4ZJc9VExezcqoxv9b+Gi3be/KHRL+7Z4Schf L18brI8rDsKfhThzmL+kny/C/w6ti5GMdAFKxXd8tYZlMExheQkrHGgVZoMI8MqMkin0XcuUFIU RiQ== X-Google-Smtp-Source: AGHT+IHFlvnZGP/uT3lo2rN1Kek/VGxFz7KPL1PDra1Y7TXq5etwmH5ZVO3EXQpkF7vZG3RxgNCe7HXm2rM= X-Received: from pjboi8.prod.google.com ([2002:a17:90b:3a08:b0:2ea:4a74:ac2]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1e43:b0:2ee:8008:b583 with SMTP id 98e67ed59e1d1-2ef69fffa82mr7837910a91.16.1733525537081; Fri, 06 Dec 2024 14:52:17 -0800 (PST) Date: Fri, 6 Dec 2024 14:52:02 -0800 In-Reply-To: <20241206225204.4008261-1-surenb@google.com> Mime-Version: 1.0 References: <20241206225204.4008261-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206225204.4008261-6-surenb@google.com> Subject: [PATCH v5 5/6] mm/slab: allow freeptr_offset to be used with ctor From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: A72C01C0009 X-Stat-Signature: rp3qhn4brn8nz4681fzzkrgmsw1wej7u X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733525519-632799 X-HE-Meta: U2FsdGVkX18bKEFsxSfH1cs6uEEs6ehNnwHzf45g/+MCC8Uf+kFobVoRgzZu1i9IeWtSG7UYQ7heMeMTL/NYaWWMvWiq29d5hl+wE94h+7HYmdNUYIjykU/oIF1At38SLOFWoTODU+oY02fJUv++UT95E+rnAuvXEHvwvg3vMIPIFbNGs0ULyn6I1BdXbqPsI0JfhoQkEi1OILZT6nAfWkWdvEw/ywLbGw/i5OXMwCok0Iy/BjufN7ygQYdCdzgpdDIAFfBdJZxglmgnYVgwNdl2XFhLyFezSAr06yPudjCQmTW0mTvQ4AZq3Za+bTESQrlOgeJANvWzZFT+h8qQU2YcJygBl/Qn10+94LYkqImYytxdThBkDWMzusRsiqx0gce0pwIGY94KsqZn0I5dOv5t27MCetQcgWRcHXK19vjsSY4n21o6IjLAXV/nRjrDJlBuqfo/rNAi/D4JNUpFE0BbvwK0OeU8Whqa6qzMauJQk+rvbKMCj+oKdGNxKDv8O3kZy6UCnM/pkoXSRtlMWrc+zyHuL9tQyiJO0hhba3dQmAWuCExEg6iayRhD+p+w6Z08WpKj6DvEDYqhEJeH1YxogRQNgkwjdMoP0IQbXG9Y0KxSoTL199zwyOxcmaTZNkDQEihy1UrH/N+jM1y7Ur6UVAGsi/mC6++tgNIEDzOfpoIbPKDr1+d0b9ge3sEvroCwDK/Ske8Vh12h0gwkvQ9thzcBqfzFkgKiOYQs2XBTd5MMAnNGNVYaUaKUWyDZ+J12qSVUTBmRIW+HH+xRoz89DcMPW8uJSrURdwd0h/2lYjZuP9iRsW860rYFG2l7TlB6uy8fFFpzPNprR+oZ1MfeST8bI+0OtTYQK74aYSi/kSQnD4W7yt8CcvK53lC9i1zrIOIihw+ko0RDpW+S15Rn4ZyGvGyNe4chwufJxyN8JrUKPe0VxjeknFS0HxduBx15XOzAlO9Ow7C3O+a h7PQ6xRT A+s/f4tBoZ4Y2nak9TLSRYiV1W8TxERO5ejT6DCbRanpQZA2IlxMoo3xnEq3IzLq3nzr+sXXjcOFjic3DpACrdvKj9cY0K/KIdAIBP9r7IQTfkUWTzRSw0RFg0a7OO+bM8PVqgG7xmxcORTofLRMtTU6NyqMFy6yCuJyOkJlGQUqg8G19Cdv10n1saSW1TfU6NPrDur75Hct0q4O7g56G6LreY+EhvXoWPRXDgUUpgd7zG586ObBiqLR1NXN9OLa7KjxzPYo5tUKsSURJKpvJ+wbyTPfsubLadKwJHlnHuNRzfWK2lK3M+npN3HAegvtG/kZ8Tcb2KEqunBmYN7Lmn1mzf+jr1WPnrtP0Y4xcm+LXVugQgXUlw8DVGuLNXCoqJT+/OE5ioTaUe2gkhcXIA4CvXdr3QioBzUSYxx7GkDEQMsIz7xwecX8NFD0QVW/gX9pLzQNs6t2fwjogrQnKQRsumqTFnFfIqSdhf+3L4ifEuviWJdrUthOY1/MEJ7s8gJVxFksK0yJF3/UdaHPjUm6gpQ5ZQ6oMfisQnvIraqKlA3+skjop+UVcTdHSM8LNkC9ab5NyOii7VKEjOGyr6aXSBw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no real reason to prevent freeptr_offset usage when a slab cache has a ctor. The only real limitation is that any field unioned with the free pointer and initialized by ctor will be overwritten since free pointer is set after @ctor invocation. Document this limitation and enable usage of freeptr_offset with ctor. Suggested-by: Vlastimil Babka Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 5 +++-- mm/slub.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 681b685b6c4e..6bad744bef5e 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -305,8 +305,9 @@ struct kmem_cache_args { * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset * is specified, %use_freeptr_offset must be set %true. * - * Note that @ctor currently isn't supported with custom free pointers - * as a @ctor requires an external free pointer. + * Note that fields unioned with free pointer cannot be initialized by + * @ctor since free pointer is set after @ctor invocation, so those + * values will be overwritten. */ unsigned int freeptr_offset; /** diff --git a/mm/slub.c b/mm/slub.c index 870a1d95521d..f62c829b7b6b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5462,7 +5462,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s) s->inuse = size; if (((flags & SLAB_TYPESAFE_BY_RCU) && !args->use_freeptr_offset) || - (flags & SLAB_POISON) || s->ctor || + (flags & SLAB_POISON) || (s->ctor && !args->use_freeptr_offset) || ((flags & SLAB_RED_ZONE) && (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) { /* From patchwork Fri Dec 6 22:52:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13897902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1A03E77179 for ; Fri, 6 Dec 2024 22:52:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E84E6B0311; Fri, 6 Dec 2024 17:52:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7993C6B0313; Fri, 6 Dec 2024 17:52:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 611316B0314; Fri, 6 Dec 2024 17:52:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3E0E86B0311 for ; Fri, 6 Dec 2024 17:52:22 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F2A22AF050 for ; Fri, 6 Dec 2024 22:52:21 +0000 (UTC) X-FDA: 82866034104.05.8F51081 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf09.hostedemail.com (Postfix) with ESMTP id 4260E140002 for ; Fri, 6 Dec 2024 22:52:08 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ExttRgZa; spf=pass (imf09.hostedemail.com: domain of 3IoBTZwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3IoBTZwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733525532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j6wEf3Fj/sCIHriciJDFTmwIAy5yi6AjoE4yT83zSv8=; b=754rc+oN6ilJK5N2N1Ci9HO6ghA5GtbbvQMNlPbtmDP4yJSNy75ty0F4P+oFPdUXWwEAPr WPDf8Uk1h0erzqt9zgRu5S+4Y8EIfrKmrUTN711DaVmAZIiWVjeDAnC54uZ7uQ9xWRIbzZ 1W+57pUnb26aOhdlishhFZnCyXFGP6I= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ExttRgZa; spf=pass (imf09.hostedemail.com: domain of 3IoBTZwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3IoBTZwYKCLEjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733525532; a=rsa-sha256; cv=none; b=XLWl6bztjLV7m04s3dEwYLHspjtqCDKdywKgihIr5oyxRvbqa66KTa7+PYczqY6NEiDGuU fCzNRHIs4FfoGGYPYxe1637NAXxGoZtC+5oc+EOlJJV9dfAYbTTuFzcWh/I6oT6QSM4yQq C6nb5IMRZBYPd0LpGw8sSaEg7PAujno= Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-72527c426a2so2078205b3a.2 for ; Fri, 06 Dec 2024 14:52:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1733525539; x=1734130339; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=j6wEf3Fj/sCIHriciJDFTmwIAy5yi6AjoE4yT83zSv8=; b=ExttRgZagR66HuAV5D0+V7dZel1+C/jAZ3J/lMkKpPuLXvXGRuXd3b6xU7+cSBiA0h UVjGNaEVfX4V4H6Aaq9BHOisu6ZWdE+/jV+cx3TLDQe3kJXgvIWxlcP7z8ZqbbXPDBrG muMhg4kvpPomEoUmcJ5YgLLKpGteaJKmAGAW6oqRgkmKVf0umbA4bR4cwGca546j6Xc/ r/3f7HgZoQr+sXRPKQXizrQiLr90LviSuocDtZRu4Vj8JoYroIU0PxMCeoTex6IsdxSL 2GsxA0HQgBwWWKjerzqLu5/N3POw+sRLwS/QrtzbOocNZ/8/oEJw93I/ZBubh0pIWBmc eXnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733525539; x=1734130339; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=j6wEf3Fj/sCIHriciJDFTmwIAy5yi6AjoE4yT83zSv8=; b=leCkF00cYnbSqynxpbWWhf/jX7A4n+hp3EQnRYma0cyc0ZkNhZcZlz2zAxwqfwvBD3 DoT5LJqprBr01lFmoBo2AsEQkQEreb6TsEUvHUv2FHbWmf/5NYVx8TxR9CWK8ZUev6qI Y3YEV4o0ZSXSTG8Ea2AeZw6iNQSf5/v8p4/byn3dzwC2ARpVslBtXab2hIuuQXcL8fMk Ln2q2MKayB+gboXoDBIDbf0vOWkMh5jLxNF01Zun5cRhcUfAJfgIZOJeZxIr/l29k6on aLVZrHLvthtZkQyQ+5JnmjiteQdWzbXdw3y+CZ5z8EYwvklvEn7bMPpjy84z8ncAQrW1 t0qA== X-Forwarded-Encrypted: i=1; AJvYcCWOofvWEteZ7nn80vkCrqzCX2oFvvHUN5RU+isLt4m4lD23wAyluy+xo0F1AitZwLuGFU7Z1tLTPw==@kvack.org X-Gm-Message-State: AOJu0YxuTqs2GzK6e+w0jeo6Het9u+CkDkMr5MRzXOyiXZWreO5LyGvk gqH1mktnDkUx46zBZE7wotgFHZHSE/dkUCodoHMb9GR78Bt6T1rdCoLkiz4/JBXCl+uESXRLeh8 tWw== X-Google-Smtp-Source: AGHT+IF/E1FLpmQ9hujXfDz8ah1g4xzG4PooG2XovTR77om7ynRlxwPBbvaTMprv3cmEsfAU/D0wit+tFCw= X-Received: from pfbeg17.prod.google.com ([2002:a05:6a00:8011:b0:724:edad:f712]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:cc2:b0:71e:55e2:2c58 with SMTP id d2e1a72fcca58-725b812e375mr6388253b3a.15.1733525538875; Fri, 06 Dec 2024 14:52:18 -0800 (PST) Date: Fri, 6 Dec 2024 14:52:03 -0800 In-Reply-To: <20241206225204.4008261-1-surenb@google.com> Mime-Version: 1.0 References: <20241206225204.4008261-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241206225204.4008261-7-surenb@google.com> Subject: [PATCH v5 6/6] docs/mm: document latest changes to vm_lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 4260E140002 X-Stat-Signature: fpx69kf78k6kfb97qrwrteg5jy16gb89 X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733525528-126516 X-HE-Meta: U2FsdGVkX18fw0mpl4SkZsWOOnfi5la4UtgVK2Px9BUT51qO0oC/PX72b3ils7H0c1DK+artprCP3bxZK+nSd2naueLCXA9cpcP0+KPQmbR+/HX8qUQbef0saDs8odeyFf90Tv8XRoTZiRcitmNiBj2jgehUh8020eZs6U20v2pBA5vnYd+KD6BlFj++NTVBH9b/BpfKz7njGf/P9MWy9YH5giILBoMZGSzg9pxPrFXhijvcgUAETFaJYSFvtbAPJLMnGbujQAhqmU41kxt2DohPaU9fE3nnGieyHkPRuzLH+Pdz77ozKv5UgK9fqLk4W5wlfDiFbfE3CBHThGcyQGs0t8NSrbJBFGCCHliaGcL4AkEelS/dfMRIBPjJr3uJEMGODbXo+w25k1KX1ajl/zbWaPFVcVZUjRxaxXE2APWlMDwwbdvLx+tZPr6NW/KCk7u7sX80pTHg03zLd7A+Gs8ATDnhI6vKZfs0WTtIDoFj7v6Y3oVVsVl+00OlDKwCHXqxygEU+Aedi965a928+1iGMntZDq9klwRzS1gK/7OALRWujEi3qnSOSVCj7BIwW2gqYyUcvBv5SAXXXysnVlYRColxk/pwaVyIVnDQ0EQqwoGqggQHMmdtxIbTH3xaaB5KbpVgj09TTFULZD1XoKDcStCKSlGB2ACRNjkCcpU7xz4665vkm/3zxi99UQnudndYJWniNuEIHpRmBglBpEkqIzBpFlsQ13sSl1gqzFqwteI3RPM5NpBxdxFvecIfgaElJPMCcGo5MeyddAhXFAzdHMlB3IAL/EqIYkdQbQQjBSUzGW35B6wPq7ekQbyOTYBUi4PLe1AsaAsGR9lZZlcLg3ybFA9vbRur7nxZpZ9YVF58Vgxu4xm4ZNbDkT7ov0COt/XBMHD/TU0MxZVQoaQ3jWdQ5mTttGNqMjZ86CINXJvmy+kpekTRZgcFZJh5jBkq6RmFu0VxzM+y2p1 XtErmadX 26etA1uY/Q+DxKajKLvA5MmARHqsvlMKb+MG3vrL3+piLP+uEZTbKzOnCwM1uVuiRyrByDobRMGPhawpC5dEgdX5VeZz17OQYq++kJtI/NVGuhcz33UVGReTEwrgSuy2w3kKShKQ4pSvCra1CFSLciW03rJNoqNdU4kNxH6BdyPPxYNMniC/WRLmxLCBVz2Ys4weq+NqGypXL/YMgnJpfNoXokxPsjKlGM1CwrlGzgQbCRsDH78ChLgLdhTd1s7W+bZk1b46HRXe/6akAzDNH3ni65uF2FHwuRuPv+cDV29ViDqbWXwYb7NLigVyg6zS9VbDgYosJwWtq5h0JdU5dbHCVKBYGJqezkYBlCW7+O1BgiRYOVo4GS/7yuwARJ6uh6FxFPD2du2dotUwNVvpEQKizPBfHKahyvsPvt0IlFrxVoG/OIaGhpQHwBwTAQCw2AswfQ27+IfjfBuip17tl+XlrrqqyHD9KOXV5sgj6xHu5NNW2ji9nP0+6UMlHkf00eI+9YE1Hu2bsyA9wTPGlj++TlW8FOVPTr7Xf8jnI9ntwxqrGgsCv7mYGQ9wtHykL1N8rLydC/36O2Y2fvfddQZ/OlmS3c4RyNNqJMpiLI5YxTiA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.003399, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the documentation to reflect that vm_lock is integrated into vma. Document newly introduced vma_start_read_locked{_nested} functions. Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes --- Documentation/mm/process_addrs.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/process_addrs.rst b/Documentation/mm/process_addrs.rst index 81417fa2ed20..92cf497a9e3c 100644 --- a/Documentation/mm/process_addrs.rst +++ b/Documentation/mm/process_addrs.rst @@ -716,7 +716,11 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`, before releasing the RCU lock via :c:func:`!rcu_read_unlock`. -VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for +In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked` +and :c:func:`!vma_start_read_locked_nested` can be used. These functions always +succeed in acquiring VMA read lock. + +VMA read locks hold the read lock on the :c:member:`!vma.vm_lock` semaphore for their duration and the caller of :c:func:`!lock_vma_under_rcu` must release it via :c:func:`!vma_end_read`. @@ -780,7 +784,7 @@ keep VMAs locked across entirely separate write operations. It also maintains correct lock ordering. Each time a VMA read lock is acquired, we acquire a read lock on the -:c:member:`!vma->vm_lock` read/write semaphore and hold it, while checking that +:c:member:`!vma.vm_lock` read/write semaphore and hold it, while checking that the sequence count of the VMA does not match that of the mm. If it does, the read lock fails. If it does not, we hold the lock, excluding @@ -790,7 +794,7 @@ Importantly, maple tree operations performed in :c:func:`!lock_vma_under_rcu` are also RCU safe, so the whole read lock operation is guaranteed to function correctly. -On the write side, we acquire a write lock on the :c:member:`!vma->vm_lock` +On the write side, we acquire a write lock on the :c:member:`!vma.vm_lock` read/write semaphore, before setting the VMA's sequence number under this lock, also simultaneously holding the mmap write lock.