From patchwork Mon Dec 16 19:24:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C63AE7717F for ; Mon, 16 Dec 2024 19:24:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD8206B00C1; Mon, 16 Dec 2024 14:24:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C603C6B00C3; Mon, 16 Dec 2024 14:24:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD7D16B00C4; Mon, 16 Dec 2024 14:24:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8B1EB6B00C1 for ; Mon, 16 Dec 2024 14:24:53 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 476784508F for ; Mon, 16 Dec 2024 19:24:53 +0000 (UTC) X-FDA: 82901798784.22.F63E864 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 325C12001A for ; Mon, 16 Dec 2024 19:24:22 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=c8i5Dshj; spf=pass (imf13.hostedemail.com: domain of 3gX5gZwYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3gX5gZwYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377077; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eye0eVNbsQT4TepH1/hu+cBmomL+3TKzfNRnkVWyI/4=; b=S4KPLin/D/tMt0POj0mB4yGc2ABzyPTDAnaz5ZWA+cpUSCCSUNtp2PHH52NYAOk8VxIH5g +Ob3nG4BscDdKCGqEKHFwrf81O0YL4Ugvv5RwBLYfDDrOib9R+c6h7EHSxlw0G6kTeqd8T AtAcPSxQp0C6wX0x8uYcxTmwk0Dz+go= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=c8i5Dshj; spf=pass (imf13.hostedemail.com: domain of 3gX5gZwYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3gX5gZwYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377077; a=rsa-sha256; cv=none; b=pj6oDFMg58W93NcOmzTwt9fW9shlW/iDlWyCNSL0R+SVIBgBpIGU980HML8aY/jXshN+bP kg+5+/zcnVaitBwDAde6VBfZB/KG96G7wzezkZ3mrbeVi+CpUuAngysSepZCX/KmxnHZnp vTGQMfj0K2lRvgz67ghQEVJfWOzXUm4= Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7fdde44f4f2so2850292a12.1 for ; Mon, 16 Dec 2024 11:24:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377090; x=1734981890; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eye0eVNbsQT4TepH1/hu+cBmomL+3TKzfNRnkVWyI/4=; b=c8i5DshjvR8XtjHzAxhEmXs3u2TkCgmlj87RYy2sSk8Ty1vy1Ov6nkc+LjrD+izg6P KNRxkhflyrsZ/3wQvHMDKJc0Qt5HLYTDmMQQsM4iG8yZnXUbKySxBGUGYxFcP3P5bnRU hHG0DrswVc+mm9gK9S6HwXp/+R1Tlrt/zQgybaELmHg1gogEet/Xd5WnYm2mtatjRrez XsC9aOwHe2BcNik27wmUXqBTseMsPOMhmIcx4NVguoGH0SCUw6yH7XpyeO/0SGzbzlTa xW84YjrPcMWE/MPy208ZPqNI5BTPXTi9lZQ1+oB+88elEXlWOsJJfvcPiyNZd8K4dm6M Hpbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377090; x=1734981890; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eye0eVNbsQT4TepH1/hu+cBmomL+3TKzfNRnkVWyI/4=; b=hiYHJgsm9H9CrWn/8KWCxnaPtaRouBJPKK2GyPboJugoGNqCoc7+XocaRM+fYovsqP ZDOvAgleWZreC7TkBFTwqrcTnrOOaz2N9kRiF+AL6HzpgBuBQr1OJM6LKBLQfCBWJCF+ fQfGCKcj7GtlDlwUR6MVMrooCzZs4jivGgyj6GcH1+EXA1VAObqgdZOY39KtbfsShpgQ Ja4tw7su2Erq4z80Tw9tQzyNXmSaQEfgaHdgX7E2E9E4oyvR1IbwkauLx+oWirAg8CpC JBrWX4onYN8zIL9JsZ5NgZTMXdwEPlxxIhn7BhG6ckaBkvk6DcabW/ZaLMMKpi6nPQrQ PaMg== X-Forwarded-Encrypted: i=1; AJvYcCWY7dTAeootYBIhsYIhHv1mTOh1OQe/Yz4UmsTdlgKh60CrxjOn0omX4/8r1D8x6ktNGkw/TkWVJg==@kvack.org X-Gm-Message-State: AOJu0YxKCoUFbBIA6W+wEmQJAfCl8ZaT7ipLCWknabWudvKVRWAfSypq wcnTKCQOnih9cXn6euSzLYWChfSNdgiDtipZt/U0PyCtvOXBRR5yx+XQOQu/AgCO0e16VIDjjZ/ Z2w== X-Google-Smtp-Source: AGHT+IFpyvj1Cl0PXtve1a3hZL9RrpgDnkSC5MatBr/H5K08KaVaYCnO8eim/kS1J5eWUMkZBi1DYQsuwp4= X-Received: from pfbdf1.prod.google.com ([2002:a05:6a00:4701:b0:728:ec44:ed90]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:9f04:b0:1e0:b5ae:8fc1 with SMTP id adf61e73a8af0-1e1dfd3da76mr21067006637.13.1734377089941; Mon, 16 Dec 2024 11:24:49 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:16 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-14-surenb@google.com> Subject: [PATCH v6 13/16] mm: introduce vma_ensure_detached() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 325C12001A X-Stat-Signature: krciot9t5khbcdgajp7cxdufummtknqp X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734377062-805715 X-HE-Meta: U2FsdGVkX1+xwgSm6/S1LHEmoUPPtieh8P466p3iS+K6HkO96IpGPX4dW2adtmuR+diujZ/MGKh2wxRisD0+etcFayOP4ptCzkEkkQUddRI5YZw9HFjfohvRd4tiGjH6NT+NoOYhzghODLTYLIREBf73HhErbKDCDwjK7CVL2Y7LkLO0d1//vXCn/8yW+DK+ejoHvHfQOc7N4TmFL0BJCZZjbe/+VsvZsINpMZmgdSFm+K1mPNtkkjz202vGhVSOpKq6yzJGKl3igTO8VCTwTgQffLV9o4lAcgzWFMXzNFq3TP05SHTLt6nLowvCS9LYMXI59g65blUB1egYOtsDRGLvwQSFH2TXvdbEWrxAMN3yvSsgQqXTjG8CyAmtNUTSgRzwKNZQz/lcU9Oxm5nVPETjjvvdC3IlWQXMPJMQ+wkGZaV8I+tgKcgJmYFf/O49ZXIaCbwj7PGEGBbFjNnoIYvpHrtan8DgAYGUAeAO8baOfcyVi3AQPw9BDvSPbaqG/mEoL3Dtm+0qB9klzGVXHUwxEFA3y01ggXi0QDy+iiqbbWhxiMP7/8+5iFMUte2/3qRzxk+D9clp+WWzYPwhwjwLLeOxZByHAueQ6L4BcDh+2pZc6V2JolGirBKLM2HqXj3/dUSFa6HQ4CRwZeIxFbIwud6Q+0vs1pFTr2i8H64KeG8VhUALd2kz8IbqWDDBkn05VpToVwK+kLom3B6SZjX0mzQ6h7YyqhnachQ0eG3h9QngEDAg8X6cwuWw5G5CJkjs63fvHBPfLSBqJJ6QTyLOp6CYoP/pwccPrF1C508dZotT1p+dN8yIEbp1rWgR/ENOozgfnW7uV9tu9+iQn2/Yp6NfuspFofDXhtRwC/aXSILumEYSriydSid0R6FpsB++eWTDyTVlKc3GBsfnETPRuf3/NwUphFG8WnsRULCqYvkm/nSoU6QdiUzPoKp+bDbyiyct0PUJ64xm8eX aQYK7Rij zSyTtgjdggdwDTbAm8x4b1+eWMn5tmnxZ3LCUXy5Q43jhm+bKiDt22oKsnhaeVJ54aZ5RoM/S2of6mFx5OMS/IeE6o9qfc0ctyMwDJgCUy4nrqMVw3B0TOwuvd45lx1am2wlC68NoATGv6wleMb8Pel79IQdjVP7ceKcKERra4ket+4+i3eMsB9uVysdpuBy1vpKaoZVBnJBgxdCNFenU6+azWobdK9PICCJPDiwZEx2JKY6nkevZjtcept983xhGbBDoEZNifO8TV4ZJGmA82b/PKMC/KFsYgX/DPAYINkc3qKqozoJc1S5t6QUrmYphoFciB8oOP5grosXbke8jiMDgeGoHChL2T0UVxRJNZPAdSTIn7o7RCJr6NeYwbINQTF1Rk0eD0NsXrCo3D+i+GL/qoEs+lIym3tUqGDy377vVQayhZRvCja3Anl4bIgmije131FilW0I0CXREh/Cq/wPkUl5Rym3npkW0yS44y6UwWm5H5oGyNYuX/BwJFu61osRO8zODIlPjd85DE4bwiB4OhwtvUjOHr/crptkRBy9UI6EelHnteANYItY86k0iD68lNcSR9e/ghJ2LIeKi0UZluQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.005883, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vma_start_read() can temporarily raise vm_refcnt of a write-locked and detached vma: // vm_refcnt==1 (attached) vma_start_write() vma->vm_lock_seq = mm->mm_lock_seq vma_start_read() vm_refcnt++; // vm_refcnt==2 vma_mark_detached() vm_refcnt--; // vm_refcnt==1 // vma is detached but vm_refcnt!=0 temporarily if (vma->vm_lock_seq == mm->mm_lock_seq) vma_refcount_put() vm_refcnt--; // vm_refcnt==0 This is currently not a problem when freeing the vma because RCU grace period should pass before kmem_cache_free(vma) gets called and by that time vma_start_read() should be done and vm_refcnt is 0. However once we introduce possibility of vma reuse before RCU grace period is over, this will become a problem (reused vma might be in non-detached state). Introduce vma_ensure_detached() for the writer to wait for readers until they exit vma_start_read(). Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 9 ++++++ mm/memory.c | 55 +++++++++++++++++++++++--------- tools/testing/vma/vma_internal.h | 8 +++++ 3 files changed, 57 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b73cf64233a4..361f26dedab1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -863,6 +863,15 @@ static inline bool is_vma_detached(struct vm_area_struct *vma) return refcount_read(&vma->vm_refcnt) == VMA_STATE_DETACHED; } +/* + * WARNING: to avoid racing with vma_mark_attached(), should be called either + * under mmap_write_lock or when the object has been isolated under + * mmap_write_lock, ensuring no competing writers. + * Should be called after marking vma as detached to wait for possible + * readers which temporarily raised vm_refcnt to drop it back and exit. + */ +void vma_ensure_detached(struct vm_area_struct *vma); + static inline void vma_mark_attached(struct vm_area_struct *vma) { vma_assert_write_locked(vma); diff --git a/mm/memory.c b/mm/memory.c index cff132003e24..534e279f98c1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6329,18 +6329,10 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, #endif #ifdef CONFIG_PER_VMA_LOCK -void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) +static inline bool __vma_enter_locked(struct vm_area_struct *vma) { - bool detached; - - /* - * If vma is detached then only vma_mark_attached() can raise the - * vm_refcnt. mmap_write_lock prevents racing with vma_mark_attached(). - */ - if (!refcount_inc_not_zero(&vma->vm_refcnt)) { - WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - return; - } + if (!refcount_inc_not_zero(&vma->vm_refcnt)) + return false; rwsem_acquire(&vma->vmlock_dep_map, 0, 0, _RET_IP_); /* vma is attached, set the writer present bit */ @@ -6350,6 +6342,22 @@ void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) refcount_read(&vma->vm_refcnt) == VMA_STATE_ATTACHED + (VMA_STATE_LOCKED + 1), TASK_UNINTERRUPTIBLE); lock_acquired(&vma->vmlock_dep_map, _RET_IP_); + + return true; +} + +static inline void __vma_exit_locked(struct vm_area_struct *vma, bool *is_detached) +{ + *is_detached = refcount_sub_and_test(VMA_STATE_LOCKED + 1, + &vma->vm_refcnt); + rwsem_release(&vma->vmlock_dep_map, _RET_IP_); +} + +void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) +{ + bool locked; + + locked = __vma_enter_locked(vma); /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -6357,13 +6365,30 @@ void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - detached = refcount_sub_and_test(VMA_STATE_LOCKED + 1, - &vma->vm_refcnt); - rwsem_release(&vma->vmlock_dep_map, _RET_IP_); - VM_BUG_ON_VMA(detached, vma); /* vma should remain attached */ + if (locked) { + bool detached; + + __vma_exit_locked(vma, &detached); + /* vma was originally attached and should remain so */ + VM_BUG_ON_VMA(detached, vma); + } } EXPORT_SYMBOL_GPL(__vma_start_write); +void vma_ensure_detached(struct vm_area_struct *vma) +{ + if (is_vma_detached(vma)) + return; + + if (__vma_enter_locked(vma)) { + bool detached; + + /* Wait for temporary readers to drop the vm_refcnt */ + __vma_exit_locked(vma, &detached); + VM_BUG_ON_VMA(!detached, vma); + } +} + /* * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be * stable and not isolated. If the VMA is not found or is being modified the diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index b55556b16060..ac0a59906fea 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -465,6 +465,14 @@ static inline bool is_vma_detached(struct vm_area_struct *vma) return refcount_read(&vma->vm_refcnt) == VMA_STATE_DETACHED; } +static inline void vma_ensure_detached(struct vm_area_struct *vma) +{ + if (is_vma_detached(vma)) + return; + + refcount_set(&vma->vm_refcnt, VMA_STATE_DETACHED); +} + static inline void vma_assert_write_locked(struct vm_area_struct *); static inline void vma_mark_attached(struct vm_area_struct *vma) {