From patchwork Mon Dec 16 19:24:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13910247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5258CE77180 for ; Mon, 16 Dec 2024 19:25:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9E646B00C3; Mon, 16 Dec 2024 14:24:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD5E16B00C4; Mon, 16 Dec 2024 14:24:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92ABB6B00C5; Mon, 16 Dec 2024 14:24:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7391D6B00C3 for ; Mon, 16 Dec 2024 14:24:56 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0F598C09C8 for ; Mon, 16 Dec 2024 19:24:55 +0000 (UTC) X-FDA: 82901799498.29.4E7460F Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 632B840017 for ; Mon, 16 Dec 2024 19:24:12 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ym6+5y6z; spf=pass (imf07.hostedemail.com: domain of 3g35gZwYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3g35gZwYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734377073; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t5fnSRoB5oFCIXa/V8gP+VbtOpDmgl8LzmUTLWBvDvo=; b=h99i5n8MdOuv41ScopfEB6hbaZRy1ZmaEiPONWvCZvZ0NXuVqivBDNylnKVypzaOF9gdpI rIFzVbyTLfyhaxgyPQu6oVCUdBfbZ9M1qZZuxp+UmjcZhGY4kkQPiC0Ooe+3M8Y9nW2r9H dlqU9mf7pMeYzv4Thg+P6kdMyuni+7s= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ym6+5y6z; spf=pass (imf07.hostedemail.com: domain of 3g35gZwYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3g35gZwYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734377073; a=rsa-sha256; cv=none; b=iamhA5wHNvR01xBcFpz3KgHyLHZC6PG0T9dgvFmfrck4xrm04cZfHZPmifnBPjSnhbMBpa L/NXPfVmIZ6O6fUuUqIG4y7wS7PM6RBhuFcng7J0vYgqhkN5GhM2oHu1xjK/VUBQna0T8E 53K/abmmffJdH2mvoL6tY9+e5jHtH/4= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7fd482c8de2so2885134a12.0 for ; Mon, 16 Dec 2024 11:24:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734377092; x=1734981892; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t5fnSRoB5oFCIXa/V8gP+VbtOpDmgl8LzmUTLWBvDvo=; b=Ym6+5y6zNkwI7h49+NgT3khcD63NCJ0PgPQXugEHvSN+EP2FO6CH0v8NVbGZwUM1wp LEbB1eV6jrl8z/p8ehxNoN1chh2g84hBz2RHiSajiS8HsZ6WDN9hHGwIlbD09BWpnqkJ FJQ3w/3hcoAXv2t3/K7SjB9v0gMbV7Fdp/sjsp1wI79gVJlsqBXuoYZXkJh/doPlF4uY +zU8b4KKzb7fMMICIlgFRG5ygwFDf700N9E9j0mHET6yuGeQIyEOvm6fCGHH4bYlLKnS T2PzXYpIIw0JLSG7Hb2FzvQksr1/0beAskgyVbOW47OGQcqxIIXTuoEm9j0j28BmnM7H pCWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734377092; x=1734981892; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t5fnSRoB5oFCIXa/V8gP+VbtOpDmgl8LzmUTLWBvDvo=; b=IO1apYi5bY6nAGirS3B52wv3owLUyIhyImc6WrOcHibrwSAKXwUJLEY5w05M45OZzK V8fvurYGErctoAxZmWbvGhu5dTHj685EZjPEmOvYI0SvJ50J2LF1d9gr9nwoStSBcTme JJt1WOzWqNSE0fjIgdArc+TSIuJICaxCKpodt8QSqLkHVpJiyzffwKyJcQMTkeJ/yhHF gKeW1BQzcAwC33qZJIFpfIP1jepkqwLsjn3ogU74TRKjBIVpPOFaabPbRDsosjBHvO7R cOTSJoKKgW/TfTXbYBi6gCkHiJR6shmBSSpQvExCrpk4EFimST2JozQ2LBh3iQ4zdUO4 huCQ== X-Forwarded-Encrypted: i=1; AJvYcCXnLiThdfuQdEaJr3BeEFlTTCunqy7eqCy/nqvkSmcntrFHtVjXBaMjgVPIQOtIfoZp5kZxTFGzWA==@kvack.org X-Gm-Message-State: AOJu0YzfMuDpm/AXLoaafhgFHU/fcbw4o+hHCweNaKqJEczAal82xH38 D1PMpxGBzqkYWS+O6MD7SFHk0DO++Rv4Y9bn7dfARSx0Np2ESbfHsnhCGSfg9cZ5mNYyA8xyaf7 xWw== X-Google-Smtp-Source: AGHT+IGeGCIjsB3KXvsuHWb/nocj10tzreW4ZVMNCmw0sPpWQ78cfKvuNdThF2i3rD3x89K8u/6hh2RJEJY= X-Received: from pggk15.prod.google.com ([2002:a63:d10f:0:b0:801:d768:c174]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:8885:b0:1e2:5cf:c8d6 with SMTP id adf61e73a8af0-1e205cfcb77mr10415260637.36.1734377091912; Mon, 16 Dec 2024 11:24:51 -0800 (PST) Date: Mon, 16 Dec 2024 11:24:17 -0800 In-Reply-To: <20241216192419.2970941-1-surenb@google.com> Mime-Version: 1.0 References: <20241216192419.2970941-1-surenb@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241216192419.2970941-15-surenb@google.com> Subject: [PATCH v6 14/16] mm: prepare lock_vma_under_rcu() for vma reuse possibility From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: peterz@infradead.org, willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, lokeshgidra@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, klarasmodin@gmail.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam05 X-Stat-Signature: qj51h8wbwzkfopkt6ye6rxah6dmjjjbu X-Rspamd-Queue-Id: 632B840017 X-Rspam-User: X-HE-Tag: 1734377052-157086 X-HE-Meta: U2FsdGVkX18tdrkVYm8KYT3JXiz+kv52cBeuSbA/k5UPnBdc1td5O3v0ynBq5uTqCrpOWzYlhutCb8DiPMvtWZ8sPBi605V7Cq6L9mq1m2WFH1NoMSEqeZ856nCk6bqIAVwGjoG1T+Fpk8JyZ/DhPq8F+X/KnrBYDzmdXzg1H1SuvKR2iFkO4n4CoY1J1GrNld+uk2IYFuLN9TaHax0tmDN1dAZhzrz9MJzw1wByx9ez0e/cVx6EoGPe9+dR4LE7TTRZ98HkHn50leAhYMq+kF6uabmnQN1pkfUARX8hKmbJmW98XOg3Jbp2jIqhjsjQ+/SZPEIMTeBKMfRzH7+w51Fh8ykEjiB5X2BLIzbbBDn9TSzPwfvHZkGN1FhvyFiy24f/2cgcPYRIQzcNqvfQRlUC9ok3GUuy13nkKQ9GY6EHPGFsXBKTAXrJZV2jb3ZUKzT69CKzNCy65MEgPK8/w3p66y0581MX0vMOgI2pvB0LWddzjc3q0OeU4pnZHeqE2ZR3LyqTtl7HStcb5UawjlT0wS0D165WKHOzQOwzMrMBpPPkJSROI8a23Vx7I+oU95A+ExMKV65ejZTA10PQ3Qsr47jHdmh+mr9STkru/tFNqBzlJ8yUvxhQvvESpPNZeyZlg7q73sAZtszY7UD+w0S5f06VgaxaUpw2HLB0rxP1sv6W+JZr1ht8cj9xW0U2PRjzL5WbRl9QwkokNJP63s0lRSpGNps/tRF564pLUZxisMNBd5Tz/x+rNyPofmtH45ZXTHnGDGsLxDddDhOWaX1WIr1EeAlXmZywZse+hBhl9GzGSeTGNHLtpR0iHlPjK03SUYVQLMmRZOIAniy8X8pPEErtwuq2Wp1cadV1QSGfDmHZZbLaOqZw0HMRTLMecl7lO+Aa8kKxIZ6AqJtp+PKoald1PcoQvnu8v0LEC/zKvCfNJ0tahtli5nGhHykAeXMcq99LD2dAelAw4gQ rvuez/Wz 6yOI58JgJayVB2+ie/TliU/TPoimeWfZlc/m+nEhYQqnUhgx23VyZmLevbSqMaZAea2FVdjaJucLQebFmDGpUmuFIDe9V3PZmHNS9TYKJFFbLzYWQn0VQ7UwIEgpye7Fhyuwsi23hbcYYpfURmhN3aY7nCLXLGDXu+FbptML8AE8/BjTbGxvRd6P+c1z8eCe5bH5eIr0LihA+Oz0JTpnniSFJDoDAHCZLuGVdcMQygmQYy1h3ljCVZByPnpUjR5tafzX5EOkGbDAFLid3fJTiJNkxO6/1C0y4pUkAwTpErZU5JHCKQb0/oigwIZZ82FeVz4AX4UIliuckSLe+xnuVlyhNsg2W1mr5wBCsV3mhFqosZMjGOiOk1LPU5Hosgkabx0NI07X4abIKbQyMUBelpANc+ZWySwIa491CtUXV5/Q3IDShiel6pHgKjwvJCGH23RQPl/6JWt05BjhlM23f6GYT3fSVcVv6ABt5XWYFE59LkyN8S9Mvv4Ez8Jmhi5yyawCnw0irOenqcYN2Rfr4TvLHzPqxrOBfMEKZ/dxdn6C2RGAG5Rt5t/VDfftbsqCc/oeFVmi6qR07vGxA5HMC7/mK4g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.016091, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Once we make vma cache SLAB_TYPESAFE_BY_RCU, it will be possible for a vma to be reused and attached to another mm after lock_vma_under_rcu() locks the vma. lock_vma_under_rcu() should ensure that vma_start_read() is using the original mm and after locking the vma it should ensure that vma->vm_mm has not changed from under us. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 10 ++++++---- mm/memory.c | 7 ++++--- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 361f26dedab1..bfd01ae07660 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -725,8 +725,10 @@ static inline void vma_refcount_put(struct vm_area_struct *vma) * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. + * False locked result is possible if mm_lock_seq overflows or if vma gets + * reused and attached to a different mm before we lock it. */ -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { int oldcnt; @@ -737,7 +739,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ - if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) + if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) return false; @@ -762,7 +764,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (oldcnt & VMA_STATE_LOCKED || - unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { + unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) { vma_refcount_put(vma); return false; } @@ -918,7 +920,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init(struct vm_area_struct *vma) {} -static inline bool vma_start_read(struct vm_area_struct *vma) +static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 534e279f98c1..2131d9769bb4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6405,7 +6405,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - if (!vma_start_read(vma)) + if (!vma_start_read(mm, vma)) goto inval; /* @@ -6415,8 +6415,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, * fields are accessible for RCU readers. */ - /* Check since vm_start/vm_end might change before we lock the VMA */ - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) + /* Check if the vma we locked is the right one. */ + if (unlikely(vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read; rcu_read_unlock();