From patchwork Wed Dec 6 10:36:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13481427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45DB5C10F04 for ; Wed, 6 Dec 2023 10:37:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBF826B00A0; Wed, 6 Dec 2023 05:37:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D710A6B00A1; Wed, 6 Dec 2023 05:37:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C11106B00A2; Wed, 6 Dec 2023 05:37:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A87186B00A0 for ; Wed, 6 Dec 2023 05:37:10 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 82339A0137 for ; Wed, 6 Dec 2023 10:37:10 +0000 (UTC) X-FDA: 81536041020.03.15B7687 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf07.hostedemail.com (Postfix) with ESMTP id BB1C440014 for ; Wed, 6 Dec 2023 10:37:08 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VIef0YmM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3005wZQYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3005wZQYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701859028; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9XWOv5VjGfqn4Ye4rb6vL096BWUdHwApYaIgHelXwhw=; b=bcuW2Fq7epTPq8wr6/btZkKch4LwyPaMLRpvMRotyznQIx7KlkebUa8UgrZav1IcFxBt7O GM1PuiGcvFASE0e2HsGTSgvcq7/a9LLXx32IC9uKblXLL5SfjvKrfWJ2q825XAqh2kfsrA dY2yhGQuBoSumOVYzEFRpHeY9IVWc6g= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VIef0YmM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3005wZQYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3005wZQYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701859028; a=rsa-sha256; cv=none; b=IS/6jJIWJpe1jxHJ11waVIlzx6uwGOQ93XK1Zdb/L1oQQQsbMWuJm6MXmdvjHec8MMZiqT CLu+1NgBQDcj3ay9Dh1OOenVbdYrRx6FkDSzEP9oswyB6Jg6XbYvaQt9SLdywnuUah2Wt/ D+jiVzU8AZ2xH8k6ic/JDS7WsEck1ww= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9a541b720aso6008369276.0 for ; Wed, 06 Dec 2023 02:37:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701859028; x=1702463828; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9XWOv5VjGfqn4Ye4rb6vL096BWUdHwApYaIgHelXwhw=; b=VIef0YmMjeSFqb/6fOL24v6C9EI7I2kM5V6D1uOD4KFIB9oHbvghykejLILpKvMON0 8JVFXRa+HtxAbEU+b+kfMWC16+aFOvb2W8g/xdHG+lxhJ7T9ibueMJsYNeI9rBnXtA8L sEgHUgH24MViLPOIPf8+pfg1POtxYLc/FmPtkYppJi6egSjEzIGn/JxqC0nB7Xb1RA9N nEeh6fLdniyZ5JUQFe0GkmPy1aURZuv0uCWWBlKKWiWqf/uSJeR/eAeV21O7wNggSS/9 VU2Blyy+xccasdIEG64INYp1l9wEjoSq0QLrxgbFX/yS32ladVK5Is/E1gq21tYbPW50 k+fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701859028; x=1702463828; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9XWOv5VjGfqn4Ye4rb6vL096BWUdHwApYaIgHelXwhw=; b=J7Tx7bBzCUUFayKqsxVGVKh2geAOnnE+0fEYTdBpgWsCExH5H6zNjQYYLGGObAC55w 6TZ4mvgH4+jYexQF30uT56HbMbYHnErKIpBQvfEWwBxst8FobVf5OK87qFf6sZD+4v7M ao3S+ojBkORvMgAOk3EF4DlZj875Da6+uHgGkuyFKtItUECXfSpXczDvVUfdSovUWc/T t3RiFScNLLLU6gP7wJjyBrvvInJNmo8bvIgVw25Z1uPwftxetWxv+QXWkZGWS+HWeELE BmzQqGOcia2E+UDitEL9nuJFnvHf4Z+wueAQwNd288JBiaOhFkD7hwals1DfJs4rXwA2 KTAA== X-Gm-Message-State: AOJu0YwGuQUJ1ZWvNT1HxLryrhDUZZKv7RlGyWqqHUVerdKZ9KZgOQKW tjoULW5wSkuVKvC4YnHziLXVPlUUJQo= X-Google-Smtp-Source: AGHT+IGCulkBJ+T0+9Lwp1vGAWeNFtPWyIYRPjeboNOQrA1z+wrU7Uo8XHJ4cQLDzSxZ9W28kHwF5PDUnGI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:73f6:b5e2:415:6ba5]) (user=surenb job=sendgmr) by 2002:a25:7690:0:b0:db4:6936:48b7 with SMTP id r138-20020a257690000000b00db4693648b7mr5580ybc.2.1701859027736; Wed, 06 Dec 2023 02:37:07 -0800 (PST) Date: Wed, 6 Dec 2023 02:36:55 -0800 In-Reply-To: <20231206103702.3873743-1-surenb@google.com> Mime-Version: 1.0 References: <20231206103702.3873743-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.rc2.451.g8631bc7472-goog Message-ID: <20231206103702.3873743-2-surenb@google.com> Subject: [PATCH v6 1/5] mm/rmap: support move to different root anon_vma in folio_move_anon_rmap() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, shuah@kernel.org, aarcange@redhat.com, lokeshgidra@google.com, peterx@redhat.com, david@redhat.com, ryan.roberts@arm.com, hughd@google.com, mhocko@suse.com, axelrasmussen@google.com, rppt@kernel.org, willy@infradead.org, Liam.Howlett@oracle.com, jannh@google.com, zhangpeng362@huawei.com, bgeffon@google.com, kaleshsingh@google.com, ngeoffray@google.com, jdduke@google.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@android.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BB1C440014 X-Stat-Signature: dzbwi376h8gj6j8zx767thi5b8upc169 X-Rspam-User: X-HE-Tag: 1701859028-979204 X-HE-Meta: U2FsdGVkX1+uSnl3gGjRDjjO5mHXGSGYVZOhAQAIL5sdZBEx3i2GBzBpIRV1sH4vesZTXefuRHCTEkcQohjIvR1kJYoTWmUvYxt9aNh9MskUkQ+fZ3ghhGaqtUtBZ84pYmNBhuI0We/6aWT8d+Nzr4w5mKK99qA9XiDxPCESuHxT4lP5j6Hw5zQkZqPinzCB3pLYxT9swMRF3Q6TeRcqcXCY2d1TCPQWTl/Suof3PyhEozmLjQ1NVq3ZAUSQTRfB40VsZRGDUFMJvcg/9yZIuNVut2Q5CNiqd9RSghmaWoRX6NUT2e5Ps8wac7tnziy2IaqF4v2rOcgnCkRZlvqLIQAPWnZSgRJ+04T0HLPpuSHA6IknMIwRYELqw6Nokez+RWMj0vYQALpFRAEzyJc07QwljugtVzfE2lnWnE0/ewlOIqFMJs/HiXxpQx23mOiJ9brpRlQGGNxz89UGFtmJl0Ld78cSQC5++7WeLlkBgcKzia/SFJM+Kc5twm7kmW/NL4bV0gsDP83U1mg/NEQ1HvzI03MN2+kQCep+T1T/4MjYhxIroNkjpS2rPLRs5m7kAyPkWUzwQELODH4dLqrlTMBNOIM6n1dAQW0CeVFJGpcLy5vtLWAGZlA2JhXg9BJNtsEYthsPmTgcyN8nfGhIfx38RmtoFnUdc/TB1SL3Bazpnw+KKm9n4L6Rnea62x8nvTMuq0Mtzj34Hoq04Xfhy0MxU06GjRxzr7zk1F1uctv8K/WCAQ/P8t3bizxpt3DfJOcyt8dMssjQ7cSSyWRoNC2sE6sxz467sbfs/UhAyEqYhyeYvn8GoIKbEgByXuEDI9I4iYgZzd1Aqz88ImvuVrGu3030sOo+9gaALBquSv5/pCzAdrWLyxo8x298WaWXoBf/mprv96/NJwyA4Oa8XDKDikhtNHfVPhRdyOZENrrOUctFjbxjs/wP3/7YxEukTDLXSctJ1JgHNef0DOx JgAmGz7D k2C+tE8asSLf0qOlMi/uN+0No/XHYceP0YYyokJ6LVuuawtOW473zt2WuZ7TP8GMYX2i43IVx10iqpPsES4HhVazGrkKrEatJrFulWZqOcgupYnSpmsWgZoZGkIsLn2ATkRdVAtO6ZNIUnpdl+MQFrixXeEb+Ajh0fNHvtr1P/a6LNVLa6KdKpj4nMIOqDWMIFuhFVlstfa5vcO0tQ7QpXqKAo3TwxJRmGpttZPplw31dt2pvQtLslgeJGvIkMc04DGbnUvlp1wwndy6y5BnNjjYMPvMddHIMWac7HdPDru0sDQ7D+poQp+GZhJG2hZWKstbxI9UrJzkPjGkAVrwwQCNLgZHljAYz7dqM008trA+31jJEOIQ8jRrOfX52iuVa16lK2VSvBG6c9/lpk0t75TfReSaaoiK6HzxpLOPQE0IgFhEUTIPHBA2lrFnE6ORyv6MuuFWYC6xVqajQsA66NKbClxChPlsTC6/ZbZIDamy/ZiEi3rzQ4emB3ZKiTERpc3klUkf4ELHdLbDOiF01jdlza9Qsj+orfrLaz3elfiGMyXw6LI+HNaSRjSDm80fCVkR5v5lLmvLyo5wjASg9748Tfys61peQkJmExQI9ss3gcxsntIXD+ogSYxmufk8OSLVfO0VeHjK7f8Hv1cUh0O0xGPEvx40kEHoNR27i3NfFrGtwaHwnboiyMA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Andrea Arcangeli For now, folio_move_anon_rmap() was only used to move a folio to a different anon_vma after fork(), whereby the root anon_vma stayed unchanged. For that, it was sufficient to hold the folio lock when calling folio_move_anon_rmap(). However, we want to make use of folio_move_anon_rmap() to move folios between VMAs that have a different root anon_vma. As folio_referenced() performs an RMAP walk without holding the folio lock but only holding the anon_vma in read mode, holding the folio lock is insufficient. When moving to an anon_vma with a different root anon_vma, we'll have to hold both, the folio lock and the anon_vma lock in write mode. Consequently, whenever we succeeded in folio_lock_anon_vma_read() to read-lock the anon_vma, we have to re-check if the mapping was changed in the meantime. If that was the case, we have to retry. Note that folio_move_anon_rmap() must only be called if the anon page is exclusive to a process, and must not be called on KSM folios. This is a preparation for UFFDIO_MOVE, which will hold the folio lock, the anon_vma lock in write mode, and the mmap_lock in read mode. Signed-off-by: Andrea Arcangeli Signed-off-by: Suren Baghdasaryan Acked-by: Peter Xu --- mm/rmap.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..525c5bc0b0b3 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -542,6 +542,7 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, struct anon_vma *root_anon_vma; unsigned long anon_mapping; +retry: rcu_read_lock(); anon_mapping = (unsigned long)READ_ONCE(folio->mapping); if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) @@ -552,6 +553,17 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON); root_anon_vma = READ_ONCE(anon_vma->root); if (down_read_trylock(&root_anon_vma->rwsem)) { + /* + * folio_move_anon_rmap() might have changed the anon_vma as we + * might not hold the folio lock here. + */ + if (unlikely((unsigned long)READ_ONCE(folio->mapping) != + anon_mapping)) { + up_read(&root_anon_vma->rwsem); + rcu_read_unlock(); + goto retry; + } + /* * If the folio is still mapped, then this anon_vma is still * its anon_vma, and holding the mutex ensures that it will @@ -586,6 +598,18 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, rcu_read_unlock(); anon_vma_lock_read(anon_vma); + /* + * folio_move_anon_rmap() might have changed the anon_vma as we might + * not hold the folio lock here. + */ + if (unlikely((unsigned long)READ_ONCE(folio->mapping) != + anon_mapping)) { + anon_vma_unlock_read(anon_vma); + put_anon_vma(anon_vma); + anon_vma = NULL; + goto retry; + } + if (atomic_dec_and_test(&anon_vma->refcount)) { /* * Oops, we held the last refcount, release the lock