From patchwork Sat Sep 23 01:31:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13396495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F662CE7A81 for ; Sat, 23 Sep 2023 01:31:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 807566B028F; Fri, 22 Sep 2023 21:31:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B6C76B02A9; Fri, 22 Sep 2023 21:31:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 630B76B02B5; Fri, 22 Sep 2023 21:31:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 530856B028F for ; Fri, 22 Sep 2023 21:31:57 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 295FB160224 for ; Sat, 23 Sep 2023 01:31:57 +0000 (UTC) X-FDA: 81266135874.28.5AC5A76 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf03.hostedemail.com (Postfix) with ESMTP id 7290D20004 for ; Sat, 23 Sep 2023 01:31:55 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2V2Y2kYR; spf=pass (imf03.hostedemail.com: domain of 3CkAOZQYKCPElnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3CkAOZQYKCPElnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695432715; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CmlRLzvRmqHtTQ7XfVaULJQdpUmF9g0LEd2VGVtMlKs=; b=mpvDDyeptyWdIPayvPgRQMMbj3HP3tV4zlks8hVLaA6mGskKHBoFCe5xeQWh4dkA+vggoh 5X561TJZpI+o/omjel7uLV0E2FZbJF9JbXnmdXNHaU7z//9X3tSrxvrGrL+by729Joet2h u9A7MwYGh6EpvyryEVlZ3GqohOG98lI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695432715; a=rsa-sha256; cv=none; b=X1jWJHvUXuMVT7tVGmQdq5kFbq0CxHOfxDdkJr4mbkdyraWs4NsYbyGNYTgM5J/O0LPV3x W0BEZk69/+ZKXRyDeBUoBa5x5TPQ3LCFmWsGZfx9Lp4/lEeRom7/xSVjxrQMS6PZWDsEOq JFCh1NpBfiQ9BFHZCO+UTWHn+40eIB8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2V2Y2kYR; spf=pass (imf03.hostedemail.com: domain of 3CkAOZQYKCPElnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3CkAOZQYKCPElnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-59beb3a8291so47974807b3.1 for ; Fri, 22 Sep 2023 18:31:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695432714; x=1696037514; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CmlRLzvRmqHtTQ7XfVaULJQdpUmF9g0LEd2VGVtMlKs=; b=2V2Y2kYRoXnGCScjOuJbyz3c+CzM6QxIvMn1JeX0IsWw4CmR1R/nj5hJPV8uwJqgJV pdQi2HH45+khpY05oAbI9xVD3iFddEW3c0LHfmO764HlYwjyu9M4wQGkC01vHdg1HHY3 y8fssxodpgh577KeWjDKtOPuuJjWtd3uZOSmryjLDTJ+lruji7ycNStLMMj5U9gpS08c 0baEWpFWePbTqA3IvURNilW/ZfcrnZEzF4S3QTugP3akL7I6/oWG7sJfqEw3EGFtYcAU x6eXB8+9nmvbbcJWK0ttk2IOoZ7jtqhmKebMQGv2CcQ0s90DB+45acaeShZ3Qhbv8oyd nJSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695432714; x=1696037514; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CmlRLzvRmqHtTQ7XfVaULJQdpUmF9g0LEd2VGVtMlKs=; b=w0rpLH8s3DFZApa7oUcoLKhEUcbP+LvCvxJhmda4oJ2eQgy4qLoHtrHHhXOmd+Eg7y oTr9uXypfQUSQxgjNvu78nVpMsm5hs1v3NSUA2t8pG3XV3Q9fjaOd9qDB3kzrzf80DJV BtdtNL5fR0W6/6MtSWkAh30hfR2bQ3q/kBbc2O+c6tHnmN6O5EOGrTmHgaXFc7K1Y12v zK7BnVaGIWtB39c8IC5cRpJf8sHLVtUCMEmJxyWfWobqCrp20TEzP3WWSusbFuckLYgd supD1O/CyFvLGNcckAUUpu2RX687jfk1hYdTnx8Q7FD652RsoYLxO6REMAS9Eo6U+dZM Y9Fw== X-Gm-Message-State: AOJu0YwbY1Tel9Xvb5Hj+qMTxryTD8LFYayFs0T6OKsnR8esqFLvA2f0 /fWkpcS1iFuDU9pucwzWXGXqrEnctvM= X-Google-Smtp-Source: AGHT+IHrox59qxxuPIk8Ga14hkjCpjYszzofG63cVT9NaC90L+xjL83ej6dG0sFL+ZY5m2NWMQmGlw9XCjA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:3398:6644:ed7f:bec9]) (user=surenb job=sendgmr) by 2002:a25:cb0f:0:b0:d7f:2cb6:7d8c with SMTP id b15-20020a25cb0f000000b00d7f2cb67d8cmr11540ybg.13.1695432714541; Fri, 22 Sep 2023 18:31:54 -0700 (PDT) Date: Fri, 22 Sep 2023 18:31:44 -0700 In-Reply-To: <20230923013148.1390521-1-surenb@google.com> Mime-Version: 1.0 References: <20230923013148.1390521-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.515.g380fc7ccd1-goog Message-ID: <20230923013148.1390521-2-surenb@google.com> Subject: [PATCH v2 1/3] userfaultfd: UFFDIO_REMAP: rmap preparation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, shuah@kernel.org, aarcange@redhat.com, lokeshgidra@google.com, peterx@redhat.com, david@redhat.com, hughd@google.com, mhocko@suse.com, axelrasmussen@google.com, rppt@kernel.org, willy@infradead.org, Liam.Howlett@oracle.com, jannh@google.com, zhangpeng362@huawei.com, bgeffon@google.com, kaleshsingh@google.com, ngeoffray@google.com, jdduke@google.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@android.com X-Stat-Signature: 1hd9gim735q7n5tr8x9b99o4tfbccraa X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7290D20004 X-Rspam-User: X-HE-Tag: 1695432715-214781 X-HE-Meta: U2FsdGVkX18g4QrpS8KpUKQ8uW+4lo4XBpSZyqUoVgsVgOAlpl1QaCeKvFbxT9gKMDsJf6/9/XPx5xJSehETCIrCac1mGglRs1LAGYu0u+x0YoB5hObevFoGhuB0qjS4a7atQpfJjMbpqpqel940X7VZZY2u4L0jhC5GPN/P8Tnc3dFsCRE9Fg2H8IbqduEtcjLjY1hN3R+r1DetZmpyGqcPde1skP443L3Xrd8io4gb2zaIQTZsYf3SQKGH7NOhkVsnHV+0CO3Ms80fJVtFqbJYUh6L+iclnO3OC6XiVg+B6r6jxWasauIeSe5PjGnUXbk4DSX+Th4L3fU8cifexd6yp6SYifF1wN/Rn3eN6YlzQr0bFn+IWywMDc0pbWKrXKVkMC9E8lIoIEje74LbBxAJsvGCqJ60KkWJjZAwnDTnsdEqSxZ3mgSvwIF+GQ/NCVbJVwzBZnNi81lOlw8/Dcs7CMsL0NbF9N2bdkRV0FxfJ8Y1jEh9Nv9XIxJ7XBnwSJil8fifrmRAem1c0O29R4rBfNgot4V++NRGbOdsSUNKaIqQhVRXUGxocLwA6PskWABBltAVHXzSSnMPY/VJO+pj4YNqcjyEeomyo/UU62sQ/ePDN+oF+cskrPnwdFMcm15flWvHPr39FZ+Ybbm6IFN9lFFZ1NxomiTIsD0pI+uQY3FqQEY/m0UTH1hcVIWAZvgv2QPUXFpfgrZUvCqz9Fd+BYuyUd5FoomeAzutmsxQ57ru6r9TFwhsF5/c3Iz3yPM0FvmQ4cDNhIf9zGpLGYN2L2T/iLj1ACbu+OPkPWyaBU/yT/TJPLElhBzdQ2NjHIoVnaiOU0fJunrZx0v05SCrBNwckIx+p6MILE5xxFtnEtbYSGmlRZGxKfFwha8qUALMBVb7skZfhQGm+pCyuZRpHoZETCjcxGIJE7zSu+kjzsWFbTjW9C/F1FxFiUQCY838ywrabR4aicPkOoh yq62jDeF XHuI4XUBszkEvKVBJT4ViV9JNeJgJV72JVPZ6k2DNy/VT7e/MJ07lOTTV4vVY6ypeK6krLUxgIqT9QLeYSi0eCltcjRvPSMiDwBy96DtWlabg6qccYm0PJm6DbgHdS+mauXiCahgA9NmO68q7vjlfObA6tERiG9Az8uMSZ8zzNyDVg0FESfklEGrfd1iiFYWF+ZgYwotFBHA+kPUgeKM7osgT+7k2GeYdlQa2ctAHjdVUVnrTDrjlI9Og6oZO8MsrITKlrt78DcQipa+EgKTOk5k9/br9Uzg9KWUSoIKBTayISBBWxTvASgmmM6UXIT0wF7bdDR+xgdeK4AqyHg4n9Biio5Bq7O9bfBxNGMpJ+1BASOG3AObD1KpU/LzjZfLi8Rm6Fkwp9YphAPBMLOlHPE3W5VSFZDWqRah0KRy/NcLDRyc9an12Iw3198XwGkw16aH15bfGplapdp5VvgNkQfJbZ8Uk+U/RVZFLUmZmIrTn1YFojL3me9eGJ8WhaENzwqdYBJoQQMj8bzOped+amcOxdsF4Pbxv9RLs71+wzGr+i3c+gJZEQtXeTkD25lD3/FnFbhxsOqk+mg6PzfxC81O+lXeoC/+MqgbudfZMOINv948lgoIBA9+n1tyrxRrjgzHSgtVXBlvDbSAE7QMtat1XdinTFR2QpoYs/JX4HhRkwd4O/laW/ky5sg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000385, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrea Arcangeli As far as the rmap code is concerned, UFFDIO_REMAP only alters the page->mapping and page->index. It does it while holding the page lock. However folio_referenced() is doing rmap walks without taking the folio lock first, so folio_lock_anon_vma_read() must be updated to re-check that the folio->mapping didn't change after we obtained the anon_vma read lock. UFFDIO_REMAP takes the anon_vma lock for writing before altering the folio->mapping, so if the folio->mapping is still the same after obtaining the anon_vma read lock (without the folio lock), the rmap walks can go ahead safely (and UFFDIO_REMAP will wait the rmap walk to complete before proceeding). UFFDIO_REMAP serializes against itself with the folio lock. All other places taking the anon_vma lock while holding the mmap_lock for writing, don't need to check if the folio->mapping has changed after taking the anon_vma lock, regardless of the folio lock, because UFFDIO_REMAP holds the mmap_lock for reading. There's one constraint enforced to allow this simplification: the source pages passed to UFFDIO_REMAP must be mapped only in one vma, but this constraint is an acceptable tradeoff for UFFDIO_REMAP users. The source addresses passed to UFFDIO_REMAP can be set as VM_DONTCOPY with MADV_DONTFORK to avoid any risk of the mapcount of the pages increasing if some thread of the process forks() before UFFDIO_REMAP run. Signed-off-by: Andrea Arcangeli Signed-off-by: Suren Baghdasaryan --- mm/rmap.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/rmap.c b/mm/rmap.c index ec7f8e6c9e48..c1ebbd23fa61 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -542,6 +542,7 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, struct anon_vma *root_anon_vma; unsigned long anon_mapping; +repeat: rcu_read_lock(); anon_mapping = (unsigned long)READ_ONCE(folio->mapping); if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) @@ -586,6 +587,18 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, rcu_read_unlock(); anon_vma_lock_read(anon_vma); + /* + * Check if UFFDIO_REMAP changed the anon_vma. This is needed + * because we don't assume the folio was locked. + */ + if (unlikely((unsigned long) READ_ONCE(folio->mapping) != + anon_mapping)) { + anon_vma_unlock_read(anon_vma); + put_anon_vma(anon_vma); + anon_vma = NULL; + goto repeat; + } + if (atomic_dec_and_test(&anon_vma->refcount)) { /* * Oops, we held the last refcount, release the lock