From patchwork Wed Jun 28 07:17:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13295270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0B09EB64DA for ; Wed, 28 Jun 2023 07:18:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12AF48D0003; Wed, 28 Jun 2023 03:18:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DCA48D0001; Wed, 28 Jun 2023 03:18:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F33358D0003; Wed, 28 Jun 2023 03:18:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E50488D0001 for ; Wed, 28 Jun 2023 03:18:07 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id ADCCF80CAC for ; Wed, 28 Jun 2023 07:18:07 +0000 (UTC) X-FDA: 80951302614.24.2E8729F Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id DE907100008 for ; Wed, 28 Jun 2023 07:18:05 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=p74+Uk0+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3rN6bZAYKCAUxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3rN6bZAYKCAUxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687936686; a=rsa-sha256; cv=none; b=qhkc+XGUjecsc3mTz9VmiPlIOz42rsse49kxNvBxI5BaVZbSfThJJ1mxiTwamyU/rQjnxX lVUyxNma2G1AE1kpAnPSJuv8M5neE6Z5Zc9RQ2v1feZDTCrJj61aevHMuwFKZW2CGtPd43 XWZi88moYm2hUPIFo2NLmYyGI4chb8o= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=p74+Uk0+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3rN6bZAYKCAUxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3rN6bZAYKCAUxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687936686; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=gPRmZ26IBsiMsbW2DFKvtKQmDaUZH2fHIbrEB332iB8=; b=pZlmdCaUTdWjlwUeTNESDDFcDVOoE4VY3q3duSKKnDKLClxyJy0MDKX7WSXrVA9kmKzB2W sIfL8JerQjlvW8JC/hUw8/lv5eSwgDosKS7uq/0hbDLm6JhiotBYp/g1c1oI3J5plq53Ip nHSBBRuE24stdYvmVCpHXCeI2g65IsI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-57003dac4a8so12872177b3.1 for ; Wed, 28 Jun 2023 00:18:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687936685; x=1690528685; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=gPRmZ26IBsiMsbW2DFKvtKQmDaUZH2fHIbrEB332iB8=; b=p74+Uk0+mievmY5sZYd+LOEEzQXCDxl0mmM922tNO4gFzK9VQ/B0Skz+U/97u0n/C7 pOrcnHNj7ORHjmEsfhuy7GKlykDD2sPss82TIhTAV/es2nhbo/8j6S96CM3gffdieF24 Bbr/X5zYZlwy2Db68vucdP4XjTO7DI3bZu6VbdJGVNnlqnrCeoah1MqzTTQG+LWlwzgF iuni3QEwQx3w495PnU/ZWYvING1HuingnRmkL01FebplcWw2BJtTvQ8E/iGk16hurnPX TB810Ddbn+iXzyjV3Ao+0WqI/M6Z3z7H89kOKH4C6lc6Zz7/25fEvffm0VfOYhhzFvnP A4Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687936685; x=1690528685; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gPRmZ26IBsiMsbW2DFKvtKQmDaUZH2fHIbrEB332iB8=; b=Xo9nH+gZyFRf1z53fHiwRJ9PQc1JplObgHY3shFYyF3XCw7zGrtVgXOJvXaRsJUzRk RkQfdpITgVGqo6cKDaqWsKF+W+oLGOwzVVHK0umIOLf60WGWJ7eEf4QufO99xsMP7wak DhfAB26dyHo+V/K7N6jyskg73BYcIa2cFEyL8GAaqRsUjMOeMp4e6emfZcI3kwYGxZPp fmX6qQPB9Nkv7s0Wf6vJsOAno596iNStNV+IFef7ZcbzzHFAIJYQzzDdqAVq2e6xpmbd eut240BfckF9B6BXUBhDaCTl5rA85AqVClE2i8k+zqwnpg+TgItFl2kjlTN5P92mX/be GLkA== X-Gm-Message-State: ABy/qLZyCXsDXuqTlb/2jbbGSh9gMFe3HJWdCHp6lJ82Bg9is3iWW/YB DhehAbNynG8q6XmPHRuUp/unK0qa06w= X-Google-Smtp-Source: APBJJlEpYSp8he1DXAXVpI2wIb6u5ggZR99ctstsOjVLxfgRvzgDR0H3VpIWbi4McUwp9/ln4tUqhFGOd9A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6664:8bd3:57fd:c83a]) (user=surenb job=sendgmr) by 2002:a05:690c:2f82:b0:56d:5db:2f07 with SMTP id ew2-20020a05690c2f8200b0056d05db2f07mr7658ywb.5.1687936684846; Wed, 28 Jun 2023 00:18:04 -0700 (PDT) Date: Wed, 28 Jun 2023 00:17:54 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230628071800.544800-1-surenb@google.com> Subject: [PATCH v4 0/6] Per-VMA lock support for swap and userfaults From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, hannes@cmpxchg.org, mhocko@suse.com, josef@toxicpanda.com, jack@suse.cz, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, michel@lespinasse.org, liam.howlett@oracle.com, jglisse@google.com, vbabka@suse.cz, minchan@google.com, dave@stgolabs.net, punit.agrawal@bytedance.com, lstoakes@gmail.com, hdanton@sina.com, apopple@nvidia.com, peterx@redhat.com, ying.huang@intel.com, david@redhat.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, viro@zeniv.linux.org.uk, brauner@kernel.org, pasha.tatashin@soleen.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DE907100008 X-Stat-Signature: 4wyi6dra8y7qfytxexf4o6t364axx7mb X-HE-Tag: 1687936685-630158 X-HE-Meta: U2FsdGVkX18BJFGTtPEVWAPQRG1vEs7BHlMFTBF45K5+H6PnaEDeyAA+2xSwzjyksVOj7Dg2H8EnTEAlG5mSZ31h1cd3qjHBjhBA51OsZ+S8T4rYobqoR8eKVXU4qKksmX+GfpFuviZW/wKbtYTEfDfuvmjA/FNZjnHdVV0UPyXlyiv8ASmBBKxAtYjXOoo3VdDs8DgXhekLX/hpJClWDkIsUcvXwyhxlmfmAGBaLdrsXVAaN5QLMZ6pCTfUo5iYZWciTyqsPceybefd25p+Iatf/Bg7SLRRtyuDF3XLC6RMy2cpobifWel1J1mveBfrDiN3mKUfY1jr71KkqGO7sjDb3nd1QUfn2i5ymYK9w9GkwUrw87x7C00w0SpQfp4mOXUPCdv3Hrd5qi9rm64EN3zQ7wMJjKv8NsLIHh1XmxuliAflnRtfsqDjmKY8t+soRN2dYszzikJqmQ5/OwL1a2Ov0lqgLR4i0MhWjhncfD8V9RmG8pZ19mEPCXE/C4hDDZi1KiDI9cvLjwOmFhOwENItESuHDPhZtWmy8511CTDP8FzQi1ttrutMRZi8rW4BriTj1OB9Gn1Qh4/dWBry3Pn767lUJq0zxIu2SB9XcK+Nj8G72YECfRXjlt7Ek71SPmfAheP4fqpEkap2K2Y5aGnLqdaRXNPsrXul0/TlX9YRgi7BGP2/O/kqm1MyKYccxEJU/2Pk36wdFIwr+XaAHxFUJJRhcRjmDEgwaa3m4PIzH9w6OOu1/dpHmN5talRyW1hCLUYI9MqzBVVEX5aQr0pNm5s1SOydjjOCvuIB4ERXmjpA+LtrRRvi1qOH+CHiRYyDKgBXwyJgwiS2v3rMGt15DJl3EG6/dBPfr8ML+CLN2RWCWM0VroQGqfCb6SyKDIPqfAF/Os6V8Ubq4f6mDzSN5w6Gzses58JAxtQWZWsktZho1e2s5+4akP3XZCEDoGvePW9fm0LreBLSwjh bF9OnV2O CHUuBtjXIZFjlyf5peSeRduv1Jy0R8pBCgup6pcKGxA6dtR5hkh+63X8+qN4L8gD9PID4crXJHa40NKgUzLd4LUStH06wDqTQtRcxcWm5SJ8PmIIZQDnq5rHRwBsgvJxR0T+PJh619neF0cWospWIPi4p7SakVmeihnUbKdjCI6pi63wFelppalh69u3MXDKM/JNh++XPJN5t5MZcekyRXknh7XAdeDe/xE9eJPSXeIF9/tNqiKsr4h+tTFkoo3BR8v+nVp4z1QuZBawBHRZWp8qHv0eE8DF/vTVWz0kAdsCiLGDsDR8C+/gxSsoBTPQdcA28592L8kdpvS+07biNuZ/fGpS9nd1BF1JXaMaiT39YaYbx8VpS/IRS9v3zpqvr9wqZknWYzrT5FYmWwllzWRXTpXUvp1s6F/Etvq/0Je9qfp61q+cttPZ/EP5Hi9WL+PWOzcG2+XUOQh0fZgk8P8rX9dHi+6u4v4qLkti/3+pmA9RY+/JM3RqQIF0wF7+NGVLHEXVpUVcY5/a1HhiYUExn+YzOm7uPcZEyEktGEmCMk4/27EldYCLlZJHG4qqAjFR++UVjVyMXPDMh2xvFpalX1d7Jf7hkN+Xa6k87PNc2/SJXCPqcAyJEnPt4OOfQr01HXA2CSJUmLiQ6n9skuSztR5Lgi6GHLWlu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When per-VMA locks were introduced in [1] several types of page faults would still fall back to mmap_lock to keep the patchset simple. Among them are swap and userfault pages. The main reason for skipping those cases was the fact that mmap_lock could be dropped while handling these faults and that required additional logic to be implemented. Implement the mechanism to allow per-VMA locks to be dropped for these cases. First, change handle_mm_fault to drop per-VMA locks when returning VM_FAULT_RETRY or VM_FAULT_COMPLETED to be consistent with the way mmap_lock is handled. Then change folio_lock_or_retry to accept vm_fault and return vm_fault_t which simplifies later patches. Finally allow swap and uffd page faults to be handled under per-VMA locks by dropping per-VMA and retrying, the same way it's done under mmap_lock. Naturally, once VMA lock is dropped that VMA should be assumed unstable and can't be used. Changes since v3 posted at [2] - Renamed folio_lock_or_retry back to folio_lock_fault, per Peter Xu - Moved per-VMA lock release to where VM_FAULT_RETRY is returned, per Peter Xu - Dropped FAULT_FLAG_LOCK_DROPPED usage, per Peter Xu - Introduced release_fault_lock() helper function, per Peter Xu - Dropped the patch releasing per-VMA lock before migration_entry_wait, per Peter Xu - Introduced assert_fault_locked() helper function, per Peter Xu - Added BUG_ON to prevent FAULT_FLAG_RETRY_NOWAIT usage with per-VMA locks Note: patch 3/8 will cause a trivial merge conflict in arch/arm64/mm/fault.c when applied over mm-unstable branch due to a patch from ARM64 tree [3] which is missing in mm-unstable. [1] https://lore.kernel.org/all/20230227173632.3292573-1-surenb@google.com/ [2] https://lore.kernel.org/all/20230627042321.1763765-1-surenb@google.com/ [3] https://lore.kernel.org/all/20230524131305.2808-1-jszhang@kernel.org/ Suren Baghdasaryan (6): swap: remove remnants of polling from read_swap_cache_async mm: add missing VM_FAULT_RESULT_TRACE name for VM_FAULT_COMPLETED mm: drop per-VMA lock when returning VM_FAULT_RETRY or VM_FAULT_COMPLETED mm: change folio_lock_or_retry to use vm_fault directly mm: handle swap page faults under per-VMA lock mm: handle userfaults under VMA lock arch/arm64/mm/fault.c | 3 ++- arch/powerpc/mm/fault.c | 3 ++- arch/s390/mm/fault.c | 3 ++- arch/x86/mm/fault.c | 3 ++- fs/userfaultfd.c | 39 ++++++++++++++++++--------------------- include/linux/mm.h | 39 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 3 ++- include/linux/pagemap.h | 9 ++++----- mm/filemap.c | 37 +++++++++++++++++++------------------ mm/madvise.c | 4 ++-- mm/memory.c | 38 ++++++++++++++++---------------------- mm/swap.h | 1 - mm/swap_state.c | 12 +++++------- 13 files changed, 113 insertions(+), 81 deletions(-)