From patchwork Mon May 1 17:50:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F40D8C77B73 for ; Mon, 1 May 2023 17:50:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B6A9900004; Mon, 1 May 2023 13:50:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 565AF900002; Mon, 1 May 2023 13:50:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42DED900004; Mon, 1 May 2023 13:50:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 36671900002 for ; Mon, 1 May 2023 13:50:33 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E6083C07A4 for ; Mon, 1 May 2023 17:50:32 +0000 (UTC) X-FDA: 80742425904.04.7EBDEFD Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf01.hostedemail.com (Postfix) with ESMTP id 1581240024 for ; Mon, 1 May 2023 17:50:30 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=dZCcwJnc; spf=pass (imf01.hostedemail.com: domain of 35vtPZAYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=35vtPZAYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682963431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Vjmy9npoFS1ltTIQSU6SYKXLbdap9UrmwfWM+Y2ixPk=; b=zP/aJjT6ZmoGs7FHJFtt6MsWmBLuC6dXqnr/lTkFXHnyMMEjfPdA2KNwS6Xt2m4CwRXsJN 3jxsvolJEWfXQI8UOgZoJlmwOYmHwfM9GLHggbb5Zi41HWhlojVqZlwVzEK/duROdXr9qX E9Mhz/mDLfStSIAHZR9716RP7F30t8M= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=dZCcwJnc; spf=pass (imf01.hostedemail.com: domain of 35vtPZAYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=35vtPZAYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682963431; a=rsa-sha256; cv=none; b=fuOwIcuL1FEUGMdxSB/LIyDmZIwQMeKe1CUZ3ieopjerM6Ngvfayd9h68BBVSl2LiZ7yh4 Umkl5cTeko4CmQ6no91fMAJVc7lahp5a+mRDQYy8IQuTmZugewlaH3OhuCuChYorQU9MWB RpmrdpLjANSj31WsFAuh5CdC6T8qxMs= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-54c2999fdc7so54682827b3.2 for ; Mon, 01 May 2023 10:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682963430; x=1685555430; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=Vjmy9npoFS1ltTIQSU6SYKXLbdap9UrmwfWM+Y2ixPk=; b=dZCcwJncaHug3B8HkBJA8I4rogyge1EVTo4b3P8k7VcTSJVpEkICoBGsnhmO7W+EaQ BIxtz97RUXIfYWYlU1CJsKlwqGVGZ7x+vS+3P/rfb8ANuZUjR7SbifV/F6aSBN5z7ZRt /AMQ4KNfWlhXtFlPUDhe7iDHC11AWgJzvh0HYI/QSVDj28sAvGrMcNoB1jrqM2PiCRxV MKWY+ho9Y/9MhG5dJ5YP+l84eWwLxXehtDV3C+sB45yhtiKFovPsBc0ojtZnWpwnmUO1 OgbB8QClj+k71UJXHVn/U9WcLkhMORsNKUzA9GKVhmcWDmrnB5+OYkqWUJRZdkLT90UK DpmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682963430; x=1685555430; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Vjmy9npoFS1ltTIQSU6SYKXLbdap9UrmwfWM+Y2ixPk=; b=bzYnilJ/hRwdemf+rS2zppkgcmriEEo664oFMCrLqcalSZXog1Bx0xK8Sb+c3p9Y/Z coK/zQXb7f3Vci/mo17XPwgv2+7hQoACDVg0whDQ5NLxFpCytYYehCqv2OXuQ3XzSNQ+ dmbBx7Lf1pFgtt1xteny7xdgzrgNgKGzN7aLdhuvjeGPXqVQ1UrcS74g89gDbYYCnsKy G1od/RQTghR9+E5ooMdOl0yGslDGN2sEohJjCKHhsYL0EhKb4EgfOlLLt6SPNn6aYo/I klEphPDAp1GkdOb1lDxI8dyaWV2MACUjen8v5Lev5waPM+H0m/pdTBvgDlCrixl7zU8H yWug== X-Gm-Message-State: AC+VfDyc8KUzJEuktfOW3tvVYRfSUTjsjmNSVCsNRz7/nrFkmOcGThOO q83LXO6iErEjL81uwN82B/8euhu5WwQ= X-Google-Smtp-Source: ACHHUZ64NZ3/9tsMewuAy2AdTfarKe08wL+z4DEH/85FRjs3ZEIBqL9XqLFWxrB85BCLpG0FPVBpKygFEUY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:9f09:0:b0:559:e830:60f1 with SMTP id s9-20020a819f09000000b00559e83060f1mr4687408ywn.8.1682963430249; Mon, 01 May 2023 10:50:30 -0700 (PDT) Date: Mon, 1 May 2023 10:50:23 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501175025.36233-1-surenb@google.com> Subject: [PATCH 1/3] mm: handle swap page faults under VMA lock if page is uncontended From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, hannes@cmpxchg.org, mhocko@suse.com, josef@toxicpanda.com, jack@suse.cz, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, michel@lespinasse.org, liam.howlett@oracle.com, jglisse@google.com, vbabka@suse.cz, minchan@google.com, dave@stgolabs.net, punit.agrawal@bytedance.com, lstoakes@gmail.com, hdanton@sina.com, apopple@nvidia.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com X-Stat-Signature: aboti4d8h7fnpuyyru6qjmb1s9a7a6km X-Rspam-User: X-Rspamd-Queue-Id: 1581240024 X-Rspamd-Server: rspam06 X-HE-Tag: 1682963430-990805 X-HE-Meta: U2FsdGVkX19mUUkv1A+DaSg2hCrXzkiA9TAqCXDKjj1v40umuRQL2kGDx1HdaFhBAzbb/dn+COXWRnNx5H0Ds9KPP5sMo0lPuWP21OrwJkiRzPxGb3+uaPCOVQDEhc76Ywpjf7Z7B94h5bbJPzamuBIX+2s0Sggh3w/WTgjAe97qQ3ZhSdinCnEkYfAe8p6XCs0tWP36baqX/W8aSX2jg/QyXSUVW11ariGrSaUcsPe8ud+cEOEiCp6TgHGFc1ZEraYCMzMrx4hkp2RrWp+WuJsOcep4xbfsGACgdXYaP0KX65/B9zw9Vr690VCwdl0VuN65uzZuRtYJCI1Nn/TxBFJMLYe3ii+55mwx+zH5n8ZMt4YbmE/f/ZU/IWphN4D1WLU10yEUeKO8O8RofB/JcjzcsOEvyo5rXZXDAH97sgSjsuyvYa8vFto2y4uBrLSaMTmyuka0jzwRCDsYnrosd6hulLGvxe4QoEpPjM8xP3M6kQZYsMw4F29Udh6Oa/euIbVFClcpVvOdWSA1KuzLMjlMp+DZsN0np61EXA/YGSUdQ7Pq5G1ZZMs619JvVcvWBicfl7ilCsMQY7mh4VMWxJS1HlX5Nh007C4EzKi/Lte/CnrBZdkPHGAqyfV18WYkG5lZIOchldxgRA0dXcxunxah2ROEwh+4II1lPjaJwSlHxtbxdSn3NSbwwfR5GAlY+/wTtaN7AwJBqFLosh63Ex/ca6/RA6dwp940rDB9+8DhJGurdeVpPVrXFI3dp8OIqvibrdxE6v0hl6sA3RKeyW9NfVRL0ASep6BQ6NMB+nwXErluFKY4Q7XShbMK/2Slx+jikdw8k62iTLQ3i4Ap/hizjS+Ily4DCFdmv6RGkUyF/Ybymv4+oRUpggusWxEg1QeBuFWOrZX8QGrknDkv/eTrBij/d97cEnugu52T6QmmwJBh1344Iiw5QkpNGpEnp+TEew9dfFMUDpy+xzL 9juze8ON ITOw2fL4s3KFhFbKUHgzU6bXMlfUiMdv3gd57Hcow/i5xmUmpxZQCkZcKqmTUhWkQ2P3NfdvyJZ/QhtulCJMvu5aogvmfHhLzXlSA+RJkHrXRwgG3TWzdU+YqXqQzZN6jxh9Qh0U2Htv/Q4IEueKNR+rizJeZ0J+V4PALS42IlBhzR9f1CDImIkRcFDsJeAmWOQw8edPpm1ubVglK6T8kchf4ByMVEnFkX2l4l3u9L9MJXGMYKogjQVJ4tyrMf9kGs32g31PJh6inWqvBF5MWV+EJWxurCBlzYhodrfPcFIWBpbGvByE/CSu9gZZlxT7YVEfYA/1IOGEZr/fjGuXYzg3pvZKVmyk58/dhNFqb1+IybvWQol43UtZf8NAtDWbJ+R80KbHVaKKH+k0+MK9Kyh8q7rxbbtTutwlsVFOqTL4hMnE0NyBR9/rumcA7J//+yVHHXXg8u02NREz0dWoTCDSu0pehooptIoKW1hZZ+xB/oZCVPCRiRVkAomkWb1YeMKUqhXmahBD7LI296mw7W+mqx+Zqrobzntb7swuZizqlSDZWsQ+kAEajGlTkpzNVxX5lEH3jN4kJ5afhXwbAyUCq6m/tAOvY325hesDso5ZqHqU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When page fault is handled under VMA lock protection, all swap page faults are retried with mmap_lock because folio_lock_or_retry implementation has to drop and reacquire mmap_lock if folio could not be immediately locked. Instead of retrying all swapped page faults, retry only when folio locking fails. Drivers implementing ops->migrate_to_ram might still rely on mmap_lock, therefore fall back to mmap_lock in this case. Signed-off-by: Suren Baghdasaryan --- mm/filemap.c | 6 ++++++ mm/memory.c | 14 +++++++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index a34abfe8c654..84f39114d4de 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1706,6 +1706,8 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) * mmap_lock has been released (mmap_read_unlock(), unless flags had both * FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in * which case mmap_lock is still held. + * If flags had FAULT_FLAG_VMA_LOCK set, meaning the operation is performed + * with VMA lock only, the VMA lock is still held. * * If neither ALLOW_RETRY nor KILLABLE are set, will always return true * with the folio locked and the mmap_lock unperturbed. @@ -1713,6 +1715,10 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) bool __folio_lock_or_retry(struct folio *folio, struct mm_struct *mm, unsigned int flags) { + /* Can't do this if not holding mmap_lock */ + if (flags & FAULT_FLAG_VMA_LOCK) + return false; + if (fault_flag_allow_retry_first(flags)) { /* * CAUTION! In this case, mmap_lock is not released diff --git a/mm/memory.c b/mm/memory.c index f69fbc251198..41f45819a923 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3711,11 +3711,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - ret = VM_FAULT_RETRY; - goto out; - } - entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { @@ -3725,6 +3720,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->page = pfn_swap_entry_to_page(entry); ret = remove_device_exclusive_entry(vmf); } else if (is_device_private_entry(entry)) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + /* + * migrate_to_ram is not yet ready to operate + * under VMA lock. + */ + ret |= VM_FAULT_RETRY; + goto out; + } + vmf->page = pfn_swap_entry_to_page(entry); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);