From patchwork Tue Mar 26 18:50:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604894 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F037DC6FD1F for ; Tue, 26 Mar 2024 19:04:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71F3F6B0087; Tue, 26 Mar 2024 15:04:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A6BF6B008A; Tue, 26 Mar 2024 15:04:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51FD26B008C; Tue, 26 Mar 2024 15:04:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 424926B0087 for ; Tue, 26 Mar 2024 15:04:25 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E1812140D3D for ; Tue, 26 Mar 2024 19:04:24 +0000 (UTC) X-FDA: 81940116048.25.965713A Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf05.hostedemail.com (Postfix) with ESMTP id 108A9100020 for ; Tue, 26 Mar 2024 19:04:22 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kYWmMjF3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479863; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MPCeiBQ6Vuv/VMKCl+70rHoy18EK+MdrkqBGmpRbWL4=; b=Dn2RqhFGz/8rZ7khkjPmR/cECpJuE3TruzfyTbF+5vbTFqGmoCLnMoBlnjrg2j2cevgBkj imq3XNskNJpNJGMpVC2twyp5i2fYm1unaOdfq501BBTiCX7r/PcN/AVA5bakN0xrmpDc43 CuZbNwOCukLJgrAVkJHRkO9SMiXSwSc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kYWmMjF3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479863; a=rsa-sha256; cv=none; b=aGPXsaTiQwAP5sxE2gvLFYDIOncEabBfOskGxLbpn0vWwnYxBMZguyT+o5Fwv3lIycWdx4 qwmHlQxvE308J4aYYusHp3xUbzKKN92s7Ul3hhJPK6JMduCi7ZR7ypGomOUTMUAqjsAd5x yD4Pkod6ImUewJK4WbHzo5geu89EHaU= Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6e6b5432439so4576690b3a.1 for ; Tue, 26 Mar 2024 12:04:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479861; x=1712084661; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=MPCeiBQ6Vuv/VMKCl+70rHoy18EK+MdrkqBGmpRbWL4=; b=kYWmMjF3yN0Zx/JzGQ+KDvNK/MXO/ggsOtEg9cMVx2yn7SCvjEYP0cDnJd76NrjzT5 nOK2Gt5/4Tlp1C6kCHM4OjqEvqcx2qzlscm+giwYTq6/aSw9EKa67s+pA8njRPtZAeNp o9J4JJBvE3lC1jGWl//L14MzCAqp/md9YWqymv7fshLnSqwrzgbVJ8sBbh+xQ7PGSoh9 RrmkSr6QMHiNBeuF2uG+rNUIxZNZL2i0Qxk0JcDfN/GiXocb7uVcNgi2Fw0UJRsyZQMx mAzivC86xhlx7Xl1fQMb6zpVBg1/eIBqIq+SKg3uoZVmZbmVt1TVruVCRXCNBvytoypE ySlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479861; x=1712084661; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=MPCeiBQ6Vuv/VMKCl+70rHoy18EK+MdrkqBGmpRbWL4=; b=W9FwKu7wg/Cvze5gox3HLhWE5d68xig2dvDLQQbpojuZ7VODWNsH5PjhrmTJmzEKzV OS0FVR4ER0T+anEgqcQkMg7ChejO8Dz4NUPY/ImUBY7fyTqh3D+H4j81ItIdyPTkvVJN vPeaqlFUaipa1QJytaw3+2JsMthNX/ZWkI//d9Cjg34utlWoA6XCp5Ve/27PT3QFuSQu xcs+ztWgFqmIsisxOOdTtA7HSvM/6h2nh6JmnRRQwrzONtXqrok3hitbjaWrWVmvMBjD cZuXLXhAXou440s62zAqB//UB3gjLzQ7q8MRRvgli6eOtQlA1VOg3EiAfcikDZKkbiJ4 aR7Q== X-Gm-Message-State: AOJu0YzOwTbfOMEGeoDAFHyD+2GTdUR3WxmWydyM3reoS9umA+wqPG6U 3VXfV4NHtN3NwVbl/Z4zo3+o+/KRRtod80B8qrW1yg1Pe3Tsa8v3HQs8VMWWPlQk3ztv X-Google-Smtp-Source: AGHT+IEvuj2NywPp4edKIveEUgd69iy3TfxxMJh0u9KQ2EoAOriC83NY/O9bepwSOY7NB56Qt1OqEA== X-Received: by 2002:a05:6a00:814:b0:6e6:830:cd13 with SMTP id m20-20020a056a00081400b006e60830cd13mr2478320pfk.23.1711479860988; Tue, 26 Mar 2024 12:04:20 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:20 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 01/10] mm/filemap: split filemap storing logic into a standalone helper Date: Wed, 27 Mar 2024 02:50:23 +0800 Message-ID: <20240326185032.72159-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 108A9100020 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 8pt3d4bbpxnzma7eoaarq3ren6pcydks X-HE-Tag: 1711479862-678238 X-HE-Meta: U2FsdGVkX1/7x4VdLP90rNydVhgDGY111KTtZ/MHn4rr8rPz5My2a9RypZlpG95aI7LCIGKdimxJB4vvsjutCxnSGOJ7Vwqep/kPcz2UB4R7yg07IC4H2jM+4HEXTbRTZe1ONyNRZqQu1jjabMct/4Gpb7WWyYcdJvwHcLX5c+r6mu13qSDJVkXWJltjy5WBCnPazFiCzXrqcGmL6MZnLibuAVhuwrZommMZIYqk4W4+NMk8WIvaHZ2F90JNjqPxxneAyPa4MppKEcJKLHSDfKXxm2l3c4YJt5A3Ax1i9WqCwjitldVqFruv9aNlpulePCmARZRf33kI01I9CgpAOZO4LmDckgJHJlUpoXqmIBI6kMK4TSfH1wgGGSnkbDf1XV284ZcS4DntkeUUkJLuD2wDFIZslZa67kfS2S4nRhKco9EttDzRZXUSMymiCWH4MK1tDF3o0F1XOyXe07qkMwzY7ffP8u10oJ2xLKEUfdEbS7lJUQhpktj+yTYW7es9Fmkk+4DQ3T0UYXLwV0YSsHZvZrtkqD6/3sxdE+zQL6R/XlT3IMYBFE6xYDzpY2sPL+IbT9tFAH/p6kmLql10vADmRuM3Kcrv+Pdx87xOv7A12do2qQhU1saaUGv7bDuqQwiRxd7j2O3mkWg26bJ8Z1ewhi7XgEj6giMIdhjQaqQ0i6yo8EbmrPZ735KG4jjmT5Xw8XVG+QmcC/tOy+NBDOa0GDWZZkppZaiFmQxHJRHSZfG+CacoqsT4ROojsgx6ygu4eSlyWzhRq6tNWNRfS8C9A9yeA+5g5tzaScdPMPbpN072bjCUPP+aexutLc7BWmNA75qHA2gPhVzySuM372tjc7mJDrTKWvcFgeKbF1xGDuH93u/KIoEDFuD18z1f4+YkaMlA4gCgPsnETSw28g32wDofkbpDM1QsBCQEolOZjnsOdp3+/4GwcIpzoct5t6ycyLXcbBwzapCFcO/ UwrvUFU0 6TahXXvzUIsszj6T7cTEjb2CjjBU8QbmNo9uFVWVMidpCh8sD1wHYMA/BI3M25xm8JOH4cVgTbD3SL1iGm43vcHU9jF9B/mbV8BcooC9w5yZG93gwOmAE/AbR8cUianlCbr3zxMKzRocmpIm7TjGFtPngE1JpTEokDAPIuJbZbFgeal39QgnkqBaVSvQNg4m5rG3x0ku1PG4yiOQXbqSj0xO53MKy+AcdP3BtyinIajcVLhET93sGXgPOVHC0d9RuLq86skd//VU+xk9/10+94o6Vv29/TCHeIeKTcdiw+vsdntrpV3ODxcB0jX7YDN79uSvXGeeeLJ+E5ScEkEkX37+A4t3hGFZ2dkqrUQTKjBaksshkeIBBJJy/+K5YkZm8ZwurNfXV8NNEO4IOhxiJa7cJkhdAAWpFw0fm/k9LiXChwBcvGPlluJKmXajKQ2GMmWAiXqyeprq8PQhNYoFDG2Z0+E+M2Rr7YO58OuzWWDU9mdewZWZXmmfDMCXWeKHGCuIXoq2ojFmYl1RKaapQVfu+eDTj+X8+xVfNlNJbQs/KvX9fNmdjWi7VYQHwdMGO9mM8X2f1urX78UvWWM8o7wS7PgW5YtmFvZK7XfgmAdhBdgYAp1t3twneTrPPtTVeW4i+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Swapcache can reuse this part for multi index support, no change of performance from page cache side except noise: Test in 8G memory cgroup and 16G brd ramdisk. echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap --rw=randread --time_based \ --ramp_time=30s --runtime=5m --group_reporting Before: bw ( MiB/s): min= 493, max= 3947, per=100.00%, avg=2625.56, stdev=25.74, samples=8651 iops : min=126454, max=1010681, avg=672142.61, stdev=6590.48, samples=8651 After: bw ( MiB/s): min= 298, max= 3840, per=100.00%, avg=2614.34, stdev=23.77, samples=8689 iops : min=76464, max=983045, avg=669270.35, stdev=6084.31, samples=8689 Test result with THP (do a THP randread then switch to 4K page in hope it issues a lot of splitting): echo 3 > /proc/sys/vm/drop_caches fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap -thp=1 --readonly \ --rw=randread --time_based --ramp_time=30s --runtime=10m \ --group_reporting fio -name=cached --numjobs=16 --filename=/mnt/test.img \ --buffered=1 --ioengine=mmap \ --rw=randread --time_based --runtime=5s --group_reporting Before: bw ( KiB/s): min= 4611, max=15370, per=100.00%, avg=8928.74, stdev=105.17, samples=19146 iops : min= 1151, max= 3842, avg=2231.27, stdev=26.29, samples=19146 READ: bw=4635B/s (4635B/s), 4635B/s-4635B/s (4635B/s-4635B/s), io=64.0KiB (65.5kB), run=14137-14137msec After: bw ( KiB/s): min= 4691, max=15666, per=100.00%, avg=8890.30, stdev=104.53, samples=19056 iops : min= 1167, max= 3913, avg=2218.68, stdev=26.15, samples=19056 READ: bw=4590B/s (4590B/s), 4590B/s-4590B/s (4590B/s-4590B/s), io=64.0KiB (65.5kB), run=14275-14275msec Signed-off-by: Kairui Song --- mm/filemap.c | 124 +++++++++++++++++++++++++++------------------------ 1 file changed, 65 insertions(+), 59 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 90b86f22a9df..0ccdc9e92764 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -848,38 +848,23 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) } EXPORT_SYMBOL_GPL(replace_page_cache_folio); -noinline int __filemap_add_folio(struct address_space *mapping, - struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) +static int __filemap_lock_store(struct xa_state *xas, struct folio *folio, + pgoff_t index, gfp_t gfp, void **shadowp) { - XA_STATE(xas, &mapping->i_pages, index); - void *alloced_shadow = NULL; - int alloced_order = 0; - bool huge; - long nr; - - VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); - mapping_set_update(&xas, mapping); - - VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); - xas_set_order(&xas, index, folio_order(folio)); - huge = folio_test_hugetlb(folio); - nr = folio_nr_pages(folio); - + void *entry, *old, *alloced_shadow = NULL; + int order, split_order, alloced_order = 0; gfp &= GFP_RECLAIM_MASK; - folio_ref_add(folio, nr); - folio->mapping = mapping; - folio->index = xas.xa_index; for (;;) { - int order = -1, split_order = 0; - void *entry, *old = NULL; + order = -1; + split_order = 0; + old = NULL; - xas_lock_irq(&xas); - xas_for_each_conflict(&xas, entry) { + xas_lock_irq(xas); + xas_for_each_conflict(xas, entry) { old = entry; if (!xa_is_value(entry)) { - xas_set_err(&xas, -EEXIST); + xas_set_err(xas, -EEXIST); goto unlock; } /* @@ -887,72 +872,93 @@ noinline int __filemap_add_folio(struct address_space *mapping, * it will be the first and only entry iterated. */ if (order == -1) - order = xas_get_order(&xas); + order = xas_get_order(xas); } /* entry may have changed before we re-acquire the lock */ if (alloced_order && (old != alloced_shadow || order != alloced_order)) { - xas_destroy(&xas); + xas_destroy(xas); alloced_order = 0; } if (old) { if (order > 0 && order > folio_order(folio)) { - /* How to handle large swap entries? */ - BUG_ON(shmem_mapping(mapping)); if (!alloced_order) { split_order = order; goto unlock; } - xas_split(&xas, old, order); - xas_reset(&xas); + xas_split(xas, old, order); + xas_reset(xas); } if (shadowp) *shadowp = old; } - xas_store(&xas, folio); - if (xas_error(&xas)) - goto unlock; - - mapping->nrpages += nr; - - /* hugetlb pages do not participate in page cache accounting */ - if (!huge) { - __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr); - if (folio_test_pmd_mappable(folio)) - __lruvec_stat_mod_folio(folio, - NR_FILE_THPS, nr); - } - + xas_store(xas, folio); + if (!xas_error(xas)) + return 0; unlock: - xas_unlock_irq(&xas); + xas_unlock_irq(xas); /* split needed, alloc here and retry. */ if (split_order) { - xas_split_alloc(&xas, old, split_order, gfp); - if (xas_error(&xas)) + xas_split_alloc(xas, old, split_order, gfp); + if (xas_error(xas)) goto error; alloced_shadow = old; alloced_order = split_order; - xas_reset(&xas); + xas_reset(xas); continue; } - if (!xas_nomem(&xas, gfp)) + if (!xas_nomem(xas, gfp)) break; } - if (xas_error(&xas)) - goto error; - - trace_mm_filemap_add_to_page_cache(folio); - return 0; error: - folio->mapping = NULL; - /* Leave page->index set: truncation relies upon it */ - folio_put_refs(folio, nr); - return xas_error(&xas); + return xas_error(xas); +} + +noinline int __filemap_add_folio(struct address_space *mapping, + struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) +{ + XA_STATE(xas, &mapping->i_pages, index); + bool huge; + long nr; + int ret; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_swapbacked(folio), folio); + mapping_set_update(&xas, mapping); + + VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); + xas_set_order(&xas, index, folio_order(folio)); + huge = folio_test_hugetlb(folio); + nr = folio_nr_pages(folio); + + folio_ref_add(folio, nr); + folio->mapping = mapping; + folio->index = xas.xa_index; + + ret = __filemap_lock_store(&xas, folio, index, gfp, shadowp); + if (!ret) { + mapping->nrpages += nr; + /* hugetlb pages do not participate in page cache accounting */ + if (!huge) { + __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr); + if (folio_test_pmd_mappable(folio)) + __lruvec_stat_mod_folio(folio, + NR_FILE_THPS, nr); + } + xas_unlock_irq(&xas); + trace_mm_filemap_add_to_page_cache(folio); + } else { + folio->mapping = NULL; + /* Leave page->index set: truncation relies upon it */ + folio_put_refs(folio, nr); + } + + return ret; } ALLOW_ERROR_INJECTION(__filemap_add_folio, ERRNO); From patchwork Tue Mar 26 18:50:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7F83C6FD1F for ; Tue, 26 Mar 2024 19:04:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BAE96B008C; Tue, 26 Mar 2024 15:04:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 743846B0093; Tue, 26 Mar 2024 15:04:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 570AD6B0095; Tue, 26 Mar 2024 15:04:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3D6C96B008C for ; Tue, 26 Mar 2024 15:04:30 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9DC0E120E4D for ; Tue, 26 Mar 2024 19:04:29 +0000 (UTC) X-FDA: 81940116258.03.D383107 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf28.hostedemail.com (Postfix) with ESMTP id A6EA4C0012 for ; Tue, 26 Mar 2024 19:04:27 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lSqk6lOB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479867; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+iVlYtZOB3fCRmg1hwgLJ8pOWaFPkCTIgs4hiVHBEmU=; b=Q2p2a/reY/vt3qmdAb6ahWHnQ6jwxo/KagnmYIp+jGZyvo4GNDf8VFlA66KlSpnm9xR4vg bsSF1mHQONhk2j4CWRZfSQUnFSHoRPz/h9tJeMXDSljMpR39qDmnVNacPI9L46DfKHAiui NujZEgAiMM+8q39puCdqCriwPhu4SQ4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lSqk6lOB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479867; a=rsa-sha256; cv=none; b=uj5RLEVLOL4d3k7vmW0bkqedvb3IlNdkNsaAC3g45hn081kSl21gMBQ7F2cWt6e3lzf8BQ 5jakKbqAgSmvPgwSGYhfpiuM2RdRNcfFVL5RzQRuYZBClmBX35opC3cXEfflu8EGPAfswB PNc1538QbPeuUEqlqN1vc1p/TMHZR/k= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6e8f51d0bf0so4729223b3a.3 for ; Tue, 26 Mar 2024 12:04:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479865; x=1712084665; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=+iVlYtZOB3fCRmg1hwgLJ8pOWaFPkCTIgs4hiVHBEmU=; b=lSqk6lOBePgsU61sfZby5ESI85aiNkpBxI+ReDxtSAWVNpAOpCEl2S3UIQDyAwo+ms MJR4V4WorxraU3E/BAdVjR08efCMrKopT4hZ0UvN7o3lNOvUKFYrcC2c+7GuEn09e0CJ khf7yMEoUQsjZZrDHaC0HzmKRUTuL5Xto6Deem5nsMbmMc8c/e1UOPB73P9uAYZ+MmXQ bgLzpiTEqCez2ksQSQYGI7P3WybGWKr0/r3PnBvJ5TNjEPMKDzFQJCRIYxVmknm3+u16 qu8Pzw0aeSCLl+2+906Gy5MPfwM/IW7s4PTQgVT6+KRiQDI1xWjHIx3nlootDyOyeVbR 5+GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479865; x=1712084665; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+iVlYtZOB3fCRmg1hwgLJ8pOWaFPkCTIgs4hiVHBEmU=; b=Vli/P9OxMqsaTBePsQKGjWDcHEep6bpQYpmH0UvEK60Zhbsi5dkP3fVuq52yQF6wHu WEf3eaWp7HLqNWuS1KJnzATxG71/UkuHPdorDcXWdkC9iliN/4/fmcsYKHU4syKQ5bKg uLvvlp1+qIGPunU6kVkxVOCzDfcDE2t9XO5WLIMdVZmOAsaWjxPFKkQyelgfbenFuNHR uglCaG1br2IMj06hvWDTe1pLitDtFomBymeawQy7B2u4XdZWLFB+BxCSBRKnm27yEPO3 4QoFj0P2wqVIPIxAN9gsdAr/1Dx4ysDQuCMgIN4qlhTsyHT7/xrc19urYb/zOR+5jDbr 5uZQ== X-Gm-Message-State: AOJu0Yy45iCgJfmxj+hzxeyqUx1l9dUJBRtMUfa6EW3TkoC+m/KvW5Nu ew749pnpNgch0Y7y8vseneN9lJR+DBQ5q2fAWyJ3WdwCQMcMrAupItxKJ1JZqSEaQPba X-Google-Smtp-Source: AGHT+IHasTHngVbHXAAz60DxTdM0n3aSiwZ9Kg40P8vfF4TYZxDB4AbSEHDL00iy0NSqMDT5XDhRaw== X-Received: by 2002:a05:6a00:21c9:b0:6e9:a70:aa7a with SMTP id t9-20020a056a0021c900b006e90a70aa7amr3745251pfj.19.1711479865567; Tue, 26 Mar 2024 12:04:25 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:24 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 02/10] mm/swap: move no readahead swapin code to a stand-alone helper Date: Wed, 27 Mar 2024 02:50:24 +0800 Message-ID: <20240326185032.72159-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: A6EA4C0012 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 4q34yemxqtp9cngmspettebdwe7kfkuc X-HE-Tag: 1711479867-171001 X-HE-Meta: U2FsdGVkX18DD+39LHp4uSB/J1qilViTPUnmRf4ydfKQ+JcEIyQbgUzUNlsk8WU0Go4fxtEGl9pY/g8uHfkbex/Oxw9x4ktCscqNv/r3xYLT8V1rnSPJPjtMMPQ4zU22GhBbWbaQcbl5R1Oefy4peGwMqZQ0p4ae87dxwVC7Ce3yPiSmtFCxVa8swL7edW0fNstL0flTXFyayLa4ewcyoNAs7ZpS7gV6njMy7UBWLKEbmm4ajVse5Cz8hG0VLTjeIh1vSd/0PyOW7M4bBIRRNCxyx6AG8IGvfsy4hd0B7Lr7bj59kb0lNnZQwMj+UfJN+9UE7a3cgANihRKmRTGogEY6KKzqs2dsFVZg3jc0rlkMSIPahTXA8Uulq5qps+BwQ2fIfuEXq0hpdi1C4Lta15QA/JunjsudFqNZxiQNyhrLe1tFukyQN6HhYv7x4b+clyxTuR+tQADpTQqHEBwmiKSdV/Hl96DAc3QBZ/0Agi40LJUdJs9+pjZGgR0pQnGiioivm3TB/JrCHQd0VUGIVyCAq1KM2faJXx7Cju7pZ3gsC/yC6eZAlvzXfFk4gv9NXfWifQrxlUkS//pwxGPx6Ly73zBp5Y8l/Sif68HsDjPcwMdSrYBGFqqEqt9/4Y0qjt2hleVSxx9zTe0B2MbUy0ywi0jBTpgiDI5oxhgERrANxoixlFiMbJ5J6b7laDQ4SGEF3D9yVO8AxssazKf5qfrAHQojBA6HsIZThrT3KsAr3qhr4HJwRDZC7etil7vtBYrKydJYAXu/o8DOFF2LIhGEcfyYcTdz9is+I9Xo29oIcwRmPz1ldSyRNUBJQul5qzQPcNJ/bw0WmUGOo855FWwPjiDf3vqWh/Nu9tycmchsHDITIwSAOI7C+4JVS94RKLIGQeiDk0Wo3xgaXRYoLxFPEiwfqe4oZp1tslICpdnMLYy66zyWLC7afEAk2EFMt4swnJ8L+Otetu/Bp61 Wwhvw5Ry gI4RM6gqhWYJ7GiCGYW2UClFlF51KWHR/TPFFwlNwG8B2ybqJFuiiwWpqKbpbzwNb3CLUjINdQnuufc7IrDRQE56j9c42YxbXleiE8hYyjCZgoNL8i2GzxY73rBnyLkttNBVPPLZT5NrXZBZ5+ckKHiShWr6dZflMaOW86qMxscHmOSLu1ZTqfSj7ywxfkyoWt2qOCQvcP9gCFzfnIhpqoGoenR+OjZfRVB9I5t1sup6Yp/0/tg98IEPJzPWyzb+qmMMy5BIsRao11tqI/AUx1olYoHmcfBdSu4lbDTFKyjNlQ9HBobeXqdEcWY1XrXHP7ccW4XwQQxrHEUZ+ssSRBfGSxtQXpU4DhP4s/H42N71OyLmzvuq6WRvzrA0MY0ToPzv8xfjCqrYxqcmYiUa/l3ekyZnP+jOgAFjrdNdr0y+jtACE8wh55k4L/I1lS5Oufpp5251p5sOdYbrI9/Ndlj3DBK3lRNBTKcTa5bAoq3gYBV5MRW95gDAsBffdAdP3eYlP6cOmXaMOLPeuFhikZzIiOx+yyXLv9REnByoUP/092gHTCNVTqiWCSgclE5wUdswrHJCqSqyo7bTB4yIL1EZYo184Q3AMJnvJaQZJRDaI/VE84e1Oz3SGsm2iPtGdIdhR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Simply move the routine to a standalone function, having a cleaner split and avoid helpers being referenced corss multiple files. Basically no feature change, but the error path is very slightly different. Previously a mem_cgroup_swapin_charge_folio fail will cause direct OOM, now we go through the error checking path in do_swap_pte, if the page is already there, just return as the page fault was handled. Signed-off-by: Kairui Song --- mm/memory.c | 42 +++------------------------------- mm/swap.h | 8 +++++++ mm/swap_state.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+), 39 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f2bc6dd15eb8..e42fadc25268 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3937,7 +3937,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swp_entry_t entry; pte_t pte; vm_fault_t ret = 0; - void *shadow = NULL; if (!pte_unmap_same(vmf)) goto out; @@ -4001,47 +4000,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { - /* - * Prevent parallel swapin from proceeding with - * the cache flag. Otherwise, another thread may - * finish swapin first, free the entry, and swapout - * reusing the same entry. It's undetectable as - * pte_same() returns true due to entry reuse. - */ - if (swapcache_prepare(entry)) { - /* Relax a bit to prevent rapid repeated page faults */ - schedule_timeout_uninterruptible(1); + /* skip swapcache and readahead */ + folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); + if (PTR_ERR(folio) == -EBUSY) goto out; - } need_clear_cache = true; - - /* skip swapcache */ - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, - vma, vmf->address, false); page = &folio->page; - if (folio) { - __folio_set_locked(folio); - __folio_set_swapbacked(folio); - - if (mem_cgroup_swapin_charge_folio(folio, - vma->vm_mm, GFP_KERNEL, - entry)) { - ret = VM_FAULT_OOM; - goto out_page; - } - mem_cgroup_swapin_uncharge_swap(entry); - - shadow = get_shadow_from_swap_cache(entry); - if (shadow) - workingset_refault(folio, shadow); - - folio_add_lru(folio); - - /* To provide entry to swap_read_folio() */ - folio->swap = entry; - swap_read_folio(folio, true, NULL); - folio->private = NULL; - } } else { page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); diff --git a/mm/swap.h b/mm/swap.h index fc2f6ade7f80..40e902812cc5 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -55,6 +55,8 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, bool skip_if_exists); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); +struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf); struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf); @@ -87,6 +89,12 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, return NULL; } +static inline struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf) +{ + return NULL; +} + static inline struct page *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, struct vm_fault *vmf) { diff --git a/mm/swap_state.c b/mm/swap_state.c index bfc7e8c58a6d..0a3fa48b3893 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -879,6 +879,66 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, return folio; } +/** + * swapin_direct - swap in folios skipping swap cache and readahead + * @entry: swap entry of this memory + * @gfp_mask: memory allocation flags + * @vmf: fault information + * + * Returns the struct folio for entry and addr after the swap entry is read + * in. + */ +struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct folio *folio; + void *shadow = NULL; + + /* + * Prevent parallel swapin from proceeding with + * the cache flag. Otherwise, another thread may + * finish swapin first, free the entry, and swapout + * reusing the same entry. It's undetectable as + * pte_same() returns true due to entry reuse. + */ + if (swapcache_prepare(entry)) { + /* Relax a bit to prevent rapid repeated page faults */ + schedule_timeout_uninterruptible(1); + return ERR_PTR(-EBUSY); + } + + /* skip swapcache */ + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, + vma, vmf->address, false); + if (folio) { + __folio_set_locked(folio); + __folio_set_swapbacked(folio); + + if (mem_cgroup_swapin_charge_folio(folio, + vma->vm_mm, GFP_KERNEL, + entry)) { + folio_unlock(folio); + folio_put(folio); + return NULL; + } + mem_cgroup_swapin_uncharge_swap(entry); + + shadow = get_shadow_from_swap_cache(entry); + if (shadow) + workingset_refault(folio, shadow); + + folio_add_lru(folio); + + /* To provide entry to swap_read_folio() */ + folio->swap = entry; + swap_read_folio(folio, true, NULL); + folio->private = NULL; + } + + return folio; +} + /** * swapin_readahead - swap in pages in hope we need them soon * @entry: swap entry of this memory From patchwork Tue Mar 26 18:50:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604896 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58143C6FD1F for ; Tue, 26 Mar 2024 19:04:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFBEA6B0093; Tue, 26 Mar 2024 15:04:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D853F6B0095; Tue, 26 Mar 2024 15:04:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB20D6B0096; Tue, 26 Mar 2024 15:04:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 973126B0093 for ; Tue, 26 Mar 2024 15:04:33 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 72A131407E9 for ; Tue, 26 Mar 2024 19:04:33 +0000 (UTC) X-FDA: 81940116426.11.2EA13E8 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf06.hostedemail.com (Postfix) with ESMTP id 95DF3180028 for ; Tue, 26 Mar 2024 19:04:31 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nNKo5Ota; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479871; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TEv2cY48RN4zMeEfANN3o0TZbotXcFB0DQsGIzjU7dM=; b=udvZKPwDXBdJ3VD/s/rydgq5PJjPSCpYIVWBynIPY/hmroSrPtFZ5skpy33FQCUFEpWyFk MU3F4PlTJphyPAERqOrm4h/JOX/kJn0AgxTW5yjH2jNaCAQPI45t0scDMN+yHMCmZgXKwp KqClioCjC9R649kMlBI31ayczrg2LU0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479871; a=rsa-sha256; cv=none; b=CUN5NpSCTpqkm/jQwaTFuedrOkjd5zC+JyXVjbQhkOYsFRLQ8yDxe17H2ibVrwrieeE+hu Sak+NsSl0piNyrND7K/9nOM65bAjHyCUFm/pslORUoILuqHRn1tEffqbD9Awj4qTpDJQud v4frk15qyX56UoS0G1V+UWKHgr3FA1M= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nNKo5Ota; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-5dbd519bde6so3466422a12.1 for ; Tue, 26 Mar 2024 12:04:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479870; x=1712084670; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=TEv2cY48RN4zMeEfANN3o0TZbotXcFB0DQsGIzjU7dM=; b=nNKo5OtaY9YALhTnCmO0YXvYCxDg0kHaPLtZ+yHAQ67g3UafjovXB21HhckcZ1u8+I JB7EdboJbBj3Kw9r0X7TS0eXTcQP0EAUZAewzgT3GYzlJqxGEkksmRRyDQ7aUJVUGfUR 6cvks8bhbxj8yttpjGMgyVltG4v1etuIjFtiJl4kcy5S2U9Dfum/FuaVmYlW8cpUOrbz mqlvR8l8AdPhiwgw+fgESN75juaM4YcCAohV/A6MqxfwEW+9zIt4HKhVFMc4hpc/54sM 2ENiHWAqJenejKI3/c3i/UzbjMLVMniYCh3K+iS8mLMX3yMbGXoGOoCmj1sAw7v4v8ah dYAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479870; x=1712084670; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=TEv2cY48RN4zMeEfANN3o0TZbotXcFB0DQsGIzjU7dM=; b=FjpoBtf8+j4sEW2tZw8zfQE8z+/uIse9bSDycovywfH78de2PK7fk65JGX/pghwIYK UoLKLvW7xlzrgHRzTakW1ZqhFGmzhV9N7Xb3HxfomgZ7Kuc2/BsN8+/lzGnj7eBSE+2U LLXuqTCrq2sKBQLi44KVxWgMbJ/kEVUS4wLpfcUqu6rOly6YOLZU7MZIXo/p4IJKjtBv V4e+W5PyXKMopxLYi9Si7EgD5mMMIkv76Yb3r/UcMA36HkxHWKmZkSrGr1ep3QRt2Zm+ xSrZHF2W7TULB6VMTajB7kJ7bjKY79s0+UJI2vycQgr3sHbjA3UGOwc4HvkwkSLbfKMd bv5g== X-Gm-Message-State: AOJu0Yy+74VW8CIkmo+uw/8TRrMBhx5ZHIwjj81RIkURtJLR8YVL3OVH Sb7M3rMz6fkuwPQnZcTQ17W4TmwIHRGKQj/PhOYFYM13WDlETCCUiC9Z5IIvtK6kS8Xo X-Google-Smtp-Source: AGHT+IEzVu/grUxN8VT3A2w/EHOVk9Am7cvRif7ZTTWBWxDgt7/3PIqhlTfiumnGRYgPCPyofkcdEA== X-Received: by 2002:a05:6a21:398c:b0:1a3:dcdf:13cf with SMTP id ad12-20020a056a21398c00b001a3dcdf13cfmr2102446pzc.56.1711479869865; Tue, 26 Mar 2024 12:04:29 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:29 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 03/10] mm/swap: convert swapin_readahead to return a folio Date: Wed, 27 Mar 2024 02:50:25 +0800 Message-ID: <20240326185032.72159-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 95DF3180028 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 596zjo4737pj9mqo34j4oyd897cznbf7 X-HE-Tag: 1711479871-406997 X-HE-Meta: U2FsdGVkX1+/E25qVPPaSoPbBnJWCyzYGZAuLpOV30OFBjOJ18a+GDBTlCqcJs6SFjn/Kg7ZZ1b+zX8ReLohkVV4mMkdUu+AvSSfG0vgf97o/RhyWgKZfNODGpMGnp5Va0trFgWWxtHeK9MCSLP0oc08SKgynUM6XrtONVqTaf8UtozofTMrt/XiUiydT/r5LZF9Wm9alhv4rNX9BIJ5mWD5JEALX7nXx1mgqHt6mnO1dmFN3EQE1CCYe44IfAo/uQ/RHsfGO1rM2ueFiOcYZXYCLk8pc0tiLZw4wGyg+HcJNXEA+JqBmn2CiTW/TIDVfmfb4Nc5ih4bQZVwwMb6fT8PlkF3XOWv4pItpwoYwvNYTvPEQDQD4hjQeFe/36sRT5R1pp1YE4kJf9F03htJgLMmRY+WT4wxDZDwM8WALj4A0KRF/3MFLARR+DG+GUr9p9cye2PLqGI7cGYs3F2AaXJeF9Kuh91HkYTJNdDoHbbHrF5AZDTCwp93m+IdrXT3DkUsAWdpz4wE3Gf1cX73AhV3Sf5/sceJpONYq+mlANy/Tx/QEKHKjmbH2qIKoVT6kPXstQny3e9NbFgc55L+UQKsu//V+jK2420kMyZDEepbqhVYu3Zrqo5f1p3t/+YxEtnjT63h3qjkIDRSnBynNsoiBiGrIYVmH3xEaL0kRHRRTrhF4vhPk2KIVH00DMCHHGVUcfTZiWxE5XnF7DC4qVoZhat6srMlAG0n/CXC4pd0eicEn2oXyLN3w0Z0zeBFcJ1ncY/8BRbn6upwpMVJ/yEhXpdKH+izgoi7q88H6zliLL0xeDvxRp58xrqq4EPZL5J7IefTxjDUyZDCrEF6FbUmCPvDL4zuaNjGUrNejR7UN9n1B6cWCNZplL1QQVc/DoIzDTecODpJib/Y5Pkh3DUDH5O6bRrU14W3tTmApmoTeXOIID2BhlbKi2vvAcPt7XX67t+/c9tYiqmfaMu J2FUhihh K2xLWjavnv5enH9LgsQ3L3oQFsDNS+TZDNvYTlk+gp5MSB3ocdULDOZyu/k1SYCS80Sgy96ZW4WjKud7hQDEARFb3BmW/BGXFY5WrAWoVqwFBJNxC5O3VkmGtzDNu45cKz1AMEat1cKVH7JNP1+llFgigMP3RRSmYui1X7JNyhivyfo2Cjni8HO5P6FP3YuaiS6AZu7LyhhJdUvLiN+gwZlhVHB2F6b1JLN0sBmaQcvvBD2IoThk6QzX8pJZkpKxtqngHSHnCXdw5BeoQkmAXPkQoa/DySJL86Ui3K7lF33touZGzjTWl46B30Xna/KVO8jAa1oaUMo33xk4d4w+s1hXGlrDXZGoxmfBweU++bFqDkyfApslOyc6EmIoH+uB3pLV1JxfzacXJIqb0nxMX4n6OoliMlVS23P0PDm1GCb9dp0y/wjaF5p1yQFD8XAt0vpevYLwhytti6Ow7ZsOOyWx+wi2jvDWllH8GyGJure9AYamS+pdJWQnLvW/tw1N27TS5NpJKhZjGrKyje250w++28oh65/2XH3l7bijAh2B61oT/59ewoK6hDK5l9qN83h4yktsiq7d5EEAy8XLmYcvJr/v8qyi3CDsiPrSTw0SFpfoGLhWiODnCdE67WETMFgc/bB3CVnag0Z71ndoncOqZD4zUIl3OC6FnVpzTFBXoW8M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Simplify the caller code logic. Signed-off-by: Kairui Song Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 8 +++----- mm/swap.h | 4 ++-- mm/swap_state.c | 6 ++---- mm/swapfile.c | 5 +---- 4 files changed, 8 insertions(+), 15 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index e42fadc25268..dfdb620a9123 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4005,12 +4005,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (PTR_ERR(folio) == -EBUSY) goto out; need_clear_cache = true; - page = &folio->page; } else { - page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, - vmf); - if (page) - folio = page_folio(page); + folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); swapcache = folio; } @@ -4027,6 +4023,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto unlock; } + page = folio_file_page(folio, swp_offset(entry)); + /* Had to read the page from swap area: Major fault */ ret = VM_FAULT_MAJOR; count_vm_event(PGMAJFAULT); diff --git a/mm/swap.h b/mm/swap.h index 40e902812cc5..aee134907a70 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -57,7 +57,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf); -struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, +struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf); static inline unsigned int folio_swap_flags(struct folio *folio) @@ -95,7 +95,7 @@ static inline struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, return NULL; } -static inline struct page *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, +static inline struct folio *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, struct vm_fault *vmf) { return NULL; diff --git a/mm/swap_state.c b/mm/swap_state.c index 0a3fa48b3893..2a9c6bdff5ea 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -951,7 +951,7 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, * it will read ahead blocks by cluster-based(ie, physical disk based) * or vma-based(ie, virtual address based on faulty address) readahead. */ -struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, +struct folio *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, struct vm_fault *vmf) { struct mempolicy *mpol; @@ -964,9 +964,7 @@ struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, swap_cluster_readahead(entry, gfp_mask, mpol, ilx); mpol_cond_put(mpol); - if (!folio) - return NULL; - return folio_file_page(folio, swp_offset(entry)); + return folio; } #ifdef CONFIG_SYSFS diff --git a/mm/swapfile.c b/mm/swapfile.c index 4919423cce76..4dd894395a0f 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1883,7 +1883,6 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, folio = swap_cache_get_folio(entry, vma, addr); if (!folio) { - struct page *page; struct vm_fault vmf = { .vma = vma, .address = addr, @@ -1891,10 +1890,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, .pmd = pmd, }; - page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, + folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, &vmf); - if (page) - folio = page_folio(page); } if (!folio) { swp_count = READ_ONCE(si->swap_map[offset]); From patchwork Tue Mar 26 18:50:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C40A2C6FD1F for ; Tue, 26 Mar 2024 19:04:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FC976B0099; Tue, 26 Mar 2024 15:04:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 483FD6B0098; Tue, 26 Mar 2024 15:04:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B01A6B0099; Tue, 26 Mar 2024 15:04:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 127866B0096 for ; Tue, 26 Mar 2024 15:04:38 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CE5B580235 for ; Tue, 26 Mar 2024 19:04:37 +0000 (UTC) X-FDA: 81940116594.29.248FF1D Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf06.hostedemail.com (Postfix) with ESMTP id 00BFF18001E for ; Tue, 26 Mar 2024 19:04:35 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mhzGtU2P; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479876; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AlnFQa5IuF1FU+kiw8JlfLUfSH7ltUVyYlgQl237ZM0=; b=tyjiUKJAHTvPppRekEYeEeu+pHBBeMKF7fdqp75Du/RCaghkJ42DIMfBvxHb/+xW3+rLx/ IeV1B6/XFMxiaLASzunWZ2EQPJRCracmYsmdoLfVoIzkao5pdB4oDfRLalkcIwarXArUSv occ445TsjGtotEuLAD1XGIn5iS8o7Rw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479876; a=rsa-sha256; cv=none; b=ZtYaUIr0747XkjQTlwhRr0JR80qcDC3Er+QqgsDIxC7t0l7epkpipFWIwHdMLK3AIuqdni 0awLqh282jCU+TzEGRiUBWmuEaOWG5ZhwHrOuxPKdfJKoQOZmq+tcdHHQpxCssKs87stBN 1HIPP/mL/5rPDM82tHROcFgTQ2HeVdU= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mhzGtU2P; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6e6b3dc3564so4184112b3a.2 for ; Tue, 26 Mar 2024 12:04:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479874; x=1712084674; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=AlnFQa5IuF1FU+kiw8JlfLUfSH7ltUVyYlgQl237ZM0=; b=mhzGtU2P+ni30/kARit+Kw1H9n3Xw5+jtEICuXnyA0ERJVf/R0JEpvQXVCFKXzW/vW azCn7ujmebpKDYPEcgwA6ZjvVQLGhPECHFQKCfFPKHFKUx+bjGHutbXkXkBfDWAgUBXB yaULKrSwjVdTX4tZq9aLo9AP3AcNhVgVz3X7ODioEGDXoF5pa0u0YtztaPrNbJaMVrK7 4CCa0nU0u9o9K8n4YoDurTu5ZXOObFLmmVA+WaGLTDI+Dh0nUkXLQceQW45m/ptTaJe1 +osJ3wI95zPFvdndGFEepuhqOvQjY7KI9VENWX+qQaxGVtbkKYxtJ4CrNT49GnZEQAh2 MceA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479874; x=1712084674; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=AlnFQa5IuF1FU+kiw8JlfLUfSH7ltUVyYlgQl237ZM0=; b=vOMjCGtGynuzed58fohAv0k90EWNA2MnJDzGaJDo770p5qwOftGnqaEVI4E9LofMWa +9p8mxs9YtDtrgwbT4NfoqTV7aAdKyDuiKotFjleuaVEB6i1WxheeHNNskVah2dKycWF vZCj5jPfUZ+eR4HbEJFqoO/GPJHaJBp5CyrBOiOP67KSwIbqme8Ijp2ATK+zst3Ob6j5 vHsUuG48qBzgTCmPAbfhxB74THax5/Zx0k8gLvibHDyg9ysLZjm3AZ/BpG7j825GdhWM Fh/MhE6gkBI7frL53OJnLXiBDaeruCCnMP0nWs1Dx09MuWYQQxVT50EIgfEFVn8ySRCF 959g== X-Gm-Message-State: AOJu0YwhNg4aRjrIdd5bIYYQc3ywV9Ytzs3XTTxtqgjLdxsXeDR5gnOI qNzR4c+l8CQ1O0CYKwCY13EdqMOZdfqGBGEZ8D+d9J190y0ylY+5BCwZ83f6FWrDhJre X-Google-Smtp-Source: AGHT+IG6+WnJXSjAIBqkKDWHn5ts/KmlVMWEdeN8j0WCY8uGY9blEEX6n3YjmXe+budCoU47FIqAOg== X-Received: by 2002:a05:6a00:3d49:b0:6e8:f8a9:490e with SMTP id lp9-20020a056a003d4900b006e8f8a9490emr2484291pfb.5.1711479874359; Tue, 26 Mar 2024 12:04:34 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.30 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:33 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 04/10] mm/swap: remove cache bypass swapin Date: Wed, 27 Mar 2024 02:50:26 +0800 Message-ID: <20240326185032.72159-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 00BFF18001E X-Rspam-User: X-Stat-Signature: yohkyoe841oqahez8ym9rjuwqgpawjyw X-Rspamd-Server: rspam03 X-HE-Tag: 1711479875-221904 X-HE-Meta: U2FsdGVkX18kDBg00+ULbqqhaCS46c4eGQGSydoZmYKSkvknOIJ5/xn/mg0q2QjoNqeewuedl0aRUTLm5KfOVxYGBsAgR+Tw3vUekQhy6xBv5jVHOsm0iyl/m9YByT3SmyrQR0HTlUIMEC0mm3ehtCXG8d9r+/D8sm/kvhGTDyAeA+pzZ8OEFhMjtGnJLhqcKlcTZq9eOo9qQJ38p7L63O6I9VwTcf4TXv2dMRisTYnIYruh9jEazVTu0hZE4uTsMu6ZvtQF/gwSdAE7ErH4lVSn6Vu3NEWpgI+bPlRLLWnJwDta+FsVGg4327jz8X0Ylb6RsuGrGi37hubptE2as7FkadgbLW3D7cBcuZKpnqQs8yYd/STDLu9xyGjYii1LoTHZuilPJrIyq9PFRT1zaYefT+6TZFhCLIcmXyBQ8fRhigIlgPaY8rODA9iwPl3auw+m0q9/zTOeDkRyMnENwl4xRn5Hynqw1gfW5Sga8S5OKtjifNJPE3NSuRPsMHqWiqJkV4/vsomHWm3qOvSndlX3PWrlb1a2BDFEzI5LtdtRCLAJ7Mz0CLmhlJsYPaCGYUp0TjK4XTpgXksiQOaoZDxqYZ/GIciUA3E4NpONL0Grj51gpUHJw+7DUbKAgvtxLP0nOzMEBTMvFqMqEHeowMyZfk7dxLTD/r1u1hks8G+ye02zDrp062k9AnruRhxhmo3i2itFgyPPweO4fnDUclkLjI7EaiLmCiyCdFZBbHb4ZVw6vdXpMKBKaJLK37WUQ5tGFs9eGOLd4ixwKuuwSDvATH/A0DA9cryoPrpLG6WR+fSrsAhPnpB9Y8N8UPXyrok5C97QTdXh3vtokUvovD2RwPQrhJRoO13zQpbZfB2Ji75qFOsMHuXW/XqUQ1LoJNZ9RRI1Uem5AImZDUxkPshn2D4ULBoZ5rD40KHQKDs3CTXz35BI04q0jLKAi0fkDdAQobL7bSur2GKaaea JxN2aUYv tC+9cSwaEMCD0lH5LF4PRPZ4Dpe7XIemdEcRh1p7nYlU1aPjK40kcuDcJ2gVgU+EKeHEfDA9BUkLoaasuCbDzMuKA1uRzHeBwJ2kybObTB99tm7KBXkqI3ttMBDufAuXUk9y8+Z7xqNL7BQNV0zM6kAAZb1OFTjZPI9mCU7BqGFf01HQ5kVpUMX1GvAp7WqZZkGpaPeN2LLGSQEct2GEoBnkk3XEgi7KYIdWRMQXyN3XbxUT9Lpn4fh/HwdnuILhGyurCJV/plG2i8Yw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song We used to have the cache bypass swapin path for better performance, but by removing it, more optimization can be applied and have an even better overall performance and less hackish. And these optimizations are not easily doable or not doable at all without this. This patch simply removes it, and the performance will drop heavily for simple swapin, things won't get this worse for real workloads but still observable. Following commits will fix this and archive a better performance. Swapout/in 30G zero pages from ZRAM (This mostly measures overhead of swap path itself, because zero pages are not compressed but simply recorded in ZRAM, and performance drops more as SWAP device is getting full): Test result of sequential swapin/out: Before (us) After (us) Swapout: 33619409 33624641 Swapin: 32393771 41614858 (-28.4%) Swapout (THP): 7817909 7795530 Swapin (THP) : 32452387 41708471 (-28.4%) Signed-off-by: Kairui Song --- mm/memory.c | 18 ++++------------- mm/swap.h | 10 +++++----- mm/swap_state.c | 53 ++++++++++--------------------------------------- mm/swapfile.c | 13 ------------ 4 files changed, 19 insertions(+), 75 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index dfdb620a9123..357d239ee2f6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3932,7 +3932,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) struct page *page; struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; - bool need_clear_cache = false; bool exclusive = false; swp_entry_t entry; pte_t pte; @@ -4000,14 +3999,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { - /* skip swapcache and readahead */ folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); - if (PTR_ERR(folio) == -EBUSY) - goto out; - need_clear_cache = true; } else { folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); - swapcache = folio; } if (!folio) { @@ -4023,6 +4017,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto unlock; } + swapcache = folio; page = folio_file_page(folio, swp_offset(entry)); /* Had to read the page from swap area: Major fault */ @@ -4187,7 +4182,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->orig_pte = pte; /* ksm created a completely new copy */ - if (unlikely(folio != swapcache && swapcache)) { + if (unlikely(folio != swapcache)) { folio_add_new_anon_rmap(folio, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { @@ -4201,7 +4196,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); folio_unlock(folio); - if (folio != swapcache && swapcache) { + if (folio != swapcache) { /* * Hold the lock to avoid the swap entry to be reused * until we take the PT lock for the pte_same() check @@ -4227,9 +4222,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); out: - /* Clear the swap cache pin for direct swapin after PTL unlock */ - if (need_clear_cache) - swapcache_clear(si, entry); if (si) put_swap_device(si); return ret; @@ -4240,12 +4232,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) folio_unlock(folio); out_release: folio_put(folio); - if (folio != swapcache && swapcache) { + if (folio != swapcache) { folio_unlock(swapcache); folio_put(swapcache); } - if (need_clear_cache) - swapcache_clear(si, entry); if (si) put_swap_device(si); return ret; diff --git a/mm/swap.h b/mm/swap.h index aee134907a70..ac9573b03432 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -41,7 +41,6 @@ void __delete_from_swap_cache(struct folio *folio, void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); -void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry); struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); struct folio *filemap_get_incore_folio(struct address_space *mapping, @@ -100,14 +99,15 @@ static inline struct folio *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, { return NULL; } - -static inline int swap_writepage(struct page *p, struct writeback_control *wbc) +static inline struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf); { - return 0; + return NULL; } -static inline void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry) +static inline int swap_writepage(struct page *p, struct writeback_control *wbc) { + return 0; } static inline struct folio *swap_cache_get_folio(swp_entry_t entry, diff --git a/mm/swap_state.c b/mm/swap_state.c index 2a9c6bdff5ea..49ef6250f676 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -880,61 +880,28 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, } /** - * swapin_direct - swap in folios skipping swap cache and readahead + * swapin_direct - swap in folios skipping readahead * @entry: swap entry of this memory * @gfp_mask: memory allocation flags * @vmf: fault information * - * Returns the struct folio for entry and addr after the swap entry is read - * in. + * Returns the folio for entry after it is read in. */ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, struct vm_fault *vmf) { - struct vm_area_struct *vma = vmf->vma; + struct mempolicy *mpol; struct folio *folio; - void *shadow = NULL; - - /* - * Prevent parallel swapin from proceeding with - * the cache flag. Otherwise, another thread may - * finish swapin first, free the entry, and swapout - * reusing the same entry. It's undetectable as - * pte_same() returns true due to entry reuse. - */ - if (swapcache_prepare(entry)) { - /* Relax a bit to prevent rapid repeated page faults */ - schedule_timeout_uninterruptible(1); - return ERR_PTR(-EBUSY); - } - - /* skip swapcache */ - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, - vma, vmf->address, false); - if (folio) { - __folio_set_locked(folio); - __folio_set_swapbacked(folio); - - if (mem_cgroup_swapin_charge_folio(folio, - vma->vm_mm, GFP_KERNEL, - entry)) { - folio_unlock(folio); - folio_put(folio); - return NULL; - } - mem_cgroup_swapin_uncharge_swap(entry); - - shadow = get_shadow_from_swap_cache(entry); - if (shadow) - workingset_refault(folio, shadow); + bool page_allocated; + pgoff_t ilx; - folio_add_lru(folio); + mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); + folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + &page_allocated, false); + mpol_cond_put(mpol); - /* To provide entry to swap_read_folio() */ - folio->swap = entry; + if (page_allocated) swap_read_folio(folio, true, NULL); - folio->private = NULL; - } return folio; } diff --git a/mm/swapfile.c b/mm/swapfile.c index 4dd894395a0f..ae8d3aa05df7 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3389,19 +3389,6 @@ int swapcache_prepare(swp_entry_t entry) return __swap_duplicate(entry, SWAP_HAS_CACHE); } -void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry) -{ - struct swap_cluster_info *ci; - unsigned long offset = swp_offset(entry); - unsigned char usage; - - ci = lock_cluster_or_swap_info(si, offset); - usage = __swap_entry_free_locked(si, offset, SWAP_HAS_CACHE); - unlock_cluster_or_swap_info(si, ci); - if (!usage) - free_swap_slot(entry); -} - struct swap_info_struct *swp_swap_info(swp_entry_t entry) { return swap_type_to_swap_info(swp_type(entry)); From patchwork Tue Mar 26 18:50:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604898 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AAC7CD1283 for ; Tue, 26 Mar 2024 19:04:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B7366B009A; Tue, 26 Mar 2024 15:04:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9180E6B009B; Tue, 26 Mar 2024 15:04:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7911E6B009C; Tue, 26 Mar 2024 15:04:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5FAFB6B009A for ; Tue, 26 Mar 2024 15:04:43 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0BEB7A0DAA for ; Tue, 26 Mar 2024 19:04:43 +0000 (UTC) X-FDA: 81940116846.19.075B35A Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf28.hostedemail.com (Postfix) with ESMTP id F2973C0007 for ; Tue, 26 Mar 2024 19:04:40 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NJgxkd9L; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479881; a=rsa-sha256; cv=none; b=Y5rLqK9lCV+Eq6A891G5LHgOnvq7U+ct7rx62cdsfOXZNTg3uBPeZx9E2LjTPtixz/Va9V EJleZf27BG4rkKit8Ole6CGWfwRmJXAjIs04eT8YAxuAvMwFE4eq39qX5DJ2Sq9ICZWRK/ OvHHYuixdgrtUjBcTaFwYUZi2KKorTU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NJgxkd9L; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479881; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QbZCOjluZtqwe+IJf5t2wPcomK28PDyLWYCalJheOPc=; b=w3K3SeRUm/gUyLrRKO1KVRA0/yz4xbwo9Jj2pFyScU99uZuXYwRDjoDWx02dkdd9tBrWon vie0hw7DCMzW5giLmf9TChOqdm8K7o+ZWwRIqUJ3kjiHO4/+xlgnhgmXhdnLXLz3GWKCyx z6oZJu12vTvYOCWekAozRjAMFCJf9J4= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1e00d1e13acso35826245ad.0 for ; Tue, 26 Mar 2024 12:04:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479879; x=1712084679; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=QbZCOjluZtqwe+IJf5t2wPcomK28PDyLWYCalJheOPc=; b=NJgxkd9LluTHriQfdTSMtZa32tzlT0Onlx+Kv0hg0hD3krPLrdbjzrAbJbdt3+3CZz qxoAdEn1BmOFazTzunrFlbSckc/4wjCPSRK5bPMySulanz0dQ3vQIRgI5tpVVO5vKYMN D5/TEX7Q/t/djfxsxRjQdKlHmVGVWxK2Tewm0RrmjwA29bQE6PQieyCvb8lcA/NX+6Pt NLeeOgWnEQlNeBHF2NtpMgMG2PRP+T7DpjavkmHm4f/gQ2mflCm8WRGlydxKifowA7ze TNtQ7KkfqY2z9lnS3iZKKqeDsUkyU/dEDdEwhDo1uJGBKghjGP7Qo2K/EORKiOrDd1pv 4BZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479879; x=1712084679; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=QbZCOjluZtqwe+IJf5t2wPcomK28PDyLWYCalJheOPc=; b=oq3wkz2i71pmhlD0Vc2zWoNsvf5+cTSrLlv3nk2quLJn2Hs+z3mefhLxJMu1kyqtxo PvCMNZbmVfGjsHc8JRQMEDk64GyoY/PPnEHA2gIKkPqof/9FO/7bl21rynpP14OpNnsD OBpK/AHlNQ/nP3N3GasxY7G6SdjA5Eb+w5bILG6oXY96PU5mINbtv2n5qgwdz0wop4/5 84hQRZK7bDMrCI6ielmYtu7GvtBs6QCgT6DR3TM4tBRsFgMQA4u90Q/3FzjZjksrLVd1 JsSXdclDQGP9jgRN1YHP4sHrwacy5z8HYSxO1vjTF+uOuRaaqu3PNifIsXgfYg4BFA6n KKBw== X-Gm-Message-State: AOJu0Yz+Dndo7m/zPMD1ZxDK//15bJSedGYKX6O2/due8UHL3/z1wqPv 1nJ4tBAieanWhT4bhsNHZz894CrGbb7ugz/EZK5K5jsq+JchjurjeN9JMU/HzwJ8bysh X-Google-Smtp-Source: AGHT+IHPrVL9xajG03QLF+CldgXO1e+DdNsfAc6WD2CDV/JcM0VHVNqqhHe8QUEsP8IyBUI6ApCsOQ== X-Received: by 2002:a05:6a20:9187:b0:1a3:c113:f441 with SMTP id v7-20020a056a20918700b001a3c113f441mr8667699pzd.15.1711479878748; Tue, 26 Mar 2024 12:04:38 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:38 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 05/10] mm/swap: clean shadow only in unmap path Date: Wed, 27 Mar 2024 02:50:27 +0800 Message-ID: <20240326185032.72159-6-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: F2973C0007 X-Stat-Signature: k41wsb4dju8ddeiz4oqfczubcisa9q6m X-HE-Tag: 1711479880-447126 X-HE-Meta: U2FsdGVkX1/MGv9TTBRO6rDxIwcQ20w/SYSKE+XsjPoY7FbPk54N7DrPqLjbsV8EpyfjkqmDeHeu3vA/DGINIlMAIMuunbYkLA90pzre5JRqn6ipCVjzbO321gV9fMiV8oQYihvOoaCtNCD3b/6pSXcUyD5fgTNGFbGSDXOSc3r0NxECuMZc6sauxY+vpblzPxSUPF5B6ES7LC4o15cN6n++Va+m+LPu6ebFWkQH0XAE4W6biWibvj+XSY+txNFB8uW1E6ImbhhDJ9oJca38yCxaICGOl09q6RmXAY0/+zgfAj/u42cqNtuI/AJv+6YHU0cK13i/23aln3ImUqFuyFOpN/lfLlmCtiafNLNQfciEnSw/5H8RroOvXFN4xQzvs8Oa/1uiQtRVVvG3vrruEb+bwQQ152lUCs3oorG1QX9GLZLU1PN4DXWuw/FqnMDpN3qLslpiIy2izKCDYIVlwfcwe2S9bly0CszT56fiQp4bPsqCK+I7S+FLjmcXNvMvTr2bxWCWKQKDEJK1XkttCyikqhou6KLGXW659ltmvyR3J+CEfXmzn/MP8GgumHOT2KDTSqvrrWTIhIBcvI6t+VjxNuQd1F/4jaqMI7xs5G8POQMfykXNK+dTLgjevvQloy39quEbVtWJjiXPnK2sG+4cM+o9c3wYeNFIubEipj9lQc9mmtMiMeVw3CsXQquC25TfpvF1R8XCsL40eJjkEP6DcIXAFBKNge49jmFInD4oQJEcYPMsvvIN8CsAL9qZjZL5Cfq8JSVfZVDbH5urm860B/HNRHQD+QQsVGLpJ8xQlxf4TH/4GrWg9zcZLTM3/FBlG71/5CnxSLi+zZW2F82CZt7jk0tcHoOYtSNri/5u12r7syycHEiB2xbNgoL8Pr0yxBcGv5jRyPCAWFUEfRJ9zEeJebBnPF8XUgB8B0Jt1DvomJvxAIsm3VR1E8Q+gBO7+6hqJfsq4umriKw lk2VPcyb PTRZ2tdaH9DrcfVvpid3mnz7m+AHa+LX2kSV+A3aeUanFswr06JZF01+cZAU0FvFkpAxEp/HBgngnmfbI1y7bEUEg8b2NW2pRcGHAb5ce0Sp/B5bkOIOWRZkatA85FL+Vn1y2L4LBr9H8tsAWiNDGnSUmJK5aQ5KhYSN9+vBJymmveDgIOujXNo2TRKgIl6X9kXsso9QaN4MfBV5S3xhYH26/WIDCyHWwADaLw2qUzORJllIUAOacHwpTOk2MAUEeAjZuWVMER7CfEJHC5u9Xmx69dikpLfRh7BUKF9Q5Pma+cv8PPPcjNV/Wr0dslD705yLyyINlSNFR/Gasis1jBB+KCyDeSsaVmT9yxKTq2kMQNuoZcYzKDBApF6hA1gDnmAlTz0SPBdl6MNCkAXeQmKuufQU3ZRHdNyV0HL9LxLyQFZbMJLDR4AMyGxk1L/HVLuiaEyGaHsYFaCVrSm8oPN0ziV01EwPml+eQfeN2+IO5ximvID0fU2KDJvKQD4yQBI6w1gq54q3v5ukby/9woLykgqKbDDRajesrMtN+orBhCe0JhudZlms0PGyVXY+LBSKOOkP1wEAtazANEJ/qUUOO8w9Wh3WhEKpD4WxDBiOBuByyByFs6e+kK4n93DLf3e7X X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song After removing the cache bypass swapin, the first thing could be gone is all the clear_shadow_from_swap_cache calls. Currently clear_shadow_from_swap_cache is being called in many paths. It's currently being called by swap_range_free which has two direct callers: - swap_free_cluster, which is only called by put_swap_folio to free up the shadow of a slot cluster. - swap_entry_free, which is only called by swapcache_free_entries to free up shadow of a slot. And these two are very commonly used everywhere in SWAP codes. Notice the shadow is only written by __delete_from_swap_cache after after a successful SWAP out, so clearly we only want to clear shadow after SWAP in (the shadow is used and no longer needed) or Unmap/MADV_FREE. After all swapin is using cached swapin path, clear_shadow_from_swap_cache is not needed for swapin anymore, because we have to insert the folio first, and this already removed the shadow. So we only need to clear the shadow for Unmap/MADV_FREE. All direct/indirect caller of swap_free_cluster and swap_entry_free are listed below: - swap_free_cluster: -> put_swap_folio (Clean the cache flag and try delete shadow, after removing the cache or error handling) -> delete_from_swap_cache -> __remove_mapping -> shmem_writepage -> folio_alloc_swap -> add_to_swap -> __read_swap_cache_async - swap_entry_free -> swapcache_free_entries -> drain_slots_cache_cpu -> free_swap_slot -> put_swap_folio (Already covered above) -> __swap_entry_free / swap_free -> free_swap_and_cache (Called by Unmap/Zap/MADV_FREE) -> madvise_free_single_vma -> unmap_page_range -> shmem_undo_range -> swap_free (Called by swapin path) -> do_swap_page (Swapin path) -> alloc_swapdev_block/free_all_swap_pages () -> try_to_unmap_one (Error handling, no shadow) -> shmem_set_folio_swapin_error (Shadow just gone) -> shmem_swapin_folio (Shmem's do_swap_page) -> unuse_pte (Swapoff, which always use swapcache) So now we only need to call clear_shadow_from_swap_cache in free_swap_and_cache because all swapin/out will went through swap cache now. Previously all above functions could invoke clear_shadow_from_swap_cache in case a cache bypass swapin left a entry with uncleared shadow. Also make clear_shadow_from_swap_cache only clear one entry for simplicity. Test result of sequential swapin/out: Before (us) After (us) Swapout: 33624641 33648529 Swapin: 41614858 40667696 (+2.3%) Swapout (THP): 7795530 7658664 Swapin (THP) : 41708471 40602278 (+2.7%) Signed-off-by: Kairui Song --- mm/swap.h | 6 ++---- mm/swap_state.c | 33 ++++++++------------------------- mm/swapfile.c | 6 ++++-- 3 files changed, 14 insertions(+), 31 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index ac9573b03432..7721ddb3bdbc 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -39,8 +39,7 @@ int add_to_swap_cache(struct folio *folio, swp_entry_t entry, void __delete_from_swap_cache(struct folio *folio, swp_entry_t entry, void *shadow); void delete_from_swap_cache(struct folio *folio); -void clear_shadow_from_swap_cache(int type, unsigned long begin, - unsigned long end); +void clear_shadow_from_swap_cache(swp_entry_t entry); struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); struct folio *filemap_get_incore_folio(struct address_space *mapping, @@ -148,8 +147,7 @@ static inline void delete_from_swap_cache(struct folio *folio) { } -static inline void clear_shadow_from_swap_cache(int type, unsigned long begin, - unsigned long end) +static inline void clear_shadow_from_swap_cache(swp_entry_t entry) { } diff --git a/mm/swap_state.c b/mm/swap_state.c index 49ef6250f676..b84e7b0ea4a5 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -245,34 +245,17 @@ void delete_from_swap_cache(struct folio *folio) folio_ref_sub(folio, folio_nr_pages(folio)); } -void clear_shadow_from_swap_cache(int type, unsigned long begin, - unsigned long end) +void clear_shadow_from_swap_cache(swp_entry_t entry) { - unsigned long curr = begin; - void *old; - - for (;;) { - swp_entry_t entry = swp_entry(type, curr); - struct address_space *address_space = swap_address_space(entry); - XA_STATE(xas, &address_space->i_pages, curr); - - xas_set_update(&xas, workingset_update_node); + struct address_space *address_space = swap_address_space(entry); + XA_STATE(xas, &address_space->i_pages, swp_offset(entry)); - xa_lock_irq(&address_space->i_pages); - xas_for_each(&xas, old, end) { - if (!xa_is_value(old)) - continue; - xas_store(&xas, NULL); - } - xa_unlock_irq(&address_space->i_pages); + xas_set_update(&xas, workingset_update_node); - /* search the next swapcache until we meet end */ - curr >>= SWAP_ADDRESS_SPACE_SHIFT; - curr++; - curr <<= SWAP_ADDRESS_SPACE_SHIFT; - if (curr > end) - break; - } + xa_lock_irq(&address_space->i_pages); + if (xa_is_value(xas_load(&xas))) + xas_store(&xas, NULL); + xa_unlock_irq(&address_space->i_pages); } /* diff --git a/mm/swapfile.c b/mm/swapfile.c index ae8d3aa05df7..bafae23c0f26 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -724,7 +724,6 @@ static void add_to_avail_list(struct swap_info_struct *p) static void swap_range_free(struct swap_info_struct *si, unsigned long offset, unsigned int nr_entries) { - unsigned long begin = offset; unsigned long end = offset + nr_entries - 1; void (*swap_slot_free_notify)(struct block_device *, unsigned long); @@ -748,7 +747,6 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, swap_slot_free_notify(si->bdev, offset); offset++; } - clear_shadow_from_swap_cache(si->type, begin, end); /* * Make sure that try_to_unuse() observes si->inuse_pages reaching 0 @@ -1605,6 +1603,8 @@ bool folio_free_swap(struct folio *folio) /* * Free the swap entry like above, but also try to * free the page cache entry if it is the last user. + * Useful when clearing the swap map and swap cache + * without reading swap content (eg. unmap, MADV_FREE) */ int free_swap_and_cache(swp_entry_t entry) { @@ -1626,6 +1626,8 @@ int free_swap_and_cache(swp_entry_t entry) !swap_page_trans_huge_swapped(p, entry)) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); + if (!count) + clear_shadow_from_swap_cache(entry); put_swap_device(p); } return p != NULL; From patchwork Tue Mar 26 18:50:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01747C54E67 for ; Tue, 26 Mar 2024 19:11:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A7CF6B0096; Tue, 26 Mar 2024 15:11:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8580B6B0098; Tue, 26 Mar 2024 15:11:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D1686B00A6; Tue, 26 Mar 2024 15:11:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5B9666B0096 for ; Tue, 26 Mar 2024 15:11:26 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2E1FD1C0AEF for ; Tue, 26 Mar 2024 19:11:26 +0000 (UTC) X-FDA: 81940133772.15.203F45A Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) by imf24.hostedemail.com (Postfix) with ESMTP id 43A2F18001B for ; Tue, 26 Mar 2024 19:11:24 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="h1VDjW/7"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.166.53 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711480284; a=rsa-sha256; cv=none; b=w9NlZATY0pcdHZjITFGfetVrlA2MXw7j/B0a6c/LOcoeh/v9P1Mqoe29grytOglqlwdvp5 5xddJqspUy5Me1tr4/cJSSxf+et92s/1b58edIkd+MyP0UDPzJFKYfreovaDKEBDVW/zDO WhfCccbqBsCTQF2gX7O+sEkka9xtsrg= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="h1VDjW/7"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.166.53 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711480284; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ztDoqKIOUdyGA4S9aoIHNjHCO+9b1c8CvnSrz3vjQuA=; b=wKcJe4po8BM+7s5qPr6LZdTQU8N9443fG8gAYfZE6/hXAjjlPg/aQUWViRELEKX4LVpCCZ MBAEHlMMYOYOaqWBWUXfPO4/983LtuealMntqmc9wOhT+ebT/ZdDa9IXhKF3Hl99mYHWM2 OiHeBa7JHKE4iOVbzGW8JNVifLYbv6I= Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7c8ae457b27so142592039f.2 for ; Tue, 26 Mar 2024 12:11:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711480283; x=1712085083; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ztDoqKIOUdyGA4S9aoIHNjHCO+9b1c8CvnSrz3vjQuA=; b=h1VDjW/7GI1UVCE6OKVizbAIO1GSZSmwXiJ1oB0boVZ7cMW08Oj1V3PjAAo/Fe51gl lDh9rWE7uObrQxHB+i0F30c3DapFNyqHInnx6XbyaAliLnLRESdv7He68NMH3z+//dt6 +E4xBQptzm1xh9d5xwoQ28/GSMgu6M3AGoYIwBaDSBZ/7N2o4MyUlE0e5gHjFsK+ozNO C8LeA9Wb+WgZ4tcourR57UlQyRPbkFQIdDiYUf0f+WresLuepclS9oxWC3nU4FI1NvfP pp9uDZ4nIMoapMpf3y8RSUOlG2kDjw89XxT0GbnqIvSc3uFHvK5TDNfFlTUb4Rzx7MRW sMeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711480283; x=1712085083; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ztDoqKIOUdyGA4S9aoIHNjHCO+9b1c8CvnSrz3vjQuA=; b=tcyUcG2yFGENPYCsRhq3VIV9Fs52Y/BJThwcPr8jYEPLSEmn8+8HdSeCdv5bmezscU 9VmIJSyB3Rw2r3P/Zan5DRdQ4RCRGcsrTau7yfuz0uZ8xfQ1kCAuT/HBMPBcVOT3fBHB mRQItfxzAxvEzRHTw0hNlNPub761+9cOKxIY5ObpjKxMX8k0Z8Q1/v+em6bvhMCx7NBQ 8YXqTKZkhDe4YEBIVA6nzYWEtXcTvjYcJjnfvhfa7ZTits+NlYJf05/2cT6NLh1BxbQ/ xCMfpm3d6OhMNazrQH5upNr3spZH39YO4NF+NXUSEVECl4YbD98NX+zLgbWX/IVSNLmr ITAg== X-Gm-Message-State: AOJu0YyQVGHLm7YFYLKE+ROs8sQulARlnxKbTpte/gWmBIhuOfFDlecV MQuRfJkOaFMTf03+WsnTSez5qniyGknsRQvNKsDgAY9v2WGRNvjkKxNqlD4nlC1pa7eo X-Google-Smtp-Source: AGHT+IFhZpRE3C2jmFm0uvP6wXg7vusrz0LzQTW8Jr4o04NUgO6I7cuBwzk1y02ru2ic2EfhCl9c0w== X-Received: by 2002:a05:6a00:21d2:b0:6e6:8df5:f77a with SMTP id t18-20020a056a0021d200b006e68df5f77amr2245769pfj.31.1711479883480; Tue, 26 Mar 2024 12:04:43 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:42 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 06/10] mm/swap: switch to use multi index entries Date: Wed, 27 Mar 2024 02:50:28 +0800 Message-ID: <20240326185032.72159-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 43A2F18001B X-Stat-Signature: qt5rh39oa4mj4kem6zqi4herecstj65f X-HE-Tag: 1711480284-378087 X-HE-Meta: U2FsdGVkX19yATz5d9bBlayouTIEfuvDv2+9GzZcfJbkz+R270q++ZgjvnQr0pKW5ygUsnV32aGFs1LJ0yefqCMgHQjf9zdnxUJdbXbIZMO8tHFa8bIk+/OE9Qi/S5jDt4iEEzWPAjiOUPrWTl4wdphIU++ys5Ab/xe2lOx8u32QgmVX875OVY8aWVBmRB8MyZgVSPrPqyDUVa/hp/fz6RokgvzCAMRgSjfYh6fBjka7ciysqLT+amM5EJAQcU6C8eQ3iMkrOtvs3eJDsTg5kCZ6eaBnw3XFWogeQZ7b02kHmD8eCNY8Ao7+XPBbW4P+S0RLNpvrNPRS1ct0rfyO6gyzUZ/xKQqiY4ZukDk9/JgQ9+d5mX+6AsjTUdB2vhbT0p+wZo5vq8AF/zo4bFm9Oo/hE+wEO9aQcVYlePuiOy1h4zwSeoywyWUSMWQvbP1STrrdjUNRTJXSjY2bWHhaxaDALkif1Do++Wj2Du8j9wHIWg+2ctLMxWGv86oxzYO4R1ijO/jy1cOzF0rEC/0dKfTdXTdoHtBR0L7F6Nnvquasbg82iUtofst7cpSoezPBn4y69W7XdOyPMd4M4XCx7/6ZKp3DDILf6Ei0LneUuUq/mpnq/82E4+sBSGni/pNHBjp6CQ3zQdgF3K+eaks3UVaxEI0w+kCPMrtG/nBuFPGLXv+5BR+BkgLbQLaHwAKBxVX6dDQACQLZg4ZVlMvMYAfByEyqx00/XXdcdRcInIDtI4vzdHYIxbBiaZVc3vH6JpneMEPTSZggqzHYb39sS3d+DgqqEg9nsGQxmmL3CBXQZDrik/0aQr4puSXrWmzCrTTOrgogihQ2ClG6ow+3Lf9LHjg7jh2WmU9uMMJrtSVqDsJ8SuR3iKlpwAT0rUmq7d7mbYjFvH42+EhgWfeZ0iVS5hhMstqXJg2z/iQFaRX5icbfQkp/DsAGplcOk8eDdoGqnMRGzIfk3qWvWuS 0rr8LZIu NGa1JW72Z6Bqpmw9txduhxFnKIYhpF9HAz1pQ4qlMk6E1NtMNgPEaEDWooujIoeEPTMMrtcxibqIDTSiX1lrsXmTGKy2fU0CmOk5AS/GwjLiu4lqEjAtc1nq4SpT6DD1Rz+826LcL4MISNBW/wLE6wN/zqh+6d5ZCWL/COJGbxmacjc3Mz0Vwa2+6+ij1sc6S9a/vwMl8WRBVwrQtRcVDyMejyS7+9oL/pAhUAh2WgB/NhWO4npsaBga5JwoatqXXBc1dleBDmblRMTNTRHZFFp+SKHfl4mV5UefBeFrlVorkg2PBVGFwAJHGc7YciUGMaaUPm8pMxOFLFPFsn3StqcR1lHknllTdUhsDU70CjPdj5hjLWqtWaF8Lf2N8y/qg0022GYQrjv6GTVuQijeGdW4g8EKNA3NnruPVyVFsTCQJp/D+bO8qaBfWcYFeAAkwMbnRL9GpDGScng3HXf7C4UqSuuLsaCkxhcitWfqOcO5K0OSLh6zh7T2giyu9D3X28bOMjFl6zszjPr6JmGeWPbZ0qyQUX9g9T5be62Ca8bDmcx8eQ2cZVtgki5TYslRs95vHfI0GNzl7/E5E6k274tVXPN/VtszISlXALsvNU9vrJIMqLdD68XxnOicdn06MbgIbTrHTjxzWRKLaHWUDxbZ84g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song From: Kairui Song Since now all explicit shadow clearing is gone and all swapin / swapout path is all using swap cache, switch swap cache to use multi index so swapping out of THP will be faster, also using less memory. Test result of sequential swapin/out of 30G zero page on ZRAM: Before (us) After (us) Swapout: 33648529 33713283 Swapin: 40667696 40954646 Swapout (THP): 7658664 6921176 (+9.7%) Swapin (THP) : 40602278 40891953 And after swapping out 30G with THP, the radix node usage dropped by a lot: Before: radix_tree_node 73728K After: radix_tree_node 7056K (-94%) Signed-off-by: Kairui Song --- mm/filemap.c | 27 +++++++++++++++++ mm/huge_memory.c | 77 +++++++++++++++++++----------------------------- mm/internal.h | 2 ++ mm/swap_state.c | 54 ++++++++++----------------------- 4 files changed, 75 insertions(+), 85 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 0ccdc9e92764..5e8e3fd26b8d 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -919,6 +919,33 @@ static int __filemap_lock_store(struct xa_state *xas, struct folio *folio, return xas_error(xas); } +int __filemap_add_swapcache(struct address_space *mapping, struct folio *folio, + pgoff_t index, gfp_t gfp, void **shadowp) +{ + XA_STATE_ORDER(xas, &mapping->i_pages, index, folio_order(folio)); + long nr; + int ret; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); + mapping_set_update(&xas, mapping); + + nr = folio_nr_pages(folio); + folio_ref_add(folio, nr); + + ret = __filemap_lock_store(&xas, folio, index, gfp, shadowp); + if (likely(!ret)) { + mapping->nrpages += nr; + __node_stat_mod_folio(folio, NR_FILE_PAGES, nr); + __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr); + xas_unlock_irq(&xas); + } else { + folio_put_refs(folio, nr); + } + + return ret; +} + noinline int __filemap_add_folio(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9859aa4f7553..4fd2f74b94a9 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2886,14 +2886,12 @@ static void __split_huge_page_tail(struct folio *folio, int tail, lru_add_page_tail(head, page_tail, lruvec, list); } -static void __split_huge_page(struct page *page, struct list_head *list, - pgoff_t end, unsigned int new_order) +static void __split_huge_page(struct address_space *mapping, struct page *page, + struct list_head *list, pgoff_t end, unsigned int new_order) { struct folio *folio = page_folio(page); struct page *head = &folio->page; struct lruvec *lruvec; - struct address_space *swap_cache = NULL; - unsigned long offset = 0; int i, nr_dropped = 0; unsigned int new_nr = 1 << new_order; int order = folio_order(folio); @@ -2902,12 +2900,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* complete memcg works before add pages to LRU */ split_page_memcg(head, order, new_order); - if (folio_test_anon(folio) && folio_test_swapcache(folio)) { - offset = swp_offset(folio->swap); - swap_cache = swap_address_space(folio->swap); - xa_lock(&swap_cache->i_pages); - } - /* lock lru list/PageCompound, ref frozen by page_ref_freeze */ lruvec = folio_lruvec_lock(folio); @@ -2919,18 +2911,18 @@ static void __split_huge_page(struct page *page, struct list_head *list, if (head[i].index >= end) { struct folio *tail = page_folio(head + i); - if (shmem_mapping(folio->mapping)) + if (shmem_mapping(mapping)) nr_dropped++; else if (folio_test_clear_dirty(tail)) folio_account_cleaned(tail, - inode_to_wb(folio->mapping->host)); + inode_to_wb(mapping->host)); __filemap_remove_folio(tail, NULL); folio_put(tail); } else if (!PageAnon(page)) { - __xa_store(&folio->mapping->i_pages, head[i].index, + __xa_store(&mapping->i_pages, head[i].index, head + i, 0); - } else if (swap_cache) { - __xa_store(&swap_cache->i_pages, offset + i, + } else if (folio_test_swapcache(folio)) { + __xa_store(&mapping->i_pages, swp_offset(folio->swap) + i, head + i, 0); } } @@ -2948,23 +2940,17 @@ static void __split_huge_page(struct page *page, struct list_head *list, split_page_owner(head, order, new_order); /* See comment in __split_huge_page_tail() */ - if (folio_test_anon(folio)) { + if (mapping) { /* Additional pin to swap cache */ - if (folio_test_swapcache(folio)) { - folio_ref_add(folio, 1 + new_nr); - xa_unlock(&swap_cache->i_pages); - } else { - folio_ref_inc(folio); - } - } else { - /* Additional pin to page cache */ folio_ref_add(folio, 1 + new_nr); - xa_unlock(&folio->mapping->i_pages); + xa_unlock(&mapping->i_pages); + } else { + folio_ref_inc(folio); } local_irq_enable(); if (nr_dropped) - shmem_uncharge(folio->mapping->host, nr_dropped); + shmem_uncharge(mapping->host, nr_dropped); remap_page(folio, nr); if (folio_test_swapcache(folio)) @@ -3043,11 +3029,12 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, struct deferred_split *ds_queue = get_deferred_split_queue(folio); /* reset xarray order to new order after split */ XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); + struct address_space *mapping = folio_mapping(folio);; struct anon_vma *anon_vma = NULL; - struct address_space *mapping = NULL; int extra_pins, ret; pgoff_t end; bool is_hzp; + gfp_t gfp; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); @@ -3079,7 +3066,6 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, } } - is_hzp = is_huge_zero_page(&folio->page); if (is_hzp) { pr_warn_ratelimited("Called split_huge_page for huge zero page\n"); @@ -3089,6 +3075,17 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, if (folio_test_writeback(folio)) return -EBUSY; + if (mapping) { + gfp = current_gfp_context(mapping_gfp_mask(mapping) & + GFP_RECLAIM_MASK); + + xas_split_alloc(&xas, folio, folio_order(folio), gfp); + if (xas_error(&xas)) { + ret = xas_error(&xas); + goto out; + } + } + if (folio_test_anon(folio)) { /* * The caller does not necessarily hold an mmap_lock that would @@ -3104,33 +3101,19 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, goto out; } end = -1; - mapping = NULL; anon_vma_lock_write(anon_vma); } else { - gfp_t gfp; - - mapping = folio->mapping; - /* Truncated ? */ if (!mapping) { ret = -EBUSY; goto out; } - gfp = current_gfp_context(mapping_gfp_mask(mapping) & - GFP_RECLAIM_MASK); - if (!filemap_release_folio(folio, gfp)) { ret = -EBUSY; goto out; } - xas_split_alloc(&xas, folio, folio_order(folio), gfp); - if (xas_error(&xas)) { - ret = xas_error(&xas); - goto out; - } - anon_vma = NULL; i_mmap_lock_read(mapping); @@ -3189,7 +3172,9 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, int nr = folio_nr_pages(folio); xas_split(&xas, folio, folio_order(folio)); - if (folio_test_pmd_mappable(folio) && + + if (!folio_test_anon(folio) && + folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) { if (folio_test_swapbacked(folio)) { __lruvec_stat_mod_folio(folio, @@ -3202,7 +3187,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, } } - __split_huge_page(page, list, end, new_order); + __split_huge_page(mapping, page, list, end, new_order); ret = 0; } else { spin_unlock(&ds_queue->split_queue_lock); @@ -3218,9 +3203,9 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, if (anon_vma) { anon_vma_unlock_write(anon_vma); put_anon_vma(anon_vma); - } - if (mapping) + } else { i_mmap_unlock_read(mapping); + } out: xas_destroy(&xas); count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED); diff --git a/mm/internal.h b/mm/internal.h index 7e486f2c502c..b2bbfd3c2b50 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1059,6 +1059,8 @@ struct migration_target_control { */ size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, struct folio *folio, loff_t fpos, size_t size); +int __filemap_add_swapcache(struct address_space *mapping, struct folio *folio, + pgoff_t index, gfp_t gfp, void **shadowp); /* * mm/vmalloc.c diff --git a/mm/swap_state.c b/mm/swap_state.c index b84e7b0ea4a5..caf69696f47c 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -90,48 +90,22 @@ int add_to_swap_cache(struct folio *folio, swp_entry_t entry, { struct address_space *address_space = swap_address_space(entry); pgoff_t idx = swp_offset(entry); - XA_STATE_ORDER(xas, &address_space->i_pages, idx, folio_order(folio)); - unsigned long i, nr = folio_nr_pages(folio); - void *old; - - xas_set_update(&xas, workingset_update_node); + int ret; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_swapcache(folio), folio); VM_BUG_ON_FOLIO(!folio_test_swapbacked(folio), folio); - folio_ref_add(folio, nr); folio_set_swapcache(folio); folio->swap = entry; - do { - xas_lock_irq(&xas); - xas_create_range(&xas); - if (xas_error(&xas)) - goto unlock; - for (i = 0; i < nr; i++) { - VM_BUG_ON_FOLIO(xas.xa_index != idx + i, folio); - if (shadowp) { - old = xas_load(&xas); - if (xa_is_value(old)) - *shadowp = old; - } - xas_store(&xas, folio); - xas_next(&xas); - } - address_space->nrpages += nr; - __node_stat_mod_folio(folio, NR_FILE_PAGES, nr); - __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr); -unlock: - xas_unlock_irq(&xas); - } while (xas_nomem(&xas, gfp)); - - if (!xas_error(&xas)) - return 0; + ret = __filemap_add_swapcache(address_space, folio, idx, gfp, shadowp); + if (ret) { + folio_clear_swapcache(folio); + folio->swap.val = 0; + } - folio_clear_swapcache(folio); - folio_ref_sub(folio, nr); - return xas_error(&xas); + return ret; } /* @@ -142,7 +116,6 @@ void __delete_from_swap_cache(struct folio *folio, swp_entry_t entry, void *shadow) { struct address_space *address_space = swap_address_space(entry); - int i; long nr = folio_nr_pages(folio); pgoff_t idx = swp_offset(entry); XA_STATE(xas, &address_space->i_pages, idx); @@ -153,11 +126,9 @@ void __delete_from_swap_cache(struct folio *folio, VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); VM_BUG_ON_FOLIO(folio_test_writeback(folio), folio); - for (i = 0; i < nr; i++) { - void *entry = xas_store(&xas, shadow); - VM_BUG_ON_PAGE(entry != folio, entry); - xas_next(&xas); - } + xas_set_order(&xas, idx, folio_order(folio)); + xas_store(&xas, shadow); + folio->swap.val = 0; folio_clear_swapcache(folio); address_space->nrpages -= nr; @@ -252,6 +223,11 @@ void clear_shadow_from_swap_cache(swp_entry_t entry) xas_set_update(&xas, workingset_update_node); + /* + * On unmap, it may delete a larger order shadow here. It's mostly + * fine since not entirely mapped folios are spiltted on swap out + * and leaves shadows with order 0. + */ xa_lock_irq(&address_space->i_pages); if (xa_is_value(xas_load(&xas))) xas_store(&xas, NULL); From patchwork Tue Mar 26 18:50:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24187C54E67 for ; Tue, 26 Mar 2024 19:04:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA1426B009C; Tue, 26 Mar 2024 15:04:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A03346B009D; Tue, 26 Mar 2024 15:04:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 856026B009E; Tue, 26 Mar 2024 15:04:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6D6CF6B009C for ; Tue, 26 Mar 2024 15:04:52 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 053C9C0DC8 for ; Tue, 26 Mar 2024 19:04:52 +0000 (UTC) X-FDA: 81940117224.29.99B8A18 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf20.hostedemail.com (Postfix) with ESMTP id 1623C1C001E for ; Tue, 26 Mar 2024 19:04:49 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BW3BWmia; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479890; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+ERJb6WoHkZLLvZFbb96DVZc9OxyF4ifaYeM2AyqivE=; b=FkIvMbqjTEfQaSpJFL1jXmsgT7pFDQjvXNCDGstqmb9+sm48Vbyu2JVDE7Fu2CyC1FON+m gY5NWqiJ5UkSYODTeoBhUHg88ZMQg7j9eS4nHzXTKKHME3fwZFj6UHnT0WMfAjxLifOfGN /IBTPc7bW1NUffrZsrAimMbGc59YKAE= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BW3BWmia; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479890; a=rsa-sha256; cv=none; b=OuuXoVMiAG9hx4PqtrdaNkw8jGk0Ze5Dqr6yH4ifZUMY6FyI1ohALLf5LEbroWMqsZxy1+ YmfF+Ixn5mF01BhH0QekJX46Mdv4WbKc1sT5KrZJGFsD3nBLK7E46FEJ1RL3lOXfoV8WCw Z4V4VPVSr3ywkMaEk1Kr4yynVTgQyPE= Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-6e6b6f86975so3800283b3a.1 for ; Tue, 26 Mar 2024 12:04:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479888; x=1712084688; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=+ERJb6WoHkZLLvZFbb96DVZc9OxyF4ifaYeM2AyqivE=; b=BW3BWmiamQo23QGHGK53gvQ+dQJNEBAIYc5jDnilLVBrM2+n2AMYd49mznQcZ4P3EJ wHS9wLawAEihBbYtLPXqzc3JvxpIW5kaVI+2AlZdkDUj9W7VbxSq+1Jiu6SVyrToSLWB 5OeK06I/6B+uQoIFqS3cNkbS7unazuiegM9TGjs4c6DYuakgRsFR0g2Ad6bPQQIK8xjN l1D5U9CgbmfaJgmCD7Ipn58bOuX3WqDE5uTHvm7CIxH6Dp7VEmXuD9KyT04+I95UdZ/t J87LDJJeuLocJ06v59Zz1N5ux/u9I6+bBZrvkhzxYMmlz0wBH9hSu7JZahxiEByBVEah T1NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479888; x=1712084688; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+ERJb6WoHkZLLvZFbb96DVZc9OxyF4ifaYeM2AyqivE=; b=pJgKSax/D1qhy8JlKkL8TMT8glEr7k/MyCctsgDZKQVayCbQpW1TS9o3eIwAue2G8A 7cuU7wQbYUG5+ZrV8es+DdqkoOO57ldo/c46W1h3dqFUFGOOB7IQwLmumzG9OlCc/sAG qucbfXR0JuWvQpCHdUfaagyFpLYTI76Ez4f26PyEjbTzGT5fl5wqWSWIac1dOB8zFqSB f1q/HhmA6psPo2wCCMUWhijZBM6LhKXsC81OyaSD9hqC2KLyZJ1Umplk8gnnGEHn5/2k j66IKrlqsThgrcOe0nqgcRV7q/k1FHNUqzBg8PKjBsKgDMZyoUbLAnyQ3FfkiA1XtICL RSqA== X-Gm-Message-State: AOJu0YwFhh0JIMeg00ADERMhqNQUoQYvVVuR/pgYSWQto8i62xOFAgx1 qT5pu9L4S4+atiNA/BBkqO5WgT2hKdsggUJHs9uJMPesdoaAafKJQsHlxlqH4Y+YntIq X-Google-Smtp-Source: AGHT+IFI5c1XPfK9gSzn3T3e+978jVi9idobPChm0l6H2mPesS3FLokLOM//jRLMf4TOY19ha0SPPg== X-Received: by 2002:a05:6a00:701e:b0:6ea:c04c:71cb with SMTP id lf30-20020a056a00701e00b006eac04c71cbmr1408795pfb.3.1711479888011; Tue, 26 Mar 2024 12:04:48 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:47 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 07/10] mm/swap: rename __read_swap_cache_async to swap_cache_alloc_or_get Date: Wed, 27 Mar 2024 02:50:29 +0800 Message-ID: <20240326185032.72159-8-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1623C1C001E X-Stat-Signature: pxukk5me5ihpqa7xi3mmpaj6yugnztxz X-HE-Tag: 1711479889-701623 X-HE-Meta: U2FsdGVkX188Np27VHtqjNoUR/+6jnpsrxq93ekWgkR/UDNYxBhchYnkKLCblDO8oqgPDO4Niel1EJV2It6crXqr0E2k6pYgZgYWqZ/JFl2Bb1A1xp+cXXgFOnDeZ227VmcG3b0jRvFJXkV2H/9MWjHiMVcro35l1kfS4c8G0/XfhpxZ/gn4mLOPNksCpkLj3vofuuf3nF7BXN/rBS0rz/lkNPfYPjznl4zTpmSyACeQnkHGUo5AbKoTRrxLdiCQOsmErHnBHLLBPwwAqGxajJWftd2u3NTagSHUniMAmuF7aBfQiCys2gpzJTTD2S3RIs3hv/AwaPwivSzx6mlXMHQ7hMTe/YIniWm3ua0RlgWJzOUL6S6Gecad5wPwQEZ9qXdD60EzdmUSdQoz+llQ7/CPmq+yyPH2CuelsRsqIonZ9wZghGMt32OLATfkuhFX7s9tcsn2ry94zxVxqn6+IvN+Jw1YpCmFo9Lop54lGl1FgWVBtcxrLwg469vssJyxk/d2fAl54Bx3dFUl9DUbi2F4usyA36c2fERroBqSOVpxahLv43Zoe06N2Ce7g9YAGrHvEFPzxlAGkH3EbdXd1OPa97EQ1xbHqPoUtRDmDckn/KC9sJK4XIwhoSYwUDGvPheUcYX4eB3PeQn/uoqptvhsOyCa8oyjs5ZchitzhzfybLBdfUAy3Km6lolklOZuYMO8Uq6B8GMnaWZn+FRWlEvpFIgNoeIKvWbp+ojsijE7WKLhrSLQr/WZVnB4FbJvFFv799g/PQjKTDnKzN9neSThJ0NOd5BQ2PmNwss7i7QPV+YX8mIRIoTIXjdb6tbejB7Zy/CAlXKOBaLv7+EvNgpBjZ+Z3pJdd3eK4b+G0m9o2+UPz559Yo78ObZPw149T10PBAYKF1iCiSPiIDPljhBPoZRdYxxm57EouU83HZkEDUqJCtk5H0+rv12au4TMh6cD1t6/5b8B31JRIcW N+139dVq QUaABmGLUoclaCvc0OUops7fViHRxCiSA9okxoSgVkuZFIzzm2MdiXEYWDoojNLHwwvPnovHQkl2lTVrw4BGQwVZbXWykJfNllppIP7amOqtK3dSG4oHVDhLYUeZk5gvg0Tbr7njutGsilPIPTmZrRp6nHGX2J0gH0F7Y+t4z7flSN41kjI8YvpulPZAsQ7fLL8GT3yQOY92YlU6Al1nyBsz9EyvW9WzTjHGQhnVcQ3t/p8UCc1xyc4IJN9MXgYTmLQOwC938R18bosRaixYFD5P0dY15DrUJFWtCbCB3i5HeCtS54tGzcce7+JOlGkEzDjiKCHXmlyIDG4LbbINUl/pPY8DhZ0KG2lnXUC0m3DqCD/JmpJJlD9MXHUFeg9+ABju29/IyQ0UW3LNjcWrmEfgLbNEForLDe3vOBGrQ2k5zoTlnb1Zz3Actpaz1IIjeZFeh8nOmiXsiylnagHPgHpQ8BLeag+bne3N6f/pcOs3nEmsNJyKbnjQT3Uqyg+GyDTeNZ3slFmNywsZqpAsRtsJZ1q0sTATY4EqOluI9nX1upeNzB3hOEGnPSbO2KC5Vt0itFArpJ2doxC9n5xUMW4kpLIJVfVTgWsgl9OY5SGyxif5hGVM81r9jmHZo5rwW6Bis3A090Jnqlmt+XDznuaS5OTxQrAWRmq0vTrRxeS/IBsU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song __read_swap_cache_async is widely used to allocate and ensure a folio is in swapcache, or get the folio if a folio is already there. Rename it to better present the usage. Signed-off-by: Kairui Song --- mm/swap.h | 2 +- mm/swap_state.c | 22 +++++++++++----------- mm/swapfile.c | 2 +- mm/zswap.c | 2 +- 4 files changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 7721ddb3bdbc..5fbbc4a42787 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -48,7 +48,7 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping, struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, struct swap_iocb **plug); -struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, +struct folio *swap_cache_alloc_or_get(swp_entry_t entry, gfp_t gfp_flags, struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, bool skip_if_exists); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, diff --git a/mm/swap_state.c b/mm/swap_state.c index caf69696f47c..cd1a16afcd9f 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -385,7 +385,7 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping, return folio; } -struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, +struct folio *swap_cache_alloc_or_get(swp_entry_t entry, gfp_t gfp_mask, struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, bool skip_if_exists) { @@ -443,12 +443,12 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, goto fail_put_swap; /* - * Protect against a recursive call to __read_swap_cache_async() + * Protect against a recursive call to swap_cache_alloc_or_get() * on the same entry waiting forever here because SWAP_HAS_CACHE * is set but the folio is not the swap cache yet. This can * happen today if mem_cgroup_swapin_charge_folio() below * triggers reclaim through zswap, which may call - * __read_swap_cache_async() in the writeback path. + * swap_cache_alloc_or_get() in the writeback path. */ if (skip_if_exists) goto fail_put_swap; @@ -457,7 +457,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * We might race against __delete_from_swap_cache(), and * stumble across a swap_map entry whose SWAP_HAS_CACHE * has not yet been cleared. Or race against another - * __read_swap_cache_async(), which has set SWAP_HAS_CACHE + * swap_cache_alloc_or_get(), which has set SWAP_HAS_CACHE * in swap_map, but not yet added its folio to swap cache. */ schedule_timeout_uninterruptible(1); @@ -505,7 +505,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * the swap entry is no longer in use. * * get/put_swap_device() aren't needed to call this function, because - * __read_swap_cache_async() call them and swap_read_folio() holds the + * swap_cache_alloc_or_get() call them and swap_read_folio() holds the * swap cache folio lock. */ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, @@ -518,7 +518,7 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct folio *folio; mpol = get_vma_policy(vma, addr, 0, &ilx); - folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, &page_allocated, false); mpol_cond_put(mpol); @@ -634,7 +634,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, blk_start_plug(&plug); for (offset = start_offset; offset <= end_offset ; offset++) { /* Ok, do the async read-ahead now */ - folio = __read_swap_cache_async( + folio = swap_cache_alloc_or_get( swp_entry(swp_type(entry), offset), gfp_mask, mpol, ilx, &page_allocated, false); if (!folio) @@ -653,7 +653,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, lru_add_drain(); /* Push any new pages onto the LRU now */ skip: /* The page was likely read above, so no need for plugging here */ - folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, &page_allocated, false); if (unlikely(page_allocated)) { zswap_folio_swapin(folio); @@ -809,7 +809,7 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, continue; pte_unmap(pte); pte = NULL; - folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, &page_allocated, false); if (!folio) continue; @@ -829,7 +829,7 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, lru_add_drain(); skip: /* The folio was likely read above, so no need for plugging here */ - folio = __read_swap_cache_async(targ_entry, gfp_mask, mpol, targ_ilx, + folio = swap_cache_alloc_or_get(targ_entry, gfp_mask, mpol, targ_ilx, &page_allocated, false); if (unlikely(page_allocated)) { zswap_folio_swapin(folio); @@ -855,7 +855,7 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, pgoff_t ilx; mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); - folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, &page_allocated, false); mpol_cond_put(mpol); diff --git a/mm/swapfile.c b/mm/swapfile.c index bafae23c0f26..332ce4e578e8 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1249,7 +1249,7 @@ static unsigned char __swap_entry_free_locked(struct swap_info_struct *p, * CPU1 CPU2 * do_swap_page() * ... swapoff+swapon - * __read_swap_cache_async() + * swap_cache_alloc_or_get() * swapcache_prepare() * __swap_duplicate() * // check swap_map diff --git a/mm/zswap.c b/mm/zswap.c index 9dec853647c8..e4d96816be70 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1126,7 +1126,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* try to allocate swap cache folio */ mpol = get_task_policy(current); - folio = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, + folio = swap_cache_alloc_or_get(swpentry, GFP_KERNEL, mpol, NO_INTERLEAVE_INDEX, &folio_was_allocated, true); if (!folio) return -ENOMEM; From patchwork Tue Mar 26 18:50:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604900 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA6BBC6FD1F for ; Tue, 26 Mar 2024 19:04:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 410896B0082; Tue, 26 Mar 2024 15:04:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 39A1C6B009D; Tue, 26 Mar 2024 15:04:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19E4F6B009E; Tue, 26 Mar 2024 15:04:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F3B136B0082 for ; Tue, 26 Mar 2024 15:04:56 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D2BC51C03BC for ; Tue, 26 Mar 2024 19:04:56 +0000 (UTC) X-FDA: 81940117392.28.96648C5 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf10.hostedemail.com (Postfix) with ESMTP id C462FC0002 for ; Tue, 26 Mar 2024 19:04:54 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mN2FX4FD; spf=pass (imf10.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479894; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eNwcqhEjkeWS6nrrKT5ddXoMcSlzEQZsZcHDot/JHBw=; b=Yuqme4AnyoocQxxufBoW9fTv9dwa0Y5znk6731g5AZ3bbiT1T7gGWxFb7xF+SMLLafFoSY BkJpIb+RGr6IXbOSL5M4KyymwEk1F4n7rOrId6xZYguV6WLmi5nD/DLl88GgtjTRmjhPlV iqI7uQXnInFTiBUvStLwTSYynELK40I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479894; a=rsa-sha256; cv=none; b=J+KiZGaSY0cXfoIH7/0lUefeiFL0bxcx8STIbI9w/yUbXPW+u+Bq+dvWY4AYo09ddIKjqd eSZmKyT+7VDmwMf40kYJQo3YfmPcExQuHd7T/7qrIxl0VLok1DhWc0h00iJqsl4efMNh0a OJP/7nSBKVjWk5+QPAh0+ofLVb7RehY= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mN2FX4FD; spf=pass (imf10.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-5e152c757a5so3019948a12.2 for ; Tue, 26 Mar 2024 12:04:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479893; x=1712084693; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=eNwcqhEjkeWS6nrrKT5ddXoMcSlzEQZsZcHDot/JHBw=; b=mN2FX4FDK0QxXbqRQz5oETa2j0rsdpAcYLf/pC16qDVLTpY8PizOxvtIVhS/ggwS2Q 5LG8oOcaun8g9QRo/HuQXAf7UMfQGCcB6AdEebQTp7qXgLPHbPWq4CnOHWYXFRUnciJp BXEcAVJLeTqTfglVqwH+qt9VW53+53nr0msHumq1e3ncACGairq71hx293B80pigxi+p 6NZc0Qd+/lEaTXEWdRwv+AUZfhyxhM+1xfF+M6f4gCx+YuFI2CIX+SWgNqnn/oFH0RIR kTYTv7pRAqYWO/fZ3dZlYgYK+A/+JAbt4RXkHfDgokZbjkab7Yb+IsrvqucwIotMIXnf jtcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479893; x=1712084693; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=eNwcqhEjkeWS6nrrKT5ddXoMcSlzEQZsZcHDot/JHBw=; b=J8DvWHR+9vJ4R4rqiYBRIvIA03O77rRrOqTip+TfjL4g+BeQv4hj6/tltmmvKz5FZJ 9MQEu97SB3rtjHDi8hTIafmEk1DguT0Ejz6XaUHbkEPHhxtiJmt8nsQUHCS/YTUvRQ8Q GS07X62UC8CmyOmSvHFRuOwczhzWBOCxjyLcHVxG+jibWRJ8TuqvFr1WR9halvxN/nOQ Z3ivPHg3LvoUvz26rtvuGh70FJfws2KI82JBPBAkIyRYplObbiNpMQq1HrOCG8rJJxjB LgPrxmThT5ufEFjkosnBqNkSGYp/KXe1URRjUqRmjwRiVm33MZvxfJeAl8hCy2U/2Xkt /Zng== X-Gm-Message-State: AOJu0YyZylsitE4NgjPSLkNJvUqhIrkJxd/juHXT7h/OtW1seHdtCSHt pxxZSN+r1I5suRjCt0VHjXfJsKkSt90t9GFUit8RlZpZOeBu1lVzqGDlKCLbSxYRnTK+ X-Google-Smtp-Source: AGHT+IE1uRgBJe04k5mjQVIO8SkGfv0fgqgAL+aS6S8PniSAjL5L4mwgIBGMDQTd4YDhvGNyjqSaCQ== X-Received: by 2002:a05:6a20:77a4:b0:1a3:6ed2:ee27 with SMTP id c36-20020a056a2077a400b001a36ed2ee27mr498166pzg.16.1711479892445; Tue, 26 Mar 2024 12:04:52 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:51 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 08/10] mm/swap: use swap cache as a synchronization layer Date: Wed, 27 Mar 2024 02:50:30 +0800 Message-ID: <20240326185032.72159-9-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: C462FC0002 X-Rspam-User: X-Stat-Signature: tw6tkwghc5kwtypra1jqttuk893imow8 X-Rspamd-Server: rspam03 X-HE-Tag: 1711479894-934975 X-HE-Meta: U2FsdGVkX1/2gAfaR4/gU2npDRmOnsU668pz3iWB9HYE6xWqy3pzSjsUCIzETb6S/80rHNpo0sysEInWRMkixh+xIILCe3q/RQjB1jSW797obQ7ZkkUp74hi+yqpl7fYInxulkWAUHre+ThQM60yIiv1gyE1qgFjfS7RV+GhLBgyOqVg0Pu4MrJGASzoS9+TuKCPl/3a8/Jw/Y78NEGEXde7yBEu5rNO014rH/mR2rIaUZfuQz3azgw2aFu4FkYz5e9ZQLBj/+jRexLPaafZBmhMgP0Y46vYNA9b1otyuczruAjf3UDXzNAOTFP09jmPqBN5JUzBKXNsygNzNC4N1X1WCHO2yhgvb0es+1rUSzPN2XwRDUUlhQsqSzPaKoDWhifZH0tn0OUzQ4O2taZemGWFYeR7OIrN6sjEv0WqPclwSHS2hdJ1epFK8oPxihSfqa32PWHBSPWBq6+QkjPY1kcK1r0GRkb8XH+Ajp2Qk3zSKNnkAlSzDDV2AKCTL9vVJFoGuR5txcf3S1cRnYeWM02lFjAaIqWhuXq5KlLkG8zf/IwqDCGR0RoqcGS//Ip/+YsBlCqFNvv65lNWGq85IwFbNkmTcNp1xeBduOGppEDqOqgSOZ/jnNKPssEhh1/1KVB6zobb73sgVgVFvv3vQPYvuNHE2SyxAUKghatJEB1R2DBCNjfM9NTwowUMpFKPmr98AhhbYgkEx+v4FK5hp+b+XPUf53twU0qVLAXpciLKqE2NbydDQT8PtCfbvX/VCuMPaNhCjHMenG0Cdn2q85oZ/dQVL3mpsHEL8pS2kiNymAESJcKPotUYRQKTmZcjdzVQmj5Qx8omg4bSYrJlyAhUSSm8apo5B4WXUlZnQbgvI5oKADTLRHAv9wYOxaPeMVDCL0rkTNoKV7xmhpZHWg4/Motg5Rh3fu1kb90odninpbQrbbTTd+/tmbGNke5R9/WAxw66sSxgeQd4Zsr nurQiqAc W2gwLlsl24VMe9+ITMQdpgCAegrES+8oZW3ykpIB9rW07mwoj/YBketPMeNHESFllJkAdhhRto1NdWJ6Mjm8NZRyVgA/tzvwzbfAVTnYg0WA7S5vGbtWCzI4cpjTwwJ/Ctp3H8CDI3j2qus0jQA4GF14nzA3XB9hHywjsmi4nN5aufbpl6aXKiomst3xYQvSHVFVVCtN0RbVlx0rfYlms2pKYFS6hC8eZ1fcvnFR7A/HpVzi+csamBzVHvf8HHB+DNvGEIRC9aluXB0V58r4sXRpBYOviBQSD3CLCQMo8ZviyvI6pq36U087SQrILwmJj8VeGyHxzJAv1GVbXqm3v5ha71oLl4KVJDkQVLN1j/IKcoM+4TTxLSzzY9Teque/sdEjOMLQ4A6CU78phc2/45wnAq4cYP8hqxlTjFGeMpgMZ5P5XY1/kn0Ioww1w41HUZb9F7wgq+d0xYicLQ751lP1dTxQWSgVQNQ+Z4AwRunqDyo7wmst+cdlnS6iXK/45QTAkZ8UDyC80Tl/VUxTv2l90LPyj0qoWzfZPVk6nxfxHEaIsn1IgmFJmI/wIBVBlAqmtSYEKJrkzu/xXxJwaVEchXYYCj1399wWtv6Oorb1kkDeWePTp59a1+sS7ZKbn3NUX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Rework how swapins are synchronized. Instead of spinning on the swap map, simply use swap cache insert as the synchronization point. Winner will insert a new locked folio, loser will just get the locked folio and wait on unlock. This way we don't need any extra mechanism and have a unified way to ensure all swapin are race free. Swap map is not removed, the HAS_CACHE bit is still set but updated with a particular order and stay eventually consistent with xarray state (it works fine with slot cache reservation), it will mostly be used for fast cache state look up. Two helpers now can be used to add a folio to swapcache: - swap_cache_add_or_get for adding a folio with entry being used (swapin). - swap_cache_add_wait for adding a folio with a freed entry (swapout). swap_cache_add_or_get add a folio to swap cache, it return NULL if folio is already swapped in or hitting OOM, it follows these steps: 1. Caller must ensure the folio is new allocated, this helper lock the folio. 2. Try to add the folio to Xarray (add_to_swap_cache). 3. If (2) success, try set SWAP_HAS_CACHE with swapcache_prepare. This step will now only fail if the entry is freed, which indicate the folio is swapped in by someone else, and if so, revert above steps and return NULL. 4. If (2) failed, try look up and return the locked folio. If a folio is returned, caller should try lock the folio and check if PG_swapcache is still set. If not, racer is hitting OOM or the folio is already swapped in, this can be tell easily (by checking page table for page table). Caller can bail out or retry conditionally. 5. If (4) failed to get a folio, the folio should have been swapped in by someone else, or racer is hitting OOM. And swap_cache_add_wait is for adding a folio with a freed entry to swap cache (for swapout path). Because swap_cache_add_or_get will revert quickly if it accidentally added a folio with freed entry to swapcache, so swap_cache_add_wait will simply wait on race. To remove a folio from swap cache, one have to following these steps: 1. First start by acquiring folio lock. 2. Check if PG_swapcache is still set, if not, this folio is removed already. 3. Call put_swap_folio() to clear SWAP_HAS_CACHE flags in SWAP map first, do this before removing folio from Xarray to ensure insertions can successfully update SWAP map. 4. Remove folio from Xarray by __delete_from_swap_cache. 5. Clear folio flag PG_swapcache, unlock and put it. Or just call delete_from_swap_cache after checking the folio is still PG_swapcache set. Note between step 3 and step 4, an entry may get loaded into swap slot cache, but this is OK because swapout will uses swap_cache_add_wait which wait for step 4. By using swap cache as the synchronization for swapin/swapout, this help removed a lot of hacks or fixes for the synchronization: schedule_timeout_uninterruptible(1) introduced by (just wait on folio): - commit 13ddaf26be32 ("mm/swap: fix race when skipping swapcache") - commit 029c4628b2eb ("mm: swap: get rid of livelock in swapin readahead") skip_if_exist introduced by (now calls always return, it never waits inside): - commit a65b0e7607cc ("zswap: make shrinking memcg-aware") and the swapoff workaround by (swap map is now consistent with xarray, and slot cache is disabled, so only need to check in swapoff now): - commit ba81f8384254 ("mm/swap: skip readahead only when swap slot cache is enabled") Test result of sequential swapin/out of 30G zero page on ZRAM: Before (us) After (us) Swapout: 33713283 33827215 Swapin: 40954646 39466754 (+3.7%) Swapout (THP): 6921176 6917709 Swapin (THP) : 40891953 39566916 (+3.3%) Signed-off-by: Kairui Song --- mm/shmem.c | 5 +- mm/swap.h | 18 ++-- mm/swap_state.c | 217 +++++++++++++++++++++++++----------------------- mm/swapfile.c | 13 ++- mm/vmscan.c | 2 +- mm/zswap.c | 2 +- 6 files changed, 132 insertions(+), 125 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 0aad0d9a621b..51e4593f9e2e 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1512,9 +1512,8 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) if (list_empty(&info->swaplist)) list_add(&info->swaplist, &shmem_swaplist); - if (add_to_swap_cache(folio, swap, - __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN, - NULL) == 0) { + if (!swap_cache_add_wait(folio, swap, + __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN)) { shmem_recalc_inode(inode, 0, 1); swap_shmem_alloc(swap); shmem_delete_from_page_cache(folio, swp_to_radix_entry(swap)); diff --git a/mm/swap.h b/mm/swap.h index 5fbbc4a42787..be2d1642b5d9 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -34,23 +34,20 @@ extern struct address_space *swapper_spaces[]; void show_swap_cache_info(void); bool add_to_swap(struct folio *folio); void *get_shadow_from_swap_cache(swp_entry_t entry); -int add_to_swap_cache(struct folio *folio, swp_entry_t entry, - gfp_t gfp, void **shadowp); void __delete_from_swap_cache(struct folio *folio, swp_entry_t entry, void *shadow); void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(swp_entry_t entry); +int swap_cache_add_wait(struct folio *folio, swp_entry_t entry, gfp_t gfp); struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); struct folio *filemap_get_incore_folio(struct address_space *mapping, pgoff_t index); - struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, struct swap_iocb **plug); struct folio *swap_cache_alloc_or_get(swp_entry_t entry, gfp_t gfp_flags, - struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, - bool skip_if_exists); + struct mempolicy *mpol, pgoff_t ilx, bool *folio_allocated); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, @@ -109,6 +106,11 @@ static inline int swap_writepage(struct page *p, struct writeback_control *wbc) return 0; } +static inline int swap_cache_add_wait(struct folio *folio, swp_entry_t entry, gfp_t gfp) +{ + return -1; +} + static inline struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr) { @@ -132,12 +134,6 @@ static inline void *get_shadow_from_swap_cache(swp_entry_t entry) return NULL; } -static inline int add_to_swap_cache(struct folio *folio, swp_entry_t entry, - gfp_t gfp_mask, void **shadowp) -{ - return -1; -} - static inline void __delete_from_swap_cache(struct folio *folio, swp_entry_t entry, void *shadow) { diff --git a/mm/swap_state.c b/mm/swap_state.c index cd1a16afcd9f..b5ea13295e17 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -85,8 +85,8 @@ void *get_shadow_from_swap_cache(swp_entry_t entry) * add_to_swap_cache resembles filemap_add_folio on swapper_space, * but sets SwapCache flag and private instead of mapping and index. */ -int add_to_swap_cache(struct folio *folio, swp_entry_t entry, - gfp_t gfp, void **shadowp) +static int add_to_swap_cache(struct folio *folio, swp_entry_t entry, + gfp_t gfp, void **shadowp) { struct address_space *address_space = swap_address_space(entry); pgoff_t idx = swp_offset(entry); @@ -169,14 +169,16 @@ bool add_to_swap(struct folio *folio) /* * Add it to the swap cache. */ - err = add_to_swap_cache(folio, entry, - __GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN, NULL); - if (err) + err = swap_cache_add_wait(folio, entry, + __GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN); + if (err) { /* - * add_to_swap_cache() doesn't return -EEXIST, so we can safely - * clear SWAP_HAS_CACHE flag. + * swap_cache_add_wait() doesn't return -EEXIST, so we can + * safely clear SWAP_HAS_CACHE flag. */ goto fail; + } + /* * Normally the folio will be dirtied in unmap because its * pte should be dirty. A special case is MADV_FREE page. The @@ -208,11 +210,12 @@ void delete_from_swap_cache(struct folio *folio) swp_entry_t entry = folio->swap; struct address_space *address_space = swap_address_space(entry); + put_swap_folio(folio, entry); + xa_lock_irq(&address_space->i_pages); __delete_from_swap_cache(folio, entry, NULL); xa_unlock_irq(&address_space->i_pages); - put_swap_folio(folio, entry); folio_ref_sub(folio, folio_nr_pages(folio)); } @@ -385,119 +388,123 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping, return folio; } -struct folio *swap_cache_alloc_or_get(swp_entry_t entry, gfp_t gfp_mask, - struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, - bool skip_if_exists) +/* + * Try add a new folio, return NULL if the entry is swapped in by someone + * else or hitting OOM. + */ +static struct folio *swap_cache_add_or_get(struct folio *folio, + swp_entry_t entry, gfp_t gfp_mask) { - struct swap_info_struct *si; - struct folio *folio; + int ret = 0; void *shadow = NULL; + struct address_space *address_space = swap_address_space(entry); - *new_page_allocated = false; - si = get_swap_device(entry); - if (!si) - return NULL; - - for (;;) { - int err; - /* - * First check the swap cache. Since this is normally - * called after swap_cache_get_folio() failed, re-calling - * that would confuse statistics. - */ - folio = filemap_get_folio(swap_address_space(entry), - swp_offset(entry)); - if (!IS_ERR(folio)) - goto got_folio; - - /* - * Just skip read ahead for unused swap slot. - * During swap_off when swap_slot_cache is disabled, - * we have to handle the race between putting - * swap entry in swap cache and marking swap slot - * as SWAP_HAS_CACHE. That's done in later part of code or - * else swap_off will be aborted if we return NULL. - */ - if (!swap_swapcount(si, entry) && swap_slot_cache_enabled) - goto fail_put_swap; - - /* - * Get a new folio to read into from swap. Allocate it now, - * before marking swap_map SWAP_HAS_CACHE, when -EEXIST will - * cause any racers to loop around until we add it to cache. - */ - folio = (struct folio *)alloc_pages_mpol(gfp_mask, 0, - mpol, ilx, numa_node_id()); - if (!folio) - goto fail_put_swap; - - /* - * Swap entry may have been freed since our caller observed it. - */ - err = swapcache_prepare(entry); - if (!err) - break; - - folio_put(folio); - if (err != -EEXIST) - goto fail_put_swap; - - /* - * Protect against a recursive call to swap_cache_alloc_or_get() - * on the same entry waiting forever here because SWAP_HAS_CACHE - * is set but the folio is not the swap cache yet. This can - * happen today if mem_cgroup_swapin_charge_folio() below - * triggers reclaim through zswap, which may call - * swap_cache_alloc_or_get() in the writeback path. - */ - if (skip_if_exists) - goto fail_put_swap; + /* If folio is NULL, simply go lookup the swapcache */ + if (folio) { + __folio_set_locked(folio); + __folio_set_swapbacked(folio); + ret = add_to_swap_cache(folio, entry, gfp_mask, &shadow); + if (ret) + __folio_clear_locked(folio); + } - /* - * We might race against __delete_from_swap_cache(), and - * stumble across a swap_map entry whose SWAP_HAS_CACHE - * has not yet been cleared. Or race against another - * swap_cache_alloc_or_get(), which has set SWAP_HAS_CACHE - * in swap_map, but not yet added its folio to swap cache. - */ - schedule_timeout_uninterruptible(1); + if (!folio || ret) { + /* If the folio is already added, return it untouched. */ + folio = filemap_get_folio(address_space, swp_offset(entry)); + /* If not, either the entry have been freed or we are OOM. */ + if (IS_ERR(folio)) + return NULL; + return folio; } /* - * The swap entry is ours to swap in. Prepare the new folio. + * The folio is now added to swap cache, try update the swap map + * to ensure the entry is still valid. If we accidentally added + * a stalled entry, undo the add. */ + ret = swapcache_prepare(entry); + if (unlikely(ret)) + goto fail_delete_cache; - __folio_set_locked(folio); - __folio_set_swapbacked(folio); - + /* Charge and shadow check */ if (mem_cgroup_swapin_charge_folio(folio, NULL, gfp_mask, entry)) - goto fail_unlock; - - /* May fail (-ENOMEM) if XArray node allocation failed. */ - if (add_to_swap_cache(folio, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) - goto fail_unlock; - + goto fail_put_flag; mem_cgroup_swapin_uncharge_swap(entry); - if (shadow) workingset_refault(folio, shadow); - /* Caller will initiate read into locked folio */ + /* Return new added folio locked */ folio_add_lru(folio); - *new_page_allocated = true; -got_folio: - put_swap_device(si); return folio; -fail_unlock: +fail_put_flag: put_swap_folio(folio, entry); +fail_delete_cache: + xa_lock_irq(&address_space->i_pages); + __delete_from_swap_cache(folio, entry, shadow); + xa_unlock_irq(&address_space->i_pages); + folio_ref_sub(folio, folio_nr_pages(folio)); folio_unlock(folio); - folio_put(folio); -fail_put_swap: - put_swap_device(si); + return NULL; } +/* + * Try to add a folio to swap cache, caller must ensure entry is freed. + * May block if swap_cache_alloc_or_get accidently loaded a freed entry + * and it will be removed very soon, so just wait and retry. + */ +int swap_cache_add_wait(struct folio *folio, swp_entry_t entry, gfp_t gfp) +{ + int ret; + struct folio *wait_folio; + + for (;;) { + ret = add_to_swap_cache(folio, entry, gfp, NULL); + if (ret != -EEXIST) + break; + wait_folio = filemap_get_folio(swap_address_space(entry), + swp_offset(entry)); + if (!IS_ERR(wait_folio)) { + folio_wait_locked(wait_folio); + folio_put(wait_folio); + } + } + + return ret; +} + +struct folio *swap_cache_alloc_or_get(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx, bool *folio_allocated) +{ + struct folio *folio, *swapcache = NULL; + struct swap_info_struct *si; + + /* Prevent swapoff from happening to us */ + si = get_swap_device(entry); + if (!si) + goto out_no_device; + + /* We are very likely the first user, alloc and try add to the swapcache. */ + folio = (struct folio *)alloc_pages_mpol(gfp_mask, 0, mpol, ilx, + numa_node_id()); + swapcache = swap_cache_add_or_get(folio, entry, gfp_mask); + if (swapcache != folio) { + folio_put(folio); + goto out_no_alloc; + } + + put_swap_device(si); + *folio_allocated = true; + return swapcache; + +out_no_alloc: + put_swap_device(si); +out_no_device: + *folio_allocated = false; + return swapcache; +} + /* * Locate a page of swap in physical memory, reserving swap cache space * and reading the disk if it is not already cached. @@ -519,7 +526,7 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, mpol = get_vma_policy(vma, addr, 0, &ilx); folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, - &page_allocated, false); + &page_allocated); mpol_cond_put(mpol); if (page_allocated) @@ -636,7 +643,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, /* Ok, do the async read-ahead now */ folio = swap_cache_alloc_or_get( swp_entry(swp_type(entry), offset), - gfp_mask, mpol, ilx, &page_allocated, false); + gfp_mask, mpol, ilx, &page_allocated); if (!folio) continue; if (page_allocated) { @@ -654,7 +661,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, skip: /* The page was likely read above, so no need for plugging here */ folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, - &page_allocated, false); + &page_allocated); if (unlikely(page_allocated)) { zswap_folio_swapin(folio); swap_read_folio(folio, false, NULL); @@ -810,7 +817,7 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, pte_unmap(pte); pte = NULL; folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, - &page_allocated, false); + &page_allocated); if (!folio) continue; if (page_allocated) { @@ -830,7 +837,7 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, skip: /* The folio was likely read above, so no need for plugging here */ folio = swap_cache_alloc_or_get(targ_entry, gfp_mask, mpol, targ_ilx, - &page_allocated, false); + &page_allocated); if (unlikely(page_allocated)) { zswap_folio_swapin(folio); swap_read_folio(folio, false, NULL); @@ -856,7 +863,7 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, - &page_allocated, false); + &page_allocated); mpol_cond_put(mpol); if (page_allocated) diff --git a/mm/swapfile.c b/mm/swapfile.c index 332ce4e578e8..8225091d42b6 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -149,9 +149,10 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, * in usual operations. */ if (folio_trylock(folio)) { - if ((flags & TTRS_ANYWAY) || + if (folio_test_swapcache(folio) && + ((flags & TTRS_ANYWAY) || ((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) || - ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio))) + ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio)))) ret = folio_free_swap(folio); folio_unlock(folio); } @@ -1344,7 +1345,8 @@ void swap_free(swp_entry_t entry) } /* - * Called after dropping swapcache to decrease refcnt to swap entries. + * Called before dropping swapcache, free the entry and ensure + * new insertion will success. */ void put_swap_folio(struct folio *folio, swp_entry_t entry) { @@ -1897,13 +1899,15 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, } if (!folio) { swp_count = READ_ONCE(si->swap_map[offset]); - if (swp_count == 0 || swp_count == SWAP_MAP_BAD) + if (swap_count(swp_count) == 0 || swp_count == SWAP_MAP_BAD) continue; return -ENOMEM; } folio_lock(folio); folio_wait_writeback(folio); + if (!folio_test_swapcache(folio)) + goto free_folio; ret = unuse_pte(vma, pmd, addr, entry, folio); if (ret < 0) { folio_unlock(folio); @@ -1912,6 +1916,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, } folio_free_swap(folio); +free_folio: folio_unlock(folio); folio_put(folio); } while (addr += PAGE_SIZE, addr != end); diff --git a/mm/vmscan.c b/mm/vmscan.c index 3ef654addd44..c3db39393428 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -732,10 +732,10 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, if (reclaimed && !mapping_exiting(mapping)) shadow = workingset_eviction(folio, target_memcg); + put_swap_folio(folio, swap); __delete_from_swap_cache(folio, swap, shadow); mem_cgroup_swapout(folio, swap); xa_unlock_irq(&mapping->i_pages); - put_swap_folio(folio, swap); } else { void (*free_folio)(struct folio *); diff --git a/mm/zswap.c b/mm/zswap.c index e4d96816be70..c80e33c74235 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1127,7 +1127,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* try to allocate swap cache folio */ mpol = get_task_policy(current); folio = swap_cache_alloc_or_get(swpentry, GFP_KERNEL, mpol, - NO_INTERLEAVE_INDEX, &folio_was_allocated, true); + NO_INTERLEAVE_INDEX, &folio_was_allocated); if (!folio) return -ENOMEM; From patchwork Tue Mar 26 18:50:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74203CD1283 for ; Tue, 26 Mar 2024 19:05:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 043116B009D; Tue, 26 Mar 2024 15:05:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F0B8C6B009E; Tue, 26 Mar 2024 15:05:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5F256B009F; Tue, 26 Mar 2024 15:05:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BB39D6B009D for ; Tue, 26 Mar 2024 15:05:00 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 966041A004B for ; Tue, 26 Mar 2024 19:05:00 +0000 (UTC) X-FDA: 81940117560.11.3AB2A0A Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf05.hostedemail.com (Postfix) with ESMTP id 8BF4B100031 for ; Tue, 26 Mar 2024 19:04:58 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=T+W8wM2S; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479898; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FiBJhEEfHQ3r2jOMm8KhSAJcpn1B/ejykzOxerrHt3Q=; b=q8C98hU3EN8ARiRY8Tb/OmUmYIJiqgJTOnlrpFe2gJ019kyxOtzojg+vwTS4lDD7Ql5ene F2q9DWmPGJksRYs2rEBv/+MgspzwFxhjc3IsHSU5itL1do6sPHfrKMPmZFOhu1vjiJi/cM QBVGTh9x3gT5z5qAiUMzNJJ2go9JE0Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479898; a=rsa-sha256; cv=none; b=xv4SxZsDujdcQCNl+/vux6b21k64/OxisrusFtE/yfkjnJ+6iyTNqjzOI5XL7guYhZsWDt lF4lo+rMXtHts5lcahM2EFX/y/r/FInIsFo2shfrj3hwDBlXhfSup3L+PA1rCNzjCAsM/j uc82cEEUjqDN6X4dAJ38Y68akEThfhQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=T+W8wM2S; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-5ce6b5e3c4eso3062785a12.2 for ; Tue, 26 Mar 2024 12:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479897; x=1712084697; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=FiBJhEEfHQ3r2jOMm8KhSAJcpn1B/ejykzOxerrHt3Q=; b=T+W8wM2SlqKTlwuUDJ5W3dWbKZ8vOnlPiQH9shqWxmp3s6VTQjNOZbRRe7wTqlsSiC ESw1Xy1QD+rvdzGQXzFRfMced8tmORbpb7nz5oJZGqVZi0KuvvYWNdkVWfKmZEMW50Ka sA0HE2I11IuTj9VagwmVr/+HG7IBx1t0ZHls1X3AyvGGfheouDNb53qihRSb5qZb8quo DHhmEcy+wsGP2QnLKOeed5j6ulgQ69yDc5qkD086nsX52zIKK2hI17Ofoq1CirgtO/qp 03NLMGrEK63ePdFDd2a2P57+/k3D2r/eE0tpBBZbRYNPd8W3Oaq2domRKttV+6mlRUSw /kmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479897; x=1712084697; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=FiBJhEEfHQ3r2jOMm8KhSAJcpn1B/ejykzOxerrHt3Q=; b=YoCuBpsI3qT6rSQGvquwHPwL1fGOZ35gPi+7RXWXWZrsbH8w09YiaCPNraPkWgNxjh g23JaPTzByiqtvED41X/5n1o7HPqTyrf5i9ieexMFSdRzwN6dACcOgnn3CFuSHD22/ut F8Ww6IqO6pVCeMnGdiysx2P5f0Ou47IXA95H/lTeOxkTI69D31sIzD1NtEIL2nJB/ocU z0XcpeShDRR4odSPAaYuTjNV+OtZey88ei9hH5hTGZT8arMw80UCuqT1wAUuA/xjH1Gw 58zM1Rpn3UCDra2W27hUkHKoFheSLEJcR/GnNyy/c7lwXcaKgaZ95DMTfnkSTtCm5oQ9 zVPQ== X-Gm-Message-State: AOJu0Yy+/BrUiMp/EVoKIGAVDHPfKi5LdtWu7QjzgHHQU+WcwQ3rOgPJ SCTH37V0s1BJaxo1t0Gz0FZTdub5fmswhXd8STCuSqW+gfSRc7UmEtpo93cupwUNv4Pu X-Google-Smtp-Source: AGHT+IFAystA0TApX/vqWOxEy6ce1dqGKGk+EmA27apR47RybsEuxh6FntAjcGODdXvlJspRq1GUwA== X-Received: by 2002:a05:6a21:3512:b0:1a3:df1a:271e with SMTP id zc18-20020a056a21351200b001a3df1a271emr752917pzb.19.1711479896808; Tue, 26 Mar 2024 12:04:56 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:04:56 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 09/10] mm/swap: delay the swap cache lookup for swapin Date: Wed, 27 Mar 2024 02:50:31 +0800 Message-ID: <20240326185032.72159-10-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: gk3s8nzt8mq7yyendmspk1or5dz5t7pq X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 8BF4B100031 X-Rspam-User: X-HE-Tag: 1711479898-961164 X-HE-Meta: U2FsdGVkX18jqp2QrJqeOnWZUwMFkESpP0uE6j542W5y8JrdN0jLk8qX2avADxUprQ3abEWkgUuA+gcbgC010WWkmKxu6pDtMCw3QJJEchCEPKDcLr04y/88NmpbIoMUZmCl7mbYdpNJi3xKIUh6t5FNh4+030HEgA2Ivf40hj8glYzCN6o2GB8D/OpoDuEtT6E+mLKIUSin3Po232eJjvM383UzQvHr/PdGRApa9977pcfDTafEcA57ERjfGPG6eCJprFOqB1iIppqFXX2x306iWnn6wQTwycM612JfgSDlvSu3OnZcTZr/LH+CnXdYzocFP8r66dHZy5lShGtWLH2zRxToc2ZjvQo785Uw1b+ot61c3cs50wzGsMZRG4rd/6IdrZQeOEi6gKazu8emSs/0yTfOyBV+JWFNyAlub10a8xTBZf8GbSnb1y3yteRNAD6mbE3auSHaf+ge03HWCnG7OI4YM73mP6CmdIzt+FJYoPWLPis/V2c3YzKUw+ZXH47dAgRVtjJSxbH6ReDxU8oyNxqJcRrqgjsW8tm6gUVXC/00Fi2L3WVRjs5ZyXZEhmoWwj/0YlQyUPH2d+5WKSC12eGW3ATyDNLOcv/WESc0Cg5ap5d2D4GlDrqZ20+qg+TtT2XPmEBiHUMooRW7U1Z77yRvlNkCXQq5tROKq276y57S6nnUVm0LfJln+rDukZ38WHwq2M24ii7PMocPRCl4XoZcubScjmzbv14vPGSLumFOI/0Z/XVO4YEUhpLWLC62NC1SIXOwccwyRZYm7xCD/hTR0rtSZ8wU8xNsdnQiFqIRAvxboZ/nGR/8kL54O8NEqz1VIIdEdHNbzn41v4HPfpDJI7qeu+ZEoKH6qvECW7acEEd/DuI28E8rPTadvInDQTy0uyrqGl79KtOnCE0Z/CwebeFQZMlg344lCXAFghkMLbYYMOWPr2sATWb3twwKxnQzVbqscUGXgzf 6c/gma5l gocChJLivo/X0x4UhOwEhPs+zvAB8lqHnsduqFpThFeQ1XiZha424i6K6vzocAqrizgqGd2FgGlD/65FpS+zjImCVE3RVjw1HjHhJg0UHbmctzFHCuQN2DMoGVEhALj/wG96YKfP76oYsiJIPZTqZlzzp9trNy4rzcOCcZKHyUzZJeKOkBeDy7KyXEZpG9A08PbBEMlmn0yhda3fKN7HtRpC9dXBffZ4z40QU1QiqmaIZ1eAhId3IW7+jR4Dme2ZKAg85Qvo0RQ+zgjJrDlMBAWBwgpBIAaKM8NmmA+Era9M3k8repixXljFVG+Rd2/CLNnBJTnALG7Z5ptwv4x0TF3F9MMko1uAIXZ2ECdLX2VA1PbGNkdhBFvPXdL93XPRnL3a4wWnJBal1M5gC8s3AI3Ok10qAr2FHFLuk4KaH393KdGRlCZSzOYQgsZSAxLiqhjN7/FNsKX2oG6BwvjRiUyl6bfoeVCi3rfrJZksSNXRYmsTUFN4nDu2fR6uVk8ZGn6GbI2v8+6I7xjjomct2raj+i2RgeRNUawr+Mc03Dt6GWEHQR4mIk7DUo41PLhSM9IEU+Rh4oUvBl2o7sb7Kn11P6CnLDkH5utieMxYAWMrtjkGWaoQfg4VhvMnuWRcPtLmDG3eEQLSPPdmfmKnMSQI2SQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song From: Kairui Song Currently we do a swap cache lookup first, then call into the ordinary swapin path. But all swapin path will call swap_cache_add_or_get, which will do a swap cache lookup again on race, because the first lookup is racy and could miss the swap cache. If the race happened (could be frequent on busy device), caller have no way of knowing that, not be able to distinguish minor / major fault, and the first lookup is redundant. So try to do swapcache lookup and readahead update late, defer it to swap_cache_alloc_or_get, and make it faster by avoiding lookup if HAS_CACHE flag is not set. This will be less accurate but the later look up will always ensure we never miss a existing swap cache. This provides 100% accuracy swap cache usage info for callers, improve minor / major page fault info, and also improve performance. Test result of sequential swapin/out of 30G zero page on ZRAM: Before (us) After (us) Swapout: 33827215 33853883 Swapin: 39466754 38336519 (+2.9%) Swapout (THP): 6917709 6814619 Swapin (THP) : 39566916 38383367 (+3.0%) Signed-off-by: Kairui Song --- mm/memory.c | 45 ++++++++---------- mm/shmem.c | 39 +++++++--------- mm/swap.h | 16 +++++-- mm/swap_state.c | 122 +++++++++++++++++++++++++++++------------------- mm/swapfile.c | 32 +++++++------ 5 files changed, 141 insertions(+), 113 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 357d239ee2f6..774a912eb46d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3932,6 +3932,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) struct page *page; struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; + bool folio_allocated = false; bool exclusive = false; swp_entry_t entry; pte_t pte; @@ -3991,35 +3992,29 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(!si)) goto out; - folio = swap_cache_get_folio(entry, vma, vmf->address); - if (folio) - page = folio_file_page(folio, swp_offset(entry)); - swapcache = folio; + if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { + folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf, &folio_allocated); + } else { + folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf, &folio_allocated); + } if (!folio) { - if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && - __swap_count(entry) == 1) { - folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); - } else { - folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); - } - - if (!folio) { - /* - * Back out if somebody else faulted in this pte - * while we released the pte lock. - */ - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); - if (likely(vmf->pte && - pte_same(ptep_get(vmf->pte), vmf->orig_pte))) - ret = VM_FAULT_OOM; - goto unlock; - } + /* + * Back out if somebody else faulted in this pte + * while we released the pte lock. + */ + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (likely(vmf->pte && + pte_same(ptep_get(vmf->pte), vmf->orig_pte))) + ret = VM_FAULT_OOM; + goto unlock; + } - swapcache = folio; - page = folio_file_page(folio, swp_offset(entry)); + swapcache = folio; + page = folio_file_page(folio, swp_offset(entry)); + if (folio_allocated) { /* Had to read the page from swap area: Major fault */ ret = VM_FAULT_MAJOR; count_vm_event(PGMAJFAULT); diff --git a/mm/shmem.c b/mm/shmem.c index 51e4593f9e2e..7884bbe28731 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1570,20 +1570,6 @@ static inline struct mempolicy *shmem_get_sbmpol(struct shmem_sb_info *sbinfo) static struct mempolicy *shmem_get_pgoff_policy(struct shmem_inode_info *info, pgoff_t index, unsigned int order, pgoff_t *ilx); -static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp, - struct shmem_inode_info *info, pgoff_t index) -{ - struct mempolicy *mpol; - pgoff_t ilx; - struct folio *folio; - - mpol = shmem_get_pgoff_policy(info, index, 0, &ilx); - folio = swap_cluster_readahead(swap, gfp, mpol, ilx); - mpol_cond_put(mpol); - - return folio; -} - /* * Make sure huge_gfp is always more limited than limit_gfp. * Some of the flags set permissions, while others set limitations. @@ -1857,9 +1843,12 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, { struct address_space *mapping = inode->i_mapping; struct shmem_inode_info *info = SHMEM_I(inode); + bool folio_allocated = false; struct swap_info_struct *si; struct folio *folio = NULL; + struct mempolicy *mpol; swp_entry_t swap; + pgoff_t ilx; int error; VM_BUG_ON(!*foliop || !xa_is_value(*foliop)); @@ -1878,22 +1867,28 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, } /* Look it up and read it in.. */ - folio = swap_cache_get_folio(swap, NULL, 0); + folio = swap_cache_try_get(swap); if (!folio) { - /* Or update major stats only when swapin succeeds?? */ - if (fault_type) { - *fault_type |= VM_FAULT_MAJOR; - count_vm_event(PGMAJFAULT); - count_memcg_event_mm(fault_mm, PGMAJFAULT); - } /* Here we actually start the io */ - folio = shmem_swapin_cluster(swap, gfp, info, index); + mpol = shmem_get_pgoff_policy(info, index, 0, &ilx); + folio = swap_cluster_readahead(swap, gfp, mpol, ilx, &folio_allocated); + mpol_cond_put(mpol); if (!folio) { error = -ENOMEM; goto failed; } + + /* Update major stats only when swapin succeeds */ + if (folio_allocated && fault_type) { + *fault_type |= VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(fault_mm, PGMAJFAULT); + } } + if (!folio_allocated) + swap_cache_update_ra(folio, NULL, 0); + /* We have to do this with folio locked to prevent races */ folio_lock(folio); if (!folio_test_swapcache(folio) || diff --git a/mm/swap.h b/mm/swap.h index be2d1642b5d9..bd872b157950 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -39,7 +39,8 @@ void __delete_from_swap_cache(struct folio *folio, void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(swp_entry_t entry); int swap_cache_add_wait(struct folio *folio, swp_entry_t entry, gfp_t gfp); -struct folio *swap_cache_get_folio(swp_entry_t entry, +struct folio *swap_cache_try_get(swp_entry_t entry); +void swap_cache_update_ra(struct folio *folio, struct vm_area_struct *vma, unsigned long addr); struct folio *filemap_get_incore_folio(struct address_space *mapping, pgoff_t index); @@ -49,16 +50,18 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct folio *swap_cache_alloc_or_get(swp_entry_t entry, gfp_t gfp_flags, struct mempolicy *mpol, pgoff_t ilx, bool *folio_allocated); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, - struct mempolicy *mpol, pgoff_t ilx); + struct mempolicy *mpol, pgoff_t ilx, bool *folio_allocated); struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); + struct vm_fault *vmf, bool *folio_allocated); struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); + struct vm_fault *vmf, bool *folio_allocated); static inline unsigned int folio_swap_flags(struct folio *folio) { return swp_swap_info(folio->swap)->flags; } + +bool __swap_has_cache(swp_entry_t entry); #else /* CONFIG_SWAP */ struct swap_iocb; static inline void swap_read_folio(struct folio *folio, bool do_poll, @@ -151,5 +154,10 @@ static inline unsigned int folio_swap_flags(struct folio *folio) { return 0; } + +static inline bool __swap_has_cache(swp_entry_t entry); +{ + return false; +} #endif /* CONFIG_SWAP */ #endif /* _MM_SWAP_H */ diff --git a/mm/swap_state.c b/mm/swap_state.c index b5ea13295e17..cf178dd1131a 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -300,54 +300,54 @@ static inline bool swap_use_vma_readahead(void) } /* - * Lookup a swap entry in the swap cache. A found folio will be returned - * unlocked and with its refcount incremented - we rely on the kernel - * lock getting page table operations atomic even if we drop the folio - * lock before returning. - * - * Caller must lock the swap device or hold a reference to keep it valid. + * Try get the swap cache, bail out quickly if swapcache bit is not set. */ -struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr) +struct folio *swap_cache_try_get(swp_entry_t entry) { struct folio *folio; - folio = filemap_get_folio(swap_address_space(entry), swp_offset(entry)); - if (!IS_ERR(folio)) { - bool vma_ra = swap_use_vma_readahead(); - bool readahead; - - /* - * At the moment, we don't support PG_readahead for anon THP - * so let's bail out rather than confusing the readahead stat. - */ - if (unlikely(folio_test_large(folio))) + if (__swap_has_cache(entry)) { + folio = filemap_get_folio(swap_address_space(entry), + swp_offset(entry)); + if (!IS_ERR(folio)) return folio; + } - readahead = folio_test_clear_readahead(folio); - if (vma && vma_ra) { - unsigned long ra_val; - int win, hits; - - ra_val = GET_SWAP_RA_VAL(vma); - win = SWAP_RA_WIN(ra_val); - hits = SWAP_RA_HITS(ra_val); - if (readahead) - hits = min_t(int, hits + 1, SWAP_RA_HITS_MAX); - atomic_long_set(&vma->swap_readahead_info, - SWAP_RA_VAL(addr, win, hits)); - } + return NULL; +} - if (readahead) { - count_vm_event(SWAP_RA_HIT); - if (!vma || !vma_ra) - atomic_inc(&swapin_readahead_hits); - } - } else { - folio = NULL; +void swap_cache_update_ra(struct folio *folio, struct vm_area_struct *vma, + unsigned long addr) +{ + bool vma_ra = swap_use_vma_readahead(); + bool readahead; + + /* + * At the moment, we don't support PG_readahead for anon THP + * so let's bail out rather than confusing the readahead stat. + */ + if (unlikely(folio_test_large(folio))) + return; + + readahead = folio_test_clear_readahead(folio); + if (vma && vma_ra) { + unsigned long ra_val; + int win, hits; + + ra_val = GET_SWAP_RA_VAL(vma); + win = SWAP_RA_WIN(ra_val); + hits = SWAP_RA_HITS(ra_val); + if (readahead) + hits = min_t(int, hits + 1, SWAP_RA_HITS_MAX); + atomic_long_set(&vma->swap_readahead_info, + SWAP_RA_VAL(addr, win, hits)); } - return folio; + if (readahead) { + count_vm_event(SWAP_RA_HIT); + if (!vma || !vma_ra) + atomic_inc(&swapin_readahead_hits); + } } /** @@ -485,6 +485,11 @@ struct folio *swap_cache_alloc_or_get(swp_entry_t entry, gfp_t gfp_mask, if (!si) goto out_no_device; + /* First do a racy check if cache is already loaded. */ + swapcache = swap_cache_try_get(entry); + if (swapcache) + goto out_no_alloc; + /* We are very likely the first user, alloc and try add to the swapcache. */ folio = (struct folio *)alloc_pages_mpol(gfp_mask, 0, mpol, ilx, numa_node_id()); @@ -614,7 +619,8 @@ static unsigned long swapin_nr_pages(unsigned long offset) * are fairly likely to have been swapped out from the same node. */ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, - struct mempolicy *mpol, pgoff_t ilx) + struct mempolicy *mpol, pgoff_t ilx, + bool *folio_allocated) { struct folio *folio; unsigned long entry_offset = swp_offset(entry); @@ -644,6 +650,10 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, folio = swap_cache_alloc_or_get( swp_entry(swp_type(entry), offset), gfp_mask, mpol, ilx, &page_allocated); + if (offset == entry_offset) { + *folio_allocated = page_allocated; + folio_allocated = NULL; + } if (!folio) continue; if (page_allocated) { @@ -666,6 +676,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, zswap_folio_swapin(folio); swap_read_folio(folio, false, NULL); } + if (folio_allocated) + *folio_allocated = page_allocated; return folio; } @@ -779,7 +791,8 @@ static void swap_ra_info(struct vm_fault *vmf, * */ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, - struct mempolicy *mpol, pgoff_t targ_ilx, struct vm_fault *vmf) + struct mempolicy *mpol, pgoff_t targ_ilx, + struct vm_fault *vmf, bool *folio_allocated) { struct blk_plug plug; struct swap_iocb *splug = NULL; @@ -818,6 +831,10 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, pte = NULL; folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, &page_allocated); + if (i == ra_info.offset) { + *folio_allocated = page_allocated; + folio_allocated = NULL; + } if (!folio) continue; if (page_allocated) { @@ -842,6 +859,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, zswap_folio_swapin(folio); swap_read_folio(folio, false, NULL); } + if (folio_allocated) + *folio_allocated = page_allocated; return folio; } @@ -854,20 +873,21 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * Returns the folio for entry after it is read in. */ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) + struct vm_fault *vmf, bool *folio_allocated) { struct mempolicy *mpol; struct folio *folio; - bool page_allocated; pgoff_t ilx; mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, - &page_allocated); + folio_allocated); mpol_cond_put(mpol); - if (page_allocated) + if (*folio_allocated) swap_read_folio(folio, true, NULL); + else if (folio) + swap_cache_update_ra(folio, vmf->vma, vmf->address); return folio; } @@ -885,18 +905,22 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, * or vma-based(ie, virtual address based on faulty address) readahead. */ struct folio *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) + struct vm_fault *vmf, bool *folio_allocated) { struct mempolicy *mpol; - pgoff_t ilx; struct folio *folio; + bool allocated; + pgoff_t ilx; mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); folio = swap_use_vma_readahead() ? - swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf) : - swap_cluster_readahead(entry, gfp_mask, mpol, ilx); + swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf, &allocated) : + swap_cluster_readahead(entry, gfp_mask, mpol, ilx, &allocated); mpol_cond_put(mpol); + if (!*folio_allocated && folio) + swap_cache_update_ra(folio, vmf->vma, vmf->address); + return folio; } diff --git a/mm/swapfile.c b/mm/swapfile.c index 8225091d42b6..ddcf2ff91c39 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1455,6 +1455,15 @@ int __swap_count(swp_entry_t entry) return swap_count(si->swap_map[offset]); } +bool __swap_has_cache(swp_entry_t entry) +{ + pgoff_t offset = swp_offset(entry); + struct swap_info_struct *si = swp_swap_info(entry); + unsigned char count = READ_ONCE(si->swap_map[offset]); + + return swap_count(count) && (count & SWAP_HAS_CACHE); +} + /* * How many references to @entry are currently swapped out? * This does not give an exact answer when swap count is continued, @@ -1862,10 +1871,18 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, struct folio *folio; unsigned long offset; unsigned char swp_count; + bool folio_allocated; swp_entry_t entry; int ret; pte_t ptent; + struct vm_fault vmf = { + .vma = vma, + .address = addr, + .real_address = addr, + .pmd = pmd, + }; + if (!pte++) { pte = pte_offset_map(pmd, addr); if (!pte) @@ -1884,19 +1901,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, offset = swp_offset(entry); pte_unmap(pte); pte = NULL; - - folio = swap_cache_get_folio(entry, vma, addr); - if (!folio) { - struct vm_fault vmf = { - .vma = vma, - .address = addr, - .real_address = addr, - .pmd = pmd, - }; - - folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, - &vmf); - } + folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, + &vmf, &folio_allocated); if (!folio) { swp_count = READ_ONCE(si->swap_map[offset]); if (swap_count(swp_count) == 0 || swp_count == SWAP_MAP_BAD) From patchwork Tue Mar 26 18:50:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13604902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54EAEC6FD1F for ; Tue, 26 Mar 2024 19:05:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D880D6B00A0; Tue, 26 Mar 2024 15:05:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D10AB6B00A2; Tue, 26 Mar 2024 15:05:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B62AF6B00A3; Tue, 26 Mar 2024 15:05:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9C3496B00A0 for ; Tue, 26 Mar 2024 15:05:05 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3E5F8120E39 for ; Tue, 26 Mar 2024 19:05:05 +0000 (UTC) X-FDA: 81940117770.23.3892AA4 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf19.hostedemail.com (Postfix) with ESMTP id 7319F1A0013 for ; Tue, 26 Mar 2024 19:05:03 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=brQqp2Zg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711479903; a=rsa-sha256; cv=none; b=V+yR2ygluOb+gzCVEqQzwnfNVbfLupYDjDMdxA4g5I+BT2x5j5yDGSkYPEMeXnn+QHJ4Hq c9h8Ht2kcsCBeuJbOPJkkpdVt4ul6RqmkSvCiQeJV4rjsGchsZp2mBSZLgDRMmfg7byTha wsWmiqhB6SNveLfMzXOZnjElCeNpTPA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=brQqp2Zg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711479903; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tWXkY/VwUtjvuoCiK/XtYUVIg6+3qA3wRWNZvOL3LW8=; b=rr9Oy66R8Q8LnUUgQ23OJdLS9LbgOIRFmAFEORCxG64GP97mvmR0HwE11iQ9Qh7PWlX6qb b8ipP4gFmSpauN7kLjsei+HjiiHGKOnmSOUwv8cNzWHXhTtZ8xiFFvxGivpSgjuNvhkAHr FCFL2YFsfXiM6M+MIk8HSvAeFJtk/80= Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-5cedfc32250so3765170a12.0 for ; Tue, 26 Mar 2024 12:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711479901; x=1712084701; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=tWXkY/VwUtjvuoCiK/XtYUVIg6+3qA3wRWNZvOL3LW8=; b=brQqp2ZgV8IlfBiVGuSv+DbGcmG6oTor8t1cdPbZsz0NMMBYqb3RWnUuEhhc2NPJzd P+G6lCj2E0u2jq3Eu8cOP+M1OtjSAE6bNNm+vJd56Sqg5qGvXQZYp+YcjRrH0HNTmj7q 090nX3fNETDjgVg3Xi6EB6oYj5V24qitQ4Ay9/ffQgb/HLlT95tYVILPrYGTJDHYfcxa Ny7Ra5gX4SIWhXwNN/o0sHMFjB50eTv9xY4KQ+0xfvfv3SJeCwwSJSGzPbIYc06R1gP4 EYp+pwdI2UI5a38AJgZDK7+DIyNaJWgPyBnDBQWH84ZURsuVbVN5BHsYbBprit5CQ3dM LfGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711479901; x=1712084701; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=tWXkY/VwUtjvuoCiK/XtYUVIg6+3qA3wRWNZvOL3LW8=; b=gRDFf68Zt8a+SHYeFVbqOQPKq32dExiT0xIeNH2C5AGKIKTbOMsNTDonHBjevthV8D 7Fo9WFf5Qo/OfQV1oTvENvZ2wFjuKcVtVl65DNkHc+8MBNJNWec2G8ROinoL939vgKHv 4vUtUqmxZBtQNezIFg1LfTzcbHuAPAQfFVCwMgNPskQKI38JW1Io3bKva/TjLAYfLODC GOs4jWs4SRGg3sjS8X8Zf1uM57oaSJmkh3HLor4WM/G4SPzh0ToaRq92ux0YCpbzmDtH dLijYsunm/FX+VmYCImJXQtGAZynnIP4I9p4o8mRBEICybBY15aKd3YllaM/ES+5VkFe iedQ== X-Gm-Message-State: AOJu0YwTRtjWTfYBz81J9jnv1W6atHOPBdqSq2walnf9UEdFwYkq858B df333ZwByKq/FQPpY/8L1+0WtjF2uqIF/ddme07uNHzPGyIkzt28J/SRUfM1j8aVjiBZ X-Google-Smtp-Source: AGHT+IH8mmais3geKY69O252RCFhbpsrIur2CeiaE4pHWZEEsNx+hS+y2YV8F+RXgoCHWzKftRyqmg== X-Received: by 2002:a05:6a21:3284:b0:1a3:68ff:5805 with SMTP id yt4-20020a056a21328400b001a368ff5805mr599508pzb.44.1711479901200; Tue, 26 Mar 2024 12:05:01 -0700 (PDT) Received: from KASONG-MB2.tencent.com ([115.171.40.106]) by smtp.gmail.com with ESMTPSA id j14-20020aa783ce000000b006ea790c2232sm6298350pfn.79.2024.03.26.12.04.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 26 Mar 2024 12:05:00 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: "Huang, Ying" , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [RFC PATCH 10/10] mm/swap: optimize synchronous swapin Date: Wed, 27 Mar 2024 02:50:32 +0800 Message-ID: <20240326185032.72159-11-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240326185032.72159-1-ryncsn@gmail.com> References: <20240326185032.72159-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7319F1A0013 X-Stat-Signature: 94opwu8r3fbbz1f4q9ubgxfgyjhdgcpc X-HE-Tag: 1711479903-690898 X-HE-Meta: U2FsdGVkX18bPpSXeMSeYXpYmRfhiYt/P1Npouosuz1BS3iAwkcDhkZCqFbiUW8ckfJRo3utCw0O55YGRDZYbXIa11dAa75S98mEZpk2SaNQy/IpTHfUWiV+qPFz8a2FIBvbAA9PkCITlO7fzL6bVVAkO004TNuPapqA/46/Xm+x+HF1rWvPJchbnPV2exklrsr+VCDgulIIiLa8BKLNPoJlva5JUs6gQLZwM5IcxPHPukboEBwMXVb1/BS9X3CGtvmwC/KctAIrpjoVK7g7cb1N2/7G3AXOdc280R7jewbWTVr4nYbVV+TynqC0EtZINE6xNflpAaHKokdvSvC1cLrwXGYwtgZjWMXbBfTzC811Xn7qlL+L1GkBgfA5c3sN0rq15jENKrmTlVjb5CZiqGKaza80c/csaWk/I2eJdDLHh3D74y3hwo9bSlck52lrQ6brwunxwn+SHD26+eVRo+Yd7Jg/KAOl1n3OEqdusOZD1V1Hs8C4pBAr+rSO7EKKED0ca8fFxUStWNf0ZuXve6YjsZppgpMpJNuUIPrAqOocL+qeLCr6/XiqhBXRQASUgaP8tcePEZJ2EgraF5aIze4CYvrugS+78MnPxkQqA/kUFJDjrOsgUVYDWfvbOHWM0puXiWmWlaSVJ6MT5LtI299MjdiIunJcs2ZXqWJj73ZfvQ18Vljm0IkDHK/3efKKX06YPEEsSTGZsbV1OzuF8L8OLNf++E+6hSnPTnmLHLZ1dMjzu2q1DntOYEwvptGIMnGikIpf4ClWSiOWAAKIhjG4PPxzOQk33HfcbADj2jV0nJeACbxG8BoRu0AawT+9JK53QNhVp4DtGJGbQbREWNOYmZVyuYcQ/2FhkYl24kT9/moLoo+6lB/7VbO65LKwIK9FTp9Eki6UHj7EolEBgMLsC89jczmTMR0KqkRCTH2oOKgSsw9DKUB09EOVFZn1JcQtmH93DkD3exqwgOj AKK9RDQa qgsspEW4yjkw3zesBWcWKIFBUy+KWwrNPPfngETD+Auhd8ln6h+Hc12iYSTSSgZklzMSvEBK/Nqh4wxQ4HtKlguI83QibjGaUUKV8NIXBdx7EbsY5ozBSCe7s72nbl+d9P87mAXwl6aVXrs2WnLDz/WHC9En6ZEtbXUkdmkgUAMWyzIXqYHSCCIWyiJgOFcWnXFt7WsOneP5EHuHyAfRD5+fX0fF9gNFSs/n7vKaznFpWleZaMdw7Hgyul0KSA94PY5/0EdggKghm8HM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Interestingly the major performance overhead of synchronous is actually from the workingset nodes update, that's because synchronous swap in keeps adding single folios into a xa_node, making the node no longer a shadow node and have to be removed from shadow_nodes, then remove the folio very shortly and making the node a shadow node again, so it has to add back to the shadow_nodes. Mark synchronous swapin folio with a special bit in swap entry embedded in folio->swap, as we still have some usable bits there. Skip workingset node update on insertion of such folio because it will be removed very quickly, and will trigger the update ensuring the workingset info is eventual consensus. Test result of sequential swapin/out of 30G zero page on ZRAM: Before (us) After (us) Swapout: 33853883 33886008 Swapin: 38336519 32465441 (+15.4%) Swapout (THP): 6814619 6899938 Swapin (THP) : 38383367 33193479 (+13.6%) Signed-off-by: Kairui Song --- include/linux/swapops.h | 5 +++- mm/filemap.c | 16 +++++++++--- mm/memory.c | 34 ++++++++++++++---------- mm/swap.h | 15 +++++++++++ mm/swap_state.c | 57 ++++++++++++++++++++++++----------------- mm/vmscan.c | 6 +++++ mm/workingset.c | 2 +- 7 files changed, 92 insertions(+), 43 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 48b700ba1d18..ebc0c3e4668d 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -25,7 +25,10 @@ * swp_entry_t's are *never* stored anywhere in their arch-dependent format. */ #define SWP_TYPE_SHIFT (BITS_PER_XA_VALUE - MAX_SWAPFILES_SHIFT) -#define SWP_OFFSET_MASK ((1UL << SWP_TYPE_SHIFT) - 1) +#define SWP_CACHE_FLAG_BITS 1 +#define SWP_CACHE_SYNCHRONOUS BIT(SWP_TYPE_SHIFT - 1) +#define SWP_OFFSET_BITS (SWP_TYPE_SHIFT - SWP_CACHE_FLAG_BITS) +#define SWP_OFFSET_MASK (BIT(SWP_OFFSET_BITS) - 1) /* * Definitions only for PFN swap entries (see is_pfn_swap_entry()). To diff --git a/mm/filemap.c b/mm/filemap.c index 5e8e3fd26b8d..ac24cc65d1da 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -923,12 +923,20 @@ int __filemap_add_swapcache(struct address_space *mapping, struct folio *folio, pgoff_t index, gfp_t gfp, void **shadowp) { XA_STATE_ORDER(xas, &mapping->i_pages, index, folio_order(folio)); + bool synchronous = swap_cache_test_synchronous(folio); long nr; int ret; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); - mapping_set_update(&xas, mapping); + + /* + * Skip node update for synchronous folio insertion, it will be + * updated on folio deletion very soon, avoid repeated LRU locking. + */ + if (!synchronous) + xas_set_update(&xas, workingset_update_node); + xas_set_lru(&xas, &shadow_nodes); nr = folio_nr_pages(folio); folio_ref_add(folio, nr); @@ -936,8 +944,10 @@ int __filemap_add_swapcache(struct address_space *mapping, struct folio *folio, ret = __filemap_lock_store(&xas, folio, index, gfp, shadowp); if (likely(!ret)) { mapping->nrpages += nr; - __node_stat_mod_folio(folio, NR_FILE_PAGES, nr); - __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr); + if (!synchronous) { + __node_stat_mod_folio(folio, NR_FILE_PAGES, nr); + __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr); + } xas_unlock_irq(&xas); } else { folio_put_refs(folio, nr); diff --git a/mm/memory.c b/mm/memory.c index 774a912eb46d..bb40202b4f29 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3933,6 +3933,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; bool folio_allocated = false; + bool synchronous_io = false; bool exclusive = false; swp_entry_t entry; pte_t pte; @@ -4032,18 +4033,19 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (ret & VM_FAULT_RETRY) goto out_release; - if (swapcache) { - /* - * Make sure folio_free_swap() or swapoff did not release the - * swapcache from under us. The page pin, and pte_same test - * below, are not enough to exclude that. Even if it is still - * swapcache, we need to check that the page's swap has not - * changed. - */ - if (unlikely(!folio_test_swapcache(folio) || - page_swap_entry(page).val != entry.val)) - goto out_page; + /* + * Make sure folio_free_swap() or swapoff did not release the + * swapcache from under us. The page pin, and pte_same test + * below, are not enough to exclude that. Even if it is still + * swapcache, we need to check that the page's swap has not + * changed. + */ + if (unlikely(!folio_test_swapcache(folio) || + (page_swap_entry(page).val & ~SWP_CACHE_SYNCHRONOUS) != entry.val)) + goto out_page; + synchronous_io = swap_cache_test_synchronous(folio); + if (!synchronous_io) { /* * KSM sometimes has to copy on read faults, for example, if * page->index of !PageKSM() pages would be nonlinear inside the @@ -4105,9 +4107,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ if (!folio_test_ksm(folio)) { exclusive = pte_swp_exclusive(vmf->orig_pte); - if (folio != swapcache) { + if (synchronous_io || folio != swapcache) { /* - * We have a fresh page that is not exposed to the + * We have a fresh page that is not sharable through the * swapcache -> certainly exclusive. */ exclusive = true; @@ -4148,7 +4150,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * yet. */ swap_free(entry); - if (should_try_to_free_swap(folio, vma, vmf->flags)) + if (synchronous_io) + delete_from_swap_cache(folio); + else if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); inc_mm_counter(vma->vm_mm, MM_ANONPAGES); @@ -4223,6 +4227,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) out_nomap: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); + if (synchronous_io) + delete_from_swap_cache(folio); out_page: folio_unlock(folio); out_release: diff --git a/mm/swap.h b/mm/swap.h index bd872b157950..9d106eebddbd 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -31,6 +31,21 @@ extern struct address_space *swapper_spaces[]; (&swapper_spaces[swp_type(entry)][swp_offset(entry) \ >> SWAP_ADDRESS_SPACE_SHIFT]) +static inline void swap_cache_mark_synchronous(struct folio *folio) +{ + folio->swap.val |= SWP_CACHE_SYNCHRONOUS; +} + +static inline bool swap_cache_test_synchronous(struct folio *folio) +{ + return folio->swap.val & SWP_CACHE_SYNCHRONOUS; +} + +static inline void swap_cache_clear_synchronous(struct folio *folio) +{ + folio->swap.val &= ~SWP_CACHE_SYNCHRONOUS; +} + void show_swap_cache_info(void); bool add_to_swap(struct folio *folio); void *get_shadow_from_swap_cache(swp_entry_t entry); diff --git a/mm/swap_state.c b/mm/swap_state.c index cf178dd1131a..b0b1b5391ac1 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -86,7 +86,7 @@ void *get_shadow_from_swap_cache(swp_entry_t entry) * but sets SwapCache flag and private instead of mapping and index. */ static int add_to_swap_cache(struct folio *folio, swp_entry_t entry, - gfp_t gfp, void **shadowp) + gfp_t gfp, bool synchronous, void **shadowp) { struct address_space *address_space = swap_address_space(entry); pgoff_t idx = swp_offset(entry); @@ -98,11 +98,12 @@ static int add_to_swap_cache(struct folio *folio, swp_entry_t entry, folio_set_swapcache(folio); folio->swap = entry; - + if (synchronous) + swap_cache_mark_synchronous(folio); ret = __filemap_add_swapcache(address_space, folio, idx, gfp, shadowp); if (ret) { - folio_clear_swapcache(folio); folio->swap.val = 0; + folio_clear_swapcache(folio); } return ret; @@ -129,11 +130,13 @@ void __delete_from_swap_cache(struct folio *folio, xas_set_order(&xas, idx, folio_order(folio)); xas_store(&xas, shadow); - folio->swap.val = 0; folio_clear_swapcache(folio); address_space->nrpages -= nr; - __node_stat_mod_folio(folio, NR_FILE_PAGES, -nr); - __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr); + if (!swap_cache_test_synchronous(folio)) { + __node_stat_mod_folio(folio, NR_FILE_PAGES, -nr); + __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr); + } + folio->swap.val = 0; } /** @@ -393,7 +396,7 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping, * else or hitting OOM. */ static struct folio *swap_cache_add_or_get(struct folio *folio, - swp_entry_t entry, gfp_t gfp_mask) + swp_entry_t entry, gfp_t gfp_mask, bool synchronous) { int ret = 0; void *shadow = NULL; @@ -403,7 +406,7 @@ static struct folio *swap_cache_add_or_get(struct folio *folio, if (folio) { __folio_set_locked(folio); __folio_set_swapbacked(folio); - ret = add_to_swap_cache(folio, entry, gfp_mask, &shadow); + ret = add_to_swap_cache(folio, entry, gfp_mask, synchronous, &shadow); if (ret) __folio_clear_locked(folio); } @@ -460,7 +463,7 @@ int swap_cache_add_wait(struct folio *folio, swp_entry_t entry, gfp_t gfp) struct folio *wait_folio; for (;;) { - ret = add_to_swap_cache(folio, entry, gfp, NULL); + ret = add_to_swap_cache(folio, entry, gfp, false, NULL); if (ret != -EEXIST) break; wait_folio = filemap_get_folio(swap_address_space(entry), @@ -493,7 +496,7 @@ struct folio *swap_cache_alloc_or_get(swp_entry_t entry, gfp_t gfp_mask, /* We are very likely the first user, alloc and try add to the swapcache. */ folio = (struct folio *)alloc_pages_mpol(gfp_mask, 0, mpol, ilx, numa_node_id()); - swapcache = swap_cache_add_or_get(folio, entry, gfp_mask); + swapcache = swap_cache_add_or_get(folio, entry, gfp_mask, false); if (swapcache != folio) { folio_put(folio); goto out_no_alloc; @@ -875,21 +878,27 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, struct vm_fault *vmf, bool *folio_allocated) { - struct mempolicy *mpol; - struct folio *folio; - pgoff_t ilx; - - mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); - folio = swap_cache_alloc_or_get(entry, gfp_mask, mpol, ilx, - folio_allocated); - mpol_cond_put(mpol); - - if (*folio_allocated) + struct folio *folio = NULL, *swapcache; + /* First do a racy check if cache is already loaded. */ + swapcache = swap_cache_try_get(entry); + if (unlikely(swapcache)) + goto out; + folio = vma_alloc_folio(gfp_mask, 0, vmf->vma, vmf->address, false); + swapcache = swap_cache_add_or_get(folio, entry, gfp_mask, true); + if (!swapcache) + goto out_nocache; + if (swapcache == folio) { swap_read_folio(folio, true, NULL); - else if (folio) - swap_cache_update_ra(folio, vmf->vma, vmf->address); - - return folio; + *folio_allocated = true; + return folio; + } +out: + swap_cache_update_ra(swapcache, vmf->vma, vmf->address); +out_nocache: + if (folio) + folio_put(folio); + *folio_allocated = false; + return swapcache; } /** diff --git a/mm/vmscan.c b/mm/vmscan.c index c3db39393428..e71b049fee01 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1228,6 +1228,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!add_to_swap(folio)) goto activate_locked_split; } + } else if (swap_cache_test_synchronous(folio)) { + /* + * We see a folio being swapped in but not activated either + * due to missing shadow or lived too short, active it. + */ + goto activate_locked; } } else if (folio_test_swapbacked(folio) && folio_test_large(folio)) { diff --git a/mm/workingset.c b/mm/workingset.c index f2a0ecaf708d..83a0b409be0f 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -753,7 +753,7 @@ static enum lru_status shadow_lru_isolate(struct list_head *item, */ if (WARN_ON_ONCE(!node->nr_values)) goto out_invalid; - if (WARN_ON_ONCE(node->count != node->nr_values)) + if (WARN_ON_ONCE(node->count != node->nr_values && mapping->host != NULL)) goto out_invalid; xa_delete_node(node, workingset_update_node); __inc_lruvec_kmem_state(node, WORKINGSET_NODERECLAIM);