From patchwork Mon Mar 25 23:50:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56FC7CD11BF for ; Mon, 25 Mar 2024 23:50:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36EBB6B0093; Mon, 25 Mar 2024 19:50:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D1856B0095; Mon, 25 Mar 2024 19:50:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 124966B0096; Mon, 25 Mar 2024 19:50:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E3C506B0093 for ; Mon, 25 Mar 2024 19:50:32 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8A4071A0324 for ; Mon, 25 Mar 2024 23:50:32 +0000 (UTC) X-FDA: 81937208304.20.8B362BD Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id B8C5B2000C for ; Mon, 25 Mar 2024 23:50:30 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=efl+5ABz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3xQ0CZgoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3xQ0CZgoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410630; a=rsa-sha256; cv=none; b=IKIW/htTgCrmT0L5CSS3NzhWf3k9kr1FNkf943AE9y+0Nrzzb/vymEXPQfizRkVGMT43IL /CVcMJYdfXdj61ewZHF9QJqsEr4jwL6ZVliOrcZf6VOH585veuKIjyREmW7ixVP+Q7tt3Q qv09zuumb4qAKpRQcERSeIZrQBm535g= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=efl+5ABz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3xQ0CZgoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3xQ0CZgoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410630; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AaCCk+fj5n3OUjEVk3zw3Mi2iE3P+5rpOrMtgAMtN6o=; b=h8HMWNfTFwdOdtkCl51Ow2lPh6iGhtcVSn034qelAzYY4YqDwUgfaztcLT3tv5NxvqKc1m U+uHZGK1KI9Otu+m+nlHbaAUh4PApDiDVzmscdPgjXoffe2aaB3C9WArIamxNg0VrCwhSS P677QbhkKWT3XWCSUI4bfwQ12VrNtpI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60ff1816749so76925737b3.3 for ; Mon, 25 Mar 2024 16:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410630; x=1712015430; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AaCCk+fj5n3OUjEVk3zw3Mi2iE3P+5rpOrMtgAMtN6o=; b=efl+5ABzGzdcBZdj+Bc5nRkUHaZ6aG9qldmAOTAhldyZW9Wv8bNvns093FZToDII6y fxSFCAIS63VRxIT/241TXqrzZGcarX5j2D1ayXRzOPTO889+JztgEolu/UCiMd9EYg23 aRiTUljTpdr6FozERCyIVSTm1Rerso0LQh0J5TuOf90I+9CqIwnO8pTusen80dreqsg9 D9c8zfpjtP91kk2SVZwEkxF4i/6k7KB849drAYSW2FLfWo6+mJnHU10yTbdAitf9m28e JyEaYFML9Sb22DYLLRB9OWgQayndoi7zfXssMGqhwmkZSQ4b7LLyCQCky9GwJlJlH4o0 CArA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410630; x=1712015430; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AaCCk+fj5n3OUjEVk3zw3Mi2iE3P+5rpOrMtgAMtN6o=; b=X4c8Yt9/MJF0cc8dlEMzWViZ+UEgfUGtGbzG3AC/x2LNe4Mpdc6GBzfQsdbTmO/od4 i94gWrAdJ/JuxyYKJFWB/eXKoF+8gtx7dWEF8yIHztwSwJ3usJl0Ij6zHzc+zDihyc1X 3cQbrxWmFGBHpcHSHwqBESZphrw6J/PGvD63j3NX33iLfWnAzKtC5Qwh62I9XIPXI2YN CqmOdR3PJhuBC+rK6A0Y/sjaTh106RWQc5XMFxGUcQrjtHtEmW5XaWXm1IEP8LqURdaI ehofzTvZrf9LhdjtCnZdqcxmNkA6ItTk6g62CkjlaxmY7jH6g8+fGed31HURtp+h53/r aBtw== X-Forwarded-Encrypted: i=1; AJvYcCWk1XLwi2ZDaPVsHC/ve7sB3w/X6+gaiPCB86jzIkfN1qKfjQ6LJ2CHGYCeiR39FEDy+4ulKltbt/vndUYZ6G+uuLs= X-Gm-Message-State: AOJu0YwFmSpo8bJOHD+70ntgi4z9OiSOW8U1w0bYqaKH5x2KBRqfmUx0 //b+LsO0edgaKpIQVddUDPvsNyFGf1HZ4s02MPY+d4NK9Lq/n64KXqMF2c457Dvunf4Y3DBBPlf N192tfuWc6VR/y5yZjw== X-Google-Smtp-Source: AGHT+IEOvQVqxv2Y0ouP0PCwiv8cpiRAswb4b3lmfWQXPAhEkV469xAOUnqSRda9DyCx6d0T4Sypd2qE7f/9L3CU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:705:b0:dcd:59a5:7545 with SMTP id k5-20020a056902070500b00dcd59a57545mr451342ybt.10.1711410629980; Mon, 25 Mar 2024 16:50:29 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:14 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-7-yosryahmed@google.com> Subject: [RFC PATCH 6/9] mm: zswap: drop support for non-zero same-filled pages handling From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B8C5B2000C X-Stat-Signature: qyad8zc7ok6tm66n6r868k6qk7xeonaw X-HE-Tag: 1711410630-8131 X-HE-Meta: U2FsdGVkX19eaX3gawRTqQ8RF4gRLz+mRfsvSSMfTGmeVlzeRfX4u+9kYwJFBrQIZfwh4N1kLq1Sokyy25LgeYM0KLoh3haCLXz+z50ePGiQAqGAYHJsNjbSyG2y4rsMN63+jFNuZT1fEPUVEoR+VUDNR2HLA5oDHUHKtjWY9uBbxxEyLH3doWZo/qS/Ac93L26TFuW7UluQT6Ic4+Gqxy0HagXuBJONaKqDZPPt5qnr/IQ2tHHViE6lE+c9EL6YfsHmPfrKTXXnElbUmOXiTfIV800Yg9HOKE3qmWpotBBuplZNMyjDp3sojanBlY5rRNcn3NE4PryKMd1uVZFade18ANL3WzmZ7OSR3iKV93GZfs37QmNLQk326PKi+jei5R2WkQLqOebJTOOYNfDi9x5spWvYyLQeTbEla7ReEkAwPSDs1V6yyGgsjA7KcK5W9A8AFr0y5Kx1zxuUaXYI88ie0KQnm1GPojlEQmppmHCiMEUCV/rtaB+n8D/ezoymRKFORAiwKfBG3k4Sre7Dgs3mz3DdXf1DWV2jRAPJvl3JVj09Hh0CiIBiinl3FL+hwC/2cTjs46/1adoLiyxK1xXKrPSPbKB+D632xUV/2KbO+Y1z1RP0bkgBwCvVZj29AMsKg1O72YadNA1Nr9R8ktjyFoip47qx3ul5CUyTpTD0JQq63q84hn68wMBGE3YmtDzWEKP4On5EzHN0Q0dG3hWHf3Cbge+HvlFQalOpZoxNSkBU4uR9PzGZyyIc0IJytjNN/60yBoKak/o82EbV/6c5/WVWZM0h8oX9XegWLsBSHj51V0sRMZ7bU5Xkjs5WKga29PMTiibyjQxKJBWWMDur8XXObt9Sly7lEBhVlgq4Re//IBe3QL0adVMErAj1wj26v/VP7lGF5FRBzTCEQsn+LvMWwitVn/BJwCd8iWcwZZtizQy7tr7wx/0Fh1BiznLgKX4LIamDbzaPkvF w+zec/JS Rwo9EEXYfEScoID0dbsWmlQSDYq2m4NM1eTBOBkLW1qvMCBaF0Wwq9OEVBlpUB1uL+lTmqLcHkM3oc/nEv00Zgm/uppPeshlEXeZgqFmzZhgJz3HEA5YtmJYsNLxYE7lV03FxfVbo8VeImqkRXAU05VnuXaHQy8reAGGGH0YeKHfmqfcabBdGV2zruBMD1pxMssIFIxfzWNs0XKMRh4U6aJq6wWi9y4qIdE4B49HQ5T/lS+ZA3V5P7ELsiHC/onVU524iw7nixp4EcHJXVnhYlIpRVrVyUnoHwlEtoUZqL8U95tLu3hgFlevMf+4Gn5U6U45AB3ZYwtZJNdim+Niy484EiK03o3tLR2hSANACXD+5mpft+2Jqgcf1NLU2SR25R6nqoJnln1mMQQik+sJlH/KfgQLrlgHSAM4g5VzAvPwoc8d+sC72Qr8G9iZjaIfVePOTD6YO+Olwh+t1QOGJYIKlzgpkTruFqjPcsc/9hqm1LNUsv5H1k9Yp8UNfbOD4okUlmlXjK8usnUn76FS0l/jIt0PtApQtu0TjYtZj2pwwPH0JEj0JIUQgAgPigWj/UNJc7SxNKGGDCJJN/+nqQBnHv/aXuH2dTtRUWKUC/KrcIQ5BN6Km0z1iGv+BxKi9DBgeB0Z8aezwrp4MixB+icI9Ql328FcFftXjaZw5HUflRo8K6d57W1BFzLLatG9XAtweduL3Rk4usRrBxWxPbIokr9OqfQ72/fuAjXMoT3NrcDaIwfxGPeFrRg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The current same-filled pages handling supports pages filled with any repeated word-sized pattern. However, in practice, most of these should be zero pages anyway. Other patterns should be nearly as common. Drop the support for non-zero same-filled pages, but keep the names of knobs exposed to userspace as "same_filled", which isn't entirely inaccurate. This yields some nice code simplification and enables a following patch that eliminates the need to allocate struct zswap_entry for those pages completely. There is also a very small performance improvement observed over 50 runs of kernel build test (kernbench) comparing the mean build time on a skylake machine when building the kernel in a cgroup v1 container with a 3G limit: base patched % diff real 70.167 69.915 -0.359% user 2953.068 2956.147 +0.104% sys 2612.811 2594.718 -0.692% This probably comes from more optimized operations like memchr_inv() and clear_highpage(). Note that the percentage of zero-filled pages during this test was only around 1.5% on average, and was not affected by this patch. Practical workloads could have a larger proportion of such pages (e.g. Johannes observed around 10% [1]), so the performance improvement should be larger. [1]https://lore.kernel.org/linux-mm/20240320210716.GH294822@cmpxchg.org/ Signed-off-by: Yosry Ahmed Reviewed-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 76 ++++++++++++++---------------------------------------- 1 file changed, 20 insertions(+), 56 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 0fc27ae950c74..413d9242cf500 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -44,8 +44,8 @@ **********************************/ /* The number of compressed pages currently stored in zswap */ atomic_t zswap_stored_pages = ATOMIC_INIT(0); -/* The number of same-value filled pages currently stored in zswap */ -static atomic_t zswap_same_filled_pages = ATOMIC_INIT(0); +/* The number of zero-filled pages currently stored in zswap */ +static atomic_t zswap_zero_filled_pages = ATOMIC_INIT(0); /* * The statistics below are not protected from concurrent access for @@ -123,9 +123,9 @@ static unsigned int zswap_accept_thr_percent = 90; /* of max pool size */ module_param_named(accept_threshold_percent, zswap_accept_thr_percent, uint, 0644); -/* Enable/disable handling non-same-value filled pages (enabled by default) */ -static bool zswap_non_same_filled_pages_enabled = true; -module_param_named(non_same_filled_pages_enabled, zswap_non_same_filled_pages_enabled, +/* Enable/disable handling non-zero-filled pages (enabled by default) */ +static bool zswap_non_zero_filled_pages_enabled = true; +module_param_named(non_same_filled_pages_enabled, zswap_non_zero_filled_pages_enabled, bool, 0644); /* Number of zpools in zswap_pool (empirically determined for scalability) */ @@ -187,11 +187,10 @@ static struct shrinker *zswap_shrinker; * * swpentry - associated swap entry, the offset indexes into the red-black tree * length - the length in bytes of the compressed page data. Needed during - * decompression. For a same value filled page length is 0, and both + * decompression. For a zero-filled page length is 0, and both * pool and lru are invalid and must be ignored. * pool - the zswap_pool the entry's data is in * handle - zpool allocation handle that stores the compressed page data - * value - value of the same-value filled pages which have same content * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages. */ @@ -199,10 +198,7 @@ struct zswap_entry { swp_entry_t swpentry; unsigned int length; struct zswap_pool *pool; - union { - unsigned long handle; - unsigned long value; - }; + unsigned long handle; struct obj_cgroup *objcg; struct list_head lru; }; @@ -805,7 +801,7 @@ static struct zpool *zswap_find_zpool(struct zswap_entry *entry) static void zswap_entry_free(struct zswap_entry *entry) { if (!entry->length) - atomic_dec(&zswap_same_filled_pages); + atomic_dec(&zswap_zero_filled_pages); else { zswap_lru_del(&zswap_list_lru, entry); zpool_free(zswap_find_zpool(entry), entry->handle); @@ -1377,43 +1373,17 @@ static void shrink_worker(struct work_struct *w) } while (zswap_total_pages() > thr); } -static bool zswap_is_folio_same_filled(struct folio *folio, unsigned long *value) +static bool zswap_is_folio_zero_filled(struct folio *folio) { - unsigned long *page; - unsigned long val; - unsigned int pos, last_pos = PAGE_SIZE / sizeof(*page) - 1; + unsigned long *kaddr; bool ret; - page = kmap_local_folio(folio, 0); - val = page[0]; - - if (val != page[last_pos]) { - ret = false; - goto out; - } - - for (pos = 1; pos < last_pos; pos++) { - if (val != page[pos]) { - ret = false; - goto out; - } - } - - *value = val; - ret = true; -out: - kunmap_local(page); + kaddr = kmap_local_folio(folio, 0); + ret = !memchr_inv(kaddr, 0, PAGE_SIZE); + kunmap_local(kaddr); return ret; } -static void zswap_fill_page(void *ptr, unsigned long value) -{ - unsigned long *page; - - page = (unsigned long *)ptr; - memset_l(page, value, PAGE_SIZE / sizeof(unsigned long)); -} - static bool zswap_check_limit(void) { unsigned long cur_pages = zswap_total_pages(); @@ -1437,7 +1407,6 @@ bool zswap_store(struct folio *folio) struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; struct zswap_entry *entry; - unsigned long value; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1470,14 +1439,13 @@ bool zswap_store(struct folio *folio) goto reject; } - if (zswap_is_folio_same_filled(folio, &value)) { + if (zswap_is_folio_zero_filled(folio)) { entry->length = 0; - entry->value = value; - atomic_inc(&zswap_same_filled_pages); + atomic_inc(&zswap_zero_filled_pages); goto insert_entry; } - if (!zswap_non_same_filled_pages_enabled) + if (!zswap_non_zero_filled_pages_enabled) goto freepage; /* if entry is successfully added, it keeps the reference */ @@ -1532,7 +1500,7 @@ bool zswap_store(struct folio *folio) store_failed: if (!entry->length) - atomic_dec(&zswap_same_filled_pages); + atomic_dec(&zswap_zero_filled_pages); else { zpool_free(zswap_find_zpool(entry), entry->handle); put_pool: @@ -1563,7 +1531,6 @@ bool zswap_load(struct folio *folio) struct page *page = &folio->page; struct xarray *tree = swap_zswap_tree(swp); struct zswap_entry *entry; - u8 *dst; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1573,11 +1540,8 @@ bool zswap_load(struct folio *folio) if (entry->length) zswap_decompress(entry, page); - else { - dst = kmap_local_page(page); - zswap_fill_page(dst, entry->value); - kunmap_local(dst); - } + else + clear_highpage(page); count_vm_event(ZSWPIN); if (entry->objcg) @@ -1679,7 +1643,7 @@ static int zswap_debugfs_init(void) debugfs_create_atomic_t("stored_pages", 0444, zswap_debugfs_root, &zswap_stored_pages); debugfs_create_atomic_t("same_filled_pages", 0444, - zswap_debugfs_root, &zswap_same_filled_pages); + zswap_debugfs_root, &zswap_zero_filled_pages); return 0; }