From patchwork Fri Oct 18 19:25:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13842274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAE31D3E180 for ; Fri, 18 Oct 2024 19:25:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41FFA6B0098; Fri, 18 Oct 2024 15:25:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CAEE6B009A; Fri, 18 Oct 2024 15:25:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2450D6B00AC; Fri, 18 Oct 2024 15:25:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F1CBA6B0098 for ; Fri, 18 Oct 2024 15:25:56 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 962BE1A089E for ; Fri, 18 Oct 2024 19:25:34 +0000 (UTC) X-FDA: 82687702650.19.FA54DBB Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf08.hostedemail.com (Postfix) with ESMTP id 3D613160014 for ; Fri, 18 Oct 2024 19:25:47 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="hRhpWc6/"; spf=pass (imf08.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729279408; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=r0anSA7wmrimpqHZPAHtNpFmhlM7GFXCjTHh2HcmL4E=; b=oQsi6OLIltnhRswmok43yDYUKMgJ4dvIxffMjfO6njQAFQ9Ksnkcug1nVlqJuGkk7lnEc4 6JNcOGuvfvmTu7aGQiXb7+VsSYxYjzdZiWHV3D4VnY7Gyz2zcImjRc03TYRiSI0oiqZ4LM 9z/WQ/gHdPpEluHTG+msT1zMPrwgOdw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729279408; a=rsa-sha256; cv=none; b=mO13BlvmDbsnacedoBWGpyJjiN95USHZ6SUWIMdJK/dbrAPiixPnSr+3oGXc1c6nns7HSU N9PExFINI8dNn3//hB5moB1gBo2gpXc7i2Ods/DDwtNJSZqfxtJWvhZnkBuH0f+BLtn19Y zHzIFvRR/LeXVL5M47u/2vmGY3DKlPA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="hRhpWc6/"; spf=pass (imf08.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-20c714cd9c8so25960925ad.0 for ; Fri, 18 Oct 2024 12:25:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729279553; x=1729884353; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=r0anSA7wmrimpqHZPAHtNpFmhlM7GFXCjTHh2HcmL4E=; b=hRhpWc6/Ns6r7vZycjn+zNxHZkhSrhBbpu9qhw50L8Ypq0YBSdrwMhnTtDqnBz4Jy/ E4nP+d31D8gnobbtDqWnXbb4DrkyRdhoCbod8b4ev+dGsvpPywE8k47U0pw4eKrP89Or azI4dUD4sZC8A6tM0SJwKcZwgcy5ZTN2U5K6yNoxzzuluqk12XmgapyFuD7dgqyPqYPK fSTaBkMhO3hwX85zzSvHENrRugoVL+sLsxHhV/2yNwvhKYWEMON8KqbAt9DVVmA3LsIJ hODHUhGBgWrKOvNvP6qt2l5aOj0f7HKGXV8gRQmbCZbYMIkPtSXnG0kUMseB6m/N7rmQ LChQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729279553; x=1729884353; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=r0anSA7wmrimpqHZPAHtNpFmhlM7GFXCjTHh2HcmL4E=; b=KIjRqrQtozJe1KfK6oFdi6TOCHLNw+T6gZTiMebfVByneikB0KqHCEIz8i4kUz5QHH RPJAG/qBAshBVeSiHy1DAwLWpaZGSVh/SZ93z/ofA4goKYm1nzyta4SlUIGBPnKRG8lK uchSTCC5epK+z0CqGBJC8mYGPWKooUp6Y4hzZmM939MYy3euspkiYmZ7+5Bgu1BVWA8K 0Q7maOUlgHM1wk5iG2cEv/bQniHH9v0PpgDjUtlbiGbsgFkVvO2Yvf9XmeXtymS1+BLV j0KaZy1weeDXB0j982pXMMsD48BSB6Of+wKbHslzhHnNC+D16zIibULShVO7dh8aCyMr /dqg== X-Gm-Message-State: AOJu0YwkR7EO1TqB6bjQhE5M0Dk0Ne7gkCLpUUkXk3CU8UkjQ888XaEy w8tICcs2BfItob6CbSArywtGupHQw5Bl3ezPoa2orOOEUB1gPiiWiZXpFwy7 X-Google-Smtp-Source: AGHT+IH+gcFUNvQJGdCfiZUKmlscDlPE7FoZXrFol71E+UA5WwDV5CwQUWLGvQyUta2qroIk60kOWw== X-Received: by 2002:a17:902:f552:b0:20b:7ec0:ee3e with SMTP id d9443c01a7336-20e5a79f505mr42301125ad.11.1729279552573; Fri, 18 Oct 2024 12:25:52 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([115.171.40.120]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e5a747769sm16284015ad.74.2024.10.18.12.25.49 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 18 Oct 2024 12:25:52 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Chris Li , Barry Song , "Huang, Ying" , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH] mm, zswap: don't touch the XArray lock if there is no entry to free Date: Sat, 19 Oct 2024 03:25:25 +0800 Message-ID: <20241018192525.95862-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3D613160014 X-Stat-Signature: jy6mbt4bpy8xigcdrp3ccxy5zesym3oa X-HE-Tag: 1729279547-323754 X-HE-Meta: U2FsdGVkX1+PXgiMT82yDflEW6Z5OLbAZnF+4nCjSdy75A7jjp6Olrjq1/FNEL9A6lDOdmqYrxOps/WWFs/cgFz78JsRzlgJTKnu2CYD0cMboMIV8Q+AW2KgU3pCLSBgFR54obFmHoQzG3OPatAEthjHCx7dcwYQM4xN4MlC7ULN3LIVQYWljYGfRhl29oOQv5js9A49TMIMFk5lAbpX2Dn0IjsOH1NGQ3XyiZ1+OWDUWo7jGQTXl/+IihK2NGNr8eyRk+RDfUksbZ6zG9/3GbHT1Otc+MtU1VtjpGZZ/sGi9idEPMZjuNe92hM6zJSaWWkzq1ngyL8up4qzc1zxH6KjaVJfSSN7Ncbi+hDrMcb5fp1JZ4oj4Vgmc6cg6XQ3cD3sPLGfrgUt+bb8Hd8FkOO+lWCRGce3abpHmsURaJECqDKj2w8CONPZ6mkQXq+GpD2aqVBSYdJKfJtiQkFvwUzh+BQbl9cH/DcNEo4DVrYjfDSGeBDGv2A60v2MmlMPdF/0zCPJhnFjl+A+FGyqBke+obCJ+ALd6ODCgnaoZ2nge1Kts0TmnFzbMq/2pfRxiqHE6crKCKVDLJPdH6Mbx4UmATyWFdzNHRRhMmNyZH7ncoexAagQRe25UVuyC3yZc8bpNTDiuVROFqfCOWF6pagventO1mCd6YlZjLYYX2GWR/orB6zE9NbAsrLzeFibfKtBagfW/5hiloVvo3E7eh8B0O8P8sK0d3kt+0JthTVcgLVe7nyA1slWAyB7bui4sqM3YqBom7TVkGZumH4WBHkkKXTO93gmNwxOm8qqQ0eL2OKkXPVz04V7TxS+CBIVqnnJnRmguqAqe9CVEgMLGhWknxTR++2UZEKNpTTFJODFUS5f8N/snKCP6m0KztcjNzsHE1p0J9gbknKNpuKfsVYabCaU6Y8ACICf2m4GJeb5Lgk3k8/Sm3cQvSk3gjfxeEEXaiRvvkqFTis50a0 +GlIORGv 7a7yZY9f8BQ0sQN51yuEj9OCpUU2IJZLojd39TBNqB6oAt48PE5ZXamJ70Q9xNScOPraL9ZtP9t1uGOcdmVLglVc0D7WtgzewXnK+b7/2U0CX6bDrIRcEP8ldyLqygff3ysBDczLnQ18RC7wNgkyJEDHick9eFeOio1mBqaswDRpq4l0L1YN64oyB6YEGb8LQNSgRnqbxFwB12ILaIclq6csg8Cq1idWcErtW6vHwjrq+FxC1EDHWDD1P5VlGpajCEDCjArzP3paPYPC5haDqyGwPj8Us7Rhgrs5QbKKU4jVuKHze+hcGK6MdY/IMSgZCohtjMJd8Yl4RbswacMxkk88T2KPH06ShdDF1bPdmPh0aOCydWoaR3APFvxaqFrHmplEVOVtMs3WXBkWQkbYAaMsmnoySQPOO6pFpiFOgJFBRimgA1AiouL/5ymajYuzx55s7A404fczOR9jQ8PEqwZsgdxfBP6JimyeDyfkjFP3MwGmyLczzYACOZWJGGnU0py52aSF4BhGA4FWEKli/o9vmXix4zKW5CYTMQIv5Npnm0eRX33oxNKiu8h77+LVliZb5OKgKXy9MDyhMu0d7aG+m2v/gM2tKjqBnI6s3jcX/BvgyHxDRy/tp3Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song zswap_invalidation now already avoids touching the XArray if the whole tree is empty, which is mostly beneficial only when zswap is disabled. This commit takes it further by optimizing the case where zswap is enabled. To reduce lock overhead, we load the XArray value locklessly first and keep the walk state. Only perform a locked erase if a entry is found, thereby minimizing unnecessary XArray lock acquisitions. Below tests are done with a 4G brd SWAP device with BLK_FEAT_SYNCHRONOUS flag dropped to simulate fast SSD device, with zswap enabled and on a 32 core system: Swapin of 4G mixed zero and 0x1 filled pages (avg of 12 test run): Before: After (-1.6%): 2315237 us 2277721 us Swapin of 2G 0x1 filled pages (avg of 24 test run): Before: After (-0.5%): 4623561 us 4600406 us Build linux kernel test with 2G memory cgroup limit (avg of 12 test run, make -j32): Before: After (-0.2%): 1334.35s 1331.63s Swapin of 2G 0x1 filled pages, but zswap disabled (avg of 24 test run): Before: After (+0.0%): 2513837 us 2514437 us zswap enabled tests are a little bit faster, zswap disabled case are identical. Suggested-by: Yosry Ahmed Signed-off-by: Kairui Song --- A previous patch [1] has been Acked and now in mm-unstable, that is a valid optimization on its own. This patch is Suggested-by Yosry during discussion. This patch is for a bit different cases (zswap disabled vs zswap enabled), so instead of a V2, I sent this as a incremental optimization and also tested it a little bit differently. Link: https://lore.kernel.org/linux-mm/20241011171950.62684-1-ryncsn@gmail.com/ [1] mm/zswap.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f6316b66fb23..a5ba80ac8861 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1641,12 +1641,21 @@ void zswap_invalidate(swp_entry_t swp) struct xarray *tree = swap_zswap_tree(swp); struct zswap_entry *entry; + XA_STATE(xas, tree, offset); + if (xa_empty(tree)) return; - entry = xa_erase(tree, offset); - if (entry) + rcu_read_lock(); + entry = xas_load(&xas); + if (entry) { + xas_lock(&xas); + WARN_ON_ONCE(xas_reload(&xas) != entry); + xas_store(&xas, NULL); + xas_unlock(&xas); zswap_entry_free(entry); + } + rcu_read_unlock(); } int zswap_swapon(int type, unsigned long nr_pages)