From patchwork Mon Apr 17 13:54:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13214054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9462C77B70 for ; Mon, 17 Apr 2023 13:54:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49F866B0071; Mon, 17 Apr 2023 09:54:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 427D28E0002; Mon, 17 Apr 2023 09:54:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C8668E0001; Mon, 17 Apr 2023 09:54:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1723A6B0071 for ; Mon, 17 Apr 2023 09:54:31 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B27C81A04D1 for ; Mon, 17 Apr 2023 13:54:30 +0000 (UTC) X-FDA: 80691027900.29.0D68CB0 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf27.hostedemail.com (Postfix) with ESMTP id AC5AE4001A for ; Mon, 17 Apr 2023 13:54:27 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="lMpY0Qu/"; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681739667; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=pMpXhhgqoqclBiBIVXBf7YGaBTn2oRSRzJmNR6lNyRQ=; b=gpvj2qV3GOTZuWrZTd4pLGYSBLg3560e4OLpiRSqGpSCSQciK0+Mvs8hug+R1qnJWp7WYZ YOdLWmUrJtbf6j734+0oeoUJgKtnLYrsHp7nvLAHL6yrgrzM9D+LCk+9NnOpkfPpA+cndc GHGb0y/Rml1vJ97PL03cjSOP7SepHB4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="lMpY0Qu/"; spf=pass (imf27.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681739667; a=rsa-sha256; cv=none; b=iN7O4W4eVgfoW/3Upr16ezqgHE5wbLIhhgu2tgEiYUsAujajQ1lUExOTxsOQxUJYC8bwn9 QQ/l0tal2lNoI7mctSetC6Ail558rv+uF0AywIc50plH8pnHxhxeL1Mhe2S+30R4QrV6ni u3enjEnozOECCbIID82oqayZ6zHQpyA= Received: by mail-pl1-f182.google.com with SMTP id w11so25714165plp.13 for ; Mon, 17 Apr 2023 06:54:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1681739666; x=1684331666; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=pMpXhhgqoqclBiBIVXBf7YGaBTn2oRSRzJmNR6lNyRQ=; b=lMpY0Qu/a56IggAJIX6VY72VSKPhHpTVPaqC9ETNpc70bd9b77rx1n3k1yco4cOgb+ 0JkFI7L/B0txbOekHNYbOhMifBI+edyW57Ko/YhwXGvQCjfvIsZYH/qe67K85JWlLjLx iYy9oAYuUWlvta1fIOsBrHZmXUBQ6nFlBlxYI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681739666; x=1684331666; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pMpXhhgqoqclBiBIVXBf7YGaBTn2oRSRzJmNR6lNyRQ=; b=PZMziu89mPyx6MCt59ga2CLhOJT/+AT6Cx7OANQhg1Xtc9JJkss5z7dEHGky4OEvnb uECt0NlNfq6tXa0aci2N6ux/Qxkeqpt5Jo+GwsSW0hbFaA8wzp3iChSFWUxiFMKAwZCP SGTJKgKj761uovaEkZkH60Bdpe+xK4BIOw2G2egVIK9gb5/TzNN0A3a7HM0HpWZVnxZo qgwmWu92xA5058q7o6VPoxmxkn8eewicVPiNSgQvUOVO7zEWaMdYkEcRhPwdIC4HA5nE ufAd5VKcebpNMIO1+NM9feuE7xmu/51BBpGhWJFyZknLIyHkl1bBrlBe+U91yKLemUll BiUQ== X-Gm-Message-State: AAQBX9fTb1pOLbbN6DDUK/wr2/im2KE5a+QpMN/brqudbxIkPFEhM6XC aasCVdPR5l9sdSsA/WWdyoiFCQ== X-Google-Smtp-Source: AKy350aiXSKRpV73fKU81i74GwN8gg8Cy6+aTaH1inRxpfH4c1p76DmZJSftIura3Gqg5bquqZXkCg== X-Received: by 2002:a17:902:e5ce:b0:1a6:f0d5:3e21 with SMTP id u14-20020a170902e5ce00b001a6f0d53e21mr1376568plf.32.1681739666455; Mon, 17 Apr 2023 06:54:26 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:97ac:e7c:d412:5251]) by smtp.gmail.com with ESMTPSA id 1-20020a170902c24100b001a0667822c8sm7769619plg.94.2023.04.17.06.54.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Apr 2023 06:54:26 -0700 (PDT) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv2] zsmalloc: allow only one active pool compaction context Date: Mon, 17 Apr 2023 22:54:20 +0900 Message-ID: <20230417135420.1836741-1-senozhatsky@chromium.org> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog MIME-Version: 1.0 X-Stat-Signature: opum4cibnus4kyqyqtiziq8seotuk8ka X-Rspam-User: X-Rspamd-Queue-Id: AC5AE4001A X-Rspamd-Server: rspam06 X-HE-Tag: 1681739667-244093 X-HE-Meta: U2FsdGVkX1+PWTGv4DDQyy/9XiOdc4UAQuBuHYRQGOQVADpysYolKJUvUv28VEW9c1mkllzq0LOIRsHKe8Czk2S4JlA6MWo6Kyvb5WRzCl8V6szJZtfRPyxKdnqXpoHV7VooY8U0BsJ0PxLWA9n+3c/f5rqKLIAAYPJeo0fbAetIqanynmFsI3FPr3ABTHUOh6lc8fRI3vdC+tKMQp0vT06C+ocS1XQ/onDmSi9JHYXOGHCKkNDl9k/cVHV6fllE0VsfRye+tuHG+msIO77/piCqcZhTAhASSPNjh+vDvqHAvvhwosbdDJAYk0cfOh0kpD3LcRu86zrwkE9SdmpXDNfl7zsPPaQAfTERQcYynUNxUPF25b5bHKDTB5k8J69Me3pQVjkFjnOeM28POZTjGNEy+NJRCIzPuicMUsaLzWMWO6w/IP1Wtg7b79SRhhFvh2YWGffhQ3N1mG7o9qVHQj4KXbHKBeWV3iI4U4XXQqkWOKHlleAYcw1ia/p3PU0xJazrnhWxEpFQCOXQGyFTjdxcqHdCweCECXuGzfHgKsKmk0RP8Sv+o/0nScUxMCOvpkOaTajV+/KCClqGsSDujhzolKaqfEKa228yv7ImS2B5vJ0hP2UTIcR4vcuw12sCdaRv+9V4l2Vme80hlUS3ReT0vx0VW8tXjwzH5aGf2z4QYi/Sil1h6864KNTYn6TzyD2flWCSR5aHOicb1ZAA6UWp7H7v+3g15lBCW/X3qmv7Uv+D5oK3t47KmGTralT1EQn4tao2Kf9tk7o7wAWmjjncRdYrB+monuj4wvdUwtosUnetxPYJrPbSqmOnsY4IFk/e9Nfpj2/ZczBIq0CRAhXOQ6aYAaS9Sgkna4a3XQph/664Wn8ag+hsPuDo31/KfD6nvNuPlREd4uAugdP63cSmka9Q4kvGBRYZ+H+pMwtugx2nGkQPMryXBBu76oaCQzBo97C0N7SwqCUWBO/ ifyF6LCZ U/ci7k2TD50fxPievtP42IhjnrQfgldNTf8Tvor5pE3TVckTSu8k3QWw7QTVNQkrWNnNYAR/Bi3zrtEXLEa17/JIMfa1pudIlC0amH4E2cH3ATV/5yLmJNzTA0O7CHd7mMQdR9i0ZyviRY2fEbXompXy6e2BXKQ7e1Dui2pzyWBXaqBmcxgQN3RCmIZjhmHEuwGIYBhlvH2WWeL/5SXXjlU7XYD6qnd+1NYbKhj8m/KVtVjsC3Wr327Z+SU0c1SJarBmVn9I9fnwSqBEo3tFEfhel+RLT/cKDZe7KuBkpjrr4WqBEcBxY+NyLpgg+0Exsc2E4UVCy7B4rSeT6VaQo82m/yKZRUh1hXC2VQwoV8q6bscbviFgSNeRKgwSrZVUBv2apDwa9rq2pgps2SSQbcI4HQdpgPUOfD1HVJVBh6ot+kyM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: zsmalloc pool can be compacted concurrently by many contexts, e.g. cc1 handle_mm_fault() do_anonymous_page() __alloc_pages_slowpath() try_to_free_pages() do_try_to_free_pages( lru_gen_shrink_node() shrink_slab() do_shrink_slab() zs_shrinker_scan() zs_compact() This creates unnecessary contention as all those processes compete for access to the same classes. A single compaction process is enough. Moreover contention that is created by multiple compaction processes impact other zsmalloc functions, e.g. zs_malloc(), since zsmalloc uses "global" pool->lock to synchronize access to pool. Introduce pool compaction mutex and permit only one compaction context at a time. This reduces overall pool->lock contention. /proc/lock-stat after make -j$((`nproc`+1)) linux kernel for &pool->lock#3: Base Patched ------------------------------------------ con-bounces 2035730 1540066 contentions 2343871 1774348 waittime-min 0.10 0.10 waittime-max 4004216.24 2745.22 waittime-total 101334168.29 67865414.91 waittime-avg 43.23 38.25 acq-bounces 2895765 2186745 acquisitions 6247686 5136943 holdtime-min 0.07 0.07 holdtime-max 2605507.97 482439.16 holdtime-total 9998599.59 5107151.01 holdtime-avg 1.60 0.99 Test run time: Base 2775.15user 1709.13system 2:13.82elapsed 3350%CPU Patched 2608.25user 1439.03system 2:03.63elapsed 3273%CPU Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index cc81dfba05a0..dfec2fc6a30f 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -264,6 +264,7 @@ struct zs_pool { struct work_struct free_work; #endif spinlock_t lock; + atomic_t compaction_in_progress; }; struct zspage { @@ -2274,6 +2275,9 @@ unsigned long zs_compact(struct zs_pool *pool) struct size_class *class; unsigned long pages_freed = 0; + if (atomic_xchg(&pool->compaction_in_progress, 1)) + return 0; + for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) { class = pool->size_class[i]; if (class->index != i) @@ -2281,6 +2285,7 @@ unsigned long zs_compact(struct zs_pool *pool) pages_freed += __zs_compact(pool, class); } atomic_long_add(pages_freed, &pool->stats.pages_compacted); + atomic_set(&pool->compaction_in_progress, 0); return pages_freed; } @@ -2388,6 +2393,7 @@ struct zs_pool *zs_create_pool(const char *name) init_deferred_free(pool); spin_lock_init(&pool->lock); + atomic_set(&pool->compaction_in_progress, 0); pool->name = kstrdup(name, GFP_KERNEL); if (!pool->name)