From patchwork Mon Apr 17 11:02:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13213699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3EA0C77B70 for ; Mon, 17 Apr 2023 11:03:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 242AC8E0005; Mon, 17 Apr 2023 07:03:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F2388E0001; Mon, 17 Apr 2023 07:03:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06F2A8E0005; Mon, 17 Apr 2023 07:03:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E614E8E0001 for ; Mon, 17 Apr 2023 07:03:58 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B8F53C0525 for ; Mon, 17 Apr 2023 11:03:58 +0000 (UTC) X-FDA: 80690598156.16.2EA7738 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf03.hostedemail.com (Postfix) with ESMTP id CCCC02000E for ; Mon, 17 Apr 2023 11:03:56 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=MzuSVH7Z; spf=pass (imf03.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681729436; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=AXWxF7DmGQlS+I9+P3uaxt1m3XBosotbfe+681ULx84=; b=clRWetI2L5JYXaX6qL0qae9km0EP7bbO9zKVNdGHewiMIP4xSwmmybBFIomo6JObxpAi/p YARDF0c/ZNbWKm6Kh6YIY/wW+xHduL5UxnaBXnRVOmb6xBe2b5q9tSW9kL/Rx9DsHVaIxj vktluE/M7ZMVdjxM+/2PG3VIi6kYAgY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=MzuSVH7Z; spf=pass (imf03.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681729436; a=rsa-sha256; cv=none; b=VkBReDQH6meljZmQGpT4V3LhTI1WdeKAxZnyr/rmyeI5tyWUz/Gfdumy2nSXJ3c5C6OAKC +iTk3WI8heVXBNNBlipSsY1OVCTk2xTVbyGhvaqB2Dl519yiia7eVdBSoax0AeMcaNuCd1 nFZNCcg2qH7Gmtg50eCx7IxXgmWFc3o= Received: by mail-pl1-f171.google.com with SMTP id lh8so12229525plb.1 for ; Mon, 17 Apr 2023 04:03:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1681729435; x=1684321435; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=AXWxF7DmGQlS+I9+P3uaxt1m3XBosotbfe+681ULx84=; b=MzuSVH7Zo2sPbGWOQ47oeP4xLii2+aOg/66pgyJLKCz+X/rFQCZEZAlJXuDiRPX63L Cux+FDAbL2z7jkjdIeQ640L8J3LRvmFN5d/WR9Ny+AaK6EMbcWe5MWSCRdLkCXf100Ae 4S6Af4C5MkMixqMRH1kFCqPRf92TDYYP6Bans= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681729435; x=1684321435; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AXWxF7DmGQlS+I9+P3uaxt1m3XBosotbfe+681ULx84=; b=LRgHMxRVlYbwzrPXodGgdbEFFfmtFs1qieddlU0UHleneaj0wCjd/a50AhahvA8zVY XXHtYHXc0MjVRsa3YhSeLwPwjGL7W2B9eIZEUa9q+MUhgQoisdifKY/2J3Uwxgy64xvy jwIjIUzYM56wcG/LCe281XIR5BjyQ+OzWKA5IRQCBlTZWMXbU+vOiVGPM9ECXunnqmu2 vb7KpFLsS0FSQVp+45piaQvtI7k+P4n9TJJhA/0plN+HN9AoC+icQ3+TK3nDtBIUitve sG5KP0EBx+Q6rjYg4GHCu5B2+MxeZzASXjgi5x3A2SX8pmipys9OIx4GCEe0RgrqQBUr vvkg== X-Gm-Message-State: AAQBX9cosTAElOTkyj5UK10pbF0JIbserQauerTDY6vOjhjU1GjPtUrI k1BsWAXg0+HRSL+yKWbx820iWA== X-Google-Smtp-Source: AKy350aXLCVk+OYIIUH+lbE2hcALZKyJLxunUGHqU3dWfL0H59xa5XS2CsoUO3I3xsT99xRfHjqmzw== X-Received: by 2002:a17:902:ac88:b0:1a6:5493:a3e8 with SMTP id h8-20020a170902ac8800b001a65493a3e8mr11813926plr.41.1681729435548; Mon, 17 Apr 2023 04:03:55 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:97ac:e7c:d412:5251]) by smtp.gmail.com with ESMTPSA id c6-20020a170902aa4600b001a221d14179sm7258666plr.302.2023.04.17.04.03.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Apr 2023 04:03:55 -0700 (PDT) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCH] zsmalloc: allow only one active pool compaction context Date: Mon, 17 Apr 2023 20:02:59 +0900 Message-ID: <20230417110259.1737315-1-senozhatsky@chromium.org> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CCCC02000E X-Stat-Signature: nw1ogaiqwoanmj7rxs1i7utje7af9iou X-Rspam-User: X-HE-Tag: 1681729436-106220 X-HE-Meta: U2FsdGVkX1/JZoL2hx3g8Up0wCCq2wLqRF8nYr16DCvCGfibJzpUesbZPygT4AvQ/haQTCTUvSGXmAbdBDVb5RwkZS1o1PivdzaNBW3OQvxkgalpqKiOES0Cf89khdQLAn5xt/xmr0ayeG+UKMIVisUyts+GeopYAf0yG8IYoWiBobpGI4FbiNJppLtemx9M8YA5kPHM9nEf+MC/D7jt7mkEdJYFiESvT5pBsGbaAy53iD1iW7I6CedLv33sKYDCX5gPoJdebHS0WPSVnaAhnORAIp6ce1PBDprbXqPGVkiWONMZIak7Q33xKz6LFLtpgEMggz+qX3AZNfONRrY90VEJaNmhkc3PKldvXJKA+nb/KUyKa7UEkf4iHeYZmwJSgi44FQGNnDg9QN1jUN8EBX9jm5+9d+WjaiXEnWTHW28GrY7eiJgSRvKSIAPnAsrdGc6PAf9zhIVPOex6Lu3eRr1UbcNw0wO+K9aQgJUT35i9gJyu76VW9tQqPkA8poEHUTBwWlaiv28yTazqo7KYn9kYWCL/NFEpp7Mzwdx1oDVPJbuAVL0nUWEArfWiQWSQqi0fQUCtNJiPryRxQ4GtT5vk6lazzsHxVLH++fegvz8Lde8OPrm+JgzlCXYcshRx1RFWBMAgwNacyxpgNGnmTucT1MaQHKuHzFqf2t3Q1b7LCjQenJTt/MhsLnx2VjFDvTDY9N3fz7dBWcusnkKcOuejOTtSTemdoK6hslv6PQGBKaJyHADqRHY+qC4gGWSyUwv07VF4/RFOdg8v6wv0xOqGpd815aHDehmR0Cxee3V6jr97jXmXxhrByD/T9mS5yiyPVre6SGd6tyln/q9U07EBmfPe/uaXoc9y5eslUjszJp/zwGjcMTC1eyrSJcxwU/ZMCTBvCbkIC8UD66USrNXPM997m36yRStzNY+X4xewRJul7xqD3f9ONe4ChpQImieUkUtyMSVmuSmX5mH ilMHh8b9 8DZgjiYLI7S3DJTfXRwrWy4TBQRSE+OLQUMbiRsjtlNrMqZ6o9DFRY3jSyQDoMOIazqHsppO26AMGCgDyMTLoZj7i2+Pc99+9wGW60jPa8D7np5NRU+fno7+94YXhbrFGo4F/ubgQXyMRltMrOsSb7D6pdDX2bbaEVFxP9bKwg69ghpX/8aOQBFR9sQp+3n9xWYiK+6TVEAb41db71G7PW8Uq7D2W9zCInEX/2c2UAMsH+0dbUDX1szrM7jT9YmrW2VPz0N0mV2BMP1HFcNjoLAbrP4bTSrPajXFVx7xA0rHDkJUOoue86ynsyGCdfLxqNIQ1Rn3MYtfQHGS8yC062pCNlYgtKWzYp+/4ZZPaNiK+xUzKGbB3AaCWbq4rXybrGa75huhySh5licxdBrYw0N+jxmyB4kDNxIoN61slddoL+3A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: zsmalloc pool can be compacted concurrently by many contexts, e.g. cc1 handle_mm_fault() do_anonymous_page() __alloc_pages_slowpath() try_to_free_pages() do_try_to_free_pages( lru_gen_shrink_node() shrink_slab() do_shrink_slab() zs_shrinker_scan() zs_compact() This creates unnecessary contention as all those processes compete for access to the same classes. A single compaction process is enough. Moreover contention that is created by multiple compaction processes impact other zsmalloc functions, e.g. zs_malloc(), since zsmalloc uses "global" pool->lock to synchronize access to pool. Introduce pool compaction mutex and permit only one compaction context at a time. /proc/lock-stat after make -j$((`nproc`+1)) linux kernel for &pool->lock#3: Base Patched -------------------------------------- con-bounces 9797655 8125860 contentions 11131185 9242153 waittime-min 0.09 0.10 waittime-max 4171695.76 3926258.74 waittime-total 506197629.16 417061026.20 waittime-avg 45.48 45.13 acq-bounces 13809103 11383480 acquisitions 21145155 18049364 holdtime-min 0.06 0.07 holdtime-max 7379928.80 3926274.89 holdtime-total 46273950.89 37279624.53 holdtime-avg 2.19 2.07 Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index cc81dfba05a0..0e036ec56c3c 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -264,6 +264,7 @@ struct zs_pool { struct work_struct free_work; #endif spinlock_t lock; + struct mutex compact_lock; }; struct zspage { @@ -2274,6 +2275,9 @@ unsigned long zs_compact(struct zs_pool *pool) struct size_class *class; unsigned long pages_freed = 0; + if (!mutex_trylock(&pool->compact_lock)) + return 0; + for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) { class = pool->size_class[i]; if (class->index != i) @@ -2281,6 +2285,7 @@ unsigned long zs_compact(struct zs_pool *pool) pages_freed += __zs_compact(pool, class); } atomic_long_add(pages_freed, &pool->stats.pages_compacted); + mutex_unlock(&pool->compact_lock); return pages_freed; } @@ -2388,6 +2393,7 @@ struct zs_pool *zs_create_pool(const char *name) init_deferred_free(pool); spin_lock_init(&pool->lock); + mutex_init(&pool->compact_lock); pool->name = kstrdup(name, GFP_KERNEL); if (!pool->name)