From patchwork Tue Apr 18 07:46:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13215155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 001E9C77B71 for ; Tue, 18 Apr 2023 07:46:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C4556B0071; Tue, 18 Apr 2023 03:46:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 774C78E0002; Tue, 18 Apr 2023 03:46:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6641F8E0001; Tue, 18 Apr 2023 03:46:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 548B46B0071 for ; Tue, 18 Apr 2023 03:46:48 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 236481A0122 for ; Tue, 18 Apr 2023 07:46:48 +0000 (UTC) X-FDA: 80693730096.01.4D5F5E3 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf30.hostedemail.com (Postfix) with ESMTP id 533D78000D for ; Tue, 18 Apr 2023 07:46:46 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=nAKwtwif; spf=pass (imf30.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681804006; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=h22ktkGMKWxVcxcaUxYu0h0WFMmYisG3d/tdFq9yk7c=; b=rEt+INvm7U87K1BUTaup7t738PBK0s2kGHM3pGS6qB25DoOxMb/bWaTqnRDNJeMmKq8ytE 1i+uiMRqFFOWiOqTGASe9fRkbOtg7/PIy0GSLxGEOy6xeUhE3v1Od9tSkI1xiBTlIkA3R/ EV3JTZxH+XdsHRjqalyOWZ9BFzQOqF4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=nAKwtwif; spf=pass (imf30.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681804006; a=rsa-sha256; cv=none; b=FRyMh49WzxCPIuBfYfkGH2wor1W4LPr+7bLSMlZ5Z7BXm4+ZigLTj+GQ8dEC4Sn02D4k0v XHwlwcy0YdO0XH5hPdZBJwS+QyG4HpPGcwLGI9a8jkMgjRSsasfHw9CyD9V39ZZMwxIPwM tnbeSTv6WAhZcSB0n7lI8Ks2dAfzMXs= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1a3df25d44eso35096955ad.2 for ; Tue, 18 Apr 2023 00:46:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1681804005; x=1684396005; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=h22ktkGMKWxVcxcaUxYu0h0WFMmYisG3d/tdFq9yk7c=; b=nAKwtwifeDZqZJY91VY4uctqfDAWvCJ8kPEnIdBPWP7nOCP+19w2A8evcSTy4aTXiK yj/dq6QyuaZW3oa0iB99dQ/GA/I0wivzBZA7HK4zAMwgfQe6ThDPlUrdvS+cWRz6iUjg rqUhYmna5js5NTsmoWbHHlyW2mNrCOezZspQY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681804005; x=1684396005; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=h22ktkGMKWxVcxcaUxYu0h0WFMmYisG3d/tdFq9yk7c=; b=JFoCZAUjzlmVYeKrNy1aQuDRPgg82efD05PB9q5iqdIx7Ib/NwtxSvWscjZd49MR4M wvlYCcC71CImJKfFDH2Kofn4+gsKRXz99SR8OSStSme9U2iApSMIb2z53/91CxhBDmDN 2Vqh1DDf5owWQWJoA7pGmBG0QRd6b86ys6nbYtsOBAOisEYBH88AMZsTbc6njeqVH3Ay O+oR8ZAysz9aUPvzlzi3JrtTL/6eU+ARcXUMBqGF9BMMIx/HRyCGWQmku+VbVu2bZgQD k9iSD6d3C7KBs8UJvzMCgAKPkPx5Hco4o62IYPn0GNbh4fDiyUuDRq1iLGPxj1/MV0ji 8ZkA== X-Gm-Message-State: AAQBX9d2hbSRJd/fM9Y1KCKflFbCz4nLvN/NIw9W5Ch7TA31fq2ldZnV aK7IKHXcOtNayckwYVvCWqjjtA== X-Google-Smtp-Source: AKy350YFjlSoPtsG95YiTUTNXmobFnXqxgrHOGzKKvaX5shNTFdX8nzsZ5QynhdG42owAFdYVfNIsw== X-Received: by 2002:a17:903:1c3:b0:1a2:a8d0:838e with SMTP id e3-20020a17090301c300b001a2a8d0838emr1507684plh.61.1681804004787; Tue, 18 Apr 2023 00:46:44 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:7e0d:abfb:57cb:cdfa]) by smtp.gmail.com with ESMTPSA id v21-20020a1709028d9500b001a527761c31sm8956975plo.79.2023.04.18.00.46.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 00:46:44 -0700 (PDT) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCHv3] zsmalloc: allow only one active pool compaction context Date: Tue, 18 Apr 2023 16:46:39 +0900 Message-ID: <20230418074639.1903197-1-senozhatsky@chromium.org> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog MIME-Version: 1.0 X-Stat-Signature: i3kyhkxr1mgazyry5ks3g1rmp5crx4uw X-Rspam-User: X-Rspamd-Queue-Id: 533D78000D X-Rspamd-Server: rspam06 X-HE-Tag: 1681804006-937270 X-HE-Meta: U2FsdGVkX188rSu7SVek6Dltj1W+IrQy4uzzVdhMft8PAUWyAATeyHRGqKbvHyNFzNzJjJILIEu+XAGoxtlgyWiFFiQ4t3tVvL+uu3nPMtCCYEs0QTAi700osxYBG0MDQkqnK4TaDjID+FHwChl04G6vrzF5FVVs4CkXumNj++NLrdAfdmbMqPqXPJKlm55NHp9w4ODJKzvn7pHZ6xzLWW5LRMCQHyemf2sumR/AKC2XWc6BaZ/CXcqv/AppjlO1X7XeMMymruz7iYCI2JXzoyIBejUZMMZLsVnKj4/PUQyJ33mjaEcBN1NitkV2l0x6U834+ytSkeZrf/dqjcdVGUQh1AOyLoG3yaN5R+bQ4xNmI30+fftIw4dbhxHpUBGVlPLo1s7b9pi/rwatCkbRD0Uxs7akR/TYsU9LMNSXDF/1KabRG50v73lOhxMyJLHmo3TbvtVI+C2pxogoaBHrMmVxgjV0D9FzB4Dkz1F7o0sU2RhBanmrVREkGkdN3ae2Q1iAY3N2j1YJi/82ZNJKa1axvqFyKrPf3AiUlQB+rBQvQ55khkChPdUf3O3UYrteoNI43GujKtEfxoRGZC7lVcKZuyQ8L9wIbttGNZqWcXSIdnuEOpaVThI665eaMo5Yqgn1Aaq3RGISTJN/yzKV9D1XT4FYChqyG3/o/VmjHPwxhhSvZUCWsuW5Uw6Wuj/IPc3xnZcXfDVg5UTfa/2tw0Nq6hv600FcDaVF/fP/kL2b8azWr8oiGe/Tbs7CrEZiQ3UGYe/CGqEri+p3Hwx28vJXhO80+7KC3jHUKWsrg4fD8vXGYegwoeQBmqdF4tWrY1TKjsvHwNb/Tf/2ucM5OXl64YCWRKDgDT3JPAbRfeojnP8BUDr130XorWVS/yTu1A5vnsJRomtB/0TNud80yNg2QQxyD0p7OV5prApuI+X4WkExjwgXLXKFUXh/9szrV2zfM0bo9F9f7QqUbt9 q2fIiLw9 2dMWpEtMCYMemo+cfn0Ug3EDssFx5pLUN6JeF4YaF800/QhUxESFAhkmndcHjwGASeRY2PDaVqloJzmVQ40WH78pT/3dfFgGmqgFLwWsl9B3ecW9oNCmpx7wLmqES1u3a7GhMJ4ShtrsftOFHW0UDiKs7bK9gZ6HSvyMst9Xxxk/3ji7G6R6u1t3artb3eLkNY9hqQvcWi29rBnauvRPiUHNNzICFSPV985bvezouc7jvMGB0mfZ3btIrXV9ihRTRgvPVcsbpD8oElB/v+7bjlxxXvy9Zl+OopdwZURojJbJ1H1A9faUEG0zAFPqnkpp3RF+ht3+gTYl3caNfaxGSSgC7rC8P74xnTroPmGw9TFM/QrsMwlBKzngYk8uBFkmbYDrde55BBivg0JsIq3gUnOoCfPXTkivJKpxPkK5LC9VtYqou3+6qo8KEnqWfobQ6uyAKRbZqn5WpzR7vOiQYz3PQ/g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: zsmalloc pool can be compacted concurrently by many contexts, e.g. cc1 handle_mm_fault() do_anonymous_page() __alloc_pages_slowpath() try_to_free_pages() do_try_to_free_pages( lru_gen_shrink_node() shrink_slab() do_shrink_slab() zs_shrinker_scan() zs_compact() Pool compaction is currently (basically) single-threaded as it is performed under pool->lock. Having multiple compaction threads results in unnecessary contention, as each thread competes for pool->lock. This, in turn, affects all zsmalloc operations such as zs_malloc(), zs_map_object(), zs_free(), etc. Introduce the pool->compaction_in_progress atomic variable, which ensures that only one compaction context can run at a time. This reduces overall pool->lock contention in (corner) cases when many contexts attempt to shrink zspool simultaneously. Fixes: c0547d0b6a4b ("zsmalloc: consolidate zs_pool's migrate_lock and size_class's locks") Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index cc81dfba05a0..44ddaf5d601e 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -264,6 +264,7 @@ struct zs_pool { struct work_struct free_work; #endif spinlock_t lock; + atomic_t compaction_in_progress; }; struct zspage { @@ -2274,6 +2275,15 @@ unsigned long zs_compact(struct zs_pool *pool) struct size_class *class; unsigned long pages_freed = 0; + /* + * Pool compaction is performed under pool->lock so it is basically + * single-threaded. Having more than one thread in __zs_compact() + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. + */ + if (atomic_xchg(&pool->compaction_in_progress, 1)) + return 0; + for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) { class = pool->size_class[i]; if (class->index != i) @@ -2281,6 +2291,7 @@ unsigned long zs_compact(struct zs_pool *pool) pages_freed += __zs_compact(pool, class); } atomic_long_add(pages_freed, &pool->stats.pages_compacted); + atomic_set(&pool->compaction_in_progress, 0); return pages_freed; } @@ -2388,6 +2399,7 @@ struct zs_pool *zs_create_pool(const char *name) init_deferred_free(pool); spin_lock_init(&pool->lock); + atomic_set(&pool->compaction_in_progress, 0); pool->name = kstrdup(name, GFP_KERNEL); if (!pool->name)