From patchwork Tue Dec 10 16:40:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13901750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66B5FE7717F for ; Tue, 10 Dec 2024 16:40:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B5E46B0260; Tue, 10 Dec 2024 11:40:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 447806B0266; Tue, 10 Dec 2024 11:40:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21D1F6B0269; Tue, 10 Dec 2024 11:40:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id ECB7E6B0260 for ; Tue, 10 Dec 2024 11:40:45 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9BF07C0946 for ; Tue, 10 Dec 2024 16:40:45 +0000 (UTC) X-FDA: 82879612998.02.B70CD31 Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by imf22.hostedemail.com (Postfix) with ESMTP id D47B0C0013 for ; Tue, 10 Dec 2024 16:40:19 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=a9yoJfC1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.46 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733848822; a=rsa-sha256; cv=none; b=Lvts40e2dDf2EgtH9EM0+0xNlpmDafGPs7LPuTUz3C/VBUsIpF2F37WelzMNuvAJ7M+u4v loLU2fu/ZvhrNze3ACFMiw7KmBreT3aOSRn62zyU7dTaT9mT6fDUr9z9ZUaCjycBzesLZq UWXnGUbvvhBj87HZx1Lg5Qq5o77c2VY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=a9yoJfC1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.46 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733848822; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8psq5tPidMqLngz98gskDzgVmEZF8iPxbnPcrOFnt1Y=; b=FLRGAyd4zTLfgV6dqhAokhMemx551fH4ltSIsRymr4/N6STwA318MrQITwKthmvwsnX6b+ cf+ONHHyeVj3kqQvkjDnKIkEYg+2jLjsTjTJzmRF3sg+zZBKNXZbVUxIRDpsQRLbEhhnOg RcvBNCzGQ0IK9Z17iOf/DkbxEdC7nfQ= Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-5401b7f7141so2277483e87.1 for ; Tue, 10 Dec 2024 08:40:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733848842; x=1734453642; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8psq5tPidMqLngz98gskDzgVmEZF8iPxbnPcrOFnt1Y=; b=a9yoJfC1gwkFbxKQdwz/D11ygHrICO2vaeibZdi3DGokKHZkwXuaBzZDV3csyjTnmD 6RZVEGzH0BUVSBsa2m2/E8bjEHW1xEbWygWakIyAbYeNGwi3wtHeUFWPO8rfk5nTlUzO M0w184RTkIp7lwpXab4Uw36AGlSZuU5w6ba7/UU8mdmdU4l4w/KKuaOTpyGBveNOyWhO j/W8yltZeQ3mkJNi4mf6s0iJfU+MRN+ieLTsRSQDmvBjB4lJ9w7PMqzHKs1O4q/jApHm mECVNbTyegjElJrDUSvKknkYaI3brxCK0R0WXI7QaUe6A2Qgggxv+nunYRj3RoZJ0cXN QJGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733848842; x=1734453642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8psq5tPidMqLngz98gskDzgVmEZF8iPxbnPcrOFnt1Y=; b=RkClmAgC6h86PMFhABHxZuFC6g6MmMS5qzXsxpSKD0Ih5wLOD32HNPn30l91Zhyqba 9+uO0gl3vNLPXVSdqMXJXP1D/aV28Jnm2ebvdF+J4PieZxO9LQOBld97MwDBtur9gdV0 DzSDL1JTxUJyBYXKQygYwr7Ht/cfGUbXWnMEaPVFefv49jnPooBJPFAPUOjgvbJDcdgi jf5WrzWgoLzp3UG9753000ll9+CFi/mBLHq0ZuA1nQ5ZxHc0WC9R4OE6eNByMPf7yVhu j+2aWK+//Ph8ofvnmtX31adxnpSrzCQLBJT+twzMxt4qtedbQVEwPR0yrFnSsQdUAL4N C/+Q== X-Gm-Message-State: AOJu0Yx1v6evxXyCfJgoZ0t83zyGkGadAwxY98WMPvQJw/r0BrY7PpTn cpPZY4g8d1d+lAI+FQdJqJxUqbCyaPC/DkOrL7J/1DP30kBnhwSYGQ9PwA== X-Gm-Gg: ASbGncs4X8hfLDAmzGYcRY+yEbO5cXGriInqBmVhnNUq6OZ8njTEZgo9jaX78UvwbCS dt5DjoThn2LEq9nn4enpbtPamb2OB9Mn4jfasMshaTVrbZ/chwfH8mOYoiL+xnhsUrrCJ5QG3Gd n3601TeC29X/qslcIXeiUTM3cMbh2vw9EtRULP0U2PcyQ5YCM60nWthAsFUaoXOyILcCw9AsQtq mUDuYfqEBmBFYBEO+BQwR6XHfG6ORld69A/gPWI24Ukyb1zoQ== X-Google-Smtp-Source: AGHT+IH7sMXWy8vZyr5EGz+tyNTQnk0FxUAmt/2tf7nxEsgjM8bPz/qNg9Wj1lm+mcn0TAZSNwIVkg== X-Received: by 2002:a05:6512:2245:b0:53e:389d:8cdd with SMTP id 2adb3069b0e04-53e389d8dfdmr5755357e87.34.1733848841502; Tue, 10 Dec 2024 08:40:41 -0800 (PST) Received: from pc638.lan ([2001:9b1:d5a0:a500:2d8:61ff:fec9:d743]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-53f93377eefsm1031875e87.67.2024.12.10.08.40.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 08:40:40 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton , Vlastimil Babka Cc: RCU , LKML , Uladzislau Rezki , Oleksiy Avramchenko Subject: [RFC v1 4/5] mm/slab: Copy a function of kvfree_rcu() initialization Date: Tue, 10 Dec 2024 17:40:34 +0100 Message-Id: <20241210164035.3391747-5-urezki@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241210164035.3391747-1-urezki@gmail.com> References: <20241210164035.3391747-1-urezki@gmail.com> MIME-Version: 1.0 X-Stat-Signature: x8a37m154qqndhss4buc4di4m1tg7976 X-Rspam-User: X-Rspamd-Queue-Id: D47B0C0013 X-Rspamd-Server: rspam08 X-HE-Tag: 1733848819-640760 X-HE-Meta: U2FsdGVkX1+6cEPiD59ZbSv7MiMDOzYuWaolHoCzipC4chKxMqAjdTzAM4YbMfStFipwNarl/CGicmuCF2np/qWLDe1Zjxfu0nKSB6MLB/t5riDeHnB6koFE99GRImjW4dp9fK3oLdrrmJTO+m7e3NMDYLHF7cLZYov45OxrkdEWbpSG9qZmq07FzM+xWcztLFWoAJEVWKHukeXI2KcF3q5UDPuY05ckeeQ0tqaZ0r/qxAOITCFEugVYkJyDciqlvaHLTUx6+OVUO4RcLy1CRiQkfSXyDruLR/YcBBUYSj6wLgICMRH25d6noOrpjKT7Fq1cW3H3bbnYpuHF7rdxxK34znvDVk7+J/rGJCtfISroHa3OraY25tHbX8795AZWuCliiBzqJ9P62JZdoNgw82ptD2lDK8nmLIPsIVI9qZGqRD2cEi16vtvDxYqvG4fa9IuFKepNrXEGpEP50rS/UcTowSUtCRHKt2AHfNsW9pbZgMQByvUZzGrQTtvTLKo/VIjxlEVf6XTCMboNomqvD695WvfwHXn7mPuNhFXGcdD7Z40cTeQoheH8TwVsJ+t1a2fQE0OvZ6/OVNTibCxhuTeAL66JxUugg4vfTRtPAGY46OJxlgAPpS6zM7+r412jR/OxuE7PTSc48o74YGWnUYbpKnmHutUxcwWNw78BJdggahCwfbj/mjhO+t8ad2WpHSqDIG+qQ+oRqmeyt2/7VwumE51jOVWzttNFtzFO/4oFXXRvzxbsaPMLs3W7CS2+XdS203NhsoOmVPLQmAzXXqXVy1fLg4kSdsnMAJvlekH4bbO0th4O8TBM0Yo23yZJysOUIbfypEqdPVcCxYJgXLeydzRcJvymNvnfecac/occ3Yf3L77gIEvPmjbCvUVcBmv+NTpup14cZ6shyqjdFWmeha3Ebz+QCDh+kknBf4FsRiVrlXZTksyqF6H0qYW0o8SqARuCgPaI+kU925r J/FSnONe sN872T7AmNQO6x2l2E5lsU34gh1Ev7ILZnjQf/XEy6zOigkbRCH32Dpjy9gjpAHa1TkEe4OFchfpJKDfXcUnvpDVqpmvS7HgVIWierZhhN4822cnrelTFSAF5DhBiLo7OgWQI1uurNupI22QzRvoV7sORdJMF1LIqZXcjYg6Fsu4a9rhRXhJ13KNZWepnoKRveCdgTqyOvSxlSby/ap15u95ZLXyZURVVSd+ObalFY0AVcj0fcjj1JdDR0wmV0G/rpNW97CxF9b0tgbGk50WEso2mCc7TIC91iVa8ewnNCTOKUQsGZwHeMq5S4du7eOMAkBpJq0r5h9nybaG7CLqMsI7Ituj03L3rAQvvYz/FaVBp5fil5dcjLRGBhNjA5r/MskWWXmokTEzQ2iMlOgREngY03weDe1X6D36/nV9lABuXMhM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.107542, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As a final step an initialization of kvfree_rcu() functionality is copied into slab_common.c from the tree.c file as well as shrinker related code. The function is temporary marked as "__maybe_unused" to eliminate a compiler warnings. Signed-off-by: Uladzislau Rezki (Sony) --- mm/slab_common.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/mm/slab_common.c b/mm/slab_common.c index e7e1d5b5f31b..cffc96bd279a 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1940,3 +1940,94 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp, return true; } + +static unsigned long +kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); + + count += krc_count(krcp); + count += READ_ONCE(krcp->nr_bkv_objs); + atomic_set(&krcp->backoff_page_cache_fill, 1); + } + + return count == 0 ? SHRINK_EMPTY : count; +} + +static unsigned long +kfree_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu, freed = 0; + + for_each_possible_cpu(cpu) { + int count; + struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); + + count = krc_count(krcp); + count += drain_page_cache(krcp); + kfree_rcu_monitor(&krcp->monitor_work.work); + + sc->nr_to_scan -= count; + freed += count; + + if (sc->nr_to_scan <= 0) + break; + } + + return freed == 0 ? SHRINK_STOP : freed; +} + +static void __init __maybe_unused +kfree_rcu_batch_init(void) +{ + int cpu; + int i, j; + struct shrinker *kfree_rcu_shrinker; + + /* Clamp it to [0:100] seconds interval. */ + if (rcu_delay_page_cache_fill_msec < 0 || + rcu_delay_page_cache_fill_msec > 100 * MSEC_PER_SEC) { + + rcu_delay_page_cache_fill_msec = + clamp(rcu_delay_page_cache_fill_msec, 0, + (int) (100 * MSEC_PER_SEC)); + + pr_info("Adjusting rcutree.rcu_delay_page_cache_fill_msec to %d ms.\n", + rcu_delay_page_cache_fill_msec); + } + + for_each_possible_cpu(cpu) { + struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); + + for (i = 0; i < KFREE_N_BATCHES; i++) { + INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); + krcp->krw_arr[i].krcp = krcp; + + for (j = 0; j < FREE_N_CHANNELS; j++) + INIT_LIST_HEAD(&krcp->krw_arr[i].bulk_head_free[j]); + } + + for (i = 0; i < FREE_N_CHANNELS; i++) + INIT_LIST_HEAD(&krcp->bulk_head[i]); + + INIT_DELAYED_WORK(&krcp->monitor_work, kfree_rcu_monitor); + INIT_DELAYED_WORK(&krcp->page_cache_work, fill_page_cache_func); + krcp->initialized = true; + } + + kfree_rcu_shrinker = shrinker_alloc(0, "rcu-slab-kfree"); + if (!kfree_rcu_shrinker) { + pr_err("Failed to allocate kfree_rcu() shrinker!\n"); + return; + } + + kfree_rcu_shrinker->count_objects = kfree_rcu_shrink_count; + kfree_rcu_shrinker->scan_objects = kfree_rcu_shrink_scan; + + shrinker_register(kfree_rcu_shrinker); +}