From patchwork Tue Dec 10 16:40:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13901748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06EFCE7717F for ; Tue, 10 Dec 2024 16:40:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3CA886B025C; Tue, 10 Dec 2024 11:40:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3567B6B025E; Tue, 10 Dec 2024 11:40:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0BBAE6B025F; Tue, 10 Dec 2024 11:40:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C993C6B025C for ; Tue, 10 Dec 2024 11:40:43 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 85DADAEA23 for ; Tue, 10 Dec 2024 16:40:43 +0000 (UTC) X-FDA: 82879612284.06.2EC3E3D Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) by imf12.hostedemail.com (Postfix) with ESMTP id D7AA34001C for ; Tue, 10 Dec 2024 16:40:31 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Oye+Yj14; spf=pass (imf12.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.47 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733848831; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ClETbVzs2/eIlopkwOYmZeoS18KY5TfMVLeWid2GulI=; b=pUCFawp9CWT7v6TBEBGmD2K+Dk/20pRpDnNR+3Igc2LM+MkbHDg+q0kLgINGpK7UKrKE6D lyqD6pcD69r+2rYuX73b+y0rxk3xyMU3qIKjpmXTrQvTJI7YrDY3oUz4sn5YgCcjnh1qUO Bk8XTVuSJGgp1DMQ7Cfft4qx6mitvic= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Oye+Yj14; spf=pass (imf12.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.47 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733848831; a=rsa-sha256; cv=none; b=fBbjLDkS5SjTdFjkbcgDeWCjAY3Q/+r4sqA0YmJlpV8AqHgpJA3GrbFsslM4ZCrlc6lQiJ lHwMahS71QpsSfvSh7KvDSgCLeaGDhDmxH/Hmfomcn8hggbStS4dRQSxiIBef6E00kCEL+ ZyMvWwWFlf/epGvsQzJZMXmBVuKQL/0= Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-5401c52000fso2401050e87.2 for ; Tue, 10 Dec 2024 08:40:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733848840; x=1734453640; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ClETbVzs2/eIlopkwOYmZeoS18KY5TfMVLeWid2GulI=; b=Oye+Yj14vF93oX7Esf2xKNQp7Q9klYVUyg3h8/UzEKxD3oRtMQg4Ld34nZdZJH72gi 5i+6h+7kStdFvwBn7mRQQsdDHwdwcliOxU6yKieJ3KgsZ8HBOK7OuyDLwqiY64vL3oCx 2IZqAbpzQeUQVDOJE9sGO2l87mdtU0U7ze/qBNYftE3YDwBfXMf/5sYlRhxJ8WtTF7uJ MPHtXeb5w/ioH0fBym1Mf2ejPweiCBoPs5pqwkCA2/PmVqdLgMHNrEczUyNiqo4w9Xpe TZf8ffBvs0+T+ckpaOf2GYzhsdmIDjBu5CWbUQof1mRw5TdCyFTnnFMDlC9LzVjL7Xrn YFsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733848840; x=1734453640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ClETbVzs2/eIlopkwOYmZeoS18KY5TfMVLeWid2GulI=; b=IgRHCmd5vspRHf34DI1eW6GA6qic4mhylwcOvHUptQ0d4Zy8LZ/sYiMSI8328/+L+v qMqm0dMmmadxhaH6hM/beDoIHP4Ns2iexVMSGmeAeK7V6CYcQA4gcweWJv1Q78GBEJ5z ZUKh6wMB0K+gQWo1zXxEguQfw7Z+mqxJHEhXhuJUbdwTx/lGc4Up6ibT4AIKMWaccx4U n0np+wv08khHVdSxW3G33rUiWvWAHwAEjqdN4JPpXemh2A/5PjFd9Hs466dS5BXK7bxB Qak9sSlbpEAIwnfjWbLRGbTZOncGycVt7w1KGpbhD0euvwz3MllXft6bwbo1maaxNPcK Eg3g== X-Gm-Message-State: AOJu0YxXLt/zXcAGLpGynvPWMPrLdNNXhyav//15aGpQH6Laj7ZqGghQ uADKz1MMUwJ1xxnIYXg3JIy/ynmDLoI1ZPByG61VZG2K6nvzEdgVoe14gw== X-Gm-Gg: ASbGncujM1Umcq3Kw8GQNbdVwhsYzLO4f+ltFEkS4C66p9vbXp3gCB09yH3vDd17GrW Iwxc4HZc+GEw2FYtDqc2QdT9xOq+ZeRQaulJJMSyjEGz92pZLRkDWVWS/O/rb99113xIaqA47Is cUvtDKckTl6fqvQBvKRpQcSn9vWI5pWxqGKbaQXuif87yNUwTD7mqylFAsGtuvZ62dI3XL52CPa LFcghbYcIzClciPddIKpBqpLQlWFa15LwiBTtMSwpaeWUxSJw== X-Google-Smtp-Source: AGHT+IFLmxwSZlSjwaR27QpJke6rURhJNzaMl3K1lpQik83HRlwxxz03FF+x+GNaY58FSlMOxRq9gA== X-Received: by 2002:a05:6512:ba6:b0:540:1fec:f322 with SMTP id 2adb3069b0e04-5402410485emr1773977e87.39.1733848839554; Tue, 10 Dec 2024 08:40:39 -0800 (PST) Received: from pc638.lan ([2001:9b1:d5a0:a500:2d8:61ff:fec9:d743]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-53f93377eefsm1031875e87.67.2024.12.10.08.40.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 08:40:38 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton , Vlastimil Babka Cc: RCU , LKML , Uladzislau Rezki , Oleksiy Avramchenko Subject: [RFC v1 2/5] mm/slab: Copy main data structures of kvfree_rcu() Date: Tue, 10 Dec 2024 17:40:32 +0100 Message-Id: <20241210164035.3391747-3-urezki@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241210164035.3391747-1-urezki@gmail.com> References: <20241210164035.3391747-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D7AA34001C X-Stat-Signature: k3pmhbgt5fjgmmsg5ixpdhc4yun3ibe4 X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733848831-733685 X-HE-Meta: U2FsdGVkX19dOoox06ZZcqnnEPri7+csyfNVbjHQavFWmrW8D6ijolRNB5AWwxPwemUxD09XdgDqWif74cY6sJnajQxQw8Yphg+4Uav/HmKcws3h9QigWlMzA31SjgFMb0Fh3c/tnb36V8YclMiqmrqysB+on0IGSg2y6SF0f+/FfhLuFfh9yqTzs9ElnJcbY3xLuayC7+3RdgaS3NXE1gCxxWhG2Yu+t2IiUZCKWQ4La/xQhnfF2DNoOTFUI7vtXiyG20ReJPZKhupiIJk+FKZ5Q+JxkwOYJ6ubjFhyrzXueG2xlEJ1wbuO6nnSLwf7ggip/fx0n1pBHN8yMUpvmTkGYnDXRGgM7udnolJjDXgpdGXGmy8uTSGhvZB8oMYsxp9zGQENTqPSS3+RoAq7whnysKL3AAyc69mBk0CptpvhH1BA95d3Y1xzLhPOWZWf2/bjf1ElCu0N3OkgqgVkHXtJ0oaAmb2/b2+3Ct0+yjDv+nV4CBkMIIRYTdGgl1jWi5bqgCuhs8da2eeAhIseA9OEf7YKCP2Q4XR8L4Vz+C0FOUH/R2vk4SFTlNvIVG3CZ75YX41MuIEg4qFLxjCn/8j+fztgvmj9Kf0dqQOD0xhl2Hda4J6lklQ/c41hT/taRsTTLuHwHyBpbG7Gc2ejysMK8jonby/txqQ+uRO4jSBz4Zunu7O2hYJQWx5l6ziWENBK+wxp3rr226V3yqskQpk0iI/HuKR8tAmHrwSsYTwJvZUDkBPHzuPLsusa16wfJJsPXSjTPa6yftruNErq1mwpxcslPlWg5oPzSroofXXT9xDDvUYWjxBLPjESL5Jfo0GkoBi9asDal/0TEPnN8dEkLo0AozNDHh32hKvUqTXtXkvS8ws/YzTFUDneFfTotBR17XcflOpYGvQjU9yFA3rygVeA9lO9EnbhKKs4KE6WfpY9l9Zl3ur8bYFy1pcNhPj9tSauycJ8UgZSEhq +Ahf7CrF cuiVmOHNFUezXFak/YJ8I3Ae9Z827BSomsquiuF0tKSG/AvzASbcLPTSBoYop6jMNG3+Z7OgUDgf/AYICKIFChkEpbFb1Dv3n43Xxdq8GK4UJ2yL50PS+8ekquffNzM5DJ63DVSE27o9asCbc37S0QDEwOTzw+cPabhtBnAaIdlV5TO0POpkWeFjBNW+zVBkcOuvSA2Vzix/wRmAHmyyJYM+lfTvQ1A9HI7TtZTp3gpN13Z3h7fM9fZ1PfJZRPND9Y6Orp7cDhrZSpTXhUXWEYmXp1kvnWq3PbAAPGPvZSxINMROKpnIiFFEENLN/5tvTKxP0L+nRcVzHWZJPhYDAnCwWw+q2fPJeZo51zWg/dbbFTPVvz6JawXuMrunoiAYqY/JB/Xgd/w7LEK4tTp0atnfwua/wt03g5OBrrgoa6Nr8qjM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch copies main data structures of kvfree_rcu() API from the kernel/rcu/tree.c into slab_common.c file. Later on, it will be removed from the tree.c. Signed-off-by: Uladzislau Rezki (Sony) --- mm/slab_common.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) diff --git a/mm/slab_common.c b/mm/slab_common.c index 893d32059915..a249fdb0d92e 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1338,3 +1338,98 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); +/* Maximum number of jiffies to wait before draining a batch. */ +#define KFREE_DRAIN_JIFFIES (5 * HZ) +#define KFREE_N_BATCHES 2 +#define FREE_N_CHANNELS 2 + +/** + * struct kvfree_rcu_bulk_data - single block to store kvfree_rcu() pointers + * @list: List node. All blocks are linked between each other + * @gp_snap: Snapshot of RCU state for objects placed to this bulk + * @nr_records: Number of active pointers in the array + * @records: Array of the kvfree_rcu() pointers + */ +struct kvfree_rcu_bulk_data { + struct list_head list; + struct rcu_gp_oldstate gp_snap; + unsigned long nr_records; + void *records[] __counted_by(nr_records); +}; + +/* + * This macro defines how many entries the "records" array + * will contain. It is based on the fact that the size of + * kvfree_rcu_bulk_data structure becomes exactly one page. + */ +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) + +/** + * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests + * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period + * @head_free: List of kfree_rcu() objects waiting for a grace period + * @head_free_gp_snap: Grace-period snapshot to check for attempted premature frees. + * @bulk_head_free: Bulk-List of kvfree_rcu() objects waiting for a grace period + * @krcp: Pointer to @kfree_rcu_cpu structure + */ + +struct kfree_rcu_cpu_work { + struct rcu_work rcu_work; + struct rcu_head *head_free; + struct rcu_gp_oldstate head_free_gp_snap; + struct list_head bulk_head_free[FREE_N_CHANNELS]; + struct kfree_rcu_cpu *krcp; +}; + +/** + * struct kfree_rcu_cpu - batch up kfree_rcu() requests for RCU grace period + * @head: List of kfree_rcu() objects not yet waiting for a grace period + * @head_gp_snap: Snapshot of RCU state for objects placed to "@head" + * @bulk_head: Bulk-List of kvfree_rcu() objects not yet waiting for a grace period + * @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period + * @lock: Synchronize access to this structure + * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES + * @initialized: The @rcu_work fields have been initialized + * @head_count: Number of objects in rcu_head singular list + * @bulk_count: Number of objects in bulk-list + * @bkvcache: + * A simple cache list that contains objects for reuse purpose. + * In order to save some per-cpu space the list is singular. + * Even though it is lockless an access has to be protected by the + * per-cpu lock. + * @page_cache_work: A work to refill the cache when it is empty + * @backoff_page_cache_fill: Delay cache refills + * @work_in_progress: Indicates that page_cache_work is running + * @hrtimer: A hrtimer for scheduling a page_cache_work + * @nr_bkv_objs: number of allocated objects at @bkvcache. + * + * This is a per-CPU structure. The reason that it is not included in + * the rcu_data structure is to permit this code to be extracted from + * the RCU files. Such extraction could allow further optimization of + * the interactions with the slab allocators. + */ +struct kfree_rcu_cpu { + // Objects queued on a linked list + // through their rcu_head structures. + struct rcu_head *head; + unsigned long head_gp_snap; + atomic_t head_count; + + // Objects queued on a bulk-list. + struct list_head bulk_head[FREE_N_CHANNELS]; + atomic_t bulk_count[FREE_N_CHANNELS]; + + struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; + raw_spinlock_t lock; + struct delayed_work monitor_work; + bool initialized; + + struct delayed_work page_cache_work; + atomic_t backoff_page_cache_fill; + atomic_t work_in_progress; + struct hrtimer hrtimer; + + struct llist_head bkvcache; + int nr_bkv_objs; +};