From patchwork Tue Dec 10 16:40:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Uladzislau Rezki (Sony)" X-Patchwork-Id: 13901747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DBF1C3DA4A for ; Tue, 10 Dec 2024 16:40:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCB5B6B0259; Tue, 10 Dec 2024 11:40:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C2D806B025C; Tue, 10 Dec 2024 11:40:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A828D6B0259; Tue, 10 Dec 2024 11:40:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 829D76B0259 for ; Tue, 10 Dec 2024 11:40:42 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3B6491A0C09 for ; Tue, 10 Dec 2024 16:40:42 +0000 (UTC) X-FDA: 82879612074.08.7128C87 Received: from mail-lf1-f49.google.com (mail-lf1-f49.google.com [209.85.167.49]) by imf08.hostedemail.com (Postfix) with ESMTP id 9AFC9160004 for ; Tue, 10 Dec 2024 16:40:25 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BbdxNRyy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.49 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733848830; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UdPE5CaBHMhT6cavsDrFmlg9kwPywztBv9Mfz0gYFxI=; b=GpyOL3yVBmWJyjS56dScpctB9yxTdTwb8lmrEaM7qv8NiniotZPHQ/vMVJVfpLPBop5w65 Q7tGtVr5BYW8hVlYDP37VKO35hu1RVKnMLwLaWZFtZXpcHwbgVP3o+54JBHwnCiHmV0+q5 3s97fO7yD5OqKPzf10Au15x9QHdnwsk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733848830; a=rsa-sha256; cv=none; b=vorIgMRumpcRqsx8xs43fcrBIh5AGNVoIonZUWx8JFm9QbYfqHANQnACcBnwazKIsHqpSY EwReMqHUysQ82FqkC9MWdhy5PZ6wUuCsmkik0KoB7n6gFOEFFvjqn5vvdTC5UcV9xK+TON A9bstEP/iDCFMnK97c9Z9o47q6aG5CU= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BbdxNRyy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.49 as permitted sender) smtp.mailfrom=urezki@gmail.com Received: by mail-lf1-f49.google.com with SMTP id 2adb3069b0e04-540218726d5so1912266e87.2 for ; Tue, 10 Dec 2024 08:40:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733848839; x=1734453639; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UdPE5CaBHMhT6cavsDrFmlg9kwPywztBv9Mfz0gYFxI=; b=BbdxNRyy1FsWnSFSMmGkq00vcI5pXoXh22p9VbNzqxonjw2mHwoJnyV85rqV4Xi7/b /r/aHwuUpq+voQYHf82I1bZJ2v2UkcTOr1GzVzb6Ow9puFYFkKw/NMQYgfzWTFns+bgo kqbSqhnXr/W9Rb14Eu5H1rhkiiNycqdQhzBHKjBQoacdoLoWjuWu1ulCf31/BH/bJ8Jd erwExfSG32zibJERHP6fQno2rJvZrYG1w6+Z5XzhnGF41dyjavMv9XfJdsO9R1ZVFRq9 DqaNtd/iF0JbxUMM+/j89Ao8JpDGsd2LW0ravs+hPottA/farx0+2lKuVaymiF/aW0Ib pSXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733848839; x=1734453639; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UdPE5CaBHMhT6cavsDrFmlg9kwPywztBv9Mfz0gYFxI=; b=VF7aJdv6dvVcDRtJH2pBO0ZAArhgrFOO6moPEZwOQ1FjowziqTH7HMXwq8zrP03MZq KlL/nfjH7To07DppC8WlaNf73kVQygd51QWelNSGFvDO8pmSjNyM92n6qf9fVdRajyxI uvGuAZCzdnTLwlmw3/mln5IW+cjkldnOT/dsV379KpnXj2VfePuDVXeAMmXsgd5mgzIx hxhPcojvee+LLz6kZJabKmHWEGJuRtTnwBTVQ/7jZ1U4Ap6zvIwLOVNUncgcArCVwCRY +ON16CUaFAPeZS/kxPRr2okagaOglqqMbXN0iB14ldAGJ3IZxtLbsfgNRvMXrmFAlH3j q1Fw== X-Gm-Message-State: AOJu0Yx0popIjJuW2Gj9D27TtRtFgVSfpm0TIFz+D3mwvs/brp5wNfmL TId9CFizSn1JpRWT3kOqgPbuAWXKEpVULYh2u4DCOG1Nrqma16MscOEm+g== X-Gm-Gg: ASbGncvCeCVYBe8XWRnIyfBwmT6Zf6Sk7OMeNFxwgQAD4cMNjoJONJujYJXSCMCfT34 SINFzbUeDeBPRHi5zJ+gwl2+mUNUgyLOD8ShvwOr5/HFsI1OPyXAFcqXabF4aP/XnFF/25JCmxr BFLX0tJcc11khfLxKdk8nPpmBleRC/bbj/RZL73BvdGOvlal7gYRDjO3MaTGyTJmIvG4k/jauA+ MbEYrrD8QHqzRcQclKh6zyCC4A6OnMQfVEdkVNIl6tOzyEeng== X-Google-Smtp-Source: AGHT+IFXg03XavY3zbgVuUO+42mxcr0qW2R92VYVB7iUi/Qw3cgWDDrElu+xTCGcLsSdSOB6pE9uDg== X-Received: by 2002:a05:6512:15a2:b0:540:1abe:d6d2 with SMTP id 2adb3069b0e04-54024107498mr1918094e87.35.1733848838529; Tue, 10 Dec 2024 08:40:38 -0800 (PST) Received: from pc638.lan ([2001:9b1:d5a0:a500:2d8:61ff:fec9:d743]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-53f93377eefsm1031875e87.67.2024.12.10.08.40.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 08:40:37 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton , Vlastimil Babka Cc: RCU , LKML , Uladzislau Rezki , Oleksiy Avramchenko Subject: [RFC v1 1/5] rcu/kvfree: Temporary reclaim over call_rcu() Date: Tue, 10 Dec 2024 17:40:31 +0100 Message-Id: <20241210164035.3391747-2-urezki@gmail.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241210164035.3391747-1-urezki@gmail.com> References: <20241210164035.3391747-1-urezki@gmail.com> MIME-Version: 1.0 X-Stat-Signature: ot94cb5bguu6beebdkipy9qo3343jexx X-Rspamd-Queue-Id: 9AFC9160004 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1733848825-360789 X-HE-Meta: U2FsdGVkX19DiqHGlG+F+BJ599RYZWYqcp0QroClPV+9vSQLGT+D362SNYW7Q7mhgp/QzJDGLKq3EndPs21CyxW3icER8RqTBtUGzFqP2jZA19orqnqh3lwbtiiQXLojQqNF7zpDP5shLb62wC5U86GAIypi8VgpmHEGlG868LZs7GuvK9Ryj+k2oSRkVuMdgDW1BlL2WC93kTUw3Grkm58zXpuWNnYNYThsBf283BKkIlhDvmB7fbOd/OGn+vn9SYlvo1+Rsdrm+tzLNK/nYiaAb6d5jvw/GQBlbjKDTVxtfYvJeSHExCdtxr/EnGGsdGh4hbYIb+2o2eBfzjyMRyuh4jZDLDzSX2uFuI1MuBvgXG3jdrQVKc/z3JUf93UuHtAML1tOcnWiM08R4r87ksdB0sEzKKpxFsWpwvp70q5H7uctxBSW8M89yp9CrJSlXamy426H1XKsYi5USyuxng4czqBcuywDkJQqftrlE5qxD3j7ieHXr8kosynq23hEzy5Ejfj/1eowRRzuLOZtAjHp0PRvRkSw19HnaHaGporHGvqbijq/yOm2EX5vaolQMdJNuIlLhd78/qULjw034O+V6T9VpIJbG4Jln6O0NzQaz44Bzh6ZczO6UFfwfyg3JgWhXNaBVa5JYPIsi3ZQwpsfCDB4+s4gxvxAvlcAR0tinjB/GId6lWlUBfPTdQxyzwfYGBU+T3ucCossUDICEEzQgwqTZcaSf7yjWMKa8tb+vn1aKbc7yBN2oy0jMwEOrok4jdfbrwVxoIHIo8+XHtx+kymdgGoCapdr6DAGogmHyojbMV/S4D6mfPoqoT3TCdxkPbLZIzKT47nzfnS3hHwGRhqFmtJIXyqrk3mwdc1UcCPjHF4ofnFRhFyI9QWJDDzHX5uEYYp54FYmjXkCbtmHpb/lCGKy+XFtJn5ainvuE3EXFxTxsyw9Xjv2+wubpECYt0a20PwhUpnh9/L YurUoRn3 OaMy0uIVtXGMHeKvgcHeEkD0EQiHaG1wjRW2uHQTq60OdOdFt703e6CqJx4OwgpjIk3V9lHOAtB6iy/3lvEKYEET45NHx221gTddXcULyXiisCrdgKY5jHhDapYyG50UyNxBS92cvqy39+lsL+5HIP9CEHALCFZ6W2X6bPEEEuv4dwg5a/24R6vJlZJHKYDzGtjFjDOpHQaSJWMm8/zchAEOaf3aC+R8eng9jjm5Qq/Ar28GtbShH8zVBdG1eAkCvuQYsVKTBh7VJEThSgRYzTyRQe/uNto2zSU7t9uxSF0PJee+6WrVjEhiqFOQbYJ2A2d/Qv43Fy/RBFCeTHBtvANev4e2khFly5NCGBYgHpejBUXXondh0iiosZ/I4H3AkQxk4gX/Xr+47/auQnAyOFuzTk9qcFMpVZTsvoUj/oPQDfXA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.036238, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is to start a smooth process of moving a main functionality to the SLAB. Therefore this patch: - adds a support(temporary) to reclaim freed objects over call_rcu(); - disconnects a main functionality of kvfree_rcu() API by using call_rcu(); - directly reclaims an object for a single-argument variant; - adds an rcu_barrier() call to the kvfree_rcu_barrier(). Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index b1f883fcd918..ab24229dfa73 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2559,13 +2559,19 @@ static void rcu_do_batch(struct rcu_data *rdp) debug_rcu_head_unqueue(rhp); rcu_lock_acquire(&rcu_callback_map); - trace_rcu_invoke_callback(rcu_state.name, rhp); f = rhp->func; - debug_rcu_head_callback(rhp); - WRITE_ONCE(rhp->func, (rcu_callback_t)0L); - f(rhp); + /* This is temporary, it will be removed when migration is over. */ + if (__is_kvfree_rcu_offset((unsigned long) f)) { + trace_rcu_invoke_kvfree_callback("", rhp, (unsigned long) f); + kvfree((void *) rhp - (unsigned long) f); + } else { + trace_rcu_invoke_callback(rcu_state.name, rhp); + debug_rcu_head_callback(rhp); + WRITE_ONCE(rhp->func, (rcu_callback_t)0L); + f(rhp); + } rcu_lock_release(&rcu_callback_map); /* @@ -3787,6 +3793,16 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) struct kfree_rcu_cpu *krcp; bool success; + if (head) { + call_rcu(head, (rcu_callback_t) ((void *) head - ptr)); + } else { + synchronize_rcu(); + kvfree(ptr); + } + + /* Disconnect the rest. */ + return; + /* * Please note there is a limitation for the head-less * variant, that is why there is a clear rule for such @@ -3871,6 +3887,9 @@ void kvfree_rcu_barrier(void) bool queued; int i, cpu; + /* Temporary. */ + rcu_barrier(); + /* * Firstly we detach objects and queue them over an RCU-batch * for all CPUs. Finally queued works are flushed for each CPU.