From patchwork Mon Mar 30 02:32:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464465 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D6C561392 for ; Mon, 30 Mar 2020 02:33:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9ABDD2078B for ; Mon, 30 Mar 2020 02:33:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="rxDfrhuk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9ABDD2078B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9179B6B0007; Sun, 29 Mar 2020 22:33:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 879CB6B0008; Sun, 29 Mar 2020 22:33:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F34C6B0037; Sun, 29 Mar 2020 22:33:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 573136B0007 for ; Sun, 29 Mar 2020 22:33:33 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 51B0F8248D7C for ; Mon, 30 Mar 2020 02:33:33 +0000 (UTC) X-FDA: 76650457506.08.badge77_59369116bf253 X-Spam-Summary: 2,0,0,5dea9318ae8824fa,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:3868:3871:3872:3874:4250:4321:4385:4390:4395:5007:6261:6653:6742:7514:7576:10004:11026:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13069:13311:13357:13894:13972:14096:14181:14384:14394:14721:21080:21444:21451:21627:21990:30054,0,RBL:209.85.160.196:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:28,LUA_SUMMARY:none X-HE-Tag: badge77_59369116bf253 X-Filterd-Recvd-Size: 4746 Received: from mail-qt1-f196.google.com (mail-qt1-f196.google.com [209.85.160.196]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:32 +0000 (UTC) Received: by mail-qt1-f196.google.com with SMTP id t9so13906375qto.9 for ; Sun, 29 Mar 2020 19:33:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=j9u1d7Nq8RT0AOHNZvylOBJ81hqFkLhdoH0L2zGb794=; b=rxDfrhuk8LqW6ZnmW+LnImOtpzM2Zw9S69J/P+yG2hd6Vw5mMl8yBevuQ6ntcQkpt1 6sS/aa7qsRAXfO8SA/uVHNRUM3XnlchLsYPPO6idkXfPZNHXzbwNfmaWP7vJXU7yG8GR cAXpIptCUsYctVOx9/SD1kmPf6JJ5CIL5InT4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=j9u1d7Nq8RT0AOHNZvylOBJ81hqFkLhdoH0L2zGb794=; b=RALkGkHyJtQAOVpea+44Bd8RW111d6FXhYKL8o3ZCTGi9ZfJR/oqOJhdolbrdlsybZ DxcvG1uNV4XHuLRJDqQQRYfBS8WNXgI+Bdq+d1ncu9fsC2R9HZROzIGX/J/t0+BVgSAr qss3V1HZh6o9rBxYOn431krh5NaiSJ7RbkG7xjeFQISwN2kxJ3fUyEt4qBbMMAnKVoFL wnXyrsHXmZwieVgr1ds85mP0akAnOZQjxJoGL1KCBo1/nxCa8t/WeH3ifEM7KU9f5vi1 x01LWchS5bCpRMYtFUgKBEbqLlaNX1phVsc9vUTVeFgSBwXnK8O4NKXx+kyQsOU5Jdi2 GZjw== X-Gm-Message-State: ANhLgQ0DAxMwc27eZhAwmPbU1zJczKcij39CUbiDr9k6ke31FSMUeIq8 uUxBOIqolm08J+rsgkD4OcooCg== X-Google-Smtp-Source: ADFU+vtJ/nFZMvAXPR8p2akuS9nHt322wcY7VnnK+V09LMOq6oHXy/eGbZOe0mEFY2W1AiDcgsCNJg== X-Received: by 2002:ac8:72ce:: with SMTP id o14mr10086333qtp.226.1585535612029; Sun, 29 Mar 2020 19:33:32 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:31 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 01/18] mm/list_lru.c: Rename kvfree_rcu() to local variant Date: Sun, 29 Mar 2020 22:32:31 -0400 Message-Id: <20200330023248.164994-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Rename kvfree_rcu() function to the kvfree_rcu_local() one. The aim is to introduce the public API that would conflict with this one. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- mm/list_lru.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 0f1f6b06b7f36..386424688f805 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -383,14 +383,14 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru) struct list_lru_memcg *memcg_lrus; /* * This is called when shrinker has already been unregistered, - * and nobody can use it. So, there is no need to use kvfree_rcu(). + * and nobody can use it. So, there is no need to use kvfree_rcu_local(). */ memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true); __memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids); kvfree(memcg_lrus); } -static void kvfree_rcu(struct rcu_head *head) +static void kvfree_rcu_local(struct rcu_head *head) { struct list_lru_memcg *mlru; @@ -429,7 +429,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru, rcu_assign_pointer(nlru->memcg_lrus, new); spin_unlock_irq(&nlru->lock); - call_rcu(&old->rcu, kvfree_rcu); + call_rcu(&old->rcu, kvfree_rcu_local); return 0; } From patchwork Mon Mar 30 02:32:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464467 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB21A13A4 for ; Mon, 30 Mar 2020 02:33:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 61D272082E for ; Mon, 30 Mar 2020 02:33:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="EZj/LdDI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 61D272082E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B627F6B0008; Sun, 29 Mar 2020 22:33:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AEB566B0037; Sun, 29 Mar 2020 22:33:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B2E26B006C; Sun, 29 Mar 2020 22:33:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 7DD396B0008 for ; Sun, 29 Mar 2020 22:33:34 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7DC6A1867 for ; Mon, 30 Mar 2020 02:33:34 +0000 (UTC) X-FDA: 76650457548.29.beds18_595c5bf626808 X-Spam-Summary: 2,0,0,fcb277b2b14d5255,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1605:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2736:2914:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:4321:4385:4390:4395:4605:5007:6261:6653:6742:7514:7576:7875:8603:9010:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13161:13229:13894:14096:14181:14394:14721:21080:21444:21451:21627:21740:21795:21987:21990:30029:30051:30054:30070,0,RBL:209.85.160.195:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: beds18_595c5bf626808 X-Filterd-Recvd-Size: 7507 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:33 +0000 (UTC) Received: by mail-qt1-f195.google.com with SMTP id g7so13882482qtj.13 for ; Sun, 29 Mar 2020 19:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SHLAtJMh4QvoU/qfwuEaXXVei4ZAh6L5O9IfzaCS1DM=; b=EZj/LdDIb8HO3LYMjwICXSEVb7Ug5CB6y+9yeeADTqwVWhn/iu4pUuRlJQp8IWjNRv ZHhEK+CiED++Nq7ZmpKIIk9AzYqhwZhx4OfWzy5quBd5oc00JjePZcq6xqYMOn+tAGOG pzdgsMtvd6/BeARibxZlYR4poF6nx+XzQnoHc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SHLAtJMh4QvoU/qfwuEaXXVei4ZAh6L5O9IfzaCS1DM=; b=Iu5C1SOexgAPKtUHISDWJool4Le8njaQ3xRS8zg6z9H5DSmFHm6iKSkk13RyEwRzma 2d4qmEFm9MJpA8IYxj1zpayW1Xh6E4a9dkHDvm2GoEZ7+8FcuVQ2gD/Q6XtUj8dcOOgR V5EHdK82AwUJFAyB8fthvAHiWrL9XoR6T+A4umlMjAROtTCT8NyQxuLk7HmVB/4k46j1 j6OkvQE8FoMlO8hwwqJQJ+JCvEuXue7h6t7VBQjcy4SDxAXo6cc7IonMPmiDklOM7XNB YgtWejR6IY0LAHMf7k6AZ/MTfKZ+68t86JnEg2mK5FGP7xo/Z+pwTH4QrImjB6oO/5YU 0xJg== X-Gm-Message-State: ANhLgQ07koM/8b7+mUoSStOYYx2r9YzXxf6SHyUBTkheeSdCnYq8DOoK TfL/qxr75qr1ShSf3YzOdli4pjBj9Co= X-Google-Smtp-Source: ADFU+vt25iE1CsRcLCeax/iZQAcxxqmJ1TSX0Yl3KB0TJZ2nCudJEfCk4XFJ5v786E3wtic9LAXdDA== X-Received: by 2002:aed:2535:: with SMTP id v50mr10066898qtc.354.1585535613043; Sun, 29 Mar 2020 19:33:33 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:32 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 02/18] rcu: Introduce kvfree_rcu() interface Date: Sun, 29 Mar 2020 22:32:32 -0400 Message-Id: <20200330023248.164994-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" kvfree_rcu() can deal with an allocated memory that is obtained via kvmalloc(). It can return two types of allocated memory or "pointers", one can belong to regular SLAB allocator and another one can be vmalloc one. It depends on requested size and memory pressure. Based on that, two streams are split, thus if a pointer belongs to vmalloc allocator it is queued to the list, otherwise SLAB one is queued into "bulk array" for further processing. The main reason of such splitting is: a) to distinguish kmalloc()/vmalloc() ptrs; b) there is no vmalloc_bulk() interface. As of now we have list_lru.c user that needs such interface, also there will be new comers. Apart of that it is preparation to have a head-less variant later. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 9 +++++++++ kernel/rcu/tiny.c | 3 ++- kernel/rcu/tree.c | 17 ++++++++++++----- 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 3598bbb5ff407..8b7128d0860e2 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -870,6 +870,15 @@ do { \ __kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ } while (0) +/** + * kvfree_rcu() - kvfree an object after a grace period. + * @ptr: pointer to kvfree + * @rhf: the name of the struct rcu_head within the type of @ptr. + * + * Same as kfree_rcu(), just simple alias. + */ +#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf) + /* * Place this after a lock-acquisition primitive to guarantee that * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index dd572ce7c7479..4b99f7b88beec 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "rcu.h" @@ -86,7 +87,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) rcu_lock_acquire(&rcu_callback_map); if (__is_kfree_rcu_offset(offset)) { trace_rcu_invoke_kfree_callback("", head, offset); - kfree((void *)head - offset); + kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); return true; } diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 4eb424eb44acb..2d10c50621c38 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2925,9 +2925,9 @@ static void kfree_rcu_work(struct work_struct *work) } /* - * Emergency case only. It can happen under low memory - * condition when an allocation gets failed, so the "bulk" - * path can not be temporary maintained. + * vmalloc() pointers end up here also emergency case. It can + * happen under low memory condition when an allocation gets + * failed, so the "bulk" path can not be temporary maintained. */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; @@ -2938,7 +2938,7 @@ static void kfree_rcu_work(struct work_struct *work) trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset); if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) - kfree((void *)head - offset); + kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3112,10 +3112,17 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) } /* + * We do not queue vmalloc pointers into array, + * instead they are just queued to the list. We + * do it because of: + * a) to distinguish kmalloc()/vmalloc() ptrs; + * b) there is no vmalloc_bulk() interface. + * * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (unlikely(!kfree_call_rcu_add_ptr_to_bulk(krcp, head, func))) { + if (is_vmalloc_addr((void *) head - (unsigned long) func) || + !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) { head->func = func; head->next = krcp->head; krcp->head = head; From patchwork Mon Mar 30 02:32:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464469 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 29CE813A4 for ; Mon, 30 Mar 2020 02:33:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D41AB20842 for ; Mon, 30 Mar 2020 02:33:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="x1VGsvdO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D41AB20842 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BB1356B0037; Sun, 29 Mar 2020 22:33:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B124F6B006C; Sun, 29 Mar 2020 22:33:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B73B6B006E; Sun, 29 Mar 2020 22:33:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id 83DA16B0037 for ; Sun, 29 Mar 2020 22:33:35 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 617FF180AD817 for ; Mon, 30 Mar 2020 02:33:35 +0000 (UTC) X-FDA: 76650457590.12.wound85_598361dcd9d51 X-Spam-Summary: 2,0,0,26a709705d8e8b66,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3871:3872:3874:4117:4250:4321:4385:4390:4395:4605:5007:6119:6261:6653:6742:7514:7576:7875:10004:11026:11473:11658:11914:12043:12114:12296:12297:12438:12517:12519:12555:12895:13894:13972:14096:14181:14394:14721:21080:21212:21444:21451:21627:21740:21772:21987:21990:30034:30054:30070:30076,0,RBL:209.85.160.193:@joelfernandes.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: wound85_598361dcd9d51 X-Filterd-Recvd-Size: 6906 Received: from mail-qt1-f193.google.com (mail-qt1-f193.google.com [209.85.160.193]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:34 +0000 (UTC) Received: by mail-qt1-f193.google.com with SMTP id a5so13897766qtw.10 for ; Sun, 29 Mar 2020 19:33:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Wx0JY2YKiW3y6xXQtIfMDOCqiN7fPBTCk6pvf651C60=; b=x1VGsvdOgbW/pXh3K2fqEFIrtX7CGT8Kj9GkfUuhJ8X66LsZN584UWKXUs/dyaR6Uy rskvRF/ZZBYw63xgUm1Ivc2mBaaKNEoUxVmZK6nYp83mdBOhL3z+bR5+bMB7SLBVqF2H 5bdDD8rjYW1c5sAQfJh2fVXftxZB5W0S4KVZM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Wx0JY2YKiW3y6xXQtIfMDOCqiN7fPBTCk6pvf651C60=; b=tH20nAsQwjzjuWMyKt+bGgNPLP4IDeWAyM2yobFQ1ybn2CvE69wd56aCdcRRI0GQ/a dcuwts3VaTsub66+gck6gbJUNEJZ1sVI4OwMM6tIxkZPjv+m0xLYwdFDNAYvi/T1yTAi TiXO9Wxbmf5vMGHi/lFdIpir4JdOvZBobAVmvSOTEbGVzL1PCymf3n14MA2gpFPR3jME PbJcVJLkw2eBs+c2doe89YgGjJZ9n0LpXej1dHkzCQPAFIiT8m00ytnQ9yMrjemMOmkX N7i9yI52ZhOytIFk6fIsLU914Z45wGZKIoN5ziIH/iGRr935KgC8+Ua2iGvB9yP9DS/p 867Q== X-Gm-Message-State: ANhLgQ0mgonm3PClJEteSLJr0bo2/zL9qamqY39lUJOa4ANklfBud2SS X2OqYNiPznLFcQEzRAgp8NSbfQ== X-Google-Smtp-Source: ADFU+vtq1MY/K8kiAE34+qHLDxZNEQyC7+n4jsYMeFFHn7w7o9ddSCQKGt3nomzKdenQfVtzBbc36g== X-Received: by 2002:ac8:6f46:: with SMTP id n6mr9692172qtv.119.1585535614134; Sun, 29 Mar 2020 19:33:34 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:33 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 03/18] rcu: Rename rcu_invoke_kfree_callback/rcu_kfree_callback Date: Sun, 29 Mar 2020 22:32:33 -0400 Message-Id: <20200330023248.164994-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Rename rcu_invoke_kfree_callback to rcu_invoke_kvfree_callback. Do the same with second trace event, that is rcu_kfree_callback, it becomes rcu_kvfree_callback. The reason is to be aligned with kvfree notation. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/trace/events/rcu.h | 8 ++++---- kernel/rcu/tiny.c | 2 +- kernel/rcu/tree.c | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index f9a7811148e2a..0ee93d0b1daa8 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -506,13 +506,13 @@ TRACE_EVENT_RCU(rcu_callback, /* * Tracepoint for the registration of a single RCU callback of the special - * kfree() form. The first argument is the RCU type, the second argument + * kvfree() form. The first argument is the RCU type, the second argument * is a pointer to the RCU callback, the third argument is the offset * of the callback within the enclosing RCU-protected data structure, * the fourth argument is the number of lazy callbacks queued, and the * fifth argument is the total number of callbacks queued. */ -TRACE_EVENT_RCU(rcu_kfree_callback, +TRACE_EVENT_RCU(rcu_kvfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset, long qlen), @@ -596,12 +596,12 @@ TRACE_EVENT_RCU(rcu_invoke_callback, /* * Tracepoint for the invocation of a single RCU callback of the special - * kfree() form. The first argument is the RCU flavor, the second + * kvfree() form. The first argument is the RCU flavor, the second * argument is a pointer to the RCU callback, and the third argument * is the offset of the callback within the enclosing RCU-protected * data structure. */ -TRACE_EVENT_RCU(rcu_invoke_kfree_callback, +TRACE_EVENT_RCU(rcu_invoke_kvfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset), diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 4b99f7b88beec..3dd8e6e207b09 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -86,7 +86,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) rcu_lock_acquire(&rcu_callback_map); if (__is_kfree_rcu_offset(offset)) { - trace_rcu_invoke_kfree_callback("", head, offset); + trace_rcu_invoke_kvfree_callback("", head, offset); kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); return true; diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 2d10c50621c38..88b744ce896c0 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2744,7 +2744,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); if (__is_kfree_rcu_offset((unsigned long)func)) - trace_rcu_kfree_callback(rcu_state.name, head, + trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, rcu_segcblist_n_cbs(&rdp->cblist)); else @@ -2935,7 +2935,7 @@ static void kfree_rcu_work(struct work_struct *work) next = head->next; debug_rcu_head_unqueue(head); rcu_lock_acquire(&rcu_callback_map); - trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset); + trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) kvfree((void *)head - offset); From patchwork Mon Mar 30 02:32:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464471 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C96ED1392 for ; Mon, 30 Mar 2020 02:33:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7F9CC20735 for ; Mon, 30 Mar 2020 02:33:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="NmZKgN+R" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7F9CC20735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9B4AF6B006C; Sun, 29 Mar 2020 22:33:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 93CA66B006E; Sun, 29 Mar 2020 22:33:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E0066B0070; Sun, 29 Mar 2020 22:33:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0029.hostedemail.com [216.40.44.29]) by kanga.kvack.org (Postfix) with ESMTP id 5E3E16B006C for ; Sun, 29 Mar 2020 22:33:36 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 5F785442C for ; Mon, 30 Mar 2020 02:33:36 +0000 (UTC) X-FDA: 76650457632.13.war82_59a75dc38fd1c X-Spam-Summary: 40,2.5,0,8f4df0230ed0f7ca,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3872:3874:4117:4250:4321:4385:4390:4395:4605:5007:6119:6261:6653:6742:7514:7576:7875:7903:10011:11026:11232:11473:11658:11914:12043:12114:12296:12297:12438:12517:12519:12555:12895:13894:14096:14181:14394:14721:21080:21433:21444:21451:21627:21740:21772:21987:21990:30054,0,RBL:209.85.222.195:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:1:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: war82_59a75dc38fd1c X-Filterd-Recvd-Size: 6171 Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:35 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id l25so17554929qki.7 for ; Sun, 29 Mar 2020 19:33:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MWGW0MR3qR5t+EjFJ/mWsxINnYeSYUXBIkqQcwzdLbg=; b=NmZKgN+R6zdEVE3IUCLOA6RtsN34sPbyKaa8mP1kEDPpsLSLDC2DRHp/Vjfoq2CUsJ 3z6jqV54GVqNx79lxtwaXeg+AIQWD+hvzHrkv8VABz1eDzoUfItrMN3CwpM0LL+n+bCv 013+kkgAk/r2Hrrd9cRQzHJsa+++wRfznqSKc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MWGW0MR3qR5t+EjFJ/mWsxINnYeSYUXBIkqQcwzdLbg=; b=PSCOqFQrEc9YqN0KS5H/RDvobyUaET8QQ9WgMBSRM3C+Slr9dTwVl8cq3Dj5GNu0Bq ab4mDai48X3NUPaNZsgZP49WOe4OikVXhbGUbG1cQ/XdWk+6EYtHIMcFD4t39+1csZCA oVK/F2Q43ZBCZkV8v39XEpkG8+y5XPTxQp026xFm5zmFULccmPps3+yPXC6aY+9yuuoo 4z4CacHlefbkGGonJ4E1C3fVlnAe5bi7d56/Z6chNSeE7E2xSehBatwiNI5/JmEx491J uyiE9mMHcxj1AMUOFGUpAGuHQa6CaoAXHcq8XXA1RfHJ7QcfsJThdm1uwnOReFijU1+p zTTg== X-Gm-Message-State: ANhLgQ3YsSy72esJwRIL7XTIq929N/7RmGS5CiNQsnGwt4IZEgLQ+n64 8lP//DOqlo1yAscT/mR1ikn4vQ== X-Google-Smtp-Source: ADFU+vvTztmnU9lpjasiqa10hSBSJgTl3WrD1Gxop2WglkZvO8iSitEjNNwX1ck2A7QnkhR469JcUg== X-Received: by 2002:a37:c43:: with SMTP id 64mr9826539qkm.47.1585535615138; Sun, 29 Mar 2020 19:33:35 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:34 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 04/18] rcu: Rename __is_kfree_rcu_offset() macro Date: Sun, 29 Mar 2020 22:32:34 -0400 Message-Id: <20200330023248.164994-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Rename __is_kfree_rcu_offset to __is_kvfree_rcu_offset. All RCU paths use kvfree() now instead of kfree(), thus rename it. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 6 +++--- kernel/rcu/tiny.c | 2 +- kernel/rcu/tree.c | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 8b7128d0860e2..c6f6a195cb1cd 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -823,16 +823,16 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) /* * Does the specified offset indicate that the corresponding rcu_head - * structure can be handled by kfree_rcu()? + * structure can be handled by kvfree_rcu()? */ -#define __is_kfree_rcu_offset(offset) ((offset) < 4096) +#define __is_kvfree_rcu_offset(offset) ((offset) < 4096) /* * Helper macro for kfree_rcu() to prevent argument-expansion eyestrain. */ #define __kfree_rcu(head, offset) \ do { \ - BUILD_BUG_ON(!__is_kfree_rcu_offset(offset)); \ + BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \ kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \ } while (0) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 3dd8e6e207b09..aa897c3f2e92c 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -85,7 +85,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) unsigned long offset = (unsigned long)head->func; rcu_lock_acquire(&rcu_callback_map); - if (__is_kfree_rcu_offset(offset)) { + if (__is_kvfree_rcu_offset(offset)) { trace_rcu_invoke_kvfree_callback("", head, offset); kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 88b744ce896c0..1209945a34bfd 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2743,7 +2743,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kfree_rcu_offset((unsigned long)func)) + if (__is_kvfree_rcu_offset((unsigned long)func)) trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, rcu_segcblist_n_cbs(&rdp->cblist)); @@ -2937,7 +2937,7 @@ static void kfree_rcu_work(struct work_struct *work) rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); - if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) + if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); From patchwork Mon Mar 30 02:32:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464473 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4AEE11392 for ; Mon, 30 Mar 2020 02:33:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0E4E92078B for ; Mon, 30 Mar 2020 02:33:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="fxT0FW2S" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E4E92078B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BB0366B006E; Sun, 29 Mar 2020 22:33:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B38DC6B0070; Sun, 29 Mar 2020 22:33:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DB016B0071; Sun, 29 Mar 2020 22:33:37 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0051.hostedemail.com [216.40.44.51]) by kanga.kvack.org (Postfix) with ESMTP id 815966B006E for ; Sun, 29 Mar 2020 22:33:37 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 813A4180AD817 for ; Mon, 30 Mar 2020 02:33:37 +0000 (UTC) X-FDA: 76650457674.27.oven40_59cf5e0622b33 X-Spam-Summary: 2,0,0,c7368cd7e166a5d7,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:421:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1605:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:2731:2914:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3873:3874:4118:4321:4385:4390:4395:4605:5007:6261:6653:6742:7514:7576:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13161:13229:13894:14096:14181:14394:14721:21080:21212:21433:21444:21451:21627:21740:30034:30054,0,RBL:209.85.160.195:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: oven40_59cf5e0622b33 X-Filterd-Recvd-Size: 7900 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:37 +0000 (UTC) Received: by mail-qt1-f195.google.com with SMTP id t17so13887442qtn.12 for ; Sun, 29 Mar 2020 19:33:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i7es2W9CEqIB4GMVROiAy2JZJ57uZN4pFanE1V1uKhA=; b=fxT0FW2SkmPY/tTctVOOiNEaCQkEJhYBD9Ljg3DTwuJpXM6Z+pQuU2xjzrWckST7Wt kTky1ILPKzjvatfadFjqzTvUwv+sQZt7v+UsnxOvCkxsgAJran4D5XewUjyU9oT8+Jl+ E6huwFcNFYQ0MuaqQftjvXLOI6/gzgK68TEnc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i7es2W9CEqIB4GMVROiAy2JZJ57uZN4pFanE1V1uKhA=; b=KdpW+XKHkE5Lg2mvSNsta8U+fE5LGhRSj3Em4dQPTlo8lxoY5MhxZMgbEy9YfemrPZ by3pI2+8vgfK9HLTzASnLu3uRJ3KmHUOWoC4ORdt/MC+XOCFFuWpdRKoTaxqK3AFMl3Y wk03V+EchJX4tbc1Vv/A6AhYYQL/1g4e1Irh/SPWGN/a41+rUIJeQI8w08jp/5R1SmbY afOnQam+jhMJch80VUxkqTqchKvdTowAVbBVoVq0QFJidLnjN+XpJJPgHyzSQ+gETl0y btHKJAEcmJay+d5GcnTysUnp5YiUPa/cxh9qc2pPIa1FXRCvH059V3HT66V6OK0qHGzx vakw== X-Gm-Message-State: ANhLgQ1ZF1gixeK67NxAypuASeQz00EMFdyrqkBbCfeP5/Vku61Gb7rP hgNGyJrXClhIsT11tCCR3BIWHQ== X-Google-Smtp-Source: ADFU+vs2ndobSCzogyfpMzEQcEldd3+b0TiV6qhu47wRBO4t65Nlw0iw94d+x/RbDFy/qTpIQuCBmQ== X-Received: by 2002:aed:3c4b:: with SMTP id u11mr9819279qte.208.1585535616115; Sun, 29 Mar 2020 19:33:36 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:35 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 05/18] rcu: Rename kfree_call_rcu() to the kvfree_call_rcu(). Date: Sun, 29 Mar 2020 22:32:35 -0400 Message-Id: <20200330023248.164994-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" The reason is, it is capable of freeing vmalloc() memory now. Do the same with __kfree_rcu() macro, it becomes __kvfree_rcu(), the reason is the same as pointed above. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 8 ++++---- include/linux/rcutiny.h | 2 +- include/linux/rcutree.h | 2 +- kernel/rcu/tree.c | 8 ++++---- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index c6f6a195cb1cd..edb6eeba49f83 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -830,10 +830,10 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) /* * Helper macro for kfree_rcu() to prevent argument-expansion eyestrain. */ -#define __kfree_rcu(head, offset) \ +#define __kvfree_rcu(head, offset) \ do { \ BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \ - kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \ + kvfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \ } while (0) /** @@ -852,7 +852,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * Because the functions are not allowed in the low-order 4096 bytes of * kernel virtual memory, offsets up to 4095 bytes can be accommodated. * If the offset is larger than 4095 bytes, a compile-time error will - * be generated in __kfree_rcu(). If this error is triggered, you can + * be generated in __kvfree_rcu(). If this error is triggered, you can * either fall back to use of call_rcu() or rearrange the structure to * position the rcu_head structure into the first 4096 bytes. * @@ -867,7 +867,7 @@ do { \ typeof (ptr) ___p = (ptr); \ \ if (___p) \ - __kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ + __kvfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ } while (0) /** diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index d77e11186afd1..5ba0bcb231976 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -34,7 +34,7 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -static inline void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) +static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { call_rcu(head, func); } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 45f3f66bb04df..3a7829d69fef8 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -33,7 +33,7 @@ static inline void rcu_virt_note_context_switch(int cpu) } void synchronize_rcu_expedited(void); -void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func); +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func); void rcu_barrier(void); bool rcu_eqs_special_set(int cpu); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 1209945a34bfd..3fb19ea039912 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3082,18 +3082,18 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, } /* - * Queue a request for lazy invocation of kfree_bulk()/kfree() after a grace + * Queue a request for lazy invocation of kfree_bulk()/kvfree() after a grace * period. Please note there are two paths are maintained, one is the main one * that uses kfree_bulk() interface and second one is emergency one, that is * used only when the main path can not be maintained temporary, due to memory * pressure. * - * Each kfree_call_rcu() request is added to a batch. The batch will be drained + * Each kvfree_call_rcu() request is added to a batch. The batch will be drained * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will * be free'd in workqueue context. This allows us to: batch requests together to * reduce the number of grace periods during heavy kfree_rcu() load. */ -void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; @@ -3142,7 +3142,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) spin_unlock(&krcp->lock); local_irq_restore(flags); } -EXPORT_SYMBOL_GPL(kfree_call_rcu); +EXPORT_SYMBOL_GPL(kvfree_call_rcu); static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) From patchwork Mon Mar 30 02:32:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464475 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 646BD13A4 for ; Mon, 30 Mar 2020 02:33:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3214F2078B for ; Mon, 30 Mar 2020 02:33:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="YFyChil3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3214F2078B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6A8296B0070; Sun, 29 Mar 2020 22:33:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 632826B0071; Sun, 29 Mar 2020 22:33:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D3B56B0072; Sun, 29 Mar 2020 22:33:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0069.hostedemail.com [216.40.44.69]) by kanga.kvack.org (Postfix) with ESMTP id 32FD06B0070 for ; Sun, 29 Mar 2020 22:33:38 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3B3BD45AA for ; Mon, 30 Mar 2020 02:33:38 +0000 (UTC) X-FDA: 76650457716.06.shake80_59edc0e83a500 X-Spam-Summary: 2,0,0,da35d940eca32d75,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:69:355:379:541:800:960:965:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3872:3874:4321:4385:4390:4395:4419:4605:5007:6261:6653:6742:7514:7576:7903:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13894:13972:14096:14181:14394:14721:21080:21444:21451:21627:21990:30054,0,RBL:209.85.160.195:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: shake80_59edc0e83a500 X-Filterd-Recvd-Size: 5205 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:37 +0000 (UTC) Received: by mail-qt1-f195.google.com with SMTP id m33so13931999qtb.3 for ; Sun, 29 Mar 2020 19:33:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y9juYIZdTD4S3/RhfF4ZU+wi+6D+Bwupgtb3fIEkI74=; b=YFyChil3lVYm7zhnne5lGhwbXXlq2EDvxsIxZFa2dtN2ItcId/sHvPqThGAh1BwFZl YWaCpA7Lt+GVmGROOWIIkfAFa//fAZAdJmSr8bHVwVDtlP8Ncb6wNX3k7OmivGSl7CnM qCJmFAVk90alncCE3ob/Cy9tUBPO6k+bzjjzU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y9juYIZdTD4S3/RhfF4ZU+wi+6D+Bwupgtb3fIEkI74=; b=eDrCAMYPpTnVmCNglkdyM+665rD4Wrdgnc4YzEjEmmxBOkXgnlbxoaJXJ2kRgyeheo Ym3AdwyYd64Qt+sruHz9njuT5Z9YLkAmUFYOrh+k23RwlDRz0m8s7QliHVP6Y22JCgNi JDAuz7bAkCvhdp1m0sNJTJ9c7ZMbkjxekFoiGS0owg2UCVcMXEEqIIS2GyuW9mqcmjI7 EsZL2icqzT4GabWHWXgr/YTCOsr5HNB3zfO04JviYfeSTE8UBhqXnKAkEw6war1D9gU7 1dmteDpxSjIaeiK33HAddKUYdQd1jUBANCsWJuZGd6BWp/FkYbFAIahAf+D0Iwqi/xam FvBQ== X-Gm-Message-State: ANhLgQ1Iy/iLdHawxq9jDQ8lJ5bxBeOs7S7VMxLfWTVg8zCya3TlW9Tp HGv42JdHixC8Qbfb76vsNwv44g== X-Google-Smtp-Source: ADFU+vscGt7vUJpfCtGgJZKczUSOnNkYu/OpdYETeb8x0PKQDoRTASn4p6B5qpZSy87hhDabIXEibQ== X-Received: by 2002:ac8:6d19:: with SMTP id o25mr10223186qtt.303.1585535617018; Sun, 29 Mar 2020 19:33:37 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:36 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 06/18] mm/list_lru.c: Remove kvfree_rcu_local() function Date: Sun, 29 Mar 2020 22:32:36 -0400 Message-Id: <20200330023248.164994-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Since there is newly introduced kvfree_rcu() API, there is no need in queuing and using call_rcu() to kvfree() an object after the GP. Remove kvfree_rcu_local() function and replace call_rcu() by new kvfree_rcu() API that does the same but in more efficient way. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- mm/list_lru.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 386424688f805..69becdb224080 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -12,6 +12,7 @@ #include #include #include +#include #include "slab.h" #ifdef CONFIG_MEMCG_KMEM @@ -383,21 +384,13 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru) struct list_lru_memcg *memcg_lrus; /* * This is called when shrinker has already been unregistered, - * and nobody can use it. So, there is no need to use kvfree_rcu_local(). + * and nobody can use it. So, there is no need to use kvfree_rcu(). */ memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true); __memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids); kvfree(memcg_lrus); } -static void kvfree_rcu_local(struct rcu_head *head) -{ - struct list_lru_memcg *mlru; - - mlru = container_of(head, struct list_lru_memcg, rcu); - kvfree(mlru); -} - static int memcg_update_list_lru_node(struct list_lru_node *nlru, int old_size, int new_size) { @@ -429,7 +422,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru, rcu_assign_pointer(nlru->memcg_lrus, new); spin_unlock_irq(&nlru->lock); - call_rcu(&old->rcu, kvfree_rcu_local); + kvfree_rcu(old, rcu); return 0; } From patchwork Mon Mar 30 02:32:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464477 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 91EC713A4 for ; Mon, 30 Mar 2020 02:33:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4881920735 for ; Mon, 30 Mar 2020 02:33:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="n8ZFtlSW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4881920735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 792106B0071; Sun, 29 Mar 2020 22:33:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 71D286B0072; Sun, 29 Mar 2020 22:33:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56E536B0073; Sun, 29 Mar 2020 22:33:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 3FB536B0071 for ; Sun, 29 Mar 2020 22:33:39 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 50AA38248D7C for ; Mon, 30 Mar 2020 02:33:39 +0000 (UTC) X-FDA: 76650457758.11.dog83_5a14697d57043 X-Spam-Summary: 2,0,0,f0b063de3db24f1c,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2731:2907:3138:3139:3140:3141:3142:3355:3865:3866:3867:3870:3871:3872:3874:4118:4250:4385:5007:6117:6119:6261:6653:6742:7875:8660:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12683:12895:12986:13148:13149:13230:13894:14110:14394:14721:21080:21444:21451:21627:21740:21789:21987:21990:30054,0,RBL:209.85.222.195:@joelfernandes.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: dog83_5a14697d57043 X-Filterd-Recvd-Size: 7069 Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:38 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id b62so17560552qkf.6 for ; Sun, 29 Mar 2020 19:33:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+SC1wb04vPFRHvA69idHFJ/D6TQHtUQk5KfrhqDEopc=; b=n8ZFtlSWpZkaejINrNFAgQe81mPm1qWw92alN9sRCcqMfUbbD+oO2me5NN9T23SnJB OGq7zOh2DqT7DRspsD/Mnk5C1Al6o2RjKHcfVq86GVhWCuMmRlVru+fvyVpWfYwn5FAV qQNh0nYHY0ZBoY2rruGIs8jCY5RCLAvmcz604= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+SC1wb04vPFRHvA69idHFJ/D6TQHtUQk5KfrhqDEopc=; b=CNoPpEOHPft7yprKKekkMBL2eihriJtxWV0ADmnorK8g1gqYPh/e8RdkWKGRLFF/L3 Sle/G40SMAqaPIQQDnUYajdW4RGU99l9t17HkIf/KGNsuA+9twOa+yLK/TXEFcj8GriZ HEocqXzVhiGbqCyuSay64YNAl3WdCevHnWIHs8KVeb+UYJcPHnZBkjsG2r2W1QZ5y5/F aEzo4+aR9ur3Pm+NS6A9/28gUw1m6y9v9qsS4KVJfzkwp/Y+0xUnMqWnfGpxSWeG/bp4 6FuxOP+Ot7LWkr+3u2j10i0mfSdsTY/qUYBvU/fz3h0sxE05K7Hs75qdRs64Vpf2G7Ek DQWA== X-Gm-Message-State: ANhLgQ1oLf0HILHUTaqAFKApBFrBUR1jKTQbT3YE2Yslen0QQaoI5hwC 4JXKIqxNcpt6YeUrhPFmKQ+Q5Q== X-Google-Smtp-Source: ADFU+vszNuc31kIyM2RoqVUpLdVvAYIJ6JetNU3ZOQhTJ5Je0FGvESstvuE4ZqfTJltehBQY4GpxHg== X-Received: by 2002:a37:db0a:: with SMTP id e10mr9787278qki.273.1585535618038; Sun, 29 Mar 2020 19:33:38 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:37 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt , "Uladzislau Rezki (Sony)" Subject: [PATCH 07/18] rcu/tree: Simplify debug_objects handling Date: Sun, 29 Mar 2020 22:32:37 -0400 Message-Id: <20200330023248.164994-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to prepare for future changes for headless RCU support, make the debug_objects handling in kfree_rcu use the final 'pointer' value of the object, instead of depending on the head. Signed-off-by: Joel Fernandes (Google) Reported-by: kbuild test robot --- kernel/rcu/tree.c | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 3fb19ea039912..95d1f5e20d5ec 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2825,7 +2825,6 @@ struct kfree_rcu_bulk_data { unsigned long nr_records; void *records[KFREE_BULK_MAX_ENTR]; struct kfree_rcu_bulk_data *next; - struct rcu_head *head_free_debug; }; /** @@ -2875,11 +2874,11 @@ struct kfree_rcu_cpu { static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc); static __always_inline void -debug_rcu_head_unqueue_bulk(struct rcu_head *head) +debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) { #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD - for (; head; head = head->next) - debug_rcu_head_unqueue(head); + for (int i = 0; i < bhead->nr_records; i++) + debug_rcu_head_unqueue((struct rcu_head *)(bhead->records[i])); #endif } @@ -2909,7 +2908,7 @@ static void kfree_rcu_work(struct work_struct *work) for (; bhead; bhead = bnext) { bnext = bhead->next; - debug_rcu_head_unqueue_bulk(bhead->head_free_debug); + debug_rcu_bhead_unqueue(bhead); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kfree_bulk_callback(rcu_state.name, @@ -2931,14 +2930,15 @@ static void kfree_rcu_work(struct work_struct *work) */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; + void *ptr = (void *)head - offset; next = head->next; - debug_rcu_head_unqueue(head); + debug_rcu_head_unqueue((struct rcu_head *)ptr); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) - kvfree((void *)head - offset); + kvfree(ptr); rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3062,18 +3062,11 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Initialize the new block. */ bnode->nr_records = 0; bnode->next = krcp->bhead; - bnode->head_free_debug = NULL; /* Attach it to the head. */ krcp->bhead = bnode; } -#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD - head->func = func; - head->next = krcp->bhead->head_free_debug; - krcp->bhead->head_free_debug = head; -#endif - /* Finally insert. */ krcp->bhead->records[krcp->bhead->nr_records++] = (void *) head - (unsigned long) func; @@ -3097,14 +3090,17 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; + void *ptr; local_irq_save(flags); // For safely calling this_cpu_ptr(). krcp = this_cpu_ptr(&krc); if (krcp->initialized) spin_lock(&krcp->lock); + ptr = (void *)head - (unsigned long)func; + // Queue the object but don't yet schedule the batch. - if (debug_rcu_head_queue(head)) { + if (debug_rcu_head_queue(ptr)) { // Probable double kfree_rcu(), just leak. WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", __func__, head); @@ -3121,8 +3117,8 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (is_vmalloc_addr((void *) head - (unsigned long) func) || - !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) { + if (is_vmalloc_addr(ptr) || + !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) { head->func = func; head->next = krcp->head; krcp->head = head; From patchwork Mon Mar 30 02:32:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464479 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 98A641392 for ; Mon, 30 Mar 2020 02:33:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5B5892078B for ; Mon, 30 Mar 2020 02:33:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="eAqEJ2hy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5B5892078B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F49E6B0072; Sun, 29 Mar 2020 22:33:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A7B76B0073; Sun, 29 Mar 2020 22:33:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51F176B0074; Sun, 29 Mar 2020 22:33:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0224.hostedemail.com [216.40.44.224]) by kanga.kvack.org (Postfix) with ESMTP id 3A0966B0072 for ; Sun, 29 Mar 2020 22:33:40 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 474834DB3 for ; Mon, 30 Mar 2020 02:33:40 +0000 (UTC) X-FDA: 76650457800.30.boys02_5a3841d45f743 X-Spam-Summary: 2,0,0,5fcc62a14da95d71,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2914:3138:3139:3140:3141:3142:3352:3865:3868:3870:3872:4321:4385:5007:6261:6653:6742:10004:11026:11658:11914:12043:12296:12297:12517:12519:12555:12895:13069:13311:13357:13894:14096:14384:14394:14721:21080:21444:21627:21740:30054,0,RBL:209.85.222.193:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: boys02_5a3841d45f743 X-Filterd-Recvd-Size: 4223 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:39 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id d11so17543938qko.3 for ; Sun, 29 Mar 2020 19:33:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VhRXFX915ZZjf6qHI5PrZD8T8+tPOI/Cg2isptxenk4=; b=eAqEJ2hyTMW80nl2AFCJp8deCuE81RBBTLR7Vl6BziuMKTXwko1qIyEOcelWccQQkl O5/4QbRDcxFSa30uXFamCDL0znuq14BuHKIBCF9g0pIf/GCsLfXA1UF2HikN7r+Yka5t f1hMLlRKNr/Ngcmd9WJd2kkyL8wzZaz0204+w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VhRXFX915ZZjf6qHI5PrZD8T8+tPOI/Cg2isptxenk4=; b=mJOXlwxvKT/wc41QgcTw+6LDZzN54+UTzHzNfLwRpdq1l+u2e+MqPlKyuWAUmWaYgz 6z9QrJ/dQAKgGa7Zw9A8A2d945xvEryDN5FCezX5vmTTDT3Hpx0xF/grXjbpBhmErNdD 93+Q3oi6FEdG02stzl6OQ4Qykk+MsH46xk49cVfReaR2SsXXd70A4NmJVA8j0n/+AznC nWvTUV1FDo+r/wdI6xkXG4Eq/Gb9LDYi/zKx/iK/n0f2N8Ju92Morjf86YKg+4d4cB8W 7JeQh3pC0vRNb/7TVv89TbS9dRKYavaFOFF//umE3Z4Sq6LqsqXOf8Xv+WRyyRcnnBzQ 9Jmg== X-Gm-Message-State: ANhLgQ1lNeTk7ghuOyBq0y30KRRo1V8jsihielvNkmu8aUwLoRBDnqzY 2wxRav8tDx02nry+86+GCmLEQw== X-Google-Smtp-Source: ADFU+vvqvHDIo+I+fVPksGXLy4Vwf61hpGLUBlvFZuL8EfaZx88zhM/5jOyQn340v+Wi0oZQiZ1hBw== X-Received: by 2002:a37:87c7:: with SMTP id j190mr9957429qkd.66.1585535619052; Sun, 29 Mar 2020 19:33:39 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:38 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt , "Uladzislau Rezki (Sony)" Subject: [PATCH 08/18] rcu/tree: Clarify emergency path comment better Date: Sun, 29 Mar 2020 22:32:38 -0400 Message-Id: <20200330023248.164994-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Clarify emergency path comment better in kfree_rcu(). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 95d1f5e20d5ec..8dfa4b32e4d00 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2924,9 +2924,10 @@ static void kfree_rcu_work(struct work_struct *work) } /* - * vmalloc() pointers end up here also emergency case. It can - * happen under low memory condition when an allocation gets - * failed, so the "bulk" path can not be temporary maintained. + * We can end up here either with 1) vmalloc() pointers or 2) were low + * on memory and could not allocate a bulk array. It can happen under + * low memory condition when an allocation gets failed, so the "bulk" + * path can not be temporarly used. */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; From patchwork Mon Mar 30 02:32:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464481 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0D5A13A4 for ; Mon, 30 Mar 2020 02:33:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7DC4C20735 for ; Mon, 30 Mar 2020 02:33:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="vate42J2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7DC4C20735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 65CA16B0073; Sun, 29 Mar 2020 22:33:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5E26B6B0074; Sun, 29 Mar 2020 22:33:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4837A6B0075; Sun, 29 Mar 2020 22:33:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0074.hostedemail.com [216.40.44.74]) by kanga.kvack.org (Postfix) with ESMTP id 2BAD66B0073 for ; Sun, 29 Mar 2020 22:33:41 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 32220181AD0A1 for ; Mon, 30 Mar 2020 02:33:41 +0000 (UTC) X-FDA: 76650457842.04.soda59_5a5d1cdd2c004 X-Spam-Summary: 10,1,0,7152cf1d00c872b1,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:404:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2907:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:4390:4395:5007:6261:6653:6742:7514:7576:7875:8603:9592:10004:11026:11473:11658:11914:12043:12297:12517:12519:12555:12895:12986:13069:13311:13357:13894:14181:14384:14394:14721:21080:21444:21451:21627:21740:30034:30054,0,RBL:209.85.222.193:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: soda59_5a5d1cdd2c004 X-Filterd-Recvd-Size: 5004 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:40 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id x3so17537394qki.4 for ; Sun, 29 Mar 2020 19:33:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1STg3B4NFmryGFsVQ39Fqcyyw2zeIxUup+slIF9OxYI=; b=vate42J2T1d4IR+p4SKAXzO/ZNqN9vu8zAMaADAJJq3jsp+Q3qXkRYHGB9gcbDfR5X yby/F2W/Z8EsSL7vC7Y/dMjfPVzClbZ7BbzFi3Jo2sSZZhn4N2rwuGJ3JSjqGl3MUieF p1dapi543VOjfTBXmPigUXE+aC4dPwbPpTuwU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1STg3B4NFmryGFsVQ39Fqcyyw2zeIxUup+slIF9OxYI=; b=F0tTKS2bkMtZ9U/xYDx72CmObvJQ3bl7EjyfYbkJLuM0Bbmi0y6u4UncjdagxTd3Xp WhKLIIWmk551C4jYwDJFXaGDfK3K7nDiWdU01ImIANvLAaFv66GnI9UtInp8tYbw/89s YsAmT6Xsis966AOV7GV+IZJ/NwnhNkOvv698VwX7CcnA//qH9alF0dvSWFWUhmvSq+CL e48vquvA1amDkGDkgyxQUz4EIrnVJJCeOtv9SKTLYfhaIIPMQ6lfcnJCGWkCnaQZy2W0 Z4PiRZZ0tmW+Lyg1zZbvo9MHv3I0PQ4HDH5hO4QgLwaKcUYvFlWShXhfWFvUZxYaPpYJ bBYg== X-Gm-Message-State: ANhLgQ3Jfv9+fY4QYEHDvF61TAxypKhMInVkZh2WSTMA+VslzYA4ecxT L6oE67ga6OFohXLVVyaegDUUHg== X-Google-Smtp-Source: ADFU+vsb7u7onAWM6SNZg0kNXedSUyJKJP7iWxDcLoGW/pjia1x5QLz+6JZM6GIVlbH8uur2r+xlOA== X-Received: by 2002:a37:6357:: with SMTP id x84mr10162791qkb.490.1585535619962; Sun, 29 Mar 2020 19:33:39 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:39 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Boqun Feng , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 09/18] rcu/tree: Simplify KFREE_BULK_MAX_ENTR macro Date: Sun, 29 Mar 2020 22:32:39 -0400 Message-Id: <20200330023248.164994-10-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" We can simplify KFREE_BULK_MAX_ENTR macro and get rid of magic numbers which were used to make the structure to be exactly one page. Suggested-by: Boqun Feng Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8dfa4b32e4d00..cfe456e68c644 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2807,13 +2807,6 @@ EXPORT_SYMBOL_GPL(call_rcu); #define KFREE_DRAIN_JIFFIES (HZ / 50) #define KFREE_N_BATCHES 2 -/* - * This macro defines how many entries the "records" array - * will contain. It is based on the fact that the size of - * kfree_rcu_bulk_data structure becomes exactly one page. - */ -#define KFREE_BULK_MAX_ENTR ((PAGE_SIZE / sizeof(void *)) - 3) - /** * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers * @nr_records: Number of active pointers in the array @@ -2827,6 +2820,14 @@ struct kfree_rcu_bulk_data { struct kfree_rcu_bulk_data *next; }; +/* + * This macro defines how many entries the "records" array + * will contain. It is based on the fact that the size of + * kfree_rcu_bulk_data structure becomes exactly one page. + */ +#define KFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *)) + /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period From patchwork Mon Mar 30 02:32:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464483 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 21A1713A4 for ; Mon, 30 Mar 2020 02:33:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C6B4220735 for ; Mon, 30 Mar 2020 02:33:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="DDiCsr4b" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C6B4220735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A9E656B0074; Sun, 29 Mar 2020 22:33:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A28DA6B0075; Sun, 29 Mar 2020 22:33:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 852DF6B0078; Sun, 29 Mar 2020 22:33:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0088.hostedemail.com [216.40.44.88]) by kanga.kvack.org (Postfix) with ESMTP id 69EA26B0074 for ; Sun, 29 Mar 2020 22:33:42 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 692AB8248D7C for ; Mon, 30 Mar 2020 02:33:42 +0000 (UTC) X-FDA: 76650457884.10.order61_5a86e2e8d962b X-Spam-Summary: 2,0,0,808fb5177e6df9ab,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:4:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2638:2693:2907:2914:2918:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:4385:5007:6117:6119:6261:6653:6742:7514:7576:7875:7903:8603:9010:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13141:13161:13229:13230:13868:13894:14096:14394:21080:21324:21433:21444:21451:21627:21740:21789:21987:21990:30034:30045:30054:30070:30091,0,RBL:209.85.219.66:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: order61_5a86e2e8d962b X-Filterd-Recvd-Size: 15403 Received: from mail-qv1-f66.google.com (mail-qv1-f66.google.com [209.85.219.66]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:41 +0000 (UTC) Received: by mail-qv1-f66.google.com with SMTP id c28so8180643qvb.10 for ; Sun, 29 Mar 2020 19:33:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9O4vayA2XFJge9v6ySy8FCcWCe4sfIKtielmms+HdCM=; b=DDiCsr4b1kQ1XOVbiJHuWE2tSr7GVhfYZ3CVtapaOGyv93A54AwPAbKXIE+Zhjk1ck jBLtStB+C08LRv3D1lnqZAY9ylWA8GQs38UiehGUwB+iDui3NcWIkJEHSNHvUzxPdrBv yUWTRTijD9XGOmKJBTuPRNKmaltJ+0cVfcpiU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9O4vayA2XFJge9v6ySy8FCcWCe4sfIKtielmms+HdCM=; b=K11wEPDkfDjT4N8KVlL1Unt0Y9hODGKffNJIvUr7/kWiu9w4qU7soifRF5JeD9Bhum p0SojIk6pZp1pKeXmzuCOpNpBPVp/F4EZHZ3Tdhw+nT+l9XITPdsAlaOxpm06/4V7qAz aHnZpHVDM2uTJ6cg1oXscwZ/lZWmyyxFhFaHrYWbb/vCjt2f2N8s5A4/lO4Xo+CWviwx 8XkBFUKtmZrhY4UgV7CkTVR5sMDcEstpp8kLiuoleaBLPijvO/Nx1Kes7+EoT8c37qOD tT9DUF8zIkMi2pTARK0DYcFMuFH1oRrJPL1aug1xymtxm7UEzKmNNoKyV59KWKyCmDCJ FqQQ== X-Gm-Message-State: ANhLgQ0PC2VtDclryvXguXhGSER6TSVOCvvMlapmYel1Ed+EvEYLwfrN R+G51EjKVSHLwn9WZh1AccPUxaE9wI8= X-Google-Smtp-Source: ADFU+vtvCwVuLdvn7Q6HnMKU6Xbgf5np5t/oJIYTqKIBRTbWBG/d57XfI2taLG0QaD7vekSJxMljEQ== X-Received: by 2002:a05:6214:294:: with SMTP id l20mr9926511qvv.46.1585535621011; Sun, 29 Mar 2020 19:33:41 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:40 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs Date: Sun, 29 Mar 2020 22:32:40 -0400 Message-Id: <20200330023248.164994-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" To do so we use an array of common kvfree_rcu_bulk_data structure. It consists of two elements, index number 0 corresponds to SLAB ptrs., whereas vmalloc pointers can be accessed by using index number 1. The reason of not mixing pointers is to have an easy way to to distinguish them. It is also the preparation patch for head-less objects support. When an object is head-less we can not queue it into any list, instead a pointer is placed directly into an array. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) Reported-by: kbuild test robot Reported-by: kbuild test robot --- kernel/rcu/tree.c | 179 ++++++++++++++++++++++++++++------------------ 1 file changed, 108 insertions(+), 71 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index cfe456e68c644..8fbc8450284db 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2808,38 +2808,36 @@ EXPORT_SYMBOL_GPL(call_rcu); #define KFREE_N_BATCHES 2 /** - * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers + * struct kvfree_rcu_bulk_data - single block to store kvfree() pointers * @nr_records: Number of active pointers in the array - * @records: Array of the kfree_rcu() pointers * @next: Next bulk object in the block chain - * @head_free_debug: For debug, when CONFIG_DEBUG_OBJECTS_RCU_HEAD is set + * @records: Array of the SLAB pointers */ -struct kfree_rcu_bulk_data { +struct kvfree_rcu_bulk_data { unsigned long nr_records; - void *records[KFREE_BULK_MAX_ENTR]; - struct kfree_rcu_bulk_data *next; + struct kvfree_rcu_bulk_data *next; + void *records[]; }; /* * This macro defines how many entries the "records" array * will contain. It is based on the fact that the size of - * kfree_rcu_bulk_data structure becomes exactly one page. + * kvfree_rcu_bulk_data become exactly one page. */ -#define KFREE_BULK_MAX_ENTR \ - ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *)) +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period * @head_free: List of kfree_rcu() objects waiting for a grace period - * @bhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period + * @bkvhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period * @krcp: Pointer to @kfree_rcu_cpu structure */ - struct kfree_rcu_cpu_work { struct rcu_work rcu_work; struct rcu_head *head_free; - struct kfree_rcu_bulk_data *bhead_free; + struct kvfree_rcu_bulk_data *bkvhead_free[2]; struct kfree_rcu_cpu *krcp; }; @@ -2861,8 +2859,9 @@ struct kfree_rcu_cpu_work { */ struct kfree_rcu_cpu { struct rcu_head *head; - struct kfree_rcu_bulk_data *bhead; - struct kfree_rcu_bulk_data *bcached; + struct kvfree_rcu_bulk_data *bkvhead[2]; + struct kvfree_rcu_bulk_data *bkvcache[2]; + struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; spinlock_t lock; struct delayed_work monitor_work; @@ -2875,7 +2874,7 @@ struct kfree_rcu_cpu { static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc); static __always_inline void -debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) +debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead) { #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD for (int i = 0; i < bhead->nr_records; i++) @@ -2890,45 +2889,77 @@ debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) static void kfree_rcu_work(struct work_struct *work) { unsigned long flags; + struct kvfree_rcu_bulk_data *bkhead, *bknext; + struct kvfree_rcu_bulk_data *bvhead, *bvnext; struct rcu_head *head, *next; - struct kfree_rcu_bulk_data *bhead, *bnext; struct kfree_rcu_cpu *krcp; struct kfree_rcu_cpu_work *krwp; + int i; krwp = container_of(to_rcu_work(work), - struct kfree_rcu_cpu_work, rcu_work); + struct kfree_rcu_cpu_work, rcu_work); + krcp = krwp->krcp; spin_lock_irqsave(&krcp->lock, flags); + /* Channel 1. */ + bkhead = krwp->bkvhead_free[0]; + krwp->bkvhead_free[0] = NULL; + + /* Channel 2. */ + bvhead = krwp->bkvhead_free[1]; + krwp->bkvhead_free[1] = NULL; + + /* Channel 3. */ head = krwp->head_free; krwp->head_free = NULL; - bhead = krwp->bhead_free; - krwp->bhead_free = NULL; spin_unlock_irqrestore(&krcp->lock, flags); - /* "bhead" is now private, so traverse locklessly. */ - for (; bhead; bhead = bnext) { - bnext = bhead->next; + /* kmalloc()/kfree() channel. */ + for (; bkhead; bkhead = bknext) { + bknext = bkhead->next; - debug_rcu_bhead_unqueue(bhead); + debug_rcu_bhead_unqueue(bkhead); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kfree_bulk_callback(rcu_state.name, - bhead->nr_records, bhead->records); + bkhead->nr_records, bkhead->records); + + kfree_bulk(bkhead->nr_records, bkhead->records); + rcu_lock_release(&rcu_callback_map); + + if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead)) + free_page((unsigned long) bkhead); + + cond_resched_tasks_rcu_qs(); + } + + /* vmalloc()/vfree() channel. */ + for (; bvhead; bvhead = bvnext) { + bvnext = bvhead->next; + + debug_rcu_bhead_unqueue(bvhead); - kfree_bulk(bhead->nr_records, bhead->records); + rcu_lock_acquire(&rcu_callback_map); + for (i = 0; i < bvhead->nr_records; i++) { + trace_rcu_invoke_kvfree_callback(rcu_state.name, + (struct rcu_head *) bvhead->records[i], 0); + vfree(bvhead->records[i]); + } rcu_lock_release(&rcu_callback_map); - if (cmpxchg(&krcp->bcached, NULL, bhead)) - free_page((unsigned long) bhead); + if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead)) + free_page((unsigned long) bvhead); cond_resched_tasks_rcu_qs(); } /* - * We can end up here either with 1) vmalloc() pointers or 2) were low - * on memory and could not allocate a bulk array. It can happen under - * low memory condition when an allocation gets failed, so the "bulk" - * path can not be temporarly used. + * This path covers emergency case only due to high + * memory pressure also means low memory condition, + * when we could not allocate a bulk array. + * + * Under that condition an object is queued to the + * list instead. */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; @@ -2965,21 +2996,34 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) krwp = &(krcp->krw_arr[i]); /* - * Try to detach bhead or head and attach it over any + * Try to detach bkvhead or head and attach it over any * available corresponding free channel. It can be that * a previous RCU batch is in progress, it means that * immediately to queue another one is not possible so * return false to tell caller to retry. */ - if ((krcp->bhead && !krwp->bhead_free) || + if ((krcp->bkvhead[0] && !krwp->bkvhead_free[0]) || + (krcp->bkvhead[1] && !krwp->bkvhead_free[1]) || (krcp->head && !krwp->head_free)) { - /* Channel 1. */ - if (!krwp->bhead_free) { - krwp->bhead_free = krcp->bhead; - krcp->bhead = NULL; + /* + * Channel 1 corresponds to SLAB ptrs. + */ + if (!krwp->bkvhead_free[0]) { + krwp->bkvhead_free[0] = krcp->bkvhead[0]; + krcp->bkvhead[0] = NULL; + } + + /* + * Channel 2 corresponds to vmalloc ptrs. + */ + if (!krwp->bkvhead_free[1]) { + krwp->bkvhead_free[1] = krcp->bkvhead[1]; + krcp->bkvhead[1] = NULL; } - /* Channel 2. */ + /* + * Channel 3 corresponds to emergency path. + */ if (!krwp->head_free) { krwp->head_free = krcp->head; krcp->head = NULL; @@ -2988,10 +3032,11 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) WRITE_ONCE(krcp->count, 0); /* - * One work is per one batch, so there are two "free channels", - * "bhead_free" and "head_free" the batch can handle. It can be - * that the work is in the pending state when two channels have - * been detached following each other, one by one. + * One work is per one batch, so there are three + * "free channels", the batch can handle. It can + * be that the work is in the pending state when + * channels have been detached following by each + * other. */ queue_rcu_work(system_wq, &krwp->rcu_work); queued = true; @@ -3036,26 +3081,25 @@ static void kfree_rcu_monitor(struct work_struct *work) } static inline bool -kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, - struct rcu_head *head, rcu_callback_t func) +kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) { - struct kfree_rcu_bulk_data *bnode; + struct kvfree_rcu_bulk_data *bnode; + int idx; if (unlikely(!krcp->initialized)) return false; lockdep_assert_held(&krcp->lock); + idx = !is_vmalloc_addr(ptr) ? 0:1; /* Check if a new block is required. */ - if (!krcp->bhead || - krcp->bhead->nr_records == KFREE_BULK_MAX_ENTR) { - bnode = xchg(&krcp->bcached, NULL); - if (!bnode) { - WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE); - - bnode = (struct kfree_rcu_bulk_data *) + if (!krcp->bkvhead[idx] || + krcp->bkvhead[idx]->nr_records == + KVFREE_BULK_MAX_ENTR) { + bnode = xchg(&krcp->bkvcache[idx], NULL); + if (!bnode) + bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); - } /* Switch to emergency path. */ if (unlikely(!bnode)) @@ -3063,30 +3107,30 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Initialize the new block. */ bnode->nr_records = 0; - bnode->next = krcp->bhead; + bnode->next = krcp->bkvhead[idx]; /* Attach it to the head. */ - krcp->bhead = bnode; + krcp->bkvhead[idx] = bnode; } /* Finally insert. */ - krcp->bhead->records[krcp->bhead->nr_records++] = - (void *) head - (unsigned long) func; + krcp->bkvhead[idx]->records + [krcp->bkvhead[idx]->nr_records++] = ptr; return true; } /* - * Queue a request for lazy invocation of kfree_bulk()/kvfree() after a grace - * period. Please note there are two paths are maintained, one is the main one - * that uses kfree_bulk() interface and second one is emergency one, that is - * used only when the main path can not be maintained temporary, due to memory - * pressure. + * Queue a request for lazy invocation of appropriate free routine after a + * grace period. Please note there are three paths are maintained, two are the + * main ones that use array of pointers interface and third one is emergency + * one, that is used only when the main path can not be maintained temporary, + * due to memory pressure. * * Each kvfree_call_rcu() request is added to a batch. The batch will be drained * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will * be free'd in workqueue context. This allows us to: batch requests together to - * reduce the number of grace periods during heavy kfree_rcu() load. + * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { @@ -3110,17 +3154,10 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) } /* - * We do not queue vmalloc pointers into array, - * instead they are just queued to the list. We - * do it because of: - * a) to distinguish kmalloc()/vmalloc() ptrs; - * b) there is no vmalloc_bulk() interface. - * * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (is_vmalloc_addr(ptr) || - !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) { + if (!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr)) { head->func = func; head->next = krcp->head; krcp->head = head; From patchwork Mon Mar 30 02:32:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464485 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B13461392 for ; Mon, 30 Mar 2020 02:34:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 74F4920A8B for ; Mon, 30 Mar 2020 02:34:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="DeRVgPSq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74F4920A8B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E4A2E6B0075; Sun, 29 Mar 2020 22:33:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D84A76B0078; Sun, 29 Mar 2020 22:33:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B88436B007B; Sun, 29 Mar 2020 22:33:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0053.hostedemail.com [216.40.44.53]) by kanga.kvack.org (Postfix) with ESMTP id A08866B0075 for ; Sun, 29 Mar 2020 22:33:43 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B0FA340D6 for ; Mon, 30 Mar 2020 02:33:43 +0000 (UTC) X-FDA: 76650457926.18.scale35_5aba6cedc2059 X-Spam-Summary: 2,0,0,ab7023bcee8c7500,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2693:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:5007:6261:6653:6742:7514:7576:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13161:13229:13894:14096:14181:14394:14721:21080:21433:21444:21451:21627:21740:30034:30054,0,RBL:209.85.222.195:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: scale35_5aba6cedc2059 X-Filterd-Recvd-Size: 5926 Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:43 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id o10so17481387qki.10 for ; Sun, 29 Mar 2020 19:33:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TgVjjqxhrtlCFDIsHkuC1uTJ/LkUZOMYLnLhSTfKyFo=; b=DeRVgPSq3h4ir1leXjxInBG2h8bq9qGUaf0q/6P5vVer49r4RyYdVmEqgmRvSGQgiR Szt5lyIL+P4fZAYle1WQ6mbTytS2I6VlQ9plfFUPqx/C6Z/OXRGykLv5/yyq2ciXUrK3 uQEVTGkaN4TtsK0qXWZ7uhAXhcNSwwyMN8s6o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TgVjjqxhrtlCFDIsHkuC1uTJ/LkUZOMYLnLhSTfKyFo=; b=JW6E8mDV1rWkaug11/C66k7iTZQWRW9yYsZvKugASfcyAzH3wuftKz0OK4hT8rBZpI wmCiXm57pU8we1YJWUgXvIpVXXzQYERM1TDpYDC4IgKiHIVJYdm2+EePEcHZPobt+LZY /8pjvXaa4c24omq67Lnsjln6j6N0sIBnjyAQfCyFLFOimJrfTbfPJYLV8dxDqkZ1JtLn VHnnoeVeQuIxgTSZTeTxqLCoor+yK1fGIbKUr9DrxfSCzMFNjf2jCzRDAeU741ZdJTAw 0WDOZOHXX0xzp/HYn52I/3S2/r2cHZG+BAQ6dgCQZXIpZGmw+TmBATgMawo5LmRe11bT 8dPA== X-Gm-Message-State: ANhLgQ3m4wWMbXt/eVgQpi/La3OuBzimLDfVVV8EVCadDq+plJhuPRGJ +nOuSMC8m2dNyeI4MRWFqVdXZ88c1Zg= X-Google-Smtp-Source: ADFU+vunMDIJ4H3Pq+Krsea4u9kPIh06c+HxsJd5C4qvfZCMXYJheHm4aD9CNDZFXtrDOkjDRGKzpg== X-Received: by 2002:a05:620a:1189:: with SMTP id b9mr7758077qkk.236.1585535622481; Sun, 29 Mar 2020 19:33:42 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:41 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 11/18] rcu/tree: Introduce expedited_drain flag Date: Sun, 29 Mar 2020 22:32:41 -0400 Message-Id: <20200330023248.164994-12-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" It is used and set to true when the bulk array can not be maintained, it happens under low memory condition and memory pressure. In that case the drain work is scheduled right away and not after KFREE_DRAIN_JIFFIES. It tends to speed up the reclaim path. On the other hand, there is no data showing the difference yet. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8fbc8450284db..3b94526f490cb 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3128,14 +3128,16 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) * due to memory pressure. * * Each kvfree_call_rcu() request is added to a batch. The batch will be drained - * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will - * be free'd in workqueue context. This allows us to: batch requests together to - * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. + * every KFREE_DRAIN_JIFFIES number of jiffies or can be scheduled right away if + * a low memory is detected. All the objects in the batch will be free'd in + * workqueue context. This allows us to: batch requests together to reduce the + * number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; + bool expedited_drain = false; void *ptr; local_irq_save(flags); // For safely calling this_cpu_ptr(). @@ -3161,6 +3163,14 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) head->func = func; head->next = krcp->head; krcp->head = head; + + /* + * There was an issue to place the pointer directly + * into array, due to memory pressure. Initiate an + * expedited drain to accelerate lazy invocation of + * appropriate free calls. + */ + expedited_drain = true; } WRITE_ONCE(krcp->count, krcp->count + 1); @@ -3169,7 +3179,9 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING && !krcp->monitor_todo) { krcp->monitor_todo = true; - schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES); + + schedule_delayed_work(&krcp->monitor_work, + expedited_drain ? 0 : KFREE_DRAIN_JIFFIES); } unlock_return: From patchwork Mon Mar 30 02:32:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464487 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 395C71392 for ; Mon, 30 Mar 2020 02:34:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EE4772078B for ; Mon, 30 Mar 2020 02:34:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="dequhaQu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE4772078B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0B3356B0078; Sun, 29 Mar 2020 22:33:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 014CC6B007B; Sun, 29 Mar 2020 22:33:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E213B6B007D; Sun, 29 Mar 2020 22:33:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id C1AE46B0078 for ; Sun, 29 Mar 2020 22:33:44 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id D5E9040D6 for ; Mon, 30 Mar 2020 02:33:44 +0000 (UTC) X-FDA: 76650457968.24.sort58_5ae20be4b0f0b X-Spam-Summary: 2,0,0,b9664e68ff3e94b4,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:1:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2895:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4049:4250:4321:4385:5007:6117:6119:6261:6653:6742:7514:7576:7875:7903:8660:9010:9592:10004:10394:11026:11473:11658:11914:12043:12291:12295:12296:12297:12438:12517:12519:12555:12683:12895:12986:13148:13161:13229:13230:13894:14096:14394:21080:21324:21433:21444:21451:21627:21740:21789:21796:21939:21987:21990:30036:30045:30054:30055:30070,0,RBL:209.85.222.194:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: sort58_5ae20be4b0f0b X-Filterd-Recvd-Size: 10001 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:44 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id x3so17537480qki.4 for ; Sun, 29 Mar 2020 19:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BgdGPXJPeX9SnpInGUH6EZTenlvWX+9TvCWRBUF+hwk=; b=dequhaQuCg4Eqs8SK1mEtFVEs26FepKbkbjAt/iTUd8AcqAnOYSdUPNVsOcRcFnf3c uSKJuhhwPHGFaJXWfClV3neUpzuC75KU5hQe/5GnHKw4jSU1iuKJYl1LlCOdsVWxjcxp sutV85pSCaoLahQrMd9R+DAWXDHWhmg94iVyo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BgdGPXJPeX9SnpInGUH6EZTenlvWX+9TvCWRBUF+hwk=; b=QpdMGrgab0i0N40WDVj/AiscFZlIe7Rao5c+tNOQIqFybtvigZ/Z8fwN/Fl9Bo+TS2 xEm8IbbDf41J/7Z88BIE/jA9RonQibaKcHQJxYny3LSZBunfzIatjc+J15M+1LJGDrxm QX6CKdkpUWPSo4ChjOf+zsZA6zJNyOoEQRp30kUOlKE62lEqHe5xlsZzhyI+KG4S+4KA CCyjTRribTGlxF7+NX/s3PHPhukcRmTjWWGh7LOFzh5n1rJQwan1J/s8vCD5TdHKe/Cv qpYmLFqrOIISI197e7uoPCqVg2ithohGPzkLC78zsE7i+khGqmBLmPxj21z/hqtZDGmM uXKA== X-Gm-Message-State: ANhLgQ3+mC09FEh/6A2alKk6ux/8QTLIot8mGRrlO+jA4FsfFymNyYP1 hheK0FeKI+qw6Vppq6MrVOxT6g== X-Google-Smtp-Source: ADFU+vt87q8g9W1XSLYZFPTn1haQJjHn0TkgzZRXHxTEV8TU+ZBXSX5+SoNzrFUyyKBnb65yliV9Vg== X-Received: by 2002:a37:4548:: with SMTP id s69mr10199428qka.63.1585535623472; Sun, 29 Mar 2020 19:33:43 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:43 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 12/18] rcu/tree: Support reclaim for head-less object Date: Sun, 29 Mar 2020 22:32:42 -0400 Message-Id: <20200330023248.164994-13-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Update the kvfree_call_rcu() with head-less support, it means an object without any rcu_head structure can be reclaimed after GP. To store pointers there are two chain-arrays maintained one for SLAB and another one is for vmalloc. Both types of objects(head-less variant and regular one) are placed there based on the type. It can be that maintaining of arrays becomes impossible due to high memory pressure. For such reason there is an emergency path. In that case objects with rcu_head inside are just queued building one way list. Later on that list is drained. As for head-less variant. Such objects do not have any rcu_head helper inside. Thus it is dynamically attached. As a result an object consists of back-pointer and regular rcu_head. It implies that emergency path can detect such object type, therefore they are tagged. So a back-pointer could be freed as well as dynamically attached wrapper. Even though such approach requires dynamic memory it needs only sizeof(unsigned long *) + sizeof(struct rcu_head) bytes, thus SLAB is used to obtain it. Finally if attaching of the rcu_head and queuing get failed, the current context has to follow might_sleep() annotation, thus below steps could be applied: a) wait until a grace period has elapsed; b) direct inlining of the kvfree() call. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 93 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 85 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 3b94526f490cb..204292378101b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2959,19 +2959,34 @@ static void kfree_rcu_work(struct work_struct *work) * when we could not allocate a bulk array. * * Under that condition an object is queued to the - * list instead. + * list instead. Please note that head-less objects + * have dynamically attached rcu_head, so they also + * contain a back-pointer that has to be freed. */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; - void *ptr = (void *)head - offset; + bool headless; + void *ptr; next = head->next; + + /* We tag the headless object, if so adjust offset. */ + headless = (((unsigned long) head - offset) & BIT(0)); + if (headless) + offset -= 1; + + ptr = (void *) head - offset; debug_rcu_head_unqueue((struct rcu_head *)ptr); + rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); - if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) + if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) { + if (headless) + kvfree((void *) *((unsigned long *) ptr)); + kvfree(ptr); + } rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3120,6 +3135,24 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) return true; } +static inline struct rcu_head *attach_rcu_head_to_object(void *obj) +{ + unsigned long *ptr; + + ptr = kmalloc(sizeof(unsigned long *) + + sizeof(struct rcu_head), GFP_NOWAIT | __GFP_NOWARN); + + if (!ptr) + ptr = kmalloc(sizeof(unsigned long *) + + sizeof(struct rcu_head), GFP_ATOMIC | __GFP_NOWARN); + + if (!ptr) + return NULL; + + ptr[0] = (unsigned long) obj; + return ((struct rcu_head *) ++ptr); +} + /* * Queue a request for lazy invocation of appropriate free routine after a * grace period. Please note there are three paths are maintained, two are the @@ -3138,20 +3171,37 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) unsigned long flags; struct kfree_rcu_cpu *krcp; bool expedited_drain = false; + bool success; void *ptr; + if (head) { + ptr = (void *) head - (unsigned long) func; + } else { + /* + * Please note there is a limitation for the head-less + * variant, that is why there is a clear rule for such + * objects: + * + * use it from might_sleep() context only. For other + * places please embed an rcu_head to your structures. + */ + might_sleep(); + ptr = (unsigned long *) func; + } + local_irq_save(flags); // For safely calling this_cpu_ptr(). krcp = this_cpu_ptr(&krc); if (krcp->initialized) spin_lock(&krcp->lock); - ptr = (void *)head - (unsigned long)func; - // Queue the object but don't yet schedule the batch. if (debug_rcu_head_queue(ptr)) { // Probable double kfree_rcu(), just leak. WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", __func__, head); + + /* Mark as success and leave. */ + success = true; goto unlock_return; } @@ -3159,7 +3209,22 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr)) { + success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr); + if (!success) { + /* Is headless object? */ + if (head == NULL) { + head = attach_rcu_head_to_object(ptr); + if (head == NULL) + goto unlock_return; + + /* + * Tag the headless object. Such objects have a back-pointer + * to the original allocated memory, that has to be freed as + * well as dynamically attached wrapper/head. + */ + func = (rcu_callback_t) (sizeof(unsigned long *) + 1); + } + head->func = func; head->next = krcp->head; krcp->head = head; @@ -3171,15 +3236,15 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * appropriate free calls. */ expedited_drain = true; + success = true; } WRITE_ONCE(krcp->count, krcp->count + 1); // Set timer to drain after KFREE_DRAIN_JIFFIES. if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING && - !krcp->monitor_todo) { + !krcp->monitor_todo) { krcp->monitor_todo = true; - schedule_delayed_work(&krcp->monitor_work, expedited_drain ? 0 : KFREE_DRAIN_JIFFIES); } @@ -3188,6 +3253,18 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) if (krcp->initialized) spin_unlock(&krcp->lock); local_irq_restore(flags); + + /* + * High memory pressure, so inline kvfree() after + * synchronize_rcu(). We can do it from might_sleep() + * context only, so the current CPU can pass the QS + * state. + */ + if (!success) { + debug_rcu_head_unqueue(ptr); + synchronize_rcu(); + kvfree(ptr); + } } EXPORT_SYMBOL_GPL(kvfree_call_rcu); From patchwork Mon Mar 30 02:32:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464489 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C946C14B4 for ; Mon, 30 Mar 2020 02:34:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8D05D2078B for ; Mon, 30 Mar 2020 02:34:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="VI/qIKph" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D05D2078B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F115A6B007B; Sun, 29 Mar 2020 22:33:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DD13D6B007D; Sun, 29 Mar 2020 22:33:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C49CD6B007E; Sun, 29 Mar 2020 22:33:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0113.hostedemail.com [216.40.44.113]) by kanga.kvack.org (Postfix) with ESMTP id AD24C6B007B for ; Sun, 29 Mar 2020 22:33:45 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A060D181AD0A1 for ; Mon, 30 Mar 2020 02:33:45 +0000 (UTC) X-FDA: 76650458010.18.cover50_5b01532723f2c X-Spam-Summary: 2,0,0,f3ee9f27f9c335d2,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3871:4321:4385:4390:4395:4605:5007:6261:6653:6742:7514:7576:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21212:21444:21451:21627:30054,0,RBL:209.85.160.194:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: cover50_5b01532723f2c X-Filterd-Recvd-Size: 4720 Received: from mail-qt1-f194.google.com (mail-qt1-f194.google.com [209.85.160.194]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:45 +0000 (UTC) Received: by mail-qt1-f194.google.com with SMTP id g7so13882710qtj.13 for ; Sun, 29 Mar 2020 19:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=b0k3soKURyDBh9v099L9R8zvO5sOH/jeMANg/8/7Rzk=; b=VI/qIKphu5cG5RcFKyvsw3jMUdxJoZtgoDsGaptYl1UYTimCaWzYL9uhYPDg//wONL j9zq93V+5DBwWreZj901i//u1yxPIc8rSnuWwcdNbo5eKT7SZpcjCpou5Xtv81k7qB4B po3VX9PxBTGTPCmy4N8gkdA6AFQrXHt7/UOg0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=b0k3soKURyDBh9v099L9R8zvO5sOH/jeMANg/8/7Rzk=; b=DvadWMg+u04tkl0Oynb5tEk6oa9m5l+LaBFz3xAJbPs4EyAtjmzTaKWCAvB0UjyfNf Ubr4/4+BSeK5XWb22UUmQFtBoWKV9XB7DF7MCH9KH49mY2U9tDdXSp2p/ArHsx5gHucJ rub1unyq5LQ0Y0N3/vT5ePOa+arER9jai+7JXbjf1OfaHRVec01XQ5dlWOPcPNX22uiq c6a5PEocCsLG5LJuZ4q/cdjeuqORCn/lhm3T2HktOwlEUvfW8rucLMzvuvaF9x+iewt3 EPj5tmBCRpdpHSxeB+8LDl3acTV/gaigN3LhmR/RKT5HEVu4nEPIo32YgqVgLDU1ousN HUJA== X-Gm-Message-State: ANhLgQ2iwabRF5gai3+WcA/ABQXaJ7/rzeJGj8LbM3pLNulcJXQbhWAU 8C6j2twf4scIJwVaapT40mimug== X-Google-Smtp-Source: ADFU+vtXYlMnNTXa8HH2h/iQhtZELpahx663+F4vLVBBsB/DDJ77YfhvA5Xkfej8EvFPpXiZ+guw3A== X-Received: by 2002:aed:24c2:: with SMTP id u2mr9893436qtc.269.1585535624468; Sun, 29 Mar 2020 19:33:44 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:44 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 13/18] rcu/tiny: Move kvfree_call_rcu() out of header Date: Sun, 29 Mar 2020 22:32:43 -0400 Message-Id: <20200330023248.164994-14-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Move inlined kvfree_call_rcu() function out of the header file. This step is a preparation for head-lees support. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcutiny.h | 6 +----- kernel/rcu/tiny.c | 6 ++++++ 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index 5ba0bcb231976..19a780afa444a 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -34,11 +34,7 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) -{ - call_rcu(head, func); -} - +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func); void rcu_qs(void); static inline void rcu_softirq_qs(void) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index aa897c3f2e92c..508c82faa45c3 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -177,6 +177,12 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } EXPORT_SYMBOL_GPL(call_rcu); +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) +{ + call_rcu(head, func); +} +EXPORT_SYMBOL_GPL(kvfree_call_rcu); + void __init rcu_init(void) { open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); From patchwork Mon Mar 30 02:32:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464491 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7ED2613A4 for ; Mon, 30 Mar 2020 02:34:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3FA2D20735 for ; Mon, 30 Mar 2020 02:34:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="Hkkf7fhI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3FA2D20735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EC0F66B007D; Sun, 29 Mar 2020 22:33:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DD5E26B007E; Sun, 29 Mar 2020 22:33:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD7E46B0080; Sun, 29 Mar 2020 22:33:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id A2C016B007D for ; Sun, 29 Mar 2020 22:33:46 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id ACC504DBF for ; Mon, 30 Mar 2020 02:33:46 +0000 (UTC) X-FDA: 76650458052.27.sail73_5b297aa7c635c X-Spam-Summary: 2,0,0,157ff7b7abbf8bdd,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2199:2393:2559:2562:2907:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:5007:6119:6261:6653:6742:7514:7576:7875:7903:8660:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13148:13211:13229:13230:13894:14096:14394:21080:21324:21433:21444:21451:21627:21740:21789:21939:21987:21990:30045:30054:30070,0,RBL:209.85.222.193:@joelfernandes.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:28,LUA_SUMMARY:none X-HE-Tag: sail73_5b297aa7c635c X-Filterd-Recvd-Size: 8774 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:46 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id h14so17551450qke.5 for ; Sun, 29 Mar 2020 19:33:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ehI/mevr7oGjGlbprsNt2erBqRqo03dz0gk8qK2323I=; b=Hkkf7fhI9UgKMG2bmf1X4tv05OtccwFMkvjxvsFynnH81KwsXpgysqGhe0jyrTAFx8 vtgbEk9Dq27acsmjbGZnmonsgzE0Eal+DdboJtO72c5rgwanQS6VMOhputaGm3E4H/d4 9/eWwrgjl5zhL/Jrdd1K0fyGGeiW0rdvBddNk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ehI/mevr7oGjGlbprsNt2erBqRqo03dz0gk8qK2323I=; b=KoERp73i7dY+YL7r6+vuJ/h7PzftstvrFja4b6dmxKBHaGxx9qTD/XgyMTL4OQfsl8 etllYsg56LU4FprcylJKHjHleNhw86Skb2iY/kDqWzQbuLhEkjFkKBWzhkRC2wGzOzzm fGz508B/q5fNGcDshGqlR1kz4jKh74en8PrDrCXj8n3FoMa0qaKO3kO5J/Y8FHtW918S y1bkEmUEW6U+OAp9gu6gniOhC9zL6WQzBsGre6FcPG9kdZCdNrRbaA9L5oN/Iz/YktKa 9+G1cFA0YWkUEi+MdE91tf+XhOgqQza46Gm2+EcPTk66KENTYMJxgCQXDHYk3Mo1Xqxe BEUA== X-Gm-Message-State: ANhLgQ2hugYPyCbLxWzeF5bDgorlP6AxqMqlLaTMqlMTYAliLWkw9w15 5qZc14ABgcHfUfC44URzlUeuLA== X-Google-Smtp-Source: ADFU+vsEu6bs9H+nTRyqOkzVOqwut6t53PnZdb4J1el0BLUwgt9JHzOjhj/hAdHYb48agOxgcw9Hfw== X-Received: by 2002:a37:2fc3:: with SMTP id v186mr9919220qkh.311.1585535625444; Sun, 29 Mar 2020 19:33:45 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:45 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 14/18] rcu/tiny: Support reclaim for head-less object Date: Sun, 29 Mar 2020 22:32:44 -0400 Message-Id: <20200330023248.164994-15-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Make a kvfree_call_rcu() function to support head-less freeing. Same as for tree-RCU, for such purpose we store pointers in array. SLAB and vmalloc ptrs. are mixed and coexist together. Under high memory pressure it can be that maintaining of arrays becomes impossible. Objects with an rcu_head are released via call_rcu(). When it comes to the head-less variant, the kvfree() call is directly inlined, i.e. we do the same as for tree-RCU: a) wait until a grace period has elapsed; b) direct inlining of the kvfree() call. Thus the current context has to follow might_sleep() annotation. Also please note that for tiny-RCU any call of synchronize_rcu() is actually a quiescent state, therefore (a) does nothing. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tiny.c | 157 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 156 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 508c82faa45c3..b1c31a935db93 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -40,6 +40,29 @@ static struct rcu_ctrlblk rcu_ctrlblk = { .curtail = &rcu_ctrlblk.rcucblist, }; +/* Can be common with tree-RCU. */ +#define KVFREE_DRAIN_JIFFIES (HZ / 50) + +/* Can be common with tree-RCU. */ +struct kvfree_rcu_bulk_data { + unsigned long nr_records; + struct kvfree_rcu_bulk_data *next; + void *records[]; +}; + +/* Can be common with tree-RCU. */ +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) + +static struct kvfree_rcu_bulk_data *kvhead; +static struct kvfree_rcu_bulk_data *kvhead_free; +static struct kvfree_rcu_bulk_data *kvcache; + +static DEFINE_STATIC_KEY_FALSE(rcu_init_done); +static struct delayed_work monitor_work; +static struct rcu_work rcu_work; +static bool monitor_todo; + void rcu_barrier(void) { wait_rcu_gp(call_rcu); @@ -177,9 +200,137 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } EXPORT_SYMBOL_GPL(call_rcu); +static inline bool +kvfree_call_rcu_add_ptr_to_bulk(void *ptr) +{ + struct kvfree_rcu_bulk_data *bnode; + + if (!kvhead || kvhead->nr_records == KVFREE_BULK_MAX_ENTR) { + bnode = xchg(&kvcache, NULL); + if (!bnode) + bnode = (struct kvfree_rcu_bulk_data *) + __get_free_page(GFP_NOWAIT | __GFP_NOWARN); + + if (unlikely(!bnode)) + return false; + + /* Initialize the new block. */ + bnode->nr_records = 0; + bnode->next = kvhead; + + /* Attach it to the bvhead. */ + kvhead = bnode; + } + + /* Done. */ + kvhead->records[kvhead->nr_records++] = ptr; + return true; +} + +static void +kvfree_rcu_work(struct work_struct *work) +{ + struct kvfree_rcu_bulk_data *kvhead_tofree, *next; + unsigned long flags; + int i; + + local_irq_save(flags); + kvhead_tofree = kvhead_free; + kvhead_free = NULL; + local_irq_restore(flags); + + /* Reclaim process. */ + for (; kvhead_tofree; kvhead_tofree = next) { + next = kvhead_tofree->next; + + for (i = 0; i < kvhead_tofree->nr_records; i++) { + debug_rcu_head_unqueue((struct rcu_head *) + kvhead_tofree->records[i]); + kvfree(kvhead_tofree->records[i]); + } + + if (cmpxchg(&kvcache, NULL, kvhead_tofree)) + free_page((unsigned long) kvhead_tofree); + } +} + +static inline bool +queue_kvfree_rcu_work(void) +{ + /* Check if the free channel is available. */ + if (kvhead_free) + return false; + + kvhead_free = kvhead; + kvhead = NULL; + + /* + * Queue the job for memory reclaim after GP. + */ + queue_rcu_work(system_wq, &rcu_work); + return true; +} + +static void kvfree_rcu_monitor(struct work_struct *work) +{ + unsigned long flags; + bool queued; + + local_irq_save(flags); + queued = queue_kvfree_rcu_work(); + if (queued) + /* Success. */ + monitor_todo = false; + local_irq_restore(flags); + + /* + * If previous RCU reclaim process is still in progress, + * schedule the work one more time to try again later. + */ + if (monitor_todo) + schedule_delayed_work(&monitor_work, + KVFREE_DRAIN_JIFFIES); +} + void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { - call_rcu(head, func); + unsigned long flags; + bool success; + void *ptr; + + if (head) { + ptr = (void *) head - (unsigned long) func; + } else { + might_sleep(); + ptr = (void *) func; + } + + if (debug_rcu_head_queue(ptr)) { + /* Probable double free, just leak. */ + WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", + __func__, head); + return; + } + + local_irq_save(flags); + success = kvfree_call_rcu_add_ptr_to_bulk(ptr); + if (static_branch_likely(&rcu_init_done)) { + if (success && !monitor_todo) { + monitor_todo = true; + schedule_delayed_work(&monitor_work, + KVFREE_DRAIN_JIFFIES); + } + } + local_irq_restore(flags); + + if (!success) { + if (!head) { + synchronize_rcu(); + kvfree(ptr); + } else { + call_rcu(head, func); + } + } } EXPORT_SYMBOL_GPL(kvfree_call_rcu); @@ -188,4 +339,8 @@ void __init rcu_init(void) open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); rcu_early_boot_tests(); srcu_init(); + + INIT_DELAYED_WORK(&monitor_work, kvfree_rcu_monitor); + INIT_RCU_WORK(&rcu_work, kvfree_rcu_work); + static_branch_enable(&rcu_init_done); } From patchwork Mon Mar 30 02:32:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464493 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 22AF51392 for ; Mon, 30 Mar 2020 02:34:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DAB0420735 for ; Mon, 30 Mar 2020 02:34:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="QzIZMvvK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DAB0420735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E9AD56B007E; Sun, 29 Mar 2020 22:33:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E20D76B0080; Sun, 29 Mar 2020 22:33:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE8166B0081; Sun, 29 Mar 2020 22:33:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0031.hostedemail.com [216.40.44.31]) by kanga.kvack.org (Postfix) with ESMTP id B3DD56B007E for ; Sun, 29 Mar 2020 22:33:47 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B6EC0181AD0A1 for ; Mon, 30 Mar 2020 02:33:47 +0000 (UTC) X-FDA: 76650458094.15.stone99_5b4db3143350d X-Spam-Summary: 2,0,0,cd11c2a7310a468a,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:4385:4390:4395:4605:5007:6119:6261:6653:6742:7514:7576:7875:7904:10004:11026:11473:11658:11914:12043:12291:12295:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13894:14181:14394:14721:21063:21080:21433:21444:21451:21627:21796:21939:30029:30036:30054:30070,0,RBL:209.85.160.196:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: stone99_5b4db3143350d X-Filterd-Recvd-Size: 6288 Received: from mail-qt1-f196.google.com (mail-qt1-f196.google.com [209.85.160.196]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:47 +0000 (UTC) Received: by mail-qt1-f196.google.com with SMTP id t17so13887626qtn.12 for ; Sun, 29 Mar 2020 19:33:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GUEUjnZByBMLXXgM/0YNDI/bxdwR+I0Cjr1aQHLknOs=; b=QzIZMvvKWmxMiOU35hOs8CLclttdCYXncAnCQ+hBWiQayLQhtd54ejH+VB98qoGytL jL4QHOy47RVa5FzCvrDWf8Y+mlYDsycomyOHalE85DcVFRzlvjF04cfiJQiaX6rxsrTb vvECi0mg+SEVN6NPXug5XDilRNcywkAzXeCnM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GUEUjnZByBMLXXgM/0YNDI/bxdwR+I0Cjr1aQHLknOs=; b=bPDBThvwUBhJkeUgoiMRcUNNLGk3/GM0wZ214+kQauFSHWPO8U5UcTLhQwZYS4FV9m uXgmRRLijIf/me4erWu10Mv8S/8GxEWQUCczBVtzgwqtZ1Yh/+2OVNiKZ5E6NW4TzDWj X5JIchDzpLFhBx6qKLLn/qd/EnQsP3SxADUdQI/1z7o33meupw+ZP615T9pIUhSSaifF 0Olb/pIFVlwHQsWbxJcHgqP0KGe9voZVWvqmFkgbGcMaFrm5eiL1s+jCEWx73kZ+VJcr fkyO0xCW5DE67STX0mrAqL2moI9+k9sQBZ9+gunsx0mK4CnUiy1eoKUTPMHwuRhpyBRG +MRw== X-Gm-Message-State: ANhLgQ3E3bZY3Xnj8pBcqQ32cnMcRcql//voY6wPHjKtTe/KmERe3/JZ ZGeMpnnZMRaelWJmvl+pfQUx9Q== X-Google-Smtp-Source: ADFU+vuY3vIdhCVs5nHfOdatShJUCKQJyQx8NufJHl0QmK67xOuIERE/nmABHpB1Lz3V3kwdOI7DTA== X-Received: by 2002:aed:3e8e:: with SMTP id n14mr10113639qtf.245.1585535626447; Sun, 29 Mar 2020 19:33:46 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:45 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 15/18] rcu: Support headless variant in the kvfree_rcu() Date: Sun, 29 Mar 2020 22:32:45 -0400 Message-Id: <20200330023248.164994-16-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Make it possible to pass one or two arguments to the kvfree_rcu() macro what corresponds to either headless case or not, so it becomes a bit versatile. As a result we obtain two ways of using that macro, below are two examples: a) kvfree_rcu(ptr, rhf); struct X { struct rcu_head rhf; unsigned char data[100]; }; void *ptr = kvmalloc(sizeof(struct X), GFP_KERNEL); if (ptr) kvfree_rcu(ptr, rhf); b) kvfree_rcu(ptr); void *ptr = kvmalloc(some_bytes, GFP_KERNEL); if (ptr) kvfree_rcu(ptr); Last one, we name it headless variant, only needs one argument, means it does not require any rcu_head to be present within the type of ptr. There is a restriction the (b) context has to fall into might_sleep() annotation. To check that, please activate the CONFIG_DEBUG_ATOMIC_SLEEP option in your kernel. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index edb6eeba49f83..7d04bbeeeef14 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -872,12 +872,42 @@ do { \ /** * kvfree_rcu() - kvfree an object after a grace period. - * @ptr: pointer to kvfree - * @rhf: the name of the struct rcu_head within the type of @ptr. * - * Same as kfree_rcu(), just simple alias. + * This macro consists of one or two arguments and it is + * based on whether an object is head-less or not. If it + * has a head then a semantic stays the same as it used + * to be before: + * + * kvfree_rcu(ptr, rhf); + * + * where @ptr is a pointer to kvfree(), @rhf is the name + * of the rcu_head structure within the type of @ptr. + * + * When it comes to head-less variant, only one argument + * is passed and that is just a pointer which has to be + * freed after a grace period. Therefore the semantic is + * + * kvfree_rcu(ptr); + * + * where @ptr is a pointer to kvfree(). + * + * Please note, head-less way of freeing is permitted to + * use from a context that has to follow might_sleep() + * annotation. Otherwise, please switch and embed the + * rcu_head structure within the type of @ptr. */ -#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf) +#define kvfree_rcu(...) KVFREE_GET_MACRO(__VA_ARGS__, \ + kvfree_rcu_arg_2, kvfree_rcu_arg_1)(__VA_ARGS__) + +#define KVFREE_GET_MACRO(_1, _2, NAME, ...) NAME +#define kvfree_rcu_arg_2(ptr, rhf) kfree_rcu(ptr, rhf) +#define kvfree_rcu_arg_1(ptr) \ +do { \ + typeof(ptr) ___p = (ptr); \ + \ + if (___p) \ + kvfree_call_rcu(NULL, (rcu_callback_t) (___p)); \ +} while (0) /* * Place this after a lock-acquisition primitive to guarantee that From patchwork Mon Mar 30 02:32:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464495 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DFF7913A4 for ; Mon, 30 Mar 2020 02:34:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AE2FC20780 for ; Mon, 30 Mar 2020 02:34:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="IoSt4QrC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AE2FC20780 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D844A6B0080; Sun, 29 Mar 2020 22:33:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D105D6B0081; Sun, 29 Mar 2020 22:33:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB29F6B0082; Sun, 29 Mar 2020 22:33:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id 9DBC86B0080 for ; Sun, 29 Mar 2020 22:33:48 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A234D8248047 for ; Mon, 30 Mar 2020 02:33:48 +0000 (UTC) X-FDA: 76650458136.07.rest38_5b720bd49781d X-Spam-Summary: 2,0,0,8744cea5cee62ec2,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3870:3876:4321:4385:4390:4395:5007:6261:6653:6742:9707:10004:11026:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13069:13311:13357:13894:14096:14384:14394:14721:21080:21444:21627:21740:30012:30054:30070,0,RBL:209.85.160.194:@joelfernandes.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: rest38_5b720bd49781d X-Filterd-Recvd-Size: 4631 Received: from mail-qt1-f194.google.com (mail-qt1-f194.google.com [209.85.160.194]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:48 +0000 (UTC) Received: by mail-qt1-f194.google.com with SMTP id e14so13956844qts.2 for ; Sun, 29 Mar 2020 19:33:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bOOZauNhsuF3feRi6dtskt8y9Qi7+H8gEQhQhgkWxqI=; b=IoSt4QrCLedpBvOuvuIWu3Cl4FfWPyXvbqqnuq/GaQ1HcU9khIkVEaqme/mPZxUhwv kD/yTyqYJNzHz7rPyQEXhmXtMbm69VHMETJQyAIAvXcj8l0UuBlX+/nqST46f4yYXTrf k0xWlyQk9oM+n+F+7at54OwU18Uv8Wr8uj0Vo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bOOZauNhsuF3feRi6dtskt8y9Qi7+H8gEQhQhgkWxqI=; b=caDCROn1tTZdk7Y51Pif0B4VmhwEZapIYLkxJs9VVYg+g6TV51h+qbyuX39lPi+WEt 6FAF+nl7QX1fytM/7jCTuNvMdbE7CZi0ZeETmkLIggDXwn4urzPpKyh9meKwW77THj5U a8fYDGYn38tSY6hFRBegqjk3iTwwq+y/l9CkoS0fd1ptmaUxghmsHA/C05yjywj1J+Dg 0b0LlF+eOQV+FIuKKELMGhuXgPJxtXs1Pl76ELkGsyoTiR37I5XdhM3+2iG3Rv+D2GtW am/ctJNOJzNFV9vmaiDwbhhsm4gLTbEZAqvp5R+X4WLpTZTUSt0vQ/Gtl+CAAXzE36IW UWJA== X-Gm-Message-State: ANhLgQ3Nh6ytpzZ7yHe1VIn9nOnyICfPFQcm57eKJYJMojZ/GULDR6et yGTbAFIj1ROCllVmvnMxRjMWmhP+vPQ= X-Google-Smtp-Source: ADFU+vvpUM4thXk+u3mpgUqeS17+XJRaBpxyDXS4TQ5h4ZRtyfMYG8HpGORF6osOZmFTIBNIXOApRA== X-Received: by 2002:ac8:2c73:: with SMTP id e48mr9339461qta.96.1585535627462; Sun, 29 Mar 2020 19:33:47 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:47 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt , "Uladzislau Rezki (Sony)" Subject: [PATCH 16/18] rcu/tree: Remove extra next variable in kfree worker function Date: Sun, 29 Mar 2020 22:32:46 -0400 Message-Id: <20200330023248.164994-17-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No change in code, small refactor. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 204292378101b..56c9e102a901d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2889,8 +2889,7 @@ debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead) static void kfree_rcu_work(struct work_struct *work) { unsigned long flags; - struct kvfree_rcu_bulk_data *bkhead, *bknext; - struct kvfree_rcu_bulk_data *bvhead, *bvnext; + struct kvfree_rcu_bulk_data *bkhead, *bvhead, *bnext; struct rcu_head *head, *next; struct kfree_rcu_cpu *krcp; struct kfree_rcu_cpu_work *krwp; @@ -2915,8 +2914,8 @@ static void kfree_rcu_work(struct work_struct *work) spin_unlock_irqrestore(&krcp->lock, flags); /* kmalloc()/kfree() channel. */ - for (; bkhead; bkhead = bknext) { - bknext = bkhead->next; + for (; bkhead; bkhead = bnext) { + bnext = bkhead->next; debug_rcu_bhead_unqueue(bkhead); @@ -2934,8 +2933,8 @@ static void kfree_rcu_work(struct work_struct *work) } /* vmalloc()/vfree() channel. */ - for (; bvhead; bvhead = bvnext) { - bvnext = bvhead->next; + for (; bvhead; bvhead = bnext) { + bnext = bvhead->next; debug_rcu_bhead_unqueue(bvhead); From patchwork Mon Mar 30 02:32:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464497 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AE6A313A4 for ; Mon, 30 Mar 2020 02:34:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 72AFA206DB for ; Mon, 30 Mar 2020 02:34:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="e/QROEfb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72AFA206DB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B60FF6B0081; Sun, 29 Mar 2020 22:33:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AED6C6B0082; Sun, 29 Mar 2020 22:33:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9432B6B0083; Sun, 29 Mar 2020 22:33:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0069.hostedemail.com [216.40.44.69]) by kanga.kvack.org (Postfix) with ESMTP id 6E0296B0081 for ; Sun, 29 Mar 2020 22:33:49 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7C269180AD81A for ; Mon, 30 Mar 2020 02:33:49 +0000 (UTC) X-FDA: 76650458178.30.comb08_5b92af30a435a X-Spam-Summary: 2,0,0,2929d9d7c511b58c,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1539:1567:1711:1714:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3622:3871:3876:4321:4385:5007:6261:6653:6742:10004:11026:11658:11914:12043:12297:12438:12517:12519:12555:12895:13069:13141:13230:13311:13357:13894:14384:14394:14721:21080:21444:21451:21627:21990:30012:30054:30070,0,RBL:209.85.160.194:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: comb08_5b92af30a435a X-Filterd-Recvd-Size: 3852 Received: from mail-qt1-f194.google.com (mail-qt1-f194.google.com [209.85.160.194]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:49 +0000 (UTC) Received: by mail-qt1-f194.google.com with SMTP id i3so13899421qtv.8 for ; Sun, 29 Mar 2020 19:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nfe8DrKWUp4DznQJ7QwjX25sDztlI0Q1Kzc2EbjhIfI=; b=e/QROEfb5RGTR9IB9T4I6da+BeTQWc1g0g4Y4IB6QTSmalMQrHGbXLRXYVq0YbVzhs tP2sTlESNLvrtJivJplrhzgd3CeNe5CxFhf+jkbqQ4hFWDiT7AJDlWveUX3cErsFRXs6 BxPFwUArDnqMXIc2Ozny0JZzGurm8NA5J/8uE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nfe8DrKWUp4DznQJ7QwjX25sDztlI0Q1Kzc2EbjhIfI=; b=HyuYHxsOxntvF0xYgQ4dc4OCJNriqBi+IGm1lien1FxQfEf5tohzdZwsYPjJo3rRjB 0sR4GBjikMG2WaEynMhN2Ny4Np7RbhUTx0LkxujqhEbbjY3vUqLRDDgO5y/RkF6DpdBd rIpMgZ3uYYHhxOoFz69vzNmJ7xp9WzGGlcBWI3IyKEnkELXxHSPmXCDCXtyZJYK/J8HF IkiU6AOxUEIbixbyOINP9NiV+/tpymuy801gvlFogJjmCMujJvU9KkAR1RAaWYpOij6X l4i+gWSzo6QPHBmwoVYFR4no/gTHYtXc+GSLcB9Y9x/WvoS+1rqA1uCwsIzAeY7w/OIO WL6w== X-Gm-Message-State: ANhLgQ3IbtT8hl1fszZrLdZRj8wpB+mKkZgHnvTyyf3geofFtnW0NTI8 3yxaIvPYNqoH0HAKI0kRXbC+hw== X-Google-Smtp-Source: ADFU+vte8V+MFLpGv25SbcL8/a36+3V3dZDn+RHDRlNYMBkkicXPxgyi30CKdwoHh7ULy/2SwK0YqQ== X-Received: by 2002:ac8:922:: with SMTP id t31mr9208234qth.95.1585535628345; Sun, 29 Mar 2020 19:33:48 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:48 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt , "Uladzislau Rezki (Sony)" Subject: [PATCH 17/18] rcu/tree: Simplify is_vmalloc_addr expression Date: Sun, 29 Mar 2020 22:32:47 -0400 Message-Id: <20200330023248.164994-18-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No code change, small refactor. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 56c9e102a901d..311d216c7faa7 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3104,7 +3104,7 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) return false; lockdep_assert_held(&krcp->lock); - idx = !is_vmalloc_addr(ptr) ? 0:1; + idx = !!is_vmalloc_addr(ptr); /* Check if a new block is required. */ if (!krcp->bkvhead[idx] || From patchwork Mon Mar 30 02:32:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464499 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78B041392 for ; Mon, 30 Mar 2020 02:34:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4582020780 for ; Mon, 30 Mar 2020 02:34:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="w1Utm7e6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4582020780 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B3E6A6B0082; Sun, 29 Mar 2020 22:33:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AC6966B0083; Sun, 29 Mar 2020 22:33:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8538A6B0085; Sun, 29 Mar 2020 22:33:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 6E7346B0082 for ; Sun, 29 Mar 2020 22:33:50 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7996A40D6 for ; Mon, 30 Mar 2020 02:33:50 +0000 (UTC) X-FDA: 76650458220.19.art11_5bb7b99f68e30 X-Spam-Summary: 2,0,0,d93b20965dc9f6ab,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3866:3867:3871:3874:4321:4385:4390:4395:5007:6119:6261:6653:6742:7875:10004:10394:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13069:13311:13357:13894:14384:14394:14721:21080:21444:21451:21627:21740:21789:30054,0,RBL:209.85.222.196:@joelfernandes.org:.lbl8.mailshell.net-62.2.0.100 66.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: art11_5bb7b99f68e30 X-Filterd-Recvd-Size: 4882 Received: from mail-qk1-f196.google.com (mail-qk1-f196.google.com [209.85.222.196]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:50 +0000 (UTC) Received: by mail-qk1-f196.google.com with SMTP id i6so17556407qke.1 for ; Sun, 29 Mar 2020 19:33:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bIwgeh9rHwo41HBlXUss+pfAH9DbFE6iCebsly3ZN4c=; b=w1Utm7e6d4/535TdQtPiMXksX4j1qcmln0qSp2Ce5ITm627q5dnSU8lnhyBq3uY9oE 0PhDmMsluIYgdTjcV+PSd7sVQ/WGe+IhJGHMYogCDZYqSvqJL1tsZLjyg/tB9D7Y4aYN zqy3An60oNrntByo8BqUBrC2oSWocnndGjLG8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bIwgeh9rHwo41HBlXUss+pfAH9DbFE6iCebsly3ZN4c=; b=j1ClIH/i3gqDC0HWl7ulOBlBX62xBRRyBl2dJN+uOgqmrZnDRSDWVz57T7V75IQQAM R+miJ0O0yQ6CgKjmQeRjIlAim3iXI32Txe+7B3BCCnArt/gFEooTQgKu5xrW9eNTKfX2 89TyRAX0CECftVPwrhoUqXlV7/WhMUQr1Asm1mDqsMvYmd5FpfgO5QySq+Vd5qNzWoud HN5HTdzWXCWTPOeGV94XssthOcFmiZyT3a+SmuTmKxcg7Lubl/3LItdPCxd+7OQr7b5h 7cNwlcO82EXZu0Wh2WBS1lSuCGkHHOtKMFOELx2rk4kz25BDxriDbn+Ea6x+ne23MhY/ Aekg== X-Gm-Message-State: ANhLgQ2+ammUql40FbX4eb7NBsW49n6E9LzpEh/QHGgJ+bVElk9xl5XB 8vvpWaLN7PiG7NQ+e0ymnNVzVw== X-Google-Smtp-Source: ADFU+vurXBFAw9FRqEBRvtnmRBiIAJYMQMQAx1nAenRxy7DRsubkevgiGpgtf0ZMZlIvcVQv3d7mpA== X-Received: by 2002:a37:a68e:: with SMTP id p136mr9993192qke.7.1585535629357; Sun, 29 Mar 2020 19:33:49 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:48 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt , "Uladzislau Rezki (Sony)" Subject: [PATCH 18/18] rcu/tree: Make kvfree_rcu() tolerate any alignment Date: Sun, 29 Mar 2020 22:32:48 -0400 Message-Id: <20200330023248.164994-19-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Handle cases where the the object being kvfree_rcu()'d is not aligned by 2-byte boundaries. Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 311d216c7faa7..d6536374d12a9 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2827,6 +2827,9 @@ struct kvfree_rcu_bulk_data { #define KVFREE_BULK_MAX_ENTR \ ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) +/* Encoding the offset of a fake rcu_head to indicate the head is a wrapper. */ +#define RCU_HEADLESS_KFREE BIT(31) + /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period @@ -2970,9 +2973,9 @@ static void kfree_rcu_work(struct work_struct *work) next = head->next; /* We tag the headless object, if so adjust offset. */ - headless = (((unsigned long) head - offset) & BIT(0)); + headless = !!(offset & RCU_HEADLESS_KFREE); if (headless) - offset -= 1; + offset &= ~(RCU_HEADLESS_KFREE); ptr = (void *) head - offset; debug_rcu_head_unqueue((struct rcu_head *)ptr); @@ -3221,7 +3224,7 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * to the original allocated memory, that has to be freed as * well as dynamically attached wrapper/head. */ - func = (rcu_callback_t) (sizeof(unsigned long *) + 1); + func = (rcu_callback_t)(sizeof(unsigned long *) | RCU_HEADLESS_KFREE); } head->func = func;