From patchwork Mon May 25 21:47:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569465 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B41890 for ; Mon, 25 May 2020 21:48:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DE8662075F for ; Mon, 25 May 2020 21:48:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YkNESquX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DE8662075F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8E0C68E0008; Mon, 25 May 2020 17:48:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8419C80061; Mon, 25 May 2020 17:48:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 691808E0013; Mon, 25 May 2020 17:48:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 4E2CD8E0008 for ; Mon, 25 May 2020 17:48:13 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id ECDA4181AC9CB for ; Mon, 25 May 2020 21:48:12 +0000 (UTC) X-FDA: 76856580024.12.trick69_2a96fe9f02c50 X-Spam-Summary: 2,0,0,cc5b743e29036889,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:2:41:69:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1606:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2691:2693:2898:3138:3139:3140:3141:3142:3355:3865:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:4390:4395:5007:6117:6119:6261:6653:7514:7576:7903:9040:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12683:12895:12986:13846:13894:14394:14687:21080:21433:21444:21451:21627:21666:21740:21990:30012:30054:30070,0,RBL:209.85.167.68:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: trick69_2a96fe9f02c50 X-Filterd-Recvd-Size: 8352 Received: from mail-lf1-f68.google.com (mail-lf1-f68.google.com [209.85.167.68]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:12 +0000 (UTC) Received: by mail-lf1-f68.google.com with SMTP id z206so7533845lfc.6 for ; Mon, 25 May 2020 14:48:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gGk+1Dv1xZWBD45WykPdecLcAYKfFE81IWJLJfIPzl4=; b=YkNESquXqQUg7n7ikcJ7p3fXioH3Tb6MRVXMx68ADYtDErZ6m8ILzBiSS96gBU5/K+ k9UbRIKwDOM3N4BQnVG/G+Tj8CfCAxkECW0eonIxdrxJC0Vx5fF7IWSYNPOe+/8UP+yt fJ4tdGUTbs+yFPo59NDWlvhSlCCqZFyBfIRrLpwmecehEm9mRb6yAqVBUgoerjSRTOBP G8E0yivmIeyNqO+QxWa/lIz721ZFs3qiyu4ayiVW9W+xMQPBkIxhCaKXQGpoxMLXDc1R bvj1/OuViAYkvPW7eieykUzB0ZkLpSny5Svu+2JDtcfZD4la8g0XljN0WF9w2EqxISt2 9RPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gGk+1Dv1xZWBD45WykPdecLcAYKfFE81IWJLJfIPzl4=; b=lvEvFMA0m4xSBrSiJHJ+dU/NEFmlUSlFsT50hddO+G+1tCB9x6Q4tx9+J0cAv+Zqzg zwCClUrq17iODz92OgePlL9X9plFw1Pjon83MU32SzrVuqkQatImzIww0jRC1ku2XDMX Y0IjtGlvvkgyANzCZ4pH3yYmneIcFZRm4XmUfhAEAde0VN4xrwv7/u+9MJyDWTHb+AhB hKZjge18dDcTpbR8ME36F0/Lk5yK5BMBqnsyPpuNmKikh8WU1b1ir2N1ZOy6JC8gK08f oWmBCSsRZoY+5bEecjXJhiEicLaNvJZUiNUxCLFfoKDy5zlz/E+YCmfbU1maiHXnn+4e 9Weg== X-Gm-Message-State: AOAM531bjW1NeIEUvqHFN/pF9g87hSDu6zjWKGNxpNexhCrN93zApJIl zCjW4mDE2YrcvVW75VMX0FI= X-Google-Smtp-Source: ABdhPJxnmRdUPLkFsmHymHpSeTqdd/iueRk3nTcSAYw1dqYbXgKIV3RatLW1vOgritqB/JVx2XTE6g== X-Received: by 2002:a19:6b14:: with SMTP id d20mr9822238lfa.202.1590443291199; Mon, 25 May 2020 14:48:11 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:10 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko , bigeasy@linutronix.de Subject: [PATCH v2 01/16] rcu/tree: Keep kfree_rcu() awake during lock contention Date: Mon, 25 May 2020 23:47:45 +0200 Message-Id: <20200525214800.93072-2-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Joel Fernandes (Google)" On PREEMPT_RT kernels, the krcp spinlock gets converted to an rt-mutex and causes kfree_rcu() callers to sleep. This makes it unusable for callers in purely atomic sections such as non-threaded IRQ handlers and raw spinlock sections. Fix it by converting the spinlock to a raw spinlock. Vetting all code paths, there is no reason to believe that the raw spinlock will hurt RT latencies as it is not held for a long time. Cc: bigeasy@linutronix.de Cc: Uladzislau Rezki Reviewed-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 6e120be29332..6e967a9d6704 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2882,7 +2882,7 @@ struct kfree_rcu_cpu { struct kfree_rcu_bulk_data *bhead; struct kfree_rcu_bulk_data *bcached; struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; - spinlock_t lock; + raw_spinlock_t lock; struct delayed_work monitor_work; bool monitor_todo; bool initialized; @@ -2915,12 +2915,12 @@ static void kfree_rcu_work(struct work_struct *work) krwp = container_of(to_rcu_work(work), struct kfree_rcu_cpu_work, rcu_work); krcp = krwp->krcp; - spin_lock_irqsave(&krcp->lock, flags); + raw_spin_lock_irqsave(&krcp->lock, flags); head = krwp->head_free; krwp->head_free = NULL; bhead = krwp->bhead_free; krwp->bhead_free = NULL; - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); /* "bhead" is now private, so traverse locklessly. */ for (; bhead; bhead = bnext) { @@ -3023,14 +3023,14 @@ static inline void kfree_rcu_drain_unlock(struct kfree_rcu_cpu *krcp, krcp->monitor_todo = false; if (queue_kfree_rcu_work(krcp)) { // Success! Our job is done here. - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); return; } // Previous RCU batch still in progress, try again later. krcp->monitor_todo = true; schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES); - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); } /* @@ -3043,11 +3043,11 @@ static void kfree_rcu_monitor(struct work_struct *work) struct kfree_rcu_cpu *krcp = container_of(work, struct kfree_rcu_cpu, monitor_work.work); - spin_lock_irqsave(&krcp->lock, flags); + raw_spin_lock_irqsave(&krcp->lock, flags); if (krcp->monitor_todo) kfree_rcu_drain_unlock(krcp, flags); else - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); } static inline bool @@ -3118,7 +3118,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) local_irq_save(flags); // For safely calling this_cpu_ptr(). krcp = this_cpu_ptr(&krc); if (krcp->initialized) - spin_lock(&krcp->lock); + raw_spin_lock(&krcp->lock); // Queue the object but don't yet schedule the batch. if (debug_rcu_head_queue(head)) { @@ -3149,7 +3149,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) unlock_return: if (krcp->initialized) - spin_unlock(&krcp->lock); + raw_spin_unlock(&krcp->lock); local_irq_restore(flags); } EXPORT_SYMBOL_GPL(kfree_call_rcu); @@ -3181,11 +3181,11 @@ kfree_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); count = krcp->count; - spin_lock_irqsave(&krcp->lock, flags); + raw_spin_lock_irqsave(&krcp->lock, flags); if (krcp->monitor_todo) kfree_rcu_drain_unlock(krcp, flags); else - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); sc->nr_to_scan -= count; freed += count; @@ -3212,15 +3212,15 @@ void __init kfree_rcu_scheduler_running(void) for_each_online_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); - spin_lock_irqsave(&krcp->lock, flags); + raw_spin_lock_irqsave(&krcp->lock, flags); if (!krcp->head || krcp->monitor_todo) { - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); continue; } krcp->monitor_todo = true; schedule_delayed_work_on(cpu, &krcp->monitor_work, KFREE_DRAIN_JIFFIES); - spin_unlock_irqrestore(&krcp->lock, flags); + raw_spin_unlock_irqrestore(&krcp->lock, flags); } } @@ -4113,7 +4113,7 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); - spin_lock_init(&krcp->lock); + raw_spin_lock_init(&krcp->lock); for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; From patchwork Mon May 25 21:47:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569467 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 24D7490 for ; Mon, 25 May 2020 21:48:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E60AC2075F for ; Mon, 25 May 2020 21:48:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oCYyc5EZ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E60AC2075F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9A2A380062; Mon, 25 May 2020 17:48:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8930F80061; Mon, 25 May 2020 17:48:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 756B680062; Mon, 25 May 2020 17:48:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0107.hostedemail.com [216.40.44.107]) by kanga.kvack.org (Postfix) with ESMTP id 5D31680061 for ; Mon, 25 May 2020 17:48:14 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1DAF7180AD815 for ; Mon, 25 May 2020 21:48:14 +0000 (UTC) X-FDA: 76856580108.13.stamp20_2ac18b4ea593f X-Spam-Summary: 2,0,0,6880b4ca705fff22,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:4250:4385:5007:6261:6653:7514:7576:9413:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13069:13149:13230:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21740:21990:30012:30054,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: stamp20_2ac18b4ea593f X-Filterd-Recvd-Size: 5235 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:13 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id v16so22189149ljc.8 for ; Mon, 25 May 2020 14:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uuS6fohvwHxuxWslRa3zOPr8O2j4OSUqCY5jL2HzmNU=; b=oCYyc5EZyHOL7pZjQCoROI4cZ7ULkiquYBF6HgEf+xj0qvTO/WfJp/9ksJNHuwAW42 jb4i+YHZK+iYQmSS0jE7yB7qXRO4BUBgpi6FlsphLeExsSAw6Ocz/pm1/jsYkK06L/62 YVvcqo08OBHInb6GIAoCCdqp79ZMpz73UTzIUY7G7MbGMjnwATjkvVA8htZ8P3cviT3S uZgnzRf82KNhueY1bygWe3QbX31XJSqrmPl/p77bhm8kKLCDURdSZcVFzSdmi6BimqWU s57xfoqdk5SaNCBIamiosFZNsms7W+239z0nlblFvmGpRWrEsokCqmQ3fGjEyGWRgLjM EroQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uuS6fohvwHxuxWslRa3zOPr8O2j4OSUqCY5jL2HzmNU=; b=jIVyDF2ZcvVG0OojQ9BIAXAZl0Rm3XXxel/W/N7BAMHe2q/J/4mqKEHZeoT7pyODgs 0zQYiiE9ssZP+hfmoPLCkbb81VujJ0CAXhQEaTxEPLnv/QmCvfwvMi3+3sDiq8eDAUR1 rZEsy/W36ThKjwAO5UdJs6wu2qYCpDKH+TQxRWh02sxCDbKgAvrR0oyWjOKffp61WTDq Smt3u40O/+vumPE0F8D7/bsmZJlwOrQoKjOh6nxz1P/CLOXSRD0fTe4XYY+//xhtc1kZ WvkoLFq05er7McxQrxgMfNTHojNZjRJuwNdEjZYSrc5jZOcNyvDJ5T4j2JESEKlEANJm U5Dg== X-Gm-Message-State: AOAM533DeG7dFeS347Wldy/YSGec6CHM2xMZjwkMST/szUv+mW+2is7X FGTCtwcj1k/VXbi5T5ZPibE= X-Google-Smtp-Source: ABdhPJzeCk2tHSENorydlmBycYrCOOWzwq7850vGWkJweiyMNVAzHv3gLgrw3ynteKptKefS6PxAKQ== X-Received: by 2002:a2e:9d45:: with SMTP id y5mr13895293ljj.258.1590443292528; Mon, 25 May 2020 14:48:12 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:11 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko , Sebastian Andrzej Siewior Subject: [PATCH v2 02/16] rcu/tree: Skip entry into the page allocator for PREEMPT_RT Date: Mon, 25 May 2020 23:47:46 +0200 Message-Id: <20200525214800.93072-3-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Joel Fernandes (Google)" To keep the kfree_rcu() code working in purely atomic sections on RT, such as non-threaded IRQ handlers and raw spinlock sections, avoid calling into the page allocator which uses sleeping locks on RT. In fact, even if the caller is preemptible, the kfree_rcu() code is not, as the krcp->lock is a raw spinlock. Calling into the page allocator is optional and avoiding it should be Ok, especially with the page pre-allocation support in future patches. Such pre-allocation would further avoid the a need for a dynamically allocated page in the first place. Cc: Sebastian Andrzej Siewior Reviewed-by: Uladzislau Rezki Co-developed-by: Uladzislau Rezki Signed-off-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 6e967a9d6704..d28010cf158b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3068,6 +3068,18 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, if (!bnode) { WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE); + /* + * To keep this path working on raw non-preemptible + * sections, prevent the optional entry into the + * allocator as it uses sleeping locks. In fact, even + * if the caller of kfree_rcu() is preemptible, this + * path still is not, as krcp->lock is a raw spinlock. + * With additional page pre-allocation in the works, + * hitting this return is going to be much less likely. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + return false; + bnode = (struct kfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); } From patchwork Mon May 25 21:47:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569469 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1481D913 for ; Mon, 25 May 2020 21:48:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CC4842075F for ; Mon, 25 May 2020 21:48:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="P3m9JiCz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC4842075F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E366980063; Mon, 25 May 2020 17:48:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D9C3380061; Mon, 25 May 2020 17:48:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC73880063; Mon, 25 May 2020 17:48:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0216.hostedemail.com [216.40.44.216]) by kanga.kvack.org (Postfix) with ESMTP id A239780061 for ; Mon, 25 May 2020 17:48:15 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 58DE3824556B for ; Mon, 25 May 2020 21:48:15 +0000 (UTC) X-FDA: 76856580150.04.fang39_2af063d16523b X-Spam-Summary: 2,0,0,e788284951e7f676,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:4390:4395:5007:6261:6653:7514:7903:9413:10004:11026:11658:11914:12043:12294:12296:12297:12438:12517:12519:12555:12679:12895:12986:13069:13311:13357:13868:13894:14096:14181:14384:14394:14687:14721:21080:21433:21444:21451:21627:21666:21740:21990:30045:30054,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: fang39_2af063d16523b X-Filterd-Recvd-Size: 4801 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:14 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id k5so22125433lji.11 for ; Mon, 25 May 2020 14:48:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4AXOThceM9HQTaD/1C2DnJHW4BibV/GSMKqCaOTg8gw=; b=P3m9JiCzUkqtxpY3mGwyXff/VaAi3lP8/Tw+HlJuwPCpcTr92OEkjgnqiZ51xr+MHa z49tq0aEIaaYGJKwcR6O6FW9FIExXQV1fFTdIfYDAjMWi+wCwWncxFYb/JFPYlqTXRBy VCUuHP7h++feYt4Np0dchSCZwta7xjAVajnIwZR8hWoZ/a7tG+J8V6FMvgGM7Rez+tsl wfN00uPkBMigNVhsfJGkkX+qjkKyweTM+CS6DU1+ugrgKsVomI/1uCAPxvj8+xcLvX8e lfWO+lPdcbH1PdVu9/kxMln3KnnoOJJqj1I0kKe/FE6oBlUYB4pysxmaQ+FYMmGNX0WT dmbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4AXOThceM9HQTaD/1C2DnJHW4BibV/GSMKqCaOTg8gw=; b=dxP2t3dhkMg4B88pD6YwElEmma0kEfVtfDtaeq9joMQNmCLaiS6oH86qwsVGJlO7s1 xChg27fKBETc3KCg4hXKOn3zmDvik4epygeYHA5/NoHrAlwwgeexZSzsSgbmd7ml35aW 3UPZFmU+Vkaa4sZdyc7JBURoaIu1bu1j9yTwjtYlW1cshU4KrVttR9m3Xqt9/dmHKdgG sof8JP4NxfhYDIfCXBgjIY5ZdUEaoXQWbEhQ+2Wppmr+7CdQ+u8IAwd9ld9Rji9kY5KV 4asfNYzjbjNOjsdKP6OpREB71UWTBdWmV92NAhepM1gJR+/nZvKJqRKfxD0I017XXHs/ U+GA== X-Gm-Message-State: AOAM532c9YahzOlvcmdGII8kZD5sXiGYHjtumU5ACYKPGBXmzwxRAncG NejWT6AZzOYzmdA+EexxHvA= X-Google-Smtp-Source: ABdhPJwoXkLN/yki/p/Nl7rk5K8p4UfzLX1z4H8VqCpUnDmQzGXhiEVBIV01t81o4vGj/zpvUKz2og== X-Received: by 2002:a2e:9910:: with SMTP id v16mr14937280lji.213.1590443293740; Mon, 25 May 2020 14:48:13 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:13 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 03/16] rcu/tree: Repeat the monitor if any free channel is busy Date: Mon, 25 May 2020 23:47:47 +0200 Message-Id: <20200525214800.93072-4-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It is possible that one of the channels cannot be detached because its free channel is busy and previously queued data has not been processed yet. On the other hand, another channel can be successfully detached causing the monitor work to stop. Prevent that by rescheduling the monitor work if there are any channels in the pending state after a detach attempt. Fixes: 34c881745549e ("rcu: Support kfree_bulk() interface in kfree_rcu()") Acked-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d28010cf158b..ae21e1bddd3e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2971,7 +2971,7 @@ static void kfree_rcu_work(struct work_struct *work) static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) { struct kfree_rcu_cpu_work *krwp; - bool queued = false; + bool repeat = false; int i; lockdep_assert_held(&krcp->lock); @@ -3009,11 +3009,14 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) * been detached following each other, one by one. */ queue_rcu_work(system_wq, &krwp->rcu_work); - queued = true; } + + /* Repeat if any "free" corresponding channel is still busy. */ + if (krcp->bhead || krcp->head) + repeat = true; } - return queued; + return !repeat; } static inline void kfree_rcu_drain_unlock(struct kfree_rcu_cpu *krcp, From patchwork Mon May 25 21:47:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569471 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DDAAE913 for ; Mon, 25 May 2020 21:48:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AB039207CB for ; Mon, 25 May 2020 21:48:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HmwS9Hf5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB039207CB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 053ED80064; Mon, 25 May 2020 17:48:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F235280061; Mon, 25 May 2020 17:48:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4D9080064; Mon, 25 May 2020 17:48:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0081.hostedemail.com [216.40.44.81]) by kanga.kvack.org (Postfix) with ESMTP id AFA7A80061 for ; Mon, 25 May 2020 17:48:16 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6725740D7 for ; Mon, 25 May 2020 21:48:16 +0000 (UTC) X-FDA: 76856580192.14.ring93_2b17d61d6f014 X-Spam-Summary: 2,0,0,6c7193b478193fa3,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2907:3138:3139:3140:3141:3142:3354:3865:3866:3867:3870:3871:3872:4118:4321:4385:5007:6117:6119:6261:6653:7514:7576:7875:8660:9413:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13148:13149:13230:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:21740:21789:21939:21987:30003:30034:30054,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: ring93_2b17d61d6f014 X-Filterd-Recvd-Size: 7057 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:15 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id q2so22118226ljm.10 for ; Mon, 25 May 2020 14:48:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s+GYAs311TuxCys/vNhpiV48XtphGpyqwqArPir1vEA=; b=HmwS9Hf5tRNKvWpc7rChtGmNKH52OEpaRZEDcq2huMBobbMyLS3W1ri6fqOH6hgQQm dG8FebHg4aifOm/2Bg1ujIP8FaUkppUxbOB2jSwctvALRBMUoibtEF+HaPmPZPfedZsm oOVCM1JwKFjxgUm/qo9NvVy0Wp18aqk63ujpfc9n8eVcRZT4l7Hg50gRWZ/SdD9MygYu Ysol6j+BrzySzgXdYFG0LsAI+D8so936BDFRNUcMfzlu/tKyy64WhJWvLbfnK66KIw1Q PHG0Zzki3cI/PupjxDSg7HRujzu71cGxzTjnE4mhSNOVjAnOIgZYytfQQzW56ODvkLKK Sq9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s+GYAs311TuxCys/vNhpiV48XtphGpyqwqArPir1vEA=; b=gj8uKA4/wS5noWF3/7gOa8ikzMHNMzJ8IWbUrtzq8x7n2RmNzNLVCBq85/e13POAlc usLp1oovGDMuw7h+5XOHWymG25YDINtu5FNXFSJuhk0CmdoLXTp/A1pr69ZQ+9GCr0og vazj5FT56vZaDWzVtKiMg/oz8vGdwUIGodc7YsM+6zMD2c+uXVilfmdcuLGdYw11vL6l s0C1a731Alq/JAHVO9i2vr0M091X9otGuuCeHBYIoYnfpIvgcfdIodycnD35k3PfP/Ut p+3G6VuGSPjK0/YK8qbHO7Utqb90NPdu24jLEyCFxsGZ56Q4QhlMe3uhBsEzaO5mXe2r lCRA== X-Gm-Message-State: AOAM532HcHsKImlfVJon7pbKyb2fj4VuSk3ns/Erm/crWjojd3c7mlUB vUm+92TQgx1F89B6Bfgr4bg= X-Google-Smtp-Source: ABdhPJw5QAHid24LqXhonjXjabD4R74L3No12FclTCqfTQczerEPg5Pgt8IKgCuBPeR0GAgrgoM5Ow== X-Received: by 2002:a2e:9081:: with SMTP id l1mr15678344ljg.81.1590443294766; Mon, 25 May 2020 14:48:14 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:14 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 04/16] rcu/tree: Make debug_objects logic independent of rcu_head Date: Mon, 25 May 2020 23:47:48 +0200 Message-Id: <20200525214800.93072-5-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Joel Fernandes (Google)" kfree_rcu()'s debug_objects logic uses the address of the object's embedded rcu_head to queue/unqueue. Instead of this, make use of the object's address itself as preparation for future headless kfree_rcu() support. Reviewed-by: Uladzislau Rezki Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index ae21e1bddd3e..776ccf858154 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2836,13 +2836,11 @@ EXPORT_SYMBOL_GPL(call_rcu); * @nr_records: Number of active pointers in the array * @records: Array of the kfree_rcu() pointers * @next: Next bulk object in the block chain - * @head_free_debug: For debug, when CONFIG_DEBUG_OBJECTS_RCU_HEAD is set */ struct kfree_rcu_bulk_data { unsigned long nr_records; void *records[KFREE_BULK_MAX_ENTR]; struct kfree_rcu_bulk_data *next; - struct rcu_head *head_free_debug; }; /** @@ -2892,11 +2890,13 @@ struct kfree_rcu_cpu { static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc); static __always_inline void -debug_rcu_head_unqueue_bulk(struct rcu_head *head) +debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) { #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD - for (; head; head = head->next) - debug_rcu_head_unqueue(head); + int i; + + for (i = 0; i < bhead->nr_records; i++) + debug_rcu_head_unqueue((struct rcu_head *)(bhead->records[i])); #endif } @@ -2926,7 +2926,7 @@ static void kfree_rcu_work(struct work_struct *work) for (; bhead; bhead = bnext) { bnext = bhead->next; - debug_rcu_head_unqueue_bulk(bhead->head_free_debug); + debug_rcu_bhead_unqueue(bhead); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kfree_bulk_callback(rcu_state.name, @@ -2948,14 +2948,15 @@ static void kfree_rcu_work(struct work_struct *work) */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; + void *ptr = (void *)head - offset; next = head->next; - debug_rcu_head_unqueue(head); + debug_rcu_head_unqueue((struct rcu_head *)ptr); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset); if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) - kfree((void *)head - offset); + kfree(ptr); rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3094,18 +3095,11 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Initialize the new block. */ bnode->nr_records = 0; bnode->next = krcp->bhead; - bnode->head_free_debug = NULL; /* Attach it to the head. */ krcp->bhead = bnode; } -#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD - head->func = func; - head->next = krcp->bhead->head_free_debug; - krcp->bhead->head_free_debug = head; -#endif - /* Finally insert. */ krcp->bhead->records[krcp->bhead->nr_records++] = (void *) head - (unsigned long) func; @@ -3129,14 +3123,17 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; + void *ptr; local_irq_save(flags); // For safely calling this_cpu_ptr(). krcp = this_cpu_ptr(&krc); if (krcp->initialized) raw_spin_lock(&krcp->lock); + ptr = (void *)head - (unsigned long)func; + // Queue the object but don't yet schedule the batch. - if (debug_rcu_head_queue(head)) { + if (debug_rcu_head_queue(ptr)) { // Probable double kfree_rcu(), just leak. WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", __func__, head); From patchwork Mon May 25 21:47:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569473 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF96D90 for ; Mon, 25 May 2020 21:48:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9C6FA2075F for ; Mon, 25 May 2020 21:48:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nQAomHcf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9C6FA2075F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2506180065; Mon, 25 May 2020 17:48:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1FD7380061; Mon, 25 May 2020 17:48:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07F8A80065; Mon, 25 May 2020 17:48:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id E16CE80061 for ; Mon, 25 May 2020 17:48:17 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A682F180AD815 for ; Mon, 25 May 2020 21:48:17 +0000 (UTC) X-FDA: 76856580234.28.suit37_2b434ea4f7362 X-Spam-Summary: 10,1,0,8fa1663eb0108200,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:404:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2907:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3874:4321:4385:4390:4395:5007:6261:6653:7514:7875:8603:9413:9592:10004:11026:11473:11658:11914:12043:12297:12517:12519:12555:12895:12986:13069:13311:13357:13894:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21740:30034:30054,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: suit37_2b434ea4f7362 X-Filterd-Recvd-Size: 5081 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:17 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id m18so22152045ljo.5 for ; Mon, 25 May 2020 14:48:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZQqXHmyUSlSMzrEScq/mc28bFFncHq5WVrswvpeHH5c=; b=nQAomHcfsNoRankxf3s+HeOEHNCHu2KmuSmpUalABmcCQtLqexcSQ3HblvcZtsuaF4 otVr2oDxWeVWUW5VOoQ2++yly0jQXYqjKZdL8iwzqlLqnci24JZ7HYKlwLwIuJGpeMGt JEqpfJKb2UHMs5+ON23CHKijnJJ2/L00Iit2ge7oba8fvFLYW1tInnxhyLssBi39T00m qi/iRX5hCt9qeOBqpcEyZFAuSuD4wbHSMCrv9jW4p3XXI+JoqECU4N39O7fBLPb8QcuO rtJbdTeEu49DYmRRX9Pp6Q0/kkZjh2qa3kaaxZ3FMJ5Ddxg1eF0jzv/99t023HnWMGHQ mRUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZQqXHmyUSlSMzrEScq/mc28bFFncHq5WVrswvpeHH5c=; b=Xn1yvb1JFiTkXpEp6RQfS+WzrqKhBeF76Qx/zYuQIJfD8G8rQCK3VphXYiIT0I0tkD aB0LtK7Xu2336bPK7cjlLwV55TLhYmIjdYyI2v0k5Q0+5oImAJpYPvo81XiIE8jlIGiR 2z1YKcX8591ZQ8HhYgwYxYT9bVlpC911TNwDyA7D86tyMf02PASUtFkpYk9E40iX0osO CVgDewGySfNjpNRByRbbnwwDuV/sSc6CdHN3idK+L5GuVG00tqIwh64WC9PU9lPqG04N XuCDEj3g0VXeMbon+ivi4eTojeUWu1bfnBrCZcKWmNjzBwEDpTnHuWsyd5ePpSe9yuem z60Q== X-Gm-Message-State: AOAM531svexqfrvd3dy3y7cyyibLjWMtYAXzF7TKvem797AvQ2Urj/4Y 5PJBcPiT4L/zy/4OXFPfuLU= X-Google-Smtp-Source: ABdhPJynkTSq/4AdeYM1T/xp9OPOdZ+/e3MJPy/AjzsKPtpdjsgFusPDyaoa2eBXUB0sJQXz2faVew== X-Received: by 2002:a05:651c:209:: with SMTP id y9mr13605918ljn.408.1590443295981; Mon, 25 May 2020 14:48:15 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:15 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko , Boqun Feng Subject: [PATCH v2 05/16] rcu/tree: Simplify KFREE_BULK_MAX_ENTR macro Date: Mon, 25 May 2020 23:47:49 +0200 Message-Id: <20200525214800.93072-6-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can simplify KFREE_BULK_MAX_ENTR macro and get rid of magic numbers which were used to make the structure to be exactly one page. Suggested-by: Boqun Feng Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 776ccf858154..6501fbcae3c7 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2824,13 +2824,6 @@ EXPORT_SYMBOL_GPL(call_rcu); #define KFREE_DRAIN_JIFFIES (HZ / 50) #define KFREE_N_BATCHES 2 -/* - * This macro defines how many entries the "records" array - * will contain. It is based on the fact that the size of - * kfree_rcu_bulk_data structure becomes exactly one page. - */ -#define KFREE_BULK_MAX_ENTR ((PAGE_SIZE / sizeof(void *)) - 3) - /** * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers * @nr_records: Number of active pointers in the array @@ -2839,10 +2832,18 @@ EXPORT_SYMBOL_GPL(call_rcu); */ struct kfree_rcu_bulk_data { unsigned long nr_records; - void *records[KFREE_BULK_MAX_ENTR]; struct kfree_rcu_bulk_data *next; + void *records[]; }; +/* + * This macro defines how many entries the "records" array + * will contain. It is based on the fact that the size of + * kfree_rcu_bulk_data structure becomes exactly one page. + */ +#define KFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *)) + /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period From patchwork Mon May 25 21:47:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569475 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B5BF913 for ; Mon, 25 May 2020 21:48:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4E7C3206C3 for ; Mon, 25 May 2020 21:48:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MiYel4cr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E7C3206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 38D2780066; Mon, 25 May 2020 17:48:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2275880061; Mon, 25 May 2020 17:48:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C74D80066; Mon, 25 May 2020 17:48:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0204.hostedemail.com [216.40.44.204]) by kanga.kvack.org (Postfix) with ESMTP id D6D2280061 for ; Mon, 25 May 2020 17:48:18 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 909812C32 for ; Mon, 25 May 2020 21:48:18 +0000 (UTC) X-FDA: 76856580276.11.form98_2b6bb832ecd20 X-Spam-Summary: 2,0,0,909084c594c08379,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2693:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3872:3874:4321:4385:4390:4395:5007:6117:6119:6261:6653:7514:9413:9592:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13255:13894:14096:14110:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21740:21990:30054:30070,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: form98_2b6bb832ecd20 X-Filterd-Recvd-Size: 5246 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:18 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id l15so21806164lje.9 for ; Mon, 25 May 2020 14:48:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=myZbGzUvCLDF53laDPj5mc/dKuerQ91BFIMGTCX8KTY=; b=MiYel4crknK5dXCl5GHpc6bBf3Hel4UK5iJXpbbn3FNgSjsxMB1Mq5QBrlw9Dm0oKt p5pJ/J6P3q28DJM/MHq/cL40IA1JX42PUYLbPaUa4l16QNFz/R9GXie4lY5P31QrVdyc U7qEY+zsEkwxvJvZKg2h3Qk5wwTiIyseRKQ+fyscqBk0nerwsTHcRG0rzs/XaAdw9cFE fcQTGKZLjzeowQ1ntpUs8Lk3cv02BJw9afWKbJxYVpCQ5OJ6i/SxwVWLDBF/0bxRFY1Z Hb+Ylu/piIEoG9HTZX1ceVG/SFPuFWQOFTYL3RG/AHpMqgDV/6b/YFyIu+6fowY1+TJV gBuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=myZbGzUvCLDF53laDPj5mc/dKuerQ91BFIMGTCX8KTY=; b=XUdjNSFfMmQ4qElIgBpe0HgJ7pF8MBQW/TGXBnadqWnd3q/MobSoFdN46f5Q8sK4D4 lyVFnFRx1or67zTsJJ4xiRV7rx1m2j2d7SaYsy0rmfQcXy2dZU0hSTVgfG0+XlC+6IZD T/M741Db8cE+tgmpJPXtH9HujghRjzcTSuoEjiLQBBotQYn5Yvc0c8eaezU4qSagzhDN V6sVkM4Pvu84E9TDvvmDGMcKYBVoM1sGFWhpnFOW9Na+0AO7WdnA+8KT1dsNjUvQNns8 GZaJo+W0PQGbyrMeOVrrImw1vAo04eOqrmTA/cZJxP99CUir5vsn/LbLJedmFwGRH84D jpIw== X-Gm-Message-State: AOAM532gVrCzITbmC4cpUl+IhzJ1ErJC8bk4eqhL5drqYFEnZ7YjdQ7L iwZWSRQkSYnHmWOH/1fAxT4= X-Google-Smtp-Source: ABdhPJwSIz48TxBYILx85UNtNxrSRT7oQBbn/6kX4IFkWPacY0Y7bMHszbNeQvbgXRgnvoSyEX311Q== X-Received: by 2002:a2e:7d19:: with SMTP id y25mr14028058ljc.255.1590443297064; Mon, 25 May 2020 14:48:17 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:16 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 06/16] rcu/tree: Move kfree_rcu_cpu locking/unlocking to separate functions Date: Mon, 25 May 2020 23:47:50 +0200 Message-Id: <20200525214800.93072-7-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce helpers to lock and unlock per-cpu "kfree_rcu_cpu" structures. That will make kfree_call_rcu() more readable and prevent programming errors. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 6501fbcae3c7..c1f4b740cf14 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2901,6 +2901,27 @@ debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) #endif } +static inline struct kfree_rcu_cpu * +krc_this_cpu_lock(unsigned long *flags) +{ + struct kfree_rcu_cpu *krcp; + + local_irq_save(*flags); // For safely calling this_cpu_ptr(). + krcp = this_cpu_ptr(&krc); + if (likely(krcp->initialized)) + raw_spin_lock(&krcp->lock); + + return krcp; +} + +static inline void +krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) +{ + if (likely(krcp->initialized)) + raw_spin_unlock(&krcp->lock); + local_irq_restore(flags); +} + /* * This function is invoked in workqueue context after a grace period. * It frees all the objects queued on ->bhead_free or ->head_free. @@ -3126,11 +3147,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) struct kfree_rcu_cpu *krcp; void *ptr; - local_irq_save(flags); // For safely calling this_cpu_ptr(). - krcp = this_cpu_ptr(&krc); - if (krcp->initialized) - raw_spin_lock(&krcp->lock); - + krcp = krc_this_cpu_lock(&flags); ptr = (void *)head - (unsigned long)func; // Queue the object but don't yet schedule the batch. @@ -3161,9 +3178,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) } unlock_return: - if (krcp->initialized) - raw_spin_unlock(&krcp->lock); - local_irq_restore(flags); + krc_this_cpu_unlock(krcp, flags); } EXPORT_SYMBOL_GPL(kfree_call_rcu); From patchwork Mon May 25 21:47:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569477 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3779F90 for ; Mon, 25 May 2020 21:48:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 05DB6206C3 for ; Mon, 25 May 2020 21:48:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BK3e8aoE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 05DB6206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6967680067; Mon, 25 May 2020 17:48:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6471680061; Mon, 25 May 2020 17:48:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BD3280067; Mon, 25 May 2020 17:48:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0105.hostedemail.com [216.40.44.105]) by kanga.kvack.org (Postfix) with ESMTP id 2ED0680061 for ; Mon, 25 May 2020 17:48:20 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E5273247A for ; Mon, 25 May 2020 21:48:19 +0000 (UTC) X-FDA: 76856580318.19.beds15_2b97eff319906 X-Spam-Summary: 2,0,0,fcf1b8d8871764b1,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2559:2562:2901:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4117:4250:4321:4385:5007:6117:6261:6653:7514:7576:9413:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12679:12895:13161:13229:13868:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:21740:21990:30034:30054:30070,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: beds15_2b97eff319906 X-Filterd-Recvd-Size: 6104 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:19 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id z13so12486756ljn.7 for ; Mon, 25 May 2020 14:48:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kkR7cKj00cDqsyl0thgtTAFScjKfcm7Y5SAgEdcKqZ8=; b=BK3e8aoEvFZvx2GLo6YbO7z9wba+ajsVH8bD0AIo5tBZKiiEdOvmUvXq1D72AQ3D0U BhAlxjuRUL0CkNAeGcjpL4A1sZb0zuAKogD4IV+qMo8itIJaDnveoNdZrbyDAzUsuGf1 ebEvHFiSdY70uJGwcfa7Y6NiawaHUmMVWj23IsqFU6dkXo/KHv7c4eWI7w7F4PLvTho3 ZnsJqv5KSYuUZUiur7lq8L5TSZ3dbhXetGiF1t7zsYYYzBSgZITwMw2g6ubCJNVoSYO3 JE6G50JexbrP95PxftqxaEKorF1thCAaidKm6BcoLb0Q+t4Hvb/2rBneVIQXVG2dsyG1 Iijw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kkR7cKj00cDqsyl0thgtTAFScjKfcm7Y5SAgEdcKqZ8=; b=gHQMEc1GqxS6XU+M7uLU8JsytZIx5j5KCAMtvdvWJNRxvhC+UmtkIAlxRWTgATnVln Lsqv73w12Rp9ooABCWP33KYL1o/Mb+m3ZuWtY21EyaUlPpSrDB4V8psQxwBbBbyf92Q/ ao1s9tvOVSovAbjPrP66tqN8w2nBVYDQHeso8Eq9qwScoK7TkgIJjQEaQgmFO8lOFiai qHds46gbjdcBKIQgBt6nv2pULPKu4/C3YEZW7aW4iu/eWnWHgfbM/tY69aZ5xoaOuYV7 SPxK6FInmcWqsQAYI8pLFuYuI5/wbry0ixemduSwaniYWtPdUH2uBj8Rm0n+d43E3Rsi CU1A== X-Gm-Message-State: AOAM531j9IVYPQLqjh9TnQvJbn2TYeHSwQDf5DPOl7SHXd555gKUX7iO GlkuEg40YZqkPHT4hGQEC/U= X-Google-Smtp-Source: ABdhPJxD27sFpaUv8BMMjBCnJmu9ZyHMVn095tD4CKJcg5exgNu/c6OWeJE4ZYEjBTu5Q8PK/El+zw== X-Received: by 2002:a05:651c:200c:: with SMTP id s12mr13228027ljo.458.1590443298228; Mon, 25 May 2020 14:48:18 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:17 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko , Sebastian Andrzej Siewior Subject: [PATCH v2 07/16] rcu/tree: Use static initializer for krc.lock Date: Mon, 25 May 2020 23:47:51 +0200 Message-Id: <20200525214800.93072-8-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Sebastian Andrzej Siewior The per-CPU variable is initialized at runtime in kfree_rcu_batch_init(). This function is invoked before 'rcu_scheduler_active' is set to 'RCU_SCHEDULER_RUNNING'. After the initialisation, '->initialized' is to true. The raw_spin_lock is only acquired if '->initialized' is set to true. The worqueue item is only used if 'rcu_scheduler_active' set to RCU_SCHEDULER_RUNNING which happens after initialisation. Use a static initializer for krc.lock and remove the runtime initialisation of the lock. Since the lock can now be always acquired, remove the '->initialized' check. Cc: Sebastian Andrzej Siewior Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tree.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index c1f4b740cf14..4b1710c1d8f6 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2868,7 +2868,7 @@ struct kfree_rcu_cpu_work { * @lock: Synchronize access to this structure * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES * @monitor_todo: Tracks whether a @monitor_work delayed work is pending - * @initialized: The @lock and @rcu_work fields have been initialized + * @initialized: The @rcu_work fields have been initialized * @count: Number of objects for which GP not started * * This is a per-CPU structure. The reason that it is not included in @@ -2888,7 +2888,9 @@ struct kfree_rcu_cpu { int count; }; -static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc); +static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { + .lock = __RAW_SPIN_LOCK_UNLOCKED(krc.lock), +}; static __always_inline void debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) @@ -2908,8 +2910,7 @@ krc_this_cpu_lock(unsigned long *flags) local_irq_save(*flags); // For safely calling this_cpu_ptr(). krcp = this_cpu_ptr(&krc); - if (likely(krcp->initialized)) - raw_spin_lock(&krcp->lock); + raw_spin_lock(&krcp->lock); return krcp; } @@ -2917,8 +2918,7 @@ krc_this_cpu_lock(unsigned long *flags) static inline void krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) { - if (likely(krcp->initialized)) - raw_spin_unlock(&krcp->lock); + raw_spin_unlock(&krcp->lock); local_irq_restore(flags); } @@ -4141,7 +4141,6 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); - raw_spin_lock_init(&krcp->lock); for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; From patchwork Mon May 25 21:47:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569479 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 37FD590 for ; Mon, 25 May 2020 21:48:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ECEEC206C3 for ; Mon, 25 May 2020 21:48:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mE47/kwi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ECEEC206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 92DB580068; Mon, 25 May 2020 17:48:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 892F480061; Mon, 25 May 2020 17:48:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 756C680068; Mon, 25 May 2020 17:48:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0172.hostedemail.com [216.40.44.172]) by kanga.kvack.org (Postfix) with ESMTP id 5D0A480061 for ; Mon, 25 May 2020 17:48:21 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 200FA181AC9CB for ; Mon, 25 May 2020 21:48:21 +0000 (UTC) X-FDA: 76856580402.21.vein54_2bc448f26634f X-Spam-Summary: 2,0,0,902cac3f02b06947,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:2:41:355:379:541:560:800:960:966:973:981:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:1801:2196:2198:2199:2200:2393:2559:2562:2693:2731:2901:2902:2907:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4049:4120:4250:4321:4385:4605:5007:6117:6119:6261:6653:7514:7875:7903:8603:9010:9040:9413:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13141:13161:13229:13230:13894:14394:14687:21080:21324:21433:21444:21451:21627:21666:21740:21939:21987:21990:30012:30034:30045:30054:30070,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:28,LUA_SUMMARY:none X-HE-Tag: vein54_2bc448f26634f X-Filterd-Recvd-Size: 9285 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:20 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id o14so22215727ljp.4 for ; Mon, 25 May 2020 14:48:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lE2u2kHS4No1brUupiLe8POVLpxEzHQVFsb+F2qAz+U=; b=mE47/kwiSbkYd41CBx5mG2U+/+3o4+EafOVHvNDHcZlUcNROQrO9SmV490BZ2yU4Zv zEimBNytZVO9xBI9ZAO3GTTIGxnCnzhXACeLE/7qZ4/kwZ/bLsFSj1W0mdusKeeHu1Yr 8G4agwb49X0+GpT+9THihZQDNVHp9/nfQarLlhOmhrc9Q/Mw1gzrEypJ2UI2Hh42AecR cnErb4qbgdAwkL/BfB/dPNy83y/kpdHbttDmJruEs4qPuZalMjTWI6R2oCS1GO8gn05j NhpZ7Sez5kjxUU/kvjQSy73y6uZZsF39QXJTZ5rYU1CsA8tT62qWRClNBE7l1JwSPiCU AS8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lE2u2kHS4No1brUupiLe8POVLpxEzHQVFsb+F2qAz+U=; b=jBENkUdpcUztpE82DKDYBH2gkCvBGqUCtC+/dbMWDpfxjK4cAMpYBykb7FQhc9+xp+ 7MP9XwrPp8301KzPgzfSCLQr87yeTYeOEmcwXqNsqFPNSY5jsxKZacuxs+6acOjjqcyu quL6bRb2xEriDqSosFiD0MMbedh2qTxjZ8aUk/JP+YTZNHSl8BmbxdHl/x3yGHkmxHqH djiq0cfwXJyrMkV6NBSAe7v99fPjKN9dix7SHWkfQtp5Y4nBVADC4LNeEvwbbMcISpiN wCDW8CXttG2WaulY+vt2EXsNN+7cR1Y53XJVd7CoIcIXmQHRz7B7DjfOicSSIU5FHoZF N5Fw== X-Gm-Message-State: AOAM531AehqqNRniKmg16g8LqtYXUgMOXeeTg7wrQZ5XF+R1z8mk6ooy clVQcjo2NrTXNQ6pBxWFfA8= X-Google-Smtp-Source: ABdhPJzFTvgrYh2DM1zyUK1I1YUtgh7ZibMe7YtuD8sPpWEUShk9Ty1z/PVNpVJRpOslAXlqcLY1Nw== X-Received: by 2002:a2e:6e17:: with SMTP id j23mr14537670ljc.106.1590443299340; Mon, 25 May 2020 14:48:19 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:18 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 08/16] rcu/tree: cache specified number of objects Date: Mon, 25 May 2020 23:47:52 +0200 Message-Id: <20200525214800.93072-9-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to reduce the dynamic need for pages in kfree_rcu(), pre-allocate a configurable number of pages per CPU and link them in a list. When kfree_rcu() reclaims objects, the object's container page is cached into a list instead of being released to the low-level page allocator. Such an approach provides O(1) access to free pages while also reducing the number of requests to the page allocator. It also makes the kfree_rcu() code to have free pages available during a low memory condition. A read-only sysfs parameter (rcu_min_cached_objs) reflects the minimum number of allowed cached pages per CPU. Signed-off-by: Uladzislau Rezki (Sony) --- .../admin-guide/kernel-parameters.txt | 8 +++ kernel/rcu/tree.c | 66 +++++++++++++++++-- 2 files changed, 70 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index ea4228779c28..b90af44ee81d 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3977,6 +3977,14 @@ latencies, which will choose a value aligned with the appropriate hardware boundaries. + rcutree.rcu_min_cached_objs= [KNL] + Minimum number of objects which are cached and + maintained per one CPU. Object size is equal + to PAGE_SIZE. The cache allows to reduce the + pressure to page allocator, also it makes the + whole algorithm to behave better in low memory + condition. + rcutree.jiffies_till_first_fqs= [KNL] Set delay from grace-period initialization to first attempt to force quiescent states. diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 4b1710c1d8f6..e2267e92de5d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -167,6 +167,15 @@ module_param(gp_init_delay, int, 0444); static int gp_cleanup_delay; module_param(gp_cleanup_delay, int, 0444); +/* + * This rcu parameter is runtime-read-only. It reflects + * a minimum allowed number of objects which can be cached + * per-CPU. Object size is equal to one page. This value + * can be changed at boot time. + */ +static int rcu_min_cached_objs = 2; +module_param(rcu_min_cached_objs, int, 0444); + /* Retrieve RCU kthreads priority for rcutorture */ int rcu_get_gp_kthreads_prio(void) { @@ -2863,7 +2872,6 @@ struct kfree_rcu_cpu_work { * struct kfree_rcu_cpu - batch up kfree_rcu() requests for RCU grace period * @head: List of kfree_rcu() objects not yet waiting for a grace period * @bhead: Bulk-List of kfree_rcu() objects not yet waiting for a grace period - * @bcached: Keeps at most one object for later reuse when build chain blocks * @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period * @lock: Synchronize access to this structure * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES @@ -2879,13 +2887,22 @@ struct kfree_rcu_cpu_work { struct kfree_rcu_cpu { struct rcu_head *head; struct kfree_rcu_bulk_data *bhead; - struct kfree_rcu_bulk_data *bcached; struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; raw_spinlock_t lock; struct delayed_work monitor_work; bool monitor_todo; bool initialized; int count; + + /* + * A simple cache list that contains objects for + * reuse purpose. In order to save some per-cpu + * space the list is singular. Even though it is + * lockless an access has to be protected by the + * per-cpu lock. + */ + struct llist_head bkvcache; + int nr_bkv_objs; }; static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { @@ -2922,6 +2939,31 @@ krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) local_irq_restore(flags); } +static inline struct kfree_rcu_bulk_data * +get_cached_bnode(struct kfree_rcu_cpu *krcp) +{ + if (!krcp->nr_bkv_objs) + return NULL; + + krcp->nr_bkv_objs--; + return (struct kfree_rcu_bulk_data *) + llist_del_first(&krcp->bkvcache); +} + +static inline bool +put_cached_bnode(struct kfree_rcu_cpu *krcp, + struct kfree_rcu_bulk_data *bnode) +{ + // Check the limit. + if (krcp->nr_bkv_objs >= rcu_min_cached_objs) + return false; + + llist_add((struct llist_node *) bnode, &krcp->bkvcache); + krcp->nr_bkv_objs++; + return true; + +} + /* * This function is invoked in workqueue context after a grace period. * It frees all the objects queued on ->bhead_free or ->head_free. @@ -2957,7 +2999,12 @@ static void kfree_rcu_work(struct work_struct *work) kfree_bulk(bhead->nr_records, bhead->records); rcu_lock_release(&rcu_callback_map); - if (cmpxchg(&krcp->bcached, NULL, bhead)) + krcp = krc_this_cpu_lock(&flags); + if (put_cached_bnode(krcp, bhead)) + bhead = NULL; + krc_this_cpu_unlock(krcp, flags); + + if (bhead) free_page((unsigned long) bhead); cond_resched_tasks_rcu_qs(); @@ -3090,7 +3137,7 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Check if a new block is required. */ if (!krcp->bhead || krcp->bhead->nr_records == KFREE_BULK_MAX_ENTR) { - bnode = xchg(&krcp->bcached, NULL); + bnode = get_cached_bnode(krcp); if (!bnode) { WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE); @@ -4140,12 +4187,23 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); + struct kfree_rcu_bulk_data *bnode; for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; } + for (i = 0; i < rcu_min_cached_objs; i++) { + bnode = (struct kfree_rcu_bulk_data *) + __get_free_page(GFP_NOWAIT | __GFP_NOWARN); + + if (bnode) + put_cached_bnode(krcp, bnode); + else + pr_err("Failed to preallocate for %d CPU!\n", cpu); + } + INIT_DELAYED_WORK(&krcp->monitor_work, kfree_rcu_monitor); krcp->initialized = true; } From patchwork Mon May 25 21:47:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569481 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42FE490 for ; Mon, 25 May 2020 21:48:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EB17020825 for ; Mon, 25 May 2020 21:48:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PqbGoUsf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EB17020825 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E604380069; Mon, 25 May 2020 17:48:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CFB1480061; Mon, 25 May 2020 17:48:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B77CB80069; Mon, 25 May 2020 17:48:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0216.hostedemail.com [216.40.44.216]) by kanga.kvack.org (Postfix) with ESMTP id 91BA980061 for ; Mon, 25 May 2020 17:48:22 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5FA5F824556B for ; Mon, 25 May 2020 21:48:22 +0000 (UTC) X-FDA: 76856580444.11.seed97_2bef471ce5852 X-Spam-Summary: 2,0,0,a2919ef2cbeabbdc,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:4:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2640:2693:2901:2907:2914:3138:3139:3140:3141:3142:3622:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:4385:5007:6117:6119:6261:6653:7514:7875:7903:7904:8603:9010:9413:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13141:13149:13172:13229:13230:13868:13894:14394:14687:21080:21324:21433:21444:21451:21627:21666:21740:21789:21987:21990:30034:30045:30054:30070:30091,0,RBL:209.85.167.68:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: seed97_2bef471ce5852 X-Filterd-Recvd-Size: 17515 Received: from mail-lf1-f68.google.com (mail-lf1-f68.google.com [209.85.167.68]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:21 +0000 (UTC) Received: by mail-lf1-f68.google.com with SMTP id d7so3273lfi.12 for ; Mon, 25 May 2020 14:48:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/NZXEBvFgqE4Wkvf8ZeZARJ+cS4M8PJVq9nwIXxQ0EY=; b=PqbGoUsfP53JPVyJbNCBCU2/RYZgNV03EwWBNTWIhd2gwQWB5A5gxwDPVKK/X9f1G/ f+ZxQFsYsEUPmqOVuqt8sjh7ZVxp3G89/n0NFoaCXQGh+SRzlC0kjtjQdTWx+lKMDX2j XvaZoDnQ320v1N5xAfz5rFAAg/MdxCRHS5DQshtRoEzT/hQVnSNuEjl8SPxjxpWxjax3 WIlg8TOLzjgeD4Vs8rlPHh21aelJAEUH7fcvJMFTJSSzrjUS2igZ3/7bQDUUQ/1drE67 KUANxR6nnT6IhYDwY/hjZMhWljb7kOES71oMa1XakrsXO7EZnjUkpoZfD5tLqm2LDXfe SwJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/NZXEBvFgqE4Wkvf8ZeZARJ+cS4M8PJVq9nwIXxQ0EY=; b=Et2uPCaLz+gcSPlTfUDHtamqwvnW7HTRJUJ6jjUmq1gnbnLa79QN35AkvVzndIraVz n+1FqEZOrkW8/ap0fjMbXINfHQOm2X9rtpLD5U3bMRy35T+xw7GQZ0qdaEek9nDjJwVA 1chk2vpKGTrPJB1xW1FtjHbh4NhZJd+udiFa64UYvKKB+nfQsIzIWCIq1Z08QRIVmkQk i3YP8pUDviK1aceZH4Bu0Ed1Q4vnAsaUAC1hTaNmD0/l9DeGMYqbwy4ucwXv9tlpkH+4 pP44tjstwidxd2GSbHWjeF9qASUdhludH+CvCvQzcOaRsqJ5+GCIv1dIcnAwCAublzWu rkPg== X-Gm-Message-State: AOAM532LSTJenD0nyLZ5dh8i5k+ygSJ0JjM5SIrFs8bcHLKnHfLGjygz J/Ie9ubZJ2N5mt9Kj8hlIik= X-Google-Smtp-Source: ABdhPJyEUtTbyAaRyaYhRXm4Pd/ECEY2+Pp7IhWI++b2yxy5y52RlKPacVXnk/qcaKBAUocojN6LuA== X-Received: by 2002:a19:f813:: with SMTP id a19mr15173962lff.212.1590443300392; Mon, 25 May 2020 14:48:20 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:19 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 09/16] rcu/tree: Maintain separate array for vmalloc ptrs Date: Mon, 25 May 2020 23:47:53 +0200 Message-Id: <20200525214800.93072-10-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To do so, we use an array of kvfree_rcu_bulk_data structures. It consists of two elements: - index number 0 corresponds to slab pointers. - index number 1 corresponds to vmalloc pointers. Keeping vmalloc pointers separated from slab pointers makes it possible to invoke the right freeing API for the right kind of pointer. It also prepares us for future headless support for vmalloc and SLAB objects. Such objects cannot be queued on a linked list and are instead directly into an array. Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) Reviewed-by: Joel Fernandes (Google) Co-developed-by: Joel Fernandes (Google) Signed-off-by: Matthew Wilcox (Oracle) --- kernel/rcu/tree.c | 173 +++++++++++++++++++++++++++------------------- 1 file changed, 100 insertions(+), 73 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e2267e92de5d..9f84ff80bc25 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -57,6 +57,8 @@ #include #include #include +#include +#include #include "../time/tick-internal.h" #include "tree.h" @@ -2832,46 +2834,47 @@ EXPORT_SYMBOL_GPL(call_rcu); /* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (HZ / 50) #define KFREE_N_BATCHES 2 +#define FREE_N_CHANNELS 2 /** - * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers + * struct kvfree_rcu_bulk_data - single block to store kvfree_rcu() pointers * @nr_records: Number of active pointers in the array - * @records: Array of the kfree_rcu() pointers * @next: Next bulk object in the block chain + * @records: Array of the kvfree_rcu() pointers */ -struct kfree_rcu_bulk_data { +struct kvfree_rcu_bulk_data { unsigned long nr_records; - struct kfree_rcu_bulk_data *next; + struct kvfree_rcu_bulk_data *next; void *records[]; }; /* * This macro defines how many entries the "records" array * will contain. It is based on the fact that the size of - * kfree_rcu_bulk_data structure becomes exactly one page. + * kvfree_rcu_bulk_data structure becomes exactly one page. */ -#define KFREE_BULK_MAX_ENTR \ - ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *)) +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period * @head_free: List of kfree_rcu() objects waiting for a grace period - * @bhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period + * @bkvhead_free: Bulk-List of kvfree_rcu() objects waiting for a grace period * @krcp: Pointer to @kfree_rcu_cpu structure */ struct kfree_rcu_cpu_work { struct rcu_work rcu_work; struct rcu_head *head_free; - struct kfree_rcu_bulk_data *bhead_free; + struct kvfree_rcu_bulk_data *bkvhead_free[FREE_N_CHANNELS]; struct kfree_rcu_cpu *krcp; }; /** * struct kfree_rcu_cpu - batch up kfree_rcu() requests for RCU grace period * @head: List of kfree_rcu() objects not yet waiting for a grace period - * @bhead: Bulk-List of kfree_rcu() objects not yet waiting for a grace period + * @bkvhead: Bulk-List of kvfree_rcu() objects not yet waiting for a grace period * @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period * @lock: Synchronize access to this structure * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES @@ -2886,7 +2889,7 @@ struct kfree_rcu_cpu_work { */ struct kfree_rcu_cpu { struct rcu_head *head; - struct kfree_rcu_bulk_data *bhead; + struct kvfree_rcu_bulk_data *bkvhead[FREE_N_CHANNELS]; struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; raw_spinlock_t lock; struct delayed_work monitor_work; @@ -2910,7 +2913,7 @@ static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc) = { }; static __always_inline void -debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) +debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead) { #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD int i; @@ -2939,20 +2942,20 @@ krc_this_cpu_unlock(struct kfree_rcu_cpu *krcp, unsigned long flags) local_irq_restore(flags); } -static inline struct kfree_rcu_bulk_data * +static inline struct kvfree_rcu_bulk_data * get_cached_bnode(struct kfree_rcu_cpu *krcp) { if (!krcp->nr_bkv_objs) return NULL; krcp->nr_bkv_objs--; - return (struct kfree_rcu_bulk_data *) + return (struct kvfree_rcu_bulk_data *) llist_del_first(&krcp->bkvcache); } static inline bool put_cached_bnode(struct kfree_rcu_cpu *krcp, - struct kfree_rcu_bulk_data *bnode) + struct kvfree_rcu_bulk_data *bnode) { // Check the limit. if (krcp->nr_bkv_objs >= rcu_min_cached_objs) @@ -2971,43 +2974,63 @@ put_cached_bnode(struct kfree_rcu_cpu *krcp, static void kfree_rcu_work(struct work_struct *work) { unsigned long flags; + struct kvfree_rcu_bulk_data *bkvhead[FREE_N_CHANNELS], *bnext; struct rcu_head *head, *next; - struct kfree_rcu_bulk_data *bhead, *bnext; struct kfree_rcu_cpu *krcp; struct kfree_rcu_cpu_work *krwp; + int i, j; krwp = container_of(to_rcu_work(work), struct kfree_rcu_cpu_work, rcu_work); krcp = krwp->krcp; + raw_spin_lock_irqsave(&krcp->lock, flags); + // Channels 1 and 2. + for (i = 0; i < FREE_N_CHANNELS; i++) { + bkvhead[i] = krwp->bkvhead_free[i]; + krwp->bkvhead_free[i] = NULL; + } + + // Channel 3. head = krwp->head_free; krwp->head_free = NULL; - bhead = krwp->bhead_free; - krwp->bhead_free = NULL; raw_spin_unlock_irqrestore(&krcp->lock, flags); - /* "bhead" is now private, so traverse locklessly. */ - for (; bhead; bhead = bnext) { - bnext = bhead->next; - - debug_rcu_bhead_unqueue(bhead); - - rcu_lock_acquire(&rcu_callback_map); - trace_rcu_invoke_kfree_bulk_callback(rcu_state.name, - bhead->nr_records, bhead->records); - - kfree_bulk(bhead->nr_records, bhead->records); - rcu_lock_release(&rcu_callback_map); + // Handle two first channels. + for (i = 0; i < FREE_N_CHANNELS; i++) { + for (; bkvhead[i]; bkvhead[i] = bnext) { + bnext = bkvhead[i]->next; + debug_rcu_bhead_unqueue(bkvhead[i]); + + rcu_lock_acquire(&rcu_callback_map); + if (i == 0) { // kmalloc() / kfree(). + trace_rcu_invoke_kfree_bulk_callback( + rcu_state.name, bkvhead[i]->nr_records, + bkvhead[i]->records); + + kfree_bulk(bkvhead[i]->nr_records, + bkvhead[i]->records); + } else { // vmalloc() / vfree(). + for (j = 0; j < bkvhead[i]->nr_records; j++) { + trace_rcu_invoke_kfree_callback( + rcu_state.name, + bkvhead[i]->records[j], 0); + + vfree(bkvhead[i]->records[j]); + } + } + rcu_lock_release(&rcu_callback_map); - krcp = krc_this_cpu_lock(&flags); - if (put_cached_bnode(krcp, bhead)) - bhead = NULL; - krc_this_cpu_unlock(krcp, flags); + krcp = krc_this_cpu_lock(&flags); + if (put_cached_bnode(krcp, bkvhead[i])) + bkvhead[i] = NULL; + krc_this_cpu_unlock(krcp, flags); - if (bhead) - free_page((unsigned long) bhead); + if (bkvhead[i]) + free_page((unsigned long) bkvhead[i]); - cond_resched_tasks_rcu_qs(); + cond_resched_tasks_rcu_qs(); + } } /* @@ -3025,7 +3048,7 @@ static void kfree_rcu_work(struct work_struct *work) trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset); if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) - kfree(ptr); + kvfree(ptr); rcu_lock_release(&rcu_callback_map); cond_resched_tasks_rcu_qs(); @@ -3042,7 +3065,7 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) { struct kfree_rcu_cpu_work *krwp; bool repeat = false; - int i; + int i, j; lockdep_assert_held(&krcp->lock); @@ -3050,21 +3073,25 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) krwp = &(krcp->krw_arr[i]); /* - * Try to detach bhead or head and attach it over any + * Try to detach bkvhead or head and attach it over any * available corresponding free channel. It can be that * a previous RCU batch is in progress, it means that * immediately to queue another one is not possible so * return false to tell caller to retry. */ - if ((krcp->bhead && !krwp->bhead_free) || + if ((krcp->bkvhead[0] && !krwp->bkvhead_free[0]) || + (krcp->bkvhead[1] && !krwp->bkvhead_free[1]) || (krcp->head && !krwp->head_free)) { - /* Channel 1. */ - if (!krwp->bhead_free) { - krwp->bhead_free = krcp->bhead; - krcp->bhead = NULL; + // Channel 1 corresponds to SLAB ptrs. + // Channel 2 corresponds to vmalloc ptrs. + for (j = 0; j < FREE_N_CHANNELS; j++) { + if (!krwp->bkvhead_free[j]) { + krwp->bkvhead_free[j] = krcp->bkvhead[j]; + krcp->bkvhead[j] = NULL; + } } - /* Channel 2. */ + // Channel 3 corresponds to emergency path. if (!krwp->head_free) { krwp->head_free = krcp->head; krcp->head = NULL; @@ -3073,16 +3100,17 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) WRITE_ONCE(krcp->count, 0); /* - * One work is per one batch, so there are two "free channels", - * "bhead_free" and "head_free" the batch can handle. It can be - * that the work is in the pending state when two channels have - * been detached following each other, one by one. + * One work is per one batch, so there are three + * "free channels", the batch can handle. It can + * be that the work is in the pending state when + * channels have been detached following by each + * other. */ queue_rcu_work(system_wq, &krwp->rcu_work); } - /* Repeat if any "free" corresponding channel is still busy. */ - if (krcp->bhead || krcp->head) + // Repeat if any "free" corresponding channel is still busy. + if (krcp->bkvhead[0] || krcp->bkvhead[1] || krcp->head) repeat = true; } @@ -3124,23 +3152,22 @@ static void kfree_rcu_monitor(struct work_struct *work) } static inline bool -kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, - struct rcu_head *head, rcu_callback_t func) +kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) { - struct kfree_rcu_bulk_data *bnode; + struct kvfree_rcu_bulk_data *bnode; + int idx; if (unlikely(!krcp->initialized)) return false; lockdep_assert_held(&krcp->lock); + idx = !!is_vmalloc_addr(ptr); /* Check if a new block is required. */ - if (!krcp->bhead || - krcp->bhead->nr_records == KFREE_BULK_MAX_ENTR) { + if (!krcp->bkvhead[idx] || + krcp->bkvhead[idx]->nr_records == KVFREE_BULK_MAX_ENTR) { bnode = get_cached_bnode(krcp); if (!bnode) { - WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE); - /* * To keep this path working on raw non-preemptible * sections, prevent the optional entry into the @@ -3153,7 +3180,7 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, if (IS_ENABLED(CONFIG_PREEMPT_RT)) return false; - bnode = (struct kfree_rcu_bulk_data *) + bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); } @@ -3163,30 +3190,30 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Initialize the new block. */ bnode->nr_records = 0; - bnode->next = krcp->bhead; + bnode->next = krcp->bkvhead[idx]; /* Attach it to the head. */ - krcp->bhead = bnode; + krcp->bkvhead[idx] = bnode; } /* Finally insert. */ - krcp->bhead->records[krcp->bhead->nr_records++] = - (void *) head - (unsigned long) func; + krcp->bkvhead[idx]->records + [krcp->bkvhead[idx]->nr_records++] = ptr; return true; } /* - * Queue a request for lazy invocation of kfree_bulk()/kfree() after a grace - * period. Please note there are two paths are maintained, one is the main one - * that uses kfree_bulk() interface and second one is emergency one, that is - * used only when the main path can not be maintained temporary, due to memory - * pressure. + * Queue a request for lazy invocation of appropriate free routine after a + * grace period. Please note there are three paths are maintained, two are the + * main ones that use array of pointers interface and third one is emergency + * one, that is used only when the main path can not be maintained temporary, + * due to memory pressure. * * Each kfree_call_rcu() request is added to a batch. The batch will be drained * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will * be free'd in workqueue context. This allows us to: batch requests together to - * reduce the number of grace periods during heavy kfree_rcu() load. + * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { @@ -3209,7 +3236,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (unlikely(!kfree_call_rcu_add_ptr_to_bulk(krcp, head, func))) { + if (unlikely(!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr))) { head->func = func; head->next = krcp->head; krcp->head = head; @@ -4187,7 +4214,7 @@ static void __init kfree_rcu_batch_init(void) for_each_possible_cpu(cpu) { struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu); - struct kfree_rcu_bulk_data *bnode; + struct kvfree_rcu_bulk_data *bnode; for (i = 0; i < KFREE_N_BATCHES; i++) { INIT_RCU_WORK(&krcp->krw_arr[i].rcu_work, kfree_rcu_work); @@ -4195,7 +4222,7 @@ static void __init kfree_rcu_batch_init(void) } for (i = 0; i < rcu_min_cached_objs; i++) { - bnode = (struct kfree_rcu_bulk_data *) + bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); if (bnode) From patchwork Mon May 25 21:47:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569483 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 421D590 for ; Mon, 25 May 2020 21:48:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1036E206C3 for ; Mon, 25 May 2020 21:48:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hPIyjT10" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1036E206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1410F8006A; Mon, 25 May 2020 17:48:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EFA1380061; Mon, 25 May 2020 17:48:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D28408006A; Mon, 25 May 2020 17:48:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id B0E0F80061 for ; Mon, 25 May 2020 17:48:23 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 701EE180AD815 for ; Mon, 25 May 2020 21:48:23 +0000 (UTC) X-FDA: 76856580486.25.news56_2c1e42ff02b09 X-Spam-Summary: 2,0,0,ee9b6adc61c75664,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1714:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3351:3867:3870:3872:4250:4385:5007:6261:6653:7514:7875:9413:10004:11026:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21987:21990:30054,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: news56_2c1e42ff02b09 X-Filterd-Recvd-Size: 4191 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:23 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id a25so10519913ljp.3 for ; Mon, 25 May 2020 14:48:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oZjrHlimgcUYomtREHYCFuAeJrpRl9IFi+y5djKRQas=; b=hPIyjT10zycop+JKy8135w3NPFuM+OtEngjMZW8KAmLmVW2Vxp0+BJgjELMVgXLFs5 jyDXvM8J2pQxwbEB2pyNztU7Se3z5Wd060ijoHg6DDb5aRWX5ZuAMUaNSzHofFIoEJNv 8aI29V6q0aJUJxkrbzIr+/lmfTa+pNBCgK3xFfTkXlN7U0Y/awGBKWtblhWlDEmhuaqL k6nPkAczXPGTLmBEXQYigDrVKl/eeB9s3UKwKW4wuEL+s+zKfha/ZpW43aFl2wLg2amm 4o+lmYtE0sBrX9HIUArLZ8k9W1xEtbHyYJUczmpOqSXo8FfhTp+nx9OiDgLI7iZo7XYS 628w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oZjrHlimgcUYomtREHYCFuAeJrpRl9IFi+y5djKRQas=; b=gbVDs/Jq3i0INq8WKbGu8yxfv+m/zalWmV7mWj+3HPzSBeHdwdXNPacRE9G/Rra3q9 OciIFIKpXc4y/2pJdB7Hy/yk1rQ5jhmQPl6EcjLO1Vqv5v6CAtOkxxLLxhSnwVGJs3X+ HQK9UUgYaKuzD25wpr6UZpcqHUkktI77x+72Axs7suDz0+Ib3Gs2q9ej8JvzELLK2iF3 1TLM2brcLluDUDiiWs37hkY1BLrkuobZEPmLkpoCP9g1i/oUCGHu2E8FwkopdWIqHKMH b29YGvrk/eqv8TushTdoeyB6OZW0Q3L/FeyIKlcxpNQZC8JwWCQDc5jIVGqy5l5t+W6X jJ0w== X-Gm-Message-State: AOAM531/IKXuqqYsyPEGNBp3uqtxoxR+6PNNn5cj+V0l6IPxt+GbUB19 QLvldC5XrrRXuv5veSQNyuc= X-Google-Smtp-Source: ABdhPJyxvtoEaRJXilbbh+ynL0lLxOteStevxPIViaI0XOVzTSChAjJHtXaOv/FWincgOqizXxgD0g== X-Received: by 2002:a2e:711c:: with SMTP id m28mr14476316ljc.104.1590443301892; Mon, 25 May 2020 14:48:21 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:20 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 10/16] rcu/tiny: support vmalloc in tiny-RCU Date: Mon, 25 May 2020 23:47:54 +0200 Message-Id: <20200525214800.93072-11-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace kfree() with kvfree() in rcu_reclaim_tiny(). This makes it possible to release either SLAB or vmalloc objects after a GP. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tiny.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index dd572ce7c747..4b99f7b88bee 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "rcu.h" @@ -86,7 +87,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) rcu_lock_acquire(&rcu_callback_map); if (__is_kfree_rcu_offset(offset)) { trace_rcu_invoke_kfree_callback("", head, offset); - kfree((void *)head - offset); + kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); return true; } From patchwork Mon May 25 21:47:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569485 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3D6EF913 for ; Mon, 25 May 2020 21:48:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0B6DB206C3 for ; Mon, 25 May 2020 21:48:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="W1kObcKM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B6DB206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 33BD98006B; Mon, 25 May 2020 17:48:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2C62A80061; Mon, 25 May 2020 17:48:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CAA38006B; Mon, 25 May 2020 17:48:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0128.hostedemail.com [216.40.44.128]) by kanga.kvack.org (Postfix) with ESMTP id E798080061 for ; Mon, 25 May 2020 17:48:24 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id AF27B180AD815 for ; Mon, 25 May 2020 21:48:24 +0000 (UTC) X-FDA: 76856580528.08.egg68_2c4bbe45b3c1f X-Spam-Summary: 2,0,0,fed708c19832ecdc,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:1:2:41:355:379:421:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:2731:2907:2914:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3873:3874:4051:4250:4321:4385:4390:4395:4605:5007:6119:6261:6653:7514:7875:7903:7904:9413:11026:11232:11473:11658:11914:12043:12114:12296:12297:12438:12517:12519:12555:12895:12986:13053:13161:13229:13894:13972:14096:14394:14687:21080:21212:21433:21444:21451:21627:21666:21740:21772:21987:21990:30034:30054:30070:30076,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:1:0,LFtime:16,LUA_SUMMARY:none X-HE-Tag: egg68_2c4bbe45b3c1f X-Filterd-Recvd-Size: 12048 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:24 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id a25so10519942ljp.3 for ; Mon, 25 May 2020 14:48:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yVqcxSq1IHpUHDxIjujLZwaSnFa1mBd5ODxuoqnxOWA=; b=W1kObcKMmw8db+H82bam2fXJtsL+RpuxspHFM85/znh15bqmGkM2LK8yiecdHxavau 5rbjeeajSMABKzCcMX0ppELr77h303g+JN4aOaln/vZw83FJj+NiX40fsk1DzGTKaaA6 x5HDk/LCR5djU6Xrczr6HmtesEspGaWmBDmbsEjjSZMIC4u02J+YBsNj3QM2dqpQxO9g LMTcTw9D5UaI8L3bqGdD1V1y6f2/BXZIxnwQjXFS790lBc+RM41Uq379XYAvH575i3Cf lwZpdL3SgNohpk8XjXVoichtVmT0Rb8IsVKrh1hS/r57tLNBg0HmPQZtdmGtc8fIkJ0z pFXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yVqcxSq1IHpUHDxIjujLZwaSnFa1mBd5ODxuoqnxOWA=; b=Ghmq7kFSNy0LRwJ+9toxkRnG/qVcXsbdUhepvabH2e0tzGt2ioXgF4EzNvTujmpdHp lK5PwuJ5D1ZDsS0/QP/jN/FBuEDQnfbRqe/d1U3Nt8OQ8clS9OsnCZ9oPJmQwA5+bFkQ jB+sSUCSvsHFYDHfa5ruwKpvNtX4QAAwmuLwhTdJTGL2137IE/OcmXC8mI41z9EXunzb J0cX9rQyyXs7DTL78QpWouSOdFAOvNeT2roEgQ3h+b4MZR4oXAXLcUZDQz9rxjy8SB5z yX4iYa28pExqJFC+SXWY3f3fG6fO+OHw7yhlzu44sK+PbV4Nck4PaVmFp4vxhpQ7VBav mgzQ== X-Gm-Message-State: AOAM531ZK/1+mOgJJtF7jRNRiAH2+PhO9bjIAH02wkjuzaFCHsTajT1r hMBpaYODTnO5MawVVZ5o02s= X-Google-Smtp-Source: ABdhPJy5wxypVh6TqdurvHEZ33pvAl6udqz8cV6qgmsGA6MyBy7ubUq3lXafMzI84W3ObCIKF2Eb6g== X-Received: by 2002:a2e:1515:: with SMTP id s21mr12431951ljd.306.1590443302949; Mon, 25 May 2020 14:48:22 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:22 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 11/16] rcu: Rename *_kfree_callback/*_kfree_rcu_offset/kfree_call_* Date: Mon, 25 May 2020 23:47:55 +0200 Message-Id: <20200525214800.93072-12-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following changes are introduced: 1. Rename rcu_invoke_kfree_callback() to rcu_invoke_kvfree_callback(), as well as the associated trace events, so the rcu_kfree_callback(), becomes rcu_kvfree_callback(). The reason is to be aligned with kvfree() notation. 2. Rename __is_kfree_rcu_offset to __is_kvfree_rcu_offset. All RCU paths use kvfree() now instead of kfree(), thus rename it. 3. Rename kfree_call_rcu() to the kvfree_call_rcu(). The reason is, it is capable of freeing vmalloc() memory now. Do the same with __kfree_rcu() macro, it becomes __kvfree_rcu(), the goal is the same. Reviewed-by: Joel Fernandes (Google) Co-developed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- include/linux/rcupdate.h | 14 +++++++------- include/linux/rcutiny.h | 2 +- include/linux/rcutree.h | 2 +- include/trace/events/rcu.h | 8 ++++---- kernel/rcu/tiny.c | 4 ++-- kernel/rcu/tree.c | 16 ++++++++-------- 6 files changed, 23 insertions(+), 23 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 659cbfa7581a..b344fc800a9b 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -828,17 +828,17 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) /* * Does the specified offset indicate that the corresponding rcu_head - * structure can be handled by kfree_rcu()? + * structure can be handled by kvfree_rcu()? */ -#define __is_kfree_rcu_offset(offset) ((offset) < 4096) +#define __is_kvfree_rcu_offset(offset) ((offset) < 4096) /* * Helper macro for kfree_rcu() to prevent argument-expansion eyestrain. */ -#define __kfree_rcu(head, offset) \ +#define __kvfree_rcu(head, offset) \ do { \ - BUILD_BUG_ON(!__is_kfree_rcu_offset(offset)); \ - kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \ + BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \ + kvfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \ } while (0) /** @@ -857,7 +857,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) * Because the functions are not allowed in the low-order 4096 bytes of * kernel virtual memory, offsets up to 4095 bytes can be accommodated. * If the offset is larger than 4095 bytes, a compile-time error will - * be generated in __kfree_rcu(). If this error is triggered, you can + * be generated in __kvfree_rcu(). If this error is triggered, you can * either fall back to use of call_rcu() or rearrange the structure to * position the rcu_head structure into the first 4096 bytes. * @@ -872,7 +872,7 @@ do { \ typeof (ptr) ___p = (ptr); \ \ if (___p) \ - __kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ + __kvfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ } while (0) /* diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index 8b851904efed..00bbd0e328c8 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -31,7 +31,7 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } -static inline void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) +static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { call_rcu(head, func); } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 9366fa4d0717..da07f8dc05c6 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -33,7 +33,7 @@ static inline void rcu_virt_note_context_switch(int cpu) } void synchronize_rcu_expedited(void); -void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func); +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func); void rcu_barrier(void); bool rcu_eqs_special_set(int cpu); diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 02dcd119f326..9d34ee810894 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -509,13 +509,13 @@ TRACE_EVENT_RCU(rcu_callback, /* * Tracepoint for the registration of a single RCU callback of the special - * kfree() form. The first argument is the RCU type, the second argument + * kvfree() form. The first argument is the RCU type, the second argument * is a pointer to the RCU callback, the third argument is the offset * of the callback within the enclosing RCU-protected data structure, * the fourth argument is the number of lazy callbacks queued, and the * fifth argument is the total number of callbacks queued. */ -TRACE_EVENT_RCU(rcu_kfree_callback, +TRACE_EVENT_RCU(rcu_kvfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset, long qlen), @@ -599,12 +599,12 @@ TRACE_EVENT_RCU(rcu_invoke_callback, /* * Tracepoint for the invocation of a single RCU callback of the special - * kfree() form. The first argument is the RCU flavor, the second + * kvfree() form. The first argument is the RCU flavor, the second * argument is a pointer to the RCU callback, and the third argument * is the offset of the callback within the enclosing RCU-protected * data structure. */ -TRACE_EVENT_RCU(rcu_invoke_kfree_callback, +TRACE_EVENT_RCU(rcu_invoke_kvfree_callback, TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset), diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 4b99f7b88bee..aa897c3f2e92 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -85,8 +85,8 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head) unsigned long offset = (unsigned long)head->func; rcu_lock_acquire(&rcu_callback_map); - if (__is_kfree_rcu_offset(offset)) { - trace_rcu_invoke_kfree_callback("", head, offset); + if (__is_kvfree_rcu_offset(offset)) { + trace_rcu_invoke_kvfree_callback("", head, offset); kvfree((void *)head - offset); rcu_lock_release(&rcu_callback_map); return true; diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 9f84ff80bc25..da29e6078392 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2771,8 +2771,8 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kfree_rcu_offset((unsigned long)func)) - trace_rcu_kfree_callback(rcu_state.name, head, + if (__is_kvfree_rcu_offset((unsigned long)func)) + trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, rcu_segcblist_n_cbs(&rdp->cblist)); else @@ -3012,7 +3012,7 @@ static void kfree_rcu_work(struct work_struct *work) bkvhead[i]->records); } else { // vmalloc() / vfree(). for (j = 0; j < bkvhead[i]->nr_records; j++) { - trace_rcu_invoke_kfree_callback( + trace_rcu_invoke_kvfree_callback( rcu_state.name, bkvhead[i]->records[j], 0); @@ -3045,9 +3045,9 @@ static void kfree_rcu_work(struct work_struct *work) next = head->next; debug_rcu_head_unqueue((struct rcu_head *)ptr); rcu_lock_acquire(&rcu_callback_map); - trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset); + trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset); - if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) + if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) kvfree(ptr); rcu_lock_release(&rcu_callback_map); @@ -3210,12 +3210,12 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) * one, that is used only when the main path can not be maintained temporary, * due to memory pressure. * - * Each kfree_call_rcu() request is added to a batch. The batch will be drained + * Each kvfree_call_rcu() request is added to a batch. The batch will be drained * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will * be free'd in workqueue context. This allows us to: batch requests together to * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ -void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) +void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; @@ -3254,7 +3254,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) unlock_return: krc_this_cpu_unlock(krcp, flags); } -EXPORT_SYMBOL_GPL(kfree_call_rcu); +EXPORT_SYMBOL_GPL(kvfree_call_rcu); static unsigned long kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) From patchwork Mon May 25 21:47:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569487 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2EC83913 for ; Mon, 25 May 2020 21:48:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F11F6206C3 for ; Mon, 25 May 2020 21:48:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NXSy4DSx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F11F6206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 169A38006C; Mon, 25 May 2020 17:48:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0528980061; Mon, 25 May 2020 17:48:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E60168006C; Mon, 25 May 2020 17:48:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0105.hostedemail.com [216.40.44.105]) by kanga.kvack.org (Postfix) with ESMTP id AA19280061 for ; Mon, 25 May 2020 17:48:25 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 7A254181AC9CB for ; Mon, 25 May 2020 21:48:25 +0000 (UTC) X-FDA: 76856580570.06.hand04_2c6caec8c6754 X-Spam-Summary: 2,0,0,0292f81b846de4f4,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3165:3352:3865:3866:3867:3868:3871:3872:4385:4390:4395:5007:6261:6653:7514:9413:10004:11026:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13069:13161:13229:13311:13357:13846:13894:13972:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054:30070,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: hand04_2c6caec8c6754 X-Filterd-Recvd-Size: 4888 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:25 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id z18so22116865lji.12 for ; Mon, 25 May 2020 14:48:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ma6ozjce0VsxuWEnBAmau4C36oX126QKppp7v2kR4Bo=; b=NXSy4DSx5DHkmao9iem3wyRQf7BFcvkq9BZwo4Xn7fNoonCwaGlYttBUNSK7XZyiKk MvFXMOtTnUGYIiaM2GCWi6QC4BB2er3giLKzJQugD8tGOH3BD2sbvjtj+V2aKbwzKrNM pfrFQpL9uwJZYM0QgCNtSI/PamuNUIIdOcIGu6z8qFvWUQZ7vQI11gTVpRK5fJqKQost VWHqXAs4hJla7bwSnB5QWpSyt8y3wvGhKapGaJz6UPZBpbC4cD74DqwzDp+LilFb/yCH ZYUO0x9KmNL3FPcmKukuuHe14Jr4U2Q1xo+/C+9lzW5T1P6m/bAMyKbWJcVeQSOqGIWR YdfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ma6ozjce0VsxuWEnBAmau4C36oX126QKppp7v2kR4Bo=; b=QXL4I2MSTBauLBUrhc1E/u7ykgKJju8z4yRF36yY+bRzGdvT+YgDshUZdNcTg/DqH8 RuNYswJ254DjTknA9nQn6Uwi7YnV5ezAl3vkSGiySZdIx0M8DcvfyxYzDOUEOoYPmq8h 51Td1kMjmnGIj8gZIasFR1NegyHrxZJ85nC4Sttmlf+8Ohk8i6wyayvfjyIDDearTd0j SWObKJib4op63g1BwmiNrd6Gj+DAQLnQBr1f5bJ6WtWcRO0CMNC4kidhA8gDXYq9lwLX xQ4+zTFtPRMzW792qB13cuythCeNOMlZzaoGbcUOFycKqOwUcaeMrGyxfNPn2lX1MMs6 47LQ== X-Gm-Message-State: AOAM532iG2K4+TGHbtJtyo9Yzid+BbocLdmt4ugxq73OrMUC5dThs3AN tLcrAiNcPoT3EZ3esssuy64= X-Google-Smtp-Source: ABdhPJzX7B4hxYlADv9D7q19Ax/D4dz4rbE7H3yxDwhkpf8rQ+Y8wFTVlnlYRsZU/BydGVrMJi0gTg== X-Received: by 2002:a2e:9891:: with SMTP id b17mr11934960ljj.320.1590443303981; Mon, 25 May 2020 14:48:23 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:23 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 12/16] mm/list_lru.c: Rename kvfree_rcu() to local variant Date: Mon, 25 May 2020 23:47:56 +0200 Message-Id: <20200525214800.93072-13-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rename kvfree_rcu() function to the kvfree_rcu_local() one. The purpose is to prevent a conflict of two same function declarations. The kvfree_rcu() will be globally visible what would lead to a build error. No functional change. Cc: linux-mm@kvack.org Cc: rcu@vger.kernel.org Cc: Andrew Morton Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- mm/list_lru.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 4d5294c39bba..42c95bcb53ca 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -373,14 +373,14 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru) struct list_lru_memcg *memcg_lrus; /* * This is called when shrinker has already been unregistered, - * and nobody can use it. So, there is no need to use kvfree_rcu(). + * and nobody can use it. So, there is no need to use kvfree_rcu_local(). */ memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true); __memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids); kvfree(memcg_lrus); } -static void kvfree_rcu(struct rcu_head *head) +static void kvfree_rcu_local(struct rcu_head *head) { struct list_lru_memcg *mlru; @@ -419,7 +419,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru, rcu_assign_pointer(nlru->memcg_lrus, new); spin_unlock_irq(&nlru->lock); - call_rcu(&old->rcu, kvfree_rcu); + call_rcu(&old->rcu, kvfree_rcu_local); return 0; } From patchwork Mon May 25 21:47:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569489 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB36C90 for ; Mon, 25 May 2020 21:48:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AFD45206C3 for ; Mon, 25 May 2020 21:48:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oK0yLXUM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AFD45206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2F05B8006D; Mon, 25 May 2020 17:48:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1DB4380061; Mon, 25 May 2020 17:48:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A4398006D; Mon, 25 May 2020 17:48:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id D138780061 for ; Mon, 25 May 2020 17:48:26 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A474C2C32 for ; Mon, 25 May 2020 21:48:26 +0000 (UTC) X-FDA: 76856580612.06.rub27_2c9416bb33741 X-Spam-Summary: 2,0,0,da7f1e1f425d22e3,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3871:3872:3874:4321:4385:4390:4395:4605:5007:6261:6653:7514:7875:7904:8603:9413:10004:11026:11658:11914:12296:12297:12517:12519:12555:12895:12986:13069:13311:13357:13894:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:30029:30054:30070,0,RBL:209.85.208.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: rub27_2c9416bb33741 X-Filterd-Recvd-Size: 4812 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:26 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id b6so22188684ljj.1 for ; Mon, 25 May 2020 14:48:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ik17svU3U29XQbLhqwF9DkU8YQ7sw6ADmoig1ExIl5o=; b=oK0yLXUMmFF5FfsG8+ulkLizK0+hpY4rBoPTkb7nV7T2QcpjbXrOHWDLqpXlwRu3m4 cpHEgZHagoLqggB6XZw3U7Z5EMBXyn73WUlxG4szDEy/5NdBis46mpdYaavBV7lVEr53 pL6JUMTe6iYJ9MXw18DSPO0y7fTGv+/w/D40aGkLANqs0eGSKrLL35ML73DOuS3iOIAu QxrYZ3+zUPz1Ive73j2u0d2N6NK7xon/ewslNnjgO8FPz7vY5iX55mBVziN6qlT1OBSU m0jOzr7OyI5uGSzkJ4MqvCculzjhE4w9LptHuuSMtyZf8ybSgnHIIbZ3ug8CNI5wmhPb qcCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ik17svU3U29XQbLhqwF9DkU8YQ7sw6ADmoig1ExIl5o=; b=UTrvltUffWvgf8mKZszRB/JndGWMcF5R9j5MraKKgMAHDeNmT95Q3but7J2ddNBA1H EhyE52AqLgguSGR/vcltMHFMZ0z9flG9b20p777tjxv5jE1in7ekvIE1sivQqVI8lOBJ aeMubFLqVc8nxDuesW8QS8yIY2dELPvFyLWQGZDKMkZ8gds74TFDG23O5pa8xX1hslPI MgGvjmL+qmYyT4U0IVzrWCj+4lnwM9rTRwlutycuSlhrneOpZtUMpZlJq/iJWpRfaoMO MJMA0CzShcni7Ml0HA2kHJkrJJuyu+6NfylJVKseIPILgmgqxtL/BtzmL1c7c+mL6Ogu /hig== X-Gm-Message-State: AOAM533TZAUCYTwXYapAHu19oNw+faFqLPL0+r4KWlQN9BJH4NfwXnxW uZwEqxjmzsrMTdu09FYU3fM= X-Google-Smtp-Source: ABdhPJzaf0nFKQVcNe6lFV+Hv17Rsr7GTfoxNLsSnpb8/RGK0v1mlJ5VKEs+R3jKwSb+XhwZeNmY6A== X-Received: by 2002:a2e:9586:: with SMTP id w6mr14219809ljh.274.1590443305005; Mon, 25 May 2020 14:48:25 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:24 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 13/16] rcu: Introduce 2 arg kvfree_rcu() interface Date: Mon, 25 May 2020 23:47:57 +0200 Message-Id: <20200525214800.93072-14-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kvmalloc() can allocate two types of objects: SLAB backed and vmalloc backed. How it behaves depends on requested object's size and memory pressure. Add a kvfree_rcu() interface that can free memory allocated via kvmalloc(). It is a simple alias to kfree_rcu() which can now handle either type of object. struct test_kvfree_rcu { struct rcu_head rcu; unsigned char array[100]; }; struct test_kvfree_rcu *p; p = kvmalloc(10 * PAGE_SIZE); if (p) kvfree_rcu(p, rcu); Signed-off-by: Uladzislau Rezki (Sony) Co-developed-by: Joel Fernandes (Google) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Joel Fernandes (Google) --- include/linux/rcupdate.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index b344fc800a9b..51b26ab02878 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -875,6 +875,15 @@ do { \ __kvfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \ } while (0) +/** + * kvfree_rcu() - kvfree an object after a grace period. + * @ptr: pointer to kvfree + * @rhf: the name of the struct rcu_head within the type of @ptr. + * + * Same as kfree_rcu(), just simple alias. + */ +#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf) + /* * Place this after a lock-acquisition primitive to guarantee that * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies From patchwork Mon May 25 21:47:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569491 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C794790 for ; Mon, 25 May 2020 21:48:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8C438207CB for ; Mon, 25 May 2020 21:48:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HjVzJp58" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C438207CB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5ADB28006E; Mon, 25 May 2020 17:48:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4EEFA80061; Mon, 25 May 2020 17:48:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 274F38006E; Mon, 25 May 2020 17:48:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0084.hostedemail.com [216.40.44.84]) by kanga.kvack.org (Postfix) with ESMTP id 07D3380061 for ; Mon, 25 May 2020 17:48:28 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BBF7D2C32 for ; Mon, 25 May 2020 21:48:27 +0000 (UTC) X-FDA: 76856580654.08.legs64_2cbdef11af32d X-Spam-Summary: 2,0,0,0167d4d784e1b724,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2199:2393:2553:2559:2562:2895:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:4605:5007:6117:6119:6261:6653:7514:7903:8660:9010:9413:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13148:13230:13894:14096:14394:14687:21080:21324:21433:21444:21451:21627:21666:21796:21939:21990:30036:30045:30054:30070:30090,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: legs64_2cbdef11af32d X-Filterd-Recvd-Size: 8219 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:27 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id l15so21806364lje.9 for ; Mon, 25 May 2020 14:48:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Eb8xq22KfAG6fp/WMem/0LimDG8xrTCxTTXXT11xHa8=; b=HjVzJp58HalgeSnFZ8TYNJeitRlinF1ZZkF3izRhYWt6NkUIcYxLzlJoz83pHJ1BhQ qs7zVnTCqZK7GtdZZCXle1gps0THusL1zrgZuTx63iamYpvBISPkP3J+gCTt7J0waglJ zm6U8EcJ2YZkrdedr9Tk4x2qU8xN8p24IB4in/1jru8iH50Eimv+4CTLSncK2ktoMFdU uWKFdC5LLSdFDv62aEdiJ3wzU26JFCT7Ptb+MgN8FySCeLwTLU23zrKdzH8jwuMmJZHx bGD0ZnkiqXtvi9Ld5J/siBePtEoJQpWfh0L8ooWrKPC1xSV+PoF4VAhi4MNb9zv/aJDy mgWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Eb8xq22KfAG6fp/WMem/0LimDG8xrTCxTTXXT11xHa8=; b=HMWoDYBXqVpMo72vKA2xoAYS2Tbb/jw3cfcv3cvJzZlARUzY0XGNHUzGDSLhiurqJM ZanvMFMlIkcjzTOQYVsoXaVBrv8zDcuPIZ3xUxIixE8FHO8yuGfMoPSkrT5pwasqJsJG QzPECYotQq2ppBPM+1iqUDhlgeKuERdlCiPpSO8iGM3ujoENxhdQhPMhu6aBtbBLz/NP F9QOPaU+H2FxmE1CWVkZSNo/56Vgg6JCY+N3feUG+7gyoQNQHrsx2cC76Nz7BSi2lzh6 akzcfizOvQYOLyVQYQQ+YJ+ruf/KrawYQBuW2k84wXC94aIRzuSyoL4+gpVADJwbYNP9 xtmg== X-Gm-Message-State: AOAM530asZ2g2M1M4DV/KAdMOpmdGzVL/uAKCCkHOgDCdh+jHp8NboQl aaC683OGFZzOpRmI1zi8Y2A= X-Google-Smtp-Source: ABdhPJxfYdA2lsbxwcu/cCC/8bfQhOexs2m2EigiwkWGZU776Hnc0bXM379NuIVU5PU462pBuLiVPQ== X-Received: by 2002:a2e:9818:: with SMTP id a24mr1882342ljj.161.1590443306082; Mon, 25 May 2020 14:48:26 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:25 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 14/16] rcu: Support reclaim for head-less object Date: Mon, 25 May 2020 23:47:58 +0200 Message-Id: <20200525214800.93072-15-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the kvfree_call_rcu() function with head-less support. This allows RCU to reclaim objects without an embedded rcu_head. tree-RCU: We introduce two chains of arrays to store SLAB-backed and vmalloc pointers, each. Storage in either of these arrays does not require embedding an rcu_head within the object. Maintaining the arrays may become impossible due to high memory pressure. For such cases there is an emergency path. Objects with rcu_head inside are just queued on a backup rcu_head list. Later on that list is drained. As for the head-less variant, as the current context can sleep, the following emergency measures are applied: a) Synchronously wait until a grace period has elapsed. b) Call kvfree(). tiny-RCU: For double argument calls, there are no new changes in behavior. For single argument call, kvfree() is directly inlined on the current stack after a synchronize_rcu() call. Note that for tiny-RCU, any call to synchronize_rcu() is actually a quiescent state, therefore it does nothing. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) Co-developed-by: Joel Fernandes (Google) --- include/linux/rcutiny.h | 18 ++++++++++++++++- kernel/rcu/tree.c | 45 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 60 insertions(+), 3 deletions(-) diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index 00bbd0e328c8..e7e4ec5938af 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -31,9 +31,25 @@ static inline void synchronize_rcu_expedited(void) synchronize_rcu(); } +/* + * Add one more declaration of kvfree() here. It is + * not so straight forward to just include + * where it is defined due to getting many compile + * errors caused by that include. + */ +extern void kvfree(const void *addr); + static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { - call_rcu(head, func); + if (head) { + call_rcu(head, func); + return; + } + + // kvfree_rcu(one_arg) call. + might_sleep(); + synchronize_rcu(); + kvfree((void *) func); } void rcu_qs(void); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index da29e6078392..2459830a3851 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3180,6 +3180,13 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) if (IS_ENABLED(CONFIG_PREEMPT_RT)) return false; + /* + * NOTE: For one argument of kvfree_rcu() we can + * drop the lock and get the page in sleepable + * context. That would allow to maintain an array + * for the CONFIG_PREEMPT_RT as well if no cached + * pages are available. + */ bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); } @@ -3219,16 +3226,33 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { unsigned long flags; struct kfree_rcu_cpu *krcp; + bool success; void *ptr; + if (head) { + ptr = (void *) head - (unsigned long) func; + } else { + /* + * Please note there is a limitation for the head-less + * variant, that is why there is a clear rule for such + * objects: it can be used from might_sleep() context + * only. For other places please embed an rcu_head to + * your data. + */ + might_sleep(); + ptr = (unsigned long *) func; + } + krcp = krc_this_cpu_lock(&flags); - ptr = (void *)head - (unsigned long)func; // Queue the object but don't yet schedule the batch. if (debug_rcu_head_queue(ptr)) { // Probable double kfree_rcu(), just leak. WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", __func__, head); + + // Mark as success and leave. + success = true; goto unlock_return; } @@ -3236,10 +3260,16 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (unlikely(!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr))) { + success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr); + if (!success) { + if (head == NULL) + // Inline if kvfree_rcu(one_arg) call. + goto unlock_return; + head->func = func; head->next = krcp->head; krcp->head = head; + success = true; } WRITE_ONCE(krcp->count, krcp->count + 1); @@ -3253,6 +3283,17 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) unlock_return: krc_this_cpu_unlock(krcp, flags); + + /* + * Inline kvfree() after synchronize_rcu(). We can do + * it from might_sleep() context only, so the current + * CPU can pass the QS state. + */ + if (!success) { + debug_rcu_head_unqueue((struct rcu_head *) ptr); + synchronize_rcu(); + kvfree(ptr); + } } EXPORT_SYMBOL_GPL(kvfree_call_rcu); From patchwork Mon May 25 21:47:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569493 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E542290 for ; Mon, 25 May 2020 21:48:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B2F94206C3 for ; Mon, 25 May 2020 21:48:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q7gfBC+1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B2F94206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 695EA8006F; Mon, 25 May 2020 17:48:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 621E780061; Mon, 25 May 2020 17:48:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BDF98006F; Mon, 25 May 2020 17:48:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0093.hostedemail.com [216.40.44.93]) by kanga.kvack.org (Postfix) with ESMTP id 2DCC980061 for ; Mon, 25 May 2020 17:48:29 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E318F181AC9CB for ; Mon, 25 May 2020 21:48:28 +0000 (UTC) X-FDA: 76856580696.25.bean91_2ce60ee53a43e X-Spam-Summary: 2,0,0,f0b8d0d5b50bad71,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:4385:4390:4395:4605:5007:6119:6261:6653:7514:7875:7903:7904:9413:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13180:13229:13894:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21939:30012:30029:30054,0,RBL:209.85.208.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: bean91_2ce60ee53a43e X-Filterd-Recvd-Size: 6138 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:28 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id c11so20000280ljn.2 for ; Mon, 25 May 2020 14:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KLIimAxShdXUGKczejB46X3ypCcUUoC/Csjl4VU2E+4=; b=Q7gfBC+1ILDdxSUjjcuyqOmy66V+6pob1eITQV4EIxudn8SXWItz7+TmuY6po5UZEl R9G15d3+16vguqYMPAqL/D0TrWStNpcWsltrDX3JiI/R75EcLxe4B/CZQwRfSgkJF1TZ 8FV3Z7DTSed4DxB7lv4jurttunSqRoAyGXlI+7863ggIzhphlfvWC5zwrr5bYTMwc6Mb 7m0VukxsGCtGwz0fvIXXVYHTJRYfFLwWdKFVqdMnDJApvbKB4sWYyrHf8HkPx+0wZkeq CMaPbWCQZQT6iYkrKqtvBoGipUmdFiDKETIBfRuYxfQkdTZraS9q1ajI3A6vUwCHazz5 Z32g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KLIimAxShdXUGKczejB46X3ypCcUUoC/Csjl4VU2E+4=; b=jDlK1xSoaBYt723FjXDDZc2tSYuR5Al/Z02tvCDPX9//PDnZFvLa8sOhprNhHmxDqw AVjfCYjUeA9p59wYTye57SxQctPqNzFNiHCfDFW3Ktl8JeS76whBk/2kstI6HPrLVwpp B5iHutZn1UTzVa5D5QME7hzM4nmnpqg8vMRPVfudhRA8GcNzo0OLJtSKbhazt0BUjZOT HU4Ydcuv01HNczpOjYbVng+SN6WHJ9TOBLAcO7E7hOKf0PwpbqoQx5V4TW0NhZ16TXUA M3PRs2HfrAlwW274drj8p8XpY2z4Yb7hoYK/IP2r9ZwJuZhKgarOSeVXeBzj+aHgdaYX LLbQ== X-Gm-Message-State: AOAM532b0qbcQSongderyBYi74KSw4JkAq6R1ImI9GoD5sSbjFBjzxPq 0h/Si4kEvfgdma/fM4AbhBRC7gU1Z9MFQw== X-Google-Smtp-Source: ABdhPJwCgQgMm4h3pLJ7oGEz9S8XTUYhEdVg429I3Qh6OkWK/z20EI55Qbu6gIqTJmV4Y79ZmOYkIA== X-Received: by 2002:a2e:9953:: with SMTP id r19mr13234097ljj.345.1590443307169; Mon, 25 May 2020 14:48:27 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:26 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 15/16] rcu: Introduce single argument kvfree_rcu() interface Date: Mon, 25 May 2020 23:47:59 +0200 Message-Id: <20200525214800.93072-16-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make kvfree_rcu() capable of freeing objects that will not embed an rcu_head within it. This saves storage overhead in such objects. Reclaiming headless objects this way requires only a single argument (pointer to the object). After this patch, there are two ways to use kvfree_rcu(): a) kvfree_rcu(ptr, rhf); struct X { struct rcu_head rhf; unsigned char data[100]; }; void *ptr = kvmalloc(sizeof(struct X), GFP_KERNEL); if (ptr) kvfree_rcu(ptr, rhf); b) kvfree_rcu(ptr); void *ptr = kvmalloc(some_bytes, GFP_KERNEL); if (ptr) kvfree_rcu(ptr); Note that the headless usage (example b) can only be used in a code that can sleep. This is enforced by the CONFIG_DEBUG_ATOMIC_SLEEP option. Co-developed-by: Joel Fernandes (Google) Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- include/linux/rcupdate.h | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 51b26ab02878..d15d46db61f7 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -877,12 +877,42 @@ do { \ /** * kvfree_rcu() - kvfree an object after a grace period. - * @ptr: pointer to kvfree - * @rhf: the name of the struct rcu_head within the type of @ptr. * - * Same as kfree_rcu(), just simple alias. + * This macro consists of one or two arguments and it is + * based on whether an object is head-less or not. If it + * has a head then a semantic stays the same as it used + * to be before: + * + * kvfree_rcu(ptr, rhf); + * + * where @ptr is a pointer to kvfree(), @rhf is the name + * of the rcu_head structure within the type of @ptr. + * + * When it comes to head-less variant, only one argument + * is passed and that is just a pointer which has to be + * freed after a grace period. Therefore the semantic is + * + * kvfree_rcu(ptr); + * + * where @ptr is a pointer to kvfree(). + * + * Please note, head-less way of freeing is permitted to + * use from a context that has to follow might_sleep() + * annotation. Otherwise, please switch and embed the + * rcu_head structure within the type of @ptr. */ -#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf) +#define kvfree_rcu(...) KVFREE_GET_MACRO(__VA_ARGS__, \ + kvfree_rcu_arg_2, kvfree_rcu_arg_1)(__VA_ARGS__) + +#define KVFREE_GET_MACRO(_1, _2, NAME, ...) NAME +#define kvfree_rcu_arg_2(ptr, rhf) kfree_rcu(ptr, rhf) +#define kvfree_rcu_arg_1(ptr) \ +do { \ + typeof(ptr) ___p = (ptr); \ + \ + if (___p) \ + kvfree_call_rcu(NULL, (rcu_callback_t) (___p)); \ +} while (0) /* * Place this after a lock-acquisition primitive to guarantee that From patchwork Mon May 25 21:48:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11569495 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2FCC590 for ; Mon, 25 May 2020 21:48:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F1AD1208A7 for ; Mon, 25 May 2020 21:48:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RE8QuoBv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F1AD1208A7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6745180070; Mon, 25 May 2020 17:48:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 583A080061; Mon, 25 May 2020 17:48:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 422D680070; Mon, 25 May 2020 17:48:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id 217C680061 for ; Mon, 25 May 2020 17:48:30 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D4ED7181AC9CB for ; Mon, 25 May 2020 21:48:29 +0000 (UTC) X-FDA: 76856580738.10.grip27_2d0d92933de63 X-Spam-Summary: 2,0,0,305f54b4fd1db1fe,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2987:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:4118:4321:4385:4390:4395:5007:6261:6653:7514:9163:9413:10004:11026:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13870:13894:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30029:30054:30056:30070,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: grip27_2d0d92933de63 X-Filterd-Recvd-Size: 7309 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 May 2020 21:48:29 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id v16so22189569ljc.8 for ; Mon, 25 May 2020 14:48:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rS1Miq61OoUAl0/Xis3Uv9UhAoqrmOOO+XDtwBPTCJI=; b=RE8QuoBvpz11xXscw/NKQU3OarX9sgIfyI5WNt5W1Biv8HvIndCiJS++QTl2IwSvJT ajSxI61vOnKp7pTICUiVOKWGNtWagy/vwzNvj89DSOgO1+F1TQUUg/15e/v3lOZCrQEN m5mKxOPbpn15RJF8FzmG7ZY9CWE+s1N+zGC/q1gjLVzvHB7Fx4E1NEgqsFq3cfrGJM7N QEbkhH5Gc8YjB0lP4gPDWNVGcgWrBEhzNk9DQIjbsLSaDERYqhnxl8VzdFMACBAbrB2+ PCCQJTs807Z/SQ+oSkfzIJ4B2zyUvxemr2+C4CFS0oO8ljLhG8tclbbQDWsng/kE3gPO XNFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rS1Miq61OoUAl0/Xis3Uv9UhAoqrmOOO+XDtwBPTCJI=; b=afckFdE6H/5fKzIKC3uMxayttuhOicneoZJNNmFc6sW32LObaPtz061slpJIFfc+W3 SgVIL96L/RuYa29VAgTwDbOMsE5i260V4vEY6K9W//WQxZk/b2Hg2ryUXMAvidmamDHe c3WWtQwb0ryrq/Uburn2jcJDCkYFTNPWtpp0gIB6zcivpHpNgbZRA1H94H73JUPxssIj lUmWT3Dp9R2L0FZN9Wa2Aqf6xNLKPCOQhFYxm6cn3zeTIkxS+6qOcG/RGp6Sm2L+S6tA k2/83BKAY2gUGvwIAXrPrDRMx3sYQhKs6ptK6TRZrkAI11q7plPeg4wErzlmGO0MktcA 8wxA== X-Gm-Message-State: AOAM531nFY8jagvjGH8dGk92VELaPUyU5D+i1n8uAyyOTLna8AaXQh+B DP2j9hfi2Iwna2PtL6yhU18= X-Google-Smtp-Source: ABdhPJzhhlCjbYamgKHrmi7RZLTeTakbFK4kj6T3b4UqYW09i9grziwtwzfUWaRmv4ROaQgmVH9S6g== X-Received: by 2002:a05:651c:105c:: with SMTP id x28mr14812208ljm.65.1590443308220; Mon, 25 May 2020 14:48:28 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id a6sm2280044lji.29.2020.05.25.14.48.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 May 2020 14:48:27 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 16/16] lib/test_vmalloc.c: Add test cases for kvfree_rcu() Date: Mon, 25 May 2020 23:48:00 +0200 Message-Id: <20200525214800.93072-17-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200525214800.93072-1-urezki@gmail.com> References: <20200525214800.93072-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce four new test cases for testing the kvfree_rcu() interface. Two of them belong to single argument functionality and another two for 2-argument functionality. The aim is to stress and check how kvfree_rcu() behaves under different load and memory conditions and analyze its performance throughput. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- lib/test_vmalloc.c | 103 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 95 insertions(+), 8 deletions(-) diff --git a/lib/test_vmalloc.c b/lib/test_vmalloc.c index 8bbefcaddfe8..ec73561cda2e 100644 --- a/lib/test_vmalloc.c +++ b/lib/test_vmalloc.c @@ -15,6 +15,8 @@ #include #include #include +#include +#include #define __param(type, name, init, msg) \ static type name = init; \ @@ -35,14 +37,18 @@ __param(int, test_loop_count, 1000000, __param(int, run_test_mask, INT_MAX, "Set tests specified in the mask.\n\n" - "\t\tid: 1, name: fix_size_alloc_test\n" - "\t\tid: 2, name: full_fit_alloc_test\n" - "\t\tid: 4, name: long_busy_list_alloc_test\n" - "\t\tid: 8, name: random_size_alloc_test\n" - "\t\tid: 16, name: fix_align_alloc_test\n" - "\t\tid: 32, name: random_size_align_alloc_test\n" - "\t\tid: 64, name: align_shift_alloc_test\n" - "\t\tid: 128, name: pcpu_alloc_test\n" + "\t\tid: 1, name: fix_size_alloc_test\n" + "\t\tid: 2, name: full_fit_alloc_test\n" + "\t\tid: 4, name: long_busy_list_alloc_test\n" + "\t\tid: 8, name: random_size_alloc_test\n" + "\t\tid: 16, name: fix_align_alloc_test\n" + "\t\tid: 32, name: random_size_align_alloc_test\n" + "\t\tid: 64, name: align_shift_alloc_test\n" + "\t\tid: 128, name: pcpu_alloc_test\n" + "\t\tid: 256, name: kvfree_rcu_1_arg_vmalloc_test\n" + "\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n" + "\t\tid: 1024, name: kvfree_rcu_1_arg_slab_test\n" + "\t\tid: 2048, name: kvfree_rcu_2_arg_slab_test\n" /* Add a new test case description here. */ ); @@ -328,6 +334,83 @@ pcpu_alloc_test(void) return rv; } +struct test_kvfree_rcu { + struct rcu_head rcu; + unsigned char array[20]; +}; + +static int +kvfree_rcu_1_arg_vmalloc_test(void) +{ + struct test_kvfree_rcu *p; + int i; + + for (i = 0; i < test_loop_count; i++) { + p = vmalloc(1 * PAGE_SIZE); + if (!p) + return -1; + + p->array[0] = 'a'; + kvfree_rcu(p); + } + + return 0; +} + +static int +kvfree_rcu_2_arg_vmalloc_test(void) +{ + struct test_kvfree_rcu *p; + int i; + + for (i = 0; i < test_loop_count; i++) { + p = vmalloc(1 * PAGE_SIZE); + if (!p) + return -1; + + p->array[0] = 'a'; + kvfree_rcu(p, rcu); + } + + return 0; +} + +static int +kvfree_rcu_1_arg_slab_test(void) +{ + struct test_kvfree_rcu *p; + int i; + + for (i = 0; i < test_loop_count; i++) { + p = kmalloc(sizeof(*p), GFP_KERNEL); + if (!p) + return -1; + + p->array[0] = 'a'; + kvfree_rcu(p); + } + + return 0; +} + +static int +kvfree_rcu_2_arg_slab_test(void) +{ + struct test_kvfree_rcu *p; + int i; + + for (i = 0; i < test_loop_count; i++) { + p = kmalloc(sizeof(*p), GFP_KERNEL); + if (!p) + return -1; + + p->array[0] = 'a'; + kvfree_rcu(p, rcu); + } + + return 0; +} + struct test_case_desc { const char *test_name; int (*test_func)(void); @@ -342,6 +425,10 @@ static struct test_case_desc test_case_array[] = { { "random_size_align_alloc_test", random_size_align_alloc_test }, { "align_shift_alloc_test", align_shift_alloc_test }, { "pcpu_alloc_test", pcpu_alloc_test }, + { "kvfree_rcu_1_arg_vmalloc_test", kvfree_rcu_1_arg_vmalloc_test }, + { "kvfree_rcu_2_arg_vmalloc_test", kvfree_rcu_2_arg_vmalloc_test }, + { "kvfree_rcu_1_arg_slab_test", kvfree_rcu_1_arg_slab_test }, + { "kvfree_rcu_2_arg_slab_test", kvfree_rcu_2_arg_slab_test }, /* Add a new test case here. */ };