From patchwork Thu Apr 2 12:32:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11470647 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6803A1667 for ; Thu, 2 Apr 2020 12:33:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0E04A20784 for ; Thu, 2 Apr 2020 12:33:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YdCj68Co" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E04A20784 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9C9188E0008; Thu, 2 Apr 2020 08:33:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 97A068E0007; Thu, 2 Apr 2020 08:33:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88E898E0008; Thu, 2 Apr 2020 08:33:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0142.hostedemail.com [216.40.44.142]) by kanga.kvack.org (Postfix) with ESMTP id 6E7BF8E0007 for ; Thu, 2 Apr 2020 08:33:05 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 22F64181AC9CB for ; Thu, 2 Apr 2020 12:33:05 +0000 (UTC) X-FDA: 76662854730.01.rock09_72353abaab327 X-Spam-Summary: 2,0,0,edada8eb4555f2d7,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3871:3872:3874:4250:4321:4385:5007:6117:6119:6261:6653:7514:7903:9413:9592:10004:10394:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13869:13894:14093:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21789:21939:21990:30005:30054:30056,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: rock09_72353abaab327 X-Filterd-Recvd-Size: 5751 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Thu, 2 Apr 2020 12:33:04 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id p14so2995500lji.11 for ; Thu, 02 Apr 2020 05:33:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=l3xG+lzh6Q2m2bU1b/cZLY6stz0OrJ21NrAPdv3f5lI=; b=YdCj68CoIGNm8klzekN6UwRYTOMrJXNAezwvbdvUizysC4JClKGycqepTK+P7kMSvz uwVO0pWBAXIY1nFRiH8r9VGbzeOqyH9dKdPayLKVc+dWktRsc/A/1Sov+46ZtTtRdUB/ xUJNK1/Gz3eateO4IDozVXnf1oN0EuAtjmu4VOSsRljrle+pIg2MY1gNdURCin4zU6/k SWHI1+MuowwyS8UwE8c8lJi2kBSm5o++E57IM3Er2yNVdlacfF/ktRO8zJygZVEjM92l tQHJkbf5Mij5Ekw+QVL+Gt0PzXldNxNVNJ3WuoTmVy3hWs8vAndrvhY6XDAQ5mGgF4yv 5pUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=l3xG+lzh6Q2m2bU1b/cZLY6stz0OrJ21NrAPdv3f5lI=; b=YSf3RW5OWEhX9GEitrI5RbG5AIqPovU05OdhJ8xBjUZQdanyaPWnT5/3R/gVwInr2E Lnm24CaeiuhVXJ+qZ13h7/OJQauZBS/Z8VSDITpdbIPb0ELZ58nWJIVkjB+LxIBqR+JB lrEObX7Rw0/laeOx/R8zLuyB0+Sjfc2HuuFowWvR7avzMdZ7QVT4wwDgJnJDk1O2i6KO M3xAB8w3HcPTbYn725qyBjY8JlZJQHvfXjPlk7Rb53ZSbhJP2aFlDhDoW8pD4hTuFVDb kWAHZHwvBcMmpRsnu2GEicCgXIsgeN1wiSDbGl5AEbXpoKAQaLbJusNLPvD2lqapUEj7 V7uA== X-Gm-Message-State: AGi0PuahHYg3QIGCFSBeKF1yyzVLAjeiaT58ZrjyyDQSv9rWmNaiayqX mbeFLRe2j9a31EJPIeUH4BE= X-Google-Smtp-Source: APiQypL/TMoPg1xpQz/u+DpsCIjhLMV4V4JUwSAbK/9BffhqsJD2f8okxquuS6KlEyeqFYyh6hqOaQ== X-Received: by 2002:a2e:8887:: with SMTP id k7mr1909764lji.193.1585830783082; Thu, 02 Apr 2020 05:33:03 -0700 (PDT) Received: from pc636.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id u25sm3833494lfo.71.2020.04.02.05.33.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Apr 2020 05:33:01 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , "Paul E . McKenney" , Joel Fernandes Cc: RCU , linux-mm@kvack.org, Andrew Morton , Uladzislau Rezki , Steven Rostedt , Oleksiy Avramchenko Subject: [PATCH 1/3] rcu/tree: use more permissive parameters when attaching a head Date: Thu, 2 Apr 2020 14:32:51 +0200 Message-Id: <20200402123253.10382-1-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It is documneted that a headless object can be reclaimed from might_sleep() context only. Because of that when a head is dynamically attached it makes sense to drop the lock and do an allocation with much more permissve flags comparing if it is done from atomic context. That is why use GFP_KERNEL flag plus some extra ones which would make an allocation most likely to be succeed. The big advantage of doing so is a direct reclaim process. Tested such approach on my local tiny system with 145MB of ram(the minimum amount the KVM system is capable of booting) and 4xCPUs. For stressing the rcuperf module was used. During tests with difference combinations i did not observe any hit of our last emergency case, when synchronize_rcu() is involved. Please note, the "dynamically attaching" path was enabled only, apart of that all types of objects were considered as headless variant during testing. Signed-off-by: Uladzislau Rezki (Sony) Suggested-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 6172e6296dd7..24f620a06219 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3148,13 +3148,10 @@ static inline struct rcu_head *attach_rcu_head_to_object(void *obj) { unsigned long *ptr; + /* Try hard to get the memory. */ ptr = kmalloc(sizeof(unsigned long *) + - sizeof(struct rcu_head), GFP_NOWAIT | __GFP_NOWARN); - - if (!ptr) - ptr = kmalloc(sizeof(unsigned long *) + - sizeof(struct rcu_head), GFP_ATOMIC | __GFP_NOWARN); - + sizeof(struct rcu_head), GFP_KERNEL | + __GFP_ATOMIC | __GFP_HIGH | __GFP_RETRY_MAYFAIL); if (!ptr) return NULL; @@ -3222,9 +3219,20 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) if (!success) { /* Is headless object? */ if (head == NULL) { + /* Drop the lock. */ + if (krcp->initialized) + spin_unlock(&krcp->lock); + local_irq_restore(flags); + head = attach_rcu_head_to_object(ptr); if (head == NULL) - goto unlock_return; + goto inline_return; + + /* Take it back. */ + local_irq_save(flags); + krcp = this_cpu_ptr(&krc); + if (krcp->initialized) + spin_lock(&krcp->lock); /* * Tag the headless object. Such objects have a back-pointer @@ -3263,6 +3271,7 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) spin_unlock(&krcp->lock); local_irq_restore(flags); +inline_return: /* * High memory pressure, so inline kvfree() after * synchronize_rcu(). We can do it from might_sleep()