From patchwork Wed Oct 16 09:54:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11192793 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 367371390 for ; Wed, 16 Oct 2019 09:54:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ED96421848 for ; Wed, 16 Oct 2019 09:54:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QE5ek4pV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED96421848 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3EC8F8E0007; Wed, 16 Oct 2019 05:54:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 34EDC8E0001; Wed, 16 Oct 2019 05:54:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23D588E0007; Wed, 16 Oct 2019 05:54:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0034.hostedemail.com [216.40.44.34]) by kanga.kvack.org (Postfix) with ESMTP id E74108E0001 for ; Wed, 16 Oct 2019 05:54:50 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 84C0C183D71E7 for ; Wed, 16 Oct 2019 09:54:50 +0000 (UTC) X-FDA: 76049188740.15.shoes16_646742778555c X-Spam-Summary: 2,0,0,93797cc7bac2900e,d41d8cd98f00b204,urezki@gmail.com,:akpm@linux-foundation.org:dwagner@suse.de:bigeasy@linutronix.de:tglx@linutronix.de::linux-kernel@vger.kernel.org:peterz@infradead.org:urezki@gmail.com:hdanton@sina.com:mhocko@suse.com:willy@infradead.org:oleksiy.avramchenko@sonymobile.com:rostedt@goodmis.org,RULES_HIT:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1437:1515:1535:1543:1605:1711:1730:1747:1777:1792:2195:2196:2199:2200:2393:2559:2562:2693:2732:2895:2903:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4117:4250:4321:4385:5007:6261:6653:6742:7514:7875:7903:8603:9413:10004:11026:11473:11658:11914:12043:12291:12295:12296:12297:12438:12517:12519:12555:12679:12683:12895:13095:13161:13229:13894:14096:14181:14394:14687:14721:14819:21080:21433:21444:21451:21627:21666:21966:30003:30034:30054:30070,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck: none,Dom X-HE-Tag: shoes16_646742778555c X-Filterd-Recvd-Size: 6856 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Wed, 16 Oct 2019 09:54:49 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id m7so23331466lji.2 for ; Wed, 16 Oct 2019 02:54:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=bdoCE4EoxSLapTmXd4+oVGs4R70YFCwt9DjIODBWD6I=; b=QE5ek4pVPZfLzXJzV28P6nJ+0ySzes/vJ33aguYDmlDCotZp11VGmfub55bpIPwULu 445YzJByyi9BRAIfThepb5GU9gtU2sJw66rXQq2NXUn/K4dZOUAT1hnRMFbHniqB+JiG weY5ynwlgmCL0ItKNN1Ahe0Uxaw+QL7EQRIy+gWWoG2biYMDsmURA9GzkizOBx9Q5MAR GlB4hTBjI240TBfNo3FXNft53oV6xc4OyN+9u6a7TZYNCfNM55vYO4oSixwfV6A2LWAu ho2FLuaKimvVybjSGJRrV8X58Dg13nlrNVPA5wSaWe7KAy3j8b6QuhgnmpgQs7k4Bdn1 mP8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=bdoCE4EoxSLapTmXd4+oVGs4R70YFCwt9DjIODBWD6I=; b=UGNH4WP/0ozm5yWsvlgGwK41dGicoVb3Oj0vyGp+gzAuBqS86fGEkSTln6Ch+EYn6w UEycB6qJZ3UIZII2tKYAFyp+TyhHdPB3S6ZgQ4tozqTI3SByNoxpv/2ea87Y8BRk7zbS 56pE8bTviSTeqbHYVJ0l6PQBmJxQChCn8DTy2pKCMOK0WXV3ufIVrBVXbCABCOsLuGTU 8nJvlxBBGIMZstn+Jo9GW/d26KSLP5EOWlIW7Iugqft8wre9DDqihPg6q7/etOiA7Dhy OGiHjBqNldr2S46nx3MHvOXezrxt640jNKfpqjU8hdjojVk90b1oPGscDwjQmNTkL1os ZOcw== X-Gm-Message-State: APjAAAVjdgPzcoYrxypF8L9AE6Oeat68nf2ZhUsgp9j58kRwPWKgdE6K XeLE2fs1TzmELrP9iaNf4OQ= X-Google-Smtp-Source: APXvYqxiDS53+QyM7r3Br5Xxnwu0u7VAzS+Uv9ERxHSO38DcMbzVXfh6Mfhp0Z22QaporVWAcWkmBQ== X-Received: by 2002:a2e:9d83:: with SMTP id c3mr25563346ljj.237.1571219688236; Wed, 16 Oct 2019 02:54:48 -0700 (PDT) Received: from pc636.semobile.internal ([37.139.158.167]) by smtp.gmail.com with ESMTPSA id b2sm886452lfq.27.2019.10.16.02.54.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Oct 2019 02:54:47 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: Daniel Wagner , Sebastian Andrzej Siewior , Thomas Gleixner , linux-mm@kvack.org, LKML , Peter Zijlstra , Uladzislau Rezki , Hillf Danton , Michal Hocko , Matthew Wilcox , Oleksiy Avramchenko , Steven Rostedt Subject: [PATCH v3 1/3] mm/vmalloc: remove preempt_disable/enable when do preloading Date: Wed, 16 Oct 2019 11:54:36 +0200 Message-Id: <20191016095438.12391-1-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some background. The preemption was disabled before to guarantee that a preloaded object is available for a CPU, it was stored for. The aim was to not allocate in atomic context when spinlock is taken later, for regular vmap allocations. But that approach conflicts with CONFIG_PREEMPT_RT philosophy. It means that calling spin_lock() with disabled preemption is forbidden in the CONFIG_PREEMPT_RT kernel. Therefore, get rid of preempt_disable() and preempt_enable() when the preload is done for splitting purpose. As a result we do not guarantee now that a CPU is preloaded, instead we minimize the case when it is not, with this change. For example i run the special test case that follows the preload pattern and path. 20 "unbind" threads run it and each does 1000000 allocations. Only 3.5 times among 1000000 a CPU was not preloaded. So it can happen but the number is negligible. V2 - > V3: - update the commit message V1 -> V2: - move __this_cpu_cmpxchg check when spin_lock is taken, as proposed by Andrew Morton - add more explanation in regard of preloading - adjust and move some comments Fixes: 82dd23e84be3 ("mm/vmalloc.c: preload a CPU with one object for split purpose") Reviewed-by: Steven Rostedt (VMware) Acked-by: Sebastian Andrzej Siewior Acked-by: Daniel Wagner Signed-off-by: Uladzislau Rezki (Sony) Acked-by: Michal Hocko --- mm/vmalloc.c | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e92ff5f7dd8b..b7b443bfdd92 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1078,31 +1078,34 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, retry: /* - * Preload this CPU with one extra vmap_area object to ensure - * that we have it available when fit type of free area is - * NE_FIT_TYPE. + * Preload this CPU with one extra vmap_area object. It is used + * when fit type of free area is NE_FIT_TYPE. Please note, it + * does not guarantee that an allocation occurs on a CPU that + * is preloaded, instead we minimize the case when it is not. + * It can happen because of cpu migration, because there is a + * race until the below spinlock is taken. * * The preload is done in non-atomic context, thus it allows us * to use more permissive allocation masks to be more stable under - * low memory condition and high memory pressure. + * low memory condition and high memory pressure. In rare case, + * if not preloaded, GFP_NOWAIT is used. * - * Even if it fails we do not really care about that. Just proceed - * as it is. "overflow" path will refill the cache we allocate from. + * Set "pva" to NULL here, because of "retry" path. */ - preempt_disable(); - if (!__this_cpu_read(ne_fit_preload_node)) { - preempt_enable(); - pva = kmem_cache_alloc_node(vmap_area_cachep, GFP_KERNEL, node); - preempt_disable(); + pva = NULL; - if (__this_cpu_cmpxchg(ne_fit_preload_node, NULL, pva)) { - if (pva) - kmem_cache_free(vmap_area_cachep, pva); - } - } + if (!this_cpu_read(ne_fit_preload_node)) + /* + * Even if it fails we do not really care about that. + * Just proceed as it is. If needed "overflow" path + * will refill the cache we allocate from. + */ + pva = kmem_cache_alloc_node(vmap_area_cachep, GFP_KERNEL, node); spin_lock(&vmap_area_lock); - preempt_enable(); + + if (pva && __this_cpu_cmpxchg(ne_fit_preload_node, NULL, pva)) + kmem_cache_free(vmap_area_cachep, pva); /* * If an allocation fails, the "vend" address is