From patchwork Sun Dec 1 01:54:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11268353 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0A87112B for ; Sun, 1 Dec 2019 01:54:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 73CAF20880 for ; Sun, 1 Dec 2019 01:54:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="w/RYgsD6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73CAF20880 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 33A6D6B0307; Sat, 30 Nov 2019 20:54:36 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 310736B0309; Sat, 30 Nov 2019 20:54:36 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24CB36B030A; Sat, 30 Nov 2019 20:54:36 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0201.hostedemail.com [216.40.44.201]) by kanga.kvack.org (Postfix) with ESMTP id 104A66B0307 for ; Sat, 30 Nov 2019 20:54:36 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id BBB1E824999B for ; Sun, 1 Dec 2019 01:54:35 +0000 (UTC) X-FDA: 76214903310.20.dolls76_2810f5e627901 X-Spam-Summary: 2,0,0,69ff1056bbbd97a7,d41d8cd98f00b204,akpm@linux-foundation.org,:akpm@linux-foundation.org:bigeasy@linutronix.de:dwagner@suse.de:hdanton@sina.com::mhocko@suse.com:mm-commits@vger.kernel.org:oleksiy.avramchenko@sonymobile.com:peterz@infradead.org:rostedt@goodmis.org:tglx@linutronix.de:torvalds@linux-foundation.org:urezki@gmail.com:willy@infradead.org,RULES_HIT:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1381:1431:1437:1535:1544:1605:1711:1730:1747:1777:1792:2195:2196:2199:2200:2393:2525:2559:2563:2682:2685:2693:2732:2859:2902:2903:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3151:3503:3504:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:5007:6261:6653:6737:7514:7576:7903:8599:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12291:12295:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13229:13846:14096:14181:14721:14819:14849:21080:21433 :21451:2 X-HE-Tag: dolls76_2810f5e627901 X-Filterd-Recvd-Size: 5903 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Sun, 1 Dec 2019 01:54:35 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2AAD6217AB; Sun, 1 Dec 2019 01:54:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1575165274; bh=IVJpqWQYhswBsHF9el9MGqjNB8khvPveAEIB7B/vAr8=; h=Date:From:To:Subject:From; b=w/RYgsD6ZVertR5BgR0PQNbzPn10Z4eOuRnCji6IIodKAURssbLGrNoYhGc4QGgAY wnmKyfWyW5l/kIFgEMgDtF4dyIFKr227NhA8bDov4rOpzo9ecs3FS29FKQH9RfpKD9 3wIACaGubpuGoLhLAqwP3uEGBIuem19bk4py3ci4= Date: Sat, 30 Nov 2019 17:54:33 -0800 From: akpm@linux-foundation.org To: akpm@linux-foundation.org, bigeasy@linutronix.de, dwagner@suse.de, hdanton@sina.com, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, oleksiy.avramchenko@sonymobile.com, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, torvalds@linux-foundation.org, urezki@gmail.com, willy@infradead.org Subject: [patch 088/158] mm/vmalloc: remove preempt_disable/enable when doing preloading Message-ID: <20191201015433.nDqmkbg4_%akpm@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Subject: mm/vmalloc: remove preempt_disable/enable when doing preloading Some background. The preemption was disabled before to guarantee that a preloaded object is available for a CPU, it was stored for. That was achieved by combining the disabling the preemption and taking the spin lock while the ne_fit_preload_node is checked. The aim was to not allocate in atomic context when spinlock is taken later, for regular vmap allocations. But that approach conflicts with CONFIG_PREEMPT_RT philosophy. It means that calling spin_lock() with disabled preemption is forbidden in the CONFIG_PREEMPT_RT kernel. Therefore, get rid of preempt_disable() and preempt_enable() when the preload is done for splitting purpose. As a result we do not guarantee now that a CPU is preloaded, instead we minimize the case when it is not, with this change, by populating the per cpu preload pointer under the vmap_area_lock. This implies that at least each caller that has done the preallocation will not fallback to an atomic allocation later. It is possible that the preallocation would be pointless or that no preallocation is done because of the race but the data shows that this is really rare. For example i run the special test case that follows the preload pattern and path. 20 "unbind" threads run it and each does 1000000 allocations. Only 3.5 times among 1000000 a CPU was not preloaded. So it can happen but the number is negligible. [mhocko@suse.com: changelog additions] Link: http://lkml.kernel.org/r/20191016095438.12391-1-urezki@gmail.com Fixes: 82dd23e84be3 ("mm/vmalloc.c: preload a CPU with one object for split purpose") Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Steven Rostedt (VMware) Acked-by: Sebastian Andrzej Siewior Acked-by: Daniel Wagner Acked-by: Michal Hocko Cc: Hillf Danton Cc: Matthew Wilcox Cc: Oleksiy Avramchenko Cc: Peter Zijlstra Cc: Thomas Gleixner Signed-off-by: Andrew Morton --- mm/vmalloc.c | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) --- a/mm/vmalloc.c~mm-vmalloc-remove-preempt_disable-enable-when-do-preloading +++ a/mm/vmalloc.c @@ -1077,31 +1077,34 @@ static struct vmap_area *alloc_vmap_area retry: /* - * Preload this CPU with one extra vmap_area object to ensure - * that we have it available when fit type of free area is - * NE_FIT_TYPE. + * Preload this CPU with one extra vmap_area object. It is used + * when fit type of free area is NE_FIT_TYPE. Please note, it + * does not guarantee that an allocation occurs on a CPU that + * is preloaded, instead we minimize the case when it is not. + * It can happen because of cpu migration, because there is a + * race until the below spinlock is taken. * * The preload is done in non-atomic context, thus it allows us * to use more permissive allocation masks to be more stable under - * low memory condition and high memory pressure. + * low memory condition and high memory pressure. In rare case, + * if not preloaded, GFP_NOWAIT is used. * - * Even if it fails we do not really care about that. Just proceed - * as it is. "overflow" path will refill the cache we allocate from. + * Set "pva" to NULL here, because of "retry" path. */ - preempt_disable(); - if (!__this_cpu_read(ne_fit_preload_node)) { - preempt_enable(); - pva = kmem_cache_alloc_node(vmap_area_cachep, GFP_KERNEL, node); - preempt_disable(); + pva = NULL; - if (__this_cpu_cmpxchg(ne_fit_preload_node, NULL, pva)) { - if (pva) - kmem_cache_free(vmap_area_cachep, pva); - } - } + if (!this_cpu_read(ne_fit_preload_node)) + /* + * Even if it fails we do not really care about that. + * Just proceed as it is. If needed "overflow" path + * will refill the cache we allocate from. + */ + pva = kmem_cache_alloc_node(vmap_area_cachep, GFP_KERNEL, node); spin_lock(&vmap_area_lock); - preempt_enable(); + + if (pva && __this_cpu_cmpxchg(ne_fit_preload_node, NULL, pva)) + kmem_cache_free(vmap_area_cachep, pva); /* * If an allocation fails, the "vend" address is