From patchwork Mon Mar 11 16:46:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC5C9C5475B for ; Mon, 11 Mar 2024 16:47:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 065296B00D1; Mon, 11 Mar 2024 12:46:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F30656B00D2; Mon, 11 Mar 2024 12:46:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA6A66B00D4; Mon, 11 Mar 2024 12:46:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C6B166B00D1 for ; Mon, 11 Mar 2024 12:46:53 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9B9AFC0508 for ; Mon, 11 Mar 2024 16:46:53 +0000 (UTC) X-FDA: 81885337506.19.5B310EB Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf28.hostedemail.com (Postfix) with ESMTP id E6834C001C for ; Mon, 11 Mar 2024 16:46:51 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b="e8tm8m/F"; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf28.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hkZD7ALZVm0P5UALUItr672MRcA6vth2fJOAHe6U29Q=; b=1yhTxT/vBGLnbXjrLdTbugThhzQ911rFgCh5gSt4ALhuLQW8y5JZjkmxN1BVizSK0V9BGU +rXq7audCYeGVBh4seNRIn5N+S63eKNeXIEtL7WGkEg7xVQurhw0fWWF72O7h5qVqON4Qi AU9CjU81/UNuH5Yj5XgmMVkz4htwTgs= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b="e8tm8m/F"; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf28.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175611; a=rsa-sha256; cv=none; b=i7ogrYHPYctHc3AnhfmzuEtw+1YUFFaV6vSFr/mwj84qjK+43rfWeZgIJuXngrOsXw0IxI 3SWnIrmP47GcF+WdpYVSsoXcWv1IZtZgaMR6SOIorauwsVnvqNgCNZJum0mKKdw0ZQlfIu GQJzmStrciOC+9hm4Z8xXQKdKfMshAA= Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-7884114feb8so119906985a.1 for ; Mon, 11 Mar 2024 09:46:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175611; x=1710780411; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=hkZD7ALZVm0P5UALUItr672MRcA6vth2fJOAHe6U29Q=; b=e8tm8m/FfWTPB+M14vv5sO2vfLcWl7sU+3fddkqoueR6qzCr+BjkzHXSJ0f7okd6qe u7813rXKkCsoArrTP06Ui3ouH4HFWJFGAs5H19wrIC64E8Kg285Z8hv8MgyxyKD256j6 78SQJLLPlmZ0L+zbucgsyrnjiKdo2JicpHr3oIbsz5GPe/XpYg1fYR1miu9jK3EGdCE6 eUeOOVxhdRsb5PouHPWZS3nB2SYtCXXVa2Ef3EQVSGL7S57XRSEQKSHSyNsLDZC77R9S /flLxfjHtz/cDpZ6V67vZfvs0BRuuQDRUYuhHE+x0tsRyMXEaLFVzpLVu7ogYoxankU3 44og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175611; x=1710780411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hkZD7ALZVm0P5UALUItr672MRcA6vth2fJOAHe6U29Q=; b=USrWmK2zL2BtZD5RMsKXhtlw8Gm1QfEUGXfOl+cBWA4DrXPsROMNf+JxwtdY+ekqnC SAwXRezMEvB/42G2vDwJjsNh+3XlvwA+JqB/DewrqU++Q3OvZZAZ3S7azV/eZousgI6S 3gM17YQel9MRAhzHUxb9Hk2EBJcM98lSq8yHoOSwcUVfGSQeP7oTZOy2Nz6H/su0SJA5 1/K3F4SYfvwTnty2cgXTYZjVsl7okALzkQBx9v4Hm61Hsz5NRaJ0CV55TJmJCNtqSDB9 grT1suUiNSRmpAinBk4dVwyJ3pF/IqoQxUkVxsVcVfJugSsFgMk51Kd+3EXogIR6YqAK /tgA== X-Forwarded-Encrypted: i=1; AJvYcCVpZriS3PChScAejTVj0l9MXVdcmBGXemL1mnzD6xIIAu5uK2PFN7Teqc8gsEaNYueM39ZiUYF1KXylVBiK9bW1HQo= X-Gm-Message-State: AOJu0YzL63ZlSmSHvT1KzOVis5w0b17wRc50EJQ4rnvAHnQqBREtR9Np 3g3Nw5A8KdG5mUrYU5irg2x5SDrLPVXMcenPtRCUU8Do1bL5LbeKMo0C3Ubug+c= X-Google-Smtp-Source: AGHT+IHTkS+2s/WrBF0S6nD4DTbIdKAkKUkplrTiXcc/c/8vBDolvcc8zD3FtIujCYDfupcwREXrKQ== X-Received: by 2002:a05:620a:15bc:b0:788:50aa:35f7 with SMTP id f28-20020a05620a15bc00b0078850aa35f7mr7780372qkk.30.1710175610969; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:50 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 08/14] fork: separate vmap stack alloction and free calls Date: Mon, 11 Mar 2024 16:46:32 +0000 Message-ID: <20240311164638.2015063-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E6834C001C X-Stat-Signature: 9gzgq34ar7exat1f4dehsguu5m8zsrch X-Rspam-User: X-HE-Tag: 1710175611-861947 X-HE-Meta: U2FsdGVkX18K1d3BQHHxXDwmDANEHf5RYkXsy2tZrK56L3FeFmVniltHu8R0iJECQF5c92vvzs6nbV//6oUqRR/v2e+PqCptci/pk3BqDtgO+hG+PWJlfPE9tiSRY0CRNO0trbKi6hWKJdnchA+XKi3ZXrECXgAAab1dJ9fL3gvgUsehFcXwT8Xp/9XKvyv8qS5C/fgSpTEh17LPZgaGvbcjS+FwiOldYmqlGdMAEYp0qfGg3Fxe/3qMItwnngywPEVhPXaxUgfxkEmoZGpo9HyS+FmFrXFm0sxu6bJO3Rh4WKF5S2CoVyZDYOrTfg9+5d1CoEd+Vt2Sz7aJT/0zidD9S9xNw2at2nQFSm2L6p997FOD/GzgTtIAwHHK6oii23MkPzHnBSuOqGd45nQHBWJmivhoEoXHsUcTpcuQvCMFaPHp49oQRHKgo3rYdgRTzM70KC7FEWssEYjW4nLeuQs6rDiGW0n3hd64IPka5+EI/eWvfJMXjphlWnb8j8T9Xtyv5gItkFBQim7LMmjn8loMIsK0JeNo/zcY1uc4/Rsjkny9B04mHActoCwsQIdiiQATIR4biUDOWIHKZRxpjpcnEM3VlA7ugvRrSiko06I9EHXSzQvZkJElYQjZlupiLl74yybNTtqiQbCulyuC8oZ6in566eux1sZLgJY3taXfPF+I+/zwd6Im5zEwKgv+zjSJCltjlej2teMyYIMn/IvAUhxWZTsDkXMn8wEph8crtQ0FgSen16PAV0QZsCcaTdEbj4EF/Aqw9GkqdF6Sf7SjawJOUqKCo+1usFZeXtwSzaDixh7FbbSkPHXFtWpqdeCE1cTE6fiIOsVsmCuyK+5p91fVjS7YfaGuaExrQbFJ1BiH+hsxAH6azWpfFXn8gkaEAU0bVMzw8C4VogGk5twJhGOuJkUG4E3YysgvTKyWDQpv9FPyMJI1ZByw97Kb1xKEenitaPOmEauL+2Y c2gsvM6O JNLtxFbVc6B9WM89CBNaRRJNej0zObh96uA9LSdmZ6FQH9KOMSjR6B7K6XLEnzMyL21a/7jOKSQcbAslIVwgM72/+waRw1Lm0iu+K5YBb4zLHfrt0ocEbFOoD94e+OafvESgt/WpBYh1P2ktQNof3o5WfYHyYMdRhVVkPP22QCDzLNUU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for the dynamic stacks, separate out the __vmalloc_node_range and vfree calls from the vmap based stack allocations. The dynamic stacks will use their own variants of these functions. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 53 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 22 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 3004e6ce6c65..bbae5f705773 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -204,6 +204,29 @@ static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area) return false; } +static inline struct vm_struct *alloc_vmap_stack(int node) +{ + void *stack; + + /* + * Allocated stacks are cached and later reused by new threads, + * so memcg accounting is performed manually on assigning/releasing + * stacks to tasks. Drop __GFP_ACCOUNT. + */ + stack = __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, + VMALLOC_START, VMALLOC_END, + THREADINFO_GFP & ~__GFP_ACCOUNT, + PAGE_KERNEL, + 0, node, __builtin_return_address(0)); + + return (stack) ? find_vm_area(stack) : NULL; +} + +static inline void free_vmap_stack(struct vm_struct *vm_area) +{ + vfree(vm_area->addr); +} + static void thread_stack_free_rcu(struct rcu_head *rh) { struct vm_stack *vm_stack = container_of(rh, struct vm_stack, rcu); @@ -212,7 +235,7 @@ static void thread_stack_free_rcu(struct rcu_head *rh) if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area)) return; - vfree(vm_area->addr); + free_vmap_stack(vm_area); } static void thread_stack_delayed_free(struct task_struct *tsk) @@ -235,7 +258,7 @@ static int free_vm_stack_cache(unsigned int cpu) if (!vm_area) continue; - vfree(vm_area->addr); + free_vmap_stack(vm_area); cached_vm_stacks[i] = NULL; } @@ -265,7 +288,6 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) { struct vm_struct *vm_area; int i, j, nr_pages; - void *stack; for (i = 0; i < NR_CACHED_STACKS; i++) { vm_area = this_cpu_xchg(cached_stacks[i], NULL); @@ -273,14 +295,13 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) continue; if (memcg_charge_kernel_stack(vm_area)) { - vfree(vm_area->addr); + free_vmap_stack(vm_area); return -ENOMEM; } /* Reset stack metadata. */ kasan_unpoison_range(vm_area->addr, THREAD_SIZE); - - stack = kasan_reset_tag(vm_area->addr); + tsk->stack = kasan_reset_tag(vm_area->addr); /* Clear stale pointers from reused stack. */ nr_pages = vm_area->nr_pages; @@ -288,26 +309,15 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) clear_page(page_address(vm_area->pages[j])); tsk->stack_vm_area = vm_area; - tsk->stack = stack; return 0; } - /* - * Allocated stacks are cached and later reused by new threads, - * so memcg accounting is performed manually on assigning/releasing - * stacks to tasks. Drop __GFP_ACCOUNT. - */ - stack = __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, - VMALLOC_START, VMALLOC_END, - THREADINFO_GFP & ~__GFP_ACCOUNT, - PAGE_KERNEL, - 0, node, __builtin_return_address(0)); - if (!stack) + vm_area = alloc_vmap_stack(node); + if (!vm_area) return -ENOMEM; - vm_area = find_vm_area(stack); if (memcg_charge_kernel_stack(vm_area)) { - vfree(stack); + free_vmap_stack(vm_area); return -ENOMEM; } /* @@ -316,8 +326,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) * so cache the vm_struct. */ tsk->stack_vm_area = vm_area; - stack = kasan_reset_tag(stack); - tsk->stack = stack; + tsk->stack = kasan_reset_tag(vm_area->addr); return 0; }