From patchwork Mon Mar 11 16:46:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF336C54E58 for ; Mon, 11 Mar 2024 16:46:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08AB46B00CA; Mon, 11 Mar 2024 12:46:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 012356B00CB; Mon, 11 Mar 2024 12:46:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D585B6B00CC; Mon, 11 Mar 2024 12:46:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C73536B00CA for ; Mon, 11 Mar 2024 12:46:48 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 934141203EC for ; Mon, 11 Mar 2024 16:46:48 +0000 (UTC) X-FDA: 81885337296.08.B4801DF Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) by imf08.hostedemail.com (Postfix) with ESMTP id F0D1616002A for ; Mon, 11 Mar 2024 16:46:46 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=QiN85uq8; spf=pass (imf08.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.167.171 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175607; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4lhjThGfB91n24i50IPCohz8k1KRXdDof+Lt/uTAI2U=; b=U/tk8nhuvYGrKlkjk5LuvYIMwbDh+3YBdpnUa1/9YlamAfgg55nWuyGz/SZ/9bSiwZgJ91 AcevNR2jD+5ImnJrsKmChaMx8teWfYSMtogJ7QHbHYjq4nxSBlJJ4UJyGYvbMzwX8IaB73 hDZf/k47Upv4WjEbvDrYz/KP3Hi3Z7s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175607; a=rsa-sha256; cv=none; b=lv7YIe0/Y3Syb4JcnMLuiuOfA46Xjyt1O4QcLI+0y+AVmXo51nemw4ymBYUBSCba3Ey6Q0 9iLX169JR5xf0k5/docSSu75P6kolOTdAbzkAptI1N7YAr/PIf+e+qWCrGyD13eYaiujcy iRr32ik2P7u7Jj6EmskL2W0FnAnuMHc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=QiN85uq8; spf=pass (imf08.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.167.171 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3c1ea5e54d3so2666753b6e.0 for ; Mon, 11 Mar 2024 09:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175606; x=1710780406; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=4lhjThGfB91n24i50IPCohz8k1KRXdDof+Lt/uTAI2U=; b=QiN85uq8UaYli18XK5rBGA+FSRzp3hOXZf591SCilHnDZwKS4m96nMqQbAcDvyVDXc H8D9v97Hco9O0/ihHiuLCSY03x0XbvvXxyWt+ytcieSuVrJA5hYrQy05AcH9mZF/xZDB UJFRdsmL3aO0XSY3AZC2cZCvtgHzO/jhhc6IsNsBjA+NfwOk0gopOmT+6C0hVQxcyYo9 ruKLS/HvAhVAUSGNpI4JrIwLR9DuaKJxTzaY2vkPR6YTzcinvFOoMxVUXpLDE8DrbAyO fN/YNjPKFVwPCvMvzGk/rYz1axls4/rxzBPTtTmRauoGZ4zeZXQ6QTXNeC5fCDXoVXF+ +dXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175606; x=1710780406; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4lhjThGfB91n24i50IPCohz8k1KRXdDof+Lt/uTAI2U=; b=acgH52WkgrOh+zFYwuo7kD5h+O/5W/2GzThEI3C+079XtmHcqFAd/jOXrbX7siO6nk yvu9WQwbz1ZkUYnSn5Dh08ktilqD9xHSQ4gNhv0gHANdnbKdYMFcSjA2eEokeEdlwgdn 1sH055seuS9OoR97DdcaFsDtDZQ5u9CwykqYSLj6aebexIZOhL1hiP18ystbtiYU5BR7 iRwaDcvvjlSAX2ZXqfU1yE2vCgXmoEDm1B72E3/vUdS2SSYucBGzQBAtbATOV1mfN7fO SaOObTPrl4ti9c7qNnBlSZCqXRwkZIulA0Jv1v6HJYofWbyG2drz1o06hYLh01Oh2Sow JO+w== X-Forwarded-Encrypted: i=1; AJvYcCW/UW7cwighXhCMM70VGgMTCmPBj+ADLFoqMS5qoFmSsI7XMnmnizj+PDiCmRyj3jxteOLPpt/bkGCLgJsRGjeZKh4= X-Gm-Message-State: AOJu0YxzfToeE8CLd/urPhZRP9kr2q9fvecPQG8omviGns/12WaYHH6a 84jCSBInSkbRErhaEF7MDZteDv3WOZkBkj6Yolc9Kbykg/MQabBz9JW40O3DVfvCOzjHt2Ph4fH BcX0= X-Google-Smtp-Source: AGHT+IEjg4Cv5VbQxeFt499tiStDwxoTwSUC+FAOlU+MVjKFmojDB4eSoAwzE63CJMnyaPIAjMQShA== X-Received: by 2002:a05:6808:ecd:b0:3c2:39d1:f105 with SMTP id q13-20020a0568080ecd00b003c239d1f105mr8453793oiv.11.1710175604451; Mon, 11 Mar 2024 09:46:44 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:44 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 01/14] task_stack.h: remove obsolete __HAVE_ARCH_KSTACK_END check Date: Mon, 11 Mar 2024 16:46:25 +0000 Message-ID: <20240311164638.2015063-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: 1iy13sra694h5e5x4k43nnkeg8noycfn X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: F0D1616002A X-Rspam-User: X-HE-Tag: 1710175606-26452 X-HE-Meta: U2FsdGVkX187xJJNtEX/nXMlkwJaBoq9Q//zcKKViNuxQ+q+nhg1u0Q2FtwWmV+r6ZmkDHQARPJoxur0ORqZ+VNoVvXOJX2OXId5Il8tKuPcFSFlWkbSGH29+7qpq7ADa/A7q45q7Xrjy0VMxlnpXwUqyBgT2K/Ho/hN1QD9iFPizI9dDkwShEC9eK3DcT3AQq0SpEtFBYGqv3JKwwWUx8CXaWqpHCqZVr9X9wkE0KwNgNfx2Jsr6bd4QGV86AAll/q1Zk0yyOrVFj1+Gh5sv1LYc+0/hbV2hgD+XhS/afQ4AgRV1vKUUL0ZzLt9g/A89zD1NKEVWveHAkoBq5uwwfIf35SwPsF6PWyxbNYm8w4j53jhwcQN9J8xxgb070Bmx9qWIJFINOpPKL8vnzwbka2/zTe2VNg0BszoxqDdOR0eCTngt1B0RFLE846n63noqNOTUcWvW6oc0fpIhy3WRuzwdA2rUca9NyRsDV50GwaUeiSbXY5PrRppEmNdQxOFRu8PuxoTKB4AwWLhPvmOlOKA9ePKdS7S+3xtB+hnlmndN+wgiS61ZawI3IgeMBERgObpXGMAJZC/SL7fdnw6DxX+j2UKXqIOSEgMzKJbPZwrdM9Ty29s15dpF0/KsPbSwVcUZ+x7sT78C//CXwF5rTlqQkNtvLWppNYHXk+FWBtKGzlCJh6v3Y4xXLkc+nIeKqbGfbHC2CqzpNBQqv9yznw9dVrnsovRqseO9RDczP+ZFJNhAnwjhq18jFIeIoqnCq4k/hXFEbl0KEP1IEYBOGlzvVZRlgv0TN6bz1dAkAozNPjfsHx5n745gyDvG7bzafsedMtMjpbu/ThSQDa6LC8HhHrkubRt7gSLrsOxNVM6tfC9HFuE+VrupY7yRaJdn+IwycY0+DTkT18GZXVHFGLtWpwMRww+uIST0wVqElXoHWVr/uKumRBpSZryjI4Ce/ZfJdrRdG3nG630cmw O0FQ0Mdi nQ9Wp/9AfFerxuRo8ojyWrF4YG3ZdhYPg/zdtT0zravfLZTk5B42EHiq7uyIpev96YMJQyVypAiRcR6GnVfg4/EVEpIkLR4Z5MqLRtoGX5f02yeA3xI5b/+26LPbpO0lR99FlquCeauNn+zsGJ5M7saug1MxpQbf+6b8mPYouZfI3F90rn4lwoyaeDRV3ZcJvYYYb+mEVXGVfIF0Fbr5zw3g5K2sRGBaQxwSH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000016, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove __HAVE_ARCH_KSTACK_END as it has been osolete since removal of metag architecture in v4.17. Signed-off-by: Pasha Tatashin --- include/linux/sched/task_stack.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_stack.h index ccd72b978e1f..860faea06883 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -116,7 +116,6 @@ static inline unsigned long stack_not_used(struct task_struct *p) #endif extern void set_task_stack_end_magic(struct task_struct *tsk); -#ifndef __HAVE_ARCH_KSTACK_END static inline int kstack_end(void *addr) { /* Reliable end of stack detection: @@ -124,6 +123,5 @@ static inline int kstack_end(void *addr) */ return !(((unsigned long)addr+sizeof(void*)-1) & (THREAD_SIZE-sizeof(void*))); } -#endif #endif /* _LINUX_SCHED_TASK_STACK_H */ From patchwork Mon Mar 11 16:46:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FD10C5475B for ; Mon, 11 Mar 2024 16:46:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8ED586B00C9; Mon, 11 Mar 2024 12:46:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 89E946B00CA; Mon, 11 Mar 2024 12:46:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A1F36B00CB; Mon, 11 Mar 2024 12:46:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5BC476B00C9 for ; Mon, 11 Mar 2024 12:46:48 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2A9AB160B0B for ; Mon, 11 Mar 2024 16:46:48 +0000 (UTC) X-FDA: 81885337296.09.B85A2A6 Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) by imf08.hostedemail.com (Postfix) with ESMTP id 3676516000E for ; Mon, 11 Mar 2024 16:46:46 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=ymJub6yR; spf=pass (imf08.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.45 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175606; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uRKHBCRgEDsAkh8Uu5g0LvaNrGLtTcS4Fv1I3hMOqvE=; b=odjG0DR8vZhTzPvb3gmN7Gw2OxgZgFCQoo03YPFji4af8TopgnOUt81+hItX837mXRzK91 JyB5Tb+WsG2+zejiYhkMfmCgU2MvthFykhflaLMCSpDhdoyuNytn4UFUk2sTRolMBpIGpW IjSwJl0eBtnHYIw/xQ4bO6kUfH3trb4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175606; a=rsa-sha256; cv=none; b=MkY/p/H9GBDYlE8bfxWHMs37/ZfJBMLmkF33gF4DeqN3OdiGaEmBPJd7aTuFRaud5prq5r g3ZquXv6vsGfD2myl056ukJ/GWQiohkYuT0U9khNpTMWgNEKC0wq8RdkZDorF8yupIMGYX Zvshntkkyozuzyq4L3t9CeMRi/1m3zw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=ymJub6yR; spf=pass (imf08.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.45 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-22181888b88so3895613fac.1 for ; Mon, 11 Mar 2024 09:46:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175605; x=1710780405; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=uRKHBCRgEDsAkh8Uu5g0LvaNrGLtTcS4Fv1I3hMOqvE=; b=ymJub6yRVXXlsScvjDvtNJ/xmeA+6ueuUB2d35MzSSCdi9YZkNI6qzxhOq+fDAFPFb VMvZKY2eMvAffWyb5EKq4sT48+uZOVlw6jabdwbBXzRBM9DHKVPBo+JmuknwHYXerkyi Yv/QE+M+2JjQuAahP+KJj1CIdPRMiWze1lAcITfo8It9++IuMjVNjf+mkRdR7FNlm/4k mCtM28YwATKorvqGJhWhaQ1WXRoQCd2IYmWPg/q8sL2gF8iWe+/MdQjSU5Xtq3cY7Ne3 rdBBNo6qneOaDbhL2AQXBhxr0Y+7P+Do5ygKQeB2G0eAU3lpcdtG1KLayTvTLRk16B3C XXkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175605; x=1710780405; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uRKHBCRgEDsAkh8Uu5g0LvaNrGLtTcS4Fv1I3hMOqvE=; b=Fl5h+9AD3hLlmzUZYpcaoaj5fU6IUSYcx6rB342rfc2aqtOQqdoO1ruik6CRvd55wR 5QHETdsvjTQlCMNGC+W3Ex5e/PZLuz/3L85mZdxBrlsk/8vF0msxTik40oYidQffAfpl FB4S+earWptlgABi3H0P/5J0uuTvMGyLGZr36m8ZQFiOy4lQd1yur8xhMBsPkBhWdZna 708Ynp6uNBPpQED01CS6wGGF2MrvgVNYHRKVOJTPnJkUfrfAK6wQlePymQKThMAtbNHP CiL+ooaPz7THu6Dxgoc5GdXNPDX6QalBjhqDl64rbM07TA2LJTDtNMBIsQILFXxBevmQ tmRQ== X-Forwarded-Encrypted: i=1; AJvYcCUa+u2mdTR7znXK2rkJooHhu8h/8uvpu6GOiKhSeJMv2QEcozNM+LeaIKzRQzXAoE2LH38lmPcBdJbS/uCaMrH7xXs= X-Gm-Message-State: AOJu0YzfBS+K41mlZQMHwsAwvAGa+2qJUDkcip7t8po9ASIbaDWlhK02 rMPkjcGKRup17H0iHAYkM64c7iyBnQVRV/T0O/DLBQ9XO5bzxPwtPvltgBXy/pc= X-Google-Smtp-Source: AGHT+IGB8z9Eh2EiOSMQqnmRI9zZB59VQQwcHf8aThINC/aQYO0+ESVp0WJ0ORdeFBqZ+2a6LmE9PQ== X-Received: by 2002:a05:6870:4721:b0:21f:4fe:9ee8 with SMTP id b33-20020a056870472100b0021f04fe9ee8mr7898763oaq.4.1710175605333; Mon, 11 Mar 2024 09:46:45 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:45 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 02/14] fork: Clean-up ifdef logic around stack allocation Date: Mon, 11 Mar 2024 16:46:26 +0000 Message-ID: <20240311164638.2015063-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3676516000E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: yxwuw8iikm3otm9a5wgi3un4uogdkk4a X-HE-Tag: 1710175606-507001 X-HE-Meta: U2FsdGVkX1/DpyUGy45Rl+A37lKK9EkGmUb4V9v5QtWSZxIQDlglgEm9GVPIt3WWS8XqZ3x0Zp/CnsOJ1rzF/VWIB6G1fzkH3qCFfxQ3Jj4bs3fJLeJGuhwTtVeztFuJhyYq04QLRQqcw9E4E/KuzdZIq/ZjPGl1W28mlaVvBHcFv4on9wRuFB7ojfwooniT9w7BPaIS1B+e/qYduKu8XoNGz5hwn3m38IwkYRqWa2WxDHZCtTYAmi9u6b2O5dfIqBS0gLpKP9pRCujOD6KjLO8W9JlRdA5i4RECl5VLnO0qA6R3iIyjAX04kz6GaRvYUfqzCUhe0djxULYppgU55h2KvVZXIzGpdcRQ4pCBUDahQmYyVrjnaw3ss+xTSqtjGHdXPiV+j3nrjyYzsGTfK+334m+mZ5tBFjoZsHiLBbzjD3Jx0bmiVo2nI1Ik2FG+6r9K8cUtSOt4HKmF8eoQk4L5K7DTazT9wvTCLZYxyk5MTvi2V+QqoDTFb2CdcSCz4SDmXA2GoYzMMRAMrng+U/Oh0eZL/jyn/xbAD1I4Q6wcLMzqXr1kjI5u3CANojimUAafAj1scqQj6pYrmMYKveEF79+LYTlTpva4K01jW25Rm8Z/FA9TZGvdyVj5DtVxWruMAsZrNzxOVP4A3J9AllyaBarB17tJva5uomoddy+4/k03Qvvvdq4GIq3c6sp003mcyUB9oXMLouDA6JB3n5++uOnh4sX2qkOZABaI0KTkPq1ejjAhbKdbRvswhhi8ehIKTGKK5PD2UBqRBFQk91Ue5g1zzC2JwRSTPG8sdmn8UK005Ca6d3ObQtjQrGXlH/9rFB4pwSbbJc270APCV94Q/5TUtYQfqBH0KgdgIdV3k2g44i9JJ3atc5JxYmpXtqe6HoPzjE2jodIqRm6wVqKN5ynPn59U+AGmVG7vgqEct1e8kevpiIEWm8mxPgfo2/59pAxu/+y36q8P8lv 5Olta55b uT9ssLg9t93D1/60M0fZXpwUGBpVNnqp4oWZFzj4jXHVJPS1dC5lnWIasNRY7mZAAfwhzlOvNunun2rInLc+BQ39Z2FJsQwxQ2eP71rjizwmz2Gj+67Hxs/W3+p0F7ATTUAvbCPpmVsKttkq+NtQj0r70yLNbdPqfyKeDc8DTPXxWzC+CGjnQjq7DU73/Y64384YuneGDVshXQG/rmv/QUkDp3LSbU/f5Bs5j2uh/yhEEfAfuVzfEIw/afqTgh64TR8Dr7LMsZibMdrdr2MhLfB5DlGSrTjlhN6/WMroSCtNDuoMcB4nQQguWWwlM6gxIe1EhVdIArK61S/EJjKbM3gynBEjqCsHGuQ5FLPPJ+o8m7SwDCUx4pXG/PKsyAER1Yaz3qTgBm3ki76nts9emp97ouYNL+eq6Ii68JL7DJx4Tha2n6/p4bTmEhTvQnxi5iM0DfN+Sp3J53+Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is unneeded OR in the ifdef functions that are used to allocate and free kernel stacks based on direct map or vmap. Adding dynamic stack support would complicate this logic even further. Therefore, clean up by Changing the order so OR is no longer needed. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 0d944e92a43f..32600bf2422a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -179,13 +179,7 @@ static inline void free_task_struct(struct task_struct *tsk) kmem_cache_free(task_struct_cachep, tsk); } -/* - * Allocate pages if THREAD_SIZE is >= PAGE_SIZE, otherwise use a - * kmemcache based allocator. - */ -# if THREAD_SIZE >= PAGE_SIZE || defined(CONFIG_VMAP_STACK) - -# ifdef CONFIG_VMAP_STACK +#ifdef CONFIG_VMAP_STACK /* * vmalloc() is a bit slow, and calling vfree() enough times will force a TLB * flush. Try to minimize the number of calls by caching stacks. @@ -337,7 +331,13 @@ static void free_thread_stack(struct task_struct *tsk) tsk->stack_vm_area = NULL; } -# else /* !CONFIG_VMAP_STACK */ +#else /* !CONFIG_VMAP_STACK */ + +/* + * Allocate pages if THREAD_SIZE is >= PAGE_SIZE, otherwise use a + * kmemcache based allocator. + */ +#if THREAD_SIZE >= PAGE_SIZE static void thread_stack_free_rcu(struct rcu_head *rh) { @@ -369,8 +369,7 @@ static void free_thread_stack(struct task_struct *tsk) tsk->stack = NULL; } -# endif /* CONFIG_VMAP_STACK */ -# else /* !(THREAD_SIZE >= PAGE_SIZE || defined(CONFIG_VMAP_STACK)) */ +#else /* !(THREAD_SIZE >= PAGE_SIZE) */ static struct kmem_cache *thread_stack_cache; @@ -409,7 +408,8 @@ void thread_stack_cache_init(void) BUG_ON(thread_stack_cache == NULL); } -# endif /* THREAD_SIZE >= PAGE_SIZE || defined(CONFIG_VMAP_STACK) */ +#endif /* THREAD_SIZE >= PAGE_SIZE */ +#endif /* CONFIG_VMAP_STACK */ /* SLAB cache for signal_struct structures (tsk->signal) */ static struct kmem_cache *signal_cachep; From patchwork Mon Mar 11 16:46:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A465CC54E58 for ; Mon, 11 Mar 2024 16:46:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1284E6B00CC; Mon, 11 Mar 2024 12:46:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B0E36B00CD; Mon, 11 Mar 2024 12:46:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF4F46B00CE; Mon, 11 Mar 2024 12:46:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CB2D86B00CC for ; Mon, 11 Mar 2024 12:46:49 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9121D140A6A for ; Mon, 11 Mar 2024 16:46:49 +0000 (UTC) X-FDA: 81885337338.27.1964A32 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) by imf07.hostedemail.com (Postfix) with ESMTP id 1FF604001E for ; Mon, 11 Mar 2024 16:46:46 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=AWN84jIn; spf=pass (imf07.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.174 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175607; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HoufURnUHz+cr/acEvSc8ejQyNfvHWk1Ds3/A5XAtBc=; b=Za9K7wgrJrnJEpJcx8H2Ee7CwvsEI+126OtmyJxaMV7pXswxWR4h6bO3ujnIfxta/ITn6n xAGU9h70wjGbwdb1YZwsjboomSzy3IgTGFoswxxfSPowNiznVlMbpvevu/x1a+k45cibmf jXJiKq9GH5csL5TrbOLWgVEBlMb9cIY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175607; a=rsa-sha256; cv=none; b=5AxgVctYAca9rBnN2eFc71oQWB5zn/Y6nakJGjnqYdTwTAbUu03eglld/YHAw56S3j21Cy 8N1vx1Zp1JTJkGZWcEZGZkW4r8dnqXUSJ13Za296gPJnDpCTqFa0Yt7Sv10CQcMsvZvNFY vsni4rKdxD5m9/ipqFmSUoZT6N3mqCA= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=AWN84jIn; spf=pass (imf07.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.174 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-7882e94d408so382722085a.0 for ; Mon, 11 Mar 2024 09:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175606; x=1710780406; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=HoufURnUHz+cr/acEvSc8ejQyNfvHWk1Ds3/A5XAtBc=; b=AWN84jIncySZxiVwwn3sTSgKlKaUOQHYXZfd47h0PxvAN1KxRiD31KKe+WlCNCtfr7 jFmvaCVPVpd0J55nsel2/TzbDpekbrckJEVLCJlhZcXviQmhWNYh+ewXoqGairrFjJHQ qRljche5PHxOdzQj9+jWsdh9VL4TT8s7/rDbDoxq5Ek1zHNN6yU+DqgHLqIk7Vtenf8G 3TFz/CwIy4VzZIFC5ijMAD427MDmR23TiYL0QBbhmrq+369Y5XIicDPwR4srlHUCHLsX 5/zEm2gYx7pMLEHQvPk/LMEwDH0DIMutw96i1V6wwPr+LhuGxTt5kZNMr+J7bcEGS6Z9 Ft3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175606; x=1710780406; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HoufURnUHz+cr/acEvSc8ejQyNfvHWk1Ds3/A5XAtBc=; b=IFMqjr0Nqv486BKrYEoL4shLnzDDxLVCPRuuoueOTlvMBFqPYXW4tedtVZR5choO4Z DN0GtJCZBjaYcVz0+CWPBQ/DpoUU0lVqkvIzACJ03R6r+/2AD4DGA8o2XlPLGD8mxMx7 KIqpYqr1EwXiqYpM0yo4wKDUk54Lau6hBde8TeKCxKa3YtpbFKbMHlcnomXgknfHaD+L zjOhCWlXMgBOUb5QBrUacTvYdQr/pC+j/knuFFPBT/bprbFoSiio3N+FEI1sSfm2rR6p KjicusGLTHud95UIpYpanRiXcT+fdT7PiVAlPICs88Mxei22zdkhee6Qg1It1RIoM2e6 JwtA== X-Forwarded-Encrypted: i=1; AJvYcCVovol0qveObtugM33pvthvCcrShhzN7sdheHGgO3gp/jxXK5qQLjiHB5S9xgEBRTeC1oo/gBoqgb9Aft3EthnXKrw= X-Gm-Message-State: AOJu0YxFPg3KNrK+NQKNufAZQPZhOUWwk7GfOBes5U8eFHn/7N55NBD4 JGhPKuxkkGhD922niz13YERov/KyIxuFiIK1TXELRR5/CVHW+Bl/sO81Re2jIKE= X-Google-Smtp-Source: AGHT+IGrhJ6GByyle2ZXj6YELCySZAA2hRd+lCiymK5OoO5mkfddy6eeNIr6RoEdx/yXV9YfoWqMCg== X-Received: by 2002:a05:620a:13cd:b0:788:3c2b:42e2 with SMTP id g13-20020a05620a13cd00b007883c2b42e2mr8567991qkl.10.1710175606139; Mon, 11 Mar 2024 09:46:46 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:45 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 03/14] fork: Clean-up naming of vm_strack/vm_struct variables in vmap stacks code Date: Mon, 11 Mar 2024 16:46:27 +0000 Message-ID: <20240311164638.2015063-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: zb95qscc3mu9w7qszaqs5xbk8qxo9tnt X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1FF604001E X-Rspam-User: X-HE-Tag: 1710175606-477291 X-HE-Meta: U2FsdGVkX1/HKVzQdwGYH16w8gDF2g5qEcKs2eK1PXsveFdy8HNC1c3kjiBcAEjDAxqp+K01HHGDTciLQV+QcSChG5AE0H7kx6wevFDkKH8N8XzMpLQOxrHpiqwWyjQuvC+VCP2l0zR0Q1VvjRWx0TtAHEFzrE3HFkfpmAumq2ta2sIMje0DjmnXTtzeIoaWXzXmybtQiNeGFAGW1njVKgzhP2Ik/1siyvVJoyOyETv6Rul+d4rxdK60sKiMVfmTCz2G2EuoWCvnlsnATo90MsPMcu2UFTrBY3vCJPktNtaw/Z2Z8d7UhoIvSAruvgCNT65tD3yNaY46F2lMxByEWH1fft4VvgLhMgzlzbWJsJDR/kaiEISZm0aKlIAXKX4k4CDx4Uf6VGSd3D8A8Q3LEssor2jZuO4IBh8IZyzesFelsw8dQEP+CGDeFNAga6jdttXUu9biB9cds9GS0DYafzDVRyWKyBNwoAWEqueNcZAk5E5LKLjxQa5Sj4b1bv1vv4sRzXb/KB2le/Z7InT0Q1VCQl/F/XxujLc/m2vsLp7n+f50f9KSRUHCHNrGb/O1/3NZWfokgJ48v5VjbgygsTQrANZ++caI8AP6eQFTO2l6GHzBzWwJWiAZ9cG3boWcdVSTYDWhdFt4QslfB0eEQL9Io/8iVgAyKhD9BeSf6X2HJtGgaiAeMhdD8iu0/FskmZ8TtJMJmSyuaEqGNXctxZieyDXBFCksltdNL/K59m9UdaNPwTSotIyczlfyJWBQHxzv+3c/n8NKPtAi+udbYAPKu5Y2UdMc0YQBDcDXxDjydI47hDZOMGA9GiU0Y7nz9KL5OZe41kp8Y/2TbWzlBG8cIwMxqa7UFSCr4pH/paooqyYt6k497XTfnq1JpyOKtq71mMdq5L1eUN1BsE49SEuXLu2ChpSHEw8OSX8yLCha7vNBKkeGgN1dERot34MeS+8cph2ZRNNCMuV/rF9 AKl4fwV5 gxHg0QxPGtaKw6/P7AI/x52G9wDXt1UJGeLNQDNw5MPj3EJ9/6i20hcUL3p+oslVuiIVA9iu4BbXM2u+S2WezrLK1lj9Nmh3tOIIP0Vg12aaWNSfhPGmBM9yHXCsYdhhiQ+2ClxY7ZkKTlwgS/QtkHET6brzpJzQGsiEAp6KaJ2+ZRS57avUIoNy+3xrRF0qdWEogicVDTm1GS+4MeHEDJWs+JWZBcLOr7puxwow4DF257uDc0wYFTaz72+6wCtMx+xrnJejRRmWwQHCx906MujFfekiGVzcD4Ek9HqNnVJZI22j6GUULcJwVruiv9FrCP03f+pZ73LNCNJR6ld6Txx9CdUcw2cqH24gWlniEtaucVZi+/9EP0y95DNZwjOBl0bjLQzCbhVMUoF6zZiRz1VTerNE3N8sbZuxtAozGS5D9B8pgab7g2tO+YLbUPMry8HpuoYc+N1Xr3VLk4xp1FaE4vXllh4UxZiFMuT7DJvi3COaLlOMcAFW+IupDJBRqA+adSVYdMvvBTF0eISo2Hvp/5JPIysUZ2LSmQcwDkv5uYos= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are two data types: "struct vm_struct" and "struct vm_stack" that have the same local variable names: vm_stack, or vm, or s, which makes code confusing to read. Change the code so the naming is consisent: struct vm_struct is always called vm_area struct vm_stack is always called vm_stack Signed-off-by: Pasha Tatashin --- kernel/fork.c | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 32600bf2422a..60e812825a7a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -192,12 +192,12 @@ struct vm_stack { struct vm_struct *stack_vm_area; }; -static bool try_release_thread_stack_to_cache(struct vm_struct *vm) +static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area) { unsigned int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) != NULL) + if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm_area) != NULL) continue; return true; } @@ -207,11 +207,12 @@ static bool try_release_thread_stack_to_cache(struct vm_struct *vm) static void thread_stack_free_rcu(struct rcu_head *rh) { struct vm_stack *vm_stack = container_of(rh, struct vm_stack, rcu); + struct vm_struct *vm_area = vm_stack->stack_vm_area; if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area)) return; - vfree(vm_stack); + vfree(vm_area->addr); } static void thread_stack_delayed_free(struct task_struct *tsk) @@ -228,12 +229,12 @@ static int free_vm_stack_cache(unsigned int cpu) int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - struct vm_struct *vm_stack = cached_vm_stacks[i]; + struct vm_struct *vm_area = cached_vm_stacks[i]; - if (!vm_stack) + if (!vm_area) continue; - vfree(vm_stack->addr); + vfree(vm_area->addr); cached_vm_stacks[i] = NULL; } @@ -263,32 +264,29 @@ static int memcg_charge_kernel_stack(struct vm_struct *vm) static int alloc_thread_stack_node(struct task_struct *tsk, int node) { - struct vm_struct *vm; + struct vm_struct *vm_area; void *stack; int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - struct vm_struct *s; - - s = this_cpu_xchg(cached_stacks[i], NULL); - - if (!s) + vm_area = this_cpu_xchg(cached_stacks[i], NULL); + if (!vm_area) continue; /* Reset stack metadata. */ - kasan_unpoison_range(s->addr, THREAD_SIZE); + kasan_unpoison_range(vm_area->addr, THREAD_SIZE); - stack = kasan_reset_tag(s->addr); + stack = kasan_reset_tag(vm_area->addr); /* Clear stale pointers from reused stack. */ memset(stack, 0, THREAD_SIZE); - if (memcg_charge_kernel_stack(s)) { - vfree(s->addr); + if (memcg_charge_kernel_stack(vm_area)) { + vfree(vm_area->addr); return -ENOMEM; } - tsk->stack_vm_area = s; + tsk->stack_vm_area = vm_area; tsk->stack = stack; return 0; } @@ -306,8 +304,8 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) if (!stack) return -ENOMEM; - vm = find_vm_area(stack); - if (memcg_charge_kernel_stack(vm)) { + vm_area = find_vm_area(stack); + if (memcg_charge_kernel_stack(vm_area)) { vfree(stack); return -ENOMEM; } @@ -316,7 +314,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) * free_thread_stack() can be called in interrupt context, * so cache the vm_struct. */ - tsk->stack_vm_area = vm; + tsk->stack_vm_area = vm_area; stack = kasan_reset_tag(stack); tsk->stack = stack; return 0; From patchwork Mon Mar 11 16:46:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8459BC54E58 for ; Mon, 11 Mar 2024 16:46:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A50836B00CD; Mon, 11 Mar 2024 12:46:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9AFB06B00CE; Mon, 11 Mar 2024 12:46:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 805B46B00CF; Mon, 11 Mar 2024 12:46:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5CD966B00CD for ; Mon, 11 Mar 2024 12:46:50 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2641BA01C3 for ; Mon, 11 Mar 2024 16:46:50 +0000 (UTC) X-FDA: 81885337380.23.2691252 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by imf06.hostedemail.com (Postfix) with ESMTP id 59AC2180003 for ; Mon, 11 Mar 2024 16:46:48 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=FL5czDVs; spf=pass (imf06.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175608; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RZjnTg3rcm4xEs2wuMm2S2zMsfw/GRvQpvq6ZsCVNYQ=; b=y8SAY9R6pvb0KNUFsKRFSKybAVHJn3+nCATPHu39H7mDy4qhMd2mLxCjNn9tMf2YhixCp9 K5zSTns58VxXx5i4jUudSSLj/nzh3S0fV8yL2prWGFZm11lo6VVY/gLUdoYelUJWzlJiho HLibseHv69yWTJIkCOtzONK1jPIwzjM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175608; a=rsa-sha256; cv=none; b=I6Lgx7bqsyqu6k7NcOGk66XNTGz2+mLcLVs9OwOKM5kGM5rr/coD73R41VdQ0yging84aC 2nNKkOtli8ZD0gjePANlRxOk/zZ2C9bINUC3KsQ8aiyr3Rvh05n162niP1Mc6WwCZQo8Xo 3Ez6zA3ICKi+X9mltYIqDlZNvF+wLCk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=FL5czDVs; spf=pass (imf06.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-78830bba6a4so170786185a.0 for ; Mon, 11 Mar 2024 09:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175607; x=1710780407; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=RZjnTg3rcm4xEs2wuMm2S2zMsfw/GRvQpvq6ZsCVNYQ=; b=FL5czDVscOELNA/+Bv28b8RW5JwhsYtggNhJely3918ZZ3L2XPNzFq7WbFnxXH5oik xPuj5kGTRffTHiDLwNI+WzpTv3CF9w9aKq6Rc/z+jBJrTx5bTx1wzUGL1mnZh47rq4/4 JNHp2/epffrRDUfoM00YGvYuLxJBglZzG346Xm6hIj2yvojPY3cpb5uXXwyeenjR2rOI Pm0dz36iES5d+qnh+j6jbCqLXTcY8dZME6zqyCXkJxB6RPqsXVNUM38Mgsglq8qjlsch RqCzukjHI0APDF76OKOwvDmHFiCA4Uugu46boIKdvemBIRAdV4b8KX/M6FkGwx76fKOX eoQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175607; x=1710780407; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RZjnTg3rcm4xEs2wuMm2S2zMsfw/GRvQpvq6ZsCVNYQ=; b=pUSOLPAo5rBJO5aslTAUqpQnq7HGt4gjxElfBZixx/fzLMtDCM7EhG+nb5WHmrg9HQ +vPRTlOplGIz2QJIcULP42Io2aG/SDIbCDkcSKFEiyQBKtORb7Y2ninXWCUAPDTbXK7K oG54qe2xFI/3G1Eb1jge6w1j50T72mOxNswQBzW+v+zdk9vkWev+xR9tok2lD4iWfnpQ lvSUlIBbrMK2nVLa572nhhpaX7InCKwQWM/JhU7VpK3aReFk7rSzorNcAZLbzMYj7fqu m95tWbDLpbBdcghU64wh/pg4szuTL8AJXouvbBaXoTmL0OX9uvMvcLwX8pXuPXLDfId4 +kTQ== X-Forwarded-Encrypted: i=1; AJvYcCV2iT2Cu77TqjrwxZDk3utUiDa/fsYoa2tTKcSBAGqyqXHsXfL5j+sNBChX/TUkzJL5OGWdmRwDDyp3OseYRNk1P4o= X-Gm-Message-State: AOJu0YxoVe0sXxPD5DSae9apEgPdRSNbxfFCgZF/2TmetuMGdM6Tj172 Yv1EQRhYPsu089KpR65ghRU5oPC7HQ2ogBTiPqNqD0TOkQDSCvQi3tG8wE4gwqE= X-Google-Smtp-Source: AGHT+IGK3ZkaISv7/cXUwYINZRSYzKZD81aDWnU5gX/aNi31ERoz+HAdpfJ4DupGmG/DqWHaLyf2Ag== X-Received: by 2002:a05:620a:3908:b0:788:7507:6e24 with SMTP id qr8-20020a05620a390800b0078875076e24mr1874945qkn.3.1710175607288; Mon, 11 Mar 2024 09:46:47 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:47 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 04/14] fork: Remove assumption that vm_area->nr_pages equals to THREAD_SIZE Date: Mon, 11 Mar 2024 16:46:28 +0000 Message-ID: <20240311164638.2015063-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: jzpq4p9okz4s4ec4sej39hj8z6sg5snx X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 59AC2180003 X-Rspam-User: X-HE-Tag: 1710175608-379542 X-HE-Meta: U2FsdGVkX1+tn8l+p8Ivg267MFLwLXz3kNUGb9hy5sI3EHWC0unmVMh4zZu/mmNIf0PXGH75hrXKZkTU+W2oJixC8FGcLW4LSa5SQWPHRankBqFhPsFWz/4upqrukPtVMnNrs2z1rVtn9YMEQkizu1sWcTEN1kW1WSGm5CSB3sxkmvs5hk/zkYPOckDFX1VZB33dLshd0Dm6ZDOFzXVNq/kT/QepdvYvZXTSwUFBbyTtECR1GGu93Dn4rT+VxIF1JTlAxipf38FXFAnAK+lCLqOBucZH2Ru6MVARr+VKA6r95G2HPxoyjdJc+zvcs7SsaZa3+ml3lvl/QGDSMdgM230Ao/O5MudspwkXyQllM3RRSuZNk8SUKnGG56aM3+8kVkwuCog6YG6UByFK1y1OXMppKu+0tcc6hbObTLgdKH8gHwiavalEJkrwzXfspG6g8AFA66nWgLarwcMmPlyPPVjMelV1nGa9Gv4XzpIC1ZmWQ6BmQ4J0ibNJhQXzgaHRu61QDnBVpyN+APB77ZqNNGqjzu0RovCBpqQIsE9y/HsznUQ0GvhdFjPVVPL5eczzCPVjS7IEGkemzKDCN1n7TuJj/UpsmeKrQ4Dutdnd0uKAqAMITyPyVh+8QxYT0uwN+d2iJjJrLZsa2MDVCOws4fFBo4L6TSdj5mZIQMhHimE6Aqk7TOCTfLrz9g5sVLaPZiBtd2prF8JlD/lbLnpiY+sYkgN+oXtzprbg7wDdYolImUC7dhPxm3/cWq3CwtCfRSlZ+Wmpbg6vNjsgBf1aiZSfg9CiMZtTryBQF5PAyDrJi8kyzDVxnY1G/LLTSGZwU7dlfzS6kz/aQRr5CgLKeYoBQAal6vdmd/FBT2DXIJ+hNCwRPlQGGaZSjfAQFcCiCFThI4a8+uDURoWKLfPhPUB8yU3WpOMhdJAEMLx+rONkv6xngdnQG/LiUdEIqQBKWfqy2GsnSfN2DuXEMxV pHxVIHYC nibzuRpHPeGQAkF4KvXRMLKlMHOV+zPTUel0Iho/PoMGhDN7nkFzK0EHJvF0LOvQ84hEynhaz1AN6uFIiYR4TD2jwngfZnjPk8INhENpLTUIDfpBsG/eZM/HC3a69g4T7Jh+u847NJfGkFTlTQ/fEtsTyZuw3hCYwm2bfSrUpnPXVWufRFlCJDoglONj58WHrFcH6/K4mPH4V6sQb+KAoqtMjxe5aiMD7RCNRzatHYTnWQkIJhmw2Q1P9DSGMuTboplAlFNE00MvaCej3QWzNoCj5cWGc04IsylLma+sxR8k2XBYf2ZrF4IBTFad/3OpPsGZ2FyH4HXiZLI6dK3vP6zz8K8GKES/oHwyrLWxYtD/MZoL7niukGPhuXcHUi50uheKMlBjvUbU7Rij5djqhXE9D37lR6g92xi4GU3iVkBj8+2x+0AS+W+Sz0UYWvFV+k4P7zo3IJ1jy4zlr2PttazxQJx+kj2PTvntHH1sl4E67n82EOdQhYe0JIW4IrMT7ayh1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In many places number of pages in the stack is detremined via (THREAD_SIZE / PAGE_SIZE). There is also a BUG_ON() that ensures that (THREAD_SIZE / PAGE_SIZE) is indeed equals to vm_area->nr_pages. However, with dynamic stacks, the number of pages in vm_area will grow with stack, therefore, use vm_area->nr_pages to determine the actual number of pages allocated in stack. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 60e812825a7a..a35f4008afa0 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -243,13 +243,11 @@ static int free_vm_stack_cache(unsigned int cpu) static int memcg_charge_kernel_stack(struct vm_struct *vm) { - int i; - int ret; + int i, ret, nr_pages; int nr_charged = 0; - BUG_ON(vm->nr_pages != THREAD_SIZE / PAGE_SIZE); - - for (i = 0; i < THREAD_SIZE / PAGE_SIZE; i++) { + nr_pages = vm->nr_pages; + for (i = 0; i < nr_pages; i++) { ret = memcg_kmem_charge_page(vm->pages[i], GFP_KERNEL, 0); if (ret) goto err; @@ -531,9 +529,10 @@ static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { struct vm_struct *vm = task_stack_vm_area(tsk); - int i; + int i, nr_pages; - for (i = 0; i < THREAD_SIZE / PAGE_SIZE; i++) + nr_pages = vm->nr_pages; + for (i = 0; i < nr_pages; i++) mod_lruvec_page_state(vm->pages[i], NR_KERNEL_STACK_KB, account * (PAGE_SIZE / 1024)); } else { @@ -551,10 +550,11 @@ void exit_task_stack_account(struct task_struct *tsk) if (IS_ENABLED(CONFIG_VMAP_STACK)) { struct vm_struct *vm; - int i; + int i, nr_pages; vm = task_stack_vm_area(tsk); - for (i = 0; i < THREAD_SIZE / PAGE_SIZE; i++) + nr_pages = vm->nr_pages; + for (i = 0; i < nr_pages; i++) memcg_kmem_uncharge_page(vm->pages[i], 0); } } From patchwork Mon Mar 11 16:46:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34486C54E58 for ; Mon, 11 Mar 2024 16:47:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB30C6B00D0; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B36C16B00D1; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 961136B00D2; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7FBA06B00D0 for ; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4F5A81404F9 for ; Mon, 11 Mar 2024 16:46:52 +0000 (UTC) X-FDA: 81885337464.01.6D6006A Received: from mail-oi1-f169.google.com (mail-oi1-f169.google.com [209.85.167.169]) by imf09.hostedemail.com (Postfix) with ESMTP id 9613914002C for ; Mon, 11 Mar 2024 16:46:50 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=CjU1balY; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf09.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.167.169 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175610; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=85s92hjYUuufTOzCHiSH6hattUyZa1ZbNxNbo2SWk2c=; b=tmcsTVSfcuXBJHFq1RlcHklbeTkY+8qu+66scyYcZUCG0wpENImKkyfOjvf1pcAe887mIb HC0gjYwd9uVHE75sQG7U874wilQHpa/dEIwnpudpi6DsM9/gcFWsnws9798u3P4TsQqnwv Xz2KZLS5e8HmG6TGLeyAPvFWw3KU340= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=CjU1balY; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf09.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.167.169 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175610; a=rsa-sha256; cv=none; b=1sChIMJKbxXL3vgn4xUDdM5/89n/ok5kH/sLnuikT/tmmPUzb2Mh+NBehj0yTlo17G3oWm Q9JiryvlOosVWpOZUM6I9xJD5nNEiwFjMtRrvRhz5/Rmu6NSUQjcNxU0/RkS0ZyMrTxO+i wSSkIHpRWHHbTfwloOc/N41CLYb2ssI= Received: by mail-oi1-f169.google.com with SMTP id 5614622812f47-3c19b7d9de7so3130545b6e.2 for ; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175610; x=1710780410; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=85s92hjYUuufTOzCHiSH6hattUyZa1ZbNxNbo2SWk2c=; b=CjU1balY7rK3bDNcI4qQ852zLFfRHcecPorCeOMGrNZoPwNJLje1m71aeeQUp3dmX6 KwaIasjil6w4fJ8trGxJKUwuQoX89xEt+lOMm4iSLCoDasJSAA3k02Dvl93dsWK3Ho0J CvUg3ntTYK0extWx/yohCcxPETh5A1gWTApr7LqAkP4I0GBD2t/nF1eZUhbcOytp8LmR L9Oco8jHm7tpAoMPqgg5tuQHIbHcqdfifPQCZL3nZVws88zq8SP7QmmG6j+qYRnAnRUK +VP4y3BUdNGuiTiL/j1mLY4G92rBxJojQIbl5FB6NXRJ8QcLAzMgc7EsyNZWFq6Fqed1 jMHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175610; x=1710780410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=85s92hjYUuufTOzCHiSH6hattUyZa1ZbNxNbo2SWk2c=; b=LORF4Rqqj3VXE/ZzOOWLZUscE8GdHpcs2j7J1KZa9osJOofg9wFeuYm5L+B9D3QcBW jifBXB1OyJkp+jQtiUfc1p6pF+4Ro1/58TdWHD5kzLWhCwI16vbZzKbrHsCKR8S7NGfh zxqC8yvIPYG94u2cpuxamiIpEDCbXEuH/+lRN0zQPDZcfIvAE20pJ83PGKzf1EwOsO3v 1hKcneTj8ilY3Wjw7QUEJkMXuOvCeuC259ghyXg7vOJKnCfeNNtKr11CG0P5XAZIuTbi DdqM8bwhcRHqvzSseyfeDIAMv2ULyLnFLwUpmPC58FxDp+z6uNgnZO3/HVTUaoQosTy7 lBKg== X-Forwarded-Encrypted: i=1; AJvYcCXYWAnfodebxf4n1Z2fHOwi0mA6oBJvZH8EsXfzSFV/knjiqSSAx1eq638zCVpmaEfsEVIk3E2Vjw8USY0BqldyGNM= X-Gm-Message-State: AOJu0YzYNRwsPRl+XItALK8XEz5llsISgHGNnGC4neh/Au5mMRyg/MiS eT2e0oWMYYNyitTbOsAgV0MDTp8ioCWAeDvlswtHQ5RZY6tdEVafQ9cGVO4MLsQ= X-Google-Smtp-Source: AGHT+IF0sA8TBoTxey5vp022o4Baok5dPg7vNZ4pcZt1EEyuyh5oG+THj2Y1Wu2BsuZNYuYtVQC9/Q== X-Received: by 2002:a05:6808:15aa:b0:3c2:1262:e04a with SMTP id t42-20020a05680815aa00b003c21262e04amr8114898oiw.1.1710175608379; Mon, 11 Mar 2024 09:46:48 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:47 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 05/14] fork: check charging success before zeroing stack Date: Mon, 11 Mar 2024 16:46:29 +0000 Message-ID: <20240311164638.2015063-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9613914002C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: rp5853m41ucse8tzjs1484nntbbdi3e3 X-HE-Tag: 1710175610-938550 X-HE-Meta: U2FsdGVkX184FeXaxgoeQgcVQXXxtIFUtBlOJQEHkr9kJCPWF4MDwjdt17LVBI0j0+F3LvFe7avnlprrHgSarQHZ2XzJBsbGUH3tNW0mWn4bywu8T+cs1D9fjbaSTRUHYFLiCkfrG+pFC7W3fmJ+22Ii2sQ4XGrNs/9h7dcwUszkMFGI0eh4LsdUj7SREClL+/j47mQZSNMKMGMpYQ3RHo6IQj/GLMPXyl1zGw7sgGmP6IN4HYT6/pk5XSWTHRL1o8nCiYrhTwAyqNCJl8Px3Cf2j9xn6M43xVPnZhPQjpw0w+ogfhNvydzkJ/c6S3YnuxLpZ0C91ppyUZCqj5ErSVaORB2/1eLi60fMsBkQxObQd/Rop5ij/AZ8yrDceKrzDgxdDfHUGQA32sP/uo1kGI+vdzR67L/5Lo3QClP6oSnKqlcGDOdLIEB+voqrbirAG7yIBhgvzwY2ajh2kBzZ6981cMjtQlsWLsBglIm1LnbacAOFXuwtPPe6ZxfOLDiJtRClzvfzu8VtKDlITY9voMM1ul7MHshAFsqoBegINTxeqMk5huzToe4dOAYR1VCN/1yoCOOIauN0AoFNvmX8byr6dA5miKZH15qWFgO1Uz5m+UTWkuw5AwVBXBSsZ1uXyaFwNB6zQBaMnTsz4yqozA58rD9zCTyPt3KCc7J3i9SSWtCAq+bkDPfeB/SEE+WYX0WlJjJB/QgHafGQBv5/NYe5BfznxxURRHHtA9bRnlkTvrLaHLgDnB+1bMBVxuwzgwjjCyBFPBHEtDGGNl0UZ2Y9R/nGJPGQYwu99TFNtqQd8e5UeW4w4yH3ozQznYyqGZAWYtLahmnAHfpPr/EDVbgf1P7giUZNawJyN7bCvuutccG+WQFRd6yPYpKXIPAqGMrxwFXYHfI2hWsYThLjXmeeq47DMejCxxZ+KVHAYtDGWrlZkdtZx7/z7eyVt4Xja5InFVSiT6CiV4do/Fj efyKM9bF dhIl5rECrPbg63SWpblJFKGu7RzuMoQexNKejwrROLvFMwhLkPHgXZiyonp7K1+DqsjDIcYBdHeYCZdP88RtODa25vT70ncIOL0eHLw0CFfjukV3Ej04XY0UiLVB5H4nM5mY75XiU6ZcdDacEvV4G8Htd4Gh9rnR+U8mZa73foawnOaY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000272, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: No need to do zero cahced stack if memcg charge fails, so move the charging attempt before the memset operation. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index a35f4008afa0..6a2f2c85e09f 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -271,6 +271,11 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) if (!vm_area) continue; + if (memcg_charge_kernel_stack(vm_area)) { + vfree(vm_area->addr); + return -ENOMEM; + } + /* Reset stack metadata. */ kasan_unpoison_range(vm_area->addr, THREAD_SIZE); @@ -279,11 +284,6 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) /* Clear stale pointers from reused stack. */ memset(stack, 0, THREAD_SIZE); - if (memcg_charge_kernel_stack(vm_area)) { - vfree(vm_area->addr); - return -ENOMEM; - } - tsk->stack_vm_area = vm_area; tsk->stack = stack; return 0; From patchwork Mon Mar 11 16:46:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89383C54E58 for ; Mon, 11 Mar 2024 16:46:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51DF86B00CF; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A7CD6B00D0; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3207A6B00D1; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1DE586B00CF for ; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D753980830 for ; Mon, 11 Mar 2024 16:46:51 +0000 (UTC) X-FDA: 81885337422.05.18E0253 Received: from mail-oa1-f54.google.com (mail-oa1-f54.google.com [209.85.160.54]) by imf15.hostedemail.com (Postfix) with ESMTP id 2CB21A0019 for ; Mon, 11 Mar 2024 16:46:49 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=vs2HnyE7; spf=pass (imf15.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.54 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175610; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WeoI5maDr3l2Ozw1ib0LWqoOcMWVTMRirMYG/aEeKfI=; b=xtAhYSdZ7rx3XVGkYiIW1bzFbACoaqsiBlkmp3fp82Jd6q1s5C9ZuA18NequZJhyYsEP05 lAjr9+CiNPUNtkzhNiKujXoffjprAdeI/a8KOqp756DLKr7V8g6LgEkJAs2JSFuwTgDQxS y0USBIT+sT4YKFJD4Jg239U7SljuVEo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175610; a=rsa-sha256; cv=none; b=xqSGyWjrZQp2sdTFNULZEm3l5y80Jm6waBiuViVsqoVEStHs/1rgkqSBG9p6M43lUIQylP mEAFzE1TTCbK8vYadkl5WQN23NUAiNRRzYeV4/7uHRDVKHbDQYBC7J9oPDADd1UJoIeRaG /QRv2mH6UUfDR/Kcf75/bbKjpDfzfYs= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=vs2HnyE7; spf=pass (imf15.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.54 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-oa1-f54.google.com with SMTP id 586e51a60fabf-21eea6aab5eso2878624fac.0 for ; Mon, 11 Mar 2024 09:46:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175609; x=1710780409; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=WeoI5maDr3l2Ozw1ib0LWqoOcMWVTMRirMYG/aEeKfI=; b=vs2HnyE7s9sCDAJLnnazCeYXed96ZPFn5WCFdgInvLvILYbIhFDB5QF9VuOdWmz2ZG VK8wrEGWd59BfH3T759syZAJsOD0HxO5RS6bEcjPBmy/3jNxZHnxfUA9v6fu1R8T5mCQ O6wUqHfnXhtPKt6eMNBcbePzVKElUEXtAli2TgDOD7k0wSD3WElzhNoXVgYXq9MbQdMw 99nL5JCZzeoLB/JF3vcfC56z4Hn/T5vizDcT9BD3AOfWk2KNhCTwKnpplE1Lg0nHPzkc fu08KjCfbNfSxH3MjHiNoAf47WHtSO9ckm3+5Fa+LM3Tv/vO7jQYqmEkNoE0QyOpAul7 1SiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175609; x=1710780409; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WeoI5maDr3l2Ozw1ib0LWqoOcMWVTMRirMYG/aEeKfI=; b=JwrPMOHnxBcyeY8QoAH+2izMqI/f7c9GwrFWg5P+paLF70f4hkyyA3qz/58lscHDB/ GSOxTzHaGqza6Yfq46FzOWnT24A5rhWZ7CPX4Uu/QRLMv2xg/31H6C6nq+lpNbuvz2eo LjfoN4u4f2PB/bVhTiXXqhZUk9cdsTsbMHgOTx2E0eM0b0ZdtR30mblGYINhpYbCo+xS st9CDWQfVxxxpBemyO9WLZ16ggaGuEfR2d9IXBKjUoiaqJe8xDdDa67EUFxWeXxWJ10O bMjdR7TR2AwGPDyIDMWfBoI9bxdz5GQrjOplpSmsnYtQ+UEDcyndFRB5dw1WBzTB+ByZ RBBg== X-Forwarded-Encrypted: i=1; AJvYcCW5w6q57baByj/PSUaMSxgsp4ViLkoSZCueEJwbhWRXdwH1A1e27lsnaKWM3SeV46hRztwMBaX9qp47Xj00h5azbgQ= X-Gm-Message-State: AOJu0YxLc3ks4DzgDzso4IoPBrgkBIJkVxxJa0wYmVcMTqxacyblGDEg +RHg/WysIPJhTWauy/xt3J2ybqFS9kwKlnlng6PZUCYjzgB5lhcdFrM+GZWZqjg= X-Google-Smtp-Source: AGHT+IF7PMDKY7qPBLzKisHlCeRFB7P9dZ8FJtUn1b4P6yLcZu1GLHtuFZbDu9s9LwmUDKjKkIy2MQ== X-Received: by 2002:a05:6870:3307:b0:221:a43d:60ef with SMTP id x7-20020a056870330700b00221a43d60efmr8059130oae.3.1710175609309; Mon, 11 Mar 2024 09:46:49 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:49 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 06/14] fork: zero vmap stack using clear_page() instead of memset() Date: Mon, 11 Mar 2024 16:46:30 +0000 Message-ID: <20240311164638.2015063-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2CB21A0019 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: prwoatabfgiukecu5qryyyp54hru7ia4 X-HE-Tag: 1710175609-844928 X-HE-Meta: U2FsdGVkX1+JP8joK4/QBkt9sg6vD1M26dbylAuTnoILw85qmyraXmOkK3xOdvVA0PWbFYBYqY2rUQLn4amuv6KvTfforxqVcpqeJoaz+PKGq5u12Is0sYWVAAVN0vMfOcXwyrHo9JJt2c+u2uPLkv8cLqJ0lY5Twi+Z+EQIrH95J89p9l/NVP7WM1CUEZBuIG75vXCojYW+5VEITMkq8XDD9Z2dXoAr2guZvu8XW2i0nLySGb/3wH59EMY+Z8nHtsCVQNhMpRBf1c53YDxia/1HXnEJ1vlK33ZweVGhQXGdT4ti79pAa4icgX8RoTRaXyc2Ntp6fkv99GlRQ4DApECZ9B+dkit6WMI2OhjOzWGdvwUsfeBA2jugn1Ar8KxEfwPK0zoKoPUiy457VgY5O/27YjoYIP8DbMXriZ9jlIs9coo/s5dND4NK1EPUbIiJYymkBVXXfovmTD3bXSjNJUK/AsQ9bDqjBjn6fk1VE5zxk3TLXy3mPLcyG5nNK8ppeu0cykOIkBQ946QRUtV3YhUuNhS5hVKnB1UqZQ7sc+nkb2Rt9B6bjws7Lvt5+I33eVkFgZJ/ws+UC2RCk7T1e+CtO8KZAwpVaqkOyenUPQeJDPB9xt5nNae03ETR24QLNpaCrwhfGBpib+L9hVucJ/bmP04scUFB0JUfGvFawB5QXly91F0QTG9opXpbG692Ryq1Sq2OooMLdcLLfze4fvZwfQxLAFe7S3EQiBNhrmZWAjS5YLPMn8bwwm16/bSd7j6GAjO3OT328hty2VqSAhJtbBZn6YObRjYGaCrZU8NlneLXBB9xVG3u7ms5gneMQqEqkOujgzHh+d/mYNr9PCSli2PWV8efVcGQvhUlyF5XbQPyixQPUG0f5EpFJS4NwWAVXjghWuYw2tioH18zjK42hzlK1I67afeb/y0gk4nidFU4T3CG7OFkOql3MzAcip0ltC9rQvF52aduJoG UZDrGcon HD8Q4yvMgcj+Zm7s+xGYLGIoqOw0Bj0SDKINHay1bF4EY7TEY9CQCJ2mSS+YqxkL71hFYLIt+ly7tEQW9HjG1ub68mcTuz3pYwTcDN6AZqxbhpeFDVHKMDW4gC+9PCOz7GQIlmdOREkxE+RhQC4YFHmuISAUIRZRV9PHCBtUCdR7JTI/5CN5YXxCXPhoDlRf4jNzWMw0DNnFRzXiwS42DnNsIu80DWpLAPCBm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preporation for dynamic kernel stacks do not zero the whole span of the stack, but instead only the pages that are part of the vm_area. This is because with dynamic stacks we might have only partially populated stacks. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 6a2f2c85e09f..41e0baee79d2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -263,8 +263,8 @@ static int memcg_charge_kernel_stack(struct vm_struct *vm) static int alloc_thread_stack_node(struct task_struct *tsk, int node) { struct vm_struct *vm_area; + int i, j, nr_pages; void *stack; - int i; for (i = 0; i < NR_CACHED_STACKS; i++) { vm_area = this_cpu_xchg(cached_stacks[i], NULL); @@ -282,7 +282,9 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) stack = kasan_reset_tag(vm_area->addr); /* Clear stale pointers from reused stack. */ - memset(stack, 0, THREAD_SIZE); + nr_pages = vm_area->nr_pages; + for (j = 0; j < nr_pages; j++) + clear_page(page_address(vm_area->pages[j])); tsk->stack_vm_area = vm_area; tsk->stack = stack; From patchwork Mon Mar 11 16:46:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BDD3C5475B for ; Mon, 11 Mar 2024 16:47:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 128F86B00D3; Mon, 11 Mar 2024 12:46:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D6A66B00D2; Mon, 11 Mar 2024 12:46:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB9266B00D3; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CA2DC6B00D2 for ; Mon, 11 Mar 2024 12:46:52 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A3C97A01F8 for ; Mon, 11 Mar 2024 16:46:52 +0000 (UTC) X-FDA: 81885337464.29.F3C8FE4 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) by imf06.hostedemail.com (Postfix) with ESMTP id DBB18180010 for ; Mon, 11 Mar 2024 16:46:50 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=DX4pdb0K; spf=pass (imf06.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.172 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175610; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rWA/mIcrC8mrdgOBIz97cH4Ahtr8zqihEDBDcy8+bis=; b=1AOy/9XS6gZqCi+v5G+80+BZl1KqqIvgiAd9osa1UOCoW+q3PpKgSfEZuA0JW3ZX8do+pQ izmOZRvvYKZmptxVASh3M5DIVNDtqYqqBqAoNNkcMfOeQURK5iTK8sJsMsEfILHHvMJcj9 cJZGnt0Zs2qPrw6JGnfQBq8zRoC2dJk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175610; a=rsa-sha256; cv=none; b=IOIloGpW6yuCWaPwOOq6biJAUI+4C2N4FMje54JWTF2a47Obi7wJmBiP0Pw7LhI41Nc7eq GItlaCTVr9FER8fpy0uq+IQLEDJCYCVH5qwtttwISUvoqMixoT1LkdLLOWT+iqb5VJGJS6 /6sMm8lwzhUmG9uJnxUxVEQ8xuPMY2k= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=DX4pdb0K; spf=pass (imf06.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.172 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-78873ff1202so24555985a.3 for ; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175610; x=1710780410; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=rWA/mIcrC8mrdgOBIz97cH4Ahtr8zqihEDBDcy8+bis=; b=DX4pdb0KjNr/ebcTqoXy/d4fZUONR6bc0FWsqFXABcYVp93OvfThWZWC9dNqQvbwEQ jrlnmOYSBa8bTxy/N/LeOOHcv7q5UagNW4vOKC45fwN1uYX+2lR7BIQ32GhFfm8i6tkZ gzRaL7zwQeJUT7dpGN+zWYmPAY1zLO9dt5Fr4NyjyzNHASnOjDlIxalNHxW7jFcply6e PuVhmGdCG2XUeH8Sc7bZrS4/hAc6nHDxWI2b4NqjZN3Rls+WDRolJe1bXHPxbja0J34f B9ACKwVW+BTL55j3ZfQeCSZ0DBOFQgsrs5zmquClDHEawwCpE7SPEkbozeKc7aps5hrg RE6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175610; x=1710780410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rWA/mIcrC8mrdgOBIz97cH4Ahtr8zqihEDBDcy8+bis=; b=nHxKMwv+/LK20+v+7v4izK/mY22SBGUmsMYuMMAgerSELahqpCHY7UDaHgQ7IhP7/Y uiGVmp0zcaawlriOF6csCjcoSVWAXczbHuvRX8Ln9Oyg8XjjntgIyW6/YKYZWwmy+Yyg rkjvMa0rAB2IMYUzcwjee8Si2Ltf4Nen5wMag1n1W4Tpny+iCSRuJRPi1IISb8ksRN6a c9cHMxIl8UOkDKnJUaxePohaPd3Atj0N0hJ0WSonZcMTLVMEX0IZYo7yMdo6mwtG6Zxq BkVCWc3pAHwyl/zW2DlzYEACh/e6YUYyb0j5/pS7wgPa8E4lCCto+GkfqmIq5av6sb2H Mdxw== X-Forwarded-Encrypted: i=1; AJvYcCXzTL7bfidFNkWl/ZWZy/edvjywyut7r/gwdyHrKAQvmO752zBtciqi59GMI8YpIikDW7hL1CVxAy02iiPJCvYG8Io= X-Gm-Message-State: AOJu0YyP8HeFTjMDbf1GufZ7oHIcWL9T06JwBOKdwRXKtzvN4cae3JWm Th74RZEs+cCTaeoO0G0hYjNdpRlY7f8F8ec+PiceVmEmKJITiccvfezP45YurdM= X-Google-Smtp-Source: AGHT+IEiaWGNP1LEtKt9OxZe9a0Ecsughs1hTC155ICY2/6rd+mLJB7PdZHKC0eSeLtAUcuC8VPmXQ== X-Received: by 2002:a05:620a:222a:b0:788:61ee:7fee with SMTP id n10-20020a05620a222a00b0078861ee7feemr6338288qkh.61.1710175610089; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:49 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 07/14] fork: use the first page in stack to store vm_stack in cached_stacks Date: Mon, 11 Mar 2024 16:46:31 +0000 Message-ID: <20240311164638.2015063-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: 1rpajoc97k6ie78t46zx4i1fax1prout X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DBB18180010 X-Rspam-User: X-HE-Tag: 1710175610-983636 X-HE-Meta: U2FsdGVkX18UeNNFj1vlJqT9wb/rwD/qY6eF/G7zB11QJnRZA2Yk8qay1uTwZFRLI2O4mX64Sj4VNR//25Vw1D9mHYBio1bDCoi4ZMSuAdpQIQXeVnlzeqR7aPZ8G5DlWpZmyDWT5kjCu/RxUpc9u6Ib6Z4Tu9EBAFKg+F2dNW/n0OzMDLhXci3Y1e+zjiqTXIe7mBRY5tr7umqUPUMFh/LTWF1EWI+PThXLNdvQalK2sBmwPhOjIBYnsdVv2PI+zo3KaPsChh/HJxTup5g/5Mn5+DQHiwyQ4jTY4WY3JUCSJ2aZZa1NvL6cItc5LcNkG+OvN+/vgwz/G3Kb5ilG/atH/rbUOaHe4+SU5jZ7oGVG50vfj8hFuGzm+S3dSiWVCbIBCAgv3Kjtp+nhqPXS5irT8ra/6T/mA2hYqi1OLpVTtMmZxNCUEhp02iFxB7+TzPt2ssBJw5DnuY6WYz3IXwmogKNk90deJEqvbMg1KlGTeFX+Obpgvzbt2jGzMIfstwYj0IkNs4VJ69BRvc/EATJQMEG+cblYQaZqMjUZliQYEaqWzDjcBD6yg4CnsBFUag3ahRnKks3AtiBF5bKzMiHOOB8TYA0V3NSPxGusFz3HvU94ijIfbvyVSpw8J9Ky1T0g9L0OeiThFyKg1p1B9pAXf6BZuMuzGsNUJQuhO2mQruPfeLdC/nsTei6zmH35UG0Wm6frnrgQvf0yQdagTs8sLMLiw9mdboFM46nV2vJRA4lEcL8vpmDD6D35WwX3F11Yh3+BPrWVUa2mhAsP4uiEfa3GmrmWgT9QjF3/QWEQYaYuvVeyVdbnjLT6p7/E27gc84XC4Pl5H3aop0QUPF2mYdMwZyc6ONkQggq+2BlbRCunRmr0SQasKXoK2qezuuzbogWgxWdAiflv1DGgjxFfyQT7UtBfHAR4RgHOFpdi06Gj/mrSf4TR9KK+gHi/QPFXv2QdomxgeJD/hHL aMUDu7TX 5sAqxYC+YZCMc6XYyn1Nwl3qSUzK+a4dSOq9tuOAhNXDdepIPt90vQ/2pjuUUmaUdNebvvmbHuaNdew3nGqWc//UxaZLTLSPjT8lg2bAm9C7wktctlM85uGVXsU9ngt2O4nXDCgorf20ewXtZABBLg25h1FP+VoUod95/7IW80isG767D6tQUogiDKGR5+j5D2jn0L59cIikGNdp6hxwpQNqkvUxL5NJ5awZR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000051, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vmap stack are stored in a per-cpu cache_stacks in order to reduce number of allocations and free calls. However, the stacks ared stored using the buttom address of the stack. Since stacks normally grow down, this is a problem with dynamic stacks, as the lower pages might not even be allocated. Instead of the first available page from vm_area. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 41e0baee79d2..3004e6ce6c65 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -217,9 +217,10 @@ static void thread_stack_free_rcu(struct rcu_head *rh) static void thread_stack_delayed_free(struct task_struct *tsk) { - struct vm_stack *vm_stack = tsk->stack; + struct vm_struct *vm_area = tsk->stack_vm_area; + struct vm_stack *vm_stack = page_address(vm_area->pages[0]); - vm_stack->stack_vm_area = tsk->stack_vm_area; + vm_stack->stack_vm_area = vm_area; call_rcu(&vm_stack->rcu, thread_stack_free_rcu); } From patchwork Mon Mar 11 16:46:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC5C9C5475B for ; Mon, 11 Mar 2024 16:47:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 065296B00D1; Mon, 11 Mar 2024 12:46:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F30656B00D2; Mon, 11 Mar 2024 12:46:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA6A66B00D4; Mon, 11 Mar 2024 12:46:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C6B166B00D1 for ; Mon, 11 Mar 2024 12:46:53 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9B9AFC0508 for ; Mon, 11 Mar 2024 16:46:53 +0000 (UTC) X-FDA: 81885337506.19.5B310EB Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf28.hostedemail.com (Postfix) with ESMTP id E6834C001C for ; Mon, 11 Mar 2024 16:46:51 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b="e8tm8m/F"; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf28.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hkZD7ALZVm0P5UALUItr672MRcA6vth2fJOAHe6U29Q=; b=1yhTxT/vBGLnbXjrLdTbugThhzQ911rFgCh5gSt4ALhuLQW8y5JZjkmxN1BVizSK0V9BGU +rXq7audCYeGVBh4seNRIn5N+S63eKNeXIEtL7WGkEg7xVQurhw0fWWF72O7h5qVqON4Qi AU9CjU81/UNuH5Yj5XgmMVkz4htwTgs= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b="e8tm8m/F"; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf28.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175611; a=rsa-sha256; cv=none; b=i7ogrYHPYctHc3AnhfmzuEtw+1YUFFaV6vSFr/mwj84qjK+43rfWeZgIJuXngrOsXw0IxI 3SWnIrmP47GcF+WdpYVSsoXcWv1IZtZgaMR6SOIorauwsVnvqNgCNZJum0mKKdw0ZQlfIu GQJzmStrciOC+9hm4Z8xXQKdKfMshAA= Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-7884114feb8so119906985a.1 for ; Mon, 11 Mar 2024 09:46:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175611; x=1710780411; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=hkZD7ALZVm0P5UALUItr672MRcA6vth2fJOAHe6U29Q=; b=e8tm8m/FfWTPB+M14vv5sO2vfLcWl7sU+3fddkqoueR6qzCr+BjkzHXSJ0f7okd6qe u7813rXKkCsoArrTP06Ui3ouH4HFWJFGAs5H19wrIC64E8Kg285Z8hv8MgyxyKD256j6 78SQJLLPlmZ0L+zbucgsyrnjiKdo2JicpHr3oIbsz5GPe/XpYg1fYR1miu9jK3EGdCE6 eUeOOVxhdRsb5PouHPWZS3nB2SYtCXXVa2Ef3EQVSGL7S57XRSEQKSHSyNsLDZC77R9S /flLxfjHtz/cDpZ6V67vZfvs0BRuuQDRUYuhHE+x0tsRyMXEaLFVzpLVu7ogYoxankU3 44og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175611; x=1710780411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hkZD7ALZVm0P5UALUItr672MRcA6vth2fJOAHe6U29Q=; b=USrWmK2zL2BtZD5RMsKXhtlw8Gm1QfEUGXfOl+cBWA4DrXPsROMNf+JxwtdY+ekqnC SAwXRezMEvB/42G2vDwJjsNh+3XlvwA+JqB/DewrqU++Q3OvZZAZ3S7azV/eZousgI6S 3gM17YQel9MRAhzHUxb9Hk2EBJcM98lSq8yHoOSwcUVfGSQeP7oTZOy2Nz6H/su0SJA5 1/K3F4SYfvwTnty2cgXTYZjVsl7okALzkQBx9v4Hm61Hsz5NRaJ0CV55TJmJCNtqSDB9 grT1suUiNSRmpAinBk4dVwyJ3pF/IqoQxUkVxsVcVfJugSsFgMk51Kd+3EXogIR6YqAK /tgA== X-Forwarded-Encrypted: i=1; AJvYcCVpZriS3PChScAejTVj0l9MXVdcmBGXemL1mnzD6xIIAu5uK2PFN7Teqc8gsEaNYueM39ZiUYF1KXylVBiK9bW1HQo= X-Gm-Message-State: AOJu0YzL63ZlSmSHvT1KzOVis5w0b17wRc50EJQ4rnvAHnQqBREtR9Np 3g3Nw5A8KdG5mUrYU5irg2x5SDrLPVXMcenPtRCUU8Do1bL5LbeKMo0C3Ubug+c= X-Google-Smtp-Source: AGHT+IHTkS+2s/WrBF0S6nD4DTbIdKAkKUkplrTiXcc/c/8vBDolvcc8zD3FtIujCYDfupcwREXrKQ== X-Received: by 2002:a05:620a:15bc:b0:788:50aa:35f7 with SMTP id f28-20020a05620a15bc00b0078850aa35f7mr7780372qkk.30.1710175610969; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:50 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 08/14] fork: separate vmap stack alloction and free calls Date: Mon, 11 Mar 2024 16:46:32 +0000 Message-ID: <20240311164638.2015063-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E6834C001C X-Stat-Signature: 9gzgq34ar7exat1f4dehsguu5m8zsrch X-Rspam-User: X-HE-Tag: 1710175611-861947 X-HE-Meta: U2FsdGVkX18K1d3BQHHxXDwmDANEHf5RYkXsy2tZrK56L3FeFmVniltHu8R0iJECQF5c92vvzs6nbV//6oUqRR/v2e+PqCptci/pk3BqDtgO+hG+PWJlfPE9tiSRY0CRNO0trbKi6hWKJdnchA+XKi3ZXrECXgAAab1dJ9fL3gvgUsehFcXwT8Xp/9XKvyv8qS5C/fgSpTEh17LPZgaGvbcjS+FwiOldYmqlGdMAEYp0qfGg3Fxe/3qMItwnngywPEVhPXaxUgfxkEmoZGpo9HyS+FmFrXFm0sxu6bJO3Rh4WKF5S2CoVyZDYOrTfg9+5d1CoEd+Vt2Sz7aJT/0zidD9S9xNw2at2nQFSm2L6p997FOD/GzgTtIAwHHK6oii23MkPzHnBSuOqGd45nQHBWJmivhoEoXHsUcTpcuQvCMFaPHp49oQRHKgo3rYdgRTzM70KC7FEWssEYjW4nLeuQs6rDiGW0n3hd64IPka5+EI/eWvfJMXjphlWnb8j8T9Xtyv5gItkFBQim7LMmjn8loMIsK0JeNo/zcY1uc4/Rsjkny9B04mHActoCwsQIdiiQATIR4biUDOWIHKZRxpjpcnEM3VlA7ugvRrSiko06I9EHXSzQvZkJElYQjZlupiLl74yybNTtqiQbCulyuC8oZ6in566eux1sZLgJY3taXfPF+I+/zwd6Im5zEwKgv+zjSJCltjlej2teMyYIMn/IvAUhxWZTsDkXMn8wEph8crtQ0FgSen16PAV0QZsCcaTdEbj4EF/Aqw9GkqdF6Sf7SjawJOUqKCo+1usFZeXtwSzaDixh7FbbSkPHXFtWpqdeCE1cTE6fiIOsVsmCuyK+5p91fVjS7YfaGuaExrQbFJ1BiH+hsxAH6azWpfFXn8gkaEAU0bVMzw8C4VogGk5twJhGOuJkUG4E3YysgvTKyWDQpv9FPyMJI1ZByw97Kb1xKEenitaPOmEauL+2Y c2gsvM6O JNLtxFbVc6B9WM89CBNaRRJNej0zObh96uA9LSdmZ6FQH9KOMSjR6B7K6XLEnzMyL21a/7jOKSQcbAslIVwgM72/+waRw1Lm0iu+K5YBb4zLHfrt0ocEbFOoD94e+OafvESgt/WpBYh1P2ktQNof3o5WfYHyYMdRhVVkPP22QCDzLNUU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for the dynamic stacks, separate out the __vmalloc_node_range and vfree calls from the vmap based stack allocations. The dynamic stacks will use their own variants of these functions. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 53 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 22 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 3004e6ce6c65..bbae5f705773 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -204,6 +204,29 @@ static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area) return false; } +static inline struct vm_struct *alloc_vmap_stack(int node) +{ + void *stack; + + /* + * Allocated stacks are cached and later reused by new threads, + * so memcg accounting is performed manually on assigning/releasing + * stacks to tasks. Drop __GFP_ACCOUNT. + */ + stack = __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, + VMALLOC_START, VMALLOC_END, + THREADINFO_GFP & ~__GFP_ACCOUNT, + PAGE_KERNEL, + 0, node, __builtin_return_address(0)); + + return (stack) ? find_vm_area(stack) : NULL; +} + +static inline void free_vmap_stack(struct vm_struct *vm_area) +{ + vfree(vm_area->addr); +} + static void thread_stack_free_rcu(struct rcu_head *rh) { struct vm_stack *vm_stack = container_of(rh, struct vm_stack, rcu); @@ -212,7 +235,7 @@ static void thread_stack_free_rcu(struct rcu_head *rh) if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area)) return; - vfree(vm_area->addr); + free_vmap_stack(vm_area); } static void thread_stack_delayed_free(struct task_struct *tsk) @@ -235,7 +258,7 @@ static int free_vm_stack_cache(unsigned int cpu) if (!vm_area) continue; - vfree(vm_area->addr); + free_vmap_stack(vm_area); cached_vm_stacks[i] = NULL; } @@ -265,7 +288,6 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) { struct vm_struct *vm_area; int i, j, nr_pages; - void *stack; for (i = 0; i < NR_CACHED_STACKS; i++) { vm_area = this_cpu_xchg(cached_stacks[i], NULL); @@ -273,14 +295,13 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) continue; if (memcg_charge_kernel_stack(vm_area)) { - vfree(vm_area->addr); + free_vmap_stack(vm_area); return -ENOMEM; } /* Reset stack metadata. */ kasan_unpoison_range(vm_area->addr, THREAD_SIZE); - - stack = kasan_reset_tag(vm_area->addr); + tsk->stack = kasan_reset_tag(vm_area->addr); /* Clear stale pointers from reused stack. */ nr_pages = vm_area->nr_pages; @@ -288,26 +309,15 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) clear_page(page_address(vm_area->pages[j])); tsk->stack_vm_area = vm_area; - tsk->stack = stack; return 0; } - /* - * Allocated stacks are cached and later reused by new threads, - * so memcg accounting is performed manually on assigning/releasing - * stacks to tasks. Drop __GFP_ACCOUNT. - */ - stack = __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, - VMALLOC_START, VMALLOC_END, - THREADINFO_GFP & ~__GFP_ACCOUNT, - PAGE_KERNEL, - 0, node, __builtin_return_address(0)); - if (!stack) + vm_area = alloc_vmap_stack(node); + if (!vm_area) return -ENOMEM; - vm_area = find_vm_area(stack); if (memcg_charge_kernel_stack(vm_area)) { - vfree(stack); + free_vmap_stack(vm_area); return -ENOMEM; } /* @@ -316,8 +326,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) * so cache the vm_struct. */ tsk->stack_vm_area = vm_area; - stack = kasan_reset_tag(stack); - tsk->stack = stack; + tsk->stack = kasan_reset_tag(vm_area->addr); return 0; } From patchwork Mon Mar 11 16:46:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A28C5C5475B for ; Mon, 11 Mar 2024 16:47:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A28596B00D4; Mon, 11 Mar 2024 12:46:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 947906B00D5; Mon, 11 Mar 2024 12:46:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 78CF46B00D6; Mon, 11 Mar 2024 12:46:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5B2EE6B00D4 for ; Mon, 11 Mar 2024 12:46:55 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 330CE14054E for ; Mon, 11 Mar 2024 16:46:55 +0000 (UTC) X-FDA: 81885337590.01.6A0ED25 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf01.hostedemail.com (Postfix) with ESMTP id 4DC974001D for ; Mon, 11 Mar 2024 16:46:53 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=O3SlzjZl; spf=pass (imf01.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175613; a=rsa-sha256; cv=none; b=y+cWuWCbRaLRnJhLY471lcbEjGzdqdEvMzMGVpT2LbMdrg3srpwmrIqvok6X1Sr0twOdAN MSb3L4oZKOGZX61mmmyjMO0sQNRsaS5jV/u5JKiYYjl18N89ssMpQibicFQH23duY4jYZY 3QpBJUXlm8QLbJ3wUGwSdX6r0QGAROI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=O3SlzjZl; spf=pass (imf01.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175613; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UUTioQgK+VFwu4bHGNUstSWJvzqUc2xbAOy5DzulCE4=; b=nmbjFpk4L3INmfz6QyXqKFWyCWlp67N3ZB4HtgyDGz1lsMwiX2zazEHM1Ki4Hn1WQy6d+Y ABD/nuMHHKX/4/x2ieIy4dtJ9ApSVXMqFCLNhyX6jOn/rzePtr7YcSteGrVrV2Ged6caUE 2DakMI5w3r8xZ0OpVpkETqD6+xoG5Y4= Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-78822adc835so198828285a.3 for ; Mon, 11 Mar 2024 09:46:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175612; x=1710780412; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=UUTioQgK+VFwu4bHGNUstSWJvzqUc2xbAOy5DzulCE4=; b=O3SlzjZl3EXLglHISO3vKH91NRgKUNmhysBrPi+uAmgFKrJWYle4ELbXVudb5/rRkg 16xJkYsHhtzTVOwxk8E2d1hbrznDoNDcU5mSyL7M/TMzcomCc8Dwi2zorPG/DJqmy5+H rS6PyUwS1TuIzJrRS2iBo9VaIoVigEamtWdRfom41zaA4UfpRx6lgFrnLQFB0Sx760sY T1OA2XY5d49yT7ORjRHLj6q3nGj7qa/XnLmvNf3KlYtFOnsZ1Bz62C593/V5GjrSqnL+ PLV9myVQyo4kYr+vt7PxQuZePYu44fi7X5JE7L4nxSUOQf9n049BiAc9JnJv7ibvhqZb N6wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175612; x=1710780412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UUTioQgK+VFwu4bHGNUstSWJvzqUc2xbAOy5DzulCE4=; b=XmB6WjW26IW9VusS5Q6vm7cd63rZVZDi+fgO7Sc7CCdIjTMGwuFsT1rbIcVJvIimo+ 1zIL/L+zsmeLiUX1aIh9TJcYlJMmnPIHNSrojtWtQJAyAMwv40OvVM11XL6c59QBjm5F AfXca12SHUshcPgd2E+Z2MPJgtb8zI9pvU4H7RT4vSNrvUuc3ohb3n2IOUc5hmdDLGFA bdhCBkf/7KQLfzFkjDNtLjpRYEh1X5j6yx4uBblYs6TPqrhcjUHCy620Z2YtplBMtUMK otRGlqdEZA2rc6AnMI5JdsSNQac5rftmx2soe1evMloWDXJQ3iw1LUtXq6Vjcld+P6zk C+7Q== X-Forwarded-Encrypted: i=1; AJvYcCV3Jg/85BbtoD7rDhLUYIxRbVmUggreVN9azfF7N/yISjEMViWknvlLbCJ/1RnoDsfSqR/UZCNgjiSGWyvJt0jMmFM= X-Gm-Message-State: AOJu0Yz6mkq5FZHrsw3DJVm7AP1vGPp+HnpFq6ERcKA/No97RPVGjpvS UmAciupZLJJKT1orgy2sBFIT7BAKOcIMRXS6QfjBrWi40IbVsOIPPhVlkZ4/RNg= X-Google-Smtp-Source: AGHT+IF4clsrNYnhAmKVCQ5b1CkyUTVuvR4XaNSojXhA81ENz+LMIfO1rJKaTKEeTeVkpkuetWxx/w== X-Received: by 2002:a05:620a:5d9a:b0:788:5909:8ce5 with SMTP id xx26-20020a05620a5d9a00b0078859098ce5mr1085134qkn.34.1710175612342; Mon, 11 Mar 2024 09:46:52 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:51 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 09/14] mm/vmalloc: Add a get_vm_area_node() and vmap_pages_range_noflush() public functions Date: Mon, 11 Mar 2024 16:46:33 +0000 Message-ID: <20240311164638.2015063-10-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4DC974001D X-Stat-Signature: 74611hjz8taic1bg7eecr5az51fz87te X-Rspam-User: X-HE-Tag: 1710175613-386306 X-HE-Meta: U2FsdGVkX1+E0pKzfKl3ydverKDc0Ich7bnxzXzf63uhiYsHDhAsI+ucbYVr7tR3xciHNYhnp5NZ80kg4kPHQ5s9CWi+fgnxuXK1Rf0MTzsty9WWYavQXL2DHQlriT/U89/UbZ+36HYEbCj5uLZwNUUErQxzH4Lklkedo/DMdgvpBS/GC+xxzlHHv3Eu2Lb6blNI09UKVFrNWGLAZzHCXroIcX2WZyY6XNw5BvTF7CJwHv9Qs7rXZ/SbBNxFzxmSfjJ9ArtBQfpJizeVD74uxVBlck27+C53Ftp5Ju+xNlRpPFGFn+k7o3NCwhN/OvNBHpGN2mu2yf2Wns9+sAr0Hj0yY3/mZrefw5dG2BaYH3lOKclvPRxDUVdTnpj+HYnLOZsSK+XE7KCfXS39o1WTSYNXoio2pHd5meoCn3S9Yrjohs6EFZaUyMzotG96akzfV+s98JpWwwZW2lEtrEA4ypy6ECwbPdjN7eIXuE3PUFM6HYTYsDGD8b9f8m0G2xDbwP339SOSBh7lZ+lO0BosN+EANaveud4oZ3YEn0JEQfbttEcgbZrC1Q74/R7PWmWPbyxecwhr1uN395k8+yDvllPv6wncSm0DCh81dvhRjbz0XSUwCrZKwTFVTnp+RZn891dmP+Htaq/nZDJjt7GkhrhkHs3M6LIDTL7FHsLgeTrm9N1WWoz5KjQruPD3P7/Wmp1lVERbgaNdqv3+yBATq57Uz7xfejCwsVtJmzNE+xSB1Xzt5Gez7Wm3NENVIyzbpvh0kVWfjtwW7N7R4Wel3snv2TmR1XMpXyeivPU4nbGy+7ew8HuAai3JswP0PDCXTDzsXeR6G7pBSXAuBacNafJqp8rpXDKkkIpbMyQZMUjUrD12Bqd6B2Xc2ymMhaO/31bo538uuQHqs2JBVmcgwo8P5fuPNjEx+GYaJeu99OxhcCSaHMVA3I3wK+febw36guEUwzFjxFZTRJYZ/Ib fijk/CId EuHR2mu1TFMuU1sE4QkMZcbGD8xQI8Y4R7bDRwyYHsNwlEfX+AVtYNnAZvU3syLzJSihxVFYnoOMXYyGbAfIqCM6IAaLzGVOMH6eFz+yFRlG2V24/H9tA2DUxSXloydnd3pQ71MTDWQWr7a4e5XUdQWlYvZ5JybASyngTLn8VeDq+Fp5MRgxxYvNylp0AOi41hzprF9E3ldhIYuvkMSslSAva/D/6XKoYwUwr8A4Fhpt1rNsc1XVN1i4gV19gQJ+8ew+mMM3Tp6gPUfnoYNcASR31pxtBM/VJNYLSnNxNsUSsWhuORV9bWkfmpmnoMC8xbjLzbmbGyULnZ0ryi8xKnpumHamw5BXGl81NPU6RI2pHjRI+EO+B2DjO/hF8O/UTjJpCGZgcvUKejeXw3qO8ury+boV6We0askMGVppFimHzGhsULW/dSaJyiqt0U5EuESuw8Es2aZkJyLALLJTNQiNKyRfUcim++FIoIoygTdv/GATuD+zQtEFsQ3PqKVPCg5M6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: get_vm_area_node() Unlike the other public get_vm_area_* variants, this one accepts node from which to allocate data structure, and also the align, which allows to create vm area with a specific alignment. This call is going to be used by dynamic stacks in order to ensure that the stack VM area of a specific alignment, and that even if there is only one page mapped, no page table allocations are going to be needed to map the other stack pages. vmap_pages_range_noflush() Is already a global function, but was exported through mm/internal.h, since we will need it from kernel/fork.c in order to map the initial stack pages, move the forward declaration of this function to the linux/vmalloc.h header. Signed-off-by: Pasha Tatashin --- include/linux/vmalloc.h | 15 +++++++++++++++ mm/internal.h | 9 --------- mm/vmalloc.c | 24 ++++++++++++++++++++++++ 3 files changed, 39 insertions(+), 9 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index c720be70c8dd..e18b6ab1584b 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -210,6 +210,9 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size, unsigned long flags, unsigned long start, unsigned long end, const void *caller); +struct vm_struct *get_vm_area_node(unsigned long size, unsigned long align, + unsigned long flags, int node, gfp_t gfp, + const void *caller); void free_vm_area(struct vm_struct *area); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); @@ -241,10 +244,22 @@ static inline void set_vm_flush_reset_perms(void *addr) vm->flags |= VM_FLUSH_RESET_PERMS; } +int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, + unsigned int page_shift); + #else static inline void set_vm_flush_reset_perms(void *addr) { } + +static inline +int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, + unsigned int page_shift) +{ + return -EINVAL; +} #endif /* for /proc/kcore */ diff --git a/mm/internal.h b/mm/internal.h index f309a010d50f..ba1e2ce68157 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -969,19 +969,10 @@ size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, */ #ifdef CONFIG_MMU void __init vmalloc_init(void); -int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned long end, - pgprot_t prot, struct page **pages, unsigned int page_shift); #else static inline void vmalloc_init(void) { } - -static inline -int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned long end, - pgprot_t prot, struct page **pages, unsigned int page_shift) -{ - return -EINVAL; -} #endif int __must_check __vmap_pages_range_noflush(unsigned long addr, diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d12a17fc0c17..7dcba463ff99 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2650,6 +2650,30 @@ struct vm_struct *get_vm_area_caller(unsigned long size, unsigned long flags, NUMA_NO_NODE, GFP_KERNEL, caller); } +/** + * get_vm_area_node - reserve a contiguous and aligned kernel virtual area + * @size: size of the area + * @align: alignment of the start address of the area + * @flags: %VM_IOREMAP for I/O mappings + * @node: NUMA node from which to allocate the area data structure + * @gfp: Flags to pass to the allocator + * @caller: Caller to be stored in the vm area data structure + * + * Search an area of @size/align in the kernel virtual mapping area, + * and reserved it for out purposes. Returns the area descriptor + * on success or %NULL on failure. + * + * Return: the area descriptor on success or %NULL on failure. + */ +struct vm_struct *get_vm_area_node(unsigned long size, unsigned long align, + unsigned long flags, int node, gfp_t gfp, + const void *caller) +{ + return __get_vm_area_node(size, align, PAGE_SHIFT, flags, + VMALLOC_START, VMALLOC_END, + node, gfp, caller); +} + /** * find_vm_area - find a continuous kernel virtual area * @addr: base address From patchwork Mon Mar 11 16:46:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9648C5475B for ; Mon, 11 Mar 2024 16:47:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0DED6B00D5; Mon, 11 Mar 2024 12:46:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CBEDE6B00D6; Mon, 11 Mar 2024 12:46:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEB1F6B00D7; Mon, 11 Mar 2024 12:46:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 97A196B00D5 for ; Mon, 11 Mar 2024 12:46:56 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6199B1604CF for ; Mon, 11 Mar 2024 16:46:56 +0000 (UTC) X-FDA: 81885337632.23.CC10EA6 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf13.hostedemail.com (Postfix) with ESMTP id 682D62001E for ; Mon, 11 Mar 2024 16:46:54 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b="uWBjwOQ/"; spf=pass (imf13.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175614; a=rsa-sha256; cv=none; b=jVIVkSvXkxOMDkZ/JeCVtHBJ/68H2YC4SMvWGuZQY+iI3vHfR5XM91V0fcumeN6nStmvts oKD0JUtagv0BeA6IZr5wB157SoQhzpUoqiry5oQ6Bq454t41hM9QlanOXcmbc6TsgHAav3 e5Q8SQZJvwvDFV7QD03fgSE8V/ZaJ1A= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b="uWBjwOQ/"; spf=pass (imf13.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175614; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2I5VvwnC3LRR3szeP871B8eKL/IZGdjRShD9rinhwbY=; b=4rCr/DXEk+kOit2XtDDJywmvwoMpiduUcZdELY/22CnAmXjdqqwcChlcwRqtz6rg5pduGm 3AY95LLgKBVpz4sYig5ZZLpjsABD8kfb/Kgq7WRh1x169+KSOG3xDHSwAPtNZOLqbkdCve bJXAhb0srlR3wxNbeqYxvkqq2WStjAI= Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-787edfea5adso203749485a.2 for ; Mon, 11 Mar 2024 09:46:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175613; x=1710780413; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=2I5VvwnC3LRR3szeP871B8eKL/IZGdjRShD9rinhwbY=; b=uWBjwOQ/5fKtRj0M1tw+1joDfiF55gtLORG3pLXJmEZsnBrfIr60G0wAVlM0PA/B+4 Lg8KQt44DaQgITVnQeEwL/3tpCrq5gImLNQyQuaaJDBDjfFCbm31iLarFQMSnbQw5B5J HIZDAvDi4Kfe/ynrCuFFMBFffkb5qty4HKJdpQ9PZaRsC5BIpoS2U/EmtLT/SU3AcEMk edmfLzhaWug0Wevi6PNXQ6iQ+UrryeaC+Anc+R7n6LMuGZQsHF9y9e2tXIk8gLntWJsg xRf3jYvEozLi/sCEaTK1IQIuWyAK4ogcxYdrHOPDwea4ZjZisNwQ798dHi9sjNx2z/ux QgWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175613; x=1710780413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2I5VvwnC3LRR3szeP871B8eKL/IZGdjRShD9rinhwbY=; b=X7UjW6wMyWpbF9StKhkg5MFDHtTNERnwwSf/uCMYrDRvJRN863/g2wID7jngO/N2lR kxEhKijs0cXtQ0r0nFB8loSQ2VyVWuosBaFCckvNJaVhUrB+bd4Jv59mwbh3G9Fhguqq mpz8Dsmvq35JvZ+e97e0B/wLAqDpOGVZV7YQlvepR3digy4Be4UMOfddchpPFx1D7sCT Eok0s0BDksJEMvvB0LCwCf3Zr5qvgeyA6NL7KABE/LgKqO3yjpKJJpYOJ1JggLXilcUX y8uUj4CzHKpEYzHo749Vp8aEmtI4uH8VedFCLg/prcXZTxZWumtBkR53fc8zI9rVzi2M p5kg== X-Forwarded-Encrypted: i=1; AJvYcCXDAmVPGrcDqMOjOrkqQR2Cd/JLXStXLbVHoSOdnsnkxOWEoh/N7kWoATtl6ciZ8mAxTp7BAvC3hxGb+GPhB8nJYrI= X-Gm-Message-State: AOJu0YzceUhu0JSZt+IHa0GT4JJ2dfOKFg1diYcGkCZYBRrg1QLgRZn+ LssSk3BWrIvtjvErt4tUEg/Zc97u+vah32dEQT4K2aQuwzpOv07H+lR2a+8Ild0= X-Google-Smtp-Source: AGHT+IH9a5LZ+7yIW2SDInIoVHV8enodZiTDdoz2gyXsxlHKxaxJ5G55cFQu5tNI4LZq2rv5sPx5sw== X-Received: by 2002:a05:620a:1a08:b0:788:4101:3d2 with SMTP id bk8-20020a05620a1a0800b00788410103d2mr10720214qkb.1.1710175613353; Mon, 11 Mar 2024 09:46:53 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:52 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 10/14] fork: Dynamic Kernel Stacks Date: Mon, 11 Mar 2024 16:46:34 +0000 Message-ID: <20240311164638.2015063-11-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 682D62001E X-Stat-Signature: maordu6pcq4fon7junis5i4gi4st5msu X-Rspam-User: X-HE-Tag: 1710175614-173971 X-HE-Meta: U2FsdGVkX18kDAZVMNjeTyPdCAha0XEdav0BJQZ9A02hBPaq3ImZ1ysATJM3tc7vgTeUEEZVA2Om5jpiQVgwUT5rOOUc3Tdxw8c4KLdUTtE3J08fLQSR3MtNdURzS/5Z+2XcDnL7dqUbX19+lA2V8FTgPdQe2ybwFRT0La5ng3cnKDEWF1SKXJUiazxbOOsCgqapLaZx2byNmjJw/Byrij/tkRUucNOgTFMSz9QbmCyJXqJHC/n5pP82tYe+UBC1eM+RGbttshT5BAJMxrIeuMP9Lj8rFkflluDD242vdlbeWHAT82yKMg8w1U/FUY+nccwssRQcZufjKLHf92z2991sC1YchgBwRvkBG5CSno831JG4tPMQ57C/uSP6PvvFzQDmySmp0yVTVoOioSmFgNayJdbqIO3WQI93A+Sh5/zpb7kgLJQp6dJu0y/5dWc0b5459MlFbA9i5J0KBbEWEfQAoawQSuR+y2WQkrpIzyVyJN4QaifehdixEJ/02VM20w3E2kum/ErAuHJnd80nK/sEIj/qw/IsnAnryLhEYvPZ7FMuZYsQgTYZEfp4DzlDG8/2FJtaceZ050TeMfUTW+poG3IMiJhEu62d2zHCZyyPCfD91PM07dPagUiQk1U4lwITNcmXjOTXITDxeiZ5rO6CFumqr74+r/LAalm7IeptcDKCT0Hxmy0m2sK/tw8e4RakZYHxGQx1cvq/x40smuXjDe1Z2zioU+T7qgvIypCdgwCgP6grZZET5ObWOESCzXoSgtOqnAiD/tVccFVwAK5zdHil5uuglQ8G0I8r/0SX1aukOOefAyGzHzRyme7lTvgLFmOl2mKoZw3Yke78WMSoouc8BpCIbXPy+DobhNL3MAEmZNLzZNiB0hPD3dwFAqD8vt5uo0o0owb37foJDCErcO80mwfoI/EHV3CK9RgPRJ5lcM8/gv9tqw9R8KDaQfeg+gSzJEmDxPxWya1 o7Xx1VN4 aNOd9L5gGR9kbR9g8Y3jAOxmpG1cArpPpe4Z/vPPyD9ucph4eFLCDVHtKZbcua3ljBTSnSNMdKHtFR+Jut6yBEY8k3KGVZTmtHJDKK+qf9/Ngy4tzKRZVt1hsr8qtlTqdZMCTAYlBZyQMbDnWhlG9zoMmf4RWysPCoQSlqm3U1ggV3Sw8C3TCMT5C8GOl8Bxy2WYYDcI8DAfLgPYxgjuzQyhHtZs8iJsDBsqT9IwUfsoz1oi3JPOPItb47NQNpNYL0offtscU6AQr0NKfYCAiX+JDjQwZmTFxlq5s1SAlhwlQ0e2BQbfi+KKS+xJz7+xO19S5CRGOW8FlpfHZOK4f/5o6pQ7S/JxAdBix0q+m/xe65Wj6NSZLpWHJqTESp7pNxTcbJtYuAR0dzHUQFy62X0zxTq47+juIzvpt2q2+5oxZ7PMpHOCmPQdZWGe68VLEdGiXSzVHxLEjmmY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The core implementation of dynamic kernel stacks. Unlike traditional kernel stacks, these stack are auto-grow as they are used. This allows to save a significant amount of memory in the fleet environments. Also, potentially the default size of kernel thread can be increased in order to prevent stack overflows without compromising on the overall memory overhead. The dynamic kernel stacks interface provides two global functions: 1. dynamic_stack_fault(). Architectures that support dynamic kernel stacks, must call this function in order to handle the fault in the stack. It allocates and maps new pages into the stack. The pages are maintained in a per-cpu data structure. 2. dynamic_stack() Must be called as a thread leaving CPU to check if the thread has allocated dynamic stack pages (tsk->flags & PF_DYNAMIC_STACK) is set. If this is the case, there are two things need to be performed: a. Charge the thread for the allocated stack pages. b. refill the per-cpu array so the next thread can also fault. Dynamic kernel threads do not support "STACK_END_MAGIC", as the last page is does not have to be faulted in. However, since they are based of vmap stacks, the guard pages always protect the dynamic kernel stacks from overflow. The average depth of a kernel thread depends on the workload, profiling, virtualization, compiler optimizations, and driver implementations. Therefore, the numbers should be tested for a specific workload. From my tests I found the following values on a freshly booted idling machines: CPU #Cores #Stacks Regular(kb) Dynamic(kb) AMD Genoa 384 5786 92576 23388 Intel Skylake 112 3182 50912 12860 AMD Rome 128 3401 54416 14784 AMD Rome 256 4908 78528 20876 Intel Haswell 72 2644 42304 10624 On all machines dynamic kernel stacks take about 25% of the original stack memory. Only 5% of active tasks performed a stack page fault in their life cycles. Signed-off-by: Pasha Tatashin --- arch/Kconfig | 34 +++++ include/linux/sched.h | 2 +- include/linux/sched/task_stack.h | 41 +++++- kernel/fork.c | 239 +++++++++++++++++++++++++++++++ kernel/sched/core.c | 1 + 5 files changed, 315 insertions(+), 2 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index a5af0edd3eb8..da3df347b069 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1241,6 +1241,40 @@ config VMAP_STACK backing virtual mappings with real shadow memory, and KASAN_VMALLOC must be enabled. +config HAVE_ARCH_DYNAMIC_STACK + def_bool n + help + An arch should select this symbol if it can support kernel stacks + dynamic growth. + + - Arch must have support for HAVE_ARCH_VMAP_STACK, in order to handle + stack related page faults + + - Arch must be able to faults from interrupt context. + - Arch must allows the kernel to handle stack faults gracefully, even + during interrupt handling. + + - Exceptions such as no pages available should be handled the same + in the consitent and predictable way. I.e. the exception should be + handled the same as when stack overflow occurs when guard pages are + touched with extra information about the allocation error. + +config DYNAMIC_STACK + default y + bool "Dynamically grow kernel stacks" + depends on THREAD_INFO_IN_TASK + depends on HAVE_ARCH_DYNAMIC_STACK + depends on VMAP_STACK + depends on !KASAN + depends on !DEBUG_STACK_USAGE + depends on !STACK_GROWSUP + help + Dynamic kernel stacks allow to save memory on machines with a lot of + threads by starting with small stacks, and grow them only when needed. + On workloads where most of the stack depth do not reach over one page + the memory saving can be subsentantial. The feature requires virtually + mapped kernel stacks in order to handle page faults. + config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET def_bool n help diff --git a/include/linux/sched.h b/include/linux/sched.h index ffe8f618ab86..d3ce3cd065ce 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1628,7 +1628,7 @@ extern struct pid *cad_pid; #define PF_USED_MATH 0x00002000 /* If unset the fpu must be initialized before use */ #define PF_USER_WORKER 0x00004000 /* Kernel thread cloned from userspace thread */ #define PF_NOFREEZE 0x00008000 /* This thread should not be frozen */ -#define PF__HOLE__00010000 0x00010000 +#define PF_DYNAMIC_STACK 0x00010000 /* This thread allocated dynamic stack pages */ #define PF_KSWAPD 0x00020000 /* I am kswapd */ #define PF_MEMALLOC_NOFS 0x00040000 /* All allocation requests will inherit GFP_NOFS */ #define PF_MEMALLOC_NOIO 0x00080000 /* All allocation requests will inherit GFP_NOIO */ diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_stack.h index 860faea06883..4934bfd65ad1 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -82,9 +82,49 @@ static inline void put_task_stack(struct task_struct *tsk) {} void exit_task_stack_account(struct task_struct *tsk); +#ifdef CONFIG_DYNAMIC_STACK + +#define task_stack_end_corrupted(task) 0 + +#ifndef THREAD_PREALLOC_PAGES +#define THREAD_PREALLOC_PAGES 1 +#endif + +#define THREAD_DYNAMIC_PAGES \ + ((THREAD_SIZE >> PAGE_SHIFT) - THREAD_PREALLOC_PAGES) + +void dynamic_stack_refill_pages(void); +bool dynamic_stack_fault(struct task_struct *tsk, unsigned long address); + +/* + * Refill and charge for the used pages. + */ +static inline void dynamic_stack(struct task_struct *tsk) +{ + if (unlikely(tsk->flags & PF_DYNAMIC_STACK)) { + dynamic_stack_refill_pages(); + tsk->flags &= ~PF_DYNAMIC_STACK; + } +} + +static inline void set_task_stack_end_magic(struct task_struct *tsk) {} + +#else /* !CONFIG_DYNAMIC_STACK */ + #define task_stack_end_corrupted(task) \ (*(end_of_stack(task)) != STACK_END_MAGIC) +void set_task_stack_end_magic(struct task_struct *tsk); +static inline void dynamic_stack(struct task_struct *tsk) {} + +static inline bool dynamic_stack_fault(struct task_struct *tsk, + unsigned long address) +{ + return false; +} + +#endif /* CONFIG_DYNAMIC_STACK */ + static inline int object_is_on_stack(const void *obj) { void *stack = task_stack_page(current); @@ -114,7 +154,6 @@ static inline unsigned long stack_not_used(struct task_struct *p) # endif } #endif -extern void set_task_stack_end_magic(struct task_struct *tsk); static inline int kstack_end(void *addr) { diff --git a/kernel/fork.c b/kernel/fork.c index bbae5f705773..63e1fd661e17 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -204,6 +204,232 @@ static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area) return false; } +#ifdef CONFIG_DYNAMIC_STACK + +static DEFINE_PER_CPU(struct page *, dynamic_stack_pages[THREAD_DYNAMIC_PAGES]); + +static struct vm_struct *alloc_vmap_stack(int node) +{ + gfp_t gfp = THREADINFO_GFP & ~__GFP_ACCOUNT; + unsigned long addr, end; + struct vm_struct *vm_area; + int err, i; + + vm_area = get_vm_area_node(THREAD_SIZE, THREAD_ALIGN, VM_MAP, node, + gfp, __builtin_return_address(0)); + if (!vm_area) + return NULL; + + vm_area->pages = kmalloc_node(sizeof(void *) * + (THREAD_SIZE >> PAGE_SHIFT), gfp, node); + if (!vm_area->pages) + goto cleanup_err; + + for (i = 0; i < THREAD_PREALLOC_PAGES; i++) { + vm_area->pages[i] = alloc_pages(gfp, 0); + if (!vm_area->pages[i]) + goto cleanup_err; + vm_area->nr_pages++; + } + + addr = (unsigned long)vm_area->addr + + (THREAD_DYNAMIC_PAGES << PAGE_SHIFT); + end = (unsigned long)vm_area->addr + THREAD_SIZE; + err = vmap_pages_range_noflush(addr, end, PAGE_KERNEL, vm_area->pages, + PAGE_SHIFT); + if (err) + goto cleanup_err; + + return vm_area; +cleanup_err: + for (i = 0; i < vm_area->nr_pages; i++) + __free_page(vm_area->pages[i]); + kfree(vm_area->pages); + kfree(vm_area); + + return NULL; +} + +static void free_vmap_stack(struct vm_struct *vm_area) +{ + int i, nr_pages; + + remove_vm_area(vm_area->addr); + + nr_pages = vm_area->nr_pages; + for (i = 0; i < nr_pages; i++) + __free_page(vm_area->pages[i]); + + kfree(vm_area->pages); + kfree(vm_area); +} + +/* + * This flag is used to pass information from fault handler to refill about + * which pages were allocated, and should be charged to memcg. + */ +#define DYNAMIC_STACK_PAGE_AQUIRED_FLAG 0x1 + +static struct page *dynamic_stack_get_page(void) +{ + struct page **pages = this_cpu_ptr(dynamic_stack_pages); + int i; + + for (i = 0; i < THREAD_DYNAMIC_PAGES; i++) { + struct page *page = pages[i]; + + if (page && !((uintptr_t)page & DYNAMIC_STACK_PAGE_AQUIRED_FLAG)) { + pages[i] = (void *)((uintptr_t)pages[i] | DYNAMIC_STACK_PAGE_AQUIRED_FLAG); + return page; + } + } + + return NULL; +} + +static int dynamic_stack_refill_pages_cpu(unsigned int cpu) +{ + struct page **pages = per_cpu_ptr(dynamic_stack_pages, cpu); + int i; + + for (i = 0; i < THREAD_DYNAMIC_PAGES; i++) { + if (pages[i]) + break; + pages[i] = alloc_pages(THREADINFO_GFP & ~__GFP_ACCOUNT, 0); + if (unlikely(!pages[i])) { + pr_err("failed to allocate dynamic stack page for cpu[%d]\n", + cpu); + } + } + + return 0; +} + +static int dynamic_stack_free_pages_cpu(unsigned int cpu) +{ + struct page **pages = per_cpu_ptr(dynamic_stack_pages, cpu); + int i; + + for (i = 0; i < THREAD_DYNAMIC_PAGES; i++) { + if (!pages[i]) + continue; + __free_page(pages[i]); + pages[i] = NULL; + } + + return 0; +} + +void dynamic_stack_refill_pages(void) +{ + struct page **pages = this_cpu_ptr(dynamic_stack_pages); + int i, ret; + + for (i = 0; i < THREAD_DYNAMIC_PAGES; i++) { + struct page *page = pages[i]; + + if (!((uintptr_t)page & DYNAMIC_STACK_PAGE_AQUIRED_FLAG)) + break; + + page = (void *)((uintptr_t)page & ~DYNAMIC_STACK_PAGE_AQUIRED_FLAG); + ret = memcg_kmem_charge_page(page, GFP_KERNEL, 0); + /* + * XXX Since stack pages were already allocated, we should never + * fail charging. Therefore, we should probably induce force + * charge and oom killing if charge fails. + */ + if (unlikely(ret)) + pr_warn_ratelimited("dynamic stack: charge for allocated page failed\n"); + + mod_lruvec_page_state(page, NR_KERNEL_STACK_KB, + PAGE_SIZE / 1024); + + page = alloc_pages(THREADINFO_GFP & ~__GFP_ACCOUNT, 0); + if (unlikely(!page)) + pr_err_ratelimited("failed to refill per-cpu dynamic stack\n"); + pages[i] = page; + } +} + +bool noinstr dynamic_stack_fault(struct task_struct *tsk, unsigned long address) +{ + unsigned long stack, hole_end, addr; + struct vm_struct *vm_area; + struct page *page; + int nr_pages; + pte_t *pte; + + /* check if address is inside the kernel stack area */ + stack = (unsigned long)tsk->stack; + if (address < stack || address >= stack + THREAD_SIZE) + return false; + + vm_area = tsk->stack_vm_area; + if (!vm_area) + return false; + + /* + * check if this stack can still grow, otherwise fault will be reported + * as guard page access. + */ + nr_pages = vm_area->nr_pages; + if (nr_pages >= (THREAD_SIZE >> PAGE_SHIFT)) + return false; + + /* Check if fault address is within the stack hole */ + hole_end = stack + THREAD_SIZE - (nr_pages << PAGE_SHIFT); + if (address >= hole_end) + return false; + + /* + * Most likely we faulted in the page right next to the last mapped + * page in the stack, however, it is possible (but very unlikely) that + * the faulted page is actually skips some pages in the stack. Make sure + * we do not create more than one holes in the stack, and map every + * page between the current fault address and the last page that is + * mapped in the stack. + */ + address = PAGE_ALIGN_DOWN(address); + for (addr = hole_end - PAGE_SIZE; addr >= address; addr -= PAGE_SIZE) { + /* Take the next page from the per-cpu list */ + page = dynamic_stack_get_page(); + if (!page) { + instrumentation_begin(); + pr_emerg("Failed to allocate a page during kernel_stack_fault\n"); + instrumentation_end(); + return false; + } + + /* Store the new page in the stack's vm_area */ + vm_area->pages[nr_pages] = page; + vm_area->nr_pages = nr_pages + 1; + + /* Add the new page entry to the page table */ + pte = virt_to_kpte(addr); + if (!pte) { + instrumentation_begin(); + pr_emerg("The PTE page table for a kernel stack is not found\n"); + instrumentation_end(); + return false; + } + + /* Make sure there are no existing mappings at this address */ + if (pte_present(*pte)) { + instrumentation_begin(); + pr_emerg("The PTE contains a mapping\n"); + instrumentation_end(); + return false; + } + set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL)); + } + + /* Refill the pcp stack pages during context switch */ + tsk->flags |= PF_DYNAMIC_STACK; + + return true; +} + +#else /* !CONFIG_DYNAMIC_STACK */ static inline struct vm_struct *alloc_vmap_stack(int node) { void *stack; @@ -226,6 +452,7 @@ static inline void free_vmap_stack(struct vm_struct *vm_area) { vfree(vm_area->addr); } +#endif /* CONFIG_DYNAMIC_STACK */ static void thread_stack_free_rcu(struct rcu_head *rh) { @@ -1083,6 +1310,16 @@ void __init fork_init(void) NULL, free_vm_stack_cache); #endif +#ifdef CONFIG_DYNAMIC_STACK + cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "fork:dynamic_stack", + dynamic_stack_refill_pages_cpu, + dynamic_stack_free_pages_cpu); + /* + * Fill the dynamic stack pages for the boot CPU, others will be filled + * as CPUs are onlined. + */ + dynamic_stack_refill_pages_cpu(smp_processor_id()); +#endif scs_init(); lockdep_init_task(&init_task); @@ -1096,6 +1333,7 @@ int __weak arch_dup_task_struct(struct task_struct *dst, return 0; } +#ifndef CONFIG_DYNAMIC_STACK void set_task_stack_end_magic(struct task_struct *tsk) { unsigned long *stackend; @@ -1103,6 +1341,7 @@ void set_task_stack_end_magic(struct task_struct *tsk) stackend = end_of_stack(tsk); *stackend = STACK_END_MAGIC; /* for overflow detection */ } +#endif static struct task_struct *dup_task_struct(struct task_struct *orig, int node) { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 9116bcc90346..20f9523c3159 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6617,6 +6617,7 @@ static void __sched notrace __schedule(unsigned int sched_mode) rq = cpu_rq(cpu); prev = rq->curr; + dynamic_stack(prev); schedule_debug(prev, !!sched_mode); if (sched_feat(HRTICK) || sched_feat(HRTICK_DL)) From patchwork Mon Mar 11 16:46:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33BE4C5475B for ; Mon, 11 Mar 2024 16:47:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A02AA6B00D6; Mon, 11 Mar 2024 12:46:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93B176B00D7; Mon, 11 Mar 2024 12:46:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71BEA6B00D8; Mon, 11 Mar 2024 12:46:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 56B836B00D6 for ; Mon, 11 Mar 2024 12:46:57 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id ED2241A040E for ; Mon, 11 Mar 2024 16:46:56 +0000 (UTC) X-FDA: 81885337632.12.2536EF0 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf10.hostedemail.com (Postfix) with ESMTP id 43BEEC001E for ; Mon, 11 Mar 2024 16:46:55 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=GGQHlpvJ; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf10.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175615; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NGLgnjdxSReHw9wlD+FW1YibHNtCAjZjXR36RT3BbYQ=; b=k31nVnO0RaE+zhKXzUW//vYxa48YVrxpV6+tIsWIyOzDNvzlUs/CtAzXZ/veCzWgTFMJf5 Mpv93KV6FpIdPbKn0BRyo55aCURD1VoSr+CcbrQ7NXaPFWm4vicczszPF7Wd6Wj38uOnrf mJ7rH5+yuFfcsYnJjzGu/TNiIWU2OE4= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=GGQHlpvJ; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf10.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175615; a=rsa-sha256; cv=none; b=zSVw4c7zF6tTEA91CL09Tlql48uSwH5lpfPbnx43xj2LK4hXrbi13HIdIaTqCJYaWX11Wb Jhs0NgqDoudB4BurRp/mjzhoffxLORDgi5PgGoC1YzWgIGRLTmpDWy8C7whHkq4UmiRNdI YhfubQhqAJl+qV1enImsVq39ctnguA0= Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-78822e21a9dso149149085a.2 for ; Mon, 11 Mar 2024 09:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175614; x=1710780414; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=NGLgnjdxSReHw9wlD+FW1YibHNtCAjZjXR36RT3BbYQ=; b=GGQHlpvJYpQMqGQcanF7Pc7eQShivWWX80uuEhmx9x4qmlZ7xslEKjFZRSmBow/o92 ccIPRAIT4/DYmCSCvkHve2XWJj4r+wQLNKJH4uFD8xBL52Pojo7lhWFFBmQQZf/Y5dG0 jPmjMpA9ZiZo7DIn7sJOc0YRsxtcaqRA+/VywY7oYECLm+pUpj6ZdfRHiUxakl/uIvkd o5HmL+lxxMAU3VN7TAAe0vv9J35IhqFwBus4/JAR7SuNNAC4wHk+rKcK1BKNzNXu1g0y 4cHL9ccpHJF3PwNVnRgRF2p1xTPLekMmBQ6oIBB+fqee5hhgJTQDMnbLCuOiySjqivCW 18TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175614; x=1710780414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NGLgnjdxSReHw9wlD+FW1YibHNtCAjZjXR36RT3BbYQ=; b=iYOrRkCJp0IIcsYEVpjZLxV7EJYI41bm3QyVlUiBK87DHb+3EHmMTtpjfzsjkGV3Hz 6vqgfdFAb6Kj9FRg8SUtAdIyCFj2191queyuyNF/vphFOZAoBhVm7UQk8GvU30SysdtX u779iCL3BwkG8BvYGBYWZuhJgwtN+cbuPVFEuO6MuctnHaW1sWbTzmnIVNsrbbIXFDA7 i34bOcIcFoo6wZ/U+drpEZAzlyhUr8taeFa4DMtfl0j9RrPrEZWE/FCS6oKkVQs4fRDv oXktIsWbmGSXM0xAyEVMjWV83XOuNYzFXywI3FTcl0Dy5VT4tE17kNlrXJ9f2GAE6hGl 7o0g== X-Forwarded-Encrypted: i=1; AJvYcCUB+ZTHfnXUNImjUbk84qFLvsr07UcF0ZszZyNaff8XUQpjb3T4Z93lRY/JUQg95As6hN4x+AJtvq5gcwlcOi/bYrY= X-Gm-Message-State: AOJu0YxZltOXIet93FgrTd+Yy0XpFkbK/dpALy/2I9R84bgCxIS0WpMq drqwbZERP7gup+T5vODHwhbFFG0bnyknQKlDQUhzDICp71vShl0wiHSEsSN4ei6bqFPOJ0/7yy+ kMSE= X-Google-Smtp-Source: AGHT+IHiFimL/uL6CiZ0/ffUHb5GjnvMvZQk5uciXsXYpJX5LXGSw9tTbmvr67JCBeLYBzWo/uqLsw== X-Received: by 2002:a05:620a:45a1:b0:788:4e44:a9b7 with SMTP id bp33-20020a05620a45a100b007884e44a9b7mr11466669qkb.78.1710175614423; Mon, 11 Mar 2024 09:46:54 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:54 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 11/14] x86: add support for Dynamic Kernel Stacks Date: Mon, 11 Mar 2024 16:46:35 +0000 Message-ID: <20240311164638.2015063-12-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: uaj1tfpqudyrtx3r65oeib9eoc45z668 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 43BEEC001E X-HE-Tag: 1710175615-3387 X-HE-Meta: U2FsdGVkX19FonC8Kg7gsMaHylFHQBtda7M2B+WhpKTcd+oTBwbZ0avXMB0Ge9SkG8oxmIc0m6OlJlgv+CIz+4Hm/irCeVX9OwbDR2rYmUeMLBj7UVNUZLLLLhCBIxxqoFdcQMHDvhMAU8rYsSvKMVRRUiCP/rk53yy8T1D9xG0gjZwx9HLiSIh6N93ctGpjd+1Uy5be9ugWEcPtkN9TBDbTKyOY4k0DkXsE+UgcPoWHcpqP/fb3Mo9JZnRJy5C4I+kl6u7Wsg8r9lJJPnBV41pLTdAd7lRkQEyjCoajbhE/dfnp+1lNBuvSdfpnIu3pOrOOJS5PSZX6NJcxQx3oX8SSmt05YjyVyGplnmL8ykz0i3PJhZOx/bKNVs9aPvDwhx+kaijVXYgL2Mj3sJlRF7mlk3+KJx05w2xzBEZJhq42uVkw7CclEjjsLI266715LRRQJFq3i2yOLrQe5ssSSUGw3nlNmT6j37x0grfYsNWR2xP8GvJ+IZtr8sQavj6ZQXv1aCz5cDncYFzG5C9Jvw8SawAxThnop2knWFicHudcLAqmPF53HtTTrBaRHblXw31QnGilBmHfMZT4WUFsByAGjbWARK9hpL+VYzYHm2AOSpcID5QuohIhIyLI192s/DYHfu7dWTyvSfuVK6w0HzaBaTF9QVJaohOOA6iVQNxsVAr+Wp+bpk6Z2D5MvkffPG04dUcTr1Jsp4vt26WLm0E5gu7kznVW3tjTa+kMgWGit/YgeV/aN11AAhe0QQhOWvS1oo1X1vObgIevrnPjwa5uZPaQn9AdOCuyUq7U8F1iR7SsHntkV1/ZUWkdegasCCoEbWtxqc9zugjiPnWnP2ywqMb+sGfY5yZvNMIkCFvFuotAEq4CrzLXoUxqP9DlA8P6Xbk5W1zf2WJoa9xet9YSFTsa56E9gkL7xYOW4u3Nbst3j6PRMlX4T/9Ktp/MWqLf3CqpJw3IodyRtis of/mJn7V shJek3JbLJFoKJYDxE5nrKfI1np4Zxi/fp4zB3atmOt97JPCyKBELdY6BB4uTSirq+OwxdGm6AyBkv+xXW2ljYiJKkacFu0ptGyKoXRLgGg6Zk6pvAEOdlurzX170wu7xI1GnQ/0aL0Nros4AvjEByPoM/TkHpsXMAmNHGHBVsH54Dps= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add dynamic_stack_fault() calls to the kernel faults, and also declare HAVE_ARCH_DYNAMIC_STACK = y, so that dynamic kernel stacks can be enabled on x86 architecture. Signed-off-by: Pasha Tatashin --- arch/x86/Kconfig | 1 + arch/x86/kernel/traps.c | 3 +++ arch/x86/mm/fault.c | 3 +++ 3 files changed, 7 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 5edec175b9bf..9bb0da3110fa 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -197,6 +197,7 @@ config X86 select HAVE_ARCH_USERFAULTFD_WP if X86_64 && USERFAULTFD select HAVE_ARCH_USERFAULTFD_MINOR if X86_64 && USERFAULTFD select HAVE_ARCH_VMAP_STACK if X86_64 + select HAVE_ARCH_DYNAMIC_STACK if X86_64 select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_WITHIN_STACK_FRAMES select HAVE_ASM_MODVERSIONS diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index c3b2f863acf0..cc05401e729f 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -413,6 +413,9 @@ DEFINE_IDTENTRY_DF(exc_double_fault) } #endif + if (dynamic_stack_fault(current, address)) + return; + irqentry_nmi_enter(regs); instrumentation_begin(); notify_die(DIE_TRAP, str, regs, error_code, X86_TRAP_DF, SIGSEGV); diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index d6375b3c633b..651c558b10eb 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1198,6 +1198,9 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, if (is_f00f_bug(regs, hw_error_code, address)) return; + if (dynamic_stack_fault(current, address)) + return; + /* Was the fault spurious, caused by lazy TLB invalidation? */ if (spurious_kernel_fault(hw_error_code, address)) return; From patchwork Mon Mar 11 16:46:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2552C5475B for ; Mon, 11 Mar 2024 16:47:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8EFD66B00D7; Mon, 11 Mar 2024 12:46:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 89ED06B00D9; Mon, 11 Mar 2024 12:46:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CAF86B00DA; Mon, 11 Mar 2024 12:46:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5816B6B00D7 for ; Mon, 11 Mar 2024 12:46:58 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 25F581C0E9B for ; Mon, 11 Mar 2024 16:46:58 +0000 (UTC) X-FDA: 81885337716.30.582E12E Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf28.hostedemail.com (Postfix) with ESMTP id 36438C0015 for ; Mon, 11 Mar 2024 16:46:56 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=Lqqt5FR+; spf=pass (imf28.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175616; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w3lfkbgOqarNxaTYrCv0KyWjfiF6t60qYNqci1EaSrU=; b=t7sT1gIy9qwfe1qEoCjgCy5wHuUcGxGF3I0TYe6rO/JLKZdNBnSmXJE/vlz38p6Gpnak3T bf9e6IC6g2cx0Ivxtc1eUZwEm+7QYQnsYaHFkjEYzKRZ839MWuL2+jyCLoeQvdZ3B9V+YE BesDZ2SGXwxnDLjvZedPaKUvh6lXJTg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175616; a=rsa-sha256; cv=none; b=VGYNwp5DcwDsK+WcO2x67tg45j4PNBqOO318h8AuEu0LfuDgovvrZLKJFNaPx0Y80Z+eci YLxgZu9Vr9Lczu014RV4bDrU0SPqQ+R+1qXSU71tN0jutnc2PB4GfSkEKf1w9v/yBuBB6K j2TLXqv0jzSWoCH5laBQQHiFCrZJ0yk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=Lqqt5FR+; spf=pass (imf28.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-78831914027so131713185a.1 for ; Mon, 11 Mar 2024 09:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175615; x=1710780415; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=w3lfkbgOqarNxaTYrCv0KyWjfiF6t60qYNqci1EaSrU=; b=Lqqt5FR+blWtxXtjmFu3BigorpeVks3hhCT0cuX3kBJu5/S2FSPzIgj5AtX9LAmb6J wZwurVGN6SsYdOJPakERzNMV8l7azLSUSji5Zv/LfZq9O6ZHzlgtjIJxwdL18FSa5CVf DONxlzrwfKk0QSvd71LU77l9LJI/lvsa1fxLl5Ar9FLd8FksIvU2c383iTDTZW3il2HL TOhc3ZdruwT3GELzRMvdmeafzRczjFGV6QZgKeHdHkIWXPnwNXOQx3CybbvQcJORZBuc zi7lDzLR7dh2LD3c4WvKHzgM6T1gSkXURLy0iwUPfJ++TYYyix7pyY21my49m3d0m9o2 UCWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175615; x=1710780415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w3lfkbgOqarNxaTYrCv0KyWjfiF6t60qYNqci1EaSrU=; b=OYopkjPnidohDsWZEPjA1FKUvwJ4CNoYUHsMTMWwatrE1K8F0thn5alsZ/Qrmejbms 6o2+cRoehxW7JTdFTPzZuK7L9NxL2bD/pfoKWMrfYZyeTgFLqp6FKypbKT7hk9BdyfO7 kRLoebWDanjpgGEELgrsAVduHXqSR+Per3hS0VBzSrBPoZCL/xIMIVk/1+k/oGf8i8tX IXQ+CLo+YqTNVfEaHwAPklCvOOGoJ00k6RGCGpvXyMVkHRWKYmibrKzQa1jUxyx1LrC8 0bRqLpkfdKmNGBpfqOI+EXo/Xr4KP8ltsrZhXEJ4NKHUtlqwvLXkYh33Wd6vCUyWQ+sU BA7w== X-Forwarded-Encrypted: i=1; AJvYcCVltbcMVbHMoGwe6aQdQ0NqhMjlEAwKXkK3T5Q2aDxu0kuyvFxug/nm4V8Wa/ZXfXOsDPQb16xwRL2ddeGbV7Ky4n0= X-Gm-Message-State: AOJu0Yygizzj4S/Q0NzE+AIEPE9l82qa1uuALK7KP4UlJQ+nsn3EDoVh 2WfpBNarnaemyl7d605NohNrLxVYY1rVIN6MlgN3m0aLYumCMKjMqtg+z/bC/Fo= X-Google-Smtp-Source: AGHT+IF4l4AZR/miARXhlieJbwXHGqXTuZHHo2Erbz9LNEfZfrLPyat4CsaJX3i07UmkC9xY9bUxgA== X-Received: by 2002:a05:620a:4609:b0:788:2740:550a with SMTP id br9-20020a05620a460900b007882740550amr9951495qkb.44.1710175615373; Mon, 11 Mar 2024 09:46:55 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:54 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 12/14] task_stack.h: Clean-up stack_not_used() implementation Date: Mon, 11 Mar 2024 16:46:36 +0000 Message-ID: <20240311164638.2015063-13-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 36438C0015 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: adcrjykh5js1s9se5g68418jqwg7j61m X-HE-Tag: 1710175616-378829 X-HE-Meta: U2FsdGVkX19WeVwQTmPL/mH2QBS6fbCpsbR/YxzbLMCjYxEdLAxI92wfZkGM1Tl/TknTiQgYrriEYwpNIpcUmCZpb7zSCHZAABkU+rK6P9QYLO+RXtxObAvMCqapdqaDfQOxSJ5xJOuylmN//TAzRRsgXhdA9nNz8wt/xLybStoTvDcCL/zTyrl0ggmBwPU1P2d8ww9y8eXWNTHBiR44hGwmvbjQgVz/CqZlibrUgnDWjR5vx/wLxlEdw66C3Ur42U3Qr9DjjWFwBUdVesWi9zi8n+9o3bpEal6WE2xYOGCmnRswCcwD7KFVgJjeBmVegd617ZKQSmXjn4HR3JQy7FaUWm4AHTRjd0fbto3iDGRT7gdbmJTQOcvnvB1plu2xjbY1zQfOaE1rQT+0KkShaZ48aoFm+KIyBewW2n5OOmlKylSwsM+1YcSk7YTXqwVJpkCmzImHuyQKUQmaMhUM3muI+4hceo7jqkNNPeMDsvvnwtogZ7xbLojNwHhBj7+GIAB/FqzXP5V55YC8ENqrYGl6BPgOnjbryLwF0B7zY05usd6X0RYS6NDzAOc4IDktB/yYM10e1HoAx0fsPMwDvyL/1Hhut8mClVBRgiA0Dg0CiKmrkR9PczEfDyZo++prLagadlC8qJEXGD+N+SuzikNvEzjcGGQr+vinjis5XHIlZyY9KtVkHF7J8s16Zv5F1tutDKVzR8UtetxWDTIbksRPGOURlTiqhnBZSIYKbxwBSZfci9tnk1fFKU601xRe4q5h1YZBEDqt/2w8e6KyHmOSRBo5mh4EBHYpikgSaTg2o8Lq81RdCzIRabYjtmntHLHRkp7QB7pWqFkCznRj7bE/Sgro0n4ReylsSBxZHinT0Y2/0U3tiQI1uMpVJrAR2S+MIP2NL1qVl61l9tRCGJe7bOMj03rtKGwiqSCNvZr04EXg0lLwP3K8n45sJxaZ5rR+i0yYFKHFHTFw8KB dto77WlD +X8/4ZkvzI1cu+sam1/3U92KZxhTNMkVOojNQadD5vuDZRMx0bHnUTB5KXB+jKmaTqBvJ9DocKoDGgqzVujxJs99BfA6NQN0NWv1T+4Lp/Mk4l6RNmjA3mU8P/fO8F/aLsV6cdeBKefREa8nrTOJDFAkf9y+1MUafJMtkhY27cjbuzY1SuPWv0quroA7Ja5+iQUdB9LnKRpYUqT2kQVXVq8Eh/Q6w9mCB/JSyNVvIYWKYlTie8pKtNYhxdA+fGnWhRmZL+rqiSUW5iuB9VhwBPrMQ5KM2dnJVfmMXVxbh2tUD5/+pygcpjz9WrbHGs0BKI6thJjL24Put+UPZeFiVwrC4FsmUFCoffrk1qd2CzYgK5FMFqtA2KGTjztlXFu2qvC/7aN0GQPrekp+crwoTPIbTaQORjO/uuIR6mL/j6gyMBcCtBzoKJF1arLhSR0gpJCMXbwSihKeJ36irmlNsoP8QPKPervKmjnAzND42bcwyC04e63DYEocaC1yiD+t32HTO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000007, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Inside small stack_not_used() function there are several ifdefs for stack growing-up vs. regular versions. Instead just implement this function two times, one for growing-up and another regular. This is needed, because there will be a third implementation of this function for dynamic stacks. Signed-off-by: Pasha Tatashin --- include/linux/sched/task_stack.h | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_stack.h index 4934bfd65ad1..396d5418ae32 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -135,25 +135,30 @@ static inline int object_is_on_stack(const void *obj) extern void thread_stack_cache_init(void); #ifdef CONFIG_DEBUG_STACK_USAGE +#ifdef CONFIG_STACK_GROWSUP static inline unsigned long stack_not_used(struct task_struct *p) { unsigned long *n = end_of_stack(p); - do { /* Skip over canary */ -# ifdef CONFIG_STACK_GROWSUP + do { /* Skip over canary */ n--; -# else - n++; -# endif } while (!*n); -# ifdef CONFIG_STACK_GROWSUP return (unsigned long)end_of_stack(p) - (unsigned long)n; -# else +} +#else /* !CONFIG_STACK_GROWSUP */ +static inline unsigned long stack_not_used(struct task_struct *p) +{ + unsigned long *n = end_of_stack(p); + + do { /* Skip over canary */ + n++; + } while (!*n); + return (unsigned long)n - (unsigned long)end_of_stack(p); -# endif } -#endif +#endif /* CONFIG_STACK_GROWSUP */ +#endif /* CONFIG_DEBUG_STACK_USAGE */ static inline int kstack_end(void *addr) { From patchwork Mon Mar 11 16:46:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3764FC5475B for ; Mon, 11 Mar 2024 16:47:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 545C16B00D9; Mon, 11 Mar 2024 12:46:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F6B56B00DA; Mon, 11 Mar 2024 12:46:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 286A46B00DB; Mon, 11 Mar 2024 12:46:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 04FFF6B00D9 for ; Mon, 11 Mar 2024 12:46:59 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C115D40245 for ; Mon, 11 Mar 2024 16:46:58 +0000 (UTC) X-FDA: 81885337716.25.EF6C0C9 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf12.hostedemail.com (Postfix) with ESMTP id 2FD344001C for ; Mon, 11 Mar 2024 16:46:56 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=yQVOfaba; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf12.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175617; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cK/ZqVdlnureqUddSXvrvjnokezl2c4iuxNaHlbvXOk=; b=5QXXU0gjEJUX+MDRIWZN83WKdaJG3kG+xjGWeHRvdvD+R0NqUiJwD/VOgE7Eqxqy1jyxM+ kUEYLsFyYraxOAwzyNQlLxC0J1ULsckhrdrbG1inQnRQxYMNpGh+3WUU6AiFalVYIj78Wv Gm5TAo82nkSPcW1xOzKxR8PWVpVnFzs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=yQVOfaba; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf12.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175617; a=rsa-sha256; cv=none; b=BOIuAnG2ef16KBY+XuW+2A3iCfXIGMmuIwoFbrGqYrvBX1WfuyfUoOtd25rftSGAcbtPmG CUVsylUKCXSMsE8kWDIr5wcls+c46NW8KUWyLppRPsjbLj9/76Wha8mjo3ZGwaqArl78wa uEbqPLAJQ/J9/5XlTKfP1iWhdOw+lwE= Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-7882d75e766so213167585a.1 for ; Mon, 11 Mar 2024 09:46:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175616; x=1710780416; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=cK/ZqVdlnureqUddSXvrvjnokezl2c4iuxNaHlbvXOk=; b=yQVOfaba7cGQ3AxH8o8ixCFlMmwIT1w/FFh1rIUyQIojAg3C56C1znvMySziX4X0y6 I9RpcfQ+gQvg3pqb9H0Rpd7Ro43K3yp1Tv2FgQWHqf1e9M75xDzEKeqJUU1/e95RL+Fz 4ffJht5ju2RlrPCMg/SX22foBdHwgo8slk+raKtNG5JGqM27sEaYaYNKIyH5jTxbtV01 3mLSTwPlIMpW6m/Jiu1pSG14p1p8e8gfdgita6TJRoNXBGiCysJ/MQaNeq8S1nEiY+mJ IFwoGWPzyuegyNwPYjBHNmvIBuQh93+u4okjHlDh8Wk5W8Ap+MEZSZ3A210i2B5FLTWy Ewcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175616; x=1710780416; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cK/ZqVdlnureqUddSXvrvjnokezl2c4iuxNaHlbvXOk=; b=aF7beIUDWHAzs/6xMsxwoICrq39jSZTJRyVHgxt6j81MG+8w6iluiLYu8b11Kdm/RU PHFjUeHCRRN2g+hg/SwqVYd0BqyNwWkC+8UOjnEfnOuSKtvBoSyhFg14BEUwWxUjC6LA ywGzt8bwsEoJ/aAs6PImhWa5zEslTIliKWMI4DMvnRY7sANQW6HpNXFL5z0ZY9RmA2r/ rBQLN7MUqEoeiOben0abWzf9vD8ETgvljpsN1cHhgMKhZR0BRDnqLgtBffr6b9VeDDHS bQ6bZbxCJjrr/Ks0J0/ol8jAK9j09Ev3d4q/SHW4SHy8XknC421gubhVlqsRAoRsTfLi kgBA== X-Forwarded-Encrypted: i=1; AJvYcCVd3Rb4aC2SQs9O1qAab2QnrXQOLZN9DDq5TGz5oEwDSB0uT+8GykEtNhImdGa1Go9+GLdlooe+w9v4RHbHW7IAcmA= X-Gm-Message-State: AOJu0Yx42nBFRAIGSL9nn05JyoU9Vlehu8nTPH8xDWhvFzse+hBHz88z 0IAHR20pQ+3gIlJDpJ4MbguM2ZLuqE7ysQziWZD8TdUBp79MA+28qz+V5HJIrDo= X-Google-Smtp-Source: AGHT+IEp6VoHaP5KrWImU0EWHAjL+QS9WD2gq+SL5008BAWJX/vKvEeMCLb5ZMa54jxJQaXt/mOFWQ== X-Received: by 2002:ae9:e701:0:b0:788:c09:ce3 with SMTP id m1-20020ae9e701000000b007880c090ce3mr7062849qka.46.1710175616304; Mon, 11 Mar 2024 09:46:56 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:55 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 13/14] task_stack.h: Add stack_not_used() support for dynamic stack Date: Mon, 11 Mar 2024 16:46:37 +0000 Message-ID: <20240311164638.2015063-14-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2FD344001C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: d7a9nwsaqxpqasz5z33uwb1uiaucuhhp X-HE-Tag: 1710175616-441745 X-HE-Meta: U2FsdGVkX1+bb45xJZ/pEf3q0kpu0togTEDC/AhG95D2zADM6v9qCyZUKsZDRzMfsBQsQvZOPH8EaFUlsBv/Q1xMBiMTx4iLkknVSeTV+UkOj+lBq9a+rfqRC9T8W/tROemH2fIJjwSqw9/aLtOSqePjhllB2ok2AeZq3ICByAujs49CdHYTKtCJdCxHhoeTLaP2a3DwOfhfq9ajLY7uxHcYiij9EmSnuNNOl4AhO6DLvdUOZ3MtnayTQZzNN14gCdrdol1EI2lZ/2ktGKqnzO2Os7AZYLC4kuT5cLE+/xHnKxeEgWsNNzmN1E4BWRngZ/F2he91kxlqGOljVXsxowCr0jegxhmp43yyFAJzbK23gNzu7LQjT1R1NtXlUO2eU9N0pg0USYDs3y9TfZVgCdbNWBl5gFokDaYXFarufrqjCVr1JEwx5u31S0WkkRhIxOn1AEd0XGqtJQJ8wiTlG5R+0LdDm5IIQAHdTGY8pm1Gne1HSItQd09zX7nPUH0LYFCnXuzJVfc/0cTq+CmTiUxVSSvM2Hxa7BEK2wPPD/YoRRnsyURhZ0EzcZ1KgS8NAPr++/aG+60yf6bJn9Ab2/XJ5N/Jby4mdEN6dPOZLl7DiYFeBxdXrFiQbQS6L47XtH4Fy7eah2dEWHVEqSjzinuUSQsMKUFLjwcFi5bI5c37JW4X7JCbKL+OSRMpghKseRAzB7WotOfev8x6oEpuKQfASFlbS/5djDL8dezl83RiGQ5NwT1D83xavdQCByJooNJwD4VNLcPjwPQHdN212BlyIz/zoooh9kDvPQhjRfUIqSX+oeX05kojIk1iaJCy8eWWLgroac13bAVk/6KCvRZIuQbmA7AI+vHWFa8negvGt4WayZzyC2wZQw5Ii1qiQwcNzpSYI+blqw1oye/9Sr3szEd4NKs/7sJzVnM11+onibdnJMrGRcBvkz3wAoZkkSg/eaUpPkcijXG3gW1 7b4l8tQb MG0a3m0ZLGQt5GpMrscckSM4YeOzJeGnEaIi6UVYCH5A8wVII5r6ooYfOE+fMREar7cHmBDpZlsZJZWwsMwJUtqW54uyLQ1uXj+Xb3P8u/hFhbgOZ+3fOG6r1s1IYQ4EVSORyyNUHy/s3FhroEA+8L9N/3AehtwrUAcFkJRkJyGCDljdSMPas7+HRF942UvjdOeA7LQ1EEA/vGbz+TKxKsdPFyTk2CitHnqOg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: CONFIG_DEBUG_STACK_USAGE is enabled by default on most architectures. Its purpose is to determine and print the maximum stack depth on thread exit. The way it works, is it starts from the buttom of the stack and searches the first non-zero word in the stack. With dynamic stack it does not work very well, as it means it faults every pages in every stack. Instead, add a specific version of stack_not_used() for dynamic stacks where instead of starting from the buttom of the stack, we start from the last page mapped in the stack. In addition to not doing uncessary page faulting, this search is optimized by skipping search through zero pages. Also, because dynamic stack does not end with MAGIC_NUMBER, there is no need to skeep the buttom most word in the stack. Signed-off-by: Pasha Tatashin --- arch/Kconfig | 1 - include/linux/sched/task_stack.h | 38 +++++++++++++++++++++++--------- 2 files changed, 27 insertions(+), 12 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index da3df347b069..759b2bb7edb6 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1266,7 +1266,6 @@ config DYNAMIC_STACK depends on HAVE_ARCH_DYNAMIC_STACK depends on VMAP_STACK depends on !KASAN - depends on !DEBUG_STACK_USAGE depends on !STACK_GROWSUP help Dynamic kernel stacks allow to save memory on machines with a lot of diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_stack.h index 396d5418ae32..c5fb679b31ee 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -9,6 +9,7 @@ #include #include #include +#include #ifdef CONFIG_THREAD_INFO_IN_TASK @@ -109,6 +110,21 @@ static inline void dynamic_stack(struct task_struct *tsk) static inline void set_task_stack_end_magic(struct task_struct *tsk) {} +#ifdef CONFIG_DEBUG_STACK_USAGE +static inline unsigned long stack_not_used(struct task_struct *p) +{ + struct vm_struct *vm_area = p->stack_vm_area; + unsigned long alloc_size = vm_area->nr_pages << PAGE_SHIFT; + unsigned long stack = (unsigned long)p->stack; + unsigned long *n = (unsigned long *)(stack + THREAD_SIZE - alloc_size); + + while (!*n) + n++; + + return (unsigned long)n - stack; +} +#endif /* CONFIG_DEBUG_STACK_USAGE */ + #else /* !CONFIG_DYNAMIC_STACK */ #define task_stack_end_corrupted(task) \ @@ -123,17 +139,6 @@ static inline bool dynamic_stack_fault(struct task_struct *tsk, return false; } -#endif /* CONFIG_DYNAMIC_STACK */ - -static inline int object_is_on_stack(const void *obj) -{ - void *stack = task_stack_page(current); - - return (obj >= stack) && (obj < (stack + THREAD_SIZE)); -} - -extern void thread_stack_cache_init(void); - #ifdef CONFIG_DEBUG_STACK_USAGE #ifdef CONFIG_STACK_GROWSUP static inline unsigned long stack_not_used(struct task_struct *p) @@ -160,6 +165,17 @@ static inline unsigned long stack_not_used(struct task_struct *p) #endif /* CONFIG_STACK_GROWSUP */ #endif /* CONFIG_DEBUG_STACK_USAGE */ +#endif /* CONFIG_DYNAMIC_STACK */ + +static inline int object_is_on_stack(const void *obj) +{ + void *stack = task_stack_page(current); + + return (obj >= stack) && (obj < (stack + THREAD_SIZE)); +} + +extern void thread_stack_cache_init(void); + static inline int kstack_end(void *addr) { /* Reliable end of stack detection: From patchwork Mon Mar 11 16:46:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD249C5475B for ; Mon, 11 Mar 2024 16:47:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5473D6B00DA; Mon, 11 Mar 2024 12:47:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F5756B00DB; Mon, 11 Mar 2024 12:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D34D6B00DC; Mon, 11 Mar 2024 12:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 199B66B00DA for ; Mon, 11 Mar 2024 12:47:00 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CE5591A0C00 for ; Mon, 11 Mar 2024 16:46:59 +0000 (UTC) X-FDA: 81885337758.14.83B7AE6 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by imf17.hostedemail.com (Postfix) with ESMTP id 0854C40007 for ; Mon, 11 Mar 2024 16:46:57 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=DUoR+5Wu; spf=pass (imf17.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.171 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175618; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q6FpZ49BpunKdsFtEl370XBCgostMe6YS8Mf7VMqlwM=; b=7wRGdQKgCsFpYRcQRsutD13eIjTlhjXgIBE5a/EWv1wsRp6hnHwSjx++w8uKT/qczeMXko NgrcJBbAtYhoe1rHoxCOk/WJqATG6bqC9q7FxiXC9jIOVf16B0PpfeNvvZ3X6lszYSWiz/ NDkHmIpWiGwUAfGwF/bf0/NpVJQiaQE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175618; a=rsa-sha256; cv=none; b=V77AlYoAxiBsH51wEBdVnsjip/Vse6smx5+a3dk4LIz2Tt+9lUv2zFTsITKSfjLcY5LLXP hWtHXYqqmox5rGbisRA5LvIyfeovBz27QLQvJ4q6al6vK/PUHWDwuyz5zkb8TumuZGsV1E m2idN/jIu4y5752cS+/KT1DzQEtxqp0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=DUoR+5Wu; spf=pass (imf17.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.171 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-7884a9a47a5so122047685a.2 for ; Mon, 11 Mar 2024 09:46:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175617; x=1710780417; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=q6FpZ49BpunKdsFtEl370XBCgostMe6YS8Mf7VMqlwM=; b=DUoR+5WuqfpiAeuoIg+PMvE8hq7GJTKv7U0E2zh+qgw9eIC7A2Pq8iSe1g/xGzogXh dWBBptybEXLZcEFIDRLB04Ll/Zjk6PNRRqk7727GS3SjA5DNyE4fCJFxIkwzO9Xvieaf 1/IHOt+m/ngNQJ1oSVocibvU2J478hSmmJ1JHzqVMleAlhi/oCQd3ELdJonY93VC0bq9 GYveYV8paNWAwUY5IWNJGeZXY4kJR2Es1iLSQ3bbNbYI/VH6/oXG8le5mVDyRj8fo91f w3w2mmgM23dBZg5E2GylvPomKg9Zh9Lno1WuXagxjdsb9CHxLTXKz83ZgHsImH6hPrkm xTiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175617; x=1710780417; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=q6FpZ49BpunKdsFtEl370XBCgostMe6YS8Mf7VMqlwM=; b=no7sRJmW39DmAA0PnwjZ+LOxrFF+W3D/2OWvE12HR00BLkwmuv0vtiTCNA5StA1p+E 3Krv+iwSClKVllyF45J4ljO5zjtvQLCYt8humlfjFA+VaovN2b2Yn3RDn7YfOMvMATc7 gqreYv9ziKfSZQMh30NPnZl+JUxutj9as/eHV6xFKTd2SMsDErPWnU7AspDGiaN79euJ ixRSa84/792ZT/+JR0WvoMNu5Y0SXg/AhXz4VaF2CRolXI+lLflEdnMXb01f6Jg4PjDq OHBFe/hoQ7miN+XuauOJ0J4g63A1BYcvQ71RmDGzh881Dz+2l1q7ICsZ7cFmu4PC2ftj MZlQ== X-Forwarded-Encrypted: i=1; AJvYcCU97nF9M0Omay1vxsQ7sT2Dlqj4b/4tCI0EGa4wntdsByYaLUWu0RghZflqi2PydeTu6Da1PQ4464AfwwQr244pojY= X-Gm-Message-State: AOJu0YxBjbepH6tC3dwReAHykDd6nr1uimceNYQVztOplUbPvdheNiX3 ZxD+UIInMYIUPhoqcAwr9hn9mQBZAmt4YXRqZIKxyMvIO/fXGFNlULhu3/vIUqU= X-Google-Smtp-Source: AGHT+IGBBQHGTyj0HK6a6nT9QQX2CzgLifOnSoon3yUl5MxZhI+SOhyqTpiNJHmFfzsFPzFEmk6ojA== X-Received: by 2002:ae9:e701:0:b0:788:2e8a:1731 with SMTP id m1-20020ae9e701000000b007882e8a1731mr6728530qka.17.1710175617252; Mon, 11 Mar 2024 09:46:57 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:56 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 14/14] fork: Dynamic Kernel Stack accounting Date: Mon, 11 Mar 2024 16:46:38 +0000 Message-ID: <20240311164638.2015063-15-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0854C40007 X-Rspam-User: X-Stat-Signature: 97hk11qyaahkm3qs9riyhiadt5r3ir9d X-Rspamd-Server: rspam03 X-HE-Tag: 1710175617-381031 X-HE-Meta: U2FsdGVkX1+T3PWynvk8NYiDkYkcDDNobitU6b5iiwRcOT9W1e46FKLEWkUwhriSXGOg392UQ9fBbKweLy5rvDQQ3CLk9PjXx1gfOabZ3pAG04N2HpQ9y4MP1zUGeRcNBHQ498pVJZ+whbinJA29g0M5o+d4xV1TyPMs6r+c5MuT0eNAYQkDtRBPxgAXX/jzYAoJPxvuHQ97pXq+LHFbtV9iqBbPkR/C6RV0xl09VcGw+3d6gbToEMcU6BoE6oCt5sRpAvqZB37LMpFUeBSXm/8f0c9S4ZtklPrhFYCV1BdlFs0fbphzSdHVB+OaOtmjGfeBo95GNpmygaR6x6wYecdN8tKQXmbbirjLNnePHjhFHQs6frd9d56nXNpHG49MiAiHGHTYd1RWfz3S3ZD+6ViBiT5iODeyiHcgot7mBgPTYFoEzR+DvGLMFJCLVgrU5fRtDHYI7xtrkSnIziuHP1CGUUoNAThknjEv5eVIAT8slnhcC7EWZzHaQWfNTAlY52wrf8pLheoDIF5vrvme5HvkH40qCBZxR1s2v0i/4SY+nSdN5m7JKpwZEaX5xnHtPVcFOGALTe5azqYxejo0qzJgfqAOfVpIcx6I/tKLfJOw0QmAuxKW3+r7kvR2Frr4bOeK05BMnox//Ut0Zd3EgPfukJgbdj10RPlcjrwxlMM6C6QGHbR+U1Us71dp7lhOtJnSKwRQGyBCr+RZzr+5ZSliwI+1PWIYvqLxJP474Pegk/KnH1j7TOPVsJMEJXNDxQdE7sHcsXiWxReihQ6JmksofP7FrJMR6a75H5v0JD7QltIX3xnPWiSoYxayhFwTwEHCyKCPl5OYH+JROs16TcxtDNFCiuogvwZPiBt5Mfte40WRSbN/nIFgbroycYMIvr1ZKMYeSYWgKzPZJ/nf/6goISV3kQReF+JMT6c6byI8/vNNaO9gX/0J+m2kpvZFDgGmGlYu9iRzZPoyN5m AQbRMAiL AWLNzQsvT4eL8KXWxIIHmUe6k0TrjitHdFY320OjxMbTXwJNFlYCYi7CHGk9eH17QpNbzS4f/AY5LeVHwzebW/q02gvmbKVlkWc2+3nHDfbn9KqKxt2Wgvj8WDczfZH0UhUokgCeHOZPJhOGZD1m12P1IX8ikKjkviTGEmZrNzJahwqYtuz/LyhntholWf+EJwTwNlFIDwYlSlOtAaysBPO42x3FwIfBqVjmu5Jvt3r5c0bX2XDRibAj4T6bUooG1Eka0kfqMB2rqujgS3rBpqN1421amv4LkuVpn45VD7CLrKfQgtDy0k5Zql91DuYWa0f7F+cWHZCE0z89FTm4qrHmRKuTYdhm6NOono36fUgGjk1w5JDy18S/qPuu6Y5DJIdX9K6A5o5Z5Tty2LkW8BHtlEarVJ6Ikep/zzRax1auRsS8vqd07V08KUIxXtSN9Lmp07IZuBrDN0io= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add an accounting of amount of stack pages that has been faulted is currently in use. Example use case: $ cat /proc/vmstat | grep stack nr_kernel_stack 18684 nr_dynamic_stacks_faults 156 The above shows that the kernel stacks use total 18684KiB, out of which 156KiB were faulted in. Given that the pre-allocated stacks are 4KiB, we can determine the total number of tasks: tasks = (nr_kernel_stack - nr_dynamic_stacks_faults) / 4 = 4632. The amount of kernel stack memory without dynamic stack on this machine woud be: 4632 * 16 KiB = 74,112 KiB Therefore, in this example dynamic stacks save: 55,428 KiB Signed-off-by: Pasha Tatashin --- include/linux/mmzone.h | 3 +++ kernel/fork.c | 13 ++++++++++++- mm/vmstat.c | 3 +++ 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a497f189d988..ba4f1d148c3f 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -198,6 +198,9 @@ enum node_stat_item { NR_FOLL_PIN_ACQUIRED, /* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED, /* pages returned via unpin_user_page() */ NR_KERNEL_STACK_KB, /* measured in KiB */ +#ifdef CONFIG_DYNAMIC_STACK + NR_DYNAMIC_STACKS_FAULTS_KB, /* KiB of faulted kernel stack memory */ +#endif #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) NR_KERNEL_SCS_KB, /* measured in KiB */ #endif diff --git a/kernel/fork.c b/kernel/fork.c index 63e1fd661e17..2520583d160a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -343,6 +343,9 @@ void dynamic_stack_refill_pages(void) mod_lruvec_page_state(page, NR_KERNEL_STACK_KB, PAGE_SIZE / 1024); + mod_lruvec_page_state(page, + NR_DYNAMIC_STACKS_FAULTS_KB, + PAGE_SIZE / 1024); page = alloc_pages(THREADINFO_GFP & ~__GFP_ACCOUNT, 0); if (unlikely(!page)) @@ -771,9 +774,17 @@ static void account_kernel_stack(struct task_struct *tsk, int account) int i, nr_pages; nr_pages = vm->nr_pages; - for (i = 0; i < nr_pages; i++) + for (i = 0; i < nr_pages; i++) { mod_lruvec_page_state(vm->pages[i], NR_KERNEL_STACK_KB, account * (PAGE_SIZE / 1024)); +#ifdef CONFIG_DYNAMIC_STACK + if (i >= THREAD_PREALLOC_PAGES) { + mod_lruvec_page_state(vm->pages[i], + NR_DYNAMIC_STACKS_FAULTS_KB, + account * (PAGE_SIZE / 1024)); + } +#endif + } } else { void *stack = task_stack_page(tsk); diff --git a/mm/vmstat.c b/mm/vmstat.c index db79935e4a54..1ad6eede3d85 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1237,6 +1237,9 @@ const char * const vmstat_text[] = { "nr_foll_pin_acquired", "nr_foll_pin_released", "nr_kernel_stack", +#ifdef CONFIG_DYNAMIC_STACK + "nr_dynamic_stacks_faults", +#endif #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) "nr_shadow_call_stack", #endif