From patchwork Mon Mar 11 16:46:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13589075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61051C54E58 for ; Mon, 11 Mar 2024 16:46:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D815A6B00C8; Mon, 11 Mar 2024 12:46:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D31266B00C9; Mon, 11 Mar 2024 12:46:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFB786B00CA; Mon, 11 Mar 2024 12:46:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B100A6B00C8 for ; Mon, 11 Mar 2024 12:46:46 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 58A0D140205 for ; Mon, 11 Mar 2024 16:46:46 +0000 (UTC) X-FDA: 81885337212.28.64E92E9 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by imf02.hostedemail.com (Postfix) with ESMTP id 9C1BA8000D for ; Mon, 11 Mar 2024 16:46:44 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=tbXmVaAM; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710175604; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=jKDy5XLn4MxCpgW/wos6NfCyYowjlJJBXq+4EzKDJes=; b=vXIFZaIJ4rSRA1Q5BglAY0msXv3fpDexANjhvCcUmjS/+eQmWgNj4HI3VR0nrIv0tbdVS8 o80VRJ1ithNiL+9qBMrUBEuo4JXYndQMPFH2O4V6gie+Me/2RaOU4yzPyPXFjc5J7RBYYS lm71V0qRkKsrEvgFhAKJhereqaQvmOU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=tbXmVaAM; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710175604; a=rsa-sha256; cv=none; b=wsG+6iKbhCo3lPrtkXFZCtRpjyb9d4fppOt8O5W7rw/x8z5P2VQe567E0B1LNlnaXOKXx6 w1f/u19b16WOrh33AWPGtkpfh6JNY7Tw74kXGEqBcO1U9r78c8nfKjcgcKIUKjP1PxSWk5 WROk+M5hV3Xh5AoOz7+JUxRtKNwZK8M= Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3c1a1e1e539so4190841b6e.1 for ; Mon, 11 Mar 2024 09:46:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175603; x=1710780403; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=jKDy5XLn4MxCpgW/wos6NfCyYowjlJJBXq+4EzKDJes=; b=tbXmVaAM1y9LSo80eVThs164EEyqSYk84U6iiy2232p2CxkPKzsxcY8DjIqiy5VTVh UvlINGv2lMZvCW7aAvE5HErJpDWqASiY9vFHXgXyDaWUzVFCnv6I8mnu3+Hn+vh7HZTm 6XQkXDzIy6GNfDlWH3frF5ieKKbPiastlZ8t4HJEoptXpQQGyXs1+VKY5trxlpDcqFBe oZzt9NXB1XOAZtRkaVyf7rJLSzePJRiKQFVVorOx+wL4v1FoYFBsyep3YkBP1t6Qfxx3 DW/rAKObrXx2CbI7nr6OcDZdZWK31VPVzOfOaSq/n7BYLWFyXzo45D9iZRNoG0+iI0Bv Sdiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175603; x=1710780403; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jKDy5XLn4MxCpgW/wos6NfCyYowjlJJBXq+4EzKDJes=; b=FhujfenFKKDNsW9g97W3qcLunA8Xjrp8R0lgDbjaYlmJnxqlXFu9zqDM+2lsbBPj9I P3uNk5j6sRGqgKzB2teDJlmLyRLpnjZCT/sM/eCUNqF6NeKse2ZAwzwsVSn9unJrOCog 67oLeZ2ZlqtizeOeHK2XJPGQ01dvC+tUS79AtlXLyenNcqzw+D2qHws6c8hfLqOhZcWR PrIfEflrdPzUtEKQV+5aKHoZS2i7v58MM7n1BlbKSiYBv9QsVwdEGZRIYOx8D9KpC+3h Hz3plocbEeaJn//HTDOqNnIiBBD9P1AEKwIvB7zZh87z239RTMoAjBR9c0yEwZ5298i5 3Heg== X-Forwarded-Encrypted: i=1; AJvYcCXOrdqrClqDZ5QXSQIPxz3aBHCv87loogkKF1MRdXzLluvfMBRxK8elUP+qZGCYP7XzeOBgLxCK5MzQIRRd7OlJRqg= X-Gm-Message-State: AOJu0YygHzgM2H6OTFAuHb6Cl3L53RrUSxZIwD7OV9hx6Om460hfv2VY upS15HatUyBov8yljjPmIHPlwrbEssNP1qzzvoWi04Tn6AgV5mndff9oavwnzTA= X-Google-Smtp-Source: AGHT+IHCf3gSyDwpxMQqAwcZOEE99Um3U9jb5cEHcYXAVoTC005pLQoY23vpn4Os61gkSYgimn/PHg== X-Received: by 2002:a05:6870:65a4:b0:221:ced3:4748 with SMTP id fp36-20020a05687065a400b00221ced34748mr6602625oab.24.1710175603623; Mon, 11 Mar 2024 09:46:43 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:43 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 00/14] Dynamic Kernel Stacks Date: Mon, 11 Mar 2024 16:46:24 +0000 Message-ID: <20240311164638.2015063-1-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: i74hxpa7utjpd5rsbysdeb5qyd3qsrqc X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9C1BA8000D X-HE-Tag: 1710175604-335715 X-HE-Meta: U2FsdGVkX1/3Qp1tO/S029rjyGkMzm9ftcgClNahL7Vmikqa2HwC032agDiVFSUwuJzYvaR1NybEmnVuDZky4dwVnozw4dtmPCKilbZYMlU3o33yaPXoU1uI+XuYUdiE+z32eGuRJA5t+VU2lP+l0Eoq/xbli9h8e/dsoYQYkJbXpNKk9ks7lMHiDCiX+nODK5WXVxwe/qwNR+53QixT9/KaXmzuxdzKRo1Criya0jdMZrLiwgXIqpqAPM3xNM0vTsYIORz/GK4+UybgeFtVp9ZORIfueECpdM2twiNw5l/TolfzxA0j+N1H/bGQAdwE12kGWtKOv+HdQXel2/rUhPld0/tR/xdtlP/B6sNcc0ULzvlhnDxDh42SSBwEBpZs7U9Ww7rJy7SHpoHll2kRL4w/y2olMoAN/sRNYkKIohiAgCN+Btl/thy+skU0gaik2wDTC8/aJstUjdnCQ8/cWo3E9LeTDOYxeLGb4HWEjko3usTJxtzIwDcpqIKMMHLBAl3h2V12TpdKoMSnG9LBev0y5BGpT1JzJWUKkwNgzcM0aVpq56Wm9lwzKpo5p3n9DgROIpjxzw4OZaF4YpAK7F5xIZrWXsQCky36g7TyjC47/o81RzRyqebw3n8oI736yt9dFQQSvgqVBqyzYXgR21WU92pvI9O3GbdH1q/IUN2zsySgUbV3XXccY03lT78E9c/JmrKjmqcjjrS8qJvs7dT0TvN4jmLDu6prtjdq+jGcoMf05JLK5rqe3Wzd7invmaY7rYmenQnBUCs7iOO5oRbxL3IjPw07hyW1Hb2fjKI5ggZt/06pGT0ivHRj5sc/z+P8vIVZp0cRu8VrrMJydoj8F/kycIywfve6Lh64jiKVxYOI8wDVDF/0L2gXliWIoEApjSRCNkQVgZome2gNTr8EsQwk92hlTznVl9xKtA86aumLsvvTToi/ERoLzrFvnYTcQct3qUAkdTjcHho lbkYx/bW FeHxD4hK9rTa0V4tOxo2cDTwB0QL5JaYKFtCDPLcA5MNma9likzK/tHdsRr0uuivSO0KnuJfcHtuXAWAm5e16oTyxvH8BhWVOIJVwlpuK3M8kUexbdHQOaLSPe6p+RVoaWnbopuPxpT9d0QCmccLo3ub6HG1fA8bQC7EH1UB9aFm7oI6vfv3GzqnKO4N3g+4SObMwhflliFQT1Dqmy074go7Yan/FM8gCvCdIvgxSX2aFewPdrrI2yFTrafWQlqeI+OpGiZxrBY9lTEAfVrcDJFxlSVy+jguI8bX4rRpFGAuVRgZWMYvtmRqT9WvNsCWZ9oy+aMTuA1Na0RBt9lhDAg8dSZyMAocWpkH2b1xgpRcCfezLiKOF6/Jh32Q1hhSGEQyYqFfyB2TKVNNE9NA6764fFadi7oxGeB+bnA3Z2EL2YlOteT+2M+D1K2dAP7ntXm++2IfRhOMcSxLRdfcR3KNsg2hFWDWZjd20 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is follow-up to the LSF/MM proposal [1]. Please provide your thoughts and comments about dynamic kernel stacks feature. This is a WIP has not been tested beside booting on some machines, and running LKDTM thread exhaust tests. The series also lacks selftests, and documentations. This feature allows to grow kernel stack dynamically, from 4KiB and up to the THREAD_SIZE. The intend is to save memory on fleet machines. From the initial experiments it shows to save on average 70-75% of the kernel stack memory. The average depth of a kernel thread depends on the workload, profiling, virtualization, compiler optimizations, and driver implementations. However, the table below shows the amount of kernel stack memory before vs. after on idling freshly booted machines: CPU #Cores #Stacks BASE(kb) Dynamic(kb) Saving AMD Genoa 384 5786 92576 23388 74.74% Intel Skylake 112 3182 50912 12860 74.74% AMD Rome 128 3401 54416 14784 72.83% AMD Rome 256 4908 78528 20876 73.42% Intel Haswell 72 2644 42304 10624 74.89% Some workloads with that have millions of threads would can benefit significantly from this feature. [1] https://lore.kernel.org/all/CA+CK2bBYt9RAVqASB2eLyRQxYT5aiL0fGhUu3TumQCyJCNTWvw@mail.gmail.com Pasha Tatashin (14): task_stack.h: remove obsolete __HAVE_ARCH_KSTACK_END check fork: Clean-up ifdef logic around stack allocation fork: Clean-up naming of vm_strack/vm_struct variables in vmap stacks code fork: Remove assumption that vm_area->nr_pages equals to THREAD_SIZE fork: check charging success before zeroing stack fork: zero vmap stack using clear_page() instead of memset() fork: use the first page in stack to store vm_stack in cached_stacks fork: separate vmap stack alloction and free calls mm/vmalloc: Add a get_vm_area_node() and vmap_pages_range_noflush() public functions fork: Dynamic Kernel Stacks x86: add support for Dynamic Kernel Stacks task_stack.h: Clean-up stack_not_used() implementation task_stack.h: Add stack_not_used() support for dynamic stack fork: Dynamic Kernel Stack accounting arch/Kconfig | 33 +++ arch/x86/Kconfig | 1 + arch/x86/kernel/traps.c | 3 + arch/x86/mm/fault.c | 3 + include/linux/mmzone.h | 3 + include/linux/sched.h | 2 +- include/linux/sched/task_stack.h | 94 ++++++-- include/linux/vmalloc.h | 15 ++ kernel/fork.c | 388 ++++++++++++++++++++++++++----- kernel/sched/core.c | 1 + mm/internal.h | 9 - mm/vmalloc.c | 24 ++ mm/vmstat.c | 3 + 13 files changed, 487 insertions(+), 92 deletions(-)