From patchwork Wed Aug 23 05:06:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mateusz Guzik X-Patchwork-Id: 13361598 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 369C4EE49A3 for ; Wed, 23 Aug 2023 05:06:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7098328003F; Wed, 23 Aug 2023 01:06:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 612E528003C; Wed, 23 Aug 2023 01:06:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43EE728003F; Wed, 23 Aug 2023 01:06:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 290E628003C for ; Wed, 23 Aug 2023 01:06:21 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 01D4040128 for ; Wed, 23 Aug 2023 05:06:21 +0000 (UTC) X-FDA: 81154183362.26.0DB2208 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by imf16.hostedemail.com (Postfix) with ESMTP id 2B12518000B for ; Wed, 23 Aug 2023 05:06:18 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="nbGMC/v6"; spf=pass (imf16.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692767179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iKHCB44AckVFwTchMQuBCPQ5Mr4hBKNgvgKNhep04i4=; b=QTukmFZ59mJ4B+Q2dTi9MeLx5COy1bZySdYtuAS/maRgMBl92EY9S1ni5K1Wa3D7aN+QjG KWebimMclwdsVMoY5+IEZfjHEr8PDaWK6rS8PPHre9tXeZ86hokR03pn4M07ytCGYX3CYO Q0jsyT5565peAhOcqmsO1O1OKACnWkU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692767179; a=rsa-sha256; cv=none; b=T7THa8SzXtx+KjjnI6lxZ707W8z/4/r1592IO/oaMU8wHOZ+j7c7qERLE/BjYhkSy4r+Zu B9SqqBSLdIn45cflmyCaZ4smVduKdK7f6kqb8JzYz+mqrL0vvhcLlyNNnRFAnErg6ySsNc Ak/oE+1QyNFzrqYDpnSKNOMn27CXKj0= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="nbGMC/v6"; spf=pass (imf16.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-99bf1f632b8so696495966b.1 for ; Tue, 22 Aug 2023 22:06:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692767178; x=1693371978; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iKHCB44AckVFwTchMQuBCPQ5Mr4hBKNgvgKNhep04i4=; b=nbGMC/v69gsCL1IBVf+ztV6UvaaYtA28bsu2aG6AvtQMBJ+xqE93v5gEURtPqyh+JM a1MdLF+6B9gcA/ErtCCRYwr6N8uk4hiVZIa5fwbicH+qxlKLgPpX8V8Jr4t+5dhgPHjO HfNCETQIIeDAprwEzHDEabadn+frN0HXrfZcoQ5LUCeQlooltrZns12hOr5bS7GeJ6eT 39KbYobKl8/hc9d0MxYsL0hN/2VjmNZuGYvtxgHwFpeJ7D1eRBtdGnd3KkB4NX4bywDL 1NZuON4UUJvw4jjxWq5Jiix4+HFHnB7c8wvbpR9KwGf/vq93yXNr+VBD4t9ellxcZLQu Gn1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692767178; x=1693371978; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iKHCB44AckVFwTchMQuBCPQ5Mr4hBKNgvgKNhep04i4=; b=S4lN93YCBDeIdRL37vb5NzSD+lxWUpsjuOCQtN7YjPZIjcgxb7Ip3I19ahFHX1Snkc HttSsRZcoWbeHH4IE3O1tgppo3KPKS3RgD//lNGu0jaH8wOscnPtei0gvnkKmLNRsxEP LSmzjevCYztQ+9S0yHGjXWuDVrduDA/afZy309wXjexVnBY5Oeum6eepUH9XMGhR4V02 O5DNBibdhac1Sjn/iRp2Upw7urvQFKU/fX5bhlx9FDazi0sVb6d3uXivMwQrQ1HM+mcF LcM1wZsTLAJefV+/elYrs3ugHbEkJIdTJZvBYF+f4MAbz0sycYwfYSoN8OVgrvIt1CR9 cRvA== X-Gm-Message-State: AOJu0YxaaiC1dMXnWGXI4QlLTbpari1qVK6iNhKArz//AtHtCS7IsEV1 6pHk2grVpYEbVY8niU3sxag= X-Google-Smtp-Source: AGHT+IE7nSULLshI01yEFzWITte3YVllrEGWbX3MHiHkvCwHHx0LceZup3+ufxgmQEyozhc7uPZ28Q== X-Received: by 2002:a17:907:60c7:b0:991:bf04:2047 with SMTP id hv7-20020a17090760c700b00991bf042047mr8784854ejc.14.1692767177624; Tue, 22 Aug 2023 22:06:17 -0700 (PDT) Received: from f.. (cst-prg-85-121.cust.vodafone.cz. [46.135.85.121]) by smtp.gmail.com with ESMTPSA id q16-20020a170906b29000b0099ddc81903asm9267401ejz.221.2023.08.22.22.06.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 22:06:17 -0700 (PDT) From: Mateusz Guzik To: linux-kernel@vger.kernel.org Cc: dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, vegard.nossum@oracle.com, linux-mm@kvack.org, Mateusz Guzik Subject: [PATCH v3 2/2] kernel/fork: group allocation/free of per-cpu counters for mm struct Date: Wed, 23 Aug 2023 07:06:09 +0200 Message-Id: <20230823050609.2228718-3-mjguzik@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823050609.2228718-1-mjguzik@gmail.com> References: <20230823050609.2228718-1-mjguzik@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2B12518000B X-Rspam-User: X-Stat-Signature: hmtt3xgtbtdy9s9ytwgdk7bjhd6b59za X-Rspamd-Server: rspam03 X-HE-Tag: 1692767178-40990 X-HE-Meta: U2FsdGVkX19UvDhfXtjwlQzocCJETw8NbO8bAZGTRfd+2GHfRluHy4iQUqrudomnduhyaG9OAQ24VJT6XgpKSfVIQkN60bwmusaxverBPAt/y8uJF06LgnZTuO62xnZ7x22CUW3FYiCflpU2UeX3lI5VaUvB+xqB0Y0fzVkJTsTV6RSSM5XiSMbN2F2bPE8GW6pH2RZ3e8RnQ/OZ4fkzKhL4Jlzr+Ty+gnw+d3eWgPQzYHa1oAwGfIIcsM81dEA6xWYNkmxgOXOAG49oL/ObypdyHuJVn0U4iQeZMVRZUtjcZ+jpJHNX1f9gxGHY1N4Dlb/796u7VO5uVgG6a9eEgMXwmpwyoCcqLiW1x1T3cuSish/gsj6nuIE0uwYaMch/mjgf3y3tyK3tqSSdkY7YOnVdN075VU41M7zHaFOh7Z1eiHMMFsMXVeHLBSErI76NqLlSOmzQ8AT/4CvUdTBxPEGd9PHUIBdM1fvMr1Hded/76mXe932/D0/XLsV8+bai4WC4Cty/7KzbIzDY1CX6qyJNWCT/SB4Px0jMxMP4kkIh8BZ6zU4Ov/fWikUj8a/03HDoP5yfWk9fk/7+IXwHc3OvIOEwir2TznA24wYwV8Jw7JgJRmp2dmtqtvCr9Rbk3NlWMN3xlcSJFy1HxEp5Ec5eDucbmvQaRVaIhQtQF/w95lM0esUrpOr50kpphJL+YRl/uofAd3xUdxp6lhahth4C7MTJ+m5RWth7n+6mf3t6GHU45Ndje042qJPzBfNBAIqWNnLSUj7pFDwBDF3LkRukazOYwS46WD/TJ+wmXB7muumudoSZAy16zjpLaFHarFCKKDrlgeHuO8Uwzi2eyysRGWP6fNrRv7N19U7xiVW005cvDPX+z9X+NyXZJGSj3b0uwxWaFZt00+L29C6vhiqhUogzzX033B5BjRFI0HT8mQYYukyCn94jQNm8D1jS0OVv+LsgCqUlyhKq2bJ gpmxRUcM jSh816m3OTBKYtlz+odaU+n/NruNvUTYE+rfxOB7oMGY2qHaO0pMdH9iA3xrxEQ9hz10vEXdHAWPPnKy8uYqAy5CmaFi6rLz9FNEfpt96FfbUQR6Qeizqp4/M047XlfpoV5NXlXdvL8ZQ+c2MreP4CTW+Chfcd4lJx4vx+Jg1iY5njbXtxbnZhWeerbnMKYKmX7uCvgH9CUutwgcF56GdmFE0wnbjcSfpH7CrrM0aYaAh3FgHpC4vfsnToDST5+mBi5gIe45lOOFhrXbQf7sotlb7PsB+21lh9dA7ztKEGhhX7K4pcj60o9CjdaQqZczoT/QY3A+9VBIDDs/NAdNSI+ipSItWUpF1RfC/EQpPMRN0TmclItxePH9/bXNCWH4/F8iFYGF2YF+jz4Uge52ukPIYwym2lSlpoEMQjbSqKA0ww36QeDXxxD7VngRJ6/ashIrRvjFJaRRtE0Kty/UYM6QNsDpn+zG0HyYMuyAMMxvgK76xPo/mFwyf7A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A trivial execve scalability test which tries to be very friendly (statically linked binaries, all separate) is predominantly bottlenecked by back-to-back per-cpu counter allocations which serialize on global locks. Ease the pain by allocating and freeing them in one go. Bench can be found here: http://apollo.backplane.com/DFlyMisc/doexec.c $ cc -static -O2 -o static-doexec doexec.c $ ./static-doexec $(nproc) Even at a very modest scale of 26 cores (ops/s): before: 133543.63 after: 186061.81 (+39%) While with the patch these allocations remain a significant problem, the primary bottleneck shifts to page release handling. Signed-off-by: Mateusz Guzik Reviewed-by: Dennis Zhou --- kernel/fork.c | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index d2e12b6d2b18..4f0ada33457e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -909,8 +909,6 @@ static void cleanup_lazy_tlbs(struct mm_struct *mm) */ void __mmdrop(struct mm_struct *mm) { - int i; - BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); @@ -925,9 +923,8 @@ void __mmdrop(struct mm_struct *mm) put_user_ns(mm->user_ns); mm_pasid_drop(mm); mm_destroy_cid(mm); + percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS); - for (i = 0; i < NR_MM_COUNTERS; i++) - percpu_counter_destroy(&mm->rss_stat[i]); free_mm(mm); } EXPORT_SYMBOL_GPL(__mmdrop); @@ -1252,8 +1249,6 @@ static void mm_init_uprobes_state(struct mm_struct *mm) static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, struct user_namespace *user_ns) { - int i; - mt_init_flags(&mm->mm_mt, MM_MT_FLAGS); mt_set_external_lock(&mm->mm_mt, &mm->mmap_lock); atomic_set(&mm->mm_users, 1); @@ -1301,17 +1296,14 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, if (mm_alloc_cid(mm)) goto fail_cid; - for (i = 0; i < NR_MM_COUNTERS; i++) - if (percpu_counter_init(&mm->rss_stat[i], 0, GFP_KERNEL_ACCOUNT)) - goto fail_pcpu; + if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT, NR_MM_COUNTERS)) + goto fail_pcpu; mm->user_ns = get_user_ns(user_ns); lru_gen_init_mm(mm); return mm; fail_pcpu: - while (i > 0) - percpu_counter_destroy(&mm->rss_stat[--i]); mm_destroy_cid(mm); fail_cid: destroy_context(mm);