From patchwork Mon May 15 14:35:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 13241624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD8AFC7EE23 for ; Mon, 15 May 2023 14:35:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12FA4900003; Mon, 15 May 2023 10:35:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DF3D900002; Mon, 15 May 2023 10:35:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1034900003; Mon, 15 May 2023 10:35:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E2ABA900002 for ; Mon, 15 May 2023 10:35:46 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A1BE6AF3D0 for ; Mon, 15 May 2023 14:35:46 +0000 (UTC) X-FDA: 80792738292.20.1C562C0 Received: from smtpout.efficios.com (smtpout.efficios.com [167.114.26.122]) by imf01.hostedemail.com (Postfix) with ESMTP id 3875640002 for ; Mon, 15 May 2023 14:35:42 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=Ygbu0By2; spf=pass (imf01.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684161343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=ZYEArWolIZt4sa6YXFI25cdVhoWmld9ONGA/zqZyRNw=; b=WF4qJAIQJ1oF6GBT6YCf2B6BQURPcYPVg/z4Vbu8aawFonJ5B1uNOtAMjADWXu6LOXhEj4 SlDgJHxIDOkiwX/Q7Dvymdy+OfMCzguFpXxPn4TxmfTTv/ZNfyWxSM/Vmmpw0jUWhSG9dv Q8cWJykh4Y8VDNcB6iE43ok7tZE53mk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684161343; a=rsa-sha256; cv=none; b=eTzn7n7hmewuxMDAt03xbmDRL20acu631UR4qd73l52hCA727Vzcpuc5Srf8k5iy0MmpM/ MyGbEQBLSjSYZoyxdOZkOMN+QbNmFBapMuzLY5LnC2+be6okhcGdWUkrlEfX0XTi/9Uo7m c7yWhIPsJM0Mt+2UM1K6TmkNslwy/yQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=Ygbu0By2; spf=pass (imf01.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1684161342; bh=55UiL7049d5WwXoq8Ga1ETl0Xcqd3k/79bhP8jRai1I=; h=From:To:Cc:Subject:Date:From; b=Ygbu0By2kzmJLHh1Fy4KXvfh+K9aNW5LbzhbEX5LeLatHUBNzOUEcNYDrcUw5Az5e v3SH4smxTxOMBPYSXoGiQuPqbVFeQMfc02K4UtkVhJMeHCcKxxgL795dpfbwruTact TdIrD0ixOzK3WpYzJKsh77uwmo9LwNOyNw2KZLr8q0TNVo0EGf8faPJezJsV3+JtuX zZFsLOUVQYckfd3VO9KJkCXjOA4Ixt05jknCAgSUVrIgCecJycLf3JuVsqt7twXxav 9Ma1bfkisrmpyWi+8puxAJFgbKbBvMCrn5rzbym0yGp/GAAAZNzShPunQjAlsiohQ7 NxI3QqSkgR+ig== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4QKhk55lzxz12dV; Mon, 15 May 2023 10:35:41 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , kernel test robot , Aaron Lu , Olivier Dion , michael.christie@oracle.com, Andrew Morton , Feng Tang , John Hubbard , Jason Gunthorpe , Peter Xu , linux-mm@kvack.org Subject: [PATCH] mm: Move mm_count into its own cache line Date: Mon, 15 May 2023 10:35:36 -0400 Message-Id: <20230515143536.114960-1-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 3875640002 X-Stat-Signature: efd1c76am5eqa1y5c9ko8eaejm94bzfr X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1684161342-171434 X-HE-Meta: U2FsdGVkX18jIJKtr9JJ55Sw9VPbwLEji6/Rmj54KyuMhTeQP5b40AxyKoXWwKCMKwml3tj15IZLaiEANu60grrXnj/RJ3t+lSjjk95EfyxUnzpqABu1gSyuHI/fFgEB+wbLXIuyExSSYbrfwqehLVomAZaYaZKc625XBT6MeZ9BcAKnDrQG5AcJAJZr5Kxw4G/C743RfVO+zsrKyzj5+ySls3B4hj0fseDkYDTWYG9o81Hn9op+9pWEN8XdxitMRuCHKcFB5qp1xeUnUsmCBzxwXL4ArbP218RPuPkHzJi52FXkgMwigDvHh1Cvq+BDXR9TupbWsGkmA26BmXqWctv+C4oJXfR1YOOTRYvlVHHL8xQ7PyKuYYhdvfe6rbGS8yDSjuNxBT0MQzR1GlN3gW5vSMOkzddS0NA0jN0QlikCPPAbsy2JKQUBD+Ow9OmuZXGyKcp0syqmeJ7ruEJJtFClZX3L783hp6GOvyc/yu5aJLRUYRELfLVbWDWdpozjhmZRdQRh+5WC2cxhDjkODHyi2gGj2xSrnjcbgsznDzWy0TgSqrs0QiNlb5Q+cAdL1XI5DOFxIYSRPQ+AxMhkVV9GkDgQ+kP2aYHnZmyXAHstW0noOFdiLTedgLlinrNHe8/X8L5QcamE6W8x1XNGSo6RWYeqVb8o7ufioLizjlwQI+UGyftky0NkUKbx30nVZSsmoYzdr54aw3pnEjMC9GX83E6gO+C2GfoOcOLBuJSSCSq46f53Tj1Uj+1W2K2iwYCAia0sUr+cOg42nQIDjbdRgLu8hywEnKQSySaq7pHuq6qaM8wa2JOn1cNIoAQJ2RsaTXZkuTCD5C6wgjI0X0gA0L1WDoN0TKh18XdHrkDekfdFO25W5myFmrfAOweQRJodJuRDwfPKzmZKvFsaAQknDCSuJ94lsnHHNPQFye9dUsx749bl3YA7bH9sx+nEaH3tg8PmwdmURcwWbxa Tt2kBKl6 SqhE9jDb0gO/PaJ3TKnnyrHmybXr8XKSiJWuhW43no81T6W7JaQP9Bhc2wC2AvHr56YJBj8nWq73QhPWKWk5t5rQqvp+ow+FC4HQPpyqj0UiJAT3n/cVPfCN2Sc/AQoQwwWqzceiGT3B9/c/eX6cAj2rRTsCMa6Lrm11XF7UAd0I5QT0klQSr+5Sbv3vbab3ktr5CZaIGo5gPvTazuxeS9ePJZTtSztkXkHOUl3eloHhjCuqhPCnghWacduZZIKR+zf2JEzyFWk+4pJYxJO5YLCyP9pq2vdfDcKq/Vg9+g/UjovWrkZpOIORDj2aexHc6HcKhxwx3P1D+aVdmOmxLg0HRX7OYIlBY+BcwhDpYfqsXtk4R2pq/cgO9x1vevyZ63fgpRTWloVi43Gib60azSlGdFPgJYJckLQO8tyGP4KhpjLAsYnOv+wSuABz90xMijLpD7t3R9qmtfUGbupIpRVA9bXaoBi15xJyjHqk/K9+ae5zB02jk4ua3K9f5zAqlyfjlCKjqFEwYFbVsNBt9+1YGq6Lp6nP2G8xeJ4aZ85xD2L3Eb4peuWg54P4mzjvhadYLsTIWKVx60iC5rnfnLd1ZSdRfaMLytWxOUGtwbvWHy9pziy0pF55sFgut3+GK1BAn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The mm_struct mm_count field is frequently updated by mmgrab/mmdrop performed by context switch. This causes false-sharing for surrounding mm_struct fields which are read-mostly. This has been observed on a 2sockets/112core/224cpu Intel Sapphire Rapids server running hackbench, and by the kernel test robot will-it-scale testcase. Move the mm_count field into its own cache line to prevent false-sharing with other mm_struct fields. Move mm_count to the first field of mm_struct to minimize the amount of padding required: rather than adding padding before and after the mm_count field, padding is only added after mm_count. Note that I noticed this odd comment in mm_struct: commit 2e3025434a6b ("mm: relocate 'write_protect_seq' in struct mm_struct") /* * With some kernel config, the current mmap_lock's offset * inside 'mm_struct' is at 0x120, which is very optimal, as * its two hot fields 'count' and 'owner' sit in 2 different * cachelines, and when mmap_lock is highly contended, both * of the 2 fields will be accessed frequently, current layout * will help to reduce cache bouncing. * * So please be careful with adding new fields before * mmap_lock, which can easily push the 2 fields into one * cacheline. */ struct rw_semaphore mmap_lock; This comment is rather odd for a few reasons: - It requires addition/removal of mm_struct fields to carefully consider field alignment of _other_ fields, - It expresses the wish to keep an "optimal" alignment for a specific kernel config. I suspect that the author of this comment may want to revisit this topic and perhaps introduce a split-struct approach for struct rw_semaphore, if the need is to place various fields of this structure in different cache lines. Fixes: 223baf9d17f2 ("sched: Fix performance regression introduced by mm_cid") Fixes: af7f588d8f73 ("sched: Introduce per-memory-map concurrency ID") Link: https://lore.kernel.org/lkml/7a0c1db1-103d-d518-ed96-1584a28fbf32@efficios.com Reported-by: kernel test robot Link: https://lore.kernel.org/oe-lkp/202305151017.27581d75-yujie.liu@intel.com Signed-off-by: Mathieu Desnoyers Cc: Peter Zijlstra Cc: Aaron Lu Cc: Olivier Dion Cc: michael.christie@oracle.com Cc: Andrew Morton Cc: Feng Tang Cc: John Hubbard Cc: Jason Gunthorpe Cc: Peter Xu Cc: linux-mm@kvack.org Reviewed-by: Aaron Lu Reviewed-by: John Hubbard --- include/linux/mm_types.h | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..de10fc797c8e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -583,6 +583,21 @@ struct mm_cid { struct kioctx_table; struct mm_struct { struct { + /* + * Fields which are often written to are placed in a separate + * cache line. + */ + struct { + /** + * @mm_count: The number of references to &struct + * mm_struct (@mm_users count as 1). + * + * Use mmgrab()/mmdrop() to modify. When this drops to + * 0, the &struct mm_struct is freed. + */ + atomic_t mm_count; + } ____cacheline_aligned_in_smp; + struct maple_tree mm_mt; #ifdef CONFIG_MMU unsigned long (*get_unmapped_area) (struct file *filp, @@ -620,14 +635,6 @@ struct mm_struct { */ atomic_t mm_users; - /** - * @mm_count: The number of references to &struct mm_struct - * (@mm_users count as 1). - * - * Use mmgrab()/mmdrop() to modify. When this drops to 0, the - * &struct mm_struct is freed. - */ - atomic_t mm_count; #ifdef CONFIG_SCHED_MM_CID /** * @pcpu_cid: Per-cpu current cid.