From patchwork Wed Dec 18 11:46:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13913537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83613E77187 for ; Wed, 18 Dec 2024 11:46:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDD166B0082; Wed, 18 Dec 2024 06:46:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C8D156B0083; Wed, 18 Dec 2024 06:46:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B545A6B0085; Wed, 18 Dec 2024 06:46:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 96C2D6B0082 for ; Wed, 18 Dec 2024 06:46:54 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1B76F120D1D for ; Wed, 18 Dec 2024 11:46:54 +0000 (UTC) X-FDA: 82907901930.13.CF92A18 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf02.hostedemail.com (Postfix) with ESMTP id A210680012 for ; Wed, 18 Dec 2024 11:45:51 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SoPgvOSj; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734522382; a=rsa-sha256; cv=none; b=xS3I3LXQXeaqHtWY8JZSDVO/EACtLBQxuoCCaQDbtDcVWBzW9cxoaTc3IBKE6yRVPkDmdX ssrLz32Ug6aeXrkOVaEVbMd79cbQm93nRQ33QHaLTuZ+0eErgemSKuifqXLlljtqA9i5d1 oj+eIPPb1LrFsSq1IMlBgI4kr6RNmTA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SoPgvOSj; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734522382; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=2S5Kxb9dEO4DhOTZgE0AX1LX3fb+5xVfddZPXU3uHZ0=; b=UNNJQqDCfGxrabbB365c+gGxnMoXrSWa54yMegkGPFoVQfak1NYPBiSvbm4K1qAetNaK/U Q8kk2FgmvhJuir10RH8ye7s3FL+VVZfng+D44pMetBuISfr0wfJA16MfMl/2lHhJ3LyVY1 /Fftijcnx5QW21f1Uwle3QK5v3YP/10= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21649a7bcdcso59633745ad.1 for ; Wed, 18 Dec 2024 03:46:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522410; x=1735127210; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=2S5Kxb9dEO4DhOTZgE0AX1LX3fb+5xVfddZPXU3uHZ0=; b=SoPgvOSjGCgehnXYdYK738XwTqUOlqw3GEJesfYF3MG/1um7iKUeEB1rABhJSEWcBm Ch87mSiNTNqdkedawWo+v2A/VvtaNoljD4PG4TegzsNSftHei7TL6zNr3lsC+gN0Fq3w 5/rTFlLSJyapZdonuJVPbyd50z9cFCASRbcuG0nSQ820SCICXw1I3ad7dHBipEy5+r6D AtXb+5kcIEbDTcZbujYlgX88ppOPaDhuTmva85tC1Or+068ysjECTBSVO8HXFvapAlMx wPBNBJnBeS0jNiPyyqQzby5meGWHSmQL6QzlUJfWOeDovEKHUEqauQd8kXAKShkEuNu8 oLhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522410; x=1735127210; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2S5Kxb9dEO4DhOTZgE0AX1LX3fb+5xVfddZPXU3uHZ0=; b=bjxW2F/W+SG7uJRR4ZoGo8WuU+kt3meUO4kj0wqmInkIKGHyz3d3ePmDKf7rd2oKa5 55y++/8ZP57TeO1LlFEe0uDXZyntLj/Fmxr8usM92X8RjTZXng/T1PkuX2RoG12dgMvX 5xdynnn8DVfGT4Q+nahSKt1dtbBPFF44Ny9WuctInpVPmIbe5B5ZYS9fMZCl4cot0pgD fNFLysfBmKGRDxA2Vm/RXazdccczHTsdWrxCUBLNag4m8KVzGjaLCa47NzqwAsymNQmd fbo03tQSvTe1uKzUwMb82jJnkj3GEYsAl+x4+oTrvteKujG++dHgBiVT+plkeNqM51Ir gPrg== X-Gm-Message-State: AOJu0YzR6KpTmgOPkNFq3fD1f2e8awRlvcoH8PjkgGilXViFbXwIdrBq 0E7GrDAFcuQ9ZAf3NFSkmH9kikp+CurqEfqbeXQ9tHkDeELJt5NgxQHsybrI X-Gm-Gg: ASbGnctUO0cMtlExYCfaCCl15k6q8YCir3kdoZpGnO10kJCGx9Qj9AKuC1PKqGSyPGr HGwa+HAvOki9BRsknnAfQDMtPRFDvO13fIDdGVT3MnDHOCeLAPKPQ139WJlQb1jC1WrZrD5+dL5 DrpBxp+zjYlMnz4vwZ1Xa+jGH042OgrwW03pDHVQyZzhU94tQL///ucdMOspVhav8EhuE0Mjf6a YPzotIDGr8B/WBu5bYUyI3ayCq5DzGte7gp6KUCDWoBrKRpO9iYLtPRRY6lJrX9C8jlbLqRO5Kd 9FcMCY25EK8= X-Google-Smtp-Source: AGHT+IHQ5mXUalnkwNNNwPSIvcAgnZ/UFKQb/CMbswOqy1c8AA2eFToGxdJ8k9lpU9VRJP3rI7w8qA== X-Received: by 2002:a17:903:22c8:b0:215:af12:b61a with SMTP id d9443c01a7336-218d726c0cfmr28120495ad.50.1734522410444; Wed, 18 Dec 2024 03:46:50 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.46.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:46:49 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 0/4] mm/swap_cgroup: remove global swap cgroup lock Date: Wed, 18 Dec 2024 19:46:29 +0800 Message-ID: <20241218114633.85196-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A210680012 X-Stat-Signature: 8urcpmroqondftyrh71s4tpqbxcck1u4 X-Rspam-User: X-HE-Tag: 1734522351-966785 X-HE-Meta: U2FsdGVkX191nqio8WN+zKsvg1OY7HgJcHzHZD5Tbz3Fs+KPyz4NgA+urppKR4eSHrOcMzPukbaQluCRRCtsEkQcWMZb6gjCQaK7czPB9f5laDzR5FSgZrR8gZWjfF33FG5Jgo2nVLGcx5M91pwflvmiHCneDCWkoCO0cpTDdWyznLcpWlk7n1sgiiDcO/+g8v7OMDyRC0i7kRfR7VDFrc+gE8G/xiuncmTlAdaX9Xqc7Iqx+9ltuVlwEFGrjz0VX8o8uS/RHZ+U70TMGluc8J92wPD+PY8eMdDZ/JxpWkorGjH1kNpULz5LRljHCFkQWCsU0qPjrSPPUAkT37NIEWAJy4gObkuMTg/sySErlKivGcvJu504ANlqE8nMLVzzhK04pV+8QIkwL2hC7TqM9+rlhC5anKJk10svy5+YV/84dqQ2uuNjNVjoyl07ex+ZdoAMHE4Wg0I5vjZ/eda2T3IWZa9O7lsaemEDspzyEseNhe/gizYd1LNaUHC5o7RgIu/a8rGdGoRWZuXr+Hwaxj9GB/7FCDutx/u5rqfgIqEbY4fk2BaJcEXC6PlvhfkybdU8TzvEvHux6990F3JnFi+jusn0pJ9QRjLFN2XRFAtncgfVYGYiGr9hZkhcnKKkMIxrHiw1B+Ds8I7CHOTvPJ0XFgowsSXV1lTCAdEVKirP8XkGiMcda4IGyDLtZ1EyFAyLO/B0N8dSQhc0TGqDYdwea2IyTaY0mFwA8z/ojRoZTr4clpViHidMCrQKQCpFowkyYS8xKa4fbPdky35/Y9etAi00jZOmT3MtcEdUQ6badQ3S5T4adWTOmw2jVu0t2J86iH+Ex+6c7ASbf/isjqmyuf1Ntyc1eGk+G/AV8Nxyy1zV0+rEBOax6hdAQZ3BblySngEBtEHkkl2lKIauBk4QIVJkmOabIoSokWxAroe25l/UX3ylGm2kszHayNk7MdsWdgWIReL813W0+Nw IQlj5Qky QQ8efJh97Xi8Zkp3Py1zz+2ZB/1L9YpOa4bfM9nHYJKDGsCUwVhMWbgOl4QbRpx1WHGvN8eqBD3vCh4A87kfwoOV1sIw/Ony4OJ6jRkofZpNR/3u+B8fgnbNZwr9CK3pfTFeC81bKccCwW44Qq6XgRxDVTDG7DfJ1+nsnEia4gSl1FngOX2+HZ4E0x2kFVIvaA9W5oMMPUYpZ1tSVVkWsbU7b3PxF8AP6Q3G1LvvZ6bL4oSGpit0HCn/7oJ2z5NgWofnMmDe8wKl8X2y1xdxWww3HQbCTlyq5mXKpX1vfwX8yZuxOoTiJIZN3DkLWdZkUU/ZslXqiCMzYmQIMhPHFOZc3RH4hfJ4jSaiOJAWlCe4aWhNpgzvfCVFblT7RgoI92t9+pd5MEtOsGRnoZZlFY/+O6f60x7Ra3qpzrw3Tu6yQdWTIh+fTapRQ9tfqVlS6GkodRH09HPagsWwTF8rM2nq1hEwiKlvPFDwxWUVBBgKTM7W9mXSmNYA7KQSeeGRRhrDou1+p067uDCmpDrxSShiEwIe5jREZM185nmJ2WQIimCQ7AVs09ZH1/gktGQ6HDXMWOxJsxD6FyyCaV8o3ISueFeAnoDuQZS3Tir+SkCfAwUUsebIlsT04PQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000139, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song This series removes the global swap cgroup lock. The critical section of this lock is very short but it's still a bottle neck for mass parallel swap workloads. Up to 10% performance gain for tmpfs build kernel test on a 48c96t system under memory pressure, and no regression for other cases: V2: https://lore.kernel.org/linux-mm/20241210092805.87281-1-ryncsn@gmail.com/ Updates since V2: - Micro optimization for bit operations in patch 3 [Chris Li] - Improve BUILD_BUG_ON to cover potential arch corner cases [Chris Li] - Introduce patch 4, make the swap_cgroup tracking code more robust [Chris Li] V1: https://lore.kernel.org/linux-mm/20241202184154.19321-1-ryncsn@gmail.com/ Updates since V1: - Collect Review and Ack. - Use bit shift instead of a mixed usage of short and atomic for emulating 2 byte xchg [Chris Li] - Merge patch 3 into patch 4 for simplicity [Roman Gushchin]. - Drop call of mem_cgroup_disabled instead in patch 1, also fix bot build error [Yosry Ahmed] - Wrap the access of the atomic_t map with helpers properly, so the emulation can be dropped to use native 2 byte xchg once available. Kairui Song (4): mm, memcontrol: avoid duplicated memcg enable check mm/swap_cgroup: remove swap_cgroup_cmpxchg mm/swap_cgroup: remove global swap cgroup lock mm/swap_cgroup: decouple swap cgroup recording and clearing include/linux/swap_cgroup.h | 14 ++-- mm/memcontrol.c | 15 ++-- mm/swap_cgroup.c | 148 +++++++++++++++++++----------------- 3 files changed, 93 insertions(+), 84 deletions(-)