From patchwork Mon Dec 2 18:41:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13891173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0422E69E81 for ; Mon, 2 Dec 2024 18:42:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15F436B0088; Mon, 2 Dec 2024 13:42:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 10F606B008A; Mon, 2 Dec 2024 13:42:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF1AE6B008C; Mon, 2 Dec 2024 13:42:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D0ED96B0088 for ; Mon, 2 Dec 2024 13:42:11 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7A3FC1C5645 for ; Mon, 2 Dec 2024 18:42:11 +0000 (UTC) X-FDA: 82850888526.12.E31F735 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf27.hostedemail.com (Postfix) with ESMTP id 4FF1740016 for ; Mon, 2 Dec 2024 18:41:54 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hPo9j6it; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733164918; a=rsa-sha256; cv=none; b=QqdUaD+VsKceGw4SxXzq7SarlzgwaSn8fOUjM05IlJQGZxi5dM2O2QV6hM/jXbhygWw5kp PmfgOlCMGGAZFYLdgRiUk1yzy0TduJAWB5G+p0X3rWOeoR5Lp25Bin+gAtAkCK3XMouWzn 3V6dL2QblV9oF6ypPYnviylXCDcHjDM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hPo9j6it; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733164918; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Fs1ZKOzA8tmw1x2UXntYJTRjKjlbwcqsiLWIOG3kp64=; b=u5dzlDbA2ju5w+ZnaBxQYQeasusL8l8FUH0DhDfjB3IRgNxIWUPflO6UQH1LDf8BsaotEq 6V/vaWNCOtzqCkfZE8YkJigFk5dIZuspMEnbxt5oIKR744HmaRKEtgP+NI2b4PixrEmVoh xWgqTSYqzsiHynUvCjH+GB1ev8I8lbs= Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-7f4325168c8so2272580a12.1 for ; Mon, 02 Dec 2024 10:42:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733164928; x=1733769728; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=Fs1ZKOzA8tmw1x2UXntYJTRjKjlbwcqsiLWIOG3kp64=; b=hPo9j6itxkMc6DFALec9C3JnDo/mK2WT2HU93xzUkaYGHuh1xRIfJ1JP4pUpeu6GgH TL/dWMxinQMAiTc8gpDaWmf32Z1yZYRZnsxMEXwGjzTFPXfl51jOStNF1y2PuYTSOLvn gei/NLZZ/WmxOwuJhtJs32mhgOZtFlOidCYN5v3HpdXu6uU6lYTwez89CxeW49YNlFyN 1LagCdF76Kba0c9DmYDnzzqeSNekzC8h3rb40Fitj4kdRC9cpRabGD9c3o9gjOSEqyhQ 01zdKCaS+LT9OSBgJ4vcizrQcDi7QTnqIbpa9oCV381ki/LPP19ctx9mDdGHvaZ6Xhlr H1YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733164928; x=1733769728; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Fs1ZKOzA8tmw1x2UXntYJTRjKjlbwcqsiLWIOG3kp64=; b=w+Fhjxej7VuJNIxRh9jotC5V1l8E2T08CQEQnooGXUrNoESyq2/2M3g61cIucfzXDu Eetw8f1pU0qNyApo0HxpBFCVDFyuwm0LDIjFDcw7tebpXAm/w4FJQCy8DSL4+pNl9XJ7 vU6+yGuqLEzW2oaNeRt3MkO8bmOw3t3bnq43TZRXdUL86GHl7RBrcdyrSAan9C9aG8I7 lmJT40Ah4dOIpJFw4KhHcAXPbZ7Y/V1XT4D3NLkm8xv/wzO4jog7GK3ybV6PS7Xth7bE D+5HrQf93WVN+dQssQxXKtvj69GgQvmZh1Zh21JbWQT4VV66CsWyk6OBzlYQop5i2k/Y P8ZA== X-Gm-Message-State: AOJu0YyDrZJvsP7vqM0CYzNvqrGt+EN30LGjkITniXJZKr98wxYn26bD 2c3SDkWmwoHqtBsnQLbz+npDKGlj6clhi78pSnW98bF1/WYXDdyou/06jPZS/JlLWg== X-Gm-Gg: ASbGncvn4jpHM4RpmAwpt6i+vQWsLSmAy6EvSisUteHIcRrDs3HBFJwa1MqwfNaIyX+ VLBQjMM35mS5OXjmIazPUUg0WZLEkQK2LSnTQvAghk1ITC0jLBDQp84SwyJb7Ra/LUKcpiavHz9 7xNYC0Jg+l9F2AI0hbnlDDR3Pi60Wstjet775y7s5O3cJpwwx5JMT228pRiTRmpcARkDdcmYL3R WXdcDMeePn9zXZugcA+w0nm2Jc29+Vp7XO+cU69/T9jbX2Iu2sAvZFbdUFnN9iL5yXp4/g= X-Google-Smtp-Source: AGHT+IHkmfGLXZBCKnLmsARIDeEIAne9TONGGqSftyDIQbDiYw0C3eN18Aa5G9nvW+L9pfDtYnWufg== X-Received: by 2002:a05:6a20:cd8f:b0:1dc:3023:bd97 with SMTP id adf61e73a8af0-1e0e0b9b301mr36331829637.41.1733164927851; Mon, 02 Dec 2024 10:42:07 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.120.120]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-725417fb7d5sm8849781b3a.94.2024.12.02.10.42.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 02 Dec 2024 10:42:07 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 0/4] mm/swap_cgroup: remove global swap cgroup lock Date: Tue, 3 Dec 2024 02:41:50 +0800 Message-ID: <20241202184154.19321-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: urgq8h6zordkidt6d6fbbiikc4ctgifd X-Rspam-User: X-Rspamd-Queue-Id: 4FF1740016 X-Rspamd-Server: rspam08 X-HE-Tag: 1733164914-744310 X-HE-Meta: U2FsdGVkX1/WaYTCb74f3iIe4D1okXzqAi5jYowlhrp8D5O3gh8q/w4ifHmFCE3i/vD7GIo+X3kFwWliUBMMfgXeYqXAV+YpIJwMoWNWmRF5nvKNAFoMuOFoOqprT/LKd6MK0uSBL2240rQOq+TlJYq6qNjgGFJ14mW4menYrBVHkUu8V03p8Jw44peWM1wmt3Qu88IUulE66vFDxp34ovqHuhjWAe3ScxoQI34GrV3lFyTU29qy5i99mmqV8WeuW6MNUsCJLJR9K+QYutAVFGDxpYk9zSuMtX/gIZ0hsZvWHGgrxs730Utz8phERRtJmXqMMz0tHkYC2C5R0K0Hp995VCPJF/7IV95ngVwxkn5foFUrlSPOvOeZEWb7PZf8FkweDOhwQoBF2p4ywl2/7YIzHqPs0DhlamADIqKQuFPYNj7lATANWr8SE6b7xLBf4W6hWNpwvRBzA9yOBAvAXOQKpiGQ+6LcIUW3KZZ3bWgxM2APTtyUx4cIQAQJOrtzlL+lIKxPSB5fQ3BlSKVrSd7bzhy1lmwiMish9JZ3X9n6Vcbofwd8hiMECAyvfYEQvtD8gMWIktMMlg5WNuBGHE4Jx4jmRAtPihCnJ4QwvGlJALeVmojZAeo8uIDsTOD3bWoNdHi8VHOW1wKuaCMjTLMfZ4W8dav8LVWNJXUgeUBA0bG2cAul4nhINuYEN+83bNTPpmSlsBgNqkoY62v1rvAdUccJQI+Ha1/YiryFUPQd+JII70kyuakAnlmVvVGUwZTeGQ4lqK/ao7yq/PeFGXXKG3z3IWsM7+enqhb7apKhzHoiF3PMT3XoFynHuK2Iqn8LdZtIZkXtz4/leBKRmlIx3rJjkdORLlrVfIjuWmcvawYUa1ZxR1ydtumfXYPEIlyXF9wmjCXyX2hN7Ot7kWwj0lOaZP/HnbA1zkDDVNA9GWIRqNaQGSnk8wgCrgA9T90T1rkbUO2xkuRWesw fpJ0wNlt v7XVzIOJr4k0RK2kTdYUxGeuacaPHmEJqmyK8NOljvliW2JESoCcRULHlQPZqDP+UYQ++BFNB/eHS3v7D2oYc7e/PHKhcv7Tt0IKwXBSByhes4g9vOVxtpji4es5nFNsLh6nh/IAgmBRZFr04TNmERsRbfOa97K9YuVdq2PlY4J7jY5rCApjGZfJXQVsrJPayZgwrqZj7zSwCwS7iYq5/dJQN7XRP16NF3nMmqlxX2LHuBX3NL0SbL/sGu1XHc+Qx1Iz/Ox+zaimV/Rxz2cF5f+zF6u51OAoN0T+ZOhPdp/H/s80kAG4sxBcO7jNTY4KYUptW/wZexReQ/XTEhwimGvk1+/RMjP8CiUvC5oSkbTVHctnFNnkiLRcQg7W1mNDssA3dqa6s2dpUFslCo8gHVqdyISY3wL2we6XAe8feaWTSsQ6eCVM5ClW+RirPJ68NkvDAUXI0EpRs7c14+wep/I1ujJXcLnA/ERdRKsK+lVNcXFexabw5Rzg/RK5A+y9yTqGmvsWX7qQYIt7EbVZPkf9/Txj9LmoQtQeJwUvVcLAjeT7G8cCEk6nqtzV2PAKuwwceAQJxXBZxIS/CFVv4als4zJX1UTNI+jBLIsWyb4kHuTs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000591, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song This series removes the global swap cgroup lock. The critical section of this lock is minimal but it's still a bottle neck for mass parallel swap workloads. Improvement of this series is more significant after the si lock rework: https://lore.kernel.org/linux-mm/20241022192451.38138-1-ryncsn@gmail.com/ But this series works very well on its own. Testing using 64G brd and build with build kernel with make -j96 in 1.5G memory cgroup using 4k folios showed below improvement (10 test run): Before this series: Sys time: 10730.08 (stdev 49.030728) Real time: 171.03 (stdev 0.850355) After this series: Sys time: 9612.24 (stdev 66.310789), -10.42% Real time: 159.78 (stdev 0.577193), -6.57% With 64k folios and 2G memcg: Before this series: Sys time: 7626.77 (stdev 43.545517) Real time: 136.22 (stdev 1.265544) After this series: Sys time: 6936.03 (stdev 39.996280), -9.06% Real time: 129.65 (stdev 0.880039), -4.82% Sequential swapout of 8G 4k zero folios (24 test run): Before this series: 5461409.12 us (stdev 183957.827084) After this commit: 5420447.26 us (stdev 196419.240317) Kairui Song (4): mm, memcontrol: avoid duplicated memcg enable check mm/swap_cgroup: remove swap_cgroup_cmpxchg mm/swap_cgroup: simplify swap cgroup definitions mm, swap_cgroup: remove global swap cgroup lock include/linux/swap_cgroup.h | 2 - mm/memcontrol.c | 2 +- mm/swap_cgroup.c | 110 ++++++++++++++++-------------------- 3 files changed, 51 insertions(+), 63 deletions(-)