From patchwork Tue Dec 10 09:28:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13901073 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EABDE7717F for ; Tue, 10 Dec 2024 09:29:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDA6F6B014F; Tue, 10 Dec 2024 04:29:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D5F4B6B0151; Tue, 10 Dec 2024 04:29:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB17C6B0152; Tue, 10 Dec 2024 04:29:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 947076B014F for ; Tue, 10 Dec 2024 04:29:06 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3A4F71604A3 for ; Tue, 10 Dec 2024 09:29:06 +0000 (UTC) X-FDA: 82878525072.16.A3B9664 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf21.hostedemail.com (Postfix) with ESMTP id 002F31C0002 for ; Tue, 10 Dec 2024 09:28:22 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="DdCU/Dul"; spf=pass (imf21.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733822934; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XXPbw4CmFZlsHQ1dqNkun8nNSKspNNgSNNzQzzqG5eQ=; b=CE/r4JCQVVcSwmpAR3Flx9fwSVatYnxSOpof1SCF3qjjdNB7GIlB6dzaPKKC0ICmQykYP/ IZm9Ml1FNTkRxr3O3GG50n/6PRQoe5+WnNZjBCy/x9/CA0NeF44/PuWOxFn6eGbwJqQg6L InCSCsipiZ5CK+BAxEIUvc/tCOVUOew= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="DdCU/Dul"; spf=pass (imf21.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733822934; a=rsa-sha256; cv=none; b=mT0wDXre6IZXYPVV45ios0RaeOXQ8F9SoZjq2eBll8KypbPqOs1f939jN5QN2vyX9jh4u8 VRzdEWTvBLS3vdOfDszksvhXCRb099FXbFAOi9OugcO9+inyGkjGli8QVOShYzmj81eZ18 G/n8m0SBgEsP2e9F6+ZwAea4j9PGmTQ= Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7fc99fc2b16so3946242a12.3 for ; Tue, 10 Dec 2024 01:29:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733822942; x=1734427742; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=XXPbw4CmFZlsHQ1dqNkun8nNSKspNNgSNNzQzzqG5eQ=; b=DdCU/DulNyDZSdvdhm2h+2WZ7AsLW4H8ZDaWo05UvhVCmIBSZWRU3BEY5c9HhaU0HR jzthYjWwTKh8FW4WKR0osW7wxEeNojWM1WPaJBbobmXsOcyEIDTeyI1jVyK1aHJRXf++ 3nS8N9JFH0s21/u5lteo1jVIDnVHFGNwgMF8XGy48tTiWQiAyY5EK0k0YrPDXYmVX24x S3H7dTWfEZb60vgjOIVPyhfsZtQjbg9B6ks2/Wb6Nda6YfLYrRCcpiPcRBBdctxZ/ZRA 8FqZHsLn2VLUrotx0J/KJWdfTxX2HGbp/Be1zEDhhi6J7zCM+FLzMPuH9X2nIUG6i8FR q+Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733822942; x=1734427742; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=XXPbw4CmFZlsHQ1dqNkun8nNSKspNNgSNNzQzzqG5eQ=; b=vohyydijEmfnimIZqtxJYwCGawiGNrKcRdOlrctj4jTwG//7mn42wkXrJTzzaVNhzi X8BMienQ5V1occkDZkWd2w/y/DiAOvYvstKu2ZjXdHmUbZAwH4UemsqdrzshlriKX/rI SvNawEFlTLc1gVc9bhgeW3rYIGTZ5uQTy2QPZmdzXiv9+xj2DxfayFYiUyInhSttI7AG WdnH7hZnskQnyajODZaPNiTO9iJl4cuvbvHKjWddmkmX22m7gzKQsXCk94pQ7E0pI/XF Yq9lhg4XHc3jsZFRjdXoiu3cA3/cKASJxaWtsycSRwb+z84ZYyS38InrV2M0b2h27NtV oA6g== X-Gm-Message-State: AOJu0YwA9As8oBal2ZRrw3ltS8I2Ruz74K/ts5ljPnnJFxs8o46K5rbc RILv0sm5REC08uz9olyE7/sbC66Z0q2J3irGXXGe1/SGIiLPGlvAndce9DeOi+Y= X-Gm-Gg: ASbGncsm2GFJQiwSYrNmiyP+xotJ2+voh0gEQbLlpHBIGK8BGCkOzw1arpGTDqgF1pa 5wskZmu064VgFJUoTzbWXMMucBl9QGJP2JuJiDxfz6q7GFa2jYMqWzX2hZubCuyOQy4F0/uH2Vd jpFuw4FUywbCC25uNhVns+NCUCFRM29y7ge7UDqOdgWJAWFLmatQ2jRvYt3aMX54iY/vv385ACg SLxzGGAQMY74sdlkvnWza+doh037uYp2I6ZaNnPuEnsLM1LXpqEFa2YFARzys2k96B8IUtRl3D6 3AELXW1u X-Google-Smtp-Source: AGHT+IHMU7qNZnjZ/bkeJFqL1owhrOCa0EJiXmL8JTWuT/MSJlktqTs5oyZfJOCFQItybeRNzpjntg== X-Received: by 2002:a17:90b:5184:b0:2ea:4c8d:c7a2 with SMTP id 98e67ed59e1d1-2ef6ab0d5b6mr24092612a91.24.1733822942554; Tue, 10 Dec 2024 01:29:02 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd1568f26asm8750095a12.9.2024.12.10.01.28.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 10 Dec 2024 01:29:01 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 1/3] mm, memcontrol: avoid duplicated memcg enable check Date: Tue, 10 Dec 2024 17:28:03 +0800 Message-ID: <20241210092805.87281-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241210092805.87281-1-ryncsn@gmail.com> References: <20241210092805.87281-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 002F31C0002 X-Stat-Signature: uhpo7q4oy8iekdc1uzuh8rt4w8ff5uzd X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733822902-4785 X-HE-Meta: U2FsdGVkX18cIhNEg2jbnmrlucKMtqaOUo99E6N3WdFnQryfqH/YQOyQ/fDfmu7y9xuBKCrBfF9Sx9x2NpVsWxmoJmbSj0zH9yOb85zEIU/hwQjrNLtWaVoT82uI6FNh+8cRcpKlNdhUSN0XFrftes1sN5YDNLhX90iDEJuPpaIAcc6dlUG+ykvD/Ll9Rxnbf2FUfldB70fmuBfr9qg6mJAgpZXygKUYLBSzeaVMiglQe5T1FfOg4pylhNGcyKE4tkbvYLy9+nCLBM3XG8HWKv5SWvExiuxKm1/5REYGF4wRmQeKlOKe2Cx4YOhhft9DRrNU2sTzsQi93zoFzoPw5n+9zFxTz6+2atvZ1y4rynmU0COULlK6KkCSyEom9eMd5fPsu2n5b/paJqJ5WQ3I5YOIbShWs4hw0zkbNYW4fBqInk78d2uKeyzl6hABtyoCQrLriQprmbFtjIw8M7zwPNitg9YCLiV2AJ3+qwOllj/7VbEDEalMJxu7FibYrS8TUSwc92O3DmfoHmQCo1xKx3bnTij3k/FTnbTEVfcW9BHQYHf7+a9C2/Vcuu+V5btPmlU8N+GJygUm9XVcB+2aDVQ0gwsygQPTFtWDxcnPrjYFx+8VqnyDMgD5fGSCKDuzu6AMDgrhPIieA8dPXKpFhaDhpu7PYecjTwzmU7i2fscuOgPscxVBIjllHnmUCFxXjp7wldaWB/vj1TAPiTIjGkzHkF+D8nqschxHCmj/6TT9uOy6p8yTNoRH7TwyQ+3SHneFHuDB3/qbwZQIn0TjrouOM4dANasGvTekUqInNK6KR3DEWR0kTgQdm2IZ8z3EYuVRVU/51YAYM6HoHL81Xm3OeNdirCn5Lie2/ew1oIgdWvu6/vZrVQ7KwKpX+XWu0lMS3YSd8hAI/84Qt8XJTZT+h4XOLMLmrmyI8JyC+JZTh1Ba0PPdYSLt/CArFxmVFdsKiygULdgP7BUvlNi xp+yYRvb GUxRvs/WCsd0bv5bJQ9cxf+d3JzSGs6baUjHRlI3SELLbcFEVG663gK0EZHz/ZLNeRQph4zVhasPq+HOOy7xEGRTa1x31tCs59WnGY/0kGBZz+WD4DucrpJ/0JZK5iyjathsAn7ekgzxGOziSwNFtYGISsAyB3tSLX7x3C7a5avTteyqkA5VXdxeWXhKy08Vryz+vBCd1JkVtZgQh1sgXuYqx04Wgj4yEpr1+lL1wAI29DkpYMZxTQX/JD3bqTBgzdtMz5ZvkVjczanYW5t2JZ7Uxts50DayeRSXD0ePT7Ff5s7s4pfgJ6ATfOb1A5GLPrepqzizUSZaRhGPFkGm5qbV+WxkAPhDaeFNqmChBHvT6nplY8JGk7mSkYD52w3jSiLiWUdt+wBTBT8H9k2cawgi3s4/hRYvpoNo90pJMYPh4pcio/hTDrzDGLgq/jwGvsC1wEJX2E8CyUMsis65aYcb3lbrznn2InMH/LMPpHiIAk8Sl7T+W0VU9CbBfgGT55/TnKX43QfbTkj5geNealyiOWs2EL9kRASTRb+ApjaqwwOA6fjBLcAYU3YYuYGr6IKnS X-Bogosity: Ham, tests=bogofilter, spamicity=0.380458, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song mem_cgroup_uncharge_swap() includes a mem_cgroup_disabled() check, so the caller doesn't need to check that. Signed-off-by: Kairui Song Reviewed-by: Yosry Ahmed Reviewed-by: Roman Gushchin Acked-by: Shakeel Butt Acked-by: Chris Li --- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..79900a486ed1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4609,7 +4609,7 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry, unsigned int nr_pages) * correspond 1:1 to page and swap slot lifetimes: we charge the * page to memory here, and uncharge swap when the slot is freed. */ - if (!mem_cgroup_disabled() && do_memsw_account()) { + if (do_memsw_account()) { /* * The swap entry might not get freed for a long time, * let's not wait for it. The page already received a From patchwork Tue Dec 10 09:28:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13901074 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8544DE7717F for ; Tue, 10 Dec 2024 09:29:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01B076B0151; Tue, 10 Dec 2024 04:29:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EE53B6B0153; Tue, 10 Dec 2024 04:29:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D60256B0154; Tue, 10 Dec 2024 04:29:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AB8116B0151 for ; Tue, 10 Dec 2024 04:29:10 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 193CBA06A2 for ; Tue, 10 Dec 2024 09:29:10 +0000 (UTC) X-FDA: 82878524652.03.BF6A63F Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf11.hostedemail.com (Postfix) with ESMTP id 5818440016 for ; Tue, 10 Dec 2024 09:28:48 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LW2t0fPs; spf=pass (imf11.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733822927; a=rsa-sha256; cv=none; b=46HB7rQKvfOavQoaqyiCQGAIadN/2petXpcWAdkn0yCWM65drQ7/p0pSJC6pu8mxbUe7+J LayZZ5uic7UARoobCiBHuHCBPZOlVK//tuPJX30O/XnyFy1wN4aW062SUZrwolfb2fwJ+P DdBKyd4876fOh8q6QMQd39+A/B20L68= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LW2t0fPs; spf=pass (imf11.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733822927; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BSCLGHYqLWfZbOCM52YJejtZzzMrW+w4kb+fWMx8dAg=; b=FkdlwdaID2dBKccdVr4aOvXZO6mM0YYiYW2P2cHgdJkyFDZAPB1UYhGgJzs5Rixb/MULPd DpWv1G03WemjEEWpEkqxvjYrb/DiGaNrbKNO6H+16ALKWqRSWfSN4shiUdcMI1mRx8baJ/ BKVKKi5eYWYovu4ihWKLoFfgqrIp8RY= Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-725ef0397aeso1658588b3a.2 for ; Tue, 10 Dec 2024 01:29:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733822946; x=1734427746; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=BSCLGHYqLWfZbOCM52YJejtZzzMrW+w4kb+fWMx8dAg=; b=LW2t0fPsVtIGBjeAaaSnnp3aqRmGtVbbif0A5sx0sIR8Knf8cypKtFVOmf3N9BNV2g SSujuWgqg94lAH5tRuChQxb15zIS0O/e71lYcyJCRW/zKPVF/xStgavac6Dizdd5II0P SzHlbf/dAZaGXX7lbMZeK/j8nRwDl1wLGkwZLQX93QNu9ybpnMJZqVxikLvqdO5lgsaB DZbazysTZ7/iOXJhTa8fTGeWNHhyn540MdAYIrktHzjLgmuntqAVJ5DSUfXk5xKhAyBo aXeYjO/8zkKfF6PzOkvivDiIqSP7zqLGVymhqrfih2i1CMaRRk6fxDpkf9ChMcEb9TIC jPvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733822946; x=1734427746; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=BSCLGHYqLWfZbOCM52YJejtZzzMrW+w4kb+fWMx8dAg=; b=OMZElTcFJ+/v292JKIhpqvURBqBN8N4IetbLygO/1fgJrp9iFf6UG4RCc6w54anpsb Lh9gOvQ1x+5YhzCoVZ7a+pbP6iFyHcoycu1O2QdhtTxU8Z/YVkXkZp/qTMNRnddvzFJb wj7PjemwFDw8NOrHbBs3kW3tjaFZ16sX3JE0aR38nxL7Kred8AZkYwsqYG/4TslDw6WW Bjj42IcsfCyQiZ5jQIGbS5oHPatc5ws2DRUr3THaDlJjS8AAecI3D7ZxP+Nb/qJWPs2T Kg0ScFS7Ei9o3a3sv5PRm3s8vFk4YDIJ1KoZpJz7G67qcLjqNm9i9Jt1vGsfBa5XPKrq /qrw== X-Gm-Message-State: AOJu0YwEAuwO5RU1qu9uDgaUb5Y1Diz11FUycGGXgGvXo6ja7/Vzo52Y yXU5LEHN0ZP23mh6N7/ugdpSHxrPv6Ete5t/gT+NDNx157hn/NTjsrtvX+Flk+M= X-Gm-Gg: ASbGncvJJbM/c44DFrrNOH3NNaUxkW/d5P9dDLbJlojiTqwPAxmuYRvTuofCFm0Yat3 OFp6YcYaQr5/MXoA5k49PuGpnoK5FQG+jhSu+ouK4oVE9D7YpxjVO4TegVVy0zKKsSiKiURMI59 mRlYbuG17qq2tuBKrE0lJfss9ydrNPO/8iSN7t+ZMODqSLMIgd41tzT8ZND83j7c8n+KH7wYTqi rdp4QT9IJ9n5JF6yTweFEl1nC7rRvo5GzlJPLVJSo0ZB7CoTfJ93VHzCO5TwpKtA9yQGkjwinAI RYas+hGc X-Google-Smtp-Source: AGHT+IHV3lqRAPpmlMOBZuYBAiL3Q5xtUsfeTC5YD+b1WYrzaAKgKUOsPSV0U0jw1or5OftGfUdPEw== X-Received: by 2002:a05:6a21:78c:b0:1db:ed8a:a607 with SMTP id adf61e73a8af0-1e1870bd7a6mr23599578637.11.1733822946475; Tue, 10 Dec 2024 01:29:06 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd1568f26asm8750095a12.9.2024.12.10.01.29.02 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 10 Dec 2024 01:29:05 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 2/3] mm/swap_cgroup: remove swap_cgroup_cmpxchg Date: Tue, 10 Dec 2024 17:28:04 +0800 Message-ID: <20241210092805.87281-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241210092805.87281-1-ryncsn@gmail.com> References: <20241210092805.87281-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 5818440016 X-Stat-Signature: opmwjhmb7w7rwecteondejcgndhswrb6 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733822928-548424 X-HE-Meta: U2FsdGVkX1/iwCQ4xMRbM6+rpjO1at8UaqWHhLziBmnJ9YJfgvSpZzbp4KbVWCUeAUwrgHoY14C1bwSLIG8gYQfg0GDrIzARMQEf/Y6vglxpAKrjSYlZdTIWfVv+BrwngswIA+hCpR+eRiVBNhMGP1J0NVep8PZEVoT8Uf4UqIAFVBsCvkhOxLpTm7/hqSGDj+vlCL1UWzPSoN4JIXu+xyjVrpelmUwgNEG9w17pVXBM8tlfr/jfZSvJ3nYtmLCfdwazk7tv4Q8hCCRF6WQIdR4KcvS3W87muJIR/AYJOHUSRMbmPowZPrkn1DSb71rS2BE5oajI7kqxo78/05wq44oycvw+Lh1/XNIA2BCYQ2w12wO+Uat0xpzLBIVjsZMEK5OXZEkEuICDQZYZOmRM/74oe/FqGO5hfaGIxIrtVA+wJ+wh7LLg36MOUUtNlWiKrq767shXP+SOLubYH8JA7tYEyS7xkyO45zJpukwYJmNZsh4qND7ieTX6VXzlBfCpks0VXvMDR5T1vqoyRmMBO6/Nlx4YXpu1DBI6+Ukd68W+AAOZgiQ7Xj0ql4hEgNbuMoGNZK6ULe1buwu70eKet2eujoFTBxtHVQ+G35s6qQidJ8nhYEdxxmFbPON08wB0AsLxj9aYaRJ0pjykT7dZuy6qSCvTFz5gPjCiC/rY/YCiDngUTzvB0/40OmCjEjnkhr8bit1vWwQoG5Sd/OlwV4He8FSLk+9NmeA5Ln4HaYOG0qk9Bbrx4wRuyFp3jVZY5aItZgK80faJY/+cVvLOtjPwlLPXhJgW3x4+2ASwRn1d9sevPWFNCruruZoyYPblRmbFM9Eq2/0NnTbFfmD+/EATfu+Ih/e6f5cQqr41VVMBk7P1qRfAfSajd0mhnIG+5Lw+vOXtHnwaYAbEONH38mR0HUCzah54Ae5a5iwkaQYr6XCM1+kx+aXJCZ7rqiIfkHXu4TiwMebK9WBdKgM 1FmyWrzW 51k7mCWbh+BNYR13ypBP0BvOE0Wzs4DzHyHhdH5CfPvq7Z0HU3V8dAblhqsJouvrgmxXBMOJ/w6VbqrB/54shmYqOuoGAsgQ8GgnFrMDT4T0cwBxI0sTKAJa/0ca8ENioMNYiR7wbT43wwkqIzbmlpdqdlZyeXi85N0My95+bLcgkH9tsu8keWkYiwYnKQAB4QWOQvDVmuGRYdGaager4YgM8yUUPEIfyavtgf+NKr0Wvx1R9A6Bi6/OD0l4UFfp4R0clhDRAl4medxv39b+2q6vd/bY1zF6xQurt6VknAEMBP5tZazdJ1Xhi4rqDrbTiuWezSRYT/fcAqO7hAPNZTqoUjibIpD0GqMZqQJ/0GrjxD8kfRUAC4Mgp4cHkOKOj0aSsr6I0QFv3Whp/uJ/byzHVgdsx2HPFC6kLnzUS1iqRXGEjuK/DIbaq9/JHA4Bdf3WvHUx5/+srJ1M0KOgMHfa5R8K58VtxdS0T5zifY5LV/DGuqV1xIhsKyxqJp4hHfAXIjDYlzo2/w1S4PMuqOqvBJFXDOluXeSaTwdhLJGCVBK3iwUH1nrxIcSDOzGnLFdajnz5Lhxd54yyEzDMIGjQUXA/L7WPYc3K2jYOEtcGit2dLlkk5pOXRoA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000143, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song This function is never used after commit 6b611388b626 ("memcg-v1: remove charge move code"). Signed-off-by: Kairui Song Reviewed-by: Yosry Ahmed Reviewed-by: Roman Gushchin Acked-by: Shakeel Butt Acked-by: Chris Li --- include/linux/swap_cgroup.h | 2 -- mm/swap_cgroup.c | 29 ----------------------------- 2 files changed, 31 deletions(-) diff --git a/include/linux/swap_cgroup.h b/include/linux/swap_cgroup.h index ae73a87775b3..d521ad1c4164 100644 --- a/include/linux/swap_cgroup.h +++ b/include/linux/swap_cgroup.h @@ -6,8 +6,6 @@ #if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP) -extern unsigned short swap_cgroup_cmpxchg(swp_entry_t ent, - unsigned short old, unsigned short new); extern unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, unsigned int nr_ents); extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent); diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index f63d1aa072a1..1770b076f6b7 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -45,35 +45,6 @@ static struct swap_cgroup *lookup_swap_cgroup(swp_entry_t ent, return &ctrl->map[offset]; } -/** - * swap_cgroup_cmpxchg - cmpxchg mem_cgroup's id for this swp_entry. - * @ent: swap entry to be cmpxchged - * @old: old id - * @new: new id - * - * Returns old id at success, 0 at failure. - * (There is no mem_cgroup using 0 as its id) - */ -unsigned short swap_cgroup_cmpxchg(swp_entry_t ent, - unsigned short old, unsigned short new) -{ - struct swap_cgroup_ctrl *ctrl; - struct swap_cgroup *sc; - unsigned long flags; - unsigned short retval; - - sc = lookup_swap_cgroup(ent, &ctrl); - - spin_lock_irqsave(&ctrl->lock, flags); - retval = sc->id; - if (retval == old) - sc->id = new; - else - retval = 0; - spin_unlock_irqrestore(&ctrl->lock, flags); - return retval; -} - /** * swap_cgroup_record - record mem_cgroup for a set of swap entries * @ent: the first swap entry to be recorded into From patchwork Tue Dec 10 09:28:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13901075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DD71E77180 for ; Tue, 10 Dec 2024 09:29:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B6686B0153; Tue, 10 Dec 2024 04:29:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0669B6B0155; Tue, 10 Dec 2024 04:29:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD3A76B0156; Tue, 10 Dec 2024 04:29:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BCC866B0153 for ; Tue, 10 Dec 2024 04:29:14 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3D33B40209 for ; Tue, 10 Dec 2024 09:29:14 +0000 (UTC) X-FDA: 82878525156.26.079F4DA Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf14.hostedemail.com (Postfix) with ESMTP id 61976100004 for ; Tue, 10 Dec 2024 09:28:48 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iBxm4684; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733822942; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1vqDO82M+XCnuau+JNGqK2Zehr2Sd7WYRXd6APN34sY=; b=6KMB8bs2PpTZUKetsXESDyd2QKQRWqVqsnj+RM57if7B486NPhvJg06Yxnpmtm0cTX/u3i ISXXkuV6ht9xkNNW2ShdQ4vS/iKx0FvQexKBfd2NPJDWj+vn68zRUPPgrwS5RbcJgzhtV0 LZV+QqY8GMefAPH+siQmVDK8isLzjW8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733822942; a=rsa-sha256; cv=none; b=XTYMIUx0tpk25/EbzzXxlek1p3+kYcEenxOeL7IkpMfxhaNc+ba5YcBrvmp/S60epTO+P5 4E2nc7IulM74mgtgPLuMM9aOTMc5uWJ7z8/IbGv+9okvk5QmkqgpqDz15j4WcwKK+461W1 2Y+TBFoge/jJ4wUdM+AOmHLXG4mmcck= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iBxm4684; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-7fd35b301bdso1995901a12.2 for ; Tue, 10 Dec 2024 01:29:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733822951; x=1734427751; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=1vqDO82M+XCnuau+JNGqK2Zehr2Sd7WYRXd6APN34sY=; b=iBxm4684G8DSyrBnp1Swa3pJNq/Jdxaek1dz5GcMb4sZZosYVwDLkJxf1tC00ivFJC DSyUds5NQQgJxumiXdxzIy0o+KZd+27hmVtFsjtPFR+FdLSuY6VPiOAY02MXS6fyrFCW G9r/WrAR876lmoXhLa4SV4NYHb4GWBjwSbsceE47pvotlSnxIM6Ec3r+2mtrMB8Qibmh Z19a39nCaNnpPFHkj1rf+66YFHySkU2f29WHmRAEOTNOJfpcPrG8XypeFH/4L18Fa9g8 s4oUQX69Lu0BRXFfXT5wHJ3ZPtgu5Bj8kdEu/tQ3Stjon1Le4X6cGxWC65O/fXdZlLBJ azxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733822951; x=1734427751; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=1vqDO82M+XCnuau+JNGqK2Zehr2Sd7WYRXd6APN34sY=; b=IntHlEPCzyk2s+poQViDiSz0z0VP/3Ofy4vd60eDDRhvX+uapSnPTIJCFpEsHUYQdC MyeLTKMRZDu2jEOzaazTKOhiRPCfsjDrorQ2M+VMEEvTlQ5N/6gOTJX1YYsLdqhW84gN wbor0r/1zCV5M5yBYmH8W2Kxiu88nACRmTwPi6tUZ/bE4wsQ3hmnpT330rd5+Ud6kdZx oaXnBEYiYIcIxSALA96jC4f011obHoQpN7/UDsWAJ7DB+o1zfSAZy/+36y6HdczsVhNZ bifuPiefpXSLsO2mKw1nsGSkFMUULN1/UwUUMrRDzmgsEPOKja+bfunKsAGne9XWsIRd +VkQ== X-Gm-Message-State: AOJu0YwkPdt3ivkoIEeWOxA7cZ0rssB6+juQV21OOloPGxOFVNQZSKSA Vwi48fclRjB2u07PDd4cdi5nG2cEPaQsCtB3OOZ72K32n/NzPTkj3octzDNLBjE= X-Gm-Gg: ASbGncsl+JqR2e8jt6ennuYJ4RqwNHFV0l1ns9AwFg1WnitBGzol58IPju6gP0HGkfV O7UGRc5tdz1RDOK2mKLU0YtzF5IUolu0cFHhgEpG88+C0X9nwGS8eMGpn1nTEYNYzHlRyrni0AX hrcWI3BC10voUYbrXIe72mzFtk2uoGaVos4D8ZuJzBIqdwwNtSt3fk1tq9qmUbR+OECUUMpcXvA aeOt4ezGq9K/iOfTCSXL50hxZ/HR9kdmcNTUJ370QRjiGj3x+gXQtv47DUeYhJt8dU1BzfF17Lr Y1M4tppl X-Google-Smtp-Source: AGHT+IHZXjaBzRsPyNk7dOYudpCeCcHTvHtx8g1LtmYlWSNCnk4Uj7a71khFvxql+RO1r55gAZOC2Q== X-Received: by 2002:a17:90b:384f:b0:2ee:b0f1:ba17 with SMTP id 98e67ed59e1d1-2efcf2c624dmr6555372a91.37.1733822950656; Tue, 10 Dec 2024 01:29:10 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd1568f26asm8750095a12.9.2024.12.10.01.29.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 10 Dec 2024 01:29:10 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 3/3] mm, swap_cgroup: remove global swap cgroup lock Date: Tue, 10 Dec 2024 17:28:05 +0800 Message-ID: <20241210092805.87281-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241210092805.87281-1-ryncsn@gmail.com> References: <20241210092805.87281-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: m4dapozd6sbnurg48ycbqzf4hipp5f6w X-Rspamd-Queue-Id: 61976100004 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1733822928-599563 X-HE-Meta: U2FsdGVkX1/F5PyWj15Skff8oUKO3uFDrpSEBk/8Qb4tGJ1BdGeCKhURX4uCJN5eNwTEWZQy65pBZIb8A6ZkN6AmemEcfK70B6U6I9HA4qzxKl6dxie5C7Cf43yPKlBrHGnQ4OZAOIzfeFYjsoMc8scIQrH74blR0Hl8HQTrMaKJOHKuGbvB/NS2ofifCh8Uy/lDXlNTqbe6J4s3HBGe/YMwu6xZyA1ut97ubo0Y62ferY7ezu53vGTrvbKeF35pnSBAVFpeDikpO7myDFKiouDG5+hj4gelZbdjVcRQ1NT1dQSqVpVEals4ksetJMHPcZuFo6HwjCxB11i3xy20A4IWrd2/iKEv/jBPQprrOkguqaiarX/kpgG+c3m31gTm9f7Wb2Thrm9BYezLdAH0EPgR6ZTlAQKh18GgTxVFd09McBptDe2075nxjyxGUohSdNn+g5heHdDjLnI2lyk7xatpNrsVs1DyOtXusdvbnPXmCB1xFKyDBtNa336G66ZvedgkjJea5TCrbFbBKA0trlQ5RAjmBUG65yOmgWWpJ8iqPFq3jfDjlwzxvebY7nRQoMK87R0DEfUl8Lm7X40RazSjQ3uGsYDowv7+jHk8ZiinYAhR8wQU0QmgoaVK2R5guBD/Xrnk7OELV7eAKm7YE1yfUbla82b0cPNXoGsMgVMIZoS8EhnLvHFtfsthY7RIAQzUDgM4ZSKbgUMjtt/eCZ0Pepj4B5W6bNo7OK2c1NHkNauNuamtbdFmW5+IYlyviJMYDzzK5mCIBpmoWbqrQhTdiwf2yIUzNGtC3n+tyIlY7cZqNm96TATHfNqgjE3etYh+DnJGnlOtqdpwaG3Ses2eYxr8rKDFiT0BiX5ou2u36OKigUMeXelHSKoeWL0PiS0ImgXLU4D5edKaLt7NTmBXUOSCpZEV6wpa53v299EolF8wQbVQ0MHp+CJXiwH5M4TFfVWm8ZynDvTXh2Q KVcBvwqS z5tnKSyWFyHzZtRNmnbOtIxgzROeNFkxmDjRISowHxuodP23puHa8xiVkIm83ujdscqsbsEpiZYRwWB3vC2XedkReqSaxFhKLh6bt+SQ2BGcnxO+xtDmFMZPbxmGNcd9yGOtTn01QUUcAUgJBi6HQRPLSGNrEpJ4VWLqWjhkw8Sf68C5tz3nwnF7DysZ+d2HqZkWRkFXwrYEjYNfEnc61RviJ4/rG3QtP+iisc4P1Us/S2J8qw/S0+7JGIkLNmtz+5eb3x4WQFZf5217/2f4EAzwbvpMc9tp3A+IXbYG+L8Z0MAqsbOWyncBiTDf/1y3EMw6ou6v2/RwLfkVuQR3yoyUz3LcF7Arb2fqxkTbLZtn3mzAF6BrZ1sjXeb3xrH00lgg7WjWRduMBo/dZKrsoCexlI8RTJUB6twH+ivyXqdRxVARgenCKGA8qfVmtdbX4wCLdphqIlVjybJyx1kbFlp3Evgc1819VQHsNTY/lrNGV6o74n3GZxhPLT0pgijnNUGMswlJUN7hu4011QtNKzpR0lT1rkK/FAej1DSjIJSOUtQ1rQFwhvmzIpAi2avhCCxUV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song commit e9e58a4ec3b1 ("memcg: avoid use cmpxchg in swap cgroup maintainance") replaced the cmpxchg/xchg with a global irq spinlock because some archs doesn't support 2 bytes cmpxchg/xchg. Clearly this won't scale well. And as commented in swap_cgroup.c, this lock is not needed for map synchronization. Emulation of 2 bytes xchg with atomic cmpxchg isn't hard, so implement it to get rid of this lock. Introduced two helpers for doing so and they can be easily dropped if a generic 2 byte xchg is support. Testing using 64G brd and build with build kernel with make -j96 in 1.5G memory cgroup using 4k folios showed below improvement (10 test run): Before this series: Sys time: 10809.46 (stdev 80.831491) Real time: 171.41 (stdev 1.239894) After this commit: Sys time: 9621.26 (stdev 34.620000), -10.42% Real time: 160.00 (stdev 0.497814), -6.57% With 64k folios and 2G memcg: Before this series: Sys time: 8231.99 (stdev 30.030994) Real time: 143.57 (stdev 0.577394) After this commit: Sys time: 7403.47 (stdev 6.270000), -10.06% Real time: 135.18 (stdev 0.605000), -5.84% Sequential swapout of 8G 64k zero folios with madvise (24 test run): Before this series: 5461409.12 us (stdev 183957.827084) After this commit: 5420447.26 us (stdev 196419.240317) Sequential swapin of 8G 4k zero folios (24 test run): Before this series: 19736958.916667 us (stdev 189027.246676) After this commit: 19662182.629630 us (stdev 172717.640614) Performance is better or at least not worse for all tests above. Signed-off-by: Kairui Song Reviewed-by: Roman Gushchin --- mm/swap_cgroup.c | 73 +++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 28 deletions(-) diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index 1770b076f6b7..a0a8547dc85d 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -7,19 +7,20 @@ static DEFINE_MUTEX(swap_cgroup_mutex); +/* Pack two cgroup id (short) of two entries in one swap_cgroup (atomic_t) */ +#define ID_PER_SC (sizeof(atomic_t) / sizeof(unsigned short)) +#define ID_SHIFT (BITS_PER_TYPE(unsigned short)) +#define ID_MASK (BIT(ID_SHIFT) - 1) struct swap_cgroup { - unsigned short id; + atomic_t ids; }; struct swap_cgroup_ctrl { struct swap_cgroup *map; - spinlock_t lock; }; static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES]; -#define SC_PER_PAGE (PAGE_SIZE/sizeof(struct swap_cgroup)) - /* * SwapCgroup implements "lookup" and "exchange" operations. * In typical usage, this swap_cgroup is accessed via memcg's charge/uncharge @@ -30,19 +31,32 @@ static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES]; * SwapCache(and its swp_entry) is under lock. * - When called via swap_free(), there is no user of this entry and no race. * Then, we don't need lock around "exchange". - * - * TODO: we can push these buffers out to HIGHMEM. */ -static struct swap_cgroup *lookup_swap_cgroup(swp_entry_t ent, - struct swap_cgroup_ctrl **ctrlp) +static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map, + pgoff_t offset) { - pgoff_t offset = swp_offset(ent); - struct swap_cgroup_ctrl *ctrl; + unsigned int shift = (offset & 1) ? 0 : ID_SHIFT; + unsigned int old_ids = atomic_read(&map[offset / ID_PER_SC].ids); - ctrl = &swap_cgroup_ctrl[swp_type(ent)]; - if (ctrlp) - *ctrlp = ctrl; - return &ctrl->map[offset]; + return (old_ids & (ID_MASK << shift)) >> shift; +} + +static unsigned short __swap_cgroup_id_xchg(struct swap_cgroup *map, + pgoff_t offset, + unsigned short new_id) +{ + unsigned short old_id; + unsigned int shift = (offset & 1) ? 0 : ID_SHIFT; + struct swap_cgroup *sc = &map[offset / ID_PER_SC]; + unsigned int new_ids, old_ids = atomic_read(&sc->ids); + + do { + old_id = (old_ids & (ID_MASK << shift)) >> shift; + new_ids = (old_ids & ~(ID_MASK << shift)); + new_ids |= ((unsigned int)new_id) << shift; + } while (!atomic_try_cmpxchg(&sc->ids, &old_ids, new_ids)); + + return old_id; } /** @@ -58,21 +72,19 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, unsigned int nr_ents) { struct swap_cgroup_ctrl *ctrl; - struct swap_cgroup *sc; - unsigned short old; - unsigned long flags; pgoff_t offset = swp_offset(ent); pgoff_t end = offset + nr_ents; + unsigned short old, iter; + struct swap_cgroup *map; - sc = lookup_swap_cgroup(ent, &ctrl); + ctrl = &swap_cgroup_ctrl[swp_type(ent)]; + map = ctrl->map; - spin_lock_irqsave(&ctrl->lock, flags); - old = sc->id; - for (; offset < end; offset++, sc++) { - VM_BUG_ON(sc->id != old); - sc->id = id; - } - spin_unlock_irqrestore(&ctrl->lock, flags); + old = __swap_cgroup_id_lookup(map, offset); + do { + iter = __swap_cgroup_id_xchg(map, offset, id); + VM_BUG_ON(iter != old); + } while (++offset != end); return old; } @@ -85,9 +97,13 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, */ unsigned short lookup_swap_cgroup_id(swp_entry_t ent) { + struct swap_cgroup_ctrl *ctrl; + if (mem_cgroup_disabled()) return 0; - return lookup_swap_cgroup(ent, NULL)->id; + + ctrl = &swap_cgroup_ctrl[swp_type(ent)]; + return __swap_cgroup_id_lookup(ctrl->map, swp_offset(ent)); } int swap_cgroup_swapon(int type, unsigned long max_pages) @@ -98,14 +114,15 @@ int swap_cgroup_swapon(int type, unsigned long max_pages) if (mem_cgroup_disabled()) return 0; - map = vcalloc(max_pages, sizeof(struct swap_cgroup)); + BUILD_BUG_ON(!ID_PER_SC); + map = vcalloc(DIV_ROUND_UP(max_pages, ID_PER_SC), + sizeof(struct swap_cgroup)); if (!map) goto nomem; ctrl = &swap_cgroup_ctrl[type]; mutex_lock(&swap_cgroup_mutex); ctrl->map = map; - spin_lock_init(&ctrl->lock); mutex_unlock(&swap_cgroup_mutex); return 0;