From patchwork Wed Dec 18 11:46:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13913538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBA3CE77187 for ; Wed, 18 Dec 2024 11:46:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 564406B0085; Wed, 18 Dec 2024 06:46:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C62E6B0088; Wed, 18 Dec 2024 06:46:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 365B76B0089; Wed, 18 Dec 2024 06:46:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 16E6A6B0085 for ; Wed, 18 Dec 2024 06:46:58 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 962AD160DA0 for ; Wed, 18 Dec 2024 11:46:57 +0000 (UTC) X-FDA: 82907902560.05.6FF8B1C Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf19.hostedemail.com (Postfix) with ESMTP id 6A5261A0008 for ; Wed, 18 Dec 2024 11:46:22 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=itdR5Yyh; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734522380; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XXPbw4CmFZlsHQ1dqNkun8nNSKspNNgSNNzQzzqG5eQ=; b=zB/9/j75C7gtlrfSKG5BAEz8VhBMIsLT15wOZh2+7i3vyWgCR6T/gM5hOU9YTW/03avHbk IiD6GJqxjASS6Nu90zEXyrlmae7b75e7kRQnhLD7QEuynzHvo66A4H+R8pRFhQtnW82MS8 semCygJf2fGEdiZB7tXG4hLTg9vN/uQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734522380; a=rsa-sha256; cv=none; b=C/nwbko06Ab6nDU4NJys8l6HWrAfB/aeT9Qn+rcts2bho8W6SDehrip7rfwel2HorSWKI2 Ec9+6Gj+/3LeOYkzYWRTIOhAvoMkLqNpQ43AJZEngQ1pnZW1krT/JCRcq+/EIjL47l+qZI vjG8lACDTQvQ39mac9M7FmK8CAXxP4Y= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=itdR5Yyh; spf=pass (imf19.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2166022c5caso50861105ad.2 for ; Wed, 18 Dec 2024 03:46:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522414; x=1735127214; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=XXPbw4CmFZlsHQ1dqNkun8nNSKspNNgSNNzQzzqG5eQ=; b=itdR5Yyhj+h/+WEqWv687WKCIImxlKpR0HdX2lhhTjtOiZxvW01fWCohTRrtgRFClL YTWlgZji4B9Yx9H0jeFohFfY3cTJdMgiOLcMODEbNO3zIZKpsl61bzEpnjiuv+3nEwAY LKFjxVl0lv+If2PyC5BOV2Qoki/9G2mtkZepYrcnIKn6x5NvCcfImtDCF/JLhL9uBO+J oRhzlZMEYpiC3nr27L8JgwZUBp/OwsZ75EEuRYFOexJCQLJP6s96U8ygxBnihbHQjqz+ 6c+ZQaoI6FLk9IzfNzDOq/AW84RWXas9K+SXWYTFJT5oymwCH4jw1VVDhgl7lCPmIGSY t3LQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522414; x=1735127214; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=XXPbw4CmFZlsHQ1dqNkun8nNSKspNNgSNNzQzzqG5eQ=; b=YrEVKfGPawjDY8m/GcdbrvOuUY29UJiQbreGHKwZ7b88nvBIlQIIX20HDLruAxcS9x MyQ0JAK3vdp5SOGY0I8AVCqfrZ0XmlLlvlFqOwVjyi8/pZYNKDT2UOX5pVEhST+IZIDm Bk/v7oXd95kH999Mdi1FTWnBvwEA+lnaCxdnOSmxij1TCR85nHFjaWMUSYwE6skssDnh lCymSUV1dE/8gnQJEQjwulTZFVKkQrouYl+2B3sOTcIEg7uJjMGqR/g0I7/qNaKucFS3 F9Zdv7Gosnq4P/QSWS5x318al1TiFvItoAqRapfCtsce1QslCMQB2aduaH1VmhRed6Ei mhYA== X-Gm-Message-State: AOJu0YwnyAnbTEx+ZMms0mqZJrN3dOKqy0aXrtZhHys991AI5lnMlVei IPPcBYT2SYTo0OVTvQJpb104NlYaSrxCSWth054WPW552cfp8kDK12Nhz/gn X-Gm-Gg: ASbGncsy0dnfC3b+hxCv7JZExA45/M5g20bZm1ARWAZffZdPYyAmR/pZjRVFAgTwt61 e4f1QTUDTM3QgnKLgObyP7OvQaXABmscIeAK4L+FsVI9+AB5/7WoQAZAL3kBHdIRH/2uD+HB6ZM mz2HE3e/57C+6tgQlhuGSlnprVGURbO5a8gGepln+bwGa9KSC+Sl4YzZC6BTOh5NKUhdagdcTNq FjiVD6Ej4hnjVkIekGCvAo1/dSN0W2FSDkcad6m4lbY/7l0miBt2qIs7QI51wWwZ9rKYT/IqVaj 5Huses+4GQE= X-Google-Smtp-Source: AGHT+IHeEFfNZXB8eHhDobzJ6aD+/g4JcIAsmjLr3dsquieQlQtdJw7HvtuvZYuwp308/geLA7elvA== X-Received: by 2002:a17:903:2348:b0:215:9642:4d6d with SMTP id d9443c01a7336-218d6fe1d79mr29573335ad.17.1734522414061; Wed, 18 Dec 2024 03:46:54 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.46.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:46:53 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 1/4] mm, memcontrol: avoid duplicated memcg enable check Date: Wed, 18 Dec 2024 19:46:30 +0800 Message-ID: <20241218114633.85196-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 6A5261A0008 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: smhmjskbyj3qwoos1usjpequ64zhcfe5 X-HE-Tag: 1734522382-886645 X-HE-Meta: U2FsdGVkX1/2Qb/LhrIIAa1m2suT/xF44zdgy7N3Kc8nVxEBf83GfB+ELRD22uT1VWdKZN8/sfEG7QjdNMBEHrIgDSd3XUvk0XOsSu/tTDkyLlvHnxqd1MZeY8z2wcnSSv5l9PP0OZkcc4mUNWkDTztk6GsEmDI7mkm7tdwZda1LlE8XeNZkWHtHu3gdIoeAKUDTlrrWf2QEX32YvaVJIp+f9gXWyn1G/Niu5BqkhxpV/Mqhyjmb9BwwFx+fyZsKNhYW2nQPTWOGDdFBJrtlh/e+jeWqpefDjE2vSQlLdpJ+UP+f60gkiAHVkDLyUkAbXnAC5khkLfq4VgqBXJn4cTU3Y0zDmUe2Yspd/bQXRm4K/OrbEpgXTGU/Mkyk53xn4ATJPozNIQaCn4Hx+ESLvBCTN5k6FPwAvOtk0AtNzBB5itk3/LTLyKNo0a8U3fgGTn4aHNVc5jFssXKKwqf36DLS9rRgQR61nHybJoGLSjSN+Ad2FRUwuSpGqsnpSUceYxEjxUyhzpbSkD9qdnYKU/ke3OeSShCA2E3I47+nj1XzO/epoUVd3BVhQYnJSyUZXkkCSmrbPHtusTNONM93ZkHGbrzFmIo2YOmPK4el5MyB03cKaWYYgnIgmZqbOayTV0eR1G7Z7zz3T9jtH/U5hUCIdsx6vygYCBa5+uNI6ccpUhMAGVXvMyC/ofzrEqLC57Mi2i+062NcJ3p4XlF5Jiot9Tlvf/Rd7ZNDJ+3zwxeJs4RDxQal1MXFyRjLzqL6D2BOpKCF27iTUYFssrH0TGSGNEt0QqUsL2DibFcu6QrlsCCe9nzOrq3W2jrw0rv8S8crlDO2bACXQ1YXfttA1cTVa7NjR7Jgh9U5yjImwh/IjnuTQIPNBq92bDaDyuRGvw3BF+wRsIvklGSDGCJau9ubwHSY7wZ1OFlvnv1yreB+ique3wwre8BdGadlmvZfWqdco8I4KocSw/dbIGJ zy8/J5oE k/nJ3twDwnUtR8q3fT3T9O2hhsbwscQ2obixVEyPm4XG5rsCPJJqsO3sgjwFw7J33bm+xRS7Z3IwQQmif8/jgpNRfMDowoC8/vHoSJ6BOGu2G8PKRcLD7gaIatjnWHiFbY+xa4vFG1Q/sryDCp9Z3W+CHZ7CeNQYljTrGkZs56hwcqznkoM4R3Ag+yO4LRF+6mb5r265yZPBJbkUtexOaXnBAjVFnwves9X28+jY08vxLYcvV+0YGmBmV9hYKXs9iNbAT0ITfESsgmY1yWQ1aBOPVn/BKDTpSZJsvpzMrgY9Qy7a+dgGXyvxjYnemvm83HTNfljhknx+XXQrmQW21I7Uu9NbeW5B6nzZet4NfDxhrccs5bfCo36kF8ozzLhFq9cAZ/pPQjblEMn8V3/zlz48w//0VyR9wYfBeMHHrPvw2hAY0fuf0yQCKAtaCYFrKW+9FSkUQqGJ5o78Hug8caTdJTVBFaD6vG+fUhN0FwlZK8D36WP/ca/i8+AqRMPjgTEvH1Wt1Ugsev0FJffKY9F0YnwasMUbrQBVIO5kyWt1EaQadswMxK21qSvsn+U9vEHlZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.038348, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song mem_cgroup_uncharge_swap() includes a mem_cgroup_disabled() check, so the caller doesn't need to check that. Signed-off-by: Kairui Song Reviewed-by: Yosry Ahmed Reviewed-by: Roman Gushchin Acked-by: Shakeel Butt Acked-by: Chris Li --- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..79900a486ed1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4609,7 +4609,7 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry, unsigned int nr_pages) * correspond 1:1 to page and swap slot lifetimes: we charge the * page to memory here, and uncharge swap when the slot is freed. */ - if (!mem_cgroup_disabled() && do_memsw_account()) { + if (do_memsw_account()) { /* * The swap entry might not get freed for a long time, * let's not wait for it. The page already received a From patchwork Wed Dec 18 11:46:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13913539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81A9EE77187 for ; Wed, 18 Dec 2024 11:47:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB32B6B0089; Wed, 18 Dec 2024 06:47:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E62516B008A; Wed, 18 Dec 2024 06:47:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D01E76B008C; Wed, 18 Dec 2024 06:47:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B08B26B0089 for ; Wed, 18 Dec 2024 06:47:01 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3D70BA0DDE for ; Wed, 18 Dec 2024 11:47:01 +0000 (UTC) X-FDA: 82907903064.02.6D490CA Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf22.hostedemail.com (Postfix) with ESMTP id 21DE1C000C for ; Wed, 18 Dec 2024 11:46:25 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fyfEJ3Ot; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734522397; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BSCLGHYqLWfZbOCM52YJejtZzzMrW+w4kb+fWMx8dAg=; b=eeisvRurnTC1e6G47P8yM2Vjmq/XPJuXJ3+Yah28bCaobs9hM9UepgjMreLfPOTckkBPtx dogDnUSbmmlIb5NfRUy6nflTxWIj3O5FeNw/mrqOPHRua9k+TeDuvJKNyDHCshOyD6hNc2 /+h60tQ8NWvFNXhuZcsdgwiFd/FiRaQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fyfEJ3Ot; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734522397; a=rsa-sha256; cv=none; b=2ac2+0bHF4eGHam2gPS5e605UDM+yT8F9gxlig7t49bJ57bEWU4qxHt+T/agOEPbvWcG2t /q84pPhsKD5fLHCrPSOu+e/mND+xnnbsBE2FFMUwldXY7n2Qz4s2Wd+e7Pd+ED8JUabguo XXPGWa1E8LRRXdKfRYA/ulkzO/p/Hkk= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2163bd70069so64637715ad.0 for ; Wed, 18 Dec 2024 03:46:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522418; x=1735127218; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=BSCLGHYqLWfZbOCM52YJejtZzzMrW+w4kb+fWMx8dAg=; b=fyfEJ3OtllSvlF0fDQKlVpSdjTIuid6Ahn5jPAP3sD3P00JGT5Eei5AaN2Z3/60Oey if1DoS6/+CzEqd/Iq4guIAPxrNkk5ajqqMbOLQ2iIcDozra/3o/edgAqEPyRZdjbvVmC 9ZbJsdpA1A4IvNoQocbuqFxgTTOyHQa4kOLgnUKd/bU7QzMcx7KVZAk1y2U1EkojokTn kpmQiiC3C/wtqMMZRMaw7tLm5MQk0ydQA9akznqg+PhLlkmliANokMZWg7/EYfnHPQAN dXS1oyQP9nHtEWf8Em4VNpt5V30EKk4vXT+E4UToYfcrYw9DGS44zov/XCsVXyVhXKEc Yl2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522418; x=1735127218; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=BSCLGHYqLWfZbOCM52YJejtZzzMrW+w4kb+fWMx8dAg=; b=juWiLaoo5qkmnwyON9hFNgqQx72Gk4CBJd0U6D1EoOxvHQmL9FFxdO4YyEwR78NG0X EjaPYgSBKiV3h6Chm1QN16JbLOWQCX+P/oHjxKMrGMVHE4pXrgGBXRKGa+fTKKx966xb x5DBKHuoA8T7NOKxzWdsALdv5k/hjwpTlIjVfgnUPHNc2RArJ/2Zt32uPqgmbI4R2oXP DrT96CyZSJeZDh+vBaVOEV9Zew9Mn5XzHrJUzJNTGWM+wy5CWMDyi1Pz8Cazxz8zEHpi h7DG2ocx+tBBbvZ/cu+wzK9K/wS5mM+kOpIgfPoSUhs65R6Ghe+X039T/PknjL63DMhR 8brw== X-Gm-Message-State: AOJu0YznXxyNEaw10CH68dTE0BjLGt3KCFxS5ZglfgWdNJ/2jhL6MfxE l4R2vfEZLy1WLxOr0kjHwWMc4rzlZIzBSyH1yYy19vM5HxrC6c0459Bdpf9S X-Gm-Gg: ASbGnctd9Z8/wOlCg+gEdUdOwugVSJ5GVjvMChOgOppxxEPoesqBcDO03grZWZsErcz AaCLIfgNsANr9NcpTRWjOcPQnTKHWSiXhseSdNrX+g1/FQrwP13UxLsdA7emivqPbWHPiQHNihp AyLAT/BFqNYkedEGJsXdKFs8I8RrXWScPNGqHt8fvDXPKVCdJV/G9frnKcAkrBOcgLeiQutgLll qHieD4k7yzXnDTNGDJxgpQTlqft/dFZ1e1L9pSfbzKHQtv1YAPNCOhHaEkx8yKoSmAhN4UHmF/p FdqcHhu3SuI= X-Google-Smtp-Source: AGHT+IFDiEO8Z+4jJiOJGyVQCnx8m44fCY5Sa4Cm3pyPpkeXJ0qEbZhG2rc7F1TYGzL+U0+D2iRi5w== X-Received: by 2002:a17:90b:2745:b0:2ee:8ea0:6b9c with SMTP id 98e67ed59e1d1-2f2e91d7dddmr4670423a91.12.1734522417699; Wed, 18 Dec 2024 03:46:57 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.46.54 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:46:57 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 2/4] mm/swap_cgroup: remove swap_cgroup_cmpxchg Date: Wed, 18 Dec 2024 19:46:31 +0800 Message-ID: <20241218114633.85196-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: 8xxsjoc6cfy36nfnai3eno4du7hd1i4p X-Rspamd-Queue-Id: 21DE1C000C X-Rspam-User: X-HE-Tag: 1734522385-584801 X-HE-Meta: U2FsdGVkX19Gf2boREuYxd20pJdF4Tuefz0JmISsbp0f5dmYIejpGl3rniZoXKS4W1P+3xiK4Qx0J+wslXYNh/3LmqMgswnh2nl7Ql+6C3miRQsYdQtqJBwRzWUdpwkSm9+uF798j7fYcEzJKcDBbXb1ath7aiUJTaAvbU2gfnreCDhCE5Fe+sBDcbKW9avGGWB/C5HU3E3UHyQNJ0xPQ8bCeFGllOxliAbMzS71fWOtmj2341v3v+cIMBuofmy2i09b2R6HtZSPrQENTyKmOA+7ngeKYK9/uASHDTDH7fZ7BeIwG2HKY4Xy+nx3rJdf9y2aKV0HEKPG1/LjHvudgIb0AL72OaTB8s6uug7X1Ec29TtOD1Cdrlbh9nVQtKFYSKG+ghw/9NaKWKsDlpvAkKXkuA53sg14MIfcq56sdoCSZxHWepTKyOrjGdVxKxsItlKEcHfnakrs00M0kPJeRzlE83Pf9D8HrRzXMFI7r69ye+ayZSarrDOpOx70TehJHGRQnikedpJb+wRSfh0CfsX/GC6GZo16EmAgLkDEkxLafODoY7lI+YPCgPGKBC72nOS2OZUVGAwxWMhtsY6yoUG89YgnOsx/HGeb2nMkJjIFdhthF+iJBklGHKQMMCTcOgABy1S6XLnPDo3mijrDWadqdJRow+S54GHpEh5Oa38IME/zEEfAGwRzO6awQL73tAeyXstgDCfqLNXJ7LxrM9VhiCa8yDZkY7UxpfaDPWChF7L83rdVhQLxhfpwJWvhZjxFziKZimwXmqCB922iunHYu1nDnS/OCsvD8EfVajix+Ti457aZd8OljUVYXVN0ycPIqz8fuqWfXxlzyMxw2jStD/PkxXiNlXQ5t1/ugvzG/6An7O5Zum5HHCusjc2qT6M1BszTvaFs5Hsff4a5qrcFqQWLH9dvSOWWFSKpO26/OSS7tUWu1h+mS30DTsCXg+iGRpqSDf8sRpvyBVH pAbWW2/f GPf6T7GlDfH7VtbKwddH0L1WHXa77hKroTRxsSNrIkmmEMJEYta2DQsPO0sHaE5GjNnP3o8QFjd2ul8u7sf1JmZMQYlfssNz8A/pyheoqhrplr98kUtbUJ2mfrq1aHb36jvKOPX/i6XBqw4TjV3uOdUr7j/P/VrpjvgZvU0UM4Vsd3/DEPxLpMDyMoaBcv3ht4ZypwUUAQfFA3IjkBR7ceZZJCugxnre56QFyxXv7A+vbUtSL9SXGHAf0nE1EvpU4Q8q7onDRMTRnAlis4Hrb3fvo0n+GYDHQFIZGuHdsgkIHcH4HdmxgUwZAbt6O5vcRq3CzQFC2jAe0mtQwwspKXUG9a/PZDSUmeBomG0iAet/bxLGPfRixwvntkfajg+edSgTLz1FbsaHgZkYIYx8YK0L3tE973ZP+Epi4jz4fC/N5j/VYqeajYODrFD2wEPrkiP20aLduN4CxrsccjSnW96tDpW8Ek3JRCcdOKNKvlmOzIEEqMvrx2ngnZO/ALbA+IPVkUhDb38IBCsGX9Ty1dUEVjhZuSVOIToLg6QuKerhzeqjV6URsVFsJsWlHqCfZPTGbFVhSFFPmbLw1+SYoc5muRYhMXPW63JFbNHaDlWaxlTw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000043, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song This function is never used after commit 6b611388b626 ("memcg-v1: remove charge move code"). Signed-off-by: Kairui Song Reviewed-by: Yosry Ahmed Reviewed-by: Roman Gushchin Acked-by: Shakeel Butt Acked-by: Chris Li --- include/linux/swap_cgroup.h | 2 -- mm/swap_cgroup.c | 29 ----------------------------- 2 files changed, 31 deletions(-) diff --git a/include/linux/swap_cgroup.h b/include/linux/swap_cgroup.h index ae73a87775b3..d521ad1c4164 100644 --- a/include/linux/swap_cgroup.h +++ b/include/linux/swap_cgroup.h @@ -6,8 +6,6 @@ #if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP) -extern unsigned short swap_cgroup_cmpxchg(swp_entry_t ent, - unsigned short old, unsigned short new); extern unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, unsigned int nr_ents); extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent); diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index f63d1aa072a1..1770b076f6b7 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -45,35 +45,6 @@ static struct swap_cgroup *lookup_swap_cgroup(swp_entry_t ent, return &ctrl->map[offset]; } -/** - * swap_cgroup_cmpxchg - cmpxchg mem_cgroup's id for this swp_entry. - * @ent: swap entry to be cmpxchged - * @old: old id - * @new: new id - * - * Returns old id at success, 0 at failure. - * (There is no mem_cgroup using 0 as its id) - */ -unsigned short swap_cgroup_cmpxchg(swp_entry_t ent, - unsigned short old, unsigned short new) -{ - struct swap_cgroup_ctrl *ctrl; - struct swap_cgroup *sc; - unsigned long flags; - unsigned short retval; - - sc = lookup_swap_cgroup(ent, &ctrl); - - spin_lock_irqsave(&ctrl->lock, flags); - retval = sc->id; - if (retval == old) - sc->id = new; - else - retval = 0; - spin_unlock_irqrestore(&ctrl->lock, flags); - return retval; -} - /** * swap_cgroup_record - record mem_cgroup for a set of swap entries * @ent: the first swap entry to be recorded into From patchwork Wed Dec 18 11:46:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13913540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79FCDE77187 for ; Wed, 18 Dec 2024 11:47:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDE876B008C; Wed, 18 Dec 2024 06:47:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C8D416B0092; Wed, 18 Dec 2024 06:47:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B090F6B0093; Wed, 18 Dec 2024 06:47:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 908076B008C for ; Wed, 18 Dec 2024 06:47:05 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1054EC0D66 for ; Wed, 18 Dec 2024 11:47:05 +0000 (UTC) X-FDA: 82907903316.29.2705A7B Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf02.hostedemail.com (Postfix) with ESMTP id 7C5ED80015 for ; Wed, 18 Dec 2024 11:46:02 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aAJVghdu; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734522390; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+Vzzy564Q7ltilyK4Qu+63SEqQV/iay9myd3eLrJl6Y=; b=CHIPuTiFwcjqtGt56cjeEGarrUAb/eXU9Ko6EheSS7GVXtHCJSv5uMzNpZ4Dkt6RQqf1i/ QEZas4Rn2DJ2m0ehCwjeK6aG5ree2cuTs+W82Wx2uO4f93zzBlIPPKnxHl3orvtswx/4dc EqvB+AwWGiGIkwSVIkhx04me8cEobXg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aAJVghdu; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734522390; a=rsa-sha256; cv=none; b=SL/BEqWB0SBRAc1W0hWh7wI1pwyjS9cfJ9cyD8dh/ESUvK0M794alrNQmj97cDY6wGKJwr 3ww7WsXfuob6yHYZPDwpmkO6B4S139IrUI4FBtvTcn9ycJw1zKUSAw/P+5tG/b0yK56rOK uai1B62Yzz+0RBIwBY2K9T/qi5GrJYU= Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-2ef87d24c2dso4938844a91.1 for ; Wed, 18 Dec 2024 03:47:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522421; x=1735127221; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=+Vzzy564Q7ltilyK4Qu+63SEqQV/iay9myd3eLrJl6Y=; b=aAJVghdu8Wv2nKeeEeXfWGiTwoWAmd7EWeDAC4zCVldBqEMkpMKBdFcKd+ia0GJwW5 qqL2NpW771BWEjjmC398xXE4CKAzaqzmuqjqlenjLIrDcqNoifq7sdkNEAxUOIPBUAh/ BA1owxWA9zNtBCWjJRi2UdR7l/k3LBm7HQCmp1XuVQnpAAlXUJNGUmiAlt4Ztih+1O5h Mky16lmmCZ9So67vROIR7TgwALEruXiII0zw9XZ82PEe9Blt9EMeCxQQ8deW/e402/1r 9DU+LXVP0IeyORKfVEXPClJdwLMXm+4YWx34GRwawffXZ1aPRXSvUD1sxWoy4l4aaYc2 mbqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522421; x=1735127221; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+Vzzy564Q7ltilyK4Qu+63SEqQV/iay9myd3eLrJl6Y=; b=FETxvAJx/aUtNFP7TNjF6dVfpHqh0Hkqtr1Xrov6tECWKgCVgWqEjy1kvQpIBO5oWw hnejO2H2hdCwlnbC6ZcanHhGxTlIJZ2ZozYnOy4tgZGaPM6/K3nRCPkUiJDUuXKIZ8F9 WqOQ5NxHfZ/11KYy25+cLsB2AnndCPGohLfqymQW1dGUnx3bQcCpoa+vNSoKkhGNiYIJ fWcYoSQNa2ZPcgZSiY3ExhvbfVVSmYPPiVZSV1eXG5Q12hYod5qI3LwakGGiC2df1+gN PmhGa+dhBZwiymoOHYlRtIOl0u5rP+J2q/HLeDoEWpLsYCAcXzWE1KMEevM9SeXbqLXv Iv8g== X-Gm-Message-State: AOJu0YzcRPT6mI8b/l6AnmAgMekk4hqacw1ukIuYby9Ao8GVyzFRvpWh vV0TvLWb6N5cBSDMi0Y3++9b92ztiDtYWgP/gIJj1XCCwuhgN4n3gvGjOgcc X-Gm-Gg: ASbGncuDTPOvCPyMAHqarJP/2mxhytdtoj0JQED5MGcXqJ5AkoOxT3qXvvKMhXAioMB 1qTS5c39hEckUMjlv8rFQUa7HKKJ1BEVxIyn3CoHtJBz/wXHuYu06I9YzEu0+3NwNXfSud6O7Le Qn6ERcBNLx8gPVHW8TVMNNciqj4HneH2lT+o7fF7xNAW7goC0i+ajVP5rOvgjSFu+au2yaZz25h BioKDwHMVMyiN/J21iV42Hlfr6KPB099NOlJJd/P3BR2DjRS7GQIY9MAjZt5Cig2q0461ILI+L+ W/MbuAQxnUM= X-Google-Smtp-Source: AGHT+IEO+JWDMXQW6FsZJE3+1Wvh/7poieUhJ+CwA1N0IXPZ0ViHOqmG3ylHlvEdt5u9FJAhum58XA== X-Received: by 2002:a17:90b:5206:b0:2ee:c04a:4281 with SMTP id 98e67ed59e1d1-2f2e91a903cmr3823034a91.6.1734522421476; Wed, 18 Dec 2024 03:47:01 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.46.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:47:01 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 3/4] mm/swap_cgroup: remove global swap cgroup lock Date: Wed, 18 Dec 2024 19:46:32 +0800 Message-ID: <20241218114633.85196-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7C5ED80015 X-Rspam-User: X-Stat-Signature: q1t95mixz17buk5xj5cuxt17qyiyxeji X-HE-Tag: 1734522362-883401 X-HE-Meta: U2FsdGVkX1/H5tF4BwIEuc3SV4rA1bU2umU2/kQ4BvpTBtMT5VXto5HF6unzf5MW2k52GTMP8gNDwY3OITVZYqE5Bg4hTUU6IIy4oW5PhOdPGQyjGolaciGx4ZTy4fjnz4MfJ7UzH4miYzepF+ojR6ekcxUdiIxn7iu0wwhhTYjpTqRPMUxHqhfrKmqzmfZS8VWYMKD8wYIcek/TanqyK3n068NJAUVg1fZtzDoqr74J1pP9mYUa3Vb7+i41/P8B8/mtTEhuPOadH2ZGlps1yfkEmdY/SEQ/K6WFqqIYmktNAcjP9vcBPaL6/JXgMHeKF3HLN1/FpizLqSHM54iBnzlKfXa0REMFpFn2dqPovB4K/9HQxrfC7PUppUz0Lvaje1LRcefaKBMEVTxDk5wZfZPmMhDTv0ff4PdcKi/sOkAz/bKKSPpIxB41pVh3a7nbZcZDVLTjcg0Ld8vFjCzCEYdQCUhjHhMPiiwe634vVuHq49BIWmvrwvKW2+ef8R0A81FOevNMh8oOBj4CdW9tov7eW8JZ37JFmAWtK7oaqEpkOM17ljUlF+fTTSyUny1uMSnphlhN9U5EaiV3PHC0WS3DsIUadRGtQDWyYezc8HDSyzAYxle+pLJt9a49k+fwBSkm7CY9lnA8pbUJdps8lV9pxr41iiFqrWJw3IXTk1imLnBFbQVw/MvsSDLH9eKiTH01DOVPffdER+/FKVSaoRoLadtCBgNSEsT//1CtkTOy5RCEi89ulTejzdXbYdludXpyzVLNVAvD4D2sXPRCQjB/os+cWVUlazmV8NEpJQx2jE099u356+uAmEfEzn78PGENBPAWMQA7+FTXB1zH0Mr5VG4ujR+HbouyCgrF8gl/4XyxONz2VI55YrTfApaPDiMnZcbtbk2I7aez/EKZQx6datcAyF35qRAAr3nksNnKA7W/SSuF6nHBb9yiMdB5O5AmYkwGkVxCCK9Wiut qNImhBOo uXgv5ty4qpMHcTjAkJWzGDZzjys26w10UarJxMOahzijZprXSRONjoqlioW7ado0wCmUkEQSmMagUrmhcK/hJihV4fu9Ceb/qJimxOEOfQQO8buc58XQykcjW4gLl1MXzvBHmZQiLt+oq/PsmyGZzK4DUhRVsM7L6LD1q3SIz48TautcvZ6ZFwLMHYp2kMhGBiWf8hE7gJuesv0/YrHPryse9oWVBfY3G3gy17cGJC5IdaD2zEg2AFgdEpAm8962LdL6Lf3/0P47wi+9F1714YBcuG6r6r9HKSvo4msFhLk7LSJB19wkMGCN6qnuNT3bymyOxE23pBZquUytrjDsw9WkCgLDuro1RrRmcVQWQidioM4RWmY3hsRaVji2SfwzUhZzBIZZoMcci/E4C5F1hvXcNotlZus/L2E3SDssuItuPMnmD0PQyXi8ijv/ffGVtWHhilqEdwULI4rHfNLxjUjiTM9ry8lENvmj2kmXe2f2m/aB+bz1zRaNm1ZWmMuTBgrhu0lD43C+orjc5acBEL9mE8JR1ZWGdRFxZUcvf2crdD1fCUDzKcUfnlo86wjTR5itG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song commit e9e58a4ec3b1 ("memcg: avoid use cmpxchg in swap cgroup maintainance") replaced the cmpxchg/xchg with a global irq spinlock because some archs doesn't support 2 bytes cmpxchg/xchg. Clearly this won't scale well. And as commented in swap_cgroup.c, this lock is not needed for map synchronization. Emulation of 2 bytes xchg with atomic cmpxchg isn't hard, so implement it to get rid of this lock. Introduced two helpers for doing so and they can be easily dropped if a generic 2 byte xchg is support. Testing using 64G brd and build with build kernel with make -j96 in 1.5G memory cgroup using 4k folios showed below improvement (6 test run): Before this series: Sys time: 10782.29 (stdev 42.353886) Real time: 171.49 (stdev 0.595541) After this commit: Sys time: 9617.23 (stdev 37.764062), -10.81% Real time: 159.65 (stdev 0.587388), -6.90% With 64k folios and 2G memcg: Before this series: Sys time: 8176.94 (stdev 26.414712) Real time: 141.98 (stdev 0.797382) After this commit: Sys time: 7358.98 (stdev 54.927593), -10.00% Real time: 134.07 (stdev 0.757463), -5.57% Sequential swapout of 8G 64k zero folios with madvise (24 test run): Before this series: 5461409.12 us (stdev 183957.827084) After this commit: 5420447.26 us (stdev 196419.240317) Sequential swapin of 8G 4k zero folios (24 test run): Before this series: 19736958.916667 us (stdev 189027.246676) After this commit: 19662182.629630 us (stdev 172717.640614) Performance is better or at least not worse for all tests above. Signed-off-by: Kairui Song Reviewed-by: Roman Gushchin --- mm/swap_cgroup.c | 77 ++++++++++++++++++++++++++++++------------------ 1 file changed, 49 insertions(+), 28 deletions(-) diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index 1770b076f6b7..cf0445cb35ed 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -7,19 +7,20 @@ static DEFINE_MUTEX(swap_cgroup_mutex); +/* Pack two cgroup id (short) of two entries in one swap_cgroup (atomic_t) */ +#define ID_PER_SC (sizeof(struct swap_cgroup) / sizeof(unsigned short)) +#define ID_SHIFT (BITS_PER_TYPE(unsigned short)) +#define ID_MASK (BIT(ID_SHIFT) - 1) struct swap_cgroup { - unsigned short id; + atomic_t ids; }; struct swap_cgroup_ctrl { struct swap_cgroup *map; - spinlock_t lock; }; static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES]; -#define SC_PER_PAGE (PAGE_SIZE/sizeof(struct swap_cgroup)) - /* * SwapCgroup implements "lookup" and "exchange" operations. * In typical usage, this swap_cgroup is accessed via memcg's charge/uncharge @@ -30,19 +31,35 @@ static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES]; * SwapCache(and its swp_entry) is under lock. * - When called via swap_free(), there is no user of this entry and no race. * Then, we don't need lock around "exchange". - * - * TODO: we can push these buffers out to HIGHMEM. */ -static struct swap_cgroup *lookup_swap_cgroup(swp_entry_t ent, - struct swap_cgroup_ctrl **ctrlp) +static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map, + pgoff_t offset) { - pgoff_t offset = swp_offset(ent); - struct swap_cgroup_ctrl *ctrl; + unsigned int shift = (offset % ID_PER_SC) * ID_SHIFT; + unsigned int old_ids = atomic_read(&map[offset / ID_PER_SC].ids); - ctrl = &swap_cgroup_ctrl[swp_type(ent)]; - if (ctrlp) - *ctrlp = ctrl; - return &ctrl->map[offset]; + BUILD_BUG_ON(!is_power_of_2(ID_PER_SC)); + BUILD_BUG_ON(sizeof(struct swap_cgroup) != sizeof(atomic_t)); + + return (old_ids >> shift) & ID_MASK; +} + +static unsigned short __swap_cgroup_id_xchg(struct swap_cgroup *map, + pgoff_t offset, + unsigned short new_id) +{ + unsigned short old_id; + struct swap_cgroup *sc = &map[offset / ID_PER_SC]; + unsigned int shift = (offset % ID_PER_SC) * ID_SHIFT; + unsigned int new_ids, old_ids = atomic_read(&sc->ids); + + do { + old_id = (old_ids >> shift) & ID_MASK; + new_ids = (old_ids & ~(ID_MASK << shift)); + new_ids |= ((unsigned int)new_id) << shift; + } while (!atomic_try_cmpxchg(&sc->ids, &old_ids, new_ids)); + + return old_id; } /** @@ -58,21 +75,19 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, unsigned int nr_ents) { struct swap_cgroup_ctrl *ctrl; - struct swap_cgroup *sc; - unsigned short old; - unsigned long flags; pgoff_t offset = swp_offset(ent); pgoff_t end = offset + nr_ents; + unsigned short old, iter; + struct swap_cgroup *map; - sc = lookup_swap_cgroup(ent, &ctrl); + ctrl = &swap_cgroup_ctrl[swp_type(ent)]; + map = ctrl->map; - spin_lock_irqsave(&ctrl->lock, flags); - old = sc->id; - for (; offset < end; offset++, sc++) { - VM_BUG_ON(sc->id != old); - sc->id = id; - } - spin_unlock_irqrestore(&ctrl->lock, flags); + old = __swap_cgroup_id_lookup(map, offset); + do { + iter = __swap_cgroup_id_xchg(map, offset, id); + VM_BUG_ON(iter != old); + } while (++offset != end); return old; } @@ -85,9 +100,13 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, */ unsigned short lookup_swap_cgroup_id(swp_entry_t ent) { + struct swap_cgroup_ctrl *ctrl; + if (mem_cgroup_disabled()) return 0; - return lookup_swap_cgroup(ent, NULL)->id; + + ctrl = &swap_cgroup_ctrl[swp_type(ent)]; + return __swap_cgroup_id_lookup(ctrl->map, swp_offset(ent)); } int swap_cgroup_swapon(int type, unsigned long max_pages) @@ -98,14 +117,16 @@ int swap_cgroup_swapon(int type, unsigned long max_pages) if (mem_cgroup_disabled()) return 0; - map = vcalloc(max_pages, sizeof(struct swap_cgroup)); + BUILD_BUG_ON(sizeof(unsigned short) * ID_PER_SC != + sizeof(struct swap_cgroup)); + map = vcalloc(DIV_ROUND_UP(max_pages, ID_PER_SC), + sizeof(struct swap_cgroup)); if (!map) goto nomem; ctrl = &swap_cgroup_ctrl[type]; mutex_lock(&swap_cgroup_mutex); ctrl->map = map; - spin_lock_init(&ctrl->lock); mutex_unlock(&swap_cgroup_mutex); return 0; From patchwork Wed Dec 18 11:46:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13913541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 337E9E77187 for ; Wed, 18 Dec 2024 11:47:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 984C66B0093; Wed, 18 Dec 2024 06:47:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9314E6B0095; Wed, 18 Dec 2024 06:47:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AB586B0096; Wed, 18 Dec 2024 06:47:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5C0796B0093 for ; Wed, 18 Dec 2024 06:47:09 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E3D0080D99 for ; Wed, 18 Dec 2024 11:47:08 +0000 (UTC) X-FDA: 82907902980.15.E5095CA Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf22.hostedemail.com (Postfix) with ESMTP id B891FC000C for ; Wed, 18 Dec 2024 11:46:33 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Zyhwkjdf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734522397; a=rsa-sha256; cv=none; b=iPeknjwp++zdCRYW1VI6mlmRuKrJFS71UHItkUJ6SU/7QkM/+H8KeHYasFUEaSJ/0W4u5v tO6L7jHIw/pgehJy06jfgdXall8ZnLBlqDfksm6UU6kWewEiE8aHFGoRsXSRCBIuktr1I4 tbFeM1WIfO4vMdVm1ND+rVnDF3/LRH4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Zyhwkjdf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734522397; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e+ni2sm3wk+IEpHJsvCxGYROd6D/iUDHnxMXaHI/HCw=; b=5G9vo0H8Mr5uCpZw261sAFkKStQJJH8Q0Kc1DBsIfMq3whqOJO/sLWx3RFvCRywPYOt0Ah wBBCg8QOoyXOV5sy+v0gqDooy+EbhRi6M8UYPF1lcr95oNpfOWHwzQTIxqmdCcEjnn/kQ3 AcFbo3r5X7J1oMwLp5Ic6HdQNlnQkCQ= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2165cb60719so51246555ad.0 for ; Wed, 18 Dec 2024 03:47:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522425; x=1735127225; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=e+ni2sm3wk+IEpHJsvCxGYROd6D/iUDHnxMXaHI/HCw=; b=ZyhwkjdfO/FSDDPdhe59ef6anX70X+dTfstWgirQhAHgHP6lIb8yzxP9tjpxe1nPy7 TkOEXVIgWAMq4+wx7euxGS0oB84nZpKK/DVoS6sDzS7b18bRgZMLQyTZ9cXfOpD2ll4U WG7G4RjFhC3VXp4WwGWQEhNc3MfeZNL6kbbVwM8GhgpxUJznl24PzgUGHWP1elULT7Cf imlHFAvCKq8r8cjAfH0Zrik1VuP9MD1N10MXGWoTu/0mL8fOOtFDTlnCO07sguEluJsp 3OKMUjKBCB6IA34bFZTTw1InSyswURAU8p3ZyZDVSRQwfycN5Qe8eFqcGnuec7xBHOPL uSeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522425; x=1735127225; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=e+ni2sm3wk+IEpHJsvCxGYROd6D/iUDHnxMXaHI/HCw=; b=uuxlfts88uLDcIKPSCYuLN0Wa0a1hC/oWeD6P5OwPzwGYft533zQ3z/D51toYR6/Xl AFqV7kKPrICQBMFLznQU7dFfqxD7GMuuROl6I2iOjGeBMNjl9Sr9ZVxweumynaCW+mEX Vm/Dd1AYazx1NF+pM0o9Yqqh4FF4Bwk+AkIR+g4xUQL2GAEiGA/hgcLCCyPaq1F3CJTg Ejpy8fSEqe4EbLf8YyfG0T3/Ute2xPNYvJ66oJ4tzOg4Q4dJPJUE8HZYnQlbZRgc/2P8 JRPsa4B8C0EaAOVPSvGJXCNSyezlCh5ZA4LC3UmR0vNiZLzbxlOGrBpMSfEzbIGHSBDE Rl7g== X-Gm-Message-State: AOJu0YznVMQMTLSud4SH2XL6BogUUaJH/CF64HW2Xzr1wdpEpINQanlt QB/r72mksufNnRrnbEELustJEV8NMvZYyV8aXrj8h1DznHQXsqHfOxvatA2r X-Gm-Gg: ASbGncs6f5LkPEKaEO081yYucxfbRuQGTdk8nv7b9wfhMiiNGrBitm2ZP/5SVcVuUxq qd358K+YygUoRdwn2psiYuV64B5k72zi45LfvZ7uCdiTNDvc/pTd+dYqI/hg0RE9RiYPoT7a3UT Kmtq2RGRxEHIbBz0iz+FMKRrTV1N6rfX9/03WPotF4eDoYLQ7oB4l5esWdBnTFX0w/vkQ1Ptvzg 5dGVejaCQL8hzhZ95IuGN9W+T1kRWCiS4e3cB/cAQlrppDzqfibyKEYc0DR7P4IE0AUNAA5q2Lg PUkOXrfvtr8= X-Google-Smtp-Source: AGHT+IFfwnmvDXUoevR3JMwJ83hwhWaRbCVWe3lKgTz6e75Uv7wy2N9Yq4LbPCY/3wUaTXBc06OKTQ== X-Received: by 2002:a17:903:183:b0:215:9894:5679 with SMTP id d9443c01a7336-218d6c4bc26mr35002555ad.0.1734522425286; Wed, 18 Dec 2024 03:47:05 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.47.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:47:04 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 4/4] mm/swap_cgroup: decouple swap cgroup recording and clearing Date: Wed, 18 Dec 2024 19:46:33 +0800 Message-ID: <20241218114633.85196-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: tuw3pj4mshj1ruckry4dx5qyrjhfqof7 X-Rspam-User: X-Rspamd-Queue-Id: B891FC000C X-Rspamd-Server: rspam08 X-HE-Tag: 1734522393-440080 X-HE-Meta: U2FsdGVkX18D3eqWlPFcKmvUlx6NbBw3PbfuVJYtaUkLAzUnH1MgONZBpuS2M/JvkCV56ijbgtUUNcbWMD8s3RJ22A+Y7SzUS/YN6hD3ZdE99j8IsBMOqB95PHXfHoB1hI7p5WoeNoujYIysyOmG+lfvFjr24Jx5pbKNqqd51qMqYJgAMmHjfsyKfScecwXervnwp0rRblMegemCYFygLsw/ZpAtk8NPCziWrmbWaxTl0dswVzZhHc6py304tG7A25SdC2DognwnGf3r9TdpUlIYu+vdk6LphQc6pmpEuI3GxGjzoCLrmOQOMQGXhjaBF/FSA/LSMi5oVyxFtZJL4heNC+b/DFq8fSFGb3AZ2Cz0fiVDWPuOo1tkQewThBS7PvdAX7NrVeius0dXSbWkcCtrT20Rc6bYgkqGEVDVhiC7OJHJ1Teb2LDWhnMgXNT8eFxe7MAYK2h7wLLcFZeVpIJLFterF5375FZ7Q1abZuD6xowUSn8gTsFmcrMOWYHcQO9+ZOwpHUFA85OF4EmgLJCw+BRx1tXf2wLRMJr4h7EOQc3NphWvUFx0IKFA+gLnHmMCVulQZh0pDCBc1W6NcCsa9BmDD03wCbBoH/B6q97mGtyAtslnErIPpn5EY/jTUabMCV7pZdDwmJzjKD06OJy6gMeKTWGWDzmxePHnKHg7khFsJLd+2//QyYY2Yg8mqi5HBM88rcycxDtW8spOjXqoDi2ib9dh73gEkBVRBYA1uK3GLtcHrLUcHajxua4ZTBReNsBYyqe9jgdfJMr6GLr+yqCj3LffJ3YbNbgq3dtbBwD/xgJOYSdWSi19jBwYARD+px/kYZcoTbvR0ZqQgdhsg3sAh8HRtFNnaEbQIMlzYriGY0Hru4Q2eIaEZQn8/mtdfTdbFB1O+dUgfdlrOUvrEJ50XCfEC6LJrEXJJ800H1+QIU0AqlnE7Y1Wa1Ak2EHKOuN52adXMRg9dCo Q7g2KKac QDm123JJyb6D7nowLjNsZDiQfa8FDAqeym9LTefjGq8NJGBsXOUGebMhdPxJ4UEy0s5yRHwJT+2e2cSQ2UlEhbirhzZonCoDWjJO4LH0R/DDJNMbZCpkARcEGYnUFHU+IM5a7t0LGXxTFSldferV8NjJdaziunSYR1Ko0lBIeO2rmWY1tguP/asY/ofAzfc8rKb+UZXXd0HWGnczoXxXojk5eFmRZVt2fWpZe7dd+5KHzQ29n2sLDfNxq+bcUXXZCL3c0kd9t74TioPVosuUvNM5VAT2c10AO7Zsm5Cqkc7cJSu/QIOXkJdJlXYeWqVgn3yhksOxyDijzOgyO15cxOZN/q6+UCRoeH0IfN5RyYClvWENsCPjxC+Rt9bQD7ruLJJpArHACGxtVdfV+XTcFBEgWSkkd+Dr0pR0fphqI83r3P63CBQVRdNa/o2l4XdnZA0OGK5H/98Oqk+oXFvahixHsVvYish7qRokdkivf10BB80pzsvuGVbsV9sk69gz55DEQPhKK6j6CJ2cOxCfiYPXqBR9mRu6M+1US0mNsEJlEuXxBgzI4MbOh2CWWzoo/CbRjFL5yv6bWFbMriaKgxHpvzVB1qWwWgTQ8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000009, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song The current implementation of swap cgroup tracking is a bit complex and fragile: On charging path, swap_cgroup_record always records an actual memcg id, and it depends on the caller to make sure all entries passed in must belong to one single folio. As folios are always charged or uncharged as a whole, and always charged and uncharged in order, swap_cgroup doesn't need an extra lock. On uncharging path, swap_cgroup_record always sets the record to zero. These entries won't be charged again until uncharging is done. So there is no extra lock needed either. Worth noting that swap cgroup clearing may happen without folio involved, eg. exiting processes will zap its page table without swapin. The xchg/cmpxchg provides atomic operations and barriers to ensure no tearing or synchronization issue of these swap cgroup records. It works but quite error-prone. Things can be much clear and robust by decoupling recording and clearing into two helpers. Recording takes the actual folio being charged as argument, and clearing always set the record to zero, and refine the debug sanity checks to better reflect their usage Benchmark even showed a very slight improvement as it saved some extra arguments and lookups: make -j96 with defconfig on tmpfs in 1.5G memory cgroup using 4k folios: Before: sys 9617.23 (stdev 37.764062) After : sys 9541.54 (stdev 42.973976) make -j96 with defconfig on tmpfs in 2G memory cgroup using 64k folios: Before: sys 7358.98 (stdev 54.927593) After : sys 7337.82 (stdev 39.398956) Suggested-by: Chris Li Signed-off-by: Kairui Song --- include/linux/swap_cgroup.h | 12 ++++--- mm/memcontrol.c | 13 +++----- mm/swap_cgroup.c | 66 +++++++++++++++++++++++-------------- 3 files changed, 55 insertions(+), 36 deletions(-) diff --git a/include/linux/swap_cgroup.h b/include/linux/swap_cgroup.h index d521ad1c4164..b5ec038069da 100644 --- a/include/linux/swap_cgroup.h +++ b/include/linux/swap_cgroup.h @@ -6,8 +6,8 @@ #if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP) -extern unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, - unsigned int nr_ents); +extern void swap_cgroup_record(struct folio *folio, swp_entry_t ent); +extern unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents); extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent); extern int swap_cgroup_swapon(int type, unsigned long max_pages); extern void swap_cgroup_swapoff(int type); @@ -15,8 +15,12 @@ extern void swap_cgroup_swapoff(int type); #else static inline -unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, - unsigned int nr_ents) +void swap_cgroup_record(struct folio *folio, swp_entry_t ent) +{ +} + +static inline +unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents) { return 0; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 79900a486ed1..ca1ea84b4bce 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4973,7 +4973,6 @@ void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) { struct mem_cgroup *memcg, *swap_memcg; unsigned int nr_entries; - unsigned short oldid; VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); VM_BUG_ON_FOLIO(folio_ref_count(folio), folio); @@ -5000,11 +4999,10 @@ void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) /* Get references for the tail pages, too */ if (nr_entries > 1) mem_cgroup_id_get_many(swap_memcg, nr_entries - 1); - oldid = swap_cgroup_record(entry, mem_cgroup_id(swap_memcg), - nr_entries); - VM_BUG_ON_FOLIO(oldid, folio); mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries); + swap_cgroup_record(folio, entry); + folio_unqueue_deferred_split(folio); folio->memcg_data = 0; @@ -5035,7 +5033,6 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) unsigned int nr_pages = folio_nr_pages(folio); struct page_counter *counter; struct mem_cgroup *memcg; - unsigned short oldid; if (do_memsw_account()) return 0; @@ -5064,10 +5061,10 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) /* Get references for the tail pages, too */ if (nr_pages > 1) mem_cgroup_id_get_many(memcg, nr_pages - 1); - oldid = swap_cgroup_record(entry, mem_cgroup_id(memcg), nr_pages); - VM_BUG_ON_FOLIO(oldid, folio); mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); + swap_cgroup_record(folio, entry); + return 0; } @@ -5081,7 +5078,7 @@ void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages) struct mem_cgroup *memcg; unsigned short id; - id = swap_cgroup_record(entry, 0, nr_pages); + id = swap_cgroup_clear(entry, nr_pages); rcu_read_lock(); memcg = mem_cgroup_from_id(id); if (memcg) { diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index cf0445cb35ed..be39078f255b 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -21,17 +21,6 @@ struct swap_cgroup_ctrl { static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES]; -/* - * SwapCgroup implements "lookup" and "exchange" operations. - * In typical usage, this swap_cgroup is accessed via memcg's charge/uncharge - * against SwapCache. At swap_free(), this is accessed directly from swap. - * - * This means, - * - we have no race in "exchange" when we're accessed via SwapCache because - * SwapCache(and its swp_entry) is under lock. - * - When called via swap_free(), there is no user of this entry and no race. - * Then, we don't need lock around "exchange". - */ static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map, pgoff_t offset) { @@ -63,29 +52,58 @@ static unsigned short __swap_cgroup_id_xchg(struct swap_cgroup *map, } /** - * swap_cgroup_record - record mem_cgroup for a set of swap entries + * swap_cgroup_record - record mem_cgroup for a set of swap entries. + * These entries must belong to one single folio, and that folio + * must be being charged for swap space (swap out), and these + * entries must not have been charged + * + * @folio: the folio that the swap entry belongs to + * @ent: the first swap entry to be recorded + */ +void swap_cgroup_record(struct folio *folio, swp_entry_t ent) +{ + unsigned int nr_ents = folio_nr_pages(folio); + struct swap_cgroup *map; + pgoff_t offset, end; + unsigned short old; + + offset = swp_offset(ent); + end = offset + nr_ents; + map = swap_cgroup_ctrl[swp_type(ent)].map; + + do { + old = __swap_cgroup_id_xchg(map, offset, + mem_cgroup_id(folio_memcg(folio))); + VM_BUG_ON(old); + } while (++offset != end); +} + +/** + * swap_cgroup_clear - clear mem_cgroup for a set of swap entries. + * These entries must be being uncharged from swap. They either + * belongs to one single folio in the swap cache (swap in for + * cgroup v1), or no longer have any users (slot freeing). + * * @ent: the first swap entry to be recorded into - * @id: mem_cgroup to be recorded * @nr_ents: number of swap entries to be recorded * - * Returns old value at success, 0 at failure. - * (Of course, old value can be 0.) + * Returns the existing old value. */ -unsigned short swap_cgroup_record(swp_entry_t ent, unsigned short id, - unsigned int nr_ents) +unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents) { - struct swap_cgroup_ctrl *ctrl; pgoff_t offset = swp_offset(ent); pgoff_t end = offset + nr_ents; - unsigned short old, iter; struct swap_cgroup *map; + unsigned short old, iter = 0; - ctrl = &swap_cgroup_ctrl[swp_type(ent)]; - map = ctrl->map; + offset = swp_offset(ent); + end = offset + nr_ents; + map = swap_cgroup_ctrl[swp_type(ent)].map; - old = __swap_cgroup_id_lookup(map, offset); do { - iter = __swap_cgroup_id_xchg(map, offset, id); + old = __swap_cgroup_id_xchg(map, offset, 0); + if (!iter) + iter = old; VM_BUG_ON(iter != old); } while (++offset != end); @@ -119,7 +137,7 @@ int swap_cgroup_swapon(int type, unsigned long max_pages) BUILD_BUG_ON(sizeof(unsigned short) * ID_PER_SC != sizeof(struct swap_cgroup)); - map = vcalloc(DIV_ROUND_UP(max_pages, ID_PER_SC), + map = vzalloc(DIV_ROUND_UP(max_pages, ID_PER_SC) * sizeof(struct swap_cgroup)); if (!map) goto nomem;