From patchwork Thu Jan 11 18:12:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13517687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D381C47077 for ; Thu, 11 Jan 2024 18:12:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C33D6B009E; Thu, 11 Jan 2024 13:12:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 073DA6B009F; Thu, 11 Jan 2024 13:12:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7E856B00A0; Thu, 11 Jan 2024 13:12:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D9AA06B009E for ; Thu, 11 Jan 2024 13:12:35 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id AC81612073F for ; Thu, 11 Jan 2024 18:12:35 +0000 (UTC) X-FDA: 81667825470.20.88452CF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 0CA4E18001F for ; Thu, 11 Jan 2024 18:12:33 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FhyOtFe6; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704996754; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3mQloxYVj81b7FxsG36DgI4RFnukVPFBF5N4HJB/q50=; b=BgINxvnhqnZATaqqtxLCQn3IHZK5kou64Fvz3n5Jo++x4/qUDi9YOStu4FafFK9nzZadAw BPMVp1g1YNKDEeS0WXYQZ6GFd3F5nfZwt/3WH2Qs1RftWpUzmrYH8iEw6FdXkNaKJvpV/n 5Cg6Ho+nJFqnEXhetkjRUZ1HUxDXSG8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FhyOtFe6; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704996754; a=rsa-sha256; cv=none; b=J3gh4sEj8ZdRIRzab1EL8EKOEFL5Sgzu4LN0apjoFaFC+NqiY266/kWGNSx+Lf/nDMJfCp D2V5lemjfphlrLmIa6cwtdNV4b7eQnakuKFodk/xkzF0hKhYmVDwx6DIia4aejjNudZ7Ex JUsOtyYTzbmci40esD+DvrgiLRaZwXo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3mQloxYVj81b7FxsG36DgI4RFnukVPFBF5N4HJB/q50=; b=FhyOtFe6kL8S/UGls9v/7hVEhe hdgLfkztUvUHgUvmV5con/y5NczYd+XVAhQdNT1Ps6UNhmhIMToP/7ZCBLbm85c/p1ula8/ZyNxai Dvte+kg/OeLEbPK2ZcD1/iF10PILenh7orq47CpPVB9Hd4yaZ2kJrtz27GLfHcPDF/F9sPVnqc0ae YIn9jCjGEgVib4OROuAY/oY0Wu4Nnp3vZ5J+FDmfksJJBBKv9Vu/865sBAF4HpU7zO5djGxS6JDpo ksSFpvv5qOkMr97Mj8iLy64ffVZH+aA/DsnaRuQPF1sJQjW+TbAagu/Vp3KlDDPhpbpWMiiV9JhJ9 ohD92AdA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rNzXR-00EWqy-F1; Thu, 11 Jan 2024 18:12:21 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/4] memcg: Convert mem_cgroup_move_charge_pte_range() to use a folio Date: Thu, 11 Jan 2024 18:12:16 +0000 Message-Id: <20240111181219.3462852-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20240111181219.3462852-1-willy@infradead.org> References: <20240111181219.3462852-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0CA4E18001F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 1wf75hc3g46fuqqpwg3awrdcixwz75u8 X-HE-Tag: 1704996753-483923 X-HE-Meta: U2FsdGVkX197iQDxKyPw/EaBDbeJQl1/Oj/QbFEKvl3Nrsz04+RBH8VkTe8Dp/38vGNVaA4oDA9/V94YDO36S3+cLcwJvusDc4IAhzIoTXo7hKVsuNjj+ZwgO/SoRH5eYh5S2hzLK35bcMZ6lCR2JQIDwEcGg6jCSJpKd2I5LHWxI6S078/ktf/ZnJn/3JsqNVCCfIia6JL8l8ZVQQmJUrdyndgkXzrSPDXQ9ROdpkIBKMuOV7yTTvBJuRWe8lK06/Q8HRWeJS3UyeBFplBm3EubvkKuS/AABdDNWMkO9vfkqY4m3OLFg+S95jVbNt0jfarFrcFPUG7Otl+46Qt3LJryjj10MwMqO3Vn/8bksh69ndO5+700XleAbiBhdVPnBXtnIIpR3tS6XGV0r5wzRQqdohvaQG71axmxalJtqYjooWUrbbbWwaCfRuIfBub3bPYooQzNwilx5vmYcbZjL/1yZMBqsOosEgGvM21eTflPI0oluIFzBbuL6Ft8HaXZtgezCjMZyQ3Zsfz0xf/lshT+WEHq2wZrydH6Zpsb7oNfctndZg8NFPNTh0t2C1zx1I5wDlU+rLhPHQN84VDIpRYwEt9xZ/icQm7ejWyWyMt9dtBhZvFaJLmBEzL+OXVc18lBrazU9aOw8RKM0HgNLWu+ziJVOpo7NFX/02f39kJnFqILVoPMjGqSEz7RvrTAzdeQhOQl9kBXLEv5vya2aZywsRtf6m976CqW+kOkdFAMSPRYocdbFASOZ6s33I94rQAyzh4JDRJ7cvSZ7QTHuS35bXQeSXH8dCAMp5Hv72p8e2vXYr3dsTZfKHHa+u6bFvJ5ZVgvdYS/7LTJ4IJFp5jecDJ0ZqCA4MBp8g3GfCOrDbcadDUKLF0I2FeZvKSZqOzOLALWxqeaIMRcuxnO6xBTecpxuv3cEf0PRmGy1tvYugAFhBGTFonX5r57gEz8+XJgBYdJc+t8vNZ/SyD cIJ1eZYJ NABQ9cMpSRgymvKpXJK2jhGb1C9kDkuPyINtfrhCb08Wa5XEfibbFDN2DHnwI7bPRjjurIuND1+HeO3osu4f6S5xLTrnH/ea1PtmPUUMmF7qskBqASUBWbenzumPWO10zhXdHBg+jI/E6HUnLiU6gI6ZH+OlfQCdqHCl1DYbStJHe2S9GyvYV+GmP4Hg99PiRdPMhsbLjl5bC4ltwVkLN2v0UKElpNyD0HgaF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove many calls to compound_head() by calling page_folio() once at the start of each stanza which receives a struct page from 'target'. There should be no change in behaviour here as all the called functions start out by converting the page to its folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Muchun Song --- mm/memcontrol.c | 49 ++++++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c4c422c81f93..c04bda961165 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5932,23 +5932,22 @@ static struct page *mc_handle_file_pte(struct vm_area_struct *vma, } /** - * mem_cgroup_move_account - move account of the page - * @page: the page + * mem_cgroup_move_account - move account of the folio + * @folio: The folio. * @compound: charge the page as compound or small page - * @from: mem_cgroup which the page is moved from. - * @to: mem_cgroup which the page is moved to. @from != @to. + * @from: mem_cgroup which the folio is moved from. + * @to: mem_cgroup which the folio is moved to. @from != @to. * - * The page must be locked and not on the LRU. + * The folio must be locked and not on the LRU. * * This function doesn't do "charge" to new cgroup and doesn't do "uncharge" * from old cgroup. */ -static int mem_cgroup_move_account(struct page *page, +static int mem_cgroup_move_account(struct folio *folio, bool compound, struct mem_cgroup *from, struct mem_cgroup *to) { - struct folio *folio = page_folio(page); struct lruvec *from_vec, *to_vec; struct pglist_data *pgdat; unsigned int nr_pages = compound ? folio_nr_pages(folio) : 1; @@ -6398,7 +6397,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, spinlock_t *ptl; enum mc_target_type target_type; union mc_target target; - struct page *page; + struct folio *folio; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -6408,26 +6407,26 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, } target_type = get_mctgt_type_thp(vma, addr, *pmd, &target); if (target_type == MC_TARGET_PAGE) { - page = target.page; - if (isolate_lru_page(page)) { - if (!mem_cgroup_move_account(page, true, + folio = page_folio(target.page); + if (folio_isolate_lru(folio)) { + if (!mem_cgroup_move_account(folio, true, mc.from, mc.to)) { mc.precharge -= HPAGE_PMD_NR; mc.moved_charge += HPAGE_PMD_NR; } - putback_lru_page(page); + folio_putback_lru(folio); } - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); } else if (target_type == MC_TARGET_DEVICE) { - page = target.page; - if (!mem_cgroup_move_account(page, true, + folio = page_folio(target.page); + if (!mem_cgroup_move_account(folio, true, mc.from, mc.to)) { mc.precharge -= HPAGE_PMD_NR; mc.moved_charge += HPAGE_PMD_NR; } - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); } spin_unlock(ptl); return 0; @@ -6450,28 +6449,28 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, device = true; fallthrough; case MC_TARGET_PAGE: - page = target.page; + folio = page_folio(target.page); /* * We can have a part of the split pmd here. Moving it * can be done but it would be too convoluted so simply * ignore such a partial THP and keep it in original * memcg. There should be somebody mapping the head. */ - if (PageTransCompound(page)) + if (folio_test_large(folio)) goto put; - if (!device && !isolate_lru_page(page)) + if (!device && !folio_isolate_lru(folio)) goto put; - if (!mem_cgroup_move_account(page, false, + if (!mem_cgroup_move_account(folio, false, mc.from, mc.to)) { mc.precharge--; /* we uncharge from mc.from later. */ mc.moved_charge++; } if (!device) - putback_lru_page(page); + folio_putback_lru(folio); put: /* get_mctgt_type() gets & locks the page */ - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); break; case MC_TARGET_SWAP: ent = target.ent; From patchwork Thu Jan 11 18:12:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13517689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB7E1C47258 for ; Thu, 11 Jan 2024 18:12:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 35C476B00A0; Thu, 11 Jan 2024 13:12:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 30E256B00A1; Thu, 11 Jan 2024 13:12:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04D866B00A2; Thu, 11 Jan 2024 13:12:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E92B26B00A0 for ; Thu, 11 Jan 2024 13:12:39 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C80BA1C0E04 for ; Thu, 11 Jan 2024 18:12:39 +0000 (UTC) X-FDA: 81667825638.05.72B2F22 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 1FC408001E for ; Thu, 11 Jan 2024 18:12:37 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="V8/B94yi"; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704996758; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ckLkDfpTy5X71kCDKuO9wk6WU/1baQA9VkLceFKNXrc=; b=X2WE4SWwWQP/BivspxB3CAfyi9yYK6XvM6ZUjTcOHEBuLsHPx5w2JcwR3HK/2ZQLaHpIpS OaYEs2vBA8z8EP6bQdq5KGLL0C4EAFSo6fW6mTgjI7HkibRCVoKVDLWOfCo47OaSwWrjgb NNHxYTUb/sg1LwJ2N2VpKk5NZ3xrER4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="V8/B94yi"; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704996758; a=rsa-sha256; cv=none; b=QzI0Q6AsI2TB5OCfk5tboDVjjc6eregi9cGq7UxmSfmdei2QHebVRfDMQz3/UTO/gte8Jo skuP1poMPuAe6vTPIW5TBtTlsxISFgmSbCz701ipcaVJ7aWwM91BemIGBDE8flRAc1eAOD h10WAU13zz1xvk3NDYvL0AOpuH24KOs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ckLkDfpTy5X71kCDKuO9wk6WU/1baQA9VkLceFKNXrc=; b=V8/B94yi0XWKJBeBSphEHnJnTP OPrAAc6rjc8oO7WITgFGTDrIkp0HsGBGgRO91ipoC07vwDNLDpLV2ml06fcYpqsaCFqDexdsbtM6Z vd+jcROAvdMoq045wg0zd7FjgflmQ0nvNj+HvVHo3lU/GzL6hAaQENEkYs2tIQbflrMb9oMsqdYif kC6jG0PMTfe/DF1zm3H5Ke2jeYCSWlNoVvRU3KcuX3yqc3q1Lh991huh5e4kmgoDPCgCXbWKv8BiN 8otdL6KQiHaqZfrDiYhSAKo0nYrnLoO2BemF1JCQ2EKJVuFkjviVrJcxJ3gXXRf3ITzpdinrjHfrm GPkj9dvw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rNzXR-00EWr0-Hx; Thu, 11 Jan 2024 18:12:21 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 2/4] memcg: Return the folio in union mc_target Date: Thu, 11 Jan 2024 18:12:17 +0000 Message-Id: <20240111181219.3462852-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20240111181219.3462852-1-willy@infradead.org> References: <20240111181219.3462852-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1FC408001E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: phq8uq4d9zn8cbuk1tzh4zen1fi6dw6r X-HE-Tag: 1704996757-505949 X-HE-Meta: U2FsdGVkX1+NUVyCwNvkFPZccbBd5s99jjCS3/N0190qPW997XBK/cEXDNGESLqT6WZzarTTn20dYaPhf479D9m/umr1QtEoAgv3P48GmctkbAFfkzeaHaGqGCxAm+cz5a/QsZunrxKr/lHnt/wPSA9xcmmYL2Fv4Jx3bGsrNMWpTLZFG/WbXcPdYtihUPw6+peKq46f9PPv5jnRwhI6YAuLe574ewYSrFo7RHeAxS+2VGE1S4M0lkPKmtxxhZsTQFkSAFp4wcfewm0CWEDuvX5JsfRGwYoRIbdlQneGySJhBZyhNGzHOywspPWVaxLTOMx8sHrgCtTsF87YEfPcrmTzILQpdsK3Ap3HOpiRaPY4rgaP6JbrAk0hh2Hz+0zRikA1HD0xDuV2252Tyn7qOkSTjHeI0fvckwMIk4prSaO+VddrWa3Zrh/95qauD+ns+pKhllZ2rVOIKeh/oQ7V97eNiXxFGbeiUyWODSNAERDZQ/2Hmg60m2xAodiCNCANCEvogfNw9i5l4e3/ONf+fSnQlqMZSGifyLHjIN7wzqTM6YOxc4sdoQlMFrjq58VSjpSxNAQ4i5IkSFuYTCM4+uM+pj7RVFnUlkCq4KBj34E2aExXJenWQv9/mzjD/bqA5Lq5QIVjxPV9UR1cjPTJbevsA8DhPePgqNB6WFCOi8haWI+/3E+xp2y79LXkNeqN2a6HnPYXkRvvV6LJ58ZPGC4UjaSNjQe08arpApuvom56FiKvZQLiiXebFvdLMCc94SmB1/AN1IW/jsHuMu0xALIaa9eq46TXai8sDYknntYGSO1OYpntgRgMTrCf+7YQpzVdljQ90PRSJ+06E8UBVEi8SM0z4izsU85K7glJRCSyfHM0sa1hoYrMW18DD6slvhBEr9TB0ghCSYtfDx2hdRO2mC0jG9rafdC/b+nagwiKOrf3ElnGPv/TjvS0z712Lfz0TDp6z6dmXU4csG1 gWu+PlS7 M4Z1DpUn2wVDuReQqWLrFpRQg8VVHGnbocQtXYWR0YYekhL51fTM7aIDUqere3z960lvR7nqB121gPi3JtqKThky9Aj0GpK/ScxuZavlPUCKvz5ud9XUITFaxilesHUmo78d5FDwa+0fgVVrCT8dvHRnqZFE5AdYoMXS/YdSRaAwbxTKCibOrJusrOwkU2EtanpSxCQC9Q9zVOO7pHet8/eJZsX2us0XyErIl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users of target.page convert it to the folio, so we can just return the folio directly and save a few calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Muchun Song --- mm/memcontrol.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c04bda961165..d14fe0740b37 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5840,7 +5840,7 @@ static int mem_cgroup_do_precharge(unsigned long count) } union mc_target { - struct page *page; + struct folio *folio; swp_entry_t ent; }; @@ -6062,7 +6062,7 @@ static int mem_cgroup_move_account(struct folio *folio, * Return: * * MC_TARGET_NONE - If the pte is not a target for move charge. * * MC_TARGET_PAGE - If the page corresponding to this pte is a target for - * move charge. If @target is not NULL, the page is stored in target->page + * move charge. If @target is not NULL, the folio is stored in target->folio * with extra refcnt taken (Caller should release it). * * MC_TARGET_SWAP - If the swap entry corresponding to this pte is a * target for charge migration. If @target is not NULL, the entry is @@ -6127,7 +6127,7 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma, is_device_coherent_page(page)) ret = MC_TARGET_DEVICE; if (target) - target->page = page; + target->folio = page_folio(page); } if (!ret || !target) { if (target) @@ -6177,7 +6177,7 @@ static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma, put_page(page); return MC_TARGET_NONE; } - target->page = page; + target->folio = page_folio(page); } } return ret; @@ -6407,7 +6407,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, } target_type = get_mctgt_type_thp(vma, addr, *pmd, &target); if (target_type == MC_TARGET_PAGE) { - folio = page_folio(target.page); + folio = target.folio; if (folio_isolate_lru(folio)) { if (!mem_cgroup_move_account(folio, true, mc.from, mc.to)) { @@ -6419,7 +6419,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, folio_unlock(folio); folio_put(folio); } else if (target_type == MC_TARGET_DEVICE) { - folio = page_folio(target.page); + folio = target.folio; if (!mem_cgroup_move_account(folio, true, mc.from, mc.to)) { mc.precharge -= HPAGE_PMD_NR; @@ -6449,7 +6449,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, device = true; fallthrough; case MC_TARGET_PAGE: - folio = page_folio(target.page); + folio = target.folio; /* * We can have a part of the split pmd here. Moving it * can be done but it would be too convoluted so simply From patchwork Thu Jan 11 18:12:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13517691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84ADBC4725D for ; Thu, 11 Jan 2024 18:12:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 950D06B00A2; Thu, 11 Jan 2024 13:12:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DA746B00A3; Thu, 11 Jan 2024 13:12:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7794D6B00A4; Thu, 11 Jan 2024 13:12:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5F61C6B00A2 for ; Thu, 11 Jan 2024 13:12:42 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 38B09A227F for ; Thu, 11 Jan 2024 18:12:42 +0000 (UTC) X-FDA: 81667825764.08.6BFD9A3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 8E602A0027 for ; Thu, 11 Jan 2024 18:12:40 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NzLVjgjx; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704996760; a=rsa-sha256; cv=none; b=BtVWxpB8Rjw+fYOUNd0raC6+rYEvmEhLdkpye8aAn/uA0ZN36GbFiZg8MyBErtwo3zFBfE DfOx8MQvGoYtUYiTozil5pvzMx0kBg7IzuoRd80yemyK/d7HXZ328JzUpP84DtBSY6At1c 5J9ybrifxhPucSd0EWvaWf3XlIOnk5E= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NzLVjgjx; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704996760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nikFvHd6fGdNzCCFcNOYGz8RI17m7m/j0Xdu8XjTLzg=; b=Pi//VzaCqHtGW7qDvqiBOg7ubyW9b5KSx0SYAYefx9iYse4NY38XNh+d3P7q8/9lO5fS9f ivwrluUZ7qlFe1oMU7LYei8GJ6qVq/cINVnddB/iUbuV3N+cj+iWyWM9Zv0mfbDd3cJgGE yOTOUhoOcBBJgdgKRd4IDISs1MV6694= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=nikFvHd6fGdNzCCFcNOYGz8RI17m7m/j0Xdu8XjTLzg=; b=NzLVjgjxox29jd/ntluwYpxv4D bVQK+95YJVxZUge/U0z3zh9L1AJkU+dtkQI0vJBHiQXWP7Od0DP/G9gDUXy87CxK3KgNxdSPTT5HR 4a1cml7L8wwn7+lcP+W5YXasDVWmtYm7YGb+S8SvcEK4vzPQgi1s8UNicyRWgpLnwwbpKMFpLJmdD KAYRPeAeqb5unTOtACQs9g9kjsUq7eGxUXvrc2QLo9HMvmTSHOmpp0wg2x87GmWZ1ORqHaqRYi+Uq 3XTcUNqJE/iRjVjPej2pgW4RSTr1eQamHjSbX5fI9IRUQ4gm3+W6wnez2+YYyTf7AJ/ACmR4HByoe kxmGEwgQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rNzXR-00EWr2-Ke; Thu, 11 Jan 2024 18:12:21 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 3/4] memcg: Use a folio in get_mctgt_type Date: Thu, 11 Jan 2024 18:12:18 +0000 Message-Id: <20240111181219.3462852-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20240111181219.3462852-1-willy@infradead.org> References: <20240111181219.3462852-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8E602A0027 X-Stat-Signature: hnqe96q7ejfahzezrdj8bakgpkfbxu78 X-Rspam-User: X-HE-Tag: 1704996760-883216 X-HE-Meta: U2FsdGVkX1/9ADLwg+GkM2F6sYN2AzuK7QtDCxhkVZfhvimFynvqY25JFVTGvGSTSgBkzMcPmK+bTf3Tk1QZQUdPuqFbhJC5IHpScmJc7ciICviGhQ+1Rx7soxI9QHDrqAxcpqFBdqptsOZEd2QWa8hCB4PYVF3CfVdAd/NoRLXThGv8hZ6NAcbLd/ers/bnoyBNPNQ5LrB/zp/J7wZcMZpXw9TzYu9e3WToKIwUYenVxrIFRDO3EIhcMvMkH3nzheriIzSLG4lfq2eOzujYxTvug10yIBN3FS1z09f74aT4FF6XyD6mPm51vjjxN/ztL7pdi/MuRuy+oZttLr9FncieoCuv+zAv+VBfRJ80rWk9CrjTQalYnKP1xiXiFXnxYu4VKoVPGMjyZAjDxrT3bE1cUKkrt15wFsl4I+CU3FUxsQ/qeFGkeIUwKPZzdclEaXiCbpQWNCrbq4vlJWEiEh2+VtF2jDeFpKGP12puT/pf27iwn6UP3DnQtoo+x4pMTyM1q4e8nTBidzsz5ntLkD8RCh+NPqFm+ausXLVk5jgQ1O1qSJWjr8Qe3DFEevI6Q/9uIuSp6qMff2f81BNTD5KGmWLgyNeXK8evdLuFcXfXeN0Vqmt7QWlfTZPvPaUhBzOc/bHFVRoMXAMz0aBXVDqvel8FVHocpzGayVfoGKvFEQUEagioOgluHtHm8UgyRdIdvr/c7zquzAznuokISgLsqNEJW54f8A7zYL58m5savB570cxNbJ90znkT4fs4HKyY2S+9m00tf19JdgTu5+LFt47oFlvdoitDdlARl4NQ85qhcaoR6G3H531XkyzLwo2ev8F9fAFncxqJCa4Zb1hIt2fzI2ZkCnB6WU3qg4SSaE7v1psmHMtemXyo3zKbSGScBhzT/ilUP278xMdQO3exAYu2BZ0MtgpQn7mJQ5da/D3QO4ft1nkMcsFVV0hZhZw6sVHTGnb4ZzSCe+i hbHao1vN Iw5jk1NN+zWqJc0WXXo//VSU/r7m6qpIIuMg6CJISSpe0cMBKDKtlFskuthDzI9Uma4G96OK3gJ77ucKy+/6poiaj0KgVFw8JeoiGgHPAYjz5o0QJBedLHl/yLOycMTSDfBC2w9/Y01f0aaqzgnf9tj6lfeQkdq9BSDnOiMaHFESQtVWKOO/tHjGvD+cTjg83o9MMcyH6gWYgIBBi3Z7/bRNxAnkOy+6g85nG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Replace seven calls to compound_head() with one. We still use the page as page_mapped() is different from folio_mapped(). Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Johannes Weiner Reviewed-by: Muchun Song --- mm/memcontrol.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d14fe0740b37..b6096c34b3e4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6076,6 +6076,7 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma, unsigned long addr, pte_t ptent, union mc_target *target) { struct page *page = NULL; + struct folio *folio; enum mc_target_type ret = MC_TARGET_NONE; swp_entry_t ent = { .val = 0 }; @@ -6090,9 +6091,11 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma, else if (is_swap_pte(ptent)) page = mc_handle_swap_pte(vma, ptent, &ent); + if (page) + folio = page_folio(page); if (target && page) { - if (!trylock_page(page)) { - put_page(page); + if (!folio_trylock(folio)) { + folio_put(folio); return ret; } /* @@ -6107,8 +6110,8 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma, * Alas, skip moving the page in this case. */ if (!pte_present(ptent) && page_mapped(page)) { - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); return ret; } } @@ -6121,18 +6124,18 @@ static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma, * mem_cgroup_move_account() checks the page is valid or * not under LRU exclusion. */ - if (page_memcg(page) == mc.from) { + if (folio_memcg(folio) == mc.from) { ret = MC_TARGET_PAGE; - if (is_device_private_page(page) || - is_device_coherent_page(page)) + if (folio_is_device_private(folio) || + folio_is_device_coherent(folio)) ret = MC_TARGET_DEVICE; if (target) - target->folio = page_folio(page); + target->folio = folio; } if (!ret || !target) { if (target) - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); } } /* From patchwork Thu Jan 11 18:12:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13517688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63D91C4725D for ; Thu, 11 Jan 2024 18:12:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECF086B009F; Thu, 11 Jan 2024 13:12:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E7D7D6B00A0; Thu, 11 Jan 2024 13:12:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1E7F6B00A1; Thu, 11 Jan 2024 13:12:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BF7C66B009F for ; Thu, 11 Jan 2024 13:12:36 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 668B680CDB for ; Thu, 11 Jan 2024 18:12:36 +0000 (UTC) X-FDA: 81667825512.13.E9180EE Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id BD63CA0029 for ; Thu, 11 Jan 2024 18:12:34 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Dly8SJF3; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704996754; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AS+uT7usg/s9J/TDR9k6OcZVDPC0GngCIBakXKVBtYw=; b=5wY0wbV5Pk1I+5Rtr9LJnpJVs3rk0TZABpfq7ugj8y44j42wG1bB0O/Y8wifuUI2MJElQc VDTKrM+vaJkw9fLS87ZF08QHXT8iEyTZzkeXd6GpBfjNWvmsKCVwl1aexNlXlHjNZ0oIsE YUrOaxVwcuHXsFyoi08FI2n+cBonvxo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Dly8SJF3; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704996754; a=rsa-sha256; cv=none; b=fUjfUh6ZkXFEQNki9nYPhwnO0jBAlMMJizG7MbwzGIhVZCfEO7X9urS9qkR1XPXBaDBGHv eKl6CXNpHaej2IYV2vUtsNwbjaEL0TXkszLQBv9hxeDe9XLi8X+YTDLh5AIxGHvO/HtM// FeVuyfV0X+b3qQRWPNHEfLoWp+z4bA0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=AS+uT7usg/s9J/TDR9k6OcZVDPC0GngCIBakXKVBtYw=; b=Dly8SJF3pkBydTQIAvm2nT5dA4 9Aq4dmuKHgGbPIE4kTwz6uoNMZl+DHU7IMGFPey88/uA1ij73+5/j0Ui5a43nr65m8kLyZ1LM7acB R2b7rD/EoU7FfPOShDM472o4/xdGeLbM/29Hi/BZczIe5rnh3n4vU9ygevgY+uC6bZ0/B5iDq3Wya QQmnVSFYLa+KRlGpgrVYbhkHDTZTnCp3ufFfm1yxTq68OiFZqYdwWuVUKxhpAJ6ve9B7heg0ghmhd e3RIXQkaboHVKAyfuR+awVljhPAuXzoPlt6d/Z6GZC0ZScBFhO+Qf2RNxscxDgMzHwx0pUHD+tqjw wF+oOoTg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rNzXR-00EWr4-NM; Thu, 11 Jan 2024 18:12:21 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 4/4] memcg: Use a folio in get_mctgt_type_thp Date: Thu, 11 Jan 2024 18:12:19 +0000 Message-Id: <20240111181219.3462852-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20240111181219.3462852-1-willy@infradead.org> References: <20240111181219.3462852-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: BD63CA0029 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: b1575xyxuuyu1yqbaewfrsn946kws6mr X-HE-Tag: 1704996754-148954 X-HE-Meta: U2FsdGVkX19uMhzyGjgKfuFc9IdW+dS91ZUcb4DF9L4o/qnk7vbzckiS1v03P114Mu584NZrVLJtuOyHUakem9sULYTU2RoFQNurDTa5O6RcS0FYOAJNRfdldoej7uYILzGZraN0ks71u0Qqkq7NyVLnprhDiT1hs3ZUqiSlHbU90mTJ66RzUFCrI/kX1/gb1OgKY7DnWJSWG9Y1N8qokChp+uWio3ej+UH702WMzhWQsrDZLBTzttlOjoL4NBen57uMVchI9rGZMvw2lkwLNyWZ0x9FZk56ECQcLWeBXLJYC9qcvaMBiYGyAEMZ3KHIqwPdEmVQBCWUbxDdgi0JjknPKcdmJd8h0a83zmHmh4hZ5Mj84b95UeUd/O9b6M0XapngViyWhMQis1dThDY1NWwdeMx2ZaXCCjTEwgEpKJteN2rcMTP5j+fFLIl4B+t7on+35VlPw4jWU7azqSVoAKRncnQq8dlKEYFmoHRMA5UyJo6iHJ79zYH1QB4Hq8mMzoQw4BbZ1MyVhNcomqH2wXagm68bDRQ+yhJtcap0o09ru3iqcsHbZ/jlz2hVeFe5O24Sz49BJ+c8OFQBJChIOdoFMZAvUjWsiDihjM6XLFqvKy6TqiY5U2LdG5ciwbxbQ66gaGI1FEj2Co3WU0BlRqFzY1/saU91juD8awuboXxvfyNBmDeR/JSS54C9vM+e3sAs+NDjr6tAVfNG7KdcNnEp2wRqjOhfsPDb0IYoLcdNcH08aGxmw3CsOa+8163HVuQb1ru+gN9bB1mNKwULrDJeHPK2HLHwTJ/FyboSKfwVesWLDkzrfSPD/zWXIZhYNoUInQHYZIzyQ6Ca6P3oL7tIbLVWke0Njaeqx9EEL1PX5/Ha0fCgcH9reRVKucKOilUuOSg8Qrkw3Be4DTP2SbbMAW52z3PcCjcojPpioULzxooB3fJmW0cpCkeVMJJ39jj7eUrARfYJGK/wZf0 Mfp5IzTG GFrX0uC5bP7mDD11hHmm073xmH5o8UwFBVF8YOEsblBtMMQjXqwAVsEIrFcZF4NzRDsgK/Y0L4zyDYtvsvuKOm024uq0+Pe29f0fUtxTzADgYqU7vIgZkbjYG7Z2J5ckngwpWkk5TSKEntmQ7cxYT32LnshKMjmCpWYIGzCXbCCaCPJawmI1gLGaO/GX04hm59vEM++Hf3iOJgfuhlJhen/fq4yeOiisucCXZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Replace five calls to compound_head() with one. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Roman Gushchin Reviewed-by: Muchun Song --- mm/memcontrol.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b6096c34b3e4..935f48c4d399 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6161,6 +6161,7 @@ static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma, unsigned long addr, pmd_t pmd, union mc_target *target) { struct page *page = NULL; + struct folio *folio; enum mc_target_type ret = MC_TARGET_NONE; if (unlikely(is_swap_pmd(pmd))) { @@ -6170,17 +6171,18 @@ static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma, } page = pmd_page(pmd); VM_BUG_ON_PAGE(!page || !PageHead(page), page); + folio = page_folio(page); if (!(mc.flags & MOVE_ANON)) return ret; - if (page_memcg(page) == mc.from) { + if (folio_memcg(folio) == mc.from) { ret = MC_TARGET_PAGE; if (target) { - get_page(page); - if (!trylock_page(page)) { - put_page(page); + folio_get(folio); + if (!folio_trylock(folio)) { + folio_put(folio); return MC_TARGET_NONE; } - target->folio = page_folio(page); + target->folio = folio; } } return ret;