From patchwork Thu Jan 11 18:12:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13517687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D381C47077 for ; Thu, 11 Jan 2024 18:12:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C33D6B009E; Thu, 11 Jan 2024 13:12:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 073DA6B009F; Thu, 11 Jan 2024 13:12:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7E856B00A0; Thu, 11 Jan 2024 13:12:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D9AA06B009E for ; Thu, 11 Jan 2024 13:12:35 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id AC81612073F for ; Thu, 11 Jan 2024 18:12:35 +0000 (UTC) X-FDA: 81667825470.20.88452CF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 0CA4E18001F for ; Thu, 11 Jan 2024 18:12:33 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FhyOtFe6; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704996754; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3mQloxYVj81b7FxsG36DgI4RFnukVPFBF5N4HJB/q50=; b=BgINxvnhqnZATaqqtxLCQn3IHZK5kou64Fvz3n5Jo++x4/qUDi9YOStu4FafFK9nzZadAw BPMVp1g1YNKDEeS0WXYQZ6GFd3F5nfZwt/3WH2Qs1RftWpUzmrYH8iEw6FdXkNaKJvpV/n 5Cg6Ho+nJFqnEXhetkjRUZ1HUxDXSG8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FhyOtFe6; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704996754; a=rsa-sha256; cv=none; b=J3gh4sEj8ZdRIRzab1EL8EKOEFL5Sgzu4LN0apjoFaFC+NqiY266/kWGNSx+Lf/nDMJfCp D2V5lemjfphlrLmIa6cwtdNV4b7eQnakuKFodk/xkzF0hKhYmVDwx6DIia4aejjNudZ7Ex JUsOtyYTzbmci40esD+DvrgiLRaZwXo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3mQloxYVj81b7FxsG36DgI4RFnukVPFBF5N4HJB/q50=; b=FhyOtFe6kL8S/UGls9v/7hVEhe hdgLfkztUvUHgUvmV5con/y5NczYd+XVAhQdNT1Ps6UNhmhIMToP/7ZCBLbm85c/p1ula8/ZyNxai Dvte+kg/OeLEbPK2ZcD1/iF10PILenh7orq47CpPVB9Hd4yaZ2kJrtz27GLfHcPDF/F9sPVnqc0ae YIn9jCjGEgVib4OROuAY/oY0Wu4Nnp3vZ5J+FDmfksJJBBKv9Vu/865sBAF4HpU7zO5djGxS6JDpo ksSFpvv5qOkMr97Mj8iLy64ffVZH+aA/DsnaRuQPF1sJQjW+TbAagu/Vp3KlDDPhpbpWMiiV9JhJ9 ohD92AdA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rNzXR-00EWqy-F1; Thu, 11 Jan 2024 18:12:21 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/4] memcg: Convert mem_cgroup_move_charge_pte_range() to use a folio Date: Thu, 11 Jan 2024 18:12:16 +0000 Message-Id: <20240111181219.3462852-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20240111181219.3462852-1-willy@infradead.org> References: <20240111181219.3462852-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0CA4E18001F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 1wf75hc3g46fuqqpwg3awrdcixwz75u8 X-HE-Tag: 1704996753-483923 X-HE-Meta: U2FsdGVkX197iQDxKyPw/EaBDbeJQl1/Oj/QbFEKvl3Nrsz04+RBH8VkTe8Dp/38vGNVaA4oDA9/V94YDO36S3+cLcwJvusDc4IAhzIoTXo7hKVsuNjj+ZwgO/SoRH5eYh5S2hzLK35bcMZ6lCR2JQIDwEcGg6jCSJpKd2I5LHWxI6S078/ktf/ZnJn/3JsqNVCCfIia6JL8l8ZVQQmJUrdyndgkXzrSPDXQ9ROdpkIBKMuOV7yTTvBJuRWe8lK06/Q8HRWeJS3UyeBFplBm3EubvkKuS/AABdDNWMkO9vfkqY4m3OLFg+S95jVbNt0jfarFrcFPUG7Otl+46Qt3LJryjj10MwMqO3Vn/8bksh69ndO5+700XleAbiBhdVPnBXtnIIpR3tS6XGV0r5wzRQqdohvaQG71axmxalJtqYjooWUrbbbWwaCfRuIfBub3bPYooQzNwilx5vmYcbZjL/1yZMBqsOosEgGvM21eTflPI0oluIFzBbuL6Ft8HaXZtgezCjMZyQ3Zsfz0xf/lshT+WEHq2wZrydH6Zpsb7oNfctndZg8NFPNTh0t2C1zx1I5wDlU+rLhPHQN84VDIpRYwEt9xZ/icQm7ejWyWyMt9dtBhZvFaJLmBEzL+OXVc18lBrazU9aOw8RKM0HgNLWu+ziJVOpo7NFX/02f39kJnFqILVoPMjGqSEz7RvrTAzdeQhOQl9kBXLEv5vya2aZywsRtf6m976CqW+kOkdFAMSPRYocdbFASOZ6s33I94rQAyzh4JDRJ7cvSZ7QTHuS35bXQeSXH8dCAMp5Hv72p8e2vXYr3dsTZfKHHa+u6bFvJ5ZVgvdYS/7LTJ4IJFp5jecDJ0ZqCA4MBp8g3GfCOrDbcadDUKLF0I2FeZvKSZqOzOLALWxqeaIMRcuxnO6xBTecpxuv3cEf0PRmGy1tvYugAFhBGTFonX5r57gEz8+XJgBYdJc+t8vNZ/SyD cIJ1eZYJ NABQ9cMpSRgymvKpXJK2jhGb1C9kDkuPyINtfrhCb08Wa5XEfibbFDN2DHnwI7bPRjjurIuND1+HeO3osu4f6S5xLTrnH/ea1PtmPUUMmF7qskBqASUBWbenzumPWO10zhXdHBg+jI/E6HUnLiU6gI6ZH+OlfQCdqHCl1DYbStJHe2S9GyvYV+GmP4Hg99PiRdPMhsbLjl5bC4ltwVkLN2v0UKElpNyD0HgaF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove many calls to compound_head() by calling page_folio() once at the start of each stanza which receives a struct page from 'target'. There should be no change in behaviour here as all the called functions start out by converting the page to its folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Muchun Song --- mm/memcontrol.c | 49 ++++++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c4c422c81f93..c04bda961165 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5932,23 +5932,22 @@ static struct page *mc_handle_file_pte(struct vm_area_struct *vma, } /** - * mem_cgroup_move_account - move account of the page - * @page: the page + * mem_cgroup_move_account - move account of the folio + * @folio: The folio. * @compound: charge the page as compound or small page - * @from: mem_cgroup which the page is moved from. - * @to: mem_cgroup which the page is moved to. @from != @to. + * @from: mem_cgroup which the folio is moved from. + * @to: mem_cgroup which the folio is moved to. @from != @to. * - * The page must be locked and not on the LRU. + * The folio must be locked and not on the LRU. * * This function doesn't do "charge" to new cgroup and doesn't do "uncharge" * from old cgroup. */ -static int mem_cgroup_move_account(struct page *page, +static int mem_cgroup_move_account(struct folio *folio, bool compound, struct mem_cgroup *from, struct mem_cgroup *to) { - struct folio *folio = page_folio(page); struct lruvec *from_vec, *to_vec; struct pglist_data *pgdat; unsigned int nr_pages = compound ? folio_nr_pages(folio) : 1; @@ -6398,7 +6397,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, spinlock_t *ptl; enum mc_target_type target_type; union mc_target target; - struct page *page; + struct folio *folio; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -6408,26 +6407,26 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, } target_type = get_mctgt_type_thp(vma, addr, *pmd, &target); if (target_type == MC_TARGET_PAGE) { - page = target.page; - if (isolate_lru_page(page)) { - if (!mem_cgroup_move_account(page, true, + folio = page_folio(target.page); + if (folio_isolate_lru(folio)) { + if (!mem_cgroup_move_account(folio, true, mc.from, mc.to)) { mc.precharge -= HPAGE_PMD_NR; mc.moved_charge += HPAGE_PMD_NR; } - putback_lru_page(page); + folio_putback_lru(folio); } - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); } else if (target_type == MC_TARGET_DEVICE) { - page = target.page; - if (!mem_cgroup_move_account(page, true, + folio = page_folio(target.page); + if (!mem_cgroup_move_account(folio, true, mc.from, mc.to)) { mc.precharge -= HPAGE_PMD_NR; mc.moved_charge += HPAGE_PMD_NR; } - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); } spin_unlock(ptl); return 0; @@ -6450,28 +6449,28 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, device = true; fallthrough; case MC_TARGET_PAGE: - page = target.page; + folio = page_folio(target.page); /* * We can have a part of the split pmd here. Moving it * can be done but it would be too convoluted so simply * ignore such a partial THP and keep it in original * memcg. There should be somebody mapping the head. */ - if (PageTransCompound(page)) + if (folio_test_large(folio)) goto put; - if (!device && !isolate_lru_page(page)) + if (!device && !folio_isolate_lru(folio)) goto put; - if (!mem_cgroup_move_account(page, false, + if (!mem_cgroup_move_account(folio, false, mc.from, mc.to)) { mc.precharge--; /* we uncharge from mc.from later. */ mc.moved_charge++; } if (!device) - putback_lru_page(page); + folio_putback_lru(folio); put: /* get_mctgt_type() gets & locks the page */ - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); break; case MC_TARGET_SWAP: ent = target.ent;