From patchwork Fri Dec 22 20:28:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13503796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10FB4C4706F for ; Fri, 22 Dec 2023 20:28:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC6D28D0013; Fri, 22 Dec 2023 15:28:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E23D28D0012; Fri, 22 Dec 2023 15:28:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC53B8D0013; Fri, 22 Dec 2023 15:28:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B886B8D0012 for ; Fri, 22 Dec 2023 15:28:23 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 937621401ED for ; Fri, 22 Dec 2023 20:28:23 +0000 (UTC) X-FDA: 81595591686.01.88D34E1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 035368001D for ; Fri, 22 Dec 2023 20:28:21 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="PmRxS/7s"; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703276902; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XJHqVmT115LC4ukb52O4YlKtYYhmqN0JTST31MbrqyQ=; b=T5KxzeUfef4o9j8D2OBvIhNNIHkTZwbd5FbXTAnj5wbLYn4PAoPGd6XaFNOnMJgFTReIg/ F2zciBn2oWGdGfPnH8zFr74BLsnV0MI+idSI+iMVkJA/1/AQ7ogbHXhqMeC79/HtqYwnyh 8Fo6wZSZ7uudKbRe7lK8esXcGXn6Rr8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="PmRxS/7s"; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703276902; a=rsa-sha256; cv=none; b=1bD1goJR2xmg8wVuFZhU2hI+gJoVphqwqjNTH/G9SPHQ0eJcDcBYB1M/mH9kw56cCXE8c4 uXNED3g3u1KI0FbwzkTDTbge9onOtN7xSg5vZ78XQ661+k05bp/xMwzdoh3Yhg1ux/FRO7 RQkyEz+TCr7sSqy8gBwbdnbIowLAvl0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=XJHqVmT115LC4ukb52O4YlKtYYhmqN0JTST31MbrqyQ=; b=PmRxS/7sdwhY88Rp00avDtdtLb sODvOn3QxZNc/mq1TRV91aMLmqdmNrFDKhLS79Br6cYOqBElruuoT+nD1VzsmFdqNVrm6A1S39pFR HyQsAhWxczbVxMV3Qd//BVvPoZVTw+YZvaam+J6qLWNEf2oX1g2X33ctZvRfr0RCMIdVhkBdYIWvB 2bI7S4apXJrxNEjw4oAWNxqsDynCH7Ju/4GgWrLifV91oh8O4oUAJfbzZozRC2y3i5QSQE8pYiFl4 0RgioaHmu5ZVLLyc/pXtozsCmXCJFvQ/AWNeV/5meUNnQFcxIpQvKbTY3BCjiLZ/xjxeEwVAZcGol CXPxB/Lg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rGm7u-008xbJ-QE; Fri, 22 Dec 2023 20:28:10 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Johannes Weiner , Vlastimil Babka Subject: [PATCH 1/4] mm: Remove inc/dec lruvec page state functions Date: Fri, 22 Dec 2023 20:28:04 +0000 Message-Id: <20231222202807.2135717-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231222202807.2135717-1-willy@infradead.org> References: <20231222202807.2135717-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 035368001D X-Rspam-User: X-Stat-Signature: 4yjrt51oms4fujc1cw5q8wnuiwwfnzrz X-Rspamd-Server: rspam01 X-HE-Tag: 1703276901-71385 X-HE-Meta: U2FsdGVkX1+m1nJf7USBOg3hKAjA9JnU7lQinS4QFuFrqQBgpgmvf1jdAsE4z/t7rehqO9G2C+nlkOjRihSA91bqvUJTrYeroNfD35V0rvf+OlAOA5rWV5j+Rzxdnf3qYUuYJv+TqfBqO0m/V/uUd+tZP3D6gLbDZgEZil6QxHgyYTUrr64UbiRgKAZe2c9WfgQANOMisVaiXzFWfz4z0ndK9D4x2MpZ1cKjz/7QfTpOEC1CdQCQIfdFjbwIfN/cCdQPRo5z5gpo4cyIFLQWtxRzHQigJJ0F1ClZLCQ5tcceDuhwbKFIRRsqUsob0iqZdtj4RyGtZjopQGOfgEtrHkrBBGg1BkjwHZQPbb/N0Q7yuyhB+zcrJh/wH8imFKk4JRlp9zohDcL9QBBhQiwtuhku1bUuB2KMy1sWRHIMbU7O536I7I9PYYfzZ0cwcwAUebskAY1Q1qvsVNxrQDm2F5wcbhzIZlVrxckVxR/lPjgzk5Xr1hNijgDs6rtTYfQeRAMhCOYgGYF2XPjIa7/BNU+pBEEk4F+SCamCU32YUVNqpdpdL/PqrSe+gvaBOb2029dM7C6g1fJlAM3uPzz4xytc5H7fKofEipuLbGcvlHP93GixNozjn483CNkB/IAd2C3N9LTuNfNOjfVk3clRPMs35HiSZchWd52QYy6FItRZj/kidtWVAsLlSFQfGXKfG+obIyy8JCRS8zzQhgxHGaNlTg8TPmOiM0ioTnticOBxPdCiCXn0MHarfjUvi/J9APGYIkUb4st0OUH+yh7g5jFJIQLpQz0ibcfJN+TX/oWhixYRZsXDmvuV6u2cmSsWWA3R0oGpn0B81dCihHmoenQ/KqkkQVkT/2Duyu9ZmQgXqO472fXe3lIGSoMmP3fp+FL7lYFGVGKJAYlpw38BAXWLqsBswndumuf43bCIZ8i2BLx7qg/PoZzq0SsjYEVrArBdJ181D0HxtcUozKz NCdUoyCE bExrat9ZYgpl2CNgpEY5neoynWKV/XVTUa0c3Q1XG6JDdb1BTerEh2LLjFAXi6mE0fY+tvGn1O9Wv+P0CDToNFjafvsco4GgLGfrc3Olm231jR5ZnkGpHmEyG4sV3FTjWLi5FXs6UGiZ7asBR8DziI7I3zLA2r2baudNbvR5/fQSNoLQk/oYqx8xRkg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All callers of these have been converted to their folio equivalents. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/vmstat.h | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index fed855bae6d8..147ae73e0ee7 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -597,18 +597,6 @@ static inline void mod_lruvec_page_state(struct page *page, #endif /* CONFIG_MEMCG */ -static inline void __inc_lruvec_page_state(struct page *page, - enum node_stat_item idx) -{ - __mod_lruvec_page_state(page, idx, 1); -} - -static inline void __dec_lruvec_page_state(struct page *page, - enum node_stat_item idx) -{ - __mod_lruvec_page_state(page, idx, -1); -} - static inline void __lruvec_stat_mod_folio(struct folio *folio, enum node_stat_item idx, int val) { @@ -627,18 +615,6 @@ static inline void __lruvec_stat_sub_folio(struct folio *folio, __lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio)); } -static inline void inc_lruvec_page_state(struct page *page, - enum node_stat_item idx) -{ - mod_lruvec_page_state(page, idx, 1); -} - -static inline void dec_lruvec_page_state(struct page *page, - enum node_stat_item idx) -{ - mod_lruvec_page_state(page, idx, -1); -} - static inline void lruvec_stat_mod_folio(struct folio *folio, enum node_stat_item idx, int val) { From patchwork Fri Dec 22 20:28:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13503793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C1BDC4706F for ; Fri, 22 Dec 2023 20:28:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F49D6B0083; Fri, 22 Dec 2023 15:28:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A7056B0085; Fri, 22 Dec 2023 15:28:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7456C6B008C; Fri, 22 Dec 2023 15:28:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6203E6B0083 for ; Fri, 22 Dec 2023 15:28:18 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 32E18A0961 for ; Fri, 22 Dec 2023 20:28:18 +0000 (UTC) X-FDA: 81595591476.11.6CE56A6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id DEA2B18001D for ; Fri, 22 Dec 2023 20:28:15 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="afe/S8DU"; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703276896; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CNl8JiBcrmMmxxRhHcEptB3LlVF+8Q3ivYeCXLGHndA=; b=DQVbiYBYe7/3BS4yZcWwRh25Eo/D9h4werrreHo0tarqv+kSGdCR53bPXiAk9vHctH7333 vSom/126qmRpZ7AnuNosQ8ax6bwd4blxLfoaMNSND++eemApFYp+nmCk0W4xp9INWBCEDG tCPBZTUqqhxKywGnMvagKVHrT3dHZF8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703276896; a=rsa-sha256; cv=none; b=Q5MbpadfOytGV24uTkPG4eXSMdNhNJPfDMwhRFE/FRjHyEonCMZVsUi8ZpuWcPqMoQL6Pd AJrY4nsLE+CwEmwUXhp6/79JHyks+guiCqY/jm5v7QxMSMEmUAFHCJnPxeS23gaid3Qb0c InQqSspL+6W5I1+mgcc9Voy2BMPUqY8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="afe/S8DU"; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=CNl8JiBcrmMmxxRhHcEptB3LlVF+8Q3ivYeCXLGHndA=; b=afe/S8DUnq6Dl27wo2+OPipbpi EjvSFraW6Oq/IFmlNqcahk8U4+D4sfPnCE4MEmhq9zFyVZYuS0pZ9GiSjwSLfjlX6++hvGBVOvH29 m19FcdxtnSknkqbqV9mdc8XMuNjonUuvjBF4DIebcV8MY0dbNBS/RqSJwGGTMgeeEzkkFJd5mQzwm 0shXyiaAcE7lwlcyITCY6bVCbXVHakc9IF2RAvGFnRzl7ndMP07jfvjvgqg9SLi35c5owb5HXBblo 8150lRm0RNIqv+zzpwG47BM4Cl4XfCAX28nP0EIcTPziSDgfnBGH97zDEjP0HvRwo83GBY5dsadby +i1mEgAQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rGm7u-008xbL-St; Fri, 22 Dec 2023 20:28:10 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Johannes Weiner , Vlastimil Babka Subject: [PATCH 2/4] slab: Convert __kmalloc_large_node() and free_large_kmalloc() to use folios Date: Fri, 22 Dec 2023 20:28:05 +0000 Message-Id: <20231222202807.2135717-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231222202807.2135717-1-willy@infradead.org> References: <20231222202807.2135717-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 99er6ds8woye1b5o9t43ncayfiyuayg4 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DEA2B18001D X-Rspam-User: X-HE-Tag: 1703276895-472945 X-HE-Meta: U2FsdGVkX1+k0SPeoTEMIaOiFPU1UP+TwsniAYwQnsn8mnLTTLwQwPQaepQue4ZB9M6n3WPst/Z+Vv+Ucq2mX8xJ3HeXap95PPsuEpJWRrviGJbi5yBdu2UFtACHFqUOGP97/ChNmbcd6OQ8wIWFsXBcIAvXqm0/XTgO70d4O+KYaVlS7Z1Gla9/VmFHTXFBX/WWMZNtzdOhpVgSRZ64Tq1s7AC9a0jCuk4zb6TePVLSazt1KJgt2cnlrCA0y/sfOFUcpJcKiVqAUF9u/REsxxXiJH14+y38snLLNUXjOh6W/YA7Aex5VEA8SdMfjXXf4W1fEpbP1WcZ9PKYpVpQmJhwdb31fVDUxyMp6SRVtVYNots+9ZGuUimwUptrn3JTTjOqNhdMSt55zBxSDJNUtHbWYyyhALckeUmhQncaM0CGvAsMGFz7Lro4E211gVxx+7v4KkQUr5LWGQFQCyfVxc9hdgCyT0yf5YeW4zYeMqTMSX+aXMKla7jnyjFEOqINIqRpP2GRfNl25FBrGIgqhPbLuLoXxK1FNwrdttCTfq2bZoRIwb3ZmYX14Qn2UB2hEZ2ofXSnLjTX/zsNLfeFkZudgYBTqiAAmCl08ARl+KXuiwNGdqmjb4PsJ2+SUDZXLysRAcR4PP+FAtaF0aRYHoRkDAK3MNBV45T0Hf/QqwQIDczyoLd0RCULhxw3sfdH4EweZq0TrmwB3MzQ3oo9DiyDzRkn+48Ul7tyW0/OAOnSnioC8z9RRjOMjuXBFm/tS6IZ++VJz09/bnVxi9l8NfhZt6NotPnNIewRuEUept6NnWt1cm+LW16syrnNvUfgLXgLNjDUs2nE8bDlPR2BIYwCycDxCJ9wVamA1Fgugxdr7zC2lwYd0M1fA2x/AiZqnRptyr//gKHSyrbC67/A22kgOYEu1mpc32mjC17cX5I2XzmzP195X6UWG0ht9fkBLvsprAqiyqZX0hEVaUR h68EouKc JSOJmU9wi6IhG5N+unVKl9Ew4neFILT95y95Ihb6FLO5T+mnhN21/xdOZPFmv9PmeOf0TX9cmDjMP5/XFAU+PyXtww3YnhqzgjiFP0/o+95IkwSOlh+1bLLCGoQOjXj5PSYlft0402r42sTUc8kAoPOFn2aonOmo/fjtrkjrH85WRq2GDUe+vxpxiYqCS5MHPUvm2hm2pxyNjumAjwXRSLjjoziJNSea10Wfr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add folio_alloc_node() to replace alloc_pages_node() and then use folio APIs throughout instead of converting back to pages. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/gfp.h | 9 +++++++++ mm/slub.c | 15 +++++++-------- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index de292a007138..d56c1d7b5c5a 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -247,6 +247,15 @@ struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) return __folio_alloc(gfp, order, nid, NULL); } +static inline +struct folio *folio_alloc_node(gfp_t gfp, unsigned int order, int nid) +{ + if (nid == NUMA_NO_NODE) + nid = numa_mem_id(); + + return __folio_alloc_node(gfp, order, nid); +} + /* * Allocate pages, preferring the node given as nid. When nid == NUMA_NO_NODE, * prefer the current CPU's closest node. Otherwise node must be valid and diff --git a/mm/slub.c b/mm/slub.c index 35aa706dc318..261f01915d9b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3919,18 +3919,17 @@ EXPORT_SYMBOL(kmem_cache_alloc_node); */ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) { - struct page *page; + struct folio *folio; void *ptr = NULL; unsigned int order = get_order(size); if (unlikely(flags & GFP_SLAB_BUG_MASK)) flags = kmalloc_fix_flags(flags); - flags |= __GFP_COMP; - page = alloc_pages_node(node, flags, order); - if (page) { - ptr = page_address(page); - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, + folio = folio_alloc_node(flags, order, node); + if (folio) { + ptr = folio_address(folio); + lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, PAGE_SIZE << order); } @@ -4379,9 +4378,9 @@ static void free_large_kmalloc(struct folio *folio, void *object) kasan_kfree_large(object); kmsan_kfree_large(object); - mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B, + lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, -(PAGE_SIZE << order)); - __free_pages(folio_page(folio, 0), order); + folio_put(folio); } /** From patchwork Fri Dec 22 20:28:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13503794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B39E3C47072 for ; Fri, 22 Dec 2023 20:28:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D60426B0085; Fri, 22 Dec 2023 15:28:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D10B96B008C; Fri, 22 Dec 2023 15:28:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD82A6B0093; Fri, 22 Dec 2023 15:28:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AAE496B008C for ; Fri, 22 Dec 2023 15:28:18 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7D2051C196D for ; Fri, 22 Dec 2023 20:28:18 +0000 (UTC) X-FDA: 81595591476.09.E9AB2B2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id E243740018 for ; Fri, 22 Dec 2023 20:28:16 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=W48EYwdf; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703276897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NccTglOgefcrO3rZHgTRFfPB5Ozz6huMlwZ2e954KZQ=; b=8GfKDiWreDihBYkErgqK8V0wo9/wl5l9XKcSSw+iVAlJDhE9BblIwNdhv4XfS5YyPRcU/i kWmx/135u0o1fIl5RvJ0xzDt5eNEeOgVcRKobjmXdvfFBw9XCAD3nFFlxK9XhVPTeXEEkx SKOqduotruVyDuna060oTg3PJI+dTqI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=W48EYwdf; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703276897; a=rsa-sha256; cv=none; b=dOxpnYxUGN1XVdr/JgodhFruLYwiG45R2YxyaI9ZNp6TuqiFwU3YDaCG73AeNxMVn9jbsS lZji1ze6VWZcvwF0gU0fnsT8Zi1ySfB9a1fbNQWGShQmiV0wQ8pRxZspfr2aWJ35BQrS1g /jMkuLcUyI3WEDlViZpnE7qUeXdAVvQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=NccTglOgefcrO3rZHgTRFfPB5Ozz6huMlwZ2e954KZQ=; b=W48EYwdfHtnKe5AxeywjFkXkwh 3Owz09ExUw+tPMpAzwAaE0PVONEM6hDPbesCwibsRKoz847aUeOlAGzHfiihH3JD3/hCTEBe2js5l Q/EwipkfRA/ZWdNmh7vRDLQuI/NmINWYbX6ndK2xIC9k5U8OhuCvpCvPUVlVfTezMf0BOJM0CxH5a ApI+FY7YzswVlQNkrs2rwSIxWwQ1MKJLhWxWtINxru7Yosor+7cwb3CH5VWTdq9H5zyS7VnjsKOr4 jnUmb7bCh8dN6FLIFDu4+HsspPnyA0L+r8xikQOpaA/ZQXvRb9Q/3rLV5sAGYNUKZE+aWvEHWKYUZ h5KFpr1A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rGm7u-008xbN-Vt; Fri, 22 Dec 2023 20:28:11 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Johannes Weiner , Vlastimil Babka Subject: [PATCH 3/4] mm/khugepaged: Use a folio more in collapse_file() Date: Fri, 22 Dec 2023 20:28:06 +0000 Message-Id: <20231222202807.2135717-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231222202807.2135717-1-willy@infradead.org> References: <20231222202807.2135717-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E243740018 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: f7fj1y4xc5kn1n6miyqxbe1w4isosatw X-HE-Tag: 1703276896-889624 X-HE-Meta: U2FsdGVkX1+k4aade2uXF9Z5LqKWQmc5xpgsORmB2A2bYnid1Rvt/9qpsymGgwM3L2HbEH2TU0ac4XfgdDIDMUUDkpanaNnLW4RnuqEFT9D+7m0BhIdUmGYAhocgR2wrExtXeWe7spUjaJG/uSaUJEFkNlMEhLpjTSoH6Yapbqe+DmOVT+2nuR9Tthg/qHmqG8WsXV5MNPioxdPmANTzOx1xaamqmZz7q1gRjW17vMp7Y7/kCVCcMe8AK6M9yug5Ujml2gYm1422bZM/Jcsql8/o/tQoOjYWjTooc/Ha/Im/tzTs5Y0GlnWgdyHD91v9icGzak9MmBtxBrX61wN64niw84smci8o+HzCkwp84TMqlBObEX4RL95Nhmx7YbOoTF/VckMcfUhZvlnulLpoFsC8kPDNUxOkfNzgo+wKhT5qCUTWPM+3DzJN/6iwQk6dFspql6T2CAOYvJtF/XLAUFIGPBveHj2pD/B99pTBS8vHnj+px+MiwJbf/ngluceMAwBMod5AfYkxQmcNfnQD1Pelj9Hw7kGQir6KDnD4f65HPkz4yhCKnEeFpgB5tr3YMHzP0CLh04n6rPzp3SBjkRCGtEg4iPA0pwxYvkPBhaw+E3+2A7lx9o/c+2ixgmWSwLOGh63aYc5wILjbupg65nygBtTRpT644C6Y5P2fgSh8H45L8H+FFINlB88h0A+ah74wYr9A/zPz4VTv2tmChYr9KbVXf/RZcrpcjbVyPIxnMUp/Q5EnvBTzeE1qKZcPWMqsABa+p6FNED3yFsPkccrGBdnboJBOnn4ddE7uPJWY5/XkG+x1m9489mehsUFp3MKdxO7I3HTlrbpCu6+Q2HCLvXqKNtMtjVTItGTYrS32zszOuKlYLNsVwrItPJXo/35NWJnKmzl74px+xRVwQtg1aSNwgWYN8clhm2h+40QPeY20Zssh3xoaeYOY7wNAEpyxvtFaocbybjeGY1Q xvds7VAi RCuJCSCxq1SswWERrQS1vY5bTGQ6VUJ0AmWOKRPPUldnl22I5I26adLVHIWzxRc+cbPRhwx+59EBdZiTIlOxQC4WSdYFYIzGDhOpVIvzv6Nq5jtT9T3fROs90MPtGz+CG/c2oMLd9uM/sxuyynLa8RVY4Th7qTe385Qr6UdMS4PyZm5HWOCG4aBD/zeEKsxOiwHoI/0l0JWUHISTEPLDSbDo6AiVftSdYlU2m X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This function is not yet fully converted to the folio API, but this removes a few uses of old APIs. Signed-off-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 13c6eadbeda3..b9b0742e4d9a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2126,23 +2126,23 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, xas_lock_irq(&xas); } - nr = thp_nr_pages(hpage); + folio = page_folio(hpage); + nr = folio_nr_pages(folio); if (is_shmem) - __mod_lruvec_page_state(hpage, NR_SHMEM_THPS, nr); + __lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, nr); else - __mod_lruvec_page_state(hpage, NR_FILE_THPS, nr); + __lruvec_stat_mod_folio(folio, NR_FILE_THPS, nr); if (nr_none) { - __mod_lruvec_page_state(hpage, NR_FILE_PAGES, nr_none); + __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr_none); /* nr_none is always 0 for non-shmem. */ - __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); + __lruvec_stat_mod_folio(folio, NR_SHMEM, nr_none); } /* * Mark hpage as uptodate before inserting it into the page cache so * that it isn't mistaken for an fallocated but unwritten page. */ - folio = page_folio(hpage); folio_mark_uptodate(folio); folio_ref_add(folio, HPAGE_PMD_NR - 1); @@ -2152,7 +2152,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, /* Join all the small entries into a single multi-index entry. */ xas_set_order(&xas, start, HPAGE_PMD_ORDER); - xas_store(&xas, hpage); + xas_store(&xas, folio); WARN_ON_ONCE(xas_error(&xas)); xas_unlock_irq(&xas); @@ -2163,7 +2163,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, retract_page_tables(mapping, start); if (cc && !cc->is_khugepaged) result = SCAN_PTE_MAPPED_HUGEPAGE; - unlock_page(hpage); + folio_unlock(folio); /* * The collapse has succeeded, so free the old pages. From patchwork Fri Dec 22 20:28:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13503797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AEFDC4706F for ; Fri, 22 Dec 2023 20:28:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 461FA8D0012; Fri, 22 Dec 2023 15:28:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3ED7F6B0098; Fri, 22 Dec 2023 15:28:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F0958D0012; Fri, 22 Dec 2023 15:28:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F25CF6B0072 for ; Fri, 22 Dec 2023 15:28:26 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C7D10C0541 for ; Fri, 22 Dec 2023 20:28:26 +0000 (UTC) X-FDA: 81595591812.06.0B0B3F1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id 2D06B40006 for ; Fri, 22 Dec 2023 20:28:24 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IR6Zcf7v; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703276905; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zbMRzMXzHw9pcMkkq4R7lfUfXVXbyGf6jYLxU1701xE=; b=4nKB+fJOIv93Yctffx9S1GbcpBjz5YO6RCy/BGuIVFKSt5iIyyaK8Psoo5MO7uSwwsJjeq Iytkf1V/R0ajl7Zu5OZXZ2vRNdc47lzDo9VhJ0wtu5q8qf8mPeAAy5YffeKgYCnQABUjA3 4Ckf0ftAliZLLjez+N0hw+z0zlRaxnw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IR6Zcf7v; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703276905; a=rsa-sha256; cv=none; b=J3uQofaUrfYjvxqMxhdrJsmJVTDn3bVTxBTY0XYfmu/B9VrOlpwpy8x1ro/0WjzKhpB+50 jcM0z29W6hdfj0oNNQ35rxdMPC445wEMBeaoetb4O0iMu1AE2NXtDTn//eGy3tssc3rjn/ nLK+vFjnIjbv5LqfJBRNrP7ujZao8/U= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=zbMRzMXzHw9pcMkkq4R7lfUfXVXbyGf6jYLxU1701xE=; b=IR6Zcf7v591A0XyD+3HZIipGTU hzz9I2zXB8tcf6tD5Kz/DW/YR1k/MGZNAtrRC9VaxHJOZAMI3SsSiVwSERhdotVKHZ8EwYy5JJcLK n6ZJd4bMzvyNLNPVh5lSOpmcV5w+NMN34DzQ+k8bDtej6AEcRQ6dWHuBXQWoXM8Ijp6Pk79rv7txF /ws+abAJN+DkWm3sS2nl02cLexix1TscdJU7rUpz5ytIpjveYrLnEfuprFR45iAJedHabvkYq3FJs n63PMz/aP57dfoLa3njA4yE72YSD891b3KcPsCX/WG+mVmm7m94UB1MqeBghWQ1FekGP5N2dveFoH AnO/3wfg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rGm7v-008xbP-25; Fri, 22 Dec 2023 20:28:11 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Johannes Weiner , Vlastimil Babka Subject: [PATCH 4/4] mm/memcontrol: Remove __mod_lruvec_page_state() Date: Fri, 22 Dec 2023 20:28:07 +0000 Message-Id: <20231222202807.2135717-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231222202807.2135717-1-willy@infradead.org> References: <20231222202807.2135717-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2D06B40006 X-Rspam-User: X-Stat-Signature: arawhxtoofh7tmngewnc1m9c39a1whbk X-Rspamd-Server: rspam01 X-HE-Tag: 1703276904-297171 X-HE-Meta: U2FsdGVkX19YVVDE+7klyR6iFPdIh7FLbFXncUIbsz3KY5LTzAWxIkUXzQoVITAO49zN2EEcJ3a4MSjj8PwnNAw2T0Bl5Wh/HAfNQfTBbFQlwL6n+hpHn9xEz9tw5UoTxvq318JvATGGu8J5hgao0d3QTQbWLWDmJ2GcVXfIZ8Pwm6k2UDYPzyvVo0MvtXEHDjyOMA3zoWAXp77Hu1HF4MYT2kre2hvZmozV+++9XeQTAfxnhyQb0lWJu9RVbyo2laMNq5CSvFwBYmqCa7+norgbiAyN0Kdl89labzWl+njWiLtCLlsBLL4MQGYB2+vx7oPPqQTYCbnVlyuFgRTZMGHbQ1hbSELghP1VTDgADJPQpDna9O3z12AWMFIhBuZVBQf1dG6AY+q2O3ny6PLq2ki5IYQjV6dwA6D5JQ+iv4iPsTpOJnyyBWf4QNrp80q39OKd4Hrto69l62SDDAtkj2WbP34H73FNmNaesG21GaDIANYpE1lSm+xyxeqcCZS1bt3exUrBuU9UcAe/ENin8J81Pj8LqX0nHK52tESY+3IVXt9xwWrEj7pKeNuzvm1RoAI8mPBV5hu2vNi80HB+w0VWYSNRI9EUGpWz/U+p0JZGjmaj2EoEG0dgPeXRGRQRkKZepOUaV26pvCOEE5rVh02Mmj7lyNQJZhMGvv6Xln7avFxDAHQDJXzC+xEd0OBqcUNzP766A03EgoblbzMQaQsQLQ7yhmekTF8YxK2EwWTyBoOamLXq8gT7jl//O0oOsP4w/ORs3Gd+97jN29nIFZgLXN7kbAUqgyeBZ58RbOYjPAxyXXHiyGS242cKWM/G/1fywqQZTAbpA8DB59+Bf/YyAoPVu2Horl13WGy9pc00CYhQ7hznDPa4L4ArsOQrE6Cmd74lgGbSBGyyflmLkIAY8Xf9d6ijxJIh0uH0eqBbo8uZb7OQJSCKLdqyckEua+bgsGxoRl5OZejVPuK cL0cv8Rs 5LZiNA+wBzwWVtvbAF8J4Rc1Snc9+pG8AohOWRTk5mplRiqozuRusxlNQaUElQAInnMTY2ST4t2vWYQ/Soy5xxATXjVWxlERlLJFEP7oVCYdws3QvGCoo+Eh1MrTxEZqWb31lL1EjqMJWVBkKMaOMxk+uKtQgRDkPB4/MtVHm1kTmheesQ7t69KZIvahhbwNufN/0mG6URZM1ysdgM/BXUoXDtLtCuebs11Q6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are no more callers of __mod_lruvec_page_state(), so convert the implementation to __lruvec_stat_mod_folio(), removing two calls to compound_head() (one explicit, one hidden inside page_memcg()). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/vmstat.h | 36 ++++++++++++++++++------------------ mm/memcontrol.c | 9 ++++----- 2 files changed, 22 insertions(+), 23 deletions(-) diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 147ae73e0ee7..343906a98d6e 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -556,19 +556,25 @@ static inline void mod_lruvec_state(struct lruvec *lruvec, local_irq_restore(flags); } -void __mod_lruvec_page_state(struct page *page, +void __lruvec_stat_mod_folio(struct folio *folio, enum node_stat_item idx, int val); -static inline void mod_lruvec_page_state(struct page *page, +static inline void lruvec_stat_mod_folio(struct folio *folio, enum node_stat_item idx, int val) { unsigned long flags; local_irq_save(flags); - __mod_lruvec_page_state(page, idx, val); + __lruvec_stat_mod_folio(folio, idx, val); local_irq_restore(flags); } +static inline void mod_lruvec_page_state(struct page *page, + enum node_stat_item idx, int val) +{ + lruvec_stat_mod_folio(page_folio(page), idx, val); +} + #else static inline void __mod_lruvec_state(struct lruvec *lruvec, @@ -583,10 +589,16 @@ static inline void mod_lruvec_state(struct lruvec *lruvec, mod_node_page_state(lruvec_pgdat(lruvec), idx, val); } -static inline void __mod_lruvec_page_state(struct page *page, - enum node_stat_item idx, int val) +static inline void __lruvec_stat_mod_folio(struct folio *folio, + enum node_stat_item idx, int val) { - __mod_node_page_state(page_pgdat(page), idx, val); + __mod_node_page_state(folio_pgdat(folio), idx, val); +} + +static inline void lruvec_stat_mod_folio(struct folio *folio, + enum node_stat_item idx, int val) +{ + mod_node_page_state(folio_pgdat(folio), idx, val); } static inline void mod_lruvec_page_state(struct page *page, @@ -597,12 +609,6 @@ static inline void mod_lruvec_page_state(struct page *page, #endif /* CONFIG_MEMCG */ -static inline void __lruvec_stat_mod_folio(struct folio *folio, - enum node_stat_item idx, int val) -{ - __mod_lruvec_page_state(&folio->page, idx, val); -} - static inline void __lruvec_stat_add_folio(struct folio *folio, enum node_stat_item idx) { @@ -615,12 +621,6 @@ static inline void __lruvec_stat_sub_folio(struct folio *folio, __lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio)); } -static inline void lruvec_stat_mod_folio(struct folio *folio, - enum node_stat_item idx, int val) -{ - mod_lruvec_page_state(&folio->page, idx, val); -} - static inline void lruvec_stat_add_folio(struct folio *folio, enum node_stat_item idx) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 36bb18d7b397..7a759554bec6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -891,16 +891,15 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, __mod_memcg_lruvec_state(lruvec, idx, val); } -void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, +void __lruvec_stat_mod_folio(struct folio *folio, enum node_stat_item idx, int val) { - struct page *head = compound_head(page); /* rmap on tail pages */ struct mem_cgroup *memcg; - pg_data_t *pgdat = page_pgdat(page); + pg_data_t *pgdat = folio_pgdat(folio); struct lruvec *lruvec; rcu_read_lock(); - memcg = page_memcg(head); + memcg = folio_memcg(folio); /* Untracked pages have no memcg, no lruvec. Update only the node */ if (!memcg) { rcu_read_unlock(); @@ -912,7 +911,7 @@ void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, __mod_lruvec_state(lruvec, idx, val); rcu_read_unlock(); } -EXPORT_SYMBOL(__mod_lruvec_page_state); +EXPORT_SYMBOL(__lruvec_stat_mod_folio); void __mod_lruvec_kmem_state(void *p, enum node_stat_item idx, int val) {