From patchwork Fri Apr 14 13:02:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13211455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAC1FC77B76 for ; Fri, 14 Apr 2023 13:03:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E157F6B007B; Fri, 14 Apr 2023 09:03:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D774C6B007D; Fri, 14 Apr 2023 09:03:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF2A16B007E; Fri, 14 Apr 2023 09:03:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A994D6B007B for ; Fri, 14 Apr 2023 09:03:25 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 88F03802A7 for ; Fri, 14 Apr 2023 13:03:25 +0000 (UTC) X-FDA: 80680012770.27.8C7BE1B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id AB0DDC000D for ; Fri, 14 Apr 2023 13:03:23 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681477403; a=rsa-sha256; cv=none; b=S3Zfe5yS9mWUZaUcHetyi8mon6ZiPor4c86/4IaTrQbQ0eusBvGsHNQDA8M/Us6GiPLqKq 9XbQ3czsMCyTvuATKT398iEAHbt9ijv6pWWUemTMpc4SJzZcWGcs+y3MoZ180SUNACQ8YS grzhWleJiR9TF/AmLVX8azXf1jWBStM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681477403; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BspBzcbIcmP3MIkdJlihJ4zGGnNWp7gX0CygQG0DWB8=; b=0e5eU4TuXhfTlb/k1TVvaLh925noaQ+vyQ00A1qxgCRsVmDymsfJ2wMYA9h3V/kpPPYQp1 5doS34VXvNa6753equVg913eNMdXz45VQwaniMmJpKNBRArvh+LlmAChwhIUbADdCv1XyI XrSj1PqYyhct9Xus0SZH+mg0BOup1CM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 417941713; Fri, 14 Apr 2023 06:04:07 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BD1443F6C4; Fri, 14 Apr 2023 06:03:21 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , Yu Zhao , "Yin, Fengwei" Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC v2 PATCH 03/17] mm: Introduce try_vma_alloc_movable_folio() Date: Fri, 14 Apr 2023 14:02:49 +0100 Message-Id: <20230414130303.2345383-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230414130303.2345383-1-ryan.roberts@arm.com> References: <20230414130303.2345383-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: AB0DDC000D X-Rspamd-Server: rspam01 X-Stat-Signature: ebs6rt876zi5aw9a7t3um36y9srqo19w X-HE-Tag: 1681477403-359601 X-HE-Meta: U2FsdGVkX1806Y+A4WApMZ7SN12i5Dgk+2O1qzkizCjUcsicOOx28TsY3kh3oL34GiabYiHLOlT8y4qWrULul7MDkHVD069Ht9RfccLDYNjaLzcBl0T6+ScTVlc2lFcY3LkVO8qfJxIWgDjKAU46l4M/zdZqMU6KRKTWKSNIQButQUqY9mO7iOH4g9RRsnpX2oSQUijh0jQFZN3XXU2bqHSi6ObU8F1AXFujC0xRCWnCJf6ch0yA0V5TLp/ab/9ApTHjMWdojJbhAG5Y7YgLgC8QPHu1bJ0SCTXbuR0XY33G5kGRqydjqjSoCROj3ceiPFB1akm08FRSQe3Svbi00e5PkoMcv2ptQfQeiav9sTj1kn36XKTruiQVmb+/lo+VKcYLSHcDL5JsI2aXgbZRk2d4K9vFxhQmy4lTMAbYHWTgBCgmVMV0cQveEsPheihzEKEpqiMMTk4hR+lgMKb8snrI3MkiieLoNFHBqziitduXdXkvSjTsGd2kF9gw7R0iah5+aYWEwLIvvdlYAi7VM55rfHPLuVRbKCCN+/1bi/BKX8gHV273EfWI1jcPthwcMVo+CMKJIYKD+Li/s7qsiQ+kXVxXxHu6yUqo5zUC2wbYDXCIV+fQCMmImXSnhokOXtts4+0JsOSf+bgfjuAEalsz4ch9wRTQaSGtZ54l0vJTOabj7CaSJ1Si7ziWs6h/odLpgpChvUSoHAjla4d5NOGUFAAWiVml4nhKPrhk73ASzQyZY9L9zyFYoKdzxmHJoYjW0GyKYZ+4cjni9MwJV7Kt+T8yeB5sexFx452x47e0znMtoSNv5PGUOshHMJBsT9NOzIp8K0LLdbED0cA+4pHrOeAzNCbGdFqw29wXYUpajs+/NvOt8TlfKUjEFA+UjQSyqaQnNOoK2QkjamxTKKZeP9/kqHWOTECT/+65AO9UACdt1zBqTMSkCeMbLo+k1HNCAC4wrP1bHBWJkwB 2LK4WL+y I3gL7EBxU72g/9IL5UmWtEF050jWLMAF4fL+7LL5fM3wUtZlaLnmJWNMa0A3USzwPnM0ic03Ad16XI0qNlTC0nkYr2N51KjlMa3HWeBzaxHBXvo0rTyiyqJP+/gmtYz68MX2CXOiwzkVTDcJIO/nmeCPTJUYXfCk6lbX7xd0t1XA/5L8qOL4v6AuEnfYk+FC3ZrQfpyC2e+dy1bZrVcfzpze4yLsr/e/n8hhF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Opportunistically attempt to allocate high-order folios in highmem, optionally zeroed. Retry with lower orders all the way to order-0, until success. Although, of note, order-1 allocations are skipped since a large folio must be at least order-2 to work with the THP machinery. The user must check what they got with folio_order(). This will be used to oportunistically allocate large folios for anonymous memory with a sensible fallback under memory pressure. For attempts to allocate non-0 orders, we set __GFP_NORETRY to prevent high latency due to reclaim, instead preferring to just try for a lower order. The same approach is used by the readahead code when allocating large folios. Signed-off-by: Ryan Roberts --- mm/memory.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) -- 2.25.1 diff --git a/mm/memory.c b/mm/memory.c index 9d5e8be49f3b..ca32f59acef2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2989,6 +2989,39 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) return 0; } +static inline struct folio *vma_alloc_movable_folio(struct vm_area_struct *vma, + unsigned long vaddr, int order, bool zeroed) +{ + gfp_t gfp = order > 0 ? __GFP_NORETRY | __GFP_NOWARN : 0; + + if (zeroed) + return vma_alloc_zeroed_movable_folio(vma, vaddr, gfp, order); + else + return vma_alloc_folio(GFP_HIGHUSER_MOVABLE | gfp, order, vma, + vaddr, false); +} + +/* + * Opportunistically attempt to allocate high-order folios, retrying with lower + * orders all the way to order-0, until success. order-1 allocations are skipped + * since a folio must be at least order-2 to work with the THP machinery. The + * user must check what they got with folio_order(). vaddr can be any virtual + * address that will be mapped by the allocated folio. + */ +static struct folio *try_vma_alloc_movable_folio(struct vm_area_struct *vma, + unsigned long vaddr, int order, bool zeroed) +{ + struct folio *folio; + + for (; order > 1; order--) { + folio = vma_alloc_movable_folio(vma, vaddr, order, zeroed); + if (folio) + return folio; + } + + return vma_alloc_movable_folio(vma, vaddr, 0, zeroed); +} + /* * Handle write page faults for pages that can be reused in the current vma *