From patchwork Tue Apr 18 19:12:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13216095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB2B3C77B75 for ; Tue, 18 Apr 2023 19:13:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FC3F900006; Tue, 18 Apr 2023 15:13:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 95C8C900003; Tue, 18 Apr 2023 15:13:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D534900006; Tue, 18 Apr 2023 15:13:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6D1A7900003 for ; Tue, 18 Apr 2023 15:13:30 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 31B4E160435 for ; Tue, 18 Apr 2023 19:13:30 +0000 (UTC) X-FDA: 80695460580.03.80E1D7F Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) by imf01.hostedemail.com (Postfix) with ESMTP id 63F6D40018 for ; Tue, 18 Apr 2023 19:13:28 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=KYPCQZ61; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681845208; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=13UUYwnQDW8sv6Z5U/gfNQWUn2NaHbpFqCKtGCMsvrI=; b=YiLzWQg1iwcE9twyW60ETWur33o3e8V1P81VJZlUVpgER9l5MondtRkniF/Y4FxnwUbdyE OyWg0a10V8xUiy2MBuGdBAbt5cP8BGo8vJJTLsXSzBm8qIOXsbbEORxVI4+r/ojuKfaQPt 3QybBOFyMhYBX5oFaIzpbmidQbxHp4Y= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=KYPCQZ61; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681845208; a=rsa-sha256; cv=none; b=GXcTKb26osYiFQMA1HNiEgJu2iKB9H/YxLxFw6oHw1646W5XMmg04zXMtj5P8Nn8QdFNmI yMz3HVQYeF2IyhmjkoZbuMmCisknZ/EHsdA83CZCvf90LoZzRWtAaaBIyOg6XMxUoZGXzU L+85ec/SMANPcc56JSvOoFKAMZ38CNo= Received: by mail-qv1-f49.google.com with SMTP id js7so10389667qvb.5 for ; Tue, 18 Apr 2023 12:13:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1681845207; x=1684437207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=13UUYwnQDW8sv6Z5U/gfNQWUn2NaHbpFqCKtGCMsvrI=; b=KYPCQZ61APdNYs9D70nl3yomvGcC38mIMM4ac/0Jxqh0MQrsikJDyqz8kNAKqw9iC1 1bmnq4F1MqaGWpFLA8ADTd1KOLEiSxDVYMfswSH15y4crGrkoG6QG/Dr8TRu4HNJqULs 3ondml+qKSu04IkF3zQ8fUI8BpomkXLoFtEZTlNb5W8qR5dw6PS9O8KDT/9Kxlz6inFB Zoapu1IejZKECDPcQcl9q/KpgYWJmZLUxb2uAQzBruh9kz4IWlq+6bWZCm5vzL8tOPu8 21csNAuzKAdA/rLnXVEEXuAuDwsTm2WP87lOxN6ifFsa6PFpUcCMCZg4Et1TQULauXd7 sXdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681845207; x=1684437207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=13UUYwnQDW8sv6Z5U/gfNQWUn2NaHbpFqCKtGCMsvrI=; b=Kg+pxnLLw9ab6nKXassXWPagtQQ/eJ1vQ1sS6amYxDxLtI5Cj1kcc7jivDiBwr0Ngg AwQ1M57uHbENj5q3pPpxmA0tv9R7DCxivizABh8lIdSkHL+h3jyJTD477qWaVK/dm9Qi UHv5LKypKO/XmcIJ1Z29lcoPw+DrGUHkkQAwVMJgv9g6woRxxltSPK8ipdbsGcSt/rl5 oDihcg2XRzEMwGSLvi0p4DYVNix/HagtZTg8TzdANaV7hakdlYvgVyfdg+V95wXyaK7V rEpd3uPG7/6Ofuinxjx6HCgAMIXvL0bDHy3okg9XWWPZMY1Pan7AEztrKBDK8YacEwQF uLGA== X-Gm-Message-State: AAQBX9eRoBUd/CvzJWvftCneMnGKHKkT8kNfZ81wPOJCl20lTIxbXsvq tZCAbejy1d1eUvI4XSYYkzFJOZ73r8Kr3JZgN0c= X-Google-Smtp-Source: AKy350bRhkIUr8N7okrlgh3Hwo07F/vQhkNz4v1tFl3I8eV2V83V9lNwfkKYujFHpq4QwdXTdItkPw== X-Received: by 2002:ad4:5946:0:b0:5ef:9b22:dc88 with SMTP id eo6-20020ad45946000000b005ef9b22dc88mr5188244qvb.0.1681845207300; Tue, 18 Apr 2023 12:13:27 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:e646]) by smtp.gmail.com with ESMTPSA id db3-20020a056214170300b005e5b30eef24sm3907199qvb.56.2023.04.18.12.13.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 12:13:26 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Kaiyang Zhao , Mel Gorman , Vlastimil Babka , David Rientjes , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [RFC PATCH 07/26] mm: page_alloc: move capture_control to the page allocator Date: Tue, 18 Apr 2023 15:12:54 -0400 Message-Id: <20230418191313.268131-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230418191313.268131-1-hannes@cmpxchg.org> References: <20230418191313.268131-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 63F6D40018 X-Stat-Signature: sxmaspmbrqjcsfnce18u3ja5rbzkpcum X-HE-Tag: 1681845208-319466 X-HE-Meta: U2FsdGVkX18/FzbSX5mEyoaRTWYRVJN3zof7EYJsY7wTwK+FkSw0Ef+ZAiA8CBNNGsI+hDYhduS0fCFLX0hc/wNHHU3CY6TjUdOUhvN9R3JOrWAUywUs6+92RO2x9rii7pRCocEbWKL5pPhKNwohD9f35vYjfNew62LsE1/aIkeFMOSOWvR1eObtolIOZ9vD8JHc7cHpCaA48XTvmMSmtiRlTMEdTH/onkZO5AVL/F5NIaLBf2W8dKGU5z03e+gQ4vy9sPAVz9BHeC4DqTrnNF/GuuLqLiAnLuGjt8r+TbZsByJR4YJn5HGZMSC0zGa/w7eyMec9y5/JP36msHY2gDvXjRP0seOUyjJl6zjPvMRUGI+eZe7V6eASQ1Dvd1zwsWHg/kaSvsvMFz7Ar3zKLpgd6PjsZ4KG8+VCCZxr1Wpia10ROG7uGQ58zyYjVKCvQDk01NXlqpcrHNQLcZ8lz/WpGql2V9Tyg5RFOjfp9ml2m3Bxp0BTHeQpOULNn+oXHnwfuTn7juqrehOhJppzSyHFehj6DCJ4XcaZfLXWUXvlOV+5dk/ZDenUPj3j5qT6ys+sdZVgE4PMyRVFfiTxBVTBdBNX0ni+22pPQPuw6CS4jkqm9qeq/6muO4rT+tyHLoqe+Yc68OLCF2/doEI25gGMdlIQITI227OT9A9uxIrUuTDsfMjJOuwUIOPsz9+72A62l7ZHmgJAyyObQ1fM2cl4QzwFBOTnm1kmbdUbUHgnK8sB+/+mqqGafUPR6kdGhKbuyxOsKo/hz6ESMsJnS1tTW7nz0Ua2uzlND+nKi7lJKVyVkgz7AT2XSm/9zxgtIdd1e80JRGSpEQ+a+tA3DuyOju0vV4IlCXtbqIEHeMe53Wlt5Nvzz9UPpA/OzJesYoreuSDut8sbJqSDjxoT3d1SrP1qS7/+W8mcDXh0hrYpe4g+5geXazNdAe+VPWgv6g2ejMt+L1ARmx5qkSW 9HkDdbWg wut+4Fopk2jvZ6pXUgzTEoPHSBLFD9RR+8cCZfeD8po/Ocx/hXfCjl0Cw3DnIg9jsnjaQSHgYuuT8mMmiFQgSJjKqPX0Kzta4+o6HJ+N6GNyYjCe5t2D7J2JZYsObqzGtEEi3dOAi9RqdcBANe1KzMVA6eT9LfhNWJusI8xEMOuHkzeqQBfqErhD0NjESQdhtreqSOb2h9IoTVl6YCMuu2GKuLOQcbTUXiaRyueFotzs0+LAI4Yo2YxJg+sCwhN0jyvJ3BQL5sv7IB31WJppX+OOtt8f6DQKBpm30DIUPikBrBtq9n35sq4yFv8Y1BOlQBLKvIw1ivYHIW9pLd4NWl5EXNWugIf5iKqrnlAk14ODQ9SOAsS995mlrrRwQDPE3qUIWGZFuhSO4f7hHKmMziSj93oZGWtbDWf/9ABZGdqG9p/Z3iDMIymt8ThTnb/kXCRov5HbUVK8836XGOcy3WK5GpiOsaVUbuIGHr0135KFFUDVuLE3sBUwN9njq7OT+X74fg+Wjy+W7xRVhrdckaogGEA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Compaction capturing is really a component of the page allocator. Later patches will also disconnect allocation request size from the compaction size, so move the setup of the capturing parameters to the "request domain", i.e. the page allocator. No functional change. Signed-off-by: Johannes Weiner Acked-by: Mel Gorman --- include/linux/compaction.h | 3 ++- mm/compaction.c | 33 ++++++++++----------------------- mm/page_alloc.c | 25 ++++++++++++++++++++++--- 3 files changed, 34 insertions(+), 27 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 52a9ff65faee..06eeb2e25833 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -56,6 +56,7 @@ enum compact_result { }; struct alloc_context; /* in mm/internal.h */ +struct capture_control; /* in mm/internal.h */ /* * Number of free order-0 pages that should be available above given watermark @@ -94,7 +95,7 @@ extern int fragmentation_index(struct zone *zone, unsigned int order); extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, enum compact_priority prio, - struct page **page); + struct capture_control *capc); extern void reset_isolation_suitable(pg_data_t *pgdat); extern enum compact_result compaction_suitable(struct zone *zone, int order, unsigned int alloc_flags, int highest_zoneidx); diff --git a/mm/compaction.c b/mm/compaction.c index 84db84e8fd3a..a2280001eea3 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2510,7 +2510,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) static enum compact_result compact_zone_order(struct zone *zone, int order, gfp_t gfp_mask, enum compact_priority prio, unsigned int alloc_flags, int highest_zoneidx, - struct page **capture) + struct capture_control *capc) { enum compact_result ret; struct compact_control cc = { @@ -2527,38 +2527,25 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY), .ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY) }; - struct capture_control capc = { - .cc = &cc, - .page = NULL, - }; - /* - * Make sure the structs are really initialized before we expose the - * capture control, in case we are interrupted and the interrupt handler - * frees a page. - */ + /* See the comment in __alloc_pages_direct_compact() */ barrier(); - WRITE_ONCE(current->capture_control, &capc); + WRITE_ONCE(capc->cc, &cc); - ret = compact_zone(&cc, &capc); + ret = compact_zone(&cc, capc); + + WRITE_ONCE(capc->cc, NULL); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); - /* - * Make sure we hide capture control first before we read the captured - * page pointer, otherwise an interrupt could free and capture a page - * and we would leak it. - */ - WRITE_ONCE(current->capture_control, NULL); - *capture = READ_ONCE(capc.page); /* * Technically, it is also possible that compaction is skipped but * the page is still captured out of luck(IRQ came and freed the page). * Returning COMPACT_SUCCESS in such cases helps in properly accounting * the COMPACT[STALL|FAIL] when compaction is skipped. */ - if (*capture) + if (capc->page) ret = COMPACT_SUCCESS; return ret; @@ -2573,13 +2560,13 @@ int sysctl_extfrag_threshold = 500; * @alloc_flags: The allocation flags of the current allocation * @ac: The context of current allocation * @prio: Determines how hard direct compaction should try to succeed - * @capture: Pointer to free page created by compaction will be stored here + * @capc: The context for capturing pages during freeing * * This is the main entry point for direct page compaction. */ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, - enum compact_priority prio, struct page **capture) + enum compact_priority prio, struct capture_control *capc) { int may_perform_io = (__force int)(gfp_mask & __GFP_IO); struct zoneref *z; @@ -2607,7 +2594,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, } status = compact_zone_order(zone, order, gfp_mask, prio, - alloc_flags, ac->highest_zoneidx, capture); + alloc_flags, ac->highest_zoneidx, capc); rc = max(status, rc); /* The allocation should succeed, stop compacting */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b9366c002334..4d20513c83be 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -944,7 +944,7 @@ static inline struct capture_control *task_capc(struct zone *zone) { struct capture_control *capc = current->capture_control; - return unlikely(capc) && + return unlikely(capc && capc->cc) && !(current->flags & PF_KTHREAD) && !capc->page && capc->cc->zone == zone ? capc : NULL; @@ -4480,22 +4480,41 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, struct page *page = NULL; unsigned long pflags; unsigned int noreclaim_flag; + struct capture_control capc = { + .page = NULL, + }; if (!order) return NULL; + /* + * Make sure the structs are really initialized before we expose the + * capture control, in case we are interrupted and the interrupt handler + * frees a page. + */ + barrier(); + WRITE_ONCE(current->capture_control, &capc); + psi_memstall_enter(&pflags); delayacct_compact_start(); noreclaim_flag = memalloc_noreclaim_save(); *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio, &page); + prio, &capc); memalloc_noreclaim_restore(noreclaim_flag); psi_memstall_leave(&pflags); delayacct_compact_end(); - if (*compact_result == COMPACT_SKIPPED) + /* + * Make sure we hide capture control first before we read the captured + * page pointer, otherwise an interrupt could free and capture a page + * and we would leak it. + */ + WRITE_ONCE(current->capture_control, NULL); + page = READ_ONCE(capc.page); + + if (!page && *compact_result == COMPACT_SKIPPED) return NULL; /* * At least in one zone compaction wasn't deferred or skipped, so let's