From patchwork Wed Sep 21 22:36:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12984274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D5EFC32771 for ; Wed, 21 Sep 2022 22:37:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 062416B0072; Wed, 21 Sep 2022 18:37:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0129F80008; Wed, 21 Sep 2022 18:37:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF5666B0074; Wed, 21 Sep 2022 18:37:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D1A1B6B0072 for ; Wed, 21 Sep 2022 18:37:17 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AC523A0185 for ; Wed, 21 Sep 2022 22:37:17 +0000 (UTC) X-FDA: 79937554914.22.D839E01 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by imf27.hostedemail.com (Postfix) with ESMTP id 5F8C74000B for ; Wed, 21 Sep 2022 22:37:17 +0000 (UTC) Received: by mail-qk1-f178.google.com with SMTP id k12so5031701qkj.8 for ; Wed, 21 Sep 2022 15:37:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=W65OPn/JICs8j0QopHJv5kNae6yu6rzXZKtwjhKQdDM=; b=BJt7G/2c59nwBSk8Q0V3/7HnKDO6LesViA0xuVg+AKKCX+h2EeeBYkeXg6TsoAjlVC Kib+ror8qjzjovAiAcLv1qxgl61JtBdSuWNin5zR6mKlz5IaSzErcjv/YpMPdKimVqk9 V6p0PisB7NFwu+6MsTMW0YBvCWFQ6g+5q/GnTMHJzHvw2ZxG+o1Aa4b6LHlvAWogYl/S Hr4kS8K6rhVR6CSn5aeSA0GnQKzcFVoawtIeiRgdn6w+ONR+upm/zFSZI7s5R9zcI7iV fFAN2mAfWX1wHgNMjY+Xen1iIvDy1Wd8PQkGAnIihgOh6KtFnNxdX7iQmG+ZNvQfEhmm IJZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=W65OPn/JICs8j0QopHJv5kNae6yu6rzXZKtwjhKQdDM=; b=k23wii1F1BSzYcqzkYyhdaUgtdT4QlEYx2+TamXdJlQNAhZjb05YatIPqLpRdDK6MR cOZRiJZxkt60RXxFk2+oPFHnyRaftAwqA04AiWMsLoWlcgoiANHzxbq6PlhRA4BxSo5/ JbtN721MqT8G/uEmstlYEcduJnSKvfPU9px9RMFLM7yfVJ8bIg/uz4C9nrCYqO8WdSXT P8tsoDvJbfCZysTQcymHUtj/8LdVdNRuHLSGioFkcuAZQ7AvqFf4aKtatu/8ULG4e5G+ pi/T6cierrI9SIQ7cHd0T8hfhd6HwlyG6rKCJB5cKCjm/DlTEgq509G2ak9qrzP8+7ju O7Ww== X-Gm-Message-State: ACrzQf0Gjj+F6rKT7kp+NUg0XjAQp5FeuoOjcGnMLg909X6EdnJC/PnA N0jiDeNPQFHAXkH6JsTwvAP0oihUDQk= X-Google-Smtp-Source: AMsMyM7dWSkzW1KyqoUVlKGHvNPTSTszHVZixiUm4p91nyjOSPM1g3zSIZHi3b8z8mqKUFlf+7gJoQ== X-Received: by 2002:a05:620a:254e:b0:6c9:cc85:c41a with SMTP id s14-20020a05620a254e00b006c9cc85c41amr275973qko.260.1663799836698; Wed, 21 Sep 2022 15:37:16 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id j13-20020a05620a288d00b006cea2984c9bsm2731540qkp.100.2022.09.21.15.37.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 15:37:14 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Mike Kravetz , Muchun Song , Florian Fainelli , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Doug Berger Subject: [PATCH 1/3] mm/hugetlb: refactor alloc_and_dissolve_huge_page Date: Wed, 21 Sep 2022 15:36:37 -0700 Message-Id: <20220921223639.1152392-2-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220921223639.1152392-1-opendmb@gmail.com> References: <20220921223639.1152392-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663799837; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W65OPn/JICs8j0QopHJv5kNae6yu6rzXZKtwjhKQdDM=; b=F+XKKpJ/XGWwNRFR7H1Hk2BSd2bkYsZt1tMIoO2kZ/AAWtgBaCZDXjkrwDTmyE5xpZOld5 brCPyGaU49fMV5Nd/Qi18c+Rgd8XwDHhTdfbv3y5mT/LTtpbTdX779Zkzpps0kGDyK4ZDO yk/8zMxtR+TW9+eKTcyEf1zJJtt76zk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="BJt7G/2c"; spf=pass (imf27.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663799837; a=rsa-sha256; cv=none; b=VsBhxUfZgNJDJ3Wm45PxBmb824UQogYclM0UszJBpOUPyat2EnDqEn2Fnlsn3BoKvTO+cm KfTHL6ojzsyEYZfQgYQgPoI4fcPLGS2XzUVzcPpg2cWFuve8Z9fxJGfrUhQgo51S0VTBhJ 0QSi7dGS1l4wsjYk66okWrbupqqreTs= Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="BJt7G/2c"; spf=pass (imf27.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: xi8eidppfoe6d6bctnm8pr3ahz7s7y78 X-Rspamd-Queue-Id: 5F8C74000B X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1663799837-607468 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The alloc_replacement_page() and replace_hugepage() functions are created from code in the alloc_and_dissolve_huge_page() function to allow their reuse by the next commit. Signed-off-by: Doug Berger --- mm/hugetlb.c | 84 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 51 insertions(+), 33 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e070b8593b37..2b60de78007c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2709,32 +2709,22 @@ void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, } /* - * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old one - * @h: struct hstate old page belongs to - * @old_page: Old page to dissolve - * @list: List to isolate the page in case we need to - * Returns 0 on success, otherwise negated error. + * Before dissolving the page, we need to allocate a new one for the + * pool to remain stable. Here, we allocate the page and 'prep' it + * by doing everything but actually updating counters and adding to + * the pool. This simplifies and let us do most of the processing + * under the lock. */ -static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, - struct list_head *list) +static struct page *alloc_replacement_page(struct hstate *h, int nid) { gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; - int nid = page_to_nid(old_page); bool alloc_retry = false; struct page *new_page; - int ret = 0; - /* - * Before dissolving the page, we need to allocate a new one for the - * pool to remain stable. Here, we allocate the page and 'prep' it - * by doing everything but actually updating counters and adding to - * the pool. This simplifies and let us do most of the processing - * under the lock. - */ alloc_retry: new_page = alloc_buddy_huge_page(h, gfp_mask, nid, NULL, NULL); if (!new_page) - return -ENOMEM; + return ERR_PTR(-ENOMEM); /* * If all goes well, this page will be directly added to the free * list in the pool. For this the ref count needs to be zero. @@ -2748,7 +2738,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, SetHPageTemporary(new_page); if (!put_page_testzero(new_page)) { if (alloc_retry) - return -EBUSY; + return ERR_PTR(-EBUSY); alloc_retry = true; goto alloc_retry; @@ -2757,6 +2747,48 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, __prep_new_huge_page(h, new_page); + return new_page; +} + +static void replace_hugepage(struct hstate *h, int nid, struct page *old_page, + struct page *new_page) +{ + lockdep_assert_held(&hugetlb_lock); + /* + * Ok, old_page is still a genuine free hugepage. Remove it from + * the freelist and decrease the counters. These will be + * incremented again when calling __prep_account_new_huge_page() + * and enqueue_huge_page() for new_page. The counters will remain + * stable since this happens under the lock. + */ + remove_hugetlb_page(h, old_page, false); + + /* + * Ref count on new page is already zero as it was dropped + * earlier. It can be directly added to the pool free list. + */ + __prep_account_new_huge_page(h, nid); + enqueue_huge_page(h, new_page); +} + +/* + * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old one + * @h: struct hstate old page belongs to + * @old_page: Old page to dissolve + * @list: List to isolate the page in case we need to + * Returns 0 on success, otherwise negated error. + */ +static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, + struct list_head *list) +{ + int nid = page_to_nid(old_page); + struct page *new_page; + int ret = 0; + + new_page = alloc_replacement_page(h, nid); + if (IS_ERR(new_page)) + return PTR_ERR(new_page); + retry: spin_lock_irq(&hugetlb_lock); if (!PageHuge(old_page)) { @@ -2783,21 +2815,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, cond_resched(); goto retry; } else { - /* - * Ok, old_page is still a genuine free hugepage. Remove it from - * the freelist and decrease the counters. These will be - * incremented again when calling __prep_account_new_huge_page() - * and enqueue_huge_page() for new_page. The counters will remain - * stable since this happens under the lock. - */ - remove_hugetlb_page(h, old_page, false); - - /* - * Ref count on new page is already zero as it was dropped - * earlier. It can be directly added to the pool free list. - */ - __prep_account_new_huge_page(h, nid); - enqueue_huge_page(h, new_page); + replace_hugepage(h, nid, old_page, new_page); /* * Pages have been replaced, we can safely free the old one. From patchwork Wed Sep 21 22:36:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12984275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99A09ECAAD8 for ; Wed, 21 Sep 2022 22:37:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 321CB80009; Wed, 21 Sep 2022 18:37:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D0F280008; Wed, 21 Sep 2022 18:37:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 199DB80009; Wed, 21 Sep 2022 18:37:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0D6DF80008 for ; Wed, 21 Sep 2022 18:37:21 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DA81D406D4 for ; Wed, 21 Sep 2022 22:37:20 +0000 (UTC) X-FDA: 79937555040.12.750BCC7 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf28.hostedemail.com (Postfix) with ESMTP id A1F73C0013 for ; Wed, 21 Sep 2022 22:37:20 +0000 (UTC) Received: by mail-qt1-f177.google.com with SMTP id w2so5161592qtv.9 for ; Wed, 21 Sep 2022 15:37:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=WraM5AHrdGzNrVxYvq6E83xWiBcfqvWRynzaGQZpK+g=; b=PnTvaZrgzYl3A2zQXVHwCw9cXtSjz0IogQa6jGd78pmS8IycFfAtj41+CxxekNv5fz kuvppT0H+KuDkkMNqV8LQ95t3WDGOEE3eo0BxBPXBVwMEMdi+LDGq26Ir9GgHzCIGXJe wJIpLMQQcCYh9MAHP/GjpezCFX2ASWnTn/xG7CipnzEwQA8CIpcidkSKxS6wO4Q/pNsu wP9V0FO0IdH0eaI7hieeh96daJY5/ByFwjIanj8gX+tgpaCUpCJLg09vQMEwHfHHY1jk G7vGR2Yv0T4hHGeTnjsx4rz30v/KPEHXAxjqbuk7dMJlw+RXie6FGHIeCoLcGZkxKyBI eHPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=WraM5AHrdGzNrVxYvq6E83xWiBcfqvWRynzaGQZpK+g=; b=trg2goK2t3QX+Ln4ZoJxIfAe/ZJhVHpsx8kUP+a8/5iugLdb4QiLiZsa2aNIa/60xb +eMYzzBL7/ZU2k1Ftl9xUFEioH0gi1JZ7byRk6FxgNRPc7/zSstj67MdurZvLpT/vdI0 TzwGKYHDS0hkj/YJNNlzZt04EcPVLIbcQFvqBG5PahgKMVx6rJ7a+Nsqu89MKzpcGEHO VBV/78JMEJ37X9WK2sfKm1bvkm8dT2F5WUeA2eAjCtelnx1a+H+EpUiVXxZNt8oAC6/x ltXCqSl08GjU1G3ANZGG++P5EXe0fc2fNEFhmmqJJT1k8sLhgEx/8i73pFW92wDNIZek 7F5w== X-Gm-Message-State: ACrzQf2YiDLBUX8HrDQCsOsVk6ZtmY+n0IuiSOLxQkIQoKeLgNrqGZle 42VViMMLFgUmIx0AsYiBuDE= X-Google-Smtp-Source: AMsMyM73KuFt2vBnW+YpQZEA07l1sWlH98yMGHcP8J/1ai2vHwUIB0wwniLwB+can+pH+2j+VRjYNQ== X-Received: by 2002:a05:622a:1391:b0:35d:fb0:d830 with SMTP id o17-20020a05622a139100b0035d0fb0d830mr497328qtk.607.1663799839911; Wed, 21 Sep 2022 15:37:19 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id j13-20020a05620a288d00b006cea2984c9bsm2731540qkp.100.2022.09.21.15.37.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 15:37:19 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Mike Kravetz , Muchun Song , Florian Fainelli , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Doug Berger Subject: [PATCH 2/3] mm/hugetlb: allow migrated hugepage to dissolve when freed Date: Wed, 21 Sep 2022 15:36:38 -0700 Message-Id: <20220921223639.1152392-3-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220921223639.1152392-1-opendmb@gmail.com> References: <20220921223639.1152392-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663799840; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WraM5AHrdGzNrVxYvq6E83xWiBcfqvWRynzaGQZpK+g=; b=sHoNNccSQeHgbWzTnyhRIuQTgll+3UNZPAh+RFsR/djTyQa4uV9h+v5d6NjbuhuCKepP1j 1WaCypf5ShKo+RIYiJnL1H4UOdyyBbDvvlbhpZTxRXyTiOcW2+f+DtZTXN6Y39bMFPx9be +NjWMhw6UzzMQ8u0W6SBOeSMBVOBDAw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PnTvaZrg; spf=pass (imf28.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663799840; a=rsa-sha256; cv=none; b=h5aSsxL7689k8gG05CDsKNjIVbK/3DlFQuUKSWHMqWtgKtWvvzzjhVByJqt0H25kUKAz88 OprNjdBvqJG3d6D7J1S4cbObR4MSrzxNUgBuFISMaMAxKuAvKpWq/YhNx1LjpZm4Hv7Kg4 hW1w54natSZjN7IS4A3+NQJNlmJpyHs= X-Stat-Signature: 6fys168m3z8npnt471ygfjqh89csgpnk X-Rspamd-Queue-Id: A1F73C0013 X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PnTvaZrg; spf=pass (imf28.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam09 X-HE-Tag: 1663799840-623993 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is no isolation mechanism for hugepages so a hugepage that is migrated is returned to its hugepage freelist. This creates problems for alloc_contig_range() because migrated hugepages can be allocated as migrate targets for subsequent hugepage migration attempts. Even if the migration succeeds the alloc_contig_range() attempt will fail because test_pages_isolated() will find the now free hugepages haven't been dissolved. A subsequent attempt by alloc_contig_range() is necessary for the isolate_migratepages_range() function to find the freed hugepage and dissolve it (assuming it has not been reallocated). A workqueue is introduced to perform the equivalent functionality of alloc_and_dissolve_huge_page() for a migrated hugepage when it is freed so that the pages can be released to the isolated page lists of the buddy allocator allowing the alloc_contig_range() attempt to succeed. The HPG_dissolve hugepage flag is introduced to allow tagging migratable hugepages that should be dissolved when freed. Signed-off-by: Doug Berger --- include/linux/hugetlb.h | 5 +++ mm/hugetlb.c | 72 ++++++++++++++++++++++++++++++++++++++--- mm/migrate.c | 1 + mm/page_alloc.c | 1 + 4 files changed, 75 insertions(+), 4 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3ec981a0d8b3..0e6e21805e51 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -222,6 +222,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, bool is_hugetlb_entry_migration(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); +void sync_hugetlb_dissolve(void); #else /* !CONFIG_HUGETLB_PAGE */ @@ -430,6 +431,8 @@ static inline vm_fault_t hugetlb_fault(struct mm_struct *mm, static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { } +static inline void sync_hugetlb_dissolve(void) { } + #endif /* !CONFIG_HUGETLB_PAGE */ /* * hugepages at page global directory. If arch support @@ -574,6 +577,7 @@ enum hugetlb_page_flags { HPG_freed, HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, + HPG_dissolve, __NR_HPAGEFLAGS, }; @@ -621,6 +625,7 @@ HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) +HPAGEFLAG(Dissolve, dissolve) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2b60de78007c..eab812760fbe 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1582,6 +1582,10 @@ static void __update_and_free_page(struct hstate *h, struct page *page) } } +static LLIST_HEAD(hpage_dissolvelist); +static void dissolve_hpage_workfn(struct work_struct *work); +static DECLARE_WORK(dissolve_hpage_work, dissolve_hpage_workfn); + /* * As update_and_free_page() can be called under any context, so we cannot * use GFP_KERNEL to allocate vmemmap pages. However, we can defer the @@ -1628,6 +1632,8 @@ static inline void flush_free_hpage_work(struct hstate *h) { if (hugetlb_vmemmap_optimizable(h)) flush_work(&free_hpage_work); + if (!hstate_is_gigantic(h)) + flush_work(&dissolve_hpage_work); } static void update_and_free_page(struct hstate *h, struct page *page, @@ -1679,7 +1685,7 @@ void free_huge_page(struct page *page) struct hstate *h = page_hstate(page); int nid = page_to_nid(page); struct hugepage_subpool *spool = hugetlb_page_subpool(page); - bool restore_reserve; + bool restore_reserve, dissolve; unsigned long flags; VM_BUG_ON_PAGE(page_count(page), page); @@ -1691,6 +1697,8 @@ void free_huge_page(struct page *page) page->mapping = NULL; restore_reserve = HPageRestoreReserve(page); ClearHPageRestoreReserve(page); + dissolve = HPageDissolve(page); + ClearHPageDissolve(page); /* * If HPageRestoreReserve was set on page, page allocation consumed a @@ -1729,6 +1737,11 @@ void free_huge_page(struct page *page) remove_hugetlb_page(h, page, true); spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page, true); + } else if (dissolve) { + spin_unlock_irqrestore(&hugetlb_lock, flags); + if (llist_add((struct llist_node *)&page->mapping, + &hpage_dissolvelist)) + schedule_work(&dissolve_hpage_work); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); @@ -2771,6 +2784,49 @@ static void replace_hugepage(struct hstate *h, int nid, struct page *old_page, enqueue_huge_page(h, new_page); } +static void dissolve_hpage_workfn(struct work_struct *work) +{ + struct llist_node *node; + + node = llist_del_all(&hpage_dissolvelist); + + while (node) { + struct page *oldpage, *newpage; + struct hstate *h; + int nid; + + oldpage = container_of((struct address_space **)node, + struct page, mapping); + node = node->next; + oldpage->mapping = NULL; + + h = page_hstate(oldpage); + nid = page_to_nid(oldpage); + + newpage = alloc_replacement_page(h, nid); + + spin_lock_irq(&hugetlb_lock); + /* finish freeing oldpage */ + arch_clear_hugepage_flags(oldpage); + enqueue_huge_page(h, oldpage); + if (IS_ERR(newpage)) { + /* cannot dissolve so just leave free */ + spin_unlock_irq(&hugetlb_lock); + goto next; + } + + replace_hugepage(h, nid, oldpage, newpage); + + /* + * Pages have been replaced, we can safely free the old one. + */ + spin_unlock_irq(&hugetlb_lock); + __update_and_free_page(h, oldpage); +next: + cond_resched(); + } +} + /* * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old one * @h: struct hstate old page belongs to @@ -2803,6 +2859,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, */ spin_unlock_irq(&hugetlb_lock); ret = isolate_hugetlb(old_page, list); + SetHPageDissolve(old_page); spin_lock_irq(&hugetlb_lock); goto free_new; } else if (!HPageFreed(old_page)) { @@ -2864,14 +2921,21 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list) if (hstate_is_gigantic(h)) return -ENOMEM; - if (page_count(head) && !isolate_hugetlb(head, list)) + if (page_count(head) && !isolate_hugetlb(head, list)) { + SetHPageDissolve(head); ret = 0; - else if (!page_count(head)) + } else if (!page_count(head)) { ret = alloc_and_dissolve_huge_page(h, head, list); - + } return ret; } +void sync_hugetlb_dissolve(void) +{ + flush_work(&free_hpage_work); + flush_work(&dissolve_hpage_work); +} + struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) { diff --git a/mm/migrate.c b/mm/migrate.c index 6a1597c92261..b6c6123e614c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -141,6 +141,7 @@ void putback_movable_pages(struct list_head *l) list_for_each_entry_safe(page, page2, l, lru) { if (unlikely(PageHuge(page))) { + ClearHPageDissolve(page); putback_active_hugepage(page); continue; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e5486d47406e..6bf76bbc0308 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -9235,6 +9235,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, if (ret && ret != -EBUSY) goto done; ret = 0; + sync_hugetlb_dissolve(); /* * Pages from [start, end) are within a pageblock_nr_pages From patchwork Wed Sep 21 22:36:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12984276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8BB0C32771 for ; Wed, 21 Sep 2022 22:37:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 607FD8000A; Wed, 21 Sep 2022 18:37:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B72D80008; Wed, 21 Sep 2022 18:37:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40A5C8000A; Wed, 21 Sep 2022 18:37:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 344F980008 for ; Wed, 21 Sep 2022 18:37:23 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 095D31C3F88 for ; Wed, 21 Sep 2022 22:37:23 +0000 (UTC) X-FDA: 79937555166.29.DD91AA1 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf17.hostedemail.com (Postfix) with ESMTP id BB9DE40057 for ; Wed, 21 Sep 2022 22:37:22 +0000 (UTC) Received: by mail-qt1-f181.google.com with SMTP id g12so5190934qts.1 for ; Wed, 21 Sep 2022 15:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=LmEVzalCJFIRHw39sUF5chZzrB9L9Ol3S17vl8cf3+Y=; b=aZdar3AJ1DG3bz5HOxv9V7SzAwaMBzRGfUiS2XhA18Fcs8H4Ak5WpiBR1K2Se7ALpk bO5X4Q3hs7tZXOHJ7kqDEFfkGSd5t1TshU98QnDHGeXRCCBrctK0s1+vmE/l9n+wMFEy KmuZhTGthHpG1FGxaAPgsnbYGSp3KhhNzHIGR4IpQ2IgFZn8JdyeTULbj5YlXASe6zfd +Jw+BUD1TUAvRH2kEJfCf3ILMgbKZCaqj3qJFLPXK0pLCCS4K5bnURD0A52RmHGkxMhB HN3Xhv0e+DFmztWDUwIiDLA5nfC0eLWMpyhpTMzmJLl6IKZ0XkD8fK0b08YlrPT44Rke zrrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=LmEVzalCJFIRHw39sUF5chZzrB9L9Ol3S17vl8cf3+Y=; b=wKWTkBpZwbw2l0BSDa58FFozmeuwOB4tbghz1//yVB3nkUMEDvmYMrIX5AgpRDNIHc 58Cx1zpQDSdPPJL5wTdgLq5dJXOtJJHDOSvlN6aObjN1N/zZnOzyLwaP9jRbIpdmKJ4B olQdVYcpOyqbBPgwQ9VEvVBFWVERqLNes/H3dQHObtTfeidPL5QfSMQZCzN17X67newj RwGI9E5uV2afIZRj9+fbu9dx7nzOA4ix99fHIWHxxh4s7wAtNfS1aobUsut02e71W5Lq J7jmSaP57sqrh8fMMo5tlDqMabkwaY2B1oq6khS4m3OvAMGZqQIdSaCpAueFPuj8GLhN N+TQ== X-Gm-Message-State: ACrzQf14N4r3Zouq2uc2V6yT6Wt6VVKpSeYXFfyGXxLc0OiSjLHdxfN6 ODb6r6Lpxd8tsuy/wY/nXqCdHAXTPO8= X-Google-Smtp-Source: AMsMyM6pGX6zsqz0C40T23b+r6f11juvY3D4Nzd4CRClOWX9iALWXgBruz8A7GfTwra9HpWNdE+06Q== X-Received: by 2002:a05:622a:1015:b0:35a:de14:45b1 with SMTP id d21-20020a05622a101500b0035ade1445b1mr569768qte.348.1663799842103; Wed, 21 Sep 2022 15:37:22 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id j13-20020a05620a288d00b006cea2984c9bsm2731540qkp.100.2022.09.21.15.37.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 15:37:21 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Mike Kravetz , Muchun Song , Florian Fainelli , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Doug Berger Subject: [PATCH 3/3] mm/hugetlb: add hugepage isolation support Date: Wed, 21 Sep 2022 15:36:39 -0700 Message-Id: <20220921223639.1152392-4-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220921223639.1152392-1-opendmb@gmail.com> References: <20220921223639.1152392-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663799842; a=rsa-sha256; cv=none; b=YbSQtfY0nJgKnOaw3nECQt+icNWEKXs5qIFzX0AFa1LonsVQ3uRxwMugn8gWg/fbQfflvb kG/G/QMQQ0pDxpnwuHM8/zUsUBzk8efMv4isV0hk2SitLcVdufucNdqnPwi1tksc8OS2Rn SzaaLgcGfSIAZXX4FRrnt8y6PBuvWKs= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=aZdar3AJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663799842; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LmEVzalCJFIRHw39sUF5chZzrB9L9Ol3S17vl8cf3+Y=; b=7A1f/QMK7CVrO5u72wqtG4sJA+X3R3aBzVnEiMUP5EroRCxNmO1fQ08xxGpaRwDY3UU4bd wIB0o+w3Lzzy+OLqKIl3RPyK1CjlFnrO7gzkzzH5m141g1YS+8Gn/2LJyf3SjPzFSHnROA ihYKLNdkv55atj8eC+kIgDXKXZv+woU= X-Stat-Signature: 9md8a7wpb5gqiasxr14szr1jh71xqda7 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: BB9DE40057 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=aZdar3AJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspam-User: X-HE-Tag: 1663799842-280256 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a range of pageblocks is isolated there is at most one hugepage that has only tail pages overlapping that range (i.e. a hugepage that overlaps the beginning of the range). However, that hugepage is the first migration target for an alloc_contig_range() attempt so it already receives special attention. Checking whether the pageblock containing the head of a hugepage is isolated is an inexpensive way to avoid hugepage allocations from isolated pageblocks which makes alloc_contig_range() more efficient. Signed-off-by: Doug Berger --- mm/hugetlb.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index eab812760fbe..3a2f0b55059d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include @@ -1135,6 +1136,10 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) if (PageHWPoison(page)) continue; + /* Check head pageblock isolation */ + if (is_migrate_isolate_page(page)) + continue; + list_move(&page->lru, &h->hugepage_activelist); set_page_refcounted(page); ClearHPageFreed(page);