From patchwork Fri Jan 22 19:52:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12040369 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A91DC433DB for ; Fri, 22 Jan 2021 19:53:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 966C723B06 for ; Fri, 22 Jan 2021 19:53:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 966C723B06 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 15C416B0007; Fri, 22 Jan 2021 14:53:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 10B8B6B000A; Fri, 22 Jan 2021 14:53:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEFD46B000C; Fri, 22 Jan 2021 14:53:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0206.hostedemail.com [216.40.44.206]) by kanga.kvack.org (Postfix) with ESMTP id CFEFB6B0007 for ; Fri, 22 Jan 2021 14:53:30 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 958CB8249980 for ; Fri, 22 Jan 2021 19:53:30 +0000 (UTC) X-FDA: 77734460580.15.skate87_63159312756e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 74C951814C707 for ; Fri, 22 Jan 2021 19:53:30 +0000 (UTC) X-HE-Tag: skate87_63159312756e X-Filterd-Recvd-Size: 21050 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Fri, 22 Jan 2021 19:53:29 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJndfM171767; Fri, 22 Jan 2021 19:53:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=h5f/U1VuxRafr9mBGsFQdTJrhqpxcwXxUuFcGTNnNds=; b=ub2C7P42OYG0Z6LBdUqzUCoz3P/kRbJcsY0Yqh+rvwXtLBlA47+ohk7MJQ80+vbfvlmt CH0x5EzuGYa3WSQsFHjgFx7oJlvUo6DKJU8wyFtL/qRTU5hd4EAtBPh5eQuqBCnLmAtd +Bc9Sn6C7AGpH0dCe36pcPFfEZtP+bi7PnBg/DTQvKS6x0noDsNEXmT39RCahl6C2jz6 KL21yM9tYC9/c6Ppevr/c544edl2kg5KznXyVKSELAuVvLW5W6PrU0Cul00sQ+KthFHk WjAXwKH/IkyMmUMDLG4F0mHDQJogP5JZTwo5EDdTkpt8ZpshcndqWqzIYAtvXCRgbnAp Tg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 3668qanv9j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:09 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJoxRd141347; Fri, 22 Jan 2021 19:53:09 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2172.outbound.protection.outlook.com [104.47.56.172]) by userp3020.oracle.com with ESMTP id 3668r1h9d4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mh2UCMqzjHDMAHMuDWSigvdUxO9R0CwydIitR2yjCOLU4t36hNd+igfhnqj8BKBSx0J4fqZNUT861hFDYvW8HjfEWXNK/bfB59S9F37WaFnvTfphe1wJnxApGji9Fqvl6fq4n6oRa2oDwuitTW9AxvASVC3NI1CyvgQxU5a+dran4MFX/BJESISiOPDHU41gVlE061F9CMNxP8XvAkVzXCVMT548gYK9L2lwUdp4jQ1rsAMHzlCanXZLNbk5ojyOomV2S5nttCQI25uvgOtsozk+DWJZ/FJm2YLlrn9CEE9HI+bDrujGehdN0IhbiInP9ivg+7LcFyEspz7sEdRPmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h5f/U1VuxRafr9mBGsFQdTJrhqpxcwXxUuFcGTNnNds=; b=l5obcFey/OBtYfqEa59hAUEq86pCtO9+iyXZwBKTHVsa44UWzumhjIBtF355Pib+eVklZTKESHNqE58fzdoi7ciZP7w28ZVRPcIsT11WlOYm/w4UZ2j3X4gcaqGAwwEbNp6Mt6SjIRpi0W+rAeI9AJV79oPNLtbiesrgisUDiHJvPASsTeFRxW40f+0gFDK8/ku6YwAHl6wJVjV3JozNAVuzzpFvJYNTE56HMewwgv2UD2MBYBIInGYA6tg5ZYYK+tawr0MDC1twkStFKjEDWs9w5aBvmJvqPSe76HjOklIAZZbzweeQ0b0aeonsfQg8BSWP0CzBZ0k+LY2qBEvdTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h5f/U1VuxRafr9mBGsFQdTJrhqpxcwXxUuFcGTNnNds=; b=XqBukEKJSOfYbN8cWfhQ1vKMI2BYzdHIzqSh2iZoGC+4GMiU+BPPA4zA2Jy4SzALXsduy1fC7fQUzGw+i8lWNA65yUIR5MnfQwIdKecbKrT3F+29zW1ELuhi5ET7L15/3kESPMGQmqe6aQ8MPvrV/1rzqd7tBzajqldlCNMTY9k= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4723.namprd10.prod.outlook.com (2603:10b6:303:9c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.12; Fri, 22 Jan 2021 19:53:06 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3784.015; Fri, 22 Jan 2021 19:53:06 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Miaohe Lin , Andrew Morton , Mike Kravetz Subject: [PATCH v3 1/5] hugetlb: use page.private for hugetlb specific page flags Date: Fri, 22 Jan 2021 11:52:27 -0800 Message-Id: <20210122195231.324857-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210122195231.324857-1-mike.kravetz@oracle.com> References: <20210122195231.324857-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.15 via Frontend Transport; Fri, 22 Jan 2021 19:53:05 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 44c18e22-1a88-447b-01ea-08d8bf0f55f0 X-MS-TrafficTypeDiagnostic: CO1PR10MB4723: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ogU7icpCxE4F26mZYN90jHR+ago/o47O0aLJuhAPKj88JuLGDqvgum1qxBMUXMK6pOd8/hQRbgPiSDlGBkcVGP020IZg0H2c4iZsAB4GXwJ5rOJXPnXYyU/Tm7VCMAoKsG4KgIwiA0rVi/l4RSKzNafXdVzP8otpiIwpDLMvquhxhLiUCYtAh5nuiXc+N/P5fliL+60gvB5SLrSaBAgn2bcMfY04gSVWd23FjcBzdhvM8UTHscvFl03tI0int7lGffGP7vGsvGPTiXqMNaPCTidd00oSdKxkqlGVmFZUBQB9T/3Qh+8AcS/zLgM4tot2/pO12bICo+ihcY0auPJc0wvG1LyYDXTEe2+fmxVyvY7WPCWdY3YwSNJpdgzMYjlg X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(396003)(376002)(366004)(346002)(956004)(16526019)(186003)(2906002)(8676002)(2616005)(6486002)(8936002)(36756003)(44832011)(52116002)(478600001)(86362001)(26005)(66946007)(107886003)(7416002)(1076003)(83380400001)(66556008)(66476007)(6666004)(7696005)(30864003)(316002)(5660300002)(54906003)(4326008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: iXJthl1QHzLnKI6xdRuQbEn+Z5/EwElRJb+WOJ/x86G0OowKfVHPynxuVhQRxf8xiBMH9WJWokkfw4bl0rZXWpHZedAU+HPTMpfjZVXGadcUHh/Shjym/QgzGWvShMrat9ywtcRIUgiCsW54HEXlIRum4mFWL6R9lAx8nZXM3Jr5rlWfulcAQOPNySR9KFNazUtR+cJpmm60WV1k+gB27M8HowgoXY73JkXR4Q1Ezzauz82P49ivh6MheUkeIAJurV5hO/E69Zre8JMyMxjeMu3KEl9vkmxm5Rdj5i+PFrv2OsjYNyY86beVnE1edrNw9x7zqpUlbQlsD1htcknscCB3EkhzUAlbq0C53puLACW+IvHSGh+apzx/lIGIlgTwzgkSnIxOp/sD1ArIYJglOnz8hCu0GWP+nbAaVsmXy9cpqYCr9RPWFQk492ZY2ccrkxRvG8oARK9HpeDh/fpl/VbcLUMTibtxN1xbpf8hDE08vjcQKhnb1qQ9D0Dxhh+Dl8+0sx30MIJe2CmjVxk9dKbAQP4voeotlY+JCt6AEDC2SphEcH7hzrTPoeaPaZ8ppITIkLEgnlzFjJHWlvr+fSMIDw3IuomSwJaqhGKeJLNa7XS2fFj/g4KyKqKE+NM5HO0ktiDgGF63cd+h1fmP1BBiALbGIFELHulK9Kcdbn1dgXtcqJKjcXvahUjTrdRItUQiJr6P69Xo2z+V4Kbv9Nt9j4c+xaT1ZDn8PVinzw77A/BnhmYmPXu/anFJEIIF3TeZxM/Wwd+SfNAR24ho3J1/GqbzKdqkI7TSAwrOisvJnbv45MmgTFbCkEY4dxpiGKEGxLDS/sYoZPraV8AraSaCGFT4KRPnYyCZmU9kaXHsk1BGZgV7JhDpDZRBqhOnSN52vbA5gp9LUx04y7iuSsNm7szgb2+mb4zgQ8vaTLzQh04rCDiSKsw/4A2tx7dH2i3SrnZrzIfEQemzth1CjMqSvyZDmIdagejLV9UlQRHKq8Qa98r13JkBMsqwxX+l3pOBxWwtAizCKDUiliDQyvkrxuPyoLrksKaC1GzvLuSiYUWkgl9tZVa+RqwMHQpA X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 44c18e22-1a88-447b-01ea-08d8bf0f55f0 X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2021 19:53:06.2111 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +yYZIM15l1MrMDbnb0L+cHrthdWikjdaWyESjk7t3P6dwYB5nM9es74vN81bwcucAeP/ZjDCx2zJvi8bQGOEYg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4723 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 adultscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 impostorscore=0 mlxscore=0 priorityscore=1501 phishscore=0 mlxlogscore=999 lowpriorityscore=0 malwarescore=0 adultscore=0 clxscore=1015 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As hugetlbfs evolved, state information about hugetlb pages was added. One 'convenient' way of doing this was to use available fields in tail pages. Over time, it has become difficult to know the meaning or contents of fields simply by looking at a small bit of code. Sometimes, the naming is just confusing. For example: The PagePrivate flag indicates a huge page reservation was consumed and needs to be restored if an error is encountered and the page is freed before it is instantiated. The page.private field contains the pointer to a subpool if the page is associated with one. In an effort to make the code more readable, use page.private to contain hugetlb specific page flags. These flags will have test, set and clear functions similar to those used for 'normal' page flags. More importantly, an enum of flag values will be created with names that actually reflect their purpose. In this patch, - Create infrastructure for hugetlb specific page flag functions - Move subpool pointer to page[1].private to make way for flags Create routines with meaningful names to modify subpool field - Use new HPageRestoreReserve flag instead of PagePrivate Conversion of other state information will happen in subsequent patches. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Acked-by: Michal Hocko --- fs/hugetlbfs/inode.c | 12 ++------ include/linux/hugetlb.h | 68 +++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 48 +++++++++++++++-------------- 3 files changed, 96 insertions(+), 32 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 21c20fd5f9ee..b00801fd6002 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -966,15 +966,9 @@ static int hugetlbfs_migrate_page(struct address_space *mapping, if (rc != MIGRATEPAGE_SUCCESS) return rc; - /* - * page_private is subpool pointer in hugetlb pages. Transfer to - * new page. PagePrivate is not associated with page_private for - * hugetlb pages and can not be set here as only page_huge_active - * pages can be migrated. - */ - if (page_private(page)) { - set_page_private(newpage, page_private(page)); - set_page_private(page, 0); + if (hugetlb_page_subpool(page)) { + hugetlb_set_page_subpool(newpage, hugetlb_page_subpool(page)); + hugetlb_set_page_subpool(page, NULL); } if (mode != MIGRATE_SYNC_NO_COPY) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index b5807f23caf8..a7eb05315c6e 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -472,6 +472,60 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, unsigned long flags); #endif /* HAVE_ARCH_HUGETLB_UNMAPPED_AREA */ +/* + * huegtlb page specific state flags. These flags are located in page.private + * of the hugetlb head page. Functions created via the below macros should be + * used to manipulate these flags. + * + * HPG_restore_reserve - Set when a hugetlb page consumes a reservation at + * allocation time. Cleared when page is fully instantiated. Free + * routine checks flag to restore a reservation on error paths. + */ +enum hugetlb_page_flags { + HPG_restore_reserve = 0, + __NR_HPAGEFLAGS, +}; + +/* + * Macros to create test, set and clear function definitions for + * hugetlb specific page flags. + */ +#ifdef CONFIG_HUGETLB_PAGE +#define TESTHPAGEFLAG(uname, flname) \ +static inline int HPage##uname(struct page *page) \ + { return test_bit(HPG_##flname, &(page->private)); } + +#define SETHPAGEFLAG(uname, flname) \ +static inline void SetHPage##uname(struct page *page) \ + { set_bit(HPG_##flname, &(page->private)); } + +#define CLEARHPAGEFLAG(uname, flname) \ +static inline void ClearHPage##uname(struct page *page) \ + { clear_bit(HPG_##flname, &(page->private)); } +#else +#define TESTHPAGEFLAG(uname, flname) \ +static inline int HPage##uname(struct page *page) \ + { return 0; } + +#define SETHPAGEFLAG(uname, flname) \ +static inline void SetHPage##uname(struct page *page) \ + { } + +#define CLEARHPAGEFLAG(uname, flname) \ +static inline void ClearHPage##uname(struct page *page) \ + { } +#endif + +#define HPAGEFLAG(uname, flname) \ + TESTHPAGEFLAG(uname, flname) \ + SETHPAGEFLAG(uname, flname) \ + CLEARHPAGEFLAG(uname, flname) \ + +/* + * Create functions associated with hugetlb page flags + */ +HPAGEFLAG(RestoreReserve, restore_reserve) + #ifdef CONFIG_HUGETLB_PAGE #define HSTATE_NAME_LEN 32 @@ -531,6 +585,20 @@ extern unsigned int default_hstate_idx; #define default_hstate (hstates[default_hstate_idx]) +/* + * hugetlb page subpool pointer located in hpage[1].private + */ +static inline struct hugepage_subpool *hugetlb_page_subpool(struct page *hpage) +{ + return (struct hugepage_subpool *)(hpage+1)->private; +} + +static inline void hugetlb_set_page_subpool(struct page *hpage, + struct hugepage_subpool *subpool) +{ + set_page_private(hpage+1, (unsigned long)subpool); +} + static inline struct hstate *hstate_file(struct file *f) { return hstate_inode(file_inode(f)); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d78111f0fa2c..8ec6138ca81b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1133,7 +1133,7 @@ static struct page *dequeue_huge_page_vma(struct hstate *h, nid = huge_node(vma, address, gfp_mask, &mpol, &nodemask); page = dequeue_huge_page_nodemask(h, gfp_mask, nid, nodemask); if (page && !avoid_reserve && vma_has_reserves(vma, chg)) { - SetPagePrivate(page); + SetHPageRestoreReserve(page); h->resv_huge_pages--; } @@ -1407,20 +1407,19 @@ static void __free_huge_page(struct page *page) */ struct hstate *h = page_hstate(page); int nid = page_to_nid(page); - struct hugepage_subpool *spool = - (struct hugepage_subpool *)page_private(page); + struct hugepage_subpool *spool = hugetlb_page_subpool(page); bool restore_reserve; VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(page_mapcount(page), page); - set_page_private(page, 0); + hugetlb_set_page_subpool(page, NULL); page->mapping = NULL; - restore_reserve = PagePrivate(page); - ClearPagePrivate(page); + restore_reserve = HPageRestoreReserve(page); + ClearHPageRestoreReserve(page); /* - * If PagePrivate() was set on page, page allocation consumed a + * If HPageRestoreReserve was set on page, page allocation consumed a * reservation. If the page was associated with a subpool, there * would have been a page reserved in the subpool before allocation * via hugepage_subpool_get_pages(). Since we are 'restoring' the @@ -2251,24 +2250,24 @@ static long vma_add_reservation(struct hstate *h, * This routine is called to restore a reservation on error paths. In the * specific error paths, a huge page was allocated (via alloc_huge_page) * and is about to be freed. If a reservation for the page existed, - * alloc_huge_page would have consumed the reservation and set PagePrivate - * in the newly allocated page. When the page is freed via free_huge_page, - * the global reservation count will be incremented if PagePrivate is set. - * However, free_huge_page can not adjust the reserve map. Adjust the - * reserve map here to be consistent with global reserve count adjustments - * to be made by free_huge_page. + * alloc_huge_page would have consumed the reservation and set + * HPageRestoreReserve in the newly allocated page. When the page is freed + * via free_huge_page, the global reservation count will be incremented if + * HPageRestoreReserve is set. However, free_huge_page can not adjust the + * reserve map. Adjust the reserve map here to be consistent with global + * reserve count adjustments to be made by free_huge_page. */ static void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, unsigned long address, struct page *page) { - if (unlikely(PagePrivate(page))) { + if (unlikely(HPageRestoreReserve(page))) { long rc = vma_needs_reservation(h, vma, address); if (unlikely(rc < 0)) { /* * Rare out of memory condition in reserve map - * manipulation. Clear PagePrivate so that + * manipulation. Clear HPageRestoreReserve so that * global reserve count will not be incremented * by free_huge_page. This will make it appear * as though the reservation for this page was @@ -2277,7 +2276,7 @@ static void restore_reserve_on_error(struct hstate *h, * is better than inconsistent global huge page * accounting of reserve counts. */ - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); } else if (rc) { rc = vma_add_reservation(h, vma, address); if (unlikely(rc < 0)) @@ -2285,7 +2284,7 @@ static void restore_reserve_on_error(struct hstate *h, * See above comment about rare out of * memory condition. */ - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); } else vma_end_reservation(h, vma, address); } @@ -2366,7 +2365,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, if (!page) goto out_uncharge_cgroup; if (!avoid_reserve && vma_has_reserves(vma, gbl_chg)) { - SetPagePrivate(page); + SetHPageRestoreReserve(page); h->resv_huge_pages--; } spin_lock(&hugetlb_lock); @@ -2384,7 +2383,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, spin_unlock(&hugetlb_lock); - set_page_private(page, (unsigned long)spool); + hugetlb_set_page_subpool(page, spool); map_commit = vma_commit_reservation(h, vma, addr); if (unlikely(map_chg > map_commit)) { @@ -3156,6 +3155,9 @@ static int __init hugetlb_init(void) { int i; + BUILD_BUG_ON(sizeof_field(struct page, private) * BITS_PER_BYTE < + __NR_HPAGEFLAGS); + if (!hugepages_supported()) { if (hugetlb_max_hstate || default_hstate_max_huge_pages) pr_warn("HugeTLB: huge pages not supported, ignoring associated command-line parameters\n"); @@ -4207,7 +4209,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, spin_lock(ptl); ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { - ClearPagePrivate(new_page); + ClearHPageRestoreReserve(new_page); /* Break COW */ huge_ptep_clear_flush(vma, haddr, ptep); @@ -4274,7 +4276,7 @@ int huge_add_to_page_cache(struct page *page, struct address_space *mapping, if (err) return err; - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); /* * set page dirty so that it will not be removed from cache/file @@ -4436,7 +4438,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto backout; if (anon_rmap) { - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); hugepage_add_new_anon_rmap(page, vma, haddr); } else page_dup_rmap(page, true); @@ -4750,7 +4752,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, if (vm_shared) { page_dup_rmap(page, true); } else { - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); hugepage_add_new_anon_rmap(page, dst_vma, dst_addr); } From patchwork Fri Jan 22 19:52:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12040373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E12F2C433E6 for ; Fri, 22 Jan 2021 19:53:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7535523B01 for ; Fri, 22 Jan 2021 19:53:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7535523B01 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0E7A46B000C; Fri, 22 Jan 2021 14:53:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 095D26B000D; Fri, 22 Jan 2021 14:53:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2BDF6B000E; Fri, 22 Jan 2021 14:53:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0109.hostedemail.com [216.40.44.109]) by kanga.kvack.org (Postfix) with ESMTP id CB46F6B000C for ; Fri, 22 Jan 2021 14:53:40 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 95A551800E8E0 for ; Fri, 22 Jan 2021 19:53:40 +0000 (UTC) X-FDA: 77734461000.04.front03_5d139392756e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 7025F801D241 for ; Fri, 22 Jan 2021 19:53:40 +0000 (UTC) X-HE-Tag: front03_5d139392756e X-Filterd-Recvd-Size: 17665 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 22 Jan 2021 19:53:39 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJmsV8170514; Fri, 22 Jan 2021 19:53:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=o4AB3ojJGlCjqbBcwbXii0wGgs998LKazPmP6JzeEbo=; b=o4Cm+ah4JRUvnLo2FkHIOtArn3mMxxTVU5Rp0M/n3pi5CBDeoKygu14tZQcm6I1vFF48 V4GNK5ml6yehuo8fL0hL8azg/U2bMScdiJc7N+eiFfJ91SKXhObWSAhiCj/SeR4o5C21 RhPgp8wj/Nq0zOzR49EJi2bXZaupQNPtD105ZkTQcJk4eQt9V69LMyRc5n+t4EtuUNGx HQ2kVJnkgiEucQaP3kRTMxQm5QpGDgjtKbhfLAn8C95JbGp7jd0prGbJ6UFlAYsFYDYz VuB4kUvoUIV3lI2vjL29qkptlEid9ATww5odujyY6OaHq4PPQOff1hZRyTR4TD3OSxBC QA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 3668qrnvey-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:12 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJoAfb135241; Fri, 22 Jan 2021 19:53:12 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2172.outbound.protection.outlook.com [104.47.56.172]) by aserp3020.oracle.com with ESMTP id 3668rj2syu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Cz2fhMZ0ddzUSxstBwVV1E7l4mmuFA8BMxvvdhhBTEgzfycRgw3D4LfydOp3WFL3NU13TcIzivU9pS7pFNbxBiEnBYAFL0UF/53/3EXHyuQE6tYfyiF6ZhI8GDV9LwCNCQ89tNvmgTM7mfP44edL9dHZ4lZuAgVtCmvcurQbLPHCvRqxApA4D1uRx1ZGzTcEMQfKyGUg1Mcial4LFreWNPm8nPVDsC0/MgACuLV+GVvVIiPGhMJfHEktH1c8WEH0rEEZu5ckO2t3NtjZDvvMJPR7uf2hbZ+Y15G/0ug9jzULC8dxjY2x31lAxg8p/B7lWGYyYHiMEWsQFCscH9igQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o4AB3ojJGlCjqbBcwbXii0wGgs998LKazPmP6JzeEbo=; b=DvZ+BJO39RK54y4v0jOpxo2CxOenztZAaODlCcBDyOOHjUqIuxDYIfnGu+WmjnTaAtA9T4QjSEGCQZfBxXLx6w+3o/C4kKfYtUH2RcmsX4nbbCjLgLM/Yp1xUI07Tj+/QjudjK08mJElavlO7Vwa1IhSpf+EuLb1XwnuDBhzMQfF5fteFN5Q0weeVrYQK+yYhmIGgyQ0cezvm43dfcfuJvGkmoYzyQ5wYntKnOnOpJtnG6+oiLGsLq3IPrHoBUEkLek6K6qvxXa9DpfBu6lYuksV3wdUp+JEXHPzpM2xPgnpZwOEN+l+hKKtHHS1aHiWbubF3xIz+EyRDLvZueK2MA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o4AB3ojJGlCjqbBcwbXii0wGgs998LKazPmP6JzeEbo=; b=lG4jWU1lfER5J+1Cb75z/FpG0TasdKLswVNIL/444CEIeE7yU4FkMZGNJ/4IBG8CB1XvRI4SdXkI5Irm6a5Xy5CxWrJqpoXxAgxFD7fJ6v8h327VlhYusngl6B3XiP9fREXePTh3nmao2nYIPmqWK9LxDhESoNr5AjChxWjlIPk= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4723.namprd10.prod.outlook.com (2603:10b6:303:9c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.12; Fri, 22 Jan 2021 19:53:10 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3784.015; Fri, 22 Jan 2021 19:53:10 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Miaohe Lin , Andrew Morton , Mike Kravetz Subject: [PATCH v3 2/5] hugetlb: convert page_huge_active() HPageMigratable flag Date: Fri, 22 Jan 2021 11:52:28 -0800 Message-Id: <20210122195231.324857-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210122195231.324857-1-mike.kravetz@oracle.com> References: <20210122195231.324857-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.15 via Frontend Transport; Fri, 22 Jan 2021 19:53:09 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c239d85b-1af1-433b-53fa-08d8bf0f582d X-MS-TrafficTypeDiagnostic: CO1PR10MB4723: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SoTwNcl6v7MrJTiFxSmnvE6sCz6XKtgyhpCKlKg4/zXwjFfFstVhrOYn/Cw7kIy9NJLTpVeMFBlg63aniN+bzFG81EG+PEaQd+7d3LhMNRc0Gz70m7VJJlphUbeOMcezpWCIQaXLYxwnBl6JhjDu7gErAMMq64e6TORGzpfiIDSrtn1dvJ3uyYRrafc1BJsM87cXFhJeMMsuUInFrX67mgV6pZdoCSIBBV4s42YNhvHWISz+pB9ZCpBe9zQlZoHKbLih5IWpr89PGvSpqSA0ISzgS8x1KOeidJOVWXL3OKNt+SGImJAUIm7nE93kljDKKpSq9NUZi1yh7bvDVt4pgWBlLfE4yZ+gq8UDcRNXhCcSgQoaHE1vrSquG1MwRl48PHGX/XejVXxscSwH+H8iI8WbmL0hMHGBAfVcbMb17M5TRZrZ0vZ5Tsu3IqRREgY7 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(396003)(376002)(366004)(346002)(956004)(16526019)(186003)(2906002)(8676002)(2616005)(6486002)(8936002)(36756003)(44832011)(52116002)(478600001)(86362001)(26005)(66946007)(107886003)(7416002)(1076003)(83380400001)(66556008)(66476007)(6666004)(7696005)(316002)(5660300002)(54906003)(4326008)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: zastUYBfdtl3ZJnu8DQFq7gtrsnaHL0AVe6TQmLam8f+TOhS/ctf9ijDKySvmZOFrSQFfbDM81XPtOBapmQu+rAFVeVfrohrDH+8XjNXObIcBcp8a8d0vHwOqFFcwtKVtoU8WtAyTV5ctH7p94E2q6drUIC53TvQVO4uc6t7ym+E7CcTHFIgNYmfWbqXY672N/5lzaiRe/KTbTGL29wUXaFEihnADPJHRR1xiABe+N+Swdx/AsSm0BJPY7UZdFiAKppiX9hW2jKIfzCwUP1m0yKOiVFLFnhH3T4Bdua+QPkG3AXAaGa77uNYb9uHzixY2NaNEPiSND34TfUWSg99Vz5MlY6tB0Jq+q5Djzx9S68J0UPZ2/4V16SAx3H0HboX3Hml18rRCDu7uOj6imVnIojEluu254nj7WTYDISwG+b0Ay+8fL649UmuZYKcjgSJPWuFwRShl1uLnnXCOVGO2+ugJCU/rXOWWpIrudw+FOHuX+Ijri3k8J2ft7mL23+aGWOOZSkXu2fslMun3/cbw/2Yga3b+mGgLOC/D7MvjPCgor2zXmTouzZibbVN4sLAl+v/wCxooDYuMrmsHp+gVyTAtHHtY0HLuaumDS5WzTL3cxD11Rs2w2eh+WBr/GDIPgIERTLwh16s7Ui/awTeyAZ4rdVV9g6jQdwskFavTJ0ObRF0ziUR/aHoUSi6XZE8181ttywr2K4m8S9e0RP4yeap3ym1ROZpZj7NNcjJ2I0sCvHoBKpYbjO2KeDu4atlubDPwxCODlNxZMs6KzuFwtM11GSeU+nQCsK9m6pZq5NKZpOquNKgd/RhNqbv23eSOO/VMT/yoqbLdxFTmP5ldR/wu1T9R52OXXqdSD7CI2FG7eAoZ5mxTKMEJRZYuGQn8G8VKOLg5Oj0cORgo4uIcqv/qSbhpTbFHtG8J2EYQ1SuEwQzc8ncUXicA8ljDqg47NesPkfN9brzdDLl7wq9k9JC3nUS6MkWerVrbFAimDfkp2D9Dqjj3AAHTbFuO6Vbo6oRuoieFFMPaVzg6bYeoXtc8jEZidZE7V1PbA5jxUFIVZbDbgukr0I1oOJCa9S1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c239d85b-1af1-433b-53fa-08d8bf0f582d X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2021 19:53:10.0094 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Cxfof56Xt/WcYJ3Bu1oqWSLk+0GdJTVe8F5YaNSCHs17iM4YbLkeqwJioWFFx8vgKQrpTBSFfPx9DP4LuRewMA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4723 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 suspectscore=0 lowpriorityscore=0 bulkscore=0 adultscore=0 spamscore=0 phishscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the new hugetlb page specific flag HPageMigratable to replace the page_huge_active interfaces. By it's name, page_huge_active implied that a huge page was on the active list. However, that is not really what code checking the flag wanted to know. It really wanted to determine if the huge page could be migrated. This happens when the page is actually added to the page cache and/or task page table. This is the reasoning behind the name change. The VM_BUG_ON_PAGE() calls in the *_huge_active() interfaces are not really necessary as we KNOW the page is a hugetlb page. Therefore, they are removed. The routine page_huge_active checked for PageHeadHuge before testing the active bit. This is unnecessary in the case where we hold a reference or lock and know it is a hugetlb head page. page_huge_active is also called without holding a reference or lock (scan_movable_pages), and can race with code freeing the page. The extra check in page_huge_active shortened the race window, but did not prevent the race. Offline code calling scan_movable_pages already deals with these races, so removing the check is acceptable. Add comment to racy code. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Reviewed-by: Muchun Song Reviewed-by: Miaohe Lin Acked-by: Michal Hocko --- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 5 +++++ include/linux/page-flags.h | 6 ----- mm/hugetlb.c | 45 ++++++++++---------------------------- mm/memory_hotplug.c | 9 +++++++- 5 files changed, 25 insertions(+), 42 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index b00801fd6002..e1d7ed2a53a9 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); - set_page_huge_active(page); + SetHPageMigratable(page); /* * unlock_page because locked by add_to_page_cache() * put_page() due to reference from alloc_huge_page() diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index a7eb05315c6e..58be44a915d1 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -480,9 +480,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HPG_restore_reserve - Set when a hugetlb page consumes a reservation at * allocation time. Cleared when page is fully instantiated. Free * routine checks flag to restore a reservation on error paths. + * HPG_migratable - Set after a newly allocated page is added to the page + * cache and/or page tables. Indicates the page is a candidate for + * migration. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, + HPG_migratable, __NR_HPAGEFLAGS, }; @@ -525,6 +529,7 @@ static inline void ClearHPage##uname(struct page *page) \ * Create functions associated with hugetlb page flags */ HPAGEFLAG(RestoreReserve, restore_reserve) +HPAGEFLAG(Migratable, migratable) #ifdef CONFIG_HUGETLB_PAGE diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index ec5d0290e0ee..db914477057b 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -592,15 +592,9 @@ static inline void ClearPageCompound(struct page *page) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(struct page *page); int PageHeadHuge(struct page *page); -bool page_huge_active(struct page *page); #else TESTPAGEFLAG_FALSE(Huge) TESTPAGEFLAG_FALSE(HeadHuge) - -static inline bool page_huge_active(struct page *page) -{ - return 0; -} #endif diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8ec6138ca81b..f1a3c8230dbf 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1353,30 +1353,6 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -/* - * Test to determine whether the hugepage is "active/in-use" (i.e. being linked - * to hstate->hugepage_activelist.) - * - * This function can be called for tail pages, but never returns true for them. - */ -bool page_huge_active(struct page *page) -{ - return PageHeadHuge(page) && PagePrivate(&page[1]); -} - -/* never called for tail page */ -void set_page_huge_active(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHeadHuge(page), page); - SetPagePrivate(&page[1]); -} - -static void clear_page_huge_active(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHeadHuge(page), page); - ClearPagePrivate(&page[1]); -} - /* * Internal hugetlb specific page flag. Do not use outside of the hugetlb * code @@ -1438,7 +1414,7 @@ static void __free_huge_page(struct page *page) } spin_lock(&hugetlb_lock); - clear_page_huge_active(page); + ClearHPageMigratable(page); hugetlb_cgroup_uncharge_page(hstate_index(h), pages_per_huge_page(h), page); hugetlb_cgroup_uncharge_page_rsvd(hstate_index(h), @@ -4218,7 +4194,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page, true); hugepage_add_new_anon_rmap(new_page, vma, haddr); - set_page_huge_active(new_page); + SetHPageMigratable(new_page); /* Make the old page be freed below */ new_page = old_page; } @@ -4455,12 +4431,12 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, spin_unlock(ptl); /* - * Only make newly allocated pages active. Existing pages found - * in the pagecache could be !page_huge_active() if they have been - * isolated for migration. + * Only set HPageMigratable in newly allocated pages. Existing pages + * found in the pagecache may not have HPageMigratableset if they have + * been isolated for migration. */ if (new_page) - set_page_huge_active(page); + SetHPageMigratable(page); unlock_page(page); out: @@ -4771,7 +4747,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, update_mmu_cache(dst_vma, dst_addr, dst_pte); spin_unlock(ptl); - set_page_huge_active(page); + SetHPageMigratable(page); if (vm_shared) unlock_page(page); ret = 0; @@ -5596,12 +5572,13 @@ bool isolate_huge_page(struct page *page, struct list_head *list) bool ret = true; spin_lock(&hugetlb_lock); - if (!PageHeadHuge(page) || !page_huge_active(page) || + if (!PageHeadHuge(page) || + !HPageMigratable(page) || !get_page_unless_zero(page)) { ret = false; goto unlock; } - clear_page_huge_active(page); + ClearHPageMigratable(page); list_move_tail(&page->lru, list); unlock: spin_unlock(&hugetlb_lock); @@ -5612,7 +5589,7 @@ void putback_active_hugepage(struct page *page) { VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - set_page_huge_active(page); + SetHPageMigratable(page); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); spin_unlock(&hugetlb_lock); put_page(page); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index f9d57b9be8c7..465164be4d2d 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1260,7 +1260,14 @@ static int scan_movable_pages(unsigned long start, unsigned long end, if (!PageHuge(page)) continue; head = compound_head(page); - if (page_huge_active(head)) + /* + * This test is racy as we hold no reference or lock. The + * hugetlb page could have been free'ed and head is no longer + * a hugetlb page before the following check. In such unlikely + * cases false positives and negatives are possible. Calling + * code must deal with these scenarios. + */ + if (HPageMigratable(head)) goto found; skip = compound_nr(head) - (page - head); pfn += skip - 1; From patchwork Fri Jan 22 19:52:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12040375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21F46C433E9 for ; Fri, 22 Jan 2021 19:53:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B7B1B23B01 for ; Fri, 22 Jan 2021 19:53:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B7B1B23B01 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 776266B000D; Fri, 22 Jan 2021 14:53:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FF4F6B000E; Fri, 22 Jan 2021 14:53:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5046E6B0010; Fri, 22 Jan 2021 14:53:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id 279416B000E for ; Fri, 22 Jan 2021 14:53:41 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D4ABC8249980 for ; Fri, 22 Jan 2021 19:53:40 +0000 (UTC) X-FDA: 77734461000.05.death32_5908b532756e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id B680D1801D300 for ; Fri, 22 Jan 2021 19:53:40 +0000 (UTC) X-HE-Tag: death32_5908b532756e X-Filterd-Recvd-Size: 13928 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Fri, 22 Jan 2021 19:53:38 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJmvn6170550; Fri, 22 Jan 2021 19:53:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=93yiyvI7Fr4YjpmwKXt61gLFVpnhbSwn2zLTRdOyRMM=; b=s/eSTXjbHlZ1DraHmI0rUCXudsnvYmuvLHlrD8230zmgyf6ABVorhk/ww2CwxZphGNpn MgDCk0prt4XE6l87ES///kxRoH+EiMnTPJ8hEx0RkGIMGdLeYYYt7F7bgTa2se3dJy5f Wzrq8Plczs/DaAShgFWmSHmY1a/mNBMIVXga3OIb81y5Rv2Gd1slzqVZ9XZxP+kXPEdY 97DaV3KxyV2KxWBMjgxg2ejBun9Cdw72xQNjPtOL3fqyEeI++o2+HlNyjHw9+KW54EDC d6n84KfMCHCqeTEskR3GG/BNEHt6cuz1xVBQObPLXEMJcvbrdgWL2fT2g8kd6YTnMuKW PA== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by aserp2130.oracle.com with ESMTP id 3668qrnvf3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:14 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJooOJ034944; Fri, 22 Jan 2021 19:53:14 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by aserp3030.oracle.com with ESMTP id 3668r0c41e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WjMt5DS4Mmzo+O5hpboaVtD5qfqTg6CulDp6OQK9p5w10j3/g0s6577oSKxtAmZlukI0hHIUxfxNo9YfT5zCnnlaHvfUm/TGDP/k+xqnp5pw1iykKAyiuaL2MBt15EZ23veIoyTsojT/B7bCNMQCa2kDIX9Lz1fq+SxLUs6hHjxoUgssWvIddV9Tl/cR3L4GEQmcyYKKSE3Ag2ZnyhAaIfBswoDmk+jhspOUqZXwGL/VZiBjkaGuP6rJp1bH5e1n+oC3kpZbVHU12dJLs9hgcqC0mRUubvb6v+utd9b5rO2deLf6JnG6v6uO1k42U8dojVzvTC6zzvalmnnsfXly3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=93yiyvI7Fr4YjpmwKXt61gLFVpnhbSwn2zLTRdOyRMM=; b=IofJHtToJ4oavjYatqx6L8Af+JfHPq6QIhuXzVTllF3PZFrobD5i0mfWaeeKDDvb1kJMjENFEM/safFBQj/gFvBXG6TuQOaU5HDZiqZNW28MDAhElcf8u0v/zD4YoecehZYJA4M0dv1IDVrw1lR6Yk/6RaLdQ/A8GHQaw6fcDmcqSIqbaJPIUai1xPhcI6GFaTU8rYQ3XCuC7hAGQwZsacZnfpwRHkQgfhmLpolRPq9LZaCm9+hcLOjPw41zRQsXezdx0uAhFukkDbWdGnWVg5PonDkyhfUpynvhnncbaXQ4uIuLsoEhCmC/uj75zR1gTEy8LzbIl8dmvM6CorB02g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=93yiyvI7Fr4YjpmwKXt61gLFVpnhbSwn2zLTRdOyRMM=; b=DftnNwntTqOCzvgBT4+h0aCGeURpzRfsgskKs/WLS6QBIPOqZ74llJCjwMAyVzsVrlLxkL77R1c9CyCe8jSNY+LyXddzdZ/m29a4oTKzgWPLu1WH2yPX5lL7hDNECFqHzjvLzxXQeM8vc1ZnZlrf2xPoO6KHQKcV0cmD7cMb+rc= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4723.namprd10.prod.outlook.com (2603:10b6:303:9c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.12; Fri, 22 Jan 2021 19:53:12 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3784.015; Fri, 22 Jan 2021 19:53:12 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Miaohe Lin , Andrew Morton , Mike Kravetz Subject: [PATCH v3 3/5] hugetlb: only set HPageMigratable for migratable hstates Date: Fri, 22 Jan 2021 11:52:29 -0800 Message-Id: <20210122195231.324857-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210122195231.324857-1-mike.kravetz@oracle.com> References: <20210122195231.324857-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.15 via Frontend Transport; Fri, 22 Jan 2021 19:53:11 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ca7a12df-413e-4915-d313-08d8bf0f595e X-MS-TrafficTypeDiagnostic: CO1PR10MB4723: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: adJatRg1t52VDhhsCHAL/PNdkBjlcGYXJeh6thIdVcy5EXlXNy6I6YUxW7UxaJx3oU0bTSRZJEW6DK3Z3LRvhAuiC00JNPYobJICzQ0Ii6L1ELQxb/8N693Ym/HD+aKpKNtpN99LmhEk4lbX9DmuQm6OSqWX1t065nzS73JnZ9tbTrQdZcgTvlAx2zO0bcGUGp/7mXL9K2FZmDGfITzJLimK5Eh3GzyGFxQhE9rl+CuoAkG/MfuDcIhCjTYMaXPbsPW9KT220rP9glA0PY8oS5xVkM2pmzBx+WsSb7PnIONjXL+tOk6iphEAJUzZxJJUS62U8zsqqyDDWtAgFdz6xLwS6KrhbOhEyaXRVvNYMl9WhWHwg9bxVJmgzuk2xO8D X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(396003)(376002)(366004)(346002)(956004)(16526019)(186003)(2906002)(8676002)(2616005)(6486002)(8936002)(36756003)(44832011)(52116002)(478600001)(86362001)(26005)(66946007)(107886003)(7416002)(1076003)(83380400001)(66556008)(66476007)(6666004)(7696005)(316002)(5660300002)(54906003)(4326008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: LUrD2lcwkLWOiQIISAt3qRSlulQOzBslhpDBxZuAHhCxxok3WrLgOpDVari8jX6VKUSJyJ6o4fKNcaWS6QfmhWII+vEI+paE9m8CdZHDmXSfDpBie856ohj53dcNVgk34wBtd48scspfoZ506o05M+HK/3OKCp6Dl4RQTr1OIlAdi99HWgN/rCzMV5U3lTB6tmQyyRsg5PtekVm56huPhFoaiP8bwpnpfamEJfSI/+17JETGIl5wzob3cfXKUALaopNXMKONC0OmhL5dUagHxvx23KEIzGpf4ZHl7NV8/1SEHRP2Q/ZNgEfy2JhvMHWX43mFEQDXH+IFgRAst+zg2Ob+H8pz1kF/9uxlAd+aVbfpRG1CzeydBEkr+94Qt8sxUmV+8lHv+AyAE8oO8YHBebN20gB/N0DueE3PwNSRWpRkd/st769VJSO9Wzs5V7H9PTgg8MVN5VWYVjpcqfGSLn2fiR3i4mrXMXzwnK/eJQtaG7DmEVQcUg1oN3apWtz5YZZ1+t4AORX43BOi6t17gey5VkZuBAk6PpoxVBgYR327m9vynnLrZCKAV0nls7zVRboogb9HXC3IG5wK1s1kknJ8w7O8/gy5LciSoh2HPomkLUth8Tj4aWWHN/fNa1u5hKAQPiMf/EvjrXJ4lf0l8rkCaiG923GNC74U2xvpt2HpLAEHWQj58W3Y40RUTvMJU9WrPe0ZYfOd4qE93e0D649b7g0FO1tFgztM+uUWqKaGbNUJYWkW6wJOEXAoTKFg+r82bGTewz6GQYlEylRi5HznZJRG932jlKm09bxqlz/2DIXtIerpn2s2d+c3Kc66GFODpLJcmvC+9S5UntrYws1vR4mZbj8VnWW9L0XDHPDV0oSUJMw+HCE63HutQNm9SPLnnJBFpWf6/I7M7/fsrJjzMSxYrYmwoHdXPgZS99f14mT1JJ+8MIEANYk9yMpebrkTviXKiYnzeyCnQPjbE5kBq1B+yQTtAUgshCaMqvM7/O1/JGnvR5FjN/KASIEaBUxp6XLrbJFvxyiG8fVRpofJlKhZDpE1JUoks+1wvWC2YxM/3Bp3rQw9j+szByTy X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ca7a12df-413e-4915-d313-08d8bf0f595e X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2021 19:53:11.9695 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0xMVXS4LRir6Sso//3FxkNDr7HCvqJ0YMQ8N0l9H71llZ2Khui/zKj8oPVJUmdgjXMNDeO5Jr7tEnB2z/PgdBw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4723 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 bulkscore=0 mlxlogscore=730 spamscore=0 suspectscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=786 mlxscore=0 suspectscore=0 lowpriorityscore=0 bulkscore=0 adultscore=0 spamscore=0 phishscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The HP_Migratable flag indicates a page is a candidate for migration. Only set the flag if the page's hstate supports migration. This allows the migration paths to detect non-migratable pages earlier. If migration is not supported for the hstate, HP_Migratable will not be set, the page will not be isolated and no attempt will be made to migrate. We should never get to unmap_and_move_huge_page for a page where migration is not supported, so throw a warning if we do. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Signed-off-by: Mike Kravetz Acked-by: Michal Hocko --- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 9 +++++++++ mm/hugetlb.c | 8 ++++---- mm/migrate.c | 9 ++++----- 4 files changed, 18 insertions(+), 10 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index e1d7ed2a53a9..93f7b8d3c5fd 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); - SetHPageMigratable(page); + SetHPageMigratableIfSupported(page); /* * unlock_page because locked by add_to_page_cache() * put_page() due to reference from alloc_huge_page() diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 58be44a915d1..cd1960541f2a 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -740,6 +740,15 @@ static inline bool hugepage_migration_supported(struct hstate *h) return arch_hugetlb_migration_supported(h); } +/* + * Only set HPageMigratable if migration supported for page + */ +static inline void SetHPageMigratableIfSupported(struct page *page) +{ + if (hugepage_migration_supported(page_hstate(page))) + SetHPageMigratable(page); +} + /* * Movability check is different as compared to migration check. * It determines whether or not a huge page should be placed on diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f1a3c8230dbf..4da1a29ac5e2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4194,7 +4194,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page, true); hugepage_add_new_anon_rmap(new_page, vma, haddr); - SetHPageMigratable(new_page); + SetHPageMigratableIfSupported(new_page); /* Make the old page be freed below */ new_page = old_page; } @@ -4436,7 +4436,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * been isolated for migration. */ if (new_page) - SetHPageMigratable(page); + SetHPageMigratableIfSupported(page); unlock_page(page); out: @@ -4747,7 +4747,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, update_mmu_cache(dst_vma, dst_addr, dst_pte); spin_unlock(ptl); - SetHPageMigratable(page); + SetHPageMigratableIfSupported(page); if (vm_shared) unlock_page(page); ret = 0; @@ -5589,7 +5589,7 @@ void putback_active_hugepage(struct page *page) { VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - SetHPageMigratable(page); + SetHPageMigratableIfSupported(page); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); spin_unlock(&hugetlb_lock); put_page(page); diff --git a/mm/migrate.c b/mm/migrate.c index a3e1acc72ad7..c8d19e83f372 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1275,13 +1275,12 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, struct address_space *mapping = NULL; /* - * Migratability of hugepages depends on architectures and their size. - * This check is necessary because some callers of hugepage migration - * like soft offline and memory hotremove don't walk through page - * tables or check whether the hugepage is pmd-based or not before - * kicking migration. + * Support for migration should be checked at isolation time. + * Therefore, we should never get here if migration is not supported + * for the page. */ if (!hugepage_migration_supported(page_hstate(hpage))) { + VM_WARN_ON(1); list_move_tail(&hpage->lru, ret); return -ENOSYS; } From patchwork Fri Jan 22 19:52:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12040379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83636C433E6 for ; Fri, 22 Jan 2021 19:55:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2151323B01 for ; Fri, 22 Jan 2021 19:55:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2151323B01 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A800B6B0008; Fri, 22 Jan 2021 14:55:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A30186B000A; Fri, 22 Jan 2021 14:55:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D3EC6B000C; Fri, 22 Jan 2021 14:55:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id 761176B0008 for ; Fri, 22 Jan 2021 14:55:30 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 32E6518175F9D for ; Fri, 22 Jan 2021 19:55:30 +0000 (UTC) X-FDA: 77734465620.01.salt78_1b122a32756e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 0117A1005D9E7 for ; Fri, 22 Jan 2021 19:55:29 +0000 (UTC) X-HE-Tag: salt78_1b122a32756e X-Filterd-Recvd-Size: 13803 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Fri, 22 Jan 2021 19:55:28 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJnetp171797; Fri, 22 Jan 2021 19:55:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=rCjBFxuyQ0KYqoX+uhkyWaw6crnEcGXAt+NWxwcHqv8=; b=MopJ/e4UconBx+N8mUHvUwPOaSRBkGdq9P3wL97/2WyN8KOdeQ0l19LGvYZr8/+xgerY IZVBVSHdrGkN9+COhOPjroiDR314f4Hw2DuVDoA7j0tKBr99wgx/ps6x9rijHw+a8DL7 kuwhTk7Q1yyx8TLn7Zj5M2BXvYRERCy+KT9XPZrxHqTSByrL0zvbvll1WOZ8gNJBILyp aGixTtyoQeZj1UePvxDlCJFBoP9EemZS0Jnfyeb4/3PtMKtEl7YTGMvD1NtdalL9qPHB Yx+3mGjefS9ns0v3E9jiMQx99DToudkSJdc02xch9gaNWFJJ1O8aAPplnQkN1/niyjrR GQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 3668qanvg2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:55:15 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJooOK034944; Fri, 22 Jan 2021 19:53:14 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by aserp3030.oracle.com with ESMTP id 3668r0c41e-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=doksZS8wsSIGWNgKTJsOmhH6CIxNXXDxaU31BrERkTNQ+pg5EmHBzDSMkH1SYevuZo6etkmVgGGmuWoRdzY/S0fDn+27no+E2GBa0iJMQ5ZNZa6phgJMnl3nggrp+vEddzTcwkt6Osk4H3LjoyTdEO0Tpq+mEiFqLRVEsE7ACX8LCnl9LHDMbeOK/xzSh00Y2E9qNRB/zxl1aqzcq5KJpUsE6m1Pg4x+DM7dN777Vy+Os9qIIyA7aLBrD+J7QBRERpo4kCaTHLqlgwePzq8jKzXZvwlJG1jOBm/20SQvMsIxbMdVmd3HbTttcrmv58+OJASUt7VyPxqx1HPxtD3nVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rCjBFxuyQ0KYqoX+uhkyWaw6crnEcGXAt+NWxwcHqv8=; b=WOGsjDXw0gKZoGMczgBqrRktUIJ4xfVHcW7UqcSbD32cfIRhubBREbi2h+NXY1VFIf59RkMTNvbs1A3J9oadCNDoa9S+E77hadhWUlr32td8AAolKxfGwkZfDPJdQ0lO3IMZbh3POUsZxMzdA4q3tfWzQssZ+OzQfejdf1z7eXXBxay7hzxFFovAsLo38jJWHH68zWu9zx7rjr0xj/bhbDd0c4Qj3Qf1TpBk8B+bJxrXiZNsMKWUmwqQOcRJUXEckpOAGYzhwzoEDn3hyTGrnQ4QSjc+LIdDBGc986zo0ultP1o6bLvwlhYoCq0pCKr+PrAhD7LunZdSgz+17pJEMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rCjBFxuyQ0KYqoX+uhkyWaw6crnEcGXAt+NWxwcHqv8=; b=d7r3gpTS7rhbm0I7Rqytj+baQaatE5IsYLTgP5UqlHba+g7iyQR2yCIm0jSqVUjWxNHBH1+rQcnu1x5Pn4AmTtypq67yjGdyNGEB9Ol92m05CKM5FZa2p2PgxSvqpb4m9YdPPMyG94ap4WlHtu7XZ4jNSBpQx0zU+ImELAFj2Co= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4723.namprd10.prod.outlook.com (2603:10b6:303:9c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.12; Fri, 22 Jan 2021 19:53:13 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3784.015; Fri, 22 Jan 2021 19:53:13 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Miaohe Lin , Andrew Morton , Mike Kravetz Subject: [PATCH v3 4/5] hugetlb: convert PageHugeTemporary() to HPageTemporary flag Date: Fri, 22 Jan 2021 11:52:30 -0800 Message-Id: <20210122195231.324857-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210122195231.324857-1-mike.kravetz@oracle.com> References: <20210122195231.324857-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.15 via Frontend Transport; Fri, 22 Jan 2021 19:53:13 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8f3c1814-4065-4b16-a768-08d8bf0f5a46 X-MS-TrafficTypeDiagnostic: CO1PR10MB4723: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zAdd/6N0IZweiyKUMuAzVyHejEq1N4TEx5QjM92jzr33H3NKHtxxiohttWjU9EIznWrfLhmKZE1dFc+5lA9EiFpWTbbLusY7qt/MTRxrA0o7scvjzN3fiJ4bKamMZhdgPouJ2GeeFKvG5/MBK7F3AQC0+qJt8KBsKJw2wHSl2OHiAIqz9cER3H/4UdLgKafH3wGpOChZqDRUJX/nOsUJ0kueLDYfa2rM+aPD9TkzEWa8Ny4D57boylpBuFifAXSvjL5XHUu80TAV9/CYNYEFdEUKjqhT37IO06+UdduK9c13zHh99aJN7eH40s+WBM38VhYILyPcSNrKw1PbY05GydPTN2CrP3km7a46IWEeAGRHCe9JYjKuJT562wipMTzJ/zH2nbJKs21YV7mN5JxiIg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(396003)(376002)(366004)(346002)(956004)(16526019)(186003)(2906002)(8676002)(2616005)(6486002)(8936002)(36756003)(44832011)(52116002)(478600001)(86362001)(26005)(66946007)(107886003)(7416002)(1076003)(83380400001)(66556008)(66476007)(6666004)(7696005)(316002)(5660300002)(54906003)(4326008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: JiRl7mTDw6ze7Je4ky7QKS1szsHMbbPfsWAlPBwVhQdbQ8hkaKyvRIFvLSVcNMmGBb6UQ8gE8cFonNTlJFuGbyRGUmKL2htj3SfpVz4TT0FRK6tUchZNWBDO55KWwEqSPD/8TjYB8eI9VwGiHZKeziz7GqBHlkzaVNbRvRkX8rilN6Inu22JvwZUoDWx3AKfWWoJL91OqUuHRbtlqt9fFfR2/UIUJ5/yTpELf4n0R4TZzKDyz5EPlEGHQGN2r27IpMrgjl7wZw06akVCrFwTGmavvgY/vK6Bn0RlqKVI3XSxJf3XbYM6wgKsE4J9FNOjVsTvFg09GinNprzNXfe1+Kw+dy/5WoQvgOayccU5z1byIJoQN/dEc5B4yQrHovtSfx3zK/pKW8gevKhUSVJrLBVY923a0jMLysdlkHjA5mvg1dNNmaHl2s9iSrtOzkBlO8E6MW0bykHglWK0gg9L4kH8yBU2D7G0/BUXsB3Ulh2MsaItfTpgCIJ4qlsmVZPfqgALWNr1hHrbXdfM5GXRdrT75i+3euO470akSkgi3zD99q6RfavJE+/XcxyaPdgeLhtxg45YQOxXn3ZJD4XByPwUJKy2g4RWVrvAloum/fMEaxvrtCh/QKl8fM8BMEGyCek1Z0CIwUv9iB237wOmZnY3KjC3cGwrQCcE2SK4OC50mZtsdV6FJYJ3uhFL4onEiZn5lWaSklgydVIWQ4B2mcvieY5ZwaaXqOlESNSbm971FoG2e4ziuto/8y3QgSxplVScbGT/deeQjvCcdPb540cP963TJMaqzkixmLOmujSv2lWBJc9poM0zaic5L9G1R4Gfryqrv7BBs683mlmCU/ynNyHAd8tgFn+CoMHjLO6YpHVZv/uPHpZ9dguC8F4o9VBIs/ZTllReJXUthH9wr86DNN1xwlkoSdOPYUZ2U8ZbwkFJZpsL8GyE/txzfctec1+Fy8yfamTJXUcDRsfGoIwymZ14SMiKmIV4mQjCI3q5vm+XsFu0f5UoF65Hx+s/ZuS970DDJw+QHWIB+n7mTL7aIc86SAkBgHuKsyo6SSm0Bzf0/B+YuwS0hoXUMZ/B X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8f3c1814-4065-4b16-a768-08d8bf0f5a46 X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2021 19:53:13.5168 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I2F1XStjIu2ZnAma9yFtvsWl0qySGpRLunVPqmmNqNS+ksWiGlHhyn6t0bSRFhfau8W3dcfQ+cnztPhX2JzPQg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4723 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 impostorscore=0 mlxscore=0 priorityscore=1501 phishscore=0 mlxlogscore=999 lowpriorityscore=0 malwarescore=0 adultscore=0 clxscore=1015 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use new hugetlb specific HPageTemporary flag to replace the PageHugeTemporary() interfaces. PageHugeTemporary does contain a PageHuge() check. However, this interface is only used within hugetlb code where we know we are dealing with a hugetlb page. Therefore, the check can be eliminated. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Reviewed-by: Muchun Song Acked-by: Michal Hocko --- include/linux/hugetlb.h | 6 ++++++ mm/hugetlb.c | 36 +++++++----------------------------- 2 files changed, 13 insertions(+), 29 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index cd1960541f2a..3c86c3a0e144 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -483,10 +483,15 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HPG_migratable - Set after a newly allocated page is added to the page * cache and/or page tables. Indicates the page is a candidate for * migration. + * HPG_temporary - - Set on a page that is temporarily allocated from the buddy + * allocator. Typically used for migration target pages when no pages + * are available in the pool. The hugetlb free page path will + * immediately free pages with this flag set to the buddy allocator. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, HPG_migratable, + HPG_temporary, __NR_HPAGEFLAGS, }; @@ -530,6 +535,7 @@ static inline void ClearHPage##uname(struct page *page) \ */ HPAGEFLAG(RestoreReserve, restore_reserve) HPAGEFLAG(Migratable, migratable) +HPAGEFLAG(Temporary, temporary) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4da1a29ac5e2..70ffa1027988 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1353,28 +1353,6 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -/* - * Internal hugetlb specific page flag. Do not use outside of the hugetlb - * code - */ -static inline bool PageHugeTemporary(struct page *page) -{ - if (!PageHuge(page)) - return false; - - return (unsigned long)page[2].mapping == -1U; -} - -static inline void SetPageHugeTemporary(struct page *page) -{ - page[2].mapping = (void *)-1U; -} - -static inline void ClearPageHugeTemporary(struct page *page) -{ - page[2].mapping = NULL; -} - static void __free_huge_page(struct page *page) { /* @@ -1422,9 +1400,9 @@ static void __free_huge_page(struct page *page) if (restore_reserve) h->resv_huge_pages++; - if (PageHugeTemporary(page)) { + if (HPageTemporary(page)) { list_del(&page->lru); - ClearPageHugeTemporary(page); + ClearHPageTemporary(page); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ @@ -1860,7 +1838,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, * codeflow */ if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) { - SetPageHugeTemporary(page); + SetHPageTemporary(page); spin_unlock(&hugetlb_lock); put_page(page); return NULL; @@ -1891,7 +1869,7 @@ static struct page *alloc_migrate_huge_page(struct hstate *h, gfp_t gfp_mask, * We do not account these pages as surplus because they are only * temporary and will be released properly on the last reference */ - SetPageHugeTemporary(page); + SetHPageTemporary(page); return page; } @@ -5612,12 +5590,12 @@ void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) * here as well otherwise the global surplus count will not match * the per-node's. */ - if (PageHugeTemporary(newpage)) { + if (HPageTemporary(newpage)) { int old_nid = page_to_nid(oldpage); int new_nid = page_to_nid(newpage); - SetPageHugeTemporary(oldpage); - ClearPageHugeTemporary(newpage); + SetHPageTemporary(oldpage); + ClearHPageTemporary(newpage); spin_lock(&hugetlb_lock); if (h->surplus_huge_pages_node[old_nid]) { From patchwork Fri Jan 22 19:52:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12040371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE233C433E6 for ; Fri, 22 Jan 2021 19:53:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5A9DE23B01 for ; Fri, 22 Jan 2021 19:53:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5A9DE23B01 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E54E96B000A; Fri, 22 Jan 2021 14:53:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DE0016B000C; Fri, 22 Jan 2021 14:53:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5D486B000D; Fri, 22 Jan 2021 14:53:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0006.hostedemail.com [216.40.44.6]) by kanga.kvack.org (Postfix) with ESMTP id A8A7C6B000A for ; Fri, 22 Jan 2021 14:53:32 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5F29A613E for ; Fri, 22 Jan 2021 19:53:32 +0000 (UTC) X-FDA: 77734460664.10.sleep57_150e7cc2756e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 3DB1216A4AA for ; Fri, 22 Jan 2021 19:53:32 +0000 (UTC) X-HE-Tag: sleep57_150e7cc2756e X-Filterd-Recvd-Size: 13032 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Fri, 22 Jan 2021 19:53:31 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJoOWA060610; Fri, 22 Jan 2021 19:53:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=hL8wQ67fCnGWMS7fCZuNZYpKCyQILeittN5nqzpGUbU=; b=EA7WToie8IRtc+Yb+WUN+/SCGHrmvrJKR2z4EYAjjZ5TD2Gf8olNrxH4GzKFNtv82rQj PiIhIXT4QxTAHeiaU6KnnvkRKO3AVLrzMlj102pis+/EupPxTGVMoGu2PNMkbBbWqOcq tY2tVdms8k3BzFIgMmxrjVFboTeAuy6dfqn3Cjh7r1R05DFauYrieyff93XbICqoPJa3 OvVjS7PklcpvzJw1eEXvdlDT6Jw5VvIOl7wiqZXkrv8HmNLXBI992TL4c3bEeniB3hXA r0/bQb9HeEFji80KoKoCo2vOqIz+vJLvfBfWkFOgiQ2kbYhrTMWG8mwT9YcwauPhJk0h ag== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2120.oracle.com with ESMTP id 3668qn5ufd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:18 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10MJnq61074517; Fri, 22 Jan 2021 19:53:17 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2171.outbound.protection.outlook.com [104.47.56.171]) by userp3030.oracle.com with ESMTP id 3668rhd7un-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 22 Jan 2021 19:53:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FwqOg0eiV95ZvUfUYXmSMV2pKC5iNKAMbaFaLt+OBZeLzbqd/W0G72UCJmq2SLT3tgi/Apr8QCfNWzRo5CYTAVpEdKp93Ug9FMqTioKGSISwcyfRHBjku7LUO8RoZ5c9FFHxiGma7SvNSSmO06iDksq1VZeE/oKtdN2tzPK9+s8kUiTZOers++W0GQDsZ6rWTetKX9ZxHTKVMeMQznIE6zbpEbsKsKzawwUusBCBFY3Kw9AAftsmgYYLE73I4KkXZot9WtlXExrXFa6mQb+rRKXRjPZEtOQTVp660bgp5iqexSTKTLT6kXyMrRTn2rn/Nrh6sxWmCfpr2GZl1lAR9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hL8wQ67fCnGWMS7fCZuNZYpKCyQILeittN5nqzpGUbU=; b=CGwrD5EI2QbtYTKDCq+aNt+R4juZDFxmWyjwLumwlj4BPsbrR+ds0sY3U+L9+1dO8WCQNRbzWxcjbuiOH26Jlp1775OhwGp4zCx4p+APOk5KeCfUsBPXVlXSekZSTpRI8mZTqkZI9w34T+3uHZP/ZgjeYN3Y0Yydn4VryDFOSP77Wt034lUz3h7xFNS6QK/oSoFeoQpULCSxe+WeIlXq91CcPyct3XeZvEU66xSDZccdpJmzRe4iwvLWRmT1bdxTjIlCuLITpX3GCwOewzfS7s7R/h9AGzAfhBl/2LkNE5GhuT+JOqiuuv7NmwkHLpx70y9zElAgnUY9oAyEhT6Juw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hL8wQ67fCnGWMS7fCZuNZYpKCyQILeittN5nqzpGUbU=; b=N8rjaKrU0dCEaOH2OBCnMWKZiWlffRVN8cXmy1uGgAf9vFZb9N15IZwiHklduk3sO30VGYn2WeGmJ6fIHscp1Bqm2lq4VQkrrgDfhMUOZGiAvc+lSDlkel2FvU/p4KfY8q4I3ZKLjmNEOgECTvlwcaERJVmKdp5U3+5hhoh079w= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4723.namprd10.prod.outlook.com (2603:10b6:303:9c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.12; Fri, 22 Jan 2021 19:53:15 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3784.015; Fri, 22 Jan 2021 19:53:15 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Miaohe Lin , Andrew Morton , Mike Kravetz Subject: [PATCH v3 5/5] hugetlb: convert PageHugeFreed to HPageFreed flag Date: Fri, 22 Jan 2021 11:52:31 -0800 Message-Id: <20210122195231.324857-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210122195231.324857-1-mike.kravetz@oracle.com> References: <20210122195231.324857-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR19CA0071.namprd19.prod.outlook.com (2603:10b6:300:94::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.15 via Frontend Transport; Fri, 22 Jan 2021 19:53:14 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9c1e2b78-c7ce-4636-aaa1-08d8bf0f5b45 X-MS-TrafficTypeDiagnostic: CO1PR10MB4723: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cfu0ZPSamDDd/8ke0V5poSM6wuvn60AkrC1ukguhjegGjpoZjiioyQ6UFOYCwZIX7105v+a14j6CkKbtMWFNMhJw/FcOMiCKqjDRq3H66VqBxEoPoT14dXhbCsX0EeYEdL/gJFsRucY7b+tr/SCXEheeAo5u8BW1/IF/oKBckO57hXWg01py4XqezBzIM7akGapzs/OROjkCLsO5ThrbTBr6Us3yOInOe6nO5N9C58kvn/m6aB5xaBku9KpQZZid9ybsNf0aCIadS3w0cPpbnsa1ZjMXK/2CK6K552kzT9qrfraRFFPFRTnZREo/FY244yq3F7q/BsFcmjTy9tFaL99tgBbcn3y7oLRa9wX/bZJDdiQuk7MUneN+fVHk1yBiAgpY/Q+4djpucLorInGU8Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(396003)(376002)(366004)(346002)(956004)(16526019)(186003)(2906002)(8676002)(2616005)(6486002)(8936002)(36756003)(44832011)(52116002)(478600001)(86362001)(26005)(66946007)(107886003)(7416002)(1076003)(83380400001)(66556008)(66476007)(6666004)(7696005)(316002)(5660300002)(54906003)(4326008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: bEzzO8RsFlB5Yw/3S/kxgT0xPbVTlu8WJpqnURmWFIoyks/mcfVV7nd5IkpQEjaR4+cd5Upfe8i42k4akMJF51jxKXgMfQiNSziVTIILhM0QW1uOf6ryC4mjpi1Wq7QSUW5bLDZvWaB9BsfLoJ6qe3cWGNqBHTDvFo+TP9p6jAsHPWXXTtob5bNvyfwZJ2Vrgs60XG/B/DfOjOWRF60Dg7lhCYF85uohi2AImuiy8CzafCwSLfCcB98JDsXreyFe0vYp+0KUsRkI3VXodFZvGNL9poXTXto86RsNIWLzQ9ApEWNzOtbeNNDFVvMl6h03a62I02Xg2EMR+AVG66HeWrUB75kGhw0GfOVOOnv5xf+V3ckLJdpD8hGJ0QNkGXcPeS0TuaOyoFtixekL/dzNDbkkgfba4AY/UNom4+oacAsMLrzubAXbLkB+04A8tEZ3m1QuAOcO26hPOMq4oFMMk2C/ahXyuFtruTJ5NOBtPgvyNG0Ucn0EGVFHzzgTtAtrc4CYNRVGZ6hPq+WT7oXYej45uV7ZE0CPR5xRF8SIj5PWd0P5tye2EEJvCqueyoOu7Cz0x2Qa7ZEFaHWVun/1b49A3BZTqrAV9zva+0JHnZDEpwMxEzUzod/aakXmZwTcwXEWZnaNm89BtYPVTIQhEKJv4W+55z2APgq5Vudwy84WpGPU7fRu6skZF6fuQM/3druB/mzJaxVwd1LWEFJUyxxL+FfFJoQRM1VR75GwfMbjI4OO+GlXKMCIaoiDrKadcl+IOpc4CgJqxfAC0iXACLBzi91nDjy9J4QgK5ZSivBP14xiuEthOTw/s3nQgARmc6qhlNOLfzRoMgC1HYquVgiOVGz56TrB3ktGU2qHa3z6uYygnZ2Zcl9OZUISvmameb/Ygw1DaSg1P1wGiix3+WZJUpaAvzcxGlY8v7QNyMk5HmqWV35ALmZVPTKRlcomK+PfMhzHiALKoGGGCL0zKFlU1gU3bUPqJLp23I+UbgYDjmMUgyApWuBuLtSzix2XCGjsBDVko/ArxF9Xj2gnxNPp4fqKrlFP0mlEdN7lB6naVb0m0Qca+GCpRzonEYA2 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9c1e2b78-c7ce-4636-aaa1-08d8bf0f5b45 X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2021 19:53:15.1251 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: O0Zvp3Swr1TOOj3Qu//TM/w462swGfncoZleSLJrkR3NfMnAoAC5+W6XMON2TCcUXgmkxZ87Eli1vtKkm3IW7w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4723 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 phishscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9872 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 priorityscore=1501 adultscore=0 impostorscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 phishscore=0 clxscore=1015 bulkscore=0 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101220101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use new hugetlb specific HPageFreed flag to replace the PageHugeFreed interfaces. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Reviewed-by: Muchun Song Acked-by: Michal Hocko --- include/linux/hugetlb.h | 3 +++ mm/hugetlb.c | 23 ++++------------------- 2 files changed, 7 insertions(+), 19 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3c86c3a0e144..f09085c0afea 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -487,11 +487,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * allocator. Typically used for migration target pages when no pages * are available in the pool. The hugetlb free page path will * immediately free pages with this flag set to the buddy allocator. + * HPG_freed - Set when page is on the free lists. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, HPG_migratable, HPG_temporary, + HPG_freed, __NR_HPAGEFLAGS, }; @@ -536,6 +538,7 @@ static inline void ClearHPage##uname(struct page *page) \ HPAGEFLAG(RestoreReserve, restore_reserve) HPAGEFLAG(Migratable, migratable) HPAGEFLAG(Temporary, temporary) +HPAGEFLAG(Freed, freed) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 70ffa1027988..c1f283cee0a9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -79,21 +79,6 @@ DEFINE_SPINLOCK(hugetlb_lock); static int num_fault_mutexes; struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp; -static inline bool PageHugeFreed(struct page *head) -{ - return page_private(head + 4) == -1UL; -} - -static inline void SetPageHugeFreed(struct page *head) -{ - set_page_private(head + 4, -1UL); -} - -static inline void ClearPageHugeFreed(struct page *head) -{ - set_page_private(head + 4, 0); -} - /* Forward declaration */ static int hugetlb_acct_memory(struct hstate *h, long delta); @@ -1043,7 +1028,7 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; - SetPageHugeFreed(page); + SetHPageFreed(page); } static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) @@ -1060,7 +1045,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) list_move(&page->lru, &h->hugepage_activelist); set_page_refcounted(page); - ClearPageHugeFreed(page); + ClearHPageFreed(page); h->free_huge_pages--; h->free_huge_pages_node[nid]--; return page; @@ -1474,7 +1459,7 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_lock(&hugetlb_lock); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; - ClearPageHugeFreed(page); + ClearHPageFreed(page); spin_unlock(&hugetlb_lock); } @@ -1747,7 +1732,7 @@ int dissolve_free_huge_page(struct page *page) * We should make sure that the page is already on the free list * when it is dissolved. */ - if (unlikely(!PageHugeFreed(head))) { + if (unlikely(!HPageFreed(head))) { spin_unlock(&hugetlb_lock); cond_resched();