From patchwork Wed Jul 21 23:05:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 932D8C6377C for ; Wed, 21 Jul 2021 23:05:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4595061241 for ; Wed, 21 Jul 2021 23:05:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4595061241 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6DE866B007D; Wed, 21 Jul 2021 19:05:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 668246B007E; Wed, 21 Jul 2021 19:05:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F5FB6B0080; Wed, 21 Jul 2021 19:05:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id 1AC8B6B007D for ; Wed, 21 Jul 2021 19:05:50 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C27A91E067 for ; Wed, 21 Jul 2021 23:05:49 +0000 (UTC) X-FDA: 78388129218.15.B2A7F4B Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf23.hostedemail.com (Postfix) with ESMTP id 4DF959001B24 for ; Wed, 21 Jul 2021 23:05:49 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvacT031904; Wed, 21 Jul 2021 23:05:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=buzV1+L2Y2ONcY0CLrJ1pQYw1k0vHhcGd6Zvf9Ttln0=; b=mxnXyuhN62yrDfddRWd9RR0nP0x6tUa9Yc2JPiiZajurwL2/47YWcXLNankF7lH4OnqM X8tie+ltDX4xw+lBV7A+PpNHZSFMv7j6ziEWb5wTZe/DP3pYY5ulyqUUhEVwFhBW67iy H/fSm3cWoC0tQzrK89DVtLlMXcJmOMrhGcRRlxeDQ13JhDnj8Mwsc2ooTPcAm0O7u2ex KRTbKJLuaunqcLz9emJDh8MEr6xq2GH3z0VTqnrcwXyfgKgkp0B2gnzcx0/90aujJRjo SPcYCtENvVv0Y1go0tS6aVBtz98Hv5u+xstKPuMBYq28egHs3dPuk2qOeoqkk3JgbH56 5g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=buzV1+L2Y2ONcY0CLrJ1pQYw1k0vHhcGd6Zvf9Ttln0=; b=ozoiLXdhbc1Qfpsm6TDWSv1b2mJoSBTqQNMuKiarmsn/YWJS9jmHu48M6sbKD83BOvbW lGhkgEHd2f+F+aWy8OGRBUw215OiAcFEq7llvNeRO/SjFLZ/0xsvBMmDVhgHoYLHQ1Go A1AQn2CpxQ5T+yGQsbD6ISbLWAveIu4sYuB9Z3RlA0vnEvzcgCa/+KS8VHzbGFspxSjl zyEjNa/2E0XHhVx3Th0iKuxPKVo7TAzC4qmOuqY6i6JHFhOguJewWENHGq1jQLauFb2P MxKYy++mEMyTB5UpmRUMal0srUxTX7PzJR6z8fsitFs9tDHlktSujUJd9nvf9mHivZzV 0g== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39wwmtuxr1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:43 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN0T2E098889; Wed, 21 Jul 2021 23:05:42 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2041.outbound.protection.outlook.com [104.47.73.41]) by userp3020.oracle.com with ESMTP id 39v8yycrpn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=am/K3UDp7yai39RL8eW7PO3rj0GxfjJWFttpwm/I2/nh70Qb1F+gt/6m29ZjY+52dXfA7E7y8SA5kJoT0yKTnc28Q0UzR1Ksjca/sz7jQTdcbFG69uM/4IXnVfIJJB76VA53SzWmR50D2HojESKVqzJhNqxUEOg/faszO0XeHC+EgecUZIFWy6HXAhIk1g4cUsQWeYA76fvR/l3SN6LeIzFpAL6LoVUytXm7fMvDO6ji1gyJj5W+OI3Kex4CdconOGgqCaJK+0GiUhOIPMw4hV48WiMXH5h1NROuEwniSUpjrPThd3YgcVkSCik8MVAxSsV0eRTfnwfbpH+KJtiXmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=buzV1+L2Y2ONcY0CLrJ1pQYw1k0vHhcGd6Zvf9Ttln0=; b=lhDhgJeX2n3NXzxd1ajrus0HIYxcuItHMPLstC1JU/MO80cQgrT/z4ODIAqM/ijARlymbwZ9qFW8BYm1CucL86IZCS8clLGrMH5LvhuSoK5DglurcAzsFhEawq2PHkoTeH891EXo+WyxdYtOPVpUeWCrXu69wTka9hCWUsXFNQEdpNL6ehnlhD/nYFVQoZfq4CmF12XrDRjgnIGXocWzEVmXC4A+lrcUeuzihruCUjP1d0FHF+hc4TU+tGvYC1qTXBk1+ZUmV4V+XciU9Q4YOVlFRSVgC892ZFf2tilnnLAHU50z3/9t3OeiNjx+dnmCpWNZdR94qHwKEI/oJKO3Cg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=buzV1+L2Y2ONcY0CLrJ1pQYw1k0vHhcGd6Zvf9Ttln0=; b=Adc7CCaT/us9PeC6py7QOkk1XPelw2y1baQ9KBT+3uoj4y+s6m/4mZbRn5PJKiWR+tq3p1NZ9MIDACDyOeortw8c4OExT+pVMuh4XsNwowGSwiPk9MZ4y1/rfoZdddpxXe82s6n5EZE5ZRhsqSVRJEW92LNoI4mjYnsRuJ3rxiA= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:38 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:38 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 8/8] hugetlb: Optimized demote vmemmap optimizatized pages Date: Wed, 21 Jul 2021 16:05:11 -0700 Message-Id: <20210721230511.201823-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:37 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6631d343-6ae6-42ce-80e4-08d94c9c0db3 X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ev/9vW5jbGZZbZnjYqRRZksDR8kUh+gccRkvsNV71AjAclRMka3re6xW3JwAV7DX+KeCyagchIwy2aJtkjTKkOxIA5HCyCps8zkm2NV3FM42PKnoocMhRp2+GP8OcCnEBAmzDugi/uSL0CLo9Nob220D58VO5rj84cTbmK4+cpyC75ZvcADKJzFHUiSY9RJUX/r/YP2RZjcVQMT4wkTNkZg0T1YMk9qTsn4RxXZnn2PnW8sBq8OHzrRvIY6t3a8w/joPPtA89kdi8kenr6AnyvqLDYfgkvKpqqZ0AHHvs318gA874nuhe+mN4nyXGl45+MiEq4a2hoiE/rgME53Vbl2881U5KIlO/jR23I5pFnX79dKek5CBixZDrUQgJxs2bCfH2INScAXzu8i/RVFtd2vyLZ9niaCmz6xAycwxyNeUCgSO2iqT8DJ150VQCKKel7wgpLeCqEvFd8ZmI9OLalke1Wi/6ed0nRTu5HDEbXE8xWzLI2QY+MTWk35mXTN2C4Z1XhGPaqXqhd8Su+AqxrzvG1iBcGtxFQ6AGe4iLDvKc05vPfL8JCirjPoUsL8/fYstT5n7l45mQf230uL8P77asGu9jK1UGs3OeQZV/X/GoB19z97uE8hl1cziGBcpYK9xyLRrud3mvInbFNDnEqv+O8ZFs2rKYRU8lnlQhCUyUY3CR48ySp3hou56CBghoWfMFjxE1fFSJyDBgiqHpA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: WggjKh0Tyxy+hgAqmEFfNi7WvLPcNhSIswkoQ9W6NS50EDmunJWGkbe3H0Wdo1Pl/6jr9/qnccasKXrIVdpTpk1Jd2aM+XR2Usc8uonhrvVgzIDkImkTG4kdXBFqHR8mKi8OvWU1BGwTsVKPrrQOx0oVkDGUFytVn6Fg/cGjaFbbIyvTlH6e944/KXJHbMNKjOXC8mpdW/Ls3f6Njr+TlWy6nyQj9OwBKoro1d4+3pfMK5P+cqVS6+2B+6SLNGKbgjkeKIUtzYj6QdpZQXdR0OwQ5EhqrXO2kQ29UwQGm9o5sPh2ary8Y2/1eUu0PhwydGi6LgDLYhp0HddJHLhls2HG3PrS0vZ0AXkKP+gOzCtNcUT/tfhCTmGjMfrzh5n/mAySrqPgFoq9L1Y/KA5hlfjtuUCJ/yM6qIxaUnGzt8p3wiJ+C1D8EuqjR7UgH7lI7SQrDF4ncXmtfSU0yM+JgAtuCdpKNbIwzFeENVOwNuegoPHgCXb/+8zB3iY2LiH0pgBSgEQZxPcDgX/mvsFHAPdJpl0vYv9dIMAEDvj0gomEhQ3E/PwLvgHo7WljfDSeZMtbHOTT83xYJtRCwIm7R+WEe93JExGunCiqpujqGGQFNJ1LtB9D6XypubDVCGydBuAqjpk7pOZH0EdcPwP88q3ULVQ7NOllbElhM3Nzi24emZfhOGRt4LGIFuV8N70KJ/FIIRupIsXBy06tA1sLXx8IIUPgYVUTscfzsWO+hQgDSFydClzzINeTH6za4V90eXJDEPz79tMBltqNf4W8s/2UylelpsAFWdbj4e7yCCeoBeGIgNlGF1XX8cbZYo6vlIT1m2MzaL9rT2s7Hkk3urrl6lu6bHUvqXMhZr3XNN1/LbblSFbg2yX/ZV9XQm/MZ+LKQ0w3I5wMZFtGCmBzWUJpKk64V/CudXeSFQLQpEEM+zPbS2PtXe9058zo5DNB0H/K/lZ3UhKLVCPIZ2wEdxisGeZGtUi8NS4YnCR6XytIyFS2M9ExBRPg0Dh0TUgXgXGNuZT59UU7ElWToNhFbxs9oITdQSsZCi+OAJD7QQPBECPDqaLoHOnz5SHcPunYKFlxN9Vdbl9aLTJ3fIF0CA37L7Pw+8ndNSW7O0cQtlBpasS816MaakbAm4Hb1ZWewNrsSgi+ccKE/TQqWMFDGQAIzi5qSJC2gT9dseCoW62vOnQLb3bX9Q+uk+NKEVReaLlnx7X+I1xvIyLjT5lRCz1CdTbcIHsWOTUIip5aigiPGMFkl469NSPI77bq92oU/YYQs8Gs51loxRGs58BDIWWs3jZ/vFfV+KpStJtvwD8TCCQWsIgL6cMzfGLs++6I X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6631d343-6ae6-42ce-80e4-08d94c9c0db3 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:38.2503 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rict4WaCU1DQ5NYurgeD2xrzvaMpIFR014t0P6zaPRtp9hXGeexPS8zU1L3QspKzhz0XCYSpXQIgSyYhcnWWGQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210137 X-Proofpoint-ORIG-GUID: FN1_UjeeCocJbyzeKN9SE_DOFfM_c1oG X-Proofpoint-GUID: FN1_UjeeCocJbyzeKN9SE_DOFfM_c1oG Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=mxnXyuhN; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=ozoiLXdh; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Adc7CCaT; spf=none (imf23.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4DF959001B24 X-Stat-Signature: nqc5yrkxfh8eafd7koi7ys16xhai9jgh X-HE-Tag: 1626908749-821942 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Put all the pieces together to optimize dthe process of demoting vmemmap optimized pages. Instead of allocating all vmemmap pages for a page to be demoted, use the demote_huge_page_vmemmap routine which will only allocate/map pages needed for the demoted pages. For vmemmap optimized pages, use the destroy_compound_gigantic_page and prep_compound_gigantic_page routines during demote. These routines can deal with vmemmap optimized pages, and know which page structs are writable. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4040dd3c6fe3..893406282c53 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3307,13 +3307,14 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) int i, nid = page_to_nid(page); struct hstate *target_hstate; bool cma_page = HPageCma(page); + bool vmemmap_optimized = HPageVmemmapOptimized(page); target_hstate = size_to_hstate(PAGE_SIZE << h->demote_order); remove_hugetlb_page_for_demote(h, page, false); spin_unlock_irq(&hugetlb_lock); - if (alloc_huge_page_vmemmap(h, page)) { + if (demote_huge_page_vmemmap(h, page)) { /* Allocation of vmemmmap failed, we can not demote page */ spin_lock_irq(&hugetlb_lock); set_page_refcounted(page); @@ -3325,16 +3326,36 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) * Use destroy_compound_gigantic_page_for_demote for all huge page * sizes as it will not ref count pages. */ - destroy_compound_gigantic_page_for_demote(page, huge_page_order(h)); + if (vmemmap_optimized) + /* + * If page is vmemmmap optimized, then demote_huge_page_vmemmap + * added vmammap for each smaller page of target order size. + * We must update/destroy all each of these smaller pages. + */ + for (i = 0; i < pages_per_huge_page(h); + i += pages_per_huge_page(target_hstate)) + destroy_compound_gigantic_page_for_demote(page + i, + huge_page_order(target_hstate)); + else + destroy_compound_gigantic_page_for_demote(page, + huge_page_order(h)); for (i = 0; i < pages_per_huge_page(h); i += pages_per_huge_page(target_hstate)) { - if (hstate_is_gigantic(target_hstate)) + /* + * Use gigantic page prep for vmemmap_optimized pages of + * all sizes as it has special vmemmap logic. The generic + * prep routine does not and should not know about hugetlb + * vmemmap optimizations. + */ + if (hstate_is_gigantic(target_hstate) || vmemmap_optimized) prep_compound_gigantic_page_for_demote(page + i, target_hstate->order); else prep_compound_page(page + i, target_hstate->order); set_page_private(page + i, 0); + if (vmemmap_optimized) + SetHPageVmemmapOptimized(page + i); set_page_refcounted(page + i); prep_new_huge_page(target_hstate, page + i, nid); if (cma_page)