From patchwork Thu Jul 6 08:58:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61BA2EB64DD for ; Thu, 6 Jul 2023 08:59:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1C0F8D0005; Thu, 6 Jul 2023 04:59:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ECBE78D0001; Thu, 6 Jul 2023 04:59:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D461A8D0005; Thu, 6 Jul 2023 04:59:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C61828D0001 for ; Thu, 6 Jul 2023 04:59:02 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 70F0AA03C4 for ; Thu, 6 Jul 2023 08:59:02 +0000 (UTC) X-FDA: 80980587324.12.FBFAD01 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf23.hostedemail.com (Postfix) with ESMTP id 23D4914000F for ; Thu, 6 Jul 2023 08:58:59 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=qj4VM0+l; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633940; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SO8r5VbGmdwJ3q2RNf1QQ4rNQ+vbk/APFGvtlYo2uGM=; b=DqjDcQ87bVMVZZWCCE/RRs9lpdCiFPCPN+H1MWp/mxyogNpkjZ2JtPDr3UvRKNxh9QkJPv 9Z3PeOqFKedAG4y8s91Kmg/KPk4I6bfjrJwa+81bDUhs40idNyAzuxj+N0Os3hd39izWHh Ux1O8NHvhvQqIPLAIW4WrQLBP60ZvKo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633940; a=rsa-sha256; cv=none; b=0AnYuWH36+IwyUyhjJcK2lj/fdYcQy3uzhRFX+GX5ZAcpbMeFpki2/SgKNBgwvkJfebv2k 9BOaJQRyJSjPj7+O3AxJfeUa1Bcfvh6DK/UQIiVEPINEY3kwkYxhci4/YFeT+yTUP1N8wk OWu4g842XSy4WYakso7GleD75apWVSw= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=qj4VM0+l; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353723.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668qadW013823; Thu, 6 Jul 2023 08:58:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=SO8r5VbGmdwJ3q2RNf1QQ4rNQ+vbk/APFGvtlYo2uGM=; b=qj4VM0+l9pDHoT5/UHKVgRVF6axZw2HM3H0Mj0bqFGArtaHyDwedi6eBBU3jr+GnhM97 J3a4JrYvtHI2tB4oQHyFYXtRJtED2T4jCNtDhjInMrKanjUDDStRgUq+zjMaZvcJZb8U uB+m+KvmYW9Yodn0cVw/qpkamfhyHvH8FqIXhmjq6KUPxkANA6CUZmeSqL7BSNYsdoJS qi+MiwR4cZu0XVX759rxKCPA1f6KorjhB5dPdtynhS8jFfx8uHDTyfCrO++Iv+6MMbt1 YKlvkitpbIB8cUFORlyBd9m0pGv/bzSnAjHLacni4F4ZDDdRsOCIxf2G+NkbeXEOcTog DQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntfpr40f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:40 +0000 Received: from m0353723.ppops.net (m0353723.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668sBQL019390; Thu, 6 Jul 2023 08:58:39 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntfpr405-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:39 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3667knaR002598; Thu, 6 Jul 2023 08:58:39 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([9.208.129.117]) by ppma05wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs5yxu5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:39 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668wbXZ65339776 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:58:38 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9C4D358059; Thu, 6 Jul 2023 08:58:37 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D486058057; Thu, 6 Jul 2023 08:58:32 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:58:32 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 01/13] mm/hugepage pud: Allow arch-specific helper function to check huge page pud support Date: Thu, 6 Jul 2023 14:28:07 +0530 Message-ID: <20230706085819.827761-2-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 3IIzO4X39zwkmKxpAzglX5tlHicNfMUx X-Proofpoint-GUID: yXhgcgU-CZmSvaocgdEUuZmGVnoTxV9y X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 mlxscore=0 phishscore=0 malwarescore=0 lowpriorityscore=0 spamscore=0 impostorscore=0 priorityscore=1501 adultscore=0 suspectscore=0 clxscore=1015 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Queue-Id: 23D4914000F X-Rspam-User: X-Stat-Signature: anepkf58dbu966jm9yq6homd8fxr3fxm X-Rspamd-Server: rspam03 X-HE-Tag: 1688633939-359458 X-HE-Meta: U2FsdGVkX18OvJ1ue0pegQT8VMjNt2+eclo223girjmjnjrVlTpDFN6Q4h+IlB/dv6XYZw1NiODLLdGMN7p9yASyy+Ynji9wHxJi2iryX9l/Ru1sIjwGrxnLZLkOo2f/yu0dco6KstJCsE33DzwFF6DhOpjZxoNphtExZ29OYVsE7SxeVUM+NAbtZrDy83kYT+bQdYZSsz0e6svLQU+55vy1L4D8nhXxf3ujQ23xCmMe4N06/v18iU7Sg1PJGWMuJj2SVZQzBBB3qcqikd7JxuG2zm6z4ziJjUA/Qs72MLZfSK+7+nK0ikbh6/QdHbKtkcJSirIwkJ+i66ihAX9P/1Q+L4Qq519nTZAxnwL7HjFsQ54dOz5p2qMy37dLr+nhW3nbcdBQn565CLNPmhbRPAvX+xIuE/s33m5LxjIHQARUuSRawkXAzfNrl3KNrK0maoM3M34/THyTMoSfy5jKz/GzbrzAFERf74au1A1aqDUlx95YriI/XH7aEkibpGT9ebe/6K/cTS94aNmBgBduvMXYe+/H2bfKWWI7HWX/44FGjgHpfbfF82V9Ny9eVniyJt2G5ujkDBUouN4zrsdf7OVuvLUtFEmDAUcUuH8Z6VA7uCkRfIFDlOZ5olEy2WRbwJ00cvT6FjI7SyEpnHUc55o4B9YSHblHwMHPSlqrUi0yuGo9D9r0MPIivC3tjGFc+jpnY8SbMXw+S3CDlkQ9hj/gIYCMoBzdcrDNr0H4VOMNdPLl4P6V1A3c2UbZYBGoLvMUK5vaISpo0OmyVQr97UyoAbnjy0a/F0HhDSLgKjxLyOD248I6spQM4OtrwNtP76SHfPp+p+l7cDwW+S/TOi2o/WyurnyZ7pkBb66TjIdaV8S6d0aZX6WXljIZNTr6TNB/gQWvRmMZxnFbNIL4z7wXuNE5gU+rbiWbRd6LTt24ZDQASWOImVxxXIjfMXxJNJA6tiFWJ+2KcfEdnpv QUphqDMR wFr6i+a/EvQiZH/NMzkCs7XJOSNZ5MjsEbw9leq96UkiadPUqSYdpaUIEBD+Ti+70aroiOYFbg8cKEXLW5ABi4pFYs52pyMVywrAz953+SnFG/RCqoQPfnCvX3aL9x2CDt3Gjg151bXBpg/YdsZqItJxK5VTVC1s9tupqC3CFk2A2I9TEuYI0jQ9VEWguBEJFSysOnV+XVS9Zm5FxED6/PoOxUDEQjFOH8uDNuTOs3KMjUps= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Architectures like powerpc would like to enable transparent huge page pud support only with radix translation. To support that add has_transparent_pud_hugepage() helper that architectures can override. Signed-off-by: Aneesh Kumar K.V --- drivers/nvdimm/pfn_devs.c | 2 +- include/linux/pgtable.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c index af7d9301520c..18ad315581ca 100644 --- a/drivers/nvdimm/pfn_devs.c +++ b/drivers/nvdimm/pfn_devs.c @@ -100,7 +100,7 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) if (has_transparent_hugepage()) { alignments[1] = HPAGE_PMD_SIZE; - if (IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)) + if (has_transparent_pud_hugepage()) alignments[2] = HPAGE_PUD_SIZE; } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 5063b482e34f..cf13f8d938a8 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1499,6 +1499,9 @@ typedef unsigned int pgtbl_mod_mask; #define has_transparent_hugepage() IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) #endif +#ifndef has_transparent_pud_hugepage +#define has_transparent_pud_hugepage() IS_BUILTIN(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +#endif /* * On some architectures it depends on the mm if the p4d/pud or pmd * layer of the page table hierarchy is folded or not. From patchwork Thu Jul 6 08:58:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 289B5EB64DD for ; Thu, 6 Jul 2023 08:59:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B4EBB8D0003; Thu, 6 Jul 2023 04:58:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B06958D0001; Thu, 6 Jul 2023 04:58:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C3A78D0003; Thu, 6 Jul 2023 04:58:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8CA9A8D0001 for ; Thu, 6 Jul 2023 04:58:59 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 524A4C0B7C for ; Thu, 6 Jul 2023 08:58:59 +0000 (UTC) X-FDA: 80980587198.05.A3C49B4 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf09.hostedemail.com (Postfix) with ESMTP id F130414001A for ; Thu, 6 Jul 2023 08:58:56 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Ultjgfk4; spf=pass (imf09.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633937; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R97SYAHPLPjbUcnqCpDHlj9OaI8fme9l4271zttXIHk=; b=1QLKG2rF8fMeY7E7L8oZxu2Ap4dtaoDvmf5L2pYw+4mzBtPKx2QzYR2f7tm/fX6Mkx1Xlh UEpfDvWnSu3alWCpRqv5emFJmIMeNbeUATJ6Rd9xrrOYP4L4Usw6vc0Hlws8S+K2yVMQNQ z2uQEm3/0h9/k822ZPHqX1hWLMhJ5UM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633937; a=rsa-sha256; cv=none; b=YUG4lcctkMCpJhhA+VVERCqjVZ3F8rqAG9NgmmwH1cv9PI9iAilt56Rx4mJkaLXkxccqD0 jTPvokICJleyHccz2bMeJP8R+5cUKPWdWoenbCk7ypnV+X21sf7g3+jNI9WIV3Q0kJen81 SdaOsNRr3w88BpFgSNEu8fMSA51a1NQ= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Ultjgfk4; spf=pass (imf09.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668khkI018514; Thu, 6 Jul 2023 08:58:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=R97SYAHPLPjbUcnqCpDHlj9OaI8fme9l4271zttXIHk=; b=Ultjgfk4yRMCJ57q21Uet95GL54qaEeA6e3WOUGwLJejtGPMHXYZi7mC4uaFZ/Wn0Bzo iLG3nZ8L2wfaOJf7ZLn/ZSzm/1Y1c8f0xhLxBV4C/8iDoRFeSz5wPAMRJPPz0Ew5kOoo rZYxAiZyQeet1IPZrDz6H5gnkL6hLFUH6AOAHA9OwNNJ8uVQTaJSlAaz6GH/3UrlVgik FNseQFf6nq/ta64hCFMGaOHNJzsiw1Du6AQRZ5j2yerqTnTtxXAHhGSjQ0VqHkCQEwDv e1/uU9yUoFksN2kXKX4cEOj0kJ7l2rdLmW0a5POCV3HsUwvPcBVE1YVP6XHuTc5Y5/oy Fg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntd80axw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:45 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668lHKZ020678; Thu, 6 Jul 2023 08:58:45 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntd80awv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:45 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3668CrUG002602; Thu, 6 Jul 2023 08:58:43 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([9.208.130.100]) by ppma05wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs5yxuc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:43 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668wg7k2622004 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:58:43 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DBBE558058; Thu, 6 Jul 2023 08:58:42 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4490858057; Thu, 6 Jul 2023 08:58:38 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:58:37 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 02/13] mm: Change pudp_huge_get_and_clear_full take vm_area_struct as arg Date: Thu, 6 Jul 2023 14:28:08 +0530 Message-ID: <20230706085819.827761-3-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: XX7uD4WEu9X0qdBPaoo0z8dwCYvAsfCc X-Proofpoint-ORIG-GUID: vG_aU0roAza1u8QnNdpZflsGLD6eLemS X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 clxscore=1015 malwarescore=0 suspectscore=0 adultscore=0 spamscore=0 bulkscore=0 impostorscore=0 mlxscore=0 priorityscore=1501 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Queue-Id: F130414001A X-Rspam-User: X-Stat-Signature: e6th3tf7tr7c9bsemgjpeea8p74yf8zm X-Rspamd-Server: rspam03 X-HE-Tag: 1688633936-800404 X-HE-Meta: U2FsdGVkX1+HfYPw6H9fwwTqviPxU9VWwlqHUUYukUk7+3QGHlQogxEW4HLLh2m9ovcMq7qbkTXc1AGMG1MFAKJrUBsvLqxmcDgmWz7eyy+JnLRW6LMNl2mftE1zfpdBV43WxhNqVOnFU9Hf8dY0mlU237ofxEsQD2xNk8VTtFUeJdYzUSVOUjdpXM7X3XzNRM59H3X7njp42e14wZCQY2bXNF/RkSQHhgWYG1+5yh48pRGPOCeryUMCBfM5qIEuxZPU9DnqJqxTipK3vkbvGVdqx0DK3WWIgu1Ls6hQqvHuyNs0bWXMmdQ2zTIGkopgBLR432oh7fqQllQVdbdGsRwU5jgiRHeD+byK8c3RjOKikwdMiCulCUz8lABrFepq/BufadBvMWMg0taiJcmJtkqgsIrEsSck5j/vRJIHvflYhDXe7C16rw7xwtK2YuA/6Z2wZzjuoX8lHFvCrXhx1fcKCFGgm3bCyPQJW07vpBPMP4r4XR63NEdhNzvjWZCdsDkHsSn9ak6p3y0WvHR/5/LYvDoDDFdCRCOVBDnN4iM6W/BSgldi20rVgWm/bfiT6ybeXEAWb/qYZo5xca790QdZE4ZzWXOQ96Ba3ymykyVrYAjnp1VjbLCTFAhQTGp5n30NNiZUekA7h1WlZj5Aje3B2tOPRrIb51NdpwZyItw+iMwlMK2f3uXqdZqenVdfFSs//zCikQGeTC0E545rMqyBHQjVJX29Aq16uyEwwzLMuTzM5KdNxtXMaOOS/JRS5XNIYCdd5UZtiPXQrSQbxa3LcIzzSVzgeQR7eIC0pDrw/nAzS6gIiwkPZmSYvKvjtk5dug9dr5D3cbGZQnNgMg/Dg2oaQ/2vkVHDeZlEl0uTsG4Cak9k/bni95fyFB4zYIPFevEUFvjfkWC+g499d5H67l9fqyYgoj8LK7ddENUHRPnksEXIYjEObRoS23E2TSBbq4MrBv5rXaFMalT EYWmUW6K NTzYkoNZyp8VzlU3X7KqymRb0tzTfbQvuHjpGqkUwB4+YXfBg8IlRgdbFeEuGIv5813Hpav3kHCEe8okGLmSytwpbXqePyvqL91BUEZD7t1fcy73W33Ecetj8gil+Pgl/Dp2qMRaXsQoTwVoiq9l1hipbmVegB0c4Z19j1JddHH6qtpQQe9r4maesKwOHLa72g4KD12ncYO2qgTbwXWMXVJdxrdNJKIKaXCUifC90pGEWK39S+263ir7LebacvhTRu/Rjov0jFBIybMilptr0MocgNQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We will use this in a later patch to do tlb flush when clearing pud entries on powerpc. This is similar to commit 93a98695f2f9 ("mm: change pmdp_huge_get_and_clear_full take vm_area_struct as arg") Signed-off-by: Aneesh Kumar K.V --- include/linux/pgtable.h | 4 ++-- mm/debug_vm_pgtable.c | 2 +- mm/huge_memory.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index cf13f8d938a8..6fd9b2831338 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -450,11 +450,11 @@ static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, #endif #ifndef __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL -static inline pud_t pudp_huge_get_and_clear_full(struct mm_struct *mm, +static inline pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long address, pud_t *pudp, int full) { - return pudp_huge_get_and_clear(mm, address, pudp); + return pudp_huge_get_and_clear(vma->vm_mm, address, pudp); } #endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index ee119e33fef1..ee2c4c1dcfc8 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -385,7 +385,7 @@ static void __init pud_advanced_tests(struct pgtable_debug_args *args) WARN_ON(!(pud_write(pud) && pud_dirty(pud))); #ifndef __PAGETABLE_PMD_FOLDED - pudp_huge_get_and_clear_full(args->mm, vaddr, args->pudp, 1); + pudp_huge_get_and_clear_full(args->vma, vaddr, args->pudp, 1); pud = READ_ONCE(*args->pudp); WARN_ON(!pud_none(pud)); #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index eb3678360b97..ba20cef681a4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1981,7 +1981,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, if (!ptl) return 0; - pudp_huge_get_and_clear_full(tlb->mm, addr, pud, tlb->fullmm); + pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); tlb_remove_pud_tlb_entry(tlb, pud, addr); if (vma_is_special_huge(vma)) { spin_unlock(ptl); From patchwork Thu Jul 6 08:58:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCA6BEB64DC for ; Thu, 6 Jul 2023 08:59:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DF688D0006; Thu, 6 Jul 2023 04:59:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 690D08D0001; Thu, 6 Jul 2023 04:59:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52F828D0006; Thu, 6 Jul 2023 04:59:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 46C028D0001 for ; Thu, 6 Jul 2023 04:59:05 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 099E6160C36 for ; Thu, 6 Jul 2023 08:59:05 +0000 (UTC) X-FDA: 80980587450.19.1EBCBCB Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf25.hostedemail.com (Postfix) with ESMTP id A7F7DA0012 for ; Thu, 6 Jul 2023 08:59:02 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=iG1zm2bc; spf=pass (imf25.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633942; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wRpOvZEenjvDphBufXXJXjaJM9fWCo+EyUTVYUwsbTE=; b=ZUpMYz/BqyQEuAYn1yhFBKqhDoSt1xZDcfHwD9ZKpkyP3//tcCIjiUwurnn2oGznDsAiwJ 6yYTF+N7sj0Uf60Nti9MgAdstB5Bgak4kOhjIi0E8O3wM7ta0NTTHNKNgt8a5En3gmYoZw n0qxibMQIjhE9VTNQ1fqZwRZQh10EDg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633942; a=rsa-sha256; cv=none; b=6e0riIBdw9QoTAYMxjW8UBXVLlUxY+7epQxZDS9oqJgVfaoZzk3d2FL5ziZZjjgSOrlv35 lWEjDZW5rHYxL2Ygjh9/jqKoMcbJggfGCeThJuaFUlOY14cyQ0hCgUEkDuL02rV+t9aPqz WoOSV8L+/EK1tPCJRxOSkN/FNWunaQs= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=iG1zm2bc; spf=pass (imf25.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668mSRt012947; Thu, 6 Jul 2023 08:58:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=wRpOvZEenjvDphBufXXJXjaJM9fWCo+EyUTVYUwsbTE=; b=iG1zm2bcmvr0m3aEc3ZhzyeVEctsTzRpc4vXyqFbQteqEDyPDMxfJAuBcfVKVMOrh7vc HJ6wFCK9Wim7gx4d+THahjsKi9pnDR2ROsQg4B+96CgKwTXFI7A1QVJZ19VEDwqrVD6Y LuplTicjvt5CL/Fx0ESGvelSpXE4wj81HLaIi6sdvRgDhdoWtWLVEKEUeo20aPTcwN7y YupKnkwm6rec/UlYb3ltTE/ziA0t4/6726wU6nxIyvhXf0Bsez69y9RgSn6v8OKcpKkk owQzgyaHzDgM6aIHLi1qTMzYUG7Lg9/W5pAhOsg9TphdUTP+t1hwaAECNx14ut1jaCnt pw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnte408pg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:51 +0000 Received: from m0360072.ppops.net (m0360072.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668oUkj024681; Thu, 6 Jul 2023 08:58:50 GMT Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnte408nv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:50 +0000 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3668U1BB004414; Thu, 6 Jul 2023 08:58:49 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([9.208.129.113]) by ppma01wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs5quwr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:49 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668wmjR66191850 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:58:48 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2DAF858058; Thu, 6 Jul 2023 08:58:48 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 73A7F58057; Thu, 6 Jul 2023 08:58:43 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:58:43 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 03/13] mm/vmemmap: Improve vmemmap_can_optimize and allow architectures to override Date: Thu, 6 Jul 2023 14:28:09 +0530 Message-ID: <20230706085819.827761-4-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: w85QwxJHUvyU5O4k89ARA3kbIUQVepph X-Proofpoint-ORIG-GUID: q7Yv54e6Sp5kY0GDyNPjEzDJrq-2NIvR X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1011 lowpriorityscore=0 bulkscore=0 phishscore=0 impostorscore=0 priorityscore=1501 malwarescore=0 adultscore=0 mlxlogscore=943 mlxscore=0 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Stat-Signature: hqmmpkyeup45unhaxw4qsb1ka6pj5y9b X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: A7F7DA0012 X-Rspam-User: X-HE-Tag: 1688633942-843042 X-HE-Meta: U2FsdGVkX19K1paxFMFcSLYqaUBqST0dh25955R9+hTnc8imnFOwPVg1tMZdBrkCyy5MTudLy1ye481JzAINI5Dq6NGVRaNfxm78wvhr21r5xFmY4dUrtPgaGPJdtEcLxqxf+RUq1qG2d6aAtKaomEoGz28C1y7NNOj26QLnUkOFdV1KSgsjPM5PGy03Juk0MxLkkH1GrvhRaiVKD/5uX77dB9/FADCACO6hAeasoXmh0P1pIeqSny16qzVzRs+DZ3w7wq+tbxT5zRT6NPEj7toIzv6JAEP+WIVKPWNpvYXZ8e4XmXCwszc5B7C1fGRjWag9X/ler8UlexaH2ozrKpiG0WJcD2VD1a6uU1EJlyRN20uBPRQUcg94v5plPpfzscHdf59LeN+Ow1ZlurTbxFO/+t3wpRX99+utFuUH+LZbEeMgcCrsPYK+CXysEWdNgtJYl9SZNaDlqfY8u4MaKVAFWZFonqxNp5eybQu6dpWCd+9XZgsXw2jlSzIv+XkD7ie/WVUWDqBv2VuI9ZSp2zOvToLNh3lbkqcw6QhedYqCEScWvHN4Qrldy78dfS8XcBCF83uvZgsQYbBAY0E03Q30QMym3/2zoEfSxn0hpboULEw+HJDyiNrmvdXgX7TUeXUBbFvFG6Eere3VzAl6Cw5z5gb/G/BGPcZKJGWWwXcEhIcC4SpJsVIZXYnxodm9vy37nuNdKHXDJMDcsyT5ZI8h5jKr/9f03qqae4IRjdonuSHPsOEpcPTWzodRt6dmsbFV7CTd1A+eRPD1zD7H9sPfeByA0+RKgM5D1GAi1E4u0FPsE+wt+aEvcYGt7mXvLRiqU7A3JJ7EOl4EEh02P7ZoUMWdf1QjbdfH2zvaslYQNNKT5VEOhMGafp2d8WegoPfib1rie4KSEIG9CGWRqZS6O3xg+wEh0K53ZwB7TEQ6oYt/o9CqUXtlHaNrbGopsL3xdOpzu360oG9QnXk ygVyuJu4 +Crsc+Gd6BfnFoGzW9jkaiEoccSaufMF7/m4PBFE3qPMxTYnp7gfdEd+c2T2pVpLlG45wcG/9bliSsaRfg3RhlHjPzmDYeYsybepbF0yPyV7VzS3Ir+4Ufq2nbYaLQxi2V42Q3WNKRKKDk4eNbgp5IEIlfA9IYEju673RHr1h+UbCCPxLSlLcH8l7WgBzJn+AyXWXwLjtHnIepTsINsFxeMkLy8AlrpLbkPpYCP9GQdo6SOwVJBW4JQk5r7F/pxlwgkX9fBqIkloVBLXXZIJJ2K9OBA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: dax vmemmap optimization requires a minimum of 2 PAGE_SIZE area within vmemmap such that tail page mapping can point to the second PAGE_SIZE area. Enforce that in vmemmap_can_optimize() function. Architectures like powerpc also want to enable vmemmap optimization conditionally (only with radix MMU translation). Hence allow architecture override. Signed-off-by: Aneesh Kumar K.V --- include/linux/mm.h | 27 +++++++++++++++++++++++---- mm/mm_init.c | 2 +- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2dd73e4f3d8e..1a2234ee14d2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3639,13 +3639,32 @@ void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap); #endif +#define VMEMMAP_RESERVE_NR 2 #ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP -static inline bool vmemmap_can_optimize(struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { - return is_power_of_2(sizeof(struct page)) && - pgmap && (pgmap_vmemmap_nr(pgmap) > 1) && !altmap; + unsigned long nr_pages; + unsigned long nr_vmemmap_pages; + + if (!pgmap || !is_power_of_2(sizeof(struct page))) + return false; + + nr_pages = pgmap_vmemmap_nr(pgmap); + nr_vmemmap_pages = ((nr_pages * sizeof(struct page)) >> PAGE_SHIFT); + /* + * For vmemmap optimization with DAX we need minimum 2 vmemmap + * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst + */ + return !altmap && (nr_vmemmap_pages > VMEMMAP_RESERVE_NR); } +/* + * If we don't have an architecture override, use the generic rule + */ +#ifndef vmemmap_can_optimize +#define vmemmap_can_optimize __vmemmap_can_optimize +#endif + #else static inline bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..245ac69b66a5 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1020,7 +1020,7 @@ static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap, if (!vmemmap_can_optimize(altmap, pgmap)) return pgmap_vmemmap_nr(pgmap); - return 2 * (PAGE_SIZE / sizeof(struct page)); + return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); } static void __ref memmap_init_compound(struct page *head, From patchwork Thu Jul 6 08:58:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FEBEEB64D9 for ; Thu, 6 Jul 2023 08:59:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 298DA8D0007; Thu, 6 Jul 2023 04:59:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 248D48D0001; Thu, 6 Jul 2023 04:59:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EC898D0007; Thu, 6 Jul 2023 04:59:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0079E8D0001 for ; Thu, 6 Jul 2023 04:59:09 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C6ABAC0910 for ; Thu, 6 Jul 2023 08:59:09 +0000 (UTC) X-FDA: 80980587618.05.C858FB0 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf25.hostedemail.com (Postfix) with ESMTP id 62078A0017 for ; Thu, 6 Jul 2023 08:59:07 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="dUDoM/ub"; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf25.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633947; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=au27mQkFBCX0j7yag6NPyyKfAD/tDcJxx1jGYEmeYuM=; b=YGMAeB3h9vo3Rx+AQ3Y7XEQ1FOsG2olZdXSaWRUq3DdeHZoNDb0Ywk42dBSCiZ3wKfWd/W qzpqCrYc0/GcSTMODXghqf/ttdoxSDmzjeJCumv44qKqBq7voJ29Ra5+VSl0MZJCy/qlXj mCE1YQGfXgIuVcUosSWZLqocxZbQrLg= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b="dUDoM/ub"; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf25.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633947; a=rsa-sha256; cv=none; b=zIkPenHl2gHliytB03dpehBWCJUGDZMDJ3iHPC9zIqW83m6S/S1XrTH1lpgKkEn9YkaqYX W4CIRsGsKEaMi5z3+Oy1wL1rUnliWr2aiAvT4KXxyUjjt10qIL2sTscwLxSRb2Xm4Zjty9 RiB+NmHCsfRuH6veZOxnRmsIv/2IpUk= Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668lZT1021944; Thu, 6 Jul 2023 08:58:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=au27mQkFBCX0j7yag6NPyyKfAD/tDcJxx1jGYEmeYuM=; b=dUDoM/ubOdN0Ov4y+9fGzKrIIHJdhGDS1aYTKSpj/5Vq2FbSXcty9KiW1e9Ww2NbZttD BzAM/rCQTVZLd2l7kgM47ESjqHqTz21pKVInFJXbVx1oJQTk/b6SmE/+rKJPRKne1cm1 BzYzBoPR4QZlJibpZ0DeXbPGitbslQs4cDV0GpG4wfv7MZUa8Dipr9z9mYs7mHsUjC5J WRDKmkh1YZYn8RYiJBB5AtmIkMBiWnSneFj+X9VUKpXTpuuGNC+wLDYL6ZhOkjq/5a3k UMty3RPNLya3T7Z49C33wpRo9WsbcG884eZT3n+GP7aMOOxzD1oqPmbv9GCEKkerHY9V 2Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntdp0871-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:56 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668lcr8022108; Thu, 6 Jul 2023 08:58:55 GMT Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntdp086j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:55 +0000 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3668GsHg016327; Thu, 6 Jul 2023 08:58:54 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([9.208.130.97]) by ppma02dal.us.ibm.com (PPS) with ESMTPS id 3rjbs6n1cf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:54 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668wrlX22020720 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:58:53 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 57B8E58057; Thu, 6 Jul 2023 08:58:53 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B709B5805D; Thu, 6 Jul 2023 08:58:48 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:58:48 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 04/13] mm/vmemmap: Allow architectures to override how vmemmap optimization works Date: Thu, 6 Jul 2023 14:28:10 +0530 Message-ID: <20230706085819.827761-5-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: -Y3c92Zldb-SvzqUci3xDTaBSNS3JySl X-Proofpoint-ORIG-GUID: XdHGGg3j-rkZUsQt9Jwc0XpxflgLdg2K X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 phishscore=0 lowpriorityscore=0 malwarescore=0 impostorscore=0 mlxlogscore=999 bulkscore=0 spamscore=0 adultscore=0 priorityscore=1501 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Queue-Id: 62078A0017 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 53hzqeabrp9558fxgrgto5crb6ryyuon X-HE-Tag: 1688633947-439591 X-HE-Meta: U2FsdGVkX1/9lIYnOcSrWIO5bK55vaSWIfmISSO/mI4tp3sGyALvcMIi4iDqvQD5xkl2OBqCE+G9P5DN1k+EJxzZAbhsuwCSjPPAHItAdIKgq7jJnyQNCyR6fFiSTXFV7nHrPyAnDBW3U8tvHVSBm2/TDlME23Fdsq9TSpWFw3LD3jrR7SuB5ZRCneKU5ZUR0nSEu9cNkCoH6Q4Qsh1MqJJ8zJyt7n9nLXFz4PoGrKWpm1MFGgL/w4uMLE6XbWc4dQ2cQ1qR6HcGZf9YKZO3/WWurAeihj7NKhITU1TmqsH/L0XB+NMtfCPzptvZ7+8jbb1FH1N9Fmkr9DqHc+G4quGFK8EByw2oN4gA/ICk/HPWk2FQNj995BCyYT8N38N5bGtX9TERS6MCz0V9/c4Jcv2BY1LRpeHQz57+ara1ioCtpPRaVFl9AvzC8V1Ef8TtJVll0HerjWymfi8P6bN0UQ9JNerv+w/V4sAESqOHI/iO+KtOdu0X84EKMSSHC9f6fWf4lLw5yaDaA2c6PF2JoSTT+pmQlfQW/QJrjZ/WHMqCnCO7y8JfCdit8YWF5dSNhUX0Qjm+p6RQLnuxLdtodM6mcuD1iLyPyTKmAta//fWqNXoS0dnhWGTk+y8XU/OO5xzQnlZoT30w2zPIvDbH7WepSwjs6WvHB5fgXBakFRar4aQ35SjydYHxTJsnsVn2UyVHFcCre9oIGgPwM8mNotmNGwXnD7omf7j0GbdmYXeRlRxEDhUeeswKgrLamJIiOlVQ8ahIRgE2p5tGk3hkig7gSyJ1JxvAUURNlVq80iFT9uGUXMTl7D8PaBp2p2ZEnv3JFZtA76tFDOvdnggU36zAj27uNh3DtRAEnQ/Rndq3u/x1RPgjf/Jx7Wl/F3MCt1d1PtaDaGRro30E1I2NQ1n7QBd/G2ZMRGOj3xc8BsH+d+0ZguxJ8Gxd/cTzwgQ6ZtJMQmniMUs4pbFoidx ts1byih3 Ow/MuQqFGT6DfMXnjhAzMnxlH8S5f1hAm9dbELJ57BYwgYXDqOJeiERhhf2FkKSiCuIogypqDvdU2gJLEdTq77ibEdwYAnrR/chusnktd4+bWwhEH4ZL2TieQqYTKk6l3bi+aaNOHUVfMPlxx41i0T9VU4Cqb3+g2G1g4un5jjS7yeIKAyLoGZQgznX6Tm4+pRQhzbk8vhd/Jvn0uwHY980MIP/f+dh3JN0tUe8vzpc1zvNo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Architectures like powerpc will like to use different page table allocators and mapping mechanisms to implement vmemmap optimization. Similar to vmemmap_populate allow architectures to implement vmemap_populate_compound_pages Signed-off-by: Aneesh Kumar K.V --- mm/sparse-vmemmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index a044a130405b..541b3f69a481 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -141,6 +141,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } +#ifndef vmemmap_populate_compound_pages pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, struct page *reuse) @@ -446,6 +447,8 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return 0; } +#endif + struct page * __meminit __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) From patchwork Thu Jul 6 08:58:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303378 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB7C8EB64D9 for ; Thu, 6 Jul 2023 08:59:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 61B358D0005; Thu, 6 Jul 2023 04:59:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CB5B8D0001; Thu, 6 Jul 2023 04:59:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 46C498D0005; Thu, 6 Jul 2023 04:59:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 397108D0001 for ; Thu, 6 Jul 2023 04:59:16 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0FBE2804D9 for ; Thu, 6 Jul 2023 08:59:16 +0000 (UTC) X-FDA: 80980587912.08.0755DCA Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf24.hostedemail.com (Postfix) with ESMTP id 606AD18001B for ; Thu, 6 Jul 2023 08:59:13 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=VTjjHpT+; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf24.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633953; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v4rgjlG+j4BFMpAKCsUxYL28soNQi0PCLcozMdYF/S8=; b=w/WtskZOHv0yRvHmUJ2GNkoLnDOecVvdiwbboUWOU7DFUZ3fnSlHKkcgQK+jxu7vIYCXdb qpCb5FAk+sqf0pDKzau0m7i7Cm4KUmbfTRNAC2y+rf9xymT0bBQw087mg0M5WWVoxeqmKK jupyQ/PzC+JtnxESGrntEabZr4fLoWM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=VTjjHpT+; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf24.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633953; a=rsa-sha256; cv=none; b=fEU/xCS5tRy6EKQ+2nBc4N1WIrNnPr42QRMMYC6kRonPzDdgzWUEikm3kZNymYUmclPTog 9ihO9mm3gurtRODu45sej3zOZosds9PQvMKnyapdb+Qv/def8LH+lFEpfF+I9ecrZhjuNF P/1kuIJhigrQbJHQLoNu4S7VU9NhBy8= Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668lleW027855; Thu, 6 Jul 2023 08:59:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=v4rgjlG+j4BFMpAKCsUxYL28soNQi0PCLcozMdYF/S8=; b=VTjjHpT+1SSkhlvAyFeUE2QUnTuR0LPpSemF8P9e2xNNGx2D4Bo9/BV+WS4rpjKiB677 9Och+mx2uScqd8AhjPMPEHrdmGHRerD0aop/r/Ajrs8CuBHW7+YDWbcKk2qORizLBeLw zJ9V5HRySviZqe/1J4Az3UTLJA2tXr9QPALvAuTWyZ8FhZg4goSPxmHyCpkI3B6ymOmA xrHpK8UshgHn1uArZf5w/ZEZoOy6tHYhZX5qcAqnHLt7ovOoIzJQj/3s5i/fJMEBFP4Y IccJLGiq+L0lRRnJv2TeNPsKPVWFJcQIJBmdl1k5I4qdLVX9DV55TgMXyM4J+3gbNAh9 0A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntdpg7wm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:01 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668lx1c028183; Thu, 6 Jul 2023 08:59:01 GMT Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntdpg7vv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:01 +0000 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36687OMW004433; Thu, 6 Jul 2023 08:58:59 GMT Received: from smtprelay02.wdc07v.mail.ibm.com ([9.208.129.120]) by ppma01wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs5quxd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:58:59 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay02.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668wweT62587240 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:58:58 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7B2CE58061; Thu, 6 Jul 2023 08:58:58 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E039058058; Thu, 6 Jul 2023 08:58:53 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:58:53 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 05/13] mm: Add __HAVE_ARCH_PUD_SAME similar to __HAVE_ARCH_P4D_SAME Date: Thu, 6 Jul 2023 14:28:11 +0530 Message-ID: <20230706085819.827761-6-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: k-Sst2htYGDWj5KXzMIJg6MIzT1S8pmY X-Proofpoint-GUID: rwBAueoCyLZfX1dRqPDOGiV00Lb5-bu3 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 priorityscore=1501 adultscore=0 lowpriorityscore=0 mlxlogscore=999 phishscore=0 impostorscore=0 suspectscore=0 bulkscore=0 spamscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Queue-Id: 606AD18001B X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: zye194o4djgbpg7ybhi87pp63t8hab7s X-HE-Tag: 1688633953-611773 X-HE-Meta: U2FsdGVkX1/wyAYr1fC0Wpzf45I9oS70ksJaTuKHQmjEbminxz3inZl6ux8Ss1d/c9YawrX+DHHC90r0JNtw3yNRmacCsNqw7okBs03oQDrEvmTA3FQUxyoY47VawfKEpY5pAiK7z0E4qUo1XYzOGOMJhinDhh8EEV7YyNlCBbK8EJG1SYucECSfgq5hvAp86/LOa/W/UPhWNSaCitpcbZk6XqjPvvakjIH1B2BXAsYXldSXHYTA39lFVS4uMO+4zrBdUUcOVRit42HsdJzaEzl3JbekRf1TOA7XH6s7NMwAKhZODg/5njKvSiYVHs8lKvS2FLQZF+hf4XzcboUHYwQM4S918gkqqoDmyc4Oouzdri257gPuHQkzn2nyRGqvxjw62Z2ifujrQodgOWOqBuEBOxKb0koIQcAw/+w57RTKnIHLn/KxmVw1Ns941299lEF/wtJdEEMVfSizX4HiWJ89GRguuGD7Fiap2UDOh6n06WDa1vzM2+3FWhGD2wnFLHSS74i9FyYy35Ck4sic9+CjHg83FVoAaP3PmuJZiiQbJg1eHlPVzFtSHINb1vypHcRupbRnaQfdFFfq7Y/jbFSEuAbNaL8vgzsooocfoDR/NNW6x1qQEZa0Rb15XaZErCqQKxBq5GwxMkM45MWmh5nSqWUEZIwn8Hi+rkThWvklQq0/MVSxYihvtXUjBsDeppCXS/oWwGj+a3ypn9HsYQDxMJwI1UMyrQQu69ieDBWoYsFEeu2D/LVJ27nPQc1zju92/AgQZ5ekC4aqXvuSg1V9S1ZjmSFLBbMZnh5advUYvyVnFgM/qmuuAG4UJdHKi2yNvLIsh6lz0ZvzWUymYbmPFUuJeVPMTTRfKQl2GNFSdHjA47aBGT3Hie3MiGjt2Z7VHZ0Io65QHMilOLL0OSYOCmK8u/RZJP2GPiSq6X8aoGs5Yd9oBrgjR38s3orY1XXtLo0bsszyMgXyTlv R56pi6h1 /Z2Bu9r6IMjl6Bk/63Q2v/y9GSx1L9zBOGiiOmlhCSJt7A1GEwYTuYU8E7pQ94REAvpOCEAElB6nn1GI24wIqRLefIchua376cflllI7oCbygzZU29QalnFqn/x2gORp1HDDrOBRXb8LyRZKe2JLLiwmCOLmBURqhpt8gFRAjmQZL7H965aD+6V/3hODHG14yRpYw/Bi5yQNBKedK4RBaalti9KurZoW8QDBa2kagRXKOdwRDzcIBSK+MmmT92L7BUyGl63OUmTZ8GpGx+2ct1pjgUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This helps architectures to override pmd_same and pud_same independently. Signed-off-by: Aneesh Kumar K.V --- include/linux/pgtable.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 6fd9b2831338..91def34f7784 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -693,7 +693,9 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) { return pmd_val(pmd_a) == pmd_val(pmd_b); } +#endif +#ifndef __HAVE_ARCH_PUD_SAME static inline int pud_same(pud_t pud_a, pud_t pud_b) { return pud_val(pud_a) == pud_val(pud_b); From patchwork Thu Jul 6 08:58:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC251EB64D9 for ; Thu, 6 Jul 2023 08:59:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 420AA8D0002; Thu, 6 Jul 2023 04:59:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CF818D0001; Thu, 6 Jul 2023 04:59:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2980A8D0002; Thu, 6 Jul 2023 04:59:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1B4FF8D0001 for ; Thu, 6 Jul 2023 04:59:20 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E6CB9C046B for ; Thu, 6 Jul 2023 08:59:19 +0000 (UTC) X-FDA: 80980588038.06.9B08114 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf23.hostedemail.com (Postfix) with ESMTP id 8CF98140018 for ; Thu, 6 Jul 2023 08:59:17 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=bi3KaxNO; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633957; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CCRKKwBNmbNEXrHZXrT40NjFgPyXV2Q+00nl9gdum18=; b=O2SofwaJKXfDouF0+0TdSspjTeGgEZHz9wid6n4n9o+hvPi9Vv4MtpP7t3saNhU8y+QqO/ UMPNWAJsigiIV/8HPLCNLoYH1ut9J0wxZ34jCNPymoOVbQxzYG6cKtxLlVN7TYKU998Bm+ ttRL4F41g5xk72A9UBIP70Unlu1yhm4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=bi3KaxNO; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf23.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633957; a=rsa-sha256; cv=none; b=oGyz5xpJN8r2HtIuSlMtIREh/MOGoWHlOKv42lMovT+6Im2Gks+z94cSkpx0YvRaZsaW/J 0GE0BSQkFb1cKSYF7aYZhEVpK3qupNkMQ56+vuT8OrIOyUJUr23w+Pp3dOr5kOO0CCCLBS b3/x7L0Ou6AP/MfeXXbZ4Ws7RwhFAt8= Received: from pps.filterd (m0353722.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668qQSf013375; Thu, 6 Jul 2023 08:59:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=CCRKKwBNmbNEXrHZXrT40NjFgPyXV2Q+00nl9gdum18=; b=bi3KaxNOU8l0U91MtNjqdDxNN6h7eDdWvXF2fobthN8F+aY5WVW+nzctW5gjy+P80BEQ +JXYfXvjNYqFrPCpuDcGi0kCFPcooS+qH5WOuahw/FFoOBsPmEqvqZ12je+koCB0kirx lMv9RJMr9gQUM/RgkuODzqnDOkvGVHxDV29tJAwzvejE/Y7yhIONafQQITCIyESOTr9k Pxb559O58Ku+pFplOpmEfMmY/IeESQlp+OP7B6HxnospMfk4R2VbUpPJzOgZJvRWp5yd 4Qidy7mWCPkHp6fhegFDJ2vh7Zlw6YmAzCsW7XYOCryiL0SJHADBdIihHLMQe8Pvx9No jw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntfrr485-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:06 +0000 Received: from m0353722.ppops.net (m0353722.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668sQO0019849; Thu, 6 Jul 2023 08:59:05 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntfrr47m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:05 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36680Yo8023003; Thu, 6 Jul 2023 08:59:04 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([9.208.129.117]) by ppma04wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs5qy1g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:04 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668x3fI1114834 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:59:03 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3A93A58064; Thu, 6 Jul 2023 08:59:03 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0997A58059; Thu, 6 Jul 2023 08:58:59 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:58:58 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 06/13] mm/huge pud: Use transparent huge pud helpers only with CONFIG_TRANSPARENT_HUGEPAGE Date: Thu, 6 Jul 2023 14:28:12 +0530 Message-ID: <20230706085819.827761-7-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 0bXOmG1rEAB8OaUVCUViSM-43wJivQJV X-Proofpoint-GUID: TUKWLIiVIJVFytlnvJjJMP4oj8752eB3 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 lowpriorityscore=0 mlxlogscore=999 spamscore=0 priorityscore=1501 malwarescore=0 suspectscore=0 impostorscore=0 adultscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 8CF98140018 X-Stat-Signature: wgezur1ogbwaapuwzk17itex75b1npzq X-HE-Tag: 1688633957-219754 X-HE-Meta: U2FsdGVkX1+V7hNKnnZr89ri7RvkDx74YP+2LDOtvlAcNTFrYFhCmNVEZ/JjWyNqGjyXZI45asGh9HB24p6VCAHcTpcic3iJNh/4MzszheZRyhru172L5mGddPNKZ4AmFmC203guJCmprjSovIJXyFMhFV89Zx4BYhtYBaCk/wMhUG0taVtQCEBcsDqtrr5nOX1TL2swIUiVl+bpH+BZvbKKJIEEb+cs2OC7C0dRlVfTDXxA2qkmuOehbJnCnBkTfZQ/T69ARWnaBcGLETYKU3CiSbXJBaTolbsv9Lr+iPgWg95MFiLCWRjW4LwIZb+mE4pDSuqQ4BA4+pjN4xP5aU6n1TDROett8Mknnw8VuTzOpULSb3dSCL+DM0U0VaZ7mbXTkRakkn5VVQ2NonMv8Rx8nkzMb/UEWRuRvMrYyrbOVpRs2mZVc1XeFxRH3spwH67GaBcIBDmakd0J0m8GKZ5C4KdsZjtZ600hLnACmEKBC4igM/c/Es+LD1/hlTQkBF2JeSru9NJTEVGsxjkGAuHq8f6EX2GkSmaf/bDmLrPHrIHuaEY9gev5xKT0h4ijlhJA68dHR0Qr97if0bIU5onttbf6Uq7JSRc6RdGjlDsmcBw6cAS5RxHVF5Usq9jtWo9cVXtARycJRogdr5P+kVMmu8FvTyoeWZOCyJi0tkr5cjkCbk49x58Ezx+/9NTm2t1cHrJkwj+739aoux7GM5B+N5oHWvi17YC43HrAh3XcXHExRrnBS/Kc+98DadWejm1ZSFosUalb8LnWgLt1r1iDSg0WOJFDEOI0iRqPcCv7k+jUGlsVkcZ4pM97+3Ccb7/dpjTRVgygN5wv41VXYwWfi/d14hBIKK9KCrKHQMWFXVf5mrc2S9GjcmlgNL6Pu5MZ1ZWUBMq9fj7J4I15lARliWkWoyQksAA2SqKirm3gbvtVXsRivzYXtVL2o6WXID393hVuGSpPhX1opxn zII6l9nO KsDTzFvpdRxi6lHEDN/LkP2fbSa8elzM9W1P9RJityB1lGdr7q2cSJxhHQR7WW8LwDsV/RxmFade67CD/fXYqxgKGR4Sdr0iNLyCXAZ9F6gvxz+FIC6ts0DLj9mKZOZL/cKMng11q5VaWK7OeKEUWQYWSYRxOwglbnBwBQoY8KGqINykgXulqNhizI/HMZV8NXWfsVjWKAZFj8/nwSiJjU9ijzrHUX226Az3GbNsLxHPMO7cBPLWRSxcWIrIL2aDE2CppnfSU0ydLDgJaOQ9UCE6+9vvl3o7NIQm+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pudp_set_wrprotect and move_huge_pud helpers are only used when CONFIG_TRANSPARENT_HUGEPAGE is enabled. Similar to pmdp_set_wrprotect and move_huge_pmd_helpers use architecture override only if CONFIG_TRANSPARENT_HUGEPAGE is set Signed-off-by: Aneesh Kumar K.V --- include/linux/pgtable.h | 2 ++ mm/mremap.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 91def34f7784..b5af3e014606 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -558,6 +558,7 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, #endif #ifndef __HAVE_ARCH_PUDP_SET_WRPROTECT #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +#ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline void pudp_set_wrprotect(struct mm_struct *mm, unsigned long address, pud_t *pudp) { @@ -571,6 +572,7 @@ static inline void pudp_set_wrprotect(struct mm_struct *mm, { BUILD_BUG(); } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ #endif diff --git a/mm/mremap.c b/mm/mremap.c index 11e06e4ab33b..056478c106ee 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -349,7 +349,7 @@ static inline bool move_normal_pud(struct vm_area_struct *vma, } #endif -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr, unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) { From patchwork Thu Jul 6 08:58:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDD53EB64D9 for ; Thu, 6 Jul 2023 08:59:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A5AA8D0007; Thu, 6 Jul 2023 04:59:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 555ED8D0001; Thu, 6 Jul 2023 04:59:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41EDE8D0007; Thu, 6 Jul 2023 04:59:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 355D78D0001 for ; Thu, 6 Jul 2023 04:59:26 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 02C43120AA0 for ; Thu, 6 Jul 2023 08:59:25 +0000 (UTC) X-FDA: 80980588332.10.9E794BA Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf27.hostedemail.com (Postfix) with ESMTP id 7CD2D40004 for ; Thu, 6 Jul 2023 08:59:23 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=KhvqwxvI; spf=pass (imf27.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633963; a=rsa-sha256; cv=none; b=VBkroNznHBwj+ETOknPL8alCUyd+U3rAzSQ+9Pr3ZNRqNHAYH8Ksq+2pzQhVT3y31kwIcE 6t1nlP25xN/3fqc/0cAAPOEzmKIfmy8Q1NW9id6BplNQaD+58hIqXwAFDKRzGbH3k5Txv0 t3lBV5vBWrAxL3Jqh0EvSgu/Gg6jJ34= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=KhvqwxvI; spf=pass (imf27.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633963; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AoFTsQX3X9wpdrWskmvUov4h2rk+jPDdNe7QeKoQYz8=; b=pMvks4sR5A3lNdKW5RXeSRXCuxfGWWgvFPH8nnqWLgocYy/8UHvkJm89JwmosGbmHl3Sha 94m3cyC8sizRv0v5XtEw+vVisSJ4csrvfsr94BDTj3lfChVYJBjxPXr4CIzps45ne3yiPs 44dU9nRgaHltjVaBZj42jZ2PBuD4lbc= Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668lcoi027627; Thu, 6 Jul 2023 08:59:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=AoFTsQX3X9wpdrWskmvUov4h2rk+jPDdNe7QeKoQYz8=; b=KhvqwxvIC0dbviJD4zdwptA47UpSfYmKjWffF51O7ey+GeMK/tGLRUT9L/JdpCU7fNuy nylCpLTDZkMSDQiktwv0/QbUV/C2nt7qhJr7h/3CNWT7Ankb+CqVOoXbtD+R9IfUmJMe nxmxNk9LBRi6gaueGDtu8h3UEha11TQleBN4As+chjSaEKUBnpPP/p2LIsesDUsl/Fcm F4NOxteuKdHL17fB7akXFxrADQlyFYGMdOFzh+/E9kF00zT53/idhLQEotVx86jGT3Gb G7xy80s/9zM81eyrHq2LsY6TfMhZEWCxhoOTKnFf/J1mE1jF/LZI78uJQ2FbOx3ZP/ma xw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntdpg81r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:11 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668lbif027611; Thu, 6 Jul 2023 08:59:10 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntdpg813-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:10 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3667sZ5Y022623; Thu, 6 Jul 2023 08:59:09 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([9.208.130.100]) by ppma04wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs5qy1w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:09 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668x85n54198742 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:59:08 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2A01E58061; Thu, 6 Jul 2023 08:59:08 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D26DE58057; Thu, 6 Jul 2023 08:59:03 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:59:03 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 07/13] mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization Date: Thu, 6 Jul 2023 14:28:13 +0530 Message-ID: <20230706085819.827761-8-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: fFcf-_WuQviJk9t4gL29Ja9EwY87Eue2 X-Proofpoint-GUID: mWdNBaQ5o1lthiivcF_IMxFvtvXpiJT1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 priorityscore=1501 adultscore=0 lowpriorityscore=0 mlxlogscore=999 phishscore=0 impostorscore=0 suspectscore=0 bulkscore=0 spamscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 7CD2D40004 X-Stat-Signature: 3muze4dr5fi8sbexrbnfwrwu1whn7qe5 X-Rspam-User: X-HE-Tag: 1688633963-594218 X-HE-Meta: U2FsdGVkX1+whFiWg4dGF3YhhIdjZh4tFuBzwJLmceGLQiBukOobZV6XQK7QNuTwdXI5ZvLTf2ZhAg7dawpOTQAS/POZVECrQmbn5VhK15Ul7lrrSgWi4UoVv6rM9rENluxP/tPLhxog+TOHFCGElC9LXw3aVMq7ORMRhCDBh/eCxiHG2bxqp0AtMKh1brk7EAKcCblMiyHXtA85jQdMnoiLKJN21gcbDiLNNcnGZaGOt9vK2PTstIvwTEAVBanKbXrhN44ImZ51Oz5gDHfMJvjudN4jGKkS9CfTW3h4hboSmVRMNgtihILL2Zxpu0+5wbziOHytOojfDd3sR3jZtYL4BEa/UiYSoqE1J0O+euLHNIINO2RUu6FXJl1RPNlBXG13ZCO02y2haUTW535Up+CE6kvd8TwMyShSE3DLAkBtFbwtPZbGxcPIOtJfgAeS8FyS0xFgUomafZJXSzvRurnG6Sax5ZzkOlWt9zQ9qlcyr/xy7decXWDQNzciIvcryGZ1Blkwog2qiE0irKxyQ3brDfvlXs4abQd4SWRB71bngdlbp8qV30eZyFhOI4smXZ+KCpJmWVD7DS4aD1+GluH1DxbYll4ppenQ48lQT++bZO+4+6WiVDGxbc8rc4BqaIdvNDMDwvP1vs5oNacWgo/no/CjkbT1zxfiyHSMUB2ZFW/j71pZtv7EAp2ti/orE0yXfbDkb64gJf8B6Yg/HI/6erhtVBBywMofgmpllI8Ia1Y1DSMElfqdpFaafuJTpXuFI4p2ww+jXuvzzi9JpwoLgOV/gOt2SnUM+V3dYBBTKV3gdPZ4+ny4DgL6m1JzW/gkghSOZH2DMuHbKrZLsvn/6544YPmf7OJpTm/kHy0KJf/R42HO0jI/6a6/whYWBLWorGp3pnM0IAzfgRaytjwhZkSZ5ors2DcjDm5FbVoUq0IRGjCwU3fS5LZuhiY2lGuqmbXLnZ9aVl2BD9s QcNztWZo WEIYlzQYt5vgIQ6vk2sZu6mVh+KqyTcwMiTGHoXPOGX21XkkTuK3aJNo1Gvu3WQR7yzT49hk+pKoVxqkJ2QDbwaWPVfK9Wr3yv7o/kksYLnYfRlCWgZkChBiEjcMH6EcObnmCOkjIhbyoXIGHdfZaDgZNmdW0YO1cX3qO2tKQlxE3A/72BCVuUvOjHBehTdNai2ysSxeWduheil8oCXstcxh9aIX52UlGUjsed5u54QRbRkQIdgDblTVWTx1zkHq/2M6gV7Mar8V8nfg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Arm disabled hugetlb vmemmap optimization [1] because hugetlb vmemmap optimization includes an update of both the permissions (writeable to read-only) and the output address (pfn) of the vmemmap ptes. That is not supported without unmapping of pte(marking it invalid) by some architectures. With DAX vmemmap optimization we don't require such pte updates and architectures can enable DAX vmemmap optimization while having hugetlb vmemmap optimization disabled. Hence split DAX optimization support into a different config. s390, loongarch and riscv don't have devdax support. So the DAX config is not enabled for them. With this change, arm64 should be able to select DAX optimization [1] commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable HUGETLB_PAGE_OPTIMIZE_VMEMMAP") Signed-off-by: Aneesh Kumar K.V --- arch/loongarch/Kconfig | 2 +- arch/riscv/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/x86/Kconfig | 3 ++- fs/Kconfig | 2 +- include/linux/mm.h | 2 +- mm/Kconfig | 5 ++++- 7 files changed, 11 insertions(+), 7 deletions(-) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index e55511af4c77..537ca2a4005a 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -59,7 +59,7 @@ config LOONGARCH select ARCH_USE_QUEUED_SPINLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_LD_ORPHAN_WARN - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select BUILDTIME_TABLE_SORT select COMMON_CLK diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index b49793cf34eb..fd9b683c4263 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -53,7 +53,7 @@ config RISCV select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT select ARCH_WANT_HUGE_PMD_SHARE if 64BIT select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 5b39918b7042..975fd06e4f4d 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -127,7 +127,7 @@ config S390 select ARCH_WANTS_NO_INSTR select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2 select DMA_OPS if PCI diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7422db409770..78224aa76409 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -128,7 +128,8 @@ config X86 select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_LD_ORPHAN_WARN - select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORT diff --git a/fs/Kconfig b/fs/Kconfig index 18d034ec7953..9c104c130a6e 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -252,7 +252,7 @@ config HUGETLB_PAGE config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE - depends on ARCH_WANT_OPTIMIZE_VMEMMAP + depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON diff --git a/include/linux/mm.h b/include/linux/mm.h index 1a2234ee14d2..83f51ec0897d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3640,7 +3640,7 @@ void vmemmap_free(unsigned long start, unsigned long end, #endif #define VMEMMAP_RESERVE_NR 2 -#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { diff --git a/mm/Kconfig b/mm/Kconfig index 09130434e30d..923bd35f81f2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -487,7 +487,10 @@ config SPARSEMEM_VMEMMAP # Select this config option from the architecture Kconfig, if it is preferred # to enable the feature of HugeTLB/dev_dax vmemmap optimization. # -config ARCH_WANT_OPTIMIZE_VMEMMAP +config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP + bool + +config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool config HAVE_MEMBLOCK_PHYS_MAP From patchwork Thu Jul 6 08:58:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17004EB64D9 for ; Thu, 6 Jul 2023 08:59:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A62328D0003; Thu, 6 Jul 2023 04:59:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A11638D0001; Thu, 6 Jul 2023 04:59:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D9468D0003; Thu, 6 Jul 2023 04:59:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7EF798D0001 for ; Thu, 6 Jul 2023 04:59:29 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 537B0C0B7C for ; Thu, 6 Jul 2023 08:59:29 +0000 (UTC) X-FDA: 80980588458.09.F805E44 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf13.hostedemail.com (Postfix) with ESMTP id B7C5120023 for ; Thu, 6 Jul 2023 08:59:26 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=iruaTUnh; spf=pass (imf13.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633966; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HJ59ItgsLbPBAoIVj3eoS5U1bMjw9YWn4IbBZYPC7LI=; b=fRh9TrWItVVEkAg2DFs0txly91LZxPhEun026u/H9gJTJlj7HhjUfJTUPfZjH0gAogjd55 zo8xosRNZo/zOOsS6mhDAgdSq+rMplvyEZjOx3tjqKIU4Xsy/P17CLEYTyRuVb6uzKYhYV 9wUvAUi7rrL5vW2v9bb+73LnqsS18bA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633966; a=rsa-sha256; cv=none; b=r/InYgJUiYNBA3Qs7PKv9TF/m+b6gVHOkkOlCfbYNl0sXr/6b3jPARI61I9rSVR5pCgxTX yHxHe7yL5vqfhxkb68vaTeIMjZPJ6Q+f4ncXwfRqK77LkWRze6GQGi+m3+xste3rhFps8r oYFKa3BUiiFwAhzmTE3Ou8tNmmZNa3s= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=iruaTUnh; spf=pass (imf13.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353727.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668vRud005067; Thu, 6 Jul 2023 08:59:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=HJ59ItgsLbPBAoIVj3eoS5U1bMjw9YWn4IbBZYPC7LI=; b=iruaTUnhh3vZ5rC4jmA3MkzkOBry/hYwBt+a7RrMVQoJT9W1/MjwWfguhSXfnIq9IAmw vfKVshpOJ8FWXU8vT12hGv7lSPaUHPtrqpQDuRILVdEzDk0h+UxKD4F7KlTD3AYcwCXX DFVJfx/A5jRufQM12XtJdx/1k1XOl84dCOwZdBlzKkhC4S8dqk9QxToqc7jjM2hUbbY8 BOosVaQvVWT3AP6Bajbfy413N9HNwkIzHkoM+FZQ8yl4vqWRf/g8/KKqaHE9c49EMIFf k8sS2rsQufyyEcYwwr07eyP1K4ce604LCPowBwEUebxOJlWrJyGF0UAkvHrasTcFHls2 ZA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntjd00qa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:16 +0000 Received: from m0353727.ppops.net (m0353727.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668xFM6009795; Thu, 6 Jul 2023 08:59:15 GMT Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntjd00py-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:15 +0000 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3668KQ1l008582; Thu, 6 Jul 2023 08:59:14 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([9.208.130.101]) by ppma01dal.us.ibm.com (PPS) with ESMTPS id 3rjbs6d49r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:14 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668xDY18323832 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:59:13 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 29B8758057; Thu, 6 Jul 2023 08:59:13 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AECC758058; Thu, 6 Jul 2023 08:59:08 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:59:08 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 08/13] powerpc/mm/trace: Convert trace event to trace event class Date: Thu, 6 Jul 2023 14:28:14 +0530 Message-ID: <20230706085819.827761-9-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: SWHhsA7jhyy6zDkZZ_IOCtlX5118KuPd X-Proofpoint-ORIG-GUID: WmutxBFBjIqixMIn3o3lYHM4RIvwGYST X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxscore=0 impostorscore=0 phishscore=0 lowpriorityscore=0 clxscore=1015 adultscore=0 bulkscore=0 suspectscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Queue-Id: B7C5120023 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: nqq5bbn3bjsarfg8gg859rbafxyu3ax9 X-HE-Tag: 1688633966-44556 X-HE-Meta: U2FsdGVkX19Ua/Ecif38GEvSOwFIaefUiu1OZDfNtPhWy/owbn3PJc/jGP/y7URCeuR9hB2IWw1+h86FDLFtwBj6d/pRx2blUXyHh+bo4y3YPBLB3t8WyBSYUF4Kfa6Z1KFhNUdyphJYeYW5czj7ttyhP8/ibmLJnJyxVagy/ArsdDMybMzEWP9cqICyzTYYlivek913ScQuTrIR31jKr6V8MfAJceYf92GsrpJyO1TWgDpRPcL0LAcVDNagnWZQHQGMn+JmUHD4FAQaWOAd/8elcIrnEbL/AUsBxX/qE8igo+qzltu8C34dB2jlLHAm+VWIgJGdz7MC3+4iygXPRtxTv6Ma1IUsuMCte9mfFKuyTe+3Jg97V2RBIn0/2aySoJVdTmpUecd+mHaKhwCHXqyPRjGdef/IELXgjSHPKSKYK/Iresp/7zrfvr4Mfh+M10RA4kt58Xg+hVS/eJ25jIP7k/owwEdCAUNvd3lZVDHwG1H3PsPsyNApWVveAN+2mkrOkGMnQPvu/M5teOesLan//tH48FXTU6KFLdhgnzio3CEAleVChcygcgacNkNhNre/wxOT6gYKei+Ky4bRVBPFnr+JfvHHbNbbOCZELTY2YMYPdfEB//WuBbjkstJFvatr3Uvp6SuN74bDMDsrpTs71Ji2PUz3J3AB9bWlVOqMfdU/xbyqmJvlHjoPlO/5Xxb3zDVKGBRH1gJY8onQRXlcYWevkFvLrCEIXIBP2sppfR7XEgDP8lka6jjHGO+YEfkc7FWVqWDIC5YNrNDKwWttbJqMRTF0Gt8EpvXst4IU0WR+BP2WF8zKENYvrH7ISjs3L/Ai9LJ4+GCqwjTakNDIkAcXEax3A2sPWBF6ZBfg8jkSLCviaaVfii4un8X2vW76095VOH4C4F+o4IWSbaDsNdfos41LBB/PdO5SN2c/F/YU7gma1AW/Gp8Wgq+0Nt32mDoOPSIT7ANzeOo k6Oq1KyB BQh4ZnxEc2vaUZv53+WGydjFwJR3Q1dCb1pbvF2qoUgdExTY9KHLWon3i7mGou64OrjLRcNEXWuNCKnRT9GyJFzH9GpeqGtglUbDakP297j1qzY85EX1l4MyGI/sSKglP1soavakfk+lJ50VRHgQF4ixPfP0XIPzKHgImj85+ZeAicec71Z5ru8iA3l16Vz9D840KlTaI3hli0fBF7S/k/V8Do4IYhPwW81/1Be+VYdUmrFE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A follow-up patch will add a pud variant for this same event. Using event class makes that addition simpler. No functional change in this patch. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/mm/book3s64/hash_pgtable.c | 2 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 2 +- include/trace/events/thp.h | 23 ++++++++++++++++------- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c index 51f48984abca..988948d69bc1 100644 --- a/arch/powerpc/mm/book3s64/hash_pgtable.c +++ b/arch/powerpc/mm/book3s64/hash_pgtable.c @@ -214,7 +214,7 @@ unsigned long hash__pmd_hugepage_update(struct mm_struct *mm, unsigned long addr old = be64_to_cpu(old_be); - trace_hugepage_update(addr, old, clr, set); + trace_hugepage_update_pmd(addr, old, clr, set); if (old & H_PAGE_HASHPTE) hpte_do_hugepage_flush(mm, addr, pmdp, old); return old; diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index e7ea492ac510..02e185d2e4d6 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -962,7 +962,7 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add #endif old = radix__pte_update(mm, addr, pmdp_ptep(pmdp), clr, set, 1); - trace_hugepage_update(addr, old, clr, set); + trace_hugepage_update_pmd(addr, old, clr, set); return old; } diff --git a/include/trace/events/thp.h b/include/trace/events/thp.h index 202b3e3e67ff..a95c78b10561 100644 --- a/include/trace/events/thp.h +++ b/include/trace/events/thp.h @@ -8,25 +8,29 @@ #include #include -TRACE_EVENT(hugepage_set_pmd, +DECLARE_EVENT_CLASS(hugepage_set, - TP_PROTO(unsigned long addr, unsigned long pmd), - TP_ARGS(addr, pmd), + TP_PROTO(unsigned long addr, unsigned long pte), + TP_ARGS(addr, pte), TP_STRUCT__entry( __field(unsigned long, addr) - __field(unsigned long, pmd) + __field(unsigned long, pte) ), TP_fast_assign( __entry->addr = addr; - __entry->pmd = pmd; + __entry->pte = pte; ), - TP_printk("Set pmd with 0x%lx with 0x%lx", __entry->addr, __entry->pmd) + TP_printk("Set page table entry with 0x%lx with 0x%lx", __entry->addr, __entry->pte) ); +DEFINE_EVENT(hugepage_set, hugepage_set_pmd, + TP_PROTO(unsigned long addr, unsigned long pmd), + TP_ARGS(addr, pmd) +); -TRACE_EVENT(hugepage_update, +DECLARE_EVENT_CLASS(hugepage_update, TP_PROTO(unsigned long addr, unsigned long pte, unsigned long clr, unsigned long set), TP_ARGS(addr, pte, clr, set), @@ -48,6 +52,11 @@ TRACE_EVENT(hugepage_update, TP_printk("hugepage update at addr 0x%lx and pte = 0x%lx clr = 0x%lx, set = 0x%lx", __entry->addr, __entry->pte, __entry->clr, __entry->set) ); +DEFINE_EVENT(hugepage_update, hugepage_update_pmd, + TP_PROTO(unsigned long addr, unsigned long pmd, unsigned long clr, unsigned long set), + TP_ARGS(addr, pmd, clr, set) +); + DECLARE_EVENT_CLASS(migration_pmd, TP_PROTO(unsigned long addr, unsigned long pmd), From patchwork Thu Jul 6 08:58:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3A16EB64D9 for ; Thu, 6 Jul 2023 08:59:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 928D38D0006; Thu, 6 Jul 2023 04:59:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D9638D0001; Thu, 6 Jul 2023 04:59:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 752368D0006; Thu, 6 Jul 2023 04:59:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 653F88D0001 for ; Thu, 6 Jul 2023 04:59:36 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BAE031605E4 for ; Thu, 6 Jul 2023 08:59:35 +0000 (UTC) X-FDA: 80980588710.13.A96D387 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf04.hostedemail.com (Postfix) with ESMTP id 533A04001F for ; Thu, 6 Jul 2023 08:59:33 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=IhwSH0rU; spf=pass (imf04.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633973; a=rsa-sha256; cv=none; b=ka406AhiCx1c4NiMEfriZPTUCC0mgVYPfCRlfrKjKgLTPs20qDo17XK5YMbrzZ1soL67Cs M3i/eZ526H7HAcWmsTDLaA9e8ovCuxezORCpKa+UdpMT1KhHCE5PgELyVrQ2ov0IIlKM7e Hx2XgfIBP7JeNYfhK+vzT2XcPgBhWJw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=IhwSH0rU; spf=pass (imf04.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633973; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ddyeBSNS2SXw9fOQS7+F0UTnbljUuHDNcF7ItvT7LqM=; b=vDAGghFbsraay+nxS5yZIut2IMyEBDxqT0ZqxHUEEEKPwh+9VyXQzhOShp1N5oHoeYHlfS tYz0TlkFb1UNehTLvycfHjuX0sdG/+xGdIHnCSqbDwetdXYU89UePKKODHjCCyy6l9/BKd YXqfrl3Es8Xt8uYNfqPJhFX8GBG5bvQ= Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668nLSg027161; Thu, 6 Jul 2023 08:59:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=ddyeBSNS2SXw9fOQS7+F0UTnbljUuHDNcF7ItvT7LqM=; b=IhwSH0rU+MsvfMVIf/a0zdzD57tn9kJlPtqjLNHCz+/SNKG/p0Y2aqoZOGFYgLAtOhZ5 UycK4D44ANNztkOgUhM0/+mQ8xVng3XVyMnu7VBlBIuDNI8Ge0DR5g/PR/lOqAMCBFju szOv/EsuCn6f0c3OhQ+YZkxOKRXPM/YkapSIBS9lmSK8on9BDJ+30AoxYe2uC9Gh3Ovf paU7OtZfsIHHu9GUdPnNMoKPMIhkEFx7DqoutzmvtyH79EC1QhythZizbNK+DgmFMCsW RSwPxgJ7goVlm3KnryYEEtyaAGumLNTJUEyXqfDaEpRf6BH0a6pA8rKWLlae1oVAckie 2g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnted08n2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:21 +0000 Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668nheR030433; Thu, 6 Jul 2023 08:59:21 GMT Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rnted08mg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:20 +0000 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3668lE2x016351; Thu, 6 Jul 2023 08:59:20 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([9.208.130.97]) by ppma02dal.us.ibm.com (PPS) with ESMTPS id 3rjbs6n1f2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:20 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668xJ0024576512 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:59:19 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DD55858058; Thu, 6 Jul 2023 08:59:18 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B291C58057; Thu, 6 Jul 2023 08:59:13 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:59:13 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 09/13] powerpc/book3s64/mm: Enable transparent pud hugepage Date: Thu, 6 Jul 2023 14:28:15 +0530 Message-ID: <20230706085819.827761-10-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: GhE2EhkyLpAXABP6lHSJ30Q9TA17TMoQ X-Proofpoint-ORIG-GUID: AhCBtrosut_OrpwTZtQsFRxPRlVD9WB3 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 impostorscore=0 bulkscore=0 spamscore=0 clxscore=1015 adultscore=0 priorityscore=1501 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 533A04001F X-Stat-Signature: 1737zipdbze5pxbp5u5pt9p3rfcdihgb X-Rspam-User: X-HE-Tag: 1688633973-268805 X-HE-Meta: U2FsdGVkX18RPSHLO9/c5dQarWJitx/sdK0e4eLYMxSGPZk8UI8GHfQm6knW8bFzBfrk3lC3yrhu+3/AmIPVmeZwJr80/OEQwGx4b1mIW7kqqf963t8yCgUEf0JatCnIEYh/HNF33WydKxtgFYncOAFkFsr3JlESg4QbIp1FBkdKJULNbpQMnlfXai/LyO+sDfAp5yf29lMpYzMEOBf7OV7ggRF/Hdo9cWDtndfpOAAS5d2Biuyvo+RNfwCcNJMCGR3nTV4aGtLDaVplS264QtRtYTcnL3HCX+9iaWa6bmzlrHNYVXt4jRbXQD06Lc+IhcP/hTROzCmCupHwebQ6JxsKoo48LfeCwAJFXR40hIJSjyORMowfSh18b4Mb+HWyTZUq9dgNEmj1Z9JHOpyf2q66zL3PfRVTHLME18r6uqryKA9t04HgLerKv7VI9J5QCjNP9oZtndI7Sed9CbiNDiIzPzb9PW73/XJ7JjN2ISnHaT7ujfWNGdSG4ay0tk7P3yvHS6nU9l9mUcpK0p99hIoGpzCb29GMfnfOG0nR5owI3CZ39xt9p5Chzrbhap2Ol3YBN43r3OKAf204GvEIEeQTqXNj68a1CW0JDiV/Yw0+XlPSfA8dhqRI847zgYrfVA7jLXiIw9c0x7RhufbCTlyfbDq+BqbKjvUjSjaX1q+6GPUJ5drQWtONv8cNyCgrGOt3xLRWrI+BpwqER+08IUvM4iza09oJVMUf0PoEXeGxkLSFWVD+ICIhsDHlrv1ruWzkUm6lBHdaUNEee5B8+7O1BSDLDESjFE3Giy4s98ZHAOW3tgKeivQng/emMsU9svny+7mZf2xKBAkrE0nzZlZgmNpQzub6osDMTfC5RdqNfolGGgF7tmOvvC/jYhjgKKG1CxgLWUXgGVq19e44XxbfsRvJ7DoGnPprqD3WDgnLsqltt6hpvCXAgIIDQkxA4FhR84jrixmOqXvxMGL IU3HRVuu DRcuLjAxN94pIeKiAM9CLw7LJitNw1CZuKZgdE5GJG9QrwCxRxUSjYcgUMKcm7mNUePF/Zbj1jQvv5UlDeB4CcK7oQG2muctNHoZ3UO7qmKf1Lr3s4QkhN1KAl/I2EIoRLfUTtoDDjuPzHoiYdhZJL38Gk9tHU8AxMWoJTAxwW05Y2t69Kfdsg61mM2YEOOqZVYlVhXu17oBWaQ2kza6Y0qM9jxtlR9ADdkq5/6rbj6mWr84= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is enabled only with radix translation and 1G hugepage size. This will be used with devdax device memory with a namespace alignment of 1G. Anon transparent hugepage is not supported even though we do have helpers checking pud_trans_huge(). We should never find that return true. The only expected pte bit combination is _PAGE_PTE | _PAGE_DEVMAP. Some of the helpers are never expected to get called on hash translation and hence is marked to call BUG() in such a case. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/book3s/64/pgtable.h | 156 ++++++++++++++++-- arch/powerpc/include/asm/book3s/64/radix.h | 36 ++++ .../include/asm/book3s/64/tlbflush-radix.h | 2 + arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 + arch/powerpc/mm/book3s64/pgtable.c | 78 +++++++++ arch/powerpc/mm/book3s64/radix_pgtable.c | 28 ++++ arch/powerpc/mm/book3s64/radix_tlb.c | 7 + arch/powerpc/platforms/Kconfig.cputype | 1 + include/trace/events/thp.h | 10 ++ 9 files changed, 315 insertions(+), 11 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 4acc9690f599..9a05de007956 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -921,8 +921,29 @@ static inline pud_t pte_pud(pte_t pte) { return __pud_raw(pte_raw(pte)); } + +static inline pte_t *pudp_ptep(pud_t *pud) +{ + return (pte_t *)pud; +} + +#define pud_pfn(pud) pte_pfn(pud_pte(pud)) +#define pud_dirty(pud) pte_dirty(pud_pte(pud)) +#define pud_young(pud) pte_young(pud_pte(pud)) +#define pud_mkold(pud) pte_pud(pte_mkold(pud_pte(pud))) +#define pud_wrprotect(pud) pte_pud(pte_wrprotect(pud_pte(pud))) +#define pud_mkdirty(pud) pte_pud(pte_mkdirty(pud_pte(pud))) +#define pud_mkclean(pud) pte_pud(pte_mkclean(pud_pte(pud))) +#define pud_mkyoung(pud) pte_pud(pte_mkyoung(pud_pte(pud))) +#define pud_mkwrite(pud) pte_pud(pte_mkwrite(pud_pte(pud))) #define pud_write(pud) pte_write(pud_pte(pud)) +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY +#define pud_soft_dirty(pmd) pte_soft_dirty(pud_pte(pud)) +#define pud_mksoft_dirty(pmd) pte_pud(pte_mksoft_dirty(pud_pte(pud))) +#define pud_clear_soft_dirty(pmd) pte_pud(pte_clear_soft_dirty(pud_pte(pud))) +#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ + static inline int pud_bad(pud_t pud) { if (radix_enabled()) @@ -1115,15 +1136,24 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write) #ifdef CONFIG_TRANSPARENT_HUGEPAGE extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); +extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot); extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); +extern void set_pud_at(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, pud_t pud); + static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) { } +static inline void update_mmu_cache_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pud) +{ +} + extern int hash__has_transparent_hugepage(void); static inline int has_transparent_hugepage(void) { @@ -1133,6 +1163,14 @@ static inline int has_transparent_hugepage(void) } #define has_transparent_hugepage has_transparent_hugepage +static inline int has_transparent_pud_hugepage(void) +{ + if (radix_enabled()) + return radix__has_transparent_pud_hugepage(); + return 0; +} +#define has_transparent_pud_hugepage has_transparent_pud_hugepage + static inline unsigned long pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, unsigned long clr, unsigned long set) @@ -1142,6 +1180,16 @@ pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, return hash__pmd_hugepage_update(mm, addr, pmdp, clr, set); } +static inline unsigned long +pud_hugepage_update(struct mm_struct *mm, unsigned long addr, pud_t *pudp, + unsigned long clr, unsigned long set) +{ + if (radix_enabled()) + return radix__pud_hugepage_update(mm, addr, pudp, clr, set); + BUG(); + return pud_val(*pudp); +} + /* * returns true for pmd migration entries, THP, devmap, hugetlb * But compile time dependent on THP config @@ -1151,6 +1199,11 @@ static inline int pmd_large(pmd_t pmd) return !!(pmd_raw(pmd) & cpu_to_be64(_PAGE_PTE)); } +static inline int pud_large(pud_t pud) +{ + return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE)); +} + /* * For radix we should always find H_PAGE_HASHPTE zero. Hence * the below will work for radix too @@ -1166,6 +1219,17 @@ static inline int __pmdp_test_and_clear_young(struct mm_struct *mm, return ((old & _PAGE_ACCESSED) != 0); } +static inline int __pudp_test_and_clear_young(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + unsigned long old; + + if ((pud_raw(*pudp) & cpu_to_be64(_PAGE_ACCESSED | H_PAGE_HASHPTE)) == 0) + return 0; + old = pud_hugepage_update(mm, addr, pudp, _PAGE_ACCESSED, 0); + return ((old & _PAGE_ACCESSED) != 0); +} + #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) @@ -1174,6 +1238,14 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_hugepage_update(mm, addr, pmdp, _PAGE_WRITE, 0); } +#define __HAVE_ARCH_PUDP_SET_WRPROTECT +static inline void pudp_set_wrprotect(struct mm_struct *mm, unsigned long addr, + pud_t *pudp) +{ + if (pud_write(*pudp)) + pud_hugepage_update(mm, addr, pudp, _PAGE_WRITE, 0); +} + /* * Only returns true for a THP. False for pmd migration entry. * We also need to return true when we come across a pte that @@ -1195,6 +1267,17 @@ static inline int pmd_trans_huge(pmd_t pmd) return hash__pmd_trans_huge(pmd); } +static inline int pud_trans_huge(pud_t pud) +{ + if (!pud_present(pud)) + return false; + + if (radix_enabled()) + return radix__pud_trans_huge(pud); + return 0; +} + + #define __HAVE_ARCH_PMD_SAME static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) { @@ -1203,6 +1286,16 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) return hash__pmd_same(pmd_a, pmd_b); } +#define __HAVE_ARCH_PUD_SAME +static inline int pud_same(pud_t pud_a, pud_t pud_b) +{ + if (radix_enabled()) + return radix__pud_same(pud_a, pud_b); + BUG(); + return 0; +} + + static inline pmd_t __pmd_mkhuge(pmd_t pmd) { if (radix_enabled()) @@ -1210,6 +1303,14 @@ static inline pmd_t __pmd_mkhuge(pmd_t pmd) return hash__pmd_mkhuge(pmd); } +static inline pud_t __pud_mkhuge(pud_t pud) +{ + if (radix_enabled()) + return radix__pud_mkhuge(pud); + BUG(); + return pud; +} + /* * pfn_pmd return a pmd_t that can be used as pmd pte entry. */ @@ -1225,14 +1326,34 @@ static inline pmd_t pmd_mkhuge(pmd_t pmd) return pmd; } +static inline pud_t pud_mkhuge(pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM + if (radix_enabled()) + WARN_ON((pud_raw(pud) & cpu_to_be64(_PAGE_PTE)) == 0); + else + WARN_ON(1); +#endif + return pud; +} + + #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS extern int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty); +#define __HAVE_ARCH_PUDP_SET_ACCESS_FLAGS +extern int pudp_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, + pud_t entry, int dirty); #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG extern int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +#define __HAVE_ARCH_PUDP_TEST_AND_CLEAR_YOUNG +extern int pudp_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp); + #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, @@ -1243,6 +1364,16 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, return hash__pmdp_huge_get_and_clear(mm, addr, pmdp); } +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR +static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + if (radix_enabled()) + return radix__pudp_huge_get_and_clear(mm, addr, pudp); + BUG(); + return *pudp; +} + static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { @@ -1257,6 +1388,11 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full); +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL +pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma, + unsigned long addr, + pud_t *pudp, int full); + #define __HAVE_ARCH_PGTABLE_DEPOSIT static inline void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, pgtable_t pgtable) @@ -1305,6 +1441,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return hash__pmd_mkdevmap(pmd); } +static inline pud_t pud_mkdevmap(pud_t pud) +{ + if (radix_enabled()) + return radix__pud_mkdevmap(pud); + BUG(); + return pud; +} + static inline int pmd_devmap(pmd_t pmd) { return pte_devmap(pmd_pte(pmd)); @@ -1312,7 +1456,7 @@ static inline int pmd_devmap(pmd_t pmd) static inline int pud_devmap(pud_t pud) { - return 0; + return pte_devmap(pud_pte(pud)); } static inline int pgd_devmap(pgd_t pgd) @@ -1321,16 +1465,6 @@ static inline int pgd_devmap(pgd_t pgd) } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline int pud_pfn(pud_t pud) -{ - /* - * Currently all calls to pud_pfn() are gated around a pud_devmap() - * check so this should never be used. If it grows another user we - * want to know about it. - */ - BUILD_BUG(); - return 0; -} #define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION pte_t ptep_modify_prot_start(struct vm_area_struct *, unsigned long, pte_t *); void ptep_modify_prot_commit(struct vm_area_struct *, unsigned long, diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 686001eda936..2ef92f36340f 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -250,6 +250,10 @@ static inline int radix__pud_bad(pud_t pud) return !!(pud_val(pud) & RADIX_PUD_BAD_BITS); } +static inline int radix__pud_same(pud_t pud_a, pud_t pud_b) +{ + return ((pud_raw(pud_a) ^ pud_raw(pud_b)) == 0); +} static inline int radix__p4d_bad(p4d_t p4d) { @@ -268,9 +272,22 @@ static inline pmd_t radix__pmd_mkhuge(pmd_t pmd) return __pmd(pmd_val(pmd) | _PAGE_PTE); } +static inline int radix__pud_trans_huge(pud_t pud) +{ + return (pud_val(pud) & (_PAGE_PTE | _PAGE_DEVMAP)) == _PAGE_PTE; +} + +static inline pud_t radix__pud_mkhuge(pud_t pud) +{ + return __pud(pud_val(pud) | _PAGE_PTE); +} + extern unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, unsigned long clr, unsigned long set); +extern unsigned long radix__pud_hugepage_update(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, unsigned long clr, + unsigned long set); extern pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, @@ -278,6 +295,9 @@ extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, extern pgtable_t radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp); extern pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp); +pud_t radix__pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp); + static inline int radix__has_transparent_hugepage(void) { /* For radix 2M at PMD level means thp */ @@ -285,6 +305,14 @@ static inline int radix__has_transparent_hugepage(void) return 1; return 0; } + +static inline int radix__has_transparent_pud_hugepage(void) +{ + /* For radix 1G at PUD level means pud hugepage support */ + if (mmu_psize_defs[MMU_PAGE_1G].shift == PUD_SHIFT) + return 1; + return 0; +} #endif static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd) @@ -292,9 +320,17 @@ static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd) return __pmd(pmd_val(pmd) | (_PAGE_PTE | _PAGE_DEVMAP)); } +static inline pud_t radix__pud_mkdevmap(pud_t pud) +{ + return __pud(pud_val(pud) | (_PAGE_PTE | _PAGE_DEVMAP)); +} + +struct vmem_altmap; extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long page_size, unsigned long phys); +int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap); extern void radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size); diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 77797a2a82eb..a38542259fab 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -68,6 +68,8 @@ void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize); extern void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +extern void radix__flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end); extern void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); extern void radix__flush_tlb_kernel_range(unsigned long start, unsigned long end); diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 0d0c1447ecf0..a01c20a8fbf7 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -50,6 +50,14 @@ static inline void flush_pmd_tlb_range(struct vm_area_struct *vma, radix__flush_pmd_tlb_range(vma, start, end); } +#define __HAVE_ARCH_FLUSH_PUD_TLB_RANGE +static inline void flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + if (radix_enabled()) + radix__flush_pud_tlb_range(vma, start, end); +} + #define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long start, diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 85c84e89e3ea..75b938268b04 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -64,11 +64,39 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, return changed; } +int pudp_set_access_flags(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp, pud_t entry, int dirty) +{ + int changed; +#ifdef CONFIG_DEBUG_VM + WARN_ON(!pud_devmap(*pudp)); + assert_spin_locked(pud_lockptr(vma->vm_mm, pudp)); +#endif + changed = !pud_same(*(pudp), entry); + if (changed) { + /* + * We can use MMU_PAGE_1G here, because only radix + * path look at the psize. + */ + __ptep_set_access_flags(vma, pudp_ptep(pudp), + pud_pte(entry), address, MMU_PAGE_1G); + } + return changed; +} + + int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { return __pmdp_test_and_clear_young(vma->vm_mm, address, pmdp); } + +int pudp_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp) +{ + return __pudp_test_and_clear_young(vma->vm_mm, address, pudp); +} + /* * set a new huge pmd. We should not be called for updating * an existing pmd entry. That should go via pmd_hugepage_update. @@ -90,6 +118,23 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr, return set_pte_at(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd)); } +void set_pud_at(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM + /* + * Make sure hardware valid bit is not set. We don't do + * tlb flush for this update. + */ + + WARN_ON(pte_hw_valid(pud_pte(*pudp))); + assert_spin_locked(pud_lockptr(mm, pudp)); + WARN_ON(!(pud_large(pud))); +#endif + trace_hugepage_set_pud(addr, pud_val(pud)); + return set_pte_at(mm, addr, pudp_ptep(pudp), pud_pte(pud)); +} + static void do_serialize(void *arg) { /* We've taken the IPI, so try to trim the mask while here */ @@ -147,11 +192,35 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, return pmd; } +pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, int full) +{ + pud_t pud; + + VM_BUG_ON(addr & ~HPAGE_PMD_MASK); + VM_BUG_ON((pud_present(*pudp) && !pud_devmap(*pudp)) || + !pud_present(*pudp)); + pud = pudp_huge_get_and_clear(vma->vm_mm, addr, pudp); + /* + * if it not a fullmm flush, then we can possibly end up converting + * this PMD pte entry to a regular level 0 PTE by a parallel page fault. + * Make sure we flush the tlb in this case. + */ + if (!full) + flush_pud_tlb_range(vma, addr, addr + HPAGE_PUD_SIZE); + return pud; +} + static pmd_t pmd_set_protbits(pmd_t pmd, pgprot_t pgprot) { return __pmd(pmd_val(pmd) | pgprot_val(pgprot)); } +static pud_t pud_set_protbits(pud_t pud, pgprot_t pgprot) +{ + return __pud(pud_val(pud) | pgprot_val(pgprot)); +} + /* * At some point we should be able to get rid of * pmd_mkhuge() and mk_huge_pmd() when we update all the @@ -166,6 +235,15 @@ pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot) return __pmd_mkhuge(pmd_set_protbits(__pmd(pmdv), pgprot)); } +pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot) +{ + unsigned long pudv; + + pudv = (pfn << PAGE_SHIFT) & PTE_RPN_MASK; + + return __pud_mkhuge(pud_set_protbits(__pud(pudv), pgprot)); +} + pmd_t mk_pmd(struct page *page, pgprot_t pgprot) { return pfn_pmd(page_to_pfn(page), pgprot); diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 02e185d2e4d6..227fea53c217 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -967,6 +967,23 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add return old; } +unsigned long radix__pud_hugepage_update(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, unsigned long clr, + unsigned long set) +{ + unsigned long old; + +#ifdef CONFIG_DEBUG_VM + WARN_ON(!pud_devmap(*pudp)); + assert_spin_locked(pud_lockptr(mm, pudp)); +#endif + + old = radix__pte_update(mm, addr, pudp_ptep(pudp), clr, set, 1); + trace_hugepage_update_pud(addr, old, clr, set); + + return old; +} + pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) @@ -1043,6 +1060,17 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm, return old_pmd; } +pud_t radix__pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + pud_t old_pud; + unsigned long old; + + old = radix__pud_hugepage_update(mm, addr, pudp, ~0UL, 0); + old_pud = __pud(old); + return old_pud; +} + #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 0bd4866d9824..79049a0194f1 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -1461,6 +1461,13 @@ void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, } EXPORT_SYMBOL(radix__flush_pmd_tlb_range); +void radix__flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + radix__flush_tlb_range_psize(vma->vm_mm, start, end, MMU_PAGE_1G); +} +EXPORT_SYMBOL(radix__flush_pud_tlb_range); + void radix__flush_tlb_all(void) { unsigned long rb,prs,r,rs; diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index 45fd975ef521..340b86ef7284 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -94,6 +94,7 @@ config PPC_BOOK3S_64 select PPC_FPU select PPC_HAVE_PMU_SUPPORT select HAVE_ARCH_TRANSPARENT_HUGEPAGE + select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE diff --git a/include/trace/events/thp.h b/include/trace/events/thp.h index a95c78b10561..f50048af5fcc 100644 --- a/include/trace/events/thp.h +++ b/include/trace/events/thp.h @@ -30,6 +30,11 @@ DEFINE_EVENT(hugepage_set, hugepage_set_pmd, TP_ARGS(addr, pmd) ); +DEFINE_EVENT(hugepage_set, hugepage_set_pud, + TP_PROTO(unsigned long addr, unsigned long pud), + TP_ARGS(addr, pud) +); + DECLARE_EVENT_CLASS(hugepage_update, TP_PROTO(unsigned long addr, unsigned long pte, unsigned long clr, unsigned long set), @@ -57,6 +62,11 @@ DEFINE_EVENT(hugepage_update, hugepage_update_pmd, TP_ARGS(addr, pmd, clr, set) ); +DEFINE_EVENT(hugepage_update, hugepage_update_pud, + TP_PROTO(unsigned long addr, unsigned long pud, unsigned long clr, unsigned long set), + TP_ARGS(addr, pud, clr, set) +); + DECLARE_EVENT_CLASS(migration_pmd, TP_PROTO(unsigned long addr, unsigned long pmd), From patchwork Thu Jul 6 08:58:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 378BEEB64D9 for ; Thu, 6 Jul 2023 08:59:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD24C8D0008; Thu, 6 Jul 2023 04:59:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C82B08D0001; Thu, 6 Jul 2023 04:59:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFBDB8D0008; Thu, 6 Jul 2023 04:59:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A1ACF8D0001 for ; Thu, 6 Jul 2023 04:59:41 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5F0FA1404AA for ; Thu, 6 Jul 2023 08:59:41 +0000 (UTC) X-FDA: 80980588962.06.C7A4B81 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf25.hostedemail.com (Postfix) with ESMTP id DAEB6A0009 for ; Thu, 6 Jul 2023 08:59:38 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=o8AewuJU; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf25.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633979; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hSmlF6vupYsKEsZ0bt9c3KNWgGbKQoXbquIrC74hYMI=; b=Y1NljoAOGsenyOE4bOFtwVVC66dA/sAt0aqveBI8rDcpesZhXhNcDmImaB8wrjpMNtuJMv 9uh8wUQaCP6tPp5LMHFrJqcRZsWup5LiUCyO6IoL1J2wdKGD/WfZiKvtzr/Dgvhd7FuNMz KifPFOKKq/KRpegtDZnP6+B64/yyIL0= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=o8AewuJU; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf25.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633979; a=rsa-sha256; cv=none; b=kn+bhzCV0SOIY27UWn8FERbWQiWL+AjVzJTQKF90TIo57y7XcLfTwUiqF2fowXpJRcJY8y 6tDil4l1C7/cjfXuclECE+1fYvebfh4rJ87MzptWmDRzAxfc6ULgIx+/5kJ87fa2Sl4AUo Rjy6FW0RBzf/Cci6mRlpF8F0OqBgQ3A= Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668kf4X018238; Thu, 6 Jul 2023 08:59:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=hSmlF6vupYsKEsZ0bt9c3KNWgGbKQoXbquIrC74hYMI=; b=o8AewuJUftoC64oiaStbE0wo4wEgSGLnEEmfs5K7LKx0sKD0isloeADEjADhVH4q6M3X fBLirLz9ja7tEwwtYN5jtcK1UNCPZnf2KTmK1dSZymSQwF302VMqei/efMb9iQ58XgGC Vgl28Gfuf/4q+VmWIHyEqIqi2YaRKhYHEs7klg7uqj1brvx4R6rJdfhq0KGKbc/mEEUk N8LoQYPYGdUj2IDLMB6KMxI/18QknVI4Wi43ysOm7NVOhOfLe4fX7M4oOnlLfBblUPw/ o6jZBC+Stfcv6JCdStcbAG5SnBPhXwVdVIIPAaqX1uQ5/jFjuBrrUKCUYnxEVb+SxvoX Tg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntd80bjp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:27 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668x7gA008936; Thu, 6 Jul 2023 08:59:27 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntd80bj2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:26 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36689xLx002667; Thu, 6 Jul 2023 08:59:25 GMT Received: from smtprelay02.wdc07v.mail.ibm.com ([9.208.129.120]) by ppma05wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs5yxwk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:25 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay02.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668xOjQ2425372 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:59:24 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 06AE758058; Thu, 6 Jul 2023 08:59:24 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7E64658057; Thu, 6 Jul 2023 08:59:19 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:59:19 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 10/13] powerpc/book3s64/vmemmap: Switch radix to use a different vmemmap handling function Date: Thu, 6 Jul 2023 14:28:16 +0530 Message-ID: <20230706085819.827761-11-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: AsKSnEkhA1RelhVgCZIg-auj5MHvpfMC X-Proofpoint-ORIG-GUID: JBLCflK0mZv0uz0ljYkB5xjimEJ7Djni X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 clxscore=1015 malwarescore=0 suspectscore=0 adultscore=0 spamscore=0 bulkscore=0 impostorscore=0 mlxscore=0 priorityscore=1501 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Queue-Id: DAEB6A0009 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: sr894ywdfbw9hwrtcsca19w18i9sjgu5 X-HE-Tag: 1688633978-307440 X-HE-Meta: U2FsdGVkX1/KHvWORCcRUGxoXbeA7Hc0Uq2PkkWs/RwpLBGM1+4Jdr9d+K+jPtbQby9P7ltDTQk6nmc8AlS8L7gBFmXuMjefVFkueP/2+Z04TYXBBe2KsGAa8LjhOc7dVjythQZr05hYKTMPOLisR2Ys5K64qA+KN7hqoVUa+Krot+pHPccvJwlesLuY9DSUnqpR/i7ba3P3NQpBTCApTGK4rCdOVpvqqAB/de4V8Lj0Ch9Nq+rqsE+ZvQ3SjrxXXg7LO8fNjEkKMdGSrMyb9+jX4r1bfq0AWyhpfCifg9L5ijJyvkgp7i56bON6XFs9w/ICwrhnnJvPvwcD0Kel1j8lEgmj7FgR6qwSg1hLme4Bjjef9WZ6070AXKEE130vfVG03DHFtnQVFRTEPTYSZ2lRk8QnxNaFK5EnT9Z3l7m2Tf+LRbdsz0R20B5ynkfc2ygZzUS4Hnvye2796AN0qRg7iIu+SH1GQVTmV5xbWaXjABheSqZIq7knC3wxw5c828lxT8+Sa84udnSjAa3hPnwx1uNbbqOGqR+OiQIEq1KTpjBiSOaEau6hJk4h4rsIMQTQAWT/90kPsyAhoib0JeHiyhxlUPI/KoGjMJaQL+l8E+BZeGGzABN7BP6qcpRgJIHBZxBvE6SlLI2tX4TB2PEEwAZKMjaZ3xoMbEW8F7yaIjniDuB902/XwTiHUK1TZrCk9/VCiLgoCzSfL+GCG8+dRCiM7FHzBffehAlTGEa01LuvWAqfDgTmp6JfZrwspR02NtCPRQNppG6JrYsQFAoD224pqv8zNMYWzTV/65OT2Hj65yGQZfXZN5kAZ1T17uDtceTHnNWB6R2JdTzS6HowTcc48XSGrOUe9REVGLuqDfKZCTUn53J8C0h3w1wjT0VMiJGesi6kYQ4q6vbePAeDtnOtKi0HZMy4DtzVqXmV+OBgpEauArEtlBTmk8fDOV4o41RxEbASvrAHhI9 8Y293Opr D59KOel0CLrIK9cn4er39LoK5mrQ6xQlM5q8Hu3/oiwOUYlclhN0r4tb7sAwvmWNbThZAKQE6B3g9kzvwPEcBcemj1ljPScsHaHyBd4zsoJQqbnzuJ/rmbkjAnnvYBN1JlG7fkiwOAvNWXSzC7JbrfTbMW5gVy8I8JhdnMe5McNdI5/JlZpkjHiBCQfcJUaI2FBNtvsPX6TV1KrR1BW6k77dFcjOynqW8UemoTxpk/+UNeiE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is in preparation to update radix to implement vmemmap optimization for devdax. Below are the rules w.r.t radix vmemmap mapping 1. First try to map things using PMD (2M) 2. With altmap if altmap cross-boundary check returns true, fall back to PAGE_SIZE 3. If we can't allocate PMD_SIZE backing memory for vmemmap, fallback to PAGE_SIZE On removing vmemmap mapping, check if every subsection that is using the vmemmap area is invalid. If found to be invalid, that implies we can safely free the vmemmap area. We don't use the PAGE_UNUSED pattern used by x86 because with 64K page size, we need to do the above check even at the PAGE_SIZE granularity. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/book3s/64/radix.h | 2 + arch/powerpc/include/asm/pgtable.h | 4 + arch/powerpc/mm/book3s64/radix_pgtable.c | 318 +++++++++++++++++++-- arch/powerpc/mm/init_64.c | 26 +- 4 files changed, 319 insertions(+), 31 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 2ef92f36340f..f1461289643a 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -331,6 +331,8 @@ extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long phys); int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); +void __ref radix__vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap); extern void radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 6a88bfdaa69b..68817ea7f994 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -165,6 +165,10 @@ static inline bool is_ioremap_addr(const void *x) return addr >= IOREMAP_BASE && addr < IOREMAP_END; } + +int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size); +bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, + unsigned long page_size); #endif /* CONFIG_PPC64 */ #endif /* __ASSEMBLY__ */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 227fea53c217..8a03e1005fd3 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -744,8 +744,57 @@ static void free_pud_table(pud_t *pud_start, p4d_t *p4d) p4d_clear(p4d); } +static bool __meminit vmemmap_pmd_is_unused(unsigned long addr, unsigned long end) +{ + unsigned long start = ALIGN_DOWN(addr, PMD_SIZE); + + return !vmemmap_populated(start, PMD_SIZE); +} + +static bool __meminit vmemmap_page_is_unused(unsigned long addr, unsigned long end) +{ + unsigned long start = ALIGN_DOWN(addr, PAGE_SIZE); + + return !vmemmap_populated(start, PAGE_SIZE); + +} + +static void __meminit free_vmemmap_pages(struct page *page, + struct vmem_altmap *altmap, + int order) +{ + unsigned int nr_pages = 1 << order; + + if (altmap) { + unsigned long alt_start, alt_end; + unsigned long base_pfn = page_to_pfn(page); + + /* + * with 1G vmemmap mmaping we can have things setup + * such that even though atlmap is specified we never + * used altmap. + */ + alt_start = altmap->base_pfn; + alt_end = altmap->base_pfn + altmap->reserve + + altmap->free + altmap->alloc + altmap->align; + + if (base_pfn >= alt_start && base_pfn < alt_end) { + vmem_altmap_free(altmap, nr_pages); + return; + } + } + + if (PageReserved(page)) { + /* allocated from memblock */ + while (nr_pages--) + free_reserved_page(page++); + } else + free_pages((unsigned long)page_address(page), order); +} + static void remove_pte_table(pte_t *pte_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pte_t *pte; @@ -759,24 +808,23 @@ static void remove_pte_table(pte_t *pte_start, unsigned long addr, if (!pte_present(*pte)) continue; - if (!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(next)) { - /* - * The vmemmap_free() and remove_section_mapping() - * codepaths call us with aligned addresses. - */ - WARN_ONCE(1, "%s: unaligned range\n", __func__); - continue; + if (PAGE_ALIGNED(addr) && PAGE_ALIGNED(next)) { + if (!direct) + free_vmemmap_pages(pte_page(*pte), altmap, 0); + pte_clear(&init_mm, addr, pte); + pages++; + } else if (!direct && vmemmap_page_is_unused(addr, next)) { + free_vmemmap_pages(pte_page(*pte), altmap, 0); + pte_clear(&init_mm, addr, pte); } - - pte_clear(&init_mm, addr, pte); - pages++; } if (direct) update_page_count(mmu_virtual_psize, -pages); } static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pte_t *pte_base; @@ -790,18 +838,21 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, continue; if (pmd_is_leaf(*pmd)) { - if (!IS_ALIGNED(addr, PMD_SIZE) || - !IS_ALIGNED(next, PMD_SIZE)) { - WARN_ONCE(1, "%s: unaligned range\n", __func__); - continue; + if (IS_ALIGNED(addr, PMD_SIZE) && + IS_ALIGNED(next, PMD_SIZE)) { + if (!direct) + free_vmemmap_pages(pmd_page(*pmd), altmap, get_order(PMD_SIZE)); + pte_clear(&init_mm, addr, (pte_t *)pmd); + pages++; + } else if (vmemmap_pmd_is_unused(addr, next)) { + free_vmemmap_pages(pmd_page(*pmd), altmap, get_order(PMD_SIZE)); + pte_clear(&init_mm, addr, (pte_t *)pmd); } - pte_clear(&init_mm, addr, (pte_t *)pmd); - pages++; continue; } pte_base = (pte_t *)pmd_page_vaddr(*pmd); - remove_pte_table(pte_base, addr, next, direct); + remove_pte_table(pte_base, addr, next, direct, altmap); free_pte_table(pte_base, pmd); } if (direct) @@ -809,7 +860,8 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, } static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pmd_t *pmd_base; @@ -834,15 +886,16 @@ static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr, } pmd_base = pud_pgtable(*pud); - remove_pmd_table(pmd_base, addr, next, direct); + remove_pmd_table(pmd_base, addr, next, direct, altmap); free_pmd_table(pmd_base, pud); } if (direct) update_page_count(MMU_PAGE_1G, -pages); } -static void __meminit remove_pagetable(unsigned long start, unsigned long end, - bool direct) +static void __meminit +remove_pagetable(unsigned long start, unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long addr, next; pud_t *pud_base; @@ -871,7 +924,7 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end, } pud_base = p4d_pgtable(*p4d); - remove_pud_table(pud_base, addr, next, direct); + remove_pud_table(pud_base, addr, next, direct, altmap); free_pud_table(pud_base, p4d); } @@ -894,7 +947,7 @@ int __meminit radix__create_section_mapping(unsigned long start, int __meminit radix__remove_section_mapping(unsigned long start, unsigned long end) { - remove_pagetable(start, end, true); + remove_pagetable(start, end, true, NULL); return 0; } #endif /* CONFIG_MEMORY_HOTPLUG */ @@ -926,10 +979,223 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start, return 0; } +int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, + unsigned long addr, unsigned long next) +{ + int large = pmd_large(*pmdp); + + if (large) + vmemmap_verify(pmdp_ptep(pmdp), node, addr, next); + + return large; +} + +void __meminit vmemmap_set_pmd(pmd_t *pmdp, void *p, int node, + unsigned long addr, unsigned long next) +{ + pte_t entry; + pte_t *ptep = pmdp_ptep(pmdp); + + VM_BUG_ON(!IS_ALIGNED(addr, PMD_SIZE)); + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + set_pte_at(&init_mm, addr, ptep, entry); + asm volatile("ptesync": : :"memory"); + + vmemmap_verify(ptep, node, addr, next); +} + +static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long addr, + int node, + struct vmem_altmap *altmap, + struct page *reuse) +{ + pte_t *pte = pte_offset_kernel(pmdp, addr); + + if (pte_none(*pte)) { + pte_t entry; + void *p; + + if (!reuse) { + /* + * make sure we don't create altmap mappings + * covering things outside the device. + */ + if (altmap && altmap_cross_boundary(altmap, addr, PAGE_SIZE)) + altmap = NULL; + + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p && altmap) + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, NULL); + if (!p) + return NULL; + } else { + /* + * When a PTE/PMD entry is freed from the init_mm + * there's a free_pages() call to this page allocated + * above. Thus this get_page() is paired with the + * put_page_testzero() on the freeing path. + * This can only called by certain ZONE_DEVICE path, + * and through vmemmap_populate_compound_pages() when + * slab is available. + */ + get_page(reuse); + p = page_to_virt(reuse); + } + + VM_BUG_ON(!PAGE_ALIGNED(addr)); + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + set_pte_at(&init_mm, addr, pte, entry); + asm volatile("ptesync": : :"memory"); + } + return pte; +} + +static inline pud_t *vmemmap_pud_alloc(p4d_t *p4dp, int node, + unsigned long address) +{ + pud_t *pud; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(p4d_none(*p4dp))) { + if (unlikely(!slab_is_available())) { + pud = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + p4d_populate(&init_mm, p4dp, pud); + /* go to the pud_offset */ + } else + return pud_alloc(&init_mm, p4dp, address); + } + return pud_offset(p4dp, address); +} + +static inline pmd_t *vmemmap_pmd_alloc(pud_t *pudp, int node, + unsigned long address) +{ + pmd_t *pmd; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(pud_none(*pudp))) { + if (unlikely(!slab_is_available())) { + pmd = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + pud_populate(&init_mm, pudp, pmd); + } else + return pmd_alloc(&init_mm, pudp, address); + } + return pmd_offset(pudp, address); +} + +static inline pte_t *vmemmap_pte_alloc(pmd_t *pmdp, int node, + unsigned long address) +{ + pte_t *pte; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(pmd_none(*pmdp))) { + if (unlikely(!slab_is_available())) { + pte = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + pmd_populate(&init_mm, pmdp, pte); + } else + return pte_alloc_kernel(pmdp, address); + } + return pte_offset_kernel(pmdp, address); +} + + + +int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) +{ + unsigned long addr; + unsigned long next; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + for (addr = start; addr < end; addr = next) { + next = pmd_addr_end(addr, end); + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return -ENOMEM; + + if (pmd_none(READ_ONCE(*pmd))) { + void *p; + + /* + * keep it simple by checking addr PMD_SIZE alignment + * and verifying the device boundary condition. + * For us to use a pmd mapping, both addr and pfn should + * be aligned. We skip if addr is not aligned and for + * pfn we hope we have extra area in the altmap that + * can help to find an aligned block. This can result + * in altmap block allocation failures, in which case + * we fallback to RAM for vmemmap allocation. + */ + if (altmap && (!IS_ALIGNED(addr, PMD_SIZE) || + altmap_cross_boundary(altmap, addr, PMD_SIZE))) { + /* + * make sure we don't create altmap mappings + * covering things outside the device. + */ + goto base_mapping; + } + + p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); + if (p) { + vmemmap_set_pmd(pmd, p, node, addr, next); + continue; + } else if (altmap) { + /* + * A vmemmap block allocation can fail due to + * alignment requirements and we trying to align + * things aggressively there by running out of + * space. Try base mapping on failure. + */ + goto base_mapping; + } + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + /* + * If a huge mapping exist due to early call to + * vmemmap_populate, let's try to use that. + */ + continue; + } +base_mapping: + /* + * Not able allocate higher order memory to back memmap + * or we found a pointer to pte page. Allocate base page + * size vmemmap + */ + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return -ENOMEM; + + pte = radix__vmemmap_pte_populate(pmd, addr, node, altmap, NULL); + if (!pte) + return -ENOMEM; + + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + next = addr + PAGE_SIZE; + } + return 0; +} + #ifdef CONFIG_MEMORY_HOTPLUG void __meminit radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size) { - remove_pagetable(start, start + page_size, false); + remove_pagetable(start, start + page_size, true, NULL); +} + +void __ref radix__vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) +{ + remove_pagetable(start, end, false, altmap); } #endif #endif diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index fe1b83020e0d..5701faca39ef 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -92,7 +92,7 @@ static struct page * __meminit vmemmap_subsection_start(unsigned long vmemmap_ad * a page table lookup here because with the hash translation we don't keep * vmemmap details in linux page table. */ -static int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size) +int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size) { struct page *start; unsigned long vmemmap_end = vmemmap_addr + vmemmap_map_size; @@ -183,8 +183,8 @@ static __meminit int vmemmap_list_populate(unsigned long phys, return 0; } -static bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, - unsigned long page_size) +bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, + unsigned long page_size) { unsigned long nr_pfn = page_size / sizeof(struct page); unsigned long start_pfn = page_to_pfn((struct page *)start); @@ -204,6 +204,11 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, bool altmap_alloc; unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix__vmemmap_populate(start, end, node, altmap); +#endif + /* Align to the page size of the linear mapping. */ start = ALIGN_DOWN(start, page_size); @@ -303,8 +308,8 @@ static unsigned long vmemmap_list_free(unsigned long start) return vmem_back->phys; } -void __ref vmemmap_free(unsigned long start, unsigned long end, - struct vmem_altmap *altmap) +void __ref __vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) { unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; unsigned long page_order = get_order(page_size); @@ -362,6 +367,17 @@ void __ref vmemmap_free(unsigned long start, unsigned long end, vmemmap_remove_mapping(start, page_size); } } + +void __ref vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) +{ +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix__vmemmap_free(start, end, altmap); +#endif + return __vmemmap_free(start, end, altmap); +} + #endif void register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page, unsigned long size) From patchwork Thu Jul 6 08:58:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96ED1EB64D9 for ; Thu, 6 Jul 2023 08:59:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E71B8E0001; Thu, 6 Jul 2023 04:59:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 296FE8D0001; Thu, 6 Jul 2023 04:59:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15E778E0001; Thu, 6 Jul 2023 04:59:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 05BC28D0001 for ; Thu, 6 Jul 2023 04:59:47 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CC6EF1A0C36 for ; Thu, 6 Jul 2023 08:59:46 +0000 (UTC) X-FDA: 80980589172.03.D3B57ED Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf18.hostedemail.com (Postfix) with ESMTP id 48D1A1C0016 for ; Thu, 6 Jul 2023 08:59:43 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=hCB+dQcg; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf18.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633984; a=rsa-sha256; cv=none; b=ujUKBP/kU8Y4fbQQZiIku+Q0O+rNEoGXEf+T2LU0dHI3Z+10l74jqN6oFGydbDCQ8IQlGY cXlV7D7C3I7E0jZ/HGSIFaLsNOKp+7o9X1yEjZprJ98JZCMXaKqxtp0Z96aD0ddrUnxk3m 07mjpYdC4egoN8D06l9tuKZgLNkqssE= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=hCB+dQcg; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf18.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633984; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iK9sMhfkY7kQSSDbBWV/XXSs1eGbiyErB2B62ou+yZA=; b=OMCiSbelZWfrFmxrUBSj2L3QND6+ynL7FQjsfORIH4lo8Dm0IHzIJQrpv6oSmHywNC2qKL o/MDQ6liCF6RbK410+0x97qCWjbQsne6fjAfAzScS5Q3pX+KpghZ0ugFihKV+ZX8kDq2zO X1A0qQTS88kkc6AdwwHshCZWQ+qIlrw= Received: from pps.filterd (m0353723.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668qXJt013779; Thu, 6 Jul 2023 08:59:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=iK9sMhfkY7kQSSDbBWV/XXSs1eGbiyErB2B62ou+yZA=; b=hCB+dQcg23/LMaph73+CViduix2kpjKVAmfJyffA5t5cLIwpqOBn4+NSPCfKIz5TykIS KMSHiPav1ZSHRPd1zKIm93GYfDpp/gLxJD/brcti425eMW0IQcygdxi7S+5gsc6UeuuX 0dZ7BqGWqKEQXV+PwarK2k69zmpLJ5wFLsbGPEu6ql0YynS0FDBkmyG1lwffyCVgtnf6 lypWBS3+HFLGBjpt+jH1YmSwQDfm3ZQQsS7k1Ddjy9xPaG3Ywr/+E3G6FsLtRq35cQJL nFribumkNogKLriYHzXsSpGlGHqiiHtMDVEub8oyGbuubXby7Eb6vj1BQbAm1k4Y6Qf2 Zg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntfpr4qb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:32 +0000 Received: from m0353723.ppops.net (m0353723.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668t9HW022254; Thu, 6 Jul 2023 08:59:31 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntfpr4pn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:31 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3668EORO030154; Thu, 6 Jul 2023 08:59:30 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([9.208.129.117]) by ppma04dal.us.ibm.com (PPS) with ESMTPS id 3rjbs654nr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:30 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668xTDh49283430 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:59:29 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1CF4E5805D; Thu, 6 Jul 2023 08:59:29 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 830DB58057; Thu, 6 Jul 2023 08:59:24 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:59:24 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 11/13] powerpc/book3s64/radix: Add support for vmemmap optimization for radix Date: Thu, 6 Jul 2023 14:28:17 +0530 Message-ID: <20230706085819.827761-12-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: I75rfvMUy3LQo8TermEdIl_HgSY-qAcz X-Proofpoint-GUID: hq5o0CWMKqVmvM-2PZgxQ5iaCpfNcN5n X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 mlxscore=0 phishscore=0 malwarescore=0 lowpriorityscore=0 spamscore=0 impostorscore=0 priorityscore=1501 adultscore=0 suspectscore=0 clxscore=1015 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 48D1A1C0016 X-Stat-Signature: cjbwzjgioks5rha6d466uojzz5phdwak X-HE-Tag: 1688633983-756990 X-HE-Meta: U2FsdGVkX18rJMbBfdAL/DPUM5D0852eMY75M1CRhPZwxwyPaYbjjWFY69vk+oRC8QuriS4HL+1ZUAv8L7g6a9RYurdPykcsSSM8bzopoOKW9F8QYrt1GXuaY3qJtBEPhIs3GDuXK87rpLNaz4gOBKSQId89jyx05zCjwLLrGlsYAkSFauEp6jz8pF3ZLfmdySFJiK44Ofrbr0Dg8jbZ9KjlrWvX/jPHk/vZcLCzplEkIwXIjLscb7Yg7Hhi7be/H2UjWe//gUhulUGwT5EZVqoChjeH3gFI95IPC1fccuomNbhZSR6BfzyHS368f6TVXE1GEXkIm717DeruaJL0vvbcneC0EyvY4aCdFT1iKaqxqAo5KulGSiYwIpEdxHXJw7QwAQjZ30TL3piuTmeqKZvojrCaLhrQl7D7a7aTjryQ1GGid3RQd7szqAskq9PNoBbQzSxXY6ZmuOp9D0jZRsTnmtCWvTkQ/uxX9CI8F09MuS2fmgiDXQH2+WIYIHqpyJ+Q8l0QUkjCeWt6SuthbZ8+gTO+3S5uuQ563EfHAHM36f/yJbf5HuZhFM/8KBEzLeqFAd4pBlkTehefH6G96UlLKK9f8AK1dtGDvBThy5tUHfkhiF7vt93BjxPq+x4obOl9ZyxzetyWnfAMsZ1o55vWDHwic1kgQF0NGHftMEE1sR28n6YgCxGU4ztKJXlrNC/E1mb55jFIE4pWyua2VnpVelb7K7SZmZtwr4zmd5p07YlwPgjpf1NuGgNS3CjVbftIjzCcWhiKP/p7VrYZ+6E1B4TOH08nKyNqZDRMRy3n6wrmIgjCV9YgcfLPAdYf8CfDm0fTHV/U0gSZ6ilgJMFYWhMwKm+i98O86EA5LgGs6kOAGHUhXHOT3VUgHSGxKabr24Qp+DlAU6qs1Qd0etJvWU79YtpwriPPZdx6bzqnLHplRrX6qwXlL6+dNud4HlErJu9Ic5DmQwqaSS2 4WEv3wS2 f98dLQ6yrJfxOi5U4o7AvY9ZGEUhNX6UQeZuz4sxvygh/IBPmsF6+zyKG36BnVwCNGY//vxSXpWx04phwKRXWJfgR9sp2sBIazD3l4QGPCaVOnTLqGXqa+G0U/GuCF2viT0wlskcsCzC2f5z3TQqgV8hjTzURgtP78Frz6R8KbAunMyZxHvFgQw11VRWZFXSPC2+G7LlOJo/SlkAY9pYfmTTYyLKbsaixauqwUL+SkyTi/hA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With 2M PMD-level mapping, we require 32 struct pages and a single vmemmap page can contain 1024 struct pages (PAGE_SIZE/sizeof(struct page)). Hence with 64K page size, we don't use vmemmap deduplication for PMD-level mapping. Signed-off-by: Aneesh Kumar K.V --- Documentation/mm/vmemmap_dedup.rst | 1 + Documentation/powerpc/index.rst | 1 + Documentation/powerpc/vmemmap_dedup.rst | 101 ++++++++++ arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/book3s/64/radix.h | 9 + arch/powerpc/mm/book3s64/radix_pgtable.c | 203 +++++++++++++++++++++ 6 files changed, 316 insertions(+) create mode 100644 Documentation/powerpc/vmemmap_dedup.rst diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_dedup.rst index a4b12ff906c4..c573e08b5043 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -210,6 +210,7 @@ the device (altmap). The following page sizes are supported in DAX: PAGE_SIZE (4K on x86_64), PMD_SIZE (2M on x86_64) and PUD_SIZE (1G on x86_64). +For powerpc equivalent details see Documentation/powerpc/vmemmap_dedup.rst The differences with HugeTLB are relatively minor. diff --git a/Documentation/powerpc/index.rst b/Documentation/powerpc/index.rst index d33b554ca7ba..a50834798454 100644 --- a/Documentation/powerpc/index.rst +++ b/Documentation/powerpc/index.rst @@ -36,6 +36,7 @@ powerpc ultravisor vas-api vcpudispatch_stats + vmemmap_dedup features diff --git a/Documentation/powerpc/vmemmap_dedup.rst b/Documentation/powerpc/vmemmap_dedup.rst new file mode 100644 index 000000000000..dc4db59fdf87 --- /dev/null +++ b/Documentation/powerpc/vmemmap_dedup.rst @@ -0,0 +1,101 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========== +Device DAX +========== + +The device-dax interface uses the tail deduplication technique explained in +Documentation/mm/vmemmap_dedup.rst + +On powerpc, vmemmap deduplication is only used with radix MMU translation. Also +with a 64K page size, only the devdax namespace with 1G alignment uses vmemmap +deduplication. + +With 2M PMD level mapping, we require 32 struct pages and a single 64K vmemmap +page can contain 1024 struct pages (64K/sizeof(struct page)). Hence there is no +vmemmap deduplication possible. + +With 1G PUD level mapping, we require 16384 struct pages and a single 64K +vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Hence we +require 16 64K pages in vmemmap to map the struct page for 1G PUD level mapping. + +Here's how things look like on device-dax after the sections are populated:: + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PUD | +-----------+ | | | + | level | | . | ----------------------+ | | + | mapping | +-----------+ | | + | | | . | ------------------------+ | + | | +-----------+ | + | | | 15 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + + +With 4K page size, 2M PMD level mapping requires 512 struct pages and a single +4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we +require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapping. + +Here's how things look like on device-dax after the sections are populated:: + + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PMD | +-----------+ | | | + | level | | 5 | ----------------------+ | | + | mapping | +-----------+ | | + | | | 6 | ------------------------+ | + | | +-----------+ | + | | | 7 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + +With 1G PUD level mapping, we require 262144 struct pages and a single 4K +vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we +require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level +mapping. + +Here's how things look like on device-dax after the sections are populated:: + + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PUD | +-----------+ | | | + | level | | . | ----------------------+ | | + | mapping | +-----------+ | | + | | | . | ------------------------+ | + | | +-----------+ | + | | | 4095 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 0b1172cbeccb..116d6add0bb0 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -174,6 +174,7 @@ config PPC select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select ARCH_WANT_LD_ORPHAN_WARN + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if PPC_RADIX_MMU select ARCH_WANTS_MODULES_DATA_IN_VMALLOC if PPC_BOOK3S_32 || PPC_8xx select ARCH_WEAK_RELEASE_ACQUIRE select BINFMT_ELF diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index f1461289643a..3195f268ed7f 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -326,6 +326,7 @@ static inline pud_t radix__pud_mkdevmap(pud_t pud) } struct vmem_altmap; +struct dev_pagemap; extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long page_size, unsigned long phys); @@ -363,5 +364,13 @@ int radix__remove_section_mapping(unsigned long start, unsigned long end); void radix__kernel_map_pages(struct page *page, int numpages, int enable); +#define vmemmap_can_optimize vmemmap_can_optimize +bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap); + +#define vmemmap_populate_compound_pages vmemmap_populate_compound_pages +int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap); #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 8a03e1005fd3..c05783466562 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -979,6 +979,15 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start, return 0; } + +bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + if (radix_enabled()) + return __vmemmap_can_optimize(altmap, pgmap); + + return false; +} + int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, unsigned long addr, unsigned long next) { @@ -1186,6 +1195,200 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in return 0; } +static pte_t * __meminit radix__vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return NULL; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return NULL; + if (pmd_leaf(*pmd)) + /* + * The second page is mapped as a hugepage due to a nearby request. + * Force our mapping to page size without deduplication + */ + return NULL; + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return NULL; + radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + return pte; +} + +static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr, + unsigned long pfn_offset, int node) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + unsigned long map_addr; + + /* the second vmemmap page which we use for duplication */ + map_addr = addr - pfn_offset * sizeof(struct page) + PAGE_SIZE; + pgd = pgd_offset_k(map_addr); + p4d = p4d_offset(pgd, map_addr); + pud = vmemmap_pud_alloc(p4d, node, map_addr); + if (!pud) + return NULL; + pmd = vmemmap_pmd_alloc(pud, node, map_addr); + if (!pmd) + return NULL; + if (pmd_leaf(*pmd)) + /* + * The second page is mapped as a hugepage due to a nearby request. + * Force our mapping to page size without deduplication + */ + return NULL; + pte = vmemmap_pte_alloc(pmd, node, map_addr); + if (!pte) + return NULL; + /* + * Check if there exist a mapping to the left + */ + if (pte_none(*pte)) { + /* + * Populate the head page vmemmap page. + * It can fall in different pmd, hence + * vmemmap_populate_address() + */ + pte = radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL, NULL); + if (!pte) + return NULL; + /* + * Populate the tail pages vmemmap page + */ + pte = radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL); + if (!pte) + return NULL; + vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE); + return pte; + } + return pte; +} + +int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap) +{ + /* + * we want to map things as base page size mapping so that + * we can save space in vmemmap. We could have huge mapping + * covering out both edges. + */ + unsigned long addr; + unsigned long addr_pfn = start_pfn; + unsigned long next; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + for (addr = start; addr < end; addr = next) { + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return -ENOMEM; + + if (pmd_leaf(READ_ONCE(*pmd))) { + /* existing huge mapping. Skip the range */ + addr_pfn += (PMD_SIZE >> PAGE_SHIFT); + next = pmd_addr_end(addr, end); + continue; + } + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return -ENOMEM; + if (!pte_none(*pte)) { + /* + * This could be because we already have a compound + * page whose VMEMMAP_RESERVE_NR pages were mapped and + * this request fall in those pages. + */ + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } else { + unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); + unsigned long pfn_offset = addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); + pte_t *tail_page_pte; + + /* + * if the address is aligned to huge page size it is the + * head mapping. + */ + if (pfn_offset == 0) { + /* Populate the head page vmemmap page */ + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + /* + * Populate the tail pages vmemmap page + * It can fall in different pmd, hence + * vmemmap_populate_address() + */ + pte = radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, NULL); + if (!pte) + return -ENOMEM; + + addr_pfn += 2; + next = addr + 2 * PAGE_SIZE; + continue; + } + /* + * get the 2nd mapping details + * Also create it if that doesn't exist + */ + tail_page_pte = vmemmap_compound_tail_page(addr, pfn_offset, node); + if (!tail_page_pte) { + + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } + + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*tail_page_pte)); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } + } + return 0; +} + + #ifdef CONFIG_MEMORY_HOTPLUG void __meminit radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size) { From patchwork Thu Jul 6 08:58:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAB89EB64DD for ; Thu, 6 Jul 2023 08:59:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6919E8E0002; Thu, 6 Jul 2023 04:59:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 644058D0001; Thu, 6 Jul 2023 04:59:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E2D68E0002; Thu, 6 Jul 2023 04:59:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3F83C8D0001 for ; Thu, 6 Jul 2023 04:59:50 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0AFD61C8F27 for ; Thu, 6 Jul 2023 08:59:50 +0000 (UTC) X-FDA: 80980589340.29.2FD3494 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf27.hostedemail.com (Postfix) with ESMTP id 988FC40012 for ; Thu, 6 Jul 2023 08:59:47 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=aM1pXunm; spf=pass (imf27.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633987; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6kxLrFH9O/dgnOxiwQllt1opEKBmkSY2lqJtQlTEuI8=; b=F2I3nVUcgWD4P/RA8MZWapS5UxezA1ux/pWhSZY9y5b+unZzkSrhVOitiVgUkva0nIntL1 NgNppqT8r133dnxVv9LkKBpcElHqOAl6l4s7SO5IQccaqoQwgOTyi4l/GPXlecRkGTkj5h CK7X9NvtFKPtY6LnL/t3MjKDVkhiKeQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633987; a=rsa-sha256; cv=none; b=Bdu0ZKLegeY5h5M15zYSsaGD2WbXMN5VVpsxBwetcfbYEbZNK6uD3XXbVqCEcsmHHN7i0a V5o6KEavYkXqMm9lr9LdjMcItzD//zs1afzTn0oCS8rtEIIhA6DI5ZCcX/a8QXKotW2R5f WQE3IoqocGeGvTjpD66QVrItUKFVNDs= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=aM1pXunm; spf=pass (imf27.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668lcjn027656; Thu, 6 Jul 2023 08:59:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=6kxLrFH9O/dgnOxiwQllt1opEKBmkSY2lqJtQlTEuI8=; b=aM1pXunmJrh1gwpCx6J3gcZns+AA7zOZ2rwspLHOD7dDdCtQz4VWJqqNSALhVh7o4lqY C/9H+6Lbqtwx7gRZleqAauJ3VTrsZlKj1qE9l29VhAcwLxXkUATRiozHE1TPFPlEEXvL T/c6nufLglrpusoAC3wecmfSreaMQWrC60iuTzdjmfAXSgWX0pfqHuP10m9CBrMTeIDu fq5nVUs3Yd6/l16Y1z1S0MDr+yHf9aTSsXU7uU4KXsGBcXfj50aDUuNhwWgsknNxpiBw hdE51QhPjVna5XmSRhjnV9uaJncLBj26pCHqzjVpdoEznSAkuJH0JdTWvjcgKgfeTN1J ow== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntdpg8ca-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:37 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668laYu027524; Thu, 6 Jul 2023 08:59:36 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntdpg8by-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:36 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3668Y8GK001878; Thu, 6 Jul 2023 08:59:35 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([9.208.130.100]) by ppma03dal.us.ibm.com (PPS) with ESMTPS id 3rjbs583t0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:35 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668xYXN55640546 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:59:34 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5A6135805D; Thu, 6 Jul 2023 08:59:34 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9FE9258057; Thu, 6 Jul 2023 08:59:29 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:59:29 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 12/13] powerpc/book3s64/radix: Remove mmu_vmemmap_psize Date: Thu, 6 Jul 2023 14:28:18 +0530 Message-ID: <20230706085819.827761-13-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: I6yQXQ_t4WZHt1CiojNjaT6B5BTrhU44 X-Proofpoint-GUID: RPQhsPkU4-f7pY4ofFUewyBBCePrbR4k X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 priorityscore=1501 adultscore=0 lowpriorityscore=0 mlxlogscore=999 phishscore=0 impostorscore=0 suspectscore=0 bulkscore=0 spamscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Stat-Signature: xkoifea7djn8u13qugwab9o9fs15s9k3 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 988FC40012 X-Rspam-User: X-HE-Tag: 1688633987-474178 X-HE-Meta: U2FsdGVkX1+ozClX/delvCPjx65/Ti05pW5S4UN4qpnlTT5kxpFBekzxqGVKxE7z7lEuXKAEtNA827pW1aZX2zS6S6DtFmvRNZxcbBtOZ6ezZWICCAEiG7p3hchkPyixJ4u1shuK4DLikEOeftREHWObdzRKXctt+Me11p7N1zPYU/596ZPVNYGQW2NSr7yu/4618yuFP6yJT6h+pl5nnLDs/1Oiqwbowpnb02sI+GGIhzMe3c7PEnjecODsmLvwSOybfTz4EqtC98qUfcaWbvl+IG/lbZRwhnNVgeQW3DsQsOF+LWxeQoZjQ5YE3Hd7Yih8itobyuwxkXY0bQGoA9y+ZUDIoMW/jqAxfor2jIJSSv+SsmIY1/7VCT1nwmy+Me5uBq47555YSRWZ3TTeOJIBZMMIDBFYpOa0CbDr9yi3tiB/ZjP7gatJVUomYHcXpy8zbDrvrhPGmeN/MHcmCnnta6svXU+PhbVMbtsLnrGqghIuW+TS5bq8abPiWDaFqfwExW4GkkLEhjyrTf2wSp4nBXqXqdI/VwkGL42AdKHuSliPwQhws4V3LlMcTLdkmcLk9zxcbJKxKZhU5bYczNv69WXBAEJj7ytQWXd7rg2X/ChjXHCcJMaYXaA7fLa1nOWciI+cjFGKZd4q1PxEO/7lyNkqA5pzthlFsHjWvFIytqflHbvSR/lNvmjSn7Uy212Yx/w6rUJnRFlccdH8PvZJ78lJThmVC6XFl1TSPH12Rsv4OvOtjIAyDBxTWFFlIUZhwkC2QMhR6053L2pzWkyyvUg5ba5pu+HiyEcPJSIRseieqQ27DiD4TWooKcM7qVcbdAuxyjVVEjqMZRMA+7PD/TXPFf+OaJmppFZpt3eEQOMEhy5/A/GFDOuQ92kiCZfijTEVZLrDNIPBOrLo79UpxxK4ZVICFkOXncAzy0I6fMxE14gm0YIe8JKZ25OPJJEvyxrRHeJgQgjgah3 BqQwqGHo Rp+72XDK46PZa0lYvPCmbdZUFuEiGAIiALD2GoISUj7yBm/7GQlkZx5QxtwyOah1nHIzJUnIbT0RIcsm+LCCLrOt9ltCvAdK4hXkaK4NLC6uNWbcQiWoNPYF1VP7nlttNbvW4Ba+0SkQDMN6BaqqARy0+Q0FEsYTLgBXPwnCxEV+zKi1P5EcqhzMaWK+i4UGrYhFsTXWb7FST+JaGTEhXO+0sDUzPEDwynomwgRRyg3SYkPAChf8BQbVFsYTWyXuTLxOiuthQOTqrQBDLvfBrI8wDPA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is not used by radix anymore. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/mm/book3s64/radix_pgtable.c | 11 ----------- arch/powerpc/mm/init_64.c | 21 ++++++++++++++------- 2 files changed, 14 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index c05783466562..dac56c883099 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -601,17 +601,6 @@ void __init radix__early_init_mmu(void) #else mmu_virtual_psize = MMU_PAGE_4K; #endif - -#ifdef CONFIG_SPARSEMEM_VMEMMAP - /* vmemmap mapping */ - if (mmu_psize_defs[MMU_PAGE_2M].shift) { - /* - * map vmemmap using 2M if available - */ - mmu_vmemmap_psize = MMU_PAGE_2M; - } else - mmu_vmemmap_psize = mmu_virtual_psize; -#endif #endif /* * initialize page table size diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index 5701faca39ef..6db7a063ba63 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -198,17 +198,12 @@ bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, return false; } -int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, - struct vmem_altmap *altmap) +int __meminit __vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) { bool altmap_alloc; unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; -#ifdef CONFIG_PPC_BOOK3S_64 - if (radix_enabled()) - return radix__vmemmap_populate(start, end, node, altmap); -#endif - /* Align to the page size of the linear mapping. */ start = ALIGN_DOWN(start, page_size); @@ -277,6 +272,18 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, return 0; } +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) +{ + +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix__vmemmap_populate(start, end, node, altmap); +#endif + + return __vmemmap_populate(start, end, node, altmap); +} + #ifdef CONFIG_MEMORY_HOTPLUG static unsigned long vmemmap_list_free(unsigned long start) { From patchwork Thu Jul 6 08:58:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13303386 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 922D8EB64DC for ; Thu, 6 Jul 2023 08:59:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 299728E0003; Thu, 6 Jul 2023 04:59:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 248E68D0001; Thu, 6 Jul 2023 04:59:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1390A8E0003; Thu, 6 Jul 2023 04:59:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 06A168D0001 for ; Thu, 6 Jul 2023 04:59:57 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BBD521404AA for ; Thu, 6 Jul 2023 08:59:56 +0000 (UTC) X-FDA: 80980589592.17.BA0F89F Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf25.hostedemail.com (Postfix) with ESMTP id 3F061A0011 for ; Thu, 6 Jul 2023 08:59:54 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=eYMJZPLf; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf25.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688633994; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R5o3btXCmicCo4Tpp9Fg5DPacjE5pAaul14OJ0FkTj4=; b=dlj8GbkoRPxNDbL9Tg2HcNJ6mDRIWH+lqozF0OJca/9Wx4f8GKjNprmeZUDNzGyuxAwsdr QW4wxN/hrf/8BTlm+Hy045jjZ+njjomu0Hv4MaiEGRB3yA1PuptT+l+E77wpL1FDbQJ50v UqLmxCyNwpnqpGv0u+efC8URzg4xR9k= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=eYMJZPLf; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf25.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688633994; a=rsa-sha256; cv=none; b=7YUIMnyP0iChq9AifZtmeROOD833uv+gEPqMx68f8w0fJauaAQuZDBiqOMQEVa6BLYEHWe TNp+VdsSC3ZdjffmkfLTY3mggUzFFh2n+Jsft5K6jrc/1qEV4c1NsoeST7b9+OnEeWIZoN S/DEvgkYW80fLs8n9bo1Siy25QJhaMU= Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3668kiT6018556; Thu, 6 Jul 2023 08:59:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=R5o3btXCmicCo4Tpp9Fg5DPacjE5pAaul14OJ0FkTj4=; b=eYMJZPLfHRMVxuwwIVyJRH/HOGVBuorO1HApiaeoNPPC63EiOICXwXIdSOKBGRWNFw9b DUhcRyK086xjQE+RTADaSqoWU3E5EShySGXIjnf33wv/XIfeMBEUtsx71AvkOWfeauMQ 80yC46K+o8CuBBcgRxBx7Fh/H/psaNiq+7Ajc6f8j37rsPAj1xvCZajDF3663wWpIIlt oR5VJnvF3+UNcruP/ev0ST7g58XmpKdB4aGsNLSkfaCv0NBVEGgd56d4UvvsjQlBY2MA SvZpkgTKnitU0zonHyDZ9NB7lsg0k6WBDNTzT3ZjfGMPJyZgxqVPNw9AvCYU8pIuv1bc 8Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntd80bt3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:42 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3668koB8018858; Thu, 6 Jul 2023 08:59:41 GMT Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rntd80bsg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:41 +0000 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3667oaNX014939; Thu, 6 Jul 2023 08:59:40 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([9.208.130.101]) by ppma02wdc.us.ibm.com (PPS) with ESMTPS id 3rjbs67x4a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 06 Jul 2023 08:59:40 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3668xdLG1376944 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 6 Jul 2023 08:59:39 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B8FF858062; Thu, 6 Jul 2023 08:59:39 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E419458059; Thu, 6 Jul 2023 08:59:34 +0000 (GMT) Received: from skywalker.in.ibm.com (unknown [9.109.212.144]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Thu, 6 Jul 2023 08:59:34 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v3 13/13] powerpc/book3s64/radix: Add debug message to give more details of vmemmap allocation Date: Thu, 6 Jul 2023 14:28:19 +0530 Message-ID: <20230706085819.827761-14-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> References: <20230706085819.827761-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: ZEf-ItYJYP9ALjhsQNJsxs1vuqIAM4_N X-Proofpoint-ORIG-GUID: Z7SWV4WQcHepAh7_Kbdsv69gkpfSrNsp X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-06_05,2023-07-06_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 clxscore=1015 malwarescore=0 suspectscore=0 adultscore=0 spamscore=0 bulkscore=0 impostorscore=0 mlxscore=0 priorityscore=1501 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307060075 X-Rspamd-Queue-Id: 3F061A0011 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: hwsrs4maztpyqp1azom67cpqrygjr3wj X-HE-Tag: 1688633994-413296 X-HE-Meta: U2FsdGVkX1+iyBj4m9bdINzt2aawpOdUjK9QfZ09e+tk2X+c2eNDYX4XnLHaGLABNqvg3pPE/PimUAsDcSzNCSkd24mvkYd7+nozoyhYWWrCqqjO5aoHQONKkLEBxKZNuCGK0R/fMJZRX3S0s0Inb21bp6Cs+FW59TVq8q1ldaIlniEpoT36AUCwjbXWghMf7k2kTJbCXZOH+gXT/pNxm/GNSb93vqvzQIP/ICkdiD7voH5kMf1BObADgqjAk41pvMk95sN49N+fhQdB0LJEUYYW3KpSjwPY9W5NZOsNnnVr1WlXmwTeTWl1LeyXwLivBZ2sPNIq65CCaIJ+yBL3Ms9FPSgR//R089Q9+ikGV7xbGL9D5CIUfcF/RkFUtKGiQmLAl+rfGHVPLozNm+cs9JthrsDQDe9zqVRirBz6zFRqtm5hFwWcflJMM9O32yWDeSJC/6ji8tKm01qadLRW6i0xQU2sB06RRVcJB5NIda36WSQbw9RlSvmHDg56ScNuETxpKjy7QDt3XsYjL84jm57bmdi9S7mBV2j1CNmwbkh5n5iOFxaSH0KeCZtcOKDaglDXeWqe1JvDxqe1nFziwyY+IuCSX41/sOo6yWiyD1i9cuiD1zIOH34o0YroJtBSRgcSlygBQnSxNLwbwxyj6FtuB4QEkQFYh8wLnFDod6y44zieHKuS58LzLX+HI4AlX6GxR8lQjRKSosb7fkgp/y9mWiVr+zek7B09SmjR4iJAvQ93ZxOQAJylx8uwhncRFXM2C45Yv5mdbntHfoMH73f4BroggZ76ikx7QQ1U5c2uOKl8ZAss3fxFQPQjbLf8ruTmU2JFo9YNtYo1PZv9ARwAoDIlnlkUgCgkUmhKquwOwvO4qsYXpsCwQLzpAvndGNZHUetyi/aLFEqkN1LKcss9gxPjKywetRxtwP7zGMslR5a/oIxvnrFTPOdD7rfayM31Lg6mFtFEFUFO3fz OBR4MELS IACSgACV2drB4OsEbLcfy9ObojMUeAAO4OHLT0Cbe/EO1gte73qLqr0yGUpvf4GU2F9HLls/2Yi+4t6VcYHbwEF6sO2XUeU07Ubpvlj6F4crZ9TnjxLBm6UeKtkTkafBXDKu2MQ21gFYI2U08HjWCmxBbXLOeQo/U4Xc2VSU7HMl4+C+SR8aXhbbb6F1/64iY93DnZJEQW3DLoCoIvIJkMPESHukG2kmmHdn5L0K7JQnw+tE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add some extra vmemmap pr_debug message that will indicate the type of vmemmap allocations. For ex: with DAX vmemmap optimization we can find the below details: [ 187.166580] radix-mmu: PAGE_SIZE vmemmap mapping [ 187.166587] radix-mmu: PAGE_SIZE vmemmap mapping [ 187.166591] radix-mmu: Tail page reuse vmemmap mapping [ 187.166594] radix-mmu: Tail page reuse vmemmap mapping [ 187.166598] radix-mmu: Tail page reuse vmemmap mapping [ 187.166601] radix-mmu: Tail page reuse vmemmap mapping [ 187.166604] radix-mmu: Tail page reuse vmemmap mapping [ 187.166608] radix-mmu: Tail page reuse vmemmap mapping [ 187.166611] radix-mmu: Tail page reuse vmemmap mapping [ 187.166614] radix-mmu: Tail page reuse vmemmap mapping [ 187.166617] radix-mmu: Tail page reuse vmemmap mapping [ 187.166620] radix-mmu: Tail page reuse vmemmap mapping [ 187.166623] radix-mmu: Tail page reuse vmemmap mapping [ 187.166626] radix-mmu: Tail page reuse vmemmap mapping [ 187.166629] radix-mmu: Tail page reuse vmemmap mapping [ 187.166632] radix-mmu: Tail page reuse vmemmap mapping And without vmemmap optimization [ 293.549931] radix-mmu: PMD_SIZE vmemmap mapping [ 293.549984] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550032] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550076] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550117] radix-mmu: PMD_SIZE vmemmap mapping Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/mm/book3s64/radix_pgtable.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index dac56c883099..a62729f70f2a 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1026,6 +1026,7 @@ static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long p = vmemmap_alloc_block_buf(PAGE_SIZE, node, NULL); if (!p) return NULL; + pr_debug("PAGE_SIZE vmemmap mapping\n"); } else { /* * When a PTE/PMD entry is freed from the init_mm @@ -1038,6 +1039,7 @@ static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long */ get_page(reuse); p = page_to_virt(reuse); + pr_debug("Tail page reuse vmemmap mapping\n"); } VM_BUG_ON(!PAGE_ALIGNED(addr)); @@ -1147,6 +1149,7 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); if (p) { vmemmap_set_pmd(pmd, p, node, addr, next); + pr_debug("PMD_SIZE vmemmap mapping\n"); continue; } else if (altmap) { /*