From patchwork Mon Jul 24 19:07:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DCF8EB64DD for ; Mon, 24 Jul 2023 19:09:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B806E6B0075; Mon, 24 Jul 2023 15:09:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B31236B0078; Mon, 24 Jul 2023 15:09:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D0DC8E0001; Mon, 24 Jul 2023 15:09:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8BC856B0075 for ; Mon, 24 Jul 2023 15:09:52 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 601AC40B6B for ; Mon, 24 Jul 2023 19:09:52 +0000 (UTC) X-FDA: 81047445024.14.4ABF26C Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf20.hostedemail.com (Postfix) with ESMTP id 061EA1C001D for ; Mon, 24 Jul 2023 19:09:48 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=m+ysZc2+; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf20.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225789; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WeZvVAViPZwcgjWudBTpiOsN3GLFM4bhGCOxEWup4xM=; b=Tie5IntDu81XOD4suADMcFLzgoOTz7SWaidVpzDtGn2VeJSdNB5tk2prnIb0/K22yamPMO vsByYF24vUjjYmt77SwkRyKOR532JS/CYPyMCKc0kVOwzPSzICmvPGb7JvL0W71oh68SOG gAeHCtM2tlKH9hvPzVeHsNIX/zqV0so= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=m+ysZc2+; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf20.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225789; a=rsa-sha256; cv=none; b=WbtnElK/WFLr76rg2oW/Pm41GDYJ6qJtMgYh8rliquHnX215OluY7olQfeFF/alvr4sNpu saiGTVurGjgSJGURNFVcnBZUO/o9wnlfrVeXZ7jHrQxkXtohGlG4YuIO6oYy6PZ2IMJ7OI DA7X8fYkQ3txegh7ZfRran1uNdYQKUQ= Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OIbmwK010747; Mon, 24 Jul 2023 19:09:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=WeZvVAViPZwcgjWudBTpiOsN3GLFM4bhGCOxEWup4xM=; b=m+ysZc2+Q3mtyz98I/drQLVWDVykgifNrOoQx1cY24zfpClm+KP0Q1PLHU22Vy1zIqBh CXNWTLYv0wIwooxbhNcvdxo9hcAvOSs9mt8uX7haNweO1YwbkbImoT1IC7obxuddfQSv IsTlzWbRbw0ANB5ftaY/CBWTjrT/5RmOvOFNE0h1XjEro6D5S5Ga/CKG45lzpdzFSauD jqMNtKCOgQUq8U46+LPyTCS15rpsbR1VCb1JM+zqRk/zu7Euzx31Vx4S2W+j3/iMkDT9 AA7ehCko5r0i0iwRxU/88EW+iYWOlelzPr/wIP59AvshfrbGqXrNTLmbxHifUzYQkZhT pw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1xcj9785-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:38 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OJ4e6h014533; Mon, 24 Jul 2023 19:09:37 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1xcj977m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:37 +0000 Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OH97lD016569; Mon, 24 Jul 2023 19:09:36 GMT Received: from smtprelay01.wdc07v.mail.ibm.com ([172.16.1.68]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 3s0v50w5pt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:36 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay01.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJ9Z7t36504226 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:09:35 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 65CC358065; Mon, 24 Jul 2023 19:09:35 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7A59F58055; Mon, 24 Jul 2023 19:09:30 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:09:30 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 01/13] mm/hugepage pud: Allow arch-specific helper function to check huge page pud support Date: Tue, 25 Jul 2023 00:37:47 +0530 Message-ID: <20230724190759.483013-2-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 0G-YvGIt8qpupwO1YmKQygBjk_qY-9QQ X-Proofpoint-GUID: 27dlQ96eQd2WGDqz-v2T9cr-ydOo2eCd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 suspectscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 spamscore=0 priorityscore=1501 phishscore=0 lowpriorityscore=0 impostorscore=0 bulkscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 061EA1C001D X-Stat-Signature: fjmuf18ubbnonrrfzuszo6nxmbpo96bf X-Rspam-User: X-HE-Tag: 1690225788-351326 X-HE-Meta: U2FsdGVkX1+0ldEybqitIl2ikbD3MTaNWJGPa7YSpWg/kBZFIkdR3PbvAbYHaIqVjlBWWCzO0F6muywP8GrLhhxJzVA8yyeHKAF4QBO+Kia8N3o/6t56tUUwhxbrjWOlTwtdnIXTEn21xzonsjcDzxD4mEiqLtKBw3W+WZZCG68fXqE2gXfiWovRM2CMjZCEq6I4BazOvNVl0G39cPWTMIpYD/sLcPoEynXy8rm6cD5KQ6YsjzEnjW+yhm3xsnRwvsUJB4lue2QUG+xnRf1c4u4OXd8thzxlNErczvkeddDBvUe9gkfApxhJJfGVS/K49aVjf2F5QH3VD8OzJG5u2QJjpTT+r2VOKyQPimMICFwM7JAzmRXFcFrw0ik46mFRBr21Hhj3jGerydwgJaihh3a+pEpacD8aUi/JX5XGBY+SOi1x5obpd2BF7Wp9CSJOMKjw/6h8DV42527qvWwjT0zN+QRaSPVKzjAnN0AmqnWcUC3JmUdXD+RtyiUVvTWnmDRtaMIpYV2PXS7IGYW1NV1O3zUwG/EJQAnAzQiFkUUa9E5hJwOfhCufCA2Og5awFG38fl/HCFsxWz3TxmaTuGnCVtpGA/NUYzbCtc11UhFgbwxArRbK8zKqGQFFkP+wCEvxyG5NpEA90mlyUGwqwRo7yU3g8Smu7/5Q0YyvYDTapAznTJLlIq8D+vkzdFsIMdZ2FJM9lyKSzp3wODnOX4XG2csO4l2MuZ40qYLvSvg00EGRH0ln8d0oJ/6/Eeeb3jD3QVI0YdkHcnJVhknX0qbB+JTZTxUW8cCFC6x5E1+tux6+eCF7WVGrbnd8wNDNKUTbE1M2flvWBbDlhWgExWvqvqra6IB4Z7GIwuxdHyQ5dJb4feGlOm3qQywA4OstRknWPndi/z2IgnSoVxiIV4EygKtt05vo/qmXUanNi8yT+TpTFLmHNZ1He/tdokKxPnAaqITZ3U8u7qcpzWn siMkD3h0 nvbqMlCkjP/jZEMPLMe0AS5VlEAY/IC0LfPlv262IWsavoEppxNyGNUJ6foKtTesFqzrx6FA87Hd5bYR1k3T8NI+M9jX5FQ0JZEeZouCTPszsRwSh2fSli02MJqziBLEQZb0l5EfLohbG7LYaEB4vFdO3ErfgAQUhTzn4UJOWTpMRAUY+3VfsbshfEO/MQLz1QfiAat6qqWHrKEIoWUkUx2E2tcfIjyZUe8lOvo6SPxx1S4fWBUt8gSMQUv7yZCA9sAFL2Gol9+Cna90C+aolH3ek4Rf0CrJe6fyyLE99GXF5JTQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Architectures like powerpc would like to enable transparent huge page pud support only with radix translation. To support that add has_transparent_pud_hugepage() helper that architectures can override. Reviewed-by: Christophe Leroy Signed-off-by: Aneesh Kumar K.V --- drivers/nvdimm/pfn_devs.c | 2 +- include/linux/pgtable.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c index af7d9301520c..18ad315581ca 100644 --- a/drivers/nvdimm/pfn_devs.c +++ b/drivers/nvdimm/pfn_devs.c @@ -100,7 +100,7 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) if (has_transparent_hugepage()) { alignments[1] = HPAGE_PMD_SIZE; - if (IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)) + if (has_transparent_pud_hugepage()) alignments[2] = HPAGE_PUD_SIZE; } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 5f36c055794b..5eb6bdf30c62 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1505,6 +1505,9 @@ typedef unsigned int pgtbl_mod_mask; #define has_transparent_hugepage() IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) #endif +#ifndef has_transparent_pud_hugepage +#define has_transparent_pud_hugepage() IS_BUILTIN(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +#endif /* * On some architectures it depends on the mm if the p4d/pud or pmd * layer of the page table hierarchy is folded or not. From patchwork Mon Jul 24 19:07:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CED0C001B0 for ; Mon, 24 Jul 2023 19:10:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A47356B0074; Mon, 24 Jul 2023 15:10:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F7986B0078; Mon, 24 Jul 2023 15:10:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8986B6B007B; Mon, 24 Jul 2023 15:10:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7A5A06B0074 for ; Mon, 24 Jul 2023 15:10:04 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4FD6580746 for ; Mon, 24 Jul 2023 19:10:04 +0000 (UTC) X-FDA: 81047445528.02.940E25A Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf08.hostedemail.com (Postfix) with ESMTP id D378B160011 for ; Mon, 24 Jul 2023 19:10:01 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=lHdXpj+y; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf08.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225802; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gCnHgXezPpuk6sMKSzDIWp04p03Sk4YUXXkML+wstSQ=; b=bK0nJO/rKxYQru6wwESTGc9plORW3C3qjoQfwvuHLiyNM6mZsHQSan8z3RNBq98UUO0+Lr jAW42d0n7++SUqeSzFJIPOCGHvgnHliBLRGYDGuQevBN4rexb8YHR+8ophqT4E++vcWgrS 30qB02oOYAma9KQxrPeogps4cqGflsE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=lHdXpj+y; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf08.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225802; a=rsa-sha256; cv=none; b=vPwyMkHw6ZAgunm1lVVsx/A9C5xMERVCmobp1A5PrZHqZZEicmHBqupV7qRTAGYBSD2DJJ /rl4mi3IGD47P0BUTWvSy7H7adUK/nGbeo4XmoESgr/Iky3S0I7OPM7YnkYxlhML5csHCV b2u8++GdHpLzjrlLPXXHCfZ3u6O/mhc= Received: from pps.filterd (m0353727.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OIbGNt032728; Mon, 24 Jul 2023 19:09:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=gCnHgXezPpuk6sMKSzDIWp04p03Sk4YUXXkML+wstSQ=; b=lHdXpj+yWdleWVdb9X4WrxNQulmfyHmaqtmnD5rDOLVSIZT24xTpN6KBD1zasYB5bKNG twitQUfYf+kUPAPlHFne1/PiNugsxzt+vKtJ1kLc+NMHcgX58LEx+I6d5MD8TC6ti7en oxRWvC/w9uvySm7pdd+CPJU5jY7f/+B1l6dMMuWvaSjl1+qYF8xwnddUD4modftWIWFH FhFWyP4x4ZB6RT1j+rNDsoNtH+4PT9GPSuiMcSK2dJFiTkxTn2l6RdhEwkARTjznejMH RMt7eI9zYhUdDY/vBzc4E33ivMr0l6q6nsp875/8QAZHPILtonhdCcqEhIdKjpt5RtPo Ew== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1wxa1ud0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:49 +0000 Received: from m0353727.ppops.net (m0353727.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OIrBS1010766; Mon, 24 Jul 2023 19:09:48 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1wxa1uck-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:48 +0000 Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OH9f0O014373; Mon, 24 Jul 2023 19:09:47 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([172.16.1.8]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3s0stxp1da-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:47 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJ9kCR1376968 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:09:46 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6EB3F58063; Mon, 24 Jul 2023 19:09:46 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 88B4D5805B; Mon, 24 Jul 2023 19:09:41 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:09:41 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 03/13] mm/vmemmap: Improve vmemmap_can_optimize and allow architectures to override Date: Tue, 25 Jul 2023 00:37:49 +0530 Message-ID: <20230724190759.483013-4-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: Dl3h_6RDGc0Vl7C4TUK_Ipy-m4OGf9uR X-Proofpoint-GUID: oytDnj1Mg_SSWRUS5_Ex18PfGl4mdKtP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 mlxscore=0 malwarescore=0 suspectscore=0 mlxlogscore=999 priorityscore=1501 impostorscore=0 lowpriorityscore=0 phishscore=0 bulkscore=0 adultscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D378B160011 X-Stat-Signature: 333ib17wsns34fkwax47qh9mmxxfyu6i X-Rspam-User: X-HE-Tag: 1690225801-776584 X-HE-Meta: U2FsdGVkX18gNc+l+LILkDMsS+UfVcw4zzNfVvh2tyIuHNd+And8e/FkImK5/fDvEHsnfqDHa+Fx3zX4Hgx1Zr0TWe7fnc8c345FGdHJTrx7ep4YBcG/fACQyne3REIGnUFZs0h/wPN92Ssc0xxxhgDPIkF0UydGULiDTCINJITPwtNcHA8igo6nn5IxCkKfi8hgmQiMas/RR5bxAz0V3tpzKmwCLXPyMwuyQ6QsBCY5/HgnhHsmY3u4BiHD1veC1PoaCpywdCP/bIigdxASGPNmfDWWxCtFapmtvGGZh3D1NSb0ipAgl6V4aIWLbFGCXV8y9drAYstCj90CN/6h81jnyYpi4d5RaFrbyP14KcwEcSwxIsVTtWRTZSS4fVmdLeyYU2m0i8W5CRi2TR3QqbpQ3qatxltHdUSKTksZPUrYylFMXA2fJ/0T8XBfKXxuZXuDpsvrqJs82SIjmLTo3KiqKxBPKRqKH8+7i/vfYJTe88FVRbzho3ZdFgYixaIE9r3xcXE0xa00aKh+vZdWyh1yhJGhaPHd9CVZJwN4twZXGpyx0hyBVObH0BJoZNoYA75fDuuwx/VASMxRV/WDO8w8HNrZxz4ZjL9W6+0PP9cc+IVpR9O/b9o/sP7WHHaakS3ePP3ddAoeTZKlhjszahrhyUGERdfRgeweI40LudPMSBj/UTV27hluNlf7gYMdaIRsglsCR0ehUsp9SxZqPnC+gkCPSXfrKHfjLyEuEtEP+8RO3DKztWSMarzg7REbzKCEf1x8HEP1ymijtHi2GKkkee6rirPQMhNcXYsvOL/8SJonA1ClO4IMo/JENJkrxvr+SJJ1Jo1aNQBacTGolfuicRLv0gzRDUaWOBsAwrP2AxbG68ipOlSqvgnCn3hrS0kvImD23fpChhFVbNAazDEUhZ3jyoCjA3xGEJepz+szWD8xDPJzFuvC1XV5cKdCsgZJwHRva6uHdKm8t8s XoPQdbfD Ka02truwoZbyEnNIjr5Mh51FBB2Di1mG/US6Ju+9HON7xLnzF5CbcJMbLOq98T3ZZgHtIzkLvBSHA+r8gSm8v9zc/DCSUS4EHGkPPh4vFq1GLPIKRSimHneDr/Mk7GRURqy7+mAdJCybighoEweTqXd++uLZnJHJtfhIUCYU2utGkYx3KAAAVx2ADHOXDH2gictMd9LmSm6UoXN4X2e1Ao4DmGbtwBDE0HgWvWO82B42DkRkKH1f5HRUCbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: dax vmemmap optimization requires a minimum of 2 PAGE_SIZE area within vmemmap such that tail page mapping can point to the second PAGE_SIZE area. Enforce that in vmemmap_can_optimize() function. Architectures like powerpc also want to enable vmemmap optimization conditionally (only with radix MMU translation). Hence allow architecture override. Reviewed-by: Christophe Leroy Signed-off-by: Aneesh Kumar K.V --- include/linux/mm.h | 27 +++++++++++++++++++++++---- mm/mm_init.c | 2 +- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a5d68baea231..fe6741539740 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3649,13 +3649,32 @@ void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap); #endif +#define VMEMMAP_RESERVE_NR 2 #ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP -static inline bool vmemmap_can_optimize(struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { - return is_power_of_2(sizeof(struct page)) && - pgmap && (pgmap_vmemmap_nr(pgmap) > 1) && !altmap; + unsigned long nr_pages; + unsigned long nr_vmemmap_pages; + + if (!pgmap || !is_power_of_2(sizeof(struct page))) + return false; + + nr_pages = pgmap_vmemmap_nr(pgmap); + nr_vmemmap_pages = ((nr_pages * sizeof(struct page)) >> PAGE_SHIFT); + /* + * For vmemmap optimization with DAX we need minimum 2 vmemmap + * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst + */ + return !altmap && (nr_vmemmap_pages > VMEMMAP_RESERVE_NR); } +/* + * If we don't have an architecture override, use the generic rule + */ +#ifndef vmemmap_can_optimize +#define vmemmap_can_optimize __vmemmap_can_optimize +#endif + #else static inline bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) diff --git a/mm/mm_init.c b/mm/mm_init.c index acb0ac194672..641c56fd08a2 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1020,7 +1020,7 @@ static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap, if (!vmemmap_can_optimize(altmap, pgmap)) return pgmap_vmemmap_nr(pgmap); - return 2 * (PAGE_SIZE / sizeof(struct page)); + return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); } static void __ref memmap_init_compound(struct page *head, From patchwork Mon Jul 24 19:07:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55149EB64DD for ; Mon, 24 Jul 2023 19:10:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D554B6B0075; Mon, 24 Jul 2023 15:10:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D05076B0078; Mon, 24 Jul 2023 15:10:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCD2E6B007B; Mon, 24 Jul 2023 15:10:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AE0E66B0075 for ; Mon, 24 Jul 2023 15:10:09 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 49A25B22CE for ; Mon, 24 Jul 2023 19:10:09 +0000 (UTC) X-FDA: 81047445738.09.2D79705 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf30.hostedemail.com (Postfix) with ESMTP id B5CA78001F for ; Mon, 24 Jul 2023 19:10:06 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=tWpaEUCU; spf=pass (imf30.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225806; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LTGdpg9vP6Z9t1EVUl4AJKSRgiUAw7qTPzDp7kO8kxk=; b=yPXrseXuOHGU/k9egLYF9w0SGX7Y3Morm3hm1pqosQWiEC5pz3wDBTwuqpCBo/wLp6caRa qyt56XvG5jStuUVmhM144SHCf6W52nw/bJshPhLVzDYds1yl9Kqjd0uKyVLrt+/fNMex8A MQL2aR4ihIiiO/ATV0SCbd5jqlfyu6U= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=tWpaEUCU; spf=pass (imf30.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225806; a=rsa-sha256; cv=none; b=pNivDWL/DctjUgrgkrjjbx1N+gEgVLDsTfMCD9ncpYjBu6lhYlxnFAGg+pkCInlCbMDpZa A79DSBkkioGzuAZ14RjGDI6ffCQUBaAJ3sfrud19/EtU7DoLptxeh5H2TAHl2moTtE4Vqs TxVGM2d904NH+5eNqAZt0fE0D4oBDQo= Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OIvTdF022075; Mon, 24 Jul 2023 19:09:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=LTGdpg9vP6Z9t1EVUl4AJKSRgiUAw7qTPzDp7kO8kxk=; b=tWpaEUCUcHII3znEroRxANvVIW19m0X5Yns6tnmykyiQ0InVVPgtJvM/YU+II+RHhARS lC8Hk3bnHyriICGoFizOMuHaI5R8PMIPOSrU9hiU1P3s7oUBAuetJRvq0qqe2bJmoZlT CP/eKrM/3XQGLvRPPcs81lFSpieBIQERKJFxvqESjxG8/TUG/NmQT/Mu02q/OlLTWU5o lTED5xuY3AkSyTeo+E8yuNXAAxEtsuQKypYSS9AVje5kDO2Kt0qEPDdhXX5+jzOSrVkU osFcEU0jV7dx0BE0prl/ltPoEm8lkzKcC47l3okIU8R0TpCZVOYReeYHnv8z5YlonErh rw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1xcj97fd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:54 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OJ4ieG015204; Mon, 24 Jul 2023 19:09:53 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1xcj97et-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:53 +0000 Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OHa25o002068; Mon, 24 Jul 2023 19:09:52 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([172.16.1.70]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3s0temns6a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:52 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJ9qwk1442526 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:09:52 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0552D58059; Mon, 24 Jul 2023 19:09:52 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0EADA58055; Mon, 24 Jul 2023 19:09:47 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:09:46 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 04/13] mm/vmemmap: Allow architectures to override how vmemmap optimization works Date: Tue, 25 Jul 2023 00:37:50 +0530 Message-ID: <20230724190759.483013-5-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: ysJLL6sReH2rTHRL1ZY8_-oWui3wUOtk X-Proofpoint-GUID: x9Nh3Sq4xQokZ0n30kdaKCzoB0ay9Jft X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 suspectscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 spamscore=0 priorityscore=1501 phishscore=0 lowpriorityscore=0 impostorscore=0 bulkscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspamd-Queue-Id: B5CA78001F X-Rspam-User: X-Stat-Signature: dtgp8h4ob46xyurg6z3ypk6p1e1bhire X-Rspamd-Server: rspam01 X-HE-Tag: 1690225806-548673 X-HE-Meta: U2FsdGVkX19iGqhv2/e1KWhm2nKCqKNJw+k+uTgjGiE5KTIuQXmHKvCv/dEbquE1hZLXovVtNUwCJ70gqK8Y3pgoN2s0Lbuo1KbPzIasmEaRCGd6l4yYYoYOWTnUffLb2uN4qbGL29GYBPAG/0p6Br5TZhmmk3gX4SCdi9dZzEbl2Ipm6OzvQunC8Ew/6ri89u9fPpI740H93rYj02PXbSzgAb0hnKh17d+QmtgMaSFz4Bb/ZMSxK7XsKnmP5CiP2biFQL0j+vaQ56qDTxQrEleMy8Aagaw7Vg2ilyyXGfT14bAhMqM+5nLLNsifjiEsFChlsNJK1xJcpCe+FGmSB74FK+ko0DfGGzt6HRvrOpQrn8FlchJ+ulTZECZqTfnF/qIL3dQHXkrQvUIwW2VU9f6OkQqnHGLLwyGcVMR6ifEdC8XijFM9s9eRbCleVmUIl3mgPDXjUAXtz3oIbBef/7RsP4iWNUq449EePm/OCk37yT3tZ4ZqxbCHbOZKXF53pDdBmFzIb1R0VUooeUIIUdikypc4c3QyAFCYAZyirU6IfUUWSFN+/KCBEDehEcIDKTQjpMb/Gsw2rOsv5V5NWI2AAEJozDua279GUCH/EoecZD9fslwzQxf63qs3ShaYeAbL+P4vcMQ9ICqaj8LvETh9lO54kOM/CRV8lpklfJxbZNjXZ3ASfZrcMfqDnWNHlYHZ1fRXn0/Hnzb4jVJKqIvIA8FrAGKtLsTsRgh9iHEjVWJZV1QkvY2aHICUHrG92Utbm/HiLgtAFQMr+5dhnPBDrqnK7aKcUSSKNPZlXiYa55RSXlVuOXW7noo8a0fylzWU4d5j9PYABD0EzpOZ9o7W1Myn3OTPbsgd2O0toVuSAAQK/PbcBIzR8R/+vUdjkmSSfA+aWCysk1ZVbUC4dTKJMXbVfdQxWrIVujEmS7HOQRybsISbnPnmE8Ov2Plr/xFDuTcxgI+Lv2aylm2 JBDsBOrh lgDCdh0D5Mr7wMJAFr85axEaXOIa8krMbu2g0ku8q/DJH90sxXItQfIBLBh9C6PED6Ir+qgmPjak+Vjuk7rN3WoUtnOZmDhom1mu5dBmZeUboXN09vI2GTKoT50W6G3MZZqyvepY9N4abizo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Architectures like powerpc will like to use different page table allocators and mapping mechanisms to implement vmemmap optimization. Similar to vmemmap_populate allow architectures to implement vmemap_populate_compound_pages Signed-off-by: Aneesh Kumar K.V --- mm/sparse-vmemmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index a044a130405b..a2cbe44c48e1 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -358,6 +358,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return 0; } +#ifndef vmemmap_populate_compound_pages /* * For compound pages bigger than section size (e.g. x86 1G compound * pages with 2M subsection size) fill the rest of sections as tail @@ -446,6 +447,8 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return 0; } +#endif + struct page * __meminit __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) From patchwork Mon Jul 24 19:07:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325272 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E0A1C41513 for ; Mon, 24 Jul 2023 19:10:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A6E8E6B0078; Mon, 24 Jul 2023 15:10:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1F038E0001; Mon, 24 Jul 2023 15:10:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BFBD6B007D; Mon, 24 Jul 2023 15:10:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7CE0D6B0078 for ; Mon, 24 Jul 2023 15:10:15 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3D8C8160B90 for ; Mon, 24 Jul 2023 19:10:15 +0000 (UTC) X-FDA: 81047445990.19.8BD0416 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf30.hostedemail.com (Postfix) with ESMTP id 70E4880022 for ; Mon, 24 Jul 2023 19:10:12 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=cAkkE5ER; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf30.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225812; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PoFFwtWHq+ffbzbCzl2ZA0XdhdJwSM3TndhjbFI8QnQ=; b=YPpDCsNfMfs9KE/xEHcXrNbqIHlbchGSrHvy5OLjasJOIpViBqPIHDvDqI0SJt+4vfPmRu O9RZ2zy4/wbFYXtFYbpXV71Ka/BVC4NtaU5hSn5KhrovZgYxnoN2JeuExz91ZG+MzCvB/U za9mE1ff3Q7kHbp7kwUItUPaSyChbuk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=cAkkE5ER; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf30.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225812; a=rsa-sha256; cv=none; b=UthM1+25lgBI1OMGPHKJkh3md50bUgbABuwyfUH9QNW8j5EnetNeyW+AvTgIwH073Yw2X7 zzW6tctecSrSUaQIrkgk0Yg+txD+cixXcnAswS5SkbdxxEJUVuif4dxPEKTWIcsoLCIbXF sJ28i/OER1Rd+5meP8VAFLcHDYc6G2s= Received: from pps.filterd (m0353727.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OIbJTu000495; Mon, 24 Jul 2023 19:10:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=PoFFwtWHq+ffbzbCzl2ZA0XdhdJwSM3TndhjbFI8QnQ=; b=cAkkE5ERFGUBHxestudUO2BuhBBK527JcL30ttWlgnijTlB0Xf/0wmFnr7xJXnbK2+nD mWrv5bBKHPciDvnOTCX6dzikDiWDDiymzedLsf9S8lMTc/ijgfoxat234L+ooQHLghw5 khdLsKsLVALTwQzkg9x3aqJKKG9SBTAmI5S7FSAmnHxv63d2ey+X10SxqHyAB8KogdO9 FumAaht8PN/0ax0VRS6BWVJbkUcHQBW+qJkY0VvbSvLyou2WtQuPCLORDZ6m1Cyht8IJ 4d3QpxmlU2GtQRVI2JLNzTyNArznS5Nh7M85A6BXSRlZt+C+Az68rosdGWnLi9mGmibj AQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1wxa1uhv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:00 +0000 Received: from m0353727.ppops.net (m0353727.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OItfwp018664; Mon, 24 Jul 2023 19:09:59 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1wxa1uh9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:59 +0000 Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OJ9RVY014365; Mon, 24 Jul 2023 19:09:58 GMT Received: from smtprelay04.dal12v.mail.ibm.com ([172.16.1.6]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3s0stxp1eq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:09:58 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay04.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJ9vvq19268120 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:09:57 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 49EB658067; Mon, 24 Jul 2023 19:09:57 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9103758055; Mon, 24 Jul 2023 19:09:52 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:09:52 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 05/13] mm: Add pud_same similar to __HAVE_ARCH_P4D_SAME Date: Tue, 25 Jul 2023 00:37:51 +0530 Message-ID: <20230724190759.483013-6-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: vCtvuxYQNZid0V10sHikjJ10jgDNJFgR X-Proofpoint-GUID: y1NIPwZaB4Pxl6lAIO3-bTdmXZChHrJD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 mlxscore=0 malwarescore=0 suspectscore=0 mlxlogscore=999 priorityscore=1501 impostorscore=0 lowpriorityscore=0 phishscore=0 bulkscore=0 adultscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspamd-Queue-Id: 70E4880022 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: hcqhiaia7563ygwhsufb67ut1gkta4q6 X-HE-Tag: 1690225812-199524 X-HE-Meta: U2FsdGVkX1/AKZCLdtrf1V5tWiFaBjwzE0tBRJOACTOujtIULeFptqaWQFTKHPPtNTnXX0sbr4JG5aGcBaYrQuaMRgIuX+OvmxJ+4d2Rl0lmGSX5wDB9rTcxH0R8xKvngcYYmAAxW8xkBxbT/iaTKkVbvutS5/RSBB1kFXN4mXkxs5fHayFmu+V+pw2YAxylR9mLXCtQy5BzGtznK6ru6ogWVLILUTie1GEbK6HblBX15C/QzwAAvaGgVuow+7ZtnJIfyrDvJvq0Wv/JnDCasWrOGDNf4i4+0mWMwLZD37B05PxXCDY2Dri/YmsWYnBGGWCzZJkJ/1EMiovy0r4Ay1s+UH9pjQh9+07BeGVq/mN+PXQFItMIrzVwVxzDpeFdnmWndqqSySHukUlClonvUrjYq+9saezSFtxQTanfMgp1MR4FEyx5i83YNtweN8F8YW9ksEeBBI3oUDest+CPBPJZoZKEOdys2p4M+sFKqs5Bk6+kyfNAVvwAquf+jcTGiDREAgKJSLHdmRr8SGc/tuFOsdMzb81N9NaZfl98s57yNwuXUstXlkp1daQ5sDBGfBXyiLpXTHGPmhGgE0zFtvUjXFjq0yCVbzrFycGzXWEgivMIjGQk1xZgJvutDobO2kjRhjY6kdMQ6VVb0SC5S3HTZ6bisTTDqSiS/uL0+bWPkFWrRRx39fLdDgYqR0uQrNri42/yGD3jo0Ac/BdFaJLORA9lNOaz3pbJZRvdqFQjAhhJWxF/e+REbyxm1ySljfll6crTvgD9EiAc01+lqeBIkIptrom5PS5hdLjxLnWhgCD4edMUO+7bDPCKtwFqB18RYHdmM19ryHH1hPqsLeCJ7/1+9EOBGf8jzBJhPbSeKtYg/PwkanrYgfPg44/ea9XL80LrwuAC1rifFo3+fDSRy+XxBHpuHkcOZBbawIkjsuCf9TinMm4VQIPf74dx54BcrmgIbsAVKDhiYb5 Ipru9oNM xjjXxLACyLS8/2FbcFpsNbAsyCRxMQIByBTGzf2QO3Gedma0hWELD8Wl5dDw9C1S0jmy2eHOvhtqH9F7V6/KQsM/QsNnvbZbQW0fvNylTFlGGueOsxs7MoU0X9ujEVCwUaJkLPrCLrxEDQO4WEEtwHdNkgQEMf94FNzd2yhGSLvdYh8ai2im6sWjlojXqyQwXOtSd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This helps architectures to override pmd_same and pud_same independently. Signed-off-by: Aneesh Kumar K.V --- include/linux/pgtable.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 124427ece520..0af8bc4ce258 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -699,11 +699,14 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) { return pmd_val(pmd_a) == pmd_val(pmd_b); } +#endif +#ifndef pud_same static inline int pud_same(pud_t pud_a, pud_t pud_b) { return pud_val(pud_a) == pud_val(pud_b); } +#define pud_same pud_same #endif #ifndef __HAVE_ARCH_P4D_SAME From patchwork Mon Jul 24 19:07:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 196C1EB64DD for ; Mon, 24 Jul 2023 19:10:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A70306B0074; Mon, 24 Jul 2023 15:10:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1EF56B007B; Mon, 24 Jul 2023 15:10:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C0188E0001; Mon, 24 Jul 2023 15:10:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7C4B46B0074 for ; Mon, 24 Jul 2023 15:10:27 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4A15E140B47 for ; Mon, 24 Jul 2023 19:10:27 +0000 (UTC) X-FDA: 81047446494.26.CFCA8A2 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf09.hostedemail.com (Postfix) with ESMTP id D4627140025 for ; Mon, 24 Jul 2023 19:10:24 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=eH2G3CVb; spf=pass (imf09.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=73e3fmXFNKjZYShJdnfcdB8KiMsBt9ayvli7p20XOWk=; b=vqjjGzVPe8F/zPergBUE1ZzJ6R91Sy/7zU4Ltho5VnRwmXGXH/tIPF3EaT2cFZnFkWckU4 QiIzqj0U0elyrqLEcU7sCIh8eCoiqoibxVQdRRGYJm4onzUi5pcQhJIC7mycqZt/7XemWo LPcCeseqZOWFpE8L55PXhJu226FpIj0= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=eH2G3CVb; spf=pass (imf09.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225824; a=rsa-sha256; cv=none; b=CsA8Cm2z23/DFxMKfQO9ADiJyGaaMtgG5q0elnBb2wuB7MJTmRedW1sudB8fW7aJq3qywE Dp0Kt4+3a6B6IQBGmGEux2bMzotdsLo+YGwBZiBm6oy9GI7nCXqWFhnXwbJLWefs2jYuoH 2B7mTM9YNjktG70RwSt8ObwgNn50F3Q= Received: from pps.filterd (m0353722.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OJ9FRe018483; Mon, 24 Jul 2023 19:10:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=73e3fmXFNKjZYShJdnfcdB8KiMsBt9ayvli7p20XOWk=; b=eH2G3CVbgWpVvCm1ZqKrserU2I8Bli/Q1MXvYNY60PyIYzlkQaC0+b7sdCBzEKZQqJvm i7OL947NoqIkLLbfbjqcWeMKkBGNnUjTTKS07NuMN6TkoFXm0LkoQqtzcrw1IAq0tv1U 4DUyhDpIWXL2YqChmh9Jjo7nOtDfiU5HcQBtYa4jRTNc23+KQhLDBNCeqZNVTKlwLCwN 5M9jQRSXQZOFBSztDdh/XCTOz2F4B9VNgKNRUYAlkhv3r+oIvlzH8YqvBkZ0T8UelOGI oA6LPYRTjajYrb+5u5jaJjex8AucYxXDqvJNseyfVNF1nwmAzYUz24hfnpaazWG/mtJ2 CQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1xh5h950-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:10 +0000 Received: from m0353722.ppops.net (m0353722.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OJ9mhH021558; Mon, 24 Jul 2023 19:10:10 GMT Received: from ppma12.dal12v.mail.ibm.com (dc.9e.1632.ip4.static.sl-reverse.com [50.22.158.220]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1xh5h94b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:10 +0000 Received: from pps.filterd (ppma12.dal12v.mail.ibm.com [127.0.0.1]) by ppma12.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OIg3QV026209; Mon, 24 Jul 2023 19:10:09 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([172.16.1.73]) by ppma12.dal12v.mail.ibm.com (PPS) with ESMTPS id 3s0serp60y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:09 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJA87548496966 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:10:08 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A621C58059; Mon, 24 Jul 2023 19:10:08 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5B6E958065; Mon, 24 Jul 2023 19:10:03 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:10:03 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 07/13] mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization Date: Tue, 25 Jul 2023 00:37:53 +0530 Message-ID: <20230724190759.483013-8-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: -aY_oPAJYWgxXf6kRS91Y2Nbx4awPOoL X-Proofpoint-GUID: 1CvGjzbNlvRdNGNOO04O5UiVC9OLMBcq X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 clxscore=1015 bulkscore=0 malwarescore=0 lowpriorityscore=0 suspectscore=0 spamscore=0 priorityscore=1501 adultscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspamd-Queue-Id: D4627140025 X-Rspam-User: X-Stat-Signature: khnc17misr7byabu86edhjo1siobx5zg X-Rspamd-Server: rspam01 X-HE-Tag: 1690225824-709200 X-HE-Meta: U2FsdGVkX1+8jzvgdrowm4aFYjd+gbCEoS6g6U+Khfqu2kwvYoMOBJx3Jd8g0El1xKBfG2MYjZikAuvBpmiNAwjt3hYU80kBqDY8+waFe4M6wmisQ1uQnDQWvRi50bFPLVwc7XUl/Xpq94M97YwFn+mq5Q6mR49mXpGFwTeBErtTktLGy8sPDJlO5NlXozC1ypbaUVG3/yU1vmDN+3uc6+bppnTeg80BSoe668MNS9shlct4L7/jHMLZq5b40PXm8PHC1u+Kc/7JRlYiutQd3Hjpfue+YoOJrwWgpzMcpLzJ+zFrFqnt8EmNVo5nrltZHVeDFe2EWcz7z3hqApe4prKSSb8Grb5SH4dTotgGHA246LOb+26jKJWfwAm9p9hCDypR9wjMRJsirN5dfB6LYPYqXv+CwFZon1Eu7ok4Q1VFaIBwTkYD0jq5vsUOHgQIoibupEg3lk3Lnh0hSaL06WbomrlxFhZudjKzQlq1ae5AOOd7d1CPPNflidmUVbzHC8i1p32rfGnlRcjUK2UOA1ocH8B+aXrhmzdscidqYwEiU40vyxZdNFsRl3eAMBcASj69jgJZ2vSAnFTysq4ppbI9KgsIObaAgAuPMhDox6dSGSqPFs09+m9oMARYJzB6APdixV0UijRQrDSmt8nY/tPYlY1Ueft5OCtj91w0hQi8+AFvgKqqDpWGotHkuOLNnRmeB4b8865sUyCnSInKJv+pxP7gfn5bLUybQNLJU4QHvwx9fIu+xMYAAUe3U+amGWj5BZMVNRhaU32rZxtm8Dv6LDtgZ5PPcmZcyBPI5q9vjx6qQI5bTETFbv9dlOS6vR/5CcaEHcr62PVr/sUfEqH0TNnvlCOnlIQ86EPrptVILqc8RO4sFjQS7A2GlTlmpcJOvW2w3KeRRUguBbPZcBC1MbFtukoPyxzkdkVQ8v4898yuVuXmTJgFscuCVn5FVPL2Z8T2Pkwk6LYvzzI GRmr4MJR bhTJZ2eFRt+JuNOYWppKrAyAv8xUW9OUk1weWOvtsoSYETfZjeWxINx7R80lxFNr2hhnsBBeHpIS7c3B8qF4zpp/QoI/MuFQmJfWj/i14Yjd4rFkGjU9sFqxmcAGoHR2GfyPgvdWlGm/bo7+Nf8aPwW8b0oofhdSw9tQK/0lsG0vuXrdZ4rglpDDkLXQz3eM3LABR84V1T0RnzjIQEgxoq+iG16/vAJywhdUnxNSK7fToevTgu8fjGYWf9LXCQWK/ChrQPBdnIGybkOY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Arm disabled hugetlb vmemmap optimization [1] because hugetlb vmemmap optimization includes an update of both the permissions (writeable to read-only) and the output address (pfn) of the vmemmap ptes. That is not supported without unmapping of pte(marking it invalid) by some architectures. With DAX vmemmap optimization we don't require such pte updates and architectures can enable DAX vmemmap optimization while having hugetlb vmemmap optimization disabled. Hence split DAX optimization support into a different config. s390, loongarch and riscv don't have devdax support. So the DAX config is not enabled for them. With this change, arm64 should be able to select DAX optimization [1] commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable HUGETLB_PAGE_OPTIMIZE_VMEMMAP") Signed-off-by: Aneesh Kumar K.V --- arch/loongarch/Kconfig | 2 +- arch/riscv/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/x86/Kconfig | 3 ++- fs/Kconfig | 2 +- include/linux/mm.h | 2 +- mm/Kconfig | 5 ++++- 7 files changed, 11 insertions(+), 7 deletions(-) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index e55511af4c77..537ca2a4005a 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -59,7 +59,7 @@ config LOONGARCH select ARCH_USE_QUEUED_SPINLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_LD_ORPHAN_WARN - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select BUILDTIME_TABLE_SORT select COMMON_CLK diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 4c07b9189c86..6943d34c1ec1 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -53,7 +53,7 @@ config RISCV select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT select ARCH_WANT_HUGE_PMD_SHARE if 64BIT select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 290b6f93b816..8ff6d1c21e38 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -127,7 +127,7 @@ config S390 select ARCH_WANTS_NO_INSTR select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2 select DMA_OPS if PCI diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7422db409770..78224aa76409 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -128,7 +128,8 @@ config X86 select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_LD_ORPHAN_WARN - select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORT diff --git a/fs/Kconfig b/fs/Kconfig index 19975b104bc3..f3be721bab6d 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -254,7 +254,7 @@ config HUGETLB_PAGE config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE - depends on ARCH_WANT_OPTIMIZE_VMEMMAP + depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON diff --git a/include/linux/mm.h b/include/linux/mm.h index fe6741539740..f8899bda941a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3650,7 +3650,7 @@ void vmemmap_free(unsigned long start, unsigned long end, #endif #define VMEMMAP_RESERVE_NR 2 -#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { diff --git a/mm/Kconfig b/mm/Kconfig index da681dda8af1..5fe49c030961 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -486,7 +486,10 @@ config SPARSEMEM_VMEMMAP # Select this config option from the architecture Kconfig, if it is preferred # to enable the feature of HugeTLB/dev_dax vmemmap optimization. # -config ARCH_WANT_OPTIMIZE_VMEMMAP +config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP + bool + +config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool config HAVE_MEMBLOCK_PHYS_MAP From patchwork Mon Jul 24 19:07:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBB9AC001B0 for ; Mon, 24 Jul 2023 19:10:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B5D16B007B; Mon, 24 Jul 2023 15:10:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 564638E0001; Mon, 24 Jul 2023 15:10:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 405346B007E; Mon, 24 Jul 2023 15:10:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 345616B007B for ; Mon, 24 Jul 2023 15:10:30 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0F3B51C9AB8 for ; Mon, 24 Jul 2023 19:10:30 +0000 (UTC) X-FDA: 81047446620.27.24F9381 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf22.hostedemail.com (Postfix) with ESMTP id 9F6A0C0013 for ; Mon, 24 Jul 2023 19:10:27 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Eg07Wzo+; spf=pass (imf22.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VNEBeAWzU/CT4iyd6EAoxgzBL6LW64vgyDe8qKmFgh0=; b=L6Z9DEwPsPDUKw1uZmIoJRpVhbH4nuVJRFtVYU4ApMcdL/ctSz2XnSwFrrJzOAggOKPs5v 4slchCb90uYPturzoKD1mJ5k/Yt/6izC5wl1TjUIcFUfPOGjRBUn90Bfjke8f/CPKRCqHQ p5mK3twzVKkOxisjsekMXAlwv5+ZiQE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225827; a=rsa-sha256; cv=none; b=ClUGRJTo/PHV5tAFkOzpQBix2VMzrW0yZlXevv9VAbDFvgHtOBzJbhod/Eg+MF72B00qmD BrUbAy0gyjGDrN8/+SwBZcHSNeNc5s/VtfgALKMgFwCvm/0h2rrlW55laj1DndmEPV8QRe lIcO6B9IeW9H3utKiYoj0MGdmqHQ0Rg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Eg07Wzo+; spf=pass (imf22.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353723.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OJ9Vks012368; Mon, 24 Jul 2023 19:10:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=VNEBeAWzU/CT4iyd6EAoxgzBL6LW64vgyDe8qKmFgh0=; b=Eg07Wzo++/WPSXNsInbAxeEDrUuN6D8+uwTP+2ehrAX8cwbotjGI1cy30M4tDh+N8G1B EkyUKwgPju2eJ42gclRm0m0FVEuk3iLbB1x4I/yk7M0IUyecoNn3yWihHK2pCDQ4uCs1 GLDQBaU0OBX5DI7sbHa/6fdZMJhLLcS02tnaMkuLhmCHmhcOhfNmPE+9B8qgZKgYbN+R YNb7+g0wxMnR0mstL88qyz8s4HZOBCQCHUxCMcIcSU47NxlvcyXkbpyquWeegBtQowrQ 4c11jLgGSHYExNEjZkPuVEQJFGurohUe3zLqeUdGsQt4vxi1OZiXyK7+YayiU2arKuOB EQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1vrvkssw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:15 +0000 Received: from m0353723.ppops.net (m0353723.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OJ9vue014568; Mon, 24 Jul 2023 19:10:15 GMT Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1vrvkssm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:15 +0000 Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OI0bji002375; Mon, 24 Jul 2023 19:10:14 GMT Received: from smtprelay01.wdc07v.mail.ibm.com ([172.16.1.68]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3s0txjnkna-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:14 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay01.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJAEnG31064600 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:10:14 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1E60158068; Mon, 24 Jul 2023 19:10:14 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4C0F558055; Mon, 24 Jul 2023 19:10:09 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:10:08 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 08/13] powerpc/mm/trace: Convert trace event to trace event class Date: Tue, 25 Jul 2023 00:37:54 +0530 Message-ID: <20230724190759.483013-9-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: tsAODs3yZXCKz5iwX_3juu_3x31vgKdm X-Proofpoint-GUID: MykmrjbFbcLZDV9QhlQqoR2j0HYK66AN X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 phishscore=0 mlxlogscore=999 clxscore=1015 spamscore=0 priorityscore=1501 impostorscore=0 lowpriorityscore=0 adultscore=0 suspectscore=0 malwarescore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspamd-Queue-Id: 9F6A0C0013 X-Rspam-User: X-Stat-Signature: m19r5bnbhxgrxesqnfrimrfzsgp6odik X-Rspamd-Server: rspam03 X-HE-Tag: 1690225827-954582 X-HE-Meta: U2FsdGVkX1/Qmm4kzxsegbvU21vfm1SwhG7j/bYVo36VwCCyTjrRX3xk+9Wxi3nkURtZl+xry7IXjmGH8VvY3FKLv+POP0PSBvFMfb/p6JDT3MKq6Mg6dDQcO2vVBicw+AzjDPn0UL4/kfGaXvus59iDcWqXnddI+KChz0ioTwXL4BPemCs06hm7+BKe9NG6w3Bzg7brkiWytCQfFnGcpNz/DmqHArGBh9MgVet7ON9ToYrpbamFVRmnRR99Pq9MdRP3Kwo9Wj95MZC7Q2el/ldadfUw6dqHQNscnRdbqM9wIFy7+lxPged+a5+JcwmOxR+Q7N0PeZV5vUPV3zM9BIId4Ktbk/Qy6+QPJC+cxX4Fz0D0QFBGXHh4ACjnTtWY/o8JnzXiV41PtnsUw9sS1P2BK2OMgkYcyd3YPpqFeW0LybeFFBGwS6DRhxSljFrSuvdfUFlttUzrSHEauczDhLizjLYoGMoZGii4TgDXKkXE4fXEVI3c1hbv3zN51Og75vC0oyQBGXt7W7rjEbs/IGl+Rvq+6WE/DgXtfOo1wiTSWT2BS6aJjC+Gx9zfNw6JihOdm+keLu00lDgswrLG0MB4j2ou4bc9lO45neV8CSvc0w2j2G6VD7J2ivCadihPaOv4kyOEkT7bNHbUvt2l4BhggL7vrksfAtTnjPsoYrzocN1q1gvZXd/FMFfiqaBiZ3cxQ6j0O9PGDOFsoFidzU7bw0QlqdMJQfe6Bsuj58oTUDt4ed9g7ESdBEp6nJipz/ZlGOmOBjFeHJJd+34q4r2aJf0CLB73WidGaq+Edbe5wMXmOQRq7dysPJqTrN6IFcJQHwR6JM0K2mCwp38NQ1e9/mGRUQOCXPnenvxvhT4McS3TP4AxJ448jYcTVs7YFqC0c6SeX+xc2sf67BvMmueAo7sQLhG3sAlS88Np8wSHCAqoqtbZdbMOz83kDO8GqsRaV1+MuMudE7tyMPc fuvB9Zy7 PgAhHm/5Lsbyw8MP+egBbOW5wFZkyMZxehNR7z4rA2FR5Q023r/PJQeGYlmjo3W4/KdO6liDPSzzmPlFfB82cGeu9m4zoYsVrECShw9j9pUkpexF4Q/8rbPAWHLR9U2XFEoevoxR5b/520njagTQjg4vUFMpXgNjEC6PMvNDfRjnQW33jqc4two8AaQEdTbVCCuzYukvV06hbOOg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A follow-up patch will add a pud variant for this same event. Using event class makes that addition simpler. No functional change in this patch. Reviewed-by: Christophe Leroy Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/mm/book3s64/hash_pgtable.c | 2 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 2 +- include/trace/events/thp.h | 23 ++++++++++++++++------- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c index 51f48984abca..988948d69bc1 100644 --- a/arch/powerpc/mm/book3s64/hash_pgtable.c +++ b/arch/powerpc/mm/book3s64/hash_pgtable.c @@ -214,7 +214,7 @@ unsigned long hash__pmd_hugepage_update(struct mm_struct *mm, unsigned long addr old = be64_to_cpu(old_be); - trace_hugepage_update(addr, old, clr, set); + trace_hugepage_update_pmd(addr, old, clr, set); if (old & H_PAGE_HASHPTE) hpte_do_hugepage_flush(mm, addr, pmdp, old); return old; diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index e7ea492ac510..02e185d2e4d6 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -962,7 +962,7 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add #endif old = radix__pte_update(mm, addr, pmdp_ptep(pmdp), clr, set, 1); - trace_hugepage_update(addr, old, clr, set); + trace_hugepage_update_pmd(addr, old, clr, set); return old; } diff --git a/include/trace/events/thp.h b/include/trace/events/thp.h index 202b3e3e67ff..a95c78b10561 100644 --- a/include/trace/events/thp.h +++ b/include/trace/events/thp.h @@ -8,25 +8,29 @@ #include #include -TRACE_EVENT(hugepage_set_pmd, +DECLARE_EVENT_CLASS(hugepage_set, - TP_PROTO(unsigned long addr, unsigned long pmd), - TP_ARGS(addr, pmd), + TP_PROTO(unsigned long addr, unsigned long pte), + TP_ARGS(addr, pte), TP_STRUCT__entry( __field(unsigned long, addr) - __field(unsigned long, pmd) + __field(unsigned long, pte) ), TP_fast_assign( __entry->addr = addr; - __entry->pmd = pmd; + __entry->pte = pte; ), - TP_printk("Set pmd with 0x%lx with 0x%lx", __entry->addr, __entry->pmd) + TP_printk("Set page table entry with 0x%lx with 0x%lx", __entry->addr, __entry->pte) ); +DEFINE_EVENT(hugepage_set, hugepage_set_pmd, + TP_PROTO(unsigned long addr, unsigned long pmd), + TP_ARGS(addr, pmd) +); -TRACE_EVENT(hugepage_update, +DECLARE_EVENT_CLASS(hugepage_update, TP_PROTO(unsigned long addr, unsigned long pte, unsigned long clr, unsigned long set), TP_ARGS(addr, pte, clr, set), @@ -48,6 +52,11 @@ TRACE_EVENT(hugepage_update, TP_printk("hugepage update at addr 0x%lx and pte = 0x%lx clr = 0x%lx, set = 0x%lx", __entry->addr, __entry->pte, __entry->clr, __entry->set) ); +DEFINE_EVENT(hugepage_update, hugepage_update_pmd, + TP_PROTO(unsigned long addr, unsigned long pmd, unsigned long clr, unsigned long set), + TP_ARGS(addr, pmd, clr, set) +); + DECLARE_EVENT_CLASS(migration_pmd, TP_PROTO(unsigned long addr, unsigned long pmd), From patchwork Mon Jul 24 19:07:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65B83EB64DD for ; Mon, 24 Jul 2023 19:10:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 05C986B007D; Mon, 24 Jul 2023 15:10:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00CAE6B007E; Mon, 24 Jul 2023 15:10:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF04E6B0080; Mon, 24 Jul 2023 15:10:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CB0CE6B007D for ; Mon, 24 Jul 2023 15:10:37 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 57092160B90 for ; Mon, 24 Jul 2023 19:10:37 +0000 (UTC) X-FDA: 81047446914.27.A17264C Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf13.hostedemail.com (Postfix) with ESMTP id DC6EE20027 for ; Mon, 24 Jul 2023 19:10:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=AMfGfKX+; spf=pass (imf13.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225835; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U9QgluxYiNw/1f7/52MDi6xL9U0M0bnrrLMKvcfIoKA=; b=KxaDjR7POmmDgDM9ixcoAVPYg+HuqP0gQSLWmx/CpFRCvq6zguzlwYRCZqT8GGLUpXxbiU vByB7cuHLk8y0GHuPiGr3js25HM0g6mze9kuTjAETRH9OiAoOCZMYrDpMOFYzVckJ5TDw+ odqJhIFBxlKIgOufY7XfAMfo+MsDNrw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=AMfGfKX+; spf=pass (imf13.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225835; a=rsa-sha256; cv=none; b=QU4fUM9CfcZ3hY1qoQY38wAgnbAI5lxoBvfNt6P41an4O//5Ppo4ja+SDbWl9DSyH9QzzX 8WkmWR2Ybgmz6elZ8iQLCdO0t6U8m+KM3POpqJPkVg4UdjWKvPEaJ0WPfDuGKdfPpwUIrt 2T20ZTuvpsc1m6HZnHFkeEU/dL4uR0A= Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OIuBTu018752; Mon, 24 Jul 2023 19:10:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=U9QgluxYiNw/1f7/52MDi6xL9U0M0bnrrLMKvcfIoKA=; b=AMfGfKX+0SEzV3Dt966MflP8GBC+Af5JiojR6n+A4/xxPr5xTZ2T7iHYlpcPflkrIJDh /d7jaPAsa26KiRgzKWYFbA2D1QIGBhJFYhBiyGYgbhiEiFxNKKX4oaOyPUAiJ2mPCrT6 LSmkJ5vMc4XsMIj2WDxXYtyxi0yJ/EpBMG7s7vpvoacvSwScQL4sJ80Ve9+95k6B1pQf X2HCBFQEoLgtE6zsZGI/UxCAr8ErkGDTi7TvcifAI1w8mFPXHOKK05iUFMceJrE8uBX4 NSgScmBkYHhU9UbHhdAuNQF7yEhHcI526qit2TJgdfeg6jY03jzyKB3awUgECWEwo7Zo Cg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1xcj97yk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:22 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OJ03OB032758; Mon, 24 Jul 2023 19:10:22 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1xcj97xy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:22 +0000 Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OH5WWt014370; Mon, 24 Jul 2023 19:10:20 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([172.16.1.72]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3s0stxp1kx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:20 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJAKBs60948976 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:10:20 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5AFE658063; Mon, 24 Jul 2023 19:10:20 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B455658055; Mon, 24 Jul 2023 19:10:14 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:10:14 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 09/13] powerpc/book3s64/mm: Enable transparent pud hugepage Date: Tue, 25 Jul 2023 00:37:55 +0530 Message-ID: <20230724190759.483013-10-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: qqfMufv328UFTdn9ZhRJqlxucMdCN-mL X-Proofpoint-GUID: G9KhVOjk67gR71syG8deK1cAausZAi00 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 suspectscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 spamscore=0 priorityscore=1501 phishscore=0 lowpriorityscore=0 impostorscore=0 bulkscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspamd-Queue-Id: DC6EE20027 X-Rspam-User: X-Stat-Signature: bt3ha5drfx9rbt5bft34nfrtntod3w9m X-Rspamd-Server: rspam01 X-HE-Tag: 1690225834-332287 X-HE-Meta: U2FsdGVkX1+xPez9NUGborkx2HTRCAXxVbtg3BMLY7Sh+raYBTSXa49g4qb6NjlXFornLRE6gOTopt0uGg51VvCOZKONsrGQD0dOTFkqy1aAUvSHKX72O6AxrEe/S4EpaoT1NJgvyGH3JLnbFvsvhFxanUF5iKCKKwg4sQy3aqFWEHufaee+WQ6zVuzA7s0nACcYkINLbipXtEdLSSfLs2syJk/kDOfLDV/Ov9zS+MMs5pSOUbC6vKSW4ZPE6r8Oo9AU3Kt6H8qjJguVGxmPwSbWAt0qstvOwJ4E17tq7e50TqfHyLLCqbQXo6LylYY9T7tYfnLOZg3GVF72x2DQs+wc8TzzhIxtDTX3Kccv9b0bddWs+NPrUnmsnVLKimNzu6fiIl59Jag+UOoA1AG98DpSpLap2qXzZCXAe2VHbsOIQxP0tw+RfXqQe5r+30XNqbERaHohC9Id/5WmOPYQPT96OalfwVaEOh6wVcabvyuEL/TGhWjc68qe1yDDb7ZxTaVwxc0M6c2LCX/M/C1YqssJW1g+pZ/mSGS5fCOdoPAjk4XVtlBoVssF5AhA0z2mK5bHP1A6uVbnuBAkNr9gB0YkLDzE43vPGi+XpwrbIIP9QHJdvNN2bMIW0QWqKM8HIzWFHNawBIx8y0V6aMfSi6TWo/UXNbPsyBvFCZzoVTEnEbcof/gGEs8gMcrSDXvFKPn34axfW043SdSAr04fmx3f1W+glTjDkKgTVkiC8EGsZa+BHtZpSFrCs4N3WF0vlMiSOhvcBblS19vSLgiWeMviQCkvX53lUdg19G1e0tTbN42Mg9K5Z1BthDcxyRjxbFTqhOfY89dp/blwrTMVnLmZDQk//6ck56ZtBOwylMd4F9H3j09h/wJ7uktW8hKO1+MID45Y2T4+t65/YQ3306jU5BVEO40s1Lnin9sFTZy05PFBTnPbm5WuIu3+Tf6wugwSzIvcPT6UoZ3HN25 mJGjZw7a zQstYaNnDZE09h9P0KFv2mdlWkP6nAlAzr+99JGkTpwfFhYyzBvz4YjiwqxRHloeqMDZcgLniz2IS/D//UXi3+8bMrgSoG2eI4jNHBnNzz3dmbFkq9t0XPn2wqAHKbH0IZh5j0ddVCCfrURE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is enabled only with radix translation and 1G hugepage size. This will be used with devdax device memory with a namespace alignment of 1G. Anon transparent hugepage is not supported even though we do have helpers checking pud_trans_huge(). We should never find that return true. The only expected pte bit combination is _PAGE_PTE | _PAGE_DEVMAP. Some of the helpers are never expected to get called on hash translation and hence is marked to call BUG() in such a case. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/book3s/64/hash.h | 9 + arch/powerpc/include/asm/book3s/64/pgtable.h | 155 ++++++++++++++++-- arch/powerpc/include/asm/book3s/64/radix.h | 36 ++++ .../include/asm/book3s/64/tlbflush-radix.h | 2 + arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 + arch/powerpc/mm/book3s64/pgtable.c | 78 +++++++++ arch/powerpc/mm/book3s64/radix_pgtable.c | 28 ++++ arch/powerpc/mm/book3s64/radix_tlb.c | 7 + arch/powerpc/platforms/Kconfig.cputype | 1 + include/trace/events/thp.h | 10 ++ 10 files changed, 323 insertions(+), 11 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/hash.h b/arch/powerpc/include/asm/book3s/64/hash.h index 17e7a778c856..efce6ef3e2a9 100644 --- a/arch/powerpc/include/asm/book3s/64/hash.h +++ b/arch/powerpc/include/asm/book3s/64/hash.h @@ -133,7 +133,16 @@ static inline int get_region_id(unsigned long ea) } #define hash__pmd_bad(pmd) (pmd_val(pmd) & H_PMD_BAD_BITS) + +/* + * pud comparison that will work with both pte and page table pointer. + */ +static inline int hash__pud_same(pud_t pud_a, pud_t pud_b) +{ + return (((pud_raw(pud_a) ^ pud_raw(pud_b)) & ~cpu_to_be64(_PAGE_HPTEFLAGS)) == 0); +} #define hash__pud_bad(pud) (pud_val(pud) & H_PUD_BAD_BITS) + static inline int hash__p4d_bad(p4d_t p4d) { return (p4d_val(p4d) == 0); diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 4acc9690f599..a8204566cfd0 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -921,8 +921,29 @@ static inline pud_t pte_pud(pte_t pte) { return __pud_raw(pte_raw(pte)); } + +static inline pte_t *pudp_ptep(pud_t *pud) +{ + return (pte_t *)pud; +} + +#define pud_pfn(pud) pte_pfn(pud_pte(pud)) +#define pud_dirty(pud) pte_dirty(pud_pte(pud)) +#define pud_young(pud) pte_young(pud_pte(pud)) +#define pud_mkold(pud) pte_pud(pte_mkold(pud_pte(pud))) +#define pud_wrprotect(pud) pte_pud(pte_wrprotect(pud_pte(pud))) +#define pud_mkdirty(pud) pte_pud(pte_mkdirty(pud_pte(pud))) +#define pud_mkclean(pud) pte_pud(pte_mkclean(pud_pte(pud))) +#define pud_mkyoung(pud) pte_pud(pte_mkyoung(pud_pte(pud))) +#define pud_mkwrite(pud) pte_pud(pte_mkwrite(pud_pte(pud))) #define pud_write(pud) pte_write(pud_pte(pud)) +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY +#define pud_soft_dirty(pmd) pte_soft_dirty(pud_pte(pud)) +#define pud_mksoft_dirty(pmd) pte_pud(pte_mksoft_dirty(pud_pte(pud))) +#define pud_clear_soft_dirty(pmd) pte_pud(pte_clear_soft_dirty(pud_pte(pud))) +#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ + static inline int pud_bad(pud_t pud) { if (radix_enabled()) @@ -1115,15 +1136,24 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write) #ifdef CONFIG_TRANSPARENT_HUGEPAGE extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); +extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot); extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); +extern void set_pud_at(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, pud_t pud); + static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) { } +static inline void update_mmu_cache_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pud) +{ +} + extern int hash__has_transparent_hugepage(void); static inline int has_transparent_hugepage(void) { @@ -1133,6 +1163,14 @@ static inline int has_transparent_hugepage(void) } #define has_transparent_hugepage has_transparent_hugepage +static inline int has_transparent_pud_hugepage(void) +{ + if (radix_enabled()) + return radix__has_transparent_pud_hugepage(); + return 0; +} +#define has_transparent_pud_hugepage has_transparent_pud_hugepage + static inline unsigned long pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, unsigned long clr, unsigned long set) @@ -1142,6 +1180,16 @@ pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, return hash__pmd_hugepage_update(mm, addr, pmdp, clr, set); } +static inline unsigned long +pud_hugepage_update(struct mm_struct *mm, unsigned long addr, pud_t *pudp, + unsigned long clr, unsigned long set) +{ + if (radix_enabled()) + return radix__pud_hugepage_update(mm, addr, pudp, clr, set); + BUG(); + return pud_val(*pudp); +} + /* * returns true for pmd migration entries, THP, devmap, hugetlb * But compile time dependent on THP config @@ -1151,6 +1199,11 @@ static inline int pmd_large(pmd_t pmd) return !!(pmd_raw(pmd) & cpu_to_be64(_PAGE_PTE)); } +static inline int pud_large(pud_t pud) +{ + return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE)); +} + /* * For radix we should always find H_PAGE_HASHPTE zero. Hence * the below will work for radix too @@ -1166,6 +1219,17 @@ static inline int __pmdp_test_and_clear_young(struct mm_struct *mm, return ((old & _PAGE_ACCESSED) != 0); } +static inline int __pudp_test_and_clear_young(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + unsigned long old; + + if ((pud_raw(*pudp) & cpu_to_be64(_PAGE_ACCESSED | H_PAGE_HASHPTE)) == 0) + return 0; + old = pud_hugepage_update(mm, addr, pudp, _PAGE_ACCESSED, 0); + return ((old & _PAGE_ACCESSED) != 0); +} + #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) @@ -1174,6 +1238,14 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_hugepage_update(mm, addr, pmdp, _PAGE_WRITE, 0); } +#define __HAVE_ARCH_PUDP_SET_WRPROTECT +static inline void pudp_set_wrprotect(struct mm_struct *mm, unsigned long addr, + pud_t *pudp) +{ + if (pud_write(*pudp)) + pud_hugepage_update(mm, addr, pudp, _PAGE_WRITE, 0); +} + /* * Only returns true for a THP. False for pmd migration entry. * We also need to return true when we come across a pte that @@ -1195,6 +1267,17 @@ static inline int pmd_trans_huge(pmd_t pmd) return hash__pmd_trans_huge(pmd); } +static inline int pud_trans_huge(pud_t pud) +{ + if (!pud_present(pud)) + return false; + + if (radix_enabled()) + return radix__pud_trans_huge(pud); + return 0; +} + + #define __HAVE_ARCH_PMD_SAME static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) { @@ -1203,6 +1286,15 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) return hash__pmd_same(pmd_a, pmd_b); } +#define pud_same pud_same +static inline int pud_same(pud_t pud_a, pud_t pud_b) +{ + if (radix_enabled()) + return radix__pud_same(pud_a, pud_b); + return hash__pud_same(pud_a, pud_b); +} + + static inline pmd_t __pmd_mkhuge(pmd_t pmd) { if (radix_enabled()) @@ -1210,6 +1302,14 @@ static inline pmd_t __pmd_mkhuge(pmd_t pmd) return hash__pmd_mkhuge(pmd); } +static inline pud_t __pud_mkhuge(pud_t pud) +{ + if (radix_enabled()) + return radix__pud_mkhuge(pud); + BUG(); + return pud; +} + /* * pfn_pmd return a pmd_t that can be used as pmd pte entry. */ @@ -1225,14 +1325,34 @@ static inline pmd_t pmd_mkhuge(pmd_t pmd) return pmd; } +static inline pud_t pud_mkhuge(pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM + if (radix_enabled()) + WARN_ON((pud_raw(pud) & cpu_to_be64(_PAGE_PTE)) == 0); + else + WARN_ON(1); +#endif + return pud; +} + + #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS extern int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty); +#define __HAVE_ARCH_PUDP_SET_ACCESS_FLAGS +extern int pudp_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, + pud_t entry, int dirty); #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG extern int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +#define __HAVE_ARCH_PUDP_TEST_AND_CLEAR_YOUNG +extern int pudp_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp); + #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, @@ -1243,6 +1363,16 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, return hash__pmdp_huge_get_and_clear(mm, addr, pmdp); } +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR +static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + if (radix_enabled()) + return radix__pudp_huge_get_and_clear(mm, addr, pudp); + BUG(); + return *pudp; +} + static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { @@ -1257,6 +1387,11 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full); +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL +pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma, + unsigned long addr, + pud_t *pudp, int full); + #define __HAVE_ARCH_PGTABLE_DEPOSIT static inline void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, pgtable_t pgtable) @@ -1305,6 +1440,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return hash__pmd_mkdevmap(pmd); } +static inline pud_t pud_mkdevmap(pud_t pud) +{ + if (radix_enabled()) + return radix__pud_mkdevmap(pud); + BUG(); + return pud; +} + static inline int pmd_devmap(pmd_t pmd) { return pte_devmap(pmd_pte(pmd)); @@ -1312,7 +1455,7 @@ static inline int pmd_devmap(pmd_t pmd) static inline int pud_devmap(pud_t pud) { - return 0; + return pte_devmap(pud_pte(pud)); } static inline int pgd_devmap(pgd_t pgd) @@ -1321,16 +1464,6 @@ static inline int pgd_devmap(pgd_t pgd) } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline int pud_pfn(pud_t pud) -{ - /* - * Currently all calls to pud_pfn() are gated around a pud_devmap() - * check so this should never be used. If it grows another user we - * want to know about it. - */ - BUILD_BUG(); - return 0; -} #define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION pte_t ptep_modify_prot_start(struct vm_area_struct *, unsigned long, pte_t *); void ptep_modify_prot_commit(struct vm_area_struct *, unsigned long, diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 686001eda936..2ef92f36340f 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -250,6 +250,10 @@ static inline int radix__pud_bad(pud_t pud) return !!(pud_val(pud) & RADIX_PUD_BAD_BITS); } +static inline int radix__pud_same(pud_t pud_a, pud_t pud_b) +{ + return ((pud_raw(pud_a) ^ pud_raw(pud_b)) == 0); +} static inline int radix__p4d_bad(p4d_t p4d) { @@ -268,9 +272,22 @@ static inline pmd_t radix__pmd_mkhuge(pmd_t pmd) return __pmd(pmd_val(pmd) | _PAGE_PTE); } +static inline int radix__pud_trans_huge(pud_t pud) +{ + return (pud_val(pud) & (_PAGE_PTE | _PAGE_DEVMAP)) == _PAGE_PTE; +} + +static inline pud_t radix__pud_mkhuge(pud_t pud) +{ + return __pud(pud_val(pud) | _PAGE_PTE); +} + extern unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, unsigned long clr, unsigned long set); +extern unsigned long radix__pud_hugepage_update(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, unsigned long clr, + unsigned long set); extern pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, @@ -278,6 +295,9 @@ extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, extern pgtable_t radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp); extern pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp); +pud_t radix__pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp); + static inline int radix__has_transparent_hugepage(void) { /* For radix 2M at PMD level means thp */ @@ -285,6 +305,14 @@ static inline int radix__has_transparent_hugepage(void) return 1; return 0; } + +static inline int radix__has_transparent_pud_hugepage(void) +{ + /* For radix 1G at PUD level means pud hugepage support */ + if (mmu_psize_defs[MMU_PAGE_1G].shift == PUD_SHIFT) + return 1; + return 0; +} #endif static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd) @@ -292,9 +320,17 @@ static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd) return __pmd(pmd_val(pmd) | (_PAGE_PTE | _PAGE_DEVMAP)); } +static inline pud_t radix__pud_mkdevmap(pud_t pud) +{ + return __pud(pud_val(pud) | (_PAGE_PTE | _PAGE_DEVMAP)); +} + +struct vmem_altmap; extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long page_size, unsigned long phys); +int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap); extern void radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size); diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 77797a2a82eb..a38542259fab 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -68,6 +68,8 @@ void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize); extern void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +extern void radix__flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end); extern void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); extern void radix__flush_tlb_kernel_range(unsigned long start, unsigned long end); diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index dca0477b0709..1950c1b825b4 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -51,6 +51,14 @@ static inline void flush_pmd_tlb_range(struct vm_area_struct *vma, radix__flush_pmd_tlb_range(vma, start, end); } +#define __HAVE_ARCH_FLUSH_PUD_TLB_RANGE +static inline void flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + if (radix_enabled()) + radix__flush_pud_tlb_range(vma, start, end); +} + #define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long start, diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 85c84e89e3ea..75b938268b04 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -64,11 +64,39 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, return changed; } +int pudp_set_access_flags(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp, pud_t entry, int dirty) +{ + int changed; +#ifdef CONFIG_DEBUG_VM + WARN_ON(!pud_devmap(*pudp)); + assert_spin_locked(pud_lockptr(vma->vm_mm, pudp)); +#endif + changed = !pud_same(*(pudp), entry); + if (changed) { + /* + * We can use MMU_PAGE_1G here, because only radix + * path look at the psize. + */ + __ptep_set_access_flags(vma, pudp_ptep(pudp), + pud_pte(entry), address, MMU_PAGE_1G); + } + return changed; +} + + int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { return __pmdp_test_and_clear_young(vma->vm_mm, address, pmdp); } + +int pudp_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp) +{ + return __pudp_test_and_clear_young(vma->vm_mm, address, pudp); +} + /* * set a new huge pmd. We should not be called for updating * an existing pmd entry. That should go via pmd_hugepage_update. @@ -90,6 +118,23 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr, return set_pte_at(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd)); } +void set_pud_at(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM + /* + * Make sure hardware valid bit is not set. We don't do + * tlb flush for this update. + */ + + WARN_ON(pte_hw_valid(pud_pte(*pudp))); + assert_spin_locked(pud_lockptr(mm, pudp)); + WARN_ON(!(pud_large(pud))); +#endif + trace_hugepage_set_pud(addr, pud_val(pud)); + return set_pte_at(mm, addr, pudp_ptep(pudp), pud_pte(pud)); +} + static void do_serialize(void *arg) { /* We've taken the IPI, so try to trim the mask while here */ @@ -147,11 +192,35 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, return pmd; } +pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, int full) +{ + pud_t pud; + + VM_BUG_ON(addr & ~HPAGE_PMD_MASK); + VM_BUG_ON((pud_present(*pudp) && !pud_devmap(*pudp)) || + !pud_present(*pudp)); + pud = pudp_huge_get_and_clear(vma->vm_mm, addr, pudp); + /* + * if it not a fullmm flush, then we can possibly end up converting + * this PMD pte entry to a regular level 0 PTE by a parallel page fault. + * Make sure we flush the tlb in this case. + */ + if (!full) + flush_pud_tlb_range(vma, addr, addr + HPAGE_PUD_SIZE); + return pud; +} + static pmd_t pmd_set_protbits(pmd_t pmd, pgprot_t pgprot) { return __pmd(pmd_val(pmd) | pgprot_val(pgprot)); } +static pud_t pud_set_protbits(pud_t pud, pgprot_t pgprot) +{ + return __pud(pud_val(pud) | pgprot_val(pgprot)); +} + /* * At some point we should be able to get rid of * pmd_mkhuge() and mk_huge_pmd() when we update all the @@ -166,6 +235,15 @@ pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot) return __pmd_mkhuge(pmd_set_protbits(__pmd(pmdv), pgprot)); } +pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot) +{ + unsigned long pudv; + + pudv = (pfn << PAGE_SHIFT) & PTE_RPN_MASK; + + return __pud_mkhuge(pud_set_protbits(__pud(pudv), pgprot)); +} + pmd_t mk_pmd(struct page *page, pgprot_t pgprot) { return pfn_pmd(page_to_pfn(page), pgprot); diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 02e185d2e4d6..227fea53c217 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -967,6 +967,23 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add return old; } +unsigned long radix__pud_hugepage_update(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, unsigned long clr, + unsigned long set) +{ + unsigned long old; + +#ifdef CONFIG_DEBUG_VM + WARN_ON(!pud_devmap(*pudp)); + assert_spin_locked(pud_lockptr(mm, pudp)); +#endif + + old = radix__pte_update(mm, addr, pudp_ptep(pudp), clr, set, 1); + trace_hugepage_update_pud(addr, old, clr, set); + + return old; +} + pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) @@ -1043,6 +1060,17 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm, return old_pmd; } +pud_t radix__pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + pud_t old_pud; + unsigned long old; + + old = radix__pud_hugepage_update(mm, addr, pudp, ~0UL, 0); + old_pud = __pud(old); + return old_pud; +} + #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 64c11a4ac34a..7ce9a4b35b80 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -1467,6 +1467,13 @@ void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, } EXPORT_SYMBOL(radix__flush_pmd_tlb_range); +void radix__flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + radix__flush_tlb_range_psize(vma->vm_mm, start, end, MMU_PAGE_1G); +} +EXPORT_SYMBOL(radix__flush_pud_tlb_range); + void radix__flush_tlb_all(void) { unsigned long rb,prs,r,rs; diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index 45fd975ef521..340b86ef7284 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -94,6 +94,7 @@ config PPC_BOOK3S_64 select PPC_FPU select PPC_HAVE_PMU_SUPPORT select HAVE_ARCH_TRANSPARENT_HUGEPAGE + select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE diff --git a/include/trace/events/thp.h b/include/trace/events/thp.h index a95c78b10561..f50048af5fcc 100644 --- a/include/trace/events/thp.h +++ b/include/trace/events/thp.h @@ -30,6 +30,11 @@ DEFINE_EVENT(hugepage_set, hugepage_set_pmd, TP_ARGS(addr, pmd) ); +DEFINE_EVENT(hugepage_set, hugepage_set_pud, + TP_PROTO(unsigned long addr, unsigned long pud), + TP_ARGS(addr, pud) +); + DECLARE_EVENT_CLASS(hugepage_update, TP_PROTO(unsigned long addr, unsigned long pte, unsigned long clr, unsigned long set), @@ -57,6 +62,11 @@ DEFINE_EVENT(hugepage_update, hugepage_update_pmd, TP_ARGS(addr, pmd, clr, set) ); +DEFINE_EVENT(hugepage_update, hugepage_update_pud, + TP_PROTO(unsigned long addr, unsigned long pud, unsigned long clr, unsigned long set), + TP_ARGS(addr, pud, clr, set) +); + DECLARE_EVENT_CLASS(migration_pmd, TP_PROTO(unsigned long addr, unsigned long pmd), From patchwork Mon Jul 24 19:07:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD4C8EB64DD for ; Mon, 24 Jul 2023 19:10:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 677EE6B0075; Mon, 24 Jul 2023 15:10:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6274F6B007E; Mon, 24 Jul 2023 15:10:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A1ED8E0001; Mon, 24 Jul 2023 15:10:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 389B16B0075 for ; Mon, 24 Jul 2023 15:10:42 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0AFFBA02A0 for ; Mon, 24 Jul 2023 19:10:42 +0000 (UTC) X-FDA: 81047447124.08.A93B6B2 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf07.hostedemail.com (Postfix) with ESMTP id 7E40140005 for ; Mon, 24 Jul 2023 19:10:39 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=dWjwtgHi; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf07.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225839; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6CDRyovNiGKkC9v1XeX1MKk9uyt80i4nHLnrRc4uvhc=; b=J3spSbofbdp7xbgMWX5ZgmoRYkGEdSf71qLmDvMHQvy+dLP2d7UvAWR9JEk7FRa7xhO9KN yQ82ZgYxEJG7+bqOrSNAW/H/9LMCCvMdeOhxebAo11AXgEM/adm4xO/GmNv2izDrd+R4J0 FED5Xu9lm0/ty/azJn1B1IJq2xcV7oM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=dWjwtgHi; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf07.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225839; a=rsa-sha256; cv=none; b=8mo5eF8F1efe5o/3KVjV1D3UDdJpHYSXJe8MkKrbwlwOwOosvT66qYhQQ57FLBvLeACoOZ Wsvtx/KXLRHbRfVAr0KcO7BnXwUfImGfDJiCmyeQ3yHKhtnPcV8h8F5JH1+kMHnE4My3Ni OiE6J76Ot4pXi78r/rxywRa3PPeqaCA= Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OI3YBX021904; Mon, 24 Jul 2023 19:10:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=6CDRyovNiGKkC9v1XeX1MKk9uyt80i4nHLnrRc4uvhc=; b=dWjwtgHiXEUty8hEM7cj5Cu9DUFs+PgW4EwuB5CAhUkA9QRXattkMmqE0jthqIN3uFQI Su2cjzu9PRSzMKCBvmVzro9+cSE1aqbz1gIBEdLEoBGIqH0+esYQegZwkpWxsIY9RGcW X0TXIKo3bmY1q2Bq1D+4YrDSBg8sOSrJsrgRV3lSXI5AVyVLgxU/hOF/tCEor/JUdrih Qyw043F9b8AVY7k4ZhuuRQUWBvv6TAwlrYNcUIxXXK0LgiSlUjNMTGuSL82KX9PyB6eS jGEAne9Dy9kczEAVxvatGIeD/1kDu54J60LgBu+Jgj1BK6WmEXOyRJve00BoQluic/9c 7Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1w6d37r0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:28 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OJ6xHB002374; Mon, 24 Jul 2023 19:10:27 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1w6d37qk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:27 +0000 Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OJ1Vcc016574; Mon, 24 Jul 2023 19:10:26 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([172.16.1.8]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 3s0v50w617-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:26 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJAPDO1245712 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:10:26 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CD41558059; Mon, 24 Jul 2023 19:10:25 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EC00558055; Mon, 24 Jul 2023 19:10:20 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:10:20 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 10/13] powerpc/book3s64/vmemmap: Switch radix to use a different vmemmap handling function Date: Tue, 25 Jul 2023 00:37:56 +0530 Message-ID: <20230724190759.483013-11-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: fBtXRgZ3f5GT2A6kPEEKBG1Z8g1iWTsM X-Proofpoint-GUID: ywU2NkuoQgSuwBuSRPvO4GwaXSC8_Ar0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 malwarescore=0 adultscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 spamscore=0 mlxlogscore=999 lowpriorityscore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7E40140005 X-Stat-Signature: gwitsngbur54hnht89d43dznkttpz57t X-HE-Tag: 1690225839-4198 X-HE-Meta: U2FsdGVkX1/P8sjRqwcPCV27cyh5RzVFv5gwpADUd2z+X36dXnOjVdfpepo5ABtMrURqxqYjXAfTQYkepcCkA6/OsHeAxYnHMQk+qeUAgDwB0sP5txiY1hkeC7IrV6T2BB92l4s1lUIyKrpD8cUJhYuFbwuMiGXperyNQU8CLBkErCxrkepfAprl3ip+zeCACfdrCu/fanQ1pO28nsbT+h9p2l0tsFho9QRw9sbYgDhqu0rFYTJaX4KO3EJN3g6/0aNTyYbZfXfLNUq1zn4R0kssCDOipALkxfRVLad5OYDbJoYAMZFzdFQsadwuloDs81/pQGkDaG8KpEfaDgN2GGxhRCPV3cdZyzFjvvmAMgq4Lw64qfupi7NdLyMwgrpYY9I7vvWP47CcuVj3y/67VFM2PGQPrJz/+N4+LzVM1LBJ2hBmVNups/GobLk/LfPbu1Psdij2TIMK0BYnQHi+PkqcrmZByO8hnlXaB2OM64vXgFdVvI5Y/k9UcxK9eI/t/UkNYGykqBO8E00yoqRYW+fLbEWjXaNfpmgGxiBlhpl2qO5NWETDGbII6NJ2LTriTKMWkGa2Sf5255R3Ws2Yun170bM/CiNRj91hWgjocoZ05Huc1ypTQIJ7nCfdWuy6oXXmcytg6P5PXCs8X82u8QccbFFo5vfnl+YJbnFBvdFIalc8OR/UKbKYKMsObfJQ2J8Z4JOwe0VVnmfzX5waeICvcry69Dwsfey8pwyq+gY778e4U/lGoGR9RzbzM6inCd8YR3wOU7C9QSuhrCLWL+8GHxnUzMlxA+QuZhSGbzUYD+/R/+70mhOlYjMe2JqRk/IKrK72XgU9DJ3JEhy2FG+ojcsoYwJCa9aNLHRsA4UamOfK5yifc59BhTEKdUn+paKehDooJtIntz04R9C7+hU2i2sXvjLKO7a0L4GHzcg7l+LHmieAbLSnVd2sx+rOL5RxJj9dP01xyqnXXix qZmJsP8Z 4/UdS75Hkc96qEjPqrVqBTyDEFJqUuBW7QSXnpZVJ1Wo7yMIcMQ+nlTrZnuo4E9ZrXF92583Nx/H/jt9nkF89e2KwPCG+ZPAoFKKj4aEku6/UeGDQKaaP0WArcVOgWtALvZ9esm/Y9VBY9CNkRLFIyqtijk4L/ko7fRMfczBtKbbsD09pWmUAUmaaR1yGEWpt+NxBxzEpeG0RI4guj9o+iyjGiaZ+q2lfefb7GzaDe+7z19E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is in preparation to update radix to implement vmemmap optimization for devdax. Below are the rules w.r.t radix vmemmap mapping 1. First try to map things using PMD (2M) 2. With altmap if altmap cross-boundary check returns true, fall back to PAGE_SIZE 3. If we can't allocate PMD_SIZE backing memory for vmemmap, fallback to PAGE_SIZE On removing vmemmap mapping, check if every subsection that is using the vmemmap area is invalid. If found to be invalid, that implies we can safely free the vmemmap area. We don't use the PAGE_UNUSED pattern used by x86 because with 64K page size, we need to do the above check even at the PAGE_SIZE granularity. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/book3s/64/radix.h | 2 + arch/powerpc/include/asm/pgtable.h | 6 + arch/powerpc/mm/book3s64/radix_pgtable.c | 325 +++++++++++++++++++-- arch/powerpc/mm/init_64.c | 26 +- 4 files changed, 328 insertions(+), 31 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 2ef92f36340f..f1461289643a 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -331,6 +331,8 @@ extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long phys); int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); +void __ref radix__vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap); extern void radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 445a22987aa3..a4893b17705a 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -157,6 +157,12 @@ static inline pgtable_t pmd_pgtable(pmd_t pmd) return (pgtable_t)pmd_page_vaddr(pmd); } +#ifdef CONFIG_PPC64 +int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size); +bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, + unsigned long page_size); +#endif /* CONFIG_PPC64 */ + #endif /* __ASSEMBLY__ */ #endif /* _ASM_POWERPC_PGTABLE_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 227fea53c217..53f8340e390c 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -744,8 +744,58 @@ static void free_pud_table(pud_t *pud_start, p4d_t *p4d) p4d_clear(p4d); } +#ifdef CONFIG_SPARSEMEM_VMEMMAP +static bool __meminit vmemmap_pmd_is_unused(unsigned long addr, unsigned long end) +{ + unsigned long start = ALIGN_DOWN(addr, PMD_SIZE); + + return !vmemmap_populated(start, PMD_SIZE); +} + +static bool __meminit vmemmap_page_is_unused(unsigned long addr, unsigned long end) +{ + unsigned long start = ALIGN_DOWN(addr, PAGE_SIZE); + + return !vmemmap_populated(start, PAGE_SIZE); + +} +#endif + +static void __meminit free_vmemmap_pages(struct page *page, + struct vmem_altmap *altmap, + int order) +{ + unsigned int nr_pages = 1 << order; + + if (altmap) { + unsigned long alt_start, alt_end; + unsigned long base_pfn = page_to_pfn(page); + + /* + * with 2M vmemmap mmaping we can have things setup + * such that even though atlmap is specified we never + * used altmap. + */ + alt_start = altmap->base_pfn; + alt_end = altmap->base_pfn + altmap->reserve + altmap->free; + + if (base_pfn >= alt_start && base_pfn < alt_end) { + vmem_altmap_free(altmap, nr_pages); + return; + } + } + + if (PageReserved(page)) { + /* allocated from memblock */ + while (nr_pages--) + free_reserved_page(page++); + } else + free_pages((unsigned long)page_address(page), order); +} + static void remove_pte_table(pte_t *pte_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pte_t *pte; @@ -759,24 +809,26 @@ static void remove_pte_table(pte_t *pte_start, unsigned long addr, if (!pte_present(*pte)) continue; - if (!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(next)) { - /* - * The vmemmap_free() and remove_section_mapping() - * codepaths call us with aligned addresses. - */ - WARN_ONCE(1, "%s: unaligned range\n", __func__); - continue; + if (PAGE_ALIGNED(addr) && PAGE_ALIGNED(next)) { + if (!direct) + free_vmemmap_pages(pte_page(*pte), altmap, 0); + pte_clear(&init_mm, addr, pte); + pages++; } - - pte_clear(&init_mm, addr, pte); - pages++; +#ifdef CONFIG_SPARSEMEM_VMEMMAP + else if (!direct && vmemmap_page_is_unused(addr, next)) { + free_vmemmap_pages(pte_page(*pte), altmap, 0); + pte_clear(&init_mm, addr, pte); + } +#endif } if (direct) update_page_count(mmu_virtual_psize, -pages); } static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pte_t *pte_base; @@ -790,18 +842,24 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, continue; if (pmd_is_leaf(*pmd)) { - if (!IS_ALIGNED(addr, PMD_SIZE) || - !IS_ALIGNED(next, PMD_SIZE)) { - WARN_ONCE(1, "%s: unaligned range\n", __func__); - continue; + if (IS_ALIGNED(addr, PMD_SIZE) && + IS_ALIGNED(next, PMD_SIZE)) { + if (!direct) + free_vmemmap_pages(pmd_page(*pmd), altmap, get_order(PMD_SIZE)); + pte_clear(&init_mm, addr, (pte_t *)pmd); + pages++; } - pte_clear(&init_mm, addr, (pte_t *)pmd); - pages++; +#ifdef CONFIG_SPARSEMEM_VMEMMAP + else if (!direct && vmemmap_pmd_is_unused(addr, next)) { + free_vmemmap_pages(pmd_page(*pmd), altmap, get_order(PMD_SIZE)); + pte_clear(&init_mm, addr, (pte_t *)pmd); + } +#endif continue; } pte_base = (pte_t *)pmd_page_vaddr(*pmd); - remove_pte_table(pte_base, addr, next, direct); + remove_pte_table(pte_base, addr, next, direct, altmap); free_pte_table(pte_base, pmd); } if (direct) @@ -809,7 +867,8 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, } static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pmd_t *pmd_base; @@ -834,15 +893,16 @@ static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr, } pmd_base = pud_pgtable(*pud); - remove_pmd_table(pmd_base, addr, next, direct); + remove_pmd_table(pmd_base, addr, next, direct, altmap); free_pmd_table(pmd_base, pud); } if (direct) update_page_count(MMU_PAGE_1G, -pages); } -static void __meminit remove_pagetable(unsigned long start, unsigned long end, - bool direct) +static void __meminit +remove_pagetable(unsigned long start, unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long addr, next; pud_t *pud_base; @@ -871,7 +931,7 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end, } pud_base = p4d_pgtable(*p4d); - remove_pud_table(pud_base, addr, next, direct); + remove_pud_table(pud_base, addr, next, direct, altmap); free_pud_table(pud_base, p4d); } @@ -894,7 +954,7 @@ int __meminit radix__create_section_mapping(unsigned long start, int __meminit radix__remove_section_mapping(unsigned long start, unsigned long end) { - remove_pagetable(start, end, true); + remove_pagetable(start, end, true, NULL); return 0; } #endif /* CONFIG_MEMORY_HOTPLUG */ @@ -926,10 +986,223 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start, return 0; } +int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, + unsigned long addr, unsigned long next) +{ + int large = pmd_large(*pmdp); + + if (large) + vmemmap_verify(pmdp_ptep(pmdp), node, addr, next); + + return large; +} + +void __meminit vmemmap_set_pmd(pmd_t *pmdp, void *p, int node, + unsigned long addr, unsigned long next) +{ + pte_t entry; + pte_t *ptep = pmdp_ptep(pmdp); + + VM_BUG_ON(!IS_ALIGNED(addr, PMD_SIZE)); + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + set_pte_at(&init_mm, addr, ptep, entry); + asm volatile("ptesync": : :"memory"); + + vmemmap_verify(ptep, node, addr, next); +} + +static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long addr, + int node, + struct vmem_altmap *altmap, + struct page *reuse) +{ + pte_t *pte = pte_offset_kernel(pmdp, addr); + + if (pte_none(*pte)) { + pte_t entry; + void *p; + + if (!reuse) { + /* + * make sure we don't create altmap mappings + * covering things outside the device. + */ + if (altmap && altmap_cross_boundary(altmap, addr, PAGE_SIZE)) + altmap = NULL; + + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p && altmap) + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, NULL); + if (!p) + return NULL; + } else { + /* + * When a PTE/PMD entry is freed from the init_mm + * there's a free_pages() call to this page allocated + * above. Thus this get_page() is paired with the + * put_page_testzero() on the freeing path. + * This can only called by certain ZONE_DEVICE path, + * and through vmemmap_populate_compound_pages() when + * slab is available. + */ + get_page(reuse); + p = page_to_virt(reuse); + } + + VM_BUG_ON(!PAGE_ALIGNED(addr)); + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + set_pte_at(&init_mm, addr, pte, entry); + asm volatile("ptesync": : :"memory"); + } + return pte; +} + +static inline pud_t *vmemmap_pud_alloc(p4d_t *p4dp, int node, + unsigned long address) +{ + pud_t *pud; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(p4d_none(*p4dp))) { + if (unlikely(!slab_is_available())) { + pud = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + p4d_populate(&init_mm, p4dp, pud); + /* go to the pud_offset */ + } else + return pud_alloc(&init_mm, p4dp, address); + } + return pud_offset(p4dp, address); +} + +static inline pmd_t *vmemmap_pmd_alloc(pud_t *pudp, int node, + unsigned long address) +{ + pmd_t *pmd; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(pud_none(*pudp))) { + if (unlikely(!slab_is_available())) { + pmd = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + pud_populate(&init_mm, pudp, pmd); + } else + return pmd_alloc(&init_mm, pudp, address); + } + return pmd_offset(pudp, address); +} + +static inline pte_t *vmemmap_pte_alloc(pmd_t *pmdp, int node, + unsigned long address) +{ + pte_t *pte; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(pmd_none(*pmdp))) { + if (unlikely(!slab_is_available())) { + pte = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + pmd_populate(&init_mm, pmdp, pte); + } else + return pte_alloc_kernel(pmdp, address); + } + return pte_offset_kernel(pmdp, address); +} + + + +int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) +{ + unsigned long addr; + unsigned long next; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + for (addr = start; addr < end; addr = next) { + next = pmd_addr_end(addr, end); + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return -ENOMEM; + + if (pmd_none(READ_ONCE(*pmd))) { + void *p; + + /* + * keep it simple by checking addr PMD_SIZE alignment + * and verifying the device boundary condition. + * For us to use a pmd mapping, both addr and pfn should + * be aligned. We skip if addr is not aligned and for + * pfn we hope we have extra area in the altmap that + * can help to find an aligned block. This can result + * in altmap block allocation failures, in which case + * we fallback to RAM for vmemmap allocation. + */ + if (altmap && (!IS_ALIGNED(addr, PMD_SIZE) || + altmap_cross_boundary(altmap, addr, PMD_SIZE))) { + /* + * make sure we don't create altmap mappings + * covering things outside the device. + */ + goto base_mapping; + } + + p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); + if (p) { + vmemmap_set_pmd(pmd, p, node, addr, next); + continue; + } else if (altmap) { + /* + * A vmemmap block allocation can fail due to + * alignment requirements and we trying to align + * things aggressively there by running out of + * space. Try base mapping on failure. + */ + goto base_mapping; + } + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + /* + * If a huge mapping exist due to early call to + * vmemmap_populate, let's try to use that. + */ + continue; + } +base_mapping: + /* + * Not able allocate higher order memory to back memmap + * or we found a pointer to pte page. Allocate base page + * size vmemmap + */ + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return -ENOMEM; + + pte = radix__vmemmap_pte_populate(pmd, addr, node, altmap, NULL); + if (!pte) + return -ENOMEM; + + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + next = addr + PAGE_SIZE; + } + return 0; +} + #ifdef CONFIG_MEMORY_HOTPLUG void __meminit radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size) { - remove_pagetable(start, start + page_size, false); + remove_pagetable(start, start + page_size, true, NULL); +} + +void __ref radix__vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) +{ + remove_pagetable(start, end, false, altmap); } #endif #endif diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index fe1b83020e0d..5701faca39ef 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -92,7 +92,7 @@ static struct page * __meminit vmemmap_subsection_start(unsigned long vmemmap_ad * a page table lookup here because with the hash translation we don't keep * vmemmap details in linux page table. */ -static int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size) +int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size) { struct page *start; unsigned long vmemmap_end = vmemmap_addr + vmemmap_map_size; @@ -183,8 +183,8 @@ static __meminit int vmemmap_list_populate(unsigned long phys, return 0; } -static bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, - unsigned long page_size) +bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, + unsigned long page_size) { unsigned long nr_pfn = page_size / sizeof(struct page); unsigned long start_pfn = page_to_pfn((struct page *)start); @@ -204,6 +204,11 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, bool altmap_alloc; unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix__vmemmap_populate(start, end, node, altmap); +#endif + /* Align to the page size of the linear mapping. */ start = ALIGN_DOWN(start, page_size); @@ -303,8 +308,8 @@ static unsigned long vmemmap_list_free(unsigned long start) return vmem_back->phys; } -void __ref vmemmap_free(unsigned long start, unsigned long end, - struct vmem_altmap *altmap) +void __ref __vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) { unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; unsigned long page_order = get_order(page_size); @@ -362,6 +367,17 @@ void __ref vmemmap_free(unsigned long start, unsigned long end, vmemmap_remove_mapping(start, page_size); } } + +void __ref vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) +{ +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix__vmemmap_free(start, end, altmap); +#endif + return __vmemmap_free(start, end, altmap); +} + #endif void register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page, unsigned long size) From patchwork Mon Jul 24 19:07:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 992D7EB64DD for ; Mon, 24 Jul 2023 19:10:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 369E16B0078; Mon, 24 Jul 2023 15:10:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 31A396B007E; Mon, 24 Jul 2023 15:10:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 194D66B0080; Mon, 24 Jul 2023 15:10:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 081D16B0078 for ; Mon, 24 Jul 2023 15:10:48 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CEB4BB21D3 for ; Mon, 24 Jul 2023 19:10:47 +0000 (UTC) X-FDA: 81047447334.20.FFA4EF9 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf26.hostedemail.com (Postfix) with ESMTP id 5A6C9140009 for ; Mon, 24 Jul 2023 19:10:45 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=GCTz3fSI; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf26.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225845; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Pf3xwM0gLXSFKTe/bRRBnDec6Iu7xBNkpvjv5p1v/QI=; b=hdRSOytbR0JzudD1mOmq3WOdzlVa4Ak9GD3DEPO1abn6cYMcErjDiU+ui53Y1NuTOWR1l4 Mt3ROG6LrFYMNs/hqYknQv3NFsQGnCyb9vSInZBYpNmwm8uhv7GaLURk2zusG6fx9U7Cg7 rP41HJt5hDA1Dfixo9f+mCnwhlFsKvY= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=GCTz3fSI; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf26.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225845; a=rsa-sha256; cv=none; b=3A1Da+Iyj8euMHIGwEdEJzHQKna9gFCrwiWVl60ZD03A6N8iaTs7owzJfrL/eSTsqh1+UV K5EphAcHkbI4jX+APh0jjXa8WfAn+9WDRKl1k4YN96fgEn9eyVVSjXwU9yRgbyxLcZGwBp ja+rk2JaMXcPR46E8XxpXpVDKj5ahHM= Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OIUrbL003100; Mon, 24 Jul 2023 19:10:34 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Pf3xwM0gLXSFKTe/bRRBnDec6Iu7xBNkpvjv5p1v/QI=; b=GCTz3fSIdd5TpMyjIbi7tXylq9d7AHf+Kjd8g80zc7JwvOXAmYbrS1VJzYg0JVDdt39F G9xaL39cwG437+cOIO5bq6CzeDl/+DqAT8esiG6pRjR6ocxJ5moKxdxFmGygy87LNB+h cfss5UsaNE9xPIkYyUVn9N6a3neFF1n659nvQyhtBmwMIeOdzl9gsnbotY7uaEPMGK6f G5WSHFAJJgrZDuaj6YIAB70ezLGOpzUrPl+CUvu6IN9463s42NyHu8Hplq6DruXNH/6f hMQCzSZyBlZQ5z/+5kpxVRiXnJmsEkouXt6wdVCKNMoMLLGjxY1gHeWaoCzhTX0qEMhy rA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1w6d37se-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:33 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OJ4ksS024877; Mon, 24 Jul 2023 19:10:33 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1w6d37s3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:33 +0000 Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OHH0JM014374; Mon, 24 Jul 2023 19:10:31 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([172.16.1.70]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3s0stxp1nx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:31 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJAVWk52035968 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:10:31 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4121D5805B; Mon, 24 Jul 2023 19:10:31 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 67B0758059; Mon, 24 Jul 2023 19:10:26 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:10:26 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 11/13] powerpc/book3s64/radix: Add support for vmemmap optimization for radix Date: Tue, 25 Jul 2023 00:37:57 +0530 Message-ID: <20230724190759.483013-12-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: vIhe8sBbq9Cums3H7h1Ilf0dvNPPDvJB X-Proofpoint-GUID: hNYWefVGD8WvHQi_f6ww-ACI53mpPRiF X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 malwarescore=0 adultscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 spamscore=0 mlxlogscore=999 lowpriorityscore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspamd-Queue-Id: 5A6C9140009 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: qkbnu7n4z3eyfeq4r8prm9o3o4bxsect X-HE-Tag: 1690225845-145531 X-HE-Meta: U2FsdGVkX1+R8bSePfA29aixvvYQbq98tJyjUBbSWI8LWPw2BsTGeA+xU6ZnoCwZqEvegfGBQaQjWO9u7sUA8UkeOUThGE8e2lJ3VCbF9S+tLI8Nl28TWW5m89DN6D51/HvJoC2/o/PxP6Ahd7JPGPtUEoZYXck5//F6WmVZSJvHX5yux0agqLInTTZujhHmExb3a5+QjigKYQjgk9zlc6QAMLzPCOYY2Z+Oibh7jsXt2OJVDBeBwrZ5Gs1K/85CxwgvLTDSzDOkvf0G0YjL31NxKMHbjjuNg5dCMoKCnaNXYT698T40It792oGNhhO8CmgxThO0TP6d7szApebr/SHUQ7u/VtrHXNZZVWYEVvQVZQa5G+3TwxBACost6oE3rNMT0ue23ZJ0NUibLuw/eTkKp8o+bZHdnnjx08mQ3WFkDiSryIdWzrL/qjFtP8zmRLYiydXV97t/zFenXIQ6ZwdzznTZpXduCqcdRuamOQcQbbErIi1MVs5OT1OxPLl/4PZcGgwF7Bj4vIz2+bEZuyee8A3Fjj6D1fj3VHVmOLK1lwHHPUN69Ywqr52yrm1OWL45UPvOX6V+QMMBanUZVFA/gnM71wa3dVxuVBdf6daFqOV9EfXhUxDQgO0XW1IYQCzuo0FzV3ZgGlxJNauIastgXgKXnlosoNaEYmX88r9u88ZXYm0ob70f29dhXUgbP6/2yi8ARHKrF1JE9vID2r08UjzYdkrxgDAwB3UwwWvbeI1P7FtR+R6q3yVEW8MCxFw92gJyGgDp79HqNyTlGHNaxa5AgpRKX9/lE5lwPp6u47BeJ/N2Z0IalCSKf+KYIY04LLNgLBxJI0tQYLN/hbhA8m7mn7cnwv+LjgxxAgdME6MKeiA98WNjjI8Ahyqw74wh7Sw/aZpBq/uE7iHJC/acV46hovQ+RJ1nm6aTSrZ8fPyLyV9oTwueoR5uKFGj0m2ZQG3xrH2KaPODF4M e5HoWyUe FDQ9mwxFiV+4WOYW8RWzGYeX5OhdSYqOXR6W7EsetWPafhWRZO4xPbiYMDytaH95zr3XaRCAKcnS9hMXiajcgM+A6OR+umDVM/oTGczipy3RIT7RAZgN2S5mmyZvbPAcSG36/Kb+pmOWf4lI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With 2M PMD-level mapping, we require 32 struct pages and a single vmemmap page can contain 1024 struct pages (PAGE_SIZE/sizeof(struct page)). Hence with 64K page size, we don't use vmemmap deduplication for PMD-level mapping. Signed-off-by: Aneesh Kumar K.V --- Documentation/mm/vmemmap_dedup.rst | 1 + Documentation/powerpc/index.rst | 1 + Documentation/powerpc/vmemmap_dedup.rst | 101 ++++++++++ arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/book3s/64/radix.h | 9 + arch/powerpc/mm/book3s64/radix_pgtable.c | 203 +++++++++++++++++++++ 6 files changed, 316 insertions(+) create mode 100644 Documentation/powerpc/vmemmap_dedup.rst diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_dedup.rst index a4b12ff906c4..c573e08b5043 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -210,6 +210,7 @@ the device (altmap). The following page sizes are supported in DAX: PAGE_SIZE (4K on x86_64), PMD_SIZE (2M on x86_64) and PUD_SIZE (1G on x86_64). +For powerpc equivalent details see Documentation/powerpc/vmemmap_dedup.rst The differences with HugeTLB are relatively minor. diff --git a/Documentation/powerpc/index.rst b/Documentation/powerpc/index.rst index d33b554ca7ba..a50834798454 100644 --- a/Documentation/powerpc/index.rst +++ b/Documentation/powerpc/index.rst @@ -36,6 +36,7 @@ powerpc ultravisor vas-api vcpudispatch_stats + vmemmap_dedup features diff --git a/Documentation/powerpc/vmemmap_dedup.rst b/Documentation/powerpc/vmemmap_dedup.rst new file mode 100644 index 000000000000..dc4db59fdf87 --- /dev/null +++ b/Documentation/powerpc/vmemmap_dedup.rst @@ -0,0 +1,101 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========== +Device DAX +========== + +The device-dax interface uses the tail deduplication technique explained in +Documentation/mm/vmemmap_dedup.rst + +On powerpc, vmemmap deduplication is only used with radix MMU translation. Also +with a 64K page size, only the devdax namespace with 1G alignment uses vmemmap +deduplication. + +With 2M PMD level mapping, we require 32 struct pages and a single 64K vmemmap +page can contain 1024 struct pages (64K/sizeof(struct page)). Hence there is no +vmemmap deduplication possible. + +With 1G PUD level mapping, we require 16384 struct pages and a single 64K +vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Hence we +require 16 64K pages in vmemmap to map the struct page for 1G PUD level mapping. + +Here's how things look like on device-dax after the sections are populated:: + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PUD | +-----------+ | | | + | level | | . | ----------------------+ | | + | mapping | +-----------+ | | + | | | . | ------------------------+ | + | | +-----------+ | + | | | 15 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + + +With 4K page size, 2M PMD level mapping requires 512 struct pages and a single +4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we +require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapping. + +Here's how things look like on device-dax after the sections are populated:: + + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PMD | +-----------+ | | | + | level | | 5 | ----------------------+ | | + | mapping | +-----------+ | | + | | | 6 | ------------------------+ | + | | +-----------+ | + | | | 7 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + +With 1G PUD level mapping, we require 262144 struct pages and a single 4K +vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we +require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level +mapping. + +Here's how things look like on device-dax after the sections are populated:: + + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PUD | +-----------+ | | | + | level | | . | ----------------------+ | | + | mapping | +-----------+ | | + | | | . | ------------------------+ | + | | +-----------+ | + | | | 4095 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 9222c138c457..d0497d13f5b4 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -174,6 +174,7 @@ config PPC select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select ARCH_WANT_LD_ORPHAN_WARN + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if PPC_RADIX_MMU select ARCH_WANTS_MODULES_DATA_IN_VMALLOC if PPC_BOOK3S_32 || PPC_8xx select ARCH_WEAK_RELEASE_ACQUIRE select BINFMT_ELF diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index f1461289643a..3195f268ed7f 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -326,6 +326,7 @@ static inline pud_t radix__pud_mkdevmap(pud_t pud) } struct vmem_altmap; +struct dev_pagemap; extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long page_size, unsigned long phys); @@ -363,5 +364,13 @@ int radix__remove_section_mapping(unsigned long start, unsigned long end); void radix__kernel_map_pages(struct page *page, int numpages, int enable); +#define vmemmap_can_optimize vmemmap_can_optimize +bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap); + +#define vmemmap_populate_compound_pages vmemmap_populate_compound_pages +int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap); #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 53f8340e390c..e5356ac37e99 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -986,6 +986,15 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start, return 0; } + +bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + if (radix_enabled()) + return __vmemmap_can_optimize(altmap, pgmap); + + return false; +} + int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, unsigned long addr, unsigned long next) { @@ -1193,6 +1202,200 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in return 0; } +static pte_t * __meminit radix__vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return NULL; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return NULL; + if (pmd_leaf(*pmd)) + /* + * The second page is mapped as a hugepage due to a nearby request. + * Force our mapping to page size without deduplication + */ + return NULL; + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return NULL; + radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + return pte; +} + +static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr, + unsigned long pfn_offset, int node) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + unsigned long map_addr; + + /* the second vmemmap page which we use for duplication */ + map_addr = addr - pfn_offset * sizeof(struct page) + PAGE_SIZE; + pgd = pgd_offset_k(map_addr); + p4d = p4d_offset(pgd, map_addr); + pud = vmemmap_pud_alloc(p4d, node, map_addr); + if (!pud) + return NULL; + pmd = vmemmap_pmd_alloc(pud, node, map_addr); + if (!pmd) + return NULL; + if (pmd_leaf(*pmd)) + /* + * The second page is mapped as a hugepage due to a nearby request. + * Force our mapping to page size without deduplication + */ + return NULL; + pte = vmemmap_pte_alloc(pmd, node, map_addr); + if (!pte) + return NULL; + /* + * Check if there exist a mapping to the left + */ + if (pte_none(*pte)) { + /* + * Populate the head page vmemmap page. + * It can fall in different pmd, hence + * vmemmap_populate_address() + */ + pte = radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL, NULL); + if (!pte) + return NULL; + /* + * Populate the tail pages vmemmap page + */ + pte = radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL); + if (!pte) + return NULL; + vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE); + return pte; + } + return pte; +} + +int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap) +{ + /* + * we want to map things as base page size mapping so that + * we can save space in vmemmap. We could have huge mapping + * covering out both edges. + */ + unsigned long addr; + unsigned long addr_pfn = start_pfn; + unsigned long next; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + for (addr = start; addr < end; addr = next) { + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return -ENOMEM; + + if (pmd_leaf(READ_ONCE(*pmd))) { + /* existing huge mapping. Skip the range */ + addr_pfn += (PMD_SIZE >> PAGE_SHIFT); + next = pmd_addr_end(addr, end); + continue; + } + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return -ENOMEM; + if (!pte_none(*pte)) { + /* + * This could be because we already have a compound + * page whose VMEMMAP_RESERVE_NR pages were mapped and + * this request fall in those pages. + */ + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } else { + unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); + unsigned long pfn_offset = addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); + pte_t *tail_page_pte; + + /* + * if the address is aligned to huge page size it is the + * head mapping. + */ + if (pfn_offset == 0) { + /* Populate the head page vmemmap page */ + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + /* + * Populate the tail pages vmemmap page + * It can fall in different pmd, hence + * vmemmap_populate_address() + */ + pte = radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, NULL); + if (!pte) + return -ENOMEM; + + addr_pfn += 2; + next = addr + 2 * PAGE_SIZE; + continue; + } + /* + * get the 2nd mapping details + * Also create it if that doesn't exist + */ + tail_page_pte = vmemmap_compound_tail_page(addr, pfn_offset, node); + if (!tail_page_pte) { + + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } + + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*tail_page_pte)); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } + } + return 0; +} + + #ifdef CONFIG_MEMORY_HOTPLUG void __meminit radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size) { From patchwork Fri Jul 28 10:23:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13331627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 833FDC0015E for ; Fri, 28 Jul 2023 10:23:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B8E48D0002; Fri, 28 Jul 2023 06:23:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 169AB6B0074; Fri, 28 Jul 2023 06:23:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 031B78D0002; Fri, 28 Jul 2023 06:23:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E9A716B0072 for ; Fri, 28 Jul 2023 06:23:55 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B35D8160981 for ; Fri, 28 Jul 2023 10:23:55 +0000 (UTC) X-FDA: 81060634830.01.136F144 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf17.hostedemail.com (Postfix) with ESMTP id 64F2140004 for ; Fri, 28 Jul 2023 10:23:53 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=RJTfHSLM; spf=pass (imf17.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690539833; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=N8ygypjq2tkjtKRO/9ldUL+Mk0QOx+wqu5nadJ8BFQQ=; b=jeEExTiADUqBEBuoCHOwvuytapHRQj1L3TmpgTTYGQLpknkYWrewM7FHWtY/dMkEztB5Gd o0Zd67+Py/FFvLff29nTt/YM4vqJUtIVVqeUFno4skG1dWDiipTJo4tj3vTQ3W/GzAnAzI ftFlQOexAaw5e7QwjKSEDbGyfdM1C1Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690539833; a=rsa-sha256; cv=none; b=Ay6+wd0EHwLbMro2gcCCtlBz/TfEBQwpofZIvlaNl6fuF8ybWdYtpbH4MRdRDSjSViYd05 yFFSGTNsEUOwn9xs4SE/dbDOT8Yk9/NN5cxPknc+Hc6cCMCpkIY6USMYMZGZVE3qLrRhr4 kH5qHHXc2uNn9R5K758DLAe/cfLtppU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=RJTfHSLM; spf=pass (imf17.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36SADKrY026312; Fri, 28 Jul 2023 10:23:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : in-reply-to : references : date : message-id : content-type : mime-version; s=pp1; bh=N8ygypjq2tkjtKRO/9ldUL+Mk0QOx+wqu5nadJ8BFQQ=; b=RJTfHSLMuETKqUzK+UhZnP9os9trkzGkWGMBQgVypJgglYPcLNtO6qraQThj3c3wMpoK JceWVt7+88xGpiuH5OawqSMFqIQOfTnGydncfI7JxltiHeIKFOEHUDzcfCkfXfD5GP2T K1zzlE4vN5KU/vG/R36568vOSyP7JQjPqc6MUrL77VdQ7WYGUjVVfXjdXbRJVR0Eds/x alH0SkPXS/EAU1PN+mU3ik69tpUK4Pi14Jl9jM2IgxE9Sz6PzxRxsTazotgeIBG48n1C 5Ff20KSClEuvpxXkyb82pFM56/EG2ERhhRLZs3CrqOeP+eh1v1fRzMsKwnvNa5smRIgF AA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s4bqw88bp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Jul 2023 10:23:42 +0000 Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36SANfnt028218; Fri, 28 Jul 2023 10:23:41 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s4bqw88bd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Jul 2023 10:23:41 +0000 Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36S8SEL6016574; Fri, 28 Jul 2023 10:23:40 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([172.16.1.72]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 3s0v51vmbg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Jul 2023 10:23:40 +0000 Received: from smtpav02.wdc07v.mail.ibm.com (smtpav02.wdc07v.mail.ibm.com [10.39.53.229]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36SANdOT6423048 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 28 Jul 2023 10:23:40 GMT Received: from smtpav02.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DD4DC5805E; Fri, 28 Jul 2023 10:23:39 +0000 (GMT) Received: from smtpav02.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4B0AB5805D; Fri, 28 Jul 2023 10:23:34 +0000 (GMT) Received: from skywalker.linux.ibm.com (unknown [9.43.12.215]) by smtpav02.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 28 Jul 2023 10:23:33 +0000 (GMT) X-Mailer: emacs 29.0.91 (via feedmail 11-beta-1 I) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon Subject: [PATCH v6 12/13 -fix] powerpc/book3s64/radix: Remove mmu_vmemmap_psize In-Reply-To: <20230724190759.483013-13-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> <20230724190759.483013-13-aneesh.kumar@linux.ibm.com> Date: Fri, 28 Jul 2023 15:53:31 +0530 Message-ID: <874jlowd0c.fsf@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-GUID: SCQVDXLFiQJenI2q91kGIzagdGrScCmn X-Proofpoint-ORIG-GUID: 7_mMA98J_lQbs-SbZcGmkjUIFdUNfHxC X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-27_10,2023-07-26_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 bulkscore=0 adultscore=0 mlxlogscore=999 lowpriorityscore=0 suspectscore=0 priorityscore=1501 phishscore=0 malwarescore=0 impostorscore=0 clxscore=1015 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307280092 X-Stat-Signature: y4efpbwunftsy38qbok48jr83o183te6 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 64F2140004 X-Rspam-User: X-HE-Tag: 1690539833-812728 X-HE-Meta: U2FsdGVkX1+yMnmha8940ZO9KNHuNZXYJcCM756h3GXo2m0hpWYIZa/JbMoFz/2TloJ40vruWm1KsPOkiyv0Uzzf6UX1q71h7kYYrPI1hP+L9tG50k2C24ZhMR/fSp1xui+7F6GkmEcGh+AkHVqXHhRFrqfKteg08mpnBM39xhGxyqHIN4oME87E4lq6OfJGSStH1iFCdApQ/BSWE8w0Sco52h9Q0mNREYLKcQJBbpqJpYK1n6BZXJQfZiSXn57R+4T8fo8fhbG6w2/gAeati5YnQfxNQc9jopwRXCNxuLB5yrSQfoC9rDa/EoJ71G21HZITxrRv6DOr6ng6ctl5Uz+hHNV8v/NKHADsIe0CtzhR9Q5dkXpJWCJkuCM0rDvgAvGK+7Ilbe4eo5kkVWxR72syoV4BB7a+UAP49HGQ+zTdkddh14uhyHHmrAlkuxcmfP2yuELPpelN55A1BHov/lzxO07JPWK5zGKFWwg5gj5ijZ3oyVK1M0VYsHIn5wxAEbum87EicaR/j6HdZ8d+5h9i+Ql61O6cLX27EeX5vWuZ2lnjpOriJFB3yZl/pcka4l33npNPBhb7o209EOqAzJmHzI98UkQRcIK8G/mBZWkgzcilGn0Ig6y9cqr6Rq+D3ArmkNf21KS1t+z2FKOoQcWZ33m1I3EXFNhJY1yYFnB4J4jsxPS5qLxbigLbXiH8cMGZUVI/xTOVpbWvrvM6Tdkx7XL6Sq5K0ruaYSI0cUDOMXNynsxKFmzlEQ/0qrKCnw9u5WB5tK/hZpsNBgzTwaCojZKzQNKZgLRKFcpUMPOlnqaBlSXhp6ps8RYymgjTSG6p24MXTCelh1ATMQhq8wxiOBbMw2BlahcIgfZebOvZI0OUA6zrjcq6MUceEYa4U8jOyqLnvWwnVqjsClDzwawwWwZXVqEJBtrFP8tw1KLILcpWXdWtbCGpTc1up2HlVnYixbG6c62mnXt4skX fVXyssQ3 BjiCer6ir4Q4lwZFzUde/t2etIi5pnl4CJP6GrZeqYXQKLK9Wb1vTjZhI0RZOhfgbzLCo3flkamr6uZsUYcYNLz8mSBDSIQwC9wQ+yr5gopyE7F5tY/MBL0FUK4SEFYvLAjexry7j8H30hrooIeKj/46MFr9mO27tr+hIsbB9N7ajthPo6lNFpjM2DkZFU83Ju85iU/sBHGGopErMQgMkPU6IWomNHXYLzD/oPho+3vILj81wcias68RVY4f20bBb0/MngbKQLQqWMWN1939sBV6t8+87ZHorXZZMJenTZcQYhjctCdYLE8waHSMN9nu3+/5GonqFHmHHcSxGTv3npLh1VTNZWA3F134H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From 2d37f0570983bfa710e73a6485e178658e8f4b38 Mon Sep 17 00:00:00 2001 From: "Aneesh Kumar K.V" Date: Fri, 28 Jul 2023 14:47:46 +0530 Subject: [PATCH] powerpc/mm: Fix kernel build error arch/powerpc/mm/init_64.c:201:15: error: no previous prototype for function '__vmemmap_populate' [-Werror,-Wmissing-prototypes] int __meminit __vmemmap_populate(unsigned long start, unsigned long end, int node, ^ arch/powerpc/mm/init_64.c:201:1: note: declare 'static' if the function is not intended to be used outside of this translation unit int __meminit __vmemmap_populate(unsigned long start, unsigned long end, int node, Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202307281617.OxcXz84j-lkp@intel.com/ Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/mm/init_64.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index 8d184e89e943..e0208cb12058 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -198,8 +198,8 @@ bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, return false; } -int __meminit __vmemmap_populate(unsigned long start, unsigned long end, int node, - struct vmem_altmap *altmap) +static int __meminit __vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) { bool altmap_alloc; unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; From patchwork Mon Jul 24 19:07:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13325278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5119EB64DD for ; Mon, 24 Jul 2023 19:10:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CE248E0002; Mon, 24 Jul 2023 15:10:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27E8F8E0001; Mon, 24 Jul 2023 15:10:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11F428E0002; Mon, 24 Jul 2023 15:10:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 002818E0001 for ; Mon, 24 Jul 2023 15:10:58 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C7C92A0B46 for ; Mon, 24 Jul 2023 19:10:58 +0000 (UTC) X-FDA: 81047447796.06.5C3D82F Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf12.hostedemail.com (Postfix) with ESMTP id 6210240013 for ; Mon, 24 Jul 2023 19:10:56 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=SJhF+xtm; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf12.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690225856; a=rsa-sha256; cv=none; b=XvNfS3UqEHR5PA4Snb1yrZ3knYNApBa9SFyMbYo3Pqcn0vEdqVAI5lCIg9i7I8F2XLT6sE bHa64wxKMvj2d+5+cRdaBoQHFlGaf+ew3eBc1Z84Jjf8DNMalf9+E/Y+H6qR/19SycNj8v OGdK1PMnajaV7FemnLkg24EktVr+2Qk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=SJhF+xtm; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf12.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690225856; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OzgVSyWVI3AAxB+6uLj8jPMnZtHSwhw7otMqlC/gYFU=; b=8XwZ8MY27iSPHIJieFGSmPxNxlqOQjyxHFP61CyOzXGdz3fCj6AvYd7JgprFeSobQRoZA5 ryvgMDm628xkv3mCfy19yxh9FmGgzwShMc4JI0YEAwYHfy67VwctQESr8tr9qScakYxKGK GPlUg85Fo2dTsOyv0Xt25BShSnCDfXk= Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36OJ9KE5021802; Mon, 24 Jul 2023 19:10:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=OzgVSyWVI3AAxB+6uLj8jPMnZtHSwhw7otMqlC/gYFU=; b=SJhF+xtm7br5nhXq0yijocNcrGjNVGUJwxuJvUdlgwgBWKPpcrcXbAeYFTqnbzUWxxDk PQEhzrW5nScaeMITi0+8M20mzz8MccUcK4d0Z00mOP5elC5siwxZuGa71q8bCMHvaXtO 1luJ9Ack1eY4awmbj5UooJBJY3vdaPf2Q5YDh5X5ZRyLQI3xpUqPg3ewQpjFfd+XbeN7 texfZWvqqN30Ca9EXFGzaexabQnlnDC4B7FWlTdmz7bJpXAB11EfDc2RsLAka6pI0kso GIww+EkkfFU8QCar/345nOgF/GpWo8zPmX672H0IVtgvnhauiLfWMxWNIWgkRrNABQYF qg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1w3eb1cf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:44 +0000 Received: from m0360072.ppops.net (m0360072.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36OJ9N1Q022158; Mon, 24 Jul 2023 19:10:43 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3s1w3eb1bh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:43 +0000 Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36OIjlFq002278; Mon, 24 Jul 2023 19:10:42 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([172.16.1.4]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3s0unj5bh8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 24 Jul 2023 19:10:42 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36OJAgLF28574434 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 24 Jul 2023 19:10:42 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2C9C258055; Mon, 24 Jul 2023 19:10:42 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 40CFB58067; Mon, 24 Jul 2023 19:10:37 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.177.66.22]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Mon, 24 Jul 2023 19:10:36 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v6 13/13] powerpc/book3s64/radix: Add debug message to give more details of vmemmap allocation Date: Tue, 25 Jul 2023 00:37:59 +0530 Message-ID: <20230724190759.483013-14-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> References: <20230724190759.483013-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 9LVlmaQWoRJPTWWZX7l3NZ4mXIM5w0dC X-Proofpoint-ORIG-GUID: DL7WikEFZZ4S5awmx-Hh0G-n0geze-Ec X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-24_14,2023-07-24_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 impostorscore=0 phishscore=0 adultscore=0 malwarescore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2307240168 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6210240013 X-Stat-Signature: pqexseywbj7k7uhsuktj9bmx74y9uido X-HE-Tag: 1690225856-204197 X-HE-Meta: U2FsdGVkX1+wcd9EvpP73T+PksXQ3sejb4l3QHWO+ZsT4mPvs9aoeXSfbsW7NyttFexkQot1C5Mx1ovKaYtbnWwweV58aT3NaFUfAQglGiyXgQmZdTVJSqm941A7wTqGNOPvaqGuKFZuQeOR4Y6Hqpa5XXUFkTZQysX4sF0uHrvc+LRXzT6e4fkAis3j3Vam9ivdDSCkAaBeSQtWAjeYE0Tp0RyO7VynbSQ58ECuKiFd1omdTy23HPmvje7HDux2cCq3a6BIrD/iBqURV9oFQRHRGNxKwWnu+3eoVzm9dKCQE7ydQDpDCda/LMw4bZhSUOmlPVdmE9dgQ88PNT6ySM14hH9WTqRdkkhvAdNXzXHDZln9TMofn5in7xIKnqk9Sm4+k1MeZ8Olxe2oAi+FGLqbNSs2uAQu4iyHuggbF54sS3ex1+nU/z832iu7K/RaO17By8sHld7japLnE69gfZc6NCDB0e0trIDtY81qRH+fDFQERPwQ+zWC9uM1hw6nkYi6h59IPnxar/Djbm+GnlNivcQGZm5HrOuIJ3lVfSUb/2Hik5XXpp9A8BtNdpb1d/hnT5qWu4LERP3PpIlhKLiHw9yPbKqrb9fRdxccS6ytTTsA249jdY528Vm7xcySn26ggYZUOiLN44djpdQbbaW3LX9aph/7DL+jsV/ACMy0FYzzwV1okPMivsYHPpvoLkuypSL7Fj2estF2crkGotI6Csa8kGeu+tI8F6LNcWgx8n1yIRkBJhsfu4BFFFdVvsNlGMx+j/XJ/y5Rm51AOHRV3QlgsEVpIjbQtm9Nh7hXfQ5sCV+bGl/ieqQ4QI3ocnfYp4c9puGME1G562tUkx2uAhdrd/P8DNzUZPvDCSpmlBE0C8Gz/C86lgdXnpZySo9K8UWeJWIpYEj2GVCDcoGbIIuyovpUg+HCa5jrIFQkGrGrxjqPjoL74/bj2RtmLV+Rd3qzWW108jYBqKZ OIPMA2hX RpDzb/X7E/rKHhl+S0j73aWJnmdUJcROD0B0mAYgExMtF6pnTgu0pyj2Il5PXfqT4ElN6HcDGPf3skhTVUnhTkvDCzum63UUmRhRAKw8GltA2MIHavZWvD9lGmL4Fn+gLwxbe31Lc76igqb6z7mD3TI+UqHEGG2ZuV8K3tjznUHAOMkDbaENvR7FMxaKIKPY1WJvBrPiybRTEy6VRrtXrFKF4ISn/u5RH+8vOhJm6quoBDYg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add some extra vmemmap pr_debug message that will indicate the type of vmemmap allocations. For ex: with DAX vmemmap optimization we can find the below details: [ 187.166580] radix-mmu: PAGE_SIZE vmemmap mapping [ 187.166587] radix-mmu: PAGE_SIZE vmemmap mapping [ 187.166591] radix-mmu: Tail page reuse vmemmap mapping [ 187.166594] radix-mmu: Tail page reuse vmemmap mapping [ 187.166598] radix-mmu: Tail page reuse vmemmap mapping [ 187.166601] radix-mmu: Tail page reuse vmemmap mapping [ 187.166604] radix-mmu: Tail page reuse vmemmap mapping [ 187.166608] radix-mmu: Tail page reuse vmemmap mapping [ 187.166611] radix-mmu: Tail page reuse vmemmap mapping [ 187.166614] radix-mmu: Tail page reuse vmemmap mapping [ 187.166617] radix-mmu: Tail page reuse vmemmap mapping [ 187.166620] radix-mmu: Tail page reuse vmemmap mapping [ 187.166623] radix-mmu: Tail page reuse vmemmap mapping [ 187.166626] radix-mmu: Tail page reuse vmemmap mapping [ 187.166629] radix-mmu: Tail page reuse vmemmap mapping [ 187.166632] radix-mmu: Tail page reuse vmemmap mapping And without vmemmap optimization [ 293.549931] radix-mmu: PMD_SIZE vmemmap mapping [ 293.549984] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550032] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550076] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550117] radix-mmu: PMD_SIZE vmemmap mapping Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/mm/book3s64/radix_pgtable.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 25b46058f556..59aaa30a7c0d 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1033,6 +1033,7 @@ static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long p = vmemmap_alloc_block_buf(PAGE_SIZE, node, NULL); if (!p) return NULL; + pr_debug("PAGE_SIZE vmemmap mapping\n"); } else { /* * When a PTE/PMD entry is freed from the init_mm @@ -1045,6 +1046,7 @@ static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long */ get_page(reuse); p = page_to_virt(reuse); + pr_debug("Tail page reuse vmemmap mapping\n"); } VM_BUG_ON(!PAGE_ALIGNED(addr)); @@ -1154,6 +1156,7 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); if (p) { vmemmap_set_pmd(pmd, p, node, addr, next); + pr_debug("PMD_SIZE vmemmap mapping\n"); continue; } else if (altmap) { /*