From patchwork Wed Dec 18 13:04:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19984E77188 for ; Wed, 18 Dec 2024 13:05:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A31226B0093; Wed, 18 Dec 2024 08:05:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E0D26B0095; Wed, 18 Dec 2024 08:05:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 881426B0096; Wed, 18 Dec 2024 08:05:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 652236B0093 for ; Wed, 18 Dec 2024 08:05:42 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DF22D1A0DFB for ; Wed, 18 Dec 2024 13:05:41 +0000 (UTC) X-FDA: 82908100212.08.5A2EDDB Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf15.hostedemail.com (Postfix) with ESMTP id 55B96A001C for ; Wed, 18 Dec 2024 13:04:53 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=T6JY6OOJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527110; a=rsa-sha256; cv=none; b=eIyFkRjtDgSbaLcdnc+7lvs2tdd9I3vfHwmj1D7Zp6+ccRAoKwRFyfrS5CZD8G3mtGYNT6 jPByu9+1Q4YEWJLsK5LiNLiq67VechrpUTlk6jrh7asWyD7z6Y4ZOz5TT7UvAnz4PDs1R+ rCn5/bqThWNIwDko3w63UEB/ItMlK1Y= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=T6JY6OOJ; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf15.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527110; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=hM2SPRzlcH/sd96KOaA26kXbxPjSpPc80adHGPO3OVAen8Dr/0v+tDJtK8hd5fK8/mLG4f M9x4LzBOEL9hSEyzezolvS9L5bS+JfIrjQgRnrS/jeY7oXTgeu3dJFf/1kBC4+bMkHoquC S2AChLnkwzhYzxAsdhXw2XjM9Gzf5w8= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21644aca3a0so76879985ad.3 for ; Wed, 18 Dec 2024 05:05:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527139; x=1735131939; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=T6JY6OOJVEQXU6v3DU6YdOuMzArKj4BhQxg044V5ZCjuo9rw9KKuBnJvUhmFwBmKSB MyDaPMbTjiYtpu56aLDf8dt69jVZ+nZBd6GkaDG/awJlxDaXxBs/2WjBvbT/g2eBxmqG G1tr7NzBqLoS/JEUtaKzfupf842z6D0iLTCovCCDjYWdtKlZH2zWar6XxGtPxYFVXHS8 2OLP7RI28c3+1iov5vUqRucHBAMhPFLIMnKJxkU2npH4rFm9Pynoe8IM0NmHsvodg9mv WNBR4oJ/GE7lkuxfUDYKgSclXuA9KbtBNEcpqRrKdtsM9cgttExVOHBKQJceCohRHssA FXGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527139; x=1735131939; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xY41yoAggnrpKZDOvMIy0+FhpPRF9QFn+bDK7HKPzWw=; b=uZIqDd8/REPrpjs7CXnaHX/jJw8+EQQ029rtTIzj/q8WIeZS8A/k43x2SUgkpgtUBB +CX+2qwY8ZmJAC62fOOFPiYzbAIP3DNYSczQMAEtSUwOjPKyJSIH8IpTWeRRkhLmDA18 tdevPcpBaUQ9jOTjdlCQeC3LnHjrS1G35GfpUYHWWJojAAET9ZHDeKTau0TPClDnQRqY i6m/aaRtvYlol1+miyMs5CJdEvb7Z5BnrZziKIjdzB8gSCWer9f85Y1BSFskZan1GfzF SDBq3ZqJ5ZGh1n0oTW1VcIWDtDquXjwJoi+NNggKpwhFO/2QbhQleQ28mhIl/dzo9vDh O+Zg== X-Gm-Message-State: AOJu0YzWk3PJTBjX5Uhj687jjdq0YDywhDWj5JleUtWB3Y/elkeJQrAD q90w7zJZrRzERIXMJL8h+nPNXH6LWYYok95Qa1V7eAUiY7fWAhyICNpk55rNvhM= X-Gm-Gg: ASbGncvEWO4WgiMdPgTSdfY5WHzsNkKMo47M1bMYjsyD34yvkdjzeTWmA08WKn1FTH8 70TAVXsPnJ+Yz/FBmdYBQcSEwJ7MSPKTgU/FEpzVhnA2zfoGd6pVmk52mocEGaWzky8IOFO0JIP CY+Ctkba1xZBftyy1EXyzwzjHGiuAvSx9fsZ69XiW5yWLPF5FTGLGSSrgDu+zHON/d0slNI7OtS qe97CMsfMVkfHnaQ9UYMZ3sREADwIbZ/Xo6+kKEQVFLmJxKUXrEu0ZOgl0llPHePGZKVcCnp6ra 3Vpe3UqsNI6ADvowZhLElQ== X-Google-Smtp-Source: AGHT+IEBex3adiI4KuoLgCHeY1D81j9Q6JTE0T6gwcWqf4Lz8nOg8PmUUMxAQoz8nfmNTBwOZklCxg== X-Received: by 2002:a17:903:2448:b0:216:6769:9eea with SMTP id d9443c01a7336-218d72477d0mr41450855ad.37.1734527138569; Wed, 18 Dec 2024 05:05:38 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:37 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , linux-s390@vger.kernel.org Subject: [PATCH v2 04/15] s390: pgtable: add statistics for PUD and P4D level page table Date: Wed, 18 Dec 2024 21:04:40 +0800 Message-Id: <64401d867f3a965f60e9c8d09099b42f21749a47.1734526570.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: e61wpinbertddfayzafrykeh9rszmbyz X-Rspam-User: X-Rspamd-Queue-Id: 55B96A001C X-Rspamd-Server: rspam08 X-HE-Tag: 1734527093-738591 X-HE-Meta: U2FsdGVkX195cfZ9V00UkjJzxMtYJkuW6P56evSHOX7YZjxlgDqzyyMBadq32p51AaWwPe6x5GSKQqKrhwITbA/lMYm7bFiZPMjoygePirh2iPjKUNHxTl2jVZ7QafGDMbhjUsSySe4dPeAJz7pr1Av9nZhxAgEdn1FDdAXolyEXp/Q3r0iMBn0u/TSIhwTn3YqKCf3x2K5u8U/t3MdI7tHWEjPp1a2gmFmd/GcCqgNfizwmVY27ajGE2jds8eEvMG6e+ajpuPXoDxK901vQr22qyLehanzV8b/q8pRhKYzmLFfyK2ew2ouEDkcwrUCHeQfhh1k8Rs7i0b3g9O+Ju9kfqAFunChbB9R07GqExMCYPx+Zbn22Vl2FE2r+/x2xY8MNlz18+xpdYqD7J3hkz9ySa0YjtMVvywtuthvLB3yfH9klf45U/GgaGgk8NOvTOEs/ARDvCrFeYKUYD+imhzG2JIKnGxfh+zyG8rVniZMoPZsZv32SKv9N1SqGqNNWDyGN572pV9d+8JlXG2jc0irebUB3vPe3dM6iq80nO2JxO/+Xqosfh+0kqQ1t2EbRhjblet40vl+DiYnb05HPyBZiOHCih3kKyo7K6YPXYl95OqOegbS7clxh2PRDtx8XPfQ0pS7qVVY+0kesZUXRyYV6GFWy+wRlf5QVbqzcwkMXijMz132qjHIWR6/iqD/Tk0X0U3745d8ff3i5Wfmt2kPoR7P+NPgpYXmog5iXfC42L0ALiAv/Hzsi5hbbBHOjIw5nwII1CdpF3+rv7sCSsUNuQIJhkpeLeU9TFQZsX0TaQMzHuG9gZCjsbJJGlhUX3vZm3WiDgsIxwD7FAthi3sRXupxayIfvlnSB4r35inVIVggw3zTPkBxmrcuN4NfCUiphps+vEZc8FNsG6opmtdGOMopF9rO8b8QEtT0OBj/8/fItegsOcLKjfMqUIDFgifXG2iymKfPdH4Xw807 qEzV2pVw qMspL0QsW1G8ZJom7huG9QHFSBsedJvGvqNQTmD0yyBsb7JUL7PUMeNOS8tfBsMyslN61GVkGcNHuU1kw5z9bOfFf5hf7HuhBPK6Lba/ri3R4zaRh8LaBLfmSdsQhvcFZ+sl4JoUU3YKLN1psdDv+UwvMGO33B1hBnAXw14LsMfzyWqbnp03GIAFZ9fpQ7WruNbDpV6MEk8Pj55rW7KF8fcG3kB4eGZcD8rQXvo6xS6OMImtqlZA43ucLlxWNUPcoTCQRH4oMr23ZDKLgU78W0pESzYom7eeB1Ar7AKTnUfKv3Mieh5AXSWORx60l7fsApYW/ZWanxI9LdiUfYNbZnsj2IuiftEHKxjSZFvkLjvK4yD1miN3je4MFsgXL3sifWV5rm99E7w9z/IxqQVBylSmjSAunJusUmJicmFgJhy4Anv1KrEKo4JYjZxflaJm1y0R+ghlg5k53Ir4ATLiEEyp3uw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Like PMD and PTE level page table, also add statistics for PUD and P4D page table. Signed-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Cc: linux-s390@vger.kernel.org --- arch/s390/include/asm/pgalloc.h | 29 +++++++++++++++++++------- arch/s390/include/asm/tlb.h | 37 +++++++++++++++++---------------- 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index 7b84ef6dc4b6d..a0c1ca5d8423c 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -53,29 +53,42 @@ static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION2_ENTRY_EMPTY); + if (!table) + return NULL; + crst_table_init(table, _REGION2_ENTRY_EMPTY); + pagetable_p4d_ctor(virt_to_ptdesc(table)); + return (p4d_t *) table; } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!mm_p4d_folded(mm)) - crst_table_free(mm, (unsigned long *) p4d); + if (mm_p4d_folded(mm)) + return; + + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + crst_table_free(mm, (unsigned long *) p4d); } static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { unsigned long *table = crst_table_alloc(mm); - if (table) - crst_table_init(table, _REGION3_ENTRY_EMPTY); + + if (!table) + return NULL; + crst_table_init(table, _REGION3_ENTRY_EMPTY); + pagetable_pud_ctor(virt_to_ptdesc(table)); + return (pud_t *) table; } static inline void pud_free(struct mm_struct *mm, pud_t *pud) { - if (!mm_pud_folded(mm)) - crst_table_free(mm, (unsigned long *) pud); + if (mm_pud_folded(mm)) + return; + + pagetable_pud_dtor(virt_to_ptdesc(pud)); + crst_table_free(mm, (unsigned long *) pud); } static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long vmaddr) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index e95b2c8081eb8..b946964afce8e 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -110,24 +110,6 @@ static inline void pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, tlb_remove_ptdesc(tlb, pmd); } -/* - * p4d_free_tlb frees a pud table and clears the CRSTE for the - * region second table entry from the tlb. - * If the mm uses a four level page table the single p4d is freed - * as the pgd. p4d_free_tlb checks the asce_limit against 8PB - * to avoid the double free of the p4d in this case. - */ -static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long address) -{ - if (mm_p4d_folded(tlb->mm)) - return; - __tlb_adjust_range(tlb, address, PAGE_SIZE); - tlb->mm->context.flush_mm = 1; - tlb->freed_tables = 1; - tlb_remove_ptdesc(tlb, p4d); -} - /* * pud_free_tlb frees a pud table and clears the CRSTE for the * region third table entry from the tlb. @@ -140,11 +122,30 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pagetable_pud_dtor(virt_to_ptdesc(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; tlb_remove_ptdesc(tlb, pud); } +/* + * p4d_free_tlb frees a p4d table and clears the CRSTE for the + * region second table entry from the tlb. + * If the mm uses a four level page table the single p4d is freed + * as the pgd. p4d_free_tlb checks the asce_limit against 8PB + * to avoid the double free of the p4d in this case. + */ +static inline void p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long address) +{ + if (mm_p4d_folded(tlb->mm)) + return; + pagetable_p4d_dtor(virt_to_ptdesc(p4d)); + __tlb_adjust_range(tlb, address, PAGE_SIZE); + tlb->mm->context.flush_mm = 1; + tlb->freed_tables = 1; + tlb_remove_ptdesc(tlb, p4d); +} #endif /* _S390_TLB_H */