From patchwork Wed Dec 20 05:18:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nanyong Sun X-Patchwork-Id: 13499491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6ADBC3DA6E for ; Wed, 20 Dec 2023 04:21:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41FBD8D0006; Tue, 19 Dec 2023 23:21:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AB968D0001; Tue, 19 Dec 2023 23:21:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 187DD8D0006; Tue, 19 Dec 2023 23:21:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 094938D0001 for ; Tue, 19 Dec 2023 23:21:49 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D0BDC4035B for ; Wed, 20 Dec 2023 04:21:48 +0000 (UTC) X-FDA: 81585898296.05.C3A6E86 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf14.hostedemail.com (Postfix) with ESMTP id 2E06B100007 for ; Wed, 20 Dec 2023 04:21:45 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of sunnanyong@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=sunnanyong@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703046107; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sdpGPIry7Om9J5JJWjA8qr98spmVPAsS0vg4JpIjNWw=; b=wfmR2XSSdYivjewD9wAOF16NfYdBKdQNZcRIN5NkGFa7hl/W2EgLpuNtWT2EbRZKWYjIZi 8WT/bMCPOyYu+6W0RSqaGgrH4Lep6mdvQqMwT8FiakC7cajigxhKg+RtLuF95CPN6XrEeQ RT4dTlj/gzMYDMk67Tj1pL+hLsXsMDE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703046107; a=rsa-sha256; cv=none; b=VaVqY+9bpc2e4B1S5dv3VaGnPSpEUtxVYwUCrJmQNyEaVDBLVEOj7xbfErk+WYS4hggnqE Vd/5wnewu0mw/8HseR8yGVfocLyBBdg2oLa+wKn7+jbxOEW9C/cAd2uAfJNrZS+gVjZ7Aj 1VH6jWuBd2MCvbTltPwLHJwYSl7Ty1k= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of sunnanyong@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=sunnanyong@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.252]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Sw0kF2gX3zsSDx; Wed, 20 Dec 2023 12:21:25 +0800 (CST) Received: from kwepemm000003.china.huawei.com (unknown [7.193.23.66]) by mail.maildlp.com (Postfix) with ESMTPS id 0F7BB18007A; Wed, 20 Dec 2023 12:21:42 +0800 (CST) Received: from huawei.com (10.175.113.32) by kwepemm000003.china.huawei.com (7.193.23.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Wed, 20 Dec 2023 12:21:41 +0800 From: Nanyong Sun To: , , , , , CC: , , , , , Subject: [PATCH v2 1/3] mm: HVO: introduce helper function to update and flush pgtable Date: Wed, 20 Dec 2023 13:18:53 +0800 Message-ID: <20231220051855.47547-2-sunnanyong@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231220051855.47547-1-sunnanyong@huawei.com> References: <20231220051855.47547-1-sunnanyong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.32] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To kwepemm000003.china.huawei.com (7.193.23.66) X-Rspamd-Queue-Id: 2E06B100007 X-Rspam-User: X-Stat-Signature: j7xmxmg4kj7yuntxoc6gp55znnddn5fk X-Rspamd-Server: rspam03 X-HE-Tag: 1703046105-91407 X-HE-Meta: U2FsdGVkX1/m53cUc7VrHFOkJs09AVIz2stLnWvU6GulBVctlMj9D6ShPmYlJD2usVc786275LR7mJCJSRWWX3VWVl6FxN6+L8u9Sj/1xaPSAeoe0hMu5HLyzhiXj7WmxiOiS/ynXNTn+czdtsBzQQeUy83+hekoKxNzSlObPEfwRSRWfW0flANAlGAq7GUtSBo+m3lRsJeccY1btsxYHAXOjo0O84EcK9hrF57QuMdpdDDPW4DGhSpgQgH1get0An3sBWahVbq/c1RTKESzWtrTkWXXNn2+GLIv3UtqbQk5n2vDEbjAl53kIRgaesYbBezZCdhPs3zNKFLjCXe1BYwb/K7/yvi1tVayStY0is5Hm8/sz/82x8ziIZQnF0JgGrzz3QxUNPjocGtxRcz5/FpFWwSdqE3BNdALPb7X+X+2zxZ34NntCrLbmbTVs5TQVkaOq7eQn8qy06lRqCeW/90aKIbiZflTitYkMENbLH/BwnuMKkWmJuUdlG1+nWPmi0rToQNqZ3ynKL83ptYtwMthfb+G3DLbM5lJE1ANGXzutCRx2WHcFcIs2y2AP0+v7LNtgdZBR3BI4glT/9Uc9y4e9yyTxrcLOHjbUgIkgCh7JXzmDE3kpWmBNsRpk7vnp4PCEqFRFcKBUm2wiSjJ0h7+qQYE0oAy4V0+ZNvDYJpORQqOAmLVJoiE4ppubuqRYW9mPYSMUwldulCzRgghpyM/Wlp64IUJD97qroeq8c1gcS6F4/EsDwazkvhlzr2r8ci05809UG6ZizowBotfI3nxogF6gH88b8Pen9WXOZk3RqTg91z9JwYPsAkoE8esaobt7R9dt5BsKJYxm+i+NmweEvAJ3NRPVSg3RHlCPZBHgdTwHjUf1dfzGVAXB5SNoivEHrRNsDSgmW/Wfetz8iuYTL6A5FW2myW/yR8755iqxB4JUqCMTfaN6oJQy/JLCXSSx6EF7cgiGPxFNML 2g3xsSjj EdnoYyxw12v7akybktCTlAHIkf6v3WbKk++H3kgow+S/KXXaMlQbpJEk9aRiQa375Y6VxURgbmRL3Xx5Ak4MzCZ/1lfmgxTDODMUkAKYRT3z1ngXNXlIk1OFqh/Th1RxV8ZTAskzVYO2LjUCNnVULAwWG93tzaVXF88WesfF0Kt7VEM7r/ub/T/TOd5E7wSrI0VsLo0ilZd4Zkgs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add pmd/pte update and tlb flush helper function to update page table. This refactoring patch is designed to facilitate each architecture to implement its own special logic in preparation for the arm64 architecture to follow the necessary break-before-make sequence when updating page tables. Signed-off-by: Nanyong Sun Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 55 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 87818ee7f01d..2187e5410a94 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -45,6 +45,37 @@ struct vmemmap_remap_walk { unsigned long flags; }; +#ifndef vmemmap_update_pmd +static inline void vmemmap_update_pmd(unsigned long addr, + pmd_t *pmdp, pte_t *ptep) +{ + pmd_populate_kernel(&init_mm, pmdp, ptep); +} +#endif + +#ifndef vmemmap_update_pte +static inline void vmemmap_update_pte(unsigned long addr, + pte_t *ptep, pte_t pte) +{ + set_pte_at(&init_mm, addr, ptep, pte); +} +#endif + +#ifndef vmemmap_flush_tlb_all +static inline void vmemmap_flush_tlb_all(void) +{ + flush_tlb_all(); +} +#endif + +#ifndef vmemmap_flush_tlb_range +static inline void vmemmap_flush_tlb_range(unsigned long start, + unsigned long end) +{ + flush_tlb_kernel_range(start, end); +} +#endif + static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; @@ -87,9 +118,9 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); - pmd_populate_kernel(&init_mm, pmd, pgtable); + vmemmap_update_pmd(start, pmd, pgtable); if (flush) - flush_tlb_kernel_range(start, start + PMD_SIZE); + vmemmap_flush_tlb_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -217,7 +248,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, } while (pgd++, addr = next, addr != end); if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) - flush_tlb_kernel_range(start, end); + vmemmap_flush_tlb_range(start, end); return 0; } @@ -263,15 +294,15 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, /* * Makes sure that preceding stores to the page contents from - * vmemmap_remap_free() become visible before the set_pte_at() - * write. + * vmemmap_remap_free() become visible before the + * vmemmap_update_pte() write. */ smp_wmb(); } entry = mk_pte(walk->reuse_page, pgprot); list_add(&page->lru, walk->vmemmap_pages); - set_pte_at(&init_mm, addr, pte, entry); + vmemmap_update_pte(addr, pte, entry); } /* @@ -310,10 +341,10 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, /* * Makes sure that preceding stores to the page contents become visible - * before the set_pte_at() write. + * before the vmemmap_update_pte() write. */ smp_wmb(); - set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + vmemmap_update_pte(addr, pte, mk_pte(page, pgprot)); } /** @@ -576,7 +607,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, } if (restored) - flush_tlb_all(); + vmemmap_flush_tlb_all(); if (!ret) ret = restored; return ret; @@ -744,7 +775,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l break; } - flush_tlb_all(); + vmemmap_flush_tlb_all(); list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize_folio(h, folio, @@ -760,7 +791,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l * allowing more vmemmap remaps to occur. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { - flush_tlb_all(); + vmemmap_flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); __hugetlb_vmemmap_optimize_folio(h, folio, @@ -769,7 +800,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l } } - flush_tlb_all(); + vmemmap_flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); }