From patchwork Tue Mar 12 00:57:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10848441 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 546EC6C2 for ; Tue, 12 Mar 2019 00:58:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D47029407 for ; Tue, 12 Mar 2019 00:58:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3B34E29418; Tue, 12 Mar 2019 00:58:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5305429401 for ; Tue, 12 Mar 2019 00:58:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 05B0D8E0003; Mon, 11 Mar 2019 20:58:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 00B838E0002; Mon, 11 Mar 2019 20:58:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E14A98E0003; Mon, 11 Mar 2019 20:58:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id B8D1A8E0002 for ; Mon, 11 Mar 2019 20:58:05 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id c188so1277506ywf.14 for ; Mon, 11 Mar 2019 17:58:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=0Yb2ivGv5k2eEZFqDaeb2AMWK/OeRIAqkI0UIANBdSk=; b=XNrxg+hrDS4Z5FM4W9EZ46pNxnP/P6/ubW0Mh/zAz9W6H300E7BjosMIQ1QzSvHyVn jmgrlK/KZN0R+CCx2wxuydP0zak0bOumTAaVBlt4D0J2W0HhVFXS1Azuib3c1Vswt/ZJ PnWvNShZkFU9GX+6UzL52PajY6EDVicAreYdfJn+1kpnUZ6Us1D33tV28vn0+o3/P/pO nJu3xoh/8xsfg5ZWIkARMtU9ICkdAQbOomCbOXBYYkm2uYlduJoDPKKtWayek4p0o546 WslQyEe/UlhGpyYVtTymkSGSqsDUbBdvh1r1hmVGSx70InWE6mW9sW/Sj/YSCOiMg4xU oMQQ== X-Gm-Message-State: APjAAAVmvhpzQvnd55Yv4dK23fViRk2qKGpWyHG15HGrjSJCcjw9VZQi aVYF3WJW35HIjlCTAJP3QCMoE2kEz5wYJ6U0/UKkzz3xLUZ9JktWGhz6ekbGGTeS5c7kNetXMeL fY1dMrNVg7QU7Pi2lkao06aFvnxgQxSLnO+j6FIAIGzJcoRgZCX7WMMsbnIHsSrUieGSSd68f9p ujgCkyPy2xmWvzyLwSmZDXGlKkj49jcXAN4DcYNnxMkLZ59m+587++J2roCWgnaGoCQVAGrLfYW QlhxXnyJblyaIQJMRyEZVRahBMW/AFPX1iB5j3QTGKiRdCbkpIj7ZxVW07044d0MylNsq/7j+Wy 23Wl7wLjQG5M+K/iXqiQ7JBElcOxC3ZxN+kwIxK9bt0tQnRqR6g7ik9QO0SRC00uFkqrEy5Mg3K 2 X-Received: by 2002:a25:d882:: with SMTP id p124mr29644311ybg.72.1552352285439; Mon, 11 Mar 2019 17:58:05 -0700 (PDT) X-Received: by 2002:a25:d882:: with SMTP id p124mr29644279ybg.72.1552352284553; Mon, 11 Mar 2019 17:58:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552352284; cv=none; d=google.com; s=arc-20160816; b=hKAjlQgTeoH3bOOSYlnOKB9Gaz6CAPPIqWB9n5IDv/MgalvYwLqfGn0ShWWD/0i1my iO+KysIwjCz/Z+1If9GwayqknQ/sbniP36QsBkv6zAEuZ3ZMxzMxu/bGlTykW2nNOeF5 MMIMwp0hY4/PAry3kujCIEFbiru0HMLT25C/kGw9QUDPVv5CTa3aUSyfqfu/YPSFOVGY MEO/nCW2kUjOjcFNg1SqnsuxHyhXyHueCAcjd2SE1IEjc94b5TMg4OI13eNGyZIS1p+m ohgerVIsi7SzMtlmLlvX8Igf1b4/DYaZ1xPcS1QGN59/BoZvmDwYK0yErwuxu+2HcxD6 DnMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=0Yb2ivGv5k2eEZFqDaeb2AMWK/OeRIAqkI0UIANBdSk=; b=ZKXAbxoKInqsnSdRkjOjSjzn1Bfv6JaFdlOccAbNSfSn+2R/t7s1w3eJqqDB2kbqY2 7IUxpbPQJWV81nOVJ61lZ2PqHDs1o+HSwCD1pzd1yEvnhJrWzumHhNVdnluwUXAGukpJ gnRaQxuUGpCmbKGa732uJdUw8HkNnDTEyUFoSAvt4/FWmfFj6UHSLjPTwDNuQJHshlNS GVrSa9CrcVfrNjM9YZZ7nJ+YhjLwMspPQBZgFZjIkpDw2e95TX/nl6HNihYalkmeWTzm +9bfTbBxBKVq/WdFQEHq4T2ZhMIiy1tuLEJ7CSquOVMYiDLTERi+u6QE8oVS2EyZnzq9 VYug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=YtoTCa1F; spf=pass (google.com: domain of 3hashxaykccsfbgohvnvvnsl.jvtspube-ttrchjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3HASHXAYKCCsfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id 184sor3409124ybm.143.2019.03.11.17.58.04 for (Google Transport Security); Mon, 11 Mar 2019 17:58:04 -0700 (PDT) Received-SPF: pass (google.com: domain of 3hashxaykccsfbgohvnvvnsl.jvtspube-ttrchjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=YtoTCa1F; spf=pass (google.com: domain of 3hashxaykccsfbgohvnvvnsl.jvtspube-ttrchjr.vyn@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3HASHXAYKCCsfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0Yb2ivGv5k2eEZFqDaeb2AMWK/OeRIAqkI0UIANBdSk=; b=YtoTCa1FsAa4SxSSy2t1RCEeLGZsCq3F9MmC82/HiNbwzc/7heOZhhMADajPy01n9b agSe/kw18UTudBiU7ksU8/SHwvlZlPyV9LhN6fTTgC8Pm6CxdX5WamzES1EemO0KXy/j decP+RthgSoxH0b7lDx3YgrKtEniFzHFTV9YD3AosSfoHuoGpJSZcij+2TA2ZsJbnnyd To+zhj0MEY+b7krt7yyKqMyeZJG/PoA+A8X8hCbGrS2l76904BOZGQ2g93sMN7LaVVvu yc2xlRSPhv8JbiOW+JO7nL/y8uBBxIr5HeyVp0+8N+nq2xCyLduG0F8nOgQDZH2I33Hs 22eQ== X-Google-Smtp-Source: APXvYqw+ll+HDtKzYX+8qYICIA45HG4qpRTQvEyf0AUjyPzgDXuQrmIOJztXx4aVqQDr6yzeNnT78JHSF9Y= X-Received: by 2002:a25:949:: with SMTP id u9mr15976942ybm.8.1552352284196; Mon, 11 Mar 2019 17:58:04 -0700 (PDT) Date: Mon, 11 Mar 2019 18:57:46 -0600 In-Reply-To: <20190310011906.254635-1-yuzhao@google.com> Message-Id: <20190312005749.30166-1-yuzhao@google.com> Mime-Version: 1.0 References: <20190310011906.254635-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v4 1/4] arm64: mm: use appropriate ctors for page tables From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland Cc: "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Joel Fernandes , "Kirill A . Shutemov" , Ard Biesheuvel , Chintan Pandya , Jun Yao , Laura Abbott , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP For pte page, use pgtable_page_ctor(); for pmd page, use pgtable_pmd_page_ctor(); and for the rest (pud, p4d and pgd), don't use any. For now, we don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK and pgtable_pmd_page_ctor() is a nop. When we do in patch 3, we make sure pmd is not folded so we won't mistakenly call pgtable_pmd_page_ctor() on pud or p4d. Acked-by: Mark Rutland Signed-off-by: Yu Zhao --- arch/arm64/mm/mmu.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b6f5aa52ac67..f704b291f2c5 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -98,7 +98,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); -static phys_addr_t __init early_pgtable_alloc(void) +static phys_addr_t __init early_pgtable_alloc(int shift) { phys_addr_t phys; void *ptr; @@ -173,7 +173,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; @@ -183,7 +183,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, if (pmd_none(pmd)) { phys_addr_t pte_phys; BUG_ON(!pgtable_alloc); - pte_phys = pgtable_alloc(); + pte_phys = pgtable_alloc(PAGE_SHIFT); __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); pmd = READ_ONCE(*pmdp); } @@ -207,7 +207,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), int flags) + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; pmd_t *pmdp; @@ -245,7 +245,7 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), int flags) + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; pud_t pud = READ_ONCE(*pudp); @@ -257,7 +257,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, if (pud_none(pud)) { phys_addr_t pmd_phys; BUG_ON(!pgtable_alloc); - pmd_phys = pgtable_alloc(); + pmd_phys = pgtable_alloc(PMD_SHIFT); __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); pud = READ_ONCE(*pudp); } @@ -293,7 +293,7 @@ static inline bool use_1G_block(unsigned long addr, unsigned long next, static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; @@ -303,7 +303,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, if (pgd_none(pgd)) { phys_addr_t pud_phys; BUG_ON(!pgtable_alloc); - pud_phys = pgtable_alloc(); + pud_phys = pgtable_alloc(PUD_SHIFT); __pgd_populate(pgdp, pud_phys, PUD_TYPE_TABLE); pgd = READ_ONCE(*pgdp); } @@ -344,7 +344,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long addr, length, end, next; @@ -370,11 +370,23 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, } while (pgdp++, addr = next, addr != end); } -static phys_addr_t pgd_pgtable_alloc(void) +static phys_addr_t pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(PGALLOC_GFP); - if (!ptr || !pgtable_page_ctor(virt_to_page(ptr))) - BUG(); + BUG_ON(!ptr); + + /* + * Call proper page table ctor in case later we need to + * call core mm functions like apply_to_page_range() on + * this pre-allocated page table. + * + * We don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK if pmd is + * folded, and if so pgtable_pmd_page_ctor() becomes nop. + */ + if (shift == PAGE_SHIFT) + BUG_ON(!pgtable_page_ctor(virt_to_page(ptr))); + else if (shift == PMD_SHIFT) + BUG_ON(!pgtable_pmd_page_ctor(virt_to_page(ptr))); /* Ensure the zeroed page is visible to the page table walker */ dsb(ishst); From patchwork Tue Mar 12 00:57:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10848443 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2A4146C2 for ; Tue, 12 Mar 2019 00:58:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 171EA292FA for ; Tue, 12 Mar 2019 00:58:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 14ECF29416; Tue, 12 Mar 2019 00:58:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 941E429417 for ; Tue, 12 Mar 2019 00:58:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F4638E0004; Mon, 11 Mar 2019 20:58:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2A4318E0002; Mon, 11 Mar 2019 20:58:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BA468E0004; Mon, 11 Mar 2019 20:58:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f72.google.com (mail-io1-f72.google.com [209.85.166.72]) by kanga.kvack.org (Postfix) with ESMTP id F0FD28E0002 for ; Mon, 11 Mar 2019 20:58:06 -0400 (EDT) Received: by mail-io1-f72.google.com with SMTP id y13so609588iol.1 for ; Mon, 11 Mar 2019 17:58:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=0N1MhcRbdixhLcojGjz1JbD9gYvKsU3sALAMDz5L+zQ=; b=RP7Nqx5aygFgILxNO+fcwNPF71czk1fIz3dSmghsuX6F+PC0IzeT0eSnBrRB1dGP00 TGtsX6VQmhQEbG5PoQdJgsh1PN7GbRZ4mYONl+D57aJNdtOD8hKtl/yvyrKhlCv/q0zt O8qVSW6APeWcJr3ANg+uTtIDjkOEBYpmffg8cw90ewX/NcJVOhJNbiKwSeV8keQ/fTOL 5C2zqyEiiRNxHS0nO7+sPDeoaWKKn06uVu+vTtIE7e6MhldcIxE4nJXwfnREZL5sSJSf iAQ+JStV1n5qoJ/xpAUZGt/tv9eK+sEQRBedOfP4YZHm7ac+h9PhxdXg20aAeKi0Yjgg 9qVg== X-Gm-Message-State: APjAAAVY9XjajN19Wd4nwV96ENJS7zfUiejzj57PpXc7ge3uUkwfT/Ko gmeEe2wFsrpuNeFZzXWadmyjEuNK9SefgEirj3auXxI62QXfB8GBRfCgfQW6lZ5arzf0xmCQuMC 64zKYIfXcRnxKYM9vUAqhO6/e033noUbKTPi0ddCyLxiZo1F1h1croYBQzn6CE3ohBQ== X-Received: by 2002:a24:68cb:: with SMTP id v194mr511715itb.145.1552352286756; Mon, 11 Mar 2019 17:58:06 -0700 (PDT) X-Received: by 2002:a24:68cb:: with SMTP id v194mr511695itb.145.1552352286052; Mon, 11 Mar 2019 17:58:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552352286; cv=none; d=google.com; s=arc-20160816; b=tgkAoMu1i1Bb9PfjF5a7cQpHA7HgK2AUR7RwrBetuXggFRZjtstFXr+CnM3RywengU UD42I1O+iFAGCuEYJwwk4Fb3AqJcgX2ZrX9U5SDDCCkEiTKR6A7JMMgPX20yCKR6tLCc yqh+o13dJte9q3MoUf5yn4A03Eq70Lx9Su6w65hT2wzQIU8WTjmUoGw9ISOzYG1cEncL 8MjKZ5xt720eZ04FTCFoD+zuPWETwICds9DCL1/mEwlst/wOaleQIkEF6pD242AvM0Uo frcJceTxgYckRziKO8uQLI24fA7JhieFr/4/zYdwkxssTdX9KxnJi+CBzXE0LoD/XOb/ E3Fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=0N1MhcRbdixhLcojGjz1JbD9gYvKsU3sALAMDz5L+zQ=; b=DXDXkt/D9hAAC/w0C3p6q2qwsyCZXWRcIHFKozL5sREAFIS+S3d55cU4M43jTsibsM 0z+h8tFyhJ2SXtjW8iWgnE5HmGUbZKGlxjoQZ0umsu5GAuUvzpfv7OwkRSN9Q8EoH+3m PA5436CqA0qJd01a86c/fihpFwHRTxL/FuL31cFjcQbVTa24DaZ+YgjHHpgZzQOk68r+ sCDbFCLafXZPHJiw5fUsBzMl+svzDkm/hSegKXcmuSjTaU2qmbySfmmu3NBB2l97mCWU nl/65oFGlthPa5Z5Br3AHORXmWE4104Cj2wx8WrhGEHgYL59rWhDStEIHNLYenFYgzg/ KUdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=m2SSklts; spf=pass (google.com: domain of 3hqshxaykccwgchpiwowwotm.kwutqvcf-uusdiks.wzo@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3HQSHXAYKCCwgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id l200sor1330153itb.24.2019.03.11.17.58.05 for (Google Transport Security); Mon, 11 Mar 2019 17:58:06 -0700 (PDT) Received-SPF: pass (google.com: domain of 3hqshxaykccwgchpiwowwotm.kwutqvcf-uusdiks.wzo@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=m2SSklts; spf=pass (google.com: domain of 3hqshxaykccwgchpiwowwotm.kwutqvcf-uusdiks.wzo@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3HQSHXAYKCCwgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0N1MhcRbdixhLcojGjz1JbD9gYvKsU3sALAMDz5L+zQ=; b=m2SSklts5axBtBFT99k5kWRqt2k7AZ/UFLe5QDWp2m4MrS9nPTzvu+lW6Cw1GTX91x q7m9lX4AdtHIlQvrmJR+9nAGmjI1Sxwo9tCXJq3a4TfLEoryCkDdcW+2PmNyW2gIWRBs wTPKiJ0yJ7eAQJXABh3sNDsdXtcGTVXBZd0z6nTiMfX/ZIfTqQq83M55OBXw3KfljcNy jwcNX81PR159z9bI7I9PlNagdpSG7F8sKSGKprXJJKepemxzBfeWNYASGdTYUFTIqFFc lrAn/W9ub2uGjX9hl5ahbzfE78V49Svncz9HAlXjfo8dyifGlJnK9/F5LsE8kRnM/321 lOaA== X-Google-Smtp-Source: APXvYqw5KDZ39HU8Icwu4S6RpSSs7BszMpNyZQ5WJg5vOVgn5k90ODytif1OzlfBC2AtFGb/6BjcJbfFcJE= X-Received: by 2002:a24:5a04:: with SMTP id v4mr546460ita.37.1552352285759; Mon, 11 Mar 2019 17:58:05 -0700 (PDT) Date: Mon, 11 Mar 2019 18:57:47 -0600 In-Reply-To: <20190312005749.30166-1-yuzhao@google.com> Message-Id: <20190312005749.30166-2-yuzhao@google.com> Mime-Version: 1.0 References: <20190310011906.254635-1-yuzhao@google.com> <20190312005749.30166-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v4 2/4] arm64: mm: don't call page table ctors for init_mm From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland Cc: "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Joel Fernandes , "Kirill A . Shutemov" , Ard Biesheuvel , Chintan Pandya , Jun Yao , Laura Abbott , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP init_mm doesn't require page table lock to be initialized at any level. Add a separate page table allocator for it, and the new one skips page table ctors. The ctors allocate memory when ALLOC_SPLIT_PTLOCKS is set. Not calling them avoids memory leak in case we call pte_free_kernel() on init_mm. Acked-by: Mark Rutland Signed-off-by: Yu Zhao --- arch/arm64/mm/mmu.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f704b291f2c5..d1dc2a2777aa 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -370,6 +370,16 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, } while (pgdp++, addr = next, addr != end); } +static phys_addr_t pgd_kernel_pgtable_alloc(int shift) +{ + void *ptr = (void *)__get_free_page(PGALLOC_GFP); + BUG_ON(!ptr); + + /* Ensure the zeroed page is visible to the page table walker */ + dsb(ishst); + return __pa(ptr); +} + static phys_addr_t pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(PGALLOC_GFP); @@ -594,7 +604,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, - prot, pgd_pgtable_alloc, 0); + prot, pgd_kernel_pgtable_alloc, 0); /* Map both the text and data into the kernel page table */ __set_fixmap(FIX_ENTRY_TRAMP_TEXT, pa_start, prot); @@ -1070,7 +1080,8 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap, flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), - size, PAGE_KERNEL, pgd_pgtable_alloc, flags); + size, PAGE_KERNEL, pgd_kernel_pgtable_alloc, + flags); return __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap, want_memblock); From patchwork Tue Mar 12 00:57:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10848445 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99FE66C2 for ; Tue, 12 Mar 2019 00:58:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87733293FD for ; Tue, 12 Mar 2019 00:58:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8523129416; Tue, 12 Mar 2019 00:58:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 01A73293FD for ; Tue, 12 Mar 2019 00:58:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BCD6C8E0005; Mon, 11 Mar 2019 20:58:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B57448E0002; Mon, 11 Mar 2019 20:58:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6E308E0005; Mon, 11 Mar 2019 20:58:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 8A56A8E0002 for ; Mon, 11 Mar 2019 20:58:08 -0400 (EDT) Received: by mail-io1-f69.google.com with SMTP id r21so576872iod.12 for ; Mon, 11 Mar 2019 17:58:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=srduQGUMd+1T0rMo1nAAhODDuDFnWSZKCOfw6uJonls=; b=RsmKcunCUGDJ/N8dkCK40CFAfXm7Ql1HLB/IqcpNEkPPQ9E15MI2WKrjVU0zbWPDkB VCo03YfI1B7QIIuDEL5ntVhh0FnLlYXJLVe4jMeV1InX9EgxcHedrHUc6PUOaAsRct7n Si/xBCnjgKEuOzL2dICJqV2NOt3d0NZso9DuWt8mZVRx/nyWr3+Ams/UxR7i11ausECL yXZ1vTRZlS5+VBR3o+wBkJQG78jB2/ndu0g6Hti+rLZ79R7MohfGQta+JDHA6I7ebcgx wNWc7ANrJMfim5BIFXA+q/I76mz179+fXEBO9nAjb7U0+k7bnA4bkt/gbsAiDOiAZc0v UwAg== X-Gm-Message-State: APjAAAWZqAeFvYq6gyduN42jlYkJ58PREf8kyki4z8qEusnSnz8H/cdi 6CpJQY0z+/TtE4S8A2C2yR8J/pOuDovEeInGrd5VXq9W9VLE+5FyYRkSatI9DOgotYz7sKzXsCv vP8kkIDlSVRE5DC5IPrqc3S2t+LblmzvDnE+fgS86SrH6dXphmwxDzS4WOjrTe/yd3A== X-Received: by 2002:a24:1f84:: with SMTP id d126mr632481itd.67.1552352288385; Mon, 11 Mar 2019 17:58:08 -0700 (PDT) X-Received: by 2002:a24:1f84:: with SMTP id d126mr632460itd.67.1552352287602; Mon, 11 Mar 2019 17:58:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552352287; cv=none; d=google.com; s=arc-20160816; b=AXmlcfT9a3Bz+o4/1czILNuPvgPvVqyyhlBYikyS/WEEgKNaPPmcsm8MB1C5+fZlor ekoS2m8uMbmrdOeepRvnBYuo5KprhgWIy21KPvcjj+OGtV/VJTPs0zE/M8mxhckgQNpm Sy9xF8qcV2hnVG4hS6xBcslDG7kzbbOOFxTCv+aq9rNlXwhFVs2AkIFz28B4Q1/q0QhA 4Mv4ryYhf8MfMfdeyYyT6LbxuBXk8onPhnZwXnm/0AfToyhEJiH/AhG0l90h4ZsvvZLm zXAJPWWD6IbPC3iqbcVRRbHlIeKZWStC/p3BGM+6x2EtQoBjEdYMFIDRGc3XRF08V+ch HMqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=srduQGUMd+1T0rMo1nAAhODDuDFnWSZKCOfw6uJonls=; b=fnIFKXxcG1uIgPxxS5Zwc0TAqjy27eRMtZ3CNrQa9UMe4iCG8WRpWjmx5niPCPbdpq bfXAEiwlPQcPBNWSC4J2FgoFExM/VI383rYfYxhuS85q7lFHRTIcP6jzoLOZ8qpg9XAQ ln5P+kPN6P3pQb1PRxAUrh95iNpRDi14rQCFhJywn0trJBOo0ZlEhAcpxjdxbsuUiTFC kfZHqjLtiWSBzMTb88icaMqlGXNSkLxoLwv76LJiA09oStRlWmHgAubqi+huPBCoovHw IQqPUIZWDCxa0oFYICZa/J+6C33ZnhlFd8+tv4755PZLxDk6rDkufW7ZUyggdEwtvRQC iFHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="oedD/egg"; spf=pass (google.com: domain of 3hwshxaykcc4iejrkyqyyqvo.mywvsxeh-wwufkmu.ybq@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3HwSHXAYKCC4iejRKYQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id f74sor1342602itf.11.2019.03.11.17.58.07 for (Google Transport Security); Mon, 11 Mar 2019 17:58:07 -0700 (PDT) Received-SPF: pass (google.com: domain of 3hwshxaykcc4iejrkyqyyqvo.mywvsxeh-wwufkmu.ybq@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="oedD/egg"; spf=pass (google.com: domain of 3hwshxaykcc4iejrkyqyyqvo.mywvsxeh-wwufkmu.ybq@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3HwSHXAYKCC4iejRKYQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=srduQGUMd+1T0rMo1nAAhODDuDFnWSZKCOfw6uJonls=; b=oedD/eggmV+SIgYB11JrwAoZLhFWdb50S/XNtFqL8BtNX+SDe5Glssl7AopEbDqcNA HCKLJ/V3bjISgT2y2KxWp3VykVnCmF5Y6Eu7pAUj8ZigjSaNi2yJ90jUlcOctuxB1Fbm A3C127QaAa54k+3O0L7KF5QaqVgnfjPJr6a85bI04USgXp04seBuyvB1LjH9V5YVSO4n 9poZ+ZPTh65xeVRUyWd030y18tK6PcucnaLZg/I1TUKe2kC6W9D9AdSqG6CQmwyqYYKV 4olxX5RbFrrKkNYKcwWg7HTaRF8nnkjP+yLqbV7GoSTMRn558HpMjY8QWgCmjB5DLMAj nSOA== X-Google-Smtp-Source: APXvYqxqCOwYBxF/7ICjEeopmZPTb2uYLa3V2NjdBJYx9miVxqzRT+/6zNK7zIhloXUYTCZtvdQJhkLhhWk= X-Received: by 2002:a24:e984:: with SMTP id f126mr583102ith.1.1552352287403; Mon, 11 Mar 2019 17:58:07 -0700 (PDT) Date: Mon, 11 Mar 2019 18:57:48 -0600 In-Reply-To: <20190312005749.30166-1-yuzhao@google.com> Message-Id: <20190312005749.30166-3-yuzhao@google.com> Mime-Version: 1.0 References: <20190310011906.254635-1-yuzhao@google.com> <20190312005749.30166-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v4 3/4] arm64: mm: call ctor for stage2 pmd page From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland Cc: "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Joel Fernandes , "Kirill A . Shutemov" , Ard Biesheuvel , Chintan Pandya , Jun Yao , Laura Abbott , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Call pgtable_pmd_page_dtor() for pmd page allocated by mmu_memory_cache_alloc() so kernel won't crash when it's freed through stage2_pmd_free()->pmd_free()->pgtable_pmd_page_dtor(). This is needed if we are going to enable split pmd pt lock. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/stage2_pgtable.h | 15 ++++++++++++--- virt/kvm/arm/mmu.c | 13 +++++++++++-- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/stage2_pgtable.h b/arch/arm64/include/asm/stage2_pgtable.h index 5412fa40825e..0d9207144257 100644 --- a/arch/arm64/include/asm/stage2_pgtable.h +++ b/arch/arm64/include/asm/stage2_pgtable.h @@ -174,10 +174,19 @@ static inline bool stage2_pud_present(struct kvm *kvm, pud_t pud) return 1; } -static inline void stage2_pud_populate(struct kvm *kvm, pud_t *pud, pmd_t *pmd) +static inline int stage2_pud_populate(struct kvm *kvm, pud_t *pud, pmd_t *pmd) { - if (kvm_stage2_has_pmd(kvm)) - pud_populate(NULL, pud, pmd); + if (!kvm_stage2_has_pmd(kvm)) + return 0; + + /* paired with pgtable_pmd_page_dtor() in pmd_free() below */ + if (!pgtable_pmd_page_ctor(virt_to_page(pmd))) { + free_page((unsigned long)pmd); + return -ENOMEM; + } + + pud_populate(NULL, pud, pmd); + return 0; } static inline pmd_t *stage2_pmd_offset(struct kvm *kvm, diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index e9d28a7ca673..11922d84be83 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1037,6 +1037,7 @@ static pud_t *stage2_get_pud(struct kvm *kvm, struct kvm_mmu_memory_cache *cache static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, phys_addr_t addr) { + int ret; pud_t *pud; pmd_t *pmd; @@ -1048,7 +1049,9 @@ static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache if (!cache) return NULL; pmd = mmu_memory_cache_alloc(cache); - stage2_pud_populate(kvm, pud, pmd); + ret = stage2_pud_populate(kvm, pud, pmd); + if (ret) + return ERR_PTR(ret); get_page(virt_to_page(pud)); } @@ -1061,6 +1064,9 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache pmd_t *pmd, old_pmd; pmd = stage2_get_pmd(kvm, cache, addr); + if (IS_ERR(pmd)) + return PTR_ERR(pmd); + VM_BUG_ON(!pmd); old_pmd = *pmd; @@ -1198,6 +1204,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pte_t *new_pte, unsigned long flags) { + int ret; pud_t *pud; pmd_t *pmd; pte_t *pte, old_pte; @@ -1227,7 +1234,9 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, if (!cache) return 0; /* ignore calls from kvm_set_spte_hva */ pmd = mmu_memory_cache_alloc(cache); - stage2_pud_populate(kvm, pud, pmd); + ret = stage2_pud_populate(kvm, pud, pmd); + if (ret) + return ret; get_page(virt_to_page(pud)); } From patchwork Tue Mar 12 00:57:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10848449 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCFAC6C2 for ; Tue, 12 Mar 2019 00:58:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CA0F92930F for ; Tue, 12 Mar 2019 00:58:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C7F5829410; Tue, 12 Mar 2019 00:58:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B39B2930F for ; Tue, 12 Mar 2019 00:58:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AF9F8E0006; Mon, 11 Mar 2019 20:58:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 762EB8E0002; Mon, 11 Mar 2019 20:58:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62AA78E0006; Mon, 11 Mar 2019 20:58:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-it1-f197.google.com (mail-it1-f197.google.com [209.85.166.197]) by kanga.kvack.org (Postfix) with ESMTP id 3690F8E0002 for ; Mon, 11 Mar 2019 20:58:10 -0400 (EDT) Received: by mail-it1-f197.google.com with SMTP id i4so819791itb.1 for ; Mon, 11 Mar 2019 17:58:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=GuInaEquY9cKCo+/R3bLL1KFcpfX5BwU8EHfq1hu08M=; b=YH0maHXCKWg7jkGbpEqwIdYy7QjEpmIE76+xicZ3efAJbeC/dP/HwS+ils4FH8c9Fn s2ulVm0dd8qKQ8iE3iqP9gq8pi/4iLtisUMFmA+/nDiPc+eRXqPNnG3aEDauNDeeQLql E/ASGzNQdb0IEbQzRcz4K7GsERjDin3EzN7DkzLGKai+aOOQH9E2bEt9q7fz89pOBsoa 8h2P2rPYVyxSJJVl5aXNGTKJteEN5fAXTKKUpJWqXFPhBE+jpTjhyPE54NvnLUr3VK8U 0JgyJ9u3rL1sGvZfo1kGWHOekR1fE3GsfVYOHn9zZFVhyAUlQ6Rh5s1inm6DxEL5erLl 3czQ== X-Gm-Message-State: APjAAAXXZolywcdnutPSnhIVb26xmfHaM56O4iN0RKelDHWqJs1+LqGw P/SVInvHFclozaWiUISXOPKuc+mmt+sgcIiqD53AsDwFTzALaFyvzOw+zy3y4YLuOh7qsdOlNRS 0thl6i1COBNsvy2kOvHA0pLs4EdAWJ3fjJMdZ6JHzO5wQUZzM+CHuEN0Kulf3MJgLNw== X-Received: by 2002:a24:ac9:: with SMTP id 192mr663657itw.15.1552352289988; Mon, 11 Mar 2019 17:58:09 -0700 (PDT) X-Received: by 2002:a24:ac9:: with SMTP id 192mr663634itw.15.1552352289178; Mon, 11 Mar 2019 17:58:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552352289; cv=none; d=google.com; s=arc-20160816; b=ZKhGkO2Od9J1lLvwnk2fIZ5l0oISDqHAs07IXupHYQmpQKjPz+YyfKocBW+ZG0GYGw LO4ijSs2qTmT6VD0QO0KhqIrCqNBrYKoPUNC9jxYvlxuivXPD6A2Ybkh3hs7dMeHCa+B nhmiiM7rU8OLy8Jk3fGg7zoSAa+vx4mYldi66wdKM5RBrl3MQ85DYi2VdAhSrI7d68C6 l8Rf72aqJ75a+OUqn811mQUEBztV4g1Nz6tBjSFAfh4SUDJtVX+vQxnWwNo62dU29fV9 n9zv55Ny1hl3lj031wgCUqswSJXJlzVVB7Gd7rP9g7+pUCrukC/r5om5Sn8nvpFER4ph DsoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=GuInaEquY9cKCo+/R3bLL1KFcpfX5BwU8EHfq1hu08M=; b=F5OqJbVSXo/VgSwyX9HDsp5dV+X45jyvCu4J9PnnonK4dQ99AuJIUvDc8M0KIfzchq s1XSIXDsDKnB7lWDrM1fD/AkN/T4xKKyTByuCdnmYpYQM/T9+jttHDSJxsA6LeE3eIsI sTHleXfl/zuM8WjZhNx3EtyVzgalhWYYqRuXwUz4CL7JCQxsaLdtG7F51h/a3QbP82CW pUI1fZ0bX0E2i9XZtg9X2W+bzZv641jZ8NeHmbTne8PBPJanc559BNdgwU2cgPgJz1Rh tMS5z316QzQq16kJt5+n7K1Sc99MFJULtJv8tPofnE3/Jp4U9M/HPxMXdFNcfP67Q6CM wl/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="HBNdWh+/"; spf=pass (google.com: domain of 3iashxaykcc8jfkslzrzzrwp.nzxwtyfi-xxvglnv.zcr@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3IASHXAYKCC8jfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id b102sor1218270itd.30.2019.03.11.17.58.09 for (Google Transport Security); Mon, 11 Mar 2019 17:58:09 -0700 (PDT) Received-SPF: pass (google.com: domain of 3iashxaykcc8jfkslzrzzrwp.nzxwtyfi-xxvglnv.zcr@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="HBNdWh+/"; spf=pass (google.com: domain of 3iashxaykcc8jfkslzrzzrwp.nzxwtyfi-xxvglnv.zcr@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3IASHXAYKCC8jfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=GuInaEquY9cKCo+/R3bLL1KFcpfX5BwU8EHfq1hu08M=; b=HBNdWh+/H/UQD3+xTCC5Gk1vTOFdi8buqG7IOzQCbqWRid+PcTT/T4iKPfX1Muy57H mA9gcEpPcKUv1DaJBopn5SU1wwyHnz+qqvQzVQiKuH4IuAZHw8WmPDHRTekZ4RoJuPoI k8fyIUxtuU7UBUmfFhSlmXIqDkwkNUo2QrkJLaPkIgWzkEwroQQwBNFou6cniBzdTUeU pguYiLIcrrFrKPeCm6lJ2yAvRHkaGHnYB9ACSfOKeP/GH7Q1zppaEI6Nk5aSRMS5Mt3C 9GOsSCQ8e1rnKd9uYUOK1sUbi9ceU8L2Dtpr15+FePDmyFg81CK03OWXDcm0/xCUaK9T fuqw== X-Google-Smtp-Source: APXvYqxhul1R8mnVTm2ANYck/U0sQICDmCd8jsCIgw/3U8ahE12sg7YNMdshwxSVxfvSc2y++gz3qAvURMY= X-Received: by 2002:a24:6b55:: with SMTP id v82mr545054itc.37.1552352288828; Mon, 11 Mar 2019 17:58:08 -0700 (PDT) Date: Mon, 11 Mar 2019 18:57:49 -0600 In-Reply-To: <20190312005749.30166-1-yuzhao@google.com> Message-Id: <20190312005749.30166-4-yuzhao@google.com> Mime-Version: 1.0 References: <20190310011906.254635-1-yuzhao@google.com> <20190312005749.30166-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v4 4/4] arm64: mm: enable per pmd page table lock From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland Cc: "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Joel Fernandes , "Kirill A . Shutemov" , Ard Biesheuvel , Chintan Pandya , Jun Yao , Laura Abbott , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Switch from per mm_struct to per pmd page table lock by enabling ARCH_ENABLE_SPLIT_PMD_PTLOCK. This provides better granularity for large system. I'm not sure if there is contention on mm->page_table_lock. Given the option comes at no cost (apart from initializing more spin locks), why not enable it now. We only do so when pmd is not folded, so we don't mistakenly call pgtable_pmd_page_ctor() on pud or p4d in pgd_pgtable_alloc(). Signed-off-by: Yu Zhao --- arch/arm64/Kconfig | 3 +++ arch/arm64/include/asm/pgalloc.h | 12 +++++++++++- arch/arm64/include/asm/tlb.h | 5 ++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index cfbf307d6dc4..a3b1b789f766 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -872,6 +872,9 @@ config ARCH_WANT_HUGE_PMD_SHARE config ARCH_HAS_CACHE_LINE_SIZE def_bool y +config ARCH_ENABLE_SPLIT_PMD_PTLOCK + def_bool y if PGTABLE_LEVELS > 2 + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help--- diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 52fa47c73bf0..dabba4b2c61f 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -33,12 +33,22 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)__get_free_page(PGALLOC_GFP); + struct page *page; + + page = alloc_page(PGALLOC_GFP); + if (!page) + return NULL; + if (!pgtable_pmd_page_ctor(page)) { + __free_page(page); + return NULL; + } + return page_address(page); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); + pgtable_pmd_page_dtor(virt_to_page(pmdp)); free_page((unsigned long)pmdp); } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 106fdc951b6e..4e3becfed387 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,7 +62,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pmdp)); + struct page *page = virt_to_page(pmdp); + + pgtable_pmd_page_dtor(page); + tlb_remove_table(tlb, page); } #endif