From patchwork Wed May 16 23:32:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kani, Toshi" X-Patchwork-Id: 10405067 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 462DD602C2 for ; Wed, 16 May 2018 23:33:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 364EF28861 for ; Wed, 16 May 2018 23:33:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A9C428859; Wed, 16 May 2018 23:33:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A207C28604 for ; Wed, 16 May 2018 23:33:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1F716B0377; Wed, 16 May 2018 19:33:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AA8096B0378; Wed, 16 May 2018 19:33:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9708D6B0379; Wed, 16 May 2018 19:33:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg0-f72.google.com (mail-pg0-f72.google.com [74.125.83.72]) by kanga.kvack.org (Postfix) with ESMTP id 4460F6B0377 for ; Wed, 16 May 2018 19:33:28 -0400 (EDT) Received: by mail-pg0-f72.google.com with SMTP id v26-v6so946485pgc.14 for ; Wed, 16 May 2018 16:33:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=o8sgoZjOxvubH3vqlI10o5Q7nh1e1mqbGOKF/LOEVSM=; b=YFksFlEq2Tl26hB0UXV3AFoaEoVLRW5TxfHMylp8Mn2gDyZSNxv7PDQcG7W1sY/3Jb MtsZ3sK9G1FPVTKtUDq7eObo51GdQCtSNXH47i1LMYeok9fnlu+ghOEdxCiK7mXHB/Ny hwS5R64yJw/RuK8sx0DW4oIl/finszwNYOJxF3lqwKzy488+NENUYNlOh1IgToiydD5P KL4LCWkkEImOPcMXqlp0sSA4T4xBjOxpgl+3vZKSb0m2YknGLDMAVhFLOczlF01dPTlp xcfl5IvS0cqM1UdZQA2ByOyttKo0TrVj++peFv62tew3PFg95ReeENBi0GidAE/dBkVu Wfsw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of toshi.kani@hpe.com designates 15.241.48.72 as permitted sender) smtp.mailfrom=toshi.kani@hpe.com X-Gm-Message-State: ALKqPwdkD1/JUbQ3Bs/flcBV3UtJhrnJI0GU3SWL/FGoyPBHMuEJxnWg mdFM51d+r+jhJjfczkOeFv5d3BANb4SQYu99mPkX4sFQAp52Uyxg6WHVYi2rXnpjxxsM2my2kyt 3h/Y2dTUVwPjbTUytS7zu2lleoME4aZgGIEh6nlUFeItFFGB+GMZ8or8icb5xnB3gpQ== X-Received: by 2002:a62:4a88:: with SMTP id c8-v6mr2938623pfj.23.1526513607946; Wed, 16 May 2018 16:33:27 -0700 (PDT) X-Google-Smtp-Source: AB8JxZoIKV2+YtKD8nszNGgUpOEJIeRWOlviZmnioq8OVHQ1aD2Od8m27F+4sIbXA7LGIiH9rPr0 X-Received: by 2002:a62:4a88:: with SMTP id c8-v6mr2938575pfj.23.1526513606989; Wed, 16 May 2018 16:33:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526513606; cv=none; d=google.com; s=arc-20160816; b=CgRI+NS0SKvQozCnCqQJ9sBDo5Dpmo+VmW+Z7KyLmK+i0fifYeAtMjeIYueTFZpkWC QznbVdhfrgi55y5kWIuBt0IV+PKMUAzBXMTBylrWB5MuPdCRvJSyLfHPpp0rJi2OA3Ds AdppIdTA0S9zMADEHJWGgpEbU9Rsc5AQTK2JjQOEogELdxTkNJDPlpMG74xcZySncRuc VfLUnkYDlGa0huhpSqdkC9+CggI2By5AYWi+ZyaiTk8Q4ap4h9LuOP6xVfdrtFe6CBAH qLbPe4Kq6Y5aQ8W4ce7q9JXmVYjRasxTTmYKwsDvlSJXNHOuHiuhq/6VGL8Ov8Bna1YR QALg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=o8sgoZjOxvubH3vqlI10o5Q7nh1e1mqbGOKF/LOEVSM=; b=WIq0YIUG63mytuqqjvWxxrBUFZMDCxWEWbqd0FpgcQ3xlPSepoihZG1Tk/wYijtk5t niWMBsw9OI3Z6MaLY640bnhXwv1WYCM7RRtviMNC8CUZOw3gbBSw6ZfE4eVGkMWfySXx G0UgE567L8/pXmONOLmsMh61WM2yE8bv6Zc/+zqg7zRwFIUJeFLe11EueOk6uPM+1Y1U jhHPmI78lFiMdib1zVX03MzHFEx0BqnzYcxY187sx2ZfT3z33W8Q2nDcm4Evmg8hwVQP rbq7kcthYXJ0SzBNn4GqO8k3z8XeQ8TpQO9kezPmF53twJ4ftCkhcXD2ZJ0WI3Xrew+x 3g+g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of toshi.kani@hpe.com designates 15.241.48.72 as permitted sender) smtp.mailfrom=toshi.kani@hpe.com Received: from g9t5008.houston.hpe.com (g9t5008.houston.hpe.com. [15.241.48.72]) by mx.google.com with ESMTPS id t13-v6si2997611pgq.358.2018.05.16.16.33.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 May 2018 16:33:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of toshi.kani@hpe.com designates 15.241.48.72 as permitted sender) client-ip=15.241.48.72; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of toshi.kani@hpe.com designates 15.241.48.72 as permitted sender) smtp.mailfrom=toshi.kani@hpe.com Received: from g9t2301.houston.hpecorp.net (g9t2301.houston.hpecorp.net [16.220.97.129]) by g9t5008.houston.hpe.com (Postfix) with ESMTP id 15C5F60; Wed, 16 May 2018 23:33:26 +0000 (UTC) Received: from misato.americas.hpqcorp.net (unknown [10.34.81.122]) by g9t2301.houston.hpecorp.net (Postfix) with ESMTP id 821878C9; Wed, 16 May 2018 23:33:25 +0000 (UTC) From: Toshi Kani To: mhocko@suse.com, akpm@linux-foundation.org, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com Cc: cpandya@codeaurora.org, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Toshi Kani , Joerg Roedel , stable@vger.kernel.org Subject: [PATCH v3 3/3] x86/mm: add TLB purge to free pmd/pte page interfaces Date: Wed, 16 May 2018 17:32:07 -0600 Message-Id: <20180516233207.1580-4-toshi.kani@hpe.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180516233207.1580-1-toshi.kani@hpe.com> References: <20180516233207.1580-1-toshi.kani@hpe.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP ioremap() calls pud_free_pmd_page() / pmd_free_pte_page() when it creates a pud / pmd map. The following preconditions are met at their entry. - All pte entries for a target pud/pmd address range have been cleared. - System-wide TLB purges have been peformed for a target pud/pmd address range. The preconditions assure that there is no stale TLB entry for the range. Speculation may not cache TLB entries since it requires all levels of page entries, including ptes, to have P & A-bits set for an associated address. However, speculation may cache pud/pmd entries (paging-structure caches) when they have P-bit set. Add a system-wide TLB purge (INVLPG) to a single page after clearing pud/pmd entry's P-bit. SDM 4.10.4.1, Operation that Invalidate TLBs and Paging-Structure Caches, states that: INVLPG invalidates all paging-structure caches associated with the current PCID regardless of the liner addresses to which they correspond. Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces") Signed-off-by: Toshi Kani Cc: Andrew Morton Cc: Michal Hocko Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin" Cc: Joerg Roedel Cc: --- arch/x86/mm/pgtable.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index f60fdf411103..7e96594c7e97 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -721,24 +721,42 @@ int pmd_clear_huge(pmd_t *pmd) * @pud: Pointer to a PUD. * @addr: Virtual address associated with pud. * - * Context: The pud range has been unmaped and TLB purged. + * Context: The pud range has been unmapped and TLB purged. * Return: 1 if clearing the entry succeeded. 0 otherwise. */ int pud_free_pmd_page(pud_t *pud, unsigned long addr) { - pmd_t *pmd; + pmd_t *pmd, *pmd_sv; + pte_t *pte; int i; if (pud_none(*pud)) return 1; pmd = (pmd_t *)pud_page_vaddr(*pud); + pmd_sv = (pmd_t *)__get_free_page(GFP_KERNEL); + if (!pmd_sv) + return 0; - for (i = 0; i < PTRS_PER_PMD; i++) - if (!pmd_free_pte_page(&pmd[i], addr + (i * PMD_SIZE))) - return 0; + for (i = 0; i < PTRS_PER_PMD; i++) { + pmd_sv[i] = pmd[i]; + if (!pmd_none(pmd[i])) + pmd_clear(&pmd[i]); + } pud_clear(pud); + + /* INVLPG to clear all paging-structure caches */ + flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1); + + for (i = 0; i < PTRS_PER_PMD; i++) { + if (!pmd_none(pmd_sv[i])) { + pte = (pte_t *)pmd_page_vaddr(pmd_sv[i]); + free_page((unsigned long)pte); + } + } + + free_page((unsigned long)pmd_sv); free_page((unsigned long)pmd); return 1; @@ -749,7 +767,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) * @pmd: Pointer to a PMD. * @addr: Virtual address associated with pmd. * - * Context: The pmd range has been unmaped and TLB purged. + * Context: The pmd range has been unmapped and TLB purged. * Return: 1 if clearing the entry succeeded. 0 otherwise. */ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) @@ -761,6 +779,10 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) pte = (pte_t *)pmd_page_vaddr(*pmd); pmd_clear(pmd); + + /* INVLPG to clear all paging-structure caches */ + flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1); + free_page((unsigned long)pte); return 1;