From patchwork Wed Jun 27 14:13:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kani, Toshi" X-Patchwork-Id: 10491765 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id AFD87602B3 for ; Wed, 27 Jun 2018 14:15:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F09C29371 for ; Wed, 27 Jun 2018 14:15:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 92ED82937C; Wed, 27 Jun 2018 14:15:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 084FB29371 for ; Wed, 27 Jun 2018 14:15:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DEFB6B0269; Wed, 27 Jun 2018 10:15:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 199B66B026B; Wed, 27 Jun 2018 10:15:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8D8E6B026C; Wed, 27 Jun 2018 10:15:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot0-f197.google.com (mail-ot0-f197.google.com [74.125.82.197]) by kanga.kvack.org (Postfix) with ESMTP id B65776B0269 for ; Wed, 27 Jun 2018 10:15:53 -0400 (EDT) Received: by mail-ot0-f197.google.com with SMTP id r26-v6so1500737otk.17 for ; Wed, 27 Jun 2018 07:15:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=w+0JXpS8hSa76Pf6iu333oEE2EIqctZ76KMwuTkIxDY=; b=UIwo8+ejmMO8g/YWSJdTciASfSTHjTIkkHbLFsxnx7lhF8rxJBq+x/69jUYdgaYK+7 4BA06w5Hojxy3rMyZ856+r5v2iAAhMcA3Gm05+JHWkPcFSt3oCa9iZqUIT+BXH4uKpGT uKNvd9Jw9x916nWISdxd+f4PWlRpMvbl/FfJy9DfoelOh9siTzfF5Af/hv3x5SSIJiGP EMepyuj6WgW1ukfIg4x3BUdpv7dFtSu1UTZG0IAITcKLyoOXFnClFMzd36Uv186CQzPd 6TrGSvzF1rwSRop3asDYqD9HMBr7KoIb6VkNwihGZw7hOcKAP6E51aHesQCSS0lajGWQ 4o3Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of toshi.kani@hpe.com designates 15.241.140.75 as permitted sender) smtp.mailfrom=toshi.kani@hpe.com X-Gm-Message-State: APt69E2DLau8WxE/uGFs0o5urIoN5WuciMqNcqEjFcIGmsx0f8Xdm19q NklhzfN2dxpD8ydQd98IDnq5RS20OHIxOs3IrXbIJurszGEdd/3bre3aV4xQqCq5Sp73Hhx5lHt vdWpRYop1VA0lfGBQmpgBflX060nj54HERMOYK3IY8HmvSR/DPxKW3Ho+kzS6aEhPMg== X-Received: by 2002:aca:1205:: with SMTP id 5-v6mr3525207ois.173.1530108953527; Wed, 27 Jun 2018 07:15:53 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcMZ61g39Y9YMNGytKJ7VtxNSz1KJj2cF+b5q3GaXvlmOka8utReztvns5jmDCysKYBPw7V X-Received: by 2002:aca:1205:: with SMTP id 5-v6mr3525153ois.173.1530108952682; Wed, 27 Jun 2018 07:15:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530108952; cv=none; d=google.com; s=arc-20160816; b=HDcTFMLTHWiA5RELqm9/UQzNcNwz5iNmbT7RU14iBCOU3OkNnFGnHTOV9Attm9rKA3 A5uYBmbQnSdbPp1aheaSQBmUsZlfuUWFaA80YLe/JFVXI6cpJzdpOib7yqEE1IxSJAVA DofqW2CK8F18kG3FsuynWRglJh96GkyFi6KRaV/no9Sm3D1yAvKamw5W2xuFqoaHb3K1 y2cQ/CSvhvwF1geTeNk0Ta+b06hapo1I4kUKBsYiRjt14DDmOitU7wjDSlbaMUyoDsVu 8trlxL4wieKvQiXw/b1P2oCMnP8jHTeHp48MusAxaOb24W5g3eqltt+Rs3/akbDK0mjQ nBXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=w+0JXpS8hSa76Pf6iu333oEE2EIqctZ76KMwuTkIxDY=; b=m0Ceozt6KL5aaC5eeZFH5bopMXIzvQXyy3jyO98F7WD6HOrdZpnsmbMqMF5qxdH2nH FgVbHr3+ClwCfSLJEgp7YermZX0w1Q6WYLRZEJ4xJcyuMguAUkQNdmyvjcQPrvg82USW hzUnq7H9AboEmHmLdhe5SeoH7mzpzFdJb319dfsixic8cbFTxBPRuuPhwycSXm92rCCK Qcqs6s2fdArJfkFRWDzjbKPC2Lx+8KoLobiv4AvbOh+J6LhyuZxGYH6VGd/q13R9XFmF my+MWDJ/AOtbkDDVQfmgfev8Xo3588ssbTxHuA/LBHRuDPE8M7g2tqbwle5xOiME+q1X auwA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of toshi.kani@hpe.com designates 15.241.140.75 as permitted sender) smtp.mailfrom=toshi.kani@hpe.com Received: from g4t3426.houston.hpe.com (g4t3426.houston.hpe.com. [15.241.140.75]) by mx.google.com with ESMTPS id w204-v6si1358249oig.335.2018.06.27.07.15.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Jun 2018 07:15:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of toshi.kani@hpe.com designates 15.241.140.75 as permitted sender) client-ip=15.241.140.75; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of toshi.kani@hpe.com designates 15.241.140.75 as permitted sender) smtp.mailfrom=toshi.kani@hpe.com Received: from g4t3433.houston.hpecorp.net (g4t3433.houston.hpecorp.net [16.208.49.245]) by g4t3426.houston.hpe.com (Postfix) with ESMTP id 31B6061; Wed, 27 Jun 2018 14:15:52 +0000 (UTC) Received: from misato.americas.hpqcorp.net (unknown [10.34.81.122]) by g4t3433.houston.hpecorp.net (Postfix) with ESMTP id 840FD4A; Wed, 27 Jun 2018 14:15:51 +0000 (UTC) From: Toshi Kani To: mhocko@suse.com, akpm@linux-foundation.org, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com Cc: cpandya@codeaurora.org, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Toshi Kani , Joerg Roedel , stable@vger.kernel.org Subject: [PATCH v4 3/3] x86/mm: add TLB purge to free pmd/pte page interfaces Date: Wed, 27 Jun 2018 08:13:48 -0600 Message-Id: <20180627141348.21777-4-toshi.kani@hpe.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180627141348.21777-1-toshi.kani@hpe.com> References: <20180627141348.21777-1-toshi.kani@hpe.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP ioremap() calls pud_free_pmd_page() / pmd_free_pte_page() when it creates a pud / pmd map. The following preconditions are met at their entry. - All pte entries for a target pud/pmd address range have been cleared. - System-wide TLB purges have been peformed for a target pud/pmd address range. The preconditions assure that there is no stale TLB entry for the range. Speculation may not cache TLB entries since it requires all levels of page entries, including ptes, to have P & A-bits set for an associated address. However, speculation may cache pud/pmd entries (paging-structure caches) when they have P-bit set. Add a system-wide TLB purge (INVLPG) to a single page after clearing pud/pmd entry's P-bit. SDM 4.10.4.1, Operation that Invalidate TLBs and Paging-Structure Caches, states that: INVLPG invalidates all paging-structure caches associated with the current PCID regardless of the liner addresses to which they correspond. Fixes: 28ee90fe6048 ("x86/mm: implement free pmd/pte page interfaces") Signed-off-by: Toshi Kani Cc: Andrew Morton Cc: Michal Hocko Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin" Cc: Joerg Roedel Cc: --- arch/x86/mm/pgtable.c | 36 ++++++++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index fbd14e506758..e3deefb891da 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -725,24 +725,44 @@ int pmd_clear_huge(pmd_t *pmd) * @pud: Pointer to a PUD. * @addr: Virtual address associated with pud. * - * Context: The pud range has been unmaped and TLB purged. + * Context: The pud range has been unmapped and TLB purged. * Return: 1 if clearing the entry succeeded. 0 otherwise. + * + * NOTE: Callers must allow a single page allocation. */ int pud_free_pmd_page(pud_t *pud, unsigned long addr) { - pmd_t *pmd; + pmd_t *pmd, *pmd_sv; + pte_t *pte; int i; if (pud_none(*pud)) return 1; pmd = (pmd_t *)pud_page_vaddr(*pud); + pmd_sv = (pmd_t *)__get_free_page(GFP_KERNEL); + if (!pmd_sv) + return 0; - for (i = 0; i < PTRS_PER_PMD; i++) - if (!pmd_free_pte_page(&pmd[i], addr + (i * PMD_SIZE))) - return 0; + for (i = 0; i < PTRS_PER_PMD; i++) { + pmd_sv[i] = pmd[i]; + if (!pmd_none(pmd[i])) + pmd_clear(&pmd[i]); + } pud_clear(pud); + + /* INVLPG to clear all paging-structure caches */ + flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1); + + for (i = 0; i < PTRS_PER_PMD; i++) { + if (!pmd_none(pmd_sv[i])) { + pte = (pte_t *)pmd_page_vaddr(pmd_sv[i]); + free_page((unsigned long)pte); + } + } + + free_page((unsigned long)pmd_sv); free_page((unsigned long)pmd); return 1; @@ -753,7 +773,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) * @pmd: Pointer to a PMD. * @addr: Virtual address associated with pmd. * - * Context: The pmd range has been unmaped and TLB purged. + * Context: The pmd range has been unmapped and TLB purged. * Return: 1 if clearing the entry succeeded. 0 otherwise. */ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) @@ -765,6 +785,10 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) pte = (pte_t *)pmd_page_vaddr(*pmd); pmd_clear(pmd); + + /* INVLPG to clear all paging-structure caches */ + flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1); + free_page((unsigned long)pte); return 1;