From patchwork Mon Apr 22 18:57:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911495 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 00BA81708 for ; Mon, 22 Apr 2019 19:00:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB596286B0 for ; Mon, 22 Apr 2019 19:00:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DF0AD287C6; Mon, 22 Apr 2019 19:00:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4CE4328793 for ; Mon, 22 Apr 2019 19:00:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA4CB6B0285; Mon, 22 Apr 2019 15:00:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E558A6B0288; Mon, 22 Apr 2019 15:00:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D45BF6B0289; Mon, 22 Apr 2019 15:00:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 98D396B0285 for ; Mon, 22 Apr 2019 15:00:39 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id ba11so674527plb.21 for ; Mon, 22 Apr 2019 12:00:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=6pDLV47YuMug94kghJNMRU0wCJe8grVkxn5O7bN8YZE=; b=e2eD5iQ0Iej3aIeIrPGIAuc5WKeiF9vkziyQLkjGMhviLeChhKqWXD2zcYfe0ctgwm dL9aJ5hgIg0K8pzdFADqmhWRjUHuSKVIhN3xVGJxSNhbj2il7uPZYVrG/MzFkzc0Zykr KeJt/zS0FuV/fcfcCUQCs0kJWfgALq8n0YcfRRf5GwVYaGFCDu5QY3DYwKYgu14ep4Cm 9Qs08RmmN5T/CJsEbRoQzsYglEjstZYyvFw7x74w8zHB1plOFWXO97EZL+55gW7y0muc e4bDETjAwGpThjViTa3Dj4PuObYE0JzMBtbHRVE3TfiiLJBNg3GS4K2fCghGWCt5VdIj q1cg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWL+6xT+W5XANFSQg5JWcdTmXmaqHRYzngHltmb/n0t/Cwb+I1V a2MoWbE+sADDUX8GX6c2Iia2/KtNm1yMx/UyQ+6Nq1gALVIL31rpy6MitYJPTr60jkJjViYrifC sYq9KYNs36be/iBF2fKKAT/6X4wV00a2vIt9sPPF3yQfACGDopAR9VsH6pDZ77D1JdA== X-Received: by 2002:a62:b418:: with SMTP id h24mr21753604pfn.145.1555959639273; Mon, 22 Apr 2019 12:00:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqxjMSUn8Ju0Q5n6PWgVvWOnQ+sNQS1MeDYEf1boOgg6PaWuGsbtsSs8/ytRMv7ZlLjPXCs9 X-Received: by 2002:a62:b418:: with SMTP id h24mr21744436pfn.145.1555959523908; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=Gr4S5GLUtIdlh0G8UcbUzEMhpeJgDmwIK0O1GqxtT2lTCnMDm0UqEOon2a5sNi2rau ZJkAkW+PM2ksh/t4cYjUe77YPRnNY7e5XVC08SYN1doMloMdFG87WB3vFA6ImdyzKIAw SD5eoi7zRLF01lfu1bkLpV/ZqRKubmshOtm3J/G9wJ06dmi+mvqZpyxjGkCPGOQ7WTtK Q60GWmYZ5IVheuEuZKY7egulb0VTUMQm7jXep2nWFERi+/6FcU0NqyMUXdFRMScz0U/k /wHLuSPOK3WJyr3bmy27/ZLvFKsjYnWFYxumZoiZikx82f1TY/fcItAE5cDnjJDd417T +gjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=6pDLV47YuMug94kghJNMRU0wCJe8grVkxn5O7bN8YZE=; b=FoUQrYC836EEPYkDL42AiE9S9DEUUN9b3yUDCoVa5gK4jn3zUFmuL8yi2Vblyojwnt cVnYn1cucw2p/5+2iENfoKa2ThDFuwTbXjQ1N/CP37lKE/uQFam0kLARc8DguLFR/OJt nJKAZj6VRzZQY/zFAzekeMfVcis8gDEBZG2Qj3qc4CLUQOGLea1bkPqw7m8wHrLzSUMv L8SH0AO+nu0oAADEQUYfHLB1rM68vqHKccr47pLiAp1jJIKxqx60uUkaaeDbs14u7fBX eQTccvKNFFVIK5cD5Jl8NymKIMdiF0NjxS5Ew/+IDjAiLw10dvhx3Se3E208dIfI1kqy YihA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id w15si615875pga.591.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417161" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , "Rafael J. Wysocki" , Pavel Machek Subject: [PATCH v4 15/23] mm: Make hibernate handle unmapped pages Date: Mon, 22 Apr 2019 11:57:57 -0700 Message-Id: <20190422185805.1169-16-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Make hibernate handle unmapped pages on the direct map when CONFIG_ARCH_HAS_SET_ALIAS is set. These functions allow for setting pages to invalid configurations, so now hibernate should check if the pages have valid mappings and handle if they are unmapped when doing a hibernate save operation. Previously this checking was already done when CONFIG_DEBUG_PAGEALLOC was configured. It does not appear to have a big hibernating performance impact. The speed of the saving operation before this change was measured as 819.02 MB/s, and after was measured at 813.32 MB/s. Before: [ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s) After: [ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s) Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Borislav Petkov Acked-by: Pavel Machek Signed-off-by: Rick Edgecombe --- arch/x86/mm/pageattr.c | 4 ---- include/linux/mm.h | 18 ++++++------------ kernel/power/snapshot.c | 5 +++-- mm/page_alloc.c | 7 +++++-- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 3574550192c6..daf4d645e537 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2257,7 +2257,6 @@ int set_direct_map_default_noflush(struct page *page) return __set_pages_p(page, 1); } -#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2302,11 +2301,8 @@ bool kernel_page_present(struct page *page) pte = lookup_address((unsigned long)page_address(page), &level); return (pte_val(*pte) & _PAGE_PRESENT); } - #endif /* CONFIG_HIBERNATION */ -#endif /* CONFIG_DEBUG_PAGEALLOC */ - int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address, unsigned numpages, unsigned long page_flags) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 6b10c21630f5..083d7b4863ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2610,37 +2610,31 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; -extern void __kernel_map_pages(struct page *page, int numpages, int enable); static inline bool debug_pagealloc_enabled(void) { - return _debug_pagealloc_enabled; + return IS_ENABLED(CONFIG_DEBUG_PAGEALLOC) && _debug_pagealloc_enabled; } +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_ARCH_HAS_SET_DIRECT_MAP) +extern void __kernel_map_pages(struct page *page, int numpages, int enable); + static inline void kernel_map_pages(struct page *page, int numpages, int enable) { - if (!debug_pagealloc_enabled()) - return; - __kernel_map_pages(page, numpages, enable); } #ifdef CONFIG_HIBERNATION extern bool kernel_page_present(struct page *page); #endif /* CONFIG_HIBERNATION */ -#else /* CONFIG_DEBUG_PAGEALLOC */ +#else /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ static inline void kernel_map_pages(struct page *page, int numpages, int enable) {} #ifdef CONFIG_HIBERNATION static inline bool kernel_page_present(struct page *page) { return true; } #endif /* CONFIG_HIBERNATION */ -static inline bool debug_pagealloc_enabled(void) -{ - return false; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ #ifdef __HAVE_ARCH_GATE_AREA extern struct vm_area_struct *get_gate_vma(struct mm_struct *mm); diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index f08a1e4ee1d4..bc9558ab1e5b 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -1342,8 +1342,9 @@ static inline void do_copy_page(long *dst, long *src) * safe_copy_page - Copy a page in a safe way. * * Check if the page we are going to copy is marked as present in the kernel - * page tables (this always is the case if CONFIG_DEBUG_PAGEALLOC is not set - * and in that case kernel_page_present() always returns 'true'). + * page tables. This always is the case if CONFIG_DEBUG_PAGEALLOC or + * CONFIG_ARCH_HAS_SET_DIRECT_MAP is not set. In that case kernel_page_present() + * always returns 'true'. */ static void safe_copy_page(void *dst, struct page *s_page) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d96ca5bc555b..34a70681a4af 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1131,7 +1131,9 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); - kernel_map_pages(page, 1 << order, 0); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 0); + kasan_free_nondeferred_pages(page, order); return true; @@ -2001,7 +2003,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_refcounted(page); arch_alloc_page(page, order); - kernel_map_pages(page, 1 << order, 1); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); kernel_poison_pages(page, 1 << order, 1); set_page_owner(page, order, gfp_flags);