From patchwork Sat May 13 22:04:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD837C7EE23 for ; Sat, 13 May 2023 22:04:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02AD56B0072; Sat, 13 May 2023 18:04:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E80806B0074; Sat, 13 May 2023 18:04:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAA936B0078; Sat, 13 May 2023 18:04:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BA5D26B0072 for ; Sat, 13 May 2023 18:04:49 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8EC5714107F for ; Sat, 13 May 2023 22:04:49 +0000 (UTC) X-FDA: 80786612298.12.B6FAD63 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf24.hostedemail.com (Postfix) with ESMTP id 52CB318000E for ; Sat, 13 May 2023 22:04:46 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=HM4raheH; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015486; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iPQ1KKQFU+70EX6OsrB6mZxP6pXOkY+MHFmNkE+acIM=; b=VfUqWrgBnyCWBWcBfxEJAMS7kkEFps6xpoW6usHXSHVYOUWED0Ru0PfwF48ZnORfT/lRlj Ijk72LWlPP/j35lB5TR5GPZZdCle3IS6novODa7T9YfcYXNwcIUyvGFg63VGCelcmjqAeV sPikswJJcaRu8RkFEU/ChizpWTaO8HQ= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=HM4raheH; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015486; a=rsa-sha256; cv=none; b=RR0I7BkAo4qf5WbrfoHxFuwLXwZE2qG9ZaEqfhHnouV0aXqVhsZi7VYcPezkuxQ81YlvlT pAn4pHondI7r2b8YTRiZcuDfzI3mJVPdQLXBrq6mI7onYSS+f7Hyxr6Ig2yM1B1diCMvC0 MWJk/zNv4bw1o54+JcUvPE2jEhLz1fE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015486; x=1715551486; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vmI9ynFUPv+78hxPgRL1E8+NKhpjn7Q2/GskPdLFBX4=; b=HM4raheHTcad7Gfh/3C5igWJES85r4JPDuG8G38soNenvGHx2RLvPvym ss2Zp+FMu8sllgpI9ShKs0O5P+m4a/h6AfPx5rvib34e32BP/IqYomhCW DhJ2aFQLeebGIVfdlDMw3OTuFcaLDJrnJ+fEKDQf3KbmETWkvG/kW1P/a 8IVq2pXak1K2ovLkfupQGqn9f25kl29BlIxPeZlYezpkpn678bOYu2v1p rh9txRkEc/jgIYtvO98rgdF86vPpMbSuvaKO3VmH/IQ9eUZxZffU6x6tD k8qwMCZ3+jJSVBFBK399XfsCDUqPBkBxFtzjT29ZjImp+xnFGXPE+2+Mk A==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="379142524" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="379142524" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="733429547" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="733429547" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:36 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id D4AF010CF77; Sun, 14 May 2023 01:04:33 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Mike Rapoport Subject: [PATCHv11 1/9] mm: Add support for unaccepted memory Date: Sun, 14 May 2023 01:04:10 +0300 Message-Id: <20230513220418.19357-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: 4fu9tqeud4595mcgk77669rb8xrhtpsj X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 52CB318000E X-Rspam-User: X-HE-Tag: 1684015486-185004 X-HE-Meta: U2FsdGVkX1/5SXSdoKrlWWtkukBzqrJrp7VYSqH3nTh8P2Ah/ZT7jxZ6GeryJ4ByXaQpQASpVc7/MWSu3uTwn3JwVwo//0B1xuntK4YjJOfUM1VlIh5gF8Htapxj40rVCTC8fhw8JRABn2R2g1hz26Lr8RKdkF30nHNt0XCd4spwyhxjipkg1DbGecelSGJR2vXxwPtqvnkXx6z0sJF9m1o5083CDumQqWjNP/2n/sXs8tNy1lreiB+TBVf3BWsqn2gpN5jb2Radvlg+k4c/NEY5313hISAbrs5C9zTDrdiixQez8BsqdhpMrBwyLrHgjDEDKnFQXAqB1r6MSAMUbGpIMc2rw9iJxZb1GuEyIAPxGb4HG8kb06AXf6mzrQjLrlinvusaDgPN0fKElEm/wx4N25KB3ULwlxV792REcxehDy3ZFshnXJvNcvyHI1ZgiwYAIIigdGlI/ctkC7RtjQJeyqZJFXxcFyiPfe7u1MOYJxMtW0keSwIznrYwqryeF7UY+QWD4/oDBykG10QzTSUR6cSKow18yAYtqe6Tzd1grJ1PSBvHno4lr8dZBFOWbsGk8s0K5Y4CI1Tw7AM1z4f27KME1aPRWSqY8Q6V2QBaOeynx0fP/ZPBm+QXP7zL6Ry5Ru6OTn563BxtbAhuTc4ug5qVrpgmfzOawzJjyTQc8awRWUzHwLLQKvqSls9sn+vEHlzo2scdFvRB46G5uUKEeQNlm2cXhfsHJNQlFedMIemo/PRmmscg8Jx3QwQDsJkd1U4YQ1c0tyEVmYAs8b+dhFxAGjlwGjZZCMi7HC6xKaAf8O/Xl+u//Pm7FPx3I8JanUYaH6Of7PIEyvlXMtw3FOCiBO8B+H1PYDY6dQMthj4hzgFEzg4LezvbFf09/rzIoi5R+AW2VGPIpIMeYABLxiXACIKhHuGUWF042E6WWShnl4iUoYV0TgDzp3XAd7eHnLo0xtcFbYOOXcY c6O1Vt7i sECR3bkLCyy3hdfkdiuijv82zj2aEhdmIHJw6/Wr8yXwf4Dju1uMtSLL1pQWfZYT9Ux9bg1YEqN3KqevyjRC+oIUIbG4WOmqFfoKKrtIx+LDLHQ/pZxa42hcFlHfyQqclkc2/R34yMkr561FTNZgSwi5MQiW44L7cYo6U8naK59dfBUE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: UEFI Specification version 2.9 introduces the concept of memory acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, require memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific to the Virtual Machine platform. There are several ways kernel can deal with unaccepted memory: 1. Accept all the memory during the boot. It is easy to implement and it doesn't have runtime cost once the system is booted. The downside is very long boot time. Accept can be parallelized to multiple CPUs to keep it manageable (i.e. via DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate memory bandwidth and does not scale beyond the point. 2. Accept a block of memory on the first use. It requires more infrastructure and changes in page allocator to make it work, but it provides good boot time. On-demand memory accept means latency spikes every time kernel steps onto a new memory block. The spikes will go away once workload data set size gets stabilized or all memory gets accepted. 3. Accept all memory in background. Introduce a thread (or multiple) that gets memory accepted proactively. It will minimize time the system experience latency spikes on memory allocation while keeping low boot time. This approach cannot function on its own. It is an extension of #2: background memory acceptance requires functional scheduler, but the page allocator may need to tap into unaccepted memory before that. The downside of the approach is that these threads also steal CPU cycles and memory bandwidth from the user's workload and may hurt user experience. The patch implements #1 and #2 for now. #2 is the default. Some workloads may want to use #1 with accept_memory=eager in kernel command line. #3 can be implemented later based on user's demands. Support of unaccepted memory requires a few changes in core-mm code: - memblock has to accept memory on allocation; - page allocator has to accept memory on the first allocation of the page; Memblock change is trivial. The page allocator is modified to accept pages. New memory gets accepted before putting pages on free lists. It is done lazily: only accept new pages when we run out of already accepted memory. The memory gets accepted until the high watermark is reached. EFI code will provide two helpers if the platform supports unaccepted memory: - accept_memory() makes a range of physical addresses accepted. - range_contains_unaccepted_memory() checks anything within the range of physical addresses requires acceptance. Signed-off-by: Kirill A. Shutemov Acked-by: Mike Rapoport # memblock Reviewed-by: Vlastimil Babka --- drivers/base/node.c | 7 ++ fs/proc/meminfo.c | 5 ++ include/linux/mm.h | 19 +++++ include/linux/mmzone.h | 8 ++ mm/internal.h | 1 + mm/memblock.c | 9 +++ mm/mm_init.c | 7 ++ mm/page_alloc.c | 173 +++++++++++++++++++++++++++++++++++++++++ mm/vmstat.c | 3 + 9 files changed, 232 insertions(+) diff --git a/drivers/base/node.c b/drivers/base/node.c index b46db17124f3..655975946ef6 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -448,6 +448,9 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d ShmemPmdMapped: %8lu kB\n" "Node %d FileHugePages: %8lu kB\n" "Node %d FilePmdMapped: %8lu kB\n" +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + "Node %d Unaccepted: %8lu kB\n" #endif , nid, K(node_page_state(pgdat, NR_FILE_DIRTY)), @@ -477,6 +480,10 @@ static ssize_t node_read_meminfo(struct device *dev, nid, K(node_page_state(pgdat, NR_SHMEM_PMDMAPPED)), nid, K(node_page_state(pgdat, NR_FILE_THPS)), nid, K(node_page_state(pgdat, NR_FILE_PMDMAPPED)) +#endif +#ifdef CONFIG_UNACCEPTED_MEMORY + , + nid, K(sum_zone_node_page_state(nid, NR_UNACCEPTED)) #endif ); len += hugetlb_report_node_meminfo(buf, len, nid); diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index b43d0bd42762..8dca4d6d96c7 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -168,6 +168,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v) global_zone_page_state(NR_FREE_CMA_PAGES)); #endif +#ifdef CONFIG_UNACCEPTED_MEMORY + show_val_kb(m, "Unaccepted: ", + global_zone_page_state(NR_UNACCEPTED)); +#endif + hugetlb_report_meminfo(m); arch_report_meminfo(m); diff --git a/include/linux/mm.h b/include/linux/mm.h index 27ce77080c79..d9174d464348 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3816,4 +3816,23 @@ madvise_set_anon_name(struct mm_struct *mm, unsigned long start, } #endif +#ifdef CONFIG_UNACCEPTED_MEMORY + +bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end); +void accept_memory(phys_addr_t start, phys_addr_t end); + +#else + +static inline bool range_contains_unaccepted_memory(phys_addr_t start, + phys_addr_t end) +{ + return false; +} + +static inline void accept_memory(phys_addr_t start, phys_addr_t end) +{ +} + +#endif + #endif /* _LINUX_MM_H */ diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a4889c9d4055..6c1c2fc13017 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -143,6 +143,9 @@ enum zone_stat_item { NR_ZSPAGES, /* allocated in zsmalloc */ #endif NR_FREE_CMA_PAGES, +#ifdef CONFIG_UNACCEPTED_MEMORY + NR_UNACCEPTED, +#endif NR_VM_ZONE_STAT_ITEMS }; enum node_stat_item { @@ -910,6 +913,11 @@ struct zone { /* free areas of different sizes */ struct free_area free_area[MAX_ORDER + 1]; +#ifdef CONFIG_UNACCEPTED_MEMORY + /* Pages to be accepted. All pages on the list are MAX_ORDER */ + struct list_head unaccepted_pages; +#endif + /* zone flags, see below */ unsigned long flags; diff --git a/mm/internal.h b/mm/internal.h index 68410c6d97ac..b1db7ba5f57d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1099,4 +1099,5 @@ struct vma_prepare { struct vm_area_struct *remove; struct vm_area_struct *remove2; }; + #endif /* __MM_INTERNAL_H */ diff --git a/mm/memblock.c b/mm/memblock.c index 3feafea06ab2..50b921119600 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -1436,6 +1436,15 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, */ kmemleak_alloc_phys(found, size, 0); + /* + * Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, + * require memory to be accepted before it can be used by the + * guest. + * + * Accept the memory of the allocated buffer. + */ + accept_memory(found, found + size); + return found; } diff --git a/mm/mm_init.c b/mm/mm_init.c index 7f7f9c677854..1cfc08e25f93 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1375,6 +1375,10 @@ static void __meminit zone_init_free_lists(struct zone *zone) INIT_LIST_HEAD(&zone->free_area[order].free_list[t]); zone->free_area[order].nr_free = 0; } + +#ifdef CONFIG_UNACCEPTED_MEMORY + INIT_LIST_HEAD(&zone->unaccepted_pages); +#endif } void __meminit init_currently_empty_zone(struct zone *zone, @@ -1960,6 +1964,9 @@ static void __init deferred_free_range(unsigned long pfn, return; } + /* Accept chunks smaller than MAX_ORDER upfront */ + accept_memory(PFN_PHYS(pfn), PFN_PHYS(pfn + nr_pages)); + for (i = 0; i < nr_pages; i++, page++, pfn++) { if (pageblock_aligned(pfn)) set_pageblock_migratetype(page, MIGRATE_MOVABLE); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 47421bedc12b..d239fba3f31c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -387,6 +387,12 @@ EXPORT_SYMBOL(nr_node_ids); EXPORT_SYMBOL(nr_online_nodes); #endif +static bool page_contains_unaccepted(struct page *page, unsigned int order); +static void accept_page(struct page *page, unsigned int order); +static bool try_to_accept_memory(struct zone *zone, unsigned int order); +static inline bool has_unaccepted_memory(void); +static bool __free_unaccepted(struct page *page); + int page_group_by_mobility_disabled __read_mostly; #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT @@ -1481,6 +1487,13 @@ void __free_pages_core(struct page *page, unsigned int order) atomic_long_add(nr_pages, &page_zone(page)->managed_pages); + if (page_contains_unaccepted(page, order)) { + if (order == MAX_ORDER && __free_unaccepted(page)) + return; + + accept_page(page, order); + } + /* * Bypass PCP and place fresh pages right to the tail, primarily * relevant for memory onlining. @@ -3159,6 +3172,9 @@ static inline long __zone_watermark_unusable_free(struct zone *z, if (!(alloc_flags & ALLOC_CMA)) unusable_free += zone_page_state(z, NR_FREE_CMA_PAGES); #endif +#ifdef CONFIG_UNACCEPTED_MEMORY + unusable_free += zone_page_state(z, NR_UNACCEPTED); +#endif return unusable_free; } @@ -3458,6 +3474,11 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask)) { int ret; + if (has_unaccepted_memory()) { + if (try_to_accept_memory(zone, order)) + goto try_this_zone; + } + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* * Watermark failed for this zone, but see if we can @@ -3510,6 +3531,11 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, return page; } else { + if (has_unaccepted_memory()) { + if (try_to_accept_memory(zone, order)) + goto try_this_zone; + } + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* Try again if zone has deferred pages */ if (deferred_pages_enabled()) { @@ -7215,3 +7241,150 @@ bool has_managed_dma(void) return false; } #endif /* CONFIG_ZONE_DMA */ + +#ifdef CONFIG_UNACCEPTED_MEMORY + +/* Counts number of zones with unaccepted pages. */ +static DEFINE_STATIC_KEY_FALSE(zones_with_unaccepted_pages); + +static bool lazy_accept = true; + +static int __init accept_memory_parse(char *p) +{ + if (!strcmp(p, "lazy")) { + lazy_accept = true; + return 0; + } else if (!strcmp(p, "eager")) { + lazy_accept = false; + return 0; + } else { + return -EINVAL; + } +} +early_param("accept_memory", accept_memory_parse); + +static bool page_contains_unaccepted(struct page *page, unsigned int order) +{ + phys_addr_t start = page_to_phys(page); + phys_addr_t end = start + (PAGE_SIZE << order); + + return range_contains_unaccepted_memory(start, end); +} + +static void accept_page(struct page *page, unsigned int order) +{ + phys_addr_t start = page_to_phys(page); + + accept_memory(start, start + (PAGE_SIZE << order)); +} + +static bool try_to_accept_memory_one(struct zone *zone) +{ + unsigned long flags; + struct page *page; + bool last; + + if (list_empty(&zone->unaccepted_pages)) + return false; + + spin_lock_irqsave(&zone->lock, flags); + page = list_first_entry_or_null(&zone->unaccepted_pages, + struct page, lru); + if (!page) { + spin_unlock_irqrestore(&zone->lock, flags); + return false; + } + + list_del(&page->lru); + last = list_empty(&zone->unaccepted_pages); + + __mod_zone_freepage_state(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); + __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); + spin_unlock_irqrestore(&zone->lock, flags); + + accept_page(page, MAX_ORDER); + + __free_pages_ok(page, MAX_ORDER, FPI_TO_TAIL); + + if (last) + static_branch_dec(&zones_with_unaccepted_pages); + + return true; +} + +static bool try_to_accept_memory(struct zone *zone, unsigned int order) +{ + long to_accept; + int ret = false; + + /* How much to accept to get to high watermark? */ + to_accept = high_wmark_pages(zone) - + (zone_page_state(zone, NR_FREE_PAGES) - + __zone_watermark_unusable_free(zone, order, 0)); + + /* Accept at least one page */ + do { + if (!try_to_accept_memory_one(zone)) + break; + ret = true; + to_accept -= MAX_ORDER_NR_PAGES; + } while (to_accept > 0); + + return ret; +} + +static inline bool has_unaccepted_memory(void) +{ + return static_branch_unlikely(&zones_with_unaccepted_pages); +} + +static bool __free_unaccepted(struct page *page) +{ + struct zone *zone = page_zone(page); + unsigned long flags; + bool first = false; + + if (!lazy_accept) + return false; + + spin_lock_irqsave(&zone->lock, flags); + first = list_empty(&zone->unaccepted_pages); + list_add_tail(&page->lru, &zone->unaccepted_pages); + __mod_zone_freepage_state(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); + __mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES); + spin_unlock_irqrestore(&zone->lock, flags); + + if (first) + static_branch_inc(&zones_with_unaccepted_pages); + + return true; +} + +#else + +static bool page_contains_unaccepted(struct page *page, unsigned int order) +{ + return false; +} + +static void accept_page(struct page *page, unsigned int order) +{ +} + +static bool try_to_accept_memory(struct zone *zone, unsigned int order) +{ + return false; +} + +static inline bool has_unaccepted_memory(void) +{ + return false; +} + +static bool __free_unaccepted(struct page *page) +{ + BUILD_BUG(); + return false; +} + +#endif /* CONFIG_UNACCEPTED_MEMORY */ diff --git a/mm/vmstat.c b/mm/vmstat.c index c28046371b45..282349cabf01 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1180,6 +1180,9 @@ const char * const vmstat_text[] = { "nr_zspages", #endif "nr_free_cma", +#ifdef CONFIG_UNACCEPTED_MEMORY + "nr_unaccepted", +#endif /* enum numa_stat_item counters */ #ifdef CONFIG_NUMA From patchwork Sat May 13 22:04:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5ED49C7EE2A for ; Sat, 13 May 2023 22:04:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84000900002; Sat, 13 May 2023 18:04:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8189C280001; Sat, 13 May 2023 18:04:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 66A63900002; Sat, 13 May 2023 18:04:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4A04D900004 for ; Sat, 13 May 2023 18:04:52 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 18F92A1179 for ; Sat, 13 May 2023 22:04:52 +0000 (UTC) X-FDA: 80786612424.03.9D56217 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf04.hostedemail.com (Postfix) with ESMTP id B7DF540008 for ; Sat, 13 May 2023 22:04:49 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UwXwwpzp; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf04.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8eER3nj4kauYEk3c3ghrCd1PHqJJZIa9/Zb2oGBx6Ro=; b=7ooaM1X1W3x18SJFuncpxtNZE72zDNa4BZMEh6VD5leZVyN8iWnd9i6ASHPhP/hjAqWwmL MLQDoGab+aO+LQ7g/jF0tbgRtQMDs8a+pmVOIU3exgIQafQ69BcbPNw+c1i5o90G6afybH yew+7EcEBk+TIAlZZFKPKTOIkrug8nA= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UwXwwpzp; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf04.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015490; a=rsa-sha256; cv=none; b=iB4C7W+QPFQHT8LoCNiuGjbiTziZBxb7ALImANNy2Q4FPv6QxObi+2gjdcW2XUop46K+O7 vvFNwnidKrx04ELORmJEHHeib+aTcNWL3b8ptb2Ce5jGDIjl5bBAxAn1x+KOpibu+boKN2 j6WVbcfq9F3CDO4bSWUM8r66dLW+aA0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015490; x=1715551490; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=F0wEDPSvMabnYNajFJWO6701ciPheAFu0TDqu98qRss=; b=UwXwwpzpv59CCd2ccaCSzmV0wTKO9unu1812lcfwmyDhFYVxZpvrCmCZ aG9zgkiqQ7NmoPq5+fITKd/ztwJ0eNdeHpWdZdJzmXuIGI4soMaiqxSy8 Uk0nj71rKH7rnyTbFkIEPAOENzExtitztgkKtpkCCzJ/l6qPnq20W6vOl gJeAz3S2+6uBcIz37ouMPB7G3YWgyV91qt7g553LALLtDwjam2bfPa+GA +UT3B5o7D2L65Q7rrOvzlrUZ9R4BiV27nYjFkjxGbGhjh8MlBc2l4JNwj u2OuVbiqrs+c3Qa/VYd1EYw6aoTfuNYf9mrR+i0TUrxsufW1Ir9pmTi6f A==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="437325178" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="437325178" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="790199649" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="790199649" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:36 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id E090C10CF78; Sun, 14 May 2023 01:04:33 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Borislav Petkov Subject: [PATCHv11 2/9] efi/x86: Get full memory map in allocate_e820() Date: Sun, 14 May 2023 01:04:11 +0300 Message-Id: <20230513220418.19357-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: bpfcucnci89d46h4gwh4zwo6un5uwaxx X-Rspam-User: X-Rspamd-Queue-Id: B7DF540008 X-Rspamd-Server: rspam07 X-HE-Tag: 1684015489-759982 X-HE-Meta: U2FsdGVkX18uaCzkfExT8J6AuID2PPQPcQDV/jsVb8paoyQQA3hXIYV6llhGZ0AGOe+P4939JYH9CU77xdf80KNnYEsveQzDpjLzUNOYjEK4ZDYiH5+QYzYG5ApXE3MpaOtvc0gAVXG1debcYDHo5Lw/A0AHdIQ0UqwgF5qbYA4iB7IRW2ktZs0fRQKP2YiK7NRkg4V/hI7Fqgg6KvyljShXra9SMd0ZQgL9+lSPUSe49z9ld6ttmkg3dSMPWN16Su0Ee7pAlwCv6DQRJMjOOyQ0yVacbO3JS6zUnSxhGC3f2Cz0MzGJeWvNhx0JOd7Si36mnMeCFftRGCg43m18V5FoOVImcvpGCHtBwDPyrmfDXqZlGWyU34GMsgsmXD95hql1h1LnX70xun3Dda6oioa+Gl7AyBKb2KDUhOxmPF8XAsF00dG800bdWJw6gqTMk130mnhBwRKgsdCNndXUYF8zwW56IWplKmV+Q0fyNAWV+tF9sqfzdnWyk2sh++fT9Zqe5XiJFp87DrOcXeaHBdF8aToyTNgWxta63QHVUXFKVwE0B+TUe4tvWPec/ufZ+08CrbkRzHxP2YKdRsMhsG4TjXLt1yNUH1ooxZ0yM+mpRbqNP4kWpc371npZxqgCmmDGX5gz4lf+LZkzJnwSXNbH1il0w2vUENMa2zn7D0YbMdkbA0mtI9MmWxZvCnte5h801E97QWNgOlbUXg8+EK3Jt1sN2VcNjolDsi3FPi19/XX9PWcMTJoo4P1dqU9uBRVqS2LVMaf1oC29d+wpW1cSjUmy926bhLm1yuPsAOTDEhQpVlHtPOrX4bLBX0/EzR4j2thVs4YGkBijIZ4C54Y3usTvwCPQNTfg+wdJnGwoaNDaAcoybAoe/dWUdNHq4K9SYIa6U73JfXZc6o3ZZuUHirarFUb6xIF2/gYAtIB793z5QwWbdH8LGsm/qFRrC00l0WwkJalC1SAlu/U KxeV3qh0 vmKha0PeaLimp2a5IK1qtPPtu4qHfI1oqC5N+1g6ofpHMPzAZ3R6GK+4DxPbiZ76cJDrTlNuWoCENa7V2BbVZ+3zkwpeHcscNUNkHf/RpUpE8qtqtbI/A62ZBYdcJHf+7vaR4Hlh49izjhiFYrfwh43g+RUa28JCuhDtrq4+Av0m38lNPPjuCG8jlcYzNT7tEkKRF6ygb4SgI+Jsa0TW+8l9ip3vvTF4NBGAuPv3RgZCpZBdA/8KrtAbNjg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently allocate_e820() is only interested in the size of map and size of memory descriptor to determine how many e820 entries the kernel needs. UEFI Specification version 2.9 introduces a new memory type -- unaccepted memory. To track unaccepted memory kernel needs to allocate a bitmap. The size of the bitmap is dependent on the maximum physical address present in the system. A full memory map is required to find the maximum address. Modify allocate_e820() to get a full memory map. Signed-off-by: Kirill A. Shutemov Reviewed-by: Borislav Petkov Acked-by: Ard Biesheuvel Reviewed-by: Tom Lendacky --- drivers/firmware/efi/libstub/x86-stub.c | 26 +++++++++++-------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index a0bfd31358ba..fff81843169c 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -681,28 +681,24 @@ static efi_status_t allocate_e820(struct boot_params *params, struct setup_data **e820ext, u32 *e820ext_size) { - unsigned long map_size, desc_size, map_key; + struct efi_boot_memmap *map; efi_status_t status; - __u32 nr_desc, desc_version; + __u32 nr_desc; - /* Only need the size of the mem map and size of each mem descriptor */ - map_size = 0; - status = efi_bs_call(get_memory_map, &map_size, NULL, &map_key, - &desc_size, &desc_version); - if (status != EFI_BUFFER_TOO_SMALL) - return (status != EFI_SUCCESS) ? status : EFI_UNSUPPORTED; + status = efi_get_memory_map(&map, false); + if (status != EFI_SUCCESS) + return status; - nr_desc = map_size / desc_size + EFI_MMAP_NR_SLACK_SLOTS; - - if (nr_desc > ARRAY_SIZE(params->e820_table)) { - u32 nr_e820ext = nr_desc - ARRAY_SIZE(params->e820_table); + nr_desc = map->map_size / map->desc_size; + if (nr_desc > ARRAY_SIZE(params->e820_table) - EFI_MMAP_NR_SLACK_SLOTS) { + u32 nr_e820ext = nr_desc - ARRAY_SIZE(params->e820_table) + + EFI_MMAP_NR_SLACK_SLOTS; status = alloc_e820ext(nr_e820ext, e820ext, e820ext_size); - if (status != EFI_SUCCESS) - return status; } - return EFI_SUCCESS; + efi_bs_call(free_pool, map); + return status; } struct exit_boot_struct { From patchwork Sat May 13 22:04:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16223C77B7D for ; Sat, 13 May 2023 22:04:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 312656B0074; Sat, 13 May 2023 18:04:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C21A900002; Sat, 13 May 2023 18:04:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C7B66B007B; Sat, 13 May 2023 18:04:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ECF106B0074 for ; Sat, 13 May 2023 18:04:50 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B5871A117B for ; Sat, 13 May 2023 22:04:50 +0000 (UTC) X-FDA: 80786612340.08.1F67DC0 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf07.hostedemail.com (Postfix) with ESMTP id 98E3840006 for ; Sat, 13 May 2023 22:04:48 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=c5BN0K0K; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf07.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=erxDDHkm9zdW4czPYzuEMILdUlPaEScqasEn1KnybQw=; b=1uZkGQ8qNjo88U/fktpXK3adLWk9d82mYv6Mfewo2E1/w5kT7nojckwI9JbLsvw3aQdtsR 0hQHRazU+OxxHAk2IlHY7NRt9Jpad4igvBLLo/o+76msRHoelvkTX7obLfRZ/YhzBQbVSH PqTFaf6UelcTvbJEUIJx2M4u537y+aE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=c5BN0K0K; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf07.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015488; a=rsa-sha256; cv=none; b=oFzRCvDtR1Gd+x4nQMo2+noVPr42z7L67zNZqFxhIpJGnVjxQRxgV0z00O0isrL4JF1bwR 0eepEz/ILSQx1m38PTd7XPvCPPV4d20Grh/rEo+L4Kn2ADmukUQoXM/3yDe0jAYP0FEpRG DYA6Pz9sbkKTMLFPqrXtf1zaG37sNr4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015488; x=1715551488; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Y0D+C8PlVRLLZHr4n5LbwmcxN7HFDk6t7kYP66WxeG4=; b=c5BN0K0Knsn9TVbhsz6LQ94hMhCp2ocjUf1e1uLNLyRrdaltoG3v3hpf UyazIUSXX7pYZGADumcCnPTFGii5L5rCmXXgKGr7XvaKvpEBT9eHZRyY5 UGqf5ux7ZOdb0OXyzxYgn5FDWEIM+BfpMaKHXFo8IH6QchF0yP0YkSJSE H6PRMIPXPGFNgW7VXzLRUCbVchYbmu65Q4f1MKNKoesmtr0w1aHX4H8kR Q7oAS6gwEAAa2fsaCWwFhcspaRmfXhro64SHvLMfkcZKORCYPhpBL/s1M ZKjyUp+zUnHyEWbo8SGQBphwvUf1Ds7L3cp4ltS1r8HC2wLGBZ7shF7Nz A==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="437325162" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="437325162" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="790199646" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="790199646" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:36 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id ED07F10CF79; Sun, 14 May 2023 01:04:33 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv11 3/9] efi/libstub: Implement support for unaccepted memory Date: Sun, 14 May 2023 01:04:12 +0300 Message-Id: <20230513220418.19357-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: xiozxnozi4874u8kyx9kwykgqah9ikoo X-Rspam-User: X-Rspamd-Queue-Id: 98E3840006 X-Rspamd-Server: rspam07 X-HE-Tag: 1684015488-802727 X-HE-Meta: U2FsdGVkX1+02hlswoJw0B+UPzJzyXXMd8l8PLRwJVkBnUrSJpH7HAbr85t/O30UHS4oEToiuyTYrQKo3vmUDjrq4POB3DKdwmZlz130t7oLrv2cd9MZDSknzoQC5t/AOB54eao6NycfuNC7k42cYQCDqRxRnsQJT3L+4bddE2jPfHOBActe3gpNeBDxBTs4pyT04ywgi/yVOLAtNetR0fMhs32cpjO6s/fHqphrav+aqWHlNFjFk+wqTiKJFXdvXtTjMO8UuiNLSCnYV7hAA9vFR2W71gIDah6ykFSP5We5R5GohiACQIvlIfrE5umHdYwMW4+eRxgrprLD1oJ4LWZFH68yWjVAkZP3jMScNN/ix6aSbym8oGxyfY6a1uyAV8tc4Up8TZ9zX1d2MESWkYQq02yyNxTMpRpeqZjMgjlEpa0630ViklH50i3vzxOKnkSQg/FBsxRjb+zI6Qj9gbddBnf3ebvroVoIhdMIYq0YfpVP0v0bS96svZCAPesUQ8gFrOKEyKvnNPICUFluqslyPk4XR/W4Tr2p0Z0CPVJJk6d1uP7BlFFKuGMOJRzPoykVT+uIDvCSg2SRs7J5B5Jp6Zqy5BKF1G8Hdlz4O2HsrCxyTEoLxfalnlrgnhVLXZDzHbsyZy4G5dQoaJ30sf+Jt2vDa0BwfluiCnJWt5rMXzrZT3JbcD46QIucnYJuprqTrm0p6aFdZgjV49VxXTEDzWD91mwK8oz3edLJzXL8FVQVwUOvppMJY9JPZFIh57FsMqjsw4eelL6f3TzVNyCdnXOmboR/Ee5WPqIieJYatkovpm+aMOJW6J0FReRxB27SklsGIScrE4W5NgqrjAw0QNqhgKe7ehzY+pNuqAVPDEPc6qcx/I9GYAOTfffn02bra935Lp64gzjHWfriVYwRfJZn42j+8jKFC/bMou0zNGpolreHVDIChM2u5qiMhQ3eVF+7fmk6wYO/7yZ 3Kdv3HYv 2LAa6d3ZfQo19z9acJpp6pCUb19T2AmDDqeM5gRJXO3S4jioVsVLgGFraxPIR20Ndgt+p7pX1s+yZ1LKX98cCNmNdHtffpxpinIcvyLIAXEx8tzRySGEVNsiTmPK34X8D1O2vABG/rZlyGLc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: UEFI Specification version 2.9 introduces the concept of memory acceptance: Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP, requiring memory to be accepted before it can be used by the guest. Accepting happens via a protocol specific for the Virtual Machine platform. Accepting memory is costly and it makes VMM allocate memory for the accepted guest physical address range. It's better to postpone memory acceptance until memory is needed. It lowers boot time and reduces memory overhead. The kernel needs to know what memory has been accepted. Firmware communicates this information via memory map: a new memory type -- EFI_UNACCEPTED_MEMORY -- indicates such memory. Range-based tracking works fine for firmware, but it gets bulky for the kernel: e820 (or whatever the arch uses) has to be modified on every page acceptance. It leads to table fragmentation and there's a limited number of entries in the e820 table. Another option is to mark such memory as usable in e820 and track if the range has been accepted in a bitmap. One bit in the bitmap represents a naturally aligned power-2-sized region of address space -- unit. For x86, unit size is 2MiB: 4k of the bitmap is enough to track 64GiB or physical address space. In the worst-case scenario -- a huge hole in the middle of the address space -- It needs 256MiB to handle 4PiB of the address space. Any unaccepted memory that is not aligned to unit_size gets accepted upfront. The bitmap is allocated and constructed in the EFI stub and passed down to the kernel via EFI configuration table. allocate_e820() allocates the bitmap if unaccepted memory is present, according to the size of unaccepted region. Signed-off-by: Kirill A. Shutemov Reviewed-by: Ard Biesheuvel --- arch/x86/boot/compressed/Makefile | 1 + arch/x86/boot/compressed/mem.c | 9 + arch/x86/include/asm/efi.h | 2 + drivers/firmware/efi/Kconfig | 14 ++ drivers/firmware/efi/efi.c | 1 + drivers/firmware/efi/libstub/Makefile | 2 + drivers/firmware/efi/libstub/bitmap.c | 41 ++++ drivers/firmware/efi/libstub/efistub.h | 6 + drivers/firmware/efi/libstub/find.c | 43 ++++ .../firmware/efi/libstub/unaccepted_memory.c | 222 ++++++++++++++++++ drivers/firmware/efi/libstub/x86-stub.c | 13 + include/linux/efi.h | 12 +- 12 files changed, 365 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/mem.c create mode 100644 drivers/firmware/efi/libstub/bitmap.c create mode 100644 drivers/firmware/efi/libstub/find.c create mode 100644 drivers/firmware/efi/libstub/unaccepted_memory.c diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 6b6cfe607bdb..cc4978123c30 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -107,6 +107,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/mem.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_mixed.o diff --git a/arch/x86/boot/compressed/mem.c b/arch/x86/boot/compressed/mem.c new file mode 100644 index 000000000000..67594fcb11d9 --- /dev/null +++ b/arch/x86/boot/compressed/mem.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include "error.h" + +void arch_accept_memory(phys_addr_t start, phys_addr_t end) +{ + /* Platform-specific memory-acceptance call goes here */ + error("Cannot accept memory"); +} diff --git a/arch/x86/include/asm/efi.h b/arch/x86/include/asm/efi.h index 419280d263d2..8b4be7cecdb8 100644 --- a/arch/x86/include/asm/efi.h +++ b/arch/x86/include/asm/efi.h @@ -31,6 +31,8 @@ extern unsigned long efi_mixed_mode_stack_pa; #define ARCH_EFI_IRQ_FLAGS_MASK X86_EFLAGS_IF +#define EFI_UNACCEPTED_UNIT_SIZE PMD_SIZE + /* * The EFI services are called through variadic functions in many cases. These * functions are implemented in assembler and support only a fixed number of diff --git a/drivers/firmware/efi/Kconfig b/drivers/firmware/efi/Kconfig index 043ca31c114e..231f1c70d1db 100644 --- a/drivers/firmware/efi/Kconfig +++ b/drivers/firmware/efi/Kconfig @@ -269,6 +269,20 @@ config EFI_COCO_SECRET virt/coco/efi_secret module to access the secrets, which in turn allows userspace programs to access the injected secrets. +config UNACCEPTED_MEMORY + bool + depends on EFI_STUB + help + Some Virtual Machine platforms, such as Intel TDX, require + some memory to be "accepted" by the guest before it can be used. + This mechanism helps prevent malicious hosts from making changes + to guest memory. + + UEFI specification v2.9 introduced EFI_UNACCEPTED_MEMORY memory type. + + This option adds support for unaccepted memory and makes such memory + usable by the kernel. + config EFI_EMBEDDED_FIRMWARE bool select CRYPTO_LIB_SHA256 diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c index abeff7dc0b58..7dce06e419c5 100644 --- a/drivers/firmware/efi/efi.c +++ b/drivers/firmware/efi/efi.c @@ -843,6 +843,7 @@ static __initdata char memory_type_name[][13] = { "MMIO Port", "PAL Code", "Persistent", + "Unaccepted", }; char * __init efi_md_typeattr_format(char *buf, size_t size, diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index 3abb2b357482..16d64a34d1e1 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -96,6 +96,8 @@ CFLAGS_arm32-stub.o := -DTEXT_OFFSET=$(TEXT_OFFSET) zboot-obj-$(CONFIG_RISCV) := lib-clz_ctz.o lib-ashldi3.o lib-$(CONFIG_EFI_ZBOOT) += zboot.o $(zboot-obj-y) +lib-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o bitmap.o find.o + extra-y := $(lib-y) lib-y := $(patsubst %.o,%.stub.o,$(lib-y)) diff --git a/drivers/firmware/efi/libstub/bitmap.c b/drivers/firmware/efi/libstub/bitmap.c new file mode 100644 index 000000000000..5c9bba0d549b --- /dev/null +++ b/drivers/firmware/efi/libstub/bitmap.c @@ -0,0 +1,41 @@ +#include + +void __bitmap_set(unsigned long *map, unsigned int start, int len) +{ + unsigned long *p = map + BIT_WORD(start); + const unsigned int size = start + len; + int bits_to_set = BITS_PER_LONG - (start % BITS_PER_LONG); + unsigned long mask_to_set = BITMAP_FIRST_WORD_MASK(start); + + while (len - bits_to_set >= 0) { + *p |= mask_to_set; + len -= bits_to_set; + bits_to_set = BITS_PER_LONG; + mask_to_set = ~0UL; + p++; + } + if (len) { + mask_to_set &= BITMAP_LAST_WORD_MASK(size); + *p |= mask_to_set; + } +} + +void __bitmap_clear(unsigned long *map, unsigned int start, int len) +{ + unsigned long *p = map + BIT_WORD(start); + const unsigned int size = start + len; + int bits_to_clear = BITS_PER_LONG - (start % BITS_PER_LONG); + unsigned long mask_to_clear = BITMAP_FIRST_WORD_MASK(start); + + while (len - bits_to_clear >= 0) { + *p &= ~mask_to_clear; + len -= bits_to_clear; + bits_to_clear = BITS_PER_LONG; + mask_to_clear = ~0UL; + p++; + } + if (len) { + mask_to_clear &= BITMAP_LAST_WORD_MASK(size); + *p &= ~mask_to_clear; + } +} diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index 67d5a20802e0..8659a01664b8 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -1133,4 +1133,10 @@ const u8 *__efi_get_smbios_string(const struct efi_smbios_record *record, void efi_remap_image(unsigned long image_base, unsigned alloc_size, unsigned long code_size); +efi_status_t allocate_unaccepted_bitmap(__u32 nr_desc, + struct efi_boot_memmap *map); +void process_unaccepted_memory(u64 start, u64 end); +void accept_memory(phys_addr_t start, phys_addr_t end); +void arch_accept_memory(phys_addr_t start, phys_addr_t end); + #endif diff --git a/drivers/firmware/efi/libstub/find.c b/drivers/firmware/efi/libstub/find.c new file mode 100644 index 000000000000..4e7740d28987 --- /dev/null +++ b/drivers/firmware/efi/libstub/find.c @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include + +/* + * Common helper for find_next_bit() function family + * @FETCH: The expression that fetches and pre-processes each word of bitmap(s) + * @MUNGE: The expression that post-processes a word containing found bit (may be empty) + * @size: The bitmap size in bits + * @start: The bitnumber to start searching at + */ +#define FIND_NEXT_BIT(FETCH, MUNGE, size, start) \ +({ \ + unsigned long mask, idx, tmp, sz = (size), __start = (start); \ + \ + if (unlikely(__start >= sz)) \ + goto out; \ + \ + mask = MUNGE(BITMAP_FIRST_WORD_MASK(__start)); \ + idx = __start / BITS_PER_LONG; \ + \ + for (tmp = (FETCH) & mask; !tmp; tmp = (FETCH)) { \ + if ((idx + 1) * BITS_PER_LONG >= sz) \ + goto out; \ + idx++; \ + } \ + \ + sz = min(idx * BITS_PER_LONG + __ffs(MUNGE(tmp)), sz); \ +out: \ + sz; \ +}) + +unsigned long _find_next_bit(const unsigned long *addr, unsigned long nbits, unsigned long start) +{ + return FIND_NEXT_BIT(addr[idx], /* nop */, nbits, start); +} + +unsigned long _find_next_zero_bit(const unsigned long *addr, unsigned long nbits, + unsigned long start) +{ + return FIND_NEXT_BIT(~addr[idx], /* nop */, nbits, start); +} diff --git a/drivers/firmware/efi/libstub/unaccepted_memory.c b/drivers/firmware/efi/libstub/unaccepted_memory.c new file mode 100644 index 000000000000..f4642c4f25dd --- /dev/null +++ b/drivers/firmware/efi/libstub/unaccepted_memory.c @@ -0,0 +1,222 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include "efistub.h" + +static struct efi_unaccepted_memory *unaccepted_table; + +efi_status_t allocate_unaccepted_bitmap(__u32 nr_desc, + struct efi_boot_memmap *map) +{ + efi_guid_t unaccepted_table_guid = LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID; + u64 unaccepted_start = ULLONG_MAX, unaccepted_end = 0, bitmap_size; + efi_status_t status; + int i; + + /* Check if the table is already installed */ + unaccepted_table = get_efi_config_table(unaccepted_table_guid); + if (unaccepted_table) { + if (unaccepted_table->version != 1) { + efi_err("Unknown version of unaccepted memory table\n"); + return EFI_UNSUPPORTED; + } + return EFI_SUCCESS; + } + + /* Check if there's any unaccepted memory and find the max address */ + for (i = 0; i < nr_desc; i++) { + efi_memory_desc_t *d; + unsigned long m = (unsigned long)map->map; + + d = efi_early_memdesc_ptr(m, map->desc_size, i); + if (d->type != EFI_UNACCEPTED_MEMORY) + continue; + + unaccepted_start = min(unaccepted_start, d->phys_addr); + unaccepted_end = max(unaccepted_end, + d->phys_addr + d->num_pages * PAGE_SIZE); + } + + if (unaccepted_start == ULLONG_MAX) + return EFI_SUCCESS; + + unaccepted_start = round_down(unaccepted_start, + EFI_UNACCEPTED_UNIT_SIZE); + unaccepted_end = round_up(unaccepted_end, EFI_UNACCEPTED_UNIT_SIZE); + + /* + * If unaccepted memory is present, allocate a bitmap to track what + * memory has to be accepted before access. + * + * One bit in the bitmap represents 2MiB in the address space: + * A 4k bitmap can track 64GiB of physical address space. + * + * In the worst case scenario -- a huge hole in the middle of the + * address space -- It needs 256MiB to handle 4PiB of the address + * space. + * + * The bitmap will be populated in setup_e820() according to the memory + * map after efi_exit_boot_services(). + */ + bitmap_size = DIV_ROUND_UP(unaccepted_end - unaccepted_start, + EFI_UNACCEPTED_UNIT_SIZE * BITS_PER_BYTE); + + status = efi_bs_call(allocate_pool, EFI_LOADER_DATA, + sizeof(*unaccepted_table) + bitmap_size, + (void **)&unaccepted_table); + if (status != EFI_SUCCESS) { + efi_err("Failed to allocate unaccepted memory config table\n"); + return status; + } + + unaccepted_table->version = 1; + unaccepted_table->unit_size = EFI_UNACCEPTED_UNIT_SIZE; + unaccepted_table->phys_base = unaccepted_start; + unaccepted_table->size = bitmap_size; + memset(unaccepted_table->bitmap, 0, bitmap_size); + + status = efi_bs_call(install_configuration_table, + &unaccepted_table_guid, unaccepted_table); + if (status != EFI_SUCCESS) { + efi_bs_call(free_pool, unaccepted_table); + efi_err("Failed to install unaccepted memory config table!\n"); + } + + return status; +} + +/* + * The accepted memory bitmap only works at unit_size granularity. Take + * unaligned start/end addresses and either: + * 1. Accepts the memory immediately and in its entirety + * 2. Accepts unaligned parts, and marks *some* aligned part unaccepted + * + * The function will never reach the bitmap_set() with zero bits to set. + */ +void process_unaccepted_memory(u64 start, u64 end) +{ + u64 unit_size = unaccepted_table->unit_size; + u64 unit_mask = unaccepted_table->unit_size - 1; + u64 bitmap_size = unaccepted_table->size; + + /* + * Ensure that at least one bit will be set in the bitmap by + * immediately accepting all regions under 2*unit_size. This is + * imprecise and may immediately accept some areas that could + * have been represented in the bitmap. But, results in simpler + * code below + * + * Consider case like this (assuming unit_size == 2MB): + * + * | 4k | 2044k | 2048k | + * ^ 0x0 ^ 2MB ^ 4MB + * + * Only the first 4k has been accepted. The 0MB->2MB region can not be + * represented in the bitmap. The 2MB->4MB region can be represented in + * the bitmap. But, the 0MB->4MB region is <2*unit_size and will be + * immediately accepted in its entirety. + */ + if (end - start < 2 * unit_size) { + arch_accept_memory(start, end); + return; + } + + /* + * No matter how the start and end are aligned, at least one unaccepted + * unit_size area will remain to be marked in the bitmap. + */ + + /* Immediately accept a phys_base) { + arch_accept_memory(start, + min(unaccepted_table->phys_base, end)); + start = unaccepted_table->phys_base; + } + + /* Nothing to record */ + if (end < unaccepted_table->phys_base) + return; + + /* Translate to offsets from the beginning of the bitmap */ + start -= unaccepted_table->phys_base; + end -= unaccepted_table->phys_base; + + /* Accept memory that doesn't fit into bitmap */ + if (end > bitmap_size * unit_size * BITS_PER_BYTE) { + unsigned long phys_start, phys_end; + + phys_start = bitmap_size * unit_size * BITS_PER_BYTE + + unaccepted_table->phys_base; + phys_end = end + unaccepted_table->phys_base; + + arch_accept_memory(phys_start, phys_end); + end = bitmap_size * unit_size * BITS_PER_BYTE; + } + + /* + * 'start' and 'end' are now both unit_size-aligned. + * Record the range as being unaccepted: + */ + bitmap_set(unaccepted_table->bitmap, + start / unit_size, (end - start) / unit_size); +} + +void accept_memory(phys_addr_t start, phys_addr_t end) +{ + unsigned long range_start, range_end; + unsigned long bitmap_size; + u64 unit_size; + + if (!unaccepted_table) + return; + + unit_size = unaccepted_table->unit_size; + + /* + * Only care for the part of the range that is represented + * in the bitmap. + */ + if (start < unaccepted_table->phys_base) + start = unaccepted_table->phys_base; + if (end < unaccepted_table->phys_base) + return; + + /* Translate to offsets from the beginning of the bitmap */ + start -= unaccepted_table->phys_base; + end -= unaccepted_table->phys_base; + + /* Make sure not to overrun the bitmap */ + if (end > unaccepted_table->size * unit_size * BITS_PER_BYTE) + end = unaccepted_table->size * unit_size * BITS_PER_BYTE; + + range_start = start / unit_size; + bitmap_size = DIV_ROUND_UP(end, unit_size); + + for_each_set_bitrange_from(range_start, range_end, + unaccepted_table->bitmap, bitmap_size) { + unsigned long phys_start, phys_end; + + phys_start = range_start * unit_size + unaccepted_table->phys_base; + phys_end = range_end * unit_size + unaccepted_table->phys_base; + + arch_accept_memory(phys_start, phys_end); + bitmap_clear(unaccepted_table->bitmap, + range_start, range_end - range_start); + } +} diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index fff81843169c..8d17cee8b98e 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -613,6 +613,16 @@ setup_e820(struct boot_params *params, struct setup_data *e820ext, u32 e820ext_s e820_type = E820_TYPE_PMEM; break; + case EFI_UNACCEPTED_MEMORY: + if (!IS_ENABLED(CONFIG_UNACCEPTED_MEMORY)) { + efi_warn_once( +"The system has unaccepted memory, but kernel does not support it\nConsider enabling CONFIG_UNACCEPTED_MEMORY\n"); + continue; + } + e820_type = E820_TYPE_RAM; + process_unaccepted_memory(d->phys_addr, + d->phys_addr + PAGE_SIZE * d->num_pages); + break; default: continue; } @@ -697,6 +707,9 @@ static efi_status_t allocate_e820(struct boot_params *params, status = alloc_e820ext(nr_e820ext, e820ext, e820ext_size); } + if (IS_ENABLED(CONFIG_UNACCEPTED_MEMORY) && status == EFI_SUCCESS) + status = allocate_unaccepted_bitmap(nr_desc, map); + efi_bs_call(free_pool, map); return status; } diff --git a/include/linux/efi.h b/include/linux/efi.h index 7aa62c92185f..29cc622910da 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -108,7 +108,8 @@ typedef struct { #define EFI_MEMORY_MAPPED_IO_PORT_SPACE 12 #define EFI_PAL_CODE 13 #define EFI_PERSISTENT_MEMORY 14 -#define EFI_MAX_MEMORY_TYPE 15 +#define EFI_UNACCEPTED_MEMORY 15 +#define EFI_MAX_MEMORY_TYPE 16 /* Attribute values: */ #define EFI_MEMORY_UC ((u64)0x0000000000000001ULL) /* uncached */ @@ -417,6 +418,7 @@ void efi_native_runtime_setup(void); #define LINUX_EFI_MOK_VARIABLE_TABLE_GUID EFI_GUID(0xc451ed2b, 0x9694, 0x45d3, 0xba, 0xba, 0xed, 0x9f, 0x89, 0x88, 0xa3, 0x89) #define LINUX_EFI_COCO_SECRET_AREA_GUID EFI_GUID(0xadf956ad, 0xe98c, 0x484c, 0xae, 0x11, 0xb5, 0x1c, 0x7d, 0x33, 0x64, 0x47) #define LINUX_EFI_BOOT_MEMMAP_GUID EFI_GUID(0x800f683f, 0xd08b, 0x423a, 0xa2, 0x93, 0x96, 0x5c, 0x3c, 0x6f, 0xe2, 0xb4) +#define LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID EFI_GUID(0xd5d1de3c, 0x105c, 0x44f9, 0x9e, 0xa9, 0xbc, 0xef, 0x98, 0x12, 0x00, 0x31) #define RISCV_EFI_BOOT_PROTOCOL_GUID EFI_GUID(0xccd15fec, 0x6f73, 0x4eec, 0x83, 0x95, 0x3e, 0x69, 0xe4, 0xb9, 0x40, 0xbf) @@ -534,6 +536,14 @@ struct efi_boot_memmap { efi_memory_desc_t map[]; }; +struct efi_unaccepted_memory { + u32 version; + u32 unit_size; + u64 phys_base; + u64 size; + unsigned long bitmap[]; +}; + /* * Architecture independent structure for describing a memory map for the * benefit of efi_memmap_init_early(), and for passing context between From patchwork Sat May 13 22:04:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34F8EC7EE2E for ; Sat, 13 May 2023 22:04:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 337BC900003; Sat, 13 May 2023 18:04:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E806900002; Sat, 13 May 2023 18:04:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16132900003; Sat, 13 May 2023 18:04:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 07B04900002 for ; Sat, 13 May 2023 18:04:52 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D46628112F for ; Sat, 13 May 2023 22:04:51 +0000 (UTC) X-FDA: 80786612382.04.83B8D66 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf24.hostedemail.com (Postfix) with ESMTP id AEB2F180008 for ; Sat, 13 May 2023 22:04:49 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=agijOIM3; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZC1H+SVlO53smocewfMX3MBnMZTnTOy5PrIz8NwD1ek=; b=8H3d1RU9wkGCuR5Rzr+JD3lTUuZR8eXzw+YVoYiMdrbfOsEKnvQnFocLlF2qwXBN5Pnkqz RGtv8Cz4tihbhIbH6YMV3CKHTtrykYchxAmH6Tqi+l3FLvZOqWCdtTgexwTGKdr3CC+BIu wT86gpTC3RQbsYfsR3fybIb6CH6GAhQ= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=agijOIM3; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015490; a=rsa-sha256; cv=none; b=GH/NiI8AimqYfdqXPjQrB2HHpSB6Uxvp0EqlEV1kXVxYzjIq8qDNjoltCIbgfp/l4ufogg 5s+CBWiOS6aVruLaoNgdZsrGUK0X+zt2XDO9HOWdRPjQB25iRD7Ui52vnYXvcm6LCB1R5N /WrE6iXDyCADPnVJhwTsXXMWJ4Dp1x4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015489; x=1715551489; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=74C6EjHk9vtTut2nxMktdKwMlSXxa+Ikleh1JSyXolE=; b=agijOIM3UwsRkCMkM4j3Gc3v0Pbg7sRfqGXQsT9+tOp5uZ3V1Zcxc5WG ei+KkbqeYLFzB980rop11JFk81fPyJ2fjkAFagl+VbChq/zBjW5Bq7HJZ X0TohWn/n+Z9BYv/91LF6/rg2OZkbQBM6mF+0jirwfiKg63zGxsaKzC4n JluokobdVmdcYYw7RK9pEFiP+VxXVrLrL5PSRuVfsZip1PePZdciLWl5p hNPLu1kqwEtgcdLcH34HiskdlX7deOYpLHXLFNmBRZZyGcADOpCJLTQZ/ KlSXdVpNrbAVn27wtUQmItSK6zAa4NTJiw1RMqUTCp0nVKFpvPoBpPXG9 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="379142507" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="379142507" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="733429545" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="733429545" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:36 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 051AE10CF7A; Sun, 14 May 2023 01:04:34 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv11 4/9] x86/boot/compressed: Handle unaccepted memory Date: Sun, 14 May 2023 01:04:13 +0300 Message-Id: <20230513220418.19357-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: 3wehjxuses316zy41fzjunnmab4zfut1 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AEB2F180008 X-Rspam-User: X-HE-Tag: 1684015489-208326 X-HE-Meta: U2FsdGVkX188BMIb+quN2/8wAOvvmwZSnnK713kxO+G1nxZ87ugAbXkXpm+I+7iXL0hl4mAUXa1ewzUklA2Z0Z7wcfCtHYitm1eVfJE7BSNNSjAUF3N0pMRBw3DLbx7puDNd3MShIPWd5xLzDdtZ2kuT6xa6BdpWFjIHuFEaT1hQj1/70bZtUHcteO9r2X+oE1mv4FDWw+lIAEPR/ca/0Lx80Zy4NNCRfrBD9OgPjBVWo9aRXXEjA6KcI/Nwj+mkQyoNKXP/EeNF7/qIp/ocXVgeVFfVyTnuU5bfhnFg+cQ+RYfVMMfw+cWRGmpm2UkcfYBPyfDtULJKoS2gpxi2cOtIfw5Ic7k4rYXC5oRFQWyIa8r2l+ZDQiQphZlsB4nQEnOrFK51f1W6QLEsWvsYzTdlo+uA/MfFoR1e4shxMhgYQndf0RT5trLSTuXoQK8qrr89woGvnrztNqL5YaaJyGVAlwwFnmA4BFwiU/mHsBR7b7j81XNVyE4fLsRQwfXi187i2dymw1XcWou7pQH4M0WkSu+3wC4UTgCIL9Vdf8uQC9GvZfgalX52w8QjbT2Enkd1s9vXaWpb2lEgrIa4N0CLX4t7ljMUNuvLinw8YxF+rd/EIK3o/58fAW4g18eK+3D0PziVwjHXbKXNWDKhYnxEowehTC/46aHPxnrmHj/YR0PLuUGV2j/wX+S5sE+s8t6twRMykOE8M3MHdh9IUdFzfuwX/PLmhKVWYXrsYQazmKGtqLaS0c5vhE2rh9ffZb4YK2F3d3QTp11VlM8NgEYlskrnaWcLAh4oLs4OEBLogoNVHjVhTa2NietP6NDaEo/0S8Vw3m08dB5qnE9qr+SzBz+mpomGrN1CUst1jfK7WJZuAKgYOdiir3hu9Rei6Lr4SqFaUUZDD7By3NBrbnEjXOLQ+coHIe/a+5bNRucQt36ot6UKPPbiJM1rWTS/ZmSgjCM6x+8sPsrlQcN qodPn5hs h2D89M4QYuknx8/gN73hlCi8hVfkYxYSdBYI1NWOLSCzccXhoZvpObcAM1lGFqRt/Uno+v9+9SP8kDVLd+En9KBtd3ht/c+8eEmniOGntyL93m+CTOWv9fqSPR/OJrpKHKnMW+rnDFAT40/0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The firmware will pre-accept the memory used to run the stub. But, the stub is responsible for accepting the memory into which it decompresses the main kernel. Accept memory just before decompression starts. The stub is also responsible for choosing a physical address in which to place the decompressed kernel image. The KASLR mechanism will randomize this physical address. Since the unaccepted memory region is relatively small, KASLR would be quite ineffective if it only used the pre-accepted area (EFI_CONVENTIONAL_MEMORY). Ensure that KASLR randomizes among the entire physical address space by also including EFI_UNACCEPTED_MEMORY. Signed-off-by: Kirill A. Shutemov Reviewed-by: Liam Merwick Reviewed-by: Tom Lendacky --- arch/x86/boot/compressed/efi.h | 1 + arch/x86/boot/compressed/kaslr.c | 35 +++++++++++++++++++++----------- arch/x86/boot/compressed/misc.c | 6 ++++++ arch/x86/boot/compressed/misc.h | 6 ++++++ 4 files changed, 36 insertions(+), 12 deletions(-) diff --git a/arch/x86/boot/compressed/efi.h b/arch/x86/boot/compressed/efi.h index 7db2f41b54cd..cf475243b6d5 100644 --- a/arch/x86/boot/compressed/efi.h +++ b/arch/x86/boot/compressed/efi.h @@ -32,6 +32,7 @@ typedef struct { } efi_table_hdr_t; #define EFI_CONVENTIONAL_MEMORY 7 +#define EFI_UNACCEPTED_MEMORY 15 #define EFI_MEMORY_MORE_RELIABLE \ ((u64)0x0000000000010000ULL) /* higher reliability */ diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c index 454757fbdfe5..749f0fe7e446 100644 --- a/arch/x86/boot/compressed/kaslr.c +++ b/arch/x86/boot/compressed/kaslr.c @@ -672,6 +672,28 @@ static bool process_mem_region(struct mem_vector *region, } #ifdef CONFIG_EFI + +/* + * Only EFI_CONVENTIONAL_MEMORY and EFI_UNACCEPTED_MEMORY (if supported) are + * guaranteed to be free. + * + * It is more conservative in picking free memory than the EFI spec allows: + * + * According to the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also free memory + * and thus available to place the kernel image into, but in practice there's + * firmware where using that memory leads to crashes. + */ +static inline bool memory_type_is_free(efi_memory_desc_t *md) +{ + if (md->type == EFI_CONVENTIONAL_MEMORY) + return true; + + if (md->type == EFI_UNACCEPTED_MEMORY) + return IS_ENABLED(CONFIG_UNACCEPTED_MEMORY); + + return false; +} + /* * Returns true if we processed the EFI memmap, which we prefer over the E820 * table if it is available. @@ -716,18 +738,7 @@ process_efi_entries(unsigned long minimum, unsigned long image_size) for (i = 0; i < nr_desc; i++) { md = efi_early_memdesc_ptr(pmap, e->efi_memdesc_size, i); - /* - * Here we are more conservative in picking free memory than - * the EFI spec allows: - * - * According to the spec, EFI_BOOT_SERVICES_{CODE|DATA} are also - * free memory and thus available to place the kernel image into, - * but in practice there's firmware where using that memory leads - * to crashes. - * - * Only EFI_CONVENTIONAL_MEMORY is guaranteed to be free. - */ - if (md->type != EFI_CONVENTIONAL_MEMORY) + if (!memory_type_is_free(md)) continue; if (efi_soft_reserve_enabled() && diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index 014ff222bf4b..eb8df0d4ad51 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -455,6 +455,12 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap, #endif debug_putstr("\nDecompressing Linux... "); + + if (IS_ENABLED(CONFIG_UNACCEPTED_MEMORY)) { + debug_putstr("Accepting memory... "); + accept_memory(__pa(output), __pa(output) + needed_size); + } + __decompress(input_data, input_len, NULL, NULL, output, output_len, NULL, error); entry_offset = parse_elf(output); diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index 2f155a0e3041..9663d1839f54 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -247,4 +247,10 @@ static inline unsigned long efi_find_vendor_table(struct boot_params *bp, } #endif /* CONFIG_EFI */ +#ifdef CONFIG_UNACCEPTED_MEMORY +void accept_memory(phys_addr_t start, phys_addr_t end); +#else +static inline void accept_memory(phys_addr_t start, phys_addr_t end) {} +#endif + #endif /* BOOT_COMPRESSED_MISC_H */ From patchwork Sat May 13 22:04:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8BE1C77B7F for ; Sat, 13 May 2023 22:04:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9051F280002; Sat, 13 May 2023 18:04:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B5FB280003; Sat, 13 May 2023 18:04:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 691E1280002; Sat, 13 May 2023 18:04:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5A36C280001 for ; Sat, 13 May 2023 18:04:56 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2F41B411B7 for ; Sat, 13 May 2023 22:04:56 +0000 (UTC) X-FDA: 80786612592.08.B578DB3 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf24.hostedemail.com (Postfix) with ESMTP id CE0E718000C for ; Sat, 13 May 2023 22:04:53 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GfnJVBW8; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BMOOHRUB7Zr7XQ5THzAtjIzZdoIftwVgnqo7Ze0X5Gc=; b=4qGhM6fvUdF3XpwUlKTNXcHravfxFATpivXAextVAnKvICJbQiDluS/PNhEQW9qxSsyvwk 9zpIir04/WCJTMjGSXJOUXj4DY7SyVQGWRgLcAIVeXLOCT4z6sMsv4kg1lLsctewF+57+6 X+h2HHnLaSZBbb7WiXKDNCQKMW0l7Mo= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GfnJVBW8; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015494; a=rsa-sha256; cv=none; b=BvQzdM+rDZBbqBe4xAzlpEfpb4xogF3xhzw2mktFrv/y30Dhe+pKHMOkT5iHc1+sBpEOSN 8TECaSX+g4IwwZv3Jmu2mMTLQ95cMbUqcKx6eZFOGIh4jncEJ55Uqm+UXkEMn4NnHslb2d qHFXIiMIv3hwgLBmNKuSsQuSNib7DvA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015493; x=1715551493; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vzNujRVXgfcy7qjArGlQaKcw6ocTEeuAnyV8Bln9kG4=; b=GfnJVBW8kpVpU87yrB3j9QostD4A0DFel3kpBJGAuS+ETFtRVfi4NN4a Br58RYMBJb2ibgTtHEvol1XNwa2czxuYZ3xET6xtLpRYym3aKwzr1xgy4 4E6hG2hyazCVCbIL9KMetsSdVv1XXvENBvD5Kh8YWe+FYqGIPpvYIWq/E bSIMnYDRaHuG3OI7VDs64sYw50o+zyESR5tdwsn7YQ5oGaLkbcs01HsSG nZZJY5jEBSAhdD6BmpQXMKISQ5SJe4m4wAqbFZ6Qj2xcIYoe0iSO0EZJm OvwV7YMABSvFD86F2hIUC6m2wT/Ctc4PhvcCE0xXsw35pBDyQuwj7EtN5 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="379142559" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="379142559" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="733429554" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="733429554" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:45 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 1087C10CF7B; Sun, 14 May 2023 01:04:34 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv11 5/9] efi: Provide helpers for unaccepted memory Date: Sun, 14 May 2023 01:04:14 +0300 Message-Id: <20230513220418.19357-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: 3ia9j78jfqkyhcwuxpnc954fqarxmd5d X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CE0E718000C X-Rspam-User: X-HE-Tag: 1684015493-578929 X-HE-Meta: U2FsdGVkX1+hTZ/oqX6knZWnNzyvlXs9+wUqk6SY0Uy+txRi4rzm8S7SRcLAFFPsurDkU/ZviEZYlXAU8ThKRKwuyAiubOUYyYos2eRN0CItac0aE7cI+JwEF23xLW9W+6OF5/uYbPIWuxSg90y8dAH2rH9bDsFCdbC9eHmU1/tYDC7AnOvRizCQhaerB6oTBjPKZXGZKVWCz8z3zMexgSM8dazvipGiDA0bK89qGhY8M2IQg9n6Flog28SJ3HzD1AqBNIoD/lc3jMuRGUPSwhy1+brGYjLurDSejl82pt07iLWs8tyZLzH3V98hhX9TUs2ZC3sYtAM9ApVFJuAAXySorMfMc1yQcOqLDfKDvjuWaj9hIyDxhb8khhfUjEo81BRzv99x/pIhpvZJ6aGWJFrs5yI+eLUamBivQP/IhhZeMAdAwyZefvlvO62tMyVyxWLw1kRv8WPyj8eqNU0L1PM7E9AdC0MGGz+4EDwgKr++8ovWymV4MgvvAB4Gr6B2n7SinEOey6aKo/cSDdePPkAvADqKr6qeQ0x7OYJnOwoX/x3o4Mk1nXK0Dsm5qijcgkTH++CNaDylfzDhmWgGY2xx96iFBUwCwxUQoUUtN2SVX64cdVMMCpf+LdpVq6dw7t7LCQkMtDcwedh0bUDg/v9byKwx663zZEnCfIBGFDa5N+J5FMrKnmqe0xOsOiBeGW0thNnKppvjWSetnLPklzpPjwxpmhN68Asyk1Y/geq/GOYcO6G7pZVTLBVKYACTNcBS7xqo0VCYotzFuPyU15nSFedKz8D0VKrLG7MBOtpm4pTD2r8GF1Bn4M63BFsGQr2UoCg1b9yiJ3pXtGAWN/c9KcfMc8ZK8cOMJwXt+FBrhLWqvfSpATueMi5zNjkbgHxfhANbiWFMYAvS7tiA3gxdJHgHWJosldXfI4Yh4csaYRTAsb+ynhWjvin0K9j+iCX7HfzCMVgS+gCwTZj Xmgz2KU/ al1sAuz/iE5l5pzeBdveMRPSlcf41b859WBTz/Xugvks6PcOgSkqLbSjMz3nyC+aP7dzKSNEiVBrM/1B1pR2D4cciCjBHjAWeRxlLxj0gb/Ulm2A9pvzm/1Boq6B2s+m1P7i42vZ7quAni/LNnES5D6Mv5hXeprdY5fP/UxFIAE/R6znvJyEK3PBSEA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Core-mm requires few helpers to support unaccepted memory: - accept_memory() checks the range of addresses against the bitmap and accept memory if needed. - range_contains_unaccepted_memory() checks if anything within the range requires acceptance. Architectural code has to provide efi_get_unaccepted_table() that returns pointer to the unaccepted memory configuration table. arch_accept_memory() handles arch-specific part of memory acceptance. Signed-off-by: Kirill A. Shutemov --- drivers/firmware/efi/Makefile | 1 + drivers/firmware/efi/efi.c | 6 ++ drivers/firmware/efi/unaccepted_memory.c | 103 +++++++++++++++++++++++ include/linux/efi.h | 1 + 4 files changed, 111 insertions(+) create mode 100644 drivers/firmware/efi/unaccepted_memory.c diff --git a/drivers/firmware/efi/Makefile b/drivers/firmware/efi/Makefile index b51f2a4c821e..e489fefd23da 100644 --- a/drivers/firmware/efi/Makefile +++ b/drivers/firmware/efi/Makefile @@ -41,3 +41,4 @@ obj-$(CONFIG_EFI_CAPSULE_LOADER) += capsule-loader.o obj-$(CONFIG_EFI_EARLYCON) += earlycon.o obj-$(CONFIG_UEFI_CPER_ARM) += cper-arm.o obj-$(CONFIG_UEFI_CPER_X86) += cper-x86.o +obj-$(CONFIG_UNACCEPTED_MEMORY) += unaccepted_memory.o diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c index 7dce06e419c5..e15a2005ed93 100644 --- a/drivers/firmware/efi/efi.c +++ b/drivers/firmware/efi/efi.c @@ -50,6 +50,9 @@ struct efi __read_mostly efi = { #ifdef CONFIG_EFI_COCO_SECRET .coco_secret = EFI_INVALID_TABLE_ADDR, #endif +#ifdef CONFIG_UNACCEPTED_MEMORY + .unaccepted = EFI_INVALID_TABLE_ADDR, +#endif }; EXPORT_SYMBOL(efi); @@ -605,6 +608,9 @@ static const efi_config_table_type_t common_tables[] __initconst = { #ifdef CONFIG_EFI_COCO_SECRET {LINUX_EFI_COCO_SECRET_AREA_GUID, &efi.coco_secret, "CocoSecret" }, #endif +#ifdef CONFIG_UNACCEPTED_MEMORY + {LINUX_EFI_UNACCEPTED_MEM_TABLE_GUID, &efi.unaccepted, "Unaccepted" }, +#endif #ifdef CONFIG_EFI_GENERIC_STUB {LINUX_EFI_SCREEN_INFO_TABLE_GUID, &screen_info_table }, #endif diff --git a/drivers/firmware/efi/unaccepted_memory.c b/drivers/firmware/efi/unaccepted_memory.c new file mode 100644 index 000000000000..bb91c41f76fb --- /dev/null +++ b/drivers/firmware/efi/unaccepted_memory.c @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include + +/* Protects unaccepted memory bitmap */ +static DEFINE_SPINLOCK(unaccepted_memory_lock); + +void accept_memory(phys_addr_t start, phys_addr_t end) +{ + struct efi_unaccepted_memory *unaccepted; + unsigned long range_start, range_end; + unsigned long flags; + u64 unit_size; + + if (efi.unaccepted == EFI_INVALID_TABLE_ADDR) + return; + + unaccepted = efi_get_unaccepted_table(); + if (!unaccepted) + return; + + unit_size = unaccepted->unit_size; + + /* + * Only care for the part of the range that is represented + * in the bitmap. + */ + if (start < unaccepted->phys_base) + start = unaccepted->phys_base; + if (end < unaccepted->phys_base) + return; + + /* Translate to offsets from the beginning of the bitmap */ + start -= unaccepted->phys_base; + end -= unaccepted->phys_base; + + /* Make sure not to overrun the bitmap */ + if (end > unaccepted->size * unit_size * BITS_PER_BYTE) + end = unaccepted->size * unit_size * BITS_PER_BYTE; + + range_start = start / unit_size; + + spin_lock_irqsave(&unaccepted_memory_lock, flags); + for_each_set_bitrange_from(range_start, range_end, unaccepted->bitmap, + DIV_ROUND_UP(end, unit_size)) { + unsigned long phys_start, phys_end; + unsigned long len = range_end - range_start; + + phys_start = range_start * unit_size + unaccepted->phys_base; + phys_end = range_end * unit_size + unaccepted->phys_base; + + arch_accept_memory(phys_start, phys_end); + bitmap_clear(unaccepted->bitmap, range_start, len); + } + spin_unlock_irqrestore(&unaccepted_memory_lock, flags); +} + +bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) +{ + struct efi_unaccepted_memory *unaccepted; + unsigned long flags; + bool ret = false; + u64 unit_size; + + unaccepted = efi_get_unaccepted_table(); + if (!unaccepted) + return false; + + unit_size = unaccepted->unit_size; + + /* + * Only care for the part of the range that is represented + * in the bitmap. + */ + if (start < unaccepted->phys_base) + start = unaccepted->phys_base; + if (end < unaccepted->phys_base) + return false; + + /* Translate to offsets from the beginning of the bitmap */ + start -= unaccepted->phys_base; + end -= unaccepted->phys_base; + + /* Make sure not to overrun the bitmap */ + if (end > unaccepted->size * unit_size * BITS_PER_BYTE) + end = unaccepted->size * unit_size * BITS_PER_BYTE; + + spin_lock_irqsave(&unaccepted_memory_lock, flags); + while (start < end) { + if (test_bit(start / unit_size, unaccepted->bitmap)) { + ret = true; + break; + } + + start += unit_size; + } + spin_unlock_irqrestore(&unaccepted_memory_lock, flags); + + return ret; +} diff --git a/include/linux/efi.h b/include/linux/efi.h index 29cc622910da..9864f9c00da2 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -646,6 +646,7 @@ extern struct efi { unsigned long tpm_final_log; /* TPM2 Final Events Log table */ unsigned long mokvar_table; /* MOK variable config table */ unsigned long coco_secret; /* Confidential computing secret table */ + unsigned long unaccepted; /* Unaccepted memory table */ efi_get_time_t *get_time; efi_set_time_t *set_time; From patchwork Sat May 13 22:04:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 267B7C7EE25 for ; Sat, 13 May 2023 22:05:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D4BBD280003; Sat, 13 May 2023 18:04:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CFC68280001; Sat, 13 May 2023 18:04:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB60F280004; Sat, 13 May 2023 18:04:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 819B4280001 for ; Sat, 13 May 2023 18:04:56 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3C22181137 for ; Sat, 13 May 2023 22:04:56 +0000 (UTC) X-FDA: 80786612592.09.E1B86A8 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf11.hostedemail.com (Postfix) with ESMTP id 2BDF24000D for ; Sat, 13 May 2023 22:04:53 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZGOarvC3; spf=none (imf11.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OQE2MEAlghjkxnRXmdNB2Xf64La8lXGjBEDZAIlZfb0=; b=QcOfpOZNBUVwd91hw7IsxL1tiL8JCpi5yQGuAmSNabkHz/PQd8I8OLeJSTWfe2urLKdIZS XeBvIKfAwJHis8NHKcX023sBzjLD5kpwVauDx/CcoFmKTpGVOI1bqla7eK69398c6gr5I4 q75yhd0qooPn9Pk1O9wtgC9Iu5wUqP4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015494; a=rsa-sha256; cv=none; b=ut6GWl1Ox7vfxNvi4s7gUCRJ6ugSFCcSfOoPOLXWPVWAVvOr3d/8zLNTmkXH4N0dviU9zI mdpbwOjXqpV7/EFhY9YyALNskkP7Lhb8Hyhu/d0J137KpqMVstyE52mMpo4lv+tdEtggR0 NW9TYpkO3jfxk/FtPvoO8qSD5h0nwGU= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZGOarvC3; spf=none (imf11.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015494; x=1715551494; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gRzQhhRAaTLGAt93TT2mPFzt+Xa/PSDkJomh6JXhxmM=; b=ZGOarvC33wEZOj2HEnomZJPih6ifrJ72XImoE38t0kU2wRlsKn6ljL5t YF/p4hrFvG/AdO4Fbewl++UNW7Jreu0+dxrBt/8WnKaRCtyWaJ07WCYCr 5UTiZVQfeDsQgWe95HVo3ZtKbdgv3HrpElsFsFchZHXgjrr3ynXgUzRR3 awWnJxjy+DW/g9kpaBl98g7DF8X1pvumATE8W+oV54B86ZIwf542pPLx9 C7esRCDm1otL0Uk6L6/RdqF4TeiHfO0lvnbhQsSrZX90T2hm0nY7ZSLXO HNZ+mfxSP1s456VFF6C7GgFQj7VRDI2WvMTomMCVt6bYMJXJRNj/9sB3O g==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="437325200" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="437325200" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="790199674" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="790199674" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:45 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 1A75810CF7C; Sun, 14 May 2023 01:04:34 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Dave Hansen Subject: [PATCHv11 6/9] efi/unaccepted: Avoid load_unaligned_zeropad() stepping into unaccepted memory Date: Sun, 14 May 2023 01:04:15 +0300 Message-Id: <20230513220418.19357-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: iq4o7yae9bwrkqjxiuc18h165xcdxh6x X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 2BDF24000D X-HE-Tag: 1684015493-74982 X-HE-Meta: U2FsdGVkX18fC2wbqZyNnkqBrgxEP2+vBrir/9kuFKETSc5Jbpe3krHlER/ZkyWPke13lwGXrdQRsHp9GHj8jTPIkHPzWHzTdmE1Y0wn4CxHzoHZScms1F06JdeADhc19Lmv4TgIuo3l1H0B6+4IytuNURrE6Kcu4ZelJtV1L+wEbi24e6y8gQSBqGAUkFFXgx9P6RYE3YkeRJzr2gFD7dDnZyJLsVv1D04YcHgmEapB4zro02UMD6/0riSpELh1VAUxboYMhIYGnVAJIPnZ6Dn1ox+He8GaywZnPx8I/41vovZ841NhoU1LWphfXye1rx4PbXsV1q3fCq914IGNlg9jX4FXJMYcPk+DLxpC5aBkNwbra7YjaiYDaaHh98vYgzRTp7MvdQhY1m7GM3l65vhSLzuW7KSAanPQn4ZaLOwt5RdWp/r8zI/FF+qPL1c6SVxzcFIKxsAtBVEku5/39obqN6VtVxA/wNVcd14pMTXHK2kdERNauKtBW2aFD8Ef5MmK8VUiOmSG4gqb3i8NGUSm5d3FqzuALP4xHWtPrgXBRLo6et00KRZMsLR2x13WIODf3kZ937oueAVB2o2cH8DMpxfzNZZd+n0ZKCkrcD6SN+B5n/uGVjzU6uOyMblr6NzHjr+IMQ+aBUxflWRwr8foeKC3H5zUmZWO+ZwurbCdmx9M/NT0OeCmpr6zdmqW0IEPeetqOGJLR0YavlJXJW/YbG1Ve+MLaJiP5lGY9SpS14BPlNyKQ5dqVgEZ7jaEkUfhzsDOwIYvxhVc0Lz8a/+EcbSwEEXt2yFlVFvBrLyTq2x9UUakaa/U6y9btlZn9YXejSH93d6A+jgbF0iQCQJipQA5zMBYwjPsnoAs4KXDu6Bln921byvfOOWgoV4+dd28ec3LzlV8VYLklnB+Z0aiPpCD+xvLbV7BJogU5NuANWFw7AqpD/xrXvTwy7DpjfrJroko+MZNTYDr+5Q GbWIasgl rNqiixdQp69qEZcg+UCeVUKSIQNexR0qeSzXkrHl4/uW0dfpVgquDnDyZzXV0r3wgShJ4qTPYUDMMxVDcnuD+WLDGVG2taS7H6ciST3+X/VTrbSUbWcjLJQ/QYbNSQJJxExnKQdLNET8R1lrJ6x9nde69aQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: load_unaligned_zeropad() can lead to unwanted loads across page boundaries. The unwanted loads are typically harmless. But, they might be made to totally unrelated or even unmapped memory. load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now #VE) to recover from these unwanted loads. But, this approach does not work for unaccepted memory. For TDX, a load from unaccepted memory will not lead to a recoverable exception within the guest. The guest will exit to the VMM where the only recourse is to terminate the guest. There are two parts to fix this issue and comprehensively avoid access to unaccepted memory. Together these ensure that an extra "guard" page is accepted in addition to the memory that needs to be used. 1. Implicitly extend the range_contains_unaccepted_memory(start, end) checks up to end+unit_size if 'end' is aligned on a unit_size boundary. 2. Implicitly extend accept_memory(start, end) to end+unit_size if 'end' is aligned on a unit_size boundary. Side note: This leads to something strange. Pages which were accepted at boot, marked by the firmware as accepted and will never _need_ to be accepted might be on unaccepted_pages list This is a cue to ensure that the next page is accepted before 'page' can be used. This is an actual, real-world problem which was discovered during TDX testing. Signed-off-by: Kirill A. Shutemov Reviewed-by: Dave Hansen Reviewed-by: Tom Lendacky --- drivers/firmware/efi/unaccepted_memory.c | 35 ++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/drivers/firmware/efi/unaccepted_memory.c b/drivers/firmware/efi/unaccepted_memory.c index bb91c41f76fb..3d1ca60916dd 100644 --- a/drivers/firmware/efi/unaccepted_memory.c +++ b/drivers/firmware/efi/unaccepted_memory.c @@ -37,6 +37,34 @@ void accept_memory(phys_addr_t start, phys_addr_t end) start -= unaccepted->phys_base; end -= unaccepted->phys_base; + /* + * load_unaligned_zeropad() can lead to unwanted loads across page + * boundaries. The unwanted loads are typically harmless. But, they + * might be made to totally unrelated or even unmapped memory. + * load_unaligned_zeropad() relies on exception fixup (#PF, #GP and now + * #VE) to recover from these unwanted loads. + * + * But, this approach does not work for unaccepted memory. For TDX, a + * load from unaccepted memory will not lead to a recoverable exception + * within the guest. The guest will exit to the VMM where the only + * recourse is to terminate the guest. + * + * There are two parts to fix this issue and comprehensively avoid + * access to unaccepted memory. Together these ensure that an extra + * "guard" page is accepted in addition to the memory that needs to be + * used: + * + * 1. Implicitly extend the range_contains_unaccepted_memory(start, end) + * checks up to end+unit_size if 'end' is aligned on a unit_size + * boundary. + * + * 2. Implicitly extend accept_memory(start, end) to end+unit_size if + * 'end' is aligned on a unit_size boundary. (immediately following + * this comment) + */ + if (!(end % unit_size)) + end += unit_size; + /* Make sure not to overrun the bitmap */ if (end > unaccepted->size * unit_size * BITS_PER_BYTE) end = unaccepted->size * unit_size * BITS_PER_BYTE; @@ -84,6 +112,13 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) start -= unaccepted->phys_base; end -= unaccepted->phys_base; + /* + * Also consider the unaccepted state of the *next* page. See fix #1 in + * the comment on load_unaligned_zeropad() in accept_memory(). + */ + if (!(end % unit_size)) + end += unit_size; + /* Make sure not to overrun the bitmap */ if (end > unaccepted->size * unit_size * BITS_PER_BYTE) end = unaccepted->size * unit_size * BITS_PER_BYTE; From patchwork Sat May 13 22:04:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9F05C7EE23 for ; Sat, 13 May 2023 22:05:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 354C4280001; Sat, 13 May 2023 18:04:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 078A8280006; Sat, 13 May 2023 18:04:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBD37280001; Sat, 13 May 2023 18:04:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C4BBE280006 for ; Sat, 13 May 2023 18:04:58 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 92680C11D4 for ; Sat, 13 May 2023 22:04:58 +0000 (UTC) X-FDA: 80786612676.28.0DA86A1 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf11.hostedemail.com (Postfix) with ESMTP id 5CCC84000D for ; Sat, 13 May 2023 22:04:56 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GexK88j2; spf=none (imf11.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015496; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cAZ5dclHmr3bPqqN6wlS/ms38HkegOt+k+1Qe8V3J/Y=; b=UWRkHmtCJdXYHuyGjt88wh7/nJ7crGiVlykfsmHC5+tyB6oqCqbA2NDDisMOi9g1s+IVQJ ill8onVXKIaBhbpEVbxgsxU9brhgY4aCtmWxjrdN8j7h967XlUVfsElKhoZQRorbhVXqEo VWi0BN600/j+5Mw3tYwaA97Vcv2slKI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015496; a=rsa-sha256; cv=none; b=TWCfgKwHiksV9ISWHEU4x8j1e35mJYdGVrCSFmdVvONPRlGInaof6OKW2pJ1SS+kdgHcgV SwWuZYK2GpmjTFIWbzXG5TvfkGlbcwyKehadhOWrtWdJbml3xKMHBvIrK9zNouHgX7xNYO h5ckm5bNRJNaTZsXc3uRSWerKIkQzkM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GexK88j2; spf=none (imf11.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.43) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015496; x=1715551496; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Eu57mUO9ct2ICW4Etq/GrS2vGWKQMEDS8IVWtzUCpBo=; b=GexK88j21PTT+Mr46pDDgUzc7pWFWifs7CYUnzUKinwSs87t2P5F6jI4 PvQkdDoZFR687zwdAA3ZvTajmuHfQUPxoDsrzMo8boh6bd0DE/GavM6/3 98phtRX3oeH1RZBgAtNrNC4WyfhTPIQFv/4Ny/uYCHWIn1zG4xLLaAtrP LgQTfSLxETvCaQOBsOIZ2HJhhswAp9ubTKHqEU+HUp+40GcaZeWQhcyKH pOaqSnHnN5EoKXQNVoPdyIW+faUZw9/ZjXLwNv3L0BjXNeaHfspVWAG8w Cr3oBc2MbX9M07R52KHHhVRWqf6kduz4n0YVw56rUg2Lynu/eNobxHfsz g==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="437325215" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="437325215" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="790199676" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="790199676" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:45 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 256A710CF7D; Sun, 14 May 2023 01:04:34 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Dave Hansen Subject: [PATCHv11 7/9] x86/tdx: Make _tdx_hypercall() and __tdx_module_call() available in boot stub Date: Sun, 14 May 2023 01:04:16 +0300 Message-Id: <20230513220418.19357-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: 9ac5r4sp3zm5jjmm3x7n384aks9ks5ge X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 5CCC84000D X-HE-Tag: 1684015496-459400 X-HE-Meta: U2FsdGVkX18fNCvFEcGAvUbowDYUOIoHOTYTdhYeaBZDGX4w+Rc2il4JodfTFunE/EQjJvew1jW3kZwg49tdOVXmeEIqcGN9UddaxF/xTiUFsFViSIhy4AWVtJG66CRQJlqiQm83XkAbFPRLgEa7SUDHVwvqUmOpiCXxBd0pcL+vqiQ5ULM86Znl63Tez/SwqXOIFFkkHgAdQttNBEHrK32ePGPBnGA4lbRwS2MyuRo7Ovf6uJ+D+VhtEOw3e1+PPkF14N+bz7iKU47hA/igXneCnpvHtbA5Uf7jSe9S6UEkRNM04Nvem/ggDGYQqKeOXNQ9+rJxAt2mSZUY4GoBHROCNIgqByOi6yrFbam/dj+dW9M5VMQIZCKrCTbR8SOguPoSPwdvCag3kMlRSStPz/PHkdWWvkMinXUCHOM1pNrYhGRjhXpzUKv6i/cbCyVY2ei9ukhOrrIzChPaxJqaIKT6BXNn4PVZCvKlW8btoco9nd5j2eGbZapGRpvpHvxGeyB0LgRIPEMM93ebCs6C7YN3dzqDdM7ZRl/Bcu6c67WHUr7VXMThINv1ZpKde1v6G8M+zWa+zjncn21P48h12t8+X6Ha0vJ8+3eq/RmTc0auVbHojrvPqCNhMnDXPYI2fs3KiwodCnhlOnzt5XEY+5XzaHIeqE0iKW6TeJyt6lO/lpC+LEj6NL0Xbt3oGE1uU23WaYExTNp9fQ55ylRgS8SsnjixFQrEYB+dkAAQu2QMp7bfChg2PdID/+wm7xf5GJONCWlXT96KTj7lPfTdlmDqhC/Bxc71s91hlaH+GeveL/UBMtLDE5a5m11LyZPcBMHq0qDWNUUzfNBr9CaXBg5t2r4HAFsRSdzgxOztFcM5SNm6wTfL4ZUSXCN09h4lQXDZVF8COloQMgrqfb4ACPMhKZClB2Y97TuP6OnLCoMixxnMLFGVRx/XrrXX776bW5bocBvWzPNkWxnbC39 EEIPtN53 jV7UwFvrFa9PTXnl6NlI1753EHfQyQN7PNTh0RndLbSgSBRNRetbeo0Q27zodDVmi92icn00CY8VHimGi9WpCZdHh691JE4hyKj5fnwQWeyalm+8HshVPARRVn+hvhmHs/MviyF44JicY9ue990xzsFU10hos2ukSO+tlR/dJOvI2daXx89zzeASBx/Ziry0L5tCe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Memory acceptance requires a hypercall and one or multiple module calls. Make helpers for the calls available in boot stub. It has to accept memory where kernel image and initrd are placed. Signed-off-by: Kirill A. Shutemov Reviewed-by: Dave Hansen --- arch/x86/coco/tdx/tdx.c | 32 ------------------- arch/x86/include/asm/shared/tdx.h | 51 +++++++++++++++++++++++++++++++ arch/x86/include/asm/tdx.h | 19 ------------ 3 files changed, 51 insertions(+), 51 deletions(-) diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index e146b599260f..e6f4c2758a68 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -14,20 +14,6 @@ #include #include -/* TDX module Call Leaf IDs */ -#define TDX_GET_INFO 1 -#define TDX_GET_VEINFO 3 -#define TDX_GET_REPORT 4 -#define TDX_ACCEPT_PAGE 6 -#define TDX_WR 8 - -/* TDCS fields. To be used by TDG.VM.WR and TDG.VM.RD module calls */ -#define TDCS_NOTIFY_ENABLES 0x9100000000000010 - -/* TDX hypercall Leaf IDs */ -#define TDVMCALL_MAP_GPA 0x10001 -#define TDVMCALL_REPORT_FATAL_ERROR 0x10003 - /* MMIO direction */ #define EPT_READ 0 #define EPT_WRITE 1 @@ -51,24 +37,6 @@ #define TDREPORT_SUBTYPE_0 0 -/* - * Wrapper for standard use of __tdx_hypercall with no output aside from - * return code. - */ -static inline u64 _tdx_hypercall(u64 fn, u64 r12, u64 r13, u64 r14, u64 r15) -{ - struct tdx_hypercall_args args = { - .r10 = TDX_HYPERCALL_STANDARD, - .r11 = fn, - .r12 = r12, - .r13 = r13, - .r14 = r14, - .r15 = r15, - }; - - return __tdx_hypercall(&args); -} - /* Called from __tdx_hypercall() for unrecoverable failure */ noinstr void __tdx_hypercall_failed(void) { diff --git a/arch/x86/include/asm/shared/tdx.h b/arch/x86/include/asm/shared/tdx.h index 2631e01f6e0f..1ff0ee822961 100644 --- a/arch/x86/include/asm/shared/tdx.h +++ b/arch/x86/include/asm/shared/tdx.h @@ -10,6 +10,20 @@ #define TDX_CPUID_LEAF_ID 0x21 #define TDX_IDENT "IntelTDX " +/* TDX module Call Leaf IDs */ +#define TDX_GET_INFO 1 +#define TDX_GET_VEINFO 3 +#define TDX_GET_REPORT 4 +#define TDX_ACCEPT_PAGE 6 +#define TDX_WR 8 + +/* TDCS fields. To be used by TDG.VM.WR and TDG.VM.RD module calls */ +#define TDCS_NOTIFY_ENABLES 0x9100000000000010 + +/* TDX hypercall Leaf IDs */ +#define TDVMCALL_MAP_GPA 0x10001 +#define TDVMCALL_REPORT_FATAL_ERROR 0x10003 + #ifndef __ASSEMBLY__ /* @@ -37,8 +51,45 @@ struct tdx_hypercall_args { u64 __tdx_hypercall(struct tdx_hypercall_args *args); u64 __tdx_hypercall_ret(struct tdx_hypercall_args *args); +/* + * Wrapper for standard use of __tdx_hypercall with no output aside from + * return code. + */ +static inline u64 _tdx_hypercall(u64 fn, u64 r12, u64 r13, u64 r14, u64 r15) +{ + struct tdx_hypercall_args args = { + .r10 = TDX_HYPERCALL_STANDARD, + .r11 = fn, + .r12 = r12, + .r13 = r13, + .r14 = r14, + .r15 = r15, + }; + + return __tdx_hypercall(&args); +} + + /* Called from __tdx_hypercall() for unrecoverable failure */ void __tdx_hypercall_failed(void); +/* + * Used in __tdx_module_call() to gather the output registers' values of the + * TDCALL instruction when requesting services from the TDX module. This is a + * software only structure and not part of the TDX module/VMM ABI + */ +struct tdx_module_output { + u64 rcx; + u64 rdx; + u64 r8; + u64 r9; + u64 r10; + u64 r11; +}; + +/* Used to communicate with the TDX module */ +u64 __tdx_module_call(u64 fn, u64 rcx, u64 rdx, u64 r8, u64 r9, + struct tdx_module_output *out); + #endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_SHARED_TDX_H */ diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 28d889c9aa16..234197ec17e4 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -20,21 +20,6 @@ #ifndef __ASSEMBLY__ -/* - * Used to gather the output registers values of the TDCALL and SEAMCALL - * instructions when requesting services from the TDX module. - * - * This is a software only structure and not part of the TDX module/VMM ABI. - */ -struct tdx_module_output { - u64 rcx; - u64 rdx; - u64 r8; - u64 r9; - u64 r10; - u64 r11; -}; - /* * Used by the #VE exception handler to gather the #VE exception * info from the TDX module. This is a software only structure @@ -55,10 +40,6 @@ struct ve_info { void __init tdx_early_init(void); -/* Used to communicate with the TDX module */ -u64 __tdx_module_call(u64 fn, u64 rcx, u64 rdx, u64 r8, u64 r9, - struct tdx_module_output *out); - void tdx_get_ve_info(struct ve_info *ve); bool tdx_handle_virt_exception(struct pt_regs *regs, struct ve_info *ve); From patchwork Sat May 13 22:04:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81068C77B7F for ; Sat, 13 May 2023 22:05:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3D39280004; Sat, 13 May 2023 18:04:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9ED14280001; Sat, 13 May 2023 18:04:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88D50280004; Sat, 13 May 2023 18:04:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 771F4280001 for ; Sat, 13 May 2023 18:04:58 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4A4D11C8525 for ; Sat, 13 May 2023 22:04:58 +0000 (UTC) X-FDA: 80786612676.30.340E627 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf14.hostedemail.com (Postfix) with ESMTP id 2A816100006 for ; Sat, 13 May 2023 22:04:54 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ah3NZ56C; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015495; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5QNweM8SAaqY0llbQtMpYPXKIrYhie9z8hDfHUvFa6Y=; b=lvwYMjiPNZPy+XU2NkdYDx+fKP78N3A550+WcnIsySLHTeLAzw6u28eWT+VWg/EfuO4XhY LfuEUhArfyYPqSVDZnw5eG/vLE1IFQpeFZ3g5Lx+r4rwldCcW0BobRlFCQpPOQyZXciqs0 324J3RRG7ClzM38MEYLgthnPE6yRfoM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ah3NZ56C; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf14.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015495; a=rsa-sha256; cv=none; b=Qv2y/DbzQvJXS71CA3Pa28pWAgoR+LdDqlKm27eIaFuje5o2N8UyFK0TbmpeixMEU8d7+J d+/v59rhGVsp19U/eYOjfgHPrqWxLJDwepeGGDFQRUdCFBfNQ6NVg8EVX8AsYvBGZig36M vqP3eKMw3KMb+1OsDfy+Sv/Xfh5lIxY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015495; x=1715551495; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZRWbbgKJS3P7pe8ruC5AYEzSpFJMHJrlR+e2/skszlg=; b=ah3NZ56CmUsdBetq7x6PhQ2LZ36WStTjZJk95B5qJRg+IDk4X/AZ4IG2 AtlNBfOeorSY1UyjEvCIG2GG8LU6U8SnYsoghrGbg5yb4dXsN22Twg9xH 8P/OXo0vCjG/vbMu3mNq//+AtvSmJFffw33p8WEvrqISOA5Gf0n/fInl+ HdVLJm9QGuTmOf08DPEsipTLaNBPWyULlPVZWpYJBhIKX0qBERg7PSiuX ZdmIbyRSmFiF+JAsbaL9Eud1P7hPApk2WmbPkNln9H8cWyQW+CnKRmAaR eSz6imkA5agVOnkL6U8DL5UtwflSShcSQD4zwzQwixLny/1J/LqKBTG7W A==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="379142571" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="379142571" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="733429556" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="733429556" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:45 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 2F03A10CF7E; Sun, 14 May 2023 01:04:34 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Dave Hansen Subject: [PATCHv11 8/9] x86/tdx: Refactor try_accept_one() Date: Sun, 14 May 2023 01:04:17 +0300 Message-Id: <20230513220418.19357-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: fgx9wukp1rkzxs737axzn8uikttjiugb X-Rspam-User: X-Rspamd-Queue-Id: 2A816100006 X-Rspamd-Server: rspam07 X-HE-Tag: 1684015494-521142 X-HE-Meta: U2FsdGVkX19JzK91gvmrhSulSI2UV/6k7GHqIbMhYqj9LOChsD/d7ZD23mbsRyjMST4L2A07ZPBjlTsXnelRddXZLk/kttlETy6TaLpunF9tmmCRXY/DUZsKh6WwbAFIuWFGrzbKIzFB6YqLkDQYEVUDJIAZJNQsrO1hF9Y13QDSVBb0jyK0bZi6KCvGoFpNIyyPyiJrMRLMsKmQNQ/728mZCGqNjbE5PXZqRjoxMoO/0Ab4vz2Mni74eyQ+GYpJavTmF/YsFWkwZqSulDCgQ6wuWKXa+QHEKuK8QhhnJw3I3sUGm/IXJI/8tmzWdVqzdsYykE34zx05Qq4tQgK4xcjS5YioaG1xfrnAM/vcjqk8V/yhaRgWIQORsrhlUJsELtxxGdIhG/jQ0jL0fG5wc+/G/wkHerF0V8AWJewohCBKIV+H1qKvdwy1/sUYO3SiTS5Nl/bEB5PQpcf8494+bTM1C/pMq7CNuOc5pSb2YZjVIvhOCS5zxfJnVN9ipiu4iuujxvQRILDKQl3z2PKvsley/4SZhhcFd/o1MTnREUCqTt9G4KrtQ2TD79HXpFleQQOVWrZ8Ivu4e80IfiF0zRo6aTaSkXZH9y4ggyqPd0XKpSgv15EK3dIRtep6VRuPs/ZrzgrlfCrp79DAqq3FYSZ+IYATO/SQgN8FZpN6AATSYVameiSQbT5qiZSbenTVtK/junknbjOIQaMaRgDiYZY2ybeeb3t8LKWmVqSsiqLKZHuQMhsCltdioW9N3+xqvCpw5tTL6btK7lh8uE2IwnzCP8300G0v320gNJxyiDmBgmQy6OaDtpGWdqnct7irdLhmXCPuEoyYmDjuAfAX4Cxkm2wAC9Vua2sZfJY9oWhftXWQW+UcuBuT0OSSEwHk42fEcBvKkCJ/sg8s36PZEjp3LI0Hg3doxUHAjM1ATlYTNO6uXrWeV+9/X870yR4XeJ0+9tPh10JzKODsTb8 EDBshP4h Fva3VhlBQXFRAIEO8XCbX0i0uSxhFAaXbMLaD4sF8kZUEhdgscm4TiLXLRV/SNjF12cHohkh3zUcVWbJihlgKXeyb9uZH+wNj1SQtY9lLUUY6P+DqEtTcWA2nm5L4uBwoUDPAhREH1xHWJdEbAkjssIBswfhHm9g89deBGLLz2rnwsUv/mF6sQfF2Og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rework try_accept_one() to return accepted size instead of modifying 'start' inside the helper. It makes 'start' in-only argument and streamlines code on the caller side. Signed-off-by: Kirill A. Shutemov Suggested-by: Borislav Petkov Reviewed-by: Dave Hansen --- arch/x86/coco/tdx/tdx.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index e6f4c2758a68..0d5fe6e24e45 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -713,18 +713,18 @@ static bool tdx_cache_flush_required(void) return true; } -static bool try_accept_one(phys_addr_t *start, unsigned long len, - enum pg_level pg_level) +static unsigned long try_accept_one(phys_addr_t start, unsigned long len, + enum pg_level pg_level) { unsigned long accept_size = page_level_size(pg_level); u64 tdcall_rcx; u8 page_size; - if (!IS_ALIGNED(*start, accept_size)) - return false; + if (!IS_ALIGNED(start, accept_size)) + return 0; if (len < accept_size) - return false; + return 0; /* * Pass the page physical address to the TDX module to accept the @@ -743,15 +743,14 @@ static bool try_accept_one(phys_addr_t *start, unsigned long len, page_size = 2; break; default: - return false; + return 0; } - tdcall_rcx = *start | page_size; + tdcall_rcx = start | page_size; if (__tdx_module_call(TDX_ACCEPT_PAGE, tdcall_rcx, 0, 0, 0, NULL)) - return false; + return 0; - *start += accept_size; - return true; + return accept_size; } /* @@ -788,21 +787,22 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc) */ while (start < end) { unsigned long len = end - start; + unsigned long accept_size; /* * Try larger accepts first. It gives chance to VMM to keep - * 1G/2M SEPT entries where possible and speeds up process by - * cutting number of hypercalls (if successful). + * 1G/2M Secure EPT entries where possible and speeds up + * process by cutting number of hypercalls (if successful). */ - if (try_accept_one(&start, len, PG_LEVEL_1G)) - continue; - - if (try_accept_one(&start, len, PG_LEVEL_2M)) - continue; - - if (!try_accept_one(&start, len, PG_LEVEL_4K)) + accept_size = try_accept_one(start, len, PG_LEVEL_1G); + if (!accept_size) + accept_size = try_accept_one(start, len, PG_LEVEL_2M); + if (!accept_size) + accept_size = try_accept_one(start, len, PG_LEVEL_4K); + if (!accept_size) return false; + start += accept_size; } return true; From patchwork Sat May 13 22:04:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13240350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 913BFC7EE2A for ; Sat, 13 May 2023 22:05:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07B49280007; Sat, 13 May 2023 18:04:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E802F280005; Sat, 13 May 2023 18:04:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8BE4280007; Sat, 13 May 2023 18:04:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AB05C280005 for ; Sat, 13 May 2023 18:04:58 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7FC30A117A for ; Sat, 13 May 2023 22:04:58 +0000 (UTC) X-FDA: 80786612676.27.4B9C923 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf24.hostedemail.com (Postfix) with ESMTP id 3E40F18000A for ; Sat, 13 May 2023 22:04:56 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F3oAwyFO; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684015496; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TZk//vcQTjKDGlwD8msh2Ls5h9OpubPnLBboQcSMSOA=; b=ns53k+RLL//p8cCq2sIk6GojdgEG2BLAuXQ5QbT55WkpDmAtoMxJJXnSrgqbIox40jwevG Hs9YL8Y1bvHfPcamt89TsuY6SI+0B+zAI+kojPRWEenn1vHrOh3nZL0mcvR49ckTYfj/LK EVK3m2NboHatdT+4xFYPyf1fuYKuL2Q= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F3oAwyFO; spf=none (imf24.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684015496; a=rsa-sha256; cv=none; b=xA3WqHn2nltBFMdNqbT0Ey3NxfQx5FV6E1fcHR1ulruPOpqh8O6ehRQTLi64v+staBLVdb DBGvZxbqanog2/TVStrpiqDfUfDRyATjF4fF62CWYqo84cp1yYiozBgbLTMkMoyXxBsKlE mHjTcBpv00duyk63/97l/SzNb7yPq5M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684015496; x=1715551496; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TXEN8aOVCx4uC+2oczJcWR+KSaKZgQy+3tVmVD2kOCc=; b=F3oAwyFO3sWSlRl9QbWTwW4NXUhVdxTNRjwTFY7Y1E9RivV4hlB+CVua JeZWCd6EeFbA4Flo8sJ+0vLO/MPuxaDOiRmu66vPB/SwXn89s+NTH4Cwx jL/K5lQgzWXplhA3JbwUg3QAsi7j6TL4Q1RSlWo3OQCtQgT3rGi0s1HMF fW6dnwE8Vh3t4QzaxzeEoKMwr270B8DxoR4evWGL6KhLNBY1pC6kDhfsB WnqotEwjQ7ssX4XOWJJSf5b5ts2HD3NVCyoRI/nkfTW7Vh4lVe4GA4H0h K2X2F3K4rYZwc+8HFnwdNhyC1d7YW5Dg3zPKnsI4Lnp+sOC3N101n+qg+ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="379142587" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="379142587" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10709"; a="733429558" X-IronPort-AV: E=Sophos;i="5.99,273,1677571200"; d="scan'208";a="733429558" Received: from sorinaau-mobl1.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.145]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 15:04:45 -0700 Received: by box.shutemov.name (Postfix, from userid 1000) id 39B2D10CF7F; Sun, 14 May 2023 01:04:34 +0300 (+03) From: "Kirill A. Shutemov" To: Borislav Petkov , Andy Lutomirski , Dave Hansen , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Dario Faggioli , Mike Rapoport , David Hildenbrand , Mel Gorman , marcelo.cerri@canonical.com, tim.gardner@canonical.com, khalid.elmously@canonical.com, philip.cox@canonical.com, aarcange@redhat.com, peterx@redhat.com, x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv11 9/9] x86/tdx: Add unaccepted memory support Date: Sun, 14 May 2023 01:04:18 +0300 Message-Id: <20230513220418.19357-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Stat-Signature: reofc9sd9phoi3h3jw3yhbyiu7eckfpf X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3E40F18000A X-Rspam-User: X-HE-Tag: 1684015496-877897 X-HE-Meta: U2FsdGVkX1/yybs68QsQVTn3jvuUH/bwVJLWI10WS55Afs5xwSJ8X2991aaOGrf90w7FuaBQ5wBjqeMbNjlRXgteV5E5j70VVrm9KrKYVfClreLD0k3WB1MKsrSWLZm4Ez8cViYdV8tqlNjpfP9ZcZfwsh0MC138YrnxF4zjz6y5VztpfyYSlihhO2g6EVfUdXPJMzKQZX9WOFukMeYOrSyths2L/IWDK/EnQ+z1VYH6DT0NS6ixsQctAfTHHTiOr/mL4SFbUgMPUczUzQp/yOgYQSK8GtF8eHXzBfFmmbZFty5HnkO8U4LoozP2PtJy9c83JUkiCbZpHb33epv3v0T2FqewL3zrONc8dGsve0vUK6B3cudjgxJxcKtUmF1o1CrodXMRlV9FuLM0VGCH6xBQilUvFohEUVoVSQIUt/wIM4sBYVRrOdqjLrmQzxgeQcZI5a0fYEMnHgn7ZZUkBH2cXzWGvR/odTmtd6dfJh7YS4paAw/HbRwIZoiV2X9V/QlVKXR7m9JipG8d3yY+LuSmpCQbn7lTJVNJRJiPTmIB1X+cTEOlmhTdmE3PoTpP7N+lrL4EcN3rYxKpQKpqUsJ4D5fIo9xxXKiPEUi8rYCmQAmQZjxsEzY0sSqQevWfRUUykn2gJWnUgponVzJIb1dMrWB5HGQ0MscRPlYHfUvHiwIMGZ9SAtZcu4j209K3wEQbfIOaq/tjAAyN8E/4cOtLWdDrbG9WzVNyxxeC/UapUlfuy+MF/kCYGzfi8Ns6bkDUyysXgvAc65l+LtO/n74E5j8Ge6ymhe5F+jtM9RGA2s5VTc6lZeBpJCzq0wpz84b0rvOyExKP8G4kGtghO6WCU6LC+q6mqLoOd9/VhKJJT/f+hIVNuxA7HFtA1Xt24Po9Bqrn9FS52oP7bFYXkTJOJWKdhHVNW9oS15omTX3FLPbTU1npujxC+xPDUZ0WEoJZpMjDexqlw+AYI// XnVsmUXI 81+ihrXbRtCIsYA+lXoWXNx/aSZXMFQgy/28M/OOOUuHHUgEe58YlKAu2CNKQ+QhThAsdqOh6+q0rrHXuubZu0y44oXhvfPClre26ieAvb5yosGEPILjkCA4jXThrBJoPzWTxIh+4USs3dAKewdDHaUKvCQCNLAZHszWVekXgbSi9310cq0CfEDEc1D6gbvNFAD83Om/8XtB7lVs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hookup TDX-specific code to accept memory. Accepting the memory is the same process as converting memory from shared to private: kernel notifies VMM with MAP_GPA hypercall and then accept pages with ACCEPT_PAGE module call. The implementation in core kernel uses tdx_enc_status_changed(). It already used for converting memory to shared and back for I/O transactions. Boot stub provides own implementation of tdx_accept_memory(). It is similar in structure to tdx_enc_status_changed(), but only cares about converting memory to private. Signed-off-by: Kirill A. Shutemov --- arch/x86/Kconfig | 2 + arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/error.c | 19 +++++ arch/x86/boot/compressed/error.h | 1 + arch/x86/boot/compressed/mem.c | 35 ++++++++- arch/x86/boot/compressed/tdx-shared.c | 2 + arch/x86/boot/compressed/tdx.c | 37 +++++++++ arch/x86/coco/tdx/Makefile | 2 +- arch/x86/coco/tdx/tdx-shared.c | 95 ++++++++++++++++++++++++ arch/x86/coco/tdx/tdx.c | 86 +-------------------- arch/x86/include/asm/shared/tdx.h | 2 + arch/x86/include/asm/tdx.h | 2 + arch/x86/include/asm/unaccepted_memory.h | 23 ++++++ 13 files changed, 221 insertions(+), 87 deletions(-) create mode 100644 arch/x86/boot/compressed/tdx-shared.c create mode 100644 arch/x86/coco/tdx/tdx-shared.c create mode 100644 arch/x86/include/asm/unaccepted_memory.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 53bab123a8ee..5c72067c06d4 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -884,9 +884,11 @@ config INTEL_TDX_GUEST bool "Intel TDX (Trust Domain Extensions) - Guest Support" depends on X86_64 && CPU_SUP_INTEL depends on X86_X2APIC + depends on EFI_STUB select ARCH_HAS_CC_PLATFORM select X86_MEM_ENCRYPT select X86_MCE + select UNACCEPTED_MEMORY help Support running as a guest under Intel TDX. Without this support, the guest kernel can not boot or run under TDX. diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index cc4978123c30..78f67e0a2666 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -107,7 +107,7 @@ endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o -vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/mem.o +vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/mem.o $(obj)/tdx-shared.o vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_mixed.o diff --git a/arch/x86/boot/compressed/error.c b/arch/x86/boot/compressed/error.c index c881878e56d3..5313c5cb2b80 100644 --- a/arch/x86/boot/compressed/error.c +++ b/arch/x86/boot/compressed/error.c @@ -22,3 +22,22 @@ void error(char *m) while (1) asm("hlt"); } + +/* EFI libstub provides vsnprintf() */ +#ifdef CONFIG_EFI_STUB +void panic(const char *fmt, ...) +{ + static char buf[1024]; + va_list args; + int len; + + va_start(args, fmt); + len = vsnprintf(buf, sizeof(buf), fmt, args); + va_end(args); + + if (len && buf[len - 1] == '\n') + buf[len - 1] = '\0'; + + error(buf); +} +#endif diff --git a/arch/x86/boot/compressed/error.h b/arch/x86/boot/compressed/error.h index 1de5821184f1..86fe33b93715 100644 --- a/arch/x86/boot/compressed/error.h +++ b/arch/x86/boot/compressed/error.h @@ -6,5 +6,6 @@ void warn(char *m); void error(char *m) __noreturn; +void panic(const char *fmt, ...) __noreturn __cold; #endif /* BOOT_COMPRESSED_ERROR_H */ diff --git a/arch/x86/boot/compressed/mem.c b/arch/x86/boot/compressed/mem.c index 67594fcb11d9..a4308d077885 100644 --- a/arch/x86/boot/compressed/mem.c +++ b/arch/x86/boot/compressed/mem.c @@ -1,9 +1,42 @@ // SPDX-License-Identifier: GPL-2.0-only +#include "../cpuflags.h" +#include "../string.h" #include "error.h" +#include "tdx.h" +#include + +/* + * accept_memory() and process_unaccepted_memory() called from EFI stub which + * runs before decompresser and its early_tdx_detect(). + * + * Enumerate TDX directly from the early users. + */ +static bool early_is_tdx_guest(void) +{ + static bool once; + static bool is_tdx; + + if (!IS_ENABLED(CONFIG_INTEL_TDX_GUEST)) + return false; + + if (!once) { + u32 eax, sig[3]; + + cpuid_count(TDX_CPUID_LEAF_ID, 0, &eax, + &sig[0], &sig[2], &sig[1]); + is_tdx = !memcmp(TDX_IDENT, sig, sizeof(sig)); + once = true; + } + + return is_tdx; +} void arch_accept_memory(phys_addr_t start, phys_addr_t end) { /* Platform-specific memory-acceptance call goes here */ - error("Cannot accept memory"); + if (early_is_tdx_guest()) + tdx_accept_memory(start, end); + else + error("Cannot accept memory: unknown platform\n"); } diff --git a/arch/x86/boot/compressed/tdx-shared.c b/arch/x86/boot/compressed/tdx-shared.c new file mode 100644 index 000000000000..5ac43762fe13 --- /dev/null +++ b/arch/x86/boot/compressed/tdx-shared.c @@ -0,0 +1,2 @@ +#include "error.h" +#include "../../coco/tdx/tdx-shared.c" diff --git a/arch/x86/boot/compressed/tdx.c b/arch/x86/boot/compressed/tdx.c index 2d81d3cc72a1..d073764eaa50 100644 --- a/arch/x86/boot/compressed/tdx.c +++ b/arch/x86/boot/compressed/tdx.c @@ -9,6 +9,9 @@ #include #include +#include + +static u64 cc_mask; /* Called from __tdx_hypercall() for unrecoverable failure */ void __tdx_hypercall_failed(void) @@ -16,6 +19,38 @@ void __tdx_hypercall_failed(void) error("TDVMCALL failed. TDX module bug?"); } +static u64 get_cc_mask(void) +{ + struct tdx_module_output out; + unsigned int gpa_width; + + /* + * TDINFO TDX module call is used to get the TD execution environment + * information like GPA width, number of available vcpus, debug mode + * information, etc. More details about the ABI can be found in TDX + * Guest-Host-Communication Interface (GHCI), section 2.4.2 TDCALL + * [TDG.VP.INFO]. + * + * The GPA width that comes out of this call is critical. TDX guests + * can not meaningfully run without it. + */ + if (__tdx_module_call(TDX_GET_INFO, 0, 0, 0, 0, &out)) + error("TDCALL GET_INFO failed (Buggy TDX module!)\n"); + + gpa_width = out.rcx & GENMASK(5, 0); + + /* + * The highest bit of a guest physical address is the "sharing" bit. + * Set it for shared pages and clear it for private pages. + */ + return BIT_ULL(gpa_width - 1); +} + +u64 cc_mkdec(u64 val) +{ + return val & ~cc_mask; +} + static inline unsigned int tdx_io_in(int size, u16 port) { struct tdx_hypercall_args args = { @@ -70,6 +105,8 @@ void early_tdx_detect(void) if (memcmp(TDX_IDENT, sig, sizeof(sig))) return; + cc_mask = get_cc_mask(); + /* Use hypercalls instead of I/O instructions */ pio_ops.f_inb = tdx_inb; pio_ops.f_outb = tdx_outb; diff --git a/arch/x86/coco/tdx/Makefile b/arch/x86/coco/tdx/Makefile index 46c55998557d..2c7dcbf1458b 100644 --- a/arch/x86/coco/tdx/Makefile +++ b/arch/x86/coco/tdx/Makefile @@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += tdx.o tdcall.o +obj-y += tdx.o tdx-shared.o tdcall.o diff --git a/arch/x86/coco/tdx/tdx-shared.c b/arch/x86/coco/tdx/tdx-shared.c new file mode 100644 index 000000000000..ee74f7bbe806 --- /dev/null +++ b/arch/x86/coco/tdx/tdx-shared.c @@ -0,0 +1,95 @@ +#include +#include + +static unsigned long try_accept_one(phys_addr_t start, unsigned long len, + enum pg_level pg_level) +{ + unsigned long accept_size = page_level_size(pg_level); + u64 tdcall_rcx; + u8 page_size; + + if (!IS_ALIGNED(start, accept_size)) + return 0; + + if (len < accept_size) + return 0; + + /* + * Pass the page physical address to the TDX module to accept the + * pending, private page. + * + * Bits 2:0 of RCX encode page size: 0 - 4K, 1 - 2M, 2 - 1G. + */ + switch (pg_level) { + case PG_LEVEL_4K: + page_size = 0; + break; + case PG_LEVEL_2M: + page_size = 1; + break; + case PG_LEVEL_1G: + page_size = 2; + break; + default: + return 0; + } + + tdcall_rcx = start | page_size; + if (__tdx_module_call(TDX_ACCEPT_PAGE, tdcall_rcx, 0, 0, 0, NULL)) + return 0; + + return accept_size; +} + +bool tdx_enc_status_changed_phys(phys_addr_t start, phys_addr_t end, bool enc) +{ + if (!enc) { + /* Set the shared (decrypted) bits: */ + start |= cc_mkdec(0); + end |= cc_mkdec(0); + } + + /* + * Notify the VMM about page mapping conversion. More info about ABI + * can be found in TDX Guest-Host-Communication Interface (GHCI), + * section "TDG.VP.VMCALL" + */ + if (_tdx_hypercall(TDVMCALL_MAP_GPA, start, end - start, 0, 0)) + return false; + + /* private->shared conversion requires only MapGPA call */ + if (!enc) + return true; + + /* + * For shared->private conversion, accept the page using + * TDX_ACCEPT_PAGE TDX module call. + */ + while (start < end) { + unsigned long len = end - start; + unsigned long accept_size; + + /* + * Try larger accepts first. It gives chance to VMM to keep + * 1G/2M Secure EPT entries where possible and speeds up + * process by cutting number of hypercalls (if successful). + */ + + accept_size = try_accept_one(start, len, PG_LEVEL_1G); + if (!accept_size) + accept_size = try_accept_one(start, len, PG_LEVEL_2M); + if (!accept_size) + accept_size = try_accept_one(start, len, PG_LEVEL_4K); + if (!accept_size) + return false; + start += accept_size; + } + + return true; +} + +void tdx_accept_memory(phys_addr_t start, phys_addr_t end) +{ + if (!tdx_enc_status_changed_phys(start, end, true)) + panic("Accepting memory failed: %#llx-%#llx\n", start, end); +} diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index 0d5fe6e24e45..32501277ef84 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -713,46 +713,6 @@ static bool tdx_cache_flush_required(void) return true; } -static unsigned long try_accept_one(phys_addr_t start, unsigned long len, - enum pg_level pg_level) -{ - unsigned long accept_size = page_level_size(pg_level); - u64 tdcall_rcx; - u8 page_size; - - if (!IS_ALIGNED(start, accept_size)) - return 0; - - if (len < accept_size) - return 0; - - /* - * Pass the page physical address to the TDX module to accept the - * pending, private page. - * - * Bits 2:0 of RCX encode page size: 0 - 4K, 1 - 2M, 2 - 1G. - */ - switch (pg_level) { - case PG_LEVEL_4K: - page_size = 0; - break; - case PG_LEVEL_2M: - page_size = 1; - break; - case PG_LEVEL_1G: - page_size = 2; - break; - default: - return 0; - } - - tdcall_rcx = start | page_size; - if (__tdx_module_call(TDX_ACCEPT_PAGE, tdcall_rcx, 0, 0, 0, NULL)) - return 0; - - return accept_size; -} - /* * Inform the VMM of the guest's intent for this physical page: shared with * the VMM or private to the guest. The VMM is expected to change its mapping @@ -761,51 +721,9 @@ static unsigned long try_accept_one(phys_addr_t start, unsigned long len, static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc) { phys_addr_t start = __pa(vaddr); - phys_addr_t end = __pa(vaddr + numpages * PAGE_SIZE); + phys_addr_t end = __pa(vaddr + numpages * PAGE_SIZE); - if (!enc) { - /* Set the shared (decrypted) bits: */ - start |= cc_mkdec(0); - end |= cc_mkdec(0); - } - - /* - * Notify the VMM about page mapping conversion. More info about ABI - * can be found in TDX Guest-Host-Communication Interface (GHCI), - * section "TDG.VP.VMCALL" - */ - if (_tdx_hypercall(TDVMCALL_MAP_GPA, start, end - start, 0, 0)) - return false; - - /* private->shared conversion requires only MapGPA call */ - if (!enc) - return true; - - /* - * For shared->private conversion, accept the page using - * TDX_ACCEPT_PAGE TDX module call. - */ - while (start < end) { - unsigned long len = end - start; - unsigned long accept_size; - - /* - * Try larger accepts first. It gives chance to VMM to keep - * 1G/2M Secure EPT entries where possible and speeds up - * process by cutting number of hypercalls (if successful). - */ - - accept_size = try_accept_one(start, len, PG_LEVEL_1G); - if (!accept_size) - accept_size = try_accept_one(start, len, PG_LEVEL_2M); - if (!accept_size) - accept_size = try_accept_one(start, len, PG_LEVEL_4K); - if (!accept_size) - return false; - start += accept_size; - } - - return true; + return tdx_enc_status_changed_phys(start, end, enc); } void __init tdx_early_init(void) diff --git a/arch/x86/include/asm/shared/tdx.h b/arch/x86/include/asm/shared/tdx.h index 1ff0ee822961..95fbe7376694 100644 --- a/arch/x86/include/asm/shared/tdx.h +++ b/arch/x86/include/asm/shared/tdx.h @@ -91,5 +91,7 @@ struct tdx_module_output { u64 __tdx_module_call(u64 fn, u64 rcx, u64 rdx, u64 r8, u64 r9, struct tdx_module_output *out); +void tdx_accept_memory(phys_addr_t start, phys_addr_t end); + #endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_SHARED_TDX_H */ diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h index 234197ec17e4..3a7340ad9a4b 100644 --- a/arch/x86/include/asm/tdx.h +++ b/arch/x86/include/asm/tdx.h @@ -50,6 +50,8 @@ bool tdx_early_handle_ve(struct pt_regs *regs); int tdx_mcall_get_report0(u8 *reportdata, u8 *tdreport); +bool tdx_enc_status_changed_phys(phys_addr_t start, phys_addr_t end, bool enc); + #else static inline void tdx_early_init(void) { }; diff --git a/arch/x86/include/asm/unaccepted_memory.h b/arch/x86/include/asm/unaccepted_memory.h new file mode 100644 index 000000000000..72b354f992bb --- /dev/null +++ b/arch/x86/include/asm/unaccepted_memory.h @@ -0,0 +1,23 @@ +#ifndef _ASM_X86_UNACCEPTED_MEMORY_H +#define _ASM_X86_UNACCEPTED_MEMORY_H + +#include +#include + +static inline void arch_accept_memory(phys_addr_t start, phys_addr_t end) +{ + /* Platform-specific memory-acceptance call goes here */ + if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) { + tdx_accept_memory(start, end); + } else { + panic("Cannot accept memory: unknown platform\n"); + } +} + +static inline struct efi_unaccepted_memory *efi_get_unaccepted_table(void) +{ + if (efi.unaccepted == EFI_INVALID_TABLE_ADDR) + return NULL; + return __va(efi.unaccepted); +} +#endif