From patchwork Sat Mar 13 16:01:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jarkko Sakkinen X-Patchwork-Id: 12136851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 595B6C433E6 for ; Sat, 13 Mar 2021 16:02:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 163AE64F21 for ; Sat, 13 Mar 2021 16:02:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233570AbhCMQCD (ORCPT ); Sat, 13 Mar 2021 11:02:03 -0500 Received: from mail.kernel.org ([198.145.29.99]:48066 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233868AbhCMQBu (ORCPT ); Sat, 13 Mar 2021 11:01:50 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 7B59264F1E; Sat, 13 Mar 2021 16:01:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1615651310; bh=HuaFV7Aup5s4tI28aR1o2B7ZXch9cKa+janu7Y/uWDY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=en7Bcj1pJwbSqsmCJ3hY0FkQ2tA65N1v8h+pQqIOAMKD9CMHRLbQSmwYKoQrJ1jUN 31Y+f9I9Y0cML9AR0oOayF1C6ZViKrjIdm+3Rr3lN8qLuC02UoZ8/J1B1M5aDpMkyn Q3jfqUq9u8GoMPpesYwT3B++ReS2f2dgmKLb1+ZUhLwde+ItegYIxTvYK+u22Q0sHv 7oit2HVyPrJDRFw+4UBGXObbAOcqIbub511CUYCOPYG64MNnWw6WR6Z+nT7PRYEd0L 4qPKSFbwvENY9uuFOCtH8Y7ccPLRJQTN+cF/17NyiH2yChUOpoIRQo+hoamUx/YAqn y32Rv0nQ7DFfg== From: Jarkko Sakkinen To: linux-sgx@vger.kernel.org Cc: Jarkko Sakkinen , Dave Hansen , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org Subject: [PATCH v4 1/3] x86/sgx: Use sgx_free_epc_page() in sgx_reclaim_pages() Date: Sat, 13 Mar 2021 18:01:17 +0200 Message-Id: <20210313160119.1318533-2-jarkko@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210313160119.1318533-1-jarkko@kernel.org> References: <20210313160119.1318533-1-jarkko@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org Replace the ad-hoc code with a sgx_free_epc_page(), in order to make sure that all the relevant checks and book keeping is done, while freeing a borrowed EPC page, and remove redundant code. EREMOVE inside sgx_free_epc_page() does not change the semantics, as EREMOVE to an uninitialize pages is a nop. Signed-off-by: Jarkko Sakkinen --- v4: * Rewrote the commit message. arch/x86/kernel/cpu/sgx/main.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index 8df81a3ed945..65004fb8a91f 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -305,7 +305,6 @@ static void sgx_reclaim_pages(void) { struct sgx_epc_page *chunk[SGX_NR_TO_SCAN]; struct sgx_backing backing[SGX_NR_TO_SCAN]; - struct sgx_epc_section *section; struct sgx_encl_page *encl_page; struct sgx_epc_page *epc_page; pgoff_t page_index; @@ -378,11 +377,7 @@ static void sgx_reclaim_pages(void) kref_put(&encl_page->encl->refcount, sgx_encl_release); epc_page->flags &= ~SGX_EPC_PAGE_RECLAIMER_TRACKED; - section = &sgx_epc_sections[epc_page->section]; - spin_lock(§ion->lock); - list_add_tail(&epc_page->list, §ion->page_list); - section->free_cnt++; - spin_unlock(§ion->lock); + sgx_free_epc_page(epc_page); } } From patchwork Sat Mar 13 16:01:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jarkko Sakkinen X-Patchwork-Id: 12136853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A1F4C433DB for ; Sat, 13 Mar 2021 16:02:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E4D5D64F1A for ; Sat, 13 Mar 2021 16:02:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234010AbhCMQCD (ORCPT ); Sat, 13 Mar 2021 11:02:03 -0500 Received: from mail.kernel.org ([198.145.29.99]:48106 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233877AbhCMQBx (ORCPT ); Sat, 13 Mar 2021 11:01:53 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id 3BD1764F1F; Sat, 13 Mar 2021 16:01:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1615651312; bh=wmiEWoBzQNM3kPcwlo3+3IWPepCWiqpC8Zche8xBDYY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IM6sBEa8fGiwbXLenDS/vV4tPQvACmAhui6k27fpOxzCFX+gWm++txshNnyhPcbtK TICkUL39V6AJpSW9TYyO8wbCpVuGDKzA19vz6qSBelNmDCUSgh3C5CeywjUOAxWy+D XDSvugAm+OUehdlXbJOMuidYSfN3NxkfZjJuL93JJPXu7R3eMyX5qVbfH+4ro/PrmW oqB3815FNbu2txcdAhTlIv0iv2Jis5+6r6FUcuLgklflOS0VH94CRubQvaYFAfFzrJ 7Xm6QKhSOxmLewWU/L+VbtNvIfIdMcKQcb3y4KLAdfdX8qrEJS+cMc9OC046lgk9oc QfD/h91OdJpwQ== From: Jarkko Sakkinen To: linux-sgx@vger.kernel.org Cc: Jarkko Sakkinen , Dave Hansen , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org Subject: [PATCH v4 2/3] x86/sgx: Replace section local dirty page lists with a global list Date: Sat, 13 Mar 2021 18:01:18 +0200 Message-Id: <20210313160119.1318533-3-jarkko@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210313160119.1318533-1-jarkko@kernel.org> References: <20210313160119.1318533-1-jarkko@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org Reset initialized EPC pages in sgx_dirty_page_list to uninitialized state, and free them using sgx_free_epc_page(). Do two passes, as for SECS pages the first round can fail, if all child pages have not yet been removed. The driver puts all pages on startup first to sgx_dirty_page_list, as the initialization could be triggered by kexec(), meaning that pages have been reserved for active enclaves before the operation. The section local lists are redundant, as sgx_free_epc_page() figures out the correction by using epc_page->section. Signed-off-by: Jarkko Sakkinen --- v4: * Open coded sgx_santize_section() to ksgxd(). * Rewrote the commit message. arch/x86/kernel/cpu/sgx/main.c | 81 ++++++++++++++++------------------ arch/x86/kernel/cpu/sgx/sgx.h | 7 --- 2 files changed, 37 insertions(+), 51 deletions(-) diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index 65004fb8a91f..cb4561444b96 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -27,39 +27,10 @@ static LIST_HEAD(sgx_active_page_list); static DEFINE_SPINLOCK(sgx_reclaimer_lock); /* - * Reset dirty EPC pages to uninitialized state. Laundry can be left with SECS - * pages whose child pages blocked EREMOVE. + * When the driver initialized, EPC pages go first here, as they could be + * initialized to an active enclave, on kexec entry. */ -static void sgx_sanitize_section(struct sgx_epc_section *section) -{ - struct sgx_epc_page *page; - LIST_HEAD(dirty); - int ret; - - /* init_laundry_list is thread-local, no need for a lock: */ - while (!list_empty(§ion->init_laundry_list)) { - if (kthread_should_stop()) - return; - - /* needed for access to ->page_list: */ - spin_lock(§ion->lock); - - page = list_first_entry(§ion->init_laundry_list, - struct sgx_epc_page, list); - - ret = __eremove(sgx_get_epc_virt_addr(page)); - if (!ret) - list_move(&page->list, §ion->page_list); - else - list_move_tail(&page->list, &dirty); - - spin_unlock(§ion->lock); - - cond_resched(); - } - - list_splice(&dirty, §ion->init_laundry_list); -} +static LIST_HEAD(sgx_dirty_page_list); static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page) { @@ -400,25 +371,48 @@ static bool sgx_should_reclaim(unsigned long watermark) static int ksgxd(void *p) { - int i; + struct sgx_epc_page *page; + LIST_HEAD(dirty); + int i, ret; set_freezable(); /* - * Sanitize pages in order to recover from kexec(). The 2nd pass is - * required for SECS pages, whose child pages blocked EREMOVE. + * Reset initialized EPC pages in sgx_dirty_page_list to uninitialized state, + * and free them using sgx_free_epc_page(). Do two passes, as for SECS pages the + * first round can fail, if all child pages have not yet been removed. The + * driver puts all pages on startup first to sgx_dirty_page_list, as the + * initialization could be triggered by kexec(), meaning that pages have been + * reserved for active enclaves before the operation. */ - for (i = 0; i < sgx_nr_epc_sections; i++) - sgx_sanitize_section(&sgx_epc_sections[i]); - for (i = 0; i < sgx_nr_epc_sections; i++) { - sgx_sanitize_section(&sgx_epc_sections[i]); + /* sgx_dirty_page_list is thread-local to ksgxd, no need for a lock: */ + for (i = 0; i < 2 && !list_empty(&sgx_dirty_page_list); i++) { + while (!list_empty(&sgx_dirty_page_list)) { + if (kthread_should_stop()) + return 0; + + page = list_first_entry(&sgx_dirty_page_list, struct sgx_epc_page, list); + + ret = __eremove(sgx_get_epc_virt_addr(page)); + if (!ret) { + /* The page is clean - move to the free list. */ + list_del(&page->list); + sgx_free_epc_page(page); + } else { + /* The page is not yet clean - move to the dirty list. */ + list_move_tail(&page->list, &dirty); + } + + cond_resched(); + } - /* Should never happen. */ - if (!list_empty(&sgx_epc_sections[i].init_laundry_list)) - WARN(1, "EPC section %d has unsanitized pages.\n", i); + list_splice(&dirty, &sgx_dirty_page_list); } + if (!list_empty(&sgx_dirty_page_list)) + WARN(1, "EPC section %d has unsanitized pages.\n", i); + while (!kthread_should_stop()) { if (try_to_freeze()) continue; @@ -632,13 +626,12 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size, section->phys_addr = phys_addr; spin_lock_init(§ion->lock); INIT_LIST_HEAD(§ion->page_list); - INIT_LIST_HEAD(§ion->init_laundry_list); for (i = 0; i < nr_pages; i++) { section->pages[i].section = index; section->pages[i].flags = 0; section->pages[i].owner = NULL; - list_add_tail(§ion->pages[i].list, §ion->init_laundry_list); + list_add_tail(§ion->pages[i].list, &sgx_dirty_page_list); } section->free_cnt = nr_pages; diff --git a/arch/x86/kernel/cpu/sgx/sgx.h b/arch/x86/kernel/cpu/sgx/sgx.h index 5fa42d143feb..bc8af0428640 100644 --- a/arch/x86/kernel/cpu/sgx/sgx.h +++ b/arch/x86/kernel/cpu/sgx/sgx.h @@ -45,13 +45,6 @@ struct sgx_epc_section { spinlock_t lock; struct list_head page_list; unsigned long free_cnt; - - /* - * Pages which need EREMOVE run on them before they can be - * used. Only safe to be accessed in ksgxd and init code. - * Not protected by locks. - */ - struct list_head init_laundry_list; }; extern struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS]; From patchwork Sat Mar 13 16:01:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jarkko Sakkinen X-Patchwork-Id: 12136857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6A01C4332B for ; Sat, 13 Mar 2021 16:02:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7050464F34 for ; Sat, 13 Mar 2021 16:02:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234019AbhCMQCD (ORCPT ); Sat, 13 Mar 2021 11:02:03 -0500 Received: from mail.kernel.org ([198.145.29.99]:48148 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233897AbhCMQBz (ORCPT ); Sat, 13 Mar 2021 11:01:55 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id E574564F19; Sat, 13 Mar 2021 16:01:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1615651315; bh=NXcloEw+fPmq0b4PZGMAPEIibJJ3oamoZi+PpoH/018=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jF416DJX4WX0it7X4reIWa64jedgrACMCs9PvQgWy+IZQBp3HHvqiy6LH3N69A1+y uct/po3L307obZvj63+6lTNAhOZ0eZflNoy7VSNCeGQXvYctEBljEpv2GtPPSfQCI1 Zt356KQdpm7Om1KlIqP4niOU8GKoUo7yV8m0eomQwCK7jXZ1BuW2heEBZhm9FVtNwf rQRS8R4RC6nuPlpC953y0I8XfwEqNNflK+ohsXQYvJtqGE6o56KoXB6RmnwHHqMxIU 8yqGdxvt3maNkMLmXjUp24bo/sRKnduAd7MO3kt+eLfdhBYg2hYAsCQwGTcoVnNDyt qvvpI7LU5B4PA== From: Jarkko Sakkinen To: linux-sgx@vger.kernel.org Cc: Jarkko Sakkinen , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Dave Hansen , linux-kernel@vger.kernel.org Subject: [PATCH v4 3/3] x86/sgx: Add a basic NUMA allocation scheme to sgx_alloc_epc_page() Date: Sat, 13 Mar 2021 18:01:19 +0200 Message-Id: <20210313160119.1318533-4-jarkko@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210313160119.1318533-1-jarkko@kernel.org> References: <20210313160119.1318533-1-jarkko@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org Background ========== EPC section is covered by one or more SRAT entries that are associated with one and only one PXM (NUMA node). The motivation behind this patch is to provide basic elements of building allocation scheme based on this premise. Just like normal RAM, enclave memory (EPC) should be covered by entries in the ACPI SRAT table. These entries allow each EPC section to be associated with a NUMA node. Use this information to implement a simple NUMA-aware allocator for enclave memory. Solution ======== Use phys_to_target_node() to associate each NUMA node with the EPC sections contained within its range. In sgx_alloc_epc_page(), first try to allocate from the NUMA node, where the CPU is executing. If that fails, allocate from other nodes, iterating them from the current node in order. Other ===== NUMA_KEEP_MEMINFO dependency is required for phys_to_target_node(). Link: https://lore.kernel.org/lkml/158188326978.894464.217282995221175417.stgit@dwillia2-desk3.amr.corp.intel.com/ Signed-off-by: Jarkko Sakkinen --- v4: * Cycle nodes instead of a global page list, starting from the node of the current thread. * Documented NUMA_KEEP_MEMINFO dependency to the commit message. * Added NUMA node pointer to struct sgx_epc_section. EPC page should reference to a section, since potentially a node could have multiple sections (Intel SDM does not say anything explicit about this). This the safest play. * Remove nodes_clear(sgx_numa_node_mask). * Appended Dave's additions to the commit message for the background section. arch/x86/Kconfig | 1 + arch/x86/kernel/cpu/sgx/main.c | 117 ++++++++++++++++++++------------- arch/x86/kernel/cpu/sgx/sgx.h | 16 +++-- 3 files changed, 84 insertions(+), 50 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 513895af8ee7..3e6152a8dd2b 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1930,6 +1930,7 @@ config X86_SGX depends on CRYPTO_SHA256=y select SRCU select MMU_NOTIFIER + select NUMA_KEEP_MEMINFO if NUMA help Intel(R) Software Guard eXtensions (SGX) is a set of CPU instructions that can be used by applications to set aside private regions of code diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index cb4561444b96..3b524a1361d6 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -18,14 +18,23 @@ static int sgx_nr_epc_sections; static struct task_struct *ksgxd_tsk; static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq); -/* - * These variables are part of the state of the reclaimer, and must be accessed - * with sgx_reclaimer_lock acquired. - */ +/* The reclaimer lock protected variables prepend the lock. */ static LIST_HEAD(sgx_active_page_list); - static DEFINE_SPINLOCK(sgx_reclaimer_lock); +/* The free page list lock protected variables prepend the lock. */ +static unsigned long sgx_nr_free_pages; + +/* Nodes with one or more EPC sections. */ +static nodemask_t sgx_numa_mask; + +/* + * Array with one list_head for each possible NUMA node. Each + * list contains all the sgx_epc_section's which are on that + * node. + */ +static struct sgx_numa_node *sgx_numa_nodes; + /* * When the driver initialized, EPC pages go first here, as they could be * initialized to an active enclave, on kexec entry. @@ -352,21 +361,9 @@ static void sgx_reclaim_pages(void) } } -static unsigned long sgx_nr_free_pages(void) -{ - unsigned long cnt = 0; - int i; - - for (i = 0; i < sgx_nr_epc_sections; i++) - cnt += sgx_epc_sections[i].free_cnt; - - return cnt; -} - static bool sgx_should_reclaim(unsigned long watermark) { - return sgx_nr_free_pages() < watermark && - !list_empty(&sgx_active_page_list); + return sgx_nr_free_pages < watermark && !list_empty(&sgx_active_page_list); } static int ksgxd(void *p) @@ -443,50 +440,63 @@ static bool __init sgx_page_reclaimer_init(void) return true; } -static struct sgx_epc_page *__sgx_alloc_epc_page_from_section(struct sgx_epc_section *section) +static struct sgx_epc_page *__sgx_alloc_epc_page_from_node(int nid) { - struct sgx_epc_page *page; + struct sgx_numa_node *node = &sgx_numa_nodes[nid]; + struct sgx_epc_page *page = NULL; + + if (!node_isset(nid, sgx_numa_mask)) + return NULL; - spin_lock(§ion->lock); + spin_lock(&node->lock); - if (list_empty(§ion->page_list)) { - spin_unlock(§ion->lock); + if (list_empty(&node->free_page_list)) { + spin_unlock(&node->lock); return NULL; } - page = list_first_entry(§ion->page_list, struct sgx_epc_page, list); + page = list_first_entry(&node->free_page_list, struct sgx_epc_page, list); list_del_init(&page->list); - section->free_cnt--; + sgx_nr_free_pages--; + + spin_unlock(&node->lock); - spin_unlock(§ion->lock); return page; } /** * __sgx_alloc_epc_page() - Allocate an EPC page * - * Iterate through EPC sections and borrow a free EPC page to the caller. When a - * page is no longer needed it must be released with sgx_free_epc_page(). + * Iterate through NUMA nodes and borrow a free EPC page to the caller. When a + * page is no longer needed it must be released with sgx_free_epc_page(). Start + * from the NUMA node, where the caller is executing. * * Return: - * an EPC page, - * -errno on error + * - an EPC page: Free EPC pages were available. + * - ERR_PTR(-ENOMEM): Run out of EPC pages. */ struct sgx_epc_page *__sgx_alloc_epc_page(void) { - struct sgx_epc_section *section; struct sgx_epc_page *page; - int i; + int nid = numa_node_id(); - for (i = 0; i < sgx_nr_epc_sections; i++) { - section = &sgx_epc_sections[i]; + /* Try to allocate EPC from the current node, first: */ + page = __sgx_alloc_epc_page_from_node(nid); + if (page) + return page; - page = __sgx_alloc_epc_page_from_section(section); + /* Then, go through the other nodes: */ + while (true) { + nid = next_node_in(nid, sgx_numa_mask); + if (nid == numa_node_id()) + break; + + page = __sgx_alloc_epc_page_from_node(nid); if (page) - return page; + break; } - return ERR_PTR(-ENOMEM); + return page; } /** @@ -592,6 +602,7 @@ struct sgx_epc_page *sgx_alloc_epc_page(void *owner, bool reclaim) void sgx_free_epc_page(struct sgx_epc_page *page) { struct sgx_epc_section *section = &sgx_epc_sections[page->section]; + struct sgx_numa_node *node = section->node; int ret; WARN_ON_ONCE(page->flags & SGX_EPC_PAGE_RECLAIMER_TRACKED); @@ -600,10 +611,12 @@ void sgx_free_epc_page(struct sgx_epc_page *page) if (WARN_ONCE(ret, "EREMOVE returned %d (0x%x)", ret, ret)) return; - spin_lock(§ion->lock); - list_add_tail(&page->list, §ion->page_list); - section->free_cnt++; - spin_unlock(§ion->lock); + spin_lock(&node->lock); + + list_add_tail(&page->list, &node->free_page_list); + sgx_nr_free_pages++; + + spin_unlock(&node->lock); } static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size, @@ -624,8 +637,6 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size, } section->phys_addr = phys_addr; - spin_lock_init(§ion->lock); - INIT_LIST_HEAD(§ion->page_list); for (i = 0; i < nr_pages; i++) { section->pages[i].section = index; @@ -634,7 +645,7 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size, list_add_tail(§ion->pages[i].list, &sgx_dirty_page_list); } - section->free_cnt = nr_pages; + sgx_nr_free_pages += nr_pages; return true; } @@ -653,8 +664,11 @@ static bool __init sgx_page_cache_init(void) { u32 eax, ebx, ecx, edx, type; u64 pa, size; + int nid; int i; + sgx_numa_nodes = kmalloc_array(num_possible_nodes(), sizeof(*sgx_numa_nodes), GFP_KERNEL); + for (i = 0; i < ARRAY_SIZE(sgx_epc_sections); i++) { cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC, &eax, &ebx, &ecx, &edx); @@ -677,6 +691,21 @@ static bool __init sgx_page_cache_init(void) break; } + nid = numa_map_to_online_node(phys_to_target_node(pa)); + if (nid == NUMA_NO_NODE) { + /* The physical address is already printed above. */ + pr_warn(FW_BUG "Unable to map EPC section to online node. Fallback to the NUMA node 0.\n"); + nid = 0; + } + + if (!node_isset(nid, sgx_numa_mask)) { + spin_lock_init(&sgx_numa_nodes[nid].lock); + INIT_LIST_HEAD(&sgx_numa_nodes[nid].free_page_list); + node_set(nid, sgx_numa_mask); + } + + sgx_epc_sections[i].node = &sgx_numa_nodes[nid]; + sgx_nr_epc_sections++; } diff --git a/arch/x86/kernel/cpu/sgx/sgx.h b/arch/x86/kernel/cpu/sgx/sgx.h index bc8af0428640..653af8ca1a25 100644 --- a/arch/x86/kernel/cpu/sgx/sgx.h +++ b/arch/x86/kernel/cpu/sgx/sgx.h @@ -29,22 +29,26 @@ struct sgx_epc_page { struct list_head list; }; +/* + * Contains the tracking data for NUMA nodes having EPC pages. Most importantly, + * the free page list local to the node is stored here. + */ +struct sgx_numa_node { + struct list_head free_page_list; + spinlock_t lock; +}; + /* * The firmware can define multiple chunks of EPC to the different areas of the * physical memory e.g. for memory areas of the each node. This structure is * used to store EPC pages for one EPC section and virtual memory area where * the pages have been mapped. - * - * 'lock' must be held before accessing 'page_list' or 'free_cnt'. */ struct sgx_epc_section { unsigned long phys_addr; void *virt_addr; struct sgx_epc_page *pages; - - spinlock_t lock; - struct list_head page_list; - unsigned long free_cnt; + struct sgx_numa_node *node; }; extern struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS];