From patchwork Mon Oct 3 22:04:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Kai" X-Patchwork-Id: 12997818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8DA0C433FE for ; Mon, 3 Oct 2022 22:04:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229979AbiJCWEo (ORCPT ); Mon, 3 Oct 2022 18:04:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229719AbiJCWEm (ORCPT ); Mon, 3 Oct 2022 18:04:42 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F02FC5F89; Mon, 3 Oct 2022 15:04:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1664834681; x=1696370681; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=b2TVdQR0wcN0WlAJV+kk7tAVgQKAON4eNa7JV7pnZQ0=; b=GRgAwq75bhTOSDcMl76lSGcDRcEdpS7yUoLe80owFuZ+KlALvXs08FvH tRMzQpqWsCu8q7Igm6xvVOYuFNBwTWPUihOIqOFmW+zp/1ka2CZCml+c/ jTvayAQV4Z+zH3wcudIANez9NPseIt4hq/oE8iOhn7WrxmatUs5i9GI7s pasOl6140mqwh70BRaPejPMEhmIOKxRhaGWmGD3V2lVqB1FJVPXn61O+p PBeIrs6bqWynzBz61Bjp4cnPN2Xk+Sxn1ZPyrrSka3foamUJTDfv8ccJ9 ne4tEvzwL4z3ofgOblvV/4aQjjD/N3ZBYVIilCzv+l+KdSjDMp6ny1RIY g==; X-IronPort-AV: E=McAfee;i="6500,9779,10489"; a="366869169" X-IronPort-AV: E=Sophos;i="5.93,366,1654585200"; d="scan'208";a="366869169" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Oct 2022 15:04:40 -0700 X-IronPort-AV: E=McAfee;i="6500,9779,10489"; a="686310953" X-IronPort-AV: E=Sophos;i="5.93,366,1654585200"; d="scan'208";a="686310953" Received: from jparcemo-mobl.amr.corp.intel.com (HELO khuang2-desk.gar.corp.intel.com) ([10.212.93.75]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Oct 2022 15:04:39 -0700 From: Kai Huang To: linux-sgx@vger.kernel.org Cc: dave.hansen@linux.intel.com, jarkko@kernel.org, tony.luck@intel.com, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] x86/sgx: Start the ksgxd() at the end of sgx_init() Date: Tue, 4 Oct 2022 11:04:27 +1300 Message-Id: <94169ef79db0695d7b42d2a78565e8dceb64cf3e.1664834225.git.kai.huang@intel.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org The ksgxd() kernel thread basically does two things: 1) sanitize all EPC pages; 2) start the page reclaimer. Currently it is created and started before initializing both the native SGX driver and the KVM driver, but there's no reason to do that. It only needs to be started when at least one of the native and the KVM driver has been initialized. Move creating and running the ksgxd() to the end of sgx_init() after at least one of the native and the KVM driver has been initialized. Also, when kernel fails to create the ksgxd(), opportunistically improve the behaviour to not disable SGX completely, but to continue to sanitize EPC pages and run w/o reclaimer. This allows SGX to continue to work when kernel is not running out of EPC (this is especially reasonable for KVM virtual EPC driver as virtual EPC pages cannot be reclaimed anyway). With above change, just remove the sgx_page_reclaimer_init() and open code its logic at the end of sgx_init() as this way is more clear. Signed-off-by: Kai Huang --- arch/x86/kernel/cpu/sgx/main.c | 41 ++++++++++++++++------------------ 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index 0aad028f04d4..713ca09f6d6e 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -420,19 +420,6 @@ static int ksgxd(void *p) return 0; } -static bool __init sgx_page_reclaimer_init(void) -{ - struct task_struct *tsk; - - tsk = kthread_run(ksgxd, NULL, "ksgxd"); - if (IS_ERR(tsk)) - return false; - - ksgxd_tsk = tsk; - - return true; -} - bool current_is_ksgxd(void) { return current == ksgxd_tsk; @@ -921,14 +908,9 @@ static int __init sgx_init(void) if (!sgx_page_cache_init()) return -ENOMEM; - if (!sgx_page_reclaimer_init()) { - ret = -ENOMEM; - goto err_page_cache; - } - ret = misc_register(&sgx_dev_provision); if (ret) - goto err_kthread; + goto err_page_cache; /* * Always try to initialize the native *and* KVM drivers. @@ -943,14 +925,29 @@ static int __init sgx_init(void) if (sgx_vepc_init() && ret) goto err_provision; + /* + * At least one of the native and the KVM driver has been + * initialized. Start the ksgxd(). + */ + ksgxd_tsk = kthread_run(ksgxd, NULL, "ksgxd"); + + /* + * If unable to create the ksgxd() thread, don't disable + * SGX completely. Instead, continue to sanitize all EPC + * pages and run w/o reclaimer. + */ + if (IS_ERR(ksgxd_tsk)) { + ksgxd_tsk = NULL; + __sgx_sanitize_pages(&sgx_dirty_page_list); + WARN_ON(__sgx_sanitize_pages(&sgx_dirty_page_list)); + pr_info("Running SGX w/o EPC page reclaimer.\n"); + } + return 0; err_provision: misc_deregister(&sgx_dev_provision); -err_kthread: - kthread_stop(ksgxd_tsk); - err_page_cache: for (i = 0; i < sgx_nr_epc_sections; i++) { vfree(sgx_epc_sections[i].pages); From patchwork Mon Oct 3 22:04:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Kai" X-Patchwork-Id: 12997820 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 317B9C433FE for ; Mon, 3 Oct 2022 22:04:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230002AbiJCWEu (ORCPT ); Mon, 3 Oct 2022 18:04:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229998AbiJCWEq (ORCPT ); Mon, 3 Oct 2022 18:04:46 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40B7763F4; Mon, 3 Oct 2022 15:04:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1664834682; x=1696370682; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6WwR4lJkD9myqJbkioJF7WxCYyfmYMBhp6kUNpLwcZg=; b=Z1OmKhcd1YBJzWwuALgAHsxI2ArVx57u6ltPX50bpGPtI4jCZV+vzlQd aTX+13GfUejHloIJmDEqo/dE4/er1QMDnIE7hzOWP3hy9BJIgBPfSGOEU KTluyq9q7uJxsQ6MVzrctlpwn8rLm8wDEkgOUsUbQxcogeT+1pZc+SAmf 5PS9V8bx3rHOFVMky1Bbpt213Y5KVz6IZYBHibB0+/zu5og611DvZYuZx WCg26X7C0n9Sk1D2/NeWeZOZoAkERIOOyLUVoiZLxKAE+ghDqmjsXv2Pd Dm/3uBli/e7iGtaEo9IAEihZ96YskzFJK1y5EL0JM6TPWAHXmAW/GSHS3 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10489"; a="366869173" X-IronPort-AV: E=Sophos;i="5.93,366,1654585200"; d="scan'208";a="366869173" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Oct 2022 15:04:42 -0700 X-IronPort-AV: E=McAfee;i="6500,9779,10489"; a="686310962" X-IronPort-AV: E=Sophos;i="5.93,366,1654585200"; d="scan'208";a="686310962" Received: from jparcemo-mobl.amr.corp.intel.com (HELO khuang2-desk.gar.corp.intel.com) ([10.212.93.75]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Oct 2022 15:04:40 -0700 From: Kai Huang To: linux-sgx@vger.kernel.org Cc: dave.hansen@linux.intel.com, jarkko@kernel.org, tony.luck@intel.com, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] x86/sgx: Only run the reclaimer when the native SGX driver is enabled Date: Tue, 4 Oct 2022 11:04:28 +1300 Message-Id: <0dd24e5549bbdee8580c2495f1df93798aadfd11.1664834225.git.kai.huang@intel.com> X-Mailer: git-send-email 2.37.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org Currently the EPC pages assigned to KVM guests cannot be reclaimed, so there's no point to run the reclaimer when the native SGX driver is not enabled. Add a function to indicate whether the native SGX driver has been initialized, and in ksgxd(), avoid running the reclaimer when it is false. In sgx_drv_init(), move the register of "/dev/sgx_enclave" misc device before initializing sgx_attributes_reserved_mask (and the other two masks) so that the new function can just use it to determine whether the SGX driver has been initialized w/o introducing a new boolean. Signed-off-by: Kai Huang --- arch/x86/kernel/cpu/sgx/driver.c | 13 +++++++++---- arch/x86/kernel/cpu/sgx/driver.h | 1 + arch/x86/kernel/cpu/sgx/main.c | 11 ++++++++++- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/cpu/sgx/driver.c b/arch/x86/kernel/cpu/sgx/driver.c index aa9b8b868867..b5e754632eed 100644 --- a/arch/x86/kernel/cpu/sgx/driver.c +++ b/arch/x86/kernel/cpu/sgx/driver.c @@ -160,6 +160,10 @@ int __init sgx_drv_init(void) return -ENODEV; } + ret = misc_register(&sgx_dev_enclave); + if (ret) + return ret; + sgx_misc_reserved_mask = ~ebx | SGX_MISC_RESERVED_MASK; cpuid_count(SGX_CPUID, 1, &eax, &ebx, &ecx, &edx); @@ -172,9 +176,10 @@ int __init sgx_drv_init(void) sgx_xfrm_reserved_mask = ~xfrm_mask; } - ret = misc_register(&sgx_dev_enclave); - if (ret) - return ret; - return 0; } + +bool sgx_drv_inited(void) +{ + return !!sgx_attributes_reserved_mask; +} diff --git a/arch/x86/kernel/cpu/sgx/driver.h b/arch/x86/kernel/cpu/sgx/driver.h index 4eddb4d571ef..159cc066e9cb 100644 --- a/arch/x86/kernel/cpu/sgx/driver.h +++ b/arch/x86/kernel/cpu/sgx/driver.h @@ -25,5 +25,6 @@ extern const struct file_operations sgx_provision_fops; long sgx_ioctl(struct file *filep, unsigned int cmd, unsigned long arg); int sgx_drv_init(void); +bool sgx_drv_inited(void); #endif /* __ARCH_X86_SGX_DRIVER_H__ */ diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index 713ca09f6d6e..0fdbc490b0f8 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -403,6 +403,14 @@ static int ksgxd(void *p) __sgx_sanitize_pages(&sgx_dirty_page_list); WARN_ON(__sgx_sanitize_pages(&sgx_dirty_page_list)); + /* + * EPC pages assigned to KVM guests cannot be reclaimed. There's + * no reason to run the reclaimer if the native SGX driver isn't + * initialized successfully (i.e. on the machines w/o SGX_LC). + */ + if (!sgx_drv_inited()) + return 0; + while (!kthread_should_stop()) { if (try_to_freeze()) continue; @@ -940,7 +948,8 @@ static int __init sgx_init(void) ksgxd_tsk = NULL; __sgx_sanitize_pages(&sgx_dirty_page_list); WARN_ON(__sgx_sanitize_pages(&sgx_dirty_page_list)); - pr_info("Running SGX w/o EPC page reclaimer.\n"); + if (sgx_drv_inited()) + pr_info("Running native SGX driver w/o EPC page reclaimer.\n"); } return 0; From patchwork Mon Oct 3 22:04:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Kai" X-Patchwork-Id: 12997819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 238D6C433F5 for ; Mon, 3 Oct 2022 22:04:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229698AbiJCWEs (ORCPT ); Mon, 3 Oct 2022 18:04:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229999AbiJCWEq (ORCPT ); Mon, 3 Oct 2022 18:04:46 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B27767652; Mon, 3 Oct 2022 15:04:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1664834683; x=1696370683; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TX27OPhzU6FkwQBBFhssDYk2QfF6ahRh6VMWmC5I3g4=; b=BPIippcQ4Y6V68R86uogfipByZScpcw5i5/HfzS/M0c5d5PzLfjtc1v9 E9VLvcYcUXRgjFlip07vXGWURz67ZZsNSiEj2qtSLjWD3FpJq6xK1MAGU 4rJMo+DKBlTCNioPWvjm0OLyC3gq4Z4gwg7GbbgIhrguO+WbtiG4OtNXi tLh1dPd02iCJKztA2r6yJzMTEeabsSkl0HQUIWmJJ4YD3uTxnpRahZW1T GHeTmEIsSIL4agl3ZT+w+lJORu9+zgaIrlgO/yTNLlVnHOaLUTHTk4yEy mB5OrPnfcyjLolGArfssngwMbT04Pxwo2A3Jsb8XkULHuGWH6UuSMavkx g==; X-IronPort-AV: E=McAfee;i="6500,9779,10489"; a="366869180" X-IronPort-AV: E=Sophos;i="5.93,366,1654585200"; d="scan'208";a="366869180" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Oct 2022 15:04:43 -0700 X-IronPort-AV: E=McAfee;i="6500,9779,10489"; a="686310975" X-IronPort-AV: E=Sophos;i="5.93,366,1654585200"; d="scan'208";a="686310975" Received: from jparcemo-mobl.amr.corp.intel.com (HELO khuang2-desk.gar.corp.intel.com) ([10.212.93.75]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Oct 2022 15:04:42 -0700 From: Kai Huang To: linux-sgx@vger.kernel.org Cc: dave.hansen@linux.intel.com, jarkko@kernel.org, tony.luck@intel.com, linux-kernel@vger.kernel.org Subject: [RESEND PATCH 3/3] x86/sgx: Add xa_store_range() return value check in sgx_setup_epc_section() Date: Tue, 4 Oct 2022 11:04:29 +1300 Message-Id: X-Mailer: git-send-email 2.37.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org In sgx_setup_epc_section(), xa_store_range() is called to store EPC pages' owner section to an Xarray using physical addresses of those EPC pages as index. Currently, the return value of xa_store_range() is not checked, but actually it can fail (i.e. due to -ENOMEM). Not checking the return value of xa_store_range() would result in the EPC section being used by SGX driver (and KVM SGX guests), but part or all of its EPC pages not being handled by the memory failure handling of EPC page. Such inconsistency should be avoided, even at the cost that this section won't be used by the kernel. Add the missing check of the return value of xa_store_range(), and when it fails, clean up and fail to initialize the EPC section. Fixes: 40e0e7843e23 ("x86/sgx: Add infrastructure to identify SGX EPC pages") Signed-off-by: Kai Huang Reviewed-by: Jarkko Sakkinen --- arch/x86/kernel/cpu/sgx/main.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index 0fdbc490b0f8..5ddf9d9296f4 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -630,8 +630,12 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size, } section->phys_addr = phys_addr; - xa_store_range(&sgx_epc_address_space, section->phys_addr, - phys_addr + size - 1, section, GFP_KERNEL); + if (xa_err(xa_store_range(&sgx_epc_address_space, section->phys_addr, + phys_addr + size - 1, section, GFP_KERNEL))) { + vfree(section->pages); + memunmap(section->virt_addr); + return false; + } for (i = 0; i < nr_pages; i++) { section->pages[i].section = index;