From patchwork Thu Jan 19 21:22:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13108789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DBE5C6379F for ; Thu, 19 Jan 2023 21:23:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2F1F6B008C; Thu, 19 Jan 2023 16:23:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D8FAB6B0092; Thu, 19 Jan 2023 16:23:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAE838E0001; Thu, 19 Jan 2023 16:23:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7CCE96B008C for ; Thu, 19 Jan 2023 16:23:45 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 48634A0275 for ; Thu, 19 Jan 2023 21:23:45 +0000 (UTC) X-FDA: 80372825610.01.C808D22 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf27.hostedemail.com (Postfix) with ESMTP id 5102A4000F for ; Thu, 19 Jan 2023 21:23:43 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Bz10HKRK; spf=pass (imf27.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674163423; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references:dkim-signature; bh=raQIy6Fe766NUbCocQ4XfRzcFOdeBSgY+sYH+kVPZqU=; b=FYntVvP38uh1+0s2VBA7bn/5Vi1O2gh+FBh0UPymBlE+0x85H49MCOyp2ntcO1Zk27psSM q7/XSoYLk8NUOtuceopCYOgacocsWapOTwYTk7FM/r/6JFtrOIVfZ/OI4txstzpnv0VtYU wh3tjFs793/YyWA4Iqc59QAdQzTxyAQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Bz10HKRK; spf=pass (imf27.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674163423; a=rsa-sha256; cv=none; b=pp5vTAncP9tKPFriu3yVYAyGQrGFTcaF9Q6DeTpBXtgUiwqdrTz6ZWuXSjVMAlfssaP7T6 NX7na3nFTE+2uUCZLGwcF1DfhY1gGPxjl0EA0OeEuYyLzNr5mbal7cQj4mC7p5ozgwxg0k dRAF8MYIKQ3o29FibCyw7ESa5kkK2kk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1674163423; x=1705699423; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=0xRZR4lTYh6NOOM79Ggd51EdLXNXPAUjGASZ+kAeGKM=; b=Bz10HKRK3VK0rFUJYz0cfitDqNY8XsMYtlnr3IlhCQiHMlBz9p8DgWRT al4Dmie3KrJRd5/tHAeG9ixgj8qHd501GFnW8Iz1Qql56pdzb6rj823do xKOiw+fFojXy8iBSOT88WBjcP/sfwpC+Dz15yGccazR0JgbgGjW+e1oEj VVB3ZWjacorrawVHVl4MB5R70yix7A+jfsjzQxmasgWlqtAB+N0+StKdF KzxDDVC8vS0aSPWKFkig90fOfLiBSU5s2zmheJ/nhQYaTzNuGI53SNuoX b+Mnx06i4bK1AMHW7uifyFPjkNUrgZ5BffaNXu+YUnqFnZtw78kV3jAeS g==; X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="323119413" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="323119413" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:42 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10595"; a="989139038" X-IronPort-AV: E=Sophos;i="5.97,230,1669104000"; d="scan'208";a="989139038" Received: from hossain3-mobl.amr.corp.intel.com (HELO rpedgeco-desk.amr.corp.intel.com) ([10.252.128.187]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2023 13:23:41 -0800 From: Rick Edgecombe To: x86@kernel.org, "H . Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H . J . Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , Weijiang Yang , "Kirill A . Shutemov" , John Allen , kcc@google.com, eranian@google.com, rppt@kernel.org, jamorris@linux.microsoft.com, dethoma@microsoft.com, akpm@linux-foundation.org, Andrew.Cooper3@citrix.com, christina.schimpe@intel.com Cc: rick.p.edgecombe@intel.com, Yu-cheng Yu Subject: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW Date: Thu, 19 Jan 2023 13:22:49 -0800 Message-Id: <20230119212317.8324-12-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230119212317.8324-1-rick.p.edgecombe@intel.com> References: <20230119212317.8324-1-rick.p.edgecombe@intel.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5102A4000F X-Stat-Signature: n15dc68h5soxty3hkzs18eqzh6bcuru8 X-Rspam-User: X-HE-Tag: 1674163423-476256 X-HE-Meta: U2FsdGVkX19AX/OBOwRkgh6bhFJMBRaSkOLyhcvY1S/eAIqycPPf38JfDiWUIWy0fLFIuOcpCVAvITwjJVNhW96Tu0wec+NAivY0w1is5roPo6qW6GOTzuAw37kfaRXgWK1iUMbSN9ijTvA0UeLEF91HzNnP9uWTwK1q67ovBg3JBbno2MP7HjHDhZnoO7cRHDcqCgscWymWEoqHu0vOd6WjguwceYPNYLCLFum0UMonzL6PiiJfLBIHzYA5zSGnohZJUFxzMoYKvDuw1Xjnqz0uHrepKCMVDts1pRqEYsYXJJ0xh6YcJmhf2HJWJ6orwsKaZdD27V6Ht8flhhNsPX+HdXzqg7hnxeK4oR48sTAvdhSiUUGBUMFU+hOUVZ7UnBD4EKoYfya31j2XElN5nEbzFfQjmDqQG5QU8YCrs1NjURGCoQIBxfHr3Q9m0DpHJlgXPcM6Q/kLo1c3UUYnSbDu+1ApQ2LSYd0fHOCELFYHdP2xyB3qaOt2F9Kx2KkjGhUacrWeZhB9k+n30feB5r/wC7qulux9fdGBZD8Y8SrUj9jDAfXTbtmwTEGDRpMvgFwVS3JrUD5Pg7PVMidm0uY5zW6TdfCuhJUMzxRMM8jEBAGferJVrnS2I4kNKyK2IA4+gwbUS/84y5jXPxUkS7fxe2lRc27KcIL93bSGgdUDiKsHmIbzXbMAPmvMj4yGb1/ShUmBUjiVydma3JTBaI1oAxaZZDjrruuWpDmyTCc6YM1H4PTsSGA0iT6mwnj5UnwICPuczVfy+AW+2+Skqq+5AKHUMYFbstvga3d+H3xkChy5js2jmu+JEiYvioZy8XeyNhP6OJk4ibr6OgXfngDZBDM5fbd92Qsed0mrtNKp8zKOc1MgJv6fSxjo+mh/ui3qAmqS5BWLaGR1vbOgMR5Ttcy5vgJ2BZwDuQTXeehRjtlvjsQVI2tbjV1fCdFV5tLKuN3wRsNfohq7SjY 5YYF44Bg ztR26fQ5QwNUTaJIBv/ly/Tf0OwNX6TLYAGnRGKAEtzr34CksEgO5smJFviheGEB5SttIs71mi42jqwc53MPKAVtaP9QPCNIiF7k5lPr/T9kleWMpmyJpBV5Ops7mpAdDhXnOOxV1QQldQdA7eXfK5drYN7hNdw9gV8lByWhf7QLs1+5XCqRxLxZFxl9dQZ3+TK5cr5u14/LJru80PRjy8iqyjICCWqxPGoC3Mji7yxarlTka+MFBwEeZG0gpFRnhxccFMYzwYN+EZP9YW1AOxrXJKw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yu-cheng Yu The Write=0,Dirty=1 PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a Write=0,Dirty=1 PTE as a shadow stack page. In order to separate the two, the software-defined _PAGE_DIRTY is changed to _PAGE_COW for the copy-on-write case, and pte_*() are updated to do this. pte_modify() takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one. However pte_modify() changes a PTE to 'newprot', but it doesn't use the pte_*(). Modify it to also move _PAGE_DIRTY to _PAGE_COW. Do this by using the pte_mkdirty() helper. Since pte_mkdirty() also sets the soft dirty bit, extract a helper that optionally doesn't set _PAGE_SOFT_DIRTY. This helper will allow future logic for deciding when to move _PAGE_DIRTY to _PAGE_COW can live in one place. Apply the same changes to pmd_modify(). Tested-by: Pengfei Xu Tested-by: John Allen Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Reviewed-by: Kees Cook --- v5: - Fix pte_modify() again, to not lose _PAGE_DIRTY, but still not set _PAGE_SOFT_DIRTY as was fixed in v4. v4: - Fix an issue in soft-dirty test, where pte_modify() would detect _PAGE_COW in pte_dirty() and set the soft dirty bit in pte_mkdirty(). v2: - Update commit log with text and suggestions from (Dave Hansen) - Drop fixup_dirty_pte() in favor of clearing the HW dirty bit along with the _PAGE_CHG_MASK masking, then calling pte_mkdirty() (Dave Hansen) arch/x86/include/asm/pgtable.h | 64 +++++++++++++++++++++++++++++----- 1 file changed, 56 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 6d2f612c04b5..7942eff2af50 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -392,9 +392,19 @@ static inline pte_t pte_mkexec(pte_t pte) return pte_clear_flags(pte, _PAGE_NX); } +static inline pte_t __pte_mkdirty(pte_t pte, bool soft) +{ + pteval_t dirty = _PAGE_DIRTY; + + if (soft) + dirty |= _PAGE_SOFT_DIRTY; + + return pte_set_flags(pte, dirty); +} + static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return __pte_mkdirty(pte, true); } static inline pte_t pte_mkyoung(pte_t pte) @@ -503,9 +513,19 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) return pmd_clear_flags(pmd, _PAGE_RW); } +static inline pmd_t __pmd_mkdirty(pmd_t pmd, bool soft) +{ + pmdval_t dirty = _PAGE_DIRTY; + + if (soft) + dirty |= _PAGE_SOFT_DIRTY; + + return pmd_set_flags(pmd, dirty); +} + static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return __pmd_mkdirty(pmd, true); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -715,26 +735,54 @@ static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) { + pteval_t _page_chg_mask_no_dirty = _PAGE_CHG_MASK & ~_PAGE_DIRTY; pteval_t val = pte_val(pte), oldval = val; + pte_t pte_result; /* * Chop off the NX bit (if present), and add the NX portion of * the newprot (if present): */ - val &= _PAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; + val &= _page_chg_mask_no_dirty; + val |= check_pgprot(newprot) & ~_page_chg_mask_no_dirty; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); - return __pte(val); + + pte_result = __pte(val); + + /* + * Dirty bit is not preserved above so it can be done + * in a special way for the shadow stack case, where it + * may need to set _PAGE_COW. __pte_mkdirty() will do this in + * the case of shadow stack. + */ + if (pte_dirty(pte)) + pte_result = __pte_mkdirty(pte_result, false); + + return pte_result; } static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) { + pteval_t _hpage_chg_mask_no_dirty = _HPAGE_CHG_MASK & ~_PAGE_DIRTY; pmdval_t val = pmd_val(pmd), oldval = val; + pmd_t pmd_result; - val &= _HPAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val &= _hpage_chg_mask_no_dirty; + val |= check_pgprot(newprot) & ~_hpage_chg_mask_no_dirty; val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); + + pmd_result = __pmd(val); + + /* + * Dirty bit is not preserved above so it can be done + * in a special way for the shadow stack case, where it + * may need to set _PAGE_COW. __pmd_mkdirty() will do this in + * the case of shadow stack. + */ + if (pmd_dirty(pmd)) + pmd_result = __pmd_mkdirty(pmd_result, false); + + return pmd_result; } /*