From patchwork Thu Feb 15 19:14:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 13559077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1025C4829E for ; Thu, 15 Feb 2024 19:14:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F31B8D0007; Thu, 15 Feb 2024 14:14:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A2C28D0001; Thu, 15 Feb 2024 14:14:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 444028D0007; Thu, 15 Feb 2024 14:14:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 357138D0001 for ; Thu, 15 Feb 2024 14:14:22 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 06F69404AA for ; Thu, 15 Feb 2024 19:14:22 +0000 (UTC) X-FDA: 81794989164.09.D6704DD Received: from smtpout.efficios.com (smtpout.efficios.com [167.114.26.122]) by imf08.hostedemail.com (Postfix) with ESMTP id 5F72016000C for ; Thu, 15 Feb 2024 19:14:19 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=GTz3kaGN; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf08.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708024459; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=pNhhSdWmeIvw+QaJwSZiQL2ICN3mPny693X8JXYQ/DM=; b=4I8cEyDDCSKIRJYxkrOTc/s9j1p6BeDn9wSiqbZgm6E9XzfW09cxFcSeynUE2fbKisfgCI RR95PJm3kKuY/wviJx9qa2Vh/PaQqvj+xmHB5/iNfWI3Gkc0W/1xRtAqtQ1MWS+aAg6e1R lhi3GK3UZgNTLq+O4OEnW/tbJtiDgIc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=GTz3kaGN; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf08.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.122 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708024459; a=rsa-sha256; cv=none; b=GSZToptz2Eml4f1qvGv5sqlK3V6ttrJ0loqAdM691TDjEvqGH1LTVjwVT12JAnEpnB4nBV WH5FAEEdnL/0ypJL+e361f9dgmeVngaogeBT4LMIxYBTNwpjVsmSzKb8HD/YYC+4EKBj7G jFFztWg+p2wPOaoZk5irCG2tZ6XziqQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1708024457; bh=GzbroZFzU/UMoQI/0o02kQB2IRVJUnv3yXo9b71JnlM=; h=From:To:Cc:Subject:Date:From; b=GTz3kaGN+Ch5s8fKnFkczu9tvWOQyyZPmBNzg8iRu4HsXmcZI9QoMKuCohuyaBYf8 cAcNxtQlBSVfF+jpWO37RFtpVyPrnLfAPkntl5v6LbkrGUjSFL6qQlqIus2TMI4J9+ jiF0XdRWYbTxMi5BGu0csIoA3MLX+oXAGN61aRhIjYhRVatN2RCMZ+3Er7/rwrahwk gPPmI+ZbEbkRO/YK3tWTiKvkCUP7kdPZnPyn079ozSFofpYUrv5lD7YAwG+OPRtYrD MFxPCjYATzw00RfsiG+pNGOrEJAKUM5Rs/nA3sFaxIDeJIpkaKgpHgwujqiXqjSWXn SemykPm5mcyCQ== Received: from thinkos.internal.efficios.com (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4TbPr950ZnzZPw; Thu, 15 Feb 2024 14:14:17 -0500 (EST) From: Mathieu Desnoyers To: Dmitry Vyukov Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Peter Oskolkov , Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , Chris Kennelly , Andrew Morton , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , linux-mm@kvack.org Subject: [RFC PATCH 1/1] sched/rseq: Consider rseq abort in page fault handler Date: Thu, 15 Feb 2024 14:14:02 -0500 Message-Id: <20240215191402.681674-1-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-Rspamd-Queue-Id: 5F72016000C X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ajy3e3ccyfortxutmdu9ss5jz668rdit X-HE-Tag: 1708024459-371540 X-HE-Meta: U2FsdGVkX1+LUjxW6OCwxf3jWFeW7sNrmKVcJDVFhVfHsTGgTgvaf4pyHBELHa5VnEis0xMh7rgnjEAMZQ3UzXvabzHHklsXQgGfiInu8CxxSlZXo5Bu0fQpj3lIzDkbuBGDiyTcvFu5sHxjdMZGkw5egaWndSNGo8kU9Hqz1b6/tIL00LXUTN9EQ19jdWzGlu+9BwDIjLip54OtO8ZNBtNBL+NP+zEw4io8CHZ8SZLS92C7/cLdN0veOeTdRR2Lq8ZHzPif/nrSxuLrSddKYzQHghxmKVBwNGNFmiaqLxFqHXatOtiZnx+8icA5xWGUrkU0E9JgjHJyDYg4xY1kIEs5Urryzl2LUOfnq0wQbNPLg6K6YqrXXAsDu/GGhjHtPn/kYn155xUQpWJ6zX2wS36LrIWDOz/uHeEkXoVdexnCysRXEfJn13RSy8TBdr0FZwo7HxHMOV6VEn7RM1e2R9UOoe3MlMNx1XTW+oHOsFR/fs0YFtO5YpoR3oR/f6QPanMv/b8yfRpMb3HNDwL5odZ7+pwC9MyTs1AIQSbqpqn42EdqzuEkrB+aYbAItmVl9fTbvT+P5hsDs76y4S6PKEsPdzl84sKZtgzA1Z66F4zU6Bkk5oxJmJtRuxthN7XH/khQhfWZmyrxkYvewGuFaZCmJ+NNIp23GGnyfmQ+Pd2d4S+bWjRcRU4vaeNJrj9IPOdvFUc6KokvCnAr7nS7Gd1jYjS15S8MKTeJtw1CkiHfEmIoJxGZ8HP7y9tTUXBhrf4tPjhOBXNWUMlg4C5BaCBAx7vV7n54T+GlBPn8Eeeu864by7/JC/T//fvJjZEqJoTE7cRe8VRSV1PrRfuUQqIehWsjctmnlArMrTTSZVhes+27RmOwN68yRpY3u9vnUkKTAI0StRtuu4mlgw2GmKyL4PBQ/OjoT1bosws+ZOe9g6ia4BmkY3SJnRCCFsX7BODq9reD7QFBw0jBct/ PaBqi1nt dka38XmSki5oUdZM0fV4X7LPgJjxT7s8Bxim/ppBUU/nA7GiLtd1OH0gl2kTu3lKWdJzA7OK7twqpYeMkmYlHRM4wmKKpSLTuaocnyjtiigj3TfZgfCdD53BYXdAwBGv7rfM7e4ny9aukF8Ri5QAcapDkq/4zTDJjjSFFM2XWLg2EwOUjUdUGsQUt0UBQBe21p80LzWaWHOj4aClCn75EXmbvYHDUHi+3j9TU1OF9havm+RJNIZCe2L13OX7E5pbEqHHyIxJe97kF+7emJOgNePOn1epxJextYGVvMyPhYukZq24GHTtf1Y9scEphdUGco4B6o5qizOF7thwrMph8c+ZiHNHFB8XWV2TQrsnpwUYGUznjOq36maL8VzedkT/Q+hdQqZcFLV+hVfuodKfu1LaaU0su80nQWzeMONcd5DTrIQs/R0dZ1h7fEqbNjdaFWPN4Km4etiILHMumDCV80QFfhCIO5rQgwCpc54A4x5iBgnIOBm4dzpX9GsRBKLI8S/TE6D+I76gQMfpNR/kWdFaN8SxZYadIiDVr1c7azYkoG8WvH9fhoQLgTiN8djVA6j5SgpFZXroQtq9lrjJ93N7TmG8QiZhakv2a6Tx8MLKRwPTGf3n0/KIZ4lwSwFkDo+nP7EbLZbY7qF0D0xM1woqtgTS59qgZaTeo1rAuOb+PMrGjJlU4HzRGLsyRmcYhGSphJej0DC/gI2k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Consider rseq abort before emitting the SIGSEGV or SIGBUS signals from the page fault handler. This allows using membarrier MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ to abort rseq critical sections which include memory accesses to memory which mapping can be munmap'd or mprotect'd after the membarrier "rseq fence" without causing SIGSEGV or SIGBUS when the page fault handler triggered by a faulting memory access within a rseq critical section is preempted before handling the page fault. The problematic scenario is: CPU 0 CPU 1 ------------------------------------------------------------------ old_p = P P = NULL - rseq c.s. begins - x = P - if (x != NULL) - v = *x - page fault - preempted membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ) munmap(old_p) (or mprotect(old_p)) - handle page fault - force_sig_fault(SIGSEGV) - rseq resume notifier - move IP to abort IP -> SIGSEGV handler runs. This is solved by postponing the force_sig_fault() to return to user-space when the page fault handler detects that rseq events will cause the thread to call the rseq resume notifier before going back to user-space. This allows the rseq resume notifier to load the userspace memory pointed by rseq->rseq_cs to compare the IP with the rseq c.s. range before either moving the IP to the abort handler or calling force_sig_fault() with the parameters previously saved by the page fault handler. Add a new AT_RSEQ_FEATURE_FLAGS getauxval(3) to allow user-space to query whether the kernel implements this behavior (flag: RSEQ_FEATURE_PAGE_FAULT_ABORT). Untested implementation submitted for early feedback. Only x86 is implemented in this PoC. Link: https://lore.kernel.org/lkml/CACT4Y+bXfekygoyhO7pCctjnL15=E=Zs31BUGXU0dk8d4rc1Cw@mail.gmail.com/ Signed-off-by: Mathieu Desnoyers Cc: Dmitry Vyukov Cc: Peter Oskolkov Cc: Peter Zijlstra Cc: "Paul E. McKenney" Cc: Boqun Feng Cc: Chris Kennelly Cc: Andrew Morton Cc: Andy Lutomirski Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: linux-mm@kvack.org --- arch/x86/mm/fault.c | 4 ++-- fs/binfmt_elf.c | 1 + include/linux/sched.h | 16 ++++++++++++++++ include/linux/sched/signal.h | 24 ++++++++++++++++++++++++ include/uapi/linux/auxvec.h | 1 + include/uapi/linux/rseq.h | 7 +++++++ kernel/rseq.c | 36 +++++++++++++++++++++++++++++++----- 7 files changed, 82 insertions(+), 7 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 679b09cfe241..42ac39680cb6 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -854,7 +854,7 @@ __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code, if (si_code == SEGV_PKUERR) force_sig_pkuerr((void __user *)address, pkey); else - force_sig_fault(SIGSEGV, si_code, (void __user *)address); + rseq_lazy_force_sig_fault(SIGSEGV, si_code, (void __user *)address); local_irq_disable(); } @@ -973,7 +973,7 @@ do_sigbus(struct pt_regs *regs, unsigned long error_code, unsigned long address, return; } #endif - force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)address); + rseq_lazy_force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)address); } static int spurious_kernel_fault_check(unsigned long error_code, pte_t *pte) diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 5397b552fbeb..8fece0911c7d 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -273,6 +273,7 @@ create_elf_tables(struct linux_binprm *bprm, const struct elfhdr *exec, #ifdef CONFIG_RSEQ NEW_AUX_ENT(AT_RSEQ_FEATURE_SIZE, offsetof(struct rseq, end)); NEW_AUX_ENT(AT_RSEQ_ALIGN, __alignof__(struct rseq)); + NEW_AUX_ENT(AT_RSEQ_FEATURE_FLAGS, RSEQ_FEATURE_FLAGS); #endif #undef NEW_AUX_ENT /* AT_NULL is zero; clear the rest too */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..39aa585ba2a3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -743,6 +743,15 @@ struct kmap_ctrl { #endif }; +#ifdef CONFIG_RSEQ +struct rseq_lazy_sig { + bool pending; + int sig; + int code; + void __user *addr; +}; +#endif + struct task_struct { #ifdef CONFIG_THREAD_INFO_IN_TASK /* @@ -1317,6 +1326,7 @@ struct task_struct { * with respect to preemption. */ unsigned long rseq_event_mask; + struct rseq_lazy_sig rseq_lazy_sig; #endif #ifdef CONFIG_SCHED_MM_CID @@ -2330,6 +2340,8 @@ unsigned long sched_cpu_util(int cpu); #ifdef CONFIG_RSEQ +#define RSEQ_FEATURE_FLAGS RSEQ_FEATURE_PAGE_FAULT_ABORT + /* * Map the event mask on the user-space ABI enum rseq_cs_flags * for direct mask checks. @@ -2390,6 +2402,8 @@ static inline void rseq_migrate(struct task_struct *t) */ static inline void rseq_fork(struct task_struct *t, unsigned long clone_flags) { + WARN_ON_ONCE(current->rseq_lazy_sig.pending); + if (clone_flags & CLONE_VM) { t->rseq = NULL; t->rseq_len = 0; @@ -2405,6 +2419,8 @@ static inline void rseq_fork(struct task_struct *t, unsigned long clone_flags) static inline void rseq_execve(struct task_struct *t) { + WARN_ON_ONCE(current->rseq_lazy_sig.pending); + t->rseq = NULL; t->rseq_len = 0; t->rseq_sig = 0; diff --git a/include/linux/sched/signal.h b/include/linux/sched/signal.h index 3499c1a8b929..0d75dfde2f9b 100644 --- a/include/linux/sched/signal.h +++ b/include/linux/sched/signal.h @@ -781,4 +781,28 @@ static inline unsigned long rlimit_max(unsigned int limit) return task_rlimit_max(current, limit); } +#ifdef CONFIG_RSEQ + +static inline int rseq_lazy_force_sig_fault(int sig, int code, void __user *addr) +{ + struct task_struct *t = current; + + if (!t->rseq_event_mask) + return force_sig_fault(sig, code, addr); + t->rseq_lazy_sig.pending = true; + t->rseq_lazy_sig.sig = sig; + t->rseq_lazy_sig.code = code; + t->rseq_lazy_sig.addr = addr; + return 0; +} + +#else + +static inline int rseq_lazy_force_sig_fault(int sig, int code, void __user *addr) +{ + return force_sig_fault(sig, code, addr); +} + +#endif + #endif /* _LINUX_SCHED_SIGNAL_H */ diff --git a/include/uapi/linux/auxvec.h b/include/uapi/linux/auxvec.h index 6991c4b8ab18..5044f367a219 100644 --- a/include/uapi/linux/auxvec.h +++ b/include/uapi/linux/auxvec.h @@ -32,6 +32,7 @@ #define AT_HWCAP2 26 /* extension of AT_HWCAP */ #define AT_RSEQ_FEATURE_SIZE 27 /* rseq supported feature size */ #define AT_RSEQ_ALIGN 28 /* rseq allocation alignment */ +#define AT_RSEQ_FEATURE_FLAGS 29 /* rseq feature flags */ #define AT_EXECFN 31 /* filename of program */ diff --git a/include/uapi/linux/rseq.h b/include/uapi/linux/rseq.h index c233aae5eac9..0fdb192e3cd3 100644 --- a/include/uapi/linux/rseq.h +++ b/include/uapi/linux/rseq.h @@ -37,6 +37,13 @@ enum rseq_cs_flags { (1U << RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE_BIT), }; +/* + * rseq feature flags. Query with getauxval(AT_RSEQ_FEATURE_FLAGS). + */ +enum rseq_feature_flags { + RSEQ_FEATURE_PAGE_FAULT_ABORT = (1U << 0), +}; + /* * struct rseq_cs is aligned on 4 * 8 bytes to ensure it is always * contained within a single cache-line. It is usually declared as diff --git a/kernel/rseq.c b/kernel/rseq.c index 9de6e35fe679..f686a97abb45 100644 --- a/kernel/rseq.c +++ b/kernel/rseq.c @@ -271,6 +271,25 @@ static bool in_rseq_cs(unsigned long ip, struct rseq_cs *rseq_cs) return ip - rseq_cs->start_ip < rseq_cs->post_commit_offset; } +static void rseq_clear_lazy_sig_fault(struct task_struct *t) +{ + if (!t->rseq_lazy_sig.pending) + return; + t->rseq_lazy_sig.pending = false; + t->rseq_lazy_sig.sig = 0; + t->rseq_lazy_sig.code = 0; + t->rseq_lazy_sig.addr = NULL; +} + +static void rseq_force_lazy_sig_fault(struct task_struct *t) +{ + if (!t->rseq_lazy_sig.pending) + return; + force_sig_fault(t->rseq_lazy_sig.sig, t->rseq_lazy_sig.code, + t->rseq_lazy_sig.addr); + rseq_clear_lazy_sig_fault(t); +} + static int rseq_ip_fixup(struct pt_regs *regs) { unsigned long ip = instruction_pointer(regs); @@ -280,25 +299,32 @@ static int rseq_ip_fixup(struct pt_regs *regs) ret = rseq_get_rseq_cs(t, &rseq_cs); if (ret) - return ret; + goto nofixup; /* * Handle potentially not being within a critical section. * If not nested over a rseq critical section, restart is useless. * Clear the rseq_cs pointer and return. */ - if (!in_rseq_cs(ip, &rseq_cs)) - return clear_rseq_cs(t); + if (!in_rseq_cs(ip, &rseq_cs)) { + ret = clear_rseq_cs(t); + goto nofixup; + } ret = rseq_need_restart(t, rseq_cs.flags); if (ret <= 0) - return ret; + goto nofixup; ret = clear_rseq_cs(t); if (ret) - return ret; + goto nofixup; + rseq_clear_lazy_sig_fault(t); trace_rseq_ip_fixup(ip, rseq_cs.start_ip, rseq_cs.post_commit_offset, rseq_cs.abort_ip); instruction_pointer_set(regs, (unsigned long)rseq_cs.abort_ip); return 0; + +nofixup: + rseq_force_lazy_sig_fault(t); + return ret; } /*