From patchwork Wed May 8 09:47:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13658417 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4F2F17E103; Wed, 8 May 2024 10:02:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=166.125.252.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715162586; cv=none; b=pg6O8BYLIVQkRMVOAHoYrBIhxyuEgnGumNcAPBFtVnYluyaT1+BDQsMQjASjQw9XsfpmK18Q46Em+W1PQjCiBzolLwUOsWIgLaE9Elqc/kIuG070+e4azVSES0xkMQrjyTtIA0NwOytdymYHGQuIX35MLp1rU8KbueyakYppmmw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715162586; c=relaxed/simple; bh=RXDdtbJ20AGlV8um9gNex7v/mq8xYJsfh3LEG9fA2MM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=PJec0x6TDWCl6N+4aVYb1hxWDv/WhdNmjFo0ljLFCilfJziEEbO80/Hc4lM0mZ1531AK1SpP55MYZEOrCsKizd5HKTieYjCH8W9Sn9eiytJ6jeZlpjLV/HqGzBsVEg6n+H7pcuMo/eI5xyGph1XEYCJYytGuCwdCEyZA/vvj7v0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com; spf=pass smtp.mailfrom=sk.com; arc=none smtp.client-ip=166.125.252.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=sk.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sk.com X-AuditID: a67dfc5b-d85ff70000001748-85-663b4a394a9a From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, 42.hyeyoo@gmail.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, max.byungchul.park@gmail.com, boqun.feng@gmail.com, longman@redhat.com, hdanton@sina.com, her0gyugyu@gmail.com Subject: [PATCH v14 07/28] dept: Distinguish each syscall context from another Date: Wed, 8 May 2024 18:47:04 +0900 Message-Id: <20240508094726.35754-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240508094726.35754-1-byungchul@sk.com> References: <20240508094726.35754-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe2xLcRTH/X739t67RuXmkrhGPCoisRjzPEMECS5BCP+wBI3dWemm6Wy2 hWS62osJkirTSPdI7VE2t8Rma9VkL2zKhq1mbJnH6DZGx6xB6/HPySfnfM/n/HMYgrPLQhl1 /GFRF6/SKCk5Ke8fmz83cuPymPlGzzw4e2o++L5mkWAut1HgvlaGwHbjOIa+uvXwfNiLYLT5 EQEmoxtBfvdLAm7UdyFwFOspaO0dB22+QQqajCcpSC8sp+DxRz+GzvPnMJRJm+HBmQIMrpF3 JJj6KLhkSseB8h7DiLWUBmvaLOgpzqPB3x0BTV3PZODwhMHFy50U1DiaSKiv7MHQettMQZft lwwe1DeS4D6bK4OrAwUUfBy2EmD1DdLwxGXBUGEIiDK+/JRBQ64LQ0bRdQxtHdUInFmvMUi2 ZxTc83kx2CUjAT+u1CHoOd1Pw4lTIzRcOn4awckT50kwdC6G0e9malWkcM87SAgG+xHBMWwh hfsFvFCV95IWDE4PLVikRMFePEcorOnDQv6QTyZIpdmUIA2do4Wc/jYsDLS00ELjhVFS6G0z 4a2hu+QrokWNOknUzVu5Vx5bU1KNtA17kw36D3QaqtqUgxiGZxfx7dkxOSjkD5rdejrIFDub b28fIYI8gZ3O23PfyoJMsF45X9SyLsjj2S38q29OKqgh2Vn8xZzIYFvBLubrXVfIv8ppfFmF 648mhF3Cd7wbQEHmApnq9LzAKXkg84vhW/P0/xYm8XeL28kzSGFBY0oRp45PilOpNYvCY1Pi 1cnh+w7FSSjwStZj/qhKNOTeXotYBinHKlwTl8VwMlVSQkpcLeIZQjlBUZe5NIZTRKtSUkXd oT26RI2YUIsmM6RyomLB8JFojt2vOiweFEWtqPs/xUxIaBpKvdyc+Soq9c2OVbeL7u9q/H4g YsOLTFPlmgW3/NrIsPzq6dhVYukdWmhcv32qczeX6vhk6/uyzWlOljSPqpZkJzIc1t9sWPvV PKPZUsLdKVlbVp6s5TP6vatXvHj4me2YUhvu6cHZfju5M2p8p2fZGOnogS3a7oNZmplP/d1P i5RkQqwqYg6hS1D9BukEyo9GAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSfUzMcRzHfX/Pd+v47TT9pA03NKE04kNm/YF+89DwB2NMN/3S6bryO5U8 rKvOU8oK3ZFj18NOKuLOKHXtVisSOWpUKrSGnheupIaO+eez197v915/fRhcbiC9GZXmqCBq lGoFJSWkYcFpy9ZsDo5c7sqYBdkZy8H1/RwBprJSCpx3SxCUPkjBoLcuFN6ODiCYePESB2OO E0Hex04cHtR3IbAXpVLQ3DMdWlzDFDTkXKAgraCMglf9kxh0GC5hUGLdBo1Z+Rg4xj8TYOyl 4LoxDZs6XzAYtxTTYNEthO6iXBomPwZCQ9cbEmpvNJBgb18C1252UFBlbyCgvrwbg+bHJgq6 Sn+T0Fj/lABndiYJd4byKegfteBgcQ3T8NphxuCefsp25tsvEp5kOjA4U3gfg5a2SgTV5z5g YC19Q0GtawADmzUHh5+36hB0Xxyk4XTGOA3XUy4iuHDaQIC+IwgmfpiokLV87cAwzuttibx9 1Ezwz/I5viK3k+b11e00b7bG87YiP76gqhfj8766SN5afJ7irV8v0Xz6YAvGDzU10fzTqxME 39NixLbP2StdFyGoVQmCGLA+XBpVdbsSxT0JP6ZP7aN1qGJrOpIwHLuSMzlTaTdTrC/X2jqO u9mTncfZMj+RbsbZASlX2LTJzTPZMO79WDWVjhiGYBdy19LXuGMZG8TVO24R/5RzuZJ7jr8a CbuKa/s8hNwsn9pUpuXSWUhqRtOKkadKkxCjVKmD/LXRUUka1TH/g7ExVjT1LZZTk9nl6Htz aA1iGaTwkDmp4Eg5qUzQJsXUII7BFZ6yurOrI+WyCGXScUGMPSDGqwVtDZrDEAov2ebdQric PaQ8KkQLQpwg/m8xRuKtQ8khjkUB++fXHNoRKs6XHA7w6X30bOmJPMZQPnskfL9uy06poTNx aW52wZWDGwdzHJKyxYW+PcK7DaJ6V8WpsZNVic9bbb4+ybd/+jeZzp5cEOlXFnhzqxjx8Ejf 5bXx9r6sFXtok27GyJ55Yfs8fI1Epk/z2/NeSeU+yRaNdwhuBgWhjVIG+uGiVvkHdT4TCikD AAA= X-CFilter-Loop: Reflected Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: It enters kernel mode on each syscall and each syscall handling should be considered independently from the point of view of Dept. Otherwise, Dept may wrongly track dependencies across different syscalls. That might be a real dependency from user mode. However, now that Dept just started to work, conservatively let Dept not track dependencies across different syscalls. Signed-off-by: Byungchul Park --- arch/arm64/kernel/syscall.c | 3 ++ arch/x86/entry/common.c | 4 +++ include/linux/dept.h | 39 ++++++++++++--------- kernel/dependency/dept.c | 67 +++++++++++++++++++------------------ 4 files changed, 64 insertions(+), 49 deletions(-) diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index ad198262b981..1b41cc443db5 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -97,6 +98,8 @@ static void el0_svc_common(struct pt_regs *regs, int scno, int sc_nr, * (Similarly for HVC and SMC elsewhere.) */ + dept_kernel_enter(); + if (flags & _TIF_MTE_ASYNC_FAULT) { /* * Process the asynchronous tag check fault before the actual diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 51cc9c7cb9bd..69d41f863c7d 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -20,6 +20,7 @@ #include #include #include +#include #ifdef CONFIG_XEN_PV #include @@ -75,6 +76,7 @@ static __always_inline bool do_syscall_x32(struct pt_regs *regs, int nr) /* Returns true to return using SYSRET, or false to use IRET */ __visible noinstr bool do_syscall_64(struct pt_regs *regs, int nr) { + dept_kernel_enter(); add_random_kstack_offset(); nr = syscall_enter_from_user_mode(regs, nr); @@ -327,6 +329,7 @@ __visible noinstr void do_int80_syscall_32(struct pt_regs *regs) { int nr = syscall_32_enter(regs); + dept_kernel_enter(); add_random_kstack_offset(); /* * Subtlety here: if ptrace pokes something larger than 2^31-1 into @@ -348,6 +351,7 @@ static noinstr bool __do_fast_syscall_32(struct pt_regs *regs) int nr = syscall_32_enter(regs); int res; + dept_kernel_enter(); add_random_kstack_offset(); /* * This cannot use syscall_enter_from_user_mode() as it has to diff --git a/include/linux/dept.h b/include/linux/dept.h index c6e2291dd843..4e359f76ac3c 100644 --- a/include/linux/dept.h +++ b/include/linux/dept.h @@ -25,11 +25,16 @@ struct task_struct; #define DEPT_MAX_SUBCLASSES_USR (DEPT_MAX_SUBCLASSES / DEPT_MAX_SUBCLASSES_EVT) #define DEPT_MAX_SUBCLASSES_CACHE 2 -#define DEPT_SIRQ 0 -#define DEPT_HIRQ 1 -#define DEPT_IRQS_NR 2 -#define DEPT_SIRQF (1UL << DEPT_SIRQ) -#define DEPT_HIRQF (1UL << DEPT_HIRQ) +enum { + DEPT_CXT_SIRQ = 0, + DEPT_CXT_HIRQ, + DEPT_CXT_IRQS_NR, + DEPT_CXT_PROCESS = DEPT_CXT_IRQS_NR, + DEPT_CXTS_NR +}; + +#define DEPT_SIRQF (1UL << DEPT_CXT_SIRQ) +#define DEPT_HIRQF (1UL << DEPT_CXT_HIRQ) struct dept_ecxt; struct dept_iecxt { @@ -94,8 +99,8 @@ struct dept_class { /* * for tracking IRQ dependencies */ - struct dept_iecxt iecxt[DEPT_IRQS_NR]; - struct dept_iwait iwait[DEPT_IRQS_NR]; + struct dept_iecxt iecxt[DEPT_CXT_IRQS_NR]; + struct dept_iwait iwait[DEPT_CXT_IRQS_NR]; /* * classified by a map embedded in task_struct, @@ -207,8 +212,8 @@ struct dept_ecxt { /* * where the IRQ-enabled happened */ - unsigned long enirq_ip[DEPT_IRQS_NR]; - struct dept_stack *enirq_stack[DEPT_IRQS_NR]; + unsigned long enirq_ip[DEPT_CXT_IRQS_NR]; + struct dept_stack *enirq_stack[DEPT_CXT_IRQS_NR]; /* * where the event context started @@ -252,8 +257,8 @@ struct dept_wait { /* * where the IRQ wait happened */ - unsigned long irq_ip[DEPT_IRQS_NR]; - struct dept_stack *irq_stack[DEPT_IRQS_NR]; + unsigned long irq_ip[DEPT_CXT_IRQS_NR]; + struct dept_stack *irq_stack[DEPT_CXT_IRQS_NR]; /* * where the wait happened @@ -406,19 +411,19 @@ struct dept_task { int wait_hist_pos; /* - * sequential id to identify each IRQ context + * sequential id to identify each context */ - unsigned int irq_id[DEPT_IRQS_NR]; + unsigned int cxt_id[DEPT_CXTS_NR]; /* * for tracking IRQ-enabled points with cross-event */ - unsigned int wgen_enirq[DEPT_IRQS_NR]; + unsigned int wgen_enirq[DEPT_CXT_IRQS_NR]; /* * for keeping up-to-date IRQ-enabled points */ - unsigned long enirq_ip[DEPT_IRQS_NR]; + unsigned long enirq_ip[DEPT_CXT_IRQS_NR]; /* * for reserving a current stack instance at each operation @@ -465,7 +470,7 @@ struct dept_task { .wait_hist = { { .wait = NULL, } }, \ .ecxt_held_pos = 0, \ .wait_hist_pos = 0, \ - .irq_id = { 0U }, \ + .cxt_id = { 0U }, \ .wgen_enirq = { 0U }, \ .enirq_ip = { 0UL }, \ .stack = NULL, \ @@ -503,6 +508,7 @@ extern void dept_event(struct dept_map *m, unsigned long e_f, unsigned long ip, extern void dept_ecxt_exit(struct dept_map *m, unsigned long e_f, unsigned long ip); extern void dept_sched_enter(void); extern void dept_sched_exit(void); +extern void dept_kernel_enter(void); static inline void dept_ecxt_enter_nokeep(struct dept_map *m) { @@ -552,6 +558,7 @@ struct dept_task { }; #define dept_ecxt_exit(m, e_f, ip) do { } while (0) #define dept_sched_enter() do { } while (0) #define dept_sched_exit() do { } while (0) +#define dept_kernel_enter() do { } while (0) #define dept_ecxt_enter_nokeep(m) do { } while (0) #define dept_key_init(k) do { (void)(k); } while (0) #define dept_key_destroy(k) do { (void)(k); } while (0) diff --git a/kernel/dependency/dept.c b/kernel/dependency/dept.c index 19406093103e..9aba9eb22760 100644 --- a/kernel/dependency/dept.c +++ b/kernel/dependency/dept.c @@ -220,9 +220,9 @@ static struct dept_class *dep_tc(struct dept_dep *d) static const char *irq_str(int irq) { - if (irq == DEPT_SIRQ) + if (irq == DEPT_CXT_SIRQ) return "softirq"; - if (irq == DEPT_HIRQ) + if (irq == DEPT_CXT_HIRQ) return "hardirq"; return "(unknown)"; } @@ -406,7 +406,7 @@ static void initialize_class(struct dept_class *c) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { struct dept_iecxt *ie = &c->iecxt[i]; struct dept_iwait *iw = &c->iwait[i]; @@ -431,7 +431,7 @@ static void initialize_ecxt(struct dept_ecxt *e) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { e->enirq_stack[i] = NULL; e->enirq_ip[i] = 0UL; } @@ -447,7 +447,7 @@ static void initialize_wait(struct dept_wait *w) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { w->irq_stack[i] = NULL; w->irq_ip[i] = 0UL; } @@ -486,7 +486,7 @@ static void destroy_ecxt(struct dept_ecxt *e) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) if (e->enirq_stack[i]) put_stack(e->enirq_stack[i]); if (e->class) @@ -502,7 +502,7 @@ static void destroy_wait(struct dept_wait *w) { int i; - for (i = 0; i < DEPT_IRQS_NR; i++) + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) if (w->irq_stack[i]) put_stack(w->irq_stack[i]); if (w->class) @@ -651,7 +651,7 @@ static void print_diagram(struct dept_dep *d) const char *tc_n = tc->sched_map ? "" : (tc->name ?: "(unknown)"); irqf = e->enirqf & w->irqf; - for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + for_each_set_bit(irq, &irqf, DEPT_CXT_IRQS_NR) { if (!firstline) pr_warn("\nor\n\n"); firstline = false; @@ -684,7 +684,7 @@ static void print_dep(struct dept_dep *d) const char *tc_n = tc->sched_map ? "" : (tc->name ?: "(unknown)"); irqf = e->enirqf & w->irqf; - for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + for_each_set_bit(irq, &irqf, DEPT_CXT_IRQS_NR) { pr_warn("%s has been enabled:\n", irq_str(irq)); print_ip_stack(e->enirq_ip[irq], e->enirq_stack[irq]); pr_warn("\n"); @@ -910,7 +910,7 @@ static void bfs(struct dept_class *c, bfs_f *cb, void *in, void **out) */ static unsigned long cur_enirqf(void); -static int cur_irq(void); +static int cur_cxt(void); static unsigned int cur_ctxt_id(void); static struct dept_iecxt *iecxt(struct dept_class *c, int irq) @@ -1458,7 +1458,7 @@ static void add_dep(struct dept_ecxt *e, struct dept_wait *w) if (d) { check_dl_bfs(d); - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { struct dept_iwait *fiw = iwait(fc, i); struct dept_iecxt *found_ie; struct dept_iwait *found_iw; @@ -1494,7 +1494,7 @@ static void add_wait(struct dept_class *c, unsigned long ip, struct dept_task *dt = dept_task(); struct dept_wait *w; unsigned int wg = 0U; - int irq; + int cxt; int i; if (DEPT_WARN_ON(!valid_class(c))) @@ -1510,9 +1510,9 @@ static void add_wait(struct dept_class *c, unsigned long ip, w->wait_stack = get_current_stack(); w->sched_sleep = sched_sleep; - irq = cur_irq(); - if (irq < DEPT_IRQS_NR) - add_iwait(c, irq, w); + cxt = cur_cxt(); + if (cxt == DEPT_CXT_HIRQ || cxt == DEPT_CXT_SIRQ) + add_iwait(c, cxt, w); /* * Avoid adding dependency between user aware nested ecxt and @@ -1593,7 +1593,7 @@ static bool add_ecxt(struct dept_map *m, struct dept_class *c, eh->sub_l = sub_l; irqf = cur_enirqf(); - for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) + for_each_set_bit(irq, &irqf, DEPT_CXT_IRQS_NR) add_iecxt(c, irq, e, false); del_ecxt(e); @@ -1745,7 +1745,7 @@ static void do_event(struct dept_map *m, struct dept_class *c, add_dep(eh->ecxt, wh->wait); } - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { struct dept_ecxt *e; if (before(dt->wgen_enirq[i], wg)) @@ -1787,7 +1787,7 @@ static void disconnect_class(struct dept_class *c) call_rcu(&d->rh, del_dep_rcu); } - for (i = 0; i < DEPT_IRQS_NR; i++) { + for (i = 0; i < DEPT_CXT_IRQS_NR; i++) { stale_iecxt(iecxt(c, i)); stale_iwait(iwait(c, i)); } @@ -1812,27 +1812,21 @@ static unsigned long cur_enirqf(void) return 0UL; } -static int cur_irq(void) +static int cur_cxt(void) { if (lockdep_softirq_context(current)) - return DEPT_SIRQ; + return DEPT_CXT_SIRQ; if (lockdep_hardirq_context()) - return DEPT_HIRQ; - return DEPT_IRQS_NR; + return DEPT_CXT_HIRQ; + return DEPT_CXT_PROCESS; } static unsigned int cur_ctxt_id(void) { struct dept_task *dt = dept_task(); - int irq = cur_irq(); + int cxt = cur_cxt(); - /* - * Normal process context - */ - if (irq == DEPT_IRQS_NR) - return 0U; - - return dt->irq_id[irq] | (1UL << irq); + return dt->cxt_id[cxt] | (1UL << cxt); } static void enirq_transition(int irq) @@ -1893,7 +1887,7 @@ static void dept_enirq(unsigned long ip) flags = dept_enter(); - for_each_set_bit(irq, &irqf, DEPT_IRQS_NR) { + for_each_set_bit(irq, &irqf, DEPT_CXT_IRQS_NR) { dt->enirq_ip[irq] = ip; enirq_transition(irq); } @@ -1939,6 +1933,13 @@ void noinstr dept_hardirqs_off(void) dept_task()->hardirqs_enabled = false; } +void noinstr dept_kernel_enter(void) +{ + struct dept_task *dt = dept_task(); + + dt->cxt_id[DEPT_CXT_PROCESS] += 1UL << DEPT_CXTS_NR; +} + /* * Ensure it's the outmost softirq context. */ @@ -1946,7 +1947,7 @@ void dept_softirq_enter(void) { struct dept_task *dt = dept_task(); - dt->irq_id[DEPT_SIRQ] += 1UL << DEPT_IRQS_NR; + dt->cxt_id[DEPT_CXT_SIRQ] += 1UL << DEPT_CXTS_NR; } /* @@ -1956,7 +1957,7 @@ void noinstr dept_hardirq_enter(void) { struct dept_task *dt = dept_task(); - dt->irq_id[DEPT_HIRQ] += 1UL << DEPT_IRQS_NR; + dt->cxt_id[DEPT_CXT_HIRQ] += 1UL << DEPT_CXTS_NR; } void dept_sched_enter(void)