From patchwork Fri Jun 9 06:45:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Whitchurch X-Patchwork-Id: 13273287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0E3CC7EE2F for ; Fri, 9 Jun 2023 06:46:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2146D8E0003; Fri, 9 Jun 2023 02:46:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C4648E0001; Fri, 9 Jun 2023 02:46:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B4728E0003; Fri, 9 Jun 2023 02:46:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EEDC48E0001 for ; Fri, 9 Jun 2023 02:46:23 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1A59940159 for ; Fri, 9 Jun 2023 06:46:23 +0000 (UTC) X-FDA: 80882275446.30.FEB2B41 Received: from smtp1.axis.com (smtp1.axis.com [195.60.68.17]) by imf04.hostedemail.com (Postfix) with ESMTP id E536940010 for ; Fri, 9 Jun 2023 06:46:20 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=axis.com header.s=axis-central1 header.b=aBvBTL82; dmarc=pass (policy=none) header.from=axis.com; spf=pass (imf04.hostedemail.com: domain of Vincent.Whitchurch@axis.com designates 195.60.68.17 as permitted sender) smtp.mailfrom=Vincent.Whitchurch@axis.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686293181; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=ZWYAkALE3Z1DeilPZzKUrfG1wIS0HdnSyXcG9sXjfuM=; b=PEpwfl42mNKi5CCEUvKty7KbwbnEvHZjuX22qdJnSR6qF5gDk8XGrLtfyfnLxsnAn1i2N5 mt4pn9RVl/l70ZAUhITW3viqci4Q2wCq6T6AwsG/IC7NDPB+efM3DUONfcH4pbM8lNqKGa eUC1Xr1qMlEUbh4K4SXUvQmORo+MVAw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=axis.com header.s=axis-central1 header.b=aBvBTL82; dmarc=pass (policy=none) header.from=axis.com; spf=pass (imf04.hostedemail.com: domain of Vincent.Whitchurch@axis.com designates 195.60.68.17 as permitted sender) smtp.mailfrom=Vincent.Whitchurch@axis.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686293181; a=rsa-sha256; cv=none; b=wS5SGSxd2vjnKgIdwkrTjFfVUr0WA4vTQ8D/ekYTZ4wPhGcVmFcVFz77r+Cnpl+mVRX75+ zXomC/RUuVWq4NPjkbC1G3U8EPnXlLWZ4aihAlXxpRO4I1IiRetN+/D5gqAp5PVOw61ecY jyxG3rg/Vr7XyFD1H1xIV3GS/wG1lKI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axis.com; q=dns/txt; s=axis-central1; t=1686293181; x=1717829181; h=from:date:subject:mime-version:content-transfer-encoding: message-id:to:cc; bh=ZWYAkALE3Z1DeilPZzKUrfG1wIS0HdnSyXcG9sXjfuM=; b=aBvBTL829Owu/OQsixRgEMwaoy2nxdlCpFHyWCk+pJDxHpLF4tT3A6vv 6Um3HvIEEzM4Hun/bRNVI5TO0Sclg7XyfPmMc/yY95EHXwByVFbUfjM2o P8E4VCXiVXiIQgQ6DRHGI2/NqYqP3oWgILgHBUu2POK3h/i39893I2TB4 vodruBFEAPTHDuwgaibcpfnluEZqkJxHF0/xuQTgPBDStKax/6E28UXdk yb+fIJ9blUVbSrwnJymt/30waqvKljjrq7QCNvBrg2F867ygWazDt3xV/ dGv6721amXFsacF3uJFKyaLR9dUP/ayqKMX6cpSM4IgaFohYZNmJ2g0hR Q==; From: Vincent Whitchurch Date: Fri, 9 Jun 2023 08:45:55 +0200 Subject: [PATCH] watchdog/mm: Allow dumping memory info in pretimeout MIME-Version: 1.0 Message-ID: <20230608-pretimeout-oom-v1-1-542cc91062d7@axis.com> X-B4-Tracking: v=1; b=H4sIAKLKgmQC/x3NQQ5AMBCF4avIrE3SIlKuIhbFlFnoSItIxN2V5 f+SL++GSIEpQpvdEOjkyOJT6DyDcbF+JuQpNRSqKFWtDG6Bdl5Jjh1FVmwaXZmqNFZPDhIabCQ cgvXj8jEncn1zUo6v/6frn+cF/Rc98HcAAAA= To: Wim Van Sebroeck , Guenter Roeck , Andrew Morton CC: , , , , Vincent Whitchurch X-Mailer: b4 0.12.2 X-Rspamd-Queue-Id: E536940010 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: hpr17cujoam8agythmremskn5wej7aj9 X-HE-Tag: 1686293180-246306 X-HE-Meta: U2FsdGVkX18Dpt432W/WSj5ChskDV/Ons2TX1umx+kuB19t2z4FNIs/ZEfFoSrBal/sXdcuxyfr4VNheUnZw3E6R6xALE8uizUbKcOpocIORPSaJa8wAyJJzkU1/ereGGTtlUplvK7prj4BlkrfFh2G6pJJQHhy0zyOFEfb8PkcHg8iCHGv/rhjhFI6xYPJcfweXWtOE4fJWTbi9M4IUBDq8jC7R9cnZcHdMaQqk2UeZEF7dwKGjXq2jKLeDpGeqMhF1kTCDRYDO1R8s9hkivMgZ11TcejzgYz/EPnRieuFADm2SfmFqznG+v0m3rlcGWWgxvXGxeHHG5VgHK8Unb5XQ2j97fEXQ0IkSB2a15AMKUNyPbV+NEZOfZAzQsCPMiWWa1VmujF6mjdrQUnm2cnn0GB272+0H9JgWVvsBUZNCFRtEfwaNzk4NFSPxzvle81mStfQeZ9pWh8tr7uLnUOyewrj146+6tD9Tpej7X4EOAAoV6FCY50YGB43v9YWP50rl8/DwqZibD4WtIiWx7V+50OGjlw6phwhmJr+utxCdwf9bwEIWsFT0/NQwJJNJYq3lUwmMNQC4Tj6e3licY5N50J9BA3nvVNzuav85rWK/R/n6XE4V69nIll1OtRHfePkZ7RhtEScQA2mi3//4eRuCAvyHDjHLarg61/6njpxJQ3xXU0XDBWxbTEPhxZ3vtulcIgen6j6yo9MVXkl5IN61HeOqXXKSQELAkU92TC9+zx0FuXCSW6SO2/wApQsq23nXKNMNAuQn5QTBo54jT8c5dFrNy99dkQ+Pfpg6KDDQMLfYmreaQXpPhnr9eNXtM4z2NcYmvK4ILvn/5M60XUQ7c/lYQuEQP0KZifainHgmrNUGzXjFTXUeDA3/wzm5eEzmi6y8IuncxlM4Me9e1HUb+jQg5jdC79ZHTlOflWkvrVphOaRxJWbvs3XU7o7kLQCuVyBp0ZEI9zDn+F8 /OrRLK41 bw8GMZRRmr9cAtCLjd+1CYPhoQMeBy3KA+BafrDAylKbPI77NhXMkc2van9nGAl2OgRNz8j1LxqJ8Fv3Rmi7MStyHBocP0dm45Mj0yWuYVvjL8I0lYrLhtB/aqB+O79EnGv+nZph37RxucH0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On my (embedded) systems, the most common cause of hitting the watchdog (pre)timeout is due to thrashing. Diagnosing these problems is hard without knowing the memory state at the point of the watchdog hit. In order to make this information available, add a module parameter to the watchdog pretimeout panic governor to ask it to dump memory info and the OOM task list (using a new helper in the OOM code) before triggering the panic. Signed-off-by: Vincent Whitchurch --- drivers/watchdog/pretimeout_panic.c | 15 +++++++++++ include/linux/oom.h | 5 ++++ include/linux/sched/task.h | 5 ++++ mm/oom_kill.c | 54 ++++++++++++++++++++++++++++++++++++- 4 files changed, 78 insertions(+), 1 deletion(-) --- base-commit: 9561de3a55bed6bdd44a12820ba81ec416e705a7 change-id: 20230608-pretimeout-oom-99148438a1df Best regards, diff --git a/drivers/watchdog/pretimeout_panic.c b/drivers/watchdog/pretimeout_panic.c index 2cc3c41d2be5b..52d686fa541c7 100644 --- a/drivers/watchdog/pretimeout_panic.c +++ b/drivers/watchdog/pretimeout_panic.c @@ -5,10 +5,15 @@ #include #include +#include +#include #include #include "watchdog_pretimeout.h" +static unsigned long dump_min_rss_bytes; +module_param(dump_min_rss_bytes, ulong, 0644); + /** * pretimeout_panic - Panic on watchdog pretimeout event * @wdd - watchdog_device @@ -17,6 +22,16 @@ */ static void pretimeout_panic(struct watchdog_device *wdd) { + /* + * Since the root cause is not certain to be low memory, only print + * tasks with RSS above a configurable limit, to avoid losing + * potentially more important messages from the log. + */ + if (dump_min_rss_bytes) { + show_mem(SHOW_MEM_FILTER_NODES, NULL); + oom_dump_tasks(DIV_ROUND_UP(dump_min_rss_bytes, PAGE_SIZE)); + } + panic("watchdog pretimeout event\n"); } diff --git a/include/linux/oom.h b/include/linux/oom.h index 7d0c9c48a0c54..1451fe2c38d78 100644 --- a/include/linux/oom.h +++ b/include/linux/oom.h @@ -52,6 +52,9 @@ struct oom_control { /* Used to print the constraint info. */ enum oom_constraint constraint; + + bool dump_trylock; + unsigned long dump_min_rss_pages; }; extern struct mutex oom_lock; @@ -102,6 +105,8 @@ long oom_badness(struct task_struct *p, extern bool out_of_memory(struct oom_control *oc); +extern void oom_dump_tasks(unsigned long min_rss_pages); + extern void exit_oom_victim(void); extern int register_oom_notifier(struct notifier_block *nb); diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h index e0f5ac90a228b..e8a68b2a3e829 100644 --- a/include/linux/sched/task.h +++ b/include/linux/sched/task.h @@ -183,6 +183,11 @@ static inline void task_lock(struct task_struct *p) spin_lock(&p->alloc_lock); } +static inline int task_trylock(struct task_struct *p) +{ + return spin_trylock(&p->alloc_lock); +} + static inline void task_unlock(struct task_struct *p) { spin_unlock(&p->alloc_lock); diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 044e1eed720ee..87e554a1c152f 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -149,6 +149,30 @@ struct task_struct *find_lock_task_mm(struct task_struct *p) return t; } +/* + * Identical to the above, except that we avoid tasks which we can't lock, to + * avoid deadlocks when called from an interrupt handler. + */ +struct task_struct *find_trylock_task_mm(struct task_struct *p) +{ + struct task_struct *t; + + rcu_read_lock(); + + for_each_thread(p, t) { + if (!task_trylock(t)) + continue; + if (likely(t->mm)) + goto found; + task_unlock(t); + } + t = NULL; +found: + rcu_read_unlock(); + + return t; +} + /* * order == -1 means the oom kill is required by sysrq, otherwise only * for display purposes. @@ -390,15 +414,26 @@ static int dump_task(struct task_struct *p, void *arg) if (!is_memcg_oom(oc) && !oom_cpuset_eligible(p, oc)) return 0; - task = find_lock_task_mm(p); + task = oc->dump_trylock ? find_trylock_task_mm(p) : + find_lock_task_mm(p); if (!task) { /* * All of p's threads have already detached their mm's. There's * no need to report them; they can't be oom killed anyway. + * + * Or we got here from an interrupt and the task lock is + * locked, in which case we're forced to ignore this task to + * avoid deadlocks. */ return 0; } + if (oc->dump_min_rss_pages && + get_mm_rss(task->mm) < oc->dump_min_rss_pages) { + task_unlock(task); + return 0; + } + pr_info("[%7d] %5d %5d %8lu %8lu %8ld %8lu %5hd %s\n", task->pid, from_kuid(&init_user_ns, task_uid(task)), task->tgid, task->mm->total_vm, get_mm_rss(task->mm), @@ -437,6 +472,23 @@ static void dump_tasks(struct oom_control *oc) } } +void oom_dump_tasks(unsigned long min_rss_pages) +{ + const gfp_t gfp_mask = GFP_KERNEL; + struct oom_control oc = { + .zonelist = node_zonelist(first_memory_node, gfp_mask), + .nodemask = NULL, + .memcg = NULL, + .gfp_mask = gfp_mask, + .order = -1, + .dump_min_rss_pages = min_rss_pages, + .dump_trylock = in_interrupt(), + }; + + dump_tasks(&oc); +} +EXPORT_SYMBOL_GPL(oom_dump_tasks); + static void dump_oom_summary(struct oom_control *oc, struct task_struct *victim) { /* one line summary of the oom killer context. */