From patchwork Sun Jun 12 18:32:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12878770 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB39FC433EF for ; Sun, 12 Jun 2022 18:33:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 385406B028F; Sun, 12 Jun 2022 14:33:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F6ED8D0131; Sun, 12 Jun 2022 14:33:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09AA56B0290; Sun, 12 Jun 2022 14:33:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E14328D0130 for ; Sun, 12 Jun 2022 14:33:13 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id BA0808058D for ; Sun, 12 Jun 2022 18:33:13 +0000 (UTC) X-FDA: 79570431066.04.DA56325 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 1C03640086 for ; Sun, 12 Jun 2022 18:33:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655058792; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1KWoYC2W2EPAu23za5OyAVoq/Bvw6gCidkrnWpWLjEc=; b=Fl/l65QHH5e6Pw6G9j0t2p1Ln3Pvm2vajSQTJr/BOocqIxb+kTZgIXxOndjHG8S9FeeW3L mO1RPAischH06kz+XcRCbI8kplzU9MBmBAmQ/uP0WNOj63ToETlVJBB9Fs2g4mAsqfTroF fDg3NnzGZTVAzWpvEWNZ/Kj23pW16HY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-588-IgY68YF-MSqy0Ejb3SgVSg-1; Sun, 12 Jun 2022 14:33:09 -0400 X-MC-Unique: IgY68YF-MSqy0Ejb3SgVSg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DB8E7185A7A4; Sun, 12 Jun 2022 18:33:08 +0000 (UTC) Received: from llong.com (unknown [10.22.8.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id A07434010E32; Sun, 12 Jun 2022 18:33:08 +0000 (UTC) From: Waiman Long To: Catalin Marinas , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Waiman Long Subject: [PATCH 1/3] mm/kmemleak: Use _irq lock/unlock variants in kmemleak_scan/_clear() Date: Sun, 12 Jun 2022 14:32:59 -0400 Message-Id: <20220612183301.981616-2-longman@redhat.com> In-Reply-To: <20220612183301.981616-1-longman@redhat.com> References: <20220612183301.981616-1-longman@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655058793; a=rsa-sha256; cv=none; b=IlsZOtlhsJCKN04W7ex7QZncYenge8NRXZtba4vZQYM02xQ+N9NWfG0mdaNlRe2ImDEbbY 5bH+AGzZqA6uymdzdoPbiofOFDS6gkYTeZSliR4mtQjHnXi+QN6KiKUkHx/dWpFAuquvoD 4gEAf7PouF6/By55MOxXs/95zwniIck= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Fl/l65QH"; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf04.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=longman@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655058793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1KWoYC2W2EPAu23za5OyAVoq/Bvw6gCidkrnWpWLjEc=; b=YZH5UluoIKKEyhcPSYzFCkB+7XD9WH7RQ+pTwjE4/KN1oO3Dbq6tj1b5toN1mbSkmHarHX kjXT+SazDCZWpfKqTHYd5aXOAtlDlx37rEcgRKxWNIqCyNATS2NIbIabIljMyZJqpX0eRK 287wcthQV6YjV9we7lVE3ZWP/FCskaY= X-Rspamd-Queue-Id: 1C03640086 X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Fl/l65QH"; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf04.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=longman@redhat.com X-Stat-Signature: 8a4oojkf1yybxeeierfci9ybghiw39gk X-Rspamd-Server: rspam02 X-HE-Tag: 1655058792-202465 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The kmemleak_scan() function is called only from the kmemleak scan thread or from write to the kmemleak debugfs file. Both are in task context and so we can directly use the simpler _irq() lock/unlock calls instead of the more complex _irqsave/_irqrestore variants. Similarly, kmemleak_clear() is called only from write to the kmemleak debugfs file. The same change can be applied. Signed-off-by: Waiman Long Reviewed-by: Muchun Song Reviewed-by: Catalin Marinas --- mm/kmemleak.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index a182f5ddaf68..dad9219c972c 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1413,7 +1413,6 @@ static void scan_gray_list(void) */ static void kmemleak_scan(void) { - unsigned long flags; struct kmemleak_object *object; struct zone *zone; int __maybe_unused i; @@ -1424,7 +1423,7 @@ static void kmemleak_scan(void) /* prepare the kmemleak_object's */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { - raw_spin_lock_irqsave(&object->lock, flags); + raw_spin_lock_irq(&object->lock); #ifdef DEBUG /* * With a few exceptions there should be a maximum of @@ -1441,7 +1440,7 @@ static void kmemleak_scan(void) if (color_gray(object) && get_object(object)) list_add_tail(&object->gray_list, &gray_list); - raw_spin_unlock_irqrestore(&object->lock, flags); + raw_spin_unlock_irq(&object->lock); } rcu_read_unlock(); @@ -1509,14 +1508,14 @@ static void kmemleak_scan(void) */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { - raw_spin_lock_irqsave(&object->lock, flags); + raw_spin_lock_irq(&object->lock); if (color_white(object) && (object->flags & OBJECT_ALLOCATED) && update_checksum(object) && get_object(object)) { /* color it gray temporarily */ object->count = object->min_count; list_add_tail(&object->gray_list, &gray_list); } - raw_spin_unlock_irqrestore(&object->lock, flags); + raw_spin_unlock_irq(&object->lock); } rcu_read_unlock(); @@ -1536,7 +1535,7 @@ static void kmemleak_scan(void) */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { - raw_spin_lock_irqsave(&object->lock, flags); + raw_spin_lock_irq(&object->lock); if (unreferenced_object(object) && !(object->flags & OBJECT_REPORTED)) { object->flags |= OBJECT_REPORTED; @@ -1546,7 +1545,7 @@ static void kmemleak_scan(void) new_leaks++; } - raw_spin_unlock_irqrestore(&object->lock, flags); + raw_spin_unlock_irq(&object->lock); } rcu_read_unlock(); @@ -1748,15 +1747,14 @@ static int dump_str_object_info(const char *str) static void kmemleak_clear(void) { struct kmemleak_object *object; - unsigned long flags; rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { - raw_spin_lock_irqsave(&object->lock, flags); + raw_spin_lock_irq(&object->lock); if ((object->flags & OBJECT_REPORTED) && unreferenced_object(object)) __paint_it(object, KMEMLEAK_GREY); - raw_spin_unlock_irqrestore(&object->lock, flags); + raw_spin_unlock_irq(&object->lock); } rcu_read_unlock(); From patchwork Sun Jun 12 18:33:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12878772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAE23CCA473 for ; Sun, 12 Jun 2022 18:33:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CB6D8D0133; Sun, 12 Jun 2022 14:33:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DDF18D0131; Sun, 12 Jun 2022 14:33:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5081F8D0133; Sun, 12 Jun 2022 14:33:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 37B6B8D0131 for ; Sun, 12 Jun 2022 14:33:16 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 155F420953 for ; Sun, 12 Jun 2022 18:33:16 +0000 (UTC) X-FDA: 79570431192.24.EBA967A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 70CE914008B for ; Sun, 12 Jun 2022 18:33:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655058794; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XR7KB+0VWMpvWvXRkdA4FDcU+xomfv79CCDTuY4yKdw=; b=eHh1/SUJkNPuY7S+1L5PtV91PM4iSLXcO6qkBTiqcOrxXVfwfql+XNyUOZobe3wjgahSFJ +PPpcN/UbX5O5BMazYJ7ZPhCUDEkNbJyMNyyug+Y/g1iaTans1t+uxezyxMgiXr8gpAWu/ K/14ZuGJxbu+P5oSaKvNevKYrqggdVQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-373-XB4a8fA1NSuW2fTRQfxo-Q-1; Sun, 12 Jun 2022 14:33:09 -0400 X-MC-Unique: XB4a8fA1NSuW2fTRQfxo-Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2B9133C02B7F; Sun, 12 Jun 2022 18:33:09 +0000 (UTC) Received: from llong.com (unknown [10.22.8.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id E76E440C1247; Sun, 12 Jun 2022 18:33:08 +0000 (UTC) From: Waiman Long To: Catalin Marinas , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Waiman Long Subject: [PATCH 2/3] mm/kmemleak: Skip unlikely objects in kmemleak_scan() without taking lock Date: Sun, 12 Jun 2022 14:33:00 -0400 Message-Id: <20220612183301.981616-3-longman@redhat.com> In-Reply-To: <20220612183301.981616-1-longman@redhat.com> References: <20220612183301.981616-1-longman@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655058795; a=rsa-sha256; cv=none; b=CEAbIXTN9fssELmJ47WDw/VrmdN1B54jWTJk0RBiPfppndDUNZyJ/PcfDOMv2U34MoSDsU aluGqvhiIzz+aezF6xDJObnh+VO5oTWmUy0D/nVgNnxAtz1Payqmx/PHk+PATJn5EMFmRY e2AZlq++2ecLWvZ1VahFYwRsAhD9vjE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="eHh1/SUJ"; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf23.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655058795; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XR7KB+0VWMpvWvXRkdA4FDcU+xomfv79CCDTuY4yKdw=; b=8ipVJ2xBhuf9ZAKqZy/vQyMdPLjnposMnBHEJy85/Nar+GDmOOasvSoPyeugP3qy4L/JA8 MVGMwJf0iP+lufbaei8YbUCY60CuUINSLARXLma2NNEmgwds9iRAnDhI5U5xJAmw0ZprH2 vcBWUg68JBR/UcS9RegmiWH1MZVLuAk= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 70CE914008B Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="eHh1/SUJ"; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf23.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com X-Stat-Signature: 5po59eowokiyqtbzyzifmm13ocdhqq8j X-HE-Tag: 1655058795-288117 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are 3 RCU-based object iteration loops in kmemleak_scan(). Because of the need to take RCU read lock, we can't insert cond_resched() into the loop like other parts of the function. As there can be millions of objects to be scanned, it takes a while to iterate all of them. The kmemleak functionality is usually enabled in a debug kernel which is much slower than a non-debug kernel. With sufficient number of kmemleak objects, the time to iterate them all may exceed 22s causing soft lockup. watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kmemleak:625] In this particular bug report, the soft lockup happen in the 2nd iteration loop. In the 2nd and 3rd loops, most of the objects are checked and then skipped under the object lock. Only a selected fews are modified. Those objects certainly need lock protection. However, the lock/unlock operation is slow especially with interrupt disabling and enabling included. We can actually do some basic check like color_white() without taking the lock and skip the object accordingly. Of course, this kind of check is racy and may miss objects that are being modified concurrently. The cost of missed objects, however, is just that they will be discovered in the next scan instead. The advantage of doing so is that iteration can be done much faster especially with LOCKDEP enabled in a debug kernel. With a debug kernel running on a 2-socket 96-thread x86-64 system (HZ=1000), the 2nd and 3rd iteration loops speedup with this patch on the first kmemleak_scan() call after bootup is shown in the table below. Before patch After patch Loop # # of objects Elapsed time # of objects Elapsed time ------ ------------ ------------ ------------ ------------ 2 2,599,850 2.392s 2,596,364 0.266s 3 2,600,176 2.171s 2,597,061 0.260s This patch reduces loop iteration times by about 88%. This will greatly reduce the chance of a soft lockup happening in the 2nd or 3rd iteration loops. Signed-off-by: Waiman Long Reviewed-by: Catalin Marinas --- mm/kmemleak.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index dad9219c972c..7dd64139a7c7 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1508,6 +1508,13 @@ static void kmemleak_scan(void) */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { + /* + * This is racy but we can save the overhead of lock/unlock + * calls. The missed objects, if any, should be caught in + * the next scan. + */ + if (!color_white(object)) + continue; raw_spin_lock_irq(&object->lock); if (color_white(object) && (object->flags & OBJECT_ALLOCATED) && update_checksum(object) && get_object(object)) { @@ -1535,6 +1542,13 @@ static void kmemleak_scan(void) */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { + /* + * This is racy but we can save the overhead of lock/unlock + * calls. The missed objects, if any, should be caught in + * the next scan. + */ + if (!color_white(object)) + continue; raw_spin_lock_irq(&object->lock); if (unreferenced_object(object) && !(object->flags & OBJECT_REPORTED)) { From patchwork Sun Jun 12 18:33:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12878771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70B47C43334 for ; Sun, 12 Jun 2022 18:33:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 779C78D0132; Sun, 12 Jun 2022 14:33:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7124B8D0131; Sun, 12 Jun 2022 14:33:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 506B08D0132; Sun, 12 Jun 2022 14:33:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3823A6B028E for ; Sun, 12 Jun 2022 14:33:14 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 10A9D8058D for ; Sun, 12 Jun 2022 18:33:14 +0000 (UTC) X-FDA: 79570431108.23.6C6A81C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id A8FEC40088 for ; Sun, 12 Jun 2022 18:33:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655058791; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FdK93Bs6pIhqQZAR3Lfyy5PxJORXQ5zTxPlLV5rbOn4=; b=fi2dssUGB86fMZGzYSElJCqomNvEO6ZoUgBRdB6Bh8BWjhfRxci04yhncYpvZRJplhKXCL It5B/nDEc8CO6lnOZYN/J7nzIzVZLh6Y41JrMEYRtFN1KC7yijVAoGmdONnrOoRBT5UP+X PXetb+6iIHlrMoiowE0YW7pBczcxdM0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-262-S9FGuFeAOLaowJXG-QadEg-1; Sun, 12 Jun 2022 14:33:09 -0400 X-MC-Unique: S9FGuFeAOLaowJXG-QadEg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6D5B7101A54E; Sun, 12 Jun 2022 18:33:09 +0000 (UTC) Received: from llong.com (unknown [10.22.8.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3729A40D282F; Sun, 12 Jun 2022 18:33:09 +0000 (UTC) From: Waiman Long To: Catalin Marinas , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Waiman Long Subject: [PATCH 3/3] mm/kmemleak: Prevent soft lockup in first object iteration loop of kmemleak_scan() Date: Sun, 12 Jun 2022 14:33:01 -0400 Message-Id: <20220612183301.981616-4-longman@redhat.com> In-Reply-To: <20220612183301.981616-1-longman@redhat.com> References: <20220612183301.981616-1-longman@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655058793; a=rsa-sha256; cv=none; b=20QiBGIS/5eokpsj/+VByXPYohGB+bdCLC1yufwSxIOFeJvbACz2ve0VXsyzzqQ2i2N78r QTFtbHrF+5AaQ/Q7Wfk6gMWk8qb5OrqKP2KHqJ/BSi+kJIAo/o+1Qp3qNGR6W6VumtGcQM 4l6j2S/bFc78n9/TDkwpU4lq6042lnI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fi2dssUG; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf27.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=longman@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655058793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FdK93Bs6pIhqQZAR3Lfyy5PxJORXQ5zTxPlLV5rbOn4=; b=hj8efCxvK/Kj4q5FWufEAhHblXOpr8sAg4WKMUtzg5svbcHqyB0QElZ0WWN9iPCBHAkCSq gdiGG4EXrqFiOpLSTwSn8M4wJ47NcyyvcJWdQUgCJd5z7BEb3HrG203KfWdfSTMSKKP4sI KKNzv43thjJAK+SQqFjH/tNVEIfWLto= X-Stat-Signature: bgh7pi1hdkpr6rcgw4resei5urnfiwby X-Rspam-User: X-Rspamd-Queue-Id: A8FEC40088 X-Rspamd-Server: rspam07 Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fi2dssUG; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf27.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=longman@redhat.com X-HE-Tag: 1655058793-708398 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The first RCU-based object iteration loop has to put almost all the objects into the gray list and so cannot skip taking the object lock. One way to avoid soft lockup is to insert occasional cond_resched() into the loop. This cannot be done while holding the RCU read lock which is to protect objects from removal. However, putting an object into the gray list means getting a reference to the object. That will prevent the object from removal as well without the need to hold the RCU read lock. So insert a cond_resched() call after every 64k objects are put into the gray list. Signed-off-by: Waiman Long --- mm/kmemleak.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 7dd64139a7c7..a7c42e134fa1 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1417,12 +1417,15 @@ static void kmemleak_scan(void) struct zone *zone; int __maybe_unused i; int new_leaks = 0; + int gray_list_cnt = 0; jiffies_last_scan = jiffies; /* prepare the kmemleak_object's */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { + bool object_pinned = false; + raw_spin_lock_irq(&object->lock); #ifdef DEBUG /* @@ -1437,10 +1440,25 @@ static void kmemleak_scan(void) #endif /* reset the reference count (whiten the object) */ object->count = 0; - if (color_gray(object) && get_object(object)) + if (color_gray(object) && get_object(object)) { list_add_tail(&object->gray_list, &gray_list); + gray_list_cnt++; + object_pinned = true; + } raw_spin_unlock_irq(&object->lock); + + /* + * With object pinned by a positive reference count, it + * won't go away and we can safely release the RCU read + * lock and do a cond_resched() to avoid soft lockup every + * 64k objects. + */ + if (object_pinned && !(gray_list_cnt & 0xffff)) { + rcu_read_unlock(); + cond_resched(); + rcu_read_lock(); + } } rcu_read_unlock();