From patchwork Wed Jan 9 17:23:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 10754613 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C97476C2 for ; Wed, 9 Jan 2019 17:24:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE10A2934D for ; Wed, 9 Jan 2019 17:24:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A1D3A29354; Wed, 9 Jan 2019 17:24:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 49B922934D for ; Wed, 9 Jan 2019 17:24:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726914AbfAIRYY (ORCPT ); Wed, 9 Jan 2019 12:24:24 -0500 Received: from mx1.redhat.com ([209.132.183.28]:47706 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726275AbfAIRYX (ORCPT ); Wed, 9 Jan 2019 12:24:23 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 55D7DCA396; Wed, 9 Jan 2019 17:24:23 +0000 (UTC) Received: from llong.com (dhcp-17-223.bos.redhat.com [10.18.17.223]) by smtp.corp.redhat.com (Postfix) with ESMTP id D95C0600CD; Wed, 9 Jan 2019 17:24:21 +0000 (UTC) From: Waiman Long To: Andrew Morton , Alexey Dobriyan , Kees Cook , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Davidlohr Bueso , Miklos Szeredi , Daniel Colascione , Dave Chinner , Randy Dunlap , Matthew Wilcox , Waiman Long Subject: [PATCH v2 1/4] /proc/stat: Extract irqs counting code into show_stat_irqs() Date: Wed, 9 Jan 2019 12:23:45 -0500 Message-Id: <1547054628-12703-2-git-send-email-longman@redhat.com> In-Reply-To: <1547054628-12703-1-git-send-email-longman@redhat.com> References: <1547054628-12703-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Wed, 09 Jan 2019 17:24:23 +0000 (UTC) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The code that generates the "intr" line of /proc/stat is now moved from show_stat() into a new function - show_stat_irqs(). There is no functional change. Signed-off-by: Waiman Long Reviewed-by: Kees Cook --- fs/proc/stat.c | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/fs/proc/stat.c b/fs/proc/stat.c index 535eda7..4b06f1b 100644 --- a/fs/proc/stat.c +++ b/fs/proc/stat.c @@ -79,12 +79,38 @@ static u64 get_iowait_time(int cpu) #endif +static u64 compute_stat_irqs_sum(void) +{ + int i; + u64 sum = 0; + + for_each_possible_cpu(i) { + sum += kstat_cpu_irqs_sum(i); + sum += arch_irq_stat_cpu(i); + } + sum += arch_irq_stat(); + return sum; +} + +/* + * Print out the "intr" line of /proc/stat. + */ +static void show_stat_irqs(struct seq_file *p) +{ + int i; + + seq_put_decimal_ull(p, "intr ", compute_stat_irqs_sum()); + for_each_irq_nr(i) + seq_put_decimal_ull(p, " ", kstat_irqs_usr(i)); + + seq_putc(p, '\n'); +} + static int show_stat(struct seq_file *p, void *v) { int i, j; u64 user, nice, system, idle, iowait, irq, softirq, steal; u64 guest, guest_nice; - u64 sum = 0; u64 sum_softirq = 0; unsigned int per_softirq_sums[NR_SOFTIRQS] = {0}; struct timespec64 boottime; @@ -105,8 +131,6 @@ static int show_stat(struct seq_file *p, void *v) steal += kcpustat_cpu(i).cpustat[CPUTIME_STEAL]; guest += kcpustat_cpu(i).cpustat[CPUTIME_GUEST]; guest_nice += kcpustat_cpu(i).cpustat[CPUTIME_GUEST_NICE]; - sum += kstat_cpu_irqs_sum(i); - sum += arch_irq_stat_cpu(i); for (j = 0; j < NR_SOFTIRQS; j++) { unsigned int softirq_stat = kstat_softirqs_cpu(j, i); @@ -115,7 +139,6 @@ static int show_stat(struct seq_file *p, void *v) sum_softirq += softirq_stat; } } - sum += arch_irq_stat(); seq_put_decimal_ull(p, "cpu ", nsec_to_clock_t(user)); seq_put_decimal_ull(p, " ", nsec_to_clock_t(nice)); @@ -154,14 +177,10 @@ static int show_stat(struct seq_file *p, void *v) seq_put_decimal_ull(p, " ", nsec_to_clock_t(guest_nice)); seq_putc(p, '\n'); } - seq_put_decimal_ull(p, "intr ", (unsigned long long)sum); - - /* sum again ? it could be updated? */ - for_each_irq_nr(j) - seq_put_decimal_ull(p, " ", kstat_irqs_usr(j)); + show_stat_irqs(p); seq_printf(p, - "\nctxt %llu\n" + "ctxt %llu\n" "btime %llu\n" "processes %lu\n" "procs_running %lu\n" From patchwork Wed Jan 9 17:23:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 10754615 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D97916C5 for ; Wed, 9 Jan 2019 17:24:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BB7C72934D for ; Wed, 9 Jan 2019 17:24:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AF79929354; Wed, 9 Jan 2019 17:24:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 55C042934D for ; Wed, 9 Jan 2019 17:24:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726962AbfAIRYl (ORCPT ); Wed, 9 Jan 2019 12:24:41 -0500 Received: from mx1.redhat.com ([209.132.183.28]:54668 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726940AbfAIRYY (ORCPT ); Wed, 9 Jan 2019 12:24:24 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 772832CD7FE; Wed, 9 Jan 2019 17:24:24 +0000 (UTC) Received: from llong.com (dhcp-17-223.bos.redhat.com [10.18.17.223]) by smtp.corp.redhat.com (Postfix) with ESMTP id 44B11600C8; Wed, 9 Jan 2019 17:24:23 +0000 (UTC) From: Waiman Long To: Andrew Morton , Alexey Dobriyan , Kees Cook , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Davidlohr Bueso , Miklos Szeredi , Daniel Colascione , Dave Chinner , Randy Dunlap , Matthew Wilcox , Waiman Long Subject: [PATCH v2 2/4] /proc/stat: Only do percpu sum of active IRQs Date: Wed, 9 Jan 2019 12:23:46 -0500 Message-Id: <1547054628-12703-3-git-send-email-longman@redhat.com> In-Reply-To: <1547054628-12703-1-git-send-email-longman@redhat.com> References: <1547054628-12703-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Wed, 09 Jan 2019 17:24:24 +0000 (UTC) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Recent computer systems may have hundreds or even thousands of IRQs available. However, most of them may not be active and their IRQ counts are zero. It is just a waste of CPU cycles to do percpu summation of those zero counts. In order to find out if an IRQ is active, we track the transition of the percpu count from 0 to 1 and atomically increment a new kstat_irq_cpus counter which counts the number of CPUs that handle this particular IRQ. The IRQ descriptor is zalloc'ed, so there is no need to initialize the new counter. On a 4-socket Broadwell server wwith 112 vCPUs and 2952 IRQs (2877 of them are 0), the system time needs to read /proc/stat 50k times was reduced from 11.200s to 8.048s. That was a execution time reduction of 28%. Signed-off-by: Waiman Long --- include/linux/irqdesc.h | 1 + kernel/irq/internals.h | 3 ++- kernel/irq/irqdesc.c | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/irqdesc.h b/include/linux/irqdesc.h index dd1e40d..86bbad2 100644 --- a/include/linux/irqdesc.h +++ b/include/linux/irqdesc.h @@ -61,6 +61,7 @@ struct irq_desc { irq_preflow_handler_t preflow_handler; #endif struct irqaction *action; /* IRQ action list */ + atomic_t kstat_irq_cpus; /* #cpus handling this IRQ */ unsigned int status_use_accessors; unsigned int core_internal_state__do_not_mess_with_it; unsigned int depth; /* nested irq disables */ diff --git a/kernel/irq/internals.h b/kernel/irq/internals.h index ca6afa2..31787c1 100644 --- a/kernel/irq/internals.h +++ b/kernel/irq/internals.h @@ -244,7 +244,8 @@ static inline void irq_state_set_masked(struct irq_desc *desc) static inline void kstat_incr_irqs_this_cpu(struct irq_desc *desc) { - __this_cpu_inc(*desc->kstat_irqs); + if (unlikely(__this_cpu_inc_return(*desc->kstat_irqs) == 1)) + atomic_inc(&desc->kstat_irq_cpus); __this_cpu_inc(kstat.irqs_sum); } diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c index ee062b7..3d2c38b 100644 --- a/kernel/irq/irqdesc.c +++ b/kernel/irq/irqdesc.c @@ -922,7 +922,7 @@ unsigned int kstat_irqs(unsigned int irq) int cpu; unsigned int sum = 0; - if (!desc || !desc->kstat_irqs) + if (!desc || !desc->kstat_irqs || !atomic_read(&desc->kstat_irq_cpus)) return 0; for_each_possible_cpu(cpu) sum += *per_cpu_ptr(desc->kstat_irqs, cpu); From patchwork Wed Jan 9 17:23:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 10754611 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0694E6C2 for ; Wed, 9 Jan 2019 17:24:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E061F2934D for ; Wed, 9 Jan 2019 17:24:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D438029354; Wed, 9 Jan 2019 17:24:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7456B2934D for ; Wed, 9 Jan 2019 17:24:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726959AbfAIRY1 (ORCPT ); Wed, 9 Jan 2019 12:24:27 -0500 Received: from mx1.redhat.com ([209.132.183.28]:33190 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726955AbfAIRY0 (ORCPT ); Wed, 9 Jan 2019 12:24:26 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DE67AC0C18A6; Wed, 9 Jan 2019 17:24:25 +0000 (UTC) Received: from llong.com (dhcp-17-223.bos.redhat.com [10.18.17.223]) by smtp.corp.redhat.com (Postfix) with ESMTP id 97885601A3; Wed, 9 Jan 2019 17:24:24 +0000 (UTC) From: Waiman Long To: Andrew Morton , Alexey Dobriyan , Kees Cook , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Davidlohr Bueso , Miklos Szeredi , Daniel Colascione , Dave Chinner , Randy Dunlap , Matthew Wilcox , Waiman Long Subject: [PATCH v2 3/4] genirq: Track the number of active IRQs Date: Wed, 9 Jan 2019 12:23:47 -0500 Message-Id: <1547054628-12703-4-git-send-email-longman@redhat.com> In-Reply-To: <1547054628-12703-1-git-send-email-longman@redhat.com> References: <1547054628-12703-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Wed, 09 Jan 2019 17:24:26 +0000 (UTC) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A new atomic variable nr_active_irqs is added to track the number of active IRQs that have one or more interrupts serviced. Signed-off-by: Waiman Long --- kernel/irq/internals.h | 7 +++++-- kernel/irq/irqdesc.c | 5 +++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/kernel/irq/internals.h b/kernel/irq/internals.h index 31787c1..239ae51 100644 --- a/kernel/irq/internals.h +++ b/kernel/irq/internals.h @@ -242,10 +242,13 @@ static inline void irq_state_set_masked(struct irq_desc *desc) #undef __irqd_to_state +extern atomic_t nr_active_irqs; /* # of active IRQs */ static inline void kstat_incr_irqs_this_cpu(struct irq_desc *desc) { - if (unlikely(__this_cpu_inc_return(*desc->kstat_irqs) == 1)) - atomic_inc(&desc->kstat_irq_cpus); + if (unlikely(__this_cpu_inc_return(*desc->kstat_irqs) == 1)) { + if (atomic_inc_return(&desc->kstat_irq_cpus) == 1) + atomic_inc(&nr_active_irqs); + } __this_cpu_inc(kstat.irqs_sum); } diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c index 3d2c38b..7e00c4d 100644 --- a/kernel/irq/irqdesc.c +++ b/kernel/irq/irqdesc.c @@ -20,6 +20,11 @@ #include "internals.h" /* + * Number of active IRQs + */ +atomic_t nr_active_irqs; + +/* * lockdep: we want to handle all irq_desc locks as a single lock-class: */ static struct lock_class_key irq_desc_lock_class; From patchwork Wed Jan 9 17:23:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 10754609 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 11B7C6C2 for ; Wed, 9 Jan 2019 17:24:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB5A52934D for ; Wed, 9 Jan 2019 17:24:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DE0C629354; Wed, 9 Jan 2019 17:24:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77EA42934D for ; Wed, 9 Jan 2019 17:24:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726973AbfAIRY2 (ORCPT ); Wed, 9 Jan 2019 12:24:28 -0500 Received: from mx1.redhat.com ([209.132.183.28]:14775 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726962AbfAIRY1 (ORCPT ); Wed, 9 Jan 2019 12:24:27 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 51C19C7A06; Wed, 9 Jan 2019 17:24:27 +0000 (UTC) Received: from llong.com (dhcp-17-223.bos.redhat.com [10.18.17.223]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0747C600C8; Wed, 9 Jan 2019 17:24:25 +0000 (UTC) From: Waiman Long To: Andrew Morton , Alexey Dobriyan , Kees Cook , Thomas Gleixner Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Davidlohr Bueso , Miklos Szeredi , Daniel Colascione , Dave Chinner , Randy Dunlap , Matthew Wilcox , Waiman Long Subject: [PATCH v2 4/4] /proc/stat: Call kstat_irqs_usr() only for active IRQs Date: Wed, 9 Jan 2019 12:23:48 -0500 Message-Id: <1547054628-12703-5-git-send-email-longman@redhat.com> In-Reply-To: <1547054628-12703-1-git-send-email-longman@redhat.com> References: <1547054628-12703-1-git-send-email-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Wed, 09 Jan 2019 17:24:27 +0000 (UTC) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP After skipping the percpu summation of non-active IRQs on a 4-socket Broadwell system with about 3k IRQs, about half of the CPU cycles were spent in the kstat_irqs() call. The majority of which were used to look up the IRQ descriptors for the corresponding IRQ numbers. We can recoup a lot of those lost cycles by calling kstat_irqs_usr() only for those IRQs that are active. A bitmap is now used to keep track of the list of the active IRQs. Changes in nr_active_irqs count will cause the code to rescan all the IRQs and repopulate the bitmap. On the same 4-socket server, the introduction of this patch further reduces the system time of reading /proc/stat 5k times from 8.048s to 5.817s. This is a another time reduction of 28%. Signed-off-by: Waiman Long --- fs/proc/stat.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/fs/proc/stat.c b/fs/proc/stat.c index 4b06f1b..5e2a398 100644 --- a/fs/proc/stat.c +++ b/fs/proc/stat.c @@ -93,6 +93,25 @@ static u64 compute_stat_irqs_sum(void) } /* + * Write the given number of space separated '0' into the sequence file. + */ +static void write_zeros(struct seq_file *p, int cnt) +{ + /* String of 16 '0's */ + static const char zeros[] = " 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0"; + + while (cnt > 0) { + if (cnt >= 16) { + seq_write(p, zeros, 32); + cnt -= 16; + } else { + seq_write(p, zeros, 2 * cnt); + cnt = 0; + } + } +} + +/* * Print out the "intr" line of /proc/stat. */ static void show_stat_irqs(struct seq_file *p) @@ -100,9 +119,74 @@ static void show_stat_irqs(struct seq_file *p) int i; seq_put_decimal_ull(p, "intr ", compute_stat_irqs_sum()); + + if (IS_ENABLED(CONFIG_SMP) && (nr_cpu_ids >= 10) && (nr_irqs >= 256)) { + /* + * On systems with 10 or more CPUs and 256 or more IRQs, + * we used a bitmap to keep track of the number of active + * IRQs and call kstat_irqs_usr() only for those IRQs. + * The bitmap will be refreshed whenever nr_active_irqs + * changes. + */ + extern atomic_t nr_active_irqs; + static DEFINE_MUTEX(irqs_mutex); + static int last_irq = -1; + static int bitmap_size, active_irqs; + static unsigned long *bitmap; + int current_irqs = atomic_read(&nr_active_irqs); + + mutex_lock(&irqs_mutex); + if (current_irqs != active_irqs) { + /* + * Rescan all the IRQs for active ones. + */ + if (nr_irqs > bitmap_size) { + static unsigned long *new_bitmap; + static int new_size; + + new_size = BITS_TO_LONGS(nr_irqs)*sizeof(long); + new_bitmap = (unsigned long *)krealloc(bitmap, + new_size, GFP_KERNEL); + if (!new_bitmap) + goto fallback; + bitmap = new_bitmap; + bitmap_size = new_size; + } + memset(bitmap, 0, bitmap_size/BITS_PER_BYTE); + last_irq = 0; + for_each_irq_nr(i) { + int cnt = kstat_irqs_usr(i); + + if (cnt) { + bitmap_set(bitmap, 0, i); + last_irq = i; + } + seq_put_decimal_ull(p, " ", cnt); + } + active_irqs = current_irqs; + mutex_unlock(&irqs_mutex); + goto out; + } + /* + * Retrieve counts from active IRQs only. + */ + for (i = 0; i <= last_irq; i++) { + int next = find_next_bit(bitmap, last_irq + 1, i); + + if (next > i) + write_zeros(p, next - i); + i = next; + seq_put_decimal_ull(p, " ", kstat_irqs_usr(i)); + } + mutex_unlock(&irqs_mutex); + write_zeros(p, nr_irqs - i); + goto out; + } +fallback: for_each_irq_nr(i) seq_put_decimal_ull(p, " ", kstat_irqs_usr(i)); +out: seq_putc(p, '\n'); }