From patchwork Sat Nov 12 23:02:35 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 9424515 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6D98160512 for ; Sat, 12 Nov 2016 23:02:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6268128DB3 for ; Sat, 12 Nov 2016 23:02:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5755928DBA; Sat, 12 Nov 2016 23:02:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, URIBL_BLACK autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 04FE928DB3 for ; Sat, 12 Nov 2016 23:02:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751382AbcKLXCt (ORCPT ); Sat, 12 Nov 2016 18:02:49 -0500 Received: from bombadil.infradead.org ([198.137.202.9]:36178 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751088AbcKLXCt (ORCPT ); Sat, 12 Nov 2016 18:02:49 -0500 Received: from [46.120.250.42] (helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.85_2 #1 (Red Hat Linux)) id 1c5hJw-0006aH-3P; Sat, 12 Nov 2016 23:02:48 +0000 From: Sagi Grimberg To: Jens Axboe , linux-block@vger.kernel.org Cc: Christoph Hellwig Subject: [PATCH 3/3] irq-poll: Reduce local_irq_save/restore operations in irq_poll_softirq Date: Sun, 13 Nov 2016 01:02:35 +0200 Message-Id: <1478991755-28153-4-git-send-email-sagi@grimberg.me> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1478991755-28153-1-git-send-email-sagi@grimberg.me> References: <1478991755-28153-1-git-send-email-sagi@grimberg.me> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP splice to a local list (and splice back when done) so we won't need to enable/disable local_irq in each iteration. Signed-off-by: Sagi Grimberg --- lib/irq_poll.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/lib/irq_poll.c b/lib/irq_poll.c index 44a5d1da4260..dc4c7ace9b41 100644 --- a/lib/irq_poll.c +++ b/lib/irq_poll.c @@ -75,13 +75,16 @@ EXPORT_SYMBOL(irq_poll_complete); static void __latent_entropy irq_poll_softirq(struct softirq_action *h) { - struct list_head *list = this_cpu_ptr(&blk_cpu_iopoll); + struct list_head *iop_list = this_cpu_ptr(&blk_cpu_iopoll); int rearm = 0, budget = irq_poll_budget; unsigned long start_time = jiffies; + LIST_HEAD(list); local_irq_disable(); + list_splice_init(iop_list, &list); + local_irq_enable(); - while (!list_empty(list)) { + while (!list_empty(&list)) { struct irq_poll *iop; int work, weight; @@ -93,14 +96,7 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) break; } - local_irq_enable(); - - /* Even though interrupts have been re-enabled, this - * access is safe because interrupts can only add new - * entries to the tail of this list, and only ->poll() - * calls can remove this head entry from the list. - */ - iop = list_entry(list->next, struct irq_poll, list); + iop = list_first_entry(&list, struct irq_poll, list); weight = iop->weight; work = 0; @@ -109,8 +105,6 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) budget -= work; - local_irq_disable(); - /* * Drivers must not modify the iopoll state, if they * consume their assigned weight (or more, some drivers can't @@ -120,13 +114,20 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) * move the instance around on the list at-will. */ if (work >= weight) { - if (unlikely(test_bit(IRQ_POLL_F_DISABLE, &iop->state))) + if (unlikely(test_bit(IRQ_POLL_F_DISABLE, &iop->state))) { + local_irq_disable(); __irq_poll_complete(iop); - else - list_move_tail(&iop->list, list); + local_irq_enable(); + } else { + list_move_tail(&iop->list, &list); + } } } + local_irq_disable(); + list_splice_tail_init(iop_list, &list); + list_splice(&list, iop_list); + if (rearm) __raise_softirq_irqoff(IRQ_POLL_SOFTIRQ);