From patchwork Sat Jan 23 20:10:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 12041925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F5DBC433DB for ; Sat, 23 Jan 2021 20:12:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 67D6922EBE for ; Sat, 23 Jan 2021 20:12:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726495AbhAWULW (ORCPT ); Sat, 23 Jan 2021 15:11:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726249AbhAWULT (ORCPT ); Sat, 23 Jan 2021 15:11:19 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 187C5C06174A; Sat, 23 Jan 2021 12:10:39 -0800 (PST) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1611432635; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RvialIUza76ZQsVU543wsyBevXPATQcHABCWvzhOsa0=; b=WvSBCSb7AKFnNe0oNkxslbemJCZo+5Wn3G8cPxu2WQHbSkRcsMxQ7zXSEi+mrqAjE3e5bP sirdN98jEqRvdfI97P28vkHMaiwZzr1+ha3bVFW+7oRtJOwKfS5oDElKJf0u0+HfuRHb38 ULM9Vc84IakrEMl0pwxTXqnFzUCALQl+Nxw4uCJc/jaYtgKg9M5SaxSevVzsVm1cWZ4cWq QKFbmRG9TRcxVHKCNxHyZuRS2fnLPuo4IMJENO1Ee3FDZpzBMxeMytVRql7ox+g1q11bld eTehjLw9gVnl0QAYMmbOpCj/xS1Z7vmjIVJGzLX4M55mO6/Yr8Nrnw4cRhTXng== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1611432635; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RvialIUza76ZQsVU543wsyBevXPATQcHABCWvzhOsa0=; b=n5lkVAuRVZqteTCZ7UWJtYsIMymG6Zjw0vi4qtRMTPqoCQw4OFzkFApI6HCVCWb74H2qAx cGgbT30Jxat9iyAg== To: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe , Thomas Gleixner , Peter Zijlstra , Ingo Molnar , Sebastian Andrzej Siewior Subject: [PATCH 1/3] smp: Process pending softirqs in flush_smp_call_function_from_idle() Date: Sat, 23 Jan 2021 21:10:25 +0100 Message-Id: <20210123201027.3262800-2-bigeasy@linutronix.de> In-Reply-To: <20210123201027.3262800-1-bigeasy@linutronix.de> References: <20210123201027.3262800-1-bigeasy@linutronix.de> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org send_call_function_single_ipi() may wake an idle CPU without sending an IPI. The woken up CPU will process the SMP-functions in flush_smp_call_function_from_idle(). Any raised softirq from within the SMP-function call will not be processed. Should the CPU have no tasks assigned, then it will go back to idle with pending softirqs and the NOHZ will rightfully complain. Process pending softirqs on return from flush_smp_call_function_queue(). Fixes: b2a02fc43a1f4 ("smp: Optimize send_call_function_single_ipi()") Reported-by: Jens Axboe Signed-off-by: Sebastian Andrzej Siewior --- kernel/smp.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/smp.c b/kernel/smp.c index 1b6070bf97bb0..aeb0adfa06063 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -449,6 +450,9 @@ void flush_smp_call_function_from_idle(void) local_irq_save(flags); flush_smp_call_function_queue(true); + if (local_softirq_pending()) + do_softirq(); + local_irq_restore(flags); } From patchwork Sat Jan 23 20:10:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 12041919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E15CDC433E0 for ; Sat, 23 Jan 2021 20:11:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A3E9E22CF6 for ; Sat, 23 Jan 2021 20:11:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726480AbhAWULU (ORCPT ); Sat, 23 Jan 2021 15:11:20 -0500 Received: from Galois.linutronix.de ([193.142.43.55]:33390 "EHLO galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726363AbhAWULS (ORCPT ); Sat, 23 Jan 2021 15:11:18 -0500 From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1611432636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FXtKHqimthZytcdJDbVm1RJiJEFrCp7q87+tAWESN50=; b=Xpr0UAX3SBHng2S2PsZcHE9cil5C2zfy62SMwHQ9oana5hWH2FqnXJFUjplVYbrPnc5c9V alABo9w6jnrrt+P3AWNKa5EUbb8GgzHTGZMd0gpalXhQhQlRWoB/E/pgTs4ok7T0aimbrV AUwJrDBw0V11RrTn4DXgGAolTAB4TKLYsDwawcrTKAjWHfzpqYA8FY7ULRH96pXTym5Gh4 CE2Fp6s0EK8dYMkCl7gzYwHpNaJDTBDms3rmhJJFpGfVjKzvbsjigGFlQeaX/7hFD3ys8Q Ocj00S7+6RK3ierIm8ChUAtTRB3QrvaekQU6Izlz3l0ztPLRZST68YwVsTbVOg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1611432636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FXtKHqimthZytcdJDbVm1RJiJEFrCp7q87+tAWESN50=; b=ouIsL2O14nSxYq0yflXVMdGLkZdVAg5NqvPVmOBSlT66bALlbVhPk8ezeM98DO7ePm+t3q oBljjHyZQ/S5G0DQ== To: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe , Thomas Gleixner , Peter Zijlstra , Ingo Molnar , Sebastian Andrzej Siewior Subject: [PATCH 2/3] blk-mq: Always complete remote completions requests in softirq Date: Sat, 23 Jan 2021 21:10:26 +0100 Message-Id: <20210123201027.3262800-3-bigeasy@linutronix.de> In-Reply-To: <20210123201027.3262800-1-bigeasy@linutronix.de> References: <20210123201027.3262800-1-bigeasy@linutronix.de> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Controllers with multiple queues have their IRQ-handelers pinned to a CPU. The core shouldn't need to complete the request on a remote CPU. Remove this case and always raise the softirq to complete the request. Signed-off-by: Sebastian Andrzej Siewior Reviewed-by: Christoph Hellwig Reviewed-by: Daniel Wagner --- block/blk-mq.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index f285a9123a8b0..90348ae518461 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -628,19 +628,7 @@ static void __blk_mq_complete_request_remote(void *data) { struct request *rq = data; - /* - * For most of single queue controllers, there is only one irq vector - * for handling I/O completion, and the only irq's affinity is set - * to all possible CPUs. On most of ARCHs, this affinity means the irq - * is handled on one specific CPU. - * - * So complete I/O requests in softirq context in case of single queue - * devices to avoid degrading I/O performance due to irqsoff latency. - */ - if (rq->q->nr_hw_queues == 1) - blk_mq_trigger_softirq(rq); - else - rq->q->mq_ops->complete(rq); + blk_mq_trigger_softirq(rq); } static inline bool blk_mq_complete_need_ipi(struct request *rq) From patchwork Sat Jan 23 20:10:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 12041921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3C3AC433DB for ; Sat, 23 Jan 2021 20:11:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C66D622EBE for ; Sat, 23 Jan 2021 20:11:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726513AbhAWULY (ORCPT ); Sat, 23 Jan 2021 15:11:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726445AbhAWULT (ORCPT ); Sat, 23 Jan 2021 15:11:19 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2737FC061786; Sat, 23 Jan 2021 12:10:39 -0800 (PST) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1611432636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SVYDIMz7sqoQeFj5ojU5tkpHrAwzSR0G0FSyZvSe3wY=; b=w5Tx+S15bTtI6ER/EJeVNbk+COnOzPAy9FBdSVlEk+4MKuONF7Hwy9Ho1EMbenLmyd6sw7 iLI5IS6gjb+QNugjVBIPvr0gC8OygA4PU32725CuMr6+3sxgZZcbQ4s9AdUnq31nSZdeUl ufYyo6a4Bj08ntz//63/rCG6QFt/qlbYJfkMj/i2M/bR0/+We5razNsCagCigMNSNJAnVA fhbOA5Ffc7Zg5R6XJlIfI5QIkukPHn5ddyTn/H4r7wjoQBitoemboz8wtnPn6BFK/oai44 3bYo47L5LS1I24R8TJKZynVZ6ZC5bQ6c/MaSgkUrAEscDcTz6zPsIVoSBHrSRw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1611432636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SVYDIMz7sqoQeFj5ojU5tkpHrAwzSR0G0FSyZvSe3wY=; b=i00Z+0eafgm7IQRt890HjnQLBhE6X8818h1Pex+jRwg53MDzfuoSzJC2842gm2AhAyfY5y zG5uGEE4RrmOw7Ag== To: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe , Thomas Gleixner , Peter Zijlstra , Ingo Molnar , Sebastian Andrzej Siewior Subject: [PATCH 3/3] blk-mq: Use llist_head for blk_cpu_done Date: Sat, 23 Jan 2021 21:10:27 +0100 Message-Id: <20210123201027.3262800-4-bigeasy@linutronix.de> In-Reply-To: <20210123201027.3262800-1-bigeasy@linutronix.de> References: <20210123201027.3262800-1-bigeasy@linutronix.de> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org With llist_head it is possible to avoid the locking (the irq-off region) when items are added. This makes it possible to add items on a remote CPU without additional locking. llist_add() returns true if the list was previously empty. This can be used to invoke the SMP function call / raise sofirq only if the first item was added (otherwise it is already pending). This simplifies the code a little and reduces the IRQ-off regions. blk_mq_raise_softirq() needs a preempt-disable section to ensure the request is enqueued on the same CPU as the softirq is raised. Some callers (USB-storage) invoke this path in preemptible context. Signed-off-by: Sebastian Andrzej Siewior Reviewed-by: Christoph Hellwig --- block/blk-mq.c | 97 ++++++++++++++++++------------------------ include/linux/blkdev.h | 2 +- 2 files changed, 42 insertions(+), 57 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 90348ae518461..463de2981df8a 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -41,7 +41,7 @@ #include "blk-mq-sched.h" #include "blk-rq-qos.h" -static DEFINE_PER_CPU(struct list_head, blk_cpu_done); +static DEFINE_PER_CPU(struct llist_head, blk_cpu_done); static void blk_mq_poll_stats_start(struct request_queue *q); static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb); @@ -567,68 +567,29 @@ void blk_mq_end_request(struct request *rq, blk_status_t error) } EXPORT_SYMBOL(blk_mq_end_request); -/* - * Softirq action handler - move entries to local list and loop over them - * while passing them to the queue registered handler. - */ -static __latent_entropy void blk_done_softirq(struct softirq_action *h) +static void blk_complete_reqs(struct llist_head *list) { - struct list_head *cpu_list, local_list; + struct llist_node *entry = llist_reverse_order(llist_del_all(list)); + struct request *rq, *next; - local_irq_disable(); - cpu_list = this_cpu_ptr(&blk_cpu_done); - list_replace_init(cpu_list, &local_list); - local_irq_enable(); - - while (!list_empty(&local_list)) { - struct request *rq; - - rq = list_entry(local_list.next, struct request, ipi_list); - list_del_init(&rq->ipi_list); + llist_for_each_entry_safe(rq, next, entry, ipi_list) rq->q->mq_ops->complete(rq); - } } -static void blk_mq_trigger_softirq(struct request *rq) +static __latent_entropy void blk_done_softirq(struct softirq_action *h) { - struct list_head *list; - unsigned long flags; - - local_irq_save(flags); - list = this_cpu_ptr(&blk_cpu_done); - list_add_tail(&rq->ipi_list, list); - - /* - * If the list only contains our just added request, signal a raise of - * the softirq. If there are already entries there, someone already - * raised the irq but it hasn't run yet. - */ - if (list->next == &rq->ipi_list) - raise_softirq_irqoff(BLOCK_SOFTIRQ); - local_irq_restore(flags); + blk_complete_reqs(this_cpu_ptr(&blk_cpu_done)); } static int blk_softirq_cpu_dead(unsigned int cpu) { - /* - * If a CPU goes away, splice its entries to the current CPU - * and trigger a run of the softirq - */ - local_irq_disable(); - list_splice_init(&per_cpu(blk_cpu_done, cpu), - this_cpu_ptr(&blk_cpu_done)); - raise_softirq_irqoff(BLOCK_SOFTIRQ); - local_irq_enable(); - + blk_complete_reqs(&per_cpu(blk_cpu_done, cpu)); return 0; } - static void __blk_mq_complete_request_remote(void *data) { - struct request *rq = data; - - blk_mq_trigger_softirq(rq); + __raise_softirq_irqoff(BLOCK_SOFTIRQ); } static inline bool blk_mq_complete_need_ipi(struct request *rq) @@ -657,6 +618,30 @@ static inline bool blk_mq_complete_need_ipi(struct request *rq) return cpu_online(rq->mq_ctx->cpu); } +static void blk_mq_complete_send_ipi(struct request *rq) +{ + struct llist_head *list; + unsigned int cpu; + + cpu = rq->mq_ctx->cpu; + list = &per_cpu(blk_cpu_done, cpu); + if (llist_add(&rq->ipi_list, list)) { + INIT_CSD(&rq->csd, __blk_mq_complete_request_remote, rq); + smp_call_function_single_async(cpu, &rq->csd); + } +} + +static void blk_mq_raise_softirq(struct request *rq) +{ + struct llist_head *list; + + preempt_disable(); + list = this_cpu_ptr(&blk_cpu_done); + if (llist_add(&rq->ipi_list, list)) + raise_softirq(BLOCK_SOFTIRQ); + preempt_enable(); +} + bool blk_mq_complete_request_remote(struct request *rq) { WRITE_ONCE(rq->state, MQ_RQ_COMPLETE); @@ -669,15 +654,15 @@ bool blk_mq_complete_request_remote(struct request *rq) return false; if (blk_mq_complete_need_ipi(rq)) { - INIT_CSD(&rq->csd, __blk_mq_complete_request_remote, rq); - smp_call_function_single_async(rq->mq_ctx->cpu, &rq->csd); - } else { - if (rq->q->nr_hw_queues > 1) - return false; - blk_mq_trigger_softirq(rq); + blk_mq_complete_send_ipi(rq); + return true; } - return true; + if (rq->q->nr_hw_queues == 1) { + blk_mq_raise_softirq(rq); + return true; + } + return false; } EXPORT_SYMBOL_GPL(blk_mq_complete_request_remote); @@ -3892,7 +3877,7 @@ static int __init blk_mq_init(void) int i; for_each_possible_cpu(i) - INIT_LIST_HEAD(&per_cpu(blk_cpu_done, i)); + init_llist_head(&per_cpu(blk_cpu_done, i)); open_softirq(BLOCK_SOFTIRQ, blk_done_softirq); cpuhp_setup_state_nocalls(CPUHP_BLOCK_SOFTIRQ_DEAD, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index f94ee3089e015..89a444c5a5833 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -153,7 +153,7 @@ struct request { */ union { struct hlist_node hash; /* merge hash */ - struct list_head ipi_list; + struct llist_node ipi_list; }; /*