From patchwork Wed Mar 27 13:21:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abel Vesa X-Patchwork-Id: 10873475 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 815F3139A for ; Wed, 27 Mar 2019 13:21:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 678C028657 for ; Wed, 27 Mar 2019 13:21:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5B99128BBF; Wed, 27 Mar 2019 13:21:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C6F5328AED for ; Wed, 27 Mar 2019 13:21:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nSeNZ9qkMGPFTJcOWqmFQ1zJ7UbvKDjeLrfV4UFKWS4=; b=LsU5MKgzOTYV/F sEMNYsFVXKL3sjCMfNScOcqj35CHcSFOpnBJk4sWxi0r2Kj1MsNIpob0tSkDRhxsU1s74+ka9wjPG HDx/IlXqDvi4LlDBVB6V8NJ5N+pukj5rbp98qaKTtgOhZ/S4IGnLiOyo+v6SbU94Yy4n3q2Tzl5qi NBM70vrFe//bnyOnPRb3WBzlTfcmjPerfiH6E7C8CznTm10XEkA4dWEJ0UgvebX82Dbk3wrtoxMhy FA9dgsg6V7fIVZklg98RfrOCvKCYxZKag6ZDfoT2k05/sNnUOI2+Ze8gIgVeIb6y28xum+i5C+iKu PIAfWTyh7sgXQzFgLweg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h98V5-0004JJ-MO; Wed, 27 Mar 2019 13:21:51 +0000 Received: from mail-db5eur01on0618.outbound.protection.outlook.com ([2a01:111:f400:fe02::618] helo=EUR01-DB5-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h98UZ-0003YO-TG for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2019 13:21:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=40SFnIjgxvNsiI+hPmydu4vsYSvQXaeVREae9BA1guk=; b=qUcDrVbwdOIPa/unlw95cgpd++wsVSUabOXqR1DNk40QlBuiVmY6WyIuAZx9oEtlddmdw0y6iWqjLZYBmDYvOHTuXm1pWCU42kNdAAT/L2lsihK0+0dE/JUViUjrPf57svLpghdSf56oW1/aQdcBy7VKMsW+x0TATUtVXK26hKQ= Received: from AM0PR04MB5779.eurprd04.prod.outlook.com (20.178.202.151) by AM0PR04MB5907.eurprd04.prod.outlook.com (20.178.202.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1730.18; Wed, 27 Mar 2019 13:21:11 +0000 Received: from AM0PR04MB5779.eurprd04.prod.outlook.com ([fe80::9579:32ab:898e:6769]) by AM0PR04MB5779.eurprd04.prod.outlook.com ([fe80::9579:32ab:898e:6769%2]) with mapi id 15.20.1730.019; Wed, 27 Mar 2019 13:21:11 +0000 From: Abel Vesa To: Sudeep Holla , Marc Zyngier , Rob Herring , Mark Rutland , Shawn Guo , Sascha Hauer , "catalin.marinas@arm.com" , Will Deacon , "Rafael J. Wysocki" , Lorenzo Pieralisi , Fabio Estevam , Lucas Stach , Aisheng Dong Subject: [RFC 3/7] smp: Poke the cores before requesting IPI Thread-Topic: [RFC 3/7] smp: Poke the cores before requesting IPI Thread-Index: AQHU5J/ysdAjuuc4sEaUwQd13R6FyA== Date: Wed, 27 Mar 2019 13:21:11 +0000 Message-ID: <1553692845-20983-4-git-send-email-abel.vesa@nxp.com> References: <1553692845-20983-1-git-send-email-abel.vesa@nxp.com> In-Reply-To: <1553692845-20983-1-git-send-email-abel.vesa@nxp.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: VI1PR0701CA0038.eurprd07.prod.outlook.com (2603:10a6:800:90::24) To AM0PR04MB5779.eurprd04.prod.outlook.com (2603:10a6:208:131::23) x-originating-ip: [212.146.100.6] x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: git-send-email 2.7.4 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3c192940-a1a6-4210-a47b-08d6b2b71494 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:AM0PR04MB5907; x-ms-traffictypediagnostic: AM0PR04MB5907: x-microsoft-antispam-prvs: x-forefront-prvs: 0989A7979C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(366004)(376002)(346002)(39860400002)(396003)(199004)(189003)(25786009)(2906002)(26005)(386003)(54906003)(8676002)(68736007)(102836004)(6512007)(71200400001)(7416002)(71190400001)(446003)(5660300002)(8936002)(50226002)(4326008)(81166006)(2616005)(476003)(81156014)(44832011)(97736004)(11346002)(6116002)(3846002)(36756003)(105586002)(86362001)(106356001)(2501003)(6486002)(76176011)(6636002)(14454004)(478600001)(6506007)(316002)(52116002)(7736002)(186003)(53936002)(110136005)(6346003)(66066001)(99286004)(14444005)(256004)(486006)(6436002)(305945005)(921003)(1121003); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR04MB5907; H:AM0PR04MB5779.eurprd04.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: nxp.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=abel.vesa@nxp.com; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: LIXKcyLEH5DsJVfS9zvLh5/J5opTWB/PenemuARDTF+AmBO4Q4NCoYADb/NIyx+O8V/QgE5dRnXNNyf7EGffnfU+iviK/h9RiIikETHSl1fhftjZmLa0zXiL7Wr2Ll2Dum+hS8d9gHZuk/je5ELQ2qXO16GEquxZOFq7IMpZ3owDpSUNpD2HAHSqrFi76Kx8yTh9jmO5sLHHuVhUqc1GLgGkKOiraoA1raHF+IlymRs5/KKRXBHtRw5C0PqhpgDs3otPCiN46+pqmER7YKA4+yHdDmEg9B7veSU2deD2Z20ePlYR4zgC+o/JiED6mEjNkAVpCGfDAUbwRqDM4hMCupQthT7KN1v6yXC3JQm3uYXsKpOIx3hGYEedtDotw/0Bgf1yhSdPfC2TkI83vJDUrXp8m2uJEYf21PAgBOfxgxk= MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3c192940-a1a6-4210-a47b-08d6b2b71494 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Mar 2019 13:21:11.6330 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR04MB5907 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190327_062120_567475_A58EF62C X-CRM114-Status: GOOD ( 18.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Abel Vesa , "linux-pm@vger.kernel.org" , dl-linux-imx , "linux-arm-kernel@lists.infradead.org" , Linux Kernel Mailing List Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Try poking the specified core(s) every time before requesting IPI, this way allowing the cpuidle driver to do its magic for the current idle state of the specified core(s), if there is such a need. Signed-off-by: Abel Vesa --- kernel/irq_work.c | 19 ++++++++++++++++--- kernel/sched/core.c | 16 +++++++++++----- kernel/smp.c | 10 +++++++++- kernel/time/tick-broadcast.c | 4 ++++ 4 files changed, 40 insertions(+), 9 deletions(-) diff --git a/kernel/irq_work.c b/kernel/irq_work.c index 6b7cdf1..deca898 100644 --- a/kernel/irq_work.c +++ b/kernel/irq_work.c @@ -17,6 +17,7 @@ #include #include #include +#include #include @@ -76,8 +77,12 @@ bool irq_work_queue_on(struct irq_work *work, int cpu) if (!irq_work_claim(work)) return false; - if (llist_add(&work->llnode, &per_cpu(raised_list, cpu))) + if (llist_add(&work->llnode, &per_cpu(raised_list, cpu))) { + /* Poke the cpu through cpuidle first */ + cpuidle_poke(cpumask_of(cpu)); + arch_send_call_function_single_ipi(cpu); + } #else /* #ifdef CONFIG_SMP */ irq_work_queue(work); @@ -99,11 +104,19 @@ bool irq_work_queue(struct irq_work *work) /* If the work is "lazy", handle it from next tick if any */ if (work->flags & IRQ_WORK_LAZY) { if (llist_add(&work->llnode, this_cpu_ptr(&lazy_list)) && - tick_nohz_tick_stopped()) + tick_nohz_tick_stopped()) { + /* Poke the cpus through cpuidle first */ + cpuidle_poke(cpumask_of(smp_processor_id())); + arch_irq_work_raise(); + } } else { - if (llist_add(&work->llnode, this_cpu_ptr(&raised_list))) + if (llist_add(&work->llnode, this_cpu_ptr(&raised_list))) { + /* Poke the cpus through cpuidle first */ + cpuidle_poke(cpumask_of(smp_processor_id())); + arch_irq_work_raise(); + } } preempt_enable(); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 4778c48..7be9dba 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -126,6 +126,12 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf) } } +static void smp_poke_and_send_reschedule(int cpu) +{ + cpuidle_poke(cpumask_of(cpu)); + smp_send_reschedule(cpu); +} + /* * RQ-clock updating methods: */ @@ -511,7 +517,7 @@ void resched_curr(struct rq *rq) } if (set_nr_and_not_polling(curr)) - smp_send_reschedule(cpu); + smp_poke_and_send_reschedule(cpu); else trace_sched_wake_idle_without_ipi(cpu); } @@ -583,7 +589,7 @@ static void wake_up_idle_cpu(int cpu) return; if (set_nr_and_not_polling(rq->idle)) - smp_send_reschedule(cpu); + smp_poke_and_send_reschedule(cpu); else trace_sched_wake_idle_without_ipi(cpu); } @@ -1471,7 +1477,7 @@ void kick_process(struct task_struct *p) preempt_disable(); cpu = task_cpu(p); if ((cpu != smp_processor_id()) && task_curr(p)) - smp_send_reschedule(cpu); + smp_poke_and_send_reschedule(cpu); preempt_enable(); } EXPORT_SYMBOL_GPL(kick_process); @@ -1836,7 +1842,7 @@ static void ttwu_queue_remote(struct task_struct *p, int cpu, int wake_flags) if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) { if (!set_nr_if_polling(rq->idle)) - smp_send_reschedule(cpu); + smp_poke_and_send_reschedule(cpu); else trace_sched_wake_idle_without_ipi(cpu); } @@ -1857,7 +1863,7 @@ void wake_up_if_idle(int cpu) } else { rq_lock_irqsave(rq, &rf); if (is_idle_task(rq->curr)) - smp_send_reschedule(cpu); + smp_poke_and_send_reschedule(cpu); /* Else CPU is not idle, do nothing here: */ rq_unlock_irqrestore(rq, &rf); } diff --git a/kernel/smp.c b/kernel/smp.c index f4cf1b0..f6b2ce7 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -175,8 +176,12 @@ static int generic_exec_single(int cpu, call_single_data_t *csd, * locking and barrier primitives. Generic code isn't really * equipped to do the right thing... */ - if (llist_add(&csd->llist, &per_cpu(call_single_queue, cpu))) + if (llist_add(&csd->llist, &per_cpu(call_single_queue, cpu))) { + /* Poke the cpus through cpuidle first */ + cpuidle_poke(cpumask_of(cpu)); + arch_send_call_function_single_ipi(cpu); + } return 0; } @@ -457,6 +462,9 @@ void smp_call_function_many(const struct cpumask *mask, __cpumask_set_cpu(cpu, cfd->cpumask_ipi); } + /* Poke the cpus through cpuidle first */ + cpuidle_poke(cfd->cpumask_ipi); + /* Send a message to all CPUs in the map */ arch_send_call_function_ipi_mask(cfd->cpumask_ipi); diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c index 0283523..8bb7b2b 100644 --- a/kernel/time/tick-broadcast.c +++ b/kernel/time/tick-broadcast.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "tick-internal.h" @@ -286,6 +287,9 @@ static bool tick_do_broadcast(struct cpumask *mask) } if (!cpumask_empty(mask)) { + /* Poke the cpus through cpuidle first */ + cpuidle_poke(mask); + /* * It might be necessary to actually check whether the devices * have different broadcast functions. For now, just use the