From patchwork Thu Jan 20 14:04:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 12718696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6B2B7C433EF for ; Thu, 20 Jan 2022 14:04:58 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.259076.446835 (Exim 4.92) (envelope-from ) id 1nAY3S-0007aX-16; Thu, 20 Jan 2022 14:04:46 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 259076.446835; Thu, 20 Jan 2022 14:04:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nAY3R-0007aN-U9; Thu, 20 Jan 2022 14:04:45 +0000 Received: by outflank-mailman (input) for mailman id 259076; Thu, 20 Jan 2022 14:04:45 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nAY3R-0007aF-2R for xen-devel@lists.xenproject.org; Thu, 20 Jan 2022 14:04:45 +0000 Received: from de-smtp-delivery-102.mimecast.com (de-smtp-delivery-102.mimecast.com [194.104.109.102]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id eaff67e9-79f9-11ec-8fa7-f31e035a9116; Thu, 20 Jan 2022 15:04:44 +0100 (CET) Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-he1eur02lp2055.outbound.protection.outlook.com [104.47.5.55]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id de-mta-41-se1l-fHKO_mJLS3QJopwKA-1; Thu, 20 Jan 2022 15:04:42 +0100 Received: from VI1PR04MB5600.eurprd04.prod.outlook.com (2603:10a6:803:e7::16) by DBBPR04MB7643.eurprd04.prod.outlook.com (2603:10a6:10:203::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4888.11; Thu, 20 Jan 2022 14:04:41 +0000 Received: from VI1PR04MB5600.eurprd04.prod.outlook.com ([fe80::5951:a489:1cf0:19fe]) by VI1PR04MB5600.eurprd04.prod.outlook.com ([fe80::5951:a489:1cf0:19fe%6]) with mapi id 15.20.4909.010; Thu, 20 Jan 2022 14:04:41 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: eaff67e9-79f9-11ec-8fa7-f31e035a9116 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=mimecast20200619; t=1642687483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7m7Y7Upa0+r0a+ciUd8pwYsa/PeoFa1xWQvHddn56eY=; b=JYpM0vyvV6atxHfdMidOI+3MZDVDKjB+ZLrwzmluhly3GYxUh2GV8aXjpwOMSF07dSomOp rDeTQoVEnIEMNgLTWY6v3WRaHBYWejAyQM96H5463KozciQ3HFAvWn1efw1Z8cHIIepDPh tkOvAp3lh5MaQ0kkmYsE8Ft+8cn4Puk= X-MC-Unique: se1l-fHKO_mJLS3QJopwKA-1 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=crEg0xyhA0n3UCTVJOOtb/U6gw8b2h+l5x83vWhOYpZgChiWCmB5ZD7cc7FiP1mx6VZor7SXiJ+CTvJUAh51/iC95TzQeve+nRCOLgZ8O7fq1qNxQDAMZBzNaa7cNOr158zUwHcS8ZptAQihExT49vqB5wWrflvWZQyRWO1sLXFVavVyPjqF7A42eEwhqzIpl2JDSlyZlOQlPkjbsjxS5gN4kczpeJFr48mW9Uy1juLsOzhnvEujDgsPz8EMPrrKLic7q4SxP2Bvqs5GPS0+U+Xp5sLsbDdsOWpIam72K21x+jjhMZuSMDLgPQXDUCiNUNCkvj4Ux8EbNnaZaz7w9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7m7Y7Upa0+r0a+ciUd8pwYsa/PeoFa1xWQvHddn56eY=; b=Z6N3unE3L3XNp6WhMcpmSk6D1SUp+YvkTzGA8Hc1ENACnhip0t7ghtDLkVLvqVkl4ITg5WjWJ5hy/WLslN99lBSLrexmaS9IhIi8uQZhrRwpNKzSfZXG/ULfmgVX3cH29ecJd7vhQT2GrUUG6JlQwiESQfskYsJpOWnrdSzcGy42XRuSN3RI3cp0W4oFs/JOeDQ6hCl/z4CFzdbUZr5+/0t922OeGpKDvuHxALmWFpejpf/LUJRspctTWAwKCkyfKb1WT5xXbYxMU4WIU8MZG8dL9uv7+Ki9d7QisDTXlqXo77P97QoJa2sFJlyJatl9HfiEVqfeFL5LpVWzGk+/SQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=suse.com; Message-ID: Date: Thu, 20 Jan 2022 15:04:39 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: [PATCH RFC v2 4/5] x86/mwait-idle: enable interrupts before C1 on Xeons Content-Language: en-US From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , Wei Liu , =?utf-8?q?Roger_Pau_Monn=C3=A9?= References: <568cb6dc-127a-9c8f-0073-2df173d4677b@suse.com> In-Reply-To: <568cb6dc-127a-9c8f-0073-2df173d4677b@suse.com> X-ClientProxiedBy: AM6PR10CA0070.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:209:80::47) To VI1PR04MB5600.eurprd04.prod.outlook.com (2603:10a6:803:e7::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aebfd843-7d0c-4d50-89ae-08d9dc1dcd66 X-MS-TrafficTypeDiagnostic: DBBPR04MB7643:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4714; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: a2bGEuN0gz7d0hWgyVJh2RxFx5OLVYMDInTyc1MCeOzhiKqO9il63aFEpHt4xgoN1Z9XlUczOAv4uiCW23ZSOJrht0p7dcqcDwtBIpDRfwAjJni82lgH4kIxarGmoKIgToclb1WOr+J8DzvZvg0fYiKaG63thISPn160hVEJloEB7tFD6gNhvSUpCb90a2HvjZWiUUPI4cim0U4KYEc92u2N6EcWEccSZ5ZlFd0SjLtkqm2yrEyEqSYmkSA28ArF1LBiH54tAWNCsHVvLgRsbltgVKNROBMHUi/b8+jboLxbAv9hyV8FZhvfUykQU2qOqJle6d/9ZFzCY7Gw0TsBJAwAtdZBLNpKmoDNGA/flh+88bmdMsJPAGqzfXJ7RZZW40Qr7wzl2LeIu6aBJD8aFscv61DFSXDCvn58FOt7WbB2L0tyVZYk/5heZuzLTljaUEdszGKVigelDw6ZaBds5JwaTA7VDypsXdxyT2YxKo8H50fgVk2q6oOy4KCQ1d9krsvnMIhhsTne7lqd4/gbpV5M9Pa4ApV2Y2dIowFsmj7V3AYFwV0f8MReOmf6KhELqhJxLgvsmU0y3JWsvNPdUpJWEd45ZPYuUtGU4IpP559Lf8MGmvrFxDCK7nCq8EV5In4G8ypdXtWq3M0BBfWiyBFv38fQcoTpCCPErs5I79Y0TF+kreHTn6mbU7kzzwnHXXTcsk00o6VFAdpNKVjbbrB5jptVuOr7c92bQ6C6S0h73B4+0uHRpbz/As+FLegn X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR04MB5600.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(6506007)(31686004)(36756003)(66476007)(508600001)(8676002)(31696002)(8936002)(38100700002)(6486002)(86362001)(186003)(2616005)(83380400001)(6916009)(66556008)(66946007)(316002)(2906002)(5660300002)(26005)(6512007)(54906003)(4326008)(43740500002)(45980500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?e0vUlIpE2U8hEwl6qk9gG2XQc0tP?= =?utf-8?q?5ImWDlnAjEOkFvVNEET4O0GUTND3dfASeptNaw9PWWg2A7ayyQfkfaFglOiAN1by/?= =?utf-8?q?hLp/oLuMc9KM838MrK3kr/OgIWLfQPntY9kbynxCKsFpGvtTk6z+RB1SxVJMneKtr?= =?utf-8?q?H01ye6lll1DY0R1JlGyAso6H/yK4OehbzjVYyQmgUXXM7X+ssGcl4cdFpTS01DRD1?= =?utf-8?q?tuHk/YOJow/ZdvGFwZAODDnWVk9Fq72uS4QdkuvlPlBgTqE/c6WXbZUJj3bZ05cYL?= =?utf-8?q?pEtGRqX9Ey0375yKZQ3xaYkdEXrgEYvRDxn/+9AVigJArGytQuoOtEcSJExvFAOpn?= =?utf-8?q?+C4sP18DMod46gpa3FowaQDVYRCyn7Jeh6kwmG+vQVyLtJkEEZQgaB7ksI06r68Il?= =?utf-8?q?JUC1d4OoQHQKTWLiCUiyoP2EEF1d87uFqBpSDPVXV4FRYJw8zqW9Xy+nE5clZBK1P?= =?utf-8?q?nMlZy167y8V2TWIw9C4XmWzHnUaQ5oTawj/DGPqQt9tW5JmAkwMQT8uxGbb89fAI8?= =?utf-8?q?Pm2EdF/e7WeD2uYIWeVphMnpnBXrYdv4xLXqjqCHZhxfLfO0mWKr/GjEV4lwysLjz?= =?utf-8?q?GFtvL1IxaaGAScELczfYhEwjnqcCWRJzpSBsCCV77zJId5USe7zT8yewpZd6p4JRj?= =?utf-8?q?4HjDCtupvkXBDwobOeZ5wcB9FJTnkVoXwYKO50/7auA+zgKjRePuPcVZMrUsSzDSZ?= =?utf-8?q?miitMgdZymKpC7PMgqRmPU6kbh5wHARmh+jKkrlHl1vPJMwXCZfbI2xkvGHqjFNkq?= =?utf-8?q?SaqG25ORVbNpeM/qiY3N2oh3+HMk9LRYofxvFmg7k1HSVfpVNf4Qv26FoPtlr6eqO?= =?utf-8?q?wAd3QK9MqsnbiPHy4a4zu6CqgLrYGz8b8p7RmBHrz5bX9ObmMGduO/Ky6f5u7Gee8?= =?utf-8?q?JvHWm+p4mLTlTrA6l8IrkVBQFfC/ODr/bh/QOpUSWBe9tkiCDWCti8lzoDMXqdS/3?= =?utf-8?q?eAZeB6qh9qPtSJNaImAW/M1SXC7D3VG9GRZ31I4FUpQ/Fh3/Bx3WnnbgyXqs4lSDG?= =?utf-8?q?L5auPwhWkYStcHqJZDGzd1tgNnJMp8GAKNIYfokVzRj7285xz1CkIGyrrb0w0UN3k?= =?utf-8?q?f0dSEoDwJaHOJTaQATOr9F4pyzUkjHDTrmKpu9skCIZI3JDFT6ztF/JbTyuNUvmrT?= =?utf-8?q?4G4XzUQbjFlpVyGDXZuJr726uoPIe+eNrJkDupVSFXkj2yLp1NUVoduJM9T+K08tf?= =?utf-8?q?lmxRR5lZoWY6yRhl3YBCHV4NHSs3CgFNhrN7xIi43QqFgPzAWFPfzkiqnjW9upBrX?= =?utf-8?q?iCjBVx6EHHQCTQTte25/2SiWnf6wh2dEMELPYj43N4TiUjJc0me1GQ3oHoao85WTi?= =?utf-8?q?7tsJtRl4v17TEJpAemAMjN/k/ruhLQvh7BUu4t+M5nyh5X8nwczSDerSSOV9auq4q?= =?utf-8?q?FMyoT8QkMxJNyT2aE51l6XR76ynTJwCsSXEqw6LdJwbf5vS/jfXoRYgcs6BnV6Z8S?= =?utf-8?q?Yk8bCl+i0rsMUY5nXvJgXLHuCyXzSZM9YRgw4dpkJ35WcLwOXgDfC/85ubJHmftoj?= =?utf-8?q?nrz6LrZ7QTBOolIuMmSu7R1m+PUcbyM8HmXTLp2MIXqY9/orIou7UDU=3D?= X-OriginatorOrg: suse.com X-MS-Exchange-CrossTenant-Network-Message-Id: aebfd843-7d0c-4d50-89ae-08d9dc1dcd66 X-MS-Exchange-CrossTenant-AuthSource: VI1PR04MB5600.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jan 2022 14:04:40.9616 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JFR3eqvSV4ZDIu1hmy/w42SVMrFV9SrQnH0g+cbhCfqVcDwDufYChi+0c751g58SEG2fNzfV2ztYDmDy7HDGqg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR04MB7643 From: Artem Bityutskiy Enable local interrupts before requesting C1 on the last two generations of Intel Xeon platforms: Sky Lake, Cascade Lake, Cooper Lake, Ice Lake. This decreases average C1 interrupt latency by about 5-10%, as measured with the 'wult' tool. The '->enter()' function of the driver enters C-states with local interrupts disabled by executing the 'monitor' and 'mwait' pair of instructions. If an interrupt happens, the CPU exits the C-state and continues executing instructions after 'mwait'. It does not jump to the interrupt handler, because local interrupts are disabled. The cpuidle subsystem enables interrupts a bit later, after doing some housekeeping. With this patch, we enable local interrupts before requesting C1. In this case, if the CPU wakes up because of an interrupt, it will jump to the interrupt handler right away. The cpuidle housekeeping will be done after the pending interrupt(s) are handled. Enabling interrupts before entering a C-state has measurable impact for faster C-states, like C1. Deeper, but slower C-states like C6 do not really benefit from this sort of change, because their latency is a lot higher comparing to the delay added by cpuidle housekeeping. This change was also tested with cyclictest and dbench. In case of Ice Lake, the average cyclictest latency decreased by 5.1%, and the average 'dbench' throughput increased by about 0.8%. Both tests were run for 4 hours with only C1 enabled (all other idle states, including 'POLL', were disabled). CPU frequency was pinned to HFM, and uncore frequency was pinned to the maximum value. The other platforms had similar single-digit percentage improvements. It is worth noting that this patch affects 'cpuidle' statistics a tiny bit. Before this patch, C1 residency did not include the interrupt handling time, but with this patch, it will include it. This is similar to what happens in case of the 'POLL' state, which also runs with interrupts enabled. Suggested-by: Len Brown Signed-off-by: Artem Bityutskiy [Linux commit: c227233ad64c77e57db738ab0e46439db71822a3] We don't have a pointer into cpuidle_state_table[] readily available. To compensate, make use of the new flag only appearing for C1 and hence only in the first table entry. Unlike Linux we want to disable IRQs again after MWAITing, as subsequently invoked functions assume so. Signed-off-by: Jan Beulich Acked-by: Roger Pau Monné --- RFC: I'm not entirely certain that we want to take this, i.e. whether we're as much worried about interrupt latency. RFC: I was going back and forth between putting the local_irq_enable() ahead of or after cpu_is_haltable(). --- v2: New. --- a/xen/arch/x86/cpu/mwait-idle.c +++ b/xen/arch/x86/cpu/mwait-idle.c @@ -108,6 +108,11 @@ static const struct cpuidle_state { #define CPUIDLE_FLAG_DISABLED 0x1 /* + * Enable interrupts before entering the C-state. On some platforms and for + * some C-states, this may measurably decrease interrupt latency. + */ +#define CPUIDLE_FLAG_IRQ_ENABLE 0x8000 +/* * Set this flag for states where the HW flushes the TLB for us * and so we don't need cross-calls to keep it consistent. * If this flag is set, SW flushes the TLB, so even if the @@ -539,7 +544,7 @@ static struct cpuidle_state __read_mostl static struct cpuidle_state __read_mostly skx_cstates[] = { { .name = "C1", - .flags = MWAIT2flg(0x00), + .flags = MWAIT2flg(0x00) | CPUIDLE_FLAG_IRQ_ENABLE, .exit_latency = 2, .target_residency = 2, }, @@ -561,7 +566,7 @@ static struct cpuidle_state __read_mostl static const struct cpuidle_state icx_cstates[] = { { .name = "C1", - .flags = MWAIT2flg(0x00), + .flags = MWAIT2flg(0x00) | CPUIDLE_FLAG_IRQ_ENABLE, .exit_latency = 1, .target_residency = 1, }, @@ -776,6 +781,7 @@ static void mwait_idle(void) unsigned int next_state; u64 before, after; u32 exp = 0, pred = 0, irq_traced[4] = { 0 }; + bool irq_enable_early = false; if (max_cstate > 0 && power && (next_state = cpuidle_current_governor->select(power)) > 0) { @@ -806,6 +812,12 @@ static void mwait_idle(void) return; } + if (cx->idx == 1 && cx->type == ACPI_STATE_C1 && + (cpuidle_state_table[0].flags & CPUIDLE_FLAG_IRQ_ENABLE)) { + ASSERT(cx->address == flg2MWAIT(cpuidle_state_table[0].flags)); + irq_enable_early = true; + } + cpufreq_dbs_timer_suspend(); rcu_idle_enter(cpu); @@ -842,9 +854,15 @@ static void mwait_idle(void) update_last_cx_stat(power, cx, before); - if (cpu_is_haltable(cpu)) + if (cpu_is_haltable(cpu)) { + if (irq_enable_early) + local_irq_enable(); + mwait_idle_with_hints(cx->address, MWAIT_ECX_INTERRUPT_BREAK); + local_irq_disable(); + } + after = alternative_call(cpuidle_get_tick); cstate_restore_tsc();