From patchwork Thu Jun 30 08:54:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 12901441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DE178C433EF for ; Thu, 30 Jun 2022 08:56:46 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.358345.587537 (Exim 4.92) (envelope-from ) id 1o6pyW-0008Hn-Ec; Thu, 30 Jun 2022 08:56:36 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 358345.587537; Thu, 30 Jun 2022 08:56:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1o6pyW-0008HR-Az; Thu, 30 Jun 2022 08:56:36 +0000 Received: by outflank-mailman (input) for mailman id 358345; Thu, 30 Jun 2022 08:56:35 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1o6pyV-0007Q3-AU for xen-devel@lists.xenproject.org; Thu, 30 Jun 2022 08:56:35 +0000 Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 880a05ab-f852-11ec-bdce-3d151da133c5; Thu, 30 Jun 2022 10:56:30 +0200 (CEST) Received: from mail-sn1anam02lp2049.outbound.protection.outlook.com (HELO NAM02-SN1-obe.outbound.protection.outlook.com) ([104.47.57.49]) by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 30 Jun 2022 04:56:31 -0400 Received: from DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) by SJ0PR03MB6699.namprd03.prod.outlook.com (2603:10b6:a03:402::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5395.14; Thu, 30 Jun 2022 08:56:29 +0000 Received: from DS7PR03MB5608.namprd03.prod.outlook.com ([fe80::40af:d5f4:95eb:d534]) by DS7PR03MB5608.namprd03.prod.outlook.com ([fe80::40af:d5f4:95eb:d534%7]) with mapi id 15.20.5395.015; Thu, 30 Jun 2022 08:56:29 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 880a05ab-f852-11ec-bdce-3d151da133c5 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1656579393; h=from:to:cc:subject:date:message-id:in-reply-to: references:content-transfer-encoding:mime-version; bh=JROiIcytnT09wIXffk8Xeuw3JUfAkJpdvq4oMXLF/HA=; b=GGYY13cWcrcO5IREOgWFjAUS+FI6b4Rs/1eUQVHop8cSYqLvVhEGq/4X YDKmVa8gWf8f7yH0bqxOi4fIyqP5bct+/Y0glm5TUWkNyg6/j4ExJrX+7 Fqfofzwo4pJcG3CUN77m58f1UpyriAjW9ALulLGkAclcGFRTb6KWHAunv U=; X-IronPort-RemoteIP: 104.47.57.49 X-IronPort-MID: 74794303 X-IronPort-Reputation: None X-IronPort-Listener: OutboundMail X-IronPort-SenderGroup: RELAY_O365 X-IronPort-MailFlowPolicy: $RELAYED IronPort-Data: A9a23:W57PUaB+nr07uBVW/13iw5YqxClBgxIJ4kV8jS/XYbTApD4n1DVSy GIYWmiCPq7fZTagc9EiO4+z908BsJ+DzN4yQQY4rX1jcSlH+JHPbTi7wuYcHM8wwunrFh8PA xA2M4GYRCwMZiaA4E/raNANlFEkvU2ybuOU5NXsZ2YgH2eIdA970Ug5w7Bi2tYx6TSEK1jlV e3a8pW31GCNg1aYAkpMg05UgEoy1BhakGpwUm0WPZinjneH/5UmJMt3yZWKB2n5WuFp8tuSH I4v+l0bElTxpH/BAvv9+lryn9ZjrrT6ZWBigVIOM0Sub4QrSoXfHc/XOdJFAXq7hQllkPgqx N9G7rqaaDwmL+7lmPowSzZGL3lxaPguFL/veRBTsOS15mieKT7X5awrC0s7e4oF5uxwHGdCs +QCLywAZQyCgOTwx6+nTu5rhYIoK8yD0IE34yk8i22GS6t5B8ySK0nJzYYwMDMYnMdBEOyYf 8MEQTFucA7Bc1tEPVJ/5JcWw7v31yWkK2YwRFS9lJUIyjOQ5hFNyLnsLN+Oc8yXR+wOtxPNz o7B1yGjav0AD/SPxDzA/n+yi+vnmSLgRJlUBLC+7uRtglCY2ioUEhJ+fVmxrOS9i0W+c8lCM EFS8S0rxYAt8GS7Q9+7WAe3yENopTYZUttUVvY8sQiLw6+MuQKBXDBYH3hGdcAss9IwSXoyz FiVktj1BDtp9rqIVXaa8bTSpjS3UcQIEVI/ieY/ZVNty7HeTEsb1Hojkv4L/HaJs+DI IronPort-HdrOrdr: A9a23:eQzNqq7/Bns5pG7x4wPXwVOBI+orL9Y04lQ7vn2ZFiY5TiXIra qTdaogviMc6Ax/ZJjvo6HjBEDmewKnyXcV2/hrAV7GZmXbUQSTXeVfBOfZowEIXheOj9K1tp 0QDJSWdueAamSS5PySiGfYLz9j+qj+zEnBv5aj854Hd3AOV0gP1XYbNu7NeXcGOTWuSKBJYq a0145inX6NaH4XZsO0Cj0sWPXCncTCkNbDbQQdDxAqxQGShXfwgYSKWySw71M7aXdi0L0i+W /Kn0jQ4biiieiyzlv523XI55pbtdP9wp9oBdCKiOISNjLw4zzYLbhJavmnhnQYseuv4FElnJ 3lpAohBd167zfrcmS8sXLWqnzd+Qdrz0Wn5U6TgHPlr8C8bik9EdB9iYVQdQacw1Y8vflnuZ g7k16xht5yN1ftjS7979/HW1VBjUyvu0cvluYVkjh2TZYeUrlMtoYSlXklXavoJBiKprzPLd MeTf01vJ1tABOnhjHizyNSKeWXLzsO9kzseDlAhiSXuwIm7kyRgXFohvD3pU1wha7Ve6M0md gsDZ4Y5I2mNvVmC56VJN1xNfdfWVa9Ni7kASa1HWnNMp0hFjbkl6PXiY9Fl91CPqZ4h6cPpA == X-IronPort-AV: E=Sophos;i="5.92,233,1650945600"; d="scan'208";a="74794303" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KSHz5XwDLfM+xRWzdkRp76wM17Quhn3EiswzWIyGE32K7MaPlcn3gII0AUy7Us4FQOlTc2+5vOdUtphq//igiruZ8YXWUgj8ZSMeV2Y18ZMvmZHXZaSi965EfjQnH3F/Q2P2VdFNvAf+VhSaib+hJQZrt34ZaRBfFXYEPLZwc3uFu1XihtaxUgw7sUVug0CAOZ85iOUdOrZzb/8VKZrNaP5VFZF6G2gspEEFmLfs0mtbhC6cxTA7f7SY1+yyOhCp1KS+xWXmPtZFXyn79bYV6qUuLZCVBGiWqy24NtrpsTZxAMruUFztBr9yiRx+8mZorju3ESgP/hKbS9gzaojgoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FloscYbijaSLVpLxOdcT4wyXo161zpQTeRJwtCFDY80=; b=c/SHP9RIFXJlgOPgWEc/cjxrrMXPhUx/ChooB05OP3EcXJAru8QhlxAZr4L1YaSOTiwZWmeETqQFKma1iTZAoGK25m3u9efGJkdkZYOpUHjX2xQTLtArrL2mINaCQoXRz/aRGlXZoqesou5rjFrv2mKA6+qFkznMli/9K4g4FHuVMTbXVSM30eMyyFhdz2E+UCG/pf838bA7G7jg1VtUnxcyfZJlZInCmhWzKiuh+uwxAyRX3YveUb7cwk8tKx0r8PY64GnZTU8nRkmKm7eE7hLpMWUmCqJ0ty9bMk69bVLIVCB+RhiJ4qQvXOiZa42aHNQNKY4XtliIB/vAg9FFcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FloscYbijaSLVpLxOdcT4wyXo161zpQTeRJwtCFDY80=; b=fuZlxv9fMBjeyh/KIYSmG6b+OA+XYvFVz/rYXS6p2nSo4NKgv4C7LGmZGaLAiw7UFYyroyI5iM5/IjGp95REvLv4CqUSAHUHAmeWzFNiIWWN++jilg3SCDobDfBav8QprKlvMZIMMbL3wDu0uiSC12RXeNx2PKX2Cz0soB/s5jA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=citrix.com; From: Roger Pau Monne To: xen-devel@lists.xenproject.org Cc: Roger Pau Monne , Jan Beulich , Andrew Cooper , Wei Liu Subject: [PATCH v2 4/6] x86/irq: fix setting irq limits Date: Thu, 30 Jun 2022 10:54:37 +0200 Message-Id: <20220630085439.83193-5-roger.pau@citrix.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220630085439.83193-1-roger.pau@citrix.com> References: <20220630085439.83193-1-roger.pau@citrix.com> X-ClientProxiedBy: LO2P265CA0017.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:62::29) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c6ac01c8-77ed-428b-f3e2-08da5a766c04 X-MS-TrafficTypeDiagnostic: SJ0PR03MB6699:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: m0nbE3sS4ZosGph5t2+YFXN0qJQwWkO/aR85HKeNB+eYkeXVs+AbFspUmbN6xKHj4I+I6P8wRhdnP/xMVWA9t7ltqvIQxLtKAL7rdM9gvs6Q0pjH3XfIfSsHwF55GdPcyLjHM6G/ThkaKryCOxTjhVn2SQyRtnE0NuolJbqSXqccJtWFGIDrB7Vv3bCGU6yT1N37Lo0t4OP6z8nAs6tvL1H+nFoBeHoL6vPOWjqZWtjWi31AJv78qVYxGgnMCeKKyVh1dYM80rILgiFTxDI/B/NgZMDoeuiYaXmH7a+wKOPWYLFZZ9yXByzixDgaufTz7Ucuemn9iouTlJDUkLGY9wVAY/NjfqtuxCdmu0K/uKJIu0VTUKhHAtjhi2mSLrlT3d0zJSWMVL97OhNvNfMMiE9bKqroVpoWSOU4TcdewB+oOy35RRuhc33h002HoIOFUbWOowFfd9s96voRL1oRr69159lMRMTXVjXOhIeLdylBAwbORj4n/ofxgHphkSjUrVlJ6pSAE1DUO7p872tocyC9yNWmnv+/31eK+qvrtsXifipMrkLSp+6RKyzo+sR8PCX6KoSl4sSnC5KGyvpoVZcO7l1iz6FaACNWukNmwEuptjZSUhqhgNkk3tXFSgtgn+x5Z0acCMY3IO/9mwYM+pl5FvdsqoacERXh3v/E5TYxi9Pk67PlJm+Kzj/JWBxXlWLbf47GSlO9UxEVIbBw0xTH42GoAfr/amwoTiud/od+RxlvSqTjwuUUiOQ3JfNT X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(39860400002)(376002)(396003)(366004)(346002)(136003)(186003)(36756003)(316002)(2906002)(6506007)(86362001)(54906003)(41300700001)(2616005)(6666004)(38100700002)(6916009)(66556008)(66476007)(26005)(66946007)(4326008)(5660300002)(8936002)(8676002)(6512007)(1076003)(478600001)(6486002)(82960400001)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?7+Pr3dnPjY4q7gNDgRL+LbYw7mNl?= =?utf-8?q?4LtYZVdX7lHQb1FPEJWnU5sbBls53dvKAzZyteKAbf1dGS6zlpA6/mw3gIwIvtmto?= =?utf-8?q?L8snaY5uJrbLvlc3lCVpbsVNMb4Ni/larKK2bN9DcFoIV8lfvMZcFQrRi44r/PKYR?= =?utf-8?q?qB+pS1YHjjahWWpPY1b7g4nK07nm7iK/5OGYYccsTNzg5940/nGCOyXiuBzWwlVVD?= =?utf-8?q?ItbFzS4tXnOIUleCxXIZCbYaqQVOHwUt4lyfoTpQ6L5aDJN+HdGgOyNO8DQuc3iwq?= =?utf-8?q?q5kHEkdExfheyT0JyzzCzOeTkq7LQNjP0tZtmczMQmY5J7GEpg8AHQ91RfmtG/3QA?= =?utf-8?q?nFKHlVcQSG8w2QkH89Nk1FLuYQHnLV0qpnJ3yS9lN72cDyYs04/538IqG0rFwsOM1?= =?utf-8?q?xKo4EKp7V6yymOMlTnG6BF8/hrMyPsytdBsJIfhAQxFxE4OKDQNCX7bcoV0skq9xm?= =?utf-8?q?9YsAmWIG5D4XjKBYiSUFOSfqLO6GV6W5/J9tDMSL+alrYZgGdAKYM9kqr5d9h/7XG?= =?utf-8?q?O8W5PaOftUJ2/x8yua3d7vnMZCCEA356phhShAkqvrPBVoS2/JY6yUcBH3tYPFmAK?= =?utf-8?q?bVq/7GmwxBBFjRrsDyzWYQ02Y3Q2u2Mklj22qCp8ci2NeWHrpCosINeKoXW+CKFgc?= =?utf-8?q?gZpAdAXCVwLpqija3WTGTkI2nuQzozDNAZkA1w2u/31motwDgVexk/KVX2mlgRLrk?= =?utf-8?q?NWxFxLKyCRIV6sd5/CdJ+gOjH0nlYomewBxVlE2D6JN7P2d9hDQxgoUAVzMPCClJK?= =?utf-8?q?1EY6lqk5otQ6wvr8JtC42mBdwQb/4aYxQU1sfpU/8rk/TI4d2TsI3D8iVgVW84Ptf?= =?utf-8?q?FheR7X7sxTh9yS5da/S7ocpQEVl+mK0PA3x6aeM6/GwBVmQHMch07P+49gGjR/7eL?= =?utf-8?q?72016oFNfB+nt1QPZwUxwrijlEhwL54kFvA9NXwIyJFFneCQUZDz2Dbcm6D4KQ0YU?= =?utf-8?q?lbnhPjUKgxsZtAYzXK0BhgGhbgUgL3FdJBgejvKk98tKy7ZhNmIZeJqEh4CM1+Flk?= =?utf-8?q?OXpyZ9P69OMzJq/upMgwagDqFm5yhicSD4QhCk/D+qB+WKyBCgBpsW8tzW29OZDQq?= =?utf-8?q?77UqeQrop/I85TauvJHST+rNHpCmklP12JcmOGYzICwAeEqAyMn0J0U/Qi4VWYxnW?= =?utf-8?q?X383p8s1y64A3kKtAwx9UHhUNPNkDqJt0G3udEju9DrA5wjdmwmOjirVidWiqHPOr?= =?utf-8?q?ZlH7QG4GnULhPQqGTEi0wjlSh1sZ0AqhyfOn7XmBDaRe8TC7gZqgz7wGKlV/IwCrE?= =?utf-8?q?RKcEuZpUvXjAlPz8xOtI+Lmf9cmVyY4Iyo0sQjFVSw/TlHhdxVvaIG3dxY6jFcG47?= =?utf-8?q?avRsWHED57XMDvMxnV3Bn7BKKKeXGR3aLssY8Tv3d06I0PyLl87pfaOOMHyWjlFhV?= =?utf-8?q?zUGfGMe8Gx3PinAhlUBvp/Q4h7F5Jh85zE6DBI0pXthcsszxoH/orf+YnFLp99Kd0?= =?utf-8?q?xIlZEETKu2taYMK/yitV7g+BDdIfXLQRYd1hm9Ol6l9Ww3JNkvlqljB+E1bZuoZeB?= =?utf-8?q?cVmUOP7ip2bJFTCPUEy9F5e66udzGVHi2g=3D=3D?= X-OriginatorOrg: citrix.com X-MS-Exchange-CrossTenant-Network-Message-Id: c6ac01c8-77ed-428b-f3e2-08da5a766c04 X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jun 2022 08:56:29.3282 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tyTho2dkLudEbOraS8MUR3y0vuNOjWwoRyDy+vYBRZxSzHvBg+chstdFe6nCZ2UF3hx4cijfEF/PY6sV3nHHaw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR03MB6699 Current code to calculate nr_irqs assumes the APIC destination mode to be physical, so all vectors on each possible CPU is available for use by a different interrupt source. This is not true when using Logical (Cluster) destination mode, where CPUs in the same cluster share the vector space. Fix by calculating the maximum Cluster ID and use it to derive the number of clusters in the system. Note the code assumes Cluster IDs to be contiguous, or else we will set nr_irqs to a number higher than the real amount of vectors (still not fatal). The number of clusters is then used instead of the number of present CPUs when calculating the value of nr_irqs. Signed-off-by: Roger Pau Monné --- xen/arch/x86/genapic/x2apic.c | 2 +- xen/arch/x86/include/asm/apic.h | 2 ++ xen/arch/x86/irq.c | 10 ++++++++-- xen/arch/x86/mpparse.c | 5 +++++ 4 files changed, 16 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/genapic/x2apic.c b/xen/arch/x86/genapic/x2apic.c index 7dfc793514..ad95564e90 100644 --- a/xen/arch/x86/genapic/x2apic.c +++ b/xen/arch/x86/genapic/x2apic.c @@ -228,7 +228,7 @@ static struct notifier_block x2apic_cpu_nfb = { .notifier_call = update_clusterinfo }; -static int8_t __initdata x2apic_phys = -1; +int8_t __initdata x2apic_phys = -1; boolean_param("x2apic_phys", x2apic_phys); const struct genapic *__init apic_x2apic_probe(void) diff --git a/xen/arch/x86/include/asm/apic.h b/xen/arch/x86/include/asm/apic.h index 7625c0ecd6..6060628836 100644 --- a/xen/arch/x86/include/asm/apic.h +++ b/xen/arch/x86/include/asm/apic.h @@ -27,6 +27,8 @@ enum apic_mode { extern bool iommu_x2apic_enabled; extern u8 apic_verbosity; extern bool directed_eoi_enabled; +extern uint16_t x2apic_max_cluster_id; +extern int8_t x2apic_phys; void check_x2apic_preenabled(void); void x2apic_bsp_setup(void); diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c index b51e25f696..b64d18c450 100644 --- a/xen/arch/x86/irq.c +++ b/xen/arch/x86/irq.c @@ -421,9 +421,15 @@ int __init init_irq_data(void) int irq, vector; if ( nr_irqs == 0 ) - nr_irqs = cpu_has_apic ? max(0U + num_present_cpus() * - NR_DYNAMIC_VECTORS, 8 * nr_irqs_gsi) + { + unsigned int vec_spaces = + (x2apic_enabled && !x2apic_phys) ? x2apic_max_cluster_id + 1 + : num_present_cpus(); + + nr_irqs = cpu_has_apic ? max(vec_spaces * NR_DYNAMIC_VECTORS, + 8 * nr_irqs_gsi) : nr_irqs_gsi; + } else if ( nr_irqs < 16 ) nr_irqs = 16; diff --git a/xen/arch/x86/mpparse.c b/xen/arch/x86/mpparse.c index d8ccab2449..dc112bffc7 100644 --- a/xen/arch/x86/mpparse.c +++ b/xen/arch/x86/mpparse.c @@ -131,6 +131,8 @@ static int __init mpf_checksum(unsigned char *mp, int len) return sum & 0xFF; } +uint16_t __initdata x2apic_max_cluster_id; + /* Return xen's logical cpu_id of the new added cpu or <0 if error */ static int MP_processor_info_x(struct mpc_config_processor *m, u32 apicid, bool hotplug) @@ -199,6 +201,9 @@ static int MP_processor_info_x(struct mpc_config_processor *m, def_to_bigsmp = true; } + x2apic_max_cluster_id = max(x2apic_max_cluster_id, + (uint16_t)(apicid >> 4)); + return cpu; }