From patchwork Fri Oct 18 06:40:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13841234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D809D3C550 for ; Fri, 18 Oct 2024 06:41:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBD976B0095; Fri, 18 Oct 2024 02:41:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1B406B0099; Fri, 18 Oct 2024 02:41:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 836796B0095; Fri, 18 Oct 2024 02:41:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 585C46B0096 for ; Fri, 18 Oct 2024 02:41:11 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D67A8160521 for ; Fri, 18 Oct 2024 06:40:57 +0000 (UTC) X-FDA: 82685775522.27.B73FDF0 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.15]) by imf24.hostedemail.com (Postfix) with ESMTP id CBBFA180011 for ; Fri, 18 Oct 2024 06:41:06 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=caziR4+x; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729233561; a=rsa-sha256; cv=none; b=ZOm9GhXlYgU8uXLqeXByWTu+n7x0YZN3CuEsC5J5e0On2GZFeViHHQraAdo7zBHMGJdF0a mo/Jg7R9t9aosggb1DaCUtz71fJrtMTYCLa9BahN0UuTSO5wrGGn8tZZ0Agy1st/jzbpMY aIKAVnpa4aP8qhtxXxBStullsa86Wfc= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=caziR4+x; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729233561; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6teQWrRk5ViUbGxsxYU0nUWeKM6iU84VA4tAM6R5lbo=; b=l5hnrCqZC9fr7weFZ8TQDh7VXm+ljMXPVnTbzBiPRmG74R8Bb8N2P63X7dd95FyupjD9fb fV0j8brmwUwEoE15KGUfnX+nnXOEVuoybim2Y1hU544v7ZeYpjoJgoUFlEMf7v0coSGSUT WTcoJI08Jb/NWkDd56Tkf0UqyOLKb6c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1729233669; x=1760769669; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GA4zNC/BA7uVqXos9F/5h3uPYjHVZ8vkHQMHlvAZyyg=; b=caziR4+x5br7ZIBcSRHRETqWoOmqIONiPZ/XPKlF8B0ai+T9d+48P3Xh 2r1xnBZBxhiuEwwDC9CMN0yJXuToIpUdoJcj/d0V+95Zg6eRNisWFlYbO IXeHOMfsFB0h0lauJSOd5JSWOmDd4UyebkcwK0BnC0psW0x83gpEOqNCh QvcTUm8UiDwC2sYHjLcMe0ZuFdQSd97xW8tNax03yxALuD1BpBbMvCyq/ FxYZO2jhEDY2MjCz23mzjv7QYUA1DO1Fc2zOL37aCFnuBMNUY1xRs5rg1 oXRmfXIMolyi/FPEUlawacjKdMZjds9Qqhj4NETH0TiieEAArg6S9TEul Q==; X-CSE-ConnectionGUID: tU66HmiPRj+onFG3fzL74g== X-CSE-MsgGUID: TkBcT5fNT+iCQiUdk9heug== X-IronPort-AV: E=McAfee;i="6700,10204,11228"; a="28884878" X-IronPort-AV: E=Sophos;i="6.11,212,1725346800"; d="scan'208";a="28884878" Received: from orviesa003.jf.intel.com ([10.64.159.143]) by fmvoesa109.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2024 23:41:03 -0700 X-CSE-ConnectionGUID: zH0hGZmDQVO7EAYVusV2Hw== X-CSE-MsgGUID: ymW4e2aWT3qeiSwr7IcC9g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="83607514" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa003.jf.intel.com with ESMTP; 17 Oct 2024 23:41:03 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org, viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, mcgrof@kernel.org, kees@kernel.org, joel.granados@kernel.org, bfoster@redhat.com, willy@infradead.org, linux-fsdevel@vger.kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [RFC PATCH v1 07/13] crypto: iaa - Change cpu-to-iaa mappings to evenly balance cores to IAAs. Date: Thu, 17 Oct 2024 23:40:55 -0700 Message-Id: <20241018064101.336232-8-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241018064101.336232-1-kanchana.p.sridhar@intel.com> References: <20241018064101.336232-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CBBFA180011 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: odi3z839akxzrdrzcdcdp9wfnmq9szn3 X-HE-Tag: 1729233666-405874 X-HE-Meta: U2FsdGVkX18Sj7nA29iU/M8841Ow1a5+9YdRZWeV4Zq9OsnXzwahgHbM3kECff8deKgEvvSMHFVIY/9Xd0Gvtzo4TEL37sYEJjITSnDQN+77kBOo76j/QfD/ZuAhXZNLlHVPI8htVxaUNETz2HIHwOPQ1XMiciOb2ezEUVOejtvPXB0MYjgvhdmyoA25KSQIqA6X7EnExn4rfF3fOGQa12ys+D/Tuc6aveBT/xlJ6u3A2Aq7aVzbq36PO6YKQsZEL2Q1+5OA9ISDYA44xtbWy5sDX6UcMhG2npD4L1W7Rrz1BbAerJTtnQioIDoH4sYeFtiVCqUIOg+u8+zmJQm/o0jp4vMlAwis2CLk7pqy6DAYDcFz/bF0LgNIm2wKVCYej9Q13vL/nV2eZxkrmV3UflL5F8+4k/NlKsYyprry6QNkPpDjxRuX2qlAMBB+YGnzFArfEgpGr4ES1+lX7jEiVT4HIpdXwzUgkV3N1n00lBReYRdJSbGbg9j5EYaDJhuy7iZIDzyZNAR9BzAyaXN6b1uFgcx8650c+R03oiWKZt1bArSuT/4+j98NgrGOWQH0wXYVmwRKEtN1mth9qQpeTXe4dxyXmpfSZJR7TVlCe0EsFxY4E75MI8Na+72PX3AZt0mxLHcQVkb4yVf57BSCLviALe8sMg6U7sMAxSvOo0QA0FlE4JON14oJnxI02ipmH9MUbQao7VbJ5eCAj1N1opde0jzzqLEyX/sf34Ss2pnBQvXHXzIrScYyNAwGxfqxYTHt4i0P0v6lzlSnxDrWLGf71C2TpuyBZBK2zhFxhzOkcVs6XW0e6heYMDSA1wyGQjn8Z0VXOCX2FBGvmm03w7iFEsptOPb429JF/5R9r3sDB6zqID5akml2NDLD+bTb8aWU/NXn52RCCzOEcC7iWImkgCOLkROGwf1fOHmgj0fXQD42TbI2+lfQ7gpHJG0J4EbJyAS64AWVIBTeroT HgYdIryG eRsZgD4ik2UTJrrvAxgD+ZFqlLT3yoOV04IPrnEyVjBpdg2gHU80Rhh2TybzYjJ55elC6PkYs2f/8b2yKeRwMiTbaIjc+HzJTtopU81vBMC+UoVaG+pRlpCjCEBKZ5fW0ydVI9hm0/U7khpH2vFFcezDUeqzzm1iI+loLtKQiAC2/iODtG9KOPcPQ3w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This change distributes the cpus more evenly among the IAAs in each socket. Old algorithm to assign cpus to IAA: ------------------------------------ If "nr_cpus" = nr_logical_cpus (includes hyper-threading), the current algorithm determines "nr_cpus_per_node" = nr_cpus / nr_nodes. Hence, on a 2-socket Sapphire Rapids server where each socket has 56 cores and 4 IAA devices, nr_cpus_per_node = 112. Further, cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa Hence, cpus_per_iaa = 224/8 = 28. The iaa_crypto driver then assigns 28 "logical" node cpus per IAA device on that node, that results in this cpu-to-iaa mapping: lscpu|grep NUMA NUMA node(s): 2 NUMA node0 CPU(s): 0-55,112-167 NUMA node1 CPU(s): 56-111,168-223 NUMA node 0: cpu 0-27 28-55 112-139 140-167 iaa iax1 iax3 iax5 iax7 NUMA node 1: cpu 56-83 84-111 168-195 196-223 iaa iax9 iax11 iax13 iax15 This appears non-optimal for a few reasons: 1) The 2 logical threads on a core will get assigned to different IAA devices. For e.g.: cpu 0: iax1 cpu 112: iax5 2) One of the logical threads on a core is assigned to an IAA that is not closest to that core. For e.g. cpu 112. 3) If numactl is used to start processes sequentially on the logical cores, some of the IAA devices on the socket could be over-subscribed, while some could be under-utilized. This patch introduces a scheme to more evenly balance the logical cores to IAA devices on a socket. New algorithm to assign cpus to IAA: ------------------------------------ We introduce a function "cpu_to_iaa()" that takes a logical cpu and returns the IAA device closest to it. If "nr_cpus" = nr_logical_cpus (includes hyper-threading), the new algorithm determines "nr_cpus_per_node" = topology_num_cores_per_package(). Hence, on a 2-socket Sapphire Rapids server where each socket has 56 cores and 4 IAA devices, nr_cpus_per_node = 56. Further, cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa Hence, cpus_per_iaa = 112/8 = 14. The iaa_crypto driver then assigns 14 "logical" node cpus per IAA device on that node, that results in this cpu-to-iaa mapping: NUMA node 0: cpu 0-13,112-125 14-27,126-139 28-41,140-153 42-55,154-167 iaa iax1 iax3 iax5 iax7 NUMA node 1: cpu 56-69,168-181 70-83,182-195 84-97,196-209 98-111,210-223 iaa iax9 iax11 iax13 iax15 This resolves the 3 issues with non-optimality of cpu-to-iaa mappings pointed out earlier with the existing approach. Originally-by: Tom Zanussi Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 84 ++++++++++++++-------- 1 file changed, 54 insertions(+), 30 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 8e6859c97970..c854a7a1aaa4 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -55,6 +55,46 @@ static struct idxd_wq *wq_table_next_wq(int cpu) return entry->wqs[entry->cur_wq]; } +/* + * Given a cpu, find the closest IAA instance. The idea is to try to + * choose the most appropriate IAA instance for a caller and spread + * available workqueues around to clients. + */ +static inline int cpu_to_iaa(int cpu) +{ + int node, n_cpus = 0, test_cpu, iaa = 0; + int nr_iaa_per_node; + const struct cpumask *node_cpus; + + if (!nr_nodes) + return 0; + + nr_iaa_per_node = nr_iaa / nr_nodes; + if (!nr_iaa_per_node) + return 0; + + for_each_online_node(node) { + node_cpus = cpumask_of_node(node); + if (!cpumask_test_cpu(cpu, node_cpus)) + continue; + + for_each_cpu(test_cpu, node_cpus) { + if ((n_cpus % nr_cpus_per_node) == 0) + iaa = node * nr_iaa_per_node; + + if (test_cpu == cpu) + return iaa; + + n_cpus++; + + if ((n_cpus % cpus_per_iaa) == 0) + iaa++; + } + } + + return -1; +} + static void wq_table_add(int cpu, struct idxd_wq *wq) { struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); @@ -895,8 +935,7 @@ static int wq_table_add_wqs(int iaa, int cpu) */ static void rebalance_wq_table(void) { - const struct cpumask *node_cpus; - int node, cpu, iaa = -1; + int cpu, iaa; if (nr_iaa == 0) return; @@ -906,37 +945,22 @@ static void rebalance_wq_table(void) clear_wq_table(); - if (nr_iaa == 1) { - for (cpu = 0; cpu < nr_cpus; cpu++) { - if (WARN_ON(wq_table_add_wqs(0, cpu))) { - pr_debug("could not add any wqs for iaa 0 to cpu %d!\n", cpu); - return; - } - } - - return; - } - - for_each_node_with_cpus(node) { - node_cpus = cpumask_of_node(node); - - for (cpu = 0; cpu < cpumask_weight(node_cpus); cpu++) { - int node_cpu = cpumask_nth(cpu, node_cpus); - - if (WARN_ON(node_cpu >= nr_cpu_ids)) { - pr_debug("node_cpu %d doesn't exist!\n", node_cpu); - return; - } + for (cpu = 0; cpu < nr_cpus; cpu++) { + iaa = cpu_to_iaa(cpu); + pr_debug("rebalance: cpu=%d iaa=%d\n", cpu, iaa); - if ((cpu % cpus_per_iaa) == 0) - iaa++; + if (WARN_ON(iaa == -1)) { + pr_debug("rebalance (cpu_to_iaa(%d)) failed!\n", cpu); + return; + } - if (WARN_ON(wq_table_add_wqs(iaa, node_cpu))) { - pr_debug("could not add any wqs for iaa %d to cpu %d!\n", iaa, cpu); - return; - } + if (WARN_ON(wq_table_add_wqs(iaa, cpu))) { + pr_debug("could not add any wqs for iaa %d to cpu %d!\n", iaa, cpu); + return; } } + + pr_debug("Finished rebalance local wqs."); } static inline int check_completion(struct device *dev, @@ -2084,7 +2108,7 @@ static int __init iaa_crypto_init_module(void) pr_err("IAA couldn't find any nodes with cpus\n"); return -ENODEV; } - nr_cpus_per_node = nr_cpus / nr_nodes; + nr_cpus_per_node = topology_num_cores_per_package(); if (crypto_has_comp("deflate-generic", 0, 0)) deflate_generic_tfm = crypto_alloc_comp("deflate-generic", 0, 0);