From patchwork Thu Jan 5 14:53:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pierre Morel X-Patchwork-Id: 13090069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 36E16C3DA7D for ; Thu, 5 Jan 2023 14:56:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pDRcf-0005lw-MH; Thu, 05 Jan 2023 09:53:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pDRcd-0005gx-Hu; Thu, 05 Jan 2023 09:53:35 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pDRcb-0004to-67; Thu, 05 Jan 2023 09:53:35 -0500 Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 305EpjjH022529; Thu, 5 Jan 2023 14:53:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=yM9j2uyN7Bpwo18nIstSzp/AMscgnmpHRKPEc1HY7XY=; b=sj5QunvCgQ1X0rJcbj+G4QN7iVSZxAf83ERDbNfsKP2ypDALPGKSYojFPMlHNZ77yJ3A /p0LKV8bpedp96ojrfQyQUJgez4IZLX2gXF7dssjBI6xl9Ow9z/I4/yPh5ywYRX40x3j 48Ski/sYLkNhnZ7A1G66BfF+8ia9+t5X7qkN97sYQ+3mPeXw0Tj0/j0cvw5factz95fz Gmz6HplnNKbQITr/NoIuPrXMduxCh+3IxSqZdXlbpQxg8PodxcuB5ZXmswdHwEV1YDDz ReIcIl0Bcqbr/yA/8c1IJUHP3MAstvyUXuw7WHLiVDKZp2MyMlEz8PYRLkl7TBhOwhuJ og== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3mx0pfg136-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 05 Jan 2023 14:53:24 +0000 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 305Er4jY028552; Thu, 5 Jan 2023 14:53:23 GMT Received: from ppma05fra.de.ibm.com (6c.4a.5195.ip4.static.sl-reverse.com [149.81.74.108]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3mx0pfg12e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 05 Jan 2023 14:53:23 +0000 Received: from pps.filterd (ppma05fra.de.ibm.com [127.0.0.1]) by ppma05fra.de.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3059IZQ3026929; Thu, 5 Jan 2023 14:53:21 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma05fra.de.ibm.com (PPS) with ESMTPS id 3mtcq6n2yj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 05 Jan 2023 14:53:21 +0000 Received: from smtpav01.fra02v.mail.ibm.com (smtpav01.fra02v.mail.ibm.com [10.20.54.100]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 305ErHOf20447610 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 5 Jan 2023 14:53:18 GMT Received: from smtpav01.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D234C2004B; Thu, 5 Jan 2023 14:53:16 +0000 (GMT) Received: from smtpav01.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CA2F320049; Thu, 5 Jan 2023 14:53:15 +0000 (GMT) Received: from li-c6ac47cc-293c-11b2-a85c-d421c8e4747b.ibm.com.com (unknown [9.171.26.113]) by smtpav01.fra02v.mail.ibm.com (Postfix) with ESMTP; Thu, 5 Jan 2023 14:53:15 +0000 (GMT) From: Pierre Morel To: qemu-s390x@nongnu.org Cc: qemu-devel@nongnu.org, borntraeger@de.ibm.com, pasic@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, mst@redhat.com, pbonzini@redhat.com, kvm@vger.kernel.org, ehabkost@redhat.com, marcel.apfelbaum@gmail.com, eblake@redhat.com, armbru@redhat.com, seiden@linux.ibm.com, nrb@linux.ibm.com, scgl@linux.ibm.com, frankja@linux.ibm.com, berrange@redhat.com, clg@kaod.org Subject: [PATCH v14 01/11] s390x/cpu topology: adding s390 specificities to CPU topology Date: Thu, 5 Jan 2023 15:53:03 +0100 Message-Id: <20230105145313.168489-2-pmorel@linux.ibm.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230105145313.168489-1-pmorel@linux.ibm.com> References: <20230105145313.168489-1-pmorel@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: -QljFR5Ep3TiHbzifM9d0O0eKlpxH-Gp X-Proofpoint-GUID: JglB97A7WW5t8aLdViL2vS-oKDfI2wro X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2023-01-05_05,2023-01-05_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxlogscore=999 clxscore=1015 phishscore=0 lowpriorityscore=0 suspectscore=0 malwarescore=0 mlxscore=0 bulkscore=0 spamscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301050114 Received-SPF: pass client-ip=148.163.158.5; envelope-from=pmorel@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org S390 adds two new SMP levels, drawers and books to the CPU topology. The S390 CPU have specific toplogy features like dedication and polarity to give to the guest indications on the host vCPUs scheduling and help the guest take the best decisions on the scheduling of threads on the vCPUs. Let us provide the SMP properties with books and drawers levels and S390 CPU with dedication and polarity, Signed-off-by: Pierre Morel --- qapi/machine.json | 14 ++++++++-- include/hw/boards.h | 10 ++++++- include/hw/s390x/cpu-topology.h | 23 ++++++++++++++++ target/s390x/cpu.h | 6 +++++ hw/core/machine-smp.c | 48 ++++++++++++++++++++++++++++----- hw/core/machine.c | 4 +++ hw/s390x/s390-virtio-ccw.c | 2 ++ softmmu/vl.c | 6 +++++ target/s390x/cpu.c | 10 +++++++ qemu-options.hx | 6 +++-- 10 files changed, 117 insertions(+), 12 deletions(-) create mode 100644 include/hw/s390x/cpu-topology.h diff --git a/qapi/machine.json b/qapi/machine.json index b9228a5e46..ff8f2b0e84 100644 --- a/qapi/machine.json +++ b/qapi/machine.json @@ -900,13 +900,15 @@ # a CPU is being hotplugged. # # @node-id: NUMA node ID the CPU belongs to -# @socket-id: socket number within node/board the CPU belongs to +# @drawer-id: drawer number within node/board the CPU belongs to +# @book-id: book number within drawer/node/board the CPU belongs to +# @socket-id: socket number within book/node/board the CPU belongs to # @die-id: die number within socket the CPU belongs to (since 4.1) # @cluster-id: cluster number within die the CPU belongs to (since 7.1) # @core-id: core number within cluster the CPU belongs to # @thread-id: thread number within core the CPU belongs to # -# Note: currently there are 6 properties that could be present +# Note: currently there are 8 properties that could be present # but management should be prepared to pass through other # properties with device_add command to allow for future # interface extension. This also requires the filed names to be kept in @@ -916,6 +918,8 @@ ## { 'struct': 'CpuInstanceProperties', 'data': { '*node-id': 'int', + '*drawer-id': 'int', + '*book-id': 'int', '*socket-id': 'int', '*die-id': 'int', '*cluster-id': 'int', @@ -1465,6 +1469,10 @@ # # @cpus: number of virtual CPUs in the virtual machine # +# @drawers: number of drawers in the CPU topology +# +# @books: number of books in the CPU topology +# # @sockets: number of sockets in the CPU topology # # @dies: number of dies per socket in the CPU topology @@ -1481,6 +1489,8 @@ ## { 'struct': 'SMPConfiguration', 'data': { '*cpus': 'int', + '*drawers': 'int', + '*books': 'int', '*sockets': 'int', '*dies': 'int', '*clusters': 'int', diff --git a/include/hw/boards.h b/include/hw/boards.h index d18d6d0073..239fc017b0 100644 --- a/include/hw/boards.h +++ b/include/hw/boards.h @@ -130,11 +130,15 @@ typedef struct { * @prefer_sockets - whether sockets are preferred over cores in smp parsing * @dies_supported - whether dies are supported by the machine * @clusters_supported - whether clusters are supported by the machine + * @books_supported - whether books are supported by the machine + * @drawers_supported - whether drawers are supported by the machine */ typedef struct { bool prefer_sockets; bool dies_supported; bool clusters_supported; + bool books_supported; + bool drawers_supported; } SMPCompatProps; /** @@ -299,7 +303,9 @@ typedef struct DeviceMemoryState { /** * CpuTopology: * @cpus: the number of present logical processors on the machine - * @sockets: the number of sockets on the machine + * @drawers: the number of drawers on the machine + * @books: the number of books in one drawer + * @sockets: the number of sockets in one book * @dies: the number of dies in one socket * @clusters: the number of clusters in one die * @cores: the number of cores in one cluster @@ -308,6 +314,8 @@ typedef struct DeviceMemoryState { */ typedef struct CpuTopology { unsigned int cpus; + unsigned int drawers; + unsigned int books; unsigned int sockets; unsigned int dies; unsigned int clusters; diff --git a/include/hw/s390x/cpu-topology.h b/include/hw/s390x/cpu-topology.h new file mode 100644 index 0000000000..d945b57fc3 --- /dev/null +++ b/include/hw/s390x/cpu-topology.h @@ -0,0 +1,23 @@ +/* + * CPU Topology + * + * Copyright IBM Corp. 2022 + * + * This work is licensed under the terms of the GNU GPL, version 2 or (at + * your option) any later version. See the COPYING file in the top-level + * directory. + */ +#ifndef HW_S390X_CPU_TOPOLOGY_H +#define HW_S390X_CPU_TOPOLOGY_H + +#define S390_TOPOLOGY_CPU_IFL 0x03 + +#define S390_TOPOLOGY_POLARITY_HORIZONTAL 0x00 +#define S390_TOPOLOGY_POLARITY_VERTICAL_LOW 0x01 +#define S390_TOPOLOGY_POLARITY_VERTICAL_MEDIUM 0x02 +#define S390_TOPOLOGY_POLARITY_VERTICAL_HIGH 0x03 + +#define S390_TOPOLOGY_SHARED 0x00 +#define S390_TOPOLOGY_DEDICATED 0x01 + +#endif diff --git a/target/s390x/cpu.h b/target/s390x/cpu.h index 7d6d01325b..39ea63a416 100644 --- a/target/s390x/cpu.h +++ b/target/s390x/cpu.h @@ -131,6 +131,12 @@ struct CPUArchState { #if !defined(CONFIG_USER_ONLY) uint32_t core_id; /* PoP "CPU address", same as cpu_index */ + int32_t socket_id; + int32_t book_id; + int32_t drawer_id; + int32_t dedicated; + int32_t polarity; + int32_t cpu_type; uint64_t cpuid; #endif diff --git a/hw/core/machine-smp.c b/hw/core/machine-smp.c index b39ed21e65..edabfdb67e 100644 --- a/hw/core/machine-smp.c +++ b/hw/core/machine-smp.c @@ -31,6 +31,14 @@ static char *cpu_hierarchy_to_string(MachineState *ms) MachineClass *mc = MACHINE_GET_CLASS(ms); GString *s = g_string_new(NULL); + if (mc->smp_props.drawers_supported) { + g_string_append_printf(s, " * drawers (%u)", ms->smp.drawers); + } + + if (mc->smp_props.books_supported) { + g_string_append_printf(s, " * books (%u)", ms->smp.books); + } + g_string_append_printf(s, "sockets (%u)", ms->smp.sockets); if (mc->smp_props.dies_supported) { @@ -73,6 +81,8 @@ void machine_parse_smp_config(MachineState *ms, { MachineClass *mc = MACHINE_GET_CLASS(ms); unsigned cpus = config->has_cpus ? config->cpus : 0; + unsigned drawers = config->has_drawers ? config->drawers : 0; + unsigned books = config->has_books ? config->books : 0; unsigned sockets = config->has_sockets ? config->sockets : 0; unsigned dies = config->has_dies ? config->dies : 0; unsigned clusters = config->has_clusters ? config->clusters : 0; @@ -85,6 +95,8 @@ void machine_parse_smp_config(MachineState *ms, * explicit configuration like "cpus=0" is not allowed. */ if ((config->has_cpus && config->cpus == 0) || + (config->has_drawers && config->drawers == 0) || + (config->has_books && config->books == 0) || (config->has_sockets && config->sockets == 0) || (config->has_dies && config->dies == 0) || (config->has_clusters && config->clusters == 0) || @@ -111,6 +123,19 @@ void machine_parse_smp_config(MachineState *ms, dies = dies > 0 ? dies : 1; clusters = clusters > 0 ? clusters : 1; + if (!mc->smp_props.books_supported && books > 1) { + error_setg(errp, "books not supported by this machine's CPU topology"); + return; + } + books = books > 0 ? books : 1; + + if (!mc->smp_props.drawers_supported && drawers > 1) { + error_setg(errp, + "drawers not supported by this machine's CPU topology"); + return; + } + drawers = drawers > 0 ? drawers : 1; + /* compute missing values based on the provided ones */ if (cpus == 0 && maxcpus == 0) { sockets = sockets > 0 ? sockets : 1; @@ -124,33 +149,41 @@ void machine_parse_smp_config(MachineState *ms, if (sockets == 0) { cores = cores > 0 ? cores : 1; threads = threads > 0 ? threads : 1; - sockets = maxcpus / (dies * clusters * cores * threads); + sockets = maxcpus / + (drawers * books * dies * clusters * cores * threads); } else if (cores == 0) { threads = threads > 0 ? threads : 1; - cores = maxcpus / (sockets * dies * clusters * threads); + cores = maxcpus / + (drawers * books * sockets * dies * clusters * threads); } } else { /* prefer cores over sockets since 6.2 */ if (cores == 0) { sockets = sockets > 0 ? sockets : 1; threads = threads > 0 ? threads : 1; - cores = maxcpus / (sockets * dies * clusters * threads); + cores = maxcpus / + (drawers * books * sockets * dies * clusters * threads); } else if (sockets == 0) { threads = threads > 0 ? threads : 1; - sockets = maxcpus / (dies * clusters * cores * threads); + sockets = maxcpus / + (drawers * books * dies * clusters * cores * threads); } } /* try to calculate omitted threads at last */ if (threads == 0) { - threads = maxcpus / (sockets * dies * clusters * cores); + threads = maxcpus / + (drawers * books * sockets * dies * clusters * cores); } } - maxcpus = maxcpus > 0 ? maxcpus : sockets * dies * clusters * cores * threads; + maxcpus = maxcpus > 0 ? maxcpus : drawers * books * sockets * dies * + clusters * cores * threads; cpus = cpus > 0 ? cpus : maxcpus; ms->smp.cpus = cpus; + ms->smp.drawers = drawers; + ms->smp.books = books; ms->smp.sockets = sockets; ms->smp.dies = dies; ms->smp.clusters = clusters; @@ -159,7 +192,8 @@ void machine_parse_smp_config(MachineState *ms, ms->smp.max_cpus = maxcpus; /* sanity-check of the computed topology */ - if (sockets * dies * clusters * cores * threads != maxcpus) { + if (drawers * books * sockets * dies * clusters * cores * threads != + maxcpus) { g_autofree char *topo_msg = cpu_hierarchy_to_string(ms); error_setg(errp, "Invalid CPU topology: " "product of the hierarchy must match maxcpus: " diff --git a/hw/core/machine.c b/hw/core/machine.c index f589b92909..3c781fed07 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -827,6 +827,8 @@ static void machine_get_smp(Object *obj, Visitor *v, const char *name, MachineState *ms = MACHINE(obj); SMPConfiguration *config = &(SMPConfiguration){ .has_cpus = true, .cpus = ms->smp.cpus, + .has_drawers = true, .drawers = ms->smp.drawers, + .has_books = true, .books = ms->smp.books, .has_sockets = true, .sockets = ms->smp.sockets, .has_dies = true, .dies = ms->smp.dies, .has_clusters = true, .clusters = ms->smp.clusters, @@ -1092,6 +1094,8 @@ static void machine_initfn(Object *obj) /* default to mc->default_cpus */ ms->smp.cpus = mc->default_cpus; ms->smp.max_cpus = mc->default_cpus; + ms->smp.drawers = 1; + ms->smp.books = 1; ms->smp.sockets = 1; ms->smp.dies = 1; ms->smp.clusters = 1; diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index f22f61b8b6..f3cc845d3b 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -733,6 +733,8 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data) mc->no_sdcard = 1; mc->max_cpus = S390_MAX_CPUS; mc->has_hotpluggable_cpus = true; + mc->smp_props.books_supported = true; + mc->smp_props.drawers_supported = true; assert(!mc->get_hotplug_handler); mc->get_hotplug_handler = s390_get_hotplug_handler; mc->cpu_index_to_instance_props = s390_cpu_index_to_props; diff --git a/softmmu/vl.c b/softmmu/vl.c index 798e1dc933..e894ab32b0 100644 --- a/softmmu/vl.c +++ b/softmmu/vl.c @@ -724,6 +724,12 @@ static QemuOptsList qemu_smp_opts = { { .name = "cpus", .type = QEMU_OPT_NUMBER, + }, { + .name = "drawers", + .type = QEMU_OPT_NUMBER, + }, { + .name = "books", + .type = QEMU_OPT_NUMBER, }, { .name = "sockets", .type = QEMU_OPT_NUMBER, diff --git a/target/s390x/cpu.c b/target/s390x/cpu.c index 96562c516d..c9f980dbb5 100644 --- a/target/s390x/cpu.c +++ b/target/s390x/cpu.c @@ -35,6 +35,7 @@ #include "fpu/softfloat-helpers.h" #include "disas/capstone.h" #include "sysemu/tcg.h" +#include "hw/s390x/cpu-topology.h" #define CR0_RESET 0xE0UL #define CR14_RESET 0xC2000000UL; @@ -257,6 +258,15 @@ static gchar *s390_gdb_arch_name(CPUState *cs) static Property s390x_cpu_properties[] = { #if !defined(CONFIG_USER_ONLY) DEFINE_PROP_UINT32("core-id", S390CPU, env.core_id, 0), + DEFINE_PROP_INT32("socket-id", S390CPU, env.socket_id, -1), + DEFINE_PROP_INT32("book-id", S390CPU, env.book_id, -1), + DEFINE_PROP_INT32("drawer-id", S390CPU, env.drawer_id, -1), + DEFINE_PROP_INT32("dedicated", S390CPU, env.dedicated, + S390_TOPOLOGY_SHARED), + DEFINE_PROP_INT32("polarity", S390CPU, env.polarity, + S390_TOPOLOGY_POLARITY_HORIZONTAL), + DEFINE_PROP_INT32("cpu-type", S390CPU, env.cpu_type, + S390_TOPOLOGY_CPU_IFL), #endif DEFINE_PROP_END_OF_LIST() }; diff --git a/qemu-options.hx b/qemu-options.hx index 7f99d15b23..8dc9a4c052 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -250,11 +250,13 @@ SRST ERST DEF("smp", HAS_ARG, QEMU_OPTION_smp, - "-smp [[cpus=]n][,maxcpus=maxcpus][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n" + "-smp [[cpus=]n][,maxcpus=maxcpus][,drawers=drawers][,books=books][,sockets=sockets][,dies=dies][,clusters=clusters][,cores=cores][,threads=threads]\n" " set the number of initial CPUs to 'n' [default=1]\n" " maxcpus= maximum number of total CPUs, including\n" " offline CPUs for hotplug, etc\n" - " sockets= number of sockets on the machine board\n" + " drawers= number of drawers on the machine board\n" + " books= number of books in one drawer\n" + " sockets= number of sockets in one book\n" " dies= number of dies in one socket\n" " clusters= number of clusters in one die\n" " cores= number of cores in one cluster\n"