From patchwork Wed Oct 13 04:58:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gavin Shan X-Patchwork-Id: 12554461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7073DC433EF for ; Wed, 13 Oct 2021 05:00:05 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 191B260C4D for ; Wed, 13 Oct 2021 05:00:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 191B260C4D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:36326 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1maWN2-0005qx-4y for qemu-devel@archiver.kernel.org; Wed, 13 Oct 2021 01:00:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47110) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maWLa-0002XX-Vx for qemu-devel@nongnu.org; Wed, 13 Oct 2021 00:58:35 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:32087) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maWLY-0004mV-Mu for qemu-devel@nongnu.org; Wed, 13 Oct 2021 00:58:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1634101111; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L4YegJM/qszkErTdodpTlQGO+o8Lz2xnY4Y8/EmBw8M=; b=Xts+WsmUNFVpNT/+NMn6QTlzKyIy4sZLxWSsCUwsjPtucM0ZKSIecH8aHmLOfSGF2mVxBQ j9o25wwcnvO86qzc4oa1BQJIrbH1Iz4SP7I38OgV+FAMbkKubHbw1QLPvCRM1P2EkYLsOE xQuOvKucB+Rc9DA8neOr+0UugDO0MSM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-595-m_q7Y-WXPeCfTTxqRjPNgQ-1; Wed, 13 Oct 2021 00:58:24 -0400 X-MC-Unique: m_q7Y-WXPeCfTTxqRjPNgQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2750C10A8E01; Wed, 13 Oct 2021 04:58:23 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-56.bne.redhat.com [10.64.54.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3E04F5D9C6; Wed, 13 Oct 2021 04:58:20 +0000 (UTC) From: Gavin Shan To: qemu-arm@nongnu.org Subject: [PATCH v3 1/2] numa: Require distance map when empty node exists Date: Wed, 13 Oct 2021 12:58:04 +0800 Message-Id: <20211013045805.192165-2-gshan@redhat.com> In-Reply-To: <20211013045805.192165-1-gshan@redhat.com> References: <20211013045805.192165-1-gshan@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=gshan@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -8 X-Spam_score: -0.9 X-Spam_bar: / X-Spam_report: (-0.9 / 5.0 requ) DKIMWL_WL_HIGH=-0.049, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robh@kernel.org, drjones@redhat.com, ehabkost@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org, shan.gavin@gmail.com, imammedo@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The following option is used to specify the distance map. It's possible the option isn't provided by user. In this case, the distance map isn't populated and exposed to platform. On the other hand, the empty NUMA node, where no memory resides, is allowed on platforms like ARM64 virt. For these empty NUMA nodes, their corresponding device-tree nodes aren't populated, but their NUMA IDs should be included in the "/distance-map" device-tree node, so that kernel can probe them properly if device-tree is used. -numa,dist,src=,dst=,val= This adds extra check after the machine is initialized, to ask for the distance map from user when empty nodes exist in device-tree. Signed-off-by: Gavin Shan Reviewed-by: Andrew Jones --- hw/core/machine.c | 4 ++++ hw/core/numa.c | 24 ++++++++++++++++++++++++ include/sysemu/numa.h | 1 + 3 files changed, 29 insertions(+) diff --git a/hw/core/machine.c b/hw/core/machine.c index b8d95eec32..c0765ad973 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -1355,6 +1355,10 @@ void machine_run_board_init(MachineState *machine) accel_init_interfaces(ACCEL_GET_CLASS(machine->accelerator)); machine_class->init(machine); phase_advance(PHASE_MACHINE_INITIALIZED); + + if (machine->numa_state) { + numa_complete_validation(machine); + } } static NotifierList machine_init_done_notifiers = diff --git a/hw/core/numa.c b/hw/core/numa.c index 510d096a88..7404b7dd38 100644 --- a/hw/core/numa.c +++ b/hw/core/numa.c @@ -727,6 +727,30 @@ void numa_complete_configuration(MachineState *ms) } } +/* + * When device-tree is used by the machine, the empty node IDs should + * be included in the distance map. So we need provide pairs of distances + * in this case. + */ +void numa_complete_validation(MachineState *ms) +{ + NodeInfo *numa_info = ms->numa_state->nodes; + int nb_numa_nodes = ms->numa_state->num_nodes; + int i; + + if (!ms->fdt || ms->numa_state->have_numa_distance) { + return; + } + + for (i = 0; i < nb_numa_nodes; i++) { + if (numa_info[i].present && !numa_info[i].node_mem) { + error_report("Empty node %d found, please provide " + "distance map.", i); + exit(EXIT_FAILURE); + } + } +} + void parse_numa_opts(MachineState *ms) { qemu_opts_foreach(qemu_find_opts("numa"), parse_numa, ms, &error_fatal); diff --git a/include/sysemu/numa.h b/include/sysemu/numa.h index 4173ef2afa..80f25ab830 100644 --- a/include/sysemu/numa.h +++ b/include/sysemu/numa.h @@ -104,6 +104,7 @@ void parse_numa_hmat_lb(NumaState *numa_state, NumaHmatLBOptions *node, void parse_numa_hmat_cache(MachineState *ms, NumaHmatCacheOptions *node, Error **errp); void numa_complete_configuration(MachineState *ms); +void numa_complete_validation(MachineState *ms); void query_numa_node_mem(NumaNodeMem node_mem[], MachineState *ms); extern QemuOptsList qemu_numa_opts; void numa_cpu_pre_plug(const struct CPUArchId *slot, DeviceState *dev, From patchwork Wed Oct 13 04:58:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gavin Shan X-Patchwork-Id: 12554459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AA23C433F5 for ; Wed, 13 Oct 2021 04:59:47 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CB84660C4D for ; Wed, 13 Oct 2021 04:59:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org CB84660C4D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:35488 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1maWMj-0005Il-Ui for qemu-devel@archiver.kernel.org; Wed, 13 Oct 2021 00:59:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47080) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maWLZ-0002TI-5F for qemu-devel@nongnu.org; Wed, 13 Oct 2021 00:58:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:48739) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maWLX-0004mN-LR for qemu-devel@nongnu.org; Wed, 13 Oct 2021 00:58:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1634101110; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aD8nzljPsEKJ570RR+jldn/7/HXpuLhVnorn+6ome1Q=; b=RUhimNnHRRGx27IravF/QFtdVAtulGM7lxub1+6lKt35IzJQFoO4PwXTwNT+5h0uI/J7Xe rGuvdkgwAfBNPXCfFAebS0fRGhwvksZpbohJasOr57nK0K5VIYKF1or5ctkxNLViZ3AMwq 8TvCdr4k3MqmYSIFncILV5Hj+BIauA4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-512-EBzFnG8fOomtjiAmD_6dKg-1; Wed, 13 Oct 2021 00:58:27 -0400 X-MC-Unique: EBzFnG8fOomtjiAmD_6dKg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6058680363B; Wed, 13 Oct 2021 04:58:26 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-56.bne.redhat.com [10.64.54.56]) by smtp.corp.redhat.com (Postfix) with ESMTP id B132E5D9C6; Wed, 13 Oct 2021 04:58:23 +0000 (UTC) From: Gavin Shan To: qemu-arm@nongnu.org Subject: [PATCH v3 2/2] hw/arm/virt: Don't create device-tree node for empty NUMA node Date: Wed, 13 Oct 2021 12:58:05 +0800 Message-Id: <20211013045805.192165-3-gshan@redhat.com> In-Reply-To: <20211013045805.192165-1-gshan@redhat.com> References: <20211013045805.192165-1-gshan@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=gshan@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=gshan@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.049, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robh@kernel.org, drjones@redhat.com, ehabkost@redhat.com, peter.maydell@linaro.org, qemu-devel@nongnu.org, shan.gavin@gmail.com, imammedo@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The empty NUMA node, where no memory resides, are allowed. For example, the following command line specifies two empty NUMA nodes. With this, QEMU fails to boot because of the conflicting device-tree node names, as the following error message indicates. /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \ -accel kvm -machine virt,gic-version=host \ -cpu host -smp 4,sockets=2,cores=2,threads=1 \ -m 1024M,slots=16,maxmem=64G \ -object memory-backend-ram,id=mem0,size=512M \ -object memory-backend-ram,id=mem1,size=512M \ -numa node,nodeid=0,cpus=0-1,memdev=mem0 \ -numa node,nodeid=1,cpus=2-3,memdev=mem1 \ -numa node,nodeid=2 \ -numa node,nodeid=3 : qemu-system-aarch64: FDT: Failed to create subnode /memory@80000000: FDT_ERR_EXISTS As specified by linux device-tree binding document, the device-tree nodes for these empty NUMA nodes shouldn't be generated. However, the corresponding NUMA node IDs should be included in the distance map device-tree node. This skips populating the device-tree nodes for these empty NUMA nodes to avoid the error, so that QEMU can be started successfully. Signed-off-by: Gavin Shan Reviewed-by: Andrew Jones --- hw/arm/boot.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hw/arm/boot.c b/hw/arm/boot.c index 57efb61ee4..4e5898fcdc 100644 --- a/hw/arm/boot.c +++ b/hw/arm/boot.c @@ -603,6 +603,10 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, mem_base = binfo->loader_start; for (i = 0; i < ms->numa_state->num_nodes; i++) { mem_len = ms->numa_state->nodes[i].node_mem; + if (!mem_len) { + continue; + } + rc = fdt_add_memory_node(fdt, acells, mem_base, scells, mem_len, i); if (rc < 0) {