From patchwork Fri Aug 5 14:16:13 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Covington X-Patchwork-Id: 9265457 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 843676048B for ; Fri, 5 Aug 2016 14:18:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7609828455 for ; Fri, 5 Aug 2016 14:18:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A84328451; Fri, 5 Aug 2016 14:18:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 064F92844A for ; Fri, 5 Aug 2016 14:18:27 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.85_2 #1 (Red Hat Linux)) id 1bVfvh-0002ln-0r; Fri, 05 Aug 2016 14:16:53 +0000 Received: from smtp.codeaurora.org ([198.145.29.96]) by bombadil.infradead.org with esmtps (Exim 4.85_2 #1 (Red Hat Linux)) id 1bVfva-0002YJ-9f for linux-arm-kernel@lists.infradead.org; Fri, 05 Aug 2016 14:16:48 +0000 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id 44AAC6135F; Fri, 5 Aug 2016 14:16:25 +0000 (UTC) Received: from [10.228.68.87] (global_nat1_iad_fw.qualcomm.com [129.46.232.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: cov@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id EADBA602A8; Fri, 5 Aug 2016 14:16:15 +0000 (UTC) Subject: Re: sysfs topology for arm64 cluster_id To: Stuart Yoder , Jon Masters , Mark Rutland , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , Don Dutile References: <54B5BC84.8090603@redhat.com> <20150114170045.GC21115@leverpostej> <54B6A4EF.7020501@redhat.com> From: Christopher Covington Message-ID: <57A49FAD.5090004@codeaurora.org> Date: Fri, 5 Aug 2016 10:16:13 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 Thunderbird/43.0 MIME-Version: 1.0 In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160805_071646_425357_EC8D5E49 X-CRM114-Status: GOOD ( 25.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Langsdorf , Vikram Sethi , Catalin Marinas , Will Deacon , Peter Newton , Mark Brown , Mark Salter , Shanker Donthineni Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Stuart, On 07/01/2016 11:54 AM, Stuart Yoder wrote: > Re-opening a thread from back in early 2015... > >> -----Original Message----- >> From: Jon Masters >> Date: Wed, Jan 14, 2015 at 11:18 AM >> Subject: Re: sysfs topology for arm64 cluster_id >> To: Mark Rutland >> Cc: "linux-arm-kernel@lists.infradead.org" >> , "linux-kernel@vger.kernel.org" >> , Don Dutile >> >> >> On 01/14/2015 12:00 PM, Mark Rutland wrote: >>> On Wed, Jan 14, 2015 at 12:47:00AM +0000, Jon Masters wrote: >>>> Hi Folks, >>>> >>>> TLDR: I would like to consider the value of adding something like >>>> "cluster_siblings" or similar in sysfs to describe ARM topology. >>>> >>>> A quick question on intended data representation in /sysfs topology >>>> before I ask the team on this end to go down the (wrong?) path. On ARM >>>> systems today, we have a hierarchical CPU topology: >>>> >>>> Socket ---- Coherent Interonnect ---- Socket >>>> | | >>>> Cluster0 ... ClusterN Cluster0 ... ClusterN >>>> | | | | >>>> Core0...CoreN Core0...CoreN Core0...CoreN Core0...CoreN >>>> | | | | | | | | >>>> T0..TN T0..Tn T0..TN T0..TN T0..TN T0..TN T0..TN T0..TN >>>> >>>> Where we might (or might not) have threads in individual cores (a la SMT >>>> - it's allowed in the architecture at any rate) and we group cores >>>> together into units of clusters usually 2-4 cores in size (though this >>>> varies between implementations, some of which have different but similar >>>> concepts, such as AppliedMicro Potenza PMDs CPU complexes of dual >>>> cores). There are multiple clusters per "socket", and there might be an >>>> arbitrary number of sockets. We'll start to enable NUMA soon. >>> >>> I have a slight disagreement with the diagram above. >> >> Thanks for the clarification - note that I was *explicitly not* saying >> that the MPIDR Affinity bits sufficiently described the system :) Nor do >> I think cpu-map does cover everything we want today. >> >>> The MPIDR_EL1.Aff* fields and the cpu-map bindings currently only >>> describe the hierarchy, without any information on the relative >>> weighting between levels, and without any mapping to HW concepts such as >>> sockets. What these happen to map to is specific to a particular system, >>> and the hierarchy may be carved up in a number of possible ways >>> (including "virtual" clusters). There are also 24 RES0 bits that could >>> potentially become additional Aff fields we may need to describe in >>> future. >> >>> "socket", "package", etc are meaningless unless the system provides a >>> mapping of Aff levels to these. We can't guess how the HW is actually >>> organised. >> >> The replies I got from you and Arnd gel with my thinking that we want >> something generic enough in Linux to handle this in a non-architectural >> way (real topology, not just hierarchies). That should also cover the >> kind of cluster-like stuff e.g. AMD with NUMA on HT on a single socket >> and other stuff. So...it sounds like we need "something" to add to our >> understanding of hierarchy, and that "something" is in sysfs. A proposal >> needs to be derived (I think Don will followup since he is keen to poke >> at this). We'll go back to the ACPI ASWG folks to add whatever is >> missing to future ACPI bindings after that discussion. > > So, whatever happened to this? > > We are running into issues with some DPDK code on arm64 that makes assumptions > about the existence of a NUMA-based system based on the physical_package_id > in sysfs. On A57 cpus since physical_package_id represents 'cluster' > things go a bit haywire. > > Granted this particular app has an x86-centric assumption in it, but what is the > longer term view of how topologies should be represented? > > This thread seemed to be heading in the direction of a solution, but > then it seems to have just stopped. Can you elaborate a little more on the specifics of the DPDK failure? Would the following change fix it? This should make physical_package_id in sysfs read as -1 (default definition from include/linux/topology.h), while preserving the cluster affinity information for kernel scheduling purposes. Thanks, Cov --- 8< --- diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index 8b57339..f1095e7 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -13,7 +13,6 @@ struct cpu_topology { extern struct cpu_topology cpu_topology[NR_CPUS]; -#define topology_physical_package_id(cpu) (cpu_topology[cpu].cluster_id) #define topology_core_id(cpu) (cpu_topology[cpu].core_id) #define topology_core_cpumask(cpu) (&cpu_topology[cpu].core_sibling) #define topology_sibling_cpumask(cpu) (&cpu_topology[cpu].thread_sibling)