From patchwork Wed Nov 14 22:49:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 10683303 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B3A8513BF for ; Wed, 14 Nov 2018 22:53:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A7FEC2AC61 for ; Wed, 14 Nov 2018 22:53:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9C57F2BF88; Wed, 14 Nov 2018 22:53:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3DBBA2AC61 for ; Wed, 14 Nov 2018 22:53:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726379AbeKOI6S (ORCPT ); Thu, 15 Nov 2018 03:58:18 -0500 Received: from mga12.intel.com ([192.55.52.136]:50985 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727084AbeKOI6R (ORCPT ); Thu, 15 Nov 2018 03:58:17 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Nov 2018 14:53:03 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,234,1539673200"; d="scan'208";a="106314883" Received: from unknown (HELO localhost.lm.intel.com) ([10.232.112.69]) by fmsmga004.fm.intel.com with ESMTP; 14 Nov 2018 14:53:03 -0800 From: Keith Busch To: linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, linux-mm@kvack.org Cc: Greg Kroah-Hartman , Rafael Wysocki , Dave Hansen , Dan Williams , Keith Busch Subject: [PATCH 3/7] doc/vm: New documentation for memory performance Date: Wed, 14 Nov 2018 15:49:16 -0700 Message-Id: <20181114224921.12123-4-keith.busch@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20181114224921.12123-2-keith.busch@intel.com> References: <20181114224921.12123-2-keith.busch@intel.com> Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Platforms may provide system memory where some physical address ranges perform differently than others. These heterogeneous memory attributes are common to the node that provides the memory and exported by the kernel. Add new documentation providing a brief overview of such systems and the attributes the kernel makes available to aid applications wishing to query this information. Signed-off-by: Keith Busch --- Documentation/vm/numaperf.rst | 71 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 Documentation/vm/numaperf.rst diff --git a/Documentation/vm/numaperf.rst b/Documentation/vm/numaperf.rst new file mode 100644 index 000000000000..5a3ecaff5474 --- /dev/null +++ b/Documentation/vm/numaperf.rst @@ -0,0 +1,71 @@ +.. _numaperf: + +================ +NUMA Performance +================ + +Some platforms may have multiple types of memory attached to a single +CPU. These disparate memory ranges share some characteristics, such as +CPU cache coherence, but may have different performance. For example, +different media types and buses affect bandwidth and latency. + +A system supporting such heterogeneous memory groups each memory type +under different "nodes" based on similar CPU locality and performance +characteristics. Some memory may share the same node as a CPU, and +others are provided as memory-only nodes. While memory only nodes do not +provide CPUs, they may still be local to one or more compute nodes. The +following diagram shows one such example of two compute noes with local +memory and a memory only node for each of compute node: + + +------------------+ +------------------+ + | Compute Node 0 +-----+ Compute Node 1 | + | Local Node0 Mem | | Local Node1 Mem | + +--------+---------+ +--------+---------+ + | | + +--------+---------+ +--------+---------+ + | Slower Node2 Mem | | Slower Node3 Mem | + +------------------+ +--------+---------+ + +A "memory initiator" is a node containing one or more devices such as +CPUs or separate memory I/O devices that can initiate memory requests. A +"memory target" is a node containing one or more CPU-accessible physical +address ranges. + +When multiple memory initiators exist, accessing the same memory +target may not perform the same as each other. The highest performing +initiator to a given target is considered to be one of that target's +local initiators. + +To aid applications matching memory targets with their initiators, +the kernel provide symlinks to each other like the following example:: + + # ls -l /sys/devices/system/node/nodeX/initiator* + /sys/devices/system/node/nodeX/targetY -> ../nodeY + + # ls -l /sys/devices/system/node/nodeY/target* + /sys/devices/system/node/nodeY/initiatorX -> ../nodeX + +Applications may wish to consider which node they want their memory to +be allocated from based on the nodes performance characteristics. If +the system provides these attributes, the kernel exports them under the +node sysfs hierarchy by appending the initiator_access directory under +the node as follows:: + + /sys/devices/system/node/nodeY/initiator_access/ + +The kernel does not provide performance attributes for non-local memory +initiators. The performance characteristics the kernel provides for +the local initiators are exported are as follows:: + + # tree /sys/devices/system/node/nodeY/initiator_access + /sys/devices/system/node/nodeY/initiator_access + |-- read_bandwidth + |-- read_latency + |-- write_bandwidth + `-- write_latency + +The bandwidth attributes are provided in MiB/second. + +The latency attributes are provided in nanoseconds. + +See also: https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf