From patchwork Sun Jan 1 09:45:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13086351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13D44C4167B for ; Sun, 1 Jan 2023 09:45:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3809A8E0003; Sun, 1 Jan 2023 04:45:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 330958E0001; Sun, 1 Jan 2023 04:45:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F82E8E0003; Sun, 1 Jan 2023 04:45:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 113FD8E0001 for ; Sun, 1 Jan 2023 04:45:44 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DED2DA070A for ; Sun, 1 Jan 2023 09:45:43 +0000 (UTC) X-FDA: 80305748166.07.7AEEB5E Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf06.hostedemail.com (Postfix) with ESMTP id 762F4180007 for ; Sun, 1 Jan 2023 09:45:42 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=LSwwzypn; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf06.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672566342; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GeMxW6/bo7t8CedbH0ljza4tMQodK6qFd732yujZOBQ=; b=mGqb6722AO2kMN/6yng2EjileGFgSbgtq9U+K+Suexld0cPxRx2lC8Ep2LV2MmZAnstDcK O+5ZnKwiyIHjuunH/yp4zx4qq2NrUfzUhNQo4SdwZvNuoRsDiJQyNQyt0oJeLYjHUAgxA3 ReB4EJF7auWg/nXFhfzSAh4X7My9l18= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=LSwwzypn; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf06.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672566342; a=rsa-sha256; cv=none; b=JuypH/Hw5rvg+1WwCC+vYH7GCh57DGdcwCgf5PZOSax6tYbhZPR3K95Qmw2Qyk29Amx78R WtcNCI03reyfy3GgKtU+d6z/fStVmT+mKo2U/SWN7p9y+PmFBAAnT6H9HJ+OH0ye2M6oZR A7KYfR+YvxvhiN8WZYi/o+5+IDvqYcE= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 66EEB60D56; Sun, 1 Jan 2023 09:45:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 82D4AC433F0; Sun, 1 Jan 2023 09:45:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672566340; bh=ZLGBlhqw5hH+SxGqvaz0vux+DdR7ulM6iEJUTeGEBI0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LSwwzypnpxZ3H85D2YBEdZVPE/TRDKRP3ryXydYoZefESGnt6H7OCoiHhlCv76bTW WNsDwPgC2L39QzADbYxOUtZ5vf8ia3V9oTIiY7BB0vnSPuWFhY7jC39amHSpkjl9qZ RnqpefIgXHJkGnQ1HoqbDXd331YBoJFJiUklCqIbOf5rn5IbhXkBJ+3w+zLmlvodBn jTk9oK8pN9j1brqpgBsL/YkXwVd+D9XQyRRvDztUu5lG0D+pTEMEkceTPHysQ+6szl 0WGgLxEBeq+937o1Ve2rDdsaqu8Ek9mJGTcCzJ1tK358+nYc9YyGg7PqyOmpi4lPZ9 BBxKhRagu/abQ== From: Mike Rapoport To: Jonathan Corbet Cc: Andrew Morton , David Hildenbrand , Johannes Weiner , Lorenzo Stoakes , "Matthew Wilcox (Oracle)" , Mel Gorman , Michal Hocko , Mike Rapoport , Vlastimil Babka , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/2] docs/mm: Page Reclaim: add page label to allow external references Date: Sun, 1 Jan 2023 11:45:22 +0200 Message-Id: <20230101094523.1522109-2-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230101094523.1522109-1-rppt@kernel.org> References: <20230101094523.1522109-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 762F4180007 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 8w5zsoefkmaa4tecm3ki9cyq9i6rojgw X-HE-Tag: 1672566342-385684 X-HE-Meta: U2FsdGVkX1/gq60CrPL8+uRlL2JUrlZlLdHlm3byGiSUlNoVj9VRCDmqfc0wW8JJwbX+3DJPpEQLZfgAkP5VfC4/lxLQx/fGra91M0ku5OK37HdsV1ielwLNbNKinPwdawypF2Nxx0N0OXz5b+5vGYKaNQ/3MGMvrQWwGFo02sIbdRdOaESEF/Dx62HNBZY4Vegkr9oqfDyhiHKdTINOc7x9vLh/W0Q3PakT1dzqcLqOhVMOcQrICG7l3peNEp5hc/Z4uFMcCmhV7FuWRIHD8V1/T5YH+h+qpqK5QMGWpK/iJZFN/Zos6Z63sOZwFHVxtutpTQZnVbVZCGOu9NDs08Ahir+EiHkW9xRybB6BOSEB8/pnzmA+KSKB2X69SEvl72O6KYI/pDz+57wN5lS20KB/IlA1SMNZa4gQbWGeFAvHMVbC8MEeVO32F2brKzJTRD59f4rEGtiWHf5WX+4GDvVv0BBfMKdy7fOqKScFV6U/mOS7D0DqbJ6vJc1hYenOteuzAOlO6Y8I9WdwVrhTm2WWIvL/HCBJfaIBYBTkjpj1UsBo+cu/lGgR11gZalUG0Xtn7UlYP4LW77r/ONAN3fSRoMIckGpRynb5P+r3wWegkGd6/YTiazBARrg878W1ZSpZUnwlBUmkVT2ZHONWVX/KkvlBh4J/JCrh9TQu7TGPSQEHKDGy8FaXoZKcdjOAZB3gMHI7pUATGVxyxtIOKUS5wrQ/aq3URsSgwOU89cqT1GzXmMOZ1nr2aHGzdFIbz6psYntzNpVAvBqet+uJOrtPPnIuv+/s0BzrP7TjEd/DhyHzLceJl81EVIS1TRVUcCfs3Sg80qgCGmmOWm5/iCMyylJDKwl4lPoZxaJanHoFiAanFMiKr2ownLNVjEgiLOeZSkV+wnfrf+HAXAx9lLQlRhRX+aQvfV8Wiyrkbpvy+I3jkQdTVnrcwl3ajS9OPWLNvfl/M5EhMv5S7wI mbvz6Ao1 bNj7vKNV3RLEXwo6n/FNR6VKaSx0PBFH6QD+U9fXzelNHwWOKF9Sxsnos6zLJ4H2GsLCVKrl3Q5/OKYg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Signed-off-by: Mike Rapoport (IBM) --- Documentation/mm/page_reclaim.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/mm/page_reclaim.rst b/Documentation/mm/page_reclaim.rst index 50a30b7f8ac3..3fccde066436 100644 --- a/Documentation/mm/page_reclaim.rst +++ b/Documentation/mm/page_reclaim.rst @@ -1,5 +1,7 @@ .. SPDX-License-Identifier: GPL-2.0 +.. _page_reclaim: + ============ Page Reclaim ============ From patchwork Sun Jan 1 09:45:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13086352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FE7EC4167B for ; Sun, 1 Jan 2023 09:45:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0721C8E0005; Sun, 1 Jan 2023 04:45:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 022748E0001; Sun, 1 Jan 2023 04:45:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E05A98E0005; Sun, 1 Jan 2023 04:45:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CFDB38E0001 for ; Sun, 1 Jan 2023 04:45:48 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A1F35404CF for ; Sun, 1 Jan 2023 09:45:48 +0000 (UTC) X-FDA: 80305748376.22.C97C3CB Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf26.hostedemail.com (Postfix) with ESMTP id 05DA8140010 for ; Sun, 1 Jan 2023 09:45:46 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=miH+Hc20; spf=pass (imf26.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672566347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=asXBjPSqL7evS2C0knrDSO1jJoTzWxHdCyo0efJjigo=; b=WuMEBi0mEuqwVyiBzWA1l1ZEMDd8mVfFibGNf+e5DQAOJI81OEd7Fc4s61wdZPzXSzrq8O SmNx8fV61tZJcLjXTONLcz1h0cF2ZqvHGJjx2q1IT91RtO7cfj3qDFPdAVqhQC+TSmK91q Ag2W2KXIcZKcFmout4a0QE5MZyzB8+M= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=miH+Hc20; spf=pass (imf26.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672566347; a=rsa-sha256; cv=none; b=a9IfXNXwYB/aDCh2nLnxMazlWYASyIM+DAdhSwAVQV3J0UCTSKmsmRoydmcXLTbUhTttaW du9ksZZZ416Hw5y0yf4bWQQ/Xb2sMl+3SXFTdz+OPCxwpUZZz50nipEnYSTDm5Vlut9ent 1XipLgk++b+jHVSdErWxqQi/KFTb+1w= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1BB7860DBF; Sun, 1 Jan 2023 09:45:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5E3E4C433EF; Sun, 1 Jan 2023 09:45:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672566344; bh=z9ZwLSxEOQtQ4Tc0WdCONTBI1jDHqtWw3QNCo6jVUzo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=miH+Hc20PPolZmCQ2J/uZ75viTP9KDzs1oLdt/pGRcUGzJykZLwj4t5sL6Z04nnFy uMa37kQwe/2Z7mCHfg3n4QByXMf8xkHAYWpVGH3qPKW2uAxpT15WuZQH9ivGpcktCG xRaW//DXutO/6t1HWXrAEO2auMgGoIjr2vWCFYvcgp5cCTmuhtA/tqWKV3ZGgKEzPV afB2mnk8tC5hTTePaW7Td+YrvHVK+UFuGCYp0DNX4pC2EY/ztnV1H2xt9jmxbPYGjS OLd+gGr1XDj6235aFJrynf9vbWtnldM6Qbu5OKATWciK1fjIeBDY/6NBYPN6OhjGAo E12pBydo9mtHg== From: Mike Rapoport To: Jonathan Corbet Cc: Andrew Morton , David Hildenbrand , Johannes Weiner , Lorenzo Stoakes , "Matthew Wilcox (Oracle)" , Mel Gorman , Michal Hocko , Mike Rapoport , Vlastimil Babka , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 2/2] docs/mm: Physical Memory: add structure, introduction and nodes description Date: Sun, 1 Jan 2023 11:45:23 +0200 Message-Id: <20230101094523.1522109-3-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230101094523.1522109-1-rppt@kernel.org> References: <20230101094523.1522109-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 05DA8140010 X-Stat-Signature: 9gdek7sr7ni5p48hyhyguhsa3xjnq9g4 X-Rspam-User: X-HE-Tag: 1672566346-905115 X-HE-Meta: U2FsdGVkX18Tbu49gXJLCTrES6ZlaeP+Vuj1/43W5BsPlDzYr/a2HjPUS62iet5++lUqyStK49FBn/zezgRxHmZhbNzfSafia5PFAm5KIPCkhm01PhSm30/BsliSc+U3Nc8axR+wkCrE9FzTitAktVhOWgEToYz9kvUUAi50V1lAoEPvje9/6svX2DjPd78WlyyeN/F+4r6Frz2xrm9xgpccdns/u0IpwsLTEQ1ITtWAlFZv4UtiJ9kRvGeQhczu0wahG0pHq+7ApNC1gv+koy3fv4YWraNuDnhJOWkHSqL3IaGmXQprJ/GK+cngWcrdnRwLtrdZYvMkiNc1QunMSfM20yFPU802AOSG1jyrqz8g36NNX5vhs3uJpyjf+hejLgaR3rge0kCc+oLXy9Bmsk8I//aKI92YtffobSdTMiCPpVNn7oeNWW/8V51e5Q5/FtUoS7WiWuEgqWCIJKSw8urbhrgBrepfZuS+kpqV2sYBJqRNRBZKBmhxntbeYNH2T+/SnGrMlH0jCrHIELmVg9mpcOwcAJB79lDk5MzsbeU2YCTtCpqbUjFr6Uv5YkzGPoikVBrdCPJRYWEOoV0fWS0vASKQlnnxde7HeLg4k7Wnd2xmroPKZcO5DBZSHTNf+ZALRSEeTr1zBAFFRLdcMevxvEyYDNSvzVEMG/nVqFNQjOK2JkPhBLgxChD9S7GVV49B6FwhEWrFS6INk5+krh2m8d59115fNQwqwJlCXqusTfPnEY0UhhWIFtEf/Olw8Z0gmN2r3h2p5F252c103gUgARAo72C2sQx5ET2yiZFUxtJQOOqJM/dQ/RtxwpbO3QxelXNPJZ1mO7AHz7m5XAE/lZTvi5Y8y6olVt4B9FRLgqmgq7wyv3sHZc20tBH/1IRDya3a12Ny9xlSDHBHHTnZLaxpt0LkfSdqKMs/CnjBEbsrqtog/0509tNBaalGkErRsKAPsecUDaUF/M5 LkK9rxhH o7gyoxSDksihmEoQmTt8TWGihiuc23YfaUznK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Signed-off-by: Mike Rapoport (IBM) --- Documentation/mm/physical_memory.rst | 322 +++++++++++++++++++++++++++ 1 file changed, 322 insertions(+) diff --git a/Documentation/mm/physical_memory.rst b/Documentation/mm/physical_memory.rst index 2ab7b8c1c863..fcf52f1db16b 100644 --- a/Documentation/mm/physical_memory.rst +++ b/Documentation/mm/physical_memory.rst @@ -3,3 +3,325 @@ =============== Physical Memory =============== + +Linux is available for a wide range of architectures so there is a need for an +architecture-independent abstraction to represent the physical memory. This +chapter describes the structures used to manage physical memory in a running +system. + +The first principal concept prevalent in the memory management is +`Non-Uniform Memory Access (NUMA) +`_. +With multi-core and multi-socket machines, memory may be arranged into banks +that incur a different cost to access depending on the “distance” from the +processor. For example, there might be a bank of memory assigned to each CPU or +a bank of memory very suitable for DMA near peripheral devices. + +Each bank is called a node and the concept is represented under Linux by a +``struct pglist_data`` even if the architecture is UMA. This structure is +always referenced to by it's typedef ``pg_data_t``. A pg_data_t structure +for a particular node can be referenced by ``NODE_DATA(nid)`` macro where +``nid`` is the ID of that node. + +For NUMA architectures, the node structures are allocated by the architecture +specific code early during boot. Usually, these structures are allocated +locally on the memory bank they represent. For UMA architectures, only one +static pg_data_t structure called ``contig_page_data`` is used. Nodes will +be discussed further in Section :ref:`Nodes ` + +Each node may be divided up into a number of blocks called zones which +represent ranges within memory. These ranges are usually determined by +architectural constraints for accessing the physical memory. A zone is +described by a ``struct zone_struct``, typedeffed to ``zone_t`` and each zone +has one of the types described below. + +`ZONE_DMA` and `ZONE_DMA32` + represent memory suitable for DMA by peripheral devices that cannot + access all of the addressable memory. Depending on the architecture, + either of these zone types or even they both can be disabled at build + time using ``CONFIG_ZONE_DMA`` and ``CONFIG_ZONE_DMA32`` configuration + options. Some 64-bit platforms may need both zones as they support + peripherals with different DMA addressing limitations. + +`ZONE_NORMAL` + is for normal memory that can be accessed by the kernel all the time. DMA + operations can be performed on pages in this zone if the DMA devices support + transfers to all addressable memory. ZONE_NORMAL is always enabled. + +`ZONE_HIGHMEM` + is the part of the physical memory that is not covered by a permanent mapping + in the kernel page tables. The memory in this zone is only accessible to the + kernel using temporary mappings. This zone is available only some 32-bit + architectures and is enabled with ``CONFIG_HIGHMEM``. + +`ZONE_MOVABLE` + is for normal accessible memory, just like ZONE_NORMAL. The difference is + that most pages in ZONE_MOVABLE are movable. That means that while virtual + addresses of these pages do not change, their content may move between + different physical pages. ZONE_MOVABLE is only enabled when one of + `kernelcore`, `movablecore` and `movable_node` parameters is present in the + kernel command line. See :ref:`Page migration ` for + additional details. + +`ZONE_DEVICE` + represents memory residing on devices such as PMEM and GPU. It has different + characteristics than RAM zone types and it exists to provide :ref:`struct + page ` and memory map services for device driver identified physical + address ranges. ZONE_DEVICE is enabled with configuration option + ``CONFIG_ZONE_DEVICE``. + +It is important to note that many kernel operations can only take place using +ZONE_NORMAL so it is the most performance critical zone. Zones are discussed +further in Section :ref:`Zones `. + +The relation between node and zone extents is determined by the physical memory +map reported by the firmware, architectural constraints for memory addressing +and certain parameters in the kernel command line. + +For example, with 32-bit kernel on an x86 UMA machine with 2 Gbytes of RAM the +entire memory will be on node 0 and there will be three zones: ZONE_DMA, +ZONE_NORMAL and ZONE_HIGHMEM:: + + 0 2G + +-------------------------------------------------------------+ + | node 0 | + +-------------------------------------------------------------+ + + 0 16M 896M 2G + +----------+-----------------------+--------------------------+ + | ZONE_DMA | ZONE_NORMAL | ZONE_HIGHMEM | + +----------+-----------------------+--------------------------+ + + +With a kernel built with ZONE_DMA disabled and ZONE_DMA32 enabled and booted +with `movablecore=80%` parameter on an arm64 machine with 16 Gbytes of RAM +equally split between two nodes, there will be ZONE_DMA32, ZONE_NORMAL and +ZONE_MOVABLE on node 0, and ZONE_NORMAL and ZONE_MOVABLE on node 1:: + + + 1G 9G 17G + +--------------------------------+ +--------------------------+ + | node 0 | | node 1 | + +--------------------------------+ +--------------------------+ + + 1G 4G 4200M 9G 9320M 17G + +---------+----------+-----------+ +------------+-------------+ + | DMA32 | NORMAL | MOVABLE | | NORMAL | MOVABLE | + +---------+----------+-----------+ +------------+-------------+ + +.. _nodes: + +Nodes +===== + +As we have mentioned, each node in memory is described by a ``pg_data_t`` which +is a typedef for a ``struct pglist_data``. When allocating a page, by default +Linux uses a node-local allocation policy to allocate memory from the node +closest to the running CPU. As processes tend to run on the same CPU, it is +likely the memory from the current node will be used. The allocation policy can +be controlled by users as described in +`Documentation/admin-guide/mm/numa_memory_policy.rst`. + +Most NUMA architectures maintain an array of pointers to the node +structures. The actual structures are allocated early during boot when +architecture specific code parses the physical memory map reported by the +firmware. The bulk of the node initialization happens slightly later in the +boot process by free_area_init() function, described later in Section +:ref:`Initialization `. + + +Along with the node structures, kernel maintains an array of ``nodemask_t`` +bitmasks called `node_states`. Each bitmask in this array represents a set of +nodes with particular properties as defined by `enum node_states`: + +`N_POSSIBLE` + The node could become online at some point. +`N_ONLINE` + The node is online. +`N_NORMAL_MEMORY` + The node has regular memory. +`N_HIGH_MEMORY` + The node has regular or high memory. When ``CONFIG_HIGHMEM`` is disabled + aliased to `N_NORMAL_MEMORY`. +`N_MEMORY` + The node has memory(regular, high, movable) +`N_CPU` + The node has one or more CPUs + +For each node that has a property described above, the bit corresponding to the +node ID in the ``node_states[]`` bitmask is set. + +For example, for node 2 with normal memory and CPUs, bit 2 will be set in :: + + node_states[N_POSSIBLE] + node_states[N_ONLINE] + node_states[N_NORMAL_MEMORY] + node_states[N_MEMORY] + node_states[N_CPU] + +For various operations possible with nodemasks please refer to +`include/linux/nodemask.h +`_. + +Among other things, nodemasks are used to provide macros for node traversal, +namely `for_each_node()` and `for_each_online_node()`. + +For instance, to call a function foo() for each online node:: + + for_each_online_node(nid) { + pg_data_t *pgdat = NODE_DATA(nid); + + foo(pgdat); + } + +Node structure +-------------- + +The struct pglist_data is declared in `include/linux/mmzone.h +`_. +Here we briefly describe fields of this structure: + +General +~~~~~~~ + +`node_zones` + The zones for this node. Not all of the zones may be populated, but it is + the full list. It is referenced by this node's node_zonelists as well as + other node's node_zonelists. + +`node_zonelists` The list of all zones in all nodes. This list defines the + order of zones that allocations are preferred from. The `node_zonelists` is + set up by build_zonelists() in mm/page_alloc.c during the initialization of + core memory management structures. + +`nr_zones` + Number of populated zones in this node. + +`node_mem_map` + For UMA systems that use FLATMEM memory model the 0's node (and the only) + `node_mem_map` is array of struct pages representing each physical frame. + +`node_page_ext` + For UMA systems that use FLATMEM memory model the 0's (and the only) node + `node_mem_map` is array of extensions of struct pages. Available only in the + kernels built with ``CONFIG_PAGE_EXTENTION`` enabled. + +`node_start_pfn` + The page frame number of the starting page frame in this node. + +`node_present_pages` + Total number of physical pages present in this node. + +`node_spanned_pages` + Total size of physical page range, including holes. + +`node_size_lock` + A lock that protects the fields defining the node extents. Only defined when + at least one of ``CONFIG_MEMORY_HOTPLUG`` or + ``CONFIG_DEFERRED_STRUCT_PAGE_INIT`` configuration options are enabled. + + pgdat_resize_lock() and pgdat_resize_unlock() are provided to manipulate + node_size_lock without checking for CONFIG_MEMORY_HOTPLUG or + CONFIG_DEFERRED_STRUCT_PAGE_INIT. + +`node_id` + The Node ID (NID) of the node, starts at 0. + +`totalreserve_pages` + This is a per~node reserve of pages that are not available to userspace + allocations. + +`first_deferred_pfn` + If memory initialization on large machines is deferred then this is the first + PFN that needs to be initialized. Defined only when + ``CONFIG_DEFERRED_STRUCT_PAGE_INIT`` is enabled + +`deferred_split_queue` + Per-node queue of huge pages that their split was deferred. Defined only when ``CONFIG_TRANSPARENT_HUGEPAGE`` is enabled. + +`__lruvec` + Per-node lruvec holding LRU lists and related parameters. Used only when memory cgroups are disabled. Should not be accessed directly, use mem_cgroup_lruvec() to look up lruvecs instead. + +Reclaim control +~~~~~~~~~~~~~~~ + +See also :ref:`Page Reclaim `. + +`kswapd` + Per-node instance of kswapd kernel thread. + +`kswapd_wait`, `pfmemalloc_wait`, `reclaim_wait` + Workqueues used to synchronize memory reclaim tasks + +`nr_writeback_throttled` + Number of tasks that are throttled waiting on dirty pages to clean. + +`nr_reclaim_start` + Number of pages written while reclaim is throttled waiting for writeback. + +`kswapd_order` + Controls the order kswapd tries to reclaim + +`kswapd_highest_zoneidx` + The highest zone index to be reclaimed by kswapd + +`kswapd_failures` + Number of runs kswapd was unable to reclaim any pages + +`min_unmapped_pages` + Minimal number of unmapped file backed pages that cannot be reclaimed. Determined by vm.min_unmapped_ratio sysctl. + Only defined when ``CONFIG_NUMA`` is enabled. + +`min_slab_pages` + Minimal number of SLAB pages that cannot be reclaimed. Determined by vm.min_slab_ratio sysctl. + Only defined when ``CONFIG_NUMA`` is enabled + +`flags` + Flags controlling reclaim behavior. + +Compaction control +~~~~~~~~~~~~~~~~~~ + +`kcompactd_max_order` + Page order that kcompactd should try to achieve. + +`kcompactd_highest_zoneidx` + The highest zone index to be compacted by kcompactd. + +`kcompactd_wait` + Workqueue used to synchronizes memory compaction tasks. + +`kcompactd` + Per-node instance of kcompactd kernel thread. + +`proactive_compact_trigger` + Determines if proactive compaction is enabled. Controlled by vm.compaction_proactiveness sysctl. + +Statistics +~~~~~~~~~~ + +`per_cpu_nodestats` + Per-CPU VM statistics for the node + +`vm_stat` + VM statistics for the node. + +.. _zones: + +Zones +===== + +.. _pages: + +Pages +===== + +.. _folios: + +Folios +====== + +.. _initialization: + +Initialization +==============