From patchwork Wed Mar 30 13:51:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Srikar Dronamraju X-Patchwork-Id: 12795864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69AFFC433F5 for ; Wed, 30 Mar 2022 13:52:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 024AA8D0003; Wed, 30 Mar 2022 09:52:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F16928D0001; Wed, 30 Mar 2022 09:52:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB82A8D0003; Wed, 30 Mar 2022 09:52:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0212.hostedemail.com [216.40.44.212]) by kanga.kvack.org (Postfix) with ESMTP id CD2D38D0001 for ; Wed, 30 Mar 2022 09:52:00 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 87AD5A0A0B for ; Wed, 30 Mar 2022 13:52:00 +0000 (UTC) X-FDA: 79301191200.28.ADB4D1A Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf31.hostedemail.com (Postfix) with ESMTP id 1132620003 for ; Wed, 30 Mar 2022 13:51:59 +0000 (UTC) Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22UBvaCc000506; Wed, 30 Mar 2022 13:51:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=pp1; bh=LwOiCD7ULa7QFysdbp5LCDxiSy4gGwG+AkFsd47R1Dg=; b=f1WEaQcSOeHnuOuofUx1qNFBYJs2zm6T9j1E+4y1khzDyXC3Xm+/lSns4JQ+X+T+n0N+ hS8GFyyYDoDg4MqAcphykAsn0SC/MYi6N4Sucr6ktJXe4vbfGdiWav/eijjUKXJUcKWb sfQnwKPKHHgXMqFiZGmzGD03TWF106TFaOhcnG4FRvJ5vH/D42PWC0PTwzxpuun9r6WO TwzSJWFVxGdOonVzEy2VPK6jYktTv9sJncCgyNTjiJv746iDlpIkVGATotxRUV84bveH rfjtSRreyLzg1rQk7VVu0zMyWjysXiSvgrP+Vs1IbJEWZDJkStmfovbyXClmpolV+omK 6Q== Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 3f4psu2j86-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 30 Mar 2022 13:51:53 +0000 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22UDiLah016604; Wed, 30 Mar 2022 13:51:51 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma04ams.nl.ibm.com with ESMTP id 3f1tf90qq0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 30 Mar 2022 13:51:51 +0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22UDpmXI40305016 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 30 Mar 2022 13:51:48 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BB3AF11C054; Wed, 30 Mar 2022 13:51:48 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 05E9111C04A; Wed, 30 Mar 2022 13:51:47 +0000 (GMT) Received: from saptagiri.in.ibm.com (unknown [9.43.100.4]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 30 Mar 2022 13:51:46 +0000 (GMT) From: Srikar Dronamraju To: Michael Ellerman Cc: linuxppc-dev , Srikar Dronamraju , linux-mm@kvack.org, Michal Hocko , Geetika Moolchandani Subject: [PATCH] powerpc/numa: Handle partially initialized numa nodes Date: Wed, 30 Mar 2022 19:21:23 +0530 Message-Id: <20220330135123.1868197-1-srikar@linux.vnet.ibm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: OPgegmOLR9akDNkqQgO8Lw_7cQSqboq8 X-Proofpoint-GUID: OPgegmOLR9akDNkqQgO8Lw_7cQSqboq8 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-30_04,2022-03-30_01,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 lowpriorityscore=0 mlxscore=0 adultscore=0 impostorscore=0 mlxlogscore=787 malwarescore=0 clxscore=1011 suspectscore=0 bulkscore=0 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203300066 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1132620003 X-Stat-Signature: i4qxfs3xsotec3srwdxg9y1ho1a8p83k Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=f1WEaQcS; dmarc=pass (policy=none) header.from=ibm.com; spf=none (imf31.hostedemail.com: domain of srikar@linux.vnet.ibm.com has no SPF policy when checking 148.163.156.1) smtp.mailfrom=srikar@linux.vnet.ibm.com X-Rspam-User: X-HE-Tag: 1648648319-676415 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With commit 09f49dca570a ("mm: handle uninitialized numa nodes gracefully") NODE_DATA for even a memoryless/cpuless node is partially initialized at boot time. Before onlining the node, current Powerpc code checks for NODE_DATA to be NULL. However since NODE_DATA is partially initialized, this check will end up always being false. This causes hotplugging a CPU to a memoryless/cpuless node to fail. Before adding CPUs $ numactl -H available: 1 nodes (4) node 4 cpus: 0 1 2 3 4 5 6 7 node 4 size: 97372 MB node 4 free: 95545 MB node distances: node 4 4: 10 $ lparstat System Configuration type=Dedicated mode=Capped smt=8 lcpu=1 mem=99709440 kB cpus=0 ent=1.00 %user %sys %wait %idle physc %entc lbusy app vcsw phint ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- 2.66 2.67 0.16 94.51 0.00 0.00 5.33 0.00 67749 0 After hotplugging 32 cores $ numactl -H node 4 cpus: 0 1 2 3 4 5 6 7 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 node 4 size: 97372 MB node 4 free: 93636 MB node distances: node 4 4: 10 $ lparstat System Configuration type=Dedicated mode=Capped smt=8 lcpu=33 mem=99709440 kB cpus=0 ent=33.00 %user %sys %wait %idle physc %entc lbusy app vcsw phint ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- 0.04 0.02 0.00 99.94 0.00 0.00 0.06 0.00 1128751 3 As we can see numactl is listing only 8 cores while lparstat is showing 33 cores. Also dmesg is showing messages like: [ 2261.318350 ] BUG: arch topology borken [ 2261.318357 ] the DIE domain not a subset of the NODE domain Fixes: 09f49dca570a ("mm: handle uninitialized numa nodes gracefully") Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-mm@kvack.org Cc: Michal Hocko Cc: Michael Ellerman Reported-by: Geetika Moolchandani Signed-off-by: Srikar Dronamraju Acked-by: Michal Hocko Acked-by: Oscar Salvador Acked-by: Mike Rapoport Signed-off-by: Oscar Salvador Reviewed-by: Srikar Dronamraju --- arch/powerpc/mm/numa.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c index b9b7fefbb64b..13022d734951 100644 --- a/arch/powerpc/mm/numa.c +++ b/arch/powerpc/mm/numa.c @@ -1436,7 +1436,7 @@ int find_and_online_cpu_nid(int cpu) if (new_nid < 0 || !node_possible(new_nid)) new_nid = first_online_node; - if (NODE_DATA(new_nid) == NULL) { + if (!node_online(new_nid)) { #ifdef CONFIG_MEMORY_HOTPLUG /* * Need to ensure that NODE_DATA is initialized for a node from