From patchwork Tue Oct 5 01:31:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 12535323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C46BC433EF for ; Tue, 5 Oct 2021 01:34:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DE33F610E6 for ; Tue, 5 Oct 2021 01:34:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DE33F610E6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id DBE17940032; Mon, 4 Oct 2021 21:31:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D481094002E; Mon, 4 Oct 2021 21:31:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBFA3940032; Mon, 4 Oct 2021 21:31:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id AB3E394002E for ; Mon, 4 Oct 2021 21:31:55 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 79CA531E7B for ; Tue, 5 Oct 2021 01:31:55 +0000 (UTC) X-FDA: 78660657390.08.F8BB588 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf15.hostedemail.com (Postfix) with ESMTP id 06ACAD000CDF for ; Tue, 5 Oct 2021 01:31:54 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1950fjVn024345; Tue, 5 Oct 2021 01:31:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=/YBgD6R6ycEEQAm6tID+saHip7SBQS0cRDXt3I5NIeo=; b=rX6zQ1POu4/iowtBt5i3YHNRU0k/z0TLmqQJ9nxlfVlczJ7uCCuCVXCbPkV8ffNI08gF LEPb9YG6M4Ez3kssn7fK5buThaq1/eQLRl3ljnaMu+1KduurvyXC7saBw455GG1BbIou T+s43cJrCIp49sXZ1BrgECcGbBbVwzn7HG/xdK7jzYSc65OpLHxz1GiHZV3gJ1X8QcXM RuPWqFX+FKv26fVEOSltOqgqIJoCX3izwmpsJYTo0X4Br9DX9HiMvYwPbial/pSfjI9J G8LyvywFyZ2KUlkUknlWCt8V890lGsJ7rHGycUddUNYjm65XgjMGpifWujZJzY/GpQNh fA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3bg454bvpk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Oct 2021 01:31:29 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1951UHTO178361; Tue, 5 Oct 2021 01:31:29 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2171.outbound.protection.outlook.com [104.47.58.171]) by aserp3020.oracle.com with ESMTP id 3bev8w0386-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Oct 2021 01:31:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YyPoD7gcfUhDkYXyRhyoolHOfym7iLGVaRwWUX96QCPHOeHaoZhHSluGOUBd/fY4baUuB6BJgAbIUYv7TznB8H/8RCrprMIlnAQmwvjyWoKAIVA6ANPI7ovoo8S63YaOSYZju37onoWK0Jh8fmbzY0lRhF+TCMXCxJus8gKapEChQiA4Xq5sGfZ1OlO+rCo+vCUChrGFmoGkk/lfYLpcWnySEXDss7wUhh8cpeEKiWIORDTYefDTdt79ZXuVs/y/6XkE17zWFrpFHSS3Aah1fKgaocWuRE7EOE3EGkpflUUoPpqi/MIatbFLUrwHbI1E2D1/osknp2WKJCrr46JC+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/YBgD6R6ycEEQAm6tID+saHip7SBQS0cRDXt3I5NIeo=; b=hGNgqQEm+eWSt8YwvYxWnxotrxRB3S6vSOFSViPsBDn3VQiDXFC7Cs8TFGUV4lWy2dVV3vXckg0rFCq6xYsmJzCyhHBzvXvaZiLnCyQL4N4JWwOn/PktgDl03uaRnuOHAAsM4OjsO/+UxhPGB2In5l0yvA/8SAcLAiJ2c84a5xUnb2Dc9qkYVxJkRoU230xDdxnLw6h6JLpCHeToLK8ViD/O+9RKjCRy/Da+aQrut9zcC1zx1+SrKLcj4sS+eNrl0c2SKuy3ALict5HU2SeHociy/OT5m8MY7us7qQmn87kUAygLWjVTqHslmHg/jtLNhdzEPFXBlq5vuHMlXllPRA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/YBgD6R6ycEEQAm6tID+saHip7SBQS0cRDXt3I5NIeo=; b=t7sg5D8o7s6qG8r6Fqd0MZufsQ8mnIAewOZJNEaDK2zudFgdLOG26sC6WbjKZyjhGm7bbgEHMKmILdYIBLGCjXyGq/gb4DNQZjVRHec1FfjAc0qBye+lObHVBBGcSZP4PqddyXGeyen39uvFjbBxVC3ZRRYq/jFLAtXHF5GFAHc= Received: from BL0PR10MB3011.namprd10.prod.outlook.com (2603:10b6:208:7e::29) by BLAPR10MB5044.namprd10.prod.outlook.com (2603:10b6:208:326::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.15; Tue, 5 Oct 2021 01:31:23 +0000 Received: from BL0PR10MB3011.namprd10.prod.outlook.com ([fe80::6d61:54c2:40f0:93a]) by BL0PR10MB3011.namprd10.prod.outlook.com ([fe80::6d61:54c2:40f0:93a%5]) with mapi id 15.20.4566.017; Tue, 5 Oct 2021 01:31:23 +0000 From: Liam Howlett To: "maple-tree@lists.infradead.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Andrew Morton , David Hildenbrand , Douglas Gilbert CC: Song Liu , Davidlohr Bueso , "Paul E . McKenney" , Matthew Wilcox , David Rientjes , Axel Rasmussen , Suren Baghdasaryan , Vlastimil Babka , Rik van Riel , Peter Zijlstra Subject: [PATCH v3 44/66] kernel/sched/fair: Use maple tree iterators instead of linked list Thread-Topic: [PATCH v3 44/66] kernel/sched/fair: Use maple tree iterators instead of linked list Thread-Index: AQHXuYiorXUzL4zFkUaYahbdRC7KyA== Date: Tue, 5 Oct 2021 01:31:03 +0000 Message-ID: <20211005012959.1110504-45-Liam.Howlett@oracle.com> References: <20211005012959.1110504-1-Liam.Howlett@oracle.com> In-Reply-To: <20211005012959.1110504-1-Liam.Howlett@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.30.2 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8ee5a201-36cf-43ed-6466-08d9879fd78b x-ms-traffictypediagnostic: BLAPR10MB5044: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:655; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: udtMORKrY3E3qS7OB7Qvkf4XXaV/GzCKb7OtYajDRjkH6gzMnuhhns4UCHY8B9f5aNhrXDARa1QwKNN2jzG2lNKjEbqYh89D6L1wfEriBhbqWkTZZssI/VG5QBs9QvpvRujDpVPn9BxfP4TJUWsMMFk0CaZEXbEQ4DP1WDJ9kJDHZ787Slc9ZG1kjDpt5/n+rQPvZBU6DY/ZbKQO5JxjgoXgOTKsxur6DhVemvSmzn2FYIwem24wMk1mTmKR2/OEUgx5zMqwPjzt/n8WexdNi78c4sHB/8pj2+fqFiqD8eAX4XS+lDn+wyWCX2Z1+x6zVl/BK3xaiO7fokmF3yLlXJ1KTVwno30CL8ePXc1vYqDwXi8TYI2kEMc27FQZ0LJjkIeXq5EBqA0AruLhK4WMm/gb7cUuPOkduIw2wkM5gROitz5LbZnIxHZjHWtbzfELaoWYd4ml4FeP5HdS4ZL0Yu8PZqaJVKyP/5ovBx+/F7cBuuU7X7E3Ctgqr/a0Gg9yZxF3Hak4lrLFHX7IO5cZV4hUat+AX8rnOeOgNVDO3Nb/u9P0v/+OgVWrVQ02tm456Iz1TqGYJjRQh8OhjwVHLD5oQo7+U39JPOOO90NX5LYImXwIlbFbEsQ9Zwk/lHtY/S18UxRuOzvCs4GcCScycwhAowGDfbgTF2Slg3Za1Qp0CLFnb04Gd8CDMwBqFr+fFQT7VjBAR7fIVnGGSDfObA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR10MB3011.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(110136005)(71200400001)(8676002)(86362001)(6666004)(6506007)(6512007)(36756003)(316002)(8936002)(54906003)(2906002)(83380400001)(186003)(66446008)(122000001)(26005)(7416002)(6486002)(4326008)(508600001)(2616005)(66476007)(44832011)(38070700005)(76116006)(5660300002)(91956017)(64756008)(66556008)(66946007)(1076003)(38100700002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?4er8KUGkn5YYQrDqkjKFlnr?= =?iso-8859-1?q?yE7u/3gCmbCvPgm93DA0u14bEznAAcjm83LRfDivdATliRZgdi+xhrapGm4f?= =?iso-8859-1?q?OjeVMZDX80S+D4SMMBMVu8zPmhhvS21A5iWS6+TMjqK5MsO76okQEKy1K+Cu?= =?iso-8859-1?q?nmpSB8Gky/o5w1lHz6nKH0swliHMe5OI6xmcFxe1WsvX8HSsWee/omWKeJ9Y?= =?iso-8859-1?q?n43sqwP80UGYq/V6RUveGGy9AJeGthMNToaMxhjhP835ld7x3N2Db7mBcx4/?= =?iso-8859-1?q?INaitxw05Lyr2nG64f6O9qeSMprCtR9HlzKbaK3z4kJIO9hOzP1dJrUsxQg6?= =?iso-8859-1?q?VcKqmp9jRlqf81u065huWTtcnLX4lZZptlBoNzODA9sXXAnV6yTZUDkUsL0/?= =?iso-8859-1?q?djptlzOrswAnrdOGhGa5gJeJbl+iiferaoPy3afICFUTwTSFlJPs0NkHbyIL?= =?iso-8859-1?q?rCET5hewjbeYEYRktUKUXmKfT/YTNsvdR9qlk4LaGWT7fqfouppLBbgTuhQ1?= =?iso-8859-1?q?D99XTcU5YkfdOkd7rJW/hJVHlh/Eqan6jy0cTAG5XmPnnHRI6cTSLlM5vaHc?= =?iso-8859-1?q?K4542tM82nndB6oSUrPIEBbEORR+3m8riafw01jSqRQx6VKhgJnMxEN60UVU?= =?iso-8859-1?q?V794vugIp6MM7y1j10yb9pHnP8IB6ZGwF94CiC4HYT28CTOlrvI/ldHTglmP?= =?iso-8859-1?q?SoS6z8Q6usA9yWZ2NhirF+qCcjxV2wYKwVavmXKaRe9szcJLxeCS6CXWhPQN?= =?iso-8859-1?q?lnQAwNsBQooidlQvOBNmZS0YPUkN46LZRmyQwrROEjhfHYtCXhTsRP5lyL3f?= =?iso-8859-1?q?VCXE1Jh7kJFUOi8BAdPu8OBfZEQGlxnE28wud0//mFlIc4mcPxmvUc2DvAOB?= =?iso-8859-1?q?3eChNstxQVBtQbBJv52Gv23+MOW2u1wbqAXyXaRN5XZQitFHFZT2w9S8hvIR?= =?iso-8859-1?q?ZLi6ZaWqdgapxrCZR4lhu0nPczSRAbXOyQakxxJWn1WtAf1vws0YOm0l94x1?= =?iso-8859-1?q?+XVepxCdP6CHTjS/FPmOsk5WS81PiVbembUGeVKucN6/jyAZ5ZTfDiY/0+p6?= =?iso-8859-1?q?6dc6QMgqtMUmt2OawbI7dvCNefCb+GP2FaCyUlx+7Pb76iN5R4PqhHqGhNUp?= =?iso-8859-1?q?4xI7VE/IBhR/cIVhUgdGNmbsulDZ0OB2QeKHNbQ9m9gntGm6GYbmCPMIiAl1?= =?iso-8859-1?q?E5piS/C+QNM0hnKRUAj3BUTq+bmuFEPqfRpi2u/k9l48nGloUoxTi/+S4KZc?= =?iso-8859-1?q?jOuPoVD2V6ZHNTxG8wjiMqOiYRZmbDddsTDJZtJISMTyVkRNbAsKf5Gw51GS?= =?iso-8859-1?q?/GtScP8tiwdWRdy6pMT6fnj9M7HatRYLhFcIiQYfF6cv4D3VXTpt31fyCkZM?= =?iso-8859-1?q?9?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BL0PR10MB3011.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8ee5a201-36cf-43ed-6466-08d9879fd78b X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Oct 2021 01:31:03.2627 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 4kZYbImqaYrA+35VtuH4YiN1F5BpSPk0Pp7ylHnBNTbFE5vfy4rXzAziiTTc8A1iEiYhfYpqpkwUmyO8EBxpVw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5044 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10127 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001 definitions=main-2110050007 X-Proofpoint-GUID: ygOcS6i5KgZXPPkXL2JxjDs3linCOWu_ X-Proofpoint-ORIG-GUID: ygOcS6i5KgZXPPkXL2JxjDs3linCOWu_ X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 06ACAD000CDF X-Stat-Signature: tot3hs7suyzu7twms1cfcxzph8tfnbgn Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=rX6zQ1PO; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=t7sg5D8o; spf=none (imf15.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=liam.howlett@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-HE-Tag: 1633397514-239052 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Signed-off-by: Liam R. Howlett --- kernel/sched/fair.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ff69f245b939..46769907c363 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2720,6 +2720,7 @@ static void task_numa_work(struct callback_head *work) unsigned long start, end; unsigned long nr_pte_updates = 0; long pages, virtpages; + MA_STATE(mas, &mm->mm_mt, 0, 0); SCHED_WARN_ON(p != container_of(work, struct task_struct, numa_work)); @@ -2772,13 +2773,17 @@ static void task_numa_work(struct callback_head *work) if (!mmap_read_trylock(mm)) return; - vma = find_vma(mm, start); + + rcu_read_lock(); + mas_set(&mas, start); + vma = mas_find(&mas, ULONG_MAX); if (!vma) { reset_ptenuma_scan(p); start = 0; - vma = mm->mmap; + mas_set(&mas, start); } - for (; vma; vma = vma->vm_next) { + + mas_for_each(&mas, vma, ULONG_MAX) { if (!vma_migratable(vma) || !vma_policy_mof(vma) || is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { continue; @@ -2823,7 +2828,9 @@ static void task_numa_work(struct callback_head *work) if (pages <= 0 || virtpages <= 0) goto out; + rcu_read_unlock(); cond_resched(); + rcu_read_lock(); } while (end != vma->vm_end); } @@ -2838,6 +2845,7 @@ static void task_numa_work(struct callback_head *work) mm->numa_scan_offset = start; else reset_ptenuma_scan(p); + rcu_read_unlock(); mmap_read_unlock(mm); /*