From patchwork Wed Dec 1 14:30:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 12650323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D83D4C433EF for ; Wed, 1 Dec 2021 16:09:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5AEC6B0073; Wed, 1 Dec 2021 11:09:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E0AC16B0074; Wed, 1 Dec 2021 11:09:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C352A6B0075; Wed, 1 Dec 2021 11:09:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0172.hostedemail.com [216.40.44.172]) by kanga.kvack.org (Postfix) with ESMTP id B57E66B0073 for ; Wed, 1 Dec 2021 11:09:19 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 6EB391808892D for ; Wed, 1 Dec 2021 16:09:09 +0000 (UTC) X-FDA: 78869709618.02.7F1B248 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf25.hostedemail.com (Postfix) with ESMTP id E35D9B000E14 for ; Wed, 1 Dec 2021 16:06:53 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1B1EGXh0017179; Wed, 1 Dec 2021 14:30:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=Xc1nPb6L1kC9bXBMRDie6o+mvdL3FssVZJRSfw/mHOg=; b=BgzLGC/EqXzHvtqyGvAZ0Ct+c0XnGrKc41EOvLRMeum1p3jqYatCCf3z3QNgZ4+P7NHi dZAbPWUwfwBQix1gR3ApQb+Kr4wr7lDjSpH39Z6QrPv3ZyyWD9yZGJeoyGOMLBFX0+v0 gl8Ibm1Ersd+RKCGqgkeO3f41/kT/SNDZofQgSTu5x+AeR/y2Gj0S1i6JOwZaV5Y37FB yVEwyTN55XMOdXkPam6VuSkl859luSsYylsOtsP1ZrTmXb/WqfO+X0SwAU46bgB34sLa X5eScm39BhUQmf2Ddrhl2oGWUVZe9B/kNwi1pRfFcdq0xMCUfflpP3NiplbJU3ctzF0I 6g== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3cp7weh2ww-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 01 Dec 2021 14:30:48 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1B1EBl7d109791; Wed, 1 Dec 2021 14:30:47 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2102.outbound.protection.outlook.com [104.47.70.102]) by aserp3020.oracle.com with ESMTP id 3cnhveqcc9-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 01 Dec 2021 14:30:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oJleJKLJjpTvQXy2WAZGNGgJcImMCIbCZHOaLEC5dZcFi+KiFF2A2wUyw7T78OiMR9NRrGv8cYnsPL+Wr3ZGl6vxKBCPPt7cAUoRc8LyCzo9KffO3zv0ZQU5BUjLVKBi3pOCoZbuI29UzaZ1BV+CJfTyMOH0P3mq5mRAOgdUXM/E6IYG/3HTMffhObPioZEnQbZY8wk2VN7IzclPjiQh+vJyhKULTYqZE6Ld9LuTluv1nkS2Qa+vbYVz3ziz/x1yScr/PKyRvwF+kRXD5vZgURuRAme48+QdyYEvN24UCd/tEqcPtyVXokVM+sK1pY7cxz0iMb4IENb0T3m3SC0reA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Xc1nPb6L1kC9bXBMRDie6o+mvdL3FssVZJRSfw/mHOg=; b=PI6OvIuZJIVtT+6E8ETW1FdYrxqtwvNeKH0wLRm6KM//n1yY6a/t9+NJtx4bSq6jJkarnN56S5GV09/hQlI498rvheRTutw1sVSHTuv4VNIdCVLrhJXUm/cTCOdMPu+w1gt2zyDuHKPDxAo7LYV5zyxjmMyZnE1kydihMxZS/nSK+rjXTaQoAhBFH8GzvRoBvp4nl2ElIX0623PmAyD5AyWsMyeO2wTC2FXi7pOBtnhj/J5c9S4sm5/lYgU7WStgbk+JI0Pd8nprKeKyTIaVUP54LYaPv0QeEIXaPq9f/EGWF9KolYgkjxxJXpbAGZbE3vb8yvWXtJWNuWWNbw1h3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Xc1nPb6L1kC9bXBMRDie6o+mvdL3FssVZJRSfw/mHOg=; b=qfmcxBzXiBSpg83QSi0QD/z6REghouocJ1aYHwD/KMeRthHsj777Cxtig0xIzVJUEvs7IobFUi7jcmWmX207JqmxJsdB3sYj1kvhSzPckfpW30axxmlGo7zFSqNPj4+MKoE26i+M9d6UK1Kxe4EkJGNj80YynXIZlB2o2WSBAEw= Received: from SN6PR10MB3022.namprd10.prod.outlook.com (2603:10b6:805:d8::25) by SA2PR10MB4508.namprd10.prod.outlook.com (2603:10b6:806:11d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.23; Wed, 1 Dec 2021 14:30:39 +0000 Received: from SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::c4d0:8291:84d:ac66]) by SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::c4d0:8291:84d:ac66%7]) with mapi id 15.20.4734.027; Wed, 1 Dec 2021 14:30:39 +0000 From: Liam Howlett To: "maple-tree@lists.infradead.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Andrew Morton CC: Song Liu , Davidlohr Bueso , "Paul E . McKenney" , Matthew Wilcox , Laurent Dufour , David Rientjes , Axel Rasmussen , Suren Baghdasaryan , Vlastimil Babka , Rik van Riel , Peter Zijlstra , Michel Lespinasse , Jerome Glisse , Minchan Kim , Joel Fernandes , Rom Lemarchand , Liam Howlett Subject: [PATCH v4 47/66] sched: Use maple tree iterator to walk VMAs Thread-Topic: [PATCH v4 47/66] sched: Use maple tree iterator to walk VMAs Thread-Index: AQHX5r/yrE5Bi998mU2IglLSh3JMzw== Date: Wed, 1 Dec 2021 14:30:11 +0000 Message-ID: <20211201142918.921493-48-Liam.Howlett@oracle.com> References: <20211201142918.921493-1-Liam.Howlett@oracle.com> In-Reply-To: <20211201142918.921493-1-Liam.Howlett@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.30.2 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fb3ebf3a-728f-4a03-a3bf-08d9b4d725c0 x-ms-traffictypediagnostic: SA2PR10MB4508: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3044; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 0p3hfCAV4rlcYtFZyj5JfDZCVXW4xj1rIHoWW8Zjgn7klMO7xYkbuNxCg7dCL4XPpRwyt/4a0Mwb8Zd/RLA0pttxOrS5SyBD9xT276z/gS85GB2VnN9844esVSrdaB7mAm/D4whKzerQqq8Um0Hda/dPdRxLDxG2beZ54KK2R8ltof4gzhppckFAvwYf2wgCgiXhMgTyM9nhPrwxdOdjo0WGwkIVsq2BTXAXWIVQxfPD9Yt7OtRydxQY6B5suQoQwewv+rBfuA0+zIJUE8Xqt/96qjfMSlp4KZ/FKdMCkFyUfkS9b8f1ZCQB/J07fgel3G3J+r7Y/7MWaNRJB9MpFqHT/5uRC8bP757TFeAPjLfFGRUCKRbuX7+4TloVz+2OTGele/Ros89iYgi5paNcRIaeJOcvOS67yZeWflwKSiccXLFIqONFewxbbzM+A6O9ICn0gzFkCg3G4dHzQNbYBmb3GbLTyquRiA76M+NIPnayjPijoMTtSZQYSJP1cBEi96iDbVHx3e+gPUDgRZ7PG8i8whe+U46Yf3diXDy3//cmvJSZdhIbC4JLYQaeM9JbCBOwj5xwSp6meV1Z3L1vn8+0nVGyWfo/5g/YkUiV9gB2MfVgpLbkTiOx6hk5qV3TC3JVIEkzL9i9fKFgE2LGjl74EDaSvzJBiI2Ff1mZVDx2Qia+ToCNZXG5WQ46pDK6 x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR10MB3022.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(186003)(26005)(7416002)(1076003)(2906002)(8936002)(54906003)(83380400001)(2616005)(38100700002)(71200400001)(66946007)(86362001)(66446008)(8676002)(6506007)(38070700005)(6512007)(6666004)(122000001)(508600001)(110136005)(6486002)(64756008)(66556008)(76116006)(91956017)(107886003)(316002)(5660300002)(44832011)(36756003)(4326008)(66476007);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?1+v4e4mu4tBRzOAftezk78g?= =?iso-8859-1?q?i/Kir4IIVncYvK6mhhRqEVYq4lZcDgnJkxUCO4xPc3K8YWy8gW0y5/AS2gr2?= =?iso-8859-1?q?n1oUOpQUgquyGTzuwemIUqdpsDCDDCviJopJ8a9Y0toiylkh6VVe3EPByqep?= =?iso-8859-1?q?/gLWyWVMmb24MPyF7TkgQNKg2zhia5gLX/tpioiq2fNMEDOUNd/QxDk3xD4O?= =?iso-8859-1?q?2yYAqwGZVoFF2FcTN/Md0i3mPYSE8uZ57tNmEIoZRWKjCpQSfHiivHN/u8f/?= =?iso-8859-1?q?PrfdExV1Ns/wzyIULVzwgJLUzzQLdYHhFoWsbXwu4SMJyNQ6sLSRx3zEjjYl?= =?iso-8859-1?q?qAygJ3hmQYuKBteOv58IebjPTLOtwpDH3rE0s46Q7UvWYWQ2jJiO3et5nraz?= =?iso-8859-1?q?E9Zgj0lTSzc8hqv5dfn42OfVa2nMD8Z8Ee6Bw8SU42bfTvDE4Lx40cETv7eO?= =?iso-8859-1?q?g1ciOwBNKLvYhYVnqLdFdfZ2XbGF+A8TX1OFtKyJA8+bV+EJgUfJ2efg2coz?= =?iso-8859-1?q?DWWeo81FpRNxNol7wTYxJBxJdIH2WIHTny8D2579dlUCKahq8jjvSzkqjHLO?= =?iso-8859-1?q?Apf6Oib1l3299sqzOZj7lH0logTzW0euZE8OgOUVjMOG2SaIQQlwP0KeDrDR?= =?iso-8859-1?q?Om3Pa65B8hk7KolLAuSRq6DaLQ8AnlvZKJ0D7w1jgWwJUA33kiSN7xd8g2fs?= =?iso-8859-1?q?fUebgWuBC4F8FaIObhK9FJEdTbRNhVhKnS8kpYXgWax7UszzmS19tbRIOW1g?= =?iso-8859-1?q?UXUB7Kj0f3j5oFV6nVfu7PS4E13ew5bnpmnOlUddyV/SOIIj2ZlXJRfFyvsI?= =?iso-8859-1?q?iGhexIVRmSJ62LevKt3VCAYrZgkc9vJPw2er8gLqukPATEg4RJu9/Ri348xE?= =?iso-8859-1?q?U+SQSGsAe1/NADYlEyPC2EgdMwzcTXeiXVDVove+I7kRn1W0NHvmi7L1mefX?= =?iso-8859-1?q?VaM9mGICYcxbJOCiDodBEmvzomzqxzE8SUMkNVuSIqjLWBkW8DEBZEmT+sWf?= =?iso-8859-1?q?C0mHfvdQBcHcSpyPl87hxKKDufQ8vbDObIZIZ8nc/gBShV0RZ92ywPjpDEY3?= =?iso-8859-1?q?wmHwd00J1MzRc1oS6BvL7ZnYeL2gGn+4NL2yIOeRar5Fm6L88sS5+7HRgUvs?= =?iso-8859-1?q?vp+9uBIxS+dmdLEtjrFPxWfLJyVpLPaFfkDXNiAiircCvMZaLo6IygAKDhps?= =?iso-8859-1?q?YkGk5X8qAsycToOmDN28xYiuM+dR2TbHOMWf/QDQjsMjB54pHSJXg3KkeUKy?= =?iso-8859-1?q?Q9ZeCER7UhyhEjTfC2xrRAlvi/uj9ZogtN44x94xgKgRxZQ+ptVfe4GEn0uc?= =?iso-8859-1?q?Qhw/Oma3VCfKOF3uQmD5KLpMGh65sc4Q++456CSwt2X+ttyAgaSx6tS3PUzU?= =?iso-8859-1?q?PfyL+zr7FT7pyerucvHOpQLsf80n6wnpHQCmgE7yPYQ7trk3IGbGqUTMl5Ol?= =?iso-8859-1?q?MAvXqgCHgdGTkrGwVUa8QYP5yeGrW29ubOHMKLx0+BabUzNJSkoG9tSUz6dE?= =?iso-8859-1?q?mYqQjU1+wDMYSaz68r+bdsl9B8m+FovnMY3RSxqIt0kMgziLgBj7DmGbR+oN?= =?iso-8859-1?q?ERmTkMODyY0BNWnSEs+Wqv0KDkJO+0ADPPZvYEpvRoS/iZlmDX0WUyXS1uCG?= =?iso-8859-1?q?OviFWIF8v+CPGYzSsXcZa9Ka8DlUmkD58UuJ3/vg2+AphtX0dHGntsPGDvlg?= =?iso-8859-1?q?=3D?= MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: fb3ebf3a-728f-4a03-a3bf-08d9b4d725c0 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Dec 2021 14:30:11.8753 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: qgglggDQCRi07Dt02d8qh2soNZp8VJFmdLgv7st2SQP7lKvM9oJcjgKC1147vjf75MKowtpA+CI9DeAyw7CuFQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR10MB4508 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10184 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 adultscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2112010081 X-Proofpoint-ORIG-GUID: qW7zYIr58mlmK_AjOVqfjFWa3ZM1zbjc X-Proofpoint-GUID: qW7zYIr58mlmK_AjOVqfjFWa3ZM1zbjc X-Rspamd-Queue-Id: E35D9B000E14 X-Stat-Signature: o48jdhs9fcgcd9ot3wz1g9451c3qogp8 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b="BgzLGC/E"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=qfmcxBzX; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf25.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=liam.howlett@oracle.com X-Rspamd-Server: rspam02 X-HE-Tag: 1638374813-772314 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" The linked list is slower than walking the VMAs using the maple tree. We can't use the VMA iterator here because it doesn't support moving to an earlier position. Signed-off-by: Matthew Wilcox (Oracle) Signed-off-by: Liam R. Howlett Acked-by: Vlastimil Babka --- kernel/sched/fair.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6e476f6d9435..39bb4a6c8507 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2672,6 +2672,7 @@ static void task_numa_work(struct callback_head *work) struct task_struct *p = current; struct mm_struct *mm = p->mm; u64 runtime = p->se.sum_exec_runtime; + MA_STATE(mas, &mm->mm_mt, 0, 0); struct vm_area_struct *vma; unsigned long start, end; unsigned long nr_pte_updates = 0; @@ -2728,13 +2729,16 @@ static void task_numa_work(struct callback_head *work) if (!mmap_read_trylock(mm)) return; - vma = find_vma(mm, start); + mas_set(&mas, start); + vma = mas_find(&mas, ULONG_MAX); if (!vma) { reset_ptenuma_scan(p); start = 0; - vma = mm->mmap; + mas_set(&mas, start); + vma = mas_find(&mas, ULONG_MAX); } - for (; vma; vma = vma->vm_next) { + + for (; vma; vma = mas_find(&mas, ULONG_MAX)) { if (!vma_migratable(vma) || !vma_policy_mof(vma) || is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { continue;