From patchwork Tue Oct 3 09:29:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13407346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2720E75438 for ; Tue, 3 Oct 2023 09:29:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4EDD1940008; Tue, 3 Oct 2023 05:29:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 476C58D0003; Tue, 3 Oct 2023 05:29:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F0EE940008; Tue, 3 Oct 2023 05:29:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1A15C8D0003 for ; Tue, 3 Oct 2023 05:29:07 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E0B89A01D3 for ; Tue, 3 Oct 2023 09:29:06 +0000 (UTC) X-FDA: 81303626292.27.9944507 Received: from mail-yb1-f180.google.com (mail-yb1-f180.google.com [209.85.219.180]) by imf25.hostedemail.com (Postfix) with ESMTP id 167F2A000E for ; Tue, 3 Oct 2023 09:29:03 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nk13zoZs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696325344; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wvvp9tOZAm0ZnFPspgZt0Lcdrx1YEcxNfvbw4PaEVR0=; b=or+htEXIJCxYp3JS4YiDzaqweWJblWmcMiW11bdtnSl6O82O+bJBD1bV9WRTcU3hpT336H ywLbQD0Kss5cjFb3BRM6igeqmhQ9Jzduh6MW6u5x4wwVKfIVL4//Hpb6D1zNGYnbCPA/PJ CqJGZdjg4PBlsMG3sVFPymqWMTm4y2A= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nk13zoZs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696325344; a=rsa-sha256; cv=none; b=oIk2iPzySnDQmTDF2+WtiM6UQrgB3DLU6DIyUHY5rehpfjzXynBkCvLZNthD3Snnpi05Lf bGB39mHD9Sb+mu4Sc1WBt6LN9iBWo7YPpYi9QApqp0CbRbqWFqqMYInH+wNEXFc0p/bF0J 8uVefV68ZM1zaOwfF+ooWoBq2bcbhDQ= Received: by mail-yb1-f180.google.com with SMTP id 3f1490d57ef6-d89491dab33so736023276.0 for ; Tue, 03 Oct 2023 02:29:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696325343; x=1696930143; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=wvvp9tOZAm0ZnFPspgZt0Lcdrx1YEcxNfvbw4PaEVR0=; b=nk13zoZsUYFof7LjGrJwvOgBHed/Olg6MhjReFQ3CBvIVtQtdoFFLb6NJsyoCY//kF qQf+bOnZYsr93ZbeoOzI5R+qB67H1AB6V5vdlQyQBwI2bVpExRQmfyZJF0/CgPl9reWV i/LjfErOD36974WUjrGXMk2pdnnVfvxM37LYs90+zI0b+Buw8jTEhPfznqtEq6yTX4uA XVpQZRnGLBsvTA/fWYc5nUPkAQDuwgjHnEAT9ZL+hDY6wwJzj8ShdfO5ekNBZ60ekepr 3YXsiJs+tpluLcVV8hrmNHv1YXItabP5zNNuyuaYuhEXCSfFY3Bg64vwtxpIlwCkiW/I 9eKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696325343; x=1696930143; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wvvp9tOZAm0ZnFPspgZt0Lcdrx1YEcxNfvbw4PaEVR0=; b=nJQ3U69mZyL4hPiRqEwlovvYCRCgsDQhv+Ks2F0bd6ovwuvTMTSEeM6lec4KJThRSF JnCeYvSwnuXUt20/+jOyYCPFYFiTblQOWspgelFGeDjg8e3jIXOQNJ9Oe8Nq5mFgMjqt lh/ca0rXXOAmJ7W+HPHu5c9ARpmGFUYEWrAC9NxeHauIssQkvR5HtvepDkEEZYLLR65A O1UtUiXQ2AkGsBCbu5LhJvDLB2IqIwq+gFIdZnCCon388q8wDYHaNBMu+Injr4WMng5N hZ5M4fAlDEYCkiHlykUPTx1mIQcXRApsLHNVzxcyzsi2rpmqj/V83yG0bC1dtHWcBUGS dqiQ== X-Gm-Message-State: AOJu0Yxj6ZQ/ZSvY+XZHC1nztaoeoV6khqSm/hmYMoxAhcqoscfJ6hoQ w1lJpjsFZcjZVs6p69g71oc96g== X-Google-Smtp-Source: AGHT+IG82Q8rMpJY3BZE97NKeDHGMxciJo3XfpIVLFLHZCNcKqiXapr+aAI39Q2ReaCSAdFAa0T/Yg== X-Received: by 2002:a25:da4e:0:b0:d7a:d628:f69d with SMTP id n75-20020a25da4e000000b00d7ad628f69dmr13139965ybf.32.1696325343062; Tue, 03 Oct 2023 02:29:03 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id w196-20020a25c7cd000000b00d1b86efc0ffsm291695ybe.6.2023.10.03.02.29.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 02:29:02 -0700 (PDT) Date: Tue, 3 Oct 2023 02:29:00 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 12/12] mempolicy: migration attempt to match interleave nodes In-Reply-To: Message-ID: <77954a5-9c9b-1c11-7d5c-3262c01b895f@google.com> References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 167F2A000E X-Stat-Signature: 9nhzo13pk15ez69s7bkk9rw5ed8hrkq7 X-HE-Tag: 1696325343-119406 X-HE-Meta: U2FsdGVkX1/G970KLd/5NrkBdE1K239kv3vxDCzc6ZyOen4oKgZl9fMWYBtdDyKQOX9zE2l3UJ1xCcScsu+EM+19F1ei/jLzvLEbXoOAm5yJNcQz1lg1xRR9lCUeskIHxcDnqTJuTEKaYEY3kVy76ZJoXPi3C5TiszmBYL2HWa9sCsfUpyrG3xb+5T8nsDaTk0SGA3RlGEJR+me2xnWwYg7Q+Q8Jy+vGdUm+qgXYd9SZi1aIreDfkXDnP/3fDiXWK+PhyS8Ul5g0apkHwXT3xwe9I0YPMlSo0OJ7Fo6qOawMMUlv1yal5A0IcgkZIfNRiUVQpyMC+1sQf4/zH3ZNsou9sRxPbpShLvyqEuwLYi2BSOjLMxDJwL+DskJn8wMIy195NCn8CebBJjM5OYVKGqoUXnOzf0xJ3I9pUynV5DIAIxrCqWsuTa/+zdXLwvak01efr54tMvFCbRHTaghY6TPEUS6ZvtwY+S3v1mCgZJhsMbDhhWCk6sOcNRiP7zC5cJ/FctW4j2m/z5A4AytZLucyQpSbAoZz1HhI90C3ZNI/shIUY6GTnKTe/lO13hfQJMOJWV0dTF2qijq8gjhaZSimQxOGdNLUf3PWMeQ1tbqH3b+UR5PSMlGA+4Wp7ZiB2ruzQoGt0BHtSX+n5XbfrILkYt2ZTU8peEKlkhGapJ59v+4tlYA3DQHyNvf/etIvLg7Uq446YqxkkCt/txU0JlOEIfRniQay8/hv1RYePcBNxpglvUTBr9drgBTvjXuFP8/7dZj3nCNkwec2nZtQBZVKHzE5hF6gjYrk6ivOy2O0qzQ4+SDvT3kVkwrZ5AglJn17QI+0e++v4NE/neW0BM3VZgQRNgmWSJeWAlNQpxQ7/5IM0L/23TKXrwhMmLvBTvIMKiuptWakYeTswQDt1VQnIxrtcFyuwm5YGQ5rnASKgnfmJYGugAW78Mr+qX1j2p9O3cvmR1ccVgwc0qP 441OgCIE sd5Fer0nnzsjRqktpHOArynhofOhT4uJBuIFDypNnLxChiI7elOH66i5Av6acS0yXMOaoeXIvtfCYEijWNUny7Y6txV468QSc6HMzTKwzy2iHnn8MxIgTS9Ssci+EGeCzW9RYfnHKiqc34zDXr9qAvu4KKihk6tpdh/FzWx3XwOTW0m3jEEJW9fmxcmnUZKGCmN8bOlsgpYgk2qT8Wh3RgK0p5w+TLmtPkhZDennS4oWd97p70X3112OTVd4Pc9fU66l9NeffPIS1yk6w6CuGPXEG8WidA6dIKFbXz3W/Sg/SpV9Dh/w1bTnXh2EGF6wPCJDnKkOY4YWquID9DT0wHE5HHQynNoG4AM7INDH8hCFzlwWGGcNXxAiIIFerI1aLo8r+I/xA/sXh9wU2H3jbTYO+r+oHyo4QlS5ddcpHNqywYQzhjcT2cUHbBkXaGLAQ2Io6a0fExoor8+k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Improve alloc_migration_target_by_mpol()'s treatment of MPOL_INTERLEAVE. Make an effort in do_mbind(), to identify the correct interleave index for the first page to be migrated, so that it and all subsequent pages from the same vma will be targeted to precisely their intended nodes. Pages from following vmas will still be interleaved from the requested nodemask, but perhaps starting from a different base. Whether this is worth doing at all, or worth improving further, is arguable: queue_folio_required() is right not to care about the precise placement on interleaved nodes; but this little effort seems appropriate. Signed-off-by: Hugh Dickins --- mm/mempolicy.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index a7b34b9c00ef..b01922e88548 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -430,6 +430,11 @@ static bool strictly_unmovable(unsigned long flags) MPOL_MF_STRICT; } +struct migration_mpol { /* for alloc_migration_target_by_mpol() */ + struct mempolicy *pol; + pgoff_t ilx; +}; + struct queue_pages { struct list_head *pagelist; unsigned long flags; @@ -1178,8 +1183,9 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, static struct folio *alloc_migration_target_by_mpol(struct folio *src, unsigned long private) { - struct mempolicy *pol = (struct mempolicy *)private; - pgoff_t ilx = 0; /* improve on this later */ + struct migration_mpol *mmpol = (struct migration_mpol *)private; + struct mempolicy *pol = mmpol->pol; + pgoff_t ilx = mmpol->ilx; struct page *page; unsigned int order; int nid = numa_node_id(); @@ -1234,6 +1240,7 @@ static long do_mbind(unsigned long start, unsigned long len, struct mm_struct *mm = current->mm; struct vm_area_struct *vma, *prev; struct vma_iterator vmi; + struct migration_mpol mmpol; struct mempolicy *new; unsigned long end; long err; @@ -1314,9 +1321,45 @@ static long do_mbind(unsigned long start, unsigned long len, new = get_task_policy(current); mpol_get(new); } + mmpol.pol = new; + mmpol.ilx = 0; + + /* + * In the interleaved case, attempt to allocate on exactly the + * targeted nodes, for the first VMA to be migrated; for later + * VMAs, the nodes will still be interleaved from the targeted + * nodemask, but one by one may be selected differently. + */ + if (new->mode == MPOL_INTERLEAVE) { + struct page *page; + unsigned int order; + unsigned long addr = -EFAULT; + + list_for_each_entry(page, &pagelist, lru) { + if (!PageKsm(page)) + break; + } + if (!list_entry_is_head(page, &pagelist, lru)) { + vma_iter_init(&vmi, mm, start); + for_each_vma_range(vmi, vma, end) { + addr = page_address_in_vma(page, vma); + if (addr != -EFAULT) + break; + } + } + if (addr != -EFAULT) { + order = compound_order(page); + /* We already know the pol, but not the ilx */ + mpol_cond_put(get_vma_policy(vma, addr, order, + &mmpol.ilx)); + /* Set base from which to increment by index */ + mmpol.ilx -= page->index >> order; + } + } + nr_failed |= migrate_pages(&pagelist, alloc_migration_target_by_mpol, NULL, - (unsigned long)new, MIGRATE_SYNC, + (unsigned long)&mmpol, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL); }