From patchwork Mon Sep 25 08:36:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13397491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28088CE7A89 for ; Mon, 25 Sep 2023 08:36:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B75C28D001B; Mon, 25 Sep 2023 04:36:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B25DD8D0001; Mon, 25 Sep 2023 04:36:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C6698D001B; Mon, 25 Sep 2023 04:36:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8A9A68D0001 for ; Mon, 25 Sep 2023 04:36:27 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 56CB71CA762 for ; Mon, 25 Sep 2023 08:36:27 +0000 (UTC) X-FDA: 81274463214.20.36DEACD Received: from mail-yb1-f180.google.com (mail-yb1-f180.google.com [209.85.219.180]) by imf07.hostedemail.com (Postfix) with ESMTP id 8772C4002A for ; Mon, 25 Sep 2023 08:36:25 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0oL+gG2h; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695630985; a=rsa-sha256; cv=none; b=qFXlgP3enb0AFrCueCufIEAMqPmtUa5cInifUq4Gr5+Z5U46iyqfGveyO1VGcPJnguWMuJ 84F8AgdDNLRDxQGXJ87Il+MZkBlGSeE0zIG8CUM1U45EEL+SeDQKPcPId/ETZGyJ+2PApD KGi0eFitClB92+Nun+iKZLaz49W9glE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0oL+gG2h; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695630985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OU+65snbrvpR+Q3ulaNy8J1cYx1gkkm9B2VLdfzl674=; b=5tXEQsmFb2j2rGyS62kMPpyY6SeT0cKPkh2Ouooz7AT8WotQzIZwOcDxj2DEVZQkG66lmS CRCP5WEmdyjkEsflqnFFaFc2+kfDbia2uJD2D76K1/OAyqvWLR0TBBVFq7wRWecAzKsPgv Z/HJpSdFzJ2VQ2YQryYRc7KAVI0b22g= Received: by mail-yb1-f180.google.com with SMTP id 3f1490d57ef6-d8164e661abso6765042276.1 for ; Mon, 25 Sep 2023 01:36:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695630984; x=1696235784; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=OU+65snbrvpR+Q3ulaNy8J1cYx1gkkm9B2VLdfzl674=; b=0oL+gG2hE0zDZR0iS6nCfmlgo1Ti2MAWzKPE4PGfnj81bEnaPKIWz9B7zn0rScTiCc /SWlqx96uj4Wptdovd4QaImgm1UUlAFqVsreIjzw7R5pO5azX+zccYzEtCUinvKbPVEX lse/sJKn2IIgwM0WN7bO2Dv3u/k1IMjFuZLFUq3dSZSp82QLC8w6tVmMbZzZ5wWeIQga mhK0zjr5Ui9cYL5rrP1Gn5vFDBEXBjf1t0TLwJsT4xI2BE+BdcN/3wgAEH81my91QCxG LfvnYTnjWh5k5sx5mvaNp7ZNcMn7GdnGRNmqQ4bhpuWdrHEuZdWnvlBefHthmuZ5gJaP LR1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695630984; x=1696235784; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OU+65snbrvpR+Q3ulaNy8J1cYx1gkkm9B2VLdfzl674=; b=OeXDN9Wu8SjXE/TCU7QgAacYK9Fs95z7uGdPjltOZaw3NW9EqJQLaK/t9TLStpTW2E 7hwz+LHokOV8TYZxc50f+Y9x4BIAo8CBg2WfQZLS5CRHiLrl/zQdKJmfpMIny2/3YPoy i+0mru1Qju73K7uz1y+QYE6+cBezgBidDEaMs/21J0IwbiDVlaz5kqkekQ8u98yLHld8 6GWmmaQaLtavLvIyCGyVM0eywRl70WVHvknQX1q4QppW9UB1XmDqv54QNr/zEq8DD5hn TlSPgeiyyCQFmkH5z3gpwMVXG7DG4NBwbWmRAjMM2tPRkwG3RfwCOJ4vdYfiOwOrLTj8 rrrQ== X-Gm-Message-State: AOJu0YxpTJORRP7e2N7rDN7MkUVUl2lS9wTfToZUd2eINEJO3FVugWHZ wtzpGKPofmRi0zGrEMzAEimrzg== X-Google-Smtp-Source: AGHT+IHXAX1QQV6PVx66keFoYyD4CtYdpXj+z0LhxW9OnhchJFVwm0d282m2rz466G2xal6CpuLvhA== X-Received: by 2002:a25:c092:0:b0:d53:f98f:8018 with SMTP id c140-20020a25c092000000b00d53f98f8018mr5028309ybf.65.1695630984516; Mon, 25 Sep 2023 01:36:24 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m17-20020a258011000000b00d85abbdc93esm2014417ybk.12.2023.09.25.01.36.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 01:36:23 -0700 (PDT) Date: Mon, 25 Sep 2023 01:36:21 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Andi Kleen , Christoph Lameter , Matthew Wilcox , Mike Kravetz , David Hildenbrand , Suren Baghdasaryan , Yang Shi , Sidhartha Kumar , Vishal Moola , Kefeng Wang , Greg Kroah-Hartman , Tejun Heo , Mel Gorman , Michal Hocko , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 12/12] mempolicy: migration attempt to match interleave nodes In-Reply-To: <2d872cef-7787-a7ca-10e-9d45a64c80b4@google.com> Message-ID: <21a2895b-4590-a0f4-c81c-67e059494583@google.com> References: <2d872cef-7787-a7ca-10e-9d45a64c80b4@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8772C4002A X-Stat-Signature: 5twmu99qymag6f6x954mum874oxnqrrt X-Rspam-User: X-HE-Tag: 1695630985-309080 X-HE-Meta: U2FsdGVkX1+oNZK3BLoYJOPg1S2BgjXiC3kedJfodb2E/xIeEl0uE5UnInObpgP8mF9lNU3+KM3eLxgfQpT3vaUP6ysJEyHPap7hiRqi534Xajkl+uKMhLiAbZO+hLUkXwS0or6t3LpfoqNrnbHrTvbvTY5F6/9ZVIo8KOqj0i/qnpfwYQIUMmumL/1MB3EWdH64d8HVYvlvNQrwgQIq7S8B1SW9IJhaT8jcYWZwkChKpMW6baLFHXYe3gD23u4RExVHlh6jgusAZPZAwBf39GxZdOlKC2f+XVt2/dEUA9RiJS3uEfKPiFaG4mw1ickDwtb6D3QlUabdHfx7d59OY0Lfhd05odfnHwUfUs2yhzR+ZyZPU5xovNUqhqJm+gsi1aPN0C/IPxSQlXGUn9tju5oUm2B+UuQq7LvymkiCfDncH5ZLKWGDGOB2jtd+dUDShZJvvXwh54GzlePZBUhiqBAlQSUaDEwHglCAzqdO58zG9pU/pXLq71tJxcRN636NYnv0z3Nqk22j4Dd9BqaE7Ncs+EPcjuR1cLcxo59nMPYIHAVX7neiUq9XSRNdpa20ArTzI0LKaAskINvvpueInMl0fquNVhGeQEZl7HxdjCBhuqciqkccMcYl7U6pKaWPjEVMh0O19QjA/qDqp7VNnomDLF4L8mSpL6Jb+ari7O/qybTHVKY/ZfPyodzohpXvdpJKr8+HuqFAmIyTH725HPy4zEQz04bOWBET4lxBe2gqbI3ZD+94YfnoM9PofewOqn8CvQtgQ1K3bmu8xv67U3mHpZeJ+iilIh853c9YrND9V5b4rzgMaziCSaVvOFFcPto7tvEM0JRvuEsFAJzQd/MYxBOQSA4xV8Is89byIRFevgVq+7gGMUlZK5VpEZclsvQesq8cPJPOlklgnJWXD1jV26V+f0hgzTwLsv6GV9LOmAWUsjHUP+T6d8uSl3HXjACBdHsfKOwx3EB9CTy QseTAPbC UnhQu6VFYWaRKGEHnUDf76pvvw/p8bPge9om2pbAPAi/PF3+lkM9QKdr7Ij8VearJ9s1LHpzQqfWRH0sI/0UU6OcDE1CBXWYo9DhOFNn04aW9jtB6/rjQjiNjcON1+vkBUcQAAh69ZwIuzhdlKtpYOAygYyWoOI7LTBNTJ340NbNbJB33jVXqQbVGzrT8MauqzaypZv0Ljd1vsDjifzn5P4LjOHwTlYd5BC5GEPC1Fnpsy56POmjZ5i7DwqeDIRXpS3NKew3/RTHzlUK9t5m4sW8o/znh3HeasTbPSqDjfda+XTlErhMn+qYD6qVS1lG3uvALTo+mxyUf8P6aHK6BuPf2sc7QfSYEiPSsFN/XARgPh48= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Improve alloc_migration_target_by_mpol()'s treatment of MPOL_INTERLEAVE. Make an effort in do_mbind(), to identify the correct interleave index for the first page to be migrated, so that it and all subsequent pages from the same vma will be targeted to precisely their intended nodes. Pages from following vmas will still be interleaved from the requested nodemask, but perhaps starting from a different base. Whether this is worth doing at all, or worth improving further, is arguable: queue_folio_required() is right not to care about the precise placement on interleaved nodes; but this little effort seems appropriate. Signed-off-by: Hugh Dickins --- mm/mempolicy.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 49 insertions(+), 3 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 74b1894d29c1..7bb9ff69879b 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -430,6 +430,11 @@ static bool strictly_unmovable(unsigned long flags) MPOL_MF_STRICT; } +struct migration_mpol { /* for alloc_migration_target_by_mpol() */ + struct mempolicy *pol; + pgoff_t ilx; +}; + struct queue_pages { struct list_head *pagelist; unsigned long flags; @@ -1175,8 +1180,9 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, static struct folio *alloc_migration_target_by_mpol(struct folio *src, unsigned long private) { - struct mempolicy *pol = (struct mempolicy *)private; - pgoff_t ilx = 0; /* improve on this later */ + struct migration_mpol *mmpol = (struct migration_mpol *)private; + struct mempolicy *pol = mmpol->pol; + pgoff_t ilx = mmpol->ilx; struct page *page; unsigned int order; int nid = numa_node_id(); @@ -1231,6 +1237,7 @@ static long do_mbind(unsigned long start, unsigned long len, struct mm_struct *mm = current->mm; struct vm_area_struct *vma, *prev; struct vma_iterator vmi; + struct migration_mpol mmpol; struct mempolicy *new; unsigned long end; long err; @@ -1311,9 +1318,48 @@ static long do_mbind(unsigned long start, unsigned long len, new = get_task_policy(current); mpol_get(new); } + mmpol.pol = new; + mmpol.ilx = 0; + + /* + * In the interleaved case, attempt to allocate on exactly the + * targeted nodes, for the first VMA to be migrated; for later + * VMAs, the nodes will still be interleaved from the targeted + * nodemask, but one by one may be selected differently. + */ + if (new->mode == MPOL_INTERLEAVE) { + struct page *page; + unsigned int order; + unsigned long addr = -EFAULT; + + list_for_each_entry(page, &pagelist, lru) { + if (!PageKsm(page)) + break; + } + if (!list_entry_is_head(page, &pagelist, lru)) { + vma_iter_init(&vmi, mm, start); + for_each_vma_range(vmi, vma, end) { + addr = page_address_in_vma(page, vma); + if (addr != -EFAULT) + break; + } + } + if (addr != -EFAULT) { + order = compound_order(page); + /* We already know the pol, but not the ilx */ + mpol_cond_put(get_vma_policy(vma, addr, order, + &mmpol.ilx)); + /* HugeTLBfs indexes in hpage_size */ + if (order && PageHuge(page)) + order = 0; + /* Set base from which to increment by index */ + mmpol.ilx -= page->index >> order; + } + } + nr_failed |= migrate_pages(&pagelist, alloc_migration_target_by_mpol, NULL, - (unsigned long)new, MIGRATE_SYNC, + (unsigned long)&mmpol, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL); }