From patchwork Thu Dec 17 18:52:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CB59C2BB48 for ; Thu, 17 Dec 2020 18:54:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 623CD224D4 for ; Thu, 17 Dec 2020 18:54:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730210AbgLQSyJ (ORCPT ); Thu, 17 Dec 2020 13:54:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729161AbgLQSyJ (ORCPT ); Thu, 17 Dec 2020 13:54:09 -0500 Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 229F9C061285 for ; Thu, 17 Dec 2020 10:52:49 -0800 (PST) Received: by mail-qt1-x832.google.com with SMTP id u21so20819817qtw.11 for ; Thu, 17 Dec 2020 10:52:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=moKlK00n8Mdrp7ywZpC6MaDpohjdb0npdJnYeg/on6s=; b=ZOxXEpzZE5eoyaS0FfOo5PWGj/4PnUFDMcm9vo0KmaZNneDdSsxRUnXw/684pkorel Zt8q9wUL1+/2aHgfijA0vUU9udDbViRaxBO3weUCGeZChY5KhViitj/diOV3z2x2tttI wda7SeXLHcPsDOl5Ebl9rKvJ0ScaFNPXEeiEs51TIgOzz8yGpGIXtvHxd5OLbkKiFJdr FUyLRdWGVEAqGfOlCyU0Hh5iMHV5x+JuVXzwNcbp1lKW5L3CjWGOeQ4prbHHN/JoUAj8 X3vysNNxwDmbtDj6EOJ2UpLam2FdfdRGtNm4VbBOsITaj3yoa47yJPcF7T5Gv3HbtL6H TndA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=moKlK00n8Mdrp7ywZpC6MaDpohjdb0npdJnYeg/on6s=; b=UprHaVERI6n6DrxckohLOWZ/psvJ+E8gIbtvO27kGlpsQLt7rLNf6MoLyLsflnhRVT 4zVIFQgo2FinHCsJjnl7U6k4jDY4vxC5/jCn38C193RAl5TkVeB2dp2TTG5a5v/JtmW4 D2CZhrsRFyItj1mwZ/1C0gDiDTth6U5eYE5iqnqSsOjFMx0eHqPcaRSmUObbwoJZvb/w +lPVzcVS82wpTkRJ0hF2vpE2lg0IlFeUqOxkutpoKNL463QbYENP9+P4YSXsRHDmHoio CUwNOV1+2UNRun0bI/b6sznDUYDQfX3wdqjolKpNFhp8o8HeaGXBlPRJJ2ApxYdfEQ2D cGzw== X-Gm-Message-State: AOAM533haHflspZaVykraaU6Ev+WUZVqdro0o7vDGOZsSCaBB89n2hHA idGVbfoqirYjKWTK6NONK3a5mg== X-Google-Smtp-Source: ABdhPJzR9Qh7lsqKjSY0T7wIxNIkDhJNQGdnPSt+BzV4Eg4mDwA9KfrzI+zYzcc82IhoDCH10Mujuw== X-Received: by 2002:aed:23d6:: with SMTP id k22mr171171qtc.226.1608231168320; Thu, 17 Dec 2020 10:52:48 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.52.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:52:47 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 01/10] mm/gup: don't pin migrated cma pages in movable zone Date: Thu, 17 Dec 2020 13:52:34 -0500 Message-Id: <20201217185243.3288048-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org In order not to fragment CMA the pinned pages are migrated. However, they are migrated to ZONE_MOVABLE, which also should not have pinned pages. Remove __GFP_MOVABLE, so pages can be migrated to zones where pinning is allowed. Signed-off-by: Pavel Tatashin Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard Acked-by: Michal Hocko --- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index 4f3cf14ac958..f2e50cdd7d67 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1565,7 +1565,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, long ret = nr_pages; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, - .gfp_mask = GFP_USER | __GFP_MOVABLE | __GFP_NOWARN, + .gfp_mask = GFP_USER | __GFP_NOWARN, }; check_again: From patchwork Thu Dec 17 18:52:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D14A6C2BBCD for ; Thu, 17 Dec 2020 18:54:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B2A3E239D3 for ; Thu, 17 Dec 2020 18:54:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730777AbgLQSyK (ORCPT ); Thu, 17 Dec 2020 13:54:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729977AbgLQSyK (ORCPT ); Thu, 17 Dec 2020 13:54:10 -0500 Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C920BC061257 for ; Thu, 17 Dec 2020 10:52:50 -0800 (PST) Received: by mail-qk1-x72b.google.com with SMTP id z11so27390006qkj.7 for ; Thu, 17 Dec 2020 10:52:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=3vDHKEzj0vF17lzh9SYpP31/FhI4ogCEdqejDZ0noAU=; b=EdFqn1VhcOpZMa3J3DFosAQqJr0UlLj2Q1rIqQzWkpAG9UR36YgESwGeCDL61GQsby 4a764EIeuSe/P0aNnH/Dxs8iJo1WUn1lcnOItCvqtGHqbyY9vLhDvhmFdF8vuur9zjjX xsS2eEr09qduALSup64cnWmdEplzUwQijRRV6F+PgHjf21hlrxI95Duk4dkSrH+7/Ozb QFbUMDNgAO5dnh4pOjC1y1EwhWY1yahxoORTL6KUO89qyCkmZ1c+tGOe/NTVvnsU5yqS KKpUaH8Q+u3CChrO/OASeiKpJwbfxJUEh+eu787k5qeewCn2F+ll2wWUMZ9SU8mcYAf1 AYKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3vDHKEzj0vF17lzh9SYpP31/FhI4ogCEdqejDZ0noAU=; b=iqA9XzMoPueQi9ajaL4ZIpUd4C+DzqoLxWRvwGK+KajMsqusEKyOQ0hjUXzeau4xDr bBgsM7i6m3+O5JoZD95TG+zWBVXBgpho79J8uu5xkeHWdtUurCpzpzXtiV4pGkScfmKO ycP/OG9KJNHYMQRw6zS1eKokDWGMgFtWLoptWy2Wt/BpLxiS4doNb3zBucan26BsZJT1 B9vZamBCHwBLGuclKybzxtJZSJKdrFytSFZO7eqgY1jfdoroL026YqgJSYr2EvShuNgd hJGu/O8pN9lbBQLSQSSyHHF587URrRJCB8VR2fxmfJkQI3jcTDrId90iIYvX2udPk2BS sKhw== X-Gm-Message-State: AOAM533hTSWcVANElG/32/HkVXNZpcqSLP3L2JMPvrV9JptIgAvb4nvW poS+JRjYgfW3d1kTq0Tn0N9kVA== X-Google-Smtp-Source: ABdhPJzxDRkkYxOXRPc2n4xnQCeM3Vw8LarCGXmJR3py3nOwsamiM0kvEZTB8gPYQhmKuvOxOnYZAg== X-Received: by 2002:a37:a241:: with SMTP id l62mr690495qke.482.1608231170066; Thu, 17 Dec 2020 10:52:50 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.52.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:52:49 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 02/10] mm cma: rename PF_MEMALLOC_NOCMA to PF_MEMALLOC_PIN Date: Thu, 17 Dec 2020 13:52:35 -0500 Message-Id: <20201217185243.3288048-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org PF_MEMALLOC_NOCMA is used ot guarantee that the allocator will not return pages that might belong to CMA region. This is currently used for long term gup to make sure that such pins are not going to be done on any CMA pages. When PF_MEMALLOC_NOCMA has been introduced we haven't realized that it is focusing on CMA pages too much and that there is larger class of pages that need the same treatment. MOVABLE zone cannot contain any long term pins as well so it makes sense to reuse and redefine this flag for that usecase as well. Rename the flag to PF_MEMALLOC_PIN which defines an allocation context which can only get pages suitable for long-term pins. Also re-name: memalloc_nocma_save()/memalloc_nocma_restore to memalloc_pin_save()/memalloc_pin_restore() and make the new functions common. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard Acked-by: Michal Hocko --- include/linux/sched.h | 2 +- include/linux/sched/mm.h | 21 +++++---------------- mm/gup.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/page_alloc.c | 4 ++-- 5 files changed, 12 insertions(+), 23 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index e5ad6d354b7b..e30d5511ccc0 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1576,7 +1576,7 @@ extern struct pid *cad_pid; #define PF_SWAPWRITE 0x00800000 /* Allowed to write to swap */ #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_mask */ #define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */ -#define PF_MEMALLOC_NOCMA 0x10000000 /* All allocation request will have _GFP_MOVABLE cleared */ +#define PF_MEMALLOC_PIN 0x10000000 /* Allocation context constrained to zones which allow long term pinning. */ #define PF_FREEZER_SKIP 0x40000000 /* Freezer should not count it as freezable */ #define PF_SUSPEND_TASK 0x80000000 /* This thread called freeze_processes() and should not be frozen */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 1ae08b8462a4..5f4dd3274734 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -270,29 +270,18 @@ static inline void memalloc_noreclaim_restore(unsigned int flags) current->flags = (current->flags & ~PF_MEMALLOC) | flags; } -#ifdef CONFIG_CMA -static inline unsigned int memalloc_nocma_save(void) +static inline unsigned int memalloc_pin_save(void) { - unsigned int flags = current->flags & PF_MEMALLOC_NOCMA; + unsigned int flags = current->flags & PF_MEMALLOC_PIN; - current->flags |= PF_MEMALLOC_NOCMA; + current->flags |= PF_MEMALLOC_PIN; return flags; } -static inline void memalloc_nocma_restore(unsigned int flags) +static inline void memalloc_pin_restore(unsigned int flags) { - current->flags = (current->flags & ~PF_MEMALLOC_NOCMA) | flags; + current->flags = (current->flags & ~PF_MEMALLOC_PIN) | flags; } -#else -static inline unsigned int memalloc_nocma_save(void) -{ - return 0; -} - -static inline void memalloc_nocma_restore(unsigned int flags) -{ -} -#endif #ifdef CONFIG_MEMCG DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); diff --git a/mm/gup.c b/mm/gup.c index f2e50cdd7d67..04602e94856b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1671,7 +1671,7 @@ static long __gup_longterm_locked(struct mm_struct *mm, long rc; if (gup_flags & FOLL_LONGTERM) - flags = memalloc_nocma_save(); + flags = memalloc_pin_save(); rc = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, NULL, gup_flags); @@ -1680,7 +1680,7 @@ static long __gup_longterm_locked(struct mm_struct *mm, if (rc > 0) rc = check_and_migrate_cma_pages(mm, start, rc, pages, vmas, gup_flags); - memalloc_nocma_restore(flags); + memalloc_pin_restore(flags); } return rc; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index cbf32d2824fd..3f5ddac5de8a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1033,10 +1033,10 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) { struct page *page; - bool nocma = !!(current->flags & PF_MEMALLOC_NOCMA); + bool pin = !!(current->flags & PF_MEMALLOC_PIN); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { - if (nocma && is_migrate_cma_page(page)) + if (pin && is_migrate_cma_page(page)) continue; if (PageHWPoison(page)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 774542e1483e..ec05396a597b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3808,8 +3808,8 @@ static inline unsigned int current_alloc_flags(gfp_t gfp_mask, #ifdef CONFIG_CMA unsigned int pflags = current->flags; - if (!(pflags & PF_MEMALLOC_NOCMA) && - gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (!(pflags & PF_MEMALLOC_PIN) && + gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; #endif From patchwork Thu Dec 17 18:52:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 574C1C2D0E4 for ; Thu, 17 Dec 2020 18:54:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D72323715 for ; Thu, 17 Dec 2020 18:54:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730762AbgLQSyK (ORCPT ); Thu, 17 Dec 2020 13:54:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730093AbgLQSyJ (ORCPT ); Thu, 17 Dec 2020 13:54:09 -0500 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 46459C0611CA for ; Thu, 17 Dec 2020 10:52:53 -0800 (PST) Received: by mail-qt1-x829.google.com with SMTP id 2so10987954qtt.10 for ; Thu, 17 Dec 2020 10:52:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=7fMarcPKbF8eZ1EkRUpeh/DuV7Asm97DUovse3OVSm8=; b=KAR7G1r8yuIWa8PDp7CdQKfOdfBpq07/k6cAWElzxpy4lPAVDkyKNpz9GBsAjB+Mpz lxfemf+m/mTOfXRnI35Ee7KfkGcGIfv8CsVkvE0jAIdadt6l+8UbKkhv4etz+g6LOVaE YkFRyvV3s4qovarOY2EKXMvRRCDjZQ9rphgdpf/4va2tXF7c1EpmTIp7c9SH3O9qckhc k8Q9ggvp7Iel3UqSuov4qqKqB0FRAf47mOMPiV7GcjAUIh2KC99oq7l4e7U+VCZ1CRcm +cfFMPGkm300n/RM66V7Tst7/0nWpLsrja6P/8KdZrEeLdVvP/swKvoLNUQf+sZI9fON 2O9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7fMarcPKbF8eZ1EkRUpeh/DuV7Asm97DUovse3OVSm8=; b=O/DbRLYK9ws8yVJaUDrZwyPch8HsMzXVm42LjSVWFgGiJj+Ssr7eObpgs3j1XzEbHn svi6z47KXHgKfxRu+6LZBL8MRK9pjEuihaGFWmCnuN2FT6BasvQHM8Sx2Bou1zimpAuX wksjs2eV5hkqeh5j1JY1JzXb5LH1WdU09GbToXfV0Yk5Q/HIW8iozJt1foI6IByO4nXZ LBrN1YX+amNNyB78CYigei2YhkNfm5YW5DzP3sQXjHf0/y0Ou1+wGCUHCB3JfvxNycz0 GCcWOcBPwshuhK6Q0qjCbnbvWg5cinSOOnCfLWqAyyi3P/L/6b4V+8g+pv710xQBcP0l gIkw== X-Gm-Message-State: AOAM532LbPFAuYlBGMbXI7C9Z/7kAUz27XUiT3EyBGdBUlmOUEj+T+U3 7zP74ht0oePsr6HpnH4P1PNG2A== X-Google-Smtp-Source: ABdhPJwomTS5oDvqiBJxZsojUgHU+ie/Ke3GQwZKAqNbibl4PlQ0cRKesaMM2s5bPJsC4PUdrNrPig== X-Received: by 2002:ac8:4f13:: with SMTP id b19mr184478qte.150.1608231172508; Thu, 17 Dec 2020 10:52:52 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.52.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:52:51 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 03/10] mm: apply per-task gfp constraints in fast path Date: Thu, 17 Dec 2020 13:52:36 -0500 Message-Id: <20201217185243.3288048-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Function current_gfp_context() is called after fast path. However, soon we will add more constraints which will also limit zones based on context. Move this call into fast path, and apply the correct constraints for all allocations. Also update .reclaim_idx based on value returned by current_gfp_context() because it soon will modify the allowed zones. Note: With this patch we will do one extra current->flags load during fast path, but we already load current->flags in fast-path: __alloc_pages_nodemask() prepare_alloc_pages() current_alloc_flags(gfp_mask, *alloc_flags); Later, when we add the zone constrain logic to current_gfp_context() we will be able to remove current->flags load from current_alloc_flags, and therefore return fast-path to the current performance level. Suggested-by: Michal Hocko Signed-off-by: Pavel Tatashin Acked-by: Michal Hocko --- mm/page_alloc.c | 15 ++++++++------- mm/vmscan.c | 10 ++++++---- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ec05396a597b..c2dea9ad0e98 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4976,6 +4976,13 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, } gfp_mask &= gfp_allowed_mask; + /* + * Apply scoped allocation constraints. This is mainly about GFP_NOFS + * resp. GFP_NOIO which has to be inherited for all allocation requests + * from a particular context which has been marked by + * memalloc_no{fs,io}_{save,restore}. + */ + gfp_mask = current_gfp_context(gfp_mask); alloc_mask = gfp_mask; if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac, &alloc_mask, &alloc_flags)) return NULL; @@ -4991,13 +4998,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, if (likely(page)) goto out; - /* - * Apply scoped allocation constraints. This is mainly about GFP_NOFS - * resp. GFP_NOIO which has to be inherited for all allocation requests - * from a particular context which has been marked by - * memalloc_no{fs,io}_{save,restore}. - */ - alloc_mask = current_gfp_context(gfp_mask); + alloc_mask = gfp_mask; ac.spread_dirty_pages = false; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 469016222cdb..d9546f5897f4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3234,11 +3234,12 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist, unsigned long try_to_free_pages(struct zonelist *zonelist, int order, gfp_t gfp_mask, nodemask_t *nodemask) { + gfp_t current_gfp_mask = current_gfp_context(gfp_mask); unsigned long nr_reclaimed; struct scan_control sc = { .nr_to_reclaim = SWAP_CLUSTER_MAX, - .gfp_mask = current_gfp_context(gfp_mask), - .reclaim_idx = gfp_zone(gfp_mask), + .gfp_mask = current_gfp_mask, + .reclaim_idx = gfp_zone(current_gfp_mask), .order = order, .nodemask = nodemask, .priority = DEF_PRIORITY, @@ -4158,17 +4159,18 @@ static int __node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned in { /* Minimum pages needed in order to stay on node */ const unsigned long nr_pages = 1 << order; + gfp_t current_gfp_mask = current_gfp_context(gfp_mask); struct task_struct *p = current; unsigned int noreclaim_flag; struct scan_control sc = { .nr_to_reclaim = max(nr_pages, SWAP_CLUSTER_MAX), - .gfp_mask = current_gfp_context(gfp_mask), + .gfp_mask = current_gfp_mask, .order = order, .priority = NODE_RECLAIM_PRIORITY, .may_writepage = !!(node_reclaim_mode & RECLAIM_WRITE), .may_unmap = !!(node_reclaim_mode & RECLAIM_UNMAP), .may_swap = 1, - .reclaim_idx = gfp_zone(gfp_mask), + .reclaim_idx = gfp_zone(current_gfp_mask), }; trace_mm_vmscan_node_reclaim_begin(pgdat->node_id, order, From patchwork Thu Dec 17 18:52:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97255C3526A for ; Thu, 17 Dec 2020 18:54:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7398123715 for ; Thu, 17 Dec 2020 18:54:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730190AbgLQSyR (ORCPT ); Thu, 17 Dec 2020 13:54:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49076 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729161AbgLQSyK (ORCPT ); Thu, 17 Dec 2020 13:54:10 -0500 Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2FB1FC0611CE for ; Thu, 17 Dec 2020 10:52:55 -0800 (PST) Received: by mail-qt1-x82f.google.com with SMTP id b9so20841060qtr.2 for ; Thu, 17 Dec 2020 10:52:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=viuX9mFc6Y6nLVnl1O/KvCl+LYXrPyDQnCgP/Za0iPk=; b=VVz78QQqx7sFfCQ/j5Q78rQykQ4c2Otp0XtoORNMo61NdLSoSK1fAaic2Ls92NM+aE LbPcwlpY1NseHvtd+DLy8yuDhjYYEB+cYX6CjrbIM3Kv+q5ieeDoVEGjPKnoxJfSmEIW FX9s6OuKDB+NIicbye04czbd3+20gJn1o4E3ihZir/YZRe9eeqaxiS0hXVPL/oDfJJ84 qmhL23t1sU4DpOXKjuaMrymP8TnmKYaRxPkFVq5Go+wvdribN9foHMu97oAI0sbnsIT3 HmPSHf3+u3y6NtfD8MPW6I4TNYKxCxAmtwrY3CuYTyKWEDgXV9QJWQHcEqoJI6eNzyWq v3bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=viuX9mFc6Y6nLVnl1O/KvCl+LYXrPyDQnCgP/Za0iPk=; b=IayNRcLOce4DC9GhBKyr0AHuPilNJ+cFnwHjQcON2kG6c6dz5sV4+DVp4yKeHt1R4K A3WzRiYQphttmM/5sSZaWBI6vpj+sBFAtdoZoygSUWIq1zOR96zfT5oyi8MSA2wqt3TC 6OePZac6JOwNfEcpASbcPeu6qAdvErBD5GOiNWaHq/fJ4w+sUY9ZBvYBfmN4y+cT9mUN vhMoTaK3PSnFuW5crrxRUu4KO5L30oVpnaPBGBktp5dmbpii6Y8ikPNpBNri1QC2lgxJ GR/ncD6NoK14ODLqb1+LquDPUtEXu8SNq320f8M/PWcbc/i/sESAL6/AyNU29YwYGSBQ 5Tog== X-Gm-Message-State: AOAM533hxtKbHBQ4R3sSwTgaUwJwFUCcdHaGWAWqkIZYmmXyc1r/HVzX 2PKco9dBHpCqgVQhsTHJk4UqTQ== X-Google-Smtp-Source: ABdhPJx6d4edswqxp8sOrVfO0BwH7R4ikTMieWBrf9/8ltVwjSEqtbN1CBbHHdBFZ4ldDeEwQ9GMkg== X-Received: by 2002:ac8:75c8:: with SMTP id z8mr169492qtq.256.1608231174446; Thu, 17 Dec 2020 10:52:54 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.52.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:52:53 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 04/10] mm: honor PF_MEMALLOC_PIN for all movable pages Date: Thu, 17 Dec 2020 13:52:37 -0500 Message-Id: <20201217185243.3288048-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org PF_MEMALLOC_PIN is only honored for CMA pages, extend this flag to work for any allocations from ZONE_MOVABLE by removing __GFP_MOVABLE from gfp_mask when this flag is passed in the current context. Add is_pinnable_page() to return true if page is in a pinnable page. A pinnable page is not in ZONE_MOVABLE and not of MIGRATE_CMA type. Signed-off-by: Pavel Tatashin Acked-by: Michal Hocko --- include/linux/mm.h | 11 +++++++++++ include/linux/sched/mm.h | 6 +++++- mm/hugetlb.c | 2 +- mm/page_alloc.c | 20 +++++++++----------- 4 files changed, 26 insertions(+), 13 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5299b90a6c40..51b3090dd072 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1109,6 +1109,17 @@ static inline bool is_zone_device_page(const struct page *page) } #endif +static inline bool is_zone_movable_page(const struct page *page) +{ + return page_zonenum(page) == ZONE_MOVABLE; +} + +/* MIGRATE_CMA and ZONE_MOVABLE do not allow pin pages */ +static inline bool is_pinnable_page(struct page *page) +{ + return !is_zone_movable_page(page) && !is_migrate_cma_page(page); +} + #ifdef CONFIG_DEV_PAGEMAP_OPS void free_devmap_managed_page(struct page *page); DECLARE_STATIC_KEY_FALSE(devmap_managed_key); diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 5f4dd3274734..a55277b0d475 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -150,12 +150,13 @@ static inline bool in_vfork(struct task_struct *tsk) * Applies per-task gfp context to the given allocation flags. * PF_MEMALLOC_NOIO implies GFP_NOIO * PF_MEMALLOC_NOFS implies GFP_NOFS + * PF_MEMALLOC_PIN implies !GFP_MOVABLE */ static inline gfp_t current_gfp_context(gfp_t flags) { unsigned int pflags = READ_ONCE(current->flags); - if (unlikely(pflags & (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS))) { + if (unlikely(pflags & (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_PIN))) { /* * NOIO implies both NOIO and NOFS and it is a weaker context * so always make sure it makes precedence @@ -164,6 +165,9 @@ static inline gfp_t current_gfp_context(gfp_t flags) flags &= ~(__GFP_IO | __GFP_FS); else if (pflags & PF_MEMALLOC_NOFS) flags &= ~__GFP_FS; + + if (pflags & PF_MEMALLOC_PIN) + flags &= ~__GFP_MOVABLE; } return flags; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3f5ddac5de8a..958ef274db82 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1036,7 +1036,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) bool pin = !!(current->flags & PF_MEMALLOC_PIN); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { - if (pin && is_migrate_cma_page(page)) + if (pin && !is_pinnable_page(page)) continue; if (PageHWPoison(page)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c2dea9ad0e98..f83fa8cf3bb8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3802,16 +3802,13 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) return alloc_flags; } -static inline unsigned int current_alloc_flags(gfp_t gfp_mask, - unsigned int alloc_flags) +/* Must be called after current_gfp_context() which can change gfp_mask */ +static inline unsigned int gpf_to_alloc_flags(gfp_t gfp_mask, + unsigned int alloc_flags) { #ifdef CONFIG_CMA - unsigned int pflags = current->flags; - - if (!(pflags & PF_MEMALLOC_PIN) && - gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; - #endif return alloc_flags; } @@ -4467,7 +4464,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask) } else if (unlikely(rt_task(current)) && !in_interrupt()) alloc_flags |= ALLOC_HARDER; - alloc_flags = current_alloc_flags(gfp_mask, alloc_flags); + alloc_flags = gpf_to_alloc_flags(gfp_mask, alloc_flags); return alloc_flags; } @@ -4769,7 +4766,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) - alloc_flags = current_alloc_flags(gfp_mask, reserve_flags); + alloc_flags = gpf_to_alloc_flags(gfp_mask, reserve_flags); /* * Reset the nodemask and zonelist iterators if memory policies can be @@ -4938,7 +4935,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, if (should_fail_alloc_page(gfp_mask, order)) return false; - *alloc_flags = current_alloc_flags(gfp_mask, *alloc_flags); + *alloc_flags = gpf_to_alloc_flags(gfp_mask, *alloc_flags); /* Dirty zone balancing only done in the fast path */ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE); @@ -4980,7 +4977,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, * Apply scoped allocation constraints. This is mainly about GFP_NOFS * resp. GFP_NOIO which has to be inherited for all allocation requests * from a particular context which has been marked by - * memalloc_no{fs,io}_{save,restore}. + * memalloc_no{fs,io}_{save,restore}. And PF_MEMALLOC_PIN which ensures + * movable zones are not used during allocation. */ gfp_mask = current_gfp_context(gfp_mask); alloc_mask = gfp_mask; From patchwork Thu Dec 17 18:52:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 945C2C3526C for ; Thu, 17 Dec 2020 18:54:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 733712399A for ; Thu, 17 Dec 2020 18:54:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731095AbgLQSyu (ORCPT ); Thu, 17 Dec 2020 13:54:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730040AbgLQSyu (ORCPT ); Thu, 17 Dec 2020 13:54:50 -0500 Received: from mail-qv1-xf31.google.com (mail-qv1-xf31.google.com [IPv6:2607:f8b0:4864:20::f31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0988CC0611E4 for ; Thu, 17 Dec 2020 10:52:57 -0800 (PST) Received: by mail-qv1-xf31.google.com with SMTP id l7so13804642qvt.4 for ; Thu, 17 Dec 2020 10:52:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=e6Ncu8c/WYeWoMyVk5R0b06plBGbLPujuz6OQkTIMmE=; b=eLX1jgZ7IuyDcsgovsbreDFQXA2MaJfYs3hIyomcThfex/wvVW3Z/nmVUoxDR5TyL0 t64YpRkFt3uba3f5F+NwIC0ERE+2Gw7ozN14LqGUNeHrLCMDnWSDA6YpTktNvpTF9K4g vkiaxpQsOTamjUV9fvIwE8wZJE3p+VMDVLxV4/kwJIJb6LaeSUoF5NhEsp/frRaHbNV0 9PGijeoxePmSMQQU0VTKb/ja5SAqjUMdxN06aeaRgeKE9Nx5C+i3HPJNvFub/pKVPzWs VFzdBSXFjFUL2BFh+DoPdA8+Ii1KvTE2a92B0fh+cVnglUzvugVbpqHty2bwO3YLaZzH 8D8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=e6Ncu8c/WYeWoMyVk5R0b06plBGbLPujuz6OQkTIMmE=; b=jhbhnPIKvOLI3+QWerHMhbLFiOCEkNaFMAHJL/QW3kpTehlR6YYiMzTuMtssOggb3T HNlupSli2hl3MTZ5S8isILewfRALL+A0N3f3j4+rR1gq+/R77jhW56OrFTUvOT9YE+C4 qYb1LLzSiRv6t2ol5PBCavgar6WMoOl1BxHHjmTYUOHW6KE++O/fx8w9Y5tNvwl+BGmf PxMpDLquCQAgxS68T3gUkcmH9/bJDnUJdllmvKlIwMZuMqKh21vN9sPYfy/cyH8+y4v2 c6ZwNDEwWTwLBUyy8fh/5hPX4h/sCSew1FbQgHq83rwJ4+5N5/Pzw/EFhl+CpVtQAdh3 D8UQ== X-Gm-Message-State: AOAM533y2IlyGuydWn7u5rWDS8LjDNak1y7eqQQZ20qUBpRaNsR9QN/b dFofKJxqpHRSOdQwGaJ+Q1m0gA== X-Google-Smtp-Source: ABdhPJwhKhgcl0hfKajMW2+KAzlYuNpe1Z7Ao4dCi2zUFsK6Bt5WiMeZZErAf8vczb45yFmrKcmNYg== X-Received: by 2002:a0c:a366:: with SMTP id u93mr626563qvu.53.1608231176250; Thu, 17 Dec 2020 10:52:56 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.52.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:52:55 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 05/10] mm/gup: migrate pinned pages out of movable zone Date: Thu, 17 Dec 2020 13:52:38 -0500 Message-Id: <20201217185243.3288048-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org We should not pin pages in ZONE_MOVABLE. Currently, we do not pin only movable CMA pages. Generalize the function that migrates CMA pages to migrate all movable pages. Use is_pinnable_page() to check which pages need to be migrated Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- include/linux/migrate.h | 1 + include/linux/mmzone.h | 11 ++++-- include/trace/events/migrate.h | 3 +- mm/gup.c | 68 ++++++++++++++-------------------- 4 files changed, 39 insertions(+), 44 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 4594838a0f7c..aae5ef0b3ba1 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -27,6 +27,7 @@ enum migrate_reason { MR_MEMPOLICY_MBIND, MR_NUMA_MISPLACED, MR_CONTIG_RANGE, + MR_LONGTERM_PIN, MR_TYPES }; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b593316bff3d..25c0c13ba4b1 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -386,9 +386,14 @@ enum zone_type { * likely to succeed, and to locally limit unmovable allocations - e.g., * to increase the number of THP/huge pages. Notable special cases are: * - * 1. Pinned pages: (long-term) pinning of movable pages might - * essentially turn such pages unmovable. Memory offlining might - * retry a long time. + * 1. Pinned pages: (long-term) pinning of movable pages is avoided + * when pages are pinned and faulted, but it is still possible that + * address space already has pages in ZONE_MOVABLE at the time when + * pages are pinned (i.e. user has touches that memory before + * pinning). In such case we try to migrate them to a different zone, + * but if migration fails the pages can still end-up pinned in + * ZONE_MOVABLE. In such case, memory offlining might retry a long + * time and will only succeed once user application unpins pages. * 2. memblock allocations: kernelcore/movablecore setups might create * situations where ZONE_MOVABLE contains unmovable allocations * after boot. Memory offlining and allocations fail early. diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h index 4d434398d64d..363b54ce104c 100644 --- a/include/trace/events/migrate.h +++ b/include/trace/events/migrate.h @@ -20,7 +20,8 @@ EM( MR_SYSCALL, "syscall_or_cpuset") \ EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind") \ EM( MR_NUMA_MISPLACED, "numa_misplaced") \ - EMe(MR_CONTIG_RANGE, "contig_range") + EM( MR_CONTIG_RANGE, "contig_range") \ + EMe(MR_LONGTERM_PIN, "longterm_pin") /* * First define the enums in the above macros to be exported to userspace diff --git a/mm/gup.c b/mm/gup.c index 04602e94856b..591d8e2dfc70 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -89,11 +89,12 @@ static __maybe_unused struct page *try_grab_compound_head(struct page *page, int orig_refs = refs; /* - * Can't do FOLL_LONGTERM + FOLL_PIN with CMA in the gup fast - * path, so fail and let the caller fall back to the slow path. + * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a + * right zone, so fail and let the caller fall back to the slow + * path. */ - if (unlikely(flags & FOLL_LONGTERM) && - is_migrate_cma_page(page)) + if (unlikely((flags & FOLL_LONGTERM) && + !is_pinnable_page(page))) return NULL; /* @@ -1549,19 +1550,18 @@ struct page *get_dump_page(unsigned long addr) } #endif /* CONFIG_ELF_CORE */ -#ifdef CONFIG_CMA -static long check_and_migrate_cma_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) +static long check_and_migrate_movable_pages(struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int gup_flags) { unsigned long i; unsigned long step; bool drain_allow = true; bool migrate_allow = true; - LIST_HEAD(cma_page_list); + LIST_HEAD(movable_page_list); long ret = nr_pages; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, @@ -1579,13 +1579,12 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, */ step = compound_nr(head) - (pages[i] - head); /* - * If we get a page from the CMA zone, since we are going to - * be pinning these entries, we might as well move them out - * of the CMA zone if possible. + * If we get a movable page, since we are going to be pinning + * these entries, try to move them out if possible. */ - if (is_migrate_cma_page(head)) { + if (!is_pinnable_page(head)) { if (PageHuge(head)) - isolate_huge_page(head, &cma_page_list); + isolate_huge_page(head, &movable_page_list); else { if (!PageLRU(head) && drain_allow) { lru_add_drain_all(); @@ -1593,7 +1592,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, } if (!isolate_lru_page(head)) { - list_add_tail(&head->lru, &cma_page_list); + list_add_tail(&head->lru, &movable_page_list); mod_node_page_state(page_pgdat(head), NR_ISOLATED_ANON + page_is_file_lru(head), @@ -1605,7 +1604,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, i += step; } - if (!list_empty(&cma_page_list)) { + if (!list_empty(&movable_page_list)) { /* * drop the above get_user_pages reference. */ @@ -1615,25 +1614,24 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, for (i = 0; i < nr_pages; i++) put_page(pages[i]); - if (migrate_pages(&cma_page_list, alloc_migration_target, NULL, - (unsigned long)&mtc, MIGRATE_SYNC, MR_CONTIG_RANGE)) { + if (migrate_pages(&movable_page_list, alloc_migration_target, NULL, + (unsigned long)&mtc, MIGRATE_SYNC, MR_LONGTERM_PIN)) { /* * some of the pages failed migration. Do get_user_pages * without migration. */ migrate_allow = false; - if (!list_empty(&cma_page_list)) - putback_movable_pages(&cma_page_list); + if (!list_empty(&movable_page_list)) + putback_movable_pages(&movable_page_list); } /* * We did migrate all the pages, Try to get the page references - * again migrating any new CMA pages which we failed to isolate - * earlier. + * again migrating any pages which we failed to isolate earlier. */ ret = __get_user_pages_locked(mm, start, nr_pages, - pages, vmas, NULL, - gup_flags); + pages, vmas, NULL, + gup_flags); if ((ret > 0) && migrate_allow) { nr_pages = ret; @@ -1644,17 +1642,6 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, return ret; } -#else -static long check_and_migrate_cma_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) -{ - return nr_pages; -} -#endif /* CONFIG_CMA */ /* * __gup_longterm_locked() is a wrapper for __get_user_pages_locked which @@ -1678,8 +1665,9 @@ static long __gup_longterm_locked(struct mm_struct *mm, if (gup_flags & FOLL_LONGTERM) { if (rc > 0) - rc = check_and_migrate_cma_pages(mm, start, rc, pages, - vmas, gup_flags); + rc = check_and_migrate_movable_pages(mm, start, rc, + pages, vmas, + gup_flags); memalloc_pin_restore(flags); } return rc; From patchwork Thu Dec 17 18:52:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4E27C3526C for ; Thu, 17 Dec 2020 18:54:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8DD65224D4 for ; Thu, 17 Dec 2020 18:54:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730877AbgLQSy0 (ORCPT ); Thu, 17 Dec 2020 13:54:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730216AbgLQSyZ (ORCPT ); Thu, 17 Dec 2020 13:54:25 -0500 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA869C0619D4 for ; Thu, 17 Dec 2020 10:52:58 -0800 (PST) Received: by mail-qt1-x82a.google.com with SMTP id g24so8428780qtq.12 for ; Thu, 17 Dec 2020 10:52:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UPZkOOUAvu9r4fFw8TJVvFyTq3W2R2uLYiLCZknh/L8=; b=KOOeZ9F311HelPB+dQ5Jxny5X6SxXyAGd3KKlieuI07e2hBrEt3RVU31ycAHwlVbHk 9xDQU/VkPBTvCiEIEwXb/bWvF/65bTrSG6bJ8OLDPC7HOLFiw2Zxhgxh1lQUOZstjQr5 nRUsykwhnsyiNEFIUNNRo7b7/kpMfRm7u0PmxNIPeHyL3cIYKFOq6O1LUn+U1qjvzFWv X5iFlzbIPzQ0QqGcu5+7fx6GsqYERnYEyi6OW2FLKNTSVDyW7ZVZTxK7klRDMMSvU/21 WRjhR/LtC6mtC0LKxI6t+b7bj0RvXcGvxCWK6d07QTYlzYVsAZs6bXx1yzp2DtAcp+N9 dzcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UPZkOOUAvu9r4fFw8TJVvFyTq3W2R2uLYiLCZknh/L8=; b=DbwqebtIvX7cj8/PpR1e9pXVLiLUXMXRkrt8pnZMTe3/t4vdJLhZTZbMTfPyqUO1I5 vHOB50UKRhg2ubB6JHmXn/SQs+Rz7qCsdpGwoOgX7hM5cb5AzHDd2ffXa1pBe0sKykEz +h5ncdgZKl499w9+h1iisZ4+fvhRsH8ivbdKhY59irjMcwOSp0FXPQyx87H65Kv3+9pk GQvakEn9giHqisSg6/3VGf4YkBpNHM4pcKVuK282b5dXWqHbUYl9Vz0MKRz5EfEaiyHv aDLRliHN7xcBuFCWCzayIFd7baSsGkF+D6Qtex5f4ioOpXHj9wvjm6Gol00ZuXjzdtQv /uUQ== X-Gm-Message-State: AOAM532eplhSSvLNSJREudYVY5pi6pg2l0XzJoLIoyWrOvfb7yAyPqNi UGHDmUL34r5siv0+1fLdR4h+XA== X-Google-Smtp-Source: ABdhPJyEyx1AuPMr4F/EQqqkrPpGXKufuGMVlxL61qVfUQnEvSqm6U4FlnMLVskQ6cx1G2kJJGaoEQ== X-Received: by 2002:ac8:108c:: with SMTP id a12mr121263qtj.275.1608231178215; Thu, 17 Dec 2020 10:52:58 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.52.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:52:57 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 06/10] memory-hotplug.rst: add a note about ZONE_MOVABLE and page pinning Date: Thu, 17 Dec 2020 13:52:39 -0500 Message-Id: <20201217185243.3288048-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Document the special handling of page pinning when ZONE_MOVABLE present. Signed-off-by: Pavel Tatashin Suggested-by: David Hildenbrand Acked-by: Michal Hocko --- Documentation/admin-guide/mm/memory-hotplug.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst index 5c4432c96c4b..c6618f99f765 100644 --- a/Documentation/admin-guide/mm/memory-hotplug.rst +++ b/Documentation/admin-guide/mm/memory-hotplug.rst @@ -357,6 +357,15 @@ creates ZONE_MOVABLE as following. Unfortunately, there is no information to show which memory block belongs to ZONE_MOVABLE. This is TBD. +.. note:: + Techniques that rely on long-term pinnings of memory (especially, RDMA and + vfio) are fundamentally problematic with ZONE_MOVABLE and, therefore, memory + hot remove. Pinned pages cannot reside on ZONE_MOVABLE, to guarantee that + memory can still get hot removed - be aware that pinning can fail even if + there is plenty of free memory in ZONE_MOVABLE. In addition, using + ZONE_MOVABLE might make page pinning more expensive, because pages have to be + migrated off that zone first. + .. _memory_hotplug_how_to_offline_memory: How to offline memory From patchwork Thu Dec 17 18:52:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA32EC2BBCA for ; Thu, 17 Dec 2020 18:54:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A87C623715 for ; Thu, 17 Dec 2020 18:54:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730885AbgLQSy0 (ORCPT ); Thu, 17 Dec 2020 13:54:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730872AbgLQSyZ (ORCPT ); Thu, 17 Dec 2020 13:54:25 -0500 Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E5A5C0619D8 for ; Thu, 17 Dec 2020 10:53:00 -0800 (PST) Received: by mail-qt1-x833.google.com with SMTP id a6so20842975qtw.6 for ; Thu, 17 Dec 2020 10:53:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Q+nZQmLz+g6ZtD9UXQKwPL3A91wbKfX5nZj6XsBF5oE=; b=JO8qsADGPR4ZcKb3Gq+FKSW5aYA37eOaMrDgEiIcr5jzVShlVNKWtFcFGT0Tx3cMaS wJK8H4/qkQAa91lkrvvrvVhvqvRTRrc0cXVG9prRf5eIUUOYkeXykED77uULERb3qi2S aKhixRDGpTBTfOPxg4C7w3zrkZQYLiR0+RfMBR9kTP4vmjFrTmFN0H68z77PVURftvfT 6zQWCURQR4W0gXHv3MUmiWL1LHTDZS/ikTWgkKkSVoypuey03I/hMAH2KRRKKfVV10dc iOwzTWGWVjztOR2Vx36yAmOnyLwBGqiXBWiq0B7YE4kuUgcKm865e7wWGRPqfZEu1nu8 aVxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q+nZQmLz+g6ZtD9UXQKwPL3A91wbKfX5nZj6XsBF5oE=; b=LxuneHaYHnBEV07Y6oRCfJg7j31h6l6RnT9PyC624yaQYXuTyGSemI0tlfj0ZF/CQP HLWwP1tXwzFzjerOkBEaSPVZr7vchhLdwDx+mce+0Fe0CyMRqiX3Ai61dY4dpDJ8p0Cm DP/f9BV+z3Vb1URdpfwyPHeYB5FSoOm/Ov8C15iKgaHpj9TVXCWQ+wFhbGv5f1lXp163 scv6A5cFNLuhPMYpjPLTTYmla1I3vnLm90DjX7lXQAaVWKA5fWvIaosNjvkSOcpVHHgf nn42r8twKfg1Hwl1oZYNcYiFtYqw+yXvjCs1ONClid8Pr/OU5PUxU01F+LedeuVYvCZY mCHA== X-Gm-Message-State: AOAM532zeDRYA8JMSWrfMPviZaVSwfRHLranVbujBrT7KlbT7I1/hgYO ZzJ9XCIMPVWARAUp3MF+H42MKg== X-Google-Smtp-Source: ABdhPJyZTZTnodRThEF3jTIBRJYgZGD6V+Z5lINIuUMU1uZ/VPO4vM0YHNy5VdMEckiPuqDN6NjqBA== X-Received: by 2002:aed:33a1:: with SMTP id v30mr196167qtd.50.1608231179869; Thu, 17 Dec 2020 10:52:59 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.52.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:52:59 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 07/10] mm/gup: change index type to long as it counts pages Date: Thu, 17 Dec 2020 13:52:40 -0500 Message-Id: <20201217185243.3288048-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org In __get_user_pages_locked() i counts number of pages which should be long. Signed-off-by: Pavel Tatashin Acked-by: Michal Hocko --- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index 591d8e2dfc70..1ebb7cc2fbe4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1481,7 +1481,7 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, { struct vm_area_struct *vma; unsigned long vm_flags; - int i; + long i; /* calculate required read or write permissions. * If FOLL_FORCE is set, we only require the "MAY" flags. From patchwork Thu Dec 17 18:52:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB1CDC3526D for ; Thu, 17 Dec 2020 18:54:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B05832399A for ; Thu, 17 Dec 2020 18:54:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731142AbgLQSyv (ORCPT ); Thu, 17 Dec 2020 13:54:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731128AbgLQSyu (ORCPT ); Thu, 17 Dec 2020 13:54:50 -0500 Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61140C0619D9 for ; Thu, 17 Dec 2020 10:53:02 -0800 (PST) Received: by mail-qk1-x732.google.com with SMTP id 19so27384493qkm.8 for ; Thu, 17 Dec 2020 10:53:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ddKWF6rz+jusoSW85/8ZXU7qwdgQHa0ysIZMt3vGuRI=; b=Y+S8xlyU6M5Ma07Mnlbu0DPfFn/l4z5e0Vd6UE2ZGYZEAAz2VUjCv85uIPBF+ol81+ x6jfcFfQXXfM7DS+Bi1SZQakCCvVq2axWBhBi5ojyTi/cRQTb6GD/pUVhV/vrnQcNmRa etvfj8gKeVup6AH3odppjx3qV8QIUj0njVsO0LCaHIzs8e4cpZjqpAs7q3EkhxdjKIFU +s2WxTbDpBRPNe2G67zpaxHjz519rXrhaV2TjNjuSBqGlt6oo06S/MB4nD3NLz8njfbS q5pG0fnAWGtmTtakVPZ/oEhADYpxQfD49mZ+SrYC19Vy0j5KIogPu+5mO4pJSH8DdOcd iRYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ddKWF6rz+jusoSW85/8ZXU7qwdgQHa0ysIZMt3vGuRI=; b=ChtYnfRo3WNnoPYKc2qpJ2ULQHqzs3VFweQNQaSrhYwBoICj7eAfic+TTYOHD0Ogph eei87GnMJbG6Q+INpmBudW2AF17zL8yvUSMo6+bRsXR1TozIqMaXZ0D9SQ7Cz3cfLx6X 1Nz4AFBOsIIDXDKFJb9p3+nfZLhuygnz/oanV5Ac3ZLEvkckW5Z/Wa2VyNTtU8ZRhthw vBicXlWEPW+1NFniyt2xf7Rnzk8P3AmkBgHwsR3y430pwMrBb7TB9WrFiIJekbn70HmC b//hLaKirZqH9FC+GvmE44uYSFEUUeUycH5v/13gwNS2qnN0BTrHXoUwLX2ZR3U3DuKN tAgg== X-Gm-Message-State: AOAM532yKM2syhqQAqIIxZoADPDOBOIfMFbWLMUpXqLvn6EarWA2kR98 5+IQCb6LbjvsWLXT2WreqKloUA== X-Google-Smtp-Source: ABdhPJxGcmSkWGboMlUiUt5OdTP7H/VwwMXdIJqjPcipfMa9wI/JO+uBMKdmiESr7baRF4zUa8CYMQ== X-Received: by 2002:a05:620a:75a:: with SMTP id i26mr730856qki.314.1608231181586; Thu, 17 Dec 2020 10:53:01 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.52.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:53:00 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 08/10] mm/gup: limit number of gup migration failures, honor failures Date: Thu, 17 Dec 2020 13:52:41 -0500 Message-Id: <20201217185243.3288048-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org check_and_migrate_movable_pages() does not honor isolation errors, and also retries migration failures indefinably. Fix both of the above issues: add a new function that checks and unpins pages range check_and_unpin_pages(). Move the retry loop from check_and_migrate_movable_pages() to __gup_longterm_locked(). Rename check_and_migrate_movable_pages() as migrate_movable_pages() and make this function accept already unpinned pages. Also, track the errors during isolation, so they can be re-tried with a different maximum limit, the isolation errors should be ephemeral. Signed-off-by: Pavel Tatashin --- mm/gup.c | 179 ++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 111 insertions(+), 68 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1ebb7cc2fbe4..70cc8b8f67c4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1550,27 +1550,57 @@ struct page *get_dump_page(unsigned long addr) } #endif /* CONFIG_ELF_CORE */ -static long check_and_migrate_movable_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) -{ - unsigned long i; - unsigned long step; - bool drain_allow = true; - bool migrate_allow = true; +/* + * Verify that there are no unpinnable (movable) pages, if so return true. + * Otherwise an unpinnable pages is found return false, and unpin all pages. + */ +static bool check_and_unpin_pages(unsigned long nr_pages, struct page **pages, + unsigned int gup_flags) +{ + unsigned long i, step; + + for (i = 0; i < nr_pages; i += step) { + struct page *head = compound_head(pages[i]); + + step = compound_nr(head) - (pages[i] - head); + if (!is_pinnable_page(head)) + break; + } + + if (i >= nr_pages) + return true; + + if (gup_flags & FOLL_PIN) { + unpin_user_pages(pages, nr_pages); + } else { + for (i = 0; i < nr_pages; i++) + put_page(pages[i]); + } + + return false; +} + +#define PINNABLE_MIGRATE_MAX 10 +#define PINNABLE_ISOLATE_MAX 100 + +/* + * Migrate pages that cannot be pinned. Return zero on success and error code + * on migration failure. If migration was successful but page isolation had + * failures return number of pages that failed to be isolated. + */ +static long migrate_movable_pages(unsigned long nr_pages, struct page **pages) +{ + unsigned long i, step; LIST_HEAD(movable_page_list); - long ret = nr_pages; + long ret = 0; + long error_count = 0; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, .gfp_mask = GFP_USER | __GFP_NOWARN, }; -check_again: - for (i = 0; i < nr_pages;) { - + lru_add_drain_all(); + for (i = 0; i < nr_pages; i += step) { struct page *head = compound_head(pages[i]); /* @@ -1583,62 +1613,42 @@ static long check_and_migrate_movable_pages(struct mm_struct *mm, * these entries, try to move them out if possible. */ if (!is_pinnable_page(head)) { - if (PageHuge(head)) - isolate_huge_page(head, &movable_page_list); - else { - if (!PageLRU(head) && drain_allow) { - lru_add_drain_all(); - drain_allow = false; - } - + if (PageHuge(head)) { + if (!isolate_huge_page(head, &movable_page_list)) + error_count += step; + } else { if (!isolate_lru_page(head)) { list_add_tail(&head->lru, &movable_page_list); mod_node_page_state(page_pgdat(head), NR_ISOLATED_ANON + page_is_file_lru(head), thp_nr_pages(head)); + } else { + error_count += step; } } } - - i += step; } if (!list_empty(&movable_page_list)) { - /* - * drop the above get_user_pages reference. - */ - if (gup_flags & FOLL_PIN) - unpin_user_pages(pages, nr_pages); - else - for (i = 0; i < nr_pages; i++) - put_page(pages[i]); + ret = migrate_pages(&movable_page_list, alloc_migration_target, + NULL, (unsigned long)&mtc, MIGRATE_SYNC, + MR_LONGTERM_PIN); + /* Assume -EBUSY failure if some pages were not migrated */ + if (ret > 0) + ret = -EBUSY; + } - if (migrate_pages(&movable_page_list, alloc_migration_target, NULL, - (unsigned long)&mtc, MIGRATE_SYNC, MR_LONGTERM_PIN)) { - /* - * some of the pages failed migration. Do get_user_pages - * without migration. - */ - migrate_allow = false; + if (ret && !list_empty(&movable_page_list)) + putback_movable_pages(&movable_page_list); - if (!list_empty(&movable_page_list)) - putback_movable_pages(&movable_page_list); - } - /* - * We did migrate all the pages, Try to get the page references - * again migrating any pages which we failed to isolate earlier. - */ - ret = __get_user_pages_locked(mm, start, nr_pages, - pages, vmas, NULL, - gup_flags); - - if ((ret > 0) && migrate_allow) { - nr_pages = ret; - drain_allow = true; - goto check_again; - } - } + /* + * Check if there were isolation errors, if so they should not be + * counted toward PINNABLE_MIGRATE_MAX, so separate them, by + * returning number of pages failed to isolate. + */ + if (!ret && error_count) + ret = error_count; return ret; } @@ -1654,22 +1664,55 @@ static long __gup_longterm_locked(struct mm_struct *mm, struct vm_area_struct **vmas, unsigned int gup_flags) { - unsigned long flags = 0; + int migrate_retry = 0; + int isolate_retry = 0; + unsigned int flags; long rc; - if (gup_flags & FOLL_LONGTERM) - flags = memalloc_pin_save(); + if (!(gup_flags & FOLL_LONGTERM)) + return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, + NULL, gup_flags); - rc = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, NULL, - gup_flags); + /* + * Without FOLL_WRITE fault handler may return zero page, which can + * be in a movable zone, and also will fail to isolate during migration, + * thus the longterm pin will fail. + */ + gup_flags &= FOLL_WRITE; - if (gup_flags & FOLL_LONGTERM) { - if (rc > 0) - rc = check_and_migrate_movable_pages(mm, start, rc, - pages, vmas, - gup_flags); - memalloc_pin_restore(flags); + flags = memalloc_pin_save(); + /* + * Migration may fail, we retry before giving up. Also, because after + * migration pages[] becomes outdated, we unpin and repin all pages + * in the range, so pages array is repopulated with new values. + * Also, because of this we cannot retry migration failures in a loop + * without pinning/unpinnig pages. + */ + for (; ; ) { + rc = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, + NULL, gup_flags); + + /* Return if error or if all pages are pinnable */ + if (rc <= 0 || check_and_unpin_pages(rc, pages, gup_flags)) + break; + + /* Some pages are not pinnable, migrate them */ + rc = migrate_movable_pages(rc, pages); + + /* + * If there is an error, and we tried maximum number of times + * bail out. Notice: we return an error code, and all pages are + * unpinned + */ + if (rc < 0 && migrate_retry++ >= PINNABLE_MIGRATE_MAX) { + break; + } else if (rc > 0 && isolate_retry++ >= PINNABLE_ISOLATE_MAX) { + rc = -EBUSY; + break; + } } + memalloc_pin_restore(flags); + return rc; } From patchwork Thu Dec 17 18:52:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A010FC2BB9A for ; Thu, 17 Dec 2020 18:54:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7670323715 for ; Thu, 17 Dec 2020 18:54:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730915AbgLQSy0 (ORCPT ); Thu, 17 Dec 2020 13:54:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730876AbgLQSyZ (ORCPT ); Thu, 17 Dec 2020 13:54:25 -0500 Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EAE6C0619DD for ; Thu, 17 Dec 2020 10:53:04 -0800 (PST) Received: by mail-qk1-x72c.google.com with SMTP id 19so27384602qkm.8 for ; Thu, 17 Dec 2020 10:53:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=COwDysI4bZZEKCXTQQYU3uK6Y/IsqJ6hL7YDBk3cg/c=; b=i7KuMhOyFcM+TyMj/X77dnuv0gWjob2WfM13WrvtO33cVvoh/4vOeACCtT/GpbiUNL WO3cYvr/PjFoEbREscOUN33YZwSNmecJNT1hQZEwCrnE+LoCOsLd3GzPR/ekMxvxcxfe kqt1qmz1bAXeIiqNlba5Jr0x++uAYRO12MC05X/E7DusvggUAxCZl7bmMdqGIH/EBYjy qOeYnM9cHS2yK+r/tRFdPdSo1tUxlF7n+EwzXOQ36HGnec+7ByAT1kdBfWxH22M2MY+v gfKGdjmZc4u1KqDuHwGN3NNZuUWdXLv/RbdNcWxp0Y9dsQDyPupyuS0RWe1BDfl/xiB5 m9Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=COwDysI4bZZEKCXTQQYU3uK6Y/IsqJ6hL7YDBk3cg/c=; b=flZbtGq9PbwszuAIMb2xodMuCXckVWCVtF0rWbL+W6HPw1ngOdECAip6bfGGofJw6s 5Qe5P7ogWw9/YJjOzWzBAXCvVRJPxlCojAjz+HPw4T3C6F2N9/HIkArdnsyrNuqqHh7D vru6gdO8e7EZE5mlXG5/5U8ng6Drte5C8hb7VAgFZPxd+r+kTFFPXBVA/L3lHE8OzdM/ /8UZob7qYhwL7CkLR5aZClEJLgKAxe9+brkyXQAfGl9qdGN5wUwewiW0CC6/E/xxcpMW 6zyhBriD5yu3QDivsC/YhdlqZDbleQDxFcc2WyYxBPXwfEL1s9JTJW3uYlpHxzJRDBS/ EI2w== X-Gm-Message-State: AOAM530qCIVrifuI73ADFysSlb0wvy8OGOUlIgJo3LauPoVKO++ZrbLf GU0ifpSxIr3OAVtMWe1KXSPf3A== X-Google-Smtp-Source: ABdhPJzWxK7UMeCHrIoSiUT4ebH815ATLU90gFdXWdc76eFy1XhzXtvmWIRlIeGJ1QgQ1s2bBj8tEg== X-Received: by 2002:a37:a241:: with SMTP id l62mr691371qke.482.1608231183357; Thu, 17 Dec 2020 10:53:03 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.53.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:53:02 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 09/10] selftests/vm: test flag is broken Date: Thu, 17 Dec 2020 13:52:42 -0500 Message-Id: <20201217185243.3288048-10-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org In gup_test both gup_flags and test_flags use the same flags field. This is broken, because gup_flags can be passed as raw value (via -F hex), which can overwrite all the test flags. Farther, in the actual gup_test.c all the passed gup_flags are erased and unconditionally replaced with FOLL_WRITE. Which means that test_flags are ignored, and code like this always performs pin dump test: 155 if (gup->flags & GUP_TEST_FLAG_DUMP_PAGES_USE_PIN) 156 nr = pin_user_pages(addr, nr, gup->flags, 157 pages + i, NULL); 158 else 159 nr = get_user_pages(addr, nr, gup->flags, 160 pages + i, NULL); 161 break; Add a new test_flags field, to allow raw gup_flags to work. Add a new subcommand for DUMP_USER_PAGES_TEST to specify that pin test should be performed. Remove unconditional overwriting of gup_flags via FOLL_WRITE. But, preserve the previous behaviour where FOLL_WRITE was the default flag, and add a new option "-W" to unset FOLL_WRITE. With the fix, dump works like this: root@virtme:/# gup_test -c ---- page #0, starting from user virt addr: 0x7f8acb9e4000 page:00000000d3d2ee27 refcount:2 mapcount:1 mapping:0000000000000000 index:0x0 pfn:0x100bcf anon flags: 0x300000000080016(referenced|uptodate|lru|swapbacked) raw: 0300000000080016 ffffd0e204021608 ffffd0e208df2e88 ffff8ea04243ec61 raw: 0000000000000000 0000000000000000 0000000200000000 0000000000000000 page dumped because: gup_test: dump_pages() test DUMP_USER_PAGES_TEST: done root@virtme:/# gup_test -c -p ---- page #0, starting from user virt addr: 0x7fd19701b000 page:00000000baed3c7d refcount:1025 mapcount:1 mapping:0000000000000000 index:0x0 pfn:0x108008 anon flags: 0x300000000080014(uptodate|lru|swapbacked) raw: 0300000000080014 ffffd0e204200188 ffffd0e205e09088 ffff8ea04243ee71 raw: 0000000000000000 0000000000000000 0000040100000000 0000000000000000 page dumped because: gup_test: dump_pages() test DUMP_USER_PAGES_TEST: done Refcount shows the difference between pin vs no-pin case. Also change type of nr from int to long, as it counts number of pages. Signed-off-by: Pavel Tatashin --- mm/gup_test.c | 9 +++------ mm/gup_test.h | 1 + tools/testing/selftests/vm/gup_test.c | 11 +++++++++-- 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/mm/gup_test.c b/mm/gup_test.c index e3cf78e5873e..24c70c5814ba 100644 --- a/mm/gup_test.c +++ b/mm/gup_test.c @@ -94,7 +94,7 @@ static int __gup_test_ioctl(unsigned int cmd, { ktime_t start_time, end_time; unsigned long i, nr_pages, addr, next; - int nr; + long nr; struct page **pages; int ret = 0; bool needs_mmap_lock = @@ -126,9 +126,6 @@ static int __gup_test_ioctl(unsigned int cmd, nr = (next - addr) / PAGE_SIZE; } - /* Filter out most gup flags: only allow a tiny subset here: */ - gup->flags &= FOLL_WRITE; - switch (cmd) { case GUP_FAST_BENCHMARK: nr = get_user_pages_fast(addr, nr, gup->flags, @@ -152,7 +149,7 @@ static int __gup_test_ioctl(unsigned int cmd, pages + i, NULL); break; case DUMP_USER_PAGES_TEST: - if (gup->flags & GUP_TEST_FLAG_DUMP_PAGES_USE_PIN) + if (gup->test_flags & GUP_TEST_FLAG_DUMP_PAGES_USE_PIN) nr = pin_user_pages(addr, nr, gup->flags, pages + i, NULL); else @@ -187,7 +184,7 @@ static int __gup_test_ioctl(unsigned int cmd, start_time = ktime_get(); - put_back_pages(cmd, pages, nr_pages, gup->flags); + put_back_pages(cmd, pages, nr_pages, gup->test_flags); end_time = ktime_get(); gup->put_delta_usec = ktime_us_delta(end_time, start_time); diff --git a/mm/gup_test.h b/mm/gup_test.h index 90a6713d50eb..b1b376b5d3b2 100644 --- a/mm/gup_test.h +++ b/mm/gup_test.h @@ -22,6 +22,7 @@ struct gup_test { __u64 size; __u32 nr_pages_per_call; __u32 flags; + __u32 test_flags; /* * Each non-zero entry is the number of the page (1-based: first page is * page 1, so that zero entries mean "do nothing") from the .addr base. diff --git a/tools/testing/selftests/vm/gup_test.c b/tools/testing/selftests/vm/gup_test.c index 6c6336dd3b7f..42c71483729f 100644 --- a/tools/testing/selftests/vm/gup_test.c +++ b/tools/testing/selftests/vm/gup_test.c @@ -37,13 +37,13 @@ int main(int argc, char **argv) { struct gup_test gup = { 0 }; unsigned long size = 128 * MB; - int i, fd, filed, opt, nr_pages = 1, thp = -1, repeats = 1, write = 0; + int i, fd, filed, opt, nr_pages = 1, thp = -1, repeats = 1, write = 1; unsigned long cmd = GUP_FAST_BENCHMARK; int flags = MAP_PRIVATE; char *file = "/dev/zero"; char *p; - while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwSH")) != -1) { + while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwWSHp")) != -1) { switch (opt) { case 'a': cmd = PIN_FAST_BENCHMARK; @@ -65,6 +65,10 @@ int main(int argc, char **argv) */ gup.which_pages[0] = 1; break; + case 'p': + /* works only with DUMP_USER_PAGES_TEST */ + gup.test_flags |= GUP_TEST_FLAG_DUMP_PAGES_USE_PIN; + break; case 'F': /* strtol, so you can pass flags in hex form */ gup.flags = strtol(optarg, 0, 0); @@ -93,6 +97,9 @@ int main(int argc, char **argv) case 'w': write = 1; break; + case 'W': + write = 0; + break; case 'f': file = optarg; break; From patchwork Thu Dec 17 18:52:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11980669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F90FC2BBCA for ; Thu, 17 Dec 2020 18:55:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 64081239D0 for ; Thu, 17 Dec 2020 18:55:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729693AbgLQSzE (ORCPT ); Thu, 17 Dec 2020 13:55:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730321AbgLQSyu (ORCPT ); Thu, 17 Dec 2020 13:54:50 -0500 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4E7EC0619DF for ; Thu, 17 Dec 2020 10:53:05 -0800 (PST) Received: by mail-qt1-x82a.google.com with SMTP id v5so11596149qtv.7 for ; Thu, 17 Dec 2020 10:53:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=fRwryvuVhXYrAhLWdCvlg8zqrU0LccdvvXixZFgTlpI=; b=gDva44QEcG/VWunZgNqEGGjC6m5HKRd5jC3iu5YSaDSrrwASQW3uz9D7uUgtQbFZS9 YN1YGfbBQ27Me0nqIB5dGg6H090zCpDkS2icZOHxNb/089r2kSUj+OVuhMTICDAYNLZT HtRlXXdMCv904IqORReSjOaIMQk32q7XdGOEXHUvYw15a7J2dm++e/DVF6j7O6nvR7DY rVzv4TB6dwm1dsjL4NUpH5/QlV9WBOeJOrkHBIuU/H4vVoDTO80obrJk96W54xN8KSgf g6/DPummsvSPaE4NAtii+AUigVRv2OtUpjv2GFzonPdM+AdN8bG3gZYeRFoX9WxZ3SM9 stxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fRwryvuVhXYrAhLWdCvlg8zqrU0LccdvvXixZFgTlpI=; b=S7LF/e5LQPEMxl0qStx8fR6VuPXr7CgwX95BbatZ0CCUwo/9IlFIMhUTqK/hU9IUzS AVdk3M19jktEjo0VBadSbOvEiVg4k4PLFMPks/9J212FOFdnGG8n9W1LkFDSak02ne8W LF+k84/qfJxRSGqg5wTgnckkp6g/H6ml/d1mR/OLWky0sVwVZYVwfqEA3sep9RZjELGl gMxywBUoGPFzsfTEYX08Z1XR8tpUKfwVtWs5P6Nwlj5d8O/DGmfzi545VHV8YIDsQLVv WMg0xn+8SJh4aq7hjFuF3r/MZgGGZIwzcrkbrrwZJ/5o4Y5AixTojX/SlePW/fuSDcxj oBKA== X-Gm-Message-State: AOAM531bEyUyVSkHdR/gfItlo3Ggm4phkmgqbaynIV8ARqG0h1dvbiHG D0cQmgU279jsNGeugZtB+SNXCA== X-Google-Smtp-Source: ABdhPJwU+kKYKSTYax2pFkwKEkMqmzxAc7GK//7UyTNHIEsyN/EH4UN0SEvfnPr8PHHMvDmMtUAQVg== X-Received: by 2002:ac8:730d:: with SMTP id x13mr171787qto.162.1608231185009; Thu, 17 Dec 2020 10:53:05 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m8sm4127434qkn.41.2020.12.17.10.53.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Dec 2020 10:53:04 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org Subject: [PATCH v4 10/10] selftests/vm: test faulting in kernel, and verify pinnable pages Date: Thu, 17 Dec 2020 13:52:43 -0500 Message-Id: <20201217185243.3288048-11-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201217185243.3288048-1-pasha.tatashin@soleen.com> References: <20201217185243.3288048-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org When pages are pinned they can be faulted in userland and migrated, and they can be faulted right in kernel without migration. In either case, the pinned pages must end-up being pinnable (not movable). Add a new test without touching pages in userland, and use FOLL_TOUCH instead. Also, verify that pinned pages are pinnable. Signed-off-by: Pavel Tatashin --- mm/gup_test.c | 6 ++++++ tools/testing/selftests/vm/gup_test.c | 17 +++++++++++++---- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/mm/gup_test.c b/mm/gup_test.c index 24c70c5814ba..24fd542091ee 100644 --- a/mm/gup_test.c +++ b/mm/gup_test.c @@ -52,6 +52,12 @@ static void verify_dma_pinned(unsigned int cmd, struct page **pages, dump_page(page, "gup_test failure"); break; + } else if (cmd == PIN_LONGTERM_BENCHMARK && + WARN(!is_pinnable_page(page), + "pages[%lu] is NOT pinnable but pinned\n", + i)) { + dump_page(page, "gup_test failure"); + break; } } break; diff --git a/tools/testing/selftests/vm/gup_test.c b/tools/testing/selftests/vm/gup_test.c index 42c71483729f..f08cc97d424d 100644 --- a/tools/testing/selftests/vm/gup_test.c +++ b/tools/testing/selftests/vm/gup_test.c @@ -13,6 +13,7 @@ /* Just the flags we need, copied from mm.h: */ #define FOLL_WRITE 0x01 /* check pte is writable */ +#define FOLL_TOUCH 0x02 /* mark page accessed */ static char *cmd_to_str(unsigned long cmd) { @@ -39,11 +40,11 @@ int main(int argc, char **argv) unsigned long size = 128 * MB; int i, fd, filed, opt, nr_pages = 1, thp = -1, repeats = 1, write = 1; unsigned long cmd = GUP_FAST_BENCHMARK; - int flags = MAP_PRIVATE; + int flags = MAP_PRIVATE, touch = 0; char *file = "/dev/zero"; char *p; - while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwWSHp")) != -1) { + while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwWSHpz")) != -1) { switch (opt) { case 'a': cmd = PIN_FAST_BENCHMARK; @@ -110,6 +111,10 @@ int main(int argc, char **argv) case 'H': flags |= (MAP_HUGETLB | MAP_ANONYMOUS); break; + case 'z': + /* fault pages in gup, do not fault in userland */ + touch = 1; + break; default: return -1; } @@ -167,8 +172,12 @@ int main(int argc, char **argv) else if (thp == 0) madvise(p, size, MADV_NOHUGEPAGE); - for (; (unsigned long)p < gup.addr + size; p += PAGE_SIZE) - p[0] = 0; + if (touch) { + gup.flags |= FOLL_TOUCH; + } else { + for (; (unsigned long)p < gup.addr + size; p += PAGE_SIZE) + p[0] = 0; + } /* Only report timing information on the *_BENCHMARK commands: */ if ((cmd == PIN_FAST_BENCHMARK) || (cmd == GUP_FAST_BENCHMARK) ||