From patchwork Wed Jul 7 18:26:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 12363465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CE00C07E95 for ; Wed, 7 Jul 2021 18:32:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1ACB861CBE for ; Wed, 7 Jul 2021 18:32:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1ACB861CBE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 048426B0078; Wed, 7 Jul 2021 14:32:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F39EE6B007B; Wed, 7 Jul 2021 14:32:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB47A6B007D; Wed, 7 Jul 2021 14:32:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0075.hostedemail.com [216.40.44.75]) by kanga.kvack.org (Postfix) with ESMTP id B9A906B0078 for ; Wed, 7 Jul 2021 14:32:11 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 29F091829B8F3 for ; Wed, 7 Jul 2021 18:32:11 +0000 (UTC) X-FDA: 78336636462.38.AC00332 Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) by imf26.hostedemail.com (Postfix) with ESMTP id 2FAF420019D4 for ; Wed, 7 Jul 2021 18:32:06 +0000 (UTC) Received: by mail-ed1-f52.google.com with SMTP id ch27so4648046edb.10 for ; Wed, 07 Jul 2021 11:32:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=X9TBuegD3B8qucF5fQVx/8HA6uQKRVlnWctSN0STdHk=; b=JHEoDNgJuf5pJ2G2aUS03u3QUdpMCJqs9w8aNqoS6YXXnRGc5XhbrWkq9jMLotAm7R MMIDxiThgDZch9j27k6rvk+khZIPCemFQhs5ih2dPYdSwp0KAdATaOFNfT8RF+OEZ8qp 2wBGe9lfLFnzzU8t0XjAm5CeHgru39N+7umFWE4iGmzYkWsbYy87npM+N2FSlzBEDRIz 6VzE12KUiSnA4FWj3q0FOh6oqoDm+r4erlymqF8pBGHoO20xi9SQD4Hziw81o/ATLmud ASX+zrIxXWyF7SZL1v+sBs1WQv21VZPsYjOEf29M6s1B2erCxlYxo7tPChw2hvom0I+2 LqkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=X9TBuegD3B8qucF5fQVx/8HA6uQKRVlnWctSN0STdHk=; b=IGu/kxuCeVvnDABiCTM0VFnoMOWsTOVDkVzQpauWCxZ5vbXaouh3FcCX9pW3eYxJte rmeZARW1CyRuLRVO1PElmGZRo2ltaIefpV/uwgs2fxFkgqE2Ywo0n9uojaxsmxLxbv1N JtCQFKLPFbm+7dcIeQiYmuDfno5Aao5Z4L/oLYY6dSjDA7FNEeR4fBPHNjjaDrNE6/4B V1+Fmz+2NlFM3+z8aYGf/KeBx5T8kp63WNX0eZ9sediLO/s8Na5NB961Z8ihZpT/fxKm wXWaFG9r7T10F/aL+rmVrpqTm0mv04MPaWmMEz426TYSMhwE4cBFFr/KmOgO3KFUvmsW TNnA== X-Gm-Message-State: AOAM532sP31rZ+g1gDXeo8RCHxBmzcJqZ3BFdCtYGbKjDDqyeLiDJhi3 CROIl3SfLYY1JZA0leAAwyKc3fIPGHKJAw== X-Google-Smtp-Source: ABdhPJwUMf4mL5nr2U320liK2/RWgXfPEJwTir2Q26O50gs74+t7zhK6gprreeCiE8yAV57csSuDHg== X-Received: by 2002:a2e:3505:: with SMTP id z5mr20238108ljz.308.1625682409508; Wed, 07 Jul 2021 11:26:49 -0700 (PDT) Received: from pc638.lan (h5ef52e3d.seluork.dyn.perspektivbredband.net. [94.245.46.61]) by smtp.gmail.com with ESMTPSA id o15sm1767963lfu.134.2021.07.07.11.26.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Jul 2021 11:26:49 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Mel Gorman , Christoph Hellwig , Matthew Wilcox , Nicholas Piggin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Oleksiy Avramchenko , Steven Rostedt Subject: [PATCH v3 1/2] mm/vmalloc: Use batched page requests in bulk-allocator Date: Wed, 7 Jul 2021 20:26:38 +0200 Message-Id: <20210707182639.31282-1-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Rspam-User: nil Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=JHEoDNgJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.52 as permitted sender) smtp.mailfrom=urezki@gmail.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2FAF420019D4 X-Stat-Signature: c6g3tx7oc7fma93nnnxxzps7ombtmm3y X-HE-Tag: 1625682726-399038 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In case of simultaneous vmalloc allocations, for example it is 1GB and 12 CPUs my system is able to hit "BUG: soft lockup" for !CONFIG_PREEMPT kernel. [ 62.512621] RIP: 0010:__alloc_pages_bulk+0xa9f/0xbb0 [ 62.512628] Code: ff 8b 44 24 48 44 29 f8 83 f8 01 0f 84 ea fe ff ff e9 07 f6 ff ff 48 8b 44 24 60 48 89 28 e9 00 f9 ff ff fb 66 0f 1f 44 00 00 e8 fd ff ff 65 48 01 51 10 e9 3e fe ff ff 48 8b 44 24 78 4d 89 [ 62.512629] RSP: 0018:ffffa7bfc29ffd20 EFLAGS: 00000206 [ 62.512631] RAX: 0000000000000200 RBX: ffffcd5405421888 RCX: ffff8c36ffdeb928 [ 62.512632] RDX: 0000000000040000 RSI: ffffa896f06b2ff8 RDI: ffffcd5405421880 [ 62.512633] RBP: ffffcd5405421880 R08: 000000000000007d R09: ffffffffffffffff [ 62.512634] R10: ffffffff9d63c084 R11: 00000000ffffffff R12: ffff8c373ffaeb80 [ 62.512635] R13: ffff8c36ffdf65f8 R14: ffff8c373ffaeb80 R15: 0000000000040000 [ 62.512637] FS: 0000000000000000(0000) GS:ffff8c36ffdc0000(0000) knlGS:0000000000000000 [ 62.512638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 62.512639] CR2: 000055c8e2fe8610 CR3: 0000000c13e10000 CR4: 00000000000006e0 [ 62.512641] Call Trace: [ 62.512646] __vmalloc_node_range+0x11c/0x2d0 [ 62.512649] ? full_fit_alloc_test+0x140/0x140 [test_vmalloc] [ 62.512654] __vmalloc_node+0x4b/0x70 [ 62.512656] ? fix_size_alloc_test+0x44/0x60 [test_vmalloc] [ 62.512659] fix_size_alloc_test+0x44/0x60 [test_vmalloc] [ 62.512662] test_func+0xe7/0x1f0 [test_vmalloc] [ 62.512666] ? fix_align_alloc_test+0x50/0x50 [test_vmalloc] [ 62.512668] kthread+0x11a/0x140 [ 62.512671] ? set_kthread_struct+0x40/0x40 [ 62.512672] ret_from_fork+0x22/0x30 To address this issue invoke a bulk-allocator many times until all pages are obtained, i.e. do batched page requests adding cond_resched() meanwhile to reschedule. Batched value is hard-coded and is 100 pages per call. Acked-by: Michal Hocko Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index aaad569e8963..a9a6d28c8baa 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2775,7 +2775,7 @@ EXPORT_SYMBOL_GPL(vmap_pfn); static inline unsigned int vm_area_alloc_pages(gfp_t gfp, int nid, - unsigned int order, unsigned long nr_pages, struct page **pages) + unsigned int order, unsigned int nr_pages, struct page **pages) { unsigned int nr_allocated = 0; @@ -2785,10 +2785,32 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * to fails, fallback to a single page allocator that is * more permissive. */ - if (!order) - nr_allocated = alloc_pages_bulk_array_node( - gfp, nid, nr_pages, pages); - else + if (!order) { + while (nr_allocated < nr_pages) { + unsigned int nr, nr_pages_request; + + /* + * A maximum allowed request is hard-coded and is 100 + * pages per call. That is done in order to prevent a + * long preemption off scenario in the bulk-allocator + * so the range is [1:100]. + */ + nr_pages_request = min(100U, nr_pages - nr_allocated); + + nr = alloc_pages_bulk_array_node(gfp, nid, + nr_pages_request, pages + nr_allocated); + + nr_allocated += nr; + cond_resched(); + + /* + * If zero or pages were obtained partly, + * fallback to a single page allocator. + */ + if (nr != nr_pages_request) + break; + } + } else /* * Compound pages required for remap_vmalloc_page if * high-order pages. From patchwork Wed Jul 7 18:26:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 12363563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B532C07E95 for ; Wed, 7 Jul 2021 18:34:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EDB8560C41 for ; Wed, 7 Jul 2021 18:34:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EDB8560C41 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D3CF26B0087; Wed, 7 Jul 2021 14:34:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D14036B0088; Wed, 7 Jul 2021 14:34:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB58E6B0089; Wed, 7 Jul 2021 14:34:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0076.hostedemail.com [216.40.44.76]) by kanga.kvack.org (Postfix) with ESMTP id 9A50F6B0087 for ; Wed, 7 Jul 2021 14:34:30 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 048CA18029D83 for ; Wed, 7 Jul 2021 18:34:30 +0000 (UTC) X-FDA: 78336642300.29.A18E350 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by imf26.hostedemail.com (Postfix) with ESMTP id 3E21920019C4 for ; Wed, 7 Jul 2021 18:34:26 +0000 (UTC) Received: by mail-ej1-f50.google.com with SMTP id bg14so4810095ejb.9 for ; Wed, 07 Jul 2021 11:34:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=g6Vax0DZu9/zEQ4mflRZs+GdOJbUoHKXVBfPpmObW7I=; b=YIFRQ16FXMx0sQBcdwP32kzHeP0LQNsMfY8IGc97+L4Y+m1GDhg1ri8h0SdC++tr7o Ui8VC7iKByCW95AM/dNs014Al/3sjoCpavddUwBWRHws7s2DbDBjfMkEQ4PcEt2YtcaZ BFSioQbqrPYX6gbbzCAqaT2B5hOCfb68Zp2T79/xKZPDp5h7rklJlompyZvqIM/cCu8v SE/aSZf7mhYvCOEgFXxR9L3VZfPMnuNESoKv10cPb4oMYGp5aTvXFQAbavnd95UB0iUV 9qWGg/GIeeIyggjI+Bk6QYjt5bNfxxQs0W2ehzjRlz5d+Vkurs36sHhC/lul6eKe4LVv s7fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=g6Vax0DZu9/zEQ4mflRZs+GdOJbUoHKXVBfPpmObW7I=; b=mEFOLjeVDOz5WyfDWCLDLb2VIyLCElEi+rTwR76W0i2/ALuOL+mP6tVDYcsXYIngQm GTA/dWARvJrvui7EihMK0gtQtPYmupkgU1wfPbOVJPuzQvq5vq8A3M/FGYTQzBUovQY7 ogtU41j1sfSnTh6pVg4Xl/RWYmrPm5TxBUJqV/Tv+496Lq3dLyIqm6Lxpx//mjkaoD0m s3w5rMV0M10HtqXktULe0kRGBUCLMJk/4krxU2cvxXxENw+of6K2aa8VlSuloEvBqpM5 rhkU/dwa9EA+nlv1DTJOvZu+roWjZ4xlD/mXcAWZeJ63MX8d97NJI3Jq2aqC94Edl+PF 6rRg== X-Gm-Message-State: AOAM533Htj3Vv8Na/46GGMb0D6WPQ7KvEjEo0NmAQ6oqbw+o40YPP5fx 7svmPAJPYsOupXlnZ0RIme/hTqpWm5MkkQ== X-Google-Smtp-Source: ABdhPJx9hsrdrNlNEmLWV6Fz2OG1hxARu0l4TtmFtTJMTyEIgiF9MFsRvCKK/sE784T3sMhyBOgT3A== X-Received: by 2002:a05:6512:2629:: with SMTP id bt41mr19591460lfb.95.1625682410291; Wed, 07 Jul 2021 11:26:50 -0700 (PDT) Received: from pc638.lan (h5ef52e3d.seluork.dyn.perspektivbredband.net. [94.245.46.61]) by smtp.gmail.com with ESMTPSA id o15sm1767963lfu.134.2021.07.07.11.26.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Jul 2021 11:26:49 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Mel Gorman , Christoph Hellwig , Matthew Wilcox , Nicholas Piggin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Oleksiy Avramchenko , Steven Rostedt Subject: [PATCH v3 2/2] mm/vmalloc: Remove gfpflags_allow_blocking() check Date: Wed, 7 Jul 2021 20:26:39 +0200 Message-Id: <20210707182639.31282-2-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210707182639.31282-1-urezki@gmail.com> References: <20210707182639.31282-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspam-User: nil Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=YIFRQ16F; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=urezki@gmail.com X-Stat-Signature: ss1oda55955hnkcdf6gbdy51ramywzjx X-Rspamd-Queue-Id: 3E21920019C4 X-Rspamd-Server: rspam01 X-HE-Tag: 1625682866-855163 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Get rid of gfpflags_allow_blocking() check from the vmalloc() path as it is supposed to be sleepable anyway. Thus remove it from the alloc_vmap_area() as well as from the vm_area_alloc_pages(). Acked-by: Michal Hocko Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a9a6d28c8baa..24a3955d5a36 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1479,6 +1479,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, int node, gfp_t gfp_mask) { struct vmap_area *va; + unsigned long freed; unsigned long addr; int purged = 0; int ret; @@ -1542,13 +1543,12 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, goto retry; } - if (gfpflags_allow_blocking(gfp_mask)) { - unsigned long freed = 0; - blocking_notifier_call_chain(&vmap_notify_list, 0, &freed); - if (freed > 0) { - purged = 0; - goto retry; - } + freed = 0; + blocking_notifier_call_chain(&vmap_notify_list, 0, &freed); + + if (freed > 0) { + purged = 0; + goto retry; } if (!(gfp_mask & __GFP_NOWARN) && printk_ratelimit()) @@ -2834,9 +2834,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, for (i = 0; i < (1U << order); i++) pages[nr_allocated + i] = page + i; - if (gfpflags_allow_blocking(gfp)) - cond_resched(); - + cond_resched(); nr_allocated += 1U << order; }