From patchwork Sun May 9 19:38:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 12246523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CF44C433ED for ; Sun, 9 May 2021 19:38:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A4E0A613AB for ; Sun, 9 May 2021 19:38:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4E0A613AB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C19006B0070; Sun, 9 May 2021 15:38:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA1656B0071; Sun, 9 May 2021 15:38:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F4496B0072; Sun, 9 May 2021 15:38:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id 7E9056B0070 for ; Sun, 9 May 2021 15:38:57 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 31F05181AF5C2 for ; Sun, 9 May 2021 19:38:57 +0000 (UTC) X-FDA: 78122705514.09.8A080AC Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) by imf29.hostedemail.com (Postfix) with ESMTP id 96C3EF4 for ; Sun, 9 May 2021 19:38:50 +0000 (UTC) Received: by mail-lj1-f173.google.com with SMTP id w4so18153298ljw.9 for ; Sun, 09 May 2021 12:38:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=P9KHoUwqxbexOhqNv8F1Ywyb8vsO2FBF8NuW/VNOiPo=; b=VaZ5jstwwQCops8VcXABpwtZedX2N9xGSqxNt94hO02IGPASRBxnIMwnDKVET16xIi qrW9cC7TB3wiCLgTDnnj9R1JMDHfvUU/N7jVA/46hjEV8jZkc9PdX4CtqaPZejlKJVUp ETz4JyWiit7zgBhH8pkTWPAL3OW++bggSY7s8K7R3cKRt8KBX9HqpHsIyJAxbyptwYUp 7tHEmMuazGZlkq8s1AxphPfoqkf71QP0FICuzJbptiSV+NoAzL5W24z1O5RNM3BfOrRn HXLZI4BQBCnWlqdDzHonZEbrNU7Wg+GhQBCjZar4gDpxl+NrGpuGnfuR35q+q74fSAVk 6hEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=P9KHoUwqxbexOhqNv8F1Ywyb8vsO2FBF8NuW/VNOiPo=; b=eszFiVI8IldqpW3TjgoHGSa0abCrsqeVvfyh7aOcBMfcR4JqrpJ7Q2MaIFQvGpbTtd tQgTHNOIiddPokb14ZLdRzsSh0DQhHqC7q8pjBTuELv1Uq4dG2GlY5HUAf1HP+SRZFtr 1WBph0rxblx78P0EwhEhCOgrRzufkV7UqMUXle4XbarkK35StlgselLHih5yqIdsRbmg FFuwclNrJ18H4YnE+Q8KpFjzoU8Kqv4KhOKE5fypdet8W5ix+WBZAzc9ERE5OrRGBb76 mBEgyknUs2WgWMzqZ/X55RnFRkN+jvncs9BcRpN5DF+s+565M93UkDphESx3kJFF+Ipq Y0WQ== X-Gm-Message-State: AOAM530GyQtAA3PN34EvwAYkQC+lWA1iVNHp6gnTLiXl8TBsmFeHwd/B gsGKZ/Eo+n6YIbdxE4l5Efc= X-Google-Smtp-Source: ABdhPJwoKB66eyiScOwS2EQBs77oKT0vsxLrKNhx3uXTnls4aHYUOxRXMZAoxnrRucSwVYo2UlrlIw== X-Received: by 2002:a2e:b5cd:: with SMTP id g13mr17743675ljn.0.1620589135179; Sun, 09 May 2021 12:38:55 -0700 (PDT) Received: from pc638.lan (h5ef52e3d.seluork.dyn.perspektivbredband.net. [94.245.46.61]) by smtp.gmail.com with ESMTPSA id m17sm1999269lfh.146.2021.05.09.12.38.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 May 2021 12:38:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Matthew Wilcox , Nicholas Piggin , Mel Gorman , Uladzislau Rezki , Hillf Danton , Michal Hocko , Oleksiy Avramchenko , Steven Rostedt Subject: [PATCH v2 1/2] mm/vmalloc: Switch to bulk allocator in __vmalloc_area_node() Date: Sun, 9 May 2021 21:38:43 +0200 Message-Id: <20210509193844.2562-1-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=VaZ5jstw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.173 as permitted sender) smtp.mailfrom=urezki@gmail.com X-Stat-Signature: 3dz3877qoja11iu9bto5bihsbojz4y5a X-Rspamd-Queue-Id: 96C3EF4 X-Rspamd-Server: rspam05 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=mail-lj1-f173.google.com; client-ip=209.85.208.173 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620589130-717755 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recently there has been introduced a page bulk allocator for users which need to get number of pages per one call request. For order-0 pages switch to __alloc_pages_bulk() instead of alloc_pages_node(), the reason is the former is not capable of allocating set of pages, thus a one call is per one page. Second, according to my tests the bulk allocator uses less cycles even for scenarios when only one page is requested. Running the "perf" on same test case shows below difference: - 45.18% __vmalloc_node - __vmalloc_node_range - 35.60% __alloc_pages - get_page_from_freelist 3.36% __list_del_entry_valid 3.00% check_preemption_disabled 1.42% prep_new_page - 31.00% __vmalloc_node - __vmalloc_node_range - 14.48% __alloc_pages_bulk 3.22% __list_del_entry_valid - 0.83% __alloc_pages get_page_from_freelist The "test_vmalloc.sh" also shows performance improvements: fix_size_alloc_test_4MB loops: 1000000 avg: 89105095 usec fix_size_alloc_test loops: 1000000 avg: 513672 usec full_fit_alloc_test loops: 1000000 avg: 748900 usec long_busy_list_alloc_test loops: 1000000 avg: 8043038 usec random_size_alloc_test loops: 1000000 avg: 4028582 usec fix_align_alloc_test loops: 1000000 avg: 1457671 usec fix_size_alloc_test_4MB loops: 1000000 avg: 62083711 usec fix_size_alloc_test loops: 1000000 avg: 449207 usec full_fit_alloc_test loops: 1000000 avg: 735985 usec long_busy_list_alloc_test loops: 1000000 avg: 5176052 usec random_size_alloc_test loops: 1000000 avg: 2589252 usec fix_align_alloc_test loops: 1000000 avg: 1365009 usec For example 4MB allocations illustrates ~30% gain, all the rest is also better. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 74 ++++++++++++++++++++++++++++------------------------ 1 file changed, 40 insertions(+), 34 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5d96fee17226..dbc6744400d5 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2766,8 +2766,6 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, unsigned long array_size; unsigned int nr_small_pages = size >> PAGE_SHIFT; unsigned int page_order; - struct page **pages; - unsigned int i; array_size = (unsigned long)nr_small_pages * sizeof(struct page *); gfp_mask |= __GFP_NOWARN; @@ -2776,13 +2774,13 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, /* Please note that the recursion is strictly bounded. */ if (array_size > PAGE_SIZE) { - pages = __vmalloc_node(array_size, 1, nested_gfp, node, + area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, area->caller); } else { - pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node(array_size, nested_gfp, node); } - if (!pages) { + if (!area->pages) { free_vm_area(area); warn_alloc(gfp_mask, NULL, "vmalloc size %lu allocation failure: " @@ -2791,43 +2789,51 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, return NULL; } - area->pages = pages; - area->nr_pages = nr_small_pages; set_vm_area_page_order(area, page_shift - PAGE_SHIFT); - page_order = vm_area_page_order(area); - /* - * Careful, we allocate and map page_order pages, but tracking is done - * per PAGE_SIZE page so as to keep the vm_struct APIs independent of - * the physical/mapped size. - */ - for (i = 0; i < area->nr_pages; i += 1U << page_order) { - struct page *page; - int p; - - /* Compound pages required for remap_vmalloc_page */ - page = alloc_pages_node(node, gfp_mask | __GFP_COMP, page_order); - if (unlikely(!page)) { - /* Successfully allocated i pages, free them in __vfree() */ - area->nr_pages = i; - atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - warn_alloc(gfp_mask, NULL, - "vmalloc size %lu allocation failure: " - "page order %u allocation failed", - area->nr_pages * PAGE_SIZE, page_order); - goto fail; - } + if (!page_order) { + area->nr_pages = __alloc_pages_bulk(gfp_mask, node, + NULL, nr_small_pages, NULL, area->pages); + } else { + /* + * Careful, we allocate and map page_order pages, but tracking is done + * per PAGE_SIZE page so as to keep the vm_struct APIs independent of + * the physical/mapped size. + */ + for (area->nr_pages = 0; area->nr_pages < nr_small_pages; + area->nr_pages += 1U << page_order) { + struct page *page; + int i; + + /* Compound pages required for remap_vmalloc_page */ + page = alloc_pages_node(node, gfp_mask | __GFP_COMP, page_order); + if (unlikely(!page)) + break; - for (p = 0; p < (1U << page_order); p++) - area->pages[i + p] = page + p; + for (i = 0; i < (1U << page_order); i++) + area->pages[area->nr_pages + i] = page + i; - if (gfpflags_allow_blocking(gfp_mask)) - cond_resched(); + if (gfpflags_allow_blocking(gfp_mask)) + cond_resched(); + } } + atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) { + /* + * If not enough pages were obtained to accomplish an + * allocation request, free them via __vfree() if any. + */ + if (area->nr_pages != nr_small_pages) { + warn_alloc(gfp_mask, NULL, + "vmalloc size %lu allocation failure: " + "page order %u allocation failed", + area->nr_pages * PAGE_SIZE, page_order); + goto fail; + } + + if (vmap_pages_range(addr, addr + size, prot, area->pages, page_shift) < 0) { warn_alloc(gfp_mask, NULL, "vmalloc size %lu allocation failure: " "failed to map pages", From patchwork Sun May 9 19:38:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 12246525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BACCC433B4 for ; Sun, 9 May 2021 19:39:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1F13C6023C for ; Sun, 9 May 2021 19:38:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1F13C6023C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 27B6A6B0071; Sun, 9 May 2021 15:38:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 205CF6B0072; Sun, 9 May 2021 15:38:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A6036B0073; Sun, 9 May 2021 15:38:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0226.hostedemail.com [216.40.44.226]) by kanga.kvack.org (Postfix) with ESMTP id E245E6B0072 for ; Sun, 9 May 2021 15:38:57 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 91049181AF5C3 for ; Sun, 9 May 2021 19:38:57 +0000 (UTC) X-FDA: 78122705514.13.5D6EA33 Received: from mail-lj1-f180.google.com (mail-lj1-f180.google.com [209.85.208.180]) by imf23.hostedemail.com (Postfix) with ESMTP id E3CFDA000380 for ; Sun, 9 May 2021 19:38:47 +0000 (UTC) Received: by mail-lj1-f180.google.com with SMTP id w4so18153314ljw.9 for ; Sun, 09 May 2021 12:38:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=S8tyYOINgJcgXO+YLeHnUlDVZZruEdZrDr6md8OI68k=; b=nzlu7ccem0ygsiY/jJ0x16lK73WjRmh51B0MtwxG3bflCWQEcPuh5mOREu5EmGPZ5B CC/WfyY/pGPTn6gfZq5uu5F26Vm4vqWf21hIl6ey2hLhXx/BoRdZOR8LMoq3HhtAG7vx giNPFVzOMbameC316jucY9r6OuSbFqBOUsBBKOSliqXCd2PCcJSk9OOGGHfeou4KN1Ch oC/MrMUjGWP/z2VL1S1h1perKLwI00rajDsMI1u0/5RLWjmzS5jumBGY80DlopOtZokG l0Pcnw0ze7ZguflS6CS+4Dke8VFGlVwB6PEwatDZ3zICvqrv1z/8ddaRnRrcl0Id9VOZ pYwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S8tyYOINgJcgXO+YLeHnUlDVZZruEdZrDr6md8OI68k=; b=hfCYFKT6CkT1tuuTKZsEM6Pc3qj5VhCcjpvSnxDjWTZBC+tVOdsY6SQik3OlnenjmL Pywjl8Jn56ygkx4XseuR+82SOpuT/OPymwYo4Fsxitba8K9B9T6f5Q0z7jc1Rfnon1Mc Wc5l8iITHuFhKnQ9l11ROGC4Z84h7jgC9LvQ7BAAxA/2vO8dSAjWFF+FkJzDKxwk1+i2 0bb1GaNZ1rcx1O2YjRgJAtBr+rGis+Yy2iqEfPxRcRkpRzTVSXdDLR3EvUBK0tJ3fvMs 04qVDYAqrmkpkYU4iVrM2i6Sc5oI3I8iJ6aRT+PiGimSJ1LXVIpgFVBaHKpKTYb/Tutv ZFWQ== X-Gm-Message-State: AOAM532JIkvuH4eurnMA5rYPfwRV0C2kW1LTBc6jJ17enSYB2paM/ZO8 l9D0/h04LYYzmb9QNHx2dn/YFgOpgVM= X-Google-Smtp-Source: ABdhPJz4+CgRWqWhVIi9vQherdRk/qSSdjAHrJnWqHztMNOv2DvPjKazUo0hPzXDsA4SYTO7xFWqbQ== X-Received: by 2002:a2e:7005:: with SMTP id l5mr945485ljc.150.1620589135987; Sun, 09 May 2021 12:38:55 -0700 (PDT) Received: from pc638.lan (h5ef52e3d.seluork.dyn.perspektivbredband.net. [94.245.46.61]) by smtp.gmail.com with ESMTPSA id m17sm1999269lfh.146.2021.05.09.12.38.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 May 2021 12:38:55 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Matthew Wilcox , Nicholas Piggin , Mel Gorman , Uladzislau Rezki , Hillf Danton , Michal Hocko , Oleksiy Avramchenko , Steven Rostedt Subject: [PATCH v2 2/2] mm/vmalloc: Print a warning message first on failure Date: Sun, 9 May 2021 21:38:44 +0200 Message-Id: <20210509193844.2562-2-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210509193844.2562-1-urezki@gmail.com> References: <20210509193844.2562-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=nzlu7cce; spf=pass (imf23.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.180 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: pcejpho1mssu3mfkbj8ums3tw5inpnhe X-Rspamd-Queue-Id: E3CFDA000380 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf23; identity=mailfrom; envelope-from=""; helo=mail-lj1-f180.google.com; client-ip=209.85.208.180 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620589127-275499 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a memory allocation for array of pages are not succeed emit a warning message as a first step and then perform the further cleanup. Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Uladzislau Rezki (Sony) --- Makefile | 2 +- mm/vmalloc.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index bb50a5ac2e13..1d658e171495 100644 --- a/Makefile +++ b/Makefile @@ -430,7 +430,7 @@ HOSTCXX = g++ endif export KBUILD_USERCFLAGS := -Wall -Wmissing-prototypes -Wstrict-prototypes \ - -O2 -fomit-frame-pointer -std=gnu89 + -O0 -g -fomit-frame-pointer -std=gnu89 export KBUILD_USERLDFLAGS := KBUILD_HOSTCFLAGS := $(KBUILD_USERCFLAGS) $(HOST_LFS_CFLAGS) $(HOSTCFLAGS) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index dbc6744400d5..1f664a17d9ea 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2781,11 +2781,11 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } if (!area->pages) { - free_vm_area(area); warn_alloc(gfp_mask, NULL, "vmalloc size %lu allocation failure: " "page array size %lu allocation failed", nr_small_pages * PAGE_SIZE, array_size); + free_vm_area(area); return NULL; }