From patchwork Mon Oct 4 14:28:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 12534131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F706C433EF for ; Mon, 4 Oct 2021 14:28:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A84D76136F for ; Mon, 4 Oct 2021 14:28:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A84D76136F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 3E2BB940037; Mon, 4 Oct 2021 10:28:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3922F94000B; Mon, 4 Oct 2021 10:28:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E59C940037; Mon, 4 Oct 2021 10:28:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0144.hostedemail.com [216.40.44.144]) by kanga.kvack.org (Postfix) with ESMTP id 0B02294000B for ; Mon, 4 Oct 2021 10:28:46 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B6B6E82499A8 for ; Mon, 4 Oct 2021 14:28:45 +0000 (UTC) X-FDA: 78658986210.29.D81AC11 Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by imf18.hostedemail.com (Postfix) with ESMTP id 62E654002807 for ; Mon, 4 Oct 2021 14:28:45 +0000 (UTC) Received: by mail-lf1-f43.google.com with SMTP id u18so72640357lfd.12 for ; Mon, 04 Oct 2021 07:28:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=eg+DHxAUJJDbSYW1GrSqHaH8uii9mKgTL9ZBpTTfMCo=; b=TudXrq7tMBLVPZhR1exlmNn2TNf7Ny2ASPNzv89E4fLTF+2J2twC+Gu5HLmZLJgsoP FduwEadxA93I6+0ubwx8WmVEb66OA/KzgM79jghdnH9mBLwlFUvsO/F9FJOTSivOP8gL tOmgYNtJl0C9wQBLgSkw80Ei9TX/dFwm+/IsCHg6GBXZY2VkvDeXes76cc/nIHzy6fZq aUyW9R0RJ0FUChGjI6tTrFF9POchK++IzKC8EE3i/ROIizPMKbGdwFJkWuRwKAdAKMzK fePk9hMYHzm1qoV4kZdUbBMxKtCtuFdBVWmUWxXtqPb1bu7PyyCw7WvCQorlrQ+o4U+m x0Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=eg+DHxAUJJDbSYW1GrSqHaH8uii9mKgTL9ZBpTTfMCo=; b=Rx3dnUIYW+rxKOArN9Qmq3f7MFGLm17+yJ3UwaKsbGqSN5+fEF0bxNos9QaeTjbbvm zo1TDYmYN8NS1LihGmPtz5Pc80SAs179CPVF3yvUeJUwCBCkkpHrv8/v8CFefMSPk/MX naioIknrFwBNuMujT0NHpHu2LKjxs7vqHzYXokl+GE4+Lg/Rhw5vX1XShNukRx3XHE5v IoK1ETf/q12YqNMx0DaDdNxtAlpft8iubHY5o+qGzU5xKveWyESlVqE9Z3w/HEGgFrdE oH6bXSMw4mtzj7ht1gn6hRuUUlibJ7i5WPtnMbgTR5dSOJBp/mEHG4ZoyRUD6PYWB+DK GtIA== X-Gm-Message-State: AOAM532LRdkgF8WVDPZgk3oHycaR/5xc0hH95egr8LqKx/fh/axpw09O 2BwBNE/w+OYBxGd5tzFIwgzlmp6TZ9TInA== X-Google-Smtp-Source: ABdhPJwlS3RRUlS3X/6IztFKpVss1d8gRJj4375bw0g0X9mYpqx/pRZZnYdJNluxo8Ofk7ZU0yQOiQ== X-Received: by 2002:ac2:4f01:: with SMTP id k1mr15147549lfr.266.1633357722615; Mon, 04 Oct 2021 07:28:42 -0700 (PDT) Received: from pc638.lan (h5ef52e3d.seluork.dyn.perspektivbredband.net. [94.245.46.61]) by smtp.gmail.com with ESMTPSA id x15sm1338299lfe.129.2021.10.04.07.28.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Oct 2021 07:28:40 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Mel Gorman , Christoph Hellwig , Matthew Wilcox , Nicholas Piggin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Oleksiy Avramchenko , Steven Rostedt , Ping Fang , David Hildenbrand Subject: [PATCH 1/2] mm/vmalloc: Do not adjust the search size for alignment overhead Date: Mon, 4 Oct 2021 16:28:28 +0200 Message-Id: <20211004142829.22222-1-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 62E654002807 X-Stat-Signature: g5dur7y8nbfcgedz8iitq4c44u15tkp3 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TudXrq7t; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=urezki@gmail.com X-HE-Tag: 1633357725-201804 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We used to include an alignment overhead into a search length, in that case we guarantee that a found area will definitely fit after applying a specific alignment that user specifies. From the other hand we do not guarantee that an area has the lowest address if an alignment is >= PAGE_SIZE. It means that, when a user specifies a special alignment together with a range that corresponds to an exact requested size then an allocation will fail. This is what happens to KASAN, it wants the free block that exactly matches a specified range during onlining memory banks: [root@vm-0 fedora]# echo online > /sys/devices/system/memory/memory82/state [root@vm-0 fedora]# echo online > /sys/devices/system/memory/memory83/state [root@vm-0 fedora]# echo online > /sys/devices/system/memory/memory85/state [root@vm-0 fedora]# echo online > /sys/devices/system/memory/memory84/state [ 223.858115] vmap allocation for size 16777216 failed: use vmalloc= to increase size [ 223.859415] bash: vmalloc: allocation failure: 16777216 bytes, mode:0x6000c0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0 [ 223.860992] CPU: 4 PID: 1644 Comm: bash Kdump: loaded Not tainted 4.18.0-339.el8.x86_64+debug #1 [ 223.862149] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 223.863580] Call Trace: [ 223.863946] dump_stack+0x8e/0xd0 [ 223.864420] warn_alloc.cold.90+0x8a/0x1b2 [ 223.864990] ? zone_watermark_ok_safe+0x300/0x300 [ 223.865626] ? slab_free_freelist_hook+0x85/0x1a0 [ 223.866264] ? __get_vm_area_node+0x240/0x2c0 [ 223.866858] ? kfree+0xdd/0x570 [ 223.867309] ? kmem_cache_alloc_node_trace+0x157/0x230 [ 223.868028] ? notifier_call_chain+0x90/0x160 [ 223.868625] __vmalloc_node_range+0x465/0x840 [ 223.869230] ? mark_held_locks+0xb7/0x120 Fix it by making sure that find_vmap_lowest_match() returns lowest start address with any given alignment value, i.e. for alignments bigger then PAGE_SIZE the algorithm rolls back toward parent nodes checking right sub-trees if the most left free block did not fit due to alignment overhead. Fixes: 68ad4a330433 ("mm/vmalloc.c: keep track of free blocks for vmap allocation") Reported-by: Ping Fang Tested-by: David Hildenbrand Reviewed-by: David Hildenbrand Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 48e717626e94..9cce45dbdee0 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1195,18 +1195,14 @@ find_vmap_lowest_match(unsigned long size, { struct vmap_area *va; struct rb_node *node; - unsigned long length; /* Start from the root. */ node = free_vmap_area_root.rb_node; - /* Adjust the search size for alignment overhead. */ - length = size + align - 1; - while (node) { va = rb_entry(node, struct vmap_area, rb_node); - if (get_subtree_max_size(node->rb_left) >= length && + if (get_subtree_max_size(node->rb_left) >= size && vstart < va->va_start) { node = node->rb_left; } else { @@ -1216,9 +1212,9 @@ find_vmap_lowest_match(unsigned long size, /* * Does not make sense to go deeper towards the right * sub-tree if it does not have a free block that is - * equal or bigger to the requested search length. + * equal or bigger to the requested search size. */ - if (get_subtree_max_size(node->rb_right) >= length) { + if (get_subtree_max_size(node->rb_right) >= size) { node = node->rb_right; continue; } @@ -1226,15 +1222,23 @@ find_vmap_lowest_match(unsigned long size, /* * OK. We roll back and find the first right sub-tree, * that will satisfy the search criteria. It can happen - * only once due to "vstart" restriction. + * due to "vstart" restriction or an alignment overhead + * that is bigger then PAGE_SIZE. */ while ((node = rb_parent(node))) { va = rb_entry(node, struct vmap_area, rb_node); if (is_within_this_va(va, size, align, vstart)) return va; - if (get_subtree_max_size(node->rb_right) >= length && + if (get_subtree_max_size(node->rb_right) >= size && vstart <= va->va_start) { + /* + * Shift the vstart forward. Please note, we update it with + * parent's start address adding "1" because we do not want + * to enter same sub-tree after it has already been checked + * and no suitable free block found there. + */ + vstart = va->va_start + 1; node = node->rb_right; break; } From patchwork Mon Oct 4 14:28:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 12534133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DED3C433FE for ; Mon, 4 Oct 2021 14:28:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AF10061251 for ; Mon, 4 Oct 2021 14:28:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AF10061251 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 3BE2A940038; Mon, 4 Oct 2021 10:28:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 36B6A94000B; Mon, 4 Oct 2021 10:28:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 149EA940038; Mon, 4 Oct 2021 10:28:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id DED3294000B for ; Mon, 4 Oct 2021 10:28:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9E1472CBDA for ; Mon, 4 Oct 2021 14:28:46 +0000 (UTC) X-FDA: 78658986252.16.85BF837 Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) by imf27.hostedemail.com (Postfix) with ESMTP id 52F887001F4E for ; Mon, 4 Oct 2021 14:28:46 +0000 (UTC) Received: by mail-lf1-f47.google.com with SMTP id e15so72464231lfr.10 for ; Mon, 04 Oct 2021 07:28:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0LHatFhVEIqMlI2Msj/gKIKEeT6D+78N3ryw0xfIhf4=; b=QGO/ulsvPQ98gRMYF3sIRAWyWai5o4ioiYmqDqdO0Wt81Wjn9/ddpCKpzIjdljWaVx ycGTegDq5EcpJGQToHEUIz8KOxgT4AZCembfMm9K5iGsjsnvAXCFmJ7bgG6BIURWIj6N DcB0vMhIbyUzmGvOUtTV4zkcje83glJbCMWnS+fmuYrkOz9rUlPBsXuttCfV2bd18AUe K7l43++pjUB+vlfDhLJ+9qb8BYAeb+n156MFLmjbnGyCapzhGHGlzJPpFOsAYcPF1/nf PY+v/abcCvFxtd8h4MEbVm96tBsrEdf2cpqQrmEKZ3vIoocRTznyKxOlBKRu4lKhv/i3 Jucw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0LHatFhVEIqMlI2Msj/gKIKEeT6D+78N3ryw0xfIhf4=; b=vGAq18UnstBTOOBX2BJRX6nfx9PExKakD/aK2kPW8Ann/xV+4pQzo5uOsxl/XPOjPm 6cvyGRfvtHbat5O35MEFz6ir3CEAbcw5p705Hi00GGfV9N26t5aQtpfDqmkuubQtsLGE J2Kcu/Ezf4FVOFjSFm40EeJ2ilXyUaX3dA8n95clo0IK3igKCpTTzhckOB9ihub1Ahpv NX/z/FjzDJe9uT1bmDnyk6stVu8Fwoi4al8R3FBSMd2X94desfsr3edE+ahpCNe72Xso y1sG6BzC849m76hg/QV53LtIjXvRzRQNyOyPve/fLhAEGeMk4KsLl5D549e26NvSqzSW 9dKQ== X-Gm-Message-State: AOAM533BzB81GY6Lfy1in43aHIRp9xV5KyOSlvtHN3qeITDqwp/eJxpT PKqEEcwJFTGIY1NqmRNJoMY= X-Google-Smtp-Source: ABdhPJzNPnvc1CO4IOvFyX1d4m2ON2hLZB/7CiFpBal3s8jAdqO42Mg2HEVq4rjm84SPVvj56OxmLQ== X-Received: by 2002:a05:6512:3082:: with SMTP id z2mr14905907lfd.657.1633357724143; Mon, 04 Oct 2021 07:28:44 -0700 (PDT) Received: from pc638.lan (h5ef52e3d.seluork.dyn.perspektivbredband.net. [94.245.46.61]) by smtp.gmail.com with ESMTPSA id x15sm1338299lfe.129.2021.10.04.07.28.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Oct 2021 07:28:43 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Mel Gorman , Christoph Hellwig , Matthew Wilcox , Nicholas Piggin , Uladzislau Rezki , Hillf Danton , Michal Hocko , Oleksiy Avramchenko , Steven Rostedt Subject: [PATCH 2/2] mm/vmalloc: Check various alignments when debugging Date: Mon, 4 Oct 2021 16:28:29 +0200 Message-Id: <20211004142829.22222-2-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20211004142829.22222-1-urezki@gmail.com> References: <20211004142829.22222-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 52F887001F4E X-Stat-Signature: egrampubincff6iens55xqm19y1ieb4a Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="QGO/ulsv"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.47 as permitted sender) smtp.mailfrom=urezki@gmail.com X-HE-Tag: 1633357726-25974 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Before we did not guarantee a free block with lowest start address for allocations with alignment >= PAGE_SIZE. Because an alignment overhead was included into a search length like below: length = size + align - 1; doing so we make sure that a bigger block would fit after applying an alignment adjustment. Now there is no such limitation, i.e. any alignment that user wants to apply will result to a lowest address of returned free area. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9cce45dbdee0..343cb5d40706 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1269,7 +1269,7 @@ find_vmap_lowest_linear_match(unsigned long size, } static void -find_vmap_lowest_match_check(unsigned long size) +find_vmap_lowest_match_check(unsigned long size, unsigned long align) { struct vmap_area *va_1, *va_2; unsigned long vstart; @@ -1278,8 +1278,8 @@ find_vmap_lowest_match_check(unsigned long size) get_random_bytes(&rnd, sizeof(rnd)); vstart = VMALLOC_START + rnd; - va_1 = find_vmap_lowest_match(size, 1, vstart); - va_2 = find_vmap_lowest_linear_match(size, 1, vstart); + va_1 = find_vmap_lowest_match(size, align, vstart); + va_2 = find_vmap_lowest_linear_match(size, align, vstart); if (va_1 != va_2) pr_emerg("not lowest: t: 0x%p, l: 0x%p, v: 0x%lx\n", @@ -1458,7 +1458,7 @@ __alloc_vmap_area(unsigned long size, unsigned long align, return vend; #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK - find_vmap_lowest_match_check(size); + find_vmap_lowest_match_check(size, align); #endif return nva_start_addr;