From patchwork Tue Dec 20 18:27:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13078089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC10FC4332F for ; Tue, 20 Dec 2022 18:27:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 238A88E0003; Tue, 20 Dec 2022 13:27:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C2A68E0001; Tue, 20 Dec 2022 13:27:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03C8E8E0003; Tue, 20 Dec 2022 13:27:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E89068E0001 for ; Tue, 20 Dec 2022 13:27:10 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A353CAB2D1 for ; Tue, 20 Dec 2022 18:27:10 +0000 (UTC) X-FDA: 80263516620.18.9CC6C6F Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by imf07.hostedemail.com (Postfix) with ESMTP id F163D40011 for ; Tue, 20 Dec 2022 18:27:08 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=He4LSkff; spf=pass (imf07.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671560829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=al6BhI4srzCs1FPe4+yTumkcC9D7WK89QS4j43wccGo=; b=cE74hdaI7edrr17ZrNSS4TkK/parg5H0wxbGa7F7w5icMH7+cNxG1HYhyx3MQxljQT3LNH fXWPQfFISkIRocpdQiLHxnX7Qbe7qeO5dwXU5uSAgcheA8BuRrOgwwRNeiAXm+BlY+qlVG 6OQtgoh8QbDWcDmfLgZEfNXM5YfTkGs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=He4LSkff; spf=pass (imf07.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671560829; a=rsa-sha256; cv=none; b=auENM7BDD6zKmrtZ9V/JBiAoUCY4yj0lXSitiC+A5D4vBmBHAGzaHb74RkQyxK/+c5JLvb L5lN7SdobKVCUdBB5sKoUACahab2aVXLB1Cl/JwHmXwhx3froim9TA7p6TkVc6BGi/YvsR yE410Ig13WoNbIGYUa0XihlM7fnm1nw= Received: by mail-lf1-f51.google.com with SMTP id bp15so19851201lfb.13 for ; Tue, 20 Dec 2022 10:27:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=al6BhI4srzCs1FPe4+yTumkcC9D7WK89QS4j43wccGo=; b=He4LSkfftmKDRS5QkA1CHeTe2kxMW4ulbRyZLHgaRHwJpmli8nEsB8nhk0unOX8AwH 2cyOvpiA9souQmUDOKwNyAeak6kcvuR/lyUn39+tCY53FnYkaHZqhCIXdQaMiTOtca+y y82yKp0un3QQE6s0pQz74OZ4WrUzX59kDfmGepGdtvtoBcoO1DXIoM+Zkby+Arbpki+W griVbhtqtjHd7tcrmPadRLvswFVF/CL3Mny5BP9ATQkcge/SUzjPE+W4cMxTJpT0N9p0 lWJ1L2iB23C70e1sngYS7nMjW5RHx2Wv1rqcS5Jzsnj5kZfAJMPMtSUhXt0RGUYMxBWh HyJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=al6BhI4srzCs1FPe4+yTumkcC9D7WK89QS4j43wccGo=; b=NIXtcByT92JYsTyhHlddXulbc05uwU65PO542/uuEEAF21eVP46rj3ryWdWC6iZgI3 NOPb350Yue2KzvZzsE8MY1zsrsmuIZxRLUi68USaJfFLtQpWdvQS44iMIIHQ+hr+tVgj 6SE24NcqRN/TArfZze4+iD5/y0mapJOtCmnqDBMQkqJCbKyBQW+YJunn9jN0VB3qTWDl v5YnH9oprMniH/mEN6aePNWuNGMYpIryyRNyW9XIFJW9HKT1Wrtr1q96j6A3gm56QWu1 27JfSZ5XcfKroz41ZPme23D97IwwgDeRLLyLF7DElrKG70P5JsrJqMbyiHtVeviYqixv +gwg== X-Gm-Message-State: ANoB5pmje3DrAcyzMSvhV7wHxlvRKbDV0vYXvOrVfIUrSFnPW1NuuHLr VKx7aTbs970JXLWgOt5A17M= X-Google-Smtp-Source: AA0mqf5bsLcJzPxHFQJ+drM/vi0aDKZw6r3yWC0LCOZJr+LNEMBjAetLFsvy96VuxR+YKa3lFTUudg== X-Received: by 2002:a05:6512:3a8b:b0:4b5:6d8d:b4cd with SMTP id q11-20020a0565123a8b00b004b56d8db4cdmr16237780lfu.54.1671560827012; Tue, 20 Dec 2022 10:27:07 -0800 (PST) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id v17-20020a056512349100b004bc1dd05351sm1538176lfr.206.2022.12.20.10.27.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 10:27:06 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: Andrew Morton Cc: linux-mm@kvack.org, LKML , Baoquan He , Christoph Hellwig , Matthew Wilcox , Nicholas Piggin , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 1/2] mm: vmalloc: Avoid a double lookup of freed VA in a tree Date: Tue, 20 Dec 2022 19:27:03 +0100 Message-Id: <20221220182704.181657-1-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-Stat-Signature: np4mjbb58ejgjacgfgud8upck7ju7x31 X-Rspam-User: X-Rspamd-Queue-Id: F163D40011 X-Rspamd-Server: rspam06 X-HE-Tag: 1671560828-308599 X-HE-Meta: U2FsdGVkX1/4Uk8wknRd7rSILxeT32bGuVtuy3ut5Rlo0l27hgW/KV7FAKgxYDM09lgc3feJ/5Uexvdq7zzZ8VssV92p95O69O+jJxO3L+Tc799CQxm2ew8pbUyHmeEplKCRAs7etuKBwudj/Zf/DX476D/6lQI+E50WWMlm1SPxbVf3eqVJcGg1O4kJw5LLCjqBUT5n6s7l0Gw2O13QZeKDK7bLSU/EQf+FSzDfjmCe0nghEW4gBzncK1v/uM61a5daiwcMRiJKjVNAbS3mW2iRywhRnkFR4L2miSyf6ToGJnHvPvn9h0RbWq/C3FQeCfsYNPcVN9lOU+QFAXoDEnXnIZZVvxuG4ROZBE/khi9eahetFL23zMTbnjvwrBts4faTgQTPIv8N3ur16xAAlLiLjGU0BBsLe3V5es3cCAJni6KhuWuOrDRBdde9FhBXV8rw/XwMluSHcwc2lJZga4VLGxgMKgs+JPXZidTfsy/2phq/1n0Qg6ld1onrkn75W5JoAIyvYKVYUbDIe4ktCAFwVzChWKF6xxsHNOxEZx5YJ3Y6WYb5xLR6NTn0Jgs6bvHNnRZbXVu9S4+ku5MydCDSiWh0KUv5NL+RXv+ejnKYriqH91Uv+4GJENpUcE7ZZN1b8SRlJSd9UsiUua4Qhk8lbHw15Xxvwf+BWm3wfa654JUC+w8rffZ5NfZqCrwnxiidLaTSYnpCcYqgZNKLQHxG+EObkzRyuuuET2cHIGmEZeT6JZ0LMIAFpjzeeBchbM5BrrheexD0g0wuqbK/Pu/Xf8fbEYxu0E68c4yvT828fPjLKCGCId6nB/kbxD+5IYHrLeJCO9ANgIv/x8FnVGZrtsDEAHicDzhWYQTjy4AXJG2xmHygMnwg5zu9ytw/UDjUJhY3g5Qw6KrmRL66kpvTXwp1XPl8xBWO+4bKoYDdv76MOq8nYO4lfVZjZdK/oWRWutd3MDrf9gWgd66 5uKRs8Gw kUByEd8tEyiyYn5vXSqbqsk6bD8yRiL28boRr7BrbwEJr+PCFR6YY0Amc+GGSgG8wFK2XeXBBzWo6iMIK7nDfgIURqKz5/lv77hqfpAVTsJj/KtqcUNsdGz9Tcysuz7iBmJ0i X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a VA is freed over a main path, for example by invoking the vfree() function, a tree is accessed two times what is odd: vfree(): __vunmap() __find_vmap_area() vm_remove_mappings() remove_vm_area() __find_vmap_area() __find_vmap_area() are called two times. Fix it by introducing a find_unlink_vmap_area() helper that finds and un-links a VA from a tree. Performance test results on a single CPU: - fix_size_alloc_test loops: 1000000 avg: 476847 usec - full_fit_alloc_test loops: 1000000 avg: 806746 usec - long_busy_list_alloc_test loops: 1000000 avg: 13552093 usec - random_size_alloc_test loops: 1000000 avg: 7441322 usec - fix_align_alloc_test loops: 1000000 avg: 1411132 usec All test took worker0=87650866284 cycles - fix_size_alloc_test loops: 1000000 avg: 490713 usec - full_fit_alloc_test loops: 1000000 avg: 579162 usec - long_busy_list_alloc_test loops: 1000000 avg: 10485448 usec - random_size_alloc_test loops: 1000000 avg: 5824449 usec - fix_align_alloc_test loops: 1000000 avg: 984735 usec All test took worker0=67952362802 cycles Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9e30f0b39203..0fc38c36e0df 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1825,9 +1825,14 @@ static void free_vmap_area_noflush(struct vmap_area *va) unsigned long va_start = va->va_start; unsigned long nr_lazy; - spin_lock(&vmap_area_lock); - unlink_va(va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + /* + * A free_vmap_block() is left. It is NOT a main free path. + */ + if (!list_empty(&va->list)) { + spin_lock(&vmap_area_lock); + unlink_va(va, &vmap_area_root); + spin_unlock(&vmap_area_lock); + } nr_lazy = atomic_long_add_return((va->va_end - va->va_start) >> PAGE_SHIFT, &vmap_lazy_nr); @@ -1871,6 +1876,19 @@ struct vmap_area *find_vmap_area(unsigned long addr) return va; } +static struct vmap_area *find_unlink_vmap_area(unsigned long addr) +{ + struct vmap_area *va; + + spin_lock(&vmap_area_lock); + va = __find_vmap_area(addr, &vmap_area_root); + if (va) + unlink_va(va, &vmap_area_root); + spin_unlock(&vmap_area_lock); + + return va; +} + /*** Per cpu kva allocator ***/ /* @@ -2236,7 +2254,7 @@ void vm_unmap_ram(const void *mem, unsigned int count) return; } - va = find_vmap_area(addr); + va = find_unlink_vmap_area(addr); BUG_ON(!va); debug_check_no_locks_freed((void *)va->va_start, (va->va_end - va->va_start)); @@ -2607,21 +2625,16 @@ struct vm_struct *remove_vm_area(const void *addr) might_sleep(); - spin_lock(&vmap_area_lock); - va = __find_vmap_area((unsigned long)addr, &vmap_area_root); - if (va && va->vm) { + va = find_unlink_vmap_area((unsigned long) addr); + if (va) { struct vm_struct *vm = va->vm; - va->vm = NULL; - spin_unlock(&vmap_area_lock); - kasan_free_module_shadow(vm); free_unmap_vmap_area(va); return vm; } - spin_unlock(&vmap_area_lock); return NULL; } @@ -2690,6 +2703,7 @@ static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) static void __vunmap(const void *addr, int deallocate_pages) { struct vm_struct *area; + struct vmap_area *va; if (!addr) return; @@ -2698,7 +2712,9 @@ static void __vunmap(const void *addr, int deallocate_pages) addr)) return; - area = find_vm_area(addr); + va = find_unlink_vmap_area((unsigned long)addr); + area = va->vm; + if (unlikely(!area)) { WARN(1, KERN_ERR "Trying to vfree() nonexistent vm area (%p)\n", addr);