From patchwork Mon May 22 11:08:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7299EC7EE2A for ; Mon, 22 May 2023 11:09:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B266280009; Mon, 22 May 2023 07:09:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 83C4B280002; Mon, 22 May 2023 07:09:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F263280009; Mon, 22 May 2023 07:09:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4C0FC280002 for ; Mon, 22 May 2023 07:09:02 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F116140D74 for ; Mon, 22 May 2023 11:09:01 +0000 (UTC) X-FDA: 80817618882.24.6DE7D9A Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) by imf10.hostedemail.com (Postfix) with ESMTP id 13782C0015 for ; Mon, 22 May 2023 11:08:59 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="Ad+N9/Bx"; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.45 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753740; a=rsa-sha256; cv=none; b=DbRhPzSjoxYdYb6XmrB6Oh8hn6GVpKhgD8nmdEm1ucwK94EFIF6RpgnI0eIjVWcBTdHM0v NN9IKyI2wpCBqJgjkj93cMsIAlsCRCOgimwtu/dXkuSQMxe8yNwrYBMHM6XfeOPhEyiAj1 9g5tYa0c65/V2QZfF9jioIm/h2HUTgo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="Ad+N9/Bx"; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.45 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=55sY0hUMVMxmKvjlYwxs4pvClntO+d/CkKt0D/BL0C26WrheBoihj9M7u/MKWuXLV7O/qS N4ToD6bUYbPqGClI5ZDhVInZoCNBZbQ69gboBuJLRrCa/UXMA+Zl1vGWymSSOSrENp8FhK NCLHOAHTy9eYBW3+Rq3O6Km5gwOD7/U= Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-96f818c48fbso542123866b.0 for ; Mon, 22 May 2023 04:08:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=Ad+N9/BxXUiItff9Ka8gp9m6Ut+hvQw/iiM/znN/E/FjnWbeKMX497Feia+wUTQL7P nn0HMTXM+eAOwCIXtFoVtxCNH0LHncvSzMz7xzwQ1Srikc1cgV966ZS1qa7YdSUE18l3 cG1SVgkhO1ucaJlN4YMO1/+KLZpOHs16VnpaYal4+QLxI/ktvW/6nIUZRewRfRy6T7uj t6Bkz1DHlo2XaUESkC1cNdMlQINss0j4ahV0Ld6LZrYiZAUobHEQcZF2xuwkm88kQ9RE EafR2f2o+2S2vnNz55Q7/rEKNFxFSEEhyB+7TfomnLLk3I+4DN8cvc7+ZtkoKPBd38H7 HuUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=SZ2bh2G1jF0Hbrlix6Sb3jcKRITsLkUPe9Wd8UngV5BHgNDreyGXIcE/mx9gMmypmF VQzynCaASSs72wU+uSLG96lx87LLyGAen/yEpwZPOgNjo9vrxhBc4ysFHol9mR0G4xb2 Uv+0SPwomeFF+SATLKmf5sQApm0afvcaGzsAdVO0Xd4p0kd+Fm2JhHv41XceMjx3VOOO 1paxDO3fgqpOt6v8+gau+JX9cs6E2PWB/ZS4tnOPQtAcsdA3931mbxl9jghnDrCAU4A2 OyJb1SL0mlNBi36YYO8nbwu5/FfoQ9l6jNQuRyTSaEUCgLugpN5jGv3TTw1lHM5y5Z60 zG1A== X-Gm-Message-State: AC+VfDz3DhAaVOPGijtrxJcxTSppir5Tz4gdlFZ2kDbegfJXPEL3Kbg6 wUMbwyfJIZKHP9iK/Xmz22vYWrAXh91rUA== X-Google-Smtp-Source: ACHHUZ4pAASV3ZfWJMFVdFF/6izNH1fGukazCQNkIlZEbt/DA1kUADDrO481FwKNM2C/1frFuAV8LA== X-Received: by 2002:a17:907:1c07:b0:96f:d8a9:d045 with SMTP id nc7-20020a1709071c0700b0096fd8a9d045mr3465283ejc.59.1684753738602; Mon, 22 May 2023 04:08:58 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:58 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 8/9] mm: vmalloc: Offload free_vmap_area_lock global lock Date: Mon, 22 May 2023 13:08:48 +0200 Message-Id: <20230522110849.2921-9-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 13782C0015 X-Stat-Signature: w4y675u5hyrwn5m1k4uqkxze4qd4jitp X-Rspam-User: X-HE-Tag: 1684753739-942994 X-HE-Meta: U2FsdGVkX18oNxWTfS5UrjNKBuWVolNgt0/7Xj9Uu2zA60MEnD27kLq+TNrC6CFcS9/INiKERy5xpdd73SMa3++8exkDe/WekDkTLZ6Gi1FMtojG1Aj9k9MOh9b2RNc5b5BzK4k8yJEA/Qr31WZHC9404E36vJ8+I2eVVgs28IcFBYr30lywYbdgRfrlI6GBnSn16wnOZk9UQpLbU1MFotaG6B+8dc+gPwjFIxLkuyYStaPIqsKJOSL8MKyui47mVhCnaHrJsW1Hl7eJicfqlyb43PKw/mW8XYqM9ZR+UA5ToSnNnTLrUXbcWE9tUwd0rKFHF9RwuoibZLCHsNsmvpuHzj6kt9UU2nQltVC7mxLvHgqdrWinFppf0nYh90fTicyND983AhNjkNqiQXbPP1xDq4eTUHe/gMO9zkvR+UbmfN74teVAikvxFbnGSiS5nexUly0vUbVA4SGAlj54N+DsUIuRUMyTOzkpfg3OyEYk2g2Ap0LUPaBxHoRY2pMIlPr34eAnvIXWKhMNJlKCmMldygPZB11a8J5BXfC1VlMZgdq87+CCFHCQUpaOAsKK6XI6euwCArtRQhqyaC2eUWA5edn467i6aiBZpgQbid/syNhOAd0/4EHeDt5D+UjqNS32Jvpr2NA9npdk7jNm9i04Atn7RX/FneW7j98SI1ESuWnPuPX1AXGq6o3MOUdTm5zxEhUBal5o5BbRW0oLZS2sXrKRRhvv/7AwW7oWoCzmaalw6B4eSZFrBcglpmOMWpKZA5+kR9il5eqeSbZsQ2R0yh/Cj6XDCYG1nGmVjlO3m8tTSnBGTOHmYRCgpDZwJrMxM2kUZDGnMYFga/mOFzvxQI7xAgHbH6fkMxAUYGRof+iaPh/ldijcxih9Mg1ZFh4bphVH5NX2cFeYS1QBhHJf1korjN58NeTbEuKCsGMP7MC4OLkdSyPOVN9VCwRLbYgIDq2WxjGU2jrlN8X qwFXo2nR BA9k/pHRsMFLJW10XICWG30wqdBh3ufXHb3wYKaFaTQY0TZOET30R82PGOYCWlOTrHMyXyWkXVb2gMtnunyQRuT0rntM3iq4vhRNwk2nWVKWfkNX7fgJQLhrLpUQCFZ5mAtiqILQuIILLXcZhCCeb40ZHaHGwJOoEnz68M3iAlIp9uZBm8EmGdlY8fv01zzLASFdSyf1qEzVtzrPYH6BFmsxyEM+1jlpn6iOKZxGPTqZA26SRNe/6VRYIay00ZhYqjeIxGUHdZ4o7anFqhYNy+KCvbNObpi+8ycDW4cw9ChwjyUtgwgXVunFIsV1sMOxZMqPsbebaToVpsbuIaYG5g9q7n243rliAm+keiQl1ra2o5HjtyXjskLAgEpJy8/kwo7WapvWIPqPJF8uBjNanjxH5LpZO68JKs9xJS1HE/PdgayG00qnOucTaojTIM9421Ty6MOM7Q1/kJXyfj7lbWn6jO6p1gTydFA90ZQ6KhvAJ5RM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a fast path of allocation sequence, that consists of per-cpu path and fallback mechanism which is used when a request can not be accomplished by fast track. A fast track pre-loads a chunk from a global vmap heap directly into its per-cpu zone, following by clipping the chunk based on allocation request. This technique allows to offload a global free_vmap_area_lock making an allocation path to be serialized to number of CPUs in a system. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index fe993c0561dd..8054b8bf6c18 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1642,6 +1642,93 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node) kmem_cache_free(vmap_area_cachep, va); } +static unsigned long +this_cpu_zone_alloc_fill(struct cpu_vmap_zone *z, + unsigned long size, unsigned long align, + gfp_t gfp_mask, int node) +{ + unsigned long addr = VMALLOC_END; + struct vmap_area *va; + + /* + * It still can race. One task sets a progress to + * 1 a second one gets preempted on entry, the first + * zeroed the progress flag and second proceed with + * an extra prefetch. + */ + if (atomic_xchg(&z->fill_in_progress, 1)) + return addr; + + va = kmem_cache_alloc_node(vmap_area_cachep, gfp_mask, node); + if (unlikely(!va)) + goto out; + + spin_lock(&free_vmap_area_lock); + addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, + cvz_size, 1, VMALLOC_START, VMALLOC_END); + spin_unlock(&free_vmap_area_lock); + + if (addr == VMALLOC_END) { + kmem_cache_free(vmap_area_cachep, va); + goto out; + } + + va->va_start = addr; + va->va_end = addr + cvz_size; + + fbl_lock(z, FREE); + va = merge_or_add_vmap_area_augment(va, + &fbl_root(z, FREE), &fbl_head(z, FREE)); + addr = va_alloc(va, &fbl_root(z, FREE), &fbl_head(z, FREE), + size, align, VMALLOC_START, VMALLOC_END); + fbl_unlock(z, FREE); + +out: + atomic_set(&z->fill_in_progress, 0); + return addr; +} + +static unsigned long +this_cpu_zone_alloc(unsigned long size, unsigned long align, gfp_t gfp_mask, int node) +{ + struct cpu_vmap_zone *z = raw_cpu_ptr(&cpu_vmap_zone); + unsigned long extra = align > PAGE_SIZE ? align : 0; + unsigned long addr = VMALLOC_END, left = 0; + + /* + * It is disabled, fallback to a global heap. + */ + if (cvz_size == ULONG_MAX) + return addr; + + /* + * Any allocation bigger/equal than one half of + * a zone-size will fallback to a global heap. + */ + if (cvz_size / (size + extra) < 3) + return addr; + + if (RB_EMPTY_ROOT(&fbl_root(z, FREE))) + goto fill; + + fbl_lock(z, FREE); + addr = __alloc_vmap_area(&fbl_root(z, FREE), &fbl_head(z, FREE), + size, align, VMALLOC_START, VMALLOC_END); + + if (addr == VMALLOC_END) + left = get_subtree_max_size(fbl_root(z, FREE).rb_node); + fbl_unlock(z, FREE); + +fill: + /* + * A low watermark is 3 pages. + */ + if (addr == VMALLOC_END && left < 4 * PAGE_SIZE) + addr = this_cpu_zone_alloc_fill(z, size, align, gfp_mask, node); + + return addr; +} + /* * Allocate a region of KVA of the specified size and alignment, within the * vstart and vend. @@ -1678,11 +1765,21 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, */ kmemleak_scan_area(&va->rb_node, SIZE_MAX, gfp_mask); + /* + * Fast path allocation, start with it. + */ + if (vstart == VMALLOC_START && vend == VMALLOC_END) + addr = this_cpu_zone_alloc(size, align, gfp_mask, node); + else + addr = vend; + retry: - preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node); - addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, - size, align, vstart, vend); - spin_unlock(&free_vmap_area_lock); + if (addr == vend) { + preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node); + addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, + size, align, vstart, vend); + spin_unlock(&free_vmap_area_lock); + } trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend); @@ -1827,6 +1924,27 @@ purge_cpu_vmap_zone(struct cpu_vmap_zone *z) return num_purged_areas; } +static void +drop_cpu_vmap_cache(struct cpu_vmap_zone *z) +{ + struct vmap_area *va, *n_va; + LIST_HEAD(free_head); + + if (RB_EMPTY_ROOT(&fbl_root(z, FREE))) + return; + + fbl_lock(z, FREE); + WRITE_ONCE(fbl(z, FREE, root.rb_node), NULL); + list_replace_init(&fbl_head(z, FREE), &free_head); + fbl_unlock(z, FREE); + + spin_lock(&free_vmap_area_lock); + list_for_each_entry_safe(va, n_va, &free_head, list) + merge_or_add_vmap_area_augment(va, + &free_vmap_area_root, &free_vmap_area_list); + spin_unlock(&free_vmap_area_lock); +} + /* * Purges all lazily-freed vmap areas. */ @@ -1868,6 +1986,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) for_each_possible_cpu(i) { z = per_cpu_ptr(&cpu_vmap_zone, i); num_purged_areas += purge_cpu_vmap_zone(z); + drop_cpu_vmap_cache(z); } }