From patchwork Mon May 22 11:08:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5674DC7EE2A for ; Mon, 22 May 2023 11:08:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AFC4D900002; Mon, 22 May 2023 07:08:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A846F280002; Mon, 22 May 2023 07:08:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94BBB900005; Mon, 22 May 2023 07:08:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 80D1C900002 for ; Mon, 22 May 2023 07:08:56 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 51BA3C0CE1 for ; Mon, 22 May 2023 11:08:56 +0000 (UTC) X-FDA: 80817618672.22.EBEE8BA Received: from mail-ej1-f47.google.com (mail-ej1-f47.google.com [209.85.218.47]) by imf10.hostedemail.com (Postfix) with ESMTP id 7916AC0005 for ; Mon, 22 May 2023 11:08:54 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=VBUTLfKb; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.47 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753734; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C3POfPMkD0T4i+gkKz2qkPU4t7/WZTiyT2qCdNLU280=; b=nnc4hHn2hMtKRnYo326LVq4K94Vkz/1BthJjeEu2HiURQ0tl6euAlbKzMeneLbBeh3zo0W iFLE0OCffLJx1uXmytLH6EpLFDoYumNyqoVXSxnJXZHSxg9rVy7ZmAWXAic7vRplL+BL0j iCPCilrtuS8q08I5wt9/XMRtFMAOrX0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=VBUTLfKb; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.47 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753734; a=rsa-sha256; cv=none; b=LLTp69JZoaU46EriGQJ5SDOhTXGM6R5SQgm2ivGYzx/3qVKP2OrHB0kZlku1f48UI9zldt bNducsJQp/GxFXSBVfnCp6qTlY79GXlBSTsxAk+1duylr9paBDbaYf8/EtJeWI9FPiWPVQ DIMrcsz+Hc6B7XJe/bKUEEpwYVBMAzs= Received: by mail-ej1-f47.google.com with SMTP id a640c23a62f3a-96fbc74fbf1so278333966b.1 for ; Mon, 22 May 2023 04:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C3POfPMkD0T4i+gkKz2qkPU4t7/WZTiyT2qCdNLU280=; b=VBUTLfKbApHu6QZhrVrOCB3BjALZb8HbF0H20xKFhc6s2ddiN7suwTGMU0bL+g+sYn RmbQ1cww5mUZenVFKfKM0obeU2G6DDRfd8SBMnH9fmFpOyakPo9ae6ksAd2XakoCBv0E 7Ov6b3cng1MnPemi03HJHLAXu9L7D+fD1mqdF1IW9NYQd6lcGKgDypewZcwaRLFcrhmr yjR9yAKjvtwtrmWZi3nCGabZYIW/EQNSzPSvJbp0e3+m5rd5yon1lw8lX3KGm/ik7TPO pykNacp5zTZge80vp2rYbryAhbquVFUZS46oKag6NjrQQcMLsIE3M8wIwP5IFy1Gt0Ul IcQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C3POfPMkD0T4i+gkKz2qkPU4t7/WZTiyT2qCdNLU280=; b=GK7GQM7h9CJIz1XxDttujFQAwMcl6BYtBO3rLt8OPEEw6H12AFoztoDqtWY/I7XpgM +OiMuCGVj6wKgv0xLAUF54k86971iE6DeNHErlK/+SGIEfBk/J+vUaQ8Z3/B0tCA7Ey/ fFRDsH0j4KVeBnBPn2pqIo5qZ2QZBjBaTcN863UXoJ+wqzjax90KtiMU1US7/xUFAK8m vAQoRsKbEZwDAMe33L39T5qMXyYvX+0HQ1R4UkLbva+/I007bolZs1FHFc9zxcq7SQGf MKBepcQyIHX2cdxeRcd/ul2WNxZ5JvBrxH+SGAiQvllAya6RYXk1OxtqgTfrGsRqeZ0F Qnng== X-Gm-Message-State: AC+VfDzVqsNPND9YjkT1O+iITYfHHVWJdUxcaJuAHmYqVgf3i7j9eBTt H428pzZpN9t+ZOqtB8IU796BvBbUIxagJQ== X-Google-Smtp-Source: ACHHUZ7sSj0PR0+CyXEW2CQQzfPJq5rycOMctGBSYZ5gIV+Lpig9VckuJmCX23ieD4b+A7HcD+ROww== X-Received: by 2002:a17:907:1c8b:b0:94e:48ac:9a51 with SMTP id nb11-20020a1709071c8b00b0094e48ac9a51mr11815922ejc.4.1684753732807; Mon, 22 May 2023 04:08:52 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:52 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 1/9] mm: vmalloc: Add va_alloc() helper Date: Mon, 22 May 2023 13:08:41 +0200 Message-Id: <20230522110849.2921-2-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7916AC0005 X-Rspam-User: X-Stat-Signature: di77r4qi4rx7yshunker3x6k5jayb3b9 X-Rspamd-Server: rspam01 X-HE-Tag: 1684753734-998636 X-HE-Meta: U2FsdGVkX1+oOL38R0pfIFfAhtJkEljFW9G8YxZ5UsZ2Gl6TM+SC0A0VQTaFXnDuxP4R1iIpJyGnoXtyMXQu9P7GaVo0GVRazwvhJE/za4GGraKVgCEovXjscF+xW39BshY/HpPQlJv2maCq38CrTSeCqc+GpVVMdGNGXxYMQHbeejNzd57d1qycWWbglYvgsaQFTtvkmNkBVbzAuaBs/8hxuE0DBUA/vvKkcS+DLZ9vnoqsXreGTKGBvUeMBPVMF1fKiD/x9zaIClkZXlxTvg9fe4Lm9aEiAbHho3HS4lTG40ahwpPWQBYLmQaTvfWVpFiIPnx6c4XDIG3mVntZzd6rV02Y5uh69f++KuRrbeUaSmWAlaWd2h7K/jlCXyTMJl6XeRGTpoUOsJhXOGBHphkYe1MXW0WVLdGpeRAhKnh9no3M1Yi2nMzpfCXU2KKJsqLC/I39U1Sj3ZwclvxOWY5xtG43wn9PxRAfobS2YQ0/+Y9P8MZpWfrc/mOwIpCq/wr8jpvQB6t8EuCqJLYw1C7OzJ+BCQ5BlmPYOUjD1/cd05mdlpVO7LSSwrx56xnq5PJkICMbfh7DyZel55UCClYEaylgqcASsbqqVYyNTCvrbl+Q52FSAExGMMvvF1KpTtYl5o/PUo7e+32tE7R9VI1S4XXbeT0s+xH/P9H2zKESzKakMFYON0bABq+C4vNAUHcqQ/t7UBfGMBE0+7o+ZSlZFiClBfZSuC+shR9wWE9oNW1x2zmLMK6lak+hbuoJVnPTr1rmexQME31XKH2NylXOwxrg5UAUvSS/XFJOX/AmJ6TfESsb9qWr2WKfz2atdRV5p1M4UsJdrbyawHD5WS9zzS0uRPrbN+zeC/sieXgv+mbcUxBSLSRxNEk5UrtDH1yIp3kRlfqvVIsjVThxmFIEJMExEFnq2XAJUsx1jwZ7HwNsHMCBbAMp49SUCSWQDKDaB3+aiKxKmdHvzf8 Ph+IJd/v dKbIhdCIjNqzAWbfEgYgM3YEN+hXMuz4pBPNanZ2hQ4rvLRgldKNhgmpeNdrZMAAWvYP7HTAvL6A59G/MeqFVEeYXm62Tsw5c3g1qvLJhrhqkIPHeYU3XwToJlghc2G3Bs/kKuEr5mrYWVAQjH3KZEcqH6tdIawASOXaPVyvF7Ibpn0JfXgpAkr7h5vipvnetT+3W0FPkJI8kyN+3hZE7rs27Zv+N0whQNELAxjGweBLnSjByXmyRZo/r/Tyy+zIqhzKS+uMgAqwvGhYj2b4/Mhd3ti6kq1SwMPUiNztj5efhUjb3KuAOjGY9BCipd3iqGeaH7iIOa2d/oWsMWRJrXX4H+zGRwI2cucZOT+H/XBEJuH3fI4gwIdoMZiRvpt4P0Ekxq1lahGPJ2yhlN+lsqcrhuLmZMI48pEjZjn3au9q9voJQHo/Ntyx7o2Gtjm7S1B4XRLZ4iM4q0yAPdxxssrSNRlBAWM7h6FWtBi2pKzu1irE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently __alloc_vmap_area() function contains an open codded logic that finds and adjusts a VA based on allocation request. Introduce a va_alloc() helper that adjusts found VA only. It will be used later at least in two places. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 31ff782d368b..409285b68a67 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1482,6 +1482,32 @@ adjust_va_to_fit_type(struct rb_root *root, struct list_head *head, return 0; } +static unsigned long +va_alloc(struct vmap_area *va, + struct rb_root *root, struct list_head *head, + unsigned long size, unsigned long align, + unsigned long vstart, unsigned long vend) +{ + unsigned long nva_start_addr; + int ret; + + if (va->va_start > vstart) + nva_start_addr = ALIGN(va->va_start, align); + else + nva_start_addr = ALIGN(vstart, align); + + /* Check the "vend" restriction. */ + if (nva_start_addr + size > vend) + return vend; + + /* Update the free vmap_area. */ + ret = adjust_va_to_fit_type(root, head, va, nva_start_addr, size); + if (WARN_ON_ONCE(ret)) + return vend; + + return nva_start_addr; +} + /* * Returns a start address of the newly allocated area, if success. * Otherwise a vend is returned that indicates failure. @@ -1494,7 +1520,6 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head, bool adjust_search_size = true; unsigned long nva_start_addr; struct vmap_area *va; - int ret; /* * Do not adjust when: @@ -1512,18 +1537,8 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head, if (unlikely(!va)) return vend; - if (va->va_start > vstart) - nva_start_addr = ALIGN(va->va_start, align); - else - nva_start_addr = ALIGN(vstart, align); - - /* Check the "vend" restriction. */ - if (nva_start_addr + size > vend) - return vend; - - /* Update the free vmap_area. */ - ret = adjust_va_to_fit_type(root, head, va, nva_start_addr, size); - if (WARN_ON_ONCE(ret)) + nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend); + if (nva_start_addr == vend) return vend; #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK From patchwork Mon May 22 11:08:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05817C77B73 for ; Mon, 22 May 2023 11:08:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F18DD280003; Mon, 22 May 2023 07:08:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EFA08280004; Mon, 22 May 2023 07:08:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD275280003; Mon, 22 May 2023 07:08:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B95A5280002 for ; Mon, 22 May 2023 07:08:57 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0DB38140D20 for ; Mon, 22 May 2023 11:08:57 +0000 (UTC) X-FDA: 80817618714.15.E1E3E6B Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by imf20.hostedemail.com (Postfix) with ESMTP id F38F51C001A for ; Mon, 22 May 2023 11:08:54 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=UI0Yt0lW; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753735; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AqE7n0km0cT3SoKgXn9+9ffSGk68u7ZexKpVeyjeuP0=; b=T4ei+JsQJ+0Tx+OC9WymVuvPgEn2G21AHsPn0bLVyajHH/x9N6Cytbphu2DqphlOoWBNvb D7+54vdSViOXRTjUqLqS1ZxsD39/Wpn6TZ4adpS+r9Ggp+6EZSC4HEz9JMb27U5/Q4uK6Y ZBH14BaL2Ho77aPf549nZvjAVoMlHPY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=UI0Yt0lW; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753735; a=rsa-sha256; cv=none; b=gwtudwNbF8aUkijzooOFycE8A2konx8XgvvMhOBR+BMIeMUGwO8gT4LZ2PD69l0tFa/e9s 5/nU75DAAfNFdEg9QyN2QAbVXazukzdNLbC7PxyQWdYMJuCn/a16WcDth5Bo0MW8nrPruq txNoHFKC4mO1WKUjUeuvcEc0AXHYBW0= Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-96f0678de80so1097955366b.3 for ; Mon, 22 May 2023 04:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AqE7n0km0cT3SoKgXn9+9ffSGk68u7ZexKpVeyjeuP0=; b=UI0Yt0lWDXh9ljMMkPAajAGV57h0505ib1ttO6nC+S6/ox2EoI9VGZd7+i0KyUT3Iz UrOeQv6HeCqlYMLmrO9xLceNit+eAAhp+ayorSX9GWMqWetCfy65fNLgsBYIBMUhhJ2L U75zeNwLejw8wEtPfmFV0nYxaTwgfXdcMVWerdFeHWF2BfssxPxEikaABdcefX8aVmFB OzcafrJAiZhR3U1i8vQurAWGDU7VgmaPUGDQ2nyXctKGB+RLSHtg6lFIAG/yBViPwShe HnTgtWcZBRbZXXW3R63z1IHpNcsR0E7wzcXRYrATiECZ/KAKp4PAjxmjbmz2ChABim6V aqqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AqE7n0km0cT3SoKgXn9+9ffSGk68u7ZexKpVeyjeuP0=; b=E2W0Gtl23cvqtsGDAeMhkFbeIOvpf31lKeZQdCLVrSVhIuwId9FDPQngH1JQUxaf3T F6yWttW+69eAJWdR2ugwsMFbYqGmorXzznOJY9KS3awBUMiBIBLdhx/YTkqF+Frc3HEB a06/VWIvw564f8SWhsBr3y0GdxnQbkju+anVMt5jTRrVJEGdDpUs3jaUlc9k2m8VseQh nX59kNJml+Bsn3ckbrpedjDkChUkqqC6WeajZvpr1321uexuBSw9Z+Yu5j4SregdoujG 7eAt1iJfzFJzEbyxvDAJpOQPxBUtXpPvJlMiOq+onS3nPBc0MVnfJDLMbH9npm6J3bQv jfCQ== X-Gm-Message-State: AC+VfDzqzw4yNaWoqKE0V3ah73KvxxfY/n/YWun1LndNVtLR2EvJQqC5 J08QF8+p3VL5cP6q3/TMuPEJ0IjI927/PA== X-Google-Smtp-Source: ACHHUZ6msUCypNMoSNhIQMunC9pTKnBPX5E3jFgX9C4tJ4LV2wmznUsQq48JjiudHWKxzuBo6hrzCQ== X-Received: by 2002:a17:907:2d23:b0:96a:928c:d382 with SMTP id gs35-20020a1709072d2300b0096a928cd382mr10346599ejc.48.1684753733572; Mon, 22 May 2023 04:08:53 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:53 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 2/9] mm: vmalloc: Rename adjust_va_to_fit_type() function Date: Mon, 22 May 2023 13:08:42 +0200 Message-Id: <20230522110849.2921-3-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: F38F51C001A X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 14zf8egr3jnfa5k55uetcrqta5nf7za8 X-HE-Tag: 1684753734-350743 X-HE-Meta: U2FsdGVkX19A6JgoZQNOquF3OV7I2K6P/VJkgEWeSE6rgUBL7Obfic8kWZSAwdkengM/ShWWA8tD/iKrM9AlPZJDUmFsmh1TM44b8fUysgCQWVaX7++0JWxGOIpEALOvDakkeVQHNZKbQEszV/LA6zBgKkbY+mTP7T3DodQVS+hpETDx0knZmVffnU8Ylfpfk4KH34HFoezhmPDMCSqmAQL/Sxeg/v/3JfWjGTOlnuwjxBquemgXvo1Sd0+tUeJpAFkqOAw/hnXtFe8kZQyHIaxhcLi88jj+iylfwb35Q5pqD6DgObIscL1K3hjeQ7YtZF6MgPf2IEphP8VpmPRfVLNftVKN0oUNpcu0Xzy5Cjcxp97q0Mn3sLJhRzR+A0S2FfLBPcMxTBQxZ5V2naLYvvpBsX5wp0Ovd8FY9Il4uhCmB36wRAlgAwl3e2L1ZmkXDOlboIZP885xx54cRpL3s1WEZepX76tSV910M6sbkNJg5DA3DYLn+nEjnaZwk0mMFzSnkHQo/6mjFdBnloLDASB84aQdMFQbgyW14dYa+hHl5GbxMwGytOtPcvSD+36mGvEaZHqt9dRrIjjbtrFfAi8oGlAeaSUEnpLc/tW4krSuUQNjHjWwuMwFWLxrj9nWylarhxL8C+yGAdw2voLtrSGijHFu7h0R1Qi17SlycJQbO5Z67YZ3V+m/7pjQ8sBlMRPGDH0KjfUFRPBruocswYbEniLyyzbXuxB0EuFRi1m7ZzmqY4ilKFul6Gg1icDq5572lXVYQpmEFtutzWoTTdrOzojQ7JXmm7If9HIo8B1JMnlxNHCfHSYUQb+NlwxIZYBvyJAHmuSjEKo8elPlC9XZb7BHO0DcK+1Zgoo3vuLWCHhbHimuwnosySZ5zDeKRVnpdifAux1hWVONofDZGmtRA73u1bjcfU4Ixby2SjE/8GrbT1oGgnYFT9CR4xgn8PhcRV+QNqOgXrKIOIW XNs3czxF OdTbsz1vwYwGV1JW70k5oPypAkuP0WnuFGKxGR5pO+gpiepWT+sWlv8W8qBg/CsPBAVZfGrICZtm12cUt5q0UiHV/WTgdfsPxK/RrJEfQ/hfmiRhegawHfXGliJLw3NVjb1LUUOC8kDNm49p1Dyaps++QRAgS+gc+sPJz9A1DS2mPAJX/rnrL1zz9JOO1jBQ6RPva7HISPJIVgsom5dUUV3mBkeFjyqs1Pw+HdrWrkZyYm+k3klecAYusdG96SNCa6Hrd7wQxOZ5UzCyBcVFPLkY/MyG9efUhMIEX9icv6E0dSIPHaJozVVPMH2fwnYXLvK5wRgvOR8Tnhmk7HTOi/NTffclRJ6/VbY7NBfrbLU9a4DV6bmui8B4b8R03KX6jwrYtWHVKNorlSb2+HRt0v8zg3wVtGW41mFDu1e9UZQ/Dosj18VDyOILVW3iiqvnlBlgXSOn7UCv75horOC5VPR3CerOHwqELnb7MesfXBgWLRSk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch renames the adjust_va_to_fit_type() function to a shorter variant and more expressive. A new name is va_clip(). There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 409285b68a67..5f900efec6a9 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1383,9 +1383,9 @@ classify_va_fit_type(struct vmap_area *va, } static __always_inline int -adjust_va_to_fit_type(struct rb_root *root, struct list_head *head, - struct vmap_area *va, unsigned long nva_start_addr, - unsigned long size) +va_clip(struct rb_root *root, struct list_head *head, + struct vmap_area *va, unsigned long nva_start_addr, + unsigned long size) { struct vmap_area *lva = NULL; enum fit_type type = classify_va_fit_type(va, nva_start_addr, size); @@ -1501,7 +1501,7 @@ va_alloc(struct vmap_area *va, return vend; /* Update the free vmap_area. */ - ret = adjust_va_to_fit_type(root, head, va, nva_start_addr, size); + ret = va_clip(root, head, va, nva_start_addr, size); if (WARN_ON_ONCE(ret)) return vend; @@ -3979,9 +3979,8 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, /* It is a BUG(), but trigger recovery instead. */ goto recovery; - ret = adjust_va_to_fit_type(&free_vmap_area_root, - &free_vmap_area_list, - va, start, size); + ret = va_clip(&free_vmap_area_root, + &free_vmap_area_list, va, start, size); if (WARN_ON_ONCE(unlikely(ret))) /* It is a BUG(), but trigger recovery instead. */ goto recovery; From patchwork Mon May 22 11:08:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 787DCC77B75 for ; Mon, 22 May 2023 11:09:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D083280004; Mon, 22 May 2023 07:08:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3821C280002; Mon, 22 May 2023 07:08:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 18DA2280005; Mon, 22 May 2023 07:08:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E3AAF280002 for ; Mon, 22 May 2023 07:08:57 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9FC50C0CE1 for ; Mon, 22 May 2023 11:08:57 +0000 (UTC) X-FDA: 80817618714.13.C740EDA Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by imf29.hostedemail.com (Postfix) with ESMTP id C86A412001A for ; Mon, 22 May 2023 11:08:55 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="GkR4/dm/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753735; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sNtfrgi/quI4TC521AyXgwFWeKCoK/ixdzMUo1Hjyag=; b=tIVKtvKipIvRDEZEARL8Vh1O6yt4eMYSSzItQseL6/fyUll0eLXMXIzkDsJ3WTsajv5uaw VqbofBJmHoPYrCt4LxP+wh8nk8kD5XvqAxGCBnFiXQjWs5hqJquX17Wg0mcVpPHhS0vsLe 8Bc7VAg9UjheJI4PRyOfwap75ioANLg= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="GkR4/dm/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.50 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753735; a=rsa-sha256; cv=none; b=Y54hnbKg5/9i5r6a2gOLwais7giRAUEXXFTKxKlODzUVydmFSeGiJg9fOGqjEz0LHHQb31 48YFroqcF2LwEK+UNj+BfULf5uR8T/wtHpxB6DoJVt7HdPA+i2F187C+eACVl1qyxkgTdU qIMoQu17fe9Lxj9MUnF2a/kQSvDrfIQ= Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-96f683e8855so488847966b.2 for ; Mon, 22 May 2023 04:08:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753734; x=1687345734; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sNtfrgi/quI4TC521AyXgwFWeKCoK/ixdzMUo1Hjyag=; b=GkR4/dm/njRpBQa+KVraZ+7JOlNk7shw4BEphYd/T4MuYuRj9UeHx7STRIkE9qZ3ZV 2oXVDGHFmqIOsQbp8tUn9Y00B46Baxi7Y8dneLOn6DV42d85pL1sEsY68+QnjQ1f2vmJ aDPoq7XcD9rLqsdzZWq3ANly5HjVSjJ29W0p5d8YhCo15pKjujxSAlIMECHBsZDAyuMN tLgaKTuxmHTfb/eGGN0rAdS+/lKXxYAVJyynl/B47Xw/ZEaCyR2lt1lpmnS2FpLh3BXG MgpEFyKwNZW3Q2xr0xt2mb+dk5JSHw7+52EgYFgJevxmwbWAu2YUPoOjuE6sjvnpUmYm ZEog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753734; x=1687345734; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sNtfrgi/quI4TC521AyXgwFWeKCoK/ixdzMUo1Hjyag=; b=DOqjbxqCj9aK8CO18cuqZJSIi2K+wbQRGczOP7CJdZdR/4Qq/ZPFYwcV68YZOJzRoH qUpbHqYnmy7krNeaidqIMz+Ax1Yz7MIgAnWbWCu6kmE4IMMvIa+l/akE8r/5XZC4v93A BxtaajxxEGPtBwhj9H5SGqAmliwej6OOFQNnlqHgNi+cvFMWvuet0mw5BD+zccz2c33j Jq3zhsStD2noscgYLPg0+R/VHgd8NuLM8R/6jGH0pP7JarmHwWhXXOTWlsTcfXP7pI2e 6BUptp9CGi6lljkqDh69U0Lbb08jLPfbqdqu/iSI6cH5GrHSXPNSQD/SNbOo/CTac0Su IJUQ== X-Gm-Message-State: AC+VfDzi5KWikHg3bYHFvbI/6yt7bBuG6KHbFGozmxguZAfgik+YYKZl 2t86l9ru4xo0QPkxt8LN42zJulB3HVGihA== X-Google-Smtp-Source: ACHHUZ6CkjutlmxyUmi96KfbjXVfsWOUvXlEtbViniM+ZZDlWRxo42QLeXCAxHA9y4X8Xg8XpIQzUA== X-Received: by 2002:a17:907:d21:b0:965:4b43:11f1 with SMTP id gn33-20020a1709070d2100b009654b4311f1mr9711913ejc.3.1684753734338; Mon, 22 May 2023 04:08:54 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 3/9] mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c Date: Mon, 22 May 2023 13:08:43 +0200 Message-Id: <20230522110849.2921-4-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C86A412001A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 895bm6bbwumciyoubt3aqbo8ysqno1mn X-HE-Tag: 1684753735-228684 X-HE-Meta: U2FsdGVkX18hmhigUOZiPUWmHuc2BQAZZdsw60GBIt9zHYfbdh0L/iOBpILq7s9mw55bHmQym/uiEl3S8G3KuQjz+tKBHKhpyNeDd3CPvh5MX2U7AUgYFtJluGlsNuKNpvlr2uCCCFtjdtoYU/8WjZuOamNou2pYWWfQtD8Bz9n9YF96roslkH1vmNMkxcj3p/fl86rVxuH60UEFlkE3mD4hx9lUue6aUd7hC0Jwwe9Of94s1caJsIJTOUO3s+Me9FKNnLV7YqbgB11YpHfNv+lkOV9H0QcY+g5cuaUpR5VzXVS2bEDuozIz/H1+96VUL0VC+0PYyIEZwmINTHni8osp5qO3u//ZMoezdTk/L5F3r/icPHbfZo0hzP2GglVTKr7C7RqlSPPc7Bja68ymmKfronLYOQY/QHBMHSJP7BiGXP90zCBCNAqPupoTGPsarLiajueJdbMo9nrC3QVLxOHcRYTvXjmijHBZ7D+CHJACNrRUmNcO+/y/KsDckEu27UNk6fj1K9o2Nqk67dHIMI21Ob41PPsinBYbvvQqmV5+oKMwH1e0AqsFhlAW+C51Ahmv3q5CcFizk+23bMp2aU3eIXyCVBrZkPKaq/rs0r50PX67jpjw+ogGjSCsagAf2iq5uIPMPCk75gAamGkNI+8AVGptbNWcxIeeWAXcMxHSFjue7FPO3ugA3UVEBkRmBcjAMnhBbokgLfu5xWRvFhTgd42QZHWXRm0EDxMQhTEP7Ti1BI/1OqJGqExUfzx5Zt127qwNd4d0IT/u0TXRwaFMBmcVEDZYwOM3chgc4ZnoDe+MGYzcsx+bNxlENQ8Arh/OXhVHTM5Z36RXgkHaKqERbx/WxlH5NV0SUpyNKnPnXN0baBIHz0s8iUklbIak3k8IqeSUSo4YVdg85Df4enOInYI5J8MYUNUqVUbjxD95F7cF9xp/XRByJEvkIQvSFAZxKYULycPSrKT825U Th6XS60s vntVrQxHvK8nyV8BmLhk07yxsfSGqyAlCtt43fRfKyekpwQZN7Tbcl2jmza/Q2o/q/j0DT/F18Nu8UP3RBFo4LgDpMf2TcjU1cU7qsxkJ9VdH4AbP/3C9QaHAl665llcJGHY75VyNhebyimB03px6lU4L/C8E1EVQw2N6I5EtH4G5EmOqL1AZCNi2L6aqtptjbVc+k20CrrnUQtZVXQ4ldzVz248dwUL1PV6iIWRSPjHbZgE5/ovjsMTrH+E2cGzbfItBASpkLvygjCpOeYXij3qGn9WnSaQcwOnWE1ecCA6XvwC5WljiX6+gAJ0rjNyvT/XKInPZa8YBUsCcRQv0Y0EkFFm742FtU5fWRkWjeY0Za8m+AqcngQ2x6NVaE48J0OCgPhnv3bNesKgGpJZ6nkCTXWV5G2I4eI5mzIzTM1prgN6l966/q4GJ0l8Wi3eH4hRqzFIe1Ng8LLTxjO3Tjqt8CQOotWOt4OZaS8W/7xgZu5M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A vmap_init_free_space() is a function that setups a vmap space and is considered as part of initialization phase. Since a main entry which is vmalloc_init(), has been moved down in vmalloc.c it makes sense to follow the pattern. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 82 ++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5f900efec6a9..19dcffb0d563 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2416,47 +2416,6 @@ void __init vm_area_register_early(struct vm_struct *vm, size_t align) kasan_populate_early_vm_area_shadow(vm->addr, vm->size); } -static void vmap_init_free_space(void) -{ - unsigned long vmap_start = 1; - const unsigned long vmap_end = ULONG_MAX; - struct vmap_area *busy, *free; - - /* - * B F B B B F - * -|-----|.....|-----|-----|-----|.....|- - * | The KVA space | - * |<--------------------------------->| - */ - list_for_each_entry(busy, &vmap_area_list, list) { - if (busy->va_start - vmap_start > 0) { - free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); - if (!WARN_ON_ONCE(!free)) { - free->va_start = vmap_start; - free->va_end = busy->va_start; - - insert_vmap_area_augment(free, NULL, - &free_vmap_area_root, - &free_vmap_area_list); - } - } - - vmap_start = busy->va_end; - } - - if (vmap_end - vmap_start > 0) { - free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); - if (!WARN_ON_ONCE(!free)) { - free->va_start = vmap_start; - free->va_end = vmap_end; - - insert_vmap_area_augment(free, NULL, - &free_vmap_area_root, - &free_vmap_area_list); - } - } -} - static inline void setup_vmalloc_vm_locked(struct vm_struct *vm, struct vmap_area *va, unsigned long flags, const void *caller) { @@ -4271,6 +4230,47 @@ module_init(proc_vmalloc_init); #endif +static void vmap_init_free_space(void) +{ + unsigned long vmap_start = 1; + const unsigned long vmap_end = ULONG_MAX; + struct vmap_area *busy, *free; + + /* + * B F B B B F + * -|-----|.....|-----|-----|-----|.....|- + * | The KVA space | + * |<--------------------------------->| + */ + list_for_each_entry(busy, &vmap_area_list, list) { + if (busy->va_start - vmap_start > 0) { + free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); + if (!WARN_ON_ONCE(!free)) { + free->va_start = vmap_start; + free->va_end = busy->va_start; + + insert_vmap_area_augment(free, NULL, + &free_vmap_area_root, + &free_vmap_area_list); + } + } + + vmap_start = busy->va_end; + } + + if (vmap_end - vmap_start > 0) { + free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); + if (!WARN_ON_ONCE(!free)) { + free->va_start = vmap_start; + free->va_end = vmap_end; + + insert_vmap_area_augment(free, NULL, + &free_vmap_area_root, + &free_vmap_area_list); + } + } +} + void __init vmalloc_init(void) { struct vmap_area *va; From patchwork Mon May 22 11:08:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAFDAC7EE2D for ; Mon, 22 May 2023 11:09:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F3D3280005; Mon, 22 May 2023 07:08:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57D9F280002; Mon, 22 May 2023 07:08:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CE85280005; Mon, 22 May 2023 07:08:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2A78D280002 for ; Mon, 22 May 2023 07:08:59 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C93A3120540 for ; Mon, 22 May 2023 11:08:58 +0000 (UTC) X-FDA: 80817618756.24.9D23464 Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by imf13.hostedemail.com (Postfix) with ESMTP id D133E20002 for ; Mon, 22 May 2023 11:08:56 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=iDb9tK4T; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.51 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753736; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=063EFU5/djz5xU31OQ5ZuEX3cdI8++KWedSs+gqqgUKHNVBRV5jy6qg+KZX+Awft5aqy02 HkeqbsT46ssII0Yn1pLWSwpf+knsnqrbLUsE+nY89IlutR6u3s6KBZuQnJ9FV7iJGcput8 78kR+E4ErPqwpEQyzTnMevMrclpoROQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=iDb9tK4T; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.51 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753736; a=rsa-sha256; cv=none; b=4If/iiQXyiFxY38U50yNwAP5b0+zOf/4qIxSo6BvyqIzU5Dgy6pUyiojeT1fcIe9xa0fSE ljRNloc441igOJxvDe8pBWMANfU4tOOJst7NWYj+yE5ufX+kwHVGfQ4U2xM+78y+hkoLzz of6dqiN4kE94vYGyYm1I2mAdJB1SR/Y= Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-965ac4dd11bso1180534866b.2 for ; Mon, 22 May 2023 04:08:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753735; x=1687345735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=iDb9tK4TOgaJNMq61xrRvzxoVUaPP0VvNEnz6NrY+zacf0FfJnwnuypFPmwH5SLNBS GyEOPDWFsssngIu4YWW0FrNovP9ZjqfzkUf/51MOpeMHMX80L0GnQUdqQZEtrUUDgAN5 CqpASsjwkoQ7U63fy74gThmTbQvI9Ie3XaHwC17zsSXEk9lHt40AkqJDIcGVhr55gFza Ab5qRCRkiZfB7QM1qkbc7/2z6/RGesk6zCBFvjSfwFLXPP4gxqH+BsOk1j0UanBK1aWA e+0Ct1vWALCAGVP46sp1nfM2tzxvrxxrGXz0I5hY5eIN0kmUd5nEpcG0c3MTS3rmdd8s WEcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753735; x=1687345735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=NpBr7Aul5t2rE5AGv6l9WKJWiEC8fdmvNYGQSZ9fjdHzTusqcNJBaxcw+0o2hsIxZB L4v1swHOR1pP9p4XHAIpPivWbxXz6Kqi9HZpvNDuM9SZUmKpYZ8hdRsN/88OkVEiioCm uEgJnZep0o/+OhdlY797lsnK4bYlEbn7nzUzrJSuCUBOSN+WjPPi+rC1sxb3KnR7Opzd uhh3U7iqrQoP1WvKobZJoLZ299jsyaeS1vay8HHS+T90TcIE5SHHgv5/bSCH672ZnTca YDy0ycQC0tCuYHSVkbpBqHGpXGSQrPhHa5q43jIIZuYqMjZ3K81A9rWU0jrjoZtLK7Cc +7qQ== X-Gm-Message-State: AC+VfDwxV3SzfmRqiR3P9//fgXxrnquqGln4/K6M5TvdA9qPp7Tl3gOV b/Wemeaw/B1h7XccUiwYa0rmot2EGgx3tA== X-Google-Smtp-Source: ACHHUZ5RSuVJjDArE0w69xzADXBpvX8mfpX9Veg7DqUbJpCMdvLgHaYw03z8mFT9WRUtE0g9JS1nlg== X-Received: by 2002:a17:907:2d0e:b0:96f:cde5:5f5e with SMTP id gs14-20020a1709072d0e00b0096fcde55f5emr3930140ejc.29.1684753735155; Mon, 22 May 2023 04:08:55 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 4/9] mm: vmalloc: Add a per-CPU-zone infrastructure Date: Mon, 22 May 2023 13:08:44 +0200 Message-Id: <20230522110849.2921-5-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D133E20002 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: kz6sopgk95qtrw71pqcu4c7apqcendbq X-HE-Tag: 1684753736-422876 X-HE-Meta: U2FsdGVkX1+PQ2Sr0EYjIyHPbn3mDqTRMRVOtZ/IMVKeU6zVWba9lhmWVL7GhKyjticvJFD1WGbArvprDEh3Gal4bK9a//56IxcWvsaS84FusMfp5jhllRMgPkxQOz2sHfNZ+MG8042tGMrLBkqvIbNyVRbe0x7naMNX/9FF4IzdRYtWHTBgCI9M4INwnG0iXm0FsUvZL+k7+z4vyicl6f+DFY2VarQ+yAcFnLs+Mv9ha/MkXgkV+hmxvzDxI9hJZofLK1vZErk+KAIAJ6l6A858r/6nSpPUJwwP/qt0O7BV9/89mXCFw9KeCwcd0F7UR96pNc/AO6kcgSZOunVDlZust0T97BoRLQPA6JHIAO/MnNdRiYwzC/B0sh9omeF0KhG1E3SbVrQlHIJ7ebveXouCbwOMaqbReB07FBfNnhmAw35zfQeEICKCssiRO9yG8BW7t1EjmQvqgLMGXzyTOUcsjnQDpIgIWC7RoIBI69rgp2AgIOvYKTchP9o3xgH5t5eAitQp3BDVvnKzTw1M2XvthVevoJxEeH1Dty+DLpo08JfVUH1mxRbS4fclm9ThrwFxiof/KhU++Ef6Ro7GHe24EcbkY66Mr9vRAFqnhpebzJfLV3YPpAU/7ILdnbN/D9nIBjyx1/ZvPIKjIRznLNVyTBXuWA5i0977R8zKobXKIlIV95lljCXmV8oEEnRCwt+nyd3ZvaQNFxvgk36B2u4tI1lkxiOeMvlUxKpKlQsDRiagZiBmnQwygkfqotlX8JHhn+zPDsxO5d9yFd0aHkK6fyxnDvKRHWyaSJLXSGW9mXMVBra/h+Imbh8wtdyxLpGwObFVqWiFc4eDq11loMDCseYc1sd8HBI9sLVr5zl6LIEVooI0vmPmpGjN4OlRbHYqDGSPDKPQsLLB6ZkHYueEyyjIcEP9CJxyUJFm0nrDsPI5PdpULAES4ABjnQN0ifEXm3aCE+653VOsOe8 JFmvOfJo XdJcuaWN5PRx+DInIMYY/ZW1/wYNjN+Xd91CZg1FbP81+s+u+yXoxtOHA0rRhaFhepntrLIS/1D0cOTP2oceWKq9R3M1R6fbTGKt9oYsyGyINySRlAIF9S/4edX9To/wjXlWMp3bdbqAzuEKGggrNZ9J/bH49PMV470qwCjhtyPPksIuyHKseB5FxM8tXnTvXASZF8/TcWfD5QRU+uAUJkZCDrDg8hQGnmRJnUftDXneI0AumX3hXFIwQXj7PbAPCDwKCCoKExRCUueXZ7itfcvqRv5PjPZz63v8KJ4A7bbvJN4LYCvs7Bx+AD1JjRbXrTS4UBtIU3J9C36CAV88xA86+FZlmu7i2P3/4tHBzQSic7SO7G4bWqWpRLIfDei4PSCPywFFlRniFREYhYKItwEUG6nni+Wefq/IGHsl4dnvvwqeclPYcT2W0tL+gIFN7FJVE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a basic skeleton that defines a cache entity for a CPU. It consist of bookkeeping data which belong to a CPU zone. A CPU-zone defines a virtual address range with its fixed size. This approach gives a possibility to map any address to a particular CPU-zone it belongs to. Below is a high-level description how it is used in alloc and free paths: alloc-path: in-front-pre-fetch occurs if a cache is empty. A chunk is placed to a free-tree of this CPU. Finally this CPU cache is clipped in order to accomplish a request. A fresh VA resides in a busy-tree of specific CPU-zone it is bound to. free-path: va->va_start is converted to a zone, so a lookup occurs in a bookkeeping zone it belongs to. Once it gets removed a VA is moved to a lazy-tree. It is freed after a TLB flushing. A "cvz_size", that reflects a zone size, is set to ULONG_MAX by this patch. The reason is we would like to follow an old behavior until all dependent changes are in place. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 19dcffb0d563..f6da2590b4de 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -773,6 +773,61 @@ static struct rb_root free_vmap_area_root = RB_ROOT; */ static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); +/* + * A per-cpu effective vmap cache logic. It allows to pre-fetch + * a memory vmap chunk with a specified size. A bookkeeping data + * is stored and accessed per-CPU. + */ +enum { FREE = 0, BUSY, LAZY, NFBL }; + +#define fbl(z, i, m) z->fbl[i].m +#define fbl_root(z, i) fbl(z, i, root) +#define fbl_head(z, i) fbl(z, i, head) + +#define fbl_lock(z, i) spin_lock(&fbl(z, i, lock)) +#define fbl_unlock(z, i) spin_unlock(&fbl(z, i, lock)) + +struct cpu_vmap_zone { + /* + * FREE, BUSY, LAZY bookkeeping data of this CPU zone. + */ + struct { + struct rb_root root; + struct list_head head; + spinlock_t lock; + } fbl[NFBL]; + + /* + * A list of outstanding lazily-freed areas, ready to + * be drained, which belong(address space) to this CPU + * zone. + */ + struct list_head purge_list; + + /* + * It controls that only one CPU can pre-fetch this + * CPU zone. + */ + atomic_t fill_in_progress; +}; + +static DEFINE_PER_CPU(struct cpu_vmap_zone, cpu_vmap_zone); + +/* Disable a per-cpu caching. */ +static __read_mostly unsigned long cvz_size = ULONG_MAX; + +static inline unsigned int +addr_to_cpu(unsigned long addr) +{ + return (addr / cvz_size) % num_possible_cpus(); +} + +static inline struct cpu_vmap_zone * +__maybe_unused addr_to_cvz(unsigned long addr) +{ + return &per_cpu(cpu_vmap_zone, addr_to_cpu(addr)); +} + static __always_inline unsigned long va_size(struct vmap_area *va) { From patchwork Mon May 22 11:08:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5418DC77B75 for ; Mon, 22 May 2023 11:09:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1FD64280006; Mon, 22 May 2023 07:09:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1ACE2280002; Mon, 22 May 2023 07:09:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF1A9280006; Mon, 22 May 2023 07:08:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D3180280002 for ; Mon, 22 May 2023 07:08:59 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 942CA1C7197 for ; Mon, 22 May 2023 11:08:59 +0000 (UTC) X-FDA: 80817618798.18.0ECDA07 Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by imf15.hostedemail.com (Postfix) with ESMTP id 9540BA000E for ; Mon, 22 May 2023 11:08:57 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=flXewQ2y; spf=pass (imf15.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753737; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DrvLnO25dwGFFlUS7GxG9ba70dEBhOOiZuR4m0pcTUM=; b=PrTG02B2+op35ULlGONwIaTzrpslmfFXyFfdwzBAOzDXso6rv1WBgpJO8VQ/91BwhJEAtW wblonHrT5k2AwqpJnWFBjrU/AjP8GZxWA0VWWyUVa4AUA7CxFusVQOgRcx5ejT+HWyYzCp YIld9TnT8bdayZKU8cclqNj05fMu35c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753737; a=rsa-sha256; cv=none; b=WVtncXmwIgzX7Lv1SGAh1GH3RxVtcrfCP2IczzPwbXOluwSgeblZ0MFIvrK1HoZLPJhmN0 snxJZTZ766m3O9bZNyVKLDvEvH0Z9/NmTr49q+1rYRq+2zDDHjdIZUGDsLMFnb29V8WpEh ouV1FHMO7xApezCCs5CuLqcOZ2PELf8= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=flXewQ2y; spf=pass (imf15.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-95fde138693so705252766b.0 for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753736; x=1687345736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DrvLnO25dwGFFlUS7GxG9ba70dEBhOOiZuR4m0pcTUM=; b=flXewQ2yArAVYlv2skHOfRQSd1S7Cqlwt9MNZZJrqFsSrWWwyouqmQjJ95oq8wCC6C ad1/P8xGdXwlSwJMGJC8rDQai6cooJJwK0WkPCEKTPeV1R4JhMhzlsAxID+3OAx7hyz8 qGIYAx4Fe+Vqa5htZtPnUE8LV6kn4rgy+MrpLn2fFSbuz9hRvORwAIU7W24PqNv/v/A+ Vvb0nbx4byHbsHsQfe2U6VreF5LdyXSZ3hV+iYjamkAgVT/lQoTYVkBdU0CzbUpSU/0c 71/85NUEE602yGx29yMZLdgzUhf6phivNIaZUU/ts0HMXqTu7rh81+XTKlpL8mliOzmO yxdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753736; x=1687345736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DrvLnO25dwGFFlUS7GxG9ba70dEBhOOiZuR4m0pcTUM=; b=QjAKx2ho3m3Z3Fe3WhTFMD0oR3c21MDWm6exZG1nFjTQvnHe67XxNgWJyfIpfjiYAT 4hmzsHoJ2YjTiJxCvXSMxElzu7LKJ2vfhIsXP345uLLDDsB8kz43elaBCvnSyPjjP5ZE /mE2XF3lVA0vVTe/hm3nOxKTjJ3W1GMQMqmpIbMJYa6XXWO4Qf59fwZ/geAJgH1iFkh7 /v85y9XrI4/HTNBArzUMkctknHzrRHSrAl6HZFHc1BYS+v+eEvhV6yhFVWnZ0HTkjU3U WnNlcaU7dA+s9r8iSNrFg5/DCEgaIiBfwKucGBG/PPqPu/72lDHexo8OP7y13YzsswBe WtrQ== X-Gm-Message-State: AC+VfDywpzsxtZmWEkk4ue1T9a559RuYoLZB6hGzkp8YVtZFmkGVjPA1 9w2xkVo+yEjCOXQrN11HOWp1VGUVfYKO8Q== X-Google-Smtp-Source: ACHHUZ7/gXy6/G0+Tbl7lFR3/PBdDHxq7cTgGtR+aeCBks9BrXjxTCNVbHQV45jJ3L/ks2X1AHOsiw== X-Received: by 2002:a17:907:961e:b0:96f:c545:6fd with SMTP id gb30-20020a170907961e00b0096fc54506fdmr4438428ejc.10.1684753736024; Mon, 22 May 2023 04:08:56 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:55 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 5/9] mm: vmalloc: Insert busy-VA per-cpu zone Date: Mon, 22 May 2023 13:08:45 +0200 Message-Id: <20230522110849.2921-6-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Stat-Signature: kynnn1z117zhi6d4quxifmyhsm89otwm X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9540BA000E X-Rspam-User: X-HE-Tag: 1684753737-86785 X-HE-Meta: U2FsdGVkX1+VeBpMY0xEZvvsSSTeyhcKl/ER0FzpcKVuCoFoX6of3429FuIF0Mro/uqJU53zEPbcEsdkX8gpdGDyXliQKdmvCF3z4BL3JAfrK2/mnmw2SUA6OKnmuDmlpu6QnLdBIsRDSOGepIRZf1dSv4BELn0YC//1mMEC+grnLzcSuj5MEx/RRwjpFCilafeCH54hl6+g2PRRiKdM4JdPnZpsePHYD8BvqmdLoc7fzBDX6mrlwyko2SITCEacvl1eVltsI2BPK9awJYYmrEn6ivKhE3jphkx0cqCve45XbVqddCOGNvlINM/O9je2deLWABxu/rNirRBkxBb6uFpP7i1VBtRFRgExl8hGBMD4k7WklHI3r3MamhMfMTKkuCdvL5SuRh82M9rqrEW7f4OGIzvUw5AhduI9McB65bJkp2+p0oQWqUh+7J8m64lrRoyQRnSJpBLREaBEPGyPrbQlY9QWdNzqzSLnt84XVbYul8+22QKoDjze0X9KPbt5Y+3k9KwhSSrx0nBVxb+40kufD+RmKjkyAtKpcrh/jTaihRwOmdX78z+KekvWNdEd+tGSk8Xvb+bZ7qh4kPAcUy/aL09tF3FdXIGvugo6cBScgeWpPA6TrTOdaYoOm47Wcpwi0XxOMDsOsc87NsaLfrj8XSfTjBJN1j84vB5xOA5WaFc3VX+tOUpY8Ul8igXlPdWOpZiVP3UimQKUv1fl14HAWl++D+xmj/Dk9Qta8G+0dg40Gf1zOPPv/h4h73Xa3R2YN7D3ydVxqv5+ZM70EXRWy86YwKFpG1jDTIPW3mLWBQ5tWbEddWlc+1VtPrY62PRuCz57P7PDQD8JY7siSMnGapipQPuThl521q0WlAe6R68X4QEdonpx+czuFpOwV5PovufySke9aUC7aFSPvCIT+Gzxlu1aG/H/yg65/lJbvSIR7gAo34IPpyW4Ev58yEtZaJXUGHWA5MAoVIH y2uLxBjH icqLTEkk50YwFkgPFdeN2kmfcLm/aVH5fB0xFENrXFi93+EQcx+JbpwTWeawghw2nChswNVMGW9ueAYPIOiTtSQYPs5Dr90JxMq0uNjzQqA5/CFK5trLCTMD/xXnmhaido2dwojFeN2TcAHiVRlCxanMkghVV3ZsmTBCPKdPvDh1d0YAEY094Afzxhb3IHSnBtUKXfZCpwxW1nqTQb5DqsxWSAIU4aXYd/dSsW7fIUwh+hj59UozeDDrrSh7H31UqIxKp3+8VGPLesx5MHDakoKhnSLtAE300mOVlA1wVtGPJ3xK4E0TGSPR50g/NxGXAevKn0w22TbkbXzotLPaTnpxSAdKCOj/YSLHh/iidO+IUsLgguIjFj0Uobqy9V7yWMcjqAeuGd6SXwmZH1210UqI5zjOKU0AmYd1i3HmJ3Ys8eyKsIh7jjbth3gvrSUzv0dmS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Store busy-VA objects per a CPU zone. A va->va_start address is converted into a correct zone where it is placed and resides. An addr_to_cvz() function is used to do a proper address conversion. Such approach balances VAs across CPUs. That is why an access becomes scalable to number of CPUs in a system. Please note: Since a zone size is set to ULONG_MAX, i.e. everything is bound thus accessed to the CPU_0 so far, this patch does not give any difference comparing with a current behavior. The global vmap_area_lock, vmap_area_root are removed as there is no need in it anymore. The vmap_area_list is still kept and is _empty_. It is exported for a kexec only. The vmallocinfo and vread() have to be reworked to be able to handle multiple zones. As a result of this patch it can handle only one zone, i.e. when cache is disabled. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 84 insertions(+), 43 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index f6da2590b4de..a9170fe19909 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -729,11 +729,9 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -static DEFINE_SPINLOCK(vmap_area_lock); static DEFINE_SPINLOCK(free_vmap_area_lock); /* Export for kexec only */ LIST_HEAD(vmap_area_list); -static struct rb_root vmap_area_root = RB_ROOT; static bool vmap_initialized __read_mostly; static struct rb_root purge_vmap_area_root = RB_ROOT; @@ -823,7 +821,7 @@ addr_to_cpu(unsigned long addr) } static inline struct cpu_vmap_zone * -__maybe_unused addr_to_cvz(unsigned long addr) +addr_to_cvz(unsigned long addr) { return &per_cpu(cpu_vmap_zone, addr_to_cpu(addr)); } @@ -859,10 +857,10 @@ unsigned long vmalloc_nr_pages(void) } /* Look up the first VA which satisfies addr < va_end, NULL if none. */ -static struct vmap_area *find_vmap_area_exceed_addr(unsigned long addr) +static struct vmap_area *find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root) { struct vmap_area *va = NULL; - struct rb_node *n = vmap_area_root.rb_node; + struct rb_node *n = root->rb_node; addr = (unsigned long)kasan_reset_tag((void *)addr); @@ -1608,12 +1606,14 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head, */ static void free_vmap_area(struct vmap_area *va) { + struct cpu_vmap_zone *z = addr_to_cvz(va->va_start); + /* * Remove from the busy tree/list. */ - spin_lock(&vmap_area_lock); - unlink_va(va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + fbl_lock(z, BUSY); + unlink_va(va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); /* * Insert/Merge it back to the free tree/list. @@ -1656,6 +1656,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, int node, gfp_t gfp_mask, unsigned long va_flags) { + struct cpu_vmap_zone *z; struct vmap_area *va; unsigned long freed; unsigned long addr; @@ -1701,9 +1702,11 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->vm = NULL; va->flags = va_flags; - spin_lock(&vmap_area_lock); - insert_vmap_area(va, &vmap_area_root, &vmap_area_list); - spin_unlock(&vmap_area_lock); + z = addr_to_cvz(va->va_start); + + fbl_lock(z, BUSY); + insert_vmap_area(va, &fbl_root(z, BUSY), &fbl_head(z, BUSY)); + fbl_unlock(z, BUSY); BUG_ON(!IS_ALIGNED(va->va_start, align)); BUG_ON(va->va_start < vstart); @@ -1926,24 +1929,26 @@ static void free_unmap_vmap_area(struct vmap_area *va) struct vmap_area *find_vmap_area(unsigned long addr) { + struct cpu_vmap_zone *z = addr_to_cvz(addr); struct vmap_area *va; - spin_lock(&vmap_area_lock); - va = __find_vmap_area(addr, &vmap_area_root); - spin_unlock(&vmap_area_lock); + fbl_lock(z, BUSY); + va = __find_vmap_area(addr, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); return va; } static struct vmap_area *find_unlink_vmap_area(unsigned long addr) { + struct cpu_vmap_zone *z = addr_to_cvz(addr); struct vmap_area *va; - spin_lock(&vmap_area_lock); - va = __find_vmap_area(addr, &vmap_area_root); + fbl_lock(z, BUSY); + va = __find_vmap_area(addr, &fbl_root(z, BUSY)); if (va) - unlink_va(va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + unlink_va(va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); return va; } @@ -2095,14 +2100,17 @@ static void *new_vmap_block(unsigned int order, gfp_t gfp_mask) static void free_vmap_block(struct vmap_block *vb) { + struct cpu_vmap_zone *z; struct vmap_block *tmp; tmp = xa_erase(&vmap_blocks, addr_to_vb_idx(vb->va->va_start)); BUG_ON(tmp != vb); - spin_lock(&vmap_area_lock); - unlink_va(vb->va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + z = addr_to_cvz(vb->va->va_start); + + fbl_lock(z, BUSY); + unlink_va(vb->va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); free_vmap_area_noflush(vb->va); kfree_rcu(vb, rcu_head); @@ -2484,9 +2492,11 @@ static inline void setup_vmalloc_vm_locked(struct vm_struct *vm, static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va, unsigned long flags, const void *caller) { - spin_lock(&vmap_area_lock); + struct cpu_vmap_zone *z = addr_to_cvz(va->va_start); + + fbl_lock(z, BUSY); setup_vmalloc_vm_locked(vm, va, flags, caller); - spin_unlock(&vmap_area_lock); + fbl_unlock(z, BUSY); } static void clear_vm_uninitialized_flag(struct vm_struct *vm) @@ -3605,6 +3615,7 @@ static void vmap_ram_vread(char *buf, char *addr, int count, unsigned long flags */ long vread(char *buf, char *addr, unsigned long count) { + struct cpu_vmap_zone *z; struct vmap_area *va; struct vm_struct *vm; char *vaddr, *buf_start = buf; @@ -3617,8 +3628,11 @@ long vread(char *buf, char *addr, unsigned long count) if ((unsigned long) addr + count < count) count = -(unsigned long) addr; - spin_lock(&vmap_area_lock); - va = find_vmap_area_exceed_addr((unsigned long)addr); + /* Hooked to CPU0 because a cache is not activated. */ + z = &per_cpu(cpu_vmap_zone, 0); + fbl_lock(z, BUSY); + + va = find_vmap_area_exceed_addr((unsigned long)addr, &fbl_root(z, BUSY)); if (!va) goto finished; @@ -3626,7 +3640,7 @@ long vread(char *buf, char *addr, unsigned long count) if ((unsigned long)addr + count <= va->va_start) goto finished; - list_for_each_entry_from(va, &vmap_area_list, list) { + list_for_each_entry_from(va, &fbl_head(z, BUSY), list) { if (!count) break; @@ -3674,7 +3688,7 @@ long vread(char *buf, char *addr, unsigned long count) count -= n; } finished: - spin_unlock(&vmap_area_lock); + fbl_unlock(z, BUSY); if (buf == buf_start) return 0; @@ -4014,14 +4028,15 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, } /* insert all vm's */ - spin_lock(&vmap_area_lock); for (area = 0; area < nr_vms; area++) { - insert_vmap_area(vas[area], &vmap_area_root, &vmap_area_list); + struct cpu_vmap_zone *z = addr_to_cvz(vas[area]->va_start); + fbl_lock(z, BUSY); + insert_vmap_area(vas[area], &fbl_root(z, BUSY), &fbl_head(z, BUSY)); setup_vmalloc_vm_locked(vms[area], vas[area], VM_ALLOC, pcpu_get_vm_areas); + fbl_unlock(z, BUSY); } - spin_unlock(&vmap_area_lock); /* * Mark allocated areas as accessible. Do it now as a best-effort @@ -4145,24 +4160,24 @@ bool vmalloc_dump_obj(void *object) #ifdef CONFIG_PROC_FS static void *s_start(struct seq_file *m, loff_t *pos) __acquires(&vmap_purge_lock) - __acquires(&vmap_area_lock) + __acquires(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) { mutex_lock(&vmap_purge_lock); - spin_lock(&vmap_area_lock); + fbl_lock((&per_cpu(cpu_vmap_zone, 0)), BUSY); - return seq_list_start(&vmap_area_list, *pos); + return seq_list_start(&fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), *pos); } static void *s_next(struct seq_file *m, void *p, loff_t *pos) { - return seq_list_next(p, &vmap_area_list, pos); + return seq_list_next(p, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), pos); } static void s_stop(struct seq_file *m, void *p) - __releases(&vmap_area_lock) + __releases(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) __releases(&vmap_purge_lock) { - spin_unlock(&vmap_area_lock); + fbl_unlock((&per_cpu(cpu_vmap_zone, 0)), BUSY); mutex_unlock(&vmap_purge_lock); } @@ -4258,7 +4273,7 @@ static int s_show(struct seq_file *m, void *p) * As a final step, dump "unpurged" areas. */ final: - if (list_is_last(&va->list, &vmap_area_list)) + if (list_is_last(&va->list, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY))) show_purge_info(m); return 0; @@ -4289,7 +4304,8 @@ static void vmap_init_free_space(void) { unsigned long vmap_start = 1; const unsigned long vmap_end = ULONG_MAX; - struct vmap_area *busy, *free; + struct vmap_area *free; + struct vm_struct *busy; /* * B F B B B F @@ -4297,12 +4313,12 @@ static void vmap_init_free_space(void) * | The KVA space | * |<--------------------------------->| */ - list_for_each_entry(busy, &vmap_area_list, list) { - if (busy->va_start - vmap_start > 0) { + for (busy = vmlist; busy; busy = busy->next) { + if (busy->addr - vmap_start > 0) { free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); if (!WARN_ON_ONCE(!free)) { free->va_start = vmap_start; - free->va_end = busy->va_start; + free->va_end = (unsigned long) busy->addr; insert_vmap_area_augment(free, NULL, &free_vmap_area_root, @@ -4310,7 +4326,7 @@ static void vmap_init_free_space(void) } } - vmap_start = busy->va_end; + vmap_start = (unsigned long) busy->addr + busy->size; } if (vmap_end - vmap_start > 0) { @@ -4326,6 +4342,22 @@ static void vmap_init_free_space(void) } } +static void vmap_init_pcpu_zones(void) +{ + struct cpu_vmap_zone *z; + int i, j; + + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + + for (j = 0; j < ARRAY_SIZE(z->fbl); j++) { + INIT_LIST_HEAD(&z->fbl[j].head); + z->fbl[j].root = RB_ROOT; + spin_lock_init(&z->fbl[j].lock); + } + } +} + void __init vmalloc_init(void) { struct vmap_area *va; @@ -4349,8 +4381,15 @@ void __init vmalloc_init(void) INIT_WORK(&p->wq, delayed_vfree_work); } + /* + * Setup per-cpu data before importing vmlist. + */ + vmap_init_pcpu_zones(); + /* Import existing vmlist entries. */ for (tmp = vmlist; tmp; tmp = tmp->next) { + struct cpu_vmap_zone *z; + va = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); if (WARN_ON_ONCE(!va)) continue; @@ -4358,7 +4397,9 @@ void __init vmalloc_init(void) va->va_start = (unsigned long)tmp->addr; va->va_end = va->va_start + tmp->size; va->vm = tmp; - insert_vmap_area(va, &vmap_area_root, &vmap_area_list); + + z = addr_to_cvz(va->va_start); + insert_vmap_area(va, &fbl_root(z, BUSY), &fbl_head(z, BUSY)); } /* From patchwork Mon May 22 11:08:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A086C77B73 for ; Mon, 22 May 2023 11:09:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBDD7280007; Mon, 22 May 2023 07:09:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D6E8A280002; Mon, 22 May 2023 07:09:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4AE2280007; Mon, 22 May 2023 07:09:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9D42D280002 for ; Mon, 22 May 2023 07:09:00 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 71C73140D16 for ; Mon, 22 May 2023 11:09:00 +0000 (UTC) X-FDA: 80817618840.13.7295EB4 Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by imf13.hostedemail.com (Postfix) with ESMTP id 8DB3C2000F for ; Mon, 22 May 2023 11:08:58 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=Fi720qzQ; spf=pass (imf13.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o/CF6SD8t4KOzLfI5LisrLifDfr954/t5B1JlATADoU=; b=W+OEecMfOhEq/X5DktkgTFjkmQAf0jSvJK2Krd+zvBsejaexG/OYTmOnSmG01nhqukbiNM SM9mBOoAM5yKpeApFtZyJFNE+qO7mvFlLD1eXZwd4zIKnbLI4OSM42gs5+RE1CykB/+KDS tHiSLQ93all7B5saTFYzolghkMypA4s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753738; a=rsa-sha256; cv=none; b=NIklumlPK8oW5aLox18xcmDLynyzX2x6FiuX91486mXEIhqkhXux4aJDurnSSJBdRQHaq6 crVOo+8wAE3Qtmzz8y7tx9FapYXb4jPqPSorStZVKOt/HY7756VEDNKdbM7eIqHI8WJDWm H841mNlibE99T+sPLQWZQBNxYihCXQA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=Fi720qzQ; spf=pass (imf13.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-965ac4dd11bso1180539566b.2 for ; Mon, 22 May 2023 04:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753737; x=1687345737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o/CF6SD8t4KOzLfI5LisrLifDfr954/t5B1JlATADoU=; b=Fi720qzQUzMzlB1l50k0vAlTsjRzBe/cg9qRDCZSutZdUCFmDwXU1qWUWjnhF6C+YJ IYmwGB/dBSBFcf+4n8lnS3JuELxCnqx/5KzHyZvkVrc0zdw6+T1veg4ZmjJxjg9CxrEV OKjrPtpIsPfNKe3Tk/UTfUWcUqfmc21TZhwMRhoz1hrlNz5snsaWIbFcpPMBqrvXzw+e BjnHOUkDc9QXislazQsjEiQDUHPub3bFS8PaRZShQXGb/Yb2GfHmsHP7ed4HMpnWLyiW rLbMYl8TEH+ya11TbsE8iAZdu8RNYvaI/k8Wdj555MFh7QE1iHfdGfjD0joUYCDSs+QZ 4VAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753737; x=1687345737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o/CF6SD8t4KOzLfI5LisrLifDfr954/t5B1JlATADoU=; b=aECyLG0jqaqEP8AtJU9jPTZoMRiKdc3A8ypeXWtaTVUVX0s1KmB/CQdknjhHuaL8KG 6vxK0gPV+DDX8BR9FsuRrztU2T4qAHJ/wfkqTP5ag/udaXsXoXp41WysRQFBUSzWieGO uOf86zW1c/1PtlwpK2ap0XY4XLWdy6pEEbEWbhdFs+AAf9RWetr8p3zHCDPbIpDAzP9P vTLBHfkuS+5O0ZIN6DW/YstRuh4p4NpmHjmZeubvboN9AiYWpWejHpnxMLA9hATvkNC4 KrLmJVDtgZRVEqZjmUwOR9QW0HVQYvQ+3X6DOJMNTrI5Nn+qIFOowe8Z35xFBRy362hV HMBw== X-Gm-Message-State: AC+VfDzv0b6/I0VZANuHSDbkiCtLsnLD9RIeRs+xq/fzCnIHLNQqi+pi hV91M6rftrmv3eG6ahuvuQEtQPch71lbfQ== X-Google-Smtp-Source: ACHHUZ4+1Ulvc5fCkI/Ge8lzhIjdkOo8SAA6ig4yBJZkl0gY7cfCTg6POEY2/5MBo0jvvQfpKEnDTQ== X-Received: by 2002:a17:907:8a12:b0:93e:fa12:aa1a with SMTP id sc18-20020a1709078a1200b0093efa12aa1amr10143550ejc.1.1684753736842; Mon, 22 May 2023 04:08:56 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:56 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 6/9] mm: vmalloc: Support multiple zones in vmallocinfo Date: Mon, 22 May 2023 13:08:46 +0200 Message-Id: <20230522110849.2921-7-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8DB3C2000F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: datzqbr1a87k3bpzfy1jsppk963rmn56 X-HE-Tag: 1684753738-404267 X-HE-Meta: U2FsdGVkX1/YNbsLuC2jOz/sHQ1OGlvZdJrqZE5UhF8HLukq2n0jNQ3WBK/OztYM8vXJt29Z8UJsCmX6UNvF06zQxG1zGMRlgA4ZDMgOfXk/Fze17z1dmZjj0RY/9dTuSWUfL1cpAhnUYJwUXY0jh2tM3UWXBnzjS9eYmGm3EmRK43hPqqznHlUIK518v8knwIq8gaV4BenYdfVwH0mHOjmVmoEsZNpbWFwnwqkMiIopfXA61bySBiCGpWjypMTfUnGp5ljswkOkWQfVKP91DxK9EMNHI1A3mOKi3BssYFpQjmUY9Bv2tHuYJTK4Qg6wnSUBMdKtkpmgNCwhJt4bPMzZgFzIlUa49uNNOWeKABSq8L4MUu22CaFk6VWacBMeEYyYDhjzF3Oo+pfKtoI2z+7xHOBBnNwGdcIcRwU+aYAsXUOzAsmNToW3bVhjlvDPXFGozYuiZ9hYWAfb+Uk+IhujoosflgUqmtyyv5A+IzhXWmX44Cqr63el3mo7dmtJV+rHu5UGXe9zO26Z0nQfpWgVI7Kxydv3SQMBzFlXPikMuH9kr6WWFDneGirJirEJGOWqE8C2dwgys70oDosnbEFE9a9wis9Tu90Fppy30bEaUqr/5bs3bXWHZl+LiWQHBpH3OqAJQN1/EPO2ZpRskJeJBa7LixaQTCHI6OGWxjrI1uCEEYtHxFGb4xJDomDvOsnu1T4khLkeBBzA+KNKs95HoZJ5o0zkD1J/rE1ckVaxXwcfaowHCq3ebGe3yWfCVJpVq27GWq2GzSlEks6xj0/FqQVbSRjbfAe4n4lTrrBXebtUagoNjUmrE2isaQJQHhUmOFUL92ldESfinP2922PEEqsM5eAh5qyzBgx37ITpRfVFW1len/htzFRWYi1/+0G6tiaDHczoWOPgYkxAOnf25RilOH6hgDPVM4zDJHCYQNPfMTk7FGjcP0lSWrqgUHKE7lMvSqkEHXDKqoJ J0puuHYK QLQ0T5IXSMwgHD8GTMeoY9Ykn1yr2GFCzxgM8FeIsdIzdonoSRaDKfLG0wCPHXMV6Bvm2RXysoHahApYhLmlA5ZUYiRKtImvD+i7EeUoZv1NkNeMb0EWCgPACw11ooupmWDm84LC71enmdlynOaWtxwe9I4kl+7dIROFKS7VREcJeABzGMIt4vMGA4GTaNBUMd/WqBJqqBngk8y1zJFSu2STdY/CsKdYUS29Alu+jGLUvOcxSrTeliHGl3Ng9AhdDKCJmFFTW3cCTPwb7e+/gY5ZKo7bNFiUmwxMPQqLfexgpPHU2IrBsCsI5frr+P6VCSxJQAa9sVX2eH4gk6TuDnf+Al7ifUfWGVDt8bVrHmFLSrx3gdlhF/Sv23FOLr3kmj/nck8GG6AK/h4lqhdixDKeH0LcJEgg76nCsNTXO8h0R6AqrzQ4bnuMOmMBb5/JMWD4b X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A global vmap area busy tree has been removed and replaced by multiple per-cpu trees/lists, therefore we need to traversal all per-cpu busy-lists in order to dump all allocated objects. Please note, after this patch, dumped addresses of allocated areas are not sequential. See an example below: 0 1 2 0 1 2 |---|---|---|---|---|---|-> vmap space There 3 CPUs dumping is done per-CPU zone, as you can see address of zone_0 can be ahead of addresses residing in the zone_1 or zone_2. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 90 ++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a9170fe19909..dd83deb5ef4f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -4159,26 +4159,18 @@ bool vmalloc_dump_obj(void *object) #ifdef CONFIG_PROC_FS static void *s_start(struct seq_file *m, loff_t *pos) - __acquires(&vmap_purge_lock) - __acquires(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) { - mutex_lock(&vmap_purge_lock); - fbl_lock((&per_cpu(cpu_vmap_zone, 0)), BUSY); - - return seq_list_start(&fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), *pos); + return *pos < 1 ? (void *)1 : NULL; } static void *s_next(struct seq_file *m, void *p, loff_t *pos) { - return seq_list_next(p, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), pos); + ++*pos; + return NULL; } static void s_stop(struct seq_file *m, void *p) - __releases(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) - __releases(&vmap_purge_lock) { - fbl_unlock((&per_cpu(cpu_vmap_zone, 0)), BUSY); - mutex_unlock(&vmap_purge_lock); } static void show_numa_info(struct seq_file *m, struct vm_struct *v) @@ -4209,6 +4201,7 @@ static void show_purge_info(struct seq_file *m) { struct vmap_area *va; + mutex_lock(&vmap_purge_lock); spin_lock(&purge_vmap_area_lock); list_for_each_entry(va, &purge_vmap_area_list, list) { seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", @@ -4216,65 +4209,72 @@ static void show_purge_info(struct seq_file *m) va->va_end - va->va_start); } spin_unlock(&purge_vmap_area_lock); + mutex_unlock(&vmap_purge_lock); } static int s_show(struct seq_file *m, void *p) { + struct cpu_vmap_zone *z; struct vmap_area *va; struct vm_struct *v; + int i; - va = list_entry(p, struct vmap_area, list); + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); - if (!va->vm) { - if (va->flags & VMAP_RAM) - seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); + fbl_lock(z, BUSY); + list_for_each_entry(va, &fbl_head(z, BUSY), list) { + if (!va->vm) { + if (va->flags & VMAP_RAM) + seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); - goto final; - } + continue; + } - v = va->vm; + v = va->vm; - seq_printf(m, "0x%pK-0x%pK %7ld", - v->addr, v->addr + v->size, v->size); + seq_printf(m, "0x%pK-0x%pK %7ld", + v->addr, v->addr + v->size, v->size); - if (v->caller) - seq_printf(m, " %pS", v->caller); + if (v->caller) + seq_printf(m, " %pS", v->caller); - if (v->nr_pages) - seq_printf(m, " pages=%d", v->nr_pages); + if (v->nr_pages) + seq_printf(m, " pages=%d", v->nr_pages); - if (v->phys_addr) - seq_printf(m, " phys=%pa", &v->phys_addr); + if (v->phys_addr) + seq_printf(m, " phys=%pa", &v->phys_addr); - if (v->flags & VM_IOREMAP) - seq_puts(m, " ioremap"); + if (v->flags & VM_IOREMAP) + seq_puts(m, " ioremap"); - if (v->flags & VM_ALLOC) - seq_puts(m, " vmalloc"); + if (v->flags & VM_ALLOC) + seq_puts(m, " vmalloc"); - if (v->flags & VM_MAP) - seq_puts(m, " vmap"); + if (v->flags & VM_MAP) + seq_puts(m, " vmap"); - if (v->flags & VM_USERMAP) - seq_puts(m, " user"); + if (v->flags & VM_USERMAP) + seq_puts(m, " user"); - if (v->flags & VM_DMA_COHERENT) - seq_puts(m, " dma-coherent"); + if (v->flags & VM_DMA_COHERENT) + seq_puts(m, " dma-coherent"); - if (is_vmalloc_addr(v->pages)) - seq_puts(m, " vpages"); + if (is_vmalloc_addr(v->pages)) + seq_puts(m, " vpages"); - show_numa_info(m, v); - seq_putc(m, '\n'); + show_numa_info(m, v); + seq_putc(m, '\n'); + } + fbl_unlock(z, BUSY); + } /* * As a final step, dump "unpurged" areas. */ -final: - if (list_is_last(&va->list, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY))) - show_purge_info(m); + show_purge_info(m); return 0; } From patchwork Mon May 22 11:08:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5454C77B75 for ; Mon, 22 May 2023 11:09:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C45E280008; Mon, 22 May 2023 07:09:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 973CA280002; Mon, 22 May 2023 07:09:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72908280008; Mon, 22 May 2023 07:09:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5E79A280002 for ; Mon, 22 May 2023 07:09:01 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 40A83160DF3 for ; Mon, 22 May 2023 11:09:01 +0000 (UTC) X-FDA: 80817618882.02.00D70D5 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by imf02.hostedemail.com (Postfix) with ESMTP id 5F60280014 for ; Mon, 22 May 2023 11:08:58 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=W+JtWfgK; spf=pass (imf02.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.42 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753739; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=O5zHBHTvGEb2Zu4p5/DPXJf8BJ5SNzb/BtA238s3igGpnUINc0PruhHGL1pVSr8XmuRAMR Tm7h2GVhd6ELP+Qcur4aFRRrIsFPpnYtS2Thy2x3Vk4I5JnuVnOxR3tToAtHtDRuEHDfhV EDqZKoMJio54EVd+CMCJ2XV65x/uMrk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=W+JtWfgK; spf=pass (imf02.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.42 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753739; a=rsa-sha256; cv=none; b=eVFl3ZqVVrecrAP/IrT/6UTj5n/NvOd95tGbwricJbdbnRG7sWPmB5ihv+l4gaz4yKHuQo 82t2KgpSa7lxymfkYpDVTLIT9QQ/nCO+KkHzioZh8PYDcAt6TaNXUWwdPS4NA8xDcHDHFR VrYu3Hobv+yQ9jkrJXDl8ujjmQ8YJQU= Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-95fde138693so705259566b.0 for ; Mon, 22 May 2023 04:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753738; x=1687345738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=W+JtWfgKVMFA959Uc5C8wiOgCsYc9+SYai8nupDO9KlapBo9VlFnbVWgJAQ004f/QK mZQgkKrt4awtWG7HH1AMVsq0BI7HDBAsUxUanMdXJuT1NjGXA8j91P6MNQVZ7/2q/uOl u8b46ysLxNwbLImafPVAYrKfHx5ge512UBdEXib49XuWenSq7+MfDhUI4cQN2Y29QWPG 8ukh8J/sV1fqVvb+o7S1LN2QlrXlkMocODGZyaNkalUBH3toAXu0mQeN6PrvNuypiv2Q zaaZIwTdJUy/uadYITY2t/C6AJEJ4AgkDslYAlw0hmiXnARf/cNIGN7mPPlYm11bAx6x 3eRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753738; x=1687345738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=HVqIffRRoQ/R9gChCVVzOCE9CDOwTzg6AkiKoP1DnQDWeBMpODsZsD2wqwDszuGCKR RZnfW9zpBH7IMD5VD4pRbLHqBxOkArLltThcYKW85hLhsXPk0pq8lQkavTC7WyMyeDge NQdVG7gnBWJYZGfHKy01zjxZiztMEXLj6vxbC7zbhJnjtEj9k8pnQw0+z2u9LwGY25Rg CFQikfhXgxGBAKNk1Fh0S/Qwa5WB1kC63wNoAmeMP0/2U/d/CxV6BcbmR7RBOnAPGK0R 7OL3obBeg6Fd7kNH0YBFMcNFrOgJyVRM3PsHxSwhAuk5PcDC8QlL37EVnKOfvV6zIor6 Gi5w== X-Gm-Message-State: AC+VfDz5xgoLpAkVSm3jIFgaDGqSEkk1ca1kOczq+yjLlzo+0YMnzNB1 DQ6DTkJI92ALczRoH+Lvzxur/dZ/cG6qCw== X-Google-Smtp-Source: ACHHUZ4lpJZI6b774mW0GGFUcH9S7bFeeu1TaXyRQOPcceEGL/AxG3iytu1ZZc95ZLsmsCWn+xcxSw== X-Received: by 2002:a17:907:3eaa:b0:96f:8d00:43be with SMTP id hs42-20020a1709073eaa00b0096f8d0043bemr6754703ejc.0.1684753737692; Mon, 22 May 2023 04:08:57 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:57 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 7/9] mm: vmalloc: Insert lazy-VA per-cpu zone Date: Mon, 22 May 2023 13:08:47 +0200 Message-Id: <20230522110849.2921-8-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5F60280014 X-Rspam-User: X-Stat-Signature: wsxdjkmfy5sm949j7xdjmssrugii375n X-Rspamd-Server: rspam01 X-HE-Tag: 1684753738-835957 X-HE-Meta: U2FsdGVkX18xxNeK2sxwAAD+C+IaaNobyZDW6ZpxQVMtA5t3BoId3m4NYVWNwev1r18yxBCg+GoOUYxZGI+nuJbRYAGdXcmfCrTbWPg+SZzXPytMAr2+zxmWBFIZFTe3q7d74b8eBWpSrpLuJ6VnepPYBCp9gwx+ga4A5x0rxQw0vzV9mv79asr9JruF3dBdchaYDa7/NOXeKsj5eeXa/ud4TPeIpZgMAHQ7pZibr3N+sxKeM6ohqwCbDWYnouwwMjjW6/IWiegUqkvJZG5RSL3/XVtIXxMP029T+6sjITD+sP6iR4BFXkgw/7+wmaLQ/JO60i/CsNptDnjJMmeUh1u65bkgY5yo2z9NIjHBN5L1Qds3R2vKMczDxJ84DjGsCYLYrokyrXqAUaF+Sx8KsZlih3qaBS/7JO0YxHx6Ky1c5ZfBXUhGji9dmogoVLHhmaF783R6pe7DjtxkN1+78hE3hjD4iSQkSFlPspnC8SSfIq6oGdSsHRfk/Xe76IaDbhAU6gdLL0L14M7kjIak5bn6nqWeTYqc8aRmEdP0h5eaUklJ9K+GLzQrJZMsmL+Fcj9+muJGC4tZImZNE6ImRTlVEup2toRVWmXc/eQ0C2wp85JLnf8IKd8WBvhtNHAJK2X3ObWJFfJg5hZD77ymvycoWKWVB+yuSPSkvFUWb20WDbKKnqmBuPdK5dtK7ck9QwVH/PCUg3i5SGVbZgn4/nugfOsussXhOG3JO+SXd1FvpWKDThokbLJEl9ZvDIqH8zAkNBYfRgxFZzaDBXwjACsydWkzbk7L2yR5EWMf7CpzzqOW9h8kGrNsiy8CuZjvr5lMIi+nlGnnLoGDkuD8SX75d4FaTBnof4XMeoC9JEkV94qb81lWZWfSDoNQsuAifQfzFfnP3VWXh7XrjL1dRNQACFwBDCAzdYS47BpPi+RjNKPr19DT6p0quuXAzIoibmvT5/wkuBYaeygStps n2sSU7vv kET8RP/S7tl/hLJoJOYJPo1pFrEIrttfrcT2W/A3NJlKnPyp8x9FBxDYm7cvN/OQEg51CUkkRzfyEVQoAWndtn51YKF3NzxP7Fw0aBcvaPv3nH22l/rhVNnd4kgqoWMXI20+YRb24VmcBlqWAG3B0OF16XLYi373fVYWqZMD86v/NjYQ/2/T4wT19hXh1WQ2Ad59pdihaztokwQKojDVwg0rNYESozA3gGCvOgIEgjKxbAqPUkz0tNV+sh0yFW7bX4xHt/NTlH5XwokVzVlDo/G4ttAtpT14wO+N9SKg1hoqfRskD2QTfYEdbx2NDsBcTfBV2/cCrV9pzLmZveC6y6Pk/LmHlScmdJnArmd5hyppDHOsIPxUR8lSBw0i8BfhCeAsxkEhLWItVjoiGt+R4lvsFxkvEjBJWlk8ZMI0NpN9ymHtpIFGEva4ZZGo7U2Z3Vdv/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similar to busy VAs, lazy ones are stored per a CPU zone also. Freed address is converted into a correct zone it belongs to and resides there for further handling. Such approach does not require to have any global locking primitive, instead an access becomes scalable to number of CPUs. This patch removes a global purge-lock, global purge-tree and list. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 ++++++++++++++++++++++++++++----------------------- 1 file changed, 71 insertions(+), 56 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index dd83deb5ef4f..fe993c0561dd 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -734,10 +734,6 @@ static DEFINE_SPINLOCK(free_vmap_area_lock); LIST_HEAD(vmap_area_list); static bool vmap_initialized __read_mostly; -static struct rb_root purge_vmap_area_root = RB_ROOT; -static LIST_HEAD(purge_vmap_area_list); -static DEFINE_SPINLOCK(purge_vmap_area_lock); - /* * This kmem_cache is used for vmap_area objects. Instead of * allocating from slab we reuse an object from this cache to @@ -1792,39 +1788,17 @@ static DEFINE_MUTEX(vmap_purge_lock); /* for per-CPU blocks */ static void purge_fragmented_blocks_allcpus(void); -/* - * Purges all lazily-freed vmap areas. - */ -static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) +static unsigned long +purge_cpu_vmap_zone(struct cpu_vmap_zone *z) { - unsigned long resched_threshold; - unsigned int num_purged_areas = 0; - struct list_head local_purge_list; + unsigned long num_purged_areas = 0; struct vmap_area *va, *n_va; - lockdep_assert_held(&vmap_purge_lock); - - spin_lock(&purge_vmap_area_lock); - purge_vmap_area_root = RB_ROOT; - list_replace_init(&purge_vmap_area_list, &local_purge_list); - spin_unlock(&purge_vmap_area_lock); - - if (unlikely(list_empty(&local_purge_list))) + if (list_empty(&z->purge_list)) goto out; - start = min(start, - list_first_entry(&local_purge_list, - struct vmap_area, list)->va_start); - - end = max(end, - list_last_entry(&local_purge_list, - struct vmap_area, list)->va_end); - - flush_tlb_kernel_range(start, end); - resched_threshold = lazy_max_pages() << 1; - spin_lock(&free_vmap_area_lock); - list_for_each_entry_safe(va, n_va, &local_purge_list, list) { + list_for_each_entry_safe(va, n_va, &z->purge_list, list) { unsigned long nr = (va->va_end - va->va_start) >> PAGE_SHIFT; unsigned long orig_start = va->va_start; unsigned long orig_end = va->va_end; @@ -1846,13 +1820,57 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) atomic_long_sub(nr, &vmap_lazy_nr); num_purged_areas++; - - if (atomic_long_read(&vmap_lazy_nr) < resched_threshold) - cond_resched_lock(&free_vmap_area_lock); } spin_unlock(&free_vmap_area_lock); out: + return num_purged_areas; +} + +/* + * Purges all lazily-freed vmap areas. + */ +static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) +{ + unsigned long num_purged_areas = 0; + struct cpu_vmap_zone *z; + int need_purge = 0; + int i; + + lockdep_assert_held(&vmap_purge_lock); + + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + INIT_LIST_HEAD(&z->purge_list); + + if (RB_EMPTY_ROOT(&fbl_root(z, LAZY))) + continue; + + fbl_lock(z, LAZY); + WRITE_ONCE(fbl(z, LAZY, root.rb_node), NULL); + list_replace_init(&fbl_head(z, LAZY), &z->purge_list); + fbl_unlock(z, LAZY); + + start = min(start, + list_first_entry(&z->purge_list, + struct vmap_area, list)->va_start); + + end = max(end, + list_last_entry(&z->purge_list, + struct vmap_area, list)->va_end); + + need_purge++; + } + + if (need_purge) { + flush_tlb_kernel_range(start, end); + + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + num_purged_areas += purge_cpu_vmap_zone(z); + } + } + trace_purge_vmap_area_lazy(start, end, num_purged_areas); return num_purged_areas > 0; } @@ -1870,16 +1888,9 @@ static void purge_vmap_area_lazy(void) static void drain_vmap_area_work(struct work_struct *work) { - unsigned long nr_lazy; - - do { - mutex_lock(&vmap_purge_lock); - __purge_vmap_area_lazy(ULONG_MAX, 0); - mutex_unlock(&vmap_purge_lock); - - /* Recheck if further work is required. */ - nr_lazy = atomic_long_read(&vmap_lazy_nr); - } while (nr_lazy > lazy_max_pages()); + mutex_lock(&vmap_purge_lock); + __purge_vmap_area_lazy(ULONG_MAX, 0); + mutex_unlock(&vmap_purge_lock); } /* @@ -1889,6 +1900,7 @@ static void drain_vmap_area_work(struct work_struct *work) */ static void free_vmap_area_noflush(struct vmap_area *va) { + struct cpu_vmap_zone *z = addr_to_cvz(va->va_start); unsigned long nr_lazy_max = lazy_max_pages(); unsigned long va_start = va->va_start; unsigned long nr_lazy; @@ -1902,10 +1914,9 @@ static void free_vmap_area_noflush(struct vmap_area *va) /* * Merge or place it to the purge tree/list. */ - spin_lock(&purge_vmap_area_lock); - merge_or_add_vmap_area(va, - &purge_vmap_area_root, &purge_vmap_area_list); - spin_unlock(&purge_vmap_area_lock); + fbl_lock(z, LAZY); + merge_or_add_vmap_area(va, &fbl_root(z, LAZY), &fbl_head(z, LAZY)); + fbl_unlock(z, LAZY); trace_free_vmap_area_noflush(va_start, nr_lazy, nr_lazy_max); @@ -4199,17 +4210,21 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v) static void show_purge_info(struct seq_file *m) { + struct cpu_vmap_zone *z; struct vmap_area *va; + int i; - mutex_lock(&vmap_purge_lock); - spin_lock(&purge_vmap_area_lock); - list_for_each_entry(va, &purge_vmap_area_list, list) { - seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + + fbl_lock(z, LAZY); + list_for_each_entry(va, &fbl_head(z, LAZY), list) { + seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } + fbl_unlock(z, LAZY); } - spin_unlock(&purge_vmap_area_lock); - mutex_unlock(&vmap_purge_lock); } static int s_show(struct seq_file *m, void *p) From patchwork Mon May 22 11:08:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7299EC7EE2A for ; Mon, 22 May 2023 11:09:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B266280009; Mon, 22 May 2023 07:09:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 83C4B280002; Mon, 22 May 2023 07:09:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F263280009; Mon, 22 May 2023 07:09:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4C0FC280002 for ; Mon, 22 May 2023 07:09:02 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F116140D74 for ; Mon, 22 May 2023 11:09:01 +0000 (UTC) X-FDA: 80817618882.24.6DE7D9A Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) by imf10.hostedemail.com (Postfix) with ESMTP id 13782C0015 for ; Mon, 22 May 2023 11:08:59 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="Ad+N9/Bx"; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.45 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753740; a=rsa-sha256; cv=none; b=DbRhPzSjoxYdYb6XmrB6Oh8hn6GVpKhgD8nmdEm1ucwK94EFIF6RpgnI0eIjVWcBTdHM0v NN9IKyI2wpCBqJgjkj93cMsIAlsCRCOgimwtu/dXkuSQMxe8yNwrYBMHM6XfeOPhEyiAj1 9g5tYa0c65/V2QZfF9jioIm/h2HUTgo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b="Ad+N9/Bx"; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.45 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=55sY0hUMVMxmKvjlYwxs4pvClntO+d/CkKt0D/BL0C26WrheBoihj9M7u/MKWuXLV7O/qS N4ToD6bUYbPqGClI5ZDhVInZoCNBZbQ69gboBuJLRrCa/UXMA+Zl1vGWymSSOSrENp8FhK NCLHOAHTy9eYBW3+Rq3O6Km5gwOD7/U= Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-96f818c48fbso542123866b.0 for ; Mon, 22 May 2023 04:08:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=Ad+N9/BxXUiItff9Ka8gp9m6Ut+hvQw/iiM/znN/E/FjnWbeKMX497Feia+wUTQL7P nn0HMTXM+eAOwCIXtFoVtxCNH0LHncvSzMz7xzwQ1Srikc1cgV966ZS1qa7YdSUE18l3 cG1SVgkhO1ucaJlN4YMO1/+KLZpOHs16VnpaYal4+QLxI/ktvW/6nIUZRewRfRy6T7uj t6Bkz1DHlo2XaUESkC1cNdMlQINss0j4ahV0Ld6LZrYiZAUobHEQcZF2xuwkm88kQ9RE EafR2f2o+2S2vnNz55Q7/rEKNFxFSEEhyB+7TfomnLLk3I+4DN8cvc7+ZtkoKPBd38H7 HuUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=SZ2bh2G1jF0Hbrlix6Sb3jcKRITsLkUPe9Wd8UngV5BHgNDreyGXIcE/mx9gMmypmF VQzynCaASSs72wU+uSLG96lx87LLyGAen/yEpwZPOgNjo9vrxhBc4ysFHol9mR0G4xb2 Uv+0SPwomeFF+SATLKmf5sQApm0afvcaGzsAdVO0Xd4p0kd+Fm2JhHv41XceMjx3VOOO 1paxDO3fgqpOt6v8+gau+JX9cs6E2PWB/ZS4tnOPQtAcsdA3931mbxl9jghnDrCAU4A2 OyJb1SL0mlNBi36YYO8nbwu5/FfoQ9l6jNQuRyTSaEUCgLugpN5jGv3TTw1lHM5y5Z60 zG1A== X-Gm-Message-State: AC+VfDz3DhAaVOPGijtrxJcxTSppir5Tz4gdlFZ2kDbegfJXPEL3Kbg6 wUMbwyfJIZKHP9iK/Xmz22vYWrAXh91rUA== X-Google-Smtp-Source: ACHHUZ4pAASV3ZfWJMFVdFF/6izNH1fGukazCQNkIlZEbt/DA1kUADDrO481FwKNM2C/1frFuAV8LA== X-Received: by 2002:a17:907:1c07:b0:96f:d8a9:d045 with SMTP id nc7-20020a1709071c0700b0096fd8a9d045mr3465283ejc.59.1684753738602; Mon, 22 May 2023 04:08:58 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:58 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 8/9] mm: vmalloc: Offload free_vmap_area_lock global lock Date: Mon, 22 May 2023 13:08:48 +0200 Message-Id: <20230522110849.2921-9-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 13782C0015 X-Stat-Signature: w4y675u5hyrwn5m1k4uqkxze4qd4jitp X-Rspam-User: X-HE-Tag: 1684753739-942994 X-HE-Meta: U2FsdGVkX18oNxWTfS5UrjNKBuWVolNgt0/7Xj9Uu2zA60MEnD27kLq+TNrC6CFcS9/INiKERy5xpdd73SMa3++8exkDe/WekDkTLZ6Gi1FMtojG1Aj9k9MOh9b2RNc5b5BzK4k8yJEA/Qr31WZHC9404E36vJ8+I2eVVgs28IcFBYr30lywYbdgRfrlI6GBnSn16wnOZk9UQpLbU1MFotaG6B+8dc+gPwjFIxLkuyYStaPIqsKJOSL8MKyui47mVhCnaHrJsW1Hl7eJicfqlyb43PKw/mW8XYqM9ZR+UA5ToSnNnTLrUXbcWE9tUwd0rKFHF9RwuoibZLCHsNsmvpuHzj6kt9UU2nQltVC7mxLvHgqdrWinFppf0nYh90fTicyND983AhNjkNqiQXbPP1xDq4eTUHe/gMO9zkvR+UbmfN74teVAikvxFbnGSiS5nexUly0vUbVA4SGAlj54N+DsUIuRUMyTOzkpfg3OyEYk2g2Ap0LUPaBxHoRY2pMIlPr34eAnvIXWKhMNJlKCmMldygPZB11a8J5BXfC1VlMZgdq87+CCFHCQUpaOAsKK6XI6euwCArtRQhqyaC2eUWA5edn467i6aiBZpgQbid/syNhOAd0/4EHeDt5D+UjqNS32Jvpr2NA9npdk7jNm9i04Atn7RX/FneW7j98SI1ESuWnPuPX1AXGq6o3MOUdTm5zxEhUBal5o5BbRW0oLZS2sXrKRRhvv/7AwW7oWoCzmaalw6B4eSZFrBcglpmOMWpKZA5+kR9il5eqeSbZsQ2R0yh/Cj6XDCYG1nGmVjlO3m8tTSnBGTOHmYRCgpDZwJrMxM2kUZDGnMYFga/mOFzvxQI7xAgHbH6fkMxAUYGRof+iaPh/ldijcxih9Mg1ZFh4bphVH5NX2cFeYS1QBhHJf1korjN58NeTbEuKCsGMP7MC4OLkdSyPOVN9VCwRLbYgIDq2WxjGU2jrlN8X qwFXo2nR BA9k/pHRsMFLJW10XICWG30wqdBh3ufXHb3wYKaFaTQY0TZOET30R82PGOYCWlOTrHMyXyWkXVb2gMtnunyQRuT0rntM3iq4vhRNwk2nWVKWfkNX7fgJQLhrLpUQCFZ5mAtiqILQuIILLXcZhCCeb40ZHaHGwJOoEnz68M3iAlIp9uZBm8EmGdlY8fv01zzLASFdSyf1qEzVtzrPYH6BFmsxyEM+1jlpn6iOKZxGPTqZA26SRNe/6VRYIay00ZhYqjeIxGUHdZ4o7anFqhYNy+KCvbNObpi+8ycDW4cw9ChwjyUtgwgXVunFIsV1sMOxZMqPsbebaToVpsbuIaYG5g9q7n243rliAm+keiQl1ra2o5HjtyXjskLAgEpJy8/kwo7WapvWIPqPJF8uBjNanjxH5LpZO68JKs9xJS1HE/PdgayG00qnOucTaojTIM9421Ty6MOM7Q1/kJXyfj7lbWn6jO6p1gTydFA90ZQ6KhvAJ5RM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a fast path of allocation sequence, that consists of per-cpu path and fallback mechanism which is used when a request can not be accomplished by fast track. A fast track pre-loads a chunk from a global vmap heap directly into its per-cpu zone, following by clipping the chunk based on allocation request. This technique allows to offload a global free_vmap_area_lock making an allocation path to be serialized to number of CPUs in a system. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index fe993c0561dd..8054b8bf6c18 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1642,6 +1642,93 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node) kmem_cache_free(vmap_area_cachep, va); } +static unsigned long +this_cpu_zone_alloc_fill(struct cpu_vmap_zone *z, + unsigned long size, unsigned long align, + gfp_t gfp_mask, int node) +{ + unsigned long addr = VMALLOC_END; + struct vmap_area *va; + + /* + * It still can race. One task sets a progress to + * 1 a second one gets preempted on entry, the first + * zeroed the progress flag and second proceed with + * an extra prefetch. + */ + if (atomic_xchg(&z->fill_in_progress, 1)) + return addr; + + va = kmem_cache_alloc_node(vmap_area_cachep, gfp_mask, node); + if (unlikely(!va)) + goto out; + + spin_lock(&free_vmap_area_lock); + addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, + cvz_size, 1, VMALLOC_START, VMALLOC_END); + spin_unlock(&free_vmap_area_lock); + + if (addr == VMALLOC_END) { + kmem_cache_free(vmap_area_cachep, va); + goto out; + } + + va->va_start = addr; + va->va_end = addr + cvz_size; + + fbl_lock(z, FREE); + va = merge_or_add_vmap_area_augment(va, + &fbl_root(z, FREE), &fbl_head(z, FREE)); + addr = va_alloc(va, &fbl_root(z, FREE), &fbl_head(z, FREE), + size, align, VMALLOC_START, VMALLOC_END); + fbl_unlock(z, FREE); + +out: + atomic_set(&z->fill_in_progress, 0); + return addr; +} + +static unsigned long +this_cpu_zone_alloc(unsigned long size, unsigned long align, gfp_t gfp_mask, int node) +{ + struct cpu_vmap_zone *z = raw_cpu_ptr(&cpu_vmap_zone); + unsigned long extra = align > PAGE_SIZE ? align : 0; + unsigned long addr = VMALLOC_END, left = 0; + + /* + * It is disabled, fallback to a global heap. + */ + if (cvz_size == ULONG_MAX) + return addr; + + /* + * Any allocation bigger/equal than one half of + * a zone-size will fallback to a global heap. + */ + if (cvz_size / (size + extra) < 3) + return addr; + + if (RB_EMPTY_ROOT(&fbl_root(z, FREE))) + goto fill; + + fbl_lock(z, FREE); + addr = __alloc_vmap_area(&fbl_root(z, FREE), &fbl_head(z, FREE), + size, align, VMALLOC_START, VMALLOC_END); + + if (addr == VMALLOC_END) + left = get_subtree_max_size(fbl_root(z, FREE).rb_node); + fbl_unlock(z, FREE); + +fill: + /* + * A low watermark is 3 pages. + */ + if (addr == VMALLOC_END && left < 4 * PAGE_SIZE) + addr = this_cpu_zone_alloc_fill(z, size, align, gfp_mask, node); + + return addr; +} + /* * Allocate a region of KVA of the specified size and alignment, within the * vstart and vend. @@ -1678,11 +1765,21 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, */ kmemleak_scan_area(&va->rb_node, SIZE_MAX, gfp_mask); + /* + * Fast path allocation, start with it. + */ + if (vstart == VMALLOC_START && vend == VMALLOC_END) + addr = this_cpu_zone_alloc(size, align, gfp_mask, node); + else + addr = vend; + retry: - preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node); - addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, - size, align, vstart, vend); - spin_unlock(&free_vmap_area_lock); + if (addr == vend) { + preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node); + addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, + size, align, vstart, vend); + spin_unlock(&free_vmap_area_lock); + } trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend); @@ -1827,6 +1924,27 @@ purge_cpu_vmap_zone(struct cpu_vmap_zone *z) return num_purged_areas; } +static void +drop_cpu_vmap_cache(struct cpu_vmap_zone *z) +{ + struct vmap_area *va, *n_va; + LIST_HEAD(free_head); + + if (RB_EMPTY_ROOT(&fbl_root(z, FREE))) + return; + + fbl_lock(z, FREE); + WRITE_ONCE(fbl(z, FREE, root.rb_node), NULL); + list_replace_init(&fbl_head(z, FREE), &free_head); + fbl_unlock(z, FREE); + + spin_lock(&free_vmap_area_lock); + list_for_each_entry_safe(va, n_va, &free_head, list) + merge_or_add_vmap_area_augment(va, + &free_vmap_area_root, &free_vmap_area_list); + spin_unlock(&free_vmap_area_lock); +} + /* * Purges all lazily-freed vmap areas. */ @@ -1868,6 +1986,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) for_each_possible_cpu(i) { z = per_cpu_ptr(&cpu_vmap_zone, i); num_purged_areas += purge_cpu_vmap_zone(z); + drop_cpu_vmap_cache(z); } } From patchwork Mon May 22 11:08:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11D43C77B73 for ; Mon, 22 May 2023 11:09:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 57DB728000A; Mon, 22 May 2023 07:09:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 506B4280002; Mon, 22 May 2023 07:09:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3ADC228000A; Mon, 22 May 2023 07:09:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1FCDB280002 for ; Mon, 22 May 2023 07:09:03 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 01DE31C7DD5 for ; Mon, 22 May 2023 11:09:02 +0000 (UTC) X-FDA: 80817618966.20.56CF4E2 Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by imf07.hostedemail.com (Postfix) with ESMTP id 113464000A for ; Mon, 22 May 2023 11:09:00 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=PvB7xNb9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753741; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f2bGuuQYL8b+xbvuLn4PcYekQUWt9yPN5iQ314l4lMI=; b=JnZSRMQkV6NeEoqULONcxNPxh+V41UlTFQapE4Bmve9lXDrplAH6DxVJL0AQ2YmY11N6+E Yhyw7y60YB1QnSumKdR22c0wddDAirfCMb+pafN3h/p17PoUnH17TSf/WfygFMOtaeDPjZ 5rONHL71MrzY30CqcFNqvw8JosnUZYI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=PvB7xNb9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753741; a=rsa-sha256; cv=none; b=sZ2/cTb7MSnoSLlR6y8TBrinwJGZML6Nedw194CcPglFEJ0hepxBvW2Wx1kf+10uPCDy40 dPeG+jPvaj3wHZJkftAWnUulQTCrqBXiG5b92SozRfMxjlQn6emOTNxtl60ZUTBA3I6Ibe wD4cfm5ZQeg4Pneeu93zae53SPpCinw= Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-96fd3a658eeso174379466b.1 for ; Mon, 22 May 2023 04:09:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f2bGuuQYL8b+xbvuLn4PcYekQUWt9yPN5iQ314l4lMI=; b=PvB7xNb9ygkZOuxzTtzRZdOhB6MfhJSa/XRAM+BYE1xoLfv+ewUOyX4VD8Bu0FXaOK AXlUTKkyIK2WVRsD39QMuEntv1HpM65yERdXpxKJXgRGMC2pW3R98Ogtx7X6mgbxBHMR ldToVDFezt0XeMtgZh1MaqlUYU2be1G2xPupT/rIWTKOSu/1KFkLMDIX1H3BLSGe29sO Fg3kOpF9fGfKN44fAOhFpBwP9Z7IQ03rJSGvaYH1M1Nh+mhzwxadyfvgGc5HplBYwh3q fzDEE5gFHzKH0XnCa9ZqMPOjoi2odYahJQNJL0GrEIc+/7U+m3d4QYr1A8eWf9czKONh 1f4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f2bGuuQYL8b+xbvuLn4PcYekQUWt9yPN5iQ314l4lMI=; b=D2jq/5G+H9bBcyLYVrOUsIeXt99VCDrVXXir4e9hFCB3y4u1zBBe2/mnpUPx5UXa21 h1QcSDosSzEgscazRWNk4je5ilTXSumVpRhZR4goMEws+GrzfdoWxLVHkgRA2ufYH21E afwFuQSzZI1PoVEmCi5n6JJX/tLGA1uGs+V12UhcezOh29mghMb/PdxLJGV4/PPTJtRm XqmC8dQShuCXvDEEQ68tR09MoOuaa6N8QkLISgaTLKd5lkitluVjTGtMwrcDcuhrh2hl 1hb5js4ihMQk8bCKNI39xzfR8oa6gMcBifsK1PfacqdS5qIa8XfjuJr5cmK8W65Ss8/q Qe1Q== X-Gm-Message-State: AC+VfDx3rXz2JT6Hj/IDaGKnlDX4qrskiYdk/UFjI5ZNq9iqxX0v7CoO 2u0/LE3GBpo+77oRh2h6f+e0eRpvqoke6Q== X-Google-Smtp-Source: ACHHUZ7OjSrTldnv+rUkfYY8jX9+e0NK8wBEQHN/GocoDZWHojfGcLerDTURKKOgb/QQDOHh1JKUqA== X-Received: by 2002:a17:907:a42a:b0:96f:a7d9:6e86 with SMTP id sg42-20020a170907a42a00b0096fa7d96e86mr6231547ejc.1.1684753739445; Mon, 22 May 2023 04:08:59 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:59 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 9/9] mm: vmalloc: Scale and activate cvz_size Date: Mon, 22 May 2023 13:08:49 +0200 Message-Id: <20230522110849.2921-10-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: sbbsa4z9fxu1sqgb6f3kew87imroia4w X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 113464000A X-HE-Tag: 1684753740-562281 X-HE-Meta: U2FsdGVkX1/aJ5TobSOxnHF53DGhs+7nUtt8jvejA7Z3G4yRGpKuySgQLNexvB8qZLEq1xQz9XQUJy7sM7PTfhaDUf/yyu7AIcmTrXQiE3pd9VtfVbWH73JTlqJwaTJtnHbD5qmgdDsKt4oXkfr8VhzdNuSFdQ2pW+hV9xG1Fh3pt/Fm7uUGZo3TITjMHUjnVJkGZ9DqqLfbpXYSXBlAYmhjghurYscsL9rllF2rtVPaO7sk/TMhLh+lsKPzodQPWQ1GIZKgf1lK0jTiJFfECDFzPwRS44P90xVS8eOSmv6j8uv0/vISUNQeMMU1KNNmXTqnlNVQaVyQOkOm7ozIYFV0bH/XqK9gRbWyRQEneC9Spk1iWzPI1Y/HDaXoSyINGaSDap5nGArV+LOdpVjhdJBAeDyx7AKYn3MscIKj+9GlPihqyKWV/yIOBbnVyK6/QTlowA91zSmqmSslOEDHVmWQjl9lGyNBey3LinAv1ngg3pxKjXffttn+kVrzmpfDMZQGHsUdaEjDz9qBTPY1neFP3B1TB4CDo4Cx+UlEQpVnTHFVZMWsqGqnf3vCIhNJO/3iOh0jxNJ0ZljJcjrAIxC6iaRIkzrmJXgCk94FTZ6cvpRkS3bSa/ocKaqTuufaYmRgabO/p3z3cU2kd726McBRm6JpMyw6p+Lp6XWCix6mCVRjbMg4rojUFhOGwu/8TclWHwojfvItyn/YW5iXvmvq6CtaoN56dvHrjN7pz5CAdh6g+8wOj6VJOwHsB/Ct6sO6MrNwll/hXGCb2ftsXffA7vN+hD0u2OaMmX1ApAiRM163wQ0GABsKgQZ4H4VvAvqBlNIxk5KoJuGLsGvvIvbPEpwDZf+970rGz2MU8BNRjkI7QjFrRgOoZML9IJDvoBN+hZl7q7ZRLjdSTnpmNPlseAKvXworRqgZZCKEdsljXLKNFgKKwCFvRTeb2eSihNcJM1ABcvIuxZ5tqVY yce51PdH fs1nb2pDSbtoe6/GleBszi/L6P6wlKPa6GJaehWcE9XNvhbhTXKMeLYwfkfBMzki4MdyIaNYahMmn9P1Sv2Bj4fvv9ebOivoTCQ4809pcFiKVVwwSi9ff7DCF1wM7/woEMUmw/TBICXiebiA2dSt3i5j61+3dkWrfQzbB6My+GKQxMG1YGDcVGgLhd+EUFEEMLA9CvjNwsuvBrOcSFQElEtM3EMavzH3x8jZqPq5bi0S1b6rr4C2snrhvCLJVCxXfm0OppHJGfEox6fTXmKpXjSnQo1psxN+6tTEa2wK2GLRLJYQPamQDggDTbRXxWnWwhKnfOG6ZkAcvpxQhhB8jZqaVXwr5xV8kEOi7FT4EgWsaws9vk0GmtDnbUw5f1H/gBSjFFeFM0nmzJlclBowWZ3BC8JF1wH/xw++O52FUa6PrIvPQ0jEvSH+2nXlNva75AzaBSL55zVQnSFwT8bIS8NH0POvmPRwFFNPE/jzqAWITJ08= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Scale a zone size to number of CPUs. Each power of two value adds extra 100 * PAGE_SIZE bytes to a zone size. A maximum zone size limit is set to 4M, it is reached if number of CPUs=1024 and PAGE_SIZE=4096 bytes. For 32-bit or single core systems cvz_size is set to ULONG_MAX. It means that a pre-loading technique is deactivated. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 8054b8bf6c18..a5e6956a8da4 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -806,8 +806,6 @@ struct cpu_vmap_zone { }; static DEFINE_PER_CPU(struct cpu_vmap_zone, cpu_vmap_zone); - -/* Disable a per-cpu caching. */ static __read_mostly unsigned long cvz_size = ULONG_MAX; static inline unsigned int @@ -4490,6 +4488,19 @@ static void vmap_init_pcpu_zones(void) spin_lock_init(&z->fbl[j].lock); } } + + /* + * Scale a zone size to number of CPUs. Each power of two + * value adds extra 100 pages to a zone size. A high-threshold + * limit is set to 4M. It can be reached if number of CPUs=1024 + * whereas a PAGE_SIZE is 4096 bytes. + */ +#if BITS_PER_LONG == 64 + if (i > 1) { + cvz_size = fls(i) * (100 * PAGE_SIZE); + cvz_size = min(cvz_size, (unsigned long) SZ_4M); + } +#endif } void __init vmalloc_init(void)