From patchwork Tue Aug 29 08:11:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13368657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 382C9C83F14 for ; Tue, 29 Aug 2023 08:11:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58596280030; Tue, 29 Aug 2023 04:11:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 535C08E001E; Tue, 29 Aug 2023 04:11:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 424A9280030; Tue, 29 Aug 2023 04:11:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 33B218E001E for ; Tue, 29 Aug 2023 04:11:48 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EA399140671 for ; Tue, 29 Aug 2023 08:11:47 +0000 (UTC) X-FDA: 81176423454.04.948F556 Received: from mail-lf1-f50.google.com (mail-lf1-f50.google.com [209.85.167.50]) by imf04.hostedemail.com (Postfix) with ESMTP id 28E9240003 for ; Tue, 29 Aug 2023 08:11:45 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=HvynVJla; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.50 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693296706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=ypiT9aaPKimQJTb8EA0BWrlL9wmne7HVxtb4pVgUIM8=; b=V8UxRakAN3IU0soWGX/0g/RmAONfHAgYbO6kj5B7gGrM65vIW7Z0pZNmWLgUtmo0zuIujf vfP8YfRQXlTAmflaOCV2J7BF7GIbQbe7y8esyv+ZlpjVac1g3h4zZzCNpIDZ3snxSI1EVD SdrumS+SJrzYWcl+eXgDG68y0I00zDk= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=HvynVJla; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.50 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693296706; a=rsa-sha256; cv=none; b=IlsamnMZVWdYA0uek7gOa6JbD+eSjK6x/x4V577cJhQ89eIhLWJeqhWx7bi7fV4PSZSefa oiBYvTbDwlmpE3QXOfpngY3SUNHcNvPt6fRATgHUdJ66WP3s83qlX8QM6OPscldJTCX0Qw /gads9213H6lO1rddP9hNZmllQ/vVpw= Received: by mail-lf1-f50.google.com with SMTP id 2adb3069b0e04-500a398cda5so6586835e87.0 for ; Tue, 29 Aug 2023 01:11:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693296704; x=1693901504; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ypiT9aaPKimQJTb8EA0BWrlL9wmne7HVxtb4pVgUIM8=; b=HvynVJlaIEJi8g772BM+SVLOetf4v/1itlq8nITpEQsMEsdeZKYVZUkuOCVFvJEYkv d77YpVpNYDArASqogFv/Thj9BXD4BHU5THBcwpnManZk2J7kOLp3SfOS0hx4fsyDTGMn OgZ2YlI8h12i6zYu9qwRG138/f59eJwoVburMpCZEYMwSmch8MXmWH0sG1M5l1/66zNj st+YOmpmT0YObALwu1Tl0gD4NQtvkbu+YqRlg+COCUcE+ww8DRYfPnEYr71v/LJZwVu7 cT83/J2X5RN4g1tW6RiR35zJ2Z5QrHOjKB5pSbHaxbVXBP1QHfjRxQa39sKwAV9abLPH LjPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693296704; x=1693901504; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ypiT9aaPKimQJTb8EA0BWrlL9wmne7HVxtb4pVgUIM8=; b=FdZ6p2QvgTNX4TaQesWa75xQcvir34VWvtXeSDQI5mzxYfX4o8eCG2yTddjD6q4gdk 30wmUIdM9F+YKEIwwf1NVeu1JBA22RSpvwlpSHFtH0DJGu1xDGfjSafctLFdC7RvVxju EUYalVpZAjQWDcdelcagACU9ZaM3SS8GPVj8MuEYSxy0P+5bkalc+ZeF+Fas/jGy56TU rCXR6pfoKHLz2bqMozxQXNTr5DzGBcDJfMqj4JL8598kbV0jh+HnfvvpyxBPwFMY5xMV BDl8qnxJRiTs3c7iXmoPEd+tApjuIIFBevO1c2Q6VYRzCEt5Kdd/c/KDqiXGUiuha3lW jmFQ== X-Gm-Message-State: AOJu0YxMAFw4p6BOVE5sr9lwR4VN56rZKfIXhE95qCTYb6/UMYx5j/yy yDHQF1j8gQs5yaF7HqmdodiaDD4ZfDiF7A== X-Google-Smtp-Source: AGHT+IHqzbyj6s/BxrQ5oaFFpJBDX6ZRBgP+tCh/LHuBIb/Nh93Z2Cj0Jn1Swx19I1gj4HbSrdGRXQ== X-Received: by 2002:a05:6512:2524:b0:4f8:766f:8dc3 with SMTP id be36-20020a056512252400b004f8766f8dc3mr20893792lfb.32.1693296703823; Tue, 29 Aug 2023 01:11:43 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id f25-20020a19ae19000000b004fbad341442sm1868026lfc.97.2023.08.29.01.11.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Aug 2023 01:11:43 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH v2 0/9] Mitigate a vmap lock contention v2 Date: Tue, 29 Aug 2023 10:11:33 +0200 Message-Id: <20230829081142.3619-1-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-Rspamd-Queue-Id: 28E9240003 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 6ghin5eqdhugdsnz1ny7gkqu4yhwxc4p X-HE-Tag: 1693296705-963054 X-HE-Meta: U2FsdGVkX187SFBzruUYP1XGB1doKLiQfh7gCw4w2nBcdXereVq4sFkMusPNDB+IDWV+TqxmzpthR6z12FujrzQooWcE1z2KAo0KVe7/rvgVVDLdNCqc+T9/eJYXNYvqSRcjFMomxmMpbmFlwS52L3Q4yuaVo0eZmKQhYrPrgycor+lupzxdxhJpRJuVGCkNjMJF7afnIhd+gcWAOeTCgXls8A8pouEbzih2SZ36r1I6J07RD6AadKyagsIAYNG2VJTFbUortIOWCJ49fohr9V9xBQSxRvMtlEAOPfxkDmjslZPRWxImnSBDHTLqtEkDQPWTZ0CsK4Kli5CdAUF+HQBrYtzPpwd5YPLfd4EZKT5B64+IH+etyST2XUkntzH1ZzEYiNpDlJRu2lfdjmPQVWYm6oHe0lODhr1FRqt/tv6xRJ5Gte92+fTd8oH86ClOMYgjpX5sRe3LSEkyMdmdJ8ix17VnIHJCyfdSdU2UYVQbJSOFdG373dEHLQGEUCs83I3S3akWgVhp6BQOrYHmSZSOPwXlxgXtEsB6zVhciXjzIe1WB1pCVyifoM4Su4dhceYtY3HLqf9DYMBFEWn83+POKveoevemiJ0L52Cgixct6BIdDrqzPkHGVgJj155l09B0bGZ2Hbc66J6a7bUg/cYTiMTOPw+fyrvSAitH1+ERIxVzE5yPrBbxFgSdcLhoV782ma4CWgJQrR5LIWil1+k1vIyhrMO3sFdOLDhfYBvrrHMqtQSh4b3/dm+kOKuR+wOiUT0ZH0cSJVO3Y7HkKoOMnfuhpmTQ/GWHNC7BnBsX7V1tghjxfHaNLWBVbe3dBDTDsqIX8V8cNrPz11bIgWMbRfjuw4dcim0yL4Ls1FdbYD/H6V/0GVWnnzBwDNMwC6qeoSKUDW71jWBindxJKZgHD6VqmmWryUJHu8Zp0oEjdKz+qngIk1xcKY2QRpLlYeo7Igen1f2/2kSRLvs FeBEBCYl 3Rd9g1Ozf0OElZiHjITpFQ7+1SpSDb5zn/pS+FSUIZnOf+A+fHDuLsOViRhY1wtYCeVI1r3sKeW2rAdvZQoQLpV/xU1ddGa3Xjii4wOlZfrVCRNs9UGiyALBCITUNOG8SF8Y97EH20/ogfcVgawQSEAzv1XHqxwAwhJOtRgC7WJ4z4rogvdQp4gO5/09UnX+MmutABLGFqRxhFxnIJ0v6qDA/yYE2vAOc5LtcfiflqOURZ8ahctLRikkFTWGM5ch1dxixpZSMu0lM51W2qajYXlHuJWKzCFFKp4wfeER441MUZMuwnVvoDMg/D+zKkzEdsn5Qff2WspoJW2LwOMWdMtZsIM5xWat4urcJsVBuGVBVbZEz4H2zU++e9VvphuaWyDMLkqZfO53/w33vmRQ3Qg5ahrc0HYl0ccYCUYJVExAXFwNA1Fm6kZIYaDBd2zKF/jzG9F3BNXPBFZWtCrmsjuDtH8UT9mgSN3COenb0/C9fHOM/0ffproYBbA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, folk! This is the v2, the series which tends to minimize the vmap lock contention. It is based on the tag: v6.5-rc6. Here you can find a documentation about it: wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env_v2.pdf even though it is a bit outdated(it follows v1), it still gives a good overview on the problem and how it can be solved. On demand and by request i can update it. The v1 is here: https://lore.kernel.org/linux-mm/ZIAqojPKjChJTssg@pc636/T/ Delta v1 -> v2: - open coded locking; - switch to array of nodes instead of per-cpu definition; - density is 2 cores per one node(not equal to number of CPUs); - VAs first go back(free path) to an owner node and later to a global heap if a block is fully freed, nid is saved in va->flags; - add helpers to drain lazily-freed areas faster, if high pressure; - picked al Reviewed-by. Test on AMD Ryzen Threadripper 3970X 32-Core Processor: sudo ./test_vmalloc.sh run_test_mask=127 nr_threads=64 94.17% 0.90% [kernel] [k] _raw_spin_lock 93.27% 93.05% [kernel] [k] native_queued_spin_lock_slowpath 74.69% 0.25% [kernel] [k] __vmalloc_node_range 72.64% 0.01% [kernel] [k] __get_vm_area_node 72.04% 0.89% [kernel] [k] alloc_vmap_area 42.17% 0.00% [kernel] [k] vmalloc 32.53% 0.00% [kernel] [k] __vmalloc_node 24.91% 0.25% [kernel] [k] vfree 24.32% 0.01% [kernel] [k] remove_vm_area 22.63% 0.21% [kernel] [k] find_unlink_vmap_area 15.51% 0.00% [unknown] [k] 0xffffffffc09a74ac 14.35% 0.00% [kernel] [k] ret_from_fork_asm 14.35% 0.00% [kernel] [k] ret_from_fork 14.35% 0.00% [kernel] [k] kthread vs 74.32% 2.42% [kernel] [k] __vmalloc_node_range 69.58% 0.01% [kernel] [k] vmalloc 54.21% 1.17% [kernel] [k] __alloc_pages_bulk 48.13% 47.91% [kernel] [k] clear_page_orig 43.60% 0.01% [unknown] [k] 0xffffffffc082f16f 32.06% 0.00% [kernel] [k] ret_from_fork_asm 32.06% 0.00% [kernel] [k] ret_from_fork 32.06% 0.00% [kernel] [k] kthread 31.30% 0.00% [unknown] [k] 0xffffffffc082f889 22.98% 4.16% [kernel] [k] vfree 14.36% 0.28% [kernel] [k] __get_vm_area_node 13.43% 3.35% [kernel] [k] alloc_vmap_area 10.86% 0.04% [kernel] [k] remove_vm_area 8.89% 2.75% [kernel] [k] _raw_spin_lock 7.19% 0.00% [unknown] [k] 0xffffffffc082fba3 6.65% 1.37% [kernel] [k] free_unref_page 6.13% 6.11% [kernel] [k] native_queued_spin_lock_slowpath On smaller systems, for example, 8xCPU Hikey960 board the contention is not that high and is approximately ~16 percent. Uladzislau Rezki (Sony) (9): mm: vmalloc: Add va_alloc() helper mm: vmalloc: Rename adjust_va_to_fit_type() function mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c mm: vmalloc: Remove global vmap_area_root rb-tree mm: vmalloc: Remove global purge_vmap_area_root rb-tree mm: vmalloc: Offload free_vmap_area_lock lock mm: vmalloc: Support multiple nodes in vread_iter mm: vmalloc: Support multiple nodes in vmallocinfo mm: vmalloc: Set nr_nodes/node_size based on CPU-cores mm/vmalloc.c | 929 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 683 insertions(+), 246 deletions(-)