From patchwork Mon May 22 11:08:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 13250144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAFDAC7EE2D for ; Mon, 22 May 2023 11:09:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F3D3280005; Mon, 22 May 2023 07:08:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57D9F280002; Mon, 22 May 2023 07:08:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CE85280005; Mon, 22 May 2023 07:08:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2A78D280002 for ; Mon, 22 May 2023 07:08:59 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C93A3120540 for ; Mon, 22 May 2023 11:08:58 +0000 (UTC) X-FDA: 80817618756.24.9D23464 Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by imf13.hostedemail.com (Postfix) with ESMTP id D133E20002 for ; Mon, 22 May 2023 11:08:56 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=iDb9tK4T; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.51 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753736; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=063EFU5/djz5xU31OQ5ZuEX3cdI8++KWedSs+gqqgUKHNVBRV5jy6qg+KZX+Awft5aqy02 HkeqbsT46ssII0Yn1pLWSwpf+knsnqrbLUsE+nY89IlutR6u3s6KBZuQnJ9FV7iJGcput8 78kR+E4ErPqwpEQyzTnMevMrclpoROQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=iDb9tK4T; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.51 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753736; a=rsa-sha256; cv=none; b=4If/iiQXyiFxY38U50yNwAP5b0+zOf/4qIxSo6BvyqIzU5Dgy6pUyiojeT1fcIe9xa0fSE ljRNloc441igOJxvDe8pBWMANfU4tOOJst7NWYj+yE5ufX+kwHVGfQ4U2xM+78y+hkoLzz of6dqiN4kE94vYGyYm1I2mAdJB1SR/Y= Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-965ac4dd11bso1180534866b.2 for ; Mon, 22 May 2023 04:08:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753735; x=1687345735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=iDb9tK4TOgaJNMq61xrRvzxoVUaPP0VvNEnz6NrY+zacf0FfJnwnuypFPmwH5SLNBS GyEOPDWFsssngIu4YWW0FrNovP9ZjqfzkUf/51MOpeMHMX80L0GnQUdqQZEtrUUDgAN5 CqpASsjwkoQ7U63fy74gThmTbQvI9Ie3XaHwC17zsSXEk9lHt40AkqJDIcGVhr55gFza Ab5qRCRkiZfB7QM1qkbc7/2z6/RGesk6zCBFvjSfwFLXPP4gxqH+BsOk1j0UanBK1aWA e+0Ct1vWALCAGVP46sp1nfM2tzxvrxxrGXz0I5hY5eIN0kmUd5nEpcG0c3MTS3rmdd8s WEcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753735; x=1687345735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=NpBr7Aul5t2rE5AGv6l9WKJWiEC8fdmvNYGQSZ9fjdHzTusqcNJBaxcw+0o2hsIxZB L4v1swHOR1pP9p4XHAIpPivWbxXz6Kqi9HZpvNDuM9SZUmKpYZ8hdRsN/88OkVEiioCm uEgJnZep0o/+OhdlY797lsnK4bYlEbn7nzUzrJSuCUBOSN+WjPPi+rC1sxb3KnR7Opzd uhh3U7iqrQoP1WvKobZJoLZ299jsyaeS1vay8HHS+T90TcIE5SHHgv5/bSCH672ZnTca YDy0ycQC0tCuYHSVkbpBqHGpXGSQrPhHa5q43jIIZuYqMjZ3K81A9rWU0jrjoZtLK7Cc +7qQ== X-Gm-Message-State: AC+VfDwxV3SzfmRqiR3P9//fgXxrnquqGln4/K6M5TvdA9qPp7Tl3gOV b/Wemeaw/B1h7XccUiwYa0rmot2EGgx3tA== X-Google-Smtp-Source: ACHHUZ5RSuVJjDArE0w69xzADXBpvX8mfpX9Veg7DqUbJpCMdvLgHaYw03z8mFT9WRUtE0g9JS1nlg== X-Received: by 2002:a17:907:2d0e:b0:96f:cde5:5f5e with SMTP id gs14-20020a1709072d0e00b0096fcde55f5emr3930140ejc.29.1684753735155; Mon, 22 May 2023 04:08:55 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 4/9] mm: vmalloc: Add a per-CPU-zone infrastructure Date: Mon, 22 May 2023 13:08:44 +0200 Message-Id: <20230522110849.2921-5-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D133E20002 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: kz6sopgk95qtrw71pqcu4c7apqcendbq X-HE-Tag: 1684753736-422876 X-HE-Meta: U2FsdGVkX1+PQ2Sr0EYjIyHPbn3mDqTRMRVOtZ/IMVKeU6zVWba9lhmWVL7GhKyjticvJFD1WGbArvprDEh3Gal4bK9a//56IxcWvsaS84FusMfp5jhllRMgPkxQOz2sHfNZ+MG8042tGMrLBkqvIbNyVRbe0x7naMNX/9FF4IzdRYtWHTBgCI9M4INwnG0iXm0FsUvZL+k7+z4vyicl6f+DFY2VarQ+yAcFnLs+Mv9ha/MkXgkV+hmxvzDxI9hJZofLK1vZErk+KAIAJ6l6A858r/6nSpPUJwwP/qt0O7BV9/89mXCFw9KeCwcd0F7UR96pNc/AO6kcgSZOunVDlZust0T97BoRLQPA6JHIAO/MnNdRiYwzC/B0sh9omeF0KhG1E3SbVrQlHIJ7ebveXouCbwOMaqbReB07FBfNnhmAw35zfQeEICKCssiRO9yG8BW7t1EjmQvqgLMGXzyTOUcsjnQDpIgIWC7RoIBI69rgp2AgIOvYKTchP9o3xgH5t5eAitQp3BDVvnKzTw1M2XvthVevoJxEeH1Dty+DLpo08JfVUH1mxRbS4fclm9ThrwFxiof/KhU++Ef6Ro7GHe24EcbkY66Mr9vRAFqnhpebzJfLV3YPpAU/7ILdnbN/D9nIBjyx1/ZvPIKjIRznLNVyTBXuWA5i0977R8zKobXKIlIV95lljCXmV8oEEnRCwt+nyd3ZvaQNFxvgk36B2u4tI1lkxiOeMvlUxKpKlQsDRiagZiBmnQwygkfqotlX8JHhn+zPDsxO5d9yFd0aHkK6fyxnDvKRHWyaSJLXSGW9mXMVBra/h+Imbh8wtdyxLpGwObFVqWiFc4eDq11loMDCseYc1sd8HBI9sLVr5zl6LIEVooI0vmPmpGjN4OlRbHYqDGSPDKPQsLLB6ZkHYueEyyjIcEP9CJxyUJFm0nrDsPI5PdpULAES4ABjnQN0ifEXm3aCE+653VOsOe8 JFmvOfJo XdJcuaWN5PRx+DInIMYY/ZW1/wYNjN+Xd91CZg1FbP81+s+u+yXoxtOHA0rRhaFhepntrLIS/1D0cOTP2oceWKq9R3M1R6fbTGKt9oYsyGyINySRlAIF9S/4edX9To/wjXlWMp3bdbqAzuEKGggrNZ9J/bH49PMV470qwCjhtyPPksIuyHKseB5FxM8tXnTvXASZF8/TcWfD5QRU+uAUJkZCDrDg8hQGnmRJnUftDXneI0AumX3hXFIwQXj7PbAPCDwKCCoKExRCUueXZ7itfcvqRv5PjPZz63v8KJ4A7bbvJN4LYCvs7Bx+AD1JjRbXrTS4UBtIU3J9C36CAV88xA86+FZlmu7i2P3/4tHBzQSic7SO7G4bWqWpRLIfDei4PSCPywFFlRniFREYhYKItwEUG6nni+Wefq/IGHsl4dnvvwqeclPYcT2W0tL+gIFN7FJVE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a basic skeleton that defines a cache entity for a CPU. It consist of bookkeeping data which belong to a CPU zone. A CPU-zone defines a virtual address range with its fixed size. This approach gives a possibility to map any address to a particular CPU-zone it belongs to. Below is a high-level description how it is used in alloc and free paths: alloc-path: in-front-pre-fetch occurs if a cache is empty. A chunk is placed to a free-tree of this CPU. Finally this CPU cache is clipped in order to accomplish a request. A fresh VA resides in a busy-tree of specific CPU-zone it is bound to. free-path: va->va_start is converted to a zone, so a lookup occurs in a bookkeeping zone it belongs to. Once it gets removed a VA is moved to a lazy-tree. It is freed after a TLB flushing. A "cvz_size", that reflects a zone size, is set to ULONG_MAX by this patch. The reason is we would like to follow an old behavior until all dependent changes are in place. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 19dcffb0d563..f6da2590b4de 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -773,6 +773,61 @@ static struct rb_root free_vmap_area_root = RB_ROOT; */ static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); +/* + * A per-cpu effective vmap cache logic. It allows to pre-fetch + * a memory vmap chunk with a specified size. A bookkeeping data + * is stored and accessed per-CPU. + */ +enum { FREE = 0, BUSY, LAZY, NFBL }; + +#define fbl(z, i, m) z->fbl[i].m +#define fbl_root(z, i) fbl(z, i, root) +#define fbl_head(z, i) fbl(z, i, head) + +#define fbl_lock(z, i) spin_lock(&fbl(z, i, lock)) +#define fbl_unlock(z, i) spin_unlock(&fbl(z, i, lock)) + +struct cpu_vmap_zone { + /* + * FREE, BUSY, LAZY bookkeeping data of this CPU zone. + */ + struct { + struct rb_root root; + struct list_head head; + spinlock_t lock; + } fbl[NFBL]; + + /* + * A list of outstanding lazily-freed areas, ready to + * be drained, which belong(address space) to this CPU + * zone. + */ + struct list_head purge_list; + + /* + * It controls that only one CPU can pre-fetch this + * CPU zone. + */ + atomic_t fill_in_progress; +}; + +static DEFINE_PER_CPU(struct cpu_vmap_zone, cpu_vmap_zone); + +/* Disable a per-cpu caching. */ +static __read_mostly unsigned long cvz_size = ULONG_MAX; + +static inline unsigned int +addr_to_cpu(unsigned long addr) +{ + return (addr / cvz_size) % num_possible_cpus(); +} + +static inline struct cpu_vmap_zone * +__maybe_unused addr_to_cvz(unsigned long addr) +{ + return &per_cpu(cpu_vmap_zone, addr_to_cpu(addr)); +} + static __always_inline unsigned long va_size(struct vmap_area *va) {