From patchwork Mon Oct 14 20:36:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13835444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73044D18145 for ; Mon, 14 Oct 2024 20:36:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D78E6B0082; Mon, 14 Oct 2024 16:36:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0874A6B0083; Mon, 14 Oct 2024 16:36:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB86D6B0089; Mon, 14 Oct 2024 16:36:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CD80D6B0082 for ; Mon, 14 Oct 2024 16:36:53 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6860DAC15A for ; Mon, 14 Oct 2024 20:36:37 +0000 (UTC) X-FDA: 82673366454.08.A5EE659 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 602FD40014 for ; Mon, 14 Oct 2024 20:36:48 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TdLJs3IG; spf=pass (imf12.hostedemail.com: domain of 34oANZwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=34oANZwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728938165; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=4U5CVtkJEYe6S9rzoTpLNep+Y3sHiaJ7kUZVFdWaC+c=; b=pg2XjLzkvgArLXUCjJRwzRf00ry0FJ8JXp9OgcKPY316HhOCUf7zhHzDXKJEZWfHgk3upb VwQFdW16noyoFIKnehmbeYT4wUHklBS5DKa1pupCE9v2NTFcfOO/qY1+wtUw9e0rIvqO7K Xd1QPuC2lhe/1dVtgTqW1maOCyZBg3Q= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TdLJs3IG; spf=pass (imf12.hostedemail.com: domain of 34oANZwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=34oANZwYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728938165; a=rsa-sha256; cv=none; b=CovaiiSbwLAF9lLjWlaFw2gv0w/uQDCvKF5OjLJTxO6BCgXZw7KJ8N2sMbvP4CELQq3IAJ nMaM+/ykqzeayutwZ5Vi9OgRg3enmskjl7c0tXWCO4H+n+U0KJsvuNbq75FdBjRj/80sCD im3eJXd7AH7aFPHWPT48Sg1Z6QZDLO4= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e292d801e59so3911196276.0 for ; Mon, 14 Oct 2024 13:36:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728938210; x=1729543010; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=4U5CVtkJEYe6S9rzoTpLNep+Y3sHiaJ7kUZVFdWaC+c=; b=TdLJs3IGftS5JIeCUJ7l4LaL+m599YmCwI6agKN5zJwAgivP5B2rA6HEFU05euqy7Y i6i0kDAgJAa5kJhI4rrT7P1FwVn8kjJlvmL+qYBXCM5f+NMExlCg02UP396kbkN3TQPr jzInskq7q7FCAQD6JJESaU6PjaTvxOG5cLZ6y29Ugat1YAJbwX2p/9RRbyBsVpmRH/0G JTXa1z3kqbg94zd5iOgTccI02szCYbr7VflWz9rN/QT+d3ucgYSMmzLE1sk3j3JJNduT 7htb+I5+KRyZ4HUufp1Xlx3+NEEhsrkBhWuiXRvEveYW3yPOVK9SA+zsKx1Hx0huJHlD IH9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728938210; x=1729543010; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=4U5CVtkJEYe6S9rzoTpLNep+Y3sHiaJ7kUZVFdWaC+c=; b=GfUY6b3eLB/ZMqbK8tlw6drlumqb8i5+Vjfp187fQ+36USnd14//dUSSJiUOxnnugO WYzRsRW9srygpKqeXGdp2MyjoGKzcituawvUH6lUY1MFWs58gd9dmiROBjycf7RHuRfD in42TuXJj8qOcKF2ffSpSbA23DPxVCz2g3IjQ/c5KGHZjbbWkY4mFBrinCxL/JFARpNY wIAbwElYz1zmjgr48yfnfgXJ4GgzEWUKEANwP8rlByqH8C5kfPpMGjuN0izb3FMXJHyI iIf3H3FnTpDau3mTBwaMm/KYjm876lkhSCMbl3usdZKtuwtWJglmdb6o7TJyfGtU6eM6 tWSg== X-Forwarded-Encrypted: i=1; AJvYcCXJ2aU8y4+KALCbgeQ3QF2hMO+1DMZhE3jtx616tQP60twY8J5VpZbHljWloN3Mu9SKUL4XdOHACA==@kvack.org X-Gm-Message-State: AOJu0YyjoDOeMjrFGSyCqjN+wFD9ms7RfV0YsnXEYTEkKBCNnC3yhjXy pksZg2JKfnKKxslXxQfLAYtq3jdzC+K7t/kKiVDe41fKj//uVCpDpE63OBcaCjOUeMjkqPhZC2P E7w== X-Google-Smtp-Source: AGHT+IH6bves0lxiwLGhNCCSLFxMFIA4Y1dq4YMuUh/nUpGxr/Xuzy4ZC8csJ8diCdDybxEmCtvJmFEe0+k= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:915:bdd7:e08a:7997]) (user=surenb job=sendgmr) by 2002:a25:c593:0:b0:e28:ede2:d060 with SMTP id 3f1490d57ef6-e2919dd2c51mr8678276.4.1728938210069; Mon, 14 Oct 2024 13:36:50 -0700 (PDT) Date: Mon, 14 Oct 2024 13:36:41 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241014203646.1952505-1-surenb@google.com> Subject: [PATCH v3 0/5] page allocation tag compression From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Stat-Signature: b5z4wtdbstbx5cowwma8c3rragcbxz1t X-Rspamd-Queue-Id: 602FD40014 X-Rspamd-Server: rspam11 X-HE-Tag: 1728938208-633126 X-HE-Meta: U2FsdGVkX18UgcKW815yPWHl4NZBI7BFXzFukX8WjU9pOz6QSnocswRIrQFSKLKumkyBfcsjwOWOp+pBoss9Kp9XBlHc20EkQwGf1YRyPoQTcuC9l8lZYw2YViPCu1OVK3FdKyeNZYT3iN59JVqWgdv53gvWTmnuDD5ZQZorCbR9k469r0d5fBdaJVUaQP5Jh8ScxVZskcCN1ySDRzLcuh7WL5Nn+jD2Jo6BdkUjzClWXB6Lr/nJDcSB2F5YMMZwunpSAxlbugIvMDSxvLUblo2wvmkQy6CnkcjmTRKT+xDOnaDu4Wl6WVZOgzgH8qMuY45AkOUMHpSijGnTCr41mJ05lE/wpDyJd127fFDCYK+60v3iZEIB1p31DbY9wzFC3R0XWo/YkOh5RGRPGqRNLVGFAJ8PhoEGrL6BPtPSktkAwuSjDI4fFjgdcVtR5aCs0aGeiIctDHXnHUXc+2QRCa0KjiJ4jgGlg3rzoOtrSlJXXt4cepqNysry6EgrqgDeqzT2uOd7umoqMQbKhUF5IksoOGZ3zqd1EC2xyH9XzRTrSscZgo87GATp7/tOT0iF9Q3ZypF15qA1a/bkfFOnkudS24FHIUIexwLRlnfeLHZt8IjwlmwIqxS3nb9GwqXOQfU94/Imgdiz/1IyCaEE4sUNozMjQqNGgv7LnLJqX5zJVR/MP1bR1NaPyQ2ahOoyBKPsV1FTDb3GtsSEsLGOBvSse64Gwh/+w5iQ1Voeh69DRq4tQkT7l/j3q5tz/xYjbjrsgGxuJsxIwqokNcT3NV9veeloFlyoeXUIy5eHyHf8hBzzV6XMualSgP54Q7Q4hQS+e08oQhRPfTbqOP1KUvp4DMqi18lzXgYQtYPRyhthvxAWSkRytpDPh1X2wrvg2GMlxPpjdaKXxIpEOyKN9gfUcxVkYNx/nxFvG5Ai3TuIHW41ei350OlxYSolWtHZEFfu8xuWIr5qdFFf3tF 5xays35W BbC9XZ4eDb30hb2P6X+uAfvmJcdr/ht+jbF1+gqGA9I/btXMqowi5flm/xGrrk4OFsPCdnJfPixdGD0jebAGoKMK8wWXSu5y+r1sRxQnyvjRMYwhiZfrsz3nPR8NGxFdNRvJjlNd7CxIIqHHmXJaQ0x90Q0syD/cUO4oRzffB1zjwi3G7gLIWyTPcQcs3gSqnypZSdlRVX+JzY/rmLUlPAap5rt+jIBp0SghzIfCI9THltF976Qel2BqXgzjpiTX28iLnDgIBT9MzmDm+JrLS9lOpME/+8hqxEV6TCTYgLbiouyZp4fXLtSI65ngzlZB+AyB4O9V5BUZy2Nj9ek1+v0PG53ASGZagKm6kBnkWNgnmUhK/8wWYNGCs7MxQEae77H3TRnhUgGZ8pvYush24LVR8Sf9D33uMd1mclYuePSX2AZom0F5CCBM0v7WYu+7NHJdCvTlkINtcylb/VL4Al5L6748jH7JtzWuCv+VgJFGM3UeHHyQ9dREmjDKlRlODUU22Gopze6amNy3RkthFgeI25k/gnqMlXHWS5GlszsAoR28ctjCtv290qvI9QVPR4Fqa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patchset implements several improvements: 1. Gracefully handles module unloading while there are used allocations allocated from that module; 2. Provides an option to store page allocation tag references in the page flags, removing dependency on page extensions and eliminating the memory overhead from storing page allocation references (~0.2% of total system memory). This also improves page allocation performance when CONFIG_MEM_ALLOC_PROFILING is enabled by eliminating page extension lookup. Page allocation performance overhead is reduced from 41% to 5.5%. Patch #1 introduces mas_for_each_rev() helper function. Patch #2 copies module tags into virtually contiguous memory which serves two purposes: - Lets us deal with the situation when module is unloaded while there are still live allocations from that module. Since we are using a copy version of the tags we can safely unload the module. Space and gaps in this contiguous memory are managed using a maple tree. - Enables simple indexing of the tags in the later patches. Patch #3 changes the way we allocate virtually contiguous memory for module tags to reserve only vitrual area and populate physical pages only as needed at module load time. Patch #4 abstracts page allocation tag reference to simplify later changes. Patch #5 adds a config to store page allocation tag references inside page flags if they fit. If the number of available page flag bits is insufficient to address all kernel allocations, profiling falls back to using page extensions with an appropriate warning. Patchset applies to mm-unstable. Changes since v2 [1]: - removed extra configs, leaving only CONFIG_PGALLOC_TAG_USE_PAGEFLAGS yes/no option, per Andrew Morton - populate physical memory for module tags only as needed, per Pasha Tatashin [1] https://lore.kernel.org/all/20240902044128.664075-1-surenb@google.com/ Suren Baghdasaryan (5): maple_tree: add mas_for_each_rev() helper alloc_tag: load module tags into separate contiguous memory alloc_tag: populate memory for module tags as needed alloc_tag: introduce pgalloc_tag_ref to abstract page tag references alloc_tag: config to store page allocation tag refs in page flags include/asm-generic/codetag.lds.h | 19 ++ include/linux/alloc_tag.h | 21 +- include/linux/codetag.h | 40 ++- include/linux/execmem.h | 11 + include/linux/maple_tree.h | 14 ++ include/linux/mm.h | 25 +- include/linux/page-flags-layout.h | 7 + include/linux/pgalloc_tag.h | 278 ++++++++++++++++++--- include/linux/vmalloc.h | 9 + kernel/module/main.c | 74 ++++-- lib/Kconfig.debug | 19 ++ lib/alloc_tag.c | 394 ++++++++++++++++++++++++++++-- lib/codetag.c | 104 +++++++- mm/execmem.c | 16 ++ mm/mm_init.c | 5 +- mm/vmalloc.c | 4 +- scripts/module.lds.S | 5 +- 17 files changed, 931 insertions(+), 114 deletions(-) base-commit: 828d7267c42c2aab3877c08b4bb00b1e56769557