From patchwork Mon Oct 14 20:36:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13835445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B94AD18144 for ; Mon, 14 Oct 2024 20:36:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCF706B0083; Mon, 14 Oct 2024 16:36:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA7946B0088; Mon, 14 Oct 2024 16:36:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFAB36B0089; Mon, 14 Oct 2024 16:36:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A12446B0083 for ; Mon, 14 Oct 2024 16:36:55 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7C259161266 for ; Mon, 14 Oct 2024 20:36:46 +0000 (UTC) X-FDA: 82673366622.10.7166473 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf08.hostedemail.com (Postfix) with ESMTP id 0F772160005 for ; Mon, 14 Oct 2024 20:36:48 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=L6c8wyum; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 35IANZwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=35IANZwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728938097; a=rsa-sha256; cv=none; b=hjzYrXNPXy8jkPw8tIWvqIdX0EXBhuAvFXhQ0kedYvre8QB1aKNvtH7HyHUd49+LKq+2fx Cj8MzmmxSMKJDV+0AgzTecaR308puwIJkyFtyjIF5EGWpldBq+nWEgEc+FTmxgsaBNZSgD RMIVdDG25/1/XczXpa3uwblIfJcwsNc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=L6c8wyum; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 35IANZwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=35IANZwYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728938097; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OBL7Vs4MVDC9S11JwgZ8NOTxmnHjmsqVGP9nKZ/sLKY=; b=eNpzjfuYGDoFtTr0AKzuzDkqtyvFtjQZHtlkAWOLWapQblLufGLX8n9hNmaH7kSL/euC9b M3n38rWB6Bc/oJxeEPx/wfLWFb/3g+sayQMdmNBqEWV4Kp84UUAjX65ca4vUimsTNAvN9i hjmZ33FzCoOmaleLqnkwpHQfuobwwvQ= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6886cd07673so85741907b3.3 for ; Mon, 14 Oct 2024 13:36:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728938213; x=1729543013; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OBL7Vs4MVDC9S11JwgZ8NOTxmnHjmsqVGP9nKZ/sLKY=; b=L6c8wyum1ny6ENQzvP36JPSzuSVohnE2bjnhsz2tlNWbOKqH16l0jMKBwPEcaRrbBl ho91zrE7H8VZ1LepFGGcuVBKle94u0gS+1uKL7Ep4pdaDc0jf10clDiIUTDRW8/qXuQm JKMGcNmuEufkqVM6GBww4/XsoaXmebIudGWxF6XJXNnf+OEptrDweElAnGyRkw9RbocH bNfG3p8RucEQP/Sm2BK1R6s5r06+iC6+bbiZ5DqjCvRqdclwh0Ukzkm9Agu3HHj1MpmW 3N6+y5XXGvtbdLU5TRplmByXGEj5qmfYvG7JQGdVhUbxtoRiJsNR9yNETNP2qozQ9iWw NipQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728938213; x=1729543013; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OBL7Vs4MVDC9S11JwgZ8NOTxmnHjmsqVGP9nKZ/sLKY=; b=pmSLmdTZZLu6m/Kzcr+CFQ4imhVjt6oAh1aTQytDGSWwZFT8Gc8MVGHRWyuzcMECk6 oHqoVnXCiuVB6vShOi9UX0StI3kTlP4WZfAv/f0itCytiC++RCik50dKlwGikleiOSkr 4AVJVJmcK/dP+NtCqYN2yns0hlLpa/O1CtsZAeWLUiq72DVvdqF+uNS8msAshsBp3tws FqBVz6ZrL8vebq6hS5RhMNVBL6/yVSGOtJG/bAWKvDaq+F3/npWiiPgX/A5sssM4whz3 SyOBJsSapZS2vrzXv9owwMuxIwowpMtltzNe5zJ98m6Wo5zYvrYMMdNeaiPWGcVHdHXe KOWQ== X-Forwarded-Encrypted: i=1; AJvYcCXJBpDXR9BVwGcEp40Yz0WKz/YSrXbG50+nmchM/lb4cwkhZOLvUzMJ4X/hbTmB9ZdrWNYNRQI7mg==@kvack.org X-Gm-Message-State: AOJu0YyVOjYD1nFyRQESfVDMnLD1kWts/uGuFwEi5k05O1Az5Svc1tyB XD2TCux8Kq519tjy+12Xd+FkRjtW3l+d+MRg81h/vUfjAs7M8i2eUjjHhnpTmcZKYE3HBWSUDR0 ehQ== X-Google-Smtp-Source: AGHT+IF2H6wJeGtAPLdmfc0iDyPr+v9IGwfu4WR2zrx8HQDicf0QvWblIiwnCcZ6i4e+0edwH2bII6UptCk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:915:bdd7:e08a:7997]) (user=surenb job=sendgmr) by 2002:a05:690c:7446:b0:62c:f6fd:5401 with SMTP id 00721157ae682-6e347b4b2edmr460147b3.6.1728938212594; Mon, 14 Oct 2024 13:36:52 -0700 (PDT) Date: Mon, 14 Oct 2024 13:36:42 -0700 In-Reply-To: <20241014203646.1952505-1-surenb@google.com> Mime-Version: 1.0 References: <20241014203646.1952505-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241014203646.1952505-2-surenb@google.com> Subject: [PATCH v3 1/5] maple_tree: add mas_for_each_rev() helper From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0F772160005 X-Stat-Signature: jf4a7ioyybosqi8km6qobsrb1guacd49 X-Rspam-User: X-HE-Tag: 1728938208-707914 X-HE-Meta: U2FsdGVkX18UZcECgk2K3IPZGLXmO/hl9Ex4W0zHp1dM5YejizrrYktagEpboeXm0hNkcQIDJLxn5BRBRBccriPvvfX5uMBQm0VJNNQe2UYSL8e1s1IOk9cAXV7i1DOiA/qc1Dtbx6o4O6fY/93MiUR6S/P5vPy/LQTWkqKN2kkCJ4XZttt2acNiTec8uFYGwWlUVG1aOpEVZjrETX4eEZzFNjnIGRgINKp6hBoHrGZPDVh47ojJVa8+nqbPximq+gcEeLPhfidkxnNjdE330rjSQvdO0fIkThW9eK3ryLEwu0hlzqL0MprfcppVTdq2wOmng5a+ycQ+OjZR+N+qjsBIXhIN70kbMeVit4TPDUXBe5RV3BUPuTeuigDhd2qgZCvOAknfLOcsfR9CxEK7Hf4BXi08EWPfp0HzhhrzRsPdFErS8pM9HpMzH1FqRKlIEAohC3JWkqyKQcMln1Gs9UGTK/4/c3ze05xACQx4eRxLG4DCeUipGXIt3hwHGckEPuQ7CBDAHp6s2CPWsDwQ1FeYhS2QRUFRLYNcftD9n+OCU/pxIlfuVg4/RhXHewM76GmidVDNyk9RH5dOT3Jex+9VuUQ1guEECNPRYPjxWjdbyZL/IZoh2EMVIDOiJWxQs4oxWSSJWOSeGNMW9NgU/Niu2hNvJG6RWQRLQU3BjrSgDUP0JIt4P6ddP+r4TnozqJF6LE3TldWXtZcEM2uE+0EfDsrbPDgJtFXweNf0U3yMGk1gJRPcOE2vV4CVmdSyj98bCEgEwPwzKI41IbcUFt1dXVhV61D07QatfdVdfuiO7pVnI93kT7KlmqsmZxyWJs9ImCNdSZ1JTt5zu2bCWInKOAQL1hjB9YdJdFr5oOHMaeinplOBsKKbbz1/kE4whmcFVcP5KA53Enjn6JIeabZtPeeSlWOt9S3ih3ld7ZVPc2WiHUao4FF0hmI2TBSu5Yfs4b3JtBw9RymQcaw 5ejHjW5a YQqwqQJtQ7OAtkYqvBOCj9bGekoK4hoHVJKzxahngNabu463/LIAli7nF60QORMM81taA5GzVWus4i8SLhgPzqm3RqCsUPc9wgVpIKmhZYJlpVfh7bEZJg1pYTjFPCnedA2bHykH32N0HmUmdoLikGW2HKQLg45YNHFt+8rvfNFp3ssik1xeZu9a+kuYqz5CtOMUg6yIFkpBljKkZ2kaa6hK5iTnPPaWDLz4oohjXXoym0FCXBY79/xHsQKyXDWnlwlg2y1GYcz0X7Zm+S57c+IjgcqvPDBgtdxM0r3ZJnxeaxF7DueONofFDNA2fDHmPfhMTi1eh6zcnmZwjiVIP4GkZgVyfnNXCWwv1Nl1UbKEhuYBlB/WlOoewDfTMbptf3dq4kqz+zFWAXaApoartOQ/OLSBmLY98a/ySW+lOEwwmTSd4V9U0ydBahzJzPdygDwfIILPl+5Vb3rPwJhhK0npLkYVc5/RVcpqb8UI1GCUnPXJTm+3p48Igw6hW5f0w26ulE4DebS4ylOTrADlZUnel3GcP1MEfQOisLDfVGRx/9Lmh5knn3UgO/3h6upFl8EKoxFhC6mSBtXFK7axA9LMDX44bkxfzArss11J8F8eYy8M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add mas_for_each_rev() function to iterate maple tree nodes in reverse order. Suggested-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/maple_tree.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index c2c11004085e..e7e2caa1a95a 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -592,6 +592,20 @@ static __always_inline void mas_reset(struct ma_state *mas) #define mas_for_each(__mas, __entry, __max) \ while (((__entry) = mas_find((__mas), (__max))) != NULL) +/** + * mas_for_each_rev() - Iterate over a range of the maple tree in reverse order. + * @__mas: Maple Tree operation state (maple_state) + * @__entry: Entry retrieved from the tree + * @__min: minimum index to retrieve from the tree + * + * When returned, mas->index and mas->last will hold the entire range for the + * entry. + * + * Note: may return the zero entry. + */ +#define mas_for_each_rev(__mas, __entry, __min) \ + while (((__entry) = mas_find_rev((__mas), (__min))) != NULL) + #ifdef CONFIG_DEBUG_MAPLE_TREE enum mt_dump_format { mt_dump_dec, From patchwork Mon Oct 14 20:36:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13835446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2B1ED18145 for ; Mon, 14 Oct 2024 20:36:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45AD76B008A; Mon, 14 Oct 2024 16:36:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E32A6B008C; Mon, 14 Oct 2024 16:36:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2101A6B0092; Mon, 14 Oct 2024 16:36:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EA5146B008A for ; Mon, 14 Oct 2024 16:36:57 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 67980141231 for ; Mon, 14 Oct 2024 20:36:49 +0000 (UTC) X-FDA: 82673366706.05.99B32F4 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 7925640004 for ; Mon, 14 Oct 2024 20:36:50 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Qs92cugm; spf=pass (imf17.hostedemail.com: domain of 35oANZwYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=35oANZwYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728938169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TKWP9Sk5eHDUMKo1jGcXlNN0cIhyXRWL15bLbS6pMlc=; b=F+0AAYdi2hd3ztLLaPvKVNDCyIRyjykBWp0S5F70eGxt9h+lyInXAu3g1IqIpWZkisnQfi KMuSsCEM3hts94NjF19sjexeC+EnV0nNPuA2lTu/gA5ZRd7x0pOmACSg9QrxH/RqMDQcnA UGcDVO9ONKJi4KfyT7x/j5IL67yTk0M= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Qs92cugm; spf=pass (imf17.hostedemail.com: domain of 35oANZwYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=35oANZwYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728938169; a=rsa-sha256; cv=none; b=X0Ocy9fVTHCMscUvJ9M+vFeFKg6pwJE9763FC+vyqQvgoNKNYt8HI/VnPVj6CEUOuLZfy8 Qqunp1Cusdym9K4yeCp51EfwGNJHwGmj+hpZtCPLxBBs/aRkAqFe+PVAI/SyaGB7bMx9lI k6S6z79pqBCuROnumzXMDOpDaCCTCRc= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e29097b84c8so7226169276.3 for ; Mon, 14 Oct 2024 13:36:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728938215; x=1729543015; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TKWP9Sk5eHDUMKo1jGcXlNN0cIhyXRWL15bLbS6pMlc=; b=Qs92cugmPTDJp1eLSihoDFYOm8h8CwRkboo/EKvfXbd6DIRMfvPnBorawDWoQ2fSRx O+xUx2q0UqF6CxZ9knjHPSXBjD6oKpYBN3Re1rZNyDJticvPlCIGU/fonfI7w4LJzjXP sicoxpWIcSMzQFZhD0jRkXPayUIL4Cx87ifqwt1NTssGwoSelzbKLm72U58mjEDBKpF6 9cW9V3qjfs9Dd+92AZQbGeU5vVfwb/NKkpg6pshbnfKTU23phjEds9qKxmzbVWgu6vWd knscAnVhguoCVIN0j9o5NPJn1cvFjh8GRO38GKKEMLzPrHwPR11anfIvPbT2YDbhKGIq +13A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728938215; x=1729543015; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TKWP9Sk5eHDUMKo1jGcXlNN0cIhyXRWL15bLbS6pMlc=; b=rtE1D9BDwKVNUZpw20QcaYW35Lw+8uxFX0ftiSy65itpPRaX6WuOrx5gUk5gi12gBh fuclUsEhfkz6xBCoHupQVitPbcYDDm2pEVudbRIH651tMmP2gI5b9dC9n4KDFtOmoQL8 pbcTZPx2YK6lu/txSrKTHNdW2q/maXoaBUKnt991YV/iLkazZ4KDt0CZq/kYPDZPpQBI S6b13d0d38cpkbCzBEp8BjUWiJ0IjLJzm+gVrYADoksND3vWmGvePFUdb8gMYxoeMRmg KbVmvbjTZ67Xfte3rrVsB9iD/z3Evb7jn9V8coyn+LFM84q1mSRVjYSo6gYgFmEGTFTz EdqA== X-Forwarded-Encrypted: i=1; AJvYcCXaHntDaOpXNL7Wjl1ss+Hu/dfNhNNbF9lakUYxwfRXcLI30cZh2Aleyf7np4PmIREtjSmGMORtdg==@kvack.org X-Gm-Message-State: AOJu0YxRjRbTAmeTilUWZ6IOWVJnWo5sZE3D+uui41dm9KpohrBkRePN ne8ZNpxSieapqbGHAb52yTi4YOpuKAw3ZrvBPnzkJJn2gXvbHTrrPnEpKRKz/O+wYXnSFyYnMRI nqA== X-Google-Smtp-Source: AGHT+IGBNIqOR+aetsuJD2kNoXsUQNEGFlkkyZON6PVRCQ5jP5534CpNDF7V47lVUomGWIrldnmJpVdzxLU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:915:bdd7:e08a:7997]) (user=surenb job=sendgmr) by 2002:a25:d094:0:b0:e28:f6f6:81a5 with SMTP id 3f1490d57ef6-e2918e5da48mr9605276.0.1728938214624; Mon, 14 Oct 2024 13:36:54 -0700 (PDT) Date: Mon, 14 Oct 2024 13:36:43 -0700 In-Reply-To: <20241014203646.1952505-1-surenb@google.com> Mime-Version: 1.0 References: <20241014203646.1952505-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241014203646.1952505-3-surenb@google.com> Subject: [PATCH v3 2/5] alloc_tag: load module tags into separate contiguous memory From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Stat-Signature: oa9fdooiyrbi7hft1tqo7n91foucew8o X-Rspamd-Queue-Id: 7925640004 X-Rspamd-Server: rspam11 X-HE-Tag: 1728938210-883867 X-HE-Meta: U2FsdGVkX18oKLy0oqKSom3jwcuXZXeys8vaWk8vqJC0KR3g9GfXYKM9m57Gs62CRAXAANW3LDogKjCGC8gt9OQojs5hV5AAihXccQtQwbMnf+VuBKLaQBZo0iWF8pKpAhtjka70AVTvh2zdPfOQZPixCxKtnaLLt6y2yI+alo9AFq4pvNxouEs2MZCwmZLVri5gQKGMmR85ceS/3xaCpYHz+KVHT523+GCd1FWCQz20En1ewAdrzlTT2UFLapAlhWBTh4TxOhqWlb7HrexUK3Gpw79apY2UDyCsJf7bV1Ll1R5NwpfhNLjUxwsa2pE8UZsSgAJqxrnahk2MesI+oHdknYU7MwwzyOXdQpIQssD6Zw/AlOzL/AHteXRDWDCWRGwDiPrZOxOExeISuZNMjBFegt2Why86uX96214oALTVqYs3wHBub6DRJ+yKA2qsi93BRIb+wAZFdvFGdI1SW/Tn8X8e+OJKWECpIKdxJdK4ilJtue5PTWemxm1JBEIQtep05mfFKXe1IPCW1OkNMVWJ/feFkKJ56hJqm/rFMAepUKJ0Zrxnyi9UcmBHb7KrmzAs0/qNsAgqGnsoBAXilxajIG0dlGaFmk5y3ELsGxj75ghltjndP77DNTunsRVlrtZKH8cGklzLulL9DvzzJkMcfp8LzIrZtGUVla2ew47EpfoS1D3ZAyLERID2T1KNJ9oWvPcjKj3DIZ5ip7QwvEGNQ3xcnlyWMfCKe5TWUUOrHJRe9gDmmpZujtibFR1gXgKTaykxm9veVJnRSXNqLNE5vHJvzHH6JWTOp2Bf9rGNCwgv4tdGKjY3ivHRx1vxAVTxh1c3LsXvUGeGK1QPb+MIVnZd/l8mnL+xqkiYs5ylCwcAxvO7Kacc9JUPSUnfFv1lnPrprQW0ghuAwEXnQ75BfuxBU5U8xxSmHcVdHbxj7LC1/ZHW8NdJbIepphNhCRmEMSyOqigAZW/PqWI ClNQAgI6 E5wxiuDfO+VuJSG9oVozR9s14jP25QSZyG8frYfRtgQOl4ksRlTLjOQCrwRRfPb7BJDur+fDIUvRymVKoQiR3E71wmanZg00zZkicglUhD1NjunWW6i9Rzu1KL/LbAlPrTvtCmalDes/n4+hWOEl0NI/cGYBL9Nu+39Rp8rmWy0DLgcZWemrDo7Oj8E0xCwAaPOKdUHtJs979jn1F25nw9FmE48Bl+tFRPIe9HsyV9xVYNWE6grLqMgBv7RXNGK7Pf+w05s5W2f5s9wzkEULCjxwKpPvPmNMvJpx7CQriKzYPn5OiA6tVCdtg95U8/C0+fzhaoIYbnNmj9GE1Hm4pFFN5TkBJgFDmwyk6I+Hmk8C3BqeoFiMBNmxv2Tv/ZgUXmFqRKTTvtNREcH+AekJ+kMBZB+P3d+tL7rzF9xldrnAT2dBPM2w2XB5++Iy0mwuZV4crjZIGdALVF2qwQhMkTWWBbF73YbsHtvXic0Wqq+PLSWEd7Gm2jztPHGH20zrjdsput+PDODSLsaSK2g2JAvV/lhlIIyZMjH2o9XM24Fft/q2SrYf2mcuDL7CDDXU3vuCHLDQDCJ0I4A6lVt6dWfDb/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a module gets unloaded there is a possibility that some of the allocations it made are still used and therefore the allocation tags corresponding to these allocations are still referenced. As such, the memory for these tags can't be freed. This is currently handled as an abnormal situation and module's data section is not being unloaded. To handle this situation without keeping module's data in memory, allow codetags with longer lifespan than the module to be loaded into their own separate memory. The in-use memory areas and gaps after module unloading in this separate memory are tracked using maple trees. Allocation tags arrange their separate memory so that it is virtually contiguous and that will allow simple allocation tag indexing later on in this patchset. The size of this virtually contiguous memory is set to store up to 100000 allocation tags. Signed-off-by: Suren Baghdasaryan --- include/asm-generic/codetag.lds.h | 19 +++ include/linux/alloc_tag.h | 13 +- include/linux/codetag.h | 37 ++++- kernel/module/main.c | 74 +++++---- lib/alloc_tag.c | 251 +++++++++++++++++++++++++++--- lib/codetag.c | 100 +++++++++++- scripts/module.lds.S | 5 +- 7 files changed, 438 insertions(+), 61 deletions(-) diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h index 64f536b80380..372c320c5043 100644 --- a/include/asm-generic/codetag.lds.h +++ b/include/asm-generic/codetag.lds.h @@ -11,4 +11,23 @@ #define CODETAG_SECTIONS() \ SECTION_WITH_BOUNDARIES(alloc_tags) +/* + * Module codetags which aren't used after module unload, therefore have the + * same lifespan as the module and can be safely unloaded with the module. + */ +#define MOD_CODETAG_SECTIONS() + +#define MOD_SEPARATE_CODETAG_SECTION(_name) \ + .codetag.##_name : { \ + SECTION_WITH_BOUNDARIES(_name) \ + } + +/* + * For codetags which might be used after module unload, therefore might stay + * longer in memory. Each such codetag type has its own section so that we can + * unload them individually once unused. + */ +#define MOD_SEPARATE_CODETAG_SECTIONS() \ + MOD_SEPARATE_CODETAG_SECTION(alloc_tags) + #endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 1f0a9ff23a2c..7431757999c5 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,13 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +struct alloc_tag_module_section { + unsigned long start_addr; + unsigned long end_addr; + /* used size */ + unsigned long size; +}; + #ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG #define CODETAG_EMPTY ((void *)1) @@ -54,6 +61,8 @@ static inline void set_codetag_empty(union codetag_ref *ref) {} #ifdef CONFIG_MEM_ALLOC_PROFILING +#define ALLOC_TAG_SECTION_NAME "alloc_tags" + struct codetag_bytes { struct codetag *ct; s64 bytes; @@ -76,7 +85,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); #define DEFINE_ALLOC_TAG(_alloc_tag) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") = { \ + __section(ALLOC_TAG_SECTION_NAME) = { \ .ct = CODE_TAG_INIT, \ .counters = &_shared_alloc_tag }; @@ -85,7 +94,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); #define DEFINE_ALLOC_TAG(_alloc_tag) \ static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") = { \ + __section(ALLOC_TAG_SECTION_NAME) = { \ .ct = CODE_TAG_INIT, \ .counters = &_alloc_tag_cntr }; diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c2a579ccd455..fb4e7adfa746 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -35,8 +35,15 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - bool (*module_unload)(struct codetag_type *cttype, + void (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); +#ifdef CONFIG_MODULES + void (*module_replaced)(struct module *mod, struct module *new_mod); + bool (*needs_section_mem)(struct module *mod, unsigned long size); + void *(*alloc_section_mem)(struct module *mod, unsigned long size, + unsigned int prepend, unsigned long align); + void (*free_section_mem)(struct module *mod, bool unused); +#endif }; struct codetag_iterator { @@ -71,11 +78,31 @@ struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); #if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) + +bool codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size); +void *codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align); +void codetag_free_module_sections(struct module *mod); +void codetag_module_replaced(struct module *mod, struct module *new_mod); void codetag_load_module(struct module *mod); -bool codetag_unload_module(struct module *mod); -#else +void codetag_unload_module(struct module *mod); + +#else /* defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) */ + +static inline bool +codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size) { return false; } +static inline void * +codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align) { return NULL; } +static inline void codetag_free_module_sections(struct module *mod) {} +static inline void codetag_module_replaced(struct module *mod, struct module *new_mod) {} static inline void codetag_load_module(struct module *mod) {} -static inline bool codetag_unload_module(struct module *mod) { return true; } -#endif +static inline void codetag_unload_module(struct module *mod) {} + +#endif /* defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) */ #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index ef54733bd7d2..5569dd052511 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1254,22 +1254,17 @@ static int module_memory_alloc(struct module *mod, enum mod_mem_type type) return 0; } -static void module_memory_free(struct module *mod, enum mod_mem_type type, - bool unload_codetags) +static void module_memory_free(struct module *mod, enum mod_mem_type type) { struct module_memory *mem = &mod->mem[type]; - void *ptr = mem->base; if (mem->is_rox) vfree(mem->rw_copy); - if (!unload_codetags && mod_mem_type_is_core_data(type)) - return; - - execmem_free(ptr); + execmem_free(mem->base); } -static void free_mod_mem(struct module *mod, bool unload_codetags) +static void free_mod_mem(struct module *mod) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1280,25 +1275,20 @@ static void free_mod_mem(struct module *mod, bool unload_codetags) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod, type, unload_codetags); + module_memory_free(mod, type); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod, MOD_DATA, unload_codetags); + module_memory_free(mod, MOD_DATA); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { - bool unload_codetags; - trace_module_free(mod); - unload_codetags = codetag_unload_module(mod); - if (!unload_codetags) - pr_warn("%s: memory allocation(s) from the module still alive, cannot unload cleanly\n", - mod->name); + codetag_unload_module(mod); mod_sysfs_teardown(mod); @@ -1341,7 +1331,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod, unload_codetags); + free_mod_mem(mod); } void *__symbol_get(const char *symbol) @@ -1606,6 +1596,14 @@ static void __layout_sections(struct module *mod, struct load_info *info, bool i if (WARN_ON_ONCE(type == MOD_INVALID)) continue; + /* + * Do not allocate codetag memory as we load it into + * preallocated contiguous memory. s->sh_entsize will + * not be used for this section, so leave it as is. + */ + if (codetag_needs_module_section(mod, sname, s->sh_size)) + continue; + s->sh_entsize = module_get_offset_and_type(mod, type, s, i); pr_debug("\t%s\n", sname); } @@ -2280,6 +2278,7 @@ static int move_module(struct module *mod, struct load_info *info) int i; enum mod_mem_type t = 0; int ret = -ENOMEM; + bool codetag_section_found = false; for_each_mod_mem_type(type) { if (!mod->mem[type].size) { @@ -2291,7 +2290,7 @@ static int move_module(struct module *mod, struct load_info *info) ret = module_memory_alloc(mod, type); if (ret) { t = type; - goto out_enomem; + goto out_err; } } @@ -2300,15 +2299,33 @@ static int move_module(struct module *mod, struct load_info *info) for (i = 0; i < info->hdr->e_shnum; i++) { void *dest; Elf_Shdr *shdr = &info->sechdrs[i]; - enum mod_mem_type type = shdr->sh_entsize >> SH_ENTSIZE_TYPE_SHIFT; - unsigned long offset = shdr->sh_entsize & SH_ENTSIZE_OFFSET_MASK; + const char *sname; unsigned long addr; if (!(shdr->sh_flags & SHF_ALLOC)) continue; - addr = (unsigned long)mod->mem[type].base + offset; - dest = mod->mem[type].rw_copy + offset; + sname = info->secstrings + shdr->sh_name; + /* + * Load codetag sections separately as they might still be used + * after module unload. + */ + if (codetag_needs_module_section(mod, sname, shdr->sh_size)) { + dest = codetag_alloc_module_section(mod, sname, shdr->sh_size, + arch_mod_section_prepend(mod, i), shdr->sh_addralign); + if (IS_ERR(dest)) { + ret = PTR_ERR(dest); + goto out_err; + } + addr = (unsigned long)dest; + codetag_section_found = true; + } else { + enum mod_mem_type type = shdr->sh_entsize >> SH_ENTSIZE_TYPE_SHIFT; + unsigned long offset = shdr->sh_entsize & SH_ENTSIZE_OFFSET_MASK; + + addr = (unsigned long)mod->mem[type].base + offset; + dest = mod->mem[type].rw_copy + offset; + } if (shdr->sh_type != SHT_NOBITS) { /* @@ -2320,7 +2337,7 @@ static int move_module(struct module *mod, struct load_info *info) if (i == info->index.mod && (WARN_ON_ONCE(shdr->sh_size != sizeof(struct module)))) { ret = -ENOEXEC; - goto out_enomem; + goto out_err; } memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size); } @@ -2336,9 +2353,12 @@ static int move_module(struct module *mod, struct load_info *info) } return 0; -out_enomem: +out_err: for (t--; t >= 0; t--) - module_memory_free(mod, t, true); + module_memory_free(mod, t); + if (codetag_section_found) + codetag_free_module_sections(mod); + return ret; } @@ -2459,6 +2479,8 @@ static struct module *layout_and_allocate(struct load_info *info, int flags) /* Module has been copied to its final place now: return it. */ mod = (void *)info->sechdrs[info->index.mod].sh_addr; kmemleak_load_module(mod, info); + codetag_module_replaced(info->mod, mod); + return mod; } @@ -2468,7 +2490,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod, true); + free_mod_mem(mod); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 81e5f9a70f22..b10e7f17eeda 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only #include +#include #include #include #include @@ -149,31 +150,231 @@ static void __init procfs_init(void) proc_create_seq("allocinfo", 0400, NULL, &allocinfo_seq_op); } -static bool alloc_tag_module_unload(struct codetag_type *cttype, - struct codetag_module *cmod) +#ifdef CONFIG_MODULES + +static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); +/* A dummy object used to indicate an unloaded module */ +static struct module unloaded_mod; +/* A dummy object used to indicate a module prepended area */ +static struct module prepend_mod; + +static struct alloc_tag_module_section module_tags; + +static bool needs_section_mem(struct module *mod, unsigned long size) +{ + return size >= sizeof(struct alloc_tag); +} + +/* Called under RCU read lock */ +static void clean_unused_module_areas_locked(void) +{ + MA_STATE(mas, &mod_area_mt, 0, module_tags.size); + struct module *val; + + mas_for_each(&mas, val, module_tags.size) { + if (val == &unloaded_mod) { + struct alloc_tag *tag; + struct alloc_tag *last; + bool unused = true; + + tag = (struct alloc_tag *)(module_tags.start_addr + mas.index); + last = (struct alloc_tag *)(module_tags.start_addr + mas.last); + while (tag <= last) { + struct alloc_tag_counters counter; + + counter = alloc_tag_read(tag); + if (counter.bytes) { + unused = false; + break; + } + tag++; + } + if (unused) + mas_erase(&mas); + } + } +} + +static void *reserve_module_tags(struct module *mod, unsigned long size, + unsigned int prepend, unsigned long align) +{ + unsigned long section_size = module_tags.end_addr - module_tags.start_addr; + MA_STATE(mas, &mod_area_mt, 0, section_size - 1); + bool cleanup_done = false; + unsigned long offset; + void *ret; + + /* If no tags return NULL */ + if (size < sizeof(struct alloc_tag)) + return NULL; + + /* + * align is always power of 2, so we can use IS_ALIGNED and ALIGN. + * align 0 or 1 means no alignment, to simplify set to 1. + */ + if (!align) + align = 1; + + mas_lock(&mas); +repeat: + /* Try finding exact size and hope the start is aligned */ + if (mas_empty_area(&mas, 0, section_size - 1, prepend + size)) + goto cleanup; + + if (IS_ALIGNED(mas.index + prepend, align)) + goto found; + + /* Try finding larger area to align later */ + mas_reset(&mas); + if (!mas_empty_area(&mas, 0, section_size - 1, + size + prepend + align - 1)) + goto found; + +cleanup: + /* No free area, try cleanup stale data and repeat the search once */ + if (!cleanup_done) { + clean_unused_module_areas_locked(); + cleanup_done = true; + mas_reset(&mas); + goto repeat; + } else { + ret = ERR_PTR(-ENOMEM); + goto out; + } + +found: + offset = mas.index; + offset += prepend; + offset = ALIGN(offset, align); + + if (offset != mas.index) { + unsigned long pad_start = mas.index; + + mas.last = offset - 1; + mas_store(&mas, &prepend_mod); + if (mas_is_err(&mas)) { + ret = ERR_PTR(xa_err(mas.node)); + goto out; + } + mas.index = offset; + mas.last = offset + size - 1; + mas_store(&mas, mod); + if (mas_is_err(&mas)) { + ret = ERR_PTR(xa_err(mas.node)); + mas.index = pad_start; + mas_erase(&mas); + goto out; + } + + } else { + mas.last = offset + size - 1; + mas_store(&mas, mod); + if (mas_is_err(&mas)) { + ret = ERR_PTR(xa_err(mas.node)); + goto out; + } + } + + if (module_tags.size < offset + size) + module_tags.size = offset + size; + + ret = (struct alloc_tag *)(module_tags.start_addr + offset); +out: + mas_unlock(&mas); + + return ret; +} + +static void release_module_tags(struct module *mod, bool unused) { - struct codetag_iterator iter = codetag_get_ct_iter(cttype); - struct alloc_tag_counters counter; - bool module_unused = true; + MA_STATE(mas, &mod_area_mt, module_tags.size, module_tags.size); + struct alloc_tag *last; struct alloc_tag *tag; - struct codetag *ct; + struct module *val; - for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { - if (iter.cmod != cmod) - continue; + if (unused) + return; + + mas_lock(&mas); + mas_for_each_rev(&mas, val, 0) + if (val == mod) + break; + + if (!val) /* module not found */ + goto out; + + tag = (struct alloc_tag *)(module_tags.start_addr + mas.index); + last = (struct alloc_tag *)(module_tags.start_addr + mas.last); + + unused = true; + while (tag <= last) { + struct alloc_tag_counters counter; - tag = ct_to_alloc_tag(ct); counter = alloc_tag_read(tag); + if (counter.bytes) { + struct codetag *ct = &tag->ct; + + pr_info("%s:%u module %s func:%s has %llu allocated at module unload\n", + ct->filename, ct->lineno, ct->modname, + ct->function, counter.bytes); + unused = false; + break; + } + tag++; + } - if (WARN(counter.bytes, - "%s:%u module %s func:%s has %llu allocated at module unload", - ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) - module_unused = false; + mas_store(&mas, unused ? NULL : &unloaded_mod); + val = mas_prev_range(&mas, 0); + if (val == &prepend_mod) + mas_store(&mas, NULL); +out: + mas_unlock(&mas); +} + +static void replace_module(struct module *mod, struct module *new_mod) +{ + MA_STATE(mas, &mod_area_mt, 0, module_tags.size); + struct module *val; + + mas_lock(&mas); + mas_for_each(&mas, val, module_tags.size) { + if (val != mod) + continue; + + mas_store_gfp(&mas, new_mod, GFP_KERNEL); + break; } + mas_unlock(&mas); +} - return module_unused; +#define MODULE_ALLOC_TAG_VMAP_SIZE (100000 * sizeof(struct alloc_tag)) + +static int __init alloc_mod_tags_mem(void) +{ + /* Allocate space to copy allocation tags */ + module_tags.start_addr = (unsigned long)execmem_alloc(EXECMEM_MODULE_DATA, + MODULE_ALLOC_TAG_VMAP_SIZE); + if (!module_tags.start_addr) + return -ENOMEM; + + module_tags.end_addr = module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_SIZE; + + return 0; +} + +static void __init free_mod_tags_mem(void) +{ + execmem_free((void *)module_tags.start_addr); + module_tags.start_addr = 0; } +#else /* CONFIG_MODULES */ + +static inline int alloc_mod_tags_mem(void) { return 0; } +static inline void free_mod_tags_mem(void) {} + +#endif /* CONFIG_MODULES */ + #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT static bool mem_profiling_support __meminitdata = true; #else @@ -255,14 +456,26 @@ static inline void sysctl_init(void) {} static int __init alloc_tag_init(void) { const struct codetag_type_desc desc = { - .section = "alloc_tags", - .tag_size = sizeof(struct alloc_tag), - .module_unload = alloc_tag_module_unload, + .section = ALLOC_TAG_SECTION_NAME, + .tag_size = sizeof(struct alloc_tag), +#ifdef CONFIG_MODULES + .needs_section_mem = needs_section_mem, + .alloc_section_mem = reserve_module_tags, + .free_section_mem = release_module_tags, + .module_replaced = replace_module, +#endif }; + int res; + + res = alloc_mod_tags_mem(); + if (res) + return res; alloc_tag_cttype = codetag_register_type(&desc); - if (IS_ERR(alloc_tag_cttype)) + if (IS_ERR(alloc_tag_cttype)) { + free_mod_tags_mem(); return PTR_ERR(alloc_tag_cttype); + } sysctl_init(); procfs_init(); diff --git a/lib/codetag.c b/lib/codetag.c index d1fbbb7c2ec3..3f3a4f4ca72d 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -207,6 +207,94 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) } #ifdef CONFIG_MODULES +#define CODETAG_SECTION_PREFIX ".codetag." + +/* Some codetag types need a separate module section */ +bool codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size) +{ + const char *type_name; + struct codetag_type *cttype; + bool ret = false; + + if (strncmp(name, CODETAG_SECTION_PREFIX, strlen(CODETAG_SECTION_PREFIX))) + return false; + + type_name = name + strlen(CODETAG_SECTION_PREFIX); + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (strcmp(type_name, cttype->desc.section) == 0) { + if (!cttype->desc.needs_section_mem) + break; + + down_write(&cttype->mod_lock); + ret = cttype->desc.needs_section_mem(mod, size); + up_write(&cttype->mod_lock); + break; + } + } + mutex_unlock(&codetag_lock); + + return ret; +} + +void *codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align) +{ + const char *type_name = name + strlen(CODETAG_SECTION_PREFIX); + struct codetag_type *cttype; + void *ret = NULL; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (strcmp(type_name, cttype->desc.section) == 0) { + if (WARN_ON(!cttype->desc.alloc_section_mem)) + break; + + down_write(&cttype->mod_lock); + ret = cttype->desc.alloc_section_mem(mod, size, prepend, align); + up_write(&cttype->mod_lock); + break; + } + } + mutex_unlock(&codetag_lock); + + return ret; +} + +void codetag_free_module_sections(struct module *mod) +{ + struct codetag_type *cttype; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (!cttype->desc.free_section_mem) + continue; + + down_write(&cttype->mod_lock); + cttype->desc.free_section_mem(mod, true); + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + +void codetag_module_replaced(struct module *mod, struct module *new_mod) +{ + struct codetag_type *cttype; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (!cttype->desc.module_replaced) + continue; + + down_write(&cttype->mod_lock); + cttype->desc.module_replaced(mod, new_mod); + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + void codetag_load_module(struct module *mod) { struct codetag_type *cttype; @@ -220,13 +308,12 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -bool codetag_unload_module(struct module *mod) +void codetag_unload_module(struct module *mod) { struct codetag_type *cttype; - bool unload_ok = true; if (!mod) - return true; + return; /* await any module's kfree_rcu() operations to complete */ kvfree_rcu_barrier(); @@ -246,18 +333,17 @@ bool codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - if (!cttype->desc.module_unload(cttype, cmod)) - unload_ok = false; + cttype->desc.module_unload(cttype, cmod); cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); kfree(cmod); } up_write(&cttype->mod_lock); + if (found && cttype->desc.free_section_mem) + cttype->desc.free_section_mem(mod, false); } mutex_unlock(&codetag_lock); - - return unload_ok; } #endif /* CONFIG_MODULES */ diff --git a/scripts/module.lds.S b/scripts/module.lds.S index 3f43edef813c..711c6e029936 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -50,7 +50,7 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) - CODETAG_SECTIONS() + MOD_CODETAG_SECTIONS() } .rodata : { @@ -59,9 +59,10 @@ SECTIONS { } #else .data : { - CODETAG_SECTIONS() + MOD_CODETAG_SECTIONS() } #endif + MOD_SEPARATE_CODETAG_SECTIONS() } /* bring in arch-specific sections */ From patchwork Mon Oct 14 20:36:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13835447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECD3FD18146 for ; Mon, 14 Oct 2024 20:37:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34A3F6B0092; Mon, 14 Oct 2024 16:37:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D4E36B0093; Mon, 14 Oct 2024 16:37:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14E386B0095; Mon, 14 Oct 2024 16:37:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DF0286B0092 for ; Mon, 14 Oct 2024 16:36:59 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 365951A0C10 for ; Mon, 14 Oct 2024 20:36:45 +0000 (UTC) X-FDA: 82673366706.19.C943D63 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 64404180009 for ; Mon, 14 Oct 2024 20:36:51 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VOQdMesi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 36IANZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=36IANZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728938061; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bOvroPog9pmluzu4zs2xrkfEGIpV84j+wuh+gRgw5PA=; b=gEVF64mYeaD0Kd2XvNLbnML5q24SxaysClUpCBR/Ff7L9M8J4dlFL4r9cLT5+J/lIR8Vqg df7pOoTOBwa/UxG07dvvpIAmiCmgEaHKPkS2H4OxqoApKwva0F5Tm8NnawqAjVQxdP1z7s GlVOQwMa8aCeGQOlNfjsofruza/3Cps= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728938061; a=rsa-sha256; cv=none; b=Ff29KacgnilNzc8DnZD0TvZZj6D+cTEnJVY5Bc0RC18iZIDZ2gIO1swN9N4mNVJOpHoz7v QpjkwtGenL2dSbIiBK9U+GIgz2E1vPEhicN/ajpC9n4oJBRedghvBSqDbvMBPbZ0Eg+ySi oOiM5uWgh0gg5P/G2IXA3aq9MKS9eHY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VOQdMesi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 36IANZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=36IANZwYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e3705b2883so26817317b3.3 for ; Mon, 14 Oct 2024 13:36:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728938217; x=1729543017; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bOvroPog9pmluzu4zs2xrkfEGIpV84j+wuh+gRgw5PA=; b=VOQdMesizs/YNQ521EcAB6zYeX7RpnLe11BvJ6gTGeHX3MvXw4WYiHczCG6GIx8yfr ZxDspMv/B1+Zq1MQYQMtKtXIQzW8cqnlWeXqKNi0O0oly936NzNP3kKe/5JDMDwNZ57V klMqxXG2FoiZPryFxzuXu5NQCbBLrGjtv4bG4n9a/QZZNodKE/MaSwkqionzusp4PDEk u8m096KVGVHluCx7GCIw+b+Aigu5aQAWCMs5Xc5llAPYco8vAN4MLSwEv0948Vim/Gjq vhF1Z2Tf7CgUpf8+MnCVw0go4vCZ4RBy5lxslXBfAk+ZhZRPksMaaUME8aQt/J/VDeAW wJWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728938217; x=1729543017; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bOvroPog9pmluzu4zs2xrkfEGIpV84j+wuh+gRgw5PA=; b=a7R/PwQgbUvjor08VBpD6ClIo684YH8BZW7UYjtAX6egvCKhC9lKDGdf9LPasz4T2Z LWFuhg+l5+4EzebU03T0n5YT0CF0i62glJqSZqLh9JQzZ8moTH4vu1Y/PrtcpxpIDIS0 DSqoe5e9tRT+hlVaLdUijrW5XT9fDGNqxEJfZJl5oQNArTeZdzVzGJWzmqCNUUn/IEYI 6BiHur6neZu7fml1jHDqRs6MPA1+SKZxI+xfCWBJgw/4zMMO+dUw2D+/1pMdPFKE5q7u aAvVi4fFCSgXgAVxX9xaciBPuUtvch+D9/pH39T2BHjLhhuLEPFjgpzFtKjrjcymH77G Zu/g== X-Forwarded-Encrypted: i=1; AJvYcCW0+h2JYwrMAAFEXC7y4wkw4nxLJA0gSL7+K8dATt8f+ol1hRTyGxh1yGaPUbhNcxHvn+KwacQTow==@kvack.org X-Gm-Message-State: AOJu0Yys2WvgAF8djuGrfKyg5sy470T5vHuB+OhQedOE2jcw8JGsH+r+ 4ZkfV1QSHOoNu8OhbjEnabGtUoc/zmsmF9ZREGAK+owd0QYGB7FCDaDYteujgt6N0Bg2Z8j1/dA 7HA== X-Google-Smtp-Source: AGHT+IHN+F4HHrAed80eKaPYhTH0JCc5Mv3vEPFTteI9kO52+1hB5IgMv4halrrXePZu5DiEH+LMAHeUbpw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:915:bdd7:e08a:7997]) (user=surenb job=sendgmr) by 2002:a05:690c:c09:b0:6e3:1702:b3e6 with SMTP id 00721157ae682-6e347b368d2mr1991517b3.4.1728938216731; Mon, 14 Oct 2024 13:36:56 -0700 (PDT) Date: Mon, 14 Oct 2024 13:36:44 -0700 In-Reply-To: <20241014203646.1952505-1-surenb@google.com> Mime-Version: 1.0 References: <20241014203646.1952505-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241014203646.1952505-4-surenb@google.com> Subject: [PATCH v3 3/5] alloc_tag: populate memory for module tags as needed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 64404180009 X-Stat-Signature: bmjjddqdaqy39i6axcodqgueneoeghtn X-Rspam-User: X-HE-Tag: 1728938211-157190 X-HE-Meta: U2FsdGVkX1/RwzR8LBZBsutH10DF+1zVG039fsjBD1uWDGI7i8zyNHKCrmnwQKMjkLKJv8JYSZqZ5o9YAcdwYOA3u5pwsOddwu0LHEf6LVPD7hDA4tRa1G3+aezlWKBhso3vt8sQLAHLu0z4Zunacw16XTqPHBxC4hjfwHCz4bTyIK3AE5/4aWWcQ+/YOeu3f/2Avrg51kLHOKqR/0IBqKV/DjLja04KXmjAL5bpPi6Qs5oFWXdQUqsHpWwbYJiLJNPu973Ja5EICc/xy9/Gn5nPk4A2JxtYNUCRbBlFny9dvAqaj5lQbsI6pMA5va6JPTgOnrBNRRYYuyn4XAA/KOlYRe3UxAsNd764xi5Q0J6iXqZWYjP7n7+tuPH8GI4tWaM8rN0ruB1E/Q51HT64Gbg+lgfdFAii4gS8fCJv3z6cNgHQRbKLcUl7e2RHjJYl7CxR1+gAmMzOiT9BpSxPLHQBWE9BArTuw/T2lgtx8uWIhXgMB37dQN9CkkzFF1lfgYYCHT5Ah+GKx5xzsUiaAPCOVwlM9dNN48mTg5Rlzn/zhz3JGKYgSYiT8Sehvw//sdyG4yS23bqXwuzAVvmpVTWiCcrjZbPCYD8ZgOdkjXYPOUx2RXdg8R3f0psJJySmLJ/CE57cTlnyeym3d8CI7lry0DUSHpom9FSRL+htGgy6Rikw1m+UHRc3hVHslZvGbGb7qcUX7XSbu8QQS93ZX42oFvlSYOInrJPOFRJxUwfl/SZBonGcbe89BAmXG62QQ7XYuXtoyQXbrl+PgTAoKTgLG1uCcjfh41Vtq2aejKNNeHQ1nB1PYrKkcRBhxjVYJyY2lyA7+/aDPnq3rM6/7vuvmwCIf1jf0w1ADryQmvi7v3m6uRDF3PpaLyftcaGMxHse9PcUoQplPXQQOX2dviwOMhzDycUqYPTY9vx2Xcdc4MzHc7QQWcQ/r2ZQi20E+fO81j2hgp33e5Dx5lP jh/lQEl8 REKVLIa5lx4s0i/Z7lmy4O7Cgty6/s9Q0CfgILBaqTkKBxzIyaS+UAGq0NhUI3cTGVogxT6ebbMua7VzntXdrWwH1Fa95JhqkDoEAw3ptRw35r9D4aGQmH3b+B4u/jsiESHWMxm2GFqhpXdALv++IWrsSaSPZ0YKPreLvldLx7KgOlEtBbrr97kA3PBgDakAJjpjg0Mi4CkM3TyF2+C1VmkwpD6019jPmeiZG4aEQtDkPBY28yU3yqBWbXBwH4Ykg4lx+drOiSbvOdw/tPXJhReKlZCqGn5ghX/03JopKeF6FlQGj7z043LGNpnMiyA9eV86XPLoaiLJoMGga1SedhRG0eEOBvIN00BEsWITN6SlIQy75Sgnra1XZV/X8nrGAqgqncg4A0iVdoQ7wwmsmVvmFxWdOFg6UF+ScdL+ID5e0EEhnBX1k7YXUg2iGJCE2IjL436CCJTZgs8Vh4mnPZJbMGr+kpxanXAzo2dkGz6SKrA04KgbZvE9q8crd3g/UQTdsPbOxbn3QZI3FFhFY9xw05EFsVzNTDUv4gd8Jr1DYOuCQjqxG3qYRMosyftfEcGTv/LOdnUKqxXcqCh5DKbkXfw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module. Signed-off-by: Suren Baghdasaryan --- include/linux/execmem.h | 11 ++++++ include/linux/vmalloc.h | 9 +++++ lib/alloc_tag.c | 84 +++++++++++++++++++++++++++++++++-------- mm/execmem.c | 16 ++++++++ mm/vmalloc.c | 4 +- 5 files changed, 106 insertions(+), 18 deletions(-) diff --git a/include/linux/execmem.h b/include/linux/execmem.h index 7436aa547818..a159a073270a 100644 --- a/include/linux/execmem.h +++ b/include/linux/execmem.h @@ -127,6 +127,17 @@ void *execmem_alloc(enum execmem_type type, size_t size); */ void execmem_free(void *ptr); +/** + * execmem_vmap - create virtual mapping for executable memory + * @type: type of the allocation + * @size: size of the virtual mapping in bytes + * + * Maps virtually contiguous area that can be populated with executable code. + * + * Return: the area descriptor on success or %NULL on failure. + */ +struct vm_struct *execmem_vmap(enum execmem_type type, size_t size); + /** * execmem_update_copy - copy an update to executable memory * @dst: destination address to update diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 9a012cd4fad2..9d64cc6f24d1 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -202,6 +202,9 @@ extern int remap_vmalloc_range_partial(struct vm_area_struct *vma, extern int remap_vmalloc_range(struct vm_area_struct *vma, void *addr, unsigned long pgoff); +int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift); + /* * Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values * and let generic vmalloc and ioremap code know when arch_sync_kernel_mappings() @@ -239,6 +242,12 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size, unsigned long flags, unsigned long start, unsigned long end, const void *caller); +struct vm_struct *__get_vm_area_node(unsigned long size, + unsigned long align, unsigned long shift, + unsigned long flags, unsigned long start, + unsigned long end, int node, gfp_t gfp_mask, + const void *caller); + void free_vm_area(struct vm_struct *area); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index b10e7f17eeda..648f32d52b8d 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -8,6 +8,7 @@ #include #include #include +#include static struct codetag_type *alloc_tag_cttype; @@ -153,6 +154,7 @@ static void __init procfs_init(void) #ifdef CONFIG_MODULES static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); +static struct vm_struct *vm_module_tags; /* A dummy object used to indicate an unloaded module */ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ @@ -195,6 +197,25 @@ static void clean_unused_module_areas_locked(void) } } +static int vm_module_tags_grow(unsigned long addr, unsigned long bytes) +{ + struct page **next_page = vm_module_tags->pages + vm_module_tags->nr_pages; + unsigned long more_pages = ALIGN(bytes, PAGE_SIZE) >> PAGE_SHIFT; + unsigned long nr; + + nr = alloc_pages_bulk_array_node(GFP_KERNEL | __GFP_NOWARN, + NUMA_NO_NODE, more_pages, next_page); + if (nr != more_pages) + return -ENOMEM; + + vm_module_tags->nr_pages += nr; + if (vmap_pages_range(addr, addr + (nr << PAGE_SHIFT), + PAGE_KERNEL, next_page, PAGE_SHIFT) < 0) + return -ENOMEM; + + return 0; +} + static void *reserve_module_tags(struct module *mod, unsigned long size, unsigned int prepend, unsigned long align) { @@ -202,7 +223,7 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, MA_STATE(mas, &mod_area_mt, 0, section_size - 1); bool cleanup_done = false; unsigned long offset; - void *ret; + void *ret = NULL; /* If no tags return NULL */ if (size < sizeof(struct alloc_tag)) @@ -239,7 +260,7 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, goto repeat; } else { ret = ERR_PTR(-ENOMEM); - goto out; + goto unlock; } found: @@ -254,7 +275,7 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, mas_store(&mas, &prepend_mod); if (mas_is_err(&mas)) { ret = ERR_PTR(xa_err(mas.node)); - goto out; + goto unlock; } mas.index = offset; mas.last = offset + size - 1; @@ -263,7 +284,7 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, ret = ERR_PTR(xa_err(mas.node)); mas.index = pad_start; mas_erase(&mas); - goto out; + goto unlock; } } else { @@ -271,18 +292,33 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, mas_store(&mas, mod); if (mas_is_err(&mas)) { ret = ERR_PTR(xa_err(mas.node)); - goto out; + goto unlock; } } +unlock: + mas_unlock(&mas); + if (IS_ERR(ret)) + return ret; - if (module_tags.size < offset + size) - module_tags.size = offset + size; + if (module_tags.size < offset + size) { + unsigned long phys_size = vm_module_tags->nr_pages << PAGE_SHIFT; - ret = (struct alloc_tag *)(module_tags.start_addr + offset); -out: - mas_unlock(&mas); + module_tags.size = offset + size; + if (phys_size < module_tags.size) { + int grow_res; + + grow_res = vm_module_tags_grow(module_tags.start_addr + phys_size, + module_tags.size - phys_size); + if (grow_res) { + static_branch_disable(&mem_alloc_profiling_key); + pr_warn("Failed to allocate tags memory for module %s. Memory profiling is disabled!\n", + mod->name); + return ERR_PTR(grow_res); + } + } + } - return ret; + return (struct alloc_tag *)(module_tags.start_addr + offset); } static void release_module_tags(struct module *mod, bool unused) @@ -351,12 +387,23 @@ static void replace_module(struct module *mod, struct module *new_mod) static int __init alloc_mod_tags_mem(void) { - /* Allocate space to copy allocation tags */ - module_tags.start_addr = (unsigned long)execmem_alloc(EXECMEM_MODULE_DATA, - MODULE_ALLOC_TAG_VMAP_SIZE); - if (!module_tags.start_addr) + /* Map space to copy allocation tags */ + vm_module_tags = execmem_vmap(EXECMEM_MODULE_DATA, MODULE_ALLOC_TAG_VMAP_SIZE); + if (!vm_module_tags) { + pr_err("Failed to map %lu bytes for module allocation tags\n", + MODULE_ALLOC_TAG_VMAP_SIZE); + module_tags.start_addr = 0; return -ENOMEM; + } + vm_module_tags->pages = kmalloc_array(get_vm_area_size(vm_module_tags) >> PAGE_SHIFT, + sizeof(struct page *), GFP_KERNEL | __GFP_ZERO); + if (!vm_module_tags->pages) { + free_vm_area(vm_module_tags); + return -ENOMEM; + } + + module_tags.start_addr = (unsigned long)vm_module_tags->addr; module_tags.end_addr = module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_SIZE; return 0; @@ -364,8 +411,13 @@ static int __init alloc_mod_tags_mem(void) static void __init free_mod_tags_mem(void) { - execmem_free((void *)module_tags.start_addr); + int i; + module_tags.start_addr = 0; + for (i = 0; i < vm_module_tags->nr_pages; i++) + __free_page(vm_module_tags->pages[i]); + kfree(vm_module_tags->pages); + free_vm_area(vm_module_tags); } #else /* CONFIG_MODULES */ diff --git a/mm/execmem.c b/mm/execmem.c index 97706d8ed720..eb346f4eaaff 100644 --- a/mm/execmem.c +++ b/mm/execmem.c @@ -366,6 +366,22 @@ void execmem_free(void *ptr) vfree(ptr); } +struct vm_struct *execmem_vmap(enum execmem_type type, size_t size) +{ + struct execmem_range *range = &execmem_info->ranges[type]; + struct vm_struct *area; + + area = __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->start, range->end, NUMA_NO_NODE, + GFP_KERNEL, __builtin_return_address(0)); + if (!area && range->fallback_start) + area = __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->fallback_start, range->fallback_end, + NUMA_NO_NODE, GFP_KERNEL, __builtin_return_address(0)); + + return area; +} + void *execmem_update_copy(void *dst, const void *src, size_t size) { return text_poke_copy(dst, src, size); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 74c0a5eae210..7ed39d104201 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -653,7 +653,7 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end, * RETURNS: * 0 on success, -errno on failure. */ -static int vmap_pages_range(unsigned long addr, unsigned long end, +int vmap_pages_range(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { int err; @@ -3106,7 +3106,7 @@ static void clear_vm_uninitialized_flag(struct vm_struct *vm) vm->flags &= ~VM_UNINITIALIZED; } -static struct vm_struct *__get_vm_area_node(unsigned long size, +struct vm_struct *__get_vm_area_node(unsigned long size, unsigned long align, unsigned long shift, unsigned long flags, unsigned long start, unsigned long end, int node, gfp_t gfp_mask, const void *caller) From patchwork Mon Oct 14 20:36:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13835448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96B1DD18144 for ; Mon, 14 Oct 2024 20:37:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C2866B0095; Mon, 14 Oct 2024 16:37:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 371276B0098; Mon, 14 Oct 2024 16:37:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14D536B0096; Mon, 14 Oct 2024 16:37:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E9C7C6B0093 for ; Mon, 14 Oct 2024 16:37:01 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9300BA129E for ; Mon, 14 Oct 2024 20:36:46 +0000 (UTC) X-FDA: 82673366874.30.9C2C6C8 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf07.hostedemail.com (Postfix) with ESMTP id 4280D40014 for ; Mon, 14 Oct 2024 20:36:50 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=q1zsZ6dh; spf=pass (imf07.hostedemail.com: domain of 36oANZwYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=36oANZwYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728938173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KDELffEPLAPDpRQ8WZJTecgr0QxXWhGmdEuRSVQA138=; b=SqSddFL/nt/5rK+bMLYx8J1Ds2wcDuWs6y8nuE48pnTWpTQ74y1H2c51IDYTOAnQ46c0NQ 9xjGA8CqXI60LByxT10JqIwpcdFES0Al7bYuWiubpZe7/PCcsobeIG7aYGYHCiKYTWaYIF Chtv+MGFNddXG1CtmGSVqH4KjlhNqKk= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=q1zsZ6dh; spf=pass (imf07.hostedemail.com: domain of 36oANZwYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=36oANZwYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728938173; a=rsa-sha256; cv=none; b=GHS1UImNolyonOE0LtL0bueC1G1EieyIz1ALfT1gLZ0noUHrBo17j25NlMx5xNwDLpR5Ea e3WZjIrTYrKlGHlHLzR4GSzUwfPjLxwSdde5QHROWdpbmArPnreFcWH5K2fhVjuayV41rm M9LqcFKXtlKgy2Mm8sTru4KUIj0QOgY= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6dbbeee08f0so77201917b3.0 for ; Mon, 14 Oct 2024 13:36:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728938219; x=1729543019; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KDELffEPLAPDpRQ8WZJTecgr0QxXWhGmdEuRSVQA138=; b=q1zsZ6dhVXl56yCAsjnsjwylb9zxyAESJ5sDXOJ+Fz9HkNSaLWm7R/e3anJMn0e/V5 NQ0yvn77pyf1F8AFx/zMP8UVvPYPPhaKmcOi1b7TnezTpgOPpkaLQwLDo9roDxqDb+Kr eEE7k7rrVFDS7Ukt2V9Jyw6mbLDIvBATcRQhb/7hJi8feXKEbA2yZp4ZUOxYz74QXKf0 g7mA5vB3wpH1G1VJJ4nmttFm6RJXlymOS5QeTG49gpUxOJ+axOKPh9YvDtQz0W1ts8c9 ZNR/DBxehXfuPsF6xFL8dZTZ/SJ1fpL+bGvImtqGHFeurHGWHVDqdOAl+vIAx3Qyu4dg DrTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728938219; x=1729543019; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KDELffEPLAPDpRQ8WZJTecgr0QxXWhGmdEuRSVQA138=; b=WyuyrhaENMaSvqmszgdYpH9T8KXCnzdTQwFhZvFSR+EYUEObWyLJVdbzsTInT/0ox8 2LJ5nP5FG42eegCVd+1DPHuDCp5MvXGZMrxwlLXntbeRjHhDl43SwnE/CW2rk/d9Zz4+ 7kUDlrq/VznV1Prupr8UHIX2XkuGYxkrHBQwlZ3n0NV5Cxqffqzt4uWJtPZlPWk9xN4y FRXJ8niBFjxixMzRg3AkIKg7mtZnNV/cvsshMAmZJneT6TG1keT4jxcMAQj3fEaMCmUt cI/HgqqmHgE0IqGc71YApOwe0UaR3NLN3jefH2Ze/9s3pjVIZzTYPtwf7qqiLIVyKGj5 /XOw== X-Forwarded-Encrypted: i=1; AJvYcCX3ZYqOHUVXG2V5ZN5UTxNz+KrKvDN9sHJh/U6FT42BsnUHzOXGgTLXsyz9tE3asw40t/ug8V3ruw==@kvack.org X-Gm-Message-State: AOJu0YzO1O80jwTmv2g23HzpkapcGkTM+6n39qIndrY/fJ8OpavCYG2O z3ODNbX+XukeKFqvWk9Fb3s9dPjhhVXeZD2c5LpQzwvucN/3x1Olh05nXwrmVHhvvbdMamaIG/A 1gw== X-Google-Smtp-Source: AGHT+IFebmDDzFfa9NO21TdKX40eCmjiTPo05JzyoY7+ZX8h6EIqKu1BLOx2oszwApr2jiSzy+GImo1RYfc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:915:bdd7:e08a:7997]) (user=surenb job=sendgmr) by 2002:a5b:40a:0:b0:e29:9c5:5fcb with SMTP id 3f1490d57ef6-e290bb22b20mr207938276.4.1728938218676; Mon, 14 Oct 2024 13:36:58 -0700 (PDT) Date: Mon, 14 Oct 2024 13:36:45 -0700 In-Reply-To: <20241014203646.1952505-1-surenb@google.com> Mime-Version: 1.0 References: <20241014203646.1952505-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241014203646.1952505-5-surenb@google.com> Subject: [PATCH v3 4/5] alloc_tag: introduce pgalloc_tag_ref to abstract page tag references From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Stat-Signature: 1k9wq5sk158xwic1yx73n1m35ek7dxu3 X-Rspamd-Queue-Id: 4280D40014 X-Rspamd-Server: rspam11 X-HE-Tag: 1728938210-686277 X-HE-Meta: U2FsdGVkX1/rEvFAFm9tjqkdGbh6gKmSCvdhmVxFznwRudeZ9Pnd8Sh3n8kQGjPWvpJSPbqib7kJdhjmRfwIlmsEbf9LpDdDoBzbB3q65k/nxKXtywWrv9vHxy7KyfHl3Ib5aDrFyvur9TJD6OJsZ/MRMZAOIkhE1Tg1A+mj0neFiFhkn1ygcdxTmH+SrRGCO1Nfi220mqQiXdKxr1CZf1tJGt1Prvrg7i1jK/AMnKxYfH0BbU2WXq4qcOgljjxYzqweE+8EWGzTPphXZouQ9JRfPr4EGAyglAYZLrgW68gbaHUnWbK7EjFg84aBfpHJYNj33W78TSQrXaHuFwwUNGnMLtCSj7Kdm2KffYptQ1PGaRJlSX+dUmm+P9fxiacqzVNWF1Bf43l4jMwtrENtJWYUikL/8m35lZVKO2SyiGp/id+U5i8OtmuN+yznNqERI4VbXLgQ/WSh7rmCV9wbbr7D29J7daVj+h/o7Pqx976ICMJSS2AQyMyvSu8u2kR86Ex15WyUlTWOayX/Qc07PqfPXPC+TPxTp+fF89Os3hzqPMQ+Qmhrwbp3Bno2Bshhh6UtEUKVeLTdJqoWsFzOfJImNOfi5F51Tl67GwalQeHkzjmyeS3t4VJ/U7y40F+pX/JBqTlOWctyWP3C7qtr1W/7IUwGR9Ofde3pkJAJHUjbasq7GFLMmZlTqS0M+5iWtmFML/2ah3tdCNzq+zQu3qBp+EHbj3X5un3AQeoBsAEPTLXpQPykDfyL0+n5+b6YCxy/gHPcLvYYkhWfzUDO6E6rr65svvsVTg0Y0Vyp2MCyVYaz4X71/IfDQ7V4r9HHaABdLSfUxgwEJi0lyaYLwMVh774Qn4W+wQ67gUwlUkdZSRrtTYLXZ0J7TQUenT3q9CuT61KBnO+9yrZEvah86o263sGQXiG1CE2zJj85mv65ozRFkCLJYincTw9lFElGpeNHia4MfU3EfeJLHLb hBxBqz6g JhuC0i6HBrCwSSqtWwwmTJ7IF4Fv44307LBHdcjEGnJgjv5IJMyr71/B6K53+38xJsqRQjsfvTBbGLu769koSL4aeM71lq5mbc3+SVpnHVFhKZrNnRuZwxhSAmlJqNOrs3VWhVWXszKC3UoB9YW1+J0BvK5jGSL6MXwMqB+lne6c8rtss4vJJKmcXMIB/8xJ0UmSrmwPQ8Dlo1mdizX7jdRcnqoEDdxzgOqQiTtbqggPZxsA98Qq8Mbtg7Jhlp5xQ8vghlFH5pW/qzbYQ4jEt2jtUUi30peFGYYiZ62Tdzt417GxpC/lm8HNsmhN1h8K8NCS11yU2HBk2TU6D0Ariwpnv76YF9n60Lt960Y2gf+MQEzH04dCg+NiNubJdrs3orqOfcwFPPPEtd4zZLdDhCLaYbpHO2u128K+mAi+x6g708v59/TBgo/1n+4qb0zYuZQFCYsItvQ8yiP1cZL5H5E6IAIqe/wZgx8zNk0+WfgaljBTgEi1rCUwV64GXqyORoGC6g9q+pD1HC91hr9tF0rQ5seiwEe9pKv/aBIQlcge1cZrT3Sw1d7RP38/knX/5EIl30m7h9DRN/YiTwm2SQiPeWg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000015, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To simplify later changes to page tag references, introduce new pgalloc_tag_ref and pgtag_ref_handle types. This allows easy replacement of page_ext as a storage of page allocation tags. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 25 +++++---- include/linux/pgalloc_tag.h | 108 ++++++++++++++++++++++++------------ lib/alloc_tag.c | 3 +- 3 files changed, 88 insertions(+), 48 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5cd22303fbc0..8efb4a6a1a70 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4180,37 +4180,38 @@ static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new return; for (i = nr_pages; i < (1 << old_order); i += nr_pages) { - union codetag_ref *ref = get_page_tag_ref(folio_page(folio, i)); + union pgtag_ref_handle handle; + union codetag_ref ref; - if (ref) { + if (get_page_tag_ref(folio_page(folio, i), &ref, &handle)) { /* Set new reference to point to the original tag */ - alloc_tag_ref_set(ref, tag); - put_page_tag_ref(ref); + alloc_tag_ref_set(&ref, tag); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) { + union pgtag_ref_handle handle; + union codetag_ref ref; struct alloc_tag *tag; - union codetag_ref *ref; tag = pgalloc_tag_get(&old->page); if (!tag) return; - ref = get_page_tag_ref(&new->page); - if (!ref) + if (!get_page_tag_ref(&new->page, &ref, &handle)) return; /* Clear the old ref to the original allocation tag. */ clear_page_tag_ref(&old->page); /* Decrement the counters of the tag on get_new_folio. */ - alloc_tag_sub(ref, folio_nr_pages(new)); - - __alloc_tag_ref_set(ref, tag); - - put_page_tag_ref(ref); + alloc_tag_sub(&ref, folio_nr_pages(new)); + __alloc_tag_ref_set(&ref, tag); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } #else /* !CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 59a3deb792a8..bc65710ee1f9 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -9,48 +9,83 @@ #ifdef CONFIG_MEM_ALLOC_PROFILING +typedef union codetag_ref pgalloc_tag_ref; + +static inline void read_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) +{ + ref->ct = pgref->ct; +} + +static inline void write_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) +{ + pgref->ct = ref->ct; +} + +union pgtag_ref_handle { + pgalloc_tag_ref *pgref; /* reference in page extension */ +}; + #include extern struct page_ext_operations page_alloc_tagging_ops; -static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +static inline pgalloc_tag_ref *pgref_from_page_ext(struct page_ext *page_ext) { - return (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); + return (pgalloc_tag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); } -static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) +static inline struct page_ext *page_ext_from_pgref(pgalloc_tag_ref *pgref) { - return (void *)ref - page_alloc_tagging_ops.offset; + return (void *)pgref - page_alloc_tagging_ops.offset; } /* Should be called only if mem_alloc_profiling_enabled() */ -static inline union codetag_ref *get_page_tag_ref(struct page *page) +static inline bool get_page_tag_ref(struct page *page, union codetag_ref *ref, + union pgtag_ref_handle *handle) { - if (page) { - struct page_ext *page_ext = page_ext_get(page); + struct page_ext *page_ext; + pgalloc_tag_ref *pgref; - if (page_ext) - return codetag_ref_from_page_ext(page_ext); - } - return NULL; + if (!page) + return false; + + page_ext = page_ext_get(page); + if (!page_ext) + return false; + + pgref = pgref_from_page_ext(page_ext); + read_pgref(pgref, ref); + handle->pgref = pgref; + return true; +} + +static inline void put_page_tag_ref(union pgtag_ref_handle handle) +{ + if (WARN_ON(!handle.pgref)) + return; + + page_ext_put(page_ext_from_pgref(handle.pgref)); } -static inline void put_page_tag_ref(union codetag_ref *ref) +static inline void update_page_tag_ref(union pgtag_ref_handle handle, + union codetag_ref *ref) { - if (WARN_ON(!ref)) + if (WARN_ON(!handle.pgref || !ref)) return; - page_ext_put(page_ext_from_codetag_ref(ref)); + write_pgref(handle.pgref, ref); } static inline void clear_page_tag_ref(struct page *page) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; - if (ref) { - set_codetag_empty(ref); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + set_codetag_empty(&ref); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -59,11 +94,13 @@ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; - if (ref) { - alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE * nr); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -71,11 +108,13 @@ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; - if (ref) { - alloc_tag_sub(ref, PAGE_SIZE * nr); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub(&ref, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -85,13 +124,14 @@ static inline struct alloc_tag *pgalloc_tag_get(struct page *page) struct alloc_tag *tag = NULL; if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); - - alloc_tag_sub_check(ref); - if (ref) { - if (ref->ct) - tag = ct_to_alloc_tag(ref->ct); - put_page_tag_ref(ref); + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub_check(&ref); + if (ref.ct) + tag = ct_to_alloc_tag(ref.ct); + put_page_tag_ref(handle); } } @@ -106,8 +146,6 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) #else /* CONFIG_MEM_ALLOC_PROFILING */ -static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } -static inline void put_page_tag_ref(union codetag_ref *ref) {} static inline void clear_page_tag_ref(struct page *page) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 648f32d52b8d..2ef762acc203 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -474,7 +475,7 @@ static __init void init_page_alloc_tagging(void) } struct page_ext_operations page_alloc_tagging_ops = { - .size = sizeof(union codetag_ref), + .size = sizeof(pgalloc_tag_ref), .need = need_page_alloc_tagging, .init = init_page_alloc_tagging, }; From patchwork Mon Oct 14 20:36:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13835449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47E0BD18145 for ; Mon, 14 Oct 2024 20:37:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BDD26B0096; Mon, 14 Oct 2024 16:37:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 545656B0098; Mon, 14 Oct 2024 16:37:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ABBE6B0099; Mon, 14 Oct 2024 16:37:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F30126B0096 for ; Mon, 14 Oct 2024 16:37:03 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4B0C21A0C10 for ; Mon, 14 Oct 2024 20:36:49 +0000 (UTC) X-FDA: 82673366874.08.91877AA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 84452C0010 for ; Mon, 14 Oct 2024 20:36:55 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=v9263ULI; spf=pass (imf28.hostedemail.com: domain of 37IANZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=37IANZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728938033; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3awKuOd0YJMzv+83YLmL6cC1+LaUmoopuJw+yqAMR2w=; b=lGOSqGFYiRIdc6spHRDI1xizkKSpQS4N0afSM7ZQUItvYe4WN7VCmlT6jozx0ieDydqhXS cpL1Ig8skLthoBIwk0Yt5+8MtlsZ57OwXwHZkkBkvSvacd5qKiHQBv5Xyqb2HIdTWSVa8/ rhNNBmtPmyg2JRZd4L4djzweY7VaMUw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=v9263ULI; spf=pass (imf28.hostedemail.com: domain of 37IANZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=37IANZwYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728938033; a=rsa-sha256; cv=none; b=Ai2FXrfXtq4FIaZtvjBDF+tTziphyi4y/FwVzcShG27qbf+2voX0EBTXIiiS5uS1MYj1D8 wxvdA/3uyojSw5CstReyXrGGYeUZKwvp9kp29ViNDyhefRldK3U8c8H5TGQmHtTxSphU6r 7UMC41rrhhC2SNvWwHUfNeYUSmPRqKE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e321d26b38so80979487b3.2 for ; Mon, 14 Oct 2024 13:37:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1728938221; x=1729543021; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3awKuOd0YJMzv+83YLmL6cC1+LaUmoopuJw+yqAMR2w=; b=v9263ULI0vEAk5xlvozwqlVOi3cvaqVStOAO25JM46XRtlFDN/n0WHHA6XRqKsmtEu cDxx28jZeCupez2Gc6hVuHPa24ckQ1A3oRxNDzRZNP0hoWSA66f3reIGgY1O+uc373ot KkD5tcZdO722dL6dfqybB1ruvyO6S8gBzLH55mkfufr6RG7V/J9YMvOsYOw3bU/Ufho+ vrg+YLBpPRabdvVS0ziv8RQbGmwxcMFOwFo+XBf1mqQN0mUlqeIT3hIEqC+yv+33uWGn 7Rd93q1kLFI+l97ugvzUpur8wcp8SQ+SL+t8hYWPqROKpUqyAPNTk73g5SrsagxD441s 8kxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728938221; x=1729543021; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3awKuOd0YJMzv+83YLmL6cC1+LaUmoopuJw+yqAMR2w=; b=XcA5y/rC/rLfmUR0zg3t6ZkRdyAnu1tjTN+BE/Ulqhzm+BiXtMDZfn/ODHHoaVRN/z igMZ9bTrDUC5JQfHnAVuhjC51cDIbC3h8uhzy48G/K5JqMWpQCwuN2SbVXguQG4+pK+A PQlVT1l1B0JNOQs/XdJcLQYz2x/nIz0Om4eyGgO/X8Pxpirhv65/tUgy006lWOSdTPLn BqppYGC1js+TumFG4KaS/iRroNZPpuxkbexjYELXFYVnDk40TuQbdv/WZAzGpJ0PdOHT +MH2hgw4p0EXNkNPO+ZyNgOI9PXehk4PI5PD7Ru7VduMUs8ymeAdQ+fq41n5qMICjtgw 7NzQ== X-Forwarded-Encrypted: i=1; AJvYcCWp370DhN8M+/xgX3eGqy34n8tX14ohwlbgqQmJn/eKZKUo7n3h+GuQVgpjZJ/HS/f6GJc9SIKXXg==@kvack.org X-Gm-Message-State: AOJu0YyZMMqTIsd/vuAR/mHpXqgd+PyflXYSE/PCHIvkMqVq02x8Xt0d wRIfMbU70nxQxYPv86R8qkPCLlinbG8YQ7GbX7lyTAU6d2ik/40iNkcmCDnq6n2zql+yM0c+wLd kYQ== X-Google-Smtp-Source: AGHT+IFg9tfmO7bztO2On8XBGLSENOaX0zqwxf4uBCk3a2PGQpcqPZvt7HPa1I/o1H4N1GNxRLUB/lu7jhs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:915:bdd7:e08a:7997]) (user=surenb job=sendgmr) by 2002:a05:690c:7446:b0:62c:f6fd:5401 with SMTP id 00721157ae682-6e347b4b2edmr460177b3.6.1728938220715; Mon, 14 Oct 2024 13:37:00 -0700 (PDT) Date: Mon, 14 Oct 2024 13:36:46 -0700 In-Reply-To: <20241014203646.1952505-1-surenb@google.com> Mime-Version: 1.0 References: <20241014203646.1952505-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241014203646.1952505-6-surenb@google.com> Subject: [PATCH v3 5/5] alloc_tag: config to store page allocation tag refs in page flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 84452C0010 X-Stat-Signature: bjkz1ahei5tjjgkgomw7t1mac3nfmboj X-Rspam-User: X-HE-Tag: 1728938215-365354 X-HE-Meta: U2FsdGVkX1/gYJVEAN4meHiCOyap/FCb8aJygpMO7mY5j6ZJKb3gAQ9wHUpis2BBv4+9p9WTm7aYU33+VOzy9zCsYcCBVpSb1/W4bW0jGnwJ6Ffh8WsoQ1qpHRA3etyX4EqdYFpgjWXz3mn3ki0/lCL03dS7QpAsQJJMp4VcahIX/PhC1xcMVs4bZraadkGloZ/r5nAqFPnOa79OU50WEdZ3zZppnb9qKcxh1PwQjmA2wqKZPQVX85a6ERPR0lKvfxbnAbd3NbRDhffUiNluIHi2+qHjIptbBrOoFwIU4ob5ym2KmI/I8KJ/FGeF3TQAf0jbzypMlIST10jy+WEByuO2OEfV4GmDWU43QBd72ulaomB2caWcq+RMydByd3J3cuiSP1gcu8TwvPMj4CKr6j0SWhCvor5qpB0Rp7Fh2SGlQ+iwEcGgkb8sQUeCWkS/PxHXeRwBEkkRBIIX/ybc0dtX8F3PwHOoxynuvvO1rjvic/T+sz3kzblZIsSJcrDRW1OuoM9HpS3CtlS+Hndj7k1Ac0Hp5IkUeFMFUUpMSKmLubmvBAR+l1ukV2ajLWg9pfTZbN5Zhcyw42HomKlEcnz4F/6y5XzVJSm7kXhuiShEHVm0HHQyAezU/mYdJYsheyM9acW7r3a9TKT+MnCajojcOBCysNuixJzmRzeoQPAR9I5JLhrHYbIK2JNcl2ljv1f9R4CD2PDIov7R3yitlrAK7dK31zEw/34efUpGNLHtKUbwRcDZl1ef9pwoQAPwwhsm2YPpSoWSJcIJqwbFMzx2TY8Q9Gxs4GZiMdf/LqcxueHL/en3jWYMjZU6kT73/9Qt9pk2jv2E4Sp1n2fnHAeZeTRii+eJ1WJZqMnbMMW0Tr1daVQk/0on5x8yV6n5QMQCCPtmgZnPWNb6nfInoYTn3lCj64RAhvUb+RpXIWU8itnvrFffNJPOu+WwvrJhUg6RlYu7KdQuP3iwhfr kvhBmf6/ +RFe08QDBse/SsD3synBvF4BMo0O7YS1sN99/o1RViSXV/jSLYhAOpcisiQeiFUMKna6ZQ14dih/xYPdqHtNg3bfnYBdM7FS0QqeYBUwV8vvDav9dc1cdkGAQ5AK9o1dYVs1fxACWVeqQ8RPFjXeN3bHwK1Tm6+yRtS6mPVpK/er3fEL25euqyCu/B/1iwu5eWtt99KFJMiupT2yGPIaCGU0b6WYedoagn6lDIW4/gm7/mzy/aMsLztT+fcZJM6dF+0MRdJfH3SLMNy8bKQTdHhphuw8365swLHn2h0JVL4Zf8OwOf761rvEgfAe+fIzlBzld43fVbH7IqG7mGzfKUwuqnXmAeYM6eYILb36vn2xrEpM6WlpBKsmac+JIwHkSwlsXl4Nj3nNSJBEjSndoyl8ZkOhRrHaO5by8tUI1Pq8HnCbI63uTf/HuBBfatK+gWJVyFsm/JF+zT6eAB3+fqlVaggHvV3gc7z342jTyWZ/L9rZlIFKJY0aadEfkYbRhTtv7gg8ahKdIzVORKDqDFMiLcDTcQnG1pDK4L//CAoJSSDG5bQIN74aEpKrbmdO0nhMaje/aWJnkf8Gzq+uP2Ifc/g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add CONFIG_PGALLOC_TAG_USE_PAGEFLAGS to store allocation tag references directly in the page flags. This eliminates memory overhead caused by page_ext and results in better performance for page allocations. If the number of available page flag bits is insufficient to address all kernel allocations, profiling falls back to using page extensions with an appropriate warning to disable this config. If dynamically loaded modules add enough tags that they can't be addressed anymore with available page flag bits, memory profiling gets disabled and a warning is issued. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 10 +- include/linux/codetag.h | 3 + include/linux/page-flags-layout.h | 7 ++ include/linux/pgalloc_tag.h | 184 +++++++++++++++++++++++++++++- lib/Kconfig.debug | 19 +++ lib/alloc_tag.c | 90 ++++++++++++++- lib/codetag.c | 4 +- mm/mm_init.c | 5 +- 8 files changed, 310 insertions(+), 12 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 7431757999c5..4f811ec0ffe0 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,8 +30,16 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +struct alloc_tag_kernel_section { + struct alloc_tag *first_tag; + unsigned long count; +}; + struct alloc_tag_module_section { - unsigned long start_addr; + union { + unsigned long start_addr; + struct alloc_tag *first_tag; + }; unsigned long end_addr; /* used size */ unsigned long size; diff --git a/include/linux/codetag.h b/include/linux/codetag.h index fb4e7adfa746..401fc297eeda 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -13,6 +13,9 @@ struct codetag_module; struct seq_buf; struct module; +#define CODETAG_SECTION_START_PREFIX "__start_" +#define CODETAG_SECTION_STOP_PREFIX "__stop_" + /* * An instance of this structure is created in a special ELF section at every * code location being tagged. At runtime, the special section is treated as diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-layout.h index 7d79818dc065..4f5c9e979bb9 100644 --- a/include/linux/page-flags-layout.h +++ b/include/linux/page-flags-layout.h @@ -111,5 +111,12 @@ ZONES_WIDTH - LRU_GEN_WIDTH - SECTIONS_WIDTH - \ NODES_WIDTH - KASAN_TAG_WIDTH - LAST_CPUPID_WIDTH) +#define NR_NON_PAGEFLAG_BITS (SECTIONS_WIDTH + NODES_WIDTH + ZONES_WIDTH + \ + LAST_CPUPID_SHIFT + KASAN_TAG_WIDTH + \ + LRU_GEN_WIDTH + LRU_REFS_WIDTH) + +#define NR_UNUSED_PAGEFLAG_BITS (BITS_PER_LONG - \ + (NR_NON_PAGEFLAG_BITS + NR_PAGEFLAGS)) + #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index bc65710ee1f9..69976cf747a6 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -9,6 +9,93 @@ #ifdef CONFIG_MEM_ALLOC_PROFILING +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS + +typedef u16 pgalloc_tag_ref; + +extern struct alloc_tag_kernel_section kernel_tags; + +#define CODETAG_ID_NULL 0 +#define CODETAG_ID_EMPTY 1 +#define CODETAG_ID_FIRST 2 + +#ifdef CONFIG_MODULES + +extern struct alloc_tag_module_section module_tags; + +static inline struct codetag *get_module_ct(pgalloc_tag_ref pgref) +{ + return &module_tags.first_tag[pgref - kernel_tags.count].ct; +} + +static inline pgalloc_tag_ref get_module_pgref(struct alloc_tag *tag) +{ + return CODETAG_ID_FIRST + kernel_tags.count + (tag - module_tags.first_tag); +} + +#else /* CONFIG_MODULES */ + +static inline struct codetag *get_module_ct(pgalloc_tag_ref pgref) +{ + pr_warn("invalid page tag reference %lu\n", (unsigned long)pgref); + return NULL; +} + +static inline pgalloc_tag_ref get_module_pgref(struct alloc_tag *tag) +{ + pr_warn("invalid page tag 0x%lx\n", (unsigned long)tag); + return CODETAG_ID_NULL; +} + +#endif /* CONFIG_MODULES */ + +static inline void read_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) +{ + pgalloc_tag_ref pgref_val = *pgref; + + switch (pgref_val) { + case (CODETAG_ID_NULL): + ref->ct = NULL; + break; + case (CODETAG_ID_EMPTY): + set_codetag_empty(ref); + break; + default: + pgref_val -= CODETAG_ID_FIRST; + ref->ct = pgref_val < kernel_tags.count ? + &kernel_tags.first_tag[pgref_val].ct : + get_module_ct(pgref_val); + break; + } +} + +static inline void write_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) +{ + struct alloc_tag *tag; + + if (!ref->ct) { + *pgref = CODETAG_ID_NULL; + return; + } + + if (is_codetag_empty(ref)) { + *pgref = CODETAG_ID_EMPTY; + return; + } + + tag = ct_to_alloc_tag(ref->ct); + if (tag >= kernel_tags.first_tag && tag < kernel_tags.first_tag + kernel_tags.count) { + *pgref = CODETAG_ID_FIRST + (tag - kernel_tags.first_tag); + return; + } + + *pgref = get_module_pgref(tag); +} + +void __init alloc_tag_sec_init(void); + +#else /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + typedef union codetag_ref pgalloc_tag_ref; static inline void read_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) @@ -21,8 +108,13 @@ static inline void write_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) pgref->ct = ref->ct; } +static inline void alloc_tag_sec_init(void) {} + +#endif /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + union pgtag_ref_handle { pgalloc_tag_ref *pgref; /* reference in page extension */ + struct page *page; /* reference in page flags */ }; #include @@ -40,8 +132,8 @@ static inline struct page_ext *page_ext_from_pgref(pgalloc_tag_ref *pgref) } /* Should be called only if mem_alloc_profiling_enabled() */ -static inline bool get_page_tag_ref(struct page *page, union codetag_ref *ref, - union pgtag_ref_handle *handle) +static inline bool page_ext_get_page_tag_ref(struct page *page, union codetag_ref *ref, + union pgtag_ref_handle *handle) { struct page_ext *page_ext; pgalloc_tag_ref *pgref; @@ -59,7 +151,7 @@ static inline bool get_page_tag_ref(struct page *page, union codetag_ref *ref, return true; } -static inline void put_page_tag_ref(union pgtag_ref_handle handle) +static inline void page_ext_put_page_tag_ref(union pgtag_ref_handle handle) { if (WARN_ON(!handle.pgref)) return; @@ -67,8 +159,8 @@ static inline void put_page_tag_ref(union pgtag_ref_handle handle) page_ext_put(page_ext_from_pgref(handle.pgref)); } -static inline void update_page_tag_ref(union pgtag_ref_handle handle, - union codetag_ref *ref) +static inline void page_ext_update_page_tag_ref(union pgtag_ref_handle handle, + union codetag_ref *ref) { if (WARN_ON(!handle.pgref || !ref)) return; @@ -76,6 +168,87 @@ static inline void update_page_tag_ref(union pgtag_ref_handle handle, write_pgref(handle.pgref, ref); } +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS + +DECLARE_STATIC_KEY_TRUE(mem_profiling_use_pageflags); +extern unsigned long alloc_tag_ref_mask; +extern int alloc_tag_ref_offs; + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline bool get_page_tag_ref(struct page *page, union codetag_ref *ref, + union pgtag_ref_handle *handle) +{ + pgalloc_tag_ref pgref; + + if (!static_key_enabled(&mem_profiling_use_pageflags)) + return page_ext_get_page_tag_ref(page, ref, handle); + + if (!page) + return false; + + pgref = (page->flags >> alloc_tag_ref_offs) & alloc_tag_ref_mask; + read_pgref(&pgref, ref); + handle->page = page; + return true; +} + +static inline void put_page_tag_ref(union pgtag_ref_handle handle) +{ + if (!static_key_enabled(&mem_profiling_use_pageflags)) { + page_ext_put_page_tag_ref(handle); + return; + } + + WARN_ON(!handle.page); +} + +static inline void update_page_tag_ref(union pgtag_ref_handle handle, union codetag_ref *ref) +{ + unsigned long old_flags, flags, val; + struct page *page = handle.page; + pgalloc_tag_ref pgref; + + if (!static_key_enabled(&mem_profiling_use_pageflags)) { + page_ext_update_page_tag_ref(handle, ref); + return; + } + + if (WARN_ON(!page || !ref)) + return; + + write_pgref(&pgref, ref); + val = (unsigned long)pgref; + val = (val & alloc_tag_ref_mask) << alloc_tag_ref_offs; + do { + old_flags = READ_ONCE(page->flags); + flags = old_flags; + flags &= ~(alloc_tag_ref_mask << alloc_tag_ref_offs); + flags |= val; + } while (unlikely(!try_cmpxchg(&page->flags, &old_flags, flags))); +} + +#else /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline bool get_page_tag_ref(struct page *page, union codetag_ref *ref, + union pgtag_ref_handle *handle) +{ + return page_ext_get_page_tag_ref(page, ref, handle); +} + +static inline void put_page_tag_ref(union pgtag_ref_handle handle) +{ + page_ext_put_page_tag_ref(handle); +} + +static inline void update_page_tag_ref(union pgtag_ref_handle handle, + union codetag_ref *ref) +{ + page_ext_update_page_tag_ref(handle, ref); +} + +#endif /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + static inline void clear_page_tag_ref(struct page *page) { if (mem_alloc_profiling_enabled()) { @@ -152,6 +325,7 @@ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL; } static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {} +static inline void alloc_tag_sec_init(void) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 7315f643817a..f4c272c30719 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1017,6 +1017,25 @@ config MEM_ALLOC_PROFILING_DEBUG Adds warnings with helpful error messages for memory allocation profiling. +config PGALLOC_TAG_USE_PAGEFLAGS + bool "Use pageflags to encode page allocation tag reference" + default n + depends on MEM_ALLOC_PROFILING + help + When set, page allocation tag references are encoded inside page + flags if they fit, otherwise they are encoded in page extensions. + + Setting this flag reduces memory and performance overhead of memory + allocation profiling but it requires enough unused page flag bits to + address all kernel allocations. If enabled but there are not enough + unused page flag bits, profiling will fall back to using page + extensions and a warning will be issued to disable this config. + If dynamically loaded modules add enough tags that they can't be + addressed anymore with available page flag bits, profiling for such + modules will be disabled and a warning will be issued. + + Say N if unsure. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 2ef762acc203..341a5c826449 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -152,6 +153,42 @@ static void __init procfs_init(void) proc_create_seq("allocinfo", 0400, NULL, &allocinfo_seq_op); } +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS + +DEFINE_STATIC_KEY_TRUE(mem_profiling_use_pageflags); +unsigned long alloc_tag_ref_mask; +int alloc_tag_ref_offs; + +#define SECTION_START(NAME) (CODETAG_SECTION_START_PREFIX NAME) +#define SECTION_STOP(NAME) (CODETAG_SECTION_STOP_PREFIX NAME) + +struct alloc_tag_kernel_section kernel_tags = { NULL, 0 }; + +void __init alloc_tag_sec_init(void) +{ + struct alloc_tag *last_codetag; + + kernel_tags.first_tag = (struct alloc_tag *)kallsyms_lookup_name( + SECTION_START(ALLOC_TAG_SECTION_NAME)); + last_codetag = (struct alloc_tag *)kallsyms_lookup_name( + SECTION_STOP(ALLOC_TAG_SECTION_NAME)); + kernel_tags.count = last_codetag - kernel_tags.first_tag; + + /* Check if kernel tags fit into page flags */ + if (kernel_tags.count > (1UL << NR_UNUSED_PAGEFLAG_BITS)) { + static_branch_disable(&mem_profiling_use_pageflags); + pr_warn("%lu kernel tags do not fit into %d available page flag bits, page extensions will be used instead!\n", + kernel_tags.count, NR_UNUSED_PAGEFLAG_BITS); + } else { + alloc_tag_ref_offs = (LRU_REFS_PGOFF - NR_UNUSED_PAGEFLAG_BITS); + alloc_tag_ref_mask = ((1UL << NR_UNUSED_PAGEFLAG_BITS) - 1); + pr_info("All unused page flags (%d bits) are used to store page tag references!\n", + NR_UNUSED_PAGEFLAG_BITS); + } +} + +#endif /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + #ifdef CONFIG_MODULES static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); @@ -161,7 +198,29 @@ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ static struct module prepend_mod; -static struct alloc_tag_module_section module_tags; +struct alloc_tag_module_section module_tags; + +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS +static inline unsigned long alloc_tag_align(unsigned long val) +{ + if (val % sizeof(struct alloc_tag) == 0) + return val; + return ((val / sizeof(struct alloc_tag)) + 1) * sizeof(struct alloc_tag); +} + +static inline bool tags_addressable(void) +{ + unsigned long tag_idx_count = CODETAG_ID_FIRST + kernel_tags.count + + module_tags.size / sizeof(struct alloc_tag); + + return tag_idx_count < (1UL << NR_UNUSED_PAGEFLAG_BITS); +} + +#else /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + +static inline bool tags_addressable(void) { return true; } + +#endif /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ static bool needs_section_mem(struct module *mod, unsigned long size) { @@ -237,6 +296,21 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, if (!align) align = 1; +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS + /* + * If alloc_tag size is not a multiple of required alignment tag + * indexing does not work. + */ + if (!IS_ALIGNED(sizeof(struct alloc_tag), align)) { + pr_err("%s: alignment %lu is incompatible with allocation tag indexing, disable CONFIG_PGALLOC_TAG_USE_PAGEFLAGS", + mod->name, align); + return ERR_PTR(-EINVAL); + } + + /* Ensure prepend consumes multiple of alloc_tag-sized blocks */ + if (prepend) + prepend = alloc_tag_align(prepend); +#endif /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ mas_lock(&mas); repeat: /* Try finding exact size and hope the start is aligned */ @@ -305,6 +379,12 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, unsigned long phys_size = vm_module_tags->nr_pages << PAGE_SHIFT; module_tags.size = offset + size; + if (!tags_addressable() && mem_alloc_profiling_enabled()) { + static_branch_disable(&mem_alloc_profiling_key); + pr_warn("With module %s there are too many tags to fit in %d page flag bits. Memory profiling is disabled!\n", + mod->name, NR_UNUSED_PAGEFLAG_BITS); + } + if (phys_size < module_tags.size) { int grow_res; @@ -406,6 +486,10 @@ static int __init alloc_mod_tags_mem(void) module_tags.start_addr = (unsigned long)vm_module_tags->addr; module_tags.end_addr = module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_SIZE; +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS + /* Ensure the base is alloc_tag aligned */ + module_tags.start_addr = alloc_tag_align(module_tags.start_addr); +#endif return 0; } @@ -467,6 +551,10 @@ early_param("sysctl.vm.mem_profiling", setup_early_mem_profiling); static __init bool need_page_alloc_tagging(void) { +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS + if (static_key_enabled(&mem_profiling_use_pageflags)) + return false; +#endif return mem_profiling_support; } diff --git a/lib/codetag.c b/lib/codetag.c index 3f3a4f4ca72d..cda5a852ef30 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -149,8 +149,8 @@ static struct codetag_range get_section_range(struct module *mod, const char *section) { return (struct codetag_range) { - get_symbol(mod, "__start_", section), - get_symbol(mod, "__stop_", section), + get_symbol(mod, CODETAG_SECTION_START_PREFIX, section), + get_symbol(mod, CODETAG_SECTION_STOP_PREFIX, section), }; } diff --git a/mm/mm_init.c b/mm/mm_init.c index 4ba5607aaf19..1c205b0a86ed 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -83,8 +83,7 @@ void __init mminit_verify_pageflags_layout(void) unsigned long or_mask, add_mask; shift = BITS_PER_LONG; - width = shift - SECTIONS_WIDTH - NODES_WIDTH - ZONES_WIDTH - - LAST_CPUPID_SHIFT - KASAN_TAG_WIDTH - LRU_GEN_WIDTH - LRU_REFS_WIDTH; + width = shift - NR_NON_PAGEFLAG_BITS; mminit_dprintk(MMINIT_TRACE, "pageflags_layout_widths", "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Gen %d Tier %d Flags %d\n", SECTIONS_WIDTH, @@ -2639,7 +2638,7 @@ void __init mm_core_init(void) BUILD_BUG_ON(MAX_ZONELISTS > 2); build_all_zonelists(NULL); page_alloc_init_cpuhp(); - + alloc_tag_sec_init(); /* * page_ext requires contiguous pages, * bigger than MAX_PAGE_ORDER unless SPARSEMEM.