From patchwork Mon May 1 16:54:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C04BC7EE29 for ; Mon, 1 May 2023 16:55:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 813F3900004; Mon, 1 May 2023 12:55:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C443900002; Mon, 1 May 2023 12:55:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 66513900004; Mon, 1 May 2023 12:55:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 57FE9900002 for ; Mon, 1 May 2023 12:55:14 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0488BA0739 for ; Mon, 1 May 2023 16:55:13 +0000 (UTC) X-FDA: 80742286548.19.1C5F5E5 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf12.hostedemail.com (Postfix) with ESMTP id 34D574001D for ; Mon, 1 May 2023 16:55:11 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=2YKjiLNW; spf=pass (imf12.hostedemail.com: domain of 37e5PZAYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=37e5PZAYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960111; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tgVoyGNNM2CEwgGlnNhwRO5zW+BCSfKlhykn/JkHIoY=; b=gWXPUTIGomXOM6MVjeRM5MfW/dYo2LT/fX6TnA+Z9WaJUgFwbr8Orw6Quj+6GkfpWWmUDV nskwEjhq5ozIZS5FRN2mnmNHSXTRm/j4BxAaLrZ+OsD2uQfVvi2BwXBL5T86gGtxMD+3Jf 4wHMZiN2fVBAGA1UhB+dOGwzZ2T57iw= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=2YKjiLNW; spf=pass (imf12.hostedemail.com: domain of 37e5PZAYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=37e5PZAYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960111; a=rsa-sha256; cv=none; b=U1DTLA5GnHJvJxne6WDrCtirEMmf0TPGpf9B4O6CX4dSlg4hHDySIePJIGYuLt9369vAck aCEzYOFa5qz1JXfYR1cxI7//8Sxl2SaHcHGO9L5LxSuNlcTrSLvnpgTFn2gLtRf2icPc/p A9WPagWeqTtLZzPzVceovjjPqHm1IiE= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-24df9b0ed7aso1439414a91.3 for ; Mon, 01 May 2023 09:55:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960110; x=1685552110; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=tgVoyGNNM2CEwgGlnNhwRO5zW+BCSfKlhykn/JkHIoY=; b=2YKjiLNW+kfuA2PzlgEprNmYD56U2DvqAYUQT3SxKi2Wkcms/FURL1MvVV4lKJqj8O 6/Ns7jrOtbP8ic5/FVYS4laNGQMz7H1sQUQlCcm/XRw/h77Gbtg2COyXGd2xvroS0fJn qwWPXD+NTjhYN8F2PIhFlV9xM7/YRbLeL8oWUbHvCo0bmT2YG7Au7XFXE7JF9tMEE+O2 Razpx64/yqsX3p0ZTAuL1qwvem7fqbeaFnGoyPVJo12mzY/mv/yC7mdLiyEupHVMtu6T xYmQq3px4UJy+E+76my+C+5ib12oMdQjlgTcOmFRh/w96xFVezYHyxkBpY3bPmKf/8tt YK/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960110; x=1685552110; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=tgVoyGNNM2CEwgGlnNhwRO5zW+BCSfKlhykn/JkHIoY=; b=UWnhy0fgl1ZggHyvqxiOIdoQXjUt50JZ3vezsiW0L87kMO2hJpgwf5UibQ89PI+3Rn EkEz2n7d3c+awBNwKcofJduVNcuaM4XTq8gAXw/KY+tc6JjKl3OaHBpQiaxyzbYTx0p/ cxKsjoQ/vYwpYLo39E5U+jwd1SMgPw/Zx/KrjJ9f9oqoaqRz3LfEq9u4ChSYAqOKkHDK TmxTDbdRxydGlLu+AAmw4HGOuj13rOZ72pdk+hzCLAks5Y7eLAfA9Qx8kmfvDzCqIs3V M1Om3qfRxQKZTpzbif0GgFMZ7Nwldi5P2yBPhQQccYO13garEDu07W01/WApttN/dxy8 tNHg== X-Gm-Message-State: AC+VfDyXGuopTOID63IXBRXcvng3UMvUxbQuENMq2bFqio3saIUa5Uhx 1OBLxOaREbH4kqZKFMSLpWznrfYVF1w= X-Google-Smtp-Source: ACHHUZ4eBSTFzzY/h621MY7B4DCDMh7rr4Q0GpDOy2/h5VKYZfljnAlN4rMUCg0PfIpVagexBlmQ2RgWd74= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a17:90a:390d:b0:246:66d6:f24e with SMTP id y13-20020a17090a390d00b0024666d6f24emr3850182pjb.2.1682960109840; Mon, 01 May 2023 09:55:09 -0700 (PDT) Date: Mon, 1 May 2023 09:54:11 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-2-surenb@google.com> Subject: [PATCH 01/40] lib/string_helpers: Drop space in string_get_size's output From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Andy Shevchenko , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , "Michael S. Tsirkin" , Jason Wang , " =?utf-8?q?Noralf_Tr=C3=B8nnes?= " X-Rspamd-Queue-Id: 34D574001D X-Stat-Signature: j1jtu4u86ddr9zyx9t98tzaxiiuteuyu X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682960111-972321 X-HE-Meta: U2FsdGVkX18YKGapKfmIOdWCGutMw16PXe3k7PeDr1MELMOQAh7xhQrkSCoEQ3jFnVSr/22cDlmf+cDmkZWbZNeMf0NvtN+s91QueLF2/PFwZaXwaP/YJ6BmEiUmerXRtmJo0gxNSu8uOyvT5G8a6TnZp9x1y74NsMzh+Fht72dN6jwfFUGJct9IpP8WKFsyii8f088fu13Cbgj6FoFwIILc+TAgyj8Qnt6hd2QF0l7C9K3vrAPnIwIW0PXp5qqeKR1rfLPzcraQdTk6CayVvKfeN58ffUBtMDeB7QjRNHSz952AtFCRZb5EOYaDFLkgPUCNfg950e9ulukWuzzeoiqvrRuDXHh5HoYt6uHiQdNzJugnBCRS99XggGNEt7+s4GNXUUzGyW/d2ilZcNiXeYg0n4qvGFvsxNt1xYk9W9x/OIbAAbERudd3Ml0oAT5jVzuusnCDKBpcjDFeui1YdoTCmL60R1ZDrgm4Uy8HHxyu7qQRF3UsnDYPP3VzL8O/NMBTNKkyrcgaAGy/+GWtuIc7tfYS585ScVrBCW9fYmzL/RJkRTY6oE1HjRnAWT1yLsSutw0rIev/B0c+MIyeFdNydEKyb0sGO8xQQXYcO1s86kH5QYNGB+TA+b9BVarrIba5jkIPhyOz7xhYaGaUq/75HSih1P1PdkP2a4bie/BdfStkk2qXxe48QzhvqHw0sCx0O8d88swsfkGfh0Y9HdGzNJd3XChNA9oEm53HH5nn8FCZapYjIjX7UGfQuELVlz06Gi4jBkQqTPtvfkOCpKnGL1ALju3qSsilru0VgfLj2EUrSKyXYw4nlMkpLIdJ13u5p5juRZOOisiGiGDU9suuKo4cnzf0K0WzMnJ7mlMuynblddKnIMtz7UnpChVhr6pgDAlgb89bNcBswtmp++TiZf6/W2bMrpD3nef0/oNcnBQ41K0nwQ8+/7omIbMXz/6kf0NTL6uQpVzqXI9 BUuaZwMo 7uswGH3O7cj4xivOAAmpwOarysWp5vud/5FSNToHa3N5dfJxe8Ubypa/zs5IdVtvWlRMG4vs/3t8aSKeQPOeen8Vw9NL5JyLRpVsgIloKKC6PPSldm2K29KArmc4eaqLoEscBixYRYdAHq70+sFbyukJi3Yde73AczVP0n4TEC6NqNa9fZuvZjTNyYBlsYl9EA14YXxcrOXtisGvfN/RVaxVvZhCqSAdUKnNFX7ui/BGEvlmGeL6p734mPxikDXPft+q++7yWqT51LZbcfgqCLB8/uh91YRwTv+s1m+Ckv5Rx2r67v/DY5YRKJAMOn0eoyWs/v16I+c/nP/+SxqKTP7B91vjSWMI042nnNEDrrYPgQCYbKw+SIWlnY0/OrYuQiSL0KDrsAY+ABrU50PomoEBSMf9UQR9qKYZDI7kdZBrKvgyBKF0pdvwivNauWpNdLkG2Zp4hLQqahidvx3VjdeBoKqh0IMcmH8uU3rZKMbIPYwfVqA7MPuE7kL3lvEMy8guZcNlvWZVKZ8gmrhjYBM6wmPmjeKx635rsdyjP2L3YvEKASJenbXVOC1q7kr+tBQGwjigD7q92f5HHxIy0pBcXJDW/QhWDHKh+ae4NPKIAPZDBWycuCcxnVglNA7G24WS/Ufai2SanzswAzYJ+1kZZVdSGfqI25FGznyKLbuOCZIQdWVMD+uuWW7B0CmkrTOyzRc/XSruV/ExBVnBbTsIZZItZZF3q2Q1h/2+/ior4J7nTo7O1netjK1J8X64ZcpCxxKsmVBFsVGs7MUS6E45hJQMZVirsTE+jiYCKAID47P/xTtC1WmfzTPbkVBWywy52Kimxz+ktLDRUdOLceptDkcd6uKtIBz3+K7dZcrXlMgxdfPX3PQOalL8Xi+D6VLbL4+wH9CEanfl7tCNCJEgTtRCuUQeJkZlv2hlqmEXISo98uHg7jh97tE/uG/KphD0zEISyWUta9gYFkqmiN8ux1zmJ lUjnNlOJ TJI/058+XUc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet Previously, string_get_size() outputted a space between the number and the units, i.e. 9.88 MiB This changes it to 9.88MiB which allows it to be parsed correctly by the 'sort -h' command. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andy Shevchenko Cc: Michael Ellerman Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: "Michael S. Tsirkin" Cc: Jason Wang Cc: "Noralf Trønnes" Cc: Jens Axboe --- lib/string_helpers.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/string_helpers.c b/lib/string_helpers.c index 230020a2e076..593b29fece32 100644 --- a/lib/string_helpers.c +++ b/lib/string_helpers.c @@ -126,8 +126,7 @@ void string_get_size(u64 size, u64 blk_size, const enum string_size_units units, else unit = units_str[units][i]; - snprintf(buf, len, "%u%s %s", (u32)size, - tmp, unit); + snprintf(buf, len, "%u%s%s", (u32)size, tmp, unit); } EXPORT_SYMBOL(string_get_size); From patchwork Mon May 1 16:54:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB4D4C77B73 for ; Mon, 1 May 2023 16:55:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C75E900005; Mon, 1 May 2023 12:55:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77A6A900002; Mon, 1 May 2023 12:55:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63E2E900005; Mon, 1 May 2023 12:55:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 57CBC900002 for ; Mon, 1 May 2023 12:55:15 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1EF601C7298 for ; Mon, 1 May 2023 16:55:15 +0000 (UTC) X-FDA: 80742286590.05.84138CB Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 54BB840009 for ; Mon, 1 May 2023 16:55:13 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ym8bWqOd; spf=pass (imf17.hostedemail.com: domain of 38O5PZAYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=38O5PZAYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960113; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OgbhlQnm6N3CIqx+6EXURMMvow4S1KQvuNCpohSnl08=; b=M3ltAAduqMNxXYtvHqZ3kHuVyTnqSY4atD7sXEmcXID64sghE8hSi9aqRw4BG4lSxI/pB/ OIJ1GwJQBjrs0jBN2vJ6xxLaG6oektDh12zDd2SGwn6tJK8WBPGtgjNiz2qRhhsnT7ZdH/ Fve8jtMw768JXY/R3bEW30RIoxk+Vfs= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ym8bWqOd; spf=pass (imf17.hostedemail.com: domain of 38O5PZAYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=38O5PZAYKCDclnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960113; a=rsa-sha256; cv=none; b=cpmCxJYbgYj5p8HmL8v/dRZbMexDRzcxpsj4Z7Zuhqeg9KWAXe24j0gYwyy8U1Uu45bFLp Zp6rOWIkfkk2bwt0tSFTJDWTqir9WwhZHrqbF/NIORfAstGCMiwFHG6K+zE/o/IA/uoz7m bmx/b5myuuyFriJOY5jurAcByW/pMxY= Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-52855ba7539so1413420a12.3 for ; Mon, 01 May 2023 09:55:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960112; x=1685552112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OgbhlQnm6N3CIqx+6EXURMMvow4S1KQvuNCpohSnl08=; b=ym8bWqOddRKLLCHNc+7Y2IQs+szb6DlXVMA8aGLldE3slVUGdWc6AeUaj2cctynpb7 /42bu00/rNI9NmOjux118bETqQ9xSRBpI0DlCMlMXfD5dyA5fGqmz9wEGRomBF9wE3qi RRE1kSFRttrNPJ5gUqGivtomoZ/CkHpvdtT+8kMIwh52/Lg8aqiW8D9XXtWx1rDhFoLd bIJ0Jt1xQTF7bUEViJLE2KzQyVYsdDW/+ihFxINfhU+/Rq1Q5G2PY50HQQ7ppVzde88x xSaqGzF/rOMvs2lVTYEukIGUCSWj1ys72INhoG28un8pv0+7d++N9bkCwOHBDc2Mepzd xe8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960112; x=1685552112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OgbhlQnm6N3CIqx+6EXURMMvow4S1KQvuNCpohSnl08=; b=Cc3Zl9ICYSQCXEXyeY6YrrwX7EZzJ32ejLO+ZZ6tpJX3kD4eJyb1JgISYlqs3XEP6/ swaN3V4I78wxw1oeXfL8tfd26YaY1f+HEnadM6ZKn+2xYYXOr9GjyWZqjcbs3h5UvaJ9 ik1rAdVhh87i87wPo+I2Di/26Y8cN9bAPxACMQHxVawf/Au6RmQ4fPuGvLZpc77OZwa+ TFEJ48ilfGAoORjk9jUmyfldO/xyMsllh70eU3vWv3iRRHrzpGFb6iVlI8VJQtVWjUsW niRBvm/2Q0qjSgMdJtVhgMWIcotL4KTqAct7RDmgQW6LIfSLxOEgfZKMYJQlz7VTn4gS jZgA== X-Gm-Message-State: AC+VfDxg2vQHJ8j0yRdLGizm+XufBlZRzqc6Fx/6QgcPudb6BD5VmREE EWxNqQrntjuV3YHPnI2ysr2H2siSZ08= X-Google-Smtp-Source: ACHHUZ5b/dFHDo+PuE3NWONWxlaUnbiu4qgjE2QFFJKcdOrghbUOD/toAldYYzf7QRUKjILKXa170Oj8yTI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a63:24f:0:b0:520:60ac:fb30 with SMTP id 76-20020a63024f000000b0052060acfb30mr3551005pgc.1.1682960112196; Mon, 01 May 2023 09:55:12 -0700 (PDT) Date: Mon, 1 May 2023 09:54:12 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-3-surenb@google.com> Subject: [PATCH 02/40] scripts/kallysms: Always include __start and __stop symbols From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: g8hy31jwafemb78zfo9y8gdgcdcdfek7 X-Rspam-User: X-Rspamd-Queue-Id: 54BB840009 X-Rspamd-Server: rspam06 X-HE-Tag: 1682960113-142920 X-HE-Meta: U2FsdGVkX1/P9q73xFDzUvDJ8w/6UstNxBxbX0ABvndE9dvX0VD2upBIHqTf9AIqRFlOSfCZT1LBxZb3GxsOZdgxJZPxxltW1XrhOaw/Y+mjRTGP3N5TVwZ2Gis5z4qELpV3RIPzhY1zhpX6iqpdVuV5MqzreL6RDt9VU3YF6gJ/tcGG+qe4fc4HKlr0GYJZQWGl+8QDE1hWfQRAeiQ0ghRg5k7C0FSj91miA2JJV/7iBh5ehU/5mxcdEc5l8apBXvzoR0lJZvpyOKDjodZdiGV9FhWQ4WAIfAsV933ZtIjCriV2vM5aVGoKpZ8AEj62IAeROwPu/g6bRp++09WwACYCe8snxk2UXeuliF14otrF0fk2gJp843QWMv8DToUzHcH3Ba17HfPFH3KM6zm7IhV77S9PEPprmVLCbrVI590w/LmMTXNGDVkt+t7cgnTxWUzkk+XwBEGNzx/w5IcJFUMjoXhRgqC/MGh8+rTj3pAM/XlRgXi99Fcwn9LN3WptnaSqjTZgEjZBp8XATnFzvPcwFyIbWzR4yIdMrQzZcBRioPGDoVXOOJuYhrLrdPAyA2DDqarbws/+mvzduq12ZjdS+JhLUtNL785nzLEf7ZiYBjCkJM2u6+LZII8aVBm+jt+On7P7nQoS5WF2BNEERkRWpKFx3q75Lz4mhUKHHbYUq8gujE/cdZJolpwsKVkyqrYclmBvJRVPgtGXz+aRx/HbKT6VxtAAzi1Mr5ZV81D/FOYQC4VwEEhZ22+y5lWDDIMOgfDotEWPqShKUXAHj+Yr0LYcbjrnxzcF0ocQaJx+Zpi1LcvWaWck/5PXwlPqYtwwC6G4vI9iwJiZ88P9Qmw7W7fonnTCH58SqQNiFyhgGusKHHGnI/Q6tozxmLS+r/BI1w41gltwzjt6pgHlnswFORoqJwzJjA8SMm2oL7Kt/ANCQYPtOOPhir4lrW248itou3tYi/Zb7g28WXX q5+Zd4mD mEoEz6XiF/6RKs7REsQDU+Xey6wdQiGJbjEZ1A7WGLceIjFd4Xp6HAQ/tJ3GSCeLnFkDCCal9Kn5mv/UUpPn05HSSI/FBiwG585fzdLZImu1HuQQkNSodaHtbfktfSgcVaZswfA8T/p7Zdli3AOJ3auzGw769VoFrYaBXJl5dnWHVjpZ6u4+xmebUuEtaqPhzYmGVC7pYA9sHXoEB0qlyVF72BzBHxy6UWk9ohqJQC+Puo+l0XCnfIfx8NxBE+kdwnVAXwVJzkk15vsgXKBpYm8zOsVSIU5v0HXkDf4OT8a6N2U70ZeCaoshudPX1Pee+lJKWYruKblNJEbp35FoWwMCqevK4jSg97+nC5N19Adb8tCcvFHmE1uUxOd4wEs5tdPinLZdMANFIooACc1OvnKfdcw/1gfNbxbONg6ONq2Am+4Og1ch7+ktXnFNoWq6dzwFIzsXa30KnrGErPgEWQUazRN3QXqKK48Qh6MLzMPk3EV03BgWsrPbraxvOql7shhNrVktrMJyMDfGsVDlAtHNPPaVuLb3eIJTyMkMvIVehge5tzap/X+wt6zJy1ydipz4SbRMR0Xf/2sRFhYVsiKu7R08ZGV2AQLuIkHxOWgXPTl6UpwaRETreWLy4b+NjNWTGEM8YqBmsjgKyit272/2YcHLnrMeN7c+HrtDSyaBrvHyDhwfcnlo18i/YjHPC9FbydX5OT8NudinnhHIcLiBjw96yDN88m+4S X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet These symbols are used to denote section boundaries: by always including them we can unify loading sections from modules with loading built-in sections, which leads to some significant cleanup. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- scripts/kallsyms.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 0d2db41177b2..7b7dbeb5bd6e 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -203,6 +203,11 @@ static int symbol_in_range(const struct sym_entry *s, return 0; } +static bool string_starts_with(const char *s, const char *prefix) +{ + return strncmp(s, prefix, strlen(prefix)) == 0; +} + static int symbol_valid(const struct sym_entry *s) { const char *name = sym_name(s); @@ -210,6 +215,14 @@ static int symbol_valid(const struct sym_entry *s) /* if --all-symbols is not specified, then symbols outside the text * and inittext sections are discarded */ if (!all_symbols) { + /* + * Symbols starting with __start and __stop are used to denote + * section boundaries, and should always be included: + */ + if (string_starts_with(name, "__start_") || + string_starts_with(name, "__stop_")) + return 1; + if (symbol_in_range(s, text_ranges, ARRAY_SIZE(text_ranges)) == 0) return 0; From patchwork Mon May 1 16:54:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04001C77B7C for ; Mon, 1 May 2023 16:55:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94C77900006; Mon, 1 May 2023 12:55:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 92501900002; Mon, 1 May 2023 12:55:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EBBF900006; Mon, 1 May 2023 12:55:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6E546900002 for ; Mon, 1 May 2023 12:55:19 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 48AEB80805 for ; Mon, 1 May 2023 16:55:18 +0000 (UTC) X-FDA: 80742286716.05.7F330D9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 6F189100023 for ; Mon, 1 May 2023 16:55:15 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=kNURb9oR; spf=pass (imf14.hostedemail.com: domain of 38u5PZAYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=38u5PZAYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960115; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bGdFaZ9yLSvpkzL04e/VHNU0a1vWusSmGZaxEHf2wm8=; b=jzXvzPQBJgLHpx+PF6jOxngGxclUbGmFdkHwoSDJfAQU8M8t12o+V22LDvAM8aeGxMYgwJ 8ibzC0AQ/GCYLsYMI5sDK0rQd57Gzle7WaCtrZPSE9L5RlrRnXCt0XdMu4Y1jTBgm79LL1 7qmhvGkko91LkxMo05vW135c4kE0wSY= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=kNURb9oR; spf=pass (imf14.hostedemail.com: domain of 38u5PZAYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=38u5PZAYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960115; a=rsa-sha256; cv=none; b=DdpiaroqRox5MgH3awoD+hRkgup+xEExlPn621ZDjaKy4Cdt3iqkpHT1b48G3gtPGWAz0+ heULVi145i99o8ZNcitxjfGVSPZrxmMFv8sa7zGzYuugw4wMeyRAeDykk4mE6yW00k8Iop O732Z+ZsHoqYfUEFnk7RdH9xoiPVI8g= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-559e55b8766so38359207b3.1 for ; Mon, 01 May 2023 09:55:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960114; x=1685552114; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bGdFaZ9yLSvpkzL04e/VHNU0a1vWusSmGZaxEHf2wm8=; b=kNURb9oRoWKMv7g0yo2L8rW1EcO17+d5GcAx/UcGQY6Nq04eWDbD4SbdCNOLXgOYzq iIefzjvXI4Snk9VN4a44AHM12D0iuNdtL0q3JhvnQXzZtf429k8SdHQyNq1DhIH3u3D/ vgRvJtIdIGprvSKRamhSV8bn0NWTicrhYnvBvcVO6D5qUdO+wBWr5u0wD0Nu7LbQQb+M IIs2pzxiZlPELIg9+H7fv7z/PKPu/SB3zB+CWnrv1/57UJqzRl/3C3lgPuiSCXQkCpOG 4JQ2sAFEorBhte2EmAzube4SlGlrch2L/NYmCg7/2eMQe55a1P+zlu78IrB4haHDMTMn h3Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960114; x=1685552114; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bGdFaZ9yLSvpkzL04e/VHNU0a1vWusSmGZaxEHf2wm8=; b=PqNJDLBpet//XbezV2RNgKen7MVdxNlWp43pNTopt1bragAU08kOaK9TIYwitAPDb1 Ms2MkvOcY6v2IWDUihVyls3k8soLA320NPKBg/LTobfZL6YYe5Q2RnYIsX+NWs9hmbWX 1v/4pxNig8OGofAAQLMHoafu5CYoJ5f4fZd5IlpZ0nxPV4T7RR03ObAmqTDc1tM8+vjB ZNVSC+tSMZPAys+iq8ZUmdFy7/g7w78A2ExGiDwPOnH4nkcbSG9J97vAbp3XJSkosq89 8nSUWruPsvz6RlWOU5PzWzCx/AA7u3l+p3jTy/wG1EBYG1xfq5G3lXgOW2ZsIpMpd/AF 8dUg== X-Gm-Message-State: AC+VfDwg086/GxL2lyH3nUcLdQZcpsGcNroWSRtLSnrn8QyEEwyc+UUO ywPAzPllGrwNQhJtGSlN7vWYon23OJ4= X-Google-Smtp-Source: ACHHUZ54ReoLXFk1oTaYjmLVtkkNt2jRchxoRiCx8rxY23YjrphJueyeWWbM6qC2nW3CU7hKMZM9JkpwZiA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a05:690c:723:b0:54f:68a1:b406 with SMTP id bt3-20020a05690c072300b0054f68a1b406mr8285886ywb.2.1682960114403; Mon, 01 May 2023 09:55:14 -0700 (PDT) Date: Mon, 1 May 2023 09:54:13 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-4-surenb@google.com> Subject: [PATCH 03/40] fs: Convert alloc_inode_sb() to a macro From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Alexander Viro X-Stat-Signature: gyafegnp998s1ap7io1rees4nod1f1o9 X-Rspam-User: X-Rspamd-Queue-Id: 6F189100023 X-Rspamd-Server: rspam06 X-HE-Tag: 1682960115-700551 X-HE-Meta: U2FsdGVkX1/8+DlvN3OXli0w2oqiBH2DElh6Vc/XusPy0vCNTsuQr4wzd/VLi2rbZnYDwZ1tFOdxXkJS3bhkrf4Odh2Wt6SzC1N4FQUip8xVEfy5c+gU7Oluqi7LUQh/MNB9zG4osNjQ84rq8Wo8ohRi4z9qHtvONTPI1vdPXDlqpHVIVjc9/INXHLcRs6RsX1fn4bZyV1lP2SfEn1tLvUnpyxTg7UUcfXLKFwitxX+OJjMn4sC2IafhAHjLmFfwVf/2eA1wvRtbnHBgwBd7J8wdADvtoGX+FUvhMUZiZ/OXHy6pmstRzZ1CLhUAi8HAKR4H2vZg+x0CQsIqEEBu1bIDsxdyCd0Dz8FjccWV0c+JOGDxa9Oe4YgURLmO6CpBEmwY/4HWiMaq/zw7VYOA/skTDDFAZ/c637KNRO2SwL3RifqeMfPKm+O/M6orEf0q0cZj8aYkUqo1Ym4QbNpURuK1ssVkg4xVEzWAalNsG6QkOd96fl0jgBK/XNkD7SVm62xbiZr4c0tRU0wpyyVwe3Ep8oKVK1wd3NckVfXH7ZpS42Ri0Uy+umP7kuZv6UsNun5pRBHYzXaoYPSMfc7yjXDbTw3ph5MCyJ4OYUBkRqmK+1N0eO/Mz/Vp1HHf7jq3Zx6gQSSjWhz3Kf2LS+9M6x9jgwHkXw+3GYt2Fc/9ZetVw6b0lvTwx4ONAGgn2+MY3fMLtko1UwH592BnM1uLZvfW4wKXRTjIcNipu92EazNoeX+1BAM5Cvi143V3G8bXoWvpAvRr0CxxksdLJ6sZkRrBKFGA+HTu/w1YDFIurxEixhhDJXDTZ93G4q+u1dbJNBBnKJgxnG4TU3vQq7u2ARl4ekTA30dlWT0r2FVFf88HunVSzEdy1okv12WIypVo6DMzrjuxfjEpVSQLlYW8Nsf0k5hSXGM7SEIq5KbB0gWv07Wi3i/8XNiS2xBGBp8JTIbkwUCJ9uQl1w9Vj4K foU6r3H2 oH1GPidii3PDIHZAlCfTrmdW3E981qD1DXGjP849KoBRkgc/8E6CntnlXrgaqzVuRNSA5dyxeysrZ9W9GSwmnXhu7UE/QAVxTjARg0GaTJjRXBkxBwgjuf9T7lchc1W8tNbakDy2rzi3H336jugoTN93Aou19ET6NGMAuwwJ7Rraz+8usdRftikmrRQcKLpT+uRyGtRfNHoaB3Tn8y2W33cumcobQIdmP486GHVJM1D8dOxEtYKkbcxgUh9fZr1/7tBc/t5Zq5lUmNxIbUDYr78jfeu4eDjHmWFWIEtAfoa1eN0hK6zCrxwM4udXoEwUtwx2E9oyoE78hy/uhySeUf8g/+PG3DlKkbsQ2jpFrt0fSvIww3qvaA4moIHyMXV3Gve6OyWfMR99ytJI9XcdpsLULxDW1kyBI9x2bvBwapsinjnVjEWlnmxgy9bmw9Rm7mi02FvizbTjvsdSextiyaKUye0cunIddb+azEzaH6WXBoO13USMfJSBKKozR40ipn5pZo+99LaBm07QDby3keZ7/DpXVglKDRNlXC7VodczW/hZXQLjMkyr9Ad2STSm92XVOnsJhxcozM6JBu+HGejJbu70gMwM9NQcYTcGSxXJy8t64Eijiv3pXB9Xx54L5yZCFglWRyE4GhgdwHKEFHqIO3d9B6NHeaDVlvr+AuaGUifHHTKRd4kEMZq06JEN7BM1nT9MNf+z3v/H1QlO7aFb4Z0f2TRxq88o5XZ2AZC8FPCnhxtGAHH3LRTWklHBncBda6V4LzOIuGiwwOK1sxVGmRVuua1k/7sax X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet We're introducing alloc tagging, which tracks memory allocations by callsite. Converting alloc_inode_sb() to a macro means allocations will be tracked by its caller, which is a bit more useful. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Alexander Viro --- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 21a981680856..4905ce14db0b 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2699,11 +2699,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap, * This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb(struct super_block *sb, struct kmem_cache *cache, gfp_t gfp) -{ - return kmem_cache_alloc_lru(cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb(_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp) extern void __insert_inode_hash(struct inode *, unsigned long hashval); static inline void insert_inode_hash(struct inode *inode) From patchwork Mon May 1 16:54:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B59A7C77B73 for ; Mon, 1 May 2023 16:55:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 46EFF900007; Mon, 1 May 2023 12:55:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E3C3900002; Mon, 1 May 2023 12:55:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 183A9900007; Mon, 1 May 2023 12:55:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0A5D6900002 for ; Mon, 1 May 2023 12:55:21 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CB842C072A for ; Mon, 1 May 2023 16:55:20 +0000 (UTC) X-FDA: 80742286800.12.C508C81 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf21.hostedemail.com (Postfix) with ESMTP id EC80F1C0013 for ; Mon, 1 May 2023 16:55:17 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=yKDBPh22; spf=pass (imf21.hostedemail.com: domain of 39O5PZAYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=39O5PZAYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960118; a=rsa-sha256; cv=none; b=zohjYQ91fONI73N/wkQtawRAZA9Lq36Azd4f2Jw/iIHq6nC9R6y4oFIuq6vH9mCsqw68IB GKVhOzqVxyYS8UMoeehc0v5Xj/AhNofUDzZDdbF1TSIHS/GlanurIXN2UKyU+k9MqdfqwA CIM8boujBiesKOBB55kUHAclsiNdgbQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=yKDBPh22; spf=pass (imf21.hostedemail.com: domain of 39O5PZAYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=39O5PZAYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960118; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MvglFAc23/TKNtr0bbBz1MPaCPYCHObzjBdGSdfY+dE=; b=kYv7YnHNd8XXqodpqWSBrzpf9nCSgz9TeKqIvzXOmhUt0TX8kMBfrrii+SotZjSej9GsfD FKI4cuYEIqIDOCAQ4YDMCShSNWPrEbbF/wRSRYTX6+lTfZx6ioRR5wpJm71yZZXHek7oDP 6ZpOSm3i8+83ggaRN4pICulF3Ij/MAg= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b8f324b3ef8so3224231276.0 for ; Mon, 01 May 2023 09:55:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960117; x=1685552117; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MvglFAc23/TKNtr0bbBz1MPaCPYCHObzjBdGSdfY+dE=; b=yKDBPh22/4OVp0gpJ5U5nHWMn8h04FMW1WN9f98nVkyHZSwAOs9UEKf2haMvXBbR8k I5P7Dha4Pldrj64han5IvUTFfOCUHNGUrIJIAXAHFHlzCILjwJdK3NkqJaf+mFjOWY9Y anqwu1P80ScHesP8cB9KZprrz91t4RxUXEXNi1EU3gVLZoQD/8xloHC2gL9j3TfYF6sZ YkdPXTXoXUqQoMaiXzOvgnurLH12U1odpmtI/6y+T+w0FuxiB/jPfmQMFC3YfiI9xt9g 5K5Lvs6m9I2be6QJ0QLu+uaPSHjNUlPR4Y/We8WulcHXa/trNfBWstA095DBG1XNwXOI RnBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960117; x=1685552117; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MvglFAc23/TKNtr0bbBz1MPaCPYCHObzjBdGSdfY+dE=; b=kBVOWIXzDwIaDZLrbP7E16esNl+RoKgy8J6nvIcp0EKb6NicbMAY2TOxo3851xbnrm zSCQDGv40RL3tTlu6Oy9Kw+D66BFhILvtZCv3CVw2xEDQ8OEbK/MIzRYC4KWOFw42d+k iYH6Vs4Ng+LSdBAKJiVSnyz+HFo/f2ISl22kZIDf+Kn2mGH6jHHuYZPvEOwILqQAj2U2 JA0SnzcZtKiqZCHEJTr9P6dQsgBu8wJGO/Vhz+Z2AlrqAREz6aSgKJPOLtjOOd1Trj9U RgeZpfYrj9eRtD2b9kfAQPynpjEat3ctHFS5g8U2kulvcpUt1FcK0FKvpQ+H8yezCMD4 lvEg== X-Gm-Message-State: AC+VfDzQCvh2byF15yx8KRJojMxDhN69KgMj2hD79TtOXvAYtboRz6qd 6TRiDbuYIPbXV/PUXLRfl2smyghODcA= X-Google-Smtp-Source: ACHHUZ7zxOn1YY1d+8Zdgat4BhX3SxSXxSfdgQr88v03ITFlUy5XmZz5otkrL0/chwvD7sDfcOf3ssnE7Vo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:c54f:0:b0:b8e:fbcb:d6ef with SMTP id v76-20020a25c54f000000b00b8efbcbd6efmr8806787ybe.4.1682960116894; Mon, 01 May 2023 09:55:16 -0700 (PDT) Date: Mon, 1 May 2023 09:54:14 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-5-surenb@google.com> Subject: [PATCH 04/40] nodemask: Split out include/linux/nodemask_types.h From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: EC80F1C0013 X-Rspamd-Server: rspam01 X-Stat-Signature: cbxi1tquij4r7esq8y6qgxeyczgikn71 X-HE-Tag: 1682960117-32300 X-HE-Meta: U2FsdGVkX193RO7XgRfwcDInijxC3nUi20OEFdPj6I4x/4nePfxVB16nQPLruP5qiic19aagXlnAC/PY4h7M2KgDxcExZNwnpSxTp+PkWLTh+KYEgn12U24pcnciB1VJp91JF0i9O04IOmcciYqJ0WXfUWWekiR3uaXJTR/t+/4BeXptnA47mTkDwSNYoi3qShpHqdn0pF6G5pwHACVfr0pRoqTYqjilsfhBSM6iGTDG8C3JDb0gvK/h9w+GGgnqOJ/lHP+9xXo+MpaIeczIYVPyya7/fQqjQNWArro/kMcN9yrooCaCGmVELw3jknGfay2vAzkorUkJzjfeq+DyC+40rS/P8DrV6gjgOUVokl86P3WICFtg1fMl7T9Q5gC3V/Ub/W0yoI63BZqukjbvGZ8nw7TgcNtZZs3YlaVEIbJBW7Wu365GhoMWyaAeM4SqTNzwtK2HWT6pWaQDMfHfRvau1gnvKXHRTjGEo36s6mv+gxiJWQaXhwiCMBfWp2ZHsYcEJF48Y3gz7BOBbTQ673/Okd5KRIUH8rmRj1kcOV+ueQF+vZYNkrwyJPXMLESUWTddnzJsiW3E4ZCZXehPPfurSkJD+Cnrxh6qmZS0IPmniOfCfEMKAj+9UjkQHUhfau6gw06IM7LnwpHTvyrSGsuR59n+HSpMqVi6bQMeL2vqnHBBDliJidUnicD1WfOWpLQ1O5qxL+ijm8zieZf1uSxeGIksOKOpEqvZWfW+CBVxOJjWxRbZoRHTsatFvYafYeXA2ezzjGiSWCVwvzDpbq878I7hv2X5lwrURsAc8lIhYOkAgB6MNifBZ2SfzhOzy0CZJRfZ7eyc8B+bNSqUFnQMR+eZ0INiHbG8LfVinAZTKo7AzjlebyogKAbvPuDxegqGUjxYc2247esA5cLk1W5U56QXMf8MNTx+x8uLiGW2H0+iVfQiHb4tH0pUz8qgUQ9voVd9HdJsvVvgNds 7LXt7kMg nSeaozgeAGi1kH9R7UYtYwfMtZa5//93R8itbXG+C0hzYozr4pDW4fhTVO1Rk/Dvddl7aVe7Tz9maYr/+ZR0qmLroB20uthfft8jMjgmmQkzkZ/xpLWp5cl5jRqv6+ioaxY29qlSGZ5hirJrKbMMVZ86bWoRcDY6uMveOFD/jqr52RuvBSa5MfwPyZGhHHbno9BGZ7GY4w76SBAnzshDHDwJPlJ4C1AuLM29XIMkZPww1763aMGofz4zsJA1WnbMQEuKYhPDllY5cWBmsQueoElP3MFSGJGSwAntMhs51ffPVyYcw02R6B/eyaHdQICE5Vn1gl6pBpendZWkB859DxS6LfJiJbBsLfihebaqDZFuz6YO1a6j8rBbvy8/uZIT4/DpNXJsJkNIfnnywPNdAZd0pbEgq90Smz5y/mRO4nqAGRN8+I0vSewfhulLYLXRDXTC4IzcuO6+JM0FAjGA9knl/kbp8pJdqHwXXMiXQtLQWkbxL1yL1yCh7U1gUVB+Nvx1dkNHyDaGSBVLwleuJd0RirUq+o9iEf1oxM7qhsJnCGSzaERwrwRnBUCUqoPCrsbP68T81BFauvDCB/8LT0FS7AD+YQ0snjmYhhxNyDxhxL99oreS2izF8jioIPKa6DUcq1QYPzMTFRb6W/d8gPd+95qvlG6cZafzRDEPwkt1FycUvgOMcIBYMnlH3rifSBvK6NRNltTiQCigI5MXd2YmsUBFpV0HtQRPWD5quggVKvdJy+tyuFvJN8+nbJb/jZMX5PGMBOKd4/kGrGSJBkNAIYS9dxtAH8wA23rZZn1pe5yQtCnIqQCOuRA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet sched.h, which defines task_struct, needs nodemask_t - but sched.h is a frequently used header and ideally shouldn't be pulling in any more code that it needs to. This splits out nodemask_types.h which has the definition sched.h needs, which will avoid a circular header dependency in the alloc tagging patch series, and as a bonus should speed up kernel build times. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Ingo Molnar Cc: Peter Zijlstra --- include/linux/nodemask.h | 2 +- include/linux/nodemask_types.h | 9 +++++++++ include/linux/sched.h | 2 +- 3 files changed, 11 insertions(+), 2 deletions(-) create mode 100644 include/linux/nodemask_types.h diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h index bb0ee80526b2..fda37b6df274 100644 --- a/include/linux/nodemask.h +++ b/include/linux/nodemask.h @@ -93,10 +93,10 @@ #include #include #include +#include #include #include -typedef struct { DECLARE_BITMAP(bits, MAX_NUMNODES); } nodemask_t; extern nodemask_t _unused_nodemask_arg_; /** diff --git a/include/linux/nodemask_types.h b/include/linux/nodemask_types.h new file mode 100644 index 000000000000..84c2f47c4237 --- /dev/null +++ b/include/linux/nodemask_types.h @@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __LINUX_NODEMASK_TYPES_H +#define __LINUX_NODEMASK_TYPES_H + +#include + +typedef struct { DECLARE_BITMAP(bits, MAX_NUMNODES); } nodemask_t; + +#endif /* __LINUX_NODEMASK_TYPES_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index eed5d65b8d1f..35e7efdea2d9 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include From patchwork Mon May 1 16:54:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 459D1C77B61 for ; Mon, 1 May 2023 16:55:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBF99900008; Mon, 1 May 2023 12:55:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD15B900002; Mon, 1 May 2023 12:55:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97290900008; Mon, 1 May 2023 12:55:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 87CCD900002 for ; Mon, 1 May 2023 12:55:22 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 56521160805 for ; Mon, 1 May 2023 16:55:22 +0000 (UTC) X-FDA: 80742286884.24.66285BD Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf07.hostedemail.com (Postfix) with ESMTP id 7B81140019 for ; Mon, 1 May 2023 16:55:20 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=WPj8rory; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 39-5PZAYKCD4xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=39-5PZAYKCD4xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960120; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Am8x4T/C3ER6nwTMzq8wqC7nUn5OpA8nmjjdL2Y+IyA=; b=sUfJwUW4vd0Z9VDKDbv27pdMctJ/8IXpXTH6TZcc10DUiklDeN+FtA6f4FPOGPFuvvOcZX NQWM6BmUFmOGEoif69SGf6ztgINMHX2I8HpQ2jM8p6rq2oGuS7ERTADyVYuN2VSoXwx9rc lO8XXQVw2HjI0FyukOOihOyxUIVevkw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=WPj8rory; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 39-5PZAYKCD4xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=39-5PZAYKCD4xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960120; a=rsa-sha256; cv=none; b=SitEEhEhrQ8IJhCppjnyB3FPZbHZw/kbV2dn0JPaoUv9n7be0j0KoRyuJVheUwA4xO52o1 ulRvR2IP6b1pV1rRuvek2M8SOZ6BJbqT6ZCjBTFQoyJ2XGPlCjxlDXJUjLjtxgYnIsPT7j +2b4aOXKCenhl+fKbibFg2NNX8R5vIc= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2473661358eso2716345a91.1 for ; Mon, 01 May 2023 09:55:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960119; x=1685552119; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Am8x4T/C3ER6nwTMzq8wqC7nUn5OpA8nmjjdL2Y+IyA=; b=WPj8roryRwAmmVOt/f6DiF72YfWfvTwfrhdft0wjazxPvIpRZjMIBwQEDTYgaFjL40 EYH/Obl5jlfmohNQjk6ksHyCvogtUr2FLXSqo8imf9pH+Av5pbwlM01Z0VPnzvuHFHvS sJJApW1h4nytu2oRGIJAzX1ILNXit4lP55uwJk+oNrWKlXyEZFC9xWXVli4TIY3SK85U B99AfIcgd6WX1rUTcNtVjahhym0hUSbpRJ8MlMIviR2dEnrK9LO4e6TK4pMF6ElbNOxK BqCNggeEEuUJjuE4WJ/nJF98lgK/t0MR2oB5re4JYDKTxZxL1IcsVMbdtZa5WL37TB8E NhaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960119; x=1685552119; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Am8x4T/C3ER6nwTMzq8wqC7nUn5OpA8nmjjdL2Y+IyA=; b=EM6lWWrPjJxix5Q4X4lTF6dD7KLpC99iiJBT2wlwCi1nM6AoNFCavgrD2baQ1fVY7g AIKS2RDTrpbrdK01XWSzcly8m75OK37neFziLSN+8ohYMIdAi6on9onuX5segDTmR2Jx gsgUbyhz2IJIQ+X/HURGedkL009N3CzhHBAlh2aQEl2b80hLN7pQtFIUImY/JFdioG65 YBgIP0WkPkIvcXvyVT3gNljZUbJCBBvfZPRL+IOVoBr1RIzdgGdcnKKBZ0rIbT8jyEJ3 3pBR2S57MyIHQLFHvY2c8X4EixZGh142hOvJG0CtKfY0woWXOuOk6mlneMfOvHa4rJ98 3ekw== X-Gm-Message-State: AC+VfDxmHgi81KX98bQazlQkzgY6LQu/aku8CdlOq7HOyk4spaVlrlYd 12QFFHrmuxwwl8YXrH4BrwdEXRbT+mc= X-Google-Smtp-Source: ACHHUZ4HYpYlqwon9ul71ywYVvGlfL5GDzgkEgI+UaCs1iOfSF39OpEKQuit70zO4oA3hBY7TJW8YvGVQSU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a17:90a:24a:b0:24d:e504:69ed with SMTP id t10-20020a17090a024a00b0024de50469edmr1685228pje.3.1682960119293; Mon, 01 May 2023 09:55:19 -0700 (PDT) Date: Mon, 1 May 2023 09:54:15 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-6-surenb@google.com> Subject: [PATCH 05/40] prandom: Remove unused include From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 7B81140019 X-Stat-Signature: jq8i4n4wsx3ncwqpex3fqh7b83eghqp3 X-HE-Tag: 1682960120-88036 X-HE-Meta: U2FsdGVkX19Mcpcd4e/IgMOpgsKSihPjMwLV107TJ5RdCZCzZW+uO7/fQTX7uQMgh8kxMe/t905ojpmbTOiH571tGWkqipDH5A85Q8hsP4jqmFey32yOIaLgxvuUDUBXUDPZ9T01RKkM5LCrNlvy/MPvyzH9DpmVNAxPaMXqXQgQzibICvHQIyh/ZrmhNQvUj9wnbgo4bLLXCsGAyd7hEqo8ICQHQ05z9ixshs+YbpihGDlB9S3j/bn0YNRudUS9u/M9LDKzjjCPHBgSqL9x7Aio2BxzX00Y5xqUR8q1j58bL7yzKDlAi/+AKBbynGkzpYLYKXCy5YugebsIXOPfpHu0Z4dyDiokS6gj3we9KQ8sGv5vnnvQgPIKTDJiLT4XKbxpvpAbgbUjnFtRVQUGjO8/CpE4Ol6eX8fRSeHaZp2QSD+c5AkrrA9jfVcdg5lk6ADAum0lrgB959NEwRycBuwBcZ3DUZoAzEFvo37qup7VzHZ5u9bxQsdP1eQht+NPNGFnB5ka3+Ujbde44D7pen0aKfJKc1nr4t7mpSJFtjl+OdOFILOslQHJ+DHQiVXGg98ZXHN4VlBFM//gDnqnkd8hK+rbWuEfESw1kDoG0rsIEIcXa0dnaOphMg1+LIRqjv5dMvPO9N2ZLOKQABuITnDqQcnmAE4N1UTuAAY6m28dclB1stt6qRZYczK6y0x2k8JcOlkvlAilR4zoq/uxDoLqIlie5uNOAIWvcbuZr0rkTum2OKkHOqzbUEwadDDFfsvSZ8mheTGrjRQXJvVzcop0yR3zGJiLQ9mGsjF5MBHoJdMJ5ecuMp4BWaQ+kl9Nq8AdPt4I1olpAUYLP4l/au/8RvVNMIyURZKBWs4zFlDPA42y8Gooqqf2y8QzUBZeOfcEl+zXRMtpBoCA9eptMIGRzUBuYJWavKWcghl0EvS119StQSzp0lPIX5521lGiI1VG0swWSxaJ8OljHuG oEQI3I33 mCL6h9Y7pVNcLuP5OwkzmJHGctwiHMj/bJegE0FoIfbHnicefnkDy4KvhMH7gdOhNmz8zi9RQYR0tU/Vm9Sr2x27vsgf+HsVY1Vg1hNOclHPpihNpBAA02rYWIi7zY7o3cDg38xNftjsd0HpdFhvUNm8DELd7jq+Bl2ScFKd8aPnkoM/m8bm1sGISGDkJ8WnA4Bcd8szoS+5qIOybQ0HTaDiqSebwOj1XebkH0w7On31UCvGHdCuJa0ObN5MZYrdTU4quO04tkbzAtAF6dF7bRj40jY59HyX1ZPX4Sgbca66uy4vWURMiWHjcCMV/1j2e05rCr9sfmfHLN+yM5DhQQfOGPV/7S8yUqq0nMs/Yer7YCCMxIGlaOrJyw5+kJq5OtDVR1YVjb6bp28ayiBO4QVeDyG36JHhqwhGBzBKJt+yeMc5C8W+gymx2Jpg/QHCI4Nhg0PgrGZrDlS3lz8kocFvzh5Lq5I4o7T6ktoWDIFhQH02eDr812RIEcH8kZQk4KuqojgFg9DJnGd+HM4V2Y07CiSxqx5Rtiae4LtcUwP6MZC/+L5T3i3hJIPyLaTw2defgz1bjSvwGFutAau9m41tKYYx8B6wCNMWLGi0f9p96WkZMpenj0I1XiQ54zzf258B4C185xg3ytQHJT9K7hiOwjz/PLJzTkJb2EGYUC3MFl29/VO0kMutUePzzMz/114xaT4vsRm9Jlft9a9AMxJZL3A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet prandom.h doesn't use percpu.h - this fixes some circular header issues. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/prandom.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/prandom.h b/include/linux/prandom.h index f2ed5b72b3d6..f7f1e5251c67 100644 --- a/include/linux/prandom.h +++ b/include/linux/prandom.h @@ -10,7 +10,6 @@ #include #include -#include #include struct rnd_state { From patchwork Mon May 1 16:54:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67E35C77B73 for ; Mon, 1 May 2023 16:55:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E581F900009; Mon, 1 May 2023 12:55:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D910F900002; Mon, 1 May 2023 12:55:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0A16900009; Mon, 1 May 2023 12:55:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B1C47900002 for ; Mon, 1 May 2023 12:55:24 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 78DFBC0779 for ; Mon, 1 May 2023 16:55:24 +0000 (UTC) X-FDA: 80742286968.03.E402672 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf01.hostedemail.com (Postfix) with ESMTP id AEE9440011 for ; Mon, 1 May 2023 16:55:22 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FUEHKTcj; spf=pass (imf01.hostedemail.com: domain of 3-e5PZAYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3-e5PZAYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960122; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AsvePdIIpK1hSWiMR4JNg9vJpL7RbQEpCCaVPwkxl+U=; b=zpANovOSueHiobpIADb/65Q0z9C5PpZQDeoF7vekXykRAxR5TB/vYUrQuCYA+w99Z5KWOg 9CQpknLQ4gSoz4vu7+lxZNmug/bLcnjhpyAI04Wz5/vk+eAUxshH8SK9lvIVUcfZ3+1VM4 eWyh8uRwoyhc5kM2iRfkvIWkn6oruEY= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FUEHKTcj; spf=pass (imf01.hostedemail.com: domain of 3-e5PZAYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3-e5PZAYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960122; a=rsa-sha256; cv=none; b=FTLzILCaqHlPRshkCuXCPB5qtiw5OzSWfu5+JBSpNSSP06Q4eAUyYMXTIS84JmrlVRyYXS ngTo09qHTkPW/P57H4g17Jfr0eFSDh1QRJH3+0GbE0YKQ8TM/nVCKg8vc4oRvcNng4D/ST ZxxgukwwVrfekHzKQ0y0HshR7CprNS0= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-559fb850e08so31595647b3.3 for ; Mon, 01 May 2023 09:55:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960122; x=1685552122; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AsvePdIIpK1hSWiMR4JNg9vJpL7RbQEpCCaVPwkxl+U=; b=FUEHKTcj/6Oaa/jf/hG74LwWnPzMzDXETDPC/OZPn9YmdSr3Je1dLjqI5qQFH1sW3q Bq/alOSo7JqKEqD9rgKcyUF43rVJZUIyQmx1G3PllnLeg5Yg39a7Bnek6w/Qmd8uXbJK EsGaYT+egztrqXkT7lGAkap+Gl5GwKP4LRCOjo5ZnVALaZHiuOt5sB/2YrOa8QMVBMrC FKm9pwPXa7lut/5XYzJLVlhPqGzsdKvhEJFlIIbig8wN8WeEjVJtPWgEipfIbTn79FXC wbTQc/DXAGa8hfnAWPDItUw4SLFFckirBkUuGFld0u0/pGt++nX/qQJEblW72Hij4gvi rVdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960122; x=1685552122; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AsvePdIIpK1hSWiMR4JNg9vJpL7RbQEpCCaVPwkxl+U=; b=dz6bWr4wtNanyabah/1EbM4Y45c714RptTzjLCvOJa022/RQgi1/OVw5geitCLAIqS bYjhCSbTNbIFpumUaAjstuLqPUwkdRBtiWEeEcjOnvTcg5bXqNZdB2gDPCgsiA65/Nax 12r+KIA6BgEbN4rtNJ5vm4LYiB7e2ai+6cRqLcZafhErXX/z9QuAX2gAMaIAHsb2DMn+ eurJQ3MqsPeKBtU66UFLCkmaaAnfE7xn10T6wf7BrGm0RR+1TNlF41RLgFWs2Y02QMW5 joUDcFFoI+E/Gje6JxaZWJ+/DqUfF5ZWEEVQuGunkC/A5rMp07yedq6+09pNSMkl14t+ pSyg== X-Gm-Message-State: AC+VfDz8NgdKFdj7j8P24lcT5toyxJghY+7ZdE4lhONwHZy6Qxg4WLOS pbwaxzifKElTk4/BN6KyXm8G7DB9/zU= X-Google-Smtp-Source: ACHHUZ7mSCpHViXCMYRNn7XHcAK4o9x0dQPzloUUcJZHS6W4xj2tbrDIYHMzSCEPzYlJGL4NreMIfcOP/Ik= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:7653:0:b0:54f:a60c:12eb with SMTP id j19-20020a817653000000b0054fa60c12ebmr8139444ywk.1.1682960121808; Mon, 01 May 2023 09:55:21 -0700 (PDT) Date: Mon, 1 May 2023 09:54:16 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-7-surenb@google.com> Subject: [PATCH 06/40] lib/string.c: strsep_no_empty() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AEE9440011 X-Stat-Signature: paw4m1ja3kw4dxnhbihfpgjns1tdgqs7 X-Rspam-User: X-HE-Tag: 1682960122-873360 X-HE-Meta: U2FsdGVkX1/wY2Ue+E3xh44c2NPr+62xNV3tE0FUp5GQZady8Vws9G8ZQfwflMOjW/l4VmfbnuNnlLO5LXQ1CR58ZDjWguvk5UA51T82qQWDKOYDTXK0ixUZ2NK5GvPv7vZiwslRet94ttTIm3WVAGpHYdglCoG24fLtRZvuhHJEuk8UIiYmhED8/X+WjVnO9jqnQ86orNysxIzg2HkRu71AOFkOZPiYUrn/kcz4SFZetf44LAld6EdTgj0buW5CPcl4glYglVVX2thbfoZoDn/5sGveLUVOD9E4S2YL1rsdjUmXahxCdDWcVK42hYfgYY1dT+YaiKT1UmBCzebFcinugKdEOYGkioQY7S0PYM4u4EYskeJCF55y/MURh04XpgafhdnMtC+0r9FZv7Zbiuc0EbjbsSAWITqjWqy06sc2lLCwX1v6p7XVbFAA/X7TZAcuGmq904esKqpbW6DePKztJ15zn1Oi5z2Vi8oqDya4xwejYLtPV/b3CxdYkvUBGLCwFJbyMF88PU1g3Vahq1WowWQY6+Dp7t+mkzozOK1shRVel8dHHc3afIcrwllTqLB01lbnw0CWqqyjF0fxjWaEhLoHrj+slx5lCE90zUJ2oWeJokJeRxHhCIb5yH7mM+QrhZSs5ZimxRA9SC0tL+Lzgng5M3maxpW2L283YFZOp/+2rOn0uNCk4WE0bfFblF5HDNO4MVRjyRpexmG7oolLKrol+qCFEDPov4AE47veQQC3I3vExeUskdpF2WAkrq3RBh5VWZfnttu2HqvJnp+kRlIF2aZ2H9nvHVqMabn+pwBa1ZF69yZ7ANGdmu/1hBmYdwhyQQptgI5XD3tLUmjVY9etemyylqY0I5bPg7UjK3avBTV58/Crcs+mqzvG6f27G5SVroXc6zcfmbyeM1hGgsVb+Vl7wBDogaGfvugiqw0fkICMWlldyQisWB2DOUOCdFgC+NaAnqbR2xX mzZilMGA Om8K+54JRxIXG3KhpU6ItxJuW7M+fq/flY4KGO1EYL6BuogzGDWLXN4t2NtE4sc1wE1VDmFP9EEh3qP9GAu6ZMbUSjVHkoDDTGGGHGuk/acuZjbwNfLpGUuUcPsX9Zx5q6eOBHmg5D3FX+HKUxU4UVqSNTC2KM0sfK/YbW1hQ00jupUnOZsljYCScYc3xgMhe0GLznRceMSpQ+oJWzBdOz+UDo2JgmgiQ3A0L53pdSshPrPp3IAHwi/pyoPYNellQuhvfc1w4oE/bCuRIqUhZy3Q3qzUQZj6z7VTH9f0fqMYevpX3A+vV1xANzaNerhKwBzq+r+ya9OCSrOsAcQXbi+H8Xq7CffT3Hh7uu2j7k5F+ZPWDpVG21W/GqFqIUMTTyJWg9zLJhlNBxIzaamc/qC0lrZ9w3ENNdJt3gHWNWvEr0iDKnD2dAn2A+fQ7ER2aQuJibvaOg4eC7pGx9OzdFv1JJyjqWQb7NPwpIjVq98jw3RCz+wePQVt7eqAXeYUvZdzOMPUGG30hoPFwODTsrUygHthxWDjwjPxrE8y36ThkBFTNtSTivcMo/u4gdW/fp2uLNytZNzi1hsoDTLEQYDPG4csYV8QmPsokqo7mEVolrBw9Ccwc3yPtyHLmatJbq0mTtkzBx6BukGMPzpnj0S9i62k9fxL94YhfTcrrmDUJMrCw1+aYMF6KUHFSmvgZC5g/umNFoO5Gx3p0Qws/tLExcjvOV3+5BL2q X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet This adds a new helper which is like strsep, except that it skips empty tokens. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/string.h | 1 + lib/string.c | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/include/linux/string.h b/include/linux/string.h index c062c581a98b..6cd5451c262c 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -96,6 +96,7 @@ extern char * strpbrk(const char *,const char *); #ifndef __HAVE_ARCH_STRSEP extern char * strsep(char **,const char *); #endif +extern char *strsep_no_empty(char **, const char *); #ifndef __HAVE_ARCH_STRSPN extern __kernel_size_t strspn(const char *,const char *); #endif diff --git a/lib/string.c b/lib/string.c index 3d55ef890106..dd4914baf45a 100644 --- a/lib/string.c +++ b/lib/string.c @@ -520,6 +520,25 @@ char *strsep(char **s, const char *ct) EXPORT_SYMBOL(strsep); #endif +/** + * strsep_no_empt - Split a string into tokens, but don't return empty tokens + * @s: The string to be searched + * @ct: The characters to search for + * + * strsep() updates @s to point after the token, ready for the next call. + */ +char *strsep_no_empty(char **s, const char *ct) +{ + char *ret; + + do { + ret = strsep(s, ct); + } while (ret && !*ret); + + return ret; +} +EXPORT_SYMBOL_GPL(strsep_no_empty); + #ifndef __HAVE_ARCH_MEMSET /** * memset - Fill a region of memory with the given value From patchwork Mon May 1 16:54:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F1A5C77B61 for ; Mon, 1 May 2023 16:55:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E3A290000A; Mon, 1 May 2023 12:55:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26E25900002; Mon, 1 May 2023 12:55:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E70B90000A; Mon, 1 May 2023 12:55:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F259B900002 for ; Mon, 1 May 2023 12:55:26 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BCE001C725E for ; Mon, 1 May 2023 16:55:26 +0000 (UTC) X-FDA: 80742287052.13.5376EB1 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf23.hostedemail.com (Postfix) with ESMTP id E48C6140015 for ; Mon, 1 May 2023 16:55:24 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=uyQWn2Ht; spf=pass (imf23.hostedemail.com: domain of 3_O5PZAYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_O5PZAYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960124; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O9K7OqGn67XevhyLqxW8/t3YPJ+QYtYh9lmzl+bkJrg=; b=B9/C43pBOCMfGX9AK6MzPbPYMN+j8p9icMKU204nC0D8f6X4QlejooecjddRJh6KX1vmdP z9+ll1zIchUX1xtGrejUwEeFrHGJts3gRGiVgTlOTDCZirxqMz1G31p58C+QL2oEjAaCHs orMxnTov+pGuWVl7b9LOm7bsCbyZLK4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=uyQWn2Ht; spf=pass (imf23.hostedemail.com: domain of 3_O5PZAYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_O5PZAYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960124; a=rsa-sha256; cv=none; b=kr1RCeFlDHtIlrHkBkUL1YWjJNfB3iUeMU88Vk98LAnr97N5yyzhdN4VK6DsEjWPm6dacO oxgtupSHzga1cWjfYpTtbq+TzSIw9+BRBcsXRBfYNkZaluR5mYRdwwhsVlXbuRcV1Bx3fe 5p/JE1TV1dYVrg1gikARwApLNEpzjio= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a77926afbso5335753276.3 for ; Mon, 01 May 2023 09:55:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960124; x=1685552124; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=O9K7OqGn67XevhyLqxW8/t3YPJ+QYtYh9lmzl+bkJrg=; b=uyQWn2HtrGHHQPYjCaEXS71WFvIKKSc7//EwH6w5zT7RuLTJsJvg4Siy51jlaOTMsf DmzxgGzQqK6buMAxRdIZF5co25tsVldyDbaF7rOabJEbLpi4XheA+l5VWP2IJDBlEIui J3jFl+rVymLiKhaaXfPqFSO8pBqa31LbX5CWrS5Wner2pA7Sqr1hkvfcI6t4MxiQZt9K qhkEpMVbiHPNZp9IQ6O2D9+a8gKMVQ/1Gtf9GajnR6izUu8XnftHHBs+v4IcA0SDdYNk NO/g09kaGZENLPill52iLrjrZMs4uUM92aRcipfW+xL+/5JdYb7kKitjTzDH4s2JKlli slTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960124; x=1685552124; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=O9K7OqGn67XevhyLqxW8/t3YPJ+QYtYh9lmzl+bkJrg=; b=lNKiWi9FdDZy7/LclSx5Ntuk9Bggbq8HUwL0XU4b8atvOEJ5ahlsniIxcYjDzBKxDx MUqNSoaz/Mfw0r9dAV+9qiGS1ClHkfiz+dEhO06ElG37sj8FndpNDQ3KExnjen0p3bCm 9tuxcRuLUbHiFs6pYbAOeirAmCGRSfqybVsMvqVqibtw/MzvnT6fW5jypTIVUscVBSbf MzhxyWxJo7EI4oZmp8k8zFH54diRh+7aZsu6Qfh0Yx9X3mLcMasw9KEjCHE2z0ZOv1Qi gzF7Z2m/drt/LUmrjlt3Zpd8m4q5kP1x7Sowjbs2I9kpPQbT1Y6XPSD7G1Bmp14nEAux 98uQ== X-Gm-Message-State: AC+VfDzQRTNJLdJB/oStO2WOvJRxXDwXqY/0I7YP46ib2zLT0DOa/7hr 70YiG3P65mztA5RBSaMQ5/ws17X/vsg= X-Google-Smtp-Source: ACHHUZ6licAmCYMGFmFEgxRhjJBE3vK82JOqHF+xQXe/hB6jcq2RRjEzB8Dg4J2EO+jXUyt83MZ91zsgyn0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a05:6902:100e:b0:b8f:47c4:58ed with SMTP id w14-20020a056902100e00b00b8f47c458edmr8682966ybt.9.1682960124020; Mon, 01 May 2023 09:55:24 -0700 (PDT) Date: Mon, 1 May 2023 09:54:17 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-8-surenb@google.com> Subject: [PATCH 07/40] Lazy percpu counters From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: 8rbaq31qwschdwxm7qpnyci1ph3m841h X-Rspam-User: X-Rspamd-Queue-Id: E48C6140015 X-Rspamd-Server: rspam06 X-HE-Tag: 1682960124-289495 X-HE-Meta: U2FsdGVkX1+A4wo7gRI7bgjMGjfQQSkNiLUsBUSXsXdTBYwacfTTRPBVkjfG53yKRV34SF0KmStDJVDuNMpzPdgFNITctEdKnKhxGodIISvvZ9l+3V2tf/31EAGzj/h2reE3bJGQiIQEejC7KDDGndbaswBxg8ZuFq0119PJ8/uzSYtF9HUEdI4F/v1a6EdBJmGQFroZJOuWdq9uizjj9FVLeyyA0F41/CwNN8sz5CSY8l9PB6pcHYnnilfOZs0fC0eXq948Fnn5ikabp306uMfkCt48sLD/XoPOSTchLF45HhyXGzsChFCqq669+hPeWvaj8nmjtjYDS4kyJLhHZYRoqdGcqe0hWndBcNWbIKwhWODFZBuNGkYHUFMax/BqE3PBZXtlSJ/mQPAAegEpYdxtaOtXeQ9y2hDEHQUhQuhCHAyu+5l6I6suNTaEKkzrDcKY45VAXb9hE1pgYEWz9c3ox300p4SL7lnXo5jN0zV0sm0I/tJbQA99UM/8EW2wHuis2x5lsWWGqQt1QnlCgTnoqA4enfLFmCY4iGw2hlQVJm//A0KF2YpCRtA2plhX+E5KcTmu50bmPmdDpJGDkyb0wezVTd3A/mBMZWpgUAzOyjsUe9OxiDqn7dygcmj33huc9ntbvlxY1trwCkX373/i5RkfiXq5vWy4SnQ2mHvyq3JQNHF9AqjU7Pr/ibRAamxWpadt1jKjuoJs5X8pl2rfC8jZ7SWaFqY+atwVyO7TjJYmbZIJWbEGEBfOWUyo0EuJGOhFaHoYJcMcTQBubN/hCZ3j4qCtjvH3JDTBApRbBl7CRQiupqi66SDecOQSw4nJWhWnxuJoUlaiH/akyKhRLJi3tSvz8dQh9UGGVHYqvnLEGkvWGJy1MuP2yEmWqF8a9gVa6zQ0E7exvy3wo2wn9FTtmftlH/ziDhET6SwlMkvT5RQY8U9J984xgY0mi8/YaP8oI51xPYpU5Xl 1uZB+5sn Z4XjXXCTj8Qp0P1rVbUiwXLqEO3sUzqMaCCBFoVGqOc5S3gf4ifC/5VSjjr8hzmRpnfFEB4NtWyy2GWJw7OiOEhDVzDr8V1/KNvWAc2UDqWjrpy5Z85CCdieDJ5tEWEbOw4KZD+o7mklu9xKcc0u6v3aJMgzSxZ4DNr8NLjj9rfF88KHMeUe+R6kYd9SuAdAdfle2gv3Ko5NIPS4mB3siWXwSVpGfBhN++KrAfg8QKOk33BZHpDTxS6uQs2KfUGBIeyqr/pUF+XUUJl55j//0YayVbwFLiBoh/7uXAVUpmCK+5vQwa+R1XpbFFDBzCy6ShSWk87jabF6uBN+wTw+PjaRWJlRkaKq/XxwBir4UFLtuObdY5q2wq/k+PGPS2FL9b156yblYx7EyydR6MpmkAJm6/bRqn2vFBnl8iyKmj8cdvOoY8bIHy0waCVuzwfL20duVPF7xpJz2Y7a1TTW4Jhey89IEyhcZskHWMnUvuG3KWg4ddcsVs6FCNIWOIK1bk47DlnhI6isFXDJ08l7j5zKO9JLkcOj2tsMIYTlz8dB3Tx2HoMmuW9h0+Tsf8zh26E1cHUd7TUHk12oyFf5SW3dYAYDrH928MJqSCJkslLhjlXtPJNdL48oqy4zm9E0Al2LfQzvbgNLsSpbQjPaCP7zkPDjoqWIWIGy9s2z9Jky6pFPgQ3+ZjhO6/g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet This patch adds lib/lazy-percpu-counter.c, which implements counters that start out as atomics, but lazily switch to percpu mode if the update rate crosses some threshold (arbitrarily set at 256 per second). Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/lazy-percpu-counter.h | 102 ++++++++++++++++++++++ lib/Kconfig | 3 + lib/Makefile | 2 + lib/lazy-percpu-counter.c | 127 ++++++++++++++++++++++++++++ 4 files changed, 234 insertions(+) create mode 100644 include/linux/lazy-percpu-counter.h create mode 100644 lib/lazy-percpu-counter.c diff --git a/include/linux/lazy-percpu-counter.h b/include/linux/lazy-percpu-counter.h new file mode 100644 index 000000000000..45ca9e2ce58b --- /dev/null +++ b/include/linux/lazy-percpu-counter.h @@ -0,0 +1,102 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Lazy percpu counters: + * (C) 2022 Kent Overstreet + * + * Lazy percpu counters start out in atomic mode, then switch to percpu mode if + * the update rate crosses some threshold. + * + * This means we don't have to decide between low memory overhead atomic + * counters and higher performance percpu counters - we can have our cake and + * eat it, too! + * + * Internally we use an atomic64_t, where the low bit indicates whether we're in + * percpu mode, and the high 8 bits are a secondary counter that's incremented + * when the counter is modified - meaning 55 bits of precision are available for + * the counter itself. + */ + +#ifndef _LINUX_LAZY_PERCPU_COUNTER_H +#define _LINUX_LAZY_PERCPU_COUNTER_H + +#include +#include + +struct lazy_percpu_counter { + atomic64_t v; + unsigned long last_wrap; +}; + +void lazy_percpu_counter_exit(struct lazy_percpu_counter *c); +void lazy_percpu_counter_add_slowpath(struct lazy_percpu_counter *c, s64 i); +void lazy_percpu_counter_add_slowpath_noupgrade(struct lazy_percpu_counter *c, s64 i); +s64 lazy_percpu_counter_read(struct lazy_percpu_counter *c); + +/* + * We use the high bits of the atomic counter for a secondary counter, which is + * incremented every time the counter is touched. When the secondary counter + * wraps, we check the time the counter last wrapped, and if it was recent + * enough that means the update frequency has crossed our threshold and we + * switch to percpu mode: + */ +#define COUNTER_MOD_BITS 8 +#define COUNTER_MOD_MASK ~(~0ULL >> COUNTER_MOD_BITS) +#define COUNTER_MOD_BITS_START (64 - COUNTER_MOD_BITS) + +/* + * We use the low bit of the counter to indicate whether we're in atomic mode + * (low bit clear), or percpu mode (low bit set, counter is a pointer to actual + * percpu counters: + */ +#define COUNTER_IS_PCPU_BIT 1 + +static inline u64 __percpu *lazy_percpu_counter_is_pcpu(u64 v) +{ + if (!(v & COUNTER_IS_PCPU_BIT)) + return NULL; + + v ^= COUNTER_IS_PCPU_BIT; + return (u64 __percpu *)(unsigned long)v; +} + +/** + * lazy_percpu_counter_add: Add a value to a lazy_percpu_counter + * + * @c: counter to modify + * @i: value to add + */ +static inline void lazy_percpu_counter_add(struct lazy_percpu_counter *c, s64 i) +{ + u64 v = atomic64_read(&c->v); + u64 __percpu *pcpu_v = lazy_percpu_counter_is_pcpu(v); + + if (likely(pcpu_v)) + this_cpu_add(*pcpu_v, i); + else + lazy_percpu_counter_add_slowpath(c, i); +} + +/** + * lazy_percpu_counter_add_noupgrade: Add a value to a lazy_percpu_counter, + * without upgrading to percpu mode + * + * @c: counter to modify + * @i: value to add + */ +static inline void lazy_percpu_counter_add_noupgrade(struct lazy_percpu_counter *c, s64 i) +{ + u64 v = atomic64_read(&c->v); + u64 __percpu *pcpu_v = lazy_percpu_counter_is_pcpu(v); + + if (likely(pcpu_v)) + this_cpu_add(*pcpu_v, i); + else + lazy_percpu_counter_add_slowpath_noupgrade(c, i); +} + +static inline void lazy_percpu_counter_sub(struct lazy_percpu_counter *c, s64 i) +{ + lazy_percpu_counter_add(c, -i); +} + +#endif /* _LINUX_LAZY_PERCPU_COUNTER_H */ diff --git a/lib/Kconfig b/lib/Kconfig index 5c2da561c516..7380292a8fcd 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -505,6 +505,9 @@ config ASSOCIATIVE_ARRAY for more information. +config LAZY_PERCPU_COUNTER + bool + config HAS_IOMEM bool depends on !NO_IOMEM diff --git a/lib/Makefile b/lib/Makefile index 876fcdeae34e..293a0858a3f8 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -164,6 +164,8 @@ obj-$(CONFIG_DEBUG_PREEMPT) += smp_processor_id.o obj-$(CONFIG_DEBUG_LIST) += list_debug.o obj-$(CONFIG_DEBUG_OBJECTS) += debugobjects.o +obj-$(CONFIG_LAZY_PERCPU_COUNTER) += lazy-percpu-counter.o + obj-$(CONFIG_BITREVERSE) += bitrev.o obj-$(CONFIG_LINEAR_RANGES) += linear_ranges.o obj-$(CONFIG_PACKING) += packing.o diff --git a/lib/lazy-percpu-counter.c b/lib/lazy-percpu-counter.c new file mode 100644 index 000000000000..4f4e32c2dc09 --- /dev/null +++ b/lib/lazy-percpu-counter.c @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include +#include + +static inline s64 lazy_percpu_counter_atomic_val(s64 v) +{ + /* Ensure output is sign extended properly: */ + return (v << COUNTER_MOD_BITS) >> + (COUNTER_MOD_BITS + COUNTER_IS_PCPU_BIT); +} + +static void lazy_percpu_counter_switch_to_pcpu(struct lazy_percpu_counter *c) +{ + u64 __percpu *pcpu_v = alloc_percpu_gfp(u64, GFP_ATOMIC|__GFP_NOWARN); + u64 old, new, v; + + if (!pcpu_v) + return; + + preempt_disable(); + v = atomic64_read(&c->v); + do { + if (lazy_percpu_counter_is_pcpu(v)) { + free_percpu(pcpu_v); + return; + } + + old = v; + new = (unsigned long)pcpu_v | 1; + + *this_cpu_ptr(pcpu_v) = lazy_percpu_counter_atomic_val(v); + } while ((v = atomic64_cmpxchg(&c->v, old, new)) != old); + preempt_enable(); +} + +/** + * lazy_percpu_counter_exit: Free resources associated with a + * lazy_percpu_counter + * + * @c: counter to exit + */ +void lazy_percpu_counter_exit(struct lazy_percpu_counter *c) +{ + free_percpu(lazy_percpu_counter_is_pcpu(atomic64_read(&c->v))); +} +EXPORT_SYMBOL_GPL(lazy_percpu_counter_exit); + +/** + * lazy_percpu_counter_read: Read current value of a lazy_percpu_counter + * + * @c: counter to read + */ +s64 lazy_percpu_counter_read(struct lazy_percpu_counter *c) +{ + s64 v = atomic64_read(&c->v); + u64 __percpu *pcpu_v = lazy_percpu_counter_is_pcpu(v); + + if (pcpu_v) { + int cpu; + + v = 0; + for_each_possible_cpu(cpu) + v += *per_cpu_ptr(pcpu_v, cpu); + } else { + v = lazy_percpu_counter_atomic_val(v); + } + + return v; +} +EXPORT_SYMBOL_GPL(lazy_percpu_counter_read); + +void lazy_percpu_counter_add_slowpath(struct lazy_percpu_counter *c, s64 i) +{ + u64 atomic_i; + u64 old, v = atomic64_read(&c->v); + u64 __percpu *pcpu_v; + + atomic_i = i << COUNTER_IS_PCPU_BIT; + atomic_i &= ~COUNTER_MOD_MASK; + atomic_i |= 1ULL << COUNTER_MOD_BITS_START; + + do { + pcpu_v = lazy_percpu_counter_is_pcpu(v); + if (pcpu_v) { + this_cpu_add(*pcpu_v, i); + return; + } + + old = v; + } while ((v = atomic64_cmpxchg(&c->v, old, old + atomic_i)) != old); + + if (unlikely(!(v & COUNTER_MOD_MASK))) { + unsigned long now = jiffies; + + if (c->last_wrap && + unlikely(time_after(c->last_wrap + HZ, now))) + lazy_percpu_counter_switch_to_pcpu(c); + else + c->last_wrap = now; + } +} +EXPORT_SYMBOL(lazy_percpu_counter_add_slowpath); + +void lazy_percpu_counter_add_slowpath_noupgrade(struct lazy_percpu_counter *c, s64 i) +{ + u64 atomic_i; + u64 old, v = atomic64_read(&c->v); + u64 __percpu *pcpu_v; + + atomic_i = i << COUNTER_IS_PCPU_BIT; + atomic_i &= ~COUNTER_MOD_MASK; + + do { + pcpu_v = lazy_percpu_counter_is_pcpu(v); + if (pcpu_v) { + this_cpu_add(*pcpu_v, i); + return; + } + + old = v; + } while ((v = atomic64_cmpxchg(&c->v, old, old + atomic_i)) != old); +} +EXPORT_SYMBOL(lazy_percpu_counter_add_slowpath_noupgrade); From patchwork Mon May 1 16:54:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F420C77B7C for ; Mon, 1 May 2023 16:55:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0749590000B; Mon, 1 May 2023 12:55:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3EB2900002; Mon, 1 May 2023 12:55:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDF8190000B; Mon, 1 May 2023 12:55:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CA865900002 for ; Mon, 1 May 2023 12:55:29 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9F42912081B for ; Mon, 1 May 2023 16:55:29 +0000 (UTC) X-FDA: 80742287178.18.E29F73F Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id 68E031C000D for ; Mon, 1 May 2023 16:55:27 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=KndAV4af; spf=pass (imf18.hostedemail.com: domain of 3_u5PZAYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3_u5PZAYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960127; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EpAPHCELRqlGyXafm3HOq1DjysfSUkIp0lW71FhLJ+Q=; b=KqT4aVDYHqY7KT4F+ubygq5fXALWx/bW69VxqaTJ1IxuOTloFtvQ8T33l6txIEPgOnq1yU EryKjXJYJDx808KEzyxLoRSF9GF89F+yaFaSTvvn4EOIR6qBHZSQIQdA8gsKunNH/G/nKI K5ERRg+yHB4FTs6ZNk9qhEuCCvGZ3xc= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=KndAV4af; spf=pass (imf18.hostedemail.com: domain of 3_u5PZAYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3_u5PZAYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960127; a=rsa-sha256; cv=none; b=0MbpQXK49r9qNwY4bP0XcXXBIwtsUp1gAKCrORJyxOpRcJ3qFDrxc8aATV3pmvj5TJVFVL Y587GMmY5rPxSwyhpIEX3XydbbEZsWV/2wXmchAI32ugu+fwhGD+rT02YCmeLPJvh7s3aw lMV/jzD7EioQzsJXAKDZy8kLwuxcCvo= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-b9968fb4a8cso5024807276.0 for ; Mon, 01 May 2023 09:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960126; x=1685552126; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EpAPHCELRqlGyXafm3HOq1DjysfSUkIp0lW71FhLJ+Q=; b=KndAV4afZjE7urj1LZ7Rf/o633ajiD8s+Zgf2L1Mk9XX6B/9PMd2nEUVnlftP7GrKt bWCCrfLe4oDbYF4trh5h6CFhIJwiPA+kfT/a0Y4OXyPznsVC8TV+5wilPJneaIud8ycS FrIUNz8/SFFMagOyadbcqEA0OzG1GIoTXbpA5wGATZUND86MP2J0BOIMLXyxPVqWZ9Wl hyCmq22BBXz06DX2ppJBr0xZldtvTeIUTTqyjK3oZrVhnhKWEQyYLcpCxTL3wMg/+F+S 6FiOvawPLieCpVMV7STZbWNFOnrCi/8x9VUaf5jjtzPaRTlRCN5z85N54yndm8M1FUN8 3eIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960126; x=1685552126; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EpAPHCELRqlGyXafm3HOq1DjysfSUkIp0lW71FhLJ+Q=; b=LiqRcm9UwP80rVv2fOs4hzAH4vUgf+6QYNC2AdISVZ2rPMCkZsl1/oS17xNrYDWk5A Oh9TpgPftM2etemlecWuKoDlYCbtZ6OpZm29KNAWobJ7iTDCsroCX9IEtSlRX3jncies 6YTWI8Aznazo2DcDYDGpOkTBGIIJMEDrRZsc5cV5bH28LNhVlduRCN2EexGC34JhQvmC LycqUT7zaURfz5AjcJClyIrKsmXGYgSfQz0g+5ewBptUG9NxDWSS012fNmcI/TcVI8vU XTss94I2fCkgZnyFtz8BH989uiLCevJDueL8a3qYThmz9fKHQVL7CPXRdIfy5SDp7P6Q KD+Q== X-Gm-Message-State: AC+VfDzbvvo1RudebxYuApyO337yOpDwBYb3UslFdjWQPt0DXWqyOIGb ESbjdJYbuKKJ99I5U9WaiYlJU7HQJco= X-Google-Smtp-Source: ACHHUZ4pUE1v0mxDvosroh8JeD02p6+n+qSCw+OmLNamZIOJjtYT8dz59YYMSGTncMrJsAgRsN3jwHLs8Lw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:db10:0:b0:b9d:b2ef:1b1e with SMTP id g16-20020a25db10000000b00b9db2ef1b1emr2548037ybf.7.1682960126449; Mon, 01 May 2023 09:55:26 -0700 (PDT) Date: Mon, 1 May 2023 09:54:18 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-9-surenb@google.com> Subject: [PATCH 08/40] mm: introduce slabobj_ext to support slab object extensions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 68E031C000D X-Stat-Signature: phodquje6qpoaxu17u6ct1ypncho4pn6 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682960127-281649 X-HE-Meta: U2FsdGVkX1995bM+VgjICsWH+ItmdbITdHFEYnTuYEXSiIxfv2jiFR5PXm+zMSZc7AKkCAqA0xfAV+7D3K29iyweH2DmebpOEkTXFQ+iuiHUWR6o1rCTpqhP24jRv2gr9OZkIJeVBBzzV8LbBTCNtHchcIqvb0E/CPYoZVCFBam9AtMo9c28OweQk5cAvBzTibFjmuN7fDMLXMMukcTk2LuKClkLAqEe54ymEyJyU0T23Slv2RYd9gj1E+7f9z96ymhEa6vrvA+L+qRnubV+Uq0RESEQK03PNk729vhl7MFaL0iXfpL91wdOXnhHv+jwt99WvgxBNnbfiK8XdaPugj1Y3rHIaB2zEmpyUTt9CJPPVkxDErkCbDJp86qArntKvx3dJhrXPr3Iz2xUjSouw5LGQgSyuroP0in0xdm/ahE4o/TI+tNgBzilPT7cYbgeRMowB3nzJtK8uSn/sGl0UKsRrBMACoXLgmR6t2K2OWnwxXMIvi4LWLj0gC+dnoZcyeBV/aWwikHmlExu9HsZCh6r68S0XN5bpEBjnZtLNyKpMCpmyvjvmEktdA0IE8IA9LsTkkX5MELgxoldJfAzCEpPH9MTYiK4FbKVXZMeWq9ewOFfwFXH6XWvYGm32RhwVWxe9UsrI5l8VRX1iphPYwCdkDYv/zhGFE3+b5hQ1vQjY5yGC5XjC6xDfbh2GTpfaQ8lX1MKdo30u23JnDHbJ2wJ5IlkV+HI75jhzdMgPKaoDVZdywRSQ0niv8A798YL7QJ/qPiKALnXCV3HbKRdocTJqaNc+WTnREzrYnc9LufIDUxTM45qIJ/owHwuD8RtaNR00H//hjrFsMHwM3NbMHaZ8uY6OnolnJXT8tmeP5JXzRE7uKjirY3WLjLq14nXAettjg2IDMEyJJUKlZJGrhIsOtr4xKds1vojdEhlYWy4YRnQcq6CrmXORko8CGlhgZkn1zHCq+jrsV9eBI+ gkyOcMOe 946H2PuAewa+QwvDuEy2lVvdKuh2HJC3WcHTJCJZHQhxEZ4IAA0CxrS22leVZ+6plLj5qQeGa7OMqOblE4WryLNkuTNDxj6IJfoXLQFrd3EqvxsHeP5lL0S2r05rl334I7b9NV/N5QcTun797I+Kn8n6r/2i9rKhPidmpG2kf4kVCuDzqIcrxpAZrnx5n0aS8+MxRv/vX1/8WRzAE01+PJr4HrYZfnf+6gGQZObIPhmY3BCXZCmrQ+3JmQ2/RmImLUHZlw6ndsjuzRMQ3hpXBV+5u+zwFsgziA9zoxypjs8wqrMtQfJIkaby97YtOz09zdbSZ3EEngH/hdrM6bd+PjbCTWLaQeALo50m/GlWVJzgK1m1fHBheQ+HbB2vxoEV+FNkHycWoVDUIL6glRACRkgksz4X6iBnW8pLFTmZ03GV0bwxLhwCEunvyO5AmsVTOjIPHYK2j3lbvDlIOXCddo4hXxkn10tmHUA7qYNaAyky/CJgo9QVlVrdP3X86oUTxYeqDe261RblEU+Sbh5slt0yTqpyErs/qHExwjpaa/VvKXO7VtN8ss0PAUIHh9bl3VhXLS5AijmlPHbOGrNLMYSu3r4jW5B81xSCF+svSA9xUNB6W41DbTbd7l+d8OJajwiR3NvFAbd9pf9cddRYnlL2Iouy+/N+w36vATJEcTssGjoVzKLSV/ha5T7/JN02NTZ7N3sjYYayDYqt+G9tLdLt35GjYctMx/1Qv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently slab pages can store only vectors of obj_cgroup pointers in page->memcg_data. Introduce slabobj_ext structure to allow more data to be stored for each slab object. Wrap obj_cgroup into slabobj_ext to support current functionality while allowing to extend slabobj_ext in the future. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 20 +++-- include/linux/mm_types.h | 4 +- init/Kconfig | 4 + mm/kfence/core.c | 14 ++-- mm/kfence/kfence.h | 4 +- mm/memcontrol.c | 56 ++------------ mm/page_owner.c | 2 +- mm/slab.h | 148 +++++++++++++++++++++++++------------ mm/slab_common.c | 47 ++++++++++++ 9 files changed, 185 insertions(+), 114 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 222d7370134c..b9fd9732a52b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -339,8 +339,8 @@ struct mem_cgroup { extern struct mem_cgroup *root_mem_cgroup; enum page_memcg_data_flags { - /* page->memcg_data is a pointer to an objcgs vector */ - MEMCG_DATA_OBJCGS = (1UL << 0), + /* page->memcg_data is a pointer to an slabobj_ext vector */ + MEMCG_DATA_OBJEXTS = (1UL << 0), /* page has been accounted as a non-slab kernel page */ MEMCG_DATA_KMEM = (1UL << 1), /* the next bit after the last actual flag */ @@ -378,7 +378,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -399,7 +399,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -496,7 +496,7 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) */ unsigned long memcg_data = READ_ONCE(folio->memcg_data); - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) return NULL; if (memcg_data & MEMCG_DATA_KMEM) { @@ -542,7 +542,7 @@ static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *ob static inline bool folio_memcg_kmem(struct folio *folio) { VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); - VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJEXTS, folio); return folio->memcg_data & MEMCG_DATA_KMEM; } @@ -1606,6 +1606,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, } #endif /* CONFIG_MEMCG */ +/* + * Extended information for slab objects stored as an array in page->memcg_data + * if MEMCG_DATA_OBJEXTS is set. + */ +struct slabobj_ext { + struct obj_cgroup *objcg; +} __aligned(8); + static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) { __mod_lruvec_kmem_state(p, idx, 1); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..e79303e1e30c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -194,7 +194,7 @@ struct page { /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif @@ -320,7 +320,7 @@ struct folio { void *private; atomic_t _mapcount; atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif /* private: the union with struct page is transitional */ diff --git a/init/Kconfig b/init/Kconfig index 32c24950c4ce..44267919a2a2 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -936,10 +936,14 @@ config CGROUP_FAVOR_DYNMODS Say N if unsure. +config SLAB_OBJ_EXT + bool + config MEMCG bool "Memory controller" select PAGE_COUNTER select EVENTFD + select SLAB_OBJ_EXT help Provides control over the memory footprint of tasks in a cgroup. diff --git a/mm/kfence/core.c b/mm/kfence/core.c index dad3c0eb70a0..aea6fa145080 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -590,9 +590,9 @@ static unsigned long kfence_init_pool(void) continue; __folio_set_slab(slab_folio(slab)); -#ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long)&kfence_metadata[i / 2 - 1].objcg | - MEMCG_DATA_OBJCGS; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = (unsigned long)&kfence_metadata[i / 2 - 1].obj_exts | + MEMCG_DATA_OBJEXTS; #endif } @@ -634,8 +634,8 @@ static unsigned long kfence_init_pool(void) if (!i || (i % 2)) continue; -#ifdef CONFIG_MEMCG - slab->memcg_data = 0; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = 0; #endif __folio_clear_slab(slab_folio(slab)); } @@ -1093,8 +1093,8 @@ void __kfence_free(void *addr) { struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr); -#ifdef CONFIG_MEMCG - KFENCE_WARN_ON(meta->objcg); +#ifdef CONFIG_MEMCG_KMEM + KFENCE_WARN_ON(meta->obj_exts.objcg); #endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index 2aafc46a4aaf..8e0d76c4ea2a 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -97,8 +97,8 @@ struct kfence_metadata { struct kfence_track free_track; /* For updating alloc_covered on frees. */ u32 alloc_stack_hash; -#ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; +#ifdef CONFIG_MEMCG_KMEM + struct slabobj_ext obj_exts; #endif }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4b27e245a055..f2a7fe718117 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2892,13 +2892,6 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) } #ifdef CONFIG_MEMCG_KMEM -/* - * The allocated objcg pointers array is not accounted directly. - * Moreover, it should not come from DMA buffer and is not readily - * reclaimable. So those GFP bits should be masked off. - */ -#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | __GFP_ACCOUNT) - /* * mod_objcg_mlstate() may be called with irq enabled, so * mod_memcg_lruvec_state() should be used. @@ -2917,62 +2910,27 @@ static inline void mod_objcg_mlstate(struct obj_cgroup *objcg, rcu_read_unlock(); } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab) -{ - unsigned int objects = objs_per_slab(s, slab); - unsigned long memcg_data; - void *vec; - - gfp &= ~OBJCGS_CLEAR_MASK; - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - slab_nid(slab)); - if (!vec) - return -ENOMEM; - - memcg_data = (unsigned long) vec | MEMCG_DATA_OBJCGS; - if (new_slab) { - /* - * If the slab is brand new and nobody can yet access its - * memcg_data, no synchronization is required and memcg_data can - * be simply assigned. - */ - slab->memcg_data = memcg_data; - } else if (cmpxchg(&slab->memcg_data, 0, memcg_data)) { - /* - * If the slab is already in use, somebody can allocate and - * assign obj_cgroups in parallel. In this case the existing - * objcg vector should be reused. - */ - kfree(vec); - return 0; - } - - kmemleak_not_leak(vec); - return 0; -} - static __always_inline struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) { /* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in - * slab->memcg_data. + * slab->obj_exts. */ if (folio_test_slab(folio)) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; struct slab *slab; unsigned int off; slab = folio_slab(folio); - objcgs = slab_objcgs(slab); - if (!objcgs) + obj_exts = slab_obj_exts(slab); + if (!obj_exts) return NULL; off = obj_to_index(slab->slab_cache, slab, p); - if (objcgs[off]) - return obj_cgroup_memcg(objcgs[off]); + if (obj_exts[off].objcg) + return obj_cgroup_memcg(obj_exts[off].objcg); return NULL; } @@ -2980,7 +2938,7 @@ struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) /* * folio_memcg_check() is used here, because in theory we can encounter * a folio where the slab flag has been cleared already, but - * slab->memcg_data has not been freed yet + * slab->obj_exts has not been freed yet * folio_memcg_check() will guarantee that a proper memory * cgroup pointer or NULL will be returned. */ diff --git a/mm/page_owner.c b/mm/page_owner.c index 31169b3e7f06..8b6086c666e6 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -372,7 +372,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, if (!memcg_data) goto out_unlock; - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) ret += scnprintf(kbuf + ret, count - ret, "Slab cache page\n"); diff --git a/mm/slab.h b/mm/slab.h index f01ac256a8f5..25d14b3a7280 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -57,8 +57,8 @@ struct slab { #endif atomic_t __page_refcount; -#ifdef CONFIG_MEMCG - unsigned long memcg_data; +#ifdef CONFIG_SLAB_OBJ_EXT + unsigned long obj_exts; #endif }; @@ -67,8 +67,8 @@ struct slab { SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ SLAB_MATCH(_refcount, __page_refcount); -#ifdef CONFIG_MEMCG -SLAB_MATCH(memcg_data, memcg_data); +#ifdef CONFIG_SLAB_OBJ_EXT +SLAB_MATCH(memcg_data, obj_exts); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <= sizeof(struct page)); @@ -390,36 +390,106 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } -#ifdef CONFIG_MEMCG_KMEM +#ifdef CONFIG_SLAB_OBJ_EXT + /* - * slab_objcgs - get the object cgroups vector associated with a slab + * slab_obj_exts - get the pointer to the slab object extension vector + * associated with a slab. * @slab: a pointer to the slab struct * - * Returns a pointer to the object cgroups vector associated with the slab, + * Returns a pointer to the object extension vector associated with the slab, * or NULL if no such vector has been associated yet. */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { - unsigned long memcg_data = READ_ONCE(slab->memcg_data); + unsigned long obj_exts = READ_ONCE(slab->obj_exts); - VM_BUG_ON_PAGE(memcg_data && !(memcg_data & MEMCG_DATA_OBJCGS), +#ifdef CONFIG_MEMCG + VM_BUG_ON_PAGE(obj_exts && !(obj_exts & MEMCG_DATA_OBJEXTS), slab_page(slab)); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, slab_page(slab)); + VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); +#else + return (struct slabobj_ext *)obj_exts; +#endif } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab); -void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, - enum node_stat_item idx, int nr); +int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab); -static inline void memcg_free_slab_cgroups(struct slab *slab) +static inline bool need_slab_obj_ext(void) { - kfree(slab_objcgs(slab)); - slab->memcg_data = 0; + /* + * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally + * inside memcg_slab_post_alloc_hook. No other users for now. + */ + return false; } +static inline void free_slab_obj_exts(struct slab *slab) +{ + struct slabobj_ext *obj_exts; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + kfree(obj_exts); + slab->obj_exts = 0; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + struct slab *slab; + + if (!p) + return NULL; + + if (!need_slab_obj_ext()) + return NULL; + + slab = virt_to_slab(p); + if (!slab_obj_exts(slab) && + WARN(alloc_slab_obj_exts(slab, s, flags, false), + "%s, %s: Failed to create slab extension vector!\n", + __func__, s->name)) + return NULL; + + return slab_obj_exts(slab) + obj_to_index(s, slab, p); +} + +#else /* CONFIG_SLAB_OBJ_EXT */ + +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) +{ + return NULL; +} + +static inline int alloc_slab_obj_exts(struct slab *slab, + struct kmem_cache *s, gfp_t gfp, + bool new_slab) +{ + return 0; +} + +static inline void free_slab_obj_exts(struct slab *slab) +{ +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + return NULL; +} + +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM +void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, + enum node_stat_item idx, int nr); + static inline size_t obj_full_size(struct kmem_cache *s) { /* @@ -487,16 +557,15 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, if (likely(p[i])) { slab = virt_to_slab(p[i]); - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, - false)) { + if (!slab_obj_exts(slab) && + alloc_slab_obj_exts(slab, s, flags, false)) { obj_cgroup_uncharge(objcg, obj_full_size(s)); continue; } off = obj_to_index(s, slab, p[i]); obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; + slab_obj_exts(slab)[off].objcg = objcg; mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), obj_full_size(s)); } else { @@ -509,14 +578,14 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; int i; if (!memcg_kmem_online()) return; - objcgs = slab_objcgs(slab); - if (!objcgs) + obj_exts = slab_obj_exts(slab); + if (!obj_exts) return; for (i = 0; i < objects; i++) { @@ -524,11 +593,11 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, unsigned int off; off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; + objcg = obj_exts[off].objcg; if (!objcg) continue; - objcgs[off] = NULL; + obj_exts[off].objcg = NULL; obj_cgroup_uncharge(objcg, obj_full_size(s)); mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), -obj_full_size(s)); @@ -537,27 +606,11 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, } #else /* CONFIG_MEMCG_KMEM */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) -{ - return NULL; -} - static inline struct mem_cgroup *memcg_from_slab_obj(void *ptr) { return NULL; } -static inline int memcg_alloc_slab_cgroups(struct slab *slab, - struct kmem_cache *s, gfp_t gfp, - bool new_slab) -{ - return 0; -} - -static inline void memcg_free_slab_cgroups(struct slab *slab) -{ -} - static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, struct obj_cgroup **objcgp, @@ -594,7 +647,7 @@ static __always_inline void account_slab(struct slab *slab, int order, struct kmem_cache *s, gfp_t gfp) { if (memcg_kmem_online() && (s->flags & SLAB_ACCOUNT)) - memcg_alloc_slab_cgroups(slab, s, gfp, true); + alloc_slab_obj_exts(slab, s, gfp, true); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), PAGE_SIZE << order); @@ -603,8 +656,7 @@ static __always_inline void account_slab(struct slab *slab, int order, static __always_inline void unaccount_slab(struct slab *slab, int order, struct kmem_cache *s) { - if (memcg_kmem_online()) - memcg_free_slab_cgroups(slab); + free_slab_obj_exts(slab); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), -(PAGE_SIZE << order)); @@ -684,6 +736,7 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, unsigned int orig_size) { unsigned int zero_size = s->object_size; + struct slabobj_ext *obj_exts; size_t i; flags &= gfp_allowed_mask; @@ -714,6 +767,7 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, kmemleak_alloc_recursive(p[i], s->object_size, 1, s->flags, flags); kmsan_slab_alloc(s, p[i], flags); + obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); diff --git a/mm/slab_common.c b/mm/slab_common.c index 607249785c07..f11cc072b01e 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -204,6 +204,53 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, return NULL; } +#ifdef CONFIG_SLAB_OBJ_EXT +/* + * The allocated objcg pointers array is not accounted directly. + * Moreover, it should not come from DMA buffer and is not readily + * reclaimable. So those GFP bits should be masked off. + */ +#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | __GFP_ACCOUNT) + +int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab) +{ + unsigned int objects = objs_per_slab(s, slab); + unsigned long obj_exts; + void *vec; + + gfp &= ~OBJCGS_CLEAR_MASK; + vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, + slab_nid(slab)); + if (!vec) + return -ENOMEM; + + obj_exts = (unsigned long)vec; +#ifdef CONFIG_MEMCG + obj_exts |= MEMCG_DATA_OBJEXTS; +#endif + if (new_slab) { + /* + * If the slab is brand new and nobody can yet access its + * obj_exts, no synchronization is required and obj_exts can + * be simply assigned. + */ + slab->obj_exts = obj_exts; + } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + /* + * If the slab is already in use, somebody can allocate and + * assign slabobj_exts in parallel. In this case the existing + * objcg vector should be reused. + */ + kfree(vec); + return 0; + } + + kmemleak_not_leak(vec); + return 0; +} +#endif /* CONFIG_SLAB_OBJ_EXT */ + static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset, From patchwork Mon May 1 16:54:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90A37C77B73 for ; Mon, 1 May 2023 16:55:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3327A90000C; Mon, 1 May 2023 12:55:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BD97900002; Mon, 1 May 2023 12:55:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C00690000C; Mon, 1 May 2023 12:55:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EC04B900002 for ; Mon, 1 May 2023 12:55:31 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C51BC407F9 for ; Mon, 1 May 2023 16:55:31 +0000 (UTC) X-FDA: 80742287262.26.2FE6491 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf28.hostedemail.com (Postfix) with ESMTP id E3311C001C for ; Mon, 1 May 2023 16:55:29 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=j+Kr4wFQ; spf=pass (imf28.hostedemail.com: domain of 3AO9PZAYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3AO9PZAYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960130; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DK6/s7foZNj5EExVH4gZckioJVPcIxI52cq1C7FfTD8=; b=XaNjW2x0Aua7BEQWkps2d5wSc/RXm+n8rGJ/EMzL9rvT/1ZuwcytEsksibQi+MVyd2QS47 s22hfbSzxi3F7Jebtk3M4aqpjCeXeJV0THqs+WpbAl+gK+tprYwQc9o53wy+Y94VOjvkx/ v690mky3ivTzEm+NqVMqBCCHDMTl7eA= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=j+Kr4wFQ; spf=pass (imf28.hostedemail.com: domain of 3AO9PZAYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3AO9PZAYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960130; a=rsa-sha256; cv=none; b=hFUU7Y1LUD/rwVCYUYmOCHtZtXWXoJ3ORbiWEnAOpSLuz7wXHy7/By5c3W87q6wX8kGxiq 018TWxa9mAvuDQco/Adh1lUVNFS9g4fFf64+fD6+5/t3mSOtXOPnqIRQQepVHjdhAu7lm6 71sA87xC+wStCfu64qnV3KFO2TzXVtA= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1aafc4aa1c6so5947355ad.1 for ; Mon, 01 May 2023 09:55:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960129; x=1685552129; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DK6/s7foZNj5EExVH4gZckioJVPcIxI52cq1C7FfTD8=; b=j+Kr4wFQ8hb6ghgASWLoQmwshyZipq1bE+D/CA603zsc4kB1n7R3CTQw1X9N4pK0pE 0zmuTlr7BciP0EtB463YkQ7YcnZO9MQaA53eLGwRosKH+IiuL9AF+xukRwd1DIpRoQoE CuOOiRq7K1VtqSyYvIO65JDUgFMtQPFNC1XFk5UKDZu8zG86BBXEpcIuAsalgfYjpqSk 4H7kaQB7eIw3KHROJbChdasmtpmhAOtDolQRdOsPV+jVQQISfSODNw3+xILO7OLY0pMp pmy+NWFCOxFQwMauo6Mv0q9Qa8hd2IlOPM48vsABoana6/QPtONXZz6Lhi9KWaOg0TwZ kOfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960129; x=1685552129; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DK6/s7foZNj5EExVH4gZckioJVPcIxI52cq1C7FfTD8=; b=fYLg1SKF58o8HWyEu8oECha9qSEQGcDTdVBBMoFG9j//R8gIkOmxSoPdtO3tHsl9U1 FwQNx89DtYSdlKwBBhXIDWNHbyHJdDRB+AGx+IlaW5+D3ibtmvWTBzSPYcVSCbRYc45Q IsyKSULm2JR7tfdunzTpOJRswFsOCkLnwN9tTqGB5i0GoHxWUOgJ1bEZOrjMRO3ai1Ts YvBtE5omA2uJQwtcX4MUMNEcPxuS+oKDcnNQ0OdQ0WaMPwuPaW0nOC3ivh2QASMMF3p2 qa3yMtInCToh0kgxbgjelfxnH1JS0B3/2fJBpjWa2VrQQejuYjo5hn1BuLXENcKSavs6 3gBg== X-Gm-Message-State: AC+VfDxDGvNRYmGI36vqkPicxFlL7TvCFPIhCs0zhisRCt0QEE9r9871 Ci5tOQNlzyLs6KL91nfqLSbbVDZ+uBQ= X-Google-Smtp-Source: ACHHUZ501oAX8t/t8FRpUPo9PQ1t5ZgNv7pUhAgWb4tueLlYCBA4gtRhgJc8gsKy6W9D1EibQZVYd1jDC2Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a17:902:6a84:b0:1a6:4543:d295 with SMTP id n4-20020a1709026a8400b001a64543d295mr4657171plk.5.1682960128744; Mon, 01 May 2023 09:55:28 -0700 (PDT) Date: Mon, 1 May 2023 09:54:19 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-10-surenb@google.com> Subject: [PATCH 09/40] mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E3311C001C X-Rspam-User: X-Stat-Signature: xy54efsggm9tpid78p9fa46spgnmmc9s X-HE-Tag: 1682960129-441444 X-HE-Meta: U2FsdGVkX1/dMwVmdzzU2IQVEHIMCsStnKw5+eQBB0P503ZAZCJ4eHnq5gky3UqkTXLCPxmX96GI52jDqhBxnK4OvGy1rwmEI3QIN6Hs343ctKaJ3NmMkEwEAVspKuP7OjwT7xj1nptvqasXfvdM+Z4FqUNEyermWNCaLkTRZwP6JZzV2DirBW5nHCM666hmYXRWIDGjgxxCY9rwNQF5ppPP+nMavrKGDJCLVFaRhYXqjhFXC4BUUiMrlbmEAHke0r5AJmH/KDIS1txL8z6TVbQQjPmzyYt3MQ63YkyuhIbROEjPAHQ9Go+IgX5WKuyh/eG31aWLmwGCHXij9niUHnRoEaiFlrBeY05LxOexQS1Gv5/0RzO9M4j1cAo7LSo+28Mpc92iv214MUnrjUqAInAIyvt3WZmnkCPM6ShfHYQOTYJQ8w/l1i8bUULCTQ5ZX+Qgtkscf6IAAueyMeL/nNCL0DFPLw5p0oKJmnaSPMJs+pTGDoduojpL3LJGf+bWrTim24uePAEwt9L6cQ9BaRKewWmPCdeCDCy9fgowFzTu8Vaajjo11WTMY23zVrKb2z+n1HERMgq7jEl4TZEy2sASsMbl/35wY8G8r03k/Vn8SOF8YEk8WBxMjjA+bAW6LHnbkE0xzlmBASpjlWfWftuNPz8Jbed5sVYPnXsAT2NCVBVVgeqbXUR5ENjGZLcym4fpkf6XNKkdYJrQ7mZuk9+wck4nMAohpsxuZ+2uLoeUxfZTGAyepBzDfm/yfDHNRCR67VqYCi789MGJzqbtmIEiQ8EvyZbvuW2T8T5ENV5/zDrvBHsLLFlKR6IUlq2KLV3+9P0sD9JPVg5lDdPLfv+u5Ik0X+tqySVSe0d2paUGmvWKG/D0VGPTwe70gacXubaXh9R/ygwIe49cDubOa7pqlR/ICFGS7YK5OHS2Y/loCwmgMU0L3vrH0ZGSs0AuGZ/UPrAlwbX5CH+6xGF PRbNL25L R+NJ6/BSYr2atvBtg+FT1fHz8w/saN7eKDaqLFdZjK59pkrZ9on+Dk6iHQeNRk9BJ9t3NARWkx3tam7VoG2AWlk+5/Mvzg7Ug98ych8h9Bt9V2KVtjovju6FEdjdeWzL1VUXhLWBpKKPssRbWR3TRHdqbq4hpHw2JKXBC1L8V+FtXwQIGOXN8cRx0IgI9I+oZM64uz6BNveY2ZnbFYF4NbQTE8AhaTT+noPYpDzCTNKBMXYdBq+MP2f+rXLAhhogDw7mgiarzVYZ4TF3enUepWaizI5HAJWIWWxTS6hUEozC0Wpd2nR0y3qr57aaDPO6WXOB5xXycJUesC/WGQm5Vlq70AhrD7NjoHls/dnctGg+X18oUCdhCjKA/HvZFmkxmj35+WIer87d72V9cC84mbA+TRXku3Ttc16TwSGz8xJCGGqLYKG81cBMKE9c9kNApoqAZUCaSKHztogStf3KvuqgLmZbRSqUGceJHfFpkprinZiDeQjoS1I2VvCnbkvV3LgKvFFUhRfUnDnQh8uRbSNhct8vY5gBeXlc+4hqMr4BB4i4wh+wD4F/34hhnybNkxDuWvQoaZ8yFti8fmOpnItWPzuO3/p/mlNM6ZXv3k11DecJA9VJqhTLu1xZ3gZaTEvMdU6dbdStjQ2Uu+ZssQazd5ukzkrCKlDna3Yj1NZZsVeZLzJHImgjTOu0RlRoV6UjaCPmYY9wUeTgbzbbGHluA6sLiaYPBzJsI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce __GFP_NO_OBJ_EXT flag in order to prevent recursive allocations when allocating slabobj_ext on a slab. Signed-off-by: Suren Baghdasaryan --- include/linux/gfp_types.h | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 6583a58670c5..aab1959130f9 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -53,8 +53,13 @@ typedef unsigned int __bitwise gfp_t; #define ___GFP_SKIP_ZERO 0 #define ___GFP_SKIP_KASAN 0 #endif +#ifdef CONFIG_SLAB_OBJ_EXT +#define ___GFP_NO_OBJ_EXT 0x4000000u +#else +#define ___GFP_NO_OBJ_EXT 0 +#endif #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x4000000u +#define ___GFP_NOLOCKDEP 0x8000000u #else #define ___GFP_NOLOCKDEP 0 #endif @@ -99,12 +104,15 @@ typedef unsigned int __bitwise gfp_t; * node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. + * + * %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) #define __GFP_WRITE ((__force gfp_t)___GFP_WRITE) #define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) #define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE) #define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT) /** * DOC: Watermark modifiers @@ -249,7 +257,7 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (26 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP)) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** From patchwork Mon May 1 16:54:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227646 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA6FFC77B7C for ; Mon, 1 May 2023 16:55:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30BE090000D; Mon, 1 May 2023 12:55:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2958E900002; Mon, 1 May 2023 12:55:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1351A90000D; Mon, 1 May 2023 12:55:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F299D900002 for ; Mon, 1 May 2023 12:55:33 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D3192407A1 for ; Mon, 1 May 2023 16:55:33 +0000 (UTC) X-FDA: 80742287346.05.65E2146 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 0FE26C0022 for ; Mon, 1 May 2023 16:55:31 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=MVmPTJvn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3A-9PZAYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3A-9PZAYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gDL+lQo0P3G6fblFMKJonaRbqsrkB9ljveMgEFO8c80=; b=oDqCHryuebUkFs1Jey+RoFDW+l1ZL4+zN0oHawf34XlaFzUBEIFm+6dxG6La5Us3o2X+NT oBHDvgeDmma05/83VM/yti33zfNYbm/MNaCC0szj6d2029hsv7jUXk/2w4vEfjSRwffLFi iXFeMI+TsNQxAJgoTHXbAKSGCcsaM6U= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=MVmPTJvn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3A-9PZAYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3A-9PZAYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960132; a=rsa-sha256; cv=none; b=y5Rm7IF5CWgBVmDG5/SHzOMGnf/h/XsydMoECjcj96t68OSjvY6PEOuMaa3vIhSpw5l5tV w0UGI//VGFXvORkWe4OFpWzWf/2Zofir0q+FMAU6QaEsWYXKhUNoCN7xmcuux9tcDJjWlB eGWvAbGq1q3lOFifn/KOvBB/ALpF9uo= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-b9a8023ccf1so5355605276.2 for ; Mon, 01 May 2023 09:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960131; x=1685552131; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gDL+lQo0P3G6fblFMKJonaRbqsrkB9ljveMgEFO8c80=; b=MVmPTJvnt/d4OHnObDpLo5Yx/m0JxigNMHefGJUQ6YHCHiKWwXI+TPB1lVnlbOYOEt 5jgACI5oKYMCB7Pq7cz3X81+bpZcuEUKDUynk+/ZyYCZwjUk5TCfCarGFU4MMvcurlND oln4td3qiCopzaVDD/Q4rMLFlXyQJx4wDKAU0Wzgmky/qZcwJ14zhRRSwMOBlU9Kez/h x9gFamkwY5h9zdy2v/nUepQsOsBIbNt92VQg8iIvlxGC803m7Adm/vEjtGzf8sBaILXc DjUHk8NLlqyBasCJKVPDW64fh5SwGVhMkreauSlsOSNHTVu6NR6T/SSX4RnGRR6l1W62 EEqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960131; x=1685552131; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gDL+lQo0P3G6fblFMKJonaRbqsrkB9ljveMgEFO8c80=; b=Jy7D0St90Gz30qAq4+ahEYoeKBedelL7lGoqCj5GjISWv9Rg55+OTfmxdLO9JrnKt7 p6KREhtNu94wR1SclWD5OciBTOjKdcRolXLPJ1vJ4KpH8hRgKRKqC1BQxe52VIGrdzVe 0vCpv9KWoqBCrbcKJW1j5eUtVHouhxQ1QM7Y2IrC0cp8yNCp/Rs/72/0KMw5qY0FhAeb LnN11t3ik7lsTfPI4Ra0+33SoJcAlTKNngUfM/tHFbQM7VvxfgG1NT9GCI96d5DgaspW 69jENx9o/HyCqmja7uxs9ZBnH3WUpZRzJF75qs8T4DzJ69X/O4Iyg+AUoa3Jtu4vV/KU QsQg== X-Gm-Message-State: AC+VfDysKKlwvpsrxhGxD4AqZb9J0b4ldYLWfgaVx1iK1l57wauTL/rz XzNRLieAIJT1X9rYbTjvVZrOQC0Tp4g= X-Google-Smtp-Source: ACHHUZ6enP7kBREmhiQ0i0LYDuH3RuGEJuyGpBzYXGRy3FeK34ct4OWkfjl1R+bRIQUodsczhvp1FyVsBhk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:6b05:0:b0:b8b:f5fb:5986 with SMTP id g5-20020a256b05000000b00b8bf5fb5986mr8475612ybc.10.1682960131180; Mon, 01 May 2023 09:55:31 -0700 (PDT) Date: Mon, 1 May 2023 09:54:20 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-11-surenb@google.com> Subject: [PATCH 10/40] mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0FE26C0022 X-Stat-Signature: m7g88oq4cbyuwbs7mp1oai1yeokrtthi X-HE-Tag: 1682960131-314032 X-HE-Meta: U2FsdGVkX1/Wq7GrxCiHtHwhqk11KerTLoJxxD6liOv4BznchZmovtnZQL+Fd+TsOlHlkmZOLwcVWyK7FVz+dTPP3Bg3LzD60BEfyjZPk6bLRQNRv2YBLDURH/arYkseH+6zK4Oz2lD0pZsz8g837b6v0oogj7gMk65AIBU7yojqq28DEVrM+sBnHS51cVfkxD8+6kQy0dc9keXPL7U1CFsB3hhYQU10o/lFnCI2RuNhMUWCns5YFr0g6uD7ip7klIVzAz1n/RFBRYUpKiYJ+fPQbf/mEL1N1dhjNLapz1h5A5Vy4Xu7v5Fn8ICgHwQEDswHv0QPrTU3Y6ThTGgb8GzeoFTzf8uexpo6VFpwYvTGcKVTr9xoZQUXXyUdgIzXOaY4iVPq9LvyB8jDvjIFpWvJXnGnKUM33NTwc0h4JPhANZ44w0NDxUGWz3ialnmQrDPca096kbNa7d1F2lfHHl3KOeIZy5FsfIrvLp/5622MBr0DGrGrXVBzWWOKkOJF1UF68XMZJbV2n4wOnrMx+0X3ly5Ic9cbz6pUIAvcnzrZhNoEeGLyMh6VtVSrgIhyJjSasaZ48F9RjfX+Zvz2gRAMPGSSFxoijTutL0kqK8jA96kTUClC6SoO5GrzLV+vHxjQ1r0Rt7eXp+iP7NBkFGEgEuh/q4yjyRY9KbkVS74v7f80jbeAW6bPPMTVDfo5Z7lwsN4ryYnGsCTkx6gt3iz28wxI3Ga0Gnzq2Ztb0DKnrmFlaGvunLML7Z1klucbetFN3YN9AvKwmtaQ9mzlmYzozskKBqMNMPVgRf9mQsdI12J/gm6HhmsxeWzOv7DdMeuoCz7tB128O4Kq39T5dLkjDkyKfXsKqge4zvWFIZHkOS2PWotRdVkB435maExCcFHGLjObjmihfqyfQzQnCMd+ItMxbVhelY172URo25AxqVk+w22a/zNyP4rrVNcJ1lc99upsxW8SfLaJSzn 5mprrZxz 1sAnubUj3PDmed4dox8en6XIo6DPegaqIXnJmEtu5qCvUOiTyb1x9N+XfpwHIU+Ky4ZiZH5CyDI9MqMg2XVOahQhZuR96of+bWXFbVElIbEhDiDCSOcFcI/ti6dMECbFhgEy/lcCQmOClLb68aNPtRvVWr5/+4rHdTcgu+pVSBMPrNE7Y26jF6l+xtWBp7YUdfU1d4GmVzQlx3ph60AOMMbmrnbZnxXxINtw9Mjam+UlU1ulZ0ideXxLtC6lok8vbxUpH3REG/vKRPs5qZ2tl6WAzNexQx2OaUDCibVqwbzQrPe9suZ/KiFBAANot9Otayxq3uJbNc9Bm6syoHieqcca+6RjUGZHQuB3AMIfIsV3nSKhyllHvmEZDxaRu+YAfpaNadmPou5/sWgcnMkqS2DD9DLkoT0fCxcP/XqXbeqRkVPaxxExb652k9zzfkmPanj34aS8UjiouCtBRtf2pEppaTWkKc9xFbzYmCbNS/VlB8YGWUmei8bmnu6EI7kf7PPJRDVKbpu3Wrf845ZACakBx1OFotlID4Ii1Le3xOzVYb6xxJiXJuYS8cqrwUArC15BcGKyeed3chsTlxrsK7gVDjuuSS/kzY88h+4PqnDqB4ZAlI5JXbsxQaPkzTS6GhoajMN7uDkXqssotm+MquPJjeUVrltUEbNZaPbg+qRqAIbXwgyJGR9tm5IcDhpgjMEJbkTk7jdbuWxfyfOrEjZwgRUALcKPpgavG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Slab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like kmem_cache and kmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. Introduce SLAB_NO_OBJ_EXT slab flag to mark these caches and avoid creating extensions for objects allocated from these slabs. Signed-off-by: Suren Baghdasaryan --- include/linux/slab.h | 7 +++++++ mm/slab.c | 2 +- mm/slub.c | 5 +++-- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 6b3e155b70bf..99a146f3cedf 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -147,6 +147,13 @@ #endif #define SLAB_TEMPORARY SLAB_RECLAIM_ACCOUNT /* Objects are short-lived */ +#ifdef CONFIG_SLAB_OBJ_EXT +/* Slab created using create_boot_cache */ +#define SLAB_NO_OBJ_EXT ((slab_flags_t __force)0x20000000U) +#else +#define SLAB_NO_OBJ_EXT 0 +#endif + /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/mm/slab.c b/mm/slab.c index bb57f7fdbae1..ccc76f7455e9 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1232,7 +1232,7 @@ void __init kmem_cache_init(void) create_boot_cache(kmem_cache, "kmem_cache", offsetof(struct kmem_cache, node) + nr_node_ids * sizeof(struct kmem_cache_node *), - SLAB_HWCACHE_ALIGN, 0, 0); + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); list_add(&kmem_cache->list, &slab_caches); slab_state = PARTIAL; diff --git a/mm/slub.c b/mm/slub.c index c87628cd8a9a..507b71372ee4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5020,7 +5020,8 @@ void __init kmem_cache_init(void) node_set(node, slab_nodes); create_boot_cache(kmem_cache_node, "kmem_cache_node", - sizeof(struct kmem_cache_node), SLAB_HWCACHE_ALIGN, 0, 0); + sizeof(struct kmem_cache_node), + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI); @@ -5030,7 +5031,7 @@ void __init kmem_cache_init(void) create_boot_cache(kmem_cache, "kmem_cache", offsetof(struct kmem_cache, node) + nr_node_ids * sizeof(struct kmem_cache_node *), - SLAB_HWCACHE_ALIGN, 0, 0); + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); kmem_cache = bootstrap(&boot_kmem_cache); kmem_cache_node = bootstrap(&boot_kmem_cache_node); From patchwork Mon May 1 16:54:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDB0FC77B61 for ; Mon, 1 May 2023 16:55:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 702DC90000E; Mon, 1 May 2023 12:55:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BD4A900002; Mon, 1 May 2023 12:55:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52D2F90000E; Mon, 1 May 2023 12:55:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3C5B9900002 for ; Mon, 1 May 2023 12:55:36 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1312DADA56 for ; Mon, 1 May 2023 16:55:36 +0000 (UTC) X-FDA: 80742287472.13.133794C Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 4E647C001C for ; Mon, 1 May 2023 16:55:34 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=tJm7WR8A; spf=pass (imf28.hostedemail.com: domain of 3Be9PZAYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3Be9PZAYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960134; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AxHF9zQp9trScXtaAl0tt4NG3XQrBAxJQ/ayq1tkXak=; b=m16Dm6GwR6USZeffpRWpQCf/Bdm7415sFJHwA+PteJqcWsl6GGFXlf92JN1Ea+Ve9zYhHO A3B3TTZfGTY4bIm7p9URraVMMXm1AyK6xg1fooxqS+G/C81PdT4RxDoVPYu7s3HHHjTi8Y yfcOUpuviSKlVmLhAmViWF40HHIdBLE= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=tJm7WR8A; spf=pass (imf28.hostedemail.com: domain of 3Be9PZAYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3Be9PZAYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960134; a=rsa-sha256; cv=none; b=JqPtGn/ChRUkEfJjZ7+YZEsE/ya4MXhW6z4vAXnqDqT0iOztzN0XnMpa0UOfdGuY+iYbaV Ek9FHXr+9zUn+9q58lxLiolUjw0mUU5cV7QH/tFe18uMX18p+Mx9VHv81cETXetrMjIJu2 FrPIxdl8JHfgqT1pu6OS6H8yPk8d+6Q= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1aafc4aa1c6so5947875ad.1 for ; Mon, 01 May 2023 09:55:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960133; x=1685552133; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AxHF9zQp9trScXtaAl0tt4NG3XQrBAxJQ/ayq1tkXak=; b=tJm7WR8AdSF6O5qWHaKU1D/ysKeRzFc3FiZT54ezTvTsHezyhXNB2hdIvEgmyHbI2P coE7FK11G2KLLdp82iDrBRtaK1XJdqiYBBu7Bzgc0CjbScOB1xMrdceY1iUR8li4Lunu QxqPry7kkIxD8xjnoCiSp2i1c6RgrlFShVMk73jBCUlai0Pj6tzx2wQKCiGt42ycRlUY Q6iiOWoraxytw320styP6cKZ8CPR54fhZHUXwBTe18PxhhkTB2K2EgrVvGyA9dqorw8v 3z+UXLp0f4HuyIt3egDvW6I8g1tJRbXw06CphA32uVzlUNPHB3VM7B/5KkyjlKHh6bO/ vApw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960133; x=1685552133; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AxHF9zQp9trScXtaAl0tt4NG3XQrBAxJQ/ayq1tkXak=; b=AQndvSYQqfVZcaj9fezGhyTzkWDxup+ErTEv01Zk0/rmMwLS8tQFvvfpFSz2+MOUOu RjnQUu4HeB/4nGy4Lv6QToTwP+UBvaOgKsCwrkSL3MN4HlkJbtL/NerCmGaYacvhVi+J lHbNtobRzgrtH9W+uII46nvL1HfgIQeWJAKY30Z2+LA1eq7OHonzh15VN6MpSFKvFTg4 rbfK/jyBIKVcsAywZQQt8F2tV76paBYrbpEQuAetXhd/kEYdyVK65IPWWQKVdAX4l6TQ oijUISrdLqIU4M4N6I3dSPFL9CzgWuV9iyPgoLCEKZhqvRm/9u1JlIfO3MB03D+1Glp4 vh8w== X-Gm-Message-State: AC+VfDzBR0407M5R7Xqgbl4jxwui9otATgoaGDyq6tu+z8+7GSV1ELd9 Bu+dj7buy05sM8GbBEaKGIogyN+/20A= X-Google-Smtp-Source: ACHHUZ40vYgpSBKgnsqUKG4z7dBiWLR/NY4OPgzSeG2TkOgPzabfXY1XoiHkCzsuVnUnxnzRXGzvUa1mMg8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a17:902:ecc5:b0:1a6:6bdb:b542 with SMTP id a5-20020a170902ecc500b001a66bdbb542mr4742101plh.9.1682960133566; Mon, 01 May 2023 09:55:33 -0700 (PDT) Date: Mon, 1 May 2023 09:54:21 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-12-surenb@google.com> Subject: [PATCH 11/40] mm: prevent slabobj_ext allocations for slabobj_ext and kmem_cache objects From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4E647C001C X-Rspam-User: X-Stat-Signature: bdgt39e848nxr4dydwih6x8u6snwph9g X-HE-Tag: 1682960134-585161 X-HE-Meta: U2FsdGVkX19wbCLqWFdsPRAWAXasZRxH9iT53eIPNI7c2iELGy0yXZc4z5fEnoDmGYcDal5srJpP5VzXf0yuKBsOQstrKpRWgYH+t52cAfaUus+tQKl2AEGRY8acDDjuMlvr1WeEw8mHUR1vmAE1Do1jYA7ICjsZILWTyHNDh0+XeVceAaEJxt1zow9Cr9rxp6VkDFuJMLTA96fDmFXbpbVaeuD2sCfRN//iXOG4oBwmFkLluuLBDSE+IQh9ZcpLF/BhgPO1hrSMrKQ1d5hj+tNrEfpS8H/FC1K1cuPhss+ABz9yqNFthPaYnkz6LRVAqsJxZwOgIYbLXM4gn0G3u9d2hhheFREa52jQvpYQIjBpQfuIKnx8R6KFPSs+Gr3xWxyoN8j/QRnltApi4xzedUhwSA3y9ywPdZG3vbjksotObAF0Ow4ZnwJtQ8ulzHDg5wDCy0fNh1kcdDOPfequfIGKO86qKPk8StE+gwW6nosUSfgR8X8mLw3Dfw1LFYkC8YHkgjds5tI1K7BpX8mBKLxh5Sd/wfeCPbaQywqGZg5Vodz5nKnrX3EKDc5AherSoIaulTxTdSTMY/U+z+tRpFMbaWX5UWXn8yv2ayCm+TWlG4OymAuJKFMMm9l3YSs0P5WpWaeYFEbwS6utL/I4hS+wCic+qSfraaUKJ/i1yrBXL4PwSvoQ8VpLbKZN5LLyNb2pLfaeCkIZr8iuZil5yK/tTK4x0VoW5VbfwC7O/0X8Y2PGvfoT84ddptFRMzgKnxtZqr4eN5hmfPxLnEqjR0tE5AadVGhRW7EDqnA03ArDtnU5hvwSYTdMtccHfXPUGCiWAwHQ8Clmik9IQY7cNrubrTdC+x4dTY96Ve53dblvliespXFnnk3eqQx8wpuSmVvhiDFv/vbg+cJt3O2aotZXP6X6sd1gPGwN1fwDIHrzKPqI2SoXNpPY3UwqCTRez6c+/7qfSshFDIyUpk8 vd4Dp1sI TTLEUEAhDXxbejb3lrX4kGyoLTTZHjtnFCLEFrWMkesOpfhSm0qgitA4NcSq5Tt4euaF752it1o6I0npwOVJAOdmSeTvjLc44tAWzlf8NrQPzSRow4qJLxRBlq9XZ3LnxiBl9vrVlrVmPecLe7E9+/gfpTQa9Q+M5cllikTNAoS+cxqjiEFMcOZrRCLYcfdCv406ryTAUibchayZO1k1t2TaCQxiAjefwTP1i6US7F5FYPkmI7Q5JgJcL67rq1W11hngwt3tDUp097ewEYBe3LRiHVrpjUIU44FkT/++9hzGrSWEe1sw1dXr8fXSG9F3TMG4T3YMOZQNjRA/SWSrPxZyI/oriNI/BBSuGq7T9nV+DFo4buJQuHVutlbUPLIIf1E+ihD6xdzz4EsGk18PsGToCynN2hu8HG2E1E98dYTOZRs8PEL3LEkIzKL2aMrXoJ0R6mtvzbrumWBguMKvBY3Rrjpx04wXfo/no3gO/vFgzqzA80Ap7vuAOAUgd0aA/GArpr45tddL/0j5wGbyEhbjRZuWVUTLZZ/ROqRnTm0mxGLoBohViuUheh25Pm+n8Hro0HHZBtNhoyQqaAIXau2lhAusvl7HUwBan0Kj6bmVsJ76Z4hbbmJ4xn48K8cwvWck5ACvHo6cx2vDHbSvdFZx/VkN5wvtJaB/QjZAvR2/q/xqislXzePDFNjDMFndNs5kNvNY/pFqI28BqhlB93usIv3bjjepflklq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use __GFP_NO_OBJ_EXT to prevent recursions when allocating slabobj_ext objects. Also prevent slabobj_ext allocations for kmem_cache objects. Signed-off-by: Suren Baghdasaryan --- mm/slab.h | 6 ++++++ mm/slab_common.c | 2 ++ 2 files changed, 8 insertions(+) diff --git a/mm/slab.h b/mm/slab.h index 25d14b3a7280..b1c22dc87047 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -450,6 +450,12 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) if (!need_slab_obj_ext()) return NULL; + if (s->flags & SLAB_NO_OBJ_EXT) + return NULL; + + if (flags & __GFP_NO_OBJ_EXT) + return NULL; + slab = virt_to_slab(p); if (!slab_obj_exts(slab) && WARN(alloc_slab_obj_exts(slab, s, flags, false), diff --git a/mm/slab_common.c b/mm/slab_common.c index f11cc072b01e..42777d66d0e3 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -220,6 +220,8 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, void *vec; gfp &= ~OBJCGS_CLEAR_MASK; + /* Prevent recursive extension vector allocation */ + gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); if (!vec) From patchwork Mon May 1 16:54:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE9C0C77B61 for ; Mon, 1 May 2023 16:55:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 57BF990000F; Mon, 1 May 2023 12:55:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 52C16900002; Mon, 1 May 2023 12:55:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F5FA90000F; Mon, 1 May 2023 12:55:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2A2D0900002 for ; Mon, 1 May 2023 12:55:39 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EACC316073E for ; Mon, 1 May 2023 16:55:38 +0000 (UTC) X-FDA: 80742287556.13.BE512FA Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf04.hostedemail.com (Postfix) with ESMTP id 2E4BF40007 for ; Mon, 1 May 2023 16:55:36 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lxWpRSi2; spf=pass (imf04.hostedemail.com: domain of 3B-9PZAYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3B-9PZAYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960137; a=rsa-sha256; cv=none; b=kwKQl2D05zlcBrJ43VYXiS1sDxcJ8fnaMVO5dTNU06Y1kzKVvAvj3KFJ1zcWhvSM55hUY8 MW+d7whasHUajp/ZkyftTT97eihH/PXg93g8ADG2upof+LVBDR4pv5yq7HZ6tboyjcGcS2 yCq4azn6byQdeyuLyAnIX1Nmld+50hI= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lxWpRSi2; spf=pass (imf04.hostedemail.com: domain of 3B-9PZAYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3B-9PZAYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960137; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xg9JlY7/fjt8XCrAgZEcEBdLHf1ezLwh/Q099RdQeBg=; b=gSowKm8170zYHG+RbakbDBv3hOAsE3hbxSE6xeKqNhRQ8mBCwB7Cn7e3Oup8rI7Ykp9JNC Awik3JM9EqE/A3zGq2JmKujUKepwnB3qc43U+YG96/dY3M9xx1c+sEUaO/ut+XScQZdFd/ wH8Nc+Y5az8D8zWpRItHovh3gXZ+18E= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-24df9b0ed7aso1439816a91.3 for ; Mon, 01 May 2023 09:55:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960136; x=1685552136; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xg9JlY7/fjt8XCrAgZEcEBdLHf1ezLwh/Q099RdQeBg=; b=lxWpRSi2RW8T5MQ6rUDBO360BpQGLeAVDuvFbZTW2oi6i0pP4Ylls+pWo4X0mmDlZ4 ZivjAV0YmO5+Csa7KFdawNwZlR87MIFCY0KvaMmh23wbSx/Zqw9mWSmtCW/DFws1pL93 HinphsKaTEMZsFj4FnMn87Gmq03vSk5t2aEjAb2xkm1oUZHHiNVFH5GCIuTzn5brrAqi 57iYlmpPjz1nsWZstv5H0Wf3fwPUmwqs17xOGhYG6xNhITzJ07bTu6Ucjsolv2WF8BDd fpFmt5mSOlDgy/NcylrZlRdpPNgsEH1oRDPs7B+h6qorwNaMNm9hhTPxfiNkqgalis48 xWGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960136; x=1685552136; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xg9JlY7/fjt8XCrAgZEcEBdLHf1ezLwh/Q099RdQeBg=; b=K0XPoikrXYkI+hbKmVYhDBMgjasj4yhDP+aYAToWAIgOYYxYcRZjNRQKrxgmul2dhr ChlttfKwTgXlEzpCJPeE9OVLR958e+YL2m5vIhreCz3UzXQcERJf2bG9en5TIj7FDxRG HFJgISw/FLIKrxt6smbYOmT/dgznuxG3pKwmou0DGBc4mIi084spDl0wTeE63kHHBBEp YmX+UsWnDbafcQ5HqBU2+vXBdnLcxgztO6KhaO6oCDCDUI1EJfM1vlFTZLlHMehcBtBD z4L9KFtpBzXAr8H3k89SvXYxExqZtzTarDZEIM7CIZpKVqKVYHihnGYKFOzAXXib+c9H Srlg== X-Gm-Message-State: AC+VfDydhd9ySZIJMBU5Ed7TUxiv0gS3KCB6my19EuugSAsJ7pUR0g6A JoW5NOi9Hhwc4xq5wcrHA8llcAA27gU= X-Google-Smtp-Source: ACHHUZ6YgiTbOdoAI7RvpBgPX+gkOC970dcPRGZpFE4ZhJnSWmLnFtQB4uP9hU7pUIktNp1PkK/6Tg72Th0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a17:90a:2d7:b0:247:5ce:5bd7 with SMTP id d23-20020a17090a02d700b0024705ce5bd7mr3861119pjd.0.1682960135879; Mon, 01 May 2023 09:55:35 -0700 (PDT) Date: Mon, 1 May 2023 09:54:22 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-13-surenb@google.com> Subject: [PATCH 12/40] slab: objext: introduce objext_flags as extension to page_memcg_data_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: 2E4BF40007 X-Rspamd-Server: rspam01 X-Stat-Signature: y7r3p4ifmebi7w5zeekwzdqx9yfhyhi1 X-HE-Tag: 1682960136-793586 X-HE-Meta: U2FsdGVkX1/m2ylWhyWUvxbHm8idcWL4/9XFJD2A8EdiSwNtRhDqoCqiFufnYlDxwjHtlLldWrXtN+pV7oCIMVMH1Yf71ZpLKfJngL7ghGofAyZmQywmhbcxs7G0k8IO1ClGEjJmCZwv9rDS0jLppDYf6YyRD8nyG6HJMwBvAl+P5y36dgCg0bMPwLBBgyhwy9bA4mF/aRtxYLefsngnCd+rrq3zNwB68UTOOf5rCIvNGaD8GbcxdFV905Vd76GLMYeZStV6EhGH3se3foZZaSZB8gMiqrxRIalQCJZKuPUeVqBSPC49e1Ay28VUwq06O1hY5aMu8fp7pnJZTqUAVuhwcU6us5uyMte7UxxQ4cqBiluqWCVU5k7rIZWyUzVdf4J4QEjMckrQ6LyDdr+rV5e3ErNcd2vf8NVUINNwuatjaeD4O9fw8qWi/x61N5Z/9Iyb483+PQ/H8LVIA8ET9j1olP46PelqZj33xH3TQIMAoL6jDOknwMWGslZmSRT42zNdVM21mpDGLe144UAFMYyXWR7jqad4IWTU4KLDKdNJe6Pq00xOy1S3xDYgQC10mdCJik9DhgczuwfgeA6Qq67NK6Nhr5DwBNXsLcGwMXul/39VywaikvG/rk7XnoLfwa0VqccM08dz67+0ucGnvGPXEgfSdqUZ3eyNi6DVjDzayhR5JVtyNimtYIBEy33xlRihfwQ4Ig1EB2uJlzIrg4zkG4qoXvD9IN3pJHxOwT1wv7DQ7+4yzRWbgD482xsVFBH4NNXT10Y+BbIJZShRv1pnjqYs16fk8GlFbKZv3lT6zxOtz4cQ9wYfvze6FDxopr45Uy8AWhj6IA8rv+SJhWir5KhVykOCeB/lwItRVOOg5yBhDzPx38l7Bm9ZStDHOvyegOuLPvuyipmyKJAyWa+Mz1xrkPRweoUpZhwQ9bb4iTfZxqrc3CdGI52zQnUug0GBeCaH4z5MM4Kzvsn GgMSdCeD 9DVhffxAaNrTSI6gTF7GEenQMqGodq/x9wlqC4I7z/7uOoFiDY/7WFyTX7C2AR9QeBG1smBOZFpf+WqkRTd3t9yMRdIbz/8U6r80z66TeDNOeLFb3q1lycTeeWwXeWwgCWroYtCJP/N6t8+rEpV71+yimu897TTFGSQZMibtdILOiBih3M3m90HOp+YBgYl8u8dC3TII7A/XGnOwKkkAnI5+Z/jqdqbnoDvlThyydNy6UDEVMJi6b8o3KBIX/KSZ5p/MH00wMEGmQVkOTufYIU46UY8IL/k/KNVJTzmEnxggsH1P9rKCYHump7dX1PQP9671LE2r2jz0UEMvdO/u5BMO3Gsga4F+VCp4eYqcBvrfQ/WFnKA3NFdwr8+GcVmCS5+psYLWdW8sWyW3yCKVNzjgVQj96pO3xf/dWMUqmjKsZnw4ppiL6L/BOcV75zw0OarggQF8IfvozXcOQyzDHKabPy0cvUWX2xHDVV3jVVlVEHFc7qZPPrNcU1xDlQgCkwMBgJ1vODF6e+i5Ud/vG+kcongl1SzYZu59Qbx2tYWuuwIBctQlzNmhW7iWOw4ACf6NVZvCYEmQNEZjnhBJoZyxpogS+klyQYHy/laU74Hldfka18B6uIbTnrh/HdEUSyRVhjQlsxUxDUuOM9qxTsVQoJ3SYGUgbOkGRW/C06em6cC/7V36JXpWsTIzHPnNdKPoEwiE1XLQ32vi7VDpw4xJtfNPwKIq1F0Y/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce objext_flags to store additional objext flags unrelated to memcg. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 29 ++++++++++++++++++++++------- mm/slab.h | 4 +--- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b9fd9732a52b..5e2da63c525f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -347,7 +347,22 @@ enum page_memcg_data_flags { __NR_MEMCG_DATA_FLAGS = (1UL << 2), }; -#define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +#define __FIRST_OBJEXT_FLAG __NR_MEMCG_DATA_FLAGS + +#else /* CONFIG_MEMCG */ + +#define __FIRST_OBJEXT_FLAG (1UL << 0) + +#endif /* CONFIG_MEMCG */ + +enum objext_flags { + /* the next bit after the last actual flag */ + __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, +}; + +#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) + +#ifdef CONFIG_MEMCG static inline bool folio_memcg_kmem(struct folio *folio); @@ -381,7 +396,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -402,7 +417,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); - return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -459,11 +474,11 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -502,11 +517,11 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } static inline struct mem_cgroup *page_memcg_check(struct page *page) diff --git a/mm/slab.h b/mm/slab.h index b1c22dc87047..bec202bdcfb8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -409,10 +409,8 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) slab_page(slab)); VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); -#else - return (struct slabobj_ext *)obj_exts; #endif + return (struct slabobj_ext *)(obj_exts & ~OBJEXTS_FLAGS_MASK); } int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, From patchwork Mon May 1 16:54:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCF69C77B7C for ; Mon, 1 May 2023 16:55:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A490900010; Mon, 1 May 2023 12:55:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 52BF2900002; Mon, 1 May 2023 12:55:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A60E900010; Mon, 1 May 2023 12:55:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2C7CA900002 for ; Mon, 1 May 2023 12:55:41 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EBC34120814 for ; Mon, 1 May 2023 16:55:40 +0000 (UTC) X-FDA: 80742287640.19.BA413BB Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 33179180020 for ; Mon, 1 May 2023 16:55:39 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1K0fFEAK; spf=pass (imf16.hostedemail.com: domain of 3Cu9PZAYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Cu9PZAYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960139; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=esQmYNSNwAFEEht8P+eJjy3027KPfc8B0JiLBlMM/Bo=; b=7lzpd/aMtEWnWd5HOMveE7FDL+TYrv4qnndXT1uVaQvsK2dHCwkzSiZI0kyIY/8X5qp6fZ x2SEl5bV7f/o42XZ3ucxi911kukiufk5ZlzbEmEYHkHu0CUEx53E+dOYc1ZD8uOgVeH5Wf /ni5q/OCNyGTBCCu06hPeI82TAgOUsc= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1K0fFEAK; spf=pass (imf16.hostedemail.com: domain of 3Cu9PZAYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Cu9PZAYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960139; a=rsa-sha256; cv=none; b=CIt8qbYsh1maJf6yJw81Qw9IcUyOHqUWwvG1Qp80z5rG37+KOQgMRkQAWTig4Z44ZxjGXs MJPfZ0CDwus/xmS0Z6EiRKA/jU6Gb2C0yUmXW5rNBcLRiZJA03/4exzX205XOHmaiKitTu 5GJlrpfgk9Dtm9bMMTLX2O60AQxT2WI= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-b9a77926afbso5336380276.3 for ; Mon, 01 May 2023 09:55:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960138; x=1685552138; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=esQmYNSNwAFEEht8P+eJjy3027KPfc8B0JiLBlMM/Bo=; b=1K0fFEAK+AaTcTrhRe/8a98Aa8YFb/wagQhTInABD8mCONGKSRgdFrxtZCXhoikIrD 95LbVzIsXi3bINaTtEnZF9G22EF21tdKrkz425iIWjDwgf5ia/D5RPk4ld7MvQ2bZrHL zTPwIsZxtdpBgp2DZGIo7+RgNrWbsBTdHAdG+30Neb5Pvht487JVGA5EbtFhDduMq7HO ShzwvYcyvz3j9I+l3PwqAphCFJcDaU5zq6rGvI+RjzLbAJs5GZMoaQi5lWkbfdI+rVbA ocsPKP0FHdN7fG1jB6JiNezvv7iCPM0MjKja2ICr+xbGSCsCD3+3JKMB5QLLD+jX5DbM FQSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960138; x=1685552138; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=esQmYNSNwAFEEht8P+eJjy3027KPfc8B0JiLBlMM/Bo=; b=lrFJtWVIBnr6ELmNTK06iu0ksvTJa2M3oncFRR4UO4O92RAYsvm78JQmZciTh/T+/k mqmu5LacNEOCjvXVqVdmXdJq3rVSt+02XEume5zQq51lrWqqDZVesEPqj0f59t1j4JjU XJ6c9cQmiv4hDxt1tW5FhUdorn0I2oVlZa98jbAKyK1dQgHnpwRmfeHLF/IxnGv8XZGl ErdK+HLGWhYhZsC7OUEmK3cnjft3U51e8ltcPcV8eOinP0ZDrUgf11m1DnS1700Hsz0D uMsGAXiu8+FCTyyiBIizqT5RnRHHDfz3r07TPoYVuDVlGpPInTYIR+WNvlQNMMYQep7H p2aA== X-Gm-Message-State: AC+VfDyR9KRmQJVG+Jl13TugqcwQO+AHrPZvVtO5dvSL+y+eEBOP3YaZ 8BPwVtLPpjeSh1nEbDBw35Wa0mloiM8= X-Google-Smtp-Source: ACHHUZ5PU+HAQ/Z79Go9ruK3jLDijCoDZh/O68HGN1mUxbgOcPOxNdjm/ucjU0E7fPKkrMRK65Bl/NGds+o= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:c00b:0:b0:b99:4887:c736 with SMTP id c11-20020a25c00b000000b00b994887c736mr8510714ybf.3.1682960138280; Mon, 01 May 2023 09:55:38 -0700 (PDT) Date: Mon, 1 May 2023 09:54:23 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-14-surenb@google.com> Subject: [PATCH 13/40] lib: code tagging framework From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 33179180020 X-Stat-Signature: ernk74yww164h1fgy1xabymbt6yookn7 X-HE-Tag: 1682960139-846808 X-HE-Meta: U2FsdGVkX1+i6jxaLhiEcDAy2o6I0E4OLaZ6sg02yUrGR51tU1am1t5F50gPbFr7ryYqtUDNv+JkwlC+6V8fvb63ooNKM41b2dXovPK+KlBLZm66QRqBOyqmVgSMLVADEJTTJUWI/k9hQeYU1KnkQinxOhz9Fnwc5hNYQU7Vw72fKXJpQPcxN9R2e+rHoFKPA9KEp7+e2sW1SiEz+xks+8/bCpY3xITuRyt8LVhNYk0hlTG15xArdltqVuL9UOWYGA0n5MWaPjsmBXipWQFZ4hnOBjXEAOFP409cVaYQ9zPq/9BEZvY8tHwN6MGA7il8RMuktzABXNOv7jAytFECliTvQo0xeyclxmLmTWxbi9rRwRqaCRnxwHI8oAmyA+0Fa418MemXGKfUq/h1i2aQodh1T81UXs5aV5LWOujuN0GidbI7sENGdmNkVy51LtMlK5aKl+p+5+pduHEDSqYqV4KaO7aZESzm6aTxmaICAqD6qccqw9m3ujeaeZbRpsf9EtxEaEYQNJOujAgDqbsiTTSgNk6ghzcqXAPhwg4f7dMoJsOG5KlDEHQ0q4MSPASfJONGH37+EjL8R55pZSEREV8JmxNYy8EgX3Y6RH6TEMKjQeYQHyOaUkcgRyd6wSdBipn543LZDBDbTjpeJEdaS6ATpLYrVfHukZZTyCi0d+IOhwsTuW1fqy8e0h19k16jlbqJCKhAJ00io50q/QSoYdmkGowmqhqwubGw6ZNeAd36TKr4bNOUKKil9Xa0rGMNWkPNSUuYAlpPrXoLsanXPNnyCq22G6fdud5QBMAmTo2GAMBsnp3sAB0gotw6slqE2rsgWcztIR3ZSHBT65UmHSjuhagJ52jt8QcKNVIs5hma1CEEhFR0eNgp7umTb40RDW5Ki8pM+Uq/u3/SalK0B/FsfSg5T6zqk+HHGefsS3kLi96uT0w2IJ5tCTDdUpBGxw4IoEWX83HFZcekTKU M477cBLh 8/SVvH0qgtAcnSwJ1m6lbTgO4vdaPZupf1aBLLqFggMXwQj78BnCqnhF9NcAyrycPmRBMoi1FHC1cl7dm/H6qTD/Gjgons1Q8yVDCIQGmnQeueHZQM+Q5pDKT7rseog6TM1/hJchLDssGTevnCBLKMDWJRgwh+f9I7Jlmp1zI54GUck2MF/+3DsqFsFtKVZ3wYucABJ1GsvAaQZPFl7QHahjQTmSk6n9G1b068XHNE9FnTcAbTXWxKzbXZFcBRVg3rLzNdvtrRJfBUkDYQJT895f3ikjH/kG8sJr9N+waHPbipTd0BW8gtha74fyHaT8iTd67tyqp8aUglbx25TyUuEavN2HW68i9hN4iJQ9RHHm4Ho3DSkSSw8amS26vQReHtNJA0A2syvrtH58eqDZO9vDpWjcnxrtJSNNmVzQgCZc6NoM9jxii8WfuO0hLCoL1UsIyMbBiSKkgUT4WOfQzYeQlSZpKGhybjaHZK0tYyj/oJVPVuEj+MnBJ5FThWK2CLgfx+CvKLEdMJ0qibv4MsH0sAfQie/FFDEAfnQ9xsHg2lNLA8g6skWT2FtrsQdlOftD/gUI6NNsdUBuaSSct0piPcd9b/mfMPfsnfqazhL+tctJGzUUAKv6i9cDMI85sE67HDPJ/CJeaasPZpoBc4/g204WBFZltoM3PrijlUAy1Hq4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add basic infrastructure to support code tagging which stores tag common information consisting of the module name, function, file name and line number. Provide functions to register a new code tag type and navigate between code tags. Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 71 ++++++++++++++ lib/Kconfig.debug | 4 + lib/Makefile | 1 + lib/codetag.c | 199 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 275 insertions(+) create mode 100644 include/linux/codetag.h create mode 100644 lib/codetag.c diff --git a/include/linux/codetag.h b/include/linux/codetag.h new file mode 100644 index 000000000000..a9d7adecc2a5 --- /dev/null +++ b/include/linux/codetag.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * code tagging framework + */ +#ifndef _LINUX_CODETAG_H +#define _LINUX_CODETAG_H + +#include + +struct codetag_iterator; +struct codetag_type; +struct seq_buf; +struct module; + +/* + * An instance of this structure is created in a special ELF section at every + * code location being tagged. At runtime, the special section is treated as + * an array of these. + */ +struct codetag { + unsigned int flags; /* used in later patches */ + unsigned int lineno; + const char *modname; + const char *function; + const char *filename; +} __aligned(8); + +union codetag_ref { + struct codetag *ct; +}; + +struct codetag_range { + struct codetag *start; + struct codetag *stop; +}; + +struct codetag_module { + struct module *mod; + struct codetag_range range; +}; + +struct codetag_type_desc { + const char *section; + size_t tag_size; +}; + +struct codetag_iterator { + struct codetag_type *cttype; + struct codetag_module *cmod; + unsigned long mod_id; + struct codetag *ct; +}; + +#define CODE_TAG_INIT { \ + .modname = KBUILD_MODNAME, \ + .function = __func__, \ + .filename = __FILE__, \ + .lineno = __LINE__, \ + .flags = 0, \ +} + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); +struct codetag *codetag_next_ct(struct codetag_iterator *iter); + +void codetag_to_text(struct seq_buf *out, struct codetag *ct); + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc); + +#endif /* _LINUX_CODETAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ce51d4dc6803..5078da7d3ffb 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -957,6 +957,10 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". +config CODE_TAGGING + bool + select KALLSYMS + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 293a0858a3f8..28d70ecf2976 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -228,6 +228,7 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ of-reconfig-notifier-error-inject.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +obj-$(CONFIG_CODE_TAGGING) += codetag.o lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/codetag.c b/lib/codetag.c new file mode 100644 index 000000000000..7708f8388e55 --- /dev/null +++ b/lib/codetag.c @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include + +struct codetag_type { + struct list_head link; + unsigned int count; + struct idr mod_idr; + struct rw_semaphore mod_lock; /* protects mod_idr */ + struct codetag_type_desc desc; +}; + +static DEFINE_MUTEX(codetag_lock); +static LIST_HEAD(codetag_types); + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock) +{ + if (lock) + down_read(&cttype->mod_lock); + else + up_read(&cttype->mod_lock); +} + +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) +{ + struct codetag_iterator iter = { + .cttype = cttype, + .cmod = NULL, + .mod_id = 0, + .ct = NULL, + }; + + return iter; +} + +static inline struct codetag *get_first_module_ct(struct codetag_module *cmod) +{ + return cmod->range.start < cmod->range.stop ? cmod->range.start : NULL; +} + +static inline +struct codetag *get_next_module_ct(struct codetag_iterator *iter) +{ + struct codetag *res = (struct codetag *) + ((char *)iter->ct + iter->cttype->desc.tag_size); + + return res < iter->cmod->range.stop ? res : NULL; +} + +struct codetag *codetag_next_ct(struct codetag_iterator *iter) +{ + struct codetag_type *cttype = iter->cttype; + struct codetag_module *cmod; + struct codetag *ct; + + lockdep_assert_held(&cttype->mod_lock); + + if (unlikely(idr_is_empty(&cttype->mod_idr))) + return NULL; + + ct = NULL; + while (true) { + cmod = idr_find(&cttype->mod_idr, iter->mod_id); + + /* If module was removed move to the next one */ + if (!cmod) + cmod = idr_get_next_ul(&cttype->mod_idr, + &iter->mod_id); + + /* Exit if no more modules */ + if (!cmod) + break; + + if (cmod != iter->cmod) { + iter->cmod = cmod; + ct = get_first_module_ct(cmod); + } else + ct = get_next_module_ct(iter); + + if (ct) + break; + + iter->mod_id++; + } + + iter->ct = ct; + return ct; +} + +void codetag_to_text(struct seq_buf *out, struct codetag *ct) +{ + seq_buf_printf(out, "%s:%u module:%s func:%s", + ct->filename, ct->lineno, + ct->modname, ct->function); +} + +static inline size_t range_size(const struct codetag_type *cttype, + const struct codetag_range *range) +{ + return ((char *)range->stop - (char *)range->start) / + cttype->desc.tag_size; +} + +static void *get_symbol(struct module *mod, const char *prefix, const char *name) +{ + char buf[64]; + int res; + + res = snprintf(buf, sizeof(buf), "%s%s", prefix, name); + if (WARN_ON(res < 1 || res > sizeof(buf))) + return NULL; + + return mod ? + (void *)find_kallsyms_symbol_value(mod, buf) : + (void *)kallsyms_lookup_name(buf); +} + +static struct codetag_range get_section_range(struct module *mod, + const char *section) +{ + return (struct codetag_range) { + get_symbol(mod, "__start_", section), + get_symbol(mod, "__stop_", section), + }; +} + +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) +{ + struct codetag_range range; + struct codetag_module *cmod; + int err; + + range = get_section_range(mod, cttype->desc.section); + if (!range.start || !range.stop) { + pr_warn("Failed to load code tags of type %s from the module %s\n", + cttype->desc.section, + mod ? mod->name : "(built-in)"); + return -EINVAL; + } + + /* Ignore empty ranges */ + if (range.start == range.stop) + return 0; + + BUG_ON(range.start > range.stop); + + cmod = kmalloc(sizeof(*cmod), GFP_KERNEL); + if (unlikely(!cmod)) + return -ENOMEM; + + cmod->mod = mod; + cmod->range = range; + + down_write(&cttype->mod_lock); + err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); + if (err >= 0) + cttype->count += range_size(cttype, &range); + up_write(&cttype->mod_lock); + + if (err < 0) { + kfree(cmod); + return err; + } + + return 0; +} + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc) +{ + struct codetag_type *cttype; + int err; + + BUG_ON(desc->tag_size <= 0); + + cttype = kzalloc(sizeof(*cttype), GFP_KERNEL); + if (unlikely(!cttype)) + return ERR_PTR(-ENOMEM); + + cttype->desc = *desc; + idr_init(&cttype->mod_idr); + init_rwsem(&cttype->mod_lock); + + err = codetag_module_init(cttype, NULL); + if (unlikely(err)) { + kfree(cttype); + return ERR_PTR(err); + } + + mutex_lock(&codetag_lock); + list_add_tail(&cttype->link, &codetag_types); + mutex_unlock(&codetag_lock); + + return cttype; +} From patchwork Mon May 1 16:54:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3A7CC77B61 for ; Mon, 1 May 2023 17:02:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37D2E90000A; Mon, 1 May 2023 13:02:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32DF0900002; Mon, 1 May 2023 13:02:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F57690000A; Mon, 1 May 2023 13:02:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0F0E7900002 for ; Mon, 1 May 2023 13:02:15 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BDD091C72A9 for ; Mon, 1 May 2023 17:02:14 +0000 (UTC) X-FDA: 80742304188.26.AE8CF72 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf27.hostedemail.com (Postfix) with ESMTP id 531274002E for ; Mon, 1 May 2023 17:02:12 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ytR2UEYh; spf=pass (imf27.hostedemail.com: domain of 3DO9PZAYKCFMDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3DO9PZAYKCFMDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fsyg89/vfc29JJa9GS5YhCGnLnEPc8LoOfUR2xEk3nU=; b=J11KTYqNKclOTfGr1fitpkhFeGpYyErqUlJ66Ww374QWkikl+Rhesx5llbQ75eXKEoUw32 4H1W6/1OlprOdIEHFUzWyu/oZ/KdP63RDHhgAZwPWANqt9+lSW8GmNBg+Es18bjaSv40e0 SHyfZJrpDqsJIV2n1NlKTsnp7sQvuQ4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ytR2UEYh; spf=pass (imf27.hostedemail.com: domain of 3DO9PZAYKCFMDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3DO9PZAYKCFMDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960532; a=rsa-sha256; cv=none; b=anfSO98540MBkH5afiGUXw1A2xcOf+FgjRyjf6w3F8DcqLUw0C5wSgqBV4CU/caC+NJS07 otSfoO/9I8rQe7cpAJdBmupo6k9e+6+gWHy3YN+I5CNcjg7YPiAG/UILPnoudBPKePDm5+ e51Wb/pon6r854kDH0f+RX5OHZi9dng= Received: by mail-ua1-f73.google.com with SMTP id a1e0cc1a2514c-77aada1df61so25928337241.1 for ; Mon, 01 May 2023 10:02:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960531; x=1685552531; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Fsyg89/vfc29JJa9GS5YhCGnLnEPc8LoOfUR2xEk3nU=; b=ytR2UEYhuAD+YRVQWgUNv88SHnd2ynRKseoU78an9eCwuDcCW2rMe12kyedJx3/bUr 8mP8kQhLhNKKNpawueKHtOKyoqY1CMn375J69k0AA1tp/1rphK+jnt9ueF5P13LzahOU y2Ah0HOFnLnZp0Dc8RXuaw3q9Go3obH7D10CJODfv9q33ibAT2zvSNGbqQVMC0kInAQq UzghzRv+xC2U3ILRtmdjoCzixF3zu6VZBbkYp8zs89knY1zIe+um7602sn4pDs7+R8j7 tQSr3D3lf0d+7ugQL+9mZDuYlDHrbRse6zGFtm4RR5gEaU3/0UuAlif+cVBJJtA7joqF lR1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960531; x=1685552531; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Fsyg89/vfc29JJa9GS5YhCGnLnEPc8LoOfUR2xEk3nU=; b=N2TL97bxndcXXNkdiOzEAcuUoa0Vso8/qQVywq/R4XKxMTtPHDYEBSp/RNE3Gd+mOO wmCtwUdHpcaIUbqAjq/Qp/AVdcmagMIh9ZtKROvXoDD4MO0ts1GmGnYi7gOVJDjpeKmi tJCp2LAvc3qth1LAJusRnw4/dx4mXr8J2BMGfQQpvqbXJlDOREfpm4b3Opry3k6oVL+a o0HRfJMCwMJlywEzFNMDNI9dHQLJ2ADKjA07KT0D/knqX4jMt6bWT3mwraZ5KhaK28RK 1BJphmZvGmEJVl2Mkw58baRJCD8NrWoM4rU9oHhCs9BDiirBx74NugMOvCMiU41vT7uv juLA== X-Gm-Message-State: AC+VfDxS2Uj7ZGaTJ+oFDKPoSt0Trz7YI+6wi2GuHJBuRCOlor23Hssp 4vmA/2V6yfuyjR8DUVbL2Naqk5ijJZw= X-Google-Smtp-Source: ACHHUZ75lgdLL8vnL0XbIs5yVvuJ1AQE8jmtL55GeRq4GA52aLYGF4UgqLBdCAr7mljybUtkaGeonaYDcH4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:8087:0:b0:b8b:f5fb:598d with SMTP id n7-20020a258087000000b00b8bf5fb598dmr8714623ybk.6.1682960140703; Mon, 01 May 2023 09:55:40 -0700 (PDT) Date: Mon, 1 May 2023 09:54:24 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-15-surenb@google.com> Subject: [PATCH 14/40] lib: code tagging module support From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 531274002E X-Stat-Signature: 5db85bjcmy7q5b71i7cze8exr477e1z9 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682960532-918257 X-HE-Meta: U2FsdGVkX180vxzNDIGyQi+0ZL5TlYar3XZTIT8M/Z+xzI46LoICiYVdFRQsAs6hGp1CXFyhYpMGqgW3BZowAqobnXdT1MjOjdZPcix5o3HYX1Az9PKzyX+u9zD74zZAqnPt6EsBueNHGqlCnAnxZ4UJhQXN32eTW1jyE3Sw66i3TCuNpdMV2oNR5OIyEJ/QnUA0Ixzv9kSEH+HGAZgzU2Coaez1fskkQsOwPPPyf/FXzEN1/2eJb5OfNTb3WC/hOg1Gzznt7QpE4KysPzlFVgMNvZX0VVHKr8B06cpavMrEHy3p4eL+K7KxF0DGAWiCnZ/q1UHwGdxZUzTW275hiiJZ88v8DjdVevi5y5qreqw8s4LL50kp/upGuX/nUjm3LhMGRN7ReSEa+gg+jApfTlV+YfeVLvbMzNnsmJmlkpzVE4BaBQijArjwhSTd2qRN09T0ezfMFFVf6JsvtQvCAzfsZLcKoI8RBtGBaAfaqeoLPmeH72rYEQLUV1925IkGv7bsmZdlM3FlJERQLjggpODMAjAZ+MHcYHScB/ma3Q2m4D4pcBIzDx30s2z/4ePcNKINST5sL6x4B8zKcTMdKEIBLHu0rNTibBsrxtXVL1auqxZBfIjxLa6F+wUQfZi9WkSdUrAds4H0tS6JorVYO5q30NVK3Z4AY4p1SJXuOeR6S21UDG4RzZw3h0LficcmqOWVZ2FT+6p8AhBL4YEyiUZp0CogSIoxL+XuP9IBvmd3dms9ycQVrdLJFbPiPMc+m5psY5IJpWGxAO+0mB7tOgC/0Z64OdSJO3/KGLSNUtzsUwuXMMlMcNn/8UBUKLoX/g0phWsxBu1j0tga7lKb99g3Zpia+b3Gbo8ekXCt0NHZ5hMAP+6c0cIWQnrYVi7ivw1JrAypOqREgNLej5oBN/mEKfkG2IlZZSJ3pPBScYtW6PACljmTsR2HzFq884M174A819g1lAc7lRY59Jd QpfjJa58 4x+jOfkfidJamSdsXOjhw282FBmld+Tu5admQd/jtZP2HUTD9OZyKQKbvq8fxg5APKCBdyY+n4eo0QveSWbmxpw1PQlcBSmIqREp5AHBdOfmx6N7PGww/b4HZpqFQxfLgfsTIJBgRgV5mKsmNmL8cUP6GeMYdeFYENfupEMmCwdRoNYuKJXf+E6m4kteTK++QGXSf3RHM4y5Uf7ipbAeYnOgsdfAUCucWSlDVpkK10vGRzu+fctIWCTHZOObkwx9+N/M0xz6wAD9H0zqplcllZ7HuEGf9ZAb8Qc3g+sEFJWjGHKr+P1+PacYY4hdD5jwGlKzhj/uJQQS+YLfhoiF9vLgzmRgukh7Ik1ljD0m8mIGmKPzwyhHt7CH3Mtz9zTNJe8QyF+UidLMaCwtgiP5F9DhskDpW6rw/uk2oagsRmYEmvIroVj9D7BZcIq9VnJzHn6J6weEqatrxgqa8k/MneVAgJWDvnOmXPU737a7YOoHqbiQTNlkseKWiP3EVRBf7+zxXCr1BZrp/3UFAnwtJKra7+JYpk0Gt7PG0BjKMCRyE/8cfvoIPTXwpyJZG19WzrUimCSnsbx+n8L5K91JV0SMWFRjG9DZQVtm4PEcZWl2jEwjdAUeqQSq4izSdZqTX649o+O4mAuPY10xLT8NXsvIeJOG6xCeLTzYnA/7J4J75F/c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add support for code tagging from dynamically loaded modules. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/codetag.h | 12 +++++++++ kernel/module/main.c | 4 +++ lib/codetag.c | 58 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 2 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index a9d7adecc2a5..386733e89b31 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -42,6 +42,10 @@ struct codetag_module { struct codetag_type_desc { const char *section; size_t tag_size; + void (*module_load)(struct codetag_type *cttype, + struct codetag_module *cmod); + void (*module_unload)(struct codetag_type *cttype, + struct codetag_module *cmod); }; struct codetag_iterator { @@ -68,4 +72,12 @@ void codetag_to_text(struct seq_buf *out, struct codetag *ct); struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); +#ifdef CONFIG_CODE_TAGGING +void codetag_load_module(struct module *mod); +void codetag_unload_module(struct module *mod); +#else +static inline void codetag_load_module(struct module *mod) {} +static inline void codetag_unload_module(struct module *mod) {} +#endif + #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index 044aa2c9e3cb..4232e7bff549 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1249,6 +1250,7 @@ static void free_module(struct module *mod) { trace_module_free(mod); + codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -2974,6 +2976,8 @@ static int load_module(struct load_info *info, const char __user *uargs, /* Get rid of temporary copy. */ free_copy(info, flags); + codetag_load_module(mod); + /* Done! */ trace_module_load(mod); diff --git a/lib/codetag.c b/lib/codetag.c index 7708f8388e55..4ea57fb37346 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -108,15 +108,20 @@ static inline size_t range_size(const struct codetag_type *cttype, static void *get_symbol(struct module *mod, const char *prefix, const char *name) { char buf[64]; + void *ret; int res; res = snprintf(buf, sizeof(buf), "%s%s", prefix, name); if (WARN_ON(res < 1 || res > sizeof(buf))) return NULL; - return mod ? + preempt_disable(); + ret = mod ? (void *)find_kallsyms_symbol_value(mod, buf) : (void *)kallsyms_lookup_name(buf); + preempt_enable(); + + return ret; } static struct codetag_range get_section_range(struct module *mod, @@ -157,8 +162,11 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); - if (err >= 0) + if (err >= 0) { cttype->count += range_size(cttype, &range); + if (cttype->desc.module_load) + cttype->desc.module_load(cttype, cmod); + } up_write(&cttype->mod_lock); if (err < 0) { @@ -197,3 +205,49 @@ codetag_register_type(const struct codetag_type_desc *desc) return cttype; } + +void codetag_load_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) + codetag_module_init(cttype, mod); + mutex_unlock(&codetag_lock); +} + +void codetag_unload_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + struct codetag_module *found = NULL; + struct codetag_module *cmod; + unsigned long mod_id, tmp; + + down_write(&cttype->mod_lock); + idr_for_each_entry_ul(&cttype->mod_idr, cmod, tmp, mod_id) { + if (cmod->mod && cmod->mod == mod) { + found = cmod; + break; + } + } + if (found) { + if (cttype->desc.module_unload) + cttype->desc.module_unload(cttype, cmod); + + cttype->count -= range_size(cttype, &cmod->range); + idr_remove(&cttype->mod_idr, mod_id); + kfree(cmod); + } + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} From patchwork Mon May 1 16:54:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 332FCC77B61 for ; Mon, 1 May 2023 16:55:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF56B900011; Mon, 1 May 2023 12:55:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA5C1900002; Mon, 1 May 2023 12:55:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B46CB900011; Mon, 1 May 2023 12:55:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A6A74900002 for ; Mon, 1 May 2023 12:55:45 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7F480A0700 for ; Mon, 1 May 2023 16:55:45 +0000 (UTC) X-FDA: 80742287850.03.D19EBDD Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf21.hostedemail.com (Postfix) with ESMTP id BC4DA1C0019 for ; Mon, 1 May 2023 16:55:43 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=QWt5E67j; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3Du9PZAYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Du9PZAYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960143; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KlR34XlXux9ljlJXPX99aK4k/cmOetFB9Q46atx2T7Y=; b=K2AUkeMXpsABDG4f7SSrljjbLwOxkVNB4PlYDXAKaLxI6QvQCLL5ub51b1SD+XeaD+7jhb K0ep5Z77JPqtLJDCAkB4XjfGeZNVKqDp6ld/SxvvNoN1MBoMcM6wGZjHt4ztCIrlH4qYQZ hziuSvs+xDtw11jMUCtfG49dQEpLprQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=QWt5E67j; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3Du9PZAYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Du9PZAYKCFUFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960143; a=rsa-sha256; cv=none; b=jGoMiYTh6gnT9XoemZELdkcRC4gh1FKXoYn2uKYtOus/MlkVX8OvLTXToD56uOBoLXrmwN CIGmqJ6at4fWApNd6x1Ja9FNMWUWhcCRP3VytnqBy+WarrK/d6YU81Q8mzeNum7F0wbssX yEyIlgB8zzmoD8JfQfR30E3Ph73e3dk= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-559f142fce7so40198147b3.3 for ; Mon, 01 May 2023 09:55:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960143; x=1685552143; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KlR34XlXux9ljlJXPX99aK4k/cmOetFB9Q46atx2T7Y=; b=QWt5E67jzX3Q9n5FFMEQTPcgcYXDs3S4iywG6hR5j3a3JlBUkYCc4DtOpxxkkjLFnu 4D+SF9vM2W+aSyyk8kJjTG69VNYflbotzNlQqwEogeuQynY5M8r1b7VoJRcfxutnecbL LuU9GuJRo0TkDfrLBSlYcY4SPMYXeI4QMPcJoAM+FNu7Tta0lyhA6wvriGeN5RlC9+S0 iffTz/pkSCC3kF9yyYctejaCym7L71sJ6nP1Y94qsXlQPHEO6LhbQR95ce2BCueglN+Q qfFo6LHwZ424yz+z+s5ncB2VWclW4myqX2EcpXj3e8kJr6XkScT1PFtCKYGoTWUopffl ivjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960143; x=1685552143; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KlR34XlXux9ljlJXPX99aK4k/cmOetFB9Q46atx2T7Y=; b=LjE+tBTCmkjylPhdPeaXaDiHo6bbtjVsxfXC+UWo0CnD2HwuMXNkgTH4kohjJ+kJc+ vowsQAQnjy5j0vTK1T1/ZbddoK1qGa81vTu3MflGIghfwfBIyKexXg4OrzDVwtlgxaBn DEcfl7i1AOzb+tkSVwvNwXGFC9LvZ7wid4YZSZf1wY/KNYqhAPxxjJvr2POwv1MqaQjS vItB3Ht22zbdErXGgoLiq3nf2Ra996kps1cuoGLVtxKukdK9onfLvOUAMkbmugGNiNP1 uf+8w2rbN2KBhrsA9qajjwknmP8EecDr1/XYBsM7m2X3YMgQ38FKgaP7ZsHBFcXPkFse 71LA== X-Gm-Message-State: AC+VfDyerQLzCnYj/kMb+W98knbOtpDT/QwV970Inxsc8mF9C3xJCv6z QqdtA+od61n15tWB1vqER+OVWL/D4kU= X-Google-Smtp-Source: ACHHUZ4HzBM+CNluGDUwOIwZDncQGZjVanEzUdQYw0pkU2rekMEn9D+HYHGjdxqKqeVKj47azkDUlaXgHqg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:b3c8:0:b0:559:e792:4e87 with SMTP id r191-20020a81b3c8000000b00559e7924e87mr4661945ywh.7.1682960142770; Mon, 01 May 2023 09:55:42 -0700 (PDT) Date: Mon, 1 May 2023 09:54:25 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-16-surenb@google.com> Subject: [PATCH 15/40] lib: prevent module unloading if memory is not freed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: BC4DA1C0019 X-Stat-Signature: s5yr4px9dt7k9tdax67df69f7x3psxsg X-HE-Tag: 1682960143-471536 X-HE-Meta: U2FsdGVkX1+K0ZznrJcQLlATy32kY1n6l7nKW02DVa7o52Nz+7w5p8DT4vUqHsNbkZlDPYgwAdCNksvg2zQ359NJbTRNKWzQD+gvw5SDmshCTg9nQ0DwIhd4mxGnh19IFaH/c/7d8BRWVJDJpWlAFtLWrYJA2MTB+SkPFIE6v0WcRT+ZWqVERqwAxemc2zaYBtPZ04OZeTavy3VezkhYjkYomOgDgz1CGQTvEujqmlG4iaZ48Up4utv+knnWKfvCosJBJXz4yPMhL05oEKre2GCsIwTrgNSuujXTr66S95mQKaQTL5vHPR4nxvxu+SfxAsaf4YQH64cVoc3EsYRweI4jvDRuMXYwLfxVZlDTi3mJgTXAOd0YWu53CCbaVZLgFDah/J+4IRcPMichfCg8CLUrKTVc+wQmjxrNUb75+qA0RYRAsWVSrxAMBPwQHQ9TnCYRsx0yzgQ6S59W2zXtIErmFWlfcGszmAaV7ixKmfc0isaPk5mDAyuyCWRa95Q5REE4OgMLo2EEfqBqAEk0RXS3EZAv0zUFqDnG/7VZ80sZrWs8D4203KBuGPC7k1ODYpj1pPOAW4uWGyhSFSImhqEDv15Cn5/bGjHvThKh1z85tyfaTxY+I0zxMoE/LEKtqAqmQbJvk4Xm8yZeGx/eoXANHpUhuJIT27Daqljv53XNQf9dZHxHhBeZb6IFtVbFRny2g2nKEoNKphv945i/7/SLmTcM/+dOK8sipkspLk/FtLMTQ0tJtCgDtjBBS2DgFUC83UR9Kzh7aorfhLLL+2BTR7glNSsixPbBJ62hChzWjnbv6eNdWt0dAEzkGAE681fwBXTrjBHhqt0AnSUt63mO9ZudPAirYWI7wCVhG8ucwK3eIZdnT1/a5LRDzVPFTeblZJXf3UOJ2eh9gDueBGV7BemkYupSBFvo0YRWcc+RxiHrx+5qKEm8lXDpPjmewiUl1UKKv0+VY6kKYUZ cKTb8y1X cg1dSqJNNg8FpjPiOGDXTbrOfaBOuBRm/0RZ/sn9/XizYxLrg92YUExQtQYv8Nh9YlQMxmknEltzLccbl8BvgWneT39a+WVspNqXovcsV3DQKvxNTVMWWt5DPVR3QByiv5l4RErVyBnUDqiARfrHwfEEdGWpmZTbqxljhUe4eC4UBycwmzuWvRsYK0K7ae1NZ/hv8W00ag5v006nUTfZVzZKvJTMhdTwwdSj1mPrrCO1ly68dEm2SjsVemgXbovNq0uBDwiwOGVtCuhqfm/mL4vdaLqAL1V+G2muG0/sYucGDfbk8mypNC2XZpKhoiGBGEyI8Regtut3GozmLs4EDSn4SAb25lrEXpfrJbHisd/7N98Y3zARrUUcFkKizldfYLT9fwg2YFLwLjtt8IgRg3YCFKzXvDseh82tU64OPVBMKk3AGuMlAbJfMxoN1GHrBCbCDmV7MgV4i8BzM0vEbpKqQSRf19VhgquLm1ukTIpr6Of1TB2ICh0ZPCZbQJzmvet2Uq7EzT9R/wIxOt7uC7s7BeyOcHcaAkRAkFcMxPlzWiH+HUdW7CmiBA+tPxXW5XEOsl/qOEIFn6AyP+wgPiwMgEYxGxgE2hcxDwkaFdmk98c9lwfqHRCAQoNvI2EPzr6w0Afm/hXksDfsuXvnzGGSx9g/0RddiiKxciy1e3ME5LIyxC+fiIJeuxQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF. Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 6 +++--- kernel/module/main.c | 23 +++++++++++++++-------- lib/codetag.c | 11 ++++++++--- 3 files changed, 26 insertions(+), 14 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index 386733e89b31..d98e4c8e86f0 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -44,7 +44,7 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - void (*module_unload)(struct codetag_type *cttype, + bool (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); }; @@ -74,10 +74,10 @@ codetag_register_type(const struct codetag_type_desc *desc); #ifdef CONFIG_CODE_TAGGING void codetag_load_module(struct module *mod); -void codetag_unload_module(struct module *mod); +bool codetag_unload_module(struct module *mod); #else static inline void codetag_load_module(struct module *mod) {} -static inline void codetag_unload_module(struct module *mod) {} +static inline bool codetag_unload_module(struct module *mod) { return true; } #endif #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index 4232e7bff549..9ff56f2bb09d 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1218,15 +1218,19 @@ static void *module_memory_alloc(unsigned int size, enum mod_mem_type type) return module_alloc(size); } -static void module_memory_free(void *ptr, enum mod_mem_type type) +static void module_memory_free(void *ptr, enum mod_mem_type type, + bool unload_codetags) { + if (!unload_codetags && mod_mem_type_is_core_data(type)) + return; + if (mod_mem_use_vmalloc(type)) vfree(ptr); else module_memfree(ptr); } -static void free_mod_mem(struct module *mod) +static void free_mod_mem(struct module *mod, bool unload_codetags) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1237,20 +1241,23 @@ static void free_mod_mem(struct module *mod) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod_mem->base, type); + module_memory_free(mod_mem->base, type, + unload_codetags); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA); + module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA, unload_codetags); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { + bool unload_codetags; + trace_module_free(mod); - codetag_unload_module(mod); + unload_codetags = codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -1292,7 +1299,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod); + free_mod_mem(mod, unload_codetags); } void *__symbol_get(const char *symbol) @@ -2294,7 +2301,7 @@ static int move_module(struct module *mod, struct load_info *info) return 0; out_enomem: for (t--; t >= 0; t--) - module_memory_free(mod->mem[t].base, t); + module_memory_free(mod->mem[t].base, t, true); return ret; } @@ -2424,7 +2431,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod); + free_mod_mem(mod, true); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/codetag.c b/lib/codetag.c index 4ea57fb37346..0ad4ea66c769 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -5,6 +5,7 @@ #include #include #include +#include struct codetag_type { struct list_head link; @@ -219,12 +220,13 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -void codetag_unload_module(struct module *mod) +bool codetag_unload_module(struct module *mod) { struct codetag_type *cttype; + bool unload_ok = true; if (!mod) - return; + return true; mutex_lock(&codetag_lock); list_for_each_entry(cttype, &codetag_types, link) { @@ -241,7 +243,8 @@ void codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - cttype->desc.module_unload(cttype, cmod); + if (!cttype->desc.module_unload(cttype, cmod)) + unload_ok = false; cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); @@ -250,4 +253,6 @@ void codetag_unload_module(struct module *mod) up_write(&cttype->mod_lock); } mutex_unlock(&codetag_lock); + + return unload_ok; } From patchwork Mon May 1 16:54:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B9E4C77B7C for ; Mon, 1 May 2023 16:55:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F4051900012; Mon, 1 May 2023 12:55:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF0C9900002; Mon, 1 May 2023 12:55:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB7CD900012; Mon, 1 May 2023 12:55:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CC048900002 for ; Mon, 1 May 2023 12:55:47 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A7A991607F3 for ; Mon, 1 May 2023 16:55:47 +0000 (UTC) X-FDA: 80742287934.12.C544C49 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id D6AA74000A for ; Mon, 1 May 2023 16:55:45 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=7DF+QMYA; spf=pass (imf17.hostedemail.com: domain of 3EO9PZAYKCFcHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3EO9PZAYKCFcHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960145; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cfU8v4uljuDyceCulpsszZZ0MEofJFrAynDM4I2qv80=; b=PHwV5Px8UAXmFbVG9m2th55zm8e73EyRovArLhZJ5YOxFVB2VxdvYkhScfOaeePuj9i8Ik 2w2BQLdbuRRAW4Kei1Q8cYjVlrFCpsNW7xWg2XsF25cMge5UrdnlprAKlNReEVNB4/lNvE 1I/SrfDHOo0Xz8pp59Mxf070sWa1cLQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=7DF+QMYA; spf=pass (imf17.hostedemail.com: domain of 3EO9PZAYKCFcHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3EO9PZAYKCFcHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960145; a=rsa-sha256; cv=none; b=B2A4MEOFl7Iv4jdv8Qw7TTvMKIWAgPHKFUEJAr/kPT3VOOB2mu6wL7qShvTJaC1V9NsGPY VJ/cjqXNm1vQAqFUeL8KfBVVi8y5Sccr2U8vDuv+ipwQDA036/qmfvKkYpSg+ZMwtBZXjY h86HG7ua5vDiwVOpXKryj3EHlIITlP8= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a6eeea78cso27814717276.0 for ; Mon, 01 May 2023 09:55:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960145; x=1685552145; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cfU8v4uljuDyceCulpsszZZ0MEofJFrAynDM4I2qv80=; b=7DF+QMYAAMLutPK+67W4THAVtEPwoqDsbe/74l1tBpyC07tcRQ6NcM5IV8D6bbGSDO mAKOZqw8S8PkrBoJSwJTCQQPpA7SwA7cyZxlWbMDiokIgQwGSl9egB8QrEXUBH/T64s1 zNCYsAsh3eGpCoX4dQnQUqKiZzSauwZNQoR9XmoXX99k0D1LyT9NYpubJVC6Qn7rUrDx j9q+sgeZA/RMhwqSDAOuZPyoK6vZlZ73Jq2IAunuj8KRGdDA/IrEELOEoHnv6XAiPWlM lMDNSxrkbvL1rl0oDCkvCwbUBvti8udh5vGgrmSEH3MVGbNpRYhf9xH2dPTrsqzR9Dja mOjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960145; x=1685552145; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cfU8v4uljuDyceCulpsszZZ0MEofJFrAynDM4I2qv80=; b=GkYgwnM5gVufJHFwxsCbW1U6OhEzp4WULfgHMLiXyK5+qjOVb4oh5aK9Lu8Qv1oaSy 0BNd2Wolfrpl/8rcoZtDOkRF0XbP3DCf7gYX7Ld2kudGH/wHAjiNjx6zeqskTxuYB0mH /DmRgWlnVAEe3MLrczslQCYs4n5E4cOAPqKw/K7v/MaJKBrP8GCD/KJCwx/BwXbwhd0L /1G2JL7zMumc2lWqGAkWuhyqtaq7nv7H5e+p78RewfaFiG6X5zMWUr+rOWqdpGkKBMy9 pQ6G5LkJInzlX0eREGKYO3KCjHRQnGvvGL11fgqQSCFZ4tPTkAj6wk9gUe6Y2lZPsJ8T Y+ig== X-Gm-Message-State: AC+VfDwj9AE1OAny12OSnVoFR78NR1QBtDGZuK49kTeyLu9zTp9emtWh /oETxuPOlu15I840AHU/yROJ8AuNujg= X-Google-Smtp-Source: ACHHUZ7KwLnOhs3Bj4j+dfecjkq9PpPEjcCUN0XtdJ++CCEZIkvWVHHnPtSXEKWIoTIOpnfnSo0AXCZYwNA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:3482:0:b0:b94:6989:7fa6 with SMTP id b124-20020a253482000000b00b9469897fa6mr8599498yba.4.1682960144921; Mon, 01 May 2023 09:55:44 -0700 (PDT) Date: Mon, 1 May 2023 09:54:26 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-17-surenb@google.com> Subject: [PATCH 16/40] lib: code tagging query helper functions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D6AA74000A X-Stat-Signature: 58efmjwzfoohbqarsrdzq6teokzbkppp X-Rspam-User: X-HE-Tag: 1682960145-58892 X-HE-Meta: U2FsdGVkX18gC8T4qoD29CSNwIkrDd4NiW4hmPqj+QEJq6nBBWcoMod85YwkvfBOWK/09KKjF8HOcQRGyCqWp6XmvYrzuhQpOKsfqT3/DX27pLrh+yrrScwts+25WS7qHHmyE6PgDwzgN/evlQflL7fT9opACpSHQZKQWfenJtD05Z3KD+kF/Ei0omg/xZb12Fv2hOQWk4badFdOvDLs0U3sgHUREtUNpqSmhFVmsw5Wv6O33twM/V0IXtGDPVFPesjoXRs4yAKkOV21S+1PlxRE4QphI16McT32KUHp60KLNQOrC5Wkz1AXMd91816L01kFds/i5Kxx6fRM8j49RMbQfDSN91VqeKdcoJ0lsbTpasAxnAdp9WSqWq9OxLltdQDyMJc0Q3NP/ZXS3Bcs0nnWXhKjfEdCrys0KwlKsXiGwYUjlZCmwA4caYadW4oRpNWopazOoDodlJsvAkAYWdWRSCRgvucKUA7r2jtPVBeMAWdjKq2L6KOPlHkpsjCzluPDAL+AnsjNZw0sj8tUsLzOpkHQNQbDa3ItJLpBjrGjl3jY0qlxDUQXAadWqEUKbPyCbW9+fk/bzNbddjD7f+BRmbxIhzgEL8hbb/EnSnd1R+ahP/t34iPAFX7DVRA2qpPeUI69z25lcuvTnzeRORI/Aa6N5gDlkAfiCsawAoerTBlqv9lrYCdgF8aQ3LieGuj29PNXIjjwTJ5XooGu6snUoOH5MRnb2RzJ/jIdff6yf0Bd6Wa/Iw21SwVdJuGXJXRCtcJouYDjcT1GefoEF930QF6+CJomT9cfXGSrxz2BNfuYvDTQxWKTkBgWGPZtJChqqyL0c1GN38+uOwbUdUM6aEy1QUY+ZmZMoIJkFBX96yFmjKJlm6d9McGnhSZRO5v2Kj5otGEtZbpYiTZ5NLLUftXu5XC3cglOI3iBxUtI0JjCTy7Dykn1X8lUZGGKiTn+aDOjFEXHWsictHf 1rl4XZkz jeCanjAZ8KXPjG0IwWWjJrg4qGwGrPI1byqs25GeW9x9ZZZ/CqgAltKE8pPyhOHd/q8vvk3opS9CrPTy3XzSnpRgUgfyISEKSOFGxLFkHv2rHHSDHwV3QBDuJtnHD0Mgyt2KmBYJpMCaG9HMS3MdQDYGTkonYwpfHCzR1h7neuBYb+HYRuGLpsDJl8LxuN1UxN3MQXbkoyIYvTgfYIm0cetk6pUOj5KRgQIdIyWapAeaOpmv62hfG38R+4Pox5IkuDSNwxz3NcFIJNv1+i30pGGTtNblwhZ6RijiH+DxArX/BmuyKlBSoyV/ISMbDsT9uwnh3ipLmlpY6ln3HVG7ZgZgJ0voKoTaTEMaZGow6U2UdcY8lPLy0aVbQK8h4zdmJVmdJP9JpmIIVRFojcTy7erkEisbxm/bpviKilNnzxXq7ioFDD3CF+BaCyhZkcgHgzuwzTIYWwGRqTf+0Jeh8ZZe8BiUoL6Qxt871KhxbODC1wV5I7C+TIFwuQ8NVbhZWa1g++6+iw+rj0cVXQ+qHiiM7AND2KWM8VS4UyR+zGi1EPOHKdRe1ZF+7tdC57HSw0SXtClhn915EBg9kp8XaC3ALmNWeQZvph/hPbJXvEPoFWqiqcf9Hy3n154TKr7EQmczM1VlSsyMiWO4rjyw3tDka3ap3dPXACfoP6IIasVKF/HKXSj4CNxpSOejdkM3BnMgTn7PBVmZjnshxFvZLzuh/Xw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet Provide codetag_query_parse() to parse codetag queries and codetag_matches_query() to check if the query affects a given codetag. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 27 ++++++++ lib/codetag.c | 135 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 162 insertions(+) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index d98e4c8e86f0..87207f199ac9 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -80,4 +80,31 @@ static inline void codetag_load_module(struct module *mod) {} static inline bool codetag_unload_module(struct module *mod) { return true; } #endif +/* Codetag query parsing */ + +struct codetag_query { + const char *filename; + const char *module; + const char *function; + const char *class; + unsigned int first_line, last_line; + unsigned int first_index, last_index; + unsigned int cur_index; + + bool match_line:1; + bool match_index:1; + + unsigned int set_enabled:1; + unsigned int enabled:2; + + unsigned int set_frequency:1; + unsigned int frequency; +}; + +char *codetag_query_parse(struct codetag_query *q, char *buf); +bool codetag_matches_query(struct codetag_query *q, + const struct codetag *ct, + const struct codetag_module *mod, + const char *class); + #endif /* _LINUX_CODETAG_H */ diff --git a/lib/codetag.c b/lib/codetag.c index 0ad4ea66c769..84f90f3b922c 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -256,3 +256,138 @@ bool codetag_unload_module(struct module *mod) return unload_ok; } + +/* Codetag query parsing */ + +#define CODETAG_QUERY_TOKENS() \ + x(func) \ + x(file) \ + x(line) \ + x(module) \ + x(class) \ + x(index) + +enum tokens { +#define x(name) TOK_##name, + CODETAG_QUERY_TOKENS() +#undef x +}; + +static const char * const token_strs[] = { +#define x(name) #name, + CODETAG_QUERY_TOKENS() +#undef x + NULL +}; + +static int parse_range(char *str, unsigned int *first, unsigned int *last) +{ + char *first_str = str; + char *last_str = strchr(first_str, '-'); + + if (last_str) + *last_str++ = '\0'; + + if (kstrtouint(first_str, 10, first)) + return -EINVAL; + + if (!last_str) + *last = *first; + else if (kstrtouint(last_str, 10, last)) + return -EINVAL; + + return 0; +} + +char *codetag_query_parse(struct codetag_query *q, char *buf) +{ + while (1) { + char *p = buf; + char *str1 = strsep_no_empty(&p, " \t\r\n"); + char *str2 = strsep_no_empty(&p, " \t\r\n"); + int ret, token; + + if (!str1 || !str2) + break; + + token = match_string(token_strs, ARRAY_SIZE(token_strs), str1); + if (token < 0) + break; + + switch (token) { + case TOK_func: + q->function = str2; + break; + case TOK_file: + q->filename = str2; + break; + case TOK_line: + ret = parse_range(str2, &q->first_line, &q->last_line); + if (ret) + return ERR_PTR(ret); + q->match_line = true; + break; + case TOK_module: + q->module = str2; + break; + case TOK_class: + q->class = str2; + break; + case TOK_index: + ret = parse_range(str2, &q->first_index, &q->last_index); + if (ret) + return ERR_PTR(ret); + q->match_index = true; + break; + } + + buf = p; + } + + return buf; +} + +bool codetag_matches_query(struct codetag_query *q, + const struct codetag *ct, + const struct codetag_module *mod, + const char *class) +{ + size_t classlen = q->class ? strlen(q->class) : 0; + + if (q->module && + (!mod->mod || + strcmp(q->module, ct->modname))) + return false; + + if (q->filename && + strcmp(q->filename, ct->filename) && + strcmp(q->filename, kbasename(ct->filename))) + return false; + + if (q->function && + strcmp(q->function, ct->function)) + return false; + + /* match against the line number range */ + if (q->match_line && + (ct->lineno < q->first_line || + ct->lineno > q->last_line)) + return false; + + /* match against the class */ + if (classlen && + (strncmp(q->class, class, classlen) || + (class[classlen] && class[classlen] != ':'))) + return false; + + /* match against the fault index */ + if (q->match_index && + (q->cur_index < q->first_index || + q->cur_index > q->last_index)) { + q->cur_index++; + return false; + } + + q->cur_index++; + return true; +} From patchwork Mon May 1 16:54:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAE63C77B61 for ; Mon, 1 May 2023 16:55:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 442E8900015; Mon, 1 May 2023 12:55:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F41C900002; Mon, 1 May 2023 12:55:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 295E1900015; Mon, 1 May 2023 12:55:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 10A93900002 for ; Mon, 1 May 2023 12:55:50 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D46B81407C0 for ; Mon, 1 May 2023 16:55:49 +0000 (UTC) X-FDA: 80742288018.28.0A64A8E Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf11.hostedemail.com (Postfix) with ESMTP id 116FA4001A for ; Mon, 1 May 2023 16:55:47 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=EtKwNJp9; spf=pass (imf11.hostedemail.com: domain of 3E-9PZAYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3E-9PZAYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960148; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n4nztol7rM1gH9OZw6qc/mLZ9u/ayJFp51XDo0Zgdb0=; b=yLVaS7Ld6ZXgqE+YpIbDdzGF1GSuhUoKWP4pggH6JTl501k8NRIXRrY+krOi7IWmux4NNg 1hFxD+DxEutzd68EDTVYS6d+H5NE+8fIsfq+SgS7vlkLg2nGAbd/DvPimz48K0MG8sdZEQ SEDZV1Tydb3BBjeRSZVtOxp3dOW3vUI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960148; a=rsa-sha256; cv=none; b=mbWutTbDFRahyssJJ2DPkaForJ5rJysODskwaeBuN713ojrtpCILKBaUYP2YKvJg87Ir+a VDxoL8AVt8Y04dg2k3lM7PHss5q98lqSf77+NeiTsl2KLTvUi2KXf3mne6lR//g1qNUlck EinfySHtr1tnitnOFPgjo3RNub9NLPY= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=EtKwNJp9; spf=pass (imf11.hostedemail.com: domain of 3E-9PZAYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3E-9PZAYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9e2b65f2eeso1390102276.2 for ; Mon, 01 May 2023 09:55:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960147; x=1685552147; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=n4nztol7rM1gH9OZw6qc/mLZ9u/ayJFp51XDo0Zgdb0=; b=EtKwNJp9DhXdPpGMnxJhURr2M+CEUDCL3ptY/gnCpcS0fCNClhBRQnKLQaL1vap9Xv flIa0VF5vnwD4+yj//NKKusPsaplJKWYoL8OXzGJevivoC8g8NK/KucCrwwcpJv7veAa xgY8hftcgpKYNDC1x16rPLL3UZWkzbNXNnzUPnYMfXTjNS/84SrVYM1+6fXXIcqe0gMq Pvc/uc9Lu7NjvCYiVn+U/2J1Q6ZWwlWHji/FlDVWwKbAq0PnDxII8zNEOzDe0Qjfw6ov uROLRXRRxNFvvQjU2j5RxMIKm9rtC5Y4CTJTP9LtI2yGnirb5FQXS5XGTlB6aZv4UEfC Y5Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960147; x=1685552147; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n4nztol7rM1gH9OZw6qc/mLZ9u/ayJFp51XDo0Zgdb0=; b=f6IF15KzYufSyH9CUWJI+bxC+2Wj5T39WRemcl7k2b3kcN6YnbnlwA3BqwuoHyLvzJ A90Z+YS0gB6U45edftJ0UypcpnxyprB69+6F/kj9DX5SERC47Vf+XFeoElyuliLzf2kT xWxTPYFQ0jpJNfJumfltFY4i9w8+dnDMt/LCc/ErRq2eWWcr6KJlbYdMma+cMocJZzFy zC5tbMElkRVLM4Sv+zDW6QNmxzax79RUtDQcBQ2P5QRLdwEzaF0pcVEkww3zv9laY2Ti 0Qs11o1BotMNdExMbFLm4XVi/w7x2Yp1CPpjUwcYOndWdaHZEPXugpK4BiGwr4jcacpG S40g== X-Gm-Message-State: AC+VfDxDAfSqXtKPKRdiv+LiHHDgaj0yjuEvviS/eRy/8/2ff3bIKTec m40xO8ug5kjGilGRopPz7YZLWHGYIqc= X-Google-Smtp-Source: ACHHUZ4asddFpQQ9HoEi7y1zPK0HoL2O96R3TqiDLfh+711URnykct4JIy5k4NiME4epWeGLHMGL7X0MSsc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:bb0b:0:b0:b9a:6508:1b5f with SMTP id z11-20020a25bb0b000000b00b9a65081b5fmr5519617ybg.11.1682960147095; Mon, 01 May 2023 09:55:47 -0700 (PDT) Date: Mon, 1 May 2023 09:54:27 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-18-surenb@google.com> Subject: [PATCH 17/40] lib: add allocation tagging support for memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: 116FA4001A X-Rspamd-Server: rspam09 X-Stat-Signature: qmkqis9eddjunz8qgn66b4c516565w6f X-HE-Tag: 1682960147-466817 X-HE-Meta: U2FsdGVkX1/cBAUkJkcCmpdGaZwRhbPVv40eAU+JUOJX/tX+x8DcnZOFdizTc3WVjgBwo61anF7BXPGWIVc+P5gThZzeqWm80LQ1g4pR8gQQHZlpCqKUGpYBzarG/CHbN7xg3GjqUNw8HJ54n/S655zyMUykI+UU1mZbOoRYq9OG3JGT5BHucr3Nh5Q2jDiMNyvSRfRJqjQm1TfdIhkruCX0zDIZdsxzFXi+IyP5lN8tnB6vJpxbIS2AnRdPLU5aS6iPjZUoTR+1vj1Z4Im7CRoY/W92Z1KBCX7bFF40Vaeam4hni6fywk7BBR++51NgoyKRzaXrkmBmvcnZXP2THbXvF+rEj38v5rYFSz8Ix3Zs1/1qtx5/s9QEDpbiFT+VK6c/8yL3V4D35gU/J5/p9JbG+j32I2CYaHuDHR7eqdCos5+5Ckd1HTZpqWIvRO2FPXnYZW7diCdT8lb6OXoo1f+yl0Nm5aaMAGzm4eJW3cgY+b+6H6SxKVAWMjy2tPu5u1LL3vy/JLq8Z2o2Txjz9L4ViJsr/peZ3T6ntmZRbD1jHxfKGjW8W/gyX4QBiubVxVkNIdbTn/TB0ePtNBJOx19G91LZznkP1eB0FbQOczDPWNduVyVr9LEJE5qY+O390pEiaiLMAJULVYC1F8txnNVDodGje0RC4XCtugyTylz4VvXjL7yYFqg0zFi9d4mMM23fGcPjbWwup9k6fBusPlNQrJ/C8Vekh3VJC22Wm2qUXsX68kKAjojvlQ18a9wy1Rx0+Zkgl52R3FMa+IowPKq/pTjaaK5u7DFcS+k/644JlhdVFdJnVatM+6eNq5YeKd142uVZKhsv1lbrDnZiREsew/whD+vBFIkyGb9Qfx5Zy2GBA/vsPia0R2A6PdIzmZD+s66VV07vWHdU5OJdvvuQV0difxZPhVINxebtAPsP3tSfQ64M1X/VQvJmYRexS2WZbfVmbkASUCWmEiI f++MNxu6 Va8BpnX08u4XId10O0obyaQg4djsf95vIC2JbTYNrd5rmjJSrekkhYx28Yv0F5qYSxoZANHIOAPKG6HY4PKtItE5GNZNYD5sKgcEwDZbM4PiGsUarvgUJSj3Md2XTDjuLj5IlMfkmsX5X0+Lsgl/twvKwribT0S7j5sLyKIeJQ97w6/76sXO3l1m/VN7+9W969n2VG/XiHJ9L12sIM9KXsuDcWRJyi9FixNicBZw7wwFEkd9e+ZzuD+QyH3zw4xA84qxbl40b2HRkHZr3SM3gyQsyKOY+GVb92gpbREGfjjP948nbbQ2YlCm0cbTRTQMeDknLRArF1ScBlYXQ/E5lpJVaRUEOY67fjCKCT7pMvbzXWyZwz+J93m5DC0CV3lbZ6gt1JWk/adxaEK2hB5squ4ldEZEqmQvZqJEzxmhQZjIV4vfHm0F8bk89ZAWcT3aIp6wRTPl20EG4REH38KbLhmuasENL43UfywZZi0+Ib5gDC8mVlpPmeqUM1O9sfCssrUHz4LNMDwz/dZgidzeiIrGbhQpv11cslEDvlG7wZURTedoq5knliku+oOeyazWRMB2JxDor4YG48y1uHb8z9hlBXRyFnM+PGaHqCRKXvyvywFybl7mXB+6lflD60WXV/sutcuRWqjmY7VrQXlN7yZu7lCj+XBePPkWFNFFy3s6E8NQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce CONFIG_MEM_ALLOC_PROFILING which provides definitions to easily instrument memory allocators. It also registers an "alloc_tags" codetag type with "allocations" defbugfs interface to output allocation tag information. CONFIG_MEM_ALLOC_PROFILING_DEBUG is provided for debugging the memory allocation profiling instrumentation. Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- .../admin-guide/kernel-parameters.txt | 2 + include/asm-generic/codetag.lds.h | 14 ++ include/asm-generic/vmlinux.lds.h | 3 + include/linux/alloc_tag.h | 105 +++++++++++ include/linux/sched.h | 24 +++ lib/Kconfig.debug | 19 ++ lib/Makefile | 2 + lib/alloc_tag.c | 177 ++++++++++++++++++ scripts/module.lds.S | 7 + 9 files changed, 353 insertions(+) create mode 100644 include/asm-generic/codetag.lds.h create mode 100644 include/linux/alloc_tag.h create mode 100644 lib/alloc_tag.c diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 9e5bab29685f..2fd8e56b7af8 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3770,6 +3770,8 @@ nomce [X86-32] Disable Machine Check Exception + nomem_profiling Disable memory allocation profiling. + nomfgpt [X86-32] Disable Multi-Function General Purpose Timer usage (for AMD Geode machines). diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h new file mode 100644 index 000000000000..64f536b80380 --- /dev/null +++ b/include/asm-generic/codetag.lds.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_GENERIC_CODETAG_LDS_H +#define __ASM_GENERIC_CODETAG_LDS_H + +#define SECTION_WITH_BOUNDARIES(_name) \ + . = ALIGN(8); \ + __start_##_name = .; \ + KEEP(*(_name)) \ + __stop_##_name = .; + +#define CODETAG_SECTIONS() \ + SECTION_WITH_BOUNDARIES(alloc_tags) + +#endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index d1f57e4868ed..985ff045c2a2 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -50,6 +50,8 @@ * [__nosave_begin, __nosave_end] for the nosave data */ +#include + #ifndef LOAD_OFFSET #define LOAD_OFFSET 0 #endif @@ -374,6 +376,7 @@ . = ALIGN(8); \ BOUNDED_SECTION_BY(__dyndbg_classes, ___dyndbg_classes) \ BOUNDED_SECTION_BY(__dyndbg, ___dyndbg) \ + CODETAG_SECTIONS() \ LIKELY_PROFILE() \ BRANCH_PROFILE() \ TRACE_PRINTKS() \ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h new file mode 100644 index 000000000000..d913f8d9a7d8 --- /dev/null +++ b/include/linux/alloc_tag.h @@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * allocation tagging + */ +#ifndef _LINUX_ALLOC_TAG_H +#define _LINUX_ALLOC_TAG_H + +#include +#include +#include +#include +#include + +/* + * An instance of this structure is created in a special ELF section at every + * allocation callsite. At runtime, the special section is treated as + * an array of these. Embedded codetag utilizes codetag framework. + */ +struct alloc_tag { + struct codetag ct; + struct lazy_percpu_counter bytes_allocated; +} __aligned(8); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) +{ + return container_of(ct, struct alloc_tag, ct); +} + +#define DEFINE_ALLOC_TAG(_alloc_tag, _old) \ + static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { .ct = CODE_TAG_INIT }; \ + struct alloc_tag * __maybe_unused _old = alloc_tag_save(&_alloc_tag) + +extern struct static_key_true mem_alloc_profiling_key; + +static inline bool mem_alloc_profiling_enabled(void) +{ + return static_branch_likely(&mem_alloc_profiling_key); +} + +static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes, + bool may_allocate) +{ + struct alloc_tag *tag; + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + /* The switch should be checked before this */ + BUG_ON(!mem_alloc_profiling_enabled()); + + WARN_ONCE(ref && !ref->ct, "alloc_tag was not set\n"); +#endif + if (!ref || !ref->ct) + return; + + tag = ct_to_alloc_tag(ref->ct); + + if (may_allocate) + lazy_percpu_counter_add(&tag->bytes_allocated, -bytes); + else + lazy_percpu_counter_add_noupgrade(&tag->bytes_allocated, -bytes); + ref->ct = NULL; +} + +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + __alloc_tag_sub(ref, bytes, true); +} + +static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) +{ + __alloc_tag_sub(ref, bytes, false); +} + +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + /* The switch should be checked before this */ + BUG_ON(!mem_alloc_profiling_enabled()); + + WARN_ONCE(ref && ref->ct, + "alloc_tag was not cleared (got tag for %s:%u)\n",\ + ref->ct->filename, ref->ct->lineno); + + WARN_ONCE(!tag, "current->alloc_tag not set"); +#endif + if (!ref || !tag) + return; + + ref->ct = &tag->ct; + lazy_percpu_counter_add(&tag->bytes_allocated, bytes); +} + +#else + +#define DEFINE_ALLOC_TAG(_alloc_tag, _old) +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} +static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) {} +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, + size_t bytes) {} + +#endif + +#endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 35e7efdea2d9..33708bf8f191 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -763,6 +763,10 @@ struct task_struct { unsigned int flags; unsigned int ptrace; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif + #ifdef CONFIG_SMP int on_cpu; struct __call_single_node wake_entry; @@ -802,6 +806,7 @@ struct task_struct { struct task_group *sched_task_group; #endif + #ifdef CONFIG_UCLAMP_TASK /* * Clamp values requested for a scheduling entity. @@ -2444,4 +2449,23 @@ static inline void sched_core_fork(struct task_struct *p) { } extern void sched_set_stop_task(int cpu, struct task_struct *stop); +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) +{ + swap(current->alloc_tag, tag); + return tag; +} + +static inline void alloc_tag_restore(struct alloc_tag *tag, struct alloc_tag *old) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN(current->alloc_tag != tag, "current->alloc_tag was changed:\n"); +#endif + current->alloc_tag = old; +} +#else +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) { return NULL; } +#define alloc_tag_restore(_tag, _old) +#endif + #endif diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 5078da7d3ffb..da0a91ea6042 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -961,6 +961,25 @@ config CODE_TAGGING bool select KALLSYMS +config MEM_ALLOC_PROFILING + bool "Enable memory allocation profiling" + default n + depends on DEBUG_FS + select CODE_TAGGING + select LAZY_PERCPU_COUNTER + help + Track allocation source code and record total allocation size + initiated at that code location. The mechanism can be used to track + memory leaks with a low performance impact. + +config MEM_ALLOC_PROFILING_DEBUG + bool "Memory allocation profiler debugging" + default n + depends on MEM_ALLOC_PROFILING + help + Adds warnings with helpful error messages for memory allocation + profiling. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 28d70ecf2976..8d09ccb4d30c 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -229,6 +229,8 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_CODE_TAGGING) += codetag.o +obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o + lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c new file mode 100644 index 000000000000..3c4cfeb79862 --- /dev/null +++ b/lib/alloc_tag.c @@ -0,0 +1,177 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include + +DEFINE_STATIC_KEY_TRUE(mem_alloc_profiling_key); + +/* + * Won't need to be exported once page allocation accounting is moved to the + * correct place: + */ +EXPORT_SYMBOL(mem_alloc_profiling_key); + +static int __init mem_alloc_profiling_disable(char *s) +{ + static_branch_disable(&mem_alloc_profiling_key); + return 1; +} +__setup("nomem_profiling", mem_alloc_profiling_disable); + +struct alloc_tag_file_iterator { + struct codetag_iterator ct_iter; + struct seq_buf buf; + char rawbuf[4096]; +}; + +struct user_buf { + char __user *buf; /* destination user buffer */ + size_t size; /* size of requested read */ + ssize_t ret; /* bytes read so far */ +}; + +static int flush_ubuf(struct user_buf *dst, struct seq_buf *src) +{ + if (src->len) { + size_t bytes = min_t(size_t, src->len, dst->size); + int err = copy_to_user(dst->buf, src->buffer, bytes); + + if (err) + return err; + + dst->ret += bytes; + dst->buf += bytes; + dst->size -= bytes; + src->len -= bytes; + memmove(src->buffer, src->buffer + bytes, src->len); + } + + return 0; +} + +static int allocations_file_open(struct inode *inode, struct file *file) +{ + struct codetag_type *cttype = inode->i_private; + struct alloc_tag_file_iterator *iter; + + iter = kzalloc(sizeof(*iter), GFP_KERNEL); + if (!iter) + return -ENOMEM; + + codetag_lock_module_list(cttype, true); + iter->ct_iter = codetag_get_ct_iter(cttype); + codetag_lock_module_list(cttype, false); + seq_buf_init(&iter->buf, iter->rawbuf, sizeof(iter->rawbuf)); + file->private_data = iter; + + return 0; +} + +static int allocations_file_release(struct inode *inode, struct file *file) +{ + struct alloc_tag_file_iterator *iter = file->private_data; + + kfree(iter); + return 0; +} + +static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + char buf[10]; + + string_get_size(lazy_percpu_counter_read(&tag->bytes_allocated), 1, + STRING_UNITS_2, buf, sizeof(buf)); + + seq_buf_printf(out, "%8s ", buf); + codetag_to_text(out, ct); + seq_buf_putc(out, '\n'); +} + +static ssize_t allocations_file_read(struct file *file, char __user *ubuf, + size_t size, loff_t *ppos) +{ + struct alloc_tag_file_iterator *iter = file->private_data; + struct user_buf buf = { .buf = ubuf, .size = size }; + struct codetag *ct; + int err = 0; + + codetag_lock_module_list(iter->ct_iter.cttype, true); + while (1) { + err = flush_ubuf(&buf, &iter->buf); + if (err || !buf.size) + break; + + ct = codetag_next_ct(&iter->ct_iter); + if (!ct) + break; + + alloc_tag_to_text(&iter->buf, ct); + } + codetag_lock_module_list(iter->ct_iter.cttype, false); + + return err ? : buf.ret; +} + +static const struct file_operations allocations_file_ops = { + .owner = THIS_MODULE, + .open = allocations_file_open, + .release = allocations_file_release, + .read = allocations_file_read, +}; + +static int __init dbgfs_init(struct codetag_type *cttype) +{ + struct dentry *file; + + file = debugfs_create_file("allocations", 0444, NULL, cttype, + &allocations_file_ops); + + return IS_ERR(file) ? PTR_ERR(file) : 0; +} + +static bool alloc_tag_module_unload(struct codetag_type *cttype, struct codetag_module *cmod) +{ + struct codetag_iterator iter = codetag_get_ct_iter(cttype); + bool module_unused = true; + struct alloc_tag *tag; + struct codetag *ct; + size_t bytes; + + for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { + if (iter.cmod != cmod) + continue; + + tag = ct_to_alloc_tag(ct); + bytes = lazy_percpu_counter_read(&tag->bytes_allocated); + + if (!WARN(bytes, "%s:%u module %s func:%s has %zu allocated at module unload", + ct->filename, ct->lineno, ct->modname, ct->function, bytes)) + lazy_percpu_counter_exit(&tag->bytes_allocated); + else + module_unused = false; + } + + return module_unused; +} + +static int __init alloc_tag_init(void) +{ + struct codetag_type *cttype; + const struct codetag_type_desc desc = { + .section = "alloc_tags", + .tag_size = sizeof(struct alloc_tag), + .module_unload = alloc_tag_module_unload, + }; + + cttype = codetag_register_type(&desc); + if (IS_ERR_OR_NULL(cttype)) + return PTR_ERR(cttype); + + return dbgfs_init(cttype); +} +module_init(alloc_tag_init); diff --git a/scripts/module.lds.S b/scripts/module.lds.S index bf5bcf2836d8..45c67a0994f3 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -9,6 +9,8 @@ #define DISCARD_EH_FRAME *(.eh_frame) #endif +#include + SECTIONS { /DISCARD/ : { *(.discard) @@ -47,12 +49,17 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) + CODETAG_SECTIONS() } .rodata : { *(.rodata .rodata.[0-9a-zA-Z_]*) *(.rodata..L*) } +#else + .data : { + CODETAG_SECTIONS() + } #endif } From patchwork Mon May 1 16:54:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 893B9C77B73 for ; Mon, 1 May 2023 16:55:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D3E2900016; Mon, 1 May 2023 12:55:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AAE7900002; Mon, 1 May 2023 12:55:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3F92900016; Mon, 1 May 2023 12:55:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E3018900002 for ; Mon, 1 May 2023 12:55:51 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AD77AA07CD for ; Mon, 1 May 2023 16:55:51 +0000 (UTC) X-FDA: 80742288102.13.42DD47A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id E100940008 for ; Mon, 1 May 2023 16:55:49 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PkT7GX0U; spf=pass (imf17.hostedemail.com: domain of 3Fe9PZAYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Fe9PZAYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960149; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ANclwoiguD8BsbfKgYnGL1LnBw+SqNV6XnzUrNDvkRU=; b=GujQJ45ak40Dghhr6f15EcNSAGvwwV3F8Lvyj1SQxnSWHd0hWX5GBELoW0POtfiR+ibhWu gPTiLgvIUDoVzlo8x86Zuwn4zeeK9CSo7ZW+mdBdNJ4EEEPGStVuODsXoahKJ1Uws/davG o4d7gKOIwbeiveLgyZ5Gf1cE5roT5Vo= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PkT7GX0U; spf=pass (imf17.hostedemail.com: domain of 3Fe9PZAYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Fe9PZAYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960149; a=rsa-sha256; cv=none; b=ja6QlE6WPTLihUTgkdem5lN2/RHpCcZ7c7GtTGkro3Ifdi/tiIjJfYAIaCam6xhEm0I0/O V989fyFU/UK6+JO6yAIrlMRAvmieEUJNpnz92BmFSV3QNvTK2wj5cQOCdm3mfhV+6r4gFU hfxZO9xm0WCJla4306rcMOmnoMSO/sU= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a6eeea78cso27815131276.0 for ; Mon, 01 May 2023 09:55:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960149; x=1685552149; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ANclwoiguD8BsbfKgYnGL1LnBw+SqNV6XnzUrNDvkRU=; b=PkT7GX0UpBWBL3ZxZZQ1u+ssYXtbTc+WLolVY9j+MP76VowH5UScBu+NGiGIbJ7Y6L eZfY/6ZYqoCRi2CKlp/WtuvsiD21usF1/P37qu0gPbOtENmIl51pQz/p9+lm6j/kWelt Ov6MNUZwC6/nK8n86rfdlgBHbFP156NpwLcMsML+lMt+4KcACeUQ1EH3vrJuTYlZHnxu 7BuZUefNEtoVSDEOw+FF7cwg3Q8gxPu6E4rHt6NEwlnEpLRRqtUVy0vsJsqLucVJRBrd gANWGNYs/01Nqy3PLDr7YbpWPHn8hbMhpjK41s03TQMV5IjienckbULG1YjCuS8h+b0d B72Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960149; x=1685552149; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ANclwoiguD8BsbfKgYnGL1LnBw+SqNV6XnzUrNDvkRU=; b=JcUVol76zYmWH+HyS53rAwrf/pPGupHBhoD00tYXWJf7Rr6luH27HpX0J657+7G22+ /C8/71fIwyFfbPMv5lt+ege9TUUL7aXNGkkgz3PTmIZSZcY3J/oex1vkFj0+jQYbCtVE PxDAmJ6x6oCrUjNy4lrdmPGeBnuhFkcZEzGcLaeZ5t3x/NyLl8S2OroYv0Y6maQMYxZ3 Q9Vf1y8+Gkc4cHP/jyIhXdRUTU8Lv6jieIigTlA91qi4pSu7pI6D75G27MNiHIY2mJ3m UkzTI41tsCchUiGXr6bI2TJp84Ex1Q6TKnHXNice+be86qfvgBVw08bo0jCdevvDGPee Tc9g== X-Gm-Message-State: AC+VfDxCrsfQmh/Ex9nP7BZ1U7vtDWcuJtzA9u8RiY7V3LxeirIqOtx8 1PgbZJdqcBrmIVS8z5fD32qKvr69pCA= X-Google-Smtp-Source: ACHHUZ641wq2FLAPkyOcOf/2AaJ42rOTMPWQjI9sBcEhs8hmmrg4PFh1J2EuWrdbxT8OAjVL7D9g50h9mjM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:dbd2:0:b0:b99:cd69:cc32 with SMTP id g201-20020a25dbd2000000b00b99cd69cc32mr11391322ybf.0.1682960149191; Mon, 01 May 2023 09:55:49 -0700 (PDT) Date: Mon, 1 May 2023 09:54:28 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-19-surenb@google.com> Subject: [PATCH 18/40] lib: introduce support for page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E100940008 X-Stat-Signature: ix3rarkjxg5si4po73i1gs4g9xq3h8gw X-Rspam-User: X-HE-Tag: 1682960149-765043 X-HE-Meta: U2FsdGVkX1/oPa7AfDkq88r5JQFXz3iLQNqLHPbwzLErxx8jyW1pK5qPwCov7EemiUxWRKFWLo+S4IPCXE/Gwhy92q0HRZDogsw31dcFUeKjM4+1LeEDVmSMpqoDuALvHPCdKAQYGrsqLoHGsPJNC5fxHHg3kEbbQduy/GJNtwN1wAgkzhJKh/fwVnL6J2+saNJZzJ4ZuZTixtkBctlD/UPmkM+mEaBur/87GByEwklmLV0F1PVH7M88jloGXkFi5NXuJZdFyi72AMFfb+JmcPDGTWW/c+kCLeWY3ZPEueV3etBUnX6oL6/Uj+7eZm/J3hB3xMjSx7arcQNmn+uSgclsBmptLxBYMv6Az8t3EQOSCAuV61vq5//Wemgo6+Jv4xWdKs8L6MkXm5Isq63V7KULTmRHjBScfZf1Xlpv4MIwLGpLm0nhtGuNGBmNpml73wLrtfk75lJs4zyhxYJNiplKGPtXAAkjBZd8unXmCZe0krt9bauEaW8XPOXq5/NfhpObyrt2aQ6nEib6hN+kKcqlxSxC1jDOFIvI3MdQfS1Oc3bX7z74+fQ6Caryi1RC1eXFFiOQZlFB67Zt2k3mdLxQ7rv03ADLHpqVW9u6FmqIaO/fFVm6mbHbgNqRPW3iP7R7nYxWSHXypCocdORBKabwRmMZX4bMIL7UoTID4zgkDbtMMtYwh62yYMmjiCrxZon7+2rBb2XkqxEOir5xHsL4RYvHLnyInOTHhCN346gEsX+8zsiRSuNUQhpHC5Ow0pOcZc7STo3hdG+lNZGwEuDoRlWK1HekyrQPTucqAOxtsR668DXWF74eg546sHZpXJJYcKgTcvq0RH/xItDrt8B2ZL8QqIYePlpQvNBqn8vJr6/N8eY8XuWYl3glcQWCoEM8lmyi+Mn0rENEV1oajgPVXUk6XKR4gz8AlBpgRzJCYhk5OTIddE6WJUVsdP1d45k36tM5j3NfT4we2q8 mUnU7v9V 57wh5ZBZg3ElqBZcwzH5Acno5JbJbfqVpjb4/QOc7224zSf9n7lg1nEiT1gqaBJSLtD643hY3Trr9HBcdYRBFpXVcRHvQ/Q+KgSaERTk3k/HJ/ALAXPhooL6EgTNULAZAUYuhJkld2qDc8m72V/OSj7DritIiJq2YNqU8i7PxCnkwP1AnEpUz4nZc1IcO2iKTB8vNGm5JW2uH9Xwmgu+xGJSm3Oy1Vere0m1rEzXzOFmWKbr5ix631Nj27Kn6JJQktHvM0Qt4F1PR9TAALlM0bhJGZ/1/gghFmYqzPHNj+1lZXo+xR8EGpeIOihEzhFgT8xgKCBYkHdPj7FDoYi5kB0gDxr0DjKs0DTT3sFIqf8yAK1q5jswLyt4hJ80eJ7oc2KQEuvSbql1mS+0o8H2oCLvBA0Lv8rVrlTozvD1xNtm3EhkjxeNqF762xma71s+YQg1HyWIHvVVoccLKFoed5aIwolrRcBu1a62YNVERBShQCTXYRQNIvjJ5fHHm3nEj8TjS4II+DDQm2n4CPUjM8CTkXXSnH3Bp4wNCZ2quY5wZ/Qh80IW2hbiSZkch6i3SrAhcz/mnQA3meXsuHxBDEKkKy0Ou5kynFlSev2VEqZoYz53hidov/Emjvh2y7q8X+FFrf4Nn9dyOGTr5TSwo2Egn/QFrsrM2H/taCoACjog0zsGJkfGV7Q5fAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a page_ext field. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/pgalloc_tag.h | 33 +++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 1 + lib/alloc_tag.c | 17 +++++++++++++++++ mm/page_ext.c | 12 +++++++++--- 4 files changed, 60 insertions(+), 3 deletions(-) create mode 100644 include/linux/pgalloc_tag.h diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h new file mode 100644 index 000000000000..f8c7b6ef9c75 --- /dev/null +++ b/include/linux/pgalloc_tag.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * page allocation tagging + */ +#ifndef _LINUX_PGALLOC_TAG_H +#define _LINUX_PGALLOC_TAG_H + +#include +#include + +extern struct page_ext_operations page_alloc_tagging_ops; +struct page_ext *lookup_page_ext(const struct page *page); + +static inline union codetag_ref *get_page_tag_ref(struct page *page) +{ + if (page && mem_alloc_profiling_enabled()) { + struct page_ext *page_ext = lookup_page_ext(page); + + if (page_ext) + return (void *)page_ext + page_alloc_tagging_ops.offset; + } + return NULL; +} + +static inline void pgalloc_tag_dec(struct page *page, unsigned int order) +{ + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) + alloc_tag_sub(ref, PAGE_SIZE << order); +} + +#endif /* _LINUX_PGALLOC_TAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index da0a91ea6042..d3aa5ee0bf0d 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -967,6 +967,7 @@ config MEM_ALLOC_PROFILING depends on DEBUG_FS select CODE_TAGGING select LAZY_PERCPU_COUNTER + select PAGE_EXTENSION help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 3c4cfeb79862..4a0b95a46b2e 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include @@ -159,6 +160,22 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, struct codetag_ return module_unused; } +static __init bool need_page_alloc_tagging(void) +{ + return true; +} + +static __init void init_page_alloc_tagging(void) +{ +} + +struct page_ext_operations page_alloc_tagging_ops = { + .size = sizeof(union codetag_ref), + .need = need_page_alloc_tagging, + .init = init_page_alloc_tagging, +}; +EXPORT_SYMBOL(page_alloc_tagging_ops); + static int __init alloc_tag_init(void) { struct codetag_type *cttype; diff --git a/mm/page_ext.c b/mm/page_ext.c index dc1626be458b..eaf054ec276c 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -10,6 +10,7 @@ #include #include #include +#include /* * struct page extension @@ -82,6 +83,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT) &page_idle_ops, #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + &page_alloc_tagging_ops, +#endif #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif @@ -90,7 +94,7 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { unsigned long page_ext_size; static unsigned long total_usage; -static struct page_ext *lookup_page_ext(const struct page *page); +struct page_ext *lookup_page_ext(const struct page *page); bool early_page_ext __meminitdata; static int __init setup_early_page_ext(char *str) @@ -199,7 +203,7 @@ void __meminit pgdat_page_ext_init(struct pglist_data *pgdat) pgdat->node_page_ext = NULL; } -static struct page_ext *lookup_page_ext(const struct page *page) +struct page_ext *lookup_page_ext(const struct page *page) { unsigned long pfn = page_to_pfn(page); unsigned long index; @@ -219,6 +223,7 @@ static struct page_ext *lookup_page_ext(const struct page *page) MAX_ORDER_NR_PAGES); return get_entry(base, index); } +EXPORT_SYMBOL(lookup_page_ext); static int __init alloc_node_page_ext(int nid) { @@ -278,7 +283,7 @@ static bool page_ext_invalid(struct page_ext *page_ext) return !page_ext || (((unsigned long)page_ext & PAGE_EXT_INVALID) == PAGE_EXT_INVALID); } -static struct page_ext *lookup_page_ext(const struct page *page) +struct page_ext *lookup_page_ext(const struct page *page) { unsigned long pfn = page_to_pfn(page); struct mem_section *section = __pfn_to_section(pfn); @@ -295,6 +300,7 @@ static struct page_ext *lookup_page_ext(const struct page *page) return NULL; return get_entry(page_ext, pfn); } +EXPORT_SYMBOL(lookup_page_ext); static void *__meminit alloc_page_ext(size_t size, int nid) { From patchwork Mon May 1 16:54:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D833C77B61 for ; Mon, 1 May 2023 16:55:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0C38900017; Mon, 1 May 2023 12:55:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BBC15900002; Mon, 1 May 2023 12:55:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0E47900017; Mon, 1 May 2023 12:55:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 92A4E900002 for ; Mon, 1 May 2023 12:55:54 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6ED60120818 for ; Mon, 1 May 2023 16:55:54 +0000 (UTC) X-FDA: 80742288228.13.00A3ABB Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 8D11F40011 for ; Mon, 1 May 2023 16:55:52 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=hhMiGEI+; spf=pass (imf04.hostedemail.com: domain of 3F-9PZAYKCF4OQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3F-9PZAYKCF4OQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960152; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LeC1gX4P3RQjW1NYELsxA3PFCP+EdMyYyajg3MbW7ME=; b=G9gTYi18mXCz/+/2xjAX+YTgkIsgxYLKC00tQi8MzlKwiOmKwhcAGtZ1Mapb/8iNJvkzhG B1V+UIzPLKPf/wr7JQe9PCuzX906TogtA3AvmPIVcWHT8FCA4+7EnfMwQBrOcZ0BNAo76y XtvRru2LcC9+As2NU8fufWj/JiCdczA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960152; a=rsa-sha256; cv=none; b=4QXpZ2TVn3P8SzrQOAxCxviRGf2l82ZSp9j3gjV1ZWGSNAcFCcZx0df8oM/92lr/JQW8zO PzZ8ixqn2fWHkhfKfuuwLvlxUcrneFzr51dwfrtgrZGpk6CXhu+wDovzpjP8tTH58oezip ductBggLarQt72Hx+Y8dKC469zhxFS4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=hhMiGEI+; spf=pass (imf04.hostedemail.com: domain of 3F-9PZAYKCF4OQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3F-9PZAYKCF4OQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-556011695d1so45700607b3.1 for ; Mon, 01 May 2023 09:55:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960151; x=1685552151; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LeC1gX4P3RQjW1NYELsxA3PFCP+EdMyYyajg3MbW7ME=; b=hhMiGEI+67cfIr24zB7jRNLa5lThUyRZeDPUTiHuwGakXI2KeF15k1Fm45C272zpQV qXSafKQv0FF+5oEv4p+76TG6tqqAwOk8ME5PBGvQt4pav/Pg3OYN+Hq4oMgw3F6ZdSvk RmJQnKNLzSXmDQCmLfbgpYZMfqGB21dxyyqy9AGFiAL7FUCD/QaAfwt7HNGi0uvHJu82 6NgN1S+GtTfGCTv2eTqAiK6fBQsiqTLf20nkiQTXY3Jq4fbsHdFYt1d0Idxgqw9T5cZH WVbgLVG1ivKLyqdpAiO34AXCSDhpXsNhycXd99B9WHPgsZw1MFzj6S84fK28jMLqmH9m yXMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960151; x=1685552151; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LeC1gX4P3RQjW1NYELsxA3PFCP+EdMyYyajg3MbW7ME=; b=Wgiz9QwWqmpRyeklyQyx5zzi9zF0gw6tSNEz1Gk7ApL0SXuA4GDt/wEFI8PKlCeONr oXje7VRa3l3kzyoEdlQmMREEiv5HEwFEZG+TYLBbImZdJAJMtO2vE4vmzEx757E8RRKg X8tn2zEqHlda7L9gFs8fnr+JDhtegdweHxhMVtNM9nMY0RfUB8IZzLK92/+FGJMKzpWz YRrG8xEFETO72MTZN2V2CbT08M276S2r73zpUH6nGVrc0L2EkiC34V3k1P5FCjy3mU7R PNl5fEbeRM0KO/AquEGzBd2++sHeMYQDPzyosWEvU+DQm4oE6E+NWQTLMMr+Ce2mEvoy o7pw== X-Gm-Message-State: AC+VfDwt2o04+tVVbkVWKHMYxb8BEongXfstP6G+VCLaSeSReYHq+xbq dzKq9zPBGsN64NfzeVe2dNhLT5KcmrY= X-Google-Smtp-Source: ACHHUZ6D5LZLB8pLCoAY1h4u51zZbpSuAM6Xxjtp7fyAseeyuJw1iO6kg4WYdKeb69llNdIv4RH/p4cKKOU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a0d:ec4c:0:b0:55a:20a3:5ce3 with SMTP id r12-20020a0dec4c000000b0055a20a35ce3mr2799132ywn.3.1682960151497; Mon, 01 May 2023 09:55:51 -0700 (PDT) Date: Mon, 1 May 2023 09:54:29 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-20-surenb@google.com> Subject: [PATCH 19/40] change alloc_pages name in dma_map_ops to avoid name conflicts From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: 8D11F40011 X-Rspamd-Server: rspam09 X-Stat-Signature: mrnop6jbobr831nqxa914tmb7cdtturn X-HE-Tag: 1682960152-773461 X-HE-Meta: U2FsdGVkX1/S2RAWhJHEgvPASOet4QhA2M5Z45cT6h9wn8qmLm6N6zrQNYfcdsiArYh73x21+B16FkMXa0JRaKp+T1OigLtKyCz2AjJVMHYTF02kHzoSkJYUjVEMjrP6pLngasR6PqIq2W7D1hfxMKpB8JJK/hfgsxF9hr5QRgcJC7Dbk6eI//i4rGnNYkxUjwD0ulh4EvV9tpMfg4Dio0jeO90Nq2/Xa9K/trz4UlG8DuebfWEJ9YVLZh3aLUoLXHVne9aJtniCvbPnUNmuKmZ1zTK4ViVskDyxjqLrgBIXdxFxRhbHHRMji+Bt8xw0In/v67kUZgNdrDJPQKAw7RdsGkpWiLEO4StJBKs6ZCBllxNH3y9SUhs4iV07FeDkTBuAnIsm8jAlPK6nSBoi7cBD5ZN1chmwkAg0cfBY6byj9FnLTs4tt8uziIngG8g7XaX2ly7xOzRN8sDYy/HHBCY5mBtPLJHI+UTuftq2TsRyWxoiAc8hPzXdQej76VbeQ4v7hHwiNhd6EithaeeF+h4upy3nukQXXwZykSpynLxg1CNx0L4KEDsNt/EKJLXdrd3Wt2k2+TqAywY2PMC60gRosZPnN5oyRrZV/YEiba5ttsY9TaWl4NmYs64ckXeDxRqNo6TaFaxk4Fl+E/7EJf6fYWfrA103VzNbgplhwrb8V3sy+Qv6sewEYwhze3Joxdo0KtQtjzkKuICadcCM2GA/+dSsemonltot4EoJoZJXN8AH5cq9vqbVEwctUQk1F0fy0AnhMCKQVX8D8gobTkRGO73rY29y5vjIL4ZUQGRH8Bgv0C8VZzMXRiS3GbHqs2MKuMS09EEdclj2w7nF6/Z/SpgfTZSBWJjnX06QACgAk0a3AgTatgFED3GVP0yReqhyntSQu0r8mzII2zbq6xJFJd2YMMjmz9pSaznpjITa1UGI+6scq4MaBbERTQLoGGMBWTlfO0SvJwloR4g LDo8zrpF 8FMPd1eG6xUw83veNUyd1eIJ4oqVA28fwTXp4/jbU1iXCxGlZqP7IKkTDQ9tLl/I+iBMebuWrzanVJeVBMrrWfKtpT82N3iO6PJXcnelY8PaCHMT6Y05wwaRImF7fgoKlJkBSHmNQIkX+ZhqYLLUM5poXXdb0I6wG5W+jBgM9CAU0b6VtMf7YE8dV7JoXwOWYVbkoaWE24a8tSEWelrRWXn86YRM6RYuKyuZY/4YHVFtQe7y1ibMo61LCOLVrpDAkHxzifyW43kHXxnQzhVv2nlxuDaBc6MJTeaXIvWDKLLYhikesCRbUIzf34Wa4aZmSG0pn0KHSPJ/iv6etq0DY5bvKnKTj+pSiKWhjflVNU+Ey+SN/hcXaxIoPzHBqOMyzJuNQNy3PsAhHLkazwo+ibhQWHJJBRY94ag0MnFDm56Db5102EKymn8eHECjVZroN8PXSWe0GtGhlqyMJas08N7LVcYXqUIjvRahOOJH44bTNRYXXfsR/lVAykn2k1RTfGPSCV0hBxMqM+O3YedZYkxnlkUNF2aJijejPz07i2uxyU3y6bP8KEF1Xa0nfxLSS/YuZ/3cbJaMUVyIxkJk5VBSq2gF1NBaBhgELFCiQUEjfAZraWYkyfxOHpt45SxKb+d9G/Y+XfD0nQTNCsf0xHb1V059hmhTKZwxcPwNVG5Pss/qHW2dbLea5Gy/4Rw7SO2uZkjfAPybTFETEPYfgLsN7f30h7W+hDe3h X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After redefining alloc_pages, all uses of that name are being replaced. Change the conflicting names to prevent preprocessor from replacing them when it's not intended. Signed-off-by: Suren Baghdasaryan --- arch/x86/kernel/amd_gart_64.c | 2 +- drivers/iommu/dma-iommu.c | 2 +- drivers/xen/grant-dma-ops.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/mapping.c | 4 ++-- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index 56a917df410d..842a0ec5eaa9 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -676,7 +676,7 @@ static const struct dma_map_ops gart_dma_ops = { .get_sgtable = dma_common_get_sgtable, .dma_supported = dma_direct_supported, .get_required_mask = dma_direct_get_required_mask, - .alloc_pages = dma_direct_alloc_pages, + .alloc_pages_op = dma_direct_alloc_pages, .free_pages = dma_direct_free_pages, }; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 7a9f0b0bddbd..76a9d5ca4eee 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1556,7 +1556,7 @@ static const struct dma_map_ops iommu_dma_ops = { .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .alloc_noncontiguous = iommu_dma_alloc_noncontiguous, .free_noncontiguous = iommu_dma_free_noncontiguous, diff --git a/drivers/xen/grant-dma-ops.c b/drivers/xen/grant-dma-ops.c index 9784a77fa3c9..6c7d984f164d 100644 --- a/drivers/xen/grant-dma-ops.c +++ b/drivers/xen/grant-dma-ops.c @@ -282,7 +282,7 @@ static int xen_grant_dma_supported(struct device *dev, u64 mask) static const struct dma_map_ops xen_grant_dma_ops = { .alloc = xen_grant_dma_alloc, .free = xen_grant_dma_free, - .alloc_pages = xen_grant_dma_alloc_pages, + .alloc_pages_op = xen_grant_dma_alloc_pages, .free_pages = xen_grant_dma_free_pages, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 67aa74d20162..5ab2616153f0 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -403,6 +403,6 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .dma_supported = xen_swiotlb_dma_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 31f114f486c4..d741940dcb3b 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -27,7 +27,7 @@ struct dma_map_ops { unsigned long attrs); void (*free)(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs); - struct page *(*alloc_pages)(struct device *dev, size_t size, + struct page *(*alloc_pages_op)(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp); void (*free_pages)(struct device *dev, size_t size, struct page *vaddr, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 9a4db5cce600..fc42930af14b 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -570,9 +570,9 @@ static struct page *__dma_alloc_pages(struct device *dev, size_t size, size = PAGE_ALIGN(size); if (dma_alloc_direct(dev, ops)) return dma_direct_alloc_pages(dev, size, dma_handle, dir, gfp); - if (!ops->alloc_pages) + if (!ops->alloc_pages_op) return NULL; - return ops->alloc_pages(dev, size, dma_handle, dir, gfp); + return ops->alloc_pages_op(dev, size, dma_handle, dir, gfp); } struct page *dma_alloc_pages(struct device *dev, size_t size, From patchwork Mon May 1 16:54:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6541EC77B73 for ; Mon, 1 May 2023 16:55:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 073BF900018; Mon, 1 May 2023 12:55:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0232B900002; Mon, 1 May 2023 12:55:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2E3B900018; Mon, 1 May 2023 12:55:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D1166900002 for ; Mon, 1 May 2023 12:55:56 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9417C407F3 for ; Mon, 1 May 2023 16:55:56 +0000 (UTC) X-FDA: 80742288312.20.E348471 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf23.hostedemail.com (Postfix) with ESMTP id B64B6140006 for ; Mon, 1 May 2023 16:55:54 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=t3klBo0f; spf=pass (imf23.hostedemail.com: domain of 3Ge9PZAYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Ge9PZAYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960154; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Mri2z+LvwzIobpEl78ZbT4L3ds7H/B8V3mrszruJwqU=; b=AGAjS1Ta+bTy/dtKMPQbFuu1ln1WNoPHibPV2jyCXxAp0g5+CP0uOFkKyL8+rKS5hfIrEZ fmmXyQPaVDdOSeuK9Tg8tr3kFSgr4EuDwOEgWRc1N5wEM21+SeQvrx2VH3PO6I+UJ75nCS rZSlO+8eGIiAj2I7aRwHu7V6rBtdWdE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=t3klBo0f; spf=pass (imf23.hostedemail.com: domain of 3Ge9PZAYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Ge9PZAYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960154; a=rsa-sha256; cv=none; b=j9RlsWWtVFqGqsvxnTAroEkwzNpB3NBQFPHpPWhv7djk5liTTzynhjx+V6HoPMVl0iXHZp SJKCOt64fkn0EYrzpB1sypVEUQAk/NeGceNaBhwc9s3Twc/HdZy0nItyHrrsOQhx6Lt3Ya XJfVA9sukURC5wvXXP737Q7RgW3CADQ= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-54c2999fdc7so53697647b3.2 for ; Mon, 01 May 2023 09:55:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960154; x=1685552154; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Mri2z+LvwzIobpEl78ZbT4L3ds7H/B8V3mrszruJwqU=; b=t3klBo0fosZzY+6Mn34UkeRIZ8VlBk56iQvcqMHnlZoMdspE+7NdZvE4k2vyn3ecSC wFY0JV6O1SpEH2hrWnI2ZwvcFG9UuRrWrZ6FC4svxPT3iRFpboq1m9ULo8cCPr6ZaPHa Y3x/6J6egKUSo5bKdDK02WHSlKy43vJBZZpKM6scw7eAqstx/nyrECEQlmlwNzLP6+c1 x6typBX0rqpw1oi/w9vTNxPJlYF7ooBhaib/snmI2nYMcyeE2e2JEAGsaa+GQO7ffHeg n1W4cCFYszTARUhVglWM9nb/Nb2Udwpan9HFpZ/k9Jx52FO/vN20r/r10gSwNfC0onxH nPNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960154; x=1685552154; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Mri2z+LvwzIobpEl78ZbT4L3ds7H/B8V3mrszruJwqU=; b=YanPtc1KLcIIzLHQJXm6wUEYFL3cQA3W+lmQ3/eQmGhzd05WEHazMi2j2pD+t/Ed55 6X5lZpJsb+XIk/91rLnSZHs26sghDY/8et5ziNibEiCYIrKpJm/PdryoIZggDe0a3S4o p7Pbrf0djYgEIHO0jaHblvyUWgKaBlxjDBQPQV38Ovz2Oz74SgtgBCdOXqVo8+//F7ox R5xoUlK7YpiKt/BG0MhasxPTgwUNuo8ljAikXat26pFwbIpxvQrNjQCEDnevCUYVaY4C LSZpEZ+pqUeVjdZ1FqulLPJZt08+skkCdhnQzpdLZ0udu08wXsf1jq5VZ3w6PMez1FS4 mzBg== X-Gm-Message-State: AC+VfDwqUPHITCRJxHGybug5gYhKitPfvd8N6qrSDrPuX9P8CO8GMeN9 lRS/+lFu9gb4FO6QUxoc3WjrcNp1BuQ= X-Google-Smtp-Source: ACHHUZ5Am/AbJJDHaPOyxBcK1szyPNP1NGMyyYGzfCHWCVUgDFAOjYjnaEfI2KDePZpQQp6gXZXY+DFEDT8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:9f09:0:b0:559:e830:60f1 with SMTP id s9-20020a819f09000000b00559e83060f1mr4600345ywn.8.1682960153830; Mon, 01 May 2023 09:55:53 -0700 (PDT) Date: Mon, 1 May 2023 09:54:30 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-21-surenb@google.com> Subject: [PATCH 20/40] mm: enable page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B64B6140006 X-Stat-Signature: rphsqi94y8df9ew454d4yes8nf3abae6 X-HE-Tag: 1682960154-361579 X-HE-Meta: U2FsdGVkX1/LAYgLvz1xElMueag4iNfplLsxqBgvOr1T+yO0zDkTIya5EUqivucfnupOsLqlpa/05ykUnLXryeXqDxopztdFHgmVrBKysMWTTtBKW67g77S4BDh4ZT+WIuXvbyL8v+r8td8m02iZEOJAjzmHEfADVI3AFeLmEkJTWSpUO2y3L0BWUKkthFswgedUElhfNl3iKYxlQKBoiEfU34n2WgLHEq0O6eyGyaRaGjijwceCvnC9ZHb374izfUmircQFApCK8PI01vX2vJ+KnAvUEY4hoEFxDDVki3t9jVwTLYBM37o0WoXyRFjzRjJ15Ba35UwzvN1MwBXBuYRYiV5uTr3duozGsSDOlDthSPwT/M3QIkHuBYB9LTKebuqZEikE6TJUPGn9MeLrOF3B1Mxt7jBOS98ij6Ww/siPaJEGNV/Fc+im+ON4ldJjwsEJZLxKVebY2DTucunsP2+LxSc9G19I3ToWF7e91JyarsEw7z7/TeXk6LB1llgS3zuRtswbvXF9t4cOzw6s1hSm4mysXTxSgSWpj0tZhqq94RkyLYPd9Flg7GQ9ZzPTF1+K8VvdF3DTC0N26xhH/Mz6+FNAI3zNYtlR8DuYZXzMHiV7x0jxsa5mt6w1ghc5oHGJ5dprdWJvPCGkNYivwrPnE4SUbUVsoW6C0xya/EtxbvXsjqKFCDETLVprF+ubXqdujLp1YXta7lKy+RWebYQ1ocf3FDK3kUqbdyppPxfggeolLk/0WJ5ySqDA4eTNNelR62wlLCbOThKaYVB89qakIjdFiWK9Sczkzu8iyOo8paRmHG23fn7PpF851PF0hO9nYZG26Ae/eldkwNBE3YOFA/y4z+91OeKnXT48KnWhL86r3Ls7bPEAfUc6fjPm1By38jmR/RHLioNt4Hg5k+hrSiMWd42C76nTCs24sU0EwFQMo9ZM+x8Szvh9w4ylmqECmwglXfnR6/tv/de 25wpGfIs G8i+h97AQjbxJGCfqCg0r3vDditMUPPWD5mwY8LEP1YnLrQ+YqgV2deAUratlMRbwbvYetBdAroTO7DKsAnhlv4AxPw8PQZ4jTCt20CHq6NARwBOhFcuWmUXLLv3VcsLk9IuhRD7Z/EqBkPSv0A4++VQZri6tO4xuy45GVmpeFad+Ockw9vkCGafcqmEFsVLhVmg9r6BiF0ZqZmDolcMet6Mrf1IAjDCopQ1rZ5mvnZ+jT90a4DcdqQvXqNUZ7engbgGPXMUZrpK8tUeU+LMRdTEovtg+42YvnZTY4IxgpZp4HM3BasuCZn/CsFV+6nYeaOq5CeAwtYrBB6/xYzXtjixKHyL6Tvye9iplpgubDfrH4rr+/QEpaK7OFyK0gmnOb955Ak0QFBm0Iey2GLHDbARHQWWwVCEL5e5gpPkzbbP950Y7T+HQelFkrvIcOn/tdY6nv6WcvrE9hVKLthBu1E2JY+P3sF94CbbwVGRU+5BxrGOzyNFe1uTzbfyY1g4Io1s7WwZM6oo9HgsjI/xduyeFqSlY9cvN4nzM9zoYboGUA8tm7j+6kD616qfCmt4iPKkYAkkjFxUWHhKPk2qkN/Qgg/gIJDtfZ95R+UOO1Svpd2HusLneFJhBHmPwgbooEo5Iy6hTyV08FrRiDvWars3gY0F8T1N+AV1fbq5U7ZAgWtQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Redefine page allocators to record allocation tags upon their invocation. Instrument post_alloc_hook and free_pages_prepare to modify current allocation tag. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 11 ++++ include/linux/gfp.h | 123 +++++++++++++++++++++++++----------- include/linux/page_ext.h | 1 - include/linux/pagemap.h | 9 ++- include/linux/pgalloc_tag.h | 38 +++++++++-- mm/compaction.c | 9 ++- mm/filemap.c | 6 +- mm/mempolicy.c | 30 ++++----- mm/mm_init.c | 1 + mm/page_alloc.c | 73 ++++++++++++--------- 10 files changed, 208 insertions(+), 93 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index d913f8d9a7d8..07922d81b641 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -102,4 +102,15 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, #endif +#define alloc_hooks(_do_alloc, _res_type, _err) \ +({ \ + _res_type _res; \ + DEFINE_ALLOC_TAG(_alloc_tag, _old); \ + \ + _res = _do_alloc; \ + alloc_tag_restore(&_alloc_tag, _old); \ + _res; \ +}) + + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/gfp.h b/include/linux/gfp.h index ed8cb537c6a7..0cb4a515109a 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -6,6 +6,8 @@ #include #include +#include +#include struct vm_area_struct; @@ -174,42 +176,57 @@ static inline void arch_free_page(struct page *page, int order) { } static inline void arch_alloc_page(struct page *page, int order) { } #endif -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, +struct page *_alloc_pages2(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +#define __alloc_pages(_gfp, _order, _preferred_nid, _nodemask) \ + alloc_hooks(_alloc_pages2(_gfp, _order, _preferred_nid, \ + _nodemask), struct page *, NULL) + +struct folio *_folio_alloc2(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); +#define __folio_alloc(_gfp, _order, _preferred_nid, _nodemask) \ + alloc_hooks(_folio_alloc2(_gfp, _order, _preferred_nid, \ + _nodemask), struct folio *, NULL) -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long _alloc_pages_bulk(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array); - -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +#define __alloc_pages_bulk(_gfp, _preferred_nid, _nodemask, _nr_pages, \ + _page_list, _page_array) \ + alloc_hooks(_alloc_pages_bulk(_gfp, _preferred_nid, \ + _nodemask, _nr_pages, \ + _page_list, _page_array), \ + unsigned long, 0) + +unsigned long _alloc_pages_bulk_array_mempolicy(gfp_t gfp, unsigned long nr_pages, struct page **page_array); +#define alloc_pages_bulk_array_mempolicy(_gfp, _nr_pages, _page_array) \ + alloc_hooks(_alloc_pages_bulk_array_mempolicy(_gfp, \ + _nr_pages, _page_array), \ + unsigned long, 0) /* Bulk allocate order-0 pages */ -static inline unsigned long -alloc_pages_bulk_list(gfp_t gfp, unsigned long nr_pages, struct list_head *list) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, list, NULL); -} +#define alloc_pages_bulk_list(_gfp, _nr_pages, _list) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, _list, NULL) -static inline unsigned long -alloc_pages_bulk_array(gfp_t gfp, unsigned long nr_pages, struct page **page_array) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, NULL, page_array); -} +#define alloc_pages_bulk_array(_gfp, _nr_pages, _page_array) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, NULL, _page_array) static inline unsigned long -alloc_pages_bulk_array_node(gfp_t gfp, int nid, unsigned long nr_pages, struct page **page_array) +_alloc_pages_bulk_array_node(gfp_t gfp, int nid, unsigned long nr_pages, struct page **page_array) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_bulk(gfp, nid, NULL, nr_pages, NULL, page_array); + return _alloc_pages_bulk(gfp, nid, NULL, nr_pages, NULL, page_array); } +#define alloc_pages_bulk_array_node(_gfp, _nid, _nr_pages, _page_array) \ + alloc_hooks(_alloc_pages_bulk_array_node(_gfp, _nid, _nr_pages, _page_array), \ + unsigned long, 0) + static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) { gfp_t warn_gfp = gfp_mask & (__GFP_THISNODE|__GFP_NOWARN); @@ -229,21 +246,25 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) * online. For more general interface, see alloc_pages_node(). */ static inline struct page * -__alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order) +_alloc_pages_node2(int nid, gfp_t gfp_mask, unsigned int order) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp_mask); - return __alloc_pages(gfp_mask, order, nid, NULL); + return _alloc_pages2(gfp_mask, order, nid, NULL); } +#define __alloc_pages_node(_nid, _gfp_mask, _order) \ + alloc_hooks(_alloc_pages_node2(_nid, _gfp_mask, _order), \ + struct page *, NULL) + static inline struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp); - return __folio_alloc(gfp, order, nid, NULL); + return _folio_alloc2(gfp, order, nid, NULL); } /* @@ -251,32 +272,45 @@ struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) * prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, +static inline struct page *_alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_node(nid, gfp_mask, order); + return _alloc_pages_node2(nid, gfp_mask, order); } +#define alloc_pages_node(_nid, _gfp_mask, _order) \ + alloc_hooks(_alloc_pages_node(_nid, _gfp_mask, _order), \ + struct page *, NULL) + #ifdef CONFIG_NUMA -struct page *alloc_pages(gfp_t gfp, unsigned int order); -struct folio *folio_alloc(gfp_t gfp, unsigned order); -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct page *_alloc_pages(gfp_t gfp, unsigned int order); +struct folio *_folio_alloc(gfp_t gfp, unsigned int order); +struct folio *_vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage); #else -static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) +static inline struct page *_alloc_pages(gfp_t gfp_mask, unsigned int order) { - return alloc_pages_node(numa_node_id(), gfp_mask, order); + return _alloc_pages_node(numa_node_id(), gfp_mask, order); } -static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order) +static inline struct folio *_folio_alloc(gfp_t gfp, unsigned int order) { return __folio_alloc_node(gfp, order, numa_node_id()); } -#define vma_alloc_folio(gfp, order, vma, addr, hugepage) \ - folio_alloc(gfp, order) +#define _vma_alloc_folio(gfp, order, vma, addr, hugepage) \ + _folio_alloc(gfp, order) #endif + +#define alloc_pages(_gfp, _order) \ + alloc_hooks(_alloc_pages(_gfp, _order), struct page *, NULL) +#define folio_alloc(_gfp, _order) \ + alloc_hooks(_folio_alloc(_gfp, _order), struct folio *, NULL) +#define vma_alloc_folio(_gfp, _order, _vma, _addr, _hugepage) \ + alloc_hooks(_vma_alloc_folio(_gfp, _order, _vma, _addr, \ + _hugepage), struct folio *, NULL) + #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0) static inline struct page *alloc_page_vma(gfp_t gfp, struct vm_area_struct *vma, unsigned long addr) @@ -286,12 +320,21 @@ static inline struct page *alloc_page_vma(gfp_t gfp, return &folio->page; } -extern unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order); -extern unsigned long get_zeroed_page(gfp_t gfp_mask); +extern unsigned long _get_free_pages(gfp_t gfp_mask, unsigned int order); +#define __get_free_pages(_gfp_mask, _order) \ + alloc_hooks(_get_free_pages(_gfp_mask, _order), unsigned long, 0) +extern unsigned long _get_zeroed_page(gfp_t gfp_mask); +#define get_zeroed_page(_gfp_mask) \ + alloc_hooks(_get_zeroed_page(_gfp_mask), unsigned long, 0) -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1); +void *_alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1); +#define alloc_pages_exact(_size, _gfp_mask) \ + alloc_hooks(_alloc_pages_exact(_size, _gfp_mask), void *, NULL) void free_pages_exact(void *virt, size_t size); -__meminit void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); + +__meminit void *_alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); +#define alloc_pages_exact_nid(_nid, _size, _gfp_mask) \ + alloc_hooks(_alloc_pages_exact_nid(_nid, _size, _gfp_mask), void *, NULL) #define __get_free_page(gfp_mask) \ __get_free_pages((gfp_mask), 0) @@ -354,10 +397,16 @@ static inline bool pm_suspended_storage(void) #ifdef CONFIG_CONTIG_ALLOC /* The below functions must be run on a range from a single zone. */ -extern int alloc_contig_range(unsigned long start, unsigned long end, +extern int _alloc_contig_range(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask); -extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask); +#define alloc_contig_range(_start, _end, _migratetype, _gfp_mask) \ + alloc_hooks(_alloc_contig_range(_start, _end, _migratetype, \ + _gfp_mask), int, -ENOMEM) +extern struct page *_alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask); +#define alloc_contig_pages(_nr_pages, _gfp_mask, _nid, _nodemask) \ + alloc_hooks(_alloc_contig_pages(_nr_pages, _gfp_mask, _nid, \ + _nodemask), struct page *, NULL) #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index 67314f648aeb..cff15ee5440e 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -4,7 +4,6 @@ #include #include -#include struct pglist_data; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index a56308a9d1a4..b2efafa001f8 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -467,14 +467,17 @@ static inline void *detach_page_private(struct page *page) } #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order); +struct folio *_filemap_alloc_folio(gfp_t gfp, unsigned int order); #else -static inline struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +static inline struct folio *_filemap_alloc_folio(gfp_t gfp, unsigned int order) { - return folio_alloc(gfp, order); + return _folio_alloc(gfp, order); } #endif +#define filemap_alloc_folio(_gfp, _order) \ + alloc_hooks(_filemap_alloc_folio(_gfp, _order), struct folio *, NULL) + static inline struct page *__page_cache_alloc(gfp_t gfp) { return &filemap_alloc_folio(gfp, 0)->page; diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index f8c7b6ef9c75..567327c1c46f 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -6,28 +6,58 @@ #define _LINUX_PGALLOC_TAG_H #include + +#ifdef CONFIG_MEM_ALLOC_PROFILING + #include extern struct page_ext_operations page_alloc_tagging_ops; -struct page_ext *lookup_page_ext(const struct page *page); +extern struct page_ext *page_ext_get(struct page *page); +extern void page_ext_put(struct page_ext *page_ext); + +static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +{ + return (void *)page_ext + page_alloc_tagging_ops.offset; +} + +static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) +{ + return (void *)ref - page_alloc_tagging_ops.offset; +} static inline union codetag_ref *get_page_tag_ref(struct page *page) { if (page && mem_alloc_profiling_enabled()) { - struct page_ext *page_ext = lookup_page_ext(page); + struct page_ext *page_ext = page_ext_get(page); if (page_ext) - return (void *)page_ext + page_alloc_tagging_ops.offset; + return codetag_ref_from_page_ext(page_ext); } return NULL; } +static inline void put_page_tag_ref(union codetag_ref *ref) +{ + if (ref) + page_ext_put(page_ext_from_codetag_ref(ref)); +} + static inline void pgalloc_tag_dec(struct page *page, unsigned int order) { union codetag_ref *ref = get_page_tag_ref(page); - if (ref) + if (ref) { alloc_tag_sub(ref, PAGE_SIZE << order); + put_page_tag_ref(ref); + } } +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } +static inline void put_page_tag_ref(union codetag_ref *ref) {} +#define pgalloc_tag_dec(__page, __size) do {} while (0) + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + #endif /* _LINUX_PGALLOC_TAG_H */ diff --git a/mm/compaction.c b/mm/compaction.c index c8bcdea15f5f..32707fb62495 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1684,7 +1684,7 @@ static void isolate_freepages(struct compact_control *cc) * This is a migrate-callback that "allocates" freepages by taking pages * from the isolated freelists in the block we are migrating to. */ -static struct page *compaction_alloc(struct page *migratepage, +static struct page *_compaction_alloc(struct page *migratepage, unsigned long data) { struct compact_control *cc = (struct compact_control *)data; @@ -1704,6 +1704,13 @@ static struct page *compaction_alloc(struct page *migratepage, return freepage; } +static struct page *compaction_alloc(struct page *migratepage, + unsigned long data) +{ + return alloc_hooks(_compaction_alloc(migratepage, data), + struct page *, NULL); +} + /* * This is a migrate-callback that "frees" freepages back to the isolated * freelist. All pages on the freelist are from the same zone, so there is no diff --git a/mm/filemap.c b/mm/filemap.c index a34abfe8c654..f0f8b782d172 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -958,7 +958,7 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, EXPORT_SYMBOL_GPL(filemap_add_folio); #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +struct folio *_filemap_alloc_folio(gfp_t gfp, unsigned int order) { int n; struct folio *folio; @@ -973,9 +973,9 @@ struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) return folio; } - return folio_alloc(gfp, order); + return _folio_alloc(gfp, order); } -EXPORT_SYMBOL(filemap_alloc_folio); +EXPORT_SYMBOL(_filemap_alloc_folio); #endif /* diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 2068b594dc88..80cd33811641 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2141,7 +2141,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, } /** - * vma_alloc_folio - Allocate a folio for a VMA. + * _vma_alloc_folio - Allocate a folio for a VMA. * @gfp: GFP flags. * @order: Order of the folio. * @vma: Pointer to VMA or NULL if not available. @@ -2155,7 +2155,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, * * Return: The folio on success or NULL if allocation fails. */ -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *_vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage) { struct mempolicy *pol; @@ -2240,10 +2240,10 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, out: return folio; } -EXPORT_SYMBOL(vma_alloc_folio); +EXPORT_SYMBOL(_vma_alloc_folio); /** - * alloc_pages - Allocate pages. + * _alloc_pages - Allocate pages. * @gfp: GFP flags. * @order: Power of two of number of pages to allocate. * @@ -2256,7 +2256,7 @@ EXPORT_SYMBOL(vma_alloc_folio); * flags are used. * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages(gfp_t gfp, unsigned order) +struct page *_alloc_pages(gfp_t gfp, unsigned int order) { struct mempolicy *pol = &default_policy; struct page *page; @@ -2274,15 +2274,15 @@ struct page *alloc_pages(gfp_t gfp, unsigned order) page = alloc_pages_preferred_many(gfp, order, policy_node(gfp, pol, numa_node_id()), pol); else - page = __alloc_pages(gfp, order, + page = _alloc_pages2(gfp, order, policy_node(gfp, pol, numa_node_id()), policy_nodemask(gfp, pol)); return page; } -EXPORT_SYMBOL(alloc_pages); +EXPORT_SYMBOL(_alloc_pages); -struct folio *folio_alloc(gfp_t gfp, unsigned order) +struct folio *_folio_alloc(gfp_t gfp, unsigned int order) { struct page *page = alloc_pages(gfp | __GFP_COMP, order); @@ -2290,7 +2290,7 @@ struct folio *folio_alloc(gfp_t gfp, unsigned order) prep_transhuge_page(page); return (struct folio *)page; } -EXPORT_SYMBOL(folio_alloc); +EXPORT_SYMBOL(_folio_alloc); static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, struct mempolicy *pol, unsigned long nr_pages, @@ -2309,13 +2309,13 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, for (i = 0; i < nodes; i++) { if (delta) { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = _alloc_pages_bulk(gfp, interleave_nodes(pol), NULL, nr_pages_per_node + 1, NULL, page_array); delta--; } else { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = _alloc_pages_bulk(gfp, interleave_nodes(pol), NULL, nr_pages_per_node, NULL, page_array); } @@ -2337,11 +2337,11 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - nr_allocated = __alloc_pages_bulk(preferred_gfp, nid, &pol->nodes, + nr_allocated = _alloc_pages_bulk(preferred_gfp, nid, &pol->nodes, nr_pages, NULL, page_array); if (nr_allocated < nr_pages) - nr_allocated += __alloc_pages_bulk(gfp, numa_node_id(), NULL, + nr_allocated += _alloc_pages_bulk(gfp, numa_node_id(), NULL, nr_pages - nr_allocated, NULL, page_array + nr_allocated); return nr_allocated; @@ -2353,7 +2353,7 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, * It can accelerate memory allocation especially interleaving * allocate memory. */ -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long _alloc_pages_bulk_array_mempolicy(gfp_t gfp, unsigned long nr_pages, struct page **page_array) { struct mempolicy *pol = &default_policy; @@ -2369,7 +2369,7 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, return alloc_pages_bulk_array_preferred_many(gfp, numa_node_id(), pol, nr_pages, page_array); - return __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()), + return _alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()), policy_nodemask(gfp, pol), nr_pages, NULL, page_array); } diff --git a/mm/mm_init.c b/mm/mm_init.c index 7f7f9c677854..42135fad4d8a 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include "internal.h" diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9de2a18519a1..edd35500f7f6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -74,6 +74,7 @@ #include #include #include +#include #include #include #include @@ -657,6 +658,7 @@ static inline bool pcp_allowed_order(unsigned int order) static inline void free_the_page(struct page *page, unsigned int order) { + if (pcp_allowed_order(order)) /* Via pcp? */ free_unref_page(page, order); else @@ -1259,6 +1261,7 @@ static __always_inline bool free_pages_prepare(struct page *page, __memcg_kmem_uncharge_page(page, order); reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_dec(page, order); return false; } @@ -1301,6 +1304,7 @@ static __always_inline bool free_pages_prepare(struct page *page, page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_dec(page, order); if (!PageHighMem(page)) { debug_check_no_locks_freed(page_address(page), @@ -1669,6 +1673,9 @@ inline void post_alloc_hook(struct page *page, unsigned int order, bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) && !should_skip_init(gfp_flags); bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS); +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref *ref; +#endif int i; set_page_private(page, 0); @@ -1721,6 +1728,14 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + ref = get_page_tag_ref(page); + if (ref) { + alloc_tag_add(ref, current->alloc_tag, PAGE_SIZE << order); + put_page_tag_ref(ref); + } +#endif } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, @@ -4568,7 +4583,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, * * Returns the number of pages on the list or array. */ -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long _alloc_pages_bulk(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array) @@ -4704,7 +4719,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, pcp_trylock_finish(UP_flags); failed: - page = __alloc_pages(gfp, 0, preferred_nid, nodemask); + page = _alloc_pages2(gfp, 0, preferred_nid, nodemask); if (page) { if (page_list) list_add(&page->lru, page_list); @@ -4715,12 +4730,12 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, goto out; } -EXPORT_SYMBOL_GPL(__alloc_pages_bulk); +EXPORT_SYMBOL_GPL(_alloc_pages_bulk); /* * This is the 'heart' of the zoned buddy allocator. */ -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, +struct page *_alloc_pages2(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { struct page *page; @@ -4783,41 +4798,41 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, return page; } -EXPORT_SYMBOL(__alloc_pages); +EXPORT_SYMBOL(_alloc_pages2); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +struct folio *_folio_alloc2(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { - struct page *page = __alloc_pages(gfp | __GFP_COMP, order, + struct page *page = _alloc_pages2(gfp | __GFP_COMP, order, preferred_nid, nodemask); if (page && order > 1) prep_transhuge_page(page); return (struct folio *)page; } -EXPORT_SYMBOL(__folio_alloc); +EXPORT_SYMBOL(_folio_alloc2); /* * Common helper functions. Never use with __GFP_HIGHMEM because the returned * address cannot represent highmem pages. Use alloc_pages and then kmap if * you need to access high mem. */ -unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order) +unsigned long _get_free_pages(gfp_t gfp_mask, unsigned int order) { struct page *page; - page = alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order); + page = _alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order); if (!page) return 0; return (unsigned long) page_address(page); } -EXPORT_SYMBOL(__get_free_pages); +EXPORT_SYMBOL(_get_free_pages); -unsigned long get_zeroed_page(gfp_t gfp_mask) +unsigned long _get_zeroed_page(gfp_t gfp_mask) { - return __get_free_page(gfp_mask | __GFP_ZERO); + return _get_free_pages(gfp_mask | __GFP_ZERO, 0); } -EXPORT_SYMBOL(get_zeroed_page); +EXPORT_SYMBOL(_get_zeroed_page); /** * __free_pages - Free pages allocated with alloc_pages(). @@ -5009,7 +5024,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, } /** - * alloc_pages_exact - allocate an exact number physically-contiguous pages. + * _alloc_pages_exact - allocate an exact number physically-contiguous pages. * @size: the number of bytes to allocate * @gfp_mask: GFP flags for the allocation, must not contain __GFP_COMP * @@ -5023,7 +5038,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, * * Return: pointer to the allocated area or %NULL in case of error. */ -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) +void *_alloc_pages_exact(size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); unsigned long addr; @@ -5031,13 +5046,13 @@ void *alloc_pages_exact(size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - addr = __get_free_pages(gfp_mask, order); + addr = _get_free_pages(gfp_mask, order); return make_alloc_exact(addr, order, size); } -EXPORT_SYMBOL(alloc_pages_exact); +EXPORT_SYMBOL(_alloc_pages_exact); /** - * alloc_pages_exact_nid - allocate an exact number of physically-contiguous + * _alloc_pages_exact_nid - allocate an exact number of physically-contiguous * pages on a node. * @nid: the preferred node ID where memory should be allocated * @size: the number of bytes to allocate @@ -5048,7 +5063,7 @@ EXPORT_SYMBOL(alloc_pages_exact); * * Return: pointer to the allocated area or %NULL in case of error. */ -void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) +void * __meminit _alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); struct page *p; @@ -5056,7 +5071,7 @@ void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - p = alloc_pages_node(nid, gfp_mask, order); + p = _alloc_pages_node(nid, gfp_mask, order); if (!p) return NULL; return make_alloc_exact((unsigned long)page_address(p), order, size); @@ -6729,7 +6744,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, } /** - * alloc_contig_range() -- tries to allocate given range of pages + * _alloc_contig_range() -- tries to allocate given range of pages * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate * @migratetype: migratetype of the underlying pageblocks (either @@ -6749,7 +6764,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * pages which PFN is in [start, end) are allocated for the caller and * need to be freed with free_contig_range(). */ -int alloc_contig_range(unsigned long start, unsigned long end, +int _alloc_contig_range(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; @@ -6873,15 +6888,15 @@ int alloc_contig_range(unsigned long start, unsigned long end, undo_isolate_page_range(start, end, migratetype); return ret; } -EXPORT_SYMBOL(alloc_contig_range); +EXPORT_SYMBOL(_alloc_contig_range); static int __alloc_contig_pages(unsigned long start_pfn, unsigned long nr_pages, gfp_t gfp_mask) { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return _alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, @@ -6916,7 +6931,7 @@ static bool zone_spans_last_pfn(const struct zone *zone, } /** - * alloc_contig_pages() -- tries to find and allocate contiguous range of pages + * _alloc_contig_pages() -- tries to find and allocate contiguous range of pages * @nr_pages: Number of contiguous pages to allocate * @gfp_mask: GFP mask to limit search and used during compaction * @nid: Target node @@ -6936,8 +6951,8 @@ static bool zone_spans_last_pfn(const struct zone *zone, * * Return: pointer to contiguous pages on success, or NULL if not successful. */ -struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask) +struct page *_alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask) { unsigned long ret, pfn, flags; struct zonelist *zonelist; From patchwork Mon May 1 16:54:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227656 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5C5AC77B7F for ; Mon, 1 May 2023 16:55:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F33D900019; Mon, 1 May 2023 12:55:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A18C900002; Mon, 1 May 2023 12:55:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41AB8900019; Mon, 1 May 2023 12:55:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 30331900002 for ; Mon, 1 May 2023 12:55:59 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EAF27ADA58 for ; Mon, 1 May 2023 16:55:58 +0000 (UTC) X-FDA: 80742288396.03.E63127C Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 139ADA0025 for ; Mon, 1 May 2023 16:55:56 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=5JJhV76m; spf=pass (imf25.hostedemail.com: domain of 3HO9PZAYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3HO9PZAYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960157; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ia0f0ycGZD+jM5YS+jW1bQxvUpAwQcUQoxrBuJXYMcE=; b=xox55RnYAlX8YWu8mpdIREbBSKTZ6OQmkVXLe5hoBkVM1OxEeN3tD8BRXM6OougjJPJISY 50/M+0ki1GUXwxPHTepyKR41OefCTDkYVq2yE0xbGb9AKdkB4bEb31I0JIR31dwd0ssSwx fJFBKNJC9qIGOaQVmybXeqMVxZGf7/Q= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=5JJhV76m; spf=pass (imf25.hostedemail.com: domain of 3HO9PZAYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3HO9PZAYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960157; a=rsa-sha256; cv=none; b=ZQCW1dFceuDplF1R2Yhm8YGrhXFrbYVSUVtXIqztkP/q3Fsvzmu5a2Q/TXX5j59T8v9Df4 xMb6JuI+zrwNGH91gBB2G/t9gV5WRY1k+XgUMNdXiywJOFrpsA1DeDHsicQW9Dm70uDV/A 9CmzXeBXX+BKMtz80Qwqgnqe9XIHKyk= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a8075bd7cso4836942276.1 for ; Mon, 01 May 2023 09:55:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960156; x=1685552156; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ia0f0ycGZD+jM5YS+jW1bQxvUpAwQcUQoxrBuJXYMcE=; b=5JJhV76m95/YBoZ/xZu42cKOJXhgJORCHh7GQ1vgchybA9Jz2W8JB87J32lzEHqVAD 5MpwShEQTLsgetTJKB81Ya9oYfauOk73A+wkcp3YnAqkuYR2uGLglMtljDk1oXUFeXUF eg/dqf1WTTeod1mNhPnzcjxqEJmrvwX1WQABkeXqhDWaPCFp/KOIQdT6Kd/lNmNqK8uA jO52yQ6iu7pCgMF+oglZnvEtlqmI75FROfYcJYL0lGgLhZXRGGNj8gM5KHULwpV7DXj0 IQGmZzm/JuCCa0BNo8vc+UrvY2fi7yQcbqUa8T0x4PnakvTfaxlxsiA9vMtpl/EDSZYr 7ZHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960156; x=1685552156; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ia0f0ycGZD+jM5YS+jW1bQxvUpAwQcUQoxrBuJXYMcE=; b=Uvvefc2y++MQgk0G48tULtwz9K55H1XXdCixHW2Ts+vtURRBSBa+9NqQADeCjO7gex LjkNvxAxga4WNMcYV/r28/Fe62aIEeXPx9kUaysc8e4aeN1xA41qZoaeUBfJZ+cwvoUB 6OQkmjbfrPyVo0VpUWGSapu21UfKEu8tGLGznNajovO0TSOJiO66RCLII6o2en1c2InP UYfyGZux639D2JZ4AhLPWWovlFkcpwCjUqczIPOVi/nHEuMmEBG+GSacHsFtf2Xjw54F sjJFACOUvwbmbXFIQWYFHYmyJSDe48wS1DVfo1rgZsDDcxRF8jnQZh/RzFDVX6BCBrhL W1xA== X-Gm-Message-State: AC+VfDziO1w6HNPS1j/rY+nG0D/wHEeXBNoi7xFMgsQO8qvZQu/nkgrA reE0x09fQ61FETB/yFdHH9oDaHxHSnM= X-Google-Smtp-Source: ACHHUZ7AZgI8CesVN/W1kPgcDGVmfHbMCJ0zKZ8nFHzmTASKNb7LK9gLjq8w5luIVN/1afaizYlyRnGrI6c= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:5d1:0:b0:b9d:52cf:4a6b with SMTP id 200-20020a2505d1000000b00b9d52cf4a6bmr4308920ybf.1.1682960156135; Mon, 01 May 2023 09:55:56 -0700 (PDT) Date: Mon, 1 May 2023 09:54:31 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-22-surenb@google.com> Subject: [PATCH 21/40] mm/page_ext: enable early_page_ext when CONFIG_MEM_ALLOC_PROFILING_DEBUG=y From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 139ADA0025 X-Stat-Signature: 589h6jrz95cxgrshoq8rt7iqgoyufsuz X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682960156-945747 X-HE-Meta: U2FsdGVkX18syy+U//IEbdmNjrLetbQSBLLlFFJwl2MHh5hBn5A65iwHgEgSGKegu0huFgVU46gD96jBR/4C/ovsfCz2bWE7uyfNpZtN39v2zGKzn9GQw0pxnhYWnkOYUA7klAOyN2OB9Fq1zlDhQonIZ2A0TuuZPAhDuyIAcPbvcKnVOcWBR9aRU5ksSB+BcqUDoXUDWZtyY3Fu4K0di/MKtv2cdjzmP7Yh6oMi+zaHBUAgPUa+xVR+7CbpavgD5cPc9ZTmR8SS83WsDdXeaMPzQ3jivP8kcNrw2uWp6VDwX7fFKL88vS/FI34GM+v5eDRt9geierKMnyYvPtGOfQMhW5qpchRQjjCYkhyyMZpTmrff9L4wKqOX90o2BiwNMRzgoSnit+COhzW25YxpDSgskW9RgQ9fW8N+4HTsncKeS/QYA/Oq1I1bTXLS4tP3FSD1L3QIvnqzcLKKKN8pVOt1ANx06dY4yofcyhvgyNx1xCcq9CnnmchZmRxt5CCUgw9ZsyN5ncTscs8a/hbZ3xxjws6pIaxJGc+p7iGuMlJzSeqQJH5P6HT1q0Zz64PmFx0Pivi8XeJoh2Fccvq16PbZWY+vhNUHRUQp/meR7IMA8CTHA5P2Q+9c5k6xJd/phcfgJYzA+LMuHgGlopP3xkmJ8Euixs9jMzms8djSiBq+EcXMYzCjKH0CG/NkRVlLtf9WJ79Om86ISz1a+NaRtYfmHlM0Wd2LU5brs77WDbtC5ZvSfswjLE34kiCcCjWE8buuynWtn76ukbCGw+p04l7zlXUctgvDCZ2cyXKe43kGMMxh86QwCUh1zQxu1/NiYMFVRcdSUKOUdenR/7eYMoySAVTtSVGUu14ns30kkVjorLob7GF1J6n7fgThZnh8hIOUWEhwyMNueOy+Ftzd/+ITFP5MxIMF4xP6hPQpwcXk79rYBMzUMuZSjzyWJYQgQp7onL6I/uwU+clVTl1 cAmwxNdH KPSLYyMBX090Ugn7+a3fv25XycPlM/KMGoLusGKDDXoiMg4L1znfueeoPCcNYT1YrlKUwlcx6iLPdZPxe63xkS38kJnw+frWNw9FuOX9sjOcbmQqZ31u8wMCHqAEeiSrEZH0oiTGLRr0xbRFiqxRtBTzya0e5WJTEeU/3lIvzFzKabyLXz1c15SMSKCQGbv2kPOHKEfeK6tPT+c4w0iy7K2nRY1AAico9OgEmDaCFEFMxVULL3Q2HPBijT80e/NKap6ee2yqZmwlKWGVbWo+QL+UYrydJJT2zAjCOOAqkNoTGgaDBXG7kbm8NyXhZYeaUp7pDQEw0/itojIPhZhDu5kpWity7Qdnz3kQJrJwEdLvYf+zy0QtgYg7On50KnN0aRYZClw1JM0r7/YsagTu29KRfWV8ck/+jwrLPHdX7gfyG0yQQ1GiZh73PJrCZc7Wf2Pac9xw1sHTCQcSzw/Dv6yo8GpOrAPaXrYW0XZ8LDrOJOfJCNbFiZxh3tDsQqkZ3FvxEPaVbhBwykevl8AzJliTXwPnVJvbLvrTvj4R7BKqf3bwLAOObbnCTV8HkkDmP7LaxeO2JJm3zJySQRpbDB/+sYCvLvTSjXtTcBONt6+AujXSz72K9BJ6iSqU2NyeJTWNBJRVEyUl/PCIVLq2879YF/BOiaQBGsb1kw7OBFGR1GklL4MgzGAPN67EK8nR3vRgPMCvc2hiwDdWDXygW0MKpF7h8+g83Sq93 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For all page allocations to be tagged, page_ext has to be initialized before the first page allocation. Early tasks allocate their stacks using page allocator before alloc_node_page_ext() initializes page_ext area, unless early_page_ext is enabled. Therefore these allocations will generate a warning when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Enable early_page_ext whenever CONFIG_MEM_ALLOC_PROFILING_DEBUG=y to ensure page_ext initialization prior to any page allocation. This will have all the negative effects associated with early_page_ext, such as possible longer boot time, therefore we enable it only when debugging with CONFIG_MEM_ALLOC_PROFILING_DEBUG enabled and not universally for CONFIG_MEM_ALLOC_PROFILING. Signed-off-by: Suren Baghdasaryan --- mm/page_ext.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/page_ext.c b/mm/page_ext.c index eaf054ec276c..55ba797f8881 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -96,7 +96,16 @@ unsigned long page_ext_size; static unsigned long total_usage; struct page_ext *lookup_page_ext(const struct page *page); +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +/* + * To ensure correct allocation tagging for pages, page_ext should be available + * before the first page allocation. Otherwise early task stacks will be + * allocated before page_ext initialization and missing tags will be flagged. + */ +bool early_page_ext __meminitdata = true; +#else bool early_page_ext __meminitdata; +#endif static int __init setup_early_page_ext(char *str) { early_page_ext = true; From patchwork Mon May 1 16:54:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 585F0C77B61 for ; Mon, 1 May 2023 16:56:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94C2990001A; Mon, 1 May 2023 12:56:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D664900002; Mon, 1 May 2023 12:56:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79DA490001A; Mon, 1 May 2023 12:56:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 67474900002 for ; Mon, 1 May 2023 12:56:01 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3B42A12075F for ; Mon, 1 May 2023 16:56:01 +0000 (UTC) X-FDA: 80742288522.10.33889CE Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 5A22C40022 for ; Mon, 1 May 2023 16:55:59 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="7gQ7i/dA"; spf=pass (imf07.hostedemail.com: domain of 3Hu9PZAYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Hu9PZAYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960159; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+1Ptlsu3HrvWcynZoOK0ZqIIaOTUmOtwKZY5J1VaLvs=; b=NKMPrie4R+WoY2NrgUfJOVL9zYAgZVHbtjCi+eVSZcCoFK02A74gVBn6eqYowSyH73WU95 7sWreKniWnjIgcOWgiTLjolX9sq23yKpRqwsp4wOxQArdhbsIV8DTN8uG7YYZgYjHfVB85 PaOkeyHHS/nAZ6izB5hbTT681jR5iAE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="7gQ7i/dA"; spf=pass (imf07.hostedemail.com: domain of 3Hu9PZAYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Hu9PZAYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960159; a=rsa-sha256; cv=none; b=V3D2qkeirT5Eyv0ILkDl7o9iHYanRGq3nw38JaJ/jTZHGncqsDFUmjC9aJYr+ON+sBrM0Z 5mnD138zzLczJMxrhxivEd09+lVddoqDhbSWT0DBM6Bti+0/dlDo7aWKQM65PrF/fDhKtf IFL1TKouSOiKfv/qmz0ZZ+3kS//q28A= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-b8f324b3ef8so3225150276.0 for ; Mon, 01 May 2023 09:55:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960158; x=1685552158; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+1Ptlsu3HrvWcynZoOK0ZqIIaOTUmOtwKZY5J1VaLvs=; b=7gQ7i/dAC1G0h1XxKulN+zlGUyuRlgFLhf3t4U3ldYpJ9ZDjgqW3Y7teulwLUtCfTD /VnlwalgQGiFZbmlj07JbUX6Sp54fAN7SxMoxkItZT8pyryAeS9XmoLet1E+NORYJiar iYyZW8dCc0lMzuXcFL3U4QE+FrlZuAGGckHXyur/pD6//ulYUoFzkf8zDD0aROPp40k/ xmTrCxx3e3E9BLeb2nij/r9kqOJAFZfyDKOq1UGUdyyy0zgGTjg9yGYmpyEbnJHJf0ZV W5qYg5oFjenWEcOuTpD8cYePEcPbJSlpmAYb7YzYWjMWXvtbqAw/EQzvEU6N9zSkiTXv C/RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960158; x=1685552158; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+1Ptlsu3HrvWcynZoOK0ZqIIaOTUmOtwKZY5J1VaLvs=; b=dPUUfiLBxHFG9jk1dOBl4XR7k+ID3cz0YKvRMn/sB+08iQPOfrOrfpG0/q5xnN6XDp ZolvvexSaSwEAibmR8rKU1e3xadukb5vMioCWVYSimKZLasdpRgjy/nP+5e/MGgaZtGf Y5GXDEtn0tlcZLxt2lzOLN5+NmcF/6slUPoPIdrlRpVFe3xSPJtHvXElcsHAy6x0PkCn RyYNvsCNj0bDvtq4mF40oSAjg4E1suSHQ7YgtPo3clwHNrZ4419kxdbvUbg4lsvnNmSJ JKylnZCTtufi1KXjOCBv6JItUHIauxMB2WiHihU75rgKUAZQFPng6zuq1TfSjjpahoCe SiGw== X-Gm-Message-State: AC+VfDwiNZcPgztKX8eo7/FR7Ag9CY8gscKK1LpU99FxJSTOmnazy0GX 6SLGgvwxtOaEXA4uMC7dlCK6au2JFds= X-Google-Smtp-Source: ACHHUZ7r1ycZ9itNH1qfXqxuYbQFcFPivnhoEDNh4tWwK8chvFvWCKKjG1ErV0+LZWXbi1M27uOVRiApR9A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a05:6902:1028:b0:b8c:607:7669 with SMTP id x8-20020a056902102800b00b8c06077669mr8930549ybt.5.1682960158430; Mon, 01 May 2023 09:55:58 -0700 (PDT) Date: Mon, 1 May 2023 09:54:32 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-23-surenb@google.com> Subject: [PATCH 22/40] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5A22C40022 X-Rspam-User: X-Stat-Signature: pbaatzeeznk6z86s6g1qzyb1qe5tm444 X-HE-Tag: 1682960159-424855 X-HE-Meta: U2FsdGVkX1/5eFFd+0Po8Z/kTbTvFyVrjG+HNC0eKq0ijcKhm3Sba5Fbecho7l/1Xu8CrZgIGnOungelcZjV2ZNk/hcaochwh0BE9pot6igH20QXHBcgOR5Jg3pFW+8oGNmArhbzb5UxRg02pG1SYzYk8/ksrMQMg6+ao+J17sQKCzwJqKWIR2/mhPcBv9X3vN3o9tBXO8oRmZPgyoBx1DdqxQGPFi/sYmO+jmnaqu/nw1Acx1gSNF3z5KZww0NlBkUVlyQKdCaNcpboavCC7hTtizJfFGe7h8EgWOcTuT4DsWqmY+h7rd8WPls94yFxqpFU8YnuLp26SIZgwB8JR+Hh0LdW4PyoANGTZDdqJSXeQVlzO1J+PQP7ANMu/Cfz//Evgi1Z5+0UcXf1Lw58r4WTY3+Jhro6ogT4rs6yMh+ciNei+ZAcgnqtOz3eAvaV4uV5W6uNoHOxI8ihOhY5PhnbjwhUMz9BhiONziMHk1Hk7G9fAWRSjGbVljPJxHFo7a/0WEMSCJd50S903tu681oAuBB3QItSMEdTkU01W62qazXuqaPRXfpNccAEupI/1Vh8iWrSevs6BiTwrMdEErBBg9GF1KoFw3Lt5Hy946Gwj9PbLscPRD7aQSC9Fhmr4qDnfurgOct+t/gQGkTFkuZbfeSBWClYb0Fm+uv83TiaoNPblaPz7pgWq7kh+N4MHMGvvMv/F7z3xVZiMdQbtWkqOczmoDKCJ9r8yJJJUKwpm5RZdNv6vSWfC4QsA8gBH7o35vSjdIyP/PfX34E8S2vL41UGA9B5Et0iJ4AcxkHaWxN9LlV4zFWSVq6iKRN4X+YEqQY3wAllZCTVp82upZ2xPKQ/+C/5bfiNFnYqLQ1x6zjMb3O6Oc80R2VSTGuVjm7KdBuRnpIw+EQWWzcG/l4ysmEhhH6FSO7WdfeJ9p20dGIuz6Izigh2nyRcE5CRIL0IiqL0PigqER7D0eH IL3ilE12 yzxO5XOXIM/mCEbZ3ppdQHKDLRE6JKE//MI9dMglqcjg4moxjBpyHO+GccpeNYDAqZUjLseVtbR+gqD7FBdDfXwicnDmkHLY0M06Akbh7fe2FbreIOytWerEMPBui4n71KSlsABj/JRvji7g2bLD5fsD2VMok05/2ehkv8f5f/v8bxZARJhlk3xugJPqcV7imrMryTlGlWkGIZomFoTCJ7MSNOaFMDPE2y9f77DkiXqpnCB5ypOnxHbzDdZNVLUPB+wdmNOWZIMMhYyn8VEcEkVRgIO+vfQrm3oZWq3X8xyz2GOpaCIUFHVpOydxBw5xp9qMqr3lkqmQ3GGEWrK50UNhOXNtPLLH7c/OQMpDH2rqXhOX5jEZ0ctVFvnKhaQTapDL+ushrNUyMX8o+GMBWXDFrgYyoTTuDQObr7JmdDfgbzC3Spk/zyZLoUubnFZ2cNaXDUi7eBKqeAyHCDg0Ahm9RaushdosxekSP54AONKYMARrli/CXaNtLi7lU92Y7M05SAkyR6u8RNcDtOQ4Z92MpQ2Z/IQ/SQDA9XZGkAKJ2jN2Q3Jv6C3Ih3lOv+BpZQLMqyteb8EcpgeC2anQ9Ikm+lfi8r4RP9veLZCGSfeHVCBqHGbXWYakIlO72/Uu3pwccnIBUD4T7IrZEP0OLuv9TyhC52Z0VTtL2owgw8HSlDS0sc7AOVxtpIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page. Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 3 files changed, 34 insertions(+) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 567327c1c46f..0cbba13869b5 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -52,11 +52,41 @@ static inline void pgalloc_tag_dec(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* New reference with 0 bytes accounted */ + alloc_tag_add(codetag_ref_from_page_ext(page_ext), tag, 0); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } static inline void put_page_tag_ref(union codetag_ref *ref) {} #define pgalloc_tag_dec(__page, __size) do {} while (0) +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 624671aaa60d..221cce0052a2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include @@ -2557,6 +2558,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, nr); + pgalloc_tag_split(head, nr); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index edd35500f7f6..8cf5a835af7f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2796,6 +2796,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); } EXPORT_SYMBOL_GPL(split_page); @@ -5012,6 +5013,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); while (page < --last) set_page_refcounted(last); From patchwork Mon May 1 16:54:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227658 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E8B7C77B73 for ; Mon, 1 May 2023 16:56:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF55F90001B; Mon, 1 May 2023 12:56:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCD35900002; Mon, 1 May 2023 12:56:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6F3190001B; Mon, 1 May 2023 12:56:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A264C900002 for ; Mon, 1 May 2023 12:56:03 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 737FF407F4 for ; Mon, 1 May 2023 16:56:03 +0000 (UTC) X-FDA: 80742288606.21.0C4CFC9 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id 8773A1C0019 for ; Mon, 1 May 2023 16:56:01 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Nr4QXyj/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3IO9PZAYKCGcXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3IO9PZAYKCGcXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3JQAU/dBFDp2j3prOKpx9pu61B2aRzHMn5ijo1mj0wk=; b=xnJfbWRI6fcJVX6NFpBG1o9nLGNBtGVNuaGqHoyfcXutfXBewrWRht9B//w6VFyMNd569Y mlu5Pp1+6TIkjavEoIHwMTToJ9SmXDbf+UMQIknFrpQwPgcwPNT73J8NAtvq7ehNWTT4uH yhS88Vx/AJIckfA2CD7i3mZjdRKLuRo= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Nr4QXyj/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3IO9PZAYKCGcXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3IO9PZAYKCGcXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960161; a=rsa-sha256; cv=none; b=Y6/HSoXWJa9CKg5bT0u0Rdonsmd14EawxlhUQXta5vVqDUH56M07kCMswJ4k3C3gLD8qSM I+wWd/74gymiBel7144HD/n24nz/8+30JVzFesDkF5g+31XZ8xvknsANy8DONZjbbiqztb 2E5eA1lfHkbQLLGdDdaAkzAVU7Gbqgc= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-b9a7df507c5so5361612276.1 for ; Mon, 01 May 2023 09:56:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960160; x=1685552160; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3JQAU/dBFDp2j3prOKpx9pu61B2aRzHMn5ijo1mj0wk=; b=Nr4QXyj/etzJlAh8i4e3XnDFrHrok+4KAhChzrGdGYpNpzqnTzNz+b7TDzIFPvjVWJ 9m6bX+drGU9A6fH4AYp0HUvbrb0fCoSPBmCg7BeIb7knPz8EfEfG9cPbzcBJ5VX+RXOe xKxhA1OiVlh84AEwzHBh+cMUHvzUPF37NpWH9RACZgYBnn9nhrwQ+ZtD6LOD5aF3vQAg h9QGbAMavYpJls9fxf3kDm4J6+Qn0mT0BuRWWAvY6omxOHY3anv02SbqbukFuexGdvmI uLtak6s/1IGom0RtuY8qFrrhQXG4Ym0F7I61Qiqk5+0EJAXYPW/dDKZJd/GArGchifG6 ytrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960160; x=1685552160; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3JQAU/dBFDp2j3prOKpx9pu61B2aRzHMn5ijo1mj0wk=; b=OEGxXtcp2jwemJTVElZgbV47CLDGo8y6xs53IU/rORB706YqIf4uWK+T53pM7WCUCU UQ6ppiiddmOTa5c6AgEhTu1U+7DrVZuMG3OHUiLwONgNVDWbSsWlriWIiXCLzQxHrm66 qOWdvoSCZIinxaNEBqRvpU2cyD6cFXztPltIaMPz0enb382Pbgu08eGSpj2cQ+nEc4C8 pZrO8AKWwzkRyzBVeaR+4P1H5+uJ4o0UpPqY44dwQQByB/HxFUzTT4UWKHnZC5gfhViK Wq8IrPXU3J5kBQSAUZptXc4CQG8QnpfUBS72gCi6XUmoN1BrheeL040KsudUbsDNcHlE AUNg== X-Gm-Message-State: AC+VfDz1EMYUKN0KaOcWQHEHZaNpCC6xaGepNkgrBZ4NRc5grwqN2FEP do24meB56PEAfqmKrcyI3RPA3k132eg= X-Google-Smtp-Source: ACHHUZ5flxShenE/t9NPL/moLDuNx9TkwM/Lf/8wW54V2Eq3VJ/wRhfgfjc0frmw2aZZoQxo3iYMAN5xHBU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:2484:0:b0:b95:e649:34b6 with SMTP id k126-20020a252484000000b00b95e64934b6mr8454589ybk.1.1682960160542; Mon, 01 May 2023 09:56:00 -0700 (PDT) Date: Mon, 1 May 2023 09:54:33 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-24-surenb@google.com> Subject: [PATCH 23/40] lib: add codetag reference into slabobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8773A1C0019 X-Stat-Signature: utr9to1zc9onm8o7nccgo1pf53tybst4 X-HE-Tag: 1682960161-625313 X-HE-Meta: U2FsdGVkX18Almyy9NwDE73D9ur4vDg3ZwZkq1csaxT2BSv9j7/rd3Sf+wFNwcFZuFsyIDY8ud+qqYJL4oiOf0rNrdyrMiJ1Jxvstj9Op/3ard66a924GXwqc3a5xRZdzRS719XRnBYyid/qeu2IdNiEjKQ2F+s1KB2gbR4UlANFfOzNuMJFicsPJWwJVqxgpND2eQ+HKzBs5Ixsl1OwEZZ0wSHot/MK5aAKGVD+0MydgoS9XCfeBXxMyzjMlHDqyMoS53d4B3s+YoiokCf8gjegslwRKIym70ziywPM2T0Kwh2E9VvmZOBxuXC96qlxRgAOCxBu3cXiqJY2ZGtts5NVPikRlivTmlpAwOnLUsoBtJ+ql2Mlmn4JEjRYdDqvwMV76HrYh6+CzWyNI1b2e+Extv9DA6BVoNQUkpBZNKM2a06wWVU//uUEYSuELJW+12otDTz6K/Te2GGpYY/zGZonJvox8m6LG/9KQ126eGTpS1O0h7w37pZGfgYB/6BXbCs39uWyo96e65SYr0oSR3qhhSZQBTyOkkmnbgxJztB6luLijCgrdskshOC3BlrqL8Phlb8kF2m1vss6pJkhOcawqXY/Ky7EKAMMiF+6q43aAGpoVrnzP2EuYLIZaqEkNE0zpTFOdZMS1Pk+rV7xsaoSnr/hNP+hLsPRTJX+bxuYGESr9mejDgLGt+lRSUf3/IXgHL+TrX+YK/tD5pkw4GLtde8Rn0xhpOPr2l32kBYebcoWoOvRYfcZgMz1fFei4qMBIUHZhlw8IT1o90KmvENwcPa91Tk1b8zcZpECGbtDia7IlVonB/ycbailAFIHO7EUWjBI/jtBiZ/gT3GHt0b4BAGZG80uqUZrDt+wiRqR+Yx4eN7FZRgh7Itu0+7JaIEdEzpXIVBk01H/+HLK5YRroaaNnP5fxIi4QjzdHCHtBzwAZyq13G953MIsoOnBWxlgLyiCC6vH+fOIpWo bZ7GqMOZ kDSKxik01UvZQTjFmKSNht+E03WeWsFxiekqWagreK1d8aj4H8bXZmPT6xjQUbdO87hYYk8Qc+tOYr+iintXMHH7DoWXjxW7w1zI6ndroiEe+uteEaS2U9iCgHRoFQVTBXC+9r1asebJrVThhGFGj96woGhNGmCACgDObJVqSaHGY8yyoEECCavMb1LOQir4WKIgZrvB7RzidHRmXwdYcz4nSuQLGhm3XT6UafCfkt4tEj07HwK7y+6kYsKtXXouxRXoZ/kdTvWSE7FC7TNy+i80yFsu33wjG2qHyedCRlfdyybXYDLsSlfVr9Xe3dk0tKLHldAuf2/unon4kB6Tu38WE7rH50svavf0mE127saNSX8QABSjnWuoz4uDU3W+JDN7pW7rGKpwdJamfcaIDsS8rFUf0cN2lmson1dP3/PLuVU4QxfuYS3Wq4EzxKEHNJWyvb7D6xuXCZgog3rydpuq9KESNl+Hwny5W9w4Jxla5oC3CCD6rqqag8czv9ivohLiVJhfZUt0fIaN7h1cedNJDu1pmyQv49AiTMwOyjAfB3h8/F4zWprLrT8dGG/DN9i7B1tjzPftmVRRBNY8Q64/NigwFvJsCT+vk/AOW6sfHivydy1AlGq9CRJNaHoLGlzxS9ensAbcpzrW/v5iLNJhc4K5oixeHPJJ+/rU12Gea6iMVU3fj+xWXIOtbZ3dtJuuHbvEJHW+dXZUIA3wdtmade7bK3QIOfjN4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To store code tag for every slab object, a codetag reference is embedded into slabobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/memcontrol.h | 5 +++++ lib/Kconfig.debug | 1 + mm/slab.h | 4 ++++ 3 files changed, 10 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 5e2da63c525f..c7f21b15b540 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1626,7 +1626,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, * if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref ref; +#endif } __aligned(8); static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index d3aa5ee0bf0d..4157c2251b07 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -968,6 +968,7 @@ config MEM_ALLOC_PROFILING select CODE_TAGGING select LAZY_PERCPU_COUNTER select PAGE_EXTENSION + select SLAB_OBJ_EXT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/mm/slab.h b/mm/slab.h index bec202bdcfb8..f953e7c81e98 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -418,6 +418,10 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, static inline bool need_slab_obj_ext(void) { +#ifdef CONFIG_MEM_ALLOC_PROFILING + if (mem_alloc_profiling_enabled()) + return true; +#endif /* * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally * inside memcg_slab_post_alloc_hook. No other users for now. From patchwork Mon May 1 16:54:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6C02C77B7C for ; Mon, 1 May 2023 16:56:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DB6B90001C; Mon, 1 May 2023 12:56:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B24B900002; Mon, 1 May 2023 12:56:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52CFF90001C; Mon, 1 May 2023 12:56:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3E5DA900002 for ; Mon, 1 May 2023 12:56:06 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1FF0EADA6D for ; Mon, 1 May 2023 16:56:06 +0000 (UTC) X-FDA: 80742288732.26.30F4DEA Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf01.hostedemail.com (Postfix) with ESMTP id C8CF24001A for ; Mon, 1 May 2023 16:56:03 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="E3/DZ7A2"; spf=pass (imf01.hostedemail.com: domain of 3Iu9PZAYKCGkZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Iu9PZAYKCGkZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960163; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UqN7FeV7aPqCmKItkyIx4xBxa3o94vtL5Phvca7QUfw=; b=69QJXvatTj3POgHfTJbkmHIlCscvnfwRgQ7q0T3jo5N0ilbpFlQHzCcB76B9yK4fALv0rX zCmsF23/mP0MNtRwRTfx02C5eahSJh2JSPYVAMSXlq3WfysWcLa+AVy1JjHgZCMBS2e0rm FWa/E83FTDZRs8qpF+EuRR1ICO+cksU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="E3/DZ7A2"; spf=pass (imf01.hostedemail.com: domain of 3Iu9PZAYKCGkZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Iu9PZAYKCGkZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960163; a=rsa-sha256; cv=none; b=Bny3oahXmKCvDhkmGYScuD/9nKpIpJ8E+Szxehn2NT06OC0vNqzvwcETmyv+ZwgpAxle3W CI0TH2eYXMC+XWQHgt/DPhEywT8JnDoaDjZ1iuB4wt4u+tjKfRkHV2g3bwn4xnxoQPfKOQ yjIztjFxDKuvqGCB2t929GOgQnPOOeA= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-b9a7766d220so3265578276.2 for ; Mon, 01 May 2023 09:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960163; x=1685552163; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UqN7FeV7aPqCmKItkyIx4xBxa3o94vtL5Phvca7QUfw=; b=E3/DZ7A2w5/KA0SPoOgDZl8cJAWCaY3BGBFQeSc/qzjQe0fM9xmbbTDocgwuOOp3LT GK3YJqraePyIrgvlz0C1S6Wec6V4sz3+3alNU4R6cNfgWKAm/KHB5K1fHdSBD6UsYReW OypIaQzRticsOqfq5jwU1kZdYTvr7RLYZbcqb3Ite92bHmcVQajevDqoiuCCwS030v+p Be3SQphLsWQFuGhQq8i0kXYiixrDLOjgisXe2ZSgniNRK9cfGXtFSqdN+Kd8F5/lQa7E /p5ZymC5neHH+BhUfAayBM2rdDwZ8KZfkYIvJtL+L0Ri9FOGTCuyj9nVUWNgD0tlqnQZ Ay6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960163; x=1685552163; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UqN7FeV7aPqCmKItkyIx4xBxa3o94vtL5Phvca7QUfw=; b=f5V2TSJd3dvI1YvHCOG1QoKQt6nIdG6O6xyuwo/u9Guxg2snS/45XeE7rbcwSDigZH MrDWIM0tvEQ2u1A7f7ztkk+gD251PAo0cG218weobfceH2kR3ZUZ95cGncxUDymKFY3t 1MOZRGkksFss/2FmaBtdHGi256C69SoSzrQspIJHrTBpbxcN7j9cLjOp+byiAdUm/HkP qvvWgRQXK3y7FUg1ETxu2HKmgABY8goBc+4MVIdEJ1UYt0iuiylq4971B9lbkvAZeoUG myaVVghYHvXwt7ppGKUjtFtBe7LfERaAs6dwHN0RN2v/VM7fixABjB2LOm3xzeWSwXIN II+Q== X-Gm-Message-State: AC+VfDwvCs2HxA4kDuqSlYRXNAFiukLkcyX+LS9beUvlY9xbYl3BPmmZ deYENC+myFKOEnjn7Rwp9b9R9J3wJ68= X-Google-Smtp-Source: ACHHUZ5uEsWRmwc91CcDamKCFDJZF1BFQrZ5fbii1YLw08AXxDN7W/lrJx2YCa2hPk82YxYc6zpx2fFuhLA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a05:6902:18d6:b0:b8f:3647:d757 with SMTP id ck22-20020a05690218d600b00b8f3647d757mr9026699ybb.11.1682960162837; Mon, 01 May 2023 09:56:02 -0700 (PDT) Date: Mon, 1 May 2023 09:54:34 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-25-surenb@google.com> Subject: [PATCH 24/40] mm/slab: add allocation accounting into slab allocation and free paths From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: 331mytoqsdhqu1tryhf94uborm5dtcd7 X-Rspam-User: X-Rspamd-Queue-Id: C8CF24001A X-Rspamd-Server: rspam06 X-HE-Tag: 1682960163-27713 X-HE-Meta: U2FsdGVkX1+qZr3tIFOjS78Y0DUH94+QirXfsAxejt+Jz4g0IgRxNFC1smZOa7ChZ6gXS/jKQjHvOuPgZ8/v57hwjgQqVlopfAFp5SqA1u8PG+Rtd+kTIDyk05KETbaIuf1YpiKke9j3Y5hy88aI9I1koxz35AL8hpT8NzmjCMnWEK44jZcYDqFfjUM5SdTa/vqQA+aT2GmRqgtiweKlwzf7UZWky1i7yajcW15o+IddFUISEoOpFjTPjYQMN7lhI0n2qAII1njqZjSKDFKKIASNQJJczvWttuTMhoRIO53TBHtxhYMgC1W6GsagrP2nOo01SAMNhX7zoDQIkq2K1zQDIJlcg7v0xfL9QasB8oOpJiHk9bBq419yRH6T4+HQNJCegII6bnsDIzvfRfeQnkq24Lm+4tfv50KIH2FZd34bv34VBinImNOkJf5zomIT3PcRRxR8sJ22j65q5+pKpKXOSZboBaobMvO10nEy/OHXgFNWn1lQ5fXJjPLCHUbrkyyvIFuI2iSZje7h45Ih5CDUqyAH5DQFkEK0hXpf6J7eALC96+OK+msPPgMRvC25qyu3S+dMkehJOX0O+utuuzEacXMYXf5C0S28Y4rF2VC+3vFqJhvmOv4yfz4YnEfAYB1JvW1c57nTVelsI3SSLXsmIAemHIiUZVPw4GXNvgXQR8KaIx6AogC9iVP1eZbvLh4l/XPsLn4BGZ4Ig2ztL/fHu+pagqdheUvTI+q6+g61CzgIpuX5TCoRRP06iH50vp3SOg5qA5bhZVBXpXmJnQ133x9zFYqARiyeGOGXnLQF0u2synGQIUEp0vLpN5aVneSxyaSUXRX2f7z2qE6K4kgvt2UXpqHoyGIlU2Tv6Ecw7qr27RUq11+tbamg4lwIMUJw/aIhYEeRwFJMr0ulX/HfFpt3v+HCh/Sq1qwfYxtfxhkAIRyDShVoU/hDGkh5WomMFvex7SoWxAfi6EP 83eGdGnH FcpWWce9quyQOmD47jNSelFQn9gFp39RlZM1Pyalq8nEBNeBdXDCESWc3uF4Ohg7+dz60GFztf6N0CYiPIJ18yy5UOMkjng7fyRSOicE4qBQ7pVlS3LaG7g+JvysKFGeBY+O3/IQ/u1NvDvnSrhgm5thseJvW/6Pp/1KYSq6Uj+EY/KPcRD6uTHp7Ch4rQ1nZVdkAJzGopBgXLSEGhA7TihTuLFAkVBPZrr1NhBPFBvW53fjzPFKsbQ3sgeeG9/gWDxPQ/3w0Z+hPEt1O5C6xp/y2JyHSQ4bpf9DapsPajHGVYnmD2a5nLIxL3NkqOiZkCBLUZYej3ncPPW43ClyBLeOth6x5fn2r88/NukKQeVf3F0bdehtWcSZcTQLpSnB5G6T3Kqc8u5son4tx0O8BYtX4XpF2rJS13p4dldTITgZl5Pfe6xzCSXl9NSOiaSjUu+PtSxXsBQaFAFm9CnC6v1R6viXi7L5EyvLXA7R6sqa5CZzrAaO1jselr8uDgCZdYNYm47yO7TQWPKIvxozpvK2exiCiZaj0s8RAzuY7oxMrMJGkQPH2qQ5h0kI+c0lnyaHs4KW/OpQU6WorXZn4NNFiZltRcz+GvleZ0Opex2/uti7iVrCwMXmpnKrOhDphyIzuhEuwDBshiulRJMXwSdMgFvsWSyBHLaxedcj+37OKliUTArMe4nsskPIrWQm9GnYqWcSYVaONHiuJ5XGuCCxTDg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Account slab allocations using codetag reference embedded into slabobj_ext. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/slab_def.h | 2 +- include/linux/slub_def.h | 4 ++-- mm/slab.c | 4 +++- mm/slab.h | 35 +++++++++++++++++++++++++++++++++++ 4 files changed, 41 insertions(+), 4 deletions(-) diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h index a61e7d55d0d3..23f14dcb8d5b 100644 --- a/include/linux/slab_def.h +++ b/include/linux/slab_def.h @@ -107,7 +107,7 @@ static inline void *nearest_obj(struct kmem_cache *cache, const struct slab *sla * reciprocal_divide(offset, cache->reciprocal_buffer_size) */ static inline unsigned int obj_to_index(const struct kmem_cache *cache, - const struct slab *slab, void *obj) + const struct slab *slab, const void *obj) { u32 offset = (obj - slab->s_mem); return reciprocal_divide(offset, cache->reciprocal_buffer_size); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index f6df03f934e5..e8be5b368857 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -176,14 +176,14 @@ static inline void *nearest_obj(struct kmem_cache *cache, const struct slab *sla /* Determine object index from a given position */ static inline unsigned int __obj_to_index(const struct kmem_cache *cache, - void *addr, void *obj) + void *addr, const void *obj) { return reciprocal_divide(kasan_reset_tag(obj) - addr, cache->reciprocal_size); } static inline unsigned int obj_to_index(const struct kmem_cache *cache, - const struct slab *slab, void *obj) + const struct slab *slab, const void *obj) { if (is_kfence_address(obj)) return 0; diff --git a/mm/slab.c b/mm/slab.c index ccc76f7455e9..026f0c08708a 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3367,9 +3367,11 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac) static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp, unsigned long caller) { + struct slab *slab = virt_to_slab(objp); bool init; - memcg_slab_free_hook(cachep, virt_to_slab(objp), &objp, 1); + memcg_slab_free_hook(cachep, slab, &objp, 1); + alloc_tagging_slab_free_hook(cachep, slab, &objp, 1); if (is_kfence_address(objp)) { kmemleak_free_recursive(objp, cachep->flags); diff --git a/mm/slab.h b/mm/slab.h index f953e7c81e98..f9442d3a10b2 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -494,6 +494,35 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) #endif /* CONFIG_SLAB_OBJ_EXT */ +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects) +{ + struct slabobj_ext *obj_exts; + int i; + + if (!mem_alloc_profiling_enabled()) + return; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + for (i = 0; i < objects; i++) { + unsigned int off = obj_to_index(s, slab, p[i]); + + alloc_tag_sub(&obj_exts[off].ref, s->size); + } +} + +#else + +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + #ifdef CONFIG_MEMCG_KMEM void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, enum node_stat_item idx, int nr); @@ -776,6 +805,12 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, s->flags, flags); kmsan_slab_alloc(s, p[i], flags); obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + /* obj_exts can be allocated for other reasons */ + if (likely(obj_exts) && mem_alloc_profiling_enabled()) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); +#endif } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); From patchwork Mon May 1 16:54:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227660 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFB87C77B73 for ; Mon, 1 May 2023 16:56:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83A8D90001D; Mon, 1 May 2023 12:56:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EAEB900002; Mon, 1 May 2023 12:56:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 664B790001D; Mon, 1 May 2023 12:56:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 56F89900002 for ; Mon, 1 May 2023 12:56:08 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 02F591407E5 for ; Mon, 1 May 2023 16:56:07 +0000 (UTC) X-FDA: 80742288816.07.28F2105 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf27.hostedemail.com (Postfix) with ESMTP id 3518E4001F for ; Mon, 1 May 2023 16:56:05 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Hn84O/Fj"; spf=pass (imf27.hostedemail.com: domain of 3Je9PZAYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Je9PZAYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960166; a=rsa-sha256; cv=none; b=QjUE+lQsN+a9ljKV6f9h5TY0ifdG4gd/AzQRuOgIXTlkdl5BymxceRlhXxYEVoW0xh1CPS Q//Ak/i68SApYZLvYiV9C5xFmFGazc9sdG1FRWLwUBuBYoU5xSgDayu24TsgWcTkaQFRZc QWpP11ko/EdqY8wM82rkxCYEJPQ7ZDw= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Hn84O/Fj"; spf=pass (imf27.hostedemail.com: domain of 3Je9PZAYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Je9PZAYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960166; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AlFKqZpStCoPEOrrPZX1wMSDo35mmKSmNx34G8mCjEo=; b=4wvA7EomSQTcPMH34itddDqVOLApHAddAu9NuvMZbT+igc3mkiVsvcxINvMBFFkMAM+b/B QBoXNx+OIO+yhPGGqqNMVScEmnDFKMqamXKVjIm8AuyCvn61zfa4+G2eLhEXKG0O1oK93k V7GEh/FeJNrq9WwAoCQQjg+2FDpXWWc= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9e50081556so490868276.3 for ; Mon, 01 May 2023 09:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960165; x=1685552165; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AlFKqZpStCoPEOrrPZX1wMSDo35mmKSmNx34G8mCjEo=; b=Hn84O/FjnBQjw81g0VTyh/uHBdrl6I0At7mqU8YKI29J+zwV878Ud7aKtkVFupHujF +P33CHtldPo27h+zuVFhAddAj0A5Kkpw2KfEVH9tijZGDu22S6vwmzU89PxAqJvFSAZJ pngWOgRWLpGBUXF1EyiUqARF3hZwzVXrTynm7hdSHxdP6sPu2MVmUUTk1QEPnZ21ugxP cu7FraBIH82U3o2syZEGdLdo8n9cfMJuxKna3vatgjXqF9IMgQ4o3Y8inTxtaKJYGM5W EpQqsvgJxtK8rynO0iG4tmpcH/aAdCa/3MR3/1NiEMIlFA2NSYb16Me/KIyQUmkmPrNV 5otA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960165; x=1685552165; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AlFKqZpStCoPEOrrPZX1wMSDo35mmKSmNx34G8mCjEo=; b=YWmG6m1u5/I0zKY6UjyxXqE7Fe3oZ82Jx3dYMySfmCUbRM98bPunVMyHJ39jVam5+m AIXbhbKbsfYhA8BW3HDHe/7FphSD1hczKyGFk1uRjkJ6t+Cw7XJP881yiSYvstwxTNGX UmuttiZ+OKaFFwFJ5KyWvZkxkeLDETBGtLo3wsUdQNl63WR6JoSahUgGfPuyqWCi8xH8 Vq74k8YYOjdMBEtsmFry7IpaDWZjwrla8xUx2raOvdbhzwGQ/J7ciYp9FhJPzWcZzIpo Dc2l1lFrMW3+MbgxUrUTL/exxFJjXTHnXVODb9PUQ7z9C2xHTT2V1dxeZLu4YUH3Sv2t 7g2g== X-Gm-Message-State: AC+VfDwtCR70T3g64r7urhEcgvcWpScRRGI36IfqzInN7S0sMsmIHvca voctjYgD6lRHgY2SG6de3di9iQus9iw= X-Google-Smtp-Source: ACHHUZ7HiRCvMneGTlvQ1D6Ierrr2x2A4iZIC6VQ7sboDBFZuTdoupQo6xhccVae7AaxEVEFzUlcuhIeo7k= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:3242:0:b0:b8f:6944:afeb with SMTP id y63-20020a253242000000b00b8f6944afebmr5782469yby.3.1682960165175; Mon, 01 May 2023 09:56:05 -0700 (PDT) Date: Mon, 1 May 2023 09:54:35 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-26-surenb@google.com> Subject: [PATCH 25/40] mm/slab: enable slab allocation tagging for kmalloc and friends From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: 3518E4001F X-Rspamd-Server: rspam01 X-Stat-Signature: 9zb3szfpikqf5rs946z54ndo111jnzko X-HE-Tag: 1682960165-585559 X-HE-Meta: U2FsdGVkX1/POtTkxs8nCa1zsw66F4Jj7vzICEsbjPRKhgVW5WCax7QbE1ktTLeCjOf14CTQ+wbyDKgwgHe5Wz6rGyvdwKb1btsTy/Vqtjh764le5ygu7jL5f8EYR7+AcoAErAOZt6DBPw3b+74OsfJYgb+dBcLGIwQ/IIabbjvKWI33LTCdMynN9hDJGhKSQXkZg/peMMjvU5dHlsOFUMZ9ECx7yyH0QLDyGL8LLse5DNEDI/5HW0hMTK0pmsKyyFkgI085pC0aekU5ZEZJIE+NWQa3eSeu2m3ppJI0ZSP+odbhfdJfpAqEadi9odA5/eQ+r+ZWoxdTK+ASfTSQAIqfoD+csn9SU7BpUXmqwUOFQtxT41ZxqmqDWE6Z0EcQupftq2DuFiVBfsYW84QFyr+rKG+v7YWBshK2KabApRCAQ6JNUxQTVcCiOI9F+mEP3j6j1IRpCtwzohO4/fmBWAzpBExFVGXTrc2GNKuQGCZ3/vKq/JgEFqHb/qTdLYB1QkDkwC1ccsZo4QwCwigImlS0nQX/UUmpW/ZL8qhOE0fImbr9IXZJuYrMHltV/Sa3Bo57r63at5V9jJjKTqRS5ItFUZYZpEubvsocWxYXEwhhzw01mKTuXCI8DnYaowIzc8CfnYqGbigf1TfE9eIyzllEPe4gMGUaN48YsvmLzYAZe3e9+GF3MVqekZFaMwOutNUAQkmP182/FoAVK9BCQ2fW4P1RQpa01kzJHSYNJtASesdCoWaRNCatCGdtM+HvOBVqSTNO/HOF/LPRwzk1aucvvkS/7VcpT5BxmCEBeMQr37HIRm58Ca0/VEqtgGLdkTMq/1faWC2E44z7RNoS0t1vKu0IPFqOUXSy/mDMF3bWbslKgc0OOPD/vtpJPNzh3RPlMGB9js/ydT6D/I7jG//4e6fooR1jSMFe6/fs/DXOPs7334Edl63ANaLtcgOXJLxDPLfQrEE9WWGrAEk 5/S/NXQC 9EQPPYn2D9l8O7jlkNcOBd2pptVvDk3mfhTK06SaC4yNnFVzKvhWQpS1L7SQU0eT2RSnfIELo+8IZmKTJp+qU0PIncrcP2lDVsXU7pMA/9A5bzFfyChQLhrHMARar1IrAbXAlBG2Hr1nMneRFipGHDoMVE5mPhp/Dmon7Ryf13of7LsRDQeyhfaqZnJ3of5p6tk9AZOWwIbRUeiWFyrvS7aGYSJ3BINhwS1qdp05KU88lkRTBlecH+79rAvsLbsKb8RsCyMg1yFsMvEq2z2V9f1nYqSQT7YzO9COm5WBra5KNueKVb6pot4Hb+6tuCq4QZ+NDOiDUVJr3E5IoYckkJmp0epyspMwdUDOsmqX+9ZpUIq1T3TeCVBCuyOsGkhOw48hAuDM5n5wcHjRFsn1dj4U7vvCO6IQyoiYhbgHqoGUDzMUTT6eVaYedC6HuJSH/8kEanzM4QggxSa/T1qzQKg3FPvS4sNd4AH5A4Ud4bx1JHJE51Xr5vI008cxLaIDdhdiI4FwRDF8AWHJzIiI382eJqFPsHlZ+TWCkMUID339wXKAd7Zq+ZfaHjicvmT7PhTDborJztTYo7HzFN+e2FWPgel2mukRIC/RNmSXoygRHz/qhco5M6K/lAhVP2z7pRLyv9g7byjCdFK8Am/vlCD7xvpg68TodtBzEZV3mQL/NYl8MK795wd0uZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Redefine kmalloc, krealloc, kzalloc, kcalloc, etc. to record allocations and deallocations done by these functions. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/slab.h | 175 ++++++++++++++++++++++--------------------- mm/slab.c | 16 ++-- mm/slab_common.c | 22 +++--- mm/slub.c | 17 +++-- mm/util.c | 10 +-- 5 files changed, 124 insertions(+), 116 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 99a146f3cedf..43c922524081 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -213,7 +213,10 @@ int kmem_cache_shrink(struct kmem_cache *s); /* * Common kmalloc functions provided by all allocators */ -void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __realloc_size(2); +void * __must_check _krealloc(const void *objp, size_t new_size, gfp_t flags) __realloc_size(2); +#define krealloc(_p, _size, _flags) \ + alloc_hooks(_krealloc(_p, _size, _flags), void*, NULL) + void kfree(const void *objp); void kfree_sensitive(const void *objp); size_t __ksize(const void *objp); @@ -451,6 +454,8 @@ static __always_inline unsigned int __kmalloc_index(size_t size, static_assert(PAGE_SHIFT <= 20); #define kmalloc_index(s) __kmalloc_index(s, true) +#include + void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); /** @@ -463,9 +468,15 @@ void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_siz * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, - gfp_t gfpflags) __assume_slab_alignment __malloc; +void *_kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc(_s, _flags) \ + alloc_hooks(_kmem_cache_alloc(_s, _flags), void*, NULL) + +void *_kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_lru(_s, _lru, _flags) \ + alloc_hooks(_kmem_cache_alloc_lru(_s, _lru, _flags), void*, NULL) + void kmem_cache_free(struct kmem_cache *s, void *objp); /* @@ -476,7 +487,9 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); * Note that interrupts must be enabled when calling these functions. */ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +int _kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +#define kmem_cache_alloc_bulk(_s, _flags, _size, _p) \ + alloc_hooks(_kmem_cache_alloc_bulk(_s, _flags, _size, _p), int, 0) static __always_inline void kfree_bulk(size_t size, void **p) { @@ -485,20 +498,32 @@ static __always_inline void kfree_bulk(size_t size, void **p) void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment - __malloc; +void *_kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment + __malloc; +#define kmem_cache_alloc_node(_s, _flags, _node) \ + alloc_hooks(_kmem_cache_alloc_node(_s, _flags, _node), void*, NULL) -void *kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) +void *_kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) __assume_kmalloc_alignment __alloc_size(3); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, +void *_kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) __assume_kmalloc_alignment __alloc_size(4); -void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment +#define kmalloc_trace(_s, _flags, _size) \ + alloc_hooks(_kmalloc_trace(_s, _flags, _size), void*, NULL) + +#define kmalloc_node_trace(_s, _gfpflags, _node, _size) \ + alloc_hooks(_kmalloc_node_trace(_s, _gfpflags, _node, _size), void*, NULL) + +void *_kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment __alloc_size(1); +#define kmalloc_large(_size, _flags) \ + alloc_hooks(_kmalloc_large(_size, _flags), void*, NULL) -void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_alignment +void *_kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_alignment __alloc_size(1); +#define kmalloc_large_node(_size, _flags, _node) \ + alloc_hooks(_kmalloc_large_node(_size, _flags, _node), void*, NULL) /** * kmalloc - allocate kernel memory @@ -554,37 +579,40 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_align * Try really hard to succeed the allocation but fail * eventually. */ -static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) +static __always_inline __alloc_size(1) void *_kmalloc(size_t size, gfp_t flags) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large(size, flags); + return _kmalloc_large(size, flags); index = kmalloc_index(size); - return kmalloc_trace( + return _kmalloc_trace( kmalloc_caches[kmalloc_type(flags)][index], flags, size); } return __kmalloc(size, flags); } +#define kmalloc(_size, _flags) alloc_hooks(_kmalloc(_size, _flags), void*, NULL) -static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) +static __always_inline __alloc_size(1) void *_kmalloc_node(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large_node(size, flags, node); + return _kmalloc_large_node(size, flags, node); index = kmalloc_index(size); - return kmalloc_node_trace( + return _kmalloc_node_trace( kmalloc_caches[kmalloc_type(flags)][index], flags, node, size); } return __kmalloc_node(size, flags, node); } +#define kmalloc_node(_size, _flags, _node) \ + alloc_hooks(_kmalloc_node(_size, _flags, _node), void*, NULL) /** * kmalloc_array - allocate memory for an array. @@ -592,16 +620,18 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *_kmalloc_array(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc(bytes, flags); - return __kmalloc(bytes, flags); + return _kmalloc(bytes, flags); + return _kmalloc(bytes, flags); } +#define kmalloc_array(_n, _size, _flags) \ + alloc_hooks(_kmalloc_array(_n, _size, _flags), void*, NULL) /** * krealloc_array - reallocate memory for an array. @@ -610,18 +640,20 @@ static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_ * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) */ -static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, - size_t new_n, - size_t new_size, - gfp_t flags) +static inline __realloc_size(2, 3) void * __must_check _krealloc_array(void *p, + size_t new_n, + size_t new_size, + gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(new_n, new_size, &bytes))) return NULL; - return krealloc(p, bytes, flags); + return _krealloc(p, bytes, flags); } +#define krealloc_array(_p, _n, _size, _flags) \ + alloc_hooks(_krealloc_array(_p, _n, _size, _flags), void*, NULL) /** * kcalloc - allocate memory for an array. The memory is set to zero. @@ -629,16 +661,14 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flags) -{ - return kmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kcalloc(_n, _size, _flags) \ + kmalloc_array(_n, _size, (_flags) | __GFP_ZERO) void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, unsigned long caller) __alloc_size(1); #define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ - _RET_IP_) + alloc_hooks(__kmalloc_node_track_caller(size, flags, node, \ + _RET_IP_), void*, NULL) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -648,11 +678,10 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, * allocator where we care about the real place the memory allocation * request comes from. */ -#define kmalloc_track_caller(size, flags) \ - __kmalloc_node_track_caller(size, flags, \ - NUMA_NO_NODE, _RET_IP_) +#define kmalloc_track_caller(size, flags) \ + kmalloc_node_track_caller(size, flags, NUMA_NO_NODE) -static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags, +static inline __alloc_size(1, 2) void *_kmalloc_array_node(size_t n, size_t size, gfp_t flags, int node) { size_t bytes; @@ -660,75 +689,53 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc_node(bytes, flags, node); + return _kmalloc_node(bytes, flags, node); return __kmalloc_node(bytes, flags, node); } +#define kmalloc_array_node(_n, _size, _flags, _node) \ + alloc_hooks(_kmalloc_array_node(_n, _size, _flags, _node), void*, NULL) -static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node) -{ - return kmalloc_array_node(n, size, flags | __GFP_ZERO, node); -} +#define kcalloc_node(_n, _size, _flags, _node) \ + kmalloc_array_node(_n, _size, (_flags) | __GFP_ZERO, _node) /* * Shortcuts */ -static inline void *kmem_cache_zalloc(struct kmem_cache *k, gfp_t flags) -{ - return kmem_cache_alloc(k, flags | __GFP_ZERO); -} +#define kmem_cache_zalloc(_k, _flags) \ + kmem_cache_alloc(_k, (_flags)|__GFP_ZERO) /** * kzalloc - allocate memory. The memory is set to zero. * @size: how many bytes of memory are required. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1) void *kzalloc(size_t size, gfp_t flags) -{ - return kmalloc(size, flags | __GFP_ZERO); -} - -/** - * kzalloc_node - allocate zeroed memory from a particular memory node. - * @size: how many bytes of memory are required. - * @flags: the type of memory to allocate (see kmalloc). - * @node: memory node from which to allocate - */ -static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int node) -{ - return kmalloc_node(size, flags | __GFP_ZERO, node); -} +#define kzalloc(_size, _flags) kmalloc(_size, (_flags)|__GFP_ZERO) +#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1); -static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags) -{ - return kvmalloc_node(size, flags, NUMA_NO_NODE); -} -static inline __alloc_size(1) void *kvzalloc_node(size_t size, gfp_t flags, int node) -{ - return kvmalloc_node(size, flags | __GFP_ZERO, node); -} -static inline __alloc_size(1) void *kvzalloc(size_t size, gfp_t flags) -{ - return kvmalloc(size, flags | __GFP_ZERO); -} +extern void *_kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node(_size, _flags, _node) \ + alloc_hooks(_kvmalloc_node(_size, _flags, _node), void*, NULL) -static inline __alloc_size(1, 2) void *kvmalloc_array(size_t n, size_t size, gfp_t flags) -{ - size_t bytes; +#define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) +#define kvzalloc(_size, _flags) kvmalloc(_size, _flags|__GFP_ZERO) - if (unlikely(check_mul_overflow(n, size, &bytes))) - return NULL; +#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) - return kvmalloc(bytes, flags); -} +#define kvmalloc_array(_n, _size, _flags) \ +({ \ + size_t _bytes; \ + \ + !check_mul_overflow(_n, _size, &_bytes) ? kvmalloc(_bytes, _flags) : NULL; \ +}) -static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t flags) -{ - return kvmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kvcalloc(_n, _size, _flags) kvmalloc_array(_n, _size, _flags|__GFP_ZERO) -extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +extern void *_kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) __realloc_size(3); + +#define kvrealloc(_p, _oldsize, _newsize, _flags) \ + alloc_hooks(_kvrealloc(_p, _oldsize, _newsize, _flags), void*, NULL) + extern void kvfree(const void *addr); extern void kvfree_sensitive(const void *addr, size_t len); diff --git a/mm/slab.c b/mm/slab.c index 026f0c08708a..e08bd3496f56 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3448,18 +3448,18 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, return ret; } -void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) +void *_kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) { return __kmem_cache_alloc_lru(cachep, NULL, flags); } -EXPORT_SYMBOL(kmem_cache_alloc); +EXPORT_SYMBOL(_kmem_cache_alloc); -void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, +void *_kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags) { return __kmem_cache_alloc_lru(cachep, lru, flags); } -EXPORT_SYMBOL(kmem_cache_alloc_lru); +EXPORT_SYMBOL(_kmem_cache_alloc_lru); static __always_inline void cache_alloc_debugcheck_after_bulk(struct kmem_cache *s, gfp_t flags, @@ -3471,7 +3471,7 @@ cache_alloc_debugcheck_after_bulk(struct kmem_cache *s, gfp_t flags, p[i] = cache_alloc_debugcheck_after(s, flags, p[i], caller); } -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, +int _kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) { struct obj_cgroup *objcg = NULL; @@ -3510,7 +3510,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, kmem_cache_free_bulk(s, i, p); return 0; } -EXPORT_SYMBOL(kmem_cache_alloc_bulk); +EXPORT_SYMBOL(_kmem_cache_alloc_bulk); /** * kmem_cache_alloc_node - Allocate an object on the specified node @@ -3525,7 +3525,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_bulk); * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) +void *_kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) { void *ret = slab_alloc_node(cachep, NULL, flags, nodeid, cachep->object_size, _RET_IP_); @@ -3533,7 +3533,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_node); +EXPORT_SYMBOL(_kmem_cache_alloc_node); void *__kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size, diff --git a/mm/slab_common.c b/mm/slab_common.c index 42777d66d0e3..a05333bbb7f1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1101,7 +1101,7 @@ size_t __ksize(const void *object) return slab_ksize(folio_slab(folio)->slab_cache); } -void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) +void *_kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) { void *ret = __kmem_cache_alloc_node(s, gfpflags, NUMA_NO_NODE, size, _RET_IP_); @@ -1111,9 +1111,9 @@ void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_trace); +EXPORT_SYMBOL(_kmalloc_trace); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, +void *_kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) { void *ret = __kmem_cache_alloc_node(s, gfpflags, node, size, _RET_IP_); @@ -1123,7 +1123,7 @@ void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_node_trace); +EXPORT_SYMBOL(_kmalloc_node_trace); gfp_t kmalloc_fix_flags(gfp_t flags) { @@ -1168,7 +1168,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) return ptr; } -void *kmalloc_large(size_t size, gfp_t flags) +void *_kmalloc_large(size_t size, gfp_t flags) { void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); @@ -1176,9 +1176,9 @@ void *kmalloc_large(size_t size, gfp_t flags) flags, NUMA_NO_NODE); return ret; } -EXPORT_SYMBOL(kmalloc_large); +EXPORT_SYMBOL(_kmalloc_large); -void *kmalloc_large_node(size_t size, gfp_t flags, int node) +void *_kmalloc_large_node(size_t size, gfp_t flags, int node) { void *ret = __kmalloc_large_node(size, flags, node); @@ -1186,7 +1186,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) flags, node); return ret; } -EXPORT_SYMBOL(kmalloc_large_node); +EXPORT_SYMBOL(_kmalloc_large_node); #ifdef CONFIG_SLAB_FREELIST_RANDOM /* Randomize a generic freelist */ @@ -1405,7 +1405,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; } - ret = kmalloc_track_caller(new_size, flags); + ret = __kmalloc_node_track_caller(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); @@ -1429,7 +1429,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) * * Return: pointer to the allocated memory or %NULL in case of error */ -void *krealloc(const void *p, size_t new_size, gfp_t flags) +void *_krealloc(const void *p, size_t new_size, gfp_t flags) { void *ret; @@ -1444,7 +1444,7 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags) return ret; } -EXPORT_SYMBOL(krealloc); +EXPORT_SYMBOL(_krealloc); /** * kfree_sensitive - Clear sensitive information in memory before freeing diff --git a/mm/slub.c b/mm/slub.c index 507b71372ee4..8f57fd086f69 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3470,18 +3470,18 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, return ret; } -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +void *_kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) { return __kmem_cache_alloc_lru(s, NULL, gfpflags); } -EXPORT_SYMBOL(kmem_cache_alloc); +EXPORT_SYMBOL(_kmem_cache_alloc); -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, +void *_kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) { return __kmem_cache_alloc_lru(s, lru, gfpflags); } -EXPORT_SYMBOL(kmem_cache_alloc_lru); +EXPORT_SYMBOL(_kmem_cache_alloc_lru); void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node, size_t orig_size, @@ -3491,7 +3491,7 @@ void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, caller, orig_size); } -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) +void *_kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); @@ -3499,7 +3499,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_node); +EXPORT_SYMBOL(_kmem_cache_alloc_node); static noinline void free_to_partial_list( struct kmem_cache *s, struct slab *slab, @@ -3779,6 +3779,7 @@ static __fastpath_inline void slab_free(struct kmem_cache *s, struct slab *slab, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); + alloc_tagging_slab_free_hook(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. @@ -4009,7 +4010,7 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, #endif /* CONFIG_SLUB_TINY */ /* Note that interrupts must be enabled when calling this function. */ -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, +int _kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) { int i; @@ -4034,7 +4035,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, slab_want_init_on_alloc(flags, s), s->object_size); return i; } -EXPORT_SYMBOL(kmem_cache_alloc_bulk); +EXPORT_SYMBOL(_kmem_cache_alloc_bulk); /* diff --git a/mm/util.c b/mm/util.c index dd12b9531ac4..e9077d1af676 100644 --- a/mm/util.c +++ b/mm/util.c @@ -579,7 +579,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node(size_t size, gfp_t flags, int node) +void *_kvmalloc_node(size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -601,7 +601,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node(size, kmalloc_flags, node); + ret = _kmalloc_node(size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -630,7 +630,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node); +EXPORT_SYMBOL(_kvmalloc_node); /** * kvfree() - Free memory. @@ -669,7 +669,7 @@ void kvfree_sensitive(const void *addr, size_t len) } EXPORT_SYMBOL(kvfree_sensitive); -void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +void *_kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) { void *newp; @@ -682,7 +682,7 @@ void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) kvfree(p); return newp; } -EXPORT_SYMBOL(kvrealloc); +EXPORT_SYMBOL(_kvrealloc); /** * __vmalloc_array - allocate memory for a virtually contiguous array. From patchwork Mon May 1 16:54:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9276C7EE29 for ; Mon, 1 May 2023 16:56:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8140190001E; Mon, 1 May 2023 12:56:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EB1C900002; Mon, 1 May 2023 12:56:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63D6790001E; Mon, 1 May 2023 12:56:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4C6CA900002 for ; Mon, 1 May 2023 12:56:10 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 039F6C07CD for ; Mon, 1 May 2023 16:56:09 +0000 (UTC) X-FDA: 80742288900.15.667AA66 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 22264A0012 for ; Mon, 1 May 2023 16:56:07 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Db2dTvDW; spf=pass (imf25.hostedemail.com: domain of 3J-9PZAYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3J-9PZAYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960168; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YfoToyHVT1LJlGJVQ6MuRV3hbhaLXh5d5qfc4HkIPik=; b=PVMEPwtfZaHbylhC+momYvjCN4KI+Az367XC+7/tQMkbjqKMidJgZOVaQi8W877NGgDjTq 91k3awxxQ0Tt1eR5kxu9z9cZkD9KjNXMlXffd7aNRLe/9XomPF+ZuED9Mk0kZTbm5ClVJ2 BQzW9mepJuZgjr1YcSzpKgILIrhJqBc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Db2dTvDW; spf=pass (imf25.hostedemail.com: domain of 3J-9PZAYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3J-9PZAYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960168; a=rsa-sha256; cv=none; b=hVIhot11oSX253dFbjzfdJnvOkSuLnv9sP6aPkqFdGSSnwe38g2Ng+kupQv5yt5W9pLWwO rsXeVo0175M4myY/4PM9JlahsObKdrvqprxWMU3G+oVl6sn5fsXmozqoGfrPwve68WMy5f TeD689kYa5AFS58iFkT3Ak2ai4ZYR+Q= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a8075bd7cso4837247276.1 for ; Mon, 01 May 2023 09:56:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960167; x=1685552167; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YfoToyHVT1LJlGJVQ6MuRV3hbhaLXh5d5qfc4HkIPik=; b=Db2dTvDWbMM6MtFyLXOOyG+p1K651ww8uL+kJDohPMMG6EWYrhfJFGluhHEW0wqXl5 UKPrByNQ1Z4QXnisfDyfKX7dbeY+3mxaMGuFVRBjDUr3JiqQISt6WeUr2I6JPah8BGyR FjdvJFEuIXgNlkkoVLLQxMPMSKPuyrjPesVcRR9te3ugJsI7tT+EXrko3AZ8ov2CcY8R JR28aghvPN6+RkIPURXyrGkWco+xEzOLoyotOADRZq9o/XJ68RcgnDEgHpMVXMMei422 +DGh+VKx3LXxPBgO4eeggvJc6/61DcK7Fvg52O8gRCKPK/nk5YYM+Q4LDqT5D1RFb4LF rBwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960167; x=1685552167; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YfoToyHVT1LJlGJVQ6MuRV3hbhaLXh5d5qfc4HkIPik=; b=F1UeJpkmON37ujJNuD8auz9nKhwK88joAOIdd6NsUXoppjTMePaeZYsrpKMoS92M6v 0pgxmy9LQNaPaDmOmmagXxA+ROnXmCehmjlCYCrHfUA/CjALMj2M7NqPMqt+ik6XltnI H3izDQ7MjW1w7s7MgRO56gXSGdrBfrKJpTKi3nvGmvqLCUzSSQuDq5DAk703dom0sSxl vk847GX98YkrHtjCswtE6fZNRwgsxu1OUn1xSpbEmBiszTg5mvrjm7iexdjzy5j2CTvd 8eYktNZGuOJ2E41fvAi2vBYgS6Lc0SdmBVREPGL72s8eic7snQS2nDGXg73F5BtY1XzG UJDQ== X-Gm-Message-State: AC+VfDwglY4GD0zMxt0ZgdLIPc8cVYKMipet5fvxNBVxTzxKAwsZDfMh qUm5dtPoivFjh/8oZ7SIU/TcIFiGvqA= X-Google-Smtp-Source: ACHHUZ40g7B5Nzrue3Inkynn2rCW8wBzh6gvt9xgi1/WmwP7LCTtOXWTAuDa8Mc648etXigh7Ci0w5kPSCE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:c0ca:0:b0:b9a:7cd6:ba7a with SMTP id c193-20020a25c0ca000000b00b9a7cd6ba7amr5449586ybf.12.1682960167516; Mon, 01 May 2023 09:56:07 -0700 (PDT) Date: Mon, 1 May 2023 09:54:36 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-27-surenb@google.com> Subject: [PATCH 26/40] mm/slub: Mark slab_free_freelist_hook() __always_inline From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 22264A0012 X-Stat-Signature: n15ba3rebmsscrd7fx1nepyoiucoudq4 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682960167-610852 X-HE-Meta: U2FsdGVkX19siJe/8FWVU5uF8awb6LaSmZfFSHtoJPYwZdRRblOazI388chLhKOPrbnW9yoCSlB6Xj87Tq9gRES5zOop4NapFtWmBNMQvjrIPhzbCmpkE2CEub9hEvqHOiu1SSBPPZ78FdYI0e8xgvDbtMw1w+D2YzmQH20ZtyHkCVTLf+Q/QL6CbqSXUnxLJGqHDruCWTFoClaoC/JwRT2Ruo56PgUff211bWzdzXgqlwkwL15Vo1LkTrhQWiCp8jZ0wL19muyfVxlMF8SRU+qqJvGx1vv9WV/lMcnXn4WN0/NO2ud36jW9r0xnn10os6eSLfw7+heunXdnlIivA7J/CtY9d6qRisQGW1NR7VCGADd1S/2p9n9WFeHi/WqRBuuDjmM/9XzvrzGZsD9ql1Fhy4XkmGM6uPmJ9I6KtZsm9kTDr5aFANvYYAHkpp8QoZcIpsVRAi+RnWO1fvof0BMko0VfXn17VcaNbJtwKCq5eDgNL34MdrwvDBI8AsfnsXovyLgfnuJfqXauakmoP08fq7lN73jKP6y4pWdx78pKVt9Vx/ZiVTtN4Wg9Mwo7v5obRGWxjZ3PW83poO1CbOw8rjvqgrapk2c8y38+n6tM917zATYiRtdIqoFChfTY3tTQCSDGirhGUEwqz+b/U0C3hWmGv7kkWqb6O1+c7ghUPsagjoYJYviRbmOSVvIKw3867wQsUqsOOxVQDyHqAduIjhnilu4oThoVaTC04xseHMFKPpQvFnotIvljN9E6TWCkw02SxST/wGLEEfKJt7N1Sw0vx7LvEJxYBGQ8rfljyOUjDjA6O3ErgiG68/1rAw5VYjJE+TFw+GBxrOqI03oucyt9JviOExtCSylvQ4VomCQxL1FoQObohDCvRzUj76CZ2nkQnyb/QOjj1fShnny9EVHslY3268VsI8T18+kJwO2QsLsYmTAiRGYW/9V59YlOWZU+5T+tlUMgilv zzgHDop+ FcMvRRVpU302K+clQ0KMZpg44dLxFYkwDyaS/MubrflJ7Hee01Qbfi8wPru4szLF/M9d2044HZlo0TGt9cRjNQgq+F2SnlBzew1pNfoseSo8uaUHBAkt5sfwCGo/s8zdpIq3/NIoeFdbzNJhd3p7N2wABi1k69ooJXDaec90Ld1QaRWIrEHcO0JQRb9BHZ+npS/QPaULoZ4g+cL1r9NFPEUt2eWCUnTq8cEo2VpPE8RHznkyfg3wNl1p6MHB9OZ26M/7EJVSmG+L9TrXNAB8AowMmaG5cabBgzZ7m080/m6Hv3naI30+qFTwheJ1rmEDsoSJO7UzuZgIOH/bbCYERDbqvv2jqYf2HhYbD1JHwEkoxlUgDsmVQ8zg7pOCtv8GUVLUxhkUpj3cwyVWbOE+LKNdpQLQiAfa9879mvkmJUsQv1sd0UP6LL6gSNzPikXFug01o1Jqq3pOeCyTSv/XrJ4Q3EBP8c5q+aIbw8ezWGjSJG5nz8tRpk1UXvH011SkFEciNwxci5eJIj9tW4QzCgbsRRXhzi0oaUSx6gNtmmMULP2S3eljfdmkpIMSirpLN8nLMmA3IPv+fsAwWvVoh3juaZkJ41sTNB02DujFUaL5gvjD0+CbYHY1jTmredD/l8kCPQyT7wb8TZlOmHzGEAN5e/gQ0h5vVQLRTsUUkuUmj9zQZGV32mLeVrwCWu4NC+yKmFTFBChYpnMgrmo73H5dTMgHWxwBuVXEL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet It seems we need to be more forceful with the compiler on this one. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 8f57fd086f69..9dd57b3384a1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1781,7 +1781,7 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, return kasan_slab_free(s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail, int *cnt) { From patchwork Mon May 1 16:54:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227662 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F63AC77B73 for ; Mon, 1 May 2023 16:56:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA599900005; Mon, 1 May 2023 12:56:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDF64900002; Mon, 1 May 2023 12:56:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8077900005; Mon, 1 May 2023 12:56:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B3B72900002 for ; Mon, 1 May 2023 12:56:12 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8CCA11A077A for ; Mon, 1 May 2023 16:56:12 +0000 (UTC) X-FDA: 80742288984.19.4A06E80 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf30.hostedemail.com (Postfix) with ESMTP id AC63680007 for ; Mon, 1 May 2023 16:56:10 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=WvFBiHp4; spf=pass (imf30.hostedemail.com: domain of 3Ke9PZAYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Ke9PZAYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960170; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sqvz5oeuoJjzyL5d6pTK46j/h0psspf4+63tRdELHAU=; b=0KpG04WPJ2slxSOv3qvRnfP+MNEz5C6q8C9o1AfPRLTuyg88ts5AVE4S9qqioFAVBR18if bGtYNE1qFsMfGQBaIIZWG9jaMiutdv8zrk7qlgwWm6yEGaOCWg8RydU0iBTvOgdC7xBAVL WZrjcl0g/Gnd+8A2K2MmVbQh2WaplDM= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=WvFBiHp4; spf=pass (imf30.hostedemail.com: domain of 3Ke9PZAYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Ke9PZAYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960170; a=rsa-sha256; cv=none; b=4kxdYq2H688++g6B+pGP+RNtvaQOcd/PNa3yoQ4/TRivJsiRPE9rMsvrVt0inFclD0spfq jwqSZc+VsICDnUqPSA1lhNEOUUE3aJvUaEKCgWo/HDJYx8bW4aY2LqlXdqNHT03H2HjJ+p z7GOYCy57oNlBn6Tkdw7WdrEVv/gnms= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-b9a7550dca3so4767345276.0 for ; Mon, 01 May 2023 09:56:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960170; x=1685552170; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sqvz5oeuoJjzyL5d6pTK46j/h0psspf4+63tRdELHAU=; b=WvFBiHp4GUaavo4X5tfWgjPPIzI4jGTUdYosSfeYytpBQUHvpJznwgCxcgtRl4137n /lBXJJHc248hdfGU59NQ6YgyXWglquQNEf83f6NZrWhqBExIs3wpePGzi3G6GXt6Q4EK YNi+IUVCZ2K+oon9eJeUuOHyA0Dti1Ev+im7AvpjkpEZ+nJa/TIMCcKnkDRhn07rC5v5 U+jUUTR0hv29E56yn3HNUOsZcTbbcZMOOzG+OmYRSipaveAVFeX44ceC3CgOIteuddHZ pJc9MAEnRvahuFY90dt+kNg7+emyX/nC+hRarTb4+wBTNE2Gp4VjzhmNP/RRS0EOHEJ7 5wQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960170; x=1685552170; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sqvz5oeuoJjzyL5d6pTK46j/h0psspf4+63tRdELHAU=; b=PSpv0+df+pJixmalSNcScfj5NLfccMpYN/1Hut1rBUKOX/mBEyOG8l0jPgYYSFosbi 9VJkuNPZYLXdAzoPsxHpnaoxaHv+SJYUI1gCiPpKWu0QSVAXhcf+sNd2JDDKPBUoxxW1 e0oda1nY0abKwC6bQr3iFYJANdunJSQq1IOtiHHUASlpDKTm8FGbZrgnpvyEklsC1/zM mLMPoSTZIdNzi+pUmWndOlana87B57bgQsU4jWTqiJV6twnMBWJtnZOPqh9zxT0Yfeca u8tiAAKPSpc+LTf7T2x1cZB1D8uI+QDoCCW7uFULw3VLefhshm5Fla0ZPHm3lxTDKb/8 vrQg== X-Gm-Message-State: AC+VfDzSdz7Zny6+TS7VKnV2j4Qj1UzI+WixzlmDdUxuJ/o2p9G3jW1F HAwdS+hD/lfwOfuLTib1DKya6mL3K+A= X-Google-Smtp-Source: ACHHUZ43Hwp6e3JER3sQv6NxKXEjKe/RSZbOmkRcgQXOYFHdzPjxH4Sg/oRvIRxv7gUZWm7HfdAOIYovC7U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:cd08:0:b0:b9a:7cfe:9bf1 with SMTP id d8-20020a25cd08000000b00b9a7cfe9bf1mr5044873ybf.8.1682960169618; Mon, 01 May 2023 09:56:09 -0700 (PDT) Date: Mon, 1 May 2023 09:54:37 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-28-surenb@google.com> Subject: [PATCH 27/40] mempool: Hook up to memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AC63680007 X-Stat-Signature: rf975dmeoq8b5bz9d998hfn6epxi8u38 X-Rspam-User: X-HE-Tag: 1682960170-143512 X-HE-Meta: U2FsdGVkX19XGPD6wOmntEBmUyuvt55EU1UMs9ab24zOwd1IQxYAgWyGs9piPHq7Rf9WL/cNpaJQ/L8FT2Z3BbwO0kwAgSBFs1XQxNJgMDfOoQ78KBjAQeQpu7LvPe43GONG122+KkA/NegLo8newZ2iGQn/B84FInSzptaERRtzzriOw2o8wx4WPJdI34xlMgNlyN2yq6bdl++rys8S4cBhkOJ2u4SJ59s2kSL13ZZ91upHaXUplMJ2NHehLCJ7A563vO3STiRXBcowkrsMWJZ7KpZs8F/JL6mI5Rx0LWADgpuKjpybc+I1aFgDnEMXmbpWELXKSdBqeV5HJ7bwfoHGzfGe9wyVI72PUxyBAS1i3ijSlCJXGwge612es2GnG8WRQpF133drUSFX4Pvp16y2sM0BRyRiC54oah6hdG7NazJjxqrD+qvC8USx1le//GaLc5r8dxZJrbTlEdMqxY0xcaXtCQUuBq/b+RY79qgZT2P8JmEE7Ywr9fvdlTFx+hRuX+H/54TrhiWl4TutV9nDE+5h4TRakhtRe/34c0lc5lTT1I5nogRQYSAUYbWb0qOd0DqLFef87j+2m0qo4bXUiQRxaIcMy6pl9w5TLApUgEKKbDsI0ZlqqEKGXATi5wbP29UwrOdi81ucfkXjDBF1HW9pSLzCC9R47j+K3e2P35e8aH5XF1teIlcEABwTcpEqZ0iRz839K2Dm+BL5RBRotDj8xJbJrGvUfHhPbPTLElrqHHBrk8VtD2cDvaL1t1VktyZaTcjy+L1QMUeQbIDidoSL/1Ym8y97A6JviGC230xNuVfI7u8iGhutuIdhb3ihyFGhAq9RcwD8UaZSEkMK9i3Rgy4zRIz1HZ/2kgI6ZzIySXvoK2pTZxQC6jVQNm/JwnyGrLP9Lrx/BbRVUGQar4w9R4l8R8xMyX6PUTXVezS4vBn+e9uScN9WsYrMZ4W13wB50iDDqixKjDe eN5UgisG VRu4j01uWWeruE130DJXzpCeTl0NEIHIZD/8bUTB3/+E927X0qcKOaWVhlPPqE2TlbTmL2sGYsPc0+ILNmAXC1757YmjOSw1VpXjNQBOS/DscKnbfHvn+U/u3mw/CkkSwSuN9sT7CcwZgBRePGsKlUkBQ+a/AtdkdzTm4KFYeIj6sukFFy2WqaNCou+wMH+Y+lFLvpv+vBLSRDdEKqLGGg9hU6LtSHn5kPRpFd8X6tKG6NPSXYA6xa2GMnV7maAljx/URnEhf2Zig9QJtdTVl4bXmi7ghJ5+dd5n/czQFAz+8u1vF5icTmzFCWAA7JUZYAG1S2sz5JrBf7/TteOaPxpVVLYjB20v2YIOPXu/nIW1sHLgKoFuFPBvy2mU3hfWcm62jeRXRBmQU937acPRRflCPEOHIlRvWepQU/4GvOrYDtDSfl2cfjMuhfcsd5x11VV5tUZG1t/PjA6J+eVh7Sr8FLUfalQdLaBiZCOG/xg04yjHtGxl+wWdUQSPggLmSeF6IALYQTzrRsa07UxyR/VhuFn8wrUXdZOqtq5bmXHMAvmcqvPmarfJjwlI0mUymR5jX4YLzNG+IY9kzprnr2JDMKiTmf1wDDjGl3eAiv9ruVzG0AsLRaIEJjLPHNT0Ejwl8bxqaK4p0U+8rF7Jfxmi+3chsBhupyP2AiJ6H3m2nKYE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet This adds hooks to mempools for correctly annotating mempool-backed allocations at the correct source line, so they show up correctly in /sys/kernel/debug/allocations. Various inline functions are converted to wrappers so that we can invoke alloc_hooks() in fewer places. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/mempool.h | 73 ++++++++++++++++++++--------------------- mm/mempool.c | 28 ++++++---------- 2 files changed, 45 insertions(+), 56 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 4aae6c06c5f2..aa6e886b01d7 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -5,6 +5,8 @@ #ifndef _LINUX_MEMPOOL_H #define _LINUX_MEMPOOL_H +#include +#include #include #include @@ -39,18 +41,32 @@ void mempool_exit(mempool_t *pool); int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id); -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + +int _mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); +#define mempool_init(...) \ + alloc_hooks(_mempool_init(__VA_ARGS__), int, -ENOMEM) extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); -extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, + +extern mempool_t *_mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int nid); +#define mempool_create_node(...) \ + alloc_hooks(_mempool_create_node(__VA_ARGS__), mempool_t *, NULL) + +#define mempool_create(_min_nr, _alloc_fn, _free_fn, _pool_data) \ + mempool_create_node(_min_nr, _alloc_fn, _free_fn, _pool_data, \ + GFP_KERNEL, NUMA_NO_NODE) extern int mempool_resize(mempool_t *pool, int new_min_nr); extern void mempool_destroy(mempool_t *pool); -extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; + +extern void *_mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; +#define mempool_alloc(_pool, _gfp) \ + alloc_hooks(_mempool_alloc((_pool), (_gfp)), void *, NULL) + extern void mempool_free(void *element, mempool_t *pool); /* @@ -61,19 +77,10 @@ extern void mempool_free(void *element, mempool_t *pool); void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); -static inline int -mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) -{ - return mempool_init(pool, min_nr, mempool_alloc_slab, - mempool_free_slab, (void *) kc); -} - -static inline mempool_t * -mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) -{ - return mempool_create(min_nr, mempool_alloc_slab, mempool_free_slab, - (void *) kc); -} +#define mempool_init_slab_pool(_pool, _min_nr, _kc) \ + mempool_init(_pool, (_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) +#define mempool_create_slab_pool(_min_nr, _kc) \ + mempool_create((_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) /* * a mempool_alloc_t and a mempool_free_t to kmalloc and kfree the @@ -82,17 +89,12 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); -static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) -{ - return mempool_init(pool, min_nr, mempool_kmalloc, - mempool_kfree, (void *) size); -} - -static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) -{ - return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, - (void *) size); -} +#define mempool_init_kmalloc_pool(_pool, _min_nr, _size) \ + mempool_init(_pool, (_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) +#define mempool_create_kmalloc_pool(_min_nr, _size) \ + mempool_create((_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) /* * A mempool_alloc_t and mempool_free_t for a simple page allocator that @@ -101,16 +103,11 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); -static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) -{ - return mempool_init(pool, min_nr, mempool_alloc_pages, - mempool_free_pages, (void *)(long)order); -} - -static inline mempool_t *mempool_create_page_pool(int min_nr, int order) -{ - return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, - (void *)(long)order); -} +#define mempool_init_page_pool(_pool, _min_nr, _order) \ + mempool_init(_pool, (_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) +#define mempool_create_page_pool(_min_nr, _order) \ + mempool_create((_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) #endif /* _LINUX_MEMPOOL_H */ diff --git a/mm/mempool.c b/mm/mempool.c index 734bcf5afbb7..4fc90735853c 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -230,17 +230,17 @@ EXPORT_SYMBOL(mempool_init_node); * * Return: %0 on success, negative error code otherwise. */ -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, +int _mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data) { return mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); } -EXPORT_SYMBOL(mempool_init); +EXPORT_SYMBOL(_mempool_init); /** - * mempool_create - create a memory pool + * mempool_create_node - create a memory pool * @min_nr: the minimum number of elements guaranteed to be * allocated for this pool. * @alloc_fn: user-defined element-allocation function. @@ -255,15 +255,7 @@ EXPORT_SYMBOL(mempool_init); * * Return: pointer to the created memory pool object or %NULL on error. */ -mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) -{ - return mempool_create_node(min_nr, alloc_fn, free_fn, pool_data, - GFP_KERNEL, NUMA_NO_NODE); -} -EXPORT_SYMBOL(mempool_create); - -mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, +mempool_t *_mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id) { @@ -281,7 +273,7 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, return pool; } -EXPORT_SYMBOL(mempool_create_node); +EXPORT_SYMBOL(_mempool_create_node); /** * mempool_resize - resize an existing memory pool @@ -377,7 +369,7 @@ EXPORT_SYMBOL(mempool_resize); * * Return: pointer to the allocated element or %NULL on error. */ -void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) +void *_mempool_alloc(mempool_t *pool, gfp_t gfp_mask) { void *element; unsigned long flags; @@ -444,7 +436,7 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) finish_wait(&pool->wait, &wait); goto repeat_alloc; } -EXPORT_SYMBOL(mempool_alloc); +EXPORT_SYMBOL(_mempool_alloc); /** * mempool_free - return an element to the pool. @@ -515,7 +507,7 @@ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data) { struct kmem_cache *mem = pool_data; VM_BUG_ON(mem->ctor); - return kmem_cache_alloc(mem, gfp_mask); + return _kmem_cache_alloc(mem, gfp_mask); } EXPORT_SYMBOL(mempool_alloc_slab); @@ -533,7 +525,7 @@ EXPORT_SYMBOL(mempool_free_slab); void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data) { size_t size = (size_t)pool_data; - return kmalloc(size, gfp_mask); + return _kmalloc(size, gfp_mask); } EXPORT_SYMBOL(mempool_kmalloc); @@ -550,7 +542,7 @@ EXPORT_SYMBOL(mempool_kfree); void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data) { int order = (int)(long)pool_data; - return alloc_pages(gfp_mask, order); + return _alloc_pages(gfp_mask, order); } EXPORT_SYMBOL(mempool_alloc_pages); From patchwork Mon May 1 16:54:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96A23C77B61 for ; Mon, 1 May 2023 16:56:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26E55900007; Mon, 1 May 2023 12:56:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 246C4900002; Mon, 1 May 2023 12:56:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07176900007; Mon, 1 May 2023 12:56:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E4334900002 for ; Mon, 1 May 2023 12:56:14 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C4A47A06D4 for ; Mon, 1 May 2023 16:56:14 +0000 (UTC) X-FDA: 80742289068.26.E4EE1DE Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf09.hostedemail.com (Postfix) with ESMTP id EED3C14001B for ; Mon, 1 May 2023 16:56:12 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Glr4JSHs; spf=pass (imf09.hostedemail.com: domain of 3K-9PZAYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3K-9PZAYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jz3+CZtZwJ9/tSYfHBM/2MHA9DEGGjB4zlxzO096/9Q=; b=grYyQCvyyKeDgJ2JENVfY6rAB1aNOd/Izth6QwRyxVPsKPZtQlbVCOclahF3YWo9C2HrpS fivS6lEQXj5hHLH1wLsDDHo+ESHE+hNyKe5/5EimFL527bgX8WczyfuO41kjivuZ/Ygnu/ ipHjvpeTAjOMd18DMLWy05sNOXeycdA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Glr4JSHs; spf=pass (imf09.hostedemail.com: domain of 3K-9PZAYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3K-9PZAYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960173; a=rsa-sha256; cv=none; b=g69iDCHw6OLmB93PthqKJmTiRSz67U+ob9TXT1ewyGR0pJWnK7gm/m+/4X5c1yRYMTgOe3 ryxvZ9DUWJQm+qP+xyp8GnJMXAUX2EuEoJytHcTswxq6txPOvUwUVQiPkCsQj69XG6mluL MINEZJ3L2JD+ambS/TDnkCk0+NZ9CmQ= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9dcfade347so3082638276.2 for ; Mon, 01 May 2023 09:56:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960172; x=1685552172; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jz3+CZtZwJ9/tSYfHBM/2MHA9DEGGjB4zlxzO096/9Q=; b=Glr4JSHsTmaAPI0AWP0PWNaIttgzeyDzPQx4tjZhzhm0zZ4p+1GCcCC3c3r4aXKXSp WIW7l4ONjqrOBXvjUIVll03e3w9s+YRWlCSxmsKNK9xzzAcM54D0O3+CWQoKF/ytzIvC kOX+0OA2rpltLDjWT1I86SdUtJ7gM1/iT7nXNjbfE5eIm0GIboQlBljhXdjDloTQNmLw 70IARjApWsmBCF7eSF3LKPA2bTZVurXLcycCJn2u8FXoQc4EQGTI1K+d32OJcow714/u 3R8ajXD58HPiHypeEQaSKmOGWRsxZiDT+trPafMrQwqkk9Lr9LEGmo4kscoUpmICusqi Lbwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960172; x=1685552172; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jz3+CZtZwJ9/tSYfHBM/2MHA9DEGGjB4zlxzO096/9Q=; b=Ib2fNfcKpmUsduiFLXyeMyjqWoy8r8Eu0NdgqFZK+jK1Vo3yeYOtsWzwpwFLwBo1Kx /UCjotjPhRMvUchff2MSwp/4ztfFO20h/Zlz2zbm4fXpHbuPpBaXwonY/l1Hp0iALnwM ZGXLqMLKohwy/4nK/7mLyU2h2x1xP8FkgjIsRC26n30/dw3ZbE2QSOq7wYHU5E0xk/vV vg1uqRXGTNdBhIpqkfDM1Nc69yoIwOprpSoXTU+hSEtq92LGu3zC/FJWubvZd5r+W3j4 LxglklnVbb6aMvBAg4R04xynkPg74tK4/spCtla17AX7ajbgqaL5bzzGKqS8/wsa2qVe khrA== X-Gm-Message-State: AC+VfDz353Dy5r2qM3Q1/EXysosSiAA2V82qel6+qa3bx+X3BcjoqfN+ pLXsfU+bayBolQLy86uFNZP6dClup9U= X-Google-Smtp-Source: ACHHUZ6QTE71DtRYGwVPEik8Avnxkjne4lD1OdWVt1RsKvPsz461K/ZnyuUPS3XSYPpuvwa5uVYwtV8n9po= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:1388:0:b0:b95:ecc5:5796 with SMTP id 130-20020a251388000000b00b95ecc55796mr5071137ybt.12.1682960171977; Mon, 01 May 2023 09:56:11 -0700 (PDT) Date: Mon, 1 May 2023 09:54:38 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-29-surenb@google.com> Subject: [PATCH 28/40] timekeeping: Fix a circular include dependency From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: EED3C14001B X-Stat-Signature: jmjcbk4wwmd6hd8hrqkdojdid4brdz3z X-Rspam-User: X-HE-Tag: 1682960172-514305 X-HE-Meta: U2FsdGVkX19g87J0VnEUD8g9npF+Pum0AygDIFFsy4z3SVDEKDTg2OfECFWQ3GFB/2TufXzaduMGVeJ1SeR1Qfq1M72qyNMHEtbeNB4jaOCSyF4YgYz+Cm48HHN+wG9h5BTdxB0z6YQ2bfyoda+I5f2W0p1ugQ5Ipg9RwYVym8Rpz8qQQq4azG766GtoYK1Ykw4del+Nuwb/9qGZl1Ai2Q0WPuTaQ0N+EP5k8Tlk5A7C+u9qKqPIzzadTUbExpfk3TTMsz6iqtG/2ZqdjtEqBM//nls7dSibjyTBLW8b2ziaLLQvdxo1cDJbT7qLhfxeHwbHVFGb/5wPyKo0wWSgCxbLkRkeY8Sq03Nzl+VgUzlpoPLGR0hFUBlF+cAzLKn0esfidUbH94HtENdrNxbMY6fN67CWMv1zMksLymMr2o0j/63fxyKJfn9fy6SDnqy+hZRe0cKr1OFfAhsknFoiurqArCvlK1xwpIDfaw/0ieiQCvRsg9EqlyrN6C7scrfK0NBX+b6O4bm3grxDOD2Aj19yArhjja1AEq5Pw3/mAB43B68GlCr0ZSuZxdyryQeJZK+2ik/es5v7kP5JDE5bLACH29lKHVVmeUdZ0blzIJiVMTaP8+8JwqlcoM/l0E4bgRRU4HCG5IatQfMDDI0FoSLqIDCilYUHmCI8+ZKniJh5SJ1cKlMAkzn9K25WKPQn/yJXGEeH4hwWNyX7aLUAhPqMHLWGPCm+WjE1bA+trQAvlirNwoYB/T1itOQkHfu/IpDRbTWb+qluTKPq+IC2iYsiRceJVlS4ToXf0XQK9aTJiTW9uScxiYw01m4VGtyXbycxMggj5x1R3FpUAb8czUsh5+5opC5dfaLiTRqVSsbGPPBp0BeB6f1ShwekX9P6DVFEefLe6Yg5Xz4mQryl8szVFk2Pl7W2w/7VQISxnucbzGHNymoD6GVdf1Ons1Sd7usSkYVQ4Ck5fuMuMnW TiJV590t Oarycxy0LS8xvu4Bor9M+QIVjFZvefqhJqQVKrADwuOFL1pRWE1b2h8fMjppLRa3QaHupseSTZnoJL1gpOgg4HU99L1Yd6MKru9J7uX5RhjnhN1PLdxIcFl9mvk6VFb18740x+naAjN2H/dD1diORUp0d8WRFiTL70oJNKZPBl5WI2kx76ORtA2FqyjrWZ0emoeAF5HTjVlmk5fHKjBa/6/R7n4vT4Su47ibhcPXO/fEEw7/vVAjK796tKiq5BYrMOQNwHohZpeok1HkkYqxPOJfHg0/GWQpuD8oxm8PbvffS8xat+g3b2TnMvqsDgW6mVGdHw62ocu3oow16QZRgMfo3Z/aIoXNNrDOSHbrSk+tT9ohI1dHi0J2nhRVXabmEbNKcu74Gmp/5YqrqOU0CVdnAHsBYesc4aFcSGRytkrexpgwGzJKzRPOReRE9ZYMHAP3GxlLoaU0rPhqlk5kYEmsdTAfeV4EzQ3p3ECXUzbdH8xewklExyYY79Xdxj8g0eBCY4RS1i5YASUANsqDeoxUXxA/WGau43KNthUSuzBB+NAF3IFxLsTn0HDqzHl8qY6uNa3amCdDPNypEwWs8qnaJgaLisAlV3jWERfRsY0UjnZWXgP6JCLaOO7fpYBdBFflIvigvwkujQVoR0G9vRZrRLhfUzKH7ZiA1VIW3MoYjIpE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet This avoids a circular header dependency in an upcoming patch by only making hrtimer.h depend on percpu-defs.h Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Thomas Gleixner Reviewed-by: Thomas Gleixner --- include/linux/hrtimer.h | 2 +- include/linux/time_namespace.h | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h index 0ee140176f10..e67349e84364 100644 --- a/include/linux/hrtimer.h +++ b/include/linux/hrtimer.h @@ -16,7 +16,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/include/linux/time_namespace.h b/include/linux/time_namespace.h index bb9d3f5542f8..d8e0cacfcae5 100644 --- a/include/linux/time_namespace.h +++ b/include/linux/time_namespace.h @@ -11,6 +11,8 @@ struct user_namespace; extern struct user_namespace init_user_ns; +struct vm_area_struct; + struct timens_offsets { struct timespec64 monotonic; struct timespec64 boottime; From patchwork Mon May 1 16:54:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD928C7EE26 for ; Mon, 1 May 2023 16:56:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A82C900003; Mon, 1 May 2023 12:56:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 358A5900002; Mon, 1 May 2023 12:56:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21EE3900003; Mon, 1 May 2023 12:56:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1380D900002 for ; Mon, 1 May 2023 12:56:17 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D93D6C07D3 for ; Mon, 1 May 2023 16:56:16 +0000 (UTC) X-FDA: 80742289152.11.81C831E Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf29.hostedemail.com (Postfix) with ESMTP id 22A52120003 for ; Mon, 1 May 2023 16:56:14 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=DmSOIqXp; spf=pass (imf29.hostedemail.com: domain of 3Lu9PZAYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Lu9PZAYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960175; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k4gyiXZ+rLWKPzhf5TCPslUr+ioI4/2asgnCyiPdOEo=; b=5mzWIkopeM2jdaYMOwHluUv+IWoxsNrXLydJrYQQPmqKyhZfAReORps7BFseuRttSGq2oj VL2cfSbhu5+Y/Ej++R3Ng1B08rHibuFTazaPu1735B0BUm/m49dtZteg7RGckY+90S0PYn QELJEDw0KaZAFLJHpFF/tfEWTPxwRpc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=DmSOIqXp; spf=pass (imf29.hostedemail.com: domain of 3Lu9PZAYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Lu9PZAYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960175; a=rsa-sha256; cv=none; b=0gCJv6z++vsS5vBDxcrpns6s6PymtOp9xebRhvDEXEbYXhrToMXGIsm7yXJTkOf65gYaUa prqB0Xeay7RPI0yiv+dwWD390n8lM/z085KUAhUZtvbSGWBkTWuyQSNM2owOByEcW1G/0H GAob0sYgvowATFAMeItM5Xa7UD0J8bI= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a792ff423so5508328276.1 for ; Mon, 01 May 2023 09:56:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960174; x=1685552174; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=k4gyiXZ+rLWKPzhf5TCPslUr+ioI4/2asgnCyiPdOEo=; b=DmSOIqXpDpgOhsbYBtSJd0pE8NBIuLLzTwqet8jr5NLvFjfEu90BXeWtYv7qlPqaG7 S7+D6TPAoWHLs26LhS/oYb03UoMwG4HhBUCufMxi2OXT8Cg0Wg5ZW9rp2ejUU7/dM0E+ g9Ptf/gGhu2ayAN03NAMSR+d1VG5nCdJ9H6WMvt6TJ0j29Cpmun2V6f3j+OoydA/EMqH hhtOdKXFKMT9O6F/RJeaiymHmIAjupnQaY2DJyokZEtCEBhHqOPrpU0cVWh3zhhtOv4w vPFJ1809YCMUcOTdA9cBgXt/0HwdTb2v9IdEptjtwIgtSwcOESs6yl5SKx9kBZVUQ0L8 /sDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960174; x=1685552174; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=k4gyiXZ+rLWKPzhf5TCPslUr+ioI4/2asgnCyiPdOEo=; b=bRQVoPHO/P0IpB1+yWBO0WuyxfbrrcJy4D2AGjPR06gAozV0YenqXFa/HKw1e5tP8m dno9Y4Jacbo47pqKtctKLJJfMlUAJTQ0PCP3ePVft79kG6WC16hmgeOO8FASPNk2g+u7 VGCB7RDnc9NppnnCF6sXRi/wdKKRtT/UdUAdx8sjrlcA1075179XXu+BGoT7FPqLwEX7 uf7os32iZhUcKFTXjEAAguUYiyKpSIoO1nEktMej3haVUN1LwYUextq/sLTKI75kXOoG lXF6zrdKjV8l3jicwWG0mOsP2yrj6UwH5J4dNa7t6U4GBsxFdRwuEhcWxH11071O/Xdg 3PdA== X-Gm-Message-State: AC+VfDyzQIBHSxnfQmT2J8K7HnUtI6ZXqgqMbmYKuaiTepIJJlXEqTdr a1XU0peRNVO1nJqm4Y/sYMC2zaEjRHE= X-Google-Smtp-Source: ACHHUZ7aGSgmCsBUI+di96C7Ib72o4O5NVupLg+6oUGpNI3RTseMV4sdkNx3JpSisCExB2ulLnUeWNtVkhg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a05:6902:1081:b0:b9d:d5dc:5971 with SMTP id v1-20020a056902108100b00b9dd5dc5971mr3225339ybu.2.1682960174070; Mon, 01 May 2023 09:56:14 -0700 (PDT) Date: Mon, 1 May 2023 09:54:39 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-30-surenb@google.com> Subject: [PATCH 29/40] mm: percpu: Introduce pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: kyg8haym9dm5uzqs65gczgfxcwi5t1fi X-Rspam-User: X-Rspamd-Queue-Id: 22A52120003 X-Rspamd-Server: rspam06 X-HE-Tag: 1682960174-736512 X-HE-Meta: U2FsdGVkX19piwuaOwbxHGP9RPKEI4sreyk/mQSoTdbnNgGOHOP46sC0pptQws3S/SOWshw8NoDPizY35MH5lNy3pJmHfEUgGZNUFn+3C204NHptAcBvo6cCfWApwSIDnJK71BWU+gjNM0xUTryEXIOxmBGDG25EMDXn7DPFufD0TMHwh+vZZMTM+AMgMBrtiOaPkA1Eb9l79JJT2XoVFUE6nVupbdR83AVIRGmfBtPByqx13r9xsTOoDTY+tcGmMlIt9R4QKlsDg4nO8OCxg0GABZyf+CdeK8BMD9Cfw8Iu1sO/yfGqrFc8TIzB2xv0maj/aF2HNDmecsid7HKSJUYI5y0VZXsa0I08nIQyo5hLt9ZdY1iXrPpopwld3VuJYtG/9IiJc2GaqwDAz0wDQp12NrYvphznK86NekAG01HG0DnvqwYEwY1bb5l2ujUELuCSOiBht/lKQRurPEayX0Y+8pvuJ+bvrhwzb8C0Y756HNc9d5Zu3VMl2+G6OBX5XSq4gIQSeQV6DhR3Stc0/RPU8dK7d4Z6LbsHCJXZZc8GFEvqH1UPG36TX7TEfPenyJ2rY3rRMfu+7p/BbVRnf/kybQbegumFL7yDhhAzVqRHrZEpYG3e0wXLhZujr6Sr6ZJ9QpE1VR40bhAVeshckAxR4h0wj1GkDHZHrFZ27OCImdFanqzkJ0FKv62RAFPQyHN8tsUctCf+b89y0dWrk9bmtMY7DmWwyeQIpgXXSzuz1EXzDOt8myUuj4BewOeS8z88LYEp0M0FimnEq2ZiQ62msMP/vpoLGrSmXUB8/B7CqXdWp2Jth2D3MJwMGo1InZC8692FPolqJ3MGR/p8blnlmVbWL3mp+VAfKDAg40VUGncjIW/g6ma1eduXT5Ti1Zxue9NLGifb2zey41yQrLnE4di9N/8FGw0zir+Z7nLc8m97YuVHQHwRcHuN9MrvRoNcA0C0xcWLfDKLMQk 3clgxbxy oLhn8dzwVb8rVzVyHG/RFkjFLOSLZ9BYdyd4npuK2xAd1oPqSsoX8Q7AaeGrebubxtqALmlUdiAOahNBswuqwtPWkIGgAA3qkKD+1HU9wPedk9P7N+Kwe8ZM4GnYkoIERBbWSs0Lki6frn/UlrWiomJ1YuCDz9n4NWAt8ENaRmWhVwzjGL0UyQwY9vLrg7fuZM5d/L1WJeseTtrNhZh5eOD/ns+ew2u9r6Yawh8jeokezFf1GhbOWAfMsnPtvkgVq+A3e2UWwHhyVe5HddW0ckbUogeBTpi4O6mJ7GNzsjjpe7UxiwsOcI/mlkJDDqbGoCpyU+ZW0B1l/gRj0aggQKCWJ2HGpoJlTJCfEdX3EHY6BayiCjKJFKsm5lLI3XwFmuRNSJN36xdH/LiRQnRkV5A4cOtsqfAd/SckxC2JUQCrqEbyyvgohZytgqSEMizcY9h3rG8VkWZzaEaaKrOBYb0KaRkXpMk1FUBL66eFsUrOJKMvYTm9piivq6InGbU2yEZlmHC8RuGSXUZUh3RSJKEMYj3VQwX4iLAzbPvdJo9OkO0bb//58pBHAb1HqPZKr3FQADjp9tj1INDuKnsh6H+GHwRE9odXzGoebW3a3VK1vK8+CEiKEWjMdWm+FJQqOqI2MGXlCfcDaf/R9jglJBy8ax/Y+trDjnBBNbqDiQyeEnt4LQycwuQYD05di4YbNRYYjhHEGPH03dwELOIkCfoVrM4vUA+7I32UxYQmkM3zKc4F5ICtXPd3urTZxGYJek/AsdoCrozIdueFSrKE5JExpvqsrUtNn5Hc/cWOUK08JongiIgRcv//Ah9Rh0RuNLFSQJW+NzOCwhVAZsyChMIyK3RzzgS0tJXkOgicZQ/0m4HoqXjJJ/kcErmbchFD/fmUeHa5J2wQo81oA4cFprIp8/FI+x0oRTahavvPbtpjQNNM5bCwUCUTnG52aupOxZCMvtbjEzgNfuJmwvdp9gz0A4A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet Upcoming alloc tagging patches require a place to stash per-allocation metadata. We already do this when memcg is enabled, so this patch generalizes the obj_cgroup * vector in struct pcpu_chunk by creating a pcpu_obj_ext type, which we will be adding to in an upcoming patch - similarly to the previous slabobj_ext patch. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andrew Morton Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: linux-mm@kvack.org --- mm/percpu-internal.h | 19 +++++++++++++++++-- mm/percpu.c | 30 +++++++++++++++--------------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index f9847c131998..2433e7b24172 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -32,6 +32,16 @@ struct pcpu_block_md { int nr_bits; /* total bits responsible for */ }; +struct pcpuobj_ext { +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *cgroup; +#endif +}; + +#ifdef CONFIG_MEMCG_KMEM +#define NEED_PCPUOBJ_EXT +#endif + struct pcpu_chunk { #ifdef CONFIG_PERCPU_STATS int nr_alloc; /* # of allocations */ @@ -57,8 +67,8 @@ struct pcpu_chunk { int end_offset; /* additional area required to have the region end page aligned */ -#ifdef CONFIG_MEMCG_KMEM - struct obj_cgroup **obj_cgroups; /* vector of object cgroups */ +#ifdef NEED_PCPUOBJ_EXT + struct pcpuobj_ext *obj_exts; /* vector of object cgroups */ #endif int nr_pages; /* # of pages served by this chunk */ @@ -67,6 +77,11 @@ struct pcpu_chunk { unsigned long populated[]; /* populated bitmap */ }; +static inline bool need_pcpuobj_ext(void) +{ + return !mem_cgroup_kmem_disabled(); +} + extern spinlock_t pcpu_lock; extern struct list_head *pcpu_chunk_lists; diff --git a/mm/percpu.c b/mm/percpu.c index 28e07ede46f6..95b26a6b718d 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1392,9 +1392,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr, panic("%s: Failed to allocate %zu bytes\n", __func__, alloc_size); -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT /* first chunk is free to use */ - chunk->obj_cgroups = NULL; + chunk->obj_exts = NULL; #endif pcpu_init_md_blocks(chunk); @@ -1463,12 +1463,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) if (!chunk->md_blocks) goto md_blocks_fail; -#ifdef CONFIG_MEMCG_KMEM - if (!mem_cgroup_kmem_disabled()) { - chunk->obj_cgroups = +#ifdef NEED_PCPUOBJ_EXT + if (need_pcpuobj_ext()) { + chunk->obj_exts = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * - sizeof(struct obj_cgroup *), gfp); - if (!chunk->obj_cgroups) + sizeof(struct pcpuobj_ext), gfp); + if (!chunk->obj_exts) goto objcg_fail; } #endif @@ -1480,7 +1480,7 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) return chunk; -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT objcg_fail: pcpu_mem_free(chunk->md_blocks); #endif @@ -1498,8 +1498,8 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) { if (!chunk) return; -#ifdef CONFIG_MEMCG_KMEM - pcpu_mem_free(chunk->obj_cgroups); +#ifdef NEED_PCPUOBJ_EXT + pcpu_mem_free(chunk->obj_exts); #endif pcpu_mem_free(chunk->md_blocks); pcpu_mem_free(chunk->bound_map); @@ -1648,8 +1648,8 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, if (!objcg) return; - if (likely(chunk && chunk->obj_cgroups)) { - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; + if (likely(chunk && chunk->obj_exts)) { + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = objcg; rcu_read_lock(); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, @@ -1665,13 +1665,13 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { struct obj_cgroup *objcg; - if (unlikely(!chunk->obj_cgroups)) + if (unlikely(!chunk->obj_exts)) return; - objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + objcg = chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup; if (!objcg) return; - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = NULL; obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); From patchwork Mon May 1 16:54:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227664 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4319C77B61 for ; Mon, 1 May 2023 16:56:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E07A90001F; Mon, 1 May 2023 12:56:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 48FFA900002; Mon, 1 May 2023 12:56:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 333F890001F; Mon, 1 May 2023 12:56:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1B0C1900002 for ; Mon, 1 May 2023 12:56:19 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E4CE3C07CD for ; Mon, 1 May 2023 16:56:18 +0000 (UTC) X-FDA: 80742289236.24.0762108 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 15D31A001F for ; Mon, 1 May 2023 16:56:16 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=yzTHI1C2; spf=pass (imf25.hostedemail.com: domain of 3MO9PZAYKCHcnpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3MO9PZAYKCHcnpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=utmd4o6Wxf9dKKkm5+p28Q6keQxRmEnIpppKzQNbjdk=; b=n1pRDj2q6zYv2Q5dpZKfF9t7IPdps3hXjxxu7bE9GfNFAqXlh4BGpIwgYsqu912arYlJm8 Ud0X/8xV+WhjYMQmZubmsRc9XeCZBC4bU9sEJgC4wpCFS29lnnPfWd2vZlsr9/bJETmSjS fzy7YPDVnKRerOCD2KLl3Y0FdoDDUSE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=yzTHI1C2; spf=pass (imf25.hostedemail.com: domain of 3MO9PZAYKCHcnpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3MO9PZAYKCHcnpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960177; a=rsa-sha256; cv=none; b=0Lm6zz/q33jfwePM9WbPh12xoI3XilRS6fL8hWNmheyA1dXQ49k1ylqGcoT+koPDXSbt2g j/pfA0ZwXc/KivVx1qkECVBKfAScmm9Uswfa1OD5fOPmZwmjeloBHEqT/fa3LiQV1xMNlk 4Iu2rTDtI+IYXZGryy3VLo7vAYTh1GY= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a8075bd7cso4837603276.1 for ; Mon, 01 May 2023 09:56:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960176; x=1685552176; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=utmd4o6Wxf9dKKkm5+p28Q6keQxRmEnIpppKzQNbjdk=; b=yzTHI1C2zJou5AH/p8lPs0VsHNeDHmv3kTFdxuBKDy961uXVpU9fU3i66ZX+sGZxop K9tHzqQGK2fyGSX0IUIKHt297nLCAVUUQ9uE+ycaL5j8vF1Uq5qjYO9IBdKFeRBRz+JV h0XCiQt2rQy/vjGuvtAMwYO/Ef8AO0zgdlUV7UkjU/4j8KZorjx6YpJycy3NjV+U1jmW 8YeCYGEoAGbwFJFnuS6l7DE88NDAQo1cCW0vxXj2MMAEUI8zWICBIQThmt4kh65kKXHV qjMw8B7CVE+Aq4G5oCD+SKPBqvWYSqi0ooDgoVvg+pFWLmMj9ApTsyrB4ZzuvS1F+m6r BcdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960176; x=1685552176; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=utmd4o6Wxf9dKKkm5+p28Q6keQxRmEnIpppKzQNbjdk=; b=Sl3sHgYcdj29bNharkPPkHyBwfzaXdBs6f8Eecoq5cLZT2UJNBwRJTUJT7MGke07LP 6+gVR519tiQRAI9Tvwl5u6D1MIJpCFuUNksUwY9LF9DJMDQcZHlyyCMD+qB58Zn84WQe uifM2IOJKaCttRtSZHmJHLiEWS6vKqWG/nQN3NIX73U9cZA2c6VYlojB13kkUpn72Pw2 +KOmrRaKe9ZCVb5ofaNUHcFWSZi98gTJCENtQ6kR8MrRTo+46ThmlkJYGPkrkA1t7Gs8 79xqL4XBp2PJgLp4b4mGZLGCCDTo0aVpSSpfwJvB68rIZM4AR/MA9YtSG0nDPlAC0FHt 1CxA== X-Gm-Message-State: AC+VfDyp4wlx1S4A/txLrpaLl9CiDDCDHxwjEdu9VfYkHjGvP3or2cgK hhrMRop1R8DvsvxYE3xLiHky4WHfDq4= X-Google-Smtp-Source: ACHHUZ6rEGe63NZRZF75NXTRoii6CEu/OB0qev1Vk2mogA+wxCpKeVLUFvVGVJNb6NKt9hysxhJvTpNQ+Ho= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a05:6902:100e:b0:b8b:f584:6b73 with SMTP id w14-20020a056902100e00b00b8bf5846b73mr5602392ybt.10.1682960176468; Mon, 01 May 2023 09:56:16 -0700 (PDT) Date: Mon, 1 May 2023 09:54:40 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-31-surenb@google.com> Subject: [PATCH 30/40] mm: percpu: Add codetag reference into pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 15D31A001F X-Stat-Signature: i6oemxrcgq8uqx1ykaem446skhejjojz X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682960176-257626 X-HE-Meta: U2FsdGVkX19RE79R6ZpBihiguQurHcJMaMn/VgaF1yr9ir4RZ5VueWfGt2tVnjXDIjuYon4iWurpDmqMhq5um51MobNQJ/il3H1U46UaYRjbcrGMv9S/5GHEwKoBYhuMAftA/taPLyR0PCPDUzV0IH5W4/08xXdfPEx0k7qRNhMFEoaY33XfzIvQ7GrsC5yX44LL0PLySKdTo+AQBk4yKDn1A3YWVeJPu8rSCFKWfROEzwPOlwj67rjInaca+4H2PjJxz8uR627zckJl+Ibci7/mRVULb97D/7p6wqbAcRdE+Kl98KJn2WCiMbGbI9KlSCAQVYLXklSdh2DQ/wWbkkhYz7u6KdmszNfyp7Is+HyvKFznjjjN7CvP4ivosjq6K1P4ZI3jT0WwDY/kYxMjFdFrroxo1475mGOTl90gjm1jhe+kzeX0QTI61X/PNlsd6Tl0GfJL7e8FRpMK4U64USrk1ymgmVYjQ9+hMOEdhvLJ4EyaXEmMOVUBE+Xxyz+1Q/vouiB+X1XXgmUbrAPbFaMXeyUQJRS6x35LJml7oJsoiHyu/eFCHihAi7SYI5tn/5nACxLvnfVc7L5CoI7KnfZ+ST5uUl7udjw/Ig84aARj51oaBFxgmMJMBzioqMltysn048hTvVMw6N65+o8s56FmIGvOdkFXEJWdzUxyUatS5tI9grOGa6r0/pNfbzcZIAqaF09BPdN8kHZPTjXt9Yj7krWpY48lFJz3zIqhUl7NnmPJvmbDjwf96TTlVz96/yTEmCEUNS1TalUruVQszxe1GgyFiL8H3CYy1f+9S53dRF7X2YmYGTsD51/1y6JM9yALRboU5FWghYKRW0W+UIf/s3MnHF+XwHB719YMS8GFvejnbHC/kAWYFTEJBxQgZ4byx6clHxe8TB3h96bGo82NmQMiIAyL+Zr/rn7dJayxh76LdEyrGMR/V8oIi838XSOARP6vkJ5kAz8/WeU Kmh9uZiK wA2rAaRQYfYYPZQpUOaq6R4I1LouP6/sb6SlbEmxBV+QETeAlCPPymfMsIahgFloTNtAb2gNNxyU1CEDJuLeH+YzXlLlTKQ/ZPxTrrP194dVFuAvxgPpx+JXiM2084fp3IQxDC6tTGIExScgenqzxzijpn8gyLw8W9osBocduxLVJPjHnHmXM8iufbIyB4RicqX5ZAh1nVlVcvKqA39vbIfy4V2Zvk5uj4BDOhc8i71wDzcr8G5AqGxn4Y/hzcWXfAc3CNWvQmo9OeLWwzsGiEI/Y2etpsPvjrDntpPrbj6d5HuxOFu/SSmy+fePLvZH9XVN6ZmG9D3H/u6AcoIO8Px+PLCM4gcH4wU/fW8aRKQ9qSEKdE8nQdXSfN3IdxyzfR3KmUymKbHiNIi2KNh2dUb+bm0RB7JjLLV9Lnkr7Yvn+ucN+P9IxbqDvlTsXA6apVkQmZunhgxRRXgyjFRVp7FTsLHieA59PrUjKqVvRIBa/5h63Xpgt8GSxFMw57nD8Blbzh6CHljZHSSCEKyCcJdQ2ToPkn7L3WuvvUvAT1RvAjmFI6wwErYmiExol8vZIrh7AwV/BFSnIQJD4wdmQQWwhASAM6GcUcWkSuSUVGEpRngJrLTeOVJSKGpdpA7KqpC51fdEgBy4rBudDAE+4lHnueKAMiaT9ScsoE1Qbqu89fdlLEOHVYmmSF4Wa7ZgZXIzEUZ02Z2SqV0FgMu0QdIe2/MMo53wRZ9L4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet To store codetag for every per-cpu allocation, a codetag reference is embedded into pcpuobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Hooks to use the newly introduced codetag are added. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/percpu-internal.h | 11 +++++++++-- mm/percpu.c | 26 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 2433e7b24172..c5d1d6723a66 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -36,9 +36,12 @@ struct pcpuobj_ext { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cgroup; #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref tag; +#endif }; -#ifdef CONFIG_MEMCG_KMEM +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEM_ALLOC_PROFILING) #define NEED_PCPUOBJ_EXT #endif @@ -79,7 +82,11 @@ struct pcpu_chunk { static inline bool need_pcpuobj_ext(void) { - return !mem_cgroup_kmem_disabled(); + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + return true; + if (!mem_cgroup_kmem_disabled()) + return true; + return false; } extern spinlock_t pcpu_lock; diff --git a/mm/percpu.c b/mm/percpu.c index 95b26a6b718d..4e2592f2e58f 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1701,6 +1701,32 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) } #endif /* CONFIG_MEMCG_KMEM */ +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { + alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) + alloc_tag_sub_noalloc(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ +} +#endif + /** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes From patchwork Mon May 1 16:54:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227666 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E915C77B61 for ; Mon, 1 May 2023 16:56:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0D59900004; Mon, 1 May 2023 12:56:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BBD1F900002; Mon, 1 May 2023 12:56:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A35E7900004; Mon, 1 May 2023 12:56:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8D57B900002 for ; Mon, 1 May 2023 12:56:21 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 474D2407F3 for ; Mon, 1 May 2023 16:56:21 +0000 (UTC) X-FDA: 80742289362.04.0832303 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 72EA240010 for ; Mon, 1 May 2023 16:56:19 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=6MUphdHN; spf=pass (imf17.hostedemail.com: domain of 3Mu9PZAYKCHkprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Mu9PZAYKCHkprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FLzZ9YV8+8xWjeDeRDWFnLJt6gby5ahlup+t8unODj4=; b=r9igLAt8ws86Xio4YnYlV1Y+sbVyn7ycIKVu+4kwm9lUbqjiyS5L8IcC4p7wBKfv0oWGTN pT3cRA3TES+UFHyyXuzxRL7LLxPCOSpvNp3cmK1hjSeUus5wNBhU/z/3vhz/uMkG9tI8wj aLDmc2Ukg1+ti/CGefJS0ZAz4kxhtOI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960179; a=rsa-sha256; cv=none; b=MQQRDRxs54s01j7esbOQ13lX2tmHRjYcpTELKTdZOwnPs+xn9n5HDXR24mkrT7mk5s/TjF I/Tu6sUASqwT2h9MTdlsU5+4Qm+b2btiorgRroJ+HXIRQe0LBJPKdhkXkQaU8euvgKy0II OUMf11LFPRiqk4r3rIUhgomBCLFFyqY= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=6MUphdHN; spf=pass (imf17.hostedemail.com: domain of 3Mu9PZAYKCHkprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Mu9PZAYKCHkprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b8f6bef3d4aso5547489276.0 for ; Mon, 01 May 2023 09:56:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960178; x=1685552178; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FLzZ9YV8+8xWjeDeRDWFnLJt6gby5ahlup+t8unODj4=; b=6MUphdHNce/Gp+joNLguOb4c4BxZ2K95cOfq0noWa5+zgK3o6mMkxvgmYTxKopfE12 gvzB4L7ekld5KXS4qJrG0iyIY1OJTcBOr1jnrj5Qj+XSjPc9d8l46YT4nC5jTNEYkgOa Ftsse48iJ0f92YHed3wWA/J0eqwt07vyXvq1REgD9bK10yNIvJKHoqfnJDmh6CoLNpLi pUPv1EUVjPqy0lNNdb2UUSVTJnBt6kuBEfOfDhudMx+qNc4MyWYtoUsdbUWBDNV9FGAp xUasOVCUh854FOcecSCfYemAj9x0uxOHD+fuFHr9oLGXAgOTof3XgPYghDgqHqQy10zL pGdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960178; x=1685552178; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FLzZ9YV8+8xWjeDeRDWFnLJt6gby5ahlup+t8unODj4=; b=R7JaiMMaDPPskEUtpY6HU6pWvP35CUprOMgM983sP9CqFv1mrM370NTpnlSML9IHdY P60rlGhcocrDPSBa4wYoiWLXWiO5GvOS/PvFjMJ5BObHg4PO33RsvBmuCRFwS34DZx9p X0dN3b6rpB1gIN/uWoZp2DlPeKRkHWkl5C6hWw1kaKtkgRhV24KGgxgRq0dmf+8D4tPu ddMHKQsJAQEtXGd+z1+ZrHw1+gwItcKl9yiueLE16wQpVEUZl/P5bHy6R7MmRYtS8I2s OxBwHS+JXh7FF8PuRdsvf801qGMDa5GsbjOOzVRt1Em+p3f3/Wy9ZBWspB80kWxoRd+z f1hw== X-Gm-Message-State: AC+VfDxTErk6pB90IP4fIS6xIMHlX5iA+3Arye7+by8rtB2Uzl1JFhpT R1EhxdHthwKDAE05Vr+YwOk1XD/+f4M= X-Google-Smtp-Source: ACHHUZ66kr0Wi3To7Fb9ZeLfyGM0k4cAx/qlYDAQpdI8t7cF2IOT9smAXhzqkgeAFGYo751C+YmUvQHpTxw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:d18e:0:b0:b9e:5008:1770 with SMTP id i136-20020a25d18e000000b00b9e50081770mr393470ybg.8.1682960178602; Mon, 01 May 2023 09:56:18 -0700 (PDT) Date: Mon, 1 May 2023 09:54:41 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-32-surenb@google.com> Subject: [PATCH 31/40] mm: percpu: enable per-cpu allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: 72EA240010 X-Rspamd-Server: rspam09 X-Stat-Signature: kndkjmececsycygm5resiyafiqmr1stf X-HE-Tag: 1682960179-39954 X-HE-Meta: U2FsdGVkX18awywGsb59Ym3LDjHF7Fy6nJ1Q2lmHRlwjoScU8ORpMdmBKepiD3SoVPHTYMGZNCcIlhgbBML96GsTWc9aXXpb1xFwo+xd5AhYw5tGP/nCdI+X7QR81Ro6mto34reRH3Yu4XhYvVrGImG9K7q41vqcHVAO3Wk8LP0MvV7xGZ1qS2YRIaJ0w95EgRKaXy9RqMQbyAEhERJpI1jpvb3K7HlQ4k4SweI+EdCn5bBMAFqr/Lrp22v0i+iVWlqvQHn58GcvBPYqxA3/wkszbCx1PVax1/EQ33NnWwQ1BPy9yUmmDWKK4UNsxd/mNuUwf2V3i+Y/DwtYtf0t90Zfqj81VsY8lMVavErNbld2AIb2D4aK2X+xOBj57A6CgdNhFVERLq7MVIMMpywauBtBq9VppeAC7+BU5zjC7Pww7L9JwyZQ4Cq2JYVyrxBzgleSxOpjIvk5VTyIDvTgJfAmhB3hZtgVYLlJTmDg1ADSrla7tDv5UqIBEkfX/uGc6CLcIrmITh1y8ucFi6mICR4L6uR/nvNThf83YCvlNLQcdySEcDuvi+xk0uyDGSQcv4Alz8q4lWf1iBznO0r5Al2poZPUqdnke2ReVvAkI2ovoTqUL26Psp+GsKrvf91f6FKMyedpLuKqjuJd8cwk2H/BemRzm0Ca0Frb5AA2QhaknPpYUQw4SvmHq7/1o/UN9Yn59exnSjt6zUB1P/t8d/R1BlhD5wYVBPJaon1Zl9KG6TB2VkFk7BcTFWHimsI4MQah92HOnOISsiakZhvYsRTiifF1kJcfaD+lDk+6pGXOuIbO4vMjx0KROfdnjUhDjGfH8nOSKMQSTg8MSiY2RXTT3SOt1TRLli1BhgfT9q7lyQu/tpMfCkboSH2lyXXhsLRY7x8/Nctr3eNFtrTSbRYOxam24P4QrX2Xi9ga0OX9A+NJkK286IoHW1lRkw0p5mN6EFrkXkRDava+kbp pe6JViW1 zcMLLhCk1fIUtyo/GF/1EA9XAJr3qzFGEmmhYF1nUSXERl0pGLUSR+oHbXgZf0oFsXPRaW3Ei1oLR69+DH6kglJunEP32YFjAq+dDUdg+3JxEdpJNmhgXIViA0aU8qGYqk9mDxUFnu7Ea7vpslIU3UIyLyQVf+8N9mMZB4E2RWWUotQcz53PU5hlot8z7IH6fGeC2AQ3LFadR/JK6dXhuOObynq9F+ue9tK5RBZX8Emi72c3y6SkN6F4BhNIX9Yw9oS4D+aQR9FQIR65Q86oQs/7EHC9u/UNs1iWPU9a0I8bcGvj3cm3fvLuh+BmbrG6E/JRbhfP01+IL0IsiJHAkAFl//AKtniALNWmKktO6nynqaEAg+WgtDLqFIEtYy77Al7IbOYBIYNLbM2hv9zX67TEzAfMh/sXmdITKpJIna5c5tSBGJWo8hrvuwXI4ebiAoRYAGR2O/Of1HSBTaQilhYbakiOH302HTQvG3r6RFhKlLZQclYYmpR9bKrvEC35KobWcmwY4+7x70Ht3KVVfTGHloZH4U4DPmvbQwmqJb35HrW+ygtwc24P3xcnFVZOhEwdccwrl7CeBVSH0VC2zDLHNAT6ChoXfrRBqlshHxJLZiLAZEc14eacD50ResFX0+zWekMEAt1V/rU2zK2gOqw9i/pUMew+ceOD9cAh4oH2OVbQyoHjX+xcwsw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Redefine __alloc_percpu, __alloc_percpu_gfp and __alloc_reserved_percpu to record allocations and deallocations done by these functions. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/percpu.h | 19 ++++++++---- mm/percpu.c | 66 +++++------------------------------------- 2 files changed, 22 insertions(+), 63 deletions(-) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 1338ea2aa720..51ec257379af 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -2,12 +2,14 @@ #ifndef __LINUX_PERCPU_H #define __LINUX_PERCPU_H +#include #include #include #include #include #include #include +#include #include @@ -116,7 +118,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); #endif -extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); extern bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr); extern bool is_kernel_percpu_address(unsigned long addr); @@ -124,10 +125,15 @@ extern bool is_kernel_percpu_address(unsigned long addr); extern void __init setup_per_cpu_areas(void); #endif -extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); -extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); -extern void free_percpu(void __percpu *__pdata); -extern phys_addr_t per_cpu_ptr_to_phys(void *addr); +extern void __percpu *__pcpu_alloc(size_t size, size_t align, bool reserved, + gfp_t gfp) __alloc_size(1); + +#define __alloc_percpu_gfp(_size, _align, _gfp) alloc_hooks( \ + __pcpu_alloc(_size, _align, false, _gfp), void __percpu *, NULL) +#define __alloc_percpu(_size, _align) alloc_hooks( \ + __pcpu_alloc(_size, _align, false, GFP_KERNEL), void __percpu *, NULL) +#define __alloc_reserved_percpu(_size, _align) alloc_hooks( \ + __pcpu_alloc(_size, _align, true, GFP_KERNEL), void __percpu *, NULL) #define alloc_percpu_gfp(type, gfp) \ (typeof(type) __percpu *)__alloc_percpu_gfp(sizeof(type), \ @@ -136,6 +142,9 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), \ __alignof__(type)) +extern void free_percpu(void __percpu *__pdata); +extern phys_addr_t per_cpu_ptr_to_phys(void *addr); + extern unsigned long pcpu_nr_pages(void); #endif /* __LINUX_PERCPU_H */ diff --git a/mm/percpu.c b/mm/percpu.c index 4e2592f2e58f..4b5cf260d8e0 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1728,7 +1728,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s #endif /** - * pcpu_alloc - the percpu allocator + * __pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes * @align: alignment of area (max PAGE_SIZE) * @reserved: allocate from the reserved chunk if available @@ -1742,8 +1742,8 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s * RETURNS: * Percpu pointer to the allocated area on success, NULL on failure. */ -static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, - gfp_t gfp) +void __percpu *__pcpu_alloc(size_t size, size_t align, bool reserved, + gfp_t gfp) { gfp_t pcpu_gfp; bool is_atomic; @@ -1909,6 +1909,8 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); + pcpu_alloc_tag_alloc_hook(chunk, off, size); + return ptr; fail_unlock: @@ -1935,61 +1937,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, return NULL; } - -/** - * __alloc_percpu_gfp - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * @gfp: allocation flags - * - * Allocate zero-filled percpu area of @size bytes aligned at @align. If - * @gfp doesn't contain %GFP_KERNEL, the allocation doesn't block and can - * be called from any context but is a lot more likely to fail. If @gfp - * has __GFP_NOWARN then no warning will be triggered on invalid or failed - * allocation requests. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) -{ - return pcpu_alloc(size, align, false, gfp); -} -EXPORT_SYMBOL_GPL(__alloc_percpu_gfp); - -/** - * __alloc_percpu - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Equivalent to __alloc_percpu_gfp(size, align, %GFP_KERNEL). - */ -void __percpu *__alloc_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, false, GFP_KERNEL); -} -EXPORT_SYMBOL_GPL(__alloc_percpu); - -/** - * __alloc_reserved_percpu - allocate reserved percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Allocate zero-filled percpu area of @size bytes aligned at @align - * from reserved percpu area if arch has set it up; otherwise, - * allocation is served from the same dynamic area. Might sleep. - * Might trigger writeouts. - * - * CONTEXT: - * Does GFP_KERNEL allocation. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_reserved_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, true, GFP_KERNEL); -} +EXPORT_SYMBOL_GPL(__pcpu_alloc); /** * pcpu_balance_free - manage the amount of free chunks @@ -2299,6 +2247,8 @@ void free_percpu(void __percpu *ptr) size = pcpu_free_area(chunk, off); + pcpu_alloc_tag_free_hook(chunk, off, size); + pcpu_memcg_free_hook(chunk, off, size); /* From patchwork Mon May 1 16:54:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A46CBC77B73 for ; Mon, 1 May 2023 17:04:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2440B900017; Mon, 1 May 2023 13:04:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F429900002; Mon, 1 May 2023 13:04:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0BC8E900017; Mon, 1 May 2023 13:04:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F0023900002 for ; Mon, 1 May 2023 13:04:07 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 95C221A07E3 for ; Mon, 1 May 2023 17:04:07 +0000 (UTC) X-FDA: 80742308934.26.062D946 Received: from mail-oi1-f201.google.com (mail-oi1-f201.google.com [209.85.167.201]) by imf07.hostedemail.com (Postfix) with ESMTP id 7EEC040033 for ; Mon, 1 May 2023 17:04:05 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=uJsYD79k; spf=pass (imf07.hostedemail.com: domain of 3NO9PZAYKCHsrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.167.201 as permitted sender) smtp.mailfrom=3NO9PZAYKCHsrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960645; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jY4Why07vSNVPpYg+gbjry+wnVwWuxvaUCerSTimV/s=; b=hf3EpocYZE9SGWUwxLY1F/NsieJeivxkbuBFdFSBGqI7fC6RRVTAlXqxG3nf+o4gUlfHuC uLiUtIMh750q5EiCSrQQS7UGM1UVYdlhst/mwJLqkcoSDhTB6ym1cZxDnwygLBhJ1c6PC4 92Q4OHpY8hVSleOH/F4mUvpvOMrdyiM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=uJsYD79k; spf=pass (imf07.hostedemail.com: domain of 3NO9PZAYKCHsrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.167.201 as permitted sender) smtp.mailfrom=3NO9PZAYKCHsrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960645; a=rsa-sha256; cv=none; b=vCYCnFla1HBFPCVfuGRjhrRjrdLCF1AY+EU1l/AGxzkz6DikJKNlwCTxNE3HG+U+PYpwF0 APVQE/Gj3jkdTVA16iKLcQ1+0WFNXC4aORzNyclmydvGHyJKeWdPUJAskGjQ4eTcUTuG1Q HNb5rv0yRKvNv/IiQDCmCEuMQRTRQ1E= Received: by mail-oi1-f201.google.com with SMTP id 5614622812f47-38e4f308c49so1895884b6e.1 for ; Mon, 01 May 2023 10:04:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960644; x=1685552644; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jY4Why07vSNVPpYg+gbjry+wnVwWuxvaUCerSTimV/s=; b=uJsYD79koNG3fLEJXfs80F3G7WdK8qVOuGnoRtlquShHeYoXOWzlD4WBjZBeq3Vrnv AKvQl520bU4LckJSArAcc7JgD+QO0eAfmPHhOiVnlYdRTYCo/tA9FSOKW5MomeGqCXjv mQ6jHpACgAwQocw/qAaHKc3ej+YL2w6K9adLmr5y+tpmoNqqGU0fQISrZEg3RBfMvBuH wGwQ0LYp6tOtxqD+XE+O7XQYRg7uPWF2zK+S+JbSSCVKHCRrA5RZ2y9mf+nvP3pu89b5 oEaWZn5QBJ5O73MGNKg1JzXEFwOA3KFCHNjQLdZ7Jyt5uDAYRYxyja3w0ty4kz0SbqvM sxnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960644; x=1685552644; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jY4Why07vSNVPpYg+gbjry+wnVwWuxvaUCerSTimV/s=; b=FHrckWA4MwFFrAIKlwA7noCnjPh8PKeqEiNu5uraUZBUznseUcy9tiDIUZs8Paq5bn 0vahXM0VTTvnJ+JLhXY3E3uO/HyUCZqG8sNtE1LohF+82kB8pATCFr1o4lINJpr9vxzj sre1I50sI4/0xToTmlBHKcYCOAjgJpMuRbmD6/Uxk23YU+TxC61sMaFq2r+k2qfkxr+L oOg7a+YaLY+OF2MixqRXLeQVJHW00hly9s4ubgP4/65QWObgzGzurVXnAEsC7Ae99Mak 1fg0wvxwvf196du4WSF/2vVZkFQA5vWZDQzvG3IrV2X/DAhr8fgeXGpthxZ0BSguNW+Q wweQ== X-Gm-Message-State: AC+VfDzHR3sAgguekkQxR5226gRls8aq67iM9JA7xry2YPqcSPeyjxfV zvB6QNbC5txcS1UvocbA/V6UD2x29Mw= X-Google-Smtp-Source: ACHHUZ6jXSxE+6iq2YUjl7qVMHUxfSxuowqD06Cn0jpzZX6iMdKT0+DoukFEFTwwgTX/LkDeTP78a66LQOo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:24c4:0:b0:997:c919:4484 with SMTP id k187-20020a2524c4000000b00997c9194484mr5789976ybk.6.1682960180824; Mon, 01 May 2023 09:56:20 -0700 (PDT) Date: Mon, 1 May 2023 09:54:42 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-33-surenb@google.com> Subject: [PATCH 32/40] arm64: Fix circular header dependency From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7EEC040033 X-Stat-Signature: atk7kagianesb4kftpmoz53tu9xqg4xf X-HE-Tag: 1682960645-90046 X-HE-Meta: U2FsdGVkX187idgTFotqkwx6j6d4AFaUZ5YTjwoFIf7a3O1vlT+M4wBGibZAC1nkMmKvy8atOqybbwYoKJIfOt9gli/76ktTOZk9FtDkzEU3VjwvJssUQH/vrXXoOBPC9sNvxyzW8GT9CDhn8CKwAcM3YYtOqT8yL0xscFCYZT3PJVTXCbw60MnF77eEmXCraa0pLvRIALHeeKDRIPb5vpC83EUznudA7qRCneHMheTXy9qNr1Ip6BCVvJ1KBwMbNA2fgrX1OIJSf7hbjbDROIJ/eqZtOlStDy8YWxtTMRyGRdTk1RYE/SPla2l90LxRtHRJsDXlxm+D+0SZZc8CXBZtnYO3goQePOTZZZdDJTRdaiagNeEI3R3CaPVr511ZomhNCAF/Iq/17LXURE9Czx5H8nRzVTgnzEVC5Tff01qAx6KOXocOtOK+iEZD8fVHundQ2Xp7e2af31msGFJHI3rL9aXOb3b+nu6QB8r/D0y1i+B9EE25QGHGKDcsDizbEbeUNHZu23fj4Qi02SVJRfeVZ8a0F2dGpQahmpdqVFdJfidEF19I8/dwY7bQxZdQjzFHVbSYJgjHjJdYRSDJ4KBJ1Sqqhv6RHIlIs55Ye/F4VXKnG4ZSUMItQgPeZ+OxYBVwVrVo7wNyERY8k8rfdYICyqlTdH6SLVO/JZ1dUKpV6dVfdvB1/yaH3/6P4DzUwHyFp7GnV6rEepwwUEs9MBd+Gd0fTO0PRCpcNejlMw0IRkxeOQgA5Y/DRLpdnUqrJ6k0nAayn/sv0UDh2ibBTk4qIteuppRVLv68fVhqIHgIOKXiGyfDwik5O1p+F/c5BCUBLbWZWcH1771e9gGjHn5Ko+Yu7uPfGGPROxeFKbsSFidI5JV6Np6sR3IgHkrcoHEn1tRGPpyYB2mWZcyJRqtseymCE4OZEvVllJtROURHQ6JzBQQ0K6NYftg4n9LaGVDCHQS/Eraj5e8vZBk RuLLRAaH a0mdGC8xtjh/B+7DVMPg+x4EFUZJ3YyMGEmM7Vf835iO78P9KGW/mztBXKDhJ8e6qvtLSriddG4kKR6ViYLr5vSXEWsVt5CEtZW2+4GLKBbLqg9p9QHZv+sgWQzTSCZoOe2LI5avB30cY85sbm69MERVl5lhC+Go3wQlMKR245izDZ8af4uQaxTMF1bwsAufq3z2WvRqUAIxpzzR0+Xm5Qx33LZbwsiyNPULhSJjKbngWR7mwwcwRAbmlSDH0qeNRWdblJ7Vd2EVPWZxSWbuJRsz2ut2DzhPs0soLg8fJUpy8SrWjh5r7PVFlBgAFpvycKZFZkqBlS5Q9tD//LZ5KDW93vqDi6nyjXrJ08ex333a4WhMXs4DAoIyLMkmcVJfNWVxrGBzMj+dbDooTfZsnbQs2YoI7swtFyidgHLWtjNqL3xEYNnNAVNAAZ8p9p8L2TUJgrTWxym4wCd/vlaVYj8OxYZFLCpLLFHCwu4RN8WrbHNGZmbq2+Ci+2OJ6cfVA2YLvtxZAwO3LC/nVZ4BIALqe7Ne3GH7SwbjINFBsUTltGD+VaCrG3U1dpaX5rgoXdZ/ZHNClBwoVuCvsWytJAR6HQocBI5lX5sRuoDq1LHihXYSxwM1Cnh+4+VvrqSQ3GcVVL61nPNT/7GPN/fQBQsWVIfReTLxrj6rKw6WYSshnpiy3CcxOAwljkDCdPz+KqRgtKGGYx9+qyo6/P3XvEQf9aw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet Replace linux/percpu.h include with asm/percpu.h to avoid circular dependency. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- arch/arm64/include/asm/spectre.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/spectre.h b/arch/arm64/include/asm/spectre.h index db7b371b367c..31823d9715ab 100644 --- a/arch/arm64/include/asm/spectre.h +++ b/arch/arm64/include/asm/spectre.h @@ -13,8 +13,8 @@ #define __BP_HARDEN_HYP_VECS_SZ ((BP_HARDEN_EL2_SLOTS - 1) * SZ_2K) #ifndef __ASSEMBLY__ - -#include +#include +#include #include #include From patchwork Mon May 1 16:54:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 722E6C77B73 for ; Mon, 1 May 2023 16:56:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10EFC900006; Mon, 1 May 2023 12:56:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C11A900002; Mon, 1 May 2023 12:56:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA270900006; Mon, 1 May 2023 12:56:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DAF43900002 for ; Mon, 1 May 2023 12:56:25 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B6C4EADA5B for ; Mon, 1 May 2023 16:56:25 +0000 (UTC) X-FDA: 80742289530.22.A6A167A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf29.hostedemail.com (Postfix) with ESMTP id D9C19120015 for ; Mon, 1 May 2023 16:56:23 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ebS+MuTL; spf=pass (imf29.hostedemail.com: domain of 3N-9PZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3N-9PZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960183; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bfMxO5ATd5n4S/fh5B/Nudv/nxDkBrDhDPER315pa0U=; b=wdQvCRndGxjXXQfzN5O7pwU6jy0CCYMUC6a5DyKxOFmWC/SwUjiHd1LUrEPH2Rx9Vlw9cQ uBT4SerOzJ97EpmoAq6fllO/Fwr5T6gA34sQztscY7o/l1tZYnjnMxql3iYLXE5L9GsDi2 M21phL2wgL8B0b0yMr3CNgbSUhtEkFk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ebS+MuTL; spf=pass (imf29.hostedemail.com: domain of 3N-9PZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3N-9PZAYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960183; a=rsa-sha256; cv=none; b=kmB/LQNVxbAjNPlR2puQn60BIjbqNC1rSQhI7irzYQcGJTbT/xIBpnV9k/HOBOjJfpv7jS HoWP4STETI0XazTaGzDWCNqMM17QQq+qJfj+nqTYAKjGGmsBVZzKNv1+FnyilpY9FBhbAO EOpJ8BX57SCW5BJztC9RwRYAcpQVHJ8= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a792ff423so5508728276.1 for ; Mon, 01 May 2023 09:56:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960183; x=1685552183; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bfMxO5ATd5n4S/fh5B/Nudv/nxDkBrDhDPER315pa0U=; b=ebS+MuTL+cQNWT4eZEJvTQOcTARn+QnzuJah4/SMgcb7+qkoUNzC7ex8MrpVKhZRXc KgeXlio4zrfdxmOppnmcCQe6TR+hOuBdIc9ihcKiKIEVRbsGHe8ObZbrrM4utujRh9wG TqBTS5rCEI/Jizsrf15UbPFacomNmio9VSdaY4o+amTMj91xth7X0uCHsGdXYXQ52AxU vnIxBaA8u69160qtwfedxuqqybJKyd5xlJm5XUDfPQCWKc/CaKvS4YCnPYvKKCkeigKq aazNcodiJ2//C8yradLEHTJBg1grANJmustwDAaYwDmuE541j3ivI8Z4B8/7N+gU15xY 5tZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960183; x=1685552183; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bfMxO5ATd5n4S/fh5B/Nudv/nxDkBrDhDPER315pa0U=; b=he1lpvw2qzUi1mfIqj8vigw+jN7bBt4clCD5etiqKg75vbLBSSOBH3VDRYomKSLB5Q u1k/yAFQI/FxzmL9v7zs3OZ7NAjw6k3XmJdvELWA+n218mQJCy9JfgKV3iS/9w2chC1H zVobUh4Sw16sPFBiFgBd27mYAU//TiD8Ir3Ofr6Ztbnd/KkZHbZGn2ibqMxxB3dBtL4v 0AJVOXTWiivGvcOEdTb/eIz4dBzemqxIAxt8GapKUfkOX401N3xF7G3Cq72b/B+u3SA/ VFA5a05z6RJysn934kK04KkTJ9qgA5sSiYu2yo39LiMCtKP0zkYrUNMpEn+fkC0VA6Cm h9kw== X-Gm-Message-State: AC+VfDwB9utf3sLa4cV4xW1WJGxhchbX4FDQoE0demFgviEdpPXH+iXu EjcFDCQUadGYv1xvTdqPiFg6t0sTILI= X-Google-Smtp-Source: ACHHUZ6mXAn8JgUNXvyGZqHJw9Jv8uWK4VnvTdJsjQhuuRu+wtW4LhhvfSOs0fXoyYvuKXJ26U7UG2NCZPo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a25:8046:0:b0:b9d:c866:d92d with SMTP id a6-20020a258046000000b00b9dc866d92dmr3485899ybn.1.1682960183189; Mon, 01 May 2023 09:56:23 -0700 (PDT) Date: Mon, 1 May 2023 09:54:43 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-34-surenb@google.com> Subject: [PATCH 33/40] move stack capture functionality into a separate function for reuse From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: abmd8te4szrta9ebzmk9qz5ngb6zq7fr X-Rspam-User: X-Rspamd-Queue-Id: D9C19120015 X-Rspamd-Server: rspam06 X-HE-Tag: 1682960183-59543 X-HE-Meta: U2FsdGVkX1/NA92+SMAIoFlhR+Kh+zMuuEvyw4Jo9qMw7fX9AKkX88vEwByd/MR88Sm9Bk/QLcF1dOzBpHQwb6XQdwUXN2zCo/+faSu+8wwX+ZFHzVwVMhdCgCjr9E6aQxyOspJTMKq7bZ31iF5LsI9tPrnklna6arV6xxwKtDeRFSNX+A4iYxWjBc2QA5t4wURmnDz1OrjMcmH1iTtmnym8AL2JHw5xDJjwpnk6eCXlp7GDYGzGRfvZaCuoSZdqxnVjny46BP6+ixfhSfR+PVZyhQV6UmHy7mVu/TACMJBg7A1gpnAq0/7jIDnnwtATlICJ1HsQCpOz9b8lo22//wLA2qXzUlYGWYPxVU9gggqIIX8UYmitFnpPyKrCJiu8iXLsQ5CtvS33u/NDc08fM/pVDjmYV7F9E8rYEjMG9/a3e983rCNn+ixWAaxHhlkohj3esjs8G23WuwSXIwdVwzA9qyHBMl0BAvDFNSFIeApxJVwQQeusbJ2kz74IipcrJ2sVWroXimWsP7YPE7i0MCx+NZ07D66UBJhDuBT1L+RkYIwU9CnO0181ujK7SF7molbKidiEx9AwECy2WZmOr75S6kF7wWD45L7be9g3aMwpO4XtZqoVBjy9Bpx8Ev7xhEP0GG9lQB7WIGdyIRIOx/zuPULbEUYHM2PsQfufQnQf/DMipRy2Il3fOgINOGN2wjO8VNts4GKnp8u9m9xDeenuQvh3L8C7VRMFX4wvciv9o/3BDVQIGqPhHchuhju8zsuVEBwbvZKrwK4wNxsus7gPf7ycbiX6dcWto8cn1/WyC8dm1eTccJW9SKog3LalLxV8RgCe2DFhgZFgCpONQGKFztokBsl+h9DVTnoH1zwGqrCRR5rQaYC8BhFODxMHmn/8FDwRwMRl35u+PXW7BK69ouHoSqOU1em3bG2rFLlQr1N5/aaNjN4lZ5J9qjZJC792bd6GUpZELzVRQB5 cV8EfKw0 K5qcPEdg096PJEqrnfmfHuHd9U/xURsrNOP33CRTrH7zXFIls/4uM3MP0lfLMQGbQFXofX2Q97zlEHMtqoXVIsQKYXLMHXv/H0gK21oN9RPM8ecWJh5zGUbiqnt2F2uPJg72V66zWBAQ60m/iggvsjzKXcfac5lq3GiZwGg21mG3KlePLR9+ybg61XQcgmdzLEuDiXHoIl3ersW/iFaBbddvEFT4UdYvSgcfAc9OzXtmgysbdO7wbJKy+fgiMB9kfeK10kpu90hC7r+9dkVY4WpyB5vj3QFci9Xfl51oaQnq1UpWDB9lBzTQ9G6WlzlakNAW3WAzFxWrA9GzlYrmM8QPWEcbweC5QvJuMjdZc2gcGVq/n4tPYut7mA4Zoef3PiAiagpDY8tDW+sBfY6sc1SubXiiPOcJY+/ev/sV1yUrjVhVA1BBEwsxyqxJohaSbZmXVdF4AfRj5jB12+4t/t7UvyPkn8DjLksbiPJwSXm0Fga5qlHJ6LP2ykRclLZ0GZqG/T2VmcQWI2W6GorKaIFE/C+oABIi6mkP9c/+E0UeiGyPi5ucg/mJyqHzeB2Xu2rTRyp+VCutpL6L3K0209WY0olK4EOGBXz+yhc0NWBIsQnOU2B/1dsJDdl9DeTXVEsd4M8cObATCVbMO3/kCLxATonQ3DJjqr7I3mgchRpNL2csiSMN88NXyMQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make save_stack() function part of stackdepot API to be used outside of page_owner. Also rename task_struct's in_page_owner to in_capture_stack flag to better convey the wider use of this flag. Signed-off-by: Suren Baghdasaryan --- include/linux/sched.h | 6 ++-- include/linux/stackdepot.h | 16 +++++++++ lib/stackdepot.c | 68 ++++++++++++++++++++++++++++++++++++++ mm/page_owner.c | 52 ++--------------------------- 4 files changed, 90 insertions(+), 52 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 33708bf8f191..6eca46ab6d78 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -942,9 +942,9 @@ struct task_struct { /* Stalled due to lack of memory */ unsigned in_memstall:1; #endif -#ifdef CONFIG_PAGE_OWNER - /* Used by page_owner=on to detect recursion in page tracking. */ - unsigned in_page_owner:1; +#ifdef CONFIG_STACKDEPOT + /* Used by stack_depot_capture_stack to detect recursion. */ + unsigned in_capture_stack:1; #endif #ifdef CONFIG_EVENTFD /* Recursion prevention for eventfd_signal() */ diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index e58306783d8e..baf7e80cf449 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -164,4 +164,20 @@ depot_stack_handle_t __must_check stack_depot_set_extra_bits( */ unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle); +/** + * stack_depot_capture_init - Initialize stack depot capture mechanism + * + * Return: Stack depot initialization status + */ +bool stack_depot_capture_init(void); + +/** + * stack_depot_capture_stack - Capture current stack trace into stack depot + * + * @flags: Allocation GFP flags + * + * Return: Handle of the stack trace stored in depot, 0 on failure + */ +depot_stack_handle_t stack_depot_capture_stack(gfp_t flags); + #endif diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 2f5aa851834e..c7e5e22fcb16 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -539,3 +539,71 @@ unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle) return parts.extra; } EXPORT_SYMBOL(stack_depot_get_extra_bits); + +static depot_stack_handle_t recursion_handle; +static depot_stack_handle_t failure_handle; + +static __always_inline depot_stack_handle_t create_custom_stack(void) +{ + unsigned long entries[4]; + unsigned int nr_entries; + + nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0); + return stack_depot_save(entries, nr_entries, GFP_KERNEL); +} + +static noinline void register_recursion_stack(void) +{ + recursion_handle = create_custom_stack(); +} + +static noinline void register_failure_stack(void) +{ + failure_handle = create_custom_stack(); +} + +bool stack_depot_capture_init(void) +{ + static DEFINE_MUTEX(stack_depot_capture_init_mutex); + static bool utility_stacks_ready; + + mutex_lock(&stack_depot_capture_init_mutex); + if (!utility_stacks_ready) { + register_recursion_stack(); + register_failure_stack(); + utility_stacks_ready = true; + } + mutex_unlock(&stack_depot_capture_init_mutex); + + return utility_stacks_ready; +} + +/* TODO: teach stack_depot_capture_stack to use off stack temporal storage */ +#define CAPTURE_STACK_DEPTH (16) + +depot_stack_handle_t stack_depot_capture_stack(gfp_t flags) +{ + unsigned long entries[CAPTURE_STACK_DEPTH]; + depot_stack_handle_t handle; + unsigned int nr_entries; + + /* + * Avoid recursion. + * + * Sometimes page metadata allocation tracking requires more + * memory to be allocated: + * - when new stack trace is saved to stack depot + * - when backtrace itself is calculated (ia64) + */ + if (current->in_capture_stack) + return recursion_handle; + current->in_capture_stack = 1; + + nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2); + handle = stack_depot_save(entries, nr_entries, flags); + if (!handle) + handle = failure_handle; + + current->in_capture_stack = 0; + return handle; +} diff --git a/mm/page_owner.c b/mm/page_owner.c index 8b6086c666e6..9fafbc290d5b 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -15,12 +15,6 @@ #include "internal.h" -/* - * TODO: teach PAGE_OWNER_STACK_DEPTH (__dump_page_owner and save_stack) - * to use off stack temporal storage - */ -#define PAGE_OWNER_STACK_DEPTH (16) - struct page_owner { unsigned short order; short last_migrate_reason; @@ -37,8 +31,6 @@ struct page_owner { static bool page_owner_enabled __initdata; DEFINE_STATIC_KEY_FALSE(page_owner_inited); -static depot_stack_handle_t dummy_handle; -static depot_stack_handle_t failure_handle; static depot_stack_handle_t early_handle; static void init_early_allocated_pages(void); @@ -68,16 +60,6 @@ static __always_inline depot_stack_handle_t create_dummy_stack(void) return stack_depot_save(entries, nr_entries, GFP_KERNEL); } -static noinline void register_dummy_stack(void) -{ - dummy_handle = create_dummy_stack(); -} - -static noinline void register_failure_stack(void) -{ - failure_handle = create_dummy_stack(); -} - static noinline void register_early_stack(void) { early_handle = create_dummy_stack(); @@ -88,8 +70,7 @@ static __init void init_page_owner(void) if (!page_owner_enabled) return; - register_dummy_stack(); - register_failure_stack(); + stack_depot_capture_init(); register_early_stack(); static_branch_enable(&page_owner_inited); init_early_allocated_pages(); @@ -107,33 +88,6 @@ static inline struct page_owner *get_page_owner(struct page_ext *page_ext) return (void *)page_ext + page_owner_ops.offset; } -static noinline depot_stack_handle_t save_stack(gfp_t flags) -{ - unsigned long entries[PAGE_OWNER_STACK_DEPTH]; - depot_stack_handle_t handle; - unsigned int nr_entries; - - /* - * Avoid recursion. - * - * Sometimes page metadata allocation tracking requires more - * memory to be allocated: - * - when new stack trace is saved to stack depot - * - when backtrace itself is calculated (ia64) - */ - if (current->in_page_owner) - return dummy_handle; - current->in_page_owner = 1; - - nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2); - handle = stack_depot_save(entries, nr_entries, flags); - if (!handle) - handle = failure_handle; - - current->in_page_owner = 0; - return handle; -} - void __reset_page_owner(struct page *page, unsigned short order) { int i; @@ -146,7 +100,7 @@ void __reset_page_owner(struct page *page, unsigned short order) if (unlikely(!page_ext)) return; - handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); + handle = stack_depot_capture_stack(GFP_NOWAIT | __GFP_NOWARN); for (i = 0; i < (1 << order); i++) { __clear_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); page_owner = get_page_owner(page_ext); @@ -189,7 +143,7 @@ noinline void __set_page_owner(struct page *page, unsigned short order, struct page_ext *page_ext; depot_stack_handle_t handle; - handle = save_stack(gfp_mask); + handle = stack_depot_capture_stack(gfp_mask); page_ext = page_ext_get(page); if (unlikely(!page_ext)) From patchwork Mon May 1 16:54:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227668 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22DA1C77B7C for ; Mon, 1 May 2023 16:56:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A849990000F; Mon, 1 May 2023 12:56:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3565900002; Mon, 1 May 2023 12:56:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FDF890000F; Mon, 1 May 2023 12:56:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 80600900002 for ; Mon, 1 May 2023 12:56:28 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4EF0A1407C0 for ; Mon, 1 May 2023 16:56:28 +0000 (UTC) X-FDA: 80742289656.29.A89B687 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf10.hostedemail.com (Postfix) with ESMTP id 6B6A6C001C for ; Mon, 1 May 2023 16:56:26 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="2XuV/LXF"; spf=pass (imf10.hostedemail.com: domain of 3Oe9PZAYKCIAwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Oe9PZAYKCIAwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960186; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tlYphv1i8nYyRhT88mmnNNep9iAUv9BwOK+gAme7uDQ=; b=w37F0pTtwIJ113IzRzs5w53AP/5NMInOuSNDyJYQ/efK1/lZx49QOmvn1optRpAEB9CmU6 vyMz4CT23HbbPuq7G21W450nR1iGn4V8XNHAWN1KiGIZlhAGpuMWjMAl3kign1GgBp0rIE 2h+7PzjV3I1+rtgGX9M1miDi1YyBn+s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960186; a=rsa-sha256; cv=none; b=ySbW/KCp0e06oST1T/wD+02C3s/OHGh16nOm3nXy5MBq2Q7iV6tPc7VWnn2b51ADPEOb+r tLowx7/7u3gBfqraL7JGX0UEq+Y6CZV7j3Y0A3WeyhVNuVTnx6dhbcgcGXj5wONDZ51T1m pCYDtwMmR+PhkENj4PsPE9XHmwHIqhI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="2XuV/LXF"; spf=pass (imf10.hostedemail.com: domain of 3Oe9PZAYKCIAwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Oe9PZAYKCIAwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-b9a79db4e7fso3326317276.0 for ; Mon, 01 May 2023 09:56:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960185; x=1685552185; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tlYphv1i8nYyRhT88mmnNNep9iAUv9BwOK+gAme7uDQ=; b=2XuV/LXFPS9qv6kO1OAu52yKSEfT1BC8Uyc51PtqvyTLOw6UNebCYkZhZP2v9FO8oS hEHZdD7KUMfSEQtQdWUpzCKOMJYHSpoe3376M6dYUuLCm/nLCzqTViK3tDrzmpk69PsF C9DS0KqR2dDtDGP7hZX5XLhu1WFJacbmtNn9E47IgAq8uRv0BvffVAbtCHATYs2yY0Ed RwD1ZD9bXshFGg1hMmTye+Eo2Jv3yEhTf/4wFmEVqV85hSy0daDAaaNAG8CJUm6nMdJU XzZ/3I4BMbhLiO6ONlK15VVriVh+DhXOoojulz1/HRxZdei9as7clRNqchBWLsgCOWe8 k8ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960185; x=1685552185; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tlYphv1i8nYyRhT88mmnNNep9iAUv9BwOK+gAme7uDQ=; b=fWuWU0de0vMXAfgTTDt/ghpQwzrzqS6jhJNsFtwKBIUct4QFkPZoUayMW9TsMumFJC liRU82jKOoPMbzRYVVQA4k81J3juwPn7Z9TSg+9gscNnacyAz6kh3dUUHwaOfzxujYBO G07yRSCwsMCMuOAyo/ZNTQJOHQH9bjXwZKXZRiidtXW+5gsr4NN2BvLz5QWcIMAeEOyL A2YUR9MbPkx1oyfmNYZwQTGaIlF4mS+/aSKlwJFuBkOetQCg7+Tfp2AZW+BAhxayAdlM tzR+ZgLxkzTUB9UywH0OvAwMpRpJ1fGpHvSXvmy56RKCvTDdZtWhiwXi7y/ql1hxGrYf lz1g== X-Gm-Message-State: AC+VfDxA9cG5ZHF0uyXrri0GMkHiPFwlSVTxIq0kkwcWUROZB8/uwFg+ PymRK/4Cr57RLJ1g9r+vNzgVf22K+/0= X-Google-Smtp-Source: ACHHUZ6xCTIdyFwNXOIyet6JY8PR9mR9OkX4G2UItacuFEKdh2Mzs7ZirEmIfEVsSkOy5x89pd9AYPIc+TM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a05:6902:1003:b0:b8f:54f5:89ff with SMTP id w3-20020a056902100300b00b8f54f589ffmr9086045ybt.11.1682960185390; Mon, 01 May 2023 09:56:25 -0700 (PDT) Date: Mon, 1 May 2023 09:54:44 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-35-surenb@google.com> Subject: [PATCH 34/40] lib: code tagging context capture support From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: 6B6A6C001C X-Rspamd-Server: rspam09 X-Stat-Signature: 9zmjyokfefpa4thzuisfn3dwsuep18r8 X-HE-Tag: 1682960186-133118 X-HE-Meta: U2FsdGVkX18APEJ5mF4pG50Fup+6S8bTebYEmPvnn1C/935qUWzntgfXIHIvbwgv1mIpI0BoWYO0QILg8CO6bnIEEJlOVNIb+ekBoZXLZeLn7xuR8LGY1jfV/BJ5eP18XcHJD9Mwz1HTTNZKhqms3xBw0AsvRc8e27RNFu960/D+kePMiSGXQJPxuroOyZP60qXCEBpLyozFPUVbrrZN/lJLKbgRHA2jp/129XBA1tVT8bvZU2a1CeFLn5ZviLIKwbxZ+4dEibDD6K4ymL/c/P+elwweQTr7v0gRM6AABAkSXZPKAnlqeYElTl4iWhT4hr8J337z24EnUOjfMsN2xLZiDpECecoIUm7UjP3nPu/qDFexG4yl5JxQFgbLW6PUI3CPXdAuy2ctQWavHVK1qsFrCH6z/TJOyUMzpZkKugYAX2RCLx2dRolztS93UoMiKB0+PElg8qi1R3vmLFEu/01zZ1njZaIW+wQ2zpFtvltN1z0nP9P1PWNeTTG2GyhcPjw0bYZiTxHqwiNWUcFTtsZcBnos8eMzLwLFNtAp7oJ90b6UlAo5Yh/XNN6HHt+v5v49apDWcGzmjWoBvcNvO2ui7ZVYPLJY5i9RPZ2tDOVkoKJKgOY6bjzgnoIUhwVdksFmtF6ONs76l3Y8vSiw7iS1lRtHgjnn6uq+rNzLtYKbyfDuLI4F6jE0CErd4uDj+KukkKabkwFFgM26nNQuaiTiduqR4uCJ56IE0PGDcrRqclB9H41dBVDx7bFezB4z3yXre8ztBcJbeSitIn9+HvmKEh0UOBhghuItXVJuYtLG2mmvrN+CjjYpWsxOLYeCrwRnJCNO5v7KFDyx7TgL0ALxdEOxviWA4W2IpSxbIXSA/d99hLmEPMEtady0Fq7rTM2AxSy9wq2OkLs78NEEehf2MIHyK013663dozyw4gk9C1U5XZdI5yV9jpD1i2UudmP/8Jj9gEgV7sBFnML fH6g5Bq9 vvoC4fMKKuG+Or4VWCARXyNiQjaMcU5l9S4B5aquGwiJfDZfShEm97Yi6TiP7CnVVZiYcdzQpbbDFomNlTjHkmAaM0OA681ZRlOKFwLNr1J0oMWUP9JkG+6x6uPFg3DYEFocpj3kgIcXMnz1dKUMIGL+xx4sAJQb8WIxmLp9xskg8gjjbu4zsedzIEHD5KFr3c/a4jfeexKfhhr21FH1g4K263C00owpC1KcWd1T/5zRckdFHgATB3FfdGi6OIN0AVwJiS3sv/5aNEhcmM6Zxp2ykJJWbsCtX5rKWoXHIimN9HdyF6FahXrFjScHaGR6el+ysKkV9J3VYeuzVcHWkt8m0EDf++godOJYJk3xDgB+F3OUJ79k3HC5ImldIy/0VFwRvIY276UXrXQqtgQqytMjMqQXNO4/Il42emmepwngucXlPXBofe4zUyd3AwuND1u3ML5rCma0SCF9a2ZIurwbakfekI4UNiv1SOZgao3/UUKuhh4QPKsw6wjXG5Fg8gECUyTyk3nliozlKBrbq5d3+tR4zmFKLeq/QkuWGs63DNPlMipvz8o0/Ur9mh+e7nbrSa+eSc2FlfJxtNJthaoXLavnP5/Haw3d5rPaIVEt/0HpTxeR3wVBMLAt2LX6U6tq8yOynr/uKsV3VD4GKe+JzywpNzQPvjgAbISWK6fTJQJM7rt7BwaV7Og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add support for code tag context capture when registering a new code tag type. When context capture for a specific code tag is enabled, codetag_ref will point to a codetag_ctx object which can be attached to an application-specific object storing code invocation context. codetag_ctx has a pointer to its codetag_with_ctx object with embedded codetag object in it. All context objects of the same code tag are placed into codetag_with_ctx.ctx_head linked list. codetag.flag is used to indicate when a context capture for the associated code tag is initialized and enabled. Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 50 +++++++++++++- include/linux/codetag_ctx.h | 48 +++++++++++++ lib/codetag.c | 134 ++++++++++++++++++++++++++++++++++++ 3 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 include/linux/codetag_ctx.h diff --git a/include/linux/codetag.h b/include/linux/codetag.h index 87207f199ac9..9ab2f017e845 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -5,8 +5,12 @@ #ifndef _LINUX_CODETAG_H #define _LINUX_CODETAG_H +#include +#include #include +struct kref; +struct codetag_ctx; struct codetag_iterator; struct codetag_type; struct seq_buf; @@ -18,15 +22,38 @@ struct module; * an array of these. */ struct codetag { - unsigned int flags; /* used in later patches */ + unsigned int flags; /* has to be the first member shared with codetag_ctx */ unsigned int lineno; const char *modname; const char *function; const char *filename; } __aligned(8); +/* codetag_with_ctx flags */ +#define CTC_FLAG_CTX_PTR (1 << 0) +#define CTC_FLAG_CTX_READY (1 << 1) +#define CTC_FLAG_CTX_ENABLED (1 << 2) + +/* + * Code tag with context capture support. Contains a list to store context for + * each tag hit, a lock protecting the list and a flag to indicate whether + * context capture is enabled for the tag. + */ +struct codetag_with_ctx { + struct codetag ct; + struct list_head ctx_head; + spinlock_t ctx_lock; +} __aligned(8); + +/* + * Tag reference can point to codetag directly or indirectly via codetag_ctx. + * Direct codetag pointer is used when context capture is disabled or not + * supported. When context capture for the tag is used, the reference points + * to the codetag_ctx through which the codetag can be reached. + */ union codetag_ref { struct codetag *ct; + struct codetag_ctx *ctx; }; struct codetag_range { @@ -46,6 +73,7 @@ struct codetag_type_desc { struct codetag_module *cmod); bool (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); + void (*free_ctx)(struct kref *ref); }; struct codetag_iterator { @@ -53,6 +81,7 @@ struct codetag_iterator { struct codetag_module *cmod; unsigned long mod_id; struct codetag *ct; + struct codetag_ctx *ctx; }; #define CODE_TAG_INIT { \ @@ -63,9 +92,28 @@ struct codetag_iterator { .flags = 0, \ } +static inline bool is_codetag_ctx_ref(union codetag_ref *ref) +{ + return !!(ref->ct->flags & CTC_FLAG_CTX_PTR); +} + +static inline +struct codetag_with_ctx *ct_to_ctc(struct codetag *ct) +{ + return container_of(ct, struct codetag_with_ctx, ct); +} + void codetag_lock_module_list(struct codetag_type *cttype, bool lock); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); +struct codetag_ctx *codetag_next_ctx(struct codetag_iterator *iter); + +bool codetag_enable_ctx(struct codetag_with_ctx *ctc, bool enable); +static inline bool codetag_ctx_enabled(struct codetag_with_ctx *ctc) +{ + return !!(ctc->ct.flags & CTC_FLAG_CTX_ENABLED); +} +bool codetag_has_ctx(struct codetag_with_ctx *ctc); void codetag_to_text(struct seq_buf *out, struct codetag *ct); diff --git a/include/linux/codetag_ctx.h b/include/linux/codetag_ctx.h new file mode 100644 index 000000000000..e741484f0e08 --- /dev/null +++ b/include/linux/codetag_ctx.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * code tag context + */ +#ifndef _LINUX_CODETAG_CTX_H +#define _LINUX_CODETAG_CTX_H + +#include +#include + +/* Code tag hit context. */ +struct codetag_ctx { + unsigned int flags; /* has to be the first member shared with codetag */ + struct codetag_with_ctx *ctc; + struct list_head node; + struct kref refcount; +} __aligned(8); + +static inline struct codetag_ctx *kref_to_ctx(struct kref *refcount) +{ + return container_of(refcount, struct codetag_ctx, refcount); +} + +static inline void add_ctx(struct codetag_ctx *ctx, + struct codetag_with_ctx *ctc) +{ + kref_init(&ctx->refcount); + spin_lock(&ctc->ctx_lock); + ctx->flags = CTC_FLAG_CTX_PTR; + ctx->ctc = ctc; + list_add_tail(&ctx->node, &ctc->ctx_head); + spin_unlock(&ctc->ctx_lock); +} + +static inline void rem_ctx(struct codetag_ctx *ctx, + void (*free_ctx)(struct kref *refcount)) +{ + struct codetag_with_ctx *ctc = ctx->ctc; + + spin_lock(&ctc->ctx_lock); + /* ctx might have been removed while we were using it */ + if (!list_empty(&ctx->node)) + list_del_init(&ctx->node); + spin_unlock(&ctc->ctx_lock); + kref_put(&ctx->refcount, free_ctx); +} + +#endif /* _LINUX_CODETAG_CTX_H */ diff --git a/lib/codetag.c b/lib/codetag.c index 84f90f3b922c..d891bbe4481d 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only #include +#include #include #include #include @@ -92,6 +93,139 @@ struct codetag *codetag_next_ct(struct codetag_iterator *iter) return ct; } +static struct codetag_ctx *next_ctx_from_ct(struct codetag_iterator *iter) +{ + struct codetag_with_ctx *ctc; + struct codetag_ctx *ctx = NULL; + struct codetag *ct = iter->ct; + + while (ct) { + if (!(ct->flags & CTC_FLAG_CTX_READY)) + goto next; + + ctc = ct_to_ctc(ct); + spin_lock(&ctc->ctx_lock); + if (!list_empty(&ctc->ctx_head)) { + ctx = list_first_entry(&ctc->ctx_head, + struct codetag_ctx, node); + kref_get(&ctx->refcount); + } + spin_unlock(&ctc->ctx_lock); + if (ctx) + break; +next: + ct = codetag_next_ct(iter); + } + + iter->ctx = ctx; + return ctx; +} + +struct codetag_ctx *codetag_next_ctx(struct codetag_iterator *iter) +{ + struct codetag_ctx *ctx = iter->ctx; + struct codetag_ctx *found = NULL; + + lockdep_assert_held(&iter->cttype->mod_lock); + + if (!ctx) + return next_ctx_from_ct(iter); + + spin_lock(&ctx->ctc->ctx_lock); + /* + * Do not advance if the object was isolated, restart at the same tag. + */ + if (!list_empty(&ctx->node)) { + if (list_is_last(&ctx->node, &ctx->ctc->ctx_head)) { + /* Finished with this tag, advance to the next */ + codetag_next_ct(iter); + } else { + found = list_next_entry(ctx, node); + kref_get(&found->refcount); + } + } + spin_unlock(&ctx->ctc->ctx_lock); + kref_put(&ctx->refcount, iter->cttype->desc.free_ctx); + + if (!found) + return next_ctx_from_ct(iter); + + iter->ctx = found; + return found; +} + +static struct codetag_type *find_cttype(struct codetag *ct) +{ + struct codetag_module *cmod; + struct codetag_type *cttype; + unsigned long mod_id; + unsigned long tmp; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + down_read(&cttype->mod_lock); + idr_for_each_entry_ul(&cttype->mod_idr, cmod, tmp, mod_id) { + if (ct >= cmod->range.start && ct < cmod->range.stop) { + up_read(&cttype->mod_lock); + goto found; + } + } + up_read(&cttype->mod_lock); + } + cttype = NULL; +found: + mutex_unlock(&codetag_lock); + + return cttype; +} + +bool codetag_enable_ctx(struct codetag_with_ctx *ctc, bool enable) +{ + struct codetag_type *cttype = find_cttype(&ctc->ct); + + if (!cttype || !cttype->desc.free_ctx) + return false; + + lockdep_assert_held(&cttype->mod_lock); + BUG_ON(!rwsem_is_locked(&cttype->mod_lock)); + + if (codetag_ctx_enabled(ctc) == enable) + return false; + + if (enable) { + /* Initialize context capture fields only once */ + if (!(ctc->ct.flags & CTC_FLAG_CTX_READY)) { + spin_lock_init(&ctc->ctx_lock); + INIT_LIST_HEAD(&ctc->ctx_head); + ctc->ct.flags |= CTC_FLAG_CTX_READY; + } + ctc->ct.flags |= CTC_FLAG_CTX_ENABLED; + } else { + /* + * The list of context objects is intentionally left untouched. + * It can be read back and if context capture is re-enablied it + * will append new objects. + */ + ctc->ct.flags &= ~CTC_FLAG_CTX_ENABLED; + } + + return true; +} + +bool codetag_has_ctx(struct codetag_with_ctx *ctc) +{ + bool no_ctx; + + if (!(ctc->ct.flags & CTC_FLAG_CTX_READY)) + return false; + + spin_lock(&ctc->ctx_lock); + no_ctx = list_empty(&ctc->ctx_head); + spin_unlock(&ctc->ctx_lock); + + return !no_ctx; +} + void codetag_to_text(struct seq_buf *out, struct codetag *ct) { seq_buf_printf(out, "%s:%u module:%s func:%s", From patchwork Mon May 1 16:54:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 560EEC77B61 for ; Mon, 1 May 2023 16:56:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7CD8900012; Mon, 1 May 2023 12:56:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E2CF7900002; Mon, 1 May 2023 12:56:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCD4C900012; Mon, 1 May 2023 12:56:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BBC2F900002 for ; Mon, 1 May 2023 12:56:30 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8740980819 for ; Mon, 1 May 2023 16:56:30 +0000 (UTC) X-FDA: 80742289740.01.FC98299 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf04.hostedemail.com (Postfix) with ESMTP id A96FD4000D for ; Mon, 1 May 2023 16:56:28 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=6G5OUnFq; spf=pass (imf04.hostedemail.com: domain of 3O-9PZAYKCIIy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3O-9PZAYKCIIy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960188; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lw2pXEBNfkStSvJmyTF2I1dH+bsBIyZ0OpKs2HHzIO8=; b=sTMR74W8gYfMLzQmrbCeGd/Ai20mcfNwP3ulItOCFy4EsRvl2fANnfNPSm/kQNkRTs5YYN oXtYqfUJntZU15sNqoAlfnDPPZjLCrX/ioQ/UgKN5TOtPyzsC1VihXej/AMhVhxknl7bGk htMYuA+TsDQZJjTjruP7op/xhd0Eftc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=6G5OUnFq; spf=pass (imf04.hostedemail.com: domain of 3O-9PZAYKCIIy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3O-9PZAYKCIIy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960188; a=rsa-sha256; cv=none; b=XeZchxnuuS2Cg46pltTXMQYvFihYjTfSrjmS3SLS+hXmpaVhwCXeL2G94D3sMfGmENPkfu kawOrFjjzCuSVXj1qpPLxFgXiheo8zT/fGLbBPVV0wbUM5Aym6+GS/iSVyjkYw2WqVFTrK TO/vtaoZJ7QPNg1d/tnWgc0tr97mOTE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-559fb5bed89so20701657b3.0 for ; Mon, 01 May 2023 09:56:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960188; x=1685552188; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lw2pXEBNfkStSvJmyTF2I1dH+bsBIyZ0OpKs2HHzIO8=; b=6G5OUnFqmhYzObxhYE56pVPUPb5/Z5fFLyg/xv1k1ELzqtOEByasEUKmpVLdsnIlKz +HpmKICNn1SXskm4W84Z1YnQTS1wv+HkfBClJgQjOTklB/JflZpG6ZVI9CkbEHZZ4VtX +VU0ocTLcHc6zEEkMfHwEXLiatB8YSa4vtZi59KquEiZLQ8PGIiDrQ9uUD2CTQfE2G7r IqwFOlnQ0w6JmNavUX7vfGowmYc1Kgs8nx0nF3FhUsDCRqWZuUIsUlCL/KEMxoHEQqWr IxEVdl0XX/ZNM2Oo2w3beOsDv/rdq05Vn6EfHRE3NqnRnISnGzk4Nz7tXYADK5fB+Pd5 tQVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960188; x=1685552188; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lw2pXEBNfkStSvJmyTF2I1dH+bsBIyZ0OpKs2HHzIO8=; b=jgTWFY0OvD3Gth8vLOss4E9Lz9RSFdlXexy0YM5R3rR84TlkAKbBH3ZOfnsB2eiyRZ /G7dRX6K+4UmC81g2Qwb50ttcBxfSK9b1wvtXAHozwyGEskgZIr+B7qtw8+swo63nKYx 0Nx6Ti927YzV7sv/FgoH8h/IFwSw7NsNh+DfgbsJ/RQVXtuPSZ8jsqe8klNICdW7x85W 6XOxffrgxnDKpQOWcDqAvp/ngPuaaIIhuYNpFGqbFkcCdGyjxHlmWezEz+FQtxHaeKNB qg++bjnG7uG7k1vgwn2Nfah1+kiKxw3Hf5tA1gW7mcfoQr075IG356//GETm+z/mvyTM hRng== X-Gm-Message-State: AC+VfDw/T8oO5WWI91ZBJMoYQCgsMWTBfHop3sisXwvCpoVVPhA8e0uL fYvuXOnctyUHJ+1P8eU8JN8f4Ss4/DM= X-Google-Smtp-Source: ACHHUZ4jn05QWo1AuUjIBZxdqElOhbGVaSLvkqDYZu4aVA/YTY7kkMHxf30EuctFEiweZjTNT8dvorYRrv0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:4304:0:b0:55a:c44:6151 with SMTP id q4-20020a814304000000b0055a0c446151mr3660331ywa.5.1682960187794; Mon, 01 May 2023 09:56:27 -0700 (PDT) Date: Mon, 1 May 2023 09:54:45 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-36-surenb@google.com> Subject: [PATCH 35/40] lib: implement context capture support for tagged allocations From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: sqe5pxwkc736ysqhpi4xt1q8d7upomgj X-Rspamd-Queue-Id: A96FD4000D X-HE-Tag: 1682960188-677493 X-HE-Meta: U2FsdGVkX18EQ5Pf1BEbeOeiGIyggiFFoTuxzXRfoX1usc5jqWN6qpWvFM8YINzK4ojj4VR9vREXDLN5fO9/Cb4Qg5IVUTVKdsKf42OU21hmyeYeXGLbYKMfiZRFF+hRnUtIBjwv6ja+m3ibu7Sk7sgHVa26xc9gzEvo2V86lkxlN5TEXyg+KkqXzKY7AH2Vgu7pgD9OnnCso9xmAYGDqZ7lYjUz6koVzBN4sPqKu29A2oxXDZJierpc9hOUEiy3HxY3174+S0LMC9ykZgZxvc2B71W4yymIe1DQn4jwYpVKIW6k4U2OcP13IOkEFruIIUBGObDENvVmU6WpCgcnZXpxsA/l2oDAbw/5fFUGeFTc5aVoFpL8ovY5VJLqO/SAOVNDf+MJ2GvhzeDnWkcy0LkXuk60saRwkuKn9bT4/fr9W8sj3Ud+vja4HqiJVSYQ/wOzCrzib6/XQex14pGdz3tiBmlalhAV6YsY3rp2ZkWXPsz40DtmSnREp7zHVLt8LC5mW/cFlbh0ltybHBm8r2VkB79Ny8q0rwdLVwEBUVuRZATNv8iG2C7Eg4QkyNkKw6WQ9A1vMiVRM5vd9Fko5TUgYrR/BrTWCdKgVFzNptS7Oy8r2DNr1bvXTxF+S+L8u/2VdAHUMra6huJUNLKYZhBJ29k3x9U/apwHueUXA+qn7pOm0aPqK9Pxzlas3hmdSbjyW5mifCk9EzNxLwNMazMAC27cCdb0Mo06XFABssJrWM97tDU94q1pFlgMtx6+t0AXGah/bgkk3QLAXGHr/Ww5BYpXyBZBhjL5/48xyJh3/WJI7XNhLondpocX7mDwH/LcUhcCLVR3FIX1X5VfcbzWmLafo+hIMTGI7y0Mh7isw7B0bN5U0f3OKaD9+m1i9rheI6MOG6+k8PQnpARHJrJDp+QLGutM3bLmhNvbPffwLMJvcSXveMIpuK2EZFV7ZL/qZQxEtoHIfBUzfAi 6y8oWfZY 2vji92t4MBYuZIoTTHWHCbRHrsIaa1snDzyVjKWvJl8f2AR/gnFpxI68kFF6b7oXN0nmg84Wt7TJz/1j+mqr6rcDLreeiFRL8qV6OnsYWdL9AOv014vJMhthHGyGzMshnPbsEFM5o13Jmx86Ak4c9uPbitxDTevcGB7YXM7x7yY3lCaikY+SYjK/B2v9bQOC+9kRx8dlFt95lDGeNOp+HTXa1p0wMkzzkjnAX9gz7yy7RSOHRpGXAyyVuSlshQvT9KpE/S3YdKDGsNZE5pS5uGRB99NF1obZ6aShzsqPMGSXvoPex6JThYenBhDTyuBRffWGgbG1r+uO4Bd+ja6PN1KrAZ0XT82bucTKcom/qTk5jlcXH6enPfa/qTXbKyYNZeNM/Dj9J1K2VtnKxeJ752NITmeYxhz5Sw2ivuFfK4m9qONkBqCA+r+n9dnhDKZaDO0UlSuUktR/b+SOWpSBxaQP+2eiCLx8hO4TgufUuhbIq51VG4RmL76iV3UIvF8JKukXtpVI+wqQfTE5/9sFB49MNqtmJ5CFEPSc/zzKMjcwiiveBQtECfpxlsYLzT7tkWS7NKbbHRzh8K1Ibx1CDd4Qfgc43YG5CiJ2TuBH3ujgR4Km//QzVScsgdOeJ4Y6p1wr5GoT/QZKnbCH6at2SSXGd7a8Tnb2rkbJauRqh1kEjK7A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement mechanisms for capturing allocation call context which consists of: - allocation size - pid, tgid and name of the allocating task - allocation timestamp - allocation call stack The patch creates allocations.ctx file which can be written to enable/disable context capture for a specific code tag. Captured context can be obtained by reading allocations.ctx file. Usage example: echo "file include/asm-generic/pgalloc.h line 63 enable" > \ /sys/kernel/debug/allocations.ctx cat allocations.ctx 91.0MiB 212 include/asm-generic/pgalloc.h:63 module:pgtable func:__pte_alloc_one size: 4096 pid: 1551 tgid: 1551 comm: cat ts: 670109646361 call stack: pte_alloc_one+0xfe/0x130 __pte_alloc+0x22/0x90 move_page_tables.part.0+0x994/0xa60 shift_arg_pages+0xa4/0x180 setup_arg_pages+0x286/0x2d0 load_elf_binary+0x4e1/0x18d0 bprm_execve+0x26b/0x660 do_execveat_common.isra.0+0x19d/0x220 __x64_sys_execve+0x2e/0x40 do_syscall_64+0x38/0x90 entry_SYSCALL_64_after_hwframe+0x63/0xcd size: 4096 pid: 1551 tgid: 1551 comm: cat ts: 670109711801 call stack: pte_alloc_one+0xfe/0x130 __do_fault+0x52/0xc0 __handle_mm_fault+0x7d9/0xdd0 handle_mm_fault+0xc0/0x2b0 do_user_addr_fault+0x1c3/0x660 exc_page_fault+0x62/0x150 asm_exc_page_fault+0x22/0x30 ... echo "file include/asm-generic/pgalloc.h line 63 disable" > \ /sys/kernel/debug/alloc_tags.ctx Note that disabling context capture will not clear already captured context but no new context will be captured. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 25 +++- include/linux/codetag.h | 3 +- include/linux/pgalloc_tag.h | 4 +- lib/Kconfig.debug | 1 + lib/alloc_tag.c | 238 +++++++++++++++++++++++++++++++++++- lib/codetag.c | 20 +-- 6 files changed, 272 insertions(+), 19 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 07922d81b641..2a3d248aae10 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -17,20 +17,29 @@ * an array of these. Embedded codetag utilizes codetag framework. */ struct alloc_tag { - struct codetag ct; + struct codetag_with_ctx ctc; struct lazy_percpu_counter bytes_allocated; } __aligned(8); #ifdef CONFIG_MEM_ALLOC_PROFILING +static inline struct alloc_tag *ctc_to_alloc_tag(struct codetag_with_ctx *ctc) +{ + return container_of(ctc, struct alloc_tag, ctc); +} + static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) { - return container_of(ct, struct alloc_tag, ct); + return container_of(ct_to_ctc(ct), struct alloc_tag, ctc); } +struct codetag_ctx *alloc_tag_create_ctx(struct alloc_tag *tag, size_t size); +void alloc_tag_free_ctx(struct codetag_ctx *ctx, struct alloc_tag **ptag); +bool alloc_tag_enable_ctx(struct alloc_tag *tag, bool enable); + #define DEFINE_ALLOC_TAG(_alloc_tag, _old) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") = { .ct = CODE_TAG_INIT }; \ + __section("alloc_tags") = { .ctc.ct = CODE_TAG_INIT }; \ struct alloc_tag * __maybe_unused _old = alloc_tag_save(&_alloc_tag) extern struct static_key_true mem_alloc_profiling_key; @@ -54,7 +63,10 @@ static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes, if (!ref || !ref->ct) return; - tag = ct_to_alloc_tag(ref->ct); + if (is_codetag_ctx_ref(ref)) + alloc_tag_free_ctx(ref->ctx, &tag); + else + tag = ct_to_alloc_tag(ref->ct); if (may_allocate) lazy_percpu_counter_add(&tag->bytes_allocated, -bytes); @@ -88,7 +100,10 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, if (!ref || !tag) return; - ref->ct = &tag->ct; + if (codetag_ctx_enabled(&tag->ctc)) + ref->ctx = alloc_tag_create_ctx(tag, bytes); + else + ref->ct = &tag->ctc.ct; lazy_percpu_counter_add(&tag->bytes_allocated, bytes); } diff --git a/include/linux/codetag.h b/include/linux/codetag.h index 9ab2f017e845..b6a2f0287a83 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -104,7 +104,8 @@ struct codetag_with_ctx *ct_to_ctc(struct codetag *ct) } void codetag_lock_module_list(struct codetag_type *cttype, bool lock); -struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); +void codetag_init_iter(struct codetag_iterator *iter, + struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); struct codetag_ctx *codetag_next_ctx(struct codetag_iterator *iter); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 0cbba13869b5..e4661bbd40c6 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -6,6 +6,7 @@ #define _LINUX_PGALLOC_TAG_H #include +#include #ifdef CONFIG_MEM_ALLOC_PROFILING @@ -70,7 +71,8 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) if (!ref->ct) goto out; - tag = ct_to_alloc_tag(ref->ct); + tag = is_codetag_ctx_ref(ref) ? ctc_to_alloc_tag(ref->ctx->ctc) + : ct_to_alloc_tag(ref->ct); page_ext = page_ext_next(page_ext); for (i = 1; i < nr; i++) { /* New reference with 0 bytes accounted */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 4157c2251b07..1b83ef17d232 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -969,6 +969,7 @@ config MEM_ALLOC_PROFILING select LAZY_PERCPU_COUNTER select PAGE_EXTENSION select SLAB_OBJ_EXT + select STACKDEPOT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 4a0b95a46b2e..675c7a08e38b 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -1,13 +1,18 @@ // SPDX-License-Identifier: GPL-2.0-only #include +#include #include #include #include #include #include +#include #include +#include #include +#define STACK_BUF_SIZE 1024 + DEFINE_STATIC_KEY_TRUE(mem_alloc_profiling_key); /* @@ -23,6 +28,16 @@ static int __init mem_alloc_profiling_disable(char *s) } __setup("nomem_profiling", mem_alloc_profiling_disable); +struct alloc_call_ctx { + struct codetag_ctx ctx; + size_t size; + pid_t pid; + pid_t tgid; + char comm[TASK_COMM_LEN]; + u64 ts_nsec; + depot_stack_handle_t stack_handle; +} __aligned(8); + struct alloc_tag_file_iterator { struct codetag_iterator ct_iter; struct seq_buf buf; @@ -64,7 +79,7 @@ static int allocations_file_open(struct inode *inode, struct file *file) return -ENOMEM; codetag_lock_module_list(cttype, true); - iter->ct_iter = codetag_get_ct_iter(cttype); + codetag_init_iter(&iter->ct_iter, cttype); codetag_lock_module_list(cttype, false); seq_buf_init(&iter->buf, iter->rawbuf, sizeof(iter->rawbuf)); file->private_data = iter; @@ -125,24 +140,240 @@ static const struct file_operations allocations_file_ops = { .read = allocations_file_read, }; +static void alloc_tag_ops_free_ctx(struct kref *refcount) +{ + kfree(container_of(kref_to_ctx(refcount), struct alloc_call_ctx, ctx)); +} + +struct codetag_ctx *alloc_tag_create_ctx(struct alloc_tag *tag, size_t size) +{ + struct alloc_call_ctx *ac_ctx; + + /* TODO: use a dedicated kmem_cache */ + ac_ctx = kmalloc(sizeof(struct alloc_call_ctx), GFP_KERNEL); + if (WARN_ON(!ac_ctx)) + return NULL; + + ac_ctx->size = size; + ac_ctx->pid = current->pid; + ac_ctx->tgid = current->tgid; + strscpy(ac_ctx->comm, current->comm, sizeof(ac_ctx->comm)); + ac_ctx->ts_nsec = local_clock(); + ac_ctx->stack_handle = + stack_depot_capture_stack(GFP_NOWAIT | __GFP_NOWARN); + add_ctx(&ac_ctx->ctx, &tag->ctc); + + return &ac_ctx->ctx; +} +EXPORT_SYMBOL_GPL(alloc_tag_create_ctx); + +void alloc_tag_free_ctx(struct codetag_ctx *ctx, struct alloc_tag **ptag) +{ + *ptag = ctc_to_alloc_tag(ctx->ctc); + rem_ctx(ctx, alloc_tag_ops_free_ctx); +} +EXPORT_SYMBOL_GPL(alloc_tag_free_ctx); + +bool alloc_tag_enable_ctx(struct alloc_tag *tag, bool enable) +{ + static bool stack_depot_ready; + + if (enable && !stack_depot_ready) { + stack_depot_init(); + stack_depot_capture_init(); + stack_depot_ready = true; + } + + return codetag_enable_ctx(&tag->ctc, enable); +} + +static void alloc_tag_ctx_to_text(struct seq_buf *out, struct codetag_ctx *ctx) +{ + struct alloc_call_ctx *ac_ctx; + char *buf; + + ac_ctx = container_of(ctx, struct alloc_call_ctx, ctx); + seq_buf_printf(out, " size: %zu\n", ac_ctx->size); + seq_buf_printf(out, " pid: %d\n", ac_ctx->pid); + seq_buf_printf(out, " tgid: %d\n", ac_ctx->tgid); + seq_buf_printf(out, " comm: %s\n", ac_ctx->comm); + seq_buf_printf(out, " ts: %llu\n", ac_ctx->ts_nsec); + + buf = kmalloc(STACK_BUF_SIZE, GFP_KERNEL); + if (buf) { + int bytes_read = stack_depot_snprint(ac_ctx->stack_handle, buf, + STACK_BUF_SIZE - 1, 8); + buf[bytes_read] = '\0'; + seq_buf_printf(out, " call stack:\n%s\n", buf); + } + kfree(buf); +} + +static ssize_t allocations_ctx_file_read(struct file *file, char __user *ubuf, + size_t size, loff_t *ppos) +{ + struct alloc_tag_file_iterator *iter = file->private_data; + struct codetag_iterator *ct_iter = &iter->ct_iter; + struct user_buf buf = { .buf = ubuf, .size = size }; + struct codetag_ctx *ctx; + struct codetag *prev_ct; + int err = 0; + + codetag_lock_module_list(ct_iter->cttype, true); + while (1) { + err = flush_ubuf(&buf, &iter->buf); + if (err || !buf.size) + break; + + prev_ct = ct_iter->ct; + ctx = codetag_next_ctx(ct_iter); + if (!ctx) + break; + + if (prev_ct != &ctx->ctc->ct) + alloc_tag_to_text(&iter->buf, &ctx->ctc->ct); + alloc_tag_ctx_to_text(&iter->buf, ctx); + } + codetag_lock_module_list(ct_iter->cttype, false); + + return err ? : buf.ret; +} + +#define CTX_CAPTURE_TOKENS() \ + x(disable, 0) \ + x(enable, 0) + +static const char * const ctx_capture_token_strs[] = { +#define x(name, nr_args) #name, + CTX_CAPTURE_TOKENS() +#undef x + NULL +}; + +enum ctx_capture_token { +#define x(name, nr_args) TOK_##name, + CTX_CAPTURE_TOKENS() +#undef x +}; + +static int enable_ctx_capture(struct codetag_type *cttype, + struct codetag_query *query, bool enable) +{ + struct codetag_iterator ct_iter; + struct codetag_with_ctx *ctc; + struct codetag *ct; + unsigned int nfound = 0; + + codetag_lock_module_list(cttype, true); + + codetag_init_iter(&ct_iter, cttype); + while ((ct = codetag_next_ct(&ct_iter))) { + if (!codetag_matches_query(query, ct, ct_iter.cmod, NULL)) + continue; + + ctc = ct_to_ctc(ct); + if (codetag_ctx_enabled(ctc) == enable) + continue; + + if (!alloc_tag_enable_ctx(ctc_to_alloc_tag(ctc), enable)) { + pr_warn("Failed to toggle context capture\n"); + continue; + } + + nfound++; + } + + codetag_lock_module_list(cttype, false); + + return nfound ? 0 : -ENOENT; +} + +static int parse_command(struct codetag_type *cttype, char *buf) +{ + struct codetag_query query = { NULL }; + char *cmd; + int ret; + int tok; + + buf = codetag_query_parse(&query, buf); + if (IS_ERR(buf)) + return PTR_ERR(buf); + + cmd = strsep_no_empty(&buf, " \t\r\n"); + if (!cmd) + return -EINVAL; /* no command */ + + tok = match_string(ctx_capture_token_strs, + ARRAY_SIZE(ctx_capture_token_strs), cmd); + if (tok < 0) + return -EINVAL; /* unknown command */ + + ret = enable_ctx_capture(cttype, &query, tok == TOK_enable); + if (ret < 0) + return ret; + + return 0; +} + +static ssize_t allocations_ctx_file_write(struct file *file, const char __user *ubuf, + size_t len, loff_t *offp) +{ + struct alloc_tag_file_iterator *iter = file->private_data; + char tmpbuf[256]; + + if (len == 0) + return 0; + /* we don't check *offp -- multiple writes() are allowed */ + if (len > sizeof(tmpbuf) - 1) + return -E2BIG; + + if (copy_from_user(tmpbuf, ubuf, len)) + return -EFAULT; + + tmpbuf[len] = '\0'; + parse_command(iter->ct_iter.cttype, tmpbuf); + + *offp += len; + return len; +} + +static const struct file_operations allocations_ctx_file_ops = { + .owner = THIS_MODULE, + .open = allocations_file_open, + .release = allocations_file_release, + .read = allocations_ctx_file_read, + .write = allocations_ctx_file_write, +}; + static int __init dbgfs_init(struct codetag_type *cttype) { struct dentry *file; + struct dentry *ctx_file; file = debugfs_create_file("allocations", 0444, NULL, cttype, &allocations_file_ops); + if (IS_ERR(file)) + return PTR_ERR(file); + + ctx_file = debugfs_create_file("allocations.ctx", 0666, NULL, cttype, + &allocations_ctx_file_ops); + if (IS_ERR(ctx_file)) { + debugfs_remove(file); + return PTR_ERR(ctx_file); + } - return IS_ERR(file) ? PTR_ERR(file) : 0; + return 0; } static bool alloc_tag_module_unload(struct codetag_type *cttype, struct codetag_module *cmod) { - struct codetag_iterator iter = codetag_get_ct_iter(cttype); + struct codetag_iterator iter; bool module_unused = true; struct alloc_tag *tag; struct codetag *ct; size_t bytes; + codetag_init_iter(&iter, cttype); for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { if (iter.cmod != cmod) continue; @@ -183,6 +414,7 @@ static int __init alloc_tag_init(void) .section = "alloc_tags", .tag_size = sizeof(struct alloc_tag), .module_unload = alloc_tag_module_unload, + .free_ctx = alloc_tag_ops_free_ctx, }; cttype = codetag_register_type(&desc); diff --git a/lib/codetag.c b/lib/codetag.c index d891bbe4481d..cbff146b3fe8 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -27,16 +27,14 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock) up_read(&cttype->mod_lock); } -struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) +void codetag_init_iter(struct codetag_iterator *iter, + struct codetag_type *cttype) { - struct codetag_iterator iter = { - .cttype = cttype, - .cmod = NULL, - .mod_id = 0, - .ct = NULL, - }; - - return iter; + iter->cttype = cttype; + iter->cmod = NULL; + iter->mod_id = 0; + iter->ct = NULL; + iter->ctx = NULL; } static inline struct codetag *get_first_module_ct(struct codetag_module *cmod) @@ -128,6 +126,10 @@ struct codetag_ctx *codetag_next_ctx(struct codetag_iterator *iter) lockdep_assert_held(&iter->cttype->mod_lock); + /* Move to the first codetag if search just started */ + if (!iter->ct) + codetag_next_ct(iter); + if (!ctx) return next_ctx_from_ct(iter); From patchwork Mon May 1 16:54:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1ECCC7EE25 for ; Mon, 1 May 2023 16:56:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 507A2900020; Mon, 1 May 2023 12:56:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DEC4900002; Mon, 1 May 2023 12:56:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 37EB0900020; Mon, 1 May 2023 12:56:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 28DE2900002 for ; Mon, 1 May 2023 12:56:33 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D0E691407E3 for ; Mon, 1 May 2023 16:56:32 +0000 (UTC) X-FDA: 80742289824.14.C20C5E9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf23.hostedemail.com (Postfix) with ESMTP id 17905140018 for ; Mon, 1 May 2023 16:56:30 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=y9PADbKN; spf=pass (imf23.hostedemail.com: domain of 3Pu9PZAYKCIU130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Pu9PZAYKCIU130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5F2YZ+zSEbu/k1s4F5etygangPIkg0EznnOkCCfoew4=; b=g6+baUcmt2bFVrL63U2FPY5toZ2ajLx7NdZZ37EQnjeo07nI1DuMnIGYvkWNI++OWmwCIW vY1VJGKTLQg83LZJe1X8uzs/FxMtF+80z+h8aa8yiqDgFGGTE0CyLlMdUtIsV6m6sewACr hhJ4W4ItHJE1JjoIJLzQfBcMknM/7E0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=y9PADbKN; spf=pass (imf23.hostedemail.com: domain of 3Pu9PZAYKCIU130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Pu9PZAYKCIU130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960191; a=rsa-sha256; cv=none; b=CfaWITUt0C2WxaJM5qhcz05E+RWgsP2OchIXDEpVF84Fpve06qZ4FIHpDqaLl4+dxSplkZ +lxrXCNm8rgXAYgZC+vYQL8KD/JVQ86mqroDUwZK9RhluHhuvu/X6bF/Buj62TkzGirmgM 7CIycdG6yNfNWgjusHS5szsIZ3pB0MM= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-55a37bf6fd0so26798707b3.2 for ; Mon, 01 May 2023 09:56:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960190; x=1685552190; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5F2YZ+zSEbu/k1s4F5etygangPIkg0EznnOkCCfoew4=; b=y9PADbKNRkycgFljwwW4rqucG3eA68YCKeeZo3WcsU1FmoEQqcIFYkjuX0vuBDZ+Ql NTwaESSbp1RZc7LHc0KY8IZ9ABdUTaz7iD+fqU6hTkv+OjleRGkRYXEn3mmMGDZeb37K RR5c6bta6TNyvXnvanAVd7N5usS73yNUZBprdVddhAocVImGlGKImrhAPkg7FdlsDO1g 0/gV7SzEHyQ+UDygmg90PVZ1tDJNusamWgID4ov2qqJT8Mlk7B+3jkcThGeG+RtvCTCf JdDB5SJtODP6opHYk8cvNwCb/U4pdSF9HDKWf6Vbz1ntZy0fwly+Flt/OMhilMEsPt5F Cb4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960190; x=1685552190; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5F2YZ+zSEbu/k1s4F5etygangPIkg0EznnOkCCfoew4=; b=aWoGgxI6U6tIvtqsHTcheD8+VzXFetsaQtGfS9a3RqWxLwnrBI9sfoFNp5lLDGWXbM Kq1UkLPFUx3rCRzyFNrGL1I/YBiS10fXfGfFSfv1t5s363d5M1uzNdlLExfTqr9UNbHV QKa8QsnK/ifAdloajThNISiig2TzHq8J+p7Fz4Yy1VEYj6Gn3WAuPuDcYbvS73dqkBkN VzJpL4ZYHSwPpUP/MzBz5wGYU7sIA0J3yY0I1NRW/j+tGbbxF7NJmEXAjbtkg1MNZwGy ecAHpzOvYT4slBUM8t1VYst1Jy3MZnbOIIBxSaeV56oWbqag30sKyxjn/m2MelheJfaV p+sw== X-Gm-Message-State: AC+VfDwaJRpV5aJrntntwWsKyt5uoPpeCbXuKKGU/2dzO6vIvmEJ09h8 dBCXCeIUebTRASBE3ohzn41Da5cT6zM= X-Google-Smtp-Source: ACHHUZ4bAJGviPj6gi0jXXmQNcf1vAdyBJ0X5d7ddmrZp75YBKgThQIRi2xp+pDFZfbHaui7RkEBtZ2XytA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:a8c4:0:b0:54d:3afc:d503 with SMTP id f187-20020a81a8c4000000b0054d3afcd503mr8631819ywh.8.1682960190091; Mon, 01 May 2023 09:56:30 -0700 (PDT) Date: Mon, 1 May 2023 09:54:46 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-37-surenb@google.com> Subject: [PATCH 36/40] lib: add memory allocations report in show_mem() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 17905140018 X-Stat-Signature: itorgy9tnmjgqy9mnqs7m7ddua4hz9qj X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682960190-191103 X-HE-Meta: U2FsdGVkX1/frRWDbVOFXNwSbsSvqOqN2lpInlW0R+T9mZ3OaRWGDhed5YGA1VaDq9MBSsATZvqDxegUm7DaEUxZYKB5+mQ8YIifAUPXyWENsmIaojhdxF9SXv+MSd1nq1IQs+MzMCxSPRdgUEv60QZnyGWToNuTiuT66lsiky3OhlumrVrPrTiGjlaatGB/uHazBPiYwaEpZhKDNJVPomnbCoFnDie2LByQi+OPxwAieaRKNUh3SRXb/vFkL5kGd9BWK+nfVI20ss1569oJHHKQeQl86UuB1ZMF8we75+eflfK9/wXKldaQPm6t6+8/ohKHZxoRtcNMcyCJfDYjDWM2BQVn9BCBOi7nM3pZZ2+Rw7EqPutXq6vW/gDC0qgqsR4xf8OZ/D0gAQBFH25u8kkMx7e2TdyjTrLsWglnreVZJPnOtwwGkknfB5mo8FZ7BvJlYGWtiBQVgaW2uWioW49JMynqDsxxjfb/A7KxNEnE0xcmHdzVf/yiDCQOi2EgrUZshNv2zMfHcCZg0Sj5n+ZLRiGA8Z94w0t/A1B2DPh9Kr7zzpAwVd4ljjriaZ4h+BOAvdWkGoZODUNox7IzOBoPgkET8yF/ScCF8XaqBQJMKomWy0DD8dhoIlYyvszTHPZuuBOt+PjDW8MdTtTzrk/X4NH1uG6q+0Jb9geCSciEiXMeNlaDUYGdvVNucSl0SbrQhmhT5ZmPS5R9ThEAQ2Ro9ITqy3h7/6CjRwCZm/QXbnKhQkvUwT4BnJYnqxzZ6R7tjNWxK5t6Ur6oO7rnsAuziYFJMxE0/RUaLUajHq5hrajhLntyTCTJXhIZRKHoUkvYc+OuWAQfaucaJnG0R+uwfxw2N3frqtJM+qSPvB3v4pjXb1pf2AcV8ncUxWvOCKpbubHw+0LgqElQPkOXi7qm/97TRizhpz6iDoT32v7RlVJVFUF61o72a+/5oSZtqaq7geDWWid78u/U4t+ axbdyEgT kLw3cE8uOZ0Lls75yd0Ri/dybHdTM/nRj291hzSdSvRtnBrGSEg9rvQ90JEb+laMlNSGaDJ3Zc8aH0rrrg2BLvX/+3wRMTFOW+3HAJX/HgmOznddaQdQre2BF+x7zoZB12JCtAUBNMM0sTDJjPj3nDv/oCeK33WSNQpxEVaCUcTwKQfo9mT280d9wU1ON5+dQAiofkiKorkSqRUGjwHhuiVjwv4Oah9THn9C+KTJrTlpFsZZtNdf+LPdAbug41xH5WPkE8KHBEMqA9JhSL+p02tTF312+1ExVPcoNT++MnBB+AjEjjhrKwxpX3ktsO8PUQu1Vnwc3ZkwBbFu99J6HIbTWoM8Lnjn4YTwwISZB2geOmBQMoiFHiu/xCLq+CEo5+ySSIgaTtZF6as8powPPG9G0vKfjENNkXqs1hYYCgtSKNC1ZkMsSmIBfSBE8IDNeII7+npPXi1eH73SeX1cBGF9ueTKOZLlZbm+792DGmxQB6q1eIpOHluCdhJapfJATYgh8ClSFsGkCkXbDB51coEXMDh+YgGlOYc6F5ZzMW2IS0XeQfbSG6Zfqio72U6f61PkxJpcr5X9JUhbXFAPGITap/dS3YH6BGJZJHESKs9yZlEy5t5AZO7rh0pRfAS1HIa4odWaHuHgLWd8z2ENrOtmL3B8A9J/gYShkj92iLbpN4tKoj6lorPWEVJ59FmJYsyQ/OHPCT2P/Gh6+F+4LzMaE+SaxBlMTakkY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Include allocations in show_mem reports. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 2 ++ lib/alloc_tag.c | 48 +++++++++++++++++++++++++++++++++++---- lib/show_mem.c | 15 ++++++++++++ 3 files changed, 60 insertions(+), 5 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 2a3d248aae10..190ab793f7e5 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -23,6 +23,8 @@ struct alloc_tag { #ifdef CONFIG_MEM_ALLOC_PROFILING +void alloc_tags_show_mem_report(struct seq_buf *s); + static inline struct alloc_tag *ctc_to_alloc_tag(struct codetag_with_ctx *ctc) { return container_of(ctc, struct alloc_tag, ctc); diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 675c7a08e38b..e2ebab8999a9 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -13,6 +13,8 @@ #define STACK_BUF_SIZE 1024 +static struct codetag_type *alloc_tag_cttype; + DEFINE_STATIC_KEY_TRUE(mem_alloc_profiling_key); /* @@ -133,6 +135,43 @@ static ssize_t allocations_file_read(struct file *file, char __user *ubuf, return err ? : buf.ret; } +void alloc_tags_show_mem_report(struct seq_buf *s) +{ + struct codetag_iterator iter; + struct codetag *ct; + struct { + struct codetag *tag; + size_t bytes; + } tags[10], n; + unsigned int i, nr = 0; + + codetag_init_iter(&iter, alloc_tag_cttype); + + codetag_lock_module_list(alloc_tag_cttype, true); + while ((ct = codetag_next_ct(&iter))) { + n.tag = ct; + n.bytes = lazy_percpu_counter_read(&ct_to_alloc_tag(ct)->bytes_allocated); + + for (i = 0; i < nr; i++) + if (n.bytes > tags[i].bytes) + break; + + if (i < ARRAY_SIZE(tags)) { + nr -= nr == ARRAY_SIZE(tags); + memmove(&tags[i + 1], + &tags[i], + sizeof(tags[0]) * (nr - i)); + nr++; + tags[i] = n; + } + } + + for (i = 0; i < nr; i++) + alloc_tag_to_text(s, tags[i].tag); + + codetag_lock_module_list(alloc_tag_cttype, false); +} + static const struct file_operations allocations_file_ops = { .owner = THIS_MODULE, .open = allocations_file_open, @@ -409,7 +448,6 @@ EXPORT_SYMBOL(page_alloc_tagging_ops); static int __init alloc_tag_init(void) { - struct codetag_type *cttype; const struct codetag_type_desc desc = { .section = "alloc_tags", .tag_size = sizeof(struct alloc_tag), @@ -417,10 +455,10 @@ static int __init alloc_tag_init(void) .free_ctx = alloc_tag_ops_free_ctx, }; - cttype = codetag_register_type(&desc); - if (IS_ERR_OR_NULL(cttype)) - return PTR_ERR(cttype); + alloc_tag_cttype = codetag_register_type(&desc); + if (IS_ERR_OR_NULL(alloc_tag_cttype)) + return PTR_ERR(alloc_tag_cttype); - return dbgfs_init(cttype); + return dbgfs_init(alloc_tag_cttype); } module_init(alloc_tag_init); diff --git a/lib/show_mem.c b/lib/show_mem.c index 1485c87be935..5c82f29168e3 100644 --- a/lib/show_mem.c +++ b/lib/show_mem.c @@ -7,6 +7,7 @@ #include #include +#include void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) { @@ -34,4 +35,18 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) #ifdef CONFIG_MEMORY_FAILURE printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { + struct seq_buf s; + char *buf = kmalloc(4096, GFP_ATOMIC); + + if (buf) { + printk("Memory allocations:\n"); + seq_buf_init(&s, buf, 4096); + alloc_tags_show_mem_report(&s); + printk("%s", buf); + kfree(buf); + } + } +#endif } From patchwork Mon May 1 16:54:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E1B3C77B73 for ; Mon, 1 May 2023 16:56:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DC1B900021; Mon, 1 May 2023 12:56:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 68D09900002; Mon, 1 May 2023 12:56:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 505AA900021; Mon, 1 May 2023 12:56:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 429C8900002 for ; Mon, 1 May 2023 12:56:35 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 27981407F2 for ; Mon, 1 May 2023 16:56:35 +0000 (UTC) X-FDA: 80742289950.18.C956837 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf03.hostedemail.com (Postfix) with ESMTP id 5E6CE2001A for ; Mon, 1 May 2023 16:56:33 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lLa+vy9Y; spf=pass (imf03.hostedemail.com: domain of 3QO9PZAYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3QO9PZAYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960193; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nH88FGoAACjgZmQtkl43JvHLPN3RafkQwEsrc5bfN4s=; b=t6LyTTlY0nAqA1yJUjVGtvgCaN9lI0NTAmz6nDvNoyjzOW75HqN+O6P4/K5SnyrciBIwhO /husoIAklj0Q+ZF3qwOxyfUaa6G2vLxwkCKPYWLxbZqZGbSyTmBh37D0zDA0Zy9DiSDqyo 6S8GmpAQUOccBAOgYTnOltbWitMBjKk= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lLa+vy9Y; spf=pass (imf03.hostedemail.com: domain of 3QO9PZAYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3QO9PZAYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960193; a=rsa-sha256; cv=none; b=BQSPqPy73ngHOVR9J672bsFhPaS8LaEcBiwAsGTSFGIbH4lI9iR6O3SimHNOLqsZ0FuQdQ +/jMJ3JTFdIIPR3KjmEAFXO7SGpWp5oFfvqHmKdhkS1ohBsOeqt2LF9utJKQwOMIxtyoYu MypxJPSD0YKwXGpRHjMsxUQdW0KUOrg= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-559f179cd38so41362047b3.0 for ; Mon, 01 May 2023 09:56:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960192; x=1685552192; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nH88FGoAACjgZmQtkl43JvHLPN3RafkQwEsrc5bfN4s=; b=lLa+vy9YS9JYxnsS/wzOxOEwwt8wSy5Uuy+3HdA356nRicp9ut8/ucfpMAH5tzjtpk TZcWKstbLNfc+ExqK3JufnNMMSOfGAmXHFXOkaH7uvY2JNRcUhOQ+mziqCIge56WJNZ/ gi3cQ2XcWFl2PGrQeodns68az+EoIqlViDNY/5PRPUy6or731mNfftlLkYP0wViktDEH g0WRqa3axpcg1emeJdFyEEwnjL1yqR7Zf3pRutm7Z9+tW9GBqkt749c0MDSZR/CpVzak uesUCRPASh4BPez2LMwWVpOzpXLQw5JYdzG+OUSaei3lVr1p9uTTjV6lehd+V/Xsz/IX /oSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960192; x=1685552192; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nH88FGoAACjgZmQtkl43JvHLPN3RafkQwEsrc5bfN4s=; b=jZEpPeJtZMmAJ5aB1wcrW8k+/P/4B7UvgK6G43ubz9r0vtdlS5Bhg0w5JlqnGl7cgY 9T7jcDfZDrht/XXi6qhOjOFX/BXkR0Ev88WImv5KWonqIWcc5b7lzMg9nyqlTLrrcmB1 YvYLC9u+ACqBAi5lussn3nvFQdyWJ7+YcB/MOVKC0EkxQE/HRrK6LLrnpqp+0vwoZ+7r 9X30pIoiECF4wB7DakM82P0hsqxNL3CZR1DcNUrOE4KxUwNWTaB/HudsF60B0vmFDDQG ZaO6U7ogMFi6MUrMmVFNOYGPNQlQ7pfDijoLn7l1+TLGCDNjDq11TOXhShnpVUBnKXsJ mTSQ== X-Gm-Message-State: AC+VfDwEaXqzvKun1nVFzwSYoIysbEmia49U8LpY+ywKNU4cVP2ZKERJ dCt0ZPdC656XOnoArankU8ULNkuZnn0= X-Google-Smtp-Source: ACHHUZ5TmDDUIxEKAG9xtiDkwRqmJC5j3DPuQpKL1gsUmhKOzqQ3R0545H3ZiUaU/3KgnQbVgGPYgv/taTQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:4011:0:b0:54f:9e1b:971c with SMTP id l17-20020a814011000000b0054f9e1b971cmr8801791ywn.1.1682960192486; Mon, 01 May 2023 09:56:32 -0700 (PDT) Date: Mon, 1 May 2023 09:54:47 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-38-surenb@google.com> Subject: [PATCH 37/40] codetag: debug: skip objext checking when it's for objext itself From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 5E6CE2001A X-Stat-Signature: 86dshca6hq5m33bgtdkfsqwmfmr1g3g7 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682960193-864748 X-HE-Meta: U2FsdGVkX1/7Rq48tpHSCjrZnC4/k0oRe30/adYgc/RCeUGoo5CIWvehKatlcu6OCm7e1npZVctalmFaufGoNtcdOMJTmXIDfNZdoXAZqeJohPvqnjxkgpPpcuOC4ACzDvn/R7YLNp8J0ceKlV9qHb1e5f6NgP8lmTwwr+fQuB2BbsOpimt/Sw+4S2Vs8wwE+biBPkvS8def7P7ay8ES7jRI0iqgxXqhjEogdHwUVtZ1nGRPPb44GfjUAI0uewCk8Fky2fj/omAwpMtgVEvl8gYxYcXTO3p61mqOKuCF5lnZbTLQ5msCP3Vm26FSLTRQwLEc3Wkhcc704yMCqw4+M8OHIKdghrWtHTTU1IefyobNvfCH8+MvratqT/tKZoFURNfIBFFRqS336D9zmbEYKhceolD+jarhNaSn3TRSk2+Nyq5LuBquEoHEXWS3LG0vdPQCkIjlnNbDM4UY6pA5I80ceiCZWG8pAa14mVj0X6xcB9GWq9xIKRGudDPXCDyEw5/m0DodhJds/F8xy0XN3DeZTUEdD6/+dpm+X9KhaCxxiASQ6DiFhDqovAYzoWRWIQuIbBlSxvjPy1QK6UJfje+ujy8EjKgH/NN0HtluEx6FZj9cWq0GI3GdsdU+lvCu/vU+lWa6clT896PF6DnqXafApKofKGVy1hrq8UBVUo9ayhdDeFctw59M9uTusj23dF6ASJsWGkSN2ball/m8ZDjnyeOs5FRNHKv2EDSyqG2gwHiud2JceDIGS2d5Yx5AeY0p0SJPQln3JhKD3Nic1xavwSUl8utp0OOf/Tz01Et8KzZgRJbLEQ6JNemE07cuuapwWer3fVKqn5HvM3hiWE3iQb97qT5n2GxIP1fEUVJqb23EHSKILEHpe1YdG+9PZn1AC6kLM2D0QhRg1N4++U6Rlaq3pegsG9ANF7yssftJimoMwqB/4kmpr3uaJnemldahSQlKO9oOoo3H7lK dqUpE/f6 h+KfmEPtYjlsTEMEuJ73Ynly1vM6Pz05YlE2tGXc+LWfQuXrE4hWQbZfSNLQHMFpgTopGZak/GAYqn2t2pbUrbPmyL3sixxFpjbOB60Jp6y4SpBJ75sMfTNyQgWx8mb2WlaCqFnlgEAMjOsjoToAp75ViQR0F0Sn6c8hTn+vbJSWesqpSj4kjftZGWcawXQKi47cB/PPevcQyU8lHE1JhIl7VZOetCm9AHktAqjeWyhvypQ2A9iGQTVy0VaCz3Ft4jfAK50EV7BQnfYnA1bqdjcaxc9eYas2MslCIM9VtPIb/H/LCBQ/BK9CAZ2tiVJQwTmNngFW1Rem0Zxn9YXkvOJjP3ETXtBI30PAjH8yDZXJ+DBxAI580VpJb3iB+zZBmU81avb3DdEfHPUiL3r2YdopD9yfg6X0eGFB8IYUvjEqHI5GXOUEtpp2HgXlP/iauL4CHVjFa67LMtGAPoE9iGN3PpVSHVDmQ+SsI/6aUpWpVFNWVBZvmLMvDrUheKBwsC+Yvb0HCwqaWOSqH6EvsUS7HjkLWd2Rte4FaClF6qFaAgbgnKvlUaHAcCFsGv30K1V1PPjGn0CwP1iRY0BaUsBC9S7Y85mDCQ86QTZzDyxSqZdupsAv18rUPtX0fklEez2TUaBKRaUCCmo1/p0FWczz8UOIbjn05QqgUS6BqwOmzEbFnoFyLd5IVrA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: objext objects are created with __GFP_NO_OBJ_EXT flag and therefore have no corresponding objext themselves (otherwise we would get an infinite recursion). When freeing these objects their codetag will be empty and when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled this will lead to false warnings. Introduce CODETAG_EMPTY special codetag value to mark allocations which intentionally lack codetag to avoid these warnings. Set objext codetags to CODETAG_EMPTY before freeing to indicate that the codetag is expected to be empty. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 28 ++++++++++++++++++++++++++++ mm/slab.h | 33 +++++++++++++++++++++++++++++++++ mm/slab_common.c | 1 + 3 files changed, 62 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 190ab793f7e5..2c3f4f3a8c93 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -51,6 +51,28 @@ static inline bool mem_alloc_profiling_enabled(void) return static_branch_likely(&mem_alloc_profiling_key); } +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +#define CODETAG_EMPTY (void *)1 + +static inline bool is_codetag_empty(union codetag_ref *ref) +{ + return ref->ct == CODETAG_EMPTY; +} + +static inline void set_codetag_empty(union codetag_ref *ref) +{ + if (ref) + ref->ct = CODETAG_EMPTY; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } +static inline void set_codetag_empty(union codetag_ref *ref) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes, bool may_allocate) { @@ -65,6 +87,11 @@ static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes, if (!ref || !ref->ct) return; + if (is_codetag_empty(ref)) { + ref->ct = NULL; + return; + } + if (is_codetag_ctx_ref(ref)) alloc_tag_free_ctx(ref->ctx, &tag); else @@ -112,6 +139,7 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, #else #define DEFINE_ALLOC_TAG(_alloc_tag, _old) +static inline void set_codetag_empty(union codetag_ref *ref) {} static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) {} static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, diff --git a/mm/slab.h b/mm/slab.h index f9442d3a10b2..50d86008a86a 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -416,6 +416,31 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab); + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) +{ + struct slabobj_ext *slab_exts; + struct slab *obj_exts_slab; + + obj_exts_slab = virt_to_slab(obj_exts); + slab_exts = slab_obj_exts(obj_exts_slab); + if (slab_exts) { + unsigned int offs = obj_to_index(obj_exts_slab->slab_cache, + obj_exts_slab, obj_exts); + /* codetag should be NULL */ + WARN_ON(slab_exts[offs].ref.ct); + set_codetag_empty(&slab_exts[offs].ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + static inline bool need_slab_obj_ext(void) { #ifdef CONFIG_MEM_ALLOC_PROFILING @@ -437,6 +462,14 @@ static inline void free_slab_obj_exts(struct slab *slab) if (!obj_exts) return; + /* + * obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its + * corresponding extension will be NULL. alloc_tag_sub() will throw a + * warning if slab has extensions but the extension of an object is + * NULL, therefore replace NULL with CODETAG_EMPTY to indicate that + * the extension for obj_exts is expected to be NULL. + */ + mark_objexts_empty(obj_exts); kfree(obj_exts); slab->obj_exts = 0; } diff --git a/mm/slab_common.c b/mm/slab_common.c index a05333bbb7f1..89265f825c43 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -244,6 +244,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, * assign slabobj_exts in parallel. In this case the existing * objcg vector should be reused. */ + mark_objexts_empty(vec); kfree(vec); return 0; } From patchwork Mon May 1 16:54:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227672 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBF63C77B61 for ; Mon, 1 May 2023 16:56:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E817900015; Mon, 1 May 2023 12:56:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0989C900002; Mon, 1 May 2023 12:56:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7AC5900015; Mon, 1 May 2023 12:56:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D9136900002 for ; Mon, 1 May 2023 12:56:37 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A84A6407E2 for ; Mon, 1 May 2023 16:56:37 +0000 (UTC) X-FDA: 80742290034.24.D0A6A9E Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf07.hostedemail.com (Postfix) with ESMTP id E034B40020 for ; Mon, 1 May 2023 16:56:35 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cL9ulyr1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3Qu9PZAYKCIk574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3Qu9PZAYKCIk574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uHz3gIMQ14LMAlJH3BHlupoStZ8/+BrUL09pT+1nDVs=; b=OB5DElKjTxGCmWUdW3b3y4PcZ48pY2Aflfq7C9v9HOhVmjymMBG+NLdz18d19aaRL3J1re mN2DcsCkUSqOajO8jvnuP3IeEbab9yFBNPtgx5FS/mSaFZK4mO6nZnmEFrX3gEfEphut+C NLxB9rlUgNgqMAEzD5u7byOKUHr1sFw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cL9ulyr1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3Qu9PZAYKCIk574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3Qu9PZAYKCIk574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960195; a=rsa-sha256; cv=none; b=rK8KNEiULg39M/u7lWuT/qJW/qxqjL/9m30SLyPcDDOhfGqbTEvUDl36gDn/0leMMQvkI9 fmU/L8GNPoN2sUBorfHt9L/S6HcMfrvY6VN8SIKR2+avrh6QKy+kGiBQyn+Uu14Si8X7J6 BhIXPiAPF9opT1PI3tA1RAygVMaFayc= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-64115ef7234so23372363b3a.1 for ; Mon, 01 May 2023 09:56:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960195; x=1685552195; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uHz3gIMQ14LMAlJH3BHlupoStZ8/+BrUL09pT+1nDVs=; b=cL9ulyr1TZmloHcY9ZAQt2916/Hy8kFnLfCTiFsQOyXG21mKZDk2kVsVGS3swW/o9A lmsPI4V+ni8bJ2cDmSOvMhTe7jiZncm+a+dX+h32N0c6xUNbGxEO59xGx2raskwk+yE+ EY40N2q/B5lxnqf+wqKkKUxKcIxmFMWDeQ/MPMZ5UoklVL3SBusyJB6cuGrGjrAev2jt I6daIpOCKx+oNZEO+zjvrtYAnmo4PPKmPdsrXndRo+l1Ieu4A+pt+fjUYtVugs+1Ncqh keNiqCygIJKFaCxV1denDbSpt/uk9T3I2i0PdEP0vYROV8UgAo/d+Ogm15HjT6dg+WeL DFBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960195; x=1685552195; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uHz3gIMQ14LMAlJH3BHlupoStZ8/+BrUL09pT+1nDVs=; b=ITvPf/KAmMU19k2yCrRW0I7StE/p2NgXnFhG9B4BtSaMbxvnWiFUQschKq7uTYHu51 Omcspg5kWXRsfU8o8RrDjdczoc+58kEjYfwzNMQ7EactvK1j8DcF20c8SzhWN1Hkt5Db JhSQworfJHiffY34ITq7KytFIOUkk48oHsEpykKdqwBWNcS7yILWq2YMJDWVuX4KZP3M y5OTS/bcDwKzxnDwa0jrFYSAZYvhpDBHCTKpltYj7XwFgqkZi2vvFWuP1AEsjEeyDbfH r0H97NIB60yMxY4vMXIMhiHVUy/JcwV717/99lrr1XeNVL6l29dSf/q3HzV9GRLZppfV ijDA== X-Gm-Message-State: AC+VfDxMAoDgm2y0KjBtbZNDhP26Sq/e8x2pj+GQHaeaAZeQGErPk3Rj 65NnWc9NQa8a9zQ3ErEbhAHeTgG7VBA= X-Google-Smtp-Source: ACHHUZ7qtjt6xJv0/W6KanE/G1kJWq4JffXkn4YmB84VAxkpL2X6o57PZ5XdLwRiQeUXxZyrcuCDjEjhXCs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a17:90a:5105:b0:244:9620:c114 with SMTP id t5-20020a17090a510500b002449620c114mr3673305pjh.1.1682960194723; Mon, 01 May 2023 09:56:34 -0700 (PDT) Date: Mon, 1 May 2023 09:54:48 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-39-surenb@google.com> Subject: [PATCH 38/40] codetag: debug: mark codetags for reserved pages as empty From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: E034B40020 X-Stat-Signature: knz3s67gisi9of945ism87yw8or631i5 X-HE-Tag: 1682960195-30836 X-HE-Meta: U2FsdGVkX1+XvmSvS8fpqdiUo9K+pFJSpy/1BuetmdpBXV20WIaJk+XSecsMVc6TK//qaTmufV+K3fWJX31fLsgp8u1/zQs8UvoCJLlkacOiRd9FvQGPkn+RzNKV9seswho8VeA673dUReScMBbDwXie416+ZVLl8wBsh6043eL4WGPPnloco6Be5TD+cilcuwa83IiV8BFsCBFxaDCe1dbB9+nc33LNw4O3hMM+a7yEezWqlTurO7LzhsFICTg92Lwpy5q3F6Eqspgmk9i43Otu9uB3elCzeY3E2UQwqWX3jc383o7KzMN2LTDVxbQzsYHmSl9v52E2FF0QriDVr1FWE+DIZjMOWIwJ8iYPgDRq+QtGNQes44SjQxzzFZcfB2ly8qc4DftPz/L93og9wZGhcP8XSZNygljfvbixK2EoWi9gxb2lbdtRiefb7CLynznZPsHVAe3tEPx60TGpzXjEUrdAwP+XpFJAFS/TWBao/VYTNFEEH+fzBmRPnramKi6mTw2Km/3cKD5SxbB8b8L2dBRU/xXINEi3BJsnqnrFjsnNc6vUA+KUdcSK72Tn+vlVdAlm98qlQUYryVmarSShCAzjtQzcmhrNZH75wwO2/oXWtYOPKwF0feBSb/ld6HdxiAlHwsTDF5DY4fEzvG+MYxyb1xyAhgWVcn3cFWEHYW8eNmh7n5thIhZQQ80Tj9Cr/zNGF52gSJgu98BGpxaqmiwgxcyRm1JyJN7xPxrOdLyxaFhA8qyGLApbdQQ3rWzvLIUtusWpiZhwxc9irOA01HMEWj/30TnTzQWLZWIbpCwQjjryw6YavsXUbrFUsqL+TIee+3SJ5exBMpRZDsUdOq1Fe3+sqW9SWylbk870+ZETfqcqmK61PFB+32ZPsckL/WXBufoOjCngDFe7zEXYD/y0pnpybnIASDjxoEPw1U1E5C8BCswRa3fMV94XQy5bNXnn9BM/ac2J06/ Saxl2mV5 A96NDQl+ST4P6zEf0Fy4xMGBsfu3GvmiOVnfAsIhVxo3vwuhF8tjsUxB6qbSMEi3JEAGXzSUQudQbyZ9sF9q2O2k38Z4uYnxayZDsUBQoLkzwxBthiv9CgOLZfwoOAXmdbcINHSRnIv/fX0XvRAha1hmvcOixC4aBzCzFFkrpUBVOAGxlC+cTr7V7TfjnGaa1XQZspdUlVSGpulxGSZ1/BlACJ05JXPIPW0qhigyVRPqWhpvx8/ky2tlW53ET24JsVo1ums+IK81/qnJlH+7P4NjpmIPqvuBuNlGs7c9VDeeg0DNJ5gI6LIcY96AACv4BNl1cA3CGbceY2jd8zqBLq30F+3xQvCaSK0f4tkdkVLj+KWs5b0aZDBkdzxHA6LCOo94OYR7cn2WV+Awpv/1vLYFzQlflcgMP79gjAWMojYG3pGvLi3wL5m5tZoSBxsxoatvHsPyqDqWa5+reIwlDEWyeMp3y+tiFjcfrCza/h/elyGfiC4DWWE3hQdEUAUf7X2KdGsmD/10uzLJ1RpVDqYEpMdvmjJ2DhhP27D0VEu7kg9mviXBkpL88GTBYf04xZe7Wqp4qGQMJ2CSGlj+dwd8q0A9te7+tTFklvOE84TdW9Orjlx6R3+0ADkf1tRr6c++7K8KaI/9wddvKJe0JqNTcg7/nmV/QdyPQcVnUO393qwRtcXl07NnqW0CF6sRpJv1IlgrkR/s0U2DD8TWrSLFVLQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To avoid debug warnings while freeing reserved pages which were not allocated with usual allocators, mark their codetags as empty before freeing. Maybe we can annotate reserved pages correctly and avoid this? Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 27ce77080c79..f5969cb85879 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -2920,6 +2921,13 @@ extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end); /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) { + union codetag_ref *ref; + + ref = get_page_tag_ref(page); + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } ClearPageReserved(page); init_page_count(page); __free_page(page); From patchwork Mon May 1 16:54:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DA71C77B61 for ; Mon, 1 May 2023 16:56:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9730A900022; Mon, 1 May 2023 12:56:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FE3A900002; Mon, 1 May 2023 12:56:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77857900022; Mon, 1 May 2023 12:56:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 601E2900002 for ; Mon, 1 May 2023 12:56:40 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2B4461407E5 for ; Mon, 1 May 2023 16:56:40 +0000 (UTC) X-FDA: 80742290160.29.9E28AB8 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 5420E180018 for ; Mon, 1 May 2023 16:56:38 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=48U8jHG6; spf=pass (imf06.hostedemail.com: domain of 3Re9PZAYKCIw8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Re9PZAYKCIw8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960198; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6mzPQystPbLbDwPdw2fPo41K7qknlkieEKK3sweddW8=; b=E/AQPbyvJuAtGS9FMXextXnBPArZHKTRBjdBTw08RXHkUAXSXSyuiFVH04lNAa31ZXj6b8 96ENJAZE+31lbHpPsRABf85xOvcRgM3/B6LnbOzAU+keL75P489v7ZcLiJOPbDp4IlT4t7 HAL5gUCb0F5haZPPCyOA8rVcgOC9Gmg= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=48U8jHG6; spf=pass (imf06.hostedemail.com: domain of 3Re9PZAYKCIw8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Re9PZAYKCIw8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960198; a=rsa-sha256; cv=none; b=2Jr0xFpt0hCKka5YaOKW8yN2YH92lpXKJPcQ7uM63V5G3DvjuTDubL5RM9e5yrxBvZeN4C MTyBcsL+JOxHnzDJRAiMRYv4nRzGXk4TJ3PV4Abx2eRmyV3PhhbNgdBVwSY8Kcj67YBCl4 0tZhUBVMVOPEaS96ltumChQYZAP2JDk= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-55a3146ed47so15573147b3.2 for ; Mon, 01 May 2023 09:56:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960197; x=1685552197; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6mzPQystPbLbDwPdw2fPo41K7qknlkieEKK3sweddW8=; b=48U8jHG63mdA66+jSc2MsTTE/SQi8BR+RKY6JNGvzt9XyZfx935vfOLln6IV1D1nZA TDOiQ/WUR049cZ5YJ1Q2SXSzODz7LrWrQUYaviS+GqGgkaK3yK5eLgyzf9e3cLjtYX0E QEA8ptoQ0G50fmCmay29a1/AGKziq1anSqyI04k35PIYMGlvV6wz4iKeKGJ75o0JVkmO ON7F5nilmKInR+kNfyJ2/HvMFyxBUo4zG20TTbLKq7fKcUY3NShJQ1fHpbVtH1cQdhj8 PGEM3ZzK108E7+nu3T246tg/tKyPHMfi0CFtYYsesaQA8cqlaVTRPDBW+c2GnGCMueBG 1wCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960197; x=1685552197; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6mzPQystPbLbDwPdw2fPo41K7qknlkieEKK3sweddW8=; b=bK5JKYWn8PDkqL5f4rPi2QgCq3Ju7XKBtOU6agRgS9UKc96zu24jW/nR8QYojuFa6M QeumSLvDwksv7I3QQriLco3AURv2vTow5aFQskiZvZ5RGFph9Y3F7r5cS1o53p6lZZzL 3Vmqb+hxCEnHHFLeaoqi/doaDACYuB0mjaSGrD/+FkJPDzndnEoNU64dqiAa9Lc+DVME n3Z19aYj6H33a1FOTIK5HGxh5mM7ffMQ2LsZl/CddYHz8/Z25pQjhaAgIs05ucTCOLh8 MLMWk/gAK7dEtsruEc3E0gQUrKBrrrkneeeWvwOVZhY9Nf3Hyn5779q//yuwUpEbd2HI X6uQ== X-Gm-Message-State: AC+VfDxL5KFDlGzpynuoDVa4CJg7RWhaPTlSgipX6k7Xk55ltr82iiBK fItzA7pLsh8KVa+LzuwSJV95hz7Zusw= X-Google-Smtp-Source: ACHHUZ4jj2W0kDCYmm99zoZd05S+xWh4GtcVfBuyUeKMJ0WYF9RUn/+Qut27GKgsUVS6rbo/+wwQvYcbMjY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:e902:0:b0:541:61aa:9e60 with SMTP id d2-20020a81e902000000b0054161aa9e60mr9069879ywm.6.1682960197340; Mon, 01 May 2023 09:56:37 -0700 (PDT) Date: Mon, 1 May 2023 09:54:49 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-40-surenb@google.com> Subject: [PATCH 39/40] codetag: debug: introduce OBJEXTS_ALLOC_FAIL to mark failed slab_ext allocations From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: uto5xoniizqo3xs8xfgkwchom16ya155 X-Rspam-User: X-Rspamd-Queue-Id: 5420E180018 X-Rspamd-Server: rspam06 X-HE-Tag: 1682960198-375560 X-HE-Meta: U2FsdGVkX19KmP6yLKy33cEqdqU8FOUji8zTTRrbZupTgA6j9LajPM+fmjKW1Eye9dilKdT2Ma3/MvSFV0756czMIxjE/TKF6d0Ui2DA8dV36qTwK8GetCwx2IYJVYd2Q1YNt8vZ/bYi4FCSLfmBFa+wHLb/Vresjowrbd/LQKfvmUbEF9Hz8BnOxOA1LcfRl9qHjZOV/xXh6rtzvUKp4haZPBRon2QF94Oh+Y9+pAVtgSORBZL9OCgXkayXGnOs4K5wqByPKJZqrNY/RzUrguP5UZh4GbfqY/RygfA7qKTaUnyqlXza6tkKnuVa3mPC+XJM6tPSoOlSqFW5slVr8Wr6DBopbnBvueR1qyIiXL9dj225GrsDNyzrH7E7xRWnt4t72GxtvxESAiZl0QL4o/lLGlC4A6zyyXjfn0jfoP8gaDgdDgKXZ3zWgdFMroXFUKP2l1pjmFDvKvXF/l1e4zI4TrW33trhWRfDZshcjEViG1i8yJxVJvBczEWt2QV2O40pu6dI6XFQqtPzW2XC2Y9ESkJ3WjsPZnSXBnOCZ6PnrCx+AHMnN3Lz+OYVMJA74F4ADNdClFqMS4b32X7IIs+UOWU8U6gue55xtVssJrwmmf4+oIiC8LdbM2XNphyV1JMAYDfrT2SvE9bcPrTuXjr64lHwSDFijP8AZi7DYZeBGOKPaZgLJsmIaXkyjAOrH+QcJ2eWF5L1SjA3DFc+peWmYX/5Lef1Cz1azVakywxM8FePSg1/to+3nG9t5hlgMpR4/nsgHPKHirN15r+/71Z/3yNn2XH+Mk6p5VnYKKd981vZSEhtacgNzEz4IH/v9MXzYCJ4WUiZ/kONXWhVumuYkghCrQaCwRR6pZNWz9YRf2Z+Dbq6KgGoS8Yfs1mA45+gZwEap758PYimKtQpCyaqeGGIRc5aSZrbhdqy3cOZvZPUHyDOHlV2+xVftglULozB6Uz3JXp7g5SJPSw PHwMiK2D BXLgKDk6jDYuSiujKo7smxeYNmiEWZC5pSMhC0n2XTPVUWE3CRcScOLqgVw3d3IqwkCGI9D+LnEBlD1XhtBnclprE3jYmHrZIsHOPfQ2QHd+/WjPjFpqiaboje5i9pVInAQO/+/dnZWf7o1F4knqYxVb9QsCMxBlsf/FQta8N44hMR98qof9hju1by5ZVeC7xvhMEnyTkz1eWpZWjXGSlOs9oyJSacIBgRXoy79xL8rT0yijicOEUUOkPpj1ncl4UGGNzViaS6d1UCmgePpjOWhn0pScb67SXS00vHQXqnMil/j8E9065bpebmtxUMW3ns+Zu0Qv+jdQ6OQC+kilRfAhab0gai2xeN6WCrbsKTwoQvEUN+oYdQGV/Gzd6tO8emN+3YqcXeCN42n9R484oeEfSSLF2oF2SC0XTqL8c0nWo82ocwT1Lao016pgiFAwWCmnkGEnK/obZe79Cqmo2qw/AesNfwO0A7Zodr9aQczE2J89B6J0U+5WLGDsGZuy/VJidoRD1/Ncdxy6tZHekbnQ2KZbOg7FjAfjVJuoRyCUGa6QiC5w2LQ5V6vrWlxcBC0TvHxQS55Re7vqSJSfFG8hu9UbPEgWt47D4Uhwu5UO+9MB5NVlLhnNDyavPg1NT1E/qlQgEGhLC9PuS9e2UefXUmFwuGA93oatK3QR3HzDF/3aKlk2KqA7BdnEcMLZhbAbHEJxJ7yjmj0qM2cBeBzziVKV4a4ynGzse X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If slabobj_ext vector allocation for a slab object fails and later on it succeeds for another object in the same slab, the slabobj_ext for the original object will be NULL and will be flagged in case when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Mark failed slabobj_ext vector allocations using a new objext_flags flag stored in the lower bits of slab->obj_exts. When new allocation succeeds it marks all tag references in the same slabobj_ext vector as empty to avoid warnings implemented by CONFIG_MEM_ALLOC_PROFILING_DEBUG checks. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 4 +++- mm/slab_common.c | 27 +++++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index c7f21b15b540..3eb8975c1462 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -356,8 +356,10 @@ enum page_memcg_data_flags { #endif /* CONFIG_MEMCG */ enum objext_flags { + /* slabobj_ext vector failed to allocate */ + OBJEXTS_ALLOC_FAIL = __FIRST_OBJEXT_FLAG, /* the next bit after the last actual flag */ - __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, + __NR_OBJEXTS_FLAGS = (__FIRST_OBJEXT_FLAG << 1), }; #define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) diff --git a/mm/slab_common.c b/mm/slab_common.c index 89265f825c43..5b7e096b70a5 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -217,21 +217,44 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, { unsigned int objects = objs_per_slab(s, slab); unsigned long obj_exts; - void *vec; + struct slabobj_ext *vec; gfp &= ~OBJCGS_CLEAR_MASK; /* Prevent recursive extension vector allocation */ gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); - if (!vec) + if (!vec) { +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + if (new_slab) { + /* Mark vectors which failed to allocate */ + slab->obj_exts = OBJEXTS_ALLOC_FAIL; +#ifdef CONFIG_MEMCG + slab->obj_exts |= MEMCG_DATA_OBJEXTS; +#endif + } +#endif return -ENOMEM; + } obj_exts = (unsigned long)vec; #ifdef CONFIG_MEMCG obj_exts |= MEMCG_DATA_OBJEXTS; #endif if (new_slab) { +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + /* + * If vector previously failed to allocate then we have live + * objects with no tag reference. Mark all references in this + * vector as empty to avoid warnings later on. + */ + if (slab->obj_exts & OBJEXTS_ALLOC_FAIL) { + unsigned int i; + + for (i = 0; i < objects; i++) + set_codetag_empty(&vec[i].ref); + } +#endif /* * If the slab is brand new and nobody can yet access its * obj_exts, no synchronization is required and obj_exts can From patchwork Mon May 1 16:54:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 571E2C77B61 for ; Mon, 1 May 2023 16:56:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E078D900023; Mon, 1 May 2023 12:56:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB7AC900002; Mon, 1 May 2023 12:56:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE292900023; Mon, 1 May 2023 12:56:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AF304900002 for ; Mon, 1 May 2023 12:56:43 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7D14CC07DF for ; Mon, 1 May 2023 16:56:43 +0000 (UTC) X-FDA: 80742290286.11.83EB186 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id AEF204001A for ; Mon, 1 May 2023 16:56:40 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=XY3g0fA5; spf=pass (imf04.hostedemail.com: domain of 3R-9PZAYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3R-9PZAYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960200; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=voe2xx017YXPnsPA1F569u968pU4cdVPLrVyFh58V3A=; b=pHxG0xCRDNRpaPY8rAnKW7uK+4iAw2FrqoLz7iwJa2b8xn7IdqqJVoAGQgr5lcC7u1Pt7I zTJ+tFJLvrGIpZsmVNubOw4O1cwRbCTjrIS29A4qNyGEQaFyrpwRc/nH6uNBb8kzLwKnmE UFvQ5MwSi/pBnv/v5TDAE4w5c9iJR2s= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=XY3g0fA5; spf=pass (imf04.hostedemail.com: domain of 3R-9PZAYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3R-9PZAYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960200; a=rsa-sha256; cv=none; b=qhr1lIoLk9s6jUVy9rdRtcVgwh/wkJNa9sb4Xs3VmS//Kj76KQWmY6yYji43LsObRPfyLl Af2Vh1knI7VwaA9SfS0O0N7rCVSFn4nryc678YIBcptUFN21F9X2xnnNeSCqVIqUwmxfAO tXMbdyXRlBEgA+EDSuJqVB/i+f50AK8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-559ffd15df9so31996347b3.3 for ; Mon, 01 May 2023 09:56:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960200; x=1685552200; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=voe2xx017YXPnsPA1F569u968pU4cdVPLrVyFh58V3A=; b=XY3g0fA5IrJ+YzYe2wUQ0ftZyFUg7ny2GrTo3NPiBlk4iPlMC/kY+SH8g6iLrwV+KV Xw+if+jca7PJgAtbAXeVriY1aFJ/EteLLOMZx1AdK6gJrXbG1uRZWDO/40Axb4AODOAW bu8B0l1k4HzrQxgM9N1NQnABkM6L6EHe4PLk5Qr9MgkaJ72bwWRTz68SbwPhkOTfAlCN aIInpOCqjYNQr/7r19P8lUnDyjtSJAnNSRwk6D0bpZ0QASPWz9H1GwXTn8bC79Kr3VsJ to3IBoyJHWuDBBn4X4nEQDSitAsGc5WJp6sVno05l5o1QoTuKQpKB8qBzlmM5zczDq7s CPoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960200; x=1685552200; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=voe2xx017YXPnsPA1F569u968pU4cdVPLrVyFh58V3A=; b=hDonPUOT3EeCSBMRJAAlNTnSuUiBSUJW3UEwpxxnVkxUWvh0710t8sG1vanpRA5G4m wqUdMM1qNPDyCioi+Qc26eOvLQzlk1tezBVWCBdNiU8JvDV9ALbemvOBVG1eAa1ku4rC x2A4gIgefG2op2oTGGRLC8ZuDCi4emT/AG5Vhz5rEZe5YuWJ/DsafKFgLUra0EqxXQOF Kv0/sJLq52PxRPqSa5kNcvWSWINpkAtnW1pNraeyS1utBFBHhC2yWQ7i2oneRf8+KGf5 Rs0SBgiXuK/mkII5R+d+1eTyYEPrzvasBuC5OPMnIK84afh5lBgaRhxsyRGNFKVTl7L2 ddxA== X-Gm-Message-State: AC+VfDxi+i2AL08le621luRNvcibrrNVNnAot89DeCdIe2NPF7MukUh3 jtkQcM1W4KiRIYFL55cjdb/jaSf4SnM= X-Google-Smtp-Source: ACHHUZ7TphX74FdXdGj2KrrIAly5W9cbCjXqr70LC9E11i/xFawgHPr35IiqOdTvLYzAxeQM5sfRxSe2CSI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a81:de0c:0:b0:559:e97a:cb21 with SMTP id k12-20020a81de0c000000b00559e97acb21mr4262900ywj.9.1682960199781; Mon, 01 May 2023 09:56:39 -0700 (PDT) Date: Mon, 1 May 2023 09:54:50 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-41-surenb@google.com> Subject: [PATCH 40/40] MAINTAINERS: Add entries for code tagging and memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: 53ktxa19yadgno3azgefpnfdiw6qq5kk X-Rspam-User: X-Rspamd-Queue-Id: AEF204001A X-Rspamd-Server: rspam06 X-HE-Tag: 1682960200-737512 X-HE-Meta: U2FsdGVkX18RV46zE2pomX4nbbAJ5rniUVebi1WOpdywOE0JM994WlnDt3jxXb337TKm7gaAzPW01Ktt+oHfe8pAxcWaVUeqW/1Yrms99PsgNiGT0/MqBS2YKlM75WrLGs2EAtQxYnQr3E+oIwmTh6DmRCUwuJTYkhZoQ5+qWmaYdqBdRpf5eG869VPoXFXrubvzS7S4UyHDYiZmL+Elr85hWCuSxM7lm0uHaM7ieRpjAj0y7EYpK6htRF7wJG2RClJNeTU5K/p1xCQnk69UYQFgoOWoJ1Ub6pzhKFA8oJTixr+GS2Z/AxtaoztPogTt74HL8JeFmCgMZ0c6G0Y+dA5sjT37oyuj3DNpqE5OF21+iIrxgpZOAn0c5XCv9mRwMP91Fq3c1eES8sRsJnZi1jUZwzgxR+MO/5gZiABDdM3z07p3nr/7He9OY2gk/Un80NWziu1VQiRavtw63on4jfCxgfe5jZvIGRQwjDvmBmlk8cAEh1cM/V/AbBIYiWv0JMlZOPAGzY5cBYpbTnmJBkWYl5RMxmmImzCi3SdF4EiheSVcJ/bHKnNCLx7e+neD7SUaz4pdsam6r2TuzecnWIogoCxWfvEs42xaxSpQX7vSI8M7Ibtncjaa4wEzQrWJBYNevY3OnhOTh0KsYqwsrc3T0KGe72k2aulZHdYJxBqzbUawlDM/9zuVhKcG/EXc+w1diBW9oxdhq3i7afyohfOEzsLqRXDOtIRmRBZO+9HMddZ6dvESDFHteWcNNuZM7OWFTaB2ZAZOEx9qoZXXg5f6KT6xOGtwBioRHfCiXYn9cwwbLJu6N77ooow8g/ZcrOkJ0w3+P833/WRH3pRM4cHj7ZuJQwhXxTGT4OXqzNkD9VgJKFShbotisAjPt2qDzpQD+BIVXfg7t4Mjy1BMOtPpY6ImCBmMRTHL4WWKP2Fl6ZZYzt+Mbo+MlUn0LubcD9sA+mojWL1AF4Dnpwz y9o0ICYK EHmkin+kK1/PNKlvMLsFv+EUXBc71d1VGCiFJ5I2WwleuJ+PaKIZJXXUzTLQnWgxKSctcI5NA3mg18Ochbiv+uS37CvVfw/WqZ//9X15w3FLqO+ORW6lZcaC6xs6Cnu41jDWcJU6WdkFooUpSVH+HOaN/cLxZsNf/BJT1mjQnIKnCUAlEWyUF4FFhuFPCy6g6s68EI6An6n1ata4WSAG3rGx6fbuJAV+9D5KQvFw93xfAO5ZhgnLXbtgRIEUsM3gpwdl1QCsc11MDQQlXJ607tvESQ4O7mdrAjZgmYcHHlCXadsjQ9rdb7SLpugWTz/w0bz9dpQqKaj/+upMhUD/FvnIztFjhGwwQYOunYfT1yIYYhcWMwoa7djHA/8fsrQGHBHZu97wL7W/WyfkBru13JhssLFrU86KZRZuXXEacsixWx7/PT162/nu2dShk3TAkwLhWKM/Zfd9BqYM1WI51VObJ9m6sCQzSyGz/IDvuhzyLZaKqV7Oq6bdOk/tFscq+vz9Ny9B6rCHBJEGTGjfmoabA4QJup9vimcYDZBq1VZ5CMMBDSQjzPDJctvlCteIfbUHgjg4GMm8LHuI/XMQqTvrIRaoxa3PUM3Teyt9duIJPfponnUPISY2QPBnrgx+p6dQIdELikViRCJLP9bKPs7/UutUglaYomUCoWUUHTNc6oHtPh/pwMcEW1I13oTpH73kQP8+8LQ3l1Psr4dGAAsdXP+4W2aRlWg7SZjMSPHigMxADwTU+9NfChzIxGO5DMZPa2OyKhNhTDrQYwp0bpXD8W9SWgpe5PUTmWf7YXZz4bRtrt92MqOixinxxTHzN0U8yobPfkDYlEkckDQd16triBIny3b2PJHGBiUqMa0TIIjXNkm7hXQcw4Fp5ZCm+WkqsgtQ/dP3+M0rATW/9YSjfd+2iBIyn3Lp6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kent Overstreet The new code & libraries added are being maintained - mark them as such. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- MAINTAINERS | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 3889d1adf71f..6f3b79266204 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5116,6 +5116,13 @@ S: Supported F: Documentation/process/code-of-conduct-interpretation.rst F: Documentation/process/code-of-conduct.rst +CODE TAGGING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/codetag.h +F: lib/codetag.c + COMEDI DRIVERS M: Ian Abbott M: H Hartley Sweeten @@ -11658,6 +11665,12 @@ S: Maintained F: Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml F: drivers/video/backlight/ktz8866.c +LAZY PERCPU COUNTERS +M: Kent Overstreet +S: Maintained +F: include/linux/lazy-percpu-counter.h +F: lib/lazy-percpu-counter.c + L3MDEV M: David Ahern L: netdev@vger.kernel.org @@ -13468,6 +13481,15 @@ F: mm/memblock.c F: mm/mm_init.c F: tools/testing/memblock/ +MEMORY ALLOCATION PROFILING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/alloc_tag.h +F: include/linux/codetag_ctx.h +F: lib/alloc_tag.c +F: lib/pgalloc_tag.c + MEMORY CONTROLLER DRIVERS M: Krzysztof Kozlowski L: linux-kernel@vger.kernel.org