From patchwork Mon Feb 12 21:38:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9787C4829F for ; Mon, 12 Feb 2024 21:39:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A8FC6B0072; Mon, 12 Feb 2024 16:39:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1595F6B0074; Mon, 12 Feb 2024 16:39:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F158E6B0075; Mon, 12 Feb 2024 16:39:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DE18D6B0072 for ; Mon, 12 Feb 2024 16:39:38 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AF4F2A0AD8 for ; Mon, 12 Feb 2024 21:39:38 +0000 (UTC) X-FDA: 81784468836.05.FECD8B6 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf01.hostedemail.com (Postfix) with ESMTP id D64D440015 for ; Mon, 12 Feb 2024 21:39:35 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="T6/cFyL9"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3FpDKZQYKCJUHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3FpDKZQYKCJUHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773975; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UEMHu7rdWBA2FK/B5eccMh9Pcx3Z9fwrCFD55ivM58g=; b=1/eU9bwQCjPpugb8tPRJdFHcdRZDks4j+/6eBekouS5Piu8vnqNnCVJaAwUNxkicfa40kC xFc1Q1SF5LHjS6YN9zFR+50/IxhDfzGQHhZtWJ2biquMIBfoS7wJcC7dlhqT53WiqlfE2o 9AEv//VKJjlyEqz1XcnXUzy0ObPAiS0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="T6/cFyL9"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3FpDKZQYKCJUHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3FpDKZQYKCJUHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773975; a=rsa-sha256; cv=none; b=p0rvWhHblID7imRGoQi/lmdswypWWgNURdiwsALi8Dgu8qUKKEs6z/kcoHcmOzWHOG0gqj jAoGBFba7rfColJSYjEEPzWqdelDH2gVZf7aPbplxc+JxoWKxjFzLNlDLrgHgo/rLhp/ls xn8hv0vTKhgXJL9eDrj/Fy8s7xMxnbM= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6077f931442so4030287b3.3 for ; Mon, 12 Feb 2024 13:39:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773975; x=1708378775; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=UEMHu7rdWBA2FK/B5eccMh9Pcx3Z9fwrCFD55ivM58g=; b=T6/cFyL9eo1hItm3n/j2/hODiRmFHr+B1brnW0h9WAhmu/6M3bjJnPxos5uswjCZDI qjY5HiKIlKdtknBwiBExXuUjyiFJF6DuGw7SVjEqF1whNWFdm0ZKtVjDkdS1zblRiDS/ f+Tb9Iqp7oOFaCy45HM6F14cLN2Zt61QOK1ZPGWQBw06aP0LbjRhRe0wiYPCVLufdZJg Rp2BW1Ck8sA/5/romK2Fcg/LWeG0zp2oO42JgNxqgwno5smfYAJqfdm6nbn7FVOreeDj 9M+x857N+56WBNsfJd1uj7e0jI11a5+TYJSCQkw+bM4S/TUvxueevz3PbhpGIsGefb4q HNFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773975; x=1708378775; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=UEMHu7rdWBA2FK/B5eccMh9Pcx3Z9fwrCFD55ivM58g=; b=izkZXTCna96iTrYAfudrVe4miLjldzc0DtJmeJLmEeS1boL/qqUVgjXmCFkxiTsb3h j3htEPObi26Ifl0kdVq5gvPDwXBqI3ztwzALLkTeAR5AvveyzmsoB9wtXmzHlsWnlA7Q xGTd/6RYgPXrWFw1mFFHITfI/7fh0hTVTawpVzr4xp26/r823zE3AhMNwvV3rZIVoDyx I6OKQ9L7NDXQEr+oEQ9ZUEUMdEQSaCfNwAIyovO7UNcWn0mghT6zmCdmvwhx6leLhdTO /8AwJQ7PVrzDHtw53LfONykGa+kTXvtDIfJ0rcbYzvOkklWG5QKF4k4WlN0f58SEsZBC ydtQ== X-Gm-Message-State: AOJu0Yx8ZwA2+jywSCi/yaABB5i/slH7EY047m1FrCBquKrIgpLqzGW6 tzPoQIitplyWHgr4YxwyBa/BSvj8jzDpSTWNESzhUVI1laKOfWlDF2tCjNcc+2T1NABaxjQMZX4 ksg== X-Google-Smtp-Source: AGHT+IHRC17IGymCwraaE36VsZx1XRfqk/4cdiLSk63iyubj5ReAhK21VAXT2xYjmsZbV1LFTPiUTjH318E= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:10c2:b0:dc6:d2c8:6e50 with SMTP id w2-20020a05690210c200b00dc6d2c86e50mr1230828ybu.7.1707773974574; Mon, 12 Feb 2024 13:39:34 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:47 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-2-surenb@google.com> Subject: [PATCH v3 01/35] lib/string_helpers: Add flags param to string_get_size() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Andy Shevchenko , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , "Michael S. Tsirkin" , Jason Wang , " =?utf-8?q?Noralf_Tr=C3=B8nnes?= " X-Rspamd-Queue-Id: D64D440015 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: e7bsn83krmoeprzoq8pnokijr3tffeew X-HE-Tag: 1707773975-168231 X-HE-Meta: U2FsdGVkX1/qnJBYczRHXzqCVRiHmLQp8NisWHyXjd8zgFzlPIxmldxYxaMEWgndE4NYWzJQqpdGpzHJ/N/ftHjH2BxWL+EiK9jR9zDLPcUzUy1wx72P8Fcf2mpqdQxlpOKSTAaZbjVcJN6QGoW65lwLX/GBmbMvWgT0jkC6tSYwbtJSSOxi24df2ysHz/I5HL26DNiS8HjKiJYXyD6sYICe6VzhKMTm0dhoLL9JyyP0Bq87BOcTY67mAlXkxUp7BCYCSucE9qdPm/2RMCmuvOuBpQ4HI77DFgQxYn3Q6Z3fu9PqKZUXxAscPedWEBNi/jxI0a6gdPXE/D/KVSHmtrfdrhgGIr7qpjs+beq24kwQFWA2N2Y5RbAW5sNqm+HIUoynsSAl7DiRBwy+4o3dUV9gyMrpH+z9JxO+DlkTb9bc8zo0jpqm3yqEQU3SYXTHV+GBSXtuT7/YsH2V0JTAJfkk0GVEgP62BXOqUzbE+OXEVbCNN60XdA5BKSQMv6+JUgsuR+ayVJpiEOzfiLljV8a7SIzzDFWMwCCdjqSx6S6Swiynnj9J9VzWA+kc5BQ5rb768oXa5qYdRqdirtR4ffElFjTVsXBx4G6dI9OeTUoRHx42SI8+4/VPUIejc0lv8yFYnVhN/FSz9VGofLjyID99tyvQ+0905FzNHqWOtwnNptEMqRQdgJ+Z14YSmnJzCt4bjZP9usiQcMnzgJEmTBUY3w+JN4tmHNqznVXDXpwxMF/zxgwO8pFiJQdK1fMKOeosncJLUkE+yzOgr7xrc9ZzK5pwsBPizicuFh2PZWE7B0uv3P4v+9C9qzXhbrvlQ42VH0RUhsJwZagQ/GXYYRCVKx26LS3YcQhmO9kO/cAZ5Pz1T0R7gL5Wbns2ErITCH+ePpGCQeg+I2F2nJTbk7Hc5dz426KJ+jh5YbReKnRjvypxLsdwPvL0JkY9TxfRQ5qmkeaA/QFsGCUJxdh Eag/Ik9v SS3jKJTuxwVU33l7GokTrkGpGzzRtevpXkSsC0yzxEJQ0uhQTkr34xRtiF77yYNUbvv6mYs2wqvKP+/02mgX4Mp88sBTQJfBE2t4ysdz30mrNglPY7YeXXo0pt0RvwKnX1D5ISmL6DTEaYB0ux+QwPhqJubUgoGRTCmiBh2h5EA/Oh6Bn6M4OqND/qBoozHa/fI2cp489xERgwYFMqJvWnZuCPDqLI27eAm9whpCnZRBiAEMBLWDY5wvnkh8L3l2a1V6F5TxWZP3vU1NV+AL5G0akt9zYgGgNsLwl00Bc66Dbv++LdFgJzvSDhEEDnX7MCMgJwoMJfWozzVdhKSvg456N5AFyYZAObg/Krsmk+YtavSCkoKLDH7OFg+CiN4yCQD0vUn/LvAPOEKhltl6eQ8qgGgJZcL5dF1eTtdYhmG1OKGnAvW9Hc4hIeQVb60biGV06QvBSgiwCehSDIAqM286F+eINP+sGenJViXnb+SKXRNXAC0TcTaBJYx3M7nAV+eLegOpqscuh6ye5AY7scNp3NcaABSQSvvLMM+2aQe8u/IN9xiYm6FXuuNAWYrkrFoRJm9VVOIb9WHuk5wnLV3cEhyjs5CSJjMbTdORHuNjAHCGNuFgq6FlHw/19YKz9rNXn1Jhv7f2ZABgcmIcVShzwoJn2nG0nw2mZZF9T4Clb2ki5d9mYcc6qrLPxB9IhNcZ+1/rCKfwp8RrKSCGJE31SFfRWKPp0SeIkozhqffjOfRXKPXBQmMkprT5LOgmcY6j61TYhxnvwlaQ6unRJIo04nGiYbQNAgvRvJMPyETL9Qiw45yhO0C3gxJBIrQIB7DDfYi1gVzzX+h/aRazyHYmhygV8Z0S+gi5plsEYXZLQ1AJFvFQx9yK0mPZEHcqFENMDAKZoDjhzYLywRuAE1wfwnH88TZRi3rOHeswwssySyLlactQ249E6j70Y3IUtye4r/FZKwvPMoZxayUSz8218UyO+ 7IaIUn7k Pc4DyAQB70s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The new flags parameter allows controlling - Whether or not the units suffix is separated by a space, for compatibility with sort -h - Whether or not to append a B suffix - we're not always printing bytes. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andy Shevchenko Cc: Michael Ellerman Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: "Michael S. Tsirkin" Cc: Jason Wang Cc: "Noralf Trønnes" Cc: Jens Axboe Reviewed-by: Kees Cook --- arch/powerpc/mm/book3s64/radix_pgtable.c | 2 +- drivers/block/virtio_blk.c | 4 ++-- drivers/gpu/drm/gud/gud_drv.c | 2 +- drivers/mmc/core/block.c | 4 ++-- drivers/mtd/spi-nor/debugfs.c | 6 ++--- .../ethernet/chelsio/cxgb4/cxgb4_debugfs.c | 4 ++-- drivers/scsi/sd.c | 8 +++---- include/linux/string_helpers.h | 11 +++++----- lib/string_helpers.c | 22 ++++++++++++++----- lib/test-string_helpers.c | 4 ++-- mm/hugetlb.c | 8 +++---- 11 files changed, 42 insertions(+), 33 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index c6a4ac766b2b..27aa5a083ff0 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -260,7 +260,7 @@ print_mapping(unsigned long start, unsigned long end, unsigned long size, bool e if (end <= start) return; - string_get_size(size, 1, STRING_UNITS_2, buf, sizeof(buf)); + string_get_size(size, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); pr_info("Mapped 0x%016lx-0x%016lx with %s pages%s\n", start, end, buf, exec ? " (exec)" : ""); diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 2bf14a0e2815..94fba7f57079 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -934,9 +934,9 @@ static void virtblk_update_capacity(struct virtio_blk *vblk, bool resize) nblocks = DIV_ROUND_UP_ULL(capacity, queue_logical_block_size(q) >> 9); string_get_size(nblocks, queue_logical_block_size(q), - STRING_UNITS_2, cap_str_2, sizeof(cap_str_2)); + STRING_SIZE_BASE2, cap_str_2, sizeof(cap_str_2)); string_get_size(nblocks, queue_logical_block_size(q), - STRING_UNITS_10, cap_str_10, sizeof(cap_str_10)); + 0, cap_str_10, sizeof(cap_str_10)); dev_notice(&vdev->dev, "[%s] %s%llu %d-byte logical blocks (%s/%s)\n", diff --git a/drivers/gpu/drm/gud/gud_drv.c b/drivers/gpu/drm/gud/gud_drv.c index 9d7bf8ee45f1..6b1748e1f666 100644 --- a/drivers/gpu/drm/gud/gud_drv.c +++ b/drivers/gpu/drm/gud/gud_drv.c @@ -329,7 +329,7 @@ static int gud_stats_debugfs(struct seq_file *m, void *data) struct gud_device *gdrm = to_gud_device(entry->dev); char buf[10]; - string_get_size(gdrm->bulk_len, 1, STRING_UNITS_2, buf, sizeof(buf)); + string_get_size(gdrm->bulk_len, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); seq_printf(m, "Max buffer size: %s\n", buf); seq_printf(m, "Number of errors: %u\n", gdrm->stats_num_errors); diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c index 32d49100dff5..1cded1e9aca4 100644 --- a/drivers/mmc/core/block.c +++ b/drivers/mmc/core/block.c @@ -2557,7 +2557,7 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct mmc_card *card, blk_queue_write_cache(md->queue.queue, cache_enabled, fua_enabled); - string_get_size((u64)size, 512, STRING_UNITS_2, + string_get_size((u64)size, 512, STRING_SIZE_BASE2, cap_str, sizeof(cap_str)); pr_info("%s: %s %s %s%s\n", md->disk->disk_name, mmc_card_id(card), mmc_card_name(card), @@ -2753,7 +2753,7 @@ static int mmc_blk_alloc_rpmb_part(struct mmc_card *card, list_add(&rpmb->node, &md->rpmbs); - string_get_size((u64)size, 512, STRING_UNITS_2, + string_get_size((u64)size, 512, STRING_SIZE_BASE2, cap_str, sizeof(cap_str)); pr_info("%s: %s %s %s, chardev (%d:%d)\n", diff --git a/drivers/mtd/spi-nor/debugfs.c b/drivers/mtd/spi-nor/debugfs.c index 2dbda6b6938a..f6c3ca430df1 100644 --- a/drivers/mtd/spi-nor/debugfs.c +++ b/drivers/mtd/spi-nor/debugfs.c @@ -85,7 +85,7 @@ static int spi_nor_params_show(struct seq_file *s, void *data) seq_printf(s, "name\t\t%s\n", info->name); seq_printf(s, "id\t\t%*ph\n", SPI_NOR_MAX_ID_LEN, nor->id); - string_get_size(params->size, 1, STRING_UNITS_2, buf, sizeof(buf)); + string_get_size(params->size, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); seq_printf(s, "size\t\t%s\n", buf); seq_printf(s, "write size\t%u\n", params->writesize); seq_printf(s, "page size\t%u\n", params->page_size); @@ -130,14 +130,14 @@ static int spi_nor_params_show(struct seq_file *s, void *data) struct spi_nor_erase_type *et = &erase_map->erase_type[i]; if (et->size) { - string_get_size(et->size, 1, STRING_UNITS_2, buf, + string_get_size(et->size, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); seq_printf(s, " %02x (%s) [%d]\n", et->opcode, buf, i); } } if (!(nor->flags & SNOR_F_NO_OP_CHIP_ERASE)) { - string_get_size(params->size, 1, STRING_UNITS_2, buf, sizeof(buf)); + string_get_size(params->size, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); seq_printf(s, " %02x (%s)\n", nor->params->die_erase_opcode, buf); } diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_debugfs.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_debugfs.c index 14e0d989c3ba..7d5fbebd36fc 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_debugfs.c +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_debugfs.c @@ -3457,8 +3457,8 @@ static void mem_region_show(struct seq_file *seq, const char *name, { char buf[40]; - string_get_size((u64)to - from + 1, 1, STRING_UNITS_2, buf, - sizeof(buf)); + string_get_size((u64)to - from + 1, 1, STRING_SIZE_BASE2, + buf, sizeof(buf)); seq_printf(seq, "%-15s %#x-%#x [%s]\n", name, from, to, buf); } diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 0833b3e6aa6e..e23bcb1d1ffa 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -2731,10 +2731,10 @@ sd_print_capacity(struct scsi_disk *sdkp, if (!sdkp->first_scan && old_capacity == sdkp->capacity) return; - string_get_size(sdkp->capacity, sector_size, - STRING_UNITS_2, cap_str_2, sizeof(cap_str_2)); - string_get_size(sdkp->capacity, sector_size, - STRING_UNITS_10, cap_str_10, sizeof(cap_str_10)); + string_get_size(sdkp->capacity, sector_size, STRING_SIZE_BASE2, + cap_str_2, sizeof(cap_str_2)); + string_get_size(sdkp->capacity, sector_size, 0, + cap_str_10, sizeof(cap_str_10)); sd_printk(KERN_NOTICE, sdkp, "%llu %d-byte logical blocks: (%s/%s)\n", diff --git a/include/linux/string_helpers.h b/include/linux/string_helpers.h index 58fb1f90eda5..a54467d891db 100644 --- a/include/linux/string_helpers.h +++ b/include/linux/string_helpers.h @@ -17,14 +17,13 @@ static inline bool string_is_terminated(const char *s, int len) return memchr(s, '\0', len) ? true : false; } -/* Descriptions of the types of units to - * print in */ -enum string_size_units { - STRING_UNITS_10, /* use powers of 10^3 (standard SI) */ - STRING_UNITS_2, /* use binary powers of 2^10 */ +enum string_size_flags { + STRING_SIZE_BASE2 = (1 << 0), + STRING_SIZE_NOSPACE = (1 << 1), + STRING_SIZE_NOBYTES = (1 << 2), }; -int string_get_size(u64 size, u64 blk_size, enum string_size_units units, +int string_get_size(u64 size, u64 blk_size, enum string_size_flags flags, char *buf, int len); int parse_int_array_user(const char __user *from, size_t count, int **array); diff --git a/lib/string_helpers.c b/lib/string_helpers.c index 7713f73e66b0..a5d7d1caed70 100644 --- a/lib/string_helpers.c +++ b/lib/string_helpers.c @@ -19,11 +19,17 @@ #include #include +enum string_size_units { + STRING_UNITS_10, /* use powers of 10^3 (standard SI) */ + STRING_UNITS_2, /* use binary powers of 2^10 */ +}; + /** * string_get_size - get the size in the specified units * @size: The size to be converted in blocks * @blk_size: Size of the block (use 1 for size in bytes) - * @units: units to use (powers of 1000 or 1024) + * @flags: units to use (powers of 1000 or 1024), whether to include space + * separator * @buf: buffer to format to * @len: length of buffer * @@ -34,14 +40,16 @@ * Return value: number of characters of output that would have been written * (which may be greater than len, if output was truncated). */ -int string_get_size(u64 size, u64 blk_size, const enum string_size_units units, +int string_get_size(u64 size, u64 blk_size, enum string_size_flags flags, char *buf, int len) { + enum string_size_units units = flags & flags & STRING_SIZE_BASE2 + ? STRING_UNITS_2 : STRING_UNITS_10; static const char *const units_10[] = { - "B", "kB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB" + "", "k", "M", "G", "T", "P", "E", "Z", "Y" }; static const char *const units_2[] = { - "B", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB", "ZiB", "YiB" + "", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi", "Yi" }; static const char *const *const units_str[] = { [STRING_UNITS_10] = units_10, @@ -128,8 +136,10 @@ int string_get_size(u64 size, u64 blk_size, const enum string_size_units units, else unit = units_str[units][i]; - return snprintf(buf, len, "%u%s %s", (u32)size, - tmp, unit); + return snprintf(buf, len, "%u%s%s%s%s", (u32)size, tmp, + (flags & STRING_SIZE_NOSPACE) ? "" : " ", + unit, + (flags & STRING_SIZE_NOBYTES) ? "" : "B"); } EXPORT_SYMBOL(string_get_size); diff --git a/lib/test-string_helpers.c b/lib/test-string_helpers.c index 9a68849a5d55..0b01ffca96fb 100644 --- a/lib/test-string_helpers.c +++ b/lib/test-string_helpers.c @@ -507,8 +507,8 @@ static __init void __test_string_get_size(const u64 size, const u64 blk_size, char buf10[string_get_size_maxbuf]; char buf2[string_get_size_maxbuf]; - string_get_size(size, blk_size, STRING_UNITS_10, buf10, sizeof(buf10)); - string_get_size(size, blk_size, STRING_UNITS_2, buf2, sizeof(buf2)); + string_get_size(size, blk_size, 0, buf10, sizeof(buf10)); + string_get_size(size, blk_size, STRING_SIZE_BASE2, buf2, sizeof(buf2)); test_string_get_size_check("STRING_UNITS_10", exp_result10, buf10, size, blk_size); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ed1581b670d4..26a8028e4bb7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3475,7 +3475,7 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) if (i == h->max_huge_pages_node[nid]) return; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_SIZE_BASE2, buf, 32); pr_warn("HugeTLB: allocating %u of page size %s failed node%d. Only allocated %lu hugepages.\n", h->max_huge_pages_node[nid], buf, nid, i); h->max_huge_pages -= (h->max_huge_pages_node[nid] - i); @@ -3561,7 +3561,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) if (i < h->max_huge_pages) { char buf[32]; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_SIZE_BASE2, buf, 32); pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", h->max_huge_pages, buf, i); h->max_huge_pages = i; @@ -3607,7 +3607,7 @@ static void __init report_hugepages(void) for_each_hstate(h) { char buf[32]; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_SIZE_BASE2, buf, 32); pr_info("HugeTLB: registered %s page size, pre-allocated %ld pages\n", buf, h->free_huge_pages); pr_info("HugeTLB: %d KiB vmemmap can be freed for a %s page\n", @@ -4527,7 +4527,7 @@ static int __init hugetlb_init(void) char buf[32]; string_get_size(huge_page_size(&default_hstate), - 1, STRING_UNITS_2, buf, 32); + 1, STRING_SIZE_BASE2, buf, 32); pr_warn("HugeTLB: Ignoring hugepages=%lu associated with %s page size\n", default_hstate.max_huge_pages, buf); pr_warn("HugeTLB: Using hugepages=%lu for number of default huge pages\n", From patchwork Mon Feb 12 21:38:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 490AAC4829E for ; Mon, 12 Feb 2024 21:39:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A7086B0074; Mon, 12 Feb 2024 16:39:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 307376B0075; Mon, 12 Feb 2024 16:39:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E3F66B007B; Mon, 12 Feb 2024 16:39:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EBE2C6B0074 for ; Mon, 12 Feb 2024 16:39:39 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8DB28120A6C for ; Mon, 12 Feb 2024 21:39:39 +0000 (UTC) X-FDA: 81784468878.02.D37A962 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id C396240013 for ; Mon, 12 Feb 2024 21:39:37 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rE9i8FyA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3GJDKZQYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3GJDKZQYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773977; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G9b+4SdACUtX02zckFp4W9j//70BE65Et6Qs8xfY7vk=; b=VLNcE7bDFCNDDUbIzBS6cmSq/ifgzorBoiYYzOJZ9oh7hCMCtrbRSzqal39T0h5IxMMHiq oGMmyqzbYPm4mDMfAm1a8k+O+hkb3Ak1yTrVYIcuIoIV3MHdIUk9PcJd8bT2jyPCDKMOLk 2D/S3qYagI7yKHeHkWgCmjOhEue8mho= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rE9i8FyA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3GJDKZQYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3GJDKZQYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773977; a=rsa-sha256; cv=none; b=gaRSvI35o4HzGqMDeS03lIwZE7zKiSinNYdR/KipaYBKT3lH9MO+1UWJv122x3tnjTSbt+ 3CLhCelSPABRiBR7VdP0+o+VV61JmN+KJW9hTqNHMXghJFz+SK60QwMWolJlGg8HjyA76Z JyqRT/P9uIgLtbfow6GznKFT1gyioHw= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60779e8f67fso4370517b3.0 for ; Mon, 12 Feb 2024 13:39:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773977; x=1708378777; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=G9b+4SdACUtX02zckFp4W9j//70BE65Et6Qs8xfY7vk=; b=rE9i8FyAnHg5bwWdhEDRYqoOjoqBAtVzHRnGG/sAZ5AK1j8c5vXsoa3ELQIk+kiuo5 Yov1xpuWcF0DjxXF+Mxpov78pY0AQTkMq0GLPkpUMRNGQu/24NgIVcATILlYRNCklx21 47WZFl+48Ib39L516JJIjlPiMJ51DX/SYhEN4JB0393QjLnMOrBCbfXFt92D0Z2wjvZI 5rySTGK/euKH/hrWKfEjAabz9Q+H3UokJVBMwRiyMADfSvyHVXeKWI/4imzcWmfJ97Sj fqynvnZDC4Oh5ctsFFo4IxOw64cTh/8x23rrGCVPNNxeGMVy+6aqvPHQlD1tjXCCYj88 F6aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773977; x=1708378777; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=G9b+4SdACUtX02zckFp4W9j//70BE65Et6Qs8xfY7vk=; b=RcyQcEXfixwuynBaK5a5JB3P8mFc52QZ2fMG/qjKkM30tOmdrNI4Osqye3yANZp495 y8xp6YBmOXA0xWxBIA6AwUOP2UjPOb8PKNMvzmhrVuCcGMzmybx8GvaMFuyRb+sLDRvK GIVBc7qcOgY1KjoiPDIcbLnQ7SQsVl/+PVl2MOI/23vMhxWiASsFuni4oGq+Z+klGPBf YDBE3fJryU+YNSkVASi1ynkPnch9D7RyBSTbm8i1OLm////bhT92p3tDVnO849K5Aw99 N1GHAdLpGDiFkUSGOwg98MV4DjvCY49Os00F6weFtkEFqSdzk0UADgjSFh+0iOb3z3Js gQHw== X-Forwarded-Encrypted: i=1; AJvYcCX8LbNfp1KlQn9Z40gDHmtw67DbHaeWPT0lIgt3eBLYF9ct8R0TraIj9DPjHbB3/X02nv3GNDYbbP/VYiUMEpCGkA8= X-Gm-Message-State: AOJu0Yw4qwrwrJFBB1xVsrFiPBrA7idfY8buQqV5nSeES/lfMUM/kt+S VlUSyiAa2QRcafZ0LtnkVwTg7kAldrMTg2WExXuolhggMFZWu0UAcOi/BGBCXqqD2iFSL3/A3eL 6pg== X-Google-Smtp-Source: AGHT+IEGxw+pumepe2rycJxKPCLRRtMyemZOKJSP2HsBcIOyAk76oq0qhFPUoU2Rl7QOiTv7xvjwgacHnLI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a0d:d495:0:b0:607:7f86:dc24 with SMTP id w143-20020a0dd495000000b006077f86dc24mr109272ywd.3.1707773976926; Mon, 12 Feb 2024 13:39:36 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:48 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-3-surenb@google.com> Subject: [PATCH v3 02/35] scripts/kallysms: Always include __start and __stop symbols From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C396240013 X-Stat-Signature: 6k3f9pwydx1a7rhp37r6or9gbxxh8t6u X-Rspam-User: X-HE-Tag: 1707773977-735870 X-HE-Meta: U2FsdGVkX18erDtYJbVqGl8X2mfvo02LPaFuZZk+0Rwcfmih6jpWBf2stBcf5m77LVOjnkQFFe2mM/cnTP2N+5lz2WkpvKN0uBdKUNV0tzc2JvTa6jk5/ygSgUSLMxzp8YiBT/7t1M6RYs+UYvOO3MBrW2njkryG8ZvVL3kmjBTg9RFNxDiPr80SfP+r00aIKCteBI8GruBE8U8lGa/SwFn2DYEEenkSCb3SqrEoJJfw7DS3QqzhEqZ1JjAaY5sInbf40y8VLx9vOHQRTES4NfbnM/oXBiqTW2ZdNtqA07cIEndhBpcOTQJ1d4pBHsb9L6eJ+HCWYVVqWtKZjGU0GyoM054cPlV1GCeSylKePTT3jwDrkqEcveWNf2s7uQMZip4MAeskJJsDeHQZgDvwtxq9TxW9Urm/hKrRMUxFlksycuYBNqh5SgJM+sxlrcjMVl+0UTxtmQWpy7oFM4XrqNYg3Ni+KDVUxuL1uUp8/NaDuMxBO2aHJmC7sCCQ6pc4eBSMIjx4tUigr9oJny0vAgLl2ldZdXjwoNvJzq9Wl0Mu5VzilHZSJ3rhLlu5FhGipGUoeKFoDbCZ+vSyVECbqSI/FT8W0TQaaUt6ga7kgf9jE3Sesjf50GrlR4F36Vs27waJK/JW+VPKR3bYsD1KNZ2inOvs3kHGtLg6dDhS9LjWCcNFWOla2CSevQaQYc8L/NTzraSGrsuo5BB8WI77jEDUsvD6HBkCikp7g3QMDZJWjyuymG46G+68tYjgcKEKeEqXcWBWv2sHP7oY8jNAsKDqTHrjF8hFIf66Sxvqc03oFBn5eNXhA20eq5FGbbrqJ5Zu2qd159Fv8CgB4xVnXEHoyxEA5pMtIn1VdvPUKPH4/hidH3gfboQtKLJwOTldlXT9cLBjE/VI83Lg7X3L+qAOlwwpQyDpkF9D590zh+a4aH9xq6/gie7VvjBhGsuEAvgVObsU7xipdo870Yp ZmQjBMBm iDmNVxpfIgO5y9KEbYXimveMCiYWSVVtk1snASR4E1TcJgTj1ouE0Oz8GiUf8PFAqqNZuFfltFszF+B0iwNdWYMozrf0vyaIcS5tRn5AqtGutUI2C9JCgRWI4Z2qFMuHJEhw/3aLIHR5yx3gNm5vUeyIK67WK9qafW0Gm3wsSgdL5m3svFUPpEyNLC8O8Dfh+JYOXccdruJvgHywT1QDBDCtUKUaxSyvfmdn632uBLRG8A6zZbvEZLt5Ppz0mjdwCrNol9eKHg4a2+dKHweTn0K2rSNXDTFdcB9oRMveGSH2xotj/ZHYGZFJUgVmdLeXzfku5ACrjijUAj93nE3SaLIg/jYqvrertEUdrPSR37Rk3n6j9bnl5cif5cosZIAK6SGiiouL0hv+vj4E2Fd6ux1sy99TFKZlKHSZZo+xe5iRkpFx0dk4IeNYGhJXJgk0WSL6pjTKJYKFRie6XGKxNDVuswYAdvSdN/haHr+ZOZvorGqz54YcNTOiIkBWItqgV9BFqtzfo0WrAbpTYAT0+GeHMe5vVbAizOxnapNYEhJAzTXCzTNcPOVMvIQvPtXWZDufBm6cJuMZzfT/oXsxOVYuUAnaZHhwnO6W28vbADpNx+cZLtI4Dlm3aiNcH9hA/HzN2vwLSsb53Usyu0i2/qIUH2KEYKr5iBz8cTSmPhtmCGC9KTitEkOe+h6mYr465wrqvoZC4Om1vYMbrOWLxQu/YBQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet These symbols are used to denote section boundaries: by always including them we can unify loading sections from modules with loading built-in sections, which leads to some significant cleanup. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- scripts/kallsyms.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 653b92f6d4c8..47978efe4797 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -204,6 +204,11 @@ static int symbol_in_range(const struct sym_entry *s, return 0; } +static bool string_starts_with(const char *s, const char *prefix) +{ + return strncmp(s, prefix, strlen(prefix)) == 0; +} + static int symbol_valid(const struct sym_entry *s) { const char *name = sym_name(s); @@ -211,6 +216,14 @@ static int symbol_valid(const struct sym_entry *s) /* if --all-symbols is not specified, then symbols outside the text * and inittext sections are discarded */ if (!all_symbols) { + /* + * Symbols starting with __start and __stop are used to denote + * section boundaries, and should always be included: + */ + if (string_starts_with(name, "__start_") || + string_starts_with(name, "__stop_")) + return 1; + if (symbol_in_range(s, text_ranges, ARRAY_SIZE(text_ranges)) == 0) return 0; From patchwork Mon Feb 12 21:38:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDD1FC4829F for ; Mon, 12 Feb 2024 21:39:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A6826B0075; Mon, 12 Feb 2024 16:39:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 359036B007B; Mon, 12 Feb 2024 16:39:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 187116B007D; Mon, 12 Feb 2024 16:39:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0513F6B0075 for ; Mon, 12 Feb 2024 16:39:42 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D113A1A0ABA for ; Mon, 12 Feb 2024 21:39:41 +0000 (UTC) X-FDA: 81784468962.19.C2997E9 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf07.hostedemail.com (Postfix) with ESMTP id 13F7840006 for ; Mon, 12 Feb 2024 21:39:39 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=P9bQTIQ3; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3GpDKZQYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3GpDKZQYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773980; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DoSimaxvysC1tsWvSL78khESKiClPBlBlt51p8Cfztk=; b=BXLSwq6bi1ARXDI1EVdCb1rOxh2fQWDRbWjiOYisZJl111ol07fYDUpHaiZ/5ta025Ao0u OA9m1yMuclTfuuZgD+fYzgTKtzBoYnZbvg3UTq9SK7mzvMMYQHXObowCTQLdcfOIwKr34W Bf4rYd0PaTKiriVkNQkZRYYbOCzMybU= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=P9bQTIQ3; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3GpDKZQYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3GpDKZQYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773980; a=rsa-sha256; cv=none; b=PSFB7wXtl6+2U9wM2D8zU+1ZXEwhTudz+VH5Bb7NQ3/07JCBgNmUkfI2EMtkwbKGmI/Tee gJ2QDeegTG6vTSHjEmXEUCPqkxjKDENRzYgXY5rRJp7zRodihABMe3yL3Vz88q73MUPsCL /+XpQhZnCwRKQA/dz4OrKXqnTLD+++w= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcc15b03287so603923276.3 for ; Mon, 12 Feb 2024 13:39:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773979; x=1708378779; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DoSimaxvysC1tsWvSL78khESKiClPBlBlt51p8Cfztk=; b=P9bQTIQ3FDvSpBDgLN57vW9qsCVAxgxz0XZt3swb2xXADqMnAWVugmn7f72fZz8OTQ hmhmsht4bu8GRpBUj1mEJb4KlVG/v8aBwiU56j8Oh5YqFgaZ8m5XKNcl02JF+4IEkfyf dp8ohpGOnMrkEpCx1FzE3i/PDEzNcc0hZ+SavbzCRYD40DfUfUf44v7EpFiR9de0YdaU m6ppnyCxxWPK8+qG7acbu4741JbIFVrC68FkhHjEGdQHxoB8aL20JritTyG6sZdjXGyv QkoTK0nUerFRKnE+zP2IugJo4xUWOntSPnxRui4ospQU4moPVysIfc4hsfZfWgEn5p8c HfZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773979; x=1708378779; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DoSimaxvysC1tsWvSL78khESKiClPBlBlt51p8Cfztk=; b=CCPk0J4oDyfht9xcrsk1rK5SRHdnu5vFKvLADf+dIlYz+JpROdc/IcU/goAo+VYVUw ZqxKgYPp/yjXUSoFZ2uYhTK9q7rKZBgyUlxsTz5+tGhseImQOnehpJdAXKSWMmgFF/ii jH6S3W7NmLCWejDaxqpoVLTyAWSB0uV+OS3yUJ3a+fo8/w0ueH9S5wINO2mUq2wAI7T8 UsbhZQ+Twgfpzf2MkAqW+ZrxottWI8TApvdaaorNg+9PXIAWYJdcgV7BKPrQHSrrA53J 8cADT/UWcAJlo5AZ/T3J65xHib4+fTpBDZNzO2nE2eioJzkbKJqXTyp9knbfg/THzmlC ht/A== X-Gm-Message-State: AOJu0YzjcM5Fv+xwMmZdEdo1aMh/G0Hma30BnBewqgIvivCOkIw4/Bhq mMCzvXP9NrGerxVxv0mFnrASr65NTVOpLRUGGcWSlO2VEFjhZ7hmnZ6mzK2YJspRWXc+p87/tDe zSg== X-Google-Smtp-Source: AGHT+IHrMi+yJjCe9l2fRgqlZ6eFfFd+JGnX1MN85wktYNynWGXKMQ9Jg1QyNSwk5gH8+cU1DTAXQiWqo+k= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:1005:b0:dcb:c2c0:b319 with SMTP id w5-20020a056902100500b00dcbc2c0b319mr80854ybt.9.1707773978951; Mon, 12 Feb 2024 13:39:38 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:49 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-4-surenb@google.com> Subject: [PATCH v3 03/35] fs: Convert alloc_inode_sb() to a macro From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Alexander Viro X-Rspamd-Queue-Id: 13F7840006 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 5y7uhe4jim3ctdm365rops83sicj144c X-HE-Tag: 1707773979-755969 X-HE-Meta: U2FsdGVkX1+sitLwXOGqNxXt/VVQyu9B35I3wgDHuiJFjO4wYJSt38CBR3KkNO1wuiqqsxb9vVhZlarW0YLnux1FQ/EEJeNL+IqdYpbAH70RZt6O7ypMVX7V6v4KhJPp8ojAAkLhkdiWW7zRFXukYBeZ+mRxD7vMe7Cwq0dnP2XSVydtQm0UEx8SwL7KBleSPaLjdz6dE54Xobpmg5dNAQgzRMPslAoe5/puTpdpG7lxR95v+qCgElvvkBzGVwd5/r1dCtJf2BqisQete/rkWb8Npasd6f/72WWV/Mv6P+jSazdeWXkaoOASXTuAxwz0kPXE4sQOLC/X7i1pYGJWwekPORrtTgGvbzX/wprVFi46nh+24YiUHjVSHNfUZ12s98E2gNCz+Af4MhYWj2drHFl6jOjz92lbu51Sg04z6T3ihiFSS578PSJieZ60if+uJUx1JXZQ/ves3vEuhRzEOYMB7Rg1Zj2k69pM+JYOwkwV9rVAyZQGIkqNkn+LdW7GRCQOcldevTzEhSvmJJjrkf/YSfDt/gvkbsTtNmEw5IdOsGbCLG5YjJHbOJKPYIboqoJBKW4inDBJKegAe5g7LKUzrccfPDqZja+KKQarG/eBozjHAWU+eQ6CgWIRzPEO8zyEKBoLvoQPNXG1kUo7w4qY7FatLp/vtqMeDUmUxKix9WKJVTSN+ugsyDY6GeX0u0J94G/Tb39udT8YW985RqlADzzGUjulIDEw9x9XZCz/TT7yqwaH2zG2TLtDShU+rbw+Mtyz/K2ocMjgxS0C1PYZsPGslcK/p4Zmi4O46mfwJcuXfSOVg6DdII0o3PIAbTdMuIBEqcdQorWVc464djyTf/m04vcnrAZUr1/5YryMDZ7nbNZLG2VkIo/qwN8WGsZf3vun0X2GvskPuHwyqTSEL5SgEgUXaeLJGC2x2L1XgErndTMMoEvgxMlZvhjxteSMiRaDxIxWdJ3ocsF sM0+OTXX 5o3oEtaUPQIR9aQDw3Ls+z26qIKqPiYOej2P1acrrvXEGKUQeKKYU4jQzLHZajchw3L6HtigODiP+YoxCtFV+7bX3exKpoANrNdm7elwp1WKvHK5GygCaN2MxMyWKblSgGyhb7QUdznvH21+BcBnuIe5IyZJVltTst9F8M/xLT58vygPpkIE6V+y193GjGs22XhRgzGJ1cg+UR2F/khr1iAXRJ8bPZE3ZYR9n+E1UyZWgFTDPpgkoSk3SpdfX51Nq8VMFQIe4cSlcnnhMwD7ZIex+QOGxHqX7IndArv3nsDObhAZui1sxK5ixoXsnn9Z/2DCBFlJhcETyDB/eJTZHgskaruuNlhjabHd1uxIOnEiexnmOiSFAXROJFSUkuUjXuPOQD3gZ7ERwGgwMBEDW8m5XCOGeZQtFUhxjvIVK26N8fgJEvOvL/T8ePM8lxnbpOoYuElTkso1QCSzytvtFvRmst6rINGs/9flajkrYnX8NTfv5L4zfKEW1dYFDaKi0EkttBVeSMZtgzVQoWfLiueoXaTTvx/0YAT8/iVTFNRPnXWUqOMybEN/XKK6u110spnDtXVFdI6ct5KWglo1RWIQt8MX5WAwLbSl74soZ9Ze72ze5eZx3nJR0H4dgADhQNmEC0gsrgTxkrMQxSwTgaNKUxfl+ZasjCwAo1bSBrt/TV7IqJF/eRE2Fth2DwuQP/DjT8tUEjLbl6pYOYp+XE9xI6b9Gi79FuVQty3pLn8n9GzG5bImBv1ex6kVw6b07JSOpuGeT1C5imH6I4arZ5dHNFePRa8iE+Ixk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet We're introducing alloc tagging, which tracks memory allocations by callsite. Converting alloc_inode_sb() to a macro means allocations will be tracked by its caller, which is a bit more useful. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Alexander Viro Reviewed-by: Kees Cook --- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index ed5966a70495..7794b4182bac 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3013,11 +3013,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap, * This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb(struct super_block *sb, struct kmem_cache *cache, gfp_t gfp) -{ - return kmem_cache_alloc_lru(cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb(_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp) extern void __insert_inode_hash(struct inode *, unsigned long hashval); static inline void insert_inode_hash(struct inode *inode) From patchwork Mon Feb 12 21:38:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3F86C4829D for ; Mon, 12 Feb 2024 21:39:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F18B66B007D; Mon, 12 Feb 2024 16:39:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EA08C6B007E; Mon, 12 Feb 2024 16:39:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA54D6B0080; Mon, 12 Feb 2024 16:39:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B69A16B007D for ; Mon, 12 Feb 2024 16:39:43 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 96762A0AA5 for ; Mon, 12 Feb 2024 21:39:43 +0000 (UTC) X-FDA: 81784469046.06.30B752A Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id C19B4100014 for ; Mon, 12 Feb 2024 21:39:41 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZQHX935S; spf=pass (imf05.hostedemail.com: domain of 3HJDKZQYKCJsNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3HJDKZQYKCJsNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773981; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S1akKlUBB/xj+27ZoPrRWZFBMZktx+X63RMMKv/Ae94=; b=hz+oHpSA+vo2hEijl0XAI5aNoWlWwzVH31WMvx5JhgiTqPKzylydxUkoPJ68aXuyHrxTTO 7nznSoBGSbzf3CPQWgNMOnnXXRCygp3ZpGBZ8CEuMBni8tutfCQo0y/uYafZ/CFXOgvHpt tV8HLzI5lp9tc2Qf8e0jM1+i/Kk7bi8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773981; a=rsa-sha256; cv=none; b=0n5HNyzfnVNlF64zpXRu5T6RtdIGWOafFj2INHGw/uOm0u7dH2J4HfT1YvyTxrqmnAhZgs 3ZLiy/qCkWzZtJqCF5A3G8mbfzWdaI0Ckf+JBr2Z22+AndZJESUhaGtIipi5+VAwx7XELE STYa55MijIVhpP5A/jff1xUfT/KZRwI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZQHX935S; spf=pass (imf05.hostedemail.com: domain of 3HJDKZQYKCJsNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3HJDKZQYKCJsNPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5efe82b835fso84258897b3.0 for ; Mon, 12 Feb 2024 13:39:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773981; x=1708378781; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=S1akKlUBB/xj+27ZoPrRWZFBMZktx+X63RMMKv/Ae94=; b=ZQHX935SpBbriGUWBbXkK/kBe3MmafWfy8pGV5020cHmyRY+aKN9JSfTylEjcZsSad 4dzeAARr+tbaT0YxP4qn2BPkPSyjJaWsn45CmiII6yIw6hzY34XLv2AWpFZnbLvgUBQk 82Qoz8TKPlI11i6ECPLSx49hY9ZSSeiJLMRG28nIPj47CYJN/GOGg86iYFWijgBQNuBl z1p8o1fEY39uowy+++5yWTBEEl51Vzr+g67oCySNuQP011C+YwxsXj6xPceDrNXD6flz gMhjrWAdC5nQ5q6N2dWUxwAITIGVDF4GZLPJtq88LfLa/WnXru1vLQjADxCWLb+TTz2H ImHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773981; x=1708378781; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=S1akKlUBB/xj+27ZoPrRWZFBMZktx+X63RMMKv/Ae94=; b=KU53qfo1J/JxiAQYTMNavefMJfcMP+sTiYIORLvx/VORmtqb2jHwLKWnUsLJNpOUBW a92RAIlScUWvKAhZjjQSxx7TkXukPFG6up7n0bdeoi+Hbrn42ONXOWkh5bur9F6/WTI1 MXudv7/1YsaEbnrUmkZ57D4yS5ek8qpj7ZooUmkfjqebLhMiCTwckKyFHA99dFnjlvNe 8cfAZUvl3VAnWd39hw6ECw1C3Oz1hw0avu7x93CQeWSEFz6E6xBVDyGAeMmQAb/A8vz0 YVMXimrVyRxpRCMQ9Vl1Jtrr4hB6u8yvQsGpUP8VELGsSx93UmNuwT1LYDAPrVsZvexV V3IA== X-Gm-Message-State: AOJu0YyhABJSU6nenTqXkmFZBxWd6AJysnt+7IQDbtANmsuKr3sEY3gi 08W+rc+5gUT89it4te3ny11I+yUjh5aTtUSr85daemoCmJvszalOoEy28aOyfnp8+T0HE6+V7af p6Q== X-Google-Smtp-Source: AGHT+IG42uM/J0f/ArRzlXl1uKup/Wz8pDYxmhL3bPE8mSLRBQvJcQ+eddRahpH6ZS5yXp+f3miFrCzzpII= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:690c:a90:b0:5ff:a9fa:2722 with SMTP id ci16-20020a05690c0a9000b005ffa9fa2722mr2210184ywb.3.1707773980866; Mon, 12 Feb 2024 13:39:40 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:50 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-5-surenb@google.com> Subject: [PATCH v3 04/35] mm: enumerate all gfp flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, " =?utf-8?b?UGV0ciBU?= =?utf-8?b?ZXNhxZnDrWs=?= " X-Rspamd-Queue-Id: C19B4100014 X-Rspam-User: X-Stat-Signature: sxdkjzwzsq9znefobkh5qxe5pfai7ssk X-Rspamd-Server: rspam03 X-HE-Tag: 1707773981-753743 X-HE-Meta: U2FsdGVkX1/4ScyNlQIR58MC2Fw8G2A/er49d2YKbB1TXdOOtg51BIvOt1uU8z195JBZDHgdlgtrIzosUP3puxDRJZWbImYQeC7lfPFBi3s5NMGfy8P5bRSF7g+pyZrC8S/Vjbzm5AklI9vTd3AzrkI7MeNMT6qyEhoNJSPP64yrVFFibDfGKutrLZnwpU21QDDGreR9+p0sdZi6wLVRz5jLM2YLYaeaI4dp4m/0zssi0xXi+DZKaw0y8W2ZCKNHl2RVWHER4aA3Wp9V6C6uDRUoWEGdB/MKsUf8XPW7hedYyCXumlqbBu9Vl4Nf5ZQVo2SxDyCoU1V0Cai0C6uWpTZ8mn11/f7OHDo+xU1AJtiLxorPTTsqqrJdhrUXAMwiwpKv2o5D4okVJYiu2HI5rJENNxN1H/GaRlYpQq8BO9rPrj83NZMgcy5E5MWKa5biA8SCqJPiQerB7UB3YUBbpMrUXbO3ceuRYwT+BYgRXnP/J9RH5cEqGaNzzBURAwoyDw5a7XMh20IIeTQHtgy2Yp98S5tMbUtt80APIx5xmA1hFeEbNRGbQkotnXYo+0GUGOvNZEeLqus/qtLCgaCVXfJXTJeUwqp8jNSfiUjio4ErSDrFMpPTsF3nnP8vyGinchSG1qoiNVWf6mgQpm7y/U1qCfE9ikYFrP/JIYrrYUX9mjn/yBCQZBqLuw1LB0m/CSSDqPeWsi8YCZpjwnU82LL9hNDsk1gxk9VvxddPKEAORjnSgI5zykvSCTt2cMVcvD72EJ+/wm6NHGFJbfej23DJl7dJ6nXnQtmSGRbJMqd+753f9nvFa87Qii16iSDH0wWUMVm85t579/olMjTZaU+W+TurdQCEwk8c0jABC5jpFnW5JZIMBoPyR4LuqEu157dzBEerZXzJsqYClmjaNa8DiByBRhdHOxamIe+E08n/eRMjyrRVUeE16esHgzrh+3Y3laDbNvP64rMzX9E A4MDPlNp EgFE2MyfGG2Cbvc+GT7LyFQLPYZCx5a7H8iWkVz7Uv6Bs6Gba/uoDO2M6IffZzsNW98JqCBe2STMn78ZEgEei7qpLWVmRZs8ESqKy16pTqDsYTMW4BpCT8PTD4bH9VIukrfveRaJS5kdgrFfjgkcpkn52fhAQrssWegXpgevLnydGXtq3OdgKx/Qh2e00UjStrJH8l1jdIyzTfA1cwUu3g9L3fWkTOFPGaAmBM4zFk9Is/wo3+e7134TYjAM4kDW8uSJUd1vGUl2dOE4vUwK2gABTZO/BU29qypk0PPO2B73vpUtBIp+oWacZp3SqB5zNRVuY9oRLeDTYeBdakerNdqgRYXVjhmCfa5iOqQbymfOXfBCHlv2spNistNiZuM3LCMQaFhLv5w9FRAkJPylpyyoUA5eRdFiBgHXTgB8Zz5Juo4xZgpY3CcrpoL3jNkFn6BEoESP6l1OOxQemc5QlzotUK9nt0Q1toLIfs6/zgBAflnnU6/mAi6DTHv3n+6LV4tvgHP+EVJdDsMGJlrqhcYWNKxMWFmf+48mlwHWBB7NCoNbIUkUercjNMmHWPFj15ugvn+6zwHhPBiqsSxbHcSIVI7q94pzuILb4Pqe5RnUJ4hDcRpBMhGaj2sWFOM+yezmM/V2+Ld7EsYodwlRTjinmu//08FPdMNCgO9KyQXdZO5g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies __GFP_BITS_SHIFT calculation. Suggested-by: Petr Tesařík Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/gfp_types.h | 90 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 28 deletions(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 1b6053da8754..868c8fb1bbc1 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -21,44 +21,78 @@ typedef unsigned int __bitwise gfp_t; * include/trace/events/mmflags.h and tools/perf/builtin-kmem.c */ +enum { + ___GFP_DMA_BIT, + ___GFP_HIGHMEM_BIT, + ___GFP_DMA32_BIT, + ___GFP_MOVABLE_BIT, + ___GFP_RECLAIMABLE_BIT, + ___GFP_HIGH_BIT, + ___GFP_IO_BIT, + ___GFP_FS_BIT, + ___GFP_ZERO_BIT, + ___GFP_UNUSED_BIT, /* 0x200u unused */ + ___GFP_DIRECT_RECLAIM_BIT, + ___GFP_KSWAPD_RECLAIM_BIT, + ___GFP_WRITE_BIT, + ___GFP_NOWARN_BIT, + ___GFP_RETRY_MAYFAIL_BIT, + ___GFP_NOFAIL_BIT, + ___GFP_NORETRY_BIT, + ___GFP_MEMALLOC_BIT, + ___GFP_COMP_BIT, + ___GFP_NOMEMALLOC_BIT, + ___GFP_HARDWALL_BIT, + ___GFP_THISNODE_BIT, + ___GFP_ACCOUNT_BIT, + ___GFP_ZEROTAGS_BIT, +#ifdef CONFIG_KASAN_HW_TAGS + ___GFP_SKIP_ZERO_BIT, + ___GFP_SKIP_KASAN_BIT, +#endif +#ifdef CONFIG_LOCKDEP + ___GFP_NOLOCKDEP_BIT, +#endif + ___GFP_LAST_BIT +}; + /* Plain integer GFP bitmasks. Do not use this directly. */ -#define ___GFP_DMA 0x01u -#define ___GFP_HIGHMEM 0x02u -#define ___GFP_DMA32 0x04u -#define ___GFP_MOVABLE 0x08u -#define ___GFP_RECLAIMABLE 0x10u -#define ___GFP_HIGH 0x20u -#define ___GFP_IO 0x40u -#define ___GFP_FS 0x80u -#define ___GFP_ZERO 0x100u +#define ___GFP_DMA BIT(___GFP_DMA_BIT) +#define ___GFP_HIGHMEM BIT(___GFP_HIGHMEM_BIT) +#define ___GFP_DMA32 BIT(___GFP_DMA32_BIT) +#define ___GFP_MOVABLE BIT(___GFP_MOVABLE_BIT) +#define ___GFP_RECLAIMABLE BIT(___GFP_RECLAIMABLE_BIT) +#define ___GFP_HIGH BIT(___GFP_HIGH_BIT) +#define ___GFP_IO BIT(___GFP_IO_BIT) +#define ___GFP_FS BIT(___GFP_FS_BIT) +#define ___GFP_ZERO BIT(___GFP_ZERO_BIT) /* 0x200u unused */ -#define ___GFP_DIRECT_RECLAIM 0x400u -#define ___GFP_KSWAPD_RECLAIM 0x800u -#define ___GFP_WRITE 0x1000u -#define ___GFP_NOWARN 0x2000u -#define ___GFP_RETRY_MAYFAIL 0x4000u -#define ___GFP_NOFAIL 0x8000u -#define ___GFP_NORETRY 0x10000u -#define ___GFP_MEMALLOC 0x20000u -#define ___GFP_COMP 0x40000u -#define ___GFP_NOMEMALLOC 0x80000u -#define ___GFP_HARDWALL 0x100000u -#define ___GFP_THISNODE 0x200000u -#define ___GFP_ACCOUNT 0x400000u -#define ___GFP_ZEROTAGS 0x800000u +#define ___GFP_DIRECT_RECLAIM BIT(___GFP_DIRECT_RECLAIM_BIT) +#define ___GFP_KSWAPD_RECLAIM BIT(___GFP_KSWAPD_RECLAIM_BIT) +#define ___GFP_WRITE BIT(___GFP_WRITE_BIT) +#define ___GFP_NOWARN BIT(___GFP_NOWARN_BIT) +#define ___GFP_RETRY_MAYFAIL BIT(___GFP_RETRY_MAYFAIL_BIT) +#define ___GFP_NOFAIL BIT(___GFP_NOFAIL_BIT) +#define ___GFP_NORETRY BIT(___GFP_NORETRY_BIT) +#define ___GFP_MEMALLOC BIT(___GFP_MEMALLOC_BIT) +#define ___GFP_COMP BIT(___GFP_COMP_BIT) +#define ___GFP_NOMEMALLOC BIT(___GFP_NOMEMALLOC_BIT) +#define ___GFP_HARDWALL BIT(___GFP_HARDWALL_BIT) +#define ___GFP_THISNODE BIT(___GFP_THISNODE_BIT) +#define ___GFP_ACCOUNT BIT(___GFP_ACCOUNT_BIT) +#define ___GFP_ZEROTAGS BIT(___GFP_ZEROTAGS_BIT) #ifdef CONFIG_KASAN_HW_TAGS -#define ___GFP_SKIP_ZERO 0x1000000u -#define ___GFP_SKIP_KASAN 0x2000000u +#define ___GFP_SKIP_ZERO BIT(___GFP_SKIP_ZERO_BIT) +#define ___GFP_SKIP_KASAN BIT(___GFP_SKIP_KASAN_BIT) #else #define ___GFP_SKIP_ZERO 0 #define ___GFP_SKIP_KASAN 0 #endif #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x4000000u +#define ___GFP_NOLOCKDEP BIT(___GFP_NOLOCKDEP_BIT) #else #define ___GFP_NOLOCKDEP 0 #endif -/* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -249,7 +283,7 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (26 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT ___GFP_LAST_BIT #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** From patchwork Mon Feb 12 21:38:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30F1DC4829E for ; Mon, 12 Feb 2024 21:39:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 490DA6B0080; Mon, 12 Feb 2024 16:39:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 41A1E6B0081; Mon, 12 Feb 2024 16:39:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21ED66B0082; Mon, 12 Feb 2024 16:39:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0B1B26B0080 for ; Mon, 12 Feb 2024 16:39:46 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D9E7F406A7 for ; Mon, 12 Feb 2024 21:39:45 +0000 (UTC) X-FDA: 81784469130.22.880DD08 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 0DE40120016 for ; Mon, 12 Feb 2024 21:39:43 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RAv33tQ2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3H5DKZQYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3H5DKZQYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773984; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+4MAAfPCe00kOecp4g2Jxa8MlwYPnJB9NakMZqL/K7o=; b=27h1Vgp/kqP0v5zCC3MCNfuqs4OaGrDdHN/u+Wc9MHgAYapqQTo9n+J8Ehg1T21/jkepeE LQYXW9VH32E0qF+BCdJABYKa9BJHCBEll4ATW40m0nwtL5R4U2J+dsf9Tku9jIAboe0vSg VtElZmisGY239CMvInDIXxJnbYOiOyA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RAv33tQ2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3H5DKZQYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3H5DKZQYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773984; a=rsa-sha256; cv=none; b=X8GYL0Ujamt3jTWob9cdQZLW4xpkb/xhtI56enfOjyIJxtpLfPoltTrhtGrBe1tmnKV7Am pzTanMjBGtH3o/NdDeWIeexHFfH8oQ/DsV1yElQcS52kfNqWJgeScxiilqgLKoCYAktc/c mS6WCTJGeeU/ZCrCJDDTvcmp7xhlb14= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b267bf11so4219147276.2 for ; Mon, 12 Feb 2024 13:39:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773983; x=1708378783; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+4MAAfPCe00kOecp4g2Jxa8MlwYPnJB9NakMZqL/K7o=; b=RAv33tQ2llHYcFVLMTQgxMKFxogjIlOCJ3/kWMD6u1OCQdTDa5YdMSLmHEaoqdPOKS Uq6nuGlfTDbAsUmICVTtlUNa3tSzBFCW/1WbPwQIXH4bZlqpCNv7YgG1m7LGKMapr+bh WrR8WHeezv3wIwvdAibHmZrqrIOFv/jnfiS+/TL+Ia59fWFGmboPrTCdoxcuDGI958TT dEe8u82/f/lN9Zts9FicuqjFIxlMsGG1cTkwQ+Kw9GQexaZeRgfXlYprCZmMCcdAibRh epE4Y9+KMcllrTEW2BUqmRcbe/wAJByVPHaAd1a3KMi4UooNieRqZpDUrIdSvfGP+uAE 7agg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773983; x=1708378783; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+4MAAfPCe00kOecp4g2Jxa8MlwYPnJB9NakMZqL/K7o=; b=jOQcd336qJM0oLjZ2XozkyqL1AEbspl4KdlaghJuGb5nH0g0mWjS08szI0i3LHH2Hi K6FdVWW+5iOTjRQLnUxIAisBtMH44pMfrARn0SkAdvp1/4r53OrZWYv6PTazn1hupsRk DfptVBpuJZQoXLU2ZU4fa4Aph1g8E+gD7OCKZcGzz8iwptvlWx69vBUrpPiz4q97Vbqj BK4oU6pgi5sTWvEY6X0H4Wda0PbTA1GzKqSbhCNiL/gSw5HVLda6wSEcKKE4aJNje8it CutgB8pSqZWvbzslKwuz72Rm51XHkrHRPN0OK3ACuZnmI9mfxyC06ilQKDnkdfkIMzP5 Klig== X-Forwarded-Encrypted: i=1; AJvYcCW88ohYf2BFfFg7JeHwjcZLFNQgDgsbN2G2PqtBnNT4nc8Z6Iw7LhYSQQJ+iEz40IPSNgcMlgyQTo5h6fk1+/9pkJg= X-Gm-Message-State: AOJu0Yx3mCRv2ltNMLj8xz9lSDYkoCNMBgdNYo6CfuPv0JzA8AvF37y+ bNw2H1Ws7uipOw3lleidlP4g4gv6wPqq2rYY50IOlH7oL2A+7EgH7kp9YZEk/snxm1XvPKFNpN5 SdQ== X-Google-Smtp-Source: AGHT+IEx2k/hmerJPpr6v0WCuO8HuxZg22OlVEzZgrJnpmE7cctHbOLVeHOYQJ7ufTWaWjLLEbzR787VVlU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a25:abcf:0:b0:dc6:a6e3:ca93 with SMTP id v73-20020a25abcf000000b00dc6a6e3ca93mr292076ybi.10.1707773983086; Mon, 12 Feb 2024 13:39:43 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:51 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-6-surenb@google.com> Subject: [PATCH v3 05/35] mm: introduce slabobj_ext to support slab object extensions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0DE40120016 X-Stat-Signature: echugd1m6r74o5h8tkyxarjqt3c3phj3 X-Rspam-User: X-HE-Tag: 1707773983-54298 X-HE-Meta: U2FsdGVkX1+RJ2lq3rWFwNrwoQHOVVupKRDsaf+M1RjkC4Diq7uEnVw9KK+njIpO057dnxa+Z5JgWK8azIs21EOavPrIoy9YzhYyJy7xRs+srM/a4F3CpuK/dI4ikkX4Fv3wmLKSgJ2f5RcF5dI0b+CeN3qqqQ1XDMLOyTFqy+JcNmy7A7e5FKHuLXjNhl49lQmvop8HKKArxI9fHJzTL2Ddf3FIwTtvd1cJenwlrs6X/nk4q046KiUieRJl4n7V76wM/ZLr2SO0BvExQLA/CDkmq0ThWAsc4KAkJVFnhw5xC+O2KHJTIeeB6lP6lhclyUIZIxZnUefWpmCsi0cDT369yPBAyncOnXBykjIUl+iV3W4jrgvxAYsEfUHRBu4Evb4R/uo3iyhMMncbysibw4WltD6/SgT1pSK4HvMG01GdqU/bx/OiH+/tKXLT1RwqeCEcQHRtjJbywdttMnnirPSlTnW8S53KDXCYduUvyAGoicou9hfMLAqcmU0zYgVvosrcaBoIMAuNmPQhG/KXx3QXWkAnUZZQcpmoQVrgRTBTw1iLGdYChTeWkFjXXQ6+renBgFARfrSFtFr3Afwc0M71g+1bzufWLDPCdK0kgtqvsMcB18UNNpbhuiN+rd7qizAiO+T3FsuCcCLDdT6iAEueTAU5smErxSh5FOinLsLCJjt2ElYdSOrOPr9xzbofkKPgNjXEv48gtDOBrvF1x7o9XsBMz3uJ4Lncd707ZXwiRlCihDTJSlT0zNrYNQO6tZWW1aus1gUOFq4z1/bbg824mWNkb+o3ExPAppvZ1q9pGxaAqyGK6VqWk3nXmmOEhuwHwgpuIwcQ3JdAICObXlQnOiS74PIt23KvMtf/WfrbS6N7R1HvyOFwN7PNxH22zF6CoYURDq48XAfkNncfXoU2lU6L+e/2bN973/kzUwpLaJPO7PUiNN/zmrGEey/oyvKBF1tvWfG7YzHeMTx QwC7ftua DpvS16aOpjg2rAfVxTXdTTii7wqqi3OaPHOSjP49hh8OiijdqJRqPeZa5PRuRzRwLlZEojdJOuw2ZfzLuRlyCcYeHfv66VbcBYS6mfpj8FhYqh5Iy79wS/DelmRuRXKX+sKfcyobUYG+yim/bnQ6/RmCFhotZjO0TO3l+KvjFPGUAy6QJ3fbeSy7Idgf2orVlbZ3dweNpawg+zaavNarSR3Mj97eWDymsWARlbWuvu2mjalJ1shI+9Vx2Z6E42afH/YaKJtQlAWj4ozisx+35VdlLoGvyui4qFXbSw7UFOm5z9kJ8iODCdv5fEpxtxvoOx9w8b9LS+U7SH2rT5RKITBCiGwGQj5ekqX0GQTkTTBRCA9ItcgI0AzxBbXaz+eKT6qqufJ28SsVGdvbkn90nqPU6Ihdb+lf4cHy5KwVjI0TWRF7xIK9n81UqLdQAp80coU6O0BH5o2OHeypccCDrkjdfuV0Imw6s0DhUQWObaKYcyqRoNdUjKgYUDqSA9vbtdoYHrsMve0f3Q2PEH0RgPZEfbLkxX5lKbxTBkQHDpadSfc4zpKTGuHsXxkOnMiqsM6aJr00vLZ9DoHXdMdVgIg1wMGLcibtEl62b5yOd3dCC3FtU8iuO9FTU9NGDTEI2K2o6Gb/GatYRI5X6Q6q/C81b3aN0tXv9txrvFgqW/q0QGiXmvf/cYvspHWZVEtZwTzkAh5OlX0vRc3w1O49fsMd2pMl2LJXMQ8cp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently slab pages can store only vectors of obj_cgroup pointers in page->memcg_data. Introduce slabobj_ext structure to allow more data to be stored for each slab object. Wrap obj_cgroup into slabobj_ext to support current functionality while allowing to extend slabobj_ext in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/memcontrol.h | 20 ++++++--- include/linux/mm_types.h | 4 +- init/Kconfig | 4 ++ mm/kfence/core.c | 14 +++--- mm/kfence/kfence.h | 4 +- mm/memcontrol.c | 56 +++-------------------- mm/page_owner.c | 2 +- mm/slab.h | 92 +++++++++++++++++++++++++++++--------- mm/slab_common.c | 48 ++++++++++++++++++++ mm/slub.c | 64 +++++++++++++------------- 10 files changed, 189 insertions(+), 119 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 20ff87f8e001..eb1dc181e412 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -348,8 +348,8 @@ struct mem_cgroup { extern struct mem_cgroup *root_mem_cgroup; enum page_memcg_data_flags { - /* page->memcg_data is a pointer to an objcgs vector */ - MEMCG_DATA_OBJCGS = (1UL << 0), + /* page->memcg_data is a pointer to an slabobj_ext vector */ + MEMCG_DATA_OBJEXTS = (1UL << 0), /* page has been accounted as a non-slab kernel page */ MEMCG_DATA_KMEM = (1UL << 1), /* the next bit after the last actual flag */ @@ -387,7 +387,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -408,7 +408,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -505,7 +505,7 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) */ unsigned long memcg_data = READ_ONCE(folio->memcg_data); - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) return NULL; if (memcg_data & MEMCG_DATA_KMEM) { @@ -551,7 +551,7 @@ static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *ob static inline bool folio_memcg_kmem(struct folio *folio) { VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); - VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJEXTS, folio); return folio->memcg_data & MEMCG_DATA_KMEM; } @@ -1633,6 +1633,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, } #endif /* CONFIG_MEMCG */ +/* + * Extended information for slab objects stored as an array in page->memcg_data + * if MEMCG_DATA_OBJEXTS is set. + */ +struct slabobj_ext { + struct obj_cgroup *objcg; +} __aligned(8); + static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) { __mod_lruvec_kmem_state(p, idx, 1); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8b611e13153e..9ff97f4e74c5 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -169,7 +169,7 @@ struct page { /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif @@ -306,7 +306,7 @@ struct folio { }; atomic_t _mapcount; atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif #if defined(WANT_PAGE_VIRTUAL) diff --git a/init/Kconfig b/init/Kconfig index deda3d14135b..8ca5285108be 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -949,10 +949,14 @@ config CGROUP_FAVOR_DYNMODS Say N if unsure. +config SLAB_OBJ_EXT + bool + config MEMCG bool "Memory controller" select PAGE_COUNTER select EVENTFD + select SLAB_OBJ_EXT help Provides control over the memory footprint of tasks in a cgroup. diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 8350f5c06f2e..964b8482275b 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -595,9 +595,9 @@ static unsigned long kfence_init_pool(void) continue; __folio_set_slab(slab_folio(slab)); -#ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long)&kfence_metadata_init[i / 2 - 1].objcg | - MEMCG_DATA_OBJCGS; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts | + MEMCG_DATA_OBJEXTS; #endif } @@ -645,8 +645,8 @@ static unsigned long kfence_init_pool(void) if (!i || (i % 2)) continue; -#ifdef CONFIG_MEMCG - slab->memcg_data = 0; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = 0; #endif __folio_clear_slab(slab_folio(slab)); } @@ -1139,8 +1139,8 @@ void __kfence_free(void *addr) { struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr); -#ifdef CONFIG_MEMCG - KFENCE_WARN_ON(meta->objcg); +#ifdef CONFIG_MEMCG_KMEM + KFENCE_WARN_ON(meta->obj_exts.objcg); #endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index f46fbb03062b..084f5f36e8e7 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -97,8 +97,8 @@ struct kfence_metadata { struct kfence_track free_track; /* For updating alloc_covered on frees. */ u32 alloc_stack_hash; -#ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; +#ifdef CONFIG_MEMCG_KMEM + struct slabobj_ext obj_exts; #endif }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1ed40f9d3a27..7021639d2a6f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2977,13 +2977,6 @@ void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg) } #ifdef CONFIG_MEMCG_KMEM -/* - * The allocated objcg pointers array is not accounted directly. - * Moreover, it should not come from DMA buffer and is not readily - * reclaimable. So those GFP bits should be masked off. - */ -#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | \ - __GFP_ACCOUNT | __GFP_NOFAIL) /* * mod_objcg_mlstate() may be called with irq enabled, so @@ -3003,62 +2996,27 @@ static inline void mod_objcg_mlstate(struct obj_cgroup *objcg, rcu_read_unlock(); } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab) -{ - unsigned int objects = objs_per_slab(s, slab); - unsigned long memcg_data; - void *vec; - - gfp &= ~OBJCGS_CLEAR_MASK; - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - slab_nid(slab)); - if (!vec) - return -ENOMEM; - - memcg_data = (unsigned long) vec | MEMCG_DATA_OBJCGS; - if (new_slab) { - /* - * If the slab is brand new and nobody can yet access its - * memcg_data, no synchronization is required and memcg_data can - * be simply assigned. - */ - slab->memcg_data = memcg_data; - } else if (cmpxchg(&slab->memcg_data, 0, memcg_data)) { - /* - * If the slab is already in use, somebody can allocate and - * assign obj_cgroups in parallel. In this case the existing - * objcg vector should be reused. - */ - kfree(vec); - return 0; - } - - kmemleak_not_leak(vec); - return 0; -} - static __always_inline struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) { /* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in - * slab->memcg_data. + * slab->obj_exts. */ if (folio_test_slab(folio)) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; struct slab *slab; unsigned int off; slab = folio_slab(folio); - objcgs = slab_objcgs(slab); - if (!objcgs) + obj_exts = slab_obj_exts(slab); + if (!obj_exts) return NULL; off = obj_to_index(slab->slab_cache, slab, p); - if (objcgs[off]) - return obj_cgroup_memcg(objcgs[off]); + if (obj_exts[off].objcg) + return obj_cgroup_memcg(obj_exts[off].objcg); return NULL; } @@ -3066,7 +3024,7 @@ struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) /* * folio_memcg_check() is used here, because in theory we can encounter * a folio where the slab flag has been cleared already, but - * slab->memcg_data has not been freed yet + * slab->obj_exts has not been freed yet * folio_memcg_check() will guarantee that a proper memory * cgroup pointer or NULL will be returned. */ diff --git a/mm/page_owner.c b/mm/page_owner.c index 5634e5d890f8..262aa7d25f40 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -377,7 +377,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, if (!memcg_data) goto out_unlock; - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) ret += scnprintf(kbuf + ret, count - ret, "Slab cache page\n"); diff --git a/mm/slab.h b/mm/slab.h index 54deeb0428c6..436a126486b5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -87,8 +87,8 @@ struct slab { unsigned int __unused; atomic_t __page_refcount; -#ifdef CONFIG_MEMCG - unsigned long memcg_data; +#ifdef CONFIG_SLAB_OBJ_EXT + unsigned long obj_exts; #endif }; @@ -97,8 +97,8 @@ struct slab { SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ SLAB_MATCH(_refcount, __page_refcount); -#ifdef CONFIG_MEMCG -SLAB_MATCH(memcg_data, memcg_data); +#ifdef CONFIG_SLAB_OBJ_EXT +SLAB_MATCH(memcg_data, obj_exts); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <= sizeof(struct page)); @@ -541,42 +541,90 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } -#ifdef CONFIG_MEMCG_KMEM +#ifdef CONFIG_SLAB_OBJ_EXT + /* - * slab_objcgs - get the object cgroups vector associated with a slab + * slab_obj_exts - get the pointer to the slab object extension vector + * associated with a slab. * @slab: a pointer to the slab struct * - * Returns a pointer to the object cgroups vector associated with the slab, + * Returns a pointer to the object extension vector associated with the slab, * or NULL if no such vector has been associated yet. */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { - unsigned long memcg_data = READ_ONCE(slab->memcg_data); + unsigned long obj_exts = READ_ONCE(slab->obj_exts); - VM_BUG_ON_PAGE(memcg_data && !(memcg_data & MEMCG_DATA_OBJCGS), +#ifdef CONFIG_MEMCG + VM_BUG_ON_PAGE(obj_exts && !(obj_exts & MEMCG_DATA_OBJEXTS), slab_page(slab)); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, slab_page(slab)); + VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); +#else + return (struct slabobj_ext *)obj_exts; +#endif } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab); -void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, - enum node_stat_item idx, int nr); -#else /* CONFIG_MEMCG_KMEM */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab); + +static inline bool need_slab_obj_ext(void) +{ + /* + * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally + * inside memcg_slab_post_alloc_hook. No other users for now. + */ + return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + struct slab *slab; + + if (!p) + return NULL; + + if (!need_slab_obj_ext()) + return NULL; + + slab = virt_to_slab(p); + if (!slab_obj_exts(slab) && + WARN(alloc_slab_obj_exts(slab, s, flags, false), + "%s, %s: Failed to create slab extension vector!\n", + __func__, s->name)) + return NULL; + + return slab_obj_exts(slab) + obj_to_index(s, slab, p); +} + +#else /* CONFIG_SLAB_OBJ_EXT */ + +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { return NULL; } -static inline int memcg_alloc_slab_cgroups(struct slab *slab, - struct kmem_cache *s, gfp_t gfp, - bool new_slab) +static inline int alloc_slab_obj_exts(struct slab *slab, + struct kmem_cache *s, gfp_t gfp, + bool new_slab) { return 0; } -#endif /* CONFIG_MEMCG_KMEM */ + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + return NULL; +} + +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM +void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, + enum node_stat_item idx, int nr); +#endif size_t __ksize(const void *objp); diff --git a/mm/slab_common.c b/mm/slab_common.c index 238293b1dbe1..6bfa1810da5e 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -201,6 +201,54 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, return NULL; } +#ifdef CONFIG_SLAB_OBJ_EXT +/* + * The allocated objcg pointers array is not accounted directly. + * Moreover, it should not come from DMA buffer and is not readily + * reclaimable. So those GFP bits should be masked off. + */ +#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | \ + __GFP_ACCOUNT | __GFP_NOFAIL) + +int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab) +{ + unsigned int objects = objs_per_slab(s, slab); + unsigned long obj_exts; + void *vec; + + gfp &= ~OBJCGS_CLEAR_MASK; + vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, + slab_nid(slab)); + if (!vec) + return -ENOMEM; + + obj_exts = (unsigned long)vec; +#ifdef CONFIG_MEMCG + obj_exts |= MEMCG_DATA_OBJEXTS; +#endif + if (new_slab) { + /* + * If the slab is brand new and nobody can yet access its + * obj_exts, no synchronization is required and obj_exts can + * be simply assigned. + */ + slab->obj_exts = obj_exts; + } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + /* + * If the slab is already in use, somebody can allocate and + * assign slabobj_exts in parallel. In this case the existing + * objcg vector should be reused. + */ + kfree(vec); + return 0; + } + + kmemleak_not_leak(vec); + return 0; +} +#endif /* CONFIG_SLAB_OBJ_EXT */ + static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset, diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..1eb1050814aa 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -683,10 +683,10 @@ static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *sla if (s->flags & __CMPXCHG_DOUBLE) { ret = __update_freelist_fast(slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); } else { ret = __update_freelist_slow(slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); } if (likely(ret)) return true; @@ -710,13 +710,13 @@ static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, if (s->flags & __CMPXCHG_DOUBLE) { ret = __update_freelist_fast(slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); } else { unsigned long flags; local_irq_save(flags); ret = __update_freelist_slow(slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); local_irq_restore(flags); } if (likely(ret)) @@ -1881,13 +1881,25 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; } -#ifdef CONFIG_MEMCG_KMEM -static inline void memcg_free_slab_cgroups(struct slab *slab) +#ifdef CONFIG_SLAB_OBJ_EXT +static inline void free_slab_obj_exts(struct slab *slab) +{ + struct slabobj_ext *obj_exts; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + kfree(obj_exts); + slab->obj_exts = 0; +} +#else +static inline void free_slab_obj_exts(struct slab *slab) { - kfree(slab_objcgs(slab)); - slab->memcg_data = 0; } +#endif +#ifdef CONFIG_MEMCG_KMEM static inline size_t obj_full_size(struct kmem_cache *s) { /* @@ -1966,15 +1978,15 @@ static void __memcg_slab_post_alloc_hook(struct kmem_cache *s, if (likely(p[i])) { slab = virt_to_slab(p[i]); - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { + if (!slab_obj_exts(slab) && + alloc_slab_obj_exts(slab, s, flags, false)) { obj_cgroup_uncharge(objcg, obj_full_size(s)); continue; } off = obj_to_index(s, slab, p[i]); obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; + slab_obj_exts(slab)[off].objcg = objcg; mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), obj_full_size(s)); } else { @@ -1995,18 +2007,18 @@ void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects, - struct obj_cgroup **objcgs) + struct slabobj_ext *obj_exts) { for (int i = 0; i < objects; i++) { struct obj_cgroup *objcg; unsigned int off; off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; + objcg = obj_exts[off].objcg; if (!objcg) continue; - objcgs[off] = NULL; + obj_exts[off].objcg = NULL; obj_cgroup_uncharge(objcg, obj_full_size(s)); mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), -obj_full_size(s)); @@ -2018,16 +2030,16 @@ static __fastpath_inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; if (!memcg_kmem_online()) return; - objcgs = slab_objcgs(slab); - if (likely(!objcgs)) + obj_exts = slab_obj_exts(slab); + if (likely(!obj_exts)) return; - __memcg_slab_free_hook(s, slab, p, objects, objcgs); + __memcg_slab_free_hook(s, slab, p, objects, obj_exts); } static inline @@ -2038,15 +2050,6 @@ void memcg_slab_alloc_error_hook(struct kmem_cache *s, int objects, obj_cgroup_uncharge(objcg, objects * obj_full_size(s)); } #else /* CONFIG_MEMCG_KMEM */ -static inline struct mem_cgroup *memcg_from_slab_obj(void *ptr) -{ - return NULL; -} - -static inline void memcg_free_slab_cgroups(struct slab *slab) -{ -} - static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, struct obj_cgroup **objcgp, @@ -2314,7 +2317,7 @@ static __always_inline void account_slab(struct slab *slab, int order, struct kmem_cache *s, gfp_t gfp) { if (memcg_kmem_online() && (s->flags & SLAB_ACCOUNT)) - memcg_alloc_slab_cgroups(slab, s, gfp, true); + alloc_slab_obj_exts(slab, s, gfp, true); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), PAGE_SIZE << order); @@ -2323,8 +2326,7 @@ static __always_inline void account_slab(struct slab *slab, int order, static __always_inline void unaccount_slab(struct slab *slab, int order, struct kmem_cache *s) { - if (memcg_kmem_online()) - memcg_free_slab_cgroups(slab); + free_slab_obj_exts(slab); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), -(PAGE_SIZE << order)); @@ -3775,6 +3777,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, unsigned int orig_size) { unsigned int zero_size = s->object_size; + struct slabobj_ext *obj_exts; bool kasan_init = init; size_t i; gfp_t init_flags = flags & gfp_allowed_mask; @@ -3817,6 +3820,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, kmemleak_alloc_recursive(p[i], s->object_size, 1, s->flags, init_flags); kmsan_slab_alloc(s, p[i], init_flags); + obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); From patchwork Mon Feb 12 21:38:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554044 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EE20C4829D for ; Mon, 12 Feb 2024 21:39:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83AF16B0082; Mon, 12 Feb 2024 16:39:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7747A6B0083; Mon, 12 Feb 2024 16:39:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CAD26B0085; Mon, 12 Feb 2024 16:39:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 442026B0082 for ; Mon, 12 Feb 2024 16:39:48 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 19AE240B16 for ; Mon, 12 Feb 2024 21:39:48 +0000 (UTC) X-FDA: 81784469256.09.ADB1309 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 5FBF5180005 for ; Mon, 12 Feb 2024 21:39:46 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=drlIHIKJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3IZDKZQYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3IZDKZQYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773986; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NFEoAydGNCIakLM+oYHcpcvSKcGKjy30qzf4+mOY1ag=; b=57HwhQyffsjjg9Z6p2UbRywwwSOoYyjyY4N7jtrJa6/a8CpzJYnWXEG/H6rkKRXZ1qjbTW 5xFOpdV9lqZ3bqNZNmLNFFZ5YChbpdUf1aUUCSQNTtBn5LRhMD/FqrplbmzomvLCMDpbnO 64YmfBU0R57rk1EVxR0bOBSKWIi8niM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=drlIHIKJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3IZDKZQYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3IZDKZQYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773986; a=rsa-sha256; cv=none; b=RPh1Z+z7fInwxb4qbhaG8J0UYw3DtY/i3zMYynbJqDYY4VHNkSf/QEyPCfolhJYSmVS1ND Wdp8mGoBQM6080C3TfxASfTQG49Br+OslHA/Qg9glBuZV/cE6eAcAJxEo8ewL3dWUbQbr/ mocWjIlMpDQyeCJaTaUfSYlyx8yILEs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6077ca6e1e1so4673997b3.2 for ; Mon, 12 Feb 2024 13:39:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773985; x=1708378785; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NFEoAydGNCIakLM+oYHcpcvSKcGKjy30qzf4+mOY1ag=; b=drlIHIKJ1KlCE4phsUHegso0p4FEd4mqaDC/GCFnk1ZIXrWYUEU9919BJue6Xs5V+F WosgaQytDyE7o9WTBWYmxeZjAVjXoqSSF/SRm++qaCQShsmIkhv6WZTrj/No3cfNQw9E aFSAdJYzQTDe5aUEhcsNnW5huC+kkNGyWIJCxEZH4pCzF8v4waEeirQexnXShpw328S8 z/W9S59RljjbYgdWvChDarzdi/5Fm7gKqc1k1GxFFwEFoErz/68o7sfqDjdlF619xOF6 u0FS7ybsyjtBQqoWM3g4ElUVFRGHu9Uwkahp1KSWtw0jFtHQk3eb9dFYVOOi5k4Q46qR PTaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773985; x=1708378785; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NFEoAydGNCIakLM+oYHcpcvSKcGKjy30qzf4+mOY1ag=; b=hGXjlHqnygyAksUQGP1gTaslzapn4XbsUIav7RHYP1tgDO429EBPCK//T3rAUL67ms rWTL6jkAwJ3hBVypMj/N4F+7v3Nj0cqFavr5T52+CIjIFjlxsW8xFMD+UIdseHqDEJLO x+cRPMmkRrIWXFVIW7ygu3Yf2sRbrL0WceIyQKk1DJngvyPfyAdUoRN0GxEOTj2JIrqs dhp1puQoz1OuKHn5X8SZ/AawQqHvYzfll/YEwtlyqv0VFFnaFQLP1ocbKwcfpEjhlV/U tWbtxBJcnvm1V+6JLXXw+bw6FI9zTPaj7N0NzHI/cHrbQmsWeY0APmGk8LmXytBWSmkV 1MZg== X-Gm-Message-State: AOJu0YzCcGwBDYdqjr7tJaoCElKn/6mV4UzPTHKKBwfTIslInPDnkwQ4 vG0qsltL7TZjBN12J/m7cU+QM7abG5p3Hu3q4J+XzgkscOoV+oLTI5/e0gKfTqrid3hVRrGBL+r chg== X-Google-Smtp-Source: AGHT+IHWeYeSfs9VSWCME9Zm5O2fUep/uRibBNlRqA9VaQU1CvkyHMwI5g7LZivFcfiaDsVBzBtyLk+b/OQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a81:a003:0:b0:607:8294:7631 with SMTP id x3-20020a81a003000000b0060782947631mr42744ywg.10.1707773985395; Mon, 12 Feb 2024 13:39:45 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:52 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-7-surenb@google.com> Subject: [PATCH v3 06/35] mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 5FBF5180005 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: swzckcw3nsfh6u8d4cbpof9k3nd3cpph X-HE-Tag: 1707773986-50095 X-HE-Meta: U2FsdGVkX1+DwJ9V/up9QHu3aGulSesaaTGFWVU3bipPXoB4slEyt4krrVmZUlFtj0/50DwzjPGqnFXVRl+9wuRc5IpxLHFX7/rn/05rRb5NSM2GPl42XH3I2/V5nrlsd1StdFfxZ95BjEM6+z5kC+fxzuSgOetLr3NE6CNhvacYigwnqKQammyuL/4DVpz3PvbMHxln6t4H+JdTPGkiAM9drgi9JgS6pxEhZhJRFNmQdFORCjWd0p5uuT98CRatyRhd1eyk30/VDjyn/f+CXWyvqeOX8d1dl7jsXQ9eTt8sQIVmICChthJ7aVYmK6zPwX6HCes3k9qPxf0VmRG7Rzm0m5trUl2Ge8HXsWDgavx80Z87jWv2Qpu2EciVI2t9nxZtLTS0kdIN5BSgd5MjlouCvqEcdoO5sCy7muzglGhU27/sFMAznQlEnJxE7ER3bGhhXrHAujE/5iZLcpYPblRJgsvoHBHP76VOU96e42uGfKzhUarS7ITi5WSIebyZT7CajUG5T5QFqvbIN89bKs/ZJHjKteDHWIRG4gUJkElDpxJ7j3chf8hblGgXVoZ0G8tton5HiYxdRblWOaefQx4uIK+hbS6CL/eCBMpQmCp6FEQH7EDvk5AbX9f9CX/opnv9wC0lf7Y1R7LC1LCswoaMBuZsUnR99FDla/MKu8B1NTSwWbSJngIWqtn5ZLqFfDJB8JRWDWBoSKYdi9UNUlXuRKgVuJlywegEhOyCXoxL6zGYEq9HdIPs6W9huSEvJf4PKjBWxXAFKBBPpMBE4Nc2br3h/B2PYTDC7LKJVeQt+T3cTh7QAuIY8lnfR3kaNXI5BA4eyWzLQPgx2awDOYD/O4RccxdvWuq4roN9k7+rrjIXZEN10fRQK+3kzJ3hRlPFwcrI7M5GCZWaaC8Ryh3HCollgkFRRWHccbS//wOoNhUsfzTYghTE69aN5Mp5NXtThomzuICP4fNR1vW mAPJLyfd iVmGWh8MaTd5yu6Nuv70ezmHJCI7+E14QUsY3TfcTk8T2s128FU3j4pk54M1vIwjN35dLMNaa6HdPfNiCTZ8spjfqpyhyfz5H0lFlrNSUInAE6SfyBFx3+o52UU3yfVKylsadDcu4bA9dPERIg6QQQsQgGoLTP5f1nHsSH87tfy5W98L8HYwWEq15pkABrsNnu2yX5GZ9B/Ra187jQtBER5N6SKdxtaefUjqMZMpebkmdaa8SwMrVGsfJaymhqux+L+p5D2085S5+BHsIYR8XfVC8v/64MGUdx9AF6Sitn/T5YUdxZUevSpKgqZnX9K3DiQhVWVtZHjXINWA/7r19MWJTtgAML8trlK6fJs06WE6xdeUnzUWZWTQZ6FejEoW4UMFCfPhBr6m2RI9GQuCZ/zXhFBV4jNSp8yyb/9xdj8vbOeCDqzqGejEHCiykkFZFQxEY5dDgvOU1FYdoRps6SBxnDrCqH07iLqJ6DbONpMe9gZdFqFqQhGFlyt/xVylJqJmgHuLnXRVYV4fvx3Ug0xbMaHlC/CoH9JeFrChSCXXQpuiN2iNCs70XdwZnjC1Ji7dbnJ0fyvL3iAmPdf0TF0Hmen2Q2Fj5+VluYYdanb/7Pc2SbQ6jnw786BosEdJBxbwYKI21ZpyJAoYlOHtLBIHQNCEEX0swFvOXLoTPDLzQNDUZryaUywEdk2wNZo4iXI1A7psW4+7qUT0zNzTBdV2+JtGDb+DNK1Bp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce __GFP_NO_OBJ_EXT flag in order to prevent recursive allocations when allocating slabobj_ext on a slab. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/gfp_types.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 868c8fb1bbc1..e36e168d8cfd 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -52,6 +52,9 @@ enum { #endif #ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, +#endif +#ifdef CONFIG_SLAB_OBJ_EXT + ___GFP_NO_OBJ_EXT_BIT, #endif ___GFP_LAST_BIT }; @@ -93,6 +96,11 @@ enum { #else #define ___GFP_NOLOCKDEP 0 #endif +#ifdef CONFIG_SLAB_OBJ_EXT +#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#else +#define ___GFP_NO_OBJ_EXT 0 +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -133,12 +141,15 @@ enum { * node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. + * + * %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) #define __GFP_WRITE ((__force gfp_t)___GFP_WRITE) #define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) #define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE) #define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT) /** * DOC: Watermark modifiers From patchwork Mon Feb 12 21:38:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BACD5C4829D for ; Mon, 12 Feb 2024 21:39:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F4946B0085; Mon, 12 Feb 2024 16:39:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A3426B0087; Mon, 12 Feb 2024 16:39:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F6056B0088; Mon, 12 Feb 2024 16:39:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0309B6B0085 for ; Mon, 12 Feb 2024 16:39:51 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8A91FA20A8 for ; Mon, 12 Feb 2024 21:39:50 +0000 (UTC) X-FDA: 81784469340.22.D5CEA20 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id A5DD3100013 for ; Mon, 12 Feb 2024 21:39:48 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vjqeCmmn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3I5DKZQYKCKIUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3I5DKZQYKCKIUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773988; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QPRNF15NS/yHs+nflQ9JCJcKpeGLPrq2C4Bm4JvXiOY=; b=WWXigp21vMkGMcYnhax+UgZJ7CblVsNpRHuZe+m+lQIl3T7fKqpg8S6Xnmn6s5RSbZsAOz RLLyPXZ4vROUGn3uEcdWSV4cNoLVU7CeE6y/ViNi3UQVuSrlfs7iQqEfYeOZDXEMbYWW5E 7oy1cdJPQmxt3KszlSuI8XWmgtHQo5s= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vjqeCmmn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3I5DKZQYKCKIUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3I5DKZQYKCKIUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773988; a=rsa-sha256; cv=none; b=1Cu2jGfaXxzDllMoTMdTUPX9y7Jl1yHHo2usLJDyx2SR8YJbKwgf3yPZsICmJlMs71LuEQ 9DoRp51ouEj5P5oNts8ZpYE3IWvilBDzTsRgstigxsf8vrmdHBALIi8prbV2JV7fJI+Fja 6CTutunOR7zJAmUHW7Rcp6nJoLMNH8k= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-604ac3c75ebso68374597b3.0 for ; Mon, 12 Feb 2024 13:39:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773988; x=1708378788; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QPRNF15NS/yHs+nflQ9JCJcKpeGLPrq2C4Bm4JvXiOY=; b=vjqeCmmn7AGhsMkhYehoZGKIwMwOBY1bfDV+6bIbzIuXJi1Vuy1/ZCQmvam5y1ZBEm 8dtLnDwo6m8y2UScswE9NGJpHCFQ2+kHK6Wxsjy8C1hJvBdruxD8MVYCBXbBtJpcS39w jm/6NrxhVpDdJxr6mNWD+yhNxuOF/iOE55NLUgDBvDZdcs4G0EAT5P6Ti59F4b46FDPO CMXGuerwm0CLokWf43PJu6dugzofqhhsufmShwXozu8IkS04x6fBSF6SjnW4cva89uio JMY+yqH5vasacP9Vmk5p37EB8ffONvBhAhY5tCyfj4Zj7XCTO9iScPdUd0xBxxNbzxsC WGEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773988; x=1708378788; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QPRNF15NS/yHs+nflQ9JCJcKpeGLPrq2C4Bm4JvXiOY=; b=Bd6zHMLoxLViKCLjdSrxR0DvrhaJB7o712AVcULFQ6sN51yBBx708Kl4yuBatYVw3U wja5ZgkVg0zLCCImC0QBWACgYZgi5ludUiaD+QUqM93AI88u47yTsW1rekktD/Vrlzk8 +rwTXxVrfkl6jMjTAAx4pHezKjDnjj9NAb80Ew6QQe1u1Q08HH5mevCmJk3xUIuE19mv NkpwM05m3c9ZbKYe92fEOjKHFAHEnogi0OdSizKsbWxbkB+g/Xpl4OvIXS431ErIoI/F 7qNYe+B5ycM63f9G01fa3TWd+Rd1jZgR/M+9445y1hiXM/DfEoasBfyowo+cHD0RKJlG 68Xw== X-Forwarded-Encrypted: i=1; AJvYcCXtHd/XvHnlqy9BJR0/cul/7REQhrMfPUSEqXix5URD+HF5vtpFxqwZ+LzqiC11Zsh4jV9yzmj/P1kltU0Yjv3r1Sc= X-Gm-Message-State: AOJu0Ywn556+c1kk4iwwKZd4JAz7dnNDQuWDSmPHMEuP90wHJgfdXHMx c8mwlXGNboLu1hTNMeC0BK4CCQZJDxWf6L6AwklK1Ofzqc0IGB4KeIQBhHDr1+KZE5vm6uY+GID 55A== X-Google-Smtp-Source: AGHT+IFIMBcV3x/W8fhRczbwea2w1q1gcT3ETfM1GWq+zDEkeIxrI9ytOpFdbEX8EDgRr+EtsMkcZCh6eRk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a25:53c3:0:b0:dcb:c4d3:6e07 with SMTP id h186-20020a2553c3000000b00dcbc4d36e07mr382686ybb.5.1707773987608; Mon, 12 Feb 2024 13:39:47 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:53 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-8-surenb@google.com> Subject: [PATCH v3 07/35] mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: A5DD3100013 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: gweq1y1it9oz55kyo6hg9t68hi5ni7o4 X-HE-Tag: 1707773988-389844 X-HE-Meta: U2FsdGVkX1/aH6IAJJ0PP45iLsQ512N/SfU/cZoMAROIRZEvk050xWLe7jzNCOWxxHAiADrAd1HvPAMc+603WoHuNmtTJbQDPw6lSlNq8rl267mcyEBmYDLb1VdvT8y1BvkCb0RSC2cdiuCguOKH5IQqV98gNQqc5SHHoFI1ym3tTHsvieQbBhei7QL3kKNvkQbOZ6VPUJ8YOR87ruCPXSslmrTbrjcb/NcBHvtFjnzUHHV/kJwCj89b/+l4uh++jcCNBa3iEtySv74jKdaXPtOfWD0omrwXGgC5YSHPWDTssJckd9GNWo/Yjk4kf4OpMQuktbOgSIgzcwmGGq2Buy8qSHNVkKpBWh7ESWqI3lbIQ8iMK9tGiMOA2crZQVpovyZqqXKGFmpe7O78/JwkKSd1Nf4LJUmBohROrJAe+u19mxOqtM0jPpbvBSOH69UkqIZYMHwV6+xMmOdvzAXaSax/cFmmBOtz+zZvP52QVfDCSS6I3+yy3kFwm9llsc6L+1Vy/priE3EBvr6FBd/cNH7vhtbjk5trPLA4y9fTS03xeKZf6/tJCFlrEvePmOggVYU0u3CEMfqA/Y/hIXHyHDol8nZ+95C94xs4VlC2cZSKV7nFudf17RpTJbTtMWROCl7c9RTTA1B3Y9Js1+K6TT+3v5n7fS/VzMjtQ/JMntwuTVaoySYgF9Tzt4q8mceZLmD9GNQaOCcJZ2ER7vjCDnz/HZ9f2ZcH0cyv3HW5ZfK28M6JWMkHVn0A0CYOSaPi1GIbzTE8eWhwd7gad3xNZttytCqwE1ovDM7SENLlgz9+unXCrPNj7SMtOJiIF8P85fYt+WmG3AU44Xzfy/KnVgxG0OfO9SBiMdLp3UnwozN4tT4QyqBSNJdlMa4H42HWRuSlWZpP1/Y9uu4M8dGCHZImpddqiRtnv084n737TLheSAEawthPZWZzAZYbcH7lNAWGNqTwuBKG+QP3fde Mdvk9SKu J0h1jlzzrAgafQ4bOwopAIGGZ2ZHtBzk98rX9NHx3BeEhsgD+Bk87t7haDsHqhu7RGTxxaZjHO3byaQDDTFEVDheGmpsxa8Y3HJTrB7ZwCe6+jNGcpgri2I/nNMAjdNbHQW7LMOqVUmHu8plUVzs0r9f/2u4LVhwtKzfROFit+7jhfD2tnvKQE1gc7Wmx6xtMbgYXIlv7lBMdu8fT1BPhl4xszA08cqn11Jv5oWzuzoRaHknlmxxmzT3UGKNhr8343a0HpWAlOd4vt6lyNADj4MA2HNDMGnkYOrVG+LuDH7AUH8rIPgSWCNU79bHmYEDVJDnNZ8APvwYPndLIPNLOR4fxgZQ/kkxNLf5Gq2wjYLBwmYHTPzL1b+VH7iZauE6zABwP0TcU/EtG+jfFZbJ9eeC8Ch5XhbzYTis6D/fbu+23q+sxJJBQ3ro+cH1Hv95zXZy1vz0qL4WEuykql3uF/E2NJyxscT3RrglDH0/StAxmwPH4OQ6MQBj9CxEJSNopKkJE4OZ53CK8SJgV/Uo0lAO+viM17c6tjeY9pwKjgyrqNqi/57aWRSoHZgyWzgooK7/AFenV17L+/taKsriqchubjFu5bOG8Y3bn6MT9g63vbDp2h6CU+/7R2jwNT+SfLeNLYIO2ILo8/6oMdDhqpwNYcQ3s1SsTL5kqhCvyjHGQFH3Ly373o/NUlm55CgkQhiPf/4d6G8F+8vEDf2Bpr6sRUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Slab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like kmem_cache and kmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. Introduce SLAB_NO_OBJ_EXT slab flag to mark these caches and avoid creating extensions for objects allocated from these slabs. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/slab.h | 7 +++++++ mm/slub.c | 5 +++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index b5f5ee8308d0..3ac2fc830f0f 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -164,6 +164,13 @@ #endif #define SLAB_TEMPORARY SLAB_RECLAIM_ACCOUNT /* Objects are short-lived */ +#ifdef CONFIG_SLAB_OBJ_EXT +/* Slab created using create_boot_cache */ +#define SLAB_NO_OBJ_EXT ((slab_flags_t __force)0x20000000U) +#else +#define SLAB_NO_OBJ_EXT 0 +#endif + /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/mm/slub.c b/mm/slub.c index 1eb1050814aa..9fd96238ed39 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5650,7 +5650,8 @@ void __init kmem_cache_init(void) node_set(node, slab_nodes); create_boot_cache(kmem_cache_node, "kmem_cache_node", - sizeof(struct kmem_cache_node), SLAB_HWCACHE_ALIGN, 0, 0); + sizeof(struct kmem_cache_node), + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI); @@ -5660,7 +5661,7 @@ void __init kmem_cache_init(void) create_boot_cache(kmem_cache, "kmem_cache", offsetof(struct kmem_cache, node) + nr_node_ids * sizeof(struct kmem_cache_node *), - SLAB_HWCACHE_ALIGN, 0, 0); + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); kmem_cache = bootstrap(&boot_kmem_cache); kmem_cache_node = bootstrap(&boot_kmem_cache_node); From patchwork Mon Feb 12 21:38:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554046 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67456C4829D for ; Mon, 12 Feb 2024 21:39:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15CFA6B0087; Mon, 12 Feb 2024 16:39:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 10BC56B0088; Mon, 12 Feb 2024 16:39:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E799F6B0089; Mon, 12 Feb 2024 16:39:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D18A36B0087 for ; Mon, 12 Feb 2024 16:39:52 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9EFB5120A6C for ; Mon, 12 Feb 2024 21:39:52 +0000 (UTC) X-FDA: 81784469424.15.8D21F52 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf13.hostedemail.com (Postfix) with ESMTP id E1D3A2000F for ; Mon, 12 Feb 2024 21:39:50 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=s+4UimYS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3JZDKZQYKCKQWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3JZDKZQYKCKQWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773990; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MHmYMVX3pDUmg31UH8awuyUsQFI7ijlFGj9IhCGVNQw=; b=35BLVeXG/yijidyQ3+Vf4lY1QRZGgZQYzU/WgtQNXQiMl6LII+D/BlIokcpbhvCrkYKXbC plqFgn5O+wOxqwj6XttzAQ7EpdFrtJtvzgEqJAybxIw4TDwVR5WQoB5QO/rQPSpXreJSKa H9ugZqfedV2xBLwnd94lLsh33SQyi98= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=s+4UimYS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3JZDKZQYKCKQWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3JZDKZQYKCKQWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773990; a=rsa-sha256; cv=none; b=sWwebsIIq04APVIadd68flFrzzi22oKBb4yoBy7Fzz09IfIq8lRHg7b+jiDzmjsO1vKO45 +hDA/e1t3CkcjbByV4vNRfi3HSdZ8gQ0fek6sYC0qWpIsfMemVc78icDulRdZq2CVNkQrt 30irKnqJaX3PPhJQq6+QhLKu258HRe8= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6077e9dd249so4426887b3.3 for ; Mon, 12 Feb 2024 13:39:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773990; x=1708378790; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MHmYMVX3pDUmg31UH8awuyUsQFI7ijlFGj9IhCGVNQw=; b=s+4UimYSWTxNf/lD8dQetJQREcmpIU/EQj7c0A5yJAmTWaNL6j3NiTQVCv5you+GTb 8HNLRW8Bai91i2dyUIQJZGEDIqQm2yAEMM4JJ/tRz8sx8XyCxNl/+ixxTrxRD1YxtP7O cedKJ5FHP8TvdjI+mWzIay9UU3pLnkQpRBNo4qPKKkQWumXDWD+CKKo0TjOosW3c3TgT pii3EJCBv3xQM+gQ07oNQma5hrBIt+TwYm3pY3HB/q8GadQjny3olyajQd4fiBKNqW31 Roylr6j9KPLUhU9d+hDGsni7PLOlkBOBKj0TfPFY+kn+al2jVcrcr6Z/zW7GU61vclrd esAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773990; x=1708378790; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MHmYMVX3pDUmg31UH8awuyUsQFI7ijlFGj9IhCGVNQw=; b=aa0zJTSRmvKNDspTLR7FzsbHKF6K4U7OWsentBHOvbKrK/G4ZhDY+nqgiBBxMqyXmG +N3znLYQfAEZI74tjDYWCRsgpZwoGnJ+MgnOdA/TrbWP2W/zZu+Aof8mDQxlFd5fMapD Fh5SH861RzZ+q7xNj6Ofx55oRvynkhAkhdJa93gQ6VvM/S20MvR07k37WhCqZBNjGwgn Ztsxt2gWAc4PClM5bj4XurLZr9wLKf1szzSgAQe+I/LSTa3lFaSzQ7snszDkcbw/9l3n 8mnZWhhvgRH/WckPQqwJEzx19+REOkrKR8CCYjPMvSIidMZ0MCmdyxFiN6AJRli0ty4W bd7A== X-Forwarded-Encrypted: i=1; AJvYcCVoibF+l0xFYYeCAymQzMz/yNgMkA+VifBBj65hoKxZkKjwJdlsQ4n6I7Z+6wle3FoYj5138oZB3gRYT6KQPdgifX4= X-Gm-Message-State: AOJu0YznyGLUP1WJqpO94EO2VeWxseF6cjVOmFNXqK1AWcU7yvIHjG1p 50pA0lLGyKqLodciltGxIa/f3WK9gsapKUsvpDgvC7YSEhQxYQoCf2xz4cxPsY0fQb4oVBQSlGY vvQ== X-Google-Smtp-Source: AGHT+IFDzjF2cGPjf9byIT/iUDxPrFM8RUYETPrPWdKRGwVNA8HPcuChMwLVorzqu+gtL/yYLkx2m1ZVYB4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:690c:fd0:b0:602:cd1a:6708 with SMTP id dg16-20020a05690c0fd000b00602cd1a6708mr1449899ywb.0.1707773989875; Mon, 12 Feb 2024 13:39:49 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:54 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-9-surenb@google.com> Subject: [PATCH v3 08/35] mm: prevent slabobj_ext allocations for slabobj_ext and kmem_cache objects From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: E1D3A2000F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ta5obciej496fz7xnfswspimc9zub779 X-HE-Tag: 1707773990-249842 X-HE-Meta: U2FsdGVkX1++YAia//sKw+gwL3rV4DqvnJf3sTepda3B8Q36XbySQiRk4mtJ3pxDJ4gxzZuuwe7wTSji9GimfP2txzHPDntsEsKADB/Cgbo5pSFyxyTEaqcpHAYBYyRAzTO81QguBCmwgWYvBJZOBMLqCzzEmrzi1wwembBPuZoAyizS8JBUyFdC60GSeBvSR7XYyaWLX5c2pqZC57rtpmX1BHBRUOGwhuHv+uksAlAYAaYppViQglazgMmajwnkqaM1gFwmUkwboUt6BGjWpY86qTMzliARS4YSYn3D81e6BMKD9oSB3GraEiroyLBmd8J8mnuoIjPaWkzBhRkPN5sNNyWjwOzkdA/pH5Mr47PFyZKXocAfjxsUykWLRYk9rlpSaZdH8rZdLJLiieuqWtUYRgS4HQNqRExFdhMf2th4kwzo1vTAqtH4LF7LS7OovXtvR/9U4dN59JVvNVJ59g2/GL9Yc76ydn2dOe+8I3II1wDejJu8AHBRuDP9dYz+MU2PtgshxB64jj6POmCICVpw+jJRylY6cKwipv6zNp+ZbjEsdDiQ/l/TayBYAHoAe/pLGz4Y/u5EUkXxTgN9SpezWcJ0OOqgTq7A+m1A5vqgIpV0Gd+qUE5sMxrYsKmQffGpLl4g7FAJr59e3RbWjXl5NLtgA2L90MI50/f/SY212SVMuh8W1G8ZdsvDGL4EwE3+V80FMbhyEZrSShlBHc1Zmen2rLTKHmWXATmhP5QuZUjzcSambTqMZRq4hN/0Q31+gzaOe5hMLd6AxJsJ5PQp2tgGK9T8pyigO5sR79BM/46kdgBQXgLmxUkCTOTz0USYBHNbuoeZXgR5QLv9j5t0X4DH9ANm/33STCugzlrKXzlNdxz9uDAfTQHy+jEz+L0wDvEJny76/wlbzLXvuCMhNuX9WneEsib5edWJZlSPntXsatZyguVv47obE6UrGBOfSQvYoxdS0+KyFT2 zrHHjydW hxTbyEG2d0nUsE1E1raNZ7PqADnAjpb80FukxyN8d2zoY6M9OTjuIYx6+KF5AbPzNlrmskdYaVZNuthxPImJSbUfbRfweTQ3A7G7uKHIhgC9ird2bgQe8oZRpj9ftR5MTw8ZLJ0EgNVpiLhrOLjDjkW7RdmJHfZdUyjr4/hT1/UewNzZY0Ve5y2Xx3yGwjbwygxhawuqMW8w/BQ2S14PLZB6cKdzib90yg0PVrC5enxK5vXav2gst+7L+PJMsVNUp57kQQi9npv/3snnXUTpHtp5nWyo1GVugknypYj+UzpXTO7wOPkTxZEQPYy1jyQSs1hjcCPh4CKQjGWLh7nES7YBnhP5bUyMLwO6NAuE97wcKueGl/5OFCgOPBZs/i75vjnrvdZAvY5Np1fAk1d2l6EwWt0eXkSjDf1L8YlNu0sq+xJUfdTuS92AH1eJ+BfQnbpWqO3CzC838R/kUAEZlHRk0X19GjHtYYlxXGCUjZSTtq+7fLSewdXu7qdWyrYT4juGkz3csNeHHLJ0Mx1dNY73YXZU2ChZ4w5rTjSJWDhxgdEC2dE81JXAvndddP13MCzK8vmQsoI8w5YlP2bzgKmkZUZk60NBbBJQ27jheNaykEl40N8t6/YPMV1iluRmR54oQYtG78IS5Gk2NA+NvLSuDRg4XRHXsWu63J3sVnzVu5oNE4yD4ZHXSqxzUVIhzw5Q5ZDs6GF4IkZT6uxGQKqK6lCE9JDbTF/JM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use __GFP_NO_OBJ_EXT to prevent recursions when allocating slabobj_ext objects. Also prevent slabobj_ext allocations for kmem_cache objects. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- mm/slab.h | 6 ++++++ mm/slab_common.c | 2 ++ 2 files changed, 8 insertions(+) diff --git a/mm/slab.h b/mm/slab.h index 436a126486b5..f4ff635091e4 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -589,6 +589,12 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) if (!need_slab_obj_ext()) return NULL; + if (s->flags & SLAB_NO_OBJ_EXT) + return NULL; + + if (flags & __GFP_NO_OBJ_EXT) + return NULL; + slab = virt_to_slab(p); if (!slab_obj_exts(slab) && WARN(alloc_slab_obj_exts(slab, s, flags, false), diff --git a/mm/slab_common.c b/mm/slab_common.c index 6bfa1810da5e..83fec2dd2e2d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -218,6 +218,8 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, void *vec; gfp &= ~OBJCGS_CLEAR_MASK; + /* Prevent recursive extension vector allocation */ + gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); if (!vec) From patchwork Mon Feb 12 21:38:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E1DDC4829D for ; Mon, 12 Feb 2024 21:40:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15F556B008A; Mon, 12 Feb 2024 16:39:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C2736B0092; Mon, 12 Feb 2024 16:39:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2BB06B008C; Mon, 12 Feb 2024 16:39:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CC5F36B0089 for ; Mon, 12 Feb 2024 16:39:55 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A2FFC802C7 for ; Mon, 12 Feb 2024 21:39:55 +0000 (UTC) X-FDA: 81784469550.29.E2E1B3A Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf21.hostedemail.com (Postfix) with ESMTP id DE35D1C000A for ; Mon, 12 Feb 2024 21:39:52 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TMFpdhCX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3KJDKZQYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3KJDKZQYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773992; a=rsa-sha256; cv=none; b=Pjzt3ORKfBWirFGokHdCe9NZHhSbeHbmsLNIWqFRgcYanh7Fq44XbAYuyuxZYvE4A31Rqc FqQ/FYc5jCajZsii7apOVxA8K/MeCmRmRSHrDgDsNAQLb8OQdr8ODJ0WP2tvnWIfroAv9L hBchOdrUDCCgBVghAHNEA4udRLtAsnU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TMFpdhCX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3KJDKZQYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3KJDKZQYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773992; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Wy0D/wmUn3AGzRHJaKJZBS4cvr5irQ3tEAXEefF/wd0=; b=0L/fq/eF3JPcshHh7lAa6SG8jCBSLO8610TDhEHJM/OQ3cE1d6kfCSyzshIdU2C/DwGXpA HWvkvN2zgnBgcKKujj632FoYOMpdfKkIUAhpjF0piDl/ZrhP87Bk8a/RRFLGoNfMLsVVjt sSJIh98lAKruZEFBOEYXgFAonY4O4O4= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-604ac3c75ebso68375527b3.0 for ; Mon, 12 Feb 2024 13:39:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773992; x=1708378792; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Wy0D/wmUn3AGzRHJaKJZBS4cvr5irQ3tEAXEefF/wd0=; b=TMFpdhCX3j1xquLdGUtOzMbfpai69IciqRQUbvKfxf6ZiE6oQvA3nICX3B1gwKLwgu X65dSIEclKcQGNRpsItziEdPHZ+ASQ1HWfoFt9DcCyOc8SKrbI89H1k5TveZ+xPTj4nX 8bnElEZWHGshNpPrxl+ZsTegYwANWHEujMJpQFSWRmJSLIlDuGW/KtllLVx9Vkxe7NiE biB+Z+6unBBoWPq+ySGqDgVGY49Sg10dHIi0mCgU8/sfkeUuF2TNfqY9ChYSjASP948o GcG3/W/ZulWyw3aqwUFPxIdFHzJago8Kn2X3DU4aSq4bq5hpmZjxqhWrBFs2wPrYEE/f xKIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773992; x=1708378792; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Wy0D/wmUn3AGzRHJaKJZBS4cvr5irQ3tEAXEefF/wd0=; b=DTcNoT9PGh/0POFkSDmUYDUVx7fvCBb0H0LWkw9VHzddchcv/Pl1ZPE35lBaK5p+BN cyObUURYSAKvs8exBjhYhGsIA9t9wtPUFoU6AEn0ZnqQ8iazIGbbvbwHEhvXIu8lgxSI IfP5FNnItpGmd+RQHB4lB7aPkZrXrow1M/vwgtPBMn6EfMpeTPsxyPe3xwPG/UmiN5r2 Tur4T/S0w/aWrswg0W2tXai6QM48HyDuz5+xJyCf5z0IN0NnHEuzDH/uuYNl1mkiRfcA aHoA4fiC4PvtpnNUzp7nbvNxsoDQokc1IA88ElFfcVJmPWImKLGxoPXs07cNOYU+bmZg AUdA== X-Gm-Message-State: AOJu0YyLBTQX24i1YW1YKsYf9Ek16rORriIOmLPAlAam/hUUqaMAGlbu KX4eoGuFd1ME0CxGFlb660OdQ5Rf2Jo/TBfWl9BTRqBNkDzrjwsZEtSNS5+sos4bx40/kNQDBeP HBg== X-Google-Smtp-Source: AGHT+IGJNldcJg1nJudvCE1Sp9g/6sTQwYzeBOB4AMRF5kjsz9nHiwJOvFtrkpc6PJTXjQSExrbdy8gvy1w= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a0d:cc91:0:b0:607:7bca:e8d5 with SMTP id o139-20020a0dcc91000000b006077bcae8d5mr286452ywd.0.1707773992000; Mon, 12 Feb 2024 13:39:52 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:55 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-10-surenb@google.com> Subject: [PATCH v3 09/35] slab: objext: introduce objext_flags as extension to page_memcg_data_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DE35D1C000A X-Stat-Signature: 54uorr6d8psoeia4ostos7btipk5crx3 X-HE-Tag: 1707773992-935615 X-HE-Meta: U2FsdGVkX19XlDQskdneA9kfneltdsmjP75f7Q8cCuOnTK9QildkI+RL5kU4G6T40vZl+vNZeqUKw6MP/7FAX6lzXmWUXWXcN40MKees77V0w8E5KW0dt9M7JuVLfwF9VDWr8fGeglzDHKEIftLSgl8nzE3X9SFh8tzI3h/XiIwQG4+13ESsu25bPYjbDyjEAhvxR98wVVkecgcvR72GGvh3zHZ8+zYCXLkrTC5GGkjTzPewq7SzqQpQcEJhKYQJM4DlQD103v3E0+34cuFhk22a3WCyW/vBg4P/Fr8lwQ2qVH0fjLy7uLGEteofizE5yc7HpQ9qyBIYy4uR3bact94FGN2IAV03hJZJeX3EBy1VLqP1zcjo01DjOnKFnZuGkRJvgDnt9mAc0GU7Nfjb0HB5MW1o1woW9A6156XH16kCjyRERx2ZbH+c4Q20kC23+9l11nG74ecBttt8lwPKD3ZkxJz1Uxb4RvargghiQ7ASTYK0awhW4qYbr3HxLY24C+at4eIgM4F1Wfkm/m6Fim+otmo9M1TkTsQzIky4paHBAVhQYMQjybL1z4gGIGbtx17uQnQoNEReP+XET1mUMR8WFYaMyLhh11arxh455bobLSGRljzYjMyr+nTgo6VxRUNVGsojnSwCu9iev0jNi/umw+966djI0G8Sv4+9hbU377KJ77TSFo06iMLFjRuhxqaPSoC9PM5V14Zc1AJE2Psq8AoCkThdqW9JOTNrULnr/nayag0Rg28JVODfObh9gka2RYxeuslArxrpyNRq5ENAB9RZcNm1Rgcg6hvvg4Yzi8BxDPDhNxHFGTYoOjK3Scfxaq+Amp9hWDzfBwyKFrM9zAhEmTxyNwQhmgPxwGHzc2OWyAHylbBRLWeCo5SvZWFeqTI8alFpZCRA6JX2Ma+MmWdltgzp15NJtrkGAacgI/LuxxQUOL9DYSYGyMT0gbkpqzmmZHSuCyMxUOu VBtmrDl6 yr3O1lQfbdk6UcpgBAevy2qWo4Hac8wHRAo4zqt4Yg30ibUygTPTaxQqmccYVp+Ahb5JciFdVmJHAk834CSKd4+xpehFmc8WqJcURyKmboIXe9PMFWq6LRB4im39qrdPUYUhriMLvaZqSX5iXkIpnIwQzGBiUR3pgyRZBJ6QHnBDLwvx06SIwOJ1lABZIz8ba2wNbTAqEcYVWwdP5n4Um5o2mP/zRx92DI02K+AX08SYfzKlPFgVHOYAUC6PnCOXJmqs9alpK9gpYwKYvD67Dcfu1cTkjP1AuKl4eNMtb0J7+KAfEWyFzgFlTjD499pyxGm037UEVCPbdUhP12IMThtI8SVjBLMme2sVhxlugLEAY4uHSLcxkmGhx5Ei4HILNvv2k8IJWHWaWQnVHqRzbokx4yxBDTKUIXU/spzIxsxTUB28WH2y6YoLqx7Ha2OSq+07OrKFB1cg4B45dVhzPryV4gkTQ7fH8GssHD7OP0EqduahoJiVCe9t+YNTkQfME+2NGlKToAvuf21/09/we4DD8IFJ8qUv1BsEJYWkHuph+e3CsHrSMSExz3tKBVoCWUBrs11ZHVuMx/O2s7XYELMFLOaSsoeUB1crm8g9V4XLmatLG7gi6MGIumJtMCYyrtMNUfr7fRDA6rpiXlGx36TIbs8SbI330wAwpdcRYblAwpByAqqUBs9C2zf8kWIdktTFlBIUdqKvQd/GDifs8PXxVdSyOojvzmyks X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce objext_flags to store additional objext flags unrelated to memcg. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/memcontrol.h | 29 ++++++++++++++++++++++------- mm/slab.h | 4 +--- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index eb1dc181e412..f3584e98b640 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -356,7 +356,22 @@ enum page_memcg_data_flags { __NR_MEMCG_DATA_FLAGS = (1UL << 2), }; -#define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +#define __FIRST_OBJEXT_FLAG __NR_MEMCG_DATA_FLAGS + +#else /* CONFIG_MEMCG */ + +#define __FIRST_OBJEXT_FLAG (1UL << 0) + +#endif /* CONFIG_MEMCG */ + +enum objext_flags { + /* the next bit after the last actual flag */ + __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, +}; + +#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) + +#ifdef CONFIG_MEMCG static inline bool folio_memcg_kmem(struct folio *folio); @@ -390,7 +405,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -411,7 +426,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); - return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -468,11 +483,11 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -511,11 +526,11 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } static inline struct mem_cgroup *page_memcg_check(struct page *page) diff --git a/mm/slab.h b/mm/slab.h index f4ff635091e4..77cf7474fe46 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -560,10 +560,8 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) slab_page(slab)); VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); -#else - return (struct slabobj_ext *)obj_exts; #endif + return (struct slabobj_ext *)(obj_exts & ~OBJEXTS_FLAGS_MASK); } int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, From patchwork Mon Feb 12 21:38:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01044C4829E for ; Mon, 12 Feb 2024 21:40:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B2176B008C; Mon, 12 Feb 2024 16:39:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 83A056B0092; Mon, 12 Feb 2024 16:39:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6633F6B0093; Mon, 12 Feb 2024 16:39:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 53BE76B008C for ; Mon, 12 Feb 2024 16:39:57 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1BE77802CC for ; Mon, 12 Feb 2024 21:39:57 +0000 (UTC) X-FDA: 81784469634.15.A84D3AD Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf02.hostedemail.com (Postfix) with ESMTP id 4E27D8000B for ; Mon, 12 Feb 2024 21:39:55 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pY02Kzmt; spf=pass (imf02.hostedemail.com: domain of 3KpDKZQYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3KpDKZQYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773995; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3+XWH7AoCI6Q7n29IBIT816HNKsBqFH+UYcKnJYD23I=; b=hvBn/bZRv4oSMCR8PckBdAOQSgbvEn3nSttKc3E8kLDaIH/XkGcDzHKZ/v33dHt/dV077I MVlZfZbkAsqayTFK3K764u3un50Ax4WWOCLxu/LG2+gf7r0nQwdIwAGgGTV7aG4koA2sGO YXE/nFfWxj0SVCQMbSpK7b7R88JneHU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773995; a=rsa-sha256; cv=none; b=ir+2xdNHaIYsACaGNbiltIc0alPJ+UEoku/db7o0ZE0uuIn/DnTkkUz4XwZKeB5pbZBmqI dObOvqmjn9TkVw8Vd1RCqNg8tsAdUCbjLSEblaHPwahYM4fjaFqcGwyJgSQLPITJBOtbEj Rv4mgATucTEcfpSVwi7X2iVi83csPF0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pY02Kzmt; spf=pass (imf02.hostedemail.com: domain of 3KpDKZQYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3KpDKZQYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6ade10cb8so8232863276.0 for ; Mon, 12 Feb 2024 13:39:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773994; x=1708378794; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3+XWH7AoCI6Q7n29IBIT816HNKsBqFH+UYcKnJYD23I=; b=pY02KzmtyZG11LipIladU1uIhl+oThxndh3TMebsi10GV9nCwtiq4bwLx0egpFehRZ WUf35A2AnAD0m813MOdl4GbtqUxWkNHcQMTpFnLRzDlcnIjLRlkLnopeTKe3cXdRjrnd on3zOvfAVEInu/rTYFzGQFzpEMghREiw479amajI048Be5cZutCce4oD2ogNJrbpG3WV bIkCcnRCEuDwfz6YUVfF6v7TM8xStkAJRNfv9YDIsex6m06fgBGZcgXMbD8PxdvPLfUR Rbq6VWr9p3YJq6swpyuwdPIRBEd5LbAYYrPjXf64oVAzidQf4nF/wRB14lq/yPHukzRh hIuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773994; x=1708378794; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3+XWH7AoCI6Q7n29IBIT816HNKsBqFH+UYcKnJYD23I=; b=cyA0gk8f+li+WX2x114bUhkExzuY2vGzlkiKehO++yXlmQ0WWo53kGYODiOj6gkpqy co3eG68ceDmvh8QiklOm2w2cXd+VpxkrpFNFTqTQK/DtdQhRHmJXt5zaMS6Wb97LjW98 II3BsojL1K0yI0hDEaJEHrakGLJZROgF4Bek8gX2Obs6K00+yEKGihcNdaV7NnoWdPB+ H23eu8hyLc759mGRk7KC5aGtxccAVtpP/ELyLxqSEwfX8wSZhJ19m5aFKWyqzSNHb8xN MBgWjJE1sv5q1iZjDqU2FrJUytH65xztwMAjj/yEIv44GXPBK+oO1LEI9tONpmjY51GQ 5kXA== X-Forwarded-Encrypted: i=1; AJvYcCX4t67YyYAL5cDMVXKVhsBXUoXL3XD8wysmB1vy0r43RZyEE6/IgPtGock0F4VDdBMFMdN9+miDpKOLEH/MUI1oWfA= X-Gm-Message-State: AOJu0Yz+l/Z3GJorarvVdkGB4lA0Arf1njfGt6SyiyzNwKs+qvbzlFyb JMMvITUKyazRRmmKGzYBmuLYdbnaodZc2qRFUizUJ6+Aw/ze3FunJmrwQBJP9mYHVvnkZ/H0CTT YqA== X-Google-Smtp-Source: AGHT+IG0KPVbCnbR173tQAyZd47dBY+49D8romdyHPgaIzYcKde9kTc683kAJqOMsgGylG5greCZPiYqZQA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a25:2614:0:b0:dc6:b7c2:176e with SMTP id m20-20020a252614000000b00dc6b7c2176emr130871ybm.4.1707773994229; Mon, 12 Feb 2024 13:39:54 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:56 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-11-surenb@google.com> Subject: [PATCH v3 10/35] lib: code tagging framework From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: gwdsjmbj4k65b34cq9m3yqe3w75bjoba X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4E27D8000B X-Rspam-User: X-HE-Tag: 1707773995-630134 X-HE-Meta: U2FsdGVkX19toZseMpFkzqQQqnjCN9n/6R6bnEqgsTqMQWnUp2rQVhBaD3Yrl+EOYgQR2HmQ5UTabwGya96QFqzwhCyCdm60fZ5bSMQJYdqpDDBtLHAQFCfzZl1xS6QDnfApPK2KznL2EAsOYcJhSCCRAYk1ZU7bEL3RKPz0TZ+mYTh/JOqGYjCz/ilne4G/9XVSqZ+n7sXWXW9aunUZqrwG646ywqjtGfkRQ9TkHKqdh9qHHddOwpvJ7zcp8fs4RjaOPZud5oh77LKIQsjvr8tULWdCFuAsIUsEPkW6qvPo+9zNQF2qz6r3dSeFN/JJK6gTalnNg2WpyE5QDdsGRGTfhWqZ+w9xDx7rzEMVJejhFSQQ0dlDJRazwmAHVWHTM2/SUp5RFMopEpZCDVFEbhX2PLCWdjsZJJs2q/OhanGx5zx02Lwib4mEMLkZuYg6v3WRMhi8AwqWzhztokQ0wHR9OdsZheDjVM4An7lZ0iAjNI6CrmWDitaLuGSh08jRLaFbwlpHlXX3ipnIM09BC8aJj6g3L83jV1gu5QUOX74TbXr1fl3TTfhKulhgvV0qd0Ka+ehfy0XAU0ANZxZMo/UUWG9k1Ym6pvrXEGFWg69aJxPuvZEtNwY8r+NpBiEnL9wz6zoDpffKZBTgHESW9LcsVHQxOEJlooNidIw9eKILR8xcmkw/eb355oZ4GLOJP7H7lesWadslhEa9XaMLumxonH9PehlVdWEcGNY0ntpPMY9A6e6+I5RndqQDEFWHYzFMj76sBXWhbmZsws8GGSiMj+NrXlLOzwfkEgWMH96noXBmkB0bCl7sb0iB9qAysudgDHfGWbQoHdW+lEDdpiKeTwSNoIyuC7s4rVOf/Nq+14YVXIINMKe0Jx+YsV3bfViKTJuJo52Co7gfiyopSWqjqilZWxxNCnQQgkttmCyy53r4pPPkDOqog+OJYdhzh9z6lWHAQMn3A+NJ39F XES0MyGr czTmegQ3MQPgBZyHBLbE71OKt4M5C00/LCQARUP0DufYUOOTFk33G/6R40CmpamFAUafHP9EqjA2uA7lF30R9eNN/uESZCwcaHeBEuJHZ6A57aP1kFbDXIrqLHmYJ10OBp2oUo+d5hXBhbfev2D6ksC08GD7jTAYv/CpjYLV6TvzVhHoxNz5o69uNIP5NEJ38ZBET3aqYOhNu4GOik6s3+t4ubp9I7d4lBjIQPoKzQlYraVwnkcn114RCpAblx8a8gGyayXmHS0a52VNrmfpQWgTIYi+ueYm9obvShh/ya9eAl/7YeyFF1qcgl1r4BgCgi4yHgHjEV68ERvZQrlWwyxNttUtmdyvfwtjoMyoEFT3kDmfgHmc0Uyt66i7d/4FzFUG1IgJk/0aCL17T6nH/YszS2XU7kvF7OAjK92Vm223RN8zjyE7dEKSaeEkIappzDrp4A6zTsib9eCCeP7MSSHky5N2i1XSMEWzM3LsrPO+3MHbXm1/rk2qQq7dPvpc/zWwWZph+Cjft2xdQNtL7k+Vs21mTR006PP5VXaqWFWo1/9/8Dc/BBu8S3j1ZZ2CjzVAIH4ObckaME/q80UJMzgCbvEFi1IT5KKalq7I2idd23mvE6OCho5UzuUxjttxRaPFmOK3zz83eyyuhA4QSTAIDcQGiW4BdCEMBVA/Smtpk77Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add basic infrastructure to support code tagging which stores tag common information consisting of the module name, function, file name and line number. Provide functions to register a new code tag type and navigate between code tags. Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 71 ++++++++++++++ lib/Kconfig.debug | 4 + lib/Makefile | 1 + lib/codetag.c | 199 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 275 insertions(+) create mode 100644 include/linux/codetag.h create mode 100644 lib/codetag.c diff --git a/include/linux/codetag.h b/include/linux/codetag.h new file mode 100644 index 000000000000..a9d7adecc2a5 --- /dev/null +++ b/include/linux/codetag.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * code tagging framework + */ +#ifndef _LINUX_CODETAG_H +#define _LINUX_CODETAG_H + +#include + +struct codetag_iterator; +struct codetag_type; +struct seq_buf; +struct module; + +/* + * An instance of this structure is created in a special ELF section at every + * code location being tagged. At runtime, the special section is treated as + * an array of these. + */ +struct codetag { + unsigned int flags; /* used in later patches */ + unsigned int lineno; + const char *modname; + const char *function; + const char *filename; +} __aligned(8); + +union codetag_ref { + struct codetag *ct; +}; + +struct codetag_range { + struct codetag *start; + struct codetag *stop; +}; + +struct codetag_module { + struct module *mod; + struct codetag_range range; +}; + +struct codetag_type_desc { + const char *section; + size_t tag_size; +}; + +struct codetag_iterator { + struct codetag_type *cttype; + struct codetag_module *cmod; + unsigned long mod_id; + struct codetag *ct; +}; + +#define CODE_TAG_INIT { \ + .modname = KBUILD_MODNAME, \ + .function = __func__, \ + .filename = __FILE__, \ + .lineno = __LINE__, \ + .flags = 0, \ +} + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); +struct codetag *codetag_next_ct(struct codetag_iterator *iter); + +void codetag_to_text(struct seq_buf *out, struct codetag *ct); + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc); + +#endif /* _LINUX_CODETAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 975a07f9f1cc..0be2d00c3696 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -968,6 +968,10 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". +config CODE_TAGGING + bool + select KALLSYMS + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 6b09731d8e61..6b48b22fdfac 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -235,6 +235,7 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ of-reconfig-notifier-error-inject.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +obj-$(CONFIG_CODE_TAGGING) += codetag.o lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/codetag.c b/lib/codetag.c new file mode 100644 index 000000000000..7708f8388e55 --- /dev/null +++ b/lib/codetag.c @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include + +struct codetag_type { + struct list_head link; + unsigned int count; + struct idr mod_idr; + struct rw_semaphore mod_lock; /* protects mod_idr */ + struct codetag_type_desc desc; +}; + +static DEFINE_MUTEX(codetag_lock); +static LIST_HEAD(codetag_types); + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock) +{ + if (lock) + down_read(&cttype->mod_lock); + else + up_read(&cttype->mod_lock); +} + +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) +{ + struct codetag_iterator iter = { + .cttype = cttype, + .cmod = NULL, + .mod_id = 0, + .ct = NULL, + }; + + return iter; +} + +static inline struct codetag *get_first_module_ct(struct codetag_module *cmod) +{ + return cmod->range.start < cmod->range.stop ? cmod->range.start : NULL; +} + +static inline +struct codetag *get_next_module_ct(struct codetag_iterator *iter) +{ + struct codetag *res = (struct codetag *) + ((char *)iter->ct + iter->cttype->desc.tag_size); + + return res < iter->cmod->range.stop ? res : NULL; +} + +struct codetag *codetag_next_ct(struct codetag_iterator *iter) +{ + struct codetag_type *cttype = iter->cttype; + struct codetag_module *cmod; + struct codetag *ct; + + lockdep_assert_held(&cttype->mod_lock); + + if (unlikely(idr_is_empty(&cttype->mod_idr))) + return NULL; + + ct = NULL; + while (true) { + cmod = idr_find(&cttype->mod_idr, iter->mod_id); + + /* If module was removed move to the next one */ + if (!cmod) + cmod = idr_get_next_ul(&cttype->mod_idr, + &iter->mod_id); + + /* Exit if no more modules */ + if (!cmod) + break; + + if (cmod != iter->cmod) { + iter->cmod = cmod; + ct = get_first_module_ct(cmod); + } else + ct = get_next_module_ct(iter); + + if (ct) + break; + + iter->mod_id++; + } + + iter->ct = ct; + return ct; +} + +void codetag_to_text(struct seq_buf *out, struct codetag *ct) +{ + seq_buf_printf(out, "%s:%u module:%s func:%s", + ct->filename, ct->lineno, + ct->modname, ct->function); +} + +static inline size_t range_size(const struct codetag_type *cttype, + const struct codetag_range *range) +{ + return ((char *)range->stop - (char *)range->start) / + cttype->desc.tag_size; +} + +static void *get_symbol(struct module *mod, const char *prefix, const char *name) +{ + char buf[64]; + int res; + + res = snprintf(buf, sizeof(buf), "%s%s", prefix, name); + if (WARN_ON(res < 1 || res > sizeof(buf))) + return NULL; + + return mod ? + (void *)find_kallsyms_symbol_value(mod, buf) : + (void *)kallsyms_lookup_name(buf); +} + +static struct codetag_range get_section_range(struct module *mod, + const char *section) +{ + return (struct codetag_range) { + get_symbol(mod, "__start_", section), + get_symbol(mod, "__stop_", section), + }; +} + +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) +{ + struct codetag_range range; + struct codetag_module *cmod; + int err; + + range = get_section_range(mod, cttype->desc.section); + if (!range.start || !range.stop) { + pr_warn("Failed to load code tags of type %s from the module %s\n", + cttype->desc.section, + mod ? mod->name : "(built-in)"); + return -EINVAL; + } + + /* Ignore empty ranges */ + if (range.start == range.stop) + return 0; + + BUG_ON(range.start > range.stop); + + cmod = kmalloc(sizeof(*cmod), GFP_KERNEL); + if (unlikely(!cmod)) + return -ENOMEM; + + cmod->mod = mod; + cmod->range = range; + + down_write(&cttype->mod_lock); + err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); + if (err >= 0) + cttype->count += range_size(cttype, &range); + up_write(&cttype->mod_lock); + + if (err < 0) { + kfree(cmod); + return err; + } + + return 0; +} + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc) +{ + struct codetag_type *cttype; + int err; + + BUG_ON(desc->tag_size <= 0); + + cttype = kzalloc(sizeof(*cttype), GFP_KERNEL); + if (unlikely(!cttype)) + return ERR_PTR(-ENOMEM); + + cttype->desc = *desc; + idr_init(&cttype->mod_idr); + init_rwsem(&cttype->mod_lock); + + err = codetag_module_init(cttype, NULL); + if (unlikely(err)) { + kfree(cttype); + return ERR_PTR(err); + } + + mutex_lock(&codetag_lock); + list_add_tail(&cttype->link, &codetag_types); + mutex_unlock(&codetag_lock); + + return cttype; +} From patchwork Mon Feb 12 21:38:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2498C4829D for ; Mon, 12 Feb 2024 21:40:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 700BD6B0092; Mon, 12 Feb 2024 16:39:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B2996B0093; Mon, 12 Feb 2024 16:39:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 505206B0095; Mon, 12 Feb 2024 16:39:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3B7E56B0092 for ; Mon, 12 Feb 2024 16:39:59 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 122A7C062E for ; Mon, 12 Feb 2024 21:39:59 +0000 (UTC) X-FDA: 81784469718.02.C5CE671 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf30.hostedemail.com (Postfix) with ESMTP id 814F28001D for ; Mon, 12 Feb 2024 21:39:57 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=O5gjYq1m; spf=pass (imf30.hostedemail.com: domain of 3LJDKZQYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3LJDKZQYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773997; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZVR5x2YILJK+zhPcdicEuOWU7vdbf7DJDXDz97hru8E=; b=D2sVYxrfs3Q5VhU67UcX2eJEI3g/dreUBs3TChMl9G1B+NlakLzqnzHwlw7Z3AS8xgBeK4 qH6zUiBsWSJFPs+0WG+rv76zfRFJhrq2alzXTh5CsMscUTxPn6Qw1NRdE9J/yxzfRo/KCR J3BMub+eXXBsHKgvhFKA+MJ2KaDxrSA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773997; a=rsa-sha256; cv=none; b=mos8WPYXFsycO4FicxO4gNa4QFA9zS420i8IOBAn0htAZ+eyUoQUiAo4zSNoCrMYmH9+MI qcBfN/MgXgqTGFAY9u7fZ989AuDzxSRiAjYjN1hHZFaYIXWXCG5iswpL/dozo60s20Z7AY d6IGl2RjoLAY/mdIkWisZ3zRN7ACWxQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=O5gjYq1m; spf=pass (imf30.hostedemail.com: domain of 3LJDKZQYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3LJDKZQYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc74610c9b7so5843885276.3 for ; Mon, 12 Feb 2024 13:39:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773996; x=1708378796; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZVR5x2YILJK+zhPcdicEuOWU7vdbf7DJDXDz97hru8E=; b=O5gjYq1mPiQTG/Ge8pMO6jkHfu5eQVAS+aEhON59oJXEVvM3XfrHogwrnW8NAdkx94 2dQ0c3wzk3WSbu4krQOi6x3FsDgGT6dKduKsQFRAOTuo09tbRwXTk1mr63Ajo6n+pySr ksXiCE8BlvrfCfUvIqFLJSLhR8TW/xPgOHRbC5FmnvNpVrN3SKqx9jmfbe4q154CsSta IbEH+bqyoSyEoLcloM1ps2fTPIeu2GQzsb4Ud0jy+9+zC+Y5e3J/I0MDwiSllvOjxP3s SZoKcnJrriIF/HubnybCfE0LjC0uN/iq+NOuJ8Br+ha92LUsY587lrWHCAmN6bJjUGaU 5Xyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773996; x=1708378796; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZVR5x2YILJK+zhPcdicEuOWU7vdbf7DJDXDz97hru8E=; b=cu/a9E9zBa2rb2XaLtVpn9ZPam3tzn2PDmWQ9XLQmYCp05zvE2yUte0GgnMMpmGyNB KEyYf+kMeANreP5frrU2XcBZu3cZ0TZYzPDoBlvTdBMwN8L9qjRuvQGDvCbKkZEbDm+1 D6znC49xd+cSdmJLKJX6oPfxcLTjxbPgBYDW4baZnL0bYrJlQQSRnQr1fcc0GHCr7s9c pDx0iv7TGgE7Q3fOO5Ct+c973GoML1Br+885R36bBaGjsS1UU9p4nPrf4uXf+ODuUEef 5LDuOgMug69Twwx+9zQKBGi2hJ3Li4vxDz2q9LcgeuYYZCFsrw1zoFFYqUjSnVOuVM9P Jn3w== X-Gm-Message-State: AOJu0YwwwMiH2NLxCMR+yafpghcL3uFkiAI9OpmXWXHpcSxufq5I2w6z zc55uOsvCOf+RDDw3P3YSAvuyus/GmsS+ZL2J/LZVbRwT+jDMzAa7wmzlyCC5MlfNUNbh+8RMTS E5A== X-Google-Smtp-Source: AGHT+IFrahWopogc+rmwq0xz3eYYu6J1qHJr+fSwJ9oyAoKj8/t+TZ8Jneh5cBWLRBbyxn7YPxVr/zwosmk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:100b:b0:dc6:fa35:b42 with SMTP id w11-20020a056902100b00b00dc6fa350b42mr2046974ybt.2.1707773996437; Mon, 12 Feb 2024 13:39:56 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:57 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-12-surenb@google.com> Subject: [PATCH v3 11/35] lib: code tagging module support From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 814F28001D X-Rspam-User: X-Stat-Signature: scpja5td7r7ojjtga7tsz3mbafx8uzy1 X-Rspamd-Server: rspam03 X-HE-Tag: 1707773997-68487 X-HE-Meta: U2FsdGVkX1+75s+2r/ME5b5ggcBsl8kj6AQJi7WIb83D1WAYpem47UmElVpLmnN4NiFMjl4VD4F2BuXboXoT+TEnDWAzmDPXIcU/VbKza9pOo0RMOFWNi+j1Ymzec3RztAucYFBlH0tJyJ4Qx+0iRqc1rMAafp0nW5+5dBqOPfBQ1V6mD+xPt8qorgamsA2SIC2LfOgEbA2kkbHbh90Btw3qeE4QS0ozyPH39dGXWzAyVOsyN2mdNhoPKfiYYQGbS4qnmQw3Bm43U9Ke8qM7aW65ZxZ6jy1BYlcdZmnkQhq6vYC9RME3IFZ6Ebo3SHVGcIZeeWXq9vwTny9kvIfJO4X1HqzNcpmSSWfxfAG+1hS9A829g3AhYebaIf5j2ApoCJwsZTcX6KkUJaDqVGXOcOHo7BVuOklmppelL4KY11BsfqZxQyQl0jxIi+9tavbWueAe+D2LK4mDTk/qZh2kPb5CYwraV8h6lfqwAf4XEDgTXIYnnyqWPhGvq3cL5VPZ+vTiUocWFfTxQ+CvsqvhkFaOTkYytXD0V5/F/8zOMCDApn7eYpi3dcwWc2LAoxVQBBNgP7Jh0SUdOJTv8lgzRFQMnZSPucRUZwgOcyS5xtKeuR/Z+R2CsKYK1IebS4/XDLJwRWFYkek6rP02yTXctu1fMZeybZ45WAMpJuxpZpqI9AynYccBOZ59AZTJNqgDa4Gtj4slnQFeMPgrVzcWxjZj2ytsmYsmFa7bb93t3z0m5tX0Zru7nbTk+PccAL1kMtBbbvr9LXbPM47Gu2NwQ3Jre3oCl+VPTI8xe9VrLlvCNasJ8GwJhc0OOXa8hcLCaxb3PvO7H/OW0/BBXvEgKL83g8uWmFmofLzeRfu60DhKOLim5FbYWtcJqnNgVCkm21OP9vAAIDLZ6/wKVNyK/v+4avqz0gF8XuS/bB4g1ak3ULGfNR9draavVH2nKJFMNFZEDxdovG8i8LurbHC GCym+g6T f0iprm1wTMrrDCW8lXmwDbEoiocvWa7BWNuXPztVmdfnz6jp59WVE/oDAP9ZhTdesCPKqecEHl6G5KK7jzoxgBymEdHuh/wlV9qAVMup1xNXGhJCTCFA0qlKHVDa/oakjaWY0pOwxXk7DmfyyVUWdQnqmPJGSq0YTQJ+ucsq4Jrndkz7kepkJaHGiJU16oP+bzP+v3BxwNeZJb7xyMrzjhw9A4/gcpbPdMMgEa+9ZBs69YtWi67sbwEiEn5zoAH/Vz3gCeWqCEbeJ6DG2ENHyWBXdLr2V4MTXASRS08ML4ryhOpcH/Xr5Qfvt0WwE10W9BOImZJZADmJ7w5Tor91CjEo+z5Ljm9vXE6FdWtRa9YAZrsdQCLmNXA83eyz1gtAqDqGqn4N3OmnmNGEmi6yRzy80DdQKdOCKOWbf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add support for code tagging from dynamically loaded modules. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/codetag.h | 12 +++++++++ kernel/module/main.c | 4 +++ lib/codetag.c | 58 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 2 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index a9d7adecc2a5..386733e89b31 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -42,6 +42,10 @@ struct codetag_module { struct codetag_type_desc { const char *section; size_t tag_size; + void (*module_load)(struct codetag_type *cttype, + struct codetag_module *cmod); + void (*module_unload)(struct codetag_type *cttype, + struct codetag_module *cmod); }; struct codetag_iterator { @@ -68,4 +72,12 @@ void codetag_to_text(struct seq_buf *out, struct codetag *ct); struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); +#ifdef CONFIG_CODE_TAGGING +void codetag_load_module(struct module *mod); +void codetag_unload_module(struct module *mod); +#else +static inline void codetag_load_module(struct module *mod) {} +static inline void codetag_unload_module(struct module *mod) {} +#endif + #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index 36681911c05a..f400ba076cc7 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1242,6 +1243,7 @@ static void free_module(struct module *mod) { trace_module_free(mod); + codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -2978,6 +2980,8 @@ static int load_module(struct load_info *info, const char __user *uargs, /* Get rid of temporary copy. */ free_copy(info, flags); + codetag_load_module(mod); + /* Done! */ trace_module_load(mod); diff --git a/lib/codetag.c b/lib/codetag.c index 7708f8388e55..4ea57fb37346 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -108,15 +108,20 @@ static inline size_t range_size(const struct codetag_type *cttype, static void *get_symbol(struct module *mod, const char *prefix, const char *name) { char buf[64]; + void *ret; int res; res = snprintf(buf, sizeof(buf), "%s%s", prefix, name); if (WARN_ON(res < 1 || res > sizeof(buf))) return NULL; - return mod ? + preempt_disable(); + ret = mod ? (void *)find_kallsyms_symbol_value(mod, buf) : (void *)kallsyms_lookup_name(buf); + preempt_enable(); + + return ret; } static struct codetag_range get_section_range(struct module *mod, @@ -157,8 +162,11 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); - if (err >= 0) + if (err >= 0) { cttype->count += range_size(cttype, &range); + if (cttype->desc.module_load) + cttype->desc.module_load(cttype, cmod); + } up_write(&cttype->mod_lock); if (err < 0) { @@ -197,3 +205,49 @@ codetag_register_type(const struct codetag_type_desc *desc) return cttype; } + +void codetag_load_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) + codetag_module_init(cttype, mod); + mutex_unlock(&codetag_lock); +} + +void codetag_unload_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + struct codetag_module *found = NULL; + struct codetag_module *cmod; + unsigned long mod_id, tmp; + + down_write(&cttype->mod_lock); + idr_for_each_entry_ul(&cttype->mod_idr, cmod, tmp, mod_id) { + if (cmod->mod && cmod->mod == mod) { + found = cmod; + break; + } + } + if (found) { + if (cttype->desc.module_unload) + cttype->desc.module_unload(cttype, cmod); + + cttype->count -= range_size(cttype, &cmod->range); + idr_remove(&cttype->mod_idr, mod_id); + kfree(cmod); + } + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} From patchwork Mon Feb 12 21:38:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1726C4829D for ; Mon, 12 Feb 2024 21:40:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04CF06B0093; Mon, 12 Feb 2024 16:40:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F3D796B0095; Mon, 12 Feb 2024 16:40:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB68E6B0096; Mon, 12 Feb 2024 16:40:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C53A06B0093 for ; Mon, 12 Feb 2024 16:40:01 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 688F0A201D for ; Mon, 12 Feb 2024 21:40:01 +0000 (UTC) X-FDA: 81784469802.19.7AB8DDD Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf26.hostedemail.com (Postfix) with ESMTP id B3D0F140003 for ; Mon, 12 Feb 2024 21:39:59 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=uID8xtGr; spf=pass (imf26.hostedemail.com: domain of 3LpDKZQYKCK0fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3LpDKZQYKCK0fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707773999; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9alS6gtOXFejwBrY9h+FRijZZu8X6fVY6jwbxDwAezo=; b=Eqx1EdiCZJtO++mSUXTFuyWIUO1ag6DS2f8WmCiUEz4Gc2y74TD4Bn77Jo+fggT++EDZ4a /S3qXPIBCYZJwi5lnHjOeS/6/Ac0kt7Z467RgVm+cM28gzh/SI2+SNhURA0xoGEEBJewkq mhpUv9Ll2cT8a6sM2+PCOZaFpsrc+XU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707773999; a=rsa-sha256; cv=none; b=A+u5uA3ezqmCk5/xdjaFD2InxiXK01S77bEGRLUjEei8fsBbSpaMIlfQxbBWeZRbP/iHgz rRBSXOPIcoBxRQL6siohB+pL7B+MlN8P3HaEOw5GRPWohzW1te/nUsyCyMNqybE139PYgK r72kxQhBK1nI3zNieNDHj3lGQpNJSBw= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=uID8xtGr; spf=pass (imf26.hostedemail.com: domain of 3LpDKZQYKCK0fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3LpDKZQYKCK0fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5efe82b835fso84262837b3.0 for ; Mon, 12 Feb 2024 13:39:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707773999; x=1708378799; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9alS6gtOXFejwBrY9h+FRijZZu8X6fVY6jwbxDwAezo=; b=uID8xtGrXQVnEXNHTcNLxTS9O/0avr/sQkLrTKc9Ke7yl0i8vLm/iK2EbDYKU2w4rh X/NvSWuSYKi/mpWl/3uyPXi8+y6cjOWZT7yV1oQsbhlNSsINq9euARHBF4K9n8DygX7N ow/Z35dr9HvYyMmar/yqpiJ2WeVuC4sr8v50mqFlXoWhp3ZLX3JebS0ZlCOVuyTVHG6Y 5wsGPoXAIfJ8xDjWsdQuIdz5vvsJcnPZYDOq/go3McJG61+YLwWrbMtG8WuoqNUaJzsM lt1+qMNcpzFxe3Q/W7ciOVo1wjT57FjRDvmSZ+GEjdxinaPc3f+8SRaOMW6WQDW7KhEA nxVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707773999; x=1708378799; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9alS6gtOXFejwBrY9h+FRijZZu8X6fVY6jwbxDwAezo=; b=MgYa88pNHkH/yY7CkIU2Lxi4AKmz3kXQyA/LmxSLalQ4uys75bDXkvNBBjrXBGVI6O REqfG9sMGdPsb8SMNiZ1cfLiYx9FoOp20kNxiueZItviAj+6WD8cXmzXu9vpRlhDYQM7 cE2mWi/fumANfJGykkgk3Lxrl7b0xAHCRvqkTuYhMsmZ2Xfhv1t4kW5sRVt68LceLwJf Kyh3QxuZ5ijITlyYyM0p1Ms+c7opb6Ch/UXPq4FDNIufV1nk7CGu6Y4E64/Lj1YfD0c8 HiUmMnhInZO6eC6U4pK+dg/CHxMnU6LB7JQ50M8PIZaYZmzqlFeClxJjTIsZa4aydTiG g6Xg== X-Gm-Message-State: AOJu0Yzpxk59EHoQf+cX5QXupBV4Zc6gwYv9VIWGp7xK07aojC3NPuM0 pyrc2AYZfny0PTIlLh0gavC6UTorPVnH6COEkptJU7rf4RT0vqMK15WJPE+zfR8ZJjss7hClodc SyQ== X-Google-Smtp-Source: AGHT+IFW0cHHZhA7bV0V2zEhIEFhckUZZIzrQEg30Nhl0ohuGdOhgEJdFPgUoRF4LKS4dpvAz4HDei+0kVU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a0d:e844:0:b0:604:a67c:7f8d with SMTP id r65-20020a0de844000000b00604a67c7f8dmr2200694ywe.5.1707773998774; Mon, 12 Feb 2024 13:39:58 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:58 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-13-surenb@google.com> Subject: [PATCH v3 12/35] lib: prevent module unloading if memory is not freed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: B3D0F140003 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 41o39ierq1cfdfegczs3djkmo788tb9a X-HE-Tag: 1707773999-944780 X-HE-Meta: U2FsdGVkX1/W/A555moEZ8T7d1ZAVbiWBcYZgMFfCUWxFMBdO4wdRwtfs46H3T+rK4FyvzcUXp+KloFb5UG7iTdHu7y5SbTp/SE5CfS26sjnwHpw28nimk+6/nWrEbacGlwsFJ/daJiNSBdy/R/FCpWPxsOYebOFUCXQElp4aWZWnzuC1vrrPQb8+KkmWMoNr/0yJMnClCdKb+u0m4x2ZljlkKy+CvMco5yfzrhzsjYCkDtYwCUV4Av6iYiphC25yussNx5Qs/76mScdeZXQQTnez5ce7AXnITPq0kzc3CJxTQW6JTmgSYHjP6utOcvyRiBxjxVDWLJtR3AGNk6OxEFWtiSo7sPUocWOJjypFT2X+aSfGnhsuOX0beXRNqMAFj5ANG2zfrkXe+5XF+4mkyvCUuQLQWaRwdQ/VhQb8cVAbIvIrscHoP855cHQ50PyhyWOHPyGqYaRUxbEl2AAC6wa1PokLNw1x2eblHRK2qfI5WZ7uNohvFKKjc/1HA9x4ki8wbLdw+Gd2LYxKsPBZLvDkP2FpIYw04wiU2L73LNZ88tYaX3JOuWhA3bPbcT6mw+f2Keoj6930XDS22YvXFL/Xq3Y1duKAf3ZZmXilF52ud/NlQ5xhO3HlkLrBoJrpz6OO3DE/EgdPyVbD3XVQA9sa9na70d1a0G1uq+Vyi7cxPmE+uqjQMymrNphXrI4DVHpDyIAd4qHFazMdLtvq7QTW32uCqti0wNAWsCrC4REQZTUG/mdibcf2tBcNCswB4jbbUZJJDuJcrDcht+TaAhNFymuyftUdlaO8sRVYOAyJeP6ZWl2aREhhxO1uQlCTzABw7FwfocYKRI/m530PWQAnzp+dhLxnxoV3jqQg1JKWiOpfYNFSz3SJq1Sg7NLlTYvEpTSbj+SYmUaiPi5LW0cxhgqjvDIDVFM8vZxKwQw+mDTqIJEqLqMcYrOmmc5in8Y+WokLpt/CfCdvP2 gEs/GNFv 5VrDLdXQhERkvgiwzNjdOT0kG+YIyPbY+39Fq7KTYLjMhrBy1pzRFLed6C5qU5YJKv9JO1F5iqJ70uTc56qy4JvkX9Vbz0n9szgW4VOOLR9aIqk34FXGjLlXUrYTtCkO7KeJUtxlDOD+03SuWj4GWsQqsMDKPYDSEHFcSY5LUDh2iSVN6zhCcVlL3tbxzYlIuNeCZ9E6iQEcfceNWp3Be829F10v5jRQ4dAw92eUwPsA/FXPhNYiqQjeCZ23XGXOCQQRowmU3bN9A6h8VHus9eonbW+glGHsxCmPu1Xx5Kkzbmj6BkrvH6f1id/D2rB9tDe7OXjNEyt2l80EEjwxk4JpgAtHHmaTmaK7g5U5FyGq/tWj8C+F/Eke72DygNkN9WaFLEesCPEauc5iqBQ/guOY32vJTzOz+7dHooW0daLjpCs8ThqsFBBk7Gf72HxKzSG8KwPAwy+0p1UDmCmcQB48OZfG15KHGXBHVP/nIW9/NwVmY6TkgSKmqy9MTT5o0KYnwaeWSKc2KRtq7ySWKCjPxmjIrsvbqLZgonpBlw4zqW/PIz80luNYIGCZeNvznwgMPzQ/+gxTKWpDWU3NAXw6z3DZA9HTF12YXMSWf0EX63dzvzcES9qkDTNf6dC3vXkm1m6O7Tf40xrR0BHyWxs/KkNEN5+SByP71Z7lJmcw9lhc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF. Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 6 +++--- kernel/module/main.c | 23 +++++++++++++++-------- lib/codetag.c | 11 ++++++++--- 3 files changed, 26 insertions(+), 14 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index 386733e89b31..d98e4c8e86f0 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -44,7 +44,7 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - void (*module_unload)(struct codetag_type *cttype, + bool (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); }; @@ -74,10 +74,10 @@ codetag_register_type(const struct codetag_type_desc *desc); #ifdef CONFIG_CODE_TAGGING void codetag_load_module(struct module *mod); -void codetag_unload_module(struct module *mod); +bool codetag_unload_module(struct module *mod); #else static inline void codetag_load_module(struct module *mod) {} -static inline void codetag_unload_module(struct module *mod) {} +static inline bool codetag_unload_module(struct module *mod) { return true; } #endif #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index f400ba076cc7..658b631e76ad 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1211,15 +1211,19 @@ static void *module_memory_alloc(unsigned int size, enum mod_mem_type type) return module_alloc(size); } -static void module_memory_free(void *ptr, enum mod_mem_type type) +static void module_memory_free(void *ptr, enum mod_mem_type type, + bool unload_codetags) { + if (!unload_codetags && mod_mem_type_is_core_data(type)) + return; + if (mod_mem_use_vmalloc(type)) vfree(ptr); else module_memfree(ptr); } -static void free_mod_mem(struct module *mod) +static void free_mod_mem(struct module *mod, bool unload_codetags) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1230,20 +1234,23 @@ static void free_mod_mem(struct module *mod) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod_mem->base, type); + module_memory_free(mod_mem->base, type, + unload_codetags); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA); + module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA, unload_codetags); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { + bool unload_codetags; + trace_module_free(mod); - codetag_unload_module(mod); + unload_codetags = codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -1285,7 +1292,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod); + free_mod_mem(mod, unload_codetags); } void *__symbol_get(const char *symbol) @@ -2298,7 +2305,7 @@ static int move_module(struct module *mod, struct load_info *info) return 0; out_enomem: for (t--; t >= 0; t--) - module_memory_free(mod->mem[t].base, t); + module_memory_free(mod->mem[t].base, t, true); return ret; } @@ -2428,7 +2435,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod); + free_mod_mem(mod, true); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/codetag.c b/lib/codetag.c index 4ea57fb37346..0ad4ea66c769 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -5,6 +5,7 @@ #include #include #include +#include struct codetag_type { struct list_head link; @@ -219,12 +220,13 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -void codetag_unload_module(struct module *mod) +bool codetag_unload_module(struct module *mod) { struct codetag_type *cttype; + bool unload_ok = true; if (!mod) - return; + return true; mutex_lock(&codetag_lock); list_for_each_entry(cttype, &codetag_types, link) { @@ -241,7 +243,8 @@ void codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - cttype->desc.module_unload(cttype, cmod); + if (!cttype->desc.module_unload(cttype, cmod)) + unload_ok = false; cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); @@ -250,4 +253,6 @@ void codetag_unload_module(struct module *mod) up_write(&cttype->mod_lock); } mutex_unlock(&codetag_lock); + + return unload_ok; } From patchwork Mon Feb 12 21:38:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 080A4C4829D for ; Mon, 12 Feb 2024 21:40:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26EE56B0095; Mon, 12 Feb 2024 16:40:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 21E776B0096; Mon, 12 Feb 2024 16:40:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F17136B0098; Mon, 12 Feb 2024 16:40:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D821A6B0095 for ; Mon, 12 Feb 2024 16:40:03 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A830CC0A36 for ; Mon, 12 Feb 2024 21:40:03 +0000 (UTC) X-FDA: 81784469886.06.AF17C65 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id B98F8100014 for ; Mon, 12 Feb 2024 21:40:01 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FTDbUvgu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3MJDKZQYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3MJDKZQYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774001; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8Z7QS+MaYqtJtKeOfyLYXivI4+YAu2oBG/bydtMi8hs=; b=euMXuYAUHZDRF8bN/j/9X912CXFRNnG5DjyQBsxjFdkkTtP4Y7WM8doFgvJIMuU08ouCIF 8N9oOFmQ7xmVCOat9jVnYUgEox8vGSXESJYQ+YqydLY2nN965yjJ8hFQJ2FDa6snZgnEl1 nyiJShdSQCC8M9xVFDNfwHbdsELI028= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FTDbUvgu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3MJDKZQYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3MJDKZQYKCK8hjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774001; a=rsa-sha256; cv=none; b=5Of6SUZUhDOxF9kBa5J72oOYjTOu6/uo/nOUasb4+wZF1e7g1/vPfx0VdLF0ZxpIgMfsZS OgBsJW5AkEUVIYJPx3r+smPakWqpNlNKxKwfmGsCoEjmK/yd6tz2ZFn3rOpf5Os+vWB7qs CG38Z2I0JyFwmqPDpAgl+gjkhXFFy9U= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5fc6463b0edso62934997b3.0 for ; Mon, 12 Feb 2024 13:40:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774001; x=1708378801; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8Z7QS+MaYqtJtKeOfyLYXivI4+YAu2oBG/bydtMi8hs=; b=FTDbUvgu8fHYCox9QUYGQ6Mbvbft6xtBkczUOMepzMmDDHWuI5ONN4IJa+FpCReDub 2qfZJZ6tcGZ2MN/TQ0DsDuTNGS+2NeVCiDefJQ1+wYxe1UpJMI9JnIpFKUO7aEdi+Y9s /7QxzX03gpPLMZvq9rbVBXjq2V1on2zE5EqOlIJcqxl5ZRbEbB7z3FNJoJO1jEe/xnRC CDfl9qwaxmH/OLH6834C3CFhNQdu+y+NcdBkGW600Gc7vQgaEvcgdkOeueZYcTeJFKYy LfmrzqQOD813OMfK/6h7jrdXf4adPY90wm2sDjHogI5lKxBo4aR+pmRKt3hWSmkBzPFT 18Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774001; x=1708378801; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8Z7QS+MaYqtJtKeOfyLYXivI4+YAu2oBG/bydtMi8hs=; b=aAKtuDYfWm4m8vQvHHJ2Llu5FN4zNp2OVzKBwgrKFNnKhE7EoMlDEwp1cxiVbP32/2 fx55EWCn4i39JcioFeUIxSr6KlyMryPdE1s4CbEEjctslwuC8aGyvNIE45BGBiip1G1x Ebc/zPqpORcfcwuRr45F2VkbwFEc9FNaaQgFlOjTALNiAI6qecEjdMoUu65tZt8MQh1y W9drLoQQuYosi4Ut8p0hTKDOpAg7m0jUxtFh42ma1jnELWSDMwXQmlnZcaItv99Ucx7I FSTPUbR5eZVKWPLtLLhUeqe8OkVlH0JW7d8OtK3+1rP48dpB148ts46Vx3Fr4VDZxouH MjOA== X-Gm-Message-State: AOJu0Yxq5mMyMjRXIAKt3dIDAPc8JzGWRo9T1It2k9c4uIIEbHJ9TrSd bLl4IIce4jsmvkuneQnvgCloQeI3bvwK5o2uRv82ROYgbEJZQ8Mj5WeTjhjLqIdxULaNGJ8Rjo/ TrQ== X-Google-Smtp-Source: AGHT+IFvMqjoVXc9TXkfAVsbZzdsmiP9oJg/iMrpDi1bTkoQckFdveXZ0cOEOyNqrIOK7Jlnp3PXZ85YLhw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:690c:908:b0:604:c668:5012 with SMTP id cb8-20020a05690c090800b00604c6685012mr1435458ywb.6.1707774000847; Mon, 12 Feb 2024 13:40:00 -0800 (PST) Date: Mon, 12 Feb 2024 13:38:59 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-14-surenb@google.com> Subject: [PATCH v3 13/35] lib: add allocation tagging support for memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: B98F8100014 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: nn8ui3mkbeq6zts9cxkwd5so6ujf8qo1 X-HE-Tag: 1707774001-154121 X-HE-Meta: U2FsdGVkX1+00vZckYZanqSRS9K1yLpVXG/FHuPeD4f/WLvKe9PYHo6PrnkSHzw4KxoReCglVJGTbWk+lxTLbFAmmNko0OdTI7gkSvQyeJFL+Cpbzhwj/iqj50d1afY0TJ3yOnFn5FZqw8T4LfuqTKToWh7OAzYe4xvUpfBhHdxhhxpP8U81ceYNC1x8rLEji4oE0Y+0VwE9WoWKWjXo1PR6OUn4rWSx1bxPxTJVkXTL2Sf9dp+QBPcx4FkMFaf7BkwEr9JLitE/EYcMzZ5gifPTna+Zwc+Xt3ZsRbSnpzq132zzkS4lmDNGFkFztm0br3Z/1sGQMd4ZjB2ct3OOZT4ifl0lb7NgcT29eQSC2VU6D/+L/9N7+Lgc1itUGYBZk/cAlcgS2d0BNEljtoeznLg43yE3T3RxCuwplN0sl01ARm9FI6xQcsBViEnv6m+dN5C1ERkylONXhkUVGmXn2H7OFrfnVKwOqI1kg/wEHfko1P/LH+hKk7ZNmoz8z9wlF+EL4PNSE77vTlMJ6YJb2r58zV6Kvr0nSY1U+F58ECDQofqd8EBX17cjZzl2ytu3Qy40yFuzEL0opAucWMacphWd9erol/FIx6a6cIhboBd975n9wZuBYpxOqJO+OtPPOgqg8iLUvztXpg+WaEy8hMdLS+gar9A/nBcW3ah08pO/HSMKzEQqPJsY89PalG/bIpCxjd8Aw1aEIBHxQEylZPYTfwx2xBbaNPNafvWHvsKgsUQ77vrppuTZNUy8CBnBKuRZzZzcs4Um4yIpLDw5eAuNRiCHbnQK2L9sOqOO/T/z5rYjJPr6chNUk5G91xnNxBDyFWFXxQELSM1fxq03BDNibgqHFQokjRXYFFhfgtPCyACuDk929NVzbMVGqrTbzxdvK7g4LGl2TjDAI8DDZjuFRXIW0iRxGJrNt0S3SCbaDLSS2iyvog7RR007gpTMAHlzXTxiZdI7K7pe4Md lKvOHg6W uSu3dbGI5vJy9Z4qBz4fOZdcmVWw7/WBxxC0nmpfgCd8s+psuQArVc10C0J6xprOqb5eFlGC2h5wKK6jaFltJy6sFlmqPFOTRUUdz5NvrHV2z82iKwbTRXZOmXRM6m/0suDx7gHF/eAbmbY6HIruGuCZMW4sPAMAat8LZFpLwu3S65nu1pRLsX935U49UxagrO1Y1WvBRlxWbjJjhf2Ug/s4eOq3GKr0c83cgaTRo7biH3JvZEu9vdSqFo0Fro9YrWZhwiUTJ5ajgeiUGXTjlycrHzgBWvvLzQONEzdCAWaPmDEByjRc9fPJMnrnyMWtc7REW/B/3HbPf23mE8Vvyxwtgre5b5laL8o+lK9175SGs8S2X/xIGqde+SuVUJEyq/6QwL7P/1gl2zHtAlMYswQs+aYExAKfGgEabP23MUVJBax2z4dPgytog2OM+24B8ZD2XKGPzhhlB6eztitV1hZpJANUGhxLPGUojl/gR34oCaWgC44UzyreUBjEjnjIS1reBYsS9E1GtE3AGQaE7aN4yZ1pHhwwlGFIfK/EyBOYyQedHg0IImTjQE9Lo+GU66iN2A+AAwYl0EmKx7TjvCQTfwzWv9LRuXU9yVG+XjVLaA/DMeRBunAif0aOfTsNnRDBrqzBETSroNLx8b4aFqWTthFgVhgwrXCuJJI223ODqNhI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce CONFIG_MEM_ALLOC_PROFILING which provides definitions to easily instrument memory allocators. It registers an "alloc_tags" codetag type with /proc/allocinfo interface to output allocation tag information when the feature is enabled. CONFIG_MEM_ALLOC_PROFILING_DEBUG is provided for debugging the memory allocation profiling instrumentation. Memory allocation profiling can be enabled or disabled at runtime using /proc/sys/vm/mem_profiling sysctl when CONFIG_MEM_ALLOC_PROFILING_DEBUG=n. CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT enables memory allocation profiling by default. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- Documentation/admin-guide/sysctl/vm.rst | 16 +++ Documentation/filesystems/proc.rst | 28 +++++ include/asm-generic/codetag.lds.h | 14 +++ include/asm-generic/vmlinux.lds.h | 3 + include/linux/alloc_tag.h | 133 ++++++++++++++++++++ include/linux/sched.h | 24 ++++ lib/Kconfig.debug | 25 ++++ lib/Makefile | 2 + lib/alloc_tag.c | 158 ++++++++++++++++++++++++ scripts/module.lds.S | 7 ++ 10 files changed, 410 insertions(+) create mode 100644 include/asm-generic/codetag.lds.h create mode 100644 include/linux/alloc_tag.h create mode 100644 lib/alloc_tag.c diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index c59889de122b..a214719492ea 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -43,6 +43,7 @@ Currently, these files are in /proc/sys/vm: - legacy_va_layout - lowmem_reserve_ratio - max_map_count +- mem_profiling (only if CONFIG_MEM_ALLOC_PROFILING=y) - memory_failure_early_kill - memory_failure_recovery - min_free_kbytes @@ -425,6 +426,21 @@ e.g., up to one or two maps per allocation. The default value is 65530. +mem_profiling +============== + +Enable memory profiling (when CONFIG_MEM_ALLOC_PROFILING=y) + +1: Enable memory profiling. + +0: Disabld memory profiling. + +Enabling memory profiling introduces a small performance overhead for all +memory allocations. + +The default value depends on CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. + + memory_failure_early_kill: ========================== diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 104c6d047d9b..40d6d18308e4 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -688,6 +688,7 @@ files are there, and which are missing. ============ =============================================================== File Content ============ =============================================================== + allocinfo Memory allocations profiling information apm Advanced power management info bootconfig Kernel command line obtained from boot config, and, if there were kernel parameters from the @@ -953,6 +954,33 @@ also be allocatable although a lot of filesystem metadata may have to be reclaimed to achieve this. +allocinfo +~~~~~~~ + +Provides information about memory allocations at all locations in the code +base. Each allocation in the code is identified by its source file, line +number, module and the function calling the allocation. The number of bytes +allocated at each location is reported. + +Example output. + +:: + + > cat /proc/allocinfo + + 153MiB mm/slub.c:1826 module:slub func:alloc_slab_page + 6.08MiB mm/slab_common.c:950 module:slab_common func:_kmalloc_order + 5.09MiB mm/memcontrol.c:2814 module:memcontrol func:alloc_slab_obj_exts + 4.54MiB mm/page_alloc.c:5777 module:page_alloc func:alloc_pages_exact + 1.32MiB include/asm-generic/pgalloc.h:63 module:pgtable func:__pte_alloc_one + 1.16MiB fs/xfs/xfs_log_priv.h:700 module:xfs func:xlog_kvmalloc + 1.00MiB mm/swap_cgroup.c:48 module:swap_cgroup func:swap_cgroup_prepare + 734KiB fs/xfs/kmem.c:20 module:xfs func:kmem_alloc + 640KiB kernel/rcu/tree.c:3184 module:tree func:fill_page_cache_func + 640KiB drivers/char/virtio_console.c:452 module:virtio_console func:alloc_buf + ... + + meminfo ~~~~~~~ diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h new file mode 100644 index 000000000000..64f536b80380 --- /dev/null +++ b/include/asm-generic/codetag.lds.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_GENERIC_CODETAG_LDS_H +#define __ASM_GENERIC_CODETAG_LDS_H + +#define SECTION_WITH_BOUNDARIES(_name) \ + . = ALIGN(8); \ + __start_##_name = .; \ + KEEP(*(_name)) \ + __stop_##_name = .; + +#define CODETAG_SECTIONS() \ + SECTION_WITH_BOUNDARIES(alloc_tags) + +#endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 5dd3a61d673d..c9997dc50c50 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -50,6 +50,8 @@ * [__nosave_begin, __nosave_end] for the nosave data */ +#include + #ifndef LOAD_OFFSET #define LOAD_OFFSET 0 #endif @@ -366,6 +368,7 @@ . = ALIGN(8); \ BOUNDED_SECTION_BY(__dyndbg_classes, ___dyndbg_classes) \ BOUNDED_SECTION_BY(__dyndbg, ___dyndbg) \ + CODETAG_SECTIONS() \ LIKELY_PROFILE() \ BRANCH_PROFILE() \ TRACE_PRINTKS() \ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h new file mode 100644 index 000000000000..cf55a149fa84 --- /dev/null +++ b/include/linux/alloc_tag.h @@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * allocation tagging + */ +#ifndef _LINUX_ALLOC_TAG_H +#define _LINUX_ALLOC_TAG_H + +#include +#include +#include +#include +#include +#include +#include + +struct alloc_tag_counters { + u64 bytes; + u64 calls; +}; + +/* + * An instance of this structure is created in a special ELF section at every + * allocation callsite. At runtime, the special section is treated as + * an array of these. Embedded codetag utilizes codetag framework. + */ +struct alloc_tag { + struct codetag ct; + struct alloc_tag_counters __percpu *counters; +} __aligned(8); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) +{ + return container_of(ct, struct alloc_tag, ct); +} + +#ifdef ARCH_NEEDS_WEAK_PER_CPU +/* + * When percpu variables are required to be defined as weak, static percpu + * variables can't be used inside a function (see comments for DECLARE_PER_CPU_SECTION). + */ +#error "Memory allocation profiling is incompatible with ARCH_NEEDS_WEAK_PER_CPU" +#endif + +#define DEFINE_ALLOC_TAG(_alloc_tag, _old) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ + static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; \ + struct alloc_tag * __maybe_unused _old = alloc_tag_save(&_alloc_tag) + +DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static inline bool mem_alloc_profiling_enabled(void) +{ + return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + &mem_alloc_profiling_key); +} + +static inline struct alloc_tag_counters alloc_tag_read(struct alloc_tag *tag) +{ + struct alloc_tag_counters v = { 0, 0 }; + struct alloc_tag_counters *counter; + int cpu; + + for_each_possible_cpu(cpu) { + counter = per_cpu_ptr(tag->counters, cpu); + v.bytes += counter->bytes; + v.calls += counter->calls; + } + + return v; +} + +static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + struct alloc_tag *tag; + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN_ONCE(ref && !ref->ct, "alloc_tag was not set\n"); +#endif + if (!ref || !ref->ct) + return; + + tag = ct_to_alloc_tag(ref->ct); + + this_cpu_sub(tag->counters->bytes, bytes); + this_cpu_dec(tag->counters->calls); + + ref->ct = NULL; +} + +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + __alloc_tag_sub(ref, bytes); +} + +static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) +{ + __alloc_tag_sub(ref, bytes); +} + +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN_ONCE(ref && ref->ct, + "alloc_tag was not cleared (got tag for %s:%u)\n",\ + ref->ct->filename, ref->ct->lineno); + + WARN_ONCE(!tag, "current->alloc_tag not set"); +#endif + if (!ref || !tag) + return; + + ref->ct = &tag->ct; + this_cpu_add(tag->counters->bytes, bytes); + this_cpu_inc(tag->counters->calls); +} + +#else + +#define DEFINE_ALLOC_TAG(_alloc_tag, _old) +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} +static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) {} +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, + size_t bytes) {} + +#endif + +#endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index ffe8f618ab86..da68a10517c8 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -770,6 +770,10 @@ struct task_struct { unsigned int flags; unsigned int ptrace; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif + #ifdef CONFIG_SMP int on_cpu; struct __call_single_node wake_entry; @@ -810,6 +814,7 @@ struct task_struct { struct task_group *sched_task_group; #endif + #ifdef CONFIG_UCLAMP_TASK /* * Clamp values requested for a scheduling entity. @@ -2183,4 +2188,23 @@ static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); } extern void sched_set_stop_task(int cpu, struct task_struct *stop); +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) +{ + swap(current->alloc_tag, tag); + return tag; +} + +static inline void alloc_tag_restore(struct alloc_tag *tag, struct alloc_tag *old) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN(current->alloc_tag != tag, "current->alloc_tag was changed:\n"); +#endif + current->alloc_tag = old; +} +#else +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) { return NULL; } +#define alloc_tag_restore(_tag, _old) +#endif + #endif diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 0be2d00c3696..78d258ca508f 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -972,6 +972,31 @@ config CODE_TAGGING bool select KALLSYMS +config MEM_ALLOC_PROFILING + bool "Enable memory allocation profiling" + default n + depends on PROC_FS + depends on !DEBUG_FORCE_WEAK_PER_CPU + select CODE_TAGGING + help + Track allocation source code and record total allocation size + initiated at that code location. The mechanism can be used to track + memory leaks with a low performance and memory impact. + +config MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + bool "Enable memory allocation profiling by default" + default y + depends on MEM_ALLOC_PROFILING + +config MEM_ALLOC_PROFILING_DEBUG + bool "Memory allocation profiler debugging" + default n + depends on MEM_ALLOC_PROFILING + select MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + help + Adds warnings with helpful error messages for memory allocation + profiling. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 6b48b22fdfac..859112f09bf5 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -236,6 +236,8 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_CODE_TAGGING) += codetag.o +obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o + lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c new file mode 100644 index 000000000000..4fc031f9cefd --- /dev/null +++ b/lib/alloc_tag.c @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include + +static struct codetag_type *alloc_tag_cttype; + +DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static void *allocinfo_start(struct seq_file *m, loff_t *pos) +{ + struct codetag_iterator *iter; + struct codetag *ct; + loff_t node = *pos; + + iter = kzalloc(sizeof(*iter), GFP_KERNEL); + m->private = iter; + if (!iter) + return NULL; + + codetag_lock_module_list(alloc_tag_cttype, true); + *iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(iter)) != NULL && node) + node--; + + return ct ? iter : NULL; +} + +static void *allocinfo_next(struct seq_file *m, void *arg, loff_t *pos) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + struct codetag *ct = codetag_next_ct(iter); + + (*pos)++; + if (!ct) + return NULL; + + return iter; +} + +static void allocinfo_stop(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)m->private; + + if (iter) { + codetag_lock_module_list(alloc_tag_cttype, false); + kfree(iter); + } +} + +static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + s64 bytes = counter.bytes; + char val[10], *p = val; + + if (bytes < 0) { + *p++ = '-'; + bytes = -bytes; + } + + string_get_size(bytes, 1, + STRING_SIZE_BASE2|STRING_SIZE_NOSPACE, + p, val + ARRAY_SIZE(val) - p); + + seq_buf_printf(out, "%8s %8llu ", val, counter.calls); + codetag_to_text(out, ct); + seq_buf_putc(out, ' '); + seq_buf_putc(out, '\n'); +} + +static int allocinfo_show(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + char *bufp; + size_t n = seq_get_buf(m, &bufp); + struct seq_buf buf; + + seq_buf_init(&buf, bufp, n); + alloc_tag_to_text(&buf, iter->ct); + seq_commit(m, seq_buf_used(&buf)); + return 0; +} + +static const struct seq_operations allocinfo_seq_op = { + .start = allocinfo_start, + .next = allocinfo_next, + .stop = allocinfo_stop, + .show = allocinfo_show, +}; + +static void __init procfs_init(void) +{ + proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); +} + +static bool alloc_tag_module_unload(struct codetag_type *cttype, + struct codetag_module *cmod) +{ + struct codetag_iterator iter = codetag_get_ct_iter(cttype); + struct alloc_tag_counters counter; + bool module_unused = true; + struct alloc_tag *tag; + struct codetag *ct; + + for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { + if (iter.cmod != cmod) + continue; + + tag = ct_to_alloc_tag(ct); + counter = alloc_tag_read(tag); + + if (WARN(counter.bytes, "%s:%u module %s func:%s has %llu allocated at module unload", + ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) + module_unused = false; + } + + return module_unused; +} + +static struct ctl_table memory_allocation_profiling_sysctls[] = { + { + .procname = "mem_profiling", + .data = &mem_alloc_profiling_key, +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + .mode = 0444, +#else + .mode = 0644, +#endif + .proc_handler = proc_do_static_key, + }, + { } +}; + +static int __init alloc_tag_init(void) +{ + const struct codetag_type_desc desc = { + .section = "alloc_tags", + .tag_size = sizeof(struct alloc_tag), + .module_unload = alloc_tag_module_unload, + }; + + alloc_tag_cttype = codetag_register_type(&desc); + if (IS_ERR_OR_NULL(alloc_tag_cttype)) + return PTR_ERR(alloc_tag_cttype); + + register_sysctl_init("vm", memory_allocation_profiling_sysctls); + procfs_init(); + + return 0; +} +module_init(alloc_tag_init); diff --git a/scripts/module.lds.S b/scripts/module.lds.S index bf5bcf2836d8..45c67a0994f3 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -9,6 +9,8 @@ #define DISCARD_EH_FRAME *(.eh_frame) #endif +#include + SECTIONS { /DISCARD/ : { *(.discard) @@ -47,12 +49,17 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) + CODETAG_SECTIONS() } .rodata : { *(.rodata .rodata.[0-9a-zA-Z_]*) *(.rodata..L*) } +#else + .data : { + CODETAG_SECTIONS() + } #endif } From patchwork Mon Feb 12 21:39:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554052 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F02B2C4829D for ; Mon, 12 Feb 2024 21:40:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 222256B0096; Mon, 12 Feb 2024 16:40:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AB476B0098; Mon, 12 Feb 2024 16:40:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF5396B0099; Mon, 12 Feb 2024 16:40:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DB3A86B0096 for ; Mon, 12 Feb 2024 16:40:05 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B27F6A1FD7 for ; Mon, 12 Feb 2024 21:40:05 +0000 (UTC) X-FDA: 81784469970.18.77E741F Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf17.hostedemail.com (Postfix) with ESMTP id E383B40017 for ; Mon, 12 Feb 2024 21:40:03 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fJRto4yO; spf=pass (imf17.hostedemail.com: domain of 3M5DKZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3M5DKZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774003; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kt8cg7NInhDJR3c6J+BP4SUIPENq39NpOtUeNq0nwQE=; b=GR+kH9b4l31kAkGBkjujHZaFZsDANHe4UsQJurhf+rFOgrNGOq9nkYbK0M1Kf+JxXva2+B mUyFrMH28pJvVmrC+O8Oz7wYA/V0Ck5yxfxECNJW4do/p0spQHjX+nD1xF7ETU1EZ9d3t3 F4jKWtJHfLeYXQ8VsvrwsaoHtpwMiKk= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fJRto4yO; spf=pass (imf17.hostedemail.com: domain of 3M5DKZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3M5DKZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774003; a=rsa-sha256; cv=none; b=GDi15MMqqCv9Zt9RV2XgGjlnuApXb6wSnuMojnafsEolx60/W63tgwhrwdHH5tBihCK2RW rlgj+ZV+tbA/MpwUCgYKM9HtvToo2b8rdoMN1py+0USaSUDr09Y8vcBEhVRonKh8Zq3l2A NHIZt+E6zomWshjJYGvEsfQumvSGR5k= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6077ca422d2so6811467b3.1 for ; Mon, 12 Feb 2024 13:40:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774003; x=1708378803; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kt8cg7NInhDJR3c6J+BP4SUIPENq39NpOtUeNq0nwQE=; b=fJRto4yOfjAymhmU00XKpEJnkuaLdVGPRBVgfzdm80pmcBfjthjU8d8df6QTUheJUF EUTYvrj75oJ7YywQ8/TxsXtaptkWXKCOkt8rV0DiSU4Ht+7sdeOJx1ssS7AZLXXxVzVK L3KFW1VpR2hPa98rzaWUCVTBk691HW7Em1YvDZmPe77ShW9WlL0Ebu1PHU3pV9AoZXOw e8RU7ZmMfcJ/2MO0Z/GMi2oNA1fZ/5TYiqsI37kZOi3BvjDyolfuh8ZrpeEEZmhqprwb UJpK1dz9pvzsRVUjIbEFJ692+0OPFNXcVVdxtz31ljdmfDqYA8E74NNLiwD/nJgNnio8 2eYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774003; x=1708378803; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kt8cg7NInhDJR3c6J+BP4SUIPENq39NpOtUeNq0nwQE=; b=uTqgaNYcCb3yniM5l7KQH3Pdk7upOMrzar9pZLaWHr9NXOhL2geExlY9WcSv8JdxNU WplRsq5llF0UmOzFvhfHo5PPgGhlwNS37tGcQFEES801JWuj9Z5e8jFjjH8Jpa538gVp oqT4WpIDQqgVeSeWOIngvDgmi5chC5GFhF7RYKUjH2MIcq+G0RCo5VSV+dg6lcinBVNM HGUrnenjxKT7iMvNjN4K3CGvSrOZCW5JPSlcuWQ9nEV6UChARj0IIueiXUxzr0BNx1fW IkRT5uyaqqCnQ3VEaHw9NitqKx9wSlP8Id+7RiMomJbzcBIEVOsmNDV8bPvzov+18EI3 DdXg== X-Forwarded-Encrypted: i=1; AJvYcCVIFWckLkjVZGt8tEdVBHFXaDAewkDpe5Wsqyfsw9X8kPGRbmIMntLTEsdMqq6GR8SdqKGNpXkX7RQDMgJpaK1Yr/4= X-Gm-Message-State: AOJu0Yy/VFmed3Fjl+0Av4WmRFGbHnv+7JfHGXIx45BLSQqPtrVy1rkG 3fWIhIbmVscRiPF/ZqXlgtftoNYQIuqm9AWXgfI+EZ0MuxIsHtVXJ89g4qQtZpgYqBgT7qP1dN7 FhQ== X-Google-Smtp-Source: AGHT+IGLgFnBLQzDqMGIcoBa9Zqezua8Q6ZnRTqobe7m2G77O75/0lc/m1eR/Azu5VyYD/x/WOMy71Jje+w= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:690c:c07:b0:604:42a3:3adc with SMTP id cl7-20020a05690c0c0700b0060442a33adcmr2207535ywb.10.1707774003026; Mon, 12 Feb 2024 13:40:03 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:00 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-15-surenb@google.com> Subject: [PATCH v3 14/35] lib: introduce support for page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: E383B40017 X-Rspam-User: X-Stat-Signature: iyyc7bwqxm3iaukhgq834qieuqcr979w X-Rspamd-Server: rspam01 X-HE-Tag: 1707774003-334261 X-HE-Meta: U2FsdGVkX19CRTJ9pdvpzEd9+irlkwyTSZzaHsuMLwbbUGYUQeVMfNJpuW/5SPFJdkMcTPXuxEaN4PycX+6n9RvU4JiyQIsBWE2ieCgYSGVTa94ShTXUdTdGSE5y9hrellsLNuxDqSvtfAsCxtY8xntOr+8C3XS/ZlezyDfCf643p1ib4hlK5nQF+T2i/uB+1N7DXsFLnGDn/FF2Y7m4bKRREgj8OJ2jqYqu4VlFV//ADbTvtXYjFZUmx96Pwl/LTdkX5jwTkUL1RiERI8ANoRvnjo0h4n+uaG19svlPpWYm9TIjMX5/AopAbfmP6+gGx9aMZcNjPP0KZMMgi/zOp7hm4M4QD/sxU6M4mDAQZLDTzJ9Akyy/sPV6jXbwvgZiBjkhtTjajjypuiIci/olyympuOMxM3KL+72R52ftDeKDnTCQ/dVw3XSYIX9eWVD/p9z+MmWb0cd8jzSC8SLgrSTjZnzRD5x5NA4y1jVHJ36fPiKbyb2mkM5UFYf8JoajKCDlFNv1WtzGne6sRqSk09iqwuniCStxZ8QYXW4Z34B31k35qYDbR5xnM0tJxLc19UbQ0FWWb+BCI30gWYSUXcst/yJ7lp6b3uAEciaGQvhhZU7gzNXzePGgUXnFRX4mX/J5vqDaflieS4SGk59r3WQayrcZrWfD85WKbDqxyj4UVZGLTdjpJdZeCuWnGGuHHTJajXY+04poYFoUmBFtzOPHPKPvx81SsGWJFZKyTvnKNSyFIWclYR/M/ZyhDrv8j5z7ec4h1B3XZ3320uFz69r+1yXHYjO9U8vU0nxVgjgUdupklsqlDWd2Sel2Ei+XaYwwSwswc1fKFWjs7Pd2lM6ou1COqX4Kp+o6L70jJoLw5tGtwvyJrqzu1fe9E3fOk8ZcHMA3eV9SKdIphtPvFOI1D9W9LF6qNldujMVXJocCPfl2arDhZIXJIW2eFHmCI9Tc2EulwRsmfCTOdXo nf580wl3 IlljkWYDFFdGcdftReb9pPVsQZOOdRhweX+RYQMN+ixlZgEiBOvWPp4mI+Eaoi0SZ0yK9GTNGSw9g1eMd4AA6ZajCni4ATtLxbrzb/juExWdXl7PM6ODmQv6SlqVtbtDS0FQ9gjOlyXzryjT8A8rRCgm8k9h+RiLsNLHiSO/RLC+oweyEXAgcpRmI2HpMlSc6Q6qwxMDtoI3nYMADfJD2SIg7ZXCPRj35otcwlvC7LvwQBZFj9OFhjUPh7X57GfeD+EQK79Eq/QZ207NMDxT83XyyzukXJRuROPnuOyreoHyecfAvUWtnV370MocY+V6ZAgHN499/YoH+G4WCcJppIYz1WCPqCP26nGpDH0T1fpoSDorh/Z1X/jRWQF07R3bG+EW9KQEBgYOpTGpmAb6/HguKIxDlL4HWapFOGrEsSqpZ/EeiZVws1TEop8qoz0fsBmgW574B4fVJ+bEDUcD2TNngRz5PGLkf8Wdjy9Qs/YJiP2/Ck/blm7y5entEZ+qkDbqdSYLmceC4a5D08cUSDnNIwahJm/XTd21ZfyeXXcd3jyxKQPEDMJz8+VoQ9KxDDPiPLQJw7TbTYbgvclEuzICYKve90LnV62W57SlrTdoFhe61i9LDBQE0r6CIWlNEZXZycjTVCQi87utqbhsyAeysuE0rmQiOAPZB3NeqBHIDZuI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a page_ext field. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/page_ext.h | 1 - include/linux/pgalloc_tag.h | 73 +++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 1 + lib/alloc_tag.c | 17 +++++++++ mm/mm_init.c | 1 + mm/page_alloc.c | 4 ++ mm/page_ext.c | 4 ++ 7 files changed, 100 insertions(+), 1 deletion(-) create mode 100644 include/linux/pgalloc_tag.h diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index be98564191e6..07e0656898f9 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -4,7 +4,6 @@ #include #include -#include struct pglist_data; diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h new file mode 100644 index 000000000000..a060c26eb449 --- /dev/null +++ b/include/linux/pgalloc_tag.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * page allocation tagging + */ +#ifndef _LINUX_PGALLOC_TAG_H +#define _LINUX_PGALLOC_TAG_H + +#include + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +#include + +extern struct page_ext_operations page_alloc_tagging_ops; +extern struct page_ext *page_ext_get(struct page *page); +extern void page_ext_put(struct page_ext *page_ext); + +static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +{ + return (void *)page_ext + page_alloc_tagging_ops.offset; +} + +static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) +{ + return (void *)ref - page_alloc_tagging_ops.offset; +} + +static inline union codetag_ref *get_page_tag_ref(struct page *page) +{ + if (page && mem_alloc_profiling_enabled()) { + struct page_ext *page_ext = page_ext_get(page); + + if (page_ext) + return codetag_ref_from_page_ext(page_ext); + } + return NULL; +} + +static inline void put_page_tag_ref(union codetag_ref *ref) +{ + page_ext_put(page_ext_from_codetag_ref(ref)); +} + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int order) +{ + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE << order); + put_page_tag_ref(ref); + } +} + +static inline void pgalloc_tag_sub(struct page *page, unsigned int order) +{ + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_sub(ref, PAGE_SIZE << order); + put_page_tag_ref(ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int order) {} +static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + +#endif /* _LINUX_PGALLOC_TAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 78d258ca508f..7bbdb0ddb011 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -978,6 +978,7 @@ config MEM_ALLOC_PROFILING depends on PROC_FS depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING + select PAGE_EXTENSION help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 4fc031f9cefd..2d5226d9262d 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -124,6 +125,22 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, return module_unused; } +static __init bool need_page_alloc_tagging(void) +{ + return true; +} + +static __init void init_page_alloc_tagging(void) +{ +} + +struct page_ext_operations page_alloc_tagging_ops = { + .size = sizeof(union codetag_ref), + .need = need_page_alloc_tagging, + .init = init_page_alloc_tagging, +}; +EXPORT_SYMBOL(page_alloc_tagging_ops); + static struct ctl_table memory_allocation_profiling_sysctls[] = { { .procname = "mem_profiling", diff --git a/mm/mm_init.c b/mm/mm_init.c index 2c19f5515e36..e9ea2919d02d 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 150d4f23b010..edb79a55a252 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -53,6 +53,7 @@ #include #include #include +#include #include #include "internal.h" #include "shuffle.h" @@ -1100,6 +1101,7 @@ static __always_inline bool free_pages_prepare(struct page *page, /* Do not let hwpoison pages hit pcplists/buddy */ reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, order); return false; } @@ -1139,6 +1141,7 @@ static __always_inline bool free_pages_prepare(struct page *page, page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, order); if (!PageHighMem(page)) { debug_check_no_locks_freed(page_address(page), @@ -1532,6 +1535,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); + pgalloc_tag_add(page, current, order); } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, diff --git a/mm/page_ext.c b/mm/page_ext.c index 4548fcc66d74..3c58fe8a24df 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -10,6 +10,7 @@ #include #include #include +#include /* * struct page extension @@ -82,6 +83,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT) &page_idle_ops, #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + &page_alloc_tagging_ops, +#endif #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif From patchwork Mon Feb 12 21:39:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91B74C4829D for ; Mon, 12 Feb 2024 21:40:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79E0F6B0099; Mon, 12 Feb 2024 16:40:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 74E5C6B009A; Mon, 12 Feb 2024 16:40:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 558966B009B; Mon, 12 Feb 2024 16:40:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4028A6B0099 for ; Mon, 12 Feb 2024 16:40:08 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1DDC0C0AA7 for ; Mon, 12 Feb 2024 21:40:08 +0000 (UTC) X-FDA: 81784470096.04.3E0A507 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf10.hostedemail.com (Postfix) with ESMTP id 63194C000E for ; Mon, 12 Feb 2024 21:40:06 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Zz5cJsWG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3NZDKZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3NZDKZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774006; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8hiMFMYdTb9wRLf05RpnPyNZaK7+4nxliVbJC61s+/Q=; b=uDmFdXXCejdcUb8OMhgdNulualzW6bAIxFhbY5OV7iG9XoSgRyxtOgH7US5vxKhY5+Z+tP jzVxj1yq8V6dCkdMJrPNkHiA0mBBvO2KsfaLYBf9nl6ATUuD7r2mkwYR7/WY9bTpfN08uA +szg0pIvUY/7KksHWu6F4Vep4Ivz5e0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Zz5cJsWG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3NZDKZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3NZDKZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774006; a=rsa-sha256; cv=none; b=7Mejl9v9LPFyjBiOUvYrRo5ZJXpGDa5iKTs/M2Am7Z1MfXHQS99qRQuvhyWJ6MOaLHduU3 s9yaOvtVEac8VxTuJTuMt3i/E1nas6U4qv+dnuOGVZ4apf1sqTPIRf4/9yigY6V250aIzW KXvvxm8mZQIwZo2dppClH74B4U4hi+o= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60761bdbd4cso19836117b3.3 for ; Mon, 12 Feb 2024 13:40:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774005; x=1708378805; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8hiMFMYdTb9wRLf05RpnPyNZaK7+4nxliVbJC61s+/Q=; b=Zz5cJsWG1jW+jTECwm1ySIOlphgzYiP6Iudl2aPkYtO3KXGyWgwyrBwCQvGXPEDGZv 77BXR7xgI5jQuz5WEpKmhmg4t3szaiEiPEeTL3nuALXIr9v+KFgl7A0W7z9ssH58/SyJ v2QSWFC3D3ohsbQqzLQa3dQYZmMdDphuondSeXAR9PPhJhlbw657Gvv0IqvUOH+wJC6U EITwhnuzjT5aHk/EYZAIwdyYda/BCYncQUYSIY806eOV93pZV5NXmWz6MMlNChYjkBiL b2UeBZ+VxpME3cFOTTOMqjnoXWKzpdxR0KkMSq5U56earL8rvi1ltdNP1MTrRxPQw2sR lZIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774005; x=1708378805; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8hiMFMYdTb9wRLf05RpnPyNZaK7+4nxliVbJC61s+/Q=; b=CKJC2FvG6kPBgZ5HdvFB+QxMRl4ufH7lXRoxDDZ/avoFJFjzC1G2D/pIdvGaA8A/a1 XOW3d5wS7xw2AZPQdbJ0sJVpmt67TKiV6ykXzlLsJt3zGV4YDv7kYHHFsqqJQFfCcQt5 mKwFLdyz05nBF4FqwvCHmbbQU/WD5NQkUc3/iYQHaGjyBEscPN/hQzgfcjAFpHzkS2+J v5mKkcrl40blHj78ROo9aRHdvk9ONbW1wEanqYBSKyMgF4QzT4N1bHGyZX6PXT2vQFc1 9WYnMg9VubB/JI48/nkD9ivhd+e6s4998FfOPIoXxuXgzQdYsaLxCEWpKYQs2Ft06r+y VEvQ== X-Gm-Message-State: AOJu0YwiY8c+lBPI66bn2+CrbkpBlAFm2q1EpFkGfSYxpEerDWavTV3e w9WyqkF0CbDjQ7SdmZzb0LSyY14XiFqL7D4C/K2DmcJo0i3+cXRrdlO9Lh1SrueO/QvBL7cqB5F paw== X-Google-Smtp-Source: AGHT+IHMymVhDhc3vqzIXqR7wXKts/Huwxqw8epMlcUK6WIr6/J3gYxRwjZRSc5OiXO09TM/bRBKSIcClBs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a0d:d650:0:b0:604:7b03:4223 with SMTP id y77-20020a0dd650000000b006047b034223mr2206277ywd.2.1707774005345; Mon, 12 Feb 2024 13:40:05 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:01 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-16-surenb@google.com> Subject: [PATCH v3 15/35] mm: percpu: increase PERCPU_MODULE_RESERVE to accommodate allocation tags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 63194C000E X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: xttxi1h1e4zr9qhdkcp3amrjdcbr8op3 X-HE-Tag: 1707774006-147234 X-HE-Meta: U2FsdGVkX1/SQnxyAFZiKVXVK7LKbu+XLti+SMgFHASjamMoXuzMoXKK8zoDbM+gw/zELAZAm/U3XxKT9T7KT8UAj2JarHQpcFUHu77dnHxP0Hd3ueTVxlyK+WH2FCDMmbXyXcpPmEp4XzS1LH1+jQfKG4X3TzcTdWiEOY1Kui+HvkJv2vaYG5wjMQZuN+Yz7Bk+yB70/jNZUjCMeZGaswPVKEISb9CKs5IEy2i8IXKZVBH/k0RJwPeEzlS7M6f8WTJrn5gGdsEUD+4opra43Z7iDXl9W5i/5MCLtKMd5QcS5+Ta/WIEjcOC4x3giGRztI91AkVcfYKbwXMk1O/KKtZrosASC1jyFpEn9d5nbD2996HtjPAz99KpX0Nz+sdYtHiu1MMS3BfvtKb2tl2tNXNVrFqTc1b2KrunliVcLy290JgFA/j7A8+AwqBn+ei/eDP4UPdPKwqQT6QVUaEjLazIT2Y/rPNuoDicsqhdxlYKsHBw3BhzvBnq75UzxZpj+kDUGV81fMn3VW7c1qM7P63z9LEF3Z3xofoDxeFBhTPgjMMzvUbJhLER2jYFrrKwa3XxkwO2SNRaZ+ZjM4NiaCQrVwPmweQjDEsxHTgyrL/ZsrrjXjarCgXqHksX98SDDiySKQAX0ThylrOStr03i8chkse6aEW2j9Yjz5m2V4MnFuWpcH0sir4dUGFY2RLck23o3ID/j0ERSUUeqcraWelOWlbTi7CovLIfmLjudBhSecTprN4bL3VoWrYK6IyuTJAzLCufdg0hHSp3PQfW0VuMdhpFCzcj02dlVzE75xCKQHw3GMGpw8oodjJ9rlVZIVwr4pEqO1kcekYACHNPz+Anw3MtOOjwa1YUgeIpMkAni8hX2Bjij45h6vv0xGcsfXILRf7hbek03HfWRNXmgTd50nvimskjflgOAm0t9+T2gbn2CW+u161NOOpXfuuR6nMJKKI0NAnvUStxL2w ZQ19Fk5l YxO7GlTVpJBEFldmOuWxF5qoM1zL+4/p0/YYmqlNsFlH6poTlkF69N416kZt/yqXz2vvKId5fmQXAVpJUw9fX6+3zBTZSFbPcOjjQy20KzkUvsjpCENMhGdMmM7/Omk2lrqW8sgrawQXiQKAG0R56Z3NNzOvyxRZA/fjOp8CDnqaXFtR9nujF398enc5XTmZkr4vvLG/V1yN9W3MUuDhUHMs+rWUMSAJnDZj9oZidlHorcvS0DIFuA0KErg7/38HWrK0pUBWYCPHUGs5NaxTFC+eA8VOfIZ5UYEn6d78hj9h+W/pgRrSi6nxFYoxi56AfuLZb4qT8qvxHZrXrTSp+G3PKYAW1dyJ+5goMt4bQxrr9ukm2Sz5TF1pfHNq97vYeHn1RI3nhBBaD9/nHfxaVhBDZSGRX9tVQzJWCxRn6RPAVSEfxCBBO3TE679khnsfZEtku2yTG7oMPfTYG6NKrVRoMl+DoPFr8jHE6m3LTRbm9TcWZ2gFmKZuW3tFBCcNTysmjnkRVEU+9hIXg8dX68CKyO9q7Du5Gv0PTqmBBcnK6xiyk79kTPDiaw3Q8NHF1LIs14LpI2upGy6qD5PacmokEl1XW3DIWMHuAgl0oNvC80V0min7xqwNR6BeDubsorXnrksEZHEnwO690sckJiNwsMSnN1JQ9ItN+cwF+bndpQITko/7KJHsJeeo+Dg/Uuv3Q6jHi3JYNu4dZaWIHi9XIO92tHnZjrIQ/qpw1SCjDDDdXWNOPJb6O+ZnMRj0OWYbVZAKG1lY6uBCbsTVYY1IryKsSD85xlN2eEB4GTWhI02VOy11kJesPS8A6Vm1rPmN1yNMnJUGILNOquCxQuS7nKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As each allocation tag generates a per-cpu variable, more space is required to store them. Increase PERCPU_MODULE_RESERVE to provide enough area. A better long-term solution would be to allocate this memory dynamically. Signed-off-by: Suren Baghdasaryan Signed-off-by: Kent Overstreet Cc: Peter Zijlstra Cc: Tejun Heo --- include/linux/percpu.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 8c677f185901..62b5eb45bd89 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -14,7 +14,11 @@ /* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES +#ifdef CONFIG_MEM_ALLOC_PROFILING +#define PERCPU_MODULE_RESERVE (8 << 12) +#else #define PERCPU_MODULE_RESERVE (8 << 10) +#endif #else #define PERCPU_MODULE_RESERVE 0 #endif From patchwork Mon Feb 12 21:39:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 991C5C4829D for ; Mon, 12 Feb 2024 21:40:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C31B86B009B; Mon, 12 Feb 2024 16:40:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BE18C6B009C; Mon, 12 Feb 2024 16:40:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A33866B009D; Mon, 12 Feb 2024 16:40:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 89CB76B009B for ; Mon, 12 Feb 2024 16:40:10 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6BE96A1E71 for ; Mon, 12 Feb 2024 21:40:10 +0000 (UTC) X-FDA: 81784470180.18.73CE3E8 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 93EB740011 for ; Mon, 12 Feb 2024 21:40:08 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JzPe5imB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3N5DKZQYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3N5DKZQYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774008; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gyt+GmM2eMHRRKEe8/8uFf8EZSlg2sZI9NA/PbSxBWI=; b=Asu/NvfnqLIOEUTmgrvr0p5oJ6U939fzHCKM+d3c4yvhVHrDZT0AIXq1lf1u4Txw8xB5rC RpodIBhCvozFgSZ72R7Svbki1uq+RVbqaiMZ8UZjlmmqnfIXATj2gt5SVrHmNNgVm0qGxw osq/UQIj9UIvpNVcOYu5KjE0Bpk3nFY= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JzPe5imB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3N5DKZQYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3N5DKZQYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774008; a=rsa-sha256; cv=none; b=fbrTNflcIEIg338d0gJ259Ekpp7YK5d+DnRVZQ/yFARYYnqRIDCifEDlKdikgcIMFdrazl ST+2SQlzwgkKBfyBJt8+CVE1OxvztyE6WUhKYUiQLAm4KQOtsse6ADFeEBz7AZXqeuCNs/ lpcT2yARsNKUI7ng0xHzmEdaHzUes2s= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-604ab15463aso4576627b3.3 for ; Mon, 12 Feb 2024 13:40:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774007; x=1708378807; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gyt+GmM2eMHRRKEe8/8uFf8EZSlg2sZI9NA/PbSxBWI=; b=JzPe5imBhkxrwDH8dDwFzaBFUhriOWz8cqS0HPRoL1wQOfhqYgFK7DBSjEonX3hdbS aivadebV8HIcAfxIRIxg0k2JhOEJ6xXbKJDajDZozSVvske9j+ZLNFdMCcPVHbq7H5qT Q2tEzYn/p6WG5he3ElneLQhx7OKagwPXh9Y4WEr2dRe0CnGWwRJsZQxLp2/2Z53S1hQL VmEBdiOFRhKHcdfjsekmwRNvz6nDMOaTtUdLy8jhHXmFV+F4dpctktsw0aIrktaiAUmo hkhqGaUHCkL5mtfJ40nSKUu3q2zc8z8bNeWpQAV1gbcHVn7Xb1eIRmWEWRPbdjEtaw62 n+4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774007; x=1708378807; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gyt+GmM2eMHRRKEe8/8uFf8EZSlg2sZI9NA/PbSxBWI=; b=OT3RDIETDcNx4SblKXh2NlRqvFW2aFHSPKTE6FMzUd6JS//8N1IARruLFlCJ+D/uHc evtUFQOGG4M42IHO1kgVbbSA5NmgXPu+NgK/FtqmOiRa2HMZdOoO5bdf+dtko63FP7ya XSLR0/U+h+MBrw9luDXh8kBWsnq2clxKxAu7JzuMneqdnbFnTeOkVRjhgNjVoae56OxE IAxSRFIN0TMHn+40j0o09yUZt4t9scj8oVU/UriRZaaAgmFAv4oPQltFZaxWnJPkRsAE Nu82inZ6RgUYkX+UOANAqzXV0jCOBxfGjFc6qPh/DL0FcewoWoHW1UHPaY51n3AzKm4i yEfw== X-Forwarded-Encrypted: i=1; AJvYcCWxlI9r/yXVhwqOSK471eW9jBc4Pj8Oxz6wQnlViilToBAOan1UaJJniKjvfS/rlGMY5bU6okZvCFwLdyxrR+b0v18= X-Gm-Message-State: AOJu0YwTyL7pOcLheseB1ToRjMHpeKORd0vORGhdL0a0wFauQq/1XR5h NZ7V1ePEIdpzS4wOtdDHjMDXDD9FCEjCISY/8njOiigUk3djUaIALaQ4Okq7JvzqcPMqUAsfstB v/g== X-Google-Smtp-Source: AGHT+IFq2nZiWlkvaCq5BbxuMd0rU4hXzsPUeXzg8XtldG/VPRi/VWxJUnymRdhbS2wacAZbpAHHkokQ9sA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a81:848a:0:b0:604:49a1:8da0 with SMTP id u132-20020a81848a000000b0060449a18da0mr1393758ywf.8.1707774007616; Mon, 12 Feb 2024 13:40:07 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:02 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-17-surenb@google.com> Subject: [PATCH v3 16/35] change alloc_pages name in dma_map_ops to avoid name conflicts From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: 9f6yij1tuskqfo4sqs3ee8o5e8stpb97 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 93EB740011 X-HE-Tag: 1707774008-322360 X-HE-Meta: U2FsdGVkX1+dcHaRwbfGaTAjbYHJWgm+DhU68StWBBDm4G4DC3EnTyY8lpNz+3JOH+CXw/2GbefzV5zRvzrzcBI0c4NFHeHP9xU7umELzSAiHNoIhW49oIFKvRL73Jq2kHi86Gdg+lov5xR2Nvi+yBrZRMo6WPUV8ogPgwMuLXTOYOjaabJ3lE8p9VYB89ipH9u4hIWnBPNOCF16Fb97konYaeD5OlQvuvGBWH75ZXfjRFJS7Owi/JiqSwx7mfGYcdi9TUQBM4nUrGG64iyYubYZDcMjRNLesCUnkJacZyiYWcZI3AqEMZO2AefZ3VTsiqs9pPYO9DQoLf75pfW0k4Ude3GUnzqvWgLUORiqDJXBRkYeg6N8E2rlE4lbZ+ZFL9ypblC6SdD6rsW1xJ9bkmB9uKm1TWAb6KRXsYWW9zfnULz8EKDQeGT2dcBm8izv679ujeWDRd1LI+mwtfE8NRy/rHr/f7Kw40bGqPxdMkGdQr2Xsj+54g4XQGLulZQS7dHNqkYPqgjnbiSmtu8e7XPLskMJxEvCQm7a6jZrdTf7HIp3tyfNYoQeIF/2Hi9oJ2+I77WxUl+GBOBAFI6tw0t22NnodfuTbwXXqpISgqEbkIp/HOVO5zQGuzFdGbCN0JH35RaaRKBDV5/8Av7Y+s/3sd8bNGj7X5vek0lJHKNsk04oUhMOlVDooG/T57QSfIa9azVTP03Kcg+0l2hmI4WzucRlvaOL8xaFpzk6TjIfn49iW2iaz97yeO3ykVN46L6JV0CSH1P2SkvaFzXnZXopiLZu2y3vXUHFT6M/zBR2Oi1nka/Mot0Bx5S1kTN4LzPah3l00rEPOfENKvX2yFpH5mFNUhh2QQ3b8gd8eovA9RLo9Txjr5msEtBPfWO4cmO4H8ySCKPAGafv7C0ZHVVpgx4MA3S/tkZwNtzxKusU6Ena7zIXk+ybUCtSkEEGXnuvSHFsjMprR9nFvyK J2CEjAj5 dusH5oW2gTVYCVKdftzBVZTl/krs2GUHBIz4cnLxqZAl8QPYWiSxa5u93rQsyF5DR3YiFST5aAoO7TBwq7UHuMOmrJjCxINVfP1r+9wCrBq806c71G6UuzbRLMt0i7jj37vBGzO4WaRCMm2F037eWHlJs13HK5VY5UTr3+rw5KTEYLd/u3PL/VWerYJjKX25dPhCe0aNi8EUOG2Ty/04XZwAgy2Dvkj7Aefhv1Oy91vBzQwfHyJzY5eBnzorhnf6+lNjxLTlEhCjcWkww/qKM/Ty2PTmJPnulg5z/YdbsoNsUA5bceileHlXqPu5okBtc//Z5hmpKThUWvhj0795vooLTK0K6A+l8J/jfFlZnoUL5G4Pzqn1EIOAKTYb6z4BlSK9JPngAiKFPhYv8dRd7XPWR938l5pBxszjcoLAn/HB1DsGue+R5QpGFULz+D1+OwNt0h0cz8R9SZsY89dmAMN/JA9/ta1+2Y3ze0Cj67eTqHpFFV9vRu14+/wiIhG+AHD5Y/ITHykJPODixdqUCQ2dGAWh/IoXkffZPlkVQgaXZvDacI4VeI+fZStlKfJ7mPSjEk9wlX+JRLtDMgwNRabXTYoyqj8EPiCMT6dSOPDR2J3RQOlU6qoPNu9ICyUWbgIHgXoWS0T8Q70CJtCBUjo/8xZNLX58ckSxUBBKK5AurVbiliqkRJKmE+OrH5y1P/PA7OMBxt6/bCWsh+B6hygrj3gH9GSzgP6SS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After redefining alloc_pages, all uses of that name are being replaced. Change the conflicting names to prevent preprocessor from replacing them when it's not intended. Signed-off-by: Suren Baghdasaryan --- arch/alpha/kernel/pci_iommu.c | 2 +- arch/mips/jazz/jazzdma.c | 2 +- arch/powerpc/kernel/dma-iommu.c | 2 +- arch/powerpc/platforms/ps3/system-bus.c | 4 ++-- arch/powerpc/platforms/pseries/vio.c | 2 +- arch/x86/kernel/amd_gart_64.c | 2 +- drivers/iommu/dma-iommu.c | 2 +- drivers/parisc/ccio-dma.c | 2 +- drivers/parisc/sba_iommu.c | 2 +- drivers/xen/grant-dma-ops.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/mapping.c | 4 ++-- 13 files changed, 15 insertions(+), 15 deletions(-) diff --git a/arch/alpha/kernel/pci_iommu.c b/arch/alpha/kernel/pci_iommu.c index c81183935e97..7fcf3e9b7103 100644 --- a/arch/alpha/kernel/pci_iommu.c +++ b/arch/alpha/kernel/pci_iommu.c @@ -929,7 +929,7 @@ const struct dma_map_ops alpha_pci_ops = { .dma_supported = alpha_pci_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; EXPORT_SYMBOL(alpha_pci_ops); diff --git a/arch/mips/jazz/jazzdma.c b/arch/mips/jazz/jazzdma.c index eabddb89d221..c97b089b9902 100644 --- a/arch/mips/jazz/jazzdma.c +++ b/arch/mips/jazz/jazzdma.c @@ -617,7 +617,7 @@ const struct dma_map_ops jazz_dma_ops = { .sync_sg_for_device = jazz_dma_sync_sg_for_device, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; EXPORT_SYMBOL(jazz_dma_ops); diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c index 8920862ffd79..f0ae39e77e37 100644 --- a/arch/powerpc/kernel/dma-iommu.c +++ b/arch/powerpc/kernel/dma-iommu.c @@ -216,6 +216,6 @@ const struct dma_map_ops dma_iommu_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/powerpc/platforms/ps3/system-bus.c b/arch/powerpc/platforms/ps3/system-bus.c index d6b5f5ecd515..56dc6b29a3e7 100644 --- a/arch/powerpc/platforms/ps3/system-bus.c +++ b/arch/powerpc/platforms/ps3/system-bus.c @@ -695,7 +695,7 @@ static const struct dma_map_ops ps3_sb_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; @@ -709,7 +709,7 @@ static const struct dma_map_ops ps3_ioc0_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/powerpc/platforms/pseries/vio.c b/arch/powerpc/platforms/pseries/vio.c index 2dc9cbc4bcd8..0c90fc4c3796 100644 --- a/arch/powerpc/platforms/pseries/vio.c +++ b/arch/powerpc/platforms/pseries/vio.c @@ -611,7 +611,7 @@ static const struct dma_map_ops vio_dma_mapping_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index 2ae98f754e59..c884deca839b 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -676,7 +676,7 @@ static const struct dma_map_ops gart_dma_ops = { .get_sgtable = dma_common_get_sgtable, .dma_supported = dma_direct_supported, .get_required_mask = dma_direct_get_required_mask, - .alloc_pages = dma_direct_alloc_pages, + .alloc_pages_op = dma_direct_alloc_pages, .free_pages = dma_direct_free_pages, }; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 50ccc4f1ef81..8a1f7f5d1bca 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1710,7 +1710,7 @@ static const struct dma_map_ops iommu_dma_ops = { .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .alloc_noncontiguous = iommu_dma_alloc_noncontiguous, .free_noncontiguous = iommu_dma_free_noncontiguous, diff --git a/drivers/parisc/ccio-dma.c b/drivers/parisc/ccio-dma.c index 9ce0d20a6c58..feef537257d0 100644 --- a/drivers/parisc/ccio-dma.c +++ b/drivers/parisc/ccio-dma.c @@ -1022,7 +1022,7 @@ static const struct dma_map_ops ccio_ops = { .map_sg = ccio_map_sg, .unmap_sg = ccio_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/drivers/parisc/sba_iommu.c b/drivers/parisc/sba_iommu.c index 784037837f65..fc3863c09f83 100644 --- a/drivers/parisc/sba_iommu.c +++ b/drivers/parisc/sba_iommu.c @@ -1090,7 +1090,7 @@ static const struct dma_map_ops sba_ops = { .map_sg = sba_map_sg, .unmap_sg = sba_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/drivers/xen/grant-dma-ops.c b/drivers/xen/grant-dma-ops.c index 76f6f26265a3..29257d2639db 100644 --- a/drivers/xen/grant-dma-ops.c +++ b/drivers/xen/grant-dma-ops.c @@ -282,7 +282,7 @@ static int xen_grant_dma_supported(struct device *dev, u64 mask) static const struct dma_map_ops xen_grant_dma_ops = { .alloc = xen_grant_dma_alloc, .free = xen_grant_dma_free, - .alloc_pages = xen_grant_dma_alloc_pages, + .alloc_pages_op = xen_grant_dma_alloc_pages, .free_pages = xen_grant_dma_free_pages, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 0e6c6c25d154..1c4ef5111651 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -403,7 +403,7 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .dma_supported = xen_swiotlb_dma_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .max_mapping_size = swiotlb_max_mapping_size, }; diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 4abc60f04209..9ee319851b5f 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -29,7 +29,7 @@ struct dma_map_ops { unsigned long attrs); void (*free)(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs); - struct page *(*alloc_pages)(struct device *dev, size_t size, + struct page *(*alloc_pages_op)(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp); void (*free_pages)(struct device *dev, size_t size, struct page *vaddr, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 58db8fd70471..5e2d51e1cdf6 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -570,9 +570,9 @@ static struct page *__dma_alloc_pages(struct device *dev, size_t size, size = PAGE_ALIGN(size); if (dma_alloc_direct(dev, ops)) return dma_direct_alloc_pages(dev, size, dma_handle, dir, gfp); - if (!ops->alloc_pages) + if (!ops->alloc_pages_op) return NULL; - return ops->alloc_pages(dev, size, dma_handle, dir, gfp); + return ops->alloc_pages_op(dev, size, dma_handle, dir, gfp); } struct page *dma_alloc_pages(struct device *dev, size_t size, From patchwork Mon Feb 12 21:39:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD01FC4829D for ; Mon, 12 Feb 2024 21:40:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D43506B009C; Mon, 12 Feb 2024 16:40:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CA5FB6B009D; Mon, 12 Feb 2024 16:40:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD0016B009E; Mon, 12 Feb 2024 16:40:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9A3706B009C for ; Mon, 12 Feb 2024 16:40:12 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7278F160B39 for ; Mon, 12 Feb 2024 21:40:12 +0000 (UTC) X-FDA: 81784470264.26.05D3323 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf11.hostedemail.com (Postfix) with ESMTP id 8FCBE40015 for ; Mon, 12 Feb 2024 21:40:10 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=syWDkLhH; spf=pass (imf11.hostedemail.com: domain of 3OZDKZQYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3OZDKZQYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GrGwB9qfJZ8baAR6T6ipxUZf1yCQtYwXa3nH2rh59j0=; b=MUefDDGmtWsWAe5uvElLNRxbq7HnfguQTl5OU33UPKmz/8za6KXkLwDTehVgIKVsBiEp6R d4D0Bz+QzNKO/cebcMiV/YtP/Y6oh/HDCvOK6JLoDKAQLqkKH+f7tP3X/yRlNr8Bp5u7sZ c57a51+KCRMtNiWqs5uW6LSnebtxmco= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=syWDkLhH; spf=pass (imf11.hostedemail.com: domain of 3OZDKZQYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3OZDKZQYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774010; a=rsa-sha256; cv=none; b=viC8ZqtQerP/ExNoJW6UUjhHyk8Hkj/C8TRxi3VedMUdUD44ne972i6ihWqH3K3wL3lihF Mt1Yxd968wVTumdBY1eLh5s2HMLdLQPK9lnXAj+9EQAn6uC015iRYExlVbr3QA/CD996Ad z/v4GNhA+Wvn6mYrFi0Zu20f2QBDn84= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60761bdbd4cso19837087b3.3 for ; Mon, 12 Feb 2024 13:40:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774009; x=1708378809; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GrGwB9qfJZ8baAR6T6ipxUZf1yCQtYwXa3nH2rh59j0=; b=syWDkLhHgDJ8sf//CvjAaAQi4Xbw4lwkaICpv0Hkv33wXW6cgpEaB//raLSx+Tydz2 wFeOP2yZfnsMTLo0cGI+0JgVenhYhyYnt3iBVG4h/v+fQDHUNe7BiVvahacCYbhAzFKl n8Vh62EVjDp6GR6uLtEEuCvkChlsJdaIFGc69+oSwayjXLro7Xwc4VAjbQmJ2GTunF54 38DzZlKd33wRXQ7RCXlbB3Wjs+vOhLB4DwFG+CqUXhy/z//BDWIGCiiYKj+NH4B3cyU2 AB9PZLh4uRvM5f/mj8Anw93QcaQWBZrWAjStvXatZqctEIt2f+WbjYrlSy+aOUrkYZPD M6EA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774009; x=1708378809; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GrGwB9qfJZ8baAR6T6ipxUZf1yCQtYwXa3nH2rh59j0=; b=gsqoGK0TzfPJzYnCjS5gmSZmCU1is3J9u/oo+LALYWlyw+CgEI6g2bo7jRTKZBweUH qzkUR1ewjYhUyzdUvb00uI7phY+jjGy2g1xF2KfWgGdeLvaU/Wkff6Ust1bmBaLarwTe mYa7Jo4ifkmJ2rDI2RMDsBX8Q9A4OfjzlmYVdsAFa6G6zGpeYdbJ5GSIo7D5Nlo7NXnJ aLW/Hpjy6zmPfcgNgWihcCuJKWUYOPwKdC/gffAp1ThhFq4/CucqRBgJW7NmqKhY8DVp FN4skGhwi4lGikn+BaV4DjXZ+rQ49C/pIWLm2Fs8OPURqFlUoukVeodCfMid3/LQ41bw damw== X-Forwarded-Encrypted: i=1; AJvYcCVh8O4bwsjHM3dlSpee1s1UM8J+tuWYfduXiO9Is0kvVpdhR8Z8kE6ciIDxsnTY8Ll4GR+WTEWoX+Ot2ELj1rlsaao= X-Gm-Message-State: AOJu0Yxe+k6Haf5ZGcwIs3cjuiMFgPw7GgzWq9JidOYRxuZkbo8j4YvT jyrPu7OzEXAwJsniRd5eGpMxka4NlaxG1M1Jar/po/7bbLYjn7IRVUJG+C+q16n9CoLBJ/AdYy8 pCg== X-Google-Smtp-Source: AGHT+IHSOOtqV+tB0AgXVysbclo7veEI+l/B8vUwDQqYprkJNFv3jRaKZjBjeFfz/9+sevD0Aijuq2/T8Xg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:690c:3505:b0:604:9b11:ff2 with SMTP id fq5-20020a05690c350500b006049b110ff2mr2208602ywb.6.1707774009704; Mon, 12 Feb 2024 13:40:09 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:03 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-18-surenb@google.com> Subject: [PATCH v3 17/35] mm: enable page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 8FCBE40015 X-Rspam-User: X-Stat-Signature: 79y358ix9grwghqa9sdmud169kp8371d X-Rspamd-Server: rspam01 X-HE-Tag: 1707774010-665981 X-HE-Meta: U2FsdGVkX19mpwLhcYHrz245WlbXR0Bh3eZsTxG7Q3VBDtrl6laZfyglWd/WKc79ie13Ibner58bVcgZ5TNVdKUCKQe5G/0zJcQ/poKHIawjA35VbPIVXhAK6JrwZfeLy1+7e9wzgwrbVDdVKvgm9U5n4LcE4ZXPPTkfrzJgFdcKkqa8oay4n0aISMcgwRpt32vnTiXi7M2nZOuI5w/ejXgTr2wt7ESFsK/2eC5I5TdyJPkrZsJZ+17zXUKgDlKtefBYpS9l2X0IUDFGkNhwz/QOIRTw/3qDkLalxf70WycobWL+/8yQ1xJiQrKVAH4LqU2amUxG7lsSDuvyjqG2964YjrBUvGwPOAntQGZYO90Z3nmj32wYBCzOBtLTGIwfOolIP2EQCnlgQ0IeMv/QTdUccEaLGQAPpIQYfrUHr+lcU46Jyc3FlyB/QE2QI3n3syjgmCzwRpbZ1g0YXeAteEMqaJistDHyKhiul9xol1fV16OnC/J65rb60M/YQpdssOkARn31hIPW/VvXA5XKJ46eK7EYtoI0HGyO9VtAtyg0KIyYmTCF2RgkXWMmhmH2hhAIYDlnLqe4XOKl+cd83wWcZuqvLXodPGzFmfPsflIoJrEsDd+MXl9HkaqFRB+ZtOKbGDVn7scln/EUmV8IwpQvULuTcvAzv3mTRCjEYFpqYMIX8x/riHMdMzFpjyH2kH5FyXJZhbNDkG+183NLahhnBkpMB8aNIBfmEUuIztwZRNkd+Z4YwKZYdHHvnnBgoshDkf5xPYiJG2sHfKYjTnx/rpI0+LDCKwwM6fH7oMUHq4wX3V/FBwnCTEDqv6Aa+80dRhB/xqGD9uqJyrK4KVduuWLxR1DNk24tlFl+MWiDNWIpDN7Mfgz6qNESISLzEA8yqm3il2n4XnFp+rY2E1KeCRm5RLDWMrVbf+rYYMMIt5VGSnVXZaz7uSpZHfsiY1Xp8wtz0XQEIVlXBci WQ2t1ZuO j63Uw8EQHBPMeBCTzzQMXSUI3aOZi41pe+CR8RIxrYAnvLY2M5CkdcYfbNaYRttt50cGZoFk6uiDsfsfIIXFEfdTTfMnEQFt8RyyHCNWnmZXjEGPGbicOZdWj2WtgSyODih6MC3871mi+15S1Jz3TE2nRo483N8OIHcbWWqq2NtzDa520eRFN7pxFqCqQk/vfEjJlh492qxvYhWhwr9IBQKmTdFus6PEEmOxxn2RjQd6mfIhBjPWDrno8GlwDhxaZN8D5hL7FWoXAsMtVb2t4+h3cYsSJcAV2QXRrUtc9BXjYPwFY/IbqL6xx4veFY2/5H9J76exFqaRhD1W7kJQmZSD/b5vqwKCPLghHS1jonLjUUAySE8AsnaJyNFnsZWz08v1dwKmZjgovPYkyP4WdFKlt5YzBPW5RLFrwYs0WloxlVYi4dM7yKTss8wFzPGncriJkxWF+/DiM+HCbI/IgIqy2iYhkXg9/yRow12RcNp7k42J8sub+vOlp4T3CLox+DtEI/ueNJAgSzAIe0lwOeQBWS6XqxX9h5ehjU11nwHezRXOlbqhoLbG1D6EHurKg70dil5yqwgxah8IW4dAJ+clHf3llRaF5YvsnhiBAEfSKetDJu6NFYzaj3yMNC9y3rTJ1+gZhzevlxDsELL3nibyaZPSrnule3tdH5+A8M3G8mIw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine page allocators to record allocation tags upon their invocation. Instrument post_alloc_hook and free_pages_prepare to modify current allocation tag. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- include/linux/alloc_tag.h | 10 +++ include/linux/gfp.h | 126 ++++++++++++++++++++++++-------------- include/linux/pagemap.h | 9 ++- mm/compaction.c | 7 ++- mm/filemap.c | 6 +- mm/mempolicy.c | 52 ++++++++-------- mm/page_alloc.c | 60 +++++++++--------- 7 files changed, 160 insertions(+), 110 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index cf55a149fa84..6fa8a94d8bc1 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -130,4 +130,14 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, #endif +#define alloc_hooks(_do_alloc) \ +({ \ + typeof(_do_alloc) _res; \ + DEFINE_ALLOC_TAG(_alloc_tag, _old); \ + \ + _res = _do_alloc; \ + alloc_tag_restore(&_alloc_tag, _old); \ + _res; \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/gfp.h b/include/linux/gfp.h index de292a007138..bc0fd5259b0b 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -6,6 +6,8 @@ #include #include +#include +#include struct vm_area_struct; struct mempolicy; @@ -175,42 +177,46 @@ static inline void arch_free_page(struct page *page, int order) { } static inline void arch_alloc_page(struct page *page, int order) { } #endif -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +#define __alloc_pages(...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) + +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); +#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__)) -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array); +#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__)) -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array); +#define alloc_pages_bulk_array_mempolicy(...) \ + alloc_hooks(alloc_pages_bulk_array_mempolicy_noprof(__VA_ARGS__)) /* Bulk allocate order-0 pages */ -static inline unsigned long -alloc_pages_bulk_list(gfp_t gfp, unsigned long nr_pages, struct list_head *list) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, list, NULL); -} +#define alloc_pages_bulk_list(_gfp, _nr_pages, _list) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, _list, NULL) -static inline unsigned long -alloc_pages_bulk_array(gfp_t gfp, unsigned long nr_pages, struct page **page_array) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, NULL, page_array); -} +#define alloc_pages_bulk_array(_gfp, _nr_pages, _page_array) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, NULL, _page_array) static inline unsigned long -alloc_pages_bulk_array_node(gfp_t gfp, int nid, unsigned long nr_pages, struct page **page_array) +alloc_pages_bulk_array_node_noprof(gfp_t gfp, int nid, unsigned long nr_pages, + struct page **page_array) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_bulk(gfp, nid, NULL, nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, NULL, nr_pages, NULL, page_array); } +#define alloc_pages_bulk_array_node(...) \ + alloc_hooks(alloc_pages_bulk_array_node_noprof(__VA_ARGS__)) + static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) { gfp_t warn_gfp = gfp_mask & (__GFP_THISNODE|__GFP_NOWARN); @@ -230,82 +236,104 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) * online. For more general interface, see alloc_pages_node(). */ static inline struct page * -__alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order) +__alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp_mask); - return __alloc_pages(gfp_mask, order, nid, NULL); + return __alloc_pages_noprof(gfp_mask, order, nid, NULL); } +#define __alloc_pages_node(...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) + static inline -struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) +struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp); - return __folio_alloc(gfp, order, nid, NULL); + return __folio_alloc_noprof(gfp, order, nid, NULL); } +#define __folio_alloc_node(...) alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__)) + /* * Allocate pages, preferring the node given as nid. When nid == NUMA_NO_NODE, * prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, - unsigned int order) +static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, + unsigned int order) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_node(nid, gfp_mask, order); + return __alloc_pages_node_noprof(nid, gfp_mask, order); } +#define alloc_pages_node(...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__)) + #ifdef CONFIG_NUMA -struct page *alloc_pages(gfp_t gfp, unsigned int order); -struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order); +struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid); -struct folio *folio_alloc(gfp_t gfp, unsigned int order); -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order); +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage); #else -static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) +static inline struct page *alloc_pages_noprof(gfp_t gfp_mask, unsigned int order) { - return alloc_pages_node(numa_node_id(), gfp_mask, order); + return alloc_pages_node_noprof(numa_node_id(), gfp_mask, order); } -static inline struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +static inline struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid) { - return alloc_pages(gfp, order); + return alloc_pages_noprof(gfp, order); } -static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order) +static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { return __folio_alloc_node(gfp, order, numa_node_id()); } -#define vma_alloc_folio(gfp, order, vma, addr, hugepage) \ - folio_alloc(gfp, order) +#define vma_alloc_folio_noprof(gfp, order, vma, addr, hugepage) \ + folio_alloc_noprof(gfp, order) #endif + +#define alloc_pages(...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__)) +#define alloc_pages_mpol(...) alloc_hooks(alloc_pages_mpol_noprof(__VA_ARGS__)) +#define folio_alloc(...) alloc_hooks(folio_alloc_noprof(__VA_ARGS__)) +#define vma_alloc_folio(...) alloc_hooks(vma_alloc_folio_noprof(__VA_ARGS__)) + #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0) -static inline struct page *alloc_page_vma(gfp_t gfp, + +static inline struct page *alloc_page_vma_noprof(gfp_t gfp, struct vm_area_struct *vma, unsigned long addr) { - struct folio *folio = vma_alloc_folio(gfp, 0, vma, addr, false); + struct folio *folio = vma_alloc_folio_noprof(gfp, 0, vma, addr, false); return &folio->page; } +#define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) + +extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); +#define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) -extern unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order); -extern unsigned long get_zeroed_page(gfp_t gfp_mask); +extern unsigned long get_zeroed_page_noprof(gfp_t gfp_mask); +#define get_zeroed_page(...) alloc_hooks(get_zeroed_page_noprof(__VA_ARGS__)) + +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) __alloc_size(1); +#define alloc_pages_exact(...) alloc_hooks(alloc_pages_exact_noprof(__VA_ARGS__)) -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1); void free_pages_exact(void *virt, size_t size); -__meminit void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); -#define __get_free_page(gfp_mask) \ - __get_free_pages((gfp_mask), 0) +__meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); +#define alloc_pages_exact_nid(...) \ + alloc_hooks(alloc_pages_exact_nid_noprof(__VA_ARGS__)) + +#define __get_free_page(gfp_mask) \ + __get_free_pages((gfp_mask), 0) -#define __get_dma_pages(gfp_mask, order) \ - __get_free_pages((gfp_mask) | GFP_DMA, (order)) +#define __get_dma_pages(gfp_mask, order) \ + __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); @@ -357,10 +385,14 @@ extern gfp_t vma_thp_gfp_mask(struct vm_area_struct *vma); #ifdef CONFIG_CONTIG_ALLOC /* The below functions must be run on a range from a single zone. */ -extern int alloc_contig_range(unsigned long start, unsigned long end, +extern int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask); -extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask); +#define alloc_contig_range(...) alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__)) + +extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask); +#define alloc_contig_pages(...) alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__)) + #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2df35e65557d..35636e67e2e1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -542,14 +542,17 @@ static inline void *detach_page_private(struct page *page) #endif #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order); +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order); #else -static inline struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +static inline struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } #endif +#define filemap_alloc_folio(...) \ + alloc_hooks(filemap_alloc_folio_noprof(__VA_ARGS__)) + static inline struct page *__page_cache_alloc(gfp_t gfp) { return &filemap_alloc_folio(gfp, 0)->page; diff --git a/mm/compaction.c b/mm/compaction.c index 4add68d40e8d..f4c0e682c979 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1781,7 +1781,7 @@ static void isolate_freepages(struct compact_control *cc) * This is a migrate-callback that "allocates" freepages by taking pages * from the isolated freelists in the block we are migrating to. */ -static struct folio *compaction_alloc(struct folio *src, unsigned long data) +static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long data) { struct compact_control *cc = (struct compact_control *)data; struct folio *dst; @@ -1800,6 +1800,11 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) return dst; } +static struct folio *compaction_alloc(struct folio *src, unsigned long data) +{ + return alloc_hooks(compaction_alloc_noprof(src, data)); +} + /* * This is a migrate-callback that "frees" freepages back to the isolated * freelist. All pages on the freelist are from the same zone, so there is no diff --git a/mm/filemap.c b/mm/filemap.c index 750e779c23db..e51e474545ad 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -957,7 +957,7 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, EXPORT_SYMBOL_GPL(filemap_add_folio); #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { int n; struct folio *folio; @@ -972,9 +972,9 @@ struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) return folio; } - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } -EXPORT_SYMBOL(filemap_alloc_folio); +EXPORT_SYMBOL(filemap_alloc_folio_noprof); #endif /* diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 10a590ee1c89..c329d00b975f 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2070,15 +2070,15 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages(preferred_gfp, order, nid, nodemask); + page = __alloc_pages_noprof(preferred_gfp, order, nid, nodemask); if (!page) - page = __alloc_pages(gfp, order, nid, NULL); + page = __alloc_pages_noprof(gfp, order, nid, NULL); return page; } /** - * alloc_pages_mpol - Allocate pages according to NUMA mempolicy. + * alloc_pages_mpol_noprof - Allocate pages according to NUMA mempolicy. * @gfp: GFP flags. * @order: Order of the page allocation. * @pol: Pointer to the NUMA mempolicy. @@ -2087,7 +2087,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, * * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid) { nodemask_t *nodemask; @@ -2117,7 +2117,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * First, try to allocate THP only on local node, but * don't reclaim unnecessarily, just compact. */ - page = __alloc_pages_node(nid, + page = __alloc_pages_node_noprof(nid, gfp | __GFP_THISNODE | __GFP_NORETRY, order); if (page || !(gfp & __GFP_DIRECT_RECLAIM)) return page; @@ -2130,7 +2130,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page = __alloc_pages(gfp, order, nid, nodemask); + page = __alloc_pages_noprof(gfp, order, nid, nodemask); if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) { /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */ @@ -2146,7 +2146,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } /** - * vma_alloc_folio - Allocate a folio for a VMA. + * vma_alloc_folio_noprof - Allocate a folio for a VMA. * @gfp: GFP flags. * @order: Order of the folio. * @vma: Pointer to VMA. @@ -2161,7 +2161,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * * Return: The folio on success or NULL if allocation fails. */ -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage) { struct mempolicy *pol; @@ -2169,15 +2169,15 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, struct page *page; pol = get_vma_policy(vma, addr, order, &ilx); - page = alloc_pages_mpol(gfp | __GFP_COMP, order, - pol, ilx, numa_node_id()); + page = alloc_pages_mpol_noprof(gfp | __GFP_COMP, order, + pol, ilx, numa_node_id()); mpol_cond_put(pol); return page_rmappable_folio(page); } -EXPORT_SYMBOL(vma_alloc_folio); +EXPORT_SYMBOL(vma_alloc_folio_noprof); /** - * alloc_pages - Allocate pages. + * alloc_pages_noprof - Allocate pages. * @gfp: GFP flags. * @order: Power of two of number of pages to allocate. * @@ -2190,7 +2190,7 @@ EXPORT_SYMBOL(vma_alloc_folio); * flags are used. * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages(gfp_t gfp, unsigned int order) +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order) { struct mempolicy *pol = &default_policy; @@ -2201,16 +2201,16 @@ struct page *alloc_pages(gfp_t gfp, unsigned int order) if (!in_interrupt() && !(gfp & __GFP_THISNODE)) pol = get_task_policy(current); - return alloc_pages_mpol(gfp, order, - pol, NO_INTERLEAVE_INDEX, numa_node_id()); + return alloc_pages_mpol_noprof(gfp, order, pol, NO_INTERLEAVE_INDEX, + numa_node_id()); } -EXPORT_SYMBOL(alloc_pages); +EXPORT_SYMBOL(alloc_pages_noprof); -struct folio *folio_alloc(gfp_t gfp, unsigned int order) +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { - return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order)); + return page_rmappable_folio(alloc_pages_noprof(gfp | __GFP_COMP, order)); } -EXPORT_SYMBOL(folio_alloc); +EXPORT_SYMBOL(folio_alloc_noprof); static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, struct mempolicy *pol, unsigned long nr_pages, @@ -2229,13 +2229,13 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, for (i = 0; i < nodes; i++) { if (delta) { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node + 1, NULL, page_array); delta--; } else { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node, NULL, page_array); } @@ -2257,11 +2257,11 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - nr_allocated = __alloc_pages_bulk(preferred_gfp, nid, &pol->nodes, + nr_allocated = alloc_pages_bulk_noprof(preferred_gfp, nid, &pol->nodes, nr_pages, NULL, page_array); if (nr_allocated < nr_pages) - nr_allocated += __alloc_pages_bulk(gfp, numa_node_id(), NULL, + nr_allocated += alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, nr_pages - nr_allocated, NULL, page_array + nr_allocated); return nr_allocated; @@ -2273,7 +2273,7 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, * It can accelerate memory allocation especially interleaving * allocate memory. */ -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array) { struct mempolicy *pol = &default_policy; @@ -2293,8 +2293,8 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, nid = numa_node_id(); nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid); - return __alloc_pages_bulk(gfp, nid, nodemask, - nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, nodemask, + nr_pages, NULL, page_array); } int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index edb79a55a252..58c0e8b948a4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4380,7 +4380,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, * * Returns the number of pages on the list or array. */ -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array) @@ -4516,7 +4516,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, pcp_trylock_finish(UP_flags); failed: - page = __alloc_pages(gfp, 0, preferred_nid, nodemask); + page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask); if (page) { if (page_list) list_add(&page->lru, page_list); @@ -4527,13 +4527,13 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, goto out; } -EXPORT_SYMBOL_GPL(__alloc_pages_bulk); +EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof); /* * This is the 'heart' of the zoned buddy allocator. */ -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, - nodemask_t *nodemask) +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, + int preferred_nid, nodemask_t *nodemask) { struct page *page; unsigned int alloc_flags = ALLOC_WMARK_LOW; @@ -4595,38 +4595,38 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, return page; } -EXPORT_SYMBOL(__alloc_pages); +EXPORT_SYMBOL(__alloc_pages_noprof); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { - struct page *page = __alloc_pages(gfp | __GFP_COMP, order, + struct page *page = __alloc_pages_noprof(gfp | __GFP_COMP, order, preferred_nid, nodemask); return page_rmappable_folio(page); } -EXPORT_SYMBOL(__folio_alloc); +EXPORT_SYMBOL(__folio_alloc_noprof); /* * Common helper functions. Never use with __GFP_HIGHMEM because the returned * address cannot represent highmem pages. Use alloc_pages and then kmap if * you need to access high mem. */ -unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order) +unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order) { struct page *page; - page = alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order); + page = alloc_pages_noprof(gfp_mask & ~__GFP_HIGHMEM, order); if (!page) return 0; return (unsigned long) page_address(page); } -EXPORT_SYMBOL(__get_free_pages); +EXPORT_SYMBOL(get_free_pages_noprof); -unsigned long get_zeroed_page(gfp_t gfp_mask) +unsigned long get_zeroed_page_noprof(gfp_t gfp_mask) { - return __get_free_page(gfp_mask | __GFP_ZERO); + return get_free_pages_noprof(gfp_mask | __GFP_ZERO, 0); } -EXPORT_SYMBOL(get_zeroed_page); +EXPORT_SYMBOL(get_zeroed_page_noprof); /** * __free_pages - Free pages allocated with alloc_pages(). @@ -4818,7 +4818,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, } /** - * alloc_pages_exact - allocate an exact number physically-contiguous pages. + * alloc_pages_exact_noprof - allocate an exact number physically-contiguous pages. * @size: the number of bytes to allocate * @gfp_mask: GFP flags for the allocation, must not contain __GFP_COMP * @@ -4832,7 +4832,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, * * Return: pointer to the allocated area or %NULL in case of error. */ -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); unsigned long addr; @@ -4840,13 +4840,13 @@ void *alloc_pages_exact(size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - addr = __get_free_pages(gfp_mask, order); + addr = get_free_pages_noprof(gfp_mask, order); return make_alloc_exact(addr, order, size); } -EXPORT_SYMBOL(alloc_pages_exact); +EXPORT_SYMBOL(alloc_pages_exact_noprof); /** - * alloc_pages_exact_nid - allocate an exact number of physically-contiguous + * alloc_pages_exact_nid_noprof - allocate an exact number of physically-contiguous * pages on a node. * @nid: the preferred node ID where memory should be allocated * @size: the number of bytes to allocate @@ -4857,7 +4857,7 @@ EXPORT_SYMBOL(alloc_pages_exact); * * Return: pointer to the allocated area or %NULL in case of error. */ -void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) +void * __meminit alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); struct page *p; @@ -4865,7 +4865,7 @@ void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - p = alloc_pages_node(nid, gfp_mask, order); + p = alloc_pages_node_noprof(nid, gfp_mask, order); if (!p) return NULL; return make_alloc_exact((unsigned long)page_address(p), order, size); @@ -6283,7 +6283,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, } /** - * alloc_contig_range() -- tries to allocate given range of pages + * alloc_contig_range_noprof() -- tries to allocate given range of pages * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate * @migratetype: migratetype of the underlying pageblocks (either @@ -6303,7 +6303,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * pages which PFN is in [start, end) are allocated for the caller and * need to be freed with free_contig_range(). */ -int alloc_contig_range(unsigned long start, unsigned long end, +int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; @@ -6427,15 +6427,15 @@ int alloc_contig_range(unsigned long start, unsigned long end, undo_isolate_page_range(start, end, migratetype); return ret; } -EXPORT_SYMBOL(alloc_contig_range); +EXPORT_SYMBOL(alloc_contig_range_noprof); static int __alloc_contig_pages(unsigned long start_pfn, unsigned long nr_pages, gfp_t gfp_mask) { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return alloc_contig_range_noprof(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, @@ -6470,7 +6470,7 @@ static bool zone_spans_last_pfn(const struct zone *zone, } /** - * alloc_contig_pages() -- tries to find and allocate contiguous range of pages + * alloc_contig_pages_noprof() -- tries to find and allocate contiguous range of pages * @nr_pages: Number of contiguous pages to allocate * @gfp_mask: GFP mask to limit search and used during compaction * @nid: Target node @@ -6490,8 +6490,8 @@ static bool zone_spans_last_pfn(const struct zone *zone, * * Return: pointer to contiguous pages on success, or NULL if not successful. */ -struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask) +struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask) { unsigned long ret, pfn, flags; struct zonelist *zonelist; From patchwork Mon Feb 12 21:39:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D6AAC4829F for ; Mon, 12 Feb 2024 21:40:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4927D6B009E; Mon, 12 Feb 2024 16:40:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 441646B009F; Mon, 12 Feb 2024 16:40:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BBB36B00A0; Mon, 12 Feb 2024 16:40:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 176746B009E for ; Mon, 12 Feb 2024 16:40:15 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DD7F514049B for ; Mon, 12 Feb 2024 21:40:14 +0000 (UTC) X-FDA: 81784470348.10.95AAC20 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 0E533C0002 for ; Mon, 12 Feb 2024 21:40:12 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZLqfeUN+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3PJDKZQYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3PJDKZQYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774013; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZrBz9DyywfwO606ZQMwpTbc2aylPEGoXqCQTswmuPUY=; b=nyr3ig6JHQgPNQNAt4s+iEI1Ug5/SXmFbufUHoe+hHjVyDGSkhHQEkLUmgUqfEKeQkqVDu DuVhh44c5dCQiCOsZ4lUEbJmEOZfR506k2FdQfYo92JmEpAcxDSs77VHnzkA/AS2JZhP8b 6O67RYLGYKtzlENhHcrGYs4WDmDdBRU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZLqfeUN+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3PJDKZQYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3PJDKZQYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774013; a=rsa-sha256; cv=none; b=W3LLdf+T3g+vFPeQwb35lN0nark+VAaRBS41NMsdS4b8rBgV8UluFkBugagEH8M7QoEMAw G8uRG48+0lPRsFDwTD6chLGjGw9uhCpOEfwkonml4tDsqXi3LHexB25vqr0jwcyNAJ26to 8G+w9o66Jp7XoH6F42xbf+PVSpZ0X6I= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-607838c0800so506607b3.1 for ; Mon, 12 Feb 2024 13:40:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774012; x=1708378812; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZrBz9DyywfwO606ZQMwpTbc2aylPEGoXqCQTswmuPUY=; b=ZLqfeUN+61Tkgsnk2Qp6vAWWz3+kANGaLkdOPkMMXgVWWADqWaCbtTx8X51NEe3Sje bhxL4sU3ycF/OWWDRE0eeKvmR0KFMKOwjyGT/YTDjuzkxe8Ssg5Eirt7SQHoqv01FKDH GKoFOruA2GnOSd1yPLo4DRboyQYXpLT98rSbOBQ5GZQZfUjkilS4e3Mi0UAjLtb0h468 at+lWYdzcADRjbzjEwqrAULDUb3NFCIl8CXh4YpLbmdzj7RICB2tFGu2LBqa1uqyu1bl nay7s2JZFz/A1Y/vYwbvMhlhQCbPAMTcPc9XDTN+B9pKbgNOR01XpqfTGw9czB+/jAvf 8QwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774012; x=1708378812; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZrBz9DyywfwO606ZQMwpTbc2aylPEGoXqCQTswmuPUY=; b=mTWdziUrZCXKNiOjE/ZTBEjggI5Vzpq/ULMYzweW/yV25DqgVJsTDGKDNg4rn/VrgZ Hk4twxNDkrsDfFxTQc0g0RzEEZ9FS7LzmSjLk1pCVLCZlV8w5bYgS6Zq7vHyZQCitDTf vhbasdeYXIzvy1qX0cpC0iSWrtoHXcEC792Z1BZQq5zeCpRNBcpACOY8UfgFMIijzzXm jLJxsvoI3UCKuLgkyNWYEfLnSocxMBhejOCFocG3CBsSdn3zdQE0JfKCRiFpPuMlYWZC uv4AviwSldUCxomu9IIIPlsbsyLBD7Ropl9S35MMdo15pMopkHDba6N/Ew3rOInyLHHu 92Lg== X-Forwarded-Encrypted: i=1; AJvYcCV8gnCmdVxq0Rfhf9ITLwC8AzLZRPbkfyWOmU6xdMMHcfAXkgUOA90Pi9Vlz40bKXSN+yqj/Sxq8dPZfsvPmxN4m3E= X-Gm-Message-State: AOJu0Yw0Ct9Rl1NwG9mmdtty+Ze2+SUIoLsjzpMWBG1V/UnhGtBKQ12j o2UqNi8A48wV2PL0U/XsQPqED+q8xR32stf7E4pkrw1/6cJ1j9CyIsQA6pVAdAflab8R8/Ocfqx UwA== X-Google-Smtp-Source: AGHT+IGdELNRz/U7g++PhfoEsyUKyEa0DYZtdFbwbl9fcNHM0h9+fVLMKP8bhlCRwXrMrONYMoTWo6orz2A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a25:ad02:0:b0:dcc:2267:796e with SMTP id y2-20020a25ad02000000b00dcc2267796emr133364ybi.2.1707774012029; Mon, 12 Feb 2024 13:40:12 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:04 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-19-surenb@google.com> Subject: [PATCH v3 18/35] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0E533C0002 X-Stat-Signature: aodsj931emju1ket3ckjogeur7f8raaq X-Rspam-User: X-HE-Tag: 1707774012-679066 X-HE-Meta: U2FsdGVkX18xpHRMC3NQ6HD/owebveaM/ffhkxPz6vtYOVHX0nJimHDWaEg7miWvDfHbwGMb0h4uOzmXCScU3l8071i3HhmJtC2btkXW8FewfK95W28Sso1abeoIEWtvcXLByS2HSUEYpGXGsyb6HSYX1IhpWvw3GCa7cTDc+pD1yolY1vi9TMnay0Q5qPUsFRwW/jtrp75CJ/oXUQPkOxjmQawq3hTUy1GUx571ROjk5gZEyD6SjGocSLxHf2fYVPhajSMBrWqSl7Dby6OC/wA/ZwKzoLBlidwgsWtq4NVrCj3z1B34OP2nCfTdZpxqI/zg95KkCwHAXLv9v2AuaIqL6Q/pYMlWAN/oAxz8Z/Y7fgHT9ywQTAOKcISnl0BujNKBrA+mhEkMnu2ml0Wy4LXBVBMBe2Cpv0U+VT20u2EihBs7BkumGRhH726Sykyt7e8F53V/Ollgc5Aqno/c3hJahDh1clS00+imic6lR8IyRDsXtRR9qV00C7H+3IFp3qj7Em8NhBNHSgUES8IekYRXRp3bLU55ojc3TfDEZnx0+5IgHka1uSOKM7moJ5QMkXFrl/WhDmqW9eSDBTTZHoERt696K3vVLA5OyuT9ZeYD13hJVz1PQx4kzEwmAI1C7gizK4PrOp2oyajCQiON+NYxD6RWRQRj/cgG5m9+AxE6NsLC/r8oNpJpBITzQxdKLrLXD1H3EOuZGbV19MmOMptx7GOsC2omakNzgnxr94iHsQsx8pP0sOfmPAAJFkDkGBOX2bexV3vzwHUO/gzhmaVBzJ4IwQxKTAGNkVf1Rok4TcQlw4bT8+hstTvFGOsjATCTb3CvwUEqHbN/jLPpnmJJi7w105KQPslcN26IDss8PkfjE8eX0xYAdPy0Svnf1K4wKOW8FU0mn4nLMWCqyt9B0cJl8QsVmuB1C+cbuSVWDf3XxH+rfiN+UMW/NLYGUFAo+Bm3cBRboNP6OlS 0yiCsrh3 v6nSpIUYiTSyYyHdp4vp/9EXqMQ1lxNhJ/8olHvTiz144igwC5rlG/Yd5OvzRAZH1CH2nlI46q4bq7ydBnY0nKm8QFfPWIQ0msA3CSQ+yWGXo1TxV6XrndNEfTDdwHafQl7IHe6zoEK5GegN1jHoPJGY9vfnl0Zg65DZTx0awoe6/o1kH9l52vCLZ+yrOLB9tyRRNVztNnV/GHZongXqRSqk+WkKOa3qsDd/V3iuKT6lG/R5T3tAcYFj1yA6Z8CV6+iaxkh/8/lpj2FE2m7dfn6JlLlK/lYnxCcEpA1qcuxk4FC3lpUzjH6vP1WuoHq1exA/d1v787GxOV4/yDf7DiUJVktfYOgkD39helm57fGapbeO+JUWiwD41UNBc5TN/FwyqPC30GYc+MjZ6XU0mHOXx/IZel/Chva8MMDIvWkbNYNIccSUf7UhnE1kUoEXn9OuzYUDQNY/2yiTH+sKvJVNq/rDvas8P8HTGuLQ28p7olI2lf5tZUTuF9ouuKMt07JQ3LIg87QidGvDoMIqW1whWVGEaUnCbBpXGmqJ71nHOM5UupLR2CEMlLKIy0ot1Wtvhf2UubjO077E7aWyyzGVJIWbnebIezoC0Kk/XmqdkdPCbWGP3IqTS6sReZ7dqd8pdU1Ns1gIGdKObDVnUKFpTvZocP6pgx6/YTPaCcS40Zdg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page. Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 3 files changed, 34 insertions(+) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index a060c26eb449..0174aff5e871 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -62,11 +62,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* New reference with 0 bytes accounted */ + alloc_tag_add(codetag_ref_from_page_ext(page_ext), tag, 0); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 94c958f7ebb5..86daae671319 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -2899,6 +2900,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, nr); + pgalloc_tag_split(head, nr); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 58c0e8b948a4..4bc5b4720fee 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2621,6 +2621,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); } EXPORT_SYMBOL_GPL(split_page); @@ -4806,6 +4807,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); while (page < --last) set_page_refcounted(last); From patchwork Mon Feb 12 21:39:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCE1BC4829E for ; Mon, 12 Feb 2024 21:40:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 442896B00A1; Mon, 12 Feb 2024 16:40:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F24D6B00A2; Mon, 12 Feb 2024 16:40:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26ABB6B00A3; Mon, 12 Feb 2024 16:40:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 12EAE6B00A1 for ; Mon, 12 Feb 2024 16:40:17 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D8E61401CB for ; Mon, 12 Feb 2024 21:40:16 +0000 (UTC) X-FDA: 81784470432.24.2EEED5F Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 44B0740011 for ; Mon, 12 Feb 2024 21:40:15 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=csVAP7LO; spf=pass (imf12.hostedemail.com: domain of 3PpDKZQYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3PpDKZQYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774015; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q3nAahWS0BuDAP/EiPEY1h7hxq9o0SR1xq9twkhL8Tk=; b=3nUWWWp1cN+rMMUv55rk2xoOx1Utq/pCJLld/6yhcoS5zPkUa07g/rFYEaHoOjLD+eF2EI JrxJN0h3FrDN7HXinXeaaGWWqn+0nyVPbKHGyPu31Rd5r6FdN1aKlFu+/7KaG/tL2CKQXj 4yfukNRRgn2vj0m+ubBA8IYEmRU2QLE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774015; a=rsa-sha256; cv=none; b=wMN2MYS3mLpQEd4rKXhqqBATH9BJ4QXUBmUsJ+KLsP4HIEFR+f1pQTE+mNSNyjA+Zixh2k wIs6G9wg6+HA4vzqLqYYGNIR8N9ZgskQSSChTIWiDncSwqFOE7GXL0TqZmjMWtQPkkLJEn IlcC3/eSJJtv1R7fsS23sZRBbGrw0vU= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=csVAP7LO; spf=pass (imf12.hostedemail.com: domain of 3PpDKZQYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3PpDKZQYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b2682870so6221582276.0 for ; Mon, 12 Feb 2024 13:40:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774014; x=1708378814; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=q3nAahWS0BuDAP/EiPEY1h7hxq9o0SR1xq9twkhL8Tk=; b=csVAP7LOvUUVichGXUrfa1haKn6p3/pVohR1y7Nv+CovuAf9ovAhajETCpN/gKrwbF IXR5CizZTi7z/6/OCsdyKYE5yUjAswITExdJFA158y7A0e/p/C8hP2lmYx1WIaaZisLM SlnakUFQLKevAlYV2qCGN6RkSyDbwSVyyDsS1FGKS3GaHOx4EF0WGj/c4JBxasNl0j4s 86ctMDtbHpu3RbdHeVgy/BUE8TuHhub+yfatZI2GL/RToAiVMpPEGYebQGAWppU+KN19 gisBGoh6GhHSS+HcP/gpLWM90/8sEFooWZ/iuaXV+M63FpPy3Cmj19q5uiDKp6EQSv8L DBoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774014; x=1708378814; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=q3nAahWS0BuDAP/EiPEY1h7hxq9o0SR1xq9twkhL8Tk=; b=hZdSUDVMjOtytdF8d6AIhJAwfWdpnkjhfZVQ+X7I/THumddkCvHETwhAGiMuOKny5n TRoKxwZ+EwvJrM5BVUMVurcqdxJutO2u2BtbUOxCmpW34TRk71fVSE62umL412zi5C4L a5dd5YqBGOEGhlkmXwVIyCY4AT0RVz7zbYyrZ96XXADKbaTrW8/e04viSzKGn+FIE8y6 MuialcwJY6KpdOdKVEnGEbBjK3VVMjH3hC2LlXxLlCKidBi9KtmJ1idXWL2R8QWX6nYX QPV6/xUM65ebzs4eMBCJdr9Ympx1T/FgsZIl4Pf+Z1gjo16CjAM/pzE9XIykAp0vtCz2 cIDg== X-Forwarded-Encrypted: i=1; AJvYcCVfG9V5f7aO5WBnsFpHs4SFzTNDRKXQDvrrU4Le+FHtS7+mPshFgh1kLj6QWwqCAX3qVAK5FiNN0+H3Xsix/YHlpf4= X-Gm-Message-State: AOJu0YzK5oYq+S0hzTqj7uH8nWTv40/yQy5NfbKCPPAZbVanzlXa60Ik T867nScECA0i66cHHL0NjhsQy5+BuZRghpnLraZuBb5y6tdfMg92e9h9rhs0lzI5s7uQ4r1MbG0 6aA== X-Google-Smtp-Source: AGHT+IFjfWZYV/B1TC2tamqn5wnV41GSoY2BgJr6sKI43uRZs97PbAp/sxw1wS58qt9V9k9WwRg1VI3jBIQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:70c:b0:dc6:fec4:1c26 with SMTP id k12-20020a056902070c00b00dc6fec41c26mr2112341ybt.1.1707774014153; Mon, 12 Feb 2024 13:40:14 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:05 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-20-surenb@google.com> Subject: [PATCH v3 19/35] mm/page_ext: enable early_page_ext when CONFIG_MEM_ALLOC_PROFILING_DEBUG=y From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: 5dz3j4kx94thkfote86839rq671bza9c X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 44B0740011 X-Rspam-User: X-HE-Tag: 1707774015-222076 X-HE-Meta: U2FsdGVkX195vA18+AgKDDBLhKJF9UOGz2ktGHkwJ3iGeyrURpc0gdxWSMITks0oLADpHah/penE+xe4i3hgRzTvR6dFpzIvLZ7g47N9+kVNIxNXQHKZXfwbbcl2WueLMgqBAhYPGgNtxZYdRe76sRtcae2saNgRXT2l8ET/juEEBN7UHuPBUq8tbnHUHcLUJu3T/3e481OMY0B6ICKNuOkgNGAIRIgNPJxw6ANZVSgZeF29Q9xxeqrY2sWopq0rhS0AXcm949g7niR07fmZizjv6jAI5ThMdn2sAQxPXLdrkkDrxbmX/3xKTzpkc90/vdhXBE8PZ37fa/TnP8hkBqeENz6SNgTzE68K85aFPe+il0MbagEVQww4C0yz3E9pxUKOz3N87rU5sp6+UCgzDBysJWcsXetv8weH6tf4D9ZH8kX3tTcyzgKMvm2YAZGau5JP31a/WKP3YK+h1qKwEUMBpVGV6zvSNgI6H860MV1NG2J0lYItgSi+knTi7H+ZnTpXLV7P69O06RB3XTZcE2AteO/lqN9Kg7/tmqkInWhgBT2iEsAPVA07duP49uh8LW09zgJZWIW1XRIWBqSTzzR4vFIhK2xBoiKz/vfWZjf5BZM/jDa8X9mtm/h0hs54HSZG9B3NnmfSRL6O0ocOCNCCiAoK7SzXUdtXYRQBUhiLAPGzroaHmaJIjvrCo0Pnp0d3ayM02IsOnQKlB0DzQkS1R1dycGdznE868/tXrcASWaYR42gnwn5RpUx/J3fcRnJx2/k1h1iMTQudbRB2QvKicI+ivZaYEEtyO7vfiJFzXoFI/h2GyFlCRp5owiQ2/QyB57uUjQGPFeC/4MhWxwZpcOaZv0nTtTUEG8YCxUkgwBa+KdLkdX6hAxsADoUAT2HkQSewnmzPGiJDfcbbcYP2mIrXu0cJUQAUOHZfPMr23Vc+zU/ESD+l90Zyy3jwHV+ZFFHweeVRbAetXiN pcR5tPIB eA0yMxSARRGrf0X5KPWbKGEgvcoaXWp9/nm7UEiC+JnKaWokwMEYUapsBLN7e85Z1FeewiB8rz2j1K1GSQO4FU6OOWnFQKpoIkC24PaqXj7Hm00NatLUmIrjF+PrDDlzM3nfP/+6qlhrcgwKCHIbDRahbeB/kcxqVTHm9D3ejbmsa43w2FGwNt9Vv5HdNZVA9x98eJPhcjydOVpXjf0alfYJpgH8uTByHYxrTdgygJj44TKf42BOeVXHlvtm0YpENLCVy+COUxIM0BrNu7j+Y8UPEZMNIPh5CNwb9kYgbNAHl6EYWCJyBK6Bm4vJJr/3cskS3PGy7UoVN04SUyBpTZR7RYj9xi+YMpviKX7LuBI2hx9qRlKFLTVvOzigTqhk7BjvVCYXNB0zCmciRSjEF68Wotdz5gUd/U63MN0oAc4vWewDDWTPOagUdNnf54ZvAXkIyCjiJNmHnCpi6yFIJlhPPYA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For all page allocations to be tagged, page_ext has to be initialized before the first page allocation. Early tasks allocate their stacks using page allocator before alloc_node_page_ext() initializes page_ext area, unless early_page_ext is enabled. Therefore these allocations will generate a warning when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Enable early_page_ext whenever CONFIG_MEM_ALLOC_PROFILING_DEBUG=y to ensure page_ext initialization prior to any page allocation. This will have all the negative effects associated with early_page_ext, such as possible longer boot time, therefore we enable it only when debugging with CONFIG_MEM_ALLOC_PROFILING_DEBUG enabled and not universally for CONFIG_MEM_ALLOC_PROFILING. Signed-off-by: Suren Baghdasaryan --- mm/page_ext.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/page_ext.c b/mm/page_ext.c index 3c58fe8a24df..e7d8f1a5589e 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -95,7 +95,16 @@ unsigned long page_ext_size; static unsigned long total_usage; +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +/* + * To ensure correct allocation tagging for pages, page_ext should be available + * before the first page allocation. Otherwise early task stacks will be + * allocated before page_ext initialization and missing tags will be flagged. + */ +bool early_page_ext __meminitdata = true; +#else bool early_page_ext __meminitdata; +#endif static int __init setup_early_page_ext(char *str) { early_page_ext = true; From patchwork Mon Feb 12 21:39:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6735C4829D for ; Mon, 12 Feb 2024 21:40:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B4C36B00A4; Mon, 12 Feb 2024 16:40:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 666A46B00A5; Mon, 12 Feb 2024 16:40:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48F936B00A6; Mon, 12 Feb 2024 16:40:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 34B2D6B00A4 for ; Mon, 12 Feb 2024 16:40:20 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F1003160255 for ; Mon, 12 Feb 2024 21:40:19 +0000 (UTC) X-FDA: 81784470558.13.F6369F9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf08.hostedemail.com (Postfix) with ESMTP id 337DC160009 for ; Mon, 12 Feb 2024 21:40:17 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=04O3LEFK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 3QJDKZQYKCL8xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3QJDKZQYKCL8xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774017; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xFf/g8Eh9xc1yf0azF3fMp/AEmHqfeMm/yqsFpU3vIo=; b=xrts/PwswQ6QPrNPYBLIjpUEq5zVCKsd0TUapU/w8MC8RoxMSqQ/c76/FbNQAdabA7lNFh OBEmSSOt1kTC99s6TeNyfu5HQPUaJblh+TSHWosvpAvyxRzULz74Opz4fbjt0rA2OmovMp zX9iQT7a/AQtWqpwoa/RHqFVh0S3HNI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=04O3LEFK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 3QJDKZQYKCL8xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3QJDKZQYKCL8xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774017; a=rsa-sha256; cv=none; b=5Rcj1Vex6B9E3wPPtc8+qfbN/ehi5SezK/imvwCBr38jMTYk4wcFpQpueEMOw4Ac30SZdb cTIyTiMmEV3pxDE3T0gsaVYSSIh/3qkZh/4XShnUvo48DaFx11xv1UEo4aU8jEwvKRkvc1 r7IIUGKQJoQB75vsiI7RbnAxxaQn7bg= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60761bdbd4cso19838387b3.3 for ; Mon, 12 Feb 2024 13:40:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774016; x=1708378816; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xFf/g8Eh9xc1yf0azF3fMp/AEmHqfeMm/yqsFpU3vIo=; b=04O3LEFKHDmlJDABQ+0V8yh1hwEqKWqQpyy7ZuiAXZ4wuip0qZ/ZPWsz0D5SHOjLOu b0dkN0ge5UF1Rs7nKjpTTR8Jf9IzP2xTNowOYJHqpyUy5ABhp//nQQ+C2lFSTjjp4+Cf 6vwDr4ocuyOsoU8EHEY1Wo6DzUxNYSRfEvtLlOjUuh58dHKkzyslOIEgOCsd6NDPV84e bEHlyu4hoHhY+riB+uTxc2rl4DId6OX3FTOYrSBY6Syu5rEDNRdJgAX/NVDRXUlqOIMh tsYmAizM+33jhNDrNYSF149Vt8VHev9+q0JBXp1ORztgIVD6M7IO0VSFKdIze2An+FtR 8cvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774016; x=1708378816; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xFf/g8Eh9xc1yf0azF3fMp/AEmHqfeMm/yqsFpU3vIo=; b=gFqOJkgyggq9eryj5pMaL6qMz8vps7L4CF9fxaZQinrp5U6k3XbbwICvF3716I5RLV 06V/M47iXehPvDP4ClyYpjBxhp+Q1Ire4IgmpNT3Mafs8NMPRTd6N+lhhgcla5ilYvHw 1G/36SvG05c6u4QNQP0Ixv8vXyWOOe5b/aHdYTzFpOfCm1Ap3+tym6wECNBLEjgs61Fu sSK7VRKPOpt/6bkjUl/8yqzLDWAiGDpj11xAZMJ0nFWbflEaBYmQ4OS0yPRBkWF9AcsK pTwPSGrYBaHpZP6j1TIsJP9MjLE3khjQVDgmK8OiURJr903zn1ANcgaHR7C/VytZlsif 0uQQ== X-Forwarded-Encrypted: i=1; AJvYcCXpR3ac1UuPiQ+skUX0SVDUfnzdGe0CFdvqdtkLgy98AxZuSATITQBUGaUayxWTGgzMzmD9t8KttqfMf94IS3oISAM= X-Gm-Message-State: AOJu0YyoYEV5RP69DCPdmh1Lww04HkSnfgd7vmnKL5hWLZeiTB+ECwtz HKIm8BcLz5gvYnChNLF6kPBnjSad0z8EQWBa/bprVLbmQYlRl3tXXKSnGHPj4SWIE81Ce3P0pmK FDw== X-Google-Smtp-Source: AGHT+IEjOAHzMTDk3G6Mkq9BQfM87/zY/tGgKD9M86mlp2G5yR63uKFbjyK/9+Jhrb/BiK/AnGAds8f8uig= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a25:8391:0:b0:dc2:1cd6:346e with SMTP id t17-20020a258391000000b00dc21cd6346emr2029085ybk.8.1707774016324; Mon, 12 Feb 2024 13:40:16 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:06 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-21-surenb@google.com> Subject: [PATCH v3 20/35] lib: add codetag reference into slabobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: wjs88jo1kc9zybqh3oyyt3hfd7xnbiqm X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 337DC160009 X-HE-Tag: 1707774017-699410 X-HE-Meta: U2FsdGVkX18we9gzbtRjh6zzEhDvm3ezZMwEOiXeocimEz1+BwgrDG2DXrGHkW3QjKU3ucA9FjX6DkmAGU5TGcX6mLOa3xxr+28zDOyu/AF8jDWBXeQXhfzV5UfXFYWzpAgYN3zarrue49lsk5wnTF8Q+zKYJT+xVJKnjCQECa4TRDG4bXuxvajWUZ9TdcwT8bGQaMGzYcxdW9vErqJMCgs7xuJloedPnxOwLo4bGDScDH8nQCm6CQxlYO66moNdUorTHLc6JkCmHxOE/T/KBmO45q2oje4ylxzOyNwvN/8W7ftoB3vzsdr9CDrcIgbATTYCmb6I6S/mloyL6fsVXG5N8qTufBVF9nYqLs1ecbsv6NFO8jXKZh9AcImEv9V10UlB2jwHcUjCUxMPrQAgQx+gDgtDBBxM+bEv82nrhz5Kos6T66nnx8w1FEydiI2kWi2eAQCWk+KbESiltFkFMZmKilG3i21lkGb37py7oSb7XM118S0JNHInwCEucgU2IukN4ZNIvaZBLHPtuDLIu6ImgtZgM5ogs4PPK+itYNzKnXejVfoQqMfMtlGXh0TMeNfPcVYhMjjk/HmJ4HmONlAEHoiySFCgIBdFKlBXY0h8wCJo8yzTzugbeqhQ42qN8xc6NR3BVqkzWjRWm0e4gwzbo4NBIiosOmqbb5Y0bnA6GsJAkpsJMVnCqMVwkkfeV45sJUIEQ9U16llLmO2xcaAzKnPd+d2sYsLp4eJNSJMvuWIlrbJJqpdCeLSmVpmSqVXa3P2p5cSSPtvU3ZLEFRD4SNnCe8uelJi9BegfcyKpw0frxXwghwIatIrwVDo73ErsatEQ//V5reEyV9Nz2ajf0vRhJJtdmyT5cUpu4Wb6IUxiI+klZRLamm2dHLOfXrbWjH6oARLvdyiYuWfXwl7GUo7INDeb9gdWrFZP3l04mGiNUzKpUVnxBdMApeBXKYPUFB2gCCdxn+bFj+F BLua4+DX hqumkkKpyDbVtw0pxFBvupRp2XlWnAlG9SAQTFzgZPJai7YyG/To2wznT/leZvfnkN+8QWSHiR0JGZl5vMjN7jPg/J7RArXtgNQY7PZWg2DGsvk0tez2U/j1V8sl6VAUtx/Xcx3hra+ep+fbpCYqapjIRLaMwdryu8NdDgLlTHzmRdG9C6KcMnQN8smCMRxVTd6+uvbK3tJ2i+luAAwJhjMJ9DCjwmRpFU82bI46tQ//rfFKVad1gjlA0P+faciqJdjboynU6S0+osuBXXX8PVPL+RrlwSWNRpERdhG+V6Jb/E7zeE3kulxF9fwltbYMuUG9vGvqVJLhIZudDfrMdRH7uu/JOnp3PNXcb3C2nmc6WHUe9Mo5KkeEW8LSW3c4iLHCKbVJEQizUsTFoGtMpB1XWtctQJiZ/nXCNdyMUFRWKwwaZ7YOlEGN2c5j3nCmtt1nHkCn2zosDU8bT3nbSFC8S0PqQDtpto+ObIRH8JBhX8o+g14kYwaDvr3U6uwQsb2kqJHeGfKUb6CXk/JIuSSg2M6Ue7I0lgS4WJvdEEToNQstOZPbk2HHIQvtC0dONGzSjU9MaFcZDDvKco1vFWKiaQ/k6XZhnuCiOovIsaXXTho/qlDuL0hdvM+tGdeBtcOC+kINlNercFOYkojBVLPnczZlBNA3uLIqJz7tsx4nN+kL6jsQnki+Hjx0rIzUr+49SXsHk4W5eflqVpxEJh/mBMg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To store code tag for every slab object, a codetag reference is embedded into slabobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/memcontrol.h | 5 +++++ lib/Kconfig.debug | 1 + mm/slab.h | 4 ++++ 3 files changed, 10 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index f3584e98b640..2b010316016c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1653,7 +1653,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, * if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref ref; +#endif } __aligned(8); static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 7bbdb0ddb011..9ecfcdb54417 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -979,6 +979,7 @@ config MEM_ALLOC_PROFILING depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION + select SLAB_OBJ_EXT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/mm/slab.h b/mm/slab.h index 77cf7474fe46..224a4b2305fb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -569,6 +569,10 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, static inline bool need_slab_obj_ext(void) { +#ifdef CONFIG_MEM_ALLOC_PROFILING + if (mem_alloc_profiling_enabled()) + return true; +#endif /* * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally * inside memcg_slab_post_alloc_hook. No other users for now. From patchwork Mon Feb 12 21:39:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A4FDC4829F for ; Mon, 12 Feb 2024 21:40:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 888F76B00A6; Mon, 12 Feb 2024 16:40:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 837C66B00A7; Mon, 12 Feb 2024 16:40:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63DC56B00A8; Mon, 12 Feb 2024 16:40:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4F1A06B00A6 for ; Mon, 12 Feb 2024 16:40:21 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 26B02801A5 for ; Mon, 12 Feb 2024 21:40:21 +0000 (UTC) X-FDA: 81784470642.26.FC990E8 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 7E67C100015 for ; Mon, 12 Feb 2024 21:40:19 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3CAOj0gW; spf=pass (imf05.hostedemail.com: domain of 3QpDKZQYKCMEz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3QpDKZQYKCMEz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774019; a=rsa-sha256; cv=none; b=3S7/9ypd6NNvZI6OzYpA3O/kwWCoz4Uin4wlSsYSL8hYPVOwUmDU0jK4vILAdUlXXksjig enuki9++Bbsw6j7lTsS+AvQH7b+fAMgnj7CEoSy2FvL4Zh3CSeVn7ujhT1jLfGbLfHj8t5 Xz5/wLzboZYbKEFS49aJqbboHUYfYr4= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3CAOj0gW; spf=pass (imf05.hostedemail.com: domain of 3QpDKZQYKCMEz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3QpDKZQYKCMEz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TxFmyXLHYuBt1ZmWD3OCbH5m3f76lMQsgX1qwMm0SnA=; b=G0dVDr9dWJeTSN7x9LyZLW/+hvr5Uy1QgTcu6Tk2nCcJqQB4RWgaPrd3D6aH0gHvPswtbY 3WjGTi5JDotdbMBlcnltiL4GMofYROADOM0uLQxaGxtQf+krkttAGgDwdvYCmNwC3d5tPB /oCoXICbT62Grb19yaWNBbmFWMwuZFE= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b26845cdso5926812276.3 for ; Mon, 12 Feb 2024 13:40:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774018; x=1708378818; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TxFmyXLHYuBt1ZmWD3OCbH5m3f76lMQsgX1qwMm0SnA=; b=3CAOj0gWQGwlngYqSHt9wEjWheIzhWc7wbxoL5dKO411iHdvPOEK4Q6Xh02fK9nwV+ xF4c6fmbVHOSlQjTUWQ5/umFN9RtTZdTNkA0jarKalEyTLZrSGFS90Q1hOiI9Z2BxJN4 ave1MUz96YyR+iRo3N/1rKWe83QNuixNKo1P2fHEocUCGpO6z9OTEEFFenvg5KDrI500 BJ82wzOIYqdMRf0GAyYAl/jc/DkM5ODvw9lBPICAGNqK5anwfM53jvsO9uEZbVbkINPp Hn5LEctSMn4jiPPDFZ38H/eDUmEtzJJr0cQ/1Vw9PrcZvk+KW7fR/Kh6ZMp5IlygZXQn k/1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774018; x=1708378818; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TxFmyXLHYuBt1ZmWD3OCbH5m3f76lMQsgX1qwMm0SnA=; b=BcnXG0YFmwZoWNWMmOvu2uU/pt4I7hlTwHFXsHxxwzU+xdOq91J/k7bGGJfT36XC7i VmnJ3HKM0a2KwuBmht6f7+RiM0jO4oNczIv/2BR4AOMxh57GhKXArQH3X0PtcP24yRjQ 1Xgu6OqmDFBtUblSVnm5/u5C5lCx4ms5qdTgt42aSb6GP7A8ticrf6joLitJ3HBEVB0a FCgk3G2BBv1jiuDMwIfYt57P0SZpBfV8HALYmy2csElxKu/vPdWyZUYGltoC7OOaEiBU UMlhFPcIpfjtEv4K+1am6vVA+eImDuuy8ldj197udcHUqYo5aQZWh8ndUVV8cNHTCfnm rk6Q== X-Forwarded-Encrypted: i=1; AJvYcCXU+5Y3Z9Q8LD/v44BQj84d//ojujKMUC6YzfpJKw7b5cVhRqkJf31cdNqWLO/N/0XBYr4uwpIz53TVJNdBctRmkI0= X-Gm-Message-State: AOJu0YzZ/3WN6JOnWGUVRpOkJtPlHb1mBHD0HX97iju6So3UlkxAEnk5 DNuC4SQPD7ccY72mbLPihHgaJ3aHW//JIlSBqjL10+zEotnLdZeXnoOxJUNdN4+Mu/hLvxSFljA 7Pg== X-Google-Smtp-Source: AGHT+IGBONdcVlXEEGFa18cuaG6bv+Dg1NI+YmaYVRfFeBb82PPmmVaHOveRp5zx3fRCNj/22b8sYtrJmjo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:2291:b0:dcc:5a91:aee9 with SMTP id dn17-20020a056902229100b00dcc5a91aee9mr85473ybb.7.1707774018612; Mon, 12 Feb 2024 13:40:18 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:07 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-22-surenb@google.com> Subject: [PATCH v3 21/35] mm/slab: add allocation accounting into slab allocation and free paths From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 7E67C100015 X-Stat-Signature: e7dfttgbppbzzaqacq1gmseix8499dqa X-Rspam-User: X-HE-Tag: 1707774019-128417 X-HE-Meta: U2FsdGVkX1+ewQ226bwJwsn6EgQBiCv29hKE/gTVnXNLJ7qa1qgiZ3PUEqMEVugqb34UQfpl8oqaE7s7+kJjzOCjpPCxvx3wyuJKCSVjo7jFAjf2ZziONdrkaGenedvH/eMbeQS3M7fqxG1PwL1cYVPPp5lxHQCt2cJjsWsfnmV7in8hfRIL4ANjae3BepCrUjQe+pUytpHJxVnvqOppBgNMC0RghQSBrSKu8VCNo/7XlPC9Yp6S9eGMDHPYfTFVxGUrbdv0sHOXFqCpmLIXmbSLpsNILTqLpG3MPI3BKfX+pxGGmMDXTMTd6tgozHnzq2FKZpx1f5copNO7dct3iqP05a2FPEzFBkf1qmc+jvkFRxLTHYDaIiznBfsCrO88+LcaIVfzxhUwpokIJbMmcazIJ5EuNquOIj1NlE62uEdlOd2mjgrHqCHrDeZKV1Ap6ED4VgynHGeh5csJyWv8tABSC5U4wLjjYdaniMpjkvfJn15FQdw3Evq6kMfwb69nUItZJYfa+GNWFpEtv02qQ6tCvCFT2j3YF8x6In3F6c+e4tAnASVnnUv/cU1/zhsMg82qFZrQqPzsMEJAKrswr42cH4L4h0mfoOApoH7VsD/K85/zx/xz9k+hzRT7MQyvA4CQ5n7Y4A61FVHrQ1CAglmx8EUtMPLiaOs+Mi3rdrneTjk6I2+yB4Y8L8tNdjT65LsRDMjCpGaP0jxzwyw7nfLclKEMM8v/GzRQRjSZx3xz0tkhge8iUBFYMNpAqcdyIGs+S8IkMTFnd2zivPAftYP77LfQAbMNjC8WutOl/yzSPbQpKmHQXf41o9G5zFYrTdodcYkEa4bcWo5drxzqB/kv92ymgF8rq8IXXmVQjDc4yXiU4FRi6uN/UVXerSLTEt/YZXWSKbIXxb1B3qCvkgJ/FME9xobuU2WDfy2VcjvDEadl5QiXXO+aWvuruOJzg690I7S6YoCmW7HBRR+ eiVmd2aR bFrAP0aUAAt8HMUPtRK9wmYrZ90Sj58+c12oLId54a0vLoqcn01LEuW+iMBCSKGK0b9rEEhTYfCxsgeJnZ40XQ/i6B5fFKOieCgiafIhnTXAfccJ5Ylqwkd6K+VuU//Jk4Ss1oHYNilUpwNFtDUjrkCyo9J66788MwHsX3HeP6GTgbsXoyLEgHPlYRW6CUlUIgMsHgu7eaIg9/GuBCFr55qA0aQvNgWGSgUMbpTVWSNcjW5TCzn3Iz9rdcm7wTaKKGeQwl5YWNIIBeV0qrZywrTfXqGiwzyHf/HPnBZLtAXPC+AaP/Kx96cu4Wks0BMqc3mm6qTtuV0B8/QSR6z1kbd03w+81IkldoOBPn5zRNn1w//5H2L4bc9OftqZ13Ot1pzxDFQeuww6SceZYEEvyZ3ruEPIEm7xrqhyW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Account slab allocations using codetag reference embedded into slabobj_ext. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- mm/slab.h | 26 ++++++++++++++++++++++++++ mm/slub.c | 5 +++++ 2 files changed, 31 insertions(+) diff --git a/mm/slab.h b/mm/slab.h index 224a4b2305fb..c4bd0d5348cb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -629,6 +629,32 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) #endif /* CONFIG_SLAB_OBJ_EXT */ +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects) +{ + struct slabobj_ext *obj_exts; + int i; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + for (i = 0; i < objects; i++) { + unsigned int off = obj_to_index(s, slab, p[i]); + + alloc_tag_sub(&obj_exts[off].ref, s->size); + } +} + +#else + +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + #ifdef CONFIG_MEMCG_KMEM void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, enum node_stat_item idx, int nr); diff --git a/mm/slub.c b/mm/slub.c index 9fd96238ed39..f4d5794c1e86 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3821,6 +3821,11 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, s->flags, init_flags); kmsan_slab_alloc(s, p[i], init_flags); obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); +#ifdef CONFIG_MEM_ALLOC_PROFILING + /* obj_exts can be allocated for other reasons */ + if (likely(obj_exts) && mem_alloc_profiling_enabled()) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); +#endif } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); From patchwork Mon Feb 12 21:39:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554060 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DA55C4829D for ; Mon, 12 Feb 2024 21:40:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 294DD6B00A8; Mon, 12 Feb 2024 16:40:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 244D56B00AA; Mon, 12 Feb 2024 16:40:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3DBA6B00A9; Mon, 12 Feb 2024 16:40:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DB5E16B00A7 for ; Mon, 12 Feb 2024 16:40:23 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C100D1209E3 for ; Mon, 12 Feb 2024 21:40:23 +0000 (UTC) X-FDA: 81784470726.09.2FCD608 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id CA76DA0008 for ; Mon, 12 Feb 2024 21:40:21 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pi7UZS0+; spf=pass (imf25.hostedemail.com: domain of 3RJDKZQYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3RJDKZQYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774021; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BW+F7aNEsn42MAqFjWDXBChBmgYKTAcf/l21NhAmWHo=; b=Y8jZeK3p4SWPR/JwjSx+3h5sQWIMeq2+aLrPWW0j0LSoKvvPOjyWhBVZFJAPQVeJDXd0n9 raZmZKVKOtijmXvyitOgRQJ1rFiW60NaViR1UYFU50S/QCUHrijiRgM1IWaGMQDcnLqtIH ypwD8MuRjd2g8rF71kvkFSSJx4JCOd0= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pi7UZS0+; spf=pass (imf25.hostedemail.com: domain of 3RJDKZQYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3RJDKZQYKCMM130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774021; a=rsa-sha256; cv=none; b=2zaeyYXQTsvop2fBYK7S48Gp+zL4dKrJlhI93/DJ3+VLy6Y81/Muz4cm7GvjcbltiqtJWj EeyQoACqyTQbCpozFgMQreAXZtHuRkLu8QtHAiBHExVbcUmNFLQ57e9K1dUpSZYCkE+O6o gGD7TuP/x6NnEb3gKr//kmp8iPbFpiA= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6ceade361so6424188276.0 for ; Mon, 12 Feb 2024 13:40:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774021; x=1708378821; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BW+F7aNEsn42MAqFjWDXBChBmgYKTAcf/l21NhAmWHo=; b=pi7UZS0+OYRsyPMDYuODS02nnu3WiTdlXbsfbQt2tyAO4PR8/bgjzUDiU1d1qsmgW4 67sQS7B6PLrTVs/98Vk2vU8rb81j+7aqlYyn4pVLLQx9AdIsU/XkodGa7tQMiSFWielG mam2ahHtj2wm1fULRMzadt3EO9qdEEk9WtqxYh7W/1BF3V/cnUmSsgapvD8R9L4n+j/Z aJzliVaFiXt1IGjjLNXc53/5BAaRX4A1x3YO4yl/DXP0VPIwv4yM2gMcVIENjg9yf4zO b0c6moLzU4/Z1s7JEG/TFcpozTForQkJ1NbdA0OM0OumxPNm3Y+B0JV/6vFh214sd6+Z x0uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774021; x=1708378821; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BW+F7aNEsn42MAqFjWDXBChBmgYKTAcf/l21NhAmWHo=; b=IFfzRgCYjoGcuCFn8bUfTueb58YyB7W+Wsw95vXx/AgYHYn6o0c04KNaRX5yDVPEJJ YhAi0E8DTSkqgknNj2y8Hqa0nIFZOcTmw6iX5AuBgR2PYyU6t6l+LpDvONvEworjulm3 CZVwRx7e78U3jVLJBYrzugO2v+R4Vetq32ybEthlEmUuOPHq063PkI1X8ODE4j7cuq6Z hKqSuwJNNcWH3Hzypacdm1mb+Xl46+Sq9HLY1bSVPFDwEBHwLcSp0+4RTrTIFu15gEmL M5LCi80hjBjNn3+iklQQQU1ajqUQQj7tDYi7UdRArj/E5DaPXHx2Gbqvery4khzhKuec WrRQ== X-Forwarded-Encrypted: i=1; AJvYcCXRx1fZ8BBmaftqrRMT84hnaNuxhZx7F0L8qNZU1eKUEguiYa1Qiik3Xmxq4VybtAMSA0YyitRRwTEEGoxKaVcpVgM= X-Gm-Message-State: AOJu0Yw/3rTP6f6TP/f2JixNpCC1CoK5qdy0Y+ZYkTT3LgHXKX+Iy/pb sDhsvu2oElEAgKDMAfAoGyM8mJjX5SYytjGPCWBrtOUZcDR/w/JicCZL4irOfmSx5KRvAMAPCQe W8A== X-Google-Smtp-Source: AGHT+IGhsTVWNh39IlIHgl7r6Rr3e1iSmOEiml6TukgPG+gmxAY5xjZC7YFwvZdpHJoCGCGJJSAE2QDryhk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:124a:b0:dc6:9e4a:f950 with SMTP id t10-20020a056902124a00b00dc69e4af950mr2042179ybu.3.1707774020742; Mon, 12 Feb 2024 13:40:20 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:08 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-23-surenb@google.com> Subject: [PATCH v3 22/35] mm/slab: enable slab allocation tagging for kmalloc and friends From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: CA76DA0008 X-Rspam-User: X-Stat-Signature: s6ktrhuxmqfdw3ggbmywa8xk871w4p49 X-Rspamd-Server: rspam01 X-HE-Tag: 1707774021-150110 X-HE-Meta: U2FsdGVkX19fAV3rz4znClZVDO6kdfSHaIYVxs6le1ewFSfHJ0n2I35E3ai7IHnN0QdkqBehBCJ3ORxSntFQEflUK+OMVXKhctA9DUa9rec4E0TfMeFZpm6c0T1qYYrG66rWpNzRy8zilp19EkcSW6hCFKHIcOaPiYBbet1FmWZzRgmUxftp2FULfY0yRmKItMeHcaUFANvVnOW5F4yoNzkv4AMU9TNMfMofFU+uCpag1/zaFh4EgvgprO/vf6omWo5XZfZ1wGGifmhYj7bAOCHtZEoqVdKP44zrAuUQD9PqR3ktDKjDsb1p2RX/9Mtj/Xd3DtnlWNzBZ3oTnF3GEmiKxIaxDxP6tKz5B6i9pPbBLw2fzoc9RutZEMYYET35TSvCtf1+DncALAaGXH9HRULEQNblGIZ2YfekjG9EwmyJw92alH40F8UY2aCp+fa2WbQklNpZsarijLJH6wG/zSFtVu26RbPwIDJCgQJeTg/JnwTlgVUFYITRBvyqca3TWRexikKN5+Xf92YR6YyzW7cRDNlpdzlNEoqncIyclp+lv4gUnG92IW4wl+zcnRAS11BAxT8tf8jqM1kA2I0Qg7BU4vORCsaRLH4KcjDCRiW2PsRlGjIJoiBkqFG09S4TPXnb4B3QfGokvJRMbPWe9w6RQa3yUwqRyZCL9OEwBkuhjPN8R3MVN7WgnDpi3+KJoycbUiVRXZfmkKOd2llwUmMOcjrHepSV8BoDhJGJq9wh2CT1kigqxYUEj6DwXQdo2PAPv/igVT69srXMv1egr2PN4Wtq4KhGwcz0A5G2qjNW9ys+DUGfTZtvwU6HdGoK5qVoLkPwR4PpGRlwTbhyn0+HfZgL6MotUWutHXcyz7QifCr9kLeV2hVSKnx7CMGeq2lePpB/NwhMEY0uCTQPo3mim7fJCer7CdXVQmK51c2B3DMuSjK+aXkB/Ke3STBSUt8uTWRVnQHdGRTsM7y zy0LBJg2 ot9qWH/XGTxiEdv3stYROTIhloTC5Y4i3drDoT+EkGcdjH4ChHy8569TkFBzAbzFWXS8jM1/KMfuHOW/fzgbHVLywi/ttBCwvwQN3tEo8S/4RdLcQOXOyD/JRR4xXN/6s5zZyJGE4h2DNL/dGLeJFAC5R1NCG2gRqC1NGTLoMZ0GfmtbLow7uJ3hs/Ac0fbA54ELn4VbKqA/bog6laP+NgAHiSmP4cMfk0zwM6vig0OKiYMrxuLs/Ill5vD3kHLF8SZVbBH06mAicL4KWPXsGndXR4X86+XBpPbYqQDCnxKqtq4woLwYYqCNHI/8aQpbtwqfL8D/UXhJs+RCsNo0CaaYdNGo3LfEN/JwuIlVPzv3AnwhTOTmMDfHY3zZNYtZyHAM6+lRV6Zq9RXjrk+aRIcIFsyEslHr2VDQ89jXpwiPy/RSQoTAx3EkoAFlj6cGsT+frE1MAbPBZ731jITRsZVAfSmyNx+Z3lAZqiwZWJzyLMAm2oqOhiH4xpAfIc1XbGdfKbOCNJHtbab2hDQsls+HxdAi73/bYnSR/VznkmgqGDJtoaVq+Oki6loqnx6/rTTNUL96f+Gn4PTzTPtrPv/Y8168CySy2ZzOwotP5SV2dfkgUgcvJqo/YANkkn0fbg9Ywc3J/ITVsXg8j8eov+Ojp1yLeAG9H17pUbzCoWnpGpZg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine kmalloc, krealloc, kzalloc, kcalloc, etc. to record allocations and deallocations done by these functions. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- include/linux/fortify-string.h | 5 +- include/linux/slab.h | 177 ++++++++++++++++----------------- include/linux/string.h | 4 +- mm/slab_common.c | 6 +- mm/slub.c | 54 +++++----- mm/util.c | 20 ++-- 6 files changed, 134 insertions(+), 132 deletions(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index 89a6888f2f9e..55f66bd8a366 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -697,9 +697,9 @@ __FORTIFY_INLINE void *memchr_inv(const void * const POS0 p, int c, size_t size) return __real_memchr_inv(p, c, size); } -extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup) +extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup_noprof) __realloc_size(2); -__FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp) +__FORTIFY_INLINE void *kmemdup_noprof(const void * const POS0 p, size_t size, gfp_t gfp) { const size_t p_size = __struct_size(p); @@ -709,6 +709,7 @@ __FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp fortify_panic(__func__); return __real_kmemdup(p, size, gfp); } +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) /** * strcpy - Copy a string into another string buffer diff --git a/include/linux/slab.h b/include/linux/slab.h index 3ac2fc830f0f..910473e07e86 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -230,7 +230,10 @@ int kmem_cache_shrink(struct kmem_cache *s); /* * Common kmalloc functions provided by all allocators */ -void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __realloc_size(2); +void * __must_check krealloc_noprof(const void *objp, size_t new_size, + gfp_t flags) __realloc_size(2); +#define krealloc(...) alloc_hooks(krealloc_noprof(__VA_ARGS__)) + void kfree(const void *objp); void kfree_sensitive(const void *objp); size_t __ksize(const void *objp); @@ -482,7 +485,10 @@ static __always_inline unsigned int __kmalloc_index(size_t size, static_assert(PAGE_SHIFT <= 20); #define kmalloc_index(s) __kmalloc_index(s, true) -void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#include + +void *__kmalloc_noprof(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#define __kmalloc(...) alloc_hooks(__kmalloc_noprof(__VA_ARGS__)) /** * kmem_cache_alloc - Allocate an object @@ -494,9 +500,14 @@ void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_siz * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, - gfp_t gfpflags) __assume_slab_alignment __malloc; +void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, + gfp_t flags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) + +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_lru(...) alloc_hooks(kmem_cache_alloc_lru_noprof(__VA_ARGS__)) + void kmem_cache_free(struct kmem_cache *s, void *objp); /* @@ -507,29 +518,40 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); * Note that interrupts must be enabled when calling these functions. */ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); + +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +#define kmem_cache_alloc_bulk(...) alloc_hooks(kmem_cache_alloc_bulk_noprof(__VA_ARGS__)) static __always_inline void kfree_bulk(size_t size, void **p) { kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment - __malloc; +#define __kmalloc_node(...) alloc_hooks(__kmalloc_node_noprof(__VA_ARGS__)) -void *kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, + int node) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) + +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t flags, size_t size) __assume_kmalloc_alignment __alloc_size(3); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, - int node, size_t size) __assume_kmalloc_alignment +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t size) __assume_kmalloc_alignment __alloc_size(4); -void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment +#define kmalloc_trace(...) alloc_hooks(kmalloc_trace_noprof(__VA_ARGS__)) + +#define kmalloc_node_trace(...) alloc_hooks(kmalloc_node_trace_noprof(__VA_ARGS__)) + +void *kmalloc_large_noprof(size_t size, gfp_t flags) __assume_page_alignment __alloc_size(1); +#define kmalloc_large(...) alloc_hooks(kmalloc_large_noprof(__VA_ARGS__)) -void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_alignment +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) __assume_page_alignment __alloc_size(1); +#define kmalloc_large_node(...) alloc_hooks(kmalloc_large_node_noprof(__VA_ARGS__)) /** * kmalloc - allocate kernel memory @@ -585,37 +607,39 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_align * Try really hard to succeed the allocation but fail * eventually. */ -static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) +static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large(size, flags); + return kmalloc_large_noprof(size, flags); index = kmalloc_index(size); - return kmalloc_trace( + return kmalloc_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, size); } - return __kmalloc(size, flags); + return __kmalloc_noprof(size, flags); } +#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) -static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) +static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large_node(size, flags, node); + return kmalloc_large_node_noprof(size, flags, node); index = kmalloc_index(size); - return kmalloc_node_trace( + return kmalloc_node_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node(size, flags, node); + return __kmalloc_node_noprof(size, flags, node); } +#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) /** * kmalloc_array - allocate memory for an array. @@ -623,16 +647,17 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc(bytes, flags); - return __kmalloc(bytes, flags); + return kmalloc_noprof(bytes, flags); + return kmalloc_noprof(bytes, flags); } +#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) /** * krealloc_array - reallocate memory for an array. @@ -641,18 +666,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_ * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) */ -static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, - size_t new_n, - size_t new_size, - gfp_t flags) +static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p, + size_t new_n, + size_t new_size, + gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(new_n, new_size, &bytes))) return NULL; - return krealloc(p, bytes, flags); + return krealloc_noprof(p, bytes, flags); } +#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__)) /** * kcalloc - allocate memory for an array. The memory is set to zero. @@ -660,16 +686,12 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flags) -{ - return kmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kcalloc(_n, _size, _flags) kmalloc_array(_n, _size, (_flags) | __GFP_ZERO) -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, unsigned long caller) __alloc_size(1); -#define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ - _RET_IP_) +#define kmalloc_node_track_caller(...) \ + alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -679,11 +701,9 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, * allocator where we care about the real place the memory allocation * request comes from. */ -#define kmalloc_track_caller(size, flags) \ - __kmalloc_node_track_caller(size, flags, \ - NUMA_NO_NODE, _RET_IP_) +#define kmalloc_track_caller(...) kmalloc_node_track_caller(__VA_ARGS__, NUMA_NO_NODE) -static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags, +static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) { size_t bytes; @@ -691,75 +711,52 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc_node(bytes, flags, node); - return __kmalloc_node(bytes, flags, node); + return kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(bytes, flags, node); } +#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) -static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node) -{ - return kmalloc_array_node(n, size, flags | __GFP_ZERO, node); -} +#define kcalloc_node(_n, _size, _flags, _node) \ + kmalloc_array_node(_n, _size, (_flags) | __GFP_ZERO, _node) /* * Shortcuts */ -static inline void *kmem_cache_zalloc(struct kmem_cache *k, gfp_t flags) -{ - return kmem_cache_alloc(k, flags | __GFP_ZERO); -} +#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO) /** * kzalloc - allocate memory. The memory is set to zero. * @size: how many bytes of memory are required. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1) void *kzalloc(size_t size, gfp_t flags) +static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) { - return kmalloc(size, flags | __GFP_ZERO); + return kmalloc_noprof(size, flags | __GFP_ZERO); } +#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) +#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -/** - * kzalloc_node - allocate zeroed memory from a particular memory node. - * @size: how many bytes of memory are required. - * @flags: the type of memory to allocate (see kmalloc). - * @node: memory node from which to allocate - */ -static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int node) -{ - return kmalloc_node(size, flags | __GFP_ZERO, node); -} +extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) -extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1); -static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags) -{ - return kvmalloc_node(size, flags, NUMA_NO_NODE); -} -static inline __alloc_size(1) void *kvzalloc_node(size_t size, gfp_t flags, int node) -{ - return kvmalloc_node(size, flags | __GFP_ZERO, node); -} -static inline __alloc_size(1) void *kvzalloc(size_t size, gfp_t flags) -{ - return kvmalloc(size, flags | __GFP_ZERO); -} +#define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) +#define kvzalloc(_size, _flags) kvmalloc(_size, _flags|__GFP_ZERO) -static inline __alloc_size(1, 2) void *kvmalloc_array(size_t n, size_t size, gfp_t flags) -{ - size_t bytes; - - if (unlikely(check_mul_overflow(n, size, &bytes))) - return NULL; +#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) - return kvmalloc(bytes, flags); -} +#define kvmalloc_array(_n, _size, _flags) \ +({ \ + size_t _bytes; \ + \ + !check_mul_overflow(_n, _size, &_bytes) ? kvmalloc(_bytes, _flags) : NULL; \ +}) -static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t flags) -{ - return kvmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kvcalloc(_n, _size, _flags) kvmalloc_array(_n, _size, _flags|__GFP_ZERO) -extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +extern void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) __realloc_size(3); +#define kvrealloc(...) alloc_hooks(kvrealloc_noprof(__VA_ARGS__)) + extern void kvfree(const void *addr); DEFINE_FREE(kvfree, void *, if (_T) kvfree(_T)) diff --git a/include/linux/string.h b/include/linux/string.h index ab148d8dbfc1..14e4fb4340f4 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -214,7 +214,9 @@ extern void kfree_const(const void *x); extern char *kstrdup(const char *s, gfp_t gfp) __malloc; extern const char *kstrdup_const(const char *s, gfp_t gfp); extern char *kstrndup(const char *s, size_t len, gfp_t gfp); -extern void *kmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +extern void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) + extern void *kvmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); extern char *kmemdup_nul(const char *s, size_t len, gfp_t gfp); diff --git a/mm/slab_common.c b/mm/slab_common.c index 83fec2dd2e2d..21b0b9e9cd9e 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1234,7 +1234,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; } - ret = kmalloc_track_caller(new_size, flags); + ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); @@ -1258,7 +1258,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) * * Return: pointer to the allocated memory or %NULL in case of error */ -void *krealloc(const void *p, size_t new_size, gfp_t flags) +void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) { void *ret; @@ -1273,7 +1273,7 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags) return ret; } -EXPORT_SYMBOL(krealloc); +EXPORT_SYMBOL(krealloc_noprof); /** * kfree_sensitive - Clear sensitive information in memory before freeing diff --git a/mm/slub.c b/mm/slub.c index f4d5794c1e86..9ea03d6e9c9d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3871,7 +3871,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list return object; } -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +void *kmem_cache_alloc_noprof(struct kmem_cache *s, gfp_t gfpflags) { void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_, s->object_size); @@ -3880,9 +3880,9 @@ void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc); +EXPORT_SYMBOL(kmem_cache_alloc_noprof); -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) { void *ret = slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, _RET_IP_, @@ -3892,10 +3892,10 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_lru); +EXPORT_SYMBOL(kmem_cache_alloc_lru_noprof); /** - * kmem_cache_alloc_node - Allocate an object on the specified node + * kmem_cache_alloc_node_noprof - Allocate an object on the specified node * @s: The cache to allocate from. * @gfpflags: See kmalloc(). * @node: node number of the target node. @@ -3907,7 +3907,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_lru); * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); @@ -3915,7 +3915,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_node); +EXPORT_SYMBOL(kmem_cache_alloc_node_noprof); /* * To avoid unnecessary overhead, we pass through large allocation requests @@ -3932,7 +3932,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) flags = kmalloc_fix_flags(flags); flags |= __GFP_COMP; - folio = (struct folio *)alloc_pages_node(node, flags, order); + folio = (struct folio *)alloc_pages_node_noprof(node, flags, order); if (folio) { ptr = folio_address(folio); lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, @@ -3947,7 +3947,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) return ptr; } -void *kmalloc_large(size_t size, gfp_t flags) +void *kmalloc_large_noprof(size_t size, gfp_t flags) { void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); @@ -3955,9 +3955,9 @@ void *kmalloc_large(size_t size, gfp_t flags) flags, NUMA_NO_NODE); return ret; } -EXPORT_SYMBOL(kmalloc_large); +EXPORT_SYMBOL(kmalloc_large_noprof); -void *kmalloc_large_node(size_t size, gfp_t flags, int node) +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) { void *ret = __kmalloc_large_node(size, flags, node); @@ -3965,7 +3965,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) flags, node); return ret; } -EXPORT_SYMBOL(kmalloc_large_node); +EXPORT_SYMBOL(kmalloc_large_node_noprof); static __always_inline void *__do_kmalloc_node(size_t size, gfp_t flags, int node, @@ -3992,26 +3992,26 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, return ret; } -void *__kmalloc_node(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) { return __do_kmalloc_node(size, flags, node, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc_node); +EXPORT_SYMBOL(__kmalloc_node_noprof); -void *__kmalloc(size_t size, gfp_t flags) +void *__kmalloc_noprof(size_t size, gfp_t flags) { return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc); +EXPORT_SYMBOL(__kmalloc_noprof); -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, - int node, unsigned long caller) +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, + int node, unsigned long caller) { return __do_kmalloc_node(size, flags, node, caller); } -EXPORT_SYMBOL(__kmalloc_node_track_caller); +EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); -void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) { void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_, size); @@ -4021,9 +4021,9 @@ void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_trace); +EXPORT_SYMBOL(kmalloc_trace_noprof); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size); @@ -4033,7 +4033,7 @@ void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_node_trace); +EXPORT_SYMBOL(kmalloc_node_trace_noprof); static noinline void free_to_partial_list( struct kmem_cache *s, struct slab *slab, @@ -4304,6 +4304,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, unsigned long addr) { memcg_slab_free_hook(s, slab, &object, 1); + alloc_tagging_slab_free_hook(s, slab, &object, 1); if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) do_slab_free(s, slab, object, object, 1, addr); @@ -4314,6 +4315,7 @@ void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); + alloc_tagging_slab_free_hook(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. @@ -4640,8 +4642,8 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, #endif /* CONFIG_SLUB_TINY */ /* Note that interrupts must be enabled when calling this function. */ -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, - void **p) +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, + void **p) { int i; struct obj_cgroup *objcg = NULL; @@ -4669,7 +4671,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, return i; } -EXPORT_SYMBOL(kmem_cache_alloc_bulk); +EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof); /* diff --git a/mm/util.c b/mm/util.c index 5a6a9802583b..291f7945190f 100644 --- a/mm/util.c +++ b/mm/util.c @@ -115,7 +115,7 @@ char *kstrndup(const char *s, size_t max, gfp_t gfp) EXPORT_SYMBOL(kstrndup); /** - * kmemdup - duplicate region of memory + * kmemdup_noprof - duplicate region of memory * * @src: memory region to duplicate * @len: memory region length @@ -124,16 +124,16 @@ EXPORT_SYMBOL(kstrndup); * Return: newly allocated copy of @src or %NULL in case of error, * result is physically contiguous. Use kfree() to free. */ -void *kmemdup(const void *src, size_t len, gfp_t gfp) +void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) { void *p; - p = kmalloc_track_caller(len, gfp); + p = kmalloc_node_track_caller_noprof(len, gfp, NUMA_NO_NODE, _RET_IP_); if (p) memcpy(p, src, len); return p; } -EXPORT_SYMBOL(kmemdup); +EXPORT_SYMBOL(kmemdup_noprof); /** * kvmemdup - duplicate region of memory @@ -577,7 +577,7 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, EXPORT_SYMBOL(vm_mmap); /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * kvmalloc_node_noprof - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. @@ -592,7 +592,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node(size_t size, gfp_t flags, int node) +void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -614,7 +614,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node(size, kmalloc_flags, node); + ret = kmalloc_node_noprof(size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -643,7 +643,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node); +EXPORT_SYMBOL(kvmalloc_node_noprof); /** * kvfree() - Free memory. @@ -682,7 +682,7 @@ void kvfree_sensitive(const void *addr, size_t len) } EXPORT_SYMBOL(kvfree_sensitive); -void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) { void *newp; @@ -695,7 +695,7 @@ void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) kvfree(p); return newp; } -EXPORT_SYMBOL(kvrealloc); +EXPORT_SYMBOL(kvrealloc_noprof); /** * __vmalloc_array - allocate memory for a virtually contiguous array. From patchwork Mon Feb 12 21:39:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47364C4829E for ; Mon, 12 Feb 2024 21:40:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2DB66B00A9; Mon, 12 Feb 2024 16:40:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DDDD86B00AA; Mon, 12 Feb 2024 16:40:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE2B46B00AB; Mon, 12 Feb 2024 16:40:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A92636B00A9 for ; Mon, 12 Feb 2024 16:40:25 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8104F160231 for ; Mon, 12 Feb 2024 21:40:25 +0000 (UTC) X-FDA: 81784470810.28.D9424F2 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf26.hostedemail.com (Postfix) with ESMTP id B54EF14000A for ; Mon, 12 Feb 2024 21:40:23 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DLcR7zfT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3RpDKZQYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3RpDKZQYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QVjGOqS2/C9DQQqaQQFc7iw3YwNnxSDsGz2z4KoUl2k=; b=J5017fs7DcofTyaLiDFDoslajdNg/5uoW6IMLk2rQisCJ1d9s/nkrwPyz0z6knSYeMfouu YqTOZCIngm1H/lX/zFjAn7+Z52Znh0dFH8jV9nYel8QHiiN5fUklWcM/LVZujzaV/SIURH +jdYhpeoRsa7s5pAQkixnB9KghF9AZo= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DLcR7zfT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3RpDKZQYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3RpDKZQYKCMU352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774023; a=rsa-sha256; cv=none; b=sG6iTZsZPsarYx9JS1sOn/IkFEm65Pst8y2TzpauxCzxqwkSrLDhiT1xeg/HJs8nMWp5FG kb1+cuh+D6XRc0TRaBMK49loZi0qX8TgMZ4mU/KKpyr5/Se+DOmRnp2eyf0cou2nZyzWdb T6L7y5xJjn7C13v4lAE3wl57jaL2I+k= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcc73148612so66135276.1 for ; Mon, 12 Feb 2024 13:40:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774023; x=1708378823; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QVjGOqS2/C9DQQqaQQFc7iw3YwNnxSDsGz2z4KoUl2k=; b=DLcR7zfTGFRoEl5+SLZrz0+tdQJkj7727qj6IorlDtWOxlW0Vvl8hBCMketjMRO79v XCNoQWRkzHhOhmZnd8ny7QicRXDZD/7WuWtLtHSSQrtjiwnIvhOCd3LehwenRhOjyjGI eqcbzF2hyz+Js3tIFsEAdOngxe0WKPf0HDUfofdPzRHC2OJPQJ2yGcqF4jj8mtxsEsyO UjHWS959GrHmYfoxITsjmzO6ibgAcCidgMsS1xYkctCa/Lp3tWMp64oaKCXUXeM1OWTy W5/eile7h81Go7KEfNWiUpjI8qRzVXqesvKxX65shnO353eih+kilY7ACrS4kZLayv1q y2Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774023; x=1708378823; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QVjGOqS2/C9DQQqaQQFc7iw3YwNnxSDsGz2z4KoUl2k=; b=P9UqOEFNxkx9rVsppazhea6DtOxc7b9FyfnkSg5N+qebtMZqzFPOb115F+8OhrzITq cn3Ff19T2EROXXaBQNYwkgQ2rrwhoJSmcrUNW+twGTGfYWajigw3e2QQSVLzFqd/bPVw MmjLFMmLdRLsxnfmMtxsyXzkzBT7OPcnyhZFAmyYClCnfSLSDpA5MbyveQB9WSG/m6Sh oVU9U6quRE+TP5Mzym1z3DEmSHWSg3/4PqP3Xl3kXJVHteILKGbxvUM9MnikBQNruytU sFbySKHDBAVwTNJWm3nUtDjH0yIVWp3gdWXV9wIqLpPZ7vqD3w/COzQJhS0rUyazx6PA Emvw== X-Forwarded-Encrypted: i=1; AJvYcCWUYH8vANeZtA4jBJcK5h1YyF5wW1aoXwHQ8NRXy8w6lLlxxKKpxrkDU6n7nEbqvD1AEcmbEH19dLnFeRtvIFbBmMA= X-Gm-Message-State: AOJu0Yy9KPvgwlhdXZKpX4tsEI7YEWSo5zPybP3ZOzws+aVD1FG/aIum DoZIvuO8EiMFgjR3Zbzy/iXrMWQbpcb8JVV49PM679bbaCSsV2z4P4wPmmvQhim70FWEST5Yn1O 1ng== X-Google-Smtp-Source: AGHT+IHcBrIgMIQ03gLdwIObn4fTPAITy7Rn+dgjwGrjWr5hIgaJ7mUN+6YCHWnvHC5rR1WgctLJs84VL90= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:709:b0:dbd:73bd:e55a with SMTP id k9-20020a056902070900b00dbd73bde55amr364194ybt.4.1707774022635; Mon, 12 Feb 2024 13:40:22 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:09 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-24-surenb@google.com> Subject: [PATCH v3 23/35] mm/slub: Mark slab_free_freelist_hook() __always_inline From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B54EF14000A X-Stat-Signature: sim86ad6e6kew4a5a59o8mrs9tsdj1e7 X-Rspam-User: X-HE-Tag: 1707774023-572641 X-HE-Meta: U2FsdGVkX18ZnlfO8F4Ocs3EelIBcK7SDT0rMT3N/RC2IbCPE+KnJ3i19vRDytf3lcPZXjSrRPCOBBYMvW+1iM3K32RJo/Uh8ccx9Cy/4l21Jdddqc3Z5cgy18XDHxxF4c1QwinZdfP8JqA7HcYnr3ov+AfFYHYQq9PqITdpR2giEODcxWE++VEO3/1wHf1/+0hpQEt2/uxnkQR1h00oQ31LEi0YByBRQx4yshWul39jvKJwhb7Dg1WO6E/egHpqfJsHxgL/fJ3y447+LxG8kfIgJ1gKMAecalh8fIU14HdjrOJZUPXlkHjDu/ptMc1aVihcSvxtqo619AR5xVSWYmLckgVwn/N8ZL1YSSPLmK7vcnEzA69ziyQIu1GYRcoim4ObwCwAGrPcxAQF+cnlOfxur1c/PY8NJzC+NiXUtdFjgn8k9grOAqUaJCNNPdLAhyZ6S/tsR1YdgIS1BTtREv7LK2LYc/ZyEe3Ev3C0m0ZadBR/09r2e+PAlp7s0V7oQzCpV5nGCr6PqNPfJCAlvvTd0kMnrour/LRUn2s4MTOh8UP2BRULmtZM8qaT24IWIdAZ1cSjXhG/cCiP4HtEtRfbD4cvSy8q1J2cp8xvFxqms/pj9OG/rJHICF+l9uEn+xJ6VztoCQrwTZFMQe7V9nIVqCzNdOA5l4cqwU4g7H7yNSU5yDM4WlisM0ctydIzrMrohVDWYd+tbcp93WjJ1ZAs7xsgpZEV5phPI7WioU+2a+IyCMtmNfXWIY5V2ZSHN2VeQunUq0ZIMZa1LluHDmgnIvyppK6HfY70pAdqsVG8l+oI2wDn/P7U7WUYBIkS2lY6StFB+PX3S0iwmWcil7LM6uALIwxY2M7xIeTWDfJ1nHpioVSS2zbTtAZ/49tgFkQmHRS1z57M6tWd2fuXWU15ujeFvtGR8w5Ay0O02VRJWnwgCSggF2HYcj+J1vLLB8a1bad7J05M/WICUgR ejNF/jxv nTQpvowAwCc5lFeGDjem0pqPspoDRFJAVhLAx78paxUVOWe7GCNj2nT5YUfMEhdziFejAyk3wlVnv8qhw2kdBfsd4sxgSNrmWRT/baf7reJDfe/dP9t2QO1dPatz9o+zYsTJLzOYYm6yKulWVHPrBfGJvCKhC+xeYHTNQBS8IZhRTELJmONTMFDUiYVs+AYHrilTrdrMOCLA+K4V4q5gnSlfnprT3zZr1Y6f2wS5t3JPrReFUx9QzPxZ24O/sas8F4m7yIXqzqFxVHvab5vYLuLevH031ztdTQdBKhW0jgoW63nt5ettaIM0Nk/xj+w7v+LDpCRP9NrUIbQaACVi8fv6uGOkUwFNKI49OFEJDb/+VSO+NaGbx65Tdk5a/56VwDDtx3kxid4+zuiw/LAIreZJMowX/pFVdvqj6BsySgn5y652MMdeXLNkrJA1+9CdnigUIWJKNZLuh8EX7LGDUDlSv3u+YWOtX2Rxcv3ktiAizg4aul24+TuJKCCCc+NG9tnqVa5iMkHr75rpgY9RchbudtDngzbgaA6aPvckx7KKtNp6TsojAHNrD4TvN/1GgRvpUWDhx9Mc0bCR7lCN7U4vLr4KL4kcsMXpi8QShEJOe5fagTMYCsUq9GWXjjzX96ZtX+MrRBfm+HnXsMX6sxY0VEX15Dxtb/F8glVo0uonBK3AYuJhnMHe0PTQW8XZEGe3pAme83IMpBgA0fL1A0s5pAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet It seems we need to be more forceful with the compiler on this one. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 9ea03d6e9c9d..4d480784942e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2124,7 +2124,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init) return !kasan_slab_free(s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail, int *cnt) { From patchwork Mon Feb 12 21:39:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1163CC4829D for ; Mon, 12 Feb 2024 21:40:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B74446B00AB; Mon, 12 Feb 2024 16:40:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AF9A06B00AC; Mon, 12 Feb 2024 16:40:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 949F96B00AD; Mon, 12 Feb 2024 16:40:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 81D416B00AB for ; Mon, 12 Feb 2024 16:40:27 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5E9D81C07CE for ; Mon, 12 Feb 2024 21:40:27 +0000 (UTC) X-FDA: 81784470894.15.12782D4 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf11.hostedemail.com (Postfix) with ESMTP id C6B7740014 for ; Mon, 12 Feb 2024 21:40:25 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=biKWZuhT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3SJDKZQYKCMc574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3SJDKZQYKCMc574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nNnBkPYeQ1kRTTlryOW31uuwa/BW+/OIiMnW19R7vGw=; b=JupyiXRL5pgdjJGokRyT/2yNPjyuG9PYQyaWgBK8LaT0KhMN2SwcZLR/se2sMdwHLWd/0q V/6wMMgCVVeyVM2aj0NrOcsfPnWCMMAZKzsyRzk2+4CNooGf8Iv7SYr2HL6MjLrIHtTx/p 4yl67jK0tQ4VkhZY2pcNQAGuIm3ASX4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=biKWZuhT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3SJDKZQYKCMc574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3SJDKZQYKCMc574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774025; a=rsa-sha256; cv=none; b=rkZZtKafVS/9nFDbMvoHkcYv58nlcQYilzpLFjP3ImDJEkYUzwWdEHc4jzhBdGWOoscxf0 WUtlRXnZvfIlb1WWv45Z5AZMwr7+hgmdlQBclz0amvRaT8BxuAza8WyFL+4ZqbG9vEzzLU cjvGf6YcZ0WyNKWlBmdABa+LIWzOT7Q= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbf216080f5so5719292276.1 for ; Mon, 12 Feb 2024 13:40:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774025; x=1708378825; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nNnBkPYeQ1kRTTlryOW31uuwa/BW+/OIiMnW19R7vGw=; b=biKWZuhT4MyfSkSM2PhY/wefTAE+V9smYo+6hI8gt71RO79Ae7DR8krgpswXhCQrCd j/5/xhJtknzAr4Qa0DzhyRFRwBJ04YJluypyW/FHEpFQhy6HHJERtyQZkH8zqksJspwt bUy87vNUrjvrDYQD/Kf5qZYeV/ofvSCABBkjrskx/dZIDB2frZygJOmVqRFZZxZoQytQ oP6Jb5of2+0azzhbtJYnaGzopz+ysy1Lh5oyPjEBXyDSkF61+LP//wxDttjvlrB3kssq qsBgz73WW9xVs8W0gNmwrJjAl5Kgsz6VEcWvpEXZOEIK87twyXurHjnjRBvmHfofTV4l D/Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774025; x=1708378825; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nNnBkPYeQ1kRTTlryOW31uuwa/BW+/OIiMnW19R7vGw=; b=a283wdQTT9PagLeelUrWZJnvzIpMmCWWcCR6C9zK1ktNEGyez+9ce39/4h7T6NJ4tK WVgLaunUArnDRZHTeTm4XV0ff5arH0ZkBzGObzyLKQToCudMwKy2MVvp8818cKtd2lEH MbZErxr1czdof+3UPoGW16Di40gDnx+5/ihFsX/NMM6um07BDJzJkViIeu7fm4rzyleI QJWM90MthR5rppDse1Mq6fr9DUtfAtQV4lcLh/usRrcbUBmN9TxNfzR4jxXy54P1GUsy efHavTWGwbWcaZKWzQjiGJi4N97iPNURu8jyATKS30RwpHlzMp0QS9WwcTbjHJjfmeKz Vsiw== X-Gm-Message-State: AOJu0YxBAguW4l+nRiZVABTaSuXlqz7BlRqA+1g3IZnOvIKzevZp7J2l sgC3XVcTA/EO1g7Z6kDWSV962j4VHnl2JvXJIurcqgHCdMDOkijnPXdhaPNEdnfIYIOr5hVi99q abQ== X-Google-Smtp-Source: AGHT+IEpiuRr80eSOkkX3iShmGwFZB235v7m7vIUsPrbIrvmqpTi8MXkiRRWkpGv6G+qILNGIJvYkoh+KuQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:154b:b0:dc7:5aad:8965 with SMTP id r11-20020a056902154b00b00dc75aad8965mr2054600ybu.0.1707774024786; Mon, 12 Feb 2024 13:40:24 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:10 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-25-surenb@google.com> Subject: [PATCH v3 24/35] mempool: Hook up to memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: C6B7740014 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 598a95ohndn5w4ma83wamauz9ob1oene X-HE-Tag: 1707774025-66956 X-HE-Meta: U2FsdGVkX1+M4zF1a0p/CUv5GS8P4Vg8vWLHckdp61QeOw9+LdJUCDyeAoeF+OY3JKHIZEfwfBeIinIRR7lxAq/bOGzkgDeFmdg514m9yfKfSMl78lUmTELwUfuibO/U9GSx8gUker+Jw0RZVeQIVY3lPREySUG9tilVsiW/dO/b2RxMy/9Ze7Ohg9NzCsTG+ZV3E1gYZvfLfclSyyvFwbDI8jNzr2Zbek+B2EMw3ZMdB0qooRX+n5iIlCZJrJQAjOJP+2LC7WQN+jvDODEGwSTGaKXx1f2SrX/UFA0jTCysUcRKkhYWk0GD1eO9/no8eAp3HnEI/FMgw9VQjnhFQd+UoW4QUDwXhzW+bvM+djSBUKivWFWwp9d6Y6xidf9WoA68Tfger4xDgVoTOSx5rD4Dyaax61tzivYMJZ+IMWNiQtJdM/GKlihxufnp62T4XNZajGnaU+0BkXazc+WETSc/lITT50qWZ+y6MDp1gSPNmYxGdBz3YfWkHY4DXw+QgzmJiBAPsf7OjG1hwkRZOJnXFacAwhbC6Q+GbA9sykAm4Ghkw1oZ6ZhgLHT1y+bb28PMVkOw/yvZc1CO+pjsm26f+t5AuwrstjdIBaRYEu/eE8exDO4QSdUfrBzVQAAdaxz9ceXhuWFQv3EQydX0gjRZ75uhjdlypYP3YNeKHt3n9aRJox7heS9+8aHbC2H8a51efFIuQZKX4E6+0lpyXgtbhjWNRS0W4wgasNTHdS5sXyJwZRjEQntQbtRKcszZP2Z3ieBryEhVPfgecTutYrLYNST+N+8bz3SSZdcGbGoZEnPogIxTrZrNxpxRmgFWzwm54zQ+LysqygG3bIhUJzOXQIUs76LQtFiPEcWHRC4+nNBAHSddT/6fTp1wXltkwrK5n0fDSAdtcVcSoPKLnH64+lq25NHI5WIE6dWZzIrjc7kvmepyUOMF3VvhRvsVGHMkDEmJ0vM3k9OBcJA T9fRF7nH rOSR1bTgnsOThBS0vcsXktAg2a0Pdwt9R+VhFvp0/j3hPXblDf/vrhUHHzTR502jS3gylqsUazxWz2pu5bVpuAKswj7f+97he5cWfpDwAUyVaZ4XLpywx5ypiKpeX0EDcqdW0/OluYDsVkCybE484DLURghYJot7+Ek/qRp3X+D8HjUuDVYDLmKW1TFVeCZjR1Pl91Kw8GydoO2O0GZhp46WdrrgSPFDedLuMmTCZs+269khjrcYflvyQzkVzTgjRtCtzJjOA16R8/mNEz+gW0wvUfIO2z8bUbCmdp7lUUgagl4eWNGdWL5t8MNzO8jtTHX35gKCWuPRcUr9FV+trn2KdNaG2Rd0IIb9wpevFPoe8Jq4yitegx1Iom17vwXd7Q++jeMgRQv004ugCQhgaJ/hAUHfMQQAIO5xg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This adds hooks to mempools for correctly annotating mempool-backed allocations at the correct source line, so they show up correctly in /sys/kernel/debug/allocations. Various inline functions are converted to wrappers so that we can invoke alloc_hooks() in fewer places. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/mempool.h | 73 ++++++++++++++++++++--------------------- mm/mempool.c | 36 ++++++++------------ 2 files changed, 49 insertions(+), 60 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 7be1e32e6d42..69e65ca515ee 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -5,6 +5,8 @@ #ifndef _LINUX_MEMPOOL_H #define _LINUX_MEMPOOL_H +#include +#include #include #include @@ -39,18 +41,32 @@ void mempool_exit(mempool_t *pool); int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id); -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); +#define mempool_init(...) \ + alloc_hooks(mempool_init_noprof(__VA_ARGS__)) extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); -extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, + +extern mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int nid); +#define mempool_create_node(...) \ + alloc_hooks(mempool_create_node_noprof(__VA_ARGS__)) + +#define mempool_create(_min_nr, _alloc_fn, _free_fn, _pool_data) \ + mempool_create_node(_min_nr, _alloc_fn, _free_fn, _pool_data, \ + GFP_KERNEL, NUMA_NO_NODE) extern int mempool_resize(mempool_t *pool, int new_min_nr); extern void mempool_destroy(mempool_t *pool); -extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; + +extern void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) __malloc; +#define mempool_alloc(...) \ + alloc_hooks(mempool_alloc_noprof(__VA_ARGS__)) + extern void *mempool_alloc_preallocated(mempool_t *pool) __malloc; extern void mempool_free(void *element, mempool_t *pool); @@ -62,19 +78,10 @@ extern void mempool_free(void *element, mempool_t *pool); void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); -static inline int -mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) -{ - return mempool_init(pool, min_nr, mempool_alloc_slab, - mempool_free_slab, (void *) kc); -} - -static inline mempool_t * -mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) -{ - return mempool_create(min_nr, mempool_alloc_slab, mempool_free_slab, - (void *) kc); -} +#define mempool_init_slab_pool(_pool, _min_nr, _kc) \ + mempool_init(_pool, (_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) +#define mempool_create_slab_pool(_min_nr, _kc) \ + mempool_create((_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) /* * a mempool_alloc_t and a mempool_free_t to kmalloc and kfree the @@ -83,17 +90,12 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); -static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) -{ - return mempool_init(pool, min_nr, mempool_kmalloc, - mempool_kfree, (void *) size); -} - -static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) -{ - return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, - (void *) size); -} +#define mempool_init_kmalloc_pool(_pool, _min_nr, _size) \ + mempool_init(_pool, (_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) +#define mempool_create_kmalloc_pool(_min_nr, _size) \ + mempool_create((_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) /* * A mempool_alloc_t and mempool_free_t for a simple page allocator that @@ -102,16 +104,11 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); -static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) -{ - return mempool_init(pool, min_nr, mempool_alloc_pages, - mempool_free_pages, (void *)(long)order); -} - -static inline mempool_t *mempool_create_page_pool(int min_nr, int order) -{ - return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, - (void *)(long)order); -} +#define mempool_init_page_pool(_pool, _min_nr, _order) \ + mempool_init(_pool, (_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) +#define mempool_create_page_pool(_min_nr, _order) \ + mempool_create((_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) #endif /* _LINUX_MEMPOOL_H */ diff --git a/mm/mempool.c b/mm/mempool.c index dbbf0e9fb424..c47ff883cf36 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -240,17 +240,17 @@ EXPORT_SYMBOL(mempool_init_node); * * Return: %0 on success, negative error code otherwise. */ -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) { return mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); } -EXPORT_SYMBOL(mempool_init); +EXPORT_SYMBOL(mempool_init_noprof); /** - * mempool_create - create a memory pool + * mempool_create_node - create a memory pool * @min_nr: the minimum number of elements guaranteed to be * allocated for this pool. * @alloc_fn: user-defined element-allocation function. @@ -265,17 +265,9 @@ EXPORT_SYMBOL(mempool_init); * * Return: pointer to the created memory pool object or %NULL on error. */ -mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) -{ - return mempool_create_node(min_nr, alloc_fn, free_fn, pool_data, - GFP_KERNEL, NUMA_NO_NODE); -} -EXPORT_SYMBOL(mempool_create); - -mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data, - gfp_t gfp_mask, int node_id) +mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) { mempool_t *pool; @@ -291,7 +283,7 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, return pool; } -EXPORT_SYMBOL(mempool_create_node); +EXPORT_SYMBOL(mempool_create_node_noprof); /** * mempool_resize - resize an existing memory pool @@ -374,7 +366,7 @@ int mempool_resize(mempool_t *pool, int new_min_nr) EXPORT_SYMBOL(mempool_resize); /** - * mempool_alloc - allocate an element from a specific memory pool + * mempool_alloc_noprof - allocate an element from a specific memory pool * @pool: pointer to the memory pool which was allocated via * mempool_create(). * @gfp_mask: the usual allocation bitmask. @@ -387,7 +379,7 @@ EXPORT_SYMBOL(mempool_resize); * * Return: pointer to the allocated element or %NULL on error. */ -void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) +void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) { void *element; unsigned long flags; @@ -454,7 +446,7 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) finish_wait(&pool->wait, &wait); goto repeat_alloc; } -EXPORT_SYMBOL(mempool_alloc); +EXPORT_SYMBOL(mempool_alloc_noprof); /** * mempool_alloc_preallocated - allocate an element from preallocated elements @@ -562,7 +554,7 @@ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data) { struct kmem_cache *mem = pool_data; VM_BUG_ON(mem->ctor); - return kmem_cache_alloc(mem, gfp_mask); + return kmem_cache_alloc_noprof(mem, gfp_mask); } EXPORT_SYMBOL(mempool_alloc_slab); @@ -580,7 +572,7 @@ EXPORT_SYMBOL(mempool_free_slab); void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data) { size_t size = (size_t)pool_data; - return kmalloc(size, gfp_mask); + return kmalloc_noprof(size, gfp_mask); } EXPORT_SYMBOL(mempool_kmalloc); @@ -597,7 +589,7 @@ EXPORT_SYMBOL(mempool_kfree); void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data) { int order = (int)(long)pool_data; - return alloc_pages(gfp_mask, order); + return alloc_pages_noprof(gfp_mask, order); } EXPORT_SYMBOL(mempool_alloc_pages); From patchwork Mon Feb 12 21:39:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4EE7C4829F for ; Mon, 12 Feb 2024 21:40:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 973786B00AC; Mon, 12 Feb 2024 16:40:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 923D66B00AD; Mon, 12 Feb 2024 16:40:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 776EC6B00AE; Mon, 12 Feb 2024 16:40:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 62D276B00AC for ; Mon, 12 Feb 2024 16:40:29 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 442621C07F8 for ; Mon, 12 Feb 2024 21:40:29 +0000 (UTC) X-FDA: 81784470978.05.D9B8C54 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf26.hostedemail.com (Postfix) with ESMTP id A88EF140007 for ; Mon, 12 Feb 2024 21:40:27 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=24Z8e9vd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3SpDKZQYKCMk796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3SpDKZQYKCMk796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774027; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Uyzb7E9FIhn/mbh3tSg2FbzBy8bNHcnwrWfMf2QZXg0=; b=8c6+L4NbbeHsXu6QbWB6P18HVYuX0M2CBgZBsCYKCjpvoq7tzRIqlNLwSoFZEH+bIAa446 BES4x0C4rVnIXI8ilYH3VXGH/Ef5W+Cf0WV+O0oEIaWTuklzDbETIH5qfVRmsLcVlG7oBS ZbgbiXzGpCpZf/LO1gjcxvcvypK9kmg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=24Z8e9vd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3SpDKZQYKCMk796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3SpDKZQYKCMk796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774027; a=rsa-sha256; cv=none; b=WePWzggj754eFIbyb5xmI/vcElGFmZi/KmKTVnOLKHTTz/6VQoXN5kRB1purHIVNenrY4X Q1zr7YjQZP4vTqW6flWF58nx0/+9FAUWVXG818Gp+LTjmP4u+hPXpc1dIgG4GqB9TsZDLn wo6KT6QaA1F+NAT2X5RrkIAr2RopDNU= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5eba564eb3fso75959537b3.1 for ; Mon, 12 Feb 2024 13:40:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774027; x=1708378827; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Uyzb7E9FIhn/mbh3tSg2FbzBy8bNHcnwrWfMf2QZXg0=; b=24Z8e9vdr5lkgeNMJ84y4TrtOcY5ORZkRi/PBFp0jUzFJYuX3g+pON3Cj91ESqeWO2 6yHtvU1VWU+IIULgcw/p0JEODT5kpvh7Yhd2f/B4sldChED9OE64NWG6ifRY5o2q6jew Z5XIVOEpFCGv7KRxaoaLhNy0iY0x+u43y2Mh6UNXNB14HwrbfoTtRhW2pjxP2jvThlKh TIZ5c7d1wNEzTQN405axe3vSQvXwvwYEQdBxXZQXdXiaTqmrkteSOxGffmjpsxDXCQdy t0fhyg2peUnhGg1xGV3dvKK6lpjp5HWJr19k8/CpR5bbAGvAY2rRUtiEimDnR8ieK23v dvsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774027; x=1708378827; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Uyzb7E9FIhn/mbh3tSg2FbzBy8bNHcnwrWfMf2QZXg0=; b=Im4zTG06V40a6dWFTOE5mH4CfuW/TJFyd22k63F7tUyeIYNd/Ggs9vcX03sT9hmlnT 6csgGjrwnJAvxu7mIyx4h2Ah4w7duBJ9FuKG75iuPVV090qr1yQrGf1vTEq6Zp+YnI79 S8oIqBdEDJ9Rjq6BrZGwNJC82igd/CaXKecK29frhYz27p28WoSgIJOsr8TMEDjcZqmK ig/C5pvmZWx7a0+IUiD8j3yAkpysX0HquxnHslGA282Wnb4KDMOV59MfV9egvlg0+rmX EndxKhgFvGos9jaH/f1Xo4/f9IttsIRKM4CVfXSx5R/1Fh0+6ZJzbIjoRjiKxoUN43YO ZNGQ== X-Forwarded-Encrypted: i=1; AJvYcCXq7H2vam8fzS575UvXcBOUvmmj3lC3lm01pCb3w0p8VY6MmPIm7jfyysZnRNlE+9qr30a/Cz8ol38EicDTLLcKOIM= X-Gm-Message-State: AOJu0YxZfxNGcpKyZfcXQwO7CCdEyjyxr9ArjYcXgJq5M4WiO/QCU8x8 mVPOzDjUa2Rn9WAtle4to4V6N9ciVJeP8KE6TwSca2ijuBM0lE378hqmXA9etx0qOsOng8Lkzw6 N0w== X-Google-Smtp-Source: AGHT+IE5NPp+ufCGF1WHGTLjC5ri/m4V06t1gtAUtmtnTKuKy22ZlehWWdmQuoopo1S/2T4p6wMzddrrcNE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:1505:b0:dc7:48ce:d17f with SMTP id q5-20020a056902150500b00dc748ced17fmr2107200ybu.10.1707774026593; Mon, 12 Feb 2024 13:40:26 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:11 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-26-surenb@google.com> Subject: [PATCH v3 25/35] xfs: Memory allocation profiling fixups From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: A88EF140007 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: cyy77s54dpokub1fdcst6zwutwrqffip X-HE-Tag: 1707774027-403286 X-HE-Meta: U2FsdGVkX184OMfPrgoB2SI7eKc7j/24j3Birg2nMGnSgWd7mvTqd5uend2moRZfRtryK7ilR3n+4W2nnd9m9PFOi8y3nEGFEj8IaRt+JoTP/4D+jpPBXSQbRpNTaYr4gf6cRNjRgbspQdAcL1sVql+/oXYiaDMyB16jM3KGj8TkqCUhxM0h97KNb6GRIXSLqfdDgv8sTrnlRKjt28tl1JcESTFM9d068bsVzHmqGPQBEuGH2orX9YjvBUZb2pAzvwVhWq9v6C0LI3oesyuZMYJMKRPPUdi1auDV3jGSa5+bRkZazyyGqDbVhV5VcdkNN0W/mjwe2I4721zGvZeFOOp546uiSBfO8IU+929TBxmZL7m6HJLTbdtBu5ONH16O9MksTTBn7YekqXnBe9H6AlGnyMoDAzrmuHUM5x2gUEs4W6dgAphZlOF2XQVulhUaD2ATapaDTxb4g6AbmI/OsbL+u0fCTgawjoDFTBgHbOT/TM6YWHCHdoE46wlCk1KcmP1MJ5oxC2TPiDoh47iiLTcW8C0PADP7/q3dnyK/b+ukVloKQGb6wKdfJ4urzS4gi25HKxOkXDvW61YOh/i4YbVj+hXOtGGp3oEnczkfs2L4qJAJ3phQDvDx3mQ4D59UFPRkVtu7BD8+f8fACr69ZYtEK7gHspYiubaiw+Hekm/keG5QiKt69md4Ro+GR9iPHT0NwL/X7FNXgk6rJIy2/t2rrJHYggTLj8ZjDLmd3rNhlZqMcG8Uvbm24zWQ4kvr9OkJ1O8LNiLvXw505sU3ljSccUFDl4nAM3546S+p7kqrXBP1wH1gldMGQ39iScrvY+SBOzlLXnf+w2JKPIQqInNnalcGD/peZ8380jkVbXyTsDGYDp3zvmCNkqg0F+UIHJedGb+e4N5W6fuSeli30IoTVtc37jJLpsT8/w7ozwxQHNJggd1jdCt9tn+8qe3pJHQc1jsRzRDodMi0rLC c1/KqXsZ QngwosrFDk2FTed3CKW8eW9OlSUqjpfvzaiPhwx4g2DXPU597TtQe6cPh7+NfHoQ2XkqzefUnBDM7ssR7PNmKi1Js56KOyg6/ZVtHt84XiLZN6aNEA37193eWfYiTrbycD52DjppomhE8Q4uwm0/uzxOvzxsZY8yv2m+pksDI+NgkKwVkRx36ucINkml/oob4uk741d5Mlxfo0NyfrIYWt1DHJcXsUrbOT6eMrQr/q8S/WUqyH2cbNqAGPGykh4wEC1qd6lO5SZzOMvgB3Z5BoGqKr7dvMjHmsalK18Fm9sAltv5NaegqJd/rgPKDknDvmfTPWISCFimQELhI3Rp7F7IeJMrVHb0Ja+iyyXJGAtsywTv3Mt0sR1QoKDZ3X6Nt02K3/OeXPqVTO/NWY8axAqrcQNOVSF37vvCWLhQf/2CZK80= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This adds an alloc_hooks() wrapper around kmem_alloc(), so that we can have allocations accounted to the proper callsite. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- fs/xfs/kmem.c | 4 ++-- fs/xfs/kmem.h | 10 ++++------ 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/fs/xfs/kmem.c b/fs/xfs/kmem.c index c557a030acfe..9aa57a4e2478 100644 --- a/fs/xfs/kmem.c +++ b/fs/xfs/kmem.c @@ -8,7 +8,7 @@ #include "xfs_trace.h" void * -kmem_alloc(size_t size, xfs_km_flags_t flags) +kmem_alloc_noprof(size_t size, xfs_km_flags_t flags) { int retries = 0; gfp_t lflags = kmem_flags_convert(flags); @@ -17,7 +17,7 @@ kmem_alloc(size_t size, xfs_km_flags_t flags) trace_kmem_alloc(size, flags, _RET_IP_); do { - ptr = kmalloc(size, lflags); + ptr = kmalloc_noprof(size, lflags); if (ptr || (flags & KM_MAYFAIL)) return ptr; if (!(++retries % 100)) diff --git a/fs/xfs/kmem.h b/fs/xfs/kmem.h index b987dc2c6851..c4cf1dc2a7af 100644 --- a/fs/xfs/kmem.h +++ b/fs/xfs/kmem.h @@ -6,6 +6,7 @@ #ifndef __XFS_SUPPORT_KMEM_H__ #define __XFS_SUPPORT_KMEM_H__ +#include #include #include #include @@ -56,18 +57,15 @@ kmem_flags_convert(xfs_km_flags_t flags) return lflags; } -extern void *kmem_alloc(size_t, xfs_km_flags_t); static inline void kmem_free(const void *ptr) { kvfree(ptr); } +extern void *kmem_alloc_noprof(size_t, xfs_km_flags_t); +#define kmem_alloc(...) alloc_hooks(kmem_alloc_noprof(__VA_ARGS__)) -static inline void * -kmem_zalloc(size_t size, xfs_km_flags_t flags) -{ - return kmem_alloc(size, flags | KM_ZERO); -} +#define kmem_zalloc(_size, _flags) kmem_alloc((_size), (_flags) | KM_ZERO) /* * Zone interfaces From patchwork Mon Feb 12 21:39:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554064 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 449AAC4829D for ; Mon, 12 Feb 2024 21:40:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C826E6B00AE; Mon, 12 Feb 2024 16:40:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C08636B00AF; Mon, 12 Feb 2024 16:40:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A34B66B00B0; Mon, 12 Feb 2024 16:40:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8FE046B00AE for ; Mon, 12 Feb 2024 16:40:31 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 51839806BD for ; Mon, 12 Feb 2024 21:40:31 +0000 (UTC) X-FDA: 81784471062.21.7BDB580 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id 7108F4000A for ; Mon, 12 Feb 2024 21:40:29 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SBXMwoHi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3TJDKZQYKCMs9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3TJDKZQYKCMs9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774029; a=rsa-sha256; cv=none; b=7U1L8+1auEUzvmmtYYYyN5f/hcxqWjMzxYx06RkLvxoac0ZGYhG8JwhKFEtShjnYTAqxtJ YtqMWdQ1lyUDM7+vzB6+v2pgo21Haa5S4t/2FJpQ57dYnQT1PDK9yTzKUGvlYnqfbTbR3D H1Am2FrSP36C+4NNFfylCyC0K78TaQM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SBXMwoHi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3TJDKZQYKCMs9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3TJDKZQYKCMs9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774029; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ATNLg2TqDpb+ESy76iaYpc/lOC/dJ7HcAs1g5FXdHcs=; b=sd+X9WgX8r/LMKAw35QfOg8PABMiBucwCuzZRhOiQnlsjSNNgmlfNtnz2MQP/wJNXX9MNG gHp+e5ABdx9C2ksGes9FeHD8roQdJJfdALzSIb5V8P/00qZBab3usMnjb0/WJm1VyoHo85 I1dFOmeajxnB12RMhPeb7dgQhhEfm0U= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-604ab15463aso4583397b3.3 for ; Mon, 12 Feb 2024 13:40:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774028; x=1708378828; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ATNLg2TqDpb+ESy76iaYpc/lOC/dJ7HcAs1g5FXdHcs=; b=SBXMwoHiM6OyZaUyQIjqbxBTEVw1hFMrhVB6PbUhzhZGSVPVZfGRoSqyr1P6C+pv+5 Wg6xxn4tWwkW1SaBi293xpl8wyatcqpkpNiMmKbvs0xb3jS1NxleQ0eZvuGNDgZt/UTJ ercNjQFbQqBQ7MQmr5QwXhegO8fukHTUh6luYKjmejbo8n/hKvqzfniVmUs2oUU9zuhT IZEhv/Uo0Gg3huGq6SGGVlWJXGJZsYus1dLvH4DvAxTRSr//q4/vL/gKnSlNwB4IUT6K Hl7jdKOIsXPi3SzbkcrWhNkSgPdyuBPxv4+z2pz/xQvchkTR3XOgpRsG2+/5W8fucmh3 O57A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774028; x=1708378828; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ATNLg2TqDpb+ESy76iaYpc/lOC/dJ7HcAs1g5FXdHcs=; b=EZq6HIIMpYdLv3YlP0OOvJ/eh+HX6kQ7IUQsMNUnlkj9QsxgBCXYRtgHAdV3l02sf6 hfuN9qvtt8v6BE5ommose5+9rHvKyjC6937a6Wm4mZ79DmYDl33qqfJpgGIJF/WzPDQ9 e4VMEop1EPgiyWs1+1Tn5g4g8bZUtszwQubucnsWUgpyjIqLyWdf0/AV8yU3c8beYc3H Flae9MwEMbIEzZ8Mr8shhYGVf+jRFHuKfNfk/scuFFVru5S8k0sKyWDFaJdkUbDa8dRK eGV/LBqkeJjdndCHaQF+6OlhddbhZ1R7p+y6x8QkGz6E5cGUQFEGYpLE3DfWOD4CPWll eudQ== X-Forwarded-Encrypted: i=1; AJvYcCVLOf+ASLuNO0PFr1q3m9eVswOSyQ9PEAAuxWbnp73R0t1qtCJ/t/tsYZC9Obq8B/CauTvIfA9SwJj4BaQI1SFfnMM= X-Gm-Message-State: AOJu0YxysEIjllO0P8AvX+hs5Pdw1kCnVXcp90Elf0TZFmqYAWe06Hmt U3acYDREatpS8Efq2uQFtYxR4YZ4Cs8rBw76Jkh1+XIl8yBhURblqSRsHN8rPNYedpoENmXmVMp Kyw== X-Google-Smtp-Source: AGHT+IFdaJ2ru42StbL1ZKAAjOQXI8b0LHOU1d1orFo3FRUTifWVI/lwUfEuIqH9ZjQMhzI7lKgTZH7BiIY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:690c:884:b0:604:d53e:4616 with SMTP id cd4-20020a05690c088400b00604d53e4616mr1398151ywb.6.1707774028573; Mon, 12 Feb 2024 13:40:28 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:12 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-27-surenb@google.com> Subject: [PATCH v3 26/35] mm: percpu: Introduce pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7108F4000A X-Stat-Signature: sakspewmotwst8qs9xomae8p6xht8jh5 X-HE-Tag: 1707774029-912802 X-HE-Meta: U2FsdGVkX19hy7/AXdsp6Lijp731aEsUdBAuRUiqgY+NKq8JRn2+/f7o1ZUNxOel7dGemEbLqsGMt7XkUm6IolhMjy9taOecyhCG28+mSu6EIzikPR9jHaqZmmrYWPWLAAW6F8H9KvIsSe/SLep9t69WxHA7WVvcw0Oa4W0FQmiuI7+en0GE3GTm33F9sBTs3sXqdYJ4QBvbWqL6OBDIsokG1L+ggkvnV1I3pdi08ok9TDs5xD9g5dn/eHWwbqwNzCLOQfk9fm6TSGaKwJP0asAQLlpNo3ovMDG8fMEPk3stNx4IMKG4kWtsTAXWjc2bhYL/KFMijYBaFs/vP4wP2d/hTBxPndhfefByBAT9+NV0Qyr1TXzKH7388Ox+EwAwiRMxf3hSSx0qygNkDSCEhcaGeO4ZJ9C77odG0+OhVilXyrs7qE6o530hxScYh9UQMKW2wUcbBO8Br8V1qbictDQkDM0kNKOXp3u5mtc9w0FffRUv0bRW4qx2Szz0EoCc5Igtv8JamXsaV3nEC4OGxxuvL7chifNAT8vWKKJ4WMv6hiN8hzMLs6xNnCU1eLdc4mR59zt7koAXyV5eViuxkSYCwONRMJ2bKNTu+xutsPc8w7neO+kMQHRlcYmfqLQYuVy+ebh5vKKGBvH6RkKJMZeX1zEKyr2stKzCxDjpjVxZHJRvl+mwz39LOJ4g++wcQAi4HLUaRiMmOJexQYhLWFtR1MOjnPRQqfEG1vomniH9KlrMHSyxt7PUILh7WVmpxtSnL7+vHGfIFQMSogIXmdxZK+MYBijZoEtfj47fZADKX8fbamjzzVwg0tUIzEAk+6t7lNfC7pIG+XyQuheBpZgCA240GlrB2q0aZrDr4xzkI1pnexOx1qSwGNYvSnkyCeIrECO74Tie8+gC7fEO5HceP3S1X0L77g1Lww3MZ9Mz0arVD4OHP64H12Ii9JZOTJgIS6hQH+MOSWFcpLU 5wMitQob ScwWRkEbN+e1e0KJCNRcZedVUOW+I39kNLsyNYRJ2++XS6e9lHsb7X893DRGrzhIPQKyghSAam7V19ZcuWBOqUVoYZXhrdm8UQMS0yJRFK2SbBMlM8V3WhrOTdcloA4ijEIBqNdI+98Mz65HrnCd6pt3IV7YK448W1I6DfNI8LU8rUXysG5RLyzyFPvLfKNxPBxEgZ6u3ihPxTb/J1byyTBYJZz+dkwTr5GBxQ7fLWhkdHTSluo4V+GDU9PGiIZ0bptnWP4l6qElDBB7c5Age51f7uk3ex168SU4pqREH8UhV4S+/MGkv/h1HKyMuPCle0KVbQJ7IkFkI4u7kkMKPmW7zplDBGXetxVRKKKCbJUiS90dQahv4GsjYSqNbGE59jcxfYuRn6YuWzwbBfwIi/RJXkToymSXjc+N/CHzlIq3OunfFcBBt+Pk6p0xalcnEBmqn6y9Ev5KxUrZtQMjt4+lzz4NyTFKeZIy+4y/EDob0PxmUvErJJDWp5jLEHQm1CQJEVm3YYnuPEjycCui/YX5lIM4ogTtAh5xi4mPGVPxIOAQLFlGDKkgOSuyv6fLtE6egb15jhO/Bg0VncNAM+heMgwSKWTMVriNo6c3fhOzi6r1KOood8LpZLTOITm+u18sUowIbPvpCthTTVxaZqgiLfQ6Q/pBSjN6oqrOZkbSQJIHCCdaRGOxKGEgbfAWJJCyIwF4eySCKQ0Qa65JiLgd6pOPTux/YwokvTQ8wjtisqfA9Jy2d8JeEb90xgEkYvQQEzFBMYH1wHEcVXXa455tAydn97GfPjQoEONHyvn3jeWFFLRs1PfIP5Op9YeF+vBFffxxwU26ThK+qMrHi6jVgErpER5crNrKH+DtZgOk8tybLzADzeRX0n+r7NnjPJ5e99obLc6bPbbUwoFhUFeoWfKbW9LO3AWM38mga9AlzBmaUqGc5SmY48DU9hSYhw7nG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Upcoming alloc tagging patches require a place to stash per-allocation metadata. We already do this when memcg is enabled, so this patch generalizes the obj_cgroup * vector in struct pcpu_chunk by creating a pcpu_obj_ext type, which we will be adding to in an upcoming patch - similarly to the previous slabobj_ext patch. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andrew Morton Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: linux-mm@kvack.org --- mm/percpu-internal.h | 19 +++++++++++++++++-- mm/percpu.c | 30 +++++++++++++++--------------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index cdd0aa597a81..e62d582f4bf3 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -32,6 +32,16 @@ struct pcpu_block_md { int nr_bits; /* total bits responsible for */ }; +struct pcpuobj_ext { +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *cgroup; +#endif +}; + +#ifdef CONFIG_MEMCG_KMEM +#define NEED_PCPUOBJ_EXT +#endif + struct pcpu_chunk { #ifdef CONFIG_PERCPU_STATS int nr_alloc; /* # of allocations */ @@ -64,8 +74,8 @@ struct pcpu_chunk { int end_offset; /* additional area required to have the region end page aligned */ -#ifdef CONFIG_MEMCG_KMEM - struct obj_cgroup **obj_cgroups; /* vector of object cgroups */ +#ifdef NEED_PCPUOBJ_EXT + struct pcpuobj_ext *obj_exts; /* vector of object cgroups */ #endif int nr_pages; /* # of pages served by this chunk */ @@ -74,6 +84,11 @@ struct pcpu_chunk { unsigned long populated[]; /* populated bitmap */ }; +static inline bool need_pcpuobj_ext(void) +{ + return !mem_cgroup_kmem_disabled(); +} + extern spinlock_t pcpu_lock; extern struct list_head *pcpu_chunk_lists; diff --git a/mm/percpu.c b/mm/percpu.c index 4e11fc1e6def..2e5edaad9cc3 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1392,9 +1392,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr, panic("%s: Failed to allocate %zu bytes\n", __func__, alloc_size); -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT /* first chunk is free to use */ - chunk->obj_cgroups = NULL; + chunk->obj_exts = NULL; #endif pcpu_init_md_blocks(chunk); @@ -1463,12 +1463,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) if (!chunk->md_blocks) goto md_blocks_fail; -#ifdef CONFIG_MEMCG_KMEM - if (!mem_cgroup_kmem_disabled()) { - chunk->obj_cgroups = +#ifdef NEED_PCPUOBJ_EXT + if (need_pcpuobj_ext()) { + chunk->obj_exts = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * - sizeof(struct obj_cgroup *), gfp); - if (!chunk->obj_cgroups) + sizeof(struct pcpuobj_ext), gfp); + if (!chunk->obj_exts) goto objcg_fail; } #endif @@ -1480,7 +1480,7 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) return chunk; -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT objcg_fail: pcpu_mem_free(chunk->md_blocks); #endif @@ -1498,8 +1498,8 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) { if (!chunk) return; -#ifdef CONFIG_MEMCG_KMEM - pcpu_mem_free(chunk->obj_cgroups); +#ifdef NEED_PCPUOBJ_EXT + pcpu_mem_free(chunk->obj_exts); #endif pcpu_mem_free(chunk->md_blocks); pcpu_mem_free(chunk->bound_map); @@ -1646,9 +1646,9 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, if (!objcg) return; - if (likely(chunk && chunk->obj_cgroups)) { + if (likely(chunk && chunk->obj_exts)) { obj_cgroup_get(objcg); - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = objcg; rcu_read_lock(); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, @@ -1663,13 +1663,13 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { struct obj_cgroup *objcg; - if (unlikely(!chunk->obj_cgroups)) + if (unlikely(!chunk->obj_exts)) return; - objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + objcg = chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup; if (!objcg) return; - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = NULL; obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); From patchwork Mon Feb 12 21:39:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E1DFC4829E for ; Mon, 12 Feb 2024 21:40:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BD566B00AF; Mon, 12 Feb 2024 16:40:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 244E66B00B0; Mon, 12 Feb 2024 16:40:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0716D6B00B2; Mon, 12 Feb 2024 16:40:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E5DF86B00AF for ; Mon, 12 Feb 2024 16:40:33 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9A9C1802C7 for ; Mon, 12 Feb 2024 21:40:33 +0000 (UTC) X-FDA: 81784471146.03.5A3C702 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf28.hostedemail.com (Postfix) with ESMTP id B980DC000F for ; Mon, 12 Feb 2024 21:40:31 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Rux7IPi6; spf=pass (imf28.hostedemail.com: domain of 3TpDKZQYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3TpDKZQYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774031; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X07z1RjJe29sLMh5N/vKSXU8uipRKYcH64glAbq+0r0=; b=5xtnMUdlgkzotomJvX17HBFK3kPN7tAP0cPHTqTplWc6p9Pg/y293PDZMnGYER4+nESuct ehrjAmiq13dPm4wnKyLenWQHZ0x7iHHka/tRGNk6VSm580kNBGijxThb0AVRSMO79a5IhN tuyB2DVRH/6TnFJfUb5LDXY3xxIyYvo= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Rux7IPi6; spf=pass (imf28.hostedemail.com: domain of 3TpDKZQYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3TpDKZQYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774031; a=rsa-sha256; cv=none; b=GLZdTaCj4jtxiu+YBTf0EX9EwiCvzOV8h3txrKYQcIeS3/O+U7yNV9QRH1h/7r+X3j5YvX w4ksc9vPhp+LcoHZ+5k44gePdy80517ZYrKsow/CgddvNUM4zQZmUxzZWSSiwCIxAdIhAU cc08eNL/0DRFu4OLKYC/p6IDb4REy/E= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc743cc50a6so306078276.2 for ; Mon, 12 Feb 2024 13:40:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774031; x=1708378831; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=X07z1RjJe29sLMh5N/vKSXU8uipRKYcH64glAbq+0r0=; b=Rux7IPi6jVnPBccGrYhBpFdq4sdl7YDFkCHwoKWMUKC0FyP1Up5MsxAR+dSa98L55x PjLYRU5OUrF7IlBt57A9n0SrvrOvKzy1CBl7sTdofzjGA1lBp8MyAa32CAG4djsUgAGw ObNEdXopAZCyiadTW7Ko2PyFLY0zzOdfkq4/laMVkc3T2cYJzpY8T9WZq0E89IAUegn9 MdwpGbiv3qbOSj7zQvmJBz4d2a+fijNIOr0vpbB6YODNkPltY9G/MP2zBLbuvEhRbWoc p09FlS37aWFcLE4oaE1xnGtjm+WW6j4Ine4K4gQhL0IL89ss4AKrnWYFwPklWpZrwyQq 2pdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774031; x=1708378831; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=X07z1RjJe29sLMh5N/vKSXU8uipRKYcH64glAbq+0r0=; b=CvwLtKcfo7Tnwld+LtPw11YAi15WknCsaTj8uUTYGmTGfSD37FurpAzcP2igzQgtf+ xzJOeEFwt8g/TpqJxDczynqixniikWVaCiqApd9TXG0sJjW8CC+Bqz2g76CNHmK9udVg Inr7UE2oe2Q8a4ZPvl+oCngdOTKrRbKge9tUEqQfWp+kCPn18WVEVRYlm74Mf+blEmpK c3x0VRo3dnIcn5ok1uGqvQw0b1yDT/sVnm8DQYuH7bmfmsNttdO1oCVWG/0GTY6nxlhA Re+0PvI1sc1wxn5d/eDtg+M4F6ZZTTaUFVzgVGe3GQiHQHiw2bkqe9fIS9Pm/G5/XWlr skcg== X-Forwarded-Encrypted: i=1; AJvYcCV7YeaCufxo6HdD17uLdJIfHzPr9O9EOmF+E5mtlW+4LfwzxJiyJVO6/KxO7eY0HXvg7itX326vhnAOKGYCga2M9fU= X-Gm-Message-State: AOJu0Yw6vvPsNOVKJwh3TQIx54tTd5buR4FjwOucn+OOC7ucA8JXha2A JtTqU4x0aatRsk9Y+lDCXNSE+qOmcWICj2xaxx9AtjU9byf9w2Triltsq94RKYZnbIxaEFZCoer SJQ== X-Google-Smtp-Source: AGHT+IE+9v5iTSehfGXvFLsyFgeYeTOkKmnm1/9Fi71HMokK15QIurDQiuu9x1J7ZwyZ5mBIdOmgWIrIEcg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:709:b0:dc7:59d9:7b46 with SMTP id k9-20020a056902070900b00dc759d97b46mr291099ybt.3.1707774030642; Mon, 12 Feb 2024 13:40:30 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:13 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-28-surenb@google.com> Subject: [PATCH v3 27/35] mm: percpu: Add codetag reference into pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: B980DC000F X-Rspam-User: X-Stat-Signature: fgj815setmzf9hte7rtjuobnosmupery X-Rspamd-Server: rspam01 X-HE-Tag: 1707774031-921104 X-HE-Meta: U2FsdGVkX1+07mIXc8V5ZK76ZNgbF7QnThDlo8d1AW02K33XbcfwNy2eTOkcEq6QPcNBZtH+ufHvtgUDPjG9biHJa1P+2Iloow9OfkdL4g3zFkhPNh5EaUyNErgNJcuqBkJrDBWB7lqK5sGO5URxBm8Z711ydIJ0hK/sK8IcGYkbIotz33awxDBfJdZYmvjlu5NUAaPJaKiImjN88bdPUWPod5OH+Z0efFpsj00tduEsQlkpE33o4sxczLcbjVqaDuLeQHNfyV9ixvBfYYVrCiqfgnWZAPV7ac0hnveJzXL6LOMWnzgtAoiLd1oRbO+WXnhVEoyAfkJibl8/p5fUzhmZlpd5y6W0EvMamVtBidSPutSJh189tjKZjOYql69NhOkkPdk8ptCaRJN4iQrJlFojdrPekn8Jyka4GlJI0eBvAeT46DMq4ZD7pdKsuGnLlfldgCp0DNv1n/ydModikxxAFx0QFCIeEDEH27zZZisMFXwYGMj1Q3vBuIYSnmD2JfupI6EjzUEWMWiA8RNTmmheM6/NvVpkMEXb4cM8vxMfwlSWSRLp9dn7xcG0UwNlAKAVhVD6+mGTBS3JJuAWRSnd58DsHoWU1KueKty/V6K/3zmmOAzGvYl5lWM7SHfQyBUmkIXtNtUjrrQriEGhYS0NxWruO2wmYpj3Z98clhsFahuoWEbNm4YHvVd/llAFuObLWgTNSPU0ADQBY5oi5pcnSAoqX2gvVUYizjGv9MMQeBXz7usVxEhKEcmF347sOXEIC4GXNK8HMaYZ7NdR/iIBs3aMCMdgJ0wOGPfmypbZvB4CNtfV3KlSmvThqrk6EWvuqw+Y8qlilxhzNLpYYe8IxN9wTFNIX2AThssss/j6RpD/qz3sX9iFQeTknav3gavhLBpXr9PI7VOmrZrQhvcrqgawGxFKiQyNx2ydCETJQ7sGgRpPNPoHEQ1hyKIdgkg4KmH21o/f6ARMpcx 0QRSPRcc rxNZ1BERLHruwrfXsmdAGo+lQGrhJTonXeHf19nKiYYnqXOrRoTa7PQ4PKnwJWnOdInHfVZyW8VCNcdgDGpbJn0+2PconVwey2J5VyAQS/aqnf10Xg3/OgRz4OjxTlIY9r+KQf3Tp6sOzeWn9zDTR1e+Cm5ovRYwB+cMFmw9BLi6orFXqlIAWYzgJU7EC37yaPQNQFAItxIlzGDFnhr9tU+P2Q1oTUT7rduk7OXscKf2Tgv595oZHpVaGuUhWutulW1Cr+vUgbpAM4bot1keGfTb4f2++SclpAIFJVzMa25pBfGAz2Oh9WamL7yM7aR7My0oyqmZZwMmHVz5FuiGE23GxTDS8TczkHc6umilSn9MrEjY7jvZ9P3Rz1J6b4Dc6JH2vxdkZvch2XtigEbWl9UXbfuSg4z85M2uLnZcHEHJf5H90co99iz6uDDD2s4gB42+LfE5rViTInYWOYXPECG83k0ZBJ7C5ALiAAJe4lumd5f3cj5fcCf1NS5kLIXImdxzlughfJMRHs9G/rLoieNnqg8KAAukZkoMNbVXywBF5HATKJWqZAXtP0sfIxSDKHCR3bcb9jVgrYDdrL74VyiJ8+rGA4gImRruxcr2UekduR/Rs+A07LqsUTPp/k4qOYRwtk/u9RNrIWG4FRqYPNDGbjNqgWiFxUM+6obBCnHjxoPAlM2giFQN6BtNr18P9V0AVnx1qDy4VkKq70WZu/L1WCw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet To store codetag for every per-cpu allocation, a codetag reference is embedded into pcpuobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Hooks to use the newly introduced codetag are added. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/percpu-internal.h | 11 +++++++++-- mm/percpu.c | 26 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index e62d582f4bf3..7e42f0ca3b7b 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -36,9 +36,12 @@ struct pcpuobj_ext { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cgroup; #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref tag; +#endif }; -#ifdef CONFIG_MEMCG_KMEM +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEM_ALLOC_PROFILING) #define NEED_PCPUOBJ_EXT #endif @@ -86,7 +89,11 @@ struct pcpu_chunk { static inline bool need_pcpuobj_ext(void) { - return !mem_cgroup_kmem_disabled(); + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + return true; + if (!mem_cgroup_kmem_disabled()) + return true; + return false; } extern spinlock_t pcpu_lock; diff --git a/mm/percpu.c b/mm/percpu.c index 2e5edaad9cc3..578531ea1f43 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1699,6 +1699,32 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) } #endif /* CONFIG_MEMCG_KMEM */ +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { + alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) + alloc_tag_sub_noalloc(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ +} +#endif + /** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes From patchwork Mon Feb 12 21:39:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C93EC4829F for ; Mon, 12 Feb 2024 21:41:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1602D6B00B0; Mon, 12 Feb 2024 16:40:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 072746B00B2; Mon, 12 Feb 2024 16:40:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDD4B6B00B3; Mon, 12 Feb 2024 16:40:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CAD0E6B00B0 for ; Mon, 12 Feb 2024 16:40:35 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A0140A04E7 for ; Mon, 12 Feb 2024 21:40:35 +0000 (UTC) X-FDA: 81784471230.19.0798FFF Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf01.hostedemail.com (Postfix) with ESMTP id ED8AA4000C for ; Mon, 12 Feb 2024 21:40:33 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=W9N0fCgf; spf=pass (imf01.hostedemail.com: domain of 3UJDKZQYKCM8DFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3UJDKZQYKCM8DFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z8tkEzFiQHW4+OeWkbnef4rwrjbu90Cr9IgJj7MTk7I=; b=YbXvqPOL7LAYVXPZt95BccLI9ZF0aHqQ9pw1d8cOhPIq80+9LvSKeZ9YnDo2lqqmPTuZd6 Pp0oAGW+FTsz2UtpDEp5nw9g5PNKz31ixGzJvJ15eAScj1A5IljhMQynS8A0H+azqlJeui mPym+twtiWwKlNv+hkd9iMarr9ReJ4c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774034; a=rsa-sha256; cv=none; b=utfSncRLv2idEHnzhtAphqpn5aMb7rxJdt9/yLt7KY2sv3gzKyjYNuObS6mrsa5jw572Ed benY2WB54Po0xaJsIbWpWWxZSp7t/NeKRs/gLCrdTiZVvmOKXKk1wiMLcn09ShgGpI/QNs IRs2NX5aPBXtfwWwYioVdkvYKBGKEBo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=W9N0fCgf; spf=pass (imf01.hostedemail.com: domain of 3UJDKZQYKCM8DFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3UJDKZQYKCM8DFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5fc6463b0edso62942457b3.0 for ; Mon, 12 Feb 2024 13:40:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774033; x=1708378833; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z8tkEzFiQHW4+OeWkbnef4rwrjbu90Cr9IgJj7MTk7I=; b=W9N0fCgfk4Dv7eb2tvtw3WR8+r7uzgCHAxM/lOlLfFxT9rZz+iH4Pqex3woYgcAKeq GzOCSfILQi9tF8lT+Yn4sjyJ4qBChKGHMPef18dcVOkbVmVHKQ0nxDw6qqKyMjQA+Kbi 7gN/TOIAnt+Cu5/n7kPPspfCrLqRViXqOmmvCpKY5WCqG8hGUVhDXk6U6xf5qsFWw8Dm YGlu9RPE07s7ZkvoZKUDSMof8ki0+kOmNdvDGvfj8vAYJ8qdgcHwKiwUNDwnXk64revu KtsQqqJaS19i52M7rycr1pGYRpabcPMxRZ7MlNK/PBHQZx/LPxDdW6CVCCsgvT/EpngM QPKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774033; x=1708378833; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z8tkEzFiQHW4+OeWkbnef4rwrjbu90Cr9IgJj7MTk7I=; b=bj5SGzCjVdMFutEnnsfIKmcUrPEkly0KDdwGk64nXsf9bKuzpTwARxhaqar70redfy 9KNzdYjT65UpnbK+XMz0CNlN45IwH2WWB/nM1P5tTe7M5BakJbOTCzVvCtiQfhPfD6Fh iijsyn3CmEkZs/Iw7jmyesBEazKbZmn4qpLRmVXQSLaLYqzuNUOtMh8VlV3AjfFQeE+w jSIkIx0EL8H3mJ2AzW+bOE0ctiIMs88Ld1PLyYgZAzMfSyM9lUHwZRKJWXBrJgagsRHo QZ+K/AyOQZrljQgbQqYfvVrFSxqpfQGeAGs385iT4kUx/gmIe0Cn6WOuAmFyklVlMPoE GKzQ== X-Forwarded-Encrypted: i=1; AJvYcCXo4/MPdjjL1kK5lb1AuHIMiqcWPqub/Kme5bdSogysN/5S7UcZ1LzIG9UzaX0vnFfE4gh/+pDJkKYMTIP146nNj+g= X-Gm-Message-State: AOJu0Yz3+AqHUL5Yjryv/Tp9g4+bxFiAqREU9jWpL8PCOZ12XNQHoVHB UNkAq3ahQi0bgOYvClmDg2Jfq2VFZPfg+UzM+m0xGg4oSo2r55/JKZskps1nxn80CXWTUTB5IZ6 APw== X-Google-Smtp-Source: AGHT+IHm6015U/EcZR8Woa1htECUFmOLxos5kLFwlm3ImstqT3JjQx2GwcIf2KWvuAyJVMYRK/0LWLgINsI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a81:a010:0:b0:607:7dee:a7fa with SMTP id x16-20020a81a010000000b006077deea7famr162033ywg.2.1707774032997; Mon, 12 Feb 2024 13:40:32 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:14 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-29-surenb@google.com> Subject: [PATCH v3 28/35] mm: percpu: enable per-cpu allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: ED8AA4000C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: en43uyj35qbkzk45k1jt91e8p7kr7w8b X-HE-Tag: 1707774033-769991 X-HE-Meta: U2FsdGVkX1/GDMuqVIhdXY0Tzr2GddhE1yXPTSl12UNPIHYTL7iVVbg0znXc6KUhvckNTfimXnGrTXahvROE9Re8o+r81ZT5FhoYsxTJl37AQP3W+ymTFX5L3keChmeJBdImAT4DqhMZaonmOiFXuLmycc9S9lmuB3m2kKTBhidVazL5QW5gsCwEv5rIF+PWbR/qZ+qcBQ4WnUVFRUqMTy94V6aVNsZ2Bn4z0bGN0APDKkT19nGefA7Btr15HAsW515KlnKC2FMXZT6FIqCpIW2ncCHMLWaVH1C8n6Ye9b6N0EiqvqAnT5K7ZQSoalse1nQenCvkR88YBCSOdyldXwInq967gGobLvQPwBy2EffpumdStyFl+A4/jBDGoCEOO8eJsBMxUXzPNmZgHjBjcaAaRv8v/zERqj5EAoiO4578LWMDRHNR182WS+Mvk8933tT61dL3TZcDXZEdUCMJgopk8gETNmUqjfuUNbXXg69EuBpQQpmrImSBZsbM4brtjTE7Key+buJtWFCUNZQY4FYhAVHLgp5N1ImOxV9Yl3P0rffTfbXFwi9OxUaz+TXF6RL3B9CSAbYeklAT+KLe9LOyk0hJ74yf4uRc4PyiJ7f2t+ZicUEPQCaIpO3B2UngwhP06zC8d71848l7/zAp9++geuTDfY0cR/nKDdfCepy1P1T2ZG6Yit5F1YKM4mBIraDorE4Gg8CZf8emzo29OK1cxYe8z+vPrdJBLlCEvUelT68742jMXzmS+TFeSBZJRP2ai1HgTdZ4AI4qvT0CLA5yjYh8ZEg/txM0Ws24uC6D6D+EE8p5hI59fv0vOLSGl9IX6CxIlk7yDrGxYgrph70Ge1vnJU2t0aydD6zrdMo2kcSAJwZmlnHoJF4TX6TPQo0BVnf0Fvx9/KIDcvmkrWtdvInsiBzd7oK1SZof/56VYWEVZ/e4MH1xOUKhyp3Lof9b5vU2bX7NqwFRrPL O6Y/c9nz TscQMPjmFuosXKzLmenKEiw25+b6is13hUllxdpfwqU6qtsXEO2YqBO4X/x+HysKS0KNT1RrE0fYLip/RApvABjhMDEb3PDdmfGNyi7kNqWmIRmdT16zh+ePXng7ChBohKyZ02azn2CeffqnFiYN4lV6EijFdohkR7xf+D/F4R1WpNgnWsnAcl0JIxWCvp0XkKJ53mvUypIPjZ0x+sn64tzpJY3vFUs6p8ahTt0v2YbplA/4WBzKD5MvEWQ7wsdGAMuYEyFNu6q5rmD0im7hH3eZm//OGv08IZ79VQT8hkq60UPC54hBnHMlJPRWyvBCEE19p1wo3UIV1Tet9sLmyNs2QMv9A5kfXBCNuvOYXnDg1WGsE20aQgnIk/pGMaXqwSf8IZJ9Xkb8ogXkDKwFZVzpiEGYZ3okFXIgFSHnNm/AFTqs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine __alloc_percpu, __alloc_percpu_gfp and __alloc_reserved_percpu to record allocations and deallocations done by these functions. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 15 +++++++++ include/linux/percpu.h | 23 +++++++++----- mm/percpu.c | 64 +++++---------------------------------- 3 files changed, 38 insertions(+), 64 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 6fa8a94d8bc1..3fe51e67e231 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -140,4 +140,19 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, _res; \ }) +/* + * workaround for a sparse bug: it complains about res_type_to_err() when + * typeof(_do_alloc) is a __percpu pointer, but gcc won't let us add a separate + * __percpu case to res_type_to_err(): + */ +#define alloc_hooks_pcpu(_do_alloc) \ +({ \ + typeof(_do_alloc) _res; \ + DEFINE_ALLOC_TAG(_alloc_tag, _old); \ + \ + _res = _do_alloc; \ + alloc_tag_restore(&_alloc_tag, _old); \ + _res; \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 62b5eb45bd89..eb4eb264136f 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -2,6 +2,7 @@ #ifndef __LINUX_PERCPU_H #define __LINUX_PERCPU_H +#include #include #include #include @@ -9,6 +10,7 @@ #include #include #include +#include #include @@ -125,7 +127,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); #endif -extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); extern bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr); extern bool is_kernel_percpu_address(unsigned long addr); @@ -133,14 +134,16 @@ extern bool is_kernel_percpu_address(unsigned long addr); extern void __init setup_per_cpu_areas(void); #endif -extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); -extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); -extern void free_percpu(void __percpu *__pdata); +extern void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, + gfp_t gfp) __alloc_size(1); extern size_t pcpu_alloc_size(void __percpu *__pdata); -DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) - -extern phys_addr_t per_cpu_ptr_to_phys(void *addr); +#define __alloc_percpu_gfp(_size, _align, _gfp) \ + alloc_hooks_pcpu(pcpu_alloc_noprof(_size, _align, false, _gfp)) +#define __alloc_percpu(_size, _align) \ + alloc_hooks_pcpu(pcpu_alloc_noprof(_size, _align, false, GFP_KERNEL)) +#define __alloc_reserved_percpu(_size, _align) \ + alloc_hooks_pcpu(pcpu_alloc_noprof(_size, _align, true, GFP_KERNEL)) #define alloc_percpu_gfp(type, gfp) \ (typeof(type) __percpu *)__alloc_percpu_gfp(sizeof(type), \ @@ -149,6 +152,12 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), \ __alignof__(type)) +extern void free_percpu(void __percpu *__pdata); + +DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) + +extern phys_addr_t per_cpu_ptr_to_phys(void *addr); + extern unsigned long pcpu_nr_pages(void); #endif /* __LINUX_PERCPU_H */ diff --git a/mm/percpu.c b/mm/percpu.c index 578531ea1f43..2badcc5e0e71 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1726,7 +1726,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s #endif /** - * pcpu_alloc - the percpu allocator + * pcpu_alloc_noprof - the percpu allocator * @size: size of area to allocate in bytes * @align: alignment of area (max PAGE_SIZE) * @reserved: allocate from the reserved chunk if available @@ -1740,7 +1740,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s * RETURNS: * Percpu pointer to the allocated area on success, NULL on failure. */ -static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, +void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, gfp_t gfp) { gfp_t pcpu_gfp; @@ -1907,6 +1907,8 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); + pcpu_alloc_tag_alloc_hook(chunk, off, size); + return ptr; fail_unlock: @@ -1935,61 +1937,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, return NULL; } - -/** - * __alloc_percpu_gfp - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * @gfp: allocation flags - * - * Allocate zero-filled percpu area of @size bytes aligned at @align. If - * @gfp doesn't contain %GFP_KERNEL, the allocation doesn't block and can - * be called from any context but is a lot more likely to fail. If @gfp - * has __GFP_NOWARN then no warning will be triggered on invalid or failed - * allocation requests. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) -{ - return pcpu_alloc(size, align, false, gfp); -} -EXPORT_SYMBOL_GPL(__alloc_percpu_gfp); - -/** - * __alloc_percpu - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Equivalent to __alloc_percpu_gfp(size, align, %GFP_KERNEL). - */ -void __percpu *__alloc_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, false, GFP_KERNEL); -} -EXPORT_SYMBOL_GPL(__alloc_percpu); - -/** - * __alloc_reserved_percpu - allocate reserved percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Allocate zero-filled percpu area of @size bytes aligned at @align - * from reserved percpu area if arch has set it up; otherwise, - * allocation is served from the same dynamic area. Might sleep. - * Might trigger writeouts. - * - * CONTEXT: - * Does GFP_KERNEL allocation. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_reserved_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, true, GFP_KERNEL); -} +EXPORT_SYMBOL_GPL(pcpu_alloc_noprof); /** * pcpu_balance_free - manage the amount of free chunks @@ -2328,6 +2276,8 @@ void free_percpu(void __percpu *ptr) spin_lock_irqsave(&pcpu_lock, flags); size = pcpu_free_area(chunk, off); + pcpu_alloc_tag_free_hook(chunk, off, size); + pcpu_memcg_free_hook(chunk, off, size); /* From patchwork Mon Feb 12 21:39:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3BC8C4829D for ; Mon, 12 Feb 2024 21:41:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A36446B00B3; Mon, 12 Feb 2024 16:40:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E5DA6B00B4; Mon, 12 Feb 2024 16:40:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8387A6B00B5; Mon, 12 Feb 2024 16:40:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6DB016B00B3 for ; Mon, 12 Feb 2024 16:40:38 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 175001A0575 for ; Mon, 12 Feb 2024 21:40:38 +0000 (UTC) X-FDA: 81784471356.28.481293D Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 3F7111C001B for ; Mon, 12 Feb 2024 21:40:36 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=I1xWkCmi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3U5DKZQYKCNIGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3U5DKZQYKCNIGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774036; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4cusAtaCctZ7K3pFyusQLmD/gTJ9EMe3N2uM61S6cPw=; b=XXFlFQw1040OOKOWiLeSyzHvIyNJeYf4cIi3/VCSvzWVa3VYWFLvWaiJRANvRPWQhdEPEv silIcqyHmkDM29rUGQ7Cdt6NGfI3Me7FFEEY20SZchMlpKQZuS6LFni2BrWDwTcfxFbfhl QU0ExeZfi9NgP7yiJSnLFwHnbgAbS0k= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=I1xWkCmi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3U5DKZQYKCNIGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3U5DKZQYKCNIGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774036; a=rsa-sha256; cv=none; b=CbKSx3GwJTgImznSm4k395vFcp/PsM4DCTZtT/QWRQkxqDl6tmEMAPSUhm7zJsNXU1amrP t2LzZj5MckuCaptjVmXS0NQbaOlJVYjf/I/LZ3RulsBEo+92fhz7bPoSV05wx1BECF0V6F XQ6y+/zAz2/HbJdmS2ESDmMEE7fzeQU= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6ade10cb8so8233952276.0 for ; Mon, 12 Feb 2024 13:40:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774035; x=1708378835; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4cusAtaCctZ7K3pFyusQLmD/gTJ9EMe3N2uM61S6cPw=; b=I1xWkCmilrfK0r7858RDjVlhOkedsgiOWsTMaPOehWBTMQz431HIQC1bBirtz9KFv3 AvM8NZwV0H2BkslvqIs4vx44EwA3xTSmElxIFKwUi/iPS97ZpQM8cHbVkZtj6KGDIvbU BgHClGrglw9fqSchqnMoAPHAI1XLS9yaDwCRBt3tC2v6dtC6XtxTgcZsKD2Jd19lt8kl xUFaBP2QaEifvRruIHGln9RHQayfiCk6R6wM7LcCcE0b0y88cU/CQPyWj1EQeT39G1Xm 6I+JCC69w6DOFz4EKDPEqCmnc039ectG9e7Arw68VGIcirT8C9pW3N466W0LT8EMryo0 hCEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774035; x=1708378835; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4cusAtaCctZ7K3pFyusQLmD/gTJ9EMe3N2uM61S6cPw=; b=u/1TKQuG4cNcCRhVfaR4g1aP2EzkC2ulssPEAQcuj7H5ablG/iXnR0ZRWVMrePA1UY 9xZ64MRLokPJdVKFxr1r3j6xOwG/AsKUzanX1PgS7hg/i9YCTL2CrlZIXdfLbz2bkSZX LvJJXr+gN05G+R71BHPaPGyK0qhZpssVUEqcjZJhTf6fODgXbLnWAup94p5pPfp/ewpn R1sELHF1UebijogA1rw4JLD2xS7AAVccEvN0h20TFIJDd94m0tYqc7BCHIqG/lKkgrs4 R4mrW3aDT1RVlBeCLJWmXk/T3TG8cc65A9HJZ5Xe1v/H5nuGI9/YNboZE7XkzY64VHR4 VH3w== X-Gm-Message-State: AOJu0YzUgxhImNsEnDto+Pk05QCn53se6I9D1JmqsqHGf4XGr0Z0raVF AyYZRMXj7ApjgKhTGbiWvXeCX/j90jpaz4GlC9cQ9HYGAIVTJAkfgBHm/Iw1LuwSUyfKPUbye3V aFA== X-Google-Smtp-Source: AGHT+IF2lAJfAQMKJa+dwJUeCTuXNowNKuBbS7XvK2jtWHY6jXD0O3VIO4SDQSUEkESVpV0NowOi7s2lO00= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:102e:b0:dc6:e884:2342 with SMTP id x14-20020a056902102e00b00dc6e8842342mr148128ybt.5.1707774035250; Mon, 12 Feb 2024 13:40:35 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:15 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-30-surenb@google.com> Subject: [PATCH v3 29/35] mm: vmalloc: Enable memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 3F7111C001B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: os5n4j1prr7zp4gmpisge3urbe8m77nq X-HE-Tag: 1707774036-745990 X-HE-Meta: U2FsdGVkX18+vlAY1D1wJElCOqjS5J/RZGpmjQkpMuWtVOVbypYSoDRVMCiN9oF+UFSG8Y9ff9qkfHDVbT+OYAo8T0E5fbZL0/+rYtmaIEWUdE1GKi0hb0s8DXthS4R4Ii9YYXKyYxTD1gVVicihOusxVXMdaEizxze8R+Yt1tFPuCIKDUb7417rD2GLwRIy6NH8GpRg5gYZpEBLuTGX/w3h0cWVUHEEu1J8rGNKUi2ikAARbKpHip95Bdebqg+pB6AQyTVHZaI4iG1rvbNyj2iYuIyqhM3Ed79ZuJWB+rZAG3zWMBmWbgZUKMa9PX/mggd1bKeoOr1CTCY0oywDjKwLNj8GZy7uvR5FCv5YyhJLdargd2jHe5moQubf/RtT7MdXp9cfPATq5/iVCIf4H94XVUeu72prDlAJ6j4286uhhqXtJ5o8T52JwPIdVur/4Py0X13ZvM3ToncAhh9Q5E5nXiYIDzqUkZC3Oto/Rtj1WQcB9z1RUEDag/Fw5/s+oG2az0lVbe23qQT/edvsWByGlbjK3+spgH7yJSRLmA/SJCWb+kqODxdrNM/3f9GqxwEZSUkdUpi9RHmcaE2rwpMbq//6JyYQkyTDoSiA03xICGhEb6Nl13qytvsDHvjVejBMrxrChRzzB+ssQ2GpMXFU8H/S720b8U8JcHVAcxiaEW47/JdRTJ/uHC33jNh4mtgliWz/ZTh4iCVoSoyaCIEapZDB0BOd6joMha16oSmTJzy7Het7k9j4p3XU8ATRqzLuPTB6mj4Tw3NQoaikr7nVK1RHqSacCRBEl6Z4uaNRAVsiJ6aKbqNiwalTjrSrTemcg4tPyXtW6wBygSQOsUZr3t+DGGkkuL/uh8DspaD8vbH/bjD/Vwdfa7Tr6+p8bUpk0E8tO3fqWzznVfSXzHDRvZpDdBdbYSJzUbiypFtqRHhvCk3QrLKv0KYQHto4bs1PJUq3R2ChevQa/DB 1kwk3ADQ 913EKCtj03df/IjKqIcQkt/ydSmovrpNnnCcD7Uu9chxVJBm6exYcnb0lOeGqOVVSoC9FOTlGpTx08o+JSKKjhSygDTQZ7z9udj1qG7rjz1lNupONXIOC4pvEd+OqOM6s2aaRTCe5ckU/8N8ozoAn5Dki9J5AuoMCbxsfRXoo/AFJXu3ezCd+d+TaqpmnzSaRmED06AWWrOCU3YTEbh0MD2lXm3ptvg71jiwHsIbo4jNT7wjn6gtd3xyLpWKOa90CegflVvoE0lnrBhvOGp+iSYt22dK+ZXlxCAO+AcF+94SqBVLOG41AX73/RbAH9cDLY1Vndi0Z21vQX+4CvkHqJRN3xi/PnjHhsQvMKHf7aiHExECCZM76mM5Oj1ssyboA9EaiGN1rrCt+PP2Yvtm4DkhSLw4ZiyG07ICoNjzYSGBY3w49fVrFppverr/8a+BY5O+puU0yiJ9lm3QBFP6L6jvAdrIqsr8TVZGperIblzjVTAvKKUgoGyexzZ6Hwee/UvO0F/EDx4XgxqjZx6AJofwTd2Mp+zGlJlOgFqR9e8ARS2J0nHXgS1tUt84/mmqBOLnYeKXPRqmwlGBiXGHdfsOV/hKT/Cg+LrAsvKHS3C01Y7PSnXUe5RQjDgJfklA/J9tdo86We2ueVKAQgV/8H+p5unVVlZEQxHrW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This wrapps all external vmalloc allocation functions with the alloc_hooks() wrapper, and switches internal allocations to _noprof variants where appropriate, for the new memory allocation profiling feature. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- drivers/staging/media/atomisp/pci/hmm/hmm.c | 2 +- include/linux/vmalloc.h | 60 ++++++++++---- kernel/kallsyms_selftest.c | 2 +- mm/util.c | 24 +++--- mm/vmalloc.c | 88 ++++++++++----------- 5 files changed, 103 insertions(+), 73 deletions(-) diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm.c b/drivers/staging/media/atomisp/pci/hmm/hmm.c index bb12644fd033..3e2899ad8517 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm.c @@ -205,7 +205,7 @@ static ia_css_ptr __hmm_alloc(size_t bytes, enum hmm_bo_type type, } dev_dbg(atomisp_dev, "pages: 0x%08x (%zu bytes), type: %d, vmalloc %p\n", - bo->start, bytes, type, vmalloc); + bo->start, bytes, type, vmalloc_noprof); return bo->start; diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index c720be70c8dd..106d78e75606 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -2,6 +2,8 @@ #ifndef _LINUX_VMALLOC_H #define _LINUX_VMALLOC_H +#include +#include #include #include #include @@ -137,26 +139,54 @@ extern unsigned long vmalloc_nr_pages(void); static inline unsigned long vmalloc_nr_pages(void) { return 0; } #endif -extern void *vmalloc(unsigned long size) __alloc_size(1); -extern void *vzalloc(unsigned long size) __alloc_size(1); -extern void *vmalloc_user(unsigned long size) __alloc_size(1); -extern void *vmalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vzalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vmalloc_32(unsigned long size) __alloc_size(1); -extern void *vmalloc_32_user(unsigned long size) __alloc_size(1); -extern void *__vmalloc(unsigned long size, gfp_t gfp_mask) __alloc_size(1); -extern void *__vmalloc_node_range(unsigned long size, unsigned long align, +extern void *vmalloc_noprof(unsigned long size) __alloc_size(1); +#define vmalloc(...) alloc_hooks(vmalloc_noprof(__VA_ARGS__)) + +extern void *vzalloc_noprof(unsigned long size) __alloc_size(1); +#define vzalloc(...) alloc_hooks(vzalloc_noprof(__VA_ARGS__)) + +extern void *vmalloc_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_user(...) alloc_hooks(vmalloc_user_noprof(__VA_ARGS__)) + +extern void *vmalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vmalloc_node(...) alloc_hooks(vmalloc_node_noprof(__VA_ARGS__)) + +extern void *vzalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vzalloc_node(...) alloc_hooks(vzalloc_node_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32(...) alloc_hooks(vmalloc_32_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32_user(...) alloc_hooks(vmalloc_32_user_noprof(__VA_ARGS__)) + +extern void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc(...) alloc_hooks(__vmalloc_noprof(__VA_ARGS__)) + +extern void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) __alloc_size(1); -void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, +#define __vmalloc_node_range(...) alloc_hooks(__vmalloc_node_range_noprof(__VA_ARGS__)) + +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) __alloc_size(1); -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc_node(...) alloc_hooks(__vmalloc_node_noprof(__VA_ARGS__)) + +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define vmalloc_huge(...) alloc_hooks(vmalloc_huge_noprof(__VA_ARGS__)) + +extern void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vmalloc_array(...) alloc_hooks(__vmalloc_array_noprof(__VA_ARGS__)) + +extern void *vmalloc_array_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vmalloc_array(...) alloc_hooks(vmalloc_array_noprof(__VA_ARGS__)) + +extern void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vcalloc(...) alloc_hooks(__vcalloc_noprof(__VA_ARGS__)) -extern void *__vmalloc_array(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vmalloc_array(size_t n, size_t size) __alloc_size(1, 2); -extern void *__vcalloc(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vcalloc(size_t n, size_t size) __alloc_size(1, 2); +extern void *vcalloc_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vcalloc(...) alloc_hooks(vcalloc_noprof(__VA_ARGS__)) extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); diff --git a/kernel/kallsyms_selftest.c b/kernel/kallsyms_selftest.c index b4cac76ea5e9..3ea9be364e32 100644 --- a/kernel/kallsyms_selftest.c +++ b/kernel/kallsyms_selftest.c @@ -82,7 +82,7 @@ static struct test_item test_items[] = { ITEM_FUNC(kallsyms_test_func_static), ITEM_FUNC(kallsyms_test_func), ITEM_FUNC(kallsyms_test_func_weak), - ITEM_FUNC(vmalloc), + ITEM_FUNC(vmalloc_noprof), ITEM_FUNC(vfree), #ifdef CONFIG_KALLSYMS_ALL ITEM_DATA(kallsyms_test_var_bss_static), diff --git a/mm/util.c b/mm/util.c index 291f7945190f..19c90036d3cc 100644 --- a/mm/util.c +++ b/mm/util.c @@ -639,7 +639,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) * about the resulting pointer, and cannot play * protection games. */ - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } @@ -698,12 +698,12 @@ void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flag EXPORT_SYMBOL(kvrealloc_noprof); /** - * __vmalloc_array - allocate memory for a virtually contiguous array. + * __vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vmalloc_array(size_t n, size_t size, gfp_t flags) +void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; @@ -711,18 +711,18 @@ void *__vmalloc_array(size_t n, size_t size, gfp_t flags) return NULL; return __vmalloc(bytes, flags); } -EXPORT_SYMBOL(__vmalloc_array); +EXPORT_SYMBOL(__vmalloc_array_noprof); /** - * vmalloc_array - allocate memory for a virtually contiguous array. + * vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vmalloc_array(size_t n, size_t size) +void *vmalloc_array_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc_array); +EXPORT_SYMBOL(vmalloc_array_noprof); /** * __vcalloc - allocate and zero memory for a virtually contiguous array. @@ -730,22 +730,22 @@ EXPORT_SYMBOL(vmalloc_array); * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vcalloc(size_t n, size_t size, gfp_t flags) +void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) { return __vmalloc_array(n, size, flags | __GFP_ZERO); } -EXPORT_SYMBOL(__vcalloc); +EXPORT_SYMBOL(__vcalloc_noprof); /** - * vcalloc - allocate and zero memory for a virtually contiguous array. + * vcalloc_noprof - allocate and zero memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vcalloc(size_t n, size_t size) +void *vcalloc_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vcalloc); +EXPORT_SYMBOL(vcalloc_noprof); struct anon_vma *folio_anon_vma(struct folio *folio) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d12a17fc0c17..5239f2c9ecae 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3025,12 +3025,12 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * but mempolicy wants to alloc memory by interleaving. */ if (IS_ENABLED(CONFIG_NUMA) && nid == NUMA_NO_NODE) - nr = alloc_pages_bulk_array_mempolicy(bulk_gfp, + nr = alloc_pages_bulk_array_mempolicy_noprof(bulk_gfp, nr_pages_request, pages + nr_allocated); else - nr = alloc_pages_bulk_array_node(bulk_gfp, nid, + nr = alloc_pages_bulk_array_node_noprof(bulk_gfp, nid, nr_pages_request, pages + nr_allocated); @@ -3060,9 +3060,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid, break; if (nid == NUMA_NO_NODE) - page = alloc_pages(alloc_gfp, order); + page = alloc_pages_noprof(alloc_gfp, order); else - page = alloc_pages_node(nid, alloc_gfp, order); + page = alloc_pages_node_noprof(nid, alloc_gfp, order); if (unlikely(!page)) { if (!nofail) break; @@ -3119,10 +3119,10 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, /* Please note that the recursion is strictly bounded. */ if (array_size > PAGE_SIZE) { - area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, + area->pages = __vmalloc_node_noprof(array_size, 1, nested_gfp, node, area->caller); } else { - area->pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node_noprof(array_size, nested_gfp, node); } if (!area->pages) { @@ -3205,7 +3205,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } /** - * __vmalloc_node_range - allocate virtually contiguous memory + * __vmalloc_node_range_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @start: vm area range start @@ -3232,7 +3232,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, * * Return: the address of the area or %NULL on failure */ -void *__vmalloc_node_range(unsigned long size, unsigned long align, +void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) @@ -3361,7 +3361,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, } /** - * __vmalloc_node - allocate virtually contiguous memory + * __vmalloc_node_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @gfp_mask: flags for the page level allocator @@ -3379,10 +3379,10 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, * * Return: pointer to the allocated memory or %NULL on error */ -void *__vmalloc_node(unsigned long size, unsigned long align, +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) { - return __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, align, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, 0, node, caller); } /* @@ -3391,15 +3391,15 @@ void *__vmalloc_node(unsigned long size, unsigned long align, * than that. */ #ifdef CONFIG_TEST_VMALLOC_MODULE -EXPORT_SYMBOL_GPL(__vmalloc_node); +EXPORT_SYMBOL_GPL(__vmalloc_node_noprof); #endif -void *__vmalloc(unsigned long size, gfp_t gfp_mask) +void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node(size, 1, gfp_mask, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, gfp_mask, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(__vmalloc); +EXPORT_SYMBOL(__vmalloc_noprof); /** * vmalloc - allocate virtually contiguous memory @@ -3413,12 +3413,12 @@ EXPORT_SYMBOL(__vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc(unsigned long size) +void *vmalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc); +EXPORT_SYMBOL(vmalloc_noprof); /** * vmalloc_huge - allocate virtually contiguous memory, allow huge pages @@ -3432,16 +3432,16 @@ EXPORT_SYMBOL(vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL_GPL(vmalloc_huge); +EXPORT_SYMBOL_GPL(vmalloc_huge_noprof); /** - * vzalloc - allocate virtually contiguous memory with zero fill + * vzalloc_noprof - allocate virtually contiguous memory with zero fill * @size: allocation size * * Allocate enough pages to cover @size from the page level @@ -3453,12 +3453,12 @@ EXPORT_SYMBOL_GPL(vmalloc_huge); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc(unsigned long size) +void *vzalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc); +EXPORT_SYMBOL(vzalloc_noprof); /** * vmalloc_user - allocate zeroed virtually contiguous memory for userspace @@ -3469,17 +3469,17 @@ EXPORT_SYMBOL(vzalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_user(unsigned long size) +void *vmalloc_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_KERNEL | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_noprof); /** - * vmalloc_node - allocate memory on a specific node + * vmalloc_node_noprof - allocate memory on a specific node * @size: allocation size * @node: numa node * @@ -3491,15 +3491,15 @@ EXPORT_SYMBOL(vmalloc_user); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_node(unsigned long size, int node) +void *vmalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_node); +EXPORT_SYMBOL(vmalloc_node_noprof); /** - * vzalloc_node - allocate memory on a specific node with zero fill + * vzalloc_node_noprof - allocate memory on a specific node with zero fill * @size: allocation size * @node: numa node * @@ -3509,12 +3509,12 @@ EXPORT_SYMBOL(vmalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc_node(unsigned long size, int node) +void *vzalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc_node); +EXPORT_SYMBOL(vzalloc_node_noprof); #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL) @@ -3529,7 +3529,7 @@ EXPORT_SYMBOL(vzalloc_node); #endif /** - * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) + * vmalloc_32_noprof - allocate virtually contiguous memory (32bit addressable) * @size: allocation size * * Allocate enough 32bit PA addressable pages to cover @size from the @@ -3537,15 +3537,15 @@ EXPORT_SYMBOL(vzalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32(unsigned long size) +void *vmalloc_32_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32); +EXPORT_SYMBOL(vmalloc_32_noprof); /** - * vmalloc_32_user - allocate zeroed virtually contiguous 32bit memory + * vmalloc_32_user_noprof - allocate zeroed virtually contiguous 32bit memory * @size: allocation size * * The resulting memory area is 32bit addressable and zeroed so it can be @@ -3553,14 +3553,14 @@ EXPORT_SYMBOL(vmalloc_32); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32_user(unsigned long size) +void *vmalloc_32_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_VMALLOC32 | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32_user); +EXPORT_SYMBOL(vmalloc_32_user_noprof); /* * Atomically zero bytes in the iterator. From patchwork Mon Feb 12 21:39:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554068 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1A53C4829F for ; Mon, 12 Feb 2024 21:41:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 863A66B00B5; Mon, 12 Feb 2024 16:40:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 813C06B00B6; Mon, 12 Feb 2024 16:40:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EF306B00B7; Mon, 12 Feb 2024 16:40:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4C30E6B00B5 for ; Mon, 12 Feb 2024 16:40:40 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 193CDA181D for ; Mon, 12 Feb 2024 21:40:40 +0000 (UTC) X-FDA: 81784471440.14.0209908 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 56007C000D for ; Mon, 12 Feb 2024 21:40:38 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e+trB7nY; spf=pass (imf22.hostedemail.com: domain of 3VZDKZQYKCNQIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3VZDKZQYKCNQIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774038; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IGO51nbRXLBlE7dZyNty79pnfUUWZlGfdlbPBWriSco=; b=Dg9j7wXTPIu5MdvpBSH4r3wND9Ku/4tR4DwABgGei1dJQz0DwL6JzHP28tzUGNt6VAi5Eo N/IfzkI+niauhg23oD8Q4BVEIs04qo6ZMur/a6e54wNproddosFrwwoTVmIGdcS6ZEL9UU Fsj9WtvyJj78dCeyYt5ZNQcBtoNd/tI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774038; a=rsa-sha256; cv=none; b=cPKL+aaa/DayckpmkB+SZylfDD8QBbnxXVZQ/3kiUnFfyKLdakTXexPF1EfeMZ0MOflZ7y /SSrhq65ZFkwPH7U9gv6jlru05p4MnnGnyI+qL3Omu3YarCAGWe7ysyswWd5r6iUZ3CR3N pL691z+Fz5AU8FvQebHQ5cV86JJ7Ku0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e+trB7nY; spf=pass (imf22.hostedemail.com: domain of 3VZDKZQYKCNQIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3VZDKZQYKCNQIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6077e1e919bso5746297b3.1 for ; Mon, 12 Feb 2024 13:40:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774037; x=1708378837; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IGO51nbRXLBlE7dZyNty79pnfUUWZlGfdlbPBWriSco=; b=e+trB7nYj0fEYD53i6T7Mw5efSZ+1AiiimEyjFK/Tu7cEEYc2airhCMv/38ZKlDfXX cT00jehFDzVQcJUmSzhetu0NE5+77/RShN5lixqFs7XpUKaV1XnyE8ue8kDMriyPbXaX 8hj4CC+vtjUucKNLiNI3q+xn8zLaeDb4o0T/Opl6rvk83fjZa4D3KpW4pMWpzhbXlgnn Bnt865iUiboB0lvLcoypZblBTrXSg6ef81qrQY17qPnKcNbGjn5KlUgpxfF24o3hztuA 6xaU5dWf1mf4PDOqf7a0zCJ5AoMI22y4MvDzxfPnxSNQO7iWMZThPUxZqFgVpQtDwA6u 6Fyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774037; x=1708378837; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IGO51nbRXLBlE7dZyNty79pnfUUWZlGfdlbPBWriSco=; b=wqWOMlnSdJ0ho55HHf1P91udZFE3m/j8lcZUahqNSIKJ+IwKqcGuKEJdZkZKuT1aez YJB5D6d1hKFyWcLnCTq23zxvhTZlE5GlLH7l4Mo5qOFsxCQN2CnXgZm9o/XKSLH/KhAP XLMC3W5/YUFlXGn7z4Ltk676bhchymAtle+6lAMo2etEIjkhGeUaSK7GQubVNz3SQVAZ R0d1wtmDiFy9vKzzkGrMX1hx/YqSxJ0KI9Runp1d2K/rL73Dq8VuMaRHfAkfhX2zNb0t /OnkTbzdp5hR3L3yrjl89cCBfVHZDHsFpeBmYnmrxuH6XC0UHGR02/Rp22KkZ2g/gLmv C83A== X-Forwarded-Encrypted: i=1; AJvYcCV0UqQUEWVPjUm/c6ZvbVLvAE/ngmtYFdY2+FvXERdnavnXFdWDXBELxyryPdTIPKp3PIy5g1ZgwzR6aVUzbUJ27nQ= X-Gm-Message-State: AOJu0Yx0llTqHyMvm8nLaYLSakfG76MNDKj3HxXqmyxFgpld1n0zpDqY NOmQIEX0oUAsxbj3CNxYAmFQrdnXhypCSDG+JFCqRACfZs0MGlouRdwuXbfQIXskzjP1JRNHB+A 1Gg== X-Google-Smtp-Source: AGHT+IHmk8+MK6zpdhADLMK0RWOIRjTnTYcSM/aIheok3v5Ff3xU8ZgKrwhYoeQuPApfz/4RhyV47R9pCGQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a25:9c08:0:b0:dc6:f21f:64ac with SMTP id c8-20020a259c08000000b00dc6f21f64acmr2107909ybo.12.1707774037425; Mon, 12 Feb 2024 13:40:37 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:16 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-31-surenb@google.com> Subject: [PATCH v3 30/35] rhashtable: Plumb through alloc tag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: w81s5rwyugw5esz5ukw1c5az7t5un3zu X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 56007C000D X-Rspam-User: X-HE-Tag: 1707774038-590218 X-HE-Meta: U2FsdGVkX19L4oaesGdT3MkAceFr/j6DPun67EdtGI9cRXybOslR4H11grGKDzKDItX40jRISeXj+6Ku+pKFkPoMIAWcgEQY1id8PGv5j91wlAS1c+SU/sWAdsTrYvWJ0k8DlJz7xXzICKNQANes5J7FQrfXbaDcgn/XYkgn7LQO5eOP1NIfJlztKnp+TkgSozeoSfhid/1VGuQkOTGSYAh7y79Y8kFMyBUy4x5ai5G+pM7rmWky4sDYeQeh/zMacZkinRrXEakMOfX7JpK3Bq7Ha2STnjIReueyM6Dwk0GNxuumxrv6z7VPDQHXjZrPAJPtp6kaFqMqoGDerLfWMfttUs1sk8J64MXHfF6q5iw0dRnSbvw8VK+1bU0auiooytQTJdo14Ic2dMi83Sf/00qXQ/Jv6OCSwQT30jS2VVElmF+YUtnol8gaRyZ0LwcSjs+PFfOh+woHyqGOZ5daxcx7wLmNVioYIe8itUS4IcCDfjMmKiEnUhebMnPLaVsegZAAeqB+PhOTERMotJSE/HicLQ60OjvDU1n61zUlceBV7asxpr/9g8Ljbqhxj536rPNyxHglI1w/LeomE7U5sczRWOLxTO6KzjCu+ox4xFI3KWMzfuiOuyNhoBAdWDFWnFHatx9SwoJiX3Cp6JT5QcisBXg4dEEFjRrmuWr4APgG0+Ai3/0C4aRe67oCD/9tOXJPUhq2V2smrf9TdkFwoKoqTFp6Yl8g+z/n50/9Q0Ho1w+gQodiQDRdsmh+YZIhWg5JUUO2/NtXo55ZAzAurnh9gieKSNowu6tTB3fvd2zqPiwBo9E7UeF0zU8lWcxJ1XO+U1KSRImgzS4diavrUCHwLnN3CpV3B7S6V3XcB57VO1XkEa6TiSad34VCwLZd9qsNfocZ77XawZ6DxqH3eFlUgieuW0SgCq8kyT9Do1DnaqqbetEieug+Kva7YGbcdLIrYFKR4pmyYepqJ+2 WtrNc6tF jRr+6Io8RxzdGqNtUgpj7QTJEvse3kmlPLSrSh30YpzmB2dL4lm/4UhkjJQmGVSH17EdHTp7zhEBXbTP6KhSUvTG0pSIpHmwyS0fmWpBydqitqQQcPynGlnHGUGrrwKGXqEk0GKlrzE3+pYnWQ250+5B/v8afaQmKOjSY5DJFXcZcBpbwyscoa/q6tyXmBt11xngcj0XjhXU3uCOyvVWcKdIjpJed2uZXkSocAWyw4lQ3mcsmNEj9dd3EIUWHF8nKYugLARuJKSduNfoSIaGNArtDAieMzxSdEhOsHe88qqSAVG0lHXJbwe9P+ht0Qt8y/rMbc971yg0qjQaiCznHGREWuEktAgvqQWfvoFbnCW/J9rhrqDVK+/sxfr2x+05Eax3TvHtuf1zdfo4u5P3WoAnHdrHlaBZyoLedmWX07t6D5sPfWceBVOCfc35BWhaXCSOw6X7tDp0C0IPvpQc1vJpIQhspV3DaSKbZJbpRTfTgjkqLXL6n9OUE/W2V9HPt0WJ4JwqX6r7Qr8zrBnEV+0sRksWtMhhMVCDssLS+bcrIhwiyhXIHb+feP/CDZli+WrBZVXwyTs3Kj33+G4SCngv47RGi4wLPf+ssxZzSJBIIP/mUMkTrGE/CqIm536n1tTywTJTILxSJ6TGBr5t7wIa8t4wvLYPrFp3oTRt7J2UPXPelR4hL7hKt9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This gives better memory allocation profiling results; rhashtable allocations will be accounted to the code that initialized the rhashtable. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/rhashtable-types.h | 11 +++++-- lib/rhashtable.c | 52 +++++++++++++++++++++++++------- 2 files changed, 50 insertions(+), 13 deletions(-) diff --git a/include/linux/rhashtable-types.h b/include/linux/rhashtable-types.h index b6f3797277ff..015c8298bebc 100644 --- a/include/linux/rhashtable-types.h +++ b/include/linux/rhashtable-types.h @@ -9,6 +9,7 @@ #ifndef _LINUX_RHASHTABLE_TYPES_H #define _LINUX_RHASHTABLE_TYPES_H +#include #include #include #include @@ -88,6 +89,9 @@ struct rhashtable { struct mutex mutex; spinlock_t lock; atomic_t nelems; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif }; /** @@ -127,9 +131,12 @@ struct rhashtable_iter { bool end_of_table; }; -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params); -int rhltable_init(struct rhltable *hlt, +#define rhashtable_init(...) alloc_hooks(rhashtable_init_noprof(__VA_ARGS__)) + +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params); +#define rhltable_init(...) alloc_hooks(rhltable_init_noprof(__VA_ARGS__)) #endif /* _LINUX_RHASHTABLE_TYPES_H */ diff --git a/lib/rhashtable.c b/lib/rhashtable.c index 6ae2ba8e06a2..b62116f332b8 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -63,6 +63,27 @@ EXPORT_SYMBOL_GPL(lockdep_rht_bucket_is_held); #define ASSERT_RHT_MUTEX(HT) #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline void rhashtable_alloc_tag_init(struct rhashtable *ht) +{ + ht->alloc_tag = current->alloc_tag; +} + +static inline struct alloc_tag *rhashtable_alloc_tag_save(struct rhashtable *ht) +{ + return alloc_tag_save(ht->alloc_tag); +} + +static inline void rhashtable_alloc_tag_restore(struct rhashtable *ht, struct alloc_tag *old) +{ + alloc_tag_restore(ht->alloc_tag, old); +} +#else +#define rhashtable_alloc_tag_init(ht) +static inline struct alloc_tag *rhashtable_alloc_tag_save(struct rhashtable *ht) { return NULL; } +#define rhashtable_alloc_tag_restore(ht, old) +#endif + static inline union nested_table *nested_table_top( const struct bucket_table *tbl) { @@ -130,7 +151,7 @@ static union nested_table *nested_table_alloc(struct rhashtable *ht, if (ntbl) return ntbl; - ntbl = kzalloc(PAGE_SIZE, GFP_ATOMIC); + ntbl = kmalloc_noprof(PAGE_SIZE, GFP_ATOMIC|__GFP_ZERO); if (ntbl && leaf) { for (i = 0; i < PAGE_SIZE / sizeof(ntbl[0]); i++) @@ -157,7 +178,7 @@ static struct bucket_table *nested_bucket_table_alloc(struct rhashtable *ht, size = sizeof(*tbl) + sizeof(tbl->buckets[0]); - tbl = kzalloc(size, gfp); + tbl = kmalloc_noprof(size, gfp|__GFP_ZERO); if (!tbl) return NULL; @@ -180,8 +201,10 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, size_t size; int i; static struct lock_class_key __key; + struct alloc_tag * __maybe_unused old = rhashtable_alloc_tag_save(ht); - tbl = kvzalloc(struct_size(tbl, buckets, nbuckets), gfp); + tbl = kvmalloc_node_noprof(struct_size(tbl, buckets, nbuckets), + gfp|__GFP_ZERO, NUMA_NO_NODE); size = nbuckets; @@ -190,6 +213,8 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, nbuckets = 0; } + rhashtable_alloc_tag_restore(ht, old); + if (tbl == NULL) return NULL; @@ -975,7 +1000,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) } /** - * rhashtable_init - initialize a new hash table + * rhashtable_init_noprof - initialize a new hash table * @ht: hash table to be initialized * @params: configuration parameters * @@ -1016,7 +1041,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) * .obj_hashfn = my_hash_fn, * }; */ -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params) { struct bucket_table *tbl; @@ -1031,6 +1056,8 @@ int rhashtable_init(struct rhashtable *ht, spin_lock_init(&ht->lock); memcpy(&ht->p, params, sizeof(*params)); + rhashtable_alloc_tag_init(ht); + if (params->min_size) ht->p.min_size = roundup_pow_of_two(params->min_size); @@ -1076,26 +1103,26 @@ int rhashtable_init(struct rhashtable *ht, return 0; } -EXPORT_SYMBOL_GPL(rhashtable_init); +EXPORT_SYMBOL_GPL(rhashtable_init_noprof); /** - * rhltable_init - initialize a new hash list table + * rhltable_init_noprof - initialize a new hash list table * @hlt: hash list table to be initialized * @params: configuration parameters * * Initializes a new hash list table. * - * See documentation for rhashtable_init. + * See documentation for rhashtable_init_noprof. */ -int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params) +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params) { int err; - err = rhashtable_init(&hlt->ht, params); + err = rhashtable_init_noprof(&hlt->ht, params); hlt->ht.rhlist = true; return err; } -EXPORT_SYMBOL_GPL(rhltable_init); +EXPORT_SYMBOL_GPL(rhltable_init_noprof); static void rhashtable_free_one(struct rhashtable *ht, struct rhash_head *obj, void (*free_fn)(void *ptr, void *arg), @@ -1222,6 +1249,7 @@ struct rhash_lock_head __rcu **rht_bucket_nested_insert( unsigned int index = hash & ((1 << tbl->nest) - 1); unsigned int size = tbl->size >> tbl->nest; union nested_table *ntbl; + struct alloc_tag * __maybe_unused old = rhashtable_alloc_tag_save(ht); ntbl = nested_table_top(tbl); hash >>= tbl->nest; @@ -1236,6 +1264,8 @@ struct rhash_lock_head __rcu **rht_bucket_nested_insert( size <= (1 << shift)); } + rhashtable_alloc_tag_restore(ht, old); + if (!ntbl) return NULL; From patchwork Mon Feb 12 21:39:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 075FFC4829E for ; Mon, 12 Feb 2024 21:41:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF9586B00B6; Mon, 12 Feb 2024 16:40:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CA8E16B00B7; Mon, 12 Feb 2024 16:40:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF85B6B00B8; Mon, 12 Feb 2024 16:40:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9B8046B00B6 for ; Mon, 12 Feb 2024 16:40:42 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6F2ED80782 for ; Mon, 12 Feb 2024 21:40:42 +0000 (UTC) X-FDA: 81784471524.29.3CC0ED8 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 9FBB940013 for ; Mon, 12 Feb 2024 21:40:40 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="3TF4t/zw"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3V5DKZQYKCNYKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3V5DKZQYKCNYKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774040; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hFi0jScQsczt4c2Plav2pAvubIeyjPFOX5wIUH8AZHM=; b=yjrq64QXYaYIo+IoBBrDXmiYwevtZ/+YK0jTw0W0iZv/UhQdl4a9u5IsHP8OxXm4iNfrwe ObTwTgKblNWbHpmBzDyyS4rzV6dmkTxo6JGOaM8H/BxjfxJR6fj7DzJuRbZlyhyOFrioky +NCSSVzG+p04TWbvp70+6/PW5HJTkFg= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="3TF4t/zw"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3V5DKZQYKCNYKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3V5DKZQYKCNYKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774040; a=rsa-sha256; cv=none; b=L2MYCMLWdiqNKyatHcKLEp1XefcX+oHpEEBIBWFSbimBz9CgjKFAc30ZFqb6ZtNMz4Bn13 lMhuaIyrf/7kLtPSk6HiShxKyUahc8ezmjRojTNVu/Rq3oH6ouBBRGpjOcb8Mm1eof25UY VaGccixE+DrgQm/w6U3+dk4jkXTPUDI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-604ad18981eso84789347b3.2 for ; Mon, 12 Feb 2024 13:40:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774040; x=1708378840; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hFi0jScQsczt4c2Plav2pAvubIeyjPFOX5wIUH8AZHM=; b=3TF4t/zwjBfJ70Ygki9A2/Li9mimSv+A0hcUbBF6dioSd7YPyELwLQbq8nauTQcpC5 D/msBRPyYaMtgM3wKPOSnnhzNNYklvsULu6pO3U+xQttudWjM5xwumrnzDzVNoVax7O1 zgT1d8tjs235lBIOgwYP1wWoWtRfqOsV3/dzcP5ZUiJ6NFDu4231kKtI23RASd4hltsj 3PpsvOlXXwe0fTqYcrO7jcLn2+tu8IWsQdfebIReZXaMgRe4mP64ZfVNyKHsp8GR5tss jc+zaNBsvewLGWqv3On9sG+xf6VOT9SJYuYH99PzjyIVUKGRUYx9ueMLoG5rX3COhPaD oj8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774040; x=1708378840; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hFi0jScQsczt4c2Plav2pAvubIeyjPFOX5wIUH8AZHM=; b=gVoNW4pflNeIKkF4r1BcBSabZDSBY2yaUsdrHp5N/ZHxXx5I/9NucuWzvVW+WCaSpI 88irTelmJN7QNkzmiJHh+e0qDKqStjGPBEfVfdhNVv1GRbI4lMeUfDKRmLwf8M7bOOA2 G9DvQ+L6vt6KUKkNil5M4F0Gi8fGuKg1+IPjRPV5OMETOYHXgF95A+l6cHFZe2Z/C4Le ETrouBRxtXRn71blaVO+KtUd7M+XB+jLARt+y4ZJKMv8KW6pUdnk4qaOXDtZ/htbSNiY emjWj/3V5WfPqE6b1easzq/AMj/erW4UI6A4txgQ+ojN1GwDHduPhcQZq2p2tlIBrblM zINQ== X-Forwarded-Encrypted: i=1; AJvYcCUR5oAJOmgVYWCYTbnyIbEilAw/mR0lnoQWjEsiWJjXwR5DVOB2rE3203TpJW5yBDPDIglyuUpFIVt+d7twgjMCFH0= X-Gm-Message-State: AOJu0Yz64OAGWDGno1HZHvq/N+1pthE49jLHMvEtxEx85MQi+3Dhx5o7 4+X9/ft07FJk/O220S56uMYZ5bxVu0i8OJMnFRkYkKcZE8ZgWVTGLjRHqZcSbmGsY43FO2VdW/J mLg== X-Google-Smtp-Source: AGHT+IG01ZIdCxyG6doWIwi6F85UI5DWfcKo01/n0teetJJReA8rQ10tG50Lh8eu7adVmXuhCLeMmXOg6Mg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:690c:b8b:b0:5ff:96b6:8ee1 with SMTP id ck11-20020a05690c0b8b00b005ff96b68ee1mr2134418ywb.7.1707774039644; Mon, 12 Feb 2024 13:40:39 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:17 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-32-surenb@google.com> Subject: [PATCH v3 31/35] lib: add memory allocations report in show_mem() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9FBB940013 X-Stat-Signature: jbis615nkkzht76tgypq8cdqgj6yy8nw X-Rspam-User: X-HE-Tag: 1707774040-657212 X-HE-Meta: U2FsdGVkX1/nEjxdJubblq+ip+k5/rVtPY+dWUsq9mFj/fwTA4DxQYYyaK+5ZHOAJXP+5crtymFQNCFunjr3D4x3jHPyN4Rjvj+Lm4sgo1BozhftSrpWhGhvUam750dSh/3dyvhTRhrDqxrTf/dX5n88Hc9nu108mPFo4dUtDhKRvKPM1qwUhF0Fr8jmP7ccZqNfWZH9HIZnmhb6G+NEWE8shByj1g6WsGadzcP610Dzr8AYPrfx7HGGvoJyeClBv1i4OTt5ziqeRnQO36gDjHXRaEOquJf7VETw9ISHEgLb8+F6iiyYB33N0e8LxJGAQaN6SP39jWD77WtIoda88Uuuaz5IjtT5mp6pNk4ZJeVUiEyUyU0gfO3NUVR45OBMVhJalBxeGNV3UnDxWtW5AedLXFl/BVnAi1+4mfYsYuI5w+G7prkx6kzPMhuztIVwN9mBcwSFc8V7D0gxbt9rxE7UO/BmUrg00L6kNGMH2Fll3nICY6ys0lTtVTVuj8lRQ9kFkJXFsS9OGIGfexQVk6088t12e1vodbSpPl+d4sqDMA5YR5Xj2B6MYItXgiTUQZS9N8tr8JZnDQ6aMyeLr0OVGjs7puPhtiLmZs5IOnBKN8pInlXZ0U5lvZkQXXZNuDWevwL18rslEu2gDc70/hMkJ/W104zGw9XGTvTpWUTqgwz5o53PhfnWBxNUHneux9ohL6Zy9MGY8BzocnK3xT/k7mjv6UEdve2+uHSRSqHfLbFaSIVsfXMbxwJB724DIuiXPHBO4oLbS/UrUtoT6f/+q9uYsZP/BFoPPXHHwbNtJN+nAXMBDlQCnGVYfvEcf3xcwN4X6J1miQHSjfwVBJpmOK8uEGmaXJrMBYZ+D3Ro5khUG/vSAalv38f/+FbIAYjQDn66Fsh+TsU4P+Iq9MYjAyrGf79jQGV5r+u2fivdrdgGYalA8mKDTBZ14PdpnqILdyvsJ4rjx3ehFZb DqhGaXrd zM7xR6f2yVeiodhAy1l6rSK9pdcBkyZZa+IeYblJA9kzvvnFqr2wxfwUKYEjcKZWHIeR0ptH0nK/u9rKIGlWYlP1ymZcL/FRqRQ6MnWo4MKslrm3t7WVqcjhnVbhSBevkaIBSWN+GAk+lWR+/aiGmDwYBhjZ9POOiTGrsV04Wm2/O1j/Dty1LKjDWqkrwDWzOgxZEZeWu9kYucEhvJWCmAtO9mooS92PqFZ6WlbJPckC12nB44gejmACvp1x9/BsRvRJ5bfAbGnE6bSOjZBcSLJD7iPlYh858atPECpUskNPCFGUYlWo8ZA1ajPuFyWCWyNLq247PfsoxaIhrbzsD2ejp6UCxdgXFWsA6Hs2dL399Zkj3h1FVkfv5oMgeMK5bLMBe5wGQiNIdzdqDoH2LCwBlnJZLBqxCnacGIcUsuGYTFfKkeK5gy8Q2zLr6EN2AAwahnt64+8gAnE0QXe7A8QtFAX5ID8RQBk6tI4kxKGH8GC6nhES2Te6mp/K9YGDboMZRn5XYDA7xXHBl5NRUcq2NWzN5wj67M3jY4msZiYZVRt2Sjnlt1HoGN3KzX15Q194BRpL4sfbDSVPRwCKR4VKrtPmTUABlzrZCjs/gOkT+7eir2HstbBiEC2BkjQ7OrrV3JKX1UX0B/scJCTAb9gT+aR5RMCqoDtl014xAMN076a+8BO2lYH+3ezC9z7+dH7jpHnC+ypPPvlFUPTLN7PGf8Oj6H+iX5AjW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Include allocations in show_mem reports. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 2 ++ lib/alloc_tag.c | 38 ++++++++++++++++++++++++++++++++++++++ mm/show_mem.c | 15 +++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 3fe51e67e231..0a5973c4ad77 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,8 @@ struct alloc_tag { #ifdef CONFIG_MEM_ALLOC_PROFILING +void alloc_tags_show_mem_report(struct seq_buf *s); + static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) { return container_of(ct, struct alloc_tag, ct); diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 2d5226d9262d..54312c213860 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -96,6 +96,44 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +void alloc_tags_show_mem_report(struct seq_buf *s) +{ + struct codetag_iterator iter; + struct codetag *ct; + struct { + struct codetag *tag; + size_t bytes; + } tags[10], n; + unsigned int i, nr = 0; + + codetag_lock_module_list(alloc_tag_cttype, true); + iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(&iter))) { + struct alloc_tag_counters counter = alloc_tag_read(ct_to_alloc_tag(ct)); + + n.tag = ct; + n.bytes = counter.bytes; + + for (i = 0; i < nr; i++) + if (n.bytes > tags[i].bytes) + break; + + if (i < ARRAY_SIZE(tags)) { + nr -= nr == ARRAY_SIZE(tags); + memmove(&tags[i + 1], + &tags[i], + sizeof(tags[0]) * (nr - i)); + nr++; + tags[i] = n; + } + } + + for (i = 0; i < nr; i++) + alloc_tag_to_text(s, tags[i].tag); + + codetag_lock_module_list(alloc_tag_cttype, false); +} + static void __init procfs_init(void) { proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); diff --git a/mm/show_mem.c b/mm/show_mem.c index 8dcfafbd283c..d514c15ca076 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -423,4 +424,18 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) #ifdef CONFIG_MEMORY_FAILURE printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { + struct seq_buf s; + char *buf = kmalloc(4096, GFP_ATOMIC); + + if (buf) { + printk("Memory allocations:\n"); + seq_buf_init(&s, buf, 4096); + alloc_tags_show_mem_report(&s); + printk("%s", buf); + kfree(buf); + } + } +#endif } From patchwork Mon Feb 12 21:39:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554070 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D40BC4829E for ; Mon, 12 Feb 2024 21:41:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6668C6B00B7; Mon, 12 Feb 2024 16:40:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F9716B00B9; Mon, 12 Feb 2024 16:40:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4427C6B00BA; Mon, 12 Feb 2024 16:40:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2B8476B00B7 for ; Mon, 12 Feb 2024 16:40:45 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0645F1406CF for ; Mon, 12 Feb 2024 21:40:45 +0000 (UTC) X-FDA: 81784471650.09.17B84A8 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 2CBA3C001A for ; Mon, 12 Feb 2024 21:40:42 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cbruGCtD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3WZDKZQYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3WZDKZQYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774043; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/sI0PWD0cr+RKARN01FIe/JdJLbOCpqo744sqtIKAHU=; b=JBgdu8LfNQCxL820niLWxWrwU3co0wfv/mV16FwfonTGLLAg7+WDMuqPfiRfPWjHTQEjAX xbA1RceRiTQ9Th5aU+i6aL5raoXwmmWJbXyCTOSDZkz6vBWp+KQ9YCSgo7fevrcFZoitI0 E7xloUmz8RVqHBi2wcVl8L5pSTTX2Oc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cbruGCtD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3WZDKZQYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3WZDKZQYKCNgMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774043; a=rsa-sha256; cv=none; b=u6NBJjKo2mT8SO3TI8hAoiuRGuW2Nkxoyt68Dx91Uhv41JGytmstOkrV6/jPYRC/4cnXDG IeokpMajLgV9Gcrs6vzxgb4veI0idcgCGlWPsvmzY+JQqw+BDFb8mxrp6an5siovhV9NY8 QgeTAhxY72lhsqrG3XdFfZakeoPuR10= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6047a047f4cso97758597b3.3 for ; Mon, 12 Feb 2024 13:40:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774042; x=1708378842; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/sI0PWD0cr+RKARN01FIe/JdJLbOCpqo744sqtIKAHU=; b=cbruGCtDsBHj1oL5+F4uVE6/C0+JAXlcGV81tVMUwO5ZMibQ18e+XPvNpRklpUR/1g 5pImFl3xUuhQfPWr3J5DvVeOlI8SoOiDBjs3uiFP/0EzP4G63hXGgqDU3YI5ExVATgvU UvIewxAbScHu8PkYkCvfpKkz0CSDpxfTm5mMeLoQEeME4wyUbcrHTRFGjt84nuF5QcFb oV/FDyuynm/F8MzWmzVujEWdoI0CY0YygBsCyIGhH+MRQzPyQthh0b1+H/S+eiSZ523A iUnfhlr8J4NHKxC5kRaDjH0MItin87TINMwpPA7MjO/UzUuuQfP3nl0lC8EjYiIX1g/v yuPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774042; x=1708378842; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/sI0PWD0cr+RKARN01FIe/JdJLbOCpqo744sqtIKAHU=; b=MBnkR0EViao2K3yh2AaOcAZtupjlsvicekMJKbPGaNU0fHQ1okQRfPWr8eboxABoSx FmCEOU6sSC9goeMM8M9+lNmGOIZj4HyEefXx1k584wxaCLj5ZZxfPKXpQ50cPXulFfxO 93oviPdAM//q5H1t+Xpzp19Qt2E9hae2uk5RDhwNG350t7bDT2Gl+ylFh5fp0sT6w18B byT2EpdrdE7ASe01tI2bWDVkRm97QXexfMgQBtq5MTyf7p+VuGtNM424ZXTP67kUGxUk EKOSFu6dfsc1ATrd79CwzAC50+4vsoCqFEBe+QV68felNPz/Eos1mQe68bz9LvAMd8n3 P1PQ== X-Forwarded-Encrypted: i=1; AJvYcCVnC9uWYA4LeiBxnDrI/Lai5f1ViSXgHmGAj7opnNZIw5zCAh/f/xweSAUCHfFKZ301WPSgJaKk7uGdbQv0TWR4o7U= X-Gm-Message-State: AOJu0YwpSnHxwxI34kx9NPrnNEBuS+69h6Bo1VD2ZBCpB5Qe5Z9PGa1n fkFHuhaKn9SFSSPZi93wKpWvKmXNwwBCPnFrUmlIv/nQC8WtaWdD1Saq6hjmffi/YHDKjYpc1MN JmA== X-Google-Smtp-Source: AGHT+IGlJug7UeocMp/yJN58BwDAI5E8aN7090iq/Hzoe26lNZkGF0b2He1z8X2tvc+mRRdeZBlUlmhSz1U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:2186:b0:dc6:cafd:dce5 with SMTP id dl6-20020a056902218600b00dc6cafddce5mr2274526ybb.12.1707774041837; Mon, 12 Feb 2024 13:40:41 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:18 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-33-surenb@google.com> Subject: [PATCH v3 32/35] codetag: debug: skip objext checking when it's for objext itself From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 2CBA3C001A X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: u9hf9tg5biddcbg3e6dufrrsdpdyw9qy X-HE-Tag: 1707774042-768903 X-HE-Meta: U2FsdGVkX1+kpj4VldmVm7HqSVdXOUEMgAsPWkfkX7gCL5m0o5/xS2/azoUf+45om3tjrfwUQ5DOITPflzTEFsrLVXcfR3NyQiFIjs2qORxJkEBtzDIQhoP9XHCPcHsJZXER9DQtEdVV4OUnBIVYrQQk8CKJQA/QpekVAoX8zaYZwOo/oe9kpARakEc00CzoiYVNxcFw+tgqtUjaBLzxtN4DEpwzV+De6NywzUzKIx6S+HVBNRZt9ZRMDp5EapDE6QjdaQMNr5l6GTTAcbe+vdEkHNar24adWDZ3dgKMe1ibRByaxOi5L9gtqsTaDCsKQzKuwJZr+S2KVJJ7PXO2frkJI4JkPN2kVaOXw0944wWiYj6BoVYxv4wElfr1DS0uxLCF3gG21rDW28H67fEr9pdI/o02/uTFCaU34HXl+nkk6jVutImzOewg9Q/3TuAzmBQ+ZUqkJsui++CF/LUNV2ZU0450Tt613LPyjja6i+BQ+2td9g9wg+vDF29EPvXsKADFdfsqSDQ6UMY+WQQc9Qdqh5XYnreHvrO68P8wrVm1/rHG08erxToNr4j00UjcKYxUGjiv8a+bRMe2jww1w+u0l5OwXvkZGJDkZSz+pjVrL3z/7TlzFK+kUs/KWITYAdtlTiv7lv8Iwu/ACT5jo9VEiTcWQDqt0Kks1g28N5VnwGR6faJ7B64KXtUBK2Ftw0yK9IZZcNAcv/k5fGNjtOfitAMPK53wUjST99rUwhln+tnAABxq9WFzlmOmCKHkKWWx+7NihFa+pGOskTYJZTxLo6ayjPRULDmFTPQpdPav3r1jy21Oh5oC6Gwl1v+cLxeU/qkqBOvi3KXeXtkumpc5pIfPkawwh1w9mqZDmfKKlhwmnLngyTPS069AtfJrSi2YvBa4cGnSvEQaFZI66dhMhwGjYaaHDQXcIilAAhwNourayz5EPLB5a4MzQAEs927bpOtCKDUWveTxbu5 rKTfxYIN bHqCYVAP1vpScx/IGvpBcNeLbc6IGNV1q4fdxo/ABRJyygO8OBzRdDV6uyOe+BzPqFiQWN1cOHe47lQr8GzGCgiVV9yFxO2Z4zw5ZswUbbLLs7N4pA+BxswApqjC4QUvnxJRe6ayhSqMiiRycJlxAEec8ACH3Tvf20g95QMxO/WaIV1FIuOfAlyely0VfbqwTvyZy2yI3Pya+iLUb8f3H9ogro2ho+KK37fWxf4mF633Sqgtg86Dulq2NDgl1p1uOJT0/ssIwwBfrkhyId/+eCk09217Vl10ixBautlF5gzizkTkv4sS9/NLbypNd5qqLedY/GDNmhoXXiwpHXpyU0wmEVUu4n0DurtdfOEwp5PhkO5CbTvWdb0BhApDKfu0nH9fez2+CFd797Pkp6/oLxOepmQOmb3SCd1ej7CWBqO0nKMLoNNd+IRThCw7D/vt/VMaE615iGaRAa1oH9kPUrgAbELgM+4nU1zZiCChSdmrEbr6Qnl0W5KdP4yteHX20qzeQj543zoZ3RofsDAgVHuvjoXelPoRq9dh0Od8Egx4a/55HbzAzWAtFE0jyDAVf4dGz4HlISozVO/LfdF7LdnALRcB5Wja/iZVuFBuIwiTmmh998neZgG4+g2rOX3DN7SowaVlVxg1Bb3YwK8i77u7zTM0zCMl08v2NF3EJMY6Kos8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: objext objects are created with __GFP_NO_OBJ_EXT flag and therefore have no corresponding objext themselves (otherwise we would get an infinite recursion). When freeing these objects their codetag will be empty and when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled this will lead to false warnings. Introduce CODETAG_EMPTY special codetag value to mark allocations which intentionally lack codetag to avoid these warnings. Set objext codetags to CODETAG_EMPTY before freeing to indicate that the codetag is expected to be empty. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 26 ++++++++++++++++++++++++++ mm/slab.h | 25 +++++++++++++++++++++++++ mm/slab_common.c | 1 + mm/slub.c | 8 ++++++++ 4 files changed, 60 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 0a5973c4ad77..1f3207097b03 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -77,6 +77,27 @@ static inline struct alloc_tag_counters alloc_tag_read(struct alloc_tag *tag) return v; } +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +#define CODETAG_EMPTY (void *)1 + +static inline bool is_codetag_empty(union codetag_ref *ref) +{ + return ref->ct == CODETAG_EMPTY; +} + +static inline void set_codetag_empty(union codetag_ref *ref) +{ + if (ref) + ref->ct = CODETAG_EMPTY; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes) { struct alloc_tag *tag; @@ -87,6 +108,11 @@ static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes) if (!ref || !ref->ct) return; + if (is_codetag_empty(ref)) { + ref->ct = NULL; + return; + } + tag = ct_to_alloc_tag(ref->ct); this_cpu_sub(tag->counters->bytes, bytes); diff --git a/mm/slab.h b/mm/slab.h index c4bd0d5348cb..cf332a839bf4 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -567,6 +567,31 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab); + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) +{ + struct slabobj_ext *slab_exts; + struct slab *obj_exts_slab; + + obj_exts_slab = virt_to_slab(obj_exts); + slab_exts = slab_obj_exts(obj_exts_slab); + if (slab_exts) { + unsigned int offs = obj_to_index(obj_exts_slab->slab_cache, + obj_exts_slab, obj_exts); + /* codetag should be NULL */ + WARN_ON(slab_exts[offs].ref.ct); + set_codetag_empty(&slab_exts[offs].ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + static inline bool need_slab_obj_ext(void) { #ifdef CONFIG_MEM_ALLOC_PROFILING diff --git a/mm/slab_common.c b/mm/slab_common.c index 21b0b9e9cd9e..d5f75d04ced2 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -242,6 +242,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, * assign slabobj_exts in parallel. In this case the existing * objcg vector should be reused. */ + mark_objexts_empty(vec); kfree(vec); return 0; } diff --git a/mm/slub.c b/mm/slub.c index 4d480784942e..1136ff18b4fe 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1890,6 +1890,14 @@ static inline void free_slab_obj_exts(struct slab *slab) if (!obj_exts) return; + /* + * obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its + * corresponding extension will be NULL. alloc_tag_sub() will throw a + * warning if slab has extensions but the extension of an object is + * NULL, therefore replace NULL with CODETAG_EMPTY to indicate that + * the extension for obj_exts is expected to be NULL. + */ + mark_objexts_empty(obj_exts); kfree(obj_exts); slab->obj_exts = 0; } From patchwork Mon Feb 12 21:39:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554071 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C901FC4829D for ; Mon, 12 Feb 2024 21:41:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2484A6B00BA; Mon, 12 Feb 2024 16:40:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D00C6B00BB; Mon, 12 Feb 2024 16:40:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA1576B00BC; Mon, 12 Feb 2024 16:40:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D49676B00BA for ; Mon, 12 Feb 2024 16:40:46 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A85DB1C125A for ; Mon, 12 Feb 2024 21:40:46 +0000 (UTC) X-FDA: 81784471692.25.C8F0807 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf15.hostedemail.com (Postfix) with ESMTP id 09E9BA000C for ; Mon, 12 Feb 2024 21:40:44 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=C1UZQ67g; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3XJDKZQYKCNsPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3XJDKZQYKCNsPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774045; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DSEwnOfj2VuhpdkIHZ8VNVuKfvBXo1I5xAUr9MF0J1M=; b=xSltOd1mniaELI2yWbweesyp5U8u+4LuMcH1kLtxresgxa97C9YfevkSe9P0rUmiYZPip1 osS+ddotRne6e8CWURiTtt310neC+jKhLiW3871qia3c2tw9OiNsd8MwHE23ZQsSSPNAKW 1z0zTozExcXzJjdzjgkVcAtkxnQQJE8= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=C1UZQ67g; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3XJDKZQYKCNsPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3XJDKZQYKCNsPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774045; a=rsa-sha256; cv=none; b=DZG4WZUBtIgAJ7zeFeFFKLRA9JwuYx1fTs1VwF2SXldp7SzSp9lUXpBiteqyl6FEtaPwEv Qxz3Z6TFWRe3bJ6j/nzbu0i/shcgJFkeByQQQ9P0J688g/OVCPj1MxKnvaRWdUqgPqxK1D WnuMACIVHvIgpz8ZQ/x8mN/5qS0LNps= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60753c3fab9so16582267b3.0 for ; Mon, 12 Feb 2024 13:40:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774044; x=1708378844; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DSEwnOfj2VuhpdkIHZ8VNVuKfvBXo1I5xAUr9MF0J1M=; b=C1UZQ67gSuC6snaJmQ0Tq4S6qRLyJLV8cAuZqBT//6L7F6AQU1e/j7nCt0OHhHPEj0 6OSLMqf+2oFi+feAIkPuMi9recQQIc78SsPxIHEXfAvt2TiVHaJ9Fpp4gDWH4ihbt7c+ e1p/fENOa5rowQGKm6Wbm0NozRXDGUErnyiOT0EiYX5aefPC9M65YUvVFTZREDyi/2dr fXJJTU21LbuAcP6q5A0C5vfy4Y//K31IEGRWAheXkxNkv81APUMUe8wbhdyj9CWQ4gpw SsTy5z8XYyGNmIQ//MWCgMqEdIBkv/6gysPAHeP272TRj/Hc5I+GiGGGjotRiiDAq662 /ciA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774044; x=1708378844; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DSEwnOfj2VuhpdkIHZ8VNVuKfvBXo1I5xAUr9MF0J1M=; b=CRvoqZOAuNuOKIREGWuJ3Oj+rrP1Aw1s5xIUKA1vZuQ3M40h4hQOktXMm6ZlUpZn3E C2Cvnw4UXeaWw58u9+er0NSOTMC6CnlofMUoj5kiT1pEN91VwWEaoHEGyQ0swF2mqSFR Mjnu3nEjOYf4baa2AjwaBp2ak93eEfmpu+CT5IiSUdAm+kyn7vzyt/kLioe7t8gVygIO 1zvKvD7RLvQXrV9AjxCNMMMvGVst8htwdD/paLQF1eFnfpwDW+K/cqcuWbWCtcz56NGs lwp3fNcYOVnHzZ3l2Q/HvbzVcAJe+6RnO+XEI81HgfctBaC+zF0mTJow9VOHbVkMmfBG UThA== X-Forwarded-Encrypted: i=1; AJvYcCWrz0KWBv5S32bA8ZKZtFAOV10Pkq98eY8JJJteshtvFD6o8YVxMYc4tKe9A4GSy98fbmy9vB3NeAehzeYoB11s1bY= X-Gm-Message-State: AOJu0YwveO+qXRBuSc/m2ollFWhyo+ZnNwYgAe72CEHLfz/tP0u4aRx4 ZT3KZIvbmJwY99SnK1mU8xS6WAFvG5ahUM0HKIY7ZVt0055GrNrtFnv7rFqKoMP0aI7ah42AA95 r8Q== X-Google-Smtp-Source: AGHT+IFtNaKOZcDNkW8O/y0IS6dfwCCkuRAaS6+IZN21h1P/6QI7aHSb74NpkK/DZ6QgX1s1BGxeLh90Utw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a0d:e8c2:0:b0:607:79be:9120 with SMTP id r185-20020a0de8c2000000b0060779be9120mr169543ywe.0.1707774044127; Mon, 12 Feb 2024 13:40:44 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:19 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-34-surenb@google.com> Subject: [PATCH v3 33/35] codetag: debug: mark codetags for reserved pages as empty From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 09E9BA000C X-Stat-Signature: aonjuh4cmy49ppiizhk78j9umybnco8g X-Rspam-User: X-HE-Tag: 1707774044-835676 X-HE-Meta: U2FsdGVkX1/ow9YOQ0jhITfin/a+/z2cLgY5EROCTX1k9bO2tczum3avWK3dJ1yGQsbLS36l3SmtIUczw4X4ksXGaWrHIqP3yU2arztyFegSEgh6L/oR5opVbJ40EkS2yrMS9piil5JobRtLHslmANB+E/3ign9+B5HEuqq7MoNOoaBiDg2ztSvFIBmfgAFDYtFzo+C3MrwHnbItHmTPs/vYBcdtcapTwjrJDlym83kktF3a8oj11kbWtZJgXzhzO33bVwPU5yXKpbn5f6CSWXYhP0a4NCrQf1WwO9oPevxjy6+o4U26UBRhRw5mNGzaZUypUEmAkU964jX0PKh7Du/Ya1nrHXNWVUU9/nJiWuAguVrbK4ktzZrnVevzfAuIvOA0uoequ2W/ufYBbheDApm8n4+Ovd3qQvqQ8fm1xJtxhNZXWOdjCcd6GoNMdRo6SXa9PB4bq6GPgFcyDqFFdofX3s15R8hO9q/kjlWr0N77kNrhY2qOaYyfutDYwSrUsPBgfXEjJXW9egMLnLdOLnzxPqKB+JohP3hEas/FW9N4NOejxjJR/GW0f+eZ9TSk3yqxReZR+Ndt60ElojkETCjv6zcMWNYivM2WrH8fImdtggL7xLKB/333WuwN3ZcwG5NaskKVy/J6SVpo3GzfqaUE8TN3/nnrjIHzS6Xn1kWlAUIoumitDCjAj8oRw5UvUGPNctBe/RiEwsYsCq3KfdiTuGCBUF3NEYywDOXPv/ugVp5KKiFgj34sJt/jHinB/TDb0+3J7simsmL533m3rV1FVG7pGUg3+BUSEdrn7HFtrsbIIXIRsNZ+OWynpEvIyiL5xEnjnY3S4MlAnl0+ZsN4LNMkmjPlpmXnSTe0vIeL9ByckHgjz8m3bQ+qQOPfedxgab+PaqTpZYuSgGUs7J5wecdGOSSNz+gNTc7uQb1vGlp9QFibd3vsQSQ8fEDja/rC/MbYmaOUQAVFfkv I6Gb14Vw 3akkxwQ4t1872Zm39f++bJ6wrthDgDLU5e46eme3tGN52l5rppUz7f/dpsNUL/oE+jLtDzvVYiGl+Lk7hRI7UmWVQunn8zOA2auNYPDVbmzWvyPC6L5fz+Bp1BhdmimyExBrhWA3zJgP3yNF335DGizNaqUTYlQvb+1lpiE7l5dn7lK91KN2s5v1x6fNDsuJt1hZvUhQSrYkrGj8sjd7DEGN5EeovqiPPUlhNxDSECIG/5w/8sZ550xt5Nf6pVKOifLa673PJi8d61Yp9IGDxjJmyabJTvJjZ1lrRJlqeM6HwKq+I9b6wvWvfYAwteK/i1QYSI55OuTZeJMXCaRc+ee9JDjKmWzFecofIRcPcEY84LepAwkw0dPpJB6RBbIDQgDmnpyBnMhmCoim1A8WtiLy7KyMovFYqZs1p1pMBxBjzFeg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To avoid debug warnings while freeing reserved pages which were not allocated with usual allocators, mark their codetags as empty before freeing. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/alloc_tag.h | 2 ++ include/linux/mm.h | 8 ++++++++ include/linux/pgalloc_tag.h | 2 ++ mm/mm_init.c | 9 +++++++++ 4 files changed, 21 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 1f3207097b03..102caf62c2a9 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -95,6 +95,7 @@ static inline void set_codetag_empty(union codetag_ref *ref) #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } +static inline void set_codetag_empty(union codetag_ref *ref) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ @@ -155,6 +156,7 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) {} static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) {} +static inline void set_codetag_empty(union codetag_ref *ref) {} #endif diff --git a/include/linux/mm.h b/include/linux/mm.h index f5a97dec5169..ac1b661987ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -3112,6 +3113,13 @@ extern void reserve_bootmem_region(phys_addr_t start, /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) { + union codetag_ref *ref; + + ref = get_page_tag_ref(page); + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } ClearPageReserved(page); init_page_count(page); __free_page(page); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 0174aff5e871..ae9b0f359264 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -93,6 +93,8 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) #else /* CONFIG_MEM_ALLOC_PROFILING */ +static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } +static inline void put_page_tag_ref(union codetag_ref *ref) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} diff --git a/mm/mm_init.c b/mm/mm_init.c index e9ea2919d02d..f5386632fe86 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2566,6 +2566,7 @@ void __init set_dma_reserve(unsigned long new_dma_reserve) void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { + union codetag_ref *ref; if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT)) { int nid = early_pfn_to_nid(pfn); @@ -2578,6 +2579,14 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn, /* KMSAN will take care of these pages. */ return; } + + /* pages were reserved and not allocated */ + ref = get_page_tag_ref(page); + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } + __free_pages_core(page, order); } From patchwork Mon Feb 12 21:39:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554072 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0373EC4829D for ; Mon, 12 Feb 2024 21:41:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A3726B00BC; Mon, 12 Feb 2024 16:40:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 72B316B00BD; Mon, 12 Feb 2024 16:40:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57B506B00BE; Mon, 12 Feb 2024 16:40:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 440DC6B00BC for ; Mon, 12 Feb 2024 16:40:49 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F089C160231 for ; Mon, 12 Feb 2024 21:40:48 +0000 (UTC) X-FDA: 81784471776.07.0878761 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf11.hostedemail.com (Postfix) with ESMTP id 5A02D4000B for ; Mon, 12 Feb 2024 21:40:47 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=wRHxLdUJ; spf=pass (imf11.hostedemail.com: domain of 3XpDKZQYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3XpDKZQYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774047; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RjVDAYkVgDU7fuXLh0MJZ7V+qIVlVtuXoydpki6LzsY=; b=TNkb6bdRImu7exuHCnj/8a7XBGriIsOPXjH5TNPtTIUAxLF5gSNft9zkzDLBXjQtLLP9OC wdWyHc05EUMXqjbXBIsF0KMKIFcWfaQ/zWCZKoWsN+6Bnp+YjtU0sa1tr4eDxSJX+CI0us LQyuEC5UtHBctIJj1qnNPsIe4rfSzVI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774047; a=rsa-sha256; cv=none; b=jROnxsr0pq8Kur7XhG/q58d1v1JGNP03ArzA0j7WwLylgc9/A28i0I28fVPVOGswUuVL1J c/5d3dMvzrx6+WeaDFe3n5Saua1+H9vLHLNNQUrWLToaQQr9R60rXaxZClOJhJwmPC9TQD YFOyn/RUa9Eka9Iomz8DseRpVWeUAOE= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=wRHxLdUJ; spf=pass (imf11.hostedemail.com: domain of 3XpDKZQYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3XpDKZQYKCN0RTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcbfe1a42a4so1073060276.2 for ; Mon, 12 Feb 2024 13:40:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774046; x=1708378846; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RjVDAYkVgDU7fuXLh0MJZ7V+qIVlVtuXoydpki6LzsY=; b=wRHxLdUJt5T0tzAUhPfudhqt8k1Brq8K8//u/1joLhaDjAkzheaEhtu7aeEEiB62FV 4hkol/uuApVTrkA0DRthOSveuHyj+GZKwazrfLwZUtW9+j1nR3WmfvDANkESJk8ahhn8 8ciwRWkNPIfdUB72P9vB6wjutoKSLqo811K/yK5mSW3ITmjNkix72tTuXRQAZP4qnmnx oSWu4cpQKuvUMaEYIGJjvWF+CMRT9AR2qXboqygRnYptZ58jufr+F/Cn5obD4MDi/m+J UTQn/7vaiJh2Yx0S151MkrEAVlsrFbNTbydHzlq8g61nlg5tqLmJbcXatmgI2p7QewDH O6VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774046; x=1708378846; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RjVDAYkVgDU7fuXLh0MJZ7V+qIVlVtuXoydpki6LzsY=; b=Pjod3NH1tSHOWBZpNI/5zfO2eVgsJjQWskjAaRLB7XqXw99V8J5DrKs+57vWbAhjLq kGT0p4j42g7/gYOxMAEXK/V1KHLBEwlfScMO6Fwc2VM2tHVBI79qCAcCMqlde22lW0js eCy7cXYk1zSElrVyAWdGKGPXuUP0gkRHAfBqi3ByYeExPMxKqIizyx4Z3J67KYFWM0Ew wE64iepCS2j/MbCNZjCvfqkCBG3rAf93qiifJSSScYWtlyCYUZiG/XhaDErmunpb4kfg 16hQCaIxx7/ZdOqERhejm4kKvAeXRnjIqjqQQ5SYWMSZi0cvZnmS60WoLUEb08hhBP0A HrWQ== X-Forwarded-Encrypted: i=1; AJvYcCXk2vB9jEJHfnlmFz2TLv9xppCTgYEORuDO2vg9x2sl6ajVNP58YICbacF/FV5oKOGKMPd5InKetxbs6ubqW0km+Ic= X-Gm-Message-State: AOJu0YyNSs9zaK4w8XBpQx1Fms5LVlT8FT9DZsoTiPsxnNnwAkeD+DCl 5cWEhbAfUKhs2WV8BKxSbPiTSzHvAw4Yfmsc1HpTlJn5555bfgiOTbrpxNangFmqv7iZIlXbIoa MjQ== X-Google-Smtp-Source: AGHT+IESoh+UWKTP+1hXTVbYaUvPuidDpjnUl+NYldJf9cgWEgRiy1LGQkGWPWQFR+H7vya0ViMEzA6ILJE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:150d:b0:dc6:d678:371d with SMTP id q13-20020a056902150d00b00dc6d678371dmr2278885ybu.3.1707774046204; Mon, 12 Feb 2024 13:40:46 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:20 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-35-surenb@google.com> Subject: [PATCH v3 34/35] codetag: debug: introduce OBJEXTS_ALLOC_FAIL to mark failed slab_ext allocations From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 5A02D4000B X-Rspam-User: X-Stat-Signature: jpnss9ugx6bzew8pwhh8byw1y5qoxjt6 X-Rspamd-Server: rspam03 X-HE-Tag: 1707774047-715124 X-HE-Meta: U2FsdGVkX1+GInjq+9woiuDiWDRdiY1e0FZ2uAomd8y6G53bwELXX8gExah0sSAX5557fD8MB/3iyR2I8KeJyp2CqjnsXmRIf4wD8Ze/umJWFdiyCsA2D4O49oI56CizuH5HVhVKk2KJBBsaEhjcjJuAQ/xZCvqT+KbD1PKGAQzy8elgNz8ZCzR03KhTPprOSCdW7cUayqrNuHGOMfXk7MUMRTPf/kCjpjh1bbLZ9cIYkHJRL5LKDHNfycTbnQ1+LLOHL9aw66enU9uXwjGagq6ViT5bvMTwneBdM7xR9tIZ42Pk/daUpuyMjxObLQzRSoY04UYV3syk1wa+M3KdIDftBCF1gFeS93bzEfkBfo8k6/+5Pu89ZfhQBQ5Q2W6GtOPx3D3bH62CC2Sg5E3rzZ0P+QHABd4DZkgvFpfaGWqNXIDcCX9vJMT5Mkv1e/BGCpRq4tV7E/5+3jQSzmbQmcctSXYREuNfHcrp7OZT/TTUm8hxeODoKEnrNs2wJ7sEvTOUa+ehz/h58qWzyAXOFmbT2aZh7VXpEcwAZnwCkQok+qs17gCKz/JFrhmLYCz5wN7XxbscyO1xudqDLR/l3jj3PrDpP1HUHuQom4J6VnbdpGninU4TRhnIjA0IwEJl+V4EzC4k5VRsoLCjTVvsScQeVRGF9MupBruiGp7P/PcLbZ03J4UTxg4ADGGM+CZv3pb+39xPHDWP8wf/CIwj5wc96lY3ks7QWLnImKZmQMpcmXtxzLj0cuerNguEns66AeVXa/Cy/pVHa0UhZVeX+rZsJxpN4DDAA599FtRoZp/sycD8a5zCKU9+2NCtFPtj3M8/ceEFrXT+nv9UwGNKXpqEqOXvOldJiHuryaN0RRBEvh5cW6N1k9oLCj40KjyxlpfZGe3Kxl2LSZVLV2v6BbIyp0YK+FlCgLI/QgyiaD5pnwa2LrrgQse2DBq+rgAxzkDoac6FY1fGSbtkGGf pIuprhRO uQtbS6kzBP78Y0m5ujfU5COc15zHvM5emPrH5Vxy3Vrk1I6cx8U/9iyI+pRy4HeGFGTJEarmqfuLe4n75vd6/LMAtLHkE0XRoGRQL5LxL1ewW1RHiCCcEcsFPqVTMVlIA7l1XPGOq0EUJScFaJ2EW9JhWZTnI2MwutfbX8fLXEL+/fol6KdeJLRT0HNkfzYStm8rp7AyMmoy59m3QOR2pp+aC9A7hg9IQCiP9/ZZLYaarA3Zq1bn0I8F2+jdRRja9Ol/z9X7wmN3fm0KsT5XOVG59Y5hvQHV7rQLvqYUyNFW+h+ohrkMp4+gErvMJQsccP51/4ApPwYlm380DJjkEjmsb+8f5k70XLHeshHpA9sOhSZLFfe1cGT4JA7kV1jWmosi564s5ihqnSwtPwqyNvx1OWnf4ZHOcwW7l2ScXUEck1+qp8WIJg0nIdAYsTGBDbtVdqyWEc/JBwkGgX4av38S6GA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If slabobj_ext vector allocation for a slab object fails and later on it succeeds for another object in the same slab, the slabobj_ext for the original object will be NULL and will be flagged in case when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Mark failed slabobj_ext vector allocations using a new objext_flags flag stored in the lower bits of slab->obj_exts. When new allocation succeeds it marks all tag references in the same slabobj_ext vector as empty to avoid warnings implemented by CONFIG_MEM_ALLOC_PROFILING_DEBUG checks. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 4 +++- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slab_common.c | 22 +++++++++++++++------- 3 files changed, 43 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 2b010316016c..f95241ca9052 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -365,8 +365,10 @@ enum page_memcg_data_flags { #endif /* CONFIG_MEMCG */ enum objext_flags { + /* slabobj_ext vector failed to allocate */ + OBJEXTS_ALLOC_FAIL = __FIRST_OBJEXT_FLAG, /* the next bit after the last actual flag */ - __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, + __NR_OBJEXTS_FLAGS = (__FIRST_OBJEXT_FLAG << 1), }; #define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) diff --git a/mm/slab.h b/mm/slab.h index cf332a839bf4..7bb3900f83ef 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -586,9 +586,34 @@ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) } } +static inline void mark_failed_objexts_alloc(struct slab *slab) +{ + slab->obj_exts = OBJEXTS_ALLOC_FAIL; +} + +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) +{ + /* + * If vector previously failed to allocate then we have live + * objects with no tag reference. Mark all references in this + * vector as empty to avoid warnings later on. + */ + if (obj_exts & OBJEXTS_ALLOC_FAIL) { + unsigned int i; + + for (i = 0; i < objects; i++) + set_codetag_empty(&vec[i].ref); + } +} + + #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} +static inline void mark_failed_objexts_alloc(struct slab *slab) {} +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ diff --git a/mm/slab_common.c b/mm/slab_common.c index d5f75d04ced2..489c7a8ba8f1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -214,29 +214,37 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab) { unsigned int objects = objs_per_slab(s, slab); - unsigned long obj_exts; - void *vec; + unsigned long new_exts; + unsigned long old_exts; + struct slabobj_ext *vec; gfp &= ~OBJCGS_CLEAR_MASK; /* Prevent recursive extension vector allocation */ gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); - if (!vec) + if (!vec) { + /* Mark vectors which failed to allocate */ + if (new_slab) + mark_failed_objexts_alloc(slab); + return -ENOMEM; + } - obj_exts = (unsigned long)vec; + new_exts = (unsigned long)vec; #ifdef CONFIG_MEMCG - obj_exts |= MEMCG_DATA_OBJEXTS; + new_exts |= MEMCG_DATA_OBJEXTS; #endif + old_exts = slab->obj_exts; + handle_failed_objexts_alloc(old_exts, vec, objects); if (new_slab) { /* * If the slab is brand new and nobody can yet access its * obj_exts, no synchronization is required and obj_exts can * be simply assigned. */ - slab->obj_exts = obj_exts; - } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + slab->obj_exts = new_exts; + } else if (cmpxchg(&slab->obj_exts, old_exts, new_exts) != old_exts) { /* * If the slab is already in use, somebody can allocate and * assign slabobj_exts in parallel. In this case the existing From patchwork Mon Feb 12 21:39:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554073 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38A15C4829E for ; Mon, 12 Feb 2024 21:41:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 508CD6B00BE; Mon, 12 Feb 2024 16:40:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 490C26B00BF; Mon, 12 Feb 2024 16:40:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26CC66B00C0; Mon, 12 Feb 2024 16:40:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 144FD6B00BE for ; Mon, 12 Feb 2024 16:40:52 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id EAB461A02DD for ; Mon, 12 Feb 2024 21:40:51 +0000 (UTC) X-FDA: 81784471902.16.A2E00E3 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf02.hostedemail.com (Postfix) with ESMTP id 43E4780005 for ; Mon, 12 Feb 2024 21:40:49 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="YyKCZr/N"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3YJDKZQYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3YJDKZQYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774049; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SBX12uj3ZhdKUrlA/PpkAMdvyZt1FRyTju3ec31OSZc=; b=EPRlGhJ9pByjMuBNupWLbK9O9mN3BG/bTDyy1uHXRMZseNIXoKJD2BkBSJmHUzfijInTXh HKTUpKayIXMYAcO+Lm73VXNQzrjvlJWYhUp9eUbkGe9EDHHHcnV63T4Zdp9nuOm8OHTQ0O RCb7BnB2cm1FPD/14/tLifJXBBYGfLg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="YyKCZr/N"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3YJDKZQYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3YJDKZQYKCN8TVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774049; a=rsa-sha256; cv=none; b=JF1ugpObei/uiYPoXULnU4pHux3djB5Mhmjw8pAS+x7wI9sYnvnT3Ws/NJmvRuZibY/wp9 JLEQJIInBRXf3yNYxsZaaUObMG1XIQFKDQ4d7ry13HOXqc5d8Aj+yaR6qxntem2I4T8mQ2 jOMwQMFgxrs4kvILonwkZxdDo+xkjZA= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6077e501cecso5399397b3.2 for ; Mon, 12 Feb 2024 13:40:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774048; x=1708378848; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SBX12uj3ZhdKUrlA/PpkAMdvyZt1FRyTju3ec31OSZc=; b=YyKCZr/N6szOKiT52xOmpJsM//U19waSGPW1+JidQmfPdyMYK8XxCwzjyXjRize577 SFEaQJlK3qpyfR0bgQBgMNl7rJmAIflq1ySgzS8XHuaizk0qY17AXMvirJr4kNzhd+SZ bvJwjMFyOs9pnPuUstt6B/ede5zwRH3Qbpa+/ZlGwoyniL5hnTx+e91pUlu/LiBgQjWK 8+OLNmy9SxEkb7gkByA8lvmuiLS7VxuIlDc/BmBSoShYBoVc5yOjt8OHAckn6cmqKcU5 kA6h9U3a++TYTkuRuOf1ltMOd68XS3vOhmuOrmmQqksgODZSu5S3R7bUsIgMhz1FgyeT Obzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774048; x=1708378848; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SBX12uj3ZhdKUrlA/PpkAMdvyZt1FRyTju3ec31OSZc=; b=oGEyRNi8nXqu1u4fk9Sx8qfPOMQecCgsxNAmlCvNsy3V7vlwNUv6toAqm8eeaiKwJZ YbKFFZgomr/hsWqWGzcQMcr5N9IXwXauHbXSUpzVXBF0bMRZX8aaZoZsa7c8aOLC1O+X EwwjaWIsrW62cpbjNCSJz8FUJnoI8LjOLXXwRswBxHZI82PNwf5d+Qgct2AKKhu5gywO dfVtGfxEDpBAJaFIj+4lHOhtGUh1AvoZurj9+jIX80ELKIrBTZMGvFVSUzs1a0pifpmB sWMpLB9JTQogGhY2tVZHlDkDqfSGBne/aKmtTrcsHXnwr0Nh7X2ybh4njV9Gz6RBxBfs UFaQ== X-Gm-Message-State: AOJu0Yy2T0bFgY9aHzzuAnorTCf1lYLhEdjc0UAEYPfghW/GeMoidb7t krzrSdhGmG7PR3maNhZEUX/jBiJt+fsgeGmBpMzLpIJ5jCBFazdIANl69ycNa6X1XIepukKsbaf E8A== X-Google-Smtp-Source: AGHT+IGnimMvKfHnjDqQ+fcufl2ID9NuWKv0TEQmpUKb7IHMbluonmQIQ8q+5RdznzBwDieXP2FUkEOc23U= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:690c:ec9:b0:604:648:6dc0 with SMTP id cs9-20020a05690c0ec900b0060406486dc0mr2354569ywb.10.1707774048308; Mon, 12 Feb 2024 13:40:48 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:21 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-36-surenb@google.com> Subject: [PATCH v3 35/35] MAINTAINERS: Add entries for code tagging and memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 43E4780005 X-Stat-Signature: wr1hjshwp8qr6fqqo5kddrjhbrpnurcq X-HE-Tag: 1707774049-80909 X-HE-Meta: U2FsdGVkX1/bgeGBk4DknUy+bGMW9HHqmXv1ugbsdNLH645zYFomNnNgJdcC2iWxoS+JdqUZdnqAsxFe0WU/hGKdq6S4OWnJNkzcfoIhOX4c7rVoCABWLxsX/bJfQME1FTY0xug6wO+fGZStnqe2Vn6277rWRNXp1fX0GsHUYcMFgGx+H+OMm8tMMfMQqTq/d96N/2sfMtrWtyr8MLxC4QRLPGNnu66wqspMajwVjg8lgKy+6TuzYhMWikgFVE+7HPnGwNqLqGmfbtlVKlYgimPfKbo/xi1OOlDyJgnf+dljKLLzATxTrRAhEbPYV0lBPSpu4ydeQ9HwHucc0Vn28pQkAqni69rQE2F91Uv++2MM6sfLeR9YmaRet4RTThvpbb18f6mbkqoJZGu0os8MGiXomdZt5rwVUkyjY3D2u3qre7Q3HVawMMGAlSKSc3Ou8KMOUleNtZESB/C8Mr4zLqII9TtCaaC5y2hWdhaGsR9QUfPq8BbDlDz1sxlPdS6ggRsIncipOpNqZPbW/fPSWfJRUvypyw9/lPmeryVq86mx9/5mesJwS2VV0WC08mLthavf5M4CY2Dy5cbllmqTC97fKFIstW5+B+s7DtygxsXSCQxlAy1U+0WCSYo5YVDm6eYzF7ZuKgWQgiuLAvPiWMPzDB6STomOWy4bLqJxUvwGvITi+B+kHPCpgNlDtu3iVkwP1A2MMds1vdu7PfmeYtw+Hj/qyG69d9esMkMcbKcQXmnlpvzNtVAWUd1qaGifATKmoCpFQgXfVoKaPRq8Gy2KxqnLu+dqozlOIcJI1ir561Hdv829D8pSAtIj5kyPgNSsFedipI+I2p/VqagQHmEKD2btYvqMHyNzuTU9OvimQjXhfJyAnIFC53ES6bBpBikNPLc3CenqPOXfUeLh2ibUERPVxh7UM38hTLLX5WcuHephJYAk6wnN9fBPQB3RE4eX5NlWgVSc3fXJ1qW nrx3171B UhD9acEzdX3NWIC/+uoAc9zMUx6tntcWNXZeuL2hE/RsR1ckFxBkipquvSB5kG1Uk4hj2ZWNNV9EHGf69iyZgmu0GObJTICKUhLDOfHjY6QHOz9qV/JJXfdNdgrPMA09wTwX/J5eDsEDXHV+k2Qn1hEHVdatahST5APIrC5ANe64xB4Eyeff7pGaGOIDtFQy8h62z+Mjd4d2AwLL+2bWTxkOin7Yj9A3n3VKZAgExFZIb/oja1Ehmg0183rxxWe+4NyghCYarU1U2HlcELlKmPuT58QLblXU+P9aDffgZCnthIhR/Tf+S8jVUU511JKdyMnGZf9IjE78XTxmLkYx9I9JSuSq8xdvP/ithQilq3xohR6Yq3LbWHZ/Xly9I6YSkintG/40KxshuzVCRIJHTZqc4qwAyzCVW3ckL9VbQhpxZ4r4E2RrpNniuHf/lXrJUK7wTLOXzGRT/KVIcX83YG3QgtsUQlSV60wAQ2Euq79+qpt5x2I71tyGc+DY5r/orjiAOJCV30uP7ovgcFO7KoC8IcsuiT+AfQeLzztJDIEYe0BNkZAMx9DPWZ0fxjDF+sDApijJ77Q3FH8rweg7XybueMLOos7VIgHkdmXQRifHbjQK+MNaueOjEyISi4mz8Z1P1d+l2c1b3YeeaxHnhbrWURY+6chp2E1Y3rWtQROfLO3fBv53sTspwDB1vr45YL/mNsI8XYpl9nK9e6lSeiUY5lfizvtUhibkAJrqXQy/Aj1rDf4b3hp6fmCmjm/Ms4rD/wy2wfI2V7aYIcIIidBopcpmcy5aTtlhk51sJGgPr0VepjPES1qUaRtbvKJ78UfHGcPb3CoU2A0GWH8/kd+DsjImFEnOAqNb+H/s3h8WiN04/CmuFXEvu2LyAu4jfys+fsOJjrSX07IcDTHoz5u36T+cFDlUa8hVB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The new code & libraries added are being maintained - mark them as such. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- MAINTAINERS | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 73d898383e51..6da139418775 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5210,6 +5210,13 @@ S: Supported F: Documentation/process/code-of-conduct-interpretation.rst F: Documentation/process/code-of-conduct.rst +CODE TAGGING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/codetag.h +F: lib/codetag.c + COMEDI DRIVERS M: Ian Abbott M: H Hartley Sweeten @@ -14056,6 +14063,15 @@ F: mm/memblock.c F: mm/mm_init.c F: tools/testing/memblock/ +MEMORY ALLOCATION PROFILING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/alloc_tag.h +F: include/linux/codetag_ctx.h +F: lib/alloc_tag.c +F: lib/pgalloc_tag.c + MEMORY CONTROLLER DRIVERS M: Krzysztof Kozlowski L: linux-kernel@vger.kernel.org