From patchwork Tue Oct 24 13:45:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0DCDC25B6C for ; Tue, 24 Oct 2023 13:46:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6FE8B6B0254; Tue, 24 Oct 2023 09:46:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 686846B0255; Tue, 24 Oct 2023 09:46:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D9ED6B0256; Tue, 24 Oct 2023 09:46:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2C9DB6B0254 for ; Tue, 24 Oct 2023 09:46:47 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ED76012030D for ; Tue, 24 Oct 2023 13:46:46 +0000 (UTC) X-FDA: 81380480412.16.A4FBC14 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf02.hostedemail.com (Postfix) with ESMTP id F3A3080019 for ; Tue, 24 Oct 2023 13:46:44 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gtCqJx+B; spf=pass (imf02.hostedemail.com: domain of 3w8o3ZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3w8o3ZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155205; a=rsa-sha256; cv=none; b=EszlQ6d33kGo6AyjhgqG1HADAuHDALSqsmY1U/SAOwF1B5s8Iwh1zbJPoKmjBOtZ8H8bu6 9G8hQgYyhfEGQISaFOHuKwUbal7h+TiYxV1rKi2YaVp0pu8XYMTe4hNMjP7/k5r7JYKFGv FSk4rL96ZHzCLhTTiVu9tkDMX34ujeY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gtCqJx+B; spf=pass (imf02.hostedemail.com: domain of 3w8o3ZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3w8o3ZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C4ig+ghmPhQV+NN9gQrUnAhQK6p3EzwOX4/nD/QGc34=; b=2gAgbeehXEDG0+QVpMqNcGCsMFRxyxpkv7bMnk4r9eYXIppQEwmWBLtBjS2qSKCgI7EQtT xfiYOL1e7UOe+hJuahKRojktOc1DvkyqFwoqnSp4iNipwwJ3B2fvb5e8O4Kjpj/xL9rwyQ lYpnwinRtpi26sSUt76ay8UkVCeCsiI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7af69a4baso59606037b3.0 for ; Tue, 24 Oct 2023 06:46:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155204; x=1698760004; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=C4ig+ghmPhQV+NN9gQrUnAhQK6p3EzwOX4/nD/QGc34=; b=gtCqJx+B5A4SJHkuOTdOV6ver2DjCd0oTV6rOa/47rJswVqnGEGNUpOfe7ylsFZLb1 mWn+rGndFCrlNSbrmh8K4G7FMW4lOsxoJTFQHvy7ink2hc/3cdDpAnbXEKhnKVziJNlQ MIShsj03vBDO8rKcr8zFCGJdeRIyXJ6JUnvSe4ROg2bHTG+gcmyR+yu1Tet5nqr79Rdr RGlU2yg05HEfQqPuyxeZe0gO1lk56wGMPbOAVhZZ2qkk8mTD0oZRpDGjftxB2l8gLhC2 gb2jycYNWC6f/i5WpswqdM4mcxiW44XZy/hB4cGXF7RkAN+Xkk8vbhFRmfeqVJXt2l05 8kRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155204; x=1698760004; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=C4ig+ghmPhQV+NN9gQrUnAhQK6p3EzwOX4/nD/QGc34=; b=jp4ABf7w37M3FDeJqm2HVRNVlDhkg9Xu0KrCYwWkbwMkoaz8VJe7/4INVf8rOJXeRD BXlKw/x1KtYiS2BPfnOCogmOVjcNLO0U0MItZx4p5Jvs0ZybQ7F7op0vXy3we+fTJR5S u4jkAehUrMzI5iwxqkFYZRrrU8nyLZsbOZ7FfiyrN10fN5z/bXp42XyS2w+o5AKttW/j Bupniyj3BJsiDnjeZlm7JthQ7RXUPOV+yNldMOFhfztTH4bjd4mo2nj5uc2oROFG4CQM 2IAPw3pSIMOCTSvbPk5JPE4825WHwccGIWWNGbkbDk2xDwdKjCNc/d/RidpOTa0bYMCa j3GQ== X-Gm-Message-State: AOJu0YwYduJJOV7fhxD5hMKJgMEYfUDz0aDnr207n50ZeVeaXTuxO+OG Eu4l3txQMxkrbai2KpDe+bIXK2/eCPk= X-Google-Smtp-Source: AGHT+IH/oTVJ1Mu9lou4puXON+M9/Y6l8UTmlhuGUIs6R4EkfIBTFxk7sH9NDcndoJzEgmocQcXHiryGytM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:d7d8:0:b0:d9a:da03:97e8 with SMTP id o207-20020a25d7d8000000b00d9ada0397e8mr212597ybg.2.1698155203948; Tue, 24 Oct 2023 06:46:43 -0700 (PDT) Date: Tue, 24 Oct 2023 06:45:58 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-2-surenb@google.com> Subject: [PATCH v2 01/39] lib/string_helpers: Add flags param to string_get_size() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Andy Shevchenko , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , "Michael S. Tsirkin" , Jason Wang , " =?utf-8?q?Noralf_Tr=C3=B8nnes?= " X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: F3A3080019 X-Stat-Signature: b48hfa7mw9gqkzznyu87cw7tpm1uyqim X-Rspam-User: X-HE-Tag: 1698155204-97687 X-HE-Meta: U2FsdGVkX18xh6VGze4nwfxIJVxnnKwyFRLVWs1urpB6vN/rcScb7Nl755PTQ8tPIsS84I3/ttBWUjiCeSNOIP4xLN2uRGZyBbges1XNKty42VXCGIogivuML8Zph1/eeMUnQXz2kgzRY5nvyHy6xKPk3XY5nqFS4eAYPBveUMljuOXNzwM0gnUdNBxOEl66c8yr3wnW3dgfsdYxYcNznsHEhNiHczTypkKe7fTeMlpfxM9wwPcWKhElIuirr7OANlPOUYaVaexRnNF6VxXUjx3d6OH7dAMtVNPL7oNKnrn59B+ENpN8OoQP9xvn01uvWjHZ+/bEgE6+zhnl9KphmCMUVf3rC2FaO402kZSXsfAHnv08MhHpPzgnjRm3wpcdK8/R5p6BFpVcm41LbD3LuUNEviY5+zAaWU6ot/3xnGiyM/bzS15sYkJR2TgO5jA8+diWDMe2VijFfuQHEBTeZ7C+FsgMMCsd4l2rSHZEqIYKIJtcoUzu7d97XLm1DWWfsvQGkm3VewQ6zhXRjmA+esIj/NTYtpK2cUozkYonlbpbEVwBiGJSFjbgU2xJKVKqQCWAgK22PJXTZOztUO+qYqE5M8tpqC/SuNzTBpAtcV2SrbeHzr6C3ZchGDE9SrP7yXIjjc3KHwKmhyqjjuAP7ErO1K31jaDQwF2k+PXw6Pg/RWsA5Qbw8g4MxAqs7MC0SU7oI9noT8TcKI1ATaHxsXUrvniBOELNVflp10YBA5mKgxWEWwlkgHBW/R2cApzYGOMo8Es7Kv+GzpnbolgeTrvTuIrXIYsNeYSDyp1kawsdXMTJRv8nXLUBhEOe9gO7e7uAHXeGBpqjXU3WwoYt87VpQuwIQeg45aUVexjq0/9vOC/ZI3yREHD9/8cCnKqK2hU+Tyu+zA+mWvGrqMOtXASEPTXj52roxG9fZWiFqWHEsZnStkuY/5G9xR7rZ4/WU8LFrRiArTuJ3wnCv3X uF8KNk6G iFLBe96fBeoDFMYUGJN3SLt2LhwEa0HOFN+ahXaBUXW2oQ3wqJzBWt7clF9mPabUDPE12Yr0wGVu10k7EyIrBmv6V4wY29x1ZP4Fw3WTDSUaaT5OokaPhLwzkSg9HW5AnHQKKas9yGwmhQUK9zYOstDAz1wyMjN8G4IhpSl6/eA/nnfpbiYu1OM7Gw8lKbbOZQVaaSQabD4SrvSWujmO5RZE82rUzZVfqL9sDK1CECSFu+JfFAq7OafTytY4EU/14DLGYjtbnx1BVgjh9lg+TzR9jPvzvojoKYBbi/TKGJzuPc4nSQvey7ifSdyfbkQRYj2PGEV+4OxMjVNRIe0l9rGGo0K4sTcwMiDSh+FZXpcEiqE34I9uuigt9JR5vkitEhhWytdM4awYeHjlI6f6OM99cv84Zmj9p96TjU+tIF2iaakTybH0xrzoFDNvKJvY4PnyxzBXAkyeCmRg9T4FQIa7pN/igBcioGdUygJThCstpHquQtis9oLyREjTvCpORCAWIGXGgiXySzyYeaxH60DTfDh4LlGKWh9oWAiRCAEukQVGZJyQa3hc8FQAKRmqVV6wHSFM1jRreaGfPrWqn7vskeOOn2pc/oZHn+yoUyu+FBtrB1OLezT8i7rm/8FIkkplVr2WJ5T2WKFSM84irmWu10mNI3M2FOMvW6fgmY5T9w4lSRPR6dwUASLY0/KfRxk/VzXkHxKfsGrHbb0pWZdjzyzM7aiTbD5+lXDGYd/vNstWNxKqkL+TDC9aiLYUxBlP4qFG0VQOjS+/NyERQUJgmEEeBlHkE/9v/sVQZFuZwJ6n5IWI5CjM+hhJzxYpLrupiF5B2JGFTURzMish51DYE7hlJDrT1O8+JJ2xYqhVEkEIdm9w5AzKEP3ZlGhGqFSo9rEumpseQcoHW5Ts/MRZND1QR1cLdlroqthQhxhXsy4+ncHBIZmb3u8QQk6Qw2K7vD1Ys29sEy4W+ouO6Qwyv8QAj ZHdunwdb XdE1M+ElXCw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The new flags parameter allows controlling - Whether or not the units suffix is separated by a space, for compatibility with sort -h - Whether or not to append a B suffix - we're not always printing bytes. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andy Shevchenko Cc: Michael Ellerman Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: "Michael S. Tsirkin" Cc: Jason Wang Cc: "Noralf Trønnes" Cc: Jens Axboe --- arch/powerpc/mm/book3s64/radix_pgtable.c | 2 +- drivers/block/virtio_blk.c | 4 ++-- drivers/gpu/drm/gud/gud_drv.c | 2 +- drivers/mmc/core/block.c | 4 ++-- drivers/mtd/spi-nor/debugfs.c | 6 ++--- .../ethernet/chelsio/cxgb4/cxgb4_debugfs.c | 4 ++-- drivers/scsi/sd.c | 8 +++---- include/linux/string_helpers.h | 13 +++++----- lib/string_helpers.c | 24 +++++++++++++------ lib/test-string_helpers.c | 4 ++-- mm/hugetlb.c | 8 +++---- 11 files changed, 44 insertions(+), 35 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index c6a4ac766b2b..27aa5a083ff0 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -260,7 +260,7 @@ print_mapping(unsigned long start, unsigned long end, unsigned long size, bool e if (end <= start) return; - string_get_size(size, 1, STRING_UNITS_2, buf, sizeof(buf)); + string_get_size(size, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); pr_info("Mapped 0x%016lx-0x%016lx with %s pages%s\n", start, end, buf, exec ? " (exec)" : ""); diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 1fe011676d07..59140424d755 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -986,9 +986,9 @@ static void virtblk_update_capacity(struct virtio_blk *vblk, bool resize) nblocks = DIV_ROUND_UP_ULL(capacity, queue_logical_block_size(q) >> 9); string_get_size(nblocks, queue_logical_block_size(q), - STRING_UNITS_2, cap_str_2, sizeof(cap_str_2)); + STRING_SIZE_BASE2, cap_str_2, sizeof(cap_str_2)); string_get_size(nblocks, queue_logical_block_size(q), - STRING_UNITS_10, cap_str_10, sizeof(cap_str_10)); + 0, cap_str_10, sizeof(cap_str_10)); dev_notice(&vdev->dev, "[%s] %s%llu %d-byte logical blocks (%s/%s)\n", diff --git a/drivers/gpu/drm/gud/gud_drv.c b/drivers/gpu/drm/gud/gud_drv.c index 9d7bf8ee45f1..6b1748e1f666 100644 --- a/drivers/gpu/drm/gud/gud_drv.c +++ b/drivers/gpu/drm/gud/gud_drv.c @@ -329,7 +329,7 @@ static int gud_stats_debugfs(struct seq_file *m, void *data) struct gud_device *gdrm = to_gud_device(entry->dev); char buf[10]; - string_get_size(gdrm->bulk_len, 1, STRING_UNITS_2, buf, sizeof(buf)); + string_get_size(gdrm->bulk_len, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); seq_printf(m, "Max buffer size: %s\n", buf); seq_printf(m, "Number of errors: %u\n", gdrm->stats_num_errors); diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c index 3a8f27c3e310..411dc8137f7c 100644 --- a/drivers/mmc/core/block.c +++ b/drivers/mmc/core/block.c @@ -2511,7 +2511,7 @@ static struct mmc_blk_data *mmc_blk_alloc_req(struct mmc_card *card, blk_queue_write_cache(md->queue.queue, cache_enabled, fua_enabled); - string_get_size((u64)size, 512, STRING_UNITS_2, + string_get_size((u64)size, 512, STRING_SIZE_BASE2, cap_str, sizeof(cap_str)); pr_info("%s: %s %s %s%s\n", md->disk->disk_name, mmc_card_id(card), mmc_card_name(card), @@ -2707,7 +2707,7 @@ static int mmc_blk_alloc_rpmb_part(struct mmc_card *card, list_add(&rpmb->node, &md->rpmbs); - string_get_size((u64)size, 512, STRING_UNITS_2, + string_get_size((u64)size, 512, STRING_SIZE_BASE2, cap_str, sizeof(cap_str)); pr_info("%s: %s %s %s, chardev (%d:%d)\n", diff --git a/drivers/mtd/spi-nor/debugfs.c b/drivers/mtd/spi-nor/debugfs.c index 6e163cb5b478..a1b61938fee2 100644 --- a/drivers/mtd/spi-nor/debugfs.c +++ b/drivers/mtd/spi-nor/debugfs.c @@ -85,7 +85,7 @@ static int spi_nor_params_show(struct seq_file *s, void *data) seq_printf(s, "name\t\t%s\n", info->name); seq_printf(s, "id\t\t%*ph\n", SPI_NOR_MAX_ID_LEN, nor->id); - string_get_size(params->size, 1, STRING_UNITS_2, buf, sizeof(buf)); + string_get_size(params->size, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); seq_printf(s, "size\t\t%s\n", buf); seq_printf(s, "write size\t%u\n", params->writesize); seq_printf(s, "page size\t%u\n", params->page_size); @@ -130,14 +130,14 @@ static int spi_nor_params_show(struct seq_file *s, void *data) struct spi_nor_erase_type *et = &erase_map->erase_type[i]; if (et->size) { - string_get_size(et->size, 1, STRING_UNITS_2, buf, + string_get_size(et->size, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); seq_printf(s, " %02x (%s) [%d]\n", et->opcode, buf, i); } } if (!(nor->flags & SNOR_F_NO_OP_CHIP_ERASE)) { - string_get_size(params->size, 1, STRING_UNITS_2, buf, sizeof(buf)); + string_get_size(params->size, 1, STRING_SIZE_BASE2, buf, sizeof(buf)); seq_printf(s, " %02x (%s)\n", SPINOR_OP_CHIP_ERASE, buf); } diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_debugfs.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_debugfs.c index 14e0d989c3ba..7d5fbebd36fc 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_debugfs.c +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_debugfs.c @@ -3457,8 +3457,8 @@ static void mem_region_show(struct seq_file *seq, const char *name, { char buf[40]; - string_get_size((u64)to - from + 1, 1, STRING_UNITS_2, buf, - sizeof(buf)); + string_get_size((u64)to - from + 1, 1, STRING_SIZE_BASE2, + buf, sizeof(buf)); seq_printf(seq, "%-15s %#x-%#x [%s]\n", name, from, to, buf); } diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 83b6a3f3863b..c37593f76b65 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -2689,10 +2689,10 @@ sd_print_capacity(struct scsi_disk *sdkp, if (!sdkp->first_scan && old_capacity == sdkp->capacity) return; - string_get_size(sdkp->capacity, sector_size, - STRING_UNITS_2, cap_str_2, sizeof(cap_str_2)); - string_get_size(sdkp->capacity, sector_size, - STRING_UNITS_10, cap_str_10, sizeof(cap_str_10)); + string_get_size(sdkp->capacity, sector_size, STRING_SIZE_BASE2, + cap_str_2, sizeof(cap_str_2)); + string_get_size(sdkp->capacity, sector_size, 0, + cap_str_10, sizeof(cap_str_10)); sd_printk(KERN_NOTICE, sdkp, "%llu %d-byte logical blocks: (%s/%s)\n", diff --git a/include/linux/string_helpers.h b/include/linux/string_helpers.h index 9d1f5bb74dd5..a54467d891db 100644 --- a/include/linux/string_helpers.h +++ b/include/linux/string_helpers.h @@ -17,15 +17,14 @@ static inline bool string_is_terminated(const char *s, int len) return memchr(s, '\0', len) ? true : false; } -/* Descriptions of the types of units to - * print in */ -enum string_size_units { - STRING_UNITS_10, /* use powers of 10^3 (standard SI) */ - STRING_UNITS_2, /* use binary powers of 2^10 */ +enum string_size_flags { + STRING_SIZE_BASE2 = (1 << 0), + STRING_SIZE_NOSPACE = (1 << 1), + STRING_SIZE_NOBYTES = (1 << 2), }; -void string_get_size(u64 size, u64 blk_size, enum string_size_units units, - char *buf, int len); +int string_get_size(u64 size, u64 blk_size, enum string_size_flags flags, + char *buf, int len); int parse_int_array_user(const char __user *from, size_t count, int **array); diff --git a/lib/string_helpers.c b/lib/string_helpers.c index 9982344cca34..b1496499b113 100644 --- a/lib/string_helpers.c +++ b/lib/string_helpers.c @@ -19,11 +19,17 @@ #include #include +enum string_size_units { + STRING_UNITS_10, /* use powers of 10^3 (standard SI) */ + STRING_UNITS_2, /* use binary powers of 2^10 */ +}; + /** * string_get_size - get the size in the specified units * @size: The size to be converted in blocks * @blk_size: Size of the block (use 1 for size in bytes) - * @units: units to use (powers of 1000 or 1024) + * @flags: units to use (powers of 1000 or 1024), whether to include space + * separator * @buf: buffer to format to * @len: length of buffer * @@ -32,14 +38,16 @@ * at least 9 bytes and will always be zero terminated. * */ -void string_get_size(u64 size, u64 blk_size, const enum string_size_units units, - char *buf, int len) +int string_get_size(u64 size, u64 blk_size, enum string_size_flags flags, + char *buf, int len) { + enum string_size_units units = flags & flags & STRING_SIZE_BASE2 + ? STRING_UNITS_2 : STRING_UNITS_10; static const char *const units_10[] = { - "B", "kB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB" + "", "k", "M", "G", "T", "P", "E", "Z", "Y" }; static const char *const units_2[] = { - "B", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB", "ZiB", "YiB" + "", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi", "Yi" }; static const char *const *const units_str[] = { [STRING_UNITS_10] = units_10, @@ -126,8 +134,10 @@ void string_get_size(u64 size, u64 blk_size, const enum string_size_units units, else unit = units_str[units][i]; - snprintf(buf, len, "%u%s %s", (u32)size, - tmp, unit); + return snprintf(buf, len, "%u%s%s%s%s", (u32)size, tmp, + (flags & STRING_SIZE_NOSPACE) ? "" : " ", + unit, + (flags & STRING_SIZE_NOBYTES) ? "" : "B"); } EXPORT_SYMBOL(string_get_size); diff --git a/lib/test-string_helpers.c b/lib/test-string_helpers.c index 9a68849a5d55..0b01ffca96fb 100644 --- a/lib/test-string_helpers.c +++ b/lib/test-string_helpers.c @@ -507,8 +507,8 @@ static __init void __test_string_get_size(const u64 size, const u64 blk_size, char buf10[string_get_size_maxbuf]; char buf2[string_get_size_maxbuf]; - string_get_size(size, blk_size, STRING_UNITS_10, buf10, sizeof(buf10)); - string_get_size(size, blk_size, STRING_UNITS_2, buf2, sizeof(buf2)); + string_get_size(size, blk_size, 0, buf10, sizeof(buf10)); + string_get_size(size, blk_size, STRING_SIZE_BASE2, buf2, sizeof(buf2)); test_string_get_size_check("STRING_UNITS_10", exp_result10, buf10, size, blk_size); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 52d26072dfda..37f2148d3b9c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3228,7 +3228,7 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) if (i == h->max_huge_pages_node[nid]) return; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_SIZE_BASE2, buf, 32); pr_warn("HugeTLB: allocating %u of page size %s failed node%d. Only allocated %lu hugepages.\n", h->max_huge_pages_node[nid], buf, nid, i); h->max_huge_pages -= (h->max_huge_pages_node[nid] - i); @@ -3290,7 +3290,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) if (i < h->max_huge_pages) { char buf[32]; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_SIZE_BASE2, buf, 32); pr_warn("HugeTLB: allocating %lu of page size %s failed. Only allocated %lu hugepages.\n", h->max_huge_pages, buf, i); h->max_huge_pages = i; @@ -3336,7 +3336,7 @@ static void __init report_hugepages(void) for_each_hstate(h) { char buf[32]; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); + string_get_size(huge_page_size(h), 1, STRING_SIZE_BASE2, buf, 32); pr_info("HugeTLB: registered %s page size, pre-allocated %ld pages\n", buf, h->free_huge_pages); pr_info("HugeTLB: %d KiB vmemmap can be freed for a %s page\n", @@ -4227,7 +4227,7 @@ static int __init hugetlb_init(void) char buf[32]; string_get_size(huge_page_size(&default_hstate), - 1, STRING_UNITS_2, buf, 32); + 1, STRING_SIZE_BASE2, buf, 32); pr_warn("HugeTLB: Ignoring hugepages=%lu associated with %s page size\n", default_hstate.max_huge_pages, buf); pr_warn("HugeTLB: Using hugepages=%lu for number of default huge pages\n", From patchwork Tue Oct 24 13:45:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34A3DC25B6D for ; Tue, 24 Oct 2023 13:46:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E6456B0256; Tue, 24 Oct 2023 09:46:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9463B6B0257; Tue, 24 Oct 2023 09:46:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7996A6B0258; Tue, 24 Oct 2023 09:46:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4D9956B0256 for ; Tue, 24 Oct 2023 09:46:49 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 223B8B5DA5 for ; Tue, 24 Oct 2023 13:46:49 +0000 (UTC) X-FDA: 81380480538.25.DA65B13 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 3DF69A0010 for ; Tue, 24 Oct 2023 13:46:47 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Pljd7L7m; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3xco3ZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3xco3ZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155207; a=rsa-sha256; cv=none; b=KtJGkCH0uo2GRO+TtJN5DADADPmQdgL4hMmf17jLDtHEuxkx71BSG7d8Y2xEPF8OaqteZ8 iPQaYwYHOg5GUWUD8+NcgLmPNCAwajBbAexwX4Yi18O+o2UxEZi2rBYgWFFYfwOfKXGJeo DnUCU+WT5IYNdLAHUGJn9ZeTeVkggys= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Pljd7L7m; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3xco3ZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3xco3ZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155207; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fObhDXE+DfK48fbfXMStFHtHAmn90M4M91A2l4CxzLM=; b=HlSzwbWzmo8TVdDVJVlUJJliYXMXOZR6onV/5q6ntVOcz9RhkGC1jdd3Ek4+hge3mCgKPN ijGFmMaE4QQXfdJSdj8taGVXxlPu0El2jpKL6TYiaOS3+shQBEMU74K+f0DFmhj4+v5hu1 g8KSOXYdRTqlp0Fms2N6nUydcJ1nZm0= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9cad450d5fso5174017276.1 for ; Tue, 24 Oct 2023 06:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155206; x=1698760006; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fObhDXE+DfK48fbfXMStFHtHAmn90M4M91A2l4CxzLM=; b=Pljd7L7mhLJQIfZuMGinpXlQx5kgBVHjLuGFlr85EfFkaiXNLqf0VlgxiALKYS48R0 dJkehTF76N0/CqDVs8ZsYV593Nn5+Dwanb8AAjQn7eYhstvSlgQ2IYXfQAM/x4xNHgIn R0bCzYdIUs/cV3h/61p4NX7qJ1semc6YHlsBfBb7U3LvIi/blyjPn3JNiCEwTzWyZb61 daNqqxjjeVUi4uoplmeAdZSCm9lNPbe2KGTWcG8Q2TEhi/wnFrjPDNJmiNP8iMrs/rlc Uc3X7JZRK3h/FHF3COU8hY36TCD+c0lrA9s8zGW6KqlBUov1PCnoME7Z8S1PNTsdn4dH Y4mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155206; x=1698760006; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fObhDXE+DfK48fbfXMStFHtHAmn90M4M91A2l4CxzLM=; b=db8oEDOaw+uKwaOMqpM1EsvwIubaoTLuTOEVfKsyNsqhNzqPADEcjAlOHaYXmq31RY 7Q5DzmdWnetxxqi7pxIYeI93YERHl47GUY6vHWkYGcmKu+w6YTnDaogrEAifkrChl4+s 0xnifUc52xyhoAk0COAkTXgciIM2hwTkksFagD8OqyAGhDQxZOZBGGLjtuXYDW3p5bsa b40Y26+jMEwqKAfiRQ1koukioz0lVMlWsbNWm674/0oqEa5qGovOb31DMLVxiUgkYSf8 +o7UkviebRj4AG41Me01U2B+KHHsAJ5/cDmMUH+JBGFSWCtT6rGfMJhswSVUZfqnlf3y hmwQ== X-Gm-Message-State: AOJu0YyvOiWoy3wBYvIYCN235d5XIjiTwmuTT8GMD/Y8NXrNu7JO6Kf7 NU3qqWQsKWzNVFHtBiYPctZtrD35LOE= X-Google-Smtp-Source: AGHT+IGzqDmmVKWAX0qOLCG4zSSOu2XUUgzUKaIXDvA8+3j+t9o+p5RCbsfa+aQ1r6gMSAIrcYIsUr6/0f0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:5008:0:b0:da0:2b01:7215 with SMTP id e8-20020a255008000000b00da02b017215mr55818ybb.10.1698155205957; Tue, 24 Oct 2023 06:46:45 -0700 (PDT) Date: Tue, 24 Oct 2023 06:45:59 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-3-surenb@google.com> Subject: [PATCH v2 02/39] scripts/kallysms: Always include __start and __stop symbols From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3DF69A0010 X-Stat-Signature: byhicsnjnatwpqbcrejfc6qztmtrwqem X-HE-Tag: 1698155207-790440 X-HE-Meta: U2FsdGVkX19746M165bXqhvnn8qt6vfB0pl4QFkqfnsMCRcrpcKY+YPIrpWqeAWBaRPc1cDJSQlyeOnJ0BtIvQFS8MNIgL+09Z6/1Cklbsxyt+NX5C9g7dLShPWwep5lQ0YY7dPun2A+jkbo65bo/3tiCs1NeXlP2hbmKMJNWjzFni7zvhWsvnVKM9RDUkfLZexmlpwoQton3y8XW+O4GUD8ObmNzMOu5jMusk0gtlDApd0/dgqxvtKfZ64ZeGCqtKx9lfcXYnzKTGgVyieYkl6AVVfXaml7C9ZoVTAKFRgL5Xx0NX7p8iAQ1sBu+1HyYKygJ5aR1FUbOm+w2+cDpY9BxUfN6V1cMfhvbFjIFl1lh/kNOBqWNvcGPUVt5vRdzvVUEQaOTTEYG6czlbyBXTZi1vvfTbjp43ciJv6MlfehDBn9UwBXtJVOufO+820Ld4SOpe+gA4b9q0fD2AYSvfy3kGC5PNt21rgcPlXds8WasjAL0Apeej7GwmQcYmo7I34loATOw4b4MsFYozyHKv9WAf2BEal/FniL/mFm0C4VWF3JZ2xyFmuIqXhGXDAI5eEvcJ8xyZ4FPy3rLEb6xHXZu5uiMkmBWeGd+Yd/q62nO8jqMymfxGqG7apv5mQcIJ1TveBUcAztPGFvAgJlBOiTa1vePHdGT7YFHknmWKghnzZnuIlK/MJtvHpS0kSjhZcmmrYLwhlQQJm6JLlRQ7ysLd857oPbWtTHcReplmz4V2intwt4MkUo2tWwAFPRO1N15uC9p8OHaxLCbQNtMLRbgexe6JEUBVJIjS9f1ojKD5liGFDulsdt8pIT5f6zEpacpfWrANXGhFF/j1BnWOvrHFE9/GxaMB2tytU62wDywc4hNCxPoJTHAGU4ss/YjJk2FNcYoq6N71VXUwyhMw5i+lsFuBy9g76h+nDRjI7vMLa4eo9Vs/4YDq7TYYTmFzr74KQyh02znD5Q6l9 X53Pwo3u nqPGB2Eig5mn+Hz5Pi8HNQ0LFnD4TkcNxwdqJFVx4367Xbr+WNCae4BIz4TEjMOzu8koq2Mzo3KjXOmM/5dG+e76B0/NtL43ctAN6cy3DInQs608fCTuRZWZ5m/a87fipqLBJo2qPQ+6bqgUDds1FGc7U0l7sC+jZyCaKXUva9v4KSTiWdgrDulvwR98Z/AmTHwrc1EmM8pr5yV7hZW3bULFrAj54Dk6grNeFUrT9CkRiPmlKHDpwH75Y/J3+XMKevZs5mlmnn0uuSTRogLpInRJmntBl6bDhGDErsyjn0H9WFOdqNf4wQ0iaaHD/QtQTUcfxwp0Yi4eNOPLJPLP5VPlPS2J3tWefKA2Nr6oD5yCSTMWalmlzn75Zcvfq2BDg0EZSQ5lXgBNLRAP2pETE+I4Z5iU6Vop5sLokPzhTsDHYX45tYUfq+f+R70483edsS6xeSL6hXizI37LrPtZW8l3HklmdAWEYCouHOWiOVYzBazpDsGoJtf2KVgEux1y84YJhjTNRusxe24ZQozoNwWeDP2hXOzmEQzbrjS1NXDgtf3g0mMRqGVqIfNaazlu3rwq4LjbkbnFcLoXx/M78g+bqzHvbH9q+NpS69gtmTJR70v9zh0hHu6NsQjOyMVTwvv6mtpIOrsWROvd/c4Am6aaPkebR23yQ/0l+GI94osQKeWKzZzJOSl2btkK14cPX4Hq5At0KPa9hd1T9W/dLJPnc//HAAL/CCUFZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet These symbols are used to denote section boundaries: by always including them we can unify loading sections from modules with loading built-in sections, which leads to some significant cleanup. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- scripts/kallsyms.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 653b92f6d4c8..47978efe4797 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -204,6 +204,11 @@ static int symbol_in_range(const struct sym_entry *s, return 0; } +static bool string_starts_with(const char *s, const char *prefix) +{ + return strncmp(s, prefix, strlen(prefix)) == 0; +} + static int symbol_valid(const struct sym_entry *s) { const char *name = sym_name(s); @@ -211,6 +216,14 @@ static int symbol_valid(const struct sym_entry *s) /* if --all-symbols is not specified, then symbols outside the text * and inittext sections are discarded */ if (!all_symbols) { + /* + * Symbols starting with __start and __stop are used to denote + * section boundaries, and should always be included: + */ + if (string_starts_with(name, "__start_") || + string_starts_with(name, "__stop_")) + return 1; + if (symbol_in_range(s, text_ranges, ARRAY_SIZE(text_ranges)) == 0) return 0; From patchwork Tue Oct 24 13:46:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F17AC07545 for ; Tue, 24 Oct 2023 13:46:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68A326B0257; Tue, 24 Oct 2023 09:46:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 612AA6B0258; Tue, 24 Oct 2023 09:46:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48D1E6B0259; Tue, 24 Oct 2023 09:46:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 18F456B0257 for ; Tue, 24 Oct 2023 09:46:51 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E89841CB793 for ; Tue, 24 Oct 2023 13:46:50 +0000 (UTC) X-FDA: 81380480580.03.575DDF6 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 36CFCC0019 for ; Tue, 24 Oct 2023 13:46:48 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2ZDPacpr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3yMo3ZQYKCG0dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3yMo3ZQYKCG0dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155209; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MWOhaEKA8zMI+Iz0e6SkNPW5Kz551oWdhyAlPdtU984=; b=GSqn93KKCfTa5garCieMoSfBoKBp/kBwQvJaCrQGnnfGUyit2agKDRPlQG1TtBTTtlxbfB xdBn3iZ99wiyQXwiLuxUM8E7qqQmuqzCzfAV/F8RDWnkVQZbc1C41JVK8JQVTuCKAqqP1Y O+rkWFgywb6o9yrYBuA3lr5WlKuiNV8= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2ZDPacpr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3yMo3ZQYKCG0dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3yMo3ZQYKCG0dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155209; a=rsa-sha256; cv=none; b=r95FZzaoqMqfhhUvnF9i9oyMzQcoSZM99hKSB6a1Nty0sT0Z7LeUKQ1H5d+PBvTpVZGwLg wgns/A13TCC6PwrtgdvUVmd1aSRdEtYUEMt91ejgGGI22vuVUxitOKRCjgn83sr3I7e2QQ hsfJZAoOdqrwJtQvOx7kONLIC/yqOhI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7ed6903a6so59226927b3.2 for ; Tue, 24 Oct 2023 06:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155208; x=1698760008; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MWOhaEKA8zMI+Iz0e6SkNPW5Kz551oWdhyAlPdtU984=; b=2ZDPacprvSLBUyHkUZt0OSl2wdnxAoEUNXPR3uHLfR8bpsTHSs59wZ5m8hrz1mX7kW n5F/LCJe9W0CuptGQbhoA6FzjlEzi5tE7k+l077TVUh0T7n/+BMkxcDbQZjRaF0pGxHI wqmqNNQUTGmPnisPUL4Wgctxn8+e0KZZ7PiF3oAJM1BtDMQdEGTYBmUGW8lY4SkRW4Fd d1bw6eBpwgUEnPUA8Uz68bh885ab6aJOJjvd4NfmYLerldGgccxAHH5vMkwLcjsZwb9o swlOnf50O5PkBrkLeXTHwwHkSfUVeuuH0T6Cu6kjZbmgi5X1gPJ6b9sgdOJFTvHDTRwZ lpCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155208; x=1698760008; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MWOhaEKA8zMI+Iz0e6SkNPW5Kz551oWdhyAlPdtU984=; b=HAamCZOdSjJyGFbnJQB/IJlUGZ6oWpCn+a9J0m47mqCHWpsajZfELxpi+Hy4T5nOOE 7/57rdRIjPhxXdAspzOiG1yN7sIKJ1pxTzRONxt4NnyFP2DpQUcrZwYp4J9VaKniX4qL eYwatxtZNTnXJ6XbyCmBFVBGO4fuPOD6s9ac5wAgHnOfaCcTB4QKct5Mb542epxvnwYZ 5cHO3a7SLw5/upBlTZBdG9VnixHLFeRLFoBsrIZ3kOfPiHUmqn0OVyBnflwIxyajY+RG YSEULjxssCN9/N0euVHlmJTkFAi7hqRWczGRrOEuWzl9tkVJ7PZYHAzgTN12A4WED4Mv zCEg== X-Gm-Message-State: AOJu0YylikmUC4bUXKspmfYQiBGmJkMy38reyjYgBIlqRkHvcnWckMeR kSFUoq1jNvR0Gbb5q09FkZ9i6L/vZbg= X-Google-Smtp-Source: AGHT+IGS8NlUfJRUa0CiIFG8N+DG1BCVzQogfv9Ui1pQm5p1SwS7Eeeej+mjL+3fA2TwJ+bOdBMbqGXulW0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:b951:0:b0:da0:1ba4:6fa2 with SMTP id s17-20020a25b951000000b00da01ba46fa2mr82321ybm.4.1698155208197; Tue, 24 Oct 2023 06:46:48 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:00 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-4-surenb@google.com> Subject: [PATCH v2 03/39] fs: Convert alloc_inode_sb() to a macro From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Alexander Viro X-Rspamd-Queue-Id: 36CFCC0019 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: wgduum9dzmbzfexgmgdrpuoxis8x5buc X-HE-Tag: 1698155208-974326 X-HE-Meta: U2FsdGVkX18HPPFV6dz1Xh23C+OxPB3CV3VV2xQJ0M1C/tJw7vUOWEW3SKJWdxjWkjJid1QUz6FsXxsE2tz1SWthwKsk/qgM7o/1u+eUMjLD+hR/wgkpyEi6eeG4KIyXNsSWW0Ui7abQTh74EFs+URMC7/w1d8Wbtwz62wKA7mra4bajSHy003702cxCADCXkXANmOJ817weShCCOaZ7l00p0dvqxo5N0J5a4mvkVtZhz7Qcp6tSN808qKn792Mio5TGOxSwdsSXLmtxZSiS8BQo7Sma+zIL6/E/l24l5ExeQxwkEsg7tVMv17Qi42whmy4obc68DYzdEqnx5dX8t52shfQWEbRD17IhUG3ccMCn1FoE9OvdMMS3lTnu3SsySM6tyg5bFr4tJTiemjjVC9vxJJ1U04YiD2HiUieFtzOP/0E7ROjwKD0uK8lyve+pQSiG0PefOU7FiqzKJtFTpg8Afk7Do7wt2bQlizpNEEswIAkaWheIaSQ3Gmf18WLcEUUkNdwemUwAS5ZDWT6yIJsfm9ROlEt0S1d9M0Vmtru3UhUNN6pq70rSIM1a42D8EZJ932ZFcQIA2/spLUXq6WWMKSDKAmjbv24tKWgB5zAZODBGJlMuB1LRkaEamJMy6AiaPSaHU6n3aHE1z5gS0ACnsJ/7tICAKqE0pDRJ8+TGtPL/6vXUJ/F1uOsyX+HSCM72P2ZqdtceTGV5HjINk0RY0x+ATQafGel4IR50Q596QT6Xlzok818ZyiOU8sabUxgn5LaIQvsnOV0NkCAumeqJazcRBMpbZRjZcbrTpKIYEZUgypjSxtXFYSLdMs8epDQtpP6hZM+sLhFDm0fyGlG5Sc8uOJyvrymH9pkaIh9w2DEvFLOYLPCTT6UHN+pA0+ZTPOzTMk5GdEjXHqooDYQkVWjpKHLqFp9xXXeCYZQxkaiS1s3OpHyfgwPC8hq1RE22KxXtN6GG3b3Rut4 E1OME3pT by40gRFjsiUpy+R9jjiYOqnVBDWq3MW2wk7vRDCMoLIYB+kqa6IYbCsLwZ2OWj1V2y40jJeX9vik19vlnxOqMUTfFjYiGpCyH3pRRd+lLnTdTfnVLKUgDqyL/3JhDeetQbIyWOpk2eYpCweprjjJGNoBFvziabyPPgJ0z7uCuqguLiGUpU3kYGDf4DYbdU7Y4VtU03ztRCSbGNeTlVFOvumLZWWlWe6HhmQqDWmp3j+i6YvxFb0s+aQairlL2p2duXOXC5DqxofeXZjs1DKGKeu2epSy0zto3ntKy8aZ94xvL4nNVrve9WVL+fE9+5VU4C9IJ/fAgUmS2ojrgQL2i7ZcxrNeI5fBZO0HVtMNL/7QksSpwgbSn6ucJo5aLqXkav5p1L/0Ti5AHFFG3kFb66lKl5MIjMuk7U2KkNYxF6RhXDJ4uIFOLWia/OpZSHjyzXCPh9X8naXX5Q5hnDCmqBv6JwovhDa92+NCMA4tW5tDnlPGsq0GNWmpnJge9GRIxrrdQO4nfv97HbNDOWhkN4k8vAm49B3gkLhwc2KzzwjFXQ6VMPTyPKrpoqDfDrKlIXZDW1cT1HeClSYg7gnGg4TwyNHgonM7CgaBtVtEkIuVDXbPeomHVHcDMJr6GVpxXWkP8khvFaXSCQVjCklBSpf/VOayB7GO+EoFyIIaGVV3nIjvhS8/SYUfkkbgYvWNgEufh8977nt6yHIhfXsXPm7dBrgXiYjnPHOMQfLTBvvoGcvwjliCYEOHiw3TLIPqW1Rpj9tQPf7r524ihmI4FLZTUzA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet We're introducing alloc tagging, which tracks memory allocations by callsite. Converting alloc_inode_sb() to a macro means allocations will be tracked by its caller, which is a bit more useful. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Alexander Viro --- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 4a40823c3c67..c545b1839e96 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2862,11 +2862,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap, * This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb(struct super_block *sb, struct kmem_cache *cache, gfp_t gfp) -{ - return kmem_cache_alloc_lru(cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb(_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp) extern void __insert_inode_hash(struct inode *, unsigned long hashval); static inline void insert_inode_hash(struct inode *inode) From patchwork Tue Oct 24 13:46:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1179C25B6C for ; Tue, 24 Oct 2023 13:46:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C557A6B0258; Tue, 24 Oct 2023 09:46:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE0356B0259; Tue, 24 Oct 2023 09:46:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E2666B025A; Tue, 24 Oct 2023 09:46:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 83DAA6B0258 for ; Tue, 24 Oct 2023 09:46:53 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 30CBCA0789 for ; Tue, 24 Oct 2023 13:46:53 +0000 (UTC) X-FDA: 81380480706.05.8E6111C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 66E1240023 for ; Tue, 24 Oct 2023 13:46:51 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=B2sDiW0X; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3yso3ZQYKCG8fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3yso3ZQYKCG8fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155211; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=griUcA7GUXZ/C94ijw+khTWkXdhOo9/SkZ3zjzjerBg=; b=nHIK/srnhgTXp+vXs41Y0cnNvt65iY27Tf7OYNTgpLLvo+hGa+b4Rg+BBZKwRmsc/TKZWW koZuJ5izKENCscJRTcXVdClk9NIiNhK1tES93fSg1SbkL682a0Whms3IfbvcY5/a2CZ9aC GxJe3AXhEaCSDDzTFH/FuK3hSaq6Ojg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=B2sDiW0X; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3yso3ZQYKCG8fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3yso3ZQYKCG8fheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155211; a=rsa-sha256; cv=none; b=0g/Qfc2DzpCoxGYgmdgaOPe0uFO9ezLL4tUcEc4pqJ+jUe9gpw2R0OnaOLyIJydYdI0I+g w/XFAkE29M8HJ/+ElipYbtW9BsHmy8Sg6QN71CJPvrgWbmMMy2Z+Fn5lV6Xr72kWSyAjd1 bcifa44Y0azJEmS1ed9apU05zalkZjs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a81cd8d267so84819747b3.1 for ; Tue, 24 Oct 2023 06:46:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155210; x=1698760010; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=griUcA7GUXZ/C94ijw+khTWkXdhOo9/SkZ3zjzjerBg=; b=B2sDiW0X+HCpv5iqfHTrmoS8Cw7AdCwbg6F14xMCi2AubCkzRmyK+ENvLw5glyHVLv sccH+y4DIEd61yEEVGjgO8hgjuyzAAF1lK+FEd/qEevhTX47iXsCNtGToiYzxBUkCcNa Ka4gyHQq9XlqlA+Hn3J1AnPuw9+bKsrIx6aVyxclabV9gVD7UdG57KCU3lRsAUFBiXmN UsriTfqPsljwu5wsPkJeRdVucVMJzJuYnAoJr4/DZYg7qx6p3mDx8sr+cFRP+iGJ3t9F 6N4INgYGhQX0pa2gC+rwwSoyXmIsyBss9JSkYtt/4mTrwf/w3blEZe5LhdeitNyxvByv 0bXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155210; x=1698760010; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=griUcA7GUXZ/C94ijw+khTWkXdhOo9/SkZ3zjzjerBg=; b=P4mW0dP4Lfpj5pEqYlX8JIXmr2BpKDdzUI7jAhlzJNfo3WXn4VbjClCGxRL1AXyuN/ QbwZ6/bQjSEI5/O1Awr2TquStZXzavyLDUjPXB1JubVw/Bo2gcKJhH77z9rXn7qfmCLC sg6Zfu2hxEUburtKM128Myi69a9QdcE3aGQKiWnIV3cdlp/b9+DRcp0v70ZYHs56c6jO iV2lahYpVOxC+XLe3dJH0rB4Nooy3s21IrIStgF5FoehdVmnSGNuU1DJ0Ro/ZwQcx9Jl lK6QhJB7Ia3CtIMenA9P1/ObjbUik2JZicJg2lEhXMdG8RpKbcW7TiO9/1LlVRryAmKZ edzw== X-Gm-Message-State: AOJu0YwWSjF9n1npdZo8YjymxXE4iCw/O+EVT0kHmzjQ9wDhtwsKejRT FwIWXq3mTn7ya5Y6E88y+bVzJ7kij6w= X-Google-Smtp-Source: AGHT+IEpwWcHxe3pWwBFpCZZha+IjIrDrruma7abx61pSV1ETO4xUZC89g08mISvSSxLbel6DAK32ekX/R8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:7485:0:b0:d9a:cbf9:1c8d with SMTP id p127-20020a257485000000b00d9acbf91c8dmr216225ybc.12.1698155210387; Tue, 24 Oct 2023 06:46:50 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:01 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-5-surenb@google.com> Subject: [PATCH v2 04/39] nodemask: Split out include/linux/nodemask_types.h From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 66E1240023 X-Stat-Signature: nums4dyfhjwhc43uqxp9zfgsue1inu81 X-Rspam-User: X-HE-Tag: 1698155211-706641 X-HE-Meta: U2FsdGVkX18001rQmN6f/Q3aL50BMaUguTpdgYc6Z9WepkFZRKNkOgpQvaTbIkMU6gMs4eaVLbBFeepjB7w7URxIFaRSJPC5rl4Yu58/Ej/Y0Yn6mvSZzgXK0VpX/BrUBO7cQma7W2iSMs+tXFm1ClNZwotPymtxTXUagKKyWanzF5WLCjvgnqxZFXuVr+3fx7pdzbw2yTPTjUkIqbpCl/1RnL9D4DGr61YOx7cCU9ur5+laVFiZNdUPw8fy/pFB7IQsQ0wBP9edg9COGVbKtdJ9tf01fuPbrTynndAqUazgkZyxnuKUPVwOZzbFdsJ+VuybQEoyss8p5e+DUiMm1446b1SMOIUA1FCxGZG+HRfpo8Cvo8npr92NL6jBMFZQog3CfQGYozDUV8yIS8rc3COtOTaI6miPCafyTY0xMfpGi4M9bAY4i4ucszHwGn1xTBknpSdUQ2pL1g9wuEobRBrRJbcVqN2SFKuDIAYuRTt5cHFXI/MEw+olLig52c9G7MqtgOjInaP9PL1X3OFMAEqBRsb7283GximRL5PtWEpKyLKFbhftjRTxE+rhDEbaIdlN9CtZ27sIvRiCYm7HQEcZshVslHNJ5UZBB5IYHyUCniw+/OuNz3bxiVhIQ87ZzNqfq41S5CpsKjIxO5FqZrRO3Dsz3Fg/HVaK6YESANnT9/wRhJETaCQ3e7euUkS9eoJB1hXlwvM/sKygcCtrgl7RC3VDDYu6fLSes+Jtq1C8tV5HvWPQBPR+drJiuyx5egXYhod1P6dHtqMPJRoXYNL1r+TH2uSgdidotF3VwVTa0DN/kPZ2v4Bfkz9ZFmHLU5J6amqAviTFAS8z/2zB4vVpGD+181AxizO/fVgiw/TslK0n7Oa9M0sIVoA2vy1Wxs86XLHBEYROpzGPmA3mwyAHbsdIWxe+L0OJot7v60shme56natgsYm7ekB03FndufDSIh2Ynh8FKdD5H0p dZjiDbVn qgk10F792KeztXx00kxwtY0QxtX9AuIPFxMl8CJPAc2CJzVanopxIs52HwF42imXDXHUdfzeA9XHZZClUgZlY/m7eB/dhSRwzjADoVVJiYBpSplLf/Re8MAbTC1pJXg1xqDv1//XDNq0cn3afVlJSS+Ax7ZsQvcMizQnnVAMb1Qzsz6XC+j5287olOCfGGtGhXOy8KToFbE+3ehFdiGBLc+jqriQucJmINzRRXuvaf5cGobkgHxgiUo3gMZmJt8qqc3Z7uPByjeK4ODHxLsBDXRTMU9Pi8GZqdhSjwGKTqjme/j2P9FPuxUJ3vHLLnku7HLtd9srkWRbWBF06Bs4l9XwGzjlXR5t5Mi7xGMukTTsn/ulLoYm0pY0xWT30O9w+75RckGQiu1gpDeIVXv8SHUuF+Re1epFYMwv43OtTKwKY+dWfkOQtUEFKemXE3BSRghPda69Y06loo62QP7WASZjmOVuCxMwEWPs9w2gIu+vS0dMlSuN2yKKsTMPWh36J1eeVdzVSbV75cJPqTgXdQyXR9+Ye2wMZgfCeUHVmb+u35i7cje4MjL1NhvdsaiBLF9Zp3EOUh7TSCc7zGrz/U0rvUvVTTePzplrRj/7Ou3cb0dfAJZgKkiJECyFuDqK5E4DtcfiH77mr7oALi2xy2W7cDko5RRnRg3Zq9IuguJGg8hAYrk7TXYuDJLCL14ioR2CIklKdxFa91bEa5coZaKlLXKgFMC9qcbuU6TqTt1jHYHzVuj39KQmrLgDTcKHgEfJsTAsbNn1aTs5YQLUrZURELK4Lca9KW+efK1OLVHMnOeVHSiHhLGVXCw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet sched.h, which defines task_struct, needs nodemask_t - but sched.h is a frequently used header and ideally shouldn't be pulling in any more code that it needs to. This splits out nodemask_types.h which has the definition sched.h needs, which will avoid a circular header dependency in the alloc tagging patch series, and as a bonus should speed up kernel build times. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Ingo Molnar Cc: Peter Zijlstra --- include/linux/nodemask.h | 2 +- include/linux/nodemask_types.h | 9 +++++++++ include/linux/sched.h | 2 +- 3 files changed, 11 insertions(+), 2 deletions(-) create mode 100644 include/linux/nodemask_types.h diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h index 8d07116caaf1..b61438313a73 100644 --- a/include/linux/nodemask.h +++ b/include/linux/nodemask.h @@ -93,10 +93,10 @@ #include #include #include +#include #include #include -typedef struct { DECLARE_BITMAP(bits, MAX_NUMNODES); } nodemask_t; extern nodemask_t _unused_nodemask_arg_; /** diff --git a/include/linux/nodemask_types.h b/include/linux/nodemask_types.h new file mode 100644 index 000000000000..84c2f47c4237 --- /dev/null +++ b/include/linux/nodemask_types.h @@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __LINUX_NODEMASK_TYPES_H +#define __LINUX_NODEMASK_TYPES_H + +#include + +typedef struct { DECLARE_BITMAP(bits, MAX_NUMNODES); } nodemask_t; + +#endif /* __LINUX_NODEMASK_TYPES_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 77f01ac385f7..12a2554a3164 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -20,7 +20,7 @@ #include #include #include -#include +#include #include #include #include From patchwork Tue Oct 24 13:46:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D44FEC07545 for ; Tue, 24 Oct 2023 13:46:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C09596B025B; Tue, 24 Oct 2023 09:46:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B442F6B025C; Tue, 24 Oct 2023 09:46:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 970816B025E; Tue, 24 Oct 2023 09:46:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7644C6B025B for ; Tue, 24 Oct 2023 09:46:55 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5773E40470 for ; Tue, 24 Oct 2023 13:46:55 +0000 (UTC) X-FDA: 81380480790.09.454F2BD Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 8482720010 for ; Tue, 24 Oct 2023 13:46:53 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=G8pFF1dc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3zMo3ZQYKCHEhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3zMo3ZQYKCHEhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155213; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r9ZaCKU38kmyhI7VEwX4KjtHXLgRMjapTHFgQ9SG6t0=; b=epOgcozy2c4TC5/rnN5DLgLpuPmGZ6T8KnQN7nHlmNLdsn6hRqgrw0ru5qQyVE7W9Np+HR suzE4et98Pup7rSnj2mTdX+t5ovMyfGH32T7E6BBme52fth+sF9UarYMV8u/9Zyp93A0dq ahQDjUe9tED7zPZUmtdmjCyVxlViEM4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=G8pFF1dc; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3zMo3ZQYKCHEhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3zMo3ZQYKCHEhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155213; a=rsa-sha256; cv=none; b=DWB1FBvNCsmkDYekGQ/c0QU7H3IuLesTXCx/1eS0AOqVemXHQmv+HtGeiTN3c9qTYhJvZ0 WUzmEZzllP9JFFCFFwG7nm69cPcMDAv5TAZpJvRNc73PTTByP8B4MiLv32WeaiRNcrNeG3 7u9comfg1JmjQ/nlI7+7u/eU6WrBEAQ= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5a828bdcfbaso61798667b3.2 for ; Tue, 24 Oct 2023 06:46:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155212; x=1698760012; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=r9ZaCKU38kmyhI7VEwX4KjtHXLgRMjapTHFgQ9SG6t0=; b=G8pFF1dcVVrfbmk0cUTEZ9XQpCJL/ldcDNgYY5nFW5h6o9y2yFaYV6GFjcyjjkPDUo tEJxTrQ9D+wPZLPvUQlri5tG19tR2G9TMWry6xrH5yVhE6UHz/3+zfkqQ+v1VDw0/7+p 3Iu2duUzlUGMzr4UEzZaWA1t2XA1g/0U2/QE6dApBeXqyXtLgWl9uqxONVNG+sQD0ybY mD7EEl7YfQaAbfshcscQn+gDXc/CsFIHIN4+uo4+aqGPute91Ia+xzuDOaovIGrEazsm 6e3FM/4C98o7AWaDmM4l5fDGQ9iyMX/cWnWSvHOdA1Ix+Bafv9+v98px3WP35niHdS9B n7Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155212; x=1698760012; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r9ZaCKU38kmyhI7VEwX4KjtHXLgRMjapTHFgQ9SG6t0=; b=hbi7GFVabAHyKXhLfSQaq5CUamz4uEcGG09qwcXhaAUDOxdvoN8l200ciba7pkGFzp DcJOlnsl4HYP/cKbs/MXgu0ubDJA4+zu+gTzuFdSsYVEUg1/9uOMefAGUXSp73nPZBzX ONe/NW+Vr+VI5pxXYZstj22h3hpIzifcvBsNG2eUB/RGZiuSEXCz/6ZJheL4cYMmVwW/ YlNTQD/CUCudB6bIoRAEeCZEnRBvrzslHu66CA2tAO9KQylx+uWMdf5k4ErYuS4TSrAD Cxb5UQL+yb8j0FCTgtyao0gqtvfEFhv7i2ppg/6UipBn58313DfWW0B8mX0II6BKJPqJ J9JQ== X-Gm-Message-State: AOJu0Yx09sphslAezLcrXh7pxTeIi2aNDVc6OADK69nB2CMEzvJuqoTy JXbpzAi3Lg7/WjV5/Pkw8CypJ9XMbhA= X-Google-Smtp-Source: AGHT+IFUpFLBsDMNDQVhWCjHvEp+oEK06bmA8hZB4NpLCdYe9bJ3XL90v8R51rmnjDRHAZifNxDo+IuLmGU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a0d:dfd0:0:b0:5a7:ad67:b4b6 with SMTP id i199-20020a0ddfd0000000b005a7ad67b4b6mr254303ywe.2.1698155212529; Tue, 24 Oct 2023 06:46:52 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:02 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-6-surenb@google.com> Subject: [PATCH v2 05/39] prandom: Remove unused include From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 8482720010 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: bdasqwb88pn65mcku3jmz68de9xhnpo4 X-HE-Tag: 1698155213-392356 X-HE-Meta: U2FsdGVkX1/sA7jd7B9tnsXGeqLiOYdJWiowlz4mUdoWx1O+BVy5jLUcC6F99DZTJhZc6rFunajJveGO1mYHs80ZHBt+EAcA53u89qA/Q6+RByXVOHXtooABLHCr3z/2RvbQ18Gwejh2iy6Qz5YaEyjHa3giKfuA/CzJo36B71ADISI2WdS28ZBUgtXE3m95J26GbM1gFko8O2khfUH6t0pP6tzGIyLSRtjxhuktd9Fm0Tuha6jNYb/uSqctC7ZoPaG62eXdzrjTfi1gjcV+WchZBQX6PxEnlxxHODeHMUnnvUZ1ax7wi4uStrm0veYE8syPSEZGufCnkGFel9gzBRUGO+7SkSbs9d35jqGSXVXOB/qeyCy0efCBsWtgd2HhiGC0YZfvQxPHTQi+rq3ThcX6c1bDd1OaLkQi4v2LEKULdGU75ro9BED/YkliMjcOHDhQmnrGsvKz8yMrz4ZZALL8tXfbZkT8Q4RXKl4I64pCMv9Cz4kADpthBH/LccWwTczitY07vtbbNH4VVn4GRUxWdHjpJMtHCzltRDpSEvDsvHtphItlUBptLib1P97fi/M1uon9F6jOpLOQ6+EHOo5wASn3MJrVhZEWW9/KpBE3Jz5N+aEoCyqLwnTnyIC0BD4tXWQrqbrJyP50l0Sb1lUHDrEFHlYPjwa6c/q7sdB5C1xLD296MlpllH3wkOYHv6nEPbTMDecdxnQzEGoxA0i88qT8aEFVOw2+mFlHfJ71WHwVRm5vzMKSA8mL8JhiG0YSoqMQc/YbvHYZuGYFPxj80Gla0mUhGfug8HFjJEaD2tD4CikuNNETHy8yXRVNri3y1Z5xFDpYs1/yT4hCZ9pgl3+ZxFOtFW1SBokGvrCk/0JztPprUGa3lSo8CUr8sIK9QADPErs6Aras14qFrjkrPiBBuJlatYCORdDss1UU/+AnHJoJNMvHSgC5x0oBBxtfkkNwFrD5iRmbB+z cQrI/b2c 3HepoTGgiFqt0kC42PWzv9fS2es7tBf6sXnOV133l9xIFCWB/Zca+9ypH6z2x+kp25bgCC+HIDJ7NzbX6rnTSCJYBWdptKmmRmvcwEmca5zjW5bvJU6F790mIEpakykTqAY0hCshC2YsH5krL+cIaW6Zdx9kvxbSGPnTQ9OwDvA3agy5/6ClmK+MlIH6mq027xH8e1Tb3PQFFD4xpydB3GMtAs6gyS4XcDg9m9n392ndxMJgHeUHMSbPQFCLOhRdvKtc4zrO/l/Smomb5MkCd9LnkbvfI+Y3yd7G/4KJ+MePraP69D4QCvqkLVbnLNYinTc4G6GqMgOKg/Tij4eQkoud+ZOzS5zsfnw4uaFOa95CYqziJhBus2vHWhNGdtLUYfpakymsqWQcT3m19Z7Re4xqQTChPsX60A/2aw5ITqwjWvwDJdDOd4HovjTu9Rs2pW35RnDbcRHWORMAggoTH9omLoBXS9l2yAjdVNN6CRcWRE7B4Ax5ipu42umwmP+yIRFe9rFjh/ePIDmaAlJ2ymBjB7jBWyfcKKpb3hoqQkTD/7XvYbPo8/nn2InBeyjH5cAc0KcD/xwRJGoJwzBOyRHk5yQpS+xtte8VP2d1p/PYp/Lc8RgT5evQq7up8plfKqkP9vHo44AplRIF0V+w5ydZy3P1Jm3jfaBLeeCBjjbsuu9/v0i7n18EuuPVfuvvcRUue+has9lchPQ6LOrvzm8UZTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000019, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet prandom.h doesn't use percpu.h - this fixes some circular header issues. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/prandom.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/prandom.h b/include/linux/prandom.h index f2ed5b72b3d6..f7f1e5251c67 100644 --- a/include/linux/prandom.h +++ b/include/linux/prandom.h @@ -10,7 +10,6 @@ #include #include -#include #include struct rnd_state { From patchwork Tue Oct 24 13:46:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E539C07545 for ; Tue, 24 Oct 2023 13:47:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BA286B025E; Tue, 24 Oct 2023 09:46:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 245CF6B025F; Tue, 24 Oct 2023 09:46:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3A316B0260; Tue, 24 Oct 2023 09:46:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DEC466B025E for ; Tue, 24 Oct 2023 09:46:57 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BE6D11A0302 for ; Tue, 24 Oct 2023 13:46:57 +0000 (UTC) X-FDA: 81380480874.13.CC3C906 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf28.hostedemail.com (Postfix) with ESMTP id D0D3BC0024 for ; Tue, 24 Oct 2023 13:46:55 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WBtb0nBo; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3zso3ZQYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3zso3ZQYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155215; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X9FRyOcPx+PqvPwKi5hgMRq683fi1N6H4UQ0tqouPmU=; b=WYH49o8XSv0hyjlGyxQqCJx14S6WHf5PeYvWZYhgWGouz4mUV0c8FTe7rxkMEAZFZji7YS 6gd1al4gqLwJIetB1NJnyOJx7vnbLtcBGTO+c6rzPYzfEBEi/A24FvRW3yXPPQTwjOkzK8 EAJMYIuDplZO0JlSR0hsi+ALZbEx/IM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WBtb0nBo; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3zso3ZQYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3zso3ZQYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155215; a=rsa-sha256; cv=none; b=kTFprB7sk8EMGOlg6qVw1/Va42u9qw6+SAWj5yEOdhjZPckp02mpwd+fGx+XE8ukrycHh+ NUofyVi0UQnK8fVUlh/anE+MSXPsdu32zK0jgIJdoXNHRosI0Enb6u0RJaRozyCExFO9Dw KT9Q0x7+CHWKQCpSRissI+GsE5fwS0M= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a8ee6a1801so57870967b3.3 for ; Tue, 24 Oct 2023 06:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155215; x=1698760015; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=X9FRyOcPx+PqvPwKi5hgMRq683fi1N6H4UQ0tqouPmU=; b=WBtb0nBoSAA94bGtZfPJdwphdeRsU5e0Q/hVAvUlo1Bao9O5H60/E4dIfF88JDd1/b 3E9Ew1oNSQzABQPu9s4NK03JyFyVldfpa/+El2CnSxyOcynMzeIqchLRpafLi18uMMYN dRHBKN8ZBnoC4djOpfJxsARtwKatIDYMmqEOxPmI8wM9s6r7VBSjscO+HR6qWZrhaL87 lHYFGnKgPT1FMuy4GUypAElGsUG66XryDjVUmGw+tYk3GLtMNgeNxms+Tmdf1bVOAtuF eDICb2lwXK0B1nqeD3ECt/mcPm7SlGW756FVnAGJ4nFBdfFj5Ss1jVNtH9VldaXj39eP D1mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155215; x=1698760015; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=X9FRyOcPx+PqvPwKi5hgMRq683fi1N6H4UQ0tqouPmU=; b=buyIDsfT/Yg6biEVBDZIgdTGnJ4HtM8tPQO/JWoJNs0OCZAu4VQ5U0RVBbVeP9OS1m MvhW14zvu8vDrhE8MwDq0xk7NkQqJ1Kh34l8EnnOp2zgQWY5MhTr6W2gf6DqMUdjv4Ie S/VwwjqoP3/q5hFPhr7fHX5fOk+5Pntfl/FJVmtLKcwxb2mZpZs1fE9T+BQ9lnqLgZnL tc5j7TdUsy5gIHgfnBy0KDTgI2FmTWAVkYdZ+8honEIxw7UA39oCPKUKCwFl8ROIJMJF kSf9eZT1Q7Mlum0eZ35meslZMTL+6cqICbnUT3UDGKHFhO2PyWx4PUQkDUJ0zGKkmNFl PdrA== X-Gm-Message-State: AOJu0YweIVc+6iHOmgvFsWR+6nP6yoS4O+5C9D8RKjpGxnHK+3Vnrl57 ytRZ8tevwykTWVERWY4OvZQ/I6fY2WI= X-Google-Smtp-Source: AGHT+IEYMUNI7PlAuFFQePhYZveJfVsgkcZviL6909NPcDbeQdTGb/wj9KBYs/s9p7NjrzoDo5b3j9oL6eI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a0d:eb8a:0:b0:5a7:b496:5983 with SMTP id u132-20020a0deb8a000000b005a7b4965983mr249771ywe.9.1698155214761; Tue, 24 Oct 2023 06:46:54 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:03 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-7-surenb@google.com> Subject: [PATCH v2 06/39] mm: enumerate all gfp flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, " =?utf-8?b?UGV0ciBU?= =?utf-8?b?ZXNhxZnDrWs=?= " X-Rspam-User: X-Stat-Signature: kctt9xjtg7q7ewy5cfp9eciweakewky4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D0D3BC0024 X-HE-Tag: 1698155215-928337 X-HE-Meta: U2FsdGVkX19lxgrmXagcYysJDjWigCfoGxWXeq+ZIgoy3Khv0HqJ77kZdXmlZmsbwi65BGbelqGJUiKZujTGPTpzy3ZDNhX6OWBMsK91e98Y85OYDAEGWgVPaB+vHg7oPuyLY23o9m+PmFOlBkBXUp8qOCkql/WGfNJ6GHNtb+COUci0DnGWH81cv0IyEJl88eLPd+TLpEq8lhLI5uZ6IJdCC5i3epi9qaFM9eopTP6lBaVYXNIgdR5PVXHVS2eyADYfgnHvn0LxS0nh8kqpYdcv/b8edQCyaXLx1qUiv5dlSWMVB6ARFfwMqul98wNUPHbspXAbbf/h9VP3W0IlZdZyga8Bc4OE1cmR/B3gQ4ustbGHrW2njDFsBg3/0bj64eqCOtSuTOc7WscKgUoDW+7Pcp/vD78oPgQIABlqTGnPkCS5AYHfICitDOv0/PabTRpoEFw8WEhDtESyuHprPTUzycc/BjLlIvU/uku2QmwQN3gnMzTbFcHMENEuMVVqPslIIm6vizhX6L6gHrhIHxmX6R/7uqpIREL2E54/fhJunCZdps/ueoH3pzsVEp+NfOHn3RFz5rG7YDutgSFqxu+L+NDDIwvglrX+yh32pWLij8GJfaMVK6ypSN7aYaG9imU4WHSWppcX826O2s3lAw6QZZZEpRuhKYcDKRME8CG9CibbqDHvQDcuc8xkB3ZmSrEcd93EDDxTNsNe0+vUJcOVBXPPiA3PJRSuaXwqfTRQTgEgZoY10zi/ksxkFX/z4sDviMWiN/mhBQXs31q9IUH4CWz2DpiX0mwEQJV1X9iNaGQ1T0HO0tgF8kB/wDKn59sEqlT6zTArj4TDTCgpUBpnIbRawgolNvURpE8sH+E1YesKAeyiaY3VOZBouF1XNnjPOO7bC47zuuB/1xrhbCM376Ke/Zy8/46n4qXJOF1jRZusbLpj8FrGZb7atpLuIKcWLg+1ED9bqEoC9Mg tdlRE/t7 C6QnVtdMv5SiRrueSN7eN25c6Zbl9xpjqZOzBA3ZJmssp//pwd8oh1wtr5HEO8Ls/2ZPYijXbkXqkr08RwToGchy7NaibpnAGu3aNt11cFLEsxFle8SFBpJJeRH4U9T4n+4bX9lWOpWzfMhpt4ZRZu+ZPnYWUVW9ZV2Cu8L8Ht4MS/KnQCUJTakNrUh4lGhX56BKQG3UAfEk1vM0HPVwQGjswNTQ207Qoc/ZV/m/1DFxbi9dU8opksyOwwljBzy6PYk0MGD9IdZ196dXoZSh+Xok6b5lP2Gq1SJrxWOC13HKsCcmzps/MpjHV1NgEP1QD/uaFeU8vfBpKkSiLhpyh1nx/NACNlWcbJFoibKHw2t3NZGe9tvwXWwxakxuYh//EyaF+TaAOaqOobasxOXW7G4I6xQoW33q/UBSyzDq7HZV9/li3DJWxYtR63AbP8tdd2s/lm7fSfeoWmaE9HIU7rNcF2SQ+dYxlEzs/goNSqP5yWYQSndIThilY+7nJrEWCFEFvwuaxs3f9x3GiZN0XCgZK3K3HS23ckDkhwb0f8yR4xlM1ONNlQc3sE5lphq+dflog5paI+LhM5sOYyIZFfeZmVSTRLFSTIu5iAgLj+nYAeAaPGvY8kgbjC/7siJ4yQNQiEd7cXIRNzz9t5lZacKky3jxzZ+LsP/N20qhLtrdMXaU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies __GFP_BITS_SHIFT calculation. Suggested-by: Petr Tesařík Signed-off-by: Suren Baghdasaryan --- include/linux/gfp_types.h | 90 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 28 deletions(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 6583a58670c5..3fbe624763d9 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -21,44 +21,78 @@ typedef unsigned int __bitwise gfp_t; * include/trace/events/mmflags.h and tools/perf/builtin-kmem.c */ +enum { + ___GFP_DMA_BIT, + ___GFP_HIGHMEM_BIT, + ___GFP_DMA32_BIT, + ___GFP_MOVABLE_BIT, + ___GFP_RECLAIMABLE_BIT, + ___GFP_HIGH_BIT, + ___GFP_IO_BIT, + ___GFP_FS_BIT, + ___GFP_ZERO_BIT, + ___GFP_UNUSED_BIT, /* 0x200u unused */ + ___GFP_DIRECT_RECLAIM_BIT, + ___GFP_KSWAPD_RECLAIM_BIT, + ___GFP_WRITE_BIT, + ___GFP_NOWARN_BIT, + ___GFP_RETRY_MAYFAIL_BIT, + ___GFP_NOFAIL_BIT, + ___GFP_NORETRY_BIT, + ___GFP_MEMALLOC_BIT, + ___GFP_COMP_BIT, + ___GFP_NOMEMALLOC_BIT, + ___GFP_HARDWALL_BIT, + ___GFP_THISNODE_BIT, + ___GFP_ACCOUNT_BIT, + ___GFP_ZEROTAGS_BIT, +#ifdef CONFIG_KASAN_HW_TAGS + ___GFP_SKIP_ZERO_BIT, + ___GFP_SKIP_KASAN_BIT, +#endif +#ifdef CONFIG_LOCKDEP + ___GFP_NOLOCKDEP_BIT, +#endif + ___GFP_LAST_BIT +}; + /* Plain integer GFP bitmasks. Do not use this directly. */ -#define ___GFP_DMA 0x01u -#define ___GFP_HIGHMEM 0x02u -#define ___GFP_DMA32 0x04u -#define ___GFP_MOVABLE 0x08u -#define ___GFP_RECLAIMABLE 0x10u -#define ___GFP_HIGH 0x20u -#define ___GFP_IO 0x40u -#define ___GFP_FS 0x80u -#define ___GFP_ZERO 0x100u +#define ___GFP_DMA BIT(___GFP_DMA_BIT) +#define ___GFP_HIGHMEM BIT(___GFP_HIGHMEM_BIT) +#define ___GFP_DMA32 BIT(___GFP_DMA32_BIT) +#define ___GFP_MOVABLE BIT(___GFP_MOVABLE_BIT) +#define ___GFP_RECLAIMABLE BIT(___GFP_RECLAIMABLE_BIT) +#define ___GFP_HIGH BIT(___GFP_HIGH_BIT) +#define ___GFP_IO BIT(___GFP_IO_BIT) +#define ___GFP_FS BIT(___GFP_FS_BIT) +#define ___GFP_ZERO BIT(___GFP_ZERO_BIT) /* 0x200u unused */ -#define ___GFP_DIRECT_RECLAIM 0x400u -#define ___GFP_KSWAPD_RECLAIM 0x800u -#define ___GFP_WRITE 0x1000u -#define ___GFP_NOWARN 0x2000u -#define ___GFP_RETRY_MAYFAIL 0x4000u -#define ___GFP_NOFAIL 0x8000u -#define ___GFP_NORETRY 0x10000u -#define ___GFP_MEMALLOC 0x20000u -#define ___GFP_COMP 0x40000u -#define ___GFP_NOMEMALLOC 0x80000u -#define ___GFP_HARDWALL 0x100000u -#define ___GFP_THISNODE 0x200000u -#define ___GFP_ACCOUNT 0x400000u -#define ___GFP_ZEROTAGS 0x800000u +#define ___GFP_DIRECT_RECLAIM BIT(___GFP_DIRECT_RECLAIM_BIT) +#define ___GFP_KSWAPD_RECLAIM BIT(___GFP_KSWAPD_RECLAIM_BIT) +#define ___GFP_WRITE BIT(___GFP_WRITE_BIT) +#define ___GFP_NOWARN BIT(___GFP_NOWARN_BIT) +#define ___GFP_RETRY_MAYFAIL BIT(___GFP_RETRY_MAYFAIL_BIT) +#define ___GFP_NOFAIL BIT(___GFP_NOFAIL_BIT) +#define ___GFP_NORETRY BIT(___GFP_NORETRY_BIT) +#define ___GFP_MEMALLOC BIT(___GFP_MEMALLOC_BIT) +#define ___GFP_COMP BIT(___GFP_COMP_BIT) +#define ___GFP_NOMEMALLOC BIT(___GFP_NOMEMALLOC_BIT) +#define ___GFP_HARDWALL BIT(___GFP_HARDWALL_BIT) +#define ___GFP_THISNODE BIT(___GFP_THISNODE_BIT) +#define ___GFP_ACCOUNT BIT(___GFP_ACCOUNT_BIT) +#define ___GFP_ZEROTAGS BIT(___GFP_ZEROTAGS_BIT) #ifdef CONFIG_KASAN_HW_TAGS -#define ___GFP_SKIP_ZERO 0x1000000u -#define ___GFP_SKIP_KASAN 0x2000000u +#define ___GFP_SKIP_ZERO BIT(___GFP_SKIP_ZERO_BIT) +#define ___GFP_SKIP_KASAN BIT(___GFP_SKIP_KASAN_BIT) #else #define ___GFP_SKIP_ZERO 0 #define ___GFP_SKIP_KASAN 0 #endif #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x4000000u +#define ___GFP_NOLOCKDEP BIT(___GFP_NOLOCKDEP_BIT) #else #define ___GFP_NOLOCKDEP 0 #endif -/* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -249,7 +283,7 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (26 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT ___GFP_LAST_BIT #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** From patchwork Tue Oct 24 13:46:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFE30C25B48 for ; Tue, 24 Oct 2023 13:47:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B5616B0260; Tue, 24 Oct 2023 09:47:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43F406B0261; Tue, 24 Oct 2023 09:47:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 269BE6B0266; Tue, 24 Oct 2023 09:47:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0ECAB6B0260 for ; Tue, 24 Oct 2023 09:47:00 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DF377B5BFF for ; Tue, 24 Oct 2023 13:46:59 +0000 (UTC) X-FDA: 81380480958.14.BDF25E0 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf28.hostedemail.com (Postfix) with ESMTP id F33A5C000F for ; Tue, 24 Oct 2023 13:46:57 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VInYczMe; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 30co3ZQYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=30co3ZQYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155218; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IRfqS9iu/xaXGtQj/xVRkZmpW45IwDBAix/OEbVBjfA=; b=4JYv2995Im1hacgzD4TxzU5W7JahQWoe4Ev/He7LHfhhbEvZQrcsr/9bdBqfnlHeRnbBYs UJekIvuQdywIw76nv55Skwq0FB2rzLK/nSGTtqMbOWfF3nWiYEuDOkG5Z773hJ0eBSoFtx 0okzzwmSXnoBuibZb0ZoMMhMqB6lMr0= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VInYczMe; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 30co3ZQYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=30co3ZQYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155218; a=rsa-sha256; cv=none; b=vX7vFRPxXTwGPWJkEv+gpWLFamwdzhMECUmWYLNQL3e53fywMyUD1kdR1/DX+mAxrbiRZ9 QLHSG84oQiWDXC8O4mICdcpW4t2EV35P8yuBlVqHUL7DULdKuqycVkOYBwmW1G1OHAnoNh /xID8fJ1HwtmgU7JLyNEdmKoUOWRHAE= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9caf486775so5283422276.2 for ; Tue, 24 Oct 2023 06:46:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155217; x=1698760017; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IRfqS9iu/xaXGtQj/xVRkZmpW45IwDBAix/OEbVBjfA=; b=VInYczMeeIWAYSVN0l93d7fKIv5qxoIzH1aUO/a010Jdc5ZYRGdhC6PHqgdTiKct33 hWP9r2mKTX5VqSwQrWwthzm1jZ4ia6UENj2Yd56V2wG6qkyCHXcDoetE7gPT29v4wQ2M BSG7txOEZtvh+GxFYZgQ1Da+OimC1y49BEk6mSRZufyGqsg6gFvY6Z4IbConmwb8Kfvw xvGqL9VITF0XTK8YyoZggajiUfydiyLKKg7P3ZqDp3PB0juGfI5frsRypy9mRw2yaGoL jrpBVBsvueGsGCvICKXDrWKtsyFcQza5xc9Omq0I7Q8ECWk11ok6ve2uv7X3i1zA7HYX ZaUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155217; x=1698760017; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IRfqS9iu/xaXGtQj/xVRkZmpW45IwDBAix/OEbVBjfA=; b=Z2P/FM4XWiwAjwaQarwPgj0i0qJ9he4q2CKzi3jpeNmMGp1avgcfLbIEQqQ/rjKXfk 2d67PpCU306VslkyqtS4nHTE1YgPusvuVqxutzpHdIo4VFKrkgkgAnmtU04RgNdEMGBe FijGi9c9//X+w5q9sL2LhHhdEOFAjyySAWriWByeCGr+q6Z6U3+JAOhruBsCU7atsmDu DiaDh5aY5rr68aZWjCLWRHah4+h06DCx3jjcUW8ldKgyGaGowEVS4IuB/jak7u5vAD8I KbPaTtG/uMmOUDeiv3kh6WbPwF3jP5TbnUmpUkX5T4Jfpk8duNxkfnMVT2uOcfdRlO4j YjyQ== X-Gm-Message-State: AOJu0YwR1/+nj9sNCEZ5jTXw8TFVNeiz0JQLCgaIa+UTC/4pXF2BGAle QRxPBHlsQ36pezRqh8MDH/CuFN/zoXI= X-Google-Smtp-Source: AGHT+IExf3DH8HKKz+2vcB2BBcqQb151AnXodOPQ8JGUuxRZrra0fIQizb58SYKBrv5Wfv4WxbjgymhW4I4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a0d:d50f:0:b0:5a7:be3f:159f with SMTP id x15-20020a0dd50f000000b005a7be3f159fmr287302ywd.5.1698155217039; Tue, 24 Oct 2023 06:46:57 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:04 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-8-surenb@google.com> Subject: [PATCH v2 07/39] mm: introduce slabobj_ext to support slab object extensions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: F33A5C000F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: dampokx7c1a3y5r3d1ewczrwnrj1ce64 X-HE-Tag: 1698155217-143639 X-HE-Meta: U2FsdGVkX18BSTHJKuVpEILBwr2oUXxMJIk8MxTkJdtNzqzA5umzA2wAsbXDdRcTEB2nP41QO9FHUwppyUsTFSswgMah2NyUnXeHwlFl1lfxkNsxRxLLuVLN75aBU+MTUUkWyBbiaNYO95t0b1T7SAH1oIY/ee4lEIfb1uwZd6TUZX51qsx1CqVUMtltNBqbDLzxfyj7cq98swcaps6TmLGwr7gm/lbAJBpc7Cn7rYStY+XTBvI62PaUUgQwZ9x7+KLNeaMcCkckVmTOATwpdYgSeGQByp1FU+TpE5Gn45l8drYW7jrh6+aFZPNudXEE15GoNg7BGcNSnWf0nh0glYwA8x+ao/Hym5MumWI7udI9qGMxvAMWn74Go42K7JRjA68yDkV99lgyIQzMAX8ivqIzPeDSfZGAeb/+o1vtVjr/eQailVQdifToZy+LSSjDkWe5PyfspCxEPAvtmHubwuUmonHTWXJh+SrWHXOo2X7slI2jGXs1Jk2NrYo3GU+oQMYfrCi9cZ4FyNTM1QnFCnFsje7a9qEOnORPVgDYPS2dKyq7p4bZ0jtmhbB+hdI3QbCoZjNJZ8qT7EksZ5tq2QX6l4Kw/BTtDqtaSos0jnWFh8GQi7TBdrhGHXlV/07fWdVXmXSuWpzAE8mCDNvUjEHuVjr4ezABk2m6sFFnTHUsDmjVZTVEA/LryUAItAKvooLlURfe6ryKSyC7VdRSk9e1CzS7vNONOR0F1z1P31sqM7q13Co1udlmkyNZCMKP8avkm/M4jI49dDCOSXLlL8AcmmL8WgsP1IrVZCVMTm8AmGQig/V5gzzw8yr/SA+bC8/eEvsyGgyYSvc9WR4kddie/2AlUUeISWH3jXJCIq1GxkEuatvmBDuFWsXZ9+wHkPhAjsavFleHYKYO0UVqDySrfUgDJpr+mBgZehIXE/tNrp3Tv9JFzU7RSJ4mnDCm55Gv8Up83AO97v6ONLT dSWWujok 79J4Yncb22xr/pgh1Evz3HwkM70e6H1prKYU4XGOnqNRwsA+J0UvGQEhIvlgG9K6DZ4t0zyNBENHGxC4D4/BAH/u7mB/6/iAb8ozOjlr4G7qoTojwoaJLO/2+KjEh3Uq8M6W3nXNkii+Qyj9b2KJp2RmPVXQ/q70Gv8FhOGn+uYtvtIyZ4usgIh8SQXObkZ7FZbP2larkMTRX/ZVe9BJUBOQ3RAek+MZUiDNzPHdkwL3PtUwEtZnERBjTLTUDvfxYY8QrGAj5O3zhGjg5CmgWeR1QQ3xWyxOvmekesf4QSH1KUkx5h31RMHpPyC4FEHsjRZq//yqj/nQ9/iHs5SV4FjeC2xYlTdMUFR5NYGkWMzK4UUMy8xQsIVpkLUkDzR0MjzUQWY/95QVBjQCO/+m/u3LWLJwQkAI907qfRxPakl1418hmEpwsQJ8W35KHuMiesFoZeZTePyMPluI7xO4O8wBN5NyKWZ5N+pbGELO3qSCnkeeoaHiIIYdA6NiBakt94VEly/NFC/by+J4L6AgU4okqnoyeAdXIw1x+5aT3+ylZXRYRHSH5zNUjt32aXXbfFBQWpdDT2wUouAf8CsVkv2y7PLT4dSIAYXZqNXbFUbK6QkPH7aAGXNimRd697xBQp2sRkRAjj2AW/wUfKLa07AhsjTMniA5/iXD2zuGfCd0dDXJMt05LVc0hIQ8RMRC/GrpNGMRSVIT48S7vuaJIHw1oltABQRQJC1sB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently slab pages can store only vectors of obj_cgroup pointers in page->memcg_data. Introduce slabobj_ext structure to allow more data to be stored for each slab object. Wrap obj_cgroup into slabobj_ext to support current functionality while allowing to extend slabobj_ext in the future. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 20 +++-- include/linux/mm_types.h | 4 +- init/Kconfig | 4 + mm/kfence/core.c | 14 ++-- mm/kfence/kfence.h | 4 +- mm/memcontrol.c | 56 ++------------ mm/page_owner.c | 2 +- mm/slab.h | 148 +++++++++++++++++++++++++------------ mm/slab_common.c | 47 ++++++++++++ 9 files changed, 185 insertions(+), 114 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e4e24da16d2c..4b17ebb7e723 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -346,8 +346,8 @@ struct mem_cgroup { extern struct mem_cgroup *root_mem_cgroup; enum page_memcg_data_flags { - /* page->memcg_data is a pointer to an objcgs vector */ - MEMCG_DATA_OBJCGS = (1UL << 0), + /* page->memcg_data is a pointer to an slabobj_ext vector */ + MEMCG_DATA_OBJEXTS = (1UL << 0), /* page has been accounted as a non-slab kernel page */ MEMCG_DATA_KMEM = (1UL << 1), /* the next bit after the last actual flag */ @@ -385,7 +385,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -406,7 +406,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -503,7 +503,7 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) */ unsigned long memcg_data = READ_ONCE(folio->memcg_data); - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) return NULL; if (memcg_data & MEMCG_DATA_KMEM) { @@ -549,7 +549,7 @@ static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *ob static inline bool folio_memcg_kmem(struct folio *folio) { VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); - VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJEXTS, folio); return folio->memcg_data & MEMCG_DATA_KMEM; } @@ -1593,6 +1593,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, } #endif /* CONFIG_MEMCG */ +/* + * Extended information for slab objects stored as an array in page->memcg_data + * if MEMCG_DATA_OBJEXTS is set. + */ +struct slabobj_ext { + struct obj_cgroup *objcg; +} __aligned(8); + static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) { __mod_lruvec_kmem_state(p, idx, 1); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 36c5b43999e6..5b55c4752c23 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -180,7 +180,7 @@ struct page { /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif @@ -315,7 +315,7 @@ struct folio { }; atomic_t _mapcount; atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif /* private: the union with struct page is transitional */ diff --git a/init/Kconfig b/init/Kconfig index 6d35728b94b2..78a7abe36037 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -937,10 +937,14 @@ config CGROUP_FAVOR_DYNMODS Say N if unsure. +config SLAB_OBJ_EXT + bool + config MEMCG bool "Memory controller" select PAGE_COUNTER select EVENTFD + select SLAB_OBJ_EXT help Provides control over the memory footprint of tasks in a cgroup. diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 3872528d0963..02b744d2e07d 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -599,9 +599,9 @@ static unsigned long kfence_init_pool(void) continue; __folio_set_slab(slab_folio(slab)); -#ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long)&kfence_metadata_init[i / 2 - 1].objcg | - MEMCG_DATA_OBJCGS; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts | + MEMCG_DATA_OBJEXTS; #endif } @@ -649,8 +649,8 @@ static unsigned long kfence_init_pool(void) if (!i || (i % 2)) continue; -#ifdef CONFIG_MEMCG - slab->memcg_data = 0; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = 0; #endif __folio_clear_slab(slab_folio(slab)); } @@ -1143,8 +1143,8 @@ void __kfence_free(void *addr) { struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr); -#ifdef CONFIG_MEMCG - KFENCE_WARN_ON(meta->objcg); +#ifdef CONFIG_MEMCG_KMEM + KFENCE_WARN_ON(meta->obj_exts.objcg); #endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index f46fbb03062b..084f5f36e8e7 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -97,8 +97,8 @@ struct kfence_metadata { struct kfence_track free_track; /* For updating alloc_covered on frees. */ u32 alloc_stack_hash; -#ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; +#ifdef CONFIG_MEMCG_KMEM + struct slabobj_ext obj_exts; #endif }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5b009b233ab8..aca777f45d34 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2859,13 +2859,6 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) } #ifdef CONFIG_MEMCG_KMEM -/* - * The allocated objcg pointers array is not accounted directly. - * Moreover, it should not come from DMA buffer and is not readily - * reclaimable. So those GFP bits should be masked off. - */ -#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | __GFP_ACCOUNT) - /* * mod_objcg_mlstate() may be called with irq enabled, so * mod_memcg_lruvec_state() should be used. @@ -2884,62 +2877,27 @@ static inline void mod_objcg_mlstate(struct obj_cgroup *objcg, rcu_read_unlock(); } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab) -{ - unsigned int objects = objs_per_slab(s, slab); - unsigned long memcg_data; - void *vec; - - gfp &= ~OBJCGS_CLEAR_MASK; - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - slab_nid(slab)); - if (!vec) - return -ENOMEM; - - memcg_data = (unsigned long) vec | MEMCG_DATA_OBJCGS; - if (new_slab) { - /* - * If the slab is brand new and nobody can yet access its - * memcg_data, no synchronization is required and memcg_data can - * be simply assigned. - */ - slab->memcg_data = memcg_data; - } else if (cmpxchg(&slab->memcg_data, 0, memcg_data)) { - /* - * If the slab is already in use, somebody can allocate and - * assign obj_cgroups in parallel. In this case the existing - * objcg vector should be reused. - */ - kfree(vec); - return 0; - } - - kmemleak_not_leak(vec); - return 0; -} - static __always_inline struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) { /* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in - * slab->memcg_data. + * slab->obj_exts. */ if (folio_test_slab(folio)) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; struct slab *slab; unsigned int off; slab = folio_slab(folio); - objcgs = slab_objcgs(slab); - if (!objcgs) + obj_exts = slab_obj_exts(slab); + if (!obj_exts) return NULL; off = obj_to_index(slab->slab_cache, slab, p); - if (objcgs[off]) - return obj_cgroup_memcg(objcgs[off]); + if (obj_exts[off].objcg) + return obj_cgroup_memcg(obj_exts[off].objcg); return NULL; } @@ -2947,7 +2905,7 @@ struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) /* * folio_memcg_check() is used here, because in theory we can encounter * a folio where the slab flag has been cleared already, but - * slab->memcg_data has not been freed yet + * slab->obj_exts has not been freed yet * folio_memcg_check() will guarantee that a proper memory * cgroup pointer or NULL will be returned. */ diff --git a/mm/page_owner.c b/mm/page_owner.c index 4e2723e1b300..de6ea5746acd 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -372,7 +372,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, if (!memcg_data) goto out_unlock; - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) ret += scnprintf(kbuf + ret, count - ret, "Slab cache page\n"); diff --git a/mm/slab.h b/mm/slab.h index 799a315695c6..5a47125469f1 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -96,8 +96,8 @@ struct slab { #endif atomic_t __page_refcount; -#ifdef CONFIG_MEMCG - unsigned long memcg_data; +#ifdef CONFIG_SLAB_OBJ_EXT + unsigned long obj_exts; #endif }; @@ -106,8 +106,8 @@ struct slab { SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ SLAB_MATCH(_refcount, __page_refcount); -#ifdef CONFIG_MEMCG -SLAB_MATCH(memcg_data, memcg_data); +#ifdef CONFIG_SLAB_OBJ_EXT +SLAB_MATCH(memcg_data, obj_exts); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <= sizeof(struct page)); @@ -429,36 +429,106 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } -#ifdef CONFIG_MEMCG_KMEM +#ifdef CONFIG_SLAB_OBJ_EXT + /* - * slab_objcgs - get the object cgroups vector associated with a slab + * slab_obj_exts - get the pointer to the slab object extension vector + * associated with a slab. * @slab: a pointer to the slab struct * - * Returns a pointer to the object cgroups vector associated with the slab, + * Returns a pointer to the object extension vector associated with the slab, * or NULL if no such vector has been associated yet. */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { - unsigned long memcg_data = READ_ONCE(slab->memcg_data); + unsigned long obj_exts = READ_ONCE(slab->obj_exts); - VM_BUG_ON_PAGE(memcg_data && !(memcg_data & MEMCG_DATA_OBJCGS), +#ifdef CONFIG_MEMCG + VM_BUG_ON_PAGE(obj_exts && !(obj_exts & MEMCG_DATA_OBJEXTS), slab_page(slab)); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, slab_page(slab)); + VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); +#else + return (struct slabobj_ext *)obj_exts; +#endif } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab); -void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, - enum node_stat_item idx, int nr); +int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab); -static inline void memcg_free_slab_cgroups(struct slab *slab) +static inline bool need_slab_obj_ext(void) { - kfree(slab_objcgs(slab)); - slab->memcg_data = 0; + /* + * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally + * inside memcg_slab_post_alloc_hook. No other users for now. + */ + return false; } +static inline void free_slab_obj_exts(struct slab *slab) +{ + struct slabobj_ext *obj_exts; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + kfree(obj_exts); + slab->obj_exts = 0; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + struct slab *slab; + + if (!p) + return NULL; + + if (!need_slab_obj_ext()) + return NULL; + + slab = virt_to_slab(p); + if (!slab_obj_exts(slab) && + WARN(alloc_slab_obj_exts(slab, s, flags, false), + "%s, %s: Failed to create slab extension vector!\n", + __func__, s->name)) + return NULL; + + return slab_obj_exts(slab) + obj_to_index(s, slab, p); +} + +#else /* CONFIG_SLAB_OBJ_EXT */ + +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) +{ + return NULL; +} + +static inline int alloc_slab_obj_exts(struct slab *slab, + struct kmem_cache *s, gfp_t gfp, + bool new_slab) +{ + return 0; +} + +static inline void free_slab_obj_exts(struct slab *slab) +{ +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + return NULL; +} + +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM +void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, + enum node_stat_item idx, int nr); + static inline size_t obj_full_size(struct kmem_cache *s) { /* @@ -526,16 +596,15 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, if (likely(p[i])) { slab = virt_to_slab(p[i]); - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, - false)) { + if (!slab_obj_exts(slab) && + alloc_slab_obj_exts(slab, s, flags, false)) { obj_cgroup_uncharge(objcg, obj_full_size(s)); continue; } off = obj_to_index(s, slab, p[i]); obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; + slab_obj_exts(slab)[off].objcg = objcg; mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), obj_full_size(s)); } else { @@ -548,14 +617,14 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; int i; if (!memcg_kmem_online()) return; - objcgs = slab_objcgs(slab); - if (!objcgs) + obj_exts = slab_obj_exts(slab); + if (!obj_exts) return; for (i = 0; i < objects; i++) { @@ -563,11 +632,11 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, unsigned int off; off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; + objcg = obj_exts[off].objcg; if (!objcg) continue; - objcgs[off] = NULL; + obj_exts[off].objcg = NULL; obj_cgroup_uncharge(objcg, obj_full_size(s)); mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), -obj_full_size(s)); @@ -576,27 +645,11 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, } #else /* CONFIG_MEMCG_KMEM */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) -{ - return NULL; -} - static inline struct mem_cgroup *memcg_from_slab_obj(void *ptr) { return NULL; } -static inline int memcg_alloc_slab_cgroups(struct slab *slab, - struct kmem_cache *s, gfp_t gfp, - bool new_slab) -{ - return 0; -} - -static inline void memcg_free_slab_cgroups(struct slab *slab) -{ -} - static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, struct obj_cgroup **objcgp, @@ -633,7 +686,7 @@ static __always_inline void account_slab(struct slab *slab, int order, struct kmem_cache *s, gfp_t gfp) { if (memcg_kmem_online() && (s->flags & SLAB_ACCOUNT)) - memcg_alloc_slab_cgroups(slab, s, gfp, true); + alloc_slab_obj_exts(slab, s, gfp, true); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), PAGE_SIZE << order); @@ -642,8 +695,7 @@ static __always_inline void account_slab(struct slab *slab, int order, static __always_inline void unaccount_slab(struct slab *slab, int order, struct kmem_cache *s) { - if (memcg_kmem_online()) - memcg_free_slab_cgroups(slab); + free_slab_obj_exts(slab); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), -(PAGE_SIZE << order)); @@ -723,6 +775,7 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, unsigned int orig_size) { unsigned int zero_size = s->object_size; + struct slabobj_ext *obj_exts; bool kasan_init = init; size_t i; @@ -765,6 +818,7 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, kmemleak_alloc_recursive(p[i], s->object_size, 1, s->flags, flags); kmsan_slab_alloc(s, p[i], flags); + obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); diff --git a/mm/slab_common.c b/mm/slab_common.c index 9bbffe82d65a..2b42a9d2c11c 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -206,6 +206,53 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, return NULL; } +#ifdef CONFIG_SLAB_OBJ_EXT +/* + * The allocated objcg pointers array is not accounted directly. + * Moreover, it should not come from DMA buffer and is not readily + * reclaimable. So those GFP bits should be masked off. + */ +#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | __GFP_ACCOUNT) + +int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab) +{ + unsigned int objects = objs_per_slab(s, slab); + unsigned long obj_exts; + void *vec; + + gfp &= ~OBJCGS_CLEAR_MASK; + vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, + slab_nid(slab)); + if (!vec) + return -ENOMEM; + + obj_exts = (unsigned long)vec; +#ifdef CONFIG_MEMCG + obj_exts |= MEMCG_DATA_OBJEXTS; +#endif + if (new_slab) { + /* + * If the slab is brand new and nobody can yet access its + * obj_exts, no synchronization is required and obj_exts can + * be simply assigned. + */ + slab->obj_exts = obj_exts; + } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + /* + * If the slab is already in use, somebody can allocate and + * assign slabobj_exts in parallel. In this case the existing + * objcg vector should be reused. + */ + kfree(vec); + return 0; + } + + kmemleak_not_leak(vec); + return 0; +} +#endif /* CONFIG_SLAB_OBJ_EXT */ + static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset, From patchwork Tue Oct 24 13:46:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0097C00A8F for ; Tue, 24 Oct 2023 13:47:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 709AA6B0261; Tue, 24 Oct 2023 09:47:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EB806B0266; Tue, 24 Oct 2023 09:47:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 440856B0269; Tue, 24 Oct 2023 09:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2C2446B0261 for ; Tue, 24 Oct 2023 09:47:02 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id F1CC9A017F for ; Tue, 24 Oct 2023 13:47:01 +0000 (UTC) X-FDA: 81380481042.20.9E754F8 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 42416100030 for ; Tue, 24 Oct 2023 13:47:00 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=E7C6lgL3; spf=pass (imf14.hostedemail.com: domain of 308o3ZQYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=308o3ZQYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=re9qyJoQbXM1lB3hu8WMyX+cG50piCMxG0UfNqXuCfM=; b=J777juFOACs9JrLvbt3Fkv1OU/Er/2hw42IBMe4WyjbMcPbPBujgHTpRJNLHWD+b1kk1S9 /9mVfP8DMACOuLNoXrmC1/O9zBdETFS4EhQbogNdl6DQxqpEof0VK14P9AEegczbUtpUzD 4uKKNqI7/wrQ4Jj3Z6VMoOaB/+yPWwg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155220; a=rsa-sha256; cv=none; b=oskjztWWQOwQkBTAgR+ccgw8pVwKW2SWtso/HR67g6ywNHo4rlCtocG/CWcmj51tULYTEw Z5F00kVtTY5HUEoa9eeH5dmPqMXqdR6yLZuZyLC3pBGODVZRiTE6jxhJMhcrZJVwv8KQSI dH/aila1R5g47Iw8rAGGXbJa9uvMh3k= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=E7C6lgL3; spf=pass (imf14.hostedemail.com: domain of 308o3ZQYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=308o3ZQYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a81cd8d267so84822307b3.1 for ; Tue, 24 Oct 2023 06:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155219; x=1698760019; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=re9qyJoQbXM1lB3hu8WMyX+cG50piCMxG0UfNqXuCfM=; b=E7C6lgL3/KxPgxT+jT1asvxZ/ITlQPmmvXUo65kNf0RozBsmpg0SVoFEtCK0NUIzuY 775ngCA9/0UPbyPPwZAz1T5911n8Haq4QpfAoRuTxOGr+NzlMlT2yxvthLtZEX6HEj7H swBTg1cyVjnyIRAvvHHF9AwpokyMOTqUNbdmTnPXKpb9vFcas8CK9hRIC2xdYBNwFSKD 236Cxkzb9JOcdpz2JVxR7Pgjp8pYhIwsEZwb1doiUlqDJwMF0jR6Yp73dr8tXWwZ4n/l qm9xEM9/kIE6P2scr9m8botVxJb4SBhMCYp7dpfiemFrFZL/HH42QEswv/GEQEXbC/TV 8SrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155219; x=1698760019; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=re9qyJoQbXM1lB3hu8WMyX+cG50piCMxG0UfNqXuCfM=; b=QkxbDq6dFvhx+Io3G4R+f6fWMqKpwLjbgeZO4cYcOxzg9WUDeayKlCtUxsAZmG0Vxy 9S3MjHml8Z9uLPRb9D1ZIX6+3b6PeXWuMWTSawoz3dAN5iy+8nE9EQZf61dH9w/QyBl8 3SlRpQIcE4mVGa/OOmA4GUKVRVqDYCFgLed5CsLxWg91Rj/oQZtVKTF2SjC9ULue3tyK xf0zqtJICyS9JmGpURrl8SImvef9SN5dhSWi7ex7+xQ+YL+KWUsCltKwlB1lg4VnQoGp Qq2R4SbsVTtWC54SI5M9Z5+CE6KgEGXFSPjyWKGrnE/UNJBFA3FEcPb0xw1j3edNrQDX 0g9Q== X-Gm-Message-State: AOJu0Yxev+XoPztgzrVg61dfYMp7MQyVx1mAblK5qBxNcp0nseCitdI2 ZTxvaECIL3bdrcrJTGim7YtEn3bqMRY= X-Google-Smtp-Source: AGHT+IGJ25fW4BnP1NJYhrzH0X+zrhr1/eHne8PBOna8hmvrezOwpbGVvOl7JHFMC4rGhoY/sRPJncXHlfA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a05:6902:1083:b0:d9a:c946:c18c with SMTP id v3-20020a056902108300b00d9ac946c18cmr311395ybu.6.1698155219188; Tue, 24 Oct 2023 06:46:59 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:05 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-9-surenb@google.com> Subject: [PATCH v2 08/39] mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 42416100030 X-Rspam-User: X-Stat-Signature: shaou4inhd3peckptq77ga9mq5dwks8c X-Rspamd-Server: rspam03 X-HE-Tag: 1698155220-987299 X-HE-Meta: U2FsdGVkX19pn1IxEXmL8C3aDGP2PqsN16gtJIpV2qa9tAkWlGPTYdRvqtKThGMja8n8TwbxVLK9LJZF8MzoJOd5zVGVDAomjDCNcJAheuN68iVz6SLm+ASh6K5GQrOgUPPg/qvuAYXcG4ET2zJn775Dn7HTxo220pc1WGUw//DNu1xf+JEVXtvE8w2gZ+LCnWV3f8XmY+uNc8YHOn3hyn/cRpki1iZNff5FvFxkQ8E3En/l93rpiWQwHbwcnq6QR6wFQxRsBqDTppkHcmL6Ih0YklAAdcexWdAHeI3cv05+9ufufYN8Qp8TosffsHgX0P6uswxkWB/D6sW6bV6HphA3EzG/4ILJWfUB4nH4p/yVPj2miktc7fJI1oSBUlgoaHUqFSCabcmms2h3a3U3RXxADsdZqiWAvh5+6lPfWD2BemSwhv7Z6o0wxhhs238F6ITZ5m0iR6sUYGAyes6DjcVdn44OYkSj4iFGWFc4b8kadeQnzZXgWr9g0rTQdqCB//aZ9EW2FJqGzUrLM0wlEPVh8r/1sj+AJZYujh9zTMLTPylktBRptFuiM9x1dIVlGM1bGYt+ckFHIs4Zmh8cWXgxJtCtX5Vqs1EpDUokocBVknhlEERFj6f35F8QKoPaYKv07VFFT/4c4quAWKRS6c9CbbJF990+MBeUgu0Le2D2QMxZCwEcE3b2aND7ECAjqHz+MejvrZ/teFHBneqAqaRURzjVtn+Apajbtu+WODZG+q5xQUTMp8dP1PZprSEsc0pxqZouH+gxFVprKgVeT4d9VlDf9t7pP4QoCwxDyS6qhlF0lN9OKr0tPEj/cNV9WJgVIepQTs2Wjf9gWN04SWX7qzbpQMLO69knn2nN4YdKguV1tKZpqIhA+88PdhOmELzwfiyjaEChlhJejbvPBcOw8KB/Zm7mOvDgqBq9t5CDCTWz7wZHu98M7tNzZCVMiW1o8LdwpFkRi8AMvzb /pSkUMUM I88XfJUG2MyuIqTSVQ+H/d8b70k9s2lXD25VygtKENKFyDW0l9ziFnbdb9fjATjwvHOqDPr4IyGYfm1yoGpcJQE0WBWNQVIopaC5R2w5A0YmCYRRv17l/U4Lp9vSPkUm0iH6JWUC9lcO1/QZieLfrMPW4Z83SqTxpoji+vsA7KzuU7DXIkLV7f/X0FEcBymuajzTIiCDvLd/4m1xHnCszh51zjlF75ywc8iZORT8okWxieOkBCL0PMzocL4B4mRS9EyJASI4JhsuwXUZMJ3D5cTO9RJuxH4njqSw7mdhH65JTFK5/qkvvPUdlxIPcKDVqKeHjQd78ugPQumxIUxpTeBtT0Px+sUPw+E0A7jCbJxj4aJNiEIV02Qs/aKoNrQfXWtd5tg59+n9xQrwExi1AyH5zUYD3eU+JnMWSOd4zM4fAqjg0AmKvb9eWfcQQcrKqrfgyhktwQZmlWlL5NghnFBMblpqlreObslQejE/95iljGTRZXUAmVqfmztvYtTwXAv3unoF/ps5QofC3JXmGtzaqoWEehmUdjaMEtzxrEHBX/JmAzSsMjmh2qtno2h0QaYDqxejwollg0GPXEngQL78+cvcNYGrqe/vCjDsavvIN2r7EdmfXNvrMOgEQxJ9xsAl2EOYwy3UrAQYCu5vF8NwYg3MSXgzIxPVRIsfvHLnVBK5o6Ok9i0kW/gL+dksNJt9RyOz25mC/H7L5MypSwrdGww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce __GFP_NO_OBJ_EXT flag in order to prevent recursive allocations when allocating slabobj_ext on a slab. Signed-off-by: Suren Baghdasaryan --- include/linux/gfp_types.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 3fbe624763d9..1c6573d69347 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -52,6 +52,9 @@ enum { #endif #ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, +#endif +#ifdef CONFIG_SLAB_OBJ_EXT + ___GFP_NO_OBJ_EXT_BIT, #endif ___GFP_LAST_BIT }; @@ -93,6 +96,11 @@ enum { #else #define ___GFP_NOLOCKDEP 0 #endif +#ifdef CONFIG_SLAB_OBJ_EXT +#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#else +#define ___GFP_NO_OBJ_EXT 0 +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -133,12 +141,15 @@ enum { * node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. + * + * %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) #define __GFP_WRITE ((__force gfp_t)___GFP_WRITE) #define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) #define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE) #define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT) /** * DOC: Watermark modifiers From patchwork Tue Oct 24 13:46:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9A3AC25B48 for ; Tue, 24 Oct 2023 13:47:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E7476B0269; Tue, 24 Oct 2023 09:47:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96F536B026A; Tue, 24 Oct 2023 09:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79A246B026B; Tue, 24 Oct 2023 09:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5D22D6B0269 for ; Tue, 24 Oct 2023 09:47:04 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 333E41A093D for ; Tue, 24 Oct 2023 13:47:04 +0000 (UTC) X-FDA: 81380481168.27.88C5FCE Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 81978C0015 for ; Tue, 24 Oct 2023 13:47:02 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d9biDiIg; spf=pass (imf28.hostedemail.com: domain of 31co3ZQYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31co3ZQYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BlkdsRQXiDN3rg6CmmMqMbDupjvE+Qo4H/i7Q99VP7E=; b=4X3MqUG840xLzZ89IVUMoyj1N3Vibt8h2RXQ8srhAf63HM2jWejXd1f4EVu5l9ZZaSo63u AuSboxrDCisXJ5QNGOHkHN7lS2uPlr6uWTNACxdElqiwj4G28/iMcDbf3eYjvxK+nwIQnJ JNLP3ZpU7SlP2qnSpKMl7qD59qIG+6s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155222; a=rsa-sha256; cv=none; b=uB4u46t06ZddjyM/NjGeZ77mbk6Z3HMVMLMZ0MDOs+F88oOZmatn9v0zyRyOqLG9rlvtMU 8GvcGBHoN46QJ/n5upUR8jFhm20X4ZeFYFvZLiE24KCET+rOZtTWhZGBhOX5a3l4UmlLzz ylwNhSJWUQsbvfZZmozdK0mGdpiSUV4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d9biDiIg; spf=pass (imf28.hostedemail.com: domain of 31co3ZQYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31co3ZQYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5a7b53a48e5so59332537b3.1 for ; Tue, 24 Oct 2023 06:47:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155221; x=1698760021; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BlkdsRQXiDN3rg6CmmMqMbDupjvE+Qo4H/i7Q99VP7E=; b=d9biDiIgURrh97vUF6hS51AyL9BGYBqpFnfutoG4qmyjwq+IMlp7nHHHquV4d+CktV 7LncsKLsuNR1hffFDbxmR8uJ3WwIJfZ/enpFodC73d1urIGfky14C6SSlOlZ9Wm4Ki8p oUyl7h6Qobc48yeTbj7V3cB84UqWe/f+0dtOU1oif67qgxd4A2SADqP/KnpJWODzgrAQ 0vak+63pwa4IdWNpSfPJQ4AY4bGcHZgovxmxgmajeLBFqVyUWg7dlkmgxSGnp7hJ1L4X PjpeI4sxbFatAvLVcA1DP9VrrU+/442PqrOzGiaVnrb6lTpyKPlCHdk4uitosKJpe2Cf ogUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155221; x=1698760021; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BlkdsRQXiDN3rg6CmmMqMbDupjvE+Qo4H/i7Q99VP7E=; b=WrspXN+g3PwNJP94a38bpenqHbcNQLiIu5ph1Ad93GkfEn9KEcwC2gTGPLog90PIrL zRZTwAIU+LQcQA+IndL2lnmoIiBClK2ir3Gw8d5iQ25nzNfKX9WiffA0nBsmvjuzud6l /fhfueMtyum1cGeRFzvEM7FFAOs8OOTvgJ5lUdSsKJxou5hpbqbeYzxaky0tVfqoSwtk X8SISP/YLIrInqiLkGZ1OLY+505ZJMrn612+7zx0c04sRb/P4WhI5tck6azXmWCKzNeb Pw/oya2/heeRQtYlsly/j7pVO1lt4q2IYVxkOexRaYcvQs3yBZL0MrTTDmxcoZLmpRDW nD2w== X-Gm-Message-State: AOJu0YxtvCASIH7YeioC78wHEzwQafztULrvmvVD5ZzvcGj2+qRnQygJ beGS1MZAm5ibk2cSxrXrXaRfDGNjDxM= X-Google-Smtp-Source: AGHT+IH4vkQmrXlidCVXKzYYty2ay4HPBInW5lCi3Eqpl3vZeKIQVqiJUvatNMDtWsMNHeDdaowbWxA4kC0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a0d:d50f:0:b0:5a7:be3f:159f with SMTP id x15-20020a0dd50f000000b005a7be3f159fmr287305ywd.5.1698155221495; Tue, 24 Oct 2023 06:47:01 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:06 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-10-surenb@google.com> Subject: [PATCH v2 09/39] mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: 9tmeheic9cfkyyaz4aickk1u4xogw54u X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 81978C0015 X-Rspam-User: X-HE-Tag: 1698155222-209666 X-HE-Meta: U2FsdGVkX1//CQTXzd6O/ADxu+xatackh9VlyQWzPWXVFifDC/40P/tEuUa84QyGNoRBeHWFuAdCDeGhutPYdN/TJEqlEn4nJfEgts3OmX2dISphew3l0XODY2oTdS15Mgst87KlvEkJ6dfJ3HYGTEAedWhnzUvJM2pibNZmhAAb9r4t1VogEFNLVhLvNEub9dlGuGlpJf8CesE201z908utsTtweDWuMdP5qNpibiAEO9ITwRYynbFTdBcV37QAXQerzYnQ/7y2ggUz1mPfPydI/P9JW07wIqyFqHaR9tNAuCYFuyNr491qdgBjCcujynqibtt6fTvZ+0e1affHtHpZoydESoBR328ouhUWpKZM9MOzGN/qrNYTh7u99WPQBckq5qMh/64L7J4GOE2bLmGhcHRbEnOvQBfY8A1HRtckumh9ac5b7ZH3iX7WfdMZNm7NcVSKllHnzBmG9JXCh0j7kRuSNKqhpQkmMe1VLYME+JHPCo2Ia8VMohKCqTQXSjExagWaLhC3tbjPwLu1f3a4V4098tsNbTTJpwxhEBojM7l/zmoupyr4zm0FwCheAyJt3WwHWEIIHXAmnNjYcQBN4dqXOJsQuRuvaEOWCePw7SyOfRlaEV+UvCAzxOQveIZW3NlFqqd6DlpaoXmz03JalUCQ2hoorfhPyEF/Ww3GG6W8T1Eg/hjlOsFVpfs2UKFL0RZVVum2P+S/y9JX18F+j6QS2nsKh6EQjgif1IuwxHJdJnDLJreDvpg403SynaLpi3dY6xDFEF7RXk3GabG+XEBqx1gnWsXzSw9BIK15Oy/wRkecf2pxDOSzlBjYw6H0fnG2s+4/7l189fn97gcQqBMdY6cKNkPFByXYXXD0uJCqnJLjVv0uHWzbk5zgdh93vtNaYyHaTHT/VR/Ta5QOzbRfHpfX052lDqU97XyKU19RAUwY/EZbD5dqDnhxIeMP1F/GJY6sRd/S878 Oagi4iBw xVwRgTfUgduqGauC6HiWtXNLFmEAOUlTQrirQKyHpVKJW8N4SFZ3OYzKfbBkDn5B8k8OQjBse79sZfhfj0L3jGZSThf1kk6cv3jYwUPYmvnxIqfclbjdxz8AtS95QjGIS6X3exocKgyzl4nqURV7R0EUI+7dsoYNnbCUS5YrhcEbLejZ7c9ZIA0IiJePrbT/60vE6KvJQzNsGh+B9vEDExL/f8rApN02Yu9cCy7Tp9y8xAEgpO7OThF2la4QYgD95v5uS7zUpRpUtrjXKgRGjt+GR0AxH9Or2i5uufVpbfL1YdzAP85g3Upv4+F/9TJh1b0Ij1dHb9vs/WcRMTAI2lLhm3SMIZc9C3i8Y4RRSJyIlmkgj2r7ALehaFpiw2cjcuCu+fY2g23lWxrmxnn5m4fckLceNbeE1yJZw4WIQtSDflzpUGl6XJD/buhx2RclqTs1LSM1Gzfxe+53ktuj26glxGEvie/1C59O/jcSpGsM6rfUswX0o7NUyB+opiIXcCGFOyJ718086lu9ARCr1sjx3PsXpjX0nAu8FknOXTGfNFy3Ka3Gk4DeJiG6331SlxyOGqejyZG/uMSgH7/fEgYT6iN5YHqFUFMR+CVM79QZibXIbtZ6VQSrRMUs+g9uV009CNuw5w4EKh4oIKjmVLLJENZ5d+bWSiIEkMPpiz5dt8dv0JAcmJ0tuwYbb9Z6Cq95s8QQODkLz1BK8Bt2kSpqkdw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Slab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like kmem_cache and kmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. Introduce SLAB_NO_OBJ_EXT slab flag to mark these caches and avoid creating extensions for objects allocated from these slabs. Signed-off-by: Suren Baghdasaryan --- include/linux/slab.h | 7 +++++++ mm/slab.c | 2 +- mm/slub.c | 5 +++-- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 8228d1276a2f..11ef3d364b2b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -164,6 +164,13 @@ #endif #define SLAB_TEMPORARY SLAB_RECLAIM_ACCOUNT /* Objects are short-lived */ +#ifdef CONFIG_SLAB_OBJ_EXT +/* Slab created using create_boot_cache */ +#define SLAB_NO_OBJ_EXT ((slab_flags_t __force)0x20000000U) +#else +#define SLAB_NO_OBJ_EXT 0 +#endif + /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/mm/slab.c b/mm/slab.c index 9ad3d0f2d1a5..cefcb7499b6c 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1232,7 +1232,7 @@ void __init kmem_cache_init(void) create_boot_cache(kmem_cache, "kmem_cache", offsetof(struct kmem_cache, node) + nr_node_ids * sizeof(struct kmem_cache_node *), - SLAB_HWCACHE_ALIGN, 0, 0); + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); list_add(&kmem_cache->list, &slab_caches); slab_state = PARTIAL; diff --git a/mm/slub.c b/mm/slub.c index f7940048138c..d16643492320 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5043,7 +5043,8 @@ void __init kmem_cache_init(void) node_set(node, slab_nodes); create_boot_cache(kmem_cache_node, "kmem_cache_node", - sizeof(struct kmem_cache_node), SLAB_HWCACHE_ALIGN, 0, 0); + sizeof(struct kmem_cache_node), + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI); @@ -5053,7 +5054,7 @@ void __init kmem_cache_init(void) create_boot_cache(kmem_cache, "kmem_cache", offsetof(struct kmem_cache, node) + nr_node_ids * sizeof(struct kmem_cache_node *), - SLAB_HWCACHE_ALIGN, 0, 0); + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); kmem_cache = bootstrap(&boot_kmem_cache); kmem_cache_node = bootstrap(&boot_kmem_cache_node); From patchwork Tue Oct 24 13:46:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18CAEC00A8F for ; Tue, 24 Oct 2023 13:47:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 223196B026B; Tue, 24 Oct 2023 09:47:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 186546B026C; Tue, 24 Oct 2023 09:47:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA18C6B026D; Tue, 24 Oct 2023 09:47:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CE1EB6B026B for ; Tue, 24 Oct 2023 09:47:06 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A96D21201F6 for ; Tue, 24 Oct 2023 13:47:06 +0000 (UTC) X-FDA: 81380481252.28.DD047C9 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf04.hostedemail.com (Postfix) with ESMTP id DBD5C4000D for ; Tue, 24 Oct 2023 13:47:04 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1lPdMVO2; spf=pass (imf04.hostedemail.com: domain of 318o3ZQYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=318o3ZQYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155224; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YHgfLVMScT5H/FxRnxJh+3xP9v/l17ghYQJMJklbWxc=; b=WS0/hVDpfR2rrQej0jer/pLt2pb36/kKVHEUQLK45Ha0neDl6xUbPXGMY1kKQiPc9hdJCS 8pEZvaO3msdGbin0XZ6N2gJf/oCpqduZB4w7r0TIRrmrmmf2RsrWKJAwbglBIinKFNtTCz p5hZYuVkppjwRzkE5jerjTkJO4Ys0UY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155224; a=rsa-sha256; cv=none; b=n+ISNYKs4vX8BJ9uOIpJP/Ctw/xDlUIqYss9VruBcwNb0Q3h4+FEqdK68epWJRKMpYTpS6 n0Lbs6KKCnmtvS28Irq1nqqwWffdfZwe9NEps+gQC2+Jf+iKLiFnFapAMuojefhb4Bi+hn 08By+GfrJVwQBJ9cXd03/6jrmTSoYPA= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1lPdMVO2; spf=pass (imf04.hostedemail.com: domain of 318o3ZQYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=318o3ZQYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9cb4de3bf0so5405249276.0 for ; Tue, 24 Oct 2023 06:47:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155224; x=1698760024; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YHgfLVMScT5H/FxRnxJh+3xP9v/l17ghYQJMJklbWxc=; b=1lPdMVO24l9dsRE8+ytJXFTZOIUTDEgsr2EvbyprNiE87UFPFJQkl6OpgEBVmPrn1U rhf2Xv2pF0Yu78XXkQ0O5+P3TgMxJNRAchT0ODrtGruuTnel6SNEhQYu1IIXwQSk0RM4 QwRLn28Bcgp4Du5myBGli5G2bOj3JxQbF6XAsYqYJu2o+Dg3wfzXAsUKL4ENgWZDeHdk gxjUfO/+wW83LQDQC0nLdabr1qgfsXeW37MbtP6Vi1g6MJbYTCKtNJZ1otWo6VScg9jX eecOCcN8x1bXQ2h6QAdLQ/U9ZWNDvFcmYQsBkvQWXJfMYm5nA2pdvEMbJhEZO6gRkLWy AZ1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155224; x=1698760024; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YHgfLVMScT5H/FxRnxJh+3xP9v/l17ghYQJMJklbWxc=; b=WMfBaP6oEEwrJ6u4aIm+sWQIai0W0ICtc7HY6V62gpqQuV0p3FWl2aIPy8Zd8P8iCP SRhaMAQy10LboExiWev0vL/5vUjxzXIe5ChRgXODjrKpdsQrO1eES03iAhwgJtosVAz/ St4KxffBqAQ3pgpbzZwCgJ5acbWxinQrzvpzBHG9hgLypO+ufqsenq+MBe1CO/8XPmCR qGqw78lgktoB0WRqAkd2s7f0ra3SxBS2vBwjqt7Xo4+woB9kmkQtlyKNNJRiOqyC7h4N +ocTT3jYYgzW3Lm5yrnya8ksmha49dpDwcPFuE/Pkct/oFiw2cOHqxnvxvHQYY3fA5NJ GkzA== X-Gm-Message-State: AOJu0Yw5vI4LDsqMqEsT3Dn1lE8WNuiXOR6fj54eF31TkpdfAOGqooXp HRdAwoAZmfITErnF8dw4VCXZiybXca4= X-Google-Smtp-Source: AGHT+IHeBKvEZxCc8z4s40S7ha5hgPZX8U0NGngnhsFR9kbksVyPLDfdSNwpUqF/X+vAqjdciG/7nvEnCYM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:297:0:b0:d9a:4c45:cfd0 with SMTP id 145-20020a250297000000b00d9a4c45cfd0mr213074ybc.2.1698155223878; Tue, 24 Oct 2023 06:47:03 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:07 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-11-surenb@google.com> Subject: [PATCH v2 10/39] mm: prevent slabobj_ext allocations for slabobj_ext and kmem_cache objects From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: DBD5C4000D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: wu5wremdzuf9qsp1piofgu8snb1mw4z9 X-HE-Tag: 1698155224-123761 X-HE-Meta: U2FsdGVkX1+UQqb17MlUV+QZQYzMQddBKD1PnQL0lVhB+g7lEOlRE9W8WwY9D3PgsR+gSmfq8Akt9VXcWqA6j1AJECDNcet/FbcoioZ5OdNqQI6qhNj/uwDI5y2PYZox3OF01owjHRGzZ/x+0Fb4hxBs60UrRmI2IuBQR4hJDJ+GhoaPKBF1b1AG1sWlsv1jodGTD7tQ6J/Bjj0dMHYgle26vVZg3lQNiJ8Id2PmYODVqDHCrhW1JmzxI3wPRSGOHogKAP3f3iZyFQj8kLQ5UKVwTIrOyn4H0i9zVURXtDbTUfL8oFk+FFk3XOqH75lTxWXvnD+5DfgHLTqb0bG0iZBiJliMT6+NkXPj8RXbLFk8Yl7FIiMD4c740XJHbYDChdTJ2qRwza1uES9ELrRuTnrpMsmSCT2MRW4+0RLI9a+skWSYvHe9AOcpBnkJkANJnemF44nZ0XasmByN+YEFOoM1vWvRYwfqZAvDZtgzXWhUH2oT8D/OaVcBFTgko1S5kzCHKYJxW6MmBMhAT2aQ9qAdMEjdbic6i0genF0JiV7m27FwIGgJ8SlgEunRXxPV46W9RRiPVv1dZ3w0fNm+GKxBgbB8hx5GPL01p4KKT00rs/kzmwGvRy0awMAaePsHg6SPD2utz+MCbQ/3Dd4lcaC8ofHAJuZj/8XYBVbCm8jmOISpPbLXwgRzhrpYcASkaHADSGnj8Ua3vkFCf0iws1WbtoLys0SqwMlYoMTDCiDXc0cLV50cGoH39ON3pUCZoZDzsuhOeaDwUU8imtULxHL8NfMUzNYG0q0l7ekFe2kueMYKujKZALdIrXuS1ml2LPzkBByikUeKrXXkq3Yo5glxonHzTCmwtnLF/4LoBHzp5dXZCfWJkupr2mzu6KcDypPHkHwROCpibFx1/ZlpJ7Ia4VtJOVfRbinsicRRaQ1zTjJ93mTAbi44rAwe2Ex874QskX6hbP5dHfCtJkQ 3h9BJLXr ImU7XJmwBcAX9ozCsTXw8BOZhAXyWuIMBEKEAPc38yHy5nwlCmsciimTDesYtCxvDjmxeB9aZ4e2AIjAFJw8LQjt8RHKW/NfpMpqYxgviKiYj5brPE2cofEe1Gc5PcgABAh2f2XFToEyUGWB/FR/1ticmb01ooWn+833R53vorYA0a3oFvjBdoPUT0cqPdnDnQczi+k+K7YN+BKJdjo46qehZmjOMYAJa8Sv+GtgjjzPFHbHwljb5e4MJ3N6WyTpnJdzzy1suttFUBAC0K3kFuBhEzZj8YybDjcHO4zsJZHtwdpL4s1Ly9ux27CxopK55dL6oDudrLlmxrLB9oGfMVVbCl4DhSe3Dj2A3YTUlOIzWS2T7B8z1aJSqyvkdfpicVTbjE+TCd8bKEv8Tmu5leqnpzLBZ3lbR94UKfNYimHW5HK7OeshVoh4Z4sqD2A54yHg/C5pbwE0jYzRDvz4cSK6UgMCD2F2/J7SKDKkJMv51AW4JgpmOt2Vl4aDYWh5WFSxxFRDFvDydEj2WVH72BRi5pJY8H/3QIxQ9WcvrPD9qj/131vtUpNry+jAgiuBXtgvc2plsEOgNfbjy4WOGNdi4lZ1xaqXYoHvK4lSGoxk5mSXM4v89eigoMhYZsxROcxYgZXi9kPxRH73/6enjHXhUQH9Xmb2zIfqTN7LvVTHln8HfO5m6Oje2BGaMg0PQIH7hVvuyXpy5SDmhtEpo59ZPfPI34eC59rdh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use __GFP_NO_OBJ_EXT to prevent recursions when allocating slabobj_ext objects. Also prevent slabobj_ext allocations for kmem_cache objects. Signed-off-by: Suren Baghdasaryan --- mm/slab.h | 6 ++++++ mm/slab_common.c | 2 ++ 2 files changed, 8 insertions(+) diff --git a/mm/slab.h b/mm/slab.h index 5a47125469f1..187acc593397 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -489,6 +489,12 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) if (!need_slab_obj_ext()) return NULL; + if (s->flags & SLAB_NO_OBJ_EXT) + return NULL; + + if (flags & __GFP_NO_OBJ_EXT) + return NULL; + slab = virt_to_slab(p); if (!slab_obj_exts(slab) && WARN(alloc_slab_obj_exts(slab, s, flags, false), diff --git a/mm/slab_common.c b/mm/slab_common.c index 2b42a9d2c11c..446f406d2703 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -222,6 +222,8 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, void *vec; gfp &= ~OBJCGS_CLEAR_MASK; + /* Prevent recursive extension vector allocation */ + gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); if (!vec) From patchwork Tue Oct 24 13:46:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68144C00A8F for ; Tue, 24 Oct 2023 13:47:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F4136B026C; Tue, 24 Oct 2023 09:47:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 37B3B6B026F; Tue, 24 Oct 2023 09:47:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17ED66B026E; Tue, 24 Oct 2023 09:47:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EFE506B026C for ; Tue, 24 Oct 2023 09:47:08 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C84E6B5D9B for ; Tue, 24 Oct 2023 13:47:08 +0000 (UTC) X-FDA: 81380481336.01.107A8BC Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf23.hostedemail.com (Postfix) with ESMTP id C9C1D14001E for ; Tue, 24 Oct 2023 13:47:06 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YG+eh7VH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 32co3ZQYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=32co3ZQYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Qg2lup2zwYWRtm8BAuIZWN5zY45A/qiqWGd2wrDGPpE=; b=xBwY0Bh056u9MBfiE/D6e2pznnCqM3ASN+zRKApzVnrSKiybNm4m8iTiUF1dHUOLcHj16d g8KU2Ljn7DxyVgDKYicU1XfsCU5HAU66EqaV+LVcVP2nmn/561aM89Qqcj+TN3YAgZDTT8 0PxOV54BsOAOfBTvVcwtQI7Ywtos5hQ= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YG+eh7VH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 32co3ZQYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=32co3ZQYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155226; a=rsa-sha256; cv=none; b=le+yXwXlDO6aQwphm6a1uAqxDXZfDIStJwl1IcKw2OrXXrRmnuiYMZuUXztLTFp0/ML+Ol fCS+lR+6fvwyzkHQ0IOkdXlad8EQNjgZzDzpap1c7980nq4baZqXwME4o2KjM9s8y/PUm+ fugUVwnQw0zHODgTYndwu7AAijkoC0Q= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5a7d1816bccso59392007b3.1 for ; Tue, 24 Oct 2023 06:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155226; x=1698760026; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Qg2lup2zwYWRtm8BAuIZWN5zY45A/qiqWGd2wrDGPpE=; b=YG+eh7VHwfFk4oJ00sQcCvyufGDSixcVa833VhvSEo5aNTBk/wJ8zVPETmN6uO/9dh pA4CQKNgrS5n5JHBQqCIrME+dTuCaj78ufniauvEcYwZWEjDIZpy5NkqeeKor75KZ4f9 dhpg+sYrsgI4532lgRIklVdlXkCihSrvQglc5GwlmzDcunD3dga6zQGYuUir/USR+WzC fCwT5XTI1Mb0RyEJqCuiuFZ2VjY+ee7e7lLw/ZaTsQiDAQfbgRAEvcDaXGmhiOj/ioMV a/2V9ATOeNIrc4K30mERysrCeGaGM64wvh/bzIFeHtxQ3QbK04Ea5x23UTL6DYBO1z62 uPDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155226; x=1698760026; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Qg2lup2zwYWRtm8BAuIZWN5zY45A/qiqWGd2wrDGPpE=; b=poHVSfqkSgUjYl0cmdJRwxcxeV7KUAXHBJ8+1/bwOICCtXDG7GuxjRTWthTAmSE36G HBgJsAYh2yvoUA5uFlgZxpx6T31XGUg4lkxkVSo0pLF8N0UD96zf4O5ebv1gjSAORYJU baFvE72SnT44+fwU/Kwih//q/1ZmGJnLEOfr3D0fw+3CcZ81nZRDX8RCdNAqbOn7XZXk HMfPNAFskIKtH+C80wSjQ0AZSMMR/F9Wdo6z4CZePwidlW2z6KNRisFxZet9/gcmH3lP stFHt2eI4rp/quOoEPGWN93GaVmW0b3Z5vLWz9sgQ1rOwtPs7GdLLneocbzZAOXdrbPP b4Pg== X-Gm-Message-State: AOJu0YzeCnXGwHbFAPCUjf9TJlTTZe0V58zfssjZ4PqvdK4Q6+K6R/tw VpgbddE7nQ+MUfO1LgVcG8WFCTOIXoQ= X-Google-Smtp-Source: AGHT+IFbSeYSocyqc9oNK2jC1GEj7y9HsYgl5xw/vFv8t55K0y/nslFyvoCz7IdF8OEyu7ekDQ1W6y46+34= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a0d:dd02:0:b0:579:f832:74b with SMTP id g2-20020a0ddd02000000b00579f832074bmr286214ywe.10.1698155225916; Tue, 24 Oct 2023 06:47:05 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:08 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-12-surenb@google.com> Subject: [PATCH v2 11/39] slab: objext: introduce objext_flags as extension to page_memcg_data_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C9C1D14001E X-Stat-Signature: iy5ny1k3ms1png53ieanyu1e63dm6uye X-HE-Tag: 1698155226-752972 X-HE-Meta: U2FsdGVkX18wOQuAb+w2FXz+Z/Tg/BaeaOefUdyb4ZCU3Lk2HLSVwawBto+teFVB6RMyVBzkRU+MxxB1b6bxNzpyCscJKoS6XlyfBpwtcL57v6AxW6xytvBu3gmJAu/TJajI/lGZl9Y4Y/fdHV6OEFYLCmRi8Iiu4BV6abnuNBBdZ/qQYklBb6+ehwlY4bi34qwUg/wtbo/u9IDogEt0tjLEwb+BNJhhpDPUfdN5LoXUecfiGeiihsthzWUlMQISe2zFPyYQrLvrqvpRlgLk0ITeZ5CjwYROHMnKdcsAiZnYgo/4L/w7KJSD/Xe7RjkG3KWn9Z08WQL6rtWKIrvzoTH1HEyS89RL2TnwDvwbk9xndF5bAne4ITiRgguQdpYHLEytvRsWBIcQ2fGJp5mABK/FZdXjNKCcn7vVBClySvzm4drAOZEPPpYJDzlo9ORi9/PfRYhZ1xBU2dVXq5fxYQy5dGN0E8Er0LdSlfRSIioN/3DaB9l9gNrC2xKaP3IQKG5KwPcC6rVW8/PZHo76kKivoaMf+h6D3HnOVQwaYa7tb12WR4wh58cpA+Qji5NOdZQcTepmndOGszxNsKbMfRv2zpSjgxdioDPE/kZHbxaRE3mbnQA50lDkLw4q3xiY/mH7Yy0Sb44gmLuCM0ifDDGj6xpnhgmgt3Ngc3UtlS74FtPSTDvduYXbnInTQZEgf3YnQj0MVpX0IarCd3VmXmgXqKOjx1y3mqWvT8ghugxAiA2W+TDurLt2EtUiPBAqTi/AH82ULq6STfwowOB3c7NNNhP9H3KOX20cNNjeRXJIjYBPFs4WLjTFJehCzQTaEnh4Os3NHMfb8dBEllXnAuVyGEE0CHPwhFbrsJ5nAAHpa1vSf8qZtJpUuvDOZAX+CnzUw4PDZfCCS0RObTG6IbJCpu0P5Rt5O3NeeTdvEOVAZc7WYD2m3//2ITvL3poJdfwONh9368Ioq1f/Gc1 BT/6W0sO xqlwclcQAq4NXM3VLfdcoFOqiFqUzAgJx5H/wtKB+SDfLymDikKbc0Y3gF7QXdyGSzxMMXG52ic34Lj3WB0qnoU1XCiy8VFtSlrnP7JBvY1FxOfK59Kg3wZSXSHcpajKLgb0jCQIybhG0O/RFtVtHcnb5AfGfPHWBzB6VxNkoBz+wzwjtNh63s09ILB8Suo7hxZRLG0gpw+Z/+hInQGuuI/8YywXTJ3AG7/evqNin7MxXVzoJ+b7K1rGmPxDnXf6PnA6dX0AnJ9WC0RZDe/5aAGuuk4uG9a8A47/3IFV8qY61y4yobdbG2A0nle2DygKbfmRtv2LP1Bp0UqJifwf1Fv215kNd7+IkRznbFG02FzEc9UrreB4vaZRdX9u65CqfnqiOEw8T9tml6OJqW5QsEzKFIE+HbUL7oyc1Mfv9u3oubMLQquC6Ic09KRI1lOuiegYQXCDTEvIBTixaoanYVe8/S/uFLLYwbu4r89HrxYdgr/Pmw4E2gzM0u9CcZUUwObHQcVMUa/JLzcOxCFGPxo/+dTjyoncNG8kbMHnJVXoDbtj92BggBk2MrvqOdu1zTg5vZ93whN7gR3uy3uYrQZB+tE23fwPCAsUbhwX8iWCCprudrlXRwgewBnd0Oboiddw70sJCXh2IC/XkDUEwzCH8htTeAh/IfIAdVnHiXz6LHZcFSTOdN64474boSR3qrKcjfrO+fu0x4aA9vhx+q5crRMEqV+j8p2i/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce objext_flags to store additional objext flags unrelated to memcg. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 29 ++++++++++++++++++++++------- mm/slab.h | 4 +--- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4b17ebb7e723..f3ede28b6fa6 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -354,7 +354,22 @@ enum page_memcg_data_flags { __NR_MEMCG_DATA_FLAGS = (1UL << 2), }; -#define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +#define __FIRST_OBJEXT_FLAG __NR_MEMCG_DATA_FLAGS + +#else /* CONFIG_MEMCG */ + +#define __FIRST_OBJEXT_FLAG (1UL << 0) + +#endif /* CONFIG_MEMCG */ + +enum objext_flags { + /* the next bit after the last actual flag */ + __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, +}; + +#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) + +#ifdef CONFIG_MEMCG static inline bool folio_memcg_kmem(struct folio *folio); @@ -388,7 +403,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -409,7 +424,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); - return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -466,11 +481,11 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -509,11 +524,11 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } static inline struct mem_cgroup *page_memcg_check(struct page *page) diff --git a/mm/slab.h b/mm/slab.h index 187acc593397..60417fd262ea 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -448,10 +448,8 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) slab_page(slab)); VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); -#else - return (struct slabobj_ext *)obj_exts; #endif + return (struct slabobj_ext *)(obj_exts & ~OBJEXTS_FLAGS_MASK); } int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, From patchwork Tue Oct 24 13:46:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6000C00A8F for ; Tue, 24 Oct 2023 13:47:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B9B96B026E; Tue, 24 Oct 2023 09:47:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43CE56B026F; Tue, 24 Oct 2023 09:47:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2421A6B0270; Tue, 24 Oct 2023 09:47:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 06BD06B026E for ; Tue, 24 Oct 2023 09:47:11 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D476116087B for ; Tue, 24 Oct 2023 13:47:10 +0000 (UTC) X-FDA: 81380481420.11.4C0DA41 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf11.hostedemail.com (Postfix) with ESMTP id 246F94002B for ; Tue, 24 Oct 2023 13:47:08 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=b2uHif12; spf=pass (imf11.hostedemail.com: domain of 33Mo3ZQYKCIExzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=33Mo3ZQYKCIExzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9NoZOXA8j9UYH0qbl3Heb4+Wi1EaC598WUGqABNW4pA=; b=be4BAYwvCG9V7bnTIK8K2gDap0sUGm2IvTJCgl+ZHLox9tiHi38jCPPyFEd2yCMj2tUDfi f2968ACaAOD5Z14x5HLtOBgUYEJ3Q/gakfPVo4BnBS+6cQRRUNS6DBKv6f4oWDqDsb6o2Z j84RJN+y6P/73pUUHU6mdGTiLGEp7A4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155229; a=rsa-sha256; cv=none; b=oeCxk0Np19d9oayh9zxiL3UUwCKp8bup0ZwdHZi+INwhELTbT/9wWA2ep0Byw1Jy5sKzOG OAM1SSqueGjGD5y+Uhy4Yid3T2NHiO657vQGX0YLyBuu56ZF87aOyvfkTRZ5+ay2fvOSJw IhexTtORl3GRDnfu8KJokqNMrK7np+4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=b2uHif12; spf=pass (imf11.hostedemail.com: domain of 33Mo3ZQYKCIExzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=33Mo3ZQYKCIExzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5abfa4c10c4so26018187b3.0 for ; Tue, 24 Oct 2023 06:47:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155228; x=1698760028; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9NoZOXA8j9UYH0qbl3Heb4+Wi1EaC598WUGqABNW4pA=; b=b2uHif12riXAogF6ua4KeNMK2Y1Q2vL/Mhgmx+UgVAbnj75Pa36jD0u0qYoElPZdET TGYeDT/Yb/MT05BxnKgThUSL57Yz6GGu+nu41/ABoId+q/QNPfWbk0Z5sTHoMDE7o4lI IuHpWvfhFUEZqWcWnpzz/p7rVxzTkO4mXnM6wUciCcjtpQy56xs9afT8KJwKuAL/li28 C6VJJSTVyjMgMxoj/ciJRYUzUDPQ2AomjuVc6uOsqUtDRWNB7R5JHKX7zkaDHflzDztK BOfi2nigqhCCrIzbcms9ehgrIYLK/WgYPmjOCS9OA/ANIjGiDTsuDXVdRHKQoYgVOM7l Jegg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155228; x=1698760028; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9NoZOXA8j9UYH0qbl3Heb4+Wi1EaC598WUGqABNW4pA=; b=C0S6zKQNBGe5JQNqRSZXPFjJGpBZL5V33bNMKqwNS2NO0FJpgSv66AUw5YvVUgkv3u oaTC/GNX2rsijslb5RcwGb5BApX/+24GzOadPJV5gkl1IIh//55DrzV/BdfnsGfuc7ks b+6OG2yvQyRYZc8dPQk1fZ1MfSbTlCeiAheZM6DW5jNRwgCWakMLQQsck0NEvBYLGhKR RljKQyncTATOsHC6Pb0rXHB8MeCwNr9AaJstCEhs9ZliKydoi5QhP1yACqgSmf4cM6ia 6C8f5NVBdf+IIctyzsS48vQgLV9aSdtdGH1hJUWiDWKdKvvkhM+aDALrG6XHjY+Hx/Rl jxcg== X-Gm-Message-State: AOJu0YxnBsozJbVn4lYMtz8oN/Ojx2UDezfK4U7IJokfwLUnjpFUV1XB 4en7v47nsurkxUi9RILMIMv3cQuEszw= X-Google-Smtp-Source: AGHT+IEp6i7OIYLqR0zo2UBT4fCb6INeX9PpoJn2QVEvfaV9Q3dpXadwEMR0perH7aOFZkHGn0YJqemKbwE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a81:a08c:0:b0:57a:e0b:f63 with SMTP id x134-20020a81a08c000000b0057a0e0b0f63mr272098ywg.7.1698155228147; Tue, 24 Oct 2023 06:47:08 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:09 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-13-surenb@google.com> Subject: [PATCH v2 12/39] lib: code tagging framework From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 246F94002B X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ou5ydxkaq3bgi8gq817enbxu37aws5ry X-HE-Tag: 1698155228-897779 X-HE-Meta: U2FsdGVkX18/xbSgXbyUUhdubE8uTRcT7q1RQF8Y+jzUWyVaT/DNpGBCgZNq2zFE246IJL1qyMcO47W5fs9MRzD/NLhd6gTg1uanRwnYBe4RjrXfcq0wzwQwCgrPXt4Wkn78zshyurc535F4GUBF6WoZz/ZbC9poo79UmlvWzKWjBXuTCHfZcMZD28KD3SUbV612dboDOHrbijuVviSIgN9ZijhtbnVDB+NMKCk5yeWWAGGPPGwH7AilBruPhgT0znhSxZECDoVuscds3O8FszP0xTz7jWaYCqPfhbCzPgpVaYvmm6iKCasghYUsLwOKG0HEsWjiV+I6wCuSYT9Y5fkcua14XXJU/qjvS6HuidhxQm78Gjd+exfCNHWy1vzBq7HXF76xl1fLpWAwwVpw0UHh3iAT7vJTWo7MSrZ67LwkZMDpFwEOCl29iLyItT0y8uCBvcdEklIHGnjvKQqoYsyJ+BQakWPKd6oGv7JnHOmxScj6T75GdEgw7EmV+rRdB8vwYD9Ao7TVx9XBva7dek0Ljj/05rpVz50H7XewRHXqtfpussmIatkNXPtMHaZEOS7qfvkKFVboc341sc7gi+Djp++qlJlcNQQEP441K/U70l0iK6JGMr1z2X4FAWpsgdMoOLV2105PIiPXzbrDFE9QJsZ9kee4rGsOK5InaQ/ByrJ4HpKMbyKJ5fNHNJTZ+uFtyRu/s/8lLmlAYApaJKxY25H8F8GIh76Fev11NCezcwZMVPfUhbKtv1wQn6seA6sl0wmF0E9tPS/w9Pdh/FttBVx/o7COn4OTZcAu2kAkRh/6CayGtZe635hfg0sVak7mg4WgPUkk6njNvR5Ebrk78XqpzuzzMB+4yOtXbsiYRMRVyvopGU3mTc9sIs36fslPE1V7La4FTcw9MsK5OSdqONPyFTaf6ulQDa8dWCDB09nflWkM2Essn50bVQUdN6Qgsw8V04lnF4blJQM IueQ+3MV 9+ATdnX0oCZEErUi9SUybIt1+g+HmanmymPV582eYAlhAPl8xxHtVDZt2nq1d9us/f01J13pZhmF+ptytORvYmauDX/60hqVuP8vpx4294EBlB9vfydlCBQgpcwZyUKUD5jksNTmjUpJ/uLPEHYBVQAaDQizQc+b/AP+U+FHAYaIPF3gr7pdFsHay6/ANeCxPFp4M5km0F42lAnVcAdFkWqckmVZj484ohnAxpVlxJ0rnKIo0obn7M5LQmPY3fnU491Fsdo77KeJhP9cAd/1hxHKNWOwAZsEkGimGBiI08vIJD9T4saLu3FVqJjQRmar06fdvnZCAGv2bGh98xWUNxNMzmOVbU6/CP+GZQmDF/cDQebZgkmSxlq9pMVUd91YACNYJ20A8RMEXtLr6d63xOsitAbyU226i4op17JT0CAXODANQ4sNP9BZUSoB11yKbljPdPP75biI2DpeJSxgHozgZSlEJYOLTIgWWwy/XI5/W734u1SCxvpw8raKCuxzpzZIFL8EpkJwfsl25aJjfKSHLQJUif2spllIfn80emrNMYIJkmXNQTMrYfSB+9uHGc68acVnMeeA+IrxdWdaGyfGz8QfmbZAdr0FLgqF2lXgg8a2T5qBe4tEr/ZmWD1OjkyWuF3LVjo/dGBjNK/0fnEBbI97tmu+wHvIMnyqSmUjns7M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add basic infrastructure to support code tagging which stores tag common information consisting of the module name, function, file name and line number. Provide functions to register a new code tag type and navigate between code tags. Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 71 ++++++++++++++ lib/Kconfig.debug | 4 + lib/Makefile | 1 + lib/codetag.c | 199 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 275 insertions(+) create mode 100644 include/linux/codetag.h create mode 100644 lib/codetag.c diff --git a/include/linux/codetag.h b/include/linux/codetag.h new file mode 100644 index 000000000000..a9d7adecc2a5 --- /dev/null +++ b/include/linux/codetag.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * code tagging framework + */ +#ifndef _LINUX_CODETAG_H +#define _LINUX_CODETAG_H + +#include + +struct codetag_iterator; +struct codetag_type; +struct seq_buf; +struct module; + +/* + * An instance of this structure is created in a special ELF section at every + * code location being tagged. At runtime, the special section is treated as + * an array of these. + */ +struct codetag { + unsigned int flags; /* used in later patches */ + unsigned int lineno; + const char *modname; + const char *function; + const char *filename; +} __aligned(8); + +union codetag_ref { + struct codetag *ct; +}; + +struct codetag_range { + struct codetag *start; + struct codetag *stop; +}; + +struct codetag_module { + struct module *mod; + struct codetag_range range; +}; + +struct codetag_type_desc { + const char *section; + size_t tag_size; +}; + +struct codetag_iterator { + struct codetag_type *cttype; + struct codetag_module *cmod; + unsigned long mod_id; + struct codetag *ct; +}; + +#define CODE_TAG_INIT { \ + .modname = KBUILD_MODNAME, \ + .function = __func__, \ + .filename = __FILE__, \ + .lineno = __LINE__, \ + .flags = 0, \ +} + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); +struct codetag *codetag_next_ct(struct codetag_iterator *iter); + +void codetag_to_text(struct seq_buf *out, struct codetag *ct); + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc); + +#endif /* _LINUX_CODETAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index fa307f93fa2e..2acbef24e93e 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -962,6 +962,10 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". +config CODE_TAGGING + bool + select KALLSYMS + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 740109b6e2c8..b50212b5b999 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -233,6 +233,7 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ of-reconfig-notifier-error-inject.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +obj-$(CONFIG_CODE_TAGGING) += codetag.o lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/codetag.c b/lib/codetag.c new file mode 100644 index 000000000000..7708f8388e55 --- /dev/null +++ b/lib/codetag.c @@ -0,0 +1,199 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include + +struct codetag_type { + struct list_head link; + unsigned int count; + struct idr mod_idr; + struct rw_semaphore mod_lock; /* protects mod_idr */ + struct codetag_type_desc desc; +}; + +static DEFINE_MUTEX(codetag_lock); +static LIST_HEAD(codetag_types); + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock) +{ + if (lock) + down_read(&cttype->mod_lock); + else + up_read(&cttype->mod_lock); +} + +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) +{ + struct codetag_iterator iter = { + .cttype = cttype, + .cmod = NULL, + .mod_id = 0, + .ct = NULL, + }; + + return iter; +} + +static inline struct codetag *get_first_module_ct(struct codetag_module *cmod) +{ + return cmod->range.start < cmod->range.stop ? cmod->range.start : NULL; +} + +static inline +struct codetag *get_next_module_ct(struct codetag_iterator *iter) +{ + struct codetag *res = (struct codetag *) + ((char *)iter->ct + iter->cttype->desc.tag_size); + + return res < iter->cmod->range.stop ? res : NULL; +} + +struct codetag *codetag_next_ct(struct codetag_iterator *iter) +{ + struct codetag_type *cttype = iter->cttype; + struct codetag_module *cmod; + struct codetag *ct; + + lockdep_assert_held(&cttype->mod_lock); + + if (unlikely(idr_is_empty(&cttype->mod_idr))) + return NULL; + + ct = NULL; + while (true) { + cmod = idr_find(&cttype->mod_idr, iter->mod_id); + + /* If module was removed move to the next one */ + if (!cmod) + cmod = idr_get_next_ul(&cttype->mod_idr, + &iter->mod_id); + + /* Exit if no more modules */ + if (!cmod) + break; + + if (cmod != iter->cmod) { + iter->cmod = cmod; + ct = get_first_module_ct(cmod); + } else + ct = get_next_module_ct(iter); + + if (ct) + break; + + iter->mod_id++; + } + + iter->ct = ct; + return ct; +} + +void codetag_to_text(struct seq_buf *out, struct codetag *ct) +{ + seq_buf_printf(out, "%s:%u module:%s func:%s", + ct->filename, ct->lineno, + ct->modname, ct->function); +} + +static inline size_t range_size(const struct codetag_type *cttype, + const struct codetag_range *range) +{ + return ((char *)range->stop - (char *)range->start) / + cttype->desc.tag_size; +} + +static void *get_symbol(struct module *mod, const char *prefix, const char *name) +{ + char buf[64]; + int res; + + res = snprintf(buf, sizeof(buf), "%s%s", prefix, name); + if (WARN_ON(res < 1 || res > sizeof(buf))) + return NULL; + + return mod ? + (void *)find_kallsyms_symbol_value(mod, buf) : + (void *)kallsyms_lookup_name(buf); +} + +static struct codetag_range get_section_range(struct module *mod, + const char *section) +{ + return (struct codetag_range) { + get_symbol(mod, "__start_", section), + get_symbol(mod, "__stop_", section), + }; +} + +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) +{ + struct codetag_range range; + struct codetag_module *cmod; + int err; + + range = get_section_range(mod, cttype->desc.section); + if (!range.start || !range.stop) { + pr_warn("Failed to load code tags of type %s from the module %s\n", + cttype->desc.section, + mod ? mod->name : "(built-in)"); + return -EINVAL; + } + + /* Ignore empty ranges */ + if (range.start == range.stop) + return 0; + + BUG_ON(range.start > range.stop); + + cmod = kmalloc(sizeof(*cmod), GFP_KERNEL); + if (unlikely(!cmod)) + return -ENOMEM; + + cmod->mod = mod; + cmod->range = range; + + down_write(&cttype->mod_lock); + err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); + if (err >= 0) + cttype->count += range_size(cttype, &range); + up_write(&cttype->mod_lock); + + if (err < 0) { + kfree(cmod); + return err; + } + + return 0; +} + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc) +{ + struct codetag_type *cttype; + int err; + + BUG_ON(desc->tag_size <= 0); + + cttype = kzalloc(sizeof(*cttype), GFP_KERNEL); + if (unlikely(!cttype)) + return ERR_PTR(-ENOMEM); + + cttype->desc = *desc; + idr_init(&cttype->mod_idr); + init_rwsem(&cttype->mod_lock); + + err = codetag_module_init(cttype, NULL); + if (unlikely(err)) { + kfree(cttype); + return ERR_PTR(err); + } + + mutex_lock(&codetag_lock); + list_add_tail(&cttype->link, &codetag_types); + mutex_unlock(&codetag_lock); + + return cttype; +} From patchwork Tue Oct 24 13:46:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434582 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27926C00A8F for ; Tue, 24 Oct 2023 13:47:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD0416B0270; Tue, 24 Oct 2023 09:47:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A597B6B0271; Tue, 24 Oct 2023 09:47:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8858B6B0272; Tue, 24 Oct 2023 09:47:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6E26D6B0270 for ; Tue, 24 Oct 2023 09:47:13 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 426FB40470 for ; Tue, 24 Oct 2023 13:47:13 +0000 (UTC) X-FDA: 81380481546.06.F6BCB73 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 74FF418002A for ; Tue, 24 Oct 2023 13:47:11 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dSdNxrMi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 33so3ZQYKCIMz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=33so3ZQYKCIMz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155231; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YavwLOEBRwC8H4iLOoVDT3e7zkyr1je9EHR8XAi9ck4=; b=j8FZd6zcMtIHFeUAQCOesmS3dmjo4IE9k482BJUYbLSfwlExT5B2uiJLzq41gW4LbtDmqU SCCBXsCaFAqXyCumt2Q4xg9Lp1espL60i2C+hLh62tjQ002uHmOZ7l4k6U3Ti70u7q0vur OtaP0uyln5pLht+cqomo9m8ZWtltUh8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dSdNxrMi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 33so3ZQYKCIMz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=33so3ZQYKCIMz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155231; a=rsa-sha256; cv=none; b=qHfrqtSyzG8/ro1e9haLcwVFSK+9lwaWLFgGAp/Vq5KXAg8j6y2lE0xqPrrYegWr6z0Mod 5PkfhO+9tQ51/4oURKyvWIXwu7RL3DsouyS19P02mDJ274mFUXPk1y8git8sbCX+jQSmhy 5MpBQHBwkRmBJB9t5oXhTmrNLGEnHPk= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5a8ead739c3so59306447b3.3 for ; Tue, 24 Oct 2023 06:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155230; x=1698760030; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YavwLOEBRwC8H4iLOoVDT3e7zkyr1je9EHR8XAi9ck4=; b=dSdNxrMi161j1FoX2yATUVM3N2zyxX7H3fnehTPQOxnwT2t12fhxmbOqm2aeuznDs6 PI80MXSaj7YtiN2//FiLfQGnSxbYjYEwUS5FX1jGzkn32vwBF2F7KKa1tyYab6cl0VDA V4/Niiwgp+Lt89hw/DaFEwYzlbkovjuMGGoJQPnqPCbaU8OAADVrcnZD5kyHOQ4jZxOQ 7G9R7F3bvXgKlIXecEepWsSZT9g+CVnD5VP4GACQn1Guv3/GXSLRomEcyB1qfsI158/J fn58DloXVqBFymvJ5Efpn3pBmgZWzh2ICHNkHphcNdGhcimI1Db5EzvJof2E/DuSFPzC bcHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155230; x=1698760030; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YavwLOEBRwC8H4iLOoVDT3e7zkyr1je9EHR8XAi9ck4=; b=VLdNhlHugksQiHCcBX9c9rUFS07f0tHpF7d43ucpvjv472T+gtpT/2hIEssh/Zfm+V YA67DJF7QwRXXsbX8I46lyPYnlFZkTryURYrlDzFYZo6fs1cc7glPk+Cof3QDzWITZ1z LwAkfiJt/OAzk3ud1k/7Ld0wiHjgO/ZF2k9SE8CyJAdq1QkJBV5tfcaKZY4fjSvAe1NR FgCpzV8jDJ4MFlo/z4D9Rr47q4DECYQMy4BsUdxqBJ7orPGk0/FbILf7IOGO0/9wGVKl bEXLNtPej9c1NmZ/4cvDUnXhc/mQZNTbfbYJQB9jrOjgnWkhYOcNb6JmoX5omIkUQmqb uCZg== X-Gm-Message-State: AOJu0YwUD0JbUWHhcA3uLaV3nxriCaugLBqWuOl92nRrWl7C/g0a8Hvu ZlqIpv4jMgmXBp2pJXc/6fwO0aFk+b0= X-Google-Smtp-Source: AGHT+IG23pgEuZi2XWkcWZyR41lqZU1TGfs7mnZJ7YZozt0qnBcTuPc8JBuvXuLPX/Ngmgob/V7gfXg7y38= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a81:4e0e:0:b0:5a2:3de0:24a9 with SMTP id c14-20020a814e0e000000b005a23de024a9mr271520ywb.1.1698155230510; Tue, 24 Oct 2023 06:47:10 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:10 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-14-surenb@google.com> Subject: [PATCH v2 13/39] lib: code tagging module support From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 74FF418002A X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: dyh99wqsthespdsqg7fccuckqxixf78o X-HE-Tag: 1698155231-295385 X-HE-Meta: U2FsdGVkX1/owtOy2xxjAf8xG/XSMpvb+8wJmC54xTTpd2DJ2rAJ5t1C0f50OI/Loi7rSfJNpOgAopYq7LxZKLDCV2/f+jcTNZOy47WC2lE6U+NYuIouJveGiIuPtCIrDWeVuSTjK1S6eIQV257vF1LL8j+BXdMG2oC/2nbaoT4jrmmxpSZdCkkPYaAHbRy/0BLjOnLIzerCEzI+5U7zH3uUpRdIPgaZiMQEA8K+TEvQib9flDSDk+2MmmfYdXF/+jM6iTxzVuDcXOZI/wHtwXgLtOQJXe2SGISKfLBEUgcXnFbaoImJ918SWVJS+K+RA8IjFaseLH0nNxm/lk6KegpjVTX7xzeJ6it2kjzdS3s8DqK1k2qkUzjjZOIhjkJXp6arf2UjWzSA7urrLV+7Eiw9UkJhDpyzVQlishymkN/ysptBHgweprgUB2PcFjhWbeT3ar12nevaNaafwjyjf6PDapf5K52EFndMTLbghwQIiC+T+azHhB5Qd/OKL8mD2zbCCEsS2uIBKEZx9WDkDYFQTvRZ7DGg8qI3jcx+VgWJQIXYgElQVnEbVAv1EdCIH4c1pL3raTbxk+SlkOP1CYy+Cb8TZqJ7TfKSX9xDzAAvazZp6h7UgnJ/UhJ/DV/4BamGG57G5Fdl3xl5Kzitwqi7uxUWDXE+IaaSsg/1QHZtP6FBMJWFjZ/zeGfgxNbkUsF+Owt+8qufSNPIRi40zFKXKqGeJa4uh7k7dO9o1PC63qb2M6eIpMbi9Jgy3GXGB/xcgiHvL+kOL2bt8XB+fK1zO3mL1Iq8pPItw+Ve104yDfpiCcDFiRnJnlyP1na0MTsrr2Nc2WbpLP3LYJW3hMg8llJT9w1V/fhI71tsaYVnmKoUJwnOvIVlgwQLsKMNrdxw2AXtbxKe9Yx3WNwk80PqTIZUftfrL7Iw1yG9J/jyobCpiB9DKgKLyxpFK58zSkEMNxyBgyFtp6eqPDp KJh3CmNO wUj0Gl+9yRu4AIFZSoZl5UsCspmEPvB8595df/tMUYzVt/VRtmksGtSzjIM72bPcOLS64XZuElZjIrlTJCbWqcdUjMlYBpwOWRmT5NJkq6Suf1Qq9EicXth2///Xd4QiqWDv6ckzI65FV5ijfbk7sJQIL3cbISJB/bUgr5hTRQL5/ppMLB/DAfs3K51a176j7XItyTfbFbMsrsDFPoOPucbXj3IL6CCImJM3qm3soJgPpNbNfq2mYaXrB8d2awyk5tb18dgCVf8tQRYRck0O0O61smJXdeaKTNd6WK7wkcG67APZlu3aauJpGd98ngDZJ8eUdKIx51JIeoHjTikZLYyfa3lf9FHfzlnTafjltRbO6FhbfB8v8miqRB2neX5jglh9xjzxlYtHu+te4MChm0+ifuB+zUmeFxuBu4eLdMhlzJF3AOPgX27qEv/n464/H+8ZAx7ZYD63epStwvAVqhhwhJQyodgLsNLOjoFFbvbuautEpbXXCP02gOsjvYxFA+2LQIie3EszcAVZBSmJTYSigyXt1ZgXMfXi5qU+BoQkvKFLN2oKChvYtTw4siuZ4WGilp1AT3RXkBwkStjr5g6AVS2VOtGktUdN2o9BHhb24KOD5lhqBS7PAJlbSVYMVTNW0ubdMOErzFJ2wr9ogk42BJijrPa2tlXKX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add support for code tagging from dynamically loaded modules. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/codetag.h | 12 +++++++++ kernel/module/main.c | 4 +++ lib/codetag.c | 58 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 2 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index a9d7adecc2a5..386733e89b31 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -42,6 +42,10 @@ struct codetag_module { struct codetag_type_desc { const char *section; size_t tag_size; + void (*module_load)(struct codetag_type *cttype, + struct codetag_module *cmod); + void (*module_unload)(struct codetag_type *cttype, + struct codetag_module *cmod); }; struct codetag_iterator { @@ -68,4 +72,12 @@ void codetag_to_text(struct seq_buf *out, struct codetag *ct); struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); +#ifdef CONFIG_CODE_TAGGING +void codetag_load_module(struct module *mod); +void codetag_unload_module(struct module *mod); +#else +static inline void codetag_load_module(struct module *mod) {} +static inline void codetag_unload_module(struct module *mod) {} +#endif + #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index 98fedfdb8db5..c0d3f562c7ab 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1242,6 +1243,7 @@ static void free_module(struct module *mod) { trace_module_free(mod); + codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -2975,6 +2977,8 @@ static int load_module(struct load_info *info, const char __user *uargs, /* Get rid of temporary copy. */ free_copy(info, flags); + codetag_load_module(mod); + /* Done! */ trace_module_load(mod); diff --git a/lib/codetag.c b/lib/codetag.c index 7708f8388e55..4ea57fb37346 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -108,15 +108,20 @@ static inline size_t range_size(const struct codetag_type *cttype, static void *get_symbol(struct module *mod, const char *prefix, const char *name) { char buf[64]; + void *ret; int res; res = snprintf(buf, sizeof(buf), "%s%s", prefix, name); if (WARN_ON(res < 1 || res > sizeof(buf))) return NULL; - return mod ? + preempt_disable(); + ret = mod ? (void *)find_kallsyms_symbol_value(mod, buf) : (void *)kallsyms_lookup_name(buf); + preempt_enable(); + + return ret; } static struct codetag_range get_section_range(struct module *mod, @@ -157,8 +162,11 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); - if (err >= 0) + if (err >= 0) { cttype->count += range_size(cttype, &range); + if (cttype->desc.module_load) + cttype->desc.module_load(cttype, cmod); + } up_write(&cttype->mod_lock); if (err < 0) { @@ -197,3 +205,49 @@ codetag_register_type(const struct codetag_type_desc *desc) return cttype; } + +void codetag_load_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) + codetag_module_init(cttype, mod); + mutex_unlock(&codetag_lock); +} + +void codetag_unload_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + struct codetag_module *found = NULL; + struct codetag_module *cmod; + unsigned long mod_id, tmp; + + down_write(&cttype->mod_lock); + idr_for_each_entry_ul(&cttype->mod_idr, cmod, tmp, mod_id) { + if (cmod->mod && cmod->mod == mod) { + found = cmod; + break; + } + } + if (found) { + if (cttype->desc.module_unload) + cttype->desc.module_unload(cttype, cmod); + + cttype->count -= range_size(cttype, &cmod->range); + idr_remove(&cttype->mod_idr, mod_id); + kfree(cmod); + } + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} From patchwork Tue Oct 24 13:46:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34B2DC00A8F for ; Tue, 24 Oct 2023 13:47:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9FA06B0272; Tue, 24 Oct 2023 09:47:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E03A96B0273; Tue, 24 Oct 2023 09:47:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C06FF6B0274; Tue, 24 Oct 2023 09:47:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A7DBF6B0272 for ; Tue, 24 Oct 2023 09:47:15 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7B2F0C03C6 for ; Tue, 24 Oct 2023 13:47:15 +0000 (UTC) X-FDA: 81380481630.14.7680D19 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf02.hostedemail.com (Postfix) with ESMTP id AB81380018 for ; Tue, 24 Oct 2023 13:47:13 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Y52j4L1O; spf=pass (imf02.hostedemail.com: domain of 34Mo3ZQYKCIU130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34Mo3ZQYKCIU130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155233; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+qMxws3YMU5LIbVoF9EJC7SgnlaxtiNdFL01V96iMG0=; b=g0OU9aHq1bf3EFcVN3N2CBf4V7Ncsfr27RndxGYV4Qx5lyK6dLZjcfy1SdTitwmiRQHn94 sp9M5UKTMOjsv1/oUUKx2kMwS7ElS/vy/Pg2X1IiN1U13qEYZZlJ1t2gVshwWI6VFze7Wj nsGYZXDc7h+5ETzh5sj0lmktmgw4Pok= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155233; a=rsa-sha256; cv=none; b=UG2+qz/vuHY23lH3+Z4+bgVhD6ONK8W92gbGvLzGiZXP8aiPluQg4PRg4cm4Uz7c8vvcgE 0kJTpXbDjyrt7DysvYqrTOhnyEUjf9CK44uyEA9n8GN4gvZbLpSTrQiimL6bdJD0Pzx8/y IOHAZfBsoeIW7eoXl71O/X6WXE+fNVY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Y52j4L1O; spf=pass (imf02.hostedemail.com: domain of 34Mo3ZQYKCIU130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34Mo3ZQYKCIU130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9c4ae201e0so4420305276.1 for ; Tue, 24 Oct 2023 06:47:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155233; x=1698760033; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+qMxws3YMU5LIbVoF9EJC7SgnlaxtiNdFL01V96iMG0=; b=Y52j4L1OvSijsyPSgNbX2CjqrI4gElom2n8coYv1DycrukENpDw8dlceVfr7pCo+jf +FEWTwd4UP0OzZzb9ocKxD4QBsi13qZ2/+wGkB/bUJU9CD3RfI/wX7C3hy+DZjtERQEC B7ggMOgreL/jKWu9T6CD8BElH5tJsCV4A6bMaTGyGp/ia9PsTnVKGwNKIQtMVIUC1TeF EOJE5vk14AKndnCiw2Rlzj4xe7vw5J62sB4Y6cCM83FW7w766j4BKTFmvBxgY0SNAXNf i6xjD+dfE73GV0a4hCC2N+FtlR4H8Rl5fHT5vGtb+3QGFXlw/PhxfsCIjHe76kv2JPmP CdLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155233; x=1698760033; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+qMxws3YMU5LIbVoF9EJC7SgnlaxtiNdFL01V96iMG0=; b=dDNSqW6XEoy7SXyxzHVvjcdDm2S3ikMEYWWQY97P4W7raytVgGKamwYWGE64uQEvbP LqZGZ0nsPTC8cX3+FlWIqoE888c03UfdZWv/szM33ivsDMNnz6dTLEkKFwps21RzgSPi 9+Sol/Ikfp24gCVd7xEhtIoKFb0y6DLQKe9aIUYL1AcKM3Pw9xp3VYDaJnoJ8HSKcjhs vjq+p2b3miDF+h8oxKp9D8X/xwOszi3Iniq7Leo2MXHuuODFSkg1GtbzkS8XCAPOiMS1 x9Zj9ud045j6w9asoufMumxiBRWE1bO5/3H0di/K9AOtINjx0Mes/ZkU/3bAkggsc4JT hQmw== X-Gm-Message-State: AOJu0YyeMcItk/3sithcfqI36DLDPaOZUCTPF98Fw+iGpIbg8EvxFxWZ hCQdeYMvZVNRn6rN6kjMo7NPp32kwYk= X-Google-Smtp-Source: AGHT+IH+f+GT4bbDivhKX/Rj+lmLO8Xm8nWs8jzCh/R0o0BCAeKtB84tFP9Lg2d1/1Vq2LZxNMEv6GNyFyw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:b28d:0:b0:d9b:59c3:6eef with SMTP id k13-20020a25b28d000000b00d9b59c36eefmr419344ybj.0.1698155232720; Tue, 24 Oct 2023 06:47:12 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:11 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-15-surenb@google.com> Subject: [PATCH v2 14/39] lib: prevent module unloading if memory is not freed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: AB81380018 X-Rspam-User: X-Stat-Signature: smhne5ed8awgpj3hkwbrceuppqmdaxj9 X-Rspamd-Server: rspam03 X-HE-Tag: 1698155233-188611 X-HE-Meta: U2FsdGVkX181g7df3vtuCT/DgoNyddSS3LR0CG4WY5ch6iKVJkX9AtGEY7ubqyqn1bhMZaWKl3wvXVRkxgorjRI1otaVUX4ihj41br9pXUfx481Gkq7s0PtL8vgrl6Y6RMq4sDFacHLRsnaMjD7C3eprqCk6H8mhjZc5hgWslYgvlwRSvNWxVsa0q6pIQuulH6GlcXk0NUtKSLK/J5kkmnrcekC7KzrSEceCOxOd14Z8s3/yRXPmfCDxHb8GHKs9PY3LcefLY00sLlOcduj4tlh7sOgw9vOgzQH4e0x69PRAaXYXRuSVF0bs8BUaAFxDA1iSZ0ByLWXZOJN8oPZf6KX5OrYgRSyj0y+LiAz2vVmYD54NoXBIMqkHgch5Ih2POAUUPO4MBwt82v8k5+QoyQ/ban0O9TAksGtW2SsVRHKWEGSQlKAqC4KqXEDlEUbBagUcgN1wTtBZAuhnRodIFLOOkolPjzgg9CTEoxw0ZRRb2bx2a8oZq+AxBjwUi2e1Lw5mwD5lLw8pcYqRYwd+LoSFRaQCKAa9IIIuQyuZq1cG+ErXqYHTZWZecW5KbsmSbhhsrrroQty7sskwtB5NzpVfSpt+bwQ4aR3UgTEnxApZrIiDDTLyKXboeYdkhJhO9aq1BlzilnSVXjG8kyBE2vKjELSQNDc4CorZhv5TE2PQ7yIDPpt/vze+JQuXsfOJt7VZ2S1mszcsM56cfsT0ETOLfpxtSMm9DBa/nXGMbyjcQIq18Yxy4LxAIhoxFvNUcqGUbC1BwCm1BhKc7myv/7uYMH6mPcYjV+8J71uhAZKPuZIyMwaCieKHa4p5jLY6Letpte8dqUhtC1gtP+tE3tFuT6u+8kaCJxXeGL2BN08f+c6U/Mp3kaHboFq50sljODctZ1FxnE7DdbUcGFek0hluKRx506PxVvD523D2Q3Q0tjlEZHx1sz/8VLDLz7QqLVq+Ys1o9lgy1/HnBNC Sd+H2y1D vP6qqRO17p3JqI6r7zSVlEqAUFliF+73wP+ukD1JkjT0d5YmD5OqhMEn7Ft1eAzO4DyPpqSFCY6MXvVRNcCf6U6ldOQpyZVtilfJrtGHsvlDdFVo1KeXraaN0UaejrxLLCkLCsZcfxDIF/WGPYyXHJ7TANsmVDf38vNutgwQZ7082LvdYS5c59Bspnb3B8Qel0VHrXdFXHMKKsZnITAjbyO6Jt3upAOPw9ioDBQB0dnbXOW7h63LcgKkYdPgIcVrTqC8ZZyOnHob8mL7z7gLhQKB5HlECJLjo1Mrs/EzNmiHJ0LM3eZfMJggfIvbKa2dZFvN5Afm/SP3magbAEMClmCdItpFwlGRAw6VwvBVoYTlUtrucetLRgto9sFPwZwbF5jteQKFAlEBjIPDK+l1eKnn2fqHPmBleWlNkjmEyO3hNFiUjkPBhp1/D26CCEtIzAx7HUW90e30+2zraH3kBZeDlGe8rZd+8c0SBeV1u1y6odF3GlEgowQDfNhh80HIspK3HOoAFsAkJeYSdvfhRmPaguuwEZ+yhT31gybhbgSbLJH8z/rLswJ/UzaUzj75s/JdJ6zPt5jer13DdLVqj1cNFBeQLwkobulB6ll9QdZ+ft+KCqA2C8xRYFYGCjxKomBIcc+BdOEQiisvmHhI20+EzzGKLtHJv6Zgx7s0/Z0J7tdM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF. Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 6 +++--- kernel/module/main.c | 23 +++++++++++++++-------- lib/codetag.c | 11 ++++++++--- 3 files changed, 26 insertions(+), 14 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index 386733e89b31..d98e4c8e86f0 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -44,7 +44,7 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - void (*module_unload)(struct codetag_type *cttype, + bool (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); }; @@ -74,10 +74,10 @@ codetag_register_type(const struct codetag_type_desc *desc); #ifdef CONFIG_CODE_TAGGING void codetag_load_module(struct module *mod); -void codetag_unload_module(struct module *mod); +bool codetag_unload_module(struct module *mod); #else static inline void codetag_load_module(struct module *mod) {} -static inline void codetag_unload_module(struct module *mod) {} +static inline bool codetag_unload_module(struct module *mod) { return true; } #endif #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index c0d3f562c7ab..079f40792ce8 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1211,15 +1211,19 @@ static void *module_memory_alloc(unsigned int size, enum mod_mem_type type) return module_alloc(size); } -static void module_memory_free(void *ptr, enum mod_mem_type type) +static void module_memory_free(void *ptr, enum mod_mem_type type, + bool unload_codetags) { + if (!unload_codetags && mod_mem_type_is_core_data(type)) + return; + if (mod_mem_use_vmalloc(type)) vfree(ptr); else module_memfree(ptr); } -static void free_mod_mem(struct module *mod) +static void free_mod_mem(struct module *mod, bool unload_codetags) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1230,20 +1234,23 @@ static void free_mod_mem(struct module *mod) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod_mem->base, type); + module_memory_free(mod_mem->base, type, + unload_codetags); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA); + module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA, unload_codetags); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { + bool unload_codetags; + trace_module_free(mod); - codetag_unload_module(mod); + unload_codetags = codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -1285,7 +1292,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod); + free_mod_mem(mod, unload_codetags); } void *__symbol_get(const char *symbol) @@ -2295,7 +2302,7 @@ static int move_module(struct module *mod, struct load_info *info) return 0; out_enomem: for (t--; t >= 0; t--) - module_memory_free(mod->mem[t].base, t); + module_memory_free(mod->mem[t].base, t, true); return ret; } @@ -2425,7 +2432,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod); + free_mod_mem(mod, true); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/codetag.c b/lib/codetag.c index 4ea57fb37346..0ad4ea66c769 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -5,6 +5,7 @@ #include #include #include +#include struct codetag_type { struct list_head link; @@ -219,12 +220,13 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -void codetag_unload_module(struct module *mod) +bool codetag_unload_module(struct module *mod) { struct codetag_type *cttype; + bool unload_ok = true; if (!mod) - return; + return true; mutex_lock(&codetag_lock); list_for_each_entry(cttype, &codetag_types, link) { @@ -241,7 +243,8 @@ void codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - cttype->desc.module_unload(cttype, cmod); + if (!cttype->desc.module_unload(cttype, cmod)) + unload_ok = false; cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); @@ -250,4 +253,6 @@ void codetag_unload_module(struct module *mod) up_write(&cttype->mod_lock); } mutex_unlock(&codetag_lock); + + return unload_ok; } From patchwork Tue Oct 24 13:46:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76B0DC00A8F for ; Tue, 24 Oct 2023 13:47:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F1246B0274; Tue, 24 Oct 2023 09:47:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 37E006B0275; Tue, 24 Oct 2023 09:47:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10BFC6B0276; Tue, 24 Oct 2023 09:47:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E90F46B0274 for ; Tue, 24 Oct 2023 09:47:17 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BC96B1CB793 for ; Tue, 24 Oct 2023 13:47:17 +0000 (UTC) X-FDA: 81380481714.24.568805B Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf06.hostedemail.com (Postfix) with ESMTP id D92A218000A for ; Tue, 24 Oct 2023 13:47:15 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YOF2PYIh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 34so3ZQYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34so3ZQYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155235; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K+jSKdm0qvzH//duzEhIZcT0jtfDLsmvFXjZdfhiaL8=; b=D2u7KyFUqdpGmBc8DzL7vNKPmGYNjDYGlVNaX/UMTSpygkphKbekda7vXsa5Bg0Mr50tY3 c2CVo3o539bA5gv97tRy2XT1JAelpiqAR7R/yNwp/9xxjjXZ5c65bqsPm7jKsqblltsF1S IRW8bRW7StKftsp0lKqQD3MY4/++wnM= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YOF2PYIh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 34so3ZQYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34so3ZQYKCIc352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155235; a=rsa-sha256; cv=none; b=Zb8yRYy6CRHfP6QZLoj2CiL7NRwTYHkg5uxKY4SxJaY6qAOn8rbzzYuw21kQkcXos58Lwe MZEdBwe8L4h07ilshxcS32sR+S3hswyU+2MP2kXAtj0zxdNQbvYGDkAGAerdt6Kw+gfSbD NPMpKdmXap3Nqs2QPkn3qHBXwRAORzc= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9caf486775so5283803276.2 for ; Tue, 24 Oct 2023 06:47:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155235; x=1698760035; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=K+jSKdm0qvzH//duzEhIZcT0jtfDLsmvFXjZdfhiaL8=; b=YOF2PYIhKCZesU835TZhoSdQQ4xEkyJwpmGSfuPE1imNh6B5b1RNJociYn/BPJ0rIu lwupRXeGbUwg24a6cUYlOpueOOQ96z+5SRtNGht+T+IGTxsVuH0jAsukL6JvjMpujLQU rwHl5b8WAvpLErT+DPzoegZ4mQNsBqbbb/n+fZgVt1xh5coI+lMSqyM+wXGLAPUeqG1x gSOl8/jr7eUSv868fTzavbELQ2M3v6a1vIzXT8IL1JhDM0fs/R8L0Kyyh/7gZckfnIh/ u46ItYnazpPZEiIeMit05nkh3GsW8gqCqhjHvx1wyY7mzu3ML9edvs72aCC6Tvl3xHC9 7r4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155235; x=1698760035; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=K+jSKdm0qvzH//duzEhIZcT0jtfDLsmvFXjZdfhiaL8=; b=IWQ/eLsUi6iK6NVR2gZUvZIVXe2z4PJUtg8xsGNJdje791kvK7TV07gqqhcV7aSWlS KHgeK7S1ISGMjDnXotTt5lPgVQ/vVpwSAN06YiJvsbCeGxYxCwmMInDguJGp1aKLjjSM DMAILBN5oQofZlzQv49FeBRqLDwoHj1JIyaa1/w0v4pz2aL+8SZcqO+BNDhd4A1ZRhdP 0y1zz8mMlq3iBxW73C0QMGDbXPlyq5Tf++uffFWK6qRhE9QpoYM3pNYMywCu5w+irfcp vDT9JU5LS1mUhvtZm0HG7HqKqwn/XzMUrqjs8M5s2UxD76UaYxLmbxWnJUpO214NZWaB QoHA== X-Gm-Message-State: AOJu0Yw+lvAqLb0hqGefYfhUz1rirM9NsObsJKANrMt+EkNXUVBWHxeo U4gOM/SRnVDOVymFXg2nPDn+kqaB3go= X-Google-Smtp-Source: AGHT+IEIa4lubk7zVV+55Kssyl5IiCjPBPehEtm0DwQdG9mxypz8Eg4MHfMNeuEtWe6Z19f/ZGvIAMikRMg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a05:6902:1083:b0:d9a:cc6b:cf1a with SMTP id v3-20020a056902108300b00d9acc6bcf1amr317480ybu.8.1698155234917; Tue, 24 Oct 2023 06:47:14 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:12 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-16-surenb@google.com> Subject: [PATCH v2 15/39] lib: add allocation tagging support for memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D92A218000A X-Stat-Signature: aiqbk7bmjz5bg8ai855a5a3o7pggt91i X-HE-Tag: 1698155235-803561 X-HE-Meta: U2FsdGVkX1/fkVORdJmGvIJXjq2i2CT8jOS6jKJMZgdFSgmcFhO27ovFQ3X86xHp28Jhici/pyT2l+S5aVYvg79Fh7JIpLxv4UK59J4EGHe3zFOEMsoUDDy8n8lrauGSH3/iP0V10pkSYPqxVZD5SXiw/3rQUDwGX94ITMZDI9vCvIhp2MKOnIsXkLLmZnvzYmAcPnZGOuiwcDxUcMePajGyn+XuIoOEvev8gwu/nL+MNwkBue5qo1NFFee71XQjMI8RTMVT9X4IK9I0QZfk+kqo5O2I+EpRYlCC2Xj7ect2U64d+BWPTp9QovT5ke2hDah/T1HIsqflQDG49u2NLRPcXgp+MQRDeR6Jg6ooIQe01IB10d9x2KbZCos/QHyqdmDTb1DyAtRudR1lHQ/8hT/+eHBYz4EyxXCTdpAG+VK5QJX6rZPtlyyHO1jLxkpnUtC1zkH9kOnws53FKZzXaIHou46xsPnrzwwNiua4rAtEZqe3WwhoHBO4/Z+9nFR3p+0cpoT823nFtHkE7v5JLpT4nhmvU9o0KBMrQwLEf5yXX5Gvv+9Bef4QXbN8gPC9aGR7f9bgu+9Enu5jIbCJKnSUzgX0k/lLvvTXSWnTr5W3F5wTp+/lBQlEfWQj86xp73T+31UsA+bKnuE/yDCTVW9/qtqMf4AYlvjh7m5I8Jp0/QESIzEo/RaXGK9Dx3/Nv/UZI7jgcC3Tys+UcX/enxo2Npf6xmdL1Q2obCgM2aDtmFgL+VpaJzwmu8ed9ERWbDRgzOkaZz0Hj8f+DCf7v0RJUvPnO2LkIxGkrgm2JdtyQvvHqAp1ri3sSh38sIt6gayRqXeJXDkkLJKceqEiwkCsmeeBPmuRcnGpnwqis71kcB58tmTlDgk56AuD1Ur/v1nMLROYRM7Og/5VV0sGmvyPed2nTksO/J+EeNG8mmQtotSypeFEy/ygEgH8gbEmfMbVpnea2pieY4oh3El mI90B5f4 XW14a06o+Zigbv0yZNpCTdZ4S1gj1eOfDmYH87Jnrn/N6UY4lKfe6tVcd3lgY85k48o0XbhDp4fQbudF5x6ccUZir3t+cfV597RbxMIrM6jR39iEvJp54pJ5547BEK7B9PA78yTuPu/g8iyaFPDOlGr9QsfUGdn7J4bpMTpYKVs8O6JLDwdgVTvlEbIoTjCSPIKLTySfgfb8omi4n75dpF/vguoIciwM/oUDh8WqneqayhXOQusBmqQUf5n7uo9FQeGZvfKTfY1k5N3cjbve/j1ybK+aJsmtrYCDreqZKLuFEEvNtB1y77lFov8o8LIlzbVtGDGcyZPPHADHgdaSI1s2jUw1IJRNwSezwItZjDIEGCGpbFe1LcAfxTbmQ7y8jRkmxuDRCCZVPZw/HpMh+G0LpLMplrW4Tw06zzIPch44TOvBk+fTbr/PnkaodqKtWGfePRqG5lbheKsf2b/SXOITTMS3GswsK7Jz7PXfhsIBCTJJWkhVy+Y6kaylDZFQEDGAPH40mqTNyAKrgSz4hoccoH8v6azbLnlDGQs8OMbv6QnVHzP6LRkWT/vmdv6VRO1NhDsoH82I+3Dj4zo7uXzyG+RWDMs9gbPaYUegQ8FcRCNGV604FVhvM02otMYEQNoKgqLGJ/nhKt+Oja5YNeZKLVkRm2UAqFlGLj4iVbRELfS8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce CONFIG_MEM_ALLOC_PROFILING which provides definitions to easily instrument memory allocators. It registers an "alloc_tags" codetag type with /proc/allocinfo interface to output allocation tag information when the feature is enabled. CONFIG_MEM_ALLOC_PROFILING_DEBUG is provided for debugging the memory allocation profiling instrumentation. Memory allocation profiling can be enabled or disabled at runtime using /proc/sys/vm/mem_profiling sysctl when CONFIG_MEM_ALLOC_PROFILING_DEBUG=n. CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT enables memory allocation profiling by default. Co-developed-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Signed-off-by: Kent Overstreet --- Documentation/admin-guide/sysctl/vm.rst | 16 +++ Documentation/filesystems/proc.rst | 28 +++++ include/asm-generic/codetag.lds.h | 14 +++ include/asm-generic/vmlinux.lds.h | 3 + include/linux/alloc_tag.h | 133 ++++++++++++++++++++ include/linux/sched.h | 24 ++++ lib/Kconfig.debug | 25 ++++ lib/Makefile | 2 + lib/alloc_tag.c | 158 ++++++++++++++++++++++++ scripts/module.lds.S | 7 ++ 10 files changed, 410 insertions(+) create mode 100644 include/asm-generic/codetag.lds.h create mode 100644 include/linux/alloc_tag.h create mode 100644 lib/alloc_tag.c diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 45ba1f4dc004..0a012ac13a38 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -43,6 +43,7 @@ Currently, these files are in /proc/sys/vm: - legacy_va_layout - lowmem_reserve_ratio - max_map_count +- mem_profiling (only if CONFIG_MEM_ALLOC_PROFILING=y) - memory_failure_early_kill - memory_failure_recovery - min_free_kbytes @@ -425,6 +426,21 @@ e.g., up to one or two maps per allocation. The default value is 65530. +mem_profiling +============== + +Enable memory profiling (when CONFIG_MEM_ALLOC_PROFILING=y) + +1: Enable memory profiling. + +0: Disabld memory profiling. + +Enabling memory profiling introduces a small performance overhead for all +memory allocations. + +The default value depends on CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. + + memory_failure_early_kill: ========================== diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 2b59cff8be17..41a7e6d95fe8 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -688,6 +688,7 @@ files are there, and which are missing. ============ =============================================================== File Content ============ =============================================================== + allocinfo Memory allocations profiling information apm Advanced power management info buddyinfo Kernel memory allocator information (see text) (2.5) bus Directory containing bus specific information @@ -947,6 +948,33 @@ also be allocatable although a lot of filesystem metadata may have to be reclaimed to achieve this. +allocinfo +~~~~~~~ + +Provides information about memory allocations at all locations in the code +base. Each allocation in the code is identified by its source file, line +number, module and the function calling the allocation. The number of bytes +allocated at each location is reported. + +Example output. + +:: + + > cat /proc/allocinfo + + 153MiB mm/slub.c:1826 module:slub func:alloc_slab_page + 6.08MiB mm/slab_common.c:950 module:slab_common func:_kmalloc_order + 5.09MiB mm/memcontrol.c:2814 module:memcontrol func:alloc_slab_obj_exts + 4.54MiB mm/page_alloc.c:5777 module:page_alloc func:alloc_pages_exact + 1.32MiB include/asm-generic/pgalloc.h:63 module:pgtable func:__pte_alloc_one + 1.16MiB fs/xfs/xfs_log_priv.h:700 module:xfs func:xlog_kvmalloc + 1.00MiB mm/swap_cgroup.c:48 module:swap_cgroup func:swap_cgroup_prepare + 734KiB fs/xfs/kmem.c:20 module:xfs func:kmem_alloc + 640KiB kernel/rcu/tree.c:3184 module:tree func:fill_page_cache_func + 640KiB drivers/char/virtio_console.c:452 module:virtio_console func:alloc_buf + ... + + meminfo ~~~~~~~ diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h new file mode 100644 index 000000000000..64f536b80380 --- /dev/null +++ b/include/asm-generic/codetag.lds.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_GENERIC_CODETAG_LDS_H +#define __ASM_GENERIC_CODETAG_LDS_H + +#define SECTION_WITH_BOUNDARIES(_name) \ + . = ALIGN(8); \ + __start_##_name = .; \ + KEEP(*(_name)) \ + __stop_##_name = .; + +#define CODETAG_SECTIONS() \ + SECTION_WITH_BOUNDARIES(alloc_tags) + +#endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 67d8dd2f1bde..f04661824939 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -50,6 +50,8 @@ * [__nosave_begin, __nosave_end] for the nosave data */ +#include + #ifndef LOAD_OFFSET #define LOAD_OFFSET 0 #endif @@ -367,6 +369,7 @@ . = ALIGN(8); \ BOUNDED_SECTION_BY(__dyndbg_classes, ___dyndbg_classes) \ BOUNDED_SECTION_BY(__dyndbg, ___dyndbg) \ + CODETAG_SECTIONS() \ LIKELY_PROFILE() \ BRANCH_PROFILE() \ TRACE_PRINTKS() \ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h new file mode 100644 index 000000000000..cf55a149fa84 --- /dev/null +++ b/include/linux/alloc_tag.h @@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * allocation tagging + */ +#ifndef _LINUX_ALLOC_TAG_H +#define _LINUX_ALLOC_TAG_H + +#include +#include +#include +#include +#include +#include +#include + +struct alloc_tag_counters { + u64 bytes; + u64 calls; +}; + +/* + * An instance of this structure is created in a special ELF section at every + * allocation callsite. At runtime, the special section is treated as + * an array of these. Embedded codetag utilizes codetag framework. + */ +struct alloc_tag { + struct codetag ct; + struct alloc_tag_counters __percpu *counters; +} __aligned(8); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) +{ + return container_of(ct, struct alloc_tag, ct); +} + +#ifdef ARCH_NEEDS_WEAK_PER_CPU +/* + * When percpu variables are required to be defined as weak, static percpu + * variables can't be used inside a function (see comments for DECLARE_PER_CPU_SECTION). + */ +#error "Memory allocation profiling is incompatible with ARCH_NEEDS_WEAK_PER_CPU" +#endif + +#define DEFINE_ALLOC_TAG(_alloc_tag, _old) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ + static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; \ + struct alloc_tag * __maybe_unused _old = alloc_tag_save(&_alloc_tag) + +DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static inline bool mem_alloc_profiling_enabled(void) +{ + return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + &mem_alloc_profiling_key); +} + +static inline struct alloc_tag_counters alloc_tag_read(struct alloc_tag *tag) +{ + struct alloc_tag_counters v = { 0, 0 }; + struct alloc_tag_counters *counter; + int cpu; + + for_each_possible_cpu(cpu) { + counter = per_cpu_ptr(tag->counters, cpu); + v.bytes += counter->bytes; + v.calls += counter->calls; + } + + return v; +} + +static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + struct alloc_tag *tag; + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN_ONCE(ref && !ref->ct, "alloc_tag was not set\n"); +#endif + if (!ref || !ref->ct) + return; + + tag = ct_to_alloc_tag(ref->ct); + + this_cpu_sub(tag->counters->bytes, bytes); + this_cpu_dec(tag->counters->calls); + + ref->ct = NULL; +} + +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + __alloc_tag_sub(ref, bytes); +} + +static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) +{ + __alloc_tag_sub(ref, bytes); +} + +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN_ONCE(ref && ref->ct, + "alloc_tag was not cleared (got tag for %s:%u)\n",\ + ref->ct->filename, ref->ct->lineno); + + WARN_ONCE(!tag, "current->alloc_tag not set"); +#endif + if (!ref || !tag) + return; + + ref->ct = &tag->ct; + this_cpu_add(tag->counters->bytes, bytes); + this_cpu_inc(tag->counters->calls); +} + +#else + +#define DEFINE_ALLOC_TAG(_alloc_tag, _old) +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} +static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) {} +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, + size_t bytes) {} + +#endif + +#endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 12a2554a3164..d35901d8de06 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -767,6 +767,10 @@ struct task_struct { unsigned int flags; unsigned int ptrace; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif + #ifdef CONFIG_SMP int on_cpu; struct __call_single_node wake_entry; @@ -806,6 +810,7 @@ struct task_struct { struct task_group *sched_task_group; #endif + #ifdef CONFIG_UCLAMP_TASK /* * Clamp values requested for a scheduling entity. @@ -2457,4 +2462,23 @@ static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); } extern void sched_set_stop_task(int cpu, struct task_struct *stop); +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) +{ + swap(current->alloc_tag, tag); + return tag; +} + +static inline void alloc_tag_restore(struct alloc_tag *tag, struct alloc_tag *old) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN(current->alloc_tag != tag, "current->alloc_tag was changed:\n"); +#endif + current->alloc_tag = old; +} +#else +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) { return NULL; } +#define alloc_tag_restore(_tag, _old) +#endif + #endif diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 2acbef24e93e..475a14e70566 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -966,6 +966,31 @@ config CODE_TAGGING bool select KALLSYMS +config MEM_ALLOC_PROFILING + bool "Enable memory allocation profiling" + default n + depends on PROC_FS + depends on !DEBUG_FORCE_WEAK_PER_CPU + select CODE_TAGGING + help + Track allocation source code and record total allocation size + initiated at that code location. The mechanism can be used to track + memory leaks with a low performance and memory impact. + +config MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + bool "Enable memory allocation profiling by default" + default y + depends on MEM_ALLOC_PROFILING + +config MEM_ALLOC_PROFILING_DEBUG + bool "Memory allocation profiler debugging" + default n + depends on MEM_ALLOC_PROFILING + select MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + help + Adds warnings with helpful error messages for memory allocation + profiling. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index b50212b5b999..4525469637fe 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -234,6 +234,8 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_CODE_TAGGING) += codetag.o +obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o + lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c new file mode 100644 index 000000000000..4fc031f9cefd --- /dev/null +++ b/lib/alloc_tag.c @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include + +static struct codetag_type *alloc_tag_cttype; + +DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static void *allocinfo_start(struct seq_file *m, loff_t *pos) +{ + struct codetag_iterator *iter; + struct codetag *ct; + loff_t node = *pos; + + iter = kzalloc(sizeof(*iter), GFP_KERNEL); + m->private = iter; + if (!iter) + return NULL; + + codetag_lock_module_list(alloc_tag_cttype, true); + *iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(iter)) != NULL && node) + node--; + + return ct ? iter : NULL; +} + +static void *allocinfo_next(struct seq_file *m, void *arg, loff_t *pos) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + struct codetag *ct = codetag_next_ct(iter); + + (*pos)++; + if (!ct) + return NULL; + + return iter; +} + +static void allocinfo_stop(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)m->private; + + if (iter) { + codetag_lock_module_list(alloc_tag_cttype, false); + kfree(iter); + } +} + +static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + s64 bytes = counter.bytes; + char val[10], *p = val; + + if (bytes < 0) { + *p++ = '-'; + bytes = -bytes; + } + + string_get_size(bytes, 1, + STRING_SIZE_BASE2|STRING_SIZE_NOSPACE, + p, val + ARRAY_SIZE(val) - p); + + seq_buf_printf(out, "%8s %8llu ", val, counter.calls); + codetag_to_text(out, ct); + seq_buf_putc(out, ' '); + seq_buf_putc(out, '\n'); +} + +static int allocinfo_show(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + char *bufp; + size_t n = seq_get_buf(m, &bufp); + struct seq_buf buf; + + seq_buf_init(&buf, bufp, n); + alloc_tag_to_text(&buf, iter->ct); + seq_commit(m, seq_buf_used(&buf)); + return 0; +} + +static const struct seq_operations allocinfo_seq_op = { + .start = allocinfo_start, + .next = allocinfo_next, + .stop = allocinfo_stop, + .show = allocinfo_show, +}; + +static void __init procfs_init(void) +{ + proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); +} + +static bool alloc_tag_module_unload(struct codetag_type *cttype, + struct codetag_module *cmod) +{ + struct codetag_iterator iter = codetag_get_ct_iter(cttype); + struct alloc_tag_counters counter; + bool module_unused = true; + struct alloc_tag *tag; + struct codetag *ct; + + for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { + if (iter.cmod != cmod) + continue; + + tag = ct_to_alloc_tag(ct); + counter = alloc_tag_read(tag); + + if (WARN(counter.bytes, "%s:%u module %s func:%s has %llu allocated at module unload", + ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) + module_unused = false; + } + + return module_unused; +} + +static struct ctl_table memory_allocation_profiling_sysctls[] = { + { + .procname = "mem_profiling", + .data = &mem_alloc_profiling_key, +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + .mode = 0444, +#else + .mode = 0644, +#endif + .proc_handler = proc_do_static_key, + }, + { } +}; + +static int __init alloc_tag_init(void) +{ + const struct codetag_type_desc desc = { + .section = "alloc_tags", + .tag_size = sizeof(struct alloc_tag), + .module_unload = alloc_tag_module_unload, + }; + + alloc_tag_cttype = codetag_register_type(&desc); + if (IS_ERR_OR_NULL(alloc_tag_cttype)) + return PTR_ERR(alloc_tag_cttype); + + register_sysctl_init("vm", memory_allocation_profiling_sysctls); + procfs_init(); + + return 0; +} +module_init(alloc_tag_init); diff --git a/scripts/module.lds.S b/scripts/module.lds.S index bf5bcf2836d8..45c67a0994f3 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -9,6 +9,8 @@ #define DISCARD_EH_FRAME *(.eh_frame) #endif +#include + SECTIONS { /DISCARD/ : { *(.discard) @@ -47,12 +49,17 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) + CODETAG_SECTIONS() } .rodata : { *(.rodata .rodata.[0-9a-zA-Z_]*) *(.rodata..L*) } +#else + .data : { + CODETAG_SECTIONS() + } #endif } From patchwork Tue Oct 24 13:46:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5968C25B48 for ; Tue, 24 Oct 2023 13:47:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B8236B0275; Tue, 24 Oct 2023 09:47:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43F796B0276; Tue, 24 Oct 2023 09:47:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21E0A6B0277; Tue, 24 Oct 2023 09:47:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0C39F6B0275 for ; Tue, 24 Oct 2023 09:47:20 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E2DDCA04E6 for ; Tue, 24 Oct 2023 13:47:19 +0000 (UTC) X-FDA: 81380481798.28.DE3855B Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 2B3A320023 for ; Tue, 24 Oct 2023 13:47:17 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JSzeVxXn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 35co3ZQYKCIo685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=35co3ZQYKCIo685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155238; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xnkv01D37mdYs16/e0X4XkgAogHBJsuQ+iVjrKs+el4=; b=uCmRxjYa89Jk7kxrANfQt7Tea915BfuG61kgg2tH+1b9HsPxNvdJe6lRzP6Sx9WEteQE9V xuBXgsJ2O6vCVEBmmTpZKFdDFmlh82KJhuRDeNE5U5lIfpyBkw//ZzVluQ3N20j6s3g7vv shUo8hedtEDY13hHiZ7FYWLTY++HxKA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JSzeVxXn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 35co3ZQYKCIo685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=35co3ZQYKCIo685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155238; a=rsa-sha256; cv=none; b=pokIYdQwswPspBZzczYJ9rkkNVZ935NPJEfLM2yhE45HIu8u43WavFsWAYvm6rQ++6fmIV a7bGJDybG1xlXAjzUMgWKeJZOFLTEpjwafjP8xDlxVjzrvBzPBdNhVluxEAtsOKFVYy+8C B6JywAaA/JYJmSTQEnF/6SVw21zd0NU= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a824ef7a83so58262227b3.0 for ; Tue, 24 Oct 2023 06:47:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155237; x=1698760037; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Xnkv01D37mdYs16/e0X4XkgAogHBJsuQ+iVjrKs+el4=; b=JSzeVxXnOQhmLqGPb0YdP53q5oPrMNjGhOSH+RlpJjWC7OhDn9epTpKI4HfNMZqvLG azknkxEwsij2LR/hKJAhJ0FKea/BQMqyFwYSR8qBdTCFvSfelxQYVngIBiCh4DHdTd3o JXeW7XtaIR6ZHQOMnZiStsiYo+fTqnS8h6xkRrQ5kfLYMBFiaKbQvj3FsbcsujRCpFSv XNDtmJvryEEFDA3L73NpZABSoB80kj0wqfbOV2Tfiq+3Km+K5cdH2tFk5IPHbFvfh7oK zNlGr/TkmxEUPMjnqUVj9GqNcvJvae4f+j2IFX5k3oD6B8BEXKZ54k1WwB9tyfhOFV48 D4xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155237; x=1698760037; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Xnkv01D37mdYs16/e0X4XkgAogHBJsuQ+iVjrKs+el4=; b=c07FRJmZvhAdxG3inRMw8YHViskO8/XVBBbEJVTTmvQw2uyoyGvjS22sAF818Tob4M 0koLmIeZVVlxvLAy0fWlWheli6zBbOTtH3JorwtbBg7DOOaNku0ZNMUOEGcSy28Y/Mse EnGpFrXznYpfHpzZs2l8P6B2B9t3mDiD8jhQ2QETiHlKC+0LK6Q4iVgtk9DC5SzP1fqZ hWpb67SYv10hmhxGby+eKJaXd52AsJoAeYJFeugXWS/X9NwFx6iqzLOKs0YEyJF6l1m+ 50OPQYpd9XXnbZs0Yn3JTMIWXz+Mef7TLtdvfFSVSKAgrYJTRmvCV2qitg3BgJsU9DyI 593w== X-Gm-Message-State: AOJu0YzCBE7FyZDnCZ4or111v8qHfGgBElj+Fger3lqao472BEGDb5tq ajqjKFQhrCgL8jvb9RqXyjNRUIvyf8Q= X-Google-Smtp-Source: AGHT+IF+McJ3mA6RssmQwfp7//Pg5OsKi1Xdwm0aQawyQdn+hvoXflHVr9pzXpd28dOZCYLaDsPpXEXkw0g= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a0d:d954:0:b0:5a7:bfcf:2cb8 with SMTP id b81-20020a0dd954000000b005a7bfcf2cb8mr268838ywe.1.1698155237063; Tue, 24 Oct 2023 06:47:17 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:13 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-17-surenb@google.com> Subject: [PATCH v2 16/39] lib: introduce support for page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 2B3A320023 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: uhgjpzrdigfxer4934k6euwb1a5aehmj X-HE-Tag: 1698155237-872264 X-HE-Meta: U2FsdGVkX183vm24nD1OZnbvI1JMTOTegzIc8HO6XvlMXENuVPkugRhw/uFZ1BhsaMqROBI+4TTzO8Aui2xTLNKBjGpjbDfWGY+GRaGfBEfVPpWm5iFhJMmUI62Ef+ZmALhd8avUMEnlV0Ke/Pc7mjSAgyQXCE8f7D2KPidZ0M7uy+IssA0cX5cz8bfSzxzS8pWjF4pwjrpYqBhso66vMUSkWXngBaoYpsBLTlvkJc8EX0xFqRN993w2zhc9BU8UDY9T06viToxy29O38lFUjlt0fXmHTst8zPnwKI3+624vZIJfK3jrw/b9kHcyCVCXAPTpDNu3Hr9kMOEVa6ETkqRpSOo2a3+vxm66DEhxl5jcgoKGFff2I1tpvvw8+mkGw3tYXxr1o4J0n9UC4abSeSA23wNR0HVC7HBxtJusnw8NVMUXyXRPZE27M2Ioehp063/IpgECP1FadytHuvM1OwdK0PhKoW2xtMM/5vMj6ecPGr100QjV4hNjKjg04MingEaC6WVRzhei8QKbjt4OenPnB5EWK2al2nND/HvYwvVtuCP7iNoGpI3P9ZrR95ioesi7GfWrBPClkbb/yNRF0/s+jlC3b7eoCtdzwxiXRPBU0lJDGyHG+VD25Ut0W08fm9P+P1QuPH9tzE/uY00NvN70DzLnS8GD4FrTugAX8S5vMLr/dwI+KHNMyVCXNCfNx4YuEFKcdLqntPap36K9it/MXzHytP9DDAIrkawAzJn73dqoqbXvuH4WvjHJvxo8CTJutZXUyZ6ZTYqa7Fsn/nYMGtJJxazmXS4aZj8a5I9yzUEzhUYSwz/zwAbLGGlHpHO2rTO0Y6pZEr9Lpo5qJgOyOnCxyg9ZSa9FBSLXkXa85OggVsBfC2hg7YS8FJJipQU8RkzBpfX6m54BS0JJj5eZKWpgFlOjMGdTPVjQ/l/i20hQf7yCzucq3ycVVlvqImzoF1ZvGWxdnNGNkOP QQX++u+X LGTK/r8tEfKkl0BbRovuzMq8qAeBqP7aXjd24nZtOmuHGSlNodn81pBXwkpJ2kPDUCMrdLR3WgRWeXVw1zfgWzBPvuyNGShArdQWPYTz1j78lhJY6ry1tpmGPwgYuHFpd4n+o157PDxXw5Z6JQIA2NweUdTUKIM0zNsIDXICq0scJDQOfrOvkNXnBEbMzzZPOODv8RtsP4ZyRyqUyU4zeH4zgnZSBoqr1d0aPkbzymtE4IrcjjZoDmrf/rgi/z/kiq5einrYJRkS+uCFq/WRaKKf6RIv7QpSQTaBcqRlHT6JNbzeI4O07+5H3BogN3LRTvPrJgbOKiabmuYYDWnGZOBrTOJanppPHFvpedWuHkHvXE8lubeJWjg04UmKX7yHoi64vAK6LmYo6rKCj9/BL1dsM+OcX0W7sIsXC4vmeHCXF5OGjMouN6ltikz0dXrIEd7ScVni9ncTu2PM2exCPwvOq40bteZF6x3RjRHP1dkTrlxyEqcOVJZPhfW3bUweZ61hmOu0ZUS9q3/cS5tJCIvzUxPNEdtOag2XDPUrvpBILlH5CR8WI7NFyqr0DAoo2ImgKX2k/Tgle6lu2Dzw9MBjKtm0fadYSDq7QA6kyl5SOf9e3Rq/hQFA1ZTDygcfly6I/JuwZnme9AGkLOuKt0uuZ4d2XOPvWUZxV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a page_ext field. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/page_ext.h | 1 - include/linux/pgalloc_tag.h | 73 +++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 1 + lib/alloc_tag.c | 17 +++++++++ mm/mm_init.c | 1 + mm/page_alloc.c | 4 ++ mm/page_ext.c | 4 ++ 7 files changed, 100 insertions(+), 1 deletion(-) create mode 100644 include/linux/pgalloc_tag.h diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index be98564191e6..07e0656898f9 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -4,7 +4,6 @@ #include #include -#include struct pglist_data; diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h new file mode 100644 index 000000000000..a060c26eb449 --- /dev/null +++ b/include/linux/pgalloc_tag.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * page allocation tagging + */ +#ifndef _LINUX_PGALLOC_TAG_H +#define _LINUX_PGALLOC_TAG_H + +#include + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +#include + +extern struct page_ext_operations page_alloc_tagging_ops; +extern struct page_ext *page_ext_get(struct page *page); +extern void page_ext_put(struct page_ext *page_ext); + +static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +{ + return (void *)page_ext + page_alloc_tagging_ops.offset; +} + +static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) +{ + return (void *)ref - page_alloc_tagging_ops.offset; +} + +static inline union codetag_ref *get_page_tag_ref(struct page *page) +{ + if (page && mem_alloc_profiling_enabled()) { + struct page_ext *page_ext = page_ext_get(page); + + if (page_ext) + return codetag_ref_from_page_ext(page_ext); + } + return NULL; +} + +static inline void put_page_tag_ref(union codetag_ref *ref) +{ + page_ext_put(page_ext_from_codetag_ref(ref)); +} + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int order) +{ + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE << order); + put_page_tag_ref(ref); + } +} + +static inline void pgalloc_tag_sub(struct page *page, unsigned int order) +{ + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_sub(ref, PAGE_SIZE << order); + put_page_tag_ref(ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int order) {} +static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + +#endif /* _LINUX_PGALLOC_TAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 475a14e70566..e1eda1450d68 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -972,6 +972,7 @@ config MEM_ALLOC_PROFILING depends on PROC_FS depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING + select PAGE_EXTENSION help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 4fc031f9cefd..2d5226d9262d 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -124,6 +125,22 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, return module_unused; } +static __init bool need_page_alloc_tagging(void) +{ + return true; +} + +static __init void init_page_alloc_tagging(void) +{ +} + +struct page_ext_operations page_alloc_tagging_ops = { + .size = sizeof(union codetag_ref), + .need = need_page_alloc_tagging, + .init = init_page_alloc_tagging, +}; +EXPORT_SYMBOL(page_alloc_tagging_ops); + static struct ctl_table memory_allocation_profiling_sysctls[] = { { .procname = "mem_profiling", diff --git a/mm/mm_init.c b/mm/mm_init.c index 50f2f34745af..8e72e431dc35 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include "internal.h" diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 95546f376302..d490d0f73e72 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include "internal.h" #include "shuffle.h" @@ -1093,6 +1094,7 @@ static __always_inline bool free_pages_prepare(struct page *page, __memcg_kmem_uncharge_page(page, order); reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, order); return false; } @@ -1135,6 +1137,7 @@ static __always_inline bool free_pages_prepare(struct page *page, page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, order); if (!PageHighMem(page)) { debug_check_no_locks_freed(page_address(page), @@ -1535,6 +1538,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); + pgalloc_tag_add(page, current, order); } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, diff --git a/mm/page_ext.c b/mm/page_ext.c index 4548fcc66d74..3c58fe8a24df 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -10,6 +10,7 @@ #include #include #include +#include /* * struct page extension @@ -82,6 +83,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT) &page_idle_ops, #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + &page_alloc_tagging_ops, +#endif #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif From patchwork Tue Oct 24 13:46:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434586 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28F24C00A8F for ; Tue, 24 Oct 2023 13:47:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FBD96B0277; Tue, 24 Oct 2023 09:47:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 883F46B0278; Tue, 24 Oct 2023 09:47:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DAF86B0279; Tue, 24 Oct 2023 09:47:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5365B6B0277 for ; Tue, 24 Oct 2023 09:47:22 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3B8154061D for ; Tue, 24 Oct 2023 13:47:22 +0000 (UTC) X-FDA: 81380481924.07.DB59746 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 4FBA9C0025 for ; Tue, 24 Oct 2023 13:47:20 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1KmXcRL+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 358o3ZQYKCIw8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=358o3ZQYKCIw8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155240; a=rsa-sha256; cv=none; b=zV1Aid9EHIW7mdE7sjAm3N53Gw8ZfpI1Q152ST48IHjJCKfWeunDkPc3CDHV0obdq7SyPN ZpjmYvXIczeoaO0MoO9M9W8QNmL89LKnDmg46sGT2rLp9ZaY5PkWL2FCELHBX7N3JuDUkD sfiqojtb5YjWmNIPGCW4DuLz0Mz6hVQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1KmXcRL+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 358o3ZQYKCIw8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=358o3ZQYKCIw8A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155240; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rU55lfnTUKEjh0uwdJqAIgtITtAkLPiyFECJvQc6NhE=; b=XTUjOlkmhCQn5UvidjertUwAt6d1waLQp1OK/CVEMV+Ac2mXGixB7bNX8ZW+46egAlMeNM ZlC6YWIsTUXXxoc6c3pUTBTNGluJ8SlskWiGJZ765rBI6NDs5q8RH07rHIAXEQymbkMoSZ i5H+f3lJHCsTslzlJFfMPIbu87NUJRk= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9a45e7e0f9so5267413276.0 for ; Tue, 24 Oct 2023 06:47:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155239; x=1698760039; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rU55lfnTUKEjh0uwdJqAIgtITtAkLPiyFECJvQc6NhE=; b=1KmXcRL+ZOZpoUCwaHDh+cVKt0oVw1AV+GmW55gmh1xcFUKo7Dy339Hukip8g/3n08 br8Iy8O7Ms1RQeaEmlvyleMrUYkCq21W7Juns7fECR6nKYX+LzLAtatf+1n0y7sPKEik IrMB95Ywxy9L9N7CP27TPpFOUZnACIzjHofSApYwIrhi7nxCwV/9YbgGI7mojpr+LPLh HNMveMxUGAB1LD6dSIkcBspxybYsUtpxfY937FZqyf1EJiU/v/0ggiF7jVfmV+8XnDRS SvwuV937loumdJTjOdDDs/NmTbHyuBIOfnkVWKK/zfja4IGi4EG/UOVxBian7tj+Sqka ZdNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155239; x=1698760039; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rU55lfnTUKEjh0uwdJqAIgtITtAkLPiyFECJvQc6NhE=; b=SoHQsXxzyf03fwrzQCYtWoj7SdLWjWGn1maDSLZ7jl0ecYe4u4klS/wFa0y8wKxLnQ jDFZyL1fxOkJYgToS6u8SIttjXLzlP8IZzKWoDNstZTFjT0mDBvSK14/5gYydejCc2ix O+goVNShLbDRJTu8wWLgaUsmag5lQCgXGiQNnaTQ5jnZeK/Yl7ETAbO4it3GV070c43u 036u5NAxqPY4vjhWQf1TJ/p7bjbljemPHr40HRBbvfhUuyXsQDetIl4nmutErjuQtlJr SYeExm9/8+Pctgv+1krjodMlgvUwoGphazdZJibi9gNU/b7UpznWwB1zsoLAQz9dp/ao X39A== X-Gm-Message-State: AOJu0Yyue1PTugyMCVvc6xi+jmcvo6SbUGpQKXqauNd/LsSQrswUMCo7 kZIRpDLu+b+Am24VAL+3xFhzxzKceCM= X-Google-Smtp-Source: AGHT+IGatz5ZTD8qO9Tv0HGrMDj6Iuk2tgVA4F+Z2fO2W9B6yDgy8GrA8nufWNVDJbI0uZZFq7uH9sjFSH4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a05:6902:105:b0:da0:3da9:ce08 with SMTP id o5-20020a056902010500b00da03da9ce08mr35563ybh.10.1698155239301; Tue, 24 Oct 2023 06:47:19 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:14 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-18-surenb@google.com> Subject: [PATCH v2 17/39] change alloc_pages name in dma_map_ops to avoid name conflicts From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 4FBA9C0025 X-Stat-Signature: a4ee4h9do8knz9y13xg9ytq1pknhqxhq X-HE-Tag: 1698155240-703400 X-HE-Meta: U2FsdGVkX1/5AYC07T5xFbcdyii9W3k38/0xuX/1HffTRXzjM0+joRXNXT9csSJdPR7CyKeLNCwRCcSNxKVs2SIbY02tHEFtiA7+9rauWMUAv/9nweZ46A4JA7TYNHtSin8MM6ShaKDAbb+W1p56TBJQhydBZdyMgZck4SSDouikWY9H1nHjHG74HuuQRn5G2K4BIvk1fsftjxWh3j24zzeU1SkoQAM+ZHFTGcYuEbqXk/hGW+QyrPf/AuKBIL/arTKYvRq3qqbghcG8NEh1QK5KqmjJbJEOTsyl1xFdpjmgCylmndTGSkE0tMEqA/6B6GxkMIigoEHN1EAERyvxNHWlX4wHrLSDQlDW0bjc1I298vs37q//xsykOnKA/lFgcuk6hNbk5tzKSOaj7Q+TD8SocssHYYjFqKIdSbQtJIg2IqSHkfKAWjXd065Pja1IFsG9sRsqbCSD7oWPhnkNtfrJneCnkgLhc2m4ip+DZvieRtURdrWIfJj/uTAIYAYgOqJiNB3Aq08TGnUgYU2K216JjN2IXdjJM819l1qJarKpXsX0vh1bYC2QacNtzZWoZmmi/PvDrUuwC8R4q/9eLU+lLL1rvpPzcUSCG3zgRDb1sNOTDl0qLPdzHkW7uaJAYkZeTIxl2bCDODTPDmSzQeQHKUrY81QE8xUigh8aH0O3Ar0GgmR8ef1faTLpMRZxBLBTCEdEXojVcpX3qvPOAYLZ4EVbSme7G3Tpf9ZM6QBvkH771MR6R0vSYuLIkj3a+KMlrvfwi2fDxr2OMfZJ3eg9sbWDboeqZazwti0D0QzegfDoh/Gk4csAvlnHNWcZs8ONcYRZwfyTWLbt0j2DgCAyRYxxtNtHPwERpKmGSZhUFkcN8wgxTjooyACrBBJjz6WkomYFNaBOdf3XTVFzVkN/DlUnF3MDeMbyepzWyF1+GbNT1Vo4DaqNcCKra5smhlsLiDk2uqCQV0uqwaV /P5KtdxP jbrYWcg+1qVqk1bWr6CAAnBpIzVYsArS5oBzDB3fMjGpcJFCX7WjiE+kvsgMsh0vAA4eXdJhArBwftbNRWqXMYWJbRxvB9vOqVs40HU2SfRxcpZ4xfNzWkJmHMy9GR4TNsqDUhYGdruCN0byOQjr2FE5WJVlTSjSxXSA0D7UmJT0b0xVThyyqYNOMZst3WZeFRuJACZ6JJJ36DIuOx5FRHOolg9E/7wr5zinb88YlyEY/0l1UQUW9YLsgX19K3ZITO8ZJepgrzh7M7wq50+7GaZ0SHt3jfD810MK1yjpNqXPHUvlO+7hW4FLJoOzMBZZaDYZf6zRoY7iEO0svyB7DeYTh9mAkIH+v/N4vpTH6DA1R7rcjB769Th86gaI9XzbgRwlrzGL5LTv6YraXCQhD3oGRxnGRSho8jcjOSvyqhQtGXQ0WooOwm9cNBXBzrmVhhbcRnxemcNtWSLCBaTsGCaLRoneXm83x8C4Wl9izKc2Anls6NnwCbljZ2GwO51/pprR1/FzJasRv3j0nWEJVCgAzaGLgXnSN+XuEgRDmQE9YlR/JhCnJXkjHON4dVtys2X/HTxH0pCMpYsdmA8IE5CI10JWjloAWeaLgfDrX8CNglFnfsH3F0W312+KiGaqiHx46n1CAxSuBgehXDBZQkstZOO4klVcaTVMOnNtopOU39es3ofE8nHr1ZGHDAuKOj+wV9sWRDnJ4VCgoaPODzoaxdRK3gfmNuovz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After redefining alloc_pages, all uses of that name are being replaced. Change the conflicting names to prevent preprocessor from replacing them when it's not intended. Signed-off-by: Suren Baghdasaryan --- arch/x86/kernel/amd_gart_64.c | 2 +- drivers/iommu/dma-iommu.c | 2 +- drivers/xen/grant-dma-ops.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/mapping.c | 4 ++-- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index 56a917df410d..842a0ec5eaa9 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -676,7 +676,7 @@ static const struct dma_map_ops gart_dma_ops = { .get_sgtable = dma_common_get_sgtable, .dma_supported = dma_direct_supported, .get_required_mask = dma_direct_get_required_mask, - .alloc_pages = dma_direct_alloc_pages, + .alloc_pages_op = dma_direct_alloc_pages, .free_pages = dma_direct_free_pages, }; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 4b1a88f514c9..28b7b2d10655 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1603,7 +1603,7 @@ static const struct dma_map_ops iommu_dma_ops = { .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .alloc_noncontiguous = iommu_dma_alloc_noncontiguous, .free_noncontiguous = iommu_dma_free_noncontiguous, diff --git a/drivers/xen/grant-dma-ops.c b/drivers/xen/grant-dma-ops.c index 76f6f26265a3..29257d2639db 100644 --- a/drivers/xen/grant-dma-ops.c +++ b/drivers/xen/grant-dma-ops.c @@ -282,7 +282,7 @@ static int xen_grant_dma_supported(struct device *dev, u64 mask) static const struct dma_map_ops xen_grant_dma_ops = { .alloc = xen_grant_dma_alloc, .free = xen_grant_dma_free, - .alloc_pages = xen_grant_dma_alloc_pages, + .alloc_pages_op = xen_grant_dma_alloc_pages, .free_pages = xen_grant_dma_free_pages, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 946bd56f0ac5..4f1e3f1fc44e 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -403,6 +403,6 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .dma_supported = xen_swiotlb_dma_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index f2fc203fb8a1..3a8a015fdd2e 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -28,7 +28,7 @@ struct dma_map_ops { unsigned long attrs); void (*free)(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs); - struct page *(*alloc_pages)(struct device *dev, size_t size, + struct page *(*alloc_pages_op)(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp); void (*free_pages)(struct device *dev, size_t size, struct page *vaddr, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index e323ca48f7f2..58e490e2cfb4 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -570,9 +570,9 @@ static struct page *__dma_alloc_pages(struct device *dev, size_t size, size = PAGE_ALIGN(size); if (dma_alloc_direct(dev, ops)) return dma_direct_alloc_pages(dev, size, dma_handle, dir, gfp); - if (!ops->alloc_pages) + if (!ops->alloc_pages_op) return NULL; - return ops->alloc_pages(dev, size, dma_handle, dir, gfp); + return ops->alloc_pages_op(dev, size, dma_handle, dir, gfp); } struct page *dma_alloc_pages(struct device *dev, size_t size, From patchwork Tue Oct 24 13:46:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43788C00A8F for ; Tue, 24 Oct 2023 13:47:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C31826B0278; Tue, 24 Oct 2023 09:47:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B94466B0279; Tue, 24 Oct 2023 09:47:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 995FD6B027A; Tue, 24 Oct 2023 09:47:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7C0C06B0278 for ; Tue, 24 Oct 2023 09:47:24 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5969FA0648 for ; Tue, 24 Oct 2023 13:47:24 +0000 (UTC) X-FDA: 81380482008.18.02EB280 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf21.hostedemail.com (Postfix) with ESMTP id 7D7B11C0022 for ; Tue, 24 Oct 2023 13:47:22 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rGFroAUS; spf=pass (imf21.hostedemail.com: domain of 36co3ZQYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=36co3ZQYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155242; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zb+0nW/ZZ6NW74eJtanLAYQ/ebma3WHtobF3vZ6u4HE=; b=PJiKIoT5t1aDoHj0+UNW0bF+6TF+Vv8RYKsdYseB0hSgkVrf+3V30AU9dHI9PJ6Gujn7Em 08AxVKA42AHW1zTRxUztEHxdNywAK3HuyX8SV7zsEvl+Ur48wnvZ4AV0cyAhAqoe84CX5G 0Dx09/baAmL2C9MLOp+bgaRDRdvG7To= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rGFroAUS; spf=pass (imf21.hostedemail.com: domain of 36co3ZQYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=36co3ZQYKCI4AC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155242; a=rsa-sha256; cv=none; b=7AtEqIySMYkz45bIULtHAKo7kjR1zW32b+Dy8VXNCNUh6ssvikikkJ89DI9Ntgv6TaBu3u dF2ysYh2ZvQmYarcSInm5AQZ4iSG86Xh2TeqmPtCjQZH8SVE/wg5v0lI7ZySjtoDH9UIjb cmsq9v8u1/rDiyft4xNIdvdnKIbLoNw= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9b9f56489fso5394883276.1 for ; Tue, 24 Oct 2023 06:47:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155241; x=1698760041; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zb+0nW/ZZ6NW74eJtanLAYQ/ebma3WHtobF3vZ6u4HE=; b=rGFroAUSRqZBvMUDRP9Z6RfF5NFFn6+wtcOC4W7zdVndugUF1QPtf2YeHPKoa1LiiT xWR4Ae6VhgZvfL5FsQxv8hMgS0NuZ45HweTzhNntBIzwzyzPf+WNle0jrKD10ryDSfuE oiYNGalJOEAZsmQM08V1w8lPsa30WOFxeWwcHpsKyHtPENwulwDeII/Qp6pdo/+8k8tD IYjbM4yaSuPgXa8eAb02uxvoPwEd7ewWR2Wng8TZ9r/EDgfNcEQOqy7D2L5qyFKDEG8I TqtJ3BFcAo8k2xF8XOJISGfJH79mQWSP5nvup18oNfAHo5RKkll6D/gMALVe3Kv70C/+ 3x2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155241; x=1698760041; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zb+0nW/ZZ6NW74eJtanLAYQ/ebma3WHtobF3vZ6u4HE=; b=luqp0nNhhhORp7NaY3XIuBfk3OTLiHUBZ3fqNJqE09+l+wmSMRl5oZeuY1Tv2836bS 7Fjt/szS3JONER00NWqAvD27Dz2Kc+jO2Na4XGrPxzTey/d8nYeD6zGbiDbc3VcERRLo B0VuJ+syrMSQ4pLv+t9eQ4IsK0u7pWm7gLHt6f8NYlcgpE4myoYpvuY/JHiFGYEY+KXG smyzXqejr7fizDeibzpYx46ncgLCkVItmRh4TXLqVyQNtr5HoJFiXKezPN5r8coUxJD5 kShJxunj1FOWKPFko8h1HW5+9dblSCxrXUv5b/0S4Y4eu6JH/pBaU0t4MdmaV5efdkL3 qbcA== X-Gm-Message-State: AOJu0YzvYLcp5d939e9evxdYg2x+ubEQQ6BdkH8XZ2uSqsBVhQKdQd6U k0s7h9cYz98gMyxYyYSGUgZfTVuQ1d0= X-Google-Smtp-Source: AGHT+IEpPcNXSaLopMEVj+acIv3rs50iHU0ea/y8V1UNdWnab+iEcPBoSFwF1WABxkknLPWrzIDF+i6tstI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a05:6902:544:b0:d13:856b:c10a with SMTP id z4-20020a056902054400b00d13856bc10amr261040ybs.3.1698155241467; Tue, 24 Oct 2023 06:47:21 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:15 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-19-surenb@google.com> Subject: [PATCH v2 18/39] change alloc_pages name in ivpu_bo_ops to avoid conflicts From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 7D7B11C0022 X-Rspam-User: X-Stat-Signature: b844x4bgcipdb3yfcnkr46wwgs4xusf7 X-Rspamd-Server: rspam01 X-HE-Tag: 1698155242-990972 X-HE-Meta: U2FsdGVkX1++2JcyFuj8uvQ8GFnnVlzGmwYpzgiiIiKsle50Ie3tRHbCvxtZBSJcOyVCcnhldHL+7+s/c4ZtKEie/4UAEcClDTPvAGUnMSJQq9zc+xSa/rzgJqhoguMGdwL/LxRz+CSxxfkPHMRy3JkXnUBClNfPFmYO1NvhaP7PajyTNSMUTj01sgdyaqGlpfdRHwQcZJE99MZtDEorubNJ7YIUp/iWXEdpFKlugz9beOrMJUq0dGxmfCI9lhZt4PQJkG0tD3l6q6bDxS6sHmNPQLN+U45sXtXw4mZms2vfVaNM0603AmRnnaJjBgncSZII9GVOuxbhWwua4zKNwd6MhzQfMMP7Yrn3z/dVTz4wKCIkMFhJApUKtJx95pFE+y+kOuHIyp9N5EZ6F7IKagn+OhGL4U0+c18qucR5YBhDIEGU7Fj/9XRgL7bRzLHo3bIwENuCNODICC5BSaCHjT0Cr68+/JnXgJwTiw2RibOh5Dzc0q1SBZXvlv3t2fXaJi+x/BDTI3Q1nsROip6VPt7Dp4NIib/ywarsCtzlmMyDAbB5/kla7UrZ3Hgq/+oTWhdQVHWaclKUVcNILaL0mlxl2RCuU01Ktd+//wx4Wpxc+M7+qaWUxAD3GYLrem+7seMhxQxKY9joK0KZabFRSxpYusVh4RDSjB0n4tjpzGVyCp3heBWTP+/U0NKs4A1jG+pXCuCbPDNT2GE0lj19pvANuX5/yj96Wlr/svKk4fBXZG8b4VvILm+UYScKFxTLWOzghuJsFejN3/9ZZVopfOLoy9bfbemoXd9BOPDptm/IsuBZH2FtjSyFmPf9OfElU2/dzq75hIryK/EsepB731IODWdwFlJ7B0FnQCFcLRyZ07H73rhE31X2rV7zFDx12ENjyEtnLZsEzpTavNV0Adza+kqNwuAueINXH8kSLk+0DvR26SremTVaTN4dRQivB0UZ8xdlljRZelpG3uh N2/FzfVi qZ96KFN5Gm5clJI5RDMdh4/3PxrXmY2e8mkzEXHXoHQtQWWR9NJ3gemvHMl/yxFySNzMXIEcCucxudOaT4u+1jdL81ixKddtsh8xPJHu/A7v9mQa2BCgGcaxp5A+zO7wyBqjlJ6pqI2EA68tCdpCFR3+hvbe/2gppFeMTR44IzmFu73P1djL27DmBD/oua/1AF2PVjlxPYZKG+IQA8aeAE0Rn3oWuxWiA2Tzx76pmxbKhginR3UQ9QkgZ1KpfNCuDdijruqP3kEb9uEbhKi7Ofnl7IGYXr9iNwWOTFqUXFcybCz2WHO7+0yh16CMQd+vaqwxLehGdwCqJeq+cBlgl4uUIUUWmT0RlAqt+qDfL6aNqifVENvz9taKUJXX/gnmgkYoRYOwyPOQMh7F+Rn29YbRb6sHQVt+bDgfpQc3k8y+JKLfGvg5ruv7wqc84abKeByMKYwTJko5oqedsMd2dRjB0jjOTOO8572Dr2K/mNyZ6s4C9t3pXHxT1fXXvemG+5c4vyMg19ayaBqguWx7vRwuwbm1lX/o1YJIIt38jKfe9LmH767MU7Gh9Ob8g60NE7meBtzwc6kPl89npjPR3QxuMXByQ32CzTV9tC8vl/ZvD6RJBD6VZiOEjXW6Q3N46O+wmJ3UzOwVzV6xabeCQDIFc7v/ycfhXre7+jmtWMuV8DrbFxxJrA7F89W4p+1OUKfWg/8M6yM1yZo0F++ZC9+0M2Dg5DK0e0na6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet After redefining alloc_pages, all uses of that name are being replaced. Change the conflicting names to prevent preprocessor from replacing them when it's not intended. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- drivers/accel/ivpu/ivpu_gem.c | 8 ++++---- drivers/accel/ivpu/ivpu_gem.h | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_gem.c b/drivers/accel/ivpu/ivpu_gem.c index d09f13b35902..d324eaf5bbe3 100644 --- a/drivers/accel/ivpu/ivpu_gem.c +++ b/drivers/accel/ivpu/ivpu_gem.c @@ -61,7 +61,7 @@ static void prime_unmap_pages_locked(struct ivpu_bo *bo) static const struct ivpu_bo_ops prime_ops = { .type = IVPU_BO_TYPE_PRIME, .name = "prime", - .alloc_pages = prime_alloc_pages_locked, + .alloc_pages_op = prime_alloc_pages_locked, .free_pages = prime_free_pages_locked, .map_pages = prime_map_pages_locked, .unmap_pages = prime_unmap_pages_locked, @@ -134,7 +134,7 @@ static void ivpu_bo_unmap_pages_locked(struct ivpu_bo *bo) static const struct ivpu_bo_ops shmem_ops = { .type = IVPU_BO_TYPE_SHMEM, .name = "shmem", - .alloc_pages = shmem_alloc_pages_locked, + .alloc_pages_op = shmem_alloc_pages_locked, .free_pages = shmem_free_pages_locked, .map_pages = ivpu_bo_map_pages_locked, .unmap_pages = ivpu_bo_unmap_pages_locked, @@ -186,7 +186,7 @@ static void internal_free_pages_locked(struct ivpu_bo *bo) static const struct ivpu_bo_ops internal_ops = { .type = IVPU_BO_TYPE_INTERNAL, .name = "internal", - .alloc_pages = internal_alloc_pages_locked, + .alloc_pages_op = internal_alloc_pages_locked, .free_pages = internal_free_pages_locked, .map_pages = ivpu_bo_map_pages_locked, .unmap_pages = ivpu_bo_unmap_pages_locked, @@ -200,7 +200,7 @@ static int __must_check ivpu_bo_alloc_and_map_pages_locked(struct ivpu_bo *bo) lockdep_assert_held(&bo->lock); drm_WARN_ON(&vdev->drm, bo->sgt); - ret = bo->ops->alloc_pages(bo); + ret = bo->ops->alloc_pages_op(bo); if (ret) { ivpu_err(vdev, "Failed to allocate pages for BO: %d", ret); return ret; diff --git a/drivers/accel/ivpu/ivpu_gem.h b/drivers/accel/ivpu/ivpu_gem.h index 6b0ceda5f253..b81cf2af0b2d 100644 --- a/drivers/accel/ivpu/ivpu_gem.h +++ b/drivers/accel/ivpu/ivpu_gem.h @@ -42,7 +42,7 @@ enum ivpu_bo_type { struct ivpu_bo_ops { enum ivpu_bo_type type; const char *name; - int (*alloc_pages)(struct ivpu_bo *bo); + int (*alloc_pages_op)(struct ivpu_bo *bo); void (*free_pages)(struct ivpu_bo *bo); int (*map_pages)(struct ivpu_bo *bo); void (*unmap_pages)(struct ivpu_bo *bo); From patchwork Tue Oct 24 13:46:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 192E1C25B6C for ; Tue, 24 Oct 2023 13:47:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA1946B027A; Tue, 24 Oct 2023 09:47:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E04F96B027B; Tue, 24 Oct 2023 09:47:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C315C6B027C; Tue, 24 Oct 2023 09:47:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A53486B027A for ; Tue, 24 Oct 2023 09:47:26 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 81C6C1A04C8 for ; Tue, 24 Oct 2023 13:47:26 +0000 (UTC) X-FDA: 81380482092.21.EEC4C5B Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf16.hostedemail.com (Postfix) with ESMTP id B178918000A for ; Tue, 24 Oct 2023 13:47:24 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e6FwhPim; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 368o3ZQYKCJACEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=368o3ZQYKCJACEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155244; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aDBkAbyEeXJgt2cs2rmRo9d2BeozU6HvXUKgeAkvqwY=; b=UZEAy4HclMN7nT0yVDJi3gRfRRozaaWrLOwV9Y4uXyFwKHH65O6yT8Nfw4rQgNx8jxm0nt Xd42AfZtXDj7lLe2ONdlDE+Bu7UU+w7yyYaZyc5yCG9p7M4MZeT3typJdz9vLWhWhbKScA foiF8KzhIs7x3wtjIvRuLQdwBWiUU+U= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e6FwhPim; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 368o3ZQYKCJACEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=368o3ZQYKCJACEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155244; a=rsa-sha256; cv=none; b=j7HiDj9WOMHuaLtS9fcHbra3mcPFlkkwVub+on0LhsHE7+OcXm2Oq6iscDI8bC6EO5OTBO AWsuOn9pIIGznrOJYORAWL80pwx1ZqrSmldW4CnOQrDemCFGsDxquA9oh99xZ8ZIE8wTSP unCl4wKD4B/XMvKecZJ+0ooTqOnsWQ0= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9cafa90160so5239345276.2 for ; Tue, 24 Oct 2023 06:47:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155244; x=1698760044; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aDBkAbyEeXJgt2cs2rmRo9d2BeozU6HvXUKgeAkvqwY=; b=e6FwhPimS2RtTZ3deaM9nqGco5any/9tLE7uoqnI/tjoCO5XZ/I6ktQH/DTMIzmvPg g5Tw/14WxO88ItSCrYFx+eGre6I+WnNzOMW12/iQKDi+das0m3FDsDRgBXzfHkag6Q8Z /mH9zZN9h4TJyccbYWVt/HhnKbnyafWH5GlsfIMQLvuBolz55c4gF2dCdDKW3+7QrLYn OqyJV6TqrLhhKDHEIBcMXZWlacwIhV3tUM2ueVlT/KadxKQDmoZpIujAjz0JlsXW6n4G 6dCGm9/tptus+Z1FjET74E8CCAHtH4+iq1nmytJ1lhFsFyTYl7qVuFOZvSz0PpNneiNk kNUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155244; x=1698760044; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aDBkAbyEeXJgt2cs2rmRo9d2BeozU6HvXUKgeAkvqwY=; b=UO9rGGXxdRBaRku4kODFSUn9EsDaZp8+Wf5dHq5YhTD7kOF415aDa5/2Y1OZ3kJig0 SAmmQsWkTTMlCCHpagSQ35EKvULfLH0ofs1l/woPjYb9yUw5lfcpMCH9rakAM3ZceW+y ZApLSHW3XSGBaRRx4F5Nn4bU6nVFssC//hALqJGKMFao89RtZuxghOACr7QQ1GQJrKVE p4VDE0vCQG637nlf7x64oAS7IB/MI5QQIiffkwX4Xm/5f2GJ4DR2A55BQp+WfN/+dp3J ndbPOJXNppfJPxbaTINrZGjaj/HqW1BVqt5eVp11B8zoophhKSrL8lgc1K+Y5tdUlvF0 4oSg== X-Gm-Message-State: AOJu0YwXnzYu4P/SlzDAS0pHR5iBF5mz9GWLySUg/tAatW8CfqwtCgKL jVidxppOS+iodMBr3UZdL7xR0tROCOM= X-Google-Smtp-Source: AGHT+IG63IbGtKdW4spHP4xgnxjx6UFQ5KUsGp6pPkRyHhjJ0vgw7TaV9Rz63MbN9DRwvIBTPlHxkdTktmk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a05:6902:105:b0:da0:3da9:ce08 with SMTP id o5-20020a056902010500b00da03da9ce08mr35574ybh.10.1698155243657; Tue, 24 Oct 2023 06:47:23 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:16 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-20-surenb@google.com> Subject: [PATCH v2 19/39] mm: enable page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: eunzqkup1g1e3dk3qtagwokjpbm3m1ic X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B178918000A X-HE-Tag: 1698155244-278321 X-HE-Meta: U2FsdGVkX1+pWkms6J+YR4ntiyMPqMguWNWlFsbtgsm0L3GRixLOcLohBVyqnSNb1g9Q/Ts+SYjsQMA50IO+wqpetxsPKl9UvPpfNetCMpPje1U4g8DbXVX/ur66/rSICdK0IvuraQB7nceueLwgu9ylDSXKl/yPMyW2dz3qkIPLS4uE0d/xeUCQiQVhCdc7AU4rqLYq4zcjnlSeC0/qQpF+ScOfMiLtnF1/hbE/lXBQDYMKUJ/6vQpyTHTc8EeEcPgET/GHlCgQoeRV60yqHocNWkpXnWygUe+AtVkd8qnKwJOaVp2cxGJGy2Gq38yK/f+xbdSTvLSHFBRXMxWfGOEM2/XjkkRaoJ9DCV7GXBye6oMBtaWaHxe21kUwCTmMEjpss2C2F/OgjMA0W8okdrGCLYTqlXwxy61eQEfcgf0YakKmvmOE0jFgG9KUUtMUja5qRIaeJC75fUpVdbyxAb6GmroelSr9l70Bp9FwhHJVXiwsA4IkZqZCCBJpKPlEPpP+yqZAgGXEkMy/ZUxvOQE8GKTP9UYPidElVy/h/BP2odgfsD8iFKkD+50LEbw3H4E1wynQ1jGz28V8K/86o+9YzgisWU46TcVq74m1DVPI7ctdSDpOF0gQpWppHADSV9tgNWGJxUoNNseZMex5x8eILHtmySilrB69bifubx55sZCkekGkKFfG0pxqrt1e47BaXPa8NOsVwMzTMbCPVILy02kjld99M5ia7x6pnkGN5Mx26M4Cf+Sksd2qWsggFW11uVllNmlAZXaMEAJ4OlFjZYF/EWQgPG4z/CXUMO7jgGS73KMCjvc/MEaUY8sHcVJK3E4aU/DFYfkW6E1D1W8OGs4CjKyL2mQDzDm9WPEJjasAgjquYXrL7nQef2wZ+fTy0SsvErn0CjYrrI2r7RMCtjW63oubu4mzWL/80apdjXWvNbEphfcjxcipGbf7d2Ar29IygQ8cj7cdE41 fRX0aP03 dr+gsHP7tSNxSYV9qdGJq1IwVrPBrEclVBf7LKotzAXNM9ZlYYUALIwP+NE4r1Ljo2ZGPcOG5HEo67hXlMTqTylgM8r6Sw5eGps8WqDEKOpC2RjdnQwCMgfmUp4YwhORiJmfvd+k2kS9N9+IXHCqi0tPxWNwYf/HuJzTUnRLKaZbSTavTkrpx4GsAwL1ixLUcozPUaIXJVVKULg7FBRD6qRmYmj6nrsTvaZ7PGW6shHkJBWhixmmF5X4vjZdTEAqYdjK5SMgDq2bWItJANZVRmJa0NpMvMiXVt0Zx4lDFbNh8wrzS+j+PdCOLZpyhmclNQ2roXqTf/6moq1+bU+rFUT8XTMB3+ggmagg+kqXM3llZvIZ9jLVxbJmdWBdVfGWh2fK25fYdw0hMBceA4TlWMts3jfjDH6l1RnjVoQ1g2TL2atWYSHYTcUnZzR6InFc9/heQFfAnQVgigWstGFol8KEoFiLg9Vakg0aRGUvpYcwXEnlciav3InTN4Lbb94fdYb1xSrbFsTb202dXy/FxnLM/w4vFUgsoJM8fJhkgDUJX8+mWIu17gvpatcVW8NiNRxqgn2e1qGyK7RUz40Uog5eN7MV5sAzOM5w4//H3NVygwlNevV9KtChVh2e3x+MvrHTna9PxiDg1l9lm5pNqj/CFbUCdJz0PLFkpIBqcgvB8GGs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine page allocators to record allocation tags upon their invocation. Instrument post_alloc_hook and free_pages_prepare to modify current allocation tag. Co-developed-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Signed-off-by: Kent Overstreet --- include/linux/alloc_tag.h | 10 ++++ include/linux/gfp.h | 111 +++++++++++++++++++++++--------------- include/linux/pagemap.h | 9 ++-- mm/compaction.c | 7 ++- mm/filemap.c | 6 +-- mm/mempolicy.c | 42 +++++++-------- mm/page_alloc.c | 60 ++++++++++----------- 7 files changed, 144 insertions(+), 101 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index cf55a149fa84..6fa8a94d8bc1 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -130,4 +130,14 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, #endif +#define alloc_hooks(_do_alloc) \ +({ \ + typeof(_do_alloc) _res; \ + DEFINE_ALLOC_TAG(_alloc_tag, _old); \ + \ + _res = _do_alloc; \ + alloc_tag_restore(&_alloc_tag, _old); \ + _res; \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 665f06675c83..20686fd1f417 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -6,6 +6,8 @@ #include #include +#include +#include struct vm_area_struct; @@ -174,42 +176,43 @@ static inline void arch_free_page(struct page *page, int order) { } static inline void arch_alloc_page(struct page *page, int order) { } #endif -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +#define __alloc_pages(...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) + +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); +#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__)) -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array); +#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__)) -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array); +#define alloc_pages_bulk_array_mempolicy(...) alloc_hooks(alloc_pages_bulk_array_mempolicy_noprof(__VA_ARGS__)) /* Bulk allocate order-0 pages */ -static inline unsigned long -alloc_pages_bulk_list(gfp_t gfp, unsigned long nr_pages, struct list_head *list) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, list, NULL); -} +#define alloc_pages_bulk_list(_gfp, _nr_pages, _list) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, _list, NULL) -static inline unsigned long -alloc_pages_bulk_array(gfp_t gfp, unsigned long nr_pages, struct page **page_array) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, NULL, page_array); -} +#define alloc_pages_bulk_array(_gfp, _nr_pages, _page_array) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, NULL, _page_array) static inline unsigned long -alloc_pages_bulk_array_node(gfp_t gfp, int nid, unsigned long nr_pages, struct page **page_array) +alloc_pages_bulk_array_node_noprof(gfp_t gfp, int nid, unsigned long nr_pages, struct page **page_array) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_bulk(gfp, nid, NULL, nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, NULL, nr_pages, NULL, page_array); } +#define alloc_pages_bulk_array_node(...) alloc_hooks(alloc_pages_bulk_array_node_noprof(__VA_ARGS__)) + static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) { gfp_t warn_gfp = gfp_mask & (__GFP_THISNODE|__GFP_NOWARN); @@ -229,21 +232,23 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) * online. For more general interface, see alloc_pages_node(). */ static inline struct page * -__alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order) +__alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp_mask); - return __alloc_pages(gfp_mask, order, nid, NULL); + return __alloc_pages_noprof(gfp_mask, order, nid, NULL); } +#define __alloc_pages_node(...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) + static inline struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp); - return __folio_alloc(gfp, order, nid, NULL); + return __folio_alloc_noprof(gfp, order, nid, NULL); } /* @@ -251,53 +256,69 @@ struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) * prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, - unsigned int order) +static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, + unsigned int order) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_node(nid, gfp_mask, order); + return __alloc_pages_node_noprof(nid, gfp_mask, order); } +#define alloc_pages_node(...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__)) + #ifdef CONFIG_NUMA -struct page *alloc_pages(gfp_t gfp, unsigned int order); -struct folio *folio_alloc(gfp_t gfp, unsigned order); -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order); +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order); +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage); #else -static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) +static inline struct page *alloc_pages_noprof(gfp_t gfp_mask, unsigned int order) { - return alloc_pages_node(numa_node_id(), gfp_mask, order); + return alloc_pages_node_noprof(numa_node_id(), gfp_mask, order); } -static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order) +static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { return __folio_alloc_node(gfp, order, numa_node_id()); } -#define vma_alloc_folio(gfp, order, vma, addr, hugepage) \ - folio_alloc(gfp, order) +#define vma_alloc_folio_noprof(gfp, order, vma, addr, hugepage) \ + folio_alloc_noprof(gfp, order) #endif + +#define alloc_pages(...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__)) +#define folio_alloc(...) alloc_hooks(folio_alloc_noprof(__VA_ARGS__)) +#define vma_alloc_folio(...) alloc_hooks(vma_alloc_folio_noprof(__VA_ARGS__)) + #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0) -static inline struct page *alloc_page_vma(gfp_t gfp, + +static inline struct page *alloc_page_vma_noprof(gfp_t gfp, struct vm_area_struct *vma, unsigned long addr) { - struct folio *folio = vma_alloc_folio(gfp, 0, vma, addr, false); + struct folio *folio = vma_alloc_folio_noprof(gfp, 0, vma, addr, false); return &folio->page; } +#define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) + +extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); +#define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) -extern unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order); -extern unsigned long get_zeroed_page(gfp_t gfp_mask); +extern unsigned long get_zeroed_page_noprof(gfp_t gfp_mask); +#define get_zeroed_page(...) alloc_hooks(get_zeroed_page_noprof(__VA_ARGS__)) + +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) __alloc_size(1); +#define alloc_pages_exact(...) alloc_hooks(alloc_pages_exact_noprof(__VA_ARGS__)) -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1); void free_pages_exact(void *virt, size_t size); -__meminit void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); -#define __get_free_page(gfp_mask) \ - __get_free_pages((gfp_mask), 0) +__meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); +#define alloc_pages_exact_nid(...) alloc_hooks(alloc_pages_exact_nid_noprof(__VA_ARGS__)) + +#define __get_free_page(gfp_mask) \ + __get_free_pages((gfp_mask), 0) -#define __get_dma_pages(gfp_mask, order) \ - __get_free_pages((gfp_mask) | GFP_DMA, (order)) +#define __get_dma_pages(gfp_mask, order) \ + __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); @@ -347,10 +368,14 @@ extern gfp_t vma_thp_gfp_mask(struct vm_area_struct *vma); #ifdef CONFIG_CONTIG_ALLOC /* The below functions must be run on a range from a single zone. */ -extern int alloc_contig_range(unsigned long start, unsigned long end, +extern int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask); -extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask); +#define alloc_contig_range(...) alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__)) + +extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask); +#define alloc_contig_pages(...) alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__)) + #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 351c3b7f93a1..cb387321f929 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -508,14 +508,17 @@ static inline void *detach_page_private(struct page *page) #endif #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order); +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order); #else -static inline struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +static inline struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } #endif +#define filemap_alloc_folio(...) \ + alloc_hooks(filemap_alloc_folio_noprof(__VA_ARGS__)) + static inline struct page *__page_cache_alloc(gfp_t gfp) { return &filemap_alloc_folio(gfp, 0)->page; diff --git a/mm/compaction.c b/mm/compaction.c index 38c8d216c6a3..74a7ec71660d 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1760,7 +1760,7 @@ static void isolate_freepages(struct compact_control *cc) * This is a migrate-callback that "allocates" freepages by taking pages * from the isolated freelists in the block we are migrating to. */ -static struct folio *compaction_alloc(struct folio *src, unsigned long data) +static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long data) { struct compact_control *cc = (struct compact_control *)data; struct folio *dst; @@ -1779,6 +1779,11 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) return dst; } +static struct folio *compaction_alloc(struct folio *src, unsigned long data) +{ + return alloc_hooks(compaction_alloc_noprof(src, data)); +} + /* * This is a migrate-callback that "frees" freepages back to the isolated * freelist. All pages on the freelist are from the same zone, so there is no diff --git a/mm/filemap.c b/mm/filemap.c index f0a15ce1bd1b..fc0e4b630b51 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -958,7 +958,7 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, EXPORT_SYMBOL_GPL(filemap_add_folio); #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { int n; struct folio *folio; @@ -973,9 +973,9 @@ struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) return folio; } - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } -EXPORT_SYMBOL(filemap_alloc_folio); +EXPORT_SYMBOL(filemap_alloc_folio_noprof); #endif /* diff --git a/mm/mempolicy.c b/mm/mempolicy.c index f1b00d6ac7ee..0c3d65fbf9e8 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2127,7 +2127,7 @@ static struct page *alloc_page_interleave(gfp_t gfp, unsigned order, { struct page *page; - page = __alloc_pages(gfp, order, nid, NULL); + page = __alloc_pages_noprof(gfp, order, nid, NULL); /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */ if (!static_branch_likely(&vm_numa_stat_key)) return page; @@ -2153,15 +2153,15 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages(preferred_gfp, order, nid, &pol->nodes); + page = __alloc_pages_noprof(preferred_gfp, order, nid, &pol->nodes); if (!page) - page = __alloc_pages(gfp, order, nid, NULL); + page = __alloc_pages_noprof(gfp, order, nid, NULL); return page; } /** - * vma_alloc_folio - Allocate a folio for a VMA. + * vma_alloc_folio_noprof - Allocate a folio for a VMA. * @gfp: GFP flags. * @order: Order of the folio. * @vma: Pointer to VMA or NULL if not available. @@ -2175,7 +2175,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, * * Return: The folio on success or NULL if allocation fails. */ -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage) { struct mempolicy *pol; @@ -2246,7 +2246,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, * memory with both reclaim and compact as well. */ if (!folio && (gfp & __GFP_DIRECT_RECLAIM)) - folio = __folio_alloc(gfp, order, hpage_node, + folio = __folio_alloc_noprof(gfp, order, hpage_node, nmask); goto out; @@ -2255,15 +2255,15 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, nmask = policy_nodemask(gfp, pol); preferred_nid = policy_node(gfp, pol, node); - folio = __folio_alloc(gfp, order, preferred_nid, nmask); + folio = __folio_alloc_noprof(gfp, order, preferred_nid, nmask); mpol_cond_put(pol); out: return folio; } -EXPORT_SYMBOL(vma_alloc_folio); +EXPORT_SYMBOL(vma_alloc_folio_noprof); /** - * alloc_pages - Allocate pages. + * alloc_pages_noprof - Allocate pages. * @gfp: GFP flags. * @order: Power of two of number of pages to allocate. * @@ -2276,7 +2276,7 @@ EXPORT_SYMBOL(vma_alloc_folio); * flags are used. * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages(gfp_t gfp, unsigned order) +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order) { struct mempolicy *pol = &default_policy; struct page *page; @@ -2294,24 +2294,24 @@ struct page *alloc_pages(gfp_t gfp, unsigned order) page = alloc_pages_preferred_many(gfp, order, policy_node(gfp, pol, numa_node_id()), pol); else - page = __alloc_pages(gfp, order, + page = __alloc_pages_noprof(gfp, order, policy_node(gfp, pol, numa_node_id()), policy_nodemask(gfp, pol)); return page; } -EXPORT_SYMBOL(alloc_pages); +EXPORT_SYMBOL(alloc_pages_noprof); -struct folio *folio_alloc(gfp_t gfp, unsigned order) +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { - struct page *page = alloc_pages(gfp | __GFP_COMP, order); + struct page *page = alloc_pages_noprof(gfp | __GFP_COMP, order); struct folio *folio = (struct folio *)page; if (folio && order > 1) folio_prep_large_rmappable(folio); return folio; } -EXPORT_SYMBOL(folio_alloc); +EXPORT_SYMBOL(folio_alloc_noprof); static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, struct mempolicy *pol, unsigned long nr_pages, @@ -2330,13 +2330,13 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, for (i = 0; i < nodes; i++) { if (delta) { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node + 1, NULL, page_array); delta--; } else { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node, NULL, page_array); } @@ -2358,11 +2358,11 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - nr_allocated = __alloc_pages_bulk(preferred_gfp, nid, &pol->nodes, + nr_allocated = alloc_pages_bulk_noprof(preferred_gfp, nid, &pol->nodes, nr_pages, NULL, page_array); if (nr_allocated < nr_pages) - nr_allocated += __alloc_pages_bulk(gfp, numa_node_id(), NULL, + nr_allocated += alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, nr_pages - nr_allocated, NULL, page_array + nr_allocated); return nr_allocated; @@ -2374,7 +2374,7 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, * It can accelerate memory allocation especially interleaving * allocate memory. */ -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array) { struct mempolicy *pol = &default_policy; @@ -2390,7 +2390,7 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, return alloc_pages_bulk_array_preferred_many(gfp, numa_node_id(), pol, nr_pages, page_array); - return __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()), + return alloc_pages_bulk_noprof(gfp, policy_node(gfp, pol, numa_node_id()), policy_nodemask(gfp, pol), nr_pages, NULL, page_array); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d490d0f73e72..63dc2f8c7901 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4239,7 +4239,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, * * Returns the number of pages on the list or array. */ -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array) @@ -4375,7 +4375,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, pcp_trylock_finish(UP_flags); failed: - page = __alloc_pages(gfp, 0, preferred_nid, nodemask); + page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask); if (page) { if (page_list) list_add(&page->lru, page_list); @@ -4386,13 +4386,13 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, goto out; } -EXPORT_SYMBOL_GPL(__alloc_pages_bulk); +EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof); /* * This is the 'heart' of the zoned buddy allocator. */ -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, - nodemask_t *nodemask) +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, + int preferred_nid, nodemask_t *nodemask) { struct page *page; unsigned int alloc_flags = ALLOC_WMARK_LOW; @@ -4454,12 +4454,12 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, return page; } -EXPORT_SYMBOL(__alloc_pages); +EXPORT_SYMBOL(__alloc_pages_noprof); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { - struct page *page = __alloc_pages(gfp | __GFP_COMP, order, + struct page *page = __alloc_pages_noprof(gfp | __GFP_COMP, order, preferred_nid, nodemask); struct folio *folio = (struct folio *)page; @@ -4467,29 +4467,29 @@ struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, folio_prep_large_rmappable(folio); return folio; } -EXPORT_SYMBOL(__folio_alloc); +EXPORT_SYMBOL(__folio_alloc_noprof); /* * Common helper functions. Never use with __GFP_HIGHMEM because the returned * address cannot represent highmem pages. Use alloc_pages and then kmap if * you need to access high mem. */ -unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order) +unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order) { struct page *page; - page = alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order); + page = alloc_pages_noprof(gfp_mask & ~__GFP_HIGHMEM, order); if (!page) return 0; return (unsigned long) page_address(page); } -EXPORT_SYMBOL(__get_free_pages); +EXPORT_SYMBOL(get_free_pages_noprof); -unsigned long get_zeroed_page(gfp_t gfp_mask) +unsigned long get_zeroed_page_noprof(gfp_t gfp_mask) { - return __get_free_page(gfp_mask | __GFP_ZERO); + return get_free_pages_noprof(gfp_mask | __GFP_ZERO, 0); } -EXPORT_SYMBOL(get_zeroed_page); +EXPORT_SYMBOL(get_zeroed_page_noprof); /** * __free_pages - Free pages allocated with alloc_pages(). @@ -4681,7 +4681,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, } /** - * alloc_pages_exact - allocate an exact number physically-contiguous pages. + * alloc_pages_exact_noprof - allocate an exact number physically-contiguous pages. * @size: the number of bytes to allocate * @gfp_mask: GFP flags for the allocation, must not contain __GFP_COMP * @@ -4695,7 +4695,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, * * Return: pointer to the allocated area or %NULL in case of error. */ -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); unsigned long addr; @@ -4703,13 +4703,13 @@ void *alloc_pages_exact(size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - addr = __get_free_pages(gfp_mask, order); + addr = get_free_pages_noprof(gfp_mask, order); return make_alloc_exact(addr, order, size); } -EXPORT_SYMBOL(alloc_pages_exact); +EXPORT_SYMBOL(alloc_pages_exact_noprof); /** - * alloc_pages_exact_nid - allocate an exact number of physically-contiguous + * alloc_pages_exact_nid_noprof - allocate an exact number of physically-contiguous * pages on a node. * @nid: the preferred node ID where memory should be allocated * @size: the number of bytes to allocate @@ -4720,7 +4720,7 @@ EXPORT_SYMBOL(alloc_pages_exact); * * Return: pointer to the allocated area or %NULL in case of error. */ -void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) +void * __meminit alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); struct page *p; @@ -4728,7 +4728,7 @@ void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - p = alloc_pages_node(nid, gfp_mask, order); + p = alloc_pages_node_noprof(nid, gfp_mask, order); if (!p) return NULL; return make_alloc_exact((unsigned long)page_address(p), order, size); @@ -6090,7 +6090,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, } /** - * alloc_contig_range() -- tries to allocate given range of pages + * alloc_contig_range_noprof() -- tries to allocate given range of pages * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate * @migratetype: migratetype of the underlying pageblocks (either @@ -6110,7 +6110,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * pages which PFN is in [start, end) are allocated for the caller and * need to be freed with free_contig_range(). */ -int alloc_contig_range(unsigned long start, unsigned long end, +int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; @@ -6234,15 +6234,15 @@ int alloc_contig_range(unsigned long start, unsigned long end, undo_isolate_page_range(start, end, migratetype); return ret; } -EXPORT_SYMBOL(alloc_contig_range); +EXPORT_SYMBOL(alloc_contig_range_noprof); static int __alloc_contig_pages(unsigned long start_pfn, unsigned long nr_pages, gfp_t gfp_mask) { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return alloc_contig_range_noprof(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, @@ -6277,7 +6277,7 @@ static bool zone_spans_last_pfn(const struct zone *zone, } /** - * alloc_contig_pages() -- tries to find and allocate contiguous range of pages + * alloc_contig_pages_noprof() -- tries to find and allocate contiguous range of pages * @nr_pages: Number of contiguous pages to allocate * @gfp_mask: GFP mask to limit search and used during compaction * @nid: Target node @@ -6297,8 +6297,8 @@ static bool zone_spans_last_pfn(const struct zone *zone, * * Return: pointer to contiguous pages on success, or NULL if not successful. */ -struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask) +struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask) { unsigned long ret, pfn, flags; struct zonelist *zonelist; From patchwork Tue Oct 24 13:46:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB1EDC25B48 for ; Tue, 24 Oct 2023 13:47:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3CB626B027C; Tue, 24 Oct 2023 09:47:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 353FF6B027D; Tue, 24 Oct 2023 09:47:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 133BE6B027E; Tue, 24 Oct 2023 09:47:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E88746B027C for ; Tue, 24 Oct 2023 09:47:28 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C9B6F405EF for ; Tue, 24 Oct 2023 13:47:28 +0000 (UTC) X-FDA: 81380482176.27.EDE2A65 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf08.hostedemail.com (Postfix) with ESMTP id 09615160029 for ; Tue, 24 Oct 2023 13:47:26 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=NdAo3mJH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 37co3ZQYKCJIEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=37co3ZQYKCJIEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155247; a=rsa-sha256; cv=none; b=UNMefj5uM7ZSH4n7ZLj9K9vjHRo//nKDZRPY4YzZ597vbeKo5U5BUaZamAjUMZGa6QSvMO z9PLPUgfoxNnuOieXxRp3VgBZjmfKq7KZZNdpRzVudF0dcvgjIeK5vt96WL9+juBBdoNS/ HXUnkjAzvSAY2Yx+AQccsvitgksLaFw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=NdAo3mJH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 37co3ZQYKCJIEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=37co3ZQYKCJIEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155247; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zhzOhwHQcxJuZ74GpFe9sYgSfMOEl4CWCuHOL/zyxyA=; b=0V+2+G9XMrlKG+eCR7qAq8s7zCd6nkQPxyBnIdbeowj4l3iWcwxpAYgL7LRs9bE3IJccDy KryvQ7KT/RWWfm7ceGdRRQG61hPCDuRcUQ9ceeRNXmapiT8kc9FYCQSVbqf4051IgaxqA6 jC42jWKHLIAVxRc13BNR3tCAtrQ7DhM= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d815354ea7fso4177407276.1 for ; Tue, 24 Oct 2023 06:47:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155246; x=1698760046; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zhzOhwHQcxJuZ74GpFe9sYgSfMOEl4CWCuHOL/zyxyA=; b=NdAo3mJH2Fn3LeIN+MKj18UmE8F4TYc2rJgcwV1KxROLoGZql6rqFrujG9AvL2pJn7 WzO9m1cdOxrvqL+mP2ShquUXjWxP72eFI0/E7wtTcJoJ1Si1eOP2iHQEfeeYSs7nlpCl raRNlC4XSPMsJzsV0W32WZ88MbguP6B0A9PBErkLoVQ3e2SuvdPrrbynd+HhVwXAOCpF BD318F7xwdSpj8Posd5h3hhnYMRnYTCcpRSAchZxqBdLObUgORrctw+8R+b7Og4Z9Onv C7k1KwsM05OyXpjag3h+gjzhKDnZ0XqdyIPo5NLjLm58FXqMfSqzC8pqcdrkXs9wEblc hy4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155246; x=1698760046; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zhzOhwHQcxJuZ74GpFe9sYgSfMOEl4CWCuHOL/zyxyA=; b=t4E2Jm/lb9YqkedDfu6JA38O9a55sgC0Elvf+kH7gd7VQpZezyAWW7xDzyv6BU2wDR zJfbpdzFTycVSFNEtTD/3CpGikysES0WJB5xBkmDGvhhdwG1tqQ7CtxFgLs9d8t0FgPm GQtKcJMT1/BdsD6mOhuzgGJbP6d42PkPzcD8oQ/wD0cDYO6oAeQLZAllX9QOHMHyhap3 aZN/2ux8G8KRIHhPq1YMBeyiL7IHscAL3BgFZa6vxM0vwU79yTovmppydqYdsmfg0Isz uRsOA4oNxTI6crMEAly/g7N9TpU6D0/LuQy4oHGmFDF2QeoaBl6nlOghHuYqc9agGQx7 o9vg== X-Gm-Message-State: AOJu0Yx1dAeo/SjjLGJRhUppCCgjv/d91aVRImblrtecPJb7lx1PB3dv Co5NTNkz5O/B/SdQYlj9YeilIE84iKk= X-Google-Smtp-Source: AGHT+IGqbReHwIMnajbCAyyRpg4FWyctkHXB3WgJzXU7P7J0pPMsKbVadURacVWEffWX34fNon1x9kvs7DE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:42c9:0:b0:d9a:bce6:acf3 with SMTP id p192-20020a2542c9000000b00d9abce6acf3mr225566yba.0.1698155245951; Tue, 24 Oct 2023 06:47:25 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:17 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-21-surenb@google.com> Subject: [PATCH v2 20/39] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 09615160029 X-Stat-Signature: um37zt5qdrisnqoat61kqfwz9p1ynasi X-HE-Tag: 1698155246-983956 X-HE-Meta: U2FsdGVkX18cOYX4vkNQ0REegUt06xEElhBKqifNrZSMlrQj5d6v/OiJhUBry2vxz6Bh80d08tBVLUpl1Pctzk4o7cSy2EMuhaTFcSLvkQjbd1gOb0axQsDSFvoAr8e/cvmL0wo079IzzYNuDIw7TB6Ph9t/97iri39aGSE5lW5eSQa5zHEQ8vP6qtBBIt1dVRb+ce/y4S7vuomlF2U8mfZFKuym1MIbTd5uJLo4VcchczM5ISNb1P50N1axJWQm7+drrznsm8+GnRLfLB289HT0FnYqDQ80kDP+/6QvEe4zT+3s1phviX++6L6mYu4shG4k7vmfv+O8556MR5bZzGcYVoE7IKdINfd2l0GvH76nSRXGipmqCgpdfzVcVc+5M1wP4vDfWluLPm+7P+8uzTcRwt++Gy+D+97Nan+Ov4jwhlmOvSWNI/pEqfSMCmqtqkHLcHTN1sU1LjIAJFslr7CuygSDhoDjIeROEeWNhHN+oMkAJTMa+3UBk7Kj9erHvErlt2zNooMNG2IJ1YgdQ0f+XlVx8+/6SpYc9BTFTYAdTSMv79ZVkx9b+6E30Nv9ZFVkyduJ6x8id+/ed8Y+JR4suHUvLttVjqJ2KCYCOMW9QT7eSwuxj6J2oJ07ZqGFIUkC7A/N2dGfYLvjO/4ayL3R2PDSJU8y7cULGkc542fGrqeE+y0tf3EpaMHS3fQzQb2Wx0McTF1+QwW8bcWQR+kJW26Jl+RAIA3w2DwN5k2lQk5JFrgWLcKB0aV5F+DSQqpIOpCnR8e8oSU9P+pFZSWatGxwNiur5MUiRLsw9i+eTklyJLDVrWGQwh78uJNe5rZp/vZAdBUeipq5VVoXVHN6Dia3tLew8iOzAEUh8gU4yxRz09vBPw7QmcHuSn8NhwCA5OfoPD00CuHvNl+fdTMM+rTxVbXxK2ap1No4yMsCIyneaDqgWvJT8OBJfg2/bH7Sg31ttmXl1T7kdkA eYpIaj60 HUEH9NDITkPxu/1K315qnCDKJqrO+k/6OYWYPxtFfQNQTYfiw0qYjca5Ji/YlTzwQZny8CzD6KYYBkPqPaIegTXXOK1+EIsx4NTm6s0XYqDDr9xsZNwNWoP/L+cR8ZlQTY+8tP4opXnQAlpolU0/zQPFhPBb+F9Mzg5fv0x7V9YwnZkOU49Yjo5JNM+2sDx0hdJye5/8hg5kGjWzYysQFy1Fgd8abzsxZok3fAh3MCUkJZMT0ObvfFrge4O5QusQ7PAkx0P19OB23crEz7zITuLv8563YpzvvEVfDy6KsOdorOJ4SS0qWtQM61tGdSZl5QQawMz4bVvT1DKebnpIWa/kE228Cv4WOYJRVlA6qPldHRFX168i2v7PLu+lAf5F5xbSohbB9LlwUDrGVi0FXKOYycW2YgFaYnHHg0OwcZIkP3nKixph5C7YjvpBqEWOWVoTKcZL9qT1Xuj9YeYYc1fhx/vidBFvIoVtUULSIwtyQa4iFK8AGAbjNiq+wy5H0p7w+HZiTuf3//qfR8PNsr95Clh+Z5yMXTyPb8HcLVv/Ko2pqBI1j5p+p1saXx8wfWOT6tORh5g5SqJ9vnL+clclqYh9FA+PlMGnQDCHPa3F6QIE2ZKe3pBzD25HGxUPXVigH8+gGvOj0LlLSgpVgAhigQU/kdcMtci9wvFsb6r9dqdc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page. Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 3 files changed, 34 insertions(+) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index a060c26eb449..0174aff5e871 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -62,11 +62,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* New reference with 0 bytes accounted */ + alloc_tag_add(codetag_ref_from_page_ext(page_ext), tag, 0); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 064fbd90822b..392b6907d875 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include @@ -2545,6 +2546,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, nr); + pgalloc_tag_split(head, nr); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 63dc2f8c7901..c4f0cd127e14 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2540,6 +2540,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); } EXPORT_SYMBOL_GPL(split_page); @@ -4669,6 +4670,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); while (page < --last) set_page_refcounted(last); From patchwork Tue Oct 24 13:46:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 808D1C25B48 for ; Tue, 24 Oct 2023 13:47:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBC696B027E; Tue, 24 Oct 2023 09:47:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF6CA6B027F; Tue, 24 Oct 2023 09:47:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 948786B0280; Tue, 24 Oct 2023 09:47:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7612F6B027E for ; Tue, 24 Oct 2023 09:47:31 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3A8EB80603 for ; Tue, 24 Oct 2023 13:47:31 +0000 (UTC) X-FDA: 81380482302.02.2ADE1D7 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 5B8A614002C for ; Tue, 24 Oct 2023 13:47:29 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=F0dI+BjF; spf=pass (imf23.hostedemail.com: domain of 38Mo3ZQYKCJUHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=38Mo3ZQYKCJUHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155249; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zC2/AjWbzXkwFGJu5UOviuzr4L4X1GYNGPTyCuvFw8o=; b=WiidZPwaJZ8jvXDqn41gi4b0qfbH2Qr+MaCiSJ6b1sxNhF41zKbrqzt33AtWJL9/3XVDu6 YfYOkKtsPQNkMyuJnxeazNLNZoTqNaA8E/BfnxpFbgcLbAyaUzPhOOPfEiawcGyAdgbW0/ DTTfufGNea2cyKSdK+AAIz4V1AsHrWE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155249; a=rsa-sha256; cv=none; b=iIbvgfo5RFElSqbYrldghP0NQU/uMFU6sXkow/t1F8z6t9+qpLwXwy1Vcpl5JtDd8kBmIX kUDmiQV2POl7LlPWHimp1lonNmSBPbzJ8qeDTX7/I2Q4wXXQaJuvxVIHHRN3SGt5UjGJtG g7+KpjB3LK5J2KLY8YFkH7BVmhZ6DcU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=F0dI+BjF; spf=pass (imf23.hostedemail.com: domain of 38Mo3ZQYKCJUHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=38Mo3ZQYKCJUHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9a3a38b96cso5080913276.0 for ; Tue, 24 Oct 2023 06:47:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155248; x=1698760048; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zC2/AjWbzXkwFGJu5UOviuzr4L4X1GYNGPTyCuvFw8o=; b=F0dI+BjFTExqjmOrUSduaMR8Mzxv5IBrWWQuPzY3ObvdKtbSeBQDpbV84Q7L8j5wnT 9582ZRJjaHcAzv1fu2y1P+l7mOBaBZLgrMkU7lPJ1usY2yMyWxz9dIM2B/XPmNpcsBAa QHXYCdMwD0bv5cF/wvf97U0lG2qSzKrFMvNKY15S6e5IjUt4r813rXsHyktLuRQniBJm cYwo1tGzVNCApfzqKEaZe2BPRCyWVvwamOzD0zCJDWrX6hfEdwhYEDLzsmdc/RolQVJh wTubUeR+JamE5qTF+QfulfC8Efg0LGwchUJ2UyB3eGJI61KdRtfS3rnwm6ksj7fSXraV JKSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155248; x=1698760048; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zC2/AjWbzXkwFGJu5UOviuzr4L4X1GYNGPTyCuvFw8o=; b=blqX+9ww9zHaHCAVddlMmfaMlw/xhzB+qnl29NeiB436DjMlSYJMmjHEKQWn2JSOBp 8Fvdk6pLf6OzttzJ1RsLUPeYdyEz3pYdmvO5XdfXi68uSETvYEjEI7lKr62GxCgEfoYp I3P/i8hcGtls5su/65L8hxobohrPlYvxs82EwMuPbP8APGjc6DNUW0ozfrt4gkQKdUgf GIBKf55IZZxJ+El66EuRdwqsjGe3ZKo6KliXGhDrqzidIcfvbcRPHb7hlbMPiiR3jvx6 blHsCQf/2HxP/vS6NGZA+tYnn+J7lBuZF+OASLc3hTi5KbcFTxCd2hmm53aWcbmorDLe I/YA== X-Gm-Message-State: AOJu0YyQuuLIxIp0+O/32m2B+Ei1PZbcNrTipyrJK0QryUjTjfYLZW+s sy9op5XDRiX20R4Bh7IcdrShw9ckfbc= X-Google-Smtp-Source: AGHT+IFtIla7jN3qQCOM5jBjQINYCYH8n10kTFpUDwIK13I4RwREYyTVERaiA7ZHusZIP5/hSujmrJfw8JM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:40c7:0:b0:da0:289e:c056 with SMTP id n190-20020a2540c7000000b00da0289ec056mr61156yba.8.1698155248287; Tue, 24 Oct 2023 06:47:28 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:18 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-22-surenb@google.com> Subject: [PATCH v2 21/39] mm/page_ext: enable early_page_ext when CONFIG_MEM_ALLOC_PROFILING_DEBUG=y From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: i4pey3tqrqkbyougam4xpdke7qez1eui X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 5B8A614002C X-Rspam-User: X-HE-Tag: 1698155249-810480 X-HE-Meta: U2FsdGVkX1/O8yNJWQCKdgPk4duX/ggzy6Y+7TH010IKGe9z/LK2hqBlU1SIp26I8aCPW8eFEtmYvtYyaV3ZKdvyWUAcywJRVV7igYnaTXETBwb2zgnXzVSMrpMPqBzrueIz5u8qo776XBkzD8F0zX0MyuHv4eviErV7oE+BnhW49YBbecBpk0vEpjxwIKhVVbm5SxrcVLc3bcZbulJJw2RYl5fQ6ZBGfFORqsoE3ZNGF+ghN371HLf7asBbzbVuk8BJTBLGvBAx+DzaUlqmLY/ot4raqhcG8zAXkDg9qYKcOju+40BLeCmur8I67rZgwWeH8bVfB2IPSNBqn0e+JASTAufJVayeGmNkRmskZcNQeFGMR6iZqOBQVb2Bj5V96/SFDvvkh4ijc+0wt1l/innEpDacBT3Vfvi/DdIO85f+JqyuqXL98fqBUd1DNotGlqw7ZdL78QoZ0EfjTz0Aw0DpjRMwTi5op3RawmiedEIRmLW3wUNJaydxcERg9FBkFtWC88oVhHceQxUOMJGWnH+dfo4E2Gdh+nHW6++3EPLWCc17owGFxGRRPK0GeXyiT4f91WumeIGRQwuqmDd43muIUJDmHRuJzbXAkVV5+5GV0XQ5Fi7CUaxmxYDDePUBUDWTjeHESSWpU2J9MK90gr5mbOfYf1NUe8WB4iW+/hdqZ8hbar0HTn8VvMud9KH5+EgDrlDCqVZBFh35vjxbLiNmPEbMByoXHHC2F/0GZM3qoGCugRUDu0fwvz9UvRVoWCHwUOlIyT9NwRYKpkUCbh3MU7c0G5eM73ws3Ikumotxz8CJJfNi3qRFhziU7gMzvO7jf3zCFZDRhCl3O0AsztR+4btk82+SdV96Wz6NUNIiW+6uBGtbZjWhS2Enb6cUJd0EIdbAZqXtvahPBGhJ6++uf3HL4I+rJHN2KbBFUrQptALnM/WzdxAgc+xOSV/IEVWSQeuKaKPA8NRZUoC xenkyiV6 RVkMD+KflP7ocRbFz+kr1LcVBeV7yq11G/owxlId+HyTKez0kOqJFpdZ9IMl6Hi+wvdS5IA6jfRV+ab/95mrv99hNkuAyw7WRz0kC9yAALaO4tmfvqDgmlxlTd73IP4w3WdWGVZKzOCVjilB98qo/PZ13xaOEUVIAIEgL4nIIraxVJ2TVNhPwhGl6hH2xSb8U49bFA2uSRcPXvjq9e1pAaRcrRFTLcCGpmCZcY23SNuVilVf6q3MwTAn42pSv2AKzNzCfeF8QmEO3YiL0tJ4LPrgHevB4BQOBDBmd7MqMdg0PaN08E3eZ5uuLSkUpBxd4H3dDGYX1Wm+AsQaMxCZqZgBTlb36yc2sHZa4GP/nozsviAiIh27iYqGAuJnnXFwxO39i7kYyI9IQRGchuWKa1vIOEVWecLDhH1RlaRtevPp/BMwTygcXrlW+X8wvXBREVtgkq4LkdiSerN9TRC8gJhq1xZV1Q160t4r7YhPRGwATXn5122eCHFddKQWvkDR/L4cDHdwsefaAkHE4f0EEyH2NMH7ZFpmaPP2QG/OKrrtNB3k5qgNLnihlPcf0hAgHhY1E8wgKq3dEWuyNoQqyBcayTTHwwKHFKqwEGUxuHl0x4QhhmIoy7C16oz64LjboUOqH/bUVoeoiJvlEVqdKtMLC9MEQ3IT0ZUFn48KkybLGluHPsUf0PkrAdN0WYVJgBFbUJoJp0jkK81um+Ghe3Qxx4gN87zeIATXX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For all page allocations to be tagged, page_ext has to be initialized before the first page allocation. Early tasks allocate their stacks using page allocator before alloc_node_page_ext() initializes page_ext area, unless early_page_ext is enabled. Therefore these allocations will generate a warning when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Enable early_page_ext whenever CONFIG_MEM_ALLOC_PROFILING_DEBUG=y to ensure page_ext initialization prior to any page allocation. This will have all the negative effects associated with early_page_ext, such as possible longer boot time, therefore we enable it only when debugging with CONFIG_MEM_ALLOC_PROFILING_DEBUG enabled and not universally for CONFIG_MEM_ALLOC_PROFILING. Signed-off-by: Suren Baghdasaryan --- mm/page_ext.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/page_ext.c b/mm/page_ext.c index 3c58fe8a24df..e7d8f1a5589e 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -95,7 +95,16 @@ unsigned long page_ext_size; static unsigned long total_usage; +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +/* + * To ensure correct allocation tagging for pages, page_ext should be available + * before the first page allocation. Otherwise early task stacks will be + * allocated before page_ext initialization and missing tags will be flagged. + */ +bool early_page_ext __meminitdata = true; +#else bool early_page_ext __meminitdata; +#endif static int __init setup_early_page_ext(char *str) { early_page_ext = true; From patchwork Tue Oct 24 13:46:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 887C4C00A8F for ; Tue, 24 Oct 2023 13:47:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA7626B027F; Tue, 24 Oct 2023 09:47:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BDFDB6B0280; Tue, 24 Oct 2023 09:47:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0A116B0281; Tue, 24 Oct 2023 09:47:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 88A116B027F for ; Tue, 24 Oct 2023 09:47:33 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6FE3D160985 for ; Tue, 24 Oct 2023 13:47:33 +0000 (UTC) X-FDA: 81380482386.06.71BCAA4 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 95E2C40021 for ; Tue, 24 Oct 2023 13:47:31 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qBm3infp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 38so3ZQYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=38so3ZQYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155251; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dyxdpQkY+52gRKectHQ0ToMTqRIUgvQpNStJHRxgij8=; b=bMq57LDwXDTwG4mLtd31bOpXJnn9BzBZ4DNXuf+Zbn+9WjcEIqKbwRM5fFb2MdmywVkg3H xPX6JL4ElEKGOeJ7q1BbWbGlozvEjMTdFAJXae7QRbIDVvOlFhVgXZkxv+3boH3IQN7A7Y ZiCa7y/6WsxKe/Y6DxFKBvluDwycwbI= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qBm3infp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 38so3ZQYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=38so3ZQYKCJcJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155251; a=rsa-sha256; cv=none; b=H3tfizt4xrHqvoasqeRTq/m5kENWME7eBE2Yd10sr1fe9vyyH7GwIc2eHfKHZzz/6YWoKp WuCrx2iERASyqlgjR+JfabzxEMMo3C6CL6TwKFFuMk2mTfaFTkaFh7EyRX+ixl1O2rKrsE QMfbM75kSGURJ2y7DVO/GKwb0mJc7cw= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a90d6ab944so49806777b3.2 for ; Tue, 24 Oct 2023 06:47:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155250; x=1698760050; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dyxdpQkY+52gRKectHQ0ToMTqRIUgvQpNStJHRxgij8=; b=qBm3infpJJqgBpj8BQDfguNnvSnuUNEnG0oiFtVArAtH1m7b/pRxcWutZFEwmu+wpb KYIwyPZJlFypY3/QFoiyfdKDQXb9pQyzJzC0YsYgXhGxOU/yUywl/zr+M9PUsVbjke7c 9IvKKftOlJIdpo4vsgMkAgtJMPYmnzgC5/wDqNXIhwQ2f/OvrqyOMdZoXhPvA5lt9eiU wg0pWqX5whJlQrDDPtx2CYpyl+YuJbjq2Ol/xV5JfmQtOqHegePSpgZRYixIMSV6hH9J 0JYv79+lpHrkaHpScUZXDhLoKcNcKiYR0kNx5sg2fpLwKlZ7rY/DmXkEX1yr5AnJ0BUf PY0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155250; x=1698760050; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dyxdpQkY+52gRKectHQ0ToMTqRIUgvQpNStJHRxgij8=; b=aNPDR2Zu5ScZUx6Fqj34cZzQN42BiUkvTRhFWm76EiXEwdLmOIxil/Vo6HZEC4nnEJ dHzYcAq8MDxMUjtg2ycmqUFo1Eu4HEsEXwR1wBRr1WUWPFqZniZb2SfSAqQlXYiVKNAm Pp0eif2GFMxaE1D/djjc9l4G6hRPsGOuIDhYaKtjRhbpO6K07fjTt0TV0KssTLkMbPXJ QD+9xO5u2a3QVvEGdeatz289uSsLzvDQf6LdyBbyTJTJnhbUj7oiABMv+f9tROVJgEOR +aCkIyG554RA5TjVW0kgU1jB4N2m7moYQa5qOEW8qe1zmbCaUkUdo8H6ajaN1aMCbT7F nQgw== X-Gm-Message-State: AOJu0Yx3r8ucwKA9hn60+PfUQe6aYPtnz+c9JMathEacvjmSZYvO5r2z LVD6lajmqTnmtjwBT4U64Yb37fvLijY= X-Google-Smtp-Source: AGHT+IH6emT7vBUijCZAXTQg5qCnjqbhy63v478Fz+jbhTeMwMkT3/M97iqix7XYtIWpa3tPCFKbfUZWH1c= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a0d:d84a:0:b0:5a8:5653:3323 with SMTP id a71-20020a0dd84a000000b005a856533323mr243310ywe.2.1698155250598; Tue, 24 Oct 2023 06:47:30 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:19 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-23-surenb@google.com> Subject: [PATCH v2 22/39] lib: add codetag reference into slabobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 95E2C40021 X-Stat-Signature: 1s9fhs3kipimb5ps4pyb1jj8z1iqp9ny X-Rspam-User: X-HE-Tag: 1698155251-380829 X-HE-Meta: U2FsdGVkX193Ku16yUIWd7oUIGF6LOpbKDvMCkXXehYK/TfJRcbK0NxIH1N3rkQylhavP533KlTJSB93ijzV7tNYPzLWAm3YLze6yd074tuixMsdKtdwkXAOCtoOEJySysEJpnElJFiBD6AL7ampcpTdsJJNxzb2M6bmbNu84+u4KoC9CEe2H5/H2lBsGRRZLb7NR7f5qkObDcdfLjBLlIAidO4ZoC46d1mc7wC+bUBT2M3Y1wpKowQbc6WNw3cwKMtTF+ZLqLWcZ3cXEbE7yNDojsSRzLWyguon1YQ7K+r7FSTuVN1eVizLXw7ePb8ib8c6iiQsEdJNcTi0qun/yxRXXis8mPnzfvUsyfH+3Si6UdO60EhN9/4xL15MCUH8135RdCCe0Bq7d3e1wujkjLMthi9flrVZ8ZONGHyog+p7zaFCqlQiU7j/y//j80SvdRND+3kpQtw8HDg1m1tp3A+EekYccYBhgEgbl3N9czO1icNbQubEZvF5qGuj1XGcEGT7cZXKo2yz3YFSP1T3pQdgPsdbR1l80Y3wFuPx6iIAhYbV7gtOfVqsolOEVlxwShrMRiLcYK2k7uCtz1C4XKScMc3FVptUUyhVxwyNhbvCEXKd3fIS6EIm5f6JfFwAdiAKnhmvNWkXcy1PzzgRTMhQiEHFC9iPnxX5ze4REqPEY5N/rod+Gq/8WGx3OGf+P/2k8baJaKOlPYdcq4e2ANrHa8SIycK4ta1TvTwLjsvnOL/TPjr1/cmqLrjXebS72nQ/8HCy8G1OUEoZU+KLNFMDYnOhFvMmy2X6JqlDoaVqI7oZFqWYQeGe4foFoAJygdmpFx4nParGg6s2tbsW8YvCf5shuWPfzZeShypC+E+1piJfhl6KH824EefV+92GkhVmJu6yPtcm0ivGVzCdrbyD+KQOMnTHDACmtPKPvcoHjCNR2dXlu7UzLSfbkmMmUuOWTTZumJUJbKRV1T+ he9c9x// oQfTXwot7KPtgK9JUkSpTSPLZQWCQ0B5u3qvT0x6Lb0Y2JJm+NBd56YEcVDioWRmBTQoouijgdvsNipwCc0V7LPviryBFvnJS78uQ3msi15zdFpdistuQtZlSg3G2HorYi/7Jpc5rM8+x49ApHGHypC4U6z9I80CS9wMOAvL7XsCZVmic2Zo98lXNbbAPWKHl8qSStnhU79QelA3DtU2nENqXEfK5ExVBqmnKqo2cBWOCnhjLVwWhGGWeYFuU/gu6kIm9L2M9fXdeICkIG1Z8FGgsIJV+gHoaw5ZcvMMLzM4iF72vYLKm6dmvLFNS8G+/KIqBfDvS/bd0j4RY1tXjfzs6nMN40CRngNZNXrPLYcOPzCQzCe6g3S2HaEx8yP7zxNP1WcXDvSZzLRKAF+VbXk0UwTJL+LdW6+soUfUAbhs1n4yF3x6urLxRnJxWBs18j1IDL5BuPQGHtb0JhnHHr4F+8XpKQpp8KhWerK/9j6JoyFEsv/FNwGYSqmCGCtpg7i2tT39l03xk4TydRqsX5AL3bp7g0wyPv5HG0hshco1xXgdBm4mfC0UOKg6QbW4JNmLpnfA6npZPYVAEkZnEZZ5CWa2AMoYPsnJ+fXFAxv95Dp/dLEdxQbQqVfTUc7fwte0iGnWsT5/l75NyEfizVMSSgsDPqYKKS5wXqjQPLLsEsH+lrkBr7gzXgDL3M7IJbI6Nn4UG6kG2mG/3bu393nRYF+7o0FcnXE/Y X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To store code tag for every slab object, a codetag reference is embedded into slabobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/memcontrol.h | 5 +++++ lib/Kconfig.debug | 1 + mm/slab.h | 4 ++++ 3 files changed, 10 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index f3ede28b6fa6..853a24b5f713 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1613,7 +1613,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, * if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref ref; +#endif } __aligned(8); static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index e1eda1450d68..482a6aae7664 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -973,6 +973,7 @@ config MEM_ALLOC_PROFILING depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION + select SLAB_OBJ_EXT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/mm/slab.h b/mm/slab.h index 60417fd262ea..293210ed10a9 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -457,6 +457,10 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, static inline bool need_slab_obj_ext(void) { +#ifdef CONFIG_MEM_ALLOC_PROFILING + if (mem_alloc_profiling_enabled()) + return true; +#endif /* * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally * inside memcg_slab_post_alloc_hook. No other users for now. From patchwork Tue Oct 24 13:46:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DFD5C07545 for ; Tue, 24 Oct 2023 13:47:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21E086B0281; Tue, 24 Oct 2023 09:47:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 135836B0282; Tue, 24 Oct 2023 09:47:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEDDB6B0283; Tue, 24 Oct 2023 09:47:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C91976B0281 for ; Tue, 24 Oct 2023 09:47:35 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8E86C1205F5 for ; Tue, 24 Oct 2023 13:47:35 +0000 (UTC) X-FDA: 81380482470.16.51524F6 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf28.hostedemail.com (Postfix) with ESMTP id D03B5C0004 for ; Tue, 24 Oct 2023 13:47:33 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OBcROUbK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 39Mo3ZQYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=39Mo3ZQYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155253; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jxcZ43bmgT/q64Ul96+yH7BdLzW8wJzWTibIuRMuFN0=; b=1F6CCmJQy2p9UNCjiJECWw+hnyUjncNk/xdSedIxdIqYoxRg1a2JhCS0okTUW113ZV6hTM 828sm+NPBIArwadZjUHdGhpe1H8IndfPAJXGEIepVgHqL7jQMPHVe5G64Ps0vy6RopNW+N 9StGOJ1DzzjPkkLe832SLL/vVsIt4VI= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OBcROUbK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 39Mo3ZQYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=39Mo3ZQYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155253; a=rsa-sha256; cv=none; b=PC4X5gQn2nfCsYk+luYbNSG4TRu2CwWAXyYaZY8nJ1ig9h516fW7hHAxxx/XvNvbVtpc2l +/c4qHSPJWD+29SJxYW2SsGnRBohwm4KWY1BybQxmlsxNorG/PcD1FNTjn8mrTt4H4EpF/ otgsn6rb9u/VWUNPtG9RRBGlnaiArfg= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-da03390793fso1013537276.3 for ; Tue, 24 Oct 2023 06:47:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155253; x=1698760053; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jxcZ43bmgT/q64Ul96+yH7BdLzW8wJzWTibIuRMuFN0=; b=OBcROUbKNdAqdzkrkQOwPJni4l3cOJwTjho2FddfM4hpGjHOIsUL2sxxylcPh2gjt/ TF0ikWjuVHc4l0n+ZzmsYURptMinpo7IILCz7GcaLjOAfZXP5MWa9RHAa56JCw//M6P7 q6vanf3tCkZcpWu0loGQ/XrUK4IFDbUak6nfUS+3tC2/Hhe5/fPMEIXWAQroY8GGaQfX HdehAvDCRld8Aqlr2d+fAc4IEIfe+YS9Pqy8axYEUca39z0qZi/OnYpcyzNywCQn9jBJ mQF8KdkGpeqQq5NM5VCoKhE2Z6mnBzjqLsZn8Rpli/7glDDQ69zNRwlgZL2Y3Atn962O JfOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155253; x=1698760053; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jxcZ43bmgT/q64Ul96+yH7BdLzW8wJzWTibIuRMuFN0=; b=rCkHULOUmKpgxXapKWmGJSccy6wwh1LypvDlpN3neUrienmxeTk2h97b16b7gdOIlo EIP8PWkub2fhxhK0Saqxzj1Iy5WW3l1x3TnUBsy1wQvkfk+NvXKJJlJ9QW6TkhUhRKv9 mSfNadlt0JtMo7EqK28HkUME4yenGyX1gomQshQHCvSJU+J9jzkNXGvWxmouOtoJFsZU HcN2o8pk5b8uJXb/+9076SiATRKDxgTTrMfj+s1Bi6CVo/7ha1C+jmbq82Kxuo/Gqh6k jiTzZXtZMUar1ipM1Eihtkeyy4YgEu2JKxiMKHCU5Mx4ZWm6a9taPP5gyeu34Cs/mpxY C4eg== X-Gm-Message-State: AOJu0Yzlsl8xqkfwwKX7nJbFvpU37cKn6tE/TYpWCW42L+7BKjsnsthq u2r4+b1avBfwjCPaeGU6VhRxP22Qft0= X-Google-Smtp-Source: AGHT+IHLiWJmtuPzRqvJDSScUKUh0y8YokFyWjfHx2rxdH2c6l653IiepUePcxPp47iOlZTgh6t0s/pZmOY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:d05:0:b0:d9a:5b63:a682 with SMTP id 5-20020a250d05000000b00d9a5b63a682mr216305ybn.13.1698155252840; Tue, 24 Oct 2023 06:47:32 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:20 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-24-surenb@google.com> Subject: [PATCH v2 23/39] mm/slab: add allocation accounting into slab allocation and free paths From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: D03B5C0004 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 5znkmu3s5ypyuq3ssh8sarbhcxp4e99i X-HE-Tag: 1698155253-593740 X-HE-Meta: U2FsdGVkX1+D77zX6fv090eCksxcWaniApENkNyjuSem2QqtvfA13UdQlJgIskomsspIVJgecMSaSBEuVHimtVb+D68OJL0rMAVSW5wIQK7GbTBqq4x80Z34LBrUspBRfYRfS9X4BY5blR2rI+vHdPlFXluompy17rt7ajmZhxWtIvc2xfhmNrvQBk3OxXTRTjhtknLmgm6GNVqXlMbLgJ3cMlIRMhZIvBvDYw4xdnEE32QriHeX3ydPwA/0kiZtCQd/3LXd+kbE6xGRU1oB1qgr4RJjHDp3RpYFbEZszdeUNi5WMxwO/SNKs9hQAMhD3XQkKrhbatPR0OQlKx8TxNS7k1OzSz8OZCuKwqC1VOM4rb6KAhs2Itcp+taS6fxoD07XqLNX/v2iKThgmHTfbHiTc9zeNgv2pa+w/AD9Si+twfQRh8G/+FPt/edht/VIWM3j1j7bz6t/lErV+0WIm2AgvdEp1O9gyqaNYWpAuKkijyQYoq3LOt1U+lPujteiIkPQoA45aPUFWDlkSlASEw4JRxNKPcyYpz8m5U57A7DWkODh6sbti3YGzGvx1Mjoi8cCN63gKaBL7l5XCy+Kzr7q8U3WXTcRBCk6UqLLDcsf+jGAGvveYFU3QT84LRL0KKkRInR9Rd+pELdtvxC0JC0mguhSSWxERWQYD0cLw6PlyoA12A+xbOGXPs2mbrXD+QGl96KDyugMyHy5DTRE0QYKqa9r26TU2TQ1aQRbQZJsRAtDiWkTfysrJjNNuiWMLhLsWNY/S630xq2wq96obgRrbq96psx5nYAh6maV/E7gZpHCqKMD0NJ9vSPhxwDLfGuhQs5vW0Yz9zTCa6SRZu/Xsd1cmcY8H60yCrPQ/o9sFJMaScd8U/RC69EDIk3VLYw06dZpf9cJO3916MzdM7hSZ04DtBD9c2Dje8ltG8KuwCoyR0LjDvBAV3K2IwDZhbc7tynmR3yIZDl8AKC oXTn8yCX cXPw3uULd5XmOSpR9al+fVNeYbKN9BzE338c1tii5kv7Ad/cSpfYKt8yvSVFbOyuX5EUUsrj8Awx7bjTlbWFy8FdKLhYWZVNrtHvwSRvKq1vdkhNbo8CF+c4xYKdCGjeD/A2UmuNbFSmmsk0kxRk1gisR0PQvkY/RJir89/xqKVm9jvjHtflUKsWrCDQwFLWYRv7Ww8tEm/LgsG3N8ZuM/ThsHHBeZ36SOTroWyVuGuRMSVvPilXjGDUSdkgXPIvo8Fw1WHcgIDagd4BmmZR8nk8RGlqMFeZbA2fDxTjfCF0dpweSNFDpinVF1LyFVVOpa7hpF188tA4IclbHLJqFPxnbW06sy7Q4GqWOoK+/NFgz10CaRuvfx3zAYxmg1r0NpDssewQpjLScnMthGGc3d5sDiqDxEvRFJkZ4lhrYLrtq2Cl+d5pY6gY5aHIAohNH3T5+mEaeSap3+tLjVMqqS8BQIK3istUzoqHYFG3AcJHKTGF+YSQCVJTkfrsVWQtNg6UioVZrH/JUv+CP+Twh7sCXKhWzrqvj5Fheb0/3OeYvAgwjLjIP8vzrQfsKvkvxlpI3zW0kovWW+tLa96lw35dhvEvm/dJ5bz7lFAREEavoju2uT670DbW5xywKq9LYEF9wFclOoqcEOGRJ8HLRTu4/dFX2lai2gqSTj8+UTztW85UGPmysG6vzHITb511xgzVSlmtAKZJwCYI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Account slab allocations using codetag reference embedded into slabobj_ext. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/slab_def.h | 2 +- include/linux/slub_def.h | 4 ++-- mm/slab.c | 4 +++- mm/slab.h | 32 ++++++++++++++++++++++++++++++++ 4 files changed, 38 insertions(+), 4 deletions(-) diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h index a61e7d55d0d3..23f14dcb8d5b 100644 --- a/include/linux/slab_def.h +++ b/include/linux/slab_def.h @@ -107,7 +107,7 @@ static inline void *nearest_obj(struct kmem_cache *cache, const struct slab *sla * reciprocal_divide(offset, cache->reciprocal_buffer_size) */ static inline unsigned int obj_to_index(const struct kmem_cache *cache, - const struct slab *slab, void *obj) + const struct slab *slab, const void *obj) { u32 offset = (obj - slab->s_mem); return reciprocal_divide(offset, cache->reciprocal_buffer_size); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index deb90cf4bffb..43fda4a5f23a 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -182,14 +182,14 @@ static inline void *nearest_obj(struct kmem_cache *cache, const struct slab *sla /* Determine object index from a given position */ static inline unsigned int __obj_to_index(const struct kmem_cache *cache, - void *addr, void *obj) + void *addr, const void *obj) { return reciprocal_divide(kasan_reset_tag(obj) - addr, cache->reciprocal_size); } static inline unsigned int obj_to_index(const struct kmem_cache *cache, - const struct slab *slab, void *obj) + const struct slab *slab, const void *obj) { if (is_kfence_address(obj)) return 0; diff --git a/mm/slab.c b/mm/slab.c index cefcb7499b6c..18923f5f05b5 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3348,9 +3348,11 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac) static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp, unsigned long caller) { + struct slab *slab = virt_to_slab(objp); bool init; - memcg_slab_free_hook(cachep, virt_to_slab(objp), &objp, 1); + memcg_slab_free_hook(cachep, slab, &objp, 1); + alloc_tagging_slab_free_hook(cachep, slab, &objp, 1); if (is_kfence_address(objp)) { kmemleak_free_recursive(objp, cachep->flags); diff --git a/mm/slab.h b/mm/slab.h index 293210ed10a9..4859ce1f8808 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -533,6 +533,32 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) #endif /* CONFIG_SLAB_OBJ_EXT */ +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects) +{ + struct slabobj_ext *obj_exts; + int i; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + for (i = 0; i < objects; i++) { + unsigned int off = obj_to_index(s, slab, p[i]); + + alloc_tag_sub(&obj_exts[off].ref, s->size); + } +} + +#else + +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + #ifdef CONFIG_MEMCG_KMEM void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, enum node_stat_item idx, int nr); @@ -827,6 +853,12 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, s->flags, flags); kmsan_slab_alloc(s, p[i], flags); obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + /* obj_exts can be allocated for other reasons */ + if (likely(obj_exts) && mem_alloc_profiling_enabled()) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); +#endif } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); From patchwork Tue Oct 24 13:46:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434594 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B15B8C07545 for ; Tue, 24 Oct 2023 13:47:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74DD96B0283; Tue, 24 Oct 2023 09:47:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D5906B0284; Tue, 24 Oct 2023 09:47:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 465936B0285; Tue, 24 Oct 2023 09:47:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2C19A6B0283 for ; Tue, 24 Oct 2023 09:47:38 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F292A12016A for ; Tue, 24 Oct 2023 13:47:37 +0000 (UTC) X-FDA: 81380482554.22.A819743 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 2DAFA120021 for ; Tue, 24 Oct 2023 13:47:35 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vtNhavO0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 398o3ZQYKCJwOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=398o3ZQYKCJwOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155256; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UgU+pcxUdPHIak08TaAB2rSrJMdjgr1bUXnQB/iAGbQ=; b=bTGLdWOq0moA6fVr0A9laZSrg9q1paQWxX0Z+Xs+5U2q0+kI6Aj1GGhP9iI+TK0vzxV3jQ bKQf/b9aGuS3HA7WiZ7EQEqDoW0BKYzvJ1K4ApBzQM5MfVmCTJIao4NKcX9EStGEVRDrdN 2CdHBLt0NPh5YeJgjezEJ8kG3mNDDUU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vtNhavO0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 398o3ZQYKCJwOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=398o3ZQYKCJwOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155256; a=rsa-sha256; cv=none; b=b/j01ryIIU8jVHqk7uVnFYXwUz/+Egc2LyOh/X35pvfohXt29r2uOr4pN0QsmaXPIWxInR isXrjvxS7hpY3if0yKlhazDWch3kzPaZXOlSBJPahztK71O8lupMsKswRS96mCluVspwgk gJP5uBI6/6C6nAA1I6bXWkGquqG9Hd0= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7a6fd18abso57842807b3.1 for ; Tue, 24 Oct 2023 06:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155255; x=1698760055; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UgU+pcxUdPHIak08TaAB2rSrJMdjgr1bUXnQB/iAGbQ=; b=vtNhavO0YKKVMXgJLgAm6Skp3cUl534NwUMavEanxpCLsDkW/0bUKoVp+JrQtPmtZg K43TbLDURlbH3UJaJFYKkOxyyMn73Abd19+oWTCnhiEAnhLCtVGnu6PStlIODqGYjhUE p0I20J35aRwjG4qrVcgOJ2OZw/r2CPJ7g5Lilk0JUne0KQJs1uJBrgVA+QKSY0pqdb0J YMbcmdmUzRhIK12SG1ueypMQGdnvXrTYk3qyz/39zF1On4gdzwXAJ4nSpkLQZzWVm4G1 e9RetQxcUoIcHzR+WESihMC+H/m5NmhwYVuZRaFTxCQqecUlL1euOi2xPEyeF0laCCtX dwPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155255; x=1698760055; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UgU+pcxUdPHIak08TaAB2rSrJMdjgr1bUXnQB/iAGbQ=; b=gJzRrJ3H0z7Iogkctj13WR42wA/9OG0Vj4o+tBLBDFpPEJwOZhAb79BgS9HZcwdoAN pNMzd8KALsfD+Ua6K6kGmA0z0wYYFJDMRwCZr1QRVkngEr0ybgb8gpH2Q/EHjbCf+Zk6 HQJwWgF+YXFEojUO5urGLS1VfreKpmzsM0RX931CwK9Yk7RHM+K3Jd07T3scab7n6GF9 3mL3ZpdosMkfTPmD7kK1lnXWAmyUzJvFmktLNM+ouWyRsgJGKZkwWwyl2k4HVZLcldi5 IVCAoD8DBHW8kIUuvl1HuqgHQXwbGFyRqUcj08ARe3YXMCVA8WyD9ZiKQ6tCoNHt7Lxu EnvA== X-Gm-Message-State: AOJu0Yya0lI8xoJspJ0gIDDD3kfBdN6hBNaSQXFj7YFiv3KwW4LwPbjf ouJSe9wD/2vwjW1cMMH5ICnfv+1zm38= X-Google-Smtp-Source: AGHT+IH99U95XbIJleJRvWe1tdmgTc3A/ryOVVoVSn5UdR3NB6RxslzPIkyuREwn+3gBRv/mIBt+xGxQfKM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a81:a157:0:b0:5a8:d81f:f5e7 with SMTP id y84-20020a81a157000000b005a8d81ff5e7mr260833ywg.8.1698155255155; Tue, 24 Oct 2023 06:47:35 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:21 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-25-surenb@google.com> Subject: [PATCH v2 24/39] mm/slab: enable slab allocation tagging for kmalloc and friends From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: sbjxs7kdnwjd61wuhpo8pwjr7gz3e1a9 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2DAFA120021 X-HE-Tag: 1698155255-578043 X-HE-Meta: U2FsdGVkX1+NeUO0e6+fzkOw82BIav3XjP3XHwF2dpbloetq49QRu2rC6Pkkd7Zgbm9UO1DrkQSwbMfXL2IO+Rv8LzC92pw3x3VXXbP1ZyOccvPbVVTcgMbhniQR3SG7rEFyBBr/gKNm1lQLOtZFR2hjPoFA7pg+j2R7l+KVBbMSASNxurROSt62P0SujmhSdDphExV83EhhzJTBAO0/WkeU5sNfurVnJqbKgPVhtDfPeRH60dWgLFjUK/B7CBMTpWy6ugD0m2LzMax4XKQabXafbRMAz1FpewehReLJfYbKB5qx1K2cgiSaHINk088Incw080Ws9KwtO8hd8bVXZ7oRx8ToJesOF0yP/xSx80mBI8nmmq2QYDQREcQic2OJz1vtTP4Q+jiTT5iDjDVR+YTpr1zZwMzBQMZZQmgv7XtlvbH5duMzH+2dFQ3LFD3pmXpcLQWhLI01QJ8RxdperwpjkSAW/UeI9/qf1ZSZoiUH0V14WjkumoaWw/DOP6EpG26m6BvZP2psPmuKBCytdqanuFEfVmv4NNz0WA2DxLnptuSIXcEZctbW+N+XUQQBK4GgJ+Nvp2CF7RR1RUQ5ntHu+b7tNRXdTdwaH1IJMxkyb8lWaNsYQiItDn/gyFx5gtDSIcsJtMcY7PpXP+3VBySSa1tfjwdE0kkFQ0PlZRgbLGoDvm6MJmNy4/WpDttuF/br8IBpQLWStFrl1QBl7i0VCPlXAL46xGHb6Kh/UKNihfBIzCTUFipwwx7g4Jxtnzy6cG+e1Q6lacdmnP5XKRWpaDMZWlG1aNhpRlQSktWz1Se4NsORCDOluRes8hyrjhw04AG2ifQHXLOiDq8U/3PDhvbpWmG75jjFJAoucucr9mGPI2xJfg6aPyBAGP+8qavSNYxshKH6HqGU4qnJyOGNdVAY0ao/acds6fJf3OJ6NP4RRu8gLe6vJUaYz8qjFKkWEF1rFWTvXG3uL1i RWWGt9K0 79zXrTF1Z4lovXLVi+FnO14sHtsvunYBYSlm9sOXXi/0pMeuj+MoW3bpBArnv/Y0iky0yQh+0xkI1GML1a+hTS3wnSsRScBw4EdwFCkvAA3HuUT4IVgoAWsQBrAeHknG5FG4+rgdgPUZNQVDpUGB1nPOt6rR1TXFNTJBFZj4RPCmd1ZcV+JIml/UtS2zC2o6t8m5VdSh3ajxrw3wCQL1q5xinheJbP8mpGt8HFZx15LQePWYpieXvKy5yaY4NL243uuaNHnwQ5DBouQCwE8sTRO7bdNRv75cW8aV3akTxhW4o5Jl/FkVN/a7T1bhoXllUaIOGtrxRIqJjftWXRrWcDT19ISaI9zxs6HUr8tmyssO5pNj31IKIfSoP3M0lKgByO6rAOey4IXMLPBQI/nOo5lIveYNjY1x/mrn3GWuVPOGcLr0KV3d7lAXvT7XVG9YbNsa5mFxMhCxSegrdvc+yKFEFTC3vV5ib4UNX87J7yBV0o4YW/jkr2bplEFovwk38aLo+ozFqwZoPnlwtrvNLntM9eYYifoCgipPNFB4b4YUxG/khpKQDtpQcr6Purc0j06uer2I/Jcgekvnb3J5mHVAUOFt9dnyu1ZtHc8T3VOLq6vUU1M06JIlhV9F+JVN3SQVZJsa2YhuPSnA8YwM130Ubm6LxGnDVKK1i1kCg02WHiL8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine kmalloc, krealloc, kzalloc, kcalloc, etc. to record allocations and deallocations done by these functions. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/fortify-string.h | 5 +- include/linux/slab.h | 173 ++++++++++++++++----------------- include/linux/string.h | 4 +- mm/slab.c | 18 ++-- mm/slab_common.c | 38 ++++---- mm/slub.c | 19 ++-- mm/util.c | 20 ++-- 7 files changed, 137 insertions(+), 140 deletions(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index da51a83b2829..11319e7634a4 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -752,9 +752,9 @@ __FORTIFY_INLINE void *memchr_inv(const void * const POS0 p, int c, size_t size) return __real_memchr_inv(p, c, size); } -extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup) +extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup_noprof) __realloc_size(2); -__FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp) +__FORTIFY_INLINE void *kmemdup_noprof(const void * const POS0 p, size_t size, gfp_t gfp) { const size_t p_size = __struct_size(p); @@ -764,6 +764,7 @@ __FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp fortify_panic(__func__); return __real_kmemdup(p, size, gfp); } +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) /** * strcpy - Copy a string into another string buffer diff --git a/include/linux/slab.h b/include/linux/slab.h index 11ef3d364b2b..0543e0f76c60 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -230,7 +230,9 @@ int kmem_cache_shrink(struct kmem_cache *s); /* * Common kmalloc functions provided by all allocators */ -void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __realloc_size(2); +void * __must_check krealloc_noprof(const void *objp, size_t new_size, gfp_t flags) __realloc_size(2); +#define krealloc(...) alloc_hooks(krealloc_noprof(__VA_ARGS__)) + void kfree(const void *objp); void kfree_sensitive(const void *objp); size_t __ksize(const void *objp); @@ -491,7 +493,10 @@ static __always_inline unsigned int __kmalloc_index(size_t size, static_assert(PAGE_SHIFT <= 20); #define kmalloc_index(s) __kmalloc_index(s, true) -void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#include + +void *__kmalloc_noprof(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#define __kmalloc(...) alloc_hooks(__kmalloc_noprof(__VA_ARGS__)) /** * kmem_cache_alloc - Allocate an object @@ -503,9 +508,13 @@ void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_siz * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, - gfp_t gfpflags) __assume_slab_alignment __malloc; +void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) + +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_lru(...) alloc_hooks(kmem_cache_alloc_lru_noprof(__VA_ARGS__)) + void kmem_cache_free(struct kmem_cache *s, void *objp); /* @@ -516,29 +525,40 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); * Note that interrupts must be enabled when calling these functions. */ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); + +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +#define kmem_cache_alloc_bulk(...) alloc_hooks(kmem_cache_alloc_bulk_noprof(__VA_ARGS__)) static __always_inline void kfree_bulk(size_t size, void **p) { kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment - __malloc; +#define __kmalloc_node(...) alloc_hooks(__kmalloc_node_noprof(__VA_ARGS__)) -void *kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment + __malloc; +#define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) + +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t flags, size_t size) __assume_kmalloc_alignment __alloc_size(3); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, - int node, size_t size) __assume_kmalloc_alignment +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t size) __assume_kmalloc_alignment __alloc_size(4); -void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment +#define kmalloc_trace(...) alloc_hooks(kmalloc_trace_noprof(__VA_ARGS__)) + +#define kmalloc_node_trace(...) alloc_hooks(kmalloc_node_trace_noprof(__VA_ARGS__)) + +void *kmalloc_large_noprof(size_t size, gfp_t flags) __assume_page_alignment __alloc_size(1); +#define kmalloc_large(...) alloc_hooks(kmalloc_large_noprof(__VA_ARGS__)) -void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_alignment +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) __assume_page_alignment __alloc_size(1); +#define kmalloc_large_node(...) alloc_hooks(kmalloc_large_node_noprof(__VA_ARGS__)) /** * kmalloc - allocate kernel memory @@ -594,37 +614,39 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_align * Try really hard to succeed the allocation but fail * eventually. */ -static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) +static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large(size, flags); + return kmalloc_large_noprof(size, flags); index = kmalloc_index(size); - return kmalloc_trace( + return kmalloc_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, size); } - return __kmalloc(size, flags); + return __kmalloc_noprof(size, flags); } +#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) -static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) +static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large_node(size, flags, node); + return kmalloc_large_node_noprof(size, flags, node); index = kmalloc_index(size); - return kmalloc_node_trace( + return kmalloc_node_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node(size, flags, node); + return __kmalloc_node_noprof(size, flags, node); } +#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) /** * kmalloc_array - allocate memory for an array. @@ -632,16 +654,17 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc(bytes, flags); - return __kmalloc(bytes, flags); + return kmalloc_noprof(bytes, flags); + return kmalloc_noprof(bytes, flags); } +#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) /** * krealloc_array - reallocate memory for an array. @@ -650,18 +673,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_ * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) */ -static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, - size_t new_n, - size_t new_size, - gfp_t flags) +static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p, + size_t new_n, + size_t new_size, + gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(new_n, new_size, &bytes))) return NULL; - return krealloc(p, bytes, flags); + return krealloc_noprof(p, bytes, flags); } +#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__)) /** * kcalloc - allocate memory for an array. The memory is set to zero. @@ -669,16 +693,11 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flags) -{ - return kmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kcalloc(_n, _size, _flags) kmalloc_array(_n, _size, (_flags) | __GFP_ZERO) -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, unsigned long caller) __alloc_size(1); -#define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ - _RET_IP_) +#define kmalloc_node_track_caller(...) alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -688,11 +707,9 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, * allocator where we care about the real place the memory allocation * request comes from. */ -#define kmalloc_track_caller(size, flags) \ - __kmalloc_node_track_caller(size, flags, \ - NUMA_NO_NODE, _RET_IP_) +#define kmalloc_track_caller(...) kmalloc_node_track_caller(__VA_ARGS__, NUMA_NO_NODE) -static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags, +static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) { size_t bytes; @@ -700,75 +717,51 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc_node(bytes, flags, node); - return __kmalloc_node(bytes, flags, node); + return kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(bytes, flags, node); } +#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) -static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node) -{ - return kmalloc_array_node(n, size, flags | __GFP_ZERO, node); -} +#define kcalloc_node(_n, _size, _flags, _node) kmalloc_array_node(_n, _size, (_flags) | __GFP_ZERO, _node) /* * Shortcuts */ -static inline void *kmem_cache_zalloc(struct kmem_cache *k, gfp_t flags) -{ - return kmem_cache_alloc(k, flags | __GFP_ZERO); -} +#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO) /** * kzalloc - allocate memory. The memory is set to zero. * @size: how many bytes of memory are required. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1) void *kzalloc(size_t size, gfp_t flags) +static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) { - return kmalloc(size, flags | __GFP_ZERO); + return kmalloc_noprof(size, flags | __GFP_ZERO); } +#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) +#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -/** - * kzalloc_node - allocate zeroed memory from a particular memory node. - * @size: how many bytes of memory are required. - * @flags: the type of memory to allocate (see kmalloc). - * @node: memory node from which to allocate - */ -static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int node) -{ - return kmalloc_node(size, flags | __GFP_ZERO, node); -} +extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) -extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1); -static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags) -{ - return kvmalloc_node(size, flags, NUMA_NO_NODE); -} -static inline __alloc_size(1) void *kvzalloc_node(size_t size, gfp_t flags, int node) -{ - return kvmalloc_node(size, flags | __GFP_ZERO, node); -} -static inline __alloc_size(1) void *kvzalloc(size_t size, gfp_t flags) -{ - return kvmalloc(size, flags | __GFP_ZERO); -} +#define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) +#define kvzalloc(_size, _flags) kvmalloc(_size, _flags|__GFP_ZERO) -static inline __alloc_size(1, 2) void *kvmalloc_array(size_t n, size_t size, gfp_t flags) -{ - size_t bytes; - - if (unlikely(check_mul_overflow(n, size, &bytes))) - return NULL; +#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) - return kvmalloc(bytes, flags); -} +#define kvmalloc_array(_n, _size, _flags) \ +({ \ + size_t _bytes; \ + \ + !check_mul_overflow(_n, _size, &_bytes) ? kvmalloc(_bytes, _flags) : NULL; \ +}) -static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t flags) -{ - return kvmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kvcalloc(_n, _size, _flags) kvmalloc_array(_n, _size, _flags|__GFP_ZERO) -extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +extern void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) __realloc_size(3); +#define kvrealloc(...) alloc_hooks(kvrealloc_noprof(__VA_ARGS__)) + extern void kvfree(const void *addr); extern void kvfree_sensitive(const void *addr, size_t len); diff --git a/include/linux/string.h b/include/linux/string.h index dbfc66400050..9516258d8117 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -176,7 +176,9 @@ extern void kfree_const(const void *x); extern char *kstrdup(const char *s, gfp_t gfp) __malloc; extern const char *kstrdup_const(const char *s, gfp_t gfp); extern char *kstrndup(const char *s, size_t len, gfp_t gfp); -extern void *kmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +extern void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) + extern void *kvmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); extern char *kmemdup_nul(const char *s, size_t len, gfp_t gfp); diff --git a/mm/slab.c b/mm/slab.c index 18923f5f05b5..f75519fa89b9 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3429,18 +3429,18 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, return ret; } -void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) +void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, gfp_t flags) { return __kmem_cache_alloc_lru(cachep, NULL, flags); } -EXPORT_SYMBOL(kmem_cache_alloc); +EXPORT_SYMBOL(kmem_cache_alloc_noprof); -void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags) { return __kmem_cache_alloc_lru(cachep, lru, flags); } -EXPORT_SYMBOL(kmem_cache_alloc_lru); +EXPORT_SYMBOL(kmem_cache_alloc_lru_noprof); static __always_inline void cache_alloc_debugcheck_after_bulk(struct kmem_cache *s, gfp_t flags, @@ -3452,8 +3452,8 @@ cache_alloc_debugcheck_after_bulk(struct kmem_cache *s, gfp_t flags, p[i] = cache_alloc_debugcheck_after(s, flags, p[i], caller); } -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, - void **p) +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, + void **p) { struct obj_cgroup *objcg = NULL; unsigned long irqflags; @@ -3491,7 +3491,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, kmem_cache_free_bulk(s, i, p); return 0; } -EXPORT_SYMBOL(kmem_cache_alloc_bulk); +EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof); /** * kmem_cache_alloc_node - Allocate an object on the specified node @@ -3506,7 +3506,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_bulk); * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) +void *kmem_cache_alloc_node_noprof(struct kmem_cache *cachep, gfp_t flags, int nodeid) { void *ret = slab_alloc_node(cachep, NULL, flags, nodeid, cachep->object_size, _RET_IP_); @@ -3514,7 +3514,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_node); +EXPORT_SYMBOL(kmem_cache_alloc_node_noprof); void *__kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size, diff --git a/mm/slab_common.c b/mm/slab_common.c index 446f406d2703..8ef5e47ff6a7 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1077,24 +1077,24 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller return ret; } -void *__kmalloc_node(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) { return __do_kmalloc_node(size, flags, node, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc_node); +EXPORT_SYMBOL(__kmalloc_node_noprof); -void *__kmalloc(size_t size, gfp_t flags) +void *__kmalloc_noprof(size_t size, gfp_t flags) { return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc); +EXPORT_SYMBOL(__kmalloc_noprof); -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, - int node, unsigned long caller) +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, + int node, unsigned long caller) { return __do_kmalloc_node(size, flags, node, caller); } -EXPORT_SYMBOL(__kmalloc_node_track_caller); +EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); /** * kfree - free previously allocated memory @@ -1161,7 +1161,7 @@ size_t __ksize(const void *object) return slab_ksize(folio_slab(folio)->slab_cache); } -void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) { void *ret = __kmem_cache_alloc_node(s, gfpflags, NUMA_NO_NODE, size, _RET_IP_); @@ -1171,9 +1171,9 @@ void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_trace); +EXPORT_SYMBOL(kmalloc_trace_noprof); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) { void *ret = __kmem_cache_alloc_node(s, gfpflags, node, size, _RET_IP_); @@ -1183,7 +1183,7 @@ void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_node_trace); +EXPORT_SYMBOL(kmalloc_node_trace_noprof); gfp_t kmalloc_fix_flags(gfp_t flags) { @@ -1213,7 +1213,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) flags = kmalloc_fix_flags(flags); flags |= __GFP_COMP; - page = alloc_pages_node(node, flags, order); + page = alloc_pages_node_noprof(node, flags, order); if (page) { ptr = page_address(page); mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, @@ -1228,7 +1228,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) return ptr; } -void *kmalloc_large(size_t size, gfp_t flags) +void *kmalloc_large_noprof(size_t size, gfp_t flags) { void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); @@ -1236,9 +1236,9 @@ void *kmalloc_large(size_t size, gfp_t flags) flags, NUMA_NO_NODE); return ret; } -EXPORT_SYMBOL(kmalloc_large); +EXPORT_SYMBOL(kmalloc_large_noprof); -void *kmalloc_large_node(size_t size, gfp_t flags, int node) +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) { void *ret = __kmalloc_large_node(size, flags, node); @@ -1246,7 +1246,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) flags, node); return ret; } -EXPORT_SYMBOL(kmalloc_large_node); +EXPORT_SYMBOL(kmalloc_large_node_noprof); #ifdef CONFIG_SLAB_FREELIST_RANDOM /* Randomize a generic freelist */ @@ -1460,7 +1460,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; } - ret = kmalloc_track_caller(new_size, flags); + ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); @@ -1484,7 +1484,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) * * Return: pointer to the allocated memory or %NULL in case of error */ -void *krealloc(const void *p, size_t new_size, gfp_t flags) +void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) { void *ret; @@ -1499,7 +1499,7 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags) return ret; } -EXPORT_SYMBOL(krealloc); +EXPORT_SYMBOL(krealloc_noprof); /** * kfree_sensitive - Clear sensitive information in memory before freeing diff --git a/mm/slub.c b/mm/slub.c index d16643492320..f5e07d8802e2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3497,18 +3497,18 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, return ret; } -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +void *kmem_cache_alloc_noprof(struct kmem_cache *s, gfp_t gfpflags) { return __kmem_cache_alloc_lru(s, NULL, gfpflags); } -EXPORT_SYMBOL(kmem_cache_alloc); +EXPORT_SYMBOL(kmem_cache_alloc_noprof); -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) { return __kmem_cache_alloc_lru(s, lru, gfpflags); } -EXPORT_SYMBOL(kmem_cache_alloc_lru); +EXPORT_SYMBOL(kmem_cache_alloc_lru_noprof); void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node, size_t orig_size, @@ -3518,7 +3518,7 @@ void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, caller, orig_size); } -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); @@ -3526,7 +3526,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_node); +EXPORT_SYMBOL(kmem_cache_alloc_node_noprof); static noinline void free_to_partial_list( struct kmem_cache *s, struct slab *slab, @@ -3802,6 +3802,7 @@ static __fastpath_inline void slab_free(struct kmem_cache *s, struct slab *slab, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); + alloc_tagging_slab_free_hook(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. @@ -4032,8 +4033,8 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, #endif /* CONFIG_SLUB_TINY */ /* Note that interrupts must be enabled when calling this function. */ -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, - void **p) +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, + void **p) { int i; struct obj_cgroup *objcg = NULL; @@ -4057,7 +4058,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, slab_want_init_on_alloc(flags, s), s->object_size); return i; } -EXPORT_SYMBOL(kmem_cache_alloc_bulk); +EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof); /* diff --git a/mm/util.c b/mm/util.c index 8cbbfd3a3d59..27ed6a5ac31a 100644 --- a/mm/util.c +++ b/mm/util.c @@ -115,7 +115,7 @@ char *kstrndup(const char *s, size_t max, gfp_t gfp) EXPORT_SYMBOL(kstrndup); /** - * kmemdup - duplicate region of memory + * kmemdup_noprof - duplicate region of memory * * @src: memory region to duplicate * @len: memory region length @@ -124,16 +124,16 @@ EXPORT_SYMBOL(kstrndup); * Return: newly allocated copy of @src or %NULL in case of error, * result is physically contiguous. Use kfree() to free. */ -void *kmemdup(const void *src, size_t len, gfp_t gfp) +void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) { void *p; - p = kmalloc_track_caller(len, gfp); + p = kmalloc_node_track_caller_noprof(len, gfp, NUMA_NO_NODE, _RET_IP_); if (p) memcpy(p, src, len); return p; } -EXPORT_SYMBOL(kmemdup); +EXPORT_SYMBOL(kmemdup_noprof); /** * kvmemdup - duplicate region of memory @@ -567,7 +567,7 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, EXPORT_SYMBOL(vm_mmap); /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * kvmalloc_node_noprof - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. @@ -582,7 +582,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node(size_t size, gfp_t flags, int node) +void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -604,7 +604,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node(size, kmalloc_flags, node); + ret = kmalloc_node_noprof(size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -633,7 +633,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node); +EXPORT_SYMBOL(kvmalloc_node_noprof); /** * kvfree() - Free memory. @@ -672,7 +672,7 @@ void kvfree_sensitive(const void *addr, size_t len) } EXPORT_SYMBOL(kvfree_sensitive); -void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) { void *newp; @@ -685,7 +685,7 @@ void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) kvfree(p); return newp; } -EXPORT_SYMBOL(kvrealloc); +EXPORT_SYMBOL(kvrealloc_noprof); /** * __vmalloc_array - allocate memory for a virtually contiguous array. From patchwork Tue Oct 24 13:46:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F351BC25B48 for ; Tue, 24 Oct 2023 13:48:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 492816B0284; Tue, 24 Oct 2023 09:47:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F08F6B0285; Tue, 24 Oct 2023 09:47:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2449F6B0286; Tue, 24 Oct 2023 09:47:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 07E5F6B0284 for ; Tue, 24 Oct 2023 09:47:40 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D9E658027E for ; Tue, 24 Oct 2023 13:47:39 +0000 (UTC) X-FDA: 81380482638.03.A3918DA Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 2B7D71A0002 for ; Tue, 24 Oct 2023 13:47:38 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4ZGTLOgy; spf=pass (imf19.hostedemail.com: domain of 3-co3ZQYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-co3ZQYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155258; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OUWmo6RLkaXHvLlOp0/KP9HFW8MtULS71POlQ8HMTCI=; b=iT3wOzQrNfC+LqhB5hWWfxf7qxVpVe+2CfYEfMr8DPa+W1cuDmPqGcae6fzXoatlGbvYPr e23B8hpQiTF3MowdhU6XGVSFoIKXhcrRcr8d24t0hcYm/wZ+c2XlhOam3SXTsdQiMl4aYE kRWGqarICIyMiJG+3K62xL10qDbVozM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155258; a=rsa-sha256; cv=none; b=nmarvL6mIIN+YPUk0rA1LS1UjHsMcVHe+ftQZnlthAQdWZCJW1HteKWDfzOOfyRdC82D32 GaCNKTftoq/AVZx39XsKBMZnwJBXBfb9yP0LdoYoO93D5z6cNXgqShb42PNhj4GyIEbHgS W3Zdh3yhgcuXC3L9q1s4H2L9uZoZLnY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4ZGTLOgy; spf=pass (imf19.hostedemail.com: domain of 3-co3ZQYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-co3ZQYKCJ4QSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7bbe0a453so55617587b3.0 for ; Tue, 24 Oct 2023 06:47:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155257; x=1698760057; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OUWmo6RLkaXHvLlOp0/KP9HFW8MtULS71POlQ8HMTCI=; b=4ZGTLOgysRoKG2TA33KSz92pt0Ka8uLrviWZSvYFHu2wp35HTQbCN8oSR89nMgsNN1 SlxffiJOQMO52sC08EooR56g9djLJiX/LD4WSY9+eN4iIZCphtSOvyhpbdhieh2ZUPqQ /vxqdgiKEr6L6ve1IkQUHHasPf+4XMCGW07lYA/sWIUrUUdSF2YB2yDNJVuE7hS5zUP3 EGjkmondPlrV6tKHXhWHniMsDgTCp0u+2bn1nsFaVB1AlaXcoZX1KYL3JQcYaVKFzb/Z 6oKnBQVPCdBI8zppGkpKb1RRgQiSRnljcKboP80JXgdtliX6cbdA8v+YsU8Nm1gkbb90 6inQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155257; x=1698760057; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OUWmo6RLkaXHvLlOp0/KP9HFW8MtULS71POlQ8HMTCI=; b=VKo6vlYF3OcbEx/Wz0MQlYVJJPCbjzjjwKFcClTepuuU047/+vzkZeXyJNklwklLyE uvRKbr6p1qwsLIHvpPp0tJQpdbhpfEWazPeLu9lrT9nzmQGqMd274qtelM0BeGhh3zmo yQB3+yy+86tBD/YF/T7eEFqEYjOS2sSpzfNNEUAIuroH4WBjIin0xYjjcGoiBy7a8oBO +QNry/Xc881GxWoHPDsRIRyJ2ifrHoeKCMAq1+RYVG91nMCgJojTkIQUNR9esedylJYA SBWfWBfvp2Gld4EliTWguXGmbz1pgIMGTIx9i0KDhIUmngfL/bqqrHTEe58wzQyYi3oE JoCQ== X-Gm-Message-State: AOJu0Yz3CnSLCRbgJZ0HwaCyn12V0i8rFgBDGpDvX6Zn7imV22wDJ7KC o4gvivgEDPMo0BVGJKm8uE7EiDfrN8Q= X-Google-Smtp-Source: AGHT+IHQ3VEMdDHth6teCnTo9GNQ4j4PajdUPKRUHryC857gV5hBaW9Bct9jFesWnWbFdQlkFZvB05Yolzc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a81:48c9:0:b0:5a7:db29:40e3 with SMTP id v192-20020a8148c9000000b005a7db2940e3mr273153ywa.7.1698155257303; Tue, 24 Oct 2023 06:47:37 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:22 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-26-surenb@google.com> Subject: [PATCH v2 25/39] mm/slub: Mark slab_free_freelist_hook() __always_inline From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 2B7D71A0002 X-Rspam-User: X-Stat-Signature: ohhif5peuwb1agu5hyxin3puqp495idc X-Rspamd-Server: rspam03 X-HE-Tag: 1698155258-602253 X-HE-Meta: U2FsdGVkX18dR3dIZiCfU6yfAVZ+78qYhd/5CyQvF6KPH7aKpQrGgDox0up418qrKCC2foHPAoZXBsWUABXdVXhQqUQ8Xq/kdxkXhArDpB0k/e89EsvrmQxvxvg1JxJOZ3bxqnN7++JLHuUj8mLCRMe5Q1YSuHfopqFJlbq11wVxPqG6nDOL8ds9gOIFB6qxBCIf2lLP4t/I7ukixIzgmd3ehVcE8A+MofEE2UMjwOU3R7otk5GXV5+GQQsLCbgHDEXIiTzSBuAG5CLnW+0qSuQWy9n0vcEgeQIV25eMxxLtSCHPtXHxlVxNAjXXUHROueOf5eb06njiCC1pDYqbfpDov7F5SIsFyQnXc5xgBRfUWD3jMhoRoqgyxc6PKzFYlL6XHNuqVDTt97/DqePwJ/ff98L7p4mj8FoL7Aew1x/y5nRn3xjjtsYuGrLCfVb/McHkXbJaiW6KO2t23a5GXvyPZBdou9vP8C92wfELNfDtdeajY7DPXeVvzFVA5EAqyTMS4MYnPrwWJqA/qtDH7jqnpe3yqnLj6jWbXCWKf8BHFFSLD4jGNdNWMByKV7UphhrXCEuTwvhIKSu68bXbYGMCEZ3XnJdT9QtBJfw6sHsR7BSqgkY3fwt/mrjLfVnlZ5tDcbXM+mY9IdRfwfMXV4trRRKBK85mZujElSsTfo/wTfyOT4Xaa8PA3cY6cwF5qsaKy+HHKwCq4tV/9jZFaFuzq70snu5HLkAC2bhT1d9zo2TqHUvPLFLabfyfT5412ZRxwTK5XkqN2KNldXgZ6v1AbIQoaaZU5w8lOmN3TRQMjmJ2yzRigtNAdlePaMPi1aMeWWkvKOUrVODP1ZC79oklKhUIdUpJufWT+UN3AjZD+9QD5F8Gu6Yfuj14YR+pjHBGIyDtaFL9DFZXaS+bBosnNVmDIBeVYbmVW4sR80tsRqVmTMMJKC9ltAgP8+DTLoMeSNyD97PshrApnKG UjXNqoLS OYA34xf1TYkiiWGxlbHdGx96oupGAua74Cgx/dMu1mDasiJN69IMwpGNZxeFj+IJTjkb4ovMYM8J/68Bp+9VHzfIwpytKsvXDpQrXThbVDjTpQTjKaz8rgrz47j1mkDfsM74tdxuDkvMoIR8/Uemkzaudd2woNkhhlfSaXX3KuZFpn5USy7G8L9QkMic9jlNyroETNS2oohFCgQHgTBpPH9TYRkxq4vE7xuiaieTeQYpmahgvLitzMeOEgpZZH3LbLRLc5bk9OywpbvzCZHlywrWO9zGdrtbLvkvOS5vAy1zwUAcexBiCRWZFfl6Xwgi7A3aEJrEogn3tNykDMFxuofLdJxD7u8rRf00Tc40DMzpQYnbaO1IowXcMq2GjJfschdiLKHfxgRpsYUiuwPT6q4aoW4ZHHk8xlYvuvlLHzHubtycbts5XV+ZEFB8R1KHkQO8wFa+WT1ZqN+bSeZ5dLYdZSn1SM01ajAaI7UyfKOEyuo2VtkPdtK9EoDYNmIH6NrqMDb1Pn4DBG2BSWAaXAhFRGRlDWkX/jslhNYVBLVCF+EEOI9dYxPF9lEPvqLS3ikzMNsXqX0dQa78vFKiEhx9QW5ijFD+oa4Vr3zeIlpZzb5gxnpyEslSjXUoqf8vJHSXSyGb3VTXnLrM0nDigncdcPXH+hGehSMCGAQynrCsdeeV7oqeEqtuhmMm/T5EUYtTTaJOQEfVYqIFnJmThuvXt+CpICXNeC0AV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet It seems we need to be more forceful with the compiler on this one. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index f5e07d8802e2..222c16cef729 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1800,7 +1800,7 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, return kasan_slab_free(s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail, int *cnt) { From patchwork Tue Oct 24 13:46:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9782BC07545 for ; Tue, 24 Oct 2023 13:48:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD08A6B0286; Tue, 24 Oct 2023 09:47:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C59046B0287; Tue, 24 Oct 2023 09:47:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A32936B0288; Tue, 24 Oct 2023 09:47:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 89A596B0286 for ; Tue, 24 Oct 2023 09:47:42 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 666C8B5D89 for ; Tue, 24 Oct 2023 13:47:42 +0000 (UTC) X-FDA: 81380482764.01.F2DA27A Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 9F67612002F for ; Tue, 24 Oct 2023 13:47:40 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=U+pQ+mEh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3-8o3ZQYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3-8o3ZQYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155260; a=rsa-sha256; cv=none; b=rWELcIBXTFsPi+VFzVlT8e83J0zBufgbBfj1BPKXVLIGi7gFK5b6eT38P+qYdnkfP8WvKv eDgmSr7vDuBZsxlv/Ccr28aslx8j3OvJtaADhUU5kxk7BrwqCqmei7e0Inik1VwRvVP6rj SlklrUMLD/Dcqjbh30XPy02m3luinF4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=U+pQ+mEh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3-8o3ZQYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3-8o3ZQYKCKASURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2MflihaSbuQmiTaEEI/kjjxwB9AeuC8B0gJ477WMkrk=; b=xBXfM93SK9dPBgc6L89igpnrKbk3G7KINmgNGaAgzw0YJZ7qOJqXegY5ZNqzyFPtjfqbg2 kZAU1k2VzlKG9pOzBs8l3VL19Q/otTxhee9y6scBPX9UDXVMigWnRgduHL0QQ51qrNHesg WYoD9tlM77s5UOJx4fyTn4Fg0YCRHDI= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9bc6447193so5058817276.3 for ; Tue, 24 Oct 2023 06:47:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155260; x=1698760060; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2MflihaSbuQmiTaEEI/kjjxwB9AeuC8B0gJ477WMkrk=; b=U+pQ+mEhMs01sCatKwa16gWy5GS0XeCZn6n6m848PJM8QW8GFOFlEjTKd0NQyZSjOh Btbh0ODYarmS9PJ6ERuU9SsXBO4xA/JkAelQSrUDqcJvSTLNNFUimeBylbST3wyZ4glR VG4Hxk9vIOVzCW61ZJLLq8KrEkrY/wBv2j9H2DH6CuiReKC21X0tQ7zu94M/v6fIKacv 2gV1mjl3DSJW/htPkkC7poybms0V+W1YA0yRUQjNnegzIGr1nz8cwmK+jY8StDnuR3Ui DbgDTxPlzQO8pEMUBOcCGAMSTXXHcMnOpbvg2Zmr/U6qqpfRCMe46IXvQhgdBapKOEGb jfxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155260; x=1698760060; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2MflihaSbuQmiTaEEI/kjjxwB9AeuC8B0gJ477WMkrk=; b=WAEZUoPcOL4C4oPu21+ZL8xZacrW6vXJvjGlP0ejmAcD0sOQFngWtBTiZN0gZZxPIh B1SHkLi5RMQ4U5svuzJdb1Cr7cFU/uA74irvkJ6frApdcKOll6OGG5pNFusbg+m3WIff KTSLW89SUooqeEHti3xye/Ht517IppuZ6W+lKzOzStnCr+aLKlQvpjuQCfV/wCBCsX6c SZHaJQ6bEdvA6whenaQYCPstu1CCfTrSXhJ5/IGD5a9kNMNfL7go5jJyHWyeFsAU70Es +fZ7/rDipLlMrPxgg9CJLOoHiqby+U4adM7/R4wwis8nWxMn3zn7Xhq9KINVJUQ/Nxnj 0Rsw== X-Gm-Message-State: AOJu0YwJW/TvrPS3Ajki2dL2rjdZk+M+MK8x6GgW5sTxC5QOj+UjWNNf 4eW6YgLTDZ8yfqis74D4EdiBZvWGdh8= X-Google-Smtp-Source: AGHT+IEn9RJ/YYka9nhFwipeWEC4kTkdUXBs4fb0JEteWf5skbl81x+dboQnU840QIKuLVpVMJxwXpn7baA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a5b:8c1:0:b0:da0:5452:29e4 with SMTP id w1-20020a5b08c1000000b00da0545229e4mr18077ybq.0.1698155259614; Tue, 24 Oct 2023 06:47:39 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:23 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-27-surenb@google.com> Subject: [PATCH v2 26/39] mempool: Hook up to memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9F67612002F X-Stat-Signature: js8788k3fy9t8beuaddip46dcgy5mfzx X-HE-Tag: 1698155260-986323 X-HE-Meta: U2FsdGVkX1/dmC0wksog7JvZUYlqPXPfkJz3oPHsOMORqq3dFYwb8Cf4AeMwg8ffSdIm3DUxJLaUaqv/c0YvSI6VlL47+zzrHOXDzQRkralIezoXuoxicaxtFqLL3qn4sa9eOEziPaLw9vRHT4QnU7JBCD/Nh/WhwAyVrzs+0Z1k9UxFGvIPZIlvri/TOErU4BDpW0VSQW8/NpKPwKM9+5WQeyek4nzd7dci4kOd4nu+AnEB7iO18S30l3NrKhqFVsqGHz15FPtlirzuMOnC1rCSEvbgUToWwWDy7TrL868EDHoMGzwD9qAevuEv9sajTzl1yQUyaX9RV3/kF+FtuGkYYuDQgQ2LDeoSIRj0wbGfdTK6Q33ytkKPhWN0d9jH9tPrbry1lvuOIcBKy68Z17g8I2qeFqgTSppfsjUugjT2UTcCK8LWoFpe5SXMiTghlbWV3BHpGoAPYk6ozRfCWi1n4/uai0iQGCbZPZTTChh6hAimigRHnilDsSeFMWh3NrBMSqDlL1ZZXgKgDdpSoMul0ujJyaYeZEbqyZEYvWe1e/PEobfgTn8nDkvW28KHBvjH9TxPOyegJUzVhnPVjRCeB7R631YvYE3oY0NKvNTH/5WUUvHF9v5POPgpmFcISTLcRp0Yqvi37yirQqc9EKUi5+rqilrs2sJJTNohL+uBigxLBcHozCV/anUM74euBvoQTn3NVfwQFNZT9T5rKGyt8LihLBf98Dj3m9bTq/ExLiXweY/TJ6z+bmHn8RppRp3HALbgoZkoWAXFQ49SlVpO83SQgFDBRy1UHXuBmEa0tCVcPkPD0Cjr/I3IDfqxVIxaWcn5EVGoNwfay669qyh4ba21FVvm5VR1mCEB3xjjJC0sZKG2IkvfyJAJvaAeVg8kLxH2p7S8m1VSAKLrhxbyWnPsqIO8tBNpzEqZ1lere0E/jDVuOOX2IqliGiZXK783D4dDiNybDsSdGNF wRnGPJRC f+OcWOtr6j7nZyvf8PkfSiGFUEX2LaOSTYSvNzEej1XAJ6f1q+DGOWCeF2EMLfE4Eq4QveuyKAcGEIknS3YT79DFgA9wva99SJ6kYa0XyqcqPCyd2zB/VRUyixkCflVPEezlYhA7TXXFHztg0aS354NQOXhm1Fyb18cflyz73yN6YWg/iCQ0I+fWtiPC2xvLkgi2OLQxAnvqVtt8MHotbRmIxsM6f5jYoWnMVOJQpv35Fd3LjLJmrDIiGyzYPKEUpLu1daL3WyFi0nf2mI1fAuKj4QeUcB9Vl405KsT+A0WVGmViT+hvEuEqlGMpTwjO9qbRoQ/Gn6q6gmUIFtvwY1z2qFmi44MevJUvGWMKprjXFy9D101QTPgQBwpEDKYwqGth1DyXiiHBQU3FaOmJtnnr0VrMd4RT3mh4e0qZm8lXjebx7+NwUqWi0ygn6fZPtHeBtZTS50BewZkTh1vKRw12U+q6urEd6VlzfoG235X4Gjt2tpCNFEarfshT7YmZq44wUAeN7YeEuk3TANjks80xo3clXEvbiFLmQ3/6fm0x1vRYnhjzuS8eRzIjVk70fbzT5s5I5D7s133Ev+LJGcl3SoypSVtqfPIJ+gddYeH1MJnZy2MOmJJ5VdV0REYNXXnlms7R97Ub3fJ2zsw6soKrqFWFBUOn/X4uw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This adds hooks to mempools for correctly annotating mempool-backed allocations at the correct source line, so they show up correctly in /sys/kernel/debug/allocations. Various inline functions are converted to wrappers so that we can invoke alloc_hooks() in fewer places. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/mempool.h | 73 ++++++++++++++++++++--------------------- mm/mempool.c | 34 ++++++++----------- 2 files changed, 48 insertions(+), 59 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 4aae6c06c5f2..9fa126aa19b5 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -5,6 +5,8 @@ #ifndef _LINUX_MEMPOOL_H #define _LINUX_MEMPOOL_H +#include +#include #include #include @@ -39,18 +41,32 @@ void mempool_exit(mempool_t *pool); int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id); -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); +#define mempool_init(...) \ + alloc_hooks(mempool_init_noprof(__VA_ARGS__)) extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); -extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, + +extern mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int nid); +#define mempool_create_node(...) \ + alloc_hooks(mempool_create_node_noprof(__VA_ARGS__)) + +#define mempool_create(_min_nr, _alloc_fn, _free_fn, _pool_data) \ + mempool_create_node(_min_nr, _alloc_fn, _free_fn, _pool_data, \ + GFP_KERNEL, NUMA_NO_NODE) extern int mempool_resize(mempool_t *pool, int new_min_nr); extern void mempool_destroy(mempool_t *pool); -extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; + +extern void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) __malloc; +#define mempool_alloc(...) \ + alloc_hooks(mempool_alloc_noprof(__VA_ARGS__)) + extern void mempool_free(void *element, mempool_t *pool); /* @@ -61,19 +77,10 @@ extern void mempool_free(void *element, mempool_t *pool); void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); -static inline int -mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) -{ - return mempool_init(pool, min_nr, mempool_alloc_slab, - mempool_free_slab, (void *) kc); -} - -static inline mempool_t * -mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) -{ - return mempool_create(min_nr, mempool_alloc_slab, mempool_free_slab, - (void *) kc); -} +#define mempool_init_slab_pool(_pool, _min_nr, _kc) \ + mempool_init(_pool, (_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) +#define mempool_create_slab_pool(_min_nr, _kc) \ + mempool_create((_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) /* * a mempool_alloc_t and a mempool_free_t to kmalloc and kfree the @@ -82,17 +89,12 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); -static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) -{ - return mempool_init(pool, min_nr, mempool_kmalloc, - mempool_kfree, (void *) size); -} - -static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) -{ - return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, - (void *) size); -} +#define mempool_init_kmalloc_pool(_pool, _min_nr, _size) \ + mempool_init(_pool, (_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) +#define mempool_create_kmalloc_pool(_min_nr, _size) \ + mempool_create((_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) /* * A mempool_alloc_t and mempool_free_t for a simple page allocator that @@ -101,16 +103,11 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); -static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) -{ - return mempool_init(pool, min_nr, mempool_alloc_pages, - mempool_free_pages, (void *)(long)order); -} - -static inline mempool_t *mempool_create_page_pool(int min_nr, int order) -{ - return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, - (void *)(long)order); -} +#define mempool_init_page_pool(_pool, _min_nr, _order) \ + mempool_init(_pool, (_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) +#define mempool_create_page_pool(_min_nr, _order) \ + mempool_create((_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) #endif /* _LINUX_MEMPOOL_H */ diff --git a/mm/mempool.c b/mm/mempool.c index 734bcf5afbb7..4fd949178449 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -230,17 +230,17 @@ EXPORT_SYMBOL(mempool_init_node); * * Return: %0 on success, negative error code otherwise. */ -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) { return mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); } -EXPORT_SYMBOL(mempool_init); +EXPORT_SYMBOL(mempool_init_noprof); /** - * mempool_create - create a memory pool + * mempool_create_node - create a memory pool * @min_nr: the minimum number of elements guaranteed to be * allocated for this pool. * @alloc_fn: user-defined element-allocation function. @@ -255,17 +255,9 @@ EXPORT_SYMBOL(mempool_init); * * Return: pointer to the created memory pool object or %NULL on error. */ -mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) -{ - return mempool_create_node(min_nr, alloc_fn, free_fn, pool_data, - GFP_KERNEL, NUMA_NO_NODE); -} -EXPORT_SYMBOL(mempool_create); - -mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data, - gfp_t gfp_mask, int node_id) +mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) { mempool_t *pool; @@ -281,7 +273,7 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, return pool; } -EXPORT_SYMBOL(mempool_create_node); +EXPORT_SYMBOL(mempool_create_node_noprof); /** * mempool_resize - resize an existing memory pool @@ -377,7 +369,7 @@ EXPORT_SYMBOL(mempool_resize); * * Return: pointer to the allocated element or %NULL on error. */ -void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) +void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) { void *element; unsigned long flags; @@ -444,7 +436,7 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) finish_wait(&pool->wait, &wait); goto repeat_alloc; } -EXPORT_SYMBOL(mempool_alloc); +EXPORT_SYMBOL(mempool_alloc_noprof); /** * mempool_free - return an element to the pool. @@ -515,7 +507,7 @@ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data) { struct kmem_cache *mem = pool_data; VM_BUG_ON(mem->ctor); - return kmem_cache_alloc(mem, gfp_mask); + return kmem_cache_alloc_noprof(mem, gfp_mask); } EXPORT_SYMBOL(mempool_alloc_slab); @@ -533,7 +525,7 @@ EXPORT_SYMBOL(mempool_free_slab); void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data) { size_t size = (size_t)pool_data; - return kmalloc(size, gfp_mask); + return kmalloc_noprof(size, gfp_mask); } EXPORT_SYMBOL(mempool_kmalloc); @@ -550,7 +542,7 @@ EXPORT_SYMBOL(mempool_kfree); void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data) { int order = (int)(long)pool_data; - return alloc_pages(gfp_mask, order); + return alloc_pages_noprof(gfp_mask, order); } EXPORT_SYMBOL(mempool_alloc_pages); From patchwork Tue Oct 24 13:46:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434596 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3494FC07545 for ; Tue, 24 Oct 2023 13:48:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04A1E6B0288; Tue, 24 Oct 2023 09:47:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F13826B0289; Tue, 24 Oct 2023 09:47:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8EA46B028A; Tue, 24 Oct 2023 09:47:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BAB936B0288 for ; Tue, 24 Oct 2023 09:47:44 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8E2F7C057E for ; Tue, 24 Oct 2023 13:47:44 +0000 (UTC) X-FDA: 81380482848.11.8BC0CCE Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id C7EB81C000D for ; Tue, 24 Oct 2023 13:47:42 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qpUmIIG0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3_co3ZQYKCKIUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3_co3ZQYKCKIUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155262; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wChZbZdPrjdzw8yNa1DdlAJuF9I/RLt/gyhxqa0kGe0=; b=s4IJzxVdAGMmIc5eYgDQgQWL8NS4eBuLvaZERwAnv/c0EJ9KnIi4S591+DRpzSWlNvXpEo v8/rvFC/9aDslTs9yOL78Jii+5TdEgWRoTi/a8p7baL49wbw8OCzmw3cZU5SVyFj3ZUPH/ AXgyh2kixmnYtGFyM9rWNUgLjwB55Sg= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qpUmIIG0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3_co3ZQYKCKIUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3_co3ZQYKCKIUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155262; a=rsa-sha256; cv=none; b=w3VInNfAA03qI6ZQ9N40dl5hKGOwpIOEFPJSqo1VA6MWFqYAggs6VTJCPcKd5IHlaIApxZ NxgZ3GVREE7UJZTDDxG4AIF/DBnT11YWkuJ4uPMzCuShH0gEGv8YBCbx5/VDPYlOiVGljQ YOcRgRzr5er0QuZr/K/W4577Tj5Mfzk= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9a541b720aso5409415276.0 for ; Tue, 24 Oct 2023 06:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155262; x=1698760062; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wChZbZdPrjdzw8yNa1DdlAJuF9I/RLt/gyhxqa0kGe0=; b=qpUmIIG0+5v/4qZPJW2GZqDDkUQ6gtoczuXfJod3j7f9POF2V9t79o0W68sU4mbZLQ x6oAQ2cKp/owiMV4BOtGDu0aYhRuKMPUD4MhekMzZ1gEbi5yxsB83R+3e5yyVMvxN0s8 6b6YkCqhiJdParpI8b23OtQ+BkhLpBPv6Ym5aIRugpx7we/rvhZbYjBs6tN6Va20G4st K7SAwmIeS3IlKn57MAk3FKD3qg0up/P59TdP7eT0wfxBLZK8S2aVWKiDWwwtJuiasxkQ o2QvGfazNfrK8q5Ziij3OQw2IAbE8TsP6dm+PyzTmkIWpo9ppxefjssWeAMqFypTf1OW +w9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155262; x=1698760062; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wChZbZdPrjdzw8yNa1DdlAJuF9I/RLt/gyhxqa0kGe0=; b=U2BQTOCAsdjoIRGUCEooTpX1ZPYQouXxs44vr9XMz7d2AhxD9/x4akPcAKxyd/Xtqw Oab986zEK/QGaEfIgb8xSc6LJ0dI1jOvWv4sv9fNjUc/NFPZRRQwr8RtjSzaGPwDai0H VVH5du0nxyMZ/gnovyZ5b0XmCgKQfnN+cJy43yn2tMJPoC4SfXrQ/yceMAciNGwBq6Q2 DgClpxLlVTC3a3ouFAcZN4vchI/t38wq7dC9+QZZeKvfI5z2pn86Hy/+PBC/Lv1XG9hA 40XcJvmhQLWePuwEQUTCR7nFJ+LyHKN/F5CG6ltDiPNaAhu4FiMmzDKPR9blG9LeeDh+ H/Qg== X-Gm-Message-State: AOJu0YzRb0P5BiKdX/VlD5ucguO96eGrgMvQJbjIHhwjjpS3RHDwKG6I CTgglmn9SyslfaVHBsbCloaJCytKeXM= X-Google-Smtp-Source: AGHT+IGcrV5EmbtfxBae51aMAnIIsiAAzPfQg3R0miJy2XM7/Ak0S71e3wg0OW88a6RB4IGUgmC4AuntPsg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:7755:0:b0:d9a:519f:d0e6 with SMTP id s82-20020a257755000000b00d9a519fd0e6mr234250ybc.6.1698155261852; Tue, 24 Oct 2023 06:47:41 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:24 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-28-surenb@google.com> Subject: [PATCH v2 27/39] xfs: Memory allocation profiling fixups From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: C7EB81C000D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: wctfg95w41g9ai86ktmhqcuaj3cbrs17 X-HE-Tag: 1698155262-243981 X-HE-Meta: U2FsdGVkX180s8P5w+LRNN2/QDCU6VREogU2UG25u+IemctT4k9PAxMMYcNoQabFJ0chDf3ruqxv3+LbCETtnXUivLRygra7dyPcfLjEoNCnO0yE5fygmaXSUHk4PF26I6qDIrDxZWMW8EXrMobjAUbtGV0FJfN7kmqdk3FPHQ8PuKL4yWf6CJ6cE22XjsFXtuRoN2PtRPPWBPxLCX04tMIiN7l/r3aMEiy34tgb0lDcr8KnZnYUPfo6oxDtMWyiIm7zUt3Fljkh4pDPv/trtQj7Bjoz+yQq1ZUPBve+GcScv7YUKHQD1zy/LlKEcb7aVwDkUItx1CTsWQYjF0EAP5c7CigHCqUAGG5sFTsNfaiHfZahRNoXmKbzjKIQ5a98vuj9688WU/xnPA4CY87RZ9PTs/ORGAvRlCM1vQlhBhZIMugWRW9MZfyn44LYAAXub6xgLi8BFa3UdR4chd08UbfPBwFcjfRSe3DjBkKTvONXTGI0zGDaCvwbblTDre1jlpAxguSuoie8pAVIXbwSfxP8M8JdiNH/sD+5EXdQTgocPQv5ZJ9PnzOK4vKvm0vhxPL8EDYef3XOSnp0EIfYOpKWZZy55SqZeys6ERNu3VXnJ1jXJGBhleCnMFc0Ko4fn9n3w+PbKhsV22/c55MW4om21iZDoHBkIiUZCteIbmAKmUiNuXjKKpRcHM+iILR/mt0Rx4OuEZ1OhHVroU82xKnMl9jNkb5DTRPctOxrePnOrSMxcIBeCzmyuRSuKSmTOePSeARQCSguegxU2JHl2txPnnJbXpND3LUfw4LeTNnQsmkJluokGjri7a0eVYnhECYpe3hyelbXmuEYEEfAK8jLq8VstgCqhphzE/VzaU4US009caE/PbVnjJW64M0kT0uMmory7JxsDdJY1fbvymMGmGe0JDwJ7ioqTYyS0H5nTsp1vXrzuqnBcmQL0LhRNbwe1zD8u+pArFYlNDk uV0uq6Sk 8eBS0LADVj6Ccf1Ioe+YSDD131yeH7yVYU6thPQF2ci4FsRpwA9d4lG+vUiA8hOLUvTHLMt9AZyqjGROFxFLKvoMJPakULn4dJ3VYH2po9TUFPFSMApRUGFnRjM/Bb5Qqe6wU1FmermGtYjNSmYa0desLk+FVyMMHzZxcv/lLyVcKYGdFRY4qpKi//vdF8rN23tFmZhlXu3Dv2jL13mw4wKqm10l6E/JlpHV8U4sAkJ7PR5tNwL7LZKS7pR3ZxWKCKgINiL8rhEF5NAqqW7p5jnsy34POyLuesbW1CecQjVqtStm+wlUBmhDyuy85O+Idy318p+pMx3LrHo6RKdQ3EQryGsKxCBV72bYr7Ah/T7l4Bhhl9clG5azbkQ+QhgkCM24K+1AjA0gUxZi+SLjWcfDSr7jZazbGXT00caRmAWhuAqaq9wrBRIvHDxXwdcDv1P/xDWQbFmUqAajaizvtpScDBQOSEKGlzTO4h2tZvsXAg2ZEKUi2jGXkg+C3XlxL8yXR8nUpvHFLpMt/ctsCVy2TG+yQVkPFqHIt0qQDXSAqjSd+4xj/vOlD3bgn2XRKo6VmshZIGCY2CI/CENPs+1GKCa+Vgy0/UVE075YmXDCoPRes+AN5JWIwgQ8EchAHjSKtT4lIQKmFhq6Zp0L6IaaAe9l1XiecetscqBuoJxCSRyE+TC3z60V2ANSONv8huzdB/SspV/Luzo96/T9pv1NBIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This adds an alloc_hooks() wrapper around kmem_alloc(), so that we can have allocations accounted to the proper callsite. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- fs/xfs/kmem.c | 4 ++-- fs/xfs/kmem.h | 10 ++++------ 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/fs/xfs/kmem.c b/fs/xfs/kmem.c index c557a030acfe..9aa57a4e2478 100644 --- a/fs/xfs/kmem.c +++ b/fs/xfs/kmem.c @@ -8,7 +8,7 @@ #include "xfs_trace.h" void * -kmem_alloc(size_t size, xfs_km_flags_t flags) +kmem_alloc_noprof(size_t size, xfs_km_flags_t flags) { int retries = 0; gfp_t lflags = kmem_flags_convert(flags); @@ -17,7 +17,7 @@ kmem_alloc(size_t size, xfs_km_flags_t flags) trace_kmem_alloc(size, flags, _RET_IP_); do { - ptr = kmalloc(size, lflags); + ptr = kmalloc_noprof(size, lflags); if (ptr || (flags & KM_MAYFAIL)) return ptr; if (!(++retries % 100)) diff --git a/fs/xfs/kmem.h b/fs/xfs/kmem.h index b987dc2c6851..c4cf1dc2a7af 100644 --- a/fs/xfs/kmem.h +++ b/fs/xfs/kmem.h @@ -6,6 +6,7 @@ #ifndef __XFS_SUPPORT_KMEM_H__ #define __XFS_SUPPORT_KMEM_H__ +#include #include #include #include @@ -56,18 +57,15 @@ kmem_flags_convert(xfs_km_flags_t flags) return lflags; } -extern void *kmem_alloc(size_t, xfs_km_flags_t); static inline void kmem_free(const void *ptr) { kvfree(ptr); } +extern void *kmem_alloc_noprof(size_t, xfs_km_flags_t); +#define kmem_alloc(...) alloc_hooks(kmem_alloc_noprof(__VA_ARGS__)) -static inline void * -kmem_zalloc(size_t size, xfs_km_flags_t flags) -{ - return kmem_alloc(size, flags | KM_ZERO); -} +#define kmem_zalloc(_size, _flags) kmem_alloc((_size), (_flags) | KM_ZERO) /* * Zone interfaces From patchwork Tue Oct 24 13:46:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62FAFC00A8F for ; Tue, 24 Oct 2023 13:48:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1F3E6B0289; Tue, 24 Oct 2023 09:47:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AAFB36B028A; Tue, 24 Oct 2023 09:47:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D4A56B028B; Tue, 24 Oct 2023 09:47:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 751346B0289 for ; Tue, 24 Oct 2023 09:47:47 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 52252A0789 for ; Tue, 24 Oct 2023 13:47:47 +0000 (UTC) X-FDA: 81380482974.18.5A2E4CC Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 343EDC0016 for ; Tue, 24 Oct 2023 13:47:45 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e62UIc+A; spf=pass (imf22.hostedemail.com: domain of 3AMs3ZQYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3AMs3ZQYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155265; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7C50BM7p5C+vJ0clcF+/KvirWmdIP5XWEnNLFYgTdS0=; b=slkfA3RfmdXd7E+fkaJBDPmjkWsHhgVfo4o3EN3N7pFIdw7MCV9IrI0CfSssi5HGWd6FuG y72Del4rbLDZ8Gv4u0JZRZEdlD5y1C/2Fr/TzqY7KBOv2RStPZu4FrrcC7H12DWLAU9Psu BoviTLneiZcxP93JCt72RtP+iRR6zrs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155265; a=rsa-sha256; cv=none; b=Jv4hoRN7sGlCoQNLdxejUMuwzsnfncuN7aWibAAMt5yLvcNfxw5Qu5KE+gKtlFhUT4cEMx lRGFt6CxmuypAWdWZoSSklFdAz3Wxrrfkydz4ZDN/4R1S2Pbe2Dm+0DYIbUKPcFdtcmW50 FqWsDr3tO9qgU8geYKh94dN5X1+QFAw= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e62UIc+A; spf=pass (imf22.hostedemail.com: domain of 3AMs3ZQYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3AMs3ZQYKCKUXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9cad450d5fso5175302276.1 for ; Tue, 24 Oct 2023 06:47:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155264; x=1698760064; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7C50BM7p5C+vJ0clcF+/KvirWmdIP5XWEnNLFYgTdS0=; b=e62UIc+AMGoDOn9izjLXeELTItCnzuTUXOH6xkhNFUT8cqcn+3z2OKXv1sx4eROyRy D8vlsuJNaeQzoiK7MsS9wgC+u58s8HrohWdq2DNDYN9r30fRcwpWwkEz6bBOeUbimk/3 A5xFgD6jwcHcW8sujGIQhCRTe2RUO+C9e8PyH+WamcXOmY3IipAXTHSMEcwrtQpOsTJW yloHaNAy3JpkwkiLq5PR/U0DywHf2fv1dbnYWPzOZX8F0pxHQ262d3akz8XNKLmRYX81 qQYUuoFCH/knMIPq0MU16EZ7Ni5nfm2c8qb3acw0lnGU/cCaZZKr4dPiNY3DtZ+6ihn9 7fMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155264; x=1698760064; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7C50BM7p5C+vJ0clcF+/KvirWmdIP5XWEnNLFYgTdS0=; b=wdNNy0EZ3M4Vgkd3qxAVvWMNTuh34Hf0NEgCIg7LhhULYNC96xepO/00B2H0Gk8ZuD rWLzdZmUrOIvek22/k9T6qzp6fwCKlrXRjx09+2/TWT/QH3Nx7Rm2kqUkbq0MK7/3Z5d 3JTeGlmWzWTCt54dKKoUp+eryUYw7dQPLFkk7/Suw0PaOO+6k3ndFbrfEOiepOVfl3xr Gd8bbUlexdwcA4LVvdSeKoL7s2N92OO+B3lVJhhXeHDFeuVhusa3pkS3JiCgc1P9sint I08tPNNZeiaL+4M5tvCMSiRJuTSL08fKJo2GEk3kEhYCq85TE237fx3nNns4YVD657Av f/yw== X-Gm-Message-State: AOJu0YzR5bt+3E0amus/LD/bbah9UzgMxM28ZGt2fH4rhLP/TGxueavV vC6xPua64kAN7zdc/grHxlZgBPGAeho= X-Google-Smtp-Source: AGHT+IFgQR1uDgRKslLBvCkBHAk0T4yEqB1Qvs6h4RCMdoan4Z+72V6WbgepeSBkK71dX/nU/j0JJWPs+jY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:d34e:0:b0:d9a:e3d9:99bd with SMTP id e75-20020a25d34e000000b00d9ae3d999bdmr212803ybf.5.1698155264079; Tue, 24 Oct 2023 06:47:44 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:25 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-29-surenb@google.com> Subject: [PATCH v2 28/39] timekeeping: Fix a circular include dependency From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 343EDC0016 X-Rspam-User: X-Stat-Signature: i9tqowkj7pggu1kpcc7jo5iec96awtei X-Rspamd-Server: rspam03 X-HE-Tag: 1698155265-749964 X-HE-Meta: U2FsdGVkX18KelSv+UL2dkW3FG9gSHgap4fDwHhFYb9bS0kCA4a/sc4LGEBY7m3W+9Mf2NWtMCSiiDsD8YtG+Mt42cUAnDvRX4XSVRlYvs3fKS5U4CStBT75xV+jeXyKalKrMI0n8NQLYNGqkqiZwoeHgTJ3U7deAjTNBaIT7W/UuKDhVhk++JuER1WnlkhYHyShp4PXY7fQiiVv88WYxm6gto5XIXyFEZ6XSc7vuaP7M5wCC/WflCCCUvJSsdi+ufEsGQyzZ48cDFmUEm9E5ZNbd64LAadYT+ngx5lZ6APmXUvlMj1Pc3dcF+b31ezNleCBy6GqW9g7+K9zXk6zig8tj0JNP8FiVKm6YpSnSalYmMrT6hRBkBoQcXdVsc9n/WzlUEow7yyfcbIB472h5QkAgbHKyWHQTLtLFOoScZBq3p70Xy7N3y4s1IgWUavZSSflsscbYPI2ZqDaybO+JOwdWGbNFIrrCInwOqxh3D8eVgIzwezw2wdg9R86RmPhblrK8Sv04KKCPORa0hPM6m2cRhTJ5enTC87uHK0mFaCvgaRxWd4ObnDPhTsn/lhRVpEtdBD5o+Tmo2o+Ypk2CwiAa2uAGkp+K614kAozcKMhMXmoiNBgTMXcje0/dlD9zx/eVtrmf9H3EaF77LeuapUEZsF1Bma3zsoOOZGPVSbywbJe74X8xM7RKZnqb1NVXYmA0owhCmfJ9HSo6QzMj2PmKUYc8cDVprt05PXGtZE4DibcMmmJwZvvzVPCkGh/Lc7fHmIPeiVHGy0e7JcioShOHjLyEgf9jU9iTZLFcNJpOq9Q8aVrjhuFh35K4FY4YpMRqkuCCRV+3LuOTlmPbgV40JEGilyQG6bMc9DJeYQWIKk+pwo1sa4GBIcNvHC4gPeMgiw6VQ6um5JisRgi/gqdOqGsVwTD/pCq5eTHPeCN/JgwTCpebJhiEe+3jayelvpig3L2Nmm8P14+bQf w1fW65fT wugxenkWu06zHlhA5za81X3D9AgkDzVfRq5VECPQmyjzOztZsQdjrdLOXzRYk6gyEVy/Pz2tM5wqvHlg4SdFBAVhWGa2uh6ERGAme5U0u1JlCx1Zy8e4oMljhI9l14iv42w5vUBFimFhwPldOD54OFwMHi2/dH8ObFZa13sPc6/Xlm9QDJinCNMz3t5QimRmpHjyBmPIPkbusJdqEkto5ljsX04V5V2AO38vdGznJjLH5mF9o+rRa2sgX3YP3z1+21cgtCeAATWG3gRNi2TVQtgHblpA9dA58itVn3LxjXp72rjFor8Mt8dVjVooPaADTgTm1R3nBqlcd7++dbOQmG/e3P+vi2pggMx77XS1OALkxwrZGTvMsgnSbDVxYwnhNOLAbE4QJbFgfMqVupzOfhf+hzpL0xOLup6EeaFKEb5HCbkNoN6o+SwXtXCzSEn4i6QVnPT9PhzdzL7ArbQjGqWSXuAME2kuCVoqsS8/sSJGDiiT9Jt+b6GOiPqIR3HENtgFOJtYX7zXL5zPr/2mEmPQiWgATzg2FRPWKOeuTf6AraXesOwtdUDbWGJ/XG9cI3o1QPJLsqJyCVNdAM72UEgIhEumFpHlBMXMsmEqqF1qQQppSzcZCnFf37sOmZ/zv/MzLjky3WLrFtyN5lVVLexL9dZ7r5r1D48Cf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This avoids a circular header dependency in an upcoming patch by only making hrtimer.h depend on percpu-defs.h Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Thomas Gleixner --- include/linux/hrtimer.h | 2 +- include/linux/time_namespace.h | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h index 0ee140176f10..e67349e84364 100644 --- a/include/linux/hrtimer.h +++ b/include/linux/hrtimer.h @@ -16,7 +16,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/include/linux/time_namespace.h b/include/linux/time_namespace.h index 03d9c5ac01d1..a9e61120d4e3 100644 --- a/include/linux/time_namespace.h +++ b/include/linux/time_namespace.h @@ -11,6 +11,8 @@ struct user_namespace; extern struct user_namespace init_user_ns; +struct vm_area_struct; + struct timens_offsets { struct timespec64 monotonic; struct timespec64 boottime; From patchwork Tue Oct 24 13:46:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434598 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6423CC00A8F for ; Tue, 24 Oct 2023 13:48:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79E276B028A; Tue, 24 Oct 2023 09:47:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 725A46B028C; Tue, 24 Oct 2023 09:47:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5533E6B028D; Tue, 24 Oct 2023 09:47:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 38AE56B028A for ; Tue, 24 Oct 2023 09:47:49 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0A238B5D8B for ; Tue, 24 Oct 2023 13:47:49 +0000 (UTC) X-FDA: 81380483058.30.D035E27 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 3A4A740009 for ; Tue, 24 Oct 2023 13:47:47 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TYv+xcNM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3Ass3ZQYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Ass3ZQYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155267; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QaUscnVR3A9ivKzFSXLwhZ9WVAKd01wEmSHK760WXQ8=; b=UXFyWZul2wTchgRMqA+84eiTccbfPLHhuS7gYExcxp35Ns1m6yTW/jhPHBxNDY+88dnSvF /5z0JCnNRC8VxoHU7cifhmOAVMikeM+TmuHaHVGLkdGK5ujX6Q4XWlT9i1fkN7ckq2W1XY 7mnsdzr6+TUEQFFGXNWnC/x4h8chMWU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TYv+xcNM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3Ass3ZQYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Ass3ZQYKCKcZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155267; a=rsa-sha256; cv=none; b=ZOPfyuQAwbKKwcqm1SW5HdObyFH50yilsLQ3yuSCy8xAOZqDkarTGTVL64F2AzyZ6IEpGB LUf/+fcNMmhPeoHQw74+Xrlk+lsw4sqLlL1MLKCie8HpVK5s8fNMd9Ff/O4rzMYsAMR37H 39AE3WmubJb93JaDq3UH4Origrt+C9I= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7d261a84bso61833407b3.3 for ; Tue, 24 Oct 2023 06:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155266; x=1698760066; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QaUscnVR3A9ivKzFSXLwhZ9WVAKd01wEmSHK760WXQ8=; b=TYv+xcNMN+69dRJOSeOZdX4R+RKtLBuf4pquIcta4LPBr6iahymeEp7y9NpO28zHMO 2uzow1gLnoe7QDY3S+WTzMdCh/eP/jlicJXYAKEhDWPJeCBoIMED8ift6e9rIF0X4jCI FmmyC6qkrPWvPkdg8+DPauJO8XcOz3PwtlppnvwuhVJV04MIJX4Fg44FLDfHWUXZ5g/g uHxoeuGO1D+hJWHbHmTBNS2mEg7uYD/1YFJuIYuFRpEyH+Maia3/ArGga8tLDKYv4lr3 QReDRF9hbX2iB2V/ICB9KEp8dGqsaOLeEYbUcA8IYnfQBCIxzsKXJKWOAIPjoGXJHLpy UjJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155266; x=1698760066; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QaUscnVR3A9ivKzFSXLwhZ9WVAKd01wEmSHK760WXQ8=; b=JmfD9uRK1CeKyyfsIooTdOXYn7ahd3dgf9VSGlmDcX3BeUaCtiGjfPMkHvazM9Frgu v6L0lwekogPfBRui39UsbG3wpPVuddOMQIJEM4fFOXU3VQaR3MCPeBVN+36V0/+Qy6Rm O+f/0mstiSei+NDLhiUAXXpxXnuy3Gz21vCwAVoo6Kcchc1/bIna8SBU2mn3wtARvewr Mi7HbIzQj9xAFJetmDOYVhuH0bNatysBnkPjmt4r2ixr0xo8PjBbI/Dp7BEa3smxjPPQ s/lLlXZzhsRD2SNp7bsF0XNmw3tNsu/r5nl9v5RfJicNQEH/f1P6wITfs9EUlxv07Z1A 09ZA== X-Gm-Message-State: AOJu0YycWA91ck6XyOid3JzUr0ouTrR9Br+Ac0U001tOStiApTm2DHAx 8loJRfhvfo7WmKA7rG7K35wMaRzzCpA= X-Google-Smtp-Source: AGHT+IH1Lkjv7kiNBXwNvQL565BH4DAdvvh1g2aWFQE+hKVCSXY8/YuVaym3f5veoHBOTW82zGAoDjJNWeM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a05:6902:105:b0:da0:3da9:ce08 with SMTP id o5-20020a056902010500b00da03da9ce08mr35592ybh.10.1698155266214; Tue, 24 Oct 2023 06:47:46 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:26 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-30-surenb@google.com> Subject: [PATCH v2 29/39] mm: percpu: Introduce pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 3A4A740009 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: p577tro81n7z75ke9s4fh5yrs3ibe1bn X-HE-Tag: 1698155267-846924 X-HE-Meta: U2FsdGVkX19/GDp/wqad6VYNqhaKFNyI4O1fBC4r1I2BzXynvoN71tl23bwfw3SYeB77BLCbA0h83wKeygwxHdFj1y3sv2pEkWf7+BCj712eYZwzTRV/6DGsjUYRHoLC6E7k0xZ0MDVsogtUsyOkk1GJgF4l8I5Yxogzs/6KAFdEXmd8Fq5F2WwMIkVAQQ6CN/tpQyTBRBeZSLtzf6TaWFjN+XqsMEZ11mct2Blc1tZvMq43dVPpHVVP+oY0lM7GSKdaZEVCDB0MQnlH7EGOaiVYgXkKTN0weWSZjFvCLlZpAbr2i4qOT0szDnq4nWUOQwaRcuQqXgsk9dkRicLIdQkK4FG960LCQC77543jTqO9Pw2xNDV2DNg7/C8fkFy/yGMUFN4+fFENoIgzIrNmvnkjEhAy5dOSE4qkhigrRjUGxuG1RlOGVbVBBs55wtp6cqt2rAM7GMjLf9hFhMkb/9YMKt1O47tig3GXuUTrqycZgjgqm9Pil07DRbc0tqquLGySHAxb99kTBq3PjrdlV1q7bn/9BAvQDxRtokCBy+66/BrfO7UAvc6YaSmIxBGPDDLWX/MvaV0yLeYSl210+ADDeV90txmecZYfnCT804YViSgNtlBg1bvmDQ8TqntWFSD1/nZMPIrvp1LJjdWMhNWBZVKUasXTzFbCWlchMgRcwG1RFxIP8hpGM+UMV4ugYWdB6hx2Ui6z8GV1J4QT+QKjfxxlYUOG5AeC7BEUWLL14JAmF4qAh/sMHb3I5avCROs7C+TQumHOW6/8My5qBIdMTNUU88Emw+9kIieIpAi3bVewPy/gzPR/c86iRH6KpHxRBbvVw20Nx4cp8DUBSSPTPdCXhjufVmqiLXNedSeHVi5jQd64LroCbGGHU0w4WSJ2XbvgJA3kvYbVOxo7BrQKIv0g4rVlfvaSKTScDNskZ+vnIuDTI1ZtJnyq/pvTUnLHslWXNIuzSOk4p+n RYA6QFID DOVvh9yt2JV47djdLCHX8PCzoZnxVl2pizPx9sHzhpLbVRcfXUBdc1Ri4QkcAU2YsJ5cG4E2/qDIVFPpBkpq60D5RBQFLN4v1+3dOqHxSvaFld0E+5RGHTIrW2Ujzh1o8GVa3IoMyA9MVo3fyxMY52bw3RA3JLJKKQMtzldf7F76T3pr7/X3H7WFjEJzz/RiyaDYSOz7OL8DGBlkxZCSa6EtTvGsY+wrEYj3bXkKagN59k17RC++lq8zKIsL5AwtEccIf7V8ibQTBx190MvXSNPxWxnx1crn0sb+vEIwK9Oom3Z2nfHPfrvkIC8rfolSwmHEqiSENE1GAvTwnLCU6vE22W7sm6/Wv0V7rTKCnzlnKVO4kNo2e9fEGwQS9/J8BkUehqcmesCdzY6Mf5RdArWAgV2g8tWBDz/CH3C+I4Os6rXb4FoQDo50cgIcJEHJRmT71c2Sazm1uhJKPusL4emQp0IQUjjOdO/gLDnF2xOhkXvrglYwcyDm/LhpDCCzkQAxuLKJeZaXk3m/motYM7VmJHi562iWeiFIKmLxhO+musjdxYncQ7kNkNI7mibuX0oROfUgPq+8pySYtXnO5I/ax0ct4vWZFVcM+PfzrKDoRnk+EAT5lxSZYY1J4UGINgRfVDgO02jwpr2rNQtaRibqoojhZgXgIhiPpLJb81+qb9DuSGCxZqEI4QLSHIc2F6s8+/jtSndUT4Ltrd9GczFFst9i0B/GcTaetd9GuSmGHn32wMoMeZyYU6rgRu/RvHdSHROrs9IDitBAf+008WIyU+eVJmnY8nwtIoigJDog+vuz9k4Raw7ltA6dxStKg9IePCWE/Z89QPTsqKIqkOayy0iselvYoJonj4sFt2OhhbfPIK+M2BenqAh96CnISA+cHnHCz2kTxHiBJLfiutei63VxbuDvis0d9CPZJne/7J7ao/nYcGn8ykA+l4iMff3eT251zAMchT8s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Upcoming alloc tagging patches require a place to stash per-allocation metadata. We already do this when memcg is enabled, so this patch generalizes the obj_cgroup * vector in struct pcpu_chunk by creating a pcpu_obj_ext type, which we will be adding to in an upcoming patch - similarly to the previous slabobj_ext patch. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andrew Morton Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: linux-mm@kvack.org --- mm/percpu-internal.h | 19 +++++++++++++++++-- mm/percpu.c | 30 +++++++++++++++--------------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index cdd0aa597a81..e62d582f4bf3 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -32,6 +32,16 @@ struct pcpu_block_md { int nr_bits; /* total bits responsible for */ }; +struct pcpuobj_ext { +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *cgroup; +#endif +}; + +#ifdef CONFIG_MEMCG_KMEM +#define NEED_PCPUOBJ_EXT +#endif + struct pcpu_chunk { #ifdef CONFIG_PERCPU_STATS int nr_alloc; /* # of allocations */ @@ -64,8 +74,8 @@ struct pcpu_chunk { int end_offset; /* additional area required to have the region end page aligned */ -#ifdef CONFIG_MEMCG_KMEM - struct obj_cgroup **obj_cgroups; /* vector of object cgroups */ +#ifdef NEED_PCPUOBJ_EXT + struct pcpuobj_ext *obj_exts; /* vector of object cgroups */ #endif int nr_pages; /* # of pages served by this chunk */ @@ -74,6 +84,11 @@ struct pcpu_chunk { unsigned long populated[]; /* populated bitmap */ }; +static inline bool need_pcpuobj_ext(void) +{ + return !mem_cgroup_kmem_disabled(); +} + extern spinlock_t pcpu_lock; extern struct list_head *pcpu_chunk_lists; diff --git a/mm/percpu.c b/mm/percpu.c index a7665de8485f..5a6202acffa3 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1392,9 +1392,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr, panic("%s: Failed to allocate %zu bytes\n", __func__, alloc_size); -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT /* first chunk is free to use */ - chunk->obj_cgroups = NULL; + chunk->obj_exts = NULL; #endif pcpu_init_md_blocks(chunk); @@ -1463,12 +1463,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) if (!chunk->md_blocks) goto md_blocks_fail; -#ifdef CONFIG_MEMCG_KMEM - if (!mem_cgroup_kmem_disabled()) { - chunk->obj_cgroups = +#ifdef NEED_PCPUOBJ_EXT + if (need_pcpuobj_ext()) { + chunk->obj_exts = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * - sizeof(struct obj_cgroup *), gfp); - if (!chunk->obj_cgroups) + sizeof(struct pcpuobj_ext), gfp); + if (!chunk->obj_exts) goto objcg_fail; } #endif @@ -1480,7 +1480,7 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) return chunk; -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT objcg_fail: pcpu_mem_free(chunk->md_blocks); #endif @@ -1498,8 +1498,8 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) { if (!chunk) return; -#ifdef CONFIG_MEMCG_KMEM - pcpu_mem_free(chunk->obj_cgroups); +#ifdef NEED_PCPUOBJ_EXT + pcpu_mem_free(chunk->obj_exts); #endif pcpu_mem_free(chunk->md_blocks); pcpu_mem_free(chunk->bound_map); @@ -1648,8 +1648,8 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, if (!objcg) return; - if (likely(chunk && chunk->obj_cgroups)) { - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; + if (likely(chunk && chunk->obj_exts)) { + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = objcg; rcu_read_lock(); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, @@ -1665,13 +1665,13 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { struct obj_cgroup *objcg; - if (unlikely(!chunk->obj_cgroups)) + if (unlikely(!chunk->obj_exts)) return; - objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + objcg = chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup; if (!objcg) return; - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = NULL; obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); From patchwork Tue Oct 24 13:46:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92C0BC00A8F for ; Tue, 24 Oct 2023 13:48:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B94636B028D; Tue, 24 Oct 2023 09:47:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1C4C6B028E; Tue, 24 Oct 2023 09:47:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F9166B028F; Tue, 24 Oct 2023 09:47:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 76E016B028D for ; Tue, 24 Oct 2023 09:47:51 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4A5671A09F7 for ; Tue, 24 Oct 2023 13:47:51 +0000 (UTC) X-FDA: 81380483142.02.F2339C1 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 6EA14180028 for ; Tue, 24 Oct 2023 13:47:49 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=v1icPGNx; spf=pass (imf06.hostedemail.com: domain of 3BMs3ZQYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BMs3ZQYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HuI3Xrhprruhc52//OZX/xia3QtNu4igKL3qfHIbn+g=; b=vY2JAMA9lTW2eyd7Bw0M5bSzEnHxjhhNa5UL+daLRRIIMPr7O1wlZXz4taEG49fYGBX0qF PmvbDDOr72ERmLaxIjoCva05W9xM70cDn7pVgtmOF/QMkwWdQIJPJLnDrE8reqYo8lWvCZ Vaq4D4h5rcEcksGkNdvOaRiEHpeH2Oc= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=v1icPGNx; spf=pass (imf06.hostedemail.com: domain of 3BMs3ZQYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BMs3ZQYKCKkbdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155269; a=rsa-sha256; cv=none; b=au0IR89pRTxYK6ncoEe6PXKKKihMBiw255xFX5P+2jHGbnSsyctn2mg3ESV0gqLd9RV6N0 GlEfyirj0FOMg2Bm8RUOsrw/bG3+rAolWYEa+tFPw8ew7apObiZsq15RVaz6/d6n7hER8Y HYFdlS+nJg/7uR5VRH2KmwAaGFNu6XE= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9c4ae201e0so4420772276.1 for ; Tue, 24 Oct 2023 06:47:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155268; x=1698760068; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HuI3Xrhprruhc52//OZX/xia3QtNu4igKL3qfHIbn+g=; b=v1icPGNxHbMXo9cu+X/4Tn8gVdflX2Vv8lkPSoi7zingS7MKNcCu4kkUVJXk3J/PN4 MU776ujMYzcJelm6clFiRw1BBIHQg204op27apwqKhKKzjr68twvFZUbr1vtS6YqWvhi Vuor9L5Y3AgQ4Zzztz/o2foviT5+QSUG9K00MpZFne8+T4YdsBTOQCPgA4/HEUeqWwks kv5gxG+u8xyGwOuP+hIsxKgut1B8/GKh4osnrIdWG6oPGSFuMXiQge2AYkzH/e9Am0as VKl6UDCtmj5JbEVp3S9iq4aDZMQNvmPSpcc3tnqrafXGOYadT7U6aocJu9yegH4wqbYv +wyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155268; x=1698760068; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HuI3Xrhprruhc52//OZX/xia3QtNu4igKL3qfHIbn+g=; b=pM+PUpcUcIMXTfX8CqG00WsSTrv3417WPrfduOUGD67buHgwAQk0gKpsnaf/pZaE0v HQJdCfaXsSrrfdJa+Yh/bnJ9F0miXCn933vG4CIHNNggqxYhAr1c2QHsiDCyuSgAJv5e gC1kLtM3Y8YjBnCTL0JdBeglmUnKiT4rqkw4K150X4KTFB9koSVcrXeQBNwWlZh4oIdY 4OFUsss3ZG6ceEGaepZlztzPBYiiTSFGRDlzIz8+5UEodHMu/WAHopfE3IhCY4UDUUnr Es96uONfyZ1miWvEwAiDZBzlhPew5sj3Iq/SnIWHE7knIFH+6w2xcAl4jH1FMYFG2ZdC +Gow== X-Gm-Message-State: AOJu0YyF4l2Izuv6HJyhbhFnwtHCpE4SReQ5KG/p5T3MvztcRc+B30tD cBrM/R7DxWfk21tLdjApMfJej6qX6Pc= X-Google-Smtp-Source: AGHT+IFJAKlMWbiV50/LrVqB8mrnoydVscOxG9uaLGUhxnV+JO976d9OMlZUiNtdsluAhIYZmngChgUkEcg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:ad41:0:b0:da0:6216:7990 with SMTP id l1-20020a25ad41000000b00da062167990mr3869ybe.3.1698155268465; Tue, 24 Oct 2023 06:47:48 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:27 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-31-surenb@google.com> Subject: [PATCH v2 30/39] mm: percpu: Add codetag reference into pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 6EA14180028 X-Rspam-User: X-Stat-Signature: 45rgob565jwarnw67kt6gqumedtewu3q X-Rspamd-Server: rspam01 X-HE-Tag: 1698155269-471574 X-HE-Meta: U2FsdGVkX1/oXJl53zE9+T/Wi6Ypp8uUeicqmCVaHrFV+fZ5ysMuoyM2Zx2lSfkSmY3xB3kyNBxZW75PObBIkIMaVC9cuNvR4DgL9sIJyZDH3Idbv9sCis4wjW+92P5ksrTzjs22Yd2yOWkeBGCTkrHtx1irmECGHJoQU/I54aIE/VSL6NNOTKYFWKGocm0YeumNYb24eGx6VS7mHZqPqLEqsDW7JW6uc4h8ho0tKiF85Ua7IdSQjA5VB4osyodQipg8v8xTROnVLjQH7DNQQOveEQAOsuLJeG+YPIIaQTamg3+7x073+BNAftrJ3hQo9DvVRmXA7w5l6tAuS60YaEwwcc3TZvY02v4TRIIUX3GVoR9z0Fdr6xHrqk3Flt4sojBMRY96aruOMajmKjqsOUl5spZoREyPd7nqFt270wCfC33cl7w1kOm1/us3aTce+KMfP6zoYVVsQqX+oaxNx5zYDVzBa+029NLTtG2H+HBsrlUL9tfvenooZ9H9XocJf7Fm48MQ7mF8WlQZVrpHkctYYi9RN7fdTDLJg6qgqFzO0tId328L4rcWjbQh79rVNrBNYha9Iaik3T2MpB1hpW4L7GvqkctV8XDmkJAGUpoVh3zh9biZfj1SDHh58jFyKqDHkp7MjcWWwdZlCAabA8i0H4Qh3SIH24oP0mYAeg6zC4MJ9epeIb1baZ7wqLhbQVyaK0jeS7eFV8WnV0GGk41onhTtajoKtcVtaFjEjEYqlkJfAT13irPAIJFunsw/KsAkHJrBpL+h72YNLx/vvrsP1/34ahPH0xJjgZBX4xkvcSRu1grszcMookI7DV/nRKO63gNCGZ17TwsEJCif0Qe4+8zehKMPzxu8hEagEp+Eg0h+LtEDalgRzTldmAl5w/Ip8JQ0wEPSb1LIVUJladdyxpRNH4ePqKroR33a0d+TBbIqotcxV2tR6WGvs0mPkNI+n41lpSsKrTIvL6w EkCCSI3G wf+9la65ze2Gt6QhBnksfOsoAott+mFTTrbsDGgxkH5zDi7UQWkVnJjdaOQEL8JNAUTGOlm3u1rtbze+XJCCYn0+tG3q2TM8RhNmalhFfBTHEm+P08O5gWrp2NQt0Ucso/sdnXWRBqC7ZOadGnbrnVC6LpFxv3769eIIitcdEGdZDFR45Apd2c2lk5vF2wkK4K326EQ65PmQGsHluhKXF5rsS0T3Yc7BMQ+AeFO2TIXh8kiIV+N5vuTtu8RSOvi1222iWt5jtTkO0kZBOle3kSO09ZD2uMy5ziTXI3GpGsKAWpIqOwFIWKoBubGIipi8RZIm4ty/HsNxh2Qm3sY8GyCnmHsm7h9L4a4j7qp55VEn4rnzikudRSLqMUk60DrnXDJInJ/8WW+CmppmYyCz+Yqh6maT9m8ihu7fmcSc9Ds6OZKEk/a897READuhGvcP6B/DP/aYBJOJhpMzB9+W3mFHyEmrCsLgDVqxHcoNRSRBp1FUs2eIhRfgk6JatyPCWVuXCJJcbFrZEg7KL0Gqo4HZNXtewj8Wy+TUQCw5IwjUVaPg3Y3z9fKZPfNsTA6L4ScYsguQGoYCu40O0ebVMbsRqaNISnkaauqGKVFm5ZKGo6RUDqxOMXCotoqsDw/QCaj8+KjIMK4OBP92VCiWBhdaHjVpplLiEnNL2Cq1JthB1bWzYS+ZEXxgPOR/Gu+Xpcxrsot4fRz5PXOr4Iv9KWFPYzQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet To store codetag for every per-cpu allocation, a codetag reference is embedded into pcpuobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Hooks to use the newly introduced codetag are added. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/percpu-internal.h | 11 +++++++++-- mm/percpu.c | 26 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index e62d582f4bf3..7e42f0ca3b7b 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -36,9 +36,12 @@ struct pcpuobj_ext { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cgroup; #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref tag; +#endif }; -#ifdef CONFIG_MEMCG_KMEM +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEM_ALLOC_PROFILING) #define NEED_PCPUOBJ_EXT #endif @@ -86,7 +89,11 @@ struct pcpu_chunk { static inline bool need_pcpuobj_ext(void) { - return !mem_cgroup_kmem_disabled(); + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + return true; + if (!mem_cgroup_kmem_disabled()) + return true; + return false; } extern spinlock_t pcpu_lock; diff --git a/mm/percpu.c b/mm/percpu.c index 5a6202acffa3..002ee5d38fd5 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1701,6 +1701,32 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) } #endif /* CONFIG_MEMCG_KMEM */ +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { + alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) + alloc_tag_sub_noalloc(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ +} +#endif + /** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes From patchwork Tue Oct 24 13:46:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434600 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D34EC25B48 for ; Tue, 24 Oct 2023 13:48:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 099086B028E; Tue, 24 Oct 2023 09:47:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 004FD6B028F; Tue, 24 Oct 2023 09:47:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D68286B0290; Tue, 24 Oct 2023 09:47:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BAA676B028E for ; Tue, 24 Oct 2023 09:47:53 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 877D88075A for ; Tue, 24 Oct 2023 13:47:53 +0000 (UTC) X-FDA: 81380483226.10.4A11627 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id D5F1C100003 for ; Tue, 24 Oct 2023 13:47:51 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2hV3EYlB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3Bss3ZQYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Bss3ZQYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155271; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=05OHaXyGuLuiu55tV7A0+NfbyrdoqwRt+v2k2i2/xoM=; b=Zcp5hR9fPRm6OvuRrPLa/gR+N5b+RShOXANzmwGMGgcFK/4jzTihFBpB5D+jf2phCysBgU GDO0KwZLGrpS1WeVYxO1GWIZ8OPW7IfPGwhsuw9UwGk6EqetRBpmJrC3iy20ZlwqrduZLi Yr67swKkJ8kNSENPBRa5AfxKa3+CES0= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2hV3EYlB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3Bss3ZQYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Bss3ZQYKCKsdfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155271; a=rsa-sha256; cv=none; b=5j3DtSgAv4WyUNJTIY0WaBLF4Ms454v5asVRZ72QdFX+5nKxhpeqfMWdQz/27Mjp7CdIhs DmoRT/VsM86Lqa0oTUryVMNzZ7WnngQSaXyLG4aPYv4WDbA3LfduZXWYqlAEHDvLUyVqRd xeVjhyIp0/O0FSE7gt0WLXmDRQF3B0A= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a824ef7a83so58268727b3.0 for ; Tue, 24 Oct 2023 06:47:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155271; x=1698760071; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=05OHaXyGuLuiu55tV7A0+NfbyrdoqwRt+v2k2i2/xoM=; b=2hV3EYlBNHTXtek2xYHXu6i6Xmy45Q9hparm278/Y0vM2O9nOlWE1Sfjt6gvL2xDN8 YyBHJeXTv2AC49LCqOr8DaAcaNDioZcWt9/hTcNJJJCCmNL5bn8sRjK1dUAJnvfzSZFe zzgcFAJGWYsAG6fSpWLHWi/2oTnlshDODUDmvbTD/e/2SyBFQdMia9qC4riuq8+UJdvp lYCRJFqqMvxLinZCnXKKe5aIBctFDLL8nYFBmeFgbpcJH2/mjd8rjGtIDTFl3M3Ni0Ci MNfgp/QAvMaCYO+fd+LUd2Be/Fy4kwxG+5Y3xNbuaqkVG3UERIM7xguOT3cKF3jQCx96 akeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155271; x=1698760071; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=05OHaXyGuLuiu55tV7A0+NfbyrdoqwRt+v2k2i2/xoM=; b=tERUggWw/IVhlwLM42RIwc86x/yktnUEMK4mT69hhKufDfHRYoHYLLttrsA157TYsx rUt1kZscvWBCBh0aUvKS+WQYygtJf5voV869KytAu6LGw5+z5rxqphATVs27XsthnGr1 EZLrDZ4kPVCi2sa9gPMx+2mbdk+fCCSZ32sdatnBKNvgs+lq3aQYSkl+75M9o9fbGNNc yj+/MPznhx5+lla2eOZysHaKr/MihsnFn1CJhVZCez8c1CDPkK794jSu+LcRVGAqfAln bbELjlcehbcyYu6XiN+AX0I2duVnHwb4Tq36aoNFf8BPjUH1rLHQD3fzX3McGtlKY8vk kf3w== X-Gm-Message-State: AOJu0Yzdpm1LvE6A8ghWnQllVnXsKWpbTJkcm0xIALjIoZFHgKvmzW+C +p+P3dtXvX1Uj+2lkCSZjZu+XmDyNOI= X-Google-Smtp-Source: AGHT+IHL3ylyGkPSwgZbZBswail77fdEm5nbxSBwW+rtci5/qxf0YkGsYjEQDGdhz4VxvAtbtJa/uTQClZE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:5008:0:b0:da0:2b01:7215 with SMTP id e8-20020a255008000000b00da02b017215mr55853ybb.10.1698155270752; Tue, 24 Oct 2023 06:47:50 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:28 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-32-surenb@google.com> Subject: [PATCH v2 31/39] mm: percpu: enable per-cpu allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D5F1C100003 X-Stat-Signature: wamm9e4kfuo9ojh5ez4uoj7ddwyesjyn X-Rspam-User: X-HE-Tag: 1698155271-473379 X-HE-Meta: U2FsdGVkX19c96dqOC6tX3iXHn331qGREgoRj0EuymspxdrKdnUeK2XYRuKPMScgYRzpZUDblYFjjKePTLVRgDtjUhOMig8g7eFIYrUeriTO5Gnw0/X9LRzyvFFAhQtDvKFxBwA9ocHIPRsUlDoYvVbDuaKFDnwKZW16l/5ZOpxw3SKu8kdbEuL7VOeJLErswCBnHDYRay6wik5AVz1g6MZkA1Vk3fXd/Lo7DlAE0wdkbRAta8vX/vkQdBSZrZMg6qo1iKlHjEfL9noLRMABrT5k900l/mezS2WWMcKVBUXw3Uf9Yzox9ytI0/rIUcGDxhJwTpNnkCXbrAt2JIuDqM90ZsMRVIWX/PLixRkDFgwQH6dlzHMYi7AZsXuLwtvfKOyOLDBaIPuoYLtcR1YRV2HlXEq1yOwbYb5IW+MalLEwQAz5rAMs1oFVZwy/7Pd/QfJY/B6TvIlmCA0hpUfjCTjSxJ9Gu7AV2hH9CFPYvE9BmNedsZ5S63toGCaIvFCFHUX7ymirlElEHV2uz0ZZaAp4wYigjbI1i703dYSvsOsPvGAoslncSC41CxWlqhX0unKD4FBM37dYOCDqeIGH51S3srfnzL6i9/1ac+y9BFwcsI6ojp53wZ0dEmxBslwHHbfp/tcE+5tYkx5J/rymKBKObB0fL8dXzHTQbWg6TDH3JSyQWsibJYdVloJSJ9QkCCI1toytH/dRyPijNBjNmX0R1Mu0mW1WZaxvrmoBXpx7JklbEBhsFMT6915svieg5xwZua8vYUpR+GVU0h+zm5JAAs38i3xYZ0/IaPv1JMeWK3SR/xPBVb1LwOztXEwhsOaeYJbsAd9tcfYLXpf6YkcaVrWSFul434DZdiEmp13nuTFDF5NGM3+sxbRoXqrrTPSE0T7+DcVZQ3g1Wfsynt9Iy1C4LekDkhYuNnQN9NTivDwbDwAHNNpt8KmN8b4lpx6mE0ZRX5PsZANITEl KVoKULCO XWVcBNgasX9hC183TumpDqRcOtATS/OUeZ35FJb2qyV4aDM5z8KgXrh0xQ6RMTV288GqHuj0QYsqsOoWkXdskvxyySxTaxnb2Rr33zdvbhx6K5cFpkgc0YFngCncPmvzPcXVEEBUCjrd9/4Oz7zn4TbAHVdnhv0VV3dr32VlKOI73nabgc1bhDEH7sonzLI/nYzLvqcn+KeAvEMBpBCPKllCq5LT7SFgZKPRzIjdx2WHV6c75OKMcCyrFY+j05toqxQ0rfZ0DEwADtHUEwgbCHTIZk1i6AleYi9rPt8Pd32eJ3nVl/asBTXB1XiXum6oqSn1ok4OPmLwuaJAjqPNok/yBLTdtXRb9ENFowWgpCbRX4xcg4pR2mD+ctZkgNcCNyo0ok6/h6NSIoCNC2peeyMGJSAzFHBhuXNpywdGGnKudS4pKdXuPf8CcaUc44CS9g1UdiYrYdQo5LvPCQx/Yqn4q7PaauFd4sg58gn8WYqolyS04bz9XR5/QeRZSGt3WuKr7aL3EKYOmMPoHHsOF3ILNMojhxUK0Bzw7aq4WAJpv3Bfn7pS1Rp0F6SCW15TfTYo0ar/PxeTMaKLtFxw7rvAY9R0ZeZjvqnK/U4ePyV/RJfP6eJu0Fq1DXnp7kMfSo3wYZ1pw74Kapnf9iC9LiSSu6s3EiBUMJv+XjPeQA79DZuU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine __alloc_percpu, __alloc_percpu_gfp and __alloc_reserved_percpu to record allocations and deallocations done by these functions. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 15 +++++++++ include/linux/percpu.h | 23 +++++++++----- mm/percpu.c | 64 +++++---------------------------------- 3 files changed, 38 insertions(+), 64 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 6fa8a94d8bc1..3fe51e67e231 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -140,4 +140,19 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, _res; \ }) +/* + * workaround for a sparse bug: it complains about res_type_to_err() when + * typeof(_do_alloc) is a __percpu pointer, but gcc won't let us add a separate + * __percpu case to res_type_to_err(): + */ +#define alloc_hooks_pcpu(_do_alloc) \ +({ \ + typeof(_do_alloc) _res; \ + DEFINE_ALLOC_TAG(_alloc_tag, _old); \ + \ + _res = _do_alloc; \ + alloc_tag_restore(&_alloc_tag, _old); \ + _res; \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 68fac2e7cbe6..338c1ef9c93d 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -2,6 +2,7 @@ #ifndef __LINUX_PERCPU_H #define __LINUX_PERCPU_H +#include #include #include #include @@ -9,6 +10,7 @@ #include #include #include +#include #include @@ -121,7 +123,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); #endif -extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); extern bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr); extern bool is_kernel_percpu_address(unsigned long addr); @@ -129,13 +130,15 @@ extern bool is_kernel_percpu_address(unsigned long addr); extern void __init setup_per_cpu_areas(void); #endif -extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); -extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); -extern void free_percpu(void __percpu *__pdata); +extern void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, + gfp_t gfp) __alloc_size(1); -DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) - -extern phys_addr_t per_cpu_ptr_to_phys(void *addr); +#define __alloc_percpu_gfp(_size, _align, _gfp) \ + alloc_hooks_pcpu(pcpu_alloc_noprof(_size, _align, false, _gfp)) +#define __alloc_percpu(_size, _align) \ + alloc_hooks_pcpu(pcpu_alloc_noprof(_size, _align, false, GFP_KERNEL)) +#define __alloc_reserved_percpu(_size, _align) \ + alloc_hooks_pcpu(pcpu_alloc_noprof(_size, _align, true, GFP_KERNEL)) #define alloc_percpu_gfp(type, gfp) \ (typeof(type) __percpu *)__alloc_percpu_gfp(sizeof(type), \ @@ -144,6 +147,12 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), \ __alignof__(type)) +extern void free_percpu(void __percpu *__pdata); + +DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) + +extern phys_addr_t per_cpu_ptr_to_phys(void *addr); + extern unsigned long pcpu_nr_pages(void); #endif /* __LINUX_PERCPU_H */ diff --git a/mm/percpu.c b/mm/percpu.c index 002ee5d38fd5..328a5b3c943b 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1728,7 +1728,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s #endif /** - * pcpu_alloc - the percpu allocator + * pcpu_alloc_noprof - the percpu allocator * @size: size of area to allocate in bytes * @align: alignment of area (max PAGE_SIZE) * @reserved: allocate from the reserved chunk if available @@ -1742,7 +1742,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s * RETURNS: * Percpu pointer to the allocated area on success, NULL on failure. */ -static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, +void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, gfp_t gfp) { gfp_t pcpu_gfp; @@ -1909,6 +1909,8 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); + pcpu_alloc_tag_alloc_hook(chunk, off, size); + return ptr; fail_unlock: @@ -1937,61 +1939,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, return NULL; } - -/** - * __alloc_percpu_gfp - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * @gfp: allocation flags - * - * Allocate zero-filled percpu area of @size bytes aligned at @align. If - * @gfp doesn't contain %GFP_KERNEL, the allocation doesn't block and can - * be called from any context but is a lot more likely to fail. If @gfp - * has __GFP_NOWARN then no warning will be triggered on invalid or failed - * allocation requests. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) -{ - return pcpu_alloc(size, align, false, gfp); -} -EXPORT_SYMBOL_GPL(__alloc_percpu_gfp); - -/** - * __alloc_percpu - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Equivalent to __alloc_percpu_gfp(size, align, %GFP_KERNEL). - */ -void __percpu *__alloc_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, false, GFP_KERNEL); -} -EXPORT_SYMBOL_GPL(__alloc_percpu); - -/** - * __alloc_reserved_percpu - allocate reserved percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Allocate zero-filled percpu area of @size bytes aligned at @align - * from reserved percpu area if arch has set it up; otherwise, - * allocation is served from the same dynamic area. Might sleep. - * Might trigger writeouts. - * - * CONTEXT: - * Does GFP_KERNEL allocation. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_reserved_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, true, GFP_KERNEL); -} +EXPORT_SYMBOL_GPL(pcpu_alloc_noprof); /** * pcpu_balance_free - manage the amount of free chunks @@ -2301,6 +2249,8 @@ void free_percpu(void __percpu *ptr) size = pcpu_free_area(chunk, off); + pcpu_alloc_tag_free_hook(chunk, off, size); + pcpu_memcg_free_hook(chunk, off, size); /* From patchwork Tue Oct 24 13:46:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEA17C25B48 for ; Tue, 24 Oct 2023 13:48:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E1246B0290; Tue, 24 Oct 2023 09:47:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 269696B0291; Tue, 24 Oct 2023 09:47:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01FDB6B0292; Tue, 24 Oct 2023 09:47:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D9D5C6B0290 for ; Tue, 24 Oct 2023 09:47:55 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B17EEC026F for ; Tue, 24 Oct 2023 13:47:55 +0000 (UTC) X-FDA: 81380483310.20.32243D8 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf22.hostedemail.com (Postfix) with ESMTP id F123CC002D for ; Tue, 24 Oct 2023 13:47:53 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=c8VwybIC; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3Ccs3ZQYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Ccs3ZQYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155274; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XaYZQGGekpRYWKofo5TkfncjTV9EZU0daTUjtZ9a4ek=; b=R9iBCYw12Wpf33WQfop0aKyAuUsM2G+dRW0zdcT2DAFVmiOnfBZ9bhQDgIxv/ZX8NTkvUR LBmHz77OSGPpen9LoRKB6eQ6x41VuzoYlk7PXPSprtB8PFrkMg98KTqwLXO/NfNn9RrMnl QRcL/9TFH+Nv0Fx17EV8p3SK9dxI16M= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=c8VwybIC; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3Ccs3ZQYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Ccs3ZQYKCK4gifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155274; a=rsa-sha256; cv=none; b=dZWEKSyqWv1Of8X6uMS/nV248syslrMG0nk1+9fIaASDuvd3vTxePUG7Pu2FLbj+FB3ZfN h41nN5C+5FQeY7zcjI0zbiacd8bDZt0MxTzGijJRivu0KtM4wvF4dcRDSJ5csLQnHqLTh/ JcEcMnNIEAksqd8bxPHra3tJ7tUuAZg= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5a7a6fd18abso57845697b3.1 for ; Tue, 24 Oct 2023 06:47:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155273; x=1698760073; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XaYZQGGekpRYWKofo5TkfncjTV9EZU0daTUjtZ9a4ek=; b=c8VwybICJQ0iiOd6KCN/XIVdcPIO3w6B4kXBFU/RTarZtsuHOdbJxlkBmEtnHJYLDW EZf2uMNjM2fNPJAjz1erQ2iRiI85Q0dmGD/fQMt7ntolH7XApQO1FFGWwEgwvs8CgQpd d5FVXmophTC3jIB56gvN6gJRdqt4juwlgk5L4S9E3VxEpuLo4o6KDrTgNoYZrSg1meO+ w7YZk+wugxNGHO2+9htDQCtFvRrVhG9eJq3OXpVBnGfKysZ9HhpjGJfAXRq2RY9jgyl7 TxsZZIxWPl0HlpeOD6tdskeBByiwhvqQGafv1ZivObHsBZwrzCsXS8Kfs66URwaR6+OS cMlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155273; x=1698760073; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XaYZQGGekpRYWKofo5TkfncjTV9EZU0daTUjtZ9a4ek=; b=FlGYhG0dfjSQ4HY4vbPCtw/HcAcumiTePMoouApG+Hc7Djds8s1EhEPO1jmKAcdbNO +2fOzdyG/k6zMaRo5T6TegfgYOQE0pu3XUj83B5DX9HQ6o2e1+RHv0pcqlCLeq/RllUe o2hcX1i+kd96dpTzhP2T/fdGzHxoRdmDGvYEXaswt/UEnavLhSF2/Cne71YIsMqGQsr9 epJfSXpyqbRHDySXVYdpuuvtpRXZ8D7EjWG2vQasJgcifxbYXJC9ftGRDuxNVxG/iif3 LIb/ybqxpWrBFROnazZOTEqZ0usz1NluqcJzOkkd4lliQ0vRKxA8CgwLRiiY2+6FFIh0 y4sQ== X-Gm-Message-State: AOJu0YzeYyo8VBk3cDYddhFg/f93uifq07DYC/vH+L0kF081urYQxHhG qUzvVhppBLmvutIKKBaOXpE8gIDK74w= X-Google-Smtp-Source: AGHT+IH/jHMj6SPf+djqQdMpidZEKeSxKuOmMnfBX0O2HbLM+zTXwQbpAuHKC7+8B6veMY9hNUZcoYeVJwM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a05:6902:168c:b0:d9a:e6ae:ddb7 with SMTP id bx12-20020a056902168c00b00d9ae6aeddb7mr221434ybb.7.1698155273022; Tue, 24 Oct 2023 06:47:53 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:29 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-33-surenb@google.com> Subject: [PATCH v2 32/39] arm64: Fix circular header dependency From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: F123CC002D X-Stat-Signature: kd5srxs7y6cn8phjesr5qub7bz9imcnw X-Rspam-User: X-HE-Tag: 1698155273-573096 X-HE-Meta: U2FsdGVkX1+uKycKUjzAp7mha2HiL++lzlLswmpcRy4d6YQ2Ua6sPBiy6RUJCzFzQ0abzeEwCyBZwv6KYh0dgcgXBRVziraZOh4NOYqhc7Cic+qrgTercnooVfCEiL6vK7/g3rJSBu4un9B8NG+6xpmD2bTSBh30mtRSQqSnJPDNFi0d7g0zzw9qmGQlPpPFxi/zrVcH1bS02sHug6N/aMLmihA8j1J3qNkRhF/QpRWp1q/JJyuDQML1INT2Z4QcJ4SdF12lw8qtyhD81K6l5Iiw1T+jxj0A9FlvOJZH/n0JJof0L0PqD0WewA0oAZ3E17uLwwYv+7LfEd6nU2EXOtikTeV7uIlefM2+4GeESIwGN+svZ/BDzLpABPJW0/vNfozBeKkWrgmTtYuxfYtSWY3djET+nIiZjqZ73/UoiUYW81v7uwiCzdIySGGIFD9mjDbygmho34OLmt+Fjqa6WiiTgTTQIHkY1keMUpNf6D5mj832kBn1yTvvYj2mcMpE6JkfaKuFLOw5eQ7mr+w7bu30qKq/+RVr7UB1vM8QiRvU22pnEb1y67WY8T1y3iNqpDkN0lEMuEFy+Q6L1S+KUgcaUCmX2w8f3WqFhhsF3Y3BAmfvbuAwHVcikHgEL3YSMtCgFMQT8WfrbgZP2wbo/DLodb7k2NVGrKSlXzHFJY8nI1r0ryLSMiYF0H7pwTFNyvp9PgXMGNFawd3sSPnKfzF3eIBziBGXBD3jOPtrzkkJYtEFONTLcdGQ6LhfM/MqdJ8kmY3QggPASVQJrkDwnAOPApdfwSDVFxlvrwUPpUwofqcCU3W6ynTiLZd64UdYbvDkPnKY7OTco7MZkdaoOnVvNNqg+AL8WYBvG5rn7V6+O/waDB9o2VxYKkeJREP723J0VnfyqBgyFncFsrDbZBfbSnsSJVKHSpGwyhrUL1BuTcR4zaADYHFDS6RMs1yqHp4WxKpuDJJaaWZIh2B INYmXZEX RoG9dwm83Qd7X77AfifDPIkr2KFD4Uw2DaMwVWsCA4HV2A00CXVY/mLWG9QNXgcfRQQGKHlUF7rYBfi6nEqg/8dMe9lMf0hWa8nL27aKf2ze9+sabL9KiJ289b/FA1vWo1crF/HRLJ8ExsAtC29Wi0A1Y/fQPXzxuU1V78abREwni8yMobVbkZGjbsglKlaBRBC2+TNk4udXxoGRlawrmEwZ4cT1YcOQQiNslvaATRK63ToqwV0qUSEQRmv5Gc5W8aTOrMq1Q8rlIRg6zHXFJVqvlDnRCpzRrbGmcwtBmK9g+O8PmRSKsQxaJoiHlEGPco7NUgvvwyQ9HQtt8q3hvtd8QknV2lz4SbyMJb7Rpr+yUZgEsKTtdqNC6HEdAl20onI73CDK8FV1xrcJJwMrIU4UVXqKPH/xfl1wKAGY2e2i4dqKLy5JgEp3kjudHjl+RLNUkovaRe39MldvTG3eow7zhuJJDc/Z0lRbj+HOsnUQ2LWKDC97Ruu+8/sQyDY8Imw9IPz2amGNXTujoy6SL3FOw5affe7pOALRImSz+vnAMdDsQea5sNpDxbawyZcmOScLNm9R7/7CbKaPQxseFD1FZA346hA8DlmRrb+yXuI4XuhBkwLNiU5zPJH4paMTCqLkbrHyLXDYx9k7aVYQHSpOA2bZ8U228f6bruOEQ8wlQ6BaxFBnH1vyi4JVRu906I37Jz2IfEv6lhfKrlH4GwZrY2QB6xQvLmm4dcWHmIvVehkw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Replace linux/percpu.h include with asm/percpu.h to avoid circular dependency. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- arch/arm64/include/asm/spectre.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/spectre.h b/arch/arm64/include/asm/spectre.h index 9cc501450486..75e837753772 100644 --- a/arch/arm64/include/asm/spectre.h +++ b/arch/arm64/include/asm/spectre.h @@ -13,8 +13,8 @@ #define __BP_HARDEN_HYP_VECS_SZ ((BP_HARDEN_EL2_SLOTS - 1) * SZ_2K) #ifndef __ASSEMBLY__ - -#include +#include +#include #include #include From patchwork Tue Oct 24 13:46:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434602 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28D9BC25B48 for ; Tue, 24 Oct 2023 13:48:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9498E6B0292; Tue, 24 Oct 2023 09:47:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D1866B0293; Tue, 24 Oct 2023 09:47:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68A326B0294; Tue, 24 Oct 2023 09:47:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4EF676B0292 for ; Tue, 24 Oct 2023 09:47:58 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DBBDCB5DA2 for ; Tue, 24 Oct 2023 13:47:57 +0000 (UTC) X-FDA: 81380483394.30.0CC0B97 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf27.hostedemail.com (Postfix) with ESMTP id ECD9540022 for ; Tue, 24 Oct 2023 13:47:55 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pARhJSFK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3C8s3ZQYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3C8s3ZQYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155276; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JDn4KyOu1aP1bbjJv2+LYLA0L02aMbiW0UOS3mV2n4g=; b=xDIijIvfMH1mOS1Qrw2cjYNpkNe68QniOcJE1qrJLjb5SxV8Rx5hdU9VZXojDjm1ZU8lxc EB4yL0nj8uTdHl9bhyZgCJoltwMfC8acUl0RZG9gbvcD8jplFyRUzthURNO0waAwij31DZ DEyT/NtMakVTap3MDDi0ryhh0yc5dRw= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pARhJSFK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3C8s3ZQYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3C8s3ZQYKCLAikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155276; a=rsa-sha256; cv=none; b=LeuZwkf47ps5kIDVbXduHXJHAXDE0pmLejAuxtTIm5AiqIJeLWV0Wyqh3jjbj9DlxneiU3 Ha2tS2NwLP2cWh2t/YAyBjGFgc4Hbmh5i0Hi0hKYaAG4LAVQX24R2ngsjvGgdCICDwlPhL DVR2uoUfLHKZ94g5MnvCA6ngYUvQhS8= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-d9a3a38b96cso5081353276.0 for ; Tue, 24 Oct 2023 06:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155275; x=1698760075; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JDn4KyOu1aP1bbjJv2+LYLA0L02aMbiW0UOS3mV2n4g=; b=pARhJSFKP1epRUTxcMClKNxX1umZ4HUobS1EUZbviTDk7BrXGaqBdpk6B5gg4+srgG zFjrjE+HopQ32xoMmYfUdetfj4YvTekkqBl+O/hXRjWHV586rM+jJfiQCsj7LDSTVx6i dvsSxk0kku2k3u260k2QPsfOzJ1wZZnXAN47wVc3AlMiTH8KhRBuCBviEvsuI0Daz/Sp +0PPw+mvCsqXFMXWTCwXg6Pr/awBRGw0UaxsQTRP6fsjO8q+sPuj8xAzvePn0MAPLTSl 1avA1ZlPbi3w3b1NBNAUAgOn1IymvrGVmZ2j2Ab6jCNb1IieYbd08YUb1bOkk1EZ8nLm nENg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155275; x=1698760075; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JDn4KyOu1aP1bbjJv2+LYLA0L02aMbiW0UOS3mV2n4g=; b=TmX/p51fi2Gxy6DwSqmv7HTfPE2TTF8FzPV602rVEwjmKNZNUWwp8e7BBm6oWboG6b HRZsteGF3xyAVdmQ4ht/zJvqGxN0dzmNdXXBawqbRCrtoXL8u2YB+tLCL4HI7SyKkiZu Ltde/w8ca/MPYsBOUP7qUT//WPgKtCEy1vQlNCf4KG+ejcLDoD5RCwVCZxCo14qmvmJN MMzZgHAcWiQT7TwgpjaNoT4r3b7yo7KApIDvNDtB/Hr6FtbRQA3ImFBKd9MNea11AC2q xi6UbkHCxTcpdcn1SjPGQkpIs593pVNJgS7XGdEU94owkETOSWXg78EMpWTMW7SWhNyB 4Vow== X-Gm-Message-State: AOJu0YwfL7wbSoHYQa3NamfvJz7rTP1xrHb1NVxRjGpmoOiM7EpXwVk1 VYS4H6Qgzsj/edmka+M5NcOTxh9zcTY= X-Google-Smtp-Source: AGHT+IEj3qXX1ASyXmKltnU3Blh4TU/UIcsAQ8zCrsk6sZWyeAyJtmCUxLTSeeZqDpp0lQL0cg2luCQWxLY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a05:6902:4ad:b0:da0:46e5:a7ef with SMTP id r13-20020a05690204ad00b00da046e5a7efmr24271ybs.3.1698155275005; Tue, 24 Oct 2023 06:47:55 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:30 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-34-surenb@google.com> Subject: [PATCH v2 33/39] mm: vmalloc: Enable memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: 4sh7ejm55awcpce1i3a4oxeq8wa8we1y X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: ECD9540022 X-HE-Tag: 1698155275-182145 X-HE-Meta: U2FsdGVkX19sRSSKqMfnPbz2QtQ0dcPueXGw2VPqxTziHTuhtG8VXv+i/hVudNQMFscSWgNEC0HgTAR3lMnQnBYLY/h+1dEViEofDSjCss8s4AuvNhzLk9YA1ZIkpOwdqx96KK6jOvlrhkItXTXA50UjI0eOmi8faxVXkeOb14BZ/0VxoBbMQIs4sllu4KSSNdVCQS1WwwfJn2Ac1aAcDWEuYz3eaPjGylPEoHJHWgGRhdcoV35F86bjPqF6eqGcRzN68VC9JSAiHHWKpsOMbfyAmCE7L+vHsrsoK9/MOqEih8BsQ8/eDYjc8a/boN6fdqc69FptQ9lpCvIJJBEQZf5K/CZ2eIqnni9RuuAwlpBoiMqzztVoYTcUjt6Re5C6KHabsU9yopQFw42nIrWxcA2ZcSkKMRcvTnxaHa7NbyeUH35SP1tMzthwGe9T6FsCOh5akozYaKoVUUdWJ+vN5/kA0dzu48xunKsCNLp8beZz16ChM89kbdeck8dgyAg1VD1+JCuf/AXT4Moas3ArzbLpbhN68RlaU736iYixV89dGrbcHvtDLJRRklLl4Cr7hVvHzAN+AmTyMYs4gCfD4clJof4AMMogn1U7ZK9ua9PVawxRNbe23o33+QNf+NSeCGkHhXfwTT6Z+Q1mRyqLCI9VfgXGjNQTy0dAYhKPZLML7g0+pa6IZAp0aerikbSF2LQtCzplDhoeOaTErplLgLeTnLXBz7I5ZrfmWg/JuBN03Jhluemzu5YIGM/0Sb2yKNAs7g8hHlD3Hn7cNx8mxlG/ZTW7A2CPcLw6oGhc0/8YvUkEvJ28XfNmfl+2NhZ2rZM6hwkTNkQbF5rilIx69TdZZpSt9AkkyGYy8eCAWo/s3g5Q7lZ+xgbn6zqkUzCr5dkZ130M9hlXFIjmmJaW3Jkbk3eA/8yNSR9TQs/CvkZj4PWCzNHJoBjhjnfJsPC/GfX/JHZiBkFdnuYLpPY edBQFjvt ETOsq6vFyaXBzOI92lpELWgux+kd+5Rv4eO7E/DUW7A25Rh8PNFnPQzICUatO1PbMekNY7TI63E0B33RED6wZS5A9YsN2pRL1Hjf8/gqtkt35UtqCM8hmE+Khde4Eooor/T8g/nqGrAZELnot9rLpO7wKko068ByVKAFwY047vp2BJlF9+ZYjoApuiw/qzIr9XSSvWf+v5S6uWebe+Wc8eAd2n2VEPrqOnkrtGTP/TGYlx/oLmdRbU2M9FSRW9O+qY/3GMInLVDpQIdkOCH2PfN7n4tAA8+w4nzGdzrZSbICbOiGIqU9sauTRbiT/Vpe3e4t2ZpMqP0MKSOvNuFSmwO1sfpw63Mv0BcRIj3yQziAb8c0WgnXajuAtnvpG0D1CSczustxSceps4ZiRexVjr481rqW1xxZCEypPW3UutDVARZdiwIFPhXyDeZ3pj/Oh19MqOCcyIL3PGdEYPIb0ypM3pKb7tDGB4I46b0cMROucBrMf/KJzp439yj2cxk2eUJMi8RIZGqrX5onYKB18ilveCCthW9nHveq05ru+qKAoRyCCw27WAZqkFAcJlXN+9dqti91Gd5IoDyHwnq0+NCVs8Exto1iizQG2s4j4ZN75Fp9MUYYCuZamxjtiCUH+XiMAqWFO+JFwQSlTneUx+VbDGTN/tLRrpEUjCF6qV8KtkZ8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This wrapps all external vmalloc allocation functions with the alloc_hooks() wrapper, and switches internal allocations to _noprof variants where appropriate, for the new memory allocation profiling feature. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- drivers/staging/media/atomisp/pci/hmm/hmm.c | 2 +- include/linux/vmalloc.h | 60 ++++++++++---- kernel/kallsyms_selftest.c | 2 +- mm/util.c | 24 +++--- mm/vmalloc.c | 88 ++++++++++----------- 5 files changed, 103 insertions(+), 73 deletions(-) diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm.c b/drivers/staging/media/atomisp/pci/hmm/hmm.c index bb12644fd033..3e2899ad8517 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm.c @@ -205,7 +205,7 @@ static ia_css_ptr __hmm_alloc(size_t bytes, enum hmm_bo_type type, } dev_dbg(atomisp_dev, "pages: 0x%08x (%zu bytes), type: %d, vmalloc %p\n", - bo->start, bytes, type, vmalloc); + bo->start, bytes, type, vmalloc_noprof); return bo->start; diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index c720be70c8dd..106d78e75606 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -2,6 +2,8 @@ #ifndef _LINUX_VMALLOC_H #define _LINUX_VMALLOC_H +#include +#include #include #include #include @@ -137,26 +139,54 @@ extern unsigned long vmalloc_nr_pages(void); static inline unsigned long vmalloc_nr_pages(void) { return 0; } #endif -extern void *vmalloc(unsigned long size) __alloc_size(1); -extern void *vzalloc(unsigned long size) __alloc_size(1); -extern void *vmalloc_user(unsigned long size) __alloc_size(1); -extern void *vmalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vzalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vmalloc_32(unsigned long size) __alloc_size(1); -extern void *vmalloc_32_user(unsigned long size) __alloc_size(1); -extern void *__vmalloc(unsigned long size, gfp_t gfp_mask) __alloc_size(1); -extern void *__vmalloc_node_range(unsigned long size, unsigned long align, +extern void *vmalloc_noprof(unsigned long size) __alloc_size(1); +#define vmalloc(...) alloc_hooks(vmalloc_noprof(__VA_ARGS__)) + +extern void *vzalloc_noprof(unsigned long size) __alloc_size(1); +#define vzalloc(...) alloc_hooks(vzalloc_noprof(__VA_ARGS__)) + +extern void *vmalloc_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_user(...) alloc_hooks(vmalloc_user_noprof(__VA_ARGS__)) + +extern void *vmalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vmalloc_node(...) alloc_hooks(vmalloc_node_noprof(__VA_ARGS__)) + +extern void *vzalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vzalloc_node(...) alloc_hooks(vzalloc_node_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32(...) alloc_hooks(vmalloc_32_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32_user(...) alloc_hooks(vmalloc_32_user_noprof(__VA_ARGS__)) + +extern void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc(...) alloc_hooks(__vmalloc_noprof(__VA_ARGS__)) + +extern void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) __alloc_size(1); -void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, +#define __vmalloc_node_range(...) alloc_hooks(__vmalloc_node_range_noprof(__VA_ARGS__)) + +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) __alloc_size(1); -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc_node(...) alloc_hooks(__vmalloc_node_noprof(__VA_ARGS__)) + +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define vmalloc_huge(...) alloc_hooks(vmalloc_huge_noprof(__VA_ARGS__)) + +extern void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vmalloc_array(...) alloc_hooks(__vmalloc_array_noprof(__VA_ARGS__)) + +extern void *vmalloc_array_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vmalloc_array(...) alloc_hooks(vmalloc_array_noprof(__VA_ARGS__)) + +extern void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vcalloc(...) alloc_hooks(__vcalloc_noprof(__VA_ARGS__)) -extern void *__vmalloc_array(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vmalloc_array(size_t n, size_t size) __alloc_size(1, 2); -extern void *__vcalloc(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vcalloc(size_t n, size_t size) __alloc_size(1, 2); +extern void *vcalloc_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vcalloc(...) alloc_hooks(vcalloc_noprof(__VA_ARGS__)) extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); diff --git a/kernel/kallsyms_selftest.c b/kernel/kallsyms_selftest.c index b4cac76ea5e9..3ea9be364e32 100644 --- a/kernel/kallsyms_selftest.c +++ b/kernel/kallsyms_selftest.c @@ -82,7 +82,7 @@ static struct test_item test_items[] = { ITEM_FUNC(kallsyms_test_func_static), ITEM_FUNC(kallsyms_test_func), ITEM_FUNC(kallsyms_test_func_weak), - ITEM_FUNC(vmalloc), + ITEM_FUNC(vmalloc_noprof), ITEM_FUNC(vfree), #ifdef CONFIG_KALLSYMS_ALL ITEM_DATA(kallsyms_test_var_bss_static), diff --git a/mm/util.c b/mm/util.c index 27ed6a5ac31a..7e6043ece040 100644 --- a/mm/util.c +++ b/mm/util.c @@ -629,7 +629,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) * about the resulting pointer, and cannot play * protection games. */ - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } @@ -688,12 +688,12 @@ void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flag EXPORT_SYMBOL(kvrealloc_noprof); /** - * __vmalloc_array - allocate memory for a virtually contiguous array. + * __vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vmalloc_array(size_t n, size_t size, gfp_t flags) +void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; @@ -701,18 +701,18 @@ void *__vmalloc_array(size_t n, size_t size, gfp_t flags) return NULL; return __vmalloc(bytes, flags); } -EXPORT_SYMBOL(__vmalloc_array); +EXPORT_SYMBOL(__vmalloc_array_noprof); /** - * vmalloc_array - allocate memory for a virtually contiguous array. + * vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vmalloc_array(size_t n, size_t size) +void *vmalloc_array_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc_array); +EXPORT_SYMBOL(vmalloc_array_noprof); /** * __vcalloc - allocate and zero memory for a virtually contiguous array. @@ -720,22 +720,22 @@ EXPORT_SYMBOL(vmalloc_array); * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vcalloc(size_t n, size_t size, gfp_t flags) +void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) { return __vmalloc_array(n, size, flags | __GFP_ZERO); } -EXPORT_SYMBOL(__vcalloc); +EXPORT_SYMBOL(__vcalloc_noprof); /** - * vcalloc - allocate and zero memory for a virtually contiguous array. + * vcalloc_noprof - allocate and zero memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vcalloc(size_t n, size_t size) +void *vcalloc_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vcalloc); +EXPORT_SYMBOL(vcalloc_noprof); struct anon_vma *folio_anon_vma(struct folio *folio) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a3fedb3ee0db..62fd33094bfd 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3025,12 +3025,12 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * but mempolicy wants to alloc memory by interleaving. */ if (IS_ENABLED(CONFIG_NUMA) && nid == NUMA_NO_NODE) - nr = alloc_pages_bulk_array_mempolicy(bulk_gfp, + nr = alloc_pages_bulk_array_mempolicy_noprof(bulk_gfp, nr_pages_request, pages + nr_allocated); else - nr = alloc_pages_bulk_array_node(bulk_gfp, nid, + nr = alloc_pages_bulk_array_node_noprof(bulk_gfp, nid, nr_pages_request, pages + nr_allocated); @@ -3060,9 +3060,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid, break; if (nid == NUMA_NO_NODE) - page = alloc_pages(alloc_gfp, order); + page = alloc_pages_noprof(alloc_gfp, order); else - page = alloc_pages_node(nid, alloc_gfp, order); + page = alloc_pages_node_noprof(nid, alloc_gfp, order); if (unlikely(!page)) { if (!nofail) break; @@ -3119,10 +3119,10 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, /* Please note that the recursion is strictly bounded. */ if (array_size > PAGE_SIZE) { - area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, + area->pages = __vmalloc_node_noprof(array_size, 1, nested_gfp, node, area->caller); } else { - area->pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node_noprof(array_size, nested_gfp, node); } if (!area->pages) { @@ -3205,7 +3205,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } /** - * __vmalloc_node_range - allocate virtually contiguous memory + * __vmalloc_node_range_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @start: vm area range start @@ -3232,7 +3232,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, * * Return: the address of the area or %NULL on failure */ -void *__vmalloc_node_range(unsigned long size, unsigned long align, +void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) @@ -3361,7 +3361,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, } /** - * __vmalloc_node - allocate virtually contiguous memory + * __vmalloc_node_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @gfp_mask: flags for the page level allocator @@ -3379,10 +3379,10 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, * * Return: pointer to the allocated memory or %NULL on error */ -void *__vmalloc_node(unsigned long size, unsigned long align, +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) { - return __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, align, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, 0, node, caller); } /* @@ -3391,15 +3391,15 @@ void *__vmalloc_node(unsigned long size, unsigned long align, * than that. */ #ifdef CONFIG_TEST_VMALLOC_MODULE -EXPORT_SYMBOL_GPL(__vmalloc_node); +EXPORT_SYMBOL_GPL(__vmalloc_node_noprof); #endif -void *__vmalloc(unsigned long size, gfp_t gfp_mask) +void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node(size, 1, gfp_mask, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, gfp_mask, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(__vmalloc); +EXPORT_SYMBOL(__vmalloc_noprof); /** * vmalloc - allocate virtually contiguous memory @@ -3413,12 +3413,12 @@ EXPORT_SYMBOL(__vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc(unsigned long size) +void *vmalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc); +EXPORT_SYMBOL(vmalloc_noprof); /** * vmalloc_huge - allocate virtually contiguous memory, allow huge pages @@ -3432,16 +3432,16 @@ EXPORT_SYMBOL(vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL_GPL(vmalloc_huge); +EXPORT_SYMBOL_GPL(vmalloc_huge_noprof); /** - * vzalloc - allocate virtually contiguous memory with zero fill + * vzalloc_noprof - allocate virtually contiguous memory with zero fill * @size: allocation size * * Allocate enough pages to cover @size from the page level @@ -3453,12 +3453,12 @@ EXPORT_SYMBOL_GPL(vmalloc_huge); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc(unsigned long size) +void *vzalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc); +EXPORT_SYMBOL(vzalloc_noprof); /** * vmalloc_user - allocate zeroed virtually contiguous memory for userspace @@ -3469,17 +3469,17 @@ EXPORT_SYMBOL(vzalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_user(unsigned long size) +void *vmalloc_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_KERNEL | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_noprof); /** - * vmalloc_node - allocate memory on a specific node + * vmalloc_node_noprof - allocate memory on a specific node * @size: allocation size * @node: numa node * @@ -3491,15 +3491,15 @@ EXPORT_SYMBOL(vmalloc_user); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_node(unsigned long size, int node) +void *vmalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_node); +EXPORT_SYMBOL(vmalloc_node_noprof); /** - * vzalloc_node - allocate memory on a specific node with zero fill + * vzalloc_node_noprof - allocate memory on a specific node with zero fill * @size: allocation size * @node: numa node * @@ -3509,12 +3509,12 @@ EXPORT_SYMBOL(vmalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc_node(unsigned long size, int node) +void *vzalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc_node); +EXPORT_SYMBOL(vzalloc_node_noprof); #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL) @@ -3529,7 +3529,7 @@ EXPORT_SYMBOL(vzalloc_node); #endif /** - * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) + * vmalloc_32_noprof - allocate virtually contiguous memory (32bit addressable) * @size: allocation size * * Allocate enough 32bit PA addressable pages to cover @size from the @@ -3537,15 +3537,15 @@ EXPORT_SYMBOL(vzalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32(unsigned long size) +void *vmalloc_32_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32); +EXPORT_SYMBOL(vmalloc_32_noprof); /** - * vmalloc_32_user - allocate zeroed virtually contiguous 32bit memory + * vmalloc_32_user_noprof - allocate zeroed virtually contiguous 32bit memory * @size: allocation size * * The resulting memory area is 32bit addressable and zeroed so it can be @@ -3553,14 +3553,14 @@ EXPORT_SYMBOL(vmalloc_32); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32_user(unsigned long size) +void *vmalloc_32_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_VMALLOC32 | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32_user); +EXPORT_SYMBOL(vmalloc_32_user_noprof); /* * Atomically zero bytes in the iterator. From patchwork Tue Oct 24 13:46:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2DC5C07545 for ; Tue, 24 Oct 2023 13:48:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74E976B0293; Tue, 24 Oct 2023 09:48:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D7F56B0294; Tue, 24 Oct 2023 09:48:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 529F66B0295; Tue, 24 Oct 2023 09:48:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 37FD26B0293 for ; Tue, 24 Oct 2023 09:48:00 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E0352160379 for ; Tue, 24 Oct 2023 13:47:59 +0000 (UTC) X-FDA: 81380483478.14.350F364 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 18BBE40004 for ; Tue, 24 Oct 2023 13:47:57 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qJAAccNh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3Dcs3ZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Dcs3ZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155278; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YqQmkpquk/zEN/BrXUtfRL7OAs/mhY9q/VQlYA3NCSg=; b=Zv+jxP4YbXl4jLc9+tgGL4ZIpO0NrkQpx7yiuaeg+ScNZnHJLM2eBciz/1aYjdskBhR1d9 /zs3FMobRXpC9vvpXJcyqEsOHmU/bkhU8KLzVOU3Dx+euPoJo2CnnQGt7+rKD+2mZZLN2S okShWfuFutq+caIzOmtsxt7PBgzM4DQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qJAAccNh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3Dcs3ZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Dcs3ZQYKCLIkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155278; a=rsa-sha256; cv=none; b=glUfmuaoWrER8jjUKbDR8clNZXFOE0ELaJ9Fj9cGzr9WRAdL86znaKKjJAEWStZkWmmXXy SqOgwJ3zmDwzFNUK6zyEC53YR/cOwR8XDG4oB3/naBpxaW+yLjPC+6yj6xYvQhb/OWNvFF JV8H/wHNCFFlUq+jQqDHLiaez4FncQs= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5a7be940fe1so59933947b3.2 for ; Tue, 24 Oct 2023 06:47:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155277; x=1698760077; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YqQmkpquk/zEN/BrXUtfRL7OAs/mhY9q/VQlYA3NCSg=; b=qJAAccNhaj/gexH3aGi0c+/OwCkZnpGyO5b4NEYjwji5AryHWm2/icq1Hv57B7beLo 737LriC3VpR8osB0t5MAS3f/ZuTX3evRXELG1GALCw4VMsprBuSuBastZzn3TsXaMgaT DTq2pe2VI/GJgw599mdzZDkXETHZaFMxBBusgRlBMx6BMKGcA60TcoR+i+SZCX4WqC4Z ZbRToXccm9FhdIg1xNHZx5t9OAunWfNpH0Hn30I38cNku/Jjam/LiCvopleQjuk58N+k 0QB9GF+UsEPuUKpu5hs9Lnx4ptUND2tUAhyPLpWZT4etvhYkarDh2TLwlC3GjwU8GoGq xZvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155277; x=1698760077; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YqQmkpquk/zEN/BrXUtfRL7OAs/mhY9q/VQlYA3NCSg=; b=pxmj/BrR4eyh0ojJPlfL/YyINCC4HC67HQTcbaL6Ptrln/tgq7QAWNlJQCxeuHoeVO 2c4e8vMQAGusTRicfKblytBDNeT7YFL76gANpkockInmFheRfGmu5UzY6S90ohXGjJ3P rIqbTXo+fXY3pB48324HySItxKv7Y5NR6IlaMcLVpOYT1HdB2iUKJF8j1c7H10kLnXLF QADM6W5ZZb2x9tyz8Upc2lkIP3pJh+An42AA492ramh5yykaNND9Iba4plAtDTALeIYT hnh+IpXfIEn3FTmDq0Soi9Ejv30K1oTcycaOrs8cdIC7ve8N2hWobWWdHmYsxnZMdliL fx0w== X-Gm-Message-State: AOJu0YyCVyBA2Sti3pyWzuqxtzKpAK4DfoEPzul8N/7eIJhG2p1knaMS RIi4fIE9JFRtUYsNn4gnEQxPIXQb/C8= X-Google-Smtp-Source: AGHT+IH6wvOVzrOaaGyY1/e57eYHLfTZ/5bwoybFh+Q+IMjNK5yyYqtlQtnc2beg11hZB/FoRq4QkUDgnBs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a81:4853:0:b0:5a8:3f07:ddd6 with SMTP id v80-20020a814853000000b005a83f07ddd6mr266393ywa.6.1698155277184; Tue, 24 Oct 2023 06:47:57 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:31 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-35-surenb@google.com> Subject: [PATCH v2 34/39] rhashtable: Plumb through alloc tag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: t5uycdeqbnjbrkmqn6pm8b84kzgk4poi X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 18BBE40004 X-HE-Tag: 1698155277-913492 X-HE-Meta: U2FsdGVkX19siWhMygcTB/KExBAtwyQdqrFYAmwXr0iIhg1lJLJ7CFAr32fRwLgvmjPqpB90It3SECTE7Iu0HH7JZ++wBNRE183B7doMWfY9Iwshg8kFeFBq6hyNHhkrSZbzbMVo0ikbeFKw9XnpplcHnFtt6EVtHUs45ynZH+u1g4WFYRFHuT9k5d2cqTcV86ub9CGqpWHJJ48ml4sibN7OoWVlErnxObi3R+WOdKRLQkvq0f7FVX7Gl41OKqkWut1IVVVcFakuowUA79TIs9ePDPqzvcalnpSlyb3l80M9HA2dqHUt3MVt/QpE4Ui/u7HoGCECEWs9J688YqHdYlCvRCrJOH//V8F4UN/pdtZSvVSC2+BAYCyUyqcQvw/ft+GoO+6K1R5rcPJcTEGfbM8GQmwGhdKI61kjxdq+GNJ+WiukMC1Gzma793WsYT7HhZW5l0ZmeUCsQyZD0NDKaFhf3omytch9TlAM30L48d4LDT/xiWIqZQOktv18oMvYmGA5V976rMu2aZq/4lhv7KZQzE14cfJAjoGn1d0ewAfCI9EUgwwKuP0OnOz3gWOJEmuNkcFHby47QZJJTtwKF8lo3MVc/DCKpfZ5mylMjpouoq9VOC8hNsPcnlS1OHHziXxtQSWN7G49+lxAOSb3wk4PRwAxlrkyv5ERw/YCoikY7ZCf6OXzZMX9+zrV3SJ979uTBvyfDtf+oA3LXfVKTeOkI/CElY4P9V61Mwpy6fZ8M/pp97weg4oZgm8dhmJCdOTLAz1uKFhp2pUqEVPkczxrOBqd6Lu1CBdizshEeGohqjSzLoKAqURZpQZsvUI+1CTnEZBa69PPmgdx0ZrfuP6/BNGnMVnvh5JCEHi5uoOcq2E5tK+AQJX8DD9SajEVVRoFzIzkpLjFuXwzuZjbG8o+64FtNoTNlgg22Lw/bX96bKcwLECRbIOdZPBR64LoBc3/aZSwH0wULWxDg5K OlCZdnHZ kBlE5X/w8ARzgpJM5DjJVUCTj7w/z0nAjc1hb+XeWtyjcOIPZcsfhiEFPhJGxNWEmE8vF+7IGYgSZua8LqbW1BuBPvECXPpRRfd+KD0ck8vjLMjWjc4dNLjb6WQKIab7WnqRDqFe57vJCDOjMO83O2hfVYhf8R7se55Cev/lXjZz7QHTSiMi0WtHE9epQboTPhlc495Rg/QMCdfZKwSrxYfjaTNrD53k/JHNq+nXZoA+oLimORk7S0BoKPpspDr8cEEEg8z0wnNkFM9inoFhleGoRJ3c5ApKmoW4o9XCpNMavr3wZfFH5bjXPUn2EngFFFgZRvnHGTFOJHVQazF+N0WVCMJCA45iR1sJ6d8fOwmZpZF96jCCOtCeMpuh7qmvpSVj613NlSQc2wKE5mk8QlX74hviLnEtiSu80V5NZtlANHO9fy1S8danVFCYgsC2AYM0Nwzr4BuMV7HIk9H5i89KYbJ+tSV7XTpr3PdLp82ngroCsKCZ/gjvzwwTk1OWGO+Ne0O7LEjtCpudyfSfnlfOnOOCODwLKHp71NtHkRlXv7ZdwpW81Il2fQe9LVKv7Q162EP0z7hJFvIZmiu1thVz3u+KVz4kUOgjClwZb/WfnWEOT08edvnx5rpj4e0mdsRNAEak3oecjZX++Zap6vLcagUZDcqX4m1OOgCuDPgJwNFA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This gives better memory allocation profiling results; rhashtable allocations will be accounted to the code that initialized the rhashtable. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/rhashtable-types.h | 11 +++++-- lib/rhashtable.c | 52 +++++++++++++++++++++++++------- 2 files changed, 50 insertions(+), 13 deletions(-) diff --git a/include/linux/rhashtable-types.h b/include/linux/rhashtable-types.h index 57467cbf4c5b..aac2984c2ef0 100644 --- a/include/linux/rhashtable-types.h +++ b/include/linux/rhashtable-types.h @@ -9,6 +9,7 @@ #ifndef _LINUX_RHASHTABLE_TYPES_H #define _LINUX_RHASHTABLE_TYPES_H +#include #include #include #include @@ -88,6 +89,9 @@ struct rhashtable { struct mutex mutex; spinlock_t lock; atomic_t nelems; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif }; /** @@ -127,9 +131,12 @@ struct rhashtable_iter { bool end_of_table; }; -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params); -int rhltable_init(struct rhltable *hlt, +#define rhashtable_init(...) alloc_hooks(rhashtable_init_noprof(__VA_ARGS__)) + +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params); +#define rhltable_init(...) alloc_hooks(rhltable_init_noprof(__VA_ARGS__)) #endif /* _LINUX_RHASHTABLE_TYPES_H */ diff --git a/lib/rhashtable.c b/lib/rhashtable.c index 6ae2ba8e06a2..b62116f332b8 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -63,6 +63,27 @@ EXPORT_SYMBOL_GPL(lockdep_rht_bucket_is_held); #define ASSERT_RHT_MUTEX(HT) #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline void rhashtable_alloc_tag_init(struct rhashtable *ht) +{ + ht->alloc_tag = current->alloc_tag; +} + +static inline struct alloc_tag *rhashtable_alloc_tag_save(struct rhashtable *ht) +{ + return alloc_tag_save(ht->alloc_tag); +} + +static inline void rhashtable_alloc_tag_restore(struct rhashtable *ht, struct alloc_tag *old) +{ + alloc_tag_restore(ht->alloc_tag, old); +} +#else +#define rhashtable_alloc_tag_init(ht) +static inline struct alloc_tag *rhashtable_alloc_tag_save(struct rhashtable *ht) { return NULL; } +#define rhashtable_alloc_tag_restore(ht, old) +#endif + static inline union nested_table *nested_table_top( const struct bucket_table *tbl) { @@ -130,7 +151,7 @@ static union nested_table *nested_table_alloc(struct rhashtable *ht, if (ntbl) return ntbl; - ntbl = kzalloc(PAGE_SIZE, GFP_ATOMIC); + ntbl = kmalloc_noprof(PAGE_SIZE, GFP_ATOMIC|__GFP_ZERO); if (ntbl && leaf) { for (i = 0; i < PAGE_SIZE / sizeof(ntbl[0]); i++) @@ -157,7 +178,7 @@ static struct bucket_table *nested_bucket_table_alloc(struct rhashtable *ht, size = sizeof(*tbl) + sizeof(tbl->buckets[0]); - tbl = kzalloc(size, gfp); + tbl = kmalloc_noprof(size, gfp|__GFP_ZERO); if (!tbl) return NULL; @@ -180,8 +201,10 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, size_t size; int i; static struct lock_class_key __key; + struct alloc_tag * __maybe_unused old = rhashtable_alloc_tag_save(ht); - tbl = kvzalloc(struct_size(tbl, buckets, nbuckets), gfp); + tbl = kvmalloc_node_noprof(struct_size(tbl, buckets, nbuckets), + gfp|__GFP_ZERO, NUMA_NO_NODE); size = nbuckets; @@ -190,6 +213,8 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, nbuckets = 0; } + rhashtable_alloc_tag_restore(ht, old); + if (tbl == NULL) return NULL; @@ -975,7 +1000,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) } /** - * rhashtable_init - initialize a new hash table + * rhashtable_init_noprof - initialize a new hash table * @ht: hash table to be initialized * @params: configuration parameters * @@ -1016,7 +1041,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) * .obj_hashfn = my_hash_fn, * }; */ -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params) { struct bucket_table *tbl; @@ -1031,6 +1056,8 @@ int rhashtable_init(struct rhashtable *ht, spin_lock_init(&ht->lock); memcpy(&ht->p, params, sizeof(*params)); + rhashtable_alloc_tag_init(ht); + if (params->min_size) ht->p.min_size = roundup_pow_of_two(params->min_size); @@ -1076,26 +1103,26 @@ int rhashtable_init(struct rhashtable *ht, return 0; } -EXPORT_SYMBOL_GPL(rhashtable_init); +EXPORT_SYMBOL_GPL(rhashtable_init_noprof); /** - * rhltable_init - initialize a new hash list table + * rhltable_init_noprof - initialize a new hash list table * @hlt: hash list table to be initialized * @params: configuration parameters * * Initializes a new hash list table. * - * See documentation for rhashtable_init. + * See documentation for rhashtable_init_noprof. */ -int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params) +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params) { int err; - err = rhashtable_init(&hlt->ht, params); + err = rhashtable_init_noprof(&hlt->ht, params); hlt->ht.rhlist = true; return err; } -EXPORT_SYMBOL_GPL(rhltable_init); +EXPORT_SYMBOL_GPL(rhltable_init_noprof); static void rhashtable_free_one(struct rhashtable *ht, struct rhash_head *obj, void (*free_fn)(void *ptr, void *arg), @@ -1222,6 +1249,7 @@ struct rhash_lock_head __rcu **rht_bucket_nested_insert( unsigned int index = hash & ((1 << tbl->nest) - 1); unsigned int size = tbl->size >> tbl->nest; union nested_table *ntbl; + struct alloc_tag * __maybe_unused old = rhashtable_alloc_tag_save(ht); ntbl = nested_table_top(tbl); hash >>= tbl->nest; @@ -1236,6 +1264,8 @@ struct rhash_lock_head __rcu **rht_bucket_nested_insert( size <= (1 << shift)); } + rhashtable_alloc_tag_restore(ht, old); + if (!ntbl) return NULL; From patchwork Tue Oct 24 13:46:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434604 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ED2FC25B6C for ; Tue, 24 Oct 2023 13:48:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D7026B0295; Tue, 24 Oct 2023 09:48:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85E276B0296; Tue, 24 Oct 2023 09:48:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6885F6B0297; Tue, 24 Oct 2023 09:48:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 514DF6B0295 for ; Tue, 24 Oct 2023 09:48:02 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 24D85C05EA for ; Tue, 24 Oct 2023 13:48:02 +0000 (UTC) X-FDA: 81380483604.16.084E74D Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 56C8E40014 for ; Tue, 24 Oct 2023 13:48:00 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=M0IOJniF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3D8s3ZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3D8s3ZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155280; a=rsa-sha256; cv=none; b=eN0qPribOYUDrs1fUO5QcTLBf1O1PEaHcKX+tJwlKQL4OnvNacBVBDIKFsvuANUzPpbtpn tmiYmXHaT/ZLYL4Zb8UJH9JYXjOcQoGTg4kgXxUHW+35dkqH1urb7+MA6YpnPbdu6oZ9EY coJQfe7xLtjc7vT5iTWWMaVev9qUYB4= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=M0IOJniF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3D8s3ZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3D8s3ZQYKCLQmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155280; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Od9SIx6nrlV00gepzGuthWnU/KWY+8bGDr+wczV24Vo=; b=CH5gwqB7oU3XDdq7qhvuZc/ax1cYsr3jolc7lYBy+C5vWuLRcI3+jB6k1rpvL17PuGvbuE K8qPsGg72AAjgF6CIcUXYypbT/9hyuOTGK3bRTrBk8GFCL/xPh5PU37aljHsKN5l62/2rM QhGCd7bgTCTGi96hROv9b648yoQTH/s= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a81a80097fso59827707b3.3 for ; Tue, 24 Oct 2023 06:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155279; x=1698760079; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Od9SIx6nrlV00gepzGuthWnU/KWY+8bGDr+wczV24Vo=; b=M0IOJniFTd9jebzGRC/Xir/OcHf65qHJYst5V0biamYmsAJg2yuJ74BTs/oeWORslA +SjiW2QJ43hPuhvUGi9hGlM0IVzpcASB13BMqiqx4sUC90VnEL8sYnIcj35Rt7zU1i87 T4gKGEDdN7Xty5UneEwDRx1ApJuOGIdl+UDo0Nfhk21csrJ0iWvEE4sWKi2GtZx5rg+j IZyKrGyRPGoP1E68DfrJDxxvWDZtJHHkmHeLaovIoNfWIxxfSPgcX7ugYJMNPYOOh8Nd WCIczr/jguEXoYVsZTmyrYtPTGvdXdZnJWaIJMA9II7Du5C18ZroEwTmr6dtACapR6F9 nKNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155279; x=1698760079; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Od9SIx6nrlV00gepzGuthWnU/KWY+8bGDr+wczV24Vo=; b=KURqpCgN6OYCPPunaloPF1/4/p1zhms5q4w++NH60h/UETWeEH/HSHcsKDRLo4YQ8x /8rSv76Bl9R6hewjZUAMt5OFfhFuPyiAeu1XvI3IG36j2JjZd2gbUv0dquSWeCxUEsHC ew3dKjpNxHHgiG/o3roibYDacNdlg6Yz4J6dljWYFOpL3Y1awWz5Wh4vPaKLhX4nDZeI ye+g1xqx5zcByenchMRfL4hxYptH8O7jLVrayGeHMwwfjY1En98m24CpukPI+p9+Jiat CBYiQI6PM5UXh4Vts3qzyGy7+LoWSeC442p3/nnkdD+dY0uNKD24y3wEtE9eMzQILhqD cuKw== X-Gm-Message-State: AOJu0YzIe18UoYV8f61qlSgqxBVSbLS7k8OCcq7UFecRoNuK9M52niKR 5+Gk2P2skYEMlgNIFxdE60Fg8vvXuSg= X-Google-Smtp-Source: AGHT+IHTNe96EdD9LLd7MUZqFW1q382ncfW7dWktSIhQfdb/UWURbia5QNDViYiMSeL9bXxhLhwSMXQz/Wc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:76cc:0:b0:d9a:68de:16a1 with SMTP id r195-20020a2576cc000000b00d9a68de16a1mr246429ybc.0.1698155279398; Tue, 24 Oct 2023 06:47:59 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:32 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-36-surenb@google.com> Subject: [PATCH v2 35/39] lib: add memory allocations report in show_mem() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 56C8E40014 X-Stat-Signature: hxi6tkhawuxzaxowh6jhozxecti8i188 X-HE-Tag: 1698155280-645463 X-HE-Meta: U2FsdGVkX18tVj4ECCbDQ/Zs2lpJvix/8Af4i1QED74AoVCtQ+ZHxsSfa7jVHpfs0l+nyTnjQOCSJnipQLrPq4gPiWt1wyxjjxwPu3JQGyyQqnUL9zvhgVO3ul/niGiTk1mD0DRcjTboWL2eF11gbrs2igBBr/k53m9LyVfpL4c4iAP59Mj6qMSqx2tpYnF7FC//D1Ls8yCuMQjZzPLcFJsRY4xRnVn7aVPRu9assCA9gGw0WRRczVSaF+I/ggS16vpWN1J/Iz4OMP1kiPgXNb74TZheCItzsuQHczg/NOn6pSIizAQUd2z+R/4V+wRWAknSe4ZbHmUX5CzKLxJTNS9AfgNFiKUm4vp/OCwuKsaqOv7HvwdyKdEXJJnLnTcoPMrZLCPYZzb/Obnf4ZD/FVKtliDes/KIhINhLFHbJi2IQeLqgkHyxy9daI/W0RPinonj8OzKpydYXgOTigZ7y6OfCA0tR8AyWyaq/CkUSbWRZ+iKceVyN9JFoAPD8QmSy3Kzxxz3KhkY2Qm632pc3TAohFtsqQ0KAUEFQZRXxZrkRICttAQJw2Wn7FQx//tb9FVZm121zBqemgRyAS4IB4CJXGE7AJXhAgN/19Fid8WWFKzPJdnfSKtnHbCZt6LfuiyiVhiETErgZHIRMGRRHJvfU+aDwjNmYcqrU5h6saMw8YRy1sTZCO8IJrRWwbsIB2tkFgxQzHPg0AqoTri6+4fqlLCfZnN3u6qGMgxWps5Vo8wK6EQGaGnWX3FVVzLA477Gzihr3zglWBsXBOT5CPoIXw5ZxWbRFPjO9DW8TfbpLLf8G00tckbqxq4AaI8B9H8yVbfgfih+nVD+Nn/y8Acel0JVwQdrEN/RHvhP2VGNFenygmSLFfgtm4kU8Ptq+tw22AKPAqQlRdwDRI/Asoa18oTIygGiANY+aQOiahW3VIYzVDt3B+85ZwSnnzdo2WyadB2KOUXm+RUdvA0 0l8OmoLC 1wV8/hIj3382/8A5BexR1RYfdgcDQeiZOU2j/e/Wtj0PFMHUIl09Tf9mGe/R6Faj1jTPBuu9b+rcQ6AKIqtWYfI1N6PQ1ui/5i8apey26r+cl1JYaEIEVNJeUhjGQbabcxO0paZJeXSYL9ZswleuIJ7VGyr4A8Ub3DZwz8cyqzUrvasl3RhRGeTwzaRq62fUNTAsY2cuBFgf31OdJagU3dzPM8WtTZ0uyHj9HZXy7qkOuk41htc+50hqZJQqkeWWyMDhWL9p7vwJle5LB8lI8QHHYw2vd9pACt59LBdRzYY9kCfxcQ+pokI8K4Sn0h2eVxpRIBplD3KQSqGkfdAZFi4ekXtbNom+zXy/TCVW/5t35xfvHuuq3CO6z2pNTpPn57tt6xdG2Ty9D6fd9kWn4VM1ToupZGXF7h28GTL8QMjhk/W5mqXHVEVa7R5tr8MgCy+71wCFm0uvIJL17GR6suegpBj4F2T98gmvUKjwzkT6GVbyiuFyQGLs4t3mIMEWT4m+8BQtdeXFbCwc7C7MVZnFVEFQdEk4X5ph03pqDlX9VHZum4CTO87LeOiwDKnGSRFJb35u/cqak1s8xuTZICa0UYt+yQZbAMUQHC5acKtBupF92ei9zmh1dWsVHk5DIf+0bjlmENaxzdpUNngfmZTNNmP/zqi12ZntJSmGkpAhpwiU+oZRrpZ1HJDkvy5f2FIV344aaRF9nP8pQ0Ss5+PYCxp4EhG06mGjS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Include allocations in show_mem reports. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 2 ++ lib/alloc_tag.c | 37 +++++++++++++++++++++++++++++++++++++ mm/show_mem.c | 15 +++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 3fe51e67e231..0a5973c4ad77 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,8 @@ struct alloc_tag { #ifdef CONFIG_MEM_ALLOC_PROFILING +void alloc_tags_show_mem_report(struct seq_buf *s); + static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) { return container_of(ct, struct alloc_tag, ct); diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 2d5226d9262d..2f7a2e3ddf55 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -96,6 +96,43 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +void alloc_tags_show_mem_report(struct seq_buf *s) +{ + struct codetag_iterator iter; + struct codetag *ct; + struct { + struct codetag *tag; + size_t bytes; + } tags[10], n; + unsigned int i, nr = 0; + + codetag_lock_module_list(alloc_tag_cttype, true); + iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(&iter))) { + struct alloc_tag_counters counter = alloc_tag_read(ct_to_alloc_tag(ct)); + n.tag = ct; + n.bytes = counter.bytes; + + for (i = 0; i < nr; i++) + if (n.bytes > tags[i].bytes) + break; + + if (i < ARRAY_SIZE(tags)) { + nr -= nr == ARRAY_SIZE(tags); + memmove(&tags[i + 1], + &tags[i], + sizeof(tags[0]) * (nr - i)); + nr++; + tags[i] = n; + } + } + + for (i = 0; i < nr; i++) + alloc_tag_to_text(s, tags[i].tag); + + codetag_lock_module_list(alloc_tag_cttype, false); +} + static void __init procfs_init(void) { proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); diff --git a/mm/show_mem.c b/mm/show_mem.c index 4b888b18bdde..660e9a78a34d 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -426,4 +427,18 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) #ifdef CONFIG_MEMORY_FAILURE printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { + struct seq_buf s; + char *buf = kmalloc(4096, GFP_ATOMIC); + + if (buf) { + printk("Memory allocations:\n"); + seq_buf_init(&s, buf, 4096); + alloc_tags_show_mem_report(&s); + printk("%s", buf); + kfree(buf); + } + } +#endif } From patchwork Tue Oct 24 13:46:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434606 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5292C07545 for ; Tue, 24 Oct 2023 13:48:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D90AA6B0296; Tue, 24 Oct 2023 09:48:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D199F6B0297; Tue, 24 Oct 2023 09:48:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B42CD6B0298; Tue, 24 Oct 2023 09:48:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9AFEF6B0296 for ; Tue, 24 Oct 2023 09:48:04 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7B692160172 for ; Tue, 24 Oct 2023 13:48:04 +0000 (UTC) X-FDA: 81380483688.29.9CB6F19 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf21.hostedemail.com (Postfix) with ESMTP id B2F1E1C0018 for ; Tue, 24 Oct 2023 13:48:02 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=iJDQLbhz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3Ecs3ZQYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Ecs3ZQYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155282; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+3jXXsjNPLXrWv+rf69UQa/6uvXnoVOikFRei2b/KT8=; b=oE1LdW9Ol6kDCu1QgH/2+g80WTtmUuOTL6247xIyOKBCMWiTh8OLC4eOvAYbteq5Uhh2vk aTtbHAG5yeI39OOHh6GmJzRVCYAUYos5RnbduGYCKD0Bp4UJ0QpXewelxgOLzUf709aYD0 YxQwo+rcmoaTYRCyNv+4fEIYNkFX3Pk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=iJDQLbhz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3Ecs3ZQYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Ecs3ZQYKCLYoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155282; a=rsa-sha256; cv=none; b=nHdh5gk0WeM3DfPev2Dna5zWPZl6bbgIeJTFhQKac7zXJ0HoR4LBxvW/Dt+CloZwSVJGRB ns2CdFa79MROFJGD2ZLRBJB4oUhq8ZqMkNwqDb2KdysgWgCuiXwcG1BBBkKT4YSnD5uYSj Y8p1TtT2fFZiBRdWTzmIvnCp/RcLjr0= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9a541b720aso5409628276.0 for ; Tue, 24 Oct 2023 06:48:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155282; x=1698760082; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+3jXXsjNPLXrWv+rf69UQa/6uvXnoVOikFRei2b/KT8=; b=iJDQLbhz/W9d3D8wBDpO8wxSCvw23tusRpCmZ2yXAvvmyjoS0+LFMXibtuZNf/Uq+s v95I5qB9l66r8YhBN6zIgpH1+QFQ8Knk+9MJETtySq8cT3dvyPuinZ9i5ySquTycDX6E 4n8uhR1lwS4QT6a0zStU1NWLWUK5k4+naPcgSlOMtNL7a9tWXVs0sV9JxUxSObfJp/+Z laqjZvmTe6J5dy52JN65eLkbuPO28G5LWCIBIo2Wu/P8jUS6yhQfoxAlK8D6DAK4heEH U12ujNryAyXaKJ7WGknEh9vGjgofpzjL4UdAXPHJxofo1lrstgxbVOkwMSoGoUP0TD65 Es2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155282; x=1698760082; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+3jXXsjNPLXrWv+rf69UQa/6uvXnoVOikFRei2b/KT8=; b=gcMiCBr3cmxSZbMklRMNA8pqFf65Our6b2lWf6/fNUykVVJ6ihGHID3bQyFnmoDHuk yOxJudP4V0MOic9zZGElXW+t+rkRtqRb/8QhCJYzz+/1w94y7Ujh6ce+LSb90ur4lbuU NjuDU2WDsJzZTy4hXtIChtKNUc1IBtesExinI6o4KuzSFjukhtyKEaFOZAFm3i5Dezkv ByCAAy5+NAZAtJZKazxS62IGvstdzNj5c8cLPe4pOk3v6b59m1YeqZFmsN9hR8LRuoXs 4XwuUH2v4LaHVNMewXPSyaoeEH3EZogofUvjuCjCQoa35GDkviSFFz0ZLK8moPlaGeWD 7GHw== X-Gm-Message-State: AOJu0YxmD9xD81h5d6NkCVW32VpGOjbmWtvAMrezX1bUf8NLIh5bAbMd Bb3X79DtY4Fjg+kzjERfzoLoIO+sFWE= X-Google-Smtp-Source: AGHT+IH8yezIA8jxKZavrTPxkBvsmi2Yxz7WtkoiuiQPT9VSDtMO7/bIUQjwMdJw5xsPHzLAEf+sR1TuKL8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:d50c:0:b0:d9c:66d1:9597 with SMTP id r12-20020a25d50c000000b00d9c66d19597mr227720ybe.13.1698155281680; Tue, 24 Oct 2023 06:48:01 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:33 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-37-surenb@google.com> Subject: [PATCH v2 36/39] codetag: debug: skip objext checking when it's for objext itself From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: B2F1E1C0018 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: gbajfno3b8kwgsz4mundmm9gdbgitwtw X-HE-Tag: 1698155282-500195 X-HE-Meta: U2FsdGVkX19n+wXryDbtJi5WXOek24CYbVBiPz/cXyZjoYPuFisuiGhEPo+Cz7Ntgp+fx3GiNzPlviXFr44X4s9uriaidwERo2mBJu21j0liVrSBBW/FvaTViRpAXh4UPLSfWqljFqI/q+WlkDml5JAcS0Q2UAOVbC7dGdgtfrMjqgXIxXcwb+Ex+ykfE0b843ZQE6bYECiyhrq2elBfhwFh1fPAeSR41t5hay5iqBSO4qNm/rucboPtBjZRWZ/t547YUbLrp+bmlAWW5dHUoA//HlpSlTcj/xSI3s6ZVdnnZ+Ca/cvlLTziw9ZJE0rmPof9HPCzuFHTWxeBPmNIpkzUp+n9Op+ubPiSM2EvJsaQILu9RfPee3tG32VZUMSw1llHcd1fHHgKFhSZNhfjqm4ajhldBPg0BEAptykKHYXpjXG5pYfGxwaXf4L36wns8K8AZzeTst9jRdxPI4YDlhDkM3li85tu9QMabN1a/2ZY+iyldm1Oj+EcDtOLqYYtsI/YV0dX1P9JdANJt9KkpiizsmTSqDLYH1sw3f+fhItKE/y6RtkPqG5/BSQ7roGYsMHO4OdyyYbg6ElTCudNwKoazraIO11E20P699L/9EGa653sgv4jffcXeIrnAYmSglXtXkjZXT8p6iYhfx8EMTnC8nxZPXrH7I0xL5yhmOtPL9nYkob52u7v6SVwV0l/Y7ax13AJegVp2MPXk/2rD2UsmUZPrQiYGYT3C6j6ecDKary9c4D8GV1ztExQ35U7xqZrv1XOpCAKIXkPnu1V6NDBjSXXL3X51XCW5I9Pv8AANeBcLJeoLnjV9b1/nR4xdda2vRCwNWHxVHCRt19Fc2CF9vgGjoK6Q06pupYtvpIRxV3185Ie4Muju6NPHCXrjYvF2ZSQBSq0eXEY1znuYIVzVBRCARD0w4I5cElhil2ESEUBOfwEQ4MtknWETPp4NypX+j1fwGxoNH3bi0T yZ79q22h 6JcwOLjQU9g/ieSWxyupS/fq4zgJDVf0Mfg/1URoQ56rNRC/ARHDC9RVST2FEiyCs1NomlEyrAko1XQ9ozrBN5BTrd4LQ9wZLutRF7x0s9k3qy8hayipIS4wV5prT3prSLt175mSZw/w75OX2G+VHgD1maBlq2TJz4k6XPV0YfmfNvO3n9kBbWRsDYK521ZCVgGQtvERBHmGvICKZrOwL0f/1bpiBguMrpuccrO6cHUut19zhgtanZIhRmgarhx6PP77QUqhH+GisePQRrBmxFfA4IFUGqIF34S6QCh6xqvcVgvoF7aaHZDEHI+s/+uyFgo9Z5ZrLNCFjnP/Bp6wgVPoRFnVtgImfudD0L87hLVaDNyq6lxDedFHXHZcSOkXyYOWApvuyd8rG1o2xKftYPKeySZMAvWUGwHinrl8UVVAsEDPNqOo19YxO48Mt/br8KwPlbXJ2EQDFsJoZRxLrSotLMCYQgY2vIB4MYFF7o8iWi1CvXyWg2IpzTr21/YjK7OhATzm9VzxbT4NLkJVGck0jFxXbzwaDEDgEFWuEOTLPVGX8HPo5LpgOeP+u5nRYQ2sqwxo/oBF32gNL7MASxhliCl0bILwkTMgTAdI13QIQJ2vcuiLoqMEJlYmld/Y+yKYVFJZZX+hbZj/WxIqo1zFAD/m7visQ5k7+78dSmvj+zko= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: objext objects are created with __GFP_NO_OBJ_EXT flag and therefore have no corresponding objext themselves (otherwise we would get an infinite recursion). When freeing these objects their codetag will be empty and when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled this will lead to false warnings. Introduce CODETAG_EMPTY special codetag value to mark allocations which intentionally lack codetag to avoid these warnings. Set objext codetags to CODETAG_EMPTY before freeing to indicate that the codetag is expected to be empty. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 26 ++++++++++++++++++++++++++ mm/slab.h | 33 +++++++++++++++++++++++++++++++++ mm/slab_common.c | 1 + 3 files changed, 60 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 0a5973c4ad77..1f3207097b03 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -77,6 +77,27 @@ static inline struct alloc_tag_counters alloc_tag_read(struct alloc_tag *tag) return v; } +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +#define CODETAG_EMPTY (void *)1 + +static inline bool is_codetag_empty(union codetag_ref *ref) +{ + return ref->ct == CODETAG_EMPTY; +} + +static inline void set_codetag_empty(union codetag_ref *ref) +{ + if (ref) + ref->ct = CODETAG_EMPTY; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes) { struct alloc_tag *tag; @@ -87,6 +108,11 @@ static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes) if (!ref || !ref->ct) return; + if (is_codetag_empty(ref)) { + ref->ct = NULL; + return; + } + tag = ct_to_alloc_tag(ref->ct); this_cpu_sub(tag->counters->bytes, bytes); diff --git a/mm/slab.h b/mm/slab.h index 4859ce1f8808..45216bad34b8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -455,6 +455,31 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab); + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) +{ + struct slabobj_ext *slab_exts; + struct slab *obj_exts_slab; + + obj_exts_slab = virt_to_slab(obj_exts); + slab_exts = slab_obj_exts(obj_exts_slab); + if (slab_exts) { + unsigned int offs = obj_to_index(obj_exts_slab->slab_cache, + obj_exts_slab, obj_exts); + /* codetag should be NULL */ + WARN_ON(slab_exts[offs].ref.ct); + set_codetag_empty(&slab_exts[offs].ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + static inline bool need_slab_obj_ext(void) { #ifdef CONFIG_MEM_ALLOC_PROFILING @@ -476,6 +501,14 @@ static inline void free_slab_obj_exts(struct slab *slab) if (!obj_exts) return; + /* + * obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its + * corresponding extension will be NULL. alloc_tag_sub() will throw a + * warning if slab has extensions but the extension of an object is + * NULL, therefore replace NULL with CODETAG_EMPTY to indicate that + * the extension for obj_exts is expected to be NULL. + */ + mark_objexts_empty(obj_exts); kfree(obj_exts); slab->obj_exts = 0; } diff --git a/mm/slab_common.c b/mm/slab_common.c index 8ef5e47ff6a7..db2cd7afc353 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -246,6 +246,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, * assign slabobj_exts in parallel. In this case the existing * objcg vector should be reused. */ + mark_objexts_empty(vec); kfree(vec); return 0; } From patchwork Tue Oct 24 13:46:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7B2FC00A8F for ; Tue, 24 Oct 2023 13:48:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37E976B0298; Tue, 24 Oct 2023 09:48:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E1D66B0299; Tue, 24 Oct 2023 09:48:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1330C6B029A; Tue, 24 Oct 2023 09:48:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EDD9A6B0298 for ; Tue, 24 Oct 2023 09:48:06 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CCFE41205CD for ; Tue, 24 Oct 2023 13:48:06 +0000 (UTC) X-FDA: 81380483772.28.BAED71E Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf26.hostedemail.com (Postfix) with ESMTP id F2889140018 for ; Tue, 24 Oct 2023 13:48:04 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rGv7NPgr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3E8s3ZQYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3E8s3ZQYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155285; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nhDAtTQGKDGtY2/txJr7AfaYGvHf9XTIsotRfIG2XI4=; b=YTHqPPylkunSrEs4jgpo5uCuwQskHFydsYEDrcukMTNNdbwsNWxG0QDoRcastQ7At5ZbuX +2Wpqa4RRrVOrMwytdk+KJK/edXzqp3hEf1++H5xk+Gqyoz6rzHGXlrzd9ufrSMrKT7wSG 1RvoBmeA/pvhRDKnn/C8FLmu6GHIabA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rGv7NPgr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3E8s3ZQYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3E8s3ZQYKCLgqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155285; a=rsa-sha256; cv=none; b=d8slMygfXuPZTx+xMH6sfXKy/9NtdWRWG+Sjluo4uDjvD+gGBPTin1v37/mX0vA9zlEE10 4a6M9uGFIiFaTv4jKz8Yhdsu0cBTjpRqVnssMKQqVGok98DU9l+4guFeeNHA5Q9HClxMbq +08R0qHMQCkBduuEsArspRBqKRTbZzU= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5a7ac9c1522so62114527b3.0 for ; Tue, 24 Oct 2023 06:48:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155284; x=1698760084; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nhDAtTQGKDGtY2/txJr7AfaYGvHf9XTIsotRfIG2XI4=; b=rGv7NPgrK3sWoNBex3wMkbmmOknlTmaJJIwZLOcp3x+OlEZdHGLJeBMTK5sxFMCcUx 2XHca4eR5kPaBolRltvdOu4zLzAv1Evn9ab5NNcZcFque1QYezfr7eKvauuAJD94PmZ6 G1I0ON1gTbqEwOhJFsKKuXl3hdOzekft5RjneCDSDKHHTLzBKD7O/SnCiAJnMIUIZu9D l7O396cWH2vaeJ9P+QZpEGQnGpDz8xOcFtg0ptgC7IB8WogapjE5a4ukP8HjN8xy6n3m y22UbgUz8y0KIFJgu5RnrTEIXH/4BLwqhZNR+34fb0C8SK1EjMhZZ3Pa0cwCZSUX/jKh L/5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155284; x=1698760084; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nhDAtTQGKDGtY2/txJr7AfaYGvHf9XTIsotRfIG2XI4=; b=DB0qfj2Mdsql8UkFT6aaTKrT1JFrJqNIxoeS0LZNBLiRwd4ekv2dvwG31URsTqya28 NDsUddqrButmrga3g5uRaByzbi8cQHOLyvivpFgbN1sLPcpCl+HqK9vAE1vDPTZ+mX7o 8j70toC7JHUionqKzrCOWGttA71CxPBI6tutOmbhWhb1fdtyv5edX6UqUkq/0feRxZ9N Hoe5I6rTkw4++3di00apgcGIpXvDkWD0vGVNV05nOjrh0HDgbBJF7MEbdYKRKsztEsoy LH4T3EZco9C5r5LeZGU73GBr/AGHdIXZJ8Q0yrUnLkJpBhKUizj9H2PlK3/UMf3kmUKd tZOA== X-Gm-Message-State: AOJu0YzoNdwYL9jO87zD8YOUdRaYSTWpBEcm7fzWYdl7hWi/PX9TDioj mD9SpewyQEm6/SK/HpFHHcUAJbrlboc= X-Google-Smtp-Source: AGHT+IEqWGHLJsK7BvvDJRxELns2H70hhRTn4578toCcoqAqCoHJoeJS8HRj9/dFP0FFd7ukqdr613N1OPo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:8541:0:b0:d89:b072:d06f with SMTP id f1-20020a258541000000b00d89b072d06fmr231639ybn.7.1698155283973; Tue, 24 Oct 2023 06:48:03 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:34 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-38-surenb@google.com> Subject: [PATCH v2 37/39] codetag: debug: mark codetags for reserved pages as empty From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: 6tuwgkdchswaqnwu1ykjfb68ax11de61 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: F2889140018 X-HE-Tag: 1698155284-535850 X-HE-Meta: U2FsdGVkX1+KlcGx8xtb5aqXIc47zeNLIXCdgsimAnxWCs1/ywHU1pGp06c0uQhL2WWdBegdpeJfYUJpFAigyHhZW6vPD9b8xxf5rNYoWIPqIV5Pydrkiv9C6eQTQG8z/1fzfQSSi7RF9rdO7PahM0D1JZEtiMUR+TLxH1BmcGw3S7Z+VIJ/jp4ufQBcyphE74UHmHDkJNrVx9LRSNFT9fXtLVsuBSYrvcZA4riarp00fBOWarKWCPzefz3S/pSUowYokdIq0slqmzb+tD2zfo8ujk3Wu9guWP3Qlx0u87dD32tcYQ3BsNPn1ofB0vgoqhqkAasXkWlPXTqkcZht/KSyd82q6mc6X53pMSrgu/emu1E26F7tP5pyOVa6K61tMUyBLcp2BcPNEuDEO6s/OC8jnNiIwiOSvOlcI97qZ1ogIxqp470FIhlXvYtp8bVBquts1M8HwmjiwWIsD2/NfsDSU3CTh6AXQXcC/4LiWHLe7Fv2IOYA6qYpmQZlq5kjmFRnq6yARpcds9e0y44zHCvxDFVDNJhYTer8kYEHwaSwofevCM+WG3jjqruhqO8q4E5A0rFwGiHcoz54OvKw0cH6Y8gDWL28MDBTGi5FZKfipy0VI2EcJ7x7fIt0FEcBAUsgN2FccKA++dw1niXJT0XXm4/Y/BRPVfpwnXU3U0M1XdC+EHYlTSramqzpHIPoGkS4iJnYOqeBeA87zWETL2K+DZt/Zi2lqjV+ym/huFIgFfUPdePqWDb6HdWuWHt6UnYu1xOPWK03uIAb/rkwZ6tfC8aUZYk0nmDE0E9daroDjpvfskpPQpDK8b4eYC8W26mMMen2BKzNF9FyAFKw3x2M0BPbq6dF6iufHSBQdXh4YpHW5vcaM4BP1jSDDVUPNuH5M0jtThJLx/J35V3iBYtMfftBbpnnbMO7K+m8KiT7sMsQqKD4zRzog7iht6U1RIaVsZJpu/s9KSN9ixf 9xWUus7y CxaOOypLJhmi1H8FyOqfXofm8iQV2ZFmqmsUf+434dfkJsD941Mr10vu46EmtDi4jPgRv4qC6JLDSqe2MqhtMgrQq+LHuycfKnr94+11Ig+6grxM8hqmN7spojS1ZQRJ06OZ4BMQ7HPGQC0bAstFhcgWXgA/OKReXtavhuUJMocgn1/mSMNoTTO6HzYzIAWPazp2ooloGoatqMILH9Il0e1nqxsa64D8Fbm+H08I9dqpJv4tizV4RME5Qi06f4rp3UALrmn363bZvQupQLPR320WEpwIV9fcatl8yGBHJ4jKXRtl23dP8I8hzu+jEKop4qVVL9XspfsTrxEwSw89lKMUSiGHS1wVbHJGt1S+rt4OPEPKoYetaPp+fiOLZ41tTnT00fP6reKgHHOz0uFWhcZMIFMlq4N4vGVenoYqZGrsaQYu4gFUx7ox671fcfowFA43g85HI2kkXq9GgxB6lro9EbaPOYqzCH+WVbBSM2+vjw8HqRgnqhsujTdwN5Eor2jt3Nnof+Fc/FeUB6GKiBebNMqYHVkRkIOUhiWdr8zRx0BosjnYAOnXUqKfMDZjC8F1uDCXGJVB7Mp+FUuJb0QN47Tjz3GEl3JM1emwYoZMqEqTWMeCK4NJSAFEJYE5n82qGMknnObk3wmHFxnB7kgdGtWb4ak0GaXnHoPFivixDISiMgEmCJ1jcow03r2tMq2M56EdoPwgaOeUs7VbLTxgRLg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To avoid debug warnings while freeing reserved pages which were not allocated with usual allocators, mark their codetags as empty before freeing. Maybe we can annotate reserved pages correctly and avoid this? Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 2 ++ include/linux/mm.h | 8 ++++++++ include/linux/pgalloc_tag.h | 2 ++ 3 files changed, 12 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 1f3207097b03..102caf62c2a9 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -95,6 +95,7 @@ static inline void set_codetag_empty(union codetag_ref *ref) #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } +static inline void set_codetag_empty(union codetag_ref *ref) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ @@ -155,6 +156,7 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) {} static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) {} +static inline void set_codetag_empty(union codetag_ref *ref) {} #endif diff --git a/include/linux/mm.h b/include/linux/mm.h index bf5d0b1b16f4..310129414833 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -3077,6 +3078,13 @@ extern void reserve_bootmem_region(phys_addr_t start, /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) { + union codetag_ref *ref; + + ref = get_page_tag_ref(page); + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } ClearPageReserved(page); init_page_count(page); __free_page(page); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 0174aff5e871..ae9b0f359264 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -93,6 +93,8 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) #else /* CONFIG_MEM_ALLOC_PROFILING */ +static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } +static inline void put_page_tag_ref(union codetag_ref *ref) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} From patchwork Tue Oct 24 13:46:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94098C00A8F for ; Tue, 24 Oct 2023 13:48:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 468A16B0299; Tue, 24 Oct 2023 09:48:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EF8E6B029B; Tue, 24 Oct 2023 09:48:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 221286B029C; Tue, 24 Oct 2023 09:48:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 07FFF6B0299 for ; Tue, 24 Oct 2023 09:48:09 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D270EC061C for ; Tue, 24 Oct 2023 13:48:08 +0000 (UTC) X-FDA: 81380483856.07.6CFF9D4 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf15.hostedemail.com (Postfix) with ESMTP id 23D86A000B for ; Tue, 24 Oct 2023 13:48:06 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VgPNi01n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3Fss3ZQYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Fss3ZQYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155287; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nwLU1aU2LNAtecsch6dr14rd1yTTpsfUN6nYD0uWJEE=; b=1T64juikDiKKkvAeFiVBnh9TdNXslvy6HVZJNxDb963CHpp6BdvAQ/UmhaNkQdoBYfYRbq Q6GplDvtYbKnH4RYzPLPB1JoAGW5g6ZNFGp/8+qFZy4hxY3o/l8wQ6Aoozg0irdRUrZwBc jfaUvwrmACCxNpAO4qzf2jlnbpF953o= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VgPNi01n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3Fss3ZQYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Fss3ZQYKCLstvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155287; a=rsa-sha256; cv=none; b=No+ww6wTKsdLzqc1D2ovBWq5KDBzWOjqCB80Qe6dVY6eilvQlY3DKLQTh/xUusblF6RqIa uet9Pyly1ydqmRBXOtyKh7xZF0cIjXSB/P2ycHD74oLpKdNV7Rsp90lrHTBsXHKARyu8al XRmRvB1ZzB3+Q9aXXzfPnTRRCOPbcO4= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9a39444700so5323376276.0 for ; Tue, 24 Oct 2023 06:48:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155286; x=1698760086; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nwLU1aU2LNAtecsch6dr14rd1yTTpsfUN6nYD0uWJEE=; b=VgPNi01nOwtEDrXmf+bx8tzFIC3eyRIbovmy3wmGQ8rzimDJGBPOs5H65tb8SlAX2+ o4hfi9rrFzPx6F4rrjUzaM7FSRRCix/IsIw8X3j5ag/xnBgAszarQ7zpoh0tcJHb8+nw w3Yq6v/u2EJcl3KXsT6eZyYBewDmZ1hqffPY9XmAO/s2db9ByURuyL8lpDXWmRellqiD WwTCkQFuVZjNXvOgbZF9QvqvMP2qkDss8NQLfNLNEo5hM3qkNyGF6xNvJtTyohnjsg5m OyrH1nykngH1bVb97F/V35OgoeNm951PWg9lguqv/OVX3oQh8OdkSxIFW2UBQI8AeTtm FnTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155286; x=1698760086; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nwLU1aU2LNAtecsch6dr14rd1yTTpsfUN6nYD0uWJEE=; b=ZKZeuIhtbv3wR5qlyAZRxzAnKTrgtf6ilOq3QzcDRQ9tbHszxS2NK/uZYMOraPSxkO MhTu6Ho0T+WRHGKBTCEZzWhZ2XHg7IAhmPbqlOEDKGy+VAiuLUFvQahTHVa6fLdvj2LQ 3HuKL9D8HeFdNS9JwV2z6+JQg0x2jc8HO0yiiIbg0VXf7dL/oHqHhNV1Av5ac/DahDsV l7+vWhJe7kQ+F4iwT+Bg1Jg1S7dU3P+Y6NYMhwirAvI8HwvKpRcPC0YcbsSVt49T8jZ+ y4xBh5lHp5wQeEI1AgjvrY5Ki/7T4rvm8Fq1Qv4ZagNKZcrcIhjDkjL85ldeGnDYPg9b JJCA== X-Gm-Message-State: AOJu0YzB0z1OaMZ9cg4WyDwhzZ/vdAm602yDpiwKHCjAbU5iwX7NHMy6 yqZ6Fhg5vM00yY18Opu4ylQJfEipoL8= X-Google-Smtp-Source: AGHT+IEQFxSAdc50qujfzmzJKJcWY3qyZfXbT1k7XuYaFuTlpAInpKqkovdUmqHR4hu8HiZ2L8Ah2MqCV0A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:d244:0:b0:d9a:4cc1:b59a with SMTP id j65-20020a25d244000000b00d9a4cc1b59amr317326ybg.1.1698155286109; Tue, 24 Oct 2023 06:48:06 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:35 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-39-surenb@google.com> Subject: [PATCH v2 38/39] codetag: debug: introduce OBJEXTS_ALLOC_FAIL to mark failed slab_ext allocations From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 23D86A000B X-Stat-Signature: ekjokbu5ufdbzd414hqi4di8h5cfi7dw X-Rspam-User: X-HE-Tag: 1698155286-603563 X-HE-Meta: U2FsdGVkX19PxMuuQOGj5Slnp2sws1SOXuutVQki399fhBrJvZUBSCb0pCUua0iOtk4gFGifNEhOT0ZSXoOC/wShlRCEUXYALY7ZJU5Be2oZx1X9QOyeozDrS3PFzvd8bMkq10fP28LhIWdImxUn227MOyqPzjgrLlRvX5WLasXrcgZ0xBoJFlaKboEigN4/N4DvstAFURs11kjLxcJCnbC3JRFEaI9Ms/UpjDLaKb51urHnyGM0KwhU7f4HHYbUKVUlf58Bm83DnrHq0+PAWhfxwBz/zB379JtjhWBjyVvJMjJupQpp9db+mw6erjjrG5yeDEaBUxxRzSd/PIcCDsEShl1hn4QvBtb0fE0jzxTI+vNwiP2XEAdstXQuypXDfpb/qNv+dyAQAAO36bRZUW5iqDq1zUC/NSiHX01iR1lPbXlH9ZV45bjVRJH/Pk6SEzlCTTr+goGza9UCmRREvC91KkuURQWMnVZNjCR0vsTxGK9FnLADNYz7FVVedCMeQAFHbweG6Dsc/voHL25InduSk+IutbbZaq/c8SzFoDdGRq931IT2UrGu/k3MEdBA2LV3/dS2cvbXamr0KDPjMKO6Q2pslvE+piYfcyZxwF+5dmDa1Ee57OtRJwRdv2ZlF1GHGOqwYnUqiqAwx9nk1YAC3mcjCr5EDeNAfRLO9S+vk2iAEAzs+UtWhaiWAGp4q4lCPXq4rI3iuFRZ/oaVOwd47gEQ/QghhY1ASZBjgTPo0SHzf1c0W8k3A9gzYWQn9ihrl9cubzJTDKQuXomrU+rkoh66m/F3bplS7mzD8lcaEuo209jsBxisLo4R5i30tqTuxEPuR2W2RRCydQbr6S3ExS7rh4Cqr+tmworQkJKgPBJUc/M6wK7ecsWxAJWOl5hJpLhir7J+B0ZigKMqEg3R8Uzlzy1+mtfQRCTU5rk21n2M5URpCNfEu1LGgoO5fZJqk12WPqLWTCBs8wP yyqRENTy F+EqGVKRGdZbY/w0c0aiBt2fgjIJ+vqfUEBHpZmxd0ii1KWvr0+SQeVZ5HetctpRKTIAJwfcpT0dNUsOCxmpbuBQzMLkZKcAv8gTTJR0Msjp5FiTjWCgJPZaOcNX+5RL0D32NkBua1A5khTEwToGh9vtpiLiZULHBkH9GXIF/DG64I1ucWHPecXIoK9ubmd6T3BE9VJvKHLZgrQdH28cDhdYM58c8DY8diZ3gMZYZfVDiDDhap37nV7NJ4ZxNuPKJisFNFYV3wE9WEhWMYL+nbD96JmIE/vzcnXJfs+d4iviwwyXVGfgp421pDU/c6pgTTGRHBD1LvDC2Y+GnIxtIyzgLVFPmMVJTMAqH4jaLmdvEEV7rP08yWmJdHWElnIuEe8ui7+G3KW+sfXLcyvoMmM3x6a5Imo4UugHGOEzy+ZIKo6x8GFPVtkRyAUOUrRlAZCT2FGgj7ab5Ct4/EGTVKBNaeTU/xG90oSk3sOerYCUpxwmTeYgNZsfL4dZziKCL0M5FQpo7SyP5YHyLihbmPW8FQc/Uq9vO2HNO4RqgHWmHd/9B9rqt65vXzS3l7WJaCNnmNSBe4wOk3vxRqcKCCzCFfJiDJ8fPxDJseq9VmD0Ly5mBVrawGkl2O+oW4uScZhhEy7AY6YhoRMXctVhcBOIHdU5ToRuge7gs2SxxLQrI5V8lYHMPrU0o/JK+magSqN7fO1T6Lvaatt7wJ6VpRhwgsw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If slabobj_ext vector allocation for a slab object fails and later on it succeeds for another object in the same slab, the slabobj_ext for the original object will be NULL and will be flagged in case when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Mark failed slabobj_ext vector allocations using a new objext_flags flag stored in the lower bits of slab->obj_exts. When new allocation succeeds it marks all tag references in the same slabobj_ext vector as empty to avoid warnings implemented by CONFIG_MEM_ALLOC_PROFILING_DEBUG checks. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 4 +++- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slab_common.c | 22 +++++++++++++++------- 3 files changed, 43 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 853a24b5f713..6b680ca424e3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -363,8 +363,10 @@ enum page_memcg_data_flags { #endif /* CONFIG_MEMCG */ enum objext_flags { + /* slabobj_ext vector failed to allocate */ + OBJEXTS_ALLOC_FAIL = __FIRST_OBJEXT_FLAG, /* the next bit after the last actual flag */ - __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, + __NR_OBJEXTS_FLAGS = (__FIRST_OBJEXT_FLAG << 1), }; #define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) diff --git a/mm/slab.h b/mm/slab.h index 45216bad34b8..1736268892e6 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -474,9 +474,34 @@ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) } } +static inline void mark_failed_objexts_alloc(struct slab *slab) +{ + slab->obj_exts = OBJEXTS_ALLOC_FAIL; +} + +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) +{ + /* + * If vector previously failed to allocate then we have live + * objects with no tag reference. Mark all references in this + * vector as empty to avoid warnings later on. + */ + if (obj_exts & OBJEXTS_ALLOC_FAIL) { + unsigned int i; + + for (i = 0; i < objects; i++) + set_codetag_empty(&vec[i].ref); + } +} + + #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} +static inline void mark_failed_objexts_alloc(struct slab *slab) {} +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ diff --git a/mm/slab_common.c b/mm/slab_common.c index db2cd7afc353..cea73314f919 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -218,29 +218,37 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab) { unsigned int objects = objs_per_slab(s, slab); - unsigned long obj_exts; - void *vec; + unsigned long new_exts; + unsigned long old_exts; + struct slabobj_ext *vec; gfp &= ~OBJCGS_CLEAR_MASK; /* Prevent recursive extension vector allocation */ gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); - if (!vec) + if (!vec) { + /* Mark vectors which failed to allocate */ + if (new_slab) + mark_failed_objexts_alloc(slab); + return -ENOMEM; + } - obj_exts = (unsigned long)vec; + new_exts = (unsigned long)vec; #ifdef CONFIG_MEMCG - obj_exts |= MEMCG_DATA_OBJEXTS; + new_exts |= MEMCG_DATA_OBJEXTS; #endif + old_exts = slab->obj_exts; + handle_failed_objexts_alloc(old_exts, vec, objects); if (new_slab) { /* * If the slab is brand new and nobody can yet access its * obj_exts, no synchronization is required and obj_exts can * be simply assigned. */ - slab->obj_exts = obj_exts; - } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + slab->obj_exts = new_exts; + } else if (cmpxchg(&slab->obj_exts, old_exts, new_exts) != old_exts) { /* * If the slab is already in use, somebody can allocate and * assign slabobj_exts in parallel. In this case the existing From patchwork Tue Oct 24 13:46:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13434608 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4B27C00A8F for ; Tue, 24 Oct 2023 13:48:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 72C016B029B; Tue, 24 Oct 2023 09:48:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 68FF36B029C; Tue, 24 Oct 2023 09:48:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DF086B029D; Tue, 24 Oct 2023 09:48:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 34CD76B029B for ; Tue, 24 Oct 2023 09:48:12 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 092C8807BC for ; Tue, 24 Oct 2023 13:48:12 +0000 (UTC) X-FDA: 81380484024.06.E7684DC Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf09.hostedemail.com (Postfix) with ESMTP id 49DFA140014 for ; Tue, 24 Oct 2023 13:48:09 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="dpQWqKS/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3GMs3ZQYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3GMs3ZQYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698155289; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5iA9XL/oyX6Tyl0rnn5vdBOVip3jqaNokzkLete42fs=; b=vCDNS52NhODcREBu8O3UPzrRQ0E63dEWHAqugxq90tNRD4mE4h+x+vvtqpcWzyU1ZP1g3I ger2cpmtrJnY6fc8ipYKYNjYGLs/agRfekmKw1HBb8Np0o4N3pFgwwHYgh9GFHPfpjNMha 1g+nehyHqv0R1KjVUBGcf0ULQAa74d0= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="dpQWqKS/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3GMs3ZQYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3GMs3ZQYKCL0vxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698155289; a=rsa-sha256; cv=none; b=fIgGHwfsEAeTkyhu21LHzrpCKS1RNQoWy81C9qV1YvSROT9IghxVB4De/AnO9m4xMcXU75 A1DSNQRPvbP/h8+X4Q0+K2ETariiHkhUFay3tPx3LeWmy5IssL8cBS1QNc6L0abY2k8s+S bXp8k8ikzwwKR8UjZKjpHpR0rhtgJSo= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-da04776a869so499113276.0 for ; Tue, 24 Oct 2023 06:48:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698155288; x=1698760088; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5iA9XL/oyX6Tyl0rnn5vdBOVip3jqaNokzkLete42fs=; b=dpQWqKS/rqmQJgxkEbU3eYhJgMU7aCpE8X4Tr7pdSaW6Pv1z7DdSH0Ya4X5u206yPC JrcqtTeZtKeQ1bD+lG7+lE4+XljDM7qOe5r6of/jXsv276AQX6LKKSp4uV778YiIPjFZ oB1USvk4NIafKvUENTfgH9xW1WSDH2XAJAR9chnsyDoC1TIvOnLetTs9IJX4mwE8Y5AE 5bS2aO51egOP6KnSEmHVJIT9v7KRhJyXensrHymjtkD8kvUM4bFuMFb97mUGVdAKyVcA BSUlxAENTdTKm/tllDTZbH/4XdD6rvm/2Wkm9AFaqgn6qKgIFSbTOGb2txWdf7EzRvPK Jy0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698155288; x=1698760088; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5iA9XL/oyX6Tyl0rnn5vdBOVip3jqaNokzkLete42fs=; b=UmNoLK1uDBIRzAJnOQxj4JYUoUAVYswS6JkSqIxamiAamSCLhsTvBeST3x4Hfqj8nj 0zzI9gwUNfBuJ7YydG7YATd070ejsSUxtja6Vn/gUGCwzXbA5MRBcFr98haEEb8R10px x7MbWG5ceTMlrlsLp38S0qjpy1Fhwfpb1nE8bPposN27bX+p6OC+Sa5cWtwbIpnNGZDV WngqaNUQTqFRfMnAO0EwZRGPJ7yAxci6Lac7NBcyuZsoct0JXrIe4FlH9+Oc5849YGmU SuiCZzVN/ClXv71bTBgnMP1JsyoAagaj3Zz2RA7XnxIDiigoC7UcQ0MrsssLYY8baVKR AmHw== X-Gm-Message-State: AOJu0YzyJakeiglaLVMnEsjAVIu2i6ubDdyxTKDe+0D7PzXpBwxEdGs6 pei0E8viBPxyfCr5Yc2cRdhIjvZuTKE= X-Google-Smtp-Source: AGHT+IG2dGz7nkvTfdEUK5vnzGjBRQVPkaal0DQpTev0jCoT6FkWge7pFcsduGaJ7nCMKZx++nF8KYSK16A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:45ba:3318:d7a5:336a]) (user=surenb job=sendgmr) by 2002:a25:ade8:0:b0:d89:dd12:163d with SMTP id d40-20020a25ade8000000b00d89dd12163dmr222107ybe.1.1698155288323; Tue, 24 Oct 2023 06:48:08 -0700 (PDT) Date: Tue, 24 Oct 2023 06:46:36 -0700 In-Reply-To: <20231024134637.3120277-1-surenb@google.com> Mime-Version: 1.0 References: <20231024134637.3120277-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.758.gaed0368e0e-goog Message-ID: <20231024134637.3120277-40-surenb@google.com> Subject: [PATCH v2 39/39] MAINTAINERS: Add entries for code tagging and memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 49DFA140014 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ier3d719g4s1azumwckie4o1i7t3865g X-HE-Tag: 1698155289-185981 X-HE-Meta: U2FsdGVkX19z3mo+XfLZjZlZ11l2f9QltKR2dYIaGeScdv1W0jHhHzF6M5SIFh5naoBwNHkfWY3xIyCIpnBaa8DvMkgOVeqO4m3HWe2hS8XnOKWSFLEQfYrWQSmoZ0FSQmUT2E/HHse7B0dOb0RowjVCm9DV7sCJdisyIAwI6SZRcRdXkPlzyBBP8Xs4/BdaVurO4w5McHnXtWfgvnu7qzJhwT4y88EpdOIL+8LVZikTTGJBuaAHhfeoOtPGOOhHZwKfmSPIBup/1lxbCfDhXmgvnVFa/+HiiikuGJNe1dA4/jkX0P3a2Vel+/AaPcyZIyDqjdCC+kelGpwhXLEPS6WwHOJoTnzSBuqpv0aerFd+wpKOLqhtx1nnm1NrG27bKmOcAYWOJn+Y/Xi8i2CT4OR+vrxfaEGMakCcKGvPyshz3QopVOPkungNjBBF5fmQfnnBHFo8i+LBpW7l5nMpYf+vsuKT/8vOfe3uAFvXm9v5cdSSIYbVyB2ZQBlFB0FMnHnwvWb3KgXgcjqXkocBHVRwwfRHlkchVCMJXI4ot26uSgo0hHZI2NUuDXK28nVe89ZbZ+TEJKTKqLQ3BPhIzZDGLzhiJgBvQh3ADTy4HmchO3W1Kl247+UGkhk0WDM9eRSaFxCJ6Ju08kCq8FBL2TEz39uSQSH5C0/ClIfFNsvlBmwVLLxyr5xsvqFh7x1j1JOQrkcjt4hCPh9aI7GvG+R1PwXHiQdvGFZKYXKTwk2iqNYhdf4AhfHJwQ8adioegjAv3IreB+PufdYmSk9MsAqJmBLeJc8IplUDAsbyNaIXjlGGVE2i+lVefakihBosGHLSaadscgYmpb7zYWFr2nKKeyxmnkK8J6nNijKirgWXHf3ysZh5XnnNZ5DlepM9+Znq/A/nmAL5fJwOS6mcu/be5xyhkS+hZYTLRIfyzE11DB0EJ4honk8XbFOkDVibz3AHzq2FxfyoX4dnbDH xIFiTC1e 46i0Mj5hi2iKdrdSduIQfNczDyIzuK9+YOvqsS0qf5BlJQ9xrY35HBinC9/DRAhWiOusrrZrTCX3NWUlpshPuUXlsfRUBsZ+59ulxqy1FuXSiHhgDg9wTiMjmRCLymRFPC4j3QqRlu+gdlseMKYZEx77S2vfUf28/VpEWBvJsT4eiOa6QFZT9HG0mlPQLlC+wa7oqSIg1Y3MEYkcVjHb9Qf7hjXVKI0kAfyhjOPqSJr84/fCM7f4qYroKGL4rWTVy/R5/awhajRYPqk3o2axYDzPMXvwuZ/BiS+ZtPxNy0PXnJ/QKLnHInGmhQgSFHq8ycPOj/W9W6U4L6dW0m+W1CTcYfJbC3OHuw/wZKbdYuXYfsS5xFoMqxkAu6yxSoMe2UKmFuKbzctb1pGxd2JZfOiMMKXp3qJa3KCYWNLAk5KAx9hBZ/1J3yuwKWKcTPxMHaSowQ2GGuMRR0hNPpU6gTfOFurhVwyFGhrH7VFBe+7ErtImJ44ftoC2Ia0xMaZCpz/IaClalrLeaFOe4v0UhrecRMJB7zYXm4rrfAMgmSRe8rLT1BZWv+xGWAOGXGXk+JbL/4CteyXq3bJJ+t0dHGiSTgRY9N27MlweVAAKW2MyWib5CfEVPCUIGpbOqNMT21huYvIet1Wv+4K2o6xscQ/dIsJecRmaDiIi1z1pRZguzqjo5qUzmRnzFlQEE5unMI1+/EyxlaTGpeV5CT9mIjVH4W3SwpIKbG5HzgQR5htrbqZB+OTBjYsdH4xVS6xWFj8GSy3RY2jkemCOsBE1+Fus2YyjJ4dnY7lEocE/CTsOvMXe2nHf5RbyLQU0hsqEdfGCszGmnOU3GFsqlh3ZJuGae/zygyJsxnkvNjb1vMR9bac9KWpPl8w5qytQ8QfYYd/inwwO+hegM6dX76rS6osJIK7qumAPuuNQy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The new code & libraries added are being maintained - mark them as such. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- MAINTAINERS | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 2894f0777537..22e51de42131 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5118,6 +5118,13 @@ S: Supported F: Documentation/process/code-of-conduct-interpretation.rst F: Documentation/process/code-of-conduct.rst +CODE TAGGING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/codetag.h +F: lib/codetag.c + COMEDI DRIVERS M: Ian Abbott M: H Hartley Sweeten @@ -13708,6 +13715,15 @@ F: mm/memblock.c F: mm/mm_init.c F: tools/testing/memblock/ +MEMORY ALLOCATION PROFILING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/alloc_tag.h +F: include/linux/codetag_ctx.h +F: lib/alloc_tag.c +F: lib/pgalloc_tag.c + MEMORY CONTROLLER DRIVERS M: Krzysztof Kozlowski L: linux-kernel@vger.kernel.org