From patchwork Fri Feb 16 00:08:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13559309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DC1BCC48BC4 for ; Fri, 16 Feb 2024 00:09:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PbMXmAPBEOHjtU44w+l2diX8HzTSkoWkox3NPDsIk80=; b=MkBcqnBcpM6Pd9 5OK4kE/PdtUBSfwS0LOwcTpTJ8RpEMyI92mxg1fSXMt+PAkR2S+vzlxDwlbHpx+5z9ABbBLIOcNCi mKU020ym7VRbTRRQJLYNOqG+xqlWb+yHcbSVI4fvzZjRXsq5f2Bun243VUdCy9T8ub2vYB/WkEqXe Y8ltq23KtQamK9MYuwsO7oRuWLPkLH7v9wqGePLdxLCW7PXj6HEjKoHYSTlh1owr06qMd8yMrYVBe H11U1+3G1K2Ix5JqvkBYf57iILsLpu16ppVh5H1+2OtUQyJ6lCbVM0tU9PPDfu2qt7hUd+K+9Jl+h dnj0NDLmZLnXTlXd0vYg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmd-00000000Yby-09T3; Fri, 16 Feb 2024 00:08:51 +0000 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmU-00000000YYN-2oyO for linux-arm-kernel@lists.infradead.org; Fri, 16 Feb 2024 00:08:43 +0000 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1d70b0e521eso12111205ad.1 for ; Thu, 15 Feb 2024 16:08:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1708042122; x=1708646922; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oULp81TNDm6rtXJp1+1NlVw8kUN7N6Lx6G6xlg9BNVs=; b=juz9ECDbLTUJLK/MIS9qE0PrCoT40x3AhrTXbwF7oFt7Wjm//4ffwzDVz0zVOC95FU Gqyidum2Dd3aVexh+DOfELrgS0pmxpse3NTS2EVzGx6uWS7YjBmpnz5T6+zQnD5cI0qf TUYs+MjVUulZWfwL8Ux48hnOns9tbD8sWVkWRQWdxKMhJOfq8w6k8r/reuRwU/Rkb64y KyTIu6POGB+s58Uajl4kvjBNhh5jti/OEtJQ2QunjOOv9xG1svMJJT4Soq5Vy2mwfNR0 8/+iisaH1r/BQrL+1rF/oNIFFfZ5KxnTUYsdDFDH5OffhEDd8FFUJ+/y8x+90/bPPOLh mMKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708042122; x=1708646922; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oULp81TNDm6rtXJp1+1NlVw8kUN7N6Lx6G6xlg9BNVs=; b=m93MZt3UJ1iCx7uIrvbCjQy1fp1wu7Dhhsp0pv0nrd6DvJ2Q2AZDy7zNh21rMT4r30 g2Q+6g9uu0oTiUlYXMwAfnZkenA8ba/6kbqQGXDOtDJj8cerAbLWLu5A/vRG6xoUhhsr u/72VIOAKMts4wWadJpdME1UjgL6A9UkKZ329DZZJUIzLuhk+8huzcURu5J/J3gZ7xXW piIST8sQ1LOs6EYG27DApp2zqDUnoWzdaaPO575zLq0nM5zjijktAOJ52pC/ty0Jaze/ vOVQFOUCKIIMVuwSXxMj3YNFl1y5gbpgB870/DAX7NLH0Nc9Iw449jfUnnOcZFO+EUK+ 8BYg== X-Forwarded-Encrypted: i=1; AJvYcCVa0xIabeFHQ74wqSzBkQEimqXq/RGilqt6zhfXIk6UKuMP3+vc++ywZDnwIu6pkWFjOD8mO1v/OIixHHfYhzoJIk3paKfKA3ihQApbT1mqtAt/ISs= X-Gm-Message-State: AOJu0Yxu3G0x81uxdAR0qdxKWoxJ881zmGqWDG5fBGS1OjB6Bnt8xH87 hlvjZL7/basPYe8VRu4ezGSkXCvpP9UqwM8gxU9yfP7rq+S5aHACiX9N8JYvQ6I= X-Google-Smtp-Source: AGHT+IG7WoMFGu0/RfCzt3XOL546mMsF4SOL7oXhsNzvqoggBob9r4cZ+ZAFJwFNFVhaS4ZZGy8jhQ== X-Received: by 2002:a17:903:1212:b0:1d9:14fb:d142 with SMTP id l18-20020a170903121200b001d914fbd142mr3570668plh.32.1708042122114; Thu, 15 Feb 2024 16:08:42 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bb6-20020a170902bc8600b001db3d365082sm1789486plb.265.2024.02.15.16.08.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 16:08:41 -0800 (PST) From: Samuel Holland To: Will Deacon , Mark Rutland , Eric Lin , Conor Dooley Cc: Palmer Dabbelt , devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Walmsley , linux-riscv@lists.infradead.org, Rob Herring , Krzysztof Kozlowski , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [PATCH v1 1/6] dt-bindings: cache: Document the sifive,perfmon-counters property Date: Thu, 15 Feb 2024 16:08:13 -0800 Message-ID: <20240216000837.1868917-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240216000837.1868917-1-samuel.holland@sifive.com> References: <20240216000837.1868917-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240215_160842_758654_2B0BE229 X-CRM114-Status: GOOD ( 10.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SiFive Composable Cache controller contains an optional PMU with a configurable number of event counters. Document a property which describes the number of available counters. Signed-off-by: Samuel Holland --- Documentation/devicetree/bindings/cache/sifive,ccache0.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Documentation/devicetree/bindings/cache/sifive,ccache0.yaml b/Documentation/devicetree/bindings/cache/sifive,ccache0.yaml index 7e8cebe21584..100eda4345de 100644 --- a/Documentation/devicetree/bindings/cache/sifive,ccache0.yaml +++ b/Documentation/devicetree/bindings/cache/sifive,ccache0.yaml @@ -81,6 +81,11 @@ properties: The reference to the reserved-memory for the L2 Loosely Integrated Memory region. The reserved memory node should be defined as per the bindings in reserved-memory.txt. + sifive,perfmon-counters: + $ref: /schemas/types.yaml#/definitions/uint32 + default: 0 + description: Number of PMU counter registers + allOf: - $ref: /schemas/cache-controller.yaml# From patchwork Fri Feb 16 00:08:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13559312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 049FAC4829E for ; Fri, 16 Feb 2024 00:09:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=J9F7PaAKBkswZI9VXH4qdCCPkaLrYOWqBR940+lsYAc=; b=icSmP4Siao19QU K2ZQfRTEVm6RtVTCr/eT5u4yIyvh5SS7msHi1Jw+fODl4fU0TkGFRAesEJRZKO20TMXxCxBlXNYYB sYgVSWa4H7mpl8sS3jpQeM/Ot16CxK1OtY1/OKq1RWTLRSZND9WdJcPg3295boZsIg1vGagz40V2w zzjloFQJgygMMVrV1T2CZ4KVJ1AZfbubIdbMwcHK1VCIv4YQn168+ZkxaYxON/dfnFbQL3ewGYSuL X790/56PDePmYAav8MM13K4k9SVzOmjfBwGHbYsM8g/flXxUwJy9czS4TY6NEawFrWFsEsPVR6cYy 5P5tgOzQ0SMZgG/uuUjg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmh-00000000YfJ-2ITt; Fri, 16 Feb 2024 00:08:55 +0000 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmX-00000000YZH-2EMw for linux-arm-kernel@lists.infradead.org; Fri, 16 Feb 2024 00:08:47 +0000 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1d953fa3286so1040695ad.2 for ; Thu, 15 Feb 2024 16:08:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1708042124; x=1708646924; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7qRMAv7CAh5HOzYBpQQNwIWdxPMEEkXjRDW/Hnn3LFw=; b=CxiM6myQMgY1Pk+DLTc1sYCQCxudBLv8pNuBQXxZfVL0e/TLTuuQym4IAZJ0kGmHFv Hn8t9OUZmEHhTzHfo082ecNtaLHhAgdgXy0HqCyT48/5PF/vjRnXLHstGEz7PuWLd8fQ r/XL/plR/6Mmd5Xrdqof8YaeA5srzSsTeHnz0TjN/2Gjl5/iu/hapDMdxcmniukOPr74 3k7PGPBn2rDM+DkaO8Gq2aODwojY3MeNKUy9Hc47WOliZQM87pZ8gd4GtpHyjZCCRzZD l+mkIAXqTWzTHT3zFVcXisms2v4JnmhZ/Kv6tWDWd7/nI4M/imrczOjrLynCOHM73bTc fb+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708042124; x=1708646924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7qRMAv7CAh5HOzYBpQQNwIWdxPMEEkXjRDW/Hnn3LFw=; b=byZwjzxJHrvs9x6iVr4FZ9VAAMb2cfHMpLi04BtJpG/2lOM7/If8WMuf/tvbHGQ7ZQ 2TPLvKa0DohfRvCn2qF3Q3dcLbOMYP9tGhhuPX6ytxNLk4R1PH2YwA6ZGVw60+gg7nOZ 9nFO/y+pN9Okl8r9CUczo2xItl/7FvaF4C6VPDZL1OtUI7tzLEi2s6J5VZ8OmHODxAkZ qUR4G7UAPywUrTsRavoSb2YYsEPkg+Rk138XgQAg9vmaL/5fQGpGjqpqaBHEVDb5/Bch KqLOBrGD8eJ8rLeUgEMGQ99SsVVomrX/6sKOKuQtdnKSrUh69m6yXJvSw8h1DmMT5YTY QrMg== X-Forwarded-Encrypted: i=1; AJvYcCWrw8D3IdMK9YDulvn/1kzrqNas4+0ZfLRThIB341vpcfuiLBURkGegvkL8l1oDe58Jesw1CwL1S3tZ8l9dm0BHR28KMQWHP7iVX+jiDrPcS5VQBRY= X-Gm-Message-State: AOJu0YwK7CtbvPX++bLXvc/UBgCw4QB8y/nxy8LmobTq76Sf6XqWdIBa MzB3O/OHVQyN5As7GK2XAzc4aZb4MIemhoGp/IVWAgbXrOlLX/I5Qqg/u20sqQA= X-Google-Smtp-Source: AGHT+IH8bcXlBYTeQrZ+7Z4/vwzRYszWCz1qlOeQMJcLWA4GkdraPy5ipvhTtWIPzUZBrC3x60oNzg== X-Received: by 2002:a17:903:904:b0:1db:a6b8:6b23 with SMTP id ll4-20020a170903090400b001dba6b86b23mr683294plb.27.1708042124375; Thu, 15 Feb 2024 16:08:44 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bb6-20020a170902bc8600b001db3d365082sm1789486plb.265.2024.02.15.16.08.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 16:08:43 -0800 (PST) From: Samuel Holland To: Will Deacon , Mark Rutland , Eric Lin , Conor Dooley Cc: Palmer Dabbelt , devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Walmsley , linux-riscv@lists.infradead.org, Rob Herring , Krzysztof Kozlowski , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [PATCH v1 2/6] drivers/perf: Add SiFive Composable Cache PMU driver Date: Thu, 15 Feb 2024 16:08:14 -0800 Message-ID: <20240216000837.1868917-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240216000837.1868917-1-samuel.holland@sifive.com> References: <20240216000837.1868917-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240215_160845_638078_82E6CB3D X-CRM114-Status: GOOD ( 25.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Lin Add a driver for the PMU found in the SiFive Composable Cache controller. This PMU provides a configurable number of counters and a variety of events. Events are grouped into sets. Each counter can count events from only one set at a time; however, it can count any number of events within that set simultaneously. The PMU hardware does not provide an overflow interrupt or a way to atomically control groups of counters. Some events can be filtered further by client ID (e.g. CPU or external DMA master). That functionality is not supported by this driver. This driver further assumes that a single Composable Cache instance is shared by all CPUs in the system. Example usage: $ perf stat -a -e sifive_ccache_pmu/inner_acquire_block_btot/, sifive_ccache_pmu/inner_acquire_block_hit/, sifive_ccache_pmu/inner_acquire_block_ntob/ ls Performance counter stats for 'system wide': 542 sifive_ccache_pmu/inner_acquire_block_btot/ 22081 sifive_ccache_pmu/inner_acquire_block_hit/ 22006 sifive_ccache_pmu/inner_acquire_block_ntob/ 0.064672432 seconds time elapsed Example using numeric event selectors: $ perf stat -a -e sifive_ccache_pmu/event=0x10001/, sifive_ccache_pmu/event=0x2002/, sifive_ccache_pmu/event=0x4001/ ls Performance counter stats for 'system wide': 478 sifive_ccache_pmu/event=0x10001/ 4717 sifive_ccache_pmu/event=0x2002/ 44966 sifive_ccache_pmu/event=0x4001/ 0.111027326 seconds time elapsed Signed-off-by: Eric Lin Co-developed-by: Samuel Holland Signed-off-by: Samuel Holland --- drivers/perf/Kconfig | 9 + drivers/perf/Makefile | 1 + drivers/perf/sifive_ccache_pmu.c | 577 +++++++++++++++++++++++++++++++ include/linux/cpuhotplug.h | 1 + 4 files changed, 588 insertions(+) create mode 100644 drivers/perf/sifive_ccache_pmu.c diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index ec6e0d9194a1..b4e4db7424b4 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -155,6 +155,15 @@ config QCOM_L3_PMU Adds the L3 cache PMU into the perf events subsystem for monitoring L3 cache events. +config SIFIVE_CCACHE_PMU + tristate "SiFive Composable Cache PMU" + depends on RISCV || COMPILE_TEST + help + Support for the Composable Cache performance monitoring unit (PMU) on + SiFive platforms. The Composable Cache PMU provides up to 64 counters + for measuring whole-system L2/L3 cache performance using the perf + events subsystem. + config THUNDERX2_PMU tristate "Cavium ThunderX2 SoC PMU UNCORE" depends on ARCH_THUNDER2 || COMPILE_TEST diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index a06338e3401c..51ef5f50ace4 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -15,6 +15,7 @@ obj-$(CONFIG_QCOM_L3_PMU) += qcom_l3_pmu.o obj-$(CONFIG_RISCV_PMU) += riscv_pmu.o obj-$(CONFIG_RISCV_PMU_LEGACY) += riscv_pmu_legacy.o obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o +obj-$(CONFIG_SIFIVE_CCACHE_PMU) += sifive_ccache_pmu.o obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o obj-$(CONFIG_ARM_SPE_PMU) += arm_spe_pmu.o diff --git a/drivers/perf/sifive_ccache_pmu.c b/drivers/perf/sifive_ccache_pmu.c new file mode 100644 index 000000000000..8c9ef0d09f48 --- /dev/null +++ b/drivers/perf/sifive_ccache_pmu.c @@ -0,0 +1,577 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * SiFive Composable Cache PMU driver + * + * Copyright (C) 2022-2024 SiFive, Inc. + * Copyright (C) Eric Lin + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#define CCACHE_SELECT_OFFSET 0x2000 +#define CCACHE_CLIENT_FILTER_OFFSET 0x2800 +#define CCACHE_COUNTER_OFFSET 0x3000 + +#define CCACHE_PMU_MAX_COUNTERS 64 + +struct sifive_ccache_pmu { + struct pmu pmu; + struct hlist_node node; + struct notifier_block cpu_pm_nb; + void __iomem *base; + DECLARE_BITMAP(used_mask, CCACHE_PMU_MAX_COUNTERS); + unsigned int cpu; + int n_counters; + struct perf_event *events[] __counted_by(n_counters); +}; + +#define to_ccache_pmu(p) (container_of(p, struct sifive_ccache_pmu, pmu)) + +#ifndef readq +static inline u64 readq(void __iomem *addr) +{ + return readl(addr) | (((u64)readl(addr + 4)) << 32); +} +#endif + +#ifndef writeq +static inline void writeq(u64 v, void __iomem *addr) +{ + writel(lower_32_bits(v), addr); + writel(upper_32_bits(v), addr + 4); +} +#endif + +/* + * sysfs attributes + * + * We export: + * - cpumask, used by perf user space and other tools to know on which CPUs to create events + * - events, used by perf user space and other tools to create events symbolically, e.g.: + * perf stat -a -e sifive_ccache_pmu/event=inner_put_partial_data_hit/ ls + * perf stat -a -e sifive_ccache_pmu/event=0x101/ ls + * - formats, used by perf user space and other tools to configure events + */ + +/* cpumask */ +static ssize_t cpumask_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + struct sifive_ccache_pmu *ccache_pmu = dev_get_drvdata(dev); + + if (ccache_pmu->cpu >= nr_cpu_ids) + return 0; + + return sysfs_emit(buf, "%d\n", ccache_pmu->cpu); +}; + +static DEVICE_ATTR_RO(cpumask); + +static struct attribute *sifive_ccache_pmu_cpumask_attrs[] = { + &dev_attr_cpumask.attr, + NULL, +}; + +static const struct attribute_group sifive_ccache_pmu_cpumask_group = { + .attrs = sifive_ccache_pmu_cpumask_attrs, +}; + +/* events */ +static ssize_t sifive_ccache_pmu_event_show(struct device *dev, struct device_attribute *attr, + char *page) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr); + return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id); +} + +#define SET_EVENT_SELECT(_event, _set) (BIT_ULL((_event) + 8) | (_set)) +#define CCACHE_PMU_EVENT_ATTR(_name, _event, _set) \ + PMU_EVENT_ATTR_ID(_name, sifive_ccache_pmu_event_show, SET_EVENT_SELECT(_event, _set)) + +enum ccache_pmu_event_set1 { + INNER_PUT_FULL_DATA = 0, + INNER_PUT_PARTIAL_DATA, + INNER_ATOMIC_DATA, + INNER_GET, + INNER_PREFETCH_READ, + INNER_PREFETCH_WRITE, + INNER_ACQUIRE_BLOCK_NTOB, + INNER_ACQUIRE_BLOCK_NTOT, + INNER_ACQUIRE_BLOCK_BTOT, + INNER_ACQUIRE_PERM_NTOT, + INNER_ACQUIRE_PERM_BTOT, + INNER_RELEASE_TTOB, + INNER_RELEASE_TTON, + INNER_RELEASE_BTON, + INNER_RELEASE_DATA_TTOB, + INNER_RELEASE_DATA_TTON, + INNER_RELEASE_DATA_BTON, + OUTER_PROBE_BLOCK_TOT, + OUTER_PROBE_BLOCK_TOB, + OUTER_PROBE_BLOCK_TON, + CCACHE_PMU_MAX_EVENT1_IDX +}; + +enum ccache_pmu_event_set2 { + INNER_PUT_FULL_DATA_HIT = 0, + INNER_PUT_PARTIAL_DATA_HIT, + INNER_ATOMIC_DATA_HIT, + INNER_GET_HIT, + INNER_PREFETCH_HIT, + INNER_ACQUIRE_BLOCK_HIT, + INNER_ACQUIRE_PERM_HIT, + INNER_RELEASE_HIT, + INNER_RELEASE_DATA_HIT, + OUTER_PROBE_HIT, + INNER_PUT_FULL_DATA_HIT_SHARED, + INNER_PUT_PARTIAL_DATA_HIT_SHARED, + INNER_ATOMIC_DATA_HIT_SHARED, + INNER_GET_HIT_SHARED, + INNER_PREFETCH_HIT_SHARED, + INNER_ACQUIRE_BLOCK_HIT_SHARED, + INNER_ACQUIRE_PERM_HIT_SHARED, + OUTER_PROBE_HIT_SHARED, + OUTER_PROBE_HIT_DIRTY, + CCACHE_PMU_MAX_EVENT2_IDX +}; + +enum ccache_pmu_event_set3 { + OUTER_ACQUIRE_BLOCK_NTOB_MISS = 0, + OUTER_ACQUIRE_BLOCK_NTOT_MISS, + OUTER_ACQUIRE_BLOCK_BTOT_MISS, + OUTER_ACQUIRE_PERM_NTOT_MISS, + OUTER_ACQUIRE_PERM_BTOT_MISS, + OUTER_RELEASE_TTOB_EVICTION, + OUTER_RELEASE_TTON_EVICTION, + OUTER_RELEASE_BTON_EVICTION, + OUTER_RELEASE_DATA_TTOB_NOT_APPLICABLE, + OUTER_RELEASE_DATA_TTON_DIRTY_EVICTION, + OUTER_RELEASE_DATA_BTON_NOT_APPLICABLE, + INNER_PROBE_BLOCK_TOT_CODE_MISS_HITS_OTHER_HARTS, + INNER_PROBE_BLOCK_TOB_LOAD_MISS_HITS_OTHER_HARTS, + INNER_PROBE_BLOCK_TON_STORE_MISS_HITS_OTHER_HARTS, + CCACHE_PMU_MAX_EVENT3_IDX +}; + +enum ccache_pmu_event_set4 { + INNER_HINT_HITS_INFLIGHT_MISS = 0, + CCACHE_PMU_MAX_EVENT4_IDX +}; + +static struct attribute *sifive_ccache_pmu_events[] = { + /* pmEventSelect1 */ + CCACHE_PMU_EVENT_ATTR(inner_put_full_data, INNER_PUT_FULL_DATA, 1), + CCACHE_PMU_EVENT_ATTR(inner_put_partial_data, INNER_PUT_PARTIAL_DATA, 1), + CCACHE_PMU_EVENT_ATTR(inner_atomic_data, INNER_ATOMIC_DATA, 1), + CCACHE_PMU_EVENT_ATTR(inner_get, INNER_GET, 1), + CCACHE_PMU_EVENT_ATTR(inner_prefetch_read, INNER_PREFETCH_READ, 1), + CCACHE_PMU_EVENT_ATTR(inner_prefetch_write, INNER_PREFETCH_WRITE, 1), + CCACHE_PMU_EVENT_ATTR(inner_acquire_block_ntob, INNER_ACQUIRE_BLOCK_NTOB, 1), + CCACHE_PMU_EVENT_ATTR(inner_acquire_block_ntot, INNER_ACQUIRE_BLOCK_NTOT, 1), + CCACHE_PMU_EVENT_ATTR(inner_acquire_block_btot, INNER_ACQUIRE_BLOCK_BTOT, 1), + CCACHE_PMU_EVENT_ATTR(inner_acquire_perm_ntot, INNER_ACQUIRE_PERM_NTOT, 1), + CCACHE_PMU_EVENT_ATTR(inner_acquire_perm_btot, INNER_ACQUIRE_PERM_BTOT, 1), + CCACHE_PMU_EVENT_ATTR(inner_release_ttob, INNER_RELEASE_TTOB, 1), + CCACHE_PMU_EVENT_ATTR(inner_release_tton, INNER_RELEASE_TTON, 1), + CCACHE_PMU_EVENT_ATTR(inner_release_bton, INNER_RELEASE_BTON, 1), + CCACHE_PMU_EVENT_ATTR(inner_release_data_ttob, INNER_RELEASE_DATA_TTOB, 1), + CCACHE_PMU_EVENT_ATTR(inner_release_data_tton, INNER_RELEASE_DATA_TTON, 1), + CCACHE_PMU_EVENT_ATTR(inner_release_data_bton, INNER_RELEASE_DATA_BTON, 1), + CCACHE_PMU_EVENT_ATTR(outer_probe_block_tot, OUTER_PROBE_BLOCK_TOT, 1), + CCACHE_PMU_EVENT_ATTR(outer_probe_block_tob, OUTER_PROBE_BLOCK_TOB, 1), + CCACHE_PMU_EVENT_ATTR(outer_probe_block_ton, OUTER_PROBE_BLOCK_TON, 1), + + /* pmEventSelect2 */ + CCACHE_PMU_EVENT_ATTR(inner_put_full_data_hit, INNER_PUT_FULL_DATA_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_put_partial_data_hit, INNER_PUT_PARTIAL_DATA_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_atomic_data_hit, INNER_ATOMIC_DATA_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_get_hit, INNER_GET_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_prefetch_hit, INNER_PREFETCH_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_acquire_block_hit, INNER_ACQUIRE_BLOCK_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_acquire_perm_hit, INNER_ACQUIRE_PERM_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_release_hit, INNER_RELEASE_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_release_data_hit, INNER_RELEASE_DATA_HIT, 2), + CCACHE_PMU_EVENT_ATTR(outer_probe_hit, OUTER_PROBE_HIT, 2), + CCACHE_PMU_EVENT_ATTR(inner_put_full_data_hit_shared, INNER_PUT_FULL_DATA_HIT_SHARED, 2), + CCACHE_PMU_EVENT_ATTR(inner_put_partial_data_hit_shared, + INNER_PUT_PARTIAL_DATA_HIT_SHARED, 2), + CCACHE_PMU_EVENT_ATTR(inner_atomic_data_hit_shared, INNER_ATOMIC_DATA_HIT_SHARED, 2), + CCACHE_PMU_EVENT_ATTR(inner_get_hit_shared, INNER_GET_HIT_SHARED, 2), + CCACHE_PMU_EVENT_ATTR(inner_prefetch_hit_shared, INNER_PREFETCH_HIT_SHARED, 2), + CCACHE_PMU_EVENT_ATTR(inner_acquire_block_hit_shared, INNER_ACQUIRE_BLOCK_HIT_SHARED, 2), + CCACHE_PMU_EVENT_ATTR(inner_acquire_perm_hit_shared, INNER_ACQUIRE_PERM_HIT_SHARED, 2), + CCACHE_PMU_EVENT_ATTR(outer_probe_hit_shared, OUTER_PROBE_HIT_SHARED, 2), + CCACHE_PMU_EVENT_ATTR(outer_probe_hit_dirty, OUTER_PROBE_HIT_DIRTY, 2), + + /* pmEventSelect3 */ + CCACHE_PMU_EVENT_ATTR(outer_acquire_block_ntob_miss, OUTER_ACQUIRE_BLOCK_NTOB_MISS, 3), + CCACHE_PMU_EVENT_ATTR(outer_acquire_block_ntot_miss, OUTER_ACQUIRE_BLOCK_NTOT_MISS, 3), + CCACHE_PMU_EVENT_ATTR(outer_acquire_block_btot_miss, OUTER_ACQUIRE_BLOCK_BTOT_MISS, 3), + CCACHE_PMU_EVENT_ATTR(outer_acquire_perm_ntot_miss, OUTER_ACQUIRE_PERM_NTOT_MISS, 3), + CCACHE_PMU_EVENT_ATTR(outer_acquire_perm_btot_miss, OUTER_ACQUIRE_PERM_BTOT_MISS, 3), + CCACHE_PMU_EVENT_ATTR(outer_release_ttob_eviction, OUTER_RELEASE_TTOB_EVICTION, 3), + CCACHE_PMU_EVENT_ATTR(outer_release_tton_eviction, OUTER_RELEASE_TTON_EVICTION, 3), + CCACHE_PMU_EVENT_ATTR(outer_release_bton_eviction, OUTER_RELEASE_BTON_EVICTION, 3), + CCACHE_PMU_EVENT_ATTR(outer_release_data_ttob_not_applicable, + OUTER_RELEASE_DATA_TTOB_NOT_APPLICABLE, 3), + CCACHE_PMU_EVENT_ATTR(outer_release_data_tton_dirty_eviction, + OUTER_RELEASE_DATA_TTON_DIRTY_EVICTION, 3), + CCACHE_PMU_EVENT_ATTR(outer_release_data_bton_not_applicable, + OUTER_RELEASE_DATA_BTON_NOT_APPLICABLE, 3), + CCACHE_PMU_EVENT_ATTR(inner_probe_block_tot_code_miss_hits_other_harts, + INNER_PROBE_BLOCK_TOT_CODE_MISS_HITS_OTHER_HARTS, 3), + CCACHE_PMU_EVENT_ATTR(inner_probe_block_tob_load_miss_hits_other_harts, + INNER_PROBE_BLOCK_TOB_LOAD_MISS_HITS_OTHER_HARTS, 3), + CCACHE_PMU_EVENT_ATTR(inner_probe_block_ton_store_miss_hits_other_harts, + INNER_PROBE_BLOCK_TON_STORE_MISS_HITS_OTHER_HARTS, 3), + + /* pm_event_select4 */ + CCACHE_PMU_EVENT_ATTR(inner_hint_hits_inflight_miss, INNER_HINT_HITS_INFLIGHT_MISS, 4), + NULL +}; + +static struct attribute_group sifive_ccache_pmu_events_group = { + .name = "events", + .attrs = sifive_ccache_pmu_events, +}; + +/* formats */ +PMU_FORMAT_ATTR(event, "config:0-63"); + +static struct attribute *sifive_ccache_pmu_formats[] = { + &format_attr_event.attr, + NULL, +}; + +static struct attribute_group sifive_ccache_pmu_format_group = { + .name = "format", + .attrs = sifive_ccache_pmu_formats, +}; + +/* + * Per PMU device attribute groups + */ + +static const struct attribute_group *sifive_ccache_pmu_attr_grps[] = { + &sifive_ccache_pmu_cpumask_group, + &sifive_ccache_pmu_events_group, + &sifive_ccache_pmu_format_group, + NULL, +}; + +/* + * Event Initialization + */ + +static int sifive_ccache_pmu_event_init(struct perf_event *event) +{ + struct sifive_ccache_pmu *ccache_pmu = to_ccache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + u64 config = event->attr.config; + u64 ev_type = config >> 8; + u64 set = config & 0xff; + + /* Check if this is a valid set and event */ + switch (set) { + case 1: + if (ev_type >= BIT_ULL(CCACHE_PMU_MAX_EVENT1_IDX)) + return -ENOENT; + break; + case 2: + if (ev_type >= BIT_ULL(CCACHE_PMU_MAX_EVENT2_IDX)) + return -ENOENT; + break; + case 3: + if (ev_type >= BIT_ULL(CCACHE_PMU_MAX_EVENT3_IDX)) + return -ENOENT; + break; + case 4: + if (ev_type >= BIT_ULL(CCACHE_PMU_MAX_EVENT4_IDX)) + return -ENOENT; + break; + default: + return -ENOENT; + } + + /* Do not allocate the hardware counter yet */ + hwc->idx = -1; + hwc->config = config; + + event->cpu = ccache_pmu->cpu; + + return 0; +} + +/* + * pmu->read: read and update the counter + */ +static void sifive_ccache_pmu_read(struct perf_event *event) +{ + struct hw_perf_event *hwc = &event->hw; + u64 prev_raw_count, new_raw_count; + u64 oldval; + + do { + prev_raw_count = local64_read(&hwc->prev_count); + new_raw_count = readq((void *)hwc->event_base); + + oldval = local64_cmpxchg(&hwc->prev_count, prev_raw_count, new_raw_count); + } while (oldval != prev_raw_count); + + local64_add(new_raw_count - prev_raw_count, &event->count); +} + +/* + * State transition functions: + * + * start()/stop() & add()/del() + */ + +/* + * pmu->start: start the event + */ +static void sifive_ccache_pmu_start(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc = &event->hw; + + if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED))) + return; + + hwc->state = 0; + + /* Set initial value to 0 */ + local64_set(&hwc->prev_count, 0); + writeq(0, (void *)hwc->event_base); + + /* Enable this counter to count events */ + writeq(hwc->config, (void *)hwc->config_base); +} + +/* + * pmu->stop: stop the counter + */ +static void sifive_ccache_pmu_stop(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc = &event->hw; + + if (hwc->state & PERF_HES_STOPPED) + return; + + /* Disable this counter to count events */ + writeq(0, (void *)hwc->config_base); + sifive_ccache_pmu_read(event); + + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; +} + +/* + * pmu->add: add the event to the PMU + */ +static int sifive_ccache_pmu_add(struct perf_event *event, int flags) +{ + struct sifive_ccache_pmu *ccache_pmu = to_ccache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + int idx; + + /* Find an available counter idx to use for this event */ + do { + idx = find_first_zero_bit(ccache_pmu->used_mask, ccache_pmu->n_counters); + if (idx >= ccache_pmu->n_counters) + return -EAGAIN; + } while (test_and_set_bit(idx, ccache_pmu->used_mask)); + + hwc->config_base = (unsigned long)ccache_pmu->base + CCACHE_SELECT_OFFSET + 8 * idx; + hwc->event_base = (unsigned long)ccache_pmu->base + CCACHE_COUNTER_OFFSET + 8 * idx; + hwc->idx = idx; + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; + + ccache_pmu->events[idx] = event; + + if (flags & PERF_EF_START) + sifive_ccache_pmu_start(event, PERF_EF_RELOAD); + + perf_event_update_userpage(event); + + return 0; +} + +/* + * pmu->del: delete the event from the PMU + */ +static void sifive_ccache_pmu_del(struct perf_event *event, int flags) +{ + struct sifive_ccache_pmu *ccache_pmu = to_ccache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + int idx = hwc->idx; + + /* Stop and release this counter */ + sifive_ccache_pmu_stop(event, PERF_EF_UPDATE); + + ccache_pmu->events[idx] = NULL; + clear_bit(idx, ccache_pmu->used_mask); + + perf_event_update_userpage(event); +} + +/* + * Driver initialization + */ + +static void sifive_ccache_pmu_hw_init(const struct sifive_ccache_pmu *ccache_pmu) +{ + /* Disable the client filter (not supported by this driver) */ + writeq(0, ccache_pmu->base + CCACHE_CLIENT_FILTER_OFFSET); +} + +static int sifive_ccache_pmu_online_cpu(unsigned int cpu, struct hlist_node *node) +{ + struct sifive_ccache_pmu *ccache_pmu = + hlist_entry_safe(node, struct sifive_ccache_pmu, node); + + if (ccache_pmu->cpu >= nr_cpu_ids) + ccache_pmu->cpu = cpu; + + return 0; +} + +static int sifive_ccache_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node) +{ + struct sifive_ccache_pmu *ccache_pmu = + hlist_entry_safe(node, struct sifive_ccache_pmu, node); + + /* Do nothing if this CPU does not own the events */ + if (cpu != ccache_pmu->cpu) + return 0; + + /* Pick a random online CPU */ + ccache_pmu->cpu = cpumask_any_but(cpu_online_mask, cpu); + if (ccache_pmu->cpu >= nr_cpu_ids) + return 0; + + /* Migrate PMU events from this CPU to the target CPU */ + perf_pmu_migrate_context(&ccache_pmu->pmu, cpu, ccache_pmu->cpu); + + return 0; +} + +static int sifive_ccache_pmu_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct sifive_ccache_pmu *ccache_pmu; + u32 n_counters; + int ret; + + /* Instances without a sifive,perfmon-counters property do not contain a PMU */ + ret = device_property_read_u32(dev, "sifive,perfmon-counters", &n_counters); + if (ret || !n_counters) + return -ENODEV; + + ccache_pmu = devm_kzalloc(dev, struct_size(ccache_pmu, events, n_counters), GFP_KERNEL); + if (!ccache_pmu) + return -ENOMEM; + + platform_set_drvdata(pdev, ccache_pmu); + + ccache_pmu->pmu = (struct pmu) { + .parent = dev, + .attr_groups = sifive_ccache_pmu_attr_grps, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, + .task_ctx_nr = perf_invalid_context, + .event_init = sifive_ccache_pmu_event_init, + .add = sifive_ccache_pmu_add, + .del = sifive_ccache_pmu_del, + .start = sifive_ccache_pmu_start, + .stop = sifive_ccache_pmu_stop, + .read = sifive_ccache_pmu_read, + }; + ccache_pmu->cpu = nr_cpu_ids; + ccache_pmu->n_counters = n_counters; + + ccache_pmu->base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(ccache_pmu->base)) + return PTR_ERR(ccache_pmu->base); + + sifive_ccache_pmu_hw_init(ccache_pmu); + + ret = cpuhp_state_add_instance(CPUHP_AP_PERF_RISCV_SIFIVE_CCACHE_ONLINE, &ccache_pmu->node); + if (ret) + return dev_err_probe(dev, ret, "Failed to add CPU hotplug instance\n"); + + ret = perf_pmu_register(&ccache_pmu->pmu, "sifive_ccache_pmu", -1); + if (ret) { + dev_err_probe(dev, ret, "Failed to register PMU\n"); + goto err_remove_instance; + } + + return 0; + +err_remove_instance: + cpuhp_state_remove_instance(CPUHP_AP_PERF_RISCV_SIFIVE_CCACHE_ONLINE, &ccache_pmu->node); + + return ret; +} + +static void sifive_ccache_pmu_remove(struct platform_device *pdev) +{ + struct sifive_ccache_pmu *ccache_pmu = platform_get_drvdata(pdev); + + perf_pmu_unregister(&ccache_pmu->pmu); + cpuhp_state_remove_instance(CPUHP_AP_PERF_RISCV_SIFIVE_CCACHE_ONLINE, &ccache_pmu->node); +} + +static const struct of_device_id sifive_ccache_pmu_of_match[] = { + { .compatible = "sifive,ccache0" }, + {} +}; +MODULE_DEVICE_TABLE(of, sifive_ccache_pmu_of_match); + +static struct platform_driver sifive_ccache_pmu_driver = { + .probe = sifive_ccache_pmu_probe, + .remove_new = sifive_ccache_pmu_remove, + .driver = { + .name = "sifive_ccache_pmu", + .of_match_table = sifive_ccache_pmu_of_match, + }, +}; + +static void __exit sifive_ccache_pmu_exit(void) +{ + platform_driver_unregister(&sifive_ccache_pmu_driver); + cpuhp_remove_multi_state(CPUHP_AP_PERF_RISCV_SIFIVE_CCACHE_ONLINE); +} +module_exit(sifive_ccache_pmu_exit); + +static int __init sifive_ccache_pmu_init(void) +{ + int ret; + + ret = cpuhp_setup_state_multi(CPUHP_AP_PERF_RISCV_SIFIVE_CCACHE_ONLINE, + "perf/sifive/ccache:online", + sifive_ccache_pmu_online_cpu, + sifive_ccache_pmu_offline_cpu); + if (ret) + return ret; + + ret = platform_driver_register(&sifive_ccache_pmu_driver); + if (ret) + goto err_remove_state; + + return 0; + +err_remove_state: + cpuhp_remove_multi_state(CPUHP_AP_PERF_RISCV_SIFIVE_CCACHE_ONLINE); + + return ret; +} +module_init(sifive_ccache_pmu_init); + +MODULE_LICENSE("GPL"); diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 172d0a743e5d..be6361fdc8ba 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -230,6 +230,7 @@ enum cpuhp_state { CPUHP_AP_PERF_POWERPC_TRACE_IMC_ONLINE, CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE, CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE, + CPUHP_AP_PERF_RISCV_SIFIVE_CCACHE_ONLINE, CPUHP_AP_PERF_CSKY_ONLINE, CPUHP_AP_WATCHDOG_ONLINE, CPUHP_AP_WORKQUEUE_ONLINE, From patchwork Fri Feb 16 00:08:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13559311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 454D2C48BC4 for ; Fri, 16 Feb 2024 00:09:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EVrvfmCfpPCJIKw9BNDttAyEAe7sSGwl119HMORxlnI=; b=ltsuJOjjMnXFBk CdreX8hC7ijaF+/gxL0WTPhcaX73uSMwp1p4PxteEMgtc2tgfJlNDUFdN9rlfm6ccDK0AT4SyMh8I WhWTTuv2KZ+eg3QV6abJMCEat3h/ymYxwMecoxj/pavDNS7dvK2zFAuApcQEEbXme1K8YVD6XQkf5 BibhCVVaNbyyj8YXNM77zf5StQpi2BnhjmkKeZxFAulzdSiH4r174AHQAYz7dcKMyUdyFhJAsBhmC Z4+HgD9V8fD+mqbgaZCCBmKQDh/5huJj4C2190oibDD4zki8rgfCbGUWaLn06zYkdEmQeqauKjhuu VRb8B3Hwl2z9IXO2yO2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmk-00000000Yhf-0pTD; Fri, 16 Feb 2024 00:08:58 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralma-00000000YaP-1LiH for linux-arm-kernel@lists.infradead.org; Fri, 16 Feb 2024 00:08:49 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-6e0aa4bb62fso841070b3a.1 for ; Thu, 15 Feb 2024 16:08:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1708042127; x=1708646927; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bVhGY5w5frFGTLRRHW7GA0aZ7SPA+FA1JdXFhjNvDZY=; b=SuxfDFVr9DzWh7QiR2QSy3FLOaDborM6UG1AVmP5XF6AATkce6cZxIr5ixBvllN5r9 lCAGOq3cdpkx0jjkonBbSlR6IJqeMo6RwZreaIwjnY8W1BX4uRW0Sv3TVCni4/+I2njb lQF8p1CZytf7h1moqcJqYHWWQydPbWM+1c2tUbGCTrBZZsoYObEge8/3U9wRq+YOl+KN 6nY2bwumYpLtK84Qx2vzhSqgT0qOevU0MnM8ufd76Ww7+DeoO9VNKGgF3wIikwd7qZFb vMmnu+k7FtU6Skjm2frHE0jsUfSgzi+5Li33YOMbl6YvithRKsJOtN0Qbga4Jy36udFh 9THA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708042127; x=1708646927; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bVhGY5w5frFGTLRRHW7GA0aZ7SPA+FA1JdXFhjNvDZY=; b=Spefr0ZTFHNRhC9u30mo8dUVorDdFWX3dMsNTQuvtkgRCfNvdKCEGDjSE9JM6M9cNw FlKvrdPNtwx4hOJiKZHw/jzke7SFSIvK/pzAGclpb+GHmpIuF96JbWi4gLliE7hSEQwl rRsVmt++k1AzFkTEorEh0rOpZfmiFd+ZuMMQ3AIyDXWbGOJIiTdoOmMFSWeX7hr2j83W zNDT38KCTDS3Z2jJCIYVtb5EIbE4sCDpgrgIADGuMgL8s46lcgA0MQsXa4LvMRcEEkBy akVQMvVGz6FNTsOI1om17RGGze5An0JXWmrNZsx9wF2yeCidypF1CG0ruF4wlhgA0/hN KqLA== X-Forwarded-Encrypted: i=1; AJvYcCXw4DQWlxCHEIqlq1u/zKDlzwtb2w+yzXU1u2P58U9eZKtfX+agOw5sFRQkIiwNnsg+eeJdS+PzL+SzjArN56+g1rV5ursJ0IApw6Xqr9rqWFfWCdI= X-Gm-Message-State: AOJu0Yy8UXi5cjkN8SRZuTcA8p80QbLqY48Zxu7djL5FE7Ucm4IQeeR7 NdwLzd324rztTqeQmsjqCSHq1PYGsFMigPfC/uO0buI+Pw4+CWS7ZpKHNr0HJKM= X-Google-Smtp-Source: AGHT+IETLEvEo995N/RIp3zMdESx5S+hcPKSxSPuG/aIgheazSPtzHakRWnlA/tfpia2Ssum3vBxWA== X-Received: by 2002:a17:902:d48b:b0:1db:8181:9d04 with SMTP id c11-20020a170902d48b00b001db81819d04mr3724031plg.38.1708042127232; Thu, 15 Feb 2024 16:08:47 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bb6-20020a170902bc8600b001db3d365082sm1789486plb.265.2024.02.15.16.08.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 16:08:45 -0800 (PST) From: Samuel Holland To: Will Deacon , Mark Rutland , Eric Lin , Conor Dooley Cc: Palmer Dabbelt , devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Walmsley , linux-riscv@lists.infradead.org, Rob Herring , Krzysztof Kozlowski , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [PATCH v1 3/6] dt-bindings: cache: Add SiFive Extensible Cache controller Date: Thu, 15 Feb 2024 16:08:15 -0800 Message-ID: <20240216000837.1868917-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240216000837.1868917-1-samuel.holland@sifive.com> References: <20240216000837.1868917-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240215_160848_399852_DAB8EBBE X-CRM114-Status: GOOD ( 14.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Lin Add YAML DT binding documentation for the SiFive Extensible Cache controller. The Extensible Cache controller interleaves cache blocks across a number of heterogeneous independently-programmed slices. Each slice contains an MMIO interface for configuration, cache maintenance, error reporting, and performance monitoring. Signed-off-by: Eric Lin Co-developed-by: Samuel Holland Signed-off-by: Samuel Holland --- .../cache/sifive,extensiblecache0.yaml | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 Documentation/devicetree/bindings/cache/sifive,extensiblecache0.yaml diff --git a/Documentation/devicetree/bindings/cache/sifive,extensiblecache0.yaml b/Documentation/devicetree/bindings/cache/sifive,extensiblecache0.yaml new file mode 100644 index 000000000000..d027114dbdba --- /dev/null +++ b/Documentation/devicetree/bindings/cache/sifive,extensiblecache0.yaml @@ -0,0 +1,136 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright (C) 2023-2024 SiFive, Inc. +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/cache/sifive,extensiblecache0.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: SiFive Extensible Cache Controller + +maintainers: + - Eric Lin + +description: + The SiFive Extensible Cache Controller provides a high-performance extensible + system (L2 or L3) cache. It is divided into several independent heterogeneous + slices, enabling a flexible topology and physical design. + +allOf: + - $ref: /schemas/cache-controller.yaml# + +select: + properties: + compatible: + contains: + enum: + - sifive,extensiblecache0 + + required: + - compatible + +properties: + compatible: + items: + - const: sifive,extensiblecache0 + - const: cache + + "#address-cells": true + "#size-cells": true + ranges: true + + interrupts: + maxItems: 1 + + cache-block-size: + const: 64 + + cache-level: true + cache-sets: true + cache-size: true + cache-unified: true + +patternProperties: + "^cache-controller@[0-9a-f]+$": + type: object + additionalProperties: false + properties: + reg: + maxItems: 1 + + cache-block-size: + const: 64 + + cache-sets: true + cache-size: true + cache-unified: true + + sifive,bm-event-counters: + $ref: /schemas/types.yaml#/definitions/uint32 + default: 0 + description: Number of bucket monitor registers in this slice + + sifive,cache-ways: + $ref: /schemas/types.yaml#/definitions/uint32 + description: Number of ways in this slice (independent of cache size) + + sifive,perfmon-counters: + $ref: /schemas/types.yaml#/definitions/uint32 + default: 0 + description: Number of PMU counter registers in this slice + + required: + - reg + - cache-block-size + - cache-sets + - cache-size + - cache-unified + - sifive,cache-ways + +required: + - compatible + - ranges + - interrupts + - cache-block-size + - cache-level + - cache-sets + - cache-size + - cache-unified + +additionalProperties: false + +examples: + - | + cache-controller@30040000 { + compatible = "sifive,extensiblecache0", "cache"; + ranges = <0x30040000 0x30040000 0x10000>; + interrupts = <0x4>; + cache-block-size = <0x40>; + cache-level = <3>; + cache-sets = <0x800>; + cache-size = <0x100000>; + cache-unified; + #address-cells = <1>; + #size-cells = <1>; + + cache-controller@30040000 { + reg = <0x30040000 0x4000>; + cache-block-size = <0x40>; + cache-sets = <0x400>; + cache-size = <0x80000>; + cache-unified; + sifive,bm-event-counters = <8>; + sifive,cache-ways = <9>; + sifive,perfmon-counters = <8>; + }; + + cache-controller@30044000 { + reg = <0x30044000 0x4000>; + cache-block-size = <0x40>; + cache-sets = <0x400>; + cache-size = <0x80000>; + cache-unified; + sifive,bm-event-counters = <8>; + sifive,cache-ways = <9>; + sifive,perfmon-counters = <8>; + }; + }; From patchwork Fri Feb 16 00:08:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13559313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9AC3BC4829E for ; Fri, 16 Feb 2024 00:09:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fUhVAoqn0emrfEPc/bIc/SkwzWtUFbGHsfihtACLJsA=; b=z8TwJPdXgr6YU0 MTihJfrpQa8S1BeDiIvsFjVJ9W3EFyGwOaSGMkhgp2NhBms6Cfg66DswAECccq3kr7HT8JFrn8q4P ihUxJOFTP8OL+nT1BLA9mDZxek4P26oVEwh5yNlUngO8491LtW8ZCoCjvS2vkNx79w+dbcwLmGfqO tCr257HiZulChmuTT0yRgZovZuZ+Sf7ja3t1it97ohnT+MUhLvGhoeyMYzGW+WWh53Esx232JliNP xVUXg1tG25d08HTPDlr3z4EIvE8wo3qKPB2EIvvHPh0DykB0whPlc+mjUF60DmPSSePxdQgl19NyL 21n+m+WloDKUwdZxRCjQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmy-00000000YuM-0GAS; Fri, 16 Feb 2024 00:09:12 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmc-00000000Ybd-24AX for linux-arm-kernel@lists.infradead.org; Fri, 16 Feb 2024 00:08:53 +0000 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1d953fa3286so1041265ad.2 for ; Thu, 15 Feb 2024 16:08:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1708042130; x=1708646930; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=97Vl//GERs+wHDqqFSQvA7qG8VAex8iXqN6MPz7a8Ck=; b=WDrm3gDblFEyJT1P3adsXhdYmMZ/tLYXn/CNcTD0KcQjsrM+c7yWQUUny1+YZmYbCh XbGVtxZBiOWV1RAH7Uv/UcqkzNBhGN7exK6fjnBr0K7BQCs+iaG/xs3oW12rodWEUV/Q W+mFUkQAtzbd1XimUAYJWPzoxRGTNzpHeH8/gSfdwfteWpEDkfBjrFwxMDsO5NV1zDRp eqvjuY+z44G54KWa96gtN59NII7rP3zl8Ur7XY522z4QXiXLahJ34upTv3KWbYhB6Yw7 y7LyxiYo2Ty0agGdd2vSSVuHwxtXQhkEXJObvN0IVKiLd85zsnSsahEzAr06vNZvYkTy NCtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708042130; x=1708646930; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=97Vl//GERs+wHDqqFSQvA7qG8VAex8iXqN6MPz7a8Ck=; b=Dbwy0nut2Q2vQGbkP99YXJGXZAkxdL4k97lWSPuq4CyHg3eYgKTJq/vmx//jx6Uh6V c8TyY9hBjn1yW8Z2iHZ7NXdDY4qfpfs8jh/AZyrxgDIWa8VqpvKJuzpaavix8IcF/SJR K8x/MHTQCr6k1ZwUE9Dpf1dcZfuYQVlP/1N+atzSa/6ImFg0nKZbEf9T3/gsOaMMtZUN ralHSmvGymMF42K0dA/uERSj2gXMf0V2D7blRKMBuRtPNyJpunwczAIyJ4CRWXD7+tEy f7x2xDFXcib445S+yKZgXdWpEVCHZE+MIkKfn9i4FhUwcAyIj40172Ggb9OblnKntfd4 pKoQ== X-Forwarded-Encrypted: i=1; AJvYcCWKfjruMePh64PSkEocXe5hJ3aXcZF3tGqD8BBs9EOjh10D5fmNlsP9R9MAHEZ2UVYP/6hmaH+FkGV8ptVXQRQ1wIw1jqZxiQQOd1kqtiHQiJ7fFzg= X-Gm-Message-State: AOJu0Yywn3mz9Xkxn+qtqEDtotD4ycd39/J2Uh/dt/12j18x3ZPUXmSK 3p0ph3uMkWH5c8l8YtAXrRD3VfPRet3EOHpFawvNDroCCcS4WEgQM7RUxC9YD1g= X-Google-Smtp-Source: AGHT+IHM3CCjf5Wmu1mQ0DXHmEVhMoKJarp5BUlbNPnrFrVhOAN6O5I6g9AI9OcGFnljzXBeMdhPNw== X-Received: by 2002:a17:903:443:b0:1db:a54b:acaf with SMTP id iw3-20020a170903044300b001dba54bacafmr558755plb.39.1708042129975; Thu, 15 Feb 2024 16:08:49 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bb6-20020a170902bc8600b001db3d365082sm1789486plb.265.2024.02.15.16.08.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 16:08:48 -0800 (PST) From: Samuel Holland To: Will Deacon , Mark Rutland , Eric Lin , Conor Dooley Cc: Palmer Dabbelt , devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Walmsley , linux-riscv@lists.infradead.org, Rob Herring , Krzysztof Kozlowski , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [PATCH v1 4/6] drivers/perf: Add SiFive Extensible Cache PMU driver Date: Thu, 15 Feb 2024 16:08:16 -0800 Message-ID: <20240216000837.1868917-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240216000837.1868917-1-samuel.holland@sifive.com> References: <20240216000837.1868917-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240215_160850_618102_2B66CE84 X-CRM114-Status: GOOD ( 25.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Lin Add a driver for the PMU found in the SiFive Extensible Cache controller. This PMU provides a configurable number of counters and a variety of events. Events are grouped into sets. Each counter can count events from only one set at a time; however, it can count any number of events within that set simultaneously. The PMU hardware does not provide an overflow interrupt. The counter inhibit register is used to atomically start/stop/read a group of counters so their values can be usefully compared. Some events can be filtered further by client ID (e.g. CPU or external DMA master). That functionality is not supported by this driver. This driver further assumes that a single Extensible Cache instance is shared by all CPUs in the system. Example usage: $ perf stat -e sifive_ecache_pmu/inner_rd_request/, sifive_ecache_pmu/inner_wr_request/, sifive_ecache_pmu/inner_rd_request_hit/, sifive_ecache_pmu/inner_wr_request_hit/ ls Performance counter stats for 'system wide': 148001 sifive_ecache_pmu/inner_rd_request/ 121064 sifive_ecache_pmu/inner_wr_request/ 113124 sifive_ecache_pmu/inner_rd_request_hit/ 120860 sifive_ecache_pmu/inner_wr_request_hit/ 0.010643962 seconds time elapsed Example combining the read/write events together within each counter: $ perf stat -e sifive_ecache_pmu/event=0x601/, sifive_ecache_pmu/event=0xc001/ ls Performance counter stats for 'system wide': 262619 sifive_ecache_pmu/event=0x601/ 224533 sifive_ecache_pmu/event=0xc001/ 0.009794808 seconds time elapsed Signed-off-by: Eric Lin Co-developed-by: Samuel Holland Signed-off-by: Samuel Holland --- drivers/perf/Kconfig | 10 + drivers/perf/Makefile | 1 + drivers/perf/sifive_ecache_pmu.c | 675 +++++++++++++++++++++++++++++++ include/linux/cpuhotplug.h | 1 + 4 files changed, 687 insertions(+) create mode 100644 drivers/perf/sifive_ecache_pmu.c diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index b4e4db7424b4..8a3b2b88d8b5 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -164,6 +164,16 @@ config SIFIVE_CCACHE_PMU for measuring whole-system L2/L3 cache performance using the perf events subsystem. +config SIFIVE_ECACHE_PMU + tristate "SiFive Extensible Cache PMU" + depends on RISCV || COMPILE_TEST + depends on OF + help + Support for the Extensible Cache performance monitoring unit (PMU) on + SiFive platforms. The Composable Cache PMU provides up to 8 counters + for measuring whole-system L2/L3 cache performance using the perf + events subsystem. + config THUNDERX2_PMU tristate "Cavium ThunderX2 SoC PMU UNCORE" depends on ARCH_THUNDER2 || COMPILE_TEST diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index 51ef5f50ace4..a51686b413f2 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -16,6 +16,7 @@ obj-$(CONFIG_RISCV_PMU) += riscv_pmu.o obj-$(CONFIG_RISCV_PMU_LEGACY) += riscv_pmu_legacy.o obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o obj-$(CONFIG_SIFIVE_CCACHE_PMU) += sifive_ccache_pmu.o +obj-$(CONFIG_SIFIVE_ECACHE_PMU) += sifive_ecache_pmu.o obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o obj-$(CONFIG_ARM_SPE_PMU) += arm_spe_pmu.o diff --git a/drivers/perf/sifive_ecache_pmu.c b/drivers/perf/sifive_ecache_pmu.c new file mode 100644 index 000000000000..51b2fa3781c9 --- /dev/null +++ b/drivers/perf/sifive_ecache_pmu.c @@ -0,0 +1,675 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * SiFive EC (Extensible Cache) PMU driver + * + * Copyright (C) 2023-2024 SiFive, Inc. + * Copyright (C) Eric Lin + * + */ + +#include +#include +#include +#include +#include +#include +#include + +#define ECACHE_SELECT_OFFSET 0x2000 +#define ECACHE_CLIENT_FILTER_OFFSET 0x2200 +#define ECACHE_COUNTER_INHIBIT_OFFSET 0x2800 +#define ECACHE_COUNTER_OFFSET 0x3000 + +#define ECACHE_PMU_MAX_COUNTERS 8 + +struct sifive_ecache_pmu_slice { + void __iomem *base; +}; + +struct sifive_ecache_pmu { + struct pmu pmu; + struct hlist_node node; + struct notifier_block cpu_pm_nb; + struct perf_event *events[ECACHE_PMU_MAX_COUNTERS]; + DECLARE_BITMAP(used_mask, ECACHE_PMU_MAX_COUNTERS); + unsigned int cpu; + unsigned int txn_flags; + int n_counters; + int n_slices; + struct sifive_ecache_pmu_slice slice[] __counted_by(n_slices); +}; + +#define to_ecache_pmu(p) (container_of(p, struct sifive_ecache_pmu, pmu)) + +/* Store the counter mask for a group in the leader's extra_reg */ +#define event_group_mask(event) (event->group_leader->hw.extra_reg.config) + +#ifndef readq +static inline u64 readq(void __iomem *addr) +{ + return readl(addr) | (((u64)readl(addr + 4)) << 32); +} +#endif + +#ifndef writeq +static inline void writeq(u64 v, void __iomem *addr) +{ + writel(lower_32_bits(v), addr); + writel(upper_32_bits(v), addr + 4); +} +#endif + +/* + * sysfs attributes + * + * We export: + * - cpumask, used by perf user space and other tools to know on which CPUs to create events + * - events, used by perf user space and other tools to create events symbolically, e.g.: + * perf stat -a -e sifive_ecache_pmu/event=inner_put_partial_data_hit/ ls + * perf stat -a -e sifive_ecache_pmu/event=0x101/ ls + * - formats, used by perf user space and other tools to configure events + */ + +/* cpumask */ +static ssize_t cpumask_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + struct sifive_ecache_pmu *ecache_pmu = dev_get_drvdata(dev); + + if (ecache_pmu->cpu >= nr_cpu_ids) + return 0; + + return sysfs_emit(buf, "%d\n", ecache_pmu->cpu); +}; + +static DEVICE_ATTR_RO(cpumask); + +static struct attribute *sifive_ecache_pmu_cpumask_attrs[] = { + &dev_attr_cpumask.attr, + NULL, +}; + +static const struct attribute_group sifive_ecache_pmu_cpumask_group = { + .attrs = sifive_ecache_pmu_cpumask_attrs, +}; + +/* events */ +static ssize_t sifive_ecache_pmu_event_show(struct device *dev, struct device_attribute *attr, + char *page) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr); + return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id); +} + +#define SET_EVENT_SELECT(_event, _set) (BIT_ULL((_event) + 8) | (_set)) +#define ECACHE_PMU_EVENT_ATTR(_name, _event, _set) \ + PMU_EVENT_ATTR_ID(_name, sifive_ecache_pmu_event_show, SET_EVENT_SELECT(_event, _set)) + +enum ecache_pmu_event_set1 { + INNER_REQUEST = 0, + INNER_RD_REQUEST, + INNER_WR_REQUEST, + INNER_PF_REQUEST, + OUTER_PRB_REQUEST, + INNER_REQUEST_HIT, + INNER_RD_REQUEST_HIT, + INNER_WR_REQUEST_HIT, + INNER_PF_REQUEST_HIT, + OUTER_PRB_REQUEST_HIT, + INNER_REQUEST_HITPF, + INNER_RD_REQUEST_HITPF, + INNER_WR_REQUEST_HITPF, + INNER_PF_REQUEST_HITPF, + OUTER_PRB_REQUEST_HITPF, + INNER_REQUEST_MISS, + INNER_RD_REQUEST_MISS, + INNER_WR_REQUEST_MISS, + INNER_PF_REQUEST_MISS, + OUTER_PRB_REQUEST_MISS, + ECACHE_PMU_MAX_EVENT1_IDX +}; + +enum ecache_pmu_event_set2 { + OUTER_REQUEST = 0, + OUTER_RD_REQUEST, + OUTER_PUT_REQUEST, + OUTER_EV_REQUEST, + OUTER_PF_REQUEST, + INNER_PRB_REQUEST, + INNER_REQUEST_WCYC, + INNER_RD_REQUEST_WCYC, + INNER_WR_REQUEST_WCYC, + INNER_PF_REQUEST_WCYC, + OUTER_PRB_REQUEST_WCYC, + OUTER_REQUEST_WCYC, + OUTER_RD_REQUEST_WCYC, + OUTER_PUT_REQUEST_WCYC, + OUTER_EV_REQUEST_WCYC, + OUTER_PF_REQUEST_WCYC, + INNER_PRB_REQUEST_WCYC, + INNER_AG_WCYC, + INNER_AP_WCYC, + INNER_AH_WCYC, + INNER_BP_WCYC, + INNER_CP_WCYC, + INNER_CX_WCYC, + INNER_DG_WCYC, + INNER_DP_WCYC, + INNER_DX_WCYC, + INNER_EG_WCYC, + OUTER_AG_WCYC, + OUTER_AP_WCYC, + OUTER_AH_WCYC, + OUTER_BP_WCYC, + OUTER_CP_WCYC, + OUTER_CX_WCYC, + OUTER_DG_WCYC, + OUTER_DP_WCYC, + OUTER_DX_WCYC, + OUTER_EG_WCYC, + ECACHE_PMU_MAX_EVENT2_IDX +}; + +static struct attribute *sifive_ecache_pmu_events[] = { + /* pmEventSelect1 */ + ECACHE_PMU_EVENT_ATTR(inner_request, INNER_REQUEST, 1), + ECACHE_PMU_EVENT_ATTR(inner_rd_request, INNER_RD_REQUEST, 1), + ECACHE_PMU_EVENT_ATTR(inner_wr_request, INNER_WR_REQUEST, 1), + ECACHE_PMU_EVENT_ATTR(inner_pf_request, INNER_PF_REQUEST, 1), + ECACHE_PMU_EVENT_ATTR(outer_prb_request, OUTER_PRB_REQUEST, 1), + ECACHE_PMU_EVENT_ATTR(inner_request_hit, INNER_REQUEST_HIT, 1), + ECACHE_PMU_EVENT_ATTR(inner_rd_request_hit, INNER_RD_REQUEST_HIT, 1), + ECACHE_PMU_EVENT_ATTR(inner_wr_request_hit, INNER_WR_REQUEST_HIT, 1), + ECACHE_PMU_EVENT_ATTR(inner_pf_request_hit, INNER_PF_REQUEST_HIT, 1), + ECACHE_PMU_EVENT_ATTR(outer_prb_request_hit, OUTER_PRB_REQUEST_HIT, 1), + ECACHE_PMU_EVENT_ATTR(inner_request_hitpf, INNER_REQUEST_HITPF, 1), + ECACHE_PMU_EVENT_ATTR(inner_rd_request_hitpf, INNER_RD_REQUEST_HITPF, 1), + ECACHE_PMU_EVENT_ATTR(inner_wr_request_hitpf, INNER_WR_REQUEST_HITPF, 1), + ECACHE_PMU_EVENT_ATTR(inner_pf_request_hitpf, INNER_PF_REQUEST_HITPF, 1), + ECACHE_PMU_EVENT_ATTR(outer_prb_request_hitpf, OUTER_PRB_REQUEST_HITPF, 1), + ECACHE_PMU_EVENT_ATTR(inner_request_miss, INNER_REQUEST_MISS, 1), + ECACHE_PMU_EVENT_ATTR(inner_rd_request_miss, INNER_RD_REQUEST_MISS, 1), + ECACHE_PMU_EVENT_ATTR(inner_wr_request_miss, INNER_WR_REQUEST_MISS, 1), + ECACHE_PMU_EVENT_ATTR(inner_pf_request_miss, INNER_PF_REQUEST_MISS, 1), + ECACHE_PMU_EVENT_ATTR(outer_prb_request_miss, OUTER_PRB_REQUEST_MISS, 1), + + /* pmEventSelect2 */ + ECACHE_PMU_EVENT_ATTR(outer_request, OUTER_REQUEST, 2), + ECACHE_PMU_EVENT_ATTR(outer_rd_request, OUTER_RD_REQUEST, 2), + ECACHE_PMU_EVENT_ATTR(outer_put_request, OUTER_PUT_REQUEST, 2), + ECACHE_PMU_EVENT_ATTR(outer_ev_request, OUTER_EV_REQUEST, 2), + ECACHE_PMU_EVENT_ATTR(outer_pf_request, OUTER_PF_REQUEST, 2), + ECACHE_PMU_EVENT_ATTR(inner_prb_request, INNER_PRB_REQUEST, 2), + ECACHE_PMU_EVENT_ATTR(inner_request_wcyc, INNER_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_rd_request_wcyc, INNER_RD_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_wr_request_wcyc, INNER_WR_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_pf_request_wcyc, INNER_PF_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_prb_request_wcyc, OUTER_PRB_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_request_wcyc, OUTER_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_rd_request_wcyc, OUTER_RD_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_put_request_wcyc, OUTER_PUT_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_ev_request_wcyc, OUTER_EV_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_pf_request_wcyc, OUTER_PF_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_prb_request_wcyc, INNER_PRB_REQUEST_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_ag_wcyc, INNER_AG_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_ap_wcyc, INNER_AP_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_ah_wcyc, INNER_AH_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_bp_wcyc, INNER_BP_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_cp_wcyc, INNER_CP_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_cx_wcyc, INNER_CX_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_dg_wcyc, INNER_DG_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_dp_wcyc, INNER_DP_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_dx_wcyc, INNER_DX_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(inner_eg_wcyc, INNER_EG_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_ag_wcyc, OUTER_AG_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_ap_wcyc, OUTER_AP_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_ah_wcyc, OUTER_AH_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_bp_wcyc, OUTER_BP_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_cp_wcyc, OUTER_CP_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_cx_wcyc, OUTER_CX_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_dg_wcyc, OUTER_DG_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_dp_wcyc, OUTER_DP_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_dx_wcyc, OUTER_DX_WCYC, 2), + ECACHE_PMU_EVENT_ATTR(outer_eg_wcyc, OUTER_EG_WCYC, 2), + NULL +}; + +static struct attribute_group sifive_ecache_pmu_events_group = { + .name = "events", + .attrs = sifive_ecache_pmu_events, +}; + +/* formats */ +PMU_FORMAT_ATTR(event, "config:0-63"); + +static struct attribute *sifive_ecache_pmu_formats[] = { + &format_attr_event.attr, + NULL, +}; + +static struct attribute_group sifive_ecache_pmu_format_group = { + .name = "format", + .attrs = sifive_ecache_pmu_formats, +}; + +/* + * Per PMU device attribute groups + */ + +static const struct attribute_group *sifive_ecache_pmu_attr_grps[] = { + &sifive_ecache_pmu_cpumask_group, + &sifive_ecache_pmu_events_group, + &sifive_ecache_pmu_format_group, + NULL, +}; + +/* + * Event Initialization + */ + +static int sifive_ecache_pmu_event_init(struct perf_event *event) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + u64 config = event->attr.config; + u64 ev_type = config >> 8; + u64 set = config & 0xff; + + /* Check if this is a valid set and event */ + switch (set) { + case 1: + if (ev_type >= BIT_ULL(ECACHE_PMU_MAX_EVENT1_IDX)) + return -ENOENT; + break; + case 2: + if (ev_type >= BIT_ULL(ECACHE_PMU_MAX_EVENT2_IDX)) + return -ENOENT; + break; + default: + return -ENOENT; + } + + /* Do not allocate the hardware counter yet */ + hwc->idx = -1; + hwc->config = config; + + event->cpu = ecache_pmu->cpu; + + return 0; +} + +/* + * Low-level functions: reading and writing counters + */ + +static void configure_counter(const struct sifive_ecache_pmu *ecache_pmu, + const struct hw_perf_event *hwc, u64 config) +{ + for (int i = 0; i < ecache_pmu->n_slices; i++) { + void __iomem *base = ecache_pmu->slice[i].base; + + if (config) + writeq(0, base + hwc->event_base); + writeq(config, base + hwc->config_base); + } +} + +static u64 read_counter(const struct sifive_ecache_pmu *ecache_pmu, const struct hw_perf_event *hwc) +{ + u64 value = 0; + + for (int i = 0; i < ecache_pmu->n_slices; i++) { + void __iomem *base = ecache_pmu->slice[i].base; + + value += readq(base + hwc->event_base); + } + + return value; +} + +static void write_inhibit(const struct sifive_ecache_pmu *ecache_pmu, u64 mask) +{ + u64 used_mask; + + /* Inhibit all unused counters in addition to the provided mask */ + bitmap_to_arr64(&used_mask, ecache_pmu->used_mask, ECACHE_PMU_MAX_COUNTERS); + mask |= ~used_mask; + + for (int i = 0; i < ecache_pmu->n_slices; i++) { + void __iomem *base = ecache_pmu->slice[i].base; + + writeq(mask, base + ECACHE_COUNTER_INHIBIT_OFFSET); + } +} + +/* + * pmu->read: read and update the counter + */ +static void sifive_ecache_pmu_read(struct perf_event *event) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + u64 prev_raw_count, new_raw_count; + u64 oldval; + + /* Inhibit the entire group during a read transaction for atomicity */ + if (ecache_pmu->txn_flags == PERF_PMU_TXN_READ && event->group_leader == event) + write_inhibit(ecache_pmu, event_group_mask(event)); + + do { + prev_raw_count = local64_read(&hwc->prev_count); + new_raw_count = read_counter(ecache_pmu, hwc); + + oldval = local64_cmpxchg(&hwc->prev_count, prev_raw_count, new_raw_count); + } while (oldval != prev_raw_count); + + local64_add(new_raw_count - prev_raw_count, &event->count); +} + +/* + * State transition functions: + * + * start()/stop() & add()/del() + */ + +/* + * pmu->start: start the event + */ +static void sifive_ecache_pmu_start(struct perf_event *event, int flags) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + + if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED))) + return; + + hwc->state = 0; + + /* Set initial value to 0 */ + local64_set(&hwc->prev_count, 0); + + /* Enable this counter to count events */ + configure_counter(ecache_pmu, hwc, hwc->config); +} + +/* + * pmu->stop: stop the counter + */ +static void sifive_ecache_pmu_stop(struct perf_event *event, int flags) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + + if (hwc->state & PERF_HES_STOPPED) + return; + + /* Disable this counter to count events */ + configure_counter(ecache_pmu, hwc, 0); + sifive_ecache_pmu_read(event); + + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; +} + +/* + * pmu->add: add the event to the PMU + */ +static int sifive_ecache_pmu_add(struct perf_event *event, int flags) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + int idx; + + /* Find an available counter idx to use for this event */ + do { + idx = find_first_zero_bit(ecache_pmu->used_mask, ecache_pmu->n_counters); + if (idx >= ecache_pmu->n_counters) + return -EAGAIN; + } while (test_and_set_bit(idx, ecache_pmu->used_mask)); + + event_group_mask(event) |= BIT_ULL(idx); + hwc->config_base = ECACHE_SELECT_OFFSET + 8 * idx; + hwc->event_base = ECACHE_COUNTER_OFFSET + 8 * idx; + hwc->idx = idx; + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; + + ecache_pmu->events[idx] = event; + + if (flags & PERF_EF_START) + sifive_ecache_pmu_start(event, PERF_EF_RELOAD); + + perf_event_update_userpage(event); + + return 0; +} + +/* + * pmu->del: delete the event from the PMU + */ +static void sifive_ecache_pmu_del(struct perf_event *event, int flags) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + int idx = hwc->idx; + + /* Stop and release this counter */ + sifive_ecache_pmu_stop(event, PERF_EF_UPDATE); + + ecache_pmu->events[idx] = NULL; + clear_bit(idx, ecache_pmu->used_mask); + + perf_event_update_userpage(event); +} + +/* + * Transaction synchronization + */ + +static void sifive_ecache_pmu_start_txn(struct pmu *pmu, unsigned int txn_flags) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(pmu); + + ecache_pmu->txn_flags = txn_flags; + + /* Inhibit any counters that were deleted since the last transaction */ + if (txn_flags == PERF_PMU_TXN_ADD) + write_inhibit(ecache_pmu, 0); +} + +static int sifive_ecache_pmu_commit_txn(struct pmu *pmu) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(pmu); + + ecache_pmu->txn_flags = 0; + + /* Successful transaction: atomically uninhibit the counters in this group */ + write_inhibit(ecache_pmu, 0); + + return 0; +} + +static void sifive_ecache_pmu_cancel_txn(struct pmu *pmu) +{ + struct sifive_ecache_pmu *ecache_pmu = to_ecache_pmu(pmu); + + ecache_pmu->txn_flags = 0; + + /* Failed transaction: leave the counters in this group inhibited */ +} + +/* + * Driver initialization + */ + +static void sifive_ecache_pmu_hw_init(const struct sifive_ecache_pmu *ecache_pmu) +{ + for (int i = 0; i < ecache_pmu->n_slices; i++) { + void __iomem *base = ecache_pmu->slice[i].base; + + /* Disable the client filter (not supported by this driver) */ + writeq(0, base + ECACHE_CLIENT_FILTER_OFFSET); + } +} + +static int sifive_ecache_pmu_online_cpu(unsigned int cpu, struct hlist_node *node) +{ + struct sifive_ecache_pmu *ecache_pmu = + hlist_entry_safe(node, struct sifive_ecache_pmu, node); + + if (ecache_pmu->cpu >= nr_cpu_ids) + ecache_pmu->cpu = cpu; + + return 0; +} + +static int sifive_ecache_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node) +{ + struct sifive_ecache_pmu *ecache_pmu = + hlist_entry_safe(node, struct sifive_ecache_pmu, node); + + /* Do nothing if this CPU does not own the events */ + if (cpu != ecache_pmu->cpu) + return 0; + + /* Pick a random online CPU */ + ecache_pmu->cpu = cpumask_any_but(cpu_online_mask, cpu); + if (ecache_pmu->cpu >= nr_cpu_ids) + return 0; + + /* Migrate PMU events from this CPU to the target CPU */ + perf_pmu_migrate_context(&ecache_pmu->pmu, cpu, ecache_pmu->cpu); + + return 0; +} + +static int sifive_ecache_pmu_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct device_node *ecache_node = dev_of_node(dev); + struct sifive_ecache_pmu *ecache_pmu; + struct device_node *slice_node; + u32 slice_counters; + int n_slices, ret; + int i = 0; + + n_slices = of_get_available_child_count(ecache_node); + if (!n_slices) + return -ENODEV; + + ecache_pmu = devm_kzalloc(dev, struct_size(ecache_pmu, slice, n_slices), GFP_KERNEL); + if (!ecache_pmu) + return -ENOMEM; + + platform_set_drvdata(pdev, ecache_pmu); + + ecache_pmu->pmu = (struct pmu) { + .parent = dev, + .attr_groups = sifive_ecache_pmu_attr_grps, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, + .task_ctx_nr = perf_invalid_context, + .event_init = sifive_ecache_pmu_event_init, + .add = sifive_ecache_pmu_add, + .del = sifive_ecache_pmu_del, + .start = sifive_ecache_pmu_start, + .stop = sifive_ecache_pmu_stop, + .read = sifive_ecache_pmu_read, + .start_txn = sifive_ecache_pmu_start_txn, + .commit_txn = sifive_ecache_pmu_commit_txn, + .cancel_txn = sifive_ecache_pmu_cancel_txn, + }; + ecache_pmu->cpu = nr_cpu_ids; + ecache_pmu->n_counters = ECACHE_PMU_MAX_COUNTERS; + ecache_pmu->n_slices = n_slices; + + for_each_available_child_of_node(ecache_node, slice_node) { + struct sifive_ecache_pmu_slice *slice = &ecache_pmu->slice[i++]; + + slice->base = devm_of_iomap(dev, slice_node, 0, NULL); + if (IS_ERR(slice->base)) + return PTR_ERR(slice->base); + + /* Get number of counters from slice node */ + ret = of_property_read_u32(slice_node, "sifive,perfmon-counters", &slice_counters); + if (ret) + return dev_err_probe(dev, ret, + "Slice %pOF missing sifive,perfmon-counters property\n", + slice_node); + + ecache_pmu->n_counters = min_t(u32, slice_counters, ecache_pmu->n_counters); + } + + sifive_ecache_pmu_hw_init(ecache_pmu); + + ret = cpuhp_state_add_instance(CPUHP_AP_PERF_RISCV_SIFIVE_ECACHE_ONLINE, &ecache_pmu->node); + if (ret) + return dev_err_probe(dev, ret, "Failed to add CPU hotplug instance\n"); + + ret = perf_pmu_register(&ecache_pmu->pmu, "sifive_ecache_pmu", -1); + if (ret) { + dev_err_probe(dev, ret, "Failed to register PMU\n"); + goto err_remove_instance; + } + + return 0; + +err_remove_instance: + cpuhp_state_remove_instance(CPUHP_AP_PERF_RISCV_SIFIVE_ECACHE_ONLINE, &ecache_pmu->node); + + return ret; +} + +static void sifive_ecache_pmu_remove(struct platform_device *pdev) +{ + struct sifive_ecache_pmu *ecache_pmu = platform_get_drvdata(pdev); + + perf_pmu_unregister(&ecache_pmu->pmu); + cpuhp_state_remove_instance(CPUHP_AP_PERF_RISCV_SIFIVE_ECACHE_ONLINE, &ecache_pmu->node); +} + +static const struct of_device_id sifive_ecache_pmu_of_match[] = { + { .compatible = "sifive,extensiblecache0" }, + {} +}; +MODULE_DEVICE_TABLE(of, sifive_ecache_pmu_of_match); + +static struct platform_driver sifive_ecache_pmu_driver = { + .probe = sifive_ecache_pmu_probe, + .remove_new = sifive_ecache_pmu_remove, + .driver = { + .name = "sifive_ecache_pmu", + .of_match_table = sifive_ecache_pmu_of_match, + }, +}; + +static void __exit sifive_ecache_pmu_exit(void) +{ + platform_driver_unregister(&sifive_ecache_pmu_driver); + cpuhp_remove_multi_state(CPUHP_AP_PERF_RISCV_SIFIVE_ECACHE_ONLINE); +} +module_exit(sifive_ecache_pmu_exit); + +static int __init sifive_ecache_pmu_init(void) +{ + int ret; + + ret = cpuhp_setup_state_multi(CPUHP_AP_PERF_RISCV_SIFIVE_ECACHE_ONLINE, + "perf/sifive/ecache:online", + sifive_ecache_pmu_online_cpu, + sifive_ecache_pmu_offline_cpu); + if (ret) + return ret; + + ret = platform_driver_register(&sifive_ecache_pmu_driver); + if (ret) + goto err_remove_state; + + return 0; + +err_remove_state: + cpuhp_remove_multi_state(CPUHP_AP_PERF_RISCV_SIFIVE_ECACHE_ONLINE); + + return ret; +} +module_init(sifive_ecache_pmu_init); + +MODULE_LICENSE("GPL"); diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index be6361fdc8ba..55bd3a5e0033 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -231,6 +231,7 @@ enum cpuhp_state { CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE, CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE, CPUHP_AP_PERF_RISCV_SIFIVE_CCACHE_ONLINE, + CPUHP_AP_PERF_RISCV_SIFIVE_ECACHE_ONLINE, CPUHP_AP_PERF_CSKY_ONLINE, CPUHP_AP_WATCHDOG_ONLINE, CPUHP_AP_WORKQUEUE_ONLINE, From patchwork Fri Feb 16 00:08:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13559371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E1C0C48BC4 for ; Fri, 16 Feb 2024 01:17:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tWBPXpglh391nhAltrjSQk1Kfxbl3TeIn5ekF1Ypvho=; b=qA7jSGl6lw6tW1 UrfvehpsmXdHVyvX9BiHwvFrFBxXDpdYR/+2j9mWLAr00l744TgA+13ZJYaHpsdVIO6ruNYPkadUV Bl1WzSWXCCVKiu382nYqx32XZrWZtZ0vvsiZrBaoISJjzixnA/7D1fI6q3rSzQgOARQgI4HJBVWLd 0SjjKzOuc3EzNx+TY3cmWJX9hSh4NiHCIzzX2FHYLXGk/7mBb6PSnLR6zOnujcOnzBAAYcZO0L4FW 6EQCysiMzNYFwD7ZdbXfpuRqUdNQR/vSpbhZ/lhc83C+tSmjIw0P4krYwDdC3MKFE5jo/wp2rUEML C33QON2Bxau9ZcpFUeRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ramr3-00000000iMc-0SMt; Fri, 16 Feb 2024 01:17:29 +0000 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmg-00000000Yd8-3gqF for linux-arm-kernel@lists.infradead.org; Fri, 16 Feb 2024 00:08:58 +0000 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1db562438e0so13114705ad.3 for ; Thu, 15 Feb 2024 16:08:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1708042133; x=1708646933; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pGHwuS0ENdmLMLz6bWQqaQXZhO2uE/iwucpykSfpWsk=; b=YdIxGRZ2bqb831ZhYX0Byh4fBuQQGIAou0161St4j+Muz9Kej22t0tja7bW+LBrCSj 3vbneukX/aQIyI8hgRODewHrK0DGggp1MI0Om32ID3np7KRxf057spx1Vwz6YK6R3Enn C5DFSw+tvcNOaBJEt46eSztZJHoBQ34G4k1Kd5CmxqxHGOeAEN2Ms3zgBCWm1blKzOKH eCW71hBhI1e5Ym8e6njcFnEcYVRAqK0/cCf2JPlDAWzRDrG8wJJE30Zc98QV4oM+9klx ZWOObc/Im9XBc+JE3KGK1h4cFBE1BRDkTyAIPeqW31dqUI8aPT+ezrnS67hz8TjDLrMy TusA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708042133; x=1708646933; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pGHwuS0ENdmLMLz6bWQqaQXZhO2uE/iwucpykSfpWsk=; b=eotaWDz4olDrUSS5CGE+hBNfBHzkXMccZKJ/PdltsRsXP3NjUNwdxdc6bBQ8D5ORRl yJINo0AFNJJCTroBW7bD++q6Hjec0DOEH3oQ0P5EDyt7o0xGmYFY8eFqccH4ArWgghO3 YubBOGzYw22SVaLiQgvZgY8I6I4Xo+KpE+hZUoVCdIuafxqM6Dld8SrPL5mfyi2x3THB VamMoIt2abJ59JLrLwzDQYRZGqSPa04FTqqv8pC9dEwy0o+RcAg1J1QZCUJ81C8mX2UA ZsgdJG5gnwe3MAaaUS9IzVvsZzWxw5KA41kwggX3fDqbT7sakWf8l8gT53E9l70Fhla+ iW9w== X-Forwarded-Encrypted: i=1; AJvYcCWLtXpPXx6ujNZwHWxcH0feZ7M+RJ6kWDBzA/mG/1b3tWzXzWkvwwsvFEzkMMUgQteijKvUOEdR4yIigkfG2Qf5c5cIQSLktLrJF1sY7BOiATlxkag= X-Gm-Message-State: AOJu0Yz2jrqgwph3NfYKpC2kqykBxwIfswh51+fp5FXMNImV0uvV08+b 9CmLdJoOgKv7LHjt7fGT1tW8t8AXXbTPtWLSv8OQrLHINhOT+rfHEgnRQ0K5SVc= X-Google-Smtp-Source: AGHT+IGthcOdIu3Gkfl9BOtBJNngVNNV+UiNIDvn/Iv90Cvzn3/WrJ1BIiVXy+QBXOWjOFCbm+O8LA== X-Received: by 2002:a17:902:784e:b0:1d9:ba26:effc with SMTP id e14-20020a170902784e00b001d9ba26effcmr3177476pln.51.1708042132885; Thu, 15 Feb 2024 16:08:52 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bb6-20020a170902bc8600b001db3d365082sm1789486plb.265.2024.02.15.16.08.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 16:08:51 -0800 (PST) From: Samuel Holland To: Will Deacon , Mark Rutland , Eric Lin , Conor Dooley Cc: Palmer Dabbelt , devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Walmsley , linux-riscv@lists.infradead.org, Rob Herring , Krzysztof Kozlowski , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [PATCH v1 5/6] dt-bindings: cache: Add SiFive Private L2 Cache controller Date: Thu, 15 Feb 2024 16:08:17 -0800 Message-ID: <20240216000837.1868917-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240216000837.1868917-1-samuel.holland@sifive.com> References: <20240216000837.1868917-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240215_160855_015178_8E7192B5 X-CRM114-Status: GOOD ( 17.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Eric Lin Add YAML DT binding documentation for the SiFive Private L2 Cache controller. Some functionality and the corresponding register bits were removed in the sifive,pl2cache1 version of the hardware, which creates the unusual situation where the newer hardware's compatible string is the fallback for the older one. Signed-off-by: Eric Lin Co-developed-by: Samuel Holland Signed-off-by: Samuel Holland --- Changes in v1: - Add back select: clause to binding - Make sifive,pl2cache1 the fallback for sifive,pl2cache0 - Fix the order of the reg property declaration - Document the sifive,perfmon-counters property .../bindings/cache/sifive,pl2cache0.yaml | 81 +++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 Documentation/devicetree/bindings/cache/sifive,pl2cache0.yaml diff --git a/Documentation/devicetree/bindings/cache/sifive,pl2cache0.yaml b/Documentation/devicetree/bindings/cache/sifive,pl2cache0.yaml new file mode 100644 index 000000000000..d89e2e5d0a97 --- /dev/null +++ b/Documentation/devicetree/bindings/cache/sifive,pl2cache0.yaml @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +# Copyright (C) 2023-2024 SiFive, Inc. +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/cache/sifive,pl2cache0.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: SiFive Private L2 Cache Controller + +maintainers: + - Eric Lin + +description: + The SiFive Private L2 Cache Controller is a per-core cache which communicates + with both the upstream L1 caches and downstream L3 cache or memory, enabling a + high-performance cache subsystem. + +allOf: + - $ref: /schemas/cache-controller.yaml# + +select: + properties: + compatible: + contains: + enum: + - sifive,pl2cache1 + + required: + - compatible + +properties: + compatible: + oneOf: + - items: + - const: sifive,pl2cache0 + - const: sifive,pl2cache1 + - const: cache + - items: + - const: sifive,pl2cache1 + - const: cache + + reg: + maxItems: 1 + + cache-block-size: true + cache-level: true + cache-sets: true + cache-size: true + cache-unified: true + + next-level-cache: true + + sifive,perfmon-counters: + $ref: /schemas/types.yaml#/definitions/uint32 + default: 0 + description: Number of PMU counter registers + +required: + - compatible + - reg + - cache-block-size + - cache-level + - cache-sets + - cache-size + - cache-unified + +additionalProperties: false + +examples: + - | + cache-controller@10104000 { + compatible = "sifive,pl2cache1", "cache"; + reg = <0x10104000 0x4000>; + cache-block-size = <64>; + cache-level = <2>; + cache-sets = <512>; + cache-size = <262144>; + cache-unified; + next-level-cache = <&L4>; + sifive,perfmon-counters = <6>; + }; From patchwork Fri Feb 16 00:08:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13559370 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6C815C4829E for ; Fri, 16 Feb 2024 01:17:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=olX1hPXm3ZMTs2aRzlwfgDqbk8epg0lKq3qdPav4lOo=; b=Ng+UySIGcz94py gWKPDSRHazSNelNH0FjrZudrUesirmlLampZaFHl4dwEG7jdtJUrIvrl1nLS/Zu94CEZc5s9G9VSl Us6lRXkxBJZkZPmpBO4FKqXUdrzBUMbOQkyx1y8uHMfpQLG8jOf51jqgq+fmX/n/1PwzM3IBKfy61 k4YLwUb3w+P+lvQS6ed3qCRbC97tl4rTt4NUFeyirWZQjH6PS0TYzEHhXf4gO7EBvnnewIjL900gv 5uRzyZlTb8II0eFyhnK5svDZ4IcDjg8zjvT40b2KCawbWxEvTQq7XKF6wEJ19bC2vYXbfOSonO4SL ANIDpRzLONc+TwtMZ+gg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ramr4-00000000iMm-1JcJ; Fri, 16 Feb 2024 01:17:30 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ralmk-00000000YgZ-1rnV for linux-arm-kernel@lists.infradead.org; Fri, 16 Feb 2024 00:09:07 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1d780a392fdso1302645ad.3 for ; Thu, 15 Feb 2024 16:08:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1708042136; x=1708646936; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p/acc0KNWqEiBrI2bUhG9VfOZxykMoq55GvaWo8V2Yc=; b=RcaKOhVF6TgtLD4+MGWOrh22bOwM8Zx3FADj3i5t5W2D1z42DVKsIxEitfzOJ56ZSj jHpyf2IAHaEVcbQ7YMx/IhBS0urLD0aOmOTrWz491Ny38N30wzGi9SzjiZX2DoiCYTFY lbcPwwUcAAt+cr9D8RWsAISeZB7mx9FJh/A9CJufzBl2j5GTRWYafYPb4nJPX1Fz6+5v xMBQfQ2CZeQ5D0M5Vnn5lJ3PIOCSHZF1o6Yd25WktScaPrAv9Ad4yuLblyLBgIEyL5ot 8QeTZxAMg7fkiAYZ3F0IK55/XVvsoAUNy7bg8ioFTk+lswPhtB6kQELf29pQa22zdVgV u+RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708042136; x=1708646936; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p/acc0KNWqEiBrI2bUhG9VfOZxykMoq55GvaWo8V2Yc=; b=TDaGBmUYolC+hlSuGueIseJGPecXp2QHbVMnoBeJ0XT9gyV2esE/DlxVKzlTCH//u8 miXeZMTbOWFPMhJstT/4mOlnc/UcY6GOIyhsjstFs6vZeHhjZpmQSOX3ojKidUqeup4M bQlhmYewFTo/uelgTBNKDgnZ/3L1JdYYNXte8usrNUAVzBE2HsJaZb4uF2H4Uxmcy/P8 YkNLw0jUnpVS5KNxFHnhUUQF0Z0oI8wNJZUDLd9vPmFyOJ6Ffuwt4YSiOUunIhYM+M9B shdWfZQZbwSHhZbwwv9p2q4bl3sS7BxGow51hSkR6U037Q4vISokNBNv1svyN5rttEhn M18g== X-Forwarded-Encrypted: i=1; AJvYcCWnLTWoy/F5Z3gB/pZleBpvZ3UMgob7m1BirEbIU7OVlF0d6zr02xb/N7Rsc7n2DzD5Gk0lMju8fp/n5KbPCrRWA9ryTTJrSdbwXKzhRaRrLOOznR0= X-Gm-Message-State: AOJu0YzBXhVpbeFDe/Dq9Z2SLUzHKJzQucmAwOOZQ4VPAt8f87b84VHq wSwoTFcn56Ie/tUcrtyA1mx9tJaWHKW9slFUgttmXnM5i2wyXYxZFHxLM+j5Szs= X-Google-Smtp-Source: AGHT+IHJy/A1S+sldWkfbkh3Ezw6XCoeo322r0XKeVNd/Im9EAWCGT0knZuiKSibPYqHNzxRxpUokQ== X-Received: by 2002:a17:902:ec89:b0:1db:aa46:4923 with SMTP id x9-20020a170902ec8900b001dbaa464923mr46470plg.40.1708042136146; Thu, 15 Feb 2024 16:08:56 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id bb6-20020a170902bc8600b001db3d365082sm1789486plb.265.2024.02.15.16.08.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 16:08:54 -0800 (PST) From: Samuel Holland To: Will Deacon , Mark Rutland , Eric Lin , Conor Dooley Cc: Palmer Dabbelt , devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Paul Walmsley , linux-riscv@lists.infradead.org, Rob Herring , Krzysztof Kozlowski , linux-arm-kernel@lists.infradead.org, Greentime Hu , Samuel Holland Subject: [PATCH v1 6/6] drivers/perf: Add SiFive Private L2 Cache PMU driver Date: Thu, 15 Feb 2024 16:08:18 -0800 Message-ID: <20240216000837.1868917-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240216000837.1868917-1-samuel.holland@sifive.com> References: <20240216000837.1868917-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240215_160859_063294_50927DCD X-CRM114-Status: GOOD ( 25.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Greentime Hu Add a driver for the PMU found in the SiFive Private L2 Cache controller. This PMU provides a configurable number of counters and a variety of events. Events are grouped into sets. Each counter can count events from only one set at a time; however, it can count any number of events within that set simultaneously. The PMU hardware does not provide an overflow interrupt or a way to atomically control groups of counters. A separate Private L2 Cache instance exists for each core, so this driver supports per-core and per-task profiling. Some events can be filtered further by client ID (e.g. CPU or external DMA master). That functionality is not supported by this driver. Example usage: $ perf stat -e sifive_pl2_pmu/inner_get/,sifive_pl2_pmu/outer_get/ ls Performance counter stats for 'ls': 95041 sifive_pl2_pmu/inner_get/ 3 sifive_pl2_pmu/outer_get/ 0.003971538 seconds time elapsed 0.000000000 seconds user 0.006315000 seconds sys Example combining multiple events together within each counter: $ perf stat -e sifive_pl2_pmu/event=0x301/, # inner_put_*_data sifive_pl2_pmu/event=0x303/ ls # outer_put_*_data Performance counter stats for 'ls': 6828 sifive_pl2_pmu/event=0x301/ 11 sifive_pl2_pmu/event=0x303/ 0.005696538 seconds time elapsed 0.000000000 seconds user 0.006337000 seconds sys Signed-off-by: Greentime Hu Co-developed-by: Eric Lin Signed-off-by: Eric Lin Co-developed-by: Samuel Holland Signed-off-by: Samuel Holland --- Changes in v1: - Add missing events to PL2 sets 2, 4, and 5 - Use event_base and config_base to precompute register addresses - Check event validity earlier, in the .event_init hook - Implement .filter for systems where only some CPUs have a PL2 - Only allocate percpu data when probing each PL2 instance - Reference count the `struct pmu` to fix unbind/bind crashes - Probe via DT since the PMU driver is now the only PL2 driver - Allow the driver to be built as a module drivers/perf/Kconfig | 10 + drivers/perf/Makefile | 1 + drivers/perf/sifive_pl2_pmu.c | 748 ++++++++++++++++++++++++++++++++++ 3 files changed, 759 insertions(+) create mode 100644 drivers/perf/sifive_pl2_pmu.c diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 8a3b2b88d8b5..bd5ebed8630b 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -174,6 +174,16 @@ config SIFIVE_ECACHE_PMU for measuring whole-system L2/L3 cache performance using the perf events subsystem. +config SIFIVE_PL2_PMU + tristate "SiFive Private L2 Cache PMU" + depends on RISCV || COMPILE_TEST + depends on OF + help + Support for the Private L2 Cache performance monitoring unit (PMU) on + SiFive platforms. The Private L2 Cache PMU provides up to 64 counters + for measuring per-program or per-hart L2 cache performance using the + perf events subsystem. + config THUNDERX2_PMU tristate "Cavium ThunderX2 SoC PMU UNCORE" depends on ARCH_THUNDER2 || COMPILE_TEST diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index a51686b413f2..d5501196dcd8 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -17,6 +17,7 @@ obj-$(CONFIG_RISCV_PMU_LEGACY) += riscv_pmu_legacy.o obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o obj-$(CONFIG_SIFIVE_CCACHE_PMU) += sifive_ccache_pmu.o obj-$(CONFIG_SIFIVE_ECACHE_PMU) += sifive_ecache_pmu.o +obj-$(CONFIG_SIFIVE_PL2_PMU) += sifive_pl2_pmu.o obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o obj-$(CONFIG_ARM_SPE_PMU) += arm_spe_pmu.o diff --git a/drivers/perf/sifive_pl2_pmu.c b/drivers/perf/sifive_pl2_pmu.c new file mode 100644 index 000000000000..d0bbac0dec06 --- /dev/null +++ b/drivers/perf/sifive_pl2_pmu.c @@ -0,0 +1,748 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * SiFive Private L2 Cache PMU driver + * + * Copyright (C) 2018-2024 SiFive, Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define PL2_SELECT_OFFSET 0x2000 +#define PL2_CLIENT_FILTER_OFFSET 0x2800 +#define PL2_COUNTER_OFFSET 0x3000 + +#define PL2_PMU_MAX_COUNTERS 64 + +struct sifive_pl2_pmu_event { + void __iomem *base; + DECLARE_BITMAP(used_mask, PL2_PMU_MAX_COUNTERS); + unsigned int cpu; + int n_counters; + struct perf_event *events[] __counted_by(n_counters); +}; + +struct sifive_pl2_pmu { + struct pmu pmu; + struct notifier_block cpu_pm_nb; + refcount_t refcount; + struct sifive_pl2_pmu_event *__percpu *event; +}; + +#define to_pl2_pmu(p) (container_of(p, struct sifive_pl2_pmu, pmu)) + +static DEFINE_MUTEX(g_mutex); +static struct sifive_pl2_pmu *g_pl2_pmu; + +#ifndef readq +static inline u64 readq(void __iomem *addr) +{ + return readl(addr) | (((u64)readl(addr + 4)) << 32); +} +#endif + +#ifndef writeq +static inline void writeq(u64 v, void __iomem *addr) +{ + writel(lower_32_bits(v), addr); + writel(upper_32_bits(v), addr + 4); +} +#endif + +/* + * sysfs attributes + * + * We export: + * - events, used by perf user space and other tools to create events symbolically, e.g.: + * perf stat -a -e sifive_pl2_pmu/event=inner_put_partial_data_hit/ ls + * perf stat -a -e sifive_pl2_pmu/event=0x101/ ls + * - formats, used by perf user space and other tools to configure events + */ + +/* events */ +static ssize_t sifive_pl2_pmu_event_show(struct device *dev, struct device_attribute *attr, + char *page) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr); + return sysfs_emit(page, "event=0x%02llx\n", pmu_attr->id); +} + +#define SET_EVENT_SELECT(_event, _set) (BIT_ULL((_event) + 8) | (_set)) +#define PL2_PMU_EVENT_ATTR(_name, _event, _set) \ + PMU_EVENT_ATTR_ID(_name, sifive_pl2_pmu_event_show, SET_EVENT_SELECT(_event, _set)) + +enum pl2_pmu_event_set1 { + INNER_PUT_FULL_DATA = 0, + INNER_PUT_PARTIAL_DATA, + INNER_ATOMIC_DATA, + INNER_GET, + INNER_PREFETCH_READ, + INNER_PREFETCH_WRITE, + INNER_ACQUIRE_BLOCK_NTOB, + INNER_ACQUIRE_BLOCK_NTOT, + INNER_ACQUIRE_BLOCK_BTOT, + INNER_ACQUIRE_PERM_NTOT, + INNER_ACQUIRE_PERM_BTOT, + INNER_RELEASE_TTOB, + INNER_RELEASE_TTON, + INNER_RELEASE_BTON, + INNER_RELEASE_DATA_TTOB, + INNER_RELEASE_DATA_TTON, + INNER_RELEASE_DATA_BTON, + INNER_RELEASE_DATA_TTOT, + INNER_PROBE_BLOCK_TOT, + INNER_PROBE_BLOCK_TOB, + INNER_PROBE_BLOCK_TON, + INNER_PROBE_PERM_TON, + INNER_PROBE_ACK_TTOB, + INNER_PROBE_ACK_TTON, + INNER_PROBE_ACK_BTON, + INNER_PROBE_ACK_TTOT, + INNER_PROBE_ACK_BTOB, + INNER_PROBE_ACK_NTON, + INNER_PROBE_ACK_DATA_TTOB, + INNER_PROBE_ACK_DATA_TTON, + INNER_PROBE_ACK_DATA_TTOT, + PL2_PMU_MAX_EVENT1_IDX +}; + +enum pl2_pmu_event_set2 { + INNER_PUT_FULL_DATA_HIT = 0, + INNER_PUT_PARTIAL_DATA_HIT, + INNER_ATOMIC_DATA_HIT, + INNER_GET_HIT, + INNER_PREFETCH_READ_HIT, + INNER_ACQUIRE_BLOCK_NTOB_HIT, + INNER_ACQUIRE_PERM_NTOT_HIT, + INNER_RELEASE_TTOB_HIT, + INNER_RELEASE_DATA_TTOB_HIT, + OUTER_PROBE_BLOCK_TOT_HIT, + INNER_PUT_FULL_DATA_HIT_SHARED, + INNER_PUT_PARTIAL_DATA_HIT_SHARED, + INNER_ATOMIC_DATA_HIT_SHARED, + INNER_GET_HIT_SHARED, + INNER_PREFETCH_READ_HIT_SHARED, + INNER_ACQUIRE_BLOCK_NTOB_HIT_SHARED, + INNER_ACQUIRE_PERM_NTOT_HIT_SHARED, + INNER_RELEASE_TTOB_HIT_SHARED, + INNER_RELEASE_DATA_TTOB_HIT_SHARED, + OUTER_PROBE_BLOCK_TOT_HIT_SHARED, + OUTER_PROBE_BLOCK_TOT_HIT_DIRTY, + PL2_PMU_MAX_EVENT2_IDX +}; + +enum pl2_pmu_event_set3 { + OUTER_PUT_FULL_DATA = 0, + OUTER_PUT_PARTIAL_DATA, + OUTER_ATOMIC_DATA, + OUTER_GET, + OUTER_PREFETCH_READ, + OUTER_PREFETCH_WRITE, + OUTER_ACQUIRE_BLOCK_NTOB, + OUTER_ACQUIRE_BLOCK_NTOT, + OUTER_ACQUIRE_BLOCK_BTOT, + OUTER_ACQUIRE_PERM_NTOT, + OUTER_ACQUIRE_PERM_BTOT, + OUTER_RELEARE_TTOB, + OUTER_RELEARE_TTON, + OUTER_RELEARE_BTON, + OUTER_RELEARE_DATA_TTOB, + OUTER_RELEARE_DATA_TTON, + OUTER_RELEARE_DATA_BTON, + OUTER_RELEARE_DATA_TTOT, + OUTER_PROBE_BLOCK_TOT, + OUTER_PROBE_BLOCK_TOB, + OUTER_PROBE_BLOCK_TON, + OUTER_PROBE_PERM_TON, + OUTER_PROBE_ACK_TTOB, + OUTER_PROBE_ACK_TTON, + OUTER_PROBE_ACK_BTON, + OUTER_PROBE_ACK_TTOT, + OUTER_PROBE_ACK_BTOB, + OUTER_PROBE_ACK_NTON, + OUTER_PROBE_ACK_DATA_TTOB, + OUTER_PROBE_ACK_DATA_TTON, + OUTER_PROBE_ACK_DATA_TTOT, + PL2_PMU_MAX_EVENT3_IDX +}; + +enum pl2_pmu_event_set4 { + INNER_HINT_HITS_MSHR = 0, + INNER_READ_HITS_MSHR, + INNER_WRITE_HITS_MSHR, + INNER_READ_REPLAY, + INNER_WRITE_REPLAY, + OUTER_PROBE_REPLAY, + REPLAY, + SLEEP_BY_MISS_QUEUE, + SLEEP_BY_EVICT_QUEUE, + SLEEP_FOR_BACK_PROBE, + SLEEP, + PL2_PMU_MAX_EVENT4_IDX +}; + +enum pl2_pmu_event_set5 { + READ_SLEEP_TIMER_EXPIRE = 0, + READ_OLDEST_TIMER_EXPIRE, + WRITE_SLEEP_TIMER_EXPIRE, + WRITE_OLDEST_TIMER_EXPIRE, + READ_SLEEP, + READ_DIR_UPDATE_WAKEUP, + READ_MISS_QUEUE_WAKEUP, + READ_EVICT_QUEUE_WAKEUP, + READ_SLEEP_TIMER_WAKEUP, + WRITE_SLEEP, + WRITE_DIR_UPDATE_WAKEUP, + WRITE_MISS_QUEUE_WAKEUP, + WRITE_EVICT_QUEUE_WAKEUP, + WRITE_SLEEP_TIMER_WAKEUP, + PL2_PMU_MAX_EVENT5_IDX +}; + +static struct attribute *sifive_pl2_pmu_events[] = { + PL2_PMU_EVENT_ATTR(inner_put_full_data, INNER_PUT_FULL_DATA, 1), + PL2_PMU_EVENT_ATTR(inner_put_partial_data, INNER_PUT_PARTIAL_DATA, 1), + PL2_PMU_EVENT_ATTR(inner_atomic_data, INNER_ATOMIC_DATA, 1), + PL2_PMU_EVENT_ATTR(inner_get, INNER_GET, 1), + PL2_PMU_EVENT_ATTR(inner_prefetch_read, INNER_PREFETCH_READ, 1), + PL2_PMU_EVENT_ATTR(inner_prefetch_write, INNER_PREFETCH_WRITE, 1), + PL2_PMU_EVENT_ATTR(inner_acquire_block_ntob, INNER_ACQUIRE_BLOCK_NTOB, 1), + PL2_PMU_EVENT_ATTR(inner_acquire_block_ntot, INNER_ACQUIRE_BLOCK_NTOT, 1), + PL2_PMU_EVENT_ATTR(inner_acquire_block_btot, INNER_ACQUIRE_BLOCK_BTOT, 1), + PL2_PMU_EVENT_ATTR(inner_acquire_perm_ntot, INNER_ACQUIRE_PERM_NTOT, 1), + PL2_PMU_EVENT_ATTR(inner_acquire_perm_btot, INNER_ACQUIRE_PERM_BTOT, 1), + PL2_PMU_EVENT_ATTR(inner_release_ttob, INNER_RELEASE_TTOB, 1), + PL2_PMU_EVENT_ATTR(inner_release_tton, INNER_RELEASE_TTON, 1), + PL2_PMU_EVENT_ATTR(inner_release_bton, INNER_RELEASE_BTON, 1), + PL2_PMU_EVENT_ATTR(inner_release_data_ttob, INNER_RELEASE_DATA_TTOB, 1), + PL2_PMU_EVENT_ATTR(inner_release_data_tton, INNER_RELEASE_DATA_TTON, 1), + PL2_PMU_EVENT_ATTR(inner_release_data_bton, INNER_RELEASE_DATA_BTON, 1), + PL2_PMU_EVENT_ATTR(inner_release_data_ttot, INNER_RELEASE_DATA_TTOT, 1), + PL2_PMU_EVENT_ATTR(inner_probe_block_tot, INNER_PROBE_BLOCK_TOT, 1), + PL2_PMU_EVENT_ATTR(inner_probe_block_tob, INNER_PROBE_BLOCK_TOB, 1), + PL2_PMU_EVENT_ATTR(inner_probe_block_ton, INNER_PROBE_BLOCK_TON, 1), + PL2_PMU_EVENT_ATTR(inner_probe_perm_ton, INNER_PROBE_PERM_TON, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_ttob, INNER_PROBE_ACK_TTOB, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_tton, INNER_PROBE_ACK_TTON, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_bton, INNER_PROBE_ACK_BTON, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_ttot, INNER_PROBE_ACK_TTOT, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_btob, INNER_PROBE_ACK_BTOB, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_nton, INNER_PROBE_ACK_NTON, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_data_ttob, INNER_PROBE_ACK_DATA_TTOB, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_data_tton, INNER_PROBE_ACK_DATA_TTON, 1), + PL2_PMU_EVENT_ATTR(inner_probe_ack_data_ttot, INNER_PROBE_ACK_DATA_TTOT, 1), + + PL2_PMU_EVENT_ATTR(inner_put_full_data_hit, INNER_PUT_FULL_DATA_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_put_partial_data_hit, INNER_PUT_PARTIAL_DATA_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_atomic_data_hit, INNER_ATOMIC_DATA_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_get_hit, INNER_GET_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_prefetch_read_hit, INNER_PREFETCH_READ_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_acquire_block_ntob_hit, INNER_ACQUIRE_BLOCK_NTOB_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_acquire_perm_ntot_hit, INNER_ACQUIRE_PERM_NTOT_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_release_ttob_hit, INNER_RELEASE_TTOB_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_release_data_ttob_hit, INNER_RELEASE_DATA_TTOB_HIT, 2), + PL2_PMU_EVENT_ATTR(outer_probe_block_tot_hit, OUTER_PROBE_BLOCK_TOT_HIT, 2), + PL2_PMU_EVENT_ATTR(inner_put_full_data_hit_shared, INNER_PUT_FULL_DATA_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(inner_put_partial_data_hit_shared, INNER_PUT_PARTIAL_DATA_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(inner_atomic_data_hit_shared, INNER_ATOMIC_DATA_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(inner_get_hit_shared, INNER_GET_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(inner_prefetch_read_hit_shared, INNER_PREFETCH_READ_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(inner_acquire_block_ntob_hit_shared, + INNER_ACQUIRE_BLOCK_NTOB_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(inner_acquire_perm_ntot_hit_shared, + INNER_ACQUIRE_PERM_NTOT_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(inner_release_ttob_hit_shared, INNER_RELEASE_TTOB_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(inner_release_data_ttob_hit_shared, + INNER_RELEASE_DATA_TTOB_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(outer_probe_block_tot_hit_shared, OUTER_PROBE_BLOCK_TOT_HIT_SHARED, 2), + PL2_PMU_EVENT_ATTR(outer_probe_block_tot_hit_dirty, OUTER_PROBE_BLOCK_TOT_HIT_DIRTY, 2), + + PL2_PMU_EVENT_ATTR(outer_put_full_data, OUTER_PUT_FULL_DATA, 3), + PL2_PMU_EVENT_ATTR(outer_put_partial_data, OUTER_PUT_PARTIAL_DATA, 3), + PL2_PMU_EVENT_ATTR(outer_atomic_data, OUTER_ATOMIC_DATA, 3), + PL2_PMU_EVENT_ATTR(outer_get, OUTER_GET, 3), + PL2_PMU_EVENT_ATTR(outer_prefetch_read, OUTER_PREFETCH_READ, 3), + PL2_PMU_EVENT_ATTR(outer_prefetch_write, OUTER_PREFETCH_WRITE, 3), + PL2_PMU_EVENT_ATTR(outer_acquire_block_ntob, OUTER_ACQUIRE_BLOCK_NTOB, 3), + PL2_PMU_EVENT_ATTR(outer_acquire_block_ntot, OUTER_ACQUIRE_BLOCK_NTOT, 3), + PL2_PMU_EVENT_ATTR(outer_acquire_block_btot, OUTER_ACQUIRE_BLOCK_BTOT, 3), + PL2_PMU_EVENT_ATTR(outer_acquire_perm_ntot, OUTER_ACQUIRE_PERM_NTOT, 3), + PL2_PMU_EVENT_ATTR(outer_acquire_perm_btot, OUTER_ACQUIRE_PERM_BTOT, 3), + PL2_PMU_EVENT_ATTR(outer_release_ttob, OUTER_RELEARE_TTOB, 3), + PL2_PMU_EVENT_ATTR(outer_release_tton, OUTER_RELEARE_TTON, 3), + PL2_PMU_EVENT_ATTR(outer_release_bton, OUTER_RELEARE_BTON, 3), + PL2_PMU_EVENT_ATTR(outer_release_data_ttob, OUTER_RELEARE_DATA_TTOB, 3), + PL2_PMU_EVENT_ATTR(outer_release_data_tton, OUTER_RELEARE_DATA_TTON, 3), + PL2_PMU_EVENT_ATTR(outer_release_data_bton, OUTER_RELEARE_DATA_BTON, 3), + PL2_PMU_EVENT_ATTR(outer_release_data_ttot, OUTER_RELEARE_DATA_TTOT, 3), + PL2_PMU_EVENT_ATTR(outer_probe_block_tot, OUTER_PROBE_BLOCK_TOT, 3), + PL2_PMU_EVENT_ATTR(outer_probe_block_tob, OUTER_PROBE_BLOCK_TOB, 3), + PL2_PMU_EVENT_ATTR(outer_probe_block_ton, OUTER_PROBE_BLOCK_TON, 3), + PL2_PMU_EVENT_ATTR(outer_probe_perm_ton, OUTER_PROBE_PERM_TON, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_ttob, OUTER_PROBE_ACK_TTOB, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_tton, OUTER_PROBE_ACK_TTON, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_bton, OUTER_PROBE_ACK_BTON, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_ttot, OUTER_PROBE_ACK_TTOT, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_btob, OUTER_PROBE_ACK_BTOB, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_nton, OUTER_PROBE_ACK_NTON, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_data_ttob, OUTER_PROBE_ACK_DATA_TTOB, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_data_tton, OUTER_PROBE_ACK_DATA_TTON, 3), + PL2_PMU_EVENT_ATTR(outer_probe_ack_data_ttot, OUTER_PROBE_ACK_DATA_TTOT, 3), + + PL2_PMU_EVENT_ATTR(inner_hint_hits_mshr, INNER_HINT_HITS_MSHR, 4), + PL2_PMU_EVENT_ATTR(inner_read_hits_mshr, INNER_READ_HITS_MSHR, 4), + PL2_PMU_EVENT_ATTR(inner_write_hits_mshr, INNER_WRITE_HITS_MSHR, 4), + PL2_PMU_EVENT_ATTR(inner_read_replay, INNER_READ_REPLAY, 4), + PL2_PMU_EVENT_ATTR(inner_write_replay, INNER_WRITE_REPLAY, 4), + PL2_PMU_EVENT_ATTR(outer_probe_replay, OUTER_PROBE_REPLAY, 4), + PL2_PMU_EVENT_ATTR(replay, REPLAY, 4), + PL2_PMU_EVENT_ATTR(sleep_by_miss_queue, SLEEP_BY_MISS_QUEUE, 4), + PL2_PMU_EVENT_ATTR(sleep_by_evict_queue, SLEEP_BY_EVICT_QUEUE, 4), + PL2_PMU_EVENT_ATTR(sleep_for_back_probe, SLEEP_FOR_BACK_PROBE, 4), + PL2_PMU_EVENT_ATTR(sleep, SLEEP, 4), + + PL2_PMU_EVENT_ATTR(read_sleep_timer_expire, READ_SLEEP_TIMER_EXPIRE, 5), + PL2_PMU_EVENT_ATTR(read_oldest_timer_expire, READ_OLDEST_TIMER_EXPIRE, 5), + PL2_PMU_EVENT_ATTR(write_sleep_timer_expire, WRITE_SLEEP_TIMER_EXPIRE, 5), + PL2_PMU_EVENT_ATTR(write_oldest_timer_expire, WRITE_OLDEST_TIMER_EXPIRE, 5), + PL2_PMU_EVENT_ATTR(read_sleep, READ_SLEEP, 5), + PL2_PMU_EVENT_ATTR(read_dir_update_wakeup, READ_DIR_UPDATE_WAKEUP, 5), + PL2_PMU_EVENT_ATTR(read_miss_queue_wakeup, READ_MISS_QUEUE_WAKEUP, 5), + PL2_PMU_EVENT_ATTR(read_evict_queue_wakeup, READ_EVICT_QUEUE_WAKEUP, 5), + PL2_PMU_EVENT_ATTR(read_sleep_timer_wakeup, READ_SLEEP_TIMER_WAKEUP, 5), + PL2_PMU_EVENT_ATTR(write_sleep, WRITE_SLEEP, 5), + PL2_PMU_EVENT_ATTR(write_dir_update_wakeup, WRITE_DIR_UPDATE_WAKEUP, 5), + PL2_PMU_EVENT_ATTR(write_miss_queue_wakeup, WRITE_MISS_QUEUE_WAKEUP, 5), + PL2_PMU_EVENT_ATTR(write_evict_queue_wakeup, WRITE_EVICT_QUEUE_WAKEUP, 5), + PL2_PMU_EVENT_ATTR(write_sleep_timer_wakeup, WRITE_SLEEP_TIMER_WAKEUP, 5), + NULL +}; + +static struct attribute_group sifive_pl2_pmu_events_group = { + .name = "events", + .attrs = sifive_pl2_pmu_events, +}; + +/* formats */ +PMU_FORMAT_ATTR(event, "config:0-63"); + +static struct attribute *sifive_pl2_pmu_formats[] = { + &format_attr_event.attr, + NULL, +}; + +static struct attribute_group sifive_pl2_pmu_format_group = { + .name = "format", + .attrs = sifive_pl2_pmu_formats, +}; + +/* + * Per PMU device attribute groups + */ + +static const struct attribute_group *sifive_pl2_pmu_attr_grps[] = { + &sifive_pl2_pmu_events_group, + &sifive_pl2_pmu_format_group, + NULL, +}; + +/* + * Event Initialization + */ + +static int sifive_pl2_pmu_event_init(struct perf_event *event) +{ + struct hw_perf_event *hwc = &event->hw; + u64 config = event->attr.config; + u64 ev_type = config >> 8; + u64 set = config & 0xff; + + /* Check if this is a valid set and event */ + switch (set) { + case 1: + if (ev_type >= BIT_ULL(PL2_PMU_MAX_EVENT1_IDX)) + return -ENOENT; + break; + case 2: + if (ev_type >= BIT_ULL(PL2_PMU_MAX_EVENT2_IDX)) + return -ENOENT; + break; + case 3: + if (ev_type >= BIT_ULL(PL2_PMU_MAX_EVENT3_IDX)) + return -ENOENT; + break; + case 4: + if (ev_type >= BIT_ULL(PL2_PMU_MAX_EVENT4_IDX)) + return -ENOENT; + break; + case 5: + if (ev_type >= BIT_ULL(PL2_PMU_MAX_EVENT5_IDX)) + return -ENOENT; + break; + default: + return -ENOENT; + } + + /* Do not allocate the hardware counter yet */ + hwc->idx = -1; + hwc->config = config; + + return 0; +} + +/* + * pmu->read: read and update the counter + */ +static void sifive_pl2_pmu_read(struct perf_event *event) +{ + struct hw_perf_event *hwc = &event->hw; + u64 prev_raw_count, new_raw_count; + u64 oldval; + + do { + prev_raw_count = local64_read(&hwc->prev_count); + new_raw_count = readq((void *)hwc->event_base); + + oldval = local64_cmpxchg(&hwc->prev_count, prev_raw_count, new_raw_count); + } while (oldval != prev_raw_count); + + local64_add(new_raw_count - prev_raw_count, &event->count); +} + +/* + * State transition functions: + * + * start()/stop() & add()/del() + */ + +/* + * pmu->start: start the event + */ +static void sifive_pl2_pmu_start(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc = &event->hw; + + if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED))) + return; + + hwc->state = 0; + + /* Set initial value to 0 */ + local64_set(&hwc->prev_count, 0); + writeq(0, (void *)hwc->event_base); + + /* Enable this counter to count events */ + writeq(hwc->config, (void *)hwc->config_base); +} + +/* + * pmu->stop: stop the counter + */ +static void sifive_pl2_pmu_stop(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc = &event->hw; + + if (hwc->state & PERF_HES_STOPPED) + return; + + /* Disable this counter to count events */ + writeq(0, (void *)hwc->config_base); + sifive_pl2_pmu_read(event); + + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; +} + +/* + * pmu->add: add the event to the PMU + */ +static int sifive_pl2_pmu_add(struct perf_event *event, int flags) +{ + struct sifive_pl2_pmu *pl2_pmu = to_pl2_pmu(event->pmu); + struct sifive_pl2_pmu_event *ptr = *this_cpu_ptr(pl2_pmu->event); + struct hw_perf_event *hwc = &event->hw; + int idx; + + /* Find an available counter idx to use for this event */ + do { + idx = find_first_zero_bit(ptr->used_mask, ptr->n_counters); + if (idx >= ptr->n_counters) + return -EAGAIN; + } while (test_and_set_bit(idx, ptr->used_mask)); + + hwc->config_base = (unsigned long)ptr->base + PL2_SELECT_OFFSET + 8 * idx; + hwc->event_base = (unsigned long)ptr->base + PL2_COUNTER_OFFSET + 8 * idx; + hwc->idx = idx; + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; + + ptr->events[idx] = event; + + if (flags & PERF_EF_START) + sifive_pl2_pmu_start(event, PERF_EF_RELOAD); + + perf_event_update_userpage(event); + + return 0; +} + +/* + * pmu->del: delete the event from the PMU + */ +static void sifive_pl2_pmu_del(struct perf_event *event, int flags) +{ + struct sifive_pl2_pmu *pl2_pmu = to_pl2_pmu(event->pmu); + struct sifive_pl2_pmu_event *ptr = *this_cpu_ptr(pl2_pmu->event); + struct hw_perf_event *hwc = &event->hw; + int idx = hwc->idx; + + /* Stop and release this counter */ + sifive_pl2_pmu_stop(event, PERF_EF_UPDATE); + + ptr->events[idx] = NULL; + clear_bit(idx, ptr->used_mask); + + perf_event_update_userpage(event); +} + +/* + * pmu->filter: check if the PMU can be used with a CPU + */ +static bool sifive_pl2_pmu_filter(struct pmu *pmu, int cpu) +{ + struct sifive_pl2_pmu *pl2_pmu = to_pl2_pmu(pmu); + struct sifive_pl2_pmu_event *ptr = *this_cpu_ptr(pl2_pmu->event); + + /* Filter out CPUs with no PL2 instance (no percpu data allocated) */ + return !ptr; +} + +/* + * Driver initialization + */ + +static void sifive_pl2_pmu_hw_init(const struct sifive_pl2_pmu_event *ptr) +{ + /* Disable the client filter (not supported by this driver) */ + writeq(0, ptr->base + PL2_CLIENT_FILTER_OFFSET); +} + +static int sifive_pl2_pmu_pm_notify(struct notifier_block *nb, unsigned long cmd, void *v) +{ + struct sifive_pl2_pmu *pl2_pmu = container_of(nb, struct sifive_pl2_pmu, cpu_pm_nb); + struct sifive_pl2_pmu_event *ptr = *this_cpu_ptr(pl2_pmu->event); + struct perf_event *event; + + if (!ptr || bitmap_empty(ptr->used_mask, PL2_PMU_MAX_COUNTERS)) + return NOTIFY_OK; + + for (int idx = 0; idx < ptr->n_counters; idx++) { + event = ptr->events[idx]; + if (!event) + continue; + + switch (cmd) { + case CPU_PM_ENTER: + /* Stop and update the counter */ + sifive_pl2_pmu_stop(event, PERF_EF_UPDATE); + break; + case CPU_PM_ENTER_FAILED: + case CPU_PM_EXIT: + /* Restore and enable the counter */ + sifive_pl2_pmu_start(event, PERF_EF_RELOAD); + break; + default: + break; + } + } + + return NOTIFY_OK; +} + +static int sifive_pl2_pmu_pm_register(struct sifive_pl2_pmu *pl2_pmu) +{ + if (!IS_ENABLED(CONFIG_CPU_PM)) + return 0; + + pl2_pmu->cpu_pm_nb.notifier_call = sifive_pl2_pmu_pm_notify; + return cpu_pm_register_notifier(&pl2_pmu->cpu_pm_nb); +} + +static void sifive_pl2_pmu_pm_unregister(struct sifive_pl2_pmu *pl2_pmu) +{ + if (!IS_ENABLED(CONFIG_CPU_PM)) + return; + + cpu_pm_unregister_notifier(&pl2_pmu->cpu_pm_nb); +} + +static struct sifive_pl2_pmu *sifive_pl2_pmu_get(void) +{ + struct sifive_pl2_pmu *pl2_pmu; + int ret; + + guard(mutex)(&g_mutex); + + pl2_pmu = g_pl2_pmu; + if (pl2_pmu) { + refcount_inc(&pl2_pmu->refcount); + return pl2_pmu; + } + + pl2_pmu = kzalloc(sizeof(*pl2_pmu), GFP_KERNEL); + if (!pl2_pmu) + return ERR_PTR(-ENOMEM); + + pl2_pmu->pmu = (struct pmu) { + .attr_groups = sifive_pl2_pmu_attr_grps, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE | PERF_PMU_CAP_NO_INTERRUPT, + .task_ctx_nr = perf_sw_context, + .event_init = sifive_pl2_pmu_event_init, + .add = sifive_pl2_pmu_add, + .del = sifive_pl2_pmu_del, + .start = sifive_pl2_pmu_start, + .stop = sifive_pl2_pmu_stop, + .read = sifive_pl2_pmu_read, + .filter = sifive_pl2_pmu_filter, + }; + + refcount_set(&pl2_pmu->refcount, 1); + + pl2_pmu->event = alloc_percpu(typeof(*pl2_pmu->event)); + if (!pl2_pmu->event) { + ret = -ENOMEM; + goto err_free; + } + + ret = sifive_pl2_pmu_pm_register(pl2_pmu); + if (ret) + goto err_free_percpu; + + ret = perf_pmu_register(&pl2_pmu->pmu, "sifive_pl2_pmu", -1); + if (ret) { + pr_err("%s: Failed to register PMU: %d\n", __func__, ret); + goto err_unregister_pm; + } + + g_pl2_pmu = pl2_pmu; + + return pl2_pmu; + +err_unregister_pm: + sifive_pl2_pmu_pm_unregister(pl2_pmu); +err_free_percpu: + free_percpu(pl2_pmu->event); +err_free: + kfree(pl2_pmu); + + return ERR_PTR(ret); +} + +static void sifive_pl2_pmu_put(void) +{ + struct sifive_pl2_pmu *pl2_pmu; + + guard(mutex)(&g_mutex); + + pl2_pmu = g_pl2_pmu; + if (!refcount_dec_and_test(&pl2_pmu->refcount)) + return; + + g_pl2_pmu = NULL; + perf_pmu_unregister(&pl2_pmu->pmu); + sifive_pl2_pmu_pm_unregister(pl2_pmu); + free_percpu(pl2_pmu->event); + kfree(pl2_pmu); +} + +static int sifive_pl2_pmu_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct device_node *np = dev_of_node(dev); + struct sifive_pl2_pmu_event *ptr; + struct sifive_pl2_pmu *pl2_pmu; + unsigned int cpu; + u32 n_counters; + int ret; + + /* Instances without a sifive,perfmon-counters property do not contain a PMU */ + ret = of_property_read_u32(np, "sifive,perfmon-counters", &n_counters); + if (ret || !n_counters) + return -ENODEV; + + /* Determine the CPU affinity of this PL2 instance */ + for_each_possible_cpu(cpu) { + struct device_node *cache_node, *cpu_node; + + cpu_node = of_cpu_device_node_get(cpu); + if (!cpu_node) + continue; + + cache_node = of_parse_phandle(cpu_node, "next-level-cache", 0); + of_node_put(cpu_node); + if (!cache_node) + continue; + + of_node_put(cache_node); + if (cache_node == np) + break; + } + if (cpu >= nr_cpu_ids) + return -ENODEV; + + ptr = devm_kzalloc(dev, struct_size(ptr, events, n_counters), GFP_KERNEL); + if (!ptr) + return -ENOMEM; + + platform_set_drvdata(pdev, ptr); + + ptr->cpu = cpu; + ptr->n_counters = n_counters; + + ptr->base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(ptr->base)) + return PTR_ERR(ptr->base); + + sifive_pl2_pmu_hw_init(ptr); + + pl2_pmu = sifive_pl2_pmu_get(); + if (IS_ERR(pl2_pmu)) + return PTR_ERR(pl2_pmu); + + *per_cpu_ptr(pl2_pmu->event, cpu) = ptr; + + return 0; +} + +static void sifive_pl2_pmu_remove(struct platform_device *pdev) +{ + struct sifive_pl2_pmu_event *ptr = platform_get_drvdata(pdev); + + *per_cpu_ptr(g_pl2_pmu->event, ptr->cpu) = NULL; + sifive_pl2_pmu_put(); +} + +static const struct of_device_id sifve_pl2_pmu_of_match[] = { + { .compatible = "sifive,pl2cache1" }, + {} +}; +MODULE_DEVICE_TABLE(of, sifve_pl2_pmu_of_match); + +static struct platform_driver sifive_pl2_pmu_driver = { + .probe = sifive_pl2_pmu_probe, + .remove_new = sifive_pl2_pmu_remove, + .driver = { + .name = "sifive_pl2_pmu", + .of_match_table = sifve_pl2_pmu_of_match, + }, +}; +module_platform_driver(sifive_pl2_pmu_driver); + +MODULE_LICENSE("GPL");