From patchwork Tue May 7 14:25:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zong Li X-Patchwork-Id: 13657226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3FC7C25B4F for ; Tue, 7 May 2024 14:26:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OKpIVbnVp7z33PVp2YpaSwA04g7sAjdgE3iCM2/fOTI=; b=fsZeKTbJdp/o4p a7UD57u6/61ZbNRMfDMMmAHN6rjDwHVEohmqy4cM50w57xEdKilSrA8yjnVgSeMvCGtANaOvD/C95 4vUoG/bpHgWRIfLzaNCWyN8MlUCXfv2t5Uu9gCE1pTT2QlqOGYucESSnH9WBZJG1o4jPksVEdOUvM dI3dvkAfjITklZMf15rlM39s3+tBiRQcRqHRKvSyVc6qXDPYQScBlgGs1+g5CycXmYrPnW73JlxMb vgqi6VhGbuEVY0a5p5lyP0oCOg88O+MvfQOxTYFSUfX0JYCho7ojcqUeenFWiFidqVLselsPoaxAy dw+9Mkhllvtj7fRvkd6g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Llm-0000000BTyb-3bBm; Tue, 07 May 2024 14:26:14 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Lli-0000000BTwL-3VaQ for linux-riscv@lists.infradead.org; Tue, 07 May 2024 14:26:12 +0000 Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-6f447976de7so2449020b3a.1 for ; Tue, 07 May 2024 07:26:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091968; x=1715696768; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=17FazZA/6yHcTXFZKWGiwqZBDbFiQzycEthhRXcysw0=; b=CH8jbjvOuvNkKtFfd6RLvsqgsR7QbEsTOhtletCz3PYW16Nh8fF6C+HcAnVF+doFSz GBAFxRmIF80nf5GlNSOLZY2uUZiTMk6IEi/m/alIIRVjYlgVyoKas87Py1otwAtO7ZGA nzAb8tDRF7Iu4BJdK2BOzWdkf65FQwQC0/rwG4+D01GjAZrG2m1Cd0iq0WB29VNKed9H 3TGs0Rv71Qj2bOJEPV9J/O3ImbgHzkhweoAUUiy+5iQZXc732IKbwkyjhgDBFPb1enUD gp5tZMft4sLhnUV6/bKUAHvVRREHsUW69/57wp7zXLWbzepfxNuYDvL2H7klLHO+hini X//A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091968; x=1715696768; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=17FazZA/6yHcTXFZKWGiwqZBDbFiQzycEthhRXcysw0=; b=pxgxwN/4H7TLNCHat/iouknv/PpluvtwFykMhz4aDw3nk1/vAeTT5Mht5uaNVxOFIl /GFOSSsvyc7zyINYHi17qToNnjbUTRwtl4tc5caGKLTop/dIPgUODm6HsyIzrSJta/iR +ONKw8z9gTmZmjvBFlI14bLuuJ/j8cEJ2abDGM0iC/tuy2FhbXmNTEqOOalPlVXU79xe LhXWIbX84tb2hCHtzcCMhHUNXLYCEL5o8hTUmLYrLYzB2QSKuXPFAscynRVbf/UGhHM5 kBv5+sFvCKR0SV795dD/uXVjRQtKLYKia7+sXt9oIRVyRLbH31PLD7PH3VtSaqu4M36v 7Ayw== X-Forwarded-Encrypted: i=1; AJvYcCUQKOsJ+GtdEeBwjrJgVU/sYEPXXmpHZ+O23V7SPOCc6YWInAyMgks2hxuo0rE4msv0Dt1jTfyc/wjxhgBLBTllO3xidRXJoUoowRH3rfwA X-Gm-Message-State: AOJu0YzfyDguBwsG8vPZR/TBSzP3xRhRwx95zzQSYeXRcXQLcotSMYaU 8BhtmOY43a1prgObU0kmioPW567m5Tg7AyDazVT98zBGEbFjFcvCufPMJOV+pnSSLoDnEj1i4Nl o X-Google-Smtp-Source: AGHT+IEVF58YFSyNvpGOqdtSAF9LBHk4AkUJDCtEDamXUUk6yuu3XQnJI085PYRdN1EgiwEXBqwp0w== X-Received: by 2002:a05:6a21:920d:b0:1af:86da:3f7 with SMTP id tl13-20020a056a21920d00b001af86da03f7mr11243607pzb.4.1715091968001; Tue, 07 May 2024 07:26:08 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:07 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 1/6] iommu/riscv: Add RISC-V IOMMU PMU support Date: Tue, 7 May 2024 22:25:55 +0800 Message-Id: <20240507142600.23844-2-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240507_072610_913295_361174A8 X-CRM114-Status: GOOD ( 25.94 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch implements the RISC-V IOMMU hardware performance monitor, it includes the counting ans sampling mode. Specification doesn't define the event ID for counting the number of clock cycles, there is no associated iohpmevt0. But we need an event for counting cycle in perf, reserve the maximum number of event ID for it now. Signed-off-by: Zong Li --- drivers/iommu/riscv/Makefile | 4 +- drivers/iommu/riscv/iommu-bits.h | 15 + drivers/iommu/riscv/iommu-pmu.c | 477 +++++++++++++++++++++++++++++++ drivers/iommu/riscv/iommu.h | 8 + 4 files changed, 502 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/riscv/iommu-pmu.c diff --git a/drivers/iommu/riscv/Makefile b/drivers/iommu/riscv/Makefile index f54c9ed17d41..1b02e07d83c9 100644 --- a/drivers/iommu/riscv/Makefile +++ b/drivers/iommu/riscv/Makefile @@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-platform.o -obj-$(CONFIG_RISCV_IOMMU_PCI) += iommu-pci.o +obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-platform.o iommu-pmu.o +obj-$(CONFIG_RISCV_IOMMU_PCI) += iommu-pci.o iommu-pmu.o diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h index 40c379222821..11351cf6c710 100644 --- a/drivers/iommu/riscv/iommu-bits.h +++ b/drivers/iommu/riscv/iommu-bits.h @@ -17,6 +17,7 @@ #include #include #include +#include /* * Chapter 5: Memory Mapped register interface @@ -210,6 +211,7 @@ enum riscv_iommu_ddtp_modes { /* 5.22 Performance monitoring event counters (31 * 64bits) */ #define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068 #define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + ((_n) * 0x8)) +#define RISCV_IOMMU_IOHPMCTR_COUNTER GENMASK_ULL(63, 0) /* 5.23 Performance monitoring event selectors (31 * 64bits) */ #define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160 @@ -251,6 +253,19 @@ enum riscv_iommu_hpmevent_id { RISCV_IOMMU_HPMEVENT_MAX = 9 }; +/* Use maximum event ID for cycle event */ +#define RISCV_IOMMU_HPMEVENT_CYCLE GENMASK_ULL(14, 0) + +#define RISCV_IOMMU_HPM_COUNTER_NUM 32 + +struct riscv_iommu_pmu { + struct pmu pmu; + void __iomem *reg; + int num_counters; + struct perf_event *events[RISCV_IOMMU_HPM_COUNTER_NUM]; + DECLARE_BITMAP(used_counters, RISCV_IOMMU_HPM_COUNTER_NUM); +}; + /* 5.24 Translation request IOVA (64bits) */ #define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258 #define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12) diff --git a/drivers/iommu/riscv/iommu-pmu.c b/drivers/iommu/riscv/iommu-pmu.c new file mode 100644 index 000000000000..6ab50763860f --- /dev/null +++ b/drivers/iommu/riscv/iommu-pmu.c @@ -0,0 +1,477 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 SiFive + * + * Authors + * Zong Li + */ + +#include + +#include "iommu.h" +#include "iommu-bits.h" + +#define to_riscv_iommu_pmu(p) (container_of(p, struct riscv_iommu_pmu, pmu)) + +#define RISCV_IOMMU_PMU_ATTR_EXTRACTOR(_name, _mask) \ + static inline u32 get_##_name(struct perf_event *event) \ + { \ + return FIELD_GET(_mask, event->attr.config); \ + } \ + +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(event, RISCV_IOMMU_IOHPMEVT_EVENT_ID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(partial_matching, RISCV_IOMMU_IOHPMEVT_DMASK); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(pid_pscid, RISCV_IOMMU_IOHPMEVT_PID_PSCID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(did_gscid, RISCV_IOMMU_IOHPMEVT_DID_GSCID); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_pid_pscid, RISCV_IOMMU_IOHPMEVT_PV_PSCV); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_did_gscid, RISCV_IOMMU_IOHPMEVT_DV_GSCV); +RISCV_IOMMU_PMU_ATTR_EXTRACTOR(filter_id_type, RISCV_IOMMU_IOHPMEVT_IDT); + +/* Formats */ +PMU_FORMAT_ATTR(event, "config:0-14"); +PMU_FORMAT_ATTR(partial_matching, "config:15"); +PMU_FORMAT_ATTR(pid_pscid, "config:16-35"); +PMU_FORMAT_ATTR(did_gscid, "config:36-59"); +PMU_FORMAT_ATTR(filter_pid_pscid, "config:60"); +PMU_FORMAT_ATTR(filter_did_gscid, "config:61"); +PMU_FORMAT_ATTR(filter_id_type, "config:62"); + +static struct attribute *riscv_iommu_pmu_formats[] = { + &format_attr_event.attr, + &format_attr_partial_matching.attr, + &format_attr_pid_pscid.attr, + &format_attr_did_gscid.attr, + &format_attr_filter_pid_pscid.attr, + &format_attr_filter_did_gscid.attr, + &format_attr_filter_id_type.attr, + NULL, +}; + +static const struct attribute_group riscv_iommu_pmu_format_group = { + .name = "format", + .attrs = riscv_iommu_pmu_formats, +}; + +/* Events */ +static ssize_t riscv_iommu_pmu_event_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr); + + return sprintf(page, "event=0x%02llx\n", pmu_attr->id); +} + +PMU_EVENT_ATTR(cycle, event_attr_cycle, + RISCV_IOMMU_HPMEVENT_CYCLE, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(dont_count, event_attr_dont_count, + RISCV_IOMMU_HPMEVENT_INVALID, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(untranslated_req, event_attr_untranslated_req, + RISCV_IOMMU_HPMEVENT_URQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(translated_req, event_attr_translated_req, + RISCV_IOMMU_HPMEVENT_TRQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(ats_trans_req, event_attr_ats_trans_req, + RISCV_IOMMU_HPMEVENT_ATS_RQ, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(tlb_miss, event_attr_tlb_miss, + RISCV_IOMMU_HPMEVENT_TLB_MISS, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(ddt_walks, event_attr_ddt_walks, + RISCV_IOMMU_HPMEVENT_DD_WALK, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(pdt_walks, event_attr_pdt_walks, + RISCV_IOMMU_HPMEVENT_PD_WALK, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(s_vs_pt_walks, event_attr_s_vs_pt_walks, + RISCV_IOMMU_HPMEVENT_S_VS_WALKS, riscv_iommu_pmu_event_show); +PMU_EVENT_ATTR(g_pt_walks, event_attr_g_pt_walks, + RISCV_IOMMU_HPMEVENT_G_WALKS, riscv_iommu_pmu_event_show); + +static struct attribute *riscv_iommu_pmu_events[] = { + &event_attr_cycle.attr.attr, + &event_attr_dont_count.attr.attr, + &event_attr_untranslated_req.attr.attr, + &event_attr_translated_req.attr.attr, + &event_attr_ats_trans_req.attr.attr, + &event_attr_tlb_miss.attr.attr, + &event_attr_ddt_walks.attr.attr, + &event_attr_pdt_walks.attr.attr, + &event_attr_s_vs_pt_walks.attr.attr, + &event_attr_g_pt_walks.attr.attr, + NULL, +}; + +static const struct attribute_group riscv_iommu_pmu_events_group = { + .name = "events", + .attrs = riscv_iommu_pmu_events, +}; + +static const struct attribute_group *riscv_iommu_pmu_attr_grps[] = { + &riscv_iommu_pmu_format_group, + &riscv_iommu_pmu_events_group, + NULL, +}; + +/* PMU Operations */ +static void riscv_iommu_pmu_set_counter(struct riscv_iommu_pmu *pmu, u32 idx, + u64 value) +{ + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return; + + writeq(FIELD_PREP(RISCV_IOMMU_IOHPMCTR_COUNTER, value), addr + idx * 8); +} + +static u64 riscv_iommu_pmu_get_counter(struct riscv_iommu_pmu *pmu, u32 idx) +{ + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES; + u64 value; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return -EINVAL; + + value = readq(addr + idx * 8); + + return FIELD_GET(RISCV_IOMMU_IOHPMCTR_COUNTER, value); +} + +static u64 riscv_iommu_pmu_get_event(struct riscv_iommu_pmu *pmu, u32 idx) +{ + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return 0; + + /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */ + if (idx == 0) + return 0; + + return readq(addr + (idx - 1) * 8); +} + +static void riscv_iommu_pmu_set_event(struct riscv_iommu_pmu *pmu, u32 idx, + u64 value) +{ + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE; + + if (WARN_ON_ONCE(idx < 0 || idx > pmu->num_counters)) + return; + + /* There is no associtated IOHPMEVT0 for IOHPMCYCLES */ + if (idx == 0) + return; + + writeq(value, addr + (idx - 1) * 8); +} + +static void riscv_iommu_pmu_enable_counter(struct riscv_iommu_pmu *pmu, u32 idx) +{ + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 value = readl(addr); + + writel(value & ~BIT(idx), addr); +} + +static void riscv_iommu_pmu_disable_counter(struct riscv_iommu_pmu *pmu, u32 idx) +{ + void __iomem *addr = pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH; + u32 value = readl(addr); + + writel(value | BIT(idx), addr); +} + +static void riscv_iommu_pmu_enable_ovf_intr(struct riscv_iommu_pmu *pmu, u32 idx) +{ + u64 value; + + if (get_event(pmu->events[idx]) == RISCV_IOMMU_HPMEVENT_CYCLE) { + value = riscv_iommu_pmu_get_counter(pmu, idx) & ~RISCV_IOMMU_IOHPMCYCLES_OVF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES); + } else { + value = riscv_iommu_pmu_get_event(pmu, idx) & ~RISCV_IOMMU_IOHPMEVT_OF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE + (idx - 1) * 8); + } +} + +static void riscv_iommu_pmu_disable_ovf_intr(struct riscv_iommu_pmu *pmu, u32 idx) +{ + u64 value; + + if (get_event(pmu->events[idx]) == RISCV_IOMMU_HPMEVENT_CYCLE) { + value = riscv_iommu_pmu_get_counter(pmu, idx) | RISCV_IOMMU_IOHPMCYCLES_OVF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMCYCLES); + } else { + value = riscv_iommu_pmu_get_event(pmu, idx) | RISCV_IOMMU_IOHPMEVT_OF; + writeq(value, pmu->reg + RISCV_IOMMU_REG_IOHPMEVT_BASE + (idx - 1) * 8); + } +} + +static void riscv_iommu_pmu_start_all(struct riscv_iommu_pmu *pmu) +{ + int idx; + + for_each_set_bit(idx, pmu->used_counters, pmu->num_counters) { + riscv_iommu_pmu_enable_ovf_intr(pmu, idx); + riscv_iommu_pmu_enable_counter(pmu, idx); + } +} + +static void riscv_iommu_pmu_stop_all(struct riscv_iommu_pmu *pmu) +{ + writel(GENMASK_ULL(pmu->num_counters - 1, 0), + pmu->reg + RISCV_IOMMU_REG_IOCOUNTINH); +} + +/* PMU APIs */ +static int riscv_iommu_pmu_set_period(struct perf_event *event) +{ + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + s64 left = local64_read(&hwc->period_left); + s64 period = hwc->sample_period; + u64 max_period = RISCV_IOMMU_IOHPMCTR_COUNTER; + int ret = 0; + + if (unlikely(left <= -period)) { + left = period; + local64_set(&hwc->period_left, left); + hwc->last_period = period; + ret = 1; + } + + if (unlikely(left <= 0)) { + left += period; + local64_set(&hwc->period_left, left); + hwc->last_period = period; + ret = 1; + } + + /* + * Limit the maximum period to prevent the counter value + * from overtaking the one we are about to program. In + * effect we are reducing max_period to account for + * interrupt latency (and we are being very conservative). + */ + if (left > (max_period >> 1)) + left = (max_period >> 1); + + local64_set(&hwc->prev_count, (u64)-left); + riscv_iommu_pmu_set_counter(pmu, hwc->idx, (u64)(-left) & max_period); + perf_event_update_userpage(event); + + return ret; +} + +static int riscv_iommu_pmu_event_init(struct perf_event *event) +{ + struct hw_perf_event *hwc = &event->hw; + + hwc->idx = -1; + hwc->config = event->attr.config; + + if (!is_sampling_event(event)) { + /* + * For non-sampling runs, limit the sample_period to half + * of the counter width. That way, the new counter value + * is far less likely to overtake the previous one unless + * you have some serious IRQ latency issues. + */ + hwc->sample_period = RISCV_IOMMU_IOHPMCTR_COUNTER >> 1; + hwc->last_period = hwc->sample_period; + local64_set(&hwc->period_left, hwc->sample_period); + } + + return 0; +} + +static void riscv_iommu_pmu_update(struct perf_event *event) +{ + struct hw_perf_event *hwc = &event->hw; + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu); + u64 delta, prev, now; + u32 idx = hwc->idx; + + do { + prev = local64_read(&hwc->prev_count); + now = riscv_iommu_pmu_get_counter(pmu, idx); + } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev); + + delta = FIELD_GET(RISCV_IOMMU_IOHPMCTR_COUNTER, now - prev); + local64_add(delta, &event->count); + local64_sub(delta, &hwc->period_left); +} + +static void riscv_iommu_pmu_start(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + + if (WARN_ON_ONCE(!(event->hw.state & PERF_HES_STOPPED))) + return; + + if (flags & PERF_EF_RELOAD) + WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE)); + + hwc->state = 0; + riscv_iommu_pmu_set_period(event); + riscv_iommu_pmu_set_event(pmu, hwc->idx, hwc->config); + riscv_iommu_pmu_enable_ovf_intr(pmu, hwc->idx); + riscv_iommu_pmu_enable_counter(pmu, hwc->idx); + + perf_event_update_userpage(event); +} + +static void riscv_iommu_pmu_stop(struct perf_event *event, int flags) +{ + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + + if (hwc->state & PERF_HES_STOPPED) + return; + + riscv_iommu_pmu_set_event(pmu, hwc->idx, RISCV_IOMMU_HPMEVENT_INVALID); + riscv_iommu_pmu_disable_counter(pmu, hwc->idx); + + if ((flags & PERF_EF_UPDATE) && !(hwc->state & PERF_HES_UPTODATE)) + riscv_iommu_pmu_update(event); + + hwc->state |= PERF_HES_STOPPED | PERF_HES_UPTODATE; +} + +static int riscv_iommu_pmu_add(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc = &event->hw; + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu); + unsigned int num_counters = pmu->num_counters; + int idx; + + /* Reserve index zero for iohpmcycles */ + if (get_event(event) == RISCV_IOMMU_HPMEVENT_CYCLE) + idx = 0; + else + idx = find_next_zero_bit(pmu->used_counters, num_counters, 1); + + if (idx == num_counters) + return -EAGAIN; + + set_bit(idx, pmu->used_counters); + + pmu->events[idx] = event; + hwc->idx = idx; + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; + + if (flags & PERF_EF_START) + riscv_iommu_pmu_start(event, flags); + + /* Propagate changes to the userspace mapping. */ + perf_event_update_userpage(event); + + return 0; +} + +static void riscv_iommu_pmu_read(struct perf_event *event) +{ + riscv_iommu_pmu_update(event); +} + +static void riscv_iommu_pmu_del(struct perf_event *event, int flags) +{ + struct hw_perf_event *hwc = &event->hw; + struct riscv_iommu_pmu *pmu = to_riscv_iommu_pmu(event->pmu); + int idx = hwc->idx; + + riscv_iommu_pmu_stop(event, PERF_EF_UPDATE); + pmu->events[idx] = NULL; + clear_bit(idx, pmu->used_counters); + perf_event_update_userpage(event); +} + +irqreturn_t riscv_iommu_pmu_handle_irq(struct riscv_iommu_pmu *pmu) +{ + struct perf_sample_data data; + struct pt_regs *regs; + u32 ovf = readl(pmu->reg + RISCV_IOMMU_REG_IOCOUNTOVF); + int idx; + + if (!ovf) + return IRQ_NONE; + + riscv_iommu_pmu_stop_all(pmu); + + regs = get_irq_regs(); + + for_each_set_bit(idx, (unsigned long *)&ovf, pmu->num_counters) { + struct perf_event *event = pmu->events[idx]; + struct hw_perf_event *hwc; + + if (WARN_ON_ONCE(!event) || !is_sampling_event(event)) + continue; + + hwc = &event->hw; + + riscv_iommu_pmu_update(event); + perf_sample_data_init(&data, 0, hwc->last_period); + if (!riscv_iommu_pmu_set_period(event)) + continue; + + if (perf_event_overflow(event, &data, regs)) + riscv_iommu_pmu_stop(event, 0); + } + + riscv_iommu_pmu_start_all(pmu); + + return IRQ_HANDLED; +} + +int riscv_iommu_pmu_init(struct riscv_iommu_pmu *pmu, void __iomem *reg, + const char *dev_name) +{ + char *name; + int ret; + + pmu->reg = reg; + pmu->num_counters = RISCV_IOMMU_HPM_COUNTER_NUM; + + pmu->pmu = (struct pmu) { + .task_ctx_nr = perf_invalid_context, + .event_init = riscv_iommu_pmu_event_init, + .add = riscv_iommu_pmu_add, + .del = riscv_iommu_pmu_del, + .start = riscv_iommu_pmu_start, + .stop = riscv_iommu_pmu_stop, + .read = riscv_iommu_pmu_read, + .attr_groups = riscv_iommu_pmu_attr_grps, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE, + .module = THIS_MODULE, + }; + + name = kasprintf(GFP_KERNEL, "riscv-iommu-pmu@%s", dev_name); + + ret = perf_pmu_register(&pmu->pmu, name, -1); + if (ret) { + pr_err("Failed to register riscv-iommu-pmu@%s: %d\n", + dev_name, ret); + return ret; + } + + /* Stop all counters and later start the counter with perf */ + riscv_iommu_pmu_stop_all(pmu); + + pr_info("riscv-iommu-pmu@%s: Registered with %d counters\n", + dev_name, pmu->num_counters); + + return 0; +} + +void riscv_iommu_pmu_uninit(struct riscv_iommu_pmu *pmu) +{ + int idx; + + /* Disable interrupt and functions */ + for_each_set_bit(idx, pmu->used_counters, pmu->num_counters) { + riscv_iommu_pmu_disable_counter(pmu, idx); + riscv_iommu_pmu_disable_ovf_intr(pmu, idx); + } + + perf_pmu_unregister(&pmu->pmu); +} diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index 03e0c45bc7e1..ff66822c1114 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -60,11 +60,19 @@ struct riscv_iommu_device { unsigned int ddt_mode; dma_addr_t ddt_phys; u64 *ddt_root; + + /* hardware performance monitor */ + struct riscv_iommu_pmu pmu; }; int riscv_iommu_init(struct riscv_iommu_device *iommu); void riscv_iommu_remove(struct riscv_iommu_device *iommu); +int riscv_iommu_pmu_init(struct riscv_iommu_pmu *pmu, void __iomem *reg, + const char *name); +void riscv_iommu_pmu_uninit(struct riscv_iommu_pmu *pmu); +irqreturn_t riscv_iommu_pmu_handle_irq(struct riscv_iommu_pmu *pmu); + #define riscv_iommu_readl(iommu, addr) \ readl_relaxed((iommu)->reg + (addr)) From patchwork Tue May 7 14:25:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zong Li X-Patchwork-Id: 13657225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1FAF1C25B77 for ; Tue, 7 May 2024 14:26:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nMHAZcki+QffYj1/75aCJZ9fxYjC2qABaV6eFl8tFyo=; b=lo0Z8gefYBqIBk okqIzUXyHBdxpth8h72GGBzPstowDevgFqAQYDF14coa+IoZVu3VXiW71SQLl4nLeJWYg1OiP0n2j cPqz2hu/5jpKIyaQEIgg6KomQxLo3NEDG45qIIkioi7vGEeZOlhtP8VpQ++g+KQGLmvy5sGr47cbx yGKOpQ/TaVF692DWsLFw/oErc8eEWjHxbe7LyFdEOOUvGrCAp64rrtSk7Aeg2yUyr1jfxmgipEW7r IM2sex1vrhG1uVN3/ilTOYGaeWnwdwVj/WT7BG1CN8Qxkl6FbWrtAgYIZDdq6ZFf1WovnieCxr4gs iQFA0yteN/5X6F+GhYOQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Llp-0000000BU0K-0ERd; Tue, 07 May 2024 14:26:17 +0000 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Lll-0000000BTwf-2maW for linux-riscv@lists.infradead.org; Tue, 07 May 2024 14:26:15 +0000 Received: by mail-pf1-x434.google.com with SMTP id d2e1a72fcca58-6f4521ad6c0so2343613b3a.0 for ; Tue, 07 May 2024 07:26:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091970; x=1715696770; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0jz4ENNoetocqIssCS0B2oXUtzcTlnoT1WDoFLx5sIU=; b=RTlgseonOBKHjrDAcKqvw6jr6OXApKhwdf33zztUq55rTJPuJU0IYtUTY06Vlkqai7 E+TM5B845zEGVrmCM82Fq7LcngCjJMYVJIpf04p40g1WaS24TtO2quN4JjAokMQBWRMm 4TnK0UAx/dELOfdUjYoCy8K82Be8Ku7cPO021JvYqtdkmX/2Y5WTAtt1PdxCFJ6fY0vC W3XNk8sdcgAQzD7J/454UKw43XSh1FVzrSF3nmPdtx2oMGm/WEXEhcPpFpUgQyGQZ+yR FW3DrKEHD5txoODqz9nAtGXPiQuFu6jVA+J2xLhTpG0dcITiM/yvvuOPZ7mDI1emhn7L p3zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091970; x=1715696770; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0jz4ENNoetocqIssCS0B2oXUtzcTlnoT1WDoFLx5sIU=; b=bSDTEhZ7astXcllfkZTYtRfe/uUTV09EVf/ryCiC9Ln7yqOyCjdT81lVyCU6CtYBHj DE3yZA559dPRGUNVGp3JPygvoyzlxX5Rl05XcKQfr7izEE0Rdop5EeuNc3tPBHEtyZ8r d2z0xZUKh6UfFPPEc7f6euE3/sCDAwauRn6Ls+dUUnE1MasB2wNyJ4Xy3t+eDxbETzDt yxKGoLShX5UlXAel6PtMQ1wDNfTDaLbC8q6uiECOTm/UvamkgXejct3FELL4/7mB5AZS UjgfTfdYuP9pLWq+5lV9cn2ctoOgDObTIe3Wp4B+kisWviVEpQsF/+KRsNHLAJAMvB/+ 3UMw== X-Forwarded-Encrypted: i=1; AJvYcCWHA6uxc642nXGkHOtnq9oa+lJuSGmRFRjubsxvBiTQFkpFNCqapO038VtYMjk+X67P91ljR92p8c43AlrZtZSIqgNUPvz/A9Av83f5iyhv X-Gm-Message-State: AOJu0YzRlyoXrz5Ur7+Y+JeFyNS5pUOra5gm34IY1DTj39EXx+cG7EeC iChSnUC4WSC9GwHXFyLoeNWzVOwpWcPpbsvzpsBwdlT7ymZmofUFC6NYUJh+5DE= X-Google-Smtp-Source: AGHT+IFFwJydlhv8Eo1uigoZzco+8G3QZGbkY7AV3d8a5KKB5Jnd/eMaJgfIknHgv405ntAxsJkBlA== X-Received: by 2002:a05:6a20:9c97:b0:1af:9ee6:25c4 with SMTP id mj23-20020a056a209c9700b001af9ee625c4mr8750903pzb.42.1715091970500; Tue, 07 May 2024 07:26:10 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:10 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 2/6] iommu/riscv: Support HPM and interrupt handling Date: Tue, 7 May 2024 22:25:56 +0800 Message-Id: <20240507142600.23844-3-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240507_072613_740631_5A6D5D97 X-CRM114-Status: GOOD ( 13.19 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch initialize the pmu stuff and uninitialize it when driver removing. The interrupt handling is also provided, this handler need to be primary handler instead of thread function, because pt_regs is empty when threading the IRQ, but pt_regs is necessary by perf_event_overflow. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu.c | 59 +++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index ec701fde520f..e0bf74a9c64d 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -526,6 +526,56 @@ static irqreturn_t riscv_iommu_fltq_process(int irq, void *data) return IRQ_HANDLED; } +/* + * IOMMU Hardware performance monitor + */ + +/* HPM interrupt primary handler */ +static irqreturn_t riscv_iommu_hpm_irq_handler(int irq, void *dev_id) +{ + struct riscv_iommu_device *iommu = (struct riscv_iommu_device *)dev_id; + + /* Process pmu irq */ + riscv_iommu_pmu_handle_irq(&iommu->pmu); + + /* Clear performance monitoring interrupt pending */ + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IPSR, RISCV_IOMMU_IPSR_PMIP); + + return IRQ_HANDLED; +} + +/* HPM initialization */ +static int riscv_iommu_hpm_enable(struct riscv_iommu_device *iommu) +{ + int rc; + + if (!(iommu->caps & RISCV_IOMMU_CAP_HPM)) + return 0; + + /* + * pt_regs is empty when threading the IRQ, but pt_regs is necessary + * by perf_event_overflow. Use primary handler instead of thread + * function for PM IRQ. + */ + rc = devm_request_irq(iommu->dev, iommu->irqs[RISCV_IOMMU_IPSR_PMIP], + riscv_iommu_hpm_irq_handler, 0, NULL, iommu); + if (rc) + return rc; + + return riscv_iommu_pmu_init(&iommu->pmu, iommu->reg, dev_name(iommu->dev)); +} + +/* HPM uninitialization */ +static void riscv_iommu_hpm_disable(struct riscv_iommu_device *iommu) +{ + if (!(iommu->caps & RISCV_IOMMU_CAP_HPM)) + return; + + devm_free_irq(iommu->dev, iommu->irqs[RISCV_IOMMU_IPSR_PMIP], iommu); + + riscv_iommu_pmu_uninit(&iommu->pmu); +} + /* Lookup and initialize device context info structure. */ static struct riscv_iommu_dc *riscv_iommu_get_dc(struct riscv_iommu_device *iommu, unsigned int devid) @@ -1551,6 +1601,9 @@ void riscv_iommu_remove(struct riscv_iommu_device *iommu) riscv_iommu_iodir_set_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF); riscv_iommu_queue_disable(&iommu->cmdq); riscv_iommu_queue_disable(&iommu->fltq); + + if (iommu->caps & RISCV_IOMMU_CAP_HPM) + riscv_iommu_pmu_uninit(&iommu->pmu); } int riscv_iommu_init(struct riscv_iommu_device *iommu) @@ -1590,6 +1643,10 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) if (rc) goto err_queue_disable; + rc = riscv_iommu_hpm_enable(iommu); + if (rc) + goto err_hpm_disable; + rc = iommu_device_sysfs_add(&iommu->iommu, NULL, NULL, "riscv-iommu@%s", dev_name(iommu->dev)); if (rc) { @@ -1608,6 +1665,8 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) err_remove_sysfs: iommu_device_sysfs_remove(&iommu->iommu); err_iodir_off: + riscv_iommu_hpm_disable(iommu); +err_hpm_disable: riscv_iommu_iodir_set_mode(iommu, RISCV_IOMMU_DDTP_MODE_OFF); err_queue_disable: riscv_iommu_queue_disable(&iommu->fltq); From patchwork Tue May 7 14:25:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zong Li X-Patchwork-Id: 13657227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BA625C25B74 for ; Tue, 7 May 2024 14:26:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zdYN0Tv4eiHTZPcJBuwT6H+zEbdy/CJKLza8PVrzmok=; b=Bbm9v5p9X8BqKO o4jrcQsG57HMurgMUd7EA5FwudoxrTz5FtmK3bZK/Fdq+eFBO21aI3pu6kLpoyKK1AzZzi9xlR9z5 y2cavmv1/Q69Zy5MVyp6gbzq+vlic2YSBgGBJdwjcdzfKLVbQdpSSdShwstdMyNwqVZSdgWDK/iKg QVmPmXurW3PusVYcBdE0uTNtkdvBeOrIrsEEHMAUqIqnLH7VMW5NKZgevdTj3iFVBepH3qkHGrtmr oCvHJqw+4Lp1hv+ZeA9Do4R0WoXZ2mMWJw5vBWZlCZGMwXP69ahMEW1rQnki6WeBAGzJ2ul1mUy3d uGKCrYedEWqA/Qig+Kmw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Lls-0000000BU2f-0xcq; Tue, 07 May 2024 14:26:20 +0000 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Lln-0000000BTya-2KT3 for linux-riscv@lists.infradead.org; Tue, 07 May 2024 14:26:18 +0000 Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-6f4178aec15so2738232b3a.0 for ; Tue, 07 May 2024 07:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091974; x=1715696774; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Q5Yiv5A35fqkZLewYyNasBxNRcaYrtDBJ52Nob0v8A4=; b=I6lnfKvW+IAwaZEkZHN08rW+/Go71BDSe0fjP4AzD54pMkGC/iT1TRgTAv6nfKwrHt DiWmw4Bwm0jYGjJ6wUybp9asla2OsBuRfF1udSwvZI6KIiWcI8PoMhbYIYBR0cpG+CmV t7/3izadW+ntoyiLJw2z6pmWaFJjQ+uqrp/GW/rlpXkIOp/eS6N7VGYDtg0DCjFM7I4R IuQB3atKmh8b/YGA4O2tOvf4fkq6ENQ+N36Tnj54ljuAWZUb5SdSrVZRJ9NmZvbfhYSF MLVcSd/h8r8LbrrLbiIrJLAEcen2ckba723rcNOEE9ZvGJFjcd9DQT3PgypZuMJm5xDW Cfig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091974; x=1715696774; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Q5Yiv5A35fqkZLewYyNasBxNRcaYrtDBJ52Nob0v8A4=; b=sGxwTxZWAP97j6FfcOYl4hlBVWiYA+bIWuXsx0z15dz0v/ikmylgVBfLovYGYpSJtj 4KlphZR+6I2vJ/Cz9pY0yB2CrumM2cx8kr8ebvjTPkxD/f4p3d88fKvlPn8FI968+Bk8 QvZZxjUipQOvMmTm4emjxVVg0l69Q+oZE4uiukLInUPpO5x1dur/NyZdyivdCDYnUPh8 aCbUheyrTh3Pd6tz3kyw/wtaldLOWdK/dlKWYxLuUufzkZAXsWm6sSTRj/831/7/wvFO rlqQBFg2drLwerLog/4u0MC+ZctR0r2BcVV6WoJ//ZU6YB6JpkhdyOCBdoUsiKU392df YHQQ== X-Forwarded-Encrypted: i=1; AJvYcCUKMf5xktlDm1XF6D5oaOqjyS2ZwdK89CHZEQ24QXzYSPYaDrq0dpUkAyCC7KvM9ECq7Vje1JNL5XaiXHYn9/zcjrpvh5JYZLXm04NQ8JFn X-Gm-Message-State: AOJu0Yzr6OdUs7dqGBG87YQeRF2Phq/2Wv2aQoZyYdulHb2I9FpcT8aO gDsk+jqGW6n0eyoH0qgt/xobIRzRMfcmyflZ2b6p61+/QIQ5r1fKUblrv8Qa4B4= X-Google-Smtp-Source: AGHT+IHdg7+7I4bSI4CruHY7MHG+tPkBhVw4YY0PCUXdy/YjCBHjgssv5B+jlzNMBIZ1jyV/U90aVQ== X-Received: by 2002:a05:6a00:4f82:b0:6ed:21b2:cb17 with SMTP id ld2-20020a056a004f8200b006ed21b2cb17mr14575610pfb.4.1715091972855; Tue, 07 May 2024 07:26:12 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:12 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 3/6] iommu/riscv: support GSCID Date: Tue, 7 May 2024 22:25:57 +0800 Message-Id: <20240507142600.23844-4-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240507_072615_700739_5CDC2D61 X-CRM114-Status: GOOD ( 19.70 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch adds a global ID Allocator for GSCID and a wrap for setting up GSCID in IOTLB invalidation command. Set up iohgatp to enable second stage table and flus stage-2 table if the GSCID is allocated. The GSCID of domain should be freed when release domain. GSCID will be allocated for parent domain in nested IOMMU process. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu-bits.h | 7 +++ drivers/iommu/riscv/iommu.c | 81 ++++++++++++++++++++++---------- 2 files changed, 62 insertions(+), 26 deletions(-) diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h index 11351cf6c710..62b1ee387357 100644 --- a/drivers/iommu/riscv/iommu-bits.h +++ b/drivers/iommu/riscv/iommu-bits.h @@ -728,6 +728,13 @@ static inline void riscv_iommu_cmd_inval_vma(struct riscv_iommu_command *cmd) cmd->dword1 = 0; } +static inline void riscv_iommu_cmd_inval_gvma(struct riscv_iommu_command *cmd) +{ + cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IOTINVAL_OPCODE) | + FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA); + cmd->dword1 = 0; +} + static inline void riscv_iommu_cmd_inval_set_addr(struct riscv_iommu_command *cmd, u64 addr) { diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index e0bf74a9c64d..d38e09b138b6 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -45,6 +45,10 @@ static DEFINE_IDA(riscv_iommu_pscids); #define RISCV_IOMMU_MAX_PSCID (BIT(20) - 1) +/* IOMMU GSCID allocation namespace. */ +static DEFINE_IDA(riscv_iommu_gscids); +#define RISCV_IOMMU_MAX_GSCID (BIT(16) - 1) + /* Device resource-managed allocations */ struct riscv_iommu_devres { void *addr; @@ -826,6 +830,7 @@ struct riscv_iommu_domain { struct list_head bonds; spinlock_t lock; /* protect bonds list updates. */ int pscid; + int gscid; int numa_node; int amo_enabled:1; unsigned int pgd_mode; @@ -919,29 +924,43 @@ static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain, rcu_read_lock(); prev = NULL; - list_for_each_entry_rcu(bond, &domain->bonds, list) { - iommu = dev_to_iommu(bond->dev); - /* - * IOTLB invalidation request can be safely omitted if already sent - * to the IOMMU for the same PSCID, and with domain->bonds list - * arranged based on the device's IOMMU, it's sufficient to check - * last device the invalidation was sent to. - */ - if (iommu == prev) - continue; - - riscv_iommu_cmd_inval_vma(&cmd); - riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid); - if (len && len >= RISCV_IOMMU_IOTLB_INVAL_LIMIT) { - for (iova = start; iova < end; iova += PAGE_SIZE) { - riscv_iommu_cmd_inval_set_addr(&cmd, iova); + /* + * Host domain needs to flush entries in stage-2 for MSI mapping. + * However, device is bound to s1 domain instead of s2 domain. + * We need to flush mapping without looping devices of s2 domain + */ + if (domain->gscid) { + riscv_iommu_cmd_inval_gvma(&cmd); + riscv_iommu_cmd_inval_set_gscid(&cmd, domain->gscid); + riscv_iommu_cmd_send(iommu, &cmd, 0); + riscv_iommu_cmd_iofence(&cmd); + riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_QUEUE_TIMEOUT); + } else { + list_for_each_entry_rcu(bond, &domain->bonds, list) { + iommu = dev_to_iommu(bond->dev); + + /* + * IOTLB invalidation request can be safely omitted if already sent + * to the IOMMU for the same PSCID, and with domain->bonds list + * arranged based on the device's IOMMU, it's sufficient to check + * last device the invalidation was sent to. + */ + if (iommu == prev) + continue; + + riscv_iommu_cmd_inval_vma(&cmd); + riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid); + if (len && len >= RISCV_IOMMU_IOTLB_INVAL_LIMIT) { + for (iova = start; iova < end; iova += PAGE_SIZE) { + riscv_iommu_cmd_inval_set_addr(&cmd, iova); + riscv_iommu_cmd_send(iommu, &cmd, 0); + } + } else { riscv_iommu_cmd_send(iommu, &cmd, 0); } - } else { - riscv_iommu_cmd_send(iommu, &cmd, 0); + prev = iommu; } - prev = iommu; } prev = NULL; @@ -972,7 +991,7 @@ static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain, * interim translation faults. */ static void riscv_iommu_iodir_update(struct riscv_iommu_device *iommu, - struct device *dev, u64 fsc, u64 ta) + struct device *dev, u64 fsc, u64 ta, u64 iohgatp) { struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); struct riscv_iommu_dc *dc; @@ -1012,6 +1031,7 @@ static void riscv_iommu_iodir_update(struct riscv_iommu_device *iommu, /* Update device context, write TC.V as the last step. */ WRITE_ONCE(dc->fsc, fsc); WRITE_ONCE(dc->ta, ta & RISCV_IOMMU_PC_TA_PSCID); + WRITE_ONCE(dc->iohgatp, iohgatp); WRITE_ONCE(dc->tc, tc); } } @@ -1271,6 +1291,9 @@ static void riscv_iommu_free_paging_domain(struct iommu_domain *iommu_domain) if ((int)domain->pscid > 0) ida_free(&riscv_iommu_pscids, domain->pscid); + if ((int)domain->gscid > 0) + ida_free(&riscv_iommu_gscids, domain->gscid); + riscv_iommu_pte_free(domain, _io_pte_entry(pfn, _PAGE_TABLE), NULL); kfree(domain); } @@ -1296,7 +1319,7 @@ static int riscv_iommu_attach_paging_domain(struct iommu_domain *iommu_domain, struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); struct riscv_iommu_device *iommu = dev_to_iommu(dev); struct riscv_iommu_info *info = dev_iommu_priv_get(dev); - u64 fsc, ta; + u64 fsc = 0, iohgatp = 0, ta; if (!riscv_iommu_pt_supported(iommu, domain->pgd_mode)) return -ENODEV; @@ -1314,12 +1337,18 @@ static int riscv_iommu_attach_paging_domain(struct iommu_domain *iommu_domain, */ riscv_iommu_iotlb_inval(domain, 0, ULONG_MAX); - fsc = FIELD_PREP(RISCV_IOMMU_PC_FSC_MODE, domain->pgd_mode) | - FIELD_PREP(RISCV_IOMMU_PC_FSC_PPN, virt_to_pfn(domain->pgd_root)); + if (domain->gscid) + iohgatp = FIELD_PREP(RISCV_IOMMU_DC_IOHGATP_MODE, domain->pgd_mode) | + FIELD_PREP(RISCV_IOMMU_DC_IOHGATP_GSCID, domain->gscid) | + FIELD_PREP(RISCV_IOMMU_DC_IOHGATP_PPN, virt_to_pfn(domain->pgd_root)); + else + fsc = FIELD_PREP(RISCV_IOMMU_PC_FSC_MODE, domain->pgd_mode) | + FIELD_PREP(RISCV_IOMMU_PC_FSC_PPN, virt_to_pfn(domain->pgd_root)); + ta = FIELD_PREP(RISCV_IOMMU_PC_TA_PSCID, domain->pscid) | RISCV_IOMMU_PC_TA_V; - riscv_iommu_iodir_update(iommu, dev, fsc, ta); + riscv_iommu_iodir_update(iommu, dev, fsc, ta, iohgatp); riscv_iommu_bond_unlink(info->domain, dev); info->domain = domain; @@ -1422,7 +1451,7 @@ static int riscv_iommu_attach_blocking_domain(struct iommu_domain *iommu_domain, struct riscv_iommu_device *iommu = dev_to_iommu(dev); struct riscv_iommu_info *info = dev_iommu_priv_get(dev); - riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, 0); + riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, 0, 0); riscv_iommu_bond_unlink(info->domain, dev); info->domain = NULL; @@ -1442,7 +1471,7 @@ static int riscv_iommu_attach_identity_domain(struct iommu_domain *iommu_domain, struct riscv_iommu_device *iommu = dev_to_iommu(dev); struct riscv_iommu_info *info = dev_iommu_priv_get(dev); - riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, RISCV_IOMMU_PC_TA_V); + riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, RISCV_IOMMU_PC_TA_V, 0); riscv_iommu_bond_unlink(info->domain, dev); info->domain = NULL; From patchwork Tue May 7 14:25:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zong Li X-Patchwork-Id: 13657228 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7B73EC10F1A for ; Tue, 7 May 2024 14:26:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FtPD/U7QY2g8pxSCO3wV+U2fheafO+8l0Qqv6I3xsAw=; b=YZtl/cWx3CO12Z dv/qhDoKhIlgF2Wyt1Virmce63p9gEbOYleucwfIrcSU4fObqr+XPTJHTdFDfMQU74KtqzlmW52tY 4GY+plfKbP+zQS1c0dRRrO7VyPvqV6h6AI1cL7k15DPRX/NIrMMWitDkHnBm5roEjvK9/aeOeu071 EfVDSOa+EXd3hTELxml0zKO6sbI6VUThqyUBeBnSqFpyFPlcGoP9kspAsNnFMOJuD4I8257biB62i hyTrLXoc+pOCFjNMI1eX7Tmzh6Z0l25KV43YkiDR4lN1dyTQ6qB1FZz895R/PkJB0z3ExR8N5/4f4 55rxSqarEM2OZYhypJtQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Llv-0000000BU4c-23Ia; Tue, 07 May 2024 14:26:23 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Llp-0000000BU0a-3mQb for linux-riscv@lists.infradead.org; Tue, 07 May 2024 14:26:19 +0000 Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-6f44dd41a5cso3082004b3a.0 for ; Tue, 07 May 2024 07:26:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091977; x=1715696777; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=b+b6a14h4JCjFlQP6t364i5UnYTPW3CwsNAJ2Ic+YBk=; b=bQ+G4VgdGkkivQoi1vC/JWfStF20/T/3nirxd4US21zfYZMkjI5qOsZhk5tIh3/igg +gE1FPCYCjsX9a+7DploWmlvqgpKpeb7OG9xNaIRDib36dTd+YwsPB31sq2ryioe7zEP 01YqpTNMeTS7SDwulCFCroFVu5VlgSNJWU1nN+y5JN2G7u/ih9KGYn/TjW9bHff2/ZLB z4zNZ4FteS0FTien8gqqRgkvBPSWHmvlBQE3i7k3w2pIa9xUuYQf5z0O2VNJe1PR+FCO pR8UzHRUa4fvQNb9OwjWHeM+vaXJa8Hg4IMNympAtYQ/qkjLbXIJYWhpqR6qrpIqGFdW nnkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091977; x=1715696777; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=b+b6a14h4JCjFlQP6t364i5UnYTPW3CwsNAJ2Ic+YBk=; b=fxv6rG5X2d2G7XiSW6DJHeiWoFBnidWxsYFx4k3Qh5rnOsMUUADhQivh49Mcu0sxhH GIlvP3XIAkugcdXXwecqvySjsOoMUDpNR9DRbJ1I8m80Zb0CX1dPCUZYrJXLsyoJFCP2 TIGCPY9Xqa6asYJVyIerOAhbMIe9/P+gLcIZU0laO02lU0HPyKBRLNZbSDAiyL49d6kW GnWBdcow3ohXqYS8D+58v0dxFAe5HVQFRLRYVeueHEfUluOn6UnZMZlft/D2XzovPOam HOTjB21dyT1xf5XWvND6jrqgzxsm1L3Tv1b1KSAwM9FR8np9CdjE+b5TuUFtcSX3VXRx FqOA== X-Forwarded-Encrypted: i=1; AJvYcCWlYWljJJeGDgUhZK2FOCZxZdB4kcNmgfIIoPulFAINaSvAjtZ4coawNndiEBWLtF5hOWMnRNRPHOir36R/XIWftGV/jfGBjn3kLS155J4b X-Gm-Message-State: AOJu0YyP2tvbEPq9mAZFaerpVefxN2SrHegWlm+jc9KSf5sMETMvh/jr 6k4jpHJ0Wczw3K9SQXuZlZ78u5YmyFClUJPKX48ro/ygwV1spw0tJKihPAm09xU= X-Google-Smtp-Source: AGHT+IGGpKiuhFlm6XAxN2dekhHNztwFPkYH1wk0fi2F0H3c/gi7fQ/EHQEjzLAwbTOOSYiKwoJeKg== X-Received: by 2002:a05:6a00:843:b0:6ea:86f2:24fb with SMTP id q3-20020a056a00084300b006ea86f224fbmr15303975pfk.25.1715091976310; Tue, 07 May 2024 07:26:16 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:16 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 4/6] iommu/riscv: support nested iommu for getting iommu hardware information Date: Tue, 7 May 2024 22:25:58 +0800 Message-Id: <20240507142600.23844-5-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240507_072618_045085_C716B0AD X-CRM114-Status: GOOD ( 12.84 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch implements .hw_info operation and the related data structures for passing the IOMMU hardware capabilities for iommufd. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu.c | 23 +++++++++++++++++++++++ include/uapi/linux/iommufd.h | 13 +++++++++++++ 2 files changed, 36 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index d38e09b138b6..072251f6ad85 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -19,6 +19,7 @@ #include #include #include +#include #include "../iommu-pages.h" #include "iommu-bits.h" @@ -1485,6 +1486,27 @@ static struct iommu_domain riscv_iommu_identity_domain = { } }; +static void *riscv_iommu_hw_info(struct device *dev, u32 *length, u32 *type) +{ + struct riscv_iommu_device *iommu = dev_to_iommu(dev); + struct iommu_hw_info_riscv_iommu *info; + + if (!iommu) + return ERR_PTR(-ENODEV); + + info = kzalloc(sizeof(*info), GFP_KERNEL); + if (!info) + return ERR_PTR(-ENOMEM); + + info->capability = iommu->caps; + info->fctl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL); + + *length = sizeof(*info); + *type = IOMMU_HW_INFO_TYPE_RISCV_IOMMU; + + return info; +} + static int riscv_iommu_device_domain_type(struct device *dev) { return 0; @@ -1560,6 +1582,7 @@ static void riscv_iommu_release_device(struct device *dev) static const struct iommu_ops riscv_iommu_ops = { .pgsize_bitmap = SZ_4K, .of_xlate = riscv_iommu_of_xlate, + .hw_info = riscv_iommu_hw_info, .identity_domain = &riscv_iommu_identity_domain, .blocked_domain = &riscv_iommu_blocking_domain, .release_domain = &riscv_iommu_blocking_domain, diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index 1dfeaa2e649e..ec9aafd7d373 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -475,15 +475,28 @@ struct iommu_hw_info_vtd { __aligned_u64 ecap_reg; }; +/** + * struct iommu_hw_info_riscv_iommu - RISCV IOMMU hardware information + * + * @capability: Value of RISC-V IOMMU capability register + * @fctl: Value of RISC-V IOMMU feature control register + */ +struct iommu_hw_info_riscv_iommu { + __aligned_u64 capability; + __u32 fctl; +}; + /** * enum iommu_hw_info_type - IOMMU Hardware Info Types * @IOMMU_HW_INFO_TYPE_NONE: Used by the drivers that do not report hardware * info * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type + * @IOMMU_HW_INFO_TYPE_RISCV_IOMMU: RISC-V iommu info type */ enum iommu_hw_info_type { IOMMU_HW_INFO_TYPE_NONE, IOMMU_HW_INFO_TYPE_INTEL_VTD, + IOMMU_HW_INFO_TYPE_RISCV_IOMMU, }; /** From patchwork Tue May 7 14:25:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zong Li X-Patchwork-Id: 13657229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7B1CDC25B5F for ; Tue, 7 May 2024 14:26:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fJOk0Z5jE0v1jyoO9W90C53sdpGc6SBhFfOzwmcGH5s=; b=yogmS/vvYMsWMr aPrK6SjxBY/+DTvQxH9dC+DsJi5bpR+DoQC8EOY/IiXsfN+aP/LA2B6uSVU42Ry6iZT1lINQXmCCe URvxZNeCSuo/Zjp5/3JSOV4gkTrrztyEGp+LySsLsfLdF7U44l8glP99DBpUzIuunz8XbTvMQy62c gA4knwNG1vxheIvEu1Hs02D6Vhqbv7p/rmx+SXjRHVfZrDdM+BSN+pJk/iu/H1sFp/I9fNehmCENY FJ178Yd61XxYVFpCtAuGMg/Lbe4LRcFstLq9gIHMJ51lkZ3/GLcLP2+Yqp3OCjIxYtzVhZztzLxIQ FG1LLCNUHL4vu8gkWaXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Llx-0000000BU5y-1t7G; Tue, 07 May 2024 14:26:25 +0000 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Llt-0000000BU3H-1M4T for linux-riscv@lists.infradead.org; Tue, 07 May 2024 14:26:24 +0000 Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-6f490b5c23bso720186b3a.3 for ; Tue, 07 May 2024 07:26:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091981; x=1715696781; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=MTOLBzQvBvlY+EmrQy97Trzu0yuGzLHkqXUAWYArG80=; b=jp/guTKJ5sUJEeKC9LxF3KPL0617o/9UeOiBP8/c9wAorcP0S9cuiUmM4p7fsC4wXy nkEyPp4Rjcim8wdDlkuGhVmzQevnfd6c39Abg/8aZyyD0BPVkdY4THiRMRMqQB9hgrC0 JJmBmW6S6+Fxdo00mijZyM+AkKqji7h1i3zLgUya/9XHQsrfJciAaZkTehu6mQ4/3tT9 F1IrBBOUJHGPgQUiQSclK5CdoBYGKKBgEk/FJYZ9L6R1ZveEr3G9fKbMnHHbE1LeLkK1 IV/E3UovFs5iccuf9orYY0lYfYo6ZHbbfTejnGDW26YiN0Hp6686VWIq5J9isXr/lKu0 V/gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091981; x=1715696781; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MTOLBzQvBvlY+EmrQy97Trzu0yuGzLHkqXUAWYArG80=; b=c7ME0zZ8Ec0tm5GKqLbTbUUnS+L3iYg9YbIBIMUCO3rttuJ5gU5b/FU1gL6EIK/SmV wugZ33wMEuke3epHfxLTCx+jcmwMyLhcftHc2VUOOY5FvPWh/QMsdxBeFyXjOpNXNqxG YyYTuuIYENiJ4DjtMgTFmWpmQ895gu5JVhX+9d2w7EM/hJ21/xN8QYluBZ3QdKIiN8QI zrOIclwkhwfWgjYp3F/H+G1z16kXu8J+fYe/lqBkb5j6mtwOtQqGUMARiEw5gSq4w9kI OGT3fYijajEUVIAZaQagWqlFGFkvS03+34hVsoysb2rFHt1IzG7GgGp/x8iZ7JoibW19 OIYA== X-Forwarded-Encrypted: i=1; AJvYcCWXHk4gKeDUiU1/uRfEPttO+14iOwzA2Lzpq/cwAGjY6rKzNP0Hmzzzxt+v1tUbVjXFbQ5gx2msfKzkYTNeF4vN32ivUkcYy686jrDpHpwS X-Gm-Message-State: AOJu0YzpfNv5NtXtjBtyjutBSy9L5fEUoyXkG7Z6/D+ppTwk3NZhuM9T mHsy9sg7/R3ZqJwNn9nQZuUT8bmLvLrGPjJBc2r7ZCC07p7fxxxHKlRNf+SpI/0= X-Google-Smtp-Source: AGHT+IFrRpCDjD/4eOupdDn8MUYNlbPGsiI9chLNqLmFjLhbFaZkNfyNZ758Eh2Q2HfbLwdvKcrXqw== X-Received: by 2002:a05:6a00:1305:b0:6ed:88e5:53d4 with SMTP id j5-20020a056a00130500b006ed88e553d4mr14682261pfu.8.1715091979690; Tue, 07 May 2024 07:26:19 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:19 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 5/6] iommu/riscv: support nested iommu for creating domains owned by userspace Date: Tue, 7 May 2024 22:25:59 +0800 Message-Id: <20240507142600.23844-6-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240507_072622_018076_8159F0E5 X-CRM114-Status: GOOD ( 24.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch implements .domain_alloc_user operation for creating domains owend by userspace, e.g. through IOMMUFD. Add s2 domain for parent domain for second stage, s1 domain will be the first stage. Don't remove IOMMU private data of dev in blocked domain, because it holds the user data of device, which is used when attaching device into s1 domain. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu.c | 227 ++++++++++++++++++++++++++++++++++- include/uapi/linux/iommufd.h | 17 +++ 2 files changed, 243 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 072251f6ad85..7eda850df475 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -827,6 +827,7 @@ static int riscv_iommu_iodir_set_mode(struct riscv_iommu_device *iommu, /* This struct contains protection domain specific IOMMU driver data. */ struct riscv_iommu_domain { + struct riscv_iommu_domain *s2; struct iommu_domain domain; struct list_head bonds; spinlock_t lock; /* protect bonds list updates. */ @@ -844,6 +845,7 @@ struct riscv_iommu_domain { /* Private IOMMU data for managed devices, dev_iommu_priv_* */ struct riscv_iommu_info { struct riscv_iommu_domain *domain; + struct riscv_iommu_dc dc_user; }; /* Linkage between an iommu_domain and attached devices. */ @@ -1454,7 +1456,6 @@ static int riscv_iommu_attach_blocking_domain(struct iommu_domain *iommu_domain, riscv_iommu_iodir_update(iommu, dev, RISCV_IOMMU_FSC_BARE, 0, 0); riscv_iommu_bond_unlink(info->domain, dev); - info->domain = NULL; return 0; } @@ -1486,6 +1487,229 @@ static struct iommu_domain riscv_iommu_identity_domain = { } }; +/** + * Nested IOMMU operations + */ + +static int riscv_iommu_attach_dev_nested(struct iommu_domain *domain, struct device *dev) +{ + struct riscv_iommu_domain *riscv_domain = iommu_domain_to_riscv(domain); + struct riscv_iommu_device *iommu = dev_to_iommu(dev); + struct riscv_iommu_info *info = dev_iommu_priv_get(dev); + + if (riscv_domain->numa_node == NUMA_NO_NODE) + riscv_domain->numa_node = dev_to_node(iommu->dev); + + riscv_iommu_bond_unlink(info->domain, dev); + + if (riscv_iommu_bond_link(riscv_domain, dev)) + return -ENOMEM; + + riscv_iommu_iotlb_inval(riscv_domain, 0, ULONG_MAX); + + riscv_iommu_iodir_update(iommu, dev, info->dc_user.fsc, info->dc_user.ta, + info->dc_user.iohgatp); + + info->domain = riscv_domain; + + return 0; +} + +static void riscv_iommu_domain_free_nested(struct iommu_domain *domain) +{ + struct riscv_iommu_domain *riscv_domain = iommu_domain_to_riscv(domain); + + kfree(riscv_domain); +} + +static const struct iommu_domain_ops riscv_iommu_nested_domain_ops = { + .attach_dev = riscv_iommu_attach_dev_nested, + .free = riscv_iommu_domain_free_nested, +}; + +static int +riscv_iommu_get_dc_user(struct device *dev, struct iommu_hwpt_riscv_iommu *user_arg) +{ + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); + struct riscv_iommu_device *iommu = dev_to_iommu(dev); + struct riscv_iommu_info *info = dev_iommu_priv_get(dev); + struct riscv_iommu_dc dc; + struct riscv_iommu_fq_record event; + u64 dc_len = sizeof(struct riscv_iommu_dc) >> (!(iommu->caps & RISCV_IOMMU_CAP_MSI_FLAT)); + u64 event_len = sizeof(struct riscv_iommu_fq_record); + void __user *event_user = NULL; + + for (int i = 0; i < fwspec->num_ids; i++) { + event.hdr = + FIELD_PREP(RISCV_IOMMU_FQ_HDR_CAUSE, RISCV_IOMMU_FQ_CAUSE_DDT_INVALID) | + FIELD_PREP(RISCV_IOMMU_FQ_HDR_DID, fwspec->ids[i]); + + /* Sanity check DC of stage-1 from user data */ + if (!user_arg->out_event_uptr || user_arg->event_len != event_len) + return -EINVAL; + + event_user = u64_to_user_ptr(user_arg->out_event_uptr); + + if (!user_arg->dc_uptr || user_arg->dc_len != dc_len) + return -EINVAL; + + if (copy_from_user(&dc, u64_to_user_ptr(user_arg->dc_uptr), dc_len)) + return -EFAULT; + + if (!(dc.tc & RISCV_IOMMU_DDTE_VALID)) { + dev_dbg(dev, "Invalid DDT from user data\n"); + if (copy_to_user(event_user, &event, event_len)) + return -EFAULT; + } + + if (!dc.fsc || dc.iohgatp) { + dev_dbg(dev, "Wrong page table from user data\n"); + if (copy_to_user(event_user, &event, event_len)) + return -EFAULT; + } + + /* Save DC of stage-1 from user data */ + memcpy(&info->dc_user, + riscv_iommu_get_dc(iommu, fwspec->ids[i]), + sizeof(struct riscv_iommu_dc)); + info->dc_user.fsc = dc.fsc; + } + + return 0; +} + +static struct iommu_domain * +riscv_iommu_domain_alloc_nested(struct device *dev, + struct iommu_domain *parent, + const struct iommu_user_data *user_data) +{ + struct riscv_iommu_domain *s2_domain = iommu_domain_to_riscv(parent); + struct riscv_iommu_domain *s1_domain; + struct riscv_iommu_device *iommu = dev_to_iommu(dev); + struct iommu_hwpt_riscv_iommu arg; + int ret, va_bits; + + if (user_data->type != IOMMU_HWPT_DATA_RISCV_IOMMU) + return ERR_PTR(-EOPNOTSUPP); + + if (parent->type != IOMMU_DOMAIN_UNMANAGED) + return ERR_PTR(-EINVAL); + + ret = iommu_copy_struct_from_user(&arg, + user_data, + IOMMU_HWPT_DATA_RISCV_IOMMU, + out_event_uptr); + if (ret) + return ERR_PTR(ret); + + s1_domain = kzalloc(sizeof(*s1_domain), GFP_KERNEL); + if (!s1_domain) + return ERR_PTR(-ENOMEM); + + spin_lock_init(&s1_domain->lock); + INIT_LIST_HEAD_RCU(&s1_domain->bonds); + + s1_domain->pscid = ida_alloc_range(&riscv_iommu_pscids, 1, + RISCV_IOMMU_MAX_PSCID, GFP_KERNEL); + if (s1_domain->pscid < 0) { + iommu_free_page(s1_domain->pgd_root); + kfree(s1_domain); + return ERR_PTR(-ENOMEM); + } + + /* Get device context of stage-1 from user*/ + ret = riscv_iommu_get_dc_user(dev, &arg); + if (ret) { + kfree(s1_domain); + return ERR_PTR(-EINVAL); + } + + if (!iommu) { + va_bits = VA_BITS; + } else if (iommu->caps & RISCV_IOMMU_CAP_S_SV57) { + va_bits = 57; + } else if (iommu->caps & RISCV_IOMMU_CAP_S_SV48) { + va_bits = 48; + } else if (iommu->caps & RISCV_IOMMU_CAP_S_SV39) { + va_bits = 39; + } else { + dev_err(dev, "cannot find supported page table mode\n"); + return ERR_PTR(-ENODEV); + } + + /* + * The ops->domain_alloc_user could be directly called by the iommufd core, + * instead of iommu core. So, this function need to set the default value of + * following data member: + * - domain->pgsize_bitmap + * - domain->geometry + * - domain->type + * - domain->ops + */ + s1_domain->s2 = s2_domain; + s1_domain->domain.type = IOMMU_DOMAIN_NESTED; + s1_domain->domain.ops = &riscv_iommu_nested_domain_ops; + s1_domain->domain.pgsize_bitmap = SZ_4K; + s1_domain->domain.geometry.aperture_start = 0; + s1_domain->domain.geometry.aperture_end = DMA_BIT_MASK(va_bits - 1); + s1_domain->domain.geometry.force_aperture = true; + + return &s1_domain->domain; +} + +static struct iommu_domain * +riscv_iommu_domain_alloc_user(struct device *dev, u32 flags, + struct iommu_domain *parent, + const struct iommu_user_data *user_data) +{ + struct iommu_domain *domain; + struct riscv_iommu_domain *riscv_domain; + + /* Allocate stage-1 domain if it has stage-2 parent domain */ + if (parent) + return riscv_iommu_domain_alloc_nested(dev, parent, user_data); + + if (flags & ~((IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING))) + return ERR_PTR(-EOPNOTSUPP); + + if (user_data) + return ERR_PTR(-EINVAL); + + /* domain_alloc_user op needs to be fully initialized */ + domain = iommu_domain_alloc(dev->bus); + if (!domain) + return ERR_PTR(-ENOMEM); + + /* + * We assume that nest-parent or g-stage only will come here + * TODO: Shadow page table doesn't be supported now. + * We currently can't distinguish g-stage and shadow + * page table here. Shadow page table shouldn't be + * put at stage-2. + */ + riscv_domain = iommu_domain_to_riscv(domain); + + /* pgd_root may be allocated in .domain_alloc_paging */ + if (riscv_domain->pgd_root) + iommu_free_page(riscv_domain->pgd_root); + + riscv_domain->pgd_root = iommu_alloc_pages_node(riscv_domain->numa_node, + GFP_KERNEL_ACCOUNT, + 2); + if (!riscv_domain->pgd_root) + return ERR_PTR(-ENOMEM); + + riscv_domain->gscid = ida_alloc_range(&riscv_iommu_gscids, 1, + RISCV_IOMMU_MAX_GSCID, GFP_KERNEL); + if (riscv_domain->gscid < 0) { + iommu_free_pages(riscv_domain->pgd_root, 2); + kfree(riscv_domain); + return ERR_PTR(-ENOMEM); + } + + return domain; +} + static void *riscv_iommu_hw_info(struct device *dev, u32 *length, u32 *type) { struct riscv_iommu_device *iommu = dev_to_iommu(dev); @@ -1587,6 +1811,7 @@ static const struct iommu_ops riscv_iommu_ops = { .blocked_domain = &riscv_iommu_blocking_domain, .release_domain = &riscv_iommu_blocking_domain, .domain_alloc_paging = riscv_iommu_alloc_paging_domain, + .domain_alloc_user = riscv_iommu_domain_alloc_user, .def_domain_type = riscv_iommu_device_domain_type, .device_group = riscv_iommu_device_group, .probe_device = riscv_iommu_probe_device, diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index ec9aafd7d373..e10b6e236647 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -390,14 +390,31 @@ struct iommu_hwpt_vtd_s1 { __u32 __reserved; }; +/** + * struct iommu_hwpt_riscv_iommu - RISCV IOMMU stage-1 device context table + * info (IOMMU_HWPT_TYPE_RISCV_IOMMU) + * @dc_len: Length of device context + * @dc_uptr: User pointer to the address of device context + * @event_len: Length of an event record + * @out_event_uptr: User pointer to the address of event record + */ +struct iommu_hwpt_riscv_iommu { + __aligned_u64 dc_len; + __aligned_u64 dc_uptr; + __aligned_u64 event_len; + __aligned_u64 out_event_uptr; +}; + /** * enum iommu_hwpt_data_type - IOMMU HWPT Data Type * @IOMMU_HWPT_DATA_NONE: no data * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table + * @IOMMU_HWPT_DATA_RISCV_IOMMU: RISC-V IOMMU device context table */ enum iommu_hwpt_data_type { IOMMU_HWPT_DATA_NONE, IOMMU_HWPT_DATA_VTD_S1, + IOMMU_HWPT_DATA_RISCV_IOMMU, }; /** From patchwork Tue May 7 14:26:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zong Li X-Patchwork-Id: 13657230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 303D1C10F1A for ; Tue, 7 May 2024 14:26:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SG4jI7fijG5ovzAs32gCFJDVrWmLcWDR2965j5ncCp8=; b=dNXkNVjAzCFA7b CkYYZlA44yuYN93rDrqPr/yyNbQ/cGhu47OQ7/maNj/g91BGlUbi2+AJn67Wr8YK7JrbT5I2dU6P0 o0hCNwK9d+H0mji61v25g+m9aFVqpl0SfpQtUnZUmExIQS0vVd3n+myIwNOjQFZ3WSk8heE9106/s buJf5Ddlto+nS6JDj2Ajd6QHpW9hoGvKc0Ntmyg490CyFbzIkCMBmNAfHcYjC6kzfEiVf6pJtgiH5 0YNdMZC0BKHtfgtDgok22dTNFWem8LjguaWycXUyZ8xImRm824jytlj6fzECek7+Zkzo11MdibHzM eoCLCKitHRAnUqeAsY2g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Lly-0000000BU6d-37fo; Tue, 07 May 2024 14:26:26 +0000 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4Llu-0000000BU3k-2OWV for linux-riscv@lists.infradead.org; Tue, 07 May 2024 14:26:24 +0000 Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-6ee13f19e7eso2645043b3a.1 for ; Tue, 07 May 2024 07:26:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1715091982; x=1715696782; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=QsG7aE0HnxZsvtjvosuXtYKV2huEonpRW6TKXMCUJnU=; b=HRvf6AQsQq4KewOz+/Q5SEGPQuDzuxZwrGYla9SvOBogAmdtAGrqsryX9XakYRL0vF OiRcVyqhU7wZjIvX4lEU73oDMFkcFHt/FyJrgp3btI66dUS7lVVMrKZUdVfHPwK4T8PR zdAE/ip/ALlZCN0YBMAanmbtFd6TGRatXS40fHjxIo4Id/d8HKAYGWSCQG0FOsKqWE8N fVkKNUbDPm8uNn2OloHB6UBzuAqba6OsvoYUleqSBf7CVKtoojk/eHScu/0WqIjjRwPv JLddudmG7PSOUqbQJ4I944qjKlZbJA7hf1WbL3KsA0d/0Pgsq5+fHNZakSHVEJRZyt++ 3dGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715091982; x=1715696782; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QsG7aE0HnxZsvtjvosuXtYKV2huEonpRW6TKXMCUJnU=; b=tQ0GNt0+HSvGpyT0rKY/id+WeuZJVH8pV5eMj0uCsrFMW5LzFvgLGizyy4v3HhtVTO rSGplaqBo9QLlJtX4pTaXrc2h2/icpMGJzZH7i6mzAm/vEVGx7XxijcIIgpTf2XZj28a UmmJUEowQsqC+GgOrZ5aDYPkJmdCeqg2KuYZPFpxg+AwEWYhBlrWrxgpB7dMsvlfQmGv G844CvhbxeeHVNAacxAJ+EE/4OFnBpXr/Q8CDML1A5BLDotg+DdOAIPCHhCCzgn0pd1f F+HElQvbf5kAWyjHfVi4IBPUvkoUNuxvtOND3thaIedZzmxnWkYhdOzbYJ19fr6toJRh QN1A== X-Forwarded-Encrypted: i=1; AJvYcCUB5tub0Rm6+67Zl79nlNCBCAFreEpUEzPN3eNurPkDzYzf1l8KNzHP0tnODvmB9bYxN8zTfkjHZYmZ0p4H4420YVb5vl5CMea+G4SbXSmb X-Gm-Message-State: AOJu0YxS6ZwhnIAjxBKAK5VkTlDozTVrscvkqJ/4dBDZtCGAJ0e4hbnX 6NSxv6Ga4Ntt9/fBScp9w1a0AkOyss9k/H8W+qw2Kv35NPU7yn4ChuUx4yx/20I= X-Google-Smtp-Source: AGHT+IESwO+1TpOehpzE3oOZnwENdjrLViV8PyyzyYTAIHr0YOeHnmv3rn25FWxqq/8kskMF3gIH5A== X-Received: by 2002:a05:6a00:17a7:b0:6ec:e726:b6f5 with SMTP id s39-20020a056a0017a700b006ece726b6f5mr16072960pfg.26.1715091982078; Tue, 07 May 2024 07:26:22 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id i22-20020aa79096000000b006f44bcbe7e3sm7687554pfa.201.2024.05.07.07.26.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 May 2024 07:26:21 -0700 (PDT) From: Zong Li To: joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, tjeznach@rivosinc.com, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, jgg@ziepe.ca, kevin.tian@intel.com, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org Cc: Zong Li Subject: [PATCH RFC RESEND 6/6] iommu/riscv: support nested iommu for flushing cache Date: Tue, 7 May 2024 22:26:00 +0800 Message-Id: <20240507142600.23844-7-zong.li@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240507142600.23844-1-zong.li@sifive.com> References: <20240507142600.23844-1-zong.li@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240507_072622_762233_4D5126D8 X-CRM114-Status: GOOD ( 15.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch implements cache_invalidate_user operation for the userspace to flush the hardware caches for a nested domain through iommufd. Signed-off-by: Zong Li --- drivers/iommu/riscv/iommu.c | 91 ++++++++++++++++++++++++++++++++++++ include/uapi/linux/iommufd.h | 9 ++++ 2 files changed, 100 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 7eda850df475..4dd58fe2242d 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -1522,9 +1522,100 @@ static void riscv_iommu_domain_free_nested(struct iommu_domain *domain) kfree(riscv_domain); } +static int riscv_iommu_fix_user_cmd(struct riscv_iommu_command *cmd, + unsigned int pscid, unsigned int gscid) +{ + u32 opcode = FIELD_GET(RISCV_IOMMU_CMD_OPCODE, cmd->dword0); + + switch (opcode) { + case RISCV_IOMMU_CMD_IOTINVAL_OPCODE: + u32 func = FIELD_GET(RISCV_IOMMU_CMD_FUNC, cmd->dword0); + + if (func != RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA && + func != RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA) { + pr_warn("The IOTINVAL function: 0x%x is not supported\n", + func); + return -EOPNOTSUPP; + } + + if (func == RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA) { + cmd->dword0 &= ~RISCV_IOMMU_CMD_FUNC; + cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_FUNC, + RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA); + } + + cmd->dword0 &= ~(RISCV_IOMMU_CMD_IOTINVAL_PSCID | + RISCV_IOMMU_CMD_IOTINVAL_GSCID); + riscv_iommu_cmd_inval_set_pscid(cmd, pscid); + riscv_iommu_cmd_inval_set_gscid(cmd, gscid); + break; + case RISCV_IOMMU_CMD_IODIR_OPCODE: + /* + * Ensure the device ID is right. We expect that VMM has + * transferred the device ID to host's from guest's. + */ + break; + default: + pr_warn("The user command: 0x%x is not supported\n", opcode); + return -EOPNOTSUPP; + } + + return 0; +} + +static int riscv_iommu_cache_invalidate_user(struct iommu_domain *domain, + struct iommu_user_data_array *array) +{ + struct riscv_iommu_domain *riscv_domain = iommu_domain_to_riscv(domain); + struct riscv_iommu_device *iommu; + struct riscv_iommu_bond *bond; + struct riscv_iommu_command cmd; + struct iommu_hwpt_riscv_iommu_invalidate inv_info; + int ret, index; + + if (!riscv_domain) + return -EINVAL; + + /* Assume attached devices in the domain go through the same IOMMU device */ + spin_lock(&riscv_domain->lock); + list_for_each_entry_rcu(bond, &riscv_domain->bonds, list) { + if (bond->dev) { + iommu = dev_to_iommu(bond->dev); + break; + } + } + spin_unlock(&riscv_domain->lock); + + if (!iommu) + return -EINVAL; + + for (index = 0; index < array->entry_num; index++) { + ret = iommu_copy_struct_from_user_array(&inv_info, array, + IOMMU_HWPT_DATA_RISCV_IOMMU, + index, cmd); + if (ret) + break; + + ret = riscv_iommu_fix_user_cmd((struct riscv_iommu_command *)inv_info.cmd, + riscv_domain->pscid, + riscv_domain->s2->gscid); + if (ret == -EOPNOTSUPP) + continue; + + riscv_iommu_cmd_send(iommu, (struct riscv_iommu_command *)inv_info.cmd, 0); + riscv_iommu_cmd_iofence(&cmd); + riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_QUEUE_TIMEOUT); + } + + array->entry_num = index; + + return ret; +} + static const struct iommu_domain_ops riscv_iommu_nested_domain_ops = { .attach_dev = riscv_iommu_attach_dev_nested, .free = riscv_iommu_domain_free_nested, + .cache_invalidate_user = riscv_iommu_cache_invalidate_user, }; static int diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index e10b6e236647..d93a8f11813d 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -689,6 +689,15 @@ struct iommu_hwpt_vtd_s1_invalidate { __u32 __reserved; }; +/** + * struct iommu_hwpt_riscv_iommu_invalidate - RISCV IOMMU cache invalidation + * (IOMMU_HWPT_TYPE_RISCV_IOMMU) + * @cmd: An array holds a command for cache invalidation + */ +struct iommu_hwpt_riscv_iommu_invalidate { + __aligned_u64 cmd[2]; +}; + /** * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE) * @size: sizeof(struct iommu_hwpt_invalidate)