| Message ID | 20240429152207.479221-1-james.clark@arm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | coresight: Use per-sink trace ID maps for Perf sessions | expand |
Hi James On Mon, 29 Apr 2024 at 16:23, James Clark <james.clark@arm.com> wrote: > > This will allow sessions with more than CORESIGHT_TRACE_IDS_MAX ETMs > as long as there are fewer than that many ETMs connected to each sink. > > Each sink owns its own trace ID map, and any Perf session connecting to > that sink will allocate from it, even if the sink is currently in use by > other users. This is similar to the existing behavior where the dynamic > trace IDs are constant as long as there is any concurrent Perf session > active. It's not completely optimal because slightly more IDs will be > used than necessary, but the optimal solution involves tracking the PIDs > of each session and allocating ID maps based on the session owner. This > is difficult to do with the combination of per-thread and per-cpu modes > and some scheduling issues. The complexity of this isn't likely to worth > it because even with multiple users they'd just see a difference in the > ordering of ID allocations rather than hitting any limits (unless the > hardware does have too many ETMs connected to one sink). > > Per-thread mode works but only until there are any overlapping IDs, at > which point Perf will error out. Both per-thread mode and sysfs mode are > left to future changes, but both can be added on top of this initial > implementation and only sysfs mode requires further driver changes. > > The HW_ID version field hasn't been bumped in order to not break Perf > which already has an error condition for other values of that field. > Instead a new minor version has been added which signifies that there > are new fields but the old fields are backwards compatible. > Looking at this overall - would it not be better to introduce the concept of a "sink ID" to allow the detection of multiple sources into the single sink that is now done by emitting multiple AUX_HWID packets with the CPU+ID extra data? This sink ID could be part of the sink csdev struct - or even the id_map struct - a simple count of sinks as the per sink maps are created would be sufficient. If this sink ID replaced the CPU+ID extra data in the HWID packets, then each packet could be emitted just once, and perf can then collate based on the sink id. Moreover, once we are ready to address the per-thread issues - then the overlap would not matter. Generate OpenCSD decode trees per sink ID, add docoders to the tree per Trace ID. Thus if a buffer has data from sink 1 trace id 5, ans sink 2, trace ID 5, then pick the right decoder for the combo. Finally in systems with ETE+TRBE were there is no use of trace IDs, a sink ID of 0x0 could potentially indicate that 1:1 relationship. Regards Mike > > James Clark (17): > perf cs-etm: Print error for new PERF_RECORD_AUX_OUTPUT_HW_ID versions > perf auxtrace: Allow number of queues to be specified > perf: cs-etm: Create decoders after both AUX and HW_ID search passes > perf: cs-etm: Allocate queues for all CPUs > perf: cs-etm: Move traceid_list to each queue > perf: cs-etm: Create decoders based on the trace ID mappings > perf: cs-etm: Support version 0.1 of HW_ID packets > coresight: Remove unused stubs > coresight: Clarify comments around the PID of the sink owner > coresight: Move struct coresight_trace_id_map to common header > coresight: Expose map argument in trace ID API > coresight: Make CPU id map a property of a trace ID map > coresight: Pass trace ID map into source enable > coresight: Use per-sink trace ID maps for Perf sessions > coresight: Remove pending trace ID release mechanism > coresight: Re-emit trace IDs when the sink changes in per-thread mode > coresight: Emit HW_IDs for all ETMs that are using the sink > > drivers/hwtracing/coresight/coresight-core.c | 10 + > drivers/hwtracing/coresight/coresight-dummy.c | 3 +- > .../hwtracing/coresight/coresight-etm-perf.c | 82 ++- > .../hwtracing/coresight/coresight-etm-perf.h | 20 +- > .../coresight/coresight-etm3x-core.c | 14 +- > .../coresight/coresight-etm4x-core.c | 14 +- > drivers/hwtracing/coresight/coresight-stm.c | 3 +- > drivers/hwtracing/coresight/coresight-sysfs.c | 3 +- > .../hwtracing/coresight/coresight-tmc-etr.c | 5 +- > drivers/hwtracing/coresight/coresight-tmc.h | 5 +- > drivers/hwtracing/coresight/coresight-tpdm.c | 3 +- > .../hwtracing/coresight/coresight-trace-id.c | 107 +-- > .../hwtracing/coresight/coresight-trace-id.h | 57 +- > include/linux/coresight-pmu.h | 17 +- > include/linux/coresight.h | 20 +- > tools/include/linux/coresight-pmu.h | 17 +- > tools/perf/util/auxtrace.c | 9 +- > tools/perf/util/auxtrace.h | 1 + > .../perf/util/cs-etm-decoder/cs-etm-decoder.c | 28 +- > tools/perf/util/cs-etm.c | 617 ++++++++++++------ > tools/perf/util/cs-etm.h | 2 +- > 21 files changed, 633 insertions(+), 404 deletions(-) > > -- > 2.34.1 >
On Mon, Apr 29, 2024 at 04:21:45PM +0100, James Clark wrote: > This will allow sessions with more than CORESIGHT_TRACE_IDS_MAX ETMs > as long as there are fewer than that many ETMs connected to each sink. > > Each sink owns its own trace ID map, and any Perf session connecting to > that sink will allocate from it, even if the sink is currently in use by > other users. This is similar to the existing behavior where the dynamic > trace IDs are constant as long as there is any concurrent Perf session > active. It's not completely optimal because slightly more IDs will be > used than necessary, but the optimal solution involves tracking the PIDs > of each session and allocating ID maps based on the session owner. This > is difficult to do with the combination of per-thread and per-cpu modes > and some scheduling issues. The complexity of this isn't likely to worth > it because even with multiple users they'd just see a difference in the > ordering of ID allocations rather than hitting any limits (unless the > hardware does have too many ETMs connected to one sink). > > Per-thread mode works but only until there are any overlapping IDs, at > which point Perf will error out. Both per-thread mode and sysfs mode are > left to future changes, but both can be added on top of this initial > implementation and only sysfs mode requires further driver changes. > > The HW_ID version field hasn't been bumped in order to not break Perf > which already has an error condition for other values of that field. > Instead a new minor version has been added which signifies that there > are new fields but the old fields are backwards compatible. I guess I can pick the tooling part now, right? Further reviewing would be nice tho. - Arnaldo
On 03/05/2024 21:23, Arnaldo Carvalho de Melo wrote: > On Mon, Apr 29, 2024 at 04:21:45PM +0100, James Clark wrote: >> This will allow sessions with more than CORESIGHT_TRACE_IDS_MAX ETMs >> as long as there are fewer than that many ETMs connected to each sink. >> >> Each sink owns its own trace ID map, and any Perf session connecting to >> that sink will allocate from it, even if the sink is currently in use by >> other users. This is similar to the existing behavior where the dynamic >> trace IDs are constant as long as there is any concurrent Perf session >> active. It's not completely optimal because slightly more IDs will be >> used than necessary, but the optimal solution involves tracking the PIDs >> of each session and allocating ID maps based on the session owner. This >> is difficult to do with the combination of per-thread and per-cpu modes >> and some scheduling issues. The complexity of this isn't likely to worth >> it because even with multiple users they'd just see a difference in the >> ordering of ID allocations rather than hitting any limits (unless the >> hardware does have too many ETMs connected to one sink). >> >> Per-thread mode works but only until there are any overlapping IDs, at >> which point Perf will error out. Both per-thread mode and sysfs mode are >> left to future changes, but both can be added on top of this initial >> implementation and only sysfs mode requires further driver changes. >> >> The HW_ID version field hasn't been bumped in order to not break Perf >> which already has an error condition for other values of that field. >> Instead a new minor version has been added which signifies that there >> are new fields but the old fields are backwards compatible. > > I guess I can pick the tooling part now, right? Further reviewing would > be nice tho. > > - Arnaldo Is it ok if we wait for the driver changes to be merged first? There might some review comments which need a format change to the packets and then a re-write of the tool changes. You could take 1 and 2 though because they're unrelated. Thanks James
Hi James, On 29-04-2024 08:51 pm, James Clark wrote: > This will allow sessions with more than CORESIGHT_TRACE_IDS_MAX ETMs > as long as there are fewer than that many ETMs connected to each sink. > > Each sink owns its own trace ID map, and any Perf session connecting to > that sink will allocate from it, even if the sink is currently in use by > other users. This is similar to the existing behavior where the dynamic > trace IDs are constant as long as there is any concurrent Perf session > active. It's not completely optimal because slightly more IDs will be > used than necessary, but the optimal solution involves tracking the PIDs > of each session and allocating ID maps based on the session owner. This > is difficult to do with the combination of per-thread and per-cpu modes > and some scheduling issues. The complexity of this isn't likely to worth > it because even with multiple users they'd just see a difference in the > ordering of ID allocations rather than hitting any limits (unless the > hardware does have too many ETMs connected to one sink). > > Per-thread mode works but only until there are any overlapping IDs, at > which point Perf will error out. Both per-thread mode and sysfs mode are > left to future changes, but both can be added on top of this initial > implementation and only sysfs mode requires further driver changes. > > The HW_ID version field hasn't been bumped in order to not break Perf > which already has an error condition for other values of that field. > Instead a new minor version has been added which signifies that there > are new fields but the old fields are backwards compatible. > > > James Clark (17): > perf cs-etm: Print error for new PERF_RECORD_AUX_OUTPUT_HW_ID versions > perf auxtrace: Allow number of queues to be specified > perf: cs-etm: Create decoders after both AUX and HW_ID search passes > perf: cs-etm: Allocate queues for all CPUs > perf: cs-etm: Move traceid_list to each queue > perf: cs-etm: Create decoders based on the trace ID mappings > perf: cs-etm: Support version 0.1 of HW_ID packets > coresight: Remove unused stubs > coresight: Clarify comments around the PID of the sink owner > coresight: Move struct coresight_trace_id_map to common header > coresight: Expose map argument in trace ID API > coresight: Make CPU id map a property of a trace ID map > coresight: Pass trace ID map into source enable > coresight: Use per-sink trace ID maps for Perf sessions > coresight: Remove pending trace ID release mechanism > coresight: Re-emit trace IDs when the sink changes in per-thread mode > coresight: Emit HW_IDs for all ETMs that are using the sink > > drivers/hwtracing/coresight/coresight-core.c | 10 + > drivers/hwtracing/coresight/coresight-dummy.c | 3 +- > .../hwtracing/coresight/coresight-etm-perf.c | 82 ++- > .../hwtracing/coresight/coresight-etm-perf.h | 20 +- > .../coresight/coresight-etm3x-core.c | 14 +- > .../coresight/coresight-etm4x-core.c | 14 +- > drivers/hwtracing/coresight/coresight-stm.c | 3 +- > drivers/hwtracing/coresight/coresight-sysfs.c | 3 +- > .../hwtracing/coresight/coresight-tmc-etr.c | 5 +- > drivers/hwtracing/coresight/coresight-tmc.h | 5 +- > drivers/hwtracing/coresight/coresight-tpdm.c | 3 +- > .../hwtracing/coresight/coresight-trace-id.c | 107 +-- > .../hwtracing/coresight/coresight-trace-id.h | 57 +- > include/linux/coresight-pmu.h | 17 +- > include/linux/coresight.h | 20 +- > tools/include/linux/coresight-pmu.h | 17 +- > tools/perf/util/auxtrace.c | 9 +- > tools/perf/util/auxtrace.h | 1 + > .../perf/util/cs-etm-decoder/cs-etm-decoder.c | 28 +- > tools/perf/util/cs-etm.c | 617 ++++++++++++------ > tools/perf/util/cs-etm.h | 2 +- > 21 files changed, 633 insertions(+), 404 deletions(-) > Thanks for implementing trace-id mapping per sink, with that we could try perf (coresight tracing) for more than 100+ CPUs. Please feel free to add, Tested-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com> Thanks, Ganapat
On Tue, May 07, 2024 at 11:01:07AM +0100, James Clark wrote: > On 03/05/2024 21:23, Arnaldo Carvalho de Melo wrote: > > I guess I can pick the tooling part now, right? Further reviewing would > > be nice tho. > Is it ok if we wait for the driver changes to be merged first? There > might some review comments which need a format change to the packets and > then a re-write of the tool changes. Ok > You could take 1 and 2 though because they're unrelated. Done. - Arnaldo
On 03/05/2024 14:40, Mike Leach wrote: > Hi James > > On Mon, 29 Apr 2024 at 16:23, James Clark <james.clark@arm.com> wrote: >> >> This will allow sessions with more than CORESIGHT_TRACE_IDS_MAX ETMs >> as long as there are fewer than that many ETMs connected to each sink. >> >> Each sink owns its own trace ID map, and any Perf session connecting to >> that sink will allocate from it, even if the sink is currently in use by >> other users. This is similar to the existing behavior where the dynamic >> trace IDs are constant as long as there is any concurrent Perf session >> active. It's not completely optimal because slightly more IDs will be >> used than necessary, but the optimal solution involves tracking the PIDs >> of each session and allocating ID maps based on the session owner. This >> is difficult to do with the combination of per-thread and per-cpu modes >> and some scheduling issues. The complexity of this isn't likely to worth >> it because even with multiple users they'd just see a difference in the >> ordering of ID allocations rather than hitting any limits (unless the >> hardware does have too many ETMs connected to one sink). >> >> Per-thread mode works but only until there are any overlapping IDs, at >> which point Perf will error out. Both per-thread mode and sysfs mode are >> left to future changes, but both can be added on top of this initial >> implementation and only sysfs mode requires further driver changes. >> >> The HW_ID version field hasn't been bumped in order to not break Perf >> which already has an error condition for other values of that field. >> Instead a new minor version has been added which signifies that there >> are new fields but the old fields are backwards compatible. >> > > Looking at this overall - would it not be better to introduce the > concept of a "sink ID" to allow the detection of multiple sources into > the single sink that is now done by emitting multiple AUX_HWID packets > with the CPU+ID extra data? I think it's worth trying so I'll give it a go and see if it ends up being cleaner. One of the reasons for emitting multiple packets was to tie events and sinks together that may not have been created when the first set of HW_IDs were output. It seems like static sink IDs could solve this with less repetition. > This sink ID could be part of the sink csdev struct - or even the > id_map struct - a simple count of sinks as the per sink maps are > created would be sufficient. If this sink ID replaced the CPU+ID extra We could also use the existing sink IDs that are used to select the sink on the perf commandline: coresight_get_sink_by_id(). > data in the HWID packets, then each packet could be emitted just once, > and perf can then collate based on the sink id. > > Moreover, once we are ready to address the per-thread issues - then > the overlap would not matter. Generate OpenCSD decode trees per sink > ID, add docoders to the tree per Trace ID. Thus if a buffer has data > from sink 1 trace id 5, ans sink 2, trace ID 5, then pick the right > decoder for the combo. The issue is more about trace in a single buffer with overlapping IDs. Whether we use a sink ID or the CPU+ID mappings, Perf still needs to know when to change the tree. In either case we'd have to re-emit the mappings to prompt the decoder change. So yes with either solution there would be multiple decode trees, it's just knowing which fragments to send to which tree that's the issue. > > Finally in systems with ETE+TRBE were there is no use of trace IDs, a > sink ID of 0x0 could potentially indicate that 1:1 relationship. > This is already indicated with the PERF_AUX_FLAG_CORESIGHT_FORMAT_RAW flag on the AUX record. But I suppose we could add sink ID 0x0 as well for consistency with the new HW_ID packets. For backwards compatibility it probably makes sense to continue using the AUX flag in Perf, but I'll see how it looks with the sink ID implementation. James > Regards > > Mike > >> >> James Clark (17): >> perf cs-etm: Print error for new PERF_RECORD_AUX_OUTPUT_HW_ID versions >> perf auxtrace: Allow number of queues to be specified >> perf: cs-etm: Create decoders after both AUX and HW_ID search passes >> perf: cs-etm: Allocate queues for all CPUs >> perf: cs-etm: Move traceid_list to each queue >> perf: cs-etm: Create decoders based on the trace ID mappings >> perf: cs-etm: Support version 0.1 of HW_ID packets >> coresight: Remove unused stubs >> coresight: Clarify comments around the PID of the sink owner >> coresight: Move struct coresight_trace_id_map to common header >> coresight: Expose map argument in trace ID API >> coresight: Make CPU id map a property of a trace ID map >> coresight: Pass trace ID map into source enable >> coresight: Use per-sink trace ID maps for Perf sessions >> coresight: Remove pending trace ID release mechanism >> coresight: Re-emit trace IDs when the sink changes in per-thread mode >> coresight: Emit HW_IDs for all ETMs that are using the sink >> >> drivers/hwtracing/coresight/coresight-core.c | 10 + >> drivers/hwtracing/coresight/coresight-dummy.c | 3 +- >> .../hwtracing/coresight/coresight-etm-perf.c | 82 ++- >> .../hwtracing/coresight/coresight-etm-perf.h | 20 +- >> .../coresight/coresight-etm3x-core.c | 14 +- >> .../coresight/coresight-etm4x-core.c | 14 +- >> drivers/hwtracing/coresight/coresight-stm.c | 3 +- >> drivers/hwtracing/coresight/coresight-sysfs.c | 3 +- >> .../hwtracing/coresight/coresight-tmc-etr.c | 5 +- >> drivers/hwtracing/coresight/coresight-tmc.h | 5 +- >> drivers/hwtracing/coresight/coresight-tpdm.c | 3 +- >> .../hwtracing/coresight/coresight-trace-id.c | 107 +-- >> .../hwtracing/coresight/coresight-trace-id.h | 57 +- >> include/linux/coresight-pmu.h | 17 +- >> include/linux/coresight.h | 20 +- >> tools/include/linux/coresight-pmu.h | 17 +- >> tools/perf/util/auxtrace.c | 9 +- >> tools/perf/util/auxtrace.h | 1 + >> .../perf/util/cs-etm-decoder/cs-etm-decoder.c | 28 +- >> tools/perf/util/cs-etm.c | 617 ++++++++++++------ >> tools/perf/util/cs-etm.h | 2 +- >> 21 files changed, 633 insertions(+), 404 deletions(-) >> >> -- >> 2.34.1 >> > >