| Message ID | 20210518235830.133834-3-matthew.brost@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | GuC submission / DRM scheduler integration plan + new uAPI | expand |
On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: > Add entry fpr i915 new parallel submission uAPI plan. > > v2: > (Daniel Vetter): > - Expand logical order explaination > - Add dummy header > - Only allow N BBs in execbuf IOCTL > - Configure parallel submission per slot not per gem context > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > Cc: Tony Ye <tony.ye@intel.com> > CC: Carl Zhang <carl.zhang@intel.com> > Cc: Daniel Vetter <daniel.vetter@intel.com> > Cc: Jason Ekstrand <jason@jlekstrand.net> > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > --- > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > 2 files changed, 196 insertions(+), 1 deletion(-) > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > new file mode 100644 > index 000000000000..8c64b983ccad > --- /dev/null > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > @@ -0,0 +1,144 @@ > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > + > +/* > + * i915_context_engines_parallel_submit: > + * > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > + * hardware contexts are created internally in the i915 run these BBs. Once a > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > + * the slots configuration). > + * > + * Their are two currently defined ways to control the placement of the > + * hardware contexts on physical engines: default behavior (no flags) and > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > + * future as new hardware / use cases arise. Details of how to use this > + * interface below above the flags. > + * > + * Returns -EINVAL if hardware context placement configuration invalid or if the > + * placement configuration isn't supported on the platform / submission > + * interface. > + * Returns -ENODEV if extension isn't supported on the platform / submission > + * inteface. > + */ > +struct i915_context_engines_parallel_submit { > + struct i915_user_extension base; > + > + __u16 engine_index; /* slot for parallel engine */ > + __u16 width; /* number of contexts per parallel engine */ > + __u16 num_siblings; /* number of siblings per context */ > + __u16 mbz16; Ok the big picture looks reasonable now, the flags still confuse me. > +/* > + * Default placement behvavior (currently unsupported): > + * > + * Rather than restricting parallel submission to a single class with a > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > + * enables parallel submission across multiple engine classes. In this case each > + * context's logical engine mask indicates where that context can placed. It is > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > + * if one context is running CS0 no other contexts can run on CS0). > + * > + * Example 1 pseudo code: > + * CSX[Y] = engine class X, logical instance Y > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > + * > + * Results in the following valid placements: > + * CS0[0], CS1[0] > + * CS0[0], CS1[1] > + * CS0[1], CS1[0] > + * CS0[1], CS1[1] > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS0[0], CS0[1] > + * VE[1] = CS1[0], CS1[1] > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=3, > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[0], CS[2] > + * CS[1], CS[0] > + * CS[1], CS[2] > + * CS[2], CS[0] > + * CS[2], CS[1] > + * > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS[0], CS[1], CS[2] > + * VE[1] = CS[0], CS[1], CS[2] > + > + * This enables a use case where all engines are created equally, we don't care > + * where they are scheduled, we just want a certain number of resources, for > + * those resources to be scheduled in parallel, and possibly across multiple > + * engine classes. > + */ So I don't really get what this does compared to setting the flag below. Is this just about running the batchbuffers the wrong way round, i.e. if you have (simplest case) width=2, num_sibglings=1, engines=CS[0], CS[1] Then both CS[0], CS[1] and CS[1], CS[0] are possible options for running 2 batches? Iow, the backend is allowed to run the batchbuffers the wrong way round, which gains us nothing, since we assume the batches take equally long and engines interchangeable. There is no scheduling scenario where this additional flexibility can help. Also we don't have flags to select the only available and then specify an entire pipe dream about what the non-flag mode does, without an implementation. What is this about? If it's just "because bonded allowed this" then I think we should just unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but welp. > + > +/* > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > + * Each context must have the same number sibling and bonds are implictly create > + * of the siblings. > + * > + * All of the below examples are in logical space. > + * > + * Example 1 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=1, > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[2], CS[3] > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS[0], CS[2] > + * VE[1] = CS[1], CS[3] > + * > + * This enables a use case where all engines are not equal and certain placement > + * rules are required (i.e. split-frame requires all contexts to be placed in a > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > + * case (logically contiguous placement, within a single engine class) is > + * supported when using GuC submission. Execlist mode could support all possible > + * bonding configurations but currently doesn't support this extension. > + */ > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) Does this map to intel_context_set_nopreempt(), so we could implement it also in execlist? Also is this just an artifact of the implementation, or is this somehow required functionality for userspace, i.e. the workload fails if e.g. GuC decides to preempt all LRC of a parallel virtual engine? Of course just preempting one is a bit a bug (but execlist I think happily does that). Cheers, Daniel > +/* > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > + * points on all hardware contexts between each set of BBs. An example use case > + * of this feature is split-frame on gen11+ hardware. When using this feature a > + * BB must be submitted on each hardware context in the parallel gem context. > + * The execbuf2 IOCTL enforces the user adheres to policy. > + */ > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > + __u64 flags; /* all undefined flags must be zero */ > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > + > + /* > + * width (i) * num_siblings (j) in length > + * index = j + i * num_siblings > + */ > + struct i915_engine_class_instance engines[0]; > +} __attribute__ ((packed)); > + > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > index 7faa46cde088..64c539486ee4 100644 > --- a/Documentation/gpu/rfc/i915_scheduler.rst > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > New parallel submission uAPI > ============================ > -Details to come in a following patch. > +The existing bonding uAPI is completely broken with GuC submission because > +whether a submission is a single context submit or parallel submit isn't known > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > +contexts in parallel with the GuC the context must be explictly registered with > +N contexts and all N contexts must be submitted in a single command to the GuC. > +These interfaces doesn't support dynamically changing between N contexts as the > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > +the legacy bonding uAPI is quite confusing and not intuitive at all. > + > +The new parallel submission uAPI consists of 3 parts: > + > +* Export engines logical mapping > +* A 'set_parallel' extension to configure contexts for parallel > + submission > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > + > +Export engines logical mapping > +------------------------------ > +Certain use cases require BBs to be placed on engine instances in logical order > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > +logical mapping with the existing query engine info IOCTL. Also the GuC > +submission interface currently only supports submitting multiple contexts to > +engines in logical order which is a new requirement compared to execlists. > +Lastly, all current platforms have at most 2 instances and the logical order is > +the same a uABI order. This will change on platforms with more than 2 instances. > + > +A single bit will be added to drm_i915_engine_info.flags indicating that the > +logical instance has been returned and a new field, > +drm_i915_engine_info.logical_instance, returns the logical instance. > + > +A 'set_parallel' extension to configure contexts for parallel submission > +------------------------------------------------------------------------ > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > +It is setup step that should be called before using any of the contexts. See > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > +similar existing examples. Once a slot is configured for parallel submission the > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > +support GuC submission. Execlist support can be added later if needed. > + > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > + > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > +------------------------------------------------------------------- > +Contexts that have been configured with the 'set_parallel' extension are allowed > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > +objects in the drm_i915_gem_exec_object2 list or the first N if > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > +submitted and how it has been configured by 'set_parallel' or other extensions. > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > +the IOCTL. > -- > 2.28.0 > > _______________________________________________ > Intel-gfx mailing list > Intel-gfx@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Oh, yeah we call that gang submit on the AMD side. Had already some internal discussions how to implement this, but so far couldn't figure out how to cleanly introduce that into the DRM scheduler. Can you briefly describe in a few words how that is supposed to work on the Intel side? Thanks, Christian. Am 19.05.21 um 01:58 schrieb Matthew Brost: > Add entry fpr i915 new parallel submission uAPI plan. > > v2: > (Daniel Vetter): > - Expand logical order explaination > - Add dummy header > - Only allow N BBs in execbuf IOCTL > - Configure parallel submission per slot not per gem context > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > Cc: Tony Ye <tony.ye@intel.com> > CC: Carl Zhang <carl.zhang@intel.com> > Cc: Daniel Vetter <daniel.vetter@intel.com> > Cc: Jason Ekstrand <jason@jlekstrand.net> > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > --- > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > 2 files changed, 196 insertions(+), 1 deletion(-) > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > new file mode 100644 > index 000000000000..8c64b983ccad > --- /dev/null > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > @@ -0,0 +1,144 @@ > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > + > +/* > + * i915_context_engines_parallel_submit: > + * > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > + * hardware contexts are created internally in the i915 run these BBs. Once a > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > + * the slots configuration). > + * > + * Their are two currently defined ways to control the placement of the > + * hardware contexts on physical engines: default behavior (no flags) and > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > + * future as new hardware / use cases arise. Details of how to use this > + * interface below above the flags. > + * > + * Returns -EINVAL if hardware context placement configuration invalid or if the > + * placement configuration isn't supported on the platform / submission > + * interface. > + * Returns -ENODEV if extension isn't supported on the platform / submission > + * inteface. > + */ > +struct i915_context_engines_parallel_submit { > + struct i915_user_extension base; > + > + __u16 engine_index; /* slot for parallel engine */ > + __u16 width; /* number of contexts per parallel engine */ > + __u16 num_siblings; /* number of siblings per context */ > + __u16 mbz16; > +/* > + * Default placement behvavior (currently unsupported): > + * > + * Rather than restricting parallel submission to a single class with a > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > + * enables parallel submission across multiple engine classes. In this case each > + * context's logical engine mask indicates where that context can placed. It is > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > + * if one context is running CS0 no other contexts can run on CS0). > + * > + * Example 1 pseudo code: > + * CSX[Y] = engine class X, logical instance Y > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > + * > + * Results in the following valid placements: > + * CS0[0], CS1[0] > + * CS0[0], CS1[1] > + * CS0[1], CS1[0] > + * CS0[1], CS1[1] > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS0[0], CS0[1] > + * VE[1] = CS1[0], CS1[1] > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=3, > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[0], CS[2] > + * CS[1], CS[0] > + * CS[1], CS[2] > + * CS[2], CS[0] > + * CS[2], CS[1] > + * > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS[0], CS[1], CS[2] > + * VE[1] = CS[0], CS[1], CS[2] > + > + * This enables a use case where all engines are created equally, we don't care > + * where they are scheduled, we just want a certain number of resources, for > + * those resources to be scheduled in parallel, and possibly across multiple > + * engine classes. > + */ > + > +/* > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > + * Each context must have the same number sibling and bonds are implictly create > + * of the siblings. > + * > + * All of the below examples are in logical space. > + * > + * Example 1 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=1, > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[2], CS[3] > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS[0], CS[2] > + * VE[1] = CS[1], CS[3] > + * > + * This enables a use case where all engines are not equal and certain placement > + * rules are required (i.e. split-frame requires all contexts to be placed in a > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > + * case (logically contiguous placement, within a single engine class) is > + * supported when using GuC submission. Execlist mode could support all possible > + * bonding configurations but currently doesn't support this extension. > + */ > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > +/* > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > + * points on all hardware contexts between each set of BBs. An example use case > + * of this feature is split-frame on gen11+ hardware. When using this feature a > + * BB must be submitted on each hardware context in the parallel gem context. > + * The execbuf2 IOCTL enforces the user adheres to policy. > + */ > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > + __u64 flags; /* all undefined flags must be zero */ > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > + > + /* > + * width (i) * num_siblings (j) in length > + * index = j + i * num_siblings > + */ > + struct i915_engine_class_instance engines[0]; > +} __attribute__ ((packed)); > + > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > index 7faa46cde088..64c539486ee4 100644 > --- a/Documentation/gpu/rfc/i915_scheduler.rst > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > New parallel submission uAPI > ============================ > -Details to come in a following patch. > +The existing bonding uAPI is completely broken with GuC submission because > +whether a submission is a single context submit or parallel submit isn't known > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > +contexts in parallel with the GuC the context must be explictly registered with > +N contexts and all N contexts must be submitted in a single command to the GuC. > +These interfaces doesn't support dynamically changing between N contexts as the > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > +the legacy bonding uAPI is quite confusing and not intuitive at all. > + > +The new parallel submission uAPI consists of 3 parts: > + > +* Export engines logical mapping > +* A 'set_parallel' extension to configure contexts for parallel > + submission > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > + > +Export engines logical mapping > +------------------------------ > +Certain use cases require BBs to be placed on engine instances in logical order > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > +logical mapping with the existing query engine info IOCTL. Also the GuC > +submission interface currently only supports submitting multiple contexts to > +engines in logical order which is a new requirement compared to execlists. > +Lastly, all current platforms have at most 2 instances and the logical order is > +the same a uABI order. This will change on platforms with more than 2 instances. > + > +A single bit will be added to drm_i915_engine_info.flags indicating that the > +logical instance has been returned and a new field, > +drm_i915_engine_info.logical_instance, returns the logical instance. > + > +A 'set_parallel' extension to configure contexts for parallel submission > +------------------------------------------------------------------------ > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > +It is setup step that should be called before using any of the contexts. See > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > +similar existing examples. Once a slot is configured for parallel submission the > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > +support GuC submission. Execlist support can be added later if needed. > + > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > + > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > +------------------------------------------------------------------- > +Contexts that have been configured with the 'set_parallel' extension are allowed > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > +objects in the drm_i915_gem_exec_object2 list or the first N if > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > +submitted and how it has been configured by 'set_parallel' or other extensions. > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > +the IOCTL.
śr., 19 maj 2021 o 01:41 Matthew Brost <matthew.brost@intel.com> napisał(a): > > Add entry fpr i915 new parallel submission uAPI plan. s/fpr/for/ > > v2: > (Daniel Vetter): > - Expand logical order explaination > - Add dummy header > - Only allow N BBs in execbuf IOCTL > - Configure parallel submission per slot not per gem context > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > Cc: Tony Ye <tony.ye@intel.com> > CC: Carl Zhang <carl.zhang@intel.com> > Cc: Daniel Vetter <daniel.vetter@intel.com> > Cc: Jason Ekstrand <jason@jlekstrand.net> > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > --- > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > 2 files changed, 196 insertions(+), 1 deletion(-) > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > new file mode 100644 > index 000000000000..8c64b983ccad > --- /dev/null > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > @@ -0,0 +1,144 @@ > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > + > +/* > + * i915_context_engines_parallel_submit: > + * > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > + * hardware contexts are created internally in the i915 run these BBs. Once a > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > + * the slots configuration). > + * > + * Their are two currently defined ways to control the placement of the s/Their/There/ > + * hardware contexts on physical engines: default behavior (no flags) and > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the IMPLICIT? This typo is repeated multiple times s/the in the/in the/ > + * future as new hardware / use cases arise. Details of how to use this > + * interface below above the flags. "below above"? :) > + * > + * Returns -EINVAL if hardware context placement configuration invalid or if the is invalid > + * placement configuration isn't supported on the platform / submission > + * interface. > + * Returns -ENODEV if extension isn't supported on the platform / submission > + * inteface. > + */ > +struct i915_context_engines_parallel_submit { > + struct i915_user_extension base; > + > + __u16 engine_index; /* slot for parallel engine */ > + __u16 width; /* number of contexts per parallel engine */ > + __u16 num_siblings; /* number of siblings per context */ > + __u16 mbz16; > +/* > + * Default placement behvavior (currently unsupported): behavior > + * > + * Rather than restricting parallel submission to a single class with a > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > + * enables parallel submission across multiple engine classes. In this case each > + * context's logical engine mask indicates where that context can placed. It is can be placed > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > + * if one context is running CS0 no other contexts can run on CS0). > + * > + * Example 1 pseudo code: > + * CSX[Y] = engine class X, logical instance Y > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > + * > + * Results in the following valid placements: > + * CS0[0], CS1[0] > + * CS0[0], CS1[1] > + * CS0[1], CS1[0] > + * CS0[1], CS1[1] > + * > + * This can also be though of as 2 virtual engines: thought (This typo repeats multiple times) > + * VE[0] = CS0[0], CS0[1] > + * VE[1] = CS1[0], CS1[1] > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=3, > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[0], CS[2] > + * CS[1], CS[0] > + * CS[1], CS[2] > + * CS[2], CS[0] > + * CS[2], CS[1] > + * > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS[0], CS[1], CS[2] > + * VE[1] = CS[0], CS[1], CS[2] > + > + * This enables a use case where all engines are created equally, we don't care > + * where they are scheduled, we just want a certain number of resources, for > + * those resources to be scheduled in parallel, and possibly across multiple > + * engine classes. > + */ > + > +/* > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > + * Each context must have the same number sibling and bonds are implictly create number of siblings > + * of the siblings. "bonds are implictly create of the siblings." -ENOPARSE (also typo in implicitly) > + * > + * All of the below examples are in logical space. > + * > + * Example 1 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=1, > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[2], CS[3] > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS[0], CS[2] > + * VE[1] = CS[1], CS[3] > + * > + * This enables a use case where all engines are not equal and certain placement > + * rules are required (i.e. split-frame requires all contexts to be placed in a > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > + * case (logically contiguous placement, within a single engine class) is > + * supported when using GuC submission. Execlist mode could support all possible > + * bonding configurations but currently doesn't support this extension. > + */ > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > +/* > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > + * points on all hardware contexts between each set of BBs. An example use case > + * of this feature is split-frame on gen11+ hardware. When using this feature a > + * BB must be submitted on each hardware context in the parallel gem context. > + * The execbuf2 IOCTL enforces the user adheres to policy. > + */ > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > + __u64 flags; /* all undefined flags must be zero */ > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > + > + /* > + * width (i) * num_siblings (j) in length > + * index = j + i * num_siblings > + */ > + struct i915_engine_class_instance engines[0]; > +} __attribute__ ((packed)); > + > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > index 7faa46cde088..64c539486ee4 100644 > --- a/Documentation/gpu/rfc/i915_scheduler.rst > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > New parallel submission uAPI > ============================ > -Details to come in a following patch. > +The existing bonding uAPI is completely broken with GuC submission because > +whether a submission is a single context submit or parallel submit isn't known > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > +contexts in parallel with the GuC the context must be explictly registered with explicitly > +N contexts and all N contexts must be submitted in a single command to the GuC. > +These interfaces doesn't support dynamically changing between N contexts as the "this interface" or "don't support", I'm not sure what "these" refers to... "changing" what? maybe "swapping"? > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > +the legacy bonding uAPI is quite confusing and not intuitive at all. > + > +The new parallel submission uAPI consists of 3 parts: > + > +* Export engines logical mapping > +* A 'set_parallel' extension to configure contexts for parallel > + submission > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > + > +Export engines logical mapping > +------------------------------ > +Certain use cases require BBs to be placed on engine instances in logical order > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > +logical mapping with the existing query engine info IOCTL. Also the GuC > +submission interface currently only supports submitting multiple contexts to > +engines in logical order which is a new requirement compared to execlists. > +Lastly, all current platforms have at most 2 instances and the logical order is 2 instances of what? > +the same a uABI order. This will change on platforms with more than 2 instances. probably s/a uABI/as uAPI/ (note: 2 typos) > + > +A single bit will be added to drm_i915_engine_info.flags indicating that the > +logical instance has been returned and a new field, > +drm_i915_engine_info.logical_instance, returns the logical instance. > + > +A 'set_parallel' extension to configure contexts for parallel submission > +------------------------------------------------------------------------ > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > +It is setup step that should be called before using any of the contexts. See > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > +similar existing examples. Once a slot is configured for parallel submission the > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only submitting > +support GuC submission. Execlist support can be added later if needed. > + > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > + > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > +------------------------------------------------------------------- > +Contexts that have been configured with the 'set_parallel' extension are allowed > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > +objects in the drm_i915_gem_exec_object2 list or the first N if > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > +submitted and how it has been configured by 'set_parallel' or other extensions. > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > +the IOCTL. worth mentioning where? :) > -- > 2.28.0
On 19/05/2021 12:10, Daniel Vetter wrote: > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: >> Add entry fpr i915 new parallel submission uAPI plan. >> >> v2: >> (Daniel Vetter): >> - Expand logical order explaination >> - Add dummy header >> - Only allow N BBs in execbuf IOCTL >> - Configure parallel submission per slot not per gem context >> >> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> >> Cc: Tony Ye <tony.ye@intel.com> >> CC: Carl Zhang <carl.zhang@intel.com> >> Cc: Daniel Vetter <daniel.vetter@intel.com> >> Cc: Jason Ekstrand <jason@jlekstrand.net> >> Signed-off-by: Matthew Brost <matthew.brost@intel.com> >> --- >> Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ >> Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- >> 2 files changed, 196 insertions(+), 1 deletion(-) >> create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h >> >> diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h >> new file mode 100644 >> index 000000000000..8c64b983ccad >> --- /dev/null >> +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h >> @@ -0,0 +1,144 @@ >> +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ >> + >> +/* >> + * i915_context_engines_parallel_submit: >> + * >> + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. >> + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple >> + * hardware contexts are created internally in the i915 run these BBs. Once a >> + * slot is configured for N BBs only N BBs can be submitted in each execbuf >> + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf >> + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on >> + * the slots configuration). >> + * >> + * Their are two currently defined ways to control the placement of the >> + * hardware contexts on physical engines: default behavior (no flags) and >> + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the >> + * future as new hardware / use cases arise. Details of how to use this >> + * interface below above the flags. >> + * >> + * Returns -EINVAL if hardware context placement configuration invalid or if the >> + * placement configuration isn't supported on the platform / submission >> + * interface. >> + * Returns -ENODEV if extension isn't supported on the platform / submission >> + * inteface. >> + */ >> +struct i915_context_engines_parallel_submit { >> + struct i915_user_extension base; >> + >> + __u16 engine_index; /* slot for parallel engine */ >> + __u16 width; /* number of contexts per parallel engine */ >> + __u16 num_siblings; /* number of siblings per context */ >> + __u16 mbz16; > > Ok the big picture looks reasonable now, the flags still confuse me. > >> +/* >> + * Default placement behvavior (currently unsupported): >> + * >> + * Rather than restricting parallel submission to a single class with a >> + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that >> + * enables parallel submission across multiple engine classes. In this case each >> + * context's logical engine mask indicates where that context can placed. It is >> + * implied in this mode that all contexts have mutual exclusive placement (e.g. >> + * if one context is running CS0 no other contexts can run on CS0). >> + * >> + * Example 1 pseudo code: >> + * CSX[Y] = engine class X, logical instance Y >> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >> + * set_engines(INVALID) >> + * set_parallel(engine_index=0, width=2, num_siblings=2, >> + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) >> + * >> + * Results in the following valid placements: >> + * CS0[0], CS1[0] >> + * CS0[0], CS1[1] >> + * CS0[1], CS1[0] >> + * CS0[1], CS1[1] >> + * >> + * This can also be though of as 2 virtual engines: >> + * VE[0] = CS0[0], CS0[1] >> + * VE[1] = CS1[0], CS1[1] >> + * >> + * Example 2 pseudo code: >> + * CS[X] = generic engine of same class, logical instance X >> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >> + * set_engines(INVALID) >> + * set_parallel(engine_index=0, width=2, num_siblings=3, >> + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) >> + * >> + * Results in the following valid placements: >> + * CS[0], CS[1] >> + * CS[0], CS[2] >> + * CS[1], CS[0] >> + * CS[1], CS[2] >> + * CS[2], CS[0] >> + * CS[2], CS[1] >> + * >> + * >> + * This can also be though of as 2 virtual engines: >> + * VE[0] = CS[0], CS[1], CS[2] >> + * VE[1] = CS[0], CS[1], CS[2] >> + >> + * This enables a use case where all engines are created equally, we don't care >> + * where they are scheduled, we just want a certain number of resources, for >> + * those resources to be scheduled in parallel, and possibly across multiple >> + * engine classes. >> + */ > > So I don't really get what this does compared to setting the flag below. > Is this just about running the batchbuffers the wrong way round, i.e. if > you have (simplest case) > > width=2, num_sibglings=1, engines=CS[0], CS[1] > > Then both > CS[0], CS[1] > and > CS[1], CS[0] > are possible options for running 2 batches? Iow, the backend is allowed to > run the batchbuffers the wrong way round, which gains us nothing, since we > assume the batches take equally long and engines interchangeable. There is > no scheduling scenario where this additional flexibility can help. > > Also we don't have flags to select the only available and then specify an > entire pipe dream about what the non-flag mode does, without an > implementation. What is this about? > > If it's just "because bonded allowed this" then I think we should just > unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but > welp. At the risk of repeating some of the information: Flexibility helps to avoid hardcoding the current limitations of media fixed function pipe into the uapi. Today's media: Run on N consecutive instances. Tomorrow's media: Run on N consecutive logical instances. (Note that the engine addressing we expose today is class:instance based which is not this new logical instance name space. It's something else. Point of engine discovery and flexible bond configuration was to punt these questions to userspace.) Tomorrow's compute: Run on any N instances. Tomorrow's + 1 media: Who knows. Consider the scheduling advantage when the "consecutiveness" is not hardcoded in the uapi: cs0: busy with 1 wide context cs1: idle cs2: busy with 1 wide context cs3: idle 2-wide job with hardcoded consecutive engines now can't run without scheduler needing to re-balance. 2-wide job with any-2 can run fine. (Also keep in mind that all 1-wide, 2-wide and 4-wide contexts can fight for four physical engines.) This was an argument for not hardcoding "consecutiveness". Then if we think about cs0 + cs1 vs cs1 + cs0 ordering inversion, for today's media there is indeed no benefit to allow the inversion (and it cannot be allowed). But we can expand the argument into parallel submissions which absolutely have to run in parallel otherwise the GPU hangs, or at least progress stalls, versus the ones which benefit from running in parallel but do not strictly depend on it. In the latter case allowing inversions of the set is a benefit. Scheduler can now put partial jobs on the GPU without having to idle, or wait for idle, on the other engines. Overall, bonds weren't the mistake, as much I am sure there is a nice camp of people who hate them pasionately. (Hey I hate relocations because I can never remember what is offset and what is delta. Or maybe that is just because I don't use them enough.) Yes they can be hard to understand at first but they were not the mistake. Mistake was the disconnect between atomic parallel submission and separate execbufs tied with a submit fence which then takes double functionality. Anyway, all that we need to do today and in the future can be expressed with bonds. All that was needed for the GuC backend is to reject the bond configuration GuC firmware cannot do. Cannot do today. Can't stress this "today" enough. Because we almost got an update of the GuC firmware already last year or so which would have broken this assumption for compute and made it more flexible. Which is why I was pushing to make the set_parallel extension more flexible than just today's media use cases. Or you may end up adding more uapi year after year. Why if bonds can do it all? Why even if bonds are not even needed for generic parallel submit in proposed GuC multi-LRC changes for compute? They are only needed for media. So all the new uapi needed to add is a proper atomic parallel submit. Literally my proposal was just to add a single flag of how wide the context is plus multi-bb execbuf extension and that's all that would be needed. There is a document somewhere which lists all possible usecases that I suggest is looked at if the uapi is getting re-designed. Regards, Tvrtko >> + >> +/* >> + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. >> + * Each context must have the same number sibling and bonds are implictly create >> + * of the siblings. >> + * >> + * All of the below examples are in logical space. >> + * >> + * Example 1 pseudo code: >> + * CS[X] = generic engine of same class, logical instance X >> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >> + * set_engines(INVALID) >> + * set_parallel(engine_index=0, width=2, num_siblings=1, >> + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) >> + * >> + * Results in the following valid placements: >> + * CS[0], CS[1] >> + * >> + * Example 2 pseudo code: >> + * CS[X] = generic engine of same class, logical instance X >> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >> + * set_engines(INVALID) >> + * set_parallel(engine_index=0, width=2, num_siblings=2, >> + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) >> + * >> + * Results in the following valid placements: >> + * CS[0], CS[1] >> + * CS[2], CS[3] >> + * >> + * This can also be though of as 2 virtual engines: >> + * VE[0] = CS[0], CS[2] >> + * VE[1] = CS[1], CS[3] >> + * >> + * This enables a use case where all engines are not equal and certain placement >> + * rules are required (i.e. split-frame requires all contexts to be placed in a >> + * logically contiguous order on the VCS engines on gen11+ platforms). This use >> + * case (logically contiguous placement, within a single engine class) is >> + * supported when using GuC submission. Execlist mode could support all possible >> + * bonding configurations but currently doesn't support this extension. >> + */ >> +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > Does this map to intel_context_set_nopreempt(), so we could implement it > also in execlist? > > Also is this just an artifact of the implementation, or is this somehow > required functionality for userspace, i.e. the workload fails if e.g. GuC > decides to preempt all LRC of a parallel virtual engine? Of course just > preempting one is a bit a bug (but execlist I think happily does that). > > Cheers, Daniel > >> +/* >> + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption >> + * points on all hardware contexts between each set of BBs. An example use case >> + * of this feature is split-frame on gen11+ hardware. When using this feature a >> + * BB must be submitted on each hardware context in the parallel gem context. >> + * The execbuf2 IOCTL enforces the user adheres to policy. >> + */ >> +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) >> +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) >> + __u64 flags; /* all undefined flags must be zero */ >> + __u64 mbz64[3]; /* reserved for future use; must be zero */ >> + >> + /* >> + * width (i) * num_siblings (j) in length >> + * index = j + i * num_siblings >> + */ >> + struct i915_engine_class_instance engines[0]; >> +} __attribute__ ((packed)); >> + >> diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst >> index 7faa46cde088..64c539486ee4 100644 >> --- a/Documentation/gpu/rfc/i915_scheduler.rst >> +++ b/Documentation/gpu/rfc/i915_scheduler.rst >> @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit >> >> New parallel submission uAPI >> ============================ >> -Details to come in a following patch. >> +The existing bonding uAPI is completely broken with GuC submission because >> +whether a submission is a single context submit or parallel submit isn't known >> +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple >> +contexts in parallel with the GuC the context must be explictly registered with >> +N contexts and all N contexts must be submitted in a single command to the GuC. >> +These interfaces doesn't support dynamically changing between N contexts as the >> +bonding uAPI does. Hence the need for a new parallel submission interface. Also >> +the legacy bonding uAPI is quite confusing and not intuitive at all. >> + >> +The new parallel submission uAPI consists of 3 parts: >> + >> +* Export engines logical mapping >> +* A 'set_parallel' extension to configure contexts for parallel >> + submission >> +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL >> + >> +Export engines logical mapping >> +------------------------------ >> +Certain use cases require BBs to be placed on engine instances in logical order >> +(e.g. split-frame on gen11+). The logical mapping of engine instances can change >> +based on fusing. Rather than making UMDs be aware of fusing, simply expose the >> +logical mapping with the existing query engine info IOCTL. Also the GuC >> +submission interface currently only supports submitting multiple contexts to >> +engines in logical order which is a new requirement compared to execlists. >> +Lastly, all current platforms have at most 2 instances and the logical order is >> +the same a uABI order. This will change on platforms with more than 2 instances. >> + >> +A single bit will be added to drm_i915_engine_info.flags indicating that the >> +logical instance has been returned and a new field, >> +drm_i915_engine_info.logical_instance, returns the logical instance. >> + >> +A 'set_parallel' extension to configure contexts for parallel submission >> +------------------------------------------------------------------------ >> +The 'set_parallel' extension configures a slot for parallel submission of N BBs. >> +It is setup step that should be called before using any of the contexts. See >> +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for >> +similar existing examples. Once a slot is configured for parallel submission the >> +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only >> +support GuC submission. Execlist support can be added later if needed. >> + >> +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and >> +i915_context_engines_parallel_submit to the uAPI to implement this extension. >> + >> +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL >> +------------------------------------------------------------------- >> +Contexts that have been configured with the 'set_parallel' extension are allowed >> +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N >> +objects in the drm_i915_gem_exec_object2 list or the first N if >> +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot >> +submitted and how it has been configured by 'set_parallel' or other extensions. >> +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of >> +the IOCTL. >> -- >> 2.28.0 >> >> _______________________________________________ >> Intel-gfx mailing list >> Intel-gfx@lists.freedesktop.org >> https://lists.freedesktop.org/mailman/listinfo/intel-gfx >
On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote: > Oh, yeah we call that gang submit on the AMD side. > > Had already some internal discussions how to implement this, but so far > couldn't figure out how to cleanly introduce that into the DRM scheduler. > > Can you briefly describe in a few words how that is supposed to work on the > Intel side? > Sure, I've done a quick PoC internally and have been able to hook this into the DRM scheduler. Basically each BB still maps to a single job as each job is somewhat unique (e.g. each job has its own ring, lrc, seqno, etc...). However all the jobs configured to run in parallel map to a single sched_entity which maintains the order each job was generated from the execbuf IOCTL (1 - N). When the backend receives jobs 1 to N - 1 it basically just updates some internal state. When the backend sees job N (last job) it actually does the submit for jobs 1 - N which with GuC submission is a simple command moving the LRC tail of the N jobs. Daniel has suggested that we create a single job for the NN BBs but that would be huge rework to the internals of the i915 and likely won't happen by the time this code first lands. Also worth noting one way a job isn't really a treated individually is the excl slot with dma-resv. In that case we create a composite fence of all jobs (dma_fence_array). Matt > Thanks, > Christian. > > Am 19.05.21 um 01:58 schrieb Matthew Brost: > > Add entry fpr i915 new parallel submission uAPI plan. > > > > v2: > > (Daniel Vetter): > > - Expand logical order explaination > > - Add dummy header > > - Only allow N BBs in execbuf IOCTL > > - Configure parallel submission per slot not per gem context > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > Cc: Tony Ye <tony.ye@intel.com> > > CC: Carl Zhang <carl.zhang@intel.com> > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > --- > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > 2 files changed, 196 insertions(+), 1 deletion(-) > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > new file mode 100644 > > index 000000000000..8c64b983ccad > > --- /dev/null > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > @@ -0,0 +1,144 @@ > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > + > > +/* > > + * i915_context_engines_parallel_submit: > > + * > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > + * the slots configuration). > > + * > > + * Their are two currently defined ways to control the placement of the > > + * hardware contexts on physical engines: default behavior (no flags) and > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > + * future as new hardware / use cases arise. Details of how to use this > > + * interface below above the flags. > > + * > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > + * placement configuration isn't supported on the platform / submission > > + * interface. > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > + * inteface. > > + */ > > +struct i915_context_engines_parallel_submit { > > + struct i915_user_extension base; > > + > > + __u16 engine_index; /* slot for parallel engine */ > > + __u16 width; /* number of contexts per parallel engine */ > > + __u16 num_siblings; /* number of siblings per context */ > > + __u16 mbz16; > > +/* > > + * Default placement behvavior (currently unsupported): > > + * > > + * Rather than restricting parallel submission to a single class with a > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > + * enables parallel submission across multiple engine classes. In this case each > > + * context's logical engine mask indicates where that context can placed. It is > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > + * if one context is running CS0 no other contexts can run on CS0). > > + * > > + * Example 1 pseudo code: > > + * CSX[Y] = engine class X, logical instance Y > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > + * > > + * Results in the following valid placements: > > + * CS0[0], CS1[0] > > + * CS0[0], CS1[1] > > + * CS0[1], CS1[0] > > + * CS0[1], CS1[1] > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS0[0], CS0[1] > > + * VE[1] = CS1[0], CS1[1] > > + * > > + * Example 2 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * CS[0], CS[2] > > + * CS[1], CS[0] > > + * CS[1], CS[2] > > + * CS[2], CS[0] > > + * CS[2], CS[1] > > + * > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS[0], CS[1], CS[2] > > + * VE[1] = CS[0], CS[1], CS[2] > > + > > + * This enables a use case where all engines are created equally, we don't care > > + * where they are scheduled, we just want a certain number of resources, for > > + * those resources to be scheduled in parallel, and possibly across multiple > > + * engine classes. > > + */ > > + > > +/* > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > + * Each context must have the same number sibling and bonds are implictly create > > + * of the siblings. > > + * > > + * All of the below examples are in logical space. > > + * > > + * Example 1 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * > > + * Example 2 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * CS[2], CS[3] > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS[0], CS[2] > > + * VE[1] = CS[1], CS[3] > > + * > > + * This enables a use case where all engines are not equal and certain placement > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > + * case (logically contiguous placement, within a single engine class) is > > + * supported when using GuC submission. Execlist mode could support all possible > > + * bonding configurations but currently doesn't support this extension. > > + */ > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > +/* > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > + * points on all hardware contexts between each set of BBs. An example use case > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > + * BB must be submitted on each hardware context in the parallel gem context. > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > + */ > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > + __u64 flags; /* all undefined flags must be zero */ > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > + > > + /* > > + * width (i) * num_siblings (j) in length > > + * index = j + i * num_siblings > > + */ > > + struct i915_engine_class_instance engines[0]; > > +} __attribute__ ((packed)); > > + > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > index 7faa46cde088..64c539486ee4 100644 > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > New parallel submission uAPI > > ============================ > > -Details to come in a following patch. > > +The existing bonding uAPI is completely broken with GuC submission because > > +whether a submission is a single context submit or parallel submit isn't known > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > +contexts in parallel with the GuC the context must be explictly registered with > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > +These interfaces doesn't support dynamically changing between N contexts as the > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > + > > +The new parallel submission uAPI consists of 3 parts: > > + > > +* Export engines logical mapping > > +* A 'set_parallel' extension to configure contexts for parallel > > + submission > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > + > > +Export engines logical mapping > > +------------------------------ > > +Certain use cases require BBs to be placed on engine instances in logical order > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > +submission interface currently only supports submitting multiple contexts to > > +engines in logical order which is a new requirement compared to execlists. > > +Lastly, all current platforms have at most 2 instances and the logical order is > > +the same a uABI order. This will change on platforms with more than 2 instances. > > + > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > +logical instance has been returned and a new field, > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > + > > +A 'set_parallel' extension to configure contexts for parallel submission > > +------------------------------------------------------------------------ > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > +It is setup step that should be called before using any of the contexts. See > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > +similar existing examples. Once a slot is configured for parallel submission the > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > +support GuC submission. Execlist support can be added later if needed. > > + > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > + > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > +------------------------------------------------------------------- > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > +the IOCTL. >
On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: > > Add entry fpr i915 new parallel submission uAPI plan. > > > > v2: > > (Daniel Vetter): > > - Expand logical order explaination > > - Add dummy header > > - Only allow N BBs in execbuf IOCTL > > - Configure parallel submission per slot not per gem context > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > Cc: Tony Ye <tony.ye@intel.com> > > CC: Carl Zhang <carl.zhang@intel.com> > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > --- > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > 2 files changed, 196 insertions(+), 1 deletion(-) > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > new file mode 100644 > > index 000000000000..8c64b983ccad > > --- /dev/null > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > @@ -0,0 +1,144 @@ > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > + > > +/* > > + * i915_context_engines_parallel_submit: > > + * > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > + * the slots configuration). > > + * > > + * Their are two currently defined ways to control the placement of the > > + * hardware contexts on physical engines: default behavior (no flags) and > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > + * future as new hardware / use cases arise. Details of how to use this > > + * interface below above the flags. > > + * > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > + * placement configuration isn't supported on the platform / submission > > + * interface. > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > + * inteface. > > + */ > > +struct i915_context_engines_parallel_submit { > > + struct i915_user_extension base; > > + > > + __u16 engine_index; /* slot for parallel engine */ > > + __u16 width; /* number of contexts per parallel engine */ > > + __u16 num_siblings; /* number of siblings per context */ > > + __u16 mbz16; > > Ok the big picture looks reasonable now, the flags still confuse me. > Yea, it is a bit confusing. > > +/* > > + * Default placement behvavior (currently unsupported): > > + * > > + * Rather than restricting parallel submission to a single class with a > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > + * enables parallel submission across multiple engine classes. In this case each > > + * context's logical engine mask indicates where that context can placed. It is > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > + * if one context is running CS0 no other contexts can run on CS0). > > + * > > + * Example 1 pseudo code: > > + * CSX[Y] = engine class X, logical instance Y > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > + * > > + * Results in the following valid placements: > > + * CS0[0], CS1[0] > > + * CS0[0], CS1[1] > > + * CS0[1], CS1[0] > > + * CS0[1], CS1[1] > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS0[0], CS0[1] > > + * VE[1] = CS1[0], CS1[1] > > + * > > + * Example 2 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * CS[0], CS[2] > > + * CS[1], CS[0] > > + * CS[1], CS[2] > > + * CS[2], CS[0] > > + * CS[2], CS[1] > > + * > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS[0], CS[1], CS[2] > > + * VE[1] = CS[0], CS[1], CS[2] > > + > > + * This enables a use case where all engines are created equally, we don't care > > + * where they are scheduled, we just want a certain number of resources, for > > + * those resources to be scheduled in parallel, and possibly across multiple > > + * engine classes. > > + */ > > So I don't really get what this does compared to setting the flag below. > Is this just about running the batchbuffers the wrong way round, i.e. if > you have (simplest case) > > width=2, num_sibglings=1, engines=CS[0], CS[1] > > Then both > CS[0], CS[1] > and > CS[1], CS[0] > are possible options for running 2 batches? Iow, the backend is allowed to > run the batchbuffers the wrong way round, which gains us nothing, since we > assume the batches take equally long and engines interchangeable. There is > no scheduling scenario where this additional flexibility can help. > > Also we don't have flags to select the only available and then specify an > entire pipe dream about what the non-flag mode does, without an > implementation. What is this about? > > If it's just "because bonded allowed this" then I think we should just > unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but > welp. > High level the flags came out of internal discussions how this interface should look. The default placement behavior is theoretically possible with execlists but has no use cases. The GuC supports / current use cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. Argued about for months and this is where we landed. At the end of the day I think we needed to show that this interface supports more placement rules than what the GuC supports / current use cases to future proof this interface. For what is it worth it seems kinda backwards that we landed on the default behavior not being supported in our current stack / HW. > > + > > +/* > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > + * Each context must have the same number sibling and bonds are implictly create > > + * of the siblings. > > + * > > + * All of the below examples are in logical space. > > + * > > + * Example 1 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * > > + * Example 2 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * CS[2], CS[3] > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS[0], CS[2] > > + * VE[1] = CS[1], CS[3] > > + * > > + * This enables a use case where all engines are not equal and certain placement > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > + * case (logically contiguous placement, within a single engine class) is > > + * supported when using GuC submission. Execlist mode could support all possible > > + * bonding configurations but currently doesn't support this extension. > > + */ > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > Does this map to intel_context_set_nopreempt(), so we could implement it > also in execlist? > intel_context_set_nopreempt is preempt is part of how this would be implemented for execlists. For GuC submission there is a algorithm between the i915 and GuC that inserts same preemption points between each set of N batches. I believe the rules are if context 1 can preempted the rest of the contexts (2 - N) can safely be preempted. A go / join algorithm, implemented with semaphores, around the batches makes sure the i915 adheres to these rules. > Also is this just an artifact of the implementation, or is this somehow > required functionality for userspace, i.e. the workload fails if e.g. GuC > decides to preempt all LRC of a parallel virtual engine? Of course just > preempting one is a bit a bug (but execlist I think happily does that). > I think it is part of the implementation / HW limitations. Basically if N batches are running in parallel if one of the batches gets preempted it can hang all the other batches even if it gets restarted. Again the idea behind this is only preempt any of these contexts between each set of N batches. The GuC has no concept of batches only contexts hence the aforementioned algorithm to insert same preemption points between each set of batches. Matt > Cheers, Daniel > > > +/* > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > + * points on all hardware contexts between each set of BBs. An example use case > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > + * BB must be submitted on each hardware context in the parallel gem context. > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > + */ > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > + __u64 flags; /* all undefined flags must be zero */ > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > + > > + /* > > + * width (i) * num_siblings (j) in length > > + * index = j + i * num_siblings > > + */ > > + struct i915_engine_class_instance engines[0]; > > +} __attribute__ ((packed)); > > + > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > index 7faa46cde088..64c539486ee4 100644 > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > > > New parallel submission uAPI > > ============================ > > -Details to come in a following patch. > > +The existing bonding uAPI is completely broken with GuC submission because > > +whether a submission is a single context submit or parallel submit isn't known > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > +contexts in parallel with the GuC the context must be explictly registered with > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > +These interfaces doesn't support dynamically changing between N contexts as the > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > + > > +The new parallel submission uAPI consists of 3 parts: > > + > > +* Export engines logical mapping > > +* A 'set_parallel' extension to configure contexts for parallel > > + submission > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > + > > +Export engines logical mapping > > +------------------------------ > > +Certain use cases require BBs to be placed on engine instances in logical order > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > +submission interface currently only supports submitting multiple contexts to > > +engines in logical order which is a new requirement compared to execlists. > > +Lastly, all current platforms have at most 2 instances and the logical order is > > +the same a uABI order. This will change on platforms with more than 2 instances. > > + > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > +logical instance has been returned and a new field, > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > + > > +A 'set_parallel' extension to configure contexts for parallel submission > > +------------------------------------------------------------------------ > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > +It is setup step that should be called before using any of the contexts. See > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > +similar existing examples. Once a slot is configured for parallel submission the > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > +support GuC submission. Execlist support can be added later if needed. > > + > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > + > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > +------------------------------------------------------------------- > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > +the IOCTL. > > -- > > 2.28.0 > > > > _______________________________________________ > > Intel-gfx mailing list > > Intel-gfx@lists.freedesktop.org > > https://lists.freedesktop.org/mailman/listinfo/intel-gfx > > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch
On Wed, May 19, 2021 at 7:19 PM Matthew Brost <matthew.brost@intel.com> wrote: > > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > v2: > > > (Daniel Vetter): > > > - Expand logical order explaination > > > - Add dummy header > > > - Only allow N BBs in execbuf IOCTL > > > - Configure parallel submission per slot not per gem context > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > Cc: Tony Ye <tony.ye@intel.com> > > > CC: Carl Zhang <carl.zhang@intel.com> > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > --- > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > new file mode 100644 > > > index 000000000000..8c64b983ccad > > > --- /dev/null > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > @@ -0,0 +1,144 @@ > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > + > > > +/* > > > + * i915_context_engines_parallel_submit: > > > + * > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > + * the slots configuration). > > > + * > > > + * Their are two currently defined ways to control the placement of the > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > + * future as new hardware / use cases arise. Details of how to use this > > > + * interface below above the flags. > > > + * > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > + * placement configuration isn't supported on the platform / submission > > > + * interface. > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > + * inteface. > > > + */ > > > +struct i915_context_engines_parallel_submit { > > > + struct i915_user_extension base; > > > + > > > + __u16 engine_index; /* slot for parallel engine */ > > > + __u16 width; /* number of contexts per parallel engine */ > > > + __u16 num_siblings; /* number of siblings per context */ > > > + __u16 mbz16; > > > > Ok the big picture looks reasonable now, the flags still confuse me. > > > > Yea, it is a bit confusing. > > > > +/* > > > + * Default placement behvavior (currently unsupported): > > > + * > > > + * Rather than restricting parallel submission to a single class with a > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > + * enables parallel submission across multiple engine classes. In this case each > > > + * context's logical engine mask indicates where that context can placed. It is > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > + * if one context is running CS0 no other contexts can run on CS0). > > > + * > > > + * Example 1 pseudo code: > > > + * CSX[Y] = engine class X, logical instance Y > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > + * set_engines(INVALID) > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > + * > > > + * Results in the following valid placements: > > > + * CS0[0], CS1[0] > > > + * CS0[0], CS1[1] > > > + * CS0[1], CS1[0] > > > + * CS0[1], CS1[1] > > > + * > > > + * This can also be though of as 2 virtual engines: > > > + * VE[0] = CS0[0], CS0[1] > > > + * VE[1] = CS1[0], CS1[1] > > > + * > > > + * Example 2 pseudo code: > > > + * CS[X] = generic engine of same class, logical instance X > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > + * set_engines(INVALID) > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > + * > > > + * Results in the following valid placements: > > > + * CS[0], CS[1] > > > + * CS[0], CS[2] > > > + * CS[1], CS[0] > > > + * CS[1], CS[2] > > > + * CS[2], CS[0] > > > + * CS[2], CS[1] > > > + * > > > + * > > > + * This can also be though of as 2 virtual engines: > > > + * VE[0] = CS[0], CS[1], CS[2] > > > + * VE[1] = CS[0], CS[1], CS[2] > > > + > > > + * This enables a use case where all engines are created equally, we don't care > > > + * where they are scheduled, we just want a certain number of resources, for > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > + * engine classes. > > > + */ > > > > So I don't really get what this does compared to setting the flag below. > > Is this just about running the batchbuffers the wrong way round, i.e. if > > you have (simplest case) > > > > width=2, num_sibglings=1, engines=CS[0], CS[1] > > > > Then both > > CS[0], CS[1] > > and > > CS[1], CS[0] > > are possible options for running 2 batches? Iow, the backend is allowed to > > run the batchbuffers the wrong way round, which gains us nothing, since we > > assume the batches take equally long and engines interchangeable. There is > > no scheduling scenario where this additional flexibility can help. > > > > Also we don't have flags to select the only available and then specify an > > entire pipe dream about what the non-flag mode does, without an > > implementation. What is this about? > > > > If it's just "because bonded allowed this" then I think we should just > > unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but > > welp. > > > > High level the flags came out of internal discussions how this interface > should look. The default placement behavior is theoretically possible > with execlists but has no use cases. The GuC supports / current use > cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. > > Argued about for months and this is where we landed. At the end of the > day I think we needed to show that this interface supports more > placement rules than what the GuC supports / current use cases to future > proof this interface. > > For what is it worth it seems kinda backwards that we landed on the > default behavior not being supported in our current stack / HW. Yeah I think that should be inverted, doesn't make sense. What I still don't get (and I've read Tvrtko's reply with the example) is what exactly is the difference between implicit and not implicit mode? Can you do a single example where the only difference is whether this flag is set, and then explain with that what are the actual differences in scheduling options that the backend is allowed to pick for the set of N patches? I'm feeling a bit dense, but I'm really not seeing what's even going on here :-( > > > + > > > +/* > > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > > + * Each context must have the same number sibling and bonds are implictly create > > > + * of the siblings. > > > + * > > > + * All of the below examples are in logical space. > > > + * > > > + * Example 1 pseudo code: > > > + * CS[X] = generic engine of same class, logical instance X > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > + * set_engines(INVALID) > > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > > + * > > > + * Results in the following valid placements: > > > + * CS[0], CS[1] > > > + * > > > + * Example 2 pseudo code: > > > + * CS[X] = generic engine of same class, logical instance X > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > + * set_engines(INVALID) > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > > + * > > > + * Results in the following valid placements: > > > + * CS[0], CS[1] > > > + * CS[2], CS[3] > > > + * > > > + * This can also be though of as 2 virtual engines: > > > + * VE[0] = CS[0], CS[2] > > > + * VE[1] = CS[1], CS[3] > > > + * > > > + * This enables a use case where all engines are not equal and certain placement > > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > > + * case (logically contiguous placement, within a single engine class) is > > > + * supported when using GuC submission. Execlist mode could support all possible > > > + * bonding configurations but currently doesn't support this extension. > > > + */ > > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > > > Does this map to intel_context_set_nopreempt(), so we could implement it > > also in execlist? > > > > intel_context_set_nopreempt is preempt is part of how this would be > implemented for execlists. For GuC submission there is a algorithm > between the i915 and GuC that inserts same preemption points between > each set of N batches. I believe the rules are if context 1 can > preempted the rest of the contexts (2 - N) can safely be preempted. A go > / join algorithm, implemented with semaphores, around the batches > makes sure the i915 adheres to these rules. > > > Also is this just an artifact of the implementation, or is this somehow > > required functionality for userspace, i.e. the workload fails if e.g. GuC > > decides to preempt all LRC of a parallel virtual engine? Of course just > > preempting one is a bit a bug (but execlist I think happily does that). > > > > I think it is part of the implementation / HW limitations. Basically if > N batches are running in parallel if one of the batches gets preempted > it can hang all the other batches even if it gets restarted. Again the > idea behind this is only preempt any of these contexts between each set > of N batches. The GuC has no concept of batches only contexts hence the > aforementioned algorithm to insert same preemption points between each > set of batches. Well backend preempting one but not the other is a bug. I don't think we need to talk about bugs :-) Now wrt a preempt mode in context creation, I think that makes sense. I'm just wondering whether this is correct here in the parallel submission (and why), or is it just an implementation artifact (current backends are buggy if we'd allow them to preempt) or what's going on. If this is just a case of "this is how current backends work, userspace doesn't actually care whether we allow preempt or not" then I think we should just document the behavior and that's it. Adding uapi for an option for which there's not even an implementation isn't great. But if we can actually choose, and userspace wants to, then I'm all fine with this. And we can always move this to a more generic place if other engines need preempt control later on too. Cheers, Daniel > Matt > > > Cheers, Daniel > > > > > +/* > > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > > + * points on all hardware contexts between each set of BBs. An example use case > > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > > + * BB must be submitted on each hardware context in the parallel gem context. > > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > > + */ > > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > > + __u64 flags; /* all undefined flags must be zero */ > > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > > + > > > + /* > > > + * width (i) * num_siblings (j) in length > > > + * index = j + i * num_siblings > > > + */ > > > + struct i915_engine_class_instance engines[0]; > > > +} __attribute__ ((packed)); > > > + > > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > > index 7faa46cde088..64c539486ee4 100644 > > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > > > > > New parallel submission uAPI > > > ============================ > > > -Details to come in a following patch. > > > +The existing bonding uAPI is completely broken with GuC submission because > > > +whether a submission is a single context submit or parallel submit isn't known > > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > > +contexts in parallel with the GuC the context must be explictly registered with > > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > > +These interfaces doesn't support dynamically changing between N contexts as the > > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > > + > > > +The new parallel submission uAPI consists of 3 parts: > > > + > > > +* Export engines logical mapping > > > +* A 'set_parallel' extension to configure contexts for parallel > > > + submission > > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > + > > > +Export engines logical mapping > > > +------------------------------ > > > +Certain use cases require BBs to be placed on engine instances in logical order > > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > > +submission interface currently only supports submitting multiple contexts to > > > +engines in logical order which is a new requirement compared to execlists. > > > +Lastly, all current platforms have at most 2 instances and the logical order is > > > +the same a uABI order. This will change on platforms with more than 2 instances. > > > + > > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > > +logical instance has been returned and a new field, > > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > > + > > > +A 'set_parallel' extension to configure contexts for parallel submission > > > +------------------------------------------------------------------------ > > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > > +It is setup step that should be called before using any of the contexts. See > > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > > +similar existing examples. Once a slot is configured for parallel submission the > > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > > +support GuC submission. Execlist support can be added later if needed. > > > + > > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > > + > > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > +------------------------------------------------------------------- > > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > > +the IOCTL. > > > -- > > > 2.28.0 > > > > > > _______________________________________________ > > > Intel-gfx mailing list > > > Intel-gfx@lists.freedesktop.org > > > https://lists.freedesktop.org/mailman/listinfo/intel-gfx > > > > -- > > Daniel Vetter > > Software Engineer, Intel Corporation > > http://blog.ffwll.ch -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
On 20/05/2021 10:54, Daniel Vetter wrote: > On Wed, May 19, 2021 at 7:19 PM Matthew Brost <matthew.brost@intel.com> wrote: >> >> On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: >>> On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: >>>> Add entry fpr i915 new parallel submission uAPI plan. >>>> >>>> v2: >>>> (Daniel Vetter): >>>> - Expand logical order explaination >>>> - Add dummy header >>>> - Only allow N BBs in execbuf IOCTL >>>> - Configure parallel submission per slot not per gem context >>>> >>>> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> >>>> Cc: Tony Ye <tony.ye@intel.com> >>>> CC: Carl Zhang <carl.zhang@intel.com> >>>> Cc: Daniel Vetter <daniel.vetter@intel.com> >>>> Cc: Jason Ekstrand <jason@jlekstrand.net> >>>> Signed-off-by: Matthew Brost <matthew.brost@intel.com> >>>> --- >>>> Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ >>>> Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- >>>> 2 files changed, 196 insertions(+), 1 deletion(-) >>>> create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h >>>> >>>> diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h >>>> new file mode 100644 >>>> index 000000000000..8c64b983ccad >>>> --- /dev/null >>>> +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h >>>> @@ -0,0 +1,144 @@ >>>> +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ >>>> + >>>> +/* >>>> + * i915_context_engines_parallel_submit: >>>> + * >>>> + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. >>>> + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple >>>> + * hardware contexts are created internally in the i915 run these BBs. Once a >>>> + * slot is configured for N BBs only N BBs can be submitted in each execbuf >>>> + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf >>>> + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on >>>> + * the slots configuration). >>>> + * >>>> + * Their are two currently defined ways to control the placement of the >>>> + * hardware contexts on physical engines: default behavior (no flags) and >>>> + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the >>>> + * future as new hardware / use cases arise. Details of how to use this >>>> + * interface below above the flags. >>>> + * >>>> + * Returns -EINVAL if hardware context placement configuration invalid or if the >>>> + * placement configuration isn't supported on the platform / submission >>>> + * interface. >>>> + * Returns -ENODEV if extension isn't supported on the platform / submission >>>> + * inteface. >>>> + */ >>>> +struct i915_context_engines_parallel_submit { >>>> + struct i915_user_extension base; >>>> + >>>> + __u16 engine_index; /* slot for parallel engine */ >>>> + __u16 width; /* number of contexts per parallel engine */ >>>> + __u16 num_siblings; /* number of siblings per context */ >>>> + __u16 mbz16; >>> >>> Ok the big picture looks reasonable now, the flags still confuse me. >>> >> >> Yea, it is a bit confusing. >> >>>> +/* >>>> + * Default placement behvavior (currently unsupported): >>>> + * >>>> + * Rather than restricting parallel submission to a single class with a >>>> + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that >>>> + * enables parallel submission across multiple engine classes. In this case each >>>> + * context's logical engine mask indicates where that context can placed. It is >>>> + * implied in this mode that all contexts have mutual exclusive placement (e.g. >>>> + * if one context is running CS0 no other contexts can run on CS0). >>>> + * >>>> + * Example 1 pseudo code: >>>> + * CSX[Y] = engine class X, logical instance Y >>>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>>> + * set_engines(INVALID) >>>> + * set_parallel(engine_index=0, width=2, num_siblings=2, >>>> + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) >>>> + * >>>> + * Results in the following valid placements: >>>> + * CS0[0], CS1[0] >>>> + * CS0[0], CS1[1] >>>> + * CS0[1], CS1[0] >>>> + * CS0[1], CS1[1] >>>> + * >>>> + * This can also be though of as 2 virtual engines: >>>> + * VE[0] = CS0[0], CS0[1] >>>> + * VE[1] = CS1[0], CS1[1] >>>> + * >>>> + * Example 2 pseudo code: >>>> + * CS[X] = generic engine of same class, logical instance X >>>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>>> + * set_engines(INVALID) >>>> + * set_parallel(engine_index=0, width=2, num_siblings=3, >>>> + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) >>>> + * >>>> + * Results in the following valid placements: >>>> + * CS[0], CS[1] >>>> + * CS[0], CS[2] >>>> + * CS[1], CS[0] >>>> + * CS[1], CS[2] >>>> + * CS[2], CS[0] >>>> + * CS[2], CS[1] >>>> + * >>>> + * >>>> + * This can also be though of as 2 virtual engines: >>>> + * VE[0] = CS[0], CS[1], CS[2] >>>> + * VE[1] = CS[0], CS[1], CS[2] >>>> + >>>> + * This enables a use case where all engines are created equally, we don't care >>>> + * where they are scheduled, we just want a certain number of resources, for >>>> + * those resources to be scheduled in parallel, and possibly across multiple >>>> + * engine classes. >>>> + */ >>> >>> So I don't really get what this does compared to setting the flag below. >>> Is this just about running the batchbuffers the wrong way round, i.e. if >>> you have (simplest case) >>> >>> width=2, num_sibglings=1, engines=CS[0], CS[1] >>> >>> Then both >>> CS[0], CS[1] >>> and >>> CS[1], CS[0] >>> are possible options for running 2 batches? Iow, the backend is allowed to >>> run the batchbuffers the wrong way round, which gains us nothing, since we >>> assume the batches take equally long and engines interchangeable. There is >>> no scheduling scenario where this additional flexibility can help. >>> >>> Also we don't have flags to select the only available and then specify an >>> entire pipe dream about what the non-flag mode does, without an >>> implementation. What is this about? >>> >>> If it's just "because bonded allowed this" then I think we should just >>> unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but >>> welp. >>> >> >> High level the flags came out of internal discussions how this interface >> should look. The default placement behavior is theoretically possible >> with execlists but has no use cases. The GuC supports / current use >> cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. >> >> Argued about for months and this is where we landed. At the end of the >> day I think we needed to show that this interface supports more >> placement rules than what the GuC supports / current use cases to future >> proof this interface. >> >> For what is it worth it seems kinda backwards that we landed on the >> default behavior not being supported in our current stack / HW. > > Yeah I think that should be inverted, doesn't make sense. > > What I still don't get (and I've read Tvrtko's reply with the example) > is what exactly is the difference between implicit and not implicit > mode? Can you do a single example where the only difference is whether > this flag is set, and then explain with that what are the actual > differences in scheduling options that the backend is allowed to pick > for the set of N patches? > > I'm feeling a bit dense, but I'm really not seeing what's even going on here :-( 2-wide compute context: .engine_map([-1, -1]) .load_balance(0: [cs0, cs1, cs2, cs3]) // place virtual engine at slot 0 .load_balance(1: [cs0, cs1, cs2, cs3]) .set_parallel() This tells the scheduler any two of the four possible engines can be used. cs0 + cs3 is fine, cs3 + cs1 also, ... any. Only implicit rule is they have to be different and that works for all. 2-wide "implicit bonds mode" aka media fixed function limitation: .engine_map([-1, -1]) .load_balance(0: [cs0, cs2]) .load_balance(1: [cs1, cs3]) .set_parallel(flags = implicit_bond) Think of implicit flag creating a "link" between vertical columns in each virtual engine slot. So valid pairs end up cs0 + cs1 and cs2 + cs3 only. You can also think of the implicit flag as a shortcut to avoid specifying bonds via the existing extension. In which case context setup would be written along the lines of: .engine_map([-1, -1]) .load_balance(0: [cs0, cs2]) .load_balance(1: [cs1, cs3]) .bond(1: master = cs0, bond = [cs1]) .bond(1: master = cs2, bond = [cs3]) .set_parallel() So the implicit flag is just a shortcut to avoid typing the bonds. Not really needed as explained in my previous reply. This was at least the "old" set_parallel. I see this latest RFC changed things a bit which I don't really follow yet. It's not perfect but needs to add very little (just one context extension, on top of multi batch execbuf which is needed anyways), doesn't need to deprecate anything, didn't require rewrites of the UMD, and it all works today and in the future. I did not really like this new uapi for all the reasons I listed already, but as not many people were seeing the advantage of not churning on the uapi, if we are churning already I did suggests a different idea. I mean if we are churning we might as well go full in. So that proposal, which didn't get any traction, was along the lines of: .engine_map([-1]) .load_balance_wide(0: width=2, engines=[[cs0, cs2], [cs1, cs3]]) This would create an explicit wide virtual engine which should work for GuC fine I think. For execlists it may require a bit of extra glue but I don't think too much. Advantage is there is one engine in the map now and it is N-wide by definition. Since no one did bite on that idea back then, I didn't really pursue is to see if it works for all use cases. But I think it should even if it probably requires further thinking to be sure. If we apply it to compute use case.. .engine_map([-1]) .load_balance_wide(0: width=2, engines=[[cs0, cs1, cs2, cs3], [cs0, cs1, cs2, cs3]]) This means the only implicit wart in there is that cs0 + cs0 obviously shouldn't be picked. But that should be fine both for execlists and hopefully for the GuC. Regards, Tvrtko
Am 19.05.21 um 18:51 schrieb Matthew Brost: > On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote: >> Oh, yeah we call that gang submit on the AMD side. >> >> Had already some internal discussions how to implement this, but so far >> couldn't figure out how to cleanly introduce that into the DRM scheduler. >> >> Can you briefly describe in a few words how that is supposed to work on the >> Intel side? >> > Sure, I've done a quick PoC internally and have been able to hook this > into the DRM scheduler. > > Basically each BB still maps to a single job as each job is somewhat > unique (e.g. each job has its own ring, lrc, seqno, etc...). However all > the jobs configured to run in parallel map to a single sched_entity > which maintains the order each job was generated from the execbuf IOCTL > (1 - N). When the backend receives jobs 1 to N - 1 it basically just > updates some internal state. When the backend sees job N (last job) it > actually does the submit for jobs 1 - N which with GuC submission is a > simple command moving the LRC tail of the N jobs. > > Daniel has suggested that we create a single job for the NN BBs but that > would be huge rework to the internals of the i915 and likely won't > happen by the time this code first lands. > > Also worth noting one way a job isn't really a treated individually is > the excl slot with dma-resv. In that case we create a composite fence of > all jobs (dma_fence_array). Yeah, that's something we have discussed as well. How do you prevent the scheduler from over committing to a single ring buffer in this scenario? Christian. > > Matt > >> Thanks, >> Christian. >> >> Am 19.05.21 um 01:58 schrieb Matthew Brost: >>> Add entry fpr i915 new parallel submission uAPI plan. >>> >>> v2: >>> (Daniel Vetter): >>> - Expand logical order explaination >>> - Add dummy header >>> - Only allow N BBs in execbuf IOCTL >>> - Configure parallel submission per slot not per gem context >>> >>> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> >>> Cc: Tony Ye <tony.ye@intel.com> >>> CC: Carl Zhang <carl.zhang@intel.com> >>> Cc: Daniel Vetter <daniel.vetter@intel.com> >>> Cc: Jason Ekstrand <jason@jlekstrand.net> >>> Signed-off-by: Matthew Brost <matthew.brost@intel.com> >>> --- >>> Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ >>> Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- >>> 2 files changed, 196 insertions(+), 1 deletion(-) >>> create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h >>> >>> diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h >>> new file mode 100644 >>> index 000000000000..8c64b983ccad >>> --- /dev/null >>> +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h >>> @@ -0,0 +1,144 @@ >>> +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ >>> + >>> +/* >>> + * i915_context_engines_parallel_submit: >>> + * >>> + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. >>> + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple >>> + * hardware contexts are created internally in the i915 run these BBs. Once a >>> + * slot is configured for N BBs only N BBs can be submitted in each execbuf >>> + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf >>> + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on >>> + * the slots configuration). >>> + * >>> + * Their are two currently defined ways to control the placement of the >>> + * hardware contexts on physical engines: default behavior (no flags) and >>> + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the >>> + * future as new hardware / use cases arise. Details of how to use this >>> + * interface below above the flags. >>> + * >>> + * Returns -EINVAL if hardware context placement configuration invalid or if the >>> + * placement configuration isn't supported on the platform / submission >>> + * interface. >>> + * Returns -ENODEV if extension isn't supported on the platform / submission >>> + * inteface. >>> + */ >>> +struct i915_context_engines_parallel_submit { >>> + struct i915_user_extension base; >>> + >>> + __u16 engine_index; /* slot for parallel engine */ >>> + __u16 width; /* number of contexts per parallel engine */ >>> + __u16 num_siblings; /* number of siblings per context */ >>> + __u16 mbz16; >>> +/* >>> + * Default placement behvavior (currently unsupported): >>> + * >>> + * Rather than restricting parallel submission to a single class with a >>> + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that >>> + * enables parallel submission across multiple engine classes. In this case each >>> + * context's logical engine mask indicates where that context can placed. It is >>> + * implied in this mode that all contexts have mutual exclusive placement (e.g. >>> + * if one context is running CS0 no other contexts can run on CS0). >>> + * >>> + * Example 1 pseudo code: >>> + * CSX[Y] = engine class X, logical instance Y >>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>> + * set_engines(INVALID) >>> + * set_parallel(engine_index=0, width=2, num_siblings=2, >>> + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) >>> + * >>> + * Results in the following valid placements: >>> + * CS0[0], CS1[0] >>> + * CS0[0], CS1[1] >>> + * CS0[1], CS1[0] >>> + * CS0[1], CS1[1] >>> + * >>> + * This can also be though of as 2 virtual engines: >>> + * VE[0] = CS0[0], CS0[1] >>> + * VE[1] = CS1[0], CS1[1] >>> + * >>> + * Example 2 pseudo code: >>> + * CS[X] = generic engine of same class, logical instance X >>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>> + * set_engines(INVALID) >>> + * set_parallel(engine_index=0, width=2, num_siblings=3, >>> + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) >>> + * >>> + * Results in the following valid placements: >>> + * CS[0], CS[1] >>> + * CS[0], CS[2] >>> + * CS[1], CS[0] >>> + * CS[1], CS[2] >>> + * CS[2], CS[0] >>> + * CS[2], CS[1] >>> + * >>> + * >>> + * This can also be though of as 2 virtual engines: >>> + * VE[0] = CS[0], CS[1], CS[2] >>> + * VE[1] = CS[0], CS[1], CS[2] >>> + >>> + * This enables a use case where all engines are created equally, we don't care >>> + * where they are scheduled, we just want a certain number of resources, for >>> + * those resources to be scheduled in parallel, and possibly across multiple >>> + * engine classes. >>> + */ >>> + >>> +/* >>> + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. >>> + * Each context must have the same number sibling and bonds are implictly create >>> + * of the siblings. >>> + * >>> + * All of the below examples are in logical space. >>> + * >>> + * Example 1 pseudo code: >>> + * CS[X] = generic engine of same class, logical instance X >>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>> + * set_engines(INVALID) >>> + * set_parallel(engine_index=0, width=2, num_siblings=1, >>> + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) >>> + * >>> + * Results in the following valid placements: >>> + * CS[0], CS[1] >>> + * >>> + * Example 2 pseudo code: >>> + * CS[X] = generic engine of same class, logical instance X >>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>> + * set_engines(INVALID) >>> + * set_parallel(engine_index=0, width=2, num_siblings=2, >>> + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) >>> + * >>> + * Results in the following valid placements: >>> + * CS[0], CS[1] >>> + * CS[2], CS[3] >>> + * >>> + * This can also be though of as 2 virtual engines: >>> + * VE[0] = CS[0], CS[2] >>> + * VE[1] = CS[1], CS[3] >>> + * >>> + * This enables a use case where all engines are not equal and certain placement >>> + * rules are required (i.e. split-frame requires all contexts to be placed in a >>> + * logically contiguous order on the VCS engines on gen11+ platforms). This use >>> + * case (logically contiguous placement, within a single engine class) is >>> + * supported when using GuC submission. Execlist mode could support all possible >>> + * bonding configurations but currently doesn't support this extension. >>> + */ >>> +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) >>> +/* >>> + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption >>> + * points on all hardware contexts between each set of BBs. An example use case >>> + * of this feature is split-frame on gen11+ hardware. When using this feature a >>> + * BB must be submitted on each hardware context in the parallel gem context. >>> + * The execbuf2 IOCTL enforces the user adheres to policy. >>> + */ >>> +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) >>> +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) >>> + __u64 flags; /* all undefined flags must be zero */ >>> + __u64 mbz64[3]; /* reserved for future use; must be zero */ >>> + >>> + /* >>> + * width (i) * num_siblings (j) in length >>> + * index = j + i * num_siblings >>> + */ >>> + struct i915_engine_class_instance engines[0]; >>> +} __attribute__ ((packed)); >>> + >>> diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst >>> index 7faa46cde088..64c539486ee4 100644 >>> --- a/Documentation/gpu/rfc/i915_scheduler.rst >>> +++ b/Documentation/gpu/rfc/i915_scheduler.rst >>> @@ -82,4 +82,55 @@ https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspec.oneapi.com%2Flevel-zero%2Flatest%2Fcore%2Fapi.html%23ze-command-queue-priorit&data=04%7C01%7Cchristian.koenig%40amd.com%7C49a7557f4e494090755608d91ae758a6%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637570403202969375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dn3PVdxYQpkpWIru5eAXHgbfuLDkppAA5daV5sHQF7s%3D&reserved=0 >>> New parallel submission uAPI >>> ============================ >>> -Details to come in a following patch. >>> +The existing bonding uAPI is completely broken with GuC submission because >>> +whether a submission is a single context submit or parallel submit isn't known >>> +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple >>> +contexts in parallel with the GuC the context must be explictly registered with >>> +N contexts and all N contexts must be submitted in a single command to the GuC. >>> +These interfaces doesn't support dynamically changing between N contexts as the >>> +bonding uAPI does. Hence the need for a new parallel submission interface. Also >>> +the legacy bonding uAPI is quite confusing and not intuitive at all. >>> + >>> +The new parallel submission uAPI consists of 3 parts: >>> + >>> +* Export engines logical mapping >>> +* A 'set_parallel' extension to configure contexts for parallel >>> + submission >>> +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL >>> + >>> +Export engines logical mapping >>> +------------------------------ >>> +Certain use cases require BBs to be placed on engine instances in logical order >>> +(e.g. split-frame on gen11+). The logical mapping of engine instances can change >>> +based on fusing. Rather than making UMDs be aware of fusing, simply expose the >>> +logical mapping with the existing query engine info IOCTL. Also the GuC >>> +submission interface currently only supports submitting multiple contexts to >>> +engines in logical order which is a new requirement compared to execlists. >>> +Lastly, all current platforms have at most 2 instances and the logical order is >>> +the same a uABI order. This will change on platforms with more than 2 instances. >>> + >>> +A single bit will be added to drm_i915_engine_info.flags indicating that the >>> +logical instance has been returned and a new field, >>> +drm_i915_engine_info.logical_instance, returns the logical instance. >>> + >>> +A 'set_parallel' extension to configure contexts for parallel submission >>> +------------------------------------------------------------------------ >>> +The 'set_parallel' extension configures a slot for parallel submission of N BBs. >>> +It is setup step that should be called before using any of the contexts. See >>> +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for >>> +similar existing examples. Once a slot is configured for parallel submission the >>> +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only >>> +support GuC submission. Execlist support can be added later if needed. >>> + >>> +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and >>> +i915_context_engines_parallel_submit to the uAPI to implement this extension. >>> + >>> +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL >>> +------------------------------------------------------------------- >>> +Contexts that have been configured with the 'set_parallel' extension are allowed >>> +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N >>> +objects in the drm_i915_gem_exec_object2 list or the first N if >>> +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot >>> +submitted and how it has been configured by 'set_parallel' or other extensions. >>> +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of >>> +the IOCTL.
On Thu, May 20, 2021 at 11:54:25AM +0200, Daniel Vetter wrote: > On Wed, May 19, 2021 at 7:19 PM Matthew Brost <matthew.brost@intel.com> wrote: > > > > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: > > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > v2: > > > > (Daniel Vetter): > > > > - Expand logical order explaination > > > > - Add dummy header > > > > - Only allow N BBs in execbuf IOCTL > > > > - Configure parallel submission per slot not per gem context > > > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > > Cc: Tony Ye <tony.ye@intel.com> > > > > CC: Carl Zhang <carl.zhang@intel.com> > > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > > --- > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > new file mode 100644 > > > > index 000000000000..8c64b983ccad > > > > --- /dev/null > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > @@ -0,0 +1,144 @@ > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > > + > > > > +/* > > > > + * i915_context_engines_parallel_submit: > > > > + * > > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > > + * the slots configuration). > > > > + * > > > > + * Their are two currently defined ways to control the placement of the > > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > > + * future as new hardware / use cases arise. Details of how to use this > > > > + * interface below above the flags. > > > > + * > > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > > + * placement configuration isn't supported on the platform / submission > > > > + * interface. > > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > > + * inteface. > > > > + */ > > > > +struct i915_context_engines_parallel_submit { > > > > + struct i915_user_extension base; > > > > + > > > > + __u16 engine_index; /* slot for parallel engine */ > > > > + __u16 width; /* number of contexts per parallel engine */ > > > > + __u16 num_siblings; /* number of siblings per context */ > > > > + __u16 mbz16; > > > > > > Ok the big picture looks reasonable now, the flags still confuse me. > > > > > > > Yea, it is a bit confusing. > > > > > > +/* > > > > + * Default placement behvavior (currently unsupported): > > > > + * > > > > + * Rather than restricting parallel submission to a single class with a > > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > > + * enables parallel submission across multiple engine classes. In this case each > > > > + * context's logical engine mask indicates where that context can placed. It is > > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > > + * if one context is running CS0 no other contexts can run on CS0). > > > > + * > > > > + * Example 1 pseudo code: > > > > + * CSX[Y] = engine class X, logical instance Y > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > + * set_engines(INVALID) > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > > + * > > > > + * Results in the following valid placements: > > > > + * CS0[0], CS1[0] > > > > + * CS0[0], CS1[1] > > > > + * CS0[1], CS1[0] > > > > + * CS0[1], CS1[1] > > > > + * > > > > + * This can also be though of as 2 virtual engines: > > > > + * VE[0] = CS0[0], CS0[1] > > > > + * VE[1] = CS1[0], CS1[1] > > > > + * > > > > + * Example 2 pseudo code: > > > > + * CS[X] = generic engine of same class, logical instance X > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > + * set_engines(INVALID) > > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > > + * > > > > + * Results in the following valid placements: > > > > + * CS[0], CS[1] > > > > + * CS[0], CS[2] > > > > + * CS[1], CS[0] > > > > + * CS[1], CS[2] > > > > + * CS[2], CS[0] > > > > + * CS[2], CS[1] > > > > + * > > > > + * > > > > + * This can also be though of as 2 virtual engines: > > > > + * VE[0] = CS[0], CS[1], CS[2] > > > > + * VE[1] = CS[0], CS[1], CS[2] > > > > + > > > > + * This enables a use case where all engines are created equally, we don't care > > > > + * where they are scheduled, we just want a certain number of resources, for > > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > > + * engine classes. > > > > + */ > > > > > > So I don't really get what this does compared to setting the flag below. > > > Is this just about running the batchbuffers the wrong way round, i.e. if > > > you have (simplest case) > > > > > > width=2, num_sibglings=1, engines=CS[0], CS[1] > > > > > > Then both > > > CS[0], CS[1] > > > and > > > CS[1], CS[0] > > > are possible options for running 2 batches? Iow, the backend is allowed to > > > run the batchbuffers the wrong way round, which gains us nothing, since we > > > assume the batches take equally long and engines interchangeable. There is > > > no scheduling scenario where this additional flexibility can help. > > > > > > Also we don't have flags to select the only available and then specify an > > > entire pipe dream about what the non-flag mode does, without an > > > implementation. What is this about? > > > > > > If it's just "because bonded allowed this" then I think we should just > > > unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but > > > welp. > > > > > > > High level the flags came out of internal discussions how this interface > > should look. The default placement behavior is theoretically possible > > with execlists but has no use cases. The GuC supports / current use > > cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. > > > > Argued about for months and this is where we landed. At the end of the > > day I think we needed to show that this interface supports more > > placement rules than what the GuC supports / current use cases to future > > proof this interface. > > > > For what is it worth it seems kinda backwards that we landed on the > > default behavior not being supported in our current stack / HW. > > Yeah I think that should be inverted, doesn't make sense. > > What I still don't get (and I've read Tvrtko's reply with the example) > is what exactly is the difference between implicit and not implicit > mode? Can you do a single example where the only difference is whether > this flag is set, and then explain with that what are the actual > differences in scheduling options that the backend is allowed to pick > for the set of N patches? > > I'm feeling a bit dense, but I'm really not seeing what's even going on here :-( > > > > > + > > > > +/* > > > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > > > + * Each context must have the same number sibling and bonds are implictly create > > > > + * of the siblings. > > > > + * > > > > + * All of the below examples are in logical space. > > > > + * > > > > + * Example 1 pseudo code: > > > > + * CS[X] = generic engine of same class, logical instance X > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > + * set_engines(INVALID) > > > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > + * > > > > + * Results in the following valid placements: > > > > + * CS[0], CS[1] > > > > + * > > > > + * Example 2 pseudo code: > > > > + * CS[X] = generic engine of same class, logical instance X > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > + * set_engines(INVALID) > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > + * > > > > + * Results in the following valid placements: > > > > + * CS[0], CS[1] > > > > + * CS[2], CS[3] > > > > + * > > > > + * This can also be though of as 2 virtual engines: > > > > + * VE[0] = CS[0], CS[2] > > > > + * VE[1] = CS[1], CS[3] > > > > + * > > > > + * This enables a use case where all engines are not equal and certain placement > > > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > > > + * case (logically contiguous placement, within a single engine class) is > > > > + * supported when using GuC submission. Execlist mode could support all possible > > > > + * bonding configurations but currently doesn't support this extension. > > > > + */ > > > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > > > > > Does this map to intel_context_set_nopreempt(), so we could implement it > > > also in execlist? > > > > > > > intel_context_set_nopreempt is preempt is part of how this would be > > implemented for execlists. For GuC submission there is a algorithm > > between the i915 and GuC that inserts same preemption points between > > each set of N batches. I believe the rules are if context 1 can > > preempted the rest of the contexts (2 - N) can safely be preempted. A go > > / join algorithm, implemented with semaphores, around the batches > > makes sure the i915 adheres to these rules. > > > > > Also is this just an artifact of the implementation, or is this somehow > > > required functionality for userspace, i.e. the workload fails if e.g. GuC > > > decides to preempt all LRC of a parallel virtual engine? Of course just > > > preempting one is a bit a bug (but execlist I think happily does that). > > > > > > > I think it is part of the implementation / HW limitations. Basically if > > N batches are running in parallel if one of the batches gets preempted > > it can hang all the other batches even if it gets restarted. Again the > > idea behind this is only preempt any of these contexts between each set > > of N batches. The GuC has no concept of batches only contexts hence the > > aforementioned algorithm to insert same preemption points between each > > set of batches. > > Well backend preempting one but not the other is a bug. I don't think > we need to talk about bugs :-) > I think it is more that backend can't preempt each context atomically as these are running on independent engines. Let say the backend tries to preempt 2 engines at the same time and only 1 responds, now we are broken with certain batches. With the algorithm implmented between the i915 and the Guc the GuC will try to preempt the first context if that works, all the other contexts can be preempted. > Now wrt a preempt mode in context creation, I think that makes sense. > I'm just wondering whether this is correct here in the parallel > submission (and why), or is it just an implementation artifact > (current backends are buggy if we'd allow them to preempt) or what's > going on. > > If this is just a case of "this is how current backends work, > userspace doesn't actually care whether we allow preempt or not" then > I think we should just document the behavior and that's it. Adding > uapi for an option for which there's not even an implementation isn't > great. > I don't think it is a case of 'this is how current backends work...', I think user space knows if it is ok for 1 of the batches to preempted and not the others without breaking the batches. Matt > But if we can actually choose, and userspace wants to, then I'm all > fine with this. And we can always move this to a more generic place if > other engines need preempt control later on too. > > Cheers, Daniel > > > > Matt > > > > > Cheers, Daniel > > > > > > > +/* > > > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > > > + * points on all hardware contexts between each set of BBs. An example use case > > > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > > > + * BB must be submitted on each hardware context in the parallel gem context. > > > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > > > + */ > > > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > > > + __u64 flags; /* all undefined flags must be zero */ > > > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > > > + > > > > + /* > > > > + * width (i) * num_siblings (j) in length > > > > + * index = j + i * num_siblings > > > > + */ > > > > + struct i915_engine_class_instance engines[0]; > > > > +} __attribute__ ((packed)); > > > > + > > > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > > > index 7faa46cde088..64c539486ee4 100644 > > > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > > > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > > > > > > > New parallel submission uAPI > > > > ============================ > > > > -Details to come in a following patch. > > > > +The existing bonding uAPI is completely broken with GuC submission because > > > > +whether a submission is a single context submit or parallel submit isn't known > > > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > > > +contexts in parallel with the GuC the context must be explictly registered with > > > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > > > +These interfaces doesn't support dynamically changing between N contexts as the > > > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > > > + > > > > +The new parallel submission uAPI consists of 3 parts: > > > > + > > > > +* Export engines logical mapping > > > > +* A 'set_parallel' extension to configure contexts for parallel > > > > + submission > > > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > + > > > > +Export engines logical mapping > > > > +------------------------------ > > > > +Certain use cases require BBs to be placed on engine instances in logical order > > > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > > > +submission interface currently only supports submitting multiple contexts to > > > > +engines in logical order which is a new requirement compared to execlists. > > > > +Lastly, all current platforms have at most 2 instances and the logical order is > > > > +the same a uABI order. This will change on platforms with more than 2 instances. > > > > + > > > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > > > +logical instance has been returned and a new field, > > > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > > > + > > > > +A 'set_parallel' extension to configure contexts for parallel submission > > > > +------------------------------------------------------------------------ > > > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > > > +It is setup step that should be called before using any of the contexts. See > > > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > > > +similar existing examples. Once a slot is configured for parallel submission the > > > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > > > +support GuC submission. Execlist support can be added later if needed. > > > > + > > > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > > > + > > > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > +------------------------------------------------------------------- > > > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > > > +the IOCTL. > > > > -- > > > > 2.28.0 > > > > > > > > _______________________________________________ > > > > Intel-gfx mailing list > > > > Intel-gfx@lists.freedesktop.org > > > > https://lists.freedesktop.org/mailman/listinfo/intel-gfx > > > > > > -- > > > Daniel Vetter > > > Software Engineer, Intel Corporation > > > http://blog.ffwll.ch > > > > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch
On Thu, May 20, 2021 at 01:11:59PM +0200, Christian König wrote: > Am 19.05.21 um 18:51 schrieb Matthew Brost: > > On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote: > > > Oh, yeah we call that gang submit on the AMD side. > > > > > > Had already some internal discussions how to implement this, but so far > > > couldn't figure out how to cleanly introduce that into the DRM scheduler. > > > > > > Can you briefly describe in a few words how that is supposed to work on the > > > Intel side? > > > > > Sure, I've done a quick PoC internally and have been able to hook this > > into the DRM scheduler. > > > > Basically each BB still maps to a single job as each job is somewhat > > unique (e.g. each job has its own ring, lrc, seqno, etc...). However all > > the jobs configured to run in parallel map to a single sched_entity > > which maintains the order each job was generated from the execbuf IOCTL > > (1 - N). When the backend receives jobs 1 to N - 1 it basically just > > updates some internal state. When the backend sees job N (last job) it > > actually does the submit for jobs 1 - N which with GuC submission is a > > simple command moving the LRC tail of the N jobs. > > > > Daniel has suggested that we create a single job for the NN BBs but that > > would be huge rework to the internals of the i915 and likely won't > > happen by the time this code first lands. > > > > Also worth noting one way a job isn't really a treated individually is > > the excl slot with dma-resv. In that case we create a composite fence of > > all jobs (dma_fence_array). > > Yeah, that's something we have discussed as well. > > How do you prevent the scheduler from over committing to a single ring > buffer in this scenario? > Each job has its own ring, the execbuf IOCTL throttles itself for each job if there isn't space in the ring. This is exactly the same as non-parallel submits. I think this is what you were asking? If not, maybe try explaining the question a bit more. Matt > Christian. > > > > > Matt > > > > > Thanks, > > > Christian. > > > > > > Am 19.05.21 um 01:58 schrieb Matthew Brost: > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > v2: > > > > (Daniel Vetter): > > > > - Expand logical order explaination > > > > - Add dummy header > > > > - Only allow N BBs in execbuf IOCTL > > > > - Configure parallel submission per slot not per gem context > > > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > > Cc: Tony Ye <tony.ye@intel.com> > > > > CC: Carl Zhang <carl.zhang@intel.com> > > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > > --- > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > new file mode 100644 > > > > index 000000000000..8c64b983ccad > > > > --- /dev/null > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > @@ -0,0 +1,144 @@ > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > > + > > > > +/* > > > > + * i915_context_engines_parallel_submit: > > > > + * > > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > > + * the slots configuration). > > > > + * > > > > + * Their are two currently defined ways to control the placement of the > > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > > + * future as new hardware / use cases arise. Details of how to use this > > > > + * interface below above the flags. > > > > + * > > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > > + * placement configuration isn't supported on the platform / submission > > > > + * interface. > > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > > + * inteface. > > > > + */ > > > > +struct i915_context_engines_parallel_submit { > > > > + struct i915_user_extension base; > > > > + > > > > + __u16 engine_index; /* slot for parallel engine */ > > > > + __u16 width; /* number of contexts per parallel engine */ > > > > + __u16 num_siblings; /* number of siblings per context */ > > > > + __u16 mbz16; > > > > +/* > > > > + * Default placement behvavior (currently unsupported): > > > > + * > > > > + * Rather than restricting parallel submission to a single class with a > > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > > + * enables parallel submission across multiple engine classes. In this case each > > > > + * context's logical engine mask indicates where that context can placed. It is > > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > > + * if one context is running CS0 no other contexts can run on CS0). > > > > + * > > > > + * Example 1 pseudo code: > > > > + * CSX[Y] = engine class X, logical instance Y > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > + * set_engines(INVALID) > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > > + * > > > > + * Results in the following valid placements: > > > > + * CS0[0], CS1[0] > > > > + * CS0[0], CS1[1] > > > > + * CS0[1], CS1[0] > > > > + * CS0[1], CS1[1] > > > > + * > > > > + * This can also be though of as 2 virtual engines: > > > > + * VE[0] = CS0[0], CS0[1] > > > > + * VE[1] = CS1[0], CS1[1] > > > > + * > > > > + * Example 2 pseudo code: > > > > + * CS[X] = generic engine of same class, logical instance X > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > + * set_engines(INVALID) > > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > > + * > > > > + * Results in the following valid placements: > > > > + * CS[0], CS[1] > > > > + * CS[0], CS[2] > > > > + * CS[1], CS[0] > > > > + * CS[1], CS[2] > > > > + * CS[2], CS[0] > > > > + * CS[2], CS[1] > > > > + * > > > > + * > > > > + * This can also be though of as 2 virtual engines: > > > > + * VE[0] = CS[0], CS[1], CS[2] > > > > + * VE[1] = CS[0], CS[1], CS[2] > > > > + > > > > + * This enables a use case where all engines are created equally, we don't care > > > > + * where they are scheduled, we just want a certain number of resources, for > > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > > + * engine classes. > > > > + */ > > > > + > > > > +/* > > > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > > > + * Each context must have the same number sibling and bonds are implictly create > > > > + * of the siblings. > > > > + * > > > > + * All of the below examples are in logical space. > > > > + * > > > > + * Example 1 pseudo code: > > > > + * CS[X] = generic engine of same class, logical instance X > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > + * set_engines(INVALID) > > > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > + * > > > > + * Results in the following valid placements: > > > > + * CS[0], CS[1] > > > > + * > > > > + * Example 2 pseudo code: > > > > + * CS[X] = generic engine of same class, logical instance X > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > + * set_engines(INVALID) > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > + * > > > > + * Results in the following valid placements: > > > > + * CS[0], CS[1] > > > > + * CS[2], CS[3] > > > > + * > > > > + * This can also be though of as 2 virtual engines: > > > > + * VE[0] = CS[0], CS[2] > > > > + * VE[1] = CS[1], CS[3] > > > > + * > > > > + * This enables a use case where all engines are not equal and certain placement > > > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > > > + * case (logically contiguous placement, within a single engine class) is > > > > + * supported when using GuC submission. Execlist mode could support all possible > > > > + * bonding configurations but currently doesn't support this extension. > > > > + */ > > > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > > > +/* > > > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > > > + * points on all hardware contexts between each set of BBs. An example use case > > > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > > > + * BB must be submitted on each hardware context in the parallel gem context. > > > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > > > + */ > > > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > > > + __u64 flags; /* all undefined flags must be zero */ > > > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > > > + > > > > + /* > > > > + * width (i) * num_siblings (j) in length > > > > + * index = j + i * num_siblings > > > > + */ > > > > + struct i915_engine_class_instance engines[0]; > > > > +} __attribute__ ((packed)); > > > > + > > > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > > > index 7faa46cde088..64c539486ee4 100644 > > > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > > > @@ -82,4 +82,55 @@ https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspec.oneapi.com%2Flevel-zero%2Flatest%2Fcore%2Fapi.html%23ze-command-queue-priorit&data=04%7C01%7Cchristian.koenig%40amd.com%7C49a7557f4e494090755608d91ae758a6%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637570403202969375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dn3PVdxYQpkpWIru5eAXHgbfuLDkppAA5daV5sHQF7s%3D&reserved=0 > > > > New parallel submission uAPI > > > > ============================ > > > > -Details to come in a following patch. > > > > +The existing bonding uAPI is completely broken with GuC submission because > > > > +whether a submission is a single context submit or parallel submit isn't known > > > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > > > +contexts in parallel with the GuC the context must be explictly registered with > > > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > > > +These interfaces doesn't support dynamically changing between N contexts as the > > > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > > > + > > > > +The new parallel submission uAPI consists of 3 parts: > > > > + > > > > +* Export engines logical mapping > > > > +* A 'set_parallel' extension to configure contexts for parallel > > > > + submission > > > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > + > > > > +Export engines logical mapping > > > > +------------------------------ > > > > +Certain use cases require BBs to be placed on engine instances in logical order > > > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > > > +submission interface currently only supports submitting multiple contexts to > > > > +engines in logical order which is a new requirement compared to execlists. > > > > +Lastly, all current platforms have at most 2 instances and the logical order is > > > > +the same a uABI order. This will change on platforms with more than 2 instances. > > > > + > > > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > > > +logical instance has been returned and a new field, > > > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > > > + > > > > +A 'set_parallel' extension to configure contexts for parallel submission > > > > +------------------------------------------------------------------------ > > > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > > > +It is setup step that should be called before using any of the contexts. See > > > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > > > +similar existing examples. Once a slot is configured for parallel submission the > > > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > > > +support GuC submission. Execlist support can be added later if needed. > > > > + > > > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > > > + > > > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > +------------------------------------------------------------------- > > > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > > > +the IOCTL. >
On Thu, May 20, 2021 at 11:57:44AM +0100, Tvrtko Ursulin wrote: > > On 20/05/2021 10:54, Daniel Vetter wrote: > > On Wed, May 19, 2021 at 7:19 PM Matthew Brost <matthew.brost@intel.com> wrote: > > > > > > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: > > > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: > > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > > > v2: > > > > > (Daniel Vetter): > > > > > - Expand logical order explaination > > > > > - Add dummy header > > > > > - Only allow N BBs in execbuf IOCTL > > > > > - Configure parallel submission per slot not per gem context > > > > > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > > > Cc: Tony Ye <tony.ye@intel.com> > > > > > CC: Carl Zhang <carl.zhang@intel.com> > > > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > > > --- > > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > new file mode 100644 > > > > > index 000000000000..8c64b983ccad > > > > > --- /dev/null > > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > @@ -0,0 +1,144 @@ > > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > > > + > > > > > +/* > > > > > + * i915_context_engines_parallel_submit: > > > > > + * > > > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > > > + * the slots configuration). > > > > > + * > > > > > + * Their are two currently defined ways to control the placement of the > > > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > > > + * future as new hardware / use cases arise. Details of how to use this > > > > > + * interface below above the flags. > > > > > + * > > > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > > > + * placement configuration isn't supported on the platform / submission > > > > > + * interface. > > > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > > > + * inteface. > > > > > + */ > > > > > +struct i915_context_engines_parallel_submit { > > > > > + struct i915_user_extension base; > > > > > + > > > > > + __u16 engine_index; /* slot for parallel engine */ > > > > > + __u16 width; /* number of contexts per parallel engine */ > > > > > + __u16 num_siblings; /* number of siblings per context */ > > > > > + __u16 mbz16; > > > > > > > > Ok the big picture looks reasonable now, the flags still confuse me. > > > > > > > > > > Yea, it is a bit confusing. > > > > > > > > +/* > > > > > + * Default placement behvavior (currently unsupported): > > > > > + * > > > > > + * Rather than restricting parallel submission to a single class with a > > > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > > > + * enables parallel submission across multiple engine classes. In this case each > > > > > + * context's logical engine mask indicates where that context can placed. It is > > > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > > > + * if one context is running CS0 no other contexts can run on CS0). > > > > > + * > > > > > + * Example 1 pseudo code: > > > > > + * CSX[Y] = engine class X, logical instance Y > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS0[0], CS1[0] > > > > > + * CS0[0], CS1[1] > > > > > + * CS0[1], CS1[0] > > > > > + * CS0[1], CS1[1] > > > > > + * > > > > > + * This can also be though of as 2 virtual engines: > > > > > + * VE[0] = CS0[0], CS0[1] > > > > > + * VE[1] = CS1[0], CS1[1] > > > > > + * > > > > > + * Example 2 pseudo code: > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS[0], CS[1] > > > > > + * CS[0], CS[2] > > > > > + * CS[1], CS[0] > > > > > + * CS[1], CS[2] > > > > > + * CS[2], CS[0] > > > > > + * CS[2], CS[1] > > > > > + * > > > > > + * > > > > > + * This can also be though of as 2 virtual engines: > > > > > + * VE[0] = CS[0], CS[1], CS[2] > > > > > + * VE[1] = CS[0], CS[1], CS[2] > > > > > + > > > > > + * This enables a use case where all engines are created equally, we don't care > > > > > + * where they are scheduled, we just want a certain number of resources, for > > > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > > > + * engine classes. > > > > > + */ > > > > > > > > So I don't really get what this does compared to setting the flag below. > > > > Is this just about running the batchbuffers the wrong way round, i.e. if > > > > you have (simplest case) > > > > > > > > width=2, num_sibglings=1, engines=CS[0], CS[1] > > > > > > > > Then both > > > > CS[0], CS[1] > > > > and > > > > CS[1], CS[0] > > > > are possible options for running 2 batches? Iow, the backend is allowed to > > > > run the batchbuffers the wrong way round, which gains us nothing, since we > > > > assume the batches take equally long and engines interchangeable. There is > > > > no scheduling scenario where this additional flexibility can help. > > > > > > > > Also we don't have flags to select the only available and then specify an > > > > entire pipe dream about what the non-flag mode does, without an > > > > implementation. What is this about? > > > > > > > > If it's just "because bonded allowed this" then I think we should just > > > > unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but > > > > welp. > > > > > > > > > > High level the flags came out of internal discussions how this interface > > > should look. The default placement behavior is theoretically possible > > > with execlists but has no use cases. The GuC supports / current use > > > cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. > > > > > > Argued about for months and this is where we landed. At the end of the > > > day I think we needed to show that this interface supports more > > > placement rules than what the GuC supports / current use cases to future > > > proof this interface. > > > > > > For what is it worth it seems kinda backwards that we landed on the > > > default behavior not being supported in our current stack / HW. > > > > Yeah I think that should be inverted, doesn't make sense. > > > > What I still don't get (and I've read Tvrtko's reply with the example) > > is what exactly is the difference between implicit and not implicit > > mode? Can you do a single example where the only difference is whether > > this flag is set, and then explain with that what are the actual > > differences in scheduling options that the backend is allowed to pick > > for the set of N patches? > > > > I'm feeling a bit dense, but I'm really not seeing what's even going on here :-( > > 2-wide compute context: > > .engine_map([-1, -1]) > .load_balance(0: [cs0, cs1, cs2, cs3]) // place virtual engine at slot 0 > .load_balance(1: [cs0, cs1, cs2, cs3]) > .set_parallel() > > This tells the scheduler any two of the four possible engines can be used. cs0 + cs3 is fine, cs3 + cs1 also, ... any. Only implicit rule is they have to be different and that works for all. > > 2-wide "implicit bonds mode" aka media fixed function limitation: > > .engine_map([-1, -1]) > .load_balance(0: [cs0, cs2]) > .load_balance(1: [cs1, cs3]) > .set_parallel(flags = implicit_bond) > > Think of implicit flag creating a "link" between vertical columns in each virtual engine slot. So valid pairs end up cs0 + cs1 and cs2 + cs3 only. > > You can also think of the implicit flag as a shortcut to avoid specifying bonds via the existing extension. In which case context setup would be written along the lines of: > > .engine_map([-1, -1]) > .load_balance(0: [cs0, cs2]) > .load_balance(1: [cs1, cs3]) > .bond(1: master = cs0, bond = [cs1]) > .bond(1: master = cs2, bond = [cs3]) > .set_parallel() > > So the implicit flag is just a shortcut to avoid typing the bonds. Not really needed as explained in my previous reply. Ah now I get both what this means, why it exists and where it's all come from. With the backstory makes a bunch more sense now. Thanks for explaining again. > This was at least the "old" set_parallel. I see this latest RFC changed > things a bit which I don't really follow yet. > > It's not perfect but needs to add very little (just one context > extension, on top of multi batch execbuf which is needed anyways), > doesn't need to deprecate anything, didn't require rewrites of the UMD, > and it all works today and in the future. > > I did not really like this new uapi for all the reasons I listed > already, but as not many people were seeing the advantage of not > churning on the uapi, if we are churning already I did suggests a > different idea. I mean if we are churning we might as well go full in. > So that proposal, which didn't get any traction, was along the lines of: > > .engine_map([-1]) > .load_balance_wide(0: width=2, engines=[[cs0, cs2], [cs1, cs3]]) > > This would create an explicit wide virtual engine which should work for > GuC fine I think. For execlists it may require a bit of extra glue but I > don't think too much. > > Advantage is there is one engine in the map now and it is N-wide by > definition. > > Since no one did bite on that idea back then, I didn't really pursue is > to see if it works for all use cases. But I think it should even if it > probably requires further thinking to be sure. > > If we apply it to compute use case.. > > .engine_map([-1]) > .load_balance_wide(0: width=2, engines=[[cs0, cs1, cs2, cs3], [cs0, cs1, cs2, cs3]]) > > This means the only implicit wart in there is that cs0 + cs0 obviously > shouldn't be picked. But that should be fine both for execlists and > hopefully for the GuC. Yeah. Another option would be to simply allow any valid pair to be listed. Gets maybe a bit too long for full combinatorials. Or we do an N-out-of-M load balance, and you just specifiy the one vector for the engine set that gets fully combined. Either way I think simple to add if/when compute comes around and asks for it. -Daniel
On Thu, May 20, 2021 at 08:10:59AM -0700, Matthew Brost wrote: > On Thu, May 20, 2021 at 11:54:25AM +0200, Daniel Vetter wrote: > > On Wed, May 19, 2021 at 7:19 PM Matthew Brost <matthew.brost@intel.com> wrote: > > > > > > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: > > > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: > > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > > > v2: > > > > > (Daniel Vetter): > > > > > - Expand logical order explaination > > > > > - Add dummy header > > > > > - Only allow N BBs in execbuf IOCTL > > > > > - Configure parallel submission per slot not per gem context > > > > > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > > > Cc: Tony Ye <tony.ye@intel.com> > > > > > CC: Carl Zhang <carl.zhang@intel.com> > > > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > > > --- > > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > new file mode 100644 > > > > > index 000000000000..8c64b983ccad > > > > > --- /dev/null > > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > @@ -0,0 +1,144 @@ > > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > > > + > > > > > +/* > > > > > + * i915_context_engines_parallel_submit: > > > > > + * > > > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > > > + * the slots configuration). > > > > > + * > > > > > + * Their are two currently defined ways to control the placement of the > > > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > > > + * future as new hardware / use cases arise. Details of how to use this > > > > > + * interface below above the flags. > > > > > + * > > > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > > > + * placement configuration isn't supported on the platform / submission > > > > > + * interface. > > > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > > > + * inteface. > > > > > + */ > > > > > +struct i915_context_engines_parallel_submit { > > > > > + struct i915_user_extension base; > > > > > + > > > > > + __u16 engine_index; /* slot for parallel engine */ > > > > > + __u16 width; /* number of contexts per parallel engine */ > > > > > + __u16 num_siblings; /* number of siblings per context */ > > > > > + __u16 mbz16; > > > > > > > > Ok the big picture looks reasonable now, the flags still confuse me. > > > > > > > > > > Yea, it is a bit confusing. > > > > > > > > +/* > > > > > + * Default placement behvavior (currently unsupported): > > > > > + * > > > > > + * Rather than restricting parallel submission to a single class with a > > > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > > > + * enables parallel submission across multiple engine classes. In this case each > > > > > + * context's logical engine mask indicates where that context can placed. It is > > > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > > > + * if one context is running CS0 no other contexts can run on CS0). > > > > > + * > > > > > + * Example 1 pseudo code: > > > > > + * CSX[Y] = engine class X, logical instance Y > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS0[0], CS1[0] > > > > > + * CS0[0], CS1[1] > > > > > + * CS0[1], CS1[0] > > > > > + * CS0[1], CS1[1] > > > > > + * > > > > > + * This can also be though of as 2 virtual engines: > > > > > + * VE[0] = CS0[0], CS0[1] > > > > > + * VE[1] = CS1[0], CS1[1] > > > > > + * > > > > > + * Example 2 pseudo code: > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS[0], CS[1] > > > > > + * CS[0], CS[2] > > > > > + * CS[1], CS[0] > > > > > + * CS[1], CS[2] > > > > > + * CS[2], CS[0] > > > > > + * CS[2], CS[1] > > > > > + * > > > > > + * > > > > > + * This can also be though of as 2 virtual engines: > > > > > + * VE[0] = CS[0], CS[1], CS[2] > > > > > + * VE[1] = CS[0], CS[1], CS[2] > > > > > + > > > > > + * This enables a use case where all engines are created equally, we don't care > > > > > + * where they are scheduled, we just want a certain number of resources, for > > > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > > > + * engine classes. > > > > > + */ > > > > > > > > So I don't really get what this does compared to setting the flag below. > > > > Is this just about running the batchbuffers the wrong way round, i.e. if > > > > you have (simplest case) > > > > > > > > width=2, num_sibglings=1, engines=CS[0], CS[1] > > > > > > > > Then both > > > > CS[0], CS[1] > > > > and > > > > CS[1], CS[0] > > > > are possible options for running 2 batches? Iow, the backend is allowed to > > > > run the batchbuffers the wrong way round, which gains us nothing, since we > > > > assume the batches take equally long and engines interchangeable. There is > > > > no scheduling scenario where this additional flexibility can help. > > > > > > > > Also we don't have flags to select the only available and then specify an > > > > entire pipe dream about what the non-flag mode does, without an > > > > implementation. What is this about? > > > > > > > > If it's just "because bonded allowed this" then I think we should just > > > > unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but > > > > welp. > > > > > > > > > > High level the flags came out of internal discussions how this interface > > > should look. The default placement behavior is theoretically possible > > > with execlists but has no use cases. The GuC supports / current use > > > cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. > > > > > > Argued about for months and this is where we landed. At the end of the > > > day I think we needed to show that this interface supports more > > > placement rules than what the GuC supports / current use cases to future > > > proof this interface. > > > > > > For what is it worth it seems kinda backwards that we landed on the > > > default behavior not being supported in our current stack / HW. > > > > Yeah I think that should be inverted, doesn't make sense. > > > > What I still don't get (and I've read Tvrtko's reply with the example) > > is what exactly is the difference between implicit and not implicit > > mode? Can you do a single example where the only difference is whether > > this flag is set, and then explain with that what are the actual > > differences in scheduling options that the backend is allowed to pick > > for the set of N patches? > > > > I'm feeling a bit dense, but I'm really not seeing what's even going on here :-( > > > > > > > + > > > > > +/* > > > > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > > > > + * Each context must have the same number sibling and bonds are implictly create > > > > > + * of the siblings. > > > > > + * > > > > > + * All of the below examples are in logical space. > > > > > + * > > > > > + * Example 1 pseudo code: > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > > > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS[0], CS[1] > > > > > + * > > > > > + * Example 2 pseudo code: > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS[0], CS[1] > > > > > + * CS[2], CS[3] > > > > > + * > > > > > + * This can also be though of as 2 virtual engines: > > > > > + * VE[0] = CS[0], CS[2] > > > > > + * VE[1] = CS[1], CS[3] > > > > > + * > > > > > + * This enables a use case where all engines are not equal and certain placement > > > > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > > > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > > > > + * case (logically contiguous placement, within a single engine class) is > > > > > + * supported when using GuC submission. Execlist mode could support all possible > > > > > + * bonding configurations but currently doesn't support this extension. > > > > > + */ > > > > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > > > > > > > Does this map to intel_context_set_nopreempt(), so we could implement it > > > > also in execlist? > > > > > > > > > > intel_context_set_nopreempt is preempt is part of how this would be > > > implemented for execlists. For GuC submission there is a algorithm > > > between the i915 and GuC that inserts same preemption points between > > > each set of N batches. I believe the rules are if context 1 can > > > preempted the rest of the contexts (2 - N) can safely be preempted. A go > > > / join algorithm, implemented with semaphores, around the batches > > > makes sure the i915 adheres to these rules. > > > > > > > Also is this just an artifact of the implementation, or is this somehow > > > > required functionality for userspace, i.e. the workload fails if e.g. GuC > > > > decides to preempt all LRC of a parallel virtual engine? Of course just > > > > preempting one is a bit a bug (but execlist I think happily does that). > > > > > > > > > > I think it is part of the implementation / HW limitations. Basically if > > > N batches are running in parallel if one of the batches gets preempted > > > it can hang all the other batches even if it gets restarted. Again the > > > idea behind this is only preempt any of these contexts between each set > > > of N batches. The GuC has no concept of batches only contexts hence the > > > aforementioned algorithm to insert same preemption points between each > > > set of batches. > > > > Well backend preempting one but not the other is a bug. I don't think > > we need to talk about bugs :-) > > > > I think it is more that backend can't preempt each context atomically as these > are running on independent engines. Let say the backend tries to preempt 2 > engines at the same time and only 1 responds, now we are broken with certain > batches. With the algorithm implmented between the i915 and the Guc the GuC will > try to preempt the first context if that works, all the other contexts can be > preempted. Ah right, we might have a workload with long non-preemptible sections, but I guess the MI_SEMAPHORE_WAIT can always preempt. So if GuC just tries on one of these, it will result in tears. > > Now wrt a preempt mode in context creation, I think that makes sense. > > I'm just wondering whether this is correct here in the parallel > > submission (and why), or is it just an implementation artifact > > (current backends are buggy if we'd allow them to preempt) or what's > > going on. > > > > If this is just a case of "this is how current backends work, > > userspace doesn't actually care whether we allow preempt or not" then > > I think we should just document the behavior and that's it. Adding > > uapi for an option for which there's not even an implementation isn't > > great. > > > > I don't think it is a case of 'this is how current backends work...', I think > user space knows if it is ok for 1 of the batches to preempted and not the > others without breaking the batches. Yeah this makes more sense as an explanation: Currently the backend (guc or execlist) figure out whether something can be preempted by just trying. Since we're dealing with N contexts that's a suboptimal approach, and we need a flag that indicates whether it's worth it or not. So makes sense to me to have this. I guess userspace does actually need both modes for now, depending upon what it's doing? -Daniel > > Matt > > > But if we can actually choose, and userspace wants to, then I'm all > > fine with this. And we can always move this to a more generic place if > > other engines need preempt control later on too. > > > > Cheers, Daniel > > > > > > > Matt > > > > > > > Cheers, Daniel > > > > > > > > > +/* > > > > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > > > > + * points on all hardware contexts between each set of BBs. An example use case > > > > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > > > > + * BB must be submitted on each hardware context in the parallel gem context. > > > > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > > > > + */ > > > > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > > > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > > > > + __u64 flags; /* all undefined flags must be zero */ > > > > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > > > > + > > > > > + /* > > > > > + * width (i) * num_siblings (j) in length > > > > > + * index = j + i * num_siblings > > > > > + */ > > > > > + struct i915_engine_class_instance engines[0]; > > > > > +} __attribute__ ((packed)); > > > > > + > > > > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > > > > index 7faa46cde088..64c539486ee4 100644 > > > > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > > > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > > > > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > > > > > > > > > New parallel submission uAPI > > > > > ============================ > > > > > -Details to come in a following patch. > > > > > +The existing bonding uAPI is completely broken with GuC submission because > > > > > +whether a submission is a single context submit or parallel submit isn't known > > > > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > > > > +contexts in parallel with the GuC the context must be explictly registered with > > > > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > > > > +These interfaces doesn't support dynamically changing between N contexts as the > > > > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > > > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > > > > + > > > > > +The new parallel submission uAPI consists of 3 parts: > > > > > + > > > > > +* Export engines logical mapping > > > > > +* A 'set_parallel' extension to configure contexts for parallel > > > > > + submission > > > > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > > + > > > > > +Export engines logical mapping > > > > > +------------------------------ > > > > > +Certain use cases require BBs to be placed on engine instances in logical order > > > > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > > > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > > > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > > > > +submission interface currently only supports submitting multiple contexts to > > > > > +engines in logical order which is a new requirement compared to execlists. > > > > > +Lastly, all current platforms have at most 2 instances and the logical order is > > > > > +the same a uABI order. This will change on platforms with more than 2 instances. > > > > > + > > > > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > > > > +logical instance has been returned and a new field, > > > > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > > > > + > > > > > +A 'set_parallel' extension to configure contexts for parallel submission > > > > > +------------------------------------------------------------------------ > > > > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > > > > +It is setup step that should be called before using any of the contexts. See > > > > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > > > > +similar existing examples. Once a slot is configured for parallel submission the > > > > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > > > > +support GuC submission. Execlist support can be added later if needed. > > > > > + > > > > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > > > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > > > > + > > > > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > > +------------------------------------------------------------------- > > > > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > > > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > > > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > > > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > > > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > > > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > > > > +the IOCTL. > > > > > -- > > > > > 2.28.0 > > > > > > > > > > _______________________________________________ > > > > > Intel-gfx mailing list > > > > > Intel-gfx@lists.freedesktop.org > > > > > https://lists.freedesktop.org/mailman/listinfo/intel-gfx > > > > > > > > -- > > > > Daniel Vetter > > > > Software Engineer, Intel Corporation > > > > http://blog.ffwll.ch > > > > > > > > -- > > Daniel Vetter > > Software Engineer, Intel Corporation > > http://blog.ffwll.ch
On Thu, May 20, 2021 at 10:46 AM Matthew Brost <matthew.brost@intel.com> wrote: > > On Thu, May 20, 2021 at 01:11:59PM +0200, Christian König wrote: > > Am 19.05.21 um 18:51 schrieb Matthew Brost: > > > On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote: > > > > Oh, yeah we call that gang submit on the AMD side. > > > > > > > > Had already some internal discussions how to implement this, but so far > > > > couldn't figure out how to cleanly introduce that into the DRM scheduler. > > > > > > > > Can you briefly describe in a few words how that is supposed to work on the > > > > Intel side? On Intel, we actually have two cases which don't fit the current drm/scheduler model well: balanced and bonded. In the balanced model, we want to submit a batch which can go to any one of some set of engines and we don't care which. It's up to the kernel to pick an engine. Imagine you had 64 identical HW compute queues, for instance. This could be done by making all the identical engines share a single drm_gpu_scheduler and round-robin around the HW queues or something. I don't know that we strictly need drm/scheduler to be aware of it but it might be nice if it grew support for this mode so we could maintain a 1:1 relationship between HW queues and drm_gpu_schedulers. That said, I'm not sure how this would play with GuC queues so maybe it doesn't help? The bonded model is like your ganged, I think. We want to submit N batches to run in parallel. And they actually have to be executing on the GPU simultaneously and not just sort-of at similar times. We need this for video. There are also potential use-cases in Vulkan or even GL that might be able to use this. One difference with the balanced mode is that bonds don't, strictly speaking, need to be on the same type of engine. Imagine, for instance, a 3D batch with a parallel compute batch doing vertex pre-processing. I'm pretty sure the bonded case is something that the mobile drivers (panfrost, etc.) would like as well for doing Vulkan on tilers where you often have to have two command buffers running in parallel. They're currently doing it by submitting a giant pile of batches where they split the batch and add sync primitives every time some GL call requires them to sync between fragment and vertex pipes. So, to sum up, I think there's likely some good collaboration to be had here for everyone. :-) --Jason > > > Sure, I've done a quick PoC internally and have been able to hook this > > > into the DRM scheduler. > > > > > > Basically each BB still maps to a single job as each job is somewhat > > > unique (e.g. each job has its own ring, lrc, seqno, etc...). However all > > > the jobs configured to run in parallel map to a single sched_entity > > > which maintains the order each job was generated from the execbuf IOCTL > > > (1 - N). When the backend receives jobs 1 to N - 1 it basically just > > > updates some internal state. When the backend sees job N (last job) it > > > actually does the submit for jobs 1 - N which with GuC submission is a > > > simple command moving the LRC tail of the N jobs. > > > > > > Daniel has suggested that we create a single job for the NN BBs but that > > > would be huge rework to the internals of the i915 and likely won't > > > happen by the time this code first lands. > > > > > > Also worth noting one way a job isn't really a treated individually is > > > the excl slot with dma-resv. In that case we create a composite fence of > > > all jobs (dma_fence_array). > > > > Yeah, that's something we have discussed as well. > > > > How do you prevent the scheduler from over committing to a single ring > > buffer in this scenario? > > > > Each job has its own ring, the execbuf IOCTL throttles itself for each > job if there isn't space in the ring. This is exactly the same as > non-parallel submits. > > I think this is what you were asking? If not, maybe try explaining the > question a bit more. > > Matt > > > Christian. > > > > > > > > Matt > > > > > > > Thanks, > > > > Christian. > > > > > > > > Am 19.05.21 um 01:58 schrieb Matthew Brost: > > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > > > v2: > > > > > (Daniel Vetter): > > > > > - Expand logical order explaination > > > > > - Add dummy header > > > > > - Only allow N BBs in execbuf IOCTL > > > > > - Configure parallel submission per slot not per gem context > > > > > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > > > Cc: Tony Ye <tony.ye@intel.com> > > > > > CC: Carl Zhang <carl.zhang@intel.com> > > > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > > > --- > > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > new file mode 100644 > > > > > index 000000000000..8c64b983ccad > > > > > --- /dev/null > > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > @@ -0,0 +1,144 @@ > > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > > > + > > > > > +/* > > > > > + * i915_context_engines_parallel_submit: > > > > > + * > > > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > > > + * the slots configuration). > > > > > + * > > > > > + * Their are two currently defined ways to control the placement of the > > > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > > > + * future as new hardware / use cases arise. Details of how to use this > > > > > + * interface below above the flags. > > > > > + * > > > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > > > + * placement configuration isn't supported on the platform / submission > > > > > + * interface. > > > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > > > + * inteface. > > > > > + */ > > > > > +struct i915_context_engines_parallel_submit { > > > > > + struct i915_user_extension base; > > > > > + > > > > > + __u16 engine_index; /* slot for parallel engine */ > > > > > + __u16 width; /* number of contexts per parallel engine */ > > > > > + __u16 num_siblings; /* number of siblings per context */ > > > > > + __u16 mbz16; > > > > > +/* > > > > > + * Default placement behvavior (currently unsupported): > > > > > + * > > > > > + * Rather than restricting parallel submission to a single class with a > > > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > > > + * enables parallel submission across multiple engine classes. In this case each > > > > > + * context's logical engine mask indicates where that context can placed. It is > > > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > > > + * if one context is running CS0 no other contexts can run on CS0). > > > > > + * > > > > > + * Example 1 pseudo code: > > > > > + * CSX[Y] = engine class X, logical instance Y > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS0[0], CS1[0] > > > > > + * CS0[0], CS1[1] > > > > > + * CS0[1], CS1[0] > > > > > + * CS0[1], CS1[1] > > > > > + * > > > > > + * This can also be though of as 2 virtual engines: > > > > > + * VE[0] = CS0[0], CS0[1] > > > > > + * VE[1] = CS1[0], CS1[1] > > > > > + * > > > > > + * Example 2 pseudo code: > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS[0], CS[1] > > > > > + * CS[0], CS[2] > > > > > + * CS[1], CS[0] > > > > > + * CS[1], CS[2] > > > > > + * CS[2], CS[0] > > > > > + * CS[2], CS[1] > > > > > + * > > > > > + * > > > > > + * This can also be though of as 2 virtual engines: > > > > > + * VE[0] = CS[0], CS[1], CS[2] > > > > > + * VE[1] = CS[0], CS[1], CS[2] > > > > > + > > > > > + * This enables a use case where all engines are created equally, we don't care > > > > > + * where they are scheduled, we just want a certain number of resources, for > > > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > > > + * engine classes. > > > > > + */ > > > > > + > > > > > +/* > > > > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > > > > + * Each context must have the same number sibling and bonds are implictly create > > > > > + * of the siblings. > > > > > + * > > > > > + * All of the below examples are in logical space. > > > > > + * > > > > > + * Example 1 pseudo code: > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > > > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS[0], CS[1] > > > > > + * > > > > > + * Example 2 pseudo code: > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > + * set_engines(INVALID) > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > > + * > > > > > + * Results in the following valid placements: > > > > > + * CS[0], CS[1] > > > > > + * CS[2], CS[3] > > > > > + * > > > > > + * This can also be though of as 2 virtual engines: > > > > > + * VE[0] = CS[0], CS[2] > > > > > + * VE[1] = CS[1], CS[3] > > > > > + * > > > > > + * This enables a use case where all engines are not equal and certain placement > > > > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > > > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > > > > + * case (logically contiguous placement, within a single engine class) is > > > > > + * supported when using GuC submission. Execlist mode could support all possible > > > > > + * bonding configurations but currently doesn't support this extension. > > > > > + */ > > > > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > > > > +/* > > > > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > > > > + * points on all hardware contexts between each set of BBs. An example use case > > > > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > > > > + * BB must be submitted on each hardware context in the parallel gem context. > > > > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > > > > + */ > > > > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > > > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > > > > + __u64 flags; /* all undefined flags must be zero */ > > > > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > > > > + > > > > > + /* > > > > > + * width (i) * num_siblings (j) in length > > > > > + * index = j + i * num_siblings > > > > > + */ > > > > > + struct i915_engine_class_instance engines[0]; > > > > > +} __attribute__ ((packed)); > > > > > + > > > > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > > > > index 7faa46cde088..64c539486ee4 100644 > > > > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > > > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > > > > @@ -82,4 +82,55 @@ https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspec.oneapi.com%2Flevel-zero%2Flatest%2Fcore%2Fapi.html%23ze-command-queue-priorit&data=04%7C01%7Cchristian.koenig%40amd.com%7C49a7557f4e494090755608d91ae758a6%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637570403202969375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dn3PVdxYQpkpWIru5eAXHgbfuLDkppAA5daV5sHQF7s%3D&reserved=0 > > > > > New parallel submission uAPI > > > > > ============================ > > > > > -Details to come in a following patch. > > > > > +The existing bonding uAPI is completely broken with GuC submission because > > > > > +whether a submission is a single context submit or parallel submit isn't known > > > > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > > > > +contexts in parallel with the GuC the context must be explictly registered with > > > > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > > > > +These interfaces doesn't support dynamically changing between N contexts as the > > > > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > > > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > > > > + > > > > > +The new parallel submission uAPI consists of 3 parts: > > > > > + > > > > > +* Export engines logical mapping > > > > > +* A 'set_parallel' extension to configure contexts for parallel > > > > > + submission > > > > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > > + > > > > > +Export engines logical mapping > > > > > +------------------------------ > > > > > +Certain use cases require BBs to be placed on engine instances in logical order > > > > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > > > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > > > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > > > > +submission interface currently only supports submitting multiple contexts to > > > > > +engines in logical order which is a new requirement compared to execlists. > > > > > +Lastly, all current platforms have at most 2 instances and the logical order is > > > > > +the same a uABI order. This will change on platforms with more than 2 instances. > > > > > + > > > > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > > > > +logical instance has been returned and a new field, > > > > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > > > > + > > > > > +A 'set_parallel' extension to configure contexts for parallel submission > > > > > +------------------------------------------------------------------------ > > > > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > > > > +It is setup step that should be called before using any of the contexts. See > > > > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > > > > +similar existing examples. Once a slot is configured for parallel submission the > > > > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > > > > +support GuC submission. Execlist support can be added later if needed. > > > > > + > > > > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > > > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > > > > + > > > > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > > +------------------------------------------------------------------- > > > > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > > > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > > > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > > > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > > > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > > > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > > > > +the IOCTL. > > > _______________________________________________ > Intel-gfx mailing list > Intel-gfx@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Am 20.05.21 um 23:38 schrieb Jason Ekstrand: > On Thu, May 20, 2021 at 10:46 AM Matthew Brost <matthew.brost@intel.com> wrote: >> On Thu, May 20, 2021 at 01:11:59PM +0200, Christian König wrote: >>> Am 19.05.21 um 18:51 schrieb Matthew Brost: >>>> On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote: >>>>> Oh, yeah we call that gang submit on the AMD side. >>>>> >>>>> Had already some internal discussions how to implement this, but so far >>>>> couldn't figure out how to cleanly introduce that into the DRM scheduler. >>>>> >>>>> Can you briefly describe in a few words how that is supposed to work on the >>>>> Intel side? > On Intel, we actually have two cases which don't fit the current > drm/scheduler model well: balanced and bonded. > > In the balanced model, we want to submit a batch which can go to any > one of some set of engines and we don't care which. It's up to the > kernel to pick an engine. Imagine you had 64 identical HW compute > queues, for instance. This could be done by making all the identical > engines share a single drm_gpu_scheduler and round-robin around the HW > queues or something. I don't know that we strictly need drm/scheduler > to be aware of it but it might be nice if it grew support for this > mode so we could maintain a 1:1 relationship between HW queues and > drm_gpu_schedulers. That said, I'm not sure how this would play with > GuC queues so maybe it doesn't help? Oh, we do have support for load balancing like that. When you call drm_sched_entity_init() you can give a list of drm_gpu_scheduler object to use round robing for scheduling. New jobs are then scheduler to the drm_gpu_scheduler instance which is idle or rather the least busy one. > The bonded model is like your ganged, I think. We want to submit N > batches to run in parallel. And they actually have to be executing on > the GPU simultaneously and not just sort-of at similar times. We need > this for video. There are also potential use-cases in Vulkan or even > GL that might be able to use this. One difference with the balanced > mode is that bonds don't, strictly speaking, need to be on the same > type of engine. Imagine, for instance, a 3D batch with a parallel > compute batch doing vertex pre-processing. > > I'm pretty sure the bonded case is something that the mobile drivers > (panfrost, etc.) would like as well for doing Vulkan on tilers where > you often have to have two command buffers running in parallel. > They're currently doing it by submitting a giant pile of batches where > they split the batch and add sync primitives every time some GL call > requires them to sync between fragment and vertex pipes. Yeah, we have exactly the same problem as well. But so far every model we discussed has some drawbacks and it is rather hard for the scheduler to guarantee that stuff runs at the same time. So if you got any ideas how to cleanly implement that then they would be rather welcomed. Regards, Christian. > > So, to sum up, I think there's likely some good collaboration to be > had here for everyone. :-) > > --Jason > >>>> Sure, I've done a quick PoC internally and have been able to hook this >>>> into the DRM scheduler. >>>> >>>> Basically each BB still maps to a single job as each job is somewhat >>>> unique (e.g. each job has its own ring, lrc, seqno, etc...). However all >>>> the jobs configured to run in parallel map to a single sched_entity >>>> which maintains the order each job was generated from the execbuf IOCTL >>>> (1 - N). When the backend receives jobs 1 to N - 1 it basically just >>>> updates some internal state. When the backend sees job N (last job) it >>>> actually does the submit for jobs 1 - N which with GuC submission is a >>>> simple command moving the LRC tail of the N jobs. >>>> >>>> Daniel has suggested that we create a single job for the NN BBs but that >>>> would be huge rework to the internals of the i915 and likely won't >>>> happen by the time this code first lands. >>>> >>>> Also worth noting one way a job isn't really a treated individually is >>>> the excl slot with dma-resv. In that case we create a composite fence of >>>> all jobs (dma_fence_array). >>> Yeah, that's something we have discussed as well. >>> >>> How do you prevent the scheduler from over committing to a single ring >>> buffer in this scenario? >>> >> Each job has its own ring, the execbuf IOCTL throttles itself for each >> job if there isn't space in the ring. This is exactly the same as >> non-parallel submits. >> >> I think this is what you were asking? If not, maybe try explaining the >> question a bit more. >> >> Matt >> >>> Christian. >>> >>>> Matt >>>> >>>>> Thanks, >>>>> Christian. >>>>> >>>>> Am 19.05.21 um 01:58 schrieb Matthew Brost: >>>>>> Add entry fpr i915 new parallel submission uAPI plan. >>>>>> >>>>>> v2: >>>>>> (Daniel Vetter): >>>>>> - Expand logical order explaination >>>>>> - Add dummy header >>>>>> - Only allow N BBs in execbuf IOCTL >>>>>> - Configure parallel submission per slot not per gem context >>>>>> >>>>>> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> >>>>>> Cc: Tony Ye <tony.ye@intel.com> >>>>>> CC: Carl Zhang <carl.zhang@intel.com> >>>>>> Cc: Daniel Vetter <daniel.vetter@intel.com> >>>>>> Cc: Jason Ekstrand <jason@jlekstrand.net> >>>>>> Signed-off-by: Matthew Brost <matthew.brost@intel.com> >>>>>> --- >>>>>> Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ >>>>>> Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- >>>>>> 2 files changed, 196 insertions(+), 1 deletion(-) >>>>>> create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h >>>>>> >>>>>> diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h >>>>>> new file mode 100644 >>>>>> index 000000000000..8c64b983ccad >>>>>> --- /dev/null >>>>>> +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h >>>>>> @@ -0,0 +1,144 @@ >>>>>> +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ >>>>>> + >>>>>> +/* >>>>>> + * i915_context_engines_parallel_submit: >>>>>> + * >>>>>> + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. >>>>>> + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple >>>>>> + * hardware contexts are created internally in the i915 run these BBs. Once a >>>>>> + * slot is configured for N BBs only N BBs can be submitted in each execbuf >>>>>> + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf >>>>>> + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on >>>>>> + * the slots configuration). >>>>>> + * >>>>>> + * Their are two currently defined ways to control the placement of the >>>>>> + * hardware contexts on physical engines: default behavior (no flags) and >>>>>> + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the >>>>>> + * future as new hardware / use cases arise. Details of how to use this >>>>>> + * interface below above the flags. >>>>>> + * >>>>>> + * Returns -EINVAL if hardware context placement configuration invalid or if the >>>>>> + * placement configuration isn't supported on the platform / submission >>>>>> + * interface. >>>>>> + * Returns -ENODEV if extension isn't supported on the platform / submission >>>>>> + * inteface. >>>>>> + */ >>>>>> +struct i915_context_engines_parallel_submit { >>>>>> + struct i915_user_extension base; >>>>>> + >>>>>> + __u16 engine_index; /* slot for parallel engine */ >>>>>> + __u16 width; /* number of contexts per parallel engine */ >>>>>> + __u16 num_siblings; /* number of siblings per context */ >>>>>> + __u16 mbz16; >>>>>> +/* >>>>>> + * Default placement behvavior (currently unsupported): >>>>>> + * >>>>>> + * Rather than restricting parallel submission to a single class with a >>>>>> + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that >>>>>> + * enables parallel submission across multiple engine classes. In this case each >>>>>> + * context's logical engine mask indicates where that context can placed. It is >>>>>> + * implied in this mode that all contexts have mutual exclusive placement (e.g. >>>>>> + * if one context is running CS0 no other contexts can run on CS0). >>>>>> + * >>>>>> + * Example 1 pseudo code: >>>>>> + * CSX[Y] = engine class X, logical instance Y >>>>>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>>>>> + * set_engines(INVALID) >>>>>> + * set_parallel(engine_index=0, width=2, num_siblings=2, >>>>>> + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) >>>>>> + * >>>>>> + * Results in the following valid placements: >>>>>> + * CS0[0], CS1[0] >>>>>> + * CS0[0], CS1[1] >>>>>> + * CS0[1], CS1[0] >>>>>> + * CS0[1], CS1[1] >>>>>> + * >>>>>> + * This can also be though of as 2 virtual engines: >>>>>> + * VE[0] = CS0[0], CS0[1] >>>>>> + * VE[1] = CS1[0], CS1[1] >>>>>> + * >>>>>> + * Example 2 pseudo code: >>>>>> + * CS[X] = generic engine of same class, logical instance X >>>>>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>>>>> + * set_engines(INVALID) >>>>>> + * set_parallel(engine_index=0, width=2, num_siblings=3, >>>>>> + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) >>>>>> + * >>>>>> + * Results in the following valid placements: >>>>>> + * CS[0], CS[1] >>>>>> + * CS[0], CS[2] >>>>>> + * CS[1], CS[0] >>>>>> + * CS[1], CS[2] >>>>>> + * CS[2], CS[0] >>>>>> + * CS[2], CS[1] >>>>>> + * >>>>>> + * >>>>>> + * This can also be though of as 2 virtual engines: >>>>>> + * VE[0] = CS[0], CS[1], CS[2] >>>>>> + * VE[1] = CS[0], CS[1], CS[2] >>>>>> + >>>>>> + * This enables a use case where all engines are created equally, we don't care >>>>>> + * where they are scheduled, we just want a certain number of resources, for >>>>>> + * those resources to be scheduled in parallel, and possibly across multiple >>>>>> + * engine classes. >>>>>> + */ >>>>>> + >>>>>> +/* >>>>>> + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. >>>>>> + * Each context must have the same number sibling and bonds are implictly create >>>>>> + * of the siblings. >>>>>> + * >>>>>> + * All of the below examples are in logical space. >>>>>> + * >>>>>> + * Example 1 pseudo code: >>>>>> + * CS[X] = generic engine of same class, logical instance X >>>>>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>>>>> + * set_engines(INVALID) >>>>>> + * set_parallel(engine_index=0, width=2, num_siblings=1, >>>>>> + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) >>>>>> + * >>>>>> + * Results in the following valid placements: >>>>>> + * CS[0], CS[1] >>>>>> + * >>>>>> + * Example 2 pseudo code: >>>>>> + * CS[X] = generic engine of same class, logical instance X >>>>>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>>>>> + * set_engines(INVALID) >>>>>> + * set_parallel(engine_index=0, width=2, num_siblings=2, >>>>>> + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) >>>>>> + * >>>>>> + * Results in the following valid placements: >>>>>> + * CS[0], CS[1] >>>>>> + * CS[2], CS[3] >>>>>> + * >>>>>> + * This can also be though of as 2 virtual engines: >>>>>> + * VE[0] = CS[0], CS[2] >>>>>> + * VE[1] = CS[1], CS[3] >>>>>> + * >>>>>> + * This enables a use case where all engines are not equal and certain placement >>>>>> + * rules are required (i.e. split-frame requires all contexts to be placed in a >>>>>> + * logically contiguous order on the VCS engines on gen11+ platforms). This use >>>>>> + * case (logically contiguous placement, within a single engine class) is >>>>>> + * supported when using GuC submission. Execlist mode could support all possible >>>>>> + * bonding configurations but currently doesn't support this extension. >>>>>> + */ >>>>>> +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) >>>>>> +/* >>>>>> + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption >>>>>> + * points on all hardware contexts between each set of BBs. An example use case >>>>>> + * of this feature is split-frame on gen11+ hardware. When using this feature a >>>>>> + * BB must be submitted on each hardware context in the parallel gem context. >>>>>> + * The execbuf2 IOCTL enforces the user adheres to policy. >>>>>> + */ >>>>>> +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) >>>>>> +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) >>>>>> + __u64 flags; /* all undefined flags must be zero */ >>>>>> + __u64 mbz64[3]; /* reserved for future use; must be zero */ >>>>>> + >>>>>> + /* >>>>>> + * width (i) * num_siblings (j) in length >>>>>> + * index = j + i * num_siblings >>>>>> + */ >>>>>> + struct i915_engine_class_instance engines[0]; >>>>>> +} __attribute__ ((packed)); >>>>>> + >>>>>> diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst >>>>>> index 7faa46cde088..64c539486ee4 100644 >>>>>> --- a/Documentation/gpu/rfc/i915_scheduler.rst >>>>>> +++ b/Documentation/gpu/rfc/i915_scheduler.rst >>>>>> @@ -82,4 +82,55 @@ https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspec.oneapi.com%2Flevel-zero%2Flatest%2Fcore%2Fapi.html%23ze-command-queue-priorit&data=04%7C01%7Cchristian.koenig%40amd.com%7C49a7557f4e494090755608d91ae758a6%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637570403202969375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dn3PVdxYQpkpWIru5eAXHgbfuLDkppAA5daV5sHQF7s%3D&reserved=0 >>>>>> New parallel submission uAPI >>>>>> ============================ >>>>>> -Details to come in a following patch. >>>>>> +The existing bonding uAPI is completely broken with GuC submission because >>>>>> +whether a submission is a single context submit or parallel submit isn't known >>>>>> +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple >>>>>> +contexts in parallel with the GuC the context must be explictly registered with >>>>>> +N contexts and all N contexts must be submitted in a single command to the GuC. >>>>>> +These interfaces doesn't support dynamically changing between N contexts as the >>>>>> +bonding uAPI does. Hence the need for a new parallel submission interface. Also >>>>>> +the legacy bonding uAPI is quite confusing and not intuitive at all. >>>>>> + >>>>>> +The new parallel submission uAPI consists of 3 parts: >>>>>> + >>>>>> +* Export engines logical mapping >>>>>> +* A 'set_parallel' extension to configure contexts for parallel >>>>>> + submission >>>>>> +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL >>>>>> + >>>>>> +Export engines logical mapping >>>>>> +------------------------------ >>>>>> +Certain use cases require BBs to be placed on engine instances in logical order >>>>>> +(e.g. split-frame on gen11+). The logical mapping of engine instances can change >>>>>> +based on fusing. Rather than making UMDs be aware of fusing, simply expose the >>>>>> +logical mapping with the existing query engine info IOCTL. Also the GuC >>>>>> +submission interface currently only supports submitting multiple contexts to >>>>>> +engines in logical order which is a new requirement compared to execlists. >>>>>> +Lastly, all current platforms have at most 2 instances and the logical order is >>>>>> +the same a uABI order. This will change on platforms with more than 2 instances. >>>>>> + >>>>>> +A single bit will be added to drm_i915_engine_info.flags indicating that the >>>>>> +logical instance has been returned and a new field, >>>>>> +drm_i915_engine_info.logical_instance, returns the logical instance. >>>>>> + >>>>>> +A 'set_parallel' extension to configure contexts for parallel submission >>>>>> +------------------------------------------------------------------------ >>>>>> +The 'set_parallel' extension configures a slot for parallel submission of N BBs. >>>>>> +It is setup step that should be called before using any of the contexts. See >>>>>> +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for >>>>>> +similar existing examples. Once a slot is configured for parallel submission the >>>>>> +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only >>>>>> +support GuC submission. Execlist support can be added later if needed. >>>>>> + >>>>>> +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and >>>>>> +i915_context_engines_parallel_submit to the uAPI to implement this extension. >>>>>> + >>>>>> +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL >>>>>> +------------------------------------------------------------------- >>>>>> +Contexts that have been configured with the 'set_parallel' extension are allowed >>>>>> +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N >>>>>> +objects in the drm_i915_gem_exec_object2 list or the first N if >>>>>> +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot >>>>>> +submitted and how it has been configured by 'set_parallel' or other extensions. >>>>>> +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of >>>>>> +the IOCTL. >> _______________________________________________ >> Intel-gfx mailing list >> Intel-gfx@lists.freedesktop.org >> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
On 19/05/2021 00:58, Matthew Brost wrote: > Add entry fpr i915 new parallel submission uAPI plan. > > v2: > (Daniel Vetter): > - Expand logical order explaination > - Add dummy header > - Only allow N BBs in execbuf IOCTL > - Configure parallel submission per slot not per gem context > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > Cc: Tony Ye <tony.ye@intel.com> > CC: Carl Zhang <carl.zhang@intel.com> > Cc: Daniel Vetter <daniel.vetter@intel.com> > Cc: Jason Ekstrand <jason@jlekstrand.net> > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > --- > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > 2 files changed, 196 insertions(+), 1 deletion(-) > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > new file mode 100644 > index 000000000000..8c64b983ccad > --- /dev/null > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > @@ -0,0 +1,144 @@ > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > + > +/* > + * i915_context_engines_parallel_submit: > + * > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > + * hardware contexts are created internally in the i915 run these BBs. Once a > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > + * the slots configuration). 1) Expand the term slot here with "slot in the context engine map" least once for clarity. 2) About where execbuf will implicitly be finding batches - suggest to also cover first/last flag here. I know you have it in the readme but I think it is good if uapi header is as self-contained as possible. > + * > + * Their are two currently defined ways to control the placement of the > + * hardware contexts on physical engines: default behavior (no flags) and > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > + * future as new hardware / use cases arise. Details of how to use this > + * interface below above the flags. > + * > + * Returns -EINVAL if hardware context placement configuration invalid or if the > + * placement configuration isn't supported on the platform / submission > + * interface. > + * Returns -ENODEV if extension isn't supported on the platform / submission > + * inteface. > + */ > +struct i915_context_engines_parallel_submit { > + struct i915_user_extension base; > + > + __u16 engine_index; /* slot for parallel engine */ > + __u16 width; /* number of contexts per parallel engine */ > + __u16 num_siblings; /* number of siblings per context */ > + __u16 mbz16; > +/* > + * Default placement behvavior (currently unsupported): > + * > + * Rather than restricting parallel submission to a single class with a > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that What do you mean with logically contiguous here? It sounds ambiguous versus logical vs "normal" engine instance numbers. > + * enables parallel submission across multiple engine classes. In this case each > + * context's logical engine mask indicates where that context can placed. It is > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > + * if one context is running CS0 no other contexts can run on CS0). I think talk about logical context and its mask is too implementation detail at the uapi level. Instead I would suggest more userspace programmer centric description. > + * > + * Example 1 pseudo code: > + * CSX[Y] = engine class X, logical instance Y > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > + * > + * Results in the following valid placements: > + * CS0[0], CS1[0] > + * CS0[0], CS1[1] > + * CS0[1], CS1[0] > + * CS0[1], CS1[1] > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS0[0], CS0[1] > + * VE[1] = CS1[0], CS1[1] Ah okay so essentially similar to what I was proposing a year ago. But then it is no longer "set_parallel" really. It is one slot in the engine map, right, with the idea to super class intel_context in the implementation? So really a wide virtual engine, as opposed to single one. In which case I think it makes sense to stay close to the existing naming of the load_balance extension for consistency. Load_balance_wide? Load_balance_parallel? Multi? I also have to say the notation "CS0[0]" - I who know this problem space am finding it hard to penetrate what that actually means. (Also uppercase IMO makes it hard to read, but maybe it is just me.) Looking a bit lower below, extension seems to be taking a 2d array of class:instance pairs, right? If so then reading these docs in order, or even just looking further down, I don't think that is explicitly called out clearly enough. So I think a paragraph or two explaining clearly how the 2d array of engines corresponds to the allowed engines for full virtual engine width. Or maybe just a 2d diagram? 2-wide virtual engine: .engines = [ /* channel 0 allowed engines: */ [cs0, cs1], /* channel 1 allowed engines: */ [cs0, cs1] ] Not sure if that's better. Also to be noted, this only allows uniform number of allowed engines per channel. I am not saying we need the non-uniform setup today but with bonds there isn't this limitation. > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=3, > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[0], CS[2] > + * CS[1], CS[0] > + * CS[1], CS[2] > + * CS[2], CS[0] > + * CS[2], CS[1] And now here you use different notation than above (CS0[0] vs CS[0]) which is confusing. > + * > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS[0], CS[1], CS[2] > + * VE[1] = CS[0], CS[1], CS[2] > + > + * This enables a use case where all engines are created equally, we don't care > + * where they are scheduled, we just want a certain number of resources, for > + * those resources to be scheduled in parallel, and possibly across multiple > + * engine classes. > + */ > + > +/* > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > + * Each context must have the same number sibling and bonds are implictly create > + * of the siblings. > + * > + * All of the below examples are in logical space. > + * > + * Example 1 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=1, > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[2], CS[3] > + * > + * This can also be though of as 2 virtual engines: > + * VE[0] = CS[0], CS[2] > + * VE[1] = CS[1], CS[3] > + * > + * This enables a use case where all engines are not equal and certain placement > + * rules are required (i.e. split-frame requires all contexts to be placed in a > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > + * case (logically contiguous placement, within a single engine class) is > + * supported when using GuC submission. Execlist mode could support all possible > + * bonding configurations but currently doesn't support this extension. > + */ > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > +/* > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > + * points on all hardware contexts between each set of BBs. An example use case > + * of this feature is split-frame on gen11+ hardware. When using this feature a > + * BB must be submitted on each hardware context in the parallel gem context. > + * The execbuf2 IOCTL enforces the user adheres to policy. > + */ Is the part about execbuf2 enforcing num_batches == context_width still valid, I mean is the latest proposal supporting num_batches < context_width? Does media needs this option? I would also have this flag as a separate patch because it strictly isn't about parallel submission per-se but a specific hw limitation on top of it. Regards, Tvrtko > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > + __u64 flags; /* all undefined flags must be zero */ > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > + > + /* > + * width (i) * num_siblings (j) in length > + * index = j + i * num_siblings > + */ > + struct i915_engine_class_instance engines[0]; > +} __attribute__ ((packed)); > + > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > index 7faa46cde088..64c539486ee4 100644 > --- a/Documentation/gpu/rfc/i915_scheduler.rst > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > New parallel submission uAPI > ============================ > -Details to come in a following patch. > +The existing bonding uAPI is completely broken with GuC submission because > +whether a submission is a single context submit or parallel submit isn't known > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > +contexts in parallel with the GuC the context must be explictly registered with > +N contexts and all N contexts must be submitted in a single command to the GuC. > +These interfaces doesn't support dynamically changing between N contexts as the > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > +the legacy bonding uAPI is quite confusing and not intuitive at all. > + > +The new parallel submission uAPI consists of 3 parts: > + > +* Export engines logical mapping > +* A 'set_parallel' extension to configure contexts for parallel > + submission > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > + > +Export engines logical mapping > +------------------------------ > +Certain use cases require BBs to be placed on engine instances in logical order > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > +logical mapping with the existing query engine info IOCTL. Also the GuC > +submission interface currently only supports submitting multiple contexts to > +engines in logical order which is a new requirement compared to execlists. > +Lastly, all current platforms have at most 2 instances and the logical order is > +the same a uABI order. This will change on platforms with more than 2 instances. > + > +A single bit will be added to drm_i915_engine_info.flags indicating that the > +logical instance has been returned and a new field, > +drm_i915_engine_info.logical_instance, returns the logical instance. > + > +A 'set_parallel' extension to configure contexts for parallel submission > +------------------------------------------------------------------------ > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > +It is setup step that should be called before using any of the contexts. See > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > +similar existing examples. Once a slot is configured for parallel submission the > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > +support GuC submission. Execlist support can be added later if needed. > + > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > + > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > +------------------------------------------------------------------- > +Contexts that have been configured with the 'set_parallel' extension are allowed > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > +objects in the drm_i915_gem_exec_object2 list or the first N if > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > +submitted and how it has been configured by 'set_parallel' or other extensions. > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > +the IOCTL. >
On 20/05/2021 20:41, Daniel Vetter wrote: > On Thu, May 20, 2021 at 11:57:44AM +0100, Tvrtko Ursulin wrote: >> >> On 20/05/2021 10:54, Daniel Vetter wrote: >>> On Wed, May 19, 2021 at 7:19 PM Matthew Brost <matthew.brost@intel.com> wrote: >>>> >>>> On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: >>>>> On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: >>>>>> Add entry fpr i915 new parallel submission uAPI plan. >>>>>> >>>>>> v2: >>>>>> (Daniel Vetter): >>>>>> - Expand logical order explaination >>>>>> - Add dummy header >>>>>> - Only allow N BBs in execbuf IOCTL >>>>>> - Configure parallel submission per slot not per gem context >>>>>> >>>>>> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> >>>>>> Cc: Tony Ye <tony.ye@intel.com> >>>>>> CC: Carl Zhang <carl.zhang@intel.com> >>>>>> Cc: Daniel Vetter <daniel.vetter@intel.com> >>>>>> Cc: Jason Ekstrand <jason@jlekstrand.net> >>>>>> Signed-off-by: Matthew Brost <matthew.brost@intel.com> >>>>>> --- >>>>>> Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ >>>>>> Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- >>>>>> 2 files changed, 196 insertions(+), 1 deletion(-) >>>>>> create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h >>>>>> >>>>>> diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h >>>>>> new file mode 100644 >>>>>> index 000000000000..8c64b983ccad >>>>>> --- /dev/null >>>>>> +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h >>>>>> @@ -0,0 +1,144 @@ >>>>>> +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ >>>>>> + >>>>>> +/* >>>>>> + * i915_context_engines_parallel_submit: >>>>>> + * >>>>>> + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. >>>>>> + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple >>>>>> + * hardware contexts are created internally in the i915 run these BBs. Once a >>>>>> + * slot is configured for N BBs only N BBs can be submitted in each execbuf >>>>>> + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf >>>>>> + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on >>>>>> + * the slots configuration). >>>>>> + * >>>>>> + * Their are two currently defined ways to control the placement of the >>>>>> + * hardware contexts on physical engines: default behavior (no flags) and >>>>>> + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the >>>>>> + * future as new hardware / use cases arise. Details of how to use this >>>>>> + * interface below above the flags. >>>>>> + * >>>>>> + * Returns -EINVAL if hardware context placement configuration invalid or if the >>>>>> + * placement configuration isn't supported on the platform / submission >>>>>> + * interface. >>>>>> + * Returns -ENODEV if extension isn't supported on the platform / submission >>>>>> + * inteface. >>>>>> + */ >>>>>> +struct i915_context_engines_parallel_submit { >>>>>> + struct i915_user_extension base; >>>>>> + >>>>>> + __u16 engine_index; /* slot for parallel engine */ >>>>>> + __u16 width; /* number of contexts per parallel engine */ >>>>>> + __u16 num_siblings; /* number of siblings per context */ >>>>>> + __u16 mbz16; >>>>> >>>>> Ok the big picture looks reasonable now, the flags still confuse me. >>>>> >>>> >>>> Yea, it is a bit confusing. >>>> >>>>>> +/* >>>>>> + * Default placement behvavior (currently unsupported): >>>>>> + * >>>>>> + * Rather than restricting parallel submission to a single class with a >>>>>> + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that >>>>>> + * enables parallel submission across multiple engine classes. In this case each >>>>>> + * context's logical engine mask indicates where that context can placed. It is >>>>>> + * implied in this mode that all contexts have mutual exclusive placement (e.g. >>>>>> + * if one context is running CS0 no other contexts can run on CS0). >>>>>> + * >>>>>> + * Example 1 pseudo code: >>>>>> + * CSX[Y] = engine class X, logical instance Y >>>>>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>>>>> + * set_engines(INVALID) >>>>>> + * set_parallel(engine_index=0, width=2, num_siblings=2, >>>>>> + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) >>>>>> + * >>>>>> + * Results in the following valid placements: >>>>>> + * CS0[0], CS1[0] >>>>>> + * CS0[0], CS1[1] >>>>>> + * CS0[1], CS1[0] >>>>>> + * CS0[1], CS1[1] >>>>>> + * >>>>>> + * This can also be though of as 2 virtual engines: >>>>>> + * VE[0] = CS0[0], CS0[1] >>>>>> + * VE[1] = CS1[0], CS1[1] >>>>>> + * >>>>>> + * Example 2 pseudo code: >>>>>> + * CS[X] = generic engine of same class, logical instance X >>>>>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>>>>> + * set_engines(INVALID) >>>>>> + * set_parallel(engine_index=0, width=2, num_siblings=3, >>>>>> + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) >>>>>> + * >>>>>> + * Results in the following valid placements: >>>>>> + * CS[0], CS[1] >>>>>> + * CS[0], CS[2] >>>>>> + * CS[1], CS[0] >>>>>> + * CS[1], CS[2] >>>>>> + * CS[2], CS[0] >>>>>> + * CS[2], CS[1] >>>>>> + * >>>>>> + * >>>>>> + * This can also be though of as 2 virtual engines: >>>>>> + * VE[0] = CS[0], CS[1], CS[2] >>>>>> + * VE[1] = CS[0], CS[1], CS[2] >>>>>> + >>>>>> + * This enables a use case where all engines are created equally, we don't care >>>>>> + * where they are scheduled, we just want a certain number of resources, for >>>>>> + * those resources to be scheduled in parallel, and possibly across multiple >>>>>> + * engine classes. >>>>>> + */ >>>>> >>>>> So I don't really get what this does compared to setting the flag below. >>>>> Is this just about running the batchbuffers the wrong way round, i.e. if >>>>> you have (simplest case) >>>>> >>>>> width=2, num_sibglings=1, engines=CS[0], CS[1] >>>>> >>>>> Then both >>>>> CS[0], CS[1] >>>>> and >>>>> CS[1], CS[0] >>>>> are possible options for running 2 batches? Iow, the backend is allowed to >>>>> run the batchbuffers the wrong way round, which gains us nothing, since we >>>>> assume the batches take equally long and engines interchangeable. There is >>>>> no scheduling scenario where this additional flexibility can help. >>>>> >>>>> Also we don't have flags to select the only available and then specify an >>>>> entire pipe dream about what the non-flag mode does, without an >>>>> implementation. What is this about? >>>>> >>>>> If it's just "because bonded allowed this" then I think we should just >>>>> unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but >>>>> welp. >>>>> >>>> >>>> High level the flags came out of internal discussions how this interface >>>> should look. The default placement behavior is theoretically possible >>>> with execlists but has no use cases. The GuC supports / current use >>>> cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. >>>> >>>> Argued about for months and this is where we landed. At the end of the >>>> day I think we needed to show that this interface supports more >>>> placement rules than what the GuC supports / current use cases to future >>>> proof this interface. >>>> >>>> For what is it worth it seems kinda backwards that we landed on the >>>> default behavior not being supported in our current stack / HW. >>> >>> Yeah I think that should be inverted, doesn't make sense. >>> >>> What I still don't get (and I've read Tvrtko's reply with the example) >>> is what exactly is the difference between implicit and not implicit >>> mode? Can you do a single example where the only difference is whether >>> this flag is set, and then explain with that what are the actual >>> differences in scheduling options that the backend is allowed to pick >>> for the set of N patches? >>> >>> I'm feeling a bit dense, but I'm really not seeing what's even going on here :-( >> >> 2-wide compute context: >> >> .engine_map([-1, -1]) >> .load_balance(0: [cs0, cs1, cs2, cs3]) // place virtual engine at slot 0 >> .load_balance(1: [cs0, cs1, cs2, cs3]) >> .set_parallel() >> >> This tells the scheduler any two of the four possible engines can be used. cs0 + cs3 is fine, cs3 + cs1 also, ... any. Only implicit rule is they have to be different and that works for all. >> >> 2-wide "implicit bonds mode" aka media fixed function limitation: >> >> .engine_map([-1, -1]) >> .load_balance(0: [cs0, cs2]) >> .load_balance(1: [cs1, cs3]) >> .set_parallel(flags = implicit_bond) >> >> Think of implicit flag creating a "link" between vertical columns in each virtual engine slot. So valid pairs end up cs0 + cs1 and cs2 + cs3 only. >> >> You can also think of the implicit flag as a shortcut to avoid specifying bonds via the existing extension. In which case context setup would be written along the lines of: >> >> .engine_map([-1, -1]) >> .load_balance(0: [cs0, cs2]) >> .load_balance(1: [cs1, cs3]) >> .bond(1: master = cs0, bond = [cs1]) >> .bond(1: master = cs2, bond = [cs3]) >> .set_parallel() >> >> So the implicit flag is just a shortcut to avoid typing the bonds. Not really needed as explained in my previous reply. > > Ah now I get both what this means, why it exists and where it's all come > from. With the backstory makes a bunch more sense now. Thanks for > explaining again. > >> This was at least the "old" set_parallel. I see this latest RFC changed >> things a bit which I don't really follow yet. >> >> It's not perfect but needs to add very little (just one context >> extension, on top of multi batch execbuf which is needed anyways), >> doesn't need to deprecate anything, didn't require rewrites of the UMD, >> and it all works today and in the future. >> >> I did not really like this new uapi for all the reasons I listed >> already, but as not many people were seeing the advantage of not >> churning on the uapi, if we are churning already I did suggests a >> different idea. I mean if we are churning we might as well go full in. >> So that proposal, which didn't get any traction, was along the lines of: >> >> .engine_map([-1]) >> .load_balance_wide(0: width=2, engines=[[cs0, cs2], [cs1, cs3]]) >> >> This would create an explicit wide virtual engine which should work for >> GuC fine I think. For execlists it may require a bit of extra glue but I >> don't think too much. >> >> Advantage is there is one engine in the map now and it is N-wide by >> definition. >> >> Since no one did bite on that idea back then, I didn't really pursue is >> to see if it works for all use cases. But I think it should even if it >> probably requires further thinking to be sure. >> >> If we apply it to compute use case.. >> >> .engine_map([-1]) >> .load_balance_wide(0: width=2, engines=[[cs0, cs1, cs2, cs3], [cs0, cs1, cs2, cs3]]) >> >> This means the only implicit wart in there is that cs0 + cs0 obviously >> shouldn't be picked. But that should be fine both for execlists and >> hopefully for the GuC. > > Yeah. Another option would be to simply allow any valid pair to be listed. > Gets maybe a bit too long for full combinatorials. Or we do an N-out-of-M > load balance, and you just specifiy the one vector for the engine set that > gets fully combined. Well main complaint against bonds were that they are too verbose. Listing all combinations is pretty much the same so... > Either way I think simple to add if/when compute comes around and asks for > it. Add what sorry lost you? Because having just read the latest proposal it seems it is actually doing pretty much what I described above. Unless I got it wrong.. Anyway, I still maintain we don't really needs to add a new context extension apart from set_parallel(bool) since bonds can express it all already. And GuC backend will have to validate the new extension anyway, as it would need to validate the bonds, since who says userspace has passed in a supported configuration even with the implicit bonds flag set. Still needs to check it.. At which point I really don't see the point in the uapi churn. All we really need is a multi batch execbuf. *If* the desire is so strong to go the single entry in engine map can be N-wide then it can be done, but actually still the GuC backend will need to validate it all makes sense for what firmware can do. So to me it's very questionable. Even the complaints about the weird stuff GuC backend does in construction-deconstruction on submit are not really an *uapi* problem. Even with just set_parallel(bool) backend could interpret it all correctly, create a single whatever it wants, and submit a single whatever it wants to the GuC. Regards, Tvrtko
On Fri, May 21, 2021 at 10:35:37AM +0200, Christian König wrote: > Am 20.05.21 um 23:38 schrieb Jason Ekstrand: > > On Thu, May 20, 2021 at 10:46 AM Matthew Brost <matthew.brost@intel.com> wrote: > > > On Thu, May 20, 2021 at 01:11:59PM +0200, Christian König wrote: > > > > Am 19.05.21 um 18:51 schrieb Matthew Brost: > > > > > On Wed, May 19, 2021 at 01:45:39PM +0200, Christian König wrote: > > > > > > Oh, yeah we call that gang submit on the AMD side. > > > > > > > > > > > > Had already some internal discussions how to implement this, but so far > > > > > > couldn't figure out how to cleanly introduce that into the DRM scheduler. > > > > > > > > > > > > Can you briefly describe in a few words how that is supposed to work on the > > > > > > Intel side? > > On Intel, we actually have two cases which don't fit the current > > drm/scheduler model well: balanced and bonded. > > > > In the balanced model, we want to submit a batch which can go to any > > one of some set of engines and we don't care which. It's up to the > > kernel to pick an engine. Imagine you had 64 identical HW compute > > queues, for instance. This could be done by making all the identical > > engines share a single drm_gpu_scheduler and round-robin around the HW > > queues or something. I don't know that we strictly need drm/scheduler > > to be aware of it but it might be nice if it grew support for this > > mode so we could maintain a 1:1 relationship between HW queues and > > drm_gpu_schedulers. That said, I'm not sure how this would play with > > GuC queues so maybe it doesn't help? > > Oh, we do have support for load balancing like that. > > When you call drm_sched_entity_init() you can give a list of > drm_gpu_scheduler object to use round robing for scheduling. > > New jobs are then scheduler to the drm_gpu_scheduler instance which is idle > or rather the least busy one. > > > The bonded model is like your ganged, I think. We want to submit N > > batches to run in parallel. And they actually have to be executing on > > the GPU simultaneously and not just sort-of at similar times. We need > > this for video. There are also potential use-cases in Vulkan or even > > GL that might be able to use this. One difference with the balanced > > mode is that bonds don't, strictly speaking, need to be on the same > > type of engine. Imagine, for instance, a 3D batch with a parallel > > compute batch doing vertex pre-processing. > > > > I'm pretty sure the bonded case is something that the mobile drivers > > (panfrost, etc.) would like as well for doing Vulkan on tilers where > > you often have to have two command buffers running in parallel. > > They're currently doing it by submitting a giant pile of batches where > > they split the batch and add sync primitives every time some GL call > > requires them to sync between fragment and vertex pipes. > > Yeah, we have exactly the same problem as well. > > But so far every model we discussed has some drawbacks and it is rather hard > for the scheduler to guarantee that stuff runs at the same time. > > So if you got any ideas how to cleanly implement that then they would be > rather welcomed. > Everything Jason said about our submission modes is correct for execlists, we have balanced + bonded models which is tightly coupled with that backend. Fortunately with GuC submission most of this complexity goes away as the GuC handles this for us. e.g. For balanced when we register a context we just give it a mask of which physical engines a context is allowed to run on. For parallel we register N contexts in a single command with the placement information + submit to all the contexts with a command which moves the tails in LRCs for us. We really don't need to bake any of this into the DRM scheduler for GuC submission. Execlists is different story but I think our plan is to do the minimum possible to plum that into the DRM scheduler (e.g. leave the balanced / bonded code in the backend). Could we update the DRM scheduler to understand balanced (seems like already does to some extent) and bonded? Yes we could but IMO the ROI on that is low for Intel. The DRM scheduler is quite clean at the moment compared to our near incomprehensible execlist backend. Execlists are basically a legacy interface so pushing features which are only required for that backend to the DRM scheduler doesn't make sense to me. That being said, if this is something AMD needs in the DRM scheduler of course we can support you. Matt > Regards, > Christian. > > > > > So, to sum up, I think there's likely some good collaboration to be > > had here for everyone. :-) > > > > --Jason > > > > > > > Sure, I've done a quick PoC internally and have been able to hook this > > > > > into the DRM scheduler. > > > > > > > > > > Basically each BB still maps to a single job as each job is somewhat > > > > > unique (e.g. each job has its own ring, lrc, seqno, etc...). However all > > > > > the jobs configured to run in parallel map to a single sched_entity > > > > > which maintains the order each job was generated from the execbuf IOCTL > > > > > (1 - N). When the backend receives jobs 1 to N - 1 it basically just > > > > > updates some internal state. When the backend sees job N (last job) it > > > > > actually does the submit for jobs 1 - N which with GuC submission is a > > > > > simple command moving the LRC tail of the N jobs. > > > > > > > > > > Daniel has suggested that we create a single job for the NN BBs but that > > > > > would be huge rework to the internals of the i915 and likely won't > > > > > happen by the time this code first lands. > > > > > > > > > > Also worth noting one way a job isn't really a treated individually is > > > > > the excl slot with dma-resv. In that case we create a composite fence of > > > > > all jobs (dma_fence_array). > > > > Yeah, that's something we have discussed as well. > > > > > > > > How do you prevent the scheduler from over committing to a single ring > > > > buffer in this scenario? > > > > > > > Each job has its own ring, the execbuf IOCTL throttles itself for each > > > job if there isn't space in the ring. This is exactly the same as > > > non-parallel submits. > > > > > > I think this is what you were asking? If not, maybe try explaining the > > > question a bit more. > > > > > > Matt > > > > > > > Christian. > > > > > > > > > Matt > > > > > > > > > > > Thanks, > > > > > > Christian. > > > > > > > > > > > > Am 19.05.21 um 01:58 schrieb Matthew Brost: > > > > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > > > > > > > v2: > > > > > > > (Daniel Vetter): > > > > > > > - Expand logical order explaination > > > > > > > - Add dummy header > > > > > > > - Only allow N BBs in execbuf IOCTL > > > > > > > - Configure parallel submission per slot not per gem context > > > > > > > > > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > > > > > Cc: Tony Ye <tony.ye@intel.com> > > > > > > > CC: Carl Zhang <carl.zhang@intel.com> > > > > > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > > > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > > > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > > > > > --- > > > > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > > > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > new file mode 100644 > > > > > > > index 000000000000..8c64b983ccad > > > > > > > --- /dev/null > > > > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > @@ -0,0 +1,144 @@ > > > > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > > > > > + > > > > > > > +/* > > > > > > > + * i915_context_engines_parallel_submit: > > > > > > > + * > > > > > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > > > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > > > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > > > > > + * the slots configuration). > > > > > > > + * > > > > > > > + * Their are two currently defined ways to control the placement of the > > > > > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > > > > > + * future as new hardware / use cases arise. Details of how to use this > > > > > > > + * interface below above the flags. > > > > > > > + * > > > > > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > > > > > + * placement configuration isn't supported on the platform / submission > > > > > > > + * interface. > > > > > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > > > > > + * inteface. > > > > > > > + */ > > > > > > > +struct i915_context_engines_parallel_submit { > > > > > > > + struct i915_user_extension base; > > > > > > > + > > > > > > > + __u16 engine_index; /* slot for parallel engine */ > > > > > > > + __u16 width; /* number of contexts per parallel engine */ > > > > > > > + __u16 num_siblings; /* number of siblings per context */ > > > > > > > + __u16 mbz16; > > > > > > > +/* > > > > > > > + * Default placement behvavior (currently unsupported): > > > > > > > + * > > > > > > > + * Rather than restricting parallel submission to a single class with a > > > > > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > > > > > + * enables parallel submission across multiple engine classes. In this case each > > > > > > > + * context's logical engine mask indicates where that context can placed. It is > > > > > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > > > > > + * if one context is running CS0 no other contexts can run on CS0). > > > > > > > + * > > > > > > > + * Example 1 pseudo code: > > > > > > > + * CSX[Y] = engine class X, logical instance Y > > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > > + * set_engines(INVALID) > > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > > > > > + * > > > > > > > + * Results in the following valid placements: > > > > > > > + * CS0[0], CS1[0] > > > > > > > + * CS0[0], CS1[1] > > > > > > > + * CS0[1], CS1[0] > > > > > > > + * CS0[1], CS1[1] > > > > > > > + * > > > > > > > + * This can also be though of as 2 virtual engines: > > > > > > > + * VE[0] = CS0[0], CS0[1] > > > > > > > + * VE[1] = CS1[0], CS1[1] > > > > > > > + * > > > > > > > + * Example 2 pseudo code: > > > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > > + * set_engines(INVALID) > > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > > > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > > > > > + * > > > > > > > + * Results in the following valid placements: > > > > > > > + * CS[0], CS[1] > > > > > > > + * CS[0], CS[2] > > > > > > > + * CS[1], CS[0] > > > > > > > + * CS[1], CS[2] > > > > > > > + * CS[2], CS[0] > > > > > > > + * CS[2], CS[1] > > > > > > > + * > > > > > > > + * > > > > > > > + * This can also be though of as 2 virtual engines: > > > > > > > + * VE[0] = CS[0], CS[1], CS[2] > > > > > > > + * VE[1] = CS[0], CS[1], CS[2] > > > > > > > + > > > > > > > + * This enables a use case where all engines are created equally, we don't care > > > > > > > + * where they are scheduled, we just want a certain number of resources, for > > > > > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > > > > > + * engine classes. > > > > > > > + */ > > > > > > > + > > > > > > > +/* > > > > > > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > > > > > > + * Each context must have the same number sibling and bonds are implictly create > > > > > > > + * of the siblings. > > > > > > > + * > > > > > > > + * All of the below examples are in logical space. > > > > > > > + * > > > > > > > + * Example 1 pseudo code: > > > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > > + * set_engines(INVALID) > > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > > > > > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > > > > + * > > > > > > > + * Results in the following valid placements: > > > > > > > + * CS[0], CS[1] > > > > > > > + * > > > > > > > + * Example 2 pseudo code: > > > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > > + * set_engines(INVALID) > > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > > > > + * > > > > > > > + * Results in the following valid placements: > > > > > > > + * CS[0], CS[1] > > > > > > > + * CS[2], CS[3] > > > > > > > + * > > > > > > > + * This can also be though of as 2 virtual engines: > > > > > > > + * VE[0] = CS[0], CS[2] > > > > > > > + * VE[1] = CS[1], CS[3] > > > > > > > + * > > > > > > > + * This enables a use case where all engines are not equal and certain placement > > > > > > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > > > > > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > > > > > > + * case (logically contiguous placement, within a single engine class) is > > > > > > > + * supported when using GuC submission. Execlist mode could support all possible > > > > > > > + * bonding configurations but currently doesn't support this extension. > > > > > > > + */ > > > > > > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > > > > > > +/* > > > > > > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > > > > > > + * points on all hardware contexts between each set of BBs. An example use case > > > > > > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > > > > > > + * BB must be submitted on each hardware context in the parallel gem context. > > > > > > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > > > > > > + */ > > > > > > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > > > > > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > > > > > > + __u64 flags; /* all undefined flags must be zero */ > > > > > > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > > > > > > + > > > > > > > + /* > > > > > > > + * width (i) * num_siblings (j) in length > > > > > > > + * index = j + i * num_siblings > > > > > > > + */ > > > > > > > + struct i915_engine_class_instance engines[0]; > > > > > > > +} __attribute__ ((packed)); > > > > > > > + > > > > > > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > > > > > > index 7faa46cde088..64c539486ee4 100644 > > > > > > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > > > > > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > > > > > > @@ -82,4 +82,55 @@ https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspec.oneapi.com%2Flevel-zero%2Flatest%2Fcore%2Fapi.html%23ze-command-queue-priorit&data=04%7C01%7Cchristian.koenig%40amd.com%7C49a7557f4e494090755608d91ae758a6%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637570403202969375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dn3PVdxYQpkpWIru5eAXHgbfuLDkppAA5daV5sHQF7s%3D&reserved=0 > > > > > > > New parallel submission uAPI > > > > > > > ============================ > > > > > > > -Details to come in a following patch. > > > > > > > +The existing bonding uAPI is completely broken with GuC submission because > > > > > > > +whether a submission is a single context submit or parallel submit isn't known > > > > > > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > > > > > > +contexts in parallel with the GuC the context must be explictly registered with > > > > > > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > > > > > > +These interfaces doesn't support dynamically changing between N contexts as the > > > > > > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > > > > > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > > > > > > + > > > > > > > +The new parallel submission uAPI consists of 3 parts: > > > > > > > + > > > > > > > +* Export engines logical mapping > > > > > > > +* A 'set_parallel' extension to configure contexts for parallel > > > > > > > + submission > > > > > > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > > > > + > > > > > > > +Export engines logical mapping > > > > > > > +------------------------------ > > > > > > > +Certain use cases require BBs to be placed on engine instances in logical order > > > > > > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > > > > > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > > > > > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > > > > > > +submission interface currently only supports submitting multiple contexts to > > > > > > > +engines in logical order which is a new requirement compared to execlists. > > > > > > > +Lastly, all current platforms have at most 2 instances and the logical order is > > > > > > > +the same a uABI order. This will change on platforms with more than 2 instances. > > > > > > > + > > > > > > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > > > > > > +logical instance has been returned and a new field, > > > > > > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > > > > > > + > > > > > > > +A 'set_parallel' extension to configure contexts for parallel submission > > > > > > > +------------------------------------------------------------------------ > > > > > > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > > > > > > +It is setup step that should be called before using any of the contexts. See > > > > > > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > > > > > > +similar existing examples. Once a slot is configured for parallel submission the > > > > > > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > > > > > > +support GuC submission. Execlist support can be added later if needed. > > > > > > > + > > > > > > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > > > > > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > > > > > > + > > > > > > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > > > > +------------------------------------------------------------------- > > > > > > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > > > > > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > > > > > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > > > > > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > > > > > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > > > > > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > > > > > > +the IOCTL. > > > _______________________________________________ > > > Intel-gfx mailing list > > > Intel-gfx@lists.freedesktop.org > > > https://lists.freedesktop.org/mailman/listinfo/intel-gfx >
On Thu, May 20, 2021 at 09:44:59PM +0200, Daniel Vetter wrote: > On Thu, May 20, 2021 at 08:10:59AM -0700, Matthew Brost wrote: > > On Thu, May 20, 2021 at 11:54:25AM +0200, Daniel Vetter wrote: > > > On Wed, May 19, 2021 at 7:19 PM Matthew Brost <matthew.brost@intel.com> wrote: > > > > > > > > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: > > > > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: > > > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > > > > > v2: > > > > > > (Daniel Vetter): > > > > > > - Expand logical order explaination > > > > > > - Add dummy header > > > > > > - Only allow N BBs in execbuf IOCTL > > > > > > - Configure parallel submission per slot not per gem context > > > > > > > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > > > > Cc: Tony Ye <tony.ye@intel.com> > > > > > > CC: Carl Zhang <carl.zhang@intel.com> > > > > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > > > > --- > > > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > new file mode 100644 > > > > > > index 000000000000..8c64b983ccad > > > > > > --- /dev/null > > > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > @@ -0,0 +1,144 @@ > > > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > > > > + > > > > > > +/* > > > > > > + * i915_context_engines_parallel_submit: > > > > > > + * > > > > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > > > > + * the slots configuration). > > > > > > + * > > > > > > + * Their are two currently defined ways to control the placement of the > > > > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > > > > + * future as new hardware / use cases arise. Details of how to use this > > > > > > + * interface below above the flags. > > > > > > + * > > > > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > > > > + * placement configuration isn't supported on the platform / submission > > > > > > + * interface. > > > > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > > > > + * inteface. > > > > > > + */ > > > > > > +struct i915_context_engines_parallel_submit { > > > > > > + struct i915_user_extension base; > > > > > > + > > > > > > + __u16 engine_index; /* slot for parallel engine */ > > > > > > + __u16 width; /* number of contexts per parallel engine */ > > > > > > + __u16 num_siblings; /* number of siblings per context */ > > > > > > + __u16 mbz16; > > > > > > > > > > Ok the big picture looks reasonable now, the flags still confuse me. > > > > > > > > > > > > > Yea, it is a bit confusing. > > > > > > > > > > +/* > > > > > > + * Default placement behvavior (currently unsupported): > > > > > > + * > > > > > > + * Rather than restricting parallel submission to a single class with a > > > > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > > > > + * enables parallel submission across multiple engine classes. In this case each > > > > > > + * context's logical engine mask indicates where that context can placed. It is > > > > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > > > > + * if one context is running CS0 no other contexts can run on CS0). > > > > > > + * > > > > > > + * Example 1 pseudo code: > > > > > > + * CSX[Y] = engine class X, logical instance Y > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > + * set_engines(INVALID) > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > > > > + * > > > > > > + * Results in the following valid placements: > > > > > > + * CS0[0], CS1[0] > > > > > > + * CS0[0], CS1[1] > > > > > > + * CS0[1], CS1[0] > > > > > > + * CS0[1], CS1[1] > > > > > > + * > > > > > > + * This can also be though of as 2 virtual engines: > > > > > > + * VE[0] = CS0[0], CS0[1] > > > > > > + * VE[1] = CS1[0], CS1[1] > > > > > > + * > > > > > > + * Example 2 pseudo code: > > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > + * set_engines(INVALID) > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > > > > + * > > > > > > + * Results in the following valid placements: > > > > > > + * CS[0], CS[1] > > > > > > + * CS[0], CS[2] > > > > > > + * CS[1], CS[0] > > > > > > + * CS[1], CS[2] > > > > > > + * CS[2], CS[0] > > > > > > + * CS[2], CS[1] > > > > > > + * > > > > > > + * > > > > > > + * This can also be though of as 2 virtual engines: > > > > > > + * VE[0] = CS[0], CS[1], CS[2] > > > > > > + * VE[1] = CS[0], CS[1], CS[2] > > > > > > + > > > > > > + * This enables a use case where all engines are created equally, we don't care > > > > > > + * where they are scheduled, we just want a certain number of resources, for > > > > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > > > > + * engine classes. > > > > > > + */ > > > > > > > > > > So I don't really get what this does compared to setting the flag below. > > > > > Is this just about running the batchbuffers the wrong way round, i.e. if > > > > > you have (simplest case) > > > > > > > > > > width=2, num_sibglings=1, engines=CS[0], CS[1] > > > > > > > > > > Then both > > > > > CS[0], CS[1] > > > > > and > > > > > CS[1], CS[0] > > > > > are possible options for running 2 batches? Iow, the backend is allowed to > > > > > run the batchbuffers the wrong way round, which gains us nothing, since we > > > > > assume the batches take equally long and engines interchangeable. There is > > > > > no scheduling scenario where this additional flexibility can help. > > > > > > > > > > Also we don't have flags to select the only available and then specify an > > > > > entire pipe dream about what the non-flag mode does, without an > > > > > implementation. What is this about? > > > > > > > > > > If it's just "because bonded allowed this" then I think we should just > > > > > unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but > > > > > welp. > > > > > > > > > > > > > High level the flags came out of internal discussions how this interface > > > > should look. The default placement behavior is theoretically possible > > > > with execlists but has no use cases. The GuC supports / current use > > > > cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. > > > > > > > > Argued about for months and this is where we landed. At the end of the > > > > day I think we needed to show that this interface supports more > > > > placement rules than what the GuC supports / current use cases to future > > > > proof this interface. > > > > > > > > For what is it worth it seems kinda backwards that we landed on the > > > > default behavior not being supported in our current stack / HW. > > > > > > Yeah I think that should be inverted, doesn't make sense. > > > > > > What I still don't get (and I've read Tvrtko's reply with the example) > > > is what exactly is the difference between implicit and not implicit > > > mode? Can you do a single example where the only difference is whether > > > this flag is set, and then explain with that what are the actual > > > differences in scheduling options that the backend is allowed to pick > > > for the set of N patches? > > > > > > I'm feeling a bit dense, but I'm really not seeing what's even going on here :-( > > > > > > > > > + > > > > > > +/* > > > > > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > > > > > + * Each context must have the same number sibling and bonds are implictly create > > > > > > + * of the siblings. > > > > > > + * > > > > > > + * All of the below examples are in logical space. > > > > > > + * > > > > > > + * Example 1 pseudo code: > > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > + * set_engines(INVALID) > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > > > > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > > > + * > > > > > > + * Results in the following valid placements: > > > > > > + * CS[0], CS[1] > > > > > > + * > > > > > > + * Example 2 pseudo code: > > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > + * set_engines(INVALID) > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > > > > > + * > > > > > > + * Results in the following valid placements: > > > > > > + * CS[0], CS[1] > > > > > > + * CS[2], CS[3] > > > > > > + * > > > > > > + * This can also be though of as 2 virtual engines: > > > > > > + * VE[0] = CS[0], CS[2] > > > > > > + * VE[1] = CS[1], CS[3] > > > > > > + * > > > > > > + * This enables a use case where all engines are not equal and certain placement > > > > > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > > > > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > > > > > + * case (logically contiguous placement, within a single engine class) is > > > > > > + * supported when using GuC submission. Execlist mode could support all possible > > > > > > + * bonding configurations but currently doesn't support this extension. > > > > > > + */ > > > > > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > > > > > > > > > Does this map to intel_context_set_nopreempt(), so we could implement it > > > > > also in execlist? > > > > > > > > > > > > > intel_context_set_nopreempt is preempt is part of how this would be > > > > implemented for execlists. For GuC submission there is a algorithm > > > > between the i915 and GuC that inserts same preemption points between > > > > each set of N batches. I believe the rules are if context 1 can > > > > preempted the rest of the contexts (2 - N) can safely be preempted. A go > > > > / join algorithm, implemented with semaphores, around the batches > > > > makes sure the i915 adheres to these rules. > > > > > > > > > Also is this just an artifact of the implementation, or is this somehow > > > > > required functionality for userspace, i.e. the workload fails if e.g. GuC > > > > > decides to preempt all LRC of a parallel virtual engine? Of course just > > > > > preempting one is a bit a bug (but execlist I think happily does that). > > > > > > > > > > > > > I think it is part of the implementation / HW limitations. Basically if > > > > N batches are running in parallel if one of the batches gets preempted > > > > it can hang all the other batches even if it gets restarted. Again the > > > > idea behind this is only preempt any of these contexts between each set > > > > of N batches. The GuC has no concept of batches only contexts hence the > > > > aforementioned algorithm to insert same preemption points between each > > > > set of batches. > > > > > > Well backend preempting one but not the other is a bug. I don't think > > > we need to talk about bugs :-) > > > > > > > I think it is more that backend can't preempt each context atomically as these > > are running on independent engines. Let say the backend tries to preempt 2 > > engines at the same time and only 1 responds, now we are broken with certain > > batches. With the algorithm implmented between the i915 and the Guc the GuC will > > try to preempt the first context if that works, all the other contexts can be > > preempted. > > Ah right, we might have a workload with long non-preemptible sections, but > I guess the MI_SEMAPHORE_WAIT can always preempt. So if GuC just tries on > one of these, it will result in tears. > > > > Now wrt a preempt mode in context creation, I think that makes sense. > > > I'm just wondering whether this is correct here in the parallel > > > submission (and why), or is it just an implementation artifact > > > (current backends are buggy if we'd allow them to preempt) or what's > > > going on. > > > > > > If this is just a case of "this is how current backends work, > > > userspace doesn't actually care whether we allow preempt or not" then > > > I think we should just document the behavior and that's it. Adding > > > uapi for an option for which there's not even an implementation isn't > > > great. > > > > > > > I don't think it is a case of 'this is how current backends work...', I think > > user space knows if it is ok for 1 of the batches to preempted and not the > > others without breaking the batches. > > Yeah this makes more sense as an explanation: Currently the backend (guc > or execlist) figure out whether something can be preempted by just trying. > Since we're dealing with N contexts that's a suboptimal approach, and we > need a flag that indicates whether it's worth it or not. > > So makes sense to me to have this. I guess userspace does actually need > both modes for now, depending upon what it's doing? > -Daniel I agree that is makes sense to have - an explict flag saying it isn't safe to preempt one of the N batches by itself or letting the driver do whatever it wants regarding preemption. The current use case is it can't preempt a single batch but it may change going forward. Lastly, there is overhead to non-preempt mid batch feature so being able to turn it off seems reasonable to me. Matt > > > > > Matt > > > > > But if we can actually choose, and userspace wants to, then I'm all > > > fine with this. And we can always move this to a more generic place if > > > other engines need preempt control later on too. > > > > > > Cheers, Daniel > > > > > > > > > > Matt > > > > > > > > > Cheers, Daniel > > > > > > > > > > > +/* > > > > > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > > > > > + * points on all hardware contexts between each set of BBs. An example use case > > > > > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > > > > > + * BB must be submitted on each hardware context in the parallel gem context. > > > > > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > > > > > + */ > > > > > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > > > > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > > > > > + __u64 flags; /* all undefined flags must be zero */ > > > > > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > > > > > + > > > > > > + /* > > > > > > + * width (i) * num_siblings (j) in length > > > > > > + * index = j + i * num_siblings > > > > > > + */ > > > > > > + struct i915_engine_class_instance engines[0]; > > > > > > +} __attribute__ ((packed)); > > > > > > + > > > > > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > > > > > index 7faa46cde088..64c539486ee4 100644 > > > > > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > > > > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > > > > > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > > > > > > > > > > > New parallel submission uAPI > > > > > > ============================ > > > > > > -Details to come in a following patch. > > > > > > +The existing bonding uAPI is completely broken with GuC submission because > > > > > > +whether a submission is a single context submit or parallel submit isn't known > > > > > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > > > > > +contexts in parallel with the GuC the context must be explictly registered with > > > > > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > > > > > +These interfaces doesn't support dynamically changing between N contexts as the > > > > > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > > > > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > > > > > + > > > > > > +The new parallel submission uAPI consists of 3 parts: > > > > > > + > > > > > > +* Export engines logical mapping > > > > > > +* A 'set_parallel' extension to configure contexts for parallel > > > > > > + submission > > > > > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > > > + > > > > > > +Export engines logical mapping > > > > > > +------------------------------ > > > > > > +Certain use cases require BBs to be placed on engine instances in logical order > > > > > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > > > > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > > > > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > > > > > +submission interface currently only supports submitting multiple contexts to > > > > > > +engines in logical order which is a new requirement compared to execlists. > > > > > > +Lastly, all current platforms have at most 2 instances and the logical order is > > > > > > +the same a uABI order. This will change on platforms with more than 2 instances. > > > > > > + > > > > > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > > > > > +logical instance has been returned and a new field, > > > > > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > > > > > + > > > > > > +A 'set_parallel' extension to configure contexts for parallel submission > > > > > > +------------------------------------------------------------------------ > > > > > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > > > > > +It is setup step that should be called before using any of the contexts. See > > > > > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > > > > > +similar existing examples. Once a slot is configured for parallel submission the > > > > > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > > > > > +support GuC submission. Execlist support can be added later if needed. > > > > > > + > > > > > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > > > > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > > > > > + > > > > > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > > > > > +------------------------------------------------------------------- > > > > > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > > > > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > > > > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > > > > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > > > > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > > > > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > > > > > +the IOCTL. > > > > > > -- > > > > > > 2.28.0 > > > > > > > > > > > > _______________________________________________ > > > > > > Intel-gfx mailing list > > > > > > Intel-gfx@lists.freedesktop.org > > > > > > https://lists.freedesktop.org/mailman/listinfo/intel-gfx > > > > > > > > > > -- > > > > > Daniel Vetter > > > > > Software Engineer, Intel Corporation > > > > > http://blog.ffwll.ch > > > > > > > > > > > > -- > > > Daniel Vetter > > > Software Engineer, Intel Corporation > > > http://blog.ffwll.ch > > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch
On Thu, May 20, 2021 at 09:41:20PM +0200, Daniel Vetter wrote: > On Thu, May 20, 2021 at 11:57:44AM +0100, Tvrtko Ursulin wrote: > > > > On 20/05/2021 10:54, Daniel Vetter wrote: > > > On Wed, May 19, 2021 at 7:19 PM Matthew Brost <matthew.brost@intel.com> wrote: > > > > > > > > On Wed, May 19, 2021 at 01:10:04PM +0200, Daniel Vetter wrote: > > > > > On Tue, May 18, 2021 at 04:58:30PM -0700, Matthew Brost wrote: > > > > > > Add entry fpr i915 new parallel submission uAPI plan. > > > > > > > > > > > > v2: > > > > > > (Daniel Vetter): > > > > > > - Expand logical order explaination > > > > > > - Add dummy header > > > > > > - Only allow N BBs in execbuf IOCTL > > > > > > - Configure parallel submission per slot not per gem context > > > > > > > > > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > > > > > Cc: Tony Ye <tony.ye@intel.com> > > > > > > CC: Carl Zhang <carl.zhang@intel.com> > > > > > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > > > > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > > > > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > > > > > --- > > > > > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > > > > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > > > > > 2 files changed, 196 insertions(+), 1 deletion(-) > > > > > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > > > > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > new file mode 100644 > > > > > > index 000000000000..8c64b983ccad > > > > > > --- /dev/null > > > > > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > > > @@ -0,0 +1,144 @@ > > > > > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > > > > > + > > > > > > +/* > > > > > > + * i915_context_engines_parallel_submit: > > > > > > + * > > > > > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > > > > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > > > > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > > > > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > > > > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > > > > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > > > > > + * the slots configuration). > > > > > > + * > > > > > > + * Their are two currently defined ways to control the placement of the > > > > > > + * hardware contexts on physical engines: default behavior (no flags) and > > > > > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > > > > > + * future as new hardware / use cases arise. Details of how to use this > > > > > > + * interface below above the flags. > > > > > > + * > > > > > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > > > > > + * placement configuration isn't supported on the platform / submission > > > > > > + * interface. > > > > > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > > > > > + * inteface. > > > > > > + */ > > > > > > +struct i915_context_engines_parallel_submit { > > > > > > + struct i915_user_extension base; > > > > > > + > > > > > > + __u16 engine_index; /* slot for parallel engine */ > > > > > > + __u16 width; /* number of contexts per parallel engine */ > > > > > > + __u16 num_siblings; /* number of siblings per context */ > > > > > > + __u16 mbz16; > > > > > > > > > > Ok the big picture looks reasonable now, the flags still confuse me. > > > > > > > > > > > > > Yea, it is a bit confusing. > > > > > > > > > > +/* > > > > > > + * Default placement behvavior (currently unsupported): > > > > > > + * > > > > > > + * Rather than restricting parallel submission to a single class with a > > > > > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > > > > > + * enables parallel submission across multiple engine classes. In this case each > > > > > > + * context's logical engine mask indicates where that context can placed. It is > > > > > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > > > > > + * if one context is running CS0 no other contexts can run on CS0). > > > > > > + * > > > > > > + * Example 1 pseudo code: > > > > > > + * CSX[Y] = engine class X, logical instance Y > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > + * set_engines(INVALID) > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > > > > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > > > > > + * > > > > > > + * Results in the following valid placements: > > > > > > + * CS0[0], CS1[0] > > > > > > + * CS0[0], CS1[1] > > > > > > + * CS0[1], CS1[0] > > > > > > + * CS0[1], CS1[1] > > > > > > + * > > > > > > + * This can also be though of as 2 virtual engines: > > > > > > + * VE[0] = CS0[0], CS0[1] > > > > > > + * VE[1] = CS1[0], CS1[1] > > > > > > + * > > > > > > + * Example 2 pseudo code: > > > > > > + * CS[X] = generic engine of same class, logical instance X > > > > > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > > > > > + * set_engines(INVALID) > > > > > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > > > > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > > > > > + * > > > > > > + * Results in the following valid placements: > > > > > > + * CS[0], CS[1] > > > > > > + * CS[0], CS[2] > > > > > > + * CS[1], CS[0] > > > > > > + * CS[1], CS[2] > > > > > > + * CS[2], CS[0] > > > > > > + * CS[2], CS[1] > > > > > > + * > > > > > > + * > > > > > > + * This can also be though of as 2 virtual engines: > > > > > > + * VE[0] = CS[0], CS[1], CS[2] > > > > > > + * VE[1] = CS[0], CS[1], CS[2] > > > > > > + > > > > > > + * This enables a use case where all engines are created equally, we don't care > > > > > > + * where they are scheduled, we just want a certain number of resources, for > > > > > > + * those resources to be scheduled in parallel, and possibly across multiple > > > > > > + * engine classes. > > > > > > + */ > > > > > > > > > > So I don't really get what this does compared to setting the flag below. > > > > > Is this just about running the batchbuffers the wrong way round, i.e. if > > > > > you have (simplest case) > > > > > > > > > > width=2, num_sibglings=1, engines=CS[0], CS[1] > > > > > > > > > > Then both > > > > > CS[0], CS[1] > > > > > and > > > > > CS[1], CS[0] > > > > > are possible options for running 2 batches? Iow, the backend is allowed to > > > > > run the batchbuffers the wrong way round, which gains us nothing, since we > > > > > assume the batches take equally long and engines interchangeable. There is > > > > > no scheduling scenario where this additional flexibility can help. > > > > > > > > > > Also we don't have flags to select the only available and then specify an > > > > > entire pipe dream about what the non-flag mode does, without an > > > > > implementation. What is this about? > > > > > > > > > > If it's just "because bonded allowed this" then I think we should just > > > > > unceremonously ditch this. Bummer for the nice kerenldoc you wrote, but > > > > > welp. > > > > > > > > > > > > > High level the flags came out of internal discussions how this interface > > > > should look. The default placement behavior is theoretically possible > > > > with execlists but has no use cases. The GuC supports / current use > > > > cases are a subset of what is possible with I915_PARALLEL_IMPLICT_BONDS. > > > > > > > > Argued about for months and this is where we landed. At the end of the > > > > day I think we needed to show that this interface supports more > > > > placement rules than what the GuC supports / current use cases to future > > > > proof this interface. > > > > > > > > For what is it worth it seems kinda backwards that we landed on the > > > > default behavior not being supported in our current stack / HW. > > > > > > Yeah I think that should be inverted, doesn't make sense. > > > > > > What I still don't get (and I've read Tvrtko's reply with the example) > > > is what exactly is the difference between implicit and not implicit > > > mode? Can you do a single example where the only difference is whether > > > this flag is set, and then explain with that what are the actual > > > differences in scheduling options that the backend is allowed to pick > > > for the set of N patches? > > > > > > I'm feeling a bit dense, but I'm really not seeing what's even going on here :-( > > > > 2-wide compute context: > > > > .engine_map([-1, -1]) > > .load_balance(0: [cs0, cs1, cs2, cs3]) // place virtual engine at slot 0 > > .load_balance(1: [cs0, cs1, cs2, cs3]) > > .set_parallel() > > > > This tells the scheduler any two of the four possible engines can be used. cs0 + cs3 is fine, cs3 + cs1 also, ... any. Only implicit rule is they have to be different and that works for all. > > > > 2-wide "implicit bonds mode" aka media fixed function limitation: > > > > .engine_map([-1, -1]) > > .load_balance(0: [cs0, cs2]) > > .load_balance(1: [cs1, cs3]) > > .set_parallel(flags = implicit_bond) > > > > Think of implicit flag creating a "link" between vertical columns in each virtual engine slot. So valid pairs end up cs0 + cs1 and cs2 + cs3 only. > > > > You can also think of the implicit flag as a shortcut to avoid specifying bonds via the existing extension. In which case context setup would be written along the lines of: > > > > .engine_map([-1, -1]) > > .load_balance(0: [cs0, cs2]) > > .load_balance(1: [cs1, cs3]) > > .bond(1: master = cs0, bond = [cs1]) > > .bond(1: master = cs2, bond = [cs3]) > > .set_parallel() > > > > So the implicit flag is just a shortcut to avoid typing the bonds. Not really needed as explained in my previous reply. > > Ah now I get both what this means, why it exists and where it's all come > from. With the backstory makes a bunch more sense now. Thanks for > explaining again. > > > This was at least the "old" set_parallel. I see this latest RFC changed > > things a bit which I don't really follow yet. > > > > It's not perfect but needs to add very little (just one context > > extension, on top of multi batch execbuf which is needed anyways), > > doesn't need to deprecate anything, didn't require rewrites of the UMD, > > and it all works today and in the future. > > > > I did not really like this new uapi for all the reasons I listed > > already, but as not many people were seeing the advantage of not > > churning on the uapi, if we are churning already I did suggests a > > different idea. I mean if we are churning we might as well go full in. > > So that proposal, which didn't get any traction, was along the lines of: > > > > .engine_map([-1]) > > .load_balance_wide(0: width=2, engines=[[cs0, cs2], [cs1, cs3]]) > > > > This would create an explicit wide virtual engine which should work for > > GuC fine I think. For execlists it may require a bit of extra glue but I > > don't think too much. > > > > Advantage is there is one engine in the map now and it is N-wide by > > definition. > > > > Since no one did bite on that idea back then, I didn't really pursue is > > to see if it works for all use cases. But I think it should even if it > > probably requires further thinking to be sure. > > > > If we apply it to compute use case.. > > > > .engine_map([-1]) > > .load_balance_wide(0: width=2, engines=[[cs0, cs1, cs2, cs3], [cs0, cs1, cs2, cs3]]) > > > > This means the only implicit wart in there is that cs0 + cs0 obviously > > shouldn't be picked. But that should be fine both for execlists and > > hopefully for the GuC. > > Yeah. Another option would be to simply allow any valid pair to be listed. > Gets maybe a bit too long for full combinatorials. Or we do an N-out-of-M > load balance, and you just specifiy the one vector for the engine set that > gets fully combined. > > Either way I think simple to add if/when compute comes around and asks for > it. I agree with this, with flags + an extendable engine list we really can make this interface describe any placement rules we want. I think what we have in placement at moment is good. Matt > -Daniel > -- > Daniel Vetter > Software Engineer, Intel Corporation > http://blog.ffwll.ch
On Fri, May 21, 2021 at 01:00:54PM +0100, Tvrtko Ursulin wrote: > > On 19/05/2021 00:58, Matthew Brost wrote: > > Add entry fpr i915 new parallel submission uAPI plan. > > > > v2: > > (Daniel Vetter): > > - Expand logical order explaination > > - Add dummy header > > - Only allow N BBs in execbuf IOCTL > > - Configure parallel submission per slot not per gem context > > > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > > Cc: Tony Ye <tony.ye@intel.com> > > CC: Carl Zhang <carl.zhang@intel.com> > > Cc: Daniel Vetter <daniel.vetter@intel.com> > > Cc: Jason Ekstrand <jason@jlekstrand.net> > > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > > --- > > Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ > > Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- > > 2 files changed, 196 insertions(+), 1 deletion(-) > > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > new file mode 100644 > > index 000000000000..8c64b983ccad > > --- /dev/null > > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > > @@ -0,0 +1,144 @@ > > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > > + > > +/* > > + * i915_context_engines_parallel_submit: > > + * > > + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. > > + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple > > + * hardware contexts are created internally in the i915 run these BBs. Once a > > + * slot is configured for N BBs only N BBs can be submitted in each execbuf > > + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf > > + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on > > + * the slots configuration). > > 1) > Expand the term slot here with "slot in the context engine map" least once > for clarity. > Sure. > 2) > About where execbuf will implicitly be finding batches - suggest to also > cover first/last flag here. I know you have it in the readme but I think it > is good if uapi header is as self-contained as possible. > Yep, good idea. > > + * > > + * Their are two currently defined ways to control the placement of the > > + * hardware contexts on physical engines: default behavior (no flags) and > > + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the > > + * future as new hardware / use cases arise. Details of how to use this > > + * interface below above the flags. > > + * > > + * Returns -EINVAL if hardware context placement configuration invalid or if the > > + * placement configuration isn't supported on the platform / submission > > + * interface. > > + * Returns -ENODEV if extension isn't supported on the platform / submission > > + * inteface. > > + */ > > +struct i915_context_engines_parallel_submit { > > + struct i915_user_extension base; > > + > > + __u16 engine_index; /* slot for parallel engine */ > > + __u16 width; /* number of contexts per parallel engine */ > > + __u16 num_siblings; /* number of siblings per context */ > > + __u16 mbz16; > > +/* > > + * Default placement behvavior (currently unsupported): > > + * > > + * Rather than restricting parallel submission to a single class with a > > + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that > > What do you mean with logically contiguous here? It sounds ambiguous versus > logical vs "normal" engine instance numbers. > This is a little backwards. I think when I wrote this originally the implicit placement comments were first. I can reword this. > > + * enables parallel submission across multiple engine classes. In this case each > > + * context's logical engine mask indicates where that context can placed. It is > > + * implied in this mode that all contexts have mutual exclusive placement (e.g. > > + * if one context is running CS0 no other contexts can run on CS0). > > I think talk about logical context and its mask is too implementation detail > at the uapi level. Instead I would suggest more userspace programmer centric > description. Ok, can you give me suggestion? Writing DOC isn't really my strength. > > > + * > > + * Example 1 pseudo code: > > + * CSX[Y] = engine class X, logical instance Y > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) > > + * > > + * Results in the following valid placements: > > + * CS0[0], CS1[0] > > + * CS0[0], CS1[1] > > + * CS0[1], CS1[0] > > + * CS0[1], CS1[1] > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS0[0], CS0[1] > > + * VE[1] = CS1[0], CS1[1] > > Ah okay so essentially similar to what I was proposing a year ago. But then > it is no longer "set_parallel" really. It is one slot in the engine map, > right, with the idea to super class intel_context in the implementation? > Yes, it is bascially a super class intel_context. In the implementation is parent-child with the parent having a linked list of child intel_contexts. > So really a wide virtual engine, as opposed to single one. In which case I > think it makes sense to stay close to the existing naming of the > load_balance extension for consistency. Load_balance_wide? > Load_balance_parallel? Multi? I like the way is named but I also don't want to argue about this as I don't really care. If someone else says this should be renamed, let's do it. > > I also have to say the notation "CS0[0]" - I who know this problem space am > finding it hard to penetrate what that actually means. (Also uppercase IMO > makes it hard to read, but maybe it is just me.) > Yea, now I think about it CS0[0] is bad because of using numbers twice. How about CSX[0] & CSY[1]? I used upper case because in the i915 all engine classes defines are upper case but agree it might be easier to read it lower case. > Looking a bit lower below, extension seems to be taking a 2d array of > class:instance pairs, right? If so then reading these docs in order, or even > just looking further down, I don't think that is explicitly called out > clearly enough. > > So I think a paragraph or two explaining clearly how the 2d array of engines > corresponds to the allowed engines for full virtual engine width. Or maybe > just a 2d diagram? > > 2-wide virtual engine: > .engines = [ > /* channel 0 allowed engines: */ [cs0, cs1], > /* channel 1 allowed engines: */ [cs0, cs1] > ] > > Not sure if that's better. > Yes, it is a 2-d array. Agree the explaination could be better. > Also to be noted, this only allows uniform number of allowed engines per > channel. I am not saying we need the non-uniform setup today but with bonds > there isn't this limitation. > Not exactly. You could do something like this. witdth = 2 siblings = 2 engines = CSX[0], CSX[1], CSY[0], INVALID This would allow a placement of: CSX[0], CSY[0] CSX[1], CSY[0] In this case the siblings is just a max value of each entry. Matt > > + * > > + * Example 2 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=3, > > + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * CS[0], CS[2] > > + * CS[1], CS[0] > > + * CS[1], CS[2] > > + * CS[2], CS[0] > > + * CS[2], CS[1] > > And now here you use different notation than above (CS0[0] vs CS[0]) which > is confusing. > Yea, a little bit. I do explain the notation in both examples though but will make this uniform. Now that I think about it CS0[0] is bad too because of using numbers twice. How about CSX[0] & CSY[1]? > > + * > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS[0], CS[1], CS[2] > > + * VE[1] = CS[0], CS[1], CS[2] > > + > > + * This enables a use case where all engines are created equally, we don't care > > + * where they are scheduled, we just want a certain number of resources, for > > + * those resources to be scheduled in parallel, and possibly across multiple > > + * engine classes. > > + */ > > + > > +/* > > + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. > > + * Each context must have the same number sibling and bonds are implictly create > > + * of the siblings. > > + * > > + * All of the below examples are in logical space. > > + * > > + * Example 1 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=1, > > + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * > > + * Example 2 pseudo code: > > + * CS[X] = generic engine of same class, logical instance X > > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > > + * set_engines(INVALID) > > + * set_parallel(engine_index=0, width=2, num_siblings=2, > > + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) > > + * > > + * Results in the following valid placements: > > + * CS[0], CS[1] > > + * CS[2], CS[3] > > + * > > + * This can also be though of as 2 virtual engines: > > + * VE[0] = CS[0], CS[2] > > + * VE[1] = CS[1], CS[3] > > + * > > + * This enables a use case where all engines are not equal and certain placement > > + * rules are required (i.e. split-frame requires all contexts to be placed in a > > + * logically contiguous order on the VCS engines on gen11+ platforms). This use > > + * case (logically contiguous placement, within a single engine class) is > > + * supported when using GuC submission. Execlist mode could support all possible > > + * bonding configurations but currently doesn't support this extension. > > + */ > > +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) > > +/* > > + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption > > + * points on all hardware contexts between each set of BBs. An example use case > > + * of this feature is split-frame on gen11+ hardware. When using this feature a > > + * BB must be submitted on each hardware context in the parallel gem context. > > + * The execbuf2 IOCTL enforces the user adheres to policy. > > + */ > Is the part about execbuf2 enforcing num_batches == context_width still > valid, I mean is the latest proposal supporting num_batches < context_width? > Does media needs this option? > With the new uAPI num_batches == context_width regardless of this flag. This is a stale comment that needs to be removed. > I would also have this flag as a separate patch because it strictly isn't > about parallel submission per-se but a specific hw limitation on top of it. > Yep, I do in the actual code but this in just the RFC header. Can break it out in the RFC header but IMO it is better to have in a single patch to review. Matt > Regards, > > Tvrtko > > > +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) > > +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) > > + __u64 flags; /* all undefined flags must be zero */ > > + __u64 mbz64[3]; /* reserved for future use; must be zero */ > > + > > + /* > > + * width (i) * num_siblings (j) in length > > + * index = j + i * num_siblings > > + */ > > + struct i915_engine_class_instance engines[0]; > > +} __attribute__ ((packed)); > > + > > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > > index 7faa46cde088..64c539486ee4 100644 > > --- a/Documentation/gpu/rfc/i915_scheduler.rst > > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > > @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit > > New parallel submission uAPI > > ============================ > > -Details to come in a following patch. > > +The existing bonding uAPI is completely broken with GuC submission because > > +whether a submission is a single context submit or parallel submit isn't known > > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > > +contexts in parallel with the GuC the context must be explictly registered with > > +N contexts and all N contexts must be submitted in a single command to the GuC. > > +These interfaces doesn't support dynamically changing between N contexts as the > > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > > +the legacy bonding uAPI is quite confusing and not intuitive at all. > > + > > +The new parallel submission uAPI consists of 3 parts: > > + > > +* Export engines logical mapping > > +* A 'set_parallel' extension to configure contexts for parallel > > + submission > > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > + > > +Export engines logical mapping > > +------------------------------ > > +Certain use cases require BBs to be placed on engine instances in logical order > > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > > +logical mapping with the existing query engine info IOCTL. Also the GuC > > +submission interface currently only supports submitting multiple contexts to > > +engines in logical order which is a new requirement compared to execlists. > > +Lastly, all current platforms have at most 2 instances and the logical order is > > +the same a uABI order. This will change on platforms with more than 2 instances. > > + > > +A single bit will be added to drm_i915_engine_info.flags indicating that the > > +logical instance has been returned and a new field, > > +drm_i915_engine_info.logical_instance, returns the logical instance. > > + > > +A 'set_parallel' extension to configure contexts for parallel submission > > +------------------------------------------------------------------------ > > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > > +It is setup step that should be called before using any of the contexts. See > > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > > +similar existing examples. Once a slot is configured for parallel submission the > > +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only > > +support GuC submission. Execlist support can be added later if needed. > > + > > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > > +i915_context_engines_parallel_submit to the uAPI to implement this extension. > > + > > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > > +------------------------------------------------------------------- > > +Contexts that have been configured with the 'set_parallel' extension are allowed > > +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N > > +objects in the drm_i915_gem_exec_object2 list or the first N if > > +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot > > +submitted and how it has been configured by 'set_parallel' or other extensions. > > +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of > > +the IOCTL. > >
On 21/05/2021 17:48, Matthew Brost wrote: > On Fri, May 21, 2021 at 01:00:54PM +0100, Tvrtko Ursulin wrote: [snip] >>> + * enables parallel submission across multiple engine classes. In this case each >>> + * context's logical engine mask indicates where that context can placed. It is >>> + * implied in this mode that all contexts have mutual exclusive placement (e.g. >>> + * if one context is running CS0 no other contexts can run on CS0). >> >> I think talk about logical context and its mask is too implementation detail >> at the uapi level. Instead I would suggest more userspace programmer centric >> description. > > Ok, can you give me suggestion? Writing DOC isn't really my strength. Yeah, not mine either. Maybe we need to hire a technical writer. :) I think in general I would just talk a bit how until now submission was along a single engine only and this is adding a wide submission model, expanding how it works with more details and only then talking about logical contexts if needed. It depends a bit whether our userspace clients still predominantly think in terms of engines, or is it contexts? I don't have an answer there. It probably isn't the most important thing and probably with a few tweaks of what you have it can be good enough. Key probably is simply coming up with as intuitive as possible diagrams, with consistent naming, showing how the wide engine is built and how it works. >>> + * >>> + * Example 1 pseudo code: >>> + * CSX[Y] = engine class X, logical instance Y >>> + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE >>> + * set_engines(INVALID) >>> + * set_parallel(engine_index=0, width=2, num_siblings=2, >>> + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) >>> + * >>> + * Results in the following valid placements: >>> + * CS0[0], CS1[0] >>> + * CS0[0], CS1[1] >>> + * CS0[1], CS1[0] >>> + * CS0[1], CS1[1] >>> + * >>> + * This can also be though of as 2 virtual engines: >>> + * VE[0] = CS0[0], CS0[1] >>> + * VE[1] = CS1[0], CS1[1] >> >> Ah okay so essentially similar to what I was proposing a year ago. But then >> it is no longer "set_parallel" really. It is one slot in the engine map, >> right, with the idea to super class intel_context in the implementation? >> > > Yes, it is bascially a super class intel_context. In the implementation is > parent-child with the parent having a linked list of child intel_contexts. > >> So really a wide virtual engine, as opposed to single one. In which case I >> think it makes sense to stay close to the existing naming of the >> load_balance extension for consistency. Load_balance_wide? >> Load_balance_parallel? Multi? > > I like the way is named but I also don't want to argue about this as I don't > really care. If someone else says this should be renamed, let's do it. I don't care too much apart from a general desire for more consistency and fewer terms in use. >> I also have to say the notation "CS0[0]" - I who know this problem space am >> finding it hard to penetrate what that actually means. (Also uppercase IMO >> makes it hard to read, but maybe it is just me.) >> > > Yea, now I think about it CS0[0] is bad because of using numbers twice. How > about CSX[0] & CSY[1]? I used upper case because in the i915 all engine classes > defines are upper case but agree it might be easier to read it lower case. What would X and Y represent? Or if I paste this part: * VE[0] = CS0[0], CS0[1] * VE[1] = CS1[0], CS1[1] Which index is engine instance and what is the other index? >> Looking a bit lower below, extension seems to be taking a 2d array of >> class:instance pairs, right? If so then reading these docs in order, or even >> just looking further down, I don't think that is explicitly called out >> clearly enough. >> >> So I think a paragraph or two explaining clearly how the 2d array of engines >> corresponds to the allowed engines for full virtual engine width. Or maybe >> just a 2d diagram? >> >> 2-wide virtual engine: >> .engines = [ >> /* channel 0 allowed engines: */ [cs0, cs1], >> /* channel 1 allowed engines: */ [cs0, cs1] >> ] >> >> Not sure if that's better. >> > > Yes, it is a 2-d array. Agree the explaination could be better. > >> Also to be noted, this only allows uniform number of allowed engines per >> channel. I am not saying we need the non-uniform setup today but with bonds >> there isn't this limitation. >> > > Not exactly. You could do something like this. > > witdth = 2 > siblings = 2 > engines = CSX[0], CSX[1], CSY[0], INVALID > > This would allow a placement of: > > CSX[0], CSY[0] > CSX[1], CSY[0] > > In this case the siblings is just a max value of each entry. Okay fair, did not think about that or saw it mentioned. Regards, Tvrtko
diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h new file mode 100644 index 000000000000..8c64b983ccad --- /dev/null +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h @@ -0,0 +1,144 @@ +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ + +/* + * i915_context_engines_parallel_submit: + * + * Setup a slot to allow multiple BBs to be submitted in a single execbuf IOCTL. + * Those BBs will then be scheduled to run on the GPU in parallel. Multiple + * hardware contexts are created internally in the i915 run these BBs. Once a + * slot is configured for N BBs only N BBs can be submitted in each execbuf + * IOCTL and this is implict behavior (e.g. the user doesn't tell the execbuf + * IOCTL there are N BBs, the execbuf IOCTL know how many BBs there are based on + * the slots configuration). + * + * Their are two currently defined ways to control the placement of the + * hardware contexts on physical engines: default behavior (no flags) and + * I915_PARALLEL_IMPLICT_BONDS (a flag). More flags may be added the in the + * future as new hardware / use cases arise. Details of how to use this + * interface below above the flags. + * + * Returns -EINVAL if hardware context placement configuration invalid or if the + * placement configuration isn't supported on the platform / submission + * interface. + * Returns -ENODEV if extension isn't supported on the platform / submission + * inteface. + */ +struct i915_context_engines_parallel_submit { + struct i915_user_extension base; + + __u16 engine_index; /* slot for parallel engine */ + __u16 width; /* number of contexts per parallel engine */ + __u16 num_siblings; /* number of siblings per context */ + __u16 mbz16; +/* + * Default placement behvavior (currently unsupported): + * + * Rather than restricting parallel submission to a single class with a + * logically contiguous placement (I915_PARALLEL_IMPLICT_BONDS), add a mode that + * enables parallel submission across multiple engine classes. In this case each + * context's logical engine mask indicates where that context can placed. It is + * implied in this mode that all contexts have mutual exclusive placement (e.g. + * if one context is running CS0 no other contexts can run on CS0). + * + * Example 1 pseudo code: + * CSX[Y] = engine class X, logical instance Y + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE + * set_engines(INVALID) + * set_parallel(engine_index=0, width=2, num_siblings=2, + * engines=CS0[0],CS0[1],CS1[0],CS1[1]) + * + * Results in the following valid placements: + * CS0[0], CS1[0] + * CS0[0], CS1[1] + * CS0[1], CS1[0] + * CS0[1], CS1[1] + * + * This can also be though of as 2 virtual engines: + * VE[0] = CS0[0], CS0[1] + * VE[1] = CS1[0], CS1[1] + * + * Example 2 pseudo code: + * CS[X] = generic engine of same class, logical instance X + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE + * set_engines(INVALID) + * set_parallel(engine_index=0, width=2, num_siblings=3, + * engines=CS[0],CS[1],CS[2],CS[0],CS[1],CS[2]) + * + * Results in the following valid placements: + * CS[0], CS[1] + * CS[0], CS[2] + * CS[1], CS[0] + * CS[1], CS[2] + * CS[2], CS[0] + * CS[2], CS[1] + * + * + * This can also be though of as 2 virtual engines: + * VE[0] = CS[0], CS[1], CS[2] + * VE[1] = CS[0], CS[1], CS[2] + + * This enables a use case where all engines are created equally, we don't care + * where they are scheduled, we just want a certain number of resources, for + * those resources to be scheduled in parallel, and possibly across multiple + * engine classes. + */ + +/* + * I915_PARALLEL_IMPLICT_BONDS - Create implict bonds between each context. + * Each context must have the same number sibling and bonds are implictly create + * of the siblings. + * + * All of the below examples are in logical space. + * + * Example 1 pseudo code: + * CS[X] = generic engine of same class, logical instance X + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE + * set_engines(INVALID) + * set_parallel(engine_index=0, width=2, num_siblings=1, + * engines=CS[0],CS[1], flags=I915_PARALLEL_IMPLICT_BONDS) + * + * Results in the following valid placements: + * CS[0], CS[1] + * + * Example 2 pseudo code: + * CS[X] = generic engine of same class, logical instance X + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE + * set_engines(INVALID) + * set_parallel(engine_index=0, width=2, num_siblings=2, + * engines=CS[0],CS[2],CS[1],CS[3], flags=I915_PARALLEL_IMPLICT_BONDS) + * + * Results in the following valid placements: + * CS[0], CS[1] + * CS[2], CS[3] + * + * This can also be though of as 2 virtual engines: + * VE[0] = CS[0], CS[2] + * VE[1] = CS[1], CS[3] + * + * This enables a use case where all engines are not equal and certain placement + * rules are required (i.e. split-frame requires all contexts to be placed in a + * logically contiguous order on the VCS engines on gen11+ platforms). This use + * case (logically contiguous placement, within a single engine class) is + * supported when using GuC submission. Execlist mode could support all possible + * bonding configurations but currently doesn't support this extension. + */ +#define I915_PARALLEL_IMPLICT_BONDS (1<<0) +/* + * Do not allow BBs to be preempted mid BB rather insert coordinated preemption + * points on all hardware contexts between each set of BBs. An example use case + * of this feature is split-frame on gen11+ hardware. When using this feature a + * BB must be submitted on each hardware context in the parallel gem context. + * The execbuf2 IOCTL enforces the user adheres to policy. + */ +#define I915_PARALLEL_NO_PREEMPT_MID_BATCH (1<<1) +#define __I915_PARALLEL_UNKNOWN_FLAGS (-(I915_PARALLEL_NO_PREEMPT_MID_BATCH << 1)) + __u64 flags; /* all undefined flags must be zero */ + __u64 mbz64[3]; /* reserved for future use; must be zero */ + + /* + * width (i) * num_siblings (j) in length + * index = j + i * num_siblings + */ + struct i915_engine_class_instance engines[0]; +} __attribute__ ((packed)); + diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst index 7faa46cde088..64c539486ee4 100644 --- a/Documentation/gpu/rfc/i915_scheduler.rst +++ b/Documentation/gpu/rfc/i915_scheduler.rst @@ -82,4 +82,55 @@ https://spec.oneapi.com/level-zero/latest/core/api.html#ze-command-queue-priorit New parallel submission uAPI ============================ -Details to come in a following patch. +The existing bonding uAPI is completely broken with GuC submission because +whether a submission is a single context submit or parallel submit isn't known +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple +contexts in parallel with the GuC the context must be explictly registered with +N contexts and all N contexts must be submitted in a single command to the GuC. +These interfaces doesn't support dynamically changing between N contexts as the +bonding uAPI does. Hence the need for a new parallel submission interface. Also +the legacy bonding uAPI is quite confusing and not intuitive at all. + +The new parallel submission uAPI consists of 3 parts: + +* Export engines logical mapping +* A 'set_parallel' extension to configure contexts for parallel + submission +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL + +Export engines logical mapping +------------------------------ +Certain use cases require BBs to be placed on engine instances in logical order +(e.g. split-frame on gen11+). The logical mapping of engine instances can change +based on fusing. Rather than making UMDs be aware of fusing, simply expose the +logical mapping with the existing query engine info IOCTL. Also the GuC +submission interface currently only supports submitting multiple contexts to +engines in logical order which is a new requirement compared to execlists. +Lastly, all current platforms have at most 2 instances and the logical order is +the same a uABI order. This will change on platforms with more than 2 instances. + +A single bit will be added to drm_i915_engine_info.flags indicating that the +logical instance has been returned and a new field, +drm_i915_engine_info.logical_instance, returns the logical instance. + +A 'set_parallel' extension to configure contexts for parallel submission +------------------------------------------------------------------------ +The 'set_parallel' extension configures a slot for parallel submission of N BBs. +It is setup step that should be called before using any of the contexts. See +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for +similar existing examples. Once a slot is configured for parallel submission the +execbuf2 IOCTL can be called submiting N BBs in a single IOCTL. Initially only +support GuC submission. Execlist support can be added later if needed. + +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and +i915_context_engines_parallel_submit to the uAPI to implement this extension. + +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL +------------------------------------------------------------------- +Contexts that have been configured with the 'set_parallel' extension are allowed +to submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N +objects in the drm_i915_gem_exec_object2 list or the first N if +I915_EXEC_BATCH_FIRST is set. The number of BBs is implict based on the slot +submitted and how it has been configured by 'set_parallel' or other extensions. +No uAPI changes in the execbuf IOCTL but worth mentioning the new behavior of +the IOCTL.
Add entry fpr i915 new parallel submission uAPI plan. v2: (Daniel Vetter): - Expand logical order explaination - Add dummy header - Only allow N BBs in execbuf IOCTL - Configure parallel submission per slot not per gem context Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Tony Ye <tony.ye@intel.com> CC: Carl Zhang <carl.zhang@intel.com> Cc: Daniel Vetter <daniel.vetter@intel.com> Cc: Jason Ekstrand <jason@jlekstrand.net> Signed-off-by: Matthew Brost <matthew.brost@intel.com> --- Documentation/gpu/rfc/i915_parallel_execbuf.h | 144 ++++++++++++++++++ Documentation/gpu/rfc/i915_scheduler.rst | 53 ++++++- 2 files changed, 196 insertions(+), 1 deletion(-) create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h