| Message ID | 20210611234044.853-3-matthew.brost@intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | GuC submission / DRM scheduler integration plan + new uAPI | expand |
Acked-by: Tony Ye <tony.ye@intel.com> Regards, Tony On 6/11/2021 4:40 PM, Matthew Brost wrote: > Add entry for i915 new parallel submission uAPI plan. > > v2: > (Daniel Vetter): > - Expand logical order explaination > - Add dummy header > - Only allow N BBs in execbuf IOCTL > - Configure parallel submission per slot not per gem context > v3: > (Marcin Ślusarz): > - Lot's of typos / bad english fixed > (Tvrtko Ursulin): > - Consistent pseudo code, clean up wording in descriptions > v4: > (Daniel Vetter) > - Drop flags > - Add kernel doc > - Reword a few things / fix typos > (Tvrtko) > - Reword a few things / fix typos > > Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> > Cc: Tony Ye <tony.ye@intel.com> > CC: Carl Zhang <carl.zhang@intel.com> > Cc: Daniel Vetter <daniel.vetter@intel.com> > Cc: Jason Ekstrand <jason@jlekstrand.net> > Signed-off-by: Matthew Brost <matthew.brost@intel.com> > Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch> > --- > Documentation/gpu/rfc/i915_parallel_execbuf.h | 117 ++++++++++++++++++ > Documentation/gpu/rfc/i915_scheduler.rst | 59 ++++++++- > 2 files changed, 175 insertions(+), 1 deletion(-) > create mode 100644 Documentation/gpu/rfc/i915_parallel_execbuf.h > > diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h > new file mode 100644 > index 000000000000..c22af3a359e4 > --- /dev/null > +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h > @@ -0,0 +1,117 @@ > +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ > + > +/** > + * struct drm_i915_context_engines_parallel_submit - Configure engine for > + * parallel submission. > + * > + * Setup a slot in the context engine map to allow multiple BBs to be submitted > + * in a single execbuf IOCTL. Those BBs will then be scheduled to run on the GPU > + * in parallel. Multiple hardware contexts are created internally in the i915 > + * run these BBs. Once a slot is configured for N BBs only N BBs can be > + * submitted in each execbuf IOCTL and this is implicit behavior e.g. The user > + * doesn't tell the execbuf IOCTL there are N BBs, the execbuf IOCTL knows how > + * many BBs there are based on the slot's configuration. The N BBs are the last > + * N buffer objects or first N if I915_EXEC_BATCH_FIRST is set. > + * > + * The default placement behavior is to create implicit bonds between each > + * context if each context maps to more than 1 physical engine (e.g. context is > + * a virtual engine). Also we only allow contexts of same engine class and these > + * contexts must be in logically contiguous order. Examples of the placement > + * behavior described below. Lastly, the default is to not allow BBs to > + * preempted mid BB rather insert coordinated preemption on all hardware > + * contexts between each set of BBs. Flags may be added in the future to change > + * bott of these default behaviors. > + * > + * Returns -EINVAL if hardware context placement configuration is invalid or if > + * the placement configuration isn't supported on the platform / submission > + * interface. > + * Returns -ENODEV if extension isn't supported on the platform / submission > + * inteface. > + * > + * .. code-block:: > + * > + * Example 1 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=1, > + * engines=CS[0],CS[1]) > + * > + * Results in the following valid placement: > + * CS[0], CS[1] > + * > + * Example 2 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS[0],CS[2],CS[1],CS[3]) > + * > + * Results in the following valid placements: > + * CS[0], CS[1] > + * CS[2], CS[3] > + * > + * This can also be thought of as 2 virtual engines described by 2-D array > + * in the engines the field with bonds placed between each index of the > + * virtual engines. e.g. CS[0] is bonded to CS[1], CS[2] is bonded to > + * CS[3]. > + * VE[0] = CS[0], CS[2] > + * VE[1] = CS[1], CS[3] > + * > + * Example 3 pseudo code: > + * CS[X] = generic engine of same class, logical instance X > + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE > + * set_engines(INVALID) > + * set_parallel(engine_index=0, width=2, num_siblings=2, > + * engines=CS[0],CS[1],CS[1],CS[3]) > + * > + * Results in the following valid and invalid placements: > + * CS[0], CS[1] > + * CS[1], CS[3] - Not logical contiguous, return -EINVAL > + */ > +struct drm_i915_context_engines_parallel_submit { > + /** > + * @base: base user extension. > + */ > + struct i915_user_extension base; > + > + /** > + * @engine_index: slot for parallel engine > + */ > + __u16 engine_index; > + > + /** > + * @width: number of contexts per parallel engine > + */ > + __u16 width; > + > + /** > + * @num_siblings: number of siblings per context > + */ > + __u16 num_siblings; > + > + /** > + * @mbz16: reserved for future use; must be zero > + */ > + __u16 mbz16; > + > + /** > + * @flags: all undefined flags must be zero, currently not defined flags > + */ > + __u64 flags; > + > + /** > + * @mbz64: reserved for future use; must be zero > + */ > + __u64 mbz64[3]; > + > + /** > + * @engines: 2-d array of engine instances to configure parallel engine > + * > + * length = width (i) * num_siblings (j) > + * index = j + i * num_siblings > + */ > + struct i915_engine_class_instance engines[0]; > + > +} __attribute__ ((packed)); > + > diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst > index 7acd386a6b49..63849b50e663 100644 > --- a/Documentation/gpu/rfc/i915_scheduler.rst > +++ b/Documentation/gpu/rfc/i915_scheduler.rst > @@ -88,4 +88,61 @@ Spec references: > > New parallel submission uAPI > ============================ > -Details to come in a following patch. > +The existing bonding uAPI is completely broken with GuC submission because > +whether a submission is a single context submit or parallel submit isn't known > +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple > +contexts in parallel with the GuC the context must be explicitly registered with > +N contexts and all N contexts must be submitted in a single command to the GuC. > +The GuC interfaces do not support dynamically changing between N contexts as the > +bonding uAPI does. Hence the need for a new parallel submission interface. Also > +the legacy bonding uAPI is quite confusing and not intuitive at all. Furthermore > +I915_SUBMIT_FENCE is by design a future fence, so not really something we should > +continue to support. > + > +The new parallel submission uAPI consists of 3 parts: > + > +* Export engines logical mapping > +* A 'set_parallel' extension to configure contexts for parallel > + submission > +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > + > +Export engines logical mapping > +------------------------------ > +Certain use cases require BBs to be placed on engine instances in logical order > +(e.g. split-frame on gen11+). The logical mapping of engine instances can change > +based on fusing. Rather than making UMDs be aware of fusing, simply expose the > +logical mapping with the existing query engine info IOCTL. Also the GuC > +submission interface currently only supports submitting multiple contexts to > +engines in logical order which is a new requirement compared to execlists. > +Lastly, all current platforms have at most 2 engine instances and the logical > +order is the same as uAPI order. This will change on platforms with more than 2 > +engine instances. > + > +A single bit will be added to drm_i915_engine_info.flags indicating that the > +logical instance has been returned and a new field, > +drm_i915_engine_info.logical_instance, returns the logical instance. > + > +A 'set_parallel' extension to configure contexts for parallel submission > +------------------------------------------------------------------------ > +The 'set_parallel' extension configures a slot for parallel submission of N BBs. > +It is a setup step that must be called before using any of the contexts. See > +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for > +similar existing examples. Once a slot is configured for parallel submission the > +execbuf2 IOCTL can be called submitting N BBs in a single IOCTL. Initially only > +supports GuC submission. Execlists supports can be added later if needed. > + > +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and > +drm_i915_context_engines_parallel_submit to the uAPI to implement this > +extension. > + > +.. kernel-doc:: Documentation/gpu/rfc/i915_parallel_execbuf.h > + :functions: drm_i915_context_engines_parallel_submit > + > +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL > +------------------------------------------------------------------- > +Contexts that have been configured with the 'set_parallel' extension can only > +submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N objects > +in in the drm_i915_gem_exec_object2 list or the first N if I915_EXEC_BATCH_FIRST > +is set. The number of BBs is implicit based on the slot submitted and how it has > +been configured by 'set_parallel' or other extensions. No uAPI changes are > +required to execbuf2 IOCTL. >
diff --git a/Documentation/gpu/rfc/i915_parallel_execbuf.h b/Documentation/gpu/rfc/i915_parallel_execbuf.h new file mode 100644 index 000000000000..c22af3a359e4 --- /dev/null +++ b/Documentation/gpu/rfc/i915_parallel_execbuf.h @@ -0,0 +1,117 @@ +#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */ + +/** + * struct drm_i915_context_engines_parallel_submit - Configure engine for + * parallel submission. + * + * Setup a slot in the context engine map to allow multiple BBs to be submitted + * in a single execbuf IOCTL. Those BBs will then be scheduled to run on the GPU + * in parallel. Multiple hardware contexts are created internally in the i915 + * run these BBs. Once a slot is configured for N BBs only N BBs can be + * submitted in each execbuf IOCTL and this is implicit behavior e.g. The user + * doesn't tell the execbuf IOCTL there are N BBs, the execbuf IOCTL knows how + * many BBs there are based on the slot's configuration. The N BBs are the last + * N buffer objects or first N if I915_EXEC_BATCH_FIRST is set. + * + * The default placement behavior is to create implicit bonds between each + * context if each context maps to more than 1 physical engine (e.g. context is + * a virtual engine). Also we only allow contexts of same engine class and these + * contexts must be in logically contiguous order. Examples of the placement + * behavior described below. Lastly, the default is to not allow BBs to + * preempted mid BB rather insert coordinated preemption on all hardware + * contexts between each set of BBs. Flags may be added in the future to change + * bott of these default behaviors. + * + * Returns -EINVAL if hardware context placement configuration is invalid or if + * the placement configuration isn't supported on the platform / submission + * interface. + * Returns -ENODEV if extension isn't supported on the platform / submission + * inteface. + * + * .. code-block:: + * + * Example 1 pseudo code: + * CS[X] = generic engine of same class, logical instance X + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE + * set_engines(INVALID) + * set_parallel(engine_index=0, width=2, num_siblings=1, + * engines=CS[0],CS[1]) + * + * Results in the following valid placement: + * CS[0], CS[1] + * + * Example 2 pseudo code: + * CS[X] = generic engine of same class, logical instance X + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE + * set_engines(INVALID) + * set_parallel(engine_index=0, width=2, num_siblings=2, + * engines=CS[0],CS[2],CS[1],CS[3]) + * + * Results in the following valid placements: + * CS[0], CS[1] + * CS[2], CS[3] + * + * This can also be thought of as 2 virtual engines described by 2-D array + * in the engines the field with bonds placed between each index of the + * virtual engines. e.g. CS[0] is bonded to CS[1], CS[2] is bonded to + * CS[3]. + * VE[0] = CS[0], CS[2] + * VE[1] = CS[1], CS[3] + * + * Example 3 pseudo code: + * CS[X] = generic engine of same class, logical instance X + * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE + * set_engines(INVALID) + * set_parallel(engine_index=0, width=2, num_siblings=2, + * engines=CS[0],CS[1],CS[1],CS[3]) + * + * Results in the following valid and invalid placements: + * CS[0], CS[1] + * CS[1], CS[3] - Not logical contiguous, return -EINVAL + */ +struct drm_i915_context_engines_parallel_submit { + /** + * @base: base user extension. + */ + struct i915_user_extension base; + + /** + * @engine_index: slot for parallel engine + */ + __u16 engine_index; + + /** + * @width: number of contexts per parallel engine + */ + __u16 width; + + /** + * @num_siblings: number of siblings per context + */ + __u16 num_siblings; + + /** + * @mbz16: reserved for future use; must be zero + */ + __u16 mbz16; + + /** + * @flags: all undefined flags must be zero, currently not defined flags + */ + __u64 flags; + + /** + * @mbz64: reserved for future use; must be zero + */ + __u64 mbz64[3]; + + /** + * @engines: 2-d array of engine instances to configure parallel engine + * + * length = width (i) * num_siblings (j) + * index = j + i * num_siblings + */ + struct i915_engine_class_instance engines[0]; + +} __attribute__ ((packed)); + diff --git a/Documentation/gpu/rfc/i915_scheduler.rst b/Documentation/gpu/rfc/i915_scheduler.rst index 7acd386a6b49..63849b50e663 100644 --- a/Documentation/gpu/rfc/i915_scheduler.rst +++ b/Documentation/gpu/rfc/i915_scheduler.rst @@ -88,4 +88,61 @@ Spec references: New parallel submission uAPI ============================ -Details to come in a following patch. +The existing bonding uAPI is completely broken with GuC submission because +whether a submission is a single context submit or parallel submit isn't known +until execbuf time activated via the I915_SUBMIT_FENCE. To submit multiple +contexts in parallel with the GuC the context must be explicitly registered with +N contexts and all N contexts must be submitted in a single command to the GuC. +The GuC interfaces do not support dynamically changing between N contexts as the +bonding uAPI does. Hence the need for a new parallel submission interface. Also +the legacy bonding uAPI is quite confusing and not intuitive at all. Furthermore +I915_SUBMIT_FENCE is by design a future fence, so not really something we should +continue to support. + +The new parallel submission uAPI consists of 3 parts: + +* Export engines logical mapping +* A 'set_parallel' extension to configure contexts for parallel + submission +* Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL + +Export engines logical mapping +------------------------------ +Certain use cases require BBs to be placed on engine instances in logical order +(e.g. split-frame on gen11+). The logical mapping of engine instances can change +based on fusing. Rather than making UMDs be aware of fusing, simply expose the +logical mapping with the existing query engine info IOCTL. Also the GuC +submission interface currently only supports submitting multiple contexts to +engines in logical order which is a new requirement compared to execlists. +Lastly, all current platforms have at most 2 engine instances and the logical +order is the same as uAPI order. This will change on platforms with more than 2 +engine instances. + +A single bit will be added to drm_i915_engine_info.flags indicating that the +logical instance has been returned and a new field, +drm_i915_engine_info.logical_instance, returns the logical instance. + +A 'set_parallel' extension to configure contexts for parallel submission +------------------------------------------------------------------------ +The 'set_parallel' extension configures a slot for parallel submission of N BBs. +It is a setup step that must be called before using any of the contexts. See +I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE or I915_CONTEXT_ENGINES_EXT_BOND for +similar existing examples. Once a slot is configured for parallel submission the +execbuf2 IOCTL can be called submitting N BBs in a single IOCTL. Initially only +supports GuC submission. Execlists supports can be added later if needed. + +Add I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT and +drm_i915_context_engines_parallel_submit to the uAPI to implement this +extension. + +.. kernel-doc:: Documentation/gpu/rfc/i915_parallel_execbuf.h + :functions: drm_i915_context_engines_parallel_submit + +Extend execbuf2 IOCTL to support submitting N BBs in a single IOCTL +------------------------------------------------------------------- +Contexts that have been configured with the 'set_parallel' extension can only +submit N BBs in a single execbuf2 IOCTL. The BBs are either the last N objects +in in the drm_i915_gem_exec_object2 list or the first N if I915_EXEC_BATCH_FIRST +is set. The number of BBs is implicit based on the slot submitted and how it has +been configured by 'set_parallel' or other extensions. No uAPI changes are +required to execbuf2 IOCTL.