| Message ID | 20190204132214.9459-16-chris@chris-wilson.co.uk (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [01/22] drm/i915/execlists: Suppress mere WAIT preemption | expand |
On 04/02/2019 13:22, Chris Wilson wrote > As kmem_caches share the same properties (size, allocation/free behaviour) > for all potential devices, we can use global caches. While this > potential has worse fragmentation behaviour (one can argue that > different devices would have different activity lifetimes, but you can > also argue that activity is temporal across the system) it is the > default behaviour of the system at large to amalgamate matching caches. > > The benefit for us is much reduced pointer dancing along the frequent > allocation paths. > > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> > --- > drivers/gpu/drm/i915/Makefile | 1 + > drivers/gpu/drm/i915/i915_active.c | 9 ++- > drivers/gpu/drm/i915/i915_active.h | 1 + > drivers/gpu/drm/i915/i915_drv.h | 3 - > drivers/gpu/drm/i915/i915_gem.c | 32 +-------- > drivers/gpu/drm/i915/i915_globals.c | 49 ++++++++++++++ > drivers/gpu/drm/i915/i915_globals.h | 14 ++++ > drivers/gpu/drm/i915/i915_pci.c | 8 ++- > drivers/gpu/drm/i915/i915_request.c | 53 ++++++++++++--- > drivers/gpu/drm/i915/i915_request.h | 10 +++ > drivers/gpu/drm/i915/i915_scheduler.c | 66 +++++++++++++++---- > drivers/gpu/drm/i915/i915_scheduler.h | 34 ++++++++-- > drivers/gpu/drm/i915/intel_guc_submission.c | 3 +- > drivers/gpu/drm/i915/intel_lrc.c | 6 +- > drivers/gpu/drm/i915/intel_ringbuffer.h | 17 ----- > drivers/gpu/drm/i915/selftests/intel_lrc.c | 2 +- > drivers/gpu/drm/i915/selftests/mock_engine.c | 48 +++++++------- > .../gpu/drm/i915/selftests/mock_gem_device.c | 26 -------- > drivers/gpu/drm/i915/selftests/mock_request.c | 12 ++-- > drivers/gpu/drm/i915/selftests/mock_request.h | 7 -- > 20 files changed, 248 insertions(+), 153 deletions(-) > create mode 100644 drivers/gpu/drm/i915/i915_globals.c > create mode 100644 drivers/gpu/drm/i915/i915_globals.h > > diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile > index 1787e1299b1b..a1d834068765 100644 > --- a/drivers/gpu/drm/i915/Makefile > +++ b/drivers/gpu/drm/i915/Makefile > @@ -77,6 +77,7 @@ i915-y += \ > i915_gem_tiling.o \ > i915_gem_userptr.o \ > i915_gemfs.o \ > + i915_globals.o \ > i915_query.o \ > i915_request.o \ > i915_scheduler.o \ > diff --git a/drivers/gpu/drm/i915/i915_active.c b/drivers/gpu/drm/i915/i915_active.c > index 64661c41532b..d23092d8c89f 100644 > --- a/drivers/gpu/drm/i915/i915_active.c > +++ b/drivers/gpu/drm/i915/i915_active.c > @@ -251,7 +251,7 @@ void i915_active_fini(struct i915_active *ref) > #include "selftests/i915_active.c" > #endif > > -int __init i915_global_active_init(void) > +int i915_global_active_init(void) These can't remain __init, since they are only called from the global __init one? > { > global.slab_cache = KMEM_CACHE(active_node, SLAB_HWCACHE_ALIGN); > if (!global.slab_cache) > @@ -260,7 +260,12 @@ int __init i915_global_active_init(void) > return 0; > } > > -void __exit i915_global_active_exit(void) > +void i915_global_active_shrink(void) > +{ > + kmem_cache_shrink(global.slab_cache); > +} > + > +void i915_global_active_exit(void) > { > kmem_cache_destroy(global.slab_cache); > } > diff --git a/drivers/gpu/drm/i915/i915_active.h b/drivers/gpu/drm/i915/i915_active.h > index 179b47aeec33..6c56d10b1f59 100644 > --- a/drivers/gpu/drm/i915/i915_active.h > +++ b/drivers/gpu/drm/i915/i915_active.h > @@ -71,6 +71,7 @@ static inline void i915_active_fini(struct i915_active *ref) { } > #endif > > int i915_global_active_init(void); > +void i915_global_active_shrink(void); > void i915_global_active_exit(void); > > #endif /* _I915_ACTIVE_H_ */ > diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h > index 3e4538ce5276..e48e3c228d9c 100644 > --- a/drivers/gpu/drm/i915/i915_drv.h > +++ b/drivers/gpu/drm/i915/i915_drv.h > @@ -1459,9 +1459,6 @@ struct drm_i915_private { > struct kmem_cache *objects; > struct kmem_cache *vmas; > struct kmem_cache *luts; > - struct kmem_cache *requests; > - struct kmem_cache *dependencies; > - struct kmem_cache *priorities; > > const struct intel_device_info __info; /* Use INTEL_INFO() to access. */ > struct intel_runtime_info __runtime; /* Use RUNTIME_INFO() to access. */ > diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c > index 2c6161c89cc7..d82e4f990586 100644 > --- a/drivers/gpu/drm/i915/i915_gem.c > +++ b/drivers/gpu/drm/i915/i915_gem.c > @@ -42,6 +42,7 @@ > #include "i915_drv.h" > #include "i915_gem_clflush.h" > #include "i915_gemfs.h" > +#include "i915_globals.h" > #include "i915_reset.h" > #include "i915_trace.h" > #include "i915_vgpu.h" > @@ -2916,12 +2917,11 @@ static void shrink_caches(struct drm_i915_private *i915) > * filled slabs to prioritise allocating from the mostly full slabs, > * with the aim of reducing fragmentation. > */ > - kmem_cache_shrink(i915->priorities); > - kmem_cache_shrink(i915->dependencies); > - kmem_cache_shrink(i915->requests); > kmem_cache_shrink(i915->luts); > kmem_cache_shrink(i915->vmas); > kmem_cache_shrink(i915->objects); > + > + i915_globals_shrink(); This is the main bit which worries me. Global caches are what we want I think, exactly for what you wrote in the commit message. But would one device going idle have the potential to inject some latency into another potentially very busy client? Perhaps we could have some sort of aggregated idle signal and defer shrinking cached to that point. Like a bitmask of global clients reporting their idle/active status to global core, and then shrink happens only if all are idle. > } > > struct sleep_rcu_work { > @@ -5264,23 +5264,6 @@ int i915_gem_init_early(struct drm_i915_private *dev_priv) > if (!dev_priv->luts) > goto err_vmas; > > - dev_priv->requests = KMEM_CACHE(i915_request, > - SLAB_HWCACHE_ALIGN | > - SLAB_RECLAIM_ACCOUNT | > - SLAB_TYPESAFE_BY_RCU); > - if (!dev_priv->requests) > - goto err_luts; > - > - dev_priv->dependencies = KMEM_CACHE(i915_dependency, > - SLAB_HWCACHE_ALIGN | > - SLAB_RECLAIM_ACCOUNT); > - if (!dev_priv->dependencies) > - goto err_requests; > - > - dev_priv->priorities = KMEM_CACHE(i915_priolist, SLAB_HWCACHE_ALIGN); > - if (!dev_priv->priorities) > - goto err_dependencies; > - > INIT_LIST_HEAD(&dev_priv->gt.active_rings); > INIT_LIST_HEAD(&dev_priv->gt.closed_vma); > > @@ -5305,12 +5288,6 @@ int i915_gem_init_early(struct drm_i915_private *dev_priv) > > return 0; > > -err_dependencies: > - kmem_cache_destroy(dev_priv->dependencies); > -err_requests: > - kmem_cache_destroy(dev_priv->requests); > -err_luts: > - kmem_cache_destroy(dev_priv->luts); > err_vmas: > kmem_cache_destroy(dev_priv->vmas); > err_objects: > @@ -5328,9 +5305,6 @@ void i915_gem_cleanup_early(struct drm_i915_private *dev_priv) > > cleanup_srcu_struct(&dev_priv->gpu_error.srcu); > > - kmem_cache_destroy(dev_priv->priorities); > - kmem_cache_destroy(dev_priv->dependencies); > - kmem_cache_destroy(dev_priv->requests); > kmem_cache_destroy(dev_priv->luts); > kmem_cache_destroy(dev_priv->vmas); > kmem_cache_destroy(dev_priv->objects); > diff --git a/drivers/gpu/drm/i915/i915_globals.c b/drivers/gpu/drm/i915/i915_globals.c > new file mode 100644 > index 000000000000..2ecf9897fd16 > --- /dev/null > +++ b/drivers/gpu/drm/i915/i915_globals.c > @@ -0,0 +1,49 @@ > +/* > + * SPDX-License-Identifier: MIT > + * > + * Copyright © 2019 Intel Corporation > + */ > + > +#include "i915_active.h" > +#include "i915_globals.h" > +#include "i915_request.h" > +#include "i915_scheduler.h" > + > +int __init i915_globals_init(void) > +{ > + int err; > + > + err = i915_global_active_init(); > + if (err) > + return err; > + > + err = i915_global_request_init(); > + if (err) > + goto err_active; > + > + err = i915_global_scheduler_init(); > + if (err) > + goto err_request; > + > + return 0; > + > +err_request: > + i915_global_request_exit(); > +err_active: > + i915_global_active_exit(); > + return err; > +} > + > +void i915_globals_shrink(void) > +{ > + i915_global_active_shrink(); > + i915_global_request_shrink(); > + i915_global_scheduler_shrink(); > +} > + > +void __exit i915_globals_exit(void) > +{ > + i915_global_scheduler_exit(); > + i915_global_request_exit(); > + i915_global_active_exit(); > +} > diff --git a/drivers/gpu/drm/i915/i915_globals.h b/drivers/gpu/drm/i915/i915_globals.h > new file mode 100644 > index 000000000000..903f52c0a1d2 > --- /dev/null > +++ b/drivers/gpu/drm/i915/i915_globals.h > @@ -0,0 +1,14 @@ > +/* > + * SPDX-License-Identifier: MIT > + * > + * Copyright © 2019 Intel Corporation > + */ > + > +#ifndef _I915_GLOBALS_H_ > +#define _I915_GLOBALS_H_ > + > +int i915_globals_init(void); > +void i915_globals_shrink(void); > +void i915_globals_exit(void); > + > +#endif /* _I915_GLOBALS_H_ */ > diff --git a/drivers/gpu/drm/i915/i915_pci.c b/drivers/gpu/drm/i915/i915_pci.c > index 852b6b4e8ed8..0d684eb530c3 100644 > --- a/drivers/gpu/drm/i915/i915_pci.c > +++ b/drivers/gpu/drm/i915/i915_pci.c > @@ -28,8 +28,8 @@ > > #include <drm/drm_drv.h> > > -#include "i915_active.h" > #include "i915_drv.h" > +#include "i915_globals.h" > #include "i915_selftest.h" > > #define PLATFORM(x) .platform = (x), .platform_mask = BIT(x) > @@ -801,7 +801,9 @@ static int __init i915_init(void) > bool use_kms = true; > int err; > > - i915_global_active_init(); > + err = i915_globals_init(); > + if (err) > + return err; > > err = i915_mock_selftests(); > if (err) > @@ -834,7 +836,7 @@ static void __exit i915_exit(void) > return; > > pci_unregister_driver(&i915_pci_driver); > - i915_global_active_exit(); > + i915_globals_exit(); > } > > module_init(i915_init); > diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c > index 04c65e6d83b9..3bb4840ba761 100644 > --- a/drivers/gpu/drm/i915/i915_request.c > +++ b/drivers/gpu/drm/i915/i915_request.c > @@ -31,6 +31,11 @@ > #include "i915_drv.h" > #include "i915_reset.h" > > +static struct i915_global_request { > + struct kmem_cache *slab_requests; > + struct kmem_cache *slab_dependencies; > +} global; > + > static const char *i915_fence_get_driver_name(struct dma_fence *fence) > { > return "i915"; > @@ -83,7 +88,7 @@ static void i915_fence_release(struct dma_fence *fence) > */ > i915_sw_fence_fini(&rq->submit); > > - kmem_cache_free(rq->i915->requests, rq); > + kmem_cache_free(global.slab_requests, rq); > } > > const struct dma_fence_ops i915_fence_ops = { > @@ -301,7 +306,7 @@ static void i915_request_retire(struct i915_request *request) > > unreserve_gt(request->i915); > > - i915_sched_node_fini(request->i915, &request->sched); > + i915_sched_node_fini(&request->sched); > i915_request_put(request); > } > > @@ -535,7 +540,7 @@ i915_request_alloc_slow(struct intel_context *ce) > ring_retire_requests(ring); > > out: > - return kmem_cache_alloc(ce->gem_context->i915->requests, GFP_KERNEL); > + return kmem_cache_alloc(global.slab_requests, GFP_KERNEL); > } > > /** > @@ -617,7 +622,7 @@ i915_request_alloc(struct intel_engine_cs *engine, struct i915_gem_context *ctx) > * > * Do not use kmem_cache_zalloc() here! > */ > - rq = kmem_cache_alloc(i915->requests, > + rq = kmem_cache_alloc(global.slab_requests, > GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN); > if (unlikely(!rq)) { > rq = i915_request_alloc_slow(ce); > @@ -701,7 +706,7 @@ i915_request_alloc(struct intel_engine_cs *engine, struct i915_gem_context *ctx) > GEM_BUG_ON(!list_empty(&rq->sched.signalers_list)); > GEM_BUG_ON(!list_empty(&rq->sched.waiters_list)); > > - kmem_cache_free(i915->requests, rq); > + kmem_cache_free(global.slab_requests, rq); > err_unreserve: > unreserve_gt(i915); > intel_context_unpin(ce); > @@ -720,9 +725,7 @@ i915_request_await_request(struct i915_request *to, struct i915_request *from) > return 0; > > if (to->engine->schedule) { > - ret = i915_sched_node_add_dependency(to->i915, > - &to->sched, > - &from->sched); > + ret = i915_sched_node_add_dependency(&to->sched, &from->sched); > if (ret < 0) > return ret; > } > @@ -1195,3 +1198,37 @@ void i915_retire_requests(struct drm_i915_private *i915) > #include "selftests/mock_request.c" > #include "selftests/i915_request.c" > #endif > + > +int i915_global_request_init(void) > +{ > + global.slab_requests = KMEM_CACHE(i915_request, > + SLAB_HWCACHE_ALIGN | > + SLAB_RECLAIM_ACCOUNT | > + SLAB_TYPESAFE_BY_RCU); > + if (!global.slab_requests) > + return -ENOMEM; > + > + global.slab_dependencies = KMEM_CACHE(i915_dependency, > + SLAB_HWCACHE_ALIGN | > + SLAB_RECLAIM_ACCOUNT); > + if (!global.slab_dependencies) > + goto err_requests; > + > + return 0; > + > +err_requests: > + kmem_cache_destroy(global.slab_requests); > + return -ENOMEM; > +} > + > +void i915_global_request_shrink(void) > +{ > + kmem_cache_shrink(global.slab_dependencies); > + kmem_cache_shrink(global.slab_requests); > +} > + > +void i915_global_request_exit(void) > +{ > + kmem_cache_destroy(global.slab_dependencies); > + kmem_cache_destroy(global.slab_requests); > +} > diff --git a/drivers/gpu/drm/i915/i915_request.h b/drivers/gpu/drm/i915/i915_request.h > index 3cffb96203b9..054bd300984b 100644 > --- a/drivers/gpu/drm/i915/i915_request.h > +++ b/drivers/gpu/drm/i915/i915_request.h > @@ -29,6 +29,7 @@ > > #include "i915_gem.h" > #include "i915_scheduler.h" > +#include "i915_selftest.h" > #include "i915_sw_fence.h" > > #include <uapi/drm/i915_drm.h> > @@ -204,6 +205,11 @@ struct i915_request { > struct drm_i915_file_private *file_priv; > /** file_priv list entry for this request */ > struct list_head client_link; > + > + I915_SELFTEST_DECLARE(struct { > + struct list_head link; > + unsigned long delay; > + } mock;) > }; > > #define I915_FENCE_GFP (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN) > @@ -786,4 +792,8 @@ i915_gem_active_retire(struct i915_gem_active *active, > #define for_each_active(mask, idx) \ > for (; mask ? idx = ffs(mask) - 1, 1 : 0; mask &= ~BIT(idx)) > > +int i915_global_request_init(void); > +void i915_global_request_shrink(void); > +void i915_global_request_exit(void); > + > #endif /* I915_REQUEST_H */ > diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c > index d01683167c77..7c1d9ef98374 100644 > --- a/drivers/gpu/drm/i915/i915_scheduler.c > +++ b/drivers/gpu/drm/i915/i915_scheduler.c > @@ -10,6 +10,11 @@ > #include "i915_request.h" > #include "i915_scheduler.h" > > +static struct i915_global_scheduler { > + struct kmem_cache *slab_dependencies; > + struct kmem_cache *slab_priorities; > +} global; > + > static DEFINE_SPINLOCK(schedule_lock); > > static const struct i915_request * > @@ -32,16 +37,15 @@ void i915_sched_node_init(struct i915_sched_node *node) > } > > static struct i915_dependency * > -i915_dependency_alloc(struct drm_i915_private *i915) > +i915_dependency_alloc(void) > { > - return kmem_cache_alloc(i915->dependencies, GFP_KERNEL); > + return kmem_cache_alloc(global.slab_dependencies, GFP_KERNEL); > } > > static void > -i915_dependency_free(struct drm_i915_private *i915, > - struct i915_dependency *dep) > +i915_dependency_free(struct i915_dependency *dep) > { > - kmem_cache_free(i915->dependencies, dep); > + kmem_cache_free(global.slab_dependencies, dep); > } > > bool __i915_sched_node_add_dependency(struct i915_sched_node *node, > @@ -68,25 +72,23 @@ bool __i915_sched_node_add_dependency(struct i915_sched_node *node, > return ret; > } > > -int i915_sched_node_add_dependency(struct drm_i915_private *i915, > - struct i915_sched_node *node, > +int i915_sched_node_add_dependency(struct i915_sched_node *node, > struct i915_sched_node *signal) > { > struct i915_dependency *dep; > > - dep = i915_dependency_alloc(i915); > + dep = i915_dependency_alloc(); > if (!dep) > return -ENOMEM; > > if (!__i915_sched_node_add_dependency(node, signal, dep, > I915_DEPENDENCY_ALLOC)) > - i915_dependency_free(i915, dep); > + i915_dependency_free(dep); > > return 0; > } > > -void i915_sched_node_fini(struct drm_i915_private *i915, > - struct i915_sched_node *node) > +void i915_sched_node_fini(struct i915_sched_node *node) > { > struct i915_dependency *dep, *tmp; > > @@ -106,7 +108,7 @@ void i915_sched_node_fini(struct drm_i915_private *i915, > > list_del(&dep->wait_link); > if (dep->flags & I915_DEPENDENCY_ALLOC) > - i915_dependency_free(i915, dep); > + i915_dependency_free(dep); > } > > /* Remove ourselves from everyone who depends upon us */ > @@ -116,7 +118,7 @@ void i915_sched_node_fini(struct drm_i915_private *i915, > > list_del(&dep->signal_link); > if (dep->flags & I915_DEPENDENCY_ALLOC) > - i915_dependency_free(i915, dep); > + i915_dependency_free(dep); > } > > spin_unlock(&schedule_lock); > @@ -193,7 +195,7 @@ i915_sched_lookup_priolist(struct intel_engine_cs *engine, int prio) > if (prio == I915_PRIORITY_NORMAL) { > p = &execlists->default_priolist; > } else { > - p = kmem_cache_alloc(engine->i915->priorities, GFP_ATOMIC); > + p = kmem_cache_alloc(global.slab_priorities, GFP_ATOMIC); > /* Convert an allocation failure to a priority bump */ > if (unlikely(!p)) { > prio = I915_PRIORITY_NORMAL; /* recurses just once */ > @@ -408,3 +410,39 @@ void i915_schedule_bump_priority(struct i915_request *rq, unsigned int bump) > > spin_unlock_bh(&schedule_lock); > } > + > +void __i915_priolist_free(struct i915_priolist *p) > +{ > + kmem_cache_free(global.slab_priorities, p); > +} > + > +int i915_global_scheduler_init(void) > +{ > + global.slab_dependencies = KMEM_CACHE(i915_dependency, > + SLAB_HWCACHE_ALIGN); > + if (!global.slab_dependencies) > + return -ENOMEM; Right, so this slab is duplicated. It could end up merged by the core, but I am thinking if this is the direction we want to go just to avoid some pointer chasing. You wouldn't consider i915->global->slab_dependencies or something along those lines? Regards, Tvrtko > + > + global.slab_priorities = KMEM_CACHE(i915_priolist, > + SLAB_HWCACHE_ALIGN); > + if (!global.slab_priorities) > + goto err_priorities; > + > + return 0; > + > +err_priorities: > + kmem_cache_destroy(global.slab_priorities); > + return -ENOMEM; > +} > + > +void i915_global_scheduler_shrink(void) > +{ > + kmem_cache_shrink(global.slab_dependencies); > + kmem_cache_shrink(global.slab_priorities); > +} > + > +void i915_global_scheduler_exit(void) > +{ > + kmem_cache_destroy(global.slab_dependencies); > + kmem_cache_destroy(global.slab_priorities); > +} > diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h > index 54bd6c89817e..5196ce07b6c2 100644 > --- a/drivers/gpu/drm/i915/i915_scheduler.h > +++ b/drivers/gpu/drm/i915/i915_scheduler.h > @@ -85,6 +85,23 @@ struct i915_dependency { > #define I915_DEPENDENCY_ALLOC BIT(0) > }; > > +struct i915_priolist { > + struct list_head requests[I915_PRIORITY_COUNT]; > + struct rb_node node; > + unsigned long used; > + int priority; > +}; > + > +#define priolist_for_each_request(it, plist, idx) \ > + for (idx = 0; idx < ARRAY_SIZE((plist)->requests); idx++) \ > + list_for_each_entry(it, &(plist)->requests[idx], sched.link) > + > +#define priolist_for_each_request_consume(it, n, plist, idx) \ > + for (; (idx = ffs((plist)->used)); (plist)->used &= ~BIT(idx - 1)) \ > + list_for_each_entry_safe(it, n, \ > + &(plist)->requests[idx - 1], \ > + sched.link) > + > void i915_sched_node_init(struct i915_sched_node *node); > > bool __i915_sched_node_add_dependency(struct i915_sched_node *node, > @@ -92,12 +109,10 @@ bool __i915_sched_node_add_dependency(struct i915_sched_node *node, > struct i915_dependency *dep, > unsigned long flags); > > -int i915_sched_node_add_dependency(struct drm_i915_private *i915, > - struct i915_sched_node *node, > +int i915_sched_node_add_dependency(struct i915_sched_node *node, > struct i915_sched_node *signal); > > -void i915_sched_node_fini(struct drm_i915_private *i915, > - struct i915_sched_node *node); > +void i915_sched_node_fini(struct i915_sched_node *node); > > void i915_schedule(struct i915_request *request, > const struct i915_sched_attr *attr); > @@ -107,4 +122,15 @@ void i915_schedule_bump_priority(struct i915_request *rq, unsigned int bump); > struct list_head * > i915_sched_lookup_priolist(struct intel_engine_cs *engine, int prio); > > +void __i915_priolist_free(struct i915_priolist *p); > +static inline void i915_priolist_free(struct i915_priolist *p) > +{ > + if (p->priority != I915_PRIORITY_NORMAL) > + __i915_priolist_free(p); > +} > + > +int i915_global_scheduler_init(void); > +void i915_global_scheduler_shrink(void); > +void i915_global_scheduler_exit(void); > + > #endif /* _I915_SCHEDULER_H_ */ > diff --git a/drivers/gpu/drm/i915/intel_guc_submission.c b/drivers/gpu/drm/i915/intel_guc_submission.c > index 8bc8aa54aa35..4cf94513615d 100644 > --- a/drivers/gpu/drm/i915/intel_guc_submission.c > +++ b/drivers/gpu/drm/i915/intel_guc_submission.c > @@ -781,8 +781,7 @@ static bool __guc_dequeue(struct intel_engine_cs *engine) > } > > rb_erase_cached(&p->node, &execlists->queue); > - if (p->priority != I915_PRIORITY_NORMAL) > - kmem_cache_free(engine->i915->priorities, p); > + i915_priolist_free(p); > } > done: > execlists->queue_priority_hint = > diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c > index 8e301f19036b..e37f207afb5a 100644 > --- a/drivers/gpu/drm/i915/intel_lrc.c > +++ b/drivers/gpu/drm/i915/intel_lrc.c > @@ -806,8 +806,7 @@ static void execlists_dequeue(struct intel_engine_cs *engine) > } > > rb_erase_cached(&p->node, &execlists->queue); > - if (p->priority != I915_PRIORITY_NORMAL) > - kmem_cache_free(engine->i915->priorities, p); > + i915_priolist_free(p); > } > > done: > @@ -966,8 +965,7 @@ static void execlists_cancel_requests(struct intel_engine_cs *engine) > } > > rb_erase_cached(&p->node, &execlists->queue); > - if (p->priority != I915_PRIORITY_NORMAL) > - kmem_cache_free(engine->i915->priorities, p); > + i915_priolist_free(p); > } > > intel_write_status_page(engine, > diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h b/drivers/gpu/drm/i915/intel_ringbuffer.h > index 8183d3441907..5dffccb6740e 100644 > --- a/drivers/gpu/drm/i915/intel_ringbuffer.h > +++ b/drivers/gpu/drm/i915/intel_ringbuffer.h > @@ -185,23 +185,6 @@ enum intel_engine_id { > #define _VECS(n) (VECS + (n)) > }; > > -struct i915_priolist { > - struct list_head requests[I915_PRIORITY_COUNT]; > - struct rb_node node; > - unsigned long used; > - int priority; > -}; > - > -#define priolist_for_each_request(it, plist, idx) \ > - for (idx = 0; idx < ARRAY_SIZE((plist)->requests); idx++) \ > - list_for_each_entry(it, &(plist)->requests[idx], sched.link) > - > -#define priolist_for_each_request_consume(it, n, plist, idx) \ > - for (; (idx = ffs((plist)->used)); (plist)->used &= ~BIT(idx - 1)) \ > - list_for_each_entry_safe(it, n, \ > - &(plist)->requests[idx - 1], \ > - sched.link) > - > struct st_preempt_hang { > struct completion completion; > unsigned int count; > diff --git a/drivers/gpu/drm/i915/selftests/intel_lrc.c b/drivers/gpu/drm/i915/selftests/intel_lrc.c > index 967cefa118ee..30ab0e04a674 100644 > --- a/drivers/gpu/drm/i915/selftests/intel_lrc.c > +++ b/drivers/gpu/drm/i915/selftests/intel_lrc.c > @@ -440,7 +440,7 @@ static struct i915_request *dummy_request(struct intel_engine_cs *engine) > static void dummy_request_free(struct i915_request *dummy) > { > i915_request_mark_complete(dummy); > - i915_sched_node_fini(dummy->engine->i915, &dummy->sched); > + i915_sched_node_fini(&dummy->sched); > kfree(dummy); > } > > diff --git a/drivers/gpu/drm/i915/selftests/mock_engine.c b/drivers/gpu/drm/i915/selftests/mock_engine.c > index 08f0cab02e0f..0d35af07867b 100644 > --- a/drivers/gpu/drm/i915/selftests/mock_engine.c > +++ b/drivers/gpu/drm/i915/selftests/mock_engine.c > @@ -76,28 +76,27 @@ static void mock_ring_free(struct intel_ring *base) > kfree(ring); > } > > -static struct mock_request *first_request(struct mock_engine *engine) > +static struct i915_request *first_request(struct mock_engine *engine) > { > return list_first_entry_or_null(&engine->hw_queue, > - struct mock_request, > - link); > + struct i915_request, > + mock.link); > } > > -static void advance(struct mock_request *request) > +static void advance(struct i915_request *request) > { > - list_del_init(&request->link); > - intel_engine_write_global_seqno(request->base.engine, > - request->base.global_seqno); > - i915_request_mark_complete(&request->base); > - GEM_BUG_ON(!i915_request_completed(&request->base)); > + list_del_init(&request->mock.link); > + intel_engine_write_global_seqno(request->engine, request->global_seqno); > + i915_request_mark_complete(request); > + GEM_BUG_ON(!i915_request_completed(request)); > > - intel_engine_queue_breadcrumbs(request->base.engine); > + intel_engine_queue_breadcrumbs(request->engine); > } > > static void hw_delay_complete(struct timer_list *t) > { > struct mock_engine *engine = from_timer(engine, t, hw_delay); > - struct mock_request *request; > + struct i915_request *request; > unsigned long flags; > > spin_lock_irqsave(&engine->hw_lock, flags); > @@ -112,8 +111,9 @@ static void hw_delay_complete(struct timer_list *t) > * requeue the timer for the next delayed request. > */ > while ((request = first_request(engine))) { > - if (request->delay) { > - mod_timer(&engine->hw_delay, jiffies + request->delay); > + if (request->mock.delay) { > + mod_timer(&engine->hw_delay, > + jiffies + request->mock.delay); > break; > } > > @@ -171,10 +171,8 @@ mock_context_pin(struct intel_engine_cs *engine, > > static int mock_request_alloc(struct i915_request *request) > { > - struct mock_request *mock = container_of(request, typeof(*mock), base); > - > - INIT_LIST_HEAD(&mock->link); > - mock->delay = 0; > + INIT_LIST_HEAD(&request->mock.link); > + request->mock.delay = 0; > > return 0; > } > @@ -192,7 +190,6 @@ static u32 *mock_emit_breadcrumb(struct i915_request *request, u32 *cs) > > static void mock_submit_request(struct i915_request *request) > { > - struct mock_request *mock = container_of(request, typeof(*mock), base); > struct mock_engine *engine = > container_of(request->engine, typeof(*engine), base); > unsigned long flags; > @@ -201,12 +198,13 @@ static void mock_submit_request(struct i915_request *request) > GEM_BUG_ON(!request->global_seqno); > > spin_lock_irqsave(&engine->hw_lock, flags); > - list_add_tail(&mock->link, &engine->hw_queue); > - if (mock->link.prev == &engine->hw_queue) { > - if (mock->delay) > - mod_timer(&engine->hw_delay, jiffies + mock->delay); > + list_add_tail(&request->mock.link, &engine->hw_queue); > + if (list_is_first(&request->mock.link, &engine->hw_queue)) { > + if (request->mock.delay) > + mod_timer(&engine->hw_delay, > + jiffies + request->mock.delay); > else > - advance(mock); > + advance(request); > } > spin_unlock_irqrestore(&engine->hw_lock, flags); > } > @@ -266,12 +264,12 @@ void mock_engine_flush(struct intel_engine_cs *engine) > { > struct mock_engine *mock = > container_of(engine, typeof(*mock), base); > - struct mock_request *request, *rn; > + struct i915_request *request, *rn; > > del_timer_sync(&mock->hw_delay); > > spin_lock_irq(&mock->hw_lock); > - list_for_each_entry_safe(request, rn, &mock->hw_queue, link) > + list_for_each_entry_safe(request, rn, &mock->hw_queue, mock.link) > advance(request); > spin_unlock_irq(&mock->hw_lock); > } > diff --git a/drivers/gpu/drm/i915/selftests/mock_gem_device.c b/drivers/gpu/drm/i915/selftests/mock_gem_device.c > index 074a0d9cbf26..17915a2d94fa 100644 > --- a/drivers/gpu/drm/i915/selftests/mock_gem_device.c > +++ b/drivers/gpu/drm/i915/selftests/mock_gem_device.c > @@ -79,9 +79,6 @@ static void mock_device_release(struct drm_device *dev) > > destroy_workqueue(i915->wq); > > - kmem_cache_destroy(i915->priorities); > - kmem_cache_destroy(i915->dependencies); > - kmem_cache_destroy(i915->requests); > kmem_cache_destroy(i915->vmas); > kmem_cache_destroy(i915->objects); > > @@ -211,23 +208,6 @@ struct drm_i915_private *mock_gem_device(void) > if (!i915->vmas) > goto err_objects; > > - i915->requests = KMEM_CACHE(mock_request, > - SLAB_HWCACHE_ALIGN | > - SLAB_RECLAIM_ACCOUNT | > - SLAB_TYPESAFE_BY_RCU); > - if (!i915->requests) > - goto err_vmas; > - > - i915->dependencies = KMEM_CACHE(i915_dependency, > - SLAB_HWCACHE_ALIGN | > - SLAB_RECLAIM_ACCOUNT); > - if (!i915->dependencies) > - goto err_requests; > - > - i915->priorities = KMEM_CACHE(i915_priolist, SLAB_HWCACHE_ALIGN); > - if (!i915->priorities) > - goto err_dependencies; > - > i915_timelines_init(i915); > > INIT_LIST_HEAD(&i915->gt.active_rings); > @@ -257,12 +237,6 @@ struct drm_i915_private *mock_gem_device(void) > err_unlock: > mutex_unlock(&i915->drm.struct_mutex); > i915_timelines_fini(i915); > - kmem_cache_destroy(i915->priorities); > -err_dependencies: > - kmem_cache_destroy(i915->dependencies); > -err_requests: > - kmem_cache_destroy(i915->requests); > -err_vmas: > kmem_cache_destroy(i915->vmas); > err_objects: > kmem_cache_destroy(i915->objects); > diff --git a/drivers/gpu/drm/i915/selftests/mock_request.c b/drivers/gpu/drm/i915/selftests/mock_request.c > index 0dc29e242597..d1a7c9608712 100644 > --- a/drivers/gpu/drm/i915/selftests/mock_request.c > +++ b/drivers/gpu/drm/i915/selftests/mock_request.c > @@ -31,29 +31,25 @@ mock_request(struct intel_engine_cs *engine, > unsigned long delay) > { > struct i915_request *request; > - struct mock_request *mock; > > /* NB the i915->requests slab cache is enlarged to fit mock_request */ > request = i915_request_alloc(engine, context); > if (IS_ERR(request)) > return NULL; > > - mock = container_of(request, typeof(*mock), base); > - mock->delay = delay; > - > - return &mock->base; > + request->mock.delay = delay; > + return request; > } > > bool mock_cancel_request(struct i915_request *request) > { > - struct mock_request *mock = container_of(request, typeof(*mock), base); > struct mock_engine *engine = > container_of(request->engine, typeof(*engine), base); > bool was_queued; > > spin_lock_irq(&engine->hw_lock); > - was_queued = !list_empty(&mock->link); > - list_del_init(&mock->link); > + was_queued = !list_empty(&request->mock.link); > + list_del_init(&request->mock.link); > spin_unlock_irq(&engine->hw_lock); > > if (was_queued) > diff --git a/drivers/gpu/drm/i915/selftests/mock_request.h b/drivers/gpu/drm/i915/selftests/mock_request.h > index 995fb728380c..4acf0211df20 100644 > --- a/drivers/gpu/drm/i915/selftests/mock_request.h > +++ b/drivers/gpu/drm/i915/selftests/mock_request.h > @@ -29,13 +29,6 @@ > > #include "../i915_request.h" > > -struct mock_request { > - struct i915_request base; > - > - struct list_head link; > - unsigned long delay; > -}; > - > struct i915_request * > mock_request(struct intel_engine_cs *engine, > struct i915_gem_context *context, >
Quoting Tvrtko Ursulin (2019-02-04 18:48:50) > > On 04/02/2019 13:22, Chris Wilson wrote > > -int __init i915_global_active_init(void) > > +int i915_global_active_init(void) > > These can't remain __init, since they are only called from the global > __init one? I ran into problems, and removed __init until it stopped complaining and I stopped caring. > > @@ -2916,12 +2917,11 @@ static void shrink_caches(struct drm_i915_private *i915) > > * filled slabs to prioritise allocating from the mostly full slabs, > > * with the aim of reducing fragmentation. > > */ > > - kmem_cache_shrink(i915->priorities); > > - kmem_cache_shrink(i915->dependencies); > > - kmem_cache_shrink(i915->requests); > > kmem_cache_shrink(i915->luts); > > kmem_cache_shrink(i915->vmas); > > kmem_cache_shrink(i915->objects); > > + > > + i915_globals_shrink(); > > This is the main bit which worries me. > > Global caches are what we want I think, exactly for what you wrote in > the commit message. But would one device going idle have the potential > to inject some latency into another potentially very busy client? > > Perhaps we could have some sort of aggregated idle signal and defer > shrinking cached to that point. Like a bitmask of global clients > reporting their idle/active status to global core, and then shrink > happens only if all are idle. I had mixed feelings too. I didn't want to completely discard the current logic, but this should be shrinking only when idle across all future stakeholders... or we demonstrate that shrinking has no effect on concurrent allocation latency. An active counter for unparking seems an easy way out. (Which today is this imaginary 1bit counter.) What I thought helped save this was this is done from a post-rcu worker, the system has to be pretty stable for us to start shrinking. We only clash with the first user to wake up. In the caller it does a loop over each cpu removing the local cpu cache and then the global slab cache. That is clearly going to increase latency for a concurrent caller. > > +int i915_global_scheduler_init(void) > > +{ > > + global.slab_dependencies = KMEM_CACHE(i915_dependency, > > + SLAB_HWCACHE_ALIGN); > > + if (!global.slab_dependencies) > > + return -ENOMEM; > > Right, so this slab is duplicated. It could end up merged by the core, > but I am thinking if this is the direction we want to go just to avoid > some pointer chasing. "some pointer chasing" :) The slab isn't necessary duplicated, that depends on compiletime policy. In debug environments or those more sensitive to performance, it will be private so that we can catch stray writes and what not. add/remove: 11/0 grow/shrink: 11/20 up/down: 595/-668 (-73) Function old new delta i915_global_request_init - 116 +116 i915_global_scheduler_init - 111 +111 igt_mock_ppgtt_misaligned_dma 679 748 +69 i915_globals_init - 53 +53 global 8 40 +32 i915_global_scheduler_shrink - 29 +29 i915_global_scheduler_exit - 29 +29 i915_global_request_shrink - 29 +29 i915_global_request_exit - 29 +29 i915_globals_shrink - 20 +20 __i915_priolist_free - 20 +20 i915_global_active_shrink - 17 +17 i915_globals_exit - 15 +15 live_suppress_wait_preempt.part.cold 202 211 +9 __err_print_to_sgl 4175 4181 +6 i915_global_active_exit 12 17 +5 intel_engine_lookup_user 54 55 +1 init_module 88 89 +1 igt_mock_ppgtt_misaligned_dma.cold 246 247 +1 i915_init 88 89 +1 gen11_irq_handler 733 734 +1 g4x_pre_enable_dp 345 346 +1 ring_request_alloc 1899 1898 -1 live_suppress_wait_preempt.part 1291 1290 -1 i915_sched_lookup_priolist 482 479 -3 i915_request_retire 1377 1373 -4 i915_request_await_dma_fence 547 543 -4 i915_fence_release 45 41 -4 __execlists_submission_tasklet 2121 2111 -10 i915_request_alloc 817 806 -11 i915_request_alloc_slow.isra 76 64 -12 i915_sched_node_add_dependency 114 101 -13 execlists_cancel_requests 690 676 -14 i915_sched_node_fini 459 444 -15 guc_submission_tasklet 1931 1916 -15 __sleep_work 106 75 -31 mock_device_release 407 371 -36 igt_mock_ppgtt_huge_fill 1108 1069 -39 i915_gem_cleanup_early 213 173 -40 igt_mock_ppgtt_huge_fill.cold 611 531 -80 mock_gem_device 1263 1102 -161 i915_gem_init_early 838 664 -174 __i915_priolist_free is the nasty one, but that's only hit for !I915_PRIORITY_NORMAL so I considered it to not be worth inlining. (I have no idea what the compiler is thinking changed in half those functions.) > You wouldn't consider i915->global->slab_dependencies or something along > those lines? I think for bits and bobs that are true globals like allocators, global variables do make sense. We either end up with pointers to a singleton, or just directly link them into the code. And I like the idea of easy wins. -Chris
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index 1787e1299b1b..a1d834068765 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -77,6 +77,7 @@ i915-y += \ i915_gem_tiling.o \ i915_gem_userptr.o \ i915_gemfs.o \ + i915_globals.o \ i915_query.o \ i915_request.o \ i915_scheduler.o \ diff --git a/drivers/gpu/drm/i915/i915_active.c b/drivers/gpu/drm/i915/i915_active.c index 64661c41532b..d23092d8c89f 100644 --- a/drivers/gpu/drm/i915/i915_active.c +++ b/drivers/gpu/drm/i915/i915_active.c @@ -251,7 +251,7 @@ void i915_active_fini(struct i915_active *ref) #include "selftests/i915_active.c" #endif -int __init i915_global_active_init(void) +int i915_global_active_init(void) { global.slab_cache = KMEM_CACHE(active_node, SLAB_HWCACHE_ALIGN); if (!global.slab_cache) @@ -260,7 +260,12 @@ int __init i915_global_active_init(void) return 0; } -void __exit i915_global_active_exit(void) +void i915_global_active_shrink(void) +{ + kmem_cache_shrink(global.slab_cache); +} + +void i915_global_active_exit(void) { kmem_cache_destroy(global.slab_cache); } diff --git a/drivers/gpu/drm/i915/i915_active.h b/drivers/gpu/drm/i915/i915_active.h index 179b47aeec33..6c56d10b1f59 100644 --- a/drivers/gpu/drm/i915/i915_active.h +++ b/drivers/gpu/drm/i915/i915_active.h @@ -71,6 +71,7 @@ static inline void i915_active_fini(struct i915_active *ref) { } #endif int i915_global_active_init(void); +void i915_global_active_shrink(void); void i915_global_active_exit(void); #endif /* _I915_ACTIVE_H_ */ diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h index 3e4538ce5276..e48e3c228d9c 100644 --- a/drivers/gpu/drm/i915/i915_drv.h +++ b/drivers/gpu/drm/i915/i915_drv.h @@ -1459,9 +1459,6 @@ struct drm_i915_private { struct kmem_cache *objects; struct kmem_cache *vmas; struct kmem_cache *luts; - struct kmem_cache *requests; - struct kmem_cache *dependencies; - struct kmem_cache *priorities; const struct intel_device_info __info; /* Use INTEL_INFO() to access. */ struct intel_runtime_info __runtime; /* Use RUNTIME_INFO() to access. */ diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c index 2c6161c89cc7..d82e4f990586 100644 --- a/drivers/gpu/drm/i915/i915_gem.c +++ b/drivers/gpu/drm/i915/i915_gem.c @@ -42,6 +42,7 @@ #include "i915_drv.h" #include "i915_gem_clflush.h" #include "i915_gemfs.h" +#include "i915_globals.h" #include "i915_reset.h" #include "i915_trace.h" #include "i915_vgpu.h" @@ -2916,12 +2917,11 @@ static void shrink_caches(struct drm_i915_private *i915) * filled slabs to prioritise allocating from the mostly full slabs, * with the aim of reducing fragmentation. */ - kmem_cache_shrink(i915->priorities); - kmem_cache_shrink(i915->dependencies); - kmem_cache_shrink(i915->requests); kmem_cache_shrink(i915->luts); kmem_cache_shrink(i915->vmas); kmem_cache_shrink(i915->objects); + + i915_globals_shrink(); } struct sleep_rcu_work { @@ -5264,23 +5264,6 @@ int i915_gem_init_early(struct drm_i915_private *dev_priv) if (!dev_priv->luts) goto err_vmas; - dev_priv->requests = KMEM_CACHE(i915_request, - SLAB_HWCACHE_ALIGN | - SLAB_RECLAIM_ACCOUNT | - SLAB_TYPESAFE_BY_RCU); - if (!dev_priv->requests) - goto err_luts; - - dev_priv->dependencies = KMEM_CACHE(i915_dependency, - SLAB_HWCACHE_ALIGN | - SLAB_RECLAIM_ACCOUNT); - if (!dev_priv->dependencies) - goto err_requests; - - dev_priv->priorities = KMEM_CACHE(i915_priolist, SLAB_HWCACHE_ALIGN); - if (!dev_priv->priorities) - goto err_dependencies; - INIT_LIST_HEAD(&dev_priv->gt.active_rings); INIT_LIST_HEAD(&dev_priv->gt.closed_vma); @@ -5305,12 +5288,6 @@ int i915_gem_init_early(struct drm_i915_private *dev_priv) return 0; -err_dependencies: - kmem_cache_destroy(dev_priv->dependencies); -err_requests: - kmem_cache_destroy(dev_priv->requests); -err_luts: - kmem_cache_destroy(dev_priv->luts); err_vmas: kmem_cache_destroy(dev_priv->vmas); err_objects: @@ -5328,9 +5305,6 @@ void i915_gem_cleanup_early(struct drm_i915_private *dev_priv) cleanup_srcu_struct(&dev_priv->gpu_error.srcu); - kmem_cache_destroy(dev_priv->priorities); - kmem_cache_destroy(dev_priv->dependencies); - kmem_cache_destroy(dev_priv->requests); kmem_cache_destroy(dev_priv->luts); kmem_cache_destroy(dev_priv->vmas); kmem_cache_destroy(dev_priv->objects); diff --git a/drivers/gpu/drm/i915/i915_globals.c b/drivers/gpu/drm/i915/i915_globals.c new file mode 100644 index 000000000000..2ecf9897fd16 --- /dev/null +++ b/drivers/gpu/drm/i915/i915_globals.c @@ -0,0 +1,49 @@ +/* + * SPDX-License-Identifier: MIT + * + * Copyright © 2019 Intel Corporation + */ + +#include "i915_active.h" +#include "i915_globals.h" +#include "i915_request.h" +#include "i915_scheduler.h" + +int __init i915_globals_init(void) +{ + int err; + + err = i915_global_active_init(); + if (err) + return err; + + err = i915_global_request_init(); + if (err) + goto err_active; + + err = i915_global_scheduler_init(); + if (err) + goto err_request; + + return 0; + +err_request: + i915_global_request_exit(); +err_active: + i915_global_active_exit(); + return err; +} + +void i915_globals_shrink(void) +{ + i915_global_active_shrink(); + i915_global_request_shrink(); + i915_global_scheduler_shrink(); +} + +void __exit i915_globals_exit(void) +{ + i915_global_scheduler_exit(); + i915_global_request_exit(); + i915_global_active_exit(); +} diff --git a/drivers/gpu/drm/i915/i915_globals.h b/drivers/gpu/drm/i915/i915_globals.h new file mode 100644 index 000000000000..903f52c0a1d2 --- /dev/null +++ b/drivers/gpu/drm/i915/i915_globals.h @@ -0,0 +1,14 @@ +/* + * SPDX-License-Identifier: MIT + * + * Copyright © 2019 Intel Corporation + */ + +#ifndef _I915_GLOBALS_H_ +#define _I915_GLOBALS_H_ + +int i915_globals_init(void); +void i915_globals_shrink(void); +void i915_globals_exit(void); + +#endif /* _I915_GLOBALS_H_ */ diff --git a/drivers/gpu/drm/i915/i915_pci.c b/drivers/gpu/drm/i915/i915_pci.c index 852b6b4e8ed8..0d684eb530c3 100644 --- a/drivers/gpu/drm/i915/i915_pci.c +++ b/drivers/gpu/drm/i915/i915_pci.c @@ -28,8 +28,8 @@ #include <drm/drm_drv.h> -#include "i915_active.h" #include "i915_drv.h" +#include "i915_globals.h" #include "i915_selftest.h" #define PLATFORM(x) .platform = (x), .platform_mask = BIT(x) @@ -801,7 +801,9 @@ static int __init i915_init(void) bool use_kms = true; int err; - i915_global_active_init(); + err = i915_globals_init(); + if (err) + return err; err = i915_mock_selftests(); if (err) @@ -834,7 +836,7 @@ static void __exit i915_exit(void) return; pci_unregister_driver(&i915_pci_driver); - i915_global_active_exit(); + i915_globals_exit(); } module_init(i915_init); diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c index 04c65e6d83b9..3bb4840ba761 100644 --- a/drivers/gpu/drm/i915/i915_request.c +++ b/drivers/gpu/drm/i915/i915_request.c @@ -31,6 +31,11 @@ #include "i915_drv.h" #include "i915_reset.h" +static struct i915_global_request { + struct kmem_cache *slab_requests; + struct kmem_cache *slab_dependencies; +} global; + static const char *i915_fence_get_driver_name(struct dma_fence *fence) { return "i915"; @@ -83,7 +88,7 @@ static void i915_fence_release(struct dma_fence *fence) */ i915_sw_fence_fini(&rq->submit); - kmem_cache_free(rq->i915->requests, rq); + kmem_cache_free(global.slab_requests, rq); } const struct dma_fence_ops i915_fence_ops = { @@ -301,7 +306,7 @@ static void i915_request_retire(struct i915_request *request) unreserve_gt(request->i915); - i915_sched_node_fini(request->i915, &request->sched); + i915_sched_node_fini(&request->sched); i915_request_put(request); } @@ -535,7 +540,7 @@ i915_request_alloc_slow(struct intel_context *ce) ring_retire_requests(ring); out: - return kmem_cache_alloc(ce->gem_context->i915->requests, GFP_KERNEL); + return kmem_cache_alloc(global.slab_requests, GFP_KERNEL); } /** @@ -617,7 +622,7 @@ i915_request_alloc(struct intel_engine_cs *engine, struct i915_gem_context *ctx) * * Do not use kmem_cache_zalloc() here! */ - rq = kmem_cache_alloc(i915->requests, + rq = kmem_cache_alloc(global.slab_requests, GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN); if (unlikely(!rq)) { rq = i915_request_alloc_slow(ce); @@ -701,7 +706,7 @@ i915_request_alloc(struct intel_engine_cs *engine, struct i915_gem_context *ctx) GEM_BUG_ON(!list_empty(&rq->sched.signalers_list)); GEM_BUG_ON(!list_empty(&rq->sched.waiters_list)); - kmem_cache_free(i915->requests, rq); + kmem_cache_free(global.slab_requests, rq); err_unreserve: unreserve_gt(i915); intel_context_unpin(ce); @@ -720,9 +725,7 @@ i915_request_await_request(struct i915_request *to, struct i915_request *from) return 0; if (to->engine->schedule) { - ret = i915_sched_node_add_dependency(to->i915, - &to->sched, - &from->sched); + ret = i915_sched_node_add_dependency(&to->sched, &from->sched); if (ret < 0) return ret; } @@ -1195,3 +1198,37 @@ void i915_retire_requests(struct drm_i915_private *i915) #include "selftests/mock_request.c" #include "selftests/i915_request.c" #endif + +int i915_global_request_init(void) +{ + global.slab_requests = KMEM_CACHE(i915_request, + SLAB_HWCACHE_ALIGN | + SLAB_RECLAIM_ACCOUNT | + SLAB_TYPESAFE_BY_RCU); + if (!global.slab_requests) + return -ENOMEM; + + global.slab_dependencies = KMEM_CACHE(i915_dependency, + SLAB_HWCACHE_ALIGN | + SLAB_RECLAIM_ACCOUNT); + if (!global.slab_dependencies) + goto err_requests; + + return 0; + +err_requests: + kmem_cache_destroy(global.slab_requests); + return -ENOMEM; +} + +void i915_global_request_shrink(void) +{ + kmem_cache_shrink(global.slab_dependencies); + kmem_cache_shrink(global.slab_requests); +} + +void i915_global_request_exit(void) +{ + kmem_cache_destroy(global.slab_dependencies); + kmem_cache_destroy(global.slab_requests); +} diff --git a/drivers/gpu/drm/i915/i915_request.h b/drivers/gpu/drm/i915/i915_request.h index 3cffb96203b9..054bd300984b 100644 --- a/drivers/gpu/drm/i915/i915_request.h +++ b/drivers/gpu/drm/i915/i915_request.h @@ -29,6 +29,7 @@ #include "i915_gem.h" #include "i915_scheduler.h" +#include "i915_selftest.h" #include "i915_sw_fence.h" #include <uapi/drm/i915_drm.h> @@ -204,6 +205,11 @@ struct i915_request { struct drm_i915_file_private *file_priv; /** file_priv list entry for this request */ struct list_head client_link; + + I915_SELFTEST_DECLARE(struct { + struct list_head link; + unsigned long delay; + } mock;) }; #define I915_FENCE_GFP (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN) @@ -786,4 +792,8 @@ i915_gem_active_retire(struct i915_gem_active *active, #define for_each_active(mask, idx) \ for (; mask ? idx = ffs(mask) - 1, 1 : 0; mask &= ~BIT(idx)) +int i915_global_request_init(void); +void i915_global_request_shrink(void); +void i915_global_request_exit(void); + #endif /* I915_REQUEST_H */ diff --git a/drivers/gpu/drm/i915/i915_scheduler.c b/drivers/gpu/drm/i915/i915_scheduler.c index d01683167c77..7c1d9ef98374 100644 --- a/drivers/gpu/drm/i915/i915_scheduler.c +++ b/drivers/gpu/drm/i915/i915_scheduler.c @@ -10,6 +10,11 @@ #include "i915_request.h" #include "i915_scheduler.h" +static struct i915_global_scheduler { + struct kmem_cache *slab_dependencies; + struct kmem_cache *slab_priorities; +} global; + static DEFINE_SPINLOCK(schedule_lock); static const struct i915_request * @@ -32,16 +37,15 @@ void i915_sched_node_init(struct i915_sched_node *node) } static struct i915_dependency * -i915_dependency_alloc(struct drm_i915_private *i915) +i915_dependency_alloc(void) { - return kmem_cache_alloc(i915->dependencies, GFP_KERNEL); + return kmem_cache_alloc(global.slab_dependencies, GFP_KERNEL); } static void -i915_dependency_free(struct drm_i915_private *i915, - struct i915_dependency *dep) +i915_dependency_free(struct i915_dependency *dep) { - kmem_cache_free(i915->dependencies, dep); + kmem_cache_free(global.slab_dependencies, dep); } bool __i915_sched_node_add_dependency(struct i915_sched_node *node, @@ -68,25 +72,23 @@ bool __i915_sched_node_add_dependency(struct i915_sched_node *node, return ret; } -int i915_sched_node_add_dependency(struct drm_i915_private *i915, - struct i915_sched_node *node, +int i915_sched_node_add_dependency(struct i915_sched_node *node, struct i915_sched_node *signal) { struct i915_dependency *dep; - dep = i915_dependency_alloc(i915); + dep = i915_dependency_alloc(); if (!dep) return -ENOMEM; if (!__i915_sched_node_add_dependency(node, signal, dep, I915_DEPENDENCY_ALLOC)) - i915_dependency_free(i915, dep); + i915_dependency_free(dep); return 0; } -void i915_sched_node_fini(struct drm_i915_private *i915, - struct i915_sched_node *node) +void i915_sched_node_fini(struct i915_sched_node *node) { struct i915_dependency *dep, *tmp; @@ -106,7 +108,7 @@ void i915_sched_node_fini(struct drm_i915_private *i915, list_del(&dep->wait_link); if (dep->flags & I915_DEPENDENCY_ALLOC) - i915_dependency_free(i915, dep); + i915_dependency_free(dep); } /* Remove ourselves from everyone who depends upon us */ @@ -116,7 +118,7 @@ void i915_sched_node_fini(struct drm_i915_private *i915, list_del(&dep->signal_link); if (dep->flags & I915_DEPENDENCY_ALLOC) - i915_dependency_free(i915, dep); + i915_dependency_free(dep); } spin_unlock(&schedule_lock); @@ -193,7 +195,7 @@ i915_sched_lookup_priolist(struct intel_engine_cs *engine, int prio) if (prio == I915_PRIORITY_NORMAL) { p = &execlists->default_priolist; } else { - p = kmem_cache_alloc(engine->i915->priorities, GFP_ATOMIC); + p = kmem_cache_alloc(global.slab_priorities, GFP_ATOMIC); /* Convert an allocation failure to a priority bump */ if (unlikely(!p)) { prio = I915_PRIORITY_NORMAL; /* recurses just once */ @@ -408,3 +410,39 @@ void i915_schedule_bump_priority(struct i915_request *rq, unsigned int bump) spin_unlock_bh(&schedule_lock); } + +void __i915_priolist_free(struct i915_priolist *p) +{ + kmem_cache_free(global.slab_priorities, p); +} + +int i915_global_scheduler_init(void) +{ + global.slab_dependencies = KMEM_CACHE(i915_dependency, + SLAB_HWCACHE_ALIGN); + if (!global.slab_dependencies) + return -ENOMEM; + + global.slab_priorities = KMEM_CACHE(i915_priolist, + SLAB_HWCACHE_ALIGN); + if (!global.slab_priorities) + goto err_priorities; + + return 0; + +err_priorities: + kmem_cache_destroy(global.slab_priorities); + return -ENOMEM; +} + +void i915_global_scheduler_shrink(void) +{ + kmem_cache_shrink(global.slab_dependencies); + kmem_cache_shrink(global.slab_priorities); +} + +void i915_global_scheduler_exit(void) +{ + kmem_cache_destroy(global.slab_dependencies); + kmem_cache_destroy(global.slab_priorities); +} diff --git a/drivers/gpu/drm/i915/i915_scheduler.h b/drivers/gpu/drm/i915/i915_scheduler.h index 54bd6c89817e..5196ce07b6c2 100644 --- a/drivers/gpu/drm/i915/i915_scheduler.h +++ b/drivers/gpu/drm/i915/i915_scheduler.h @@ -85,6 +85,23 @@ struct i915_dependency { #define I915_DEPENDENCY_ALLOC BIT(0) }; +struct i915_priolist { + struct list_head requests[I915_PRIORITY_COUNT]; + struct rb_node node; + unsigned long used; + int priority; +}; + +#define priolist_for_each_request(it, plist, idx) \ + for (idx = 0; idx < ARRAY_SIZE((plist)->requests); idx++) \ + list_for_each_entry(it, &(plist)->requests[idx], sched.link) + +#define priolist_for_each_request_consume(it, n, plist, idx) \ + for (; (idx = ffs((plist)->used)); (plist)->used &= ~BIT(idx - 1)) \ + list_for_each_entry_safe(it, n, \ + &(plist)->requests[idx - 1], \ + sched.link) + void i915_sched_node_init(struct i915_sched_node *node); bool __i915_sched_node_add_dependency(struct i915_sched_node *node, @@ -92,12 +109,10 @@ bool __i915_sched_node_add_dependency(struct i915_sched_node *node, struct i915_dependency *dep, unsigned long flags); -int i915_sched_node_add_dependency(struct drm_i915_private *i915, - struct i915_sched_node *node, +int i915_sched_node_add_dependency(struct i915_sched_node *node, struct i915_sched_node *signal); -void i915_sched_node_fini(struct drm_i915_private *i915, - struct i915_sched_node *node); +void i915_sched_node_fini(struct i915_sched_node *node); void i915_schedule(struct i915_request *request, const struct i915_sched_attr *attr); @@ -107,4 +122,15 @@ void i915_schedule_bump_priority(struct i915_request *rq, unsigned int bump); struct list_head * i915_sched_lookup_priolist(struct intel_engine_cs *engine, int prio); +void __i915_priolist_free(struct i915_priolist *p); +static inline void i915_priolist_free(struct i915_priolist *p) +{ + if (p->priority != I915_PRIORITY_NORMAL) + __i915_priolist_free(p); +} + +int i915_global_scheduler_init(void); +void i915_global_scheduler_shrink(void); +void i915_global_scheduler_exit(void); + #endif /* _I915_SCHEDULER_H_ */ diff --git a/drivers/gpu/drm/i915/intel_guc_submission.c b/drivers/gpu/drm/i915/intel_guc_submission.c index 8bc8aa54aa35..4cf94513615d 100644 --- a/drivers/gpu/drm/i915/intel_guc_submission.c +++ b/drivers/gpu/drm/i915/intel_guc_submission.c @@ -781,8 +781,7 @@ static bool __guc_dequeue(struct intel_engine_cs *engine) } rb_erase_cached(&p->node, &execlists->queue); - if (p->priority != I915_PRIORITY_NORMAL) - kmem_cache_free(engine->i915->priorities, p); + i915_priolist_free(p); } done: execlists->queue_priority_hint = diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c index 8e301f19036b..e37f207afb5a 100644 --- a/drivers/gpu/drm/i915/intel_lrc.c +++ b/drivers/gpu/drm/i915/intel_lrc.c @@ -806,8 +806,7 @@ static void execlists_dequeue(struct intel_engine_cs *engine) } rb_erase_cached(&p->node, &execlists->queue); - if (p->priority != I915_PRIORITY_NORMAL) - kmem_cache_free(engine->i915->priorities, p); + i915_priolist_free(p); } done: @@ -966,8 +965,7 @@ static void execlists_cancel_requests(struct intel_engine_cs *engine) } rb_erase_cached(&p->node, &execlists->queue); - if (p->priority != I915_PRIORITY_NORMAL) - kmem_cache_free(engine->i915->priorities, p); + i915_priolist_free(p); } intel_write_status_page(engine, diff --git a/drivers/gpu/drm/i915/intel_ringbuffer.h b/drivers/gpu/drm/i915/intel_ringbuffer.h index 8183d3441907..5dffccb6740e 100644 --- a/drivers/gpu/drm/i915/intel_ringbuffer.h +++ b/drivers/gpu/drm/i915/intel_ringbuffer.h @@ -185,23 +185,6 @@ enum intel_engine_id { #define _VECS(n) (VECS + (n)) }; -struct i915_priolist { - struct list_head requests[I915_PRIORITY_COUNT]; - struct rb_node node; - unsigned long used; - int priority; -}; - -#define priolist_for_each_request(it, plist, idx) \ - for (idx = 0; idx < ARRAY_SIZE((plist)->requests); idx++) \ - list_for_each_entry(it, &(plist)->requests[idx], sched.link) - -#define priolist_for_each_request_consume(it, n, plist, idx) \ - for (; (idx = ffs((plist)->used)); (plist)->used &= ~BIT(idx - 1)) \ - list_for_each_entry_safe(it, n, \ - &(plist)->requests[idx - 1], \ - sched.link) - struct st_preempt_hang { struct completion completion; unsigned int count; diff --git a/drivers/gpu/drm/i915/selftests/intel_lrc.c b/drivers/gpu/drm/i915/selftests/intel_lrc.c index 967cefa118ee..30ab0e04a674 100644 --- a/drivers/gpu/drm/i915/selftests/intel_lrc.c +++ b/drivers/gpu/drm/i915/selftests/intel_lrc.c @@ -440,7 +440,7 @@ static struct i915_request *dummy_request(struct intel_engine_cs *engine) static void dummy_request_free(struct i915_request *dummy) { i915_request_mark_complete(dummy); - i915_sched_node_fini(dummy->engine->i915, &dummy->sched); + i915_sched_node_fini(&dummy->sched); kfree(dummy); } diff --git a/drivers/gpu/drm/i915/selftests/mock_engine.c b/drivers/gpu/drm/i915/selftests/mock_engine.c index 08f0cab02e0f..0d35af07867b 100644 --- a/drivers/gpu/drm/i915/selftests/mock_engine.c +++ b/drivers/gpu/drm/i915/selftests/mock_engine.c @@ -76,28 +76,27 @@ static void mock_ring_free(struct intel_ring *base) kfree(ring); } -static struct mock_request *first_request(struct mock_engine *engine) +static struct i915_request *first_request(struct mock_engine *engine) { return list_first_entry_or_null(&engine->hw_queue, - struct mock_request, - link); + struct i915_request, + mock.link); } -static void advance(struct mock_request *request) +static void advance(struct i915_request *request) { - list_del_init(&request->link); - intel_engine_write_global_seqno(request->base.engine, - request->base.global_seqno); - i915_request_mark_complete(&request->base); - GEM_BUG_ON(!i915_request_completed(&request->base)); + list_del_init(&request->mock.link); + intel_engine_write_global_seqno(request->engine, request->global_seqno); + i915_request_mark_complete(request); + GEM_BUG_ON(!i915_request_completed(request)); - intel_engine_queue_breadcrumbs(request->base.engine); + intel_engine_queue_breadcrumbs(request->engine); } static void hw_delay_complete(struct timer_list *t) { struct mock_engine *engine = from_timer(engine, t, hw_delay); - struct mock_request *request; + struct i915_request *request; unsigned long flags; spin_lock_irqsave(&engine->hw_lock, flags); @@ -112,8 +111,9 @@ static void hw_delay_complete(struct timer_list *t) * requeue the timer for the next delayed request. */ while ((request = first_request(engine))) { - if (request->delay) { - mod_timer(&engine->hw_delay, jiffies + request->delay); + if (request->mock.delay) { + mod_timer(&engine->hw_delay, + jiffies + request->mock.delay); break; } @@ -171,10 +171,8 @@ mock_context_pin(struct intel_engine_cs *engine, static int mock_request_alloc(struct i915_request *request) { - struct mock_request *mock = container_of(request, typeof(*mock), base); - - INIT_LIST_HEAD(&mock->link); - mock->delay = 0; + INIT_LIST_HEAD(&request->mock.link); + request->mock.delay = 0; return 0; } @@ -192,7 +190,6 @@ static u32 *mock_emit_breadcrumb(struct i915_request *request, u32 *cs) static void mock_submit_request(struct i915_request *request) { - struct mock_request *mock = container_of(request, typeof(*mock), base); struct mock_engine *engine = container_of(request->engine, typeof(*engine), base); unsigned long flags; @@ -201,12 +198,13 @@ static void mock_submit_request(struct i915_request *request) GEM_BUG_ON(!request->global_seqno); spin_lock_irqsave(&engine->hw_lock, flags); - list_add_tail(&mock->link, &engine->hw_queue); - if (mock->link.prev == &engine->hw_queue) { - if (mock->delay) - mod_timer(&engine->hw_delay, jiffies + mock->delay); + list_add_tail(&request->mock.link, &engine->hw_queue); + if (list_is_first(&request->mock.link, &engine->hw_queue)) { + if (request->mock.delay) + mod_timer(&engine->hw_delay, + jiffies + request->mock.delay); else - advance(mock); + advance(request); } spin_unlock_irqrestore(&engine->hw_lock, flags); } @@ -266,12 +264,12 @@ void mock_engine_flush(struct intel_engine_cs *engine) { struct mock_engine *mock = container_of(engine, typeof(*mock), base); - struct mock_request *request, *rn; + struct i915_request *request, *rn; del_timer_sync(&mock->hw_delay); spin_lock_irq(&mock->hw_lock); - list_for_each_entry_safe(request, rn, &mock->hw_queue, link) + list_for_each_entry_safe(request, rn, &mock->hw_queue, mock.link) advance(request); spin_unlock_irq(&mock->hw_lock); } diff --git a/drivers/gpu/drm/i915/selftests/mock_gem_device.c b/drivers/gpu/drm/i915/selftests/mock_gem_device.c index 074a0d9cbf26..17915a2d94fa 100644 --- a/drivers/gpu/drm/i915/selftests/mock_gem_device.c +++ b/drivers/gpu/drm/i915/selftests/mock_gem_device.c @@ -79,9 +79,6 @@ static void mock_device_release(struct drm_device *dev) destroy_workqueue(i915->wq); - kmem_cache_destroy(i915->priorities); - kmem_cache_destroy(i915->dependencies); - kmem_cache_destroy(i915->requests); kmem_cache_destroy(i915->vmas); kmem_cache_destroy(i915->objects); @@ -211,23 +208,6 @@ struct drm_i915_private *mock_gem_device(void) if (!i915->vmas) goto err_objects; - i915->requests = KMEM_CACHE(mock_request, - SLAB_HWCACHE_ALIGN | - SLAB_RECLAIM_ACCOUNT | - SLAB_TYPESAFE_BY_RCU); - if (!i915->requests) - goto err_vmas; - - i915->dependencies = KMEM_CACHE(i915_dependency, - SLAB_HWCACHE_ALIGN | - SLAB_RECLAIM_ACCOUNT); - if (!i915->dependencies) - goto err_requests; - - i915->priorities = KMEM_CACHE(i915_priolist, SLAB_HWCACHE_ALIGN); - if (!i915->priorities) - goto err_dependencies; - i915_timelines_init(i915); INIT_LIST_HEAD(&i915->gt.active_rings); @@ -257,12 +237,6 @@ struct drm_i915_private *mock_gem_device(void) err_unlock: mutex_unlock(&i915->drm.struct_mutex); i915_timelines_fini(i915); - kmem_cache_destroy(i915->priorities); -err_dependencies: - kmem_cache_destroy(i915->dependencies); -err_requests: - kmem_cache_destroy(i915->requests); -err_vmas: kmem_cache_destroy(i915->vmas); err_objects: kmem_cache_destroy(i915->objects); diff --git a/drivers/gpu/drm/i915/selftests/mock_request.c b/drivers/gpu/drm/i915/selftests/mock_request.c index 0dc29e242597..d1a7c9608712 100644 --- a/drivers/gpu/drm/i915/selftests/mock_request.c +++ b/drivers/gpu/drm/i915/selftests/mock_request.c @@ -31,29 +31,25 @@ mock_request(struct intel_engine_cs *engine, unsigned long delay) { struct i915_request *request; - struct mock_request *mock; /* NB the i915->requests slab cache is enlarged to fit mock_request */ request = i915_request_alloc(engine, context); if (IS_ERR(request)) return NULL; - mock = container_of(request, typeof(*mock), base); - mock->delay = delay; - - return &mock->base; + request->mock.delay = delay; + return request; } bool mock_cancel_request(struct i915_request *request) { - struct mock_request *mock = container_of(request, typeof(*mock), base); struct mock_engine *engine = container_of(request->engine, typeof(*engine), base); bool was_queued; spin_lock_irq(&engine->hw_lock); - was_queued = !list_empty(&mock->link); - list_del_init(&mock->link); + was_queued = !list_empty(&request->mock.link); + list_del_init(&request->mock.link); spin_unlock_irq(&engine->hw_lock); if (was_queued) diff --git a/drivers/gpu/drm/i915/selftests/mock_request.h b/drivers/gpu/drm/i915/selftests/mock_request.h index 995fb728380c..4acf0211df20 100644 --- a/drivers/gpu/drm/i915/selftests/mock_request.h +++ b/drivers/gpu/drm/i915/selftests/mock_request.h @@ -29,13 +29,6 @@ #include "../i915_request.h" -struct mock_request { - struct i915_request base; - - struct list_head link; - unsigned long delay; -}; - struct i915_request * mock_request(struct intel_engine_cs *engine, struct i915_gem_context *context,
As kmem_caches share the same properties (size, allocation/free behaviour) for all potential devices, we can use global caches. While this potential has worse fragmentation behaviour (one can argue that different devices would have different activity lifetimes, but you can also argue that activity is temporal across the system) it is the default behaviour of the system at large to amalgamate matching caches. The benefit for us is much reduced pointer dancing along the frequent allocation paths. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> --- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/i915_active.c | 9 ++- drivers/gpu/drm/i915/i915_active.h | 1 + drivers/gpu/drm/i915/i915_drv.h | 3 - drivers/gpu/drm/i915/i915_gem.c | 32 +-------- drivers/gpu/drm/i915/i915_globals.c | 49 ++++++++++++++ drivers/gpu/drm/i915/i915_globals.h | 14 ++++ drivers/gpu/drm/i915/i915_pci.c | 8 ++- drivers/gpu/drm/i915/i915_request.c | 53 ++++++++++++--- drivers/gpu/drm/i915/i915_request.h | 10 +++ drivers/gpu/drm/i915/i915_scheduler.c | 66 +++++++++++++++---- drivers/gpu/drm/i915/i915_scheduler.h | 34 ++++++++-- drivers/gpu/drm/i915/intel_guc_submission.c | 3 +- drivers/gpu/drm/i915/intel_lrc.c | 6 +- drivers/gpu/drm/i915/intel_ringbuffer.h | 17 ----- drivers/gpu/drm/i915/selftests/intel_lrc.c | 2 +- drivers/gpu/drm/i915/selftests/mock_engine.c | 48 +++++++------- .../gpu/drm/i915/selftests/mock_gem_device.c | 26 -------- drivers/gpu/drm/i915/selftests/mock_request.c | 12 ++-- drivers/gpu/drm/i915/selftests/mock_request.h | 7 -- 20 files changed, 248 insertions(+), 153 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_globals.c create mode 100644 drivers/gpu/drm/i915/i915_globals.h