From patchwork Thu Feb 14 14:57:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Auld X-Patchwork-Id: 10812903 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F72313B5 for ; Thu, 14 Feb 2019 14:57:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E0262E900 for ; Thu, 14 Feb 2019 14:57:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 525632E907; Thu, 14 Feb 2019 14:57:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 96FC02E900 for ; Thu, 14 Feb 2019 14:57:53 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3AF206E90E; Thu, 14 Feb 2019 14:57:52 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4A5376E90E for ; Thu, 14 Feb 2019 14:57:49 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Feb 2019 06:57:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,369,1544515200"; d="scan'208";a="114920430" Received: from jhbrinke-mobl3.amr.corp.intel.com (HELO mwahaha.ger.corp.intel.com) ([10.252.11.151]) by orsmga007.jf.intel.com with ESMTP; 14 Feb 2019 06:57:48 -0800 From: Matthew Auld To: intel-gfx@lists.freedesktop.org Date: Thu, 14 Feb 2019 14:57:03 +0000 Message-Id: <20190214145740.14521-6-matthew.auld@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190214145740.14521-1-matthew.auld@intel.com> References: <20190214145740.14521-1-matthew.auld@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [RFC PATCH 05/42] drm/i915/region: support basic eviction X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP Support basic eviction for regions. Signed-off-by: Matthew Auld Cc: Joonas Lahtinen Cc: Abdiel Janulgue --- drivers/gpu/drm/i915/i915_drv.h | 2 + drivers/gpu/drm/i915/i915_gem.c | 16 ++++ drivers/gpu/drm/i915/i915_gem_object.h | 7 ++ drivers/gpu/drm/i915/i915_gem_shrinker.c | 59 ++++++++++++++ drivers/gpu/drm/i915/intel_memory_region.c | 40 +++++++++- drivers/gpu/drm/i915/intel_memory_region.h | 7 ++ .../drm/i915/selftests/intel_memory_region.c | 76 +++++++++++++++++++ drivers/gpu/drm/i915/selftests/mock_region.c | 1 + 8 files changed, 204 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h index 0bea7d889284..3df27769b978 100644 --- a/drivers/gpu/drm/i915/i915_drv.h +++ b/drivers/gpu/drm/i915/i915_drv.h @@ -3196,6 +3196,8 @@ void i915_gem_shrinker_register(struct drm_i915_private *i915); void i915_gem_shrinker_unregister(struct drm_i915_private *i915); void i915_gem_shrinker_taints_mutex(struct drm_i915_private *i915, struct mutex *mutex); +int i915_gem_shrink_memory_region(struct intel_memory_region *mem, + resource_size_t target); /* i915_gem_tiling.c */ static inline bool i915_gem_object_needs_bit17_swizzle(struct drm_i915_gem_object *obj) diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c index 92768ab294a4..7f044b643a75 100644 --- a/drivers/gpu/drm/i915/i915_gem.c +++ b/drivers/gpu/drm/i915/i915_gem.c @@ -4095,6 +4095,22 @@ i915_gem_madvise_ioctl(struct drm_device *dev, void *data, !i915_gem_object_has_pages(obj)) i915_gem_object_truncate(obj); + if (obj->memory_region) { + mutex_lock(&obj->memory_region->obj_lock); + + switch (obj->mm.madv) { + case I915_MADV_WILLNEED: + list_move(&obj->region_link, &obj->memory_region->objects); + break; + default: + list_move(&obj->region_link, + &obj->memory_region->purgeable); + break; + } + + mutex_unlock(&obj->memory_region->obj_lock); + } + args->retained = obj->mm.madv != __I915_MADV_PURGED; mutex_unlock(&obj->mm.lock); diff --git a/drivers/gpu/drm/i915/i915_gem_object.h b/drivers/gpu/drm/i915/i915_gem_object.h index ac52f61e8ad1..76947a6f49f1 100644 --- a/drivers/gpu/drm/i915/i915_gem_object.h +++ b/drivers/gpu/drm/i915/i915_gem_object.h @@ -95,6 +95,13 @@ struct drm_i915_gem_object { * List of memory region blocks allocated for this object. */ struct list_head blocks; + /** + * Element within memory_region->objects or memory_region->purgeable if + * the object is marked as DONTNEED. Access is protected by + * memory_region->obj_lock. + */ + struct list_head region_link; + struct list_head tmp_link; struct { /** diff --git a/drivers/gpu/drm/i915/i915_gem_shrinker.c b/drivers/gpu/drm/i915/i915_gem_shrinker.c index 6da795c7e62e..713c6c93cf30 100644 --- a/drivers/gpu/drm/i915/i915_gem_shrinker.c +++ b/drivers/gpu/drm/i915/i915_gem_shrinker.c @@ -308,6 +308,65 @@ unsigned long i915_gem_shrink_all(struct drm_i915_private *i915) return freed; } +int i915_gem_shrink_memory_region(struct intel_memory_region *mem, + resource_size_t target) +{ + struct drm_i915_private *i915 = mem->i915; + struct drm_i915_gem_object *obj, *on; + resource_size_t found; + LIST_HEAD(purgeable); + bool unlock; + int err; + + if (!shrinker_lock(i915, 0, &unlock)) + return 0; + + i915_retire_requests(i915); + + err = 0; + found = 0; + + mutex_lock(&mem->obj_lock); + + list_for_each_entry(obj, &mem->purgeable, region_link) { + if (!i915_gem_object_has_pages(obj)) + continue; + + if (READ_ONCE(obj->pin_global)) + continue; + + if (atomic_read(&obj->mm.pages_pin_count) > obj->bind_count) + continue; + + list_add(&obj->tmp_link, &purgeable); + + found += obj->base.size; + if (found >= target) + goto found; + } + + err = -ENOSPC; +found: + mutex_unlock(&mem->obj_lock); + + list_for_each_entry_safe(obj, on, &purgeable, tmp_link) { + if (!err) + err = i915_gem_object_unbind(obj); + if (!err) { + __i915_gem_object_put_pages(obj, + I915_MM_SHRINKER); + if (!i915_gem_object_has_pages(obj)) + obj->mm.madv = __I915_MADV_PURGED; + } + + list_del(&obj->tmp_link); + } + + shrinker_unlock(i915, unlock); + + return err; +} + static unsigned long i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) { diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c index 405d6d51194f..f7fdc3e942e6 100644 --- a/drivers/gpu/drm/i915/intel_memory_region.c +++ b/drivers/gpu/drm/i915/intel_memory_region.c @@ -89,7 +89,8 @@ i915_memory_region_get_pages_buddy(struct drm_i915_gem_object *obj) unsigned int order; u64 block_size; u64 offset; - + bool retry = false; +retry: order = fls(n_pages) - 1; GEM_BUG_ON(order > mem->mm.max_order); @@ -98,9 +99,25 @@ i915_memory_region_get_pages_buddy(struct drm_i915_gem_object *obj) if (!IS_ERR(block)) break; - /* XXX: some kind of eviction pass, local to the device */ - if (!order--) - goto err_free_blocks; + if (!order--) { + resource_size_t target; + int err; + + if (retry) + goto err_free_blocks; + + target = n_pages * mem->mm.min_size; + + mutex_unlock(&mem->mm_lock); + err = i915_gem_shrink_memory_region(mem, + target); + mutex_lock(&mem->mm_lock); + if (err) + goto err_free_blocks; + + retry = true; + goto retry; + } } while (1); n_pages -= 1 << order; @@ -151,6 +168,13 @@ void i915_memory_region_release_buddy(struct intel_memory_region *mem) i915_gem_buddy_fini(&mem->mm); } +void i915_gem_object_release_memory_region(struct drm_i915_gem_object *obj) +{ + mutex_lock(&obj->memory_region->obj_lock); + list_del(&obj->region_link); + mutex_unlock(&obj->memory_region->obj_lock); +} + struct drm_i915_gem_object * i915_gem_object_create_region(struct intel_memory_region *mem, resource_size_t size, @@ -179,6 +203,10 @@ i915_gem_object_create_region(struct intel_memory_region *mem, INIT_LIST_HEAD(&obj->blocks); obj->memory_region = mem; + mutex_lock(&mem->obj_lock); + list_add(&obj->region_link, &mem->objects); + mutex_unlock(&mem->obj_lock); + i915_gem_object_set_cache_coherency(obj, obj->cache_level); return obj; @@ -205,6 +233,10 @@ intel_memory_region_create(struct drm_i915_private *i915, mem->min_page_size = min_page_size; mem->ops = ops; + mutex_init(&mem->obj_lock); + INIT_LIST_HEAD(&mem->objects); + INIT_LIST_HEAD(&mem->purgeable); + mutex_init(&mem->mm_lock); if (ops->init) { diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h index 6d8a954ca75e..b1546afb0b6e 100644 --- a/drivers/gpu/drm/i915/intel_memory_region.h +++ b/drivers/gpu/drm/i915/intel_memory_region.h @@ -99,6 +99,11 @@ struct intel_memory_region { unsigned int type; unsigned int instance; unsigned int id; + + /* Protects access to objects and purgeable */ + struct mutex obj_lock; + struct list_head objects; + struct list_head purgeable; }; int i915_memory_region_init_buddy(struct intel_memory_region *mem); @@ -108,6 +113,8 @@ int i915_memory_region_get_pages_buddy(struct drm_i915_gem_object *obj); void i915_memory_region_put_pages_buddy(struct drm_i915_gem_object *obj, struct sg_table *pages); +void i915_gem_object_release_memory_region(struct drm_i915_gem_object *obj); + struct intel_memory_region * intel_memory_region_create(struct drm_i915_private *i915, resource_size_t start, diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c index 2b8d28216d87..1cea381d2d5e 100644 --- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c +++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c @@ -93,10 +93,86 @@ static int igt_mock_fill(void *arg) return err; } +static void igt_mark_evictable(struct drm_i915_gem_object *obj) +{ + i915_gem_object_unpin_pages(obj); + obj->mm.madv = I915_MADV_DONTNEED; + list_move(&obj->region_link, &obj->memory_region->purgeable); +} + +static int igt_mock_shrink(void *arg) +{ + struct intel_memory_region *mem = arg; + struct drm_i915_gem_object *obj; + unsigned long n_objects; + LIST_HEAD(objects); + resource_size_t target; + resource_size_t total; + int err = 0; + + target = mem->mm.min_size; + total = resource_size(&mem->region); + n_objects = total / target; + + while (n_objects--) { + obj = i915_gem_object_create_region(mem, + target, + 0); + if (IS_ERR(obj)) { + err = PTR_ERR(obj); + goto err_close_objects; + } + + list_add(&obj->st_link, &objects); + + err = i915_gem_object_pin_pages(obj); + if (err) + goto err_close_objects; + + /* + * Make half of the region evictable, though do so in a + * horribly fragmented fashion. + */ + if (n_objects % 2) + igt_mark_evictable(obj); + } + + while (target <= total / 2) { + obj = i915_gem_object_create_region(mem, target, 0); + if (IS_ERR(obj)) { + err = PTR_ERR(obj); + goto err_close_objects; + } + + list_add(&obj->st_link, &objects); + + /* Provoke the shrinker to start violently swinging its axe! */ + err = i915_gem_object_pin_pages(obj); + if (err) { + pr_err("failed to shrink for target=%pa", &target); + goto err_close_objects; + } + + /* Again, half of the region should remain evictable */ + igt_mark_evictable(obj); + + target <<= 1; + } + +err_close_objects: + close_objects(&objects); + + if (err == -ENOMEM) + err = 0; + + return err; +} + int intel_memory_region_mock_selftests(void) { static const struct i915_subtest tests[] = { SUBTEST(igt_mock_fill), + SUBTEST(igt_mock_shrink), }; struct intel_memory_region *mem; struct drm_i915_private *i915; diff --git a/drivers/gpu/drm/i915/selftests/mock_region.c b/drivers/gpu/drm/i915/selftests/mock_region.c index 2c83711f780d..11e9f379aaca 100644 --- a/drivers/gpu/drm/i915/selftests/mock_region.c +++ b/drivers/gpu/drm/i915/selftests/mock_region.c @@ -27,6 +27,7 @@ static const struct drm_i915_gem_object_ops mock_region_obj_ops = { .get_pages = i915_memory_region_get_pages_buddy, .put_pages = i915_memory_region_put_pages_buddy, + .release = i915_gem_object_release_memory_region, }; static struct drm_i915_gem_object *