From patchwork Fri Nov 8 20:49:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11235525 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E189C1599 for ; Fri, 8 Nov 2019 20:49:57 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CAFEC214DB for ; Fri, 8 Nov 2019 20:49:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CAFEC214DB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4D37D6FA77; Fri, 8 Nov 2019 20:49:56 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id C34E06FA73; Fri, 8 Nov 2019 20:49:52 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from haswell.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 19134342-1500050 for multiple; Fri, 08 Nov 2019 20:49:36 +0000 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 8 Nov 2019 20:49:32 +0000 Message-Id: <20191108204932.6197-3-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191108204932.6197-1-chris@chris-wilson.co.uk> References: <20191108204932.6197-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH i-g-t 3/3] i915/gem_exec_scheduler: Exercise priority inversion from resource contention X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: igt-dev@lists.freedesktop.org Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" One of the hardest priority inversion tasks to both handle and to simulate in testing is inversion due to resource contention. The challenge is that a high priority context should never be blocked by a low priority context, even if both are starving for resources -- ideally, at least for some RT OSes, the higher priority context has first pick of the meagre resources so that it can be executed with minimum latency. userfaultfd allows us to handle a page fault in userspace, and so arbitrary impose a delay on the fault handler, creating a situation where a low priority context is blocked waiting for the fault. This blocked context should not prevent a high priority context from being executed. While the userfault tries to emulate a slow fault (e.g. from a failing swap device), it is unfortunately limited to a single object type: the userptr. Hopefully, we will find other ways to impose other starvation conditions on global resources. Signed-off-by: Chris Wilson Cc: Joonas Lahtinen Cc: Tvrtko Ursulin --- tests/i915/gem_exec_schedule.c | 155 +++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) diff --git a/tests/i915/gem_exec_schedule.c b/tests/i915/gem_exec_schedule.c index 84581bffe..0af7b4c6d 100644 --- a/tests/i915/gem_exec_schedule.c +++ b/tests/i915/gem_exec_schedule.c @@ -23,10 +23,16 @@ #include "config.h" +#include + +#include #include #include +#include +#include #include #include +#include #include "igt.h" #include "igt_rand.h" @@ -1625,6 +1631,152 @@ static void test_pi_ringfull(int fd, unsigned int engine, unsigned int flags) munmap(result, 4096); } +static int userfaultfd(int flags) +{ + return syscall(SYS_userfaultfd, flags); +} + +struct ufd_thread { + pthread_t thread; + uint32_t batch; + uint32_t *page; + unsigned int engine; + int i915; +}; + +static uint32_t create_userptr(int i915, void *page) +{ + uint32_t handle; + + gem_userptr(i915, page, 4096, 0, 0, &handle); + return handle; +} + +static void *ufd_thread(void *arg) +{ + struct ufd_thread *t = arg; + struct drm_i915_gem_exec_object2 obj[2] = { + { .handle = create_userptr(t->i915, t->page) }, + { .handle = t->batch }, + }; + struct drm_i915_gem_execbuffer2 eb = { + .buffers_ptr = to_user_pointer(obj), + .buffer_count = ARRAY_SIZE(obj), + .flags = t->engine, + .rsvd1 = gem_context_create(t->i915), + }; + gem_context_set_priority(t->i915, eb.rsvd1, MIN_PRIO); + + igt_debug("submitting fault\n"); + gem_execbuf(t->i915, &eb); + gem_sync(t->i915, obj[0].handle); + gem_close(t->i915, obj[0].handle); + + gem_context_destroy(t->i915, eb.rsvd1); + + t->i915 = -1; + return NULL; +} + +static void test_pi_userfault(int i915, unsigned int engine) +{ + struct uffdio_api api = { .api = UFFD_API }; + struct uffdio_register reg; + struct uffdio_copy copy; + struct uffd_msg msg; + struct ufd_thread t; + char poison[4096]; + int ufd; + + /* + * Resource contention can easily lead to priority inversion problems, + * that we wish to avoid. Here, we simulate one simple form of resource + * starvation by using an arbitrary slow userspace fault handler to cause + * the low priority context to block waiting for its resource. While it + * is blocked, it should not prevent a higher priority context from + * executing. + * + * This is only a very simple scenario, in more general tests we will + * need to simulate contention on the shared resource such that both + * low and high priority contexts are starving and must fight over + * the meagre resources. One step at a time. + */ + + ufd = userfaultfd(0); + igt_require_f(ufd != -1, "kernel support for userfaultfd\n"); + igt_require_f(ioctl(ufd, UFFDIO_API, &api) == 0 && api.api == UFFD_API, + "userfaultfd API v%lld:%lld\n", UFFD_API, api.api); + + t.i915 = i915; + t.engine = engine; + + t.page = mmap(NULL, 4096, PROT_WRITE, MAP_SHARED | MAP_ANON, 0, 0); + igt_assert(t.page != MAP_FAILED); + + t.batch = gem_create(i915, 4096); + memset(poison, 0xff, sizeof(poison)); + gem_write(i915, t.batch, 0, poison, 4096); + + /* Register our fault handler for t.page */ + memset(®, 0, sizeof(reg)); + reg.mode = UFFDIO_REGISTER_MODE_MISSING; + reg.range.start = to_user_pointer(t.page); + reg.range.len = 4096; + do_ioctl(ufd, UFFDIO_REGISTER, ®); + igt_assert(reg.ioctls == UFFD_API_RANGE_IOCTLS); + + /* Kick off the low priority submission */ + igt_assert(pthread_create(&t.thread, NULL, ufd_thread, &t) == 0); + + /* Wait until the low priority thread is blocked on a fault */ + igt_assert_eq(read(ufd, &msg, sizeof(msg)), sizeof(msg)); + igt_assert_eq(msg.event, UFFD_EVENT_PAGEFAULT); + igt_assert(from_user_pointer(msg.arg.pagefault.address) == t.page); + + /* While the low priority context is blocked; execute a vip */ + if (1) { + const uint32_t bbe = MI_BATCH_BUFFER_END; + struct drm_i915_gem_exec_object2 obj = { + .handle = t.batch, + }; + struct pollfd pfd; + struct drm_i915_gem_execbuffer2 eb = { + .buffers_ptr = to_user_pointer(&obj), + .buffer_count = 1, + .flags = engine | I915_EXEC_FENCE_OUT, + .rsvd1 = gem_context_create(i915), + }; + gem_context_set_priority(i915, eb.rsvd1, MAX_PRIO); + gem_write(i915, obj.handle, 0, &bbe, sizeof(bbe)); + gem_execbuf_wr(i915, &eb); + + memset(&pfd, 0, sizeof(pfd)); + pfd.fd = eb.rsvd2 >> 32; + pfd.events = POLLIN; + poll(&pfd, 1, -1); + igt_assert_eq(sync_fence_status(pfd.fd), 1); + close(pfd.fd); + + gem_context_destroy(i915, eb.rsvd1); + } + + /* Confirm the low priority context is still waiting */ + igt_assert_eq(t.i915, i915); + + /* Service the fault; releasing the low priority context */ + memset(©, 0, sizeof(copy)); + copy.dst = msg.arg.pagefault.address; + copy.src = to_user_pointer(memset(poison, 0xc5, sizeof(poison))); + copy.len = 4096; + do_ioctl(ufd, UFFDIO_COPY, ©); + + pthread_join(t.thread, NULL); + + gem_close(i915, t.batch); + munmap(t.page, 4096); + close(ufd); +} + static void measure_semaphore_power(int i915) { struct rapl gpu, pkg; @@ -1864,6 +2016,9 @@ igt_main igt_subtest_f("pi-common-%s", e->name) test_pi_ringfull(fd, eb_ring(e), SHARED); + + igt_subtest_f("pi-userfault-%s", e->name) + test_pi_userfault(fd, eb_ring(e)); } } }