From patchwork Fri Sep 29 12:39:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tvrtko Ursulin X-Patchwork-Id: 9977963 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 098DD6034B for ; Fri, 29 Sep 2017 12:40:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F34052980D for ; Fri, 29 Sep 2017 12:40:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E6EF429858; Fri, 29 Sep 2017 12:40:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3D4E42980D for ; Fri, 29 Sep 2017 12:40:03 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id CC0206EB77; Fri, 29 Sep 2017 12:40:02 +0000 (UTC) X-Original-To: Intel-gfx@lists.freedesktop.org Delivered-To: Intel-gfx@lists.freedesktop.org Received: from mail-wr0-x242.google.com (mail-wr0-x242.google.com [IPv6:2a00:1450:400c:c0c::242]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7DDDA6EB41 for ; Fri, 29 Sep 2017 12:39:53 +0000 (UTC) Received: by mail-wr0-x242.google.com with SMTP id z1so1212551wre.1 for ; Fri, 29 Sep 2017 05:39:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ursulin-net.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UGwdSYlORDN/vUL91n2lCthkZI4IB3NfK/KgTsWFp4Q=; b=EiDS5GFPSvZYINHEE7WC06WtYA+dW4tCrWYjqdYSgRLp5ThaDb7CyRnvjfG8wXMJ3c kry9AIcXoPTiAZo1LHObpYVfxtDtnbxygBvFi5UjohKQVFYDogrEL7aFAHVcz/rAb6M2 KjscTJB+BgRxwfp6BOPDET2ZVLHxkD2LVC40YL33HmTRwzScVDCMTL70pBbSwl/O53jw mglKFAKw4bU3cpg1nys3c2xKhvgojzjg7QXS1ODBXtYZ9v0BeGK5FBhaDELsoqyHDNxz HpUdOD3PmsM+CBWY7jWDSU555yMV4EWdiucXtGhgw1tb8j8UEZGT3mpJ6TDS588l4eD/ LwIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UGwdSYlORDN/vUL91n2lCthkZI4IB3NfK/KgTsWFp4Q=; b=nyZxtRFu3EtTHb9jnWNB5D3CSS13mci36+Fkd60T+9yMgSUO8jiYl0VEW2LwNAoZ6J CreCHHFc6vGuLr8WAADHKEJWCrWHGV383hY7A9STHlFwavVPIV3T5QUPU3Xr3i2MZ4eY ViCuoDiMcGd6fovDBPFjw+QWdbcNeTbjeDGY9CtoFFBfrk6HCXo0q1R7gfPrwNxswgHI dnorwvn31dkOy08+8oSMU2o6+jQv2jGuC2/jzoo0b7OFaRmFi+Wf9bjV+X5XQJ7wybMq jnrD6VK+VGzwA6Rqo4eLOHu3vgb6DQwZtmrkr5/4hOm2svOK2jRy8DO/CZe9TdSUYVk5 Js0Q== X-Gm-Message-State: AHPjjUh1zuhWFnliWi6v7FFa5ycGIAWFJFUCkvieMKAMaN+i5RYhwNNj 7sdnNx2SSSbBGQ9oZDnK/pO23ps2 X-Google-Smtp-Source: AOwi7QBqVk7DdIyCvL87Id2IPw6BMDNueEBygVRPb3B7OPhhgQuiNYd9xqWHIbtKaXRosxE5+m7Gsg== X-Received: by 10.223.157.198 with SMTP id q6mr7436273wre.102.1506688791912; Fri, 29 Sep 2017 05:39:51 -0700 (PDT) Received: from t460p.intel ([95.146.151.158]) by smtp.gmail.com with ESMTPSA id w5sm1959691wrg.65.2017.09.29.05.39.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Sep 2017 05:39:51 -0700 (PDT) From: Tvrtko Ursulin X-Google-Original-From: Tvrtko Ursulin To: Intel-gfx@lists.freedesktop.org Date: Fri, 29 Sep 2017 13:39:38 +0100 Message-Id: <20170929123939.3312-7-tvrtko.ursulin@linux.intel.com> X-Mailer: git-send-email 2.9.5 In-Reply-To: <20170929123939.3312-1-tvrtko.ursulin@linux.intel.com> References: <20170929123939.3312-1-tvrtko.ursulin@linux.intel.com> Subject: [Intel-gfx] [PATCH i-g-t 6/7] gem_wsim: Busy stats balancers X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP From: Tvrtko Ursulin Add busy and busy-avg balancers which make balancing decisions by looking at engine busyness via the i915 PMU. And thus are able to make decisions on the actual instantaneous load of the system, and not use metrics that lag behind by a batch or two. In doing so, each client should be able to greedily maximise their own usage of the system, leading to improved load balancing even in the face of other uncooperative clients. On the other hand, we are only using the instantaneous load without coupling in the predictive factor for dispatch and execution length. v2: * Commit text. (Chris Wilson) * Rename get_stats to get_pmu_stats. (Chris Wilson) * Fix PMU readout in VCS remap mode. Signed-off-by: Tvrtko Ursulin Reviewed-by: Chris Wilson --- benchmarks/Makefile.am | 2 +- benchmarks/gem_wsim.c | 142 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 143 insertions(+), 1 deletion(-) diff --git a/benchmarks/Makefile.am b/benchmarks/Makefile.am index 5c2cd19d6b20..2c15468d3be4 100644 --- a/benchmarks/Makefile.am +++ b/benchmarks/Makefile.am @@ -19,6 +19,6 @@ gem_latency_CFLAGS = $(AM_CFLAGS) $(THREAD_CFLAGS) gem_latency_LDADD = $(LDADD) -lpthread gem_syslatency_CFLAGS = $(AM_CFLAGS) $(THREAD_CFLAGS) gem_syslatency_LDADD = $(LDADD) -lpthread -lrt -gem_wsim_LDADD = $(LDADD) -lpthread +gem_wsim_LDADD = $(LDADD) $(top_builddir)/lib/libigt_perf.la -lpthread EXTRA_DIST=README diff --git a/benchmarks/gem_wsim.c b/benchmarks/gem_wsim.c index 82fe6ba9ec5f..8b2cd90659a9 100644 --- a/benchmarks/gem_wsim.c +++ b/benchmarks/gem_wsim.c @@ -50,6 +50,7 @@ #include "intel_io.h" #include "igt_aux.h" #include "igt_rand.h" +#include "igt_perf.h" #include "sw_sync.h" #include "ewma.h" @@ -188,6 +189,16 @@ struct workload uint32_t last[NUM_ENGINES]; } rt; }; + + struct busy_balancer { + int fd; + bool first; + unsigned int num_engines; + unsigned int engine_map[5]; + uint64_t t_prev; + uint64_t prev[5]; + double busy[5]; + } busy_balancer; }; static const unsigned int nop_calibration_us = 1000; @@ -993,6 +1004,8 @@ struct workload_balancer { unsigned int flags; unsigned int min_gen; + int (*init)(const struct workload_balancer *balancer, + struct workload *wrk); unsigned int (*get_qd)(const struct workload_balancer *balancer, struct workload *wrk, enum intel_engine_id engine); @@ -1242,6 +1255,108 @@ context_balance(const struct workload_balancer *balancer, return get_vcs_engine(wrk->ctx_list[w->context].static_vcs); } +static unsigned int +get_engine_busy(const struct workload_balancer *balancer, + struct workload *wrk, enum intel_engine_id engine) +{ + struct busy_balancer *bb = &wrk->busy_balancer; + + if (engine == VCS2 && (wrk->flags & VCS2REMAP)) + engine = BCS; + + return bb->busy[bb->engine_map[engine]]; +} + +static void +get_pmu_stats(const struct workload_balancer *b, struct workload *wrk) +{ + struct busy_balancer *bb = &wrk->busy_balancer; + uint64_t val[7]; + unsigned int i; + + igt_assert_eq(read(bb->fd, val, sizeof(val)), + (2 + bb->num_engines) * sizeof(uint64_t)); + + if (!bb->first) { + for (i = 0; i < bb->num_engines; i++) { + double d; + + d = (val[2 + i] - bb->prev[i]) * 100; + d /= val[1] - bb->t_prev; + bb->busy[i] = d; + } + } + + for (i = 0; i < bb->num_engines; i++) + bb->prev[i] = val[2 + i]; + + bb->t_prev = val[1]; + bb->first = false; +} + +static enum intel_engine_id +busy_avg_balance(const struct workload_balancer *balancer, + struct workload *wrk, struct w_step *w) +{ + get_pmu_stats(balancer, wrk); + + return qdavg_balance(balancer, wrk, w); +} + +static enum intel_engine_id +busy_balance(const struct workload_balancer *balancer, + struct workload *wrk, struct w_step *w) +{ + get_pmu_stats(balancer, wrk); + + return qd_balance(balancer, wrk, w); +} + +static int +busy_init(const struct workload_balancer *balancer, struct workload *wrk) +{ + struct busy_balancer *bb = &wrk->busy_balancer; + struct engine_desc { + unsigned class, inst; + enum intel_engine_id id; + } *d, engines[] = { + { I915_ENGINE_CLASS_RENDER, 0, RCS }, + { I915_ENGINE_CLASS_COPY, 0, BCS }, + { I915_ENGINE_CLASS_VIDEO, 0, VCS1 }, + { I915_ENGINE_CLASS_VIDEO, 1, VCS2 }, + { I915_ENGINE_CLASS_VIDEO_ENHANCE, 0, VECS }, + { 0, 0, VCS } + }; + + bb->num_engines = 0; + bb->first = true; + bb->fd = -1; + + for (d = &engines[0]; d->id != VCS; d++) { + int pfd; + + pfd = perf_i915_open_group(I915_PMU_ENGINE_BUSY(d->class, + d->inst), + bb->fd); + if (pfd < 0) { + if (d->id != VCS2) + return -(10 + bb->num_engines); + else + continue; + } + + if (bb->num_engines == 0) + bb->fd = pfd; + + bb->engine_map[d->id] = bb->num_engines++; + } + + if (bb->num_engines < 5 && !(wrk->flags & VCS2REMAP)) + return -1; + + return 0; +} + static const struct workload_balancer all_balancers[] = { { .id = 0, @@ -1315,6 +1430,22 @@ static const struct workload_balancer all_balancers[] = { .desc = "Static round-robin VCS assignment at context creation.", .balance = context_balance, }, + { + .id = 9, + .name = "busy", + .desc = "Engine busyness based balancing.", + .init = busy_init, + .get_qd = get_engine_busy, + .balance = busy_balance, + }, + { + .id = 10, + .name = "busy-avg", + .desc = "Average engine busyness based balancing.", + .init = busy_init, + .get_qd = get_engine_busy, + .balance = busy_avg_balance, + }, }; static unsigned int @@ -2226,6 +2357,17 @@ int main(int argc, char **argv) (verbose > 0 && master_workload == i); prepare_workload(i, w[i], flags_); + + if (balancer && balancer->init) { + int ret = balancer->init(balancer, w[i]); + if (ret) { + if (verbose) + fprintf(stderr, + "Failed to initialize balancing! (%u=%d)\n", + i, ret); + return 1; + } + } } gem_quiescent_gpu(fd);