From patchwork Fri Sep 7 11:14:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tvrtko Ursulin X-Patchwork-Id: 10592079 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC9DE6CB for ; Fri, 7 Sep 2018 11:14:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D1F2F2ADAE for ; Fri, 7 Sep 2018 11:14:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C5E932AEC9; Fri, 7 Sep 2018 11:14:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C6DFC2AEA0 for ; Fri, 7 Sep 2018 11:14:30 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3AEDB6E8C4; Fri, 7 Sep 2018 11:14:29 +0000 (UTC) X-Original-To: Intel-gfx@lists.freedesktop.org Delivered-To: Intel-gfx@lists.freedesktop.org Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4DE486E8C4 for ; Fri, 7 Sep 2018 11:14:27 +0000 (UTC) Received: by mail-wr1-x441.google.com with SMTP id z96-v6so14560725wrb.8 for ; Fri, 07 Sep 2018 04:14:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=BB7pf/peE7KcYK5eH1wLCv45gyBRRKKEmx4E4KtGZSc=; b=So6cyHrSa1qJvhi4m/GNAoHGE6s82Ud5YrPqFJisBuzPkquGzfePzo2p77KYFH17t8 KWWJWP6sXdUATQI1rAGwYKB/oFJrbT0ijefSBaPwy8tMeYni25d4et5KlWTzI3aDbGfF rjDzr33RbRVZVrFYeXlYkJy6HTFeVgPffwBn70VyNfUhEvju614V1ju1QWKXMo/pXT69 jcb6ult6ohMx9sV9Gq5W6FdKB+QjpWrdPVRRqN+x1IAfQPs+9AVV5rYVPys1TEaO3ZaH mt9qqaovDD4TmdoLCD47OokoXuFg9oiO0rEyFOmKV14mZo5DFDfVwONkJvkBOh3SPO2J OPMQ== X-Gm-Message-State: APzg51BV3Ms1z8dOaocPsfmkIXBHutOj9Y5UiJQ41AWoVmipfEbA6fRe XgfX2f9ePJAqB+G4u2qoUMhns7mzQp4= X-Google-Smtp-Source: ANB0Vdb5oeId9z6Fn3TN+6qPtZxTLtsLxG+jKVa0jQ0SU8qwTtJ+O3OjKkj41AgrerfLXOqNhWDIfg== X-Received: by 2002:a5d:438d:: with SMTP id i13-v6mr6038820wrq.156.1536318865740; Fri, 07 Sep 2018 04:14:25 -0700 (PDT) Received: from localhost.localdomain ([95.144.165.37]) by smtp.gmail.com with ESMTPSA id 144-v6sm10810115wma.19.2018.09.07.04.14.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 07 Sep 2018 04:14:25 -0700 (PDT) From: Tvrtko Ursulin X-Google-Original-From: Tvrtko Ursulin To: igt-dev@lists.freedesktop.org Date: Fri, 7 Sep 2018 12:14:20 +0100 Message-Id: <20180907111420.12970-1-tvrtko.ursulin@linux.intel.com> X-Mailer: git-send-email 2.17.1 Subject: [Intel-gfx] [PATH i-g-t] igt: Test tagging support X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Intel-gfx@lists.freedesktop.org MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP From: Tvrtko Ursulin Proposal to add test tags as a replacement for separate test list which can be difficult to maintain and get out of date. Putting this maintanenace inline with tests makes it easier to remember to update the (now implicit) lists, and also enables richer test selection possibilities for the test runner. Current method of implying tags from test/subtest names has a couple of problems one of which is where some names can use the same token for different meanings. (One example is the "default" token.) It also creates a name clash between naming and tagging. Test tags are strings of tokens separated by spaces, meaning of which we decide by creating a convetion and helped by the suitable helper macros. For example tags can be things like: gem, kms, basic, guc, flip, semaphore, bz12345678, gt4e, reset, etc.. At runtime this would look something like this (abbreviated for readability): @ tests/gem_sync --list-subtests-with-tags ... forked-each TAGS="gem " forked-store-each TAGS="gem " basic-all TAGS="gem basic " basic-store-all TAGS="gem basic " @ tests/gem_concurrent_blit --list-subtests-with-tags ... 16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress " 16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress " 16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress " @ tests/kms_flip --list-subtests-with-tags | grep basic basic-plain-flip TAGS="kms basic " basic-flip-vs-dpms TAGS="kms basic " Test runner could then enable usages like: ./run-tests --include gem --exclude stress Or on simulation for example: ./run-tests --exclude stress,reset,rc6,rps,vblank Which I think could really be useful and more flexible than name based selection. The simulation example could also replace the sprinkle of igt_skip_on_simulation which often lacks context, which tags could provide. This RFC implementation automatically adds various tags based on both binary and subtest names. This is to enable easy transition to the tag based system. Idea is that all current tests get some useful tags with minimum amount of work. Unfortunately this doesn't work for test binaries which at the moment do not support the subtests enumeration so those are the ones which would need identifying and converting. Source code usage, for tests which would want to override the default tag generation would look something like: igt_gem_subtest("basic", "each") ... igt_gem_subtest("manual", "special-setup-test") ... igt_gem_stress_subtest_f("tag", "name-%s", ...) { ... We can of course polish the details of the API if the idea will be deemed interesting. v2: More default tags for untagged tests for easier migration. Signed-off-by: Tvrtko Ursulin --- lib/igt_core.c | 115 +++++++++++++++++++++++++++++++++++-- lib/igt_core.h | 29 +++++++++- tests/gem_concurrent_all.c | 34 +++++------ 3 files changed, 152 insertions(+), 26 deletions(-) diff --git a/lib/igt_core.c b/lib/igt_core.c index 23bb858fd886..daae1a3e4b56 100644 --- a/lib/igt_core.c +++ b/lib/igt_core.c @@ -258,10 +258,12 @@ static unsigned int exit_handler_count; const char *igt_interactive_debug; /* subtests helpers */ -static bool list_subtests = false; -static char *run_single_subtest = NULL; +static bool list_subtests, list_subtests_tags; +static char *run_single_subtest; static bool run_single_subtest_found = false; -static const char *in_subtest = NULL; +static const char *in_subtest; +static char *subtest_tags; +static bool subtest_tags_allocated; static struct timespec subtest_time; static clockid_t igt_clock = (clockid_t)-1; static bool in_fixture = false; @@ -281,6 +283,7 @@ bool test_child; enum { OPT_LIST_SUBTESTS, + OPT_LIST_SUBTESTS_TAGS, OPT_RUN_SUBTEST, OPT_DESCRIPTION, OPT_DEBUG, @@ -549,6 +552,7 @@ static void print_usage(const char *help_str, bool output_on_stderr) fprintf(f, "Usage: %s [OPTIONS]\n", command_str); fprintf(f, " --list-subtests\n" + " --list-subtests-with-tags\n" " --run-subtest \n" " --debug[=log-domain]\n" " --interactive-debug[=domain]\n" @@ -659,6 +663,7 @@ static int common_init(int *argc, char **argv, int c, option_index = 0, i, x; static struct option long_options[] = { {"list-subtests", 0, 0, OPT_LIST_SUBTESTS}, + {"list-subtests-with-tags", 0, 0, OPT_LIST_SUBTESTS_TAGS}, {"run-subtest", 1, 0, OPT_RUN_SUBTEST}, {"help-description", 0, 0, OPT_DESCRIPTION}, {"debug", optional_argument, 0, OPT_DEBUG}, @@ -753,6 +758,12 @@ static int common_init(int *argc, char **argv, if (!run_single_subtest) list_subtests = true; break; + case OPT_LIST_SUBTESTS_TAGS: + if (!run_single_subtest) { + list_subtests = true; + list_subtests_tags = true; + } + break; case OPT_RUN_SUBTEST: if (!list_subtests) run_single_subtest = strdup(optarg); @@ -884,19 +895,100 @@ void igt_simple_init_parse_opts(int *argc, char **argv, extra_opt_handler, handler_data); } +#define __IGT_TAGS_MAX (4096) + +static char * __get_tag_str(char *tags) +{ + if (tags) + return tags; + + tags = malloc(__IGT_TAGS_MAX); + igt_assert(tags); + + memset(tags, 0, __IGT_TAGS_MAX); + + return tags; +} + +static char * __add_tag(char *tags, unsigned int *pos, const char *tag) +{ + size_t left = __IGT_TAGS_MAX - *pos; + char _tag[128]; + int ret; + + if (!tag) + return tags; + + tags = __get_tag_str(tags); + + sprintf(_tag, "%s ", tag); + if (!strncmp(tags, _tag, strlen(_tag))) + return tags; + + sprintf(_tag, " %s ", tag); + if (strstr(tags, _tag)) + return tags; + + ret = snprintf(tags + *pos, left, "%s ", tag); + igt_assert(ret > 0 && ret < left); + + *pos += ret; + + return tags; +} + /* * Note: Testcases which use these helpers MUST NOT output anything to stdout * outside of places protected by igt_run_subtest checks - the piglit * runner adds every line to the subtest list. */ -bool __igt_run_subtest(const char *subtest_name) +bool __igt_run_subtest(const char *subtest_name, const char *tags) { + unsigned int tag_idx = 0; + char *_tags = NULL; + const char *tag; int i; assert(!in_subtest); + assert(!subtest_tags); assert(!in_fixture); assert(test_with_subtests); + if (!tags) { + if (!strncmp(command_str, "gem_", 4)) + tag = "gem"; + else if (!strncmp(command_str, "prime_", 4)) + tag = "gem"; + else if (!strncmp(command_str, "kms_", 4)) + tag = "kms"; + else if (!strncmp(command_str, "pm_", 3)) + tag = "pm"; + else if (!strncmp(command_str, "drv_", 3)) + tag = "drv"; + else if (!strncmp(command_str, "debugfs_", 3)) + tag = "drv"; + else if (!strncmp(command_str, "core_", 4)) + tag = "drm"; + else if (!strncmp(command_str, "drm_", 4)) + tag = "drm"; + else if (!strncmp(command_str, "sw_", 4)) + tag = "external"; + else + tag = NULL; + + _tags = __add_tag(_tags, &tag_idx, tag); + + if (!strncmp(subtest_name, "basic-", 6)) + tag = "basic"; + else + tag = NULL; + + _tags = __add_tag(_tags, &tag_idx, tag); + + tags = _tags; + subtest_tags_allocated = _tags; + } + /* check the subtest name only contains a-z, A-Z, 0-9, '-' and '_' */ for (i = 0; subtest_name[i] != '\0'; i++) if (subtest_name[i] != '_' && subtest_name[i] != '-' @@ -907,7 +999,10 @@ bool __igt_run_subtest(const char *subtest_name) } if (list_subtests) { - printf("%s\n", subtest_name); + if (list_subtests_tags && tags) + printf("%s TAGS=\"%s\"\n", subtest_name, tags); + else + printf("%s\n", subtest_name); return false; } @@ -940,7 +1035,10 @@ bool __igt_run_subtest(const char *subtest_name) _igt_log_buffer_reset(); + subtest_tags = (char *)tags; + igt_gettime(&subtest_time); + return (in_subtest = subtest_name); } @@ -997,7 +1095,12 @@ static void exit_subtest(const char *result) igt_terminate_spin_batches(); - in_subtest = NULL; + if (subtest_tags_allocated) + free(subtest_tags); + + in_subtest = subtest_tags = NULL; + subtest_tags_allocated = false; + siglongjmp(igt_subtest_jmpbuf, 1); } diff --git a/lib/igt_core.h b/lib/igt_core.h index b80e1702a16f..300ff739d97b 100644 --- a/lib/igt_core.h +++ b/lib/igt_core.h @@ -145,7 +145,7 @@ int igt_subtest_init_parse_opts(int *argc, char **argv, #define igt_subtest_init(argc, argv) \ igt_subtest_init_parse_opts(&argc, argv, NULL, NULL, NULL, NULL, NULL); -bool __igt_run_subtest(const char *subtest_name); +bool __igt_run_subtest(const char *subtest_name, const char *subtest_tags); #define __igt_tokencat2(x, y) x ## y /** @@ -158,6 +158,29 @@ bool __igt_run_subtest(const char *subtest_name); */ #define igt_tokencat(x, y) __igt_tokencat2(x, y) +#define igt_tagged_subtest(tags, name) \ + for (; __igt_run_subtest((name), (tags)) && \ + (sigsetjmp(igt_subtest_jmpbuf, 1) == 0); \ + igt_success()) + +#define igt_gem_subtest(tags, name) igt_tagged_subtest("gem "tags, name) + +#define __igt_tagged_subtest_f(tags, tmp, format...) \ + for (char tmp [256]; \ + snprintf( tmp , sizeof( tmp ), format), \ + __igt_run_subtest(tmp, tags) && \ + (sigsetjmp(igt_subtest_jmpbuf, 1) == 0); \ + igt_success()) + +#define igt_tagged_subtest_f(tags, f...) \ + __igt_tagged_subtest_f(tags, igt_tokencat(__tmpchar, __LINE__), f) + +#define igt_gem_subtest_f(tags, f...) \ + igt_tagged_subtest_f("gem "tags, f) + +#define igt_gem_stress_subtest_f(tags, f...) \ + igt_tagged_subtest_f("gem stress "tags, f) + /** * igt_subtest: * @name: name of the subtest @@ -169,14 +192,14 @@ bool __igt_run_subtest(const char *subtest_name); * * This is a simpler version of igt_subtest_f() */ -#define igt_subtest(name) for (; __igt_run_subtest((name)) && \ +#define igt_subtest(name) for (; __igt_run_subtest((name), NULL) && \ (sigsetjmp(igt_subtest_jmpbuf, 1) == 0); \ igt_success()) #define __igt_subtest_f(tmp, format...) \ for (char tmp [256]; \ snprintf( tmp , sizeof( tmp ), \ format), \ - __igt_run_subtest( tmp ) && \ + __igt_run_subtest(tmp, NULL) && \ (sigsetjmp(igt_subtest_jmpbuf, 1) == 0); \ igt_success()) diff --git a/tests/gem_concurrent_all.c b/tests/gem_concurrent_all.c index 4ac08c1b1c90..c1d29421b8ba 100644 --- a/tests/gem_concurrent_all.c +++ b/tests/gem_concurrent_all.c @@ -1481,47 +1481,47 @@ run_mode(const char *prefix, igt_subtest_group { igt_fixture p->require(); - igt_subtest_f("%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_basic0, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-sanitycheck1%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheck1%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_basic1, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-sanitycheckN%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheckN%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_basicN, p->copy, h->hang); } /* try to overwrite the source values */ - igt_subtest_f("%s-%s-%s-overwrite-source-one%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source-one%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_overwrite_source__one, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-overwrite-source%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_overwrite_source, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-overwrite-source-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_overwrite_source_read_bcs, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-overwrite-source-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { igt_require(rendercopy); buffers_create(&buffers); run_wrap_func(&buffers, @@ -1529,7 +1529,7 @@ run_mode(const char *prefix, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-overwrite-source-rev%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source-rev%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_overwrite_source__rev, @@ -1537,21 +1537,21 @@ run_mode(const char *prefix, } /* try to intermix copies with GPU copies*/ - igt_subtest_f("%s-%s-%s-intermix-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-intermix-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { igt_require(rendercopy); buffers_create(&buffers); run_wrap_func(&buffers, do_intermix_rcs, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-intermix-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-intermix-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { igt_require(rendercopy); buffers_create(&buffers); run_wrap_func(&buffers, do_intermix_bcs, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-intermix-both%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-intermix-both%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { igt_require(rendercopy); buffers_create(&buffers); run_wrap_func(&buffers, @@ -1560,7 +1560,7 @@ run_mode(const char *prefix, } /* try to read the results before the copy completes */ - igt_subtest_f("%s-%s-%s-early-read%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-early-read%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_early_read, @@ -1568,13 +1568,13 @@ run_mode(const char *prefix, } /* concurrent reads */ - igt_subtest_f("%s-%s-%s-read-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-read-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_read_read_bcs, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-read-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-read-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { igt_require(rendercopy); buffers_create(&buffers); run_wrap_func(&buffers, @@ -1583,13 +1583,13 @@ run_mode(const char *prefix, } /* split copying between rings */ - igt_subtest_f("%s-%s-%s-write-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-write-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_write_read_bcs, p->copy, h->hang); } - igt_subtest_f("%s-%s-%s-write-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-write-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { igt_require(rendercopy); buffers_create(&buffers); run_wrap_func(&buffers, @@ -1598,7 +1598,7 @@ run_mode(const char *prefix, } /* and finally try to trick the kernel into loosing the pending write */ - igt_subtest_f("%s-%s-%s-gpu-read-after-write%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { + igt_gem_stress_subtest_f("", "%s-%s-%s-gpu-read-after-write%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) { buffers_create(&buffers); run_wrap_func(&buffers, do_gpu_read_after_write,