From patchwork Sun Feb 12 23:28:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luc Van Oostenryck X-Patchwork-Id: 9568631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6BF2E60572 for ; Sun, 12 Feb 2017 23:29:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 60DA2280FC for ; Sun, 12 Feb 2017 23:29:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 55F5F28113; Sun, 12 Feb 2017 23:29:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A592280FC for ; Sun, 12 Feb 2017 23:29:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751131AbdBLX3R (ORCPT ); Sun, 12 Feb 2017 18:29:17 -0500 Received: from mail-wr0-f195.google.com ([209.85.128.195]:33135 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751256AbdBLX3Q (ORCPT ); Sun, 12 Feb 2017 18:29:16 -0500 Received: by mail-wr0-f195.google.com with SMTP id i10so21879553wrb.0 for ; Sun, 12 Feb 2017 15:29:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MzkC4Ju8XMRHZ4ZlfSb41KqN8ZQhSC5pkJ9ppmXa+QE=; b=SONx55sFsYsnj47QLJ16r03RCMbRKHJSRy+HFpnaOxQt4mbZUzIKcARXL8WmA9kPpm K1YH6uIGOTSp4TUQuOvXRR3efCZM/vVXguan4fj0sYOINXG/sNqGlXpBVX4D41rqUd99 a9fgAfXcT2pDbt+aqzTdWo/VICd6ARqKZkTiuGkA9aeufDf51x//taYMESQwSPBfkdci Ob7ZFuSMb8rBJIAJFdxWj5Iesn5x6kyTNsY922QUPvLGbz5sGSGFjSskZKg1juTBb9Dg n+ilEfePzwAJE+qN2JzwYgwg8M1OeKmtH4Y//bZG+Ik8oUuOa9nnxUL7/1n9R6tsVTDT JotA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MzkC4Ju8XMRHZ4ZlfSb41KqN8ZQhSC5pkJ9ppmXa+QE=; b=oNaILjzEBlfMDowvA86cbINqKpYQYMxRUANh22sGIVqWTkKxKI0MEerqyvNzQlYRK9 8wvgZUoVLXu0Bz0kCHEUFCGqFiJkyEHMJOzcUC9CmB6iAAvfzXVsT6nXL9zCWxbGKglA YA9SznPrdb/pkHTA8/SZ8Vi1Ny/qwOc1i0I4/4qJvvlvP/iq9oPt3RW4BsfX0TakaFvY W70JUpnYFHPyahRzC1V433r4IvOeSzzcMaznbPpTia2Prn3xQ7cYocQ9oe19Z2lRPf+c f/3WGufxEqksZp8XzOj4TIrd3xhr0WgN7c7zmrchn3WMxvFWq8RavmpOh36rAa+X5f7r /Z3A== X-Gm-Message-State: AMke39n4D2r+ZDsMm+wS2J0eZoFOeMcgmIeJEVX2Nhz39wKsZ6lQagAtURDcYjINq+d05g== X-Received: by 10.223.151.53 with SMTP id r50mr16061801wrb.129.1486942155368; Sun, 12 Feb 2017 15:29:15 -0800 (PST) Received: from localhost.localdomain ([2a02:a03f:870:3e00:1cde:6e8b:d488:2b72]) by smtp.gmail.com with ESMTPSA id 45sm8440605wrx.60.2017.02.12.15.29.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 12 Feb 2017 15:29:14 -0800 (PST) From: Luc Van Oostenryck To: linux-sparse@vger.kernel.org Cc: Christopher Li , Luc Van Oostenryck Subject: [PATCH v2 06/14] testsuite: report as error tests known to fail but which succeed Date: Mon, 13 Feb 2017 00:28:56 +0100 Message-Id: <20170212232904.49647-7-luc.vanoostenryck@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170212232904.49647-1-luc.vanoostenryck@gmail.com> References: <20170208222805.c77ilwchxpg2vnx4@macpro.local> <20170212232904.49647-1-luc.vanoostenryck@gmail.com> Sender: linux-sparse-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-sparse@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Such situation may simply show that what was tested is now fixed and that it's juste the test annotation which need to be adapted, but can be a sign that something else is broken. Reporting the exact result (failure/success, known-to-fail/expect-to-succeed) make the testsuite more useful and allow to use more efficiently git-bisect or other automated testing tools. Signed-off-by: Luc Van Oostenryck --- validation/test-suite | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/validation/test-suite b/validation/test-suite index 0d874e075..8692bfe98 100755 --- a/validation/test-suite +++ b/validation/test-suite @@ -11,6 +11,10 @@ if [ ! -x "$default_path/sparse-llvm" ]; then disabled_cmds="sparsec sparsei sparse-llvm" fi +# flags: +# - some tests gave an unexpected result +failed=0 + # counts: # - tests that have not been converted to test-suite format # - tests that are disabled @@ -163,18 +167,27 @@ do_test() test_failed=1 fi - if [ "$test_failed" -eq "1" ]; then - ko_tests=`expr $ko_tests + 1` - get_tag "check-known-to-fail" $file - if [ "$?" -eq "0" ]; then + get_tag "check-known-to-fail" $file + must_fail=`expr "$?" = 0` + known_ko_tests=`expr $known_ko_tests + $must_fail` + + [ "$test_failed" -eq "$must_fail" ] || failed=1 + + if [ "$must_fail" -eq "1" ]; then + if [ "$test_failed" -eq "1" ]; then echo "info: test '$file' is known to fail" - known_ko_tests=`expr $known_ko_tests + 1` + else + echo "info: test '$file' is known to fail but succeed!" + test_failed=1 fi - return 1 + fi + + if [ "$test_failed" -eq "1" ]; then + ko_tests=`expr $ko_tests + 1` else ok_tests=`expr $ok_tests + 1` - return 0 fi + return $test_failed } do_test_suite() @@ -273,5 +286,5 @@ case "$1" in ;; esac -exit 0 +exit $failed