From patchwork Thu Nov 17 16:13:58 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Li X-Patchwork-Id: 9434607 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D30D660469 for ; Thu, 17 Nov 2016 17:08:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF29D2966C for ; Thu, 17 Nov 2016 17:08:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B400C2966F; Thu, 17 Nov 2016 17:08:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID,T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 682C72966C for ; Thu, 17 Nov 2016 17:08:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933105AbcKQRIe (ORCPT ); Thu, 17 Nov 2016 12:08:34 -0500 Received: from mail-it0-f65.google.com ([209.85.214.65]:35027 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934401AbcKQRI2 (ORCPT ); Thu, 17 Nov 2016 12:08:28 -0500 Received: by mail-it0-f65.google.com with SMTP id b123so16515900itb.2 for ; Thu, 17 Nov 2016 09:08:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=ZP/BPOxhjAACSmdQRZ3bzfPaLkVKf0WRvS2AxwRa+XY=; b=BMpSBb0oI/YG0SeJuZFqyuZ+/TVqu3dXwXxKdcvdo8PGsG/iFgdgmtDJZUYik1jyBS nxBGOUPgHR9ni3FwghATQl9cR726YYkecrgGYkL5RCwEObzrBeInIdCUvM5+HNGcMFoI AAIvIe9WUuL8TIA35tWLkgwVmhXE+j/Lm/K0b0/NpyrkuWBAIciEumvPnt7SsoORVXiq 2zrNkdyWcXn/0EWelr26MwxSIWv+iYRS2OhrY4uhFofuWocfFDgdmcw+9mmE2Y5Ll2jV u8LRZhJhQjJ5e673wJqO4ygrMoX4agBgD7S3rCclG3wORSmKU64h7BNVLv1BsuumQCTj uZSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=ZP/BPOxhjAACSmdQRZ3bzfPaLkVKf0WRvS2AxwRa+XY=; b=lmTQlh+a5EovgA0qZOSnLk/rVHm3XLaacLQGokPzEPsVRhosKLNyjW0vdkE99GXdzZ rl2zmAR+3CiwqV+rg5zP1st/ZQqYb+97olYwrORAMGlp5wYPw1Qjo9rc/flTd/f/dRUl w4EAC2tblUD8ldQLivyxAw2BcYeysheWYv5vcZ8nRFLgcjXKLkcdTzOgX8rPdzELxLc/ QgNn303BsT720+WAG9VsB2jQZGBWQqBsdcFnc8zGBk9z9T+H4auhdYccB9ZHzdCNlL6H HrbJM7pwEZzZK4I2zTrQXf49gr75eLMHJu+18Y7IYTaoEAc5JFT1KRmxzHmcJ5f2vRZh l5CA== X-Gm-Message-State: AKaTC00vIZi5XVknJma/3SJSsCWkC+dLICYxytQcicj/fkMTeqldFGmsleYYPZugo1iWdwFL5MrWqa3gRrLR5A== X-Received: by 10.107.19.22 with SMTP id b22mr3552891ioj.236.1479399238580; Thu, 17 Nov 2016 08:13:58 -0800 (PST) MIME-Version: 1.0 Received: by 10.36.209.68 with HTTP; Thu, 17 Nov 2016 08:13:58 -0800 (PST) In-Reply-To: <20161102214509.36571-4-luc.vanoostenryck@gmail.com> References: <20161102214509.36571-1-luc.vanoostenryck@gmail.com> <20161102214509.36571-4-luc.vanoostenryck@gmail.com> From: Christopher Li Date: Fri, 18 Nov 2016 00:13:58 +0800 X-Google-Sender-Auth: Iiup_lA6QyhwNC1jxhkDQTzEklc Message-ID: Subject: Re: [PATCH 3/5] testsuite: report as error tests known to fail but which succeed To: Luc Van Oostenryck Cc: Linux-Sparse Sender: linux-sparse-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-sparse@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Thu, Nov 3, 2016 at 5:45 AM, Luc Van Oostenryck wrote: > Such situation may simply show that what was tested is now fixed > and that it's juste the test annotation which need to be adapted, > but can be a sign that something else is broken. > > Reporting the exact result (failure/success, known-to-fail/expect-to-succeed) > make the testsuite more useful and allow to use more efficiently > git-bisect or other automated testing tools. I like what you are doing as a result. But I think the implementation has room to improve. I don't like deep and messy "else" statement. I attach a patch which I modify base on yours. I hope it is easier to read. Care to review it? Chris --- .pc/9409919-3-5-testsuite-report-as-error-tests-known-to-fail-but-which-succeed.patch/validation/test-suite 2016-10-24 17:37:29.303846000 +0800 +++ validation/test-suite 2016-11-17 23:56:35.404392454 +0800 @@ -161,18 +161,25 @@ do_test() test_failed=1 fi - if [ "$test_failed" -eq "1" ]; then - ko_tests=`expr $ko_tests + 1` - get_tag "check-known-to-fail" $file - if [ "$?" -eq "0" ]; then + get_tag "check-known-to-fail" $file + must_fail=`expr "$?" = 0` + known_ko_tests=`expr $known_ko_tests + $must_fail` + + if [ "$must_fail" -eq "1" ]; then + if [ "$test_failed" -eq "1" ]; then echo "info: test '$file' is known to fail" - known_ko_tests=`expr $known_ko_tests + 1` + else + echo "info: test '$file' is known to fail but succeed!" + test_failed=1 fi - return 1 + fi + + if [ "$test_failed" -eq "1" ]; then + ko_tests=`expr $ko_tests + 1` else ok_tests=`expr $ok_tests + 1` - return 0 fi + return $test_failed } do_test_suite()