From patchwork Mon Mar 16 20:21:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heidi Fahim X-Patchwork-Id: 11441247 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7B28713B1 for ; Mon, 16 Mar 2020 20:21:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 419F120679 for ; Mon, 16 Mar 2020 20:21:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lL1UaDJ3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732503AbgCPUVq (ORCPT ); Mon, 16 Mar 2020 16:21:46 -0400 Received: from mail-pg1-f202.google.com ([209.85.215.202]:36301 "EHLO mail-pg1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732436AbgCPUVp (ORCPT ); Mon, 16 Mar 2020 16:21:45 -0400 Received: by mail-pg1-f202.google.com with SMTP id b130so12879355pga.3 for ; Mon, 16 Mar 2020 13:21:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=0dKXpM+vAToFQ99KFoY4s/h1UvCiGXVGJzVOlRPgN/w=; b=lL1UaDJ33tqQdzKUxp7HqRoiLyuZO479lvU6K9jUmKLpg0o1p4LSgewSvJ2wJzWsYK RVeji6SAfZdkg8w/OOlb/aA5GppJiCyzOIuTDimNye627x0kRPDMO/sVk0Ka/uQMLYGI V8sbQrXBatkbxDemqrL+u4d1aTYPkVNi8PM7Ahm8XZDhy4T2IXt04dCDAOmnJclbME92 z2eXHlgn6AfKzhPa5Pujnaja7PShVvF7S0L6WAPUX6EpwT9ZudwrRuaAoneqKLhbXouO bXHeb6ZYoEZZbQUJn2C7DSsftxZAP7oUeRO9KtsztW9zQ+fmMas08bfd9uzqAZqq1g10 N8XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=0dKXpM+vAToFQ99KFoY4s/h1UvCiGXVGJzVOlRPgN/w=; b=Nr48Kx3l7hf566JQoioYdmlLmCdfk3n7j59h3gCPidKW63FLmG1VgpG7CbzVhqPsaF iN4dxdJnSXcJDG/DHZ2+gtaeEmVLqEEP6AW+BRLr0W6M7k7Q8Y8yrJxae6F4ektYyIJd Ib+DKQs9L7FLQ3nonCmmBEMqwmRTIitHCbRNUDDhdZZ1EMpU/hmmvptnzI7jKfUzD7aA Qwj0EcTGtYwffMi8HxiML9gRziavUFL4OifqofY46dkbVFTfOLcEhcMYMTYMs2bNg0U1 lAC21qp4JsS6YKe1CdFAtJ+A0yBk/XNyiojPWNXhyxuNXfI8BAvSSNVoFWcaMwKm06h6 U64w== X-Gm-Message-State: ANhLgQ1uolZfLU4VpzUK3s1TxIlR9WZZYMyao+Yh5JetQV1ppLeDDqG9 qfj11dityZ2yXjKkMlPE+pSVFKP8gIwdxMTm X-Google-Smtp-Source: ADFU+vsklmZkx4fIL8aNLf15/1C/GMKjVqewxq+sntb7vgeAeK+izs7ZtDWLg1kkj0HyrUNTJ7JSPV82/CJAvhX0 X-Received: by 2002:a17:90b:3683:: with SMTP id mj3mr1254907pjb.153.1584390103472; Mon, 16 Mar 2020 13:21:43 -0700 (PDT) Date: Mon, 16 Mar 2020 13:21:24 -0700 Message-Id: <20200316202125.15852-1-heidifahim@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog Subject: [PATCH v3 1/2] kunit: kunit_parser: make parser more robust From: Heidi Fahim To: shuah@kernel.org, brendanhiggins@google.com, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com Cc: Heidi Fahim Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Previously, kunit_parser did not properly handle kunit TAP output that - had any prefixes (generated from different configs e.g. CONFIG_PRINTK_TIME) - had unrelated kernel output mixed in the middle of it, which has shown up when testing with allyesconfig To remove prefixes, the parser looks for the first line that includes TAP output, "TAP version 14". It then determines the length of the string before this sequence, and strips that number of characters off the beginning of the following lines until the last KUnit output line is reached. These fixes have been tested with additional tests in the KUnitParseTest and their associated logs have also been added. Signed-off-by: Heidi Fahim Reviewed-by: Brendan Higgins --- Changelog v3: - Addressing Shuah's git apply issues with whitespaces. Marked all files in test_data as binaries in new kunit/ .gitattributes file. The irregular whitespaces here are necessary as they are used for testing the KUnit parser. - Important: please note that now git apply works, however the .log files in /test_data/ may get marked as Untracked files. These should be included in the commit as they are required for kunit_tool_test.py. tools/testing/kunit/.gitattributes | 1 + tools/testing/kunit/kunit_parser.py | 40 +++++----- tools/testing/kunit/kunit_tool_test.py | 69 ++++++++++++++++++ .../test_data/test_config_printk_time.log | Bin 0 -> 1584 bytes .../test_data/test_interrupted_tap_output.log | Bin 0 -> 1982 bytes .../test_data/test_kernel_panic_interrupt.log | Bin 0 -> 1321 bytes .../test_data/test_multiple_prefixes.log | Bin 0 -> 1832 bytes ..._output_with_prefix_isolated_correctly.log | Bin 0 -> 1655 bytes .../kunit/test_data/test_pound_no_prefix.log | Bin 0 -> 1193 bytes .../kunit/test_data/test_pound_sign.log | Bin 0 -> 1655 bytes 10 files changed, 90 insertions(+), 20 deletions(-) create mode 100644 tools/testing/kunit/.gitattributes create mode 100644 tools/testing/kunit/test_data/test_config_printk_time.log create mode 100644 tools/testing/kunit/test_data/test_interrupted_tap_output.log create mode 100644 tools/testing/kunit/test_data/test_kernel_panic_interrupt.log create mode 100644 tools/testing/kunit/test_data/test_multiple_prefixes.log create mode 100644 tools/testing/kunit/test_data/test_output_with_prefix_isolated_correctly.log create mode 100644 tools/testing/kunit/test_data/test_pound_no_prefix.log create mode 100644 tools/testing/kunit/test_data/test_pound_sign.log GIT binary patch literal 1655 zcmah}O^?$s5IvV)F;XueHpOY1wnXB979k zUt&$>z4`Rp106PFD@Nx5DwAt>j6tHdSSZ++G6)|)X^|8veY1b*ZG=v-jNzahZ|^*5 zoU$NALGF+lEFssqT z!#;1vGe@mEEGa5G%tNVMXoCOLa#cI+1ob?ij}ql*<%^-L{c^W(-oo1ssIqbmN6jTs zI(lD2SfA=X@ICh{K7jT*(VDv$Nxmzil@FtHNd-~uU>hxI95MX*x{Kld zEQ{dqiW@QuBw4jSYCP#6AvHL@{kXYd@4EE2+xr+8Y`=YpNr!H+Bqg;cQPG2gJBsB5 zX)&p2G+WMCU}P8^l$fXKTC&YLQbC#<*TA&5^gAcyv0EF!ip|+(m2Kz}CUU-Fux63; jvVXaBGrY#ID3eNFT 0 else line + elif kunit_end_re.search(line): break elif started: - yield line + yield line[prefix_len:] if prefix_len > 0 else line def raw_output(kernel_output): for line in kernel_output: @@ -108,18 +110,16 @@ OK_NOT_OK_SUBTEST = re.compile(r'^\t(ok|not ok) [0-9]+ - (.*)$') OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) [0-9]+ - (.*)$') -def parse_ok_not_ok_test_case(lines: List[str], - test_case: TestCase, - expecting_test_case: bool) -> bool: +def parse_ok_not_ok_test_case(lines: List[str], test_case: TestCase) -> bool: save_non_diagnositic(lines, test_case) if not lines: - if expecting_test_case: - test_case.status = TestStatus.TEST_CRASHED - return True - else: - return False + test_case.status = TestStatus.TEST_CRASHED + return True line = lines[0] match = OK_NOT_OK_SUBTEST.match(line) + while not match and lines: + line = lines.pop(0) + match = OK_NOT_OK_SUBTEST.match(line) if match: test_case.log.append(lines.pop(0)) test_case.name = match.group(2) @@ -150,12 +150,12 @@ def parse_diagnostic(lines: List[str], test_case: TestCase) -> bool: else: return False -def parse_test_case(lines: List[str], expecting_test_case: bool) -> TestCase: +def parse_test_case(lines: List[str]) -> TestCase: test_case = TestCase() save_non_diagnositic(lines, test_case) while parse_diagnostic(lines, test_case): pass - if parse_ok_not_ok_test_case(lines, test_case, expecting_test_case): + if parse_ok_not_ok_test_case(lines, test_case): return test_case else: return None @@ -234,11 +234,11 @@ def parse_test_suite(lines: List[str]) -> TestSuite: expected_test_case_num = parse_subtest_plan(lines) if not expected_test_case_num: return None - test_case = parse_test_case(lines, expected_test_case_num > 0) - expected_test_case_num -= 1 - while test_case: + while expected_test_case_num > 0: + test_case = parse_test_case(lines) + if not test_case: + break test_suite.cases.append(test_case) - test_case = parse_test_case(lines, expected_test_case_num > 0) expected_test_case_num -= 1 if parse_ok_not_ok_test_suite(lines, test_suite): test_suite.status = bubble_up_test_case_errors(test_suite) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index cba97756ac4a..0efae697f396 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -108,6 +108,36 @@ class KUnitParserTest(unittest.TestCase): self.assertContains('ok 1 - example', result) file.close() + def test_output_with_prefix_isolated_correctly(self): + log_path = get_absolute_path( + 'test_data/test_pound_sign.log') + with open(log_path) as file: + result = kunit_parser.isolate_kunit_output(file.readlines()) + self.assertContains('TAP version 14\n', result) + self.assertContains(' # Subtest: kunit-resource-test', result) + self.assertContains(' 1..5', result) + self.assertContains(' ok 1 - kunit_resource_test_init_resources', result) + self.assertContains(' ok 2 - kunit_resource_test_alloc_resource', result) + self.assertContains(' ok 3 - kunit_resource_test_destroy_resource', result) + self.assertContains(' foo bar #', result) + self.assertContains(' ok 4 - kunit_resource_test_cleanup_resources', result) + self.assertContains(' ok 5 - kunit_resource_test_proper_free_ordering', result) + self.assertContains('ok 1 - kunit-resource-test', result) + self.assertContains(' foo bar # non-kunit output', result) + self.assertContains(' # Subtest: kunit-try-catch-test', result) + self.assertContains(' 1..2', result) + self.assertContains(' ok 1 - kunit_test_try_catch_successful_try_no_catch', + result) + self.assertContains(' ok 2 - kunit_test_try_catch_unsuccessful_try_does_catch', + result) + self.assertContains('ok 2 - kunit-try-catch-test', result) + self.assertContains(' # Subtest: string-stream-test', result) + self.assertContains(' 1..3', result) + self.assertContains(' ok 1 - string_stream_test_empty_on_creation', result) + self.assertContains(' ok 2 - string_stream_test_not_empty_after_add', result) + self.assertContains(' ok 3 - string_stream_test_get_string', result) + self.assertContains('ok 3 - string-stream-test', result) + def test_parse_successful_test_log(self): all_passed_log = get_absolute_path( 'test_data/test_is_test_passed-all_passed.log') @@ -150,6 +180,45 @@ class KUnitParserTest(unittest.TestCase): result.status) file.close() + def test_ignores_prefix_printk_time(self): + prefix_log = get_absolute_path( + 'test_data/test_config_printk_time.log') + with open(prefix_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual('kunit-resource-test', result.suites[0].name) + + def test_ignores_multiple_prefixes(self): + prefix_log = get_absolute_path( + 'test_data/test_multiple_prefixes.log') + with open(prefix_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual('kunit-resource-test', result.suites[0].name) + + def test_prefix_mixed_kernel_output(self): + mixed_prefix_log = get_absolute_path( + 'test_data/test_interrupted_tap_output.log') + with open(mixed_prefix_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual('kunit-resource-test', result.suites[0].name) + + def test_prefix_poundsign(self): + pound_log = get_absolute_path('test_data/test_pound_sign.log') + with open(pound_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual('kunit-resource-test', result.suites[0].name) + + def test_kernel_panic_end(self): + panic_log = get_absolute_path('test_data/test_kernel_panic_interrupt.log') + with open(panic_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual('kunit-resource-test', result.suites[0].name) + + def test_pound_no_prefix(self): + pound_log = get_absolute_path('test_data/test_pound_no_prefix.log') + with open(pound_log) as file: + result = kunit_parser.parse_run_tests(file.readlines()) + self.assertEqual('kunit-resource-test', result.suites[0].name) + class StrContains(str): def __eq__(self, other): return self in other diff --git a/tools/testing/kunit/test_data/test_config_printk_time.log b/tools/testing/kunit/test_data/test_config_printk_time.log new file mode 100644 index 0000000000000000000000000000000000000000..c02ca773946d641291e27d44d73174cc16a17d9d GIT binary patch literal 1584 zcmai!-)q}25XYapzv7@T8>=%piIdE*htV+@8zqo+FQEuUKBp$O&PeiF|Mxq^U79*h zs}~FTKHvK#-5Pj_k(dcycTn5H_+1K`jH41^UFj&k12k$=V)kzR(%LGMQI*0Lz3ldK z(|UBERmtX%TdJTig_lINlSTGXJWi5N;&CeAq44*ht=Y<8@I~~se}7%VU$m|u1M#%~ z_u>*(&4yk16m|^@L>3)2R$~+nPt4=dC^y^)CYDjs4Rz;38qZzYxAj@0F+G)x&<*QwCyL`dCmh7*>d7vEleL$_OFG2z??d#nTljtjZ9ZkNm*D-^a#H~OYwd7vmgtfWvPH*5DjZL(I!MOc* zZy)C6f=sy((vk=QgI1SDV&H{a>VtlNvL)0h3?OLm{2!iseC^X(WV?{V9Y~5?%QdtM zIog4Y$gt7XPHcL5F%RLE^7>OapbwB)g37@&>kq)-GO-S+e#?H_>cu=b6LN(iT%(IE z9EjK+tP#FY+EC5piok8`;o%`XoL*g?pWgid?*fpdt@Pa)qRq63FzQiNXhFU;Fujx0hFd>uD?GdFr`uyIxOliVuJ{_5EAR0Ab2aCJuu!;NtvNgWwEYgCwQ_0 zTo_%T;ob|bO;lI39l>Vje$e?_gW?S3S~$7dIVr=oxxGz$&43UKuUWV)r9^AvGUHyT zbR+NLc$=Ae$lf?g)nbV>+HM|lWla-&5X(mFG+I>QfLcVn zXo%pNY&qd~OaznuxX%J32egsIFp7>On;lIsh!W!}*xzdUxf3#N#|AKELpGbnGdc_w rayVsh#3s_;p6f%~!%YmE#1iFNm430IoQg#U=CA6uosqXj&w%{_Pb*9~ literal 0 HcmV?d00001 diff --git a/tools/testing/kunit/test_data/test_kernel_panic_interrupt.log b/tools/testing/kunit/test_data/test_kernel_panic_interrupt.log new file mode 100644 index 0000000000000000000000000000000000000000..c045eee75f27fefaabf3ba073d9e282721c19a67 GIT binary patch literal 1321 zcmai!-)q}25XYapzv7@T8=EuPiIdc@htV+@gAvHOmr#T(%c-%gGm^a2|NYK!=cfMA z>c!%GpYPr2bUFYnwqR>U>mDkTX_rSJQCln&97^f=_n@?h3zfV(y!19gCs;;sRE~Fh z9yd-|kfI=WNDY>d>CAD&%JMAv$R3yBXZE;aaLU{RNTZLub@;-D+P}Z<;IA?^Lk<3F z`1k4>_H|65Tm*Sd0-jM8Ya^A{8b|K_ViAsvn1c^q$MgR@Z~1dhy~$3I!3;^2qwLWs zaydgL6xd+%D!ZCtWub(wtDc=qIh=tjgOxHoHA-wazu3v=`I;&EMZ_Kd}!({oCDL1PpdRzC|=Sw^-8b z+mop1_P{R^a)MqDn)`6ES}@N@-#aQ1B*{jy?Iu)Sk{Q>)^{Cr>CuANCqrsX5Y`adk qwDDsZtQl-rDBFjs-@L}KFqKO7uB5^OxfbyR^W-{?vuvZ$^4UKw_JoN5 literal 0 HcmV?d00001 diff --git a/tools/testing/kunit/test_data/test_multiple_prefixes.log b/tools/testing/kunit/test_data/test_multiple_prefixes.log new file mode 100644 index 0000000000000000000000000000000000000000..bc48407dcc36c44665c7d2ac620e42e7caf98481 GIT binary patch literal 1832 zcma)-UvJYe5Wt_ur#NXZAeQ2^O;;lEfQFC|XsWc0mua$G=h8(@e39+2<=b;^pet<7 z>X$ed-|znUF1Nr-OvFO)y1NZ;*XbR!wlaR3gC^ssM!0Qs!eTUPU1RZPcka2CRi>aO zhYfl;?B%-m=s>HQ)u6Q0KxHZ~iCQPC;=OpBWgo=jT);!+??GF$k>S865wmgqc?G}F zx{3^>^Vq4AQ@HBZY})6r?HHI?bktdmNf1WW(#a%?exxm=kPJNemnQ>n=ks3V#t730 zfipyUj~FA&B7}k+vD6>Mm`518MrFEo&ln9Qi$GdSEm~PxL`k+lzDt`}K!-U!vPREJ z>usNC<@NomwVlS?I63rVC@E9rjw)TJH9A*zwG2W-gM5tjROQ|Ecw%+tKUzWP#(C`8 z9+w;$>p_nAQgHqgexPbz&CqNts&M-UoE&gNrm$&!FUd&FCOkLh7$b!80>hA(14osf zFOMpsQzq0Z^uP<0Fn9)W%!Pk+k4egp%6Xs?xIdv))>k1wm&f>*1Z7pD>z&7j$7fd; z&`@VQSR#K0TnsE?+A-*Nv<8EDQ)v#Ia|Nz9n#-EQ&#zZG-0bEucsj5Jr30Rko!J_C zFv*(7%l7TZ<(YUFGjp-J$w9#C+n1aN-32u~%H4x%xtZaTZFzsWIhpz)oHRp zH7$yBEtcmQCPig^52w*4yYfnJW3dA)#8fO7<&rn)T2B`O&PArfe`&vod48xXvQneo XtMDf=)eE&A(fy zUt&$>z4`Rp106PFD@Nx5DwAt>j6tHdSSZ++G6)|)X^|8veY1b*ZG=v-jNzahZ|^*5 zoU$NALGF+lEFssqT z!#;1vGe@mEEGa5G%tNVMXoCOLa#cI+1ob?ij}ql*<%^-L{c^W(-oo1ssIqbmN6jTs zI(lD2SfA=X@ICh{K7jT*(VDv$Nxmzil@FtHNd-~uU>hxI95MX*x{Kld zEQ{dqiW@QuBw4jSYCP#6AvHL@{kXYd@4EE2+xr+8Y`=YpNr!H+Bqg;cQPG2gJBsB5 zX)&p2G+WMCU}P8^l$fXKTC&YLQbC#<*TA&5^gAcyv0EF!ip|+(m2Kz}CUU-Fux63; jvVXaBGrY#ID3eNFTtt2D7x%05gScM{xU2jvXlo8~06vKbetq7-KeVnQ1IapqHy5z)axUjHICc!h z79Dj~V?1m!kSrD{zvu`F#DS5Ij2xJ;_O#^!yFA0PXVMcK*wq=f;Nq7084V>7Kc{l+DZE0Dm{a@mRhv3w1|>yf&57h5zMdrJeHXgFl6E}JD|?D-4NP~RK4wE zt-QXS#kvf4IK?`q$)p@f={l{^xw5O}uxTjl2h}OyXR6LT(-nm7QOQ@Dgz*bcGuNCv zXAtuXRWtLvI(ZI=3`pn%Y+Bz-GE%e5bB~?8zCs)FGALB(Iet}eyAO>vG{h_XFacqE z{5B`*sm&ji^FSqVt3j)*uLdJ7i{K|gS=H!z=ds~=>*@j;>Wl|V`)nb>?X3&cr$Q`SIDZMAulOIpvz#v03lUqL#0ON%qPWsxdzE5bRMz)!!40$bN+0Q%a X-Patchwork-Id: 11441249 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 024D71668 for ; Mon, 16 Mar 2020 20:21:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CE07020719 for ; Mon, 16 Mar 2020 20:21:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hHtzqGW2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732536AbgCPUVu (ORCPT ); Mon, 16 Mar 2020 16:21:50 -0400 Received: from mail-pj1-f73.google.com ([209.85.216.73]:41174 "EHLO mail-pj1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732532AbgCPUVu (ORCPT ); Mon, 16 Mar 2020 16:21:50 -0400 Received: by mail-pj1-f73.google.com with SMTP id k3so11760090pjo.6 for ; Mon, 16 Mar 2020 13:21:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=To5N8tUaCvz6MVLHlfCQ9mbVyH/J+Wx9dmrV0++j4lg=; b=hHtzqGW2wqp+frqqcV5F6ULbiTUd+KNsCjFgkkR6VUHzqMqAzmW7BZgNddhTlWR097 NvSNHNUslVQPh2yVlBZ7OcDGTTHbPpPbc+2em1gzfwMUcJ4rV2omEabS55+EDGJ2vzYC vcqBpYFw6pKPRdsp4KkS75br6p391Xz14pXKasPdT3Q1qVDMAyeuT5apvhPvuWbnzCFV xCZ3NB5Q+cz3dpyEtsDkTkcHGChY6aNbT54XXP4xamsR8SaNgaQvUqyUcYJ/Ad+8TjLo DTePvbC0eoXKXnSxDtdqZr14znb7z+ESuzRGlV/vVFu2Ekq9NKqw4Qd48Kh8wI3VhEiz t+aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=To5N8tUaCvz6MVLHlfCQ9mbVyH/J+Wx9dmrV0++j4lg=; b=YwSR7MwxCl9lUHL/C5lyweKsyAP/MdVVRYBjXrd/cLpaKUJcpno9OocgSKAs2HMnF7 HfEjgzzvJcOnD15DOZvENstHEim0LZFjmf+mvrskPzv9OIbFZbqPtvtFOm0M1xRagq+n yMKmXm2CKSpwofnHaY9ydwJ2BDZ6U55HAOEoS23PMVgirktcPcT4qbqnQKqGDSSnFOZM b9A51qNUqlJC5n6DXmYHJh3woQUm4ugM2UjaVO4nZLVBMGALbo1/ij/MasVr0NkQ98v7 gBZdkDIq/G9jrKlJ1JLh5W3VfhANEnzFLzc23H4TLHrlpjLgzVmflMf1NoCq4PrxWmgV YQ7A== X-Gm-Message-State: ANhLgQ1BnSTxOX0g2UgcQYXcwMssXlt1dcRmQH0hXgpIxpZ0IUdHHnGs 5144Zsb1QYmQ0IZtSiTBTrH+CP+b6oFA3XpU X-Google-Smtp-Source: ADFU+vudic1Hqys5mhWZo4n3tXuf/uO0RkDAaANduCIY7/w+eDZqBZGdeokIqwjtq1a/MhwWU6G9/YSRXcGUiNi6 X-Received: by 2002:a65:5905:: with SMTP id f5mr1411993pgu.87.1584390108022; Mon, 16 Mar 2020 13:21:48 -0700 (PDT) Date: Mon, 16 Mar 2020 13:21:25 -0700 In-Reply-To: <20200316202125.15852-1-heidifahim@google.com> Message-Id: <20200316202125.15852-2-heidifahim@google.com> Mime-Version: 1.0 References: <20200316202125.15852-1-heidifahim@google.com> X-Mailer: git-send-email 2.25.1.481.gfbce0eb801-goog Subject: [PATCH v3 2/2] kunit: Run all KUnit tests through allyesconfig From: Heidi Fahim To: shuah@kernel.org, brendanhiggins@google.com, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com Cc: Heidi Fahim Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Implemented the functionality to run all KUnit tests through kunit_tool by specifying an --alltests flag, which builds UML with allyesconfig enabled, and consequently runs every KUnit test. A new function was added to kunit_kernel: make_allyesconfig. Firstly, if --alltests is specified, kunit.py triggers build_um_kernel which call make_allyesconfig. This function calls the make command, disables the broken configs that would otherwise prevent UML from building, then starts the kernel with all possible configurations enabled. All stdout and stderr is sent to test.log and read from there then fed through kunit_parser to parse the tests to the user. Also added a signal_handler in case kunit is interrupted while running. Tested: Run under different conditions such as testing with --raw_output, testing program interrupt then immediately running kunit again without --alltests and making sure to clean the console. Signed-off-by: Heidi Fahim Reviewed-by: Brendan Higgins --- Changelog: v3: - Minor change since v2 in message for program interruption. Changed "Cleaning config." to "Cleaning console." .../kunit/configs/broken_on_uml.config | 41 ++++++++++++ tools/testing/kunit/kunit.py | 30 +++++---- tools/testing/kunit/kunit_kernel.py | 62 ++++++++++++++----- tools/testing/kunit/kunit_parser.py | 1 + tools/testing/kunit/kunit_tool_test.py | 17 ++--- 5 files changed, 115 insertions(+), 36 deletions(-) create mode 100644 tools/testing/kunit/configs/broken_on_uml.config diff --git a/tools/testing/kunit/configs/broken_on_uml.config b/tools/testing/kunit/configs/broken_on_uml.config new file mode 100644 index 000000000000..239b9f03da2c --- /dev/null +++ b/tools/testing/kunit/configs/broken_on_uml.config @@ -0,0 +1,41 @@ +# These are currently broken on UML and prevent allyesconfig from building +# CONFIG_STATIC_LINK is not set +# CONFIG_UML_NET_VECTOR is not set +# CONFIG_UML_NET_VDE is not set +# CONFIG_UML_NET_PCAP is not set +# CONFIG_NET_PTP_CLASSIFY is not set +# CONFIG_IP_VS is not set +# CONFIG_BRIDGE_EBT_BROUTE is not set +# CONFIG_BRIDGE_EBT_T_FILTER is not set +# CONFIG_BRIDGE_EBT_T_NAT is not set +# CONFIG_MTD_NAND_CADENCE is not set +# CONFIG_MTD_NAND_NANDSIM is not set +# CONFIG_BLK_DEV_NULL_BLK is not set +# CONFIG_BLK_DEV_RAM is not set +# CONFIG_SCSI_DEBUG is not set +# CONFIG_NET_VENDOR_XILINX is not set +# CONFIG_NULL_TTY is not set +# CONFIG_PTP_1588_CLOCK is not set +# CONFIG_PINCTRL_EQUILIBRIUM is not set +# CONFIG_DMABUF_SELFTESTS is not set +# CONFIG_COMEDI is not set +# CONFIG_XIL_AXIS_FIFO is not set +# CONFIG_EXFAT_FS is not set +# CONFIG_STM_DUMMY is not set +# CONFIG_FSI_MASTER_ASPEED is not set +# CONFIG_JFS_FS is not set +# CONFIG_UBIFS_FS is not set +# CONFIG_CRAMFS is not set +# CONFIG_CRYPTO_DEV_SAFEXCEL is not set +# CONFIG_CRYPTO_DEV_AMLOGIC_GXL is not set +# CONFIG_KCOV is not set +# CONFIG_LKDTM is not set +# CONFIG_REED_SOLOMON_TEST is not set +# CONFIG_TEST_RHASHTABLE is not set +# CONFIG_TEST_MEMINIT is not set +# CONFIG_NETWORK_PHY_TIMESTAMPING is not set +# CONFIG_DEBUG_INFO_BTF is not set +# CONFIG_PTP_1588_CLOCK_INES is not set +# CONFIG_QCOM_CPR is not set +# CONFIG_RESET_BRCMSTB_RESCAL is not set +# CONFIG_RESET_INTEL_GW is not set diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 180ad1e1b04f..650bb4cfc544 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -22,7 +22,9 @@ import kunit_parser KunitResult = namedtuple('KunitResult', ['status','result']) -KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs', 'build_dir', 'defconfig']) +KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs', + 'build_dir', 'defconfig', + 'alltests']) KernelDirectoryPath = sys.argv[0].split('tools/testing/kunit/')[0] @@ -55,24 +57,23 @@ def run_tests(linux: kunit_kernel.LinuxSourceTree, kunit_parser.print_with_timestamp('Building KUnit Kernel ...') build_start = time.time() - success = linux.build_um_kernel(request.jobs, request.build_dir) + success = linux.build_um_kernel(request.alltests, + request.jobs, + request.build_dir) build_end = time.time() if not success: return KunitResult(KunitStatus.BUILD_FAILURE, 'could not build kernel') kunit_parser.print_with_timestamp('Starting KUnit Kernel ...') test_start = time.time() - - test_result = kunit_parser.TestResult(kunit_parser.TestStatus.SUCCESS, - [], - 'Tests not Parsed.') + kunit_output = linux.run_kernel( + timeout=None if request.alltests else request.timeout, + build_dir=request.build_dir) if request.raw_output: - kunit_parser.raw_output( - linux.run_kernel(timeout=request.timeout, - build_dir=request.build_dir)) + raw_output = kunit_parser.raw_output(kunit_output) + isolated = list(kunit_parser.isolate_kunit_output(raw_output)) + test_result = kunit_parser.parse_test_result(isolated) else: - kunit_output = linux.run_kernel(timeout=request.timeout, - build_dir=request.build_dir) test_result = kunit_parser.parse_run_tests(kunit_output) test_end = time.time() @@ -120,6 +121,10 @@ def main(argv, linux=None): help='Uses a default .kunitconfig.', action='store_true') + run_parser.add_argument('--alltests', + help='Run all KUnit tests through allyesconfig', + action='store_true') + cli_args = parser.parse_args(argv) if cli_args.subcommand == 'run': @@ -143,7 +148,8 @@ def main(argv, linux=None): cli_args.timeout, cli_args.jobs, cli_args.build_dir, - cli_args.defconfig) + cli_args.defconfig, + cli_args.alltests) result = run_tests(linux, request) if result.status != KunitStatus.SUCCESS: sys.exit(1) diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index d99ae75ef72f..415d8f3c4fe4 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -10,11 +10,16 @@ import logging import subprocess import os +import signal + +from contextlib import ExitStack import kunit_config +import kunit_parser KCONFIG_PATH = '.config' kunitconfig_path = '.kunitconfig' +BROKEN_ALLCONFIG_PATH = 'tools/testing/kunit/configs/broken_on_uml.config' class ConfigError(Exception): """Represents an error trying to configure the Linux kernel.""" @@ -40,12 +45,29 @@ class LinuxSourceTreeOperations(object): if build_dir: command += ['O=' + build_dir] try: - subprocess.check_output(command) + subprocess.check_output(command, stderr=subprocess.PIPE) except OSError as e: raise ConfigError('Could not call make command: ' + e) except subprocess.CalledProcessError as e: raise ConfigError(e.output) + def make_allyesconfig(self): + kunit_parser.print_with_timestamp( + 'Enabling all CONFIGs for UML...') + process = subprocess.Popen( + ['make', 'ARCH=um', 'allyesconfig'], + stdout=subprocess.DEVNULL, + stderr=subprocess.STDOUT) + process.wait() + kunit_parser.print_with_timestamp( + 'Disabling broken configs to run KUnit tests...') + with ExitStack() as es: + config = open(KCONFIG_PATH, 'a') + disable = open(BROKEN_ALLCONFIG_PATH, 'r').read() + config.write(disable) + kunit_parser.print_with_timestamp( + 'Starting Kernel with all configs takes a few minutes...') + def make(self, jobs, build_dir): command = ['make', 'ARCH=um', '--jobs=' + str(jobs)] if build_dir: @@ -57,18 +79,16 @@ class LinuxSourceTreeOperations(object): except subprocess.CalledProcessError as e: raise BuildError(e.output) - def linux_bin(self, params, timeout, build_dir): + def linux_bin(self, params, timeout, build_dir, outfile): """Runs the Linux UML binary. Must be named 'linux'.""" linux_bin = './linux' if build_dir: linux_bin = os.path.join(build_dir, 'linux') - process = subprocess.Popen( - [linux_bin] + params, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - process.wait(timeout=timeout) - return process + with open(outfile, 'w') as output: + process = subprocess.Popen([linux_bin] + params, + stdout=output, + stderr=subprocess.STDOUT) + process.wait(timeout) def get_kconfig_path(build_dir): @@ -84,6 +104,7 @@ class LinuxSourceTree(object): self._kconfig = kunit_config.Kconfig() self._kconfig.read_from_file(kunitconfig_path) self._ops = LinuxSourceTreeOperations() + signal.signal(signal.SIGINT, self.signal_handler) def clean(self): try: @@ -135,7 +156,9 @@ class LinuxSourceTree(object): print('Generating .config ...') return self.build_config(build_dir) - def build_um_kernel(self, jobs, build_dir): + def build_um_kernel(self, alltests, jobs, build_dir): + if alltests: + self._ops.make_allyesconfig() try: self._ops.make_olddefconfig(build_dir) self._ops.make(jobs, build_dir) @@ -144,10 +167,15 @@ class LinuxSourceTree(object): return False return self.validate_config(build_dir) - def run_kernel(self, args=[], timeout=None, build_dir=''): - args.extend(['mem=256M']) - process = self._ops.linux_bin(args, timeout, build_dir) - with open(os.path.join(build_dir, 'test.log'), 'w') as f: - for line in process.stdout: - f.write(line.rstrip().decode('ascii') + '\n') - yield line.rstrip().decode('ascii') + def run_kernel(self, args=[], build_dir='', timeout=None): + args.extend(['mem=1G']) + outfile = 'test.log' + self._ops.linux_bin(args, timeout, build_dir, outfile) + subprocess.call(['stty', 'sane']) + with open(outfile, 'r') as file: + for line in file: + yield line + + def signal_handler(self, sig, frame): + logging.error('Build interruption occurred. Cleaning console.') + subprocess.call(['stty', 'sane']) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index adf86747b07f..02815062d5a6 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -65,6 +65,7 @@ def isolate_kunit_output(kernel_output): def raw_output(kernel_output): for line in kernel_output: print(line) + yield line DIVIDER = '=' * 60 diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 0efae697f396..ce47e87b633a 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -243,7 +243,8 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run'], self.linux_source_mock) assert self.linux_source_mock.build_reconfig.call_count == 1 assert self.linux_source_mock.run_kernel.call_count == 1 - self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='', timeout=300) + self.linux_source_mock.run_kernel.assert_called_once_with( + build_dir='', timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_passes_args_fail(self): @@ -258,25 +259,27 @@ class KUnitMainTest(unittest.TestCase): def test_run_raw_output(self): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) - kunit.main(['run', '--raw_output'], self.linux_source_mock) + with self.assertRaises(SystemExit) as e: + kunit.main(['run', '--raw_output'], self.linux_source_mock) + assert type(e.exception) == SystemExit + assert e.exception.code == 1 assert self.linux_source_mock.build_reconfig.call_count == 1 assert self.linux_source_mock.run_kernel.call_count == 1 - for kall in self.print_mock.call_args_list: - assert kall != mock.call(StrContains('Testing complete.')) - assert kall != mock.call(StrContains(' 0 tests run')) def test_run_timeout(self): timeout = 3453 kunit.main(['run', '--timeout', str(timeout)], self.linux_source_mock) assert self.linux_source_mock.build_reconfig.call_count == 1 - self.linux_source_mock.run_kernel.assert_called_once_with(build_dir='', timeout=timeout) + self.linux_source_mock.run_kernel.assert_called_once_with( + build_dir='', timeout=timeout) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_builddir(self): build_dir = '.kunit' kunit.main(['run', '--build_dir', build_dir], self.linux_source_mock) assert self.linux_source_mock.build_reconfig.call_count == 1 - self.linux_source_mock.run_kernel.assert_called_once_with(build_dir=build_dir, timeout=300) + self.linux_source_mock.run_kernel.assert_called_once_with( + build_dir=build_dir, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) if __name__ == '__main__':