| Message ID | 20230501181610.2617032-1-dlatypov@google.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | e30f65c4b3d671115bf2a9d9ef142285387f2aff |
| Delegated to: | Brendan Higgins |
| Headers | show |
| Series | kunit: tool: undo type subscripts for subprocess.Popen | expand |
On Mon, 1 May 2023 11:16:10 -0700 Daniel Latypov <dlatypov@google.com> wrote: > Writing `subprocess.Popen[str]` requires python 3.9+. > kunit.py has an assertion that the python version is 3.7+, so we should > try to stay backwards compatible. > > This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix > pre-existing `mypy --strict` errors and update run_checks.py"), since > mypy complains like so > > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] > > Note: `mypy --strict --python-version 3.7` does not work. > > We could annotate each file with comments like > `# mypy: disable-error-code="type-arg" > but then we might still get nudged to break back-compat in other files. > > This patch adds a `mypy.ini` file since it seems like the only way to > disable specific error codes for all our files. > > Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, > but I think being explicit is better, particularly since most kernel > devs won't be familiar with how mypy works. Thank you for fast fix! > > Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") > Reported-by: SeongJae Park <sj@kernel.org> > Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org > Signed-off-by: Daniel Latypov <dlatypov@google.com> Tested-by: SeongJae Park <sj@kernel.org> Thanks, SJ [...]
On Tue, 2 May 2023 at 02:16, 'Daniel Latypov' via KUnit Development <kunit-dev@googlegroups.com> wrote: > > Writing `subprocess.Popen[str]` requires python 3.9+. > kunit.py has an assertion that the python version is 3.7+, so we should > try to stay backwards compatible. > > This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix > pre-existing `mypy --strict` errors and update run_checks.py"), since > mypy complains like so > > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] > > Note: `mypy --strict --python-version 3.7` does not work. > > We could annotate each file with comments like > `# mypy: disable-error-code="type-arg" > but then we might still get nudged to break back-compat in other files. > > This patch adds a `mypy.ini` file since it seems like the only way to > disable specific error codes for all our files. > > Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, > but I think being explicit is better, particularly since most kernel > devs won't be familiar with how mypy works. > > Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") > Reported-by: SeongJae Park <sj@kernel.org> > Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org > Signed-off-by: Daniel Latypov <dlatypov@google.com> > --- Thanks for jumping on this. Looks good to me! Reviewed-by: David Gow <davidgow@google.com> Cheers, -- David
Hi David and Brendan, On Tue, 2 May 2023 08:04:20 +0800 David Gow <davidgow@google.com> wrote: > [-- Attachment #1: Type: text/plain, Size: 1473 bytes --] > > On Tue, 2 May 2023 at 02:16, 'Daniel Latypov' via KUnit Development > <kunit-dev@googlegroups.com> wrote: > > > > Writing `subprocess.Popen[str]` requires python 3.9+. > > kunit.py has an assertion that the python version is 3.7+, so we should > > try to stay backwards compatible. > > > > This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix > > pre-existing `mypy --strict` errors and update run_checks.py"), since > > mypy complains like so > > > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] > > > > Note: `mypy --strict --python-version 3.7` does not work. > > > > We could annotate each file with comments like > > `# mypy: disable-error-code="type-arg" > > but then we might still get nudged to break back-compat in other files. > > > > This patch adds a `mypy.ini` file since it seems like the only way to > > disable specific error codes for all our files. > > > > Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, > > but I think being explicit is better, particularly since most kernel > > devs won't be familiar with how mypy works. > > > > Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") > > Reported-by: SeongJae Park <sj@kernel.org> > > Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > --- > > Thanks for jumping on this. > > Looks good to me! > > Reviewed-by: David Gow <davidgow@google.com> Looks like this patch is still not merged in the mainline. May I ask the ETA, or any concern if you have? Thanks, SJ > > Cheers, > -- David
On Sat, 10 Jun 2023 at 03:09, SeongJae Park <sj@kernel.org> wrote: > > Hi David and Brendan, > > On Tue, 2 May 2023 08:04:20 +0800 David Gow <davidgow@google.com> wrote: > > > [-- Attachment #1: Type: text/plain, Size: 1473 bytes --] > > > > On Tue, 2 May 2023 at 02:16, 'Daniel Latypov' via KUnit Development > > <kunit-dev@googlegroups.com> wrote: > > > > > > Writing `subprocess.Popen[str]` requires python 3.9+. > > > kunit.py has an assertion that the python version is 3.7+, so we should > > > try to stay backwards compatible. > > > > > > This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix > > > pre-existing `mypy --strict` errors and update run_checks.py"), since > > > mypy complains like so > > > > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] > > > > > > Note: `mypy --strict --python-version 3.7` does not work. > > > > > > We could annotate each file with comments like > > > `# mypy: disable-error-code="type-arg" > > > but then we might still get nudged to break back-compat in other files. > > > > > > This patch adds a `mypy.ini` file since it seems like the only way to > > > disable specific error codes for all our files. > > > > > > Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, > > > but I think being explicit is better, particularly since most kernel > > > devs won't be familiar with how mypy works. > > > > > > Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") > > > Reported-by: SeongJae Park <sj@kernel.org> > > > Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org > > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > > --- > > > > Thanks for jumping on this. > > > > Looks good to me! > > > > Reviewed-by: David Gow <davidgow@google.com> > > Looks like this patch is still not merged in the mainline. May I ask the ETA, > or any concern if you have? > > We've got this queued for 6.5 in the kselftest/kunit tree[1], so it should land during the merge window. But I'll look into getting it applied as a fix for 6.4, beforehand. -- David [1]: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=kunit&id=e30f65c4b3d671115bf2a9d9ef142285387f2aff
On Sat, 10 Jun 2023 12:15:55 +0800 David Gow <davidgow@google.com> wrote: > [-- Attachment #1: Type: text/plain, Size: 2275 bytes --] > > On Sat, 10 Jun 2023 at 03:09, SeongJae Park <sj@kernel.org> wrote: > > > > Hi David and Brendan, > > > > On Tue, 2 May 2023 08:04:20 +0800 David Gow <davidgow@google.com> wrote: > > > > > [-- Attachment #1: Type: text/plain, Size: 1473 bytes --] > > > > > > On Tue, 2 May 2023 at 02:16, 'Daniel Latypov' via KUnit Development > > > <kunit-dev@googlegroups.com> wrote: > > > > > > > > Writing `subprocess.Popen[str]` requires python 3.9+. > > > > kunit.py has an assertion that the python version is 3.7+, so we should > > > > try to stay backwards compatible. > > > > > > > > This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix > > > > pre-existing `mypy --strict` errors and update run_checks.py"), since > > > > mypy complains like so > > > > > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] > > > > > > > > Note: `mypy --strict --python-version 3.7` does not work. > > > > > > > > We could annotate each file with comments like > > > > `# mypy: disable-error-code="type-arg" > > > > but then we might still get nudged to break back-compat in other files. > > > > > > > > This patch adds a `mypy.ini` file since it seems like the only way to > > > > disable specific error codes for all our files. > > > > > > > > Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, > > > > but I think being explicit is better, particularly since most kernel > > > > devs won't be familiar with how mypy works. > > > > > > > > Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") > > > > Reported-by: SeongJae Park <sj@kernel.org> > > > > Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org > > > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > > > --- > > > > > > Thanks for jumping on this. > > > > > > Looks good to me! > > > > > > Reviewed-by: David Gow <davidgow@google.com> > > > > Looks like this patch is still not merged in the mainline. May I ask the ETA, > > or any concern if you have? > > > > > > We've got this queued for 6.5 in the kselftest/kunit tree[1], so it > should land during the merge window. But I'll look into getting it > applied as a fix for 6.4, beforehand. Thank you for the kind answer, Gow! I was thinking this would be treated as a fix, and hence merged into the mainline before next merge window. I'm actually getting my personal test suite failures due to absence of this fix. It's not a critical problem, but it would definitely better for me if this could be merged into the mainline as early as possible. Thanks, SJ > > -- David > > [1]: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=kunit&id=e30f65c4b3d671115bf2a9d9ef142285387f2aff
Hi Greg and Sasha, On Sat, 10 Jun 2023 17:56:18 +0000 SeongJae Park <sj@kernel.org> wrote: > On Sat, 10 Jun 2023 12:15:55 +0800 David Gow <davidgow@google.com> wrote: > > > [-- Attachment #1: Type: text/plain, Size: 2275 bytes --] > > > > On Sat, 10 Jun 2023 at 03:09, SeongJae Park <sj@kernel.org> wrote: > > > > > > Hi David and Brendan, > > > > > > On Tue, 2 May 2023 08:04:20 +0800 David Gow <davidgow@google.com> wrote: > > > > > > > [-- Attachment #1: Type: text/plain, Size: 1473 bytes --] > > > > > > > > On Tue, 2 May 2023 at 02:16, 'Daniel Latypov' via KUnit Development > > > > <kunit-dev@googlegroups.com> wrote: > > > > > > > > > > Writing `subprocess.Popen[str]` requires python 3.9+. > > > > > kunit.py has an assertion that the python version is 3.7+, so we should > > > > > try to stay backwards compatible. > > > > > > > > > > This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix > > > > > pre-existing `mypy --strict` errors and update run_checks.py"), since > > > > > mypy complains like so > > > > > > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] > > > > > > > > > > Note: `mypy --strict --python-version 3.7` does not work. > > > > > > > > > > We could annotate each file with comments like > > > > > `# mypy: disable-error-code="type-arg" > > > > > but then we might still get nudged to break back-compat in other files. > > > > > > > > > > This patch adds a `mypy.ini` file since it seems like the only way to > > > > > disable specific error codes for all our files. > > > > > > > > > > Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, > > > > > but I think being explicit is better, particularly since most kernel > > > > > devs won't be familiar with how mypy works. > > > > > > > > > > Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") > > > > > Reported-by: SeongJae Park <sj@kernel.org> > > > > > Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org > > > > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > > > > --- > > > > > > > > Thanks for jumping on this. > > > > > > > > Looks good to me! > > > > > > > > Reviewed-by: David Gow <davidgow@google.com> > > > > > > Looks like this patch is still not merged in the mainline. May I ask the ETA, > > > or any concern if you have? > > > > > > > > > > We've got this queued for 6.5 in the kselftest/kunit tree[1], so it > > should land during the merge window. But I'll look into getting it > > applied as a fix for 6.4, beforehand. > > Thank you for the kind answer, Gow! I was thinking this would be treated as a > fix, and hence merged into the mainline before next merge window. I'm actually > getting my personal test suite failures due to absence of this fix. It's not a > critical problem, but it would definitely better for me if this could be merged > into the mainline as early as possible. This patch is now in the mainline (e30f65c4b3d671115bf2a9d9ef142285387f2aff). However, this fix is not in 6.4.y yet, so the original issue is reproducible on 6.4.y. Could you please add this to 6.4.y? I confirmed the mainline commit can cleanly applied on latest 6.1.y tree, and it fixes the issue. Thanks, SJ > > > Thanks, > SJ > > > > > -- David > > > > [1]: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/commit/?h=kunit&id=e30f65c4b3d671115bf2a9d9ef142285387f2aff
On Mon, Jul 03, 2023 at 07:27:04PM +0000, SeongJae Park wrote: > Hi Greg and Sasha, > > On Sat, 10 Jun 2023 17:56:18 +0000 SeongJae Park <sj@kernel.org> wrote: > > > On Sat, 10 Jun 2023 12:15:55 +0800 David Gow <davidgow@google.com> wrote: > > > > > [-- Attachment #1: Type: text/plain, Size: 2275 bytes --] > > > > > > On Sat, 10 Jun 2023 at 03:09, SeongJae Park <sj@kernel.org> wrote: > > > > > > > > Hi David and Brendan, > > > > > > > > On Tue, 2 May 2023 08:04:20 +0800 David Gow <davidgow@google.com> wrote: > > > > > > > > > [-- Attachment #1: Type: text/plain, Size: 1473 bytes --] > > > > > > > > > > On Tue, 2 May 2023 at 02:16, 'Daniel Latypov' via KUnit Development > > > > > <kunit-dev@googlegroups.com> wrote: > > > > > > > > > > > > Writing `subprocess.Popen[str]` requires python 3.9+. > > > > > > kunit.py has an assertion that the python version is 3.7+, so we should > > > > > > try to stay backwards compatible. > > > > > > > > > > > > This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix > > > > > > pre-existing `mypy --strict` errors and update run_checks.py"), since > > > > > > mypy complains like so > > > > > > > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] > > > > > > > > > > > > Note: `mypy --strict --python-version 3.7` does not work. > > > > > > > > > > > > We could annotate each file with comments like > > > > > > `# mypy: disable-error-code="type-arg" > > > > > > but then we might still get nudged to break back-compat in other files. > > > > > > > > > > > > This patch adds a `mypy.ini` file since it seems like the only way to > > > > > > disable specific error codes for all our files. > > > > > > > > > > > > Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, > > > > > > but I think being explicit is better, particularly since most kernel > > > > > > devs won't be familiar with how mypy works. > > > > > > > > > > > > Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") > > > > > > Reported-by: SeongJae Park <sj@kernel.org> > > > > > > Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org > > > > > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > > > > > --- > > > > > > > > > > Thanks for jumping on this. > > > > > > > > > > Looks good to me! > > > > > > > > > > Reviewed-by: David Gow <davidgow@google.com> > > > > > > > > Looks like this patch is still not merged in the mainline. May I ask the ETA, > > > > or any concern if you have? > > > > > > > > > > > > > > We've got this queued for 6.5 in the kselftest/kunit tree[1], so it > > > should land during the merge window. But I'll look into getting it > > > applied as a fix for 6.4, beforehand. > > > > Thank you for the kind answer, Gow! I was thinking this would be treated as a > > fix, and hence merged into the mainline before next merge window. I'm actually > > getting my personal test suite failures due to absence of this fix. It's not a > > critical problem, but it would definitely better for me if this could be merged > > into the mainline as early as possible. > > This patch is now in the mainline (e30f65c4b3d671115bf2a9d9ef142285387f2aff). > However, this fix is not in 6.4.y yet, so the original issue is reproducible on > 6.4.y. Could you please add this to 6.4.y? I confirmed the mainline commit > can cleanly applied on latest 6.1.y tree, and it fixes the issue. As this was not specifically tagged with a "cc: stable..." marking, that is why it was not picked up automatically. Also, we do not normally add patches to any stable releases until it is in a released kernel from Linus (i.e. a -rc release), unless you have a specific reason for it to be merged earlier. Should this be merged "now" into the stable trees and not wait for 6.5-rc1? thanks, greg k-h
On Mon, 3 Jul 2023 21:44:53 +0200 Greg KH <gregkh@linuxfoundation.org> wrote: > On Mon, Jul 03, 2023 at 07:27:04PM +0000, SeongJae Park wrote: [...] > > This patch is now in the mainline (e30f65c4b3d671115bf2a9d9ef142285387f2aff). > > However, this fix is not in 6.4.y yet, so the original issue is reproducible on > > 6.4.y. Could you please add this to 6.4.y? I confirmed the mainline commit > > can cleanly applied on latest 6.1.y tree, and it fixes the issue. > > As this was not specifically tagged with a "cc: stable..." marking, that > is why it was not picked up automatically. I understand that. I thought this would be merged into the mainline before v6.4 release, but I was wrong. > Also, we do not normally add patches to any stable releases until it is in a > released kernel from Linus (i.e. a -rc release), unless you have a specific > reason for it to be merged earlier. > > Should this be merged "now" into the stable trees and not wait for > 6.5-rc1? The problem is not urgent, so I can wait until 6.5-rc1. Thanks, SJ > > thanks, > > greg k-h
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index f01f94106129..7f648802caf6 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -92,7 +92,7 @@ class LinuxSourceTreeOperations: if stderr: # likely only due to build warnings print(stderr.decode()) - def start(self, params: List[str], build_dir: str) -> subprocess.Popen[str]: + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: raise RuntimeError('not implemented!') @@ -113,7 +113,7 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations): kconfig.merge_in_entries(base_kunitconfig) return kconfig - def start(self, params: List[str], build_dir: str) -> subprocess.Popen[str]: + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: kernel_path = os.path.join(build_dir, self._kernel_path) qemu_command = ['qemu-system-' + self._qemu_arch, '-nodefaults', @@ -142,7 +142,7 @@ class LinuxSourceTreeOperationsUml(LinuxSourceTreeOperations): kconfig.merge_in_entries(base_kunitconfig) return kconfig - def start(self, params: List[str], build_dir: str) -> subprocess.Popen[str]: + def start(self, params: List[str], build_dir: str) -> subprocess.Popen: """Runs the Linux UML binary. Must be named 'linux'.""" linux_bin = os.path.join(build_dir, 'linux') params.extend(['mem=1G', 'console=tty', 'kunit_shutdown=halt']) diff --git a/tools/testing/kunit/mypy.ini b/tools/testing/kunit/mypy.ini new file mode 100644 index 000000000000..ddd288309efa --- /dev/null +++ b/tools/testing/kunit/mypy.ini @@ -0,0 +1,6 @@ +[mypy] +strict = True + +# E.g. we can't write subprocess.Popen[str] until Python 3.9+. +# But kunit.py tries to support Python 3.7+, so let's disable it. +disable_error_code = type-arg diff --git a/tools/testing/kunit/run_checks.py b/tools/testing/kunit/run_checks.py index 8208c3b3135e..c6d494ea3373 100755 --- a/tools/testing/kunit/run_checks.py +++ b/tools/testing/kunit/run_checks.py @@ -23,7 +23,7 @@ commands: Dict[str, Sequence[str]] = { 'kunit_tool_test.py': ['./kunit_tool_test.py'], 'kunit smoke test': ['./kunit.py', 'run', '--kunitconfig=lib/kunit', '--build_dir=kunit_run_checks'], 'pytype': ['/bin/sh', '-c', 'pytype *.py'], - 'mypy': ['mypy', '--strict', '--exclude', '_test.py$', '--exclude', 'qemu_configs/', '.'], + 'mypy': ['mypy', '--config-file', 'mypy.ini', '--exclude', '_test.py$', '--exclude', 'qemu_configs/', '.'], } # The user might not have mypy or pytype installed, skip them if so.
Writing `subprocess.Popen[str]` requires python 3.9+. kunit.py has an assertion that the python version is 3.7+, so we should try to stay backwards compatible. This conflicts a bit with commit 1da2e6220e11 ("kunit: tool: fix pre-existing `mypy --strict` errors and update run_checks.py"), since mypy complains like so > kunit_kernel.py:95: error: Missing type parameters for generic type "Popen" [type-arg] Note: `mypy --strict --python-version 3.7` does not work. We could annotate each file with comments like `# mypy: disable-error-code="type-arg" but then we might still get nudged to break back-compat in other files. This patch adds a `mypy.ini` file since it seems like the only way to disable specific error codes for all our files. Note: run_checks.py doesn't need to specify `--config_file mypy.ini`, but I think being explicit is better, particularly since most kernel devs won't be familiar with how mypy works. Fixes: 695e26030858 ("kunit: tool: add subscripts for type annotations where appropriate") Reported-by: SeongJae Park <sj@kernel.org> Link: https://lore.kernel.org/linux-kselftest/20230501171520.138753-1-sj@kernel.org Signed-off-by: Daniel Latypov <dlatypov@google.com> --- tools/testing/kunit/kunit_kernel.py | 6 +++--- tools/testing/kunit/mypy.ini | 6 ++++++ tools/testing/kunit/run_checks.py | 2 +- 3 files changed, 10 insertions(+), 4 deletions(-) create mode 100644 tools/testing/kunit/mypy.ini base-commit: a42077b787680cbc365a96446b30f32399fa3f6f