| Message ID | YkImfPbNOzQBq5ZD@marsc.168.1.7 (mailing list archive) |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
| Series | Documentation: dev-tools: Add a section for static analysis tools | expand |
On 3/28/22 3:19 PM, Marcelo Schmitt wrote: > Complement the Kernel Testing Guide documentation page by adding a > section about static analysis tools. > > Signed-off-by: Marcelo Schmitt <marcelo.schmitt1@gmail.com> > --- > Hey everyone, > > I think this patch can be a good addition to the documentation as > discussed in the thread for the testing guide documentation page: > Link: https://lore.kernel.org/linux-doc/CABVgOS=2iYtqTVdxwH=mcFpcSuLP4cpJ4s6PKP4Gc-SH6jidgQ@mail.gmail.com/ > > If you think it would be worth it, I can try making something more > elaborated. Maybe provide some guidance on when to use each tool. > I've been studying how Linux device drivers are tested. > Here's a post I wrote talking about some testing tools. > Link: https://marcelosc.gitlab.io/how-is-linux-tested/ > > Best regards, > Marcelo > This a good addition to the testing overview doc. Please see a couple of comments below > Documentation/dev-tools/testing-overview.rst | 29 ++++++++++++++++++++ > 1 file changed, 29 insertions(+) > > diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst > index 65feb81edb14..b00511109a9d 100644 > --- a/Documentation/dev-tools/testing-overview.rst > +++ b/Documentation/dev-tools/testing-overview.rst > @@ -115,3 +115,32 @@ that none of these errors are occurring during the test. > Some of these tools integrate with KUnit or kselftest and will > automatically fail tests if an issue is detected. > > +Static Analysis Tools > +====================== > + > +In addition to testing a running kernel, one may also scout for bugs by > +analyzing the source code semantics. Three tools are well known for serving this > +purpose. > + > +Sparse can help test the kernel by performing type-checking, lock checking, > +value range checking, in addition to reporting various errors and warnings while > +examining the code. See the Documentation/dev-tools/sparse.rst documentation > +page for details on how to use it. > +> +Smatch extends Sparse and provides additional checks for programming logic > +mistakes such as missing breaks in switch statements, unused return values on > +error checking, forgetting to set an error code in the return of an error path, > +etc. Smatch also has tests against more serious issues such as integer > +overflows, null pointer dereferences, and memory leaks. See the project page at > +http://smatch.sourceforge.net/. > + > +We also have Coccinelle as an option within static analyzers. Coccinelle is > +often used to aid collateral evolution of source code, but it can also help to > +avoid certain bugs that have been expressed semantically. The types of tests > +available include API tests, tests for correct usage of kernel iterators, checks > +for the soundness of free operations, analysis of locking behavior, and further > +tests known to help keep consistent kernel usage. See the > +Documentation/dev-tools/coccinelle.rst documentation page for details. > + > +These static analysis tools support running tests on the whole source tree or > +over a specific file or directory. > You can also add a couple of sentences along the lines to clarify why static analysis is beneficial: "It is easier to detect and fix problems during the development process." Also add a word of caution: "Static analysis tools suffer from false positives and errors and warns need to be evaluated carefully before attempting to fix them." Reviewed-by: Shuah Khan <skhan@linuxfoundation.org> thanks, -- Shuah
On Mon, Mar 28, 2022 at 4:20 PM Marcelo Schmitt <marcelo.schmitt1@gmail.com> wrote: > > Complement the Kernel Testing Guide documentation page by adding a > section about static analysis tools. > > Signed-off-by: Marcelo Schmitt <marcelo.schmitt1@gmail.com> > --- > Hey everyone, > > I think this patch can be a good addition to the documentation as > discussed in the thread for the testing guide documentation page: > Link: https://lore.kernel.org/linux-doc/CABVgOS=2iYtqTVdxwH=mcFpcSuLP4cpJ4s6PKP4Gc-SH6jidgQ@mail.gmail.com/ > > If you think it would be worth it, I can try making something more Nice! This is definitely worth including, but I'm out of my depth here. Some ideas below in case they're helpful. Acked-by: Daniel Latypov <dlatypov@google.com> > elaborated. Maybe provide some guidance on when to use each tool. > I've been studying how Linux device drivers are tested. > Here's a post I wrote talking about some testing tools. > Link: https://marcelosc.gitlab.io/how-is-linux-tested/ > > Best regards, > Marcelo > > Documentation/dev-tools/testing-overview.rst | 29 ++++++++++++++++++++ > 1 file changed, 29 insertions(+) > > diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst > index 65feb81edb14..b00511109a9d 100644 > --- a/Documentation/dev-tools/testing-overview.rst > +++ b/Documentation/dev-tools/testing-overview.rst > @@ -115,3 +115,32 @@ that none of these errors are occurring during the test. > Some of these tools integrate with KUnit or kselftest and will > automatically fail tests if an issue is detected. > > +Static Analysis Tools > +====================== > + > +In addition to testing a running kernel, one may also scout for bugs by > +analyzing the source code semantics. Three tools are well known for serving this > +purpose. > + > +Sparse can help test the kernel by performing type-checking, lock checking, > +value range checking, in addition to reporting various errors and warnings while > +examining the code. See the Documentation/dev-tools/sparse.rst documentation > +page for details on how to use it. > + > +Smatch extends Sparse and provides additional checks for programming logic Coming from a place of ignorance, when should I use sparse vs smatch? Is there an existing consensus on this, or is that a controversial question? I assume other newcomers like me will have the same question and be unsure if they should spend time learning about sparse. In general, I think the main point of this page is to explain both what the tools do and give an idea of _why_ someone might want to consider using them. > +mistakes such as missing breaks in switch statements, unused return values on > +error checking, forgetting to set an error code in the return of an error path, > +etc. Smatch also has tests against more serious issues such as integer > +overflows, null pointer dereferences, and memory leaks. See the project page at > +http://smatch.sourceforge.net/. > + > +We also have Coccinelle as an option within static analyzers. Coccinelle is > +often used to aid collateral evolution of source code, but it can also help to > +avoid certain bugs that have been expressed semantically. The types of tests > +available include API tests, tests for correct usage of kernel iterators, checks > +for the soundness of free operations, analysis of locking behavior, and further > +tests known to help keep consistent kernel usage. See the > +Documentation/dev-tools/coccinelle.rst documentation page for details. > + > +These static analysis tools support running tests on the whole source tree or > +over a specific file or directory. Personal preference: I'd personally put this up before we go into details about each tool. E.g. perhaps like In addition to testing a running kernel, one can also analyze kernel source code directly (the whole tree or specific files) using **static analysis** tools. Some tools commonly used in the kernel are ... > -- > 2.35.1 >
On Tue, Mar 29, 2022 at 5:20 AM Marcelo Schmitt <marcelo.schmitt1@gmail.com> wrote: > > Complement the Kernel Testing Guide documentation page by adding a > section about static analysis tools. > > Signed-off-by: Marcelo Schmitt <marcelo.schmitt1@gmail.com> > --- > Hey everyone, > > I think this patch can be a good addition to the documentation as > discussed in the thread for the testing guide documentation page: > Link: https://lore.kernel.org/linux-doc/CABVgOS=2iYtqTVdxwH=mcFpcSuLP4cpJ4s6PKP4Gc-SH6jidgQ@mail.gmail.com/ > > If you think it would be worth it, I can try making something more > elaborated. Maybe provide some guidance on when to use each tool. > I've been studying how Linux device drivers are tested. > Here's a post I wrote talking about some testing tools. > Link: https://marcelosc.gitlab.io/how-is-linux-tested/ > > Best regards, > Marcelo > Thanks a lot for adding to this, and for writing that blog post, which I think is an excellent overview itself. I'd definitely like to see more of it work its way into this document. (While we've largely stuck to tools which are in-tree thus far, I think we'd definitely benefit from discussion of, for example, the different CI systems.) One thing which isn't totally clear is when to use one of these tools instead of another. This is a bit awkward, given that there is a bit more overlap (and the existing documentation is less clear), but comparing the "calling this specific kernel function incorrectly is causing this very specific bug" nature of Coccinelle with the "there's a bunch of typechecking so you can statically prove you didn't forget an endianness conversion somewhere" things sparse does. (Take those with a grain of salt, though, as I confess to not being an expert with any of these tools...) I've added a few other ideas inline, below, which you can take or leave as you see fit. Either way, I think this is a great improvement on not talking about static analysis tools at all, so thanks. Reviewed-by: David Gow <davidgow@google.com> Cheers, -- David > Documentation/dev-tools/testing-overview.rst | 29 ++++++++++++++++++++ > 1 file changed, 29 insertions(+) > > diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst > index 65feb81edb14..b00511109a9d 100644 > --- a/Documentation/dev-tools/testing-overview.rst > +++ b/Documentation/dev-tools/testing-overview.rst > @@ -115,3 +115,32 @@ that none of these errors are occurring during the test. > Some of these tools integrate with KUnit or kselftest and will > automatically fail tests if an issue is detected. > > +Static Analysis Tools > +====================== > + > +In addition to testing a running kernel, one may also scout for bugs by > +analyzing the source code semantics. Three tools are well known for serving this > +purpose. It might be worth being a bit more explicit that these run _at compile time_. Maybe this would also be the right place to document CONFIG_WERROR? Though that is slightly different yet again... > + > +Sparse can help test the kernel by performing type-checking, lock checking, > +value range checking, in addition to reporting various errors and warnings while > +examining the code. See the Documentation/dev-tools/sparse.rst documentation > +page for details on how to use it. It'd be nice to give an example of the extended type-checking here. e.g., endianness checks: https://lwn.net/Articles/205624/ > + > +Smatch extends Sparse and provides additional checks for programming logic > +mistakes such as missing breaks in switch statements, unused return values on > +error checking, forgetting to set an error code in the return of an error path, > +etc. Smatch also has tests against more serious issues such as integer > +overflows, null pointer dereferences, and memory leaks. See the project page at > +http://smatch.sourceforge.net/. > + > +We also have Coccinelle as an option within static analyzers. Coccinelle is Nit: _maybe_ it's worth simplifying this to just "Coccinelle is another static analyzer" or similar, so that all of these paragraphs start with the name of the tool being talked about. That makes it a bit easier to skim the page. > +often used to aid collateral evolution of source code, but it can also help to Nit: It's not entirely obvious what "collateral evolution of source code" is. Would "refactoring" be close enough? > +avoid certain bugs that have been expressed semantically. The types of tests > +available include API tests, tests for correct usage of kernel iterators, checks > +for the soundness of free operations, analysis of locking behavior, and further > +tests known to help keep consistent kernel usage. See the > +Documentation/dev-tools/coccinelle.rst documentation page for details. Maybe talk about how Coccinelle is matching specific (and often kernel-specific) patterns? > + > +These static analysis tools support running tests on the whole source tree or > +over a specific file or directory. > -- > 2.35.1 >
On Mon, 28 Mar 2022, Marcelo Schmitt wrote: > Complement the Kernel Testing Guide documentation page by adding a > section about static analysis tools. > > Signed-off-by: Marcelo Schmitt <marcelo.schmitt1@gmail.com> > --- > Hey everyone, > > I think this patch can be a good addition to the documentation as > discussed in the thread for the testing guide documentation page: > Link: https://lore.kernel.org/linux-doc/CABVgOS=2iYtqTVdxwH=mcFpcSuLP4cpJ4s6PKP4Gc-SH6jidgQ@mail.gmail.com/ > > If you think it would be worth it, I can try making something more > elaborated. Maybe provide some guidance on when to use each tool. > I've been studying how Linux device drivers are tested. > Here's a post I wrote talking about some testing tools. > Link: https://marcelosc.gitlab.io/how-is-linux-tested/ > > Best regards, > Marcelo > > Documentation/dev-tools/testing-overview.rst | 29 ++++++++++++++++++++ > 1 file changed, 29 insertions(+) > > diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst > index 65feb81edb14..b00511109a9d 100644 > --- a/Documentation/dev-tools/testing-overview.rst > +++ b/Documentation/dev-tools/testing-overview.rst > @@ -115,3 +115,32 @@ that none of these errors are occurring during the test. > Some of these tools integrate with KUnit or kselftest and will > automatically fail tests if an issue is detected. > > +Static Analysis Tools > +====================== > + > +In addition to testing a running kernel, one may also scout for bugs by > +analyzing the source code semantics. Three tools are well known for serving this > +purpose. > + > +Sparse can help test the kernel by performing type-checking, lock checking, > +value range checking, in addition to reporting various errors and warnings while > +examining the code. See the Documentation/dev-tools/sparse.rst documentation > +page for details on how to use it. > + > +Smatch extends Sparse and provides additional checks for programming logic > +mistakes such as missing breaks in switch statements, unused return values on > +error checking, forgetting to set an error code in the return of an error path, > +etc. Smatch also has tests against more serious issues such as integer > +overflows, null pointer dereferences, and memory leaks. See the project page at > +http://smatch.sourceforge.net/. > + > +We also have Coccinelle as an option within static analyzers. Coccinelle is > +often used to aid collateral evolution of source code, but it can also help to > +avoid certain bugs that have been expressed semantically. The types of tests "avoid certain bugs that have been expressed semantically" would be better as something like "avoid certain bugs that occur in common patterns" julia > +available include API tests, tests for correct usage of kernel iterators, checks > +for the soundness of free operations, analysis of locking behavior, and further > +tests known to help keep consistent kernel usage. See the > +Documentation/dev-tools/coccinelle.rst documentation page for details. > + > +These static analysis tools support running tests on the whole source tree or > +over a specific file or directory. > -- > 2.35.1 > >
On Mon, Mar 28, 2022 at 05:37:28PM -0500, Daniel Latypov wrote: > On Mon, Mar 28, 2022 at 4:20 PM Marcelo Schmitt > <marcelo.schmitt1@gmail.com> wrote: > > > > Complement the Kernel Testing Guide documentation page by adding a > > section about static analysis tools. > > > > Signed-off-by: Marcelo Schmitt <marcelo.schmitt1@gmail.com> > > > > --- > > Hey everyone, > > > > I think this patch can be a good addition to the documentation as > > discussed in the thread for the testing guide documentation page: > > Link: https://lore.kernel.org/linux-doc/CABVgOS=2iYtqTVdxwH=mcFpcSuLP4cpJ4s6PKP4Gc-SH6jidgQ@mail.gmail.com/ > > > > If you think it would be worth it, I can try making something more > > Nice! > This is definitely worth including, but I'm out of my depth here. > Some ideas below in case they're helpful. > > Acked-by: Daniel Latypov <dlatypov@google.com> > > > elaborated. Maybe provide some guidance on when to use each tool. > > I've been studying how Linux device drivers are tested. > > Here's a post I wrote talking about some testing tools. > > Link: https://marcelosc.gitlab.io/how-is-linux-tested/ > > > > Best regards, > > Marcelo > > > > Documentation/dev-tools/testing-overview.rst | 29 ++++++++++++++++++++ > > 1 file changed, 29 insertions(+) > > > > diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst > > index 65feb81edb14..b00511109a9d 100644 > > --- a/Documentation/dev-tools/testing-overview.rst > > +++ b/Documentation/dev-tools/testing-overview.rst > > @@ -115,3 +115,32 @@ that none of these errors are occurring during the test. > > Some of these tools integrate with KUnit or kselftest and will > > automatically fail tests if an issue is detected. > > > > +Static Analysis Tools > > +====================== > > + > > +In addition to testing a running kernel, one may also scout for bugs by > > +analyzing the source code semantics. Three tools are well known for serving this > > +purpose. > > + > > +Sparse can help test the kernel by performing type-checking, lock checking, > > +value range checking, in addition to reporting various errors and warnings while > > +examining the code. See the Documentation/dev-tools/sparse.rst documentation > > +page for details on how to use it. > > + > > +Smatch extends Sparse and provides additional checks for programming logic > > Coming from a place of ignorance, when should I use sparse vs smatch? > Is there an existing consensus on this, or is that a controversial question? I am the author of Smatch. Sparse is a compiler which can print static checker warnings. Smatch uses Sparse as a C front end. Sparse is useful for type checking, detecting places which use __user pointers improperly or endian bugs. Sparse is much faster than Smatch. Smatch does flow analysis and if you build the DB, then it does cross function analysis. Smatch tries to answer questions like where is this buffer allocated? How big is it? Can this index be controlled by the user? Is this variable larger than that variable? I feel like it's generally easier to write checks in Smatch than it is to write checks in Sparse. Coccinelle is probably the easiest for writing checks. It works before the pre-compiler so it easier to check for bugs in macros using Coccinelle. Coccinelle also writes patches fixes for you which no one else does. There are sometimes overlap in their checks. But I'm not going to re-implement Sparse's check in Smatch because Sparse already does that well. If it's just way easier to write the check with Coccinelle then do that. With Coccinelle you can do a mass conversion from kmalloc(x * size, GFP_KERNEL) to kmalloc_array(x, size, GFP_KERNEL); and that's really useful. If you just created a Smatch warning and try to push the work of converting on to the maintainers they would be annoyed. You'd have to argue about each warning if can really overflow or not. regards, dan carpenter
Acked-by: Dan Carpenter <dan.carpenter@oracle.com>
regards,
dan carpenter
On Tue, 29 Mar 2022, Dan Carpenter wrote: > On Mon, Mar 28, 2022 at 05:37:28PM -0500, Daniel Latypov wrote: > > On Mon, Mar 28, 2022 at 4:20 PM Marcelo Schmitt > > <marcelo.schmitt1@gmail.com> wrote: > > > > > > Complement the Kernel Testing Guide documentation page by adding a > > > section about static analysis tools. > > > > > > Signed-off-by: Marcelo Schmitt <marcelo.schmitt1@gmail.com> > > > > > > > --- > > > Hey everyone, > > > > > > I think this patch can be a good addition to the documentation as > > > discussed in the thread for the testing guide documentation page: > > > Link: https://lore.kernel.org/linux-doc/CABVgOS=2iYtqTVdxwH=mcFpcSuLP4cpJ4s6PKP4Gc-SH6jidgQ@mail.gmail.com/ > > > > > > If you think it would be worth it, I can try making something more > > > > Nice! > > This is definitely worth including, but I'm out of my depth here. > > Some ideas below in case they're helpful. > > > > Acked-by: Daniel Latypov <dlatypov@google.com> > > > > > elaborated. Maybe provide some guidance on when to use each tool. > > > I've been studying how Linux device drivers are tested. > > > Here's a post I wrote talking about some testing tools. > > > Link: https://marcelosc.gitlab.io/how-is-linux-tested/ > > > > > > Best regards, > > > Marcelo > > > > > > Documentation/dev-tools/testing-overview.rst | 29 ++++++++++++++++++++ > > > 1 file changed, 29 insertions(+) > > > > > > diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst > > > index 65feb81edb14..b00511109a9d 100644 > > > --- a/Documentation/dev-tools/testing-overview.rst > > > +++ b/Documentation/dev-tools/testing-overview.rst > > > @@ -115,3 +115,32 @@ that none of these errors are occurring during the test. > > > Some of these tools integrate with KUnit or kselftest and will > > > automatically fail tests if an issue is detected. > > > > > > +Static Analysis Tools > > > +====================== > > > + > > > +In addition to testing a running kernel, one may also scout for bugs by > > > +analyzing the source code semantics. Three tools are well known for serving this > > > +purpose. > > > + > > > +Sparse can help test the kernel by performing type-checking, lock checking, > > > +value range checking, in addition to reporting various errors and warnings while > > > +examining the code. See the Documentation/dev-tools/sparse.rst documentation > > > +page for details on how to use it. > > > + > > > +Smatch extends Sparse and provides additional checks for programming logic > > > > Coming from a place of ignorance, when should I use sparse vs smatch? > > Is there an existing consensus on this, or is that a controversial question? > > I am the author of Smatch. > > Sparse is a compiler which can print static checker warnings. Smatch > uses Sparse as a C front end. > > Sparse is useful for type checking, detecting places which use __user > pointers improperly or endian bugs. Sparse is much faster than Smatch. > > Smatch does flow analysis and if you build the DB, then it does > cross function analysis. Smatch tries to answer questions like where is > this buffer allocated? How big is it? Can this index be controlled by > the user? Is this variable larger than that variable? I feel like it's > generally easier to write checks in Smatch than it is to write checks in > Sparse. > > Coccinelle is probably the easiest for writing checks. It works before > the pre-compiler so it easier to check for bugs in macros using > Coccinelle. Coccinelle also writes patches fixes for you which no one > else does. > > There are sometimes overlap in their checks. But I'm not going to > re-implement Sparse's check in Smatch because Sparse already does that > well. If it's just way easier to write the check with Coccinelle then > do that. > > With Coccinelle you can do a mass conversion from > kmalloc(x * size, GFP_KERNEL) to kmalloc_array(x, size, GFP_KERNEL); and > that's really useful. If you just created a Smatch warning and try to > push the work of converting on to the maintainers they would be annoyed. > You'd have to argue about each warning if can really overflow or not. Coccinelle does no analysis of variable values, which is the strong point of smatch. On the other hand, as Dan points out, it allows you to do simple things in a simple way. julia
diff --git a/Documentation/dev-tools/testing-overview.rst b/Documentation/dev-tools/testing-overview.rst index 65feb81edb14..b00511109a9d 100644 --- a/Documentation/dev-tools/testing-overview.rst +++ b/Documentation/dev-tools/testing-overview.rst @@ -115,3 +115,32 @@ that none of these errors are occurring during the test. Some of these tools integrate with KUnit or kselftest and will automatically fail tests if an issue is detected. +Static Analysis Tools +====================== + +In addition to testing a running kernel, one may also scout for bugs by +analyzing the source code semantics. Three tools are well known for serving this +purpose. + +Sparse can help test the kernel by performing type-checking, lock checking, +value range checking, in addition to reporting various errors and warnings while +examining the code. See the Documentation/dev-tools/sparse.rst documentation +page for details on how to use it. + +Smatch extends Sparse and provides additional checks for programming logic +mistakes such as missing breaks in switch statements, unused return values on +error checking, forgetting to set an error code in the return of an error path, +etc. Smatch also has tests against more serious issues such as integer +overflows, null pointer dereferences, and memory leaks. See the project page at +http://smatch.sourceforge.net/. + +We also have Coccinelle as an option within static analyzers. Coccinelle is +often used to aid collateral evolution of source code, but it can also help to +avoid certain bugs that have been expressed semantically. The types of tests +available include API tests, tests for correct usage of kernel iterators, checks +for the soundness of free operations, analysis of locking behavior, and further +tests known to help keep consistent kernel usage. See the +Documentation/dev-tools/coccinelle.rst documentation page for details. + +These static analysis tools support running tests on the whole source tree or +over a specific file or directory.
Complement the Kernel Testing Guide documentation page by adding a section about static analysis tools. Signed-off-by: Marcelo Schmitt <marcelo.schmitt1@gmail.com> --- Hey everyone, I think this patch can be a good addition to the documentation as discussed in the thread for the testing guide documentation page: Link: https://lore.kernel.org/linux-doc/CABVgOS=2iYtqTVdxwH=mcFpcSuLP4cpJ4s6PKP4Gc-SH6jidgQ@mail.gmail.com/ If you think it would be worth it, I can try making something more elaborated. Maybe provide some guidance on when to use each tool. I've been studying how Linux device drivers are tested. Here's a post I wrote talking about some testing tools. Link: https://marcelosc.gitlab.io/how-is-linux-tested/ Best regards, Marcelo Documentation/dev-tools/testing-overview.rst | 29 ++++++++++++++++++++ 1 file changed, 29 insertions(+)