From patchwork Wed Mar 1 15:36:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13156144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5CD6C6FA9D for ; Wed, 1 Mar 2023 15:36:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DA2B6B0073; Wed, 1 Mar 2023 10:36:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 662BB6B0074; Wed, 1 Mar 2023 10:36:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DD676B0075; Wed, 1 Mar 2023 10:36:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 38C3A6B0073 for ; Wed, 1 Mar 2023 10:36:17 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 147FB1A08A6 for ; Wed, 1 Mar 2023 15:36:17 +0000 (UTC) X-FDA: 80520730794.23.83E930A Received: from out-21.mta0.migadu.com (out-21.mta0.migadu.com [91.218.175.21]) by imf21.hostedemail.com (Postfix) with ESMTP id 4E1A81C0027 for ; Wed, 1 Mar 2023 15:36:14 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="rhUgbe/z"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of andrey.konovalov@linux.dev designates 91.218.175.21 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677684974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=B997K50RBKGWXSrBUG5MoDdxxD/GnEcdYW4O1Qk10so=; b=8Nl7vcEsuJol69b2g0MiFzt/ZtaIyn/wK+qtVjTXQlsmSkgvb7CidJKqaKKWwXdlsysQMT cZ9fb55exJARvRNQQInQJ1uLK+VQCenZrUN7Kye9D+nCD6YNrPvT+Rjmsk/amBDxndpeuR DpHXI2gc+VQJhMp0F9+tuvzBLqKvN28= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="rhUgbe/z"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of andrey.konovalov@linux.dev designates 91.218.175.21 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677684974; a=rsa-sha256; cv=none; b=i2rqj+Qx8LqCs9A4L+gg+9xOvPeeN98kl80DmTg6hMuNWsi4bqk0fXiJuaVvPE52q1Mick 242JjPsbr0XbQdrKFoxd/5ZZZjqcoMGcPA004ui69Ggol1ZPBPgSDwhGBhnK7ELEiupM8q UgNNIu5PfbwR63rgtE+mIuQAThiyrmQ= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1677684971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=B997K50RBKGWXSrBUG5MoDdxxD/GnEcdYW4O1Qk10so=; b=rhUgbe/zMFtHzohaoUHjz00r58lwNDU8QUQcr8CBPdudv8AUSkZ4hGI7C9I77mRdSKGjq4 exXPFbuLZKakbjwqikFgfSVZWZ/dP6zRZNQ0wMVBwAAEGxnmUtRFxNPCp6JNPKwjL8r2cw FzNdpfR5PEz61bWQbZvvae3+9xbQKxk= From: andrey.konovalov@linux.dev To: Andrew Morton Cc: Andrey Konovalov , Dmitry Vyukov , Marco Elver , Alexander Potapenko , kasan-dev@googlegroups.com, linux-mm@kvack.org, Bagas Sanjaya , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v3] kcov: improve documentation Date: Wed, 1 Mar 2023 16:36:09 +0100 Message-Id: <72be5c215c275f35891229b90622ed859f196a46.1677684837.git.andreyknvl@google.com> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 4E1A81C0027 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: unjnwfnfpnu8z9osm3etimzhhso8q6jy X-HE-Tag: 1677684974-658937 X-HE-Meta: U2FsdGVkX19DVORPpYRUtSb9navfrJe7mn4XtZdRi1mjAQCzCQ8BbqAlBp88L+U3aj0GazaX3mdJzHe+ycPzt9T49UoemVhJXjiHKfriN2b4bQcRehI8F+AWMk7Efdj7DXyWouOEr06sqEKMOBI0zFFv9wn77T/ludpcDOnDl2DOVFpSmVIJkuA86m0dFhSyVroccbS9vhYxfq33pRX+wCp3zUsa+Cvgr6KxKUCnyUftgRCRscFujuLqgKw39ezxwNJ9Sjr6k2Y1S/sf+ZgFdrjhAawMxzh0EiLknmZtNX3nh2xPNpEOOC64GyKif+68ERc4ua/+YEMdgQPDHY1siTnjFalJ6juvJUuVB53s8Lu2kj4F8nx/XoxcF9NonCZXtJmdSzQM4UloZ6yreAv8Wnl+Qel8GvlXsAomdhFgQaa0OhrlXKD82nsfidEsv7G6TBLxaGjOdM8XX5OA4RoCgRpdXtizXipIS2ROiWLcCg8iZLFmWmciV+TdO/Rg3fzTSlgz3j+L8dWmPWzTqKiU2wUpXCrAQJXI9d4mRqDi5TcG0jlKdzUAcA37oX84xle/dAbrwgR5IwwwUIzJnxcgt5wJtwtUUMx6bzWY0g+KGIm1h+CJqDVTkdAzHpJMiWEQe2lleVhNYXUUaWfXFYdaIBal9WwEVUMSWxIo6/MRFcjMVwBOjeYdpKPvLiZnqF1bSk2DdP37TYctDe60xxN6cInCQPt2Mz9yeoE/od0MY377Lyc3Asi42tLN5JTH7QunZreoACRX7kSwDB07gExrcKGab+NwTlMbMqZk19hzADgeUtqWCxku10tlYZQ5CzAdx+1ugjToLJABzVk/OYqIfo/V7w740o0edE3vPCPhSLUN7iXKbPcPdWkxveKJSfqHlmCNgF10JcmMLg9dUcC9yVMU3qCs85ZJzq5Z4JHHbQB6wyAbQXPXTN/GHB3oJDgS0l9BpjJLhyFHu7wbD0Y CRCtTuGl mdIsT1ZLB8D0PWPOLWg+tKuVtH6sSmb7JjNXZTth/ISGs203lk2gIncktyjQpgdyaz0LF4SeEGV/54inOLkW4BV/SHz0LHL1RKfJY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Improve KCOV documentation: - Use KCOV instead of kcov, as the former is more widely-used. - Mention Clang in compiler requirements. - Use ``annotations`` for inline code. - Rework remote coverage collection documentation for better clarity. - Various smaller changes. Reviewed-by: Dmitry Vyukov Reviewed-by: Bagas Sanjaya Signed-off-by: Andrey Konovalov --- Changes v2->v3: - Fix ``annotation`` for KCOV_REMOTE_ENABLE. Changes v1->v2: - Add GCC version requirement for comparison operands collection. --- Documentation/dev-tools/kcov.rst | 169 +++++++++++++++++++------------ 1 file changed, 102 insertions(+), 67 deletions(-) diff --git a/Documentation/dev-tools/kcov.rst b/Documentation/dev-tools/kcov.rst index d83c9ab49427..6611434e2dd2 100644 --- a/Documentation/dev-tools/kcov.rst +++ b/Documentation/dev-tools/kcov.rst @@ -1,42 +1,50 @@ -kcov: code coverage for fuzzing +KCOV: code coverage for fuzzing =============================== -kcov exposes kernel code coverage information in a form suitable for coverage- -guided fuzzing (randomized testing). Coverage data of a running kernel is -exported via the "kcov" debugfs file. Coverage collection is enabled on a task -basis, and thus it can capture precise coverage of a single system call. +KCOV collects and exposes kernel code coverage information in a form suitable +for coverage-guided fuzzing. Coverage data of a running kernel is exported via +the ``kcov`` debugfs file. Coverage collection is enabled on a task basis, and +thus KCOV can capture precise coverage of a single system call. -Note that kcov does not aim to collect as much coverage as possible. It aims -to collect more or less stable coverage that is function of syscall inputs. -To achieve this goal it does not collect coverage in soft/hard interrupts -and instrumentation of some inherently non-deterministic parts of kernel is -disabled (e.g. scheduler, locking). +Note that KCOV does not aim to collect as much coverage as possible. It aims +to collect more or less stable coverage that is a function of syscall inputs. +To achieve this goal, it does not collect coverage in soft/hard interrupts +(unless remove coverage collection is enabled, see below) and from some +inherently non-deterministic parts of the kernel (e.g. scheduler, locking). -kcov is also able to collect comparison operands from the instrumented code -(this feature currently requires that the kernel is compiled with clang). +Besides collecting code coverage, KCOV can also collect comparison operands. +See the "Comparison operands collection" section for details. + +Besides collecting coverage data from syscall handlers, KCOV can also collect +coverage for annotated parts of the kernel executing in background kernel +tasks or soft interrupts. See the "Remote coverage collection" section for +details. Prerequisites ------------- -Configure the kernel with:: +KCOV relies on compiler instrumentation and requires GCC 6.1.0 or later +or any Clang version supported by the kernel. - CONFIG_KCOV=y +Collecting comparison operands is supported with GCC 8+ or with Clang. -CONFIG_KCOV requires gcc 6.1.0 or later. +To enable KCOV, configure the kernel with:: -If the comparison operands need to be collected, set:: + CONFIG_KCOV=y + +To enable comparison operands collection, set:: CONFIG_KCOV_ENABLE_COMPARISONS=y -Profiling data will only become accessible once debugfs has been mounted:: +Coverage data only becomes accessible once debugfs has been mounted:: mount -t debugfs none /sys/kernel/debug Coverage collection ------------------- -The following program demonstrates coverage collection from within a test -program using kcov: +The following program demonstrates how to use KCOV to collect coverage for a +single syscall from within a test program: .. code-block:: c @@ -84,7 +92,7 @@ program using kcov: perror("ioctl"), exit(1); /* Reset coverage from the tail of the ioctl() call. */ __atomic_store_n(&cover[0], 0, __ATOMIC_RELAXED); - /* That's the target syscal call. */ + /* Call the target syscall call. */ read(-1, NULL, 0); /* Read number of PCs collected. */ n = __atomic_load_n(&cover[0], __ATOMIC_RELAXED); @@ -103,7 +111,7 @@ program using kcov: return 0; } -After piping through addr2line output of the program looks as follows:: +After piping through ``addr2line`` the output of the program looks as follows:: SyS_read fs/read_write.c:562 @@ -121,12 +129,13 @@ After piping through addr2line output of the program looks as follows:: fs/read_write.c:562 If a program needs to collect coverage from several threads (independently), -it needs to open /sys/kernel/debug/kcov in each thread separately. +it needs to open ``/sys/kernel/debug/kcov`` in each thread separately. The interface is fine-grained to allow efficient forking of test processes. -That is, a parent process opens /sys/kernel/debug/kcov, enables trace mode, -mmaps coverage buffer and then forks child processes in a loop. Child processes -only need to enable coverage (disable happens automatically on thread end). +That is, a parent process opens ``/sys/kernel/debug/kcov``, enables trace mode, +mmaps coverage buffer, and then forks child processes in a loop. The child +processes only need to enable coverage (it gets disabled automatically when +a thread exits). Comparison operands collection ------------------------------ @@ -205,52 +214,78 @@ Comparison operands collection is similar to coverage collection: return 0; } -Note that the kcov modes (coverage collection or comparison operands) are -mutually exclusive. +Note that the KCOV modes (collection of code coverage or comparison operands) +are mutually exclusive. Remote coverage collection -------------------------- -With KCOV_ENABLE coverage is collected only for syscalls that are issued -from the current process. With KCOV_REMOTE_ENABLE it's possible to collect -coverage for arbitrary parts of the kernel code, provided that those parts -are annotated with kcov_remote_start()/kcov_remote_stop(). - -This allows to collect coverage from two types of kernel background -threads: the global ones, that are spawned during kernel boot in a limited -number of instances (e.g. one USB hub_event() worker thread is spawned per -USB HCD); and the local ones, that are spawned when a user interacts with -some kernel interface (e.g. vhost workers); as well as from soft -interrupts. - -To enable collecting coverage from a global background thread or from a -softirq, a unique global handle must be assigned and passed to the -corresponding kcov_remote_start() call. Then a userspace process can pass -a list of such handles to the KCOV_REMOTE_ENABLE ioctl in the handles -array field of the kcov_remote_arg struct. This will attach the used kcov -device to the code sections, that are referenced by those handles. - -Since there might be many local background threads spawned from different -userspace processes, we can't use a single global handle per annotation. -Instead, the userspace process passes a non-zero handle through the -common_handle field of the kcov_remote_arg struct. This common handle gets -saved to the kcov_handle field in the current task_struct and needs to be -passed to the newly spawned threads via custom annotations. Those threads -should in turn be annotated with kcov_remote_start()/kcov_remote_stop(). - -Internally kcov stores handles as u64 integers. The top byte of a handle -is used to denote the id of a subsystem that this handle belongs to, and -the lower 4 bytes are used to denote the id of a thread instance within -that subsystem. A reserved value 0 is used as a subsystem id for common -handles as they don't belong to a particular subsystem. The bytes 4-7 are -currently reserved and must be zero. In the future the number of bytes -used for the subsystem or handle ids might be increased. - -When a particular userspace process collects coverage via a common -handle, kcov will collect coverage for each code section that is annotated -to use the common handle obtained as kcov_handle from the current -task_struct. However non common handles allow to collect coverage -selectively from different subsystems. +Besides collecting coverage data from handlers of syscalls issued from a +userspace process, KCOV can also collect coverage for parts of the kernel +executing in other contexts - so-called "remote" coverage. + +Using KCOV to collect remote coverage requires: + +1. Modifying kernel code to annotate the code section from where coverage + should be collected with ``kcov_remote_start`` and ``kcov_remote_stop``. + +2. Using ``KCOV_REMOTE_ENABLE`` instead of ``KCOV_ENABLE`` in the userspace + process that collects coverage. + +Both ``kcov_remote_start`` and ``kcov_remote_stop`` annotations and the +``KCOV_REMOTE_ENABLE`` ioctl accept handles that identify particular coverage +collection sections. The way a handle is used depends on the context where the +matching code section executes. + +KCOV supports collecting remote coverage from the following contexts: + +1. Global kernel background tasks. These are the tasks that are spawned during + kernel boot in a limited number of instances (e.g. one USB ``hub_event`` + worker is spawned per one USB HCD). + +2. Local kernel background tasks. These are spawned when a userspace process + interacts with some kernel interface and are usually killed when the process + exits (e.g. vhost workers). + +3. Soft interrupts. + +For #1 and #3, a unique global handle must be chosen and passed to the +corresponding ``kcov_remote_start`` call. Then a userspace process must pass +this handle to ``KCOV_REMOTE_ENABLE`` in the ``handles`` array field of the +``kcov_remote_arg`` struct. This will attach the used KCOV device to the code +section referenced by this handle. Multiple global handles identifying +different code sections can be passed at once. + +For #2, the userspace process instead must pass a non-zero handle through the +``common_handle`` field of the ``kcov_remote_arg`` struct. This common handle +gets saved to the ``kcov_handle`` field in the current ``task_struct`` and +needs to be passed to the newly spawned local tasks via custom kernel code +modifications. Those tasks should in turn use the passed handle in their +``kcov_remote_start`` and ``kcov_remote_stop`` annotations. + +KCOV follows a predefined format for both global and common handles. Each +handle is a ``u64`` integer. Currently, only the one top and the lower 4 bytes +are used. Bytes 4-7 are reserved and must be zero. + +For global handles, the top byte of the handle denotes the id of a subsystem +this handle belongs to. For example, KCOV uses ``1`` as the USB subsystem id. +The lower 4 bytes of a global handle denote the id of a task instance within +that subsystem. For example, each ``hub_event`` worker uses the USB bus number +as the task instance id. + +For common handles, a reserved value ``0`` is used as a subsystem id, as such +handles don't belong to a particular subsystem. The lower 4 bytes of a common +handle identify a collective instance of all local tasks spawned by the +userspace process that passed a common handle to ``KCOV_REMOTE_ENABLE``. + +In practice, any value can be used for common handle instance id if coverage +is only collected from a single userspace process on the system. However, if +common handles are used by multiple processes, unique instance ids must be +used for each process. One option is to use the process id as the common +handle instance id. + +The following program demonstrates using KCOV to collect coverage from both +local tasks spawned by the process and the global task that handles USB bus #1: .. code-block:: c