From patchwork Fri May 13 15:52:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Adamson X-Patchwork-Id: 12849024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79D82C433F5 for ; Fri, 13 May 2022 15:53:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345140AbiEMPxQ (ORCPT ); Fri, 13 May 2022 11:53:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1380626AbiEMPxO (ORCPT ); Fri, 13 May 2022 11:53:14 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48AD920CA76 for ; Fri, 13 May 2022 08:53:13 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24DFdDOq005776; Fri, 13 May 2022 15:53:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2021-07-09; bh=dzDDomZrhd1Fm0N161zQGvrwgmKjqLx6L/iQYjbjKzg=; b=COhETa82qKBBZHgRGh96dQIpHC0M6cGcuOXKWJUUfitif9zYJgtH8VURBMZNW2+ozGqo Wj3ZFWcWAR7zxeQYl73yPOssaaraMjm1TTbRnWEqiLPhywWmGliNTw0I1HPOfbJZjhbi OiF0ig4dKNt96lH9UWpACfO1/Enj/cC+T0XBUJqt2N6/WQtkHicBfrLo/Pq8M0CPBXj9 qeHZ0X14nljyZw/YsK6SfCgNKmazjb7l7DSX+T1NMvwaYri0eGdrgvUs7NTtNwV0T8uq df8Hcjk5IqJMtfW9753yTMyF5RZTLzEifNrW2zOn+nFSuqYKP0YicS5nelwHIxTRXcp2 IA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fwhatr09c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 13 May 2022 15:53:07 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 24DFjZJL012167; Fri, 13 May 2022 15:53:07 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fwf76625r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 13 May 2022 15:53:07 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 24DFr1xQ034295; Fri, 13 May 2022 15:53:06 GMT Received: from dhcp-10-159-226-103.vpn.oracle.com (dhcp-10-159-226-103.vpn.oracle.com [10.159.226.103]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fwf766243-2; Fri, 13 May 2022 15:53:06 +0000 From: Alan Adamson To: linux-block@vger.kernel.org Cc: alan.adamson@oracle.com, linux-nvme@lists.infradead.org, osandov@fb.com Subject: [PATCH v3 blktests 1/2] tests/nvme: add helper routines to use error injector Date: Fri, 13 May 2022 08:52:51 -0700 Message-Id: <20220513155252.14332-2-alan.adamson@oracle.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220513155252.14332-1-alan.adamson@oracle.com> References: <20220513155252.14332-1-alan.adamson@oracle.com> MIME-Version: 1.0 X-Proofpoint-GUID: SudgCAGsnfO4Emzjuj_lh-2ZGFTWvXYi X-Proofpoint-ORIG-GUID: SudgCAGsnfO4Emzjuj_lh-2ZGFTWvXYi Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org nvme tests can use these helper routines to setup and use the nvme error injector. Signed-off-by: Alan Adamson --- tests/nvme/rc | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/tests/nvme/rc b/tests/nvme/rc index 3c38408a0bfe..c49a3c5d78da 100644 --- a/tests/nvme/rc +++ b/tests/nvme/rc @@ -375,3 +375,47 @@ _discovery_genctr() { _nvme_discover "${nvme_trtype}" | sed -n -e 's/^.*Generation counter \([0-9]\+\).*$/\1/p' } + +declare -A NS_DEV_FAULT_INJECT_SAVE +declare -A CTRL_DEV_FAULT_INJECT_SAVE + +_nvme_err_inject_setup() +{ + local a + + for a in /sys/kernel/debug/"$1"/fault_inject/*; do + NS_DEV_FAULT_INJECT_SAVE[${a}]=$(<"${a}") + done + + for a in /sys/kernel/debug/"$2"/fault_inject/*; do + CTRL_DEV_FAULT_INJECT_SAVE[${a}]=$(<"${a}") + done +} + +_nvme_err_inject_cleanup() +{ + local a + + for a in /sys/kernel/debug/"${ns_dev}"/fault_inject/*; do + echo "${NS_DEV_FAULT_INJECT_SAVE[${a}]}" > "${a}" + echo ${a} $(<"${a}") >> /tmp/iii + done + for a in /sys/kernel/debug/"${ctrl_dev}"/fault_inject/*; do + echo "${CTRL_DEV_FAULT_INJECT_SAVE[${a}]}" > "${a}" + done +} + +_nvme_enable_err_inject() +{ + echo "$2" > /sys/kernel/debug/"$1"/fault_inject/verbose + echo "$3" > /sys/kernel/debug/"$1"/fault_inject/probability + echo "$4" > /sys/kernel/debug/"$1"/fault_inject/dont_retry + echo "$5" > /sys/kernel/debug/"$1"/fault_inject/status + echo "$6" > /sys/kernel/debug/"$1"/fault_inject/times +} + +_nvme_clear_err_inject() +{ + echo 0 > /sys/kernel/debug/"$1"/fault_inject/probability + echo 0 > /sys/kernel/debug/"$1"/fault_inject/times +} From patchwork Fri May 13 15:52:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Adamson X-Patchwork-Id: 12849025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3555C4332F for ; Fri, 13 May 2022 15:53:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345493AbiEMPxR (ORCPT ); Fri, 13 May 2022 11:53:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382144AbiEMPxP (ORCPT ); Fri, 13 May 2022 11:53:15 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CD9B20D24D for ; Fri, 13 May 2022 08:53:14 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24DFdC7O005769; Fri, 13 May 2022 15:53:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2021-07-09; bh=+wn5OkM5Ai+3Pwa2ZmApyaSDNLntVr1QkElfVJ9EUuE=; b=KR0O2tvyB7lDBqxYj09Bd/cTUL8lOUhE2dGIoy/qRP7ddrthGsnEqo71D1ysgLqHO+3q PmlYBwnPa/iQOykVTZKmFb0KAT5DrATkT5kbEFBWlAI4HiSOJAVa6ZK6pCxRytaP545u MDV0EFg+fUKVAYtJwJLdaQWab4vbv9TjAzkW3+pLDmauh8FPxB1bHHUiIIMBWSg2yoKO tB2Px3+lIQH2XWXf57Zms+cK+GhBfVn1/BoLbcrx4+44qpR7gAFh6DGok+GqvsDaiUip 6ZQTfPYl16StMNOfowh6N00MLj/ePykXO29mjbHk/xjKiiJ7CD7G0qI/PzBbzcJkKZwL ag== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fwhatr09d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 13 May 2022 15:53:08 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 24DFjZHc012200; Fri, 13 May 2022 15:53:07 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fwf76626h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 13 May 2022 15:53:07 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 24DFr1xS034295; Fri, 13 May 2022 15:53:07 GMT Received: from dhcp-10-159-226-103.vpn.oracle.com (dhcp-10-159-226-103.vpn.oracle.com [10.159.226.103]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fwf766243-3; Fri, 13 May 2022 15:53:07 +0000 From: Alan Adamson To: linux-block@vger.kernel.org Cc: alan.adamson@oracle.com, linux-nvme@lists.infradead.org, osandov@fb.com Subject: [PATCH v3 blktests 2/2] tests/nvme: add tests for error logging Date: Fri, 13 May 2022 08:52:52 -0700 Message-Id: <20220513155252.14332-3-alan.adamson@oracle.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220513155252.14332-1-alan.adamson@oracle.com> References: <20220513155252.14332-1-alan.adamson@oracle.com> MIME-Version: 1.0 X-Proofpoint-GUID: o64VwStvWTXNi9ilZa1VG9iHxGbQC7ue X-Proofpoint-ORIG-GUID: o64VwStvWTXNi9ilZa1VG9iHxGbQC7ue Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Test nvme error logging by injecting errors. Kernel must have FAULT_INJECTION and FAULT_INJECTION_DEBUG_FS configured to use error injector. Tests can be run with or without NVME_VERBOSE_ERRORS configured. Signed-off-by: Alan Adamson Signed-off-by: Chaitanya Kulkarni --- tests/nvme/039 | 152 +++++++++++++++++++++++++++++++++++++++++++++ tests/nvme/039.out | 7 +++ 2 files changed, 159 insertions(+) create mode 100755 tests/nvme/039 create mode 100644 tests/nvme/039.out diff --git a/tests/nvme/039 b/tests/nvme/039 new file mode 100755 index 000000000000..dd216cbb2ef0 --- /dev/null +++ b/tests/nvme/039 @@ -0,0 +1,152 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-3.0+ +# Copyright (C) 2022 Oracle and/or its affiliates +# +# Test nvme error logging by injecting errors. Kernel must have FAULT_INJECTION +# and FAULT_INJECTION_DEBUG_FS configured to use error injector. Tests can be +# run with or without NVME_VERBOSE_ERRORS configured. +# +# Test for commit bd83fe6f2cd2 ("nvme: add verbose error logging"). + +. tests/nvme/rc +DESCRIPTION="test error logging" +QUICK=1 + +requires() { + _nvme_requires + _have_kernel_option FAULT_INJECTION && \ + _have_kernel_option FAULT_INJECTION_DEBUG_FS +} + +inject_unrec_read_read() +{ + # Inject a 'Unrecovered Read Error' (0x281) status error on a READ + _nvme_enable_err_inject "$1" 0 100 1 0x281 1 + + dd if=/dev/"$1" of=/dev/null bs=512 count=1 iflag=direct \ + 2> /dev/null 1>&2 + + _nvme_clear_err_inject "$1" + if ${nvme_verbose_errors}; then + dmesg -t | tail -2 | grep "Unrecovered Read Error (" | \ + sed 's/nvme.*://g' + else + dmesg -t | tail -2 | grep "Cmd(" | sed 's/I\/O Cmd/Read/g' | \ + sed 's/I\/O Error/Unrecovered Read Error/g' | \ + sed 's/nvme.*://g' + fi +} + +inject_invalid_status_read() +{ + # Inject an invalid status (0x375) on a READ + _nvme_enable_err_inject "$1" 0 100 1 0x375 1 + + dd if=/dev/"$1" of=/dev/null bs=512 count=1 iflag=direct \ + 2> /dev/null 1>&2 + + _nvme_clear_err_inject "$1" + + if ${nvme_verbose_errors}; then + dmesg -t | tail -2 | grep "Unknown (" | \ + sed 's/nvme.*://g' + else + dmesg -t | tail -2 | grep "Cmd(" | sed 's/I\/O Cmd/Read/g' | \ + sed 's/I\/O Error/Unknown/g' | \ + sed 's/nvme.*://g' + fi +} + +inject_write_fault_write() +{ + # Inject a 'Write Fault' 0x280 status error on a WRITE + _nvme_enable_err_inject "$1" 0 100 1 0x280 1 + + dd if=/dev/zero of=/dev/"$1" bs=512 count=1 oflag=direct \ + 2> /dev/null 1>&2 + + _nvme_clear_err_inject "$1" + + if ${nvme_verbose_errors}; then + dmesg -t | tail -2 | grep "Write Fault (" | \ + sed 's/nvme.*://g' + else + dmesg -t | tail -2 | grep "Cmd(" | sed 's/I\/O Cmd/Write/g' | \ + sed 's/I\/O Error/Write Fault/g' | \ + sed 's/nvme.*://g' + fi +} + +inject_access_denied_identify() +{ + # Inject a 'Access Denied' (0x286) status error on an + # Identify admin command + _nvme_enable_err_inject "$1" 0 100 1 0x286 1 + + nvme admin-passthru /dev/"$1" --opcode=0x06 --data-len=4096 \ + --cdw10=1 -r 2> /dev/null 1>&2 + + _nvme_clear_err_inject "$1" + + if ${nvme_verbose_errors}; then + dmesg -t | tail -1 | grep "Access Denied (" | \ + sed 's/nvme.*://g' + else + dmesg -t | tail -1 | grep "Admin Cmd(" | \ + sed 's/Admin Cmd/Identify/g' | \ + sed 's/I\/O Error/Access Denied/g' | \ + sed 's/nvme.*://g' + fi +} + +inject_invalid_cmd_admin() +{ + # Inject a 'Invalid Command Opcode' (0x1) on an invalid command (0x96) + _nvme_enable_err_inject "$1" 0 100 1 0x1 1 + + nvme admin-passthru /dev/"$1" --opcode=0x96 --data-len=4096 \ + --cdw10=1 -r 2> /dev/null 1>&2 + + _nvme_clear_err_inject "$1" + + if ${nvme_verbose_errors}; then + dmesg -t | tail -1 | grep "Invalid Command Opcode (" | \ + sed 's/nvme.*://g' + else + dmesg -t | tail -1 | grep "Admin Cmd(" | \ + sed 's/Admin Cmd/Unknown/g' | \ + sed 's/I\/O Error/Invalid Command Opcode/g' | \ + sed 's/nvme.*://g' + fi +} + +test_device() { + local nvme_verbose_errors + local ns_dev + local ctrl_dev + + echo "Running ${TEST_NAME}" + + if _have_kernel_option NVME_VERBOSE_ERRORS; then + nvme_verbose_errors=true + else + unset SKIP_REASON + nvme_verbose_errors=false + fi + + ns_dev=${TEST_DEV##*/} + ctrl_dev=${ns_dev%n*} + + _nvme_err_inject_setup "${ns_dev}" "${ctrl_dev}" + + inject_unrec_read_read "${ns_dev}" + inject_invalid_status_read "${ns_dev}" + inject_write_fault_write "${ns_dev}" + + inject_access_denied_identify "${ctrl_dev}" + inject_invalid_cmd_admin "${ctrl_dev}" + + _nvme_err_inject_cleanup "${ns_dev}" "${ctrl_dev}" + + echo "Test complete" +} diff --git a/tests/nvme/039.out b/tests/nvme/039.out new file mode 100644 index 000000000000..162935eb1d7b --- /dev/null +++ b/tests/nvme/039.out @@ -0,0 +1,7 @@ +Running nvme/039 + Read(0x2) @ LBA 0, 1 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) DNR + Read(0x2) @ LBA 0, 1 blocks, Unknown (sct 0x3 / sc 0x75) DNR + Write(0x1) @ LBA 0, 1 blocks, Write Fault (sct 0x2 / sc 0x80) DNR + Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR + Unknown(0x96), Invalid Command Opcode (sct 0x0 / sc 0x1) DNR +Test complete