From patchwork Thu Apr 18 00:13:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luis Chamberlain X-Patchwork-Id: 13634024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECB1CC4345F for ; Thu, 18 Apr 2024 00:14:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D30A6B0087; Wed, 17 Apr 2024 20:14:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 683776B0088; Wed, 17 Apr 2024 20:14:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54A7D6B0089; Wed, 17 Apr 2024 20:14:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 35CFE6B0087 for ; Wed, 17 Apr 2024 20:14:04 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id EBA281A1286 for ; Thu, 18 Apr 2024 00:14:03 +0000 (UTC) X-FDA: 82020729966.03.8812C68 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf18.hostedemail.com (Postfix) with ESMTP id E590E1C000D for ; Thu, 18 Apr 2024 00:14:00 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=ecONMVNC; spf=none (imf18.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713399241; a=rsa-sha256; cv=none; b=wWH8KzXtQIkKv5N23b2baJtT38p/xNVOzz5Y+JH+sNdzgagCylm69dwcOQ/sZ2c3dMeazV fEjvlQDolAQ01oE/jL1phrx09Yy9wcIlGQiZXQp8czNkkMJ9is/IlhUKFQhkGAhTQ2hn9H 57qC6ZA11oImUfGhc/xQFE9t/EB7cgY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=ecONMVNC; spf=none (imf18.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713399241; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Fc9W3NDlawC6i8CUcs20Bxa//1KfA2ETQ6jFvKSlCKk=; b=79IQmuI7a0FEPuiVVEO/ThqjKkcaIlf3ynG/JICIiR1p+e5e95eXvWjjWEYn7r1r4WLUgi rM4NdI4796brtfH9HYXslMcNjgpQikTHCQ670jlid7HFLJbMNaTd5ywPlI8GaFHNNAIAJz fYzQ6bnmxpSoYnVeKlzTcaezWLS748A= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-Type: Content-ID:Content-Description:In-Reply-To:References; bh=Fc9W3NDlawC6i8CUcs20Bxa//1KfA2ETQ6jFvKSlCKk=; b=ecONMVNC6Mu9hrfLzfZ1uco50U N/UcJ7LR6OTvkC1P4rsF/uYOt1uKo9EIR89BANXn2fZpy3Q1FpB/qC8dXzHCb4MSRDv0io9cKgC4t uScHC6O6mXYXiF7OXUbeaI7YYeqclAwB5JbcA4FHUJ0ziWm6h5azTAiAvLkvx68RZMDCkVdkQBXVn FF1Nicxfk3c37vJTQIw0ftzmHP301rNhMysOhNeaP4xczcDA75CX0J5beFvgnzbfLuy3hmWxO5ssz XJWPKqqMLSjCFj10TybpdjLhnA7povBP64w680mUKnlFH0Blkom4sfPQPMueqUz+/73DPQxKJz7km fbbbgKeg==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxFPZ-00000000OwW-3Abv; Thu, 18 Apr 2024 00:13:57 +0000 From: Luis Chamberlain To: fstests@vger.kernel.org Cc: kdevops@lists.linux.dev, linux-xfs@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, willy@infradead.org, david@redhat.com, linmiaohe@huawei.com, muchun.song@linux.dev, osalvador@suse.de, Luis Chamberlain Subject: [PATCH] fstests: add fsstress + compaction test Date: Wed, 17 Apr 2024 17:13:56 -0700 Message-ID: <20240418001356.95857-1-mcgrof@kernel.org> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E590E1C000D X-Stat-Signature: bhnabnxpwgrajd7oqjo6tcgrny96npse X-Rspam-User: X-HE-Tag: 1713399240-462434 X-HE-Meta: U2FsdGVkX1/rHEUkIsdLZSws4DulUVsN+5S3CF8tfFr091MLowmrEyn7D42f9FDFcMyebsIw4Vt08AFMBBea1KCDSNzuepA8Y8/y2imQ5TGmY8PvqgrKff0U+365jfHt38G3cEjDeJf/GSIRmHImAVsqnucATHcOY6hwK3i4p9GkLu3NyMnMfAbSyywA4WLVZREeAYyvZfq1KIS2IUVUHGzuWo2xczs7FMJM2fRT35AFyI+xsFEVladi46keywi+ex3N/oPABDLgL9biIY/aJYoZ1++UAfehpmvO9btqxFlz7Df92rIjd1GJGXwgjoPmpr4Yf/bB4UHK+57XaQgYzf14SWkw/3Cr8wlzY0BqgqXEhyn2NdlRxgyUAP6fe09dZNLiJ4271oEO+4jQj04B3mzcrqendGgJfI5pBVky6O6tVddHgCJcY+f2LMdB9CLSnkp69F4ydBXpwslWLFmaTlG7ngNYiNYsh3xPRVIgxIDu6+FylEnF0bWhPzFkP+WoIyIvjBjgUTjtl7m0kmCCnHQliBHYev10GQeT6zqz6m0Mr3iNZuJvS/Vb1Rao+VF60kOgUtqAQtokGhSd1FKL4cmKS63GjyjJA6hUA4yVZGpJpX4SwCPrmOgn+98dpBEqc6jT2typhM37YlKYzksIlR1BPzmMx13rF7faPROEWlLkkh/XwbBH/YPdd55kFA5nUlNborg5/9EGM7McI8JSroq4Hai5QcMdn++ohW2g/eJxZuEl+phntmfiNVxCMOqnKJLXnPt+Uz/RdRl7deSNbDwb+WpevwWdLvYhjM25iRJUhBBrapGDAiuLeSmPu8+qNYfbS/tG8XW0dc9Q9FXYkrZOwymmYgvohEwbUxY6H7og8QIR3w4eO+lvR6J0kdO9ERTHa5mYHJT4VS0KuHoVJbmMNbkAmjmRg9Ic3IU1IApt/v+d7fermnny43H9vQ4H1FHqYC3UdYMebaNyyi0 2k6YCja1 aUEZ0uW96HTJnkAHm9A3V6+Kdj4fubdLNsWgshkz+CJNzbKpowjlMWU+o7oJL6UYoHs6QiMKdWhwM0vBXdxakfGNpk2pue50UKTUuuTunAxneiijNN8+B+zaIFYG0oQzPzc2ITBe86kjXPPxRkLEhi09bcgj/bAhuwIR8/Gqq5CsFda8q18ocSiyssufiMj+NiEixx0Bcng4ddaatF390qkv8Z110dwtJLhWmw4BStZGuzROUaPEOzncyg/JdqP62MskpLzcDtqFPhMlLuop07ISa+sTzmRK8suFkIazIOvc6XI/x0eGDEZV2mIFWJC7pMiUCs1MyuUaUVPoNBW/nI3orvF95HtWx1NMeFSzHxRWsem6lkQeDlOYH1GPr9EnfMiniB49KEGE2Nnt1lgZ8bXF8d8D/rZbpSFf2oBznRFSRZ/5JW4Fo/K1SaxLbBfH938kw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Running compaction while we run fsstress can crash older kernels as per korg#218227 [0], the fix for that [0] has been posted [1] but that patch is not yet on v6.9-rc4 and the patch requires changes for v6.9. Today I find that v6.9-rc4 is also hitting an unrecoverable hung task between compaction and fsstress while running generic/476 on the following kdevops test sections [2]: * xfs_nocrc * xfs_nocrc_2k * xfs_nocrc_4k Analyzing the trace I see the guest uses loopback block devices for the fstests TEST_DEV, the loopback file uses sparsefiles on a btrfs partition. The contention based on traces [3] [4] seems to be that we have somehow have fsstress + compaction race on folio_wait_bit_common(). We have this happening: a) kthread compaction --> migrate_pages_batch() --> folio_wait_bit_common() b) workqueue on btrfs writeback wb_workfn --> extent_write_cache_pages() --> folio_wait_bit_common() c) workqueue on loopback loop_rootcg_workfn() --> filemap_fdatawrite_wbc() --> folio_wait_bit_common() d) kthread xfsaild --> blk_mq_submit_bio() --> wbt_wait() I tried to reproduce but couldn't easily do so, so I wrote this test to help, and with this I have 100% failure rate so far out of 2 runs. Given we also have korg#218227 and that patch likely needing backporting, folks will want a reproducer for this issue. This should hopefully help with that case and this new separate issue. To reproduce with kdevops just: make defconfig-xfs_nocrc_2k -j $(nproc) make -j $(nproc) make fstests make linux make fstests-baseline TESTS=generic/733 tail -f guestfs/*-xfs-nocrc-2k/console.log [0] https://bugzilla.kernel.org/show_bug.cgi?id=218227 [1] https://lore.kernel.org/all/7ee2bb8c-441a-418b-ba3a-d305f69d31c8@suse.cz/T/#u [2] https://github.com/linux-kdevops/kdevops/blob/main/playbooks/roles/fstests/templates/xfs/xfs.config [3] https://gist.github.com/mcgrof/4dfa3264f513ce6ca398414326cfab84 [4] https://gist.github.com/mcgrof/f40a9f31a43793dac928ce287cfacfeb Signed-off-by: Luis Chamberlain --- Note: kdevops uses its own fork of fstests which has this merged already, so the above should just work. If it's your first time using kdevops be sure to just read the README for the first time users: https://github.com/linux-kdevops/kdevops/blob/main/docs/kdevops-first-run.md common/rc | 7 ++++++ tests/generic/744 | 56 +++++++++++++++++++++++++++++++++++++++++++ tests/generic/744.out | 2 ++ 3 files changed, 65 insertions(+) create mode 100755 tests/generic/744 create mode 100644 tests/generic/744.out diff --git a/common/rc b/common/rc index b7b77ac1b46d..d4432f5ce259 100644 --- a/common/rc +++ b/common/rc @@ -120,6 +120,13 @@ _require_hugepages() _notrun "Kernel does not report huge page size" } +# Requires CONFIG_COMPACTION +_require_compaction() +{ + if [ ! -f /proc/sys/vm/compact_memory ]; then + _notrun "Need compaction enabled CONFIG_COMPACTION=y" + fi +} # Get hugepagesize in bytes _get_hugepagesize() { diff --git a/tests/generic/744 b/tests/generic/744 new file mode 100755 index 000000000000..2b3c0c7e92fb --- /dev/null +++ b/tests/generic/744 @@ -0,0 +1,56 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Luis Chamberlain. All Rights Reserved. +# +# FS QA Test 744 +# +# fsstress + compaction test +# +. ./common/preamble +_begin_fstest auto rw long_rw stress soak smoketest + +_cleanup() +{ + cd / + rm -f $tmp.* + $KILLALL_PROG -9 fsstress > /dev/null 2>&1 +} + +# Import common functions. + +# real QA test starts here + +# Modify as appropriate. +_supported_fs generic + +_require_scratch +_require_compaction +_require_command "$KILLALL_PROG" "killall" + +echo "Silence is golden." + +_scratch_mkfs > $seqres.full 2>&1 +_scratch_mount >> $seqres.full 2>&1 + +nr_cpus=$((LOAD_FACTOR * 4)) +nr_ops=$((25000 * nr_cpus * TIME_FACTOR)) +fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus) + +# start a background getxattr loop for the existing xattr +runfile="$tmp.getfattr" +touch $runfile +while [ -e $runfile ]; do + echo 1 > /proc/sys/vm/compact_memory + sleep 15 +done & +getfattr_pid=$! + +test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION") + +$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full + +rm -f $runfile +wait > /dev/null 2>&1 + +status=0 +exit diff --git a/tests/generic/744.out b/tests/generic/744.out new file mode 100644 index 000000000000..205c684fa995 --- /dev/null +++ b/tests/generic/744.out @@ -0,0 +1,2 @@ +QA output created by 744 +Silence is golden