From patchwork Tue Apr 4 23:16:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13201031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D51BC6FD1D for ; Tue, 4 Apr 2023 23:16:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235365AbjDDXQ6 (ORCPT ); Tue, 4 Apr 2023 19:16:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235754AbjDDXQ5 (ORCPT ); Tue, 4 Apr 2023 19:16:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4B0C1BB; Tue, 4 Apr 2023 16:16:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 521F56393D; Tue, 4 Apr 2023 23:16:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B7D38C433D2; Tue, 4 Apr 2023 23:16:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680650215; bh=j+qk8nPYdtuWY+Jy6NATbc9nnNe0iOt+89wCDUIWN5U=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=vKmMtF/8ZCXgM2Oi1Q2vcvVmjj8xnfsoqELkd8358dg7bpxJSscmi2AHkAW1oSfx8 FKeiYy3/JpXAhjdHv7MMeBmN/fMAHW8McpCOU+w754TxNd8fKJQygebG61xuZeDz6F t9E8+DDCD93Iw3jBfbexsUEzP4ep4+HH6qHXxIdcbQqOxGVi20zSPzM9He0wae65aW U3YvZx8Y4mQKt6PFe2fq650f4FdRXkT0ie2HvYtSdydTBNszCW25Djz3fLy8g8uU0l Pa3ijW7DFkNdwtx6wC9o2wYghvfIofB8NfsDXDNp3513SXkXGs2UoJLtRUeehV1EZF q7SfmWkf3jwPQ== Subject: [PATCH 1/3] common/populate: fix btree-format xattr creation on xfs From: "Darrick J. Wong" To: djwong@kernel.org, zlang@redhat.com Cc: linux-xfs@vger.kernel.org, fstests@vger.kernel.org, guan@eryu.me Date: Tue, 04 Apr 2023 16:16:55 -0700 Message-ID: <168065021529.494608.15409038797376185356.stgit@frogsfrogsfrogs> In-Reply-To: <168065020955.494608.9615705289123811403.stgit@frogsfrogsfrogs> References: <168065020955.494608.9615705289123811403.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org From: Darrick J. Wong Currently, we set a large number of extended attributes when trying to force the attr fork to be in BTREE format. This doesn't work reliably because userspace has no control over where xattr leaf and dabtree blocks are mapped, and contiguous mappings can prevent the file from having a btree format attr fork. However, we /do/ have one small knob for controlling attr fork mappings in the form of creating remote value xattrs and then deleting them to leave holes in the mappings. Create a separate helper function that exploits this property to try to create a sparse attr fork with enough mappings to give us the btree attr fork that we want. Signed-off-by: Darrick J. Wong --- common/populate | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/common/populate b/common/populate index b6f510f396..144a3f5186 100644 --- a/common/populate +++ b/common/populate @@ -155,6 +155,57 @@ __populate_create_attr() { done } +# Create an extended attr structure and ensure that the fork is btree format +__populate_xfs_create_btree_attr() { + local name="$1" + local isize="$2" + local dblksz="$3" + local icore_size="$(_xfs_get_inode_core_bytes $SCRATCH_MNT)" + # We need enough extents to guarantee that the attr fork is in btree + # format. Cycling the mount to use xfs_db is too slow, so watch for + # when the number of holes that we can punch in the attr fork by + # deleting remote xattrs exceeds the number of extent mappings that can + # fit in the inode core. + local max_nextents="$(((isize - icore_size) / 16))" + local nr + local i + local incr + local bigval + + # Add about one block's worth of attrs in betweeen creating punchable + # remote value blocks. + incr=$(( (dblksz / 16) / 100 * 100 )) + bigval="$(perl -e "print \"@\" x $dblksz;")" + + touch "${name}" + + # We cannot control the mapping behaviors of the attr fork leaf and + # dabtree blocks, but we do know that remote values are stored in a + # single extent, and that those mappings are removed if the xattr is + # deleted. + # + # The extended attribute structure tends to grow from offset zero + # upwards, so we try to set up a sparse attr fork mapping by + # iteratively creating at least one leaf block's worth of local attrs, + # and then one remote attr, until the number of remote xattrs exceeds + # the number of mappings that fit in the inode core... + for ((nr = 0; nr < (incr * max_nextents); nr += incr)); do + # Simulate a getfattr dump file so we can bulk-add attrs. + ( + echo "# file: ${name}"; + seq --format "user.%08g=\"abcdefgh\"" "${nr}" "$((nr + incr + 1))" + echo "user.v$(printf "%.08d" "$nr")=\"${bigval}\"" + echo + ) | setfattr --restore - + done + + # ... and in the second loop we delete all the remote attrs to + # fragment the attr fork mappings. + for ((i = 0; i < nr; i += incr)); do + setfattr -x "user.v$(printf "%.08d" "$i")" "${name}" + done +} + # Fill up some percentage of the remaining free space __populate_fill_fs() { dir="$1" @@ -327,7 +378,7 @@ _scratch_xfs_populate() { # BTREE echo "+ btree attr" - __populate_create_attr "${SCRATCH_MNT}/ATTR.FMT_BTREE" "$((64 * blksz / 40))" true + __populate_xfs_create_btree_attr "${SCRATCH_MNT}/ATTR.FMT_BTREE" "$isize" "$dblksz" # trusted namespace touch ${SCRATCH_MNT}/ATTR.TRUSTED From patchwork Tue Apr 4 23:17:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13201032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32102C6FD1D for ; Tue, 4 Apr 2023 23:17:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236064AbjDDXRE (ORCPT ); Tue, 4 Apr 2023 19:17:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235754AbjDDXRD (ORCPT ); Tue, 4 Apr 2023 19:17:03 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72C384200; Tue, 4 Apr 2023 16:17:02 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0E90B639B7; Tue, 4 Apr 2023 23:17:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 73E37C433D2; Tue, 4 Apr 2023 23:17:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680650221; bh=/nm612KZkoTN/JgnmYvXu87DPGq5WXcxJSyNQ90IAUU=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=mqu6ailpNyrIHStIl1xj+Lnd4Zjnoy4mKSyQCXTJ1GPUPDdCaa6Lww/CEsEhuloKf AIazlYdnOgHpbHeUCnZyOMvgKoPRsq/2XMyABkpYNcdjmgGJQi87NyOqbGKCKTI7oa 4lxWX7fKKLnw1qy7KIiE1/MWk66gvRCpWDmXY10P8Wps5QsK7WbaPyShgc6kLaOtMI 5TiQw6tOCh3iHgGChXcIVUFUMMPz/uMf3fO3k5nt5z8MuV+oBze1W/6UeqLsxXBHmk pIfClFt5OBpqgL/ryHa0nu2rHkN7rleEj5p7jGbSLZTtkKkST0U54SdrQHcdGwuQNm NfUyYdNPMYl8Q== Subject: [PATCH 2/3] populate: create fewer subdirs when constructing directories From: "Darrick J. Wong" To: djwong@kernel.org, zlang@redhat.com Cc: linux-xfs@vger.kernel.org, fstests@vger.kernel.org, guan@eryu.me Date: Tue, 04 Apr 2023 16:17:00 -0700 Message-ID: <168065022088.494608.9054319548766410710.stgit@frogsfrogsfrogs> In-Reply-To: <168065020955.494608.9615705289123811403.stgit@frogsfrogsfrogs> References: <168065020955.494608.9615705289123811403.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org From: Darrick J. Wong Based on some surveys of aged filesystems, I've noticed that the proportion of directory children that are subdirectories tends to be more in the 5-10% range, not the 95% that the current code generates. Rework popdir.pl so that we can specify arbitrary percentages of children files, and lower the ratio dramatically. This shouldn't have any substantive changes in the shape of the directories that gets generated; it just gets us a more realistic sample. Signed-off-by: Darrick J. Wong --- common/populate | 4 ++-- src/popdir.pl | 15 ++++++++++----- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/common/populate b/common/populate index 144a3f5186..524e0c32cb 100644 --- a/common/populate +++ b/common/populate @@ -308,7 +308,7 @@ _scratch_xfs_populate() { # Fill up the root inode chunk echo "+ fill root ino chunk" - $here/src/popdir.pl --dir "${SCRATCH_MNT}" --start 1 --end 64 --format "dummy%u" --file-mult 1 + $here/src/popdir.pl --dir "${SCRATCH_MNT}" --start 1 --end 64 --format "dummy%u" --file-pct 100 # Regular files # - FMT_EXTENTS @@ -422,7 +422,7 @@ _scratch_xfs_populate() { local rec_per_btblock=16 local nr="$(( 2 * (blksz / rec_per_btblock) * ino_per_rec ))" local dir="${SCRATCH_MNT}/INOBT" - __populate_create_dir "${dir}" "${nr}" true --file-mult 1 + __populate_create_dir "${dir}" "${nr}" true --file-pct 100 # Reverse-mapping btree is_rmapbt="$(_xfs_has_feature "$SCRATCH_MNT" rmapbt -v)" diff --git a/src/popdir.pl b/src/popdir.pl index dc0c046b7d..e89095aafe 100755 --- a/src/popdir.pl +++ b/src/popdir.pl @@ -11,7 +11,7 @@ use File::Basename; $progname=$0; GetOptions("start=i" => \$start, "end=i" => \$end, - "file-mult=i" => \$file_mult, + "file-pct=i" => \$file_pct, "incr=i" => \$incr, "format=s" => \$format, "dir=s" => \$dir, @@ -30,8 +30,7 @@ Options: --start=num create names starting with this number (0) --incr=num increment file number by this much (1) --end=num stop at this file number (100) - --file-mult create a regular file when file number is a multiple - of this quantity (20) + --file-pct create this percentage of regular files (90 percent) --remove remove instead of creating --format=str printf formatting string for file name ("%08d") --verbose verbose output @@ -47,17 +46,23 @@ if (defined $dir) { } $start = 0 if (!defined $start); $end = 100 if (!defined $end); -$file_mult = 20 if (!defined $file_mult); +$file_pct = 90 if (!defined $file_pct); $format = "%08d" if (!defined $format); $incr = 1 if (!defined $incr); +if ($file_pct < 0) { + $file_pct = 0; +} elsif ($file_pct > 100) { + $file_pct = 100; +} + for ($i = $start; $i <= $end; $i += $incr) { $fname = sprintf($format, $i); if ($remove) { $verbose && print "rm $fname\n"; unlink($fname) or rmdir($fname) or die("unlink $fname"); - } elsif ($file_mult == 0 or ($i % $file_mult) == 0) { + } elsif (($i % 100) < $file_pct) { # create a file $verbose && print "touch $fname\n"; open(DONTCARE, ">$fname") or die("touch $fname"); From patchwork Tue Apr 4 23:17:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13201033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1038BC76188 for ; Tue, 4 Apr 2023 23:17:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235754AbjDDXRK (ORCPT ); Tue, 4 Apr 2023 19:17:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230455AbjDDXRJ (ORCPT ); Tue, 4 Apr 2023 19:17:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B2F44200; Tue, 4 Apr 2023 16:17:08 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id ABD92638ED; Tue, 4 Apr 2023 23:17:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 13708C433EF; Tue, 4 Apr 2023 23:17:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680650227; bh=/nKlwZxCQrgiPCN8nqKQXsGE3xozZu0KoZoBHX5uLPA=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=Sfnf4YXnEcdDoh12PvJQglKVflrbgVDMBI9HEQTiNcN/85YHNKs9rc760YOancy5o 7eXJWZDqsFPMySBrDJCtftfSyLDuuRHiCWWVRljgmD1TrXbvo2JdUOG/GnABxDpsVi wcVulm92IwaHNWVmGaGslMSnk3nUT9U7SrRSS+KBbCZaiR5EpRZWYbxj62Jr1mqRLg PsDJrrkBqbvOMgB9IdFQACqBnBMHMhLmPa1fVTa8H/NirwwzREpkEEtbKml/Tu0YKr rcVRxsJIuetEVr/AOwVOZOiONKSGsbIuc+JRbq3brIx88s5XDbsUzaCURyJKEdv8fm rIaR8jO5/2Efw== Subject: [PATCH 3/3] populate: create fewer holes in directories and xattrs From: "Darrick J. Wong" To: djwong@kernel.org, zlang@redhat.com Cc: linux-xfs@vger.kernel.org, fstests@vger.kernel.org, guan@eryu.me Date: Tue, 04 Apr 2023 16:17:06 -0700 Message-ID: <168065022660.494608.3347185703277760781.stgit@frogsfrogsfrogs> In-Reply-To: <168065020955.494608.9615705289123811403.stgit@frogsfrogsfrogs> References: <168065020955.494608.9615705289123811403.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: fstests@vger.kernel.org From: Darrick J. Wong The same study of aging filesystem metadumps also showed that the rate of file and xattr deletion is a lot lower than what this script does. Decrease the deletion interval from every other file/name to every 19th item to get us to a more reasonable 5% deletion rate, but with a prime number. ;) Signed-off-by: Darrick J. Wong --- common/populate | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/common/populate b/common/populate index 524e0c32cb..3d233073c9 100644 --- a/common/populate +++ b/common/populate @@ -89,7 +89,7 @@ __populate_create_dir() { $here/src/popdir.pl --dir "${name}" --end "${nr}" "$@" test -z "${missing}" && return - $here/src/popdir.pl --dir "${name}" --start 1 --incr 2 --end "${nr}" --remove "$@" + $here/src/popdir.pl --dir "${name}" --start 1 --incr 19 --end "${nr}" --remove "$@" } # Create a large directory and ensure that it's a btree format @@ -123,7 +123,7 @@ __populate_xfs_create_btree_dir() { done test -z "${missing}" && return - $here/src/popdir.pl --dir "${name}" --start 1 --incr 2 --end "${nr}" --remove + $here/src/popdir.pl --dir "${name}" --start 1 --incr 19 --end "${nr}" --remove } # Add a bunch of attrs to a file @@ -138,7 +138,7 @@ __populate_create_attr() { ${PYTHON3_PROG} $here/src/popattr.py --file "${name}" --end "${nr}" test -z "${missing}" && return - ${PYTHON3_PROG} $here/src/popattr.py --file "${name}" --start 1 --incr 2 --end "${nr}" --remove + ${PYTHON3_PROG} $here/src/popattr.py --file "${name}" --start 1 --incr 19 --end "${nr}" --remove return fi